Xen ARM community call - meeting minutes and date for the next one

Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

Hi all,

Apologies for the late sending, you will find at the end of the e-mail a
summary of the discussion from the previous call. Feel free to reply if I
missed some parts.

I suggest to have the next call on the 5th April at 5PM UTC. Any opinions?

Also do you have any specific topic you would like to talk during this call?

Cheers,

== Attendees ==

Apologies if I misspelled any name.

Stefano, Aporeto
Julien, ARM
Oleksandr, EPAM
Artem, EPAM
Thanasis, OnApp
Volodymir, EPAM

== Xen on ARM status ==

Over 100 patches in-flight for Xen on ARM:
    - PV protocols: Some are already accepted
    - NUMA support
    - GICv3 ITS support
    - Exposing and emulating a PL011 for guest
    - guest SMC forwarding for Xilinx platform
    - Interrupt latency improvement

== PV protocols ==

* PV protocols written by Stefano was merged after 10 months

Stefano: PV protocols review are moving faster
Attendees agreed

* Audio protocol: close to be accepted
* Display protocol: minor issue, a bit more design is required

Hopefully both will be ready for Xen 4.9

Oleksandr: What to do when the backend die?

(I cannot find any notes on it some I am not sure if we answered the question
during the call. I suspect it has been asked to bring up the subject on the ML).

== Interrupt latency ==

Stefano: Some improvement has been done but it is not possible to know whether
it is good. Do you have any specific IRQ latency requirements?

Artem: There is no hard latency requirements in automotive, although many
requirements depends on latency. For example:
    * Scheduling
    * GPU (implementation is sentive to interrupt latency)

Automotive is using a set of benchmark to find the virtualization overhead. This
should be low.

ACTION: Artem to send a list of benchmark

== SMC/HVC handling in Xen ==

Artem: Please review the proposal on the mailing list. See:

https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg00430.html

== Deprivilege mode ==

EPAM are working on adding support for OP-TEE in Xen to allow multiple guest
access the trusted firmware.

During the discussion on the design, it was suggested to move the SMC handling
in a separate domain. This was tested using the VM event API and Mini-OS
(upstream with Chen Baozi's series to support ARM64). The first results
shows it is 10 times slower than handling SMC calls directly in the hypervisor.

Volodymir is working on another approach to deprivilege the execution by
implementing a Xen EL0.

== big.LITTLE support ==

Thanasis: Document discussed on the ML. Xen will split CPUs at boot time
(big vs little). A series will be sent on the on the ML soon.

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Stefano Stabellini]

On Tue, 28 Mar 2017, Julien Grall wrote:
> Hi all,
> 
> Apologies for the late sending, you will find at the end of the e-mail a
> summary of the discussion from the previous call. Feel free to reply if I
> missed some parts.
> 
> I suggest to have the next call on the 5th April at 5PM UTC. Any opinions?

Works for me.


> Also do you have any specific topic you would like to talk during this call?
> 
> Cheers,
> 
> == Attendees ==
> 
> Apologies if I misspelled any name.
> 
> Stefano, Aporeto
> Julien, ARM
> Oleksandr, EPAM
> Artem, EPAM
> Thanasis, OnApp
> Volodymir, EPAM
> 
> == Xen on ARM status ==
> 
> Over 100 patches in-flight for Xen on ARM:
>     - PV protocols: Some are already accepted
>     - NUMA support
>     - GICv3 ITS support
>     - Exposing and emulating a PL011 for guest
>     - guest SMC forwarding for Xilinx platform
>     - Interrupt latency improvement
> 
> == PV protocols ==
> 
> * PV protocols written by Stefano was merged after 10 months
> 
> Stefano: PV protocols review are moving faster
> Attendees agreed
> 
> * Audio protocol: close to be accepted
> * Display protocol: minor issue, a bit more design is required
> 
> Hopefully both will be ready for Xen 4.9
> 
> Oleksandr: What to do when the backend die?
> 
> (I cannot find any notes on it some I am not sure if we answered the question
> during the call. I suspect it has been asked to bring up the subject on the ML).
> 
> == Interrupt latency ==
> 
> Stefano: Some improvement has been done but it is not possible to know whether
> it is good. Do you have any specific IRQ latency requirements?
> 
> Artem: There is no hard latency requirements in automotive, although many
> requirements depends on latency. For example:
>     * Scheduling
>     * GPU (implementation is sentive to interrupt latency)
> 
> Automotive is using a set of benchmark to find the virtualization overhead. This
> should be low.
> 
> ACTION: Artem to send a list of benchmark
> 
> == SMC/HVC handling in Xen ==
> 
> Artem: Please review the proposal on the mailing list. See:
> 
> https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg00430.html
> 
> == Deprivilege mode ==
> 
> EPAM are working on adding support for OP-TEE in Xen to allow multiple guest
> access the trusted firmware.
> 
> During the discussion on the design, it was suggested to move the SMC handling
> in a separate domain. This was tested using the VM event API and Mini-OS
> (upstream with Chen Baozi's series to support ARM64). The first results
> shows it is 10 times slower than handling SMC calls directly in the hypervisor.
> 
> Volodymir is working on another approach to deprivilege the execution by
> implementing a Xen EL0.
> 
> == big.LITTLE support ==
> 
> Thanasis: Document discussed on the ML. Xen will split CPUs at boot time
> (big vs little). A series will be sent on the on the ML soon.
> 
> -- 
> Julien Grall
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Artem Mygaiev]

[-- Attachment #1.1: Type: text/plain, Size: 4065 bytes --]

Hello Julien

5th works for us. We will collect topics internally and send to this thread.

ARTEM MYGAIEV 
Director, Technology Solutions
 
Office: +380 44 390 5457 x 65570   Cell: +380 67 921 1131   Email: artem_mygaiev@epam.com 
Kyiv, Ukraine (GMT+3)   epam.com 
 
CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies. 
 

-----Original Message-----
From: Julien Grall [mailto:julien.grall@arm.com] 
Sent: 28 березня 2017 р. 18:23
To: xen-devel <xen-devel@lists.xenproject.org>
Cc: Edgar E. Iglesias <edgar.iglesias@xilinx.com>; lars.kurth@citrix.com; Stefano Stabellini <sstabellini@kernel.org>; Artem Mygaiev <Artem_Mygaiev@epam.com>; Volodymyr Babchuk <vlad.babchuk@gmail.com>; anastassios.nanos@onapp.com; thkatsios@cslab.ece.ntua.gr
Subject: Xen ARM community call - meeting minutes and date for the next one

Hi all,

Apologies for the late sending, you will find at the end of the e-mail a summary of the discussion from the previous call. Feel free to reply if I missed some parts.

I suggest to have the next call on the 5th April at 5PM UTC. Any opinions?

Also do you have any specific topic you would like to talk during this call?

Cheers,

== Attendees ==

Apologies if I misspelled any name.

Stefano, Aporeto
Julien, ARM
Oleksandr, EPAM
Artem, EPAM
Thanasis, OnApp
Volodymir, EPAM

== Xen on ARM status ==

Over 100 patches in-flight for Xen on ARM:
    - PV protocols: Some are already accepted
    - NUMA support
    - GICv3 ITS support
    - Exposing and emulating a PL011 for guest
    - guest SMC forwarding for Xilinx platform
    - Interrupt latency improvement

== PV protocols ==

* PV protocols written by Stefano was merged after 10 months

Stefano: PV protocols review are moving faster Attendees agreed

* Audio protocol: close to be accepted
* Display protocol: minor issue, a bit more design is required

Hopefully both will be ready for Xen 4.9

Oleksandr: What to do when the backend die?

(I cannot find any notes on it some I am not sure if we answered the question during the call. I suspect it has been asked to bring up the subject on the ML).

== Interrupt latency ==

Stefano: Some improvement has been done but it is not possible to know whether it is good. Do you have any specific IRQ latency requirements?

Artem: There is no hard latency requirements in automotive, although many requirements depends on latency. For example:
    * Scheduling
    * GPU (implementation is sentive to interrupt latency)

Automotive is using a set of benchmark to find the virtualization overhead. This should be low.

ACTION: Artem to send a list of benchmark

== SMC/HVC handling in Xen ==

Artem: Please review the proposal on the mailing list. See:

https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg00430.html

== Deprivilege mode ==

EPAM are working on adding support for OP-TEE in Xen to allow multiple guest access the trusted firmware.

During the discussion on the design, it was suggested to move the SMC handling in a separate domain. This was tested using the VM event API and Mini-OS (upstream with Chen Baozi's series to support ARM64). The first results shows it is 10 times slower than handling SMC calls directly in the hypervisor.

Volodymir is working on another approach to deprivilege the execution by implementing a Xen EL0.

== big.LITTLE support ==

Thanasis: Document discussed on the ML. Xen will split CPUs at boot time (big vs little). A series will be sent on the on the ML soon.

--
Julien Grall

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 4640 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Edgar E. Iglesias]

On Tue, Mar 28, 2017 at 04:23:03PM +0100, Julien Grall wrote:
> Hi all,
> 
> Apologies for the late sending, you will find at the end of the e-mail a
> summary of the discussion from the previous call. Feel free to reply if I
> missed some parts.
> 
> I suggest to have the next call on the 5th April at 5PM UTC. Any opinions?


That works for me, thanks!

Edgar


> 
> Also do you have any specific topic you would like to talk during this call?
> 
> Cheers,
> 
> == Attendees ==
> 
> Apologies if I misspelled any name.
> 
> Stefano, Aporeto
> Julien, ARM
> Oleksandr, EPAM
> Artem, EPAM
> Thanasis, OnApp
> Volodymir, EPAM
> 
> == Xen on ARM status ==
> 
> Over 100 patches in-flight for Xen on ARM:
>     - PV protocols: Some are already accepted
>     - NUMA support
>     - GICv3 ITS support
>     - Exposing and emulating a PL011 for guest
>     - guest SMC forwarding for Xilinx platform
>     - Interrupt latency improvement
> 
> == PV protocols ==
> 
> * PV protocols written by Stefano was merged after 10 months
> 
> Stefano: PV protocols review are moving faster
> Attendees agreed
> 
> * Audio protocol: close to be accepted
> * Display protocol: minor issue, a bit more design is required
> 
> Hopefully both will be ready for Xen 4.9
> 
> Oleksandr: What to do when the backend die?
> 
> (I cannot find any notes on it some I am not sure if we answered the question
> during the call. I suspect it has been asked to bring up the subject on the ML).
> 
> == Interrupt latency ==
> 
> Stefano: Some improvement has been done but it is not possible to know whether
> it is good. Do you have any specific IRQ latency requirements?
> 
> Artem: There is no hard latency requirements in automotive, although many
> requirements depends on latency. For example:
>     * Scheduling
>     * GPU (implementation is sentive to interrupt latency)
> 
> Automotive is using a set of benchmark to find the virtualization overhead. This
> should be low.
> 
> ACTION: Artem to send a list of benchmark
> 
> == SMC/HVC handling in Xen ==
> 
> Artem: Please review the proposal on the mailing list. See:
> 
> https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg00430.html
> 
> == Deprivilege mode ==
> 
> EPAM are working on adding support for OP-TEE in Xen to allow multiple guest
> access the trusted firmware.
> 
> During the discussion on the design, it was suggested to move the SMC handling
> in a separate domain. This was tested using the VM event API and Mini-OS
> (upstream with Chen Baozi's series to support ARM64). The first results
> shows it is 10 times slower than handling SMC calls directly in the hypervisor.
> 
> Volodymir is working on another approach to deprivilege the execution by
> implementing a Xen EL0.
> 
> == big.LITTLE support ==
> 
> Thanasis: Document discussed on the ML. Xen will split CPUs at boot time
> (big vs little). A series will be sent on the on the ML soon.
> 
> -- 
> Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

Hi,

On 28/03/2017 16:23, Julien Grall wrote:
> Hi all,
>
> Apologies for the late sending, you will find at the end of the e-mail a
> summary of the discussion from the previous call. Feel free to reply if I
> missed some parts.
>
> I suggest to have the next call on the 5th April at 5PM UTC. Any opinions?

Apologize, I forgot that we switched timezone last Sunday. I was meant
to say 5pm BST (e.g UTC + 1). Is it fine for everyone to do the meeting
one hour later?

Cheers,

--
Julien Grall
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Stefano Stabellini]

On Wed, 29 Mar 2017, Julien Grall wrote:
> Hi,
> 
> On 28/03/2017 16:23, Julien Grall wrote:
> > Hi all,
> > 
> > Apologies for the late sending, you will find at the end of the e-mail a
> > summary of the discussion from the previous call. Feel free to reply if I
> > missed some parts.
> > 
> > I suggest to have the next call on the 5th April at 5PM UTC. Any opinions?
> 
> Apologize, I forgot that we switched timezone last Sunday. I was meant
> to say 5pm BST (e.g UTC + 1). Is it fine for everyone to do the meeting
> one hour later?

I didn't even realize that the timezone was different. Yes, please :-)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Volodymyr Babchuk]

Hi Julien,

5pm UTC+1 will be fine for me.

I just finished my EL0 PoC and want to share benchmark results.

My benchmark setup is primitive, but results are reproducible. I have
wrote small driver that calls SMC 10.000.000  times in a loop. So,
benchmark looks like this:

root@xenaarch64:~# time cat /proc/smc_test
 Will call SMC 10000000 time(s)
Done!

real 1m51.428s
user 0m0.020s
sys 1m51.240s

I compared three types of SMC "handlers":

1. Right in hypervisor. I lust put `advance_pc` into the
`do_trap_smc()`. This is the base.
2. Handling in MiniOS, using `monitor.c` framework.
3. Handling in EL0 XEN application.

In all three cases there was no actual handling. Just variants of `return 0;`.
Results:

1. Hypervisor:
    real 0m10.757s
    user 0m0.000s
    sys 0m10.752s

    10.757s = 1x (base value)
    1.07us per call

2. MiniOs:
    real 1m51.428s
    user 0m0.020s
    sys 1m51.240s

    111.428s = 10.35x slower than hypervisor handler
    11.1us per call

3. EL0:
    real 1m3.525s
    user 0m0.000s
    sys 1m3.516s

    63.525s = 5.9x slower than hypervisor handler
    6.35us par call

As you can see, handling in hypervisor mode is obviously the fastest
way. In my first implementation EL0 approach was as slow as MiniOs.
Then I simplified context switching (for example we don't need to
save/restore GIC context). That gave me 1.7x boost. Now profiler shows
that hypervisor spends time in spinlocks and p2m code. But, anyways,
10 000 000 traps in 60 seconds is a good result :)

Testing was done on Renesas H3 board with 4xA57 cores.

On 28 March 2017 at 18:23, Julien Grall <julien.grall@arm.com> wrote:
> Hi all,
>
> Apologies for the late sending, you will find at the end of the e-mail a
> summary of the discussion from the previous call. Feel free to reply if I
> missed some parts.
>
> I suggest to have the next call on the 5th April at 5PM UTC. Any opinions?
>
> Also do you have any specific topic you would like to talk during this call?
>
> Cheers,
>
> == Attendees ==
>
> Apologies if I misspelled any name.
>
> Stefano, Aporeto
> Julien, ARM
> Oleksandr, EPAM
> Artem, EPAM
> Thanasis, OnApp
> Volodymir, EPAM
>
> == Xen on ARM status ==
>
> Over 100 patches in-flight for Xen on ARM:
>     - PV protocols: Some are already accepted
>     - NUMA support
>     - GICv3 ITS support
>     - Exposing and emulating a PL011 for guest
>     - guest SMC forwarding for Xilinx platform
>     - Interrupt latency improvement
>
> == PV protocols ==
>
> * PV protocols written by Stefano was merged after 10 months
>
> Stefano: PV protocols review are moving faster
> Attendees agreed
>
> * Audio protocol: close to be accepted
> * Display protocol: minor issue, a bit more design is required
>
> Hopefully both will be ready for Xen 4.9
>
> Oleksandr: What to do when the backend die?
>
> (I cannot find any notes on it some I am not sure if we answered the question
> during the call. I suspect it has been asked to bring up the subject on the ML).
>
> == Interrupt latency ==
>
> Stefano: Some improvement has been done but it is not possible to know whether
> it is good. Do you have any specific IRQ latency requirements?
>
> Artem: There is no hard latency requirements in automotive, although many
> requirements depends on latency. For example:
>     * Scheduling
>     * GPU (implementation is sentive to interrupt latency)
>
> Automotive is using a set of benchmark to find the virtualization overhead. This
> should be low.
>
> ACTION: Artem to send a list of benchmark
>
> == SMC/HVC handling in Xen ==
>
> Artem: Please review the proposal on the mailing list. See:
>
> https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg00430.html
>
> == Deprivilege mode ==
>
> EPAM are working on adding support for OP-TEE in Xen to allow multiple guest
> access the trusted firmware.
>
> During the discussion on the design, it was suggested to move the SMC handling
> in a separate domain. This was tested using the VM event API and Mini-OS
> (upstream with Chen Baozi's series to support ARM64). The first results
> shows it is 10 times slower than handling SMC calls directly in the hypervisor.
>
> Volodymir is working on another approach to deprivilege the execution by
> implementing a Xen EL0.
>
> == big.LITTLE support ==
>
> Thanasis: Document discussed on the ML. Xen will split CPUs at boot time
> (big vs little). A series will be sent on the on the ML soon.
>
> --
> Julien Grall



-- 
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@gmail.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Stefano Stabellini]

On Thu, 30 Mar 2017, Volodymyr Babchuk wrote:
> Hi Julien,
> 
> 5pm UTC+1 will be fine for me.
> 
> I just finished my EL0 PoC and want to share benchmark results.
> 
> My benchmark setup is primitive, but results are reproducible. I have
> wrote small driver that calls SMC 10.000.000  times in a loop. So,
> benchmark looks like this:
> 
> root@xenaarch64:~# time cat /proc/smc_test
>  Will call SMC 10000000 time(s)
> Done!
> 
> real 1m51.428s
> user 0m0.020s
> sys 1m51.240s
> 
> I compared three types of SMC "handlers":
> 
> 1. Right in hypervisor. I lust put `advance_pc` into the
> `do_trap_smc()`. This is the base.
> 2. Handling in MiniOS, using `monitor.c` framework.
> 3. Handling in EL0 XEN application.
> 
> In all three cases there was no actual handling. Just variants of `return 0;`.
> Results:
> 
> 1. Hypervisor:
>     real 0m10.757s
>     user 0m0.000s
>     sys 0m10.752s
> 
>     10.757s = 1x (base value)
>     1.07us per call

1u is incredibly good actually


> 2. MiniOs:
>     real 1m51.428s
>     user 0m0.020s
>     sys 1m51.240s
> 
>     111.428s = 10.35x slower than hypervisor handler
>     11.1us per call
> 
> 3. EL0:
>     real 1m3.525s
>     user 0m0.000s
>     sys 1m3.516s
> 
>     63.525s = 5.9x slower than hypervisor handler
>     6.35us par call
> 
> As you can see, handling in hypervisor mode is obviously the fastest
> way. In my first implementation EL0 approach was as slow as MiniOs.
> Then I simplified context switching (for example we don't need to
> save/restore GIC context). That gave me 1.7x boost. Now profiler shows
> that hypervisor spends time in spinlocks and p2m code. But, anyways,
> 10 000 000 traps in 60 seconds is a good result :)
> 
> Testing was done on Renesas H3 board with 4xA57 cores.

This is a very good result! I am also quite happy with your scientific
approach. I wish more people started their projects doing experiments
like this one, before putting their head down to write thousands of
lines of code. I encourage you to submit a talk to Xen Summit about this
and/or write a blog post for https://blog.xenproject.org/.

I think that 6u per call is very good. Seeing that in the hypervisor
case the time is only 1u per call, it makes me think we might be able to
go down a bit more. I also assume that it is quite reliable (having
vcpus pinned to pcpus), while in the MiniOs case, unless you have enough
free pcpus to guarantee that MiniOs is already running, we should see
spikes of latency, right? Because if MiniOs is not running, or it cannot
run immediately, it should take much longer?  In fact, where was MiniOs
running in your setup?


> On 28 March 2017 at 18:23, Julien Grall <julien.grall@arm.com> wrote:
> > Hi all,
> >
> > Apologies for the late sending, you will find at the end of the e-mail a
> > summary of the discussion from the previous call. Feel free to reply if I
> > missed some parts.
> >
> > I suggest to have the next call on the 5th April at 5PM UTC. Any opinions?
> >
> > Also do you have any specific topic you would like to talk during this call?
> >
> > Cheers,
> >
> > == Attendees ==
> >
> > Apologies if I misspelled any name.
> >
> > Stefano, Aporeto
> > Julien, ARM
> > Oleksandr, EPAM
> > Artem, EPAM
> > Thanasis, OnApp
> > Volodymir, EPAM
> >
> > == Xen on ARM status ==
> >
> > Over 100 patches in-flight for Xen on ARM:
> >     - PV protocols: Some are already accepted
> >     - NUMA support
> >     - GICv3 ITS support
> >     - Exposing and emulating a PL011 for guest
> >     - guest SMC forwarding for Xilinx platform
> >     - Interrupt latency improvement
> >
> > == PV protocols ==
> >
> > * PV protocols written by Stefano was merged after 10 months
> >
> > Stefano: PV protocols review are moving faster
> > Attendees agreed
> >
> > * Audio protocol: close to be accepted
> > * Display protocol: minor issue, a bit more design is required
> >
> > Hopefully both will be ready for Xen 4.9
> >
> > Oleksandr: What to do when the backend die?
> >
> > (I cannot find any notes on it some I am not sure if we answered the question
> > during the call. I suspect it has been asked to bring up the subject on the ML).
> >
> > == Interrupt latency ==
> >
> > Stefano: Some improvement has been done but it is not possible to know whether
> > it is good. Do you have any specific IRQ latency requirements?
> >
> > Artem: There is no hard latency requirements in automotive, although many
> > requirements depends on latency. For example:
> >     * Scheduling
> >     * GPU (implementation is sentive to interrupt latency)
> >
> > Automotive is using a set of benchmark to find the virtualization overhead. This
> > should be low.
> >
> > ACTION: Artem to send a list of benchmark
> >
> > == SMC/HVC handling in Xen ==
> >
> > Artem: Please review the proposal on the mailing list. See:
> >
> > https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg00430.html
> >
> > == Deprivilege mode ==
> >
> > EPAM are working on adding support for OP-TEE in Xen to allow multiple guest
> > access the trusted firmware.
> >
> > During the discussion on the design, it was suggested to move the SMC handling
> > in a separate domain. This was tested using the VM event API and Mini-OS
> > (upstream with Chen Baozi's series to support ARM64). The first results
> > shows it is 10 times slower than handling SMC calls directly in the hypervisor.
> >
> > Volodymir is working on another approach to deprivilege the execution by
> > implementing a Xen EL0.
> >
> > == big.LITTLE support ==
> >
> > Thanasis: Document discussed on the ML. Xen will split CPUs at boot time
> > (big vs little). A series will be sent on the on the ML soon.
> >
> > --
> > Julien Grall
> 
> 
> 
> -- 
> WBR Volodymyr Babchuk aka lorc [+380976646013]
> mailto: vlad.babchuk@gmail.com
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Volodymyr Babchuk]

Hi Stefano,

On 30 March 2017 at 21:52, Stefano Stabellini <sstabellini@kernel.org> wrote:
> On Thu, 30 Mar 2017, Volodymyr Babchuk wrote:
>> Hi Julien,
>>
>> 5pm UTC+1 will be fine for me.
>>
>> I just finished my EL0 PoC and want to share benchmark results.
>>
>> My benchmark setup is primitive, but results are reproducible. I have
>> wrote small driver that calls SMC 10.000.000  times in a loop. So,
>> benchmark looks like this:
>>
>> root@xenaarch64:~# time cat /proc/smc_test
>>  Will call SMC 10000000 time(s)
>> Done!
>>
>> real 1m51.428s
>> user 0m0.020s
>> sys 1m51.240s
>>
>> I compared three types of SMC "handlers":
>>
>> 1. Right in hypervisor. I lust put `advance_pc` into the
>> `do_trap_smc()`. This is the base.
>> 2. Handling in MiniOS, using `monitor.c` framework.
>> 3. Handling in EL0 XEN application.
>>
>> In all three cases there was no actual handling. Just variants of `return 0;`.
>> Results:
>>
>> 1. Hypervisor:
>>     real 0m10.757s
>>     user 0m0.000s
>>     sys 0m10.752s
>>
>>     10.757s = 1x (base value)
>>     1.07us per call
>
> 1u is incredibly good actually

Yep. But handler was right in the traps.c. There is very short path:
guest_sync (in entry.S) -> do_trap_hypervisor() -> do_trap_smc().
I assume that most of simple traps are handled within 1-2us.

>
>> 2. MiniOs:
>>     real 1m51.428s
>>     user 0m0.020s
>>     sys 1m51.240s
>>
>>     111.428s = 10.35x slower than hypervisor handler
>>     11.1us per call
>>
>> 3. EL0:
>>     real 1m3.525s
>>     user 0m0.000s
>>     sys 1m3.516s
>>
>>     63.525s = 5.9x slower than hypervisor handler
>>     6.35us par call
>>
>> As you can see, handling in hypervisor mode is obviously the fastest
>> way. In my first implementation EL0 approach was as slow as MiniOs.
>> Then I simplified context switching (for example we don't need to
>> save/restore GIC context). That gave me 1.7x boost. Now profiler shows
>> that hypervisor spends time in spinlocks and p2m code. But, anyways,
>> 10 000 000 traps in 60 seconds is a good result :)
>>
>> Testing was done on Renesas H3 board with 4xA57 cores.
>
> This is a very good result! I am also quite happy with your scientific
> approach.
Thanks!

> I wish more people started their projects doing experiments
> like this one, before putting their head down to write thousands of
> lines of code. I encourage you to submit a talk to Xen Summit about this
> and/or write a blog post for https://blog.xenproject.org/.
Yep. Actually Artem is pushing me to talk at Xen Summit :)

> I think that 6u per call is very good. Seeing that in the hypervisor
> case the time is only 1u per call, it makes me think we might be able to
> go down a bit more. I also assume that it is quite reliable (having
> vcpus pinned to pcpus), while in the MiniOs case, unless you have enough
> free pcpus to guarantee that MiniOs is already running, we should see
> spikes of latency, right? Because if MiniOs is not running, or it cannot
> run immediately, it should take much longer?  In fact, where was MiniOs
> running in your setup?

Yes, there was MiniOS for aarch64 running. I published links to
corresponding patches in the ML some time ago.

I didn't pinned vcpus to cpus because I had only 3 domais running on 4 CPUs:
dom0, domU which I used for benchmarking and MiniOS. I'm sure that at
leas two cores were idle. But you are right, under heavy load there
will be latency spikes, because we can't guarantee that MiniOs will be
scheduled immediately to serve handling request.
My design of EL0 app is better in this case, because for hypervisor it
looks like another function call. It still can be preempted, thought.
But at least entry to EL0 handler is done immediately.

And yes, my profiler shows that there are ways to further decrease
latency. Most obvious way is to get rid of 2nd stage translation and
thus eliminate p2m code from the call chain. Currently hypervisor
spends 20% of time in spinlocks code and about ~10-15% in p2m. So
there definitely are areas to improve :)
>
>> On 28 March 2017 at 18:23, Julien Grall <julien.grall@arm.com> wrote:
>> > Hi all,
>> >
>> > Apologies for the late sending, you will find at the end of the e-mail a
>> > summary of the discussion from the previous call. Feel free to reply if I
>> > missed some parts.
>> >
>> > I suggest to have the next call on the 5th April at 5PM UTC. Any opinions?
>> >
>> > Also do you have any specific topic you would like to talk during this call?
>> >
>> > Cheers,
>> >
>> > == Attendees ==
>> >
>> > Apologies if I misspelled any name.
>> >
>> > Stefano, Aporeto
>> > Julien, ARM
>> > Oleksandr, EPAM
>> > Artem, EPAM
>> > Thanasis, OnApp
>> > Volodymir, EPAM
>> >
>> > == Xen on ARM status ==
>> >
>> > Over 100 patches in-flight for Xen on ARM:
>> >     - PV protocols: Some are already accepted
>> >     - NUMA support
>> >     - GICv3 ITS support
>> >     - Exposing and emulating a PL011 for guest
>> >     - guest SMC forwarding for Xilinx platform
>> >     - Interrupt latency improvement
>> >
>> > == PV protocols ==
>> >
>> > * PV protocols written by Stefano was merged after 10 months
>> >
>> > Stefano: PV protocols review are moving faster
>> > Attendees agreed
>> >
>> > * Audio protocol: close to be accepted
>> > * Display protocol: minor issue, a bit more design is required
>> >
>> > Hopefully both will be ready for Xen 4.9
>> >
>> > Oleksandr: What to do when the backend die?
>> >
>> > (I cannot find any notes on it some I am not sure if we answered the question
>> > during the call. I suspect it has been asked to bring up the subject on the ML).
>> >
>> > == Interrupt latency ==
>> >
>> > Stefano: Some improvement has been done but it is not possible to know whether
>> > it is good. Do you have any specific IRQ latency requirements?
>> >
>> > Artem: There is no hard latency requirements in automotive, although many
>> > requirements depends on latency. For example:
>> >     * Scheduling
>> >     * GPU (implementation is sentive to interrupt latency)
>> >
>> > Automotive is using a set of benchmark to find the virtualization overhead. This
>> > should be low.
>> >
>> > ACTION: Artem to send a list of benchmark
>> >
>> > == SMC/HVC handling in Xen ==
>> >
>> > Artem: Please review the proposal on the mailing list. See:
>> >
>> > https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg00430.html
>> >
>> > == Deprivilege mode ==
>> >
>> > EPAM are working on adding support for OP-TEE in Xen to allow multiple guest
>> > access the trusted firmware.
>> >
>> > During the discussion on the design, it was suggested to move the SMC handling
>> > in a separate domain. This was tested using the VM event API and Mini-OS
>> > (upstream with Chen Baozi's series to support ARM64). The first results
>> > shows it is 10 times slower than handling SMC calls directly in the hypervisor.
>> >
>> > Volodymir is working on another approach to deprivilege the execution by
>> > implementing a Xen EL0.
>> >
>> > == big.LITTLE support ==
>> >
>> > Thanasis: Document discussed on the ML. Xen will split CPUs at boot time
>> > (big vs little). A series will be sent on the on the ML soon.
>> >
>> > --
>> > Julien Grall
>>
>>
>>
>> --
>> WBR Volodymyr Babchuk aka lorc [+380976646013]
>> mailto: vlad.babchuk@gmail.com
>>



-- 
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@gmail.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

Hi Volodymyr

On 30/03/2017 20:19, Volodymyr Babchuk wrote:
> On 30 March 2017 at 21:52, Stefano Stabellini <sstabellini@kernel.org> wrote:
>> On Thu, 30 Mar 2017, Volodymyr Babchuk wrote:
> And yes, my profiler shows that there are ways to further decrease
> latency. Most obvious way is to get rid of 2nd stage translation and
> thus eliminate p2m code from the call chain. Currently hypervisor
> spends 20% of time in spinlocks code and about ~10-15% in p2m. So
> there definitely are areas to improve :)

Correct me if I am wrong. You are suggesting to remove stage-2 MMU 
translation, right? If so, what are the benefits to have some Xen code 
running at EL0?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Volodymyr Babchuk]

Hi Julien,



On 30 March 2017 at 22:57, Julien Grall <julien.grall@arm.com> wrote:
> Hi Volodymyr
>
> On 30/03/2017 20:19, Volodymyr Babchuk wrote:
>>
>> On 30 March 2017 at 21:52, Stefano Stabellini <sstabellini@kernel.org>
>> wrote:
>>>
>>> On Thu, 30 Mar 2017, Volodymyr Babchuk wrote:
>>
>> And yes, my profiler shows that there are ways to further decrease
>> latency. Most obvious way is to get rid of 2nd stage translation and
>> thus eliminate p2m code from the call chain. Currently hypervisor
>> spends 20% of time in spinlocks code and about ~10-15% in p2m. So
>> there definitely are areas to improve :)
>
>
> Correct me if I am wrong. You are suggesting to remove stage-2 MMU
> translation, right? If so, what are the benefits to have some Xen code
> running at EL0?
Because that will be loadable code :) Also, this will provide some
degree of isolation as this code will communicate with Xen only via
SVCs.

Look, we need two stages in the first place because conventional guest
wants to control own MMU. But Xen native app (lets call it in this
way) should not control MMU. In my hack I had to create stage-1 table
with 1:1 mapping to make things work. Actually.... it just came to me
that I can try to disable stage 1 MMU and leave only stage 2. Not sure
if it is possible, need to check TRM...
But, anyways, my initial idea was to disable second stage MMU (drop VM
bit in HCR_EL2) and program only TTBR0_EL1 with friends. With this
approach there will be no need to save/restore p2m context when I
switch from guest context to app context and back.

-- 
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@gmail.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

On 03/30/2017 09:17 PM, Volodymyr Babchuk wrote:
> Hi Julien,

Hi Volodymyr,

> On 30 March 2017 at 22:57, Julien Grall <julien.grall@arm.com> wrote:
>> Hi Volodymyr
>>
>> On 30/03/2017 20:19, Volodymyr Babchuk wrote:
>>>
>>> On 30 March 2017 at 21:52, Stefano Stabellini <sstabellini@kernel.org>
>>> wrote:
>>>>
>>>> On Thu, 30 Mar 2017, Volodymyr Babchuk wrote:
>>>
>>> And yes, my profiler shows that there are ways to further decrease
>>> latency. Most obvious way is to get rid of 2nd stage translation and
>>> thus eliminate p2m code from the call chain. Currently hypervisor
>>> spends 20% of time in spinlocks code and about ~10-15% in p2m. So
>>> there definitely are areas to improve :)
>>
>>
>> Correct me if I am wrong. You are suggesting to remove stage-2 MMU
>> translation, right? If so, what are the benefits to have some Xen code
>> running at EL0?
> Because that will be loadable code :) Also, this will provide some
> degree of isolation as this code will communicate with Xen only via
> SVCs.Hi
>
> Look, we need two stages in the first place because conventional guest
> wants to control own MMU. But Xen native app (lets call it in this
> way) should not control MMU. In my hack I had to create stage-1 table
> with 1:1 mapping to make things work. Actually.... it just came to me
> that I can try to disable stage 1 MMU and leave only stage 2. Not sure
> if it is possible, need to check TRM...

I cannot see why stage-1 page table cannot be disabled.

> But, anyways, my initial idea was to disable second stage MMU (drop VM
> bit in HCR_EL2) and program only TTBR0_EL1 with friends. With this
> approach there will be no need to save/restore p2m context when I
> switch from guest context to app context and back.

Even if you disable stage-2 translation, you still have to program 
VTTBR_EL2.VMID because, from my understanding the TLBs will still be 
tagged with the VMID (see the note in page D4-1823 in ARM DDI 
0487A.k_iss10775).

If you keep the VMID of the guest, you would need to flush the TLBs at 
every context switch to avoid TLB conflict.

To prevent that, you would need a specific VMID for the EL0 app, and 
therefore save/restore a part of the p21m context everytime.

The benefits of using stage-2 page table, is you can leverage the p2m 
code rather than creating another version to handle stage-1.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

Hi Volodymyr

On 30/03/2017 20:19, Volodymyr Babchuk wrote:
> On 30 March 2017 at 21:52, Stefano Stabellini <sstabellini@kernel.org> wrote:
>> On Thu, 30 Mar 2017, Volodymyr Babchuk wrote:
> And yes, my profiler shows that there are ways to further decrease
> latency. Most obvious way is to get rid of 2nd stage translation and
> thus eliminate p2m code from the call chain. Currently hypervisor
> spends 20% of time in spinlocks code and about ~10-15% in p2m. So
> there definitely are areas to improve :)

Correct me if I am wrong. You are suggesting to remove stage-2 MMU 
translation, right? If so, what are the benefits to have some Xen code 
running at EL0?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

Hi,

Quick reminder the community call will be tomorrow at 5pm BST (i.e UTC + 1).

Let me know if you have any specific topic you would like to talk during 
this call.

For joining the call, please use either:

Call            +44 1223 406065 (Local dial in)
and enter the access code below followed by # key.
Participant code: 4915191

Mobile Auto Dial:
         VoIP: voip://+441223406065;4915191#
         iOS devices: +44 1223 406065,4915191 and press #
         Other devices: +44 1223 406065x4915191#

Additional Calling Information:

UK +44 1142828002
US CA +1 4085761502
US TX +1 5123141073
JP +81 453455355
DE +49 8945604050
NO +47 73187518
SE +46 46313131
FR +33 497235101
TW +886 35657119
HU +36 13275600
IE +353 91337900

Toll Free

UK 0800 1412084
US +1 8668801148
CN +86 4006782367
IN 0008009868365
IN +918049282778
TW 08000 22065
HU 0680981587
IE 1800800022
KF +972732558877

Regards,

On 28/03/17 16:23, Julien Grall wrote:
> Hi all,
>
> Apologies for the late sending, you will find at the end of the e-mail a
> summary of the discussion from the previous call. Feel free to reply if I
> missed some parts.
>
> I suggest to have the next call on the 5th April at 5PM UTC. Any opinions?
>
> Also do you have any specific topic you would like to talk during this call?
>
> Cheers,
>
> == Attendees ==
>
> Apologies if I misspelled any name.
>
> Stefano, Aporeto
> Julien, ARM
> Oleksandr, EPAM
> Artem, EPAM
> Thanasis, OnApp
> Volodymir, EPAM
>
> == Xen on ARM status ==
>
> Over 100 patches in-flight for Xen on ARM:
>     - PV protocols: Some are already accepted
>     - NUMA support
>     - GICv3 ITS support
>     - Exposing and emulating a PL011 for guest
>     - guest SMC forwarding for Xilinx platform
>     - Interrupt latency improvement
>
> == PV protocols ==
>
> * PV protocols written by Stefano was merged after 10 months
>
> Stefano: PV protocols review are moving faster
> Attendees agreed
>
> * Audio protocol: close to be accepted
> * Display protocol: minor issue, a bit more design is required
>
> Hopefully both will be ready for Xen 4.9
>
> Oleksandr: What to do when the backend die?
>
> (I cannot find any notes on it some I am not sure if we answered the question
> during the call. I suspect it has been asked to bring up the subject on the ML).
>
> == Interrupt latency ==
>
> Stefano: Some improvement has been done but it is not possible to know whether
> it is good. Do you have any specific IRQ latency requirements?
>
> Artem: There is no hard latency requirements in automotive, although many
> requirements depends on latency. For example:
>     * Scheduling
>     * GPU (implementation is sentive to interrupt latency)
>
> Automotive is using a set of benchmark to find the virtualization overhead. This
> should be low.
>
> ACTION: Artem to send a list of benchmark
>
> == SMC/HVC handling in Xen ==
>
> Artem: Please review the proposal on the mailing list. See:
>
> https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg00430.html
>
> == Deprivilege mode ==
>
> EPAM are working on adding support for OP-TEE in Xen to allow multiple guest
> access the trusted firmware.
>
> During the discussion on the design, it was suggested to move the SMC handling
> in a separate domain. This was tested using the VM event API and Mini-OS
> (upstream with Chen Baozi's series to support ARM64). The first results
> shows it is 10 times slower than handling SMC calls directly in the hypervisor.
>
> Volodymir is working on another approach to deprivilege the execution by
> implementing a Xen EL0.
>
> == big.LITTLE support ==
>
> Thanasis: Document discussed on the ML. Xen will split CPUs at boot time
> (big vs little). A series will be sent on the on the ML soon.
>

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

Hi all,

Below the meeting minutes for the previous Xen community call. Feel free to
reply if I missed some parts.

I suggest to have the next call on the 3rd May at 5PM BST. Any opinions.

Also, do you have any specific topic you would like to talk during this 
call?

Cheers,

== Attendees ==

Stefano, Aporeto
Julien, ARM
Oleksandr, EPAM
Volodymir, EPAM
Edgar, Xilinx
Anasatasios, OnApp

== SMC and EL0 App ==

Some SMC call may be sensitive to latency (e.g power down).

Stefano: We would need to be able to decide whether SMC are emulated in 
EL0 or EL2.

Edgar: Shall I wait EL0 App support before resending the series to 
forward SMC on Zynqmp [1]?

Answer: No need to wait, the current approach is fine for now.

Xen EL0 update from Volodymir:
     * Thread started on xen-devel (see [2])
     * Stefano suggested to create and share a small example
     * Plan -> Post the patches on github + writing document

== IOMMU ==

Xen does not support the generic IOMMU bindings.

ACTION: Needs to be fixed ASAP. Potentially on Xen 4.9?

== Reserve region support in DT ==

Reserve region is not mapped in DOM0, though they should be mapped. DMA 
API will allocate memory for those regions.

There was a discussion about that few months ago (see [3]). This would 
need to be followed-up.

== Cross-compiling ==

Anastasios: We are looking at support Xen in Buildroot. The use case is 
a small initramfs.

Potential task: Keep Xen updated on buildroot. Anastasios could help on 
that.

Stefano: Xen is compiling fine with Yocto.

ACTION: Stefano to write a wiki page about using meta-virtualization on 
Yocto.

Release update
--------------

Stefano might keep a branch around to keep patches which target the next
release.

[1] 
https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00634.html
[2] 
https://lists.xenproject.org/archives/html/xen-devel/2017-04/msg01002.html
[3] https://lists.xen.org/archives/html/xen-devel/2016-11/msg02295.html

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Stefano Stabellini]

On Thu, 20 Apr 2017, Julien Grall wrote:
> == Cross-compiling ==
> 
> Anastasios: We are looking at support Xen in Buildroot. The use case is a
> small initramfs.
> 
> Potential task: Keep Xen updated on buildroot. Anastasios could help on that.
> 
> Stefano: Xen is compiling fine with Yocto.
> 
> ACTION: Stefano to write a wiki page about using meta-virtualization on Yocto.

I wrote the following wiki page, let me know if it works for you:

https://wiki.xenproject.org/wiki/Xen_on_ARM_and_Yocto

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

Hi all,

A quick reminder, the Community Call will be tomorrow (Wednesday 3rd 
May) at 5PM BST.

For joining the call, please use either:

Call            +44 1223 406065 (Local dial in)
and enter the access code below followed by # key.
Participant code: 4915191

Mobile Auto Dial:
         VoIP: voip://+441223406065;4915191#
         iOS devices: +44 1223 406065,4915191 and press #
         Other devices: +44 1223 406065x4915191#

Additional Calling Information:

UK +44 1142828002
US CA +1 4085761502
US TX +1 5123141073
JP +81 453455355
DE +49 8945604050
NO +47 73187518
SE +46 46313131
FR +33 497235101
TW +886 35657119
HU +36 13275600
IE +353 91337900

Toll Free

UK 0800 1412084
US +1 8668801148
CN +86 4006782367
IN 0008009868365
IN +918049282778
TW 08000 22065
HU 0680981587
IE 1800800022
KF +972732558877

Regards,

On 20/04/17 17:44, Julien Grall wrote:
> Hi all,
>
> Below the meeting minutes for the previous Xen community call. Feel free to
> reply if I missed some parts.
>
> I suggest to have the next call on the 3rd May at 5PM BST. Any opinions.
>
> Also, do you have any specific topic you would like to talk during this
> call?
>
> Cheers,
>
> == Attendees ==
>
> Stefano, Aporeto
> Julien, ARM
> Oleksandr, EPAM
> Volodymir, EPAM
> Edgar, Xilinx
> Anasatasios, OnApp
>
> == SMC and EL0 App ==
>
> Some SMC call may be sensitive to latency (e.g power down).
>
> Stefano: We would need to be able to decide whether SMC are emulated in
> EL0 or EL2.
>
> Edgar: Shall I wait EL0 App support before resending the series to
> forward SMC on Zynqmp [1]?
>
> Answer: No need to wait, the current approach is fine for now.
>
> Xen EL0 update from Volodymir:
>     * Thread started on xen-devel (see [2])
>     * Stefano suggested to create and share a small example
>     * Plan -> Post the patches on github + writing document
>
> == IOMMU ==
>
> Xen does not support the generic IOMMU bindings.
>
> ACTION: Needs to be fixed ASAP. Potentially on Xen 4.9?
>
> == Reserve region support in DT ==
>
> Reserve region is not mapped in DOM0, though they should be mapped. DMA
> API will allocate memory for those regions.
>
> There was a discussion about that few months ago (see [3]). This would
> need to be followed-up.
>
> == Cross-compiling ==
>
> Anastasios: We are looking at support Xen in Buildroot. The use case is
> a small initramfs.
>
> Potential task: Keep Xen updated on buildroot. Anastasios could help on
> that.
>
> Stefano: Xen is compiling fine with Yocto.
>
> ACTION: Stefano to write a wiki page about using meta-virtualization on
> Yocto.
>
> Release update
> --------------
>
> Stefano might keep a branch around to keep patches which target the next
> release.
>
> [1]
> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00634.html
> [2]
> https://lists.xenproject.org/archives/html/xen-devel/2017-04/msg01002.html
> [3] https://lists.xen.org/archives/html/xen-devel/2016-11/msg02295.html
>

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Volodymyr Babchuk]

Hello guys,

I'm following the OP-TEE discussion.
At [1] you can find my patch series for OP-TEE calls handling. It is
very old thing, but it worked (and even had nice features like ability
to handle domain crash in the middle of OP-TEE session).
If you want latest version, just take a look at [2]. There are not all
features implemented. My estimation is that full handler will be 3-4
times bigger (about 1000-1500 lines of code, I think).

[1] https://github.com/lorc/xen/commits/staging-4.7
[2] https://github.com/lorc/xen/blob/f0f22838a63b5c29b21a1dc2d3f5db81323aa3ac/xen/arch/arm/optee/smc.c


On 2 May 2017 at 21:00, Julien Grall <julien.grall@arm.com> wrote:
> Hi all,
>
> A quick reminder, the Community Call will be tomorrow (Wednesday 3rd May) at
> 5PM BST.
>
> For joining the call, please use either:
>
> Call            +44 1223 406065 (Local dial in)
> and enter the access code below followed by # key.
> Participant code: 4915191
>
> Mobile Auto Dial:
>         VoIP: voip://+441223406065;4915191#
>         iOS devices: +44 1223 406065,4915191 and press #
>         Other devices: +44 1223 406065x4915191#
>
> Additional Calling Information:
>
> UK +44 1142828002
> US CA +1 4085761502
> US TX +1 5123141073
> JP +81 453455355
> DE +49 8945604050
> NO +47 73187518
> SE +46 46313131
> FR +33 497235101
> TW +886 35657119
> HU +36 13275600
> IE +353 91337900
>
> Toll Free
>
> UK 0800 1412084
> US +1 8668801148
> CN +86 4006782367
> IN 0008009868365
> IN +918049282778
> TW 08000 22065
> HU 0680981587
> IE 1800800022
> KF +972732558877
>
> Regards,
>
>
> On 20/04/17 17:44, Julien Grall wrote:
>>
>> Hi all,
>>
>> Below the meeting minutes for the previous Xen community call. Feel free
>> to
>> reply if I missed some parts.
>>
>> I suggest to have the next call on the 3rd May at 5PM BST. Any opinions.
>>
>> Also, do you have any specific topic you would like to talk during this
>> call?
>>
>> Cheers,
>>
>> == Attendees ==
>>
>> Stefano, Aporeto
>> Julien, ARM
>> Oleksandr, EPAM
>> Volodymir, EPAM
>> Edgar, Xilinx
>> Anasatasios, OnApp
>>
>> == SMC and EL0 App ==
>>
>> Some SMC call may be sensitive to latency (e.g power down).
>>
>> Stefano: We would need to be able to decide whether SMC are emulated in
>> EL0 or EL2.
>>
>> Edgar: Shall I wait EL0 App support before resending the series to
>> forward SMC on Zynqmp [1]?
>>
>> Answer: No need to wait, the current approach is fine for now.
>>
>> Xen EL0 update from Volodymir:
>>     * Thread started on xen-devel (see [2])
>>     * Stefano suggested to create and share a small example
>>     * Plan -> Post the patches on github + writing document
>>
>> == IOMMU ==
>>
>> Xen does not support the generic IOMMU bindings.
>>
>> ACTION: Needs to be fixed ASAP. Potentially on Xen 4.9?
>>
>> == Reserve region support in DT ==
>>
>> Reserve region is not mapped in DOM0, though they should be mapped. DMA
>> API will allocate memory for those regions.
>>
>> There was a discussion about that few months ago (see [3]). This would
>> need to be followed-up.
>>
>> == Cross-compiling ==
>>
>> Anastasios: We are looking at support Xen in Buildroot. The use case is
>> a small initramfs.
>>
>> Potential task: Keep Xen updated on buildroot. Anastasios could help on
>> that.
>>
>> Stefano: Xen is compiling fine with Yocto.
>>
>> ACTION: Stefano to write a wiki page about using meta-virtualization on
>> Yocto.
>>
>> Release update
>> --------------
>>
>> Stefano might keep a branch around to keep patches which target the next
>> release.
>>
>> [1]
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00634.html
>> [2]
>> https://lists.xenproject.org/archives/html/xen-devel/2017-04/msg01002.html
>> [3] https://lists.xen.org/archives/html/xen-devel/2016-11/msg02295.html
>>
>
> --
> Julien Grall



-- 
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@gmail.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Artem Mygaiev]

[-- Attachment #1.1: Type: text/plain, Size: 1843 bytes --]

Lets make it clear - there must be one translation table for EL0 apps...

On Mar 30, 2017 23:18, Volodymyr Babchuk <vlad.babchuk@gmail.com> wrote:
Hi Julien,



On 30 March 2017 at 22:57, Julien Grall <julien.grall@arm.com> wrote:
> Hi Volodymyr
>
> On 30/03/2017 20:19, Volodymyr Babchuk wrote:
>>
>> On 30 March 2017 at 21:52, Stefano Stabellini <sstabellini@kernel.org>
>> wrote:
>>>
>>> On Thu, 30 Mar 2017, Volodymyr Babchuk wrote:
>>
>> And yes, my profiler shows that there are ways to further decrease
>> latency. Most obvious way is to get rid of 2nd stage translation and
>> thus eliminate p2m code from the call chain. Currently hypervisor
>> spends 20% of time in spinlocks code and about ~10-15% in p2m. So
>> there definitely are areas to improve :)
>
>
> Correct me if I am wrong. You are suggesting to remove stage-2 MMU
> translation, right? If so, what are the benefits to have some Xen code
> running at EL0?
Because that will be loadable code :) Also, this will provide some
degree of isolation as this code will communicate with Xen only via
SVCs.

Look, we need two stages in the first place because conventional guest
wants to control own MMU. But Xen native app (lets call it in this
way) should not control MMU. In my hack I had to create stage-1 table
with 1:1 mapping to make things work. Actually.... it just came to me
that I can try to disable stage 1 MMU and leave only stage 2. Not sure
if it is possible, need to check TRM...
But, anyways, my initial idea was to disable second stage MMU (drop VM
bit in HCR_EL2) and program only TTBR0_EL1 with friends. With this
approach there will be no need to save/restore p2m context when I
switch from guest context to app context and back.

--
WBR Volodymyr Babchuk aka lorc [+380976646013]
mailto: vlad.babchuk@gmail.com


[-- Attachment #1.2: Type: text/html, Size: 2568 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

Hi all,

Apologies for the late sending. You will find at the end of the mail a summary
of the discussion. Feel free to reply if I missed some parts.

At the end of the call we agreed to have another meeting before the end of the
year. Given that the Christmas is approaching and some people may take holidays
around, I would suggest to do the meeting on Wednesday 14th December at 4pm
UTC. Any opinions?

Also, do you have any specific topic you would like to talk during the next call?

Cheers,

== Attendees ==

Apologies if I misspelled any name.

* Stefano Stabellini:
    Aporeto, Xen ARM maintainer
* Julien Grall
    ARM, Xen ARM maintainer
* Artem Mygaiev, Andrii Anisov, Alexandre Agizim:
    EPAM, Embedded and automotive application for Xen
* Steward Hildebrand:
    Dornerworks, help customer to use Xen in their product
* Edgar Iglesias:
    Xilinx, Embedded and datacenter application
* Meng Xu:
    University of Pennsylvania, RT Xen. Looking for improving performance for
    real-time application in virtualised environment
* Bosch (sorry I forgot the name of the attendees): Car multimedia
* Anastassios Nanos:
    OnApp, using Xen on lower power server
* Dario Faggioli:
    Citrix, scheduler maintainer. Interested in following big.LITTLE discussion

== Features ==

=== Co-processor architecture ===

Artem:  EPAM is working on a co-processor framework to share resources between
        guests (see a RFC of the design document [1]). A prototype has been
        created by sharing a GPU between guests, the overhead is ~5% compare
        to native.
Julien: concerned about context switch time
Stefano: concerned about the size of the emulator and security impact
Edgar:  it might not be possible to know the FPGA accelerator when building
        Xen.
Stefano: having the emulator in Xen EL1 would be better for protection
Andrii: it is not necessary to implement a full co-processor emulator. It is
        sufficient to emulation the behavior on register write.

ACTION: Continue the discussion on the mailing list.

=== big.LITTLE ===

Anastassios:    interested in knowing the status of big.LITTLE support in Xen
Dario:          went through the thread on the ML. The consensus seems to be
                based on vcpu pin/affinity.
Anastassios:    There are issue the way xen handles boot code. Wrong errata
                for cpus.
Julien:         Core could have different errata and features. Errata already
                works today.
                We need a summary of the discussion.

Dario stepped in to write a summary.

Anastassios:    What is the best board to work big.LITTLE with Xen?

?:              Renesas

ACTION: Dario to write a summary of the big.LITTLE discussions.

=== Secure Call Monitor (SMC) from guests ===

Andrii/Artem:   EPAM is working on allowing a guest to make call to TEE (e.g
                OPTEE). They are working in collaboration with Linaro to
                make OPTEE  virtualization aware. A design document has been
                posted on the ML (see [3]).
Edgar:          interested on this. Trusted world need to be accessed to
                manage FPGA and for power management

=== Running unmodified baremetal software in a guest ===

Edgar:          Xilinx is working on running unmodified baremetal software
                in a guest. Piece of work required:
                    - Mapping memory area with different memory attribut
                      to domU
                    - Replicate host memory map
                    - Exposing a vUART to guest (see [4] for the discussion)
Steward:        vUART is very important, especially for baremetal guests
Stefano:        it would need to be loggable
Julien:         we would have the same issue in the future to be compliant
                with the VM Spec (see [5]).

=== Areas of concern ===

Bosch: problem with xen-swiotlb. It does not work properly on renesas board.
Stefano: please report the error on the ML

ACTION: Bosch to send a bug report regarding xen-swiotlb

Edgar:  IOMMU could not be used by the guest (Stage-1). This would be useful
        to implement driver in userspace.
Julien: When will it be required?
Edgar:  It is a trend

Julien: Should we organized another Community Call? When?
Artem: Once per month is a good start

All: Agreed on a call before Christmas

[1] https://lists.xenproject.org/archives/html/xen-devel/2016-10/msg01966.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01802.html
[3] https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg02220.html
[4] https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg00846.html
[5] http://people.linaro.org/~christoffer.dall/VMSystemSpecificationForARM-v2.0.pdf

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Stefano Stabellini]

On Tue, 6 Dec 2016, Julien Grall wrote:
> Hi all,
> 
> Apologies for the late sending. You will find at the end of the mail a summary
> of the discussion. Feel free to reply if I missed some parts.
> 
> At the end of the call we agreed to have another meeting before the end of the
> year. Given that the Christmas is approaching and some people may take holidays
> around, I would suggest to do the meeting on Wednesday 14th December at 4pm
> UTC. Any opinions?

Would it be possible to move this call to 5pm UTC (this one, not
necessarily all the future calls)?



> Also, do you have any specific topic you would like to talk during the next call?
> 
> Cheers,
> 
> == Attendees ==
> 
> Apologies if I misspelled any name.
> 
> * Stefano Stabellini:
>     Aporeto, Xen ARM maintainer
> * Julien Grall
>     ARM, Xen ARM maintainer
> * Artem Mygaiev, Andrii Anisov, Alexandre Agizim:
>     EPAM, Embedded and automotive application for Xen
> * Steward Hildebrand:
>     Dornerworks, help customer to use Xen in their product
> * Edgar Iglesias:
>     Xilinx, Embedded and datacenter application
> * Meng Xu:
>     University of Pennsylvania, RT Xen. Looking for improving performance for
>     real-time application in virtualised environment
> * Bosch (sorry I forgot the name of the attendees): Car multimedia
> * Anastassios Nanos:
>     OnApp, using Xen on lower power server
> * Dario Faggioli:
>     Citrix, scheduler maintainer. Interested in following big.LITTLE discussion
> 
> == Features ==
> 
> === Co-processor architecture ===
> 
> Artem:  EPAM is working on a co-processor framework to share resources between
>         guests (see a RFC of the design document [1]). A prototype has been
>         created by sharing a GPU between guests, the overhead is ~5% compare
>         to native.
> Julien: concerned about context switch time
> Stefano: concerned about the size of the emulator and security impact
> Edgar:  it might not be possible to know the FPGA accelerator when building
>         Xen.
> Stefano: having the emulator in Xen EL1 would be better for protection
> Andrii: it is not necessary to implement a full co-processor emulator. It is
>         sufficient to emulation the behavior on register write.
> 
> ACTION: Continue the discussion on the mailing list.
> 
> === big.LITTLE ===
> 
> Anastassios:    interested in knowing the status of big.LITTLE support in Xen
> Dario:          went through the thread on the ML. The consensus seems to be
>                 based on vcpu pin/affinity.
> Anastassios:    There are issue the way xen handles boot code. Wrong errata
>                 for cpus.
> Julien:         Core could have different errata and features. Errata already
>                 works today.
>                 We need a summary of the discussion.
> 
> Dario stepped in to write a summary.
> 
> Anastassios:    What is the best board to work big.LITTLE with Xen?
> 
> ?:              Renesas
> 
> ACTION: Dario to write a summary of the big.LITTLE discussions.
> 
> === Secure Call Monitor (SMC) from guests ===
> 
> Andrii/Artem:   EPAM is working on allowing a guest to make call to TEE (e.g
>                 OPTEE). They are working in collaboration with Linaro to
>                 make OPTEE  virtualization aware. A design document has been
>                 posted on the ML (see [3]).
> Edgar:          interested on this. Trusted world need to be accessed to
>                 manage FPGA and for power management
> 
> === Running unmodified baremetal software in a guest ===
> 
> Edgar:          Xilinx is working on running unmodified baremetal software
>                 in a guest. Piece of work required:
>                     - Mapping memory area with different memory attribut
>                       to domU
>                     - Replicate host memory map
>                     - Exposing a vUART to guest (see [4] for the discussion)
> Steward:        vUART is very important, especially for baremetal guests
> Stefano:        it would need to be loggable
> Julien:         we would have the same issue in the future to be compliant
>                 with the VM Spec (see [5]).
> 
> === Areas of concern ===
> 
> Bosch: problem with xen-swiotlb. It does not work properly on renesas board.
> Stefano: please report the error on the ML
> 
> ACTION: Bosch to send a bug report regarding xen-swiotlb
> 
> Edgar:  IOMMU could not be used by the guest (Stage-1). This would be useful
>         to implement driver in userspace.
> Julien: When will it be required?
> Edgar:  It is a trend
> 
> Julien: Should we organized another Community Call? When?
> Artem: Once per month is a good start
> 
> All: Agreed on a call before Christmas
> 
> [1] https://lists.xenproject.org/archives/html/xen-devel/2016-10/msg01966.html
> [2] https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01802.html
> [3] https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg02220.html
> [4] https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg00846.html
> [5] http://people.linaro.org/~christoffer.dall/VMSystemSpecificationForARM-v2.0.pdf
> 
> -- 
> Julien Grall
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Edgar E. Iglesias]

On Tue, Dec 06, 2016 at 11:02:20AM -0800, Stefano Stabellini wrote:
> On Tue, 6 Dec 2016, Julien Grall wrote:
> > Hi all,
> > 
> > Apologies for the late sending. You will find at the end of the mail a summary
> > of the discussion. Feel free to reply if I missed some parts.
> > 
> > At the end of the call we agreed to have another meeting before the end of the
> > year. Given that the Christmas is approaching and some people may take holidays
> > around, I would suggest to do the meeting on Wednesday 14th December at 4pm
> > UTC. Any opinions?
> 
> Would it be possible to move this call to 5pm UTC (this one, not
> necessarily all the future calls)?

Hi All,

I'm up for a call the 14th. 4 or 5pm UTC both work for me.

Best regards,
Edgar


> 
> 
> 
> > Also, do you have any specific topic you would like to talk during the next call?
> > 
> > Cheers,
> > 
> > == Attendees ==
> > 
> > Apologies if I misspelled any name.
> > 
> > * Stefano Stabellini:
> >     Aporeto, Xen ARM maintainer
> > * Julien Grall
> >     ARM, Xen ARM maintainer
> > * Artem Mygaiev, Andrii Anisov, Alexandre Agizim:
> >     EPAM, Embedded and automotive application for Xen
> > * Steward Hildebrand:
> >     Dornerworks, help customer to use Xen in their product
> > * Edgar Iglesias:
> >     Xilinx, Embedded and datacenter application
> > * Meng Xu:
> >     University of Pennsylvania, RT Xen. Looking for improving performance for
> >     real-time application in virtualised environment
> > * Bosch (sorry I forgot the name of the attendees): Car multimedia
> > * Anastassios Nanos:
> >     OnApp, using Xen on lower power server
> > * Dario Faggioli:
> >     Citrix, scheduler maintainer. Interested in following big.LITTLE discussion
> > 
> > == Features ==
> > 
> > === Co-processor architecture ===
> > 
> > Artem:  EPAM is working on a co-processor framework to share resources between
> >         guests (see a RFC of the design document [1]). A prototype has been
> >         created by sharing a GPU between guests, the overhead is ~5% compare
> >         to native.
> > Julien: concerned about context switch time
> > Stefano: concerned about the size of the emulator and security impact
> > Edgar:  it might not be possible to know the FPGA accelerator when building
> >         Xen.
> > Stefano: having the emulator in Xen EL1 would be better for protection
> > Andrii: it is not necessary to implement a full co-processor emulator. It is
> >         sufficient to emulation the behavior on register write.
> > 
> > ACTION: Continue the discussion on the mailing list.
> > 
> > === big.LITTLE ===
> > 
> > Anastassios:    interested in knowing the status of big.LITTLE support in Xen
> > Dario:          went through the thread on the ML. The consensus seems to be
> >                 based on vcpu pin/affinity.
> > Anastassios:    There are issue the way xen handles boot code. Wrong errata
> >                 for cpus.
> > Julien:         Core could have different errata and features. Errata already
> >                 works today.
> >                 We need a summary of the discussion.
> > 
> > Dario stepped in to write a summary.
> > 
> > Anastassios:    What is the best board to work big.LITTLE with Xen?
> > 
> > ?:              Renesas
> > 
> > ACTION: Dario to write a summary of the big.LITTLE discussions.
> > 
> > === Secure Call Monitor (SMC) from guests ===
> > 
> > Andrii/Artem:   EPAM is working on allowing a guest to make call to TEE (e.g
> >                 OPTEE). They are working in collaboration with Linaro to
> >                 make OPTEE  virtualization aware. A design document has been
> >                 posted on the ML (see [3]).
> > Edgar:          interested on this. Trusted world need to be accessed to
> >                 manage FPGA and for power management
> > 
> > === Running unmodified baremetal software in a guest ===
> > 
> > Edgar:          Xilinx is working on running unmodified baremetal software
> >                 in a guest. Piece of work required:
> >                     - Mapping memory area with different memory attribut
> >                       to domU
> >                     - Replicate host memory map
> >                     - Exposing a vUART to guest (see [4] for the discussion)
> > Steward:        vUART is very important, especially for baremetal guests
> > Stefano:        it would need to be loggable
> > Julien:         we would have the same issue in the future to be compliant
> >                 with the VM Spec (see [5]).
> > 
> > === Areas of concern ===
> > 
> > Bosch: problem with xen-swiotlb. It does not work properly on renesas board.
> > Stefano: please report the error on the ML
> > 
> > ACTION: Bosch to send a bug report regarding xen-swiotlb
> > 
> > Edgar:  IOMMU could not be used by the guest (Stage-1). This would be useful
> >         to implement driver in userspace.
> > Julien: When will it be required?
> > Edgar:  It is a trend
> > 
> > Julien: Should we organized another Community Call? When?
> > Artem: Once per month is a good start
> > 
> > All: Agreed on a call before Christmas
> > 
> > [1] https://lists.xenproject.org/archives/html/xen-devel/2016-10/msg01966.html
> > [2] https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01802.html
> > [3] https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg02220.html
> > [4] https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg00846.html
> > [5] http://people.linaro.org/~christoffer.dall/VMSystemSpecificationForARM-v2.0.pdf
> > 
> > -- 
> > Julien Grall
> > 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3949 bytes --]

On Tue, 2016-12-06 at 13:48 +0000, Julien Grall wrote:
> === big.LITTLE ===
> 
> Anastassios:    interested in knowing the status of big.LITTLE
> support in Xen
> Dario:          went through the thread on the ML. The consensus
> seems to be
>                 based on vcpu pin/affinity.
> Anastassios:    There are issue the way xen handles boot code. Wrong
> errata
>                 for cpus.
> Julien:         Core could have different errata and features. Errata
> already
>                 works today.
>                 We need a summary of the discussion.
> 
> Dario stepped in to write a summary.
> 
> Anastassios:    What is the best board to work big.LITTLE with Xen?
> 
> ?:              Renesas
> 
> ACTION: Dario to write a summary of the big.LITTLE discussions.
> 
Here it is. I took the chance and wrote it as a design / feature
document, but it still has some TODO, for things that were either not
clear or not fully agreed upon in the thread.

 [DOC RFC] Heterogeneous Multi Processing Support in Xen
 <1481135379.3445.142.camel@citrix.com>

(I wanted to post a link to the message in the archives, but I'm not
seeing it there yet... oh, well.)

Hope this helps. :-)

Regards,
Dario

> === Secure Call Monitor (SMC) from guests ===
> 
> Andrii/Artem:   EPAM is working on allowing a guest to make call to
> TEE (e.g
>                 OPTEE). They are working in collaboration with Linaro
> to
>                 make OPTEE  virtualization aware. A design document
> has been
>                 posted on the ML (see [3]).
> Edgar:          interested on this. Trusted world need to be accessed
> to
>                 manage FPGA and for power management
> 
> === Running unmodified baremetal software in a guest ===
> 
> Edgar:          Xilinx is working on running unmodified baremetal
> software
>                 in a guest. Piece of work required:
>                     - Mapping memory area with different memory
> attribut
>                       to domU
>                     - Replicate host memory map
>                     - Exposing a vUART to guest (see [4] for the
> discussion)
> Steward:        vUART is very important, especially for baremetal
> guests
> Stefano:        it would need to be loggable
> Julien:         we would have the same issue in the future to be
> compliant
>                 with the VM Spec (see [5]).
> 
> === Areas of concern ===
> 
> Bosch: problem with xen-swiotlb. It does not work properly on renesas
> board.
> Stefano: please report the error on the ML
> 
> ACTION: Bosch to send a bug report regarding xen-swiotlb
> 
> Edgar:  IOMMU could not be used by the guest (Stage-1). This would be
> useful
>         to implement driver in userspace.
> Julien: When will it be required?
> Edgar:  It is a trend
> 
> Julien: Should we organized another Community Call? When?
> Artem: Once per month is a good start
> 
> All: Agreed on a call before Christmas
> 
> [1] https://lists.xenproject.org/archives/html/xen-devel/2016-10/msg0
> 1966.html
> [2] https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg0
> 1802.html
> [3] https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg0
> 2220.html
> [4] https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg0
> 0846.html
> [5] http://people.linaro.org/~christoffer.dall/VMSystemSpecificationF
> orARM-v2.0.pdf
> 
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 641 bytes --]

On Wed, 2016-12-07 at 19:38 +0100, Dario Faggioli wrote:
>  [DOC RFC] Heterogeneous Multi Processing Support in Xen
>  <1481135379.3445.142.camel@citrix.com>
> 
> (I wanted to post a link to the message in the archives, but I'm not
> seeing it there yet... oh, well.)
>
It appeared:

https://lists.xen.org/archives/html/xen-devel/2016-12/msg00826.html

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

Hi all,

Stefano suggested to move the call at 5pm and I haven't seen any 
disagreement.

So the call will be tomorrow (Wednesday 14th December at 5pm).

The conference details are the same as last time. For reminder:

Use either of the following to join the call:
Call            +44 1223 406065 (Local dial in)
and enter the access code below followed by # key.
Participant code: 4915191

Mobile Auto Dial:
         VoIP: voip://+441223406065;4915191#
         iOS devices: +44 1223 406065,4915191 and press #
         Other devices: +44 1223 406065x4915191#

Additional Calling Information:

UK +44 1142828002
US CA +1 4085761502
US TX +1 5123141073
JP +81 453455355
DE +49 8945604050
NO +47 73187518
SE +46 46313131
FR +33 497235101
TW +886 35657119
HU +36 13275600
IE +353 91337900

Toll Free

UK 0800 1412084
US +1 8668801148
CN +86 4006782367
IN 0008009868365
IN +918049282778
TW 08000 22065
HU 0680981587
IE 1800800022
KF +972732558877

Cheers,

On 06/12/16 13:48, Julien Grall wrote:
> Hi all,
>
> Apologies for the late sending. You will find at the end of the mail a summary
> of the discussion. Feel free to reply if I missed some parts.
>
> At the end of the call we agreed to have another meeting before the end of the
> year. Given that the Christmas is approaching and some people may take holidays
> around, I would suggest to do the meeting on Wednesday 14th December at 4pm
> UTC. Any opinions?
>
> Also, do you have any specific topic you would like to talk during the next call?
>
> Cheers,
>
> == Attendees ==
>
> Apologies if I misspelled any name.
>
> * Stefano Stabellini:
>     Aporeto, Xen ARM maintainer
> * Julien Grall
>     ARM, Xen ARM maintainer
> * Artem Mygaiev, Andrii Anisov, Alexandre Agizim:
>     EPAM, Embedded and automotive application for Xen
> * Steward Hildebrand:
>     Dornerworks, help customer to use Xen in their product
> * Edgar Iglesias:
>     Xilinx, Embedded and datacenter application
> * Meng Xu:
>     University of Pennsylvania, RT Xen. Looking for improving performance for
>     real-time application in virtualised environment
> * Bosch (sorry I forgot the name of the attendees): Car multimedia
> * Anastassios Nanos:
>     OnApp, using Xen on lower power server
> * Dario Faggioli:
>     Citrix, scheduler maintainer. Interested in following big.LITTLE discussion
>
> == Features ==
>
> === Co-processor architecture ===
>
> Artem:  EPAM is working on a co-processor framework to share resources between
>         guests (see a RFC of the design document [1]). A prototype has been
>         created by sharing a GPU between guests, the overhead is ~5% compare
>         to native.
> Julien: concerned about context switch time
> Stefano: concerned about the size of the emulator and security impact
> Edgar:  it might not be possible to know the FPGA accelerator when building
>         Xen.
> Stefano: having the emulator in Xen EL1 would be better for protection
> Andrii: it is not necessary to implement a full co-processor emulator. It is
>         sufficient to emulation the behavior on register write.
>
> ACTION: Continue the discussion on the mailing list.
>
> === big.LITTLE ===
>
> Anastassios:    interested in knowing the status of big.LITTLE support in Xen
> Dario:          went through the thread on the ML. The consensus seems to be
>                 based on vcpu pin/affinity.
> Anastassios:    There are issue the way xen handles boot code. Wrong errata
>                 for cpus.
> Julien:         Core could have different errata and features. Errata already
>                 works today.
>                 We need a summary of the discussion.
>
> Dario stepped in to write a summary.
>
> Anastassios:    What is the best board to work big.LITTLE with Xen?
>
> ?:              Renesas
>
> ACTION: Dario to write a summary of the big.LITTLE discussions.
>
> === Secure Call Monitor (SMC) from guests ===
>
> Andrii/Artem:   EPAM is working on allowing a guest to make call to TEE (e.g
>                 OPTEE). They are working in collaboration with Linaro to
>                 make OPTEE  virtualization aware. A design document has been
>                 posted on the ML (see [3]).
> Edgar:          interested on this. Trusted world need to be accessed to
>                 manage FPGA and for power management
>
> === Running unmodified baremetal software in a guest ===
>
> Edgar:          Xilinx is working on running unmodified baremetal software
>                 in a guest. Piece of work required:
>                     - Mapping memory area with different memory attribut
>                       to domU
>                     - Replicate host memory map
>                     - Exposing a vUART to guest (see [4] for the discussion)
> Steward:        vUART is very important, especially for baremetal guests
> Stefano:        it would need to be loggable
> Julien:         we would have the same issue in the future to be compliant
>                 with the VM Spec (see [5]).
>
> === Areas of concern ===
>
> Bosch: problem with xen-swiotlb. It does not work properly on renesas board.
> Stefano: please report the error on the ML
>
> ACTION: Bosch to send a bug report regarding xen-swiotlb
>
> Edgar:  IOMMU could not be used by the guest (Stage-1). This would be useful
>         to implement driver in userspace.
> Julien: When will it be required?
> Edgar:  It is a trend
>
> Julien: Should we organized another Community Call? When?
> Artem: Once per month is a good start
>
> All: Agreed on a call before Christmas
>
> [1] https://lists.xenproject.org/archives/html/xen-devel/2016-10/msg01966.html
> [2] https://lists.xenproject.org/archives/html/xen-devel/2016-09/msg01802.html
> [3] https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg02220.html
> [4] https://lists.xenproject.org/archives/html/xen-devel/2016-11/msg00846.html
> [5] http://people.linaro.org/~christoffer.dall/VMSystemSpecificationForARM-v2.0.pdf
>

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 876 bytes --]

On Tue, 2016-12-13 at 19:00 +0000, Julien Grall wrote:
> Hi all,
> 
> Stefano suggested to move the call at 5pm and I haven't seen any 
> disagreement.
> 
> So the call will be tomorrow (Wednesday 14th December at 5pm).
> 
Hey, I'm not sure I will be able join today.

Both my interest and my contribution to the discussion is all about
big.LITTLE (and, in general, heterogeneous processing support), for
which I sent a design doc a few days ago.

I'll keep an eye out for the minute. Feel free to forward to me any
question and/or bring me in any discussion related to such topic.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Edgar E. Iglesias]

On Wed, Dec 14, 2016 at 03:59:00PM +0100, Dario Faggioli wrote:
> On Tue, 2016-12-13 at 19:00 +0000, Julien Grall wrote:
> > Hi all,
> > 
> > Stefano suggested to move the call at 5pm and I haven't seen any 
> > disagreement.
> > 
> > So the call will be tomorrow (Wednesday 14th December at 5pm).


Hi all,

Thanks for the call today. I'm sending a link to EEMI the
power management framework that Xilinx co-developed with Aggios.
I don't have a link to the ARM specs for SCMI. I found some slides
though. Perhaps Julien has specs?

SCMI:
http://www.slideshare.net/linaroorg/las16200-scmi-system-management-and-control-interface

EEMI:
https://www.xilinx.com/support/documentation/user_guides/ug1200-eemi-api.pdf

Roughly, EEMI works top down by having the various SW layers request
power down/up and other settings to the layer beneath. The various
layers are responsible for recounting so that we don't shut down
something that is in use and also to isolate so that we don't
turn on or off anything that we don't own.

At the moment we tunnel API calls via SMC calls to EL3. For calls
that ATF at EL3 can implement it answers directly. If not, these
calls further propagate over a shared memory / IPI interface to
a dedicated tiny core (PMU) with power management firmware.

The PMU is responsible for mediating power/clock request changes
and making sure that the requesters don't mess with devices or
settings that don't belong to them.

So top down the idea is:

EL0  User-space uses devices (open/close or whatever).
EL1  OS/Kernel refcounts per process and requests power down/up of devs when not used.
EL2  Hypervisor (or dom0 or something not yet implemented) refcounts per guest and forwards the request down the chain if appropriate.
EL3  Trusted Firmware mediates between Normal and Secure world.

PMU  Mediates between different CPU clusters and Programmable logic.

Best regards,
Edgar





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

On 14/12/16 21:18, Edgar E. Iglesias wrote:
> On Wed, Dec 14, 2016 at 03:59:00PM +0100, Dario Faggioli wrote:
>> On Tue, 2016-12-13 at 19:00 +0000, Julien Grall wrote:
>>> Hi all,
>>>
>>> Stefano suggested to move the call at 5pm and I haven't seen any
>>> disagreement.
>>>
>>> So the call will be tomorrow (Wednesday 14th December at 5pm).
>
>
> Hi all,

Hi Edgar,

>
> Thanks for the call today. I'm sending a link to EEMI the
> power management framework that Xilinx co-developed with Aggios.
> I don't have a link to the ARM specs for SCMI. I found some slides
> though. Perhaps Julien has specs?

Thank you for the link, I will add them in the meeting minutes.

Regarding SCMI, the spec is not yet public but can be downloaded under 
NDA (see the link at the end of the slides you sent).

Cheers,

-- 
Julien Grall


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Andrii Anisov]

Julien, Stefano,

Are there any updates about:

> ACTION: Bosch to send a bug report regarding xen-swiotlb
>
> Edgar:  IOMMU could not be used by the guest (Stage-1). This would be useful
>         to implement driver in userspace.
> Julien: When will it be required?
> Edgar:  It is a trend

Any mailing threads or discussions?

-- 

*Andrii Anisov*

*Lead Systems Engineer*

*Office: *+380 44 390 5457 <tel:+380%2044%20390%205457> *x* 66766 
<tel:66766>*Cell: *+380 50 5738852 <tel:+380%2050%205738852>*Email: 
*andrii_anisov@epam.com <mailto:andrii_anisov@epam.com>

*Kyiv**,* *Ukraine *(GMT+3)*epam.com <http://www.epam.com>*

CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or 
entity(ies) to which it is addressed and contains information that is 
legally privileged and confidential. If you are not the intended 
recipient, or the person responsible for delivering the message to the 
intended recipient, you are hereby notified that any dissemination, 
distribution or copying of this communication is strictly prohibited. 
All unintended recipients are obliged to delete this message and destroy 
any printed copies.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Andrii Anisov]

Sorry for the mess,

I mean the xen-swiotlb issue on renesas board:

 > Bosch: problem with xen-swiotlb. It does not work properly on renesas 
board.
 > Stefano: please report the error on the ML
 >
 > ACTION: Bosch to send a bug report regarding xen-swiotlb

-- 

*Andrii Anisov*

*Lead Systems Engineer*

*Office: *+380 44 390 5457 <tel:+380%2044%20390%205457> *x* 66766 
<tel:66766>*Cell: *+380 50 5738852 <tel:+380%2050%205738852>*Email: 
*andrii_anisov@epam.com <mailto:andrii_anisov@epam.com>

*Kyiv**,* *Ukraine *(GMT+3)*epam.com <http://www.epam.com>*

CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or 
entity(ies) to which it is addressed and contains information that is 
legally privileged and confidential. If you are not the intended 
recipient, or the person responsible for delivering the message to the 
intended recipient, you are hereby notified that any dissemination, 
distribution or copying of this communication is strictly prohibited. 
All unintended recipients are obliged to delete this message and destroy 
any printed copies.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Julien Grall]

Hi Andrii,

On 20/12/2016 19:00, Andrii Anisov wrote:
> Sorry for the mess,
>
> I mean the xen-swiotlb issue on renesas board:
>
>> Bosch: problem with xen-swiotlb. It does not work properly on renesas
> board.
>> Stefano: please report the error on the ML
>>
>> ACTION: Bosch to send a bug report regarding xen-swiotlb

No news so far. Dirk, do you have any update on this?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Dirk Behme]

On 20.12.2016 19:01, Julien Grall wrote:
> Hi Andrii,
>
> On 20/12/2016 19:00, Andrii Anisov wrote:
>> Sorry for the mess,
>>
>> I mean the xen-swiotlb issue on renesas board:
>>
>>> Bosch: problem with xen-swiotlb. It does not work properly on renesas
>> board.
>>> Stefano: please report the error on the ML
>>>
>>> ACTION: Bosch to send a bug report regarding xen-swiotlb
>
> No news so far. Dirk, do you have any update on this?


We will try to update as soon as the vacation season is over, 
beginning of next year. Together with Pooya (the Bosch guy who 
attended the first call) I'll try to summarize the issues we see, then.

I don't have the details atm. But if I remember correctly, I've heard 
about two (?) issues (haven't done the tests myself).

The first one was related to (e.g. USB) DMA depending how we configure 
the memory assignments on the Renesas Salvator-X board. Depending on 
how much memory we assign to Xen and Linux, e.g. USB wasn't working 
(trying to use an USB stick as rootfs).

The other issue I heard about was some root file system corruptions 
after two or three re-boots we haven't observed in the native Linux 
case. The plan was to do some further analysis, first, before we blame 
Xen regarding this, though.

As mentioned, Pooya will have the details and correct me if I'm 
totally wrong here ;)

Best regards

Dirk

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Andrii Anisov]

Hello Dirk,

Except those two issues did you face some issues with PVR KM?
 
ANDRII ANISOV
Lead Systems Engineer
   
Office: +380  44 390 5457 x 66766   Cell: +380  50 573 8852   Email: andrii_anisov@epam.com
Kyiv, Ukraine (GMT+3)   epam.com
   
CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the  message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.
     

​    
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Pooya.Keshavarzi]

Hi,

On 01/03/2017 12:33 PM, Dirk Behme wrote:
> On 20.12.2016 19:01, Julien Grall wrote:
>> Hi Andrii,
>>
>> On 20/12/2016 19:00, Andrii Anisov wrote:
>>> Sorry for the mess,
>>>
>>> I mean the xen-swiotlb issue on renesas board:
>>>
>>>> Bosch: problem with xen-swiotlb. It does not work properly on renesas
>>> board.
>>>> Stefano: please report the error on the ML
>>>>
>>>> ACTION: Bosch to send a bug report regarding xen-swiotlb
>>
>> No news so far. Dirk, do you have any update on this?
> 
> 
> We will try to update as soon as the vacation season is over, beginning of next year. Together with Pooya (the Bosch guy who attended the first call) I'll try to summarize the issues we see, then.
> 
> I don't have the details atm. But if I remember correctly, I've heard about two (?) issues (haven't done the tests myself).
> 
> The first one was related to (e.g. USB) DMA depending how we configure the memory assignments on the Renesas Salvator-X board. Depending on how much memory we assign to Xen and Linux, e.g. USB wasn't working (trying to use an USB stick as rootfs).
> 
> The other issue I heard about was some root file system corruptions after two or three re-boots we haven't observed in the native Linux case. The plan was to do some further analysis, first, before we blame Xen regarding this, though.
> 
> As mentioned, Pooya will have the details and correct me if I'm totally wrong here ;)
> 

Firstly sorry for the late reply on this.

Regarding the problem with swiotlb-xen here are some more details:

If we limit Dom0's memory such that only low-memory (up to 32-bit addressable memory) is available to Dom0, then swiotlb-xen does not have to use bounce buffers and the devices (e.g. USB, ethernet) would work.

But when there is some high memory also available to Dom0, the followings happen:
 - If the the device address happens to be in the device's DMA window (see xen_swiotlb_map_page()), then the device would work.
 - Otherwise if it has to allocate and map a bounce buffer, then the device would not work.

Kind regards,
Pooya

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Stefano Stabellini]

On Thu, 12 Jan 2017, Pooya.Keshavarzi wrote:
> > The other issue I heard about was some root file system corruptions after two or three re-boots we haven't observed in the native Linux case. The plan was to do some further analysis, first, before we blame Xen regarding this, though.
> > 
> > As mentioned, Pooya will have the details and correct me if I'm totally wrong here ;)
> > 
> 
> Firstly sorry for the late reply on this.
> 
> Regarding the problem with swiotlb-xen here are some more details:
> 
> If we limit Dom0's memory such that only low-memory (up to 32-bit addressable memory) is available to Dom0, then swiotlb-xen does not have to use bounce buffers and the devices (e.g. USB, ethernet) would work.
> 
> But when there is some high memory also available to Dom0, the followings happen:
>  - If the the device address happens to be in the device's DMA window (see xen_swiotlb_map_page()), then the device would work.
>  - Otherwise if it has to allocate and map a bounce buffer, then the device would not work.

From what you wrote it looks like the xen_swiotlb_map_page path: 

	if (dma_capable(dev, dev_addr, size) &&
	    !range_straddles_page_boundary(phys, size) &&
		!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
		!swiotlb_force) {
		/* we are not interested in the dma_addr returned by
		 * xen_dma_map_page, only in the potential cache flushes executed
		 * by the function. */
		xen_dma_map_page(dev, page, dev_addr, offset, size, dir, attrs);
		return dev_addr;
	}

works, but the other does not. Does it match your understanding? Have
you done any digging to find the reason why the bounce buffer code path
is broken on your platform?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Pooya.Keshavarzi]

On 01/12/2017 07:50 PM, Stefano Stabellini wrote:
> On Thu, 12 Jan 2017, Pooya.Keshavarzi wrote:
>>
>> Firstly sorry for the late reply on this.
>>
>> Regarding the problem with swiotlb-xen here are some more details:
>>
>> If we limit Dom0's memory such that only low-memory (up to 32-bit addressable memory) is available to Dom0, then swiotlb-xen does not have to use bounce buffers and the devices (e.g. USB, ethernet) would work.
>>
>> But when there is some high memory also available to Dom0, the followings happen:
>>  - If the the device address happens to be in the device's DMA window (see xen_swiotlb_map_page()), then the device would work.
>>  - Otherwise if it has to allocate and map a bounce buffer, then the device would not work.
> 
> From what you wrote it looks like the xen_swiotlb_map_page path: 
> 
> 	if (dma_capable(dev, dev_addr, size) &&
> 	    !range_straddles_page_boundary(phys, size) &&
> 		!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
> 		!swiotlb_force) {
> 		/* we are not interested in the dma_addr returned by
> 		 * xen_dma_map_page, only in the potential cache flushes executed
> 		 * by the function. */
> 		xen_dma_map_page(dev, page, dev_addr, offset, size, dir, attrs);
> 		return dev_addr;
> 	}
> 
> works, but the other does not. Does it match your understanding? Have
> you done any digging to find the reason why the bounce buffer code path
> is broken on your platform?

Yes, The above path works but the other one doesn't.
I did some digging but failed to find out what's the problem. The returned address from swiotlb_tbl_map_single() is within the memory range allocated earlier for Xen software IO TLB and is dma capable, so it seem to be OK.

What's your suggestion for further digging?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Stefano Stabellini]

On Fri, 13 Jan 2017, Pooya.Keshavarzi wrote:
> On 01/12/2017 07:50 PM, Stefano Stabellini wrote:
> > On Thu, 12 Jan 2017, Pooya.Keshavarzi wrote:
> >>
> >> Firstly sorry for the late reply on this.
> >>
> >> Regarding the problem with swiotlb-xen here are some more details:
> >>
> >> If we limit Dom0's memory such that only low-memory (up to 32-bit addressable memory) is available to Dom0, then swiotlb-xen does not have to use bounce buffers and the devices (e.g. USB, ethernet) would work.
> >>
> >> But when there is some high memory also available to Dom0, the followings happen:
> >>  - If the the device address happens to be in the device's DMA window (see xen_swiotlb_map_page()), then the device would work.
> >>  - Otherwise if it has to allocate and map a bounce buffer, then the device would not work.
> > 
> > From what you wrote it looks like the xen_swiotlb_map_page path: 
> > 
> > 	if (dma_capable(dev, dev_addr, size) &&
> > 	    !range_straddles_page_boundary(phys, size) &&
> > 		!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
> > 		!swiotlb_force) {
> > 		/* we are not interested in the dma_addr returned by
> > 		 * xen_dma_map_page, only in the potential cache flushes executed
> > 		 * by the function. */
> > 		xen_dma_map_page(dev, page, dev_addr, offset, size, dir, attrs);
> > 		return dev_addr;
> > 	}
> > 
> > works, but the other does not. Does it match your understanding? Have
> > you done any digging to find the reason why the bounce buffer code path
> > is broken on your platform?
> 
> Yes, The above path works but the other one doesn't.
> I did some digging but failed to find out what's the problem. The returned address from swiotlb_tbl_map_single() is within the memory range allocated earlier for Xen software IO TLB and is dma capable, so it seem to be OK.
> 
> What's your suggestion for further digging?

Is the device DMA coherent?
I take that it fails even without running any guests, correct?


A few things come to mind:

- In xen_dma_map_page, does it take the local path or the foreign path
  (if(local)...) when it fails?

- Check that xen_swiotlb_init initializes the swiotlb memory region
  appriopriately and that it falls in a memory range supported by the
  device.

- Check that xen_phys_to_bus(map) returns the right dev_addr. As a
  reference, I know that on some arm32 platforms it wouldn't return the
  right value.

- Does the following patch fixes the issue for you?

diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 87e6035..17c65fa 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -409,9 +409,9 @@ dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
 	if (map == SWIOTLB_MAP_ERROR)
 		return DMA_ERROR_CODE;
 
+	dev_addr = xen_phys_to_bus(map);
 	xen_dma_map_page(dev, pfn_to_page(map >> PAGE_SHIFT),
 					dev_addr, map & ~PAGE_MASK, size, dir, attrs);
-	dev_addr = xen_phys_to_bus(map);
 
 	/*
 	 * Ensure that the address returned is DMA'ble

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Pooya Keshavarzi]

Hi Stefano,

On 01/13/2017 07:39 PM, Stefano Stabellini wrote:
> On Fri, 13 Jan 2017, Pooya.Keshavarzi wrote:
>> On 01/12/2017 07:50 PM, Stefano Stabellini wrote:
>>> On Thu, 12 Jan 2017, Pooya.Keshavarzi wrote:
>>>>
>>>> Firstly sorry for the late reply on this.
>>>>
>>>> Regarding the problem with swiotlb-xen here are some more details:
>>>>
>>>> If we limit Dom0's memory such that only low-memory (up to 32-bit addressable memory) is available to Dom0, then swiotlb-xen does not have to use bounce buffers and the devices (e.g. USB, ethernet) would work.
>>>>
>>>> But when there is some high memory also available to Dom0, the followings happen:
>>>>  - If the the device address happens to be in the device's DMA window (see xen_swiotlb_map_page()), then the device would work.
>>>>  - Otherwise if it has to allocate and map a bounce buffer, then the device would not work.
>>>
>>> From what you wrote it looks like the xen_swiotlb_map_page path: 
>>>
>>> 	if (dma_capable(dev, dev_addr, size) &&
>>> 	    !range_straddles_page_boundary(phys, size) &&
>>> 		!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
>>> 		!swiotlb_force) {
>>> 		/* we are not interested in the dma_addr returned by
>>> 		 * xen_dma_map_page, only in the potential cache flushes executed
>>> 		 * by the function. */
>>> 		xen_dma_map_page(dev, page, dev_addr, offset, size, dir, attrs);
>>> 		return dev_addr;
>>> 	}
>>>
>>> works, but the other does not. Does it match your understanding? Have
>>> you done any digging to find the reason why the bounce buffer code path
>>> is broken on your platform?
>>
>> Yes, The above path works but the other one doesn't.
>> I did some digging but failed to find out what's the problem. The returned address from swiotlb_tbl_map_single() is within the memory range allocated earlier for Xen software IO TLB and is dma capable, so it seem to be OK.
>>
>> What's your suggestion for further digging?
> 
> Is the device DMA coherent?
No.

> I take that it fails even without running any guests, correct?
Yes, fails only with Dom0.

> A few things come to mind:
> 
> - In xen_dma_map_page, does it take the local path or the foreign path
>   (if(local)...) when it fails?
When it fails, it takes the foreign path.

> - Check that xen_swiotlb_init initializes the swiotlb memory region
>   appriopriately and that it falls in a memory range supported by the
>   device.
The only thing is this warning:
xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB

The allocation in the first memory bank can be solved by this, and then swiotlb-xen allocates in the proper memory range.
https://patchwork.kernel.org/patch/9347329/
We had a similar issue.


> - Check that xen_phys_to_bus(map) returns the right dev_addr. As a
>   reference, I know that on some arm32 platforms it wouldn't return the
>   right value.
It returns the same address as map.

 
> - Does the following patch fixes the issue for you?
Yes :) it seems that it solves the issue for Ethernet.
And now it takes the local path in xen_dma_map_page.

But why?

 
> diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> index 87e6035..17c65fa 100644
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -409,9 +409,9 @@ dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
>  	if (map == SWIOTLB_MAP_ERROR)
>  		return DMA_ERROR_CODE;
>  
> +	dev_addr = xen_phys_to_bus(map);
>  	xen_dma_map_page(dev, pfn_to_page(map >> PAGE_SHIFT),
>  					dev_addr, map & ~PAGE_MASK, size, dir, attrs);
> -	dev_addr = xen_phys_to_bus(map);
>  
>  	/*
>  	 * Ensure that the address returned is DMA'ble 


And following patch solves the issue for USB, MMC, SDC.

diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 7399782..69b5c62 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -567,6 +567,7 @@ xen_swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
 				sg_dma_len(sgl) = 0;
 				return 0;
 			}
+			dev_addr = xen_phys_to_bus(map);
 			xen_dma_map_page(hwdev, pfn_to_page(map >> PAGE_SHIFT),
 						dev_addr,
 						map & ~PAGE_MASK,


BR,
Pooya

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Stefano Stabellini]

On Thu, 19 Jan 2017, Pooya Keshavarzi wrote:
> Hi Stefano,
> 
> On 01/13/2017 07:39 PM, Stefano Stabellini wrote:
> > On Fri, 13 Jan 2017, Pooya.Keshavarzi wrote:
> >> On 01/12/2017 07:50 PM, Stefano Stabellini wrote:
> >>> On Thu, 12 Jan 2017, Pooya.Keshavarzi wrote:
> >>>>
> >>>> Firstly sorry for the late reply on this.
> >>>>
> >>>> Regarding the problem with swiotlb-xen here are some more details:
> >>>>
> >>>> If we limit Dom0's memory such that only low-memory (up to 32-bit addressable memory) is available to Dom0, then swiotlb-xen does not have to use bounce buffers and the devices (e.g. USB, ethernet) would work.
> >>>>
> >>>> But when there is some high memory also available to Dom0, the followings happen:
> >>>>  - If the the device address happens to be in the device's DMA window (see xen_swiotlb_map_page()), then the device would work.
> >>>>  - Otherwise if it has to allocate and map a bounce buffer, then the device would not work.
> >>>
> >>> From what you wrote it looks like the xen_swiotlb_map_page path: 
> >>>
> >>> 	if (dma_capable(dev, dev_addr, size) &&
> >>> 	    !range_straddles_page_boundary(phys, size) &&
> >>> 		!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
> >>> 		!swiotlb_force) {
> >>> 		/* we are not interested in the dma_addr returned by
> >>> 		 * xen_dma_map_page, only in the potential cache flushes executed
> >>> 		 * by the function. */
> >>> 		xen_dma_map_page(dev, page, dev_addr, offset, size, dir, attrs);
> >>> 		return dev_addr;
> >>> 	}
> >>>
> >>> works, but the other does not. Does it match your understanding? Have
> >>> you done any digging to find the reason why the bounce buffer code path
> >>> is broken on your platform?
> >>
> >> Yes, The above path works but the other one doesn't.
> >> I did some digging but failed to find out what's the problem. The returned address from swiotlb_tbl_map_single() is within the memory range allocated earlier for Xen software IO TLB and is dma capable, so it seem to be OK.
> >>
> >> What's your suggestion for further digging?
> > 
> > Is the device DMA coherent?
> No.
> 
> > I take that it fails even without running any guests, correct?
> Yes, fails only with Dom0.
> 
> > A few things come to mind:
> > 
> > - In xen_dma_map_page, does it take the local path or the foreign path
> >   (if(local)...) when it fails?
> When it fails, it takes the foreign path.
> 
> > - Check that xen_swiotlb_init initializes the swiotlb memory region
> >   appriopriately and that it falls in a memory range supported by the
> >   device.
> The only thing is this warning:
> xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
> 
> The allocation in the first memory bank can be solved by this, and then swiotlb-xen allocates in the proper memory range.
> https://patchwork.kernel.org/patch/9347329/
> We had a similar issue.
> 
> 
> > - Check that xen_phys_to_bus(map) returns the right dev_addr. As a
> >   reference, I know that on some arm32 platforms it wouldn't return the
> >   right value.
> It returns the same address as map.
> 
>  
> > - Does the following patch fixes the issue for you?
> Yes :) it seems that it solves the issue for Ethernet.
> And now it takes the local path in xen_dma_map_page.
> 
> But why?

This is a genuine bug in swiotlb-xen. xen_swiotlb_map_page swaps the
original page for one from the swiotlb pool. The call to
swiotlb_tbl_map_single returns the new page. However, we are calling
xen_dma_map_page passing the dev_addr of the original page, rather than
the new one. xen_dma_map_page wrongly assumes that the page is foreign
because phys_addr and dma_addr don't match.

Thank you for reporting the issue and for testing. I'll submit a patch
shortly. I'll add your signed-off-by to it, if that's OK.


  
> > diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> > index 87e6035..17c65fa 100644
> > --- a/drivers/xen/swiotlb-xen.c
> > +++ b/drivers/xen/swiotlb-xen.c
> > @@ -409,9 +409,9 @@ dma_addr_t xen_swiotlb_map_page(struct device *dev, struct page *page,
> >  	if (map == SWIOTLB_MAP_ERROR)
> >  		return DMA_ERROR_CODE;
> >  
> > +	dev_addr = xen_phys_to_bus(map);
> >  	xen_dma_map_page(dev, pfn_to_page(map >> PAGE_SHIFT),
> >  					dev_addr, map & ~PAGE_MASK, size, dir, attrs);
> > -	dev_addr = xen_phys_to_bus(map);
> >  
> >  	/*
> >  	 * Ensure that the address returned is DMA'ble 
> 
> 
> And following patch solves the issue for USB, MMC, SDC.
> 
> diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> index 7399782..69b5c62 100644
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -567,6 +567,7 @@ xen_swiotlb_map_sg_attrs(struct device *hwdev, struct scatterlist *sgl,
>  				sg_dma_len(sgl) = 0;
>  				return 0;
>  			}
> +			dev_addr = xen_phys_to_bus(map);
>  			xen_dma_map_page(hwdev, pfn_to_page(map >> PAGE_SHIFT),
>  						dev_addr,
>  						map & ~PAGE_MASK,
> 
> 
> BR,
> Pooya
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Andrii Anisov]

Dear Pooya,

On our site we did not face those issues 'cause we limited dom0 memory space to the 32-bit addressable range.
BTW, it looks you use nfs root in you setups, do you?

Issues we faced and got fixes/workarounds was as following:
MMC driver did not work due to dma_mmap fixed with the code similar to https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg00874.html
PVR KM suffered from two issues fixed by https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg00873.html and https://github.com/xen-troops/pvr_km/commit/b413e02af3df6e69afc4070a247fa740e32819ba
  
 
  ANDRII ANISOV
  Lead Systems Engineer
   
  Office: +380  44 390 5457 x 66766   Cell: +380  50 573 8852   Email: andrii_anisov@epam.com
  Kyiv, Ukraine (GMT+3)   epam.com
   
  CONFIDENTIALITY CAUTION AND DISCLAIMER
 This message is intended only for the use of the individual(s) or entity(ies) to which it is addressed and contains information that is legally privileged and confidential. If you are not the intended recipient, or the person responsible for delivering the  message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. All unintended recipients are obliged to delete this message and destroy any printed copies.




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Pooya.Keshavarzi]

Dear Andrii,

On 01/13/2017 11:47 AM, Andrii Anisov wrote:
> 
> Dear Pooya,
> 
> On our site we did not face those issues 'cause we limited dom0 memory space to the 32-bit addressable range.
Could you give it a try? Maybe there is something wrong on our side and it works for you.

> BTW, it looks you use nfs root in you setups, do you?
yes, we've tried from nfs and also USB to mount root.

> 
> Issues we faced and got fixes/workarounds was as following:
> MMC driver did not work due to dma_mmap fixed with the code similar to https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg00874.html
If I'm not wrong I think MMC worked for us. I have to double check this since it's been a while.

> PVR KM suffered from two issues fixed by https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg00873.html and https://github.com/xen-troops/pvr_km/commit/b413e02af3df6e69afc4070a247fa740e32819ba
>   

BR,
Pooya

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Xen ARM community call - meeting minutes and date for the next one

[Konrad Rzeszutek Wilk]

On Tue, Dec 20, 2016 at 08:00:27PM +0200, Andrii Anisov wrote:
> Sorry for the mess,
> 
> I mean the xen-swiotlb issue on renesas board:

Can you make sure to CC me on it (since I am the maintainer of that code).
> 
> > Bosch: problem with xen-swiotlb. It does not work properly on renesas
> board.
> > Stefano: please report the error on the ML
> >
> > ACTION: Bosch to send a bug report regarding xen-swiotlb

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel