RT-Xen on ARM

RT-Xen on ARM

[Andrii Anisov]

Dear Meng Xu,

We are going to evaluate an RTDS scheduler on ARM.

Basically I'm going to repeat use-cases described in 
https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf in some 
amount.

Do you have any recommendations or suggestions?


BTW, even following 
https://xenbits.xen.org/docs/unstable/features/sched_rtds.html I've 
faced several issues, not rtds one, but nasty:

- no xentop for ARM

- root@salvator-x-h3-xt:/scripts# xl sched-rtds -d DomU -v all -p 10000 
-b 2500
   (XEN) FLASK: Denying unknown domctl_scheduler_op: 2.
   libxl: error: libxl_sched.c:663:sched_rtds_vcpu_set_all: Domain 
2:Setting vcpu sched rtds: Operation not permitted
   libxl_vcpu_sched_params_set_all failed.

-- 

*Andrii Anisov*



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Wei Liu]

On Mon, Jul 03, 2017 at 02:03:33PM +0300, Andrii Anisov wrote:
> Dear Meng Xu,
> 
> We are going to evaluate an RTDS scheduler on ARM.
> 
> Basically I'm going to repeat use-cases described in
> https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf in some
> amount.
> 
> Do you have any recommendations or suggestions?
> 
> 
> BTW, even following
> https://xenbits.xen.org/docs/unstable/features/sched_rtds.html I've faced
> several issues, not rtds one, but nasty:
> 
> - no xentop for ARM

This is a bit strange, fc658208e026242420b9924a9e4bfa581479e1f5 seems to
imply xentop should work on ARM.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

On 03.07.17 16:03, Wei Liu wrote:
> This is a bit strange, fc658208e026242420b9924a9e4bfa581479e1f5 seems to
> imply xentop should work on ARM.
Yep it is built.
But yocto missed its installation. That issue is on our site :)

-- 

*Andrii Anisov*



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

On Mon, Jul 3, 2017 at 7:03 AM, Andrii Anisov <andrii_anisov@epam.com> wrote:
>
> Dear Meng Xu,


Hi Andrii,
>
>
> We are going to evaluate an RTDS scheduler on ARM.
>
>
> Basically I'm going to repeat use-cases described in https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf in some amount.


I see. Please don't hesitate to ask if you have any question about the results.

>
> Do you have any recommendations or suggestions?


Which experiment/use case do you plan to run?
What are the requirements (or performance guarantees) you want to have
from RTDS?


The configuration for the VCPUs depends on the tasks and the OS
scheduler running on the VCPUs.
The VCPU's utilization (budget/period) is usually larger than the
tasks' utilizations (\sum e_i / p_i), where e_i is the task's
worst-case execution time, and p_i is the task's period.
The VCPU's parameters can be calculated by the CARTS tool [1],

[1] https://rtg.cis.upenn.edu/carts/index.php


>
>
> BTW, even following https://xenbits.xen.org/docs/unstable/features/sched_rtds.html I've faced several issues, not rtds one, but nasty:
>
> - no xentop for ARM


Did you try "sudo xl top"?
IIRC, sudo xl top should work.
>
>
> - root@salvator-x-h3-xt:/scripts# xl sched-rtds -d DomU -v all -p 10000 -b 2500
>   (XEN) FLASK: Denying unknown domctl_scheduler_op: 2.
>   libxl: error: libxl_sched.c:663:sched_rtds_vcpu_set_all: Domain 2:Setting vcpu sched rtds: Operation not permitted
>   libxl_vcpu_sched_params_set_all failed.


Which version of Xen or commit point did you use?

Thanks,

Meng



-- 
-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

Hello Meng Xu,


On 03.07.17 16:35, Meng Xu wrote:
>> Do you have any recommendations or suggestions?
> Which experiment/use case do you plan to run?
> What are the requirements (or performance guarantees) you want to have
> from RTDS?
Currently we have no defined target use-cases.
That's why we are going to keep configuration (of guests and workloads) 
close to [1] for evaluation, but on our target SoC.
I'm wondering if there are known issues or specifics for ARM.

[1] https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf

> The configuration for the VCPUs depends on the tasks and the OS
> scheduler running on the VCPUs.
> The VCPU's utilization (budget/period) is usually larger than the
> tasks' utilizations (\sum e_i / p_i), where e_i is the task's
> worst-case execution time, and p_i is the task's period.
> The VCPU's parameters can be calculated by the CARTS tool [1],
>
> [1] https://rtg.cis.upenn.edu/carts/index.php
Got it.

> Did you try "sudo xl top"?
> IIRC, sudo xl top should work.
It was an issue of our yocto build. Xentop was not installed.
>> - root@salvator-x-h3-xt:/scripts# xl sched-rtds -d DomU -v all -p 10000 -b 2500
>>    (XEN) FLASK: Denying unknown domctl_scheduler_op: 2.
>>    libxl: error: libxl_sched.c:663:sched_rtds_vcpu_set_all: Domain 2:Setting vcpu sched rtds: Operation not permitted
>>    libxl_vcpu_sched_params_set_all failed.
> Which version of Xen or commit point did you use?
We have our stuff based on top of 4.9.0-rc6.

-- 

*Andrii Anisov*

*Lead Systems Engineer*

*Office: *+380 44 390 5457 <tel:+380%2044%20390%205457> *x* 66766 
<tel:66766>*Cell: *+380 50 5738852 <tel:+380%2050%205738852>*Email: 
*andrii_anisov@epam.com <mailto:andrii_anisov@epam.com>

*Kyiv**,* *Ukraine *(GMT+3)*epam.com <http://www.epam.com>*

CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or 
entity(ies) to which it is addressed and contains information that is 
legally privileged and confidential. If you are not the intended 
recipient, or the person responsible for delivering the message to the 
intended recipient, you are hereby notified that any dissemination, 
distribution or copying of this communication is strictly prohibited. 
All unintended recipients are obliged to delete this message and destroy 
any printed copies.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

On Mon, Jul 3, 2017 at 10:58 AM, Andrii Anisov <andrii_anisov@epam.com> wrote:
> Hello Meng Xu,
>
>
> On 03.07.17 16:35, Meng Xu wrote:
>>>
>>> Do you have any recommendations or suggestions?
>>
>> Which experiment/use case do you plan to run?
>> What are the requirements (or performance guarantees) you want to have
>> from RTDS?
>
> Currently we have no defined target use-cases.
> That's why we are going to keep configuration (of guests and workloads)
> close to [1] for evaluation, but on our target SoC.
> I'm wondering if there are known issues or specifics for ARM.

As far as I know, there is no known issue for ARM as long as that
version Xen runs on the ARM board.

I assume you have your own workloads to run, which are periodically
activated task.
The workloads in [1] are independent periodic CPU-intensive tasks: the
task does some computation for every period.
If your workloads are similar to the tasks, it should be ok.

One thing in my mind that may affect your evaluations for your real
workload is what you want to achieve.

The RTDS uses the EDF scheduling, under which the priorities of the
VCPUs (or VMs) are dynamically changed based on their (absolute)
deadlines. This provides better real-time performance for the
*overall* system.
If you want to make one VM highest priority and let that VM preempt
other VMs whenever the highest priority VM is active, it's better to
use the RM or FP scheduling, instead of the EDF scheduling.

Once the scheduling policy is determined, you will need to configure
the VCPUs' parameters based on the systems' workload.
This requires the workload's timing parameters for the CARTS tool to
compute the VCPUs' parameters.


Best,

Meng


-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

On 03.07.17 21:42, Meng Xu wrote:
> As far as I know, there is no known issue for ARM as long as that
> version Xen runs on the ARM board.
  That's good.
> I assume you have your own workloads to run, which are periodically
> activated task.
> The workloads in [1] are independent periodic CPU-intensive tasks: the
> task does some computation for every period.
> If your workloads are similar to the tasks, it should be ok.
Actually now we have just a high-level use case without any specific 
parameters defined.
I.e. in an automotive system there should be a domain dedicated to 
instrumental cluster beside IVI domain. IC domain should be RT.
So we are just evaluating and experimenting with an existing functionality.

> One thing in my mind that may affect your evaluations for your real
> workload is what you want to achieve.
>
> The RTDS uses the EDF scheduling, under which the priorities of the
> VCPUs (or VMs) are dynamically changed based on their (absolute)
> deadlines. This provides better real-time performance for the
> *overall* system.
In case we would have a driver domain and IC domain would draw to pv 
display baked by backend in a driver domain. Driver domain should be RT 
capable as well.
So it seems two domains should be RT beside non-RT IVI domain.

> If you want to make one VM highest priority and let that VM preempt
> other VMs whenever the highest priority VM is active, it's better to
> use the RM or FP scheduling, instead of the EDF scheduling.
So you are suggesting to introduce more RT schedulers with different 
algorithms. Did I get you right?

> Once the scheduling policy is determined, you will need to configure
> the VCPUs' parameters based on the systems' workload.
> This requires the workload's timing parameters for the CARTS tool to
> compute the VCPUs' parameters.
Thank you for suggestions.

-- 

*Andrii Anisov*



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

On Tue, Jul 4, 2017 at 8:28 AM, Andrii Anisov <andrii_anisov@epam.com> wrote:
>
>
> On 03.07.17 21:42, Meng Xu wrote:
>>
>> As far as I know, there is no known issue for ARM as long as that
>> version Xen runs on the ARM board.
>
>  That's good.
>>
>> I assume you have your own workloads to run, which are periodically
>> activated task.
>> The workloads in [1] are independent periodic CPU-intensive tasks: the
>> task does some computation for every period.
>> If your workloads are similar to the tasks, it should be ok.
>
> Actually now we have just a high-level use case without any specific parameters defined.
> I.e. in an automotive system there should be a domain dedicated to instrumental cluster beside IVI domain. IC domain should be RT.
> So we are just evaluating and experimenting with an existing functionality.
>
>> One thing in my mind that may affect your evaluations for your real
>> workload is what you want to achieve.
>>
>> The RTDS uses the EDF scheduling, under which the priorities of the
>> VCPUs (or VMs) are dynamically changed based on their (absolute)
>> deadlines. This provides better real-time performance for the
>> *overall* system.
>
> In case we would have a driver domain and IC domain would draw to pv display baked by backend in a driver domain. Driver domain should be RT capable as well.
> So it seems two domains should be RT beside non-RT IVI domain.
>
>> If you want to make one VM highest priority and let that VM preempt
>> other VMs whenever the highest priority VM is active, it's better to
>> use the RM or FP scheduling, instead of the EDF scheduling.
>
> So you are suggesting to introduce more RT schedulers with different algorithms. Did I get you right?

The EDF scheduling cares about the overall system's RT performance. If
you want to guarantee the *soft* real-time performance of the IVI
domains and allow the IVI domain to delay the two RT domains in some
scheduling periods, the EDF scheduling is better than the RM
scheduling. Note that we need to reserve enough CPU resources to make
sure the delay from the IVI domain to the two RT domains won't cause
the deadline miss of the two RT domains.

The RM scheduling will guarantees a domain always has a higher
priority than another domain. If you want to eliminate the CPU delay
from the IVI domain to the other two RT domains, can tolerate some
deadline misses of the IVI domain, and want to consolidate the three
domains to *fewer* cores, the RM scheduling should be a better choice,
IMO.

Supporting the RM scheduling policy in the RTDS scheduler is not
difficult. Actually, the RTDS scheduler was designed to be able to
extend to other scheduling policies, such as RM scheduling. In the
RT-Xen project[1], it supports both RM and EDF scheduling policy. We
just choose to upstream the EDF first.

Currently, we are working on synchronizing the RT-Xen with the latest
Xen: we want to implement the RM scheduling policy in the latest Xen.
I'm also teaching/training a master student how to implement the
scheduling policies in the RTDS scheduler so that we can have more
contributors.

I personally am very interested in the realistic use case, especially
the automotive use cases, for the RTDS scheduler. If you have any use
case that we can help to test, please don't hesitate to ask.

[1] https://github.com/PennPanda/RT-Xen

Best,

Meng

-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 5930 bytes --]

On Tue, 2017-07-04 at 11:12 -0400, Meng Xu wrote:
> On Tue, Jul 4, 2017 at 8:28 AM, Andrii Anisov <andrii_anisov@epam.com
> > wrote:
> > 
> > So you are suggesting to introduce more RT schedulers with
> > different algorithms. Did I get you right?
> 
> The EDF scheduling cares about the overall system's RT performance.
> If
> you want to guarantee the *soft* real-time performance of the IVI
> domains and allow the IVI domain to delay the two RT domains in some
> scheduling periods, the EDF scheduling is better than the RM
> scheduling. Note that we need to reserve enough CPU resources to make
> sure the delay from the IVI domain to the two RT domains won't cause
> the deadline miss of the two RT domains.
> 
This is technically correct, but, at the same time, I don't think it is
the best way to describe why and how one should use the RTDS scheduler.

In fact, what scheduling and prioritization strategy is used,
internally in the scheduler, is (for now) not exposed to the user, and
it hence should not have an impact in deciding whether or not to adopt
the scheduler... Unless we've done things in a very wrong way! :-P

What I'd say, as a description of what RTDS can give, to people
interested in using it, would be as follows.

RTDS gives you the chance to provide your VMs, guarantees of CPU
utilization that is precise, and has a well defined and strictly
enforced granularity. In fact, by using RTDS, it's possible to specify
two things:
- that a VM should at least be able to execute for a certain U% of 
  total CPU time
- that a VM will be able to exploit this 'reservation' with a time 
  granularity of P milliseconds.

U, in fact, is expressed as U=B/P, P (called period) is how frequently
a VM is given a chance to run, while B (called budget) is for how long
it will be able to run, on every time interval of length P.

So, if, as an example, a VM has a budget of 10 milliseconds and a
period of 100 milliseconds, this means:
- the VM will be granted 10% CPU execution time;
- if an event for the VM arrives at time t1, the VM itself will be
  able to start processing process it no later than t2=t1+2*P-2*B

That's why, IMO, the period matters (a lot!). If one "just" knows that
a VM will roughly need, say, 40% CPU time, then it does not matter if
the scheduling parameters are B/P=4/10, or B/P=40/100, or
B/P=40000/100000.
OTOH, if one also cares about the latency, doing the math and setting
the period properly.

In fact, this capability of specifying the granularity of a
reservation, is one of the main differences between RTDS (and, in
general, or real time scheduling algorithms) and other general purpose
algorithm. In fact, it is possible with general purpose algorithms too
(for example, using weights, in Credit1 and Credit2, or using `nice' in
Linux's CFS) to specify a certain utilization of a VM (task). But, in
those algorithms, it's impossible to specify precisely, and on a per-VM 
basis, the granularity of such reservation.

The caveat is that, unfortunately, the guarantee does not extend to
letting you exploit the full capacity. What I mean is that, while on
uniprocessor systems all that I have said above stays true, with the
only constraint of not giving, to the various VMs cumulatively, more
than 100% utilization, on multiprocessors, that is not true. Therefore,
if you have 4 pCPUs, and you assign the parameters to the various VMs
in such a way that the sum of B/P of all of them is <= 400%, it's not
guaranteed that _all_ of them will actually get their B, in every
interval of length P.

Knowing what the upper bound is, for a given number of pCPU, is not
easy. A necessary and sufficient limit has (to the best of my
knowledge, which may not be updated to the current state of the art of
RT academic literature) yet to be found. There are various limits, and
various ways of computing them, none of which is suitable to be
implemented inside an hypervisor... so Xen won't tell you whether or
not your overall set of parameters is feasible or not. :-(

(Perhaps we could, at least, keep track of the total utilization and at
least warn the user when we overcome full capacity. Say, if with 4
pCPUs, we go over 400%, we can well print a warning saying that
deadlines will be missed. Meng?)

These limits also depends on the actual scheduling policy (e.g.,
Eearliest Deadline First vs Rate Monotonic), but (again, to the best of
my knowledge) it has not been determined yet whether one is always
better than the other (again, for SMPs, in UPs, EDF wins), and so it's
again improper to bother with what algorithm to choose.

> Supporting the RM scheduling policy in the RTDS scheduler is not
> difficult. Actually, the RTDS scheduler was designed to be able to
> extend to other scheduling policies, such as RM scheduling. In the
> RT-Xen project[1], it supports both RM and EDF scheduling policy. We
> just choose to upstream the EDF first.
> 
Exactly. And I'm ok having RM, but we'll have to be careful about how
we document/advertise it, or we risk confusing people. :-)

In fact, I think that, whether or not Andrii will find RTDS
useful, depends really really really little, if at all, by the fact
that we implemented EDF or RM!

> I personally am very interested in the realistic use case, especially
> the automotive use cases, for the RTDS scheduler. If you have any use
> case that we can help to test, please don't hesitate to ask.
> 
Indeed! :-)

If you, or anyone from your team, have questions about this, don't
hesitate to fish me during the summit in Budapest. :-D

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

On Wed, Jul 5, 2017 at 4:29 AM, Dario Faggioli
<dario.faggioli@citrix.com> wrote:
> On Tue, 2017-07-04 at 11:12 -0400, Meng Xu wrote:
>> On Tue, Jul 4, 2017 at 8:28 AM, Andrii Anisov <andrii_anisov@epam.com
>> > wrote:
>> >
>> > So you are suggesting to introduce more RT schedulers with
>> > different algorithms. Did I get you right?
>>
>> The EDF scheduling cares about the overall system's RT performance.
>> If
>> you want to guarantee the *soft* real-time performance of the IVI
>> domains and allow the IVI domain to delay the two RT domains in some
>> scheduling periods, the EDF scheduling is better than the RM
>> scheduling. Note that we need to reserve enough CPU resources to make
>> sure the delay from the IVI domain to the two RT domains won't cause
>> the deadline miss of the two RT domains.
>>
> This is technically correct, but, at the same time, I don't think it is
> the best way to describe why and how one should use the RTDS scheduler.

Thank you very much, Dario, for giving a better explanation in the
user's perspectives.

>
> In fact, what scheduling and prioritization strategy is used,
> internally in the scheduler, is (for now) not exposed to the user, and
> it hence should not have an impact in deciding whether or not to adopt
> the scheduler... Unless we've done things in a very wrong way! :-P
>
> What I'd say, as a description of what RTDS can give, to people
> interested in using it, would be as follows.
>
> RTDS gives you the chance to provide your VMs, guarantees of CPU
> utilization that is precise, and has a well defined and strictly
> enforced granularity. In fact, by using RTDS, it's possible to specify
> two things:
> - that a VM should at least be able to execute for a certain U% of
>   total CPU time
> - that a VM will be able to exploit this 'reservation' with a time
>   granularity of P milliseconds.
>
> U, in fact, is expressed as U=B/P, P (called period) is how frequently
> a VM is given a chance to run, while B (called budget) is for how long
> it will be able to run, on every time interval of length P.
>
> So, if, as an example, a VM has a budget of 10 milliseconds and a
> period of 100 milliseconds, this means:
> - the VM will be granted 10% CPU execution time;
> - if an event for the VM arrives at time t1, the VM itself will be
>   able to start processing process it no later than t2=t1+2*P-2*B
>
> That's why, IMO, the period matters (a lot!). If one "just" knows that
> a VM will roughly need, say, 40% CPU time, then it does not matter if
> the scheduling parameters are B/P=4/10, or B/P=40/100, or
> B/P=40000/100000.
> OTOH, if one also cares about the latency, doing the math and setting
> the period properly.
>
> In fact, this capability of specifying the granularity of a
> reservation, is one of the main differences between RTDS (and, in
> general, or real time scheduling algorithms) and other general purpose
> algorithm. In fact, it is possible with general purpose algorithms too
> (for example, using weights, in Credit1 and Credit2, or using `nice' in
> Linux's CFS) to specify a certain utilization of a VM (task). But, in
> those algorithms, it's impossible to specify precisely, and on a per-VM
> basis, the granularity of such reservation.
>
> The caveat is that, unfortunately, the guarantee does not extend to
> letting you exploit the full capacity. What I mean is that, while on
> uniprocessor systems all that I have said above stays true, with the
> only constraint of not giving, to the various VMs cumulatively, more
> than 100% utilization, on multiprocessors, that is not true. Therefore,
> if you have 4 pCPUs, and you assign the parameters to the various VMs
> in such a way that the sum of B/P of all of them is <= 400%, it's not
> guaranteed that _all_ of them will actually get their B, in every
> interval of length P.
>
> Knowing what the upper bound is, for a given number of pCPU, is not
> easy. A necessary and sufficient limit has (to the best of my
> knowledge, which may not be updated to the current state of the art of
> RT academic literature) yet to be found. There are various limits, and
> various ways of computing them, none of which is suitable to be
> implemented inside an hypervisor... so Xen won't tell you whether or
> not your overall set of parameters is feasible or not. :-(
>
> (Perhaps we could, at least, keep track of the total utilization and at
> least warn the user when we overcome full capacity. Say, if with 4
> pCPUs, we go over 400%, we can well print a warning saying that
> deadlines will be missed. Meng?)

The total utilization can help answer if the VCPU parameters are
feasible or not.
But I'm thinking there may exist a better (yet optimal) approach to
answer the question: If all VCPUs on K cores are globally scheduled or
completely partitioned onto each of the K cores, we can use
Utilization Bound of the EDF scheduling algorithm for checking if the
VCPU's performance can be safely provided.
This requires the VCPUs' parameters (which also computes the total
utilization), which are easy to get.

Another thing is where this schedulability check should be provided:
in Xen kernel, in Xen toolstack, or as a separate utility tool?
In my opinion, a separate utility tool seems to be better than the
other tool approaches?

Best,

Meng

-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1740 bytes --]

On Fri, 2017-07-07 at 14:29 -0400, Meng Xu wrote:
> On Wed, Jul 5, 2017 at 4:29 AM, Dario Faggioli
> <dario.faggioli@citrix.com> wrote:
> > 
> The total utilization can help answer if the VCPU parameters are
> feasible or not.
>
I'm just saying that we could keep track of utilization and, if on an
host with N CPUs, we reach more than (N*100)%, we can warn the user
that deadlines will be missed.

This is a simple enough check, and it can live in the hypervisor.

> But I'm thinking there may exist a better (yet optimal) approach to
> answer the question: If all VCPUs on K cores are globally scheduled
> or
> completely partitioned onto each of the K cores, we can use
> Utilization Bound of the EDF scheduling algorithm for checking if the
> VCPU's performance can be safely provided.
> This requires the VCPUs' parameters (which also computes the total
> utilization), which are easy to get.
> 
I know there's math we can use, I'm just saying we don't want that in
Xen.

> Another thing is where this schedulability check should be provided:
> in Xen kernel, in Xen toolstack, or as a separate utility tool?
> In my opinion, a separate utility tool seems to be better than the
> other tool approaches?
> 
Exactly. As said above, you don't put something as complex as that
inside Xen. It can well live in toolstack, IMO, as far as we also add a
(global, non per-domain) for telling whether we want admission or
control not.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1917 bytes --]

On Tue, 2017-07-04 at 15:28 +0300, Andrii Anisov wrote:
> On 03.07.17 21:42, Meng Xu wrote:
> > 
> > The RTDS uses the EDF scheduling, under which the priorities of the
> > VCPUs (or VMs) are dynamically changed based on their (absolute)
> > deadlines. This provides better real-time performance for the
> > *overall* system.
> 
> In case we would have a driver domain and IC domain would draw to pv 
> display baked by backend in a driver domain. Driver domain should be
> RT 
> capable as well.
> So it seems two domains should be RT beside non-RT IVI domain.
> 
Currently (and this is not changing anytime soon), the only way of
using different schedulers for different domains is by means of
cpupools.

I.e., you create, for instance, an RTDS pool, and a Credit or Credit2
pool. In the RTDS pool, you put the RT domains, so the IC domain and
the driver domains, and you subdivide resources between them according
to the utilization and latency requirements that each one of them has.

You then put all the non-RT domains in the other pool, and you control
their behavior via weighs, cap, pinning (etc).

Of course, this is just an example, and whether or not it is the best
way of going, as well as the specific parameters, really depends on the
characteristics of your platform and of your workload.

Trying to reason on as detailed as possible scenario, and then actually
testing and benchmarking every envisioned solution, is the only way to
actually tell what will work best.

As I said in the other email, I'm more than up discussing this, either
via email or in person (e.g., at the Summit).

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1613 bytes --]

On Mon, 2017-07-03 at 14:42 -0400, Meng Xu wrote:
> On Mon, Jul 3, 2017 at 10:58 AM, Andrii Anisov <andrii_anisov@epam.co
> m> wrote:
> > 
> Once the scheduling policy is determined, you will need to configure
> the VCPUs' parameters based on the systems' workload.
> This requires the workload's timing parameters for the CARTS tool to
> compute the VCPUs' parameters.
> 
Yes, this is an interesting thing that Meng is mentioning.

RTDS allows you to specify the parameters (budget and period, or,
depending on how you prefer to see things, utilization and latency) on
a per-vCPU basis.

This may look cumbersome and overly complicated (and, in fact, if you
don't need it, you can ignore it :-D), but it may actually be really
important in a truly RT scenario.

Whether or not it is useful, almost entirely depends on what the VM is
doing, and how you decide to control and configure things inside it.
For example, if you have a particular task in the VM that you need must
absolutely execute for at least 10ms every 100ms, you can:
- inside the VM, pin the task to vCPU 0, and give it top priority;
- at the Xen level, give (with RTDS) to vCPU 0 budget=10ms and 
  period=100ms (or maybe budget of 12ms, to allow for some overhead
  :-P).

This is something that no other scheduler allows you to do. :-)

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

On Wed, Jul 5, 2017 at 4:51 AM, Dario Faggioli
<dario.faggioli@citrix.com> wrote:
> On Mon, 2017-07-03 at 14:42 -0400, Meng Xu wrote:
>> On Mon, Jul 3, 2017 at 10:58 AM, Andrii Anisov <andrii_anisov@epam.co
>> m> wrote:
>> >
>> Once the scheduling policy is determined, you will need to configure
>> the VCPUs' parameters based on the systems' workload.
>> This requires the workload's timing parameters for the CARTS tool to
>> compute the VCPUs' parameters.
>>
> Yes, this is an interesting thing that Meng is mentioning.
>
> RTDS allows you to specify the parameters (budget and period, or,
> depending on how you prefer to see things, utilization and latency) on
> a per-vCPU basis.
>
> This may look cumbersome and overly complicated (and, in fact, if you
> don't need it, you can ignore it :-D), but it may actually be really
> important in a truly RT scenario.
>
> Whether or not it is useful, almost entirely depends on what the VM is
> doing, and how you decide to control and configure things inside it.

Exactly.

Andrii,
If you encountered any question/difficulty in choosing the proper VCPU
parameters for your workload, please don't hesitate to ping me and
Dario.

I'm also trying to make it *easier* for users to *correctly* configure
the VCPUs on RTDS.
The more we understand the real use cases, the more we can help
improve the scheduler and its related tools.

> For example, if you have a particular task in the VM that you need must
> absolutely execute for at least 10ms every 100ms, you can:
> - inside the VM, pin the task to vCPU 0, and give it top priority;
> - at the Xen level, give (with RTDS) to vCPU 0 budget=10ms and
>   period=100ms (or maybe budget of 12ms, to allow for some overhead
>   :-P).

This assumes that the start time of the task's each period is
synchronized with the start time of the VCPU's each period.
If this assumption does not hold, the VCPU need a larger budget to
guarantee the task will always have 10ms in *any* 100ms time interval.
The budget can be computed by CARTS
(https://rtg.cis.upenn.edu/carts/).

>
> This is something that no other scheduler allows you to do. :-)

Exactly.
If you want to reason about the real-time timing properties required
by some safety-critical systems' standard, such as the ISO-26262 for
automotive systems, the above computation and analysis will be
required.


-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

[sorry, my finger slips. let me rephrase my last sentence.]

>
>> For example, if you have a particular task in the VM that you need must
>> absolutely execute for at least 10ms every 100ms, you can:
>> - inside the VM, pin the task to vCPU 0, and give it top priority;
>> - at the Xen level, give (with RTDS) to vCPU 0 budget=10ms and
>>   period=100ms (or maybe budget of 12ms, to allow for some overhead
>>   :-P).
>
> This assumes that the start time of the task's each period is
> synchronized with the start time of the VCPU's each period.
> If this assumption does not hold, the VCPU need a larger budget to
> guarantee the task will always have 10ms in *any* 100ms time interval.
> The budget can be computed by CARTS
> (https://rtg.cis.upenn.edu/carts/).
>
>>
>> This is something that no other scheduler allows you to do. :-)
>
> Exactly.
> If you want to reason about the real-time timing properties required
> by some safety-critical systems' standard, such as the ISO-26262 for
> automotive systems, the above computation and analysis will be
> required.

I mean:
If you want to argue about the real-time timing properties required
by some safety-critical systems' standard, such as the ISO-26262 for
automotive systems, the system support (such as the RTDS scheduler),
the correct configuration of the system, and the analysis  computation
and analysis will be required.

Thanks,

Meng
-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

Hello Meng Xu,

On 07.07.17 21:43, Meng Xu wrote:
> Andrii,
> If you encountered any question/difficulty in choosing the proper VCPU
> parameters for your workload, please don't hesitate to ping me and
> Dario.
Thank you. I'll keep you in touch when we have something specified.

-- 

*Andrii Anisov*



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

Hello Meng Xu,

I've get back to this stuff.


On 03.07.17 17:58, Andrii Anisov wrote:
> That's why we are going to keep configuration (of guests and 
> workloads) close to [1] for evaluation, but on our target SoC.
> I'm wondering if there are known issues or specifics for ARM.
>
> [1] https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf
Currently I have a setup with dom0 and domU's with Litmus-RT. Following 
the document I need workload tasks.
Maybe you have mentioned workload tasks sources you can share, so that 
would shorten my steps.

-- 

*Andrii Anisov*



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

Hi Andrii,

On Tue, Aug 1, 2017 at 4:02 AM, Andrii Anisov <andrii_anisov@epam.com> wrote:
> Hello Meng Xu,
>
> I've get back to this stuff.

Sorry for the late response. I'm not sure if you have already solved this.

>
>
> On 03.07.17 17:58, Andrii Anisov wrote:
>>
>> That's why we are going to keep configuration (of guests and workloads)
>> close to [1] for evaluation, but on our target SoC.
>> I'm wondering if there are known issues or specifics for ARM.
>>
>> [1] https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf
>
> Currently I have a setup with dom0 and domU's with Litmus-RT.

Great!

> Following the
> document I need workload tasks.
> Maybe you have mentioned workload tasks sources you can share, so that would
> shorten my steps.

Sure. The workload we used in the paper is mainly the cpu-intensive task.
We first calibrate a busy-loop of multiplications that runs for 1ms.
Then for a task that executes for exe(ms), we simply let the task
execute the 1ms busy loop for exe times.
It is also good to run the same task for several times to make sure
the task's execution time is table from different runs.

The Section 4.1 and 4.2 in [1] explained the whole experiment steps.
If you have any question or confusion on a specific step, please feel
free to let me know.
We may schedule a meeting to clarify all the questions or confusions
you may have.

[1] https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf

Best regards,

Meng




>
> --
>
> *Andrii Anisov*
>
>



-- 
-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

Hello Meng Xu,


On 18.08.17 23:43, Meng Xu wrote:
> The Section 4.1 and 4.2 in [1] explained the whole experiment steps.
> If you have any question or confusion on a specific step, please feel
> free to let me know.
 From the document it is not really clear if you ran one guest RT domain 
or several simultaneously for your experiments.

-- 

*Andrii Anisov*



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

On 21.08.17 11:07, Andrii Anisov wrote:
> Hello Meng Xu,
>
>
> On 18.08.17 23:43, Meng Xu wrote:
>> The Section 4.1 and 4.2 in [1] explained the whole experiment steps.
>> If you have any question or confusion on a specific step, please feel
>> free to let me know.
> From the document it is not really clear if you ran one guest RT 
> domain or several simultaneously for your experiments.
Also it is not described XEN RT scheduler setup like vcpus period/budget 
configuration for each guest domain.
It is not obvious if the configured set of vcpus in the experiment setup 
utilized all the pcpus bandwidth.

-- 

*Andrii Anisov*



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

On Mon, Aug 21, 2017 at 4:16 AM, Andrii Anisov <andrii_anisov@epam.com> wrote:
>
> On 21.08.17 11:07, Andrii Anisov wrote:
>>
>> Hello Meng Xu,
>>
>>
>> On 18.08.17 23:43, Meng Xu wrote:
>>>
>>> The Section 4.1 and 4.2 in [1] explained the whole experiment steps.
>>> If you have any question or confusion on a specific step, please feel
>>> free to let me know.
>>
>> From the document it is not really clear if you ran one guest RT domain or
>> several simultaneously for your experiments.
>
> Also it is not described XEN RT scheduler setup like vcpus period/budget
> configuration for each guest domain.
> It is not obvious if the configured set of vcpus in the experiment setup
> utilized all the pcpus bandwidth.
>

Given the set of tasks in each VM, we compute the VCPUs' periods and
budgets, using the CARTS tool [1]. Note that each task has a period
and a worst-case execution time (wcet).

The configured set of vcpus in the experiment setup may not use all
pcpus bandwidth. For example, if we have one task (period = 10ms, wcet
= 2ms) on a VCPU, the VCPU of the task will not be configured with
100% bandwidth. If the VCPU is the only VCPU on a pcpu, that pcpu
bandwidth won't be fully used because there is not enough workload to
fully use all pcpu bandwidth.

[1] https://rtg.cis.upenn.edu/carts/

Best,

Meng
-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

Dear Meng Xu,


On 22.08.17 05:02, Meng Xu wrote:
> Given the set of tasks in each VM, we compute the VCPUs' periods and
> budgets, using the CARTS tool [1]. Note that each task has a period
> and a worst-case execution time (wcet).
>
> [1] https://rtg.cis.upenn.edu/carts/

In a CARTS tool documentation [1] it is said that:
     "At the same time, it is also accompanied by  a lightweight 
command‐line  option  that  enables  our  tool  to  be integrated with 
other existing toolchains."

But there is no CLI usage description in the document. Could you please 
provide such a description? My intention is to have a set of generated 
input xml's, feed them to CARTS and get models ready to be parsed to 
receive domains configuration for each test case.

[1] https://rtg.cis.upenn.edu/carts/docs/userguide.pdf

-- 

*Andrii Anisov*



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

Hi Andrii,

On Wed, Sep 27, 2017 at 8:37 AM, Andrii Anisov <andrii_anisov@epam.com> wrote:
>
> Dear Meng Xu,
>
>
> On 22.08.17 05:02, Meng Xu wrote:
>>
>> Given the set of tasks in each VM, we compute the VCPUs' periods and
>> budgets, using the CARTS tool [1]. Note that each task has a period
>> and a worst-case execution time (wcet).
>>
>> [1] https://rtg.cis.upenn.edu/carts/
>
>
> In a CARTS tool documentation [1] it is said that:
>     "At the same time, it is also accompanied by  a lightweight command‐line  option  that  enables  our  tool  to  be integrated with other existing toolchains."
>
> But there is no CLI usage description in the document. Could you please provide such a description?


The command is:
java -jar carts.jar inputfile outputfile

An example command is:
java -jar carts.jar 1-1.10-in.xml MPR2 1-1.10-out.xml

Best,

Meng


-- 
Meng Xu
Ph.D. Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

Hello,


On 27.09.17 16:57, Meng Xu wrote:
> The command is:
> java -jar carts.jar inputfile outputfile
 From the next example, I would say the command is:
     java -jar carts.jar inputfile interface_type outputfile

> An example command is:
> java -jar carts.jar 1-1.10-in.xml MPR2 1-1.10-out.xml
Thanks a lot. It does work.

Could you please clarify a bit more points to me:
     - As I understand the upstreamed rtds employs gEDF only. Is it 
correct?
     - Could you please provide an example input xml for CARTS described 
a system with 2 RT domains with 2 VCPUs each, running on a 2PCPUs, with 
gEDF scheduling at VMM level (for XEN based setup). For pEDF at both VMM 
and domain level, my understanding is that the os_scheduler represents 
XEN, and VCPUs are represented by components with tasks running on them.
     - I did not get a concept of min_period/max_period for a 
component/os_scheduler in CARTS description files. If I have them 
different, CARTS gives calculation for all periods in between, but did 
not provide the best period to get system schedulable.

-- 

*Andrii Anisov*


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

On Wed, Sep 27, 2017 at 10:37 AM, Andrii Anisov <andrii_anisov@epam.com> wrote:
> Hello,
>
>
> On 27.09.17 16:57, Meng Xu wrote:
>>
>> The command is:
>> java -jar carts.jar inputfile outputfile
>
> From the next example, I would say the command is:
>     java -jar carts.jar inputfile interface_type outputfile
>
>> An example command is:
>> java -jar carts.jar 1-1.10-in.xml MPR2 1-1.10-out.xml
>
> Thanks a lot. It does work.
>
> Could you please clarify a bit more points to me:
>     - As I understand the upstreamed rtds employs gEDF only. Is it correct?

RTDS can support gEDF and partitioned EDF.
To support pEDF, you can set each VCPU's affinity (hard and soft
affninity)  to only one core using "xl vcpu-pin" command. The VCPUs on
each core will be scheduled by pEDF scheduling algorithm.

Note that:
When you use gEDF scheduler in VM or VMM (i.e., Xen), you should use
MPR2 model to compute the resource interface (i.e., VCPU parameters).
When you use pEDF scheduler, you should use PRM model to compute.


>     - Could you please provide an example input xml for CARTS described a
> system with 2 RT domains with 2 VCPUs each, running on a 2PCPUs, with gEDF
> scheduling at VMM level (for XEN based setup).

Hmm, if you use the gEDF scheduling algorithm, this may not be
possible. Let me explain why.
In the MPR2 model, it computes the interface with the minimum number
of cores. To get 2 VCPUs for a VM, the total utilization (i.e., budget
/ period) of these two VCPUs must be larger than 1.0. Since you ask
for 2 domains, the total utilization of these 4 VCPUs will be larger
than 2.0, which are definitely not schedulable on two cores.

If you are considering VCPUs with very low utilization, you may use
PRM model to compute each VCPU's parameters; after that, you can treat
these VCPUs as tasks, create another xml file, and ask CARTS to
compute the resource interface for these VCPUs.

(Unfortunately, the current CARTS implementation does not support
mixing MPR model in one XML file, although it is supported in theory.
This can be worked around by using the above approach.)

> For pEDF at both VMM and
> domain level, my understanding is that the os_scheduler represents XEN, and
> VCPUs are represented by components with tasks running on them.

Yes, if you analyze the entire system that uses one type of scheduler
with only one type of model (i.e., PRM or MPR2).

If you mixed the scheduling algorithm or the interface model, you can
compute each VM or VCPU's parameters first. Then you treat VCPUs as
tasks and create another XML which will be used to compute the number
of cores to schedule all these VCPUs.

>     - I did not get a concept of min_period/max_period for a
> component/os_scheduler in CARTS description files. If I have them different,
> CARTS gives calculation for all periods in between, but did not provide the
> best period to get system schedulable.

You should set them to the same value.
min_period/max_period range is used for other models. I never used it.

Best,

Meng

-- 
Meng Xu
Ph.D. Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

Hello,


On 27.09.17 22:57, Meng Xu wrote:
> Note that:
> When you use gEDF scheduler in VM or VMM (i.e., Xen), you should use
> MPR2 model
I guess you mean DMPR in CARTS terms.

>   to compute the resource interface (i.e., VCPU parameters).
> When you use pEDF scheduler, you should use PRM model to compute.
>>      - Could you please provide an example input xml for CARTS described a
>> system with 2 RT domains with 2 VCPUs each, running on a 2PCPUs, with gEDF
>> scheduling at VMM level (for XEN based setup).
> Hmm, if you use the gEDF scheduling algorithm, this may not be
> possible. Let me explain why.
> In the MPR2 model, it computes the interface with the minimum number
> of cores. To get 2 VCPUs for a VM, the total utilization (i.e., budget
> / period) of these two VCPUs must be larger than 1.0. Since you ask
> for 2 domains, the total utilization of these 4 VCPUs will be larger
> than 2.0, which are definitely not schedulable on two cores.
Well, if we are speaking about test-cases similar to described in [1], 
where the whole real time tasks set utilization is taken from 
1.1...(PCPU*1)-0.1, there is no problem with having VCPU number greater 
than PCPUs. For sure if we take number of domains  more that 1.

> If you are considering VCPUs with very low utilization, you may use
> PRM model to compute each VCPU's parameters; after that, you can treat
> these VCPUs as tasks, create another xml file, and ask CARTS to
> compute the resource interface for these VCPUs.
Sounds terrible for getting it scripted :(
> (Unfortunately, the current CARTS implementation does not support
> mixing MPR model in one XML file, although it is supported in theory.
> This can be worked around by using the above approach.)
>
>> For pEDF at both VMM and
>> domain level, my understanding is that the os_scheduler represents XEN, and
>> VCPUs are represented by components with tasks running on them.
> Yes, if you analyze the entire system that uses one type of scheduler
> with only one type of model (i.e., PRM or MPR2).
>
> If you mixed the scheduling algorithm or the interface model, you can
> compute each VM or VCPU's parameters first. Then you treat VCPUs as
> tasks and create another XML which will be used to compute the number
> of cores to schedule all these VCPUs.
>
>>      - I did not get a concept of min_period/max_period for a
>> component/os_scheduler in CARTS description files. If I have them different,
>> CARTS gives calculation for all periods in between, but did not provide the
>> best period to get system schedulable.
> You should set them to the same value.
Ok, how to chose the value for some taskset in a component?

-- 

*Andrii Anisov*



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 5051 bytes --]

On Thu, 2017-09-28 at 12:18 +0300, Andrii Anisov wrote:
> > >      - Could you please provide an example input xml for CARTS
> > > described a
> > > system with 2 RT domains with 2 VCPUs each, running on a 2PCPUs,
> > > with gEDF
> > > scheduling at VMM level (for XEN based setup).
> > 
> > Hmm, if you use the gEDF scheduling algorithm, this may not be
> > possible. Let me explain why.
> > In the MPR2 model, it computes the interface with the minimum
> > number
> > of cores. To get 2 VCPUs for a VM, the total utilization (i.e.,
> > budget
> > / period) of these two VCPUs must be larger than 1.0. Since you ask
> > for 2 domains, the total utilization of these 4 VCPUs will be
> > larger
> > than 2.0, which are definitely not schedulable on two cores.
> 
> Well, if we are speaking about test-cases similar to described in
> [1], 
>
Missing the link?

> where the whole real time tasks set utilization is taken from 
> 1.1...(PCPU*1)-0.1, there is no problem with having VCPU number
> greater 
> than PCPUs. For sure if we take number of domains  more that 1.
> 
I may miss/have misunderstood some of the background information, but
anyway. Also, I've been exposed to the supply/demand concepts that are
the basis of Meng's CARTS tool, during my PhD on this things, but it's
been a while, so bear with me.

Point is, if you have 4 vCPUs in total (2 domains with 2 each), and 2
pCPUs, they can't run all the time all together. The choices are
between a general purpose scheduler, or an RT one.

The GP scheduler --like Credit1 or Credit2, if you are on Xen-- will
give you fairness, but without any precise temporal guarantee. This
means that, in this case, each vCPU will be given the chance to run for
1/2 CPU capacity (at there are 2 CPUs, 4 vCPUs total, i.e., 2/4=1/2).
You can influence this with weights, but that's still "not real-time".

I.e., if you say that d1v0 and d1v1 have double the weights of d2v0 and
d2v1, if 2 is the total available CPU capacity, d1's vCPUs will run for

Now, let's say that, in d1, there is a critical real-time application
that absolutely needs to read the status of a sensor every 10ms. Is
there a set of weights able to provide you with the guarantee that it
will always be able to, whatever it is that the other vCPUs are doing?

Well, no, there isn't. And this is where RTDS becomes interesting. So,
is there a way to tell to RTDS that you want to be sure that d1 will
have a chance to run every 10ms? Yes, there is: you give to it a period
of 10ms. Now you need to esteem how much computation time d1's vCPUs
require (for reading that sensors, but, let's say, also for doing other
stuff, since it's a 2 vCPUs VM). Let's assume that you're quite
confident that it will be enough to giving 20% overall capacity to each
vCPU. This means budget for d1v0 and d1v1 will be 2ms (as 2/10*100=20).

Now, there is another domain, and it may host applications with real-
time requirements as well, here's where things become interesting.
According to what I remember from SMP real-time scheduling theory, you
just can't assume that giving d2v0 and d2v1 budget and period of 80ms
and 100ms, respectively, will result in a schedulable system, even
though you have 2 CPUs, and 2/10+2/10+80/100+80/100=2 (for Meng, I'm
talking in general, about the fact that, under gEDF, with M CPUs, U<M
is necessary but not sufficient, but I haven't done the math for this
specific case :-P).

And here's where tools like CARTS may be of great help. So, basically,
if also the applications that you want to run in d2 have their own
timing requirement, you use those to come up with an indication of the
period the VM's vCPUs need to have. Then you use a tool, passing all
these information to it as parameters (not that I've never used CARTS,
so I don't know what parameters it actually takes, I'm talking in
general), and the tool gives you the scheduling parameter at each
level: OS scheduler --which would be Linux/Android, inside the VMs--
and hypervisor scheduler --which would be the budget and period of the
vCPUs to configure in RTDS.

But you can't (or at least, I don't think) expect the tools to just
tell you "how to set parameters for scheduling 4 vCPUs on 2 pCPUs",
without also telling it what you want/need to achieve. So, basically,
you have to define the term "schedule", in the above sentence.

Otherwise, which I think is what Meng is telling you when he says "this
may not be possible", I guess the tool will try to see if it is
possible to schedule 4 vCPUs over 2 pCPUs, in such a way that whatever
real-time activities, running on those 4 vCPUs, will all and always
meet their deadline... and the answer will (probably?) be 'no'.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1516 bytes --]

On Thu, 2017-09-28 at 18:01 +0200, Dario Faggioli wrote:
> On Thu, 2017-09-28 at 12:18 +0300, Andrii Anisov wrote:
> > > > 
> The GP scheduler --like Credit1 or Credit2, if you are on Xen-- will
> give you fairness, but without any precise temporal guarantee. This
> means that, in this case, each vCPU will be given the chance to run
> for
> 1/2 CPU capacity (at there are 2 CPUs, 4 vCPUs total, i.e., 2/4=1/2).
> You can influence this with weights, but that's still "not real-
> time".
> 
> I.e., if you say that d1v0 and d1v1 have double the weights of d2v0
> and
> d2v1, if 2 is the total available CPU capacity, d1's vCPUs will run
> for
> 
Err... Sorry, I hit send, before finishing this paragraph.

I meant to say: if you set d1 to have twice as much the weight of d2
(because weights, in Credit1 and Credit2, are set on a per-domain
basis), d1 will execute for about ~130% of CPU time (subdivided,
theoretically equally, between d1v0 and d1v1), and d2 will execute for
~70% (subdivided between d2v0 and d2v1).

So, basically, the 2x ratio between the domain's execution time is
achievable, with a general purpose scheduler, but it's not possible to
specify, and have it enforced, any time granularity. 

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

Hello Dario,


On 28.09.17 19:01, Dario Faggioli wrote:
> On Thu, 2017-09-28 at 12:18 +0300, Andrii Anisov wrote:
>>>>       - Could you please provide an example input xml for CARTS
>>>> described a
>>>> system with 2 RT domains with 2 VCPUs each, running on a 2PCPUs,
>>>> with gEDF
>>>> scheduling at VMM level (for XEN based setup).
>>> Hmm, if you use the gEDF scheduling algorithm, this may not be
>>> possible. Let me explain why.
>>> In the MPR2 model, it computes the interface with the minimum
>>> number
>>> of cores. To get 2 VCPUs for a VM, the total utilization (i.e.,
>>> budget
>>> / period) of these two VCPUs must be larger than 1.0. Since you ask
>>> for 2 domains, the total utilization of these 4 VCPUs will be
>>> larger
>>> than 2.0, which are definitely not schedulable on two cores.
>> Well, if we are speaking about test-cases similar to described in
>> [1],
>>
> Missing the link?
Yep, I've missed the link.

[1] https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf

-- 

*Andrii Anisov*

*Lead Systems Engineer*

*Office: *+380 44 390 5457 <tel:+380%2044%20390%205457> *x* 66766 
<tel:66766>*Cell: *+380 50 5738852 <tel:+380%2050%205738852>*Email: 
*andrii_anisov@epam.com <mailto:andrii_anisov@epam.com>

*Kyiv**,* *Ukraine *(GMT+3)*epam.com <http://www.epam.com>*

CONFIDENTIALITY CAUTION AND DISCLAIMER
This message is intended only for the use of the individual(s) or 
entity(ies) to which it is addressed and contains information that is 
legally privileged and confidential. If you are not the intended 
recipient, or the person responsible for delivering the message to the 
intended recipient, you are hereby notified that any dissemination, 
distribution or copying of this communication is strictly prohibited. 
All unintended recipients are obliged to delete this message and destroy 
any printed copies.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

Hi Andrii,

I'm sorry for replying to this thread late. I was busy with a paper
deadline until last Saturday morning.

I saw Dario's thorough answer which explains the high-level idea of
the real-time analysis that is the theoretical foundation of the
analysis tool, e.g., CARTs.
Hopefully, he answered your question.
If not, please feel free to ask.

I just added some very quick comment about your questions/comments below.

On Thu, Sep 28, 2017 at 5:18 AM, Andrii Anisov <andrii_anisov@epam.com> wrote:
> Hello,
>
>
> On 27.09.17 22:57, Meng Xu wrote:
>>
>> Note that:
>> When you use gEDF scheduler in VM or VMM (i.e., Xen), you should use
>> MPR2 model
>
> I guess you mean DMPR in CARTS terms.
>
>>   to compute the resource interface (i.e., VCPU parameters).
>> When you use pEDF scheduler, you should use PRM model to compute.
>>>
>>>      - Could you please provide an example input xml for CARTS described
>>> a
>>> system with 2 RT domains with 2 VCPUs each, running on a 2PCPUs, with
>>> gEDF
>>> scheduling at VMM level (for XEN based setup).
>>
>> Hmm, if you use the gEDF scheduling algorithm, this may not be
>> possible. Let me explain why.
>> In the MPR2 model, it computes the interface with the minimum number
>> of cores. To get 2 VCPUs for a VM, the total utilization (i.e., budget
>> / period) of these two VCPUs must be larger than 1.0. Since you ask
>> for 2 domains, the total utilization of these 4 VCPUs will be larger
>> than 2.0, which are definitely not schedulable on two cores.
>
> Well, if we are speaking about test-cases similar to described in [1], where
> the whole real time tasks set utilization is taken from 1.1...(PCPU*1)-0.1,
> there is no problem with having VCPU number greater than PCPUs. For sure if
> we take number of domains  more that 1.

The number of VCPUs can be larger than the number of PCPUs.

>
>> If you are considering VCPUs with very low utilization, you may use
>> PRM model to compute each VCPU's parameters; after that, you can treat
>> these VCPUs as tasks, create another xml file, and ask CARTS to
>> compute the resource interface for these VCPUs.
>
> Sounds terrible for getting it scripted :(

If you use python to parse the xml file, it should not be very
difficuly. Python has api to parse the xml. :)

>>
>> (Unfortunately, the current CARTS implementation does not support
>> mixing MPR model in one XML file, although it is supported in theory.
>> This can be worked around by using the above approach.)
>>
>>> For pEDF at both VMM and
>>> domain level, my understanding is that the os_scheduler represents XEN,
>>> and
>>> VCPUs are represented by components with tasks running on them.
>>
>> Yes, if you analyze the entire system that uses one type of scheduler
>> with only one type of model (i.e., PRM or MPR2).
>>
>> If you mixed the scheduling algorithm or the interface model, you can
>> compute each VM or VCPU's parameters first. Then you treat VCPUs as
>> tasks and create another XML which will be used to compute the number
>> of cores to schedule all these VCPUs.
>>
>>>      - I did not get a concept of min_period/max_period for a
>>> component/os_scheduler in CARTS description files. If I have them
>>> different,
>>> CARTS gives calculation for all periods in between, but did not provide
>>> the
>>> best period to get system schedulable.
>>
>> You should set them to the same value.
>
> Ok, how to chose the value for some taskset in a component?

Tasks' periods and execution time depends on the tasks' requirement.
As Dario mentioned, if a sensor needs to process every 100ms, the
sensor task's period is 100ms. Its execution time is the worst-case
execution time of the sensor task.

As to the component (or VM)'s period, it's better to be smaller than
its tasks' periods. Usually, I may want to set to a value divisible by
its tasks' periods.
You may try different values for components' periods, because the
VCPU's bandwidth (budget/period) will be different for different
components' periods.
You can choose the component's period that produces a smaller VCPU's
bandwidth, which may help make VCPUs easiler to be scheduled on PCPUs.

Best,

Meng


-- 
Meng Xu
Ph.D. Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

On Mon, Aug 21, 2017 at 4:07 AM, Andrii Anisov <andrii_anisov@epam.com> wrote:
>
> Hello Meng Xu,
>
>
> On 18.08.17 23:43, Meng Xu wrote:
>>
>> The Section 4.1 and 4.2 in [1] explained the whole experiment steps.
>> If you have any question or confusion on a specific step, please feel
>> free to let me know.
>
> From the document it is not really clear if you ran one guest RT domain or several simultaneously for your experiments.
>

We run 4 VMs simultaneously.


Meng


-- 
-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Andrii Anisov]

On 18.08.17 23:43, Meng Xu wrote:
> Sure. The workload we used in the paper is mainly the cpu-intensive task.
> We first calibrate a busy-loop of multiplications that runs for 1ms.
> Then for a task that executes for exe(ms), we simply let the task
> execute the 1ms busy loop for exe times.
I'm a bit confused, why didn't you ran the system with rtspin from 
LITMUS-RT, any issues with it?
BTW, I've found set experimental patches (scripts and functional 
changes) on your github: https://github.com/PennPanda/liblitmus .
Are they related to the mentioned document [1]?

> [1] https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf

-- 

*Andrii Anisov*



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Meng Xu]

On Mon, Aug 21, 2017 at 4:38 AM, Andrii Anisov <andrii_anisov@epam.com> wrote:
>
> On 18.08.17 23:43, Meng Xu wrote:
>>
>> Sure. The workload we used in the paper is mainly the cpu-intensive task.
>> We first calibrate a busy-loop of multiplications that runs for 1ms.
>> Then for a task that executes for exe(ms), we simply let the task
>> execute the 1ms busy loop for exe times.
>
> I'm a bit confused, why didn't you ran the system with rtspin from
> LITMUS-RT, any issues with it?

The task we are using should do same amount of calculation for the
same amount of time. For example, suppose it takes 1ms to run the
following piece of code:
for( i = 0; i < 1 million; i++)
     sum += i;
This piece of code can be viewed as the "payload" of a realistic workload.

Suppose the task is scheduled to run at t0, preempted at t1, resumes
at t2, and finishes at t3. We have (t1 - t0) + (t3 - t2) = 1ms and we
are sure the task did the addition for 1million times.

However, if we use the rtspin, the rtspin will check if (t2-t0) > 1ms.
If so, it will claim it finishes its workload although it hasn't
finished its workload, i.e., doing addition for 1million times.

Since we want to compare if tasks can finish their "workload" by their
deadline under different scheduling algorithms, we should fix the
"amount of workload" a task does under different scheduling policies.
rtspin() does not achieve our purpose. That's why we don't use it.

Note that rtspin() is initially designed to test the scheduling
overload of LITMUS. It does not perform the same amount of workload
for the same assigned wcet.

> BTW, I've found set experimental patches (scripts and functional changes) on
> your github: https://github.com/PennPanda/liblitmus .
> Are they related to the mentioned document [1]?

Not really. The liblitmus repo under my repo. is for another project.
It is not for [1]'s purpose.

The idea of creating the real-time task is similar, though.
The real-time task is based on the bin/base_task.c in liblitmus.
It needs to fill out the job() function as follows:

static int job(int wcet)
{
    for (i = 0; i < wcet; i++)
          loop_for_one_1ms()
}

 loop_for_one_1ms() {
     /* iterations value differs across machines */
     for (j = 0; j < iterations; j++ )
           result  = result + j * j;
  }


>
>> [1] https://www.cis.upenn.edu/~linhphan/papers/emsoft14-rt-xen.pdf
>
>
> --

Hope it helps clear the confusion.

Thanks,

Meng



-- 
-----------
Meng Xu
PhD Candidate in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: RT-Xen on ARM

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1064 bytes --]

On Mon, 2017-07-03 at 14:03 +0300, Andrii Anisov wrote:
> - root@salvator-x-h3-xt:/scripts# xl sched-rtds -d DomU -v all -p
> 10000 
> -b 2500
>    (XEN) FLASK: Denying unknown domctl_scheduler_op: 2.
>    libxl: error: libxl_sched.c:663:sched_rtds_vcpu_set_all: Domain 
> 2:Setting vcpu sched rtds: Operation not permitted
>    libxl_vcpu_sched_params_set_all failed.
> 
Mmm... this looks to me to mean that we're missing the XSM/Flask bits
of XEN_DOMCTL_SCHEDOP_putvcpuinfo and XEN_DOMCTL_SCHEDOP_getvcpuinfo.

Meng, can you guys check (e.g., by building a Flask enabled hypervisor
and testing the basic functionalities of RTDS) and, if it actually does
not work, fix this?

It should be ok to test on x86, as I don't think this is something ARM
specific.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel