Re: [PATCH] sched_rt: Improved nested virtualization performance

Re: [PATCH] sched_rt: Improved nested virtualization performance

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3453 bytes --]

Hi,

Thanks Tianyang for the report and for the patch, and Meng for helping
getting the patch itself on the list and to me, and for commenting.

On Wed, 2015-11-18 at 22:38 -0500, Meng Xu wrote:
> [cc. Dario...]
> 
> 2015-11-18 22:24 GMT-05:00 Tianyang Chen <tianyangpenn@gmail.com>:
> > In nested virtualization, choosing the smaller value for the
> > time slice between the MAX_SCHEDULE and the budget will cause
> > high host CPU usage.
> 
> Just add one comment:
> I also experienced this issue when I develop Xen in a virtual box on
> MacBook Air.
> 
> The root problem is not the returned time slice in rt_schedule(). The
> root problem is that the CPU speed for Xen in nested virtualization
> is
> not constant, which breaks the budget accounting in the RTDS
> scheduler, which assumes that the RTDS scheduler runs on homogenous
> CPUs.
> 
I actually think the root problem _is_ the the returned timeslice in
rt_schedule(), both in nested and non-nested virtualization scenarios.

The reason why I want this scheduler to become event-driver, and start
using timers for things like replenishments and activations as well as
for budget enforcement, is to avoid having the rt_schedule() function
(and all the other functions it itself calls, as of now) invoked with
sub-millisecond periodicity.

This makes very few sense, no matter whether the hypervisor is running
on bare metal or in a VM.

The proper way forward is to, _first_of_all_, fix the above, by
following up on Dagaen's work. After that, we can see whether nested
virt is still an issue and think to proper solutions.

> I'm not sure if we should always return the MAX_SCHEDULE as the time
> slice.  A VCPU may have a little bit more budget (< 1ms) than it is
> assigned on native machine if we always return MAX_SCHEDULE.
> 
With the current design, if there is a vCPU running, returning anything
that is bigger than its budget, defeats any chance of enforcing such
budget for that vCPU, making using this scheduler useless.

So, no, I don't think we should always return MAX_SCHEDULE, I think we
must fix the lack of event-driven-ess of the scheduler.

> What do you think, Dario? Do you think the nested virtualization
> situation is a concern right now?
> 
It probably is, but it is a consequence of how the scheduler has been
designed and implemented. After all, this scheduler is still
experimental for a reason, and things like this is exactly why I won't
be comfortable in declaring it !experimental, until we start using
timers and events properly in it.

So, while we are here, Meng, can I ask whether you have a prognosis for
that work? I don't like putting pressure on people, but the thing about
experimental features is that it is ok for them to stay in, but some
progress toward putting them out of such status should be made in every
release, or we'll have to consider throwing them out.

It would be therefore really good if we could get something in the 4.7
timeframe...

Thanks and Regards,
Dario

PS. If you're receiving multiple copy of this email, sorry, it's my
fault, when trying using the proper address for xen-devel! :-(
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH] sched_rt: Improved nested virtualization performance

[Meng Xu]

Hi Dario,

2015-11-19 4:51 GMT-05:00 Dario Faggioli <dario.faggioli@citrix.com>:
> Hi,
>
> Thanks Tianyang for the report and for the patch, and Meng for helping
> getting the patch itself on the list and to me, and for commenting.
>
> On Wed, 2015-11-18 at 22:38 -0500, Meng Xu wrote:
>> [cc. Dario...]
>>
>> 2015-11-18 22:24 GMT-05:00 Tianyang Chen <tianyangpenn@gmail.com>:
>> > In nested virtualization, choosing the smaller value for the
>> > time slice between the MAX_SCHEDULE and the budget will cause
>> > high host CPU usage.
>>
>> Just add one comment:
>> I also experienced this issue when I develop Xen in a virtual box on
>> MacBook Air.
>>
>> The root problem is not the returned time slice in rt_schedule(). The
>> root problem is that the CPU speed for Xen in nested virtualization
>> is
>> not constant, which breaks the budget accounting in the RTDS
>> scheduler, which assumes that the RTDS scheduler runs on homogenous
>> CPUs.
>>
> I actually think the root problem _is_ the the returned timeslice in
> rt_schedule(), both in nested and non-nested virtualization scenarios.
>
> The reason why I want this scheduler to become event-driver, and start
> using timers for things like replenishments and activations as well as
> for budget enforcement, is to avoid having the rt_schedule() function
> (and all the other functions it itself calls, as of now) invoked with
> sub-millisecond periodicity.
>
> This makes very few sense, no matter whether the hypervisor is running
> on bare metal or in a VM.
>
> The proper way forward is to, _first_of_all_, fix the above, by
> following up on Dagaen's work. After that, we can see whether nested
> virt is still an issue and think to proper solutions.

Yes. Agree. :-) That may fix the root problem.

>
>> I'm not sure if we should always return the MAX_SCHEDULE as the time
>> slice.  A VCPU may have a little bit more budget (< 1ms) than it is
>> assigned on native machine if we always return MAX_SCHEDULE.
>>
> With the current design, if there is a vCPU running, returning anything
> that is bigger than its budget, defeats any chance of enforcing such
> budget for that vCPU, making using this scheduler useless.
>
> So, no, I don't think we should always return MAX_SCHEDULE, I think we
> must fix the lack of event-driven-ess of the scheduler.
>
>> What do you think, Dario? Do you think the nested virtualization
>> situation is a concern right now?
>>
> It probably is, but it is a consequence of how the scheduler has been
> designed and implemented. After all, this scheduler is still
> experimental for a reason, and things like this is exactly why I won't
> be comfortable in declaring it !experimental, until we start using
> timers and events properly in it.
>
> So, while we are here, Meng, can I ask whether you have a prognosis for
> that work? I don't like putting pressure on people, but the thing about
> experimental features is that it is ok for them to stay in, but some
> progress toward putting them out of such status should be made in every
> release, or we'll have to consider throwing them out.

Yes. Tianyang is a master student in our department and working with me on this.
I'm asking him to update RT-Xen which is based on Xen 4.3 to a newer
version of RT-Xen which will be based on Xen 4.6.
I hope this can help him get familiar with the development process and
be finished within this year.
This updating work will finish before the end of this semester.

Starting from next January, he and I will work on picking up Dagaen's
work. The goal here is to meet the release cycle of Xen 4.7.

>
> It would be therefore really good if we could get something in the 4.7
> timeframe...

Yes. That is what I and Tianyang want too. This will also be
beneficial for his job hunting next year. :-D

>
> Thanks and Regards,

Thank you very much for your help!
>
> PS. If you're receiving multiple copy of this email, sorry, it's my
> fault, when trying using the proper address for xen-devel! :-(

That's totally not a problem at all. ;-)

Meng


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

Re: [PATCH] sched_rt: Improved nested virtualization performance

[Meng Xu]

2015-11-18 22:50 GMT-05:00 Tianyang Chen <tianyangpenn@gmail.com>:
> In nested virtualization, choosing the smaller value for the
> time slice between the MAX_SCHEDULE and the budget will cause
> high host CPU usage.

Just add one comment:
I also experienced this issue when I develop Xen in a virtual box on
MacBook Air.

The root problem is not the returned time slice in rt_schedule(). The
root problem is that the CPU speed for Xen in nested virtualization is
not constant, which breaks the budget accounting in the RTDS
scheduler, which assumes that the RTDS scheduler runs on homogenous
CPUs.

I'm not sure if we should always return the MAX_SCHEDULE as the time
slice.  A VCPU may have a little bit more budget (< 1ms) than it is
assigned on native machine if we always return MAX_SCHEDULE.

What do you think, Dario? Do you think the nested virtualization
situation is a concern right now?


>
> Signed-off-by: Tianyang Chen <tianyangpenn@gmail.com>
> ---
>  xen/common/sched_rt.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
> index 4372486..9da3f35 100644
> --- a/xen/common/sched_rt.c
> +++ b/xen/common/sched_rt.c
> @@ -889,7 +889,7 @@ rt_schedule(const struct scheduler *ops, s_time_t now, bool_t tasklet_work_sched
>          }
>      }
>
> -    ret.time = MIN(snext->budget, MAX_SCHEDULE); /* sched quantum */
> +    ret.time = MAX_SCHEDULE; /*should be used in nested virtualization*/
>      ret.task = snext->vcpu;
>

Tianyang,
The comment for this line should not be changed. The new comment
should be in the commit message since this is the explanation of the
whole patch.

Thanks,

Meng


-----------
Meng Xu
PhD Student in Computer and Information Science
University of Pennsylvania
http://www.cis.upenn.edu/~mengxu/

[PATCH] sched_rt: Improved nested virtualization performance

[Tianyang Chen]

In nested virtualization, choosing the smaller value for the
time slice between the MAX_SCHEDULE and the budget will cause
high host CPU usage.

Signed-off-by: Tianyang Chen <tianyangpenn@gmail.com>
---
 xen/common/sched_rt.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c
index 4372486..9da3f35 100644
--- a/xen/common/sched_rt.c
+++ b/xen/common/sched_rt.c
@@ -889,7 +889,7 @@ rt_schedule(const struct scheduler *ops, s_time_t now, bool_t tasklet_work_sched
         }
     }
 
-    ret.time = MIN(snext->budget, MAX_SCHEDULE); /* sched quantum */
+    ret.time = MAX_SCHEDULE; /*should be used in nested virtualization*/
     ret.task = snext->vcpu;
 
     /* TRACE */
-- 
1.7.9.5