about CPU QoS in KVM

about CPU QoS in KVM

[Gonglei (Arei)]

Hi all,

Currently, KVM do the CPU resource reservation by the cgroup mechanism
which can't do entire accurate separation because the capacity of the Linux
scheduler. Take the public cloud as an example, some customers rent one vm
with 8 CPUs paid by enough money, they want to get enough response
speed on CPU scheduling. So we (the cloud platform providers ) reserve 1GHz
CPU resources by cgroup for those VM's vcpu/pcpu. 

But the actual effects can't meet those requirements because the cgroup is
limiting share usage of other processes in order to attach the reservation proportion,
but the scheduler can't assure that. This mechanism is different with Xen,
We can directly change the CPU weight on Xen hypervisor so that we can
get entire accurate control on CPU resources based on accurate capacity (upper limit),
share (weight) and reservation.

So my question is do we have a good method to do CPU reservation in KVM?

Thanks,
-Gonglei

about CPU QoS in KVM

[Gonglei (Arei)]

Hi all,

Currently, KVM do the CPU resource reservation by the cgroup mechanism
which can't do entire accurate separation because the capacity of the Linux
scheduler. Take the public cloud as an example, some customers rent one vm
with 8 CPUs paid by enough money, they want to get enough response
speed on CPU scheduling. So we (the cloud platform providers ) reserve 1GHz
CPU resources by cgroup for those VM's vcpu/pcpu. 

But the actual effects can't meet those requirements because the cgroup is
limiting share usage of other processes in order to attach the reservation proportion,
but the scheduler can't assure that. This mechanism is different with Xen,
We can directly change the CPU weight on Xen hypervisor so that we can
get entire accurate control on CPU resources based on accurate capacity (upper limit),
share (weight) and reservation.

So my question is do we have a good method to do CPU reservation in KVM?

Thanks,
-Gonglei

Re: about CPU QoS in KVM

[Paolo Bonzini]

Adding Rik.

Paolo

On 20/04/2017 15:32, Gonglei (Arei) wrote:
> Hi all,
> 
> Currently, KVM do the CPU resource reservation by the cgroup mechanism
> which can't do entire accurate separation because the capacity of the Linux
> scheduler. Take the public cloud as an example, some customers rent one vm
> with 8 CPUs paid by enough money, they want to get enough response
> speed on CPU scheduling. So we (the cloud platform providers ) reserve 1GHz
> CPU resources by cgroup for those VM's vcpu/pcpu. 
> 
> But the actual effects can't meet those requirements because the cgroup is
> limiting share usage of other processes in order to attach the reservation proportion,
> but the scheduler can't assure that. This mechanism is different with Xen,
> We can directly change the CPU weight on Xen hypervisor so that we can
> get entire accurate control on CPU resources based on accurate capacity (upper limit),
> share (weight) and reservation.
> 
> So my question is do we have a good method to do CPU reservation in KVM?
> 
> Thanks,
> -Gonglei
> 
> 

Re: about CPU QoS in KVM

[Paolo Bonzini]

Adding Rik.

Paolo

On 20/04/2017 15:32, Gonglei (Arei) wrote:
> Hi all,
> 
> Currently, KVM do the CPU resource reservation by the cgroup mechanism
> which can't do entire accurate separation because the capacity of the Linux
> scheduler. Take the public cloud as an example, some customers rent one vm
> with 8 CPUs paid by enough money, they want to get enough response
> speed on CPU scheduling. So we (the cloud platform providers ) reserve 1GHz
> CPU resources by cgroup for those VM's vcpu/pcpu. 
> 
> But the actual effects can't meet those requirements because the cgroup is
> limiting share usage of other processes in order to attach the reservation proportion,
> but the scheduler can't assure that. This mechanism is different with Xen,
> We can directly change the CPU weight on Xen hypervisor so that we can
> get entire accurate control on CPU resources based on accurate capacity (upper limit),
> share (weight) and reservation.
> 
> So my question is do we have a good method to do CPU reservation in KVM?
> 
> Thanks,
> -Gonglei
> 
> 

Re: about CPU QoS in KVM

[Juri Lelli]

Hi,

just noticed this and thought I might just jump in. :)

On 20/04/17 16:00, Paolo Bonzini wrote:
> Adding Rik.
> 
> Paolo
> 
> On 20/04/2017 15:32, Gonglei (Arei) wrote:
> > Hi all,
> > 
> > Currently, KVM do the CPU resource reservation by the cgroup mechanism
> > which can't do entire accurate separation because the capacity of the Linux
> > scheduler. Take the public cloud as an example, some customers rent one vm
> > with 8 CPUs paid by enough money, they want to get enough response
> > speed on CPU scheduling. So we (the cloud platform providers ) reserve 1GHz
> > CPU resources by cgroup for those VM's vcpu/pcpu. 
> > 
> > But the actual effects can't meet those requirements because the cgroup is
> > limiting share usage of other processes in order to attach the reservation proportion,
> > but the scheduler can't assure that. This mechanism is different with Xen,
> > We can directly change the CPU weight on Xen hypervisor so that we can
> > get entire accurate control on CPU resources based on accurate capacity (upper limit),
> > share (weight) and reservation.
> > 
> > So my question is do we have a good method to do CPU reservation in KVM?
> > 
> > Thanks,
> > -Gonglei

Not entirely sure what your particular requirements are Gonglei, but you
might be interested to know that there has been research work [1,2,3,
just to name a few] that used a mainline real-time scheduling policy
(SCHED_DEADLINE) to provide Qos support to virtual machines (KVM).

I won't enter in too much detail, but the basic idea is to use
reservation based scheduling mechanisms to enforce temporal isolation and
guaranteed CPU bandwidth to VM's vcpu(s).

I'm Cc-ing Tommaso, Luca and Carlo, whom can provide more information as
needed.

Best,

- Juri

[1] - http://retis.sssup.it/~nino/publication/rtlws14sdnnfs.pdf
[2] - http://retis.sssup.it/~tommaso/publications/VHPC-2010.pdf
[3] - http://retis.sssup.it/~tommaso/publications/RTSOAA-2009-RTV.pdf

Skimming through Tommaso and Luca publications might be interesting as
well. I'm pretty sure I missed the most importants papers. :)
http://retis.sssup.it/~tommaso/eng/publications.html
https://scholar.google.co.uk/citations?user=C3a6glEAAAAJ&hl=en

Re: about CPU QoS in KVM

[Luca Abeni]

Hi all,

On Tue, 25 Apr 2017 11:13:03 +0100
Juri Lelli <juri.lelli@arm.com> wrote:
[...]
> > > Currently, KVM do the CPU resource reservation by the cgroup
> > > mechanism which can't do entire accurate separation because the
> > > capacity of the Linux scheduler. Take the public cloud as an
> > > example, some customers rent one vm with 8 CPUs paid by enough
> > > money, they want to get enough response speed on CPU scheduling.
> > > So we (the cloud platform providers ) reserve 1GHz CPU resources
> > > by cgroup for those VM's vcpu/pcpu. 
> > > 
> > > But the actual effects can't meet those requirements because the
> > > cgroup is limiting share usage of other processes in order to
> > > attach the reservation proportion, but the scheduler can't assure
> > > that. This mechanism is different with Xen, We can directly
> > > change the CPU weight on Xen hypervisor so that we can get entire
> > > accurate control on CPU resources based on accurate capacity
> > > (upper limit), share (weight) and reservation.
> > > 
> > > So my question is do we have a good method to do CPU reservation
> > > in KVM?
> > > 
> > > Thanks,
> > > -Gonglei  
> 
> Not entirely sure what your particular requirements are Gonglei, but
> you might be interested to know that there has been research work
> [1,2,3, just to name a few] that used a mainline real-time scheduling
> policy (SCHED_DEADLINE) to provide Qos support to virtual machines
> (KVM).
> 
> I won't enter in too much detail, but the basic idea is to use
> reservation based scheduling mechanisms to enforce temporal isolation
> and guaranteed CPU bandwidth to VM's vcpu(s).

Since I've been CC-ed, I add some details to what Juri wrote:
if the guest workload is mostly CPU intensive, just scheduling the vCPU
threads with SCHED_DEADLINE allows deterministic CPU allocation.

I recently showed some experiments about this to some students: if some
periodic real-time tasks are executed in the guest and KVM's vCPU
threads are scheduled with SCHED_DEADLINE, then the number of missed
deadlines measured in the guest matches with the number of missed
deadlines expected from theoretical analysis (using real-time
hierarchical scheduling analysis).
[I can provide more details if needed: here is a quick and incomplete
summary I wrote for the students:
http://retis.sssup.it/~luca/CBSD/h-sched_experiments.txt
up to item 4 it is just theoretical schedulability analysis; the
interesting parts start from item 5]

I think this shows that (for CPU-intensive workloads) it is possible to
deterministically control the QoS of a KVM-based VM by using
SCHED_DEADLINE.


If your guest workload includes more I/O, then you have to schedule
more threads with SCHED_DEADLINE (for example, I have some experiments
with a lot of network traffic in the guest; I used vhost-net and I've
been able to control the network throughput by scheduling the vhost-net
kernel thread with SCHED_DEADLINE and changing the runtime and period
associated to thread).



				Luca

> 
> I'm Cc-ing Tommaso, Luca and Carlo, whom can provide more information
> as needed.
> 
> Best,
> 
> - Juri
> 
> [1] - http://retis.sssup.it/~nino/publication/rtlws14sdnnfs.pdf
> [2] - http://retis.sssup.it/~tommaso/publications/VHPC-2010.pdf
> [3] - http://retis.sssup.it/~tommaso/publications/RTSOAA-2009-RTV.pdf
> 
> Skimming through Tommaso and Luca publications might be interesting as
> well. I'm pretty sure I missed the most importants papers. :)
> http://retis.sssup.it/~tommaso/eng/publications.html
> https://scholar.google.co.uk/citations?user=C3a6glEAAAAJ&hl=en