Core Scheduling - Concurrent VMs

Core Scheduling - Concurrent VMs

[Rajendran, Jaishankar]

Refer to the below experiments performed using Core Scheduling for Concurrent VMs and we found the benchmark scores executed in different VMs are 
degrading after enablement of Core Scheduling. Both the host and guest are enabled with Core Scheduler

Environment:
++++++++++
Platform - CML-NUC - i5-10600 @ 3.3 Ghz
Core Assignment for Host OS : No .of Cores     : 4   No .of Threads : 4 ( i.e. 8 Logical Cores)
Host OS - Ubuntu 20.1 with Chromium Kernel 5.4
Guest OS - Android OS running as VM using qemu/kvm with Chromium Kernel 5.4
VM's are assigned with 4 cores and 4 Threads (i.e. 8 Logical Cores)

Kernel Config:
+++++++++++
Host and Guest OS Kernel is enabled with CONFIG_SCHED_CORE=y

CGROUP Mapping:
++++++++++++++++
We did follow the CGroup approach for Core Scheduling based on the below document
https://lkml.org/lkml/2021/1/22/1469 
 and configured the CGroup. But not able to find cpu.core_tag file in CGroups?

VM1 (Android OS running using qemu/KVM) is executed and its default cgroup is changed to caas1
VM2 (Android OS running using qemu/KVM) is executed and its default cgroup is changed to caas2
The changes are validated using htop command.

WorkLoad:
+++++++++
GeekBench 5.3.1 - Multi core Test executed on both the VMs
 
Observations:
++++++++++++
As per core CORE_SCHEDULER documentation, (vCPU) Threads of VM1 & VM2 (belonging to two different cgroups ) should not be scheduled on the same core at a given time.
But we observe that vCPU Threads of different VMs are scheduled in the same core ( Used HTOP and ps commands)

Benchmark Scores:
+++++++++++++++
With the above changes we are seeing degradation in the Multi-Threaded scores 

Without Core Scheduling:
++++++++++++++++++++
GeekBench - MultiCore Test- Concurrent Run (Two VMs) - Performance Gap is 2%
With Core Scheduling 
+++++++++++++++++
GeekBench- MultiCoreTes - Concurrent Run (Two VMs) - Performance Gap is 18%

Pl advice? 

Thanks,
Jaishankar Rajendran / Raju 

Re: Core Scheduling - Concurrent VMs

[Joel Fernandes]

Hi,
+Dario Faggioli and +Vineeth Pillai as well.

On Mon, Jul 26, 2021 at 1:41 AM Rajendran, Jaishankar
<jaishankar.rajendran@intel.com> wrote:
>
> Refer to the below experiments performed using Core Scheduling for Concurrent VMs and we found the benchmark scores executed in different VMs are
> degrading after enablement of Core Scheduling. Both the host and guest are enabled with Core Scheduler

Why are you running core scheduling within the guest? Only the host
needs it and you tag vCPU threads in the host. Within the guest you
don't need it (and it probably doesn't make sense unless you pin the
vCPU threads to individual physical hardware threads).

>
> Environment:
> ++++++++++
> Platform - CML-NUC - i5-10600 @ 3.3 Ghz
> Core Assignment for Host OS : No .of Cores     : 4   No .of Threads : 4 ( i.e. 8 Logical Cores)
> Host OS - Ubuntu 20.1 with Chromium Kernel 5.4
> Guest OS - Android OS running as VM using qemu/kvm with Chromium Kernel 5.4
> VM's are assigned with 4 cores and 4 Threads (i.e. 8 Logical Cores)
>
> Kernel Config:
> +++++++++++
> Host and Guest OS Kernel is enabled with CONFIG_SCHED_CORE=y
>
> CGROUP Mapping:
> ++++++++++++++++
> We did follow the CGroup approach for Core Scheduling based on the below document
> https://lkml.org/lkml/2021/1/22/1469
>  and configured the CGroup. But not able to find cpu.core_tag file in CGroups?
>
> VM1 (Android OS running using qemu/KVM) is executed and its default cgroup is changed to caas1
> VM2 (Android OS running using qemu/KVM) is executed and its default cgroup is changed to caas2

It is not clear to me how the individual vCPU threads are tagged so it
is hard to say much.

> The changes are validated using htop command.
>
> WorkLoad:
> +++++++++
> GeekBench 5.3.1 - Multi core Test executed on both the VMs
>
> Observations:
> ++++++++++++
> As per core CORE_SCHEDULER documentation, (vCPU) Threads of VM1 & VM2 (belonging to two different cgroups ) should not be scheduled on the same core at a given time.
> But we observe that vCPU Threads of different VMs are scheduled in the same core ( Used HTOP and ps commands)

I don't think these commands can really tell you that core scheduler
is working or not. They are too coarse grained. You would need to look
at it through ftrace scheduling events. Core scheduling will still
allow differently tagged threads to use a core, it is just that it
wont allow it simultaneously.

Also just to note, in ChromeOS, we are working on some changes to tag
the whole Android VM with same tag (all vCPU threads in a single VM
can share a core) which should improve performance. Previously we were
putting different tags for each vCPU thread of the Android VM which
seemed overkill.

Re: Core Scheduling - Concurrent VMs

[Peter Zijlstra]

On Mon, Jul 26, 2021 at 11:57:21AM -0400, Joel Fernandes wrote:
> Hi,
> +Dario Faggioli and +Vineeth Pillai as well.
> 
> On Mon, Jul 26, 2021 at 1:41 AM Rajendran, Jaishankar
> <jaishankar.rajendran@intel.com> wrote:
> >
> > Refer to the below experiments performed using Core Scheduling for Concurrent VMs and we found the benchmark scores executed in different VMs are
> > degrading after enablement of Core Scheduling. Both the host and guest are enabled with Core Scheduler
> 
> Why are you running core scheduling within the guest? Only the host
> needs it and you tag vCPU threads in the host. Within the guest you
> don't need it (and it probably doesn't make sense unless you pin the
> vCPU threads to individual physical hardware threads).

Correct, in-guest core-scheduling only works if there's an in-guest
topology, which requires pinning vcpu threads and sacrificing a chicken
on the altar of qemu to get the right incantation such that the guest
actually receives the right topology.

Nested core scheduling hurts my head.

> I don't think these commands can really tell you that core scheduler
> is working or not. They are too coarse grained. You would need to look
> at it through ftrace scheduling events. Core scheduling will still
> allow differently tagged threads to use a core, it is just that it
> wont allow it simultaneously.

Right, you need tracing to verify it actually works, no real alternative
there.