Re: [RFC PATCH v4 00/19] Core scheduling v4

[Joel Fernandes <joel@joelfernandes.org>]

Hi Julien, Peter, all,

On Fri, Feb 21, 2020 at 06:20:57PM -0500, Julien Desfossez wrote:
> On 18-Feb-2020 04:58:02 PM, Vineeth Remanan Pillai wrote:
> > > Yes, this makes sense, patch updated at here, I put your name there if
> > > you don't mind.
> > > https://github.com/aubreyli/linux/tree/coresched_v4-v5.5.2-rc2
> > >
> > > Thanks Aubrey!
> 
> Just a quick note, I ran a very cpu-intensive benchmark (9x12 vcpus VMs
> running linpack), all affined to an 18 cores NUMA node (36 hardware
> threads). Each VM is running in its own cgroup/tag with core scheduling
> enabled. We know it already performed much better than nosmt, so for
> this case, I measured various co-scheduling statistics:
> - how much time the process spends co-scheduled with idle, a compatible
>   or an incompatible task
> - how long does the process spends running in a inefficient
>   configuration (more than 1 thread running alone on a core)
> 
> And I am very happy to report than even though the 9 VMs were configured
> to float on the whole NUMA node, the scheduler / load-balancer did a
> very good job at keeping an efficient configuration:
> 
> Process 10667 (qemu-system-x86), 10 seconds trace:
>   - total runtime: 46451472309 ns,
>   - local neighbors (total: 45713285084 ns, 98.411 % of process runtime):
>   - idle neighbors (total: 484054061 ns, 1.042 % of process runtime):
>   - foreign neighbors (total: 4191002 ns, 0.009 % of process runtime):
>   - unknown neighbors  (total: 92042503 ns, 0.198 % of process runtime)
>   - inefficient periods (total: 464832 ns, 0.001 % of process runtime):
>     - number of periods: 48
>     - min period duration: 1424 ns
>     - max period duration: 116988 ns
>     - average period duration: 9684.000 ns
>     - stdev: 19282.130
> 
> I thought you would enjoy seeing this :-)

Looks quite interesting. We are trying apply this work to ChromeOS. What we
want to do is selectively marking tasks, instead of grouping sets of trusted
tasks. I have a patch that adds a prctl which a task can call, and it works
well (task calls prctl and gets a cookie which gives it a dedicated core).

However, I have the following questions, in particular there are 4 scenarios
where I feel the current patches do not resolve MDS/L1TF, would you guys
please share your thoughts?

1. HT1 is running either hostile guest or host code.
   HT2 is running an interrupt handler (victim).

   In this case I see there is a possible MDS issue between HT1 and HT2.

2. HT1 is executing hostile host code, and gets interrupted by a victim
   interrupt. HT2 is idle.

   In this case, I see there is a possible MDS issue between interrupt and
   the host code on the same HT1.

3. HT1 is executing hostile guest code, HT2 is executing a victim interrupt
   handler on the host.

   In this case, I see there is a possible L1TF issue between HT1 and HT2.
   This issue does not happen if HT1 is running host code, since the host
   kernel takes care of inverting PTE bits.

4. HT1 is idle, and HT2 is running a victim process. Now HT1 starts running
   hostile code on guest or host. HT2 is being forced idle. However, there is
   an overlap between HT1 starting to execute hostile code and HT2's victim
   process getting scheduled out.
   Speaking to Vineeth, we discussed an idea to monitor the core_sched_seq
   counter of the sibling being idled to detect that it is now idle.
   However we discussed today that looking at this data, it is not really an
   issue since it is such a small window.

My concern is now cases 1, 2 to which there does not seem a good solution,
short of disabling interrupts. For 3, we could still possibly do something on
the guest side, such as using shadow page tables. Any thoughts on all this?

thanks,

- Joel