[MODERATED] L1D-Fault KVM mitigation

[MODERATED] L1D-Fault KVM mitigation

[Joerg Roedel]

Hey,

I've been looking into the mitigation for the L1D fault issue in KVM,
and since the hardware seems to speculate with the GPA as an HPA, it
seems we have to disable SMT to be fully secure here because otherwise
two different guests running on HT siblings could spy on each other.

I'd like to discuss how we mitigate this, the big hammer would be not
initializing the HT siblings at boot on affected machines, but that is
probably a bit too eager as it also penalizes people not using KVM.

Another option is to just print a fat warning and/or refuse to load the
KVM modules on affected machines when HT is enabled.

So what are the opinions on how we should best mitigate this issue?


Regards,

	Joerg

[MODERATED] Re: L1D-Fault KVM mitigation

[Peter Zijlstra]

On Tue, Apr 24, 2018 at 11:06:30AM +0200, speck for Joerg Roedel wrote:
> Hey,
> 
> I've been looking into the mitigation for the L1D fault issue in KVM,
> and since the hardware seems to speculate with the GPA as an HPA, it
> seems we have to disable SMT to be fully secure here because otherwise
> two different guests running on HT siblings could spy on each other.
> 
> I'd like to discuss how we mitigate this, the big hammer would be not
> initializing the HT siblings at boot on affected machines, but that is
> probably a bit too eager as it also penalizes people not using KVM.
> 
> Another option is to just print a fat warning and/or refuse to load the
> KVM modules on affected machines when HT is enabled.
> 
> So what are the opinions on how we should best mitigate this issue?

Another option, that is being explored, is to co-schedule siblings.
So ensure all siblings either run vcpus of the _same_ VM or idle.

Of course, this is all rather intrusive and ugly and brings with it
setup costs as well, because you'd have to sync up on VMENTER, VMEXIT
and interrupts (on the idle CPUs).

Another complication is that on overcommitted systems the regular load
balancer will happily migrate vcpu tasks around. So it is fairly tricky
to ensure runnable vcpu threads of the same VM are in fact around to be
ran on a core.

Not to mention that Linus has basically said: "No way, Jose".

I know that I worked a little with Tim on this, and I know Google did
their own thing (but have not seen patches from them -- is pjt on this
list?). I've also heard Amazon was also working on things (are they
here?). And I think RHT was also looking into something (mingo, bonzini
-- are you guys reading?)

In any case, if any of that is to go fly we need very solid numbers to
convince Linus to reconsider.

Another idea that I had was to only allow trusted guest kernels, as in
trusted computing, key verified images etc.. Of course, they too can be
compromised, but hopefully it avoids the most egregious hostile guest
scenarios.

[MODERATED] Re: L1D-Fault KVM mitigation

[David Woodhouse]

On Tue, 2018-04-24 at 11:35 +0200, speck for Peter Zijlstra wrote:
> 
> Another option, that is being explored, is to co-schedule siblings.
> So ensure all siblings either run vcpus of the _same_ VM or idle.
> 
> Of course, this is all rather intrusive and ugly and brings with it
> setup costs as well, because you'd have to sync up on VMENTER, VMEXIT
> and interrupts (on the idle CPUs).

I hate to suggest more microcode hacks but... if there was an MSR bit
which, when set, would pause any HT sibling that was currently in VMX
non-root mode, then we could set that up to be automatically set on
vmexit and it would automatically pause the problematic siblings.
Meaning that co-ordinating vmexits with them might actually be
feasible?

The precise definition of 'pause' in the above could survive some
bikeshedding, but basically it shouldn't run any more guest
instructions, but it *should* be allowed to vmexit on interrupts, etc.

[MODERATED] Re: L1D-Fault KVM mitigation

[Peter Zijlstra]

On Tue, Apr 24, 2018 at 10:48:12AM +0100, speck for David Woodhouse wrote:
> On Tue, 2018-04-24 at 11:35 +0200, speck for Peter Zijlstra wrote:
> > 
> > Another option, that is being explored, is to co-schedule siblings.
> > So ensure all siblings either run vcpus of the _same_ VM or idle.
> > 
> > Of course, this is all rather intrusive and ugly and brings with it
> > setup costs as well, because you'd have to sync up on VMENTER, VMEXIT
> > and interrupts (on the idle CPUs).
> 
> I hate to suggest more microcode hacks but... if there was an MSR bit
> which, when set, would pause any HT sibling that was currently in VMX
> non-root mode, then we could set that up to be automatically set on
> vmexit and it would automatically pause the problematic siblings.
> Meaning that co-ordinating vmexits with them might actually be
> feasible?

Not sure I'm following. The above assumes a sibling is running a VCPU of
another VM, right? But it could equally well run any regular old task
(including idle).

So only pausing siblings in VMX mode wouldn't help anything. The !VMX
tasks could still be loading stuff into L1.

[MODERATED] Re: L1D-Fault KVM mitigation

[David Woodhouse]

On Tue, 2018-04-24 at 13:04 +0200, speck for Peter Zijlstra wrote:
> Not sure I'm following. The above assumes a sibling is running a VCPU of
> another VM, right? But it could equally well run any regular old task
> (including idle).
> 
> So only pausing siblings in VMX mode wouldn't help anything. The !VMX
> tasks could still be loading stuff into L1.

Er, yeah... I may have briefly forgotten that some people sometimes run
actual userspace, not just VM guests.

It's ring 3 *and* VMX non-root which would need to be paused on HT
siblings. And it would need to be triggered on any transition back into
the kernel from userspace too, not just vmexit. Which makes it a little
bit harder.

[MODERATED] Re: L1D-Fault KVM mitigation

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 2425 bytes --]

On 04/24/2018 07:16 AM, speck for David Woodhouse wrote:
> On Tue, 2018-04-24 at 13:04 +0200, speck for Peter Zijlstra wrote:
>> Not sure I'm following. The above assumes a sibling is running a
>> VCPU of another VM, right? But it could equally well run any regular
>> old task (including idle).

>> So only pausing siblings in VMX mode wouldn't help anything. The
>> !VMX tasks could still be loading stuff into L1.

> Er, yeah... I may have briefly forgotten that some people sometimes
> run actual userspace, not just VM guests.

:)

More than once over the past few months, I've pointed out that Annapurna
was the right way to go. Personally, I think the only possible way to
make any of this safe is by moving "all" userspace off host like AMZ.

> It's ring 3 *and* VMX non-root which would need to be paused on HT
> siblings. And it would need to be triggered on any transition back
> into the kernel from userspace too, not just vmexit. Which makes it a
> little bit harder.

The additional ucode hack (pausing the sibling thread) has been
discussed at some length already. It's a good idea, except for the host
side as you note, and there we have all manner of interrupts to handle.
In the case of a traditional OS on the host, that's a lot of code
happily pulling secrets into the L1 and lots of paths to track if we
wanted to do like one of the others (dunno if I can say who, but someone
is tracking all secrets loaded into the L1 and scrubbing them after, but
they also refactored enough other stuff to make that just about work).

On our end, we've discussed automatic handling at boot along the lines
that have been pondered this morning here. The problem is that you don't
want to penalize until people run KVM, and you don't want to them hot
unplug or do crazy things that might affect tunings (I already got shot
down, rightly, for suggesting that one). So, instead, it's going to have
to be messaging, and maybe tainting as insecure of some kind.

I've requested that RHEL's installer be modified to effectively add a
checkbox that defaults to enabled but very prominently offers to disable
HT if you're using virtualization. We'll then need some guidance
coordinated across the industry that for total security (even in the
face of some of the gang scheduling proposals) you need to !HT.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

[MODERATED] Re: L1D-Fault KVM mitigation

[David Woodhouse]

On Tue, 2018-04-24 at 13:04 +0200, speck for Peter Zijlstra wrote:
> On Tue, Apr 24, 2018 at 10:48:12AM +0100, speck for David Woodhouse wrote:
> > 
> > On Tue, 2018-04-24 at 11:35 +0200, speck for Peter Zijlstra wrote:
> > > 
> > > 
> > > Another option, that is being explored, is to co-schedule siblings.
> > > So ensure all siblings either run vcpus of the _same_ VM or idle.
> > > 
> > > Of course, this is all rather intrusive and ugly and brings with it
> > > setup costs as well, because you'd have to sync up on VMENTER, VMEXIT
> > > and interrupts (on the idle CPUs).
>
> > I hate to suggest more microcode hacks but... if there was an MSR bit
> > which, when set, would pause any HT sibling that was currently in VMX
> > non-root mode, then we could set that up to be automatically set on
> > vmexit and it would automatically pause the problematic siblings.
> > Meaning that co-ordinating vmexits with them might actually be
> > feasible?

> Not sure I'm following. The above assumes a sibling is running a VCPU of
> another VM, right? But it could equally well run any regular old task
> (including idle).
> 
> So only pausing siblings in VMX mode wouldn't help anything. The !VMX
> tasks could still be loading stuff into L1.

That's OK because it's only the VMX tasks which can abuse it, isn't it?

Let's assume we've fixed the problem for normal tasks, by flipping the
top bit in absent PTEs that actually contain swap pointers, etc.

The only thing we have left is VM guests. The microcode bit would say
that *if* a CPU thread is in non-root mode then *it* gets paused unless
its sibling is also in non-root mode for the same VMID.

So when both siblings are actually in the VM, they get to run. If one
sibling comes *out* of the VM to the host kernel or to run (host)
userspace, then the other one doesn't execute any guest instructions.
It can take exceptions which cause a vmexit though.

We'd also want a vCPU to be able to run if its sibling is actually in
the host but *idle* (and has flushed the L1. Perhaps we actually
automatically flush the L1 when resuming a sibling that got paused).

It does still depend on gang scheduling (or at least forced sibling
idle which is a subset of that), or a singleton vCPU might *never* get
run. But we were going to have to do something along those lines
anyway. The microcode trick just makes it a lot easier because we don't
have to *explicitly* pause the sibling vCPUs and manage their state on
every vmexit/entry. And avoids potential race conditions with managing
that in software.

[MODERATED] Re: L1D-Fault KVM mitigation

[Peter Zijlstra]

On Wed, May 23, 2018 at 10:45:45AM +0100, speck for David Woodhouse wrote:

> That's OK because it's only the VMX tasks which can abuse it, isn't it?

If, like you outline below, this is an (optional) ucode assist to
co-scheduling matching VCPU threads, then yes.

> Let's assume we've fixed the problem for normal tasks, by flipping the
> top bit in absent PTEs that actually contain swap pointers, etc.
> 
> The only thing we have left is VM guests. The microcode bit would say
> that *if* a CPU thread is in non-root mode then *it* gets paused unless
> its sibling is also in non-root mode for the same VMID.
> 
> So when both siblings are actually in the VM, they get to run. If one
> sibling comes *out* of the VM to the host kernel or to run (host)
> userspace, then the other one doesn't execute any guest instructions.
> It can take exceptions which cause a vmexit though.

Would it make sense to time limit the being 'stuck', much like PLE ?

> We'd also want a vCPU to be able to run if its sibling is actually in
> the host but *idle* (and has flushed the L1. Perhaps we actually
> automatically flush the L1 when resuming a sibling that got paused).

Right, idle is a wildcard which matches with any VCPU. We don't care
about the cache state of the sibling though. L1 is shared and since
VMENTER must flush L1, that is sufficient.

> It does still depend on gang scheduling (or at least forced sibling
> idle which is a subset of that), or a singleton vCPU might *never* get
> run. But we were going to have to do something along those lines
> anyway.

Linus has opinions on that.. but yes, without that all that remains is
disabling HT afaict.

> The microcode trick just makes it a lot easier because we don't
> have to *explicitly* pause the sibling vCPUs and manage their state on
> every vmexit/entry. And avoids potential race conditions with managing
> that in software.

Yes, it would certainly help and avoid a fair bit of ugly. It would, for
instance, avoid having to modify irq_enter() / irq_exit(), which would
otherwise be required (and possibly leak all data touched up until that
point is reached).

But even with all that, adding L1-flush to every VMENTER will hurt lots.
Consider for example the PIO emulation used when booting a guest from a
disk image. That causes VMEXIT/VMENTER at stupendous rates.

Also, none of this readily addresses the problem of load-balancing
shredding the VCPU localities required for this.

[MODERATED] Re: L1D-Fault KVM mitigation

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

On 05/24/2018 05:45 AM, speck for Peter Zijlstra wrote:
> On Wed, May 23, 2018 at 10:45:45AM +0100, speck for David Woodhouse wrote:

>> The microcode trick just makes it a lot easier because we don't
>> have to *explicitly* pause the sibling vCPUs and manage their state on
>> every vmexit/entry. And avoids potential race conditions with managing
>> that in software.
> 
> Yes, it would certainly help and avoid a fair bit of ugly.

We specifically requested this microcode change a few months ago and
were told that it isn't possible to implement across the fleet. If that
changes/changed, then great, wonderful, we're all for it.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

On Thu, 24 May 2018, speck for Peter Zijlstra wrote:
> On Wed, May 23, 2018 at 10:45:45AM +0100, speck for David Woodhouse wrote:
> > The microcode trick just makes it a lot easier because we don't
> > have to *explicitly* pause the sibling vCPUs and manage their state on
> > every vmexit/entry. And avoids potential race conditions with managing
> > that in software.
> 
> Yes, it would certainly help and avoid a fair bit of ugly. It would, for
> instance, avoid having to modify irq_enter() / irq_exit(), which would
> otherwise be required (and possibly leak all data touched up until that
> point is reached).
> 
> But even with all that, adding L1-flush to every VMENTER will hurt lots.
> Consider for example the PIO emulation used when booting a guest from a
> disk image. That causes VMEXIT/VMENTER at stupendous rates.

Just did a test on SKL Client where I have ucode. It does not have HT so
its not suffering from any HT side effects when L1D is flushed.

Boot time from a disk image is ~1s measured from the first vcpu enter.

With L1D Flush on vmenter the boot time is about 5-10% slower. And that has
lots of PIO operations in the early boot.

For a kernel build the L1D Flush has an overhead of < 1%.

Netperf guest to host has a slight drop of the throughput in the 2%
range. Host to guest surprisingly goes up by ~3%. Fun stuff!

Now I isolated two host CPUs and pinned the two vCPUs on it to be able to
measure the overhead. Running cyclictest with a period of 25us in the guest
on a isolated guest CPU and monitoring the behaviour with perf on the host
for the corresponding host CPU gives

No Flush	      	       Flush

1.31 insn per cycle	       1.14 insn per cycle

2e6 L1-dcache-load-misses/sec  26e6 L1-dcache-load-misses/sec

In that simple test the L1D misses go up by a factor of 13.

Now with the whole gang scheduling the numbers I heard through the
grapevine are in the range of factor 130, i.e. 13k% for a simple boot from
disk image. 13 minutes instead of 6 seconds...

That's not surprising at all, though the magnitude is way higher than I
expected. I don't see a realistic chance for vmexit heavy workloads to work
with that synchronization thing at all, whether it's ucode assisted or not.
 
The only workload types which will ever benefit from that co-scheduling
stuff are CPU bound workloads which more or less never vmexit. But are
those workloads really workloads which benefit from HT? Compute workloads
tend to use floating point or vector instructions which are not really HT
friendly.

Can the virt folks who know what runs on their clowdy offerings please shed
some light on this? Has anyone made a proper analysis of clowd workloads
and their behaviour on HT and their vmexit rates?

Thanks,

	tglx

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

On Thu, 24 May 2018, speck for Thomas Gleixner wrote:
> On Thu, 24 May 2018, speck for Peter Zijlstra wrote:
> > On Wed, May 23, 2018 at 10:45:45AM +0100, speck for David Woodhouse wrote:
> > > The microcode trick just makes it a lot easier because we don't
> > > have to *explicitly* pause the sibling vCPUs and manage their state on
> > > every vmexit/entry. And avoids potential race conditions with managing
> > > that in software.
> > 
> > Yes, it would certainly help and avoid a fair bit of ugly. It would, for
> > instance, avoid having to modify irq_enter() / irq_exit(), which would
> > otherwise be required (and possibly leak all data touched up until that
> > point is reached).
> > 
> > But even with all that, adding L1-flush to every VMENTER will hurt lots.
> > Consider for example the PIO emulation used when booting a guest from a
> > disk image. That causes VMEXIT/VMENTER at stupendous rates.
> 
> Just did a test on SKL Client where I have ucode. It does not have HT so
> its not suffering from any HT side effects when L1D is flushed.
> 
> Boot time from a disk image is ~1s measured from the first vcpu enter.
> 
> With L1D Flush on vmenter the boot time is about 5-10% slower. And that has
> lots of PIO operations in the early boot.
> 
> For a kernel build the L1D Flush has an overhead of < 1%.
> 
> Netperf guest to host has a slight drop of the throughput in the 2%
> range. Host to guest surprisingly goes up by ~3%. Fun stuff!
> 
> Now I isolated two host CPUs and pinned the two vCPUs on it to be able to
> measure the overhead. Running cyclictest with a period of 25us in the guest
> on a isolated guest CPU and monitoring the behaviour with perf on the host
> for the corresponding host CPU gives
> 
> No Flush	      	       Flush
> 
> 1.31 insn per cycle	       1.14 insn per cycle
> 
> 2e6 L1-dcache-load-misses/sec  26e6 L1-dcache-load-misses/sec
> 
> In that simple test the L1D misses go up by a factor of 13.
> 
> Now with the whole gang scheduling the numbers I heard through the
> grapevine are in the range of factor 130, i.e. 13k% for a simple boot from
> disk image. 13 minutes instead of 6 seconds...
> 
> That's not surprising at all, though the magnitude is way higher than I
> expected. I don't see a realistic chance for vmexit heavy workloads to work
> with that synchronization thing at all, whether it's ucode assisted or not.

That said, I think we should stage the host side mitigations plus the L1
flush on vmenter ASAP so we are not standing there with our pants down when
the cat comes out of the bag early. That means HT off, but it's still
better than having absolutely nothing.

The gang scheduling nonsense can be added on top when it should
surprisingly turn out to be usable at all.

Thanks,

	tglx

[MODERATED] Re: L1D-Fault KVM mitigation

[Jiri Kosina]

On Thu, 24 May 2018, speck for Thomas Gleixner wrote:

> That said, I think we should stage the host side mitigations plus the L1
> flush on vmenter ASAP so we are not standing there with our pants down when
> the cat comes out of the bag early. 

Agreed.

> That means HT off, but it's still better than having absolutely nothing.

Will we actually be enforcing switching SMT off (before anything better 
exists) by either offlining all the siblings or forcing them to idle at 
the moment first virtual machine gets started, from the kernel directly?

This seems like this policy would better be enforced by userspace 
(libvirt?), but kernel should probably at least warn on affected CPUs if 
it detects this is being violated.

Thanks,

-- 
Jiri Kosina
SUSE Labs

[MODERATED] Re: L1D-Fault KVM mitigation

[Dave Hansen]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

On 05/24/2018 08:38 AM, speck for Jiri Kosina wrote:
>> That means HT off, but it's still better than having absolutely nothing.
> Will we actually be enforcing switching SMT off (before anything better 
> exists) by either offlining all the siblings or forcing them to idle at 
> the moment first virtual machine gets started, from the kernel directly?
> 
> This seems like this policy would better be enforced by userspace 
> (libvirt?), but kernel should probably at least warn on affected CPUs if 
> it detects this is being violated.

The most straightforward thing is to do trigger the same behavior as
"noht" as part of our arch/x86/kernel/cpu/bugs.c code whenever KVM is
compile-time enabled.

I think we have to do that by default, but allow folks to override it if
they want, like if they know KVM will never get used.

[MODERATED] Re: L1D-Fault KVM mitigation

[Linus Torvalds]

On Thu, 24 May 2018, speck for Dave Hansen wrote:
> 
> The most straightforward thing is to do trigger the same behavior as
> "noht" as part of our arch/x86/kernel/cpu/bugs.c code whenever KVM is
> compile-time enabled.

Absolutely not.

The whole HT issue is only for people who run untrusted loads. 

Lots of people use kvm for testing, but also for just running qemu etc. 
They have absolutely zero reason to disable HT in general.

The only people who worry are Amazon etc.

We should document the hell out of it, and happily the tech press will be 
very happy to inform everybody about the issue. But most people won't care 
one whit, and will run only "good" loads (ie operating systems that have 
been fixed) in their vm's and not random garbage.

Don't try to make this HT workaround be anything but what it is: an issue 
for the Amazons and googles of the world.

               Linus

[MODERATED] Encrypted Message

[Tim Chen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/rfc822-headers; protected-headers="v1", Size: 134 bytes --]

From: Tim Chen <tim.c.chen@linux.intel.com>
To: speck for Thomas Gleixner <speck@linutronix.de>
Subject: Re: L1D-Fault KVM mitigation

[-- Attachment #2: Type: text/plain, Size: 3619 bytes --]

On 05/24/2018 08:33 AM, speck for Thomas Gleixner wrote:
> On Thu, 24 May 2018, speck for Thomas Gleixner wrote:
>> On Thu, 24 May 2018, speck for Peter Zijlstra wrote:
>>> On Wed, May 23, 2018 at 10:45:45AM +0100, speck for David Woodhouse wrote:
>>>> The microcode trick just makes it a lot easier because we don't
>>>> have to *explicitly* pause the sibling vCPUs and manage their state on
>>>> every vmexit/entry. And avoids potential race conditions with managing
>>>> that in software.
>>>
>>> Yes, it would certainly help and avoid a fair bit of ugly. It would, for
>>> instance, avoid having to modify irq_enter() / irq_exit(), which would
>>> otherwise be required (and possibly leak all data touched up until that
>>> point is reached).
>>>
>>> But even with all that, adding L1-flush to every VMENTER will hurt lots.
>>> Consider for example the PIO emulation used when booting a guest from a
>>> disk image. That causes VMEXIT/VMENTER at stupendous rates.
>>
>> Just did a test on SKL Client where I have ucode. It does not have HT so
>> its not suffering from any HT side effects when L1D is flushed.
>>
>> Boot time from a disk image is ~1s measured from the first vcpu enter.
>>
>> With L1D Flush on vmenter the boot time is about 5-10% slower. And that has
>> lots of PIO operations in the early boot.
>>
>> For a kernel build the L1D Flush has an overhead of < 1%.
>>
>> Netperf guest to host has a slight drop of the throughput in the 2%
>> range. Host to guest surprisingly goes up by ~3%. Fun stuff!
>>
>> Now I isolated two host CPUs and pinned the two vCPUs on it to be able to
>> measure the overhead. Running cyclictest with a period of 25us in the guest
>> on a isolated guest CPU and monitoring the behaviour with perf on the host
>> for the corresponding host CPU gives
>>
>> No Flush	      	       Flush
>>
>> 1.31 insn per cycle	       1.14 insn per cycle
>>
>> 2e6 L1-dcache-load-misses/sec  26e6 L1-dcache-load-misses/sec
>>
>> In that simple test the L1D misses go up by a factor of 13.
>>
>> Now with the whole gang scheduling the numbers I heard through the
>> grapevine are in the range of factor 130, i.e. 13k% for a simple boot from
>> disk image. 13 minutes instead of 6 seconds...

The performance is highly dependent on how often we VM exit.
Working with Peter Z on his prototype, the performance ranges from
no regression for a network loop back, ~20% regression for kernel compile
to ~100% regression on File IO.  PIO brings out the worse aspect
of the synchronization overhead as we VM exit on every dword PIO read in, and the
kernel and initrd image was about 50 MB for the experiment, and led to
13 min of load time.

We may need to do the co-scheduling only when VM exit rate is low, and
turn off the SMT when VM exit rate becomes too high.

(Note: I haven't added in the L1 flush on VM entry for my experiment, that is on
the todo).

Tim


>>
>> That's not surprising at all, though the magnitude is way higher than I
>> expected. I don't see a realistic chance for vmexit heavy workloads to work
>> with that synchronization thing at all, whether it's ucode assisted or not.
> 
> That said, I think we should stage the host side mitigations plus the L1
> flush on vmenter ASAP so we are not standing there with our pants down when
> the cat comes out of the bag early. That means HT off, but it's still
> better than having absolutely nothing.
> 
> The gang scheduling nonsense can be added on top when it should
> surprisingly turn out to be usable at all.
> 
> Thanks,
> 
> 	tglx
> 

[MODERATED] Re: L1D-Fault KVM mitigation

[Linus Torvalds]

On Thu, 24 May 2018, speck for Tim Chen wrote:
> 
> We may need to do the co-scheduling only when VM exit rate is low, and
> turn off the SMT when VM exit rate becomes too high.

I don't think there's any way to actually turn off HT dynamically, is 
there?

Yes, you can park the sibling in idle, but afaik, some core resources will 
still be statically partitioned, so it's different from the "turn off HT 
at boot" case.

But if there is actually a way to turn HT off dynamically, maybe we could 
make it be CPU hotplug. That would certainly be _so_ much better than the 
nasty "turn it on/off in BIOS" even for other uses.

               Linus

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

On Thu, 24 May 2018, speck for Linus Torvalds wrote:
> On Thu, 24 May 2018, speck for Tim Chen wrote:
> > 
> > We may need to do the co-scheduling only when VM exit rate is low, and
> > turn off the SMT when VM exit rate becomes too high.
> 
> I don't think there's any way to actually turn off HT dynamically, is 
> there?
> 
> Yes, you can park the sibling in idle, but afaik, some core resources will 
> still be statically partitioned, so it's different from the "turn off HT 
> at boot" case.
> 
> But if there is actually a way to turn HT off dynamically, maybe we could 
> make it be CPU hotplug. That would certainly be _so_ much better than the 
> nasty "turn it on/off in BIOS" even for other uses.

CPU hotplug is pretty much the only solution for runtime HT disable. In the
current state it won't give the boot time allocated per cpu resources back
and nr_present_cpus() will still be the same as before shutting them down,
but everything else will be cleaned out. I need to check the scheduler
topology stuff, but that should see the change as well. We could optimize
that with the static key which is already available (sched_smt_present),
but I have to look which nastiness is involved when toggling that at
runtime.

So yes, it's different from the case where HT is disabled in the BIOS, but
I don't think it really matters much in practice.

I'll implement a knob in sysfs for that and see how that behaves.

Thanks,

	tglx

[MODERATED] Re: L1D-Fault KVM mitigation

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1849 bytes --]

On 25/05/2018 10:31, speck for Thomas Gleixner wrote:
> On Thu, 24 May 2018, speck for Linus Torvalds wrote:
>> On Thu, 24 May 2018, speck for Tim Chen wrote:
>>>
>>> We may need to do the co-scheduling only when VM exit rate is low, and
>>> turn off the SMT when VM exit rate becomes too high.
>>
>> I don't think there's any way to actually turn off HT dynamically, is 
>> there?
>>
>> Yes, you can park the sibling in idle, but afaik, some core resources will 
>> still be statically partitioned, so it's different from the "turn off HT 
>> at boot" case.
>>
>> But if there is actually a way to turn HT off dynamically, maybe we could 
>> make it be CPU hotplug. That would certainly be _so_ much better than the 
>> nasty "turn it on/off in BIOS" even for other uses.
> 
> CPU hotplug is pretty much the only solution for runtime HT disable. In the
> current state it won't give the boot time allocated per cpu resources back
> and nr_present_cpus() will still be the same as before shutting them down,
> but everything else will be cleaned out. I need to check the scheduler
> topology stuff, but that should see the change as well. We could optimize
> that with the static key which is already available (sched_smt_present),
> but I have to look which nastiness is involved when toggling that at
> runtime.
> 
> So yes, it's different from the case where HT is disabled in the BIOS, but
> I don't think it really matters much in practice.
> 
> I'll implement a knob in sysfs for that and see how that behaves.

/me is still alive though slightly jet-lagged

That was also Ingo's suggestion when I talked to him (about a month back).

Also, PIO workloads are not necessarily _that_ bad because of course KVM
will do only one vmexit per "rep ins".  But some bootloaders are better
than others.

Paolo

[MODERATED] Encrypted Message

[Tim Chen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/rfc822-headers; protected-headers="v1", Size: 127 bytes --]

From: Tim Chen <tim.c.chen@linux.intel.com>
To: speck for Tim Chen <speck@linutronix.de>
Subject: Re: L1D-Fault KVM mitigation

[-- Attachment #2: Type: text/plain, Size: 3260 bytes --]

On 05/24/2018 04:18 PM, speck for Tim Chen wrote:
> On 05/24/2018 08:33 AM, speck for Thomas Gleixner wrote:
>> On Thu, 24 May 2018, speck for Thomas Gleixner wrote:
>>> On Thu, 24 May 2018, speck for Peter Zijlstra wrote:
>>>> On Wed, May 23, 2018 at 10:45:45AM +0100, speck for David Woodhouse wrote:
>>>>> The microcode trick just makes it a lot easier because we don't
>>>>> have to *explicitly* pause the sibling vCPUs and manage their state on
>>>>> every vmexit/entry. And avoids potential race conditions with managing
>>>>> that in software.
>>>>
>>>> Yes, it would certainly help and avoid a fair bit of ugly. It would, for
>>>> instance, avoid having to modify irq_enter() / irq_exit(), which would
>>>> otherwise be required (and possibly leak all data touched up until that
>>>> point is reached).
>>>>
>>>> But even with all that, adding L1-flush to every VMENTER will hurt lots.
>>>> Consider for example the PIO emulation used when booting a guest from a
>>>> disk image. That causes VMEXIT/VMENTER at stupendous rates.
>>>
>>> Just did a test on SKL Client where I have ucode. It does not have HT so
>>> its not suffering from any HT side effects when L1D is flushed.
>>>
>>> Boot time from a disk image is ~1s measured from the first vcpu enter.
>>>
>>> With L1D Flush on vmenter the boot time is about 5-10% slower. And that has
>>> lots of PIO operations in the early boot.
>>>
>>> For a kernel build the L1D Flush has an overhead of < 1%.
>>>
>>> Netperf guest to host has a slight drop of the throughput in the 2%
>>> range. Host to guest surprisingly goes up by ~3%. Fun stuff!
>>>
>>> Now I isolated two host CPUs and pinned the two vCPUs on it to be able to
>>> measure the overhead. Running cyclictest with a period of 25us in the guest
>>> on a isolated guest CPU and monitoring the behaviour with perf on the host
>>> for the corresponding host CPU gives
>>>
>>> No Flush	      	       Flush
>>>
>>> 1.31 insn per cycle	       1.14 insn per cycle
>>>
>>> 2e6 L1-dcache-load-misses/sec  26e6 L1-dcache-load-misses/sec
>>>
>>> In that simple test the L1D misses go up by a factor of 13.
>>>
>>> Now with the whole gang scheduling the numbers I heard through the
>>> grapevine are in the range of factor 130, i.e. 13k% for a simple boot from
>>> disk image. 13 minutes instead of 6 seconds...
> 
> The performance is highly dependent on how often we VM exit.
> Working with Peter Z on his prototype, the performance ranges from
> no regression for a network loop back, ~20% regression for kernel compile
> to ~100% regression on File IO.  PIO brings out the worse aspect
> of the synchronization overhead as we VM exit on every dword PIO read in, and the
> kernel and initrd image was about 50 MB for the experiment, and led to
> 13 min of load time.
> 
> We may need to do the co-scheduling only when VM exit rate is low, and
> turn off the SMT when VM exit rate becomes too high.
> 
> (Note: I haven't added in the L1 flush on VM entry for my experiment, that is on
> the todo).

As a post note, I added in the L1 flush and the performance numbers
pretty much stay the same.  So the synchronization overhead is
dominant and L1 flush overhead is secondary.

Tim

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

On Thu, 24 May 2018, speck for Tim Chen wrote:

>> Now with the whole gang scheduling the numbers I heard through the
>> grapevine are in the range of factor 130, i.e. 13k% for a simple boot from
>> disk image. 13 minutes instead of 6 seconds...

> The performance is highly dependent on how often we VM exit.

That's pretty obvious.

> Working with Peter Z on his prototype, the performance ranges from
> no regression for a network loop back, ~20% regression for kernel compile
> to ~100% regression on File IO.

These numbers are not that interesting when you do not provide comparisons
vs. single threaded. See below.

> PIO brings out the worse aspect of the synchronization overhead as we VM
> exit on every dword PIO read in, and the kernel and initrd image was
> about 50 MB for the experiment, and led to 13 min of load time.
>
> We may need to do the co-scheduling only when VM exit rate is low, and
> turn off the SMT when VM exit rate becomes too high.

You cannot do that during runtime. That will destroy placement schemes and
whatever. The SMT off decision needs to be done at a quiescent moment,
i.e. before starting VMs.

The PIO case _IS_ interesting because it highlights the problem with the
synchronization overhead. And it does not matter at all whether you VMEXIT
because of a PIO access or due to any other reason. So even if you optimize
it then you still have a gazillion of vm_exits on boot. The simple boot
tests I did have ~250k vm_exits in 5 seconds and only half of them are PIO.

Removing the PIO access makes the boot faster because you avoid 50% of the
vmexits, but the rest of the vmexits will still get a massive overhead,
unless you have a scenario where two vCPUs of a guest are runnable and
ready to enter at the same time and vmexit at the same time. Any other
scenario will lose due to the busy waiting synchronization overhead. Just
look at traces and do the math.

I did the following test:

 - Two CPUs (siblings) on the host (HSW-EX) fully isolated

 - One guest with two vCPUs affine to the isolated host CPUs. idle=poll on
   the guest command line to avoid the single vCPU case.

 - No L1 Flush

 - Running a kernel compile on the guest in the regular virtio disk backed
   filesystem. Modified the build skript to stop before the final linkage
   because that is single threaded.

Time:  88 seconds

vmexits:   vCPU0         86.218
           vCPU1         85.703
           total        171.921

That's about 2 vmexits per ms.

Running the same compile single threaded (offlining vCPU1 in the guest)
increases the time to 107 seconds.

    107 / 88  = 1.22

I.e. it's 20% slower than the one using two threads. That means that it is
the same slowdown as having two threads synchronized (your number).

So if I take the above example and assume that the overhead of
synchronization is ~20% then the average vmenter/vmexit time is close to
50us.

Next I did an experiment with synchronizing the vmenter/vmexit. It's
probably more stupid than what you have as the overhead I observe is way
higher, but then I don't know how and what you tested exactly, so it's hard
to compare.

Nevertheless it gave me very interesting insights via tracing the
synchronization mechanics. The interesting thing is that halfways
synchronous vmexits on both vCPUs are rather cheap. The slightly async ones
make the big difference and at some points in the trace the stuff starts to
ping pong in and out of guest mode without really making progress for a
while. So there is not only the overhead itself, it's timing dependend
overhead which can accumulate rather fast. And there is absolutely nothing
you can do about that.

So I can see the usefulness for scenarious which David Woodhouse described
where vCPU and host CPU have a fixed relationship and the guests exit once
in a while. But that should really be done with ucode assisantance which
avoids all the nasty synchronization hackery more or less completely.

But if anyone believes that the gang scheduling scheme with full software
synchronization can be applied to random usecases, then he's probably
working for the marketing department and authoring the L1 terminal fuckup
press release and whitepaper.

I'm surely open for a suprising clever trick which makes this all work, but
I certainly won't hold by breath.

Thanks,

        tglx

[MODERATED] Re: L1D-Fault KVM mitigation

[Andi Kleen]

> The PIO case _IS_ interesting because it highlights the problem with the
> synchronization overhead. And it does not matter at all whether you VMEXIT
> because of a PIO access or due to any other reason. So even if you optimize
> it then you still have a gazillion of vm_exits on boot. The simple boot
> tests I did have ~250k vm_exits in 5 seconds and only half of them are PIO.

Keep in mind that we don't need to synchronize when the other CPU is idle
in the guest, so it's only a problem when all the CPUs are busy.

That should be the common case for boot.

Right now something doesn't seem to be working right with this, so 
the PIO overhead is still high. 

> Nevertheless it gave me very interesting insights via tracing the
> synchronization mechanics. The interesting thing is that halfways
> synchronous vmexits on both vCPUs are rather cheap. The slightly async ones

What's an async vmexit? One that blocks?

I didn't think we had that many of those. What exactly you are seeing?

-Andi

[MODERATED] Re: L1D-Fault KVM mitigation

[Linus Torvalds]

On Sat, 26 May 2018, speck for Andi Kleen wrote:
> 
> Keep in mind that we don't need to synchronize when the other CPU is idle
> in the guest, so it's only a problem when all the CPUs are busy.

What? No.

Maybe you mean "idle in the _host_"?

Which is not necessarily at all the same thing. 

> That should be the common case for boot.

Again, no. The guest booting has absolutely nothing to do with the host 
being idle or not.

If the argument is that "most of the time you have idle host CPU 
resources", then the solution to that is simple: "turn off HT".

                    Linus

[MODERATED] Re: L1D-Fault KVM mitigation

[Andi Kleen]

On Sat, May 26, 2018 at 01:48:01PM -0700, speck for Linus Torvalds wrote:
> 
> 
> On Sat, 26 May 2018, speck for Andi Kleen wrote:
> > 
> > Keep in mind that we don't need to synchronize when the other CPU is idle
> > in the guest, so it's only a problem when all the CPUs are busy.
> 
> What? No.
> 
> Maybe you mean "idle in the _host_"?

It's both actually. Either idle in the host, or idle in the guest.

When it's idle in the host we don't need to synchronize, unless 
there is an interrupt (which does its own synchronization) because
the idle loop has nothing valuable to leak.

And if it's idle in the guest then the vcpu is blocked and
also obviously doesn't need to be synchronized.

Both cases can be reasonably common.

-Andi

[MODERATED] Re: L1D-Fault KVM mitigation

[Linus Torvalds]

On Sun, 27 May 2018, speck for Andi Kleen wrote:
> > 
> > Maybe you mean "idle in the _host_"?
> 
> It's both actually. Either idle in the host, or idle in the guest.

It really really isn't.

Andi, if one sibling does a vmexit, and the other sibling is still in vmx 
mode, we have to force-exit the other sibling. It doesn't matter one whit 
whether it's idle or not - because we won't know.

Or is there some magical sideband that I am not aware of?

> When it's idle in the host we don't need to synchronize, unless 
> there is an interrupt (which does its own synchronization) because
> the idle loop has nothing valuable to leak.

Right. The only case that doesn't need synchronization is "other sibling 
is idle in the _host_".

> And if it's idle in the guest then the vcpu is blocked and
> also obviously doesn't need to be synchronized.

If the other sibling is idle inside the guest, then we don't even know 
that.

Of course, if somebody is doing exit-on-{hlt/mwait/pause}, and we end up 
actually being in the host and _emulating_ the idle, then that might be 
acceptable. Except even then we'd be potentially leaking a *lot* of host 
caches. The vmx emulation code ends up having a not all that insignificant 
footprint in that (host percpu state, thread state etc).

But why would you emulate halt/mwait/pause anyway? That sounds insane to 
me. The reason you would want exit-on-halt is so that the host can do 
something else if a vcpu goes idle, not so that it can just stay in some 
emulated idle state.

If you want to go to low-power mode, you'd just let the halt/mwait happen 
inside the guest.

But to be honest, I haven't actually checked what kvm users (or xen, or 
whatever) really do. Am I missing something?

           Linus

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

On Sun, 27 May 2018, speck for Linus Torvalds wrote:
> On Sun, 27 May 2018, speck for Andi Kleen wrote:
> > > 
> > > Maybe you mean "idle in the _host_"?
> > 
> > It's both actually. Either idle in the host, or idle in the guest.
> 
> It really really isn't.
> 
> Andi, if one sibling does a vmexit, and the other sibling is still in vmx 
> mode, we have to force-exit the other sibling. It doesn't matter one whit 
> whether it's idle or not - because we won't know.
> 
> Or is there some magical sideband that I am not aware of?
> 
> > When it's idle in the host we don't need to synchronize, unless 
> > there is an interrupt (which does its own synchronization) because
> > the idle loop has nothing valuable to leak.
> 
> Right. The only case that doesn't need synchronization is "other sibling 
> is idle in the _host_".

It still needs synchronization as you have to enforce that the other
sibling _IS_ in forced idle and not trying to do something else. Unless it
has reached forced idle state you can't vmenter.

Thanks,

	tglx

[MODERATED] Re: L1D-Fault KVM mitigation

[Andrew Cooper]

[-- Attachment #1: Type: text/plain, Size: 2256 bytes --]

On 27/05/2018 19:49, speck for Linus Torvalds wrote:
>
> On Sun, 27 May 2018, speck for Andi Kleen wrote:
>>> Maybe you mean "idle in the _host_"?
>> It's both actually. Either idle in the host, or idle in the guest.
> It really really isn't.
>
> Andi, if one sibling does a vmexit, and the other sibling is still in vmx 
> mode, we have to force-exit the other sibling. It doesn't matter one whit 
> whether it's idle or not - because we won't know.
>
> Or is there some magical sideband that I am not aware of?
>
>> When it's idle in the host we don't need to synchronize, unless 
>> there is an interrupt (which does its own synchronization) because
>> the idle loop has nothing valuable to leak.
> Right. The only case that doesn't need synchronization is "other sibling 
> is idle in the _host_".
>
>> And if it's idle in the guest then the vcpu is blocked and
>> also obviously doesn't need to be synchronized.
> If the other sibling is idle inside the guest, then we don't even know 
> that.
>
> Of course, if somebody is doing exit-on-{hlt/mwait/pause}, and we end up 
> actually being in the host and _emulating_ the idle, then that might be 
> acceptable. Except even then we'd be potentially leaking a *lot* of host 
> caches. The vmx emulation code ends up having a not all that insignificant 
> footprint in that (host percpu state, thread state etc).
>
> But why would you emulate halt/mwait/pause anyway? That sounds insane to 
> me. The reason you would want exit-on-halt is so that the host can do 
> something else if a vcpu goes idle, not so that it can just stay in some 
> emulated idle state.
>
> If you want to go to low-power mode, you'd just let the halt/mwait happen 
> inside the guest.
>
> But to be honest, I haven't actually checked what kvm users (or xen, or 
> whatever) really do. Am I missing something?

Xen doesn't ever vcpus enter idle themselves.  We trap HLT/etc and will
either schedule another VM, or choose to idle the host if there really
is nothing else to do.

I've never come across a plausible usecase for letting non-root mode
idle the cores into a low power state.  They simply aren't in a position
to know whether other work needs doing or not.

~Andrew

[MODERATED] Re: L1D-Fault KVM mitigation

[Linus Torvalds]

On Sun, 27 May 2018, speck for Andrew Cooper wrote:
> 
> Xen doesn't ever vcpus enter idle themselves.  We trap HLT/etc and will
> either schedule another VM, or choose to idle the host if there really
> is nothing else to do.

Right. But then there is never any such thing as "guest idle". There is 
only "host idle" or "host is doing something else entirely".

You _could_ have a "idle polling" mode which is separate from the regular 
host idle loop, I guess.

> I've never come across a plausible usecase for letting non-root mode
> idle the cores into a low power state.  They simply aren't in a position
> to know whether other work needs doing or not.

Afaik, it mainly makes sense when there is no actual host OS at all, just 
the bare-metal hypervisor used for partitioning resources, not scheduling 
them.

                 Linus

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Sun, 27 May 2018, speck for Linus Torvalds wrote:
> On Sun, 27 May 2018, speck for Andrew Cooper wrote:
> > 
> > Xen doesn't ever vcpus enter idle themselves.  We trap HLT/etc and will
> > either schedule another VM, or choose to idle the host if there really
> > is nothing else to do.
> 
> Right. But then there is never any such thing as "guest idle". There is 
> only "host idle" or "host is doing something else entirely".
> 
> You _could_ have a "idle polling" mode which is separate from the regular 
> host idle loop, I guess.
> 
> > I've never come across a plausible usecase for letting non-root mode
> > idle the cores into a low power state.  They simply aren't in a position
> > to know whether other work needs doing or not.
> 
> Afaik, it mainly makes sense when there is no actual host OS at all, just 
> the bare-metal hypervisor used for partitioning resources, not scheduling 
> them.

Right. That's what the Jailhouse hypervisor does. It's a zero vmexit setup.

Thanks,

	tglx

[MODERATED] Re: L1D-Fault KVM mitigation

[Andrew Cooper]

[-- Attachment #1: Type: text/plain, Size: 1886 bytes --]

On 27/05/2018 20:41, speck for Thomas Gleixner wrote:
> On Sun, 27 May 2018, speck for Linus Torvalds wrote:
>> On Sun, 27 May 2018, speck for Andrew Cooper wrote:
>>> Xen doesn't ever vcpus enter idle themselves.  We trap HLT/etc and will
>>> either schedule another VM, or choose to idle the host if there really
>>> is nothing else to do.
>> Right. But then there is never any such thing as "guest idle". There is 
>> only "host idle" or "host is doing something else entirely".
>>
>> You _could_ have a "idle polling" mode which is separate from the regular 
>> host idle loop, I guess.
>>
>>> I've never come across a plausible usecase for letting non-root mode
>>> idle the cores into a low power state.  They simply aren't in a position
>>> to know whether other work needs doing or not.
>> Afaik, it mainly makes sense when there is no actual host OS at all, just 
>> the bare-metal hypervisor used for partitioning resources, not scheduling 
>> them.
> Right. That's what the Jailhouse hypervisor does. It's a zero vmexit setup.

Jailhouse also has static assignment of resources, which means they can
arrange never to have two different VMs on the same sibling hyperthreads.

They still need to disable hyperthreads or find a working
synchronisation algorithm for entry/exit, but they don't have the added
gang scheduling problem of ensuring that two hyperthreads are always
occupied by vcpus of the same VM.

FWIW, my gut feeling at the moment is that the overhead of
synchronisation will outweigh disabling hyperthreading, but I'd like to
be proved wrong.  Others in the Xen community are looking to extend
shadow paging to be as performant as EPT is currently (because at that
point, the hypervisor control every PTE accessible to the pagewalk), and
again, I'd like to see this succeed, but my gut feeling is that it wont.

~Andrew

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 1804 bytes --]

On Sun, 27 May 2018, speck for Andrew Cooper wrote:
> On 27/05/2018 20:41, speck for Thomas Gleixner wrote:
> > On Sun, 27 May 2018, speck for Linus Torvalds wrote:
> >> Afaik, it mainly makes sense when there is no actual host OS at all, just 
> >> the bare-metal hypervisor used for partitioning resources, not scheduling 
> >> them.
> > Right. That's what the Jailhouse hypervisor does. It's a zero vmexit setup.
> 
> Jailhouse also has static assignment of resources, which means they can
> arrange never to have two different VMs on the same sibling hyperthreads.

Right.

> They still need to disable hyperthreads or find a working
> synchronisation algorithm for entry/exit, but they don't have the added
> gang scheduling problem of ensuring that two hyperthreads are always
> occupied by vcpus of the same VM.

If Jailhouse exits, then there is something badly wrong and the guest will
usually be terminated. There are a few odd cases where an exit actually is
non fatal, but that should be fixable.

> FWIW, my gut feeling at the moment is that the overhead of
> synchronisation will outweigh disabling hyperthreading, but I'd like to
> be proved wrong.  Others in the Xen community are looking to extend
> shadow paging to be as performant as EPT is currently (because at that
> point, the hypervisor control every PTE accessible to the pagewalk), and
> again, I'd like to see this succeed, but my gut feeling is that it wont.

It might be a viable solution for some of the common scenarios like mass
hosting which tends to have a lot of single vcpu guests; there the overhead
of shadow page tables might be less than the overhead of forcing siblings
into idle and putting restrictions on load balancing etc. At least worth to
investigate.

Thanks,

	tglx

[MODERATED] Re: L1D-Fault KVM mitigation

[Andrew Cooper]

[-- Attachment #1: Type: text/plain, Size: 1779 bytes --]

On 28/05/2018 07:47, speck for Thomas Gleixner wrote:
> On Sun, 27 May 2018, speck for Andrew Cooper wrote:
>> On 27/05/2018 20:41, speck for Thomas Gleixner wrote:
>>
>> FWIW, my gut feeling at the moment is that the overhead of
>> synchronisation will outweigh disabling hyperthreading, but I'd like to
>> be proved wrong.  Others in the Xen community are looking to extend
>> shadow paging to be as performant as EPT is currently (because at that
>> point, the hypervisor control every PTE accessible to the pagewalk), and
>> again, I'd like to see this succeed, but my gut feeling is that it wont.
> It might be a viable solution for some of the common scenarios like mass
> hosting which tends to have a lot of single vcpu guests; there the overhead
> of shadow page tables might be less than the overhead of forcing siblings
> into idle and putting restrictions on load balancing etc. At least worth to
> investigate.

Sadly, KPTI has taken what as a manageable performance difference
between shadow and EPT, and wrecked it.  A number of common tasks are
between 6 and 16 times slower than EPT, due to all the CR3 vmexits.

The CR3-target feature is attractive because it does let writes to CR3
happen, and can even filter on the NOFLUSH bit being set.  The problem
is the lack of a GPA => HPA translation, so a naive hypervisor which
tries to use this has its guests wandering off their shadows, and
everything explodes.

As of this morning, Juergen and I are experimenting with the hypervisor
providing the translation table to the guest, so it can write the
correctly-translated CR3 and avoid the vmexit.  I've also asked whether
this would be feasible to do in microcode, to avoid having to make any
guest modifications.

~Andrew

[MODERATED] Re: L1D-Fault KVM mitigation

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1379 bytes --]

On 27/05/2018 20:49, speck for Linus Torvalds wrote:
> But why would you emulate halt/mwait/pause anyway? That sounds insane to 
> me. The reason you would want exit-on-halt is so that the host can do 
> something else if a vcpu goes idle, not so that it can just stay in some 
> emulated idle state.

When hlt/mwait is emulated, the thread goes to sleep.  When pause is
emulated, the thread checks if there is another CPU to yield to, but
otherwise stays running.

KVM recently grew a new mode where hlt/mwait/pause is passed directly to
the guest.  It was (partly) contributed by Amazon because that's what
they're doing in their KVM-based cloud stuff - it does all I/O in custom
hardware so all interrupts will be VT-d posted interrupts and avoid the
overhead of the vCPU thread going to sleep and back running.

There is a problem though.  The only way to know if the guest is in
hlt/mwait/pause, is to cause a vmexit, e.g. with an IPI, and read the VM
control state (which you can only do from the CPU that was running it!).
 So even for the "idle in the guest" case you pretty much have to do
synchronization.

Paolo

> If you want to go to low-power mode, you'd just let the halt/mwait happen 
> inside the guest.
> 
> But to be honest, I haven't actually checked what kvm users (or xen, or 
> whatever) really do. Am I missing something?

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

On Mon, 28 May 2018, speck for Paolo Bonzini wrote:
> On 27/05/2018 20:49, speck for Linus Torvalds wrote:
> > But why would you emulate halt/mwait/pause anyway? That sounds insane to 
> > me. The reason you would want exit-on-halt is so that the host can do 
> > something else if a vcpu goes idle, not so that it can just stay in some 
> > emulated idle state.
> 
> When hlt/mwait is emulated, the thread goes to sleep.  When pause is
> emulated, the thread checks if there is another CPU to yield to, but
> otherwise stays running.
> 
> KVM recently grew a new mode where hlt/mwait/pause is passed directly to
> the guest.  It was (partly) contributed by Amazon because that's what
> they're doing in their KVM-based cloud stuff - it does all I/O in custom
> hardware so all interrupts will be VT-d posted interrupts and avoid the
> overhead of the vCPU thread going to sleep and back running.
> 
> There is a problem though.  The only way to know if the guest is in
> hlt/mwait/pause, is to cause a vmexit, e.g. with an IPI, and read the VM
> control state (which you can only do from the CPU that was running it!).
>  So even for the "idle in the guest" case you pretty much have to do
> synchronization.

Right, but there ucode assist would be handy. If the sibling is in
hlt/mwait then the other could do a vmexit undisturbed and when the
mwait/hlt resumes then it either forces a vmexit if the other sibling is
still in the host or it lets it continue when the other sibling reentered
guest mode again.

Thanks,

	tglx

[MODERATED] Re: L1D-Fault KVM mitigation

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

On 28/05/2018 17:56, speck for Thomas Gleixner wrote:
>> There is a problem though.  The only way to know if the guest is in
>> hlt/mwait/pause, is to cause a vmexit, e.g. with an IPI, and read the VM
>> control state (which you can only do from the CPU that was running it!).
>>  So even for the "idle in the guest" case you pretty much have to do
>> synchronization.
> Right, but there ucode assist would be handy. If the sibling is in
> hlt/mwait then the other could do a vmexit undisturbed and when the
> mwait/hlt resumes then it either forces a vmexit if the other sibling is
> still in the host or it lets it continue when the other sibling reentered
> guest mode again.

Yes, but I am not sure how microcode can do it since it could
potentially block a sibling forever.

Perhaps some kind of "sibling not in VMX non-root mode" vmexit, to
automate the IPI part of the synchronization, would be useful.  When the
hypervisor receives it, it can decide whether to poll for completion of
the sibling's vmexit, or to go to sleep.

Paolo

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

On Sat, 26 May 2018, speck for Andi Kleen wrote:
> > The PIO case _IS_ interesting because it highlights the problem with the
> > synchronization overhead. And it does not matter at all whether you VMEXIT
> > because of a PIO access or due to any other reason. So even if you optimize
> > it then you still have a gazillion of vm_exits on boot. The simple boot
> > tests I did have ~250k vm_exits in 5 seconds and only half of them are PIO.
> 
> Keep in mind that we don't need to synchronize when the other CPU is idle
> in the guest, so it's only a problem when all the CPUs are busy.

No. It does not matter at all what a guest CPU does. The allowed states are:

	CPU0			CPU1

	In host			In host

	In guest		In host forced idle

	In host forced idle	In guest

	In guest       		In guest

Whatever the guest mode does is irrelevant.

> That should be the common case for boot.

> > Nevertheless it gave me very interesting insights via tracing the
> > synchronization mechanics. The interesting thing is that halfways
> > synchronous vmexits on both vCPUs are rather cheap. The slightly async ones
> 
> What's an async vmexit? One that blocks?

No, I'm talking about timing. Sorry I should have said simultaneous. Let me
rephrase.

If the vmexits of both guest CPUs happen almost at the same time,
i.e. simultaneous then the overhead is pretty small. That's the case for
the tick. But that's pretty much the only event which has that property.

All other vmexits I see are singular events of one guest. There you have a
choice of busy waiting for the other vCPU to vmexit as well or force it out
via an IPI. The method I use is IPI as busy waiting would be horribly slow
for obvious reasons.

Now there are situations which show the following behaviour:

CPU0	  vmexit
	  IPI CPU1
CPU0	  sync_exit
CPU1	  vmexit
CPU1	  sync_exit
CPU1	  sync_enter

CPU0	  do_stuff
CPU0	  sync_enter

CPU0	  vmenter
CPU1	  vmenter

CPU0	  vmexit	immediate after vmenter	
	  IPI CPU1
....

and this ping pong goes on 10 times in a row taking 2+ milliseconds and the
progress made is the guest is minimal. The reason for this are interrupts
targeted to one of the vCPUs or operations in one of the guest threads
which cause several exits in a row. And there is nothing you can do about
that. It's completely workload dependent.

So unless you have a fully controlled scenario where the guests almost
never exit, the whole synchronization stuff is doomed. But fully controlled
means a 1:1 relationship of physical and virtual CPUs like David mentioned.
Yes, that stuff can benefit, but then we rather want ucode assistance than
the whole wait/IPI dance in software.

Thanks,

	tglx

[MODERATED] Re: L1D-Fault KVM mitigation

[Linus Torvalds]

On Sun, 27 May 2018, speck for Thomas Gleixner wrote:
> 
> No. It does not matter at all what a guest CPU does. The allowed states are:

It's even tighter than that.

> 	CPU0			CPU1
> 
> 	In host			In host
> 
> 	In guest		In host forced idle
> 
> 	In host forced idle	In guest
> 
> 	In guest       		In guest

That last case is "both in SAME guest".

You mustn't have two different VM's running on two sibling CPU's, or one 
might siphon data from the other.

                 Linus

[MODERATED] Re: L1D-Fault KVM mitigation

[Andi Kleen]

On Sun, May 27, 2018 at 05:42:46PM +0200, speck for Thomas Gleixner wrote:
> Whatever the guest mode does is irrelevant.

Idle in the guest means already exited to the host because
HLT exit is always enabled.

> 
> and this ping pong goes on 10 times in a row taking 2+ milliseconds and the
> progress made is the guest is minimal. The reason for this are interrupts
> targeted to one of the vCPUs or operations in one of the guest threads
> which cause several exits in a row. And there is nothing you can do about
> that. It's completely workload dependent.

There's a threshold of the exit rate somewhere where HT off is probably 
better, but I don't think we really have characterized well so far where
exactly this threshold is.

My suspicion is that a lot of workloads will be below the threshold.
We'll see what works out.

> 
> So unless you have a fully controlled scenario where the guests almost
> never exit, the whole synchronization stuff is doomed. But fully controlled
> means a 1:1 relationship of physical and virtual CPUs like David mentioned.
> Yes, that stuff can benefit, but then we rather want ucode assistance than
> the whole wait/IPI dance in software.

Right that would be an obvious optimization. However in the traces
I looked so far the IPI was actually not the most expensive part.

I think it's because IPIs between siblings are not very expensive.

-Andi

[MODERATED] Encrypted Message

[Tim Chen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/rfc822-headers; protected-headers="v1", Size: 134 bytes --]

From: Tim Chen <tim.c.chen@linux.intel.com>
To: speck for Thomas Gleixner <speck@linutronix.de>
Subject: Re: L1D-Fault KVM mitigation

[-- Attachment #2: Type: text/plain, Size: 1799 bytes --]

On 05/26/2018 12:14 PM, speck for Thomas Gleixner wrote:
> On Thu, 24 May 2018, speck for Tim Chen wrote:
> 
 of load time.
>>
>> We may need to do the co-scheduling only when VM exit rate is low, and
>> turn off the SMT when VM exit rate becomes too high.
> 
> You cannot do that during runtime. That will destroy placement schemes and
> whatever. The SMT off decision needs to be done at a quiescent moment,
> i.e. before starting VMs.

Taking the SMT offline is a bit much and too big a hammer.  Andi and I thought about
having the scheduler forcing the other thread in idle instead for high
VM exit rate scenario. We don't have
to bother about doing sync with the other idle thread.

But we have issues about fairness, as we will be starving the
other run queue.

> 

> Running the same compile single threaded (offlining vCPU1 in the guest)
> increases the time to 107 seconds.
> 
>     107 / 88  = 1.22
> 
> I.e. it's 20% slower than the one using two threads. That means that it is
> the same slowdown as having two threads synchronized (your number).

yes, with compile workload, the HT speedup was mostly eaten up by
overhead.

> 
> So if I take the above example and assume that the overhead of
> synchronization is ~20% then the average vmenter/vmexit time is close to
> 50us.
> 

> 
> So I can see the usefulness for scenarious which David Woodhouse described
> where vCPU and host CPU have a fixed relationship and the guests exit once
> in a while. But that should really be done with ucode assisantance which
> avoids all the nasty synchronization hackery more or less completely.

The ucode guys are looking into such possibilities.  It is tough as they
have to work within the constraint of limited ucode headroom.

Thanks.

Tim

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

On Tue, 29 May 2018, speck for Tim Chen wrote:
On 05/26/2018 12:14 PM, speck for Thomas Gleixner wrote:
> On Thu, 24 May 2018, speck for Tim Chen wrote:
> 
> > > We may need to do the co-scheduling only when VM exit rate is low, and
> > > turn off the SMT when VM exit rate becomes too high.
> > 
> > You cannot do that during runtime. That will destroy placement schemes and
> > whatever. The SMT off decision needs to be done at a quiescent moment,
> > i.e. before starting VMs.

> Taking the SMT offline is a bit much and too big a hammer.

Sorry, that's bullshit. It massively depends on the workload and the
scenario. I've explained it a gazillion times by now that there are enough
workloads which will massively lose with SMT on and the extra overhead. Its
trivial enough to figure that out without implementing all bells and
whistels.

> Andi and I thought about having the scheduler forcing the other thread in
> idle instead for high VM exit rate scenario. We don't have to bother
> about doing sync with the other idle thread.

You still have to make sure that the other idle thread _IS_ idle. It's not
the full synchronizaiton scheme, but it's extra work in a hotpath when the
guest is exit heavy. And you still have the problem of interrupt and
softirqs being served on the 'idle' sibling. It's not that simple.

> But we have issues about fairness, as we will be starving the
> other run queue.

That's more than obvious. And you will create even worse issues because
workloads which have a placement scheme, i.e. vCPU affinities will have no
chance to migrate to another CPU. Not to talk about wreckaging the load
balancer completely.

> > I.e. it's 20% slower than the one using two threads. That means that it is
> > the same slowdown as having two threads synchronized (your number).

> yes, with compile workload, the HT speedup was mostly eaten up by
> overhead.

So where is the point of the exercise?

You will not find a generic solution for this problem ever simply because
the workloads and guest scenarios are too different. There are clearly
scenarios which can benefit, but at the same time there are scenarios which
will be way worse off than with SMT disabled.

I completely understand that Intel wants to avoid the 'disable SMT'
solution by all means, but this cannot be done with something which is
obvioulsy creating more problems than it solves in the first place.

At some point reality has to kick in and you have to admit that there is no
generic solution and the only solution for a lot of use cases will be to
disable SMT. The solution for special workloads like the fully partitioned
ones David mentioned do not need the extra mess all over the place
especially not when there is ucode assist at least to the point which fits
into the patch space and some of it really should not take a huge amount of
effort, like the forced sibling vmexit to avoid the whole IPI machinery.

Thanks,

	tglx

[MODERATED] Encrypted Message

[Tim Chen]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/rfc822-headers; protected-headers="v1", Size: 134 bytes --]

From: Tim Chen <tim.c.chen@linux.intel.com>
To: speck for Thomas Gleixner <speck@linutronix.de>
Subject: Re: L1D-Fault KVM mitigation

[-- Attachment #2: Type: text/plain, Size: 1626 bytes --]

On 05/29/2018 02:14 PM, speck for Thomas Gleixner wrote:
> 
>> yes, with compile workload, the HT speedup was mostly eaten up by
>> overhead.
> 
> So where is the point of the exercise?
> 
> You will not find a generic solution for this problem ever simply because
> the workloads and guest scenarios are too different. There are clearly
> scenarios which can benefit, but at the same time there are scenarios which
> will be way worse off than with SMT disabled.
> 
> I completely understand that Intel wants to avoid the 'disable SMT'
> solution by all means, but this cannot be done with something which is
> obvioulsy creating more problems than it solves in the first place.
> 
> At some point reality has to kick in and you have to admit that there is no
> generic solution and the only solution for a lot of use cases will be to
> disable SMT. The solution for special workloads like the fully partitioned
> ones David mentioned do not need the extra mess all over the place
> especially not when there is ucode assist at least to the point which fits
> into the patch space and some of it really should not take a huge amount of
> effort, like the forced sibling vmexit to avoid the whole IPI machinery.
> 

Having to sync on VM entry and on VM exit and on interrupt to idle sibling
sucks. Hopefully the ucode guys can come up with something
to provide an option that forces the sibling to vmexit on vmexit,
and on interrupt to idle sibling. This should cut the sync overhead in half.
Then only VM entry needs to be synced should we still want to
do co-scheduling.

Thanks.

Tim

[MODERATED] Re: L1D-Fault KVM mitigation

[Andi Kleen]

> Now with the whole gang scheduling the numbers I heard through the
> grapevine are in the range of factor 130, i.e. 13k% for a simple boot from
> disk image. 13 minutes instead of 6 seconds...

That's unoptimized, and also for the extreme PIO case (which does one
exit for every dword). With some optimizations we hope to do better.

Also the PIO case is really not that interesting, although it would
be nice to get rid of the slowdown so that users don't
have to fix their boot loaders.

But yes PIO will suffer a bit that's unavoidable. As long 
as the slow down is not too bad it should be acceptable. If it's 
a problem they can always use some other mechanism to load
the kernel.

> 
> That's not surprising at all, though the magnitude is way higher than I
> expected. I don't see a realistic chance for vmexit heavy workloads to work
> with that synchronization thing at all, whether it's ucode assisted or not.

Nothing should be anywhere near as VMEXIT intensive as PIO.

The worst realistic one is likely IO intensive with lots of 
small transactions. But even there you are nowhere near
"one exit for every 2 bytes in a long loop" like PIO.

>  
> The only workload types which will ever benefit from that co-scheduling
> stuff are CPU bound workloads which more or less never vmexit. But are
> those workloads really workloads which benefit from HT?

That's a very extreme conclusion which I don't think is backed at all
by your data.

> Compute workloads
> tend to use floating point or vector instructions which are not really HT
> friendly.

There are plenty of compute workloads that benefit from HT: usually
everything that is not completely memory bandwidth dominated per single
thread.

> 
> Can the virt folks who know what runs on their clowdy offerings please shed
> some light on this? Has anyone made a proper analysis of clowd workloads
> and their behaviour on HT and their vmexit rates?

My understanding is that most cloud providers only sell cores, so they
are already good for other guests.

Usually they use some other affinity mechanism to reach a similar
effect.

But we still have to fix the general case e.g. just for "someone
runs a KVM guest on a random system with HT on"

-Andi

[MODERATED] Re: L1D-Fault KVM mitigation

[Linus Torvalds]

On Thu, 24 May 2018, speck for Peter Zijlstra wrote:
> 
> > It does still depend on gang scheduling (or at least forced sibling
> > idle which is a subset of that), or a singleton vCPU might *never* get
> > run. But we were going to have to do something along those lines
> > anyway.
> 
> Linus has opinions on that.. 

Let's call them "beliefs".

I don't believe for a moment that anybody can come up with anything 
remotely reasonable for gang scheduling.

I'm willing to entertain the possibility that some really smart person can 
solve the problem cleanly and without messing up anything else.

I don't think it's remotely _likely_ to happen, but I'm willing to 
consider it within the realm of possibilities.

So right now, I consider the gang scheduling a pipe dream by people who 
underestimate how hard and ugly it would be, and often have political 
reasons why they are pushing the idea (ie they want to claim it's not a 
hardware deficiency, but just a small software problem).

And I do have opinions on those kinds of _people_.

           Linus

[MODERATED] Re: L1D-Fault KVM mitigation

[David Woodhouse]

On Thu, 2018-05-24 at 08:38 -0700, speck for Linus Torvalds wrote:
> 
> On Thu, 24 May 2018, speck for Peter Zijlstra wrote:
> > 
> > 
> > > 
> > > It does still depend on gang scheduling (or at least forced sibling
> > > idle which is a subset of that), or a singleton vCPU might *never* get
> > > run. But we were going to have to do something along those lines
> > > anyway.
> > 
> > Linus has opinions on that.. 
>
> Let's call them "beliefs".
> 
> I don't believe for a moment that anybody can come up with anything 
> remotely reasonable for gang scheduling.
> 
> I'm willing to entertain the possibility that some really smart person can 
> solve the problem cleanly and without messing up anything else.
> 
> I don't think it's remotely _likely_ to happen, but I'm willing to 
> consider it within the realm of possibilities.
> 
> So right now, I consider the gang scheduling a pipe dream by people who 
> underestimate how hard and ugly it would be, and often have political 
> reasons why they are pushing the idea (ie they want to claim it's not a 
> hardware deficiency, but just a small software problem).

Gang scheduling for the general case is probably a pipe dream.

But there are use cases where we were actually pinning sibling vCPUs to
the sibling pCPUs anyway because HT was *already* a nightmare for
information leakage paranoia.

In those cases, "pause the vCPU when the sibling pCPU vmexits to
anything other than idle" is the necessary and mostly sufficient
missing piece. Because we effectively already *have* gang scheduling to
the extent that it matters.

[MODERATED] Re: L1D-Fault KVM mitigation

[Linus Torvalds]

On Thu, 24 May 2018, speck for David Woodhouse wrote:
> 
> In those cases, "pause the vCPU when the sibling pCPU vmexits to
> anything other than idle" is the necessary and mostly sufficient
> missing piece. Because we effectively already *have* gang scheduling to
> the extent that it matters.

I'll believe it when I see the patch and actual numbers on real loads. 

As far as I can tell, if you want to avoid the data leak, you need to

 - on vmenter, the sibling *must* be in idle, or in "waiting to vmenter 
   the same VM". If the sibling is doing anything else, you don't know 
   what the f*ck it's doing, so you need to do an IPI or a sleep to 
   synchronize with it.

   And regardless, you're going to have to do (probably several) atomic 
   locked accesses just to make sure this is race-free. And you have to 
   clear your L1 caches before that "waiting to vmenter" _and_ before the 
   "going to idle" state, becasue otherwise there's a big race the other 
   way waiting to happen (which is trivial for the guest to hit, since the 
   guest trivially controls vmenter/vmexit by choice of instructions).

 - on exit from idle, you need to IPI the other CPU if it's in VM mode, 
   since you're now starting to touch L1 cache lines. I guess you can do a 
   synchronous wait, depending on why you're exiting idle, but probably 
   not without a big hit on latencies. You're basically primarily in an 
   interrupt handler, for chrissake!

 - on vmexit, you know that the sibling CPU can't have been doing anything 
   else (unless the exit was becuise the sibling IPI'd you to _get_ you 
   out of vmx). But if it was in vmx mode, since you're now going to 
   pollute the L1 caches, you need to either wait for it synchronously to 
   exit its VM, or you need to IPI it to force it to exit. Presumably, you 
   want to wait for it most of the time, but if you exited due to an 
   interrupt or something, that may not be an option, so maybe the IPI is 
   the common case, the same as exit from idle above?

Am I missing something fundamental? Because those three are just the basic 
requirements that have nothing to do with *scheduling*. They are 
fundamental to the whole "I must not touch any interesting data while 
another sibling is in VM mode".

Honestly, I don't care one whit what the actual VM entry/exit code does, 
since it doesn't hurt anything else, and the sane workaround remains to 
just disable HT. But I care deeply about the enter/exit from idle, 
especially if it now involves locked operations and L1 cache flushing, 
just in case.

But note how none of the above has anything to do with the *scheduler*. 
The gang scheduling is just to make the above fuck-up perform much better.

And the above all happens when we expect real things to be going on in the 
system, ie the host actually has timers and networking going on, and the 
actual _paying_ primary clients on something like Amazon cloud don't tend 
to be so much about CPU in the first place (those will be using the lowest 
cost bulk tiers), so they likely have a fair amount of real code going 
on too.

So absolutely none of this is *rare*. Interrupts, vmenter, and vmexit 
happen all the time, and whoever came up with the gang scheduling scheme 
was presumably looking a lot at the best-case numbers.

Now you want to add gang schedulign on *top* of the above disaster? 

Yeah, good luck. 

Maybe I've explained my skepticism sufficiently above. I don't have any 
scheduling frequency numbers and average times in VMX, and presumably you 
do, and maybe I'm being just unreasonably pessimistic. But it all really 
sounds like a complete pipe dream to me.  My guess is that the cost of all 
of the above is going to be *much* bigger than the advantage of HT.

Last I saw, HT on Intel processors was on the order of 30% win. Are you 
seriously claiming that the above costs are going to be so small that you 
make up for it? Because that's a hard limit. If it's more expensive than 
30%, it makes no sense to do. 

That's different from all the other mitigation we've done. It has to be 
cheap, and it has to be cheap for the *bad* case - the case where you 
actually have lots of VMX activity.

The only case I see making up for it is when vm entry/exit are so rare 
(because you don't actually do much virtualization at all) that you're 
willing to make them much much slower just to keep HT for the 95% of the 
time when you're not doing any virtualization at all.

But that case doesn't say "cloud provider" to me. It says "desktop machine 
that occasionally runs virtualbox". And that case doesn't care about the 
security issue in the first place, since it's not running other peoples 
untrusted shit to begin with.

So I really find it unlikely that anybody comes up with something 
worthwhile. 

              Linus

[MODERATED] Re: L1D-Fault KVM mitigation

[David Woodhouse]

On Thu, 2018-05-24 at 09:35 -0700, speck for Linus Torvalds wrote:
> 
> The only case I see making up for it is when vm entry/exit are so rare 
> (because you don't actually do much virtualization at all) that you're 
> willing to make them much much slower just to keep HT for the 95% of the 
> time when you're not doing any virtualization at all.

With decently designed hardware and SR-IOV passthrough for all I/O, and
posted interrupts... 

[MODERATED] Re: L1D-Fault KVM mitigation

[Linus Torvalds]

On Thu, 24 May 2018, speck for David Woodhouse wrote:
> 
> With decently designed hardware and SR-IOV passthrough for all I/O, and
> posted interrupts... 

If we had decently designed hardware, we wouldn't be having this 
discussion in the first place.

Anyway, I'll wait for the numbers (and the patch - even good numbers with 
a hacky patch doesn't necessarily mean "worth upstreamning").

                 Linus

[MODERATED] Re: L1D-Fault KVM mitigation

[David Woodhouse]

On Thu, 2018-05-24 at 09:57 -0700, speck for Linus Torvalds wrote:
> 
> 
> On Thu, 24 May 2018, speck for David Woodhouse wrote:
> > 
> > With decently designed hardware and SR-IOV passthrough for all I/O, and
> > posted interrupts... 
> 
> If we had decently designed hardware, we wouldn't be having this 
> discussion in the first place.

Touché.

But with decently designed hardware *around* the CPUs in question…

…actually, who am I kidding? That's bullshit too. There are plenty of
skeletons in *that* cupboard.

There may be no such thing as "decently designed hardware" but we
*have* managed to eliminate vmexits to the point where they are no
longer the first and only thing we ever have to care about. Which
really does fit into the category you mentioned:

> The only case I see making up for it is when vm entry/exit are so rare 
> (because you don't actually do much virtualization at all) that you're 
> willing to make them much much slower just to keep HT for the 95% of the 
> time when you're not doing any virtualization at all.

My point was that you can do virtualisation without taking vmexits.
Much of the last decade or so of Intel CPU design and platform design
have been about achieving exactly that.

[MODERATED] Re: ***UNCHECKED*** Re: L1D-Fault KVM mitigation

[Joerg Roedel]

On Tue, Apr 24, 2018 at 11:35:37AM +0200, speck for Peter Zijlstra wrote:
> Another option, that is being explored, is to co-schedule siblings.
> So ensure all siblings either run vcpus of the _same_ VM or idle.
> 
> Of course, this is all rather intrusive and ugly and brings with it
> setup costs as well, because you'd have to sync up on VMENTER, VMEXIT
> and interrupts (on the idle CPUs).

Not to mention that it is going to be a maintenance nightmare for the
years to come.

And even if we end up with gang-scheduling in the end, which I don't see
coming yet, we need a simpler plan to have a mitigation when the embargo
lifts.


Regards,

	Joerg

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

On Tue, 24 Apr 2018, speck for Joerg Roedel wrote:
> On Tue, Apr 24, 2018 at 11:35:37AM +0200, speck for Peter Zijlstra wrote:
> > Another option, that is being explored, is to co-schedule siblings.
> > So ensure all siblings either run vcpus of the _same_ VM or idle.
> > 
> > Of course, this is all rather intrusive and ugly and brings with it
> > setup costs as well, because you'd have to sync up on VMENTER, VMEXIT
> > and interrupts (on the idle CPUs).
> 
> Not to mention that it is going to be a maintenance nightmare for the
> years to come.
> 
> And even if we end up with gang-scheduling in the end, which I don't see
> coming yet, we need a simpler plan to have a mitigation when the embargo
> lifts.

There are other problems aside of those Peter mentioned.

There is only a particular class of workloads which can benefit of that:

 multi-vcpu guests where all VCPUs are doing CPU bound computations in
 parallel

Everything else, which is vmexit heavy due to I/O or whatever reasons is
going to suffer badly from the synchronization costs on VMENTER/EXIT.

Not to talk about the single VCPU guests which are pretty widespread in
hosting environments. There the crippled gang-scheduling of VPCU and idle
is not beneficial at all because it's basically the same as HT off just
with the ugly synchronization overhead plus increased power consumption.

Aside of that NMIs are an interesting problem as well. You certainly don't
want to do a synchronization dance from NMI, but ignoring NMI if you are
paranoid is not working either because it can make very interesting data
cache hot depending on the stuff which is monitored.

I've yet to see any numbers which show the effect of this gang-scheduling
mess on a broad range of workloads.

Thanks,

	tglx

[MODERATED] Re: L1D-Fault KVM mitigation

[Andi Kleen]

> Aside of that NMIs are an interesting problem as well. You certainly don't
> want to do a synchronization dance from NMI, but ignoring NMI if you are
> paranoid is not working either because it can make very interesting data
> cache hot depending on the stuff which is monitored.

Like what interesting data? 

AFAIK NMIs do access only some non interesting kernel data structures, 
and potentially the current context if you're talking about perf's stack walking.
The current context would of course already be leakable without NMIs.

-Andi

[MODERATED] Re: L1D-Fault KVM mitigation

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 3787 bytes --]

On 24/04/2018 11:35, speck for Peter Zijlstra wrote:
> I know that I worked a little with Tim on this, and I know Google did
> their own thing (but have not seen patches from them -- is pjt on this
> list?). I've also heard Amazon was also working on things (are they
> here?). And I think RHT was also looking into something (mingo, bonzini
> -- are you guys reading?)

Yes, I am.  First of all: the cost of doing an L1D flush on every
vmentry is absolutely horrible on KVM microbenchmarks, but seems a
little better (around 6% worst case) on syscall microbenchmarks.
"Message-passing" workloads with vCPUs repeatedly going to sleep are the
worst.

First of all, hyperthreading in general doesn't exactly shine when
running many small virtual machines since the VMs are unlikely to share
any code or data and you'll be able to use half the normal amount of L1
cache.  Perhaps KSM shares the guest kernels and recovers some of the
icache (assuming that there are kernel-heavy benchmarks that _also_
benefit from hyperthreading), but it's more likely to give worse than
improved performance.

Hyperthreading may provide slightly better jitter when you run two
different guests on the siblings.  But with gang scheduling you wouldn't
do that, so that's not an issue.  As a result, in the overcommitted case
the main issue is having to explain to the customers that disabling
hyperthreading is not that bad.

Even in the non-overcommitted case, there is a possibility that host
IRQs or NMIs happen, which as Thomas pointed out can also pollute the cache.

The only case where hyperthreading may be salvaged is the case where 1)
all guest CPUs are pinned to a single physical CPU and memory is also
reserved because you use 1GB hugetlbfs 2) host IRQs are either using
VT-d posted interrupts or are pinned away from those physical CPUs that
run guests, 3) you are using nohz_full and other similar fine-tuned
configuration to ensure that the guest CPUs run smoothly.  This includes
NFV usecases, but big databases like SAP are also run like this sometimes.

In this case, you'd need to add synchronization around vmexit.  However,
because these workloads _will_ actually do vmexits, sometimes a lot of
them (e.g. unless you use nohz_full in the guest as well, you'll have
vmexits to program the LAPIC timer).  Either all of them will have to
suffer from the synchronization cost, or you have to arbitrarily decide
that some vmexits are "confined" and unlikely to pollute the cache; in
that case you skip the synchronization and the L1D flush.  For example
you could say "anything that does not do get_user_pages is confined".

Because you've done this arbitrary choice, the synchronization is total
security theater unless you know what you're doing: no two guests on the
same core, no interrupt handlers that can run during a vmexit and
pollute the L1 cache (if that happens, the other sibling would be able
to read that data), etc..

BUT: 1) I'm not saying hyperthreading is valuable in those cases, only
that it can be salvaged; 2) if you're paranoid you're more likely to
disable HT anyway.  So while I do plan to test what happens when we do
synchronization, it's all but certain that we're going to ship it.  And
even that would only be if it is acceptable upstream---I'm not going to
make it a special Red Hat only patch.

Ingo suggested, for ease of testing and also for ease of deployment, a
knob to easily online/offline all siblings but the first on each core.
There's still the chance that some userspace daemon is started before
hyperthreading is software-disabled that way, and is confused by the
number of CPUs suddenly halving, so it would have to be both on the
kernel command line and in debugfs.

Paolo

[MODERATED] Re: L1D-Fault KVM mitigation

[Konrad Rzeszutek Wilk]

On Tue, Apr 24, 2018 at 02:53:15PM +0200, speck for Paolo Bonzini wrote:
> On 24/04/2018 11:35, speck for Peter Zijlstra wrote:
> > I know that I worked a little with Tim on this, and I know Google did
> > their own thing (but have not seen patches from them -- is pjt on this
> > list?). I've also heard Amazon was also working on things (are they
> > here?). And I think RHT was also looking into something (mingo, bonzini
> > -- are you guys reading?)
> 
> Yes, I am.  First of all: the cost of doing an L1D flush on every

..snip..

> Ingo suggested, for ease of testing and also for ease of deployment, a
> knob to easily online/offline all siblings but the first on each core.
> There's still the chance that some userspace daemon is started before
> hyperthreading is software-disabled that way, and is confused by the
> number of CPUs suddenly halving, so it would have to be both on the
> kernel command line and in debugfs.

Are there any patches that you would be willing to share so folks
can review/test/etc? I was going to start doing this next but I suspect
you have already most of this ?

[MODERATED] Re: L1D-Fault KVM mitigation

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]

On 03/05/2018 18:20, speck for Konrad Rzeszutek Wilk wrote:
> On Tue, Apr 24, 2018 at 02:53:15PM +0200, speck for Paolo Bonzini wrote:
>> On 24/04/2018 11:35, speck for Peter Zijlstra wrote:
>>> I know that I worked a little with Tim on this, and I know Google did
>>> their own thing (but have not seen patches from them -- is pjt on this
>>> list?). I've also heard Amazon was also working on things (are they
>>> here?). And I think RHT was also looking into something (mingo, bonzini
>>> -- are you guys reading?)
>>
>> Yes, I am.  First of all: the cost of doing an L1D flush on every
> 
> ..snip..
> 
>> Ingo suggested, for ease of testing and also for ease of deployment, a
>> knob to easily online/offline all siblings but the first on each core.
>> There's still the chance that some userspace daemon is started before
>> hyperthreading is software-disabled that way, and is confused by the
>> number of CPUs suddenly halving, so it would have to be both on the
>> kernel command line and in debugfs.
> 
> Are there any patches that you would be willing to share so folks
> can review/test/etc? I was going to start doing this next but I suspect
> you have already most of this ?
> 

Yes, I'll send them out tomorrow.

Paolo

[MODERATED] Re: L1D-Fault KVM mitigation

[Jiri Kosina]

On Mon, 7 May 2018, speck for Paolo Bonzini wrote:

> > Are there any patches that you would be willing to share so folks can 
> > review/test/etc? I was going to start doing this next but I suspect
> > you have already most of this ?
> 
> Yes, I'll send them out tomorrow.

Hi Paolo,

have you sent those somewhere and I missed that, or is it still pending?

Thanks,

-- 
Jiri Kosina
SUSE Labs

[MODERATED] Re: L1D-Fault KVM mitigation

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 512 bytes --]

On 16/05/2018 10:51, speck for Jiri Kosina wrote:
> On Mon, 7 May 2018, speck for Paolo Bonzini wrote:
> 
>>> Are there any patches that you would be willing to share so folks can 
>>> review/test/etc? I was going to start doing this next but I suspect
>>> you have already most of this ?
>>
>> Yes, I'll send them out tomorrow.
> 
> Hi Paolo,
> 
> have you sent those somewhere and I missed that, or is it still pending?

I'm figuring out right now the scripts to send patches here. :)

Paolo

[MODERATED] Re: L1D-Fault KVM mitigation

[David Woodhouse]

On Wed, 2018-05-16 at 10:53 +0200, speck for Paolo Bonzini wrote:
> On 16/05/2018 10:51, speck for Jiri Kosina wrote:
> > 
> > On Mon, 7 May 2018, speck for Paolo Bonzini wrote:
> > 
> > > 
> > > > 
> > > > Are there any patches that you would be willing to share so
> > > > folks can 
> > > > review/test/etc? I was going to start doing this next but I
> > > > suspect
> > > > you have already most of this ?
> > > Yes, I'll send them out tomorrow.
> >
> > Hi Paolo,
> > 
> > have you sent those somewhere and I missed that, or is it still
> > pending?
>
> I'm figuring out right now the scripts to send patches here. :)

Did you work it out? Is this one also part of the release at 21:00 UTC
today, or do we have a bit longer?

Re: L1D-Fault KVM mitigation

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 893 bytes --]

On Mon, 21 May 2018, speck for David Woodhouse wrote:

> On Wed, 2018-05-16 at 10:53 +0200, speck for Paolo Bonzini wrote:
> > On 16/05/2018 10:51, speck for Jiri Kosina wrote:
> > > 
> > > On Mon, 7 May 2018, speck for Paolo Bonzini wrote:
> > > 
> > > > 
> > > > > 
> > > > > Are there any patches that you would be willing to share so
> > > > > folks can 
> > > > > review/test/etc? I was going to start doing this next but I
> > > > > suspect
> > > > > you have already most of this ?
> > > > Yes, I'll send them out tomorrow.
> > >
> > > Hi Paolo,
> > > 
> > > have you sent those somewhere and I missed that, or is it still
> > > pending?
> >
> > I'm figuring out right now the scripts to send patches here. :)
> 
> Did you work it out? Is this one also part of the release at 21:00 UTC
> today, or do we have a bit longer?

That's not part of today. Today is GPZ4 only.

Thanks,

	tglx