[MODERATED] [PATCH 1/1] SMTDOC 0

[MODERATED] [PATCH 1/1] SMTDOC 0

[Andi Kleen]

Add a document describing when and how to disable L1TF for KVM. The goal
is to avoid disabling as much as possible to keep the performance
benefits from SMT.

The warning message from KVM when SMT is on can point to this
document.

---

An open item is still how to handle interrupts for the "confined cores"
case. Right now it describes the sledge hammer of changing the affinity
of all interrupts, but that's likely not needed in all cases.
I'll send another email on this.

Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 Documentation/virtual/kvm/l1tf-mitigation.txt | 68 +++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
 create mode 100644 Documentation/virtual/kvm/l1tf-mitigation.txt

diff --git a/Documentation/virtual/kvm/l1tf-mitigation.txt b/Documentation/virtual/kvm/l1tf-mitigation.txt
new file mode 100644
index 000000000000..876a28499973
--- /dev/null
+++ b/Documentation/virtual/kvm/l1tf-mitigation.txt
@@ -0,0 +1,68 @@
+L1TF mitigation strategies in KVM
+
+On some Intel CPUs a side channel attack may make it possible to read
+arbitrary data in the L1D cache of the CPU core in the guest, potentially allowing
+to read data from outside the VM.
+
+The Linux kernel automatically detects the conditions for this problem
+and reports the status in /sys/devices/system/cpu/vulnerabilities/l1tf.
+
+With HyperThreading (SMT) enabled the L1D cache of a core is shared between the two
+CPU threads in a core.
+
+KVM mitigates this problems transparently whe SMT is disabled. With SMT
+enabled additional actions may be needed.  This document describes
+these additional actions.
+
+- When the system does not support HyperThreading no further
+effort is needed.
+
+This can be determined with
+
+% lscpu | grep Thread
+
+If the reported Thread number is 1 then HyperThreading is not
+available.
+
+- When the guest is not running any untrusted code then
+no further action is needed.
+
+- When the guest kernel is controlled by the host owner the only
+mitigation needed is to update the guest kernel to a kernel version
+with L1TF page table mitigation integrated.
+
+This will prevent any untrusted user programs running in the guest attacking
+the host, other guests, or other processes running inside the guest.
+
+When this is done no further action is needed.
+
+- When the guest is already confined to an dedicated core that
+does not run any host processes or other guests on either of its threads
+no further action is needed.
+
+The core could still in some cases process interrupts.
+
+The interrupts can be redirected to other cores using the
+/proc/irq/NUMBER/smp_affinity interface.
+
+XXX discuss interrupt security here
+
+- When the guest is not confined it can be confined to dedicated cores
+through exclusive cpusets. This does not necessarily need to be dedicated
+cores for each guest, but could be also dedicated cores for a group
+of VMs controlled by the same entity.
+
+For more details on cpusets please see Documentation/cgroup-v1/cpusets.txt
+
+- If this all fails SMT can be disabled on all cores
+
+SMT improves performance so it should be only disabled when absolutely
+needed, and reenabled after the untrusted guest is finished.
+
+To disable SMT:
+
+echo off > /sys/devices/system/cpu/smt/control
+
+To reenable after running untrusted guest
+
+echo on > /sys/devices/system/cpu/smt/control
-- 
2.14.4

[MODERATED] Re: [PATCH 1/1] SMTDOC 0

[Josh Poimboeuf]

On Thu, Jul 05, 2018 at 03:31:29PM -0700, speck for Andi Kleen wrote:
> From: Andi Kleen <ak@linux.intel.com>
> Subject:  x86/kvm/l1tf: Document when to disable SMT for KVM
> 
> Add a document describing when and how to disable L1TF for KVM. The goal
> is to avoid disabling as much as possible to keep the performance
> benefits from SMT.
> 
> The warning message from KVM when SMT is on can point to this
> document.
> 
> ---
> 
> An open item is still how to handle interrupts for the "confined cores"
> case. Right now it describes the sledge hammer of changing the affinity
> of all interrupts, but that's likely not needed in all cases.
> I'll send another email on this.
> 
> Signed-off-by: Andi Kleen <ak@linux.intel.com>
> ---
>  Documentation/virtual/kvm/l1tf-mitigation.txt | 68 +++++++++++++++++++++++++++
>  1 file changed, 68 insertions(+)
>  create mode 100644 Documentation/virtual/kvm/l1tf-mitigation.txt
> 
> diff --git a/Documentation/virtual/kvm/l1tf-mitigation.txt b/Documentation/virtual/kvm/l1tf-mitigation.txt
> new file mode 100644
> index 000000000000..876a28499973
> --- /dev/null
> +++ b/Documentation/virtual/kvm/l1tf-mitigation.txt
> @@ -0,0 +1,68 @@
> +L1TF mitigation strategies in KVM
> +
> +On some Intel CPUs a side channel attack may make it possible to read
> +arbitrary data in the L1D cache of the CPU core in the guest, potentially allowing
> +to read data from outside the VM.
> +
> +The Linux kernel automatically detects the conditions for this problem
> +and reports the status in /sys/devices/system/cpu/vulnerabilities/l1tf.
> +
> +With HyperThreading (SMT) enabled the L1D cache of a core is shared between the two
> +CPU threads in a core.
> +
> +KVM mitigates this problems transparently whe SMT is disabled. With SMT

"when"

Also, this statement will no longer be true after Jiri's patch, since
the default is l1tf=novirt, which means the L1 flushing is disabled by
default.

> +enabled additional actions may be needed.  This document describes
> +these additional actions.
> +
> +- When the system does not support HyperThreading no further
> +effort is needed.

Ditto

> +
> +This can be determined with
> +
> +% lscpu | grep Thread
> +
> +If the reported Thread number is 1 then HyperThreading is not
> +available.
> +
> +- When the guest is not running any untrusted code then
> +no further action is needed.
> +
> +- When the guest kernel is controlled by the host owner the only
> +mitigation needed is to update the guest kernel to a kernel version
> +with L1TF page table mitigation integrated.
> +
> +This will prevent any untrusted user programs running in the guest attacking
> +the host, other guests, or other processes running inside the guest.
> +
> +When this is done no further action is needed.
> +
> +- When the guest is already confined to an dedicated core that

s/an/a/

> +does not run any host processes or other guests on either of its threads
> +no further action is needed.
> +
> +The core could still in some cases process interrupts.
> +
> +The interrupts can be redirected to other cores using the
> +/proc/irq/NUMBER/smp_affinity interface.
> +
> +XXX discuss interrupt security here
> +
> +- When the guest is not confined it can be confined to dedicated cores
> +through exclusive cpusets. This does not necessarily need to be dedicated
> +cores for each guest, but could be also dedicated cores for a group
> +of VMs controlled by the same entity.
> +
> +For more details on cpusets please see Documentation/cgroup-v1/cpusets.txt
> +
> +- If this all fails SMT can be disabled on all cores
> +
> +SMT improves performance so it should be only disabled when absolutely
> +needed, and reenabled after the untrusted guest is finished.
> +
> +To disable SMT:
> +
> +echo off > /sys/devices/system/cpu/smt/control
> +
> +To reenable after running untrusted guest
> +
> +echo on > /sys/devices/system/cpu/smt/control

Even after re-enabling SMT, the threads will still be offline.  They
will also need to use sysfs to re-online them
(/sys/devices/system/cpu/cpu*/online).

-- 
Josh

[MODERATED] Re: [PATCH 1/1] SMTDOC 0

[Andi Kleen]

> Also, this statement will no longer be true after Jiri's patch, since
> the default is l1tf=novirt, which means the L1 flushing is disabled by
> default.


What?!? Of course L1 flushing in KVM be always enabled. Why would you 
ever want to disable it?  

AFAIK it doesn't make much performance difference anyways.

I thought novirt just means keep SMT on.

Jiri?

-Andi

[MODERATED] Re: [PATCH 1/1] SMTDOC 0

[Jiri Kosina]

On Fri, 6 Jul 2018, speck for Andi Kleen wrote:

> What?!? Of course L1 flushing in KVM be always enabled. Why would you 
> ever want to disable it?

Well, you don't want the flushes if you trust the VMs not to snoop on each 
other (and the hypervisor). That's *exactly the same* argument why we are 
not disabling HT by default, isn't it?

If the VMs are not trusted, enabling conditionally just half of the 
mitigations doesn't bring any additional security.

So the *whole and only* point of ('machine/system independent') 'l1tf='
parameter really is to let the system owner/admin to globally take a 
simple choice between

	- I absolutely do want to isolate VMs and am OK with paying the 
          performance price if it's needed on a particular HW 
	  (l1tf=full[,novirt])

	- My VMs+hypervisor absolutely are friends with each other no 
          matter what (l1tf="novirt,nowarn")
	
	- dear kernel, please let me know in case something potentially 
          dangerous is happening, but keep going (l1tf="novirt" or 
          l1tf="full" with SMT on) -- the default

I am afraid that if we don't start from this point, the state space 
explosion happens immediately.

Or are you suggesting to flush L1D cache on *every* VM boundary crossing, 
unconditionally, by default? If so, please provide the full decision 
matrix understandable for end users (admins) so that each and every of 
them is able to make a qualified decision on the combination of 
smt/flushes/process afinity/irq afinity parameters they have to use so 
that they are "feeling proven safe". 
I am afraid the error rate will be enormous, while with the proposed 
semantics I am trying to get to is as close as possible to clearly 
understandable "untrusted VMs->full mitigation" message as possible.

I also believe (Linus, if I am interpreting you in a wrong way, please 
speak up) that was why Linus (and others) originally suggested not to 
disable HT by default -- exactly because the 'default' case (with 
exception to the cloud providers) is that you *don't* require the full VM 
isolation (ignoring all the "HT, interrupts, caches, flushes" details) *by 
design*.

> AFAIK it doesn't make much performance difference anyways.

Okay, so if there is a solid proof that all the added vmenter/vmexit 
overhead is completely (or even in amortized sense) for free, so be it. 
Has anyone really done such analysis yet?

I can't really see us just adding L1D flush to every VM boundary without 
having even hand-wawy numbers available. Heck, we're accompanyning every 
tiny scheduler change by almost-a-master-thesis-in-statistics changelog, 
and now we'd just do this all of a sudden?

> I thought novirt just means keep SMT on.

Then please read a changelog, and the text being added to 
kernel-parameters.txt, and the code.

Anyway, a simple question: which of the statements below you believe 
should be the default kernel view of the world, if it detects L1TF 
vulnerable CPU?

- the VMs are mutually not trusted (wrt. other VMs, and *also* wrt. the 
  hypervisor)

- the VMs and the hypervisor mutually trust each other

- something else (please explain in terms of decision matrix / tree)

Thanks,

-- 
Jiri Kosina
SUSE Labs

[MODERATED] Re: [PATCH 1/1] SMTDOC 0

[Jiri Kosina]

On Sat, 7 Jul 2018, speck for Jiri Kosina wrote:

> 	- I absolutely do want to isolate VMs and am OK with paying the 
>           performance price if it's needed on a particular HW 
> 	  (l1tf=full[,novirt])

full[,force] here, sorry for the noise.

-- 
Jiri Kosina
SUSE Labs

[MODERATED] Re: [PATCH 1/1] SMTDOC 0

[Linus Torvalds]

On Sat, 7 Jul 2018, speck for Jiri Kosina wrote:
> 
> I also believe (Linus, if I am interpreting you in a wrong way, please 
> speak up) that was why Linus (and others) originally suggested not to 
> disable HT by default -- exactly because the 'default' case (with 
> exception to the cloud providers) is that you *don't* require the full VM 
> isolation (ignoring all the "HT, interrupts, caches, flushes" details) *by 
> design*.

No, the *true* default is that you don't do VM's at all, even if you have 
the capability.

And those people who do the occasional "run some windows VM for xyz" don't 
care about the isolation much, not "by design", but simply because it's 
not an issue. They don't run crazy untrusted code. 

The cloud people really are special. They can't trust their customers not 
to be actively evil. Almost nobody else cares.

                   Linus

[MODERATED] Re: [PATCH 1/1] SMTDOC 0

[Andi Kleen]

On Sat, Jul 07, 2018 at 01:00:59AM +0200, speck for Jiri Kosina wrote:
> On Fri, 6 Jul 2018, speck for Andi Kleen wrote:
> 
> > What?!? Of course L1 flushing in KVM be always enabled. Why would you 
> > ever want to disable it?
> 
> Well, you don't want the flushes if you trust the VMs not to snoop on each 
> other (and the hypervisor). That's *exactly the same* argument why we are 
> not disabling HT by default, isn't it?

HT off can be very expensive, L1 flush on entry is not.

So the trade offs are very different here. iirc the flush overhead
was low enough that it's probably ok to just have it on by default,

Maybe an obscure tuanble somewhere to turn it off if someone is extremely
sensitive to performance. But most users probably don't need
to bother with that.

> If the VMs are not trusted, enabling conditionally just half of the 
> mitigations doesn't bring any additional security.

Everything helps, it slows down attackers and since the L1 cache
is very transistent it may well make things impossible.

-Andi

[MODERATED] Re: [PATCH 1/1] SMTDOC 0

[Linus Torvalds]

On Fri, 6 Jul 2018, speck for Andi Kleen wrote:
> 
> HT off can be very expensive, L1 flush on entry is not.

What?

What were the 50-100% slowdown reports I saw? I think they were for 
disk-IO intensive stuff, but I seem to recall a huge slowdown from the L1 
flush.

No?

           Linus

Re: [PATCH 1/1] SMTDOC 0

[Thomas Gleixner]

On Fri, 6 Jul 2018, speck for Linus Torvalds wrote:
> On Fri, 6 Jul 2018, speck for Andi Kleen wrote:
> > 
> > HT off can be very expensive, L1 flush on entry is not.
> 
> What?
> 
> What were the 50-100% slowdown reports I saw? I think they were for 
> disk-IO intensive stuff, but I seem to recall a huge slowdown from the L1 
> flush.

I posted numbers once:

Bootup of a stock debian guest is 10% slower with L1D Flush on every VMENTER

A kernel compile in the same guest (all virtio) slows down by < 1%. The
number of VMEXITs is minimal, so no surprise.

With a testcase which took an interrupt every 25us, the slowdown depends on
the computational work done in the guest and ranges between 20 and 70%.
The number of L1D misses increases by a factor of 13. noflush 2e6, flush
260e6.

And yes, we should clearly describe facts and not the wishful thinking of
Intel. I'm so tired of this marketing drivel.

Thanks,

	tglx

Re: [PATCH 1/1] SMTDOC 0

[Thomas Gleixner]

On Fri, 6 Jul 2018, speck for Josh Poimboeuf wrote:
> On Thu, Jul 05, 2018 at 03:31:29PM -0700, speck for Andi Kleen wrote:
> > +To disable SMT:
> > +
> > +echo off > /sys/devices/system/cpu/smt/control
> > +
> > +To reenable after running untrusted guest
> > +
> > +echo on > /sys/devices/system/cpu/smt/control
> 
> Even after re-enabling SMT, the threads will still be offline.  They
> will also need to use sysfs to re-online them
> (/sys/devices/system/cpu/cpu*/online).

It's easy enough to make that automatic. There was an issue when I tried
first, but that's not longer the case and I simply forgot to write the
patch. I'll whip that up later today.

Thanks,

	tglx

[MODERATED] Re: [PATCH 1/1] SMTDOC 0

[Linus Torvalds]

On Sat, 7 Jul 2018, speck for Thomas Gleixner wrote:
> 
> Bootup of a stock debian guest is 10% slower with L1D Flush on every VMENTER
> 
> A kernel compile in the same guest (all virtio) slows down by < 1%. The
> number of VMEXITs is minimal, so no surprise.

Yeah, compiles are generally fairly easy, and 90% of the time just 
CPU-intensive in user space.

> With a testcase which took an interrupt every 25us, the slowdown depends on
> the computational work done in the guest and ranges between 20 and 70%.
> The number of L1D misses increases by a factor of 13. noflush 2e6, flush
> 260e6.

I assum eyou tested the non-MSR one, doing the nasty flush by hand. That's 
probably worse, because not only does the flush (obviously) cause a lot of 
subsequent D$ fills, but the flush itself basically forces the same number 
of L1 misses in order to do the flush (and probably a lot of L2 misses 
too, to read the stupid pointless flush area).

So the "just invalidate the cache with an MSR write" is probably quite 
noticeably better.

But it's probably better by a factor of 2x in the best case, not a factor 
of 13x ;)

                Linus

Re: [PATCH 1/1] SMTDOC 0

[Thomas Gleixner]

On Sat, 7 Jul 2018, speck for Linus Torvalds wrote:
> On Sat, 7 Jul 2018, speck for Thomas Gleixner wrote:
> > 
> > Bootup of a stock debian guest is 10% slower with L1D Flush on every VMENTER
> > 
> > A kernel compile in the same guest (all virtio) slows down by < 1%. The
> > number of VMEXITs is minimal, so no surprise.
> 
> Yeah, compiles are generally fairly easy, and 90% of the time just 
> CPU-intensive in user space.
> 
> > With a testcase which took an interrupt every 25us, the slowdown depends on
> > the computational work done in the guest and ranges between 20 and 70%.
> > The number of L1D misses increases by a factor of 13. noflush 2e6, flush
> > 260e6.
> 
> I assum eyou tested the non-MSR one, doing the nasty flush by hand. That's 
> probably worse, because not only does the flush (obviously) cause a lot of 
> subsequent D$ fills, but the flush itself basically forces the same number 
> of L1 misses in order to do the flush (and probably a lot of L2 misses 
> too, to read the stupid pointless flush area).
> 
> So the "just invalidate the cache with an MSR write" is probably quite 
> noticeably better.
> 
> But it's probably better by a factor of 2x in the best case, not a factor 
> of 13x ;)

No, that was with the MSR based flush. The SW one is roughly in the same
ballpark on that machine. It's slightly worse, but not a huge difference.

Thanks,

	tglx

[MODERATED] Re: [PATCH 1/1] SMTDOC 0

[Andi Kleen]

> Bootup of a stock debian guest is 10% slower with L1D Flush on every VMENTER

Most of that is the PIO overhead, which is not terrible interesting.
It does an exit for every PIO to load kernel+initrd.

This one should go away anyways with Paolo's optimizations to not
do L1 flush for such "fast exit" cases

> And yes, we should clearly describe facts and not the wishful thinking of
> Intel. I'm so tired of this marketing drivel.

The benchmarks I saw didn't show significant overhead for "real" workloads.

Of course you can always construct some worst case, but basing
your decisions purely on worst cases is usually not a good strategy.

-Andi

Re: [PATCH 1/1] SMTDOC 0

[Thomas Gleixner]

On Sat, 7 Jul 2018, speck for Andi Kleen wrote:
> > And yes, we should clearly describe facts and not the wishful thinking of
> > Intel. I'm so tired of this marketing drivel.
> 
> The benchmarks I saw didn't show significant overhead for "real" workloads.
> 
> Of course you can always construct some worst case, but basing
> your decisions purely on worst cases is usually not a good strategy.

It's still wrong not to clearly tell that this can have significant impact
under certain conditions. VMEXIT heavy scenarios exist in reality.

Thanks,

	tglx

[MODERATED] Re: [PATCH 1/1] SMTDOC 0

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

On 07/08/2018 06:22 AM, speck for Thomas Gleixner wrote:
> On Sat, 7 Jul 2018, speck for Andi Kleen wrote:
>>> And yes, we should clearly describe facts and not the wishful thinking of
>>> Intel. I'm so tired of this marketing drivel.
>>
>> The benchmarks I saw didn't show significant overhead for "real" workloads.
>>
>> Of course you can always construct some worst case, but basing
>> your decisions purely on worst cases is usually not a good strategy.
> 
> It's still wrong not to clearly tell that this can have significant impact
> under certain conditions. VMEXIT heavy scenarios exist in reality.

Just to add, according to the numbers I've seen, it takes upwards of 512
cycles (double on older parts) to refill the L1D on Skylake. That's not
huge, but it's not free. And it's not a complete mitigation on its own.
It seems to make most sense to go with what Jiri and Linus have said.

In case it's helpful to have a comparison example though, and because
not everyone here may be aware of it, POWER9 is vulnerable to original
Meltdown and the currently upstream mitigation is to do an L1D$ flush on
every return to userspace or hypervisor exit. Essentially, the same as
is being described here, with also a similarly fast L2->L1 refill. They
added a fast L1D$ flush instruction via millicode similar to Intel's.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop