[MODERATED] [PATCH 1/1] SMTDOCv2 0

[MODERATED] [PATCH 1/1] SMTDOCv2 0

[Andi Kleen]

Add a document describing when and how to disable L1TF for KVM. The goal
is to avoid disabling as much as possible to keep the performance
benefits from SMT.

The warning message from KVM when SMT is on can point to this
document.

---

An open item is still how to handle interrupts for the "confined cores"
case. Right now it describes the sledge hammer of changing the affinity
of all interrupts, but that's likely not needed in all cases.
I'll send another email on this.

v2: Fix typos, fix SMT on sequence.

Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 Documentation/virtual/kvm/l1tf-mitigation.txt | 69 +++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)
 create mode 100644 Documentation/virtual/kvm/l1tf-mitigation.txt

diff --git a/Documentation/virtual/kvm/l1tf-mitigation.txt b/Documentation/virtual/kvm/l1tf-mitigation.txt
new file mode 100644
index 000000000000..c8d72d0057f0
--- /dev/null
+++ b/Documentation/virtual/kvm/l1tf-mitigation.txt
@@ -0,0 +1,69 @@
+L1TF mitigation strategies in KVM
+
+On some Intel CPUs a side channel attack may make it possible to read
+arbitrary data in the L1D cache of the CPU core in the guest, potentially allowing
+to read data from outside the VM.
+
+The Linux kernel automatically detects the conditions for this problem
+and reports the status in /sys/devices/system/cpu/vulnerabilities/l1tf.
+
+With HyperThreading (SMT) enabled the L1D cache of a core is shared between the two
+CPU threads in a core.
+
+KVM mitigates this problems transparently when SMT is disabled. With SMT
+enabled additional actions may be needed.  This document describes
+these additional actions.
+
+- When the system does not support HyperThreading no further
+effort is needed.
+
+This can be determined with
+
+% lscpu | grep Thread
+
+If the reported Thread number is 1 then HyperThreading is not
+available.
+
+- When the guest is not running any untrusted code then
+no further action is needed.
+
+- When the guest kernel is controlled by the host owner the only
+mitigation needed is to update the guest kernel to a kernel version
+with L1TF page table mitigation integrated.
+
+This will prevent any untrusted user programs running in the guest attacking
+the host, other guests, or other processes running inside the guest.
+
+When this is done no further action is needed.
+
+- When the guest is already confined to a dedicated core that
+does not run any host processes or other guests on either of its threads
+no further action is needed.
+
+The core could still in some cases process interrupts.
+
+The interrupts can be redirected to other cores using the
+/proc/irq/NUMBER/smp_affinity interface.
+
+- When the guest is not confined it can be confined to dedicated cores
+through exclusive cpusets. This does not necessarily need to be dedicated
+cores for each guest, but could be also dedicated cores for a group
+of VMs controlled by the same entity.
+
+For more details on cpusets please see Documentation/cgroup-v1/cpusets.txt
+
+- If this all fails SMT can be disabled on all cores
+
+SMT improves performance so it should be only disabled when absolutely
+needed, and reenabled after the untrusted guest is finished.
+
+To disable SMT:
+
+echo off > /sys/devices/system/cpu/smt/control
+
+To reenable after running untrusted guest
+
+echo on > /sys/devices/system/cpu/smt/control
+for i in /sys/devices/system/cpu/cpu*/online ; do
+    echo 1 > $i
+done
-- 
2.14.4

Re: [PATCH 1/1] SMTDOCv2 0

[Thomas Gleixner]

On Fri, 6 Jul 2018, speck for Andi Kleen wrote:
> --- /dev/null
> +++ b/Documentation/virtual/kvm/l1tf-mitigation.txt

I seem to remember that I asked for RST format. Aside of that we really
want to start a proper documentation section about all that speculation
nonsense and not having a random text file here and there.

> @@ -0,0 +1,69 @@
> +L1TF mitigation strategies in KVM
> +On some Intel CPUs a side channel attack may make it possible to read

On some? L1TF affects every fricking Intel CPU with EPT except a few ATOMs.

> +arbitrary data in the L1D cache of the CPU core in the guest, potentially allowing
> +to read data from outside the VM.

This really wants to be explained so that the admin who is supposed to read
that understands the mechanims at least at the high level. That
'may/potentially' weasel wording is not at all helpful to make decisions
about the mitigations.

> +The Linux kernel automatically detects the conditions for this problem
> +and reports the status in /sys/devices/system/cpu/vulnerabilities/l1tf.
> +
> +With HyperThreading (SMT) enabled the L1D cache of a core is shared between the two
> +CPU threads in a core.

And the consequence of this is? I surely know, but the people who seek
advice not necessarily so.

> +KVM mitigates this problems transparently when SMT is disabled. With SMT

Wrong. This depends on L1D Flush and contrary to your believe there are
valid reasons to make L1D flushing controllable.

> +enabled additional actions may be needed.  This document describes
> +these additional actions.
> +
> +- When the system does not support HyperThreading no further
> +effort is needed.
> +
> +This can be determined with
> +
> +% lscpu | grep Thread
> +
> +If the reported Thread number is 1 then HyperThreading is not
> +available.

Groan.

/sys/devices/system/cpu/smt/control
/sys/devices/system/cpu/smt/active

are there for exactly this.

> +- When the guest is not running any untrusted code then
> +no further action is needed.
> +
> +- When the guest kernel is controlled by the host owner the only
> +mitigation needed is to update the guest kernel to a kernel version
> +with L1TF page table mitigation integrated.
> +
> +This will prevent any untrusted user programs running in the guest attacking
> +the host, other guests, or other processes running inside the guest.
> +
> +When this is done no further action is needed.
> +
> +- When the guest is already confined to a dedicated core that
> +does not run any host processes or other guests on either of its threads
> +no further action is needed.
> +
> +The core could still in some cases process interrupts.
> +
> +The interrupts can be redirected to other cores using the
> +/proc/irq/NUMBER/smp_affinity interface.
> +
> +- When the guest is not confined it can be confined to dedicated cores
> +through exclusive cpusets. This does not necessarily need to be dedicated
> +cores for each guest, but could be also dedicated cores for a group
> +of VMs controlled by the same entity.
> +
> +For more details on cpusets please see Documentation/cgroup-v1/cpusets.txt
> +
> +- If this all fails SMT can be disabled on all cores

Fails? How should someone who is not familiar with the deep details of that
decided that the above fails?

> +
> +SMT improves performance so it should be only disabled when absolutely

SMT improves performance depending on the workload. There are workloads
which are negatively impacted by SMT.

That said, this all wants to be properly structured and explained in
detail. A hastily cobbled together string of syllables does not make a
document which is intended to provide guidance to people who do not have a
deep technical insight into this.

Thanks,

	tglx

[MODERATED] Re: [PATCH 1/1] SMTDOCv2 0

[Andi Kleen]

On Sat, Jul 07, 2018 at 11:39:02AM +0200, speck for Thomas Gleixner wrote:
> On Fri, 6 Jul 2018, speck for Andi Kleen wrote:
> > --- /dev/null
> > +++ b/Documentation/virtual/kvm/l1tf-mitigation.txt
> 
> I seem to remember that I asked for RST format. Aside of that we really

I probably lost that somewhere in your flood of ad hominems.

> > @@ -0,0 +1,69 @@
> > +L1TF mitigation strategies in KVM
> > +On some Intel CPUs a side channel attack may make it possible to read
> 
> On some? L1TF affects every fricking Intel CPU with EPT except a few ATOMs.

Think what will the state be when someone reads this in two years.

> 
> > +arbitrary data in the L1D cache of the CPU core in the guest, potentially allowing
> > +to read data from outside the VM.
> 
> This really wants to be explained so that the admin who is supposed to read
> that understands the mechanims at least at the high level. That
> 'may/potentially' weasel wording is not at all helpful to make decisions
> about the mitigations.

There will be an Intel white paper with details, we can point to that later.
I don't think going into micro architectural details here will
help any admin.

> 
> Fails? How should someone who is not familiar with the deep details of that
> decided that the above fails?

It's a list of actions all with preconditions. For each it is clear
if you can or cannot do the action. 

> 
> > +
> > +SMT improves performance so it should be only disabled when absolutely
> 
> SMT improves performance depending on the workload. There are workloads
> which are negatively impacted by SMT.
> 
> That said, this all wants to be properly structured and explained in
> detail. A hastily cobbled together string of syllables does not make a
> document which is intended to provide guidance to people who do not have a
> deep technical insight into this.

We can perhaps expand some things slightly.

But I doubt any admin is interested in reading a novel on this here. I
certainly wouldn't be. There will be plenty of other material elsewhere for
people who want all the gory details.

-Andi

Re: [PATCH 1/1] SMTDOCv2 0

[Thomas Gleixner]

On Sat, 7 Jul 2018, speck for Andi Kleen wrote:
> On Sat, Jul 07, 2018 at 11:39:02AM +0200, speck for Thomas Gleixner wrote:
> > On some? L1TF affects every fricking Intel CPU with EPT except a few ATOMs.
> 
> Think what will the state be when someone reads this in two years.

It's not rocket science to spell it out in a way which is still valid in
two years.

> > > +arbitrary data in the L1D cache of the CPU core in the guest, potentially allowing
> > > +to read data from outside the VM.
> > 
> > This really wants to be explained so that the admin who is supposed to read
> > that understands the mechanims at least at the high level. That
> > 'may/potentially' weasel wording is not at all helpful to make decisions
> > about the mitigations.
> 
> There will be an Intel white paper with details, we can point to that later.
> I don't think going into micro architectural details here will
> help any admin.

It's not about micro architectural details, but there needs to be some high
level explanation so the administrator can make a judgement about the
different mitigation methods.

> > Fails? How should someone who is not familiar with the deep details of that
> > decided that the above fails?
> 
> It's a list of actions all with preconditions. For each it is clear
> if you can or cannot do the action. 

Oh well..

> > > +SMT improves performance so it should be only disabled when absolutely
> > 
> > SMT improves performance depending on the workload. There are workloads
> > which are negatively impacted by SMT.
> > 
> > That said, this all wants to be properly structured and explained in
> > detail. A hastily cobbled together string of syllables does not make a
> > document which is intended to provide guidance to people who do not have a
> > deep technical insight into this.
> 
> We can perhaps expand some things slightly.
> 
> But I doubt any admin is interested in reading a novel on this here. I
> certainly wouldn't be. There will be plenty of other material elsewhere for
> people who want all the gory details.

Nobody asks you to write a novel. But a consice documentation is surely
due. Aside of that it does not matter how much other material will be
available. This is about documentation the kernel provides and I consider
it bad practice to throw a few bones into a text file just to claim that we
have documentation. You might be content with that scrap paper, but most
people prefer and appreciate proper written documentation.

Thanks,

	tglx