[MODERATED] Cache load gadgets

[MODERATED] Cache load gadgets

[David Woodhouse]

If we aren't disabling hyperthreading, then we get to lose sleep over
what a VM guest can trick us into loading into L1$ while a sibling runs
the attack.

For this they only need half a Spectre-v1 gadget; they only need to
load something into L1. We've tweaked our scanner and trawled through
the false positives, and we have a handful of patches which look
something like the one below.

Should we be getting these into the tree for next Tuesday's release?

From: Julian Stecklina <jsteckli@amazon.de>
Date: Mon, 6 Aug 2018 13:55:43 +0200
Subject: [PATCH] kvm, x86: fix potential cache load gadget
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

APIC IDs are guest provided. The array access in the Local APIC code is
a potential cache load gadget.

Signed-off-by: Julian Stecklina <jsteckli@amazon.de>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Martin Pohlack <mpohlack@amazon.de>
Reviewed-by: Jan H. Schönherr <jschoenh@amazon.de>
---
 arch/x86/kvm/lapic.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 55cf55f..e919283 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -28,6 +28,7 @@
 #include <linux/export.h>
 #include <linux/math64.h>
 #include <linux/slab.h>
+#include <linux/nospec.h>
 #include <asm/processor.h>
 #include <asm/msr.h>
 #include <asm/page.h>
@@ -133,7 +134,7 @@ static inline bool kvm_apic_map_get_logical_dest(struct kvm_apic_map *map,
 		if (offset <= max_apic_id) {
 			u8 cluster_size = min(max_apic_id - offset + 1, 16U);
 
-			*cluster = &map->phys_map[offset];
+			*cluster = &map->phys_map[array_index_nospec(offset, max_apic_id + 1)];
 			*mask = dest_id & (0xffff >> (16 - cluster_size));
 		} else {
 			*mask = 0;
@@ -821,7 +822,7 @@ static inline bool kvm_apic_map_get_dest_lapic(struct kvm *kvm,
 		if (irq->dest_id > map->max_apic_id) {
 			*bitmap = 0;
 		} else {
-			*dst = &map->phys_map[irq->dest_id];
+			*dst = &map->phys_map[array_index_nospec(irq->dest_id, map->max_apic_id + 1)];
 			*bitmap = 1;
 		}
 		return true;
-- 
2.7.4

[MODERATED] Re: Cache load gadgets

[Andi Kleen]

On Wed, Aug 08, 2018 at 02:53:35PM +0100, speck for David Woodhouse wrote:
> If we aren't disabling hyperthreading, then we get to lose sleep over
> what a VM guest can trick us into loading into L1$ while a sibling runs
> the attack.
> 
> For this they only need half a Spectre-v1 gadget; they only need to
> load something into L1. We've tweaked our scanner and trawled through
> the false positives, and we have a handful of patches which look
> something like the one below.
> 
> Should we be getting these into the tree for next Tuesday's release?

I don't think it's absolutely needed for Tuesday, can go in slower.

There will be a range of enhancements for L1TF beyond Tuesday anyways.

-Andi

[MODERATED] Re: Cache load gadgets

[Josh Poimboeuf]

On Wed, Aug 08, 2018 at 07:53:16AM -0700, speck for Andi Kleen wrote:
> On Wed, Aug 08, 2018 at 02:53:35PM +0100, speck for David Woodhouse wrote:
> > If we aren't disabling hyperthreading, then we get to lose sleep over
> > what a VM guest can trick us into loading into L1$ while a sibling runs
> > the attack.
> > 
> > For this they only need half a Spectre-v1 gadget; they only need to
> > load something into L1. We've tweaked our scanner and trawled through
> > the false positives, and we have a handful of patches which look
> > something like the one below.
> > 
> > Should we be getting these into the tree for next Tuesday's release?
> 
> I don't think it's absolutely needed for Tuesday, can go in slower.
> 
> There will be a range of enhancements for L1TF beyond Tuesday anyways.

Yet this patch is trivial, no harm in merging it now.  We already have a
similar one in RHEL (and several hundred more like it).

-- 
Josh

[MODERATED] Re: Cache load gadgets

[Andi Kleen]

On Wed, Aug 08, 2018 at 10:34:29AM -0500, speck for Josh Poimboeuf wrote:
> On Wed, Aug 08, 2018 at 07:53:16AM -0700, speck for Andi Kleen wrote:
> > On Wed, Aug 08, 2018 at 02:53:35PM +0100, speck for David Woodhouse wrote:
> > > If we aren't disabling hyperthreading, then we get to lose sleep over
> > > what a VM guest can trick us into loading into L1$ while a sibling runs
> > > the attack.
> > > 
> > > For this they only need half a Spectre-v1 gadget; they only need to
> > > load something into L1. We've tweaked our scanner and trawled through
> > > the false positives, and we have a handful of patches which look
> > > something like the one below.
> > > 
> > > Should we be getting these into the tree for next Tuesday's release?
> > 
> > I don't think it's absolutely needed for Tuesday, can go in slower.
> > 
> > There will be a range of enhancements for L1TF beyond Tuesday anyways.
> 
> Yet this patch is trivial, no harm in merging it now.  We already have a
> similar one in RHEL (and several hundred more like it).

The main concern with such an approach is that we need something
scalable to get new code that goes into the kernel scanned
and fixed.

Otherwise it'll quickly bitrot.

-Andi

[MODERATED] Re: Cache load gadgets

[Josh Poimboeuf]

On Wed, Aug 08, 2018 at 09:50:46AM -0700, speck for Andi Kleen wrote:
> On Wed, Aug 08, 2018 at 10:34:29AM -0500, speck for Josh Poimboeuf wrote:
> > On Wed, Aug 08, 2018 at 07:53:16AM -0700, speck for Andi Kleen wrote:
> > > On Wed, Aug 08, 2018 at 02:53:35PM +0100, speck for David Woodhouse wrote:
> > > > If we aren't disabling hyperthreading, then we get to lose sleep over
> > > > what a VM guest can trick us into loading into L1$ while a sibling runs
> > > > the attack.
> > > > 
> > > > For this they only need half a Spectre-v1 gadget; they only need to
> > > > load something into L1. We've tweaked our scanner and trawled through
> > > > the false positives, and we have a handful of patches which look
> > > > something like the one below.
> > > > 
> > > > Should we be getting these into the tree for next Tuesday's release?
> > > 
> > > I don't think it's absolutely needed for Tuesday, can go in slower.
> > > 
> > > There will be a range of enhancements for L1TF beyond Tuesday anyways.
> > 
> > Yet this patch is trivial, no harm in merging it now.  We already have a
> > similar one in RHEL (and several hundred more like it).
> 
> The main concern with such an approach is that we need something
> scalable to get new code that goes into the kernel scanned
> and fixed.
> 
> Otherwise it'll quickly bitrot.

Yeah, long term we need to have compiler/tooling support to fix these
automatically.  In the mean time we have to plug the holes manually (and
significant work is already going on upstream to do so).

-- 
Josh

Re: Cache load gadgets

[Thomas Gleixner]

On Wed, 8 Aug 2018, speck for David Woodhouse wrote:

> If we aren't disabling hyperthreading, then we get to lose sleep over
> what a VM guest can trick us into loading into L1$ while a sibling runs
> the attack.
> 
> For this they only need half a Spectre-v1 gadget; they only need to
> load something into L1. We've tweaked our scanner and trawled through
> the false positives, and we have a handful of patches which look
> something like the one below.
>
> Should we be getting these into the tree for next Tuesday's release?

They look simple enough. So yes, why not as it needs to be fixed anyway.
 
> From: Julian Stecklina <jsteckli@amazon.de>
> Date: Mon, 6 Aug 2018 13:55:43 +0200
> Subject: [PATCH] kvm, x86: fix potential cache load gadget
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
> 
> APIC IDs are guest provided. The array access in the Local APIC code is
> a potential cache load gadget.

This wants a bit more explanation. Connecting cache load gadget to L1TF
won't be immediately clear for a reader who did not have to rack his nerves
with this crap and it will make those who had to scratching their head in 6
month from now after we drained all the horrors from our memory.

Thanks,

	tglx

Re: Cache load gadgets

[Thomas Gleixner]

On Wed, 8 Aug 2018, speck for Josh Poimboeuf wrote:
> On Wed, Aug 08, 2018 at 09:50:46AM -0700, speck for Andi Kleen wrote:
> > On Wed, Aug 08, 2018 at 10:34:29AM -0500, speck for Josh Poimboeuf wrote:
> > > On Wed, Aug 08, 2018 at 07:53:16AM -0700, speck for Andi Kleen wrote:
> > > > On Wed, Aug 08, 2018 at 02:53:35PM +0100, speck for David Woodhouse wrote:
> > > > > If we aren't disabling hyperthreading, then we get to lose sleep over
> > > > > what a VM guest can trick us into loading into L1$ while a sibling runs
> > > > > the attack.
> > > > > 
> > > > > For this they only need half a Spectre-v1 gadget; they only need to
> > > > > load something into L1. We've tweaked our scanner and trawled through
> > > > > the false positives, and we have a handful of patches which look
> > > > > something like the one below.
> > > > > 
> > > > > Should we be getting these into the tree for next Tuesday's release?
> > > > 
> > > > I don't think it's absolutely needed for Tuesday, can go in slower.
> > > > 
> > > > There will be a range of enhancements for L1TF beyond Tuesday anyways.
> > > 
> > > Yet this patch is trivial, no harm in merging it now.  We already have a
> > > similar one in RHEL (and several hundred more like it).
> > 
> > The main concern with such an approach is that we need something
> > scalable to get new code that goes into the kernel scanned
> > and fixed.
> > 
> > Otherwise it'll quickly bitrot.
> 
> Yeah, long term we need to have compiler/tooling support to fix these
> automatically.  In the mean time we have to plug the holes manually (and
> significant work is already going on upstream to do so).

Right. We need both. The current fixes and tooling to catch new ones.

Thanks,

	tglx

[MODERATED] Re: Cache load gadgets

[David Woodhouse]

On Thu, 2018-08-09 at 21:00 +0200, speck for Thomas Gleixner wrote:
> On Wed, 8 Aug 2018, speck for David Woodhouse wrote:
> 
> > If we aren't disabling hyperthreading, then we get to lose sleep over
> > what a VM guest can trick us into loading into L1$ while a sibling runs
> > the attack.
> > 
> > For this they only need half a Spectre-v1 gadget; they only need to
> > load something into L1. We've tweaked our scanner and trawled through
> > the false positives, and we have a handful of patches which look
> > something like the one below.
> >
> > Should we be getting these into the tree for next Tuesday's release?
> 
> They look simple enough. So yes, why not as it needs to be fixed anyway.

Well... a lot of people are busy making production builds and testing
them, and getting things ready to release on Tuesday.

If we pile more stuff on now, we're basically making a lot of people
throw it all away and work through the weekend. Which is fine if it's
necessary... but we should only really do if it if *is* necessary.

I'm inclined to suggest the gadget fixes can wait until Tuesday. 

[MODERATED] Re: Cache load gadgets

[Konrad Rzeszutek Wilk]

On Wed, Aug 08, 2018 at 10:34:29AM -0500, speck for Josh Poimboeuf wrote:
> On Wed, Aug 08, 2018 at 07:53:16AM -0700, speck for Andi Kleen wrote:
> > On Wed, Aug 08, 2018 at 02:53:35PM +0100, speck for David Woodhouse wrote:
> > > If we aren't disabling hyperthreading, then we get to lose sleep over
> > > what a VM guest can trick us into loading into L1$ while a sibling runs
> > > the attack.
> > > 
> > > For this they only need half a Spectre-v1 gadget; they only need to
> > > load something into L1. We've tweaked our scanner and trawled through
> > > the false positives, and we have a handful of patches which look
> > > something like the one below.
> > > 
> > > Should we be getting these into the tree for next Tuesday's release?
> > 
> > I don't think it's absolutely needed for Tuesday, can go in slower.
> > 
> > There will be a range of enhancements for L1TF beyond Tuesday anyways.
> 
> Yet this patch is trivial, no harm in merging it now.  We already have a
> similar one in RHEL (and several hundred more like it).

Josh, and David,  would it be possible to include me (konrad.wilk@oracle.com),
jamie.iles@oracle.com, liran.alon@oracle.com on anything that is sent
after Tuesday?

As in we want to help in analyzing and reviewing them to minimize the
effect on the upstream kernel for this particular nasty attack and the
above folks aren't reading every LKML email :-). 

Thanks!
> 
> -- 
> Josh

[MODERATED] Re: Cache load gadgets

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1462 bytes --]

On 09/08/2018 21:20, speck for David Woodhouse wrote:
> On Thu, 2018-08-09 at 21:00 +0200, speck for Thomas Gleixner wrote:
>> On Wed, 8 Aug 2018, speck for David Woodhouse wrote:
>>
>>> If we aren't disabling hyperthreading, then we get to lose sleep over
>>> what a VM guest can trick us into loading into L1$ while a sibling runs
>>> the attack.
>>>  
>>> For this they only need half a Spectre-v1 gadget; they only need to
>>> load something into L1. We've tweaked our scanner and trawled through
>>> the false positives, and we have a handful of patches which look
>>> something like the one below.
>>>
>>> Should we be getting these into the tree for next Tuesday's release?
>>
>> They look simple enough. So yes, why not as it needs to be fixed anyway.
> 
> Well... a lot of people are busy making production builds and testing
> them, and getting things ready to release on Tuesday.
> 
> If we pile more stuff on now, we're basically making a lot of people
> throw it all away and work through the weekend. Which is fine if it's
> necessary... but we should only really do if it if *is* necessary.
> 
> I'm inclined to suggest the gadget fixes can wait until Tuesday. 
> 

Furthermore, this is totally unmaintainable.  Can the shiny compiler
plugin infrastructure be used to mark the variable instead of the uses?
Yeah, I know that smells a lot like "volatile" but in this case it may
be a better thing to do.

Paolo

[MODERATED] Re: Cache load gadgets

[David Woodhouse]

On Mon, 2018-08-13 at 11:17 -0400, speck for Konrad Rzeszutek Wilk wrote:
> On Wed, Aug 08, 2018 at 10:34:29AM -0500, speck for Josh Poimboeuf wrote:
> > 
> > On Wed, Aug 08, 2018 at 07:53:16AM -0700, speck for Andi Kleen wrote:
> > > 
> > > On Wed, Aug 08, 2018 at 02:53:35PM +0100, speck for David Woodhouse wrote:
> > > > 
> > > > If we aren't disabling hyperthreading, then we get to lose sleep over
> > > > what a VM guest can trick us into loading into L1$ while a sibling runs
> > > > the attack.
> > > > 
> > > > For this they only need half a Spectre-v1 gadget; they only need to
> > > > load something into L1. We've tweaked our scanner and trawled through
> > > > the false positives, and we have a handful of patches which look
> > > > something like the one below.
> > > > 
> > > > Should we be getting these into the tree for next Tuesday's release?
> > > I don't think it's absolutely needed for Tuesday, can go in slower.
> > > 
> > > There will be a range of enhancements for L1TF beyond Tuesday anyways.
> >
> > Yet this patch is trivial, no harm in merging it now.  We already have a
> > similar one in RHEL (and several hundred more like it).
>
> Josh, and David,  would it be possible to include me (konrad.wilk@oracle.com),
> jamie.iles@oracle.com, liran.alon@oracle.com on anything that is sent
> after Tuesday?
> 
> As in we want to help in analyzing and reviewing them to minimize the
> effect on the upstream kernel for this particular nasty attack and the
> above folks aren't reading every LKML email :-). 

Even the gadget fixes are a band-aid. As Josh says, there are hundreds
of them. It's not massively feasible to close every gap, if this is
what we consider the attack surface to be.

I think we need to take a serious look at actually removing sensitive
pages from the physical 1:1 map so that they *can't* be accessed.

Maybe start with a way for kernel code to request an "otherwise
unmapped" page, which would need to be explicitly mapped for each
access (after ensuring that no potentially malicious guest is running).
Or which could be mapped into a per-process address, in some cases
(like a VMCS for a given vCPU).

We could look at doing something similar for anonymous user pages too,
and poking those out of the 1:1 map — at a cost of having to deal with
get_user_pages() et al.

It's distinctly non-trivial but it is potentially a way forward that
gives us some confidence that an attacker *can't* just read arbitrary
memory, pulled in via cache load gadgets or other means.

If we're going to have certain pages accessible only when a malicious
(or just different) guest can't be running on the sibling... perhaps we
also end up wanting coscheduling.

[MODERATED] Re: Cache load gadgets

[Josh Poimboeuf]

On Mon, Aug 13, 2018 at 11:17:27AM -0400, speck for Konrad Rzeszutek Wilk wrote:
> On Wed, Aug 08, 2018 at 10:34:29AM -0500, speck for Josh Poimboeuf wrote:
> > On Wed, Aug 08, 2018 at 07:53:16AM -0700, speck for Andi Kleen wrote:
> > > On Wed, Aug 08, 2018 at 02:53:35PM +0100, speck for David Woodhouse wrote:
> > > > If we aren't disabling hyperthreading, then we get to lose sleep over
> > > > what a VM guest can trick us into loading into L1$ while a sibling runs
> > > > the attack.
> > > > 
> > > > For this they only need half a Spectre-v1 gadget; they only need to
> > > > load something into L1. We've tweaked our scanner and trawled through
> > > > the false positives, and we have a handful of patches which look
> > > > something like the one below.
> > > > 
> > > > Should we be getting these into the tree for next Tuesday's release?
> > > 
> > > I don't think it's absolutely needed for Tuesday, can go in slower.
> > > 
> > > There will be a range of enhancements for L1TF beyond Tuesday anyways.
> > 
> > Yet this patch is trivial, no harm in merging it now.  We already have a
> > similar one in RHEL (and several hundred more like it).
> 
> Josh, and David,  would it be possible to include me (konrad.wilk@oracle.com),
> jamie.iles@oracle.com, liran.alon@oracle.com on anything that is sent
> after Tuesday?
> 
> As in we want to help in analyzing and reviewing them to minimize the
> effect on the upstream kernel for this particular nasty attack and the
> above folks aren't reading every LKML email :-). 

Sure, I asked Jeremy Cline (a Red Hatter whose been posting some of
these) to add you all to CC for future postings.  And I'll do so as
well.

Though there have been others doing these, so the best way to catch all
of them would be to create a mail filter for, e.g. "array_index_nospec"
and "barrier_nospec".

-- 
Josh

[MODERATED] Re: Cache load gadgets

[Greg KH]

On Mon, Aug 13, 2018 at 12:23:03PM -0500, speck for Josh Poimboeuf wrote:
> On Mon, Aug 13, 2018 at 11:17:27AM -0400, speck for Konrad Rzeszutek Wilk wrote:
> > On Wed, Aug 08, 2018 at 10:34:29AM -0500, speck for Josh Poimboeuf wrote:
> > > On Wed, Aug 08, 2018 at 07:53:16AM -0700, speck for Andi Kleen wrote:
> > > > On Wed, Aug 08, 2018 at 02:53:35PM +0100, speck for David Woodhouse wrote:
> > > > > If we aren't disabling hyperthreading, then we get to lose sleep over
> > > > > what a VM guest can trick us into loading into L1$ while a sibling runs
> > > > > the attack.
> > > > > 
> > > > > For this they only need half a Spectre-v1 gadget; they only need to
> > > > > load something into L1. We've tweaked our scanner and trawled through
> > > > > the false positives, and we have a handful of patches which look
> > > > > something like the one below.
> > > > > 
> > > > > Should we be getting these into the tree for next Tuesday's release?
> > > > 
> > > > I don't think it's absolutely needed for Tuesday, can go in slower.
> > > > 
> > > > There will be a range of enhancements for L1TF beyond Tuesday anyways.
> > > 
> > > Yet this patch is trivial, no harm in merging it now.  We already have a
> > > similar one in RHEL (and several hundred more like it).
> > 
> > Josh, and David,  would it be possible to include me (konrad.wilk@oracle.com),
> > jamie.iles@oracle.com, liran.alon@oracle.com on anything that is sent
> > after Tuesday?
> > 
> > As in we want to help in analyzing and reviewing them to minimize the
> > effect on the upstream kernel for this particular nasty attack and the
> > above folks aren't reading every LKML email :-). 
> 
> Sure, I asked Jeremy Cline (a Red Hatter whose been posting some of
> these) to add you all to CC for future postings.  And I'll do so as
> well.

And you all need to remember to cc: stable on these, I'm not seeing that
happen :(

> Though there have been others doing these, so the best way to catch all
> of them would be to create a mail filter for, e.g. "array_index_nospec"
> and "barrier_nospec".

I do search for these and not many are making it into the tree.
Certainly not 300...

thanks,

greg k-h

[MODERATED] Re: Cache load gadgets

[Luck, Tony]

On Mon, Aug 13, 2018 at 06:40:27PM +0200, speck for David Woodhouse wrote:
> Even the gadget fixes are a band-aid. As Josh says, there are hundreds
> of them. It's not massively feasible to close every gap, if this is
> what we consider the attack surface to be.
> 
> I think we need to take a serious look at actually removing sensitive
> pages from the physical 1:1 map so that they *can't* be accessed.
> 
> Maybe start with a way for kernel code to request an "otherwise
> unmapped" page, which would need to be explicitly mapped for each
> access (after ensuring that no potentially malicious guest is running).
> Or which could be mapped into a per-process address, in some cases
> (like a VMCS for a given vCPU).

Could we build on top of KPTI?  I.e. on entry to the kernel we currently
have nothing important mapped. If we added code to that to validate user
input (i.e. the "front-end" to system calls). If out of range values are
found then we have a fast -EINVAL back to the user. If everything is OK,
then we do the KPTI switch to the full kernel address space.

ioctl would be a pain, but most other system calls would just require
refactoring into a front-end validity check and a back end that does
the real work.

-Tony

[MODERATED] Re: Cache load gadgets

[Josh Poimboeuf]

On Mon, Aug 13, 2018 at 08:59:22PM +0200, speck for Greg KH wrote:
> On Mon, Aug 13, 2018 at 12:23:03PM -0500, speck for Josh Poimboeuf wrote:
> > On Mon, Aug 13, 2018 at 11:17:27AM -0400, speck for Konrad Rzeszutek Wilk wrote:
> > > On Wed, Aug 08, 2018 at 10:34:29AM -0500, speck for Josh Poimboeuf wrote:
> > > > On Wed, Aug 08, 2018 at 07:53:16AM -0700, speck for Andi Kleen wrote:
> > > > > On Wed, Aug 08, 2018 at 02:53:35PM +0100, speck for David Woodhouse wrote:
> > > > > > If we aren't disabling hyperthreading, then we get to lose sleep over
> > > > > > what a VM guest can trick us into loading into L1$ while a sibling runs
> > > > > > the attack.
> > > > > > 
> > > > > > For this they only need half a Spectre-v1 gadget; they only need to
> > > > > > load something into L1. We've tweaked our scanner and trawled through
> > > > > > the false positives, and we have a handful of patches which look
> > > > > > something like the one below.
> > > > > > 
> > > > > > Should we be getting these into the tree for next Tuesday's release?
> > > > > 
> > > > > I don't think it's absolutely needed for Tuesday, can go in slower.
> > > > > 
> > > > > There will be a range of enhancements for L1TF beyond Tuesday anyways.
> > > > 
> > > > Yet this patch is trivial, no harm in merging it now.  We already have a
> > > > similar one in RHEL (and several hundred more like it).
> > > 
> > > Josh, and David,  would it be possible to include me (konrad.wilk@oracle.com),
> > > jamie.iles@oracle.com, liran.alon@oracle.com on anything that is sent
> > > after Tuesday?
> > > 
> > > As in we want to help in analyzing and reviewing them to minimize the
> > > effect on the upstream kernel for this particular nasty attack and the
> > > above folks aren't reading every LKML email :-). 
> > 
> > Sure, I asked Jeremy Cline (a Red Hatter whose been posting some of
> > these) to add you all to CC for future postings.  And I'll do so as
> > well.
> 
> And you all need to remember to cc: stable on these, I'm not seeing that
> happen :(

Ok, I relayed that message to Jeremy.

> > Though there have been others doing these, so the best way to catch all
> > of them would be to create a mail filter for, e.g. "array_index_nospec"
> > and "barrier_nospec".
> 
> I do search for these and not many are making it into the tree.
> Certainly not 300...

It will take time.  Jeremy has been out the last few weeks and I've been
preoccupied with L1TF.

And BTW, it will be a lot more than 300...

-- 
Josh

[MODERATED] Re: Cache load gadgets

[Linus Torvalds]

On Mon, 13 Aug 2018, speck for Josh Poimboeuf wrote:
> > 
> > I do search for these and not many are making it into the tree.
> > Certainly not 300...
> 
> It will take time.  Jeremy has been out the last few weeks and I've been
> preoccupied with L1TF.
> 
> And BTW, it will be a lot more than 300...

I do not believe I want to see these idiotic cache gadgets annotated or 
"fixed" in the kernel. Definitely not by hand, and I doubt it's even 
fixable in a compiler.

Just give it up. At some point, there is nothing we can do. It's trivial 
to find code that loads something off a random pointer and populates a 
cacheline. It's not a "gadget", it's just a fact of life.

            Linus

[MODERATED] Re: Cache load gadgets

[Josh Poimboeuf]

On Mon, Aug 13, 2018 at 01:20:48PM -0700, speck for Linus Torvalds wrote:
> 
> 
> On Mon, 13 Aug 2018, speck for Josh Poimboeuf wrote:
> > > 
> > > I do search for these and not many are making it into the tree.
> > > Certainly not 300...
> > 
> > It will take time.  Jeremy has been out the last few weeks and I've been
> > preoccupied with L1TF.
> > 
> > And BTW, it will be a lot more than 300...
> 
> I do not believe I want to see these idiotic cache gadgets annotated or 
> "fixed" in the kernel. Definitely not by hand, and I doubt it's even 
> fixable in a compiler.
> 
> Just give it up. At some point, there is nothing we can do. It's trivial 
> to find code that loads something off a random pointer and populates a 
> cacheline. It's not a "gadget", it's just a fact of life.

Well, but we're talking about Spectre v1 here.  It's not just some
random pointer; the user can force a speculative load from *any*
address.

Are you suggesting we just give up on Spectre v1?

-- 
Josh

[MODERATED] Re: Cache load gadgets

[Linus Torvalds]

On Mon, 13 Aug 2018, speck for Josh Poimboeuf wrote:
> 
> Well, but we're talking about Spectre v1 here.

No we're not. We're just talking about loading an arbitrary cacheline, 
without any actual data leak. There is no spectre there.

That *will* continue to happen. Exactly because of what I said: those 
things are *everywhere*. And it's not a leak on its own. It just fills the 
cacheline.

The data leak is due to non-spectre-v1 bugs, ie "smt + L1TF".

That's the one people should fix, or just decide "I don't care".

Trying to avoid filling the L1 cache is a pointless stupid exercize. Don't 
do it. 

                Linus

[MODERATED] Re: Cache load gadgets

[David Woodhouse]

On Mon, 2018-08-13 at 13:56 -0700, speck for Linus Torvalds wrote:
> 
> On Mon, 13 Aug 2018, speck for Josh Poimboeuf wrote:
> > 
> > Well, but we're talking about Spectre v1 here.
> 
> No we're not. We're just talking about loading an arbitrary cacheline, 
> without any actual data leak. There is no spectre there.
> 
> That *will* continue to happen. Exactly because of what I said: those 
> things are *everywhere*. And it's not a leak on its own. It just fills the 
> cacheline.
> 
> The data leak is due to non-spectre-v1 bugs, ie "smt + L1TF".
> 
> That's the one people should fix, or just decide "I don't care".
> 
> Trying to avoid filling the L1 cache is a pointless stupid exercize. Don't 
> do it.

Right. The only way to stop that — assuming we do care — is to take
certain pages out of the kernel memory map altogether so they *cannot*
be in L1$ while some other potentially malicious process/guest is
running.

[MODERATED] Re: Cache load gadgets

[Josh Poimboeuf]

On Mon, Aug 13, 2018 at 01:56:49PM -0700, speck for Linus Torvalds wrote:
> 
> 
> On Mon, 13 Aug 2018, speck for Josh Poimboeuf wrote:
> > 
> > Well, but we're talking about Spectre v1 here.
> 
> No we're not. We're just talking about loading an arbitrary cacheline, 
> without any actual data leak. There is no spectre there.
> 
> That *will* continue to happen. Exactly because of what I said: those 
> things are *everywhere*. And it's not a leak on its own. It just fills the 
> cacheline.
> 
> The data leak is due to non-spectre-v1 bugs, ie "smt + L1TF".
> 
> That's the one people should fix, or just decide "I don't care".
> 
> Trying to avoid filling the L1 cache is a pointless stupid exercize. Don't 
> do it. 

Ok, so I think the two discussions got mixed up a bit and I
misunderstood your point.  The patch which was posted earlier was a
Spectre v1 fix, but I believe it was suggested that it has a higher
priority because of L1TF.

Taking L1TF out of the equation, it sounds like you have no objections
to the Spectre v1 fixes.  We will continue doing those...

-- 
Josh

[MODERATED] Re: Cache load gadgets

[Linus Torvalds]

On Mon, 13 Aug 2018, speck for Josh Poimboeuf wrote:
> 
> Ok, so I think the two discussions got mixed up a bit and I
> misunderstood your point.  The patch which was posted earlier was a
> Spectre v1 fix, but I believe it was suggested that it has a higher
> priority because of L1TF.

Actually, no. The patch that was posted earlier (although maybe there were 
multiple ones and I'm only remembering one) was *not* a spectre-v1 fix, it 
was literally just a "don't speculatively fill cacheline using a 
user-supplied index" fix.

Noiw, it used the same "nonspec" helpers that the spectre-v1 fixes do use, 
so it *looks* the same, but it didn't actually have the required 
indirection through memory to turn the loaded cacheline into a data leak.

Unless I read the patch wrong, and/or mis-reemember.

So it _looked_ like "spectre-v1", but it was really just a "cachelines get 
filled by various things".

Note that some of those things that fill cachelines we don't have any 
control over. Do  you know how the L2 -> L1 prefetch logic works? There 
are literally papers about improving dependent load performance by having 
logic that makes a prefetcher kick in when you see a value being loaded 
that *looks* like a pointer. 

Seriously - this is things like the memory subsystem doing cache 
prefetching on its own (with no instructions - just based on earlier 
patterns and the size of the load that it can see) based on what kind of 
data it sees being loaded from memory, _just_in_case_ it's a pointer.

Is that just an academic paper? Maybe. And maybe not. I don't know. 

My point is, for all we know, the hardware will prefetch things just 
because we load an 8-byte value from user space. Maybe the value is 
actuall just a bitmap, but if user space *crafts* it to look like a kernel 
pointer, and resources are idle, and maybe the prefetch logic kicks in and 
we'll be populating cachelines.

No, I don't actually think Intel or AMD do that. But it's literally 
possible. Filling caches down speculative paths is going to happen 
everywhere. 

> Taking L1TF out of the equation, it sounds like you have no objections
> to the Spectre v1 fixes.  We will continue doing those...

The real Spectre-v1 ones where we have a "load a speculative value from 
memory, then use that value as an index to load/store another value, so 
that people can figuyre out the first speculative value" by all means.

So the "full spectre-v1" case we should very much strive to fix. It's the 
"half-spectre" that I think is a dead end because just loading a cacheline 
can happen so many ways.

             Linus

[MODERATED] Re: Cache load gadgets

[Josh Poimboeuf]

On Mon, Aug 13, 2018 at 02:41:31PM -0700, speck for Linus Torvalds wrote:
> > Taking L1TF out of the equation, it sounds like you have no objections
> > to the Spectre v1 fixes.  We will continue doing those...
> 
> The real Spectre-v1 ones where we have a "load a speculative value from 
> memory, then use that value as an index to load/store another value, so 
> that people can figuyre out the first speculative value" by all means.
> 
> So the "full spectre-v1" case we should very much strive to fix. It's the 
> "half-spectre" that I think is a dead end because just loading a cacheline 
> can happen so many ways.

Well, we've only been fixing half spectres because we don't know how to
reliably find full spectres.

And half spectre isn't "cachelines get filled by various things".  It's
"cachelines are completely attacker controlled", because the array index
can be *any* value, positive or negative.  So it doesn't just leak some
random adjacent data, it leaks whatever the attacker wants in the L1.

There are already about 75 of these half spectre fixes in the kernel
already.

-- 
Josh

[MODERATED] Re: Cache load gadgets

[Linus Torvalds]

On Mon, 13 Aug 2018, speck for Josh Poimboeuf wrote:
> 
> There are already about 75 of these half spectre fixes in the kernel
> already.

... and we should stop with the crazy voodoo programming then.

             Linus

[MODERATED] Re: Cache load gadgets

[Julian Stecklina]

[-- Attachment #1: Type: text/plain, Size: 2104 bytes --]

On 08/13/2018 09:27 PM, speck for Luck, Tony wrote:
> On Mon, Aug 13, 2018 at 06:40:27PM +0200, speck for David Woodhouse wrote:
>> Even the gadget fixes are a band-aid. As Josh says, there are hundreds
>> of them. It's not massively feasible to close every gap, if this is
>> what we consider the attack surface to be.
>>
>> I think we need to take a serious look at actually removing sensitive
>> pages from the physical 1:1 map so that they *can't* be accessed.
>>
>> Maybe start with a way for kernel code to request an "otherwise
>> unmapped" page, which would need to be explicitly mapped for each
>> access (after ensuring that no potentially malicious guest is running).
>> Or which could be mapped into a per-process address, in some cases
>> (like a VMCS for a given vCPU).
> Could we build on top of KPTI?  I.e. on entry to the kernel we currently
> have nothing important mapped. If we added code to that to validate user
> input (i.e. the "front-end" to system calls). If out of range values are
> found then we have a fast -EINVAL back to the user. If everything is OK,
> then we do the KPTI switch to the full kernel address space.
>
> ioctl would be a pain, but most other system calls would just require
> refactoring into a front-end validity check and a back end that does
> the real work.

That's unfortunately not sufficient. You could still end up pulling
arbitrary memory in the cache by misspeculating in the kernel. In the
long term, fixing gadgets and making all input validation
"speculation-safe" is a losing battle.

The viable long term plan is to remove as many mappings from the global
kernel address space as possible. Specifically, while being in the
context of process A, memory of process B should not be visible in the
kernel. If that's the case, no gadget can speculatively pull B's memory
into the L1 cache. At the moment, this is not the case. The big problem
is the 1:1 mapping of all physical memory. And after that, there is disk
cache, userspace register state or vCPU state floating around in the kernel.

Julian

[MODERATED] Re: Cache load gadgets

[Andi Kleen]

> That's unfortunately not sufficient. You could still end up pulling
> arbitrary memory in the cache by misspeculating in the kernel. In the

You claim that, but where are the exploits?

There is lots of theoretical speculation on how attacks could
be chained, but in practice everyone who tried it found it
quite difficult. 

All the newer attacks are usually somewhat fragile to start
with, and if you start combining them they become more and more
difficult. At some point theoretical turns into impractical.

> The viable long term plan is to remove as many mappings from the global

viable?

> kernel address space as possible. Specifically, while being in the
> context of process A, memory of process B should not be visible in the
> kernel.

You're basically proposing a major rewrite on how Linux works, with
sprinkling kmap everywhere. kmap on 32bit was an absolutely nightmare,
and it was only used in very limited places.

I doubt there is any practical way to pull this off given the size of the
code base, even ignoring the likely performance loss.

-Andi

[MODERATED] Re: Cache load gadgets

[Julian Stecklina]

[-- Attachment #1: Type: text/plain, Size: 1010 bytes --]

On 08/14/2018 05:00 PM, speck for Andi Kleen wrote:
>> That's unfortunately not sufficient. You could still end up pulling
>> arbitrary memory in the cache by misspeculating in the kernel. In the
> You claim that, but where are the exploits?
>
> There is lots of theoretical speculation on how attacks could
> be chained, but in practice everyone who tried it found it
> quite difficult. 
>
> All the newer attacks are usually somewhat fragile to start
> with, and if you start combining them they become more and more
> difficult. At some point theoretical turns into impractical.

L1TF in combination with a suitable cache load gadget on a HT-enabled
system is very practical. As we are not going to fix pure cache load
gadgets in the kernel, this leaves two options for the security minded:
a) Turn off HT and take a large performance hit for some workloads or b)
carry their own patches on top of upstream.

I guess there are quite some people in the b) camp at the moment.

Julian

[MODERATED] Re: Cache load gadgets

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

On 17/08/2018 11:11, speck for Julian Stecklina wrote:
> On 08/14/2018 05:00 PM, speck for Andi Kleen wrote:
>>> That's unfortunately not sufficient. You could still end up pulling
>>> arbitrary memory in the cache by misspeculating in the kernel. In the
>> You claim that, but where are the exploits?
>>
>> There is lots of theoretical speculation on how attacks could
>> be chained, but in practice everyone who tried it found it
>> quite difficult. 
>>
>> All the newer attacks are usually somewhat fragile to start
>> with, and if you start combining them they become more and more
>> difficult. At some point theoretical turns into impractical.
> 
> L1TF in combination with a suitable cache load gadget on a HT-enabled
> system is very practical. As we are not going to fix pure cache load
> gadgets in the kernel, this leaves two options for the security minded:
> a) Turn off HT and take a large performance hit for some workloads or b)
> carry their own patches on top of upstream.
> 
> I guess there are quite some people in the b) camp at the moment.

KVM is special in that it has the EPT variant of L1TF, and fixing the
most blatant cache load gadgets there is probably feasible.  Doing it in
general is hard and unmaintainable though.  At least we should use
sparse static analysis to track those tainted values.

Paolo

[MODERATED] Re: Cache load gadgets

[Norbert Manthey]

[-- Attachment #1.1: Type: text/plain, Size: 1819 bytes --]



On 08/17/2018 11:18 AM, speck for Paolo Bonzini wrote:
> On 17/08/2018 11:11, speck for Julian Stecklina wrote:
>> On 08/14/2018 05:00 PM, speck for Andi Kleen wrote:
>>>> That's unfortunately not sufficient. You could still end up pulling
>>>> arbitrary memory in the cache by misspeculating in the kernel. In the
>>> You claim that, but where are the exploits?
>>>
>>> There is lots of theoretical speculation on how attacks could
>>> be chained, but in practice everyone who tried it found it
>>> quite difficult. 
>>>
>>> All the newer attacks are usually somewhat fragile to start
>>> with, and if you start combining them they become more and more
>>> difficult. At some point theoretical turns into impractical.
>>
>> L1TF in combination with a suitable cache load gadget on a HT-enabled
>> system is very practical. As we are not going to fix pure cache load
>> gadgets in the kernel, this leaves two options for the security minded:
>> a) Turn off HT and take a large performance hit for some workloads or b)
>> carry their own patches on top of upstream.
>>
>> I guess there are quite some people in the b) camp at the moment.
> 
> KVM is special in that it has the EPT variant of L1TF, and fixing the
> most blatant cache load gadgets there is probably feasible.  Doing it in
> general is hard and unmaintainable though.  At least we should use
> sparse static analysis to track those tainted values.
> 
> Paolo
> 

Do you mean you want to run the smatch tool and raise spectre related
candidates, e.g. similarly to what syzbot is doing, and run that against
the development branches, as well as selected stable branches?

Is there anybody doing continuous analysis like that already? And if
not, what would be the expectations for such a system?

Best,
Norbert


[-- Attachment #2.1: Type: text/plain, Size: 244 bytes --]

Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

[-- Attachment #2.2: Type: text/html, Size: 291 bytes --]

[MODERATED] Re: Cache load gadgets

[Julian Stecklina]

[-- Attachment #1: Type: text/plain, Size: 1791 bytes --]

On 08/17/2018 11:18 AM, speck for Paolo Bonzini wrote:
> On 17/08/2018 11:11, speck for Julian Stecklina wrote:
>> L1TF in combination with a suitable cache load gadget on a HT-enabled
>> system is very practical. As we are not going to fix pure cache load
>> gadgets in the kernel, this leaves two options for the security minded:
>> a) Turn off HT and take a large performance hit for some workloads or b)
>> carry their own patches on top of upstream.
>>
>> I guess there are quite some people in the b) camp at the moment.
> KVM is special in that it has the EPT variant of L1TF, and fixing the
> most blatant cache load gadgets there is probably feasible.  Doing it in
> general is hard and unmaintainable though.  At least we should use
> sparse static analysis to track those tainted values.

In general, I agree. For our usecase, we would also need to fix parts of
VFIO and maybe other common system call entry paths. It's just
labor-intense and error prone.

The last days, I have been experimenting with making certain KVM memory
allocations process-local in the kernel, so they cannot be pulled in
with cache load gadgets from other processes in the first place. It
looks promising so far. Of course, this doesn't address the big linear
memory mapping of physical memory, but it's a start.

I'm also hoping someone picks up the speculative load hardening compiler
pass in clang and fixes it for the kernel. Maybe the performance impact
is not so bad, if some hot code paths are manually checked. The problem
is mostly that they assume that RSP is positive, which is obviously not
the case in the kernel, but it looks fixable:
https://docs.google.com/document/d/1wwcfv3UV9ZnZVcGiGuoITT_61e_Ko3TmoCS3uXLcJR0/edit#heading=h.phdehs44eom6


Julian

[MODERATED] Re: Cache load gadgets

[Konrad Rzeszutek Wilk]

On Fri, Aug 17, 2018 at 10:00:13PM +0200, speck for Julian Stecklina wrote:
> On 08/17/2018 11:18 AM, speck for Paolo Bonzini wrote:
> > On 17/08/2018 11:11, speck for Julian Stecklina wrote:
> >> L1TF in combination with a suitable cache load gadget on a HT-enabled
> >> system is very practical. As we are not going to fix pure cache load
> >> gadgets in the kernel, this leaves two options for the security minded:
> >> a) Turn off HT and take a large performance hit for some workloads or b)
> >> carry their own patches on top of upstream.
> >>
> >> I guess there are quite some people in the b) camp at the moment.
> > KVM is special in that it has the EPT variant of L1TF, and fixing the
> > most blatant cache load gadgets there is probably feasible.  Doing it in
> > general is hard and unmaintainable though.  At least we should use
> > sparse static analysis to track those tainted values.
> 
> In general, I agree. For our usecase, we would also need to fix parts of
> VFIO and maybe other common system call entry paths. It's just
> labor-intense and error prone.
> 
> The last days, I have been experimenting with making certain KVM memory
> allocations process-local in the kernel, so they cannot be pulled in
> with cache load gadgets from other processes in the first place. It
> looks promising so far. Of course, this doesn't address the big linear
> memory mapping of physical memory, but it's a start.

Just in case you are not aware this was posted some time ago:

https://marc.info/?l=linux-kernel&m=151923574116661&w=2

Any chance you could drop a git bundle of what you have done so far?

> 
> I'm also hoping someone picks up the speculative load hardening compiler
> pass in clang and fixes it for the kernel. Maybe the performance impact
> is not so bad, if some hot code paths are manually checked. The problem
> is mostly that they assume that RSP is positive, which is obviously not
> the case in the kernel, but it looks fixable:
> https://docs.google.com/document/d/1wwcfv3UV9ZnZVcGiGuoITT_61e_Ko3TmoCS3uXLcJR0/edit#heading=h.phdehs44eom6
>

Let me ask the compiler folks in Oracle. 
> 
> Julian
> 
>