[MODERATED] ARCH_CAPABILITIES note

[MODERATED] ARCH_CAPABILITIES note

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 643 bytes --]

Hi all,

I've pinged Intel asking them to clarify ARCH_CAPABILITIES to ensure
that we don't open up exposure in future silicon to past exploits.

One topic I can't touch on here, but another is the SSBD. So for v4,
we're currently (probably) safe (enough?) against kernel stack exploit
by virtue of the serializing lfence in the retpoline entry code. BUT if
future silicon says "hey, I'm fine" and we disable the retpoline
mitigation, we'll /still/ need to do the lfence on kernel entry.

There are similar others where we just need to be /really/ careful.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

[MODERATED] Re: ARCH_CAPABILITIES note

[Dave Hansen]

[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

On 06/05/2018 09:41 AM, speck for Jon Masters wrote:
> One topic I can't touch on here, but another is the SSBD. So for v4,
> we're currently (probably) safe (enough?) against kernel stack exploit
> by virtue of the serializing lfence in the retpoline entry code. BUT if
> future silicon says "hey, I'm fine" and we disable the retpoline
> mitigation, we'll /still/ need to do the lfence on kernel entry.

Totally agree that we need to be really careful about mitigations that
interact between the variants.  But, I have a bunch of questions.

Just to be clear: you are talking about the *syscall* entry path, and
the retpoline on the indirect call to the syscall table entry, right?

Is there a reason you are not concerned about the interrupt entry paths
for SSB?

Wouldn't this be an issue already on distros (not mainline) that disable
retpoline and use IBRS?

The retpoline LFENCE is only speculative and not ever retired, so I'd
first ask: Is a *speculative*, non-retired LFENCE a sufficient SSB
mitigation today, and will it remain that way?  Agreed?

Who inside Intel were you asking, btw?

[MODERATED] Re: ARCH_CAPABILITIES note

[Linus Torvalds]

On Tue, 5 Jun 2018, speck for Dave Hansen wrote:
> 
> Is there a reason you are not concerned about the interrupt entry paths
> for SSB?

Interrupts and exceptions are serializing, both on entry and exit (iret).
 
Only the fast system call and  sysret instructions are not already 
guaranteed to be much stronger than lfence.

                Linus

[MODERATED] Re: ARCH_CAPABILITIES note

[David Woodhouse]

On Tue, 2018-06-05 at 12:41 -0400, speck for Jon Masters wrote:
> Hi all,
> 
> I've pinged Intel asking them to clarify ARCH_CAPABILITIES to ensure
> that we don't open up exposure in future silicon to past exploits.
> 
> One topic I can't touch on here, but another is the SSBD. So for v4,
> we're currently (probably) safe (enough?) against kernel stack exploit
> by virtue of the serializing lfence in the retpoline entry code. BUT if
> future silicon says "hey, I'm fine" and we disable the retpoline
> mitigation, we'll /still/ need to do the lfence on kernel entry.
> 
> There are similar others where we just need to be /really/ careful.

I'm also concerned about the backward compatibility of SPEC_CTRL.

If a VM host exposes IBRS+STIPB to guests today by trapping the first
access and then making it passthrough, and (rightly) faults on any
other bits being set, then subsequent writes by the guest (when it's no
longer trapped) will let those other bits be set.

We already have this with the SSBD bit on pre-4.17 kernels. Although in
4.17 I note we actually let the guest set the SSBD bit on the first
access without faulting, even when it's not supposed to be present in
the guest (and in fact even on microcode where it *isn't* present at
all).

We could do with a mask of which bits the guest is allowed to set (and
perhaps for future SPEC_CTRL bits, a different mask of which bits the
guest is allowed to *clear*, to ensure the hypervisor remains
protected?)

[MODERATED] Re: ARCH_CAPABILITIES note

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 2381 bytes --]

On 06/05/2018 02:27 PM, speck for Dave Hansen wrote:
> On 06/05/2018 09:41 AM, speck for Jon Masters wrote:
>> One topic I can't touch on here, but another is the SSBD. So for v4,
>> we're currently (probably) safe (enough?) against kernel stack exploit
>> by virtue of the serializing lfence in the retpoline entry code. BUT if
>> future silicon says "hey, I'm fine" and we disable the retpoline
>> mitigation, we'll /still/ need to do the lfence on kernel entry.
> 
> Totally agree that we need to be really careful about mitigations that
> interact between the variants.  But, I have a bunch of questions.

:)

> Just to be clear: you are talking about the *syscall* entry path, and
> the retpoline on the indirect call to the syscall table entry, right?

Yup

> Is there a reason you are not concerned about the interrupt entry paths
> for SSB?

Linus got there first

> Wouldn't this be an issue already on distros (not mainline) that disable
> retpoline and use IBRS?

Yeah, it would. Thanks for the reminder. Josh, Waiman, I'll ping
internally about getting that bit cleaned up in our standup.

> The retpoline LFENCE is only speculative and not ever retired

It's serializing though. The problem is not so much the use of the
lfence (provided the SDM doesn't change) but that it only applies right
up /until/ syscall dispatch, but not once the syscall handler has been
dispatched. The handler might do something dangerous. I audited a few,
but there's a lot more work that needs to be done there I think.

I had a lot of similar dialogue with IBM during the embargo process,
which is what lead to the upstream stf patches that were posted. They do
that precisely to protect the kernel stack. Similarly, this is why Arm
leave their mitigation on generally in the kernel (although for Arm,
we'll ultimately be doing things a little differently downstream unless
they also pick up the optimizations that exist for various uarches).

> Who inside Intel were you asking, btw?

Ronak, Asit, and the core team folks. I'll forward you a mail that
mentions this and another attack vector also currently coincidentally
fixed by the retpoline code, but not being tracked. I'm not at liberty
to (yet) share the details of that other attack here but Intel know.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

Re: ARCH_CAPABILITIES note

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 1974 bytes --]

On Tue, 5 Jun 2018, speck for Jon Masters wrote:
> On 06/05/2018 02:27 PM, speck for Dave Hansen wrote:
> > On 06/05/2018 09:41 AM, speck for Jon Masters wrote:
> >> One topic I can't touch on here, but another is the SSBD. So for v4,
> >> we're currently (probably) safe (enough?) against kernel stack exploit
> >> by virtue of the serializing lfence in the retpoline entry code. BUT if
> >> future silicon says "hey, I'm fine" and we disable the retpoline
> >> mitigation, we'll /still/ need to do the lfence on kernel entry.
> > 
> > Totally agree that we need to be really careful about mitigations that
> > interact between the variants.  But, I have a bunch of questions.
> 
> :)
> 
> > Just to be clear: you are talking about the *syscall* entry path, and
> > the retpoline on the indirect call to the syscall table entry, right?
> 
> Yup

And how is that supposed to protect anything?

The attack happens further down the syscall and the store target can be
anything: kernel stack, function pointers or just a variable. If the load
gadget gets speculated on any of those, you lost.

Te LFENCE in the entry code is not protecting any of these, except a
straight return to user space speculation.

The other retpolines which are injected by GCC might mitigate some of the
store target species, but I'm not seeing how they guarantee to mitigate all
of them.

The only sane option is to prevent the speculative out of bounds write when
the index is user space controlled.

> Ronak, Asit, and the core team folks. I'll forward you a mail that
> mentions this and another attack vector also currently coincidentally
> fixed by the retpoline code, but not being tracked. I'm not at liberty
> to (yet) share the details of that other attack here but Intel know.

Has the next secrecy foofaraw started or did I miss that the previous one
never ended?

/me shrugs and mutters the Götz von Berlichingen Euphemism.

Thanks,

	tglx

[MODERATED] Re: ARCH_CAPABILITIES note

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 2508 bytes --]

On 06/05/2018 03:19 PM, speck for Thomas Gleixner wrote:
> On Tue, 5 Jun 2018, speck for Jon Masters wrote:
>> On 06/05/2018 02:27 PM, speck for Dave Hansen wrote:
>>> On 06/05/2018 09:41 AM, speck for Jon Masters wrote:
>>>> One topic I can't touch on here, but another is the SSBD. So for v4,
>>>> we're currently (probably) safe (enough?) against kernel stack exploit
>>>> by virtue of the serializing lfence in the retpoline entry code. BUT if
>>>> future silicon says "hey, I'm fine" and we disable the retpoline
>>>> mitigation, we'll /still/ need to do the lfence on kernel entry.
>>>
>>> Totally agree that we need to be really careful about mitigations that
>>> interact between the variants.  But, I have a bunch of questions.
>>
>> :)
>>
>>> Just to be clear: you are talking about the *syscall* entry path, and
>>> the retpoline on the indirect call to the syscall table entry, right?
>>
>> Yup
> 
> And how is that supposed to protect anything?
> 
> The attack happens further down the syscall and the store target can be
> anything: kernel stack, function pointers or just a variable. If the load
> gadget gets speculated on any of those, you lost.

Agree.

> Te LFENCE in the entry code is not protecting any of these, except a
> straight return to user space speculation.

Right. My earlier concern was simply the trampoline entry code saving
userspace registers onto the stack, etc. As I said, a lot more work
needs doing on the syscall handlers to understand what should be done.
It's not just out of bounds checks, it's also anywhere you might be able
to do a stack attack where some user controlled value is stored to the
stack then some other safe value is stored, followed by a second load.

>> Ronak, Asit, and the core team folks. I'll forward you a mail that
>> mentions this and another attack vector also currently coincidentally
>> fixed by the retpoline code, but not being tracked. I'm not at liberty
>> to (yet) share the details of that other attack here but Intel know.
> 
> Has the next secrecy foofaraw started or did I miss that the previous one
> never ended?

There are a few vulnerability reports I'm aware of which I don't own.
They're not Intel controlled, and we've asked that these be shared. The
one that's relevant to this thread is one we can hopefully discuss soon,
but happens to be mitigated by the existing Spectre-v2 fixes in Linux.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

[MODERATED] Re: ARCH_CAPABILITIES note

[Linus Torvalds]

On Tue, 5 Jun 2018, speck for Thomas Gleixner wrote:
> 
> Te LFENCE in the entry code is not protecting any of these, except a
> straight return to user space speculation.

I think people are just talking about the plain SSB attack where a user 
store in the store buffer is bypassed, and the kernel reads an old stale 
value.

I do not actually believe this is an attack at all. Users can use threads 
to show kernel multiple values at any kernel memory read time, so the 
attack is bogus.

The other possible store buffer attack is one that does *NOT* bypass the 
store buffer, but instead depends on the kernel reading a "trusted" kernel 
value that was in the store buffer because user space polluted *kernel* 
data, and the store is still speculative, and has not been killed yet 
(because Intel stupidly kills memory ops in _retirement_, not at execution 
time, which is why Meltdown happened).

IOW, I believe a user process could in theory do

 - slow load followed by conditional that is mispredicted

 - within the covers of that slow load and mispredict, do:

    - speculative store to kernel space

    - speculative system call

    - kernel will now *HIT* in the store buffer and see the user-supplied 
      value for something that should be safe

but I'm not sure that the above is actually possible.

But *if* the above is possible, then it allows user space to possibly 
pollute some very early kernel data structure reads, if user space just 
knows the addresses.

And by "very early kernel data striucture reads" I mean things like the 
percpu stack pointer read in the entry trampoline. But we have those %cr3 
movement things there that probably stop things like that.

Anyway, that happens even before the retpoline.

And again, I think this whole thing is confused by there being so many 
*different* issues. The store buffer *bypass*, the possible store buffer 
*poisoning*, the possible interactions with loading return addresses from 
stack as a result of such poisoning, the odd (and still not fully 
explained) second version of "v1" that I haven't seen the whitepaper for, 
etc etc etc.

I really would prefer that people give way more concrete examples, and 
stop with the handwaving. *Anything* can happen when you combine four 
different security issues. Can we agree that we always give real hard 
examples, not some hand-wavy "read after read after store buffer bypass"?

                  Linus

[MODERATED] Re: ARCH_CAPABILITIES note

[Dave Hansen]

[-- Attachment #1: Type: text/plain, Size: 1414 bytes --]

On 06/05/2018 11:48 AM, speck for Jon Masters wrote:
> On 06/05/2018 02:27 PM, speck for Dave Hansen wrote:
>> The retpoline LFENCE is only speculative and not ever retired
> 
> It's serializing though. The problem is not so much the use of the
> lfence (provided the SDM doesn't change) but that it only applies right
> up /until/ syscall dispatch, but not once the syscall handler has been
> dispatched. The handler might do something dangerous. I audited a few,
> but there's a lot more work that needs to be done there I think.

Just trying to be precise: Are you saying that LFENCE is an
architecturally "serializing instruction", or that it shares some
properties will fully "serializing instructions"?

Also, do we agree that the speculative retpoline LFENCE has different
behavior than a retired LFENCE?

From my perspective, the speculative LFENCE only has an impact on the
loads in that _particular_ speculative execution path.  It has no
defined impact on the retired execution path, or *other* speculation
paths that can occur.

The processor could easily speculate into an LFENCE, decide it does not
want to go down that speculative path, and restart speculating elsewhere
as if it had never seen the LFENCE.

BTW, I this is related to, but separate from the question of the
mitigation value of the existing retpoline LFENCE for mitigating things
other than Spectre v2.

[MODERATED] Re: ARCH_CAPABILITIES note

[Linus Torvalds]

On Wed, 6 Jun 2018, speck for Dave Hansen wrote:
> 
> Just trying to be precise: Are you saying that LFENCE is an
> architecturally "serializing instruction", or that it shares some
> properties will fully "serializing instructions"?

Heh. Bring it up with the Intel documentation people.

Intel should really stop using "serializing instruction" entirely, 
because the rules are so bad, and so odd, and so pointless.

A _true_ serializing instruction will serialize even instruction fetch and 
make sure any I$ access is coherent with D$ etc. And no, lfence does not 
do that.

But lfence - despite the name and the history of being a memory ordering 
device - does what you care about if you care about instruction execution 
ordering: it will block execution until all preceding instructions have 
retired.

Intel really does a horrible job of documenting these kinds of 
serializations. They fixed their memory ordering documentation, but any 
ordering that is not about _normal_ cached memory ordering is imnsho very 
very lacking.

They have some language about non-cached ordering (ie PCIe and uncached 
RAM with WC/WT/whatever), but it's not at all as clear or well architected 
as their memory ordering.

And when it comes to "serializing" instructions that actually impact 
instruction flow, it's a complete toss-up. "lfence" isn't serializing, but 
it serializes execution. But Intel at no point really talks about what 
"serializing" means, although they in a few places talk about when you 
need to use a serializing instruction (but even then there are almost 
always exceptions).

So you need a "real" serializing instruction if you want to synchronize 
with the TLB walker, or synchronize with another CPU modifying your 
instructions under you - aka cross-modifying code - or even the _same_ CPU 
doing self-modifying code through another virtual address. 

And iirc there's some verbiage how you need a "real" serializing 
instruction if you change execution modes that affect decoding (ie real 
mode to protected mode), except most of the time you don't, and just a 
"jump" is sufficient for legacy reasons.

And again, lfence is *not* a serializing instruction, but lfence *does* 
stop execution until all preceding instructions have completed. Note how 
that is only about the front-end, and also note that this is not really 
what lfence was actually originally specified for.

Also, afaik, lfence doesn't necessarily wait for the store buffer to 
drain, but since it waits for all preceding stores to be stable, it at 
least guarantees that stores _are_ in the store buffer, and there are no 
speculative store buffer entries, so it avoids the store buffer bypass 
(and the SB poisoning that I haven't actually seen verified if Intel has 
problems with or not).

And no - none of the above is really spelled out very well anywhere that I 
can find. You need to just search for "serializing" in the SDK and see 
where Intel says you need a serializing instruction.

                    Linus

[MODERATED] Re: ARCH_CAPABILITIES note

[Linus Torvalds]

On Wed, 6 Jun 2018, speck for Dave Hansen wrote:
> 
> From my perspective, the speculative LFENCE only has an impact on the
> loads in that _particular_ speculative execution path.  It has no
> defined impact on the retired execution path, or *other* speculation
> paths that can occur.
> 
> The processor could easily speculate into an LFENCE, decide it does not
> want to go down that speculative path, and restart speculating elsewhere
> as if it had never seen the LFENCE.

.. and no, I don't know of anything that *guarantees* that the lfence in 
the retpoline sequence effectively ends up always serializing execution as 
if it was in the non-speculative sequence.

The retpoline code sequence is obviously designed to always get to the 
lfence, with the exceptions being:

 (a) cpu's that don't speculate or don't have an RSB at all (but then 
     retpoline wouldn't be used)

 (b) interrupt at just the right point messing up the RSB that retpoline 
     sets up (but then the interrupt itself is a serializing event)

So I think we can basically agree on the lfence being "started".

But your worry is what if the lfence was in the next cacheline, and wasn't 
even fetched or didn't get far enough in the pipeline to wait for anything 
before the mis-predict was noticed and the "ret" continues at the target?

At that point, I don't think it's obvious that we couldn't have other 
speculation still outstanding.

So I agree that it's a worry. We can only ask Intel architects.

              Linus

[MODERATED] Re: ARCH_CAPABILITIES note

[Dave Hansen]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

On 06/06/2018 09:04 AM, speck for Linus Torvalds wrote:
> But your worry is what if the lfence was in the next cacheline, and wasn't 
> even fetched or didn't get far enough in the pipeline to wait for anything 
> before the mis-predict was noticed and the "ret" continues at the target?

Yeah, either it does not enter the pipeline, or the processor has thrown
it away before it has had a chance to do enough mitigation (for things
other than Spectre v2).

> At that point, I don't think it's obvious that we couldn't have other 
> speculation still outstanding.
> 
> So I agree that it's a worry. We can only ask Intel architects.

Yep, we'll do that.

For folks on this list: if you believe you are gleaning any mitigation
value from the retpoline LFENCE (or PAUSE for that matter) for issues
other than Spectre v2, please let Intel folks know (you can start with
those of us on this list).  We'll try to pin down what guarantees it can
actually provide.

[MODERATED] Re: ARCH_CAPABILITIES note

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

On 06/06/2018 11:50 AM, speck for Linus Torvalds wrote:

> Also, afaik, lfence doesn't necessarily wait for the store buffer to 
> drain, but since it waits for all preceding stores to be stable, it at 
> least guarantees that stores _are_ in the store buffer, and there are no 
> speculative store buffer entries, so it avoids the store buffer bypass 
> (and the SB poisoning that I haven't actually seen verified if Intel has 
> problems with or not).

Since SSB is public now, I fished with friends at Graz to see what
they'd investigated there. They said they did a bunch of testing already
and are working on more attacks using speculative store buffer function
pointer indirection into kernel address space.

Jon.


-- 
Computer Architect | Sent from my Fedora powered laptop