linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: David Woodhouse <dwmw2@infradead.org>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: "KarimAllah Ahmed" <karahmed@amazon.de>,
	"Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
	"Andi Kleen" <ak@linux.intel.com>,
	"Andrea Arcangeli" <aarcange@redhat.com>,
	"Andy Lutomirski" <luto@kernel.org>,
	"Arjan van de Ven" <arjan@linux.intel.com>,
	"Ashok Raj" <ashok.raj@intel.com>,
	"Asit Mallick" <asit.k.mallick@intel.com>,
	"Borislav Petkov" <bp@suse.de>,
	"Dan Williams" <dan.j.williams@intel.com>,
	"Dave Hansen" <dave.hansen@intel.com>,
	"Greg Kroah-Hartman" <gregkh@linuxfoundation.org>,
	"H . Peter Anvin" <hpa@zytor.com>,
	"Ingo Molnar" <mingo@redhat.com>,
	"Janakarajan Natarajan" <Janakarajan.Natarajan@amd.com>,
	"Joerg Roedel" <joro@8bytes.org>,
	"Jun Nakajima" <jun.nakajima@intel.com>,
	"Laura Abbott" <labbott@redhat.com>,
	"Masami Hiramatsu" <mhiramat@kernel.org>,
	"Paolo Bonzini" <pbonzini@redhat.com>,
	"Peter Zijlstra" <peterz@infradead.org>,
	"Radim Krčmář" <rkrcmar@redhat.com>,
	"Thomas Gleixner" <tglx@linutronix.de>,
	"Tim Chen" <tim.c.chen@linux.intel.com>,
	"Tom Lendacky" <thomas.lendacky@amd.com>,
	"KVM list" <kvm@vger.kernel.org>,
	"the arch/x86 maintainers" <x86@kernel.org>,
	"Arjan Van De Ven" <arjan.van.de.ven@intel.com>
Subject: Re: [RFC 09/10] x86/enter: Create macros to restrict/unrestrict Indirect Branch Speculation
Date: Mon, 22 Jan 2018 16:27:06 +0000	[thread overview]
Message-ID: <1516638426.9521.20.camel@infradead.org> (raw)
In-Reply-To: <CA+55aFwoxXmY0u84iPT-RP8DoG+Qb8t-mh2r8RLhKVAH3jm6qg@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 5893 bytes --]

On Sun, 2018-01-21 at 14:27 -0800, Linus Torvalds wrote:
> On Sun, Jan 21, 2018 at 2:00 PM, David Woodhouse <dwmw2@infradead.org> wrote:
> >>
> >> The patches do things like add the garbage MSR writes to the kernel
> >> entry/exit points. That's insane. That says "we're trying to protect
> >> the kernel".  We already have retpoline there, with less overhead.
> >
> > You're looking at IBRS usage, not IBPB. They are different things.
> 
> Ehh. Odd intel naming detail.
> 
> If you look at this series, it very much does that kernel entry/exit
> stuff. It was patch 10/10, iirc. In fact, the patch I was replying to
> was explicitly setting that garbage up.
> 
> And I really don't want to see these garbage patches just mindlessly
> sent around.

I think we've covered the technical part of this now, not that you like
it — not that any of us *like* it. But since the peanut gallery is
paying lots of attention it's probably worth explaining it a little
more for their benefit.

This is all about Spectre variant 2, where the CPU can be tricked into
mispredicting the target of an indirect branch. And I'm specifically
looking at what we can do on *current* hardware, where we're limited to
the hacks they can manage to add in the microcode.

The new microcode from Intel and AMD adds three new features.

One new feature (IBPB) is a complete barrier for branch prediction.
After frobbing this, no branch targets learned earlier are going to be
used. It's kind of expensive (order of magnitude ~4000 cycles).

The second (STIBP) protects a hyperthread sibling from following branch
predictions which were learned on another sibling. You *might* want
this when running unrelated processes in userspace, for example. Or
different VM guests running on HT siblings.

The third feature (IBRS) is more complicated. It's designed to be
set when you enter a more privileged execution mode (i.e. the kernel).
It prevents branch targets learned in a less-privileged execution mode,
BEFORE IT WAS MOST RECENTLY SET, from taking effect. But it's not just
a 'set-and-forget' feature, it also has barrier-like semantics and
needs to be set on *each* entry into the kernel (from userspace or a VM
guest). It's *also* expensive. And a vile hack, but for a while it was
the only option we had.

Even with IBRS, the CPU cannot tell the difference between different
userspace processes, and between different VM guests. So in addition to
IBRS to protect the kernel, we need the full IBPB barrier on context
switch and vmexit. And maybe STIBP while they're running.

Then along came Paul with the cunning plan of "oh, indirect branches
can be exploited? Screw it, let's not have any of *those* then", which
is retpoline. And it's a *lot* faster than frobbing IBRS on every entry
into the kernel. It's a massive performance win.

So now we *mostly* don't need IBRS. We build with retpoline, use IBPB
on context switches/vmexit (which is in the first part of this patch
series before IBRS is added), and we're safe. We even refactored the
patch series to put retpoline first.

But wait, why did I say "mostly"? Well, not everyone has a retpoline
compiler yet... but OK, screw them; they need to update.

Then there's Skylake, and that generation of CPU cores. For complicated
reasons they actually end up being vulnerable not just on indirect
branches, but also on a 'ret' in some circumstances (such as 16+ CALLs
in a deep chain).

The IBRS solution, ugly though it is, did address that. Retpoline
doesn't. There are patches being floated to detect and prevent deep
stacks, and deal with some of the other special cases that bite on SKL,
but those are icky too. And in fact IBRS performance isn't anywhere
near as bad on this generation of CPUs as it is on earlier CPUs
*anyway*, which makes it not quite so insane to *contemplate* using it
as Intel proposed.

That's why my initial idea, as implemented in this RFC patchset, was to
stick with IBRS on Skylake, and use retpoline everywhere else. I'll
give you "garbage patches", but they weren't being "just mindlessly
sent around". If we're going to drop IBRS support and accept the
caveats, then let's do it as a conscious decision having seen what it
would look like, not just drop it quietly because poor Davey is too
scared that Linus might shout at him again. :)

I have seen *hand-wavy* analyses of the Skylake thing that mean I'm not
actually lying awake at night fretting about it, but nothing concrete
that really says it's OK.

If you view retpoline as a performance optimisation, which is how it
first arrived, then it's rather unconventional to say "well, it only
opens a *little* bit of a security hole but it does go nice and fast so
let's do it".

But fine, I'm content with ditching the use of IBRS to protect the
kernel, and I'm not even surprised. There's a *reason* we put it last
in the series, as both the most contentious and most dispensable part.
I'd be *happier* with a coherent analysis showing Skylake is still OK,
but hey-ho, screw Skylake.

The early part of the series adds the new feature bits and detects when
it can turn KPTI off on non-Meltdown-vulnerable Intel CPUs, and also
supports the IBPB barrier that we need to make retpoline complete. That
much I think we definitely *do* want. There have been a bunch of us
working on this behind the scenes; one of us will probably post that
bit in the next day or so.

I think we also want to expose IBRS to VM guests, even if we don't use
it ourselves. Because Windows guests (and RHEL guests; yay!) do use it.

If we can be done with the shouty part, I'd actually quite like to have
a sensible discussion about when, if ever, we do IBPB on context switch
(ptraceability and dumpable have both been suggested) and when, if
ever, we set STIPB in userspace.




[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5213 bytes --]

  reply	other threads:[~2018-01-22 16:27 UTC|newest]

Thread overview: 161+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-01-20 19:22 [RFC 00/10] Speculation Control feature support KarimAllah Ahmed
2018-01-20 19:22 ` [RFC 01/10] x86/speculation: Add basic support for IBPB KarimAllah Ahmed
2018-01-20 19:22 ` [RFC 02/10] x86/kvm: Add IBPB support KarimAllah Ahmed
2018-01-20 20:18   ` Woodhouse, David
2018-01-22 18:56   ` Jim Mattson
2018-01-22 19:31     ` Jim Mattson
2018-01-20 19:22 ` [RFC 03/10] x86/speculation: Use Indirect Branch Prediction Barrier in context switch KarimAllah Ahmed
2018-01-20 19:22 ` [RFC 04/10] x86/mm: Only flush indirect branches when switching into non dumpable process KarimAllah Ahmed
2018-01-20 21:06   ` Woodhouse, David
2018-01-22 18:29     ` Tim Chen
2018-01-21 11:22   ` Peter Zijlstra
2018-01-21 12:04     ` David Woodhouse
2018-01-21 14:07       ` H.J. Lu
2018-01-22 10:19       ` Peter Zijlstra
2018-01-22 10:23         ` David Woodhouse
2018-01-21 16:21     ` Ingo Molnar
2018-01-21 16:25       ` Arjan van de Ven
2018-01-21 22:20       ` Woodhouse, David
2018-01-29  6:35     ` Jon Masters
2018-01-29 14:07       ` Peter Zijlstra
2018-01-20 19:22 ` [RFC 05/10] x86/speculation: Add basic IBRS support infrastructure KarimAllah Ahmed
2018-01-21 14:31   ` Thomas Gleixner
2018-01-21 14:56     ` Borislav Petkov
2018-01-22  9:51       ` Peter Zijlstra
2018-01-22 12:06         ` Borislav Petkov
2018-01-22 13:30           ` Greg Kroah-Hartman
2018-01-22 13:37             ` Woodhouse, David
2018-01-21 15:25     ` David Woodhouse
2018-01-23 20:58     ` David Woodhouse
2018-01-23 22:43       ` Johannes Erdfelt
2018-01-24  8:47       ` Peter Zijlstra
2018-01-24  9:02         ` David Woodhouse
2018-01-24  9:10           ` Greg Kroah-Hartman
2018-01-24 15:09             ` Arjan van de Ven
2018-01-24 15:18               ` David Woodhouse
2018-01-24  9:34           ` Peter Zijlstra
2018-01-24 10:49           ` Henrique de Moraes Holschuh
2018-01-24 12:30             ` David Woodhouse
2018-01-24 12:14         ` David Woodhouse
2018-01-24 12:29           ` Peter Zijlstra
2018-01-24 12:58             ` David Woodhouse
2018-01-29 20:14   ` [RFC,05/10] " Eduardo Habkost
2018-01-29 20:17     ` David Woodhouse
2018-01-29 20:42       ` Eduardo Habkost
2018-01-29 20:44         ` Arjan van de Ven
2018-01-29 21:02           ` David Woodhouse
2018-01-29 21:37             ` Jim Mattson
2018-01-29 21:50               ` Eduardo Habkost
2018-01-29 22:12                 ` Jim Mattson
2018-01-30  1:22                   ` Eduardo Habkost
2018-01-29 22:25                 ` Andi Kleen
2018-01-30  1:37                   ` Eduardo Habkost
2018-01-29 21:37             ` Andi Kleen
2018-01-29 21:44             ` Eduardo Habkost
2018-01-29 22:10               ` Konrad Rzeszutek Wilk
2018-01-30  1:12                 ` Eduardo Habkost
2018-01-30  0:23             ` Linus Torvalds
2018-01-30  1:03               ` Jim Mattson
2018-01-30  3:13                 ` Andi Kleen
2018-01-31 15:03                   ` Paolo Bonzini
2018-01-31 15:07                     ` Dr. David Alan Gilbert
2018-01-30  1:32               ` Arjan van de Ven
2018-01-30  3:32                 ` Linus Torvalds
2018-01-30 12:04                   ` Eduardo Habkost
2018-01-30 13:54                   ` Arjan van de Ven
2018-01-30  8:22               ` David Woodhouse
2018-01-30 11:35               ` David Woodhouse
2018-01-30 11:56               ` Dr. David Alan Gilbert
2018-01-30 12:11               ` Christian Borntraeger
2018-01-30 14:46                 ` Christophe de Dinechin
2018-01-30 14:52                   ` Christian Borntraeger
2018-01-30 14:56                     ` Christophe de Dinechin
2018-01-30 15:33                       ` Christian Borntraeger
2018-01-30 20:46               ` Alan Cox
2018-01-31 10:05                 ` Christophe de Dinechin
2018-01-31 10:15                   ` Thomas Gleixner
2018-01-31 11:04                     ` Dr. David Alan Gilbert
2018-01-31 11:52                       ` Borislav Petkov
2018-01-31 12:30                         ` Dr. David Alan Gilbert
2018-01-31 13:18                           ` Borislav Petkov
2018-01-31 14:04                             ` Dr. David Alan Gilbert
2018-01-31 14:44                               ` Eduardo Habkost
2018-01-31 16:28                                 ` Borislav Petkov
2018-01-31 11:07                     ` Christophe de Dinechin
2018-01-31 15:00                     ` Eduardo Habkost
2018-01-31 15:11                     ` Arjan van de Ven
2018-01-31 10:03   ` [RFC 05/10] " Christophe de Dinechin
2018-01-20 19:22 ` [RFC 06/10] x86/speculation: Add inlines to control Indirect Branch Speculation KarimAllah Ahmed
2018-01-20 19:22 ` [RFC 07/10] x86: Simplify spectre_v2 command line parsing KarimAllah Ahmed
2018-01-20 19:22 ` [RFC 08/10] x86/idle: Control Indirect Branch Speculation in idle KarimAllah Ahmed
2018-01-20 19:23 ` [RFC 09/10] x86/enter: Create macros to restrict/unrestrict Indirect Branch Speculation KarimAllah Ahmed
2018-01-21 19:14   ` Andy Lutomirski
2018-01-23 16:12     ` Tom Lendacky
2018-01-23 16:20       ` Woodhouse, David
2018-01-23 22:37         ` Tom Lendacky
2018-01-23 22:49           ` Andi Kleen
2018-01-23 23:14             ` Woodhouse, David
2018-01-23 23:22               ` Andi Kleen
2018-01-24  0:47               ` Tim Chen
2018-01-24  1:00                 ` Andy Lutomirski
2018-01-24  1:22                   ` David Woodhouse
2018-01-24  1:59                   ` Van De Ven, Arjan
2018-01-24  3:25                     ` Andy Lutomirski
2018-01-21 19:34   ` Linus Torvalds
2018-01-21 20:28     ` David Woodhouse
2018-01-21 21:35       ` Linus Torvalds
2018-01-21 22:00         ` David Woodhouse
2018-01-21 22:27           ` Linus Torvalds
2018-01-22 16:27             ` David Woodhouse [this message]
2018-01-23  7:29               ` Ingo Molnar
2018-01-23  7:53                 ` Ingo Molnar
2018-01-23  9:27                   ` Ingo Molnar
2018-01-23  9:37                     ` David Woodhouse
2018-01-23 15:01                     ` Dave Hansen
2018-01-23  9:30                   ` David Woodhouse
2018-01-23 10:15                     ` Ingo Molnar
2018-01-23 10:27                       ` David Woodhouse
2018-01-23 10:44                         ` Ingo Molnar
2018-01-23 10:57                           ` David Woodhouse
2018-01-23 10:23                     ` Ingo Molnar
2018-01-23 10:35                       ` David Woodhouse
2018-02-04 18:43                       ` Thomas Gleixner
2018-02-04 20:22                         ` David Woodhouse
2018-02-06  9:14                         ` David Woodhouse
2018-01-25 16:19                     ` Mason
2018-01-25 17:16                       ` Greg Kroah-Hartman
2018-01-29 11:59                         ` Mason
2018-01-24  0:05                 ` Andi Kleen
2018-01-23 20:16       ` Pavel Machek
2018-01-20 19:23 ` [RFC 10/10] x86/enter: Use IBRS on syscall and interrupts KarimAllah Ahmed
2018-01-21 13:50   ` Konrad Rzeszutek Wilk
2018-01-21 14:40     ` KarimAllah Ahmed
2018-01-21 17:22     ` Dave Hansen
2018-01-21 14:02 ` [RFC 00/10] Speculation Control feature support Konrad Rzeszutek Wilk
2018-01-22 21:27   ` David Woodhouse
2018-01-22 22:15 [RFC 09/10] x86/enter: Create macros to restrict/unrestrict Indirect Branch Speculation Luke Kenneth Casson Leighton
2018-01-23 11:13 Liran Alon
2018-01-25 22:20 ` Dave Hansen
2018-01-26  2:11 Liran Alon
2018-01-26  2:23 ` Dave Hansen
2018-01-26  9:11   ` David Woodhouse
2018-01-26 17:19     ` Linus Torvalds
2018-01-26 17:27       ` Borislav Petkov
2018-01-26 17:29       ` David Woodhouse
2018-01-26 17:31         ` David Woodhouse
2018-01-26 17:59       ` Andi Kleen
2018-01-26 18:11         ` David Woodhouse
2018-01-26 18:12           ` Arjan van de Ven
2018-01-26 18:26             ` David Woodhouse
2018-01-26 18:28               ` Van De Ven, Arjan
2018-01-26 18:43                 ` David Woodhouse
2018-01-26 18:44                   ` Van De Ven, Arjan
2018-01-26 18:53                     ` David Woodhouse
2018-01-26 19:02         ` Konrad Rzeszutek Wilk
2018-01-26 19:11           ` Hansen, Dave
2018-01-27 13:42             ` Konrad Rzeszutek Wilk
2018-01-27 15:55               ` Dave Hansen
2018-01-26 19:11           ` David Woodhouse
2018-01-26  8:46 ` David Woodhouse
2018-01-26  2:50 Liran Alon
2018-01-26  2:55 ` Van De Ven, Arjan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1516638426.9521.20.camel@infradead.org \
    --to=dwmw2@infradead.org \
    --cc=Janakarajan.Natarajan@amd.com \
    --cc=aarcange@redhat.com \
    --cc=ak@linux.intel.com \
    --cc=arjan.van.de.ven@intel.com \
    --cc=arjan@linux.intel.com \
    --cc=ashok.raj@intel.com \
    --cc=asit.k.mallick@intel.com \
    --cc=bp@suse.de \
    --cc=dan.j.williams@intel.com \
    --cc=dave.hansen@intel.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=hpa@zytor.com \
    --cc=joro@8bytes.org \
    --cc=jun.nakajima@intel.com \
    --cc=karahmed@amazon.de \
    --cc=kvm@vger.kernel.org \
    --cc=labbott@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=luto@kernel.org \
    --cc=mhiramat@kernel.org \
    --cc=mingo@redhat.com \
    --cc=pbonzini@redhat.com \
    --cc=peterz@infradead.org \
    --cc=rkrcmar@redhat.com \
    --cc=tglx@linutronix.de \
    --cc=thomas.lendacky@amd.com \
    --cc=tim.c.chen@linux.intel.com \
    --cc=torvalds@linux-foundation.org \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).