[MODERATED] Date/Time?

[MODERATED] Date/Time?

[Greg KH]

Ok, last I heard it's the 21st, this Monday, right?

Any specific timezone that applies to?

Jon, I know you were talking to people working on other arches, any
ideas when I should be expecting to see patches I can backport to get
into devices/distros?

thanks,

greg k-h

[MODERATED] Re: Date/Time?

[Borislav Petkov]

On Sat, May 19, 2018 at 07:26:27PM +0200, speck for Greg KH wrote:
> Ok, last I heard it's the 21st, this Monday, right?
> 
> Any specific timezone that applies to?

Says here: "May 21st 2100 UTC".

-- 
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 

[MODERATED] Re: Date/Time?

[David Woodhouse]

On Sat, 2018-05-19 at 19:26 +0200, speck for Greg KH wrote:
> Ok, last I heard it's the 21st, this Monday, right?
> 
> Any specific timezone that applies to?
> 
> Jon, I know you were talking to people working on other arches, any
> ideas when I should be expecting to see patches I can backport to get
> into devices/distros?

I'm putting together a 4.9 backport over the weekend and Monday
morning.

[MODERATED] Re: Date/Time?

[Linus Torvalds]

On Sat, 19 May 2018, speck for Borislav Petkov wrote:
> 
> Says here: "May 21st 2100 UTC".

What an odd time. It's noon in the Aleutian Islands and absolutely nowhere 
else that I can find.

I guess it's "after trading closes", but that would be 4pm eastern. And 
2100 UTC is 5pm EDT, so it's _really_ after.

So I wonder why that particular choice. 

Whatever. It's 2pm for me. I guess it's 11pm for Greg in Paris.

Are we sure that's not a typo, and it's supposed to be noon UTC, 
transposing the two digits?

             Linus

[MODERATED] Re: Date/Time?

[Greg KH]

On Sat, May 19, 2018 at 07:21:11PM +0100, speck for David Woodhouse wrote:
> On Sat, 2018-05-19 at 19:26 +0200, speck for Greg KH wrote:
> > Ok, last I heard it's the 21st, this Monday, right?
> > 
> > Any specific timezone that applies to?
> > 
> > Jon, I know you were talking to people working on other arches, any
> > ideas when I should be expecting to see patches I can backport to get
> > into devices/distros?
> 
> I'm putting together a 4.9 backport over the weekend and Monday
> morning.

Thanks for this, I started it but ran into some problems and then got
distracted when someone mentioned that you might be willing to do a
patch that can be tested well :)

greg k-h

[MODERATED] Re: Re: Date/Time?

[Borislav Petkov]

On Sat, May 19, 2018 at 11:45:38AM -0700, speck for Linus Torvalds wrote:
> So I wonder why that particular choice. 
> 
> Whatever. It's 2pm for me. I guess it's 11pm for Greg in Paris.
> 
> Are we sure that's not a typo, and it's supposed to be noon UTC, 
> transposing the two digits?

Another post from our sec. guys here says:

"May 21 1400 Pacific

which is around

CRD: 2018-05-21 21:00 UTC

or 2300 CET"

and 14:00 pacific is 2pm your time, right?

Maybe someone from Intel could confirm.

As to why, I think maybe because they didn't want to totally destroy the
holiday of people where Monday the 21st is an official holiday.

:-)

-- 
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 

[MODERATED] Re: Re: Date/Time?

[Linus Torvalds]

On Sat, 19 May 2018, speck for Borislav Petkov wrote:
> 
> Another post from our sec. guys here says:
> 
> "May 21 1400 Pacific

Ok, so that matches. 

> As to why, I think maybe because they didn't want to totally destroy the
> holiday of people where Monday the 21st is an official holiday.

End of Shavuot? Pentecost Monday? Random day of Ramadan? Canadian 
{Victoria, National Patriots'} Day?

Oh, it's "National Strawberries and Cream Day" in the US. Yes, we wouldn't 
want to interrupt _that_.

Anyway, not being religious, Canadian, _or_ particularly fanatical about 
Strawberries and Cream, I guess I really don't care. 2pm works fine for 
me.

[ Hmm, 2100UTC literally is "midnight in Jerusalem". So it actually 
  probably *is* "End of Shavout", and the time was picked for Orthodox 
  Jews.

  And I thought I was kidding when I looked up the date at first. ]

             Linus

[MODERATED] Re: Date/Time?

[Borislav Petkov]

On Sat, May 19, 2018 at 12:48:34PM -0700, speck for Linus Torvalds wrote:
> End of Shavuot? Pentecost Monday? Random day of Ramadan? Canadian 
> {Victoria, National Patriots'} Day?
> 
> Oh, it's "National Strawberries and Cream Day" in the US. Yes, we wouldn't 
> want to interrupt _that_.

LOL!

Strawberries and cream just don't taste as good with store bypass chunks
speculatively sprinkled ontop. ;-P

But yeah, it is Pentecost Monday here in .de.

> Anyway, not being religious, Canadian, _or_ particularly fanatical about 
> Strawberries and Cream, I guess I really don't care. 2pm works fine for 
> me.

Ah, you wanna know when to push out the speck pile you've gotten from
tglx, right.

-- 
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 

[MODERATED] Re: Date/Time?

[Jiri Kosina]

On Sat, 19 May 2018, speck for Greg KH wrote:

> Jon, I know you were talking to people working on other arches, any
> ideas when I should be expecting to see patches I can backport to get
> into devices/distros?

FWIW we have 4.4 port, let me know if you want it.

But it's based on our SLE kernel, which also contains the IBRS stuff, so 
it'd need some minor tweaking so that it applies to kernel without IBRS.

-- 
Jiri Kosina
SUSE Labs

[MODERATED] Re: Date/Time?

[Greg KH]

On Sun, May 20, 2018 at 07:45:20PM +0200, speck for Jiri Kosina wrote:
> On Sat, 19 May 2018, speck for Greg KH wrote:
> 
> > Jon, I know you were talking to people working on other arches, any
> > ideas when I should be expecting to see patches I can backport to get
> > into devices/distros?
> 
> FWIW we have 4.4 port, let me know if you want it.
> 
> But it's based on our SLE kernel, which also contains the IBRS stuff, so 
> it'd need some minor tweaking so that it applies to kernel without IBRS.

Why wouldn't I also want the IBRS stuff for the upstream 4.4.y?  Am I
missing patches that I should have backported but didn't?

And yes, I do want it if you have it :)

thanks,

greg k-h

[MODERATED] Re: Date/Time?

[Borislav Petkov]

On Sun, May 20, 2018 at 08:19:34PM +0200, speck for Greg KH wrote:
> Why wouldn't I also want the IBRS stuff for the upstream 4.4.y?

Because the IBRS stuff is not upstream.

-- 
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- 

[MODERATED] Re: Date/Time?

[Linus Torvalds]

On Sun, 20 May 2018, speck for Greg KH wrote:
> 
> Why wouldn't I also want the IBRS stuff for the upstream 4.4.y?  Am I
> missing patches that I should have backported but didn't?

I think Jiri is talking about the bat-shit-crazy stuff that I refused to 
take upstream. The stuff that adds thousands of cycles to every kernel 
entry and doesn't actually fix any real problem except for "but but in 
theory" PR problems.

             Linus

[MODERATED] Re: Date/Time?

[Jiri Kosina]

On Sun, 20 May 2018, speck for Linus Torvalds wrote:

> > Why wouldn't I also want the IBRS stuff for the upstream 4.4.y?  Am I
> > missing patches that I should have backported but didn't?
> 
> I think Jiri is talking about the bat-shit-crazy stuff that I refused to 
> take upstream. The stuff that adds thousands of cycles to every kernel 
> entry and doesn't actually fix any real problem except for "but but in 
> theory" PR problems.

Yes, that is exactly the one :)

We are providing the option of toggling the IBRS on kernel entry/exit only 
on CPUs that can fall back to using (potentially) poisoned indirect branch 
predictor in case of RSB underflow (Skylake; fortunately on those CPUs the 
IBRS overhead is considerably lower than on other CPUs) if the user/admin 
wants to.

And that is the same MSR as SSBD, so there will be some minor conflicts to 
resolve.

-- 
Jiri Kosina
SUSE Labs

[MODERATED] Re: Date/Time?

[David Woodhouse]

On Sat, 2018-05-19 at 19:21 +0100, speck for David Woodhouse wrote:
> On Sat, 2018-05-19 at 19:26 +0200, speck for Greg KH wrote:
> > 
> > Ok, last I heard it's the 21st, this Monday, right?
> > 
> > Any specific timezone that applies to?
> > 
> > Jon, I know you were talking to people working on other arches, any
> > ideas when I should be expecting to see patches I can backport to get
> > into devices/distros?
>
> I'm putting together a 4.9 backport over the weekend and Monday
> morning.

As I go through this, I note we seem to have have gained a bunch of
this kind of thing (comments added by me):

 if (…)
   wrmsr();
 /* else speculate happily over that conditional branch as
    if the wrmsr had never happened ... */


Are we sure those are all OK?

[MODERATED] Re: Date/Time?

[Greg KH]

On Sun, May 20, 2018 at 08:35:27PM +0200, speck for Borislav Petkov wrote:
> On Sun, May 20, 2018 at 08:19:34PM +0200, speck for Greg KH wrote:
> > Why wouldn't I also want the IBRS stuff for the upstream 4.4.y?
> 
> Because the IBRS stuff is not upstream.

Ah, that stuff.  Yeah, no, I don't want that "batshit crazy stuff" in
the 4.4.y tree, to quote someone :)

thanks,

greg k-h

[MODERATED] Re: Date/Time?

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

On 05/19/2018 02:45 PM, speck for Linus Torvalds wrote:
> 
> 
> On Sat, 19 May 2018, speck for Borislav Petkov wrote:
>>
>> Says here: "May 21st 2100 UTC".
> 
> What an odd time. It's noon in the Aleutian Islands and absolutely nowhere 
> else that I can find.
> 
> I guess it's "after trading closes", but that would be 4pm eastern. And 
> 2100 UTC is 5pm EDT, so it's _really_ after.
> 
> So I wonder why that particular choice. 
> 
> Whatever. It's 2pm for me. I guess it's 11pm for Greg in Paris.
> 
> Are we sure that's not a typo, and it's supposed to be noon UTC, 
> transposing the two digits?

It's 2pm PDT because Intel asked GPZ if "21st" could mean "the afternoon
of the 21st" in order to allow various CERT briefings and the like. It
was apparently deemed reasonable by the Google powers that be.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

[MODERATED] Re: Date/Time?

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 3246 bytes --]

On 05/19/2018 01:26 PM, speck for Greg KH wrote:

> Jon, I know you were talking to people working on other arches, any
> ideas when I should be expecting to see patches I can backport to get
> into devices/distros?

Sorry for lag. In flight back from a funeral. Here's the latest below.
Then please see my rant (not directed at anyone here) at the end that
makes me think I should maintain a tree next time to get these to you.

* IBM p. They've an "stf-flush" patch series that protects the kernel
stack from attack by doing (effectively) a store-to-forward buffer flush
on entry into the kernel, patched with optimal code using alternatives.
It's /very/ similar to the rfi-flush code, but it doesn't rely upon
updated millicode this time. Backporting to older distros will be easier
because the heavy lifting on the entry macros was done for older kernels
the last time around. IBM know to contact Thomas, and should have
already, but I will specifically remind them AGAIN to also ping Greg. I
left the patch sending process to them. They know the parameters, etc.

That's just the kernel part. There will updated patches for userspace
and the prctl will be wired up later on. I imagine that will need some
kind of millicode but I don't yet have the plan fully from IBM there.

* IBM z. I probably shouldn't say here whether they are or are not
impacted by any of this. I can tell you that they aren't impacted by the
kernel attack since all entry into the kernel is fully serializing. They
have taken the view that they don't intend to provide any patches at
this time. I have requested an escalation within IBM on this topic.

* ARM. They have a new SMCC (Secure Monitor Call) interface (similar to
the one they did for branch predictor invalidation for Spectre-v2) that
will be wired up with kernel patches. Each of the vendors will implement
the new SMCC in ATF (Arm Trusted Firmware) on their parts, using the
best back-end mitigation (similar to microcode). Arm know to ping Thomas
with those patches, yet this has not happened yet (to my knowledge), nor
to Greg. Will has point on this in any case. He and the team are working
hard on this. Still, I am saddened by these patches not being available
prior to disclosure since this is not how we do things in the server
space. But in anticipation that this would happen, I worked directly
with Cavium (the only server-class CPU vendor with production RHEL
support for which we need an answer tomorrow on our end) to create a
firmware knob that can be used in the short term until this is fixed. I
also have pinged all of the Arm server vendors to let them know I expect
all of the SMCC wiring to be in place within the next few weeks.

And now my little rant. It is clear (to me) that Intel know how to
handle some of this better than others. The next time there is an issue,
I am going to cajole all of the vendors as usual, but I will also make
sure Greg, Linus, and Thomas are read in early enough that we can go
figure out the best way to handle the other arches. If that's me hosting
a git tree, great. Whatever you need. But we need to fix this next time.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

[MODERATED] Re: Date/Time?

[Jiri Kosina]

On Sun, 20 May 2018, speck for Jon Masters wrote:

> * IBM p. They've an "stf-flush" patch series that protects the kernel
> stack from attack by doing (effectively) a store-to-forward buffer flush
> on entry into the kernel, patched with optimal code using alternatives.
> It's /very/ similar to the rfi-flush code, but it doesn't rely upon
> updated millicode this time. Backporting to older distros will be easier
> because the heavy lifting on the entry macros was done for older kernels
> the last time around. IBM know to contact Thomas, and should have
> already, but I will specifically remind them AGAIN to also ping Greg. I
> left the patch sending process to them. They know the parameters, etc.

FWIW we already have the port of this to 4.12, 4.4 and 3.0 in case -stable 
would like to have these before IBM submits it properly.

-- 
Jiri Kosina
SUSE Labs

[MODERATED] Re: Date/Time?

[Linus Torvalds]

On Sun, 20 May 2018, speck for Jon Masters wrote:
> 
> And now my little rant. It is clear (to me) that Intel know how to
> handle some of this better than others.

Hahhahahaaaa! <Wipes tears from eyes, not sure whether from laughing or 
crying>.

Yeah, Intel has been great. Snort.

They did learn something from last time, I think, and that made them more 
open to some things. But I really think it's mostly because Thomas has 
been pushing it. Look at where this mailing list is hosted, for example, 
just to pick one random detail.

             Linus

[MODERATED] Re: Date/Time?

[David Woodhouse]

On Sun, 2018-05-20 at 14:48 -0700, speck for Linus Torvalds wrote:
> They did learn something from last time, I think, and that made them more 
> open to some things. But I really think it's mostly because Thomas has 
> been pushing it. Look at where this mailing list is hosted, for example, 
> just to pick one random detail.

OK, it was only one example of many but really... if Intel ever offer
to host anything email-related for you, say no. :)

There's a reason I was giving Intel people functional email accounts
long before I ever started working there, and continue to do so even
after I stopped.

[MODERATED] Re: Date/Time?

[David Woodhouse]

On Sat, 2018-05-19 at 19:21 +0100, speck for David Woodhouse wrote:
> On Sat, 2018-05-19 at 19:26 +0200, speck for Greg KH wrote:
> > 
> > Ok, last I heard it's the 21st, this Monday, right?
> > 
> > Any specific timezone that applies to?
> > 
> > Jon, I know you were talking to people working on other arches, any
> > ideas when I should be expecting to see patches I can backport to get
> > into devices/distros?
>
> I'm putting together a 4.9 backport over the weekend and Monday
> morning.

I've just pushed this to the linux-4.9.y branch of Thomas's git tree. 
It's basically untested except for the fact that it builds (for
x86_64). I'll do some testing in the morning.

[MODERATED] Re: Date/Time?

[Greg KH]

On Sun, May 20, 2018 at 11:09:08PM +0200, speck for Jiri Kosina wrote:
> On Sun, 20 May 2018, speck for Jon Masters wrote:
> 
> > * IBM p. They've an "stf-flush" patch series that protects the kernel
> > stack from attack by doing (effectively) a store-to-forward buffer flush
> > on entry into the kernel, patched with optimal code using alternatives.
> > It's /very/ similar to the rfi-flush code, but it doesn't rely upon
> > updated millicode this time. Backporting to older distros will be easier
> > because the heavy lifting on the entry macros was done for older kernels
> > the last time around. IBM know to contact Thomas, and should have
> > already, but I will specifically remind them AGAIN to also ping Greg. I
> > left the patch sending process to them. They know the parameters, etc.
> 
> FWIW we already have the port of this to 4.12, 4.4 and 3.0 in case -stable 
> would like to have these before IBM submits it properly.

Thanks for the offer, but I'll wait until they hit Linus's tree.

Odds are Debian might want these patches, if you have them, so maybe
posting them somewhere after the deadline will be helpful for them.

thanks,

greg k-h

[MODERATED] Re: Date/Time?

[Greg KH]

On Sun, May 20, 2018 at 04:59:04PM -0400, speck for Jon Masters wrote:
> On 05/19/2018 01:26 PM, speck for Greg KH wrote:
> 
> > Jon, I know you were talking to people working on other arches, any
> > ideas when I should be expecting to see patches I can backport to get
> > into devices/distros?
> 
> Sorry for lag. In flight back from a funeral. Here's the latest below.
> Then please see my rant (not directed at anyone here) at the end that
> makes me think I should maintain a tree next time to get these to you.

<snip of the summary>

Thanks for this, much appreciated.

Sad to see there is no other arch that is going to be ready, wonderful :(

> And now my little rant. It is clear (to me) that Intel know how to
> handle some of this better than others. The next time there is an issue,
> I am going to cajole all of the vendors as usual, but I will also make
> sure Greg, Linus, and Thomas are read in early enough that we can go
> figure out the best way to handle the other arches. If that's me hosting
> a git tree, great. Whatever you need. But we need to fix this next time.

As Linus said, hah!  Intel only "handled this better" because of Thomas
pushing them (the effort it took to get me here was crazy...)

And I doubt we need another git tree, we already have one, that should
have worked for any arch that needed it.  The problem seems to be that
those developers were not "allowed" here for us to work together,
exactly like last time.

Oh well, baby steps, maybe next time will be better, but I'm not holding
my breath.

greg k-h

[MODERATED] Re: Date/Time?

[David Woodhouse]

On Mon, 2018-05-21 at 08:16 +0200, speck for Greg KH wrote:

> And I doubt we need another git tree, we already have one, that should
> have worked for any arch that needed it.  The problem seems to be that
> those developers were not "allowed" here for us to work together,
> exactly like last time.

If that is 100% known to be the case, then it should be part of our
messaging when we release.

"Here are patches for x86. Other architectures are not yet covered
because their maintainers were not permitted to work with the pre-
embargo team for corporate bullshit reasons."

I'm sure Linus can phrase that sentiment just as well as I can... :)

After the requisite whining from pointyhairs at the companies
concerned, it will lead to their engineers being allowed to do the
right thing next time.

We *do* have to be sure it's true, of course.

[MODERATED] Re: Date/Time?

[Greg KH]

On Mon, May 21, 2018 at 12:01:02AM +0100, speck for David Woodhouse wrote:
> 
> 
> On Sat, 2018-05-19 at 19:21 +0100, speck for David Woodhouse wrote:
> > On Sat, 2018-05-19 at 19:26 +0200, speck for Greg KH wrote:
> > > 
> > > Ok, last I heard it's the 21st, this Monday, right?
> > > 
> > > Any specific timezone that applies to?
> > > 
> > > Jon, I know you were talking to people working on other arches, any
> > > ideas when I should be expecting to see patches I can backport to get
> > > into devices/distros?
> >
> > I'm putting together a 4.9 backport over the weekend and Monday
> > morning.
> 
> I've just pushed this to the linux-4.9.y branch of Thomas's git tree. 
> It's basically untested except for the fact that it builds (for
> x86_64). I'll do some testing in the morning.

At first glance, it looks good (53 patches!!!).  I've grabbed the 1
prerequisite patch that is already upstream now.

Thanks a lot for these.

greg k-h

Re: Date/Time?

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 2138 bytes --]

On Mon, 21 May 2018, speck for David Woodhouse wrote:
> On Mon, 2018-05-21 at 08:16 +0200, speck for Greg KH wrote:
> 
> > And I doubt we need another git tree, we already have one, that should
> > have worked for any arch that needed it.  The problem seems to be that
> > those developers were not "allowed" here for us to work together,
> > exactly like last time.
> 
> If that is 100% known to be the case, then it should be part of our
> messaging when we release.

We could have created yet another crypto mailing list to have the others on
board for GPZ4 because they are not allowed to hear about the L1 still
embargoed (Intel only) crap.

I pondered it for a bit, but then decided not to as its already bad enough
to have one crypto list to deal with. And dealing with the extra nonsense
of figuring out which list is for what and then having parallel discussions
about the same thing on both lists is just not workable.

I gave them access to the sysfs/prctl/seccomp and command line bits though,
so they could align their work on ours. I offered them to merge their stuff
through the sekrit git repo, but they have their own plans it seems.

I told Intel that we want the others here and that we can stop talking
about the L1 fuckup until GPZ4 is public and Intel agreed. That was at
least 4 weeks ago.

But the only two people who were allowed on the list are Tom Lendacky and
Alexei Starovoitov. The ARM/PowerPC people were still under discussion
@Intel end of last week. Useful...

Getting Tom in took only 3 weeks. For Greg it was 6 weeks and I had to push
hard to get it done. Alexei was pushed through by Dave Hansen, no idea how
long it took him to get there.

So in general the situation was more workable than last time, but far from
being great.

The right thing for stuff like this is treating us as adult and responsible
people and let us work it out on our own without being babysitted by Intel
and their disfunctional and agonizingly slow bureaucraZy. 

Seriously, the whole foofaraw about the L1 fuckup is just an idiotic
secrecy spectacle.

Thanks,

	tglx

[MODERATED] Re: Date/Time?

[Greg KH]

On Sun, May 20, 2018 at 04:59:04PM -0400, speck for Jon Masters wrote:
> * ARM. They have a new SMCC (Secure Monitor Call) interface (similar to
> the one they did for branch predictor invalidation for Spectre-v2) that
> will be wired up with kernel patches. Each of the vendors will implement
> the new SMCC in ATF (Arm Trusted Firmware) on their parts, using the
> best back-end mitigation (similar to microcode). Arm know to ping Thomas
> with those patches, yet this has not happened yet (to my knowledge), nor
> to Greg. Will has point on this in any case. He and the team are working
> hard on this. Still, I am saddened by these patches not being available
> prior to disclosure since this is not how we do things in the server
> space. But in anticipation that this would happen, I worked directly
> with Cavium (the only server-class CPU vendor with production RHEL
> support for which we need an answer tomorrow on our end) to create a
> firmware knob that can be used in the short term until this is fixed. I
> also have pinged all of the Arm server vendors to let them know I expect
> all of the SMCC wiring to be in place within the next few weeks.

Just heard from ARM, they are going to wait a week or so and then send
patches for inclusion in 4.18 and then send some backports for the older
kernels then.  The kernel patches are useless without firmware updates
and I don't know what the state of them are.

They also complained that they can not finalize their patches as they do
not know what we agreed on for the prctl() interface.  Which sucks as
that was hashed out here weeks ago.

Another proof of how when we are not "allowed" to talk to the right
people, Linux users suffer.  That is not ok.

thanks,

greg k-h

[MODERATED] Re: Date/Time?

[Konrad Rzeszutek Wilk]

> They also complained that they can not finalize their patches as they do
> not know what we agreed on for the prctl() interface.  Which sucks as
> that was hashed out here weeks ago.

They weren't on keybase? I've religiously been telling folks what is done
upstream along with the patches so that this would _not_ occur!

This is on the 'spectre_meltdown' keybase channel.
<sighs>
> 
> Another proof of how when we are not "allowed" to talk to the right
> people, Linux users suffer.  That is not ok.
> 
> thanks,
> 
> greg k-h

[MODERATED] Re: Date/Time?

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 2620 bytes --]

On 05/21/2018 12:46 PM, speck for Greg KH wrote:
> On Sun, May 20, 2018 at 04:59:04PM -0400, speck for Jon Masters wrote:
>> * ARM. They have a new SMCC (Secure Monitor Call) interface (similar to
>> the one they did for branch predictor invalidation for Spectre-v2) that
>> will be wired up with kernel patches. Each of the vendors will implement
>> the new SMCC in ATF (Arm Trusted Firmware) on their parts, using the
>> best back-end mitigation (similar to microcode). Arm know to ping Thomas
>> with those patches, yet this has not happened yet (to my knowledge), nor
>> to Greg. Will has point on this in any case. He and the team are working
>> hard on this. Still, I am saddened by these patches not being available
>> prior to disclosure since this is not how we do things in the server
>> space. But in anticipation that this would happen, I worked directly
>> with Cavium (the only server-class CPU vendor with production RHEL
>> support for which we need an answer tomorrow on our end) to create a
>> firmware knob that can be used in the short term until this is fixed. I
>> also have pinged all of the Arm server vendors to let them know I expect
>> all of the SMCC wiring to be in place within the next few weeks.
> 
> Just heard from ARM, they are going to wait a week or so and then send
> patches for inclusion in 4.18 and then send some backports for the older
> kernels then.  The kernel patches are useless without firmware updates
> and I don't know what the state of them are.

They're on the keybase channel and have been receiving all of the
updated patches. I also specifically sent them a copy previously.

> They also complained that they can not finalize their patches as they do
> not know what we agreed on for the prctl() interface.  Which sucks as
> that was hashed out here weeks ago.

Nah. As Konrad says, several folk from Arm are on keybase. They also
could have pinged us at any time if this was blocking progress. I know
resources are tight and committed, but since none of us were pinged, I'm
calling BS on that. I've also not been given even an early draft, nor
know anyone who has, even though I asked their kernel lead working on
this for a copy. I've made my peace with where we are today, sadly.

> Another proof of how when we are not "allowed" to talk to the right
> people, Linux users suffer.  That is not ok.

One of the problems does seem to be that we didn't have a dedicated list
for v4 earlier. I think it's important to tackle that next time around.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

[MODERATED] Re: Date/Time?

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 2581 bytes --]

On 05/21/2018 12:46 PM, speck for Greg KH wrote:
> On Sun, May 20, 2018 at 04:59:04PM -0400, speck for Jon Masters wrote:
>> * ARM. They have a new SMCC (Secure Monitor Call) interface (similar to
>> the one they did for branch predictor invalidation for Spectre-v2) that
>> will be wired up with kernel patches. Each of the vendors will implement
>> the new SMCC in ATF (Arm Trusted Firmware) on their parts, using the
>> best back-end mitigation (similar to microcode). Arm know to ping Thomas
>> with those patches, yet this has not happened yet (to my knowledge), nor
>> to Greg. Will has point on this in any case. He and the team are working
>> hard on this. Still, I am saddened by these patches not being available
>> prior to disclosure since this is not how we do things in the server
>> space. But in anticipation that this would happen, I worked directly
>> with Cavium (the only server-class CPU vendor with production RHEL
>> support for which we need an answer tomorrow on our end) to create a
>> firmware knob that can be used in the short term until this is fixed. I
>> also have pinged all of the Arm server vendors to let them know I expect
>> all of the SMCC wiring to be in place within the next few weeks.
> 
> Just heard from ARM, they are going to wait a week or so and then send
> patches for inclusion in 4.18 and then send some backports for the older
> kernels then.  The kernel patches are useless without firmware updates
> and I don't know what the state of them are.

For the firmware, I'm told that there's a reference ATF (Arm Trusted
Firmware) in flight that implements the new SMCC. I'm aware that some of
the client folks have this working. On the server side, people are
waiting for Arm to release the reference to them to incorporate. One of
the server vendors is not impacted at all, while three others are to
differing levels, and all can implement the SMCC but may also provide
more optimal per-platform fixes via the errata infrastructure.

I've been working with Cavium (as the only supported platform with RHEL
today - we only ship subscriptions with Tier 1 OEMs, the others you can
kick the tires on but no support yet) on plans and we came to a decision
that we would use the emergency knob in the short term, then give Arm a
few weeks to get the ATF reference out. If they don't have that done
within the next few weeks, at least for Cavium/HPE it'll be done
separately implementing the SMCC as we're not waiting around.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

Re: Date/Time?

[Thomas Gleixner]

On Mon, 21 May 2018, speck for Greg KH wrote:
> On Sun, May 20, 2018 at 04:59:04PM -0400, speck for Jon Masters wrote:
> > * ARM. They have a new SMCC (Secure Monitor Call) interface (similar to
> > the one they did for branch predictor invalidation for Spectre-v2) that
> > will be wired up with kernel patches. Each of the vendors will implement
> > the new SMCC in ATF (Arm Trusted Firmware) on their parts, using the
> > best back-end mitigation (similar to microcode). Arm know to ping Thomas
> > with those patches, yet this has not happened yet (to my knowledge), nor
> > to Greg. Will has point on this in any case. He and the team are working
> > hard on this. Still, I am saddened by these patches not being available
> > prior to disclosure since this is not how we do things in the server
> > space. But in anticipation that this would happen, I worked directly
> > with Cavium (the only server-class CPU vendor with production RHEL
> > support for which we need an answer tomorrow on our end) to create a
> > firmware knob that can be used in the short term until this is fixed. I
> > also have pinged all of the Arm server vendors to let them know I expect
> > all of the SMCC wiring to be in place within the next few weeks.
> 
> Just heard from ARM, they are going to wait a week or so and then send
> patches for inclusion in 4.18 and then send some backports for the older
> kernels then.  The kernel patches are useless without firmware updates
> and I don't know what the state of them are.
> 
> They also complained that they can not finalize their patches as they do
> not know what we agreed on for the prctl() interface.  Which sucks as
> that was hashed out here weeks ago.

That's odd as I gave them access to the patches on May 9th...
 
Thanks,

	tglx

Re: Date/Time?

[Thomas Gleixner]

On Mon, 21 May 2018, speck for Jon Masters wrote:
> One of the problems does seem to be that we didn't have a dedicated list
> for v4 earlier. I think it's important to tackle that next time around.

Crap. What's important is that _WE_ can decide and act on our own without
having to ask Intel about every fart.

Thanks,

	tglx

[MODERATED] Re: Date/Time?

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

On 05/21/2018 02:56 PM, speck for Thomas Gleixner wrote:
> On Mon, 21 May 2018, speck for Jon Masters wrote:
>> One of the problems does seem to be that we didn't have a dedicated list
>> for v4 earlier. I think it's important to tackle that next time around.
> 
> Crap. What's important is that _WE_ can decide and act on our own without
> having to ask Intel about every fart.

Sure. My point is Intel didn't own v4. GPZ own that one. Intel just took
point in doing disclosures to various folks once again. I actually
consider what they are trying to do to be quite noble in intention, but
(as I have said to Intel's execs with all due intended respect) it's not
their place to do this for the whole industry. And it's important to
understand the separation between v4 and any other past/future issue.

This is why we need a better framework (and some folks are working on
something) for joint industry issues that removes single vendors from
owning architectural embargoes that impact multiple others.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

[MODERATED] Re: Date/Time?

[Greg KH]

On Mon, May 21, 2018 at 01:12:40PM -0400, speck for Konrad Rzeszutek Wilk wrote:
> > They also complained that they can not finalize their patches as they do
> > not know what we agreed on for the prctl() interface.  Which sucks as
> > that was hashed out here weeks ago.
> 
> They weren't on keybase? I've religiously been telling folks what is done
> upstream along with the patches so that this would _not_ occur!
> 
> This is on the 'spectre_meltdown' keybase channel.

Before an hour ago, I didn't know what "keybase" was.  Now that I do
know, I have no idea how in the world anyone submitted kernel patches
using it for ARM people to be basing their code on.

Come on people, be sane here.  The ARM developers said they only got the
patches through a back-channel, which sounds like what the keybase chat
was.  But it wasn't good enough and I don't blame them.

ugh.  Oh well, good thing that no one cares about ARM servers :(

thanks,

greg k-h

[MODERATED] Re: Date/Time?

[Jiri Kosina]

On Mon, 21 May 2018, speck for Greg KH wrote:

> Before an hour ago, I didn't know what "keybase" was.  Now that I do 
> know, I have no idea how in the world anyone submitted kernel patches 
> using it for ARM people to be basing their code on.

If Thomas wouldn't set up this list, apparently everything would be 
happening just on this keybase (whatever that is). I was a bit shocked in 
the early phases indeed.

I find it rather ironic that for discussing stuff that allows external 
websites to steal encryption keys from your browser ... you use ... 
*drumroll* ... external website :)

-- 
Jiri Kosina
SUSE Labs

[MODERATED] Re: Date/Time?

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 2038 bytes --]

On 05/21/2018 03:48 PM, speck for Jiri Kosina wrote:
> On Mon, 21 May 2018, speck for Greg KH wrote:
> 
>> Before an hour ago, I didn't know what "keybase" was.  Now that I do 
>> know, I have no idea how in the world anyone submitted kernel patches 
>> using it for ARM people to be basing their code on.
> 
> If Thomas wouldn't set up this list, apparently everything would be 
> happening just on this keybase (whatever that is). I was a bit shocked in 
> the early phases indeed.

> I find it rather ironic that for discussing stuff that allows external 
> websites to steal encryption keys from your browser ... you use ... 
> *drumroll* ... external website :)

Let's skip the FUD tho. It's not a "website". It's an open source
end-to-end encrypted chat application. Its purpose was never to share
patches anyway, just for coordination of issues (you can attach files,
but that was only for convenience). Keybase also wasn't my idea, but I
don't have a problem with it as a useful means of getting many parties
collaborating on coordinated disclosure (think of it as IRC but with an
attempt at actual security that Google, Intel, and others were ok with
using). At the same time, it's great tglx setup this list for Linux
patch discussion. They aren't both trying to solve the same thing.

Specifically on the other arches, at the time they needed to know what
to work on, there was no mailing list anyway (this list wasn't setup for
v4 specific discussion at that time, so those folks could not have been
added to it anyway). They received a copy of the work in progress, by
email as well, and were part of the ongoing discussion so did know each
time Konrad mentioned an updated patch was available. Various of us also
sent copies of the patches. Folks certainly knew who to ping for help.

Regardless of how things should or should not be done, folks were
certainly well aware that things were coming today, months ago.

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop

[MODERATED] Re: Date/Time?

[Greg KH]

On Mon, May 21, 2018 at 03:16:56PM -0400, speck for Jon Masters wrote:
> 
> This is why we need a better framework (and some folks are working on
> something) for joint industry issues that removes single vendors from
> owning architectural embargoes that impact multiple others.

Might I ask who those "some folks" are?

I've been talking to a few groups about this recently, CERT included
(it's actually their mandate to be the ones to do this, but well...)

We should take this off-list...

thanks,

greg k-h

[MODERATED] Re: Date/Time?

[Peter Zijlstra]

On Mon, May 21, 2018 at 04:30:27PM -0400, speck for Jon Masters wrote:
> Let's skip the FUD tho. It's not a "website". It's an open source
> end-to-end encrypted chat application.

But it is not packaged by any distro afaik, so you basically have to
trust their webshite to fetch a binary, which is pretty dodgy if you ask
me.

Also, keybase is pretty terrible UI wise..

[MODERATED] Re: Date/Time?

[Greg KH]

On Tue, May 22, 2018 at 10:00:35AM +0200, speck for Peter Zijlstra wrote:
> On Mon, May 21, 2018 at 04:30:27PM -0400, speck for Jon Masters wrote:
> > Let's skip the FUD tho. It's not a "website". It's an open source
> > end-to-end encrypted chat application.
> 
> But it is not packaged by any distro afaik, so you basically have to
> trust their webshite to fetch a binary, which is pretty dodgy if you ask
> me.

You mean you don't trust the "run a random binary from the internet as
root?" instructions they provide you:
	https://keybase.io/docs/the_app/install_linux
:(

And Jon, it's not "FUD" it's the apparent hypocrisy of some companies
INSISTING on keeping things "secure" and then they go and place the
patches created here up on a random location/site and insist on all
security people from all companies run some random binary as root just
to talk about when they might be releasing things and what to do next
and when they should all meet up for beer when they are in town.

If I wanted to Trojan a whole bunch of people that really should know
better, this would be the place to do it.

ugh.

greg k-h

[MODERATED] Re: Date/Time?

[Andi Kleen]

> If I wanted to Trojan a whole bunch of people that really should know
> better, this would be the place to do it.

Yes I agree. That looks pretty suspicious. Much easier to trust good old gpg.
 
-Andi

[MODERATED] Re: Date/Time?

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

On 05/22/2018 05:26 AM, speck for Greg KH wrote:
> On Tue, May 22, 2018 at 10:00:35AM +0200, speck for Peter Zijlstra wrote:
>> On Mon, May 21, 2018 at 04:30:27PM -0400, speck for Jon Masters wrote:
>>> Let's skip the FUD tho. It's not a "website". It's an open source
>>> end-to-end encrypted chat application.
>>
>> But it is not packaged by any distro afaik, so you basically have to
>> trust their webshite to fetch a binary, which is pretty dodgy if you ask
>> me.
> 
> You mean you don't trust the "run a random binary from the internet as
> root?" instructions they provide you:
> 	https://keybase.io/docs/the_app/install_linux

Conversely, encrypted GPG email is so secure. Just last week, I recall
reading about how great it is that there are totally no problems.

Do I like the "run this binary" instructions? No. But you can also
download the source from github and build it. Which of course is totally
secure because we're all going to audit every line of the source before
we build it, just like people who rebuild Android because that's way
more secure than trusting the images they download. Just like how we all
audited our email and GPG implementations, as well as IRC clients, and
all of the other software we're running, completely.

So anyway, the above aside, I get it. The companies involved were
willing to use Keybase (which they suggested), so it was better than the
alternative of having no communications channel. There's a lot we can
learn collectively as an industry :)

Jon.

-- 
Computer Architect | Sent from my Fedora powered laptop