BUG: early intel microcode update violating alignment rules

BUG: early intel microcode update violating alignment rules

[Henrique de Moraes Holschuh]

Hello x86 maintainers,

While adding some paranoia to the Intel Microcode driver, I noticed the
early Intel microcode update driver is not enforcing a 16-byte alignment
when it attempts to send the microcode update data to the processor via
WRMSR 79H.

This 16-byte alignment requirement is stated in the Intel SDM, vol 3A,
section 9.11.6, page 9-34.

Apparently at least some of the recent processors don't need the full
16-byte alignment, as a microcode update that is NOT aligned to 16 bytes,
but which is aligned to 4 bytes seems to work fine on a Xeon X5550 and on a
2nd-gen Core i5.

I detected the problem by adding an alignment test in
apply_microcode_early(), on file arch/x86/kernel/microcode_intel_early.c:

	if ((unsigned long)(mc_intel->bits) % 16)
		return -1;

Which fired on the next reboot, skipping the early update on the BSP and
APs.

This will only happen when the microcode update file data inside the early
initramfs archive is not 16-byte aligned.  Unfortunately, the cpio format
used in the early initramfs aligns the member file data just to 4-byte
boundaries.

Is there a way to fix this in the kernel for the BSP?  We already have
several microcode updates that are >16KiB in size and they seem to be
getting bigger and bigger, so doing it on the stack is out.

Maybe there is a 64KiB scratch area that could be used just to get the BSP
microcode update done?  vmalloc isn't available that early, but it can be
used to fix the issue for the APs.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: BUG: early intel microcode update violating alignment rules

[Borislav Petkov]

On Sat, Aug 09, 2014 at 08:19:11PM -0300, Henrique de Moraes Holschuh wrote:
> Is there a way to fix this in the kernel for the BSP?

I think you're looking at this the wrong way around. :-) The thing that
needs fixing is the SDM since some CPUs seem to accept 16-byte unaligned
microcode just fine.

Re: BUG: early intel microcode update violating alignment rules

[Henrique de Moraes Holschuh]

On Mon, 11 Aug 2014, Borislav Petkov wrote:
> On Sat, Aug 09, 2014 at 08:19:11PM -0300, Henrique de Moraes Holschuh wrote:
> > Is there a way to fix this in the kernel for the BSP?
> 
> I think you're looking at this the wrong way around. :-) The thing that
> needs fixing is the SDM since some CPUs seem to accept 16-byte unaligned
> microcode just fine.

I often wonder how much of the Intel SDM is really a fairy tale...  it
certainly has enough legends from times long past inside ;-)  But just like
old stories, should you forget all about them, they sometimes grow fangs
back and get you when you're least prepared.

Now, seriously, we're neither aligning the thing, nor checking any of it for
alignment, so userspace can mess with us at will.  Unless it is trying to be
actively malicious, we'll get 4-byte alignment out of userspace for the data
inside the early initramfs (assuming the use of the common cpio tools: GNU
cpio and GNU pax), but that's it.

I can easily propose fixes to reject incorrectly aligned data (and will do
so), but you *really* don't want to know the kind of crap I came up with to
try to align the microcode update for the BSP: Standard Lovecraftian Mythos
Safety Procedures apply!  So I am turning to you for ideas...

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: BUG: early intel microcode update violating alignment rules

[Borislav Petkov]

On Mon, Aug 11, 2014 at 10:16:13AM -0300, Henrique de Moraes Holschuh wrote:
> On Mon, 11 Aug 2014, Borislav Petkov wrote:
> > On Sat, Aug 09, 2014 at 08:19:11PM -0300, Henrique de Moraes Holschuh wrote:
> > > Is there a way to fix this in the kernel for the BSP?
> > 
> > I think you're looking at this the wrong way around. :-) The thing that
> > needs fixing is the SDM since some CPUs seem to accept 16-byte unaligned
> > microcode just fine.
> 
> I often wonder how much of the Intel SDM is really a fairy tale...  it
> certainly has enough legends from times long past inside ;-)  But just like
> old stories, should you forget all about them, they sometimes grow fangs
> back and get you when you're least prepared.
> 
> Now, seriously, we're neither aligning the thing, nor checking any of it for
> alignment, so userspace can mess with us at will.  Unless it is trying to be
> actively malicious, we'll get 4-byte alignment out of userspace for the data
> inside the early initramfs (assuming the use of the common cpio tools: GNU
> cpio and GNU pax), but that's it.
> 
> I can easily propose fixes to reject incorrectly aligned data (and will do
> so), but you *really* don't want to know the kind of crap I came up with to
> try to align the microcode update for the BSP: Standard Lovecraftian Mythos
> Safety Procedures apply!  So I am turning to you for ideas...

It seems to me you're looking for issues where there are none. We simply
have to ask Intel people what's with the 16-byte alignment and fix
the SDM, apparently. If the processor accepts the non-16-byte-aligned
update, why do you care?

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: BUG: early intel microcode update violating alignment rules

[H. Peter Anvin]

On 08/11/2014 07:00 AM, Borislav Petkov wrote:
>>
>> I often wonder how much of the Intel SDM is really a fairy tale...  it
>> certainly has enough legends from times long past inside ;-)  But just like
>> old stories, should you forget all about them, they sometimes grow fangs
>> back and get you when you're least prepared.
>>
> 
> It seems to me you're looking for issues where there are none. We simply
> have to ask Intel people what's with the 16-byte alignment and fix
> the SDM, apparently. If the processor accepts the non-16-byte-aligned
> update, why do you care?
> 

The SDM is the contract between the hardware and the software.  This
doesn't mean that not following the SDM doesn't work, but following the
procedure in the SDM is what his guaranteed to work.

So Henrique is right, we should fix this if we can.

	-hpa

Re: BUG: early intel microcode update violating alignment rules

[H. Peter Anvin]

On 08/09/2014 04:19 PM, Henrique de Moraes Holschuh wrote:
> 
> This will only happen when the microcode update file data inside the early
> initramfs archive is not 16-byte aligned.  Unfortunately, the cpio format
> used in the early initramfs aligns the member file data just to 4-byte
> boundaries.
> 
> Is there a way to fix this in the kernel for the BSP?  We already have
> several microcode updates that are >16KiB in size and they seem to be
> getting bigger and bigger, so doing it on the stack is out.
> 
> Maybe there is a 64KiB scratch area that could be used just to get the BSP
> microcode update done?  vmalloc isn't available that early, but it can be
> used to fix the issue for the APs.
> 

We could put a buffer in the initdata region (we really could use an
initbss region!) or in the brk.

	-hpa

Re: BUG: early intel microcode update violating alignment rules

[Henrique de Moraes Holschuh]

On Mon, Aug 11, 2014, at 11:51, H. Peter Anvin wrote:
> We could put a buffer in the initdata region (we really could use an
> initbss region!) or in the brk.

That sounds much better than the hideous crap I came up with.  The
buffer would need to be at least 64KiB in size to be on the safe side. 
The largest public microcode update ATM is 23KiB.

I am not sure if we might need more than 64KiB: the Intel SDM mentions
that in real mode the update data must not cross a segment boundary, and
also must not exceed a segment limit.  I am a bit rusty on real mode,
but doesn't that mean, in practice, that microcode update data size is
limited in size to 64KiB?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: BUG: early intel microcode update violating alignment rules

[H. Peter Anvin]

I don't think so.  I wouldn't count on that in the long run.

I'm having some discussions about this integrally at the moment.

On August 11, 2014 11:18:32 AM PDT, Henrique de Moraes Holschuh <hmh@hmh.eng.br> wrote:
>On Mon, Aug 11, 2014, at 11:51, H. Peter Anvin wrote:
>> We could put a buffer in the initdata region (we really could use an
>> initbss region!) or in the brk.
>
>That sounds much better than the hideous crap I came up with.  The
>buffer would need to be at least 64KiB in size to be on the safe side. 
>The largest public microcode update ATM is 23KiB.
>
>I am not sure if we might need more than 64KiB: the Intel SDM mentions
>that in real mode the update data must not cross a segment boundary,
>and
>also must not exceed a segment limit.  I am a bit rusty on real mode,
>but doesn't that mean, in practice, that microcode update data size is
>limited in size to 64KiB?

-- 
Sent from my mobile phone.  Please pardon brevity and lack of formatting.

Re: BUG: early intel microcode update violating alignment rules

[Bill Davidsen]

Borislav Petkov wrote:
> On Mon, Aug 11, 2014 at 10:16:13AM -0300, Henrique de Moraes Holschuh wrote:
>> On Mon, 11 Aug 2014, Borislav Petkov wrote:
>>> On Sat, Aug 09, 2014 at 08:19:11PM -0300, Henrique de Moraes Holschuh wrote:
>>>> Is there a way to fix this in the kernel for the BSP?
>>>
>>> I think you're looking at this the wrong way around. :-) The thing that
>>> needs fixing is the SDM since some CPUs seem to accept 16-byte unaligned
>>> microcode just fine.
>>
>> I often wonder how much of the Intel SDM is really a fairy tale...  it
>> certainly has enough legends from times long past inside ;-)  But just like
>> old stories, should you forget all about them, they sometimes grow fangs
>> back and get you when you're least prepared.
>>
>> Now, seriously, we're neither aligning the thing, nor checking any of it for
>> alignment, so userspace can mess with us at will.  Unless it is trying to be
>> actively malicious, we'll get 4-byte alignment out of userspace for the data
>> inside the early initramfs (assuming the use of the common cpio tools: GNU
>> cpio and GNU pax), but that's it.
>>
>> I can easily propose fixes to reject incorrectly aligned data (and will do
>> so), but you *really* don't want to know the kind of crap I came up with to
>> try to align the microcode update for the BSP: Standard Lovecraftian Mythos
>> Safety Procedures apply!  So I am turning to you for ideas...
>
> It seems to me you're looking for issues where there are none. We simply
> have to ask Intel people what's with the 16-byte alignment and fix
> the SDM, apparently. If the processor accepts the non-16-byte-aligned
> update, why do you care?
>
Because if the requirement is enforced in some future revision, and updates then 
fail in some insane way, the vendor is justified in claiming "I told you so."

Don't suppose you have anything in memory right after the microcode which you 
could put on the stack (15 bytes) slide the image up into alignment, load it, 
and put everything back. Haven't looked at the code or data, just tossing out an 
idea I used for something else back when.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: BUG: early intel microcode update violating alignment rules

[Borislav Petkov]

On Mon, Aug 11, 2014 at 02:52:39PM -0400, Bill Davidsen wrote:
> Because if the requirement is enforced in some future revision, and
> updates then fail in some insane way, the vendor is justified in
> claiming "I told you so."

... provided that requirement is not some old relic which is not needed
anymore. I was questioning it, considering the fact that CPUs don't
enforce it (if they did, we'd be seeing explosions left and right).

Anyway, hpa is checking the whole deal internally as we speak.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: BUG: early intel microcode update violating alignment rules

[Henrique de Moraes Holschuh]

On Mon, 11 Aug 2014, Bill Davidsen wrote:
> Don't suppose you have anything in memory right after the microcode
> which you could put on the stack (15 bytes) slide the image up into
> alignment, load it, and put everything back. Haven't looked at the
> code or data, just tossing out an idea I used for something else
> back when.

Right after?  We don't know what will be there.  But we know what is right
before: there's enough bytes worth of microcode metadata header to align the
real microcode data to the previous 16-byte boundary.

This is a fix I should be fully capable of writing, actually.  But it will
be slower (two memcpy, two memove) and much nastier than the proper solution
using a scratch area (one memcpy).

That said, it is reported[1] that an Intel microcode update will take
anything from a couple hundred-thousand to several million cycles per
processor core.  I wouldn't lose any sleep over adding two 16-byte memcpy
and two ~32KiB memove to something that is already that expensive.  And it
will be easy to backport.

Anyway, I don't know enough to either add an initbss, or change the brk as
HPA hinted, or how easy would it be to backport that, so I will wait for
further directions.

Meanwhile, I prepared an iucode-tool v1.0.3 release that works around this
kernel issue for the benefit of the linux distros that use it[2].  It's at
https://gitorious.org/iucode-tool if anyone around here is interested.

[1] http://inertiawar.com/microcode/

[2] Debian, Ubuntu and their derivatives.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

early microcode: how to disable at runtime?

[Henrique de Moraes Holschuh]

Currently, there is no way to disable early microcode update at runtime.  If
an early initramfs with microcode update data is available, it will always
be used.

This can be a very big deal when things go wrong: it is hard for the regular
user to recover from an initramfs image that crashes the system, and the
early initramfs has no "disable" trigger.

In the general case, booting from rescue media will be required to fix the
system.

Unfortunately, the BSP microcode update is handled before early_param() is
available, so a kernel command line parameter to disable early microcode
updates for the BSP isn't easily possible.

Other than duplicating much of the code in parse_early_param/parse_args to
search for a "noearlymcu" parameter, is there any other way we could add a
disable switch for the early microcode update system?

Otherwise, distros will have to generate a rescue initramfs image
automatically that has the entire early initramfs stripped, or somehow get
the bootloaders (and mkinitramfs/dracut) to deal (correctly) with two
initramfs images.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: early microcode: how to disable at runtime?

[H. Peter Anvin]

On 08/31/2014 12:45 PM, Henrique de Moraes Holschuh wrote:
> Currently, there is no way to disable early microcode update at runtime.  If
> an early initramfs with microcode update data is available, it will always
> be used.
> 
> This can be a very big deal when things go wrong: it is hard for the regular
> user to recover from an initramfs image that crashes the system, and the
> early initramfs has no "disable" trigger.
> 
> In the general case, booting from rescue media will be required to fix the
> system.
> 
> Unfortunately, the BSP microcode update is handled before early_param() is
> available, so a kernel command line parameter to disable early microcode
> updates for the BSP isn't easily possible.
> 
> Other than duplicating much of the code in parse_early_param/parse_args to
> search for a "noearlymcu" parameter, is there any other way we could add a
> disable switch for the early microcode update system?
> 

Probably more like something more like arch/x86/boot/cmdline.c.  Maybe
we could even make it sharable.

	-hpa

Re: early microcode: how to disable at runtime?

[Henrique de Moraes Holschuh]

On Sun, 31 Aug 2014, H. Peter Anvin wrote:
> On 08/31/2014 12:45 PM, Henrique de Moraes Holschuh wrote:
> > Currently, there is no way to disable early microcode update at runtime.  If
> > an early initramfs with microcode update data is available, it will always
> > be used.
> > 
> > This can be a very big deal when things go wrong: it is hard for the regular
> > user to recover from an initramfs image that crashes the system, and the
> > early initramfs has no "disable" trigger.
> > 
> > In the general case, booting from rescue media will be required to fix the
> > system.
> > 
> > Unfortunately, the BSP microcode update is handled before early_param() is
> > available, so a kernel command line parameter to disable early microcode
> > updates for the BSP isn't easily possible.
> > 
> > Other than duplicating much of the code in parse_early_param/parse_args to
> > search for a "noearlymcu" parameter, is there any other way we could add a
> > disable switch for the early microcode update system?
> > 
> 
> Probably more like something more like arch/x86/boot/cmdline.c.  Maybe
> we could even make it sharable.

Looks like that would work, yes.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: early microcode: how to disable at runtime?

[Borislav Petkov]

On Sun, Aug 31, 2014 at 05:11:26PM -0300, Henrique de Moraes Holschuh wrote:
> > Probably more like something more like arch/x86/boot/cmdline.c.  Maybe
> > we could even make it sharable.
> 
> Looks like that would work, yes.

Guys, guys, please relax and check the code first: "dis_ucode_ldr".

-- 
Regards/Gruss,
    Boris.
--

Re: early microcode: how to disable at runtime?

[Henrique de Moraes Holschuh]

On Mon, 01 Sep 2014, Borislav Petkov wrote:
> On Sun, Aug 31, 2014 at 05:11:26PM -0300, Henrique de Moraes Holschuh wrote:
> > > Probably more like something more like arch/x86/boot/cmdline.c.  Maybe
> > > we could even make it sharable.
> > 
> > Looks like that would work, yes.
> 
> Guys, guys, please relax and check the code first: "dis_ucode_ldr".

Thanks!

Now we just need it added to Documentation/kernel-parameters.txt and to get
it backported to -stable.

This parameter is not a "debugging aid", it is a "save the day" bug
workaround aid...

I can't say I like the parameter name, though :p

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: early microcode: how to disable at runtime?

[H. Peter Anvin]

On 08/31/2014 10:13 PM, Borislav Petkov wrote:
> On Sun, Aug 31, 2014 at 05:11:26PM -0300, Henrique de Moraes Holschuh wrote:
>>> Probably more like something more like arch/x86/boot/cmdline.c.  Maybe
>>> we could even make it sharable.
>>
>> Looks like that would work, yes.
> 
> Guys, guys, please relax and check the code first: "dis_ucode_ldr".
> 

Yeah.  It is an absolutely horrific option name (something like
"noearlyucode" would have been so much nicer; we generally use the no-
prefix and no underscores) but I should have objected to that one much
sooner.

	-hpa

Re: early microcode: how to disable at runtime?

[Henrique de Moraes Holschuh]

On Mon, 01 Sep 2014, H. Peter Anvin wrote:
> On 08/31/2014 10:13 PM, Borislav Petkov wrote:
> > On Sun, Aug 31, 2014 at 05:11:26PM -0300, Henrique de Moraes Holschuh wrote:
> >>> Probably more like something more like arch/x86/boot/cmdline.c.  Maybe
> >>> we could even make it sharable.
> >>
> >> Looks like that would work, yes.
> > 
> > Guys, guys, please relax and check the code first: "dis_ucode_ldr".
> > 
> 
> Yeah.  It is an absolutely horrific option name (something like
> "noearlyucode" would have been so much nicer; we generally use the no-
> prefix and no underscores) but I should have objected to that one much
> sooner.

It wasn't properly documented, it is obscure as all heck, and early
microcode updates active by default is a very recent development on distros.

You cannot even find "dis_ucode_ldr" being cited by anyone through google
searches.

I believe it is actually possible to fix this still: either by directly
renaming, or by using a deprecation period and testing for a new name and
the "dis_ucode_ldr" name.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: early microcode: how to disable at runtime?

[H. Peter Anvin]

No, it is worse to rename it and have older kernels behave differently.  Kernel options are basically super obscure to most people anyway.

On September 1, 2014 10:43:22 AM PDT, Henrique de Moraes Holschuh <hmh@hmh.eng.br> wrote:
>On Mon, 01 Sep 2014, H. Peter Anvin wrote:
>> On 08/31/2014 10:13 PM, Borislav Petkov wrote:
>> > On Sun, Aug 31, 2014 at 05:11:26PM -0300, Henrique de Moraes
>Holschuh wrote:
>> >>> Probably more like something more like arch/x86/boot/cmdline.c. 
>Maybe
>> >>> we could even make it sharable.
>> >>
>> >> Looks like that would work, yes.
>> > 
>> > Guys, guys, please relax and check the code first: "dis_ucode_ldr".
>> > 
>> 
>> Yeah.  It is an absolutely horrific option name (something like
>> "noearlyucode" would have been so much nicer; we generally use the
>no-
>> prefix and no underscores) but I should have objected to that one
>much
>> sooner.
>
>It wasn't properly documented, it is obscure as all heck, and early
>microcode updates active by default is a very recent development on
>distros.
>
>You cannot even find "dis_ucode_ldr" being cited by anyone through
>google
>searches.
>
>I believe it is actually possible to fix this still: either by directly
>renaming, or by using a deprecation period and testing for a new name
>and
>the "dis_ucode_ldr" name.

-- 
Sent from my mobile phone.  Please pardon brevity and lack of formatting.

Re: early microcode: how to disable at runtime?

[Henrique de Moraes Holschuh]

On Mon, 01 Sep 2014, H. Peter Anvin wrote:
> No, it is worse to rename it and have older kernels behave differently.
> Kernel options are basically super obscure to most people anyway.

Ok, so rename is out, although as long as the rename was accepted for
3.16-stable, it would be just 3.16 and 3.16.1 (and maybe 3.16.2) that would
not have the new name.

Options left are "keep it as is", or "add new and [optionally] deprecate
old".  I can't really tell for sure which one you'd prefer from your
previous answer.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: early microcode: how to disable at runtime?

[Borislav Petkov]

On Mon, Sep 01, 2014 at 04:59:21PM -0300, Henrique de Moraes Holschuh wrote:
> On Mon, 01 Sep 2014, H. Peter Anvin wrote:
> > No, it is worse to rename it and have older kernels behave differently.
> > Kernel options are basically super obscure to most people anyway.
> 
> Ok, so rename is out, although as long as the rename was accepted for
> 3.16-stable, it would be just 3.16 and 3.16.1 (and maybe 3.16.2) that would
> not have the new name.
> 
> Options left are "keep it as is", or "add new and [optionally] deprecate
> old".  I can't really tell for sure which one you'd prefer from your
> previous answer.

Let me answer to all the concerns here:

* the parameter is not in kernel-parameters.txt for the simple reason
that we don't want people to disable ucode loaders left and right.
The absolutely normal, 99% situation should be loading works fine or
we don't load the ucode at all - this option is only for the very,
very, very! obscure and strange situation where we want to rule out any
microcode interference. Just in case.

This situation should almost never happen.

* The param naming could be argued about - maybe it is ugly and
unreadable for that same reason.

> This can be a very big deal when things go wrong: it is hard for the
> regular user to recover from an initramfs image that crashes the
> system, and the early initramfs has no "disable" trigger.

This maybe is a serious problem but disabling microcode loading is not
the proper solution. If the microcode in the initrd is corrupted, it
should simply not get loaded and system should continue as much as it
can - it *should* *not* be a requirement to disable the ucode loader in
order to workaround corrupted initrds.

And frankly, I'm trying to imagine how a corrupted microcode in an
initrd would ever fail the booting. So Henrique, if you have something
which is *not* hypothetical, please say so. It needs to be fixed
properly and not with disabling the ucode loader.

-- 
Regards/Gruss,
    Boris.
--

Re: early microcode: how to disable at runtime?

[Henrique de Moraes Holschuh]

On Tue, 02 Sep 2014, Borislav Petkov wrote:
> > This can be a very big deal when things go wrong: it is hard for the
> > regular user to recover from an initramfs image that crashes the
> > system, and the early initramfs has no "disable" trigger.
> 
> This maybe is a serious problem but disabling microcode loading is not
> the proper solution. If the microcode in the initrd is corrupted, it
> should simply not get loaded and system should continue as much as it
> can - it *should* *not* be a requirement to disable the ucode loader in
> order to workaround corrupted initrds.

Things do go wrong in other ways, not just corrupt microcode data/initramfs
images.

This stuff runs too early.  It is easy to break, and annoying to debug.

> And frankly, I'm trying to imagine how a corrupted microcode in an
> initrd would ever fail the booting. So Henrique, if you have something
> which is *not* hypothetical, please say so. It needs to be fixed
> properly and not with disabling the ucode loader.

I am not worried about corrupted microcode, or even corrupted initramfs
images.  I am just worried about regular bugs introduced by a kernel update
causing systems to fail to reboot later.

I don't want to tell an user he has to use rescue media to get his system
back into working order when a kernel parameter would have been enough to
get it to boot.

Although, on the corrupted microcode topic, I have this very strong feeling
that at least the Intel driver would benefit from a careful audit on the
microcode container handling if we want to be sure it won't do stupid things
when fed specially crafted hostile data.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: early microcode: how to disable at runtime?

[Borislav Petkov]

On Tue, Sep 02, 2014 at 10:16:51AM -0300, Henrique de Moraes Holschuh wrote:
> Things do go wrong in other ways, not just corrupt microcode data/initramfs
> images.
> 
> This stuff runs too early.  It is easy to break, and annoying to debug.

How is this stuff easy to break? Please stop with the conjectures and
give concrete issues, if any.

> Although, on the corrupted microcode topic, I have this very strong
> feeling that at least the Intel driver would benefit from a careful
> audit on the microcode container handling if we want to be sure it
> won't do stupid things when fed specially crafted hostile data.

If loading the microcode can disrupt the system (and not simply be
ignored if faulty) then we have to fix that. Disabling the loader is only
a stop-gap measure in such situation anyway - not a real solution.

-- 
Regards/Gruss,
    Boris.
--

Re: early microcode: how to disable at runtime?

[Henrique de Moraes Holschuh]

On Wed, 03 Sep 2014, Borislav Petkov wrote:
> On Tue, Sep 02, 2014 at 10:16:51AM -0300, Henrique de Moraes Holschuh wrote:
> > Things do go wrong in other ways, not just corrupt microcode data/initramfs
> > images.
> > 
> > This stuff runs too early.  It is easy to break, and annoying to debug.
> 
> How is this stuff easy to break? Please stop with the conjectures and
> give concrete issues, if any.

Half the things you'd usually do when writing kernel code cannot be done
that early, that's how.  It is not that the code is bad or especially
fragile (at least not the core or the AMD driver.  The Intel driver is
convoluted).

You have fixed several such bugs recently, commit ids:
75a1ba5b2c529db60ca49626bcaf0bddf4548438,
84516098b58e05821780dc0b89abcee434b4dca5,
5335ba5cf475369f88db8e6835764efdcad8ab96.

> > Although, on the corrupted microcode topic, I have this very strong
> > feeling that at least the Intel driver would benefit from a careful
> > audit on the microcode container handling if we want to be sure it
> > won't do stupid things when fed specially crafted hostile data.
> 
> If loading the microcode can disrupt the system (and not simply be
> ignored if faulty) then we have to fix that. Disabling the loader is only
> a stop-gap measure in such situation anyway - not a real solution.

Yes.  It is a stop gap solution, I never claimed it to be anything else.

What I did claim is that it is going to be a far more user-friendly stop gap
than the "boot from rescue media" option.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: early microcode: how to disable at runtime?

[Borislav Petkov]

On Wed, Sep 03, 2014 at 09:43:02AM -0300, Henrique de Moraes Holschuh wrote:
> Half the things you'd usually do when writing kernel code cannot be done
> that early, that's how.  It is not that the code is bad or especially
> fragile (at least not the core or the AMD driver.  The Intel driver is
> convoluted).

Right, it could use some cleaning up.

> You have fixed several such bugs recently, commit ids:
> 75a1ba5b2c529db60ca49626bcaf0bddf4548438,
> 84516098b58e05821780dc0b89abcee434b4dca5,
> 5335ba5cf475369f88db8e6835764efdcad8ab96.

Those were unfortunate because the original patches didn't get tested
enough and got applied. This is not happening again on my watch.

> Yes. It is a stop gap solution, I never claimed it to be anything
> else.
>
> What I did claim is that it is going to be a far more user-friendly
> stop gap than the "boot from rescue media" option.

Ok, we have it now and we can use it if needed. All is good.

-- 
Regards/Gruss,
    Boris.
--

Re: early microcode: how to disable at runtime?

[Henrique de Moraes Holschuh]

On Thu, 04 Sep 2014, Borislav Petkov wrote:
> > Yes. It is a stop gap solution, I never claimed it to be anything
> > else.
> >
> > What I did claim is that it is going to be a far more user-friendly
> > stop gap than the "boot from rescue media" option.
> 
> Ok, we have it now and we can use it if needed. All is good.

Almost :-)

Can we get it backported to -stable, please (3.10 and later)?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

RE: early microcode: how to disable at runtime?

[Yu, Fenghua]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 847 bytes --]

> From: Borislav Petkov [mailto:bp@alien8.de]
> > What I did claim is that it is going to be a far more user-friendly
> > stop gap than the "boot from rescue media" option.
> 
> Ok, we have it now and we can use it if needed. All is good.

Quoted comment from commit 65cef1311d5d212fd3d48a43678536dc878ca288:
"...This is useful mostly in debugging situations where we want to turn off microcode
    loading, both early from the initrd and late, ...."

In case this comment may cause confusion, I need to confirm that:

The parameter "dis_ucode_ldr" only turns off early microcode loading from the initrd.
It does not trun off loading microcode later during run time.

Thanks.

-Fenghua

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: early microcode: how to disable at runtime?

[Borislav Petkov]

On Thu, Sep 04, 2014 at 02:45:10PM -0300, Henrique de Moraes Holschuh wrote:
> Can we get it backported to -stable, please (3.10 and later)?

I don't see how stable rules apply here.

-- 
Regards/Gruss,
    Boris.
--

Re: early microcode: how to disable at runtime?

[Borislav Petkov]

On Thu, Sep 04, 2014 at 06:37:05PM +0000, Yu, Fenghua wrote:
> In case this comment may cause confusion, I need to confirm that:
> 
> The parameter "dis_ucode_ldr" only turns off early microcode loading from the initrd.
> It does not trun off loading microcode later during run time.

Are you asking or are you saying?

Because it should disable loading of the microcode machinery later too -
see microcode_init().

-- 
Regards/Gruss,
    Boris.
--

Re: early microcode: how to disable at runtime?

[Henrique de Moraes Holschuh]

On Fri, 05 Sep 2014, Borislav Petkov wrote:
> On Thu, Sep 04, 2014 at 02:45:10PM -0300, Henrique de Moraes Holschuh wrote:
> > Can we get it backported to -stable, please (3.10 and later)?
> 
> I don't see how stable rules apply here.

It would be better if this functionality was generally available where early
microcode loading is supported.  As far as stable/longterm kernel.org
kernels are concerned, I believe that'd be 3.10, 3.12 and 3.14.

But it is no big deal as far as I'm concerned.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh