Loading Pentium III microcode under Linux - catch 22!

Loading Pentium III microcode under Linux - catch 22!

[Chris Rankin]

Tigran,

I have an i840 motherboard with a pair of 933 MHz PIII
Coppermine CPUs, and I use your microcode driver to
load the latest Intel microcode into my CPUs. This is
very important because these CPUs are buggy without
their microcode, and I would prefer to have the BIOS
load it except that this would prevent me from booting
into memtest. I have tried this before - memtest
crashes with an "Unexpected Interrupt" error after a
few minutes. (No i840 workarounds enabled?) Since I
suspect that DOS would do the same thing and I would
boot into DOS to flash firmware, I have decided that
crashes like this would be a Bad Thing.

I have modified by boot scripts to load the microcode
as soon as the root filesystem has been successfully
mounted. However, this means that kernel always boots
on buggy CPUs! For example, last night my boot failed
just after releasing the unused kernel memory. I
suspect that the record temperatures that my part of
the world is currently experiencing is adversely
influencing things. My boot-ups are usually fine.

In an ideal world, I would like Linux to load the
microcode *before* the kernel boots, which begs the
question of "How?". Can you suggest anything, please?
I remember talk of boot-time RAM disks, and wondered
if the microcode could be placed on one of these
somehow? Or would that be ruled out immediately by the
microcode's non-GPL nature?

Any suggestions gratefully received,
Thanks,
Chris Rankin


________________________________________________________________________
Want to chat instantly with your online friends?  Get the FREE Yahoo!
Messenger http://uk.messenger.yahoo.com/

Re: Loading Pentium III microcode under Linux - catch 22!

[Måns Rullgård]

Chris Rankin <rankincj@yahoo.com> writes:

> I have an i840 motherboard with a pair of 933 MHz PIII
> Coppermine CPUs, and I use your microcode driver to
> load the latest Intel microcode into my CPUs. This is
> very important because these CPUs are buggy without
> their microcode, and I would prefer to have the BIOS
> load it except that this would prevent me from booting
> into memtest. I have tried this before - memtest
> crashes with an "Unexpected Interrupt" error after a
> few minutes. (No i840 workarounds enabled?) Since I
> suspect that DOS would do the same thing and I would
> boot into DOS to flash firmware, I have decided that
> crashes like this would be a Bad Thing.

If the microcode in the CPUs is buggy, they are faulty and you should
demand to get them replaced at no cost.

> In an ideal world, I would like Linux to load the
> microcode *before* the kernel boots, which begs the
> question of "How?". Can you suggest anything, please?
> I remember talk of boot-time RAM disks, and wondered
> if the microcode could be placed on one of these
> somehow? Or would that be ruled out immediately by the
> microcode's non-GPL nature?

I guess it would be possible to compile the microcode into the kernel
and have some code in arch/i386/* load it as early as possible.  As
long as you don't distribute the compiled kernel you should be fine
wrt licensing.

-- 
Måns Rullgård
mru@users.sf.net

Re: Loading Pentium III microcode under Linux - catch 22!

[Alan Cox]

On Iau, 2003-08-07 at 13:28, Chris Rankin wrote:
> I have modified by boot scripts to load the microcode
> as soon as the root filesystem has been successfully
> mounted. However, this means that kernel always boots
> on buggy CPUs! For example, last night my boot failed
> just after releasing the unused kernel memory. I
> suspect that the record temperatures that my part of
> the world is currently experiencing is adversely
> influencing things. My boot-ups are usually fine

As far as I am aware none of the microcode updates even apply
to 933Mhz era PIII, just the ones the BIOS ships with by default
nowdays. Also the kind of stuff the errata fix are obscure 
ultra-weird corner cases people just don't hit.

Thus I'd be very suprised if loading the microcode any earlier
was neccessary - certainly nobody else has reported needing to.

> In an ideal world, I would like Linux to load the
> microcode *before* the kernel boots, which begs the
> question of "How?". Can you suggest anything, please?

The kernel can't load the microcode until it has booted, it can
load it very early after that from initrd.

Re: Loading Pentium III microcode under Linux - catch 22!

[Chris Rankin]

 --- Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> As far as I am aware none of the microcode updates
> even apply to 933Mhz era PIII, just the ones the
> BIOS ships with by default nowdays. Also the kind of
> stuff the errata fix are obscure ultra-weird corner
> cases people just don't hit.

Lucky me, eh?

My CPUs *do* take microcode, and they are 933 MHz...
;-). I upgraded from a pair of 733 MHz CPUs bought in
July 2000, and my current BIOS just doesn't have any
microcode for them. Without the update, I used to come
back at the end of the day, switch on the KVM and be
unable to use the keyboard and mouse.

Anyway, I wasn't aware that Intel had released a
changelist for their microcode updates. Goodness knows
what bugs they're fixing.

> Thus I'd be very suprised if loading the microcode
> any earlier was neccessary - certainly nobody else
> has reported needing to.

My machine is currently hanging during boot-up (while
fscking the root partition, or releasing the "init"
memory), and CPU malfunction is the leading candidate
explanation. I have already replaced the 300W PSU with
a 400W one and tested the memory.

> > In an ideal world, I would like Linux to load the
> > microcode *before* the kernel boots, which begs
> > the question of "How?". Can you suggest anything,
> > please?
> 
> The kernel can't load the microcode until it has
> booted,

Yes, that's the "catch-22" bit. I was originally
thinking about either a bootstrapping floppy disk, or
maybe hacking some code into the boot-up sequence
itself.

> it can load it very early after that from initrd.

OK, I'll look into that.

Cheers,
Chris



________________________________________________________________________
Want to chat instantly with your online friends?  Get the FREE Yahoo!
Messenger http://uk.messenger.yahoo.com/

Re: Loading Pentium III microcode under Linux - catch 22!

[Alan Cox]

On Iau, 2003-08-07 at 15:38, Chris Rankin wrote:
> July 2000, and my current BIOS just doesn't have any
> microcode for them. Without the update, I used to come
> back at the end of the day, switch on the KVM and be
> unable to use the keyboard and mouse.

Sounds believable

> Anyway, I wasn't aware that Intel had released a
> changelist for their microcode updates. Goodness knows
> what bugs they're fixing.

Generally speaking its the ones that end  "Can be worked around by
the BIOS" in the big list of errata. Some of those are however
other things like setting timing registers or turning off features
via semi-secret mtrr registers

> memory), and CPU malfunction is the leading candidate
> explanation. I have already replaced the 300W PSU with
> a 400W one and tested the memory.

Ok

> Yes, that's the "catch-22" bit. I was originally
> thinking about either a bootstrapping floppy disk, or
> maybe hacking some code into the boot-up sequence
> itself.
> 
> > it can load it very early after that from initrd.
> 
> OK, I'll look into that.

Looking at it you can do it in initrd fine, or you can do it
as the first thing you do once the real root fs is mounted
from init's scripts (/etc/rc.sysinit normally)

Re: Loading Pentium III microcode under Linux - catch 22!

[Tomas Szepe]

> [alan@lxorguk.ukuu.org.uk]
> 
> On Iau, 2003-08-07 at 15:38, Chris Rankin wrote:
> > July 2000, and my current BIOS just doesn't have any
> > microcode for them. Without the update, I used to come
> > back at the end of the day, switch on the KVM and be
> > unable to use the keyboard and mouse.
> 
> Sounds believable

Sure does.  There are machines (like the HP tc2100) which
always freeze entirely as soon as a PS/2 keyboard is attached.

> > > it can load it very early after that from initrd.
> > 
> > OK, I'll look into that.
> 
> Looking at it you can do it in initrd fine, or you can do it
> as the first thing you do once the real root fs is mounted
> from init's scripts (/etc/rc.sysinit normally)

and /etc/rc.d/rc.S on BSDish systems I believe.

-- 
Tomas Szepe <szepe@pinerecords.com>

Re: Loading Pentium III microcode under Linux - catch 22!

[Chris Rankin]

 --- Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> Looking at it you can do it in initrd fine, or you
> can do it as the first thing you do once the real
> root fs is mounted from init's scripts
> (/etc/rc.sysinit normally)

I'm doing the latter right already. The problem is
that even using ext3, I'm occasionally fscking the
root partition with dodgy CPUs after a lock-up.
Hopefully, initrd will allow me to load the microcode
earlier in the boot sequence.

Cheers,
Chris



________________________________________________________________________
Want to chat instantly with your online friends?  Get the FREE Yahoo!
Messenger http://uk.messenger.yahoo.com/

Re: Loading Pentium III microcode under Linux - catch 22!

[Eyal Lebedinsky]

Chris Rankin wrote:
> 
>  --- Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> > As far as I am aware none of the microcode updates
> > even apply to 933Mhz era PIII, just the ones the
> > BIOS ships with by default nowdays. Also the kind of
> > stuff the errata fix are obscure ultra-weird corner
> > cases people just don't hit.
> 
> Lucky me, eh?
> 
> My CPUs *do* take microcode, and they are 933 MHz...
> ;-). I upgraded from a pair of 733 MHz CPUs bought in

Some random ideas:
	- reinstall the slower, stable CPUs and burn the BIOS
		If you are a linux hacker then you kept these
		"just in case".. or else
	- get a loaner CPU and use it to burn the BIOS
		If you are a linux hacker then you have no
		life and no friends to get a loaner from but
		you do have a roomfull of computer parts...
		otherwise
	- install only one CPU, it may be more stable
	- run your CPU slower than spec, it may be more stable
		most microcode bugs are not timining sensitive
		but it is worth a try

Good luck.

--
Eyal Lebedinsky (eyal@eyal.emu.id.au) <http://samba.org/eyal/>

Re: Loading Pentium III microcode under Linux - catch 22!

[Tigran Aivazian]

On 7 Aug 2003, Alan Cox wrote:
> > In an ideal world, I would like Linux to load the
> > microcode *before* the kernel boots, which begs the
> > question of "How?". Can you suggest anything, please?
> 
> The kernel can't load the microcode until it has booted, it can
> load it very early after that from initrd.

Well, it could even do it a little bit earlier, i.e. if we allowed linking
microcode data into the kernel image (as some scsi firmware data is, etc
and as __initdata so it's thrown away later) then it could update it very
very early, i.e. even before mounting the root filesystem. All it needs is
the ability to do IPIs and get the data from somewhere.

I could implement this, but if you tell me that this is not allowed
because of the GPL issues (microcode data chunks are copyrighted by Intel)
then obviously I won't waste time writing the code to do this.

Alternatively, yes, he can do it from within initrd. Hope that is early 
enough for him.

Kind regards
Tigran

Re: Loading Pentium III microcode under Linux - catch 22!

[Alan Cox]

On Iau, 2003-08-07 at 16:57, Tigran Aivazian wrote:
> I could implement this, but if you tell me that this is not allowed
> because of the GPL issues (microcode data chunks are copyrighted by Intel)
> then obviously I won't waste time writing the code to do this.

Thats something we are moving away from, so that we can load firmware
from the initrd. Another interesting place to tackle it might be in lilo
or grub ?

Re: Loading Pentium III microcode under Linux - catch 22!

[Chris Rankin]

 --- Tigran Aivazian <tigran@veritas.com> wrote:
> Alternatively, yes, he can do it from within initrd.
> Hope that is early 
> enough for him.

Hi,

Yes, I think I'll go the initrd route. However,
hammering on the machine last night reproduced the
malfunction regardless of whether the microcode was
loaded or not!

(Addressing list in general:)

This board (Supermicro PIIIDME, i840 chipset) has been
stable for almost 3 years, since between 2.4.0-pre6
and 2.4.0-pre9 IIRC. All recent hangs and lock-ups
have been attributable to the IO subsystem in 2.4.20,
and it just ran 2.4.21 for 3 solid weeks. The only
thing that has really changed recently is the weather,
but the lm_sensors package (2.6.5) reports that
everything is fine.

Booting with "acpi=off" has so far prevented the
kernel from hanging just after the "Freeing unused
kernel memory" message. However, the kernel still hung
later: the first time was when reading lm_sensors
after one CPU had reached the magic BIOS setting of 55
degrees C under high load. This hang took out the
keyboard, mouse and serial console straight away, but
left the ethernet interface running for a few minutes.
The second hang was more spontaneous, and killed the
box outright.

Do ATX power supplies have temperature sensors built
into them? I have just upgraded my PSU from 300W to a
brand new 400W one, so doubt that it's faulty. Could
the power supply be triggering a temperature-related
ACPI event? I seem to recall the boot-log mentioning
assorted S and C events being supported. (Can't
remember which ones exactly: S0 and S3?)

And I have already run memtest for 2 hours without
problems.

Cheers,
Chris


________________________________________________________________________
Want to chat instantly with your online friends?  Get the FREE Yahoo!
Messenger http://uk.messenger.yahoo.com/

RE: Loading Pentium III microcode under Linux - catch 22!

[Nakajima, Jun]

One issue with doing it in lilo or grub is that we need to apply
microcode update to each (physical) CPU. So we need to change the
current driver structure if we want to move it to the boot loader. So
one of the other ways is:
1. Boot loader copies update to memory
2. Kernel applies update soon after it brings the processors (including
BP, and APs) online in init_intel() or something like that. 

We don't need IPI or even the microcode update driver if we do this. But
I think putting it in initrd should be sufficient.

Thanks,
Jun

> -----Original Message-----
> From: Alan Cox [mailto:alan@lxorguk.ukuu.org.uk]
> Sent: Thursday, August 07, 2003 8:55 AM
> To: Tigran Aivazian
> Cc: Chris Rankin; Linux Kernel Mailing List
> Subject: Re: Loading Pentium III microcode under Linux - catch 22!
> 
> On Iau, 2003-08-07 at 16:57, Tigran Aivazian wrote:
> > I could implement this, but if you tell me that this is not allowed
> > because of the GPL issues (microcode data chunks are copyrighted by
> Intel)
> > then obviously I won't waste time writing the code to do this.
> 
> Thats something we are moving away from, so that we can load firmware
> from the initrd. Another interesting place to tackle it might be in
lilo
> or grub ?
> 
> -
> To unsubscribe from this list: send the line "unsubscribe
linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

RE: Loading Pentium III microcode under Linux - catch 22!

[Alan Cox]

On Iau, 2003-08-07 at 18:40, Nakajima, Jun wrote:
> One issue with doing it in lilo or grub is that we need to apply
> microcode update to each (physical) CPU. So we need to change the
> current driver structure if we want to move it to the boot loader.

Not really. The boot loader loads the kernel and initrd image and runs
the kernel. The initrd is a file system which can contain things like
the microcode loader and the microcode sets, just like it can contain
anything else.