RE: Loading Pentium III microcode under Linux - catch 22!

RE: Loading Pentium III microcode under Linux - catch 22!

[Nakajima, Jun]

One issue with doing it in lilo or grub is that we need to apply
microcode update to each (physical) CPU. So we need to change the
current driver structure if we want to move it to the boot loader. So
one of the other ways is:
1. Boot loader copies update to memory
2. Kernel applies update soon after it brings the processors (including
BP, and APs) online in init_intel() or something like that. 

We don't need IPI or even the microcode update driver if we do this. But
I think putting it in initrd should be sufficient.

Thanks,
Jun

> -----Original Message-----
> From: Alan Cox [mailto:alan@lxorguk.ukuu.org.uk]
> Sent: Thursday, August 07, 2003 8:55 AM
> To: Tigran Aivazian
> Cc: Chris Rankin; Linux Kernel Mailing List
> Subject: Re: Loading Pentium III microcode under Linux - catch 22!
> 
> On Iau, 2003-08-07 at 16:57, Tigran Aivazian wrote:
> > I could implement this, but if you tell me that this is not allowed
> > because of the GPL issues (microcode data chunks are copyrighted by
> Intel)
> > then obviously I won't waste time writing the code to do this.
> 
> Thats something we are moving away from, so that we can load firmware
> from the initrd. Another interesting place to tackle it might be in
lilo
> or grub ?
> 
> -
> To unsubscribe from this list: send the line "unsubscribe
linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

RE: Loading Pentium III microcode under Linux - catch 22! BOARD NOW STABLE!

[Chris Rankin]

 --- "Nakajima, Jun" <jun.nakajima@intel.com> wrote:
> We don't need IPI or even the microcode update
> driver if we do this. But I think putting it in
> initrd should be sufficient.

Well, I don't know about "sufficient". Someone on
"www.hardwareanalysis.com" suggested underclocking my
CPUs. So my dual 933 MHz CPUs are now running at 700
MHz (FSB changed from 133 MHz to 100 MHz) and
everything has been fine for the past 3 hours; same
load as when everything hit the fan last week too. Of
course, the CPUs run cooler like this, so I can't
ignore the thermal element...

$ more /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 8
model name      : Pentium III (Coppermine)
stepping        : 6
cpu MHz         : 696.033
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8
apic sep mtrr pge mca cmov 
pat pse36 mmx fxsr sse
bogomips        : 1389.36

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 8
model name      : Pentium III (Coppermine)
stepping        : 6
cpu MHz         : 696.033
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8
apic sep mtrr pge mca cmov 
pat pse36 mmx fxsr sse
bogomips        : 1389.36

The correct microcode has been loaded successfully:

IA-32 Microcode Update Driver: v1.11
<tigran@veritas.com>
microcode: CPU0 updated from revision 0 to 7,
date=05052000
microcode: CPU1 updated from revision 0 to 7,
date=05052000

This was not done from initrd (yet). However, I should
probably point out that 4 weeks ago, I *must* have
successfully loaded the microcode from rc.sysinit in
order for the box to survive for the next 3 weeks.

Now throughout all these problems, Linux has never
locked up before this message has appeared in the boot
log:

Freeing unused kernel memory: 108k freed

However, it seems that it can crash at any point
afterwards. (I never had to wait too long either,
although it seemed that the more work the box did, the
quicker it crashed.)

The conclusions from all this seem to be:
- Linux was crashing due to an incorrect
CPU/motherboard interaction.
- the current microcode for these CPUs does not fix
the problem, although it might reduce it (literally
weather permitting).

I spoke to Supermicro about this i840 board (PIIIDME)
last year, and all BIOS development has stopped. The
best they could offer me was my existing BIOS
containing the new microcode (which I already have).

Given the information at hand, my options would seem
to be:
a) reinstall the original 733 MHz CPUs, accepting that
this is the best that the board can do. These CPUs
would need the FSB putting back to 133 MHz so it's not
exactly a "slam-dunk" success. However, I have run
them  successfully in hot weather before, so I'm
reasonably confident that they would work now.
b) relocate close to one of the Earth's Poles, where
seasonal temperatures are more favourable to CPUs.

Stablising the 933 MHZ CPUs on this board properly
would appear to need extra help from Intel. Judging by
what I've seen, Linux does seem to have a window in
its boot sequence when any extra workaround could
safely be applied.

Cheers,
Chris

P.S. I'm now going to set up an initrd anyway out of
sheer bloody-mindedness ... ;-)


________________________________________________________________________
Want to chat instantly with your online friends?  Get the FREE Yahoo!
Messenger http://uk.messenger.yahoo.com/

RE: Loading Pentium III microcode under Linux - catch 22!

[Alan Cox]

On Iau, 2003-08-07 at 18:40, Nakajima, Jun wrote:
> One issue with doing it in lilo or grub is that we need to apply
> microcode update to each (physical) CPU. So we need to change the
> current driver structure if we want to move it to the boot loader.

Not really. The boot loader loads the kernel and initrd image and runs
the kernel. The initrd is a file system which can contain things like
the microcode loader and the microcode sets, just like it can contain
anything else.

Re: Loading Pentium III microcode under Linux - catch 22!

[Chris Rankin]

 --- Tigran Aivazian <tigran@veritas.com> wrote:
> Alternatively, yes, he can do it from within initrd.
> Hope that is early 
> enough for him.

Hi,

Yes, I think I'll go the initrd route. However,
hammering on the machine last night reproduced the
malfunction regardless of whether the microcode was
loaded or not!

(Addressing list in general:)

This board (Supermicro PIIIDME, i840 chipset) has been
stable for almost 3 years, since between 2.4.0-pre6
and 2.4.0-pre9 IIRC. All recent hangs and lock-ups
have been attributable to the IO subsystem in 2.4.20,
and it just ran 2.4.21 for 3 solid weeks. The only
thing that has really changed recently is the weather,
but the lm_sensors package (2.6.5) reports that
everything is fine.

Booting with "acpi=off" has so far prevented the
kernel from hanging just after the "Freeing unused
kernel memory" message. However, the kernel still hung
later: the first time was when reading lm_sensors
after one CPU had reached the magic BIOS setting of 55
degrees C under high load. This hang took out the
keyboard, mouse and serial console straight away, but
left the ethernet interface running for a few minutes.
The second hang was more spontaneous, and killed the
box outright.

Do ATX power supplies have temperature sensors built
into them? I have just upgraded my PSU from 300W to a
brand new 400W one, so doubt that it's faulty. Could
the power supply be triggering a temperature-related
ACPI event? I seem to recall the boot-log mentioning
assorted S and C events being supported. (Can't
remember which ones exactly: S0 and S3?)

And I have already run memtest for 2 hours without
problems.

Cheers,
Chris


________________________________________________________________________
Want to chat instantly with your online friends?  Get the FREE Yahoo!
Messenger http://uk.messenger.yahoo.com/

Re: Loading Pentium III microcode under Linux - catch 22!

[Eyal Lebedinsky]

Chris Rankin wrote:
> 
>  --- Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> > As far as I am aware none of the microcode updates
> > even apply to 933Mhz era PIII, just the ones the
> > BIOS ships with by default nowdays. Also the kind of
> > stuff the errata fix are obscure ultra-weird corner
> > cases people just don't hit.
> 
> Lucky me, eh?
> 
> My CPUs *do* take microcode, and they are 933 MHz...
> ;-). I upgraded from a pair of 733 MHz CPUs bought in

Some random ideas:
	- reinstall the slower, stable CPUs and burn the BIOS
		If you are a linux hacker then you kept these
		"just in case".. or else
	- get a loaner CPU and use it to burn the BIOS
		If you are a linux hacker then you have no
		life and no friends to get a loaner from but
		you do have a roomfull of computer parts...
		otherwise
	- install only one CPU, it may be more stable
	- run your CPU slower than spec, it may be more stable
		most microcode bugs are not timining sensitive
		but it is worth a try

Good luck.

--
Eyal Lebedinsky (eyal@eyal.emu.id.au) <http://samba.org/eyal/>

Re: Loading Pentium III microcode under Linux - catch 22!

[Tigran Aivazian]

On 7 Aug 2003, Alan Cox wrote:
> > In an ideal world, I would like Linux to load the
> > microcode *before* the kernel boots, which begs the
> > question of "How?". Can you suggest anything, please?
> 
> The kernel can't load the microcode until it has booted, it can
> load it very early after that from initrd.

Well, it could even do it a little bit earlier, i.e. if we allowed linking
microcode data into the kernel image (as some scsi firmware data is, etc
and as __initdata so it's thrown away later) then it could update it very
very early, i.e. even before mounting the root filesystem. All it needs is
the ability to do IPIs and get the data from somewhere.

I could implement this, but if you tell me that this is not allowed
because of the GPL issues (microcode data chunks are copyrighted by Intel)
then obviously I won't waste time writing the code to do this.

Alternatively, yes, he can do it from within initrd. Hope that is early 
enough for him.

Kind regards
Tigran

Re: Loading Pentium III microcode under Linux - catch 22!

[Alan Cox]

On Iau, 2003-08-07 at 16:57, Tigran Aivazian wrote:
> I could implement this, but if you tell me that this is not allowed
> because of the GPL issues (microcode data chunks are copyrighted by Intel)
> then obviously I won't waste time writing the code to do this.

Thats something we are moving away from, so that we can load firmware
from the initrd. Another interesting place to tackle it might be in lilo
or grub ?

Re: Loading Pentium III microcode under Linux - catch 22!

[Chris Rankin]

 --- Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> Looking at it you can do it in initrd fine, or you
> can do it as the first thing you do once the real
> root fs is mounted from init's scripts
> (/etc/rc.sysinit normally)

I'm doing the latter right already. The problem is
that even using ext3, I'm occasionally fscking the
root partition with dodgy CPUs after a lock-up.
Hopefully, initrd will allow me to load the microcode
earlier in the boot sequence.

Cheers,
Chris



________________________________________________________________________
Want to chat instantly with your online friends?  Get the FREE Yahoo!
Messenger http://uk.messenger.yahoo.com/

Re: Loading Pentium III microcode under Linux - catch 22!

[Tomas Szepe]

> [alan@lxorguk.ukuu.org.uk]
> 
> On Iau, 2003-08-07 at 15:38, Chris Rankin wrote:
> > July 2000, and my current BIOS just doesn't have any
> > microcode for them. Without the update, I used to come
> > back at the end of the day, switch on the KVM and be
> > unable to use the keyboard and mouse.
> 
> Sounds believable

Sure does.  There are machines (like the HP tc2100) which
always freeze entirely as soon as a PS/2 keyboard is attached.

> > > it can load it very early after that from initrd.
> > 
> > OK, I'll look into that.
> 
> Looking at it you can do it in initrd fine, or you can do it
> as the first thing you do once the real root fs is mounted
> from init's scripts (/etc/rc.sysinit normally)

and /etc/rc.d/rc.S on BSDish systems I believe.

-- 
Tomas Szepe <szepe@pinerecords.com>

Re: Loading Pentium III microcode under Linux - catch 22!

[Alan Cox]

On Iau, 2003-08-07 at 15:38, Chris Rankin wrote:
> July 2000, and my current BIOS just doesn't have any
> microcode for them. Without the update, I used to come
> back at the end of the day, switch on the KVM and be
> unable to use the keyboard and mouse.

Sounds believable

> Anyway, I wasn't aware that Intel had released a
> changelist for their microcode updates. Goodness knows
> what bugs they're fixing.

Generally speaking its the ones that end  "Can be worked around by
the BIOS" in the big list of errata. Some of those are however
other things like setting timing registers or turning off features
via semi-secret mtrr registers

> memory), and CPU malfunction is the leading candidate
> explanation. I have already replaced the 300W PSU with
> a 400W one and tested the memory.

Ok

> Yes, that's the "catch-22" bit. I was originally
> thinking about either a bootstrapping floppy disk, or
> maybe hacking some code into the boot-up sequence
> itself.
> 
> > it can load it very early after that from initrd.
> 
> OK, I'll look into that.

Looking at it you can do it in initrd fine, or you can do it
as the first thing you do once the real root fs is mounted
from init's scripts (/etc/rc.sysinit normally)

Re: Loading Pentium III microcode under Linux - catch 22!

[Chris Rankin]

 --- Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> As far as I am aware none of the microcode updates
> even apply to 933Mhz era PIII, just the ones the
> BIOS ships with by default nowdays. Also the kind of
> stuff the errata fix are obscure ultra-weird corner
> cases people just don't hit.

Lucky me, eh?

My CPUs *do* take microcode, and they are 933 MHz...
;-). I upgraded from a pair of 733 MHz CPUs bought in
July 2000, and my current BIOS just doesn't have any
microcode for them. Without the update, I used to come
back at the end of the day, switch on the KVM and be
unable to use the keyboard and mouse.

Anyway, I wasn't aware that Intel had released a
changelist for their microcode updates. Goodness knows
what bugs they're fixing.

> Thus I'd be very suprised if loading the microcode
> any earlier was neccessary - certainly nobody else
> has reported needing to.

My machine is currently hanging during boot-up (while
fscking the root partition, or releasing the "init"
memory), and CPU malfunction is the leading candidate
explanation. I have already replaced the 300W PSU with
a 400W one and tested the memory.

> > In an ideal world, I would like Linux to load the
> > microcode *before* the kernel boots, which begs
> > the question of "How?". Can you suggest anything,
> > please?
> 
> The kernel can't load the microcode until it has
> booted,

Yes, that's the "catch-22" bit. I was originally
thinking about either a bootstrapping floppy disk, or
maybe hacking some code into the boot-up sequence
itself.

> it can load it very early after that from initrd.

OK, I'll look into that.

Cheers,
Chris



________________________________________________________________________
Want to chat instantly with your online friends?  Get the FREE Yahoo!
Messenger http://uk.messenger.yahoo.com/

Re: Loading Pentium III microcode under Linux - catch 22!

[Alan Cox]

On Iau, 2003-08-07 at 13:28, Chris Rankin wrote:
> I have modified by boot scripts to load the microcode
> as soon as the root filesystem has been successfully
> mounted. However, this means that kernel always boots
> on buggy CPUs! For example, last night my boot failed
> just after releasing the unused kernel memory. I
> suspect that the record temperatures that my part of
> the world is currently experiencing is adversely
> influencing things. My boot-ups are usually fine

As far as I am aware none of the microcode updates even apply
to 933Mhz era PIII, just the ones the BIOS ships with by default
nowdays. Also the kind of stuff the errata fix are obscure 
ultra-weird corner cases people just don't hit.

Thus I'd be very suprised if loading the microcode any earlier
was neccessary - certainly nobody else has reported needing to.

> In an ideal world, I would like Linux to load the
> microcode *before* the kernel boots, which begs the
> question of "How?". Can you suggest anything, please?

The kernel can't load the microcode until it has booted, it can
load it very early after that from initrd.

Re: Loading Pentium III microcode under Linux - catch 22!

[Måns Rullgård]

Chris Rankin <rankincj@yahoo.com> writes:

> I have an i840 motherboard with a pair of 933 MHz PIII
> Coppermine CPUs, and I use your microcode driver to
> load the latest Intel microcode into my CPUs. This is
> very important because these CPUs are buggy without
> their microcode, and I would prefer to have the BIOS
> load it except that this would prevent me from booting
> into memtest. I have tried this before - memtest
> crashes with an "Unexpected Interrupt" error after a
> few minutes. (No i840 workarounds enabled?) Since I
> suspect that DOS would do the same thing and I would
> boot into DOS to flash firmware, I have decided that
> crashes like this would be a Bad Thing.

If the microcode in the CPUs is buggy, they are faulty and you should
demand to get them replaced at no cost.

> In an ideal world, I would like Linux to load the
> microcode *before* the kernel boots, which begs the
> question of "How?". Can you suggest anything, please?
> I remember talk of boot-time RAM disks, and wondered
> if the microcode could be placed on one of these
> somehow? Or would that be ruled out immediately by the
> microcode's non-GPL nature?

I guess it would be possible to compile the microcode into the kernel
and have some code in arch/i386/* load it as early as possible.  As
long as you don't distribute the compiled kernel you should be fine
wrt licensing.

-- 
Måns Rullgård
mru@users.sf.net

Loading Pentium III microcode under Linux - catch 22!

[Chris Rankin]

Tigran,

I have an i840 motherboard with a pair of 933 MHz PIII
Coppermine CPUs, and I use your microcode driver to
load the latest Intel microcode into my CPUs. This is
very important because these CPUs are buggy without
their microcode, and I would prefer to have the BIOS
load it except that this would prevent me from booting
into memtest. I have tried this before - memtest
crashes with an "Unexpected Interrupt" error after a
few minutes. (No i840 workarounds enabled?) Since I
suspect that DOS would do the same thing and I would
boot into DOS to flash firmware, I have decided that
crashes like this would be a Bad Thing.

I have modified by boot scripts to load the microcode
as soon as the root filesystem has been successfully
mounted. However, this means that kernel always boots
on buggy CPUs! For example, last night my boot failed
just after releasing the unused kernel memory. I
suspect that the record temperatures that my part of
the world is currently experiencing is adversely
influencing things. My boot-ups are usually fine.

In an ideal world, I would like Linux to load the
microcode *before* the kernel boots, which begs the
question of "How?". Can you suggest anything, please?
I remember talk of boot-time RAM disks, and wondered
if the microcode could be placed on one of these
somehow? Or would that be ruled out immediately by the
microcode's non-GPL nature?

Any suggestions gratefully received,
Thanks,
Chris Rankin


________________________________________________________________________
Want to chat instantly with your online friends?  Get the FREE Yahoo!
Messenger http://uk.messenger.yahoo.com/