Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

I am trying to draw AMD into the picture using their ask AMD web form
but I think it is broken.

Could asia support please take this seriously and forward it to the appropriate
AMD technical personel. I believe the issue is not restricted to linux but
to any code which executes the same way.

The ask AMD submission follows:

Subject Details:
Possible CPU ERRATA: re: bus disconnect and apic timer interrupt

Greetings:
I and many others have been tracking down a hard lockup problem on linux and 
nforce2 chipset.

Please find continuing discussion including a copy of this submission here:

http://linux.derkeiler.com/Mailing-Lists/Kernel/2003-12/1528.html

My current level of knowledge (best estimate) on the problem is that if a cpu 
disconnect cycle is in progress or has occurred and the local apic timer interrupt
is the trigger to return to a connected state then an undocumented timing
constraint exists. The constraint is that the local apic acknowledge will not be
correctly received by the local apic if it occurs earlier than about 500us 
after the processor continues execution. That is if the processor issues
the ack earlier than 500us after resuming execution then an unrecoverable
hard lockup of the system occurs.

Possible causes include a slow start to the local system bus in relation to
the reconnection of the cpu to the local apic as per earlier model athlon CPU's?
Or system bus connect disconnect signal timing problems with the nforce2 northbridge?

What I would like to know is:

a) Can you please isolate- verify cause assuming you have hardware testing facilities.

b) Does this problem affect all local apic interrupt sources including those which
 have come from an io-apic.

c) Is there is a chipset independent way of finding out if we are coming out of
 a disconnect state prior to issuing the local apic acknowledge. 
 i.e. is there a readable status bit within the processor that we can use to see
 if it is safe to immediately ACK the local apic or if we should wait for 500ns or so.

I have experienced this problem on XP2500 barton and XP2200 thoroughbred cores.
Others have experienced it on other model barton cores. At least 4 makes of
motherboard are involved.

So far it appears to affect all current and pending linux releases for the nforce2 chipsets. 
One could say this relates to a good quantity of potential AMD athlon cpu sales
and bugs with nforce2 and AMD may sour uptake of nforce3 and x64.........

Regards
Ross Dickson.
Director.
Dat's Creative Pty Ltd
Gold Coast
Australia


I don't know if it got through, I received this after the submit button

The page cannot be displayed
There is a problem with the page you are trying to reach and it cannot be displayed.
Please try the following:
Click the Refresh button, or try again later; it does not normally take a long time for an application to restart.
Open the 139.95.253.214 home page, and then look for links to the information you want.
HTTP Error 500-12 Application Restarting
 Internet Information Services
Technical Information (for support personnel)

Background:
 The request cannot be processed while the Web site is restarting. 

More information:
 Microsoft Support 

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Maciej W. Rozycki]

On Sat, 7 Feb 2004, Len Brown wrote:

> > > Int: type 3, pol 0, trig 0, bus 0, irq 0, 2-0
> > > should be:
> > > Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-2
> > > 
> > > type should be mp_INT, not mp_ExtINT, and dstirq should be 2, not 0;
> > > yes?
> > 
> >  Nope -- it should be:
> > 
> > Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-0
> > 
> > as the ACPI spec explicitly defines a 1:1 map -- see section 5.2.10.7 (I
> > have rev.2.0c): "This means that I/O APIC interrupt inputs 0-15 must be
> > mapped to global system interrupts 0-15 and have identical sources as the
> > 8259 IRQs 0-15 unless overrides are used."
> 
> ah, here is the one i was looking for -- the one from the over-ride, and
> it looks correct.
> 
> > > ACPI: INT_SRC_OVR (bus[0] irq[0x0] global_irq[0x2] polarity[0x0]
> > trigger[0x0])
> > > Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-2
> 
> I'm not sure what is going on with pin0 above -- (and below), i think
> there is a workaround in this system that isn't in the base kernel? 

 The system reports the timer IRQ to be routed to pin 2, even though it is
connected to pin 0.  There's no workaround in the kernel as working it
around cannot be handled in a generic way.

> Indeed, I've lost track of the original failure on this thread, was it
> simply that the timer came out as XT-PIC mode instead of IOAPIC-edge?

 The ACPI table had no entry for the timer IRQ, so Linux reverted to 
the ExtINT (XT-PIC) mode for the IRQ as a last resort.  It's only thanks 
to ancient EISA systems the fallback is there at all and ACPI-based 
systems happened to work.

> > > > > IRQ to pin mappings:
> > > > > IRQ0 -> 0:2-> 0:0
> > > > [...]
> > > > > IRQ9 -> 0:9-> 0:9
> > > > 
> > > >  These two entries are wrong -- the interrupts are set up as if they
> > > > were
> > > > connected to multiple I/O APIC inputs.  The first entry is a result of
> > > > your hack, but the second one suggests a bug somewhere.
> > > 
> > > Right, this means that there are multiple irq_2_pin entries, which isn't
> > > what we want.  Indeed, I've not seen a system were we _do_ want them --
> > > do system actually exist where the same interrupt wires simultaneously
> > > go to multiple interrupt input pins?
> > 
> >  I'm told such systems exist(ed), at least in the MP-table era so code
> > has been added to handle them.  Technically, there's nothing wrong with
> > that, except that the interrupt has to be level-triggered at the APIC
> > level, of course.
> > 
> Hmm, i wonder if any bad things go wrong for IRQ9 -> 0:9-> 0:9 -- I
> guess we do all the APIC tickling twice because we thing there are two
> pins associated with the IRQ.

 That entry appears due to conditions making the following entry being 
reported in the log:

IOAPIC[0]: Set PCI routing entry (2-9 -> 0x71 -> IRQ 9 Mode:1 Active:0)

I don't know why the SCI interrupt is set up the second time here.  That's 
the reason the interrupt is routed twice.

 As a side note, I wonder why entries for PCI interrupts are processed so
late...

> I suspect that we have some failing systems due to this bug for when the
> interrupt is _not_ identity mapped like this example -- b/c we'd ack the
> wrong pin as a side-effect of an interrupt...

 As you can see there's only one pin recorded twice.

> BTW. I've always disliked the IRQ -> pin mapping, it should really be
> IRQ <- pin mapping -- or maybe more accurately the vector should be in
> there...

 Well, using vectors wouldn't be that bad -- I've already discovered this
once. :-)  It would require a clean-up of ISA drivers, but actually that
is desireable regardless, as there are non-i386 systems supporting the ISA
bus out there that want to use numbers outside the 0-15 range for ISA
interrupts.

> > > This is a check for a bad BIOS that sets the timer to level triggered. 
> > > I don't see this as the case for the nforce2 system on hand.  Indeed,
> > > I've _never_ see this case, and I wonder if this clause can be deleted
> > > altogether...  Did I miss something?
> > 
> >  Note that in principle the timer can be set up for a level-triggered 
> > operation -- this used to be the case for MCA systems that had additional 
> > circuitry to ack the interrupt at the source.  I'm not sure if ACPI 
> > supports such a configuration -- the MPS certainly did.
> 
> >  Anyway, this workaround, if left in place, should be triggered for the
> > timer source (bus_irq == 0) regardless of the I/O APIC pin it's routed to
> > (global_irq).
> 
> this is an ACPI specific routine, (and IMHO it is thus mis-named --
> along with all the acpi-specific stuff in mpparse.c;-)  I'd be inclined
> to squak about a bogus entry rather than using a silent workaround for a
> system that may not actually exist.

 Someone has introduced it for a reason, I suppose -- can't the originator 
be tracked down?

> >  It's still correct -- we are still replacing an [IOAPIC.PIN -> IRQ] entry
> > (perhaps the arrow should be reversed), except we disregard old IOAPIC.PIN
> > information without examining it.
> 
> yes, not looking at the IOAPIC.PIN -- that is what i meant by obsolete
> comment. (and the fact that the arrow points from pin to IRQ is the only
> part of the comment i like;-)

 If interpreted as a vector as opposed to a physical IRQ line, then I 
agree -- I'd leave that as is, then.

> >  While being able to track an ExtINT source could be useful for the NMI
> > watchdog under certain circumstances, the ACPI spec does not recognize
> > this kind of interrupt and does not provide a means of expressing it in 
> > the tables.  So we cannot set any entry to ExtINT and the timer IRQ is an 
> > ordinary interrupt.  We assume LINT0 interrupts are ExtINT ones 
> > implicitly.
> 
> ACPI has an additional MADT entry type for specifying the NMI.

 That's for LINT1 that's (almost?) universally configured for NMI
interrupts, indeed.  OTOH, LINT0 of the BSP is typically configured as
ExtINT for the virtual wire mode, although I've met a system using a
through-I/O-APIC variation of the mode as well as one using the PIC
compatibility mode (we used to support both variations; I hope it hasn't
got broken) -- both of them had LINT0 inputs simply masked out.  The
MP-table provides a way to report any sensible interrupt configurations
for LINT inputs (including ExtINT, Fixed, SMI, etc.), not only the NMI.

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Len Brown]

On Sat, 2004-02-07 at 07:41, Maciej W. Rozycki wrote:
> On Sat, 7 Feb 2004, Len Brown wrote:
> 
> > Maciej -- Sorry for the delayed response.  December was pretty busy and
> > there was  a window where I didn't catch up on e-mail unless [PATCH] was
> > in the subject.
> 
>  Sorry about that -- I could have added it to the subject, indeed.
> 
> > > > ACPI: IOAPIC (id[0x02] address[0xfec00000] global_irq_base[0x0])
> > > > IOAPIC[0]: Assigned apic_id 2
> > > > IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, IRQ 0-23
> > > > Bus #0 is ISA
> > > > Int: type 3, pol 0, trig 0, bus 0, irq 0, 2-0
> > > 
> > >  I've browsed the relevant part of the ACPI spec and the above entry
> > > is 
> > > incorrect.  It looks like INTIN0 is now the preferred line for the
> > > 8254 
> > > timer; at least it is the default one when using ACPI tables.  This is
> > > a 
> > > bug in Linux.
> > 
> > Int: type 3, pol 0, trig 0, bus 0, irq 0, 2-0
> > should be:
> > Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-2
> > 
> > type should be mp_INT, not mp_ExtINT, and dstirq should be 2, not 0;
> > yes?
> 
>  Nope -- it should be:
> 
> Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-0
> 
> as the ACPI spec explicitly defines a 1:1 map -- see section 5.2.10.7 (I
> have rev.2.0c): "This means that I/O APIC interrupt inputs 0-15 must be
> mapped to global system interrupts 0-15 and have identical sources as the
> 8259 IRQs 0-15 unless overrides are used."

ah, here is the one i was looking for -- the one from the over-ride, and
it looks correct.

> > ACPI: INT_SRC_OVR (bus[0] irq[0x0] global_irq[0x2] polarity[0x0]
> trigger[0x0])
> > Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-2

I'm not sure what is going on with pin0 above -- (and below), i think
there is a workaround in this system that isn't in the base kernel? 
Indeed, I've lost track of the original failure on this thread, was it
simply that the timer came out as XT-PIC mode instead of IOAPIC-edge?


> > > > IRQ to pin mappings:
> > > > IRQ0 -> 0:2-> 0:0
> > > [...]
> > > > IRQ9 -> 0:9-> 0:9
> > > 
> > >  These two entries are wrong -- the interrupts are set up as if they
> > > were
> > > connected to multiple I/O APIC inputs.  The first entry is a result of
> > > your hack, but the second one suggests a bug somewhere.
> > 
> > Right, this means that there are multiple irq_2_pin entries, which isn't
> > what we want.  Indeed, I've not seen a system were we _do_ want them --
> > do system actually exist where the same interrupt wires simultaneously
> > go to multiple interrupt input pins?
> 
>  I'm told such systems exist(ed), at least in the MP-table era so code
> has been added to handle them.  Technically, there's nothing wrong with
> that, except that the interrupt has to be level-triggered at the APIC
> level, of course.
> 
Hmm, i wonder if any bad things go wrong for IRQ9 -> 0:9-> 0:9 -- I
guess we do all the APIC tickling twice because we thing there are two
pins associated with the IRQ.

I suspect that we have some failing systems due to this bug for when the
interrupt is _not_ identity mapped like this example -- b/c we'd ack the
wrong pin as a side-effect of an interrupt...

BTW. I've always disliked the IRQ -> pin mapping, it should really be
IRQ <- pin mapping -- or maybe more accurately the vector should be in
there...

> > > @@ -940,7 +940,7 @@ void __init mp_override_legacy_irq (
> > >          *      erroneously sets the trigger to level, resulting in a
> > > HUGE 
> > >          *      increase of timer interrupts!
> > >          */
> > > -       if ((bus_irq == 0) && (global_irq == 2) && (trigger == 3))
> > > +       if ((bus_irq == 0) && (trigger == 3))
> > >                 trigger = 1;
> > 
> > This is a check for a bad BIOS that sets the timer to level triggered. 
> > I don't see this as the case for the nforce2 system on hand.  Indeed,
> > I've _never_ see this case, and I wonder if this clause can be deleted
> > altogether...  Did I miss something?
> 
>  Note that in principle the timer can be set up for a level-triggered 
> operation -- this used to be the case for MCA systems that had additional 
> circuitry to ack the interrupt at the source.  I'm not sure if ACPI 
> supports such a configuration -- the MPS certainly did.

>  Anyway, this workaround, if left in place, should be triggered for the
> timer source (bus_irq == 0) regardless of the I/O APIC pin it's routed to
> (global_irq).

this is an ACPI specific routine, (and IMHO it is thus mis-named --
along with all the acpi-specific stuff in mpparse.c;-)  I'd be inclined
to squak about a bogus entry rather than using a silent workaround for a
system that may not actually exist.

> > >         intsrc.mpc_type = MP_INTSRC;
> > > @@ -961,7 +961,7 @@ void __init mp_override_legacy_irq (
> > >          * Otherwise create a new entry (e.g. global_irq == 2).
> > >          */
> > >         for (i = 0; i < mp_irq_entries; i++) {
> > > -               if ((mp_irqs[i].mpc_dstapic == intsrc.mpc_dstapic) 
> > > +               if ((mp_irqs[i].mpc_srcbus == intsrc.mpc_srcbus) 
> > >                         && (mp_irqs[i].mpc_srcbusirq ==
> > > intsrc.mpc_srcbusirq)) {
> > >                         mp_irqs[i] = intsrc;
> > >                         found = 1;
> > 
> > This makes sense.  If we're over-riding the destination, we don't care
> > what the destination was before, ya?
> 
>  Worse even -- we may replace a wrong entry.

oy!

> > this used to check dstapic and dstirq.
> > it was partially fixed last summer when checking srcbus replaced
> > checking dstirq.  Unclear why dstapic wasn't changed to srcbus at the
> > same time.
> > 
> > i think with this change the comment above the code is no longer
> > correct, yes?
> 
>  It's still correct -- we are still replacing an [IOAPIC.PIN -> IRQ] entry
> (perhaps the arrow should be reversed), except we disregard old IOAPIC.PIN
> information without examining it.

yes, not looking at the IOAPIC.PIN -- that is what i meant by obsolete
comment. (and the fact that the arrow points from pin to IRQ is the only
part of the comment i like;-)

> > > -               intsrc.mpc_irqtype = i ? mp_INT : mp_ExtINT;   /*
> > > 8259A to #0 */
> > > +               intsrc.mpc_irqtype = mp_INT;
> > 
> > I believe this change is correct.  When this code runs, the system is in
> > APIC mode and there is no concept of vectored external PIC interrupts.
> 
>  While being able to track an ExtINT source could be useful for the NMI
> watchdog under certain circumstances, the ACPI spec does not recognize
> this kind of interrupt and does not provide a means of expressing it in 
> the tables.  So we cannot set any entry to ExtINT and the timer IRQ is an 
> ordinary interrupt.  We assume LINT0 interrupts are ExtINT ones 
> implicitly.

ACPI has an additional MADT entry type for specifying the NMI.

thanks,
-Len

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Maciej W. Rozycki]

On Sat, 7 Feb 2004, Len Brown wrote:

> Maciej -- Sorry for the delayed response.  December was pretty busy and
> there was  a window where I didn't catch up on e-mail unless [PATCH] was
> in the subject.

 Sorry about that -- I could have added it to the subject, indeed.

> > > ACPI: IOAPIC (id[0x02] address[0xfec00000] global_irq_base[0x0])
> > > IOAPIC[0]: Assigned apic_id 2
> > > IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, IRQ 0-23
> > > Bus #0 is ISA
> > > Int: type 3, pol 0, trig 0, bus 0, irq 0, 2-0
> > 
> >  I've browsed the relevant part of the ACPI spec and the above entry
> > is 
> > incorrect.  It looks like INTIN0 is now the preferred line for the
> > 8254 
> > timer; at least it is the default one when using ACPI tables.  This is
> > a 
> > bug in Linux.
> 
> Int: type 3, pol 0, trig 0, bus 0, irq 0, 2-0
> should be:
> Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-2
> 
> type should be mp_INT, not mp_ExtINT, and dstirq should be 2, not 0;
> yes?

 Nope -- it should be:

Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-0

as the ACPI spec explicitly defines a 1:1 map -- see section 5.2.10.7 (I
have rev.2.0c): "This means that I/O APIC interrupt inputs 0-15 must be
mapped to global system interrupts 0-15 and have identical sources as the
8259 IRQs 0-15 unless overrides are used."

> > > ACPI: INT_SRC_OVR (bus[0] irq[0x9] global_irq[0x9] polarity[0x1]
> > trigger[0x3])
> > > Int: type 0, pol 1, trig 3, bus 0, irq 9, 2-9
> > 
> >  And yet another explicit entry which has an effect on configuration
> > as
> > reported below.
> 
> This is the ACPI SCI.  Mapping IRQ9 ot INTIN9 is a no-op.  The effect of
> this entry is to explicitly setting the polarity/triger to high/level. 
> If this were not present it would still be in IRQ9, but would be set to
> the default, which is low/level.

 Yep, just explaining the bits to Ross...

> > > IRQ to pin mappings:
> > > IRQ0 -> 0:2-> 0:0
> > [...]
> > > IRQ9 -> 0:9-> 0:9
> > 
> >  These two entries are wrong -- the interrupts are set up as if they
> > were
> > connected to multiple I/O APIC inputs.  The first entry is a result of
> > your hack, but the second one suggests a bug somewhere.
> 
> Right, this means that there are multiple irq_2_pin entries, which isn't
> what we want.  Indeed, I've not seen a system were we _do_ want them --
> do system actually exist where the same interrupt wires simultaneously
> go to multiple interrupt input pins?

 I'm told such systems exist(ed), at least in the MP-table era so code
has been added to handle them.  Technically, there's nothing wrong with
that, except that the interrupt has to be level-triggered at the APIC
level, of course.

> > @@ -940,7 +940,7 @@ void __init mp_override_legacy_irq (
> >          *      erroneously sets the trigger to level, resulting in a
> > HUGE 
> >          *      increase of timer interrupts!
> >          */
> > -       if ((bus_irq == 0) && (global_irq == 2) && (trigger == 3))
> > +       if ((bus_irq == 0) && (trigger == 3))
> >                 trigger = 1;
> 
> This is a check for a bad BIOS that sets the timer to level triggered. 
> I don't see this as the case for the nforce2 system on hand.  Indeed,
> I've _never_ see this case, and I wonder if this clause can be deleted
> altogether...  Did I miss something?

 Note that in principle the timer can be set up for a level-triggered 
operation -- this used to be the case for MCA systems that had additional 
circuitry to ack the interrupt at the source.  I'm not sure if ACPI 
supports such a configuration -- the MPS certainly did.

 Anyway, this workaround, if left in place, should be triggered for the
timer source (bus_irq == 0) regardless of the I/O APIC pin it's routed to
(global_irq).

> >         intsrc.mpc_type = MP_INTSRC;
> > @@ -961,7 +961,7 @@ void __init mp_override_legacy_irq (
> >          * Otherwise create a new entry (e.g. global_irq == 2).
> >          */
> >         for (i = 0; i < mp_irq_entries; i++) {
> > -               if ((mp_irqs[i].mpc_dstapic == intsrc.mpc_dstapic) 
> > +               if ((mp_irqs[i].mpc_srcbus == intsrc.mpc_srcbus) 
> >                         && (mp_irqs[i].mpc_srcbusirq ==
> > intsrc.mpc_srcbusirq)) {
> >                         mp_irqs[i] = intsrc;
> >                         found = 1;
> 
> This makes sense.  If we're over-riding the destination, we don't care
> what the destination was before, ya?

 Worse even -- we may replace a wrong entry.

> this used to check dstapic and dstirq.
> it was partially fixed last summer when checking srcbus replaced
> checking dstirq.  Unclear why dstapic wasn't changed to srcbus at the
> same time.
> 
> i think with this change the comment above the code is no longer
> correct, yes?

 It's still correct -- we are still replacing an [IOAPIC.PIN -> IRQ] entry
(perhaps the arrow should be reversed), except we disregard old IOAPIC.PIN
information without examining it.

> > -               intsrc.mpc_irqtype = i ? mp_INT : mp_ExtINT;   /*
> > 8259A to #0 */
> > +               intsrc.mpc_irqtype = mp_INT;
> 
> I believe this change is correct.  When this code runs, the system is in
> APIC mode and there is no concept of vectored external PIC interrupts.

 While being able to track an ExtINT source could be useful for the NMI
watchdog under certain circumstances, the ACPI spec does not recognize
this kind of interrupt and does not provide a means of expressing it in 
the tables.  So we cannot set any entry to ExtINT and the timer IRQ is an 
ordinary interrupt.  We assume LINT0 interrupts are ExtINT ones 
implicitly.

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Len Brown]

Maciej -- Sorry for the delayed response.  December was pretty busy and
there was  a window where I didn't catch up on e-mail unless [PATCH] was
in the subject.

On Thu, 2003-12-11 at 10:15, Maciej W. Rozycki wrote:
> On Thu, 11 Dec 2003, Ross Dickson wrote:
> 
> > ACPI: IOAPIC (id[0x02] address[0xfec00000] global_irq_base[0x0])
> > IOAPIC[0]: Assigned apic_id 2
> > IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, IRQ 0-23
> > Bus #0 is ISA
> > Int: type 3, pol 0, trig 0, bus 0, irq 0, 2-0
> 
>  I've browsed the relevant part of the ACPI spec and the above entry
> is 
> incorrect.  It looks like INTIN0 is now the preferred line for the
> 8254 
> timer; at least it is the default one when using ACPI tables.  This is
> a 
> bug in Linux.

Int: type 3, pol 0, trig 0, bus 0, irq 0, 2-0
should be:
Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-2

type should be mp_INT, not mp_ExtINT, and dstirq should be 2, not 0;
yes?

> > ACPI: INT_SRC_OVR (bus[0] irq[0x0] global_irq[0x2] polarity[0x0]
> trigger[0x0])
> > Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-2
> 
>  Now this is an explicit entry stating the 8254 timer is connected to
> INTIN2.  If this is not the case, the BIOS is buggy and the solution
> is to
> fix it.  I don't consider it possible to be worked around in Linux
> except
> maybe with a command line option added manually.

Agreed.

> > ACPI: INT_SRC_OVR (bus[0] irq[0x9] global_irq[0x9] polarity[0x1]
> trigger[0x3])
> > Int: type 0, pol 1, trig 3, bus 0, irq 9, 2-9
> 
>  And yet another explicit entry which has an effect on configuration
> as
> reported below.

This is the ACPI SCI.  Mapping IRQ9 ot INTIN9 is a no-op.  The effect of
this entry is to explicitly setting the polarity/triger to high/level. 
If this were not present it would still be in IRQ9, but would be set to
the default, which is low/level.

> > IRQ to pin mappings:
> > IRQ0 -> 0:2-> 0:0
> [...]
> > IRQ9 -> 0:9-> 0:9
> 
>  These two entries are wrong -- the interrupts are set up as if they
> were
> connected to multiple I/O APIC inputs.  The first entry is a result of
> your hack, but the second one suggests a bug somewhere.

Right, this means that there are multiple irq_2_pin entries, which isn't
what we want.  Indeed, I've not seen a system were we _do_ want them --
do system actually exist where the same interrupt wires simultaneously
go to multiple interrupt input pins?

> patch-mips-2.6.0-test11-20031209-acpi-irq0-1
> diff -up --recursive --new-file
> linux-mips-2.6.0-test11-20031209.macro/arch/i386/kernel/mpparse.c
> linux-mips-2.6.0-test11-20031209/arch/i386/kernel/mpparse.c
> 
> ---
> linux-mips-2.6.0-test11-20031209.macro/arch/i386/kernel/mpparse.c  
> 2003-11-25 04:57:01.000000000 +0000
> +++ linux-mips-2.6.0-test11-20031209/arch/i386/kernel/mpparse.c
> 2003-12-11 09:43:26.000000000 +0000
> @@ -940,7 +940,7 @@ void __init mp_override_legacy_irq (
>          *      erroneously sets the trigger to level, resulting in a
> HUGE 
>          *      increase of timer interrupts!
>          */
> -       if ((bus_irq == 0) && (global_irq == 2) && (trigger == 3))
> +       if ((bus_irq == 0) && (trigger == 3))
>                 trigger = 1;
>  

This is a check for a bad BIOS that sets the timer to level triggered. 
I don't see this as the case for the nforce2 system on hand.  Indeed,
I've _never_ see this case, and I wonder if this clause can be deleted
altogether...  Did I miss something?

>         intsrc.mpc_type = MP_INTSRC;
> @@ -961,7 +961,7 @@ void __init mp_override_legacy_irq (
>          * Otherwise create a new entry (e.g. global_irq == 2).
>          */
>         for (i = 0; i < mp_irq_entries; i++) {
> -               if ((mp_irqs[i].mpc_dstapic == intsrc.mpc_dstapic) 
> +               if ((mp_irqs[i].mpc_srcbus == intsrc.mpc_srcbus) 
>                         && (mp_irqs[i].mpc_srcbusirq ==
> intsrc.mpc_srcbusirq)) {
>                         mp_irqs[i] = intsrc;
>                         found = 1;

This makes sense.  If we're over-riding the destination, we don't care
what the destination was before, ya?

this used to check dstapic and dstirq.
it was partially fixed last summer when checking srcbus replaced
checking dstirq.  Unclear why dstapic wasn't changed to srcbus at the
same time.

i think with this change the comment above the code is no longer
correct, yes?


> @@ -1008,9 +1008,10 @@ void __init mp_config_acpi_legacy_irqs (
>          */
>         for (i = 0; i < 16; i++) {
>  
> -               if (i == 2) continue;                   /* Don't
> connect IRQ2 */
> +               if (i == 2)
> +                       continue;                       /* Don't
> connect IRQ2 */

white-space -- ok.

>  
> -               intsrc.mpc_irqtype = i ? mp_INT : mp_ExtINT;   /*
> 8259A to #0 */
> +               intsrc.mpc_irqtype = mp_INT;

I believe this change is correct.  When this code runs, the system is in
APIC mode and there is no concept of vectored external PIC interrupts.

>                 intsrc.mpc_srcbusirq = i;                  /* Identity
> mapped */
>                 intsrc.mpc_dstirq = i;
>  

thanks,
-Len

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Josh McKinney]

On approximately Mon, Dec 15, 2003 at 11:54:32PM +1000, Ross Dickson wrote:
> <snip>
> 
> Out of curiosity of the 10 lines with predelay count like follows
> 
> ..APIC TIMER ack delay, predelay count: 20769 
> 
> Do any of them exceed your safe count of 20779? and any really close
> in value to the reload count of 20791?
> 

I have an Asus A7N8X deluxe rev. 2, so no BIOS updates. I am running
2.6.0-test11-wli-2 with your(Ross Dickson) two patches that I diffed
for 2.6.  Default delay times, 19hrs uptime, grep tested.

..APIC TIMER ack delay, reload:25053, safe:25038
..APIC TIMER ack delay, predelay count:24988 
..APIC TIMER ack delay, predelay count:24954 
..APIC TIMER ack delay, predelay count:25011 
..APIC TIMER ack delay, predelay count:24970 
..APIC TIMER ack delay, predelay count:25032 
..APIC TIMER ack delay, predelay count:24991 
..APIC TIMER ack delay, predelay count:24949 
..APIC TIMER ack delay, predelay count:25013 
..APIC TIMER ack delay, predelay count:24971 
..APIC TIMER ack delay, predelay count:24928 

> On our pheonix bios's we regularly see 2 or 3 of them exceed the safe count
> (indication of potential lockup without the patch) often with one of them 
> within 4 counts of the reload value (really quick).
> 
> Can you also advise if your bios setting of the  "C1 disconnect" is set to on, off,
> or auto? - trying to gain a clue as to how award can have disconnect running
> and avoid lockups.
> 

athcool version 0.3.1

nVIDIA nForce2 (10de 01e0) found
'Halt Disconnect and Stop Grant Disconnect' bit is enabled.

> Also are you running with DDR333 or DDR400 ram and how many sticks?
> 

1 stick, DDR400 Corsair, 5-2-2-2

> I have heard lockups are not supposed to happen at all if the fsb (host bus
> clock speed) matches the ddr speed. One of my systems went about 4 hours
> (xp2500 333fsb, DDR333) without the apic delay patch on a pheonix bios
> before lockup.
> 
I run sync memory timings and have seen pretty good uptimes without
the patches.

> So far it appears to be safe with a barton core cpu to read the local apic
> timer count register as the v2 apic delay patch does. 
> 

I am running the barton core here, so far so good it seems.  Time will
only tell I guess.  We are getting in the right direction I hope.

> So far I cannot use my v2 apic delay patch for long periods with my throughbred
> core XP2200 without hard lockups (pheonix bios, fsb266, DDR400 ram).
> 
> Regards
> Ross.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Josh McKinney		     |	Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
                             | They that can give up essential liberty
Linux, the choice       -o)  | to obtain a little temporary safety deserve 
of the GNU generation    /\  | neither liberty or safety. 
                        _\_v |                          -Benjamin Franklin

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

<snip>
> I stayed with 600UL 100ndelay just to see if anything 
> breaks with amd XP3000+ and patches with a bios 
> that doesn't crash with nforce2 but needs help from 
> patches on other points(to get edge timer on and 
> to use nmi_watchdog=1 rather than =2). Also hope 
> we get a clue about what Award bios update does 
> that Phoenix does not do so far. 
 
>/usr/src/kernel-source-2.6.0/include/asm-i386/apic.h 
> #define APIC_DEBUG 1 

>...but I don't see any 

>calibrating APIC timer ... 
> ..... CPU clock speed is 2079.0146 MHz. 
> ..... host bus clock speed is 332.0663 MHz. 
> NET: Registered protocol family 16 
> ..APIC TIMER ack delay, reload:20791, safe:20779 
> ..APIC TIMER ack delay, predelay count: 20769 

>etc 

Hi bob, if the award bios has completely stabilised your system then that is
great news and it should make the apic delay patch unnecessary for your system.

The second patch, the io-apic patch is a workaround to enable the 8254 
connection to the io-apic INTIN0 because that is where it appears to be
wired to on nforce2 mobos.

According to Maciej Rozycki it looks like bios lies and says it is wired
to INTIN2 so nothing happens when that is tested first.

Out of curiosity of the 10 lines with predelay count like follows

..APIC TIMER ack delay, predelay count: 20769 

Do any of them exceed your safe count of 20779? and any really close
in value to the reload count of 20791?

On our pheonix bios's we regularly see 2 or 3 of them exceed the safe count
(indication of potential lockup without the patch) often with one of them 
within 4 counts of the reload value (really quick).

Can you also advise if your bios setting of the  "C1 disconnect" is set to on, off,
or auto? - trying to gain a clue as to how award can have disconnect running
and avoid lockups.

Also are you running with DDR333 or DDR400 ram and how many sticks?

I have heard lockups are not supposed to happen at all if the fsb (host bus
clock speed) matches the ddr speed. One of my systems went about 4 hours
(xp2500 333fsb, DDR333) without the apic delay patch on a pheonix bios
before lockup.

So far it appears to be safe with a barton core cpu to read the local apic
timer count register as the v2 apic delay patch does. 

So far I cannot use my v2 apic delay patch for long periods with my throughbred
core XP2200 without hard lockups (pheonix bios, fsb266, DDR400 ram).

Regards
Ross.

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Maciej W. Rozycki]

On Mon, 15 Dec 2003 ross.alexander@uk.neceur.com wrote:

> 2) If the local apic is running is it necessary to use the 8254 as a 
> timer?

 The 8254 timer is used for timekeeping -- it would be unnecessarily ugly 
to stuff this fuctionality to the APIC timer on one of CPUs.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Bob]

Ross Dickson wrote:

>It also reports its existence on boot if used. e.g.
>
>..APIC TIMER ack delay, reload:16701, safe:16691
>
>And if  
>
>#define APIC_DEBUG 0
>
>is set to 1 in 
>
>/usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h
>  
>
I'm trying the two new patches with 2.6 and my Award
bios which fixes the crash problem itself but didn't turn
on edge timer without a patch, and didn't turn on
nmi_watchdog=1 with the first patch, only =2.

nmi_watchdog gets a lot more ticks with =1, too.

bob@where  cat /proc/interrupts
           CPU0      
  0:     924534    IO-APIC-edge  timer
  1:       1070    IO-APIC-edge  i8042
  2:          0          XT-PIC  cascade
  8:          1    IO-APIC-edge  rtc
  9:          0   IO-APIC-level  acpi
 12:      14077    IO-APIC-edge  i8042
 14:         10    IO-APIC-edge  ide0
 15:         10    IO-APIC-edge  ide1
 16:      44086   IO-APIC-level  ide2, ide3, eth0
 17:          0   IO-APIC-level  yenta, yenta
 19:      35570   IO-APIC-level  ide4, ide5
 21:          0   IO-APIC-level  NVidia nForce2
NMI:     924555
LOC:     924432
ERR:          0
MIS:         22

I stayed with 600UL 100ndelay just to see if anything
breaks with amd XP3000+ and patches with a bios
that doesn't crash with nforce2 but needs help from
patches on other points(to get edge timer on and
to use nmi_watchdog=1 rather than =2). Also hope
we get a clue about what Award bios update does
that Phoenix does not do so far.

/usr/src/kernel-source-2.6.0/include/asm-i386/apic.h
#define APIC_DEBUG 1

...but I don't see any

calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769

etc

in the area of "NET: Registered" I see the following--

Dec 15 05:34:30 where kernel: PCI: Setting latency timer of device 
0000:00:06.0 to 64
Dec 15 05:34:30 where kernel: intel8x0: clocking to 47414
Dec 15 05:34:30 where kernel: ALSA device list:
Dec 15 05:34:30 where kernel:   #0: Dummy 1
Dec 15 05:34:30 where kernel:   #1: Virtual MIDI Card 1
Dec 15 05:34:30 where kernel:   #2: NVidia nForce2 at 0xe5080000, irq 21
Dec 15 05:34:30 where kernel: NET: Registered protocol family 2
Dec 15 05:34:30 where kernel: IP: routing cache hash table of 8192 
buckets, 64Kbytes
Dec 15 05:34:30 where kernel: TCP: Hash tables configured (established 
262144 bind 65536)
Dec 15 05:34:30 where kernel: ip_conntrack version 2.1 (8191 buckets, 
65528 max) - 296 bytes per conntrack
Dec 15 05:34:30 where kernel: ip_tables: (C) 2000-2002 Netfilter core team
Dec 15 05:34:30 where kernel: ipt_recent v0.3.1: Stephen Frost 
<sfrost@snowman.net>.  http://snowman.net/projects/ipt_rece
nt/
Dec 15 05:34:30 where kernel: NET: Registered protocol family 1
Dec 15 05:34:30 where kernel: NET: Registered protocol family 17
Dec 15 05:34:30 where kernel: BIOS EDD facility v0.10 2003-Oct-11, 4 
devices found
Dec 15 05:34:30 where kernel: Please report your BIOS at 
http://domsch.com/linux/edd30/results.html
Dec 15 05:34:30 where kernel: md: Autodetecting RAID arrays.

Dec 15 05:34:30 where kernel: NFORCE2: chipset revision 162

log starts with this acpi stuff--

Dec 15 05:34:30 where kernel: klogd 1.4.1#10, log source = /proc/kmsg 
started.
Dec 15 05:34:30 where kernel: Inspecting /boot/System.map-2.6.0.nf2
Dec 15 05:34:30 where kernel: Loaded 33440 symbols from 
/boot/System.map-2.6.0.nf2.
Dec 15 05:34:30 where kernel: Symbols match kernel version 2.6.0.
Dec 15 05:34:30 where kernel: No module symbols loaded - kernel modules 
not enabled.
Dec 15 05:34:30 where kernel: IRQs 3 4 *5 6 7 10 11 12 14 15)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [LFIR] (IRQs 3 4 
5 6 7 10 11 12 14 15)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [L3CM] (IRQs 3 4 
5 6 7 10 11 12 14 15)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [LIDE] (IRQs 3 4 
5 6 7 10 11 12 14 15)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC1] (IRQs *16)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC2] (IRQs *17)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC3] (IRQs *18)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC4] (IRQs *19)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC5] (IRQs 16)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCF] (IRQs 20 
21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCG] (IRQs 20 
21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCH] (IRQs 20 
21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCI] (IRQs 20 
21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCJ] (IRQs 20 
21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCK] (IRQs 20 
21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCS] (IRQs *23)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCL] (IRQs 20 
21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCM] (IRQs 20 
21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [AP3C] (IRQs 20 
21 22)
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCZ] (IRQs 20 
21 22)
Dec 15 05:34:30 where kernel: Linux Plug and Play Support v0.97 (c) Adam 
Belay
Dec 15 05:34:30 where kernel: pnp: the driver 'system' has been registered
Dec 15 05:34:30 where kernel: PnPBIOS: Scanning system for PnP BIOS 
support...
Dec 15 05:34:30 where kernel: PnPBIOS: Found PnP BIOS installation 
structure at 0xc00fc660
Dec 15 05:34:30 where kernel: PnPBIOS: PnP BIOS version 1.0, entry 
0xf0000:0xc690, dseg 0xf0000
Dec 15 05:34:30 where kernel: pnp: match found with the PnP device 
'00:07' and the driver 'system'
Dec 15 05:34:30 where kernel: pnp: match found with the PnP device 
'00:08' and the driver 'system'
Dec 15 05:34:30 where kernel: PnPBIOS: 13 nodes reported by PnP BIOS; 13 
recorded by driver
Dec 15 05:34:30 where kernel: SCSI subsystem initialized
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCS] enabled at 
IRQ 23
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-23 -> 
0xa9 -> IRQ 23 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:00:01[A] -> 2-23 -> IRQ 23
Dec 15 05:34:30 where kernel: Pin 2-23 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCF] enabled at 
IRQ 20
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-20 -> 
0xb1 -> IRQ 20 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:00:02[A] -> 2-20 -> IRQ 20
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCG] enabled at 
IRQ 22
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-22 -> 
0xb9 -> IRQ 22 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:00:02[B] -> 2-22 -> IRQ 22
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCL] enabled at 
IRQ 21
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-21 -> 
0xc1 -> IRQ 21 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:00:02[C] -> 2-21 -> IRQ 21
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCH] enabled at 
IRQ 20
Dec 15 05:34:30 where kernel: Pin 2-20 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCI] enabled at 
IRQ 22
Dec 15 05:34:30 where kernel: Pin 2-22 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCJ] enabled at 
IRQ 21
Dec 15 05:34:30 where kernel: Pin 2-21 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCK] enabled at 
IRQ 20
Dec 15 05:34:30 where kernel: Pin 2-20 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCM] enabled at 
IRQ 22
Dec 15 05:34:30 where kernel: Pin 2-22 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [AP3C] enabled at 
IRQ 21
Dec 15 05:34:30 where kernel: Pin 2-21 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APCZ] enabled at 
IRQ 20
Dec 15 05:34:30 where kernel: Pin 2-20 already programmed
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC4] enabled at 
IRQ 19
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-19 -> 
0xc9 -> IRQ 19 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:01:06[A] -> 2-19 -> IRQ 19
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC1] enabled at 
IRQ 16
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-16 -> 
0xd1 -> IRQ 16 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:01:06[B] -> 2-16 -> IRQ 16
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC2] enabled at 
IRQ 17
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-17 -> 
0xd9 -> IRQ 17 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:01:06[C] -> 2-17 -> IRQ 17
Dec 15 05:34:30 where kernel: ACPI: PCI Interrupt Link [APC3] enabled at 
IRQ 18
Dec 15 05:34:30 where kernel: IOAPIC[0]: Set PCI routing entry (2-18 -> 
0xe1 -> IRQ 18 Mode:1 Active:0)
Dec 15 05:34:30 where kernel: 00:01:06[D] -> 2-18 -> IRQ 18
Dec 15 05:34:30 where kernel: Pin 2-16 already programmed
Dec 15 05:34:30 where kernel: Pin 2-17 already programmed
Dec 15 05:34:30 where kernel: Pin 2-18 already programmed
Dec 15 05:34:30 where kernel: Pin 2-19 already programmed
Dec 15 05:34:30 where kernel: Pin 2-17 already programmed
Dec 15 05:34:30 where kernel: Pin 2-18 already programmed
Dec 15 05:34:30 where kernel: Pin 2-19 already programmed
Dec 15 05:34:30 where kernel: Pin 2-16 already programmed
Dec 15 05:34:30 where kernel: Pin 2-19 already programmed
Dec 15 05:34:30 where kernel: Pin 2-16 already programmed
Dec 15 05:34:30 where kernel: Pin 2-17 already programmed
Dec 15 05:34:30 where kernel: Pin 2-18 already programmed
Dec 15 05:34:30 where kernel: Pin 2-16 already programmed
Dec 15 05:34:30 where kernel: Pin 2-17 already programmed
Dec 15 05:34:30 where kernel: Pin 2-18 already programmed
Dec 15 05:34:30 where kernel: Pin 2-19 already programmed
Dec 15 05:34:30 where kernel: Pin 2-18 already programmed
Dec 15 05:34:30 where last message repeated 3 times
Dec 15 05:34:30 where kernel: Pin 2-19 already programmed
Dec 15 05:34:30 where kernel: PCI: Using ACPI for IRQ routing

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[ross.alexander]

Hi all,

Just a quick note saying I've been running my nforce2 for nearly four
days with considerable I/O and no trouble.

Processor: AMD Athlon XP 2700+
MB: ASUS A7N8X Deluxe
APCI: Turned off in kernel config
Boot params: acpi=off noapic
Kernel: 2.6.0-test11-bk7 + disconnect quirk patch

I still have a considerable number of spurious IRQs (but < 1%
compared to IRQ0 / LINT).  I'm not running the timer patch
at all.

I will try with latest bk kernel + disconnect  + acpi compiled in.

Questions.

1) Does anybody know the state of play ala 2.6.0 release and which
patches we can get in?

2) If the local apic is running is it necessary to use the 8254 as a 
timer?

3) Does anybody, anywhere, have any nvidia documentation and
is it worth trying to bully them into releasing some?

Cheers,

Ross

---------------------------------------------------------------------------------
Ross Alexander                           "We demand clearly defined
MIS - NEC Europe Limited            boundaries of uncertainty and
Work ph: +44 20 8752 3394         doubt."

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Jamie Lokier]

Ross Dickson wrote:
> Hmmm It may well be the settling time of the cpu PLL (phase lock
> loop) changing from low power disconnect speed up to bus speed -
> higher speed bus on the same type of silicon may take longer? as
> also it may take more power to run faster so the internal bus bit
> drivers may also take more time to settle? These sorts of things are
> mentioned in earlier model athlon errata. Especially if there is perhaps
> a marginal northbridge timing in the area Ian has thought about?

If you're waiting for a PLL to settle, it could vary tremendously.

Is there some way to sense the disconnect state and put in a proper
delay e.g. 10 microseconds when it is sensed?  I'd guess the
disconnect state isn't encounted much under system load, which is the
only time you really care about fast timer interrupts.

> I am now going to try increasing the wait loop delay from 100ns to 400ns
> in case the apic does not like being hammered repetitively during the delay
> time. - It could be that the bus between the cpu core and the local apic is
> marginal on either timing (PLL) or current and if we hammer it we may
> be asking for incorrect reads?

It's possible that hammering it will consume more bus power and push
a stabilising PLL and/or power supply a bit far.

If it's purely marginal timing, then doing fewer reads just means
you'll hit the marginal state less often, not avoid it completely.

-- Jamie

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

Resend due to no CC...

On Sun, 2003-12-14 at 14:11, Ross Dickson wrote:
> I had a lockup on a boot so I am trying a bit more conservative with
> 
> 1000UL and ndelay(400) 
> 
> I don't think anyone should try any less than this but hey?
> 
> This gives my system a safety margin of 16 apic counts.
> The v1 patch on my system typically gave a safety delay of 13 counts.
> 
> The performance hit with the v2 is still less than with the v1.
> With v2 additional delay would only have been present on 2 out of 10
> instead of 10 out of 10 with v1.

13h 18min here now, with 800UL 100ns

Works like a charm. It seems like we are just avoiding a timer race with
some values...  ie more delay might not solve the problem.

-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

On Sunday 14 December 2003 21:24, Ross Dickson wrote:
<snip>
> 
> The v2 io-apic mods seems OK so far.
> 
> I am not yet sure the v2 apic patch is going to be stable enough for everyone.
> I suspect it is the apic reads.
> 
> Ian had problems at 600ns so now has been trying 800ns. I have yet to hear how
> his machine is.
> 
> I have also gone from 600ns to 800ns and had my first hard lockup
> after about 6 hrs on 800ns delay timeout. 
> 
> I am using a heavily patched 2.4.23 kern so it could be a coincidence
> but I am suspicious that it may not be that safe to be reading the apic
> registers at all during the delay timeout as the v2 apic patch does.
> 
> I am now going to try increasing the wait loop delay from 100ns to 400ns
> in case the apic does not like being hammered repetitively during the delay
> time. - It could be that the bus between the cpu core and the local apic is
> marginal on either timing (PLL) or current and if we hammer it we may
> be asking for incorrect reads?
> 
> I am about to recompile with the following values that differ from my
> original v2 posting. 
> ( 800UL and the ndelay(400) )

<snip>
I had a lockup on a boot so I am trying a bit more conservative with

1000UL and ndelay(400) 

I don't think anyone should try any less than this but hey?

This gives my system a safety margin of 16 apic counts.
The v1 patch on my system typically gave a safety delay of 13 counts.

The performance hit with the v2 is still less than with the v1.
With v2 additional delay would only have been present on 2 out of 10
instead of 10 out of 10 with v1.

..APIC TIMER ack delay, reload:16701, safe:16685
..APIC TIMER ack delay, predelay count:16657
..APIC TIMER ack delay, predelay count:16664
..APIC TIMER ack delay, predelay count:16677
..APIC TIMER ack delay, predelay count:16691
..APIC TIMER ack delay, predelay count:16636
..APIC TIMER ack delay, predelay count:16649
..APIC TIMER ack delay, predelay count:16660
..APIC TIMER ack delay, predelay count:16671
..APIC TIMER ack delay, predelay count:16686
..APIC TIMER ack delay, predelay count:16635

Although as previously mentioned on my system it takes 28 apic counts
to write an io byte to the south bridge such as an ack in xtpic mode
so v1 was not what you would call slow.

If 28 counts was a benchmark then I could still try v2 up to about 1750UL and
still be quicker than the xtpic.

Ross.

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

On Sunday 14 December 2003 14:27, Jamie Lokier wrote:
> Ross Dickson wrote:
> > The v1 patch ((cpu_khz >> 12)+200) gave 700ns additional delay to
> > your existing code path so I think I was being too optimistic in the
> > initial v2 timing. I made the initial timing CPU freq dependent on
> > the assumption that a faster cpu would get to the delay point
> > quicker.
> 
> Maybe it scales with bus speed, not CPU internal speed?

Very good point.

Hmmm It may well be the settling time of the cpu PLL (phase lock
loop) changing from low power disconnect speed up to bus speed -
higher speed bus on the same type of silicon may take longer? as
also it may take more power to run faster so the internal bus bit
drivers may also take more time to settle? These sorts of things are
mentioned in earlier model athlon errata. Especially if there is perhaps
a marginal northbridge timing in the area Ian has thought about?

The nforce2 ram controller is also unique and kind of predicts and caches
reads so maybe the apic timer irq code is ready to run a lot sooner than 
other chipsets so the normal delays wrt loading from memory are not there?
Are we compensating for that? We don't really know unfortunately.

> 
> -- Jamie
> 
> 
> 

Josh - Thanks very much for the patches.

The v2 io-apic mods seems OK so far.

I am not yet sure the v2 apic patch is going to be stable enough for everyone.
I suspect it is the apic reads.

Ian had problems at 600ns so now has been trying 800ns. I have yet to hear how
his machine is.

I have also gone from 600ns to 800ns and had my first hard lockup
after about 6 hrs on 800ns delay timeout. 

I am using a heavily patched 2.4.23 kern so it could be a coincidence
but I am suspicious that it may not be that safe to be reading the apic
registers at all during the delay timeout as the v2 apic patch does.

I am now going to try increasing the wait loop delay from 100ns to 400ns
in case the apic does not like being hammered repetitively during the delay
time. - It could be that the bus between the cpu core and the local apic is
marginal on either timing (PLL) or current and if we hammer it we may
be asking for incorrect reads?

I am about to recompile with the following values that differ from my
original v2 posting. 
( 800UL and the ndelay(400) )

#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
	/*
	 * on AMDXP & nforce2 chipset we need about 800ns?
	 * from timer irq start to apic irq ack to prevent
	 * hard lockups, use apic timer itself.
	 * C1 disconnect bit related.  Ross Dickson.
	 */
	{
		static unsigned int passno, safecnt;
		if(!passno) { /* calculate timing */
			safecnt = apic_read(APIC_TMICT) -
				( (800UL * apic_read(APIC_TMICT) ) /
				(1000000000UL/HZ) );
			printk("..APIC TIMER ack delay, reload:%u, safe:%u\n",
				apic_read(APIC_TMICT), safecnt);
			passno++;
		}
#if APIC_DEBUG
		if(passno<12) {
			unsigned int at1 = apic_read(APIC_TMCCT);
			if( passno > 1 )
				Dprintk("..APIC TIMER ack delay, predelay count:%u \n", at1 );
			passno++;
		}
# endif
		/* delay only if required */
		while( apic_read(APIC_TMCCT) > safecnt )
			ndelay(400);
	}
#endif


Regards
Ross.

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Josh McKinney]

[-- Attachment #1: Type: text/plain, Size: 1235 bytes --]

On approximately Sat, Dec 13, 2003 at 10:38:41PM +0100, Ian Kumlien wrote:
> On Sat, 2003-12-13 at 19:07, Ross Dickson wrote:
> > Greetings,
> > I have updated the apic timer ack delay patch and the io-apic edge patch
> > although I do not think anyone had any problems with the first release.
> 
<snip>
> 
> Right now i'm compiling and hope that the patch doesn't need more
> workups. (Josh McKinney patches missed some things, more on that further
> down)
> 
<snip>
> 
> apic part is missing a closing ) and the last part of the #ifdef =)
> So, either rediff or let the users fix that up =)
> 
> -- 
> Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

Oops.  I was playing kernel hacker while I should have been doing
other things.  Patch rediffed and both attached.  Thanks for catching
my mess up.

-- 
Josh McKinney		     |	Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
                             | They that can give up essential liberty
Linux, the choice       -o)  | to obtain a little temporary safety deserve 
of the GNU generation    /\  | neither liberty or safety. 
                        _\_v |                          -Benjamin Franklin

[-- Attachment #2: 2.6.0-test11-ioapic_v2.patch --]
[-- Type: text/plain, Size: 2587 bytes --]

diff -urN linux-2.6.0-test11/arch/i386/kernel/io_apic.c linux-2.6.0-test11-nf2/arch/i386/kernel/io_apic.c
--- linux-2.6.0-test11/arch/i386/kernel/io_apic.c	2003-11-26 15:43:32.000000000 -0500
+++ linux-2.6.0-test11-nf2/arch/i386/kernel/io_apic.c	2003-12-13 15:14:25.000000000 -0500
@@ -2128,6 +2128,54 @@
 		printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n");
 	}
 
+#ifdef CONFIG_ACPI_BOOT && CONFIG_X86_UP_IOAPIC
+        /* for nforce2 try vector 0 on pin0
+         * Note 8259a is already masked, also by default
+         * the io_apic_set_pci_routing call disables the 8259 irq 0
+         * so we must be connected directly to the 8254 timer if this works
+         * Note2: this violates the above comment re Subtle but works!
+         */
+        printk(KERN_INFO "..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...\n");
+        if (pin1 != -1) {
+                extern spinlock_t i8259A_lock;
+                unsigned long flags;
+                int tok, saved_timer_ack = timer_ack;
+                /*
+                 * Ok, does IRQ0 through the IOAPIC work?
+                 */
+                io_apic_set_pci_routing ( 0, 0, 0, 0, 0); /* connect pin */
+                unmask_IO_APIC_irq(0);
+                timer_ack = 0;
+
+                /*
+
+
+
+                 * Ok, does IRQ0 through the IOAPIC work?
+                 */
+                spin_lock_irqsave(&i8259A_lock, flags);
+                Dprintk("..TIMER check 8259 ints disabled, imr1:%02x, imr2:%02x\n", inb(0x21), inb(0xA1));
+                tok = timer_irq_works();
+                spin_unlock_irqrestore(&i8259A_lock, flags);
+                if (tok) {
+                        if (nmi_watchdog == NMI_IO_APIC) {
+                                disable_8259A_irq(0);
+                                setup_nmi();
+                                enable_8259A_irq(0);
+                                check_nmi_watchdog();
+                        }
+                        printk(KERN_INFO "..TIMER: works OK on apic pin0 irq0\n" );
+                        return;
+                }
+                /* failed */
+                timer_ack = saved_timer_ack;
+                clear_IO_APIC_pin(0, 0);
+                io_apic_set_pci_routing ( 0, pin1, 0, 0, 0);
+                printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC Pin 0\n");
+        }
+/* end new stuff for nforce2 */
+#endif
+
 	printk(KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... ");
 	if (pin2 != -1) {
 		printk("\n..... (found pin %d) ...", pin2);

[-- Attachment #3: 2.6.0-test11-apic-delay_v2.patch --]
[-- Type: text/plain, Size: 1700 bytes --]

diff -urN linux-2.6.0-test11/arch/i386/kernel/apic.c linux-2.6.0-test11-nf2/arch/i386/kernel/apic.c
--- linux-2.6.0-test11/arch/i386/kernel/apic.c	2003-11-26 15:46:07.000000000 -0500
+++ linux-2.6.0-test11-nf2/arch/i386/kernel/apic.c	2003-12-13 23:48:30.000000000 -0500
@@ -1089,6 +1089,37 @@
 	 */
 	irq_stat[cpu].apic_timer_irqs++;
 
+#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
+        /*
+         * on 2200XP & nforce2 chipset we need 600ns?
+         * from timer irq start to apic irq ack to prevent
+         * hard lockups, use apic timer itself.
+         * C1 disconnect bit related.  Ross Dickson.
+         */
+        {
+                static unsigned int passno, safecnt;
+                if(!passno) { /* calculate timing */
+                        safecnt = apic_read(APIC_TMICT) -
+                                ( (600UL * apic_read(APIC_TMICT) ) /
+                                (1000000000UL/HZ) );
+                        printk("..APIC TIMER ack delay, reload:%u, safe:%u\n",
+                                apic_read(APIC_TMICT), safecnt);
+                        passno++;
+                }
+#if APIC_DEBUG
+                if(passno<12) {
+                        unsigned int at1 = apic_read(APIC_TMCCT);
+                        if( passno > 1 )
+                                Dprintk("..APIC TIMER ack delay, predelay count:%u \n", at1 );
+                        passno++;
+                }
+# endif
+                /* delay only if required */
+                while( apic_read(APIC_TMCCT) > safecnt )
+                        ndelay(100);
+        }
+#endif
+
 	/*
 	 * NOTE! We'd better ACK the irq immediately,
 	 * because timer handling can be slow.

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Jamie Lokier]

Ross Dickson wrote:
> The v1 patch ((cpu_khz >> 12)+200) gave 700ns additional delay to
> your existing code path so I think I was being too optimistic in the
> initial v2 timing. I made the initial timing CPU freq dependent on
> the assumption that a faster cpu would get to the delay point
> quicker.

Maybe it scales with bus speed, not CPU internal speed?

-- Jamie

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

On Sunday 14 December 2003 09:21, you wrote:
> On Sun, 2003-12-14 at 00:16, Ross Dickson wrote:
> > On Sunday 14 December 2003 08:28, you wrote:
> > > On Sat, 2003-12-13 at 19:07, Ross Dickson wrote:
> > > > ..APIC TIMER ack delay, reload:16701, safe:16691
> > > 
> > > calibrating APIC timer ...
> > > ..... CPU clock speed is 2079.0146 MHz.
> > > ..... host bus clock speed is 332.0663 MHz.
> > > NET: Registered protocol family 16
> > > ..APIC TIMER ack delay, reload:20791, safe:20779
> > > ..APIC TIMER ack delay, predelay count: 20769
> > > ..APIC TIMER ack delay, predelay count: 20786
> > > ..APIC TIMER ack delay, predelay count: 20716
> > > ..APIC TIMER ack delay, predelay count: 20731
> > > ..APIC TIMER ack delay, predelay count: 20747
> > > ..APIC TIMER ack delay, predelay count: 20762
> > > ..APIC TIMER ack delay, predelay count: 20780
> > > ..APIC TIMER ack delay, predelay count: 20729
> > > ..APIC TIMER ack delay, predelay count: 20740
> > > ..APIC TIMER ack delay, predelay count: 20757
> > 
> > Thanks Ian.
> > From this we see your local apic is indeed counting 1.2 times faster than mine
> > ratio of 333/266 fsb. So the reload:20791 - safe:20779 gives 12 counts time.
> > Given 20791 is 1ms on your system then your 12 counts is 577ns
> > But more importantly from the ack delay theory as your machine like mine is
> > prone to lockups then a lockup could likely have occured at count:20786 having
> > only 240ns time expired. Next worst case was less likely to lockup at count:20780.
> 
> I just had a lockup running with preempt, now trying with preempt
> disabled. This is a clean 2.6.0-test11 with just io-apic and apic v2
> patches.

Thanks for testing, thats not good news.
Try rebooting with kernel args to get back in.

acpi=off noapic

My 600ns is likely too small - try putting 800UL or 1000UL in place of the 600UL 

The v1 patch ((cpu_khz >> 12)+200) gave 700ns additional delay to your existing
code path so I think I was being too optimistic in the initial v2 timing. I made the
initial timing CPU freq dependent on the assumption that a faster cpu would get to
the delay point quicker. 

On my system the v1 patch had 13 counts of total delay whereas my initial the v2
gave only 10 so I think 800UL is the smallest value we should use.

If that does not fix it then it could also be that reading the timer so early may also
be dangerous? or something else? and so then we would go back to v1 apic patch.

> 
> > The only ones any delay would have been added to by the patch would be the
> > count:20786 and count:20780 and it would have been just enough to wait until 
> > the counter got below the safe:20779 so the patch contributes little overhead.
>  
> > > Survived my greptest which no non patched kernel has ever done on this
> > > machine.
> > > 
> > > Has anyone got that extended ringbuffer to work? I haven't been able to
> > > get a complete "boot" dmesg in ages because of all the output all the
> > > drivers make... Does it need a updated dmesg?
> > 
> > This may be what you have already tried:
> > I am not sure where it is in the 2.6 config or indeed if it is different but it is
> > CONFIG_LOG_BUF_SHIFT under kernel hacking on 2.4.23 maybe try 16 for 64K.
> > To match dmesg output try 
> > 
> > dmesg -s65536
> > 
> > (unless dmesg can automatically pick up the expanded ring buffer size on 2.6?)
> 
> Ahhh great!, no, it doesn't auto detect it... Maybe there is a newer
> version, i hate mdk for being so nice to new versions and ignoring the
> old.

My wishful thinking, I don't think any version autodetects but it would be a nice feature.
Ross.
> 
> -- 
> Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
> 

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 308 bytes --]

Hi again, 

Unfortunately that kernel deadlocked as well... Both times while
downloading, is there some patches that i have missed for 2.6?

Please send em all if you have been successful in running it past the
'crash deadline'.

-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 2702 bytes --]

On Sun, 2003-12-14 at 00:16, Ross Dickson wrote:
> On Sunday 14 December 2003 08:28, you wrote:
> > On Sat, 2003-12-13 at 19:07, Ross Dickson wrote:
> > > ..APIC TIMER ack delay, reload:16701, safe:16691
> > 
> > calibrating APIC timer ...
> > ..... CPU clock speed is 2079.0146 MHz.
> > ..... host bus clock speed is 332.0663 MHz.
> > NET: Registered protocol family 16
> > ..APIC TIMER ack delay, reload:20791, safe:20779
> > ..APIC TIMER ack delay, predelay count: 20769
> > ..APIC TIMER ack delay, predelay count: 20786
> > ..APIC TIMER ack delay, predelay count: 20716
> > ..APIC TIMER ack delay, predelay count: 20731
> > ..APIC TIMER ack delay, predelay count: 20747
> > ..APIC TIMER ack delay, predelay count: 20762
> > ..APIC TIMER ack delay, predelay count: 20780
> > ..APIC TIMER ack delay, predelay count: 20729
> > ..APIC TIMER ack delay, predelay count: 20740
> > ..APIC TIMER ack delay, predelay count: 20757
> 
> Thanks Ian.
> From this we see your local apic is indeed counting 1.2 times faster than mine
> ratio of 333/266 fsb. So the reload:20791 - safe:20779 gives 12 counts time.
> Given 20791 is 1ms on your system then your 12 counts is 577ns
> But more importantly from the ack delay theory as your machine like mine is
> prone to lockups then a lockup could likely have occured at count:20786 having
> only 240ns time expired. Next worst case was less likely to lockup at count:20780.

I just had a lockup running with preempt, now trying with preempt
disabled. This is a clean 2.6.0-test11 with just io-apic and apic v2
patches.

> The only ones any delay would have been added to by the patch would be the
> count:20786 and count:20780 and it would have been just enough to wait until 
> the counter got below the safe:20779 so the patch contributes little overhead.
 
> > Survived my greptest which no non patched kernel has ever done on this
> > machine.
> > 
> > Has anyone got that extended ringbuffer to work? I haven't been able to
> > get a complete "boot" dmesg in ages because of all the output all the
> > drivers make... Does it need a updated dmesg?
> 
> This may be what you have already tried:
> I am not sure where it is in the 2.6 config or indeed if it is different but it is
> CONFIG_LOG_BUF_SHIFT under kernel hacking on 2.4.23 maybe try 16 for 64K.
> To match dmesg output try 
> 
> dmesg -s65536
> 
> (unless dmesg can automatically pick up the expanded ring buffer size on 2.6?)

Ahhh great!, no, it doesn't auto detect it... Maybe there is a newer
version, i hate mdk for being so nice to new versions and ignoring the
old.

-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

On Sunday 14 December 2003 08:28, you wrote:
> On Sat, 2003-12-13 at 19:07, Ross Dickson wrote:
> > ..APIC TIMER ack delay, reload:16701, safe:16691
> 
> calibrating APIC timer ...
> ..... CPU clock speed is 2079.0146 MHz.
> ..... host bus clock speed is 332.0663 MHz.
> NET: Registered protocol family 16
> ..APIC TIMER ack delay, reload:20791, safe:20779
> ..APIC TIMER ack delay, predelay count: 20769
> ..APIC TIMER ack delay, predelay count: 20786
> ..APIC TIMER ack delay, predelay count: 20716
> ..APIC TIMER ack delay, predelay count: 20731
> ..APIC TIMER ack delay, predelay count: 20747
> ..APIC TIMER ack delay, predelay count: 20762
> ..APIC TIMER ack delay, predelay count: 20780
> ..APIC TIMER ack delay, predelay count: 20729
> ..APIC TIMER ack delay, predelay count: 20740
> ..APIC TIMER ack delay, predelay count: 20757

Thanks Ian.
>From this we see your local apic is indeed counting 1.2 times faster than mine
ratio of 333/266 fsb. So the reload:20791 - safe:20779 gives 12 counts time.
Given 20791 is 1ms on your system then your 12 counts is 577ns
But more importantly from the ack delay theory as your machine like mine is
prone to lockups then a lockup could likely have occured at count:20786 having
only 240ns time expired. Next worst case was less likely to lockup at count:20780.

The only ones any delay would have been added to by the patch would be the
count:20786 and count:20780 and it would have been just enough to wait until 
the counter got below the safe:20779 so the patch contributes little overhead.
 
> ---
> 
> Survived my greptest which no non patched kernel has ever done on this
> machine.
> 
> Has anyone got that extended ringbuffer to work? I haven't been able to
> get a complete "boot" dmesg in ages because of all the output all the
> drivers make... Does it need a updated dmesg?

This may be what you have already tried:
I am not sure where it is in the 2.6 config or indeed if it is different but it is
CONFIG_LOG_BUF_SHIFT under kernel hacking on 2.4.23 maybe try 16 for 64K.
To match dmesg output try 

dmesg -s65536

(unless dmesg can automatically pick up the expanded ring buffer size on 2.6?)


> (I mean, is this like the non updated gnome-terminal in mdk 9.1 that
> deadlocks on 2.6 if you run some ncurces apps in a "larger than usual"
> window?)
> 
> -- 
> Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
> 

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

On Sat, 2003-12-13 at 19:07, Ross Dickson wrote:
> ..APIC TIMER ack delay, reload:16701, safe:16691

calibrating APIC timer ...
..... CPU clock speed is 2079.0146 MHz.
..... host bus clock speed is 332.0663 MHz.
NET: Registered protocol family 16
..APIC TIMER ack delay, reload:20791, safe:20779
..APIC TIMER ack delay, predelay count: 20769
..APIC TIMER ack delay, predelay count: 20786
..APIC TIMER ack delay, predelay count: 20716
..APIC TIMER ack delay, predelay count: 20731
..APIC TIMER ack delay, predelay count: 20747
..APIC TIMER ack delay, predelay count: 20762
..APIC TIMER ack delay, predelay count: 20780
..APIC TIMER ack delay, predelay count: 20729
..APIC TIMER ack delay, predelay count: 20740
..APIC TIMER ack delay, predelay count: 20757
---

Survived my greptest which no non patched kernel has ever done on this
machine.

Has anyone got that extended ringbuffer to work? I haven't been able to
get a complete "boot" dmesg in ages because of all the output all the
drivers make... Does it need a updated dmesg?
(I mean, is this like the non updated gnome-terminal in mdk 9.1 that
deadlocks on 2.6 if you run some ncurces apps in a "larger than usual"
window?)

-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 4112 bytes --]

On Sat, 2003-12-13 at 19:07, Ross Dickson wrote:
> Greetings,
> I have updated the apic timer ack delay patch and the io-apic edge patch
> although I do not think anyone had any problems with the first release.

I have 3days of uptime here... It's low due to me wanting to test dual
channel memory...

> These patches may not be required if your bios is sorted out - mine are
> not yet.

Neither is mine =P

> The apic ack delay no longer adds a small delay to every timer int so I 
> believe its performance hit is barely detectable. 

Niiice

> In fact the busier the system gets with irqs- the more likelyhood 
> of the delay time expiring naturally and the less the impact of the patch.
> 
> It now only delays the ones that would be too quick and most likely
> cause a hard lockup. It also reports its existence on boot if used. e.g.
> 
> ..APIC TIMER ack delay, reload:16701, safe:16691
> 
> And if  
> 
> #define APIC_DEBUG 0
> 
> is set to 1 in 
> 
> /usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h

Right now i'm compiling and hope that the patch doesn't need more
workups. (Josh McKinney patches missed some things, more on that further
down)

> Then you can report at boot ten pre delay apic timer times as well to get a 
> feel as to whether the delay had to kick in or not. Note that ten is a very
> small sample size but it gives an idea of the timing numbers.
> 
> The apic timer counts down from the reload value and interrupts at zero.
> The reload value corresponds to the time between HZ at the rate the 
> counter counts. Nothing new so far.
> e.g. My system is running 1000Hz so the time is 1ms.
> The reload value is 16701 so that represents 1ms.
> 
> The safe count it calculates for my system is 16691 so any number smaller
> than that is expected to be when it is safe to ack the local apic after an apic
> timer interrupt. If you want to experiment with the delay time in nano seconds
> just change the 600UL. If I set mine to 400UL then my hard lockups return.
> Interestingly I can read the local apic timer safely right after the interrupt - 
> I just can't safely ack it then. It is as if the C1 disconnect logic within the
> processor only screws with the ACK irq cancellation logic and then only for
> a short time after an irq.

Did you see the 240usecond delay that happens after a disconnect? But
according to the amd northbridge spec, the pci bus will be disabled or
something at that time...

> The io-apic edge patch has been cleaned up a little. If the above debug is on
> then it will display the 8259a xtpic interrupt mask. I get hex fb and hex ff 
> indicating that the only int enabled on the 8259A xtpic which handles irq 0-7
> is the cascade irq 2 which is OK because on the other 8259A irq 8-15 are masked.
> This masked xtpic should always be the case. We want the 8259A off during
> our test to see if the 8254 timer is connected directly to pin 0 of the ioapic.
> The other messages are as per previous version.
> 
> 
> Lastly I need sleep (4 am) so they are not yet done as patch files. I have
> put them into two text files and bzipped them as an attachment. Anyhow they
> are small and insert in the same places as their previous versions.
> 
> Thanks Jesse for rediffing the original patches for 2.6 would you like to repeat
> the favour with these please?
> 
> Again they are still experimental.... so I am hoping Ian, Jesse and others will
> put them to the test.

Compiling 2.6.0 atm, since i want to test that somemore on this
machine... 

Josh McKinney wrote:
> I am not Jesse but I thought I would diff them for 2.6 myself.  Thanks
> Ross and others for putting so much time and effort into this.  I
> haven't had time to test them, but I soon will.  Uptime is:  
>  15:19:45 up 2 days,  3:53,  7 users,  load average: 0.05, 0.28, 0.37
> with the original v1 patches.

apic part is missing a closing ) and the last part of the #ifdef =)
So, either rediff or let the users fix that up =)

-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Josh McKinney]

[-- Attachment #1: Type: text/plain, Size: 1243 bytes --]

On approximately Sun, Dec 14, 2003 at 04:07:28AM +1000, Ross Dickson wrote:
> Greetings,
> I have updated the apic timer ack delay patch and the io-apic edge patch
> although I do not think anyone had any problems with the first release.
> 
<snip>
> 
> Thanks Jesse for rediffing the original patches for 2.6 would you like to repeat
> the favour with these please?
> 
> Again they are still experimental.... so I am hoping Ian, Jesse and others will
> put them to the test.
> 
> Thanks
> Ross Dickson.
> 

I am not Jesse but I thought I would diff them for 2.6 myself.  Thanks
Ross and others for putting so much time and effort into this.  I
haven't had time to test them, but I soon will.  Uptime is:  
 15:19:45 up 2 days,  3:53,  7 users,  load average: 0.05, 0.28, 0.37
with the original v1 patches.

Patches attached.

-- 
Josh McKinney		     |	Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
                             | They that can give up essential liberty
Linux, the choice       -o)  | to obtain a little temporary safety deserve 
of the GNU generation    /\  | neither liberty or safety. 
                        _\_v |                          -Benjamin Franklin

[-- Attachment #2: 2.6.0-test11-apic_delay_v2.patch --]
[-- Type: text/plain, Size: 1207 bytes --]

diff -urN linux-2.6.0-test11/arch/i386/kernel/apic.c linux-2.6.0-test11-nf2/arch/i386/kernel/apic.c
--- linux-2.6.0-test11/arch/i386/kernel/apic.c	2003-11-26 15:46:07.000000000 -0500
+++ linux-2.6.0-test11-nf2/arch/i386/kernel/apic.c	2003-12-13 15:04:38.000000000 -0500
@@ -1089,6 +1089,32 @@
 	 */
 	irq_stat[cpu].apic_timer_irqs++;
 
+#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
+	/*
+	 * on 2200XP & nforce2 chipset we need 600ns?
+	 * from timer irq start to apic irq ack to prevent
+	 * hard lockups, use apic timer itself.
+	 * C1 disconnect bit related.  Ross Dickson.
+	 */
+	{
+	  static unsigned int passno, safecnt;
+	  if(!passno {/*calculate timing*/
+	    safecnt = apic_read(APIC_TMICT) -
+	      ( (600UL * apic_read(APIC_TMICT) ) /
+		(1000000000UL/HZ) );
+	    printk("..APIC TIMER ack delay, reload:%u, safe:%u\n",
+		   apic_read(APIC_TMICT), safecnt);
+	    passno++;
+	  }
+#if APIC_DEBUG
+	     if(passno<12) {
+	       unsigned int at1 = apic_read(APIC_TMCCT);
+	       if(passno>1)
+		 Dprintk("..APIC TIMER ack delay, predelay count: %u \n", at1 );
+	       passno++;
+	     }
+#endif
+
 	/*
 	 * NOTE! We'd better ACK the irq immediately,
 	 * because timer handling can be slow.

[-- Attachment #3: 2.6.0-test11-ioapic_v2.patch --]
[-- Type: text/plain, Size: 2587 bytes --]

diff -urN linux-2.6.0-test11/arch/i386/kernel/io_apic.c linux-2.6.0-test11-nf2/arch/i386/kernel/io_apic.c
--- linux-2.6.0-test11/arch/i386/kernel/io_apic.c	2003-11-26 15:43:32.000000000 -0500
+++ linux-2.6.0-test11-nf2/arch/i386/kernel/io_apic.c	2003-12-13 15:14:25.000000000 -0500
@@ -2128,6 +2128,54 @@
 		printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n");
 	}
 
+#ifdef CONFIG_ACPI_BOOT && CONFIG_X86_UP_IOAPIC
+        /* for nforce2 try vector 0 on pin0
+         * Note 8259a is already masked, also by default
+         * the io_apic_set_pci_routing call disables the 8259 irq 0
+         * so we must be connected directly to the 8254 timer if this works
+         * Note2: this violates the above comment re Subtle but works!
+         */
+        printk(KERN_INFO "..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...\n");
+        if (pin1 != -1) {
+                extern spinlock_t i8259A_lock;
+                unsigned long flags;
+                int tok, saved_timer_ack = timer_ack;
+                /*
+                 * Ok, does IRQ0 through the IOAPIC work?
+                 */
+                io_apic_set_pci_routing ( 0, 0, 0, 0, 0); /* connect pin */
+                unmask_IO_APIC_irq(0);
+                timer_ack = 0;
+
+                /*
+
+
+
+                 * Ok, does IRQ0 through the IOAPIC work?
+                 */
+                spin_lock_irqsave(&i8259A_lock, flags);
+                Dprintk("..TIMER check 8259 ints disabled, imr1:%02x, imr2:%02x\n", inb(0x21), inb(0xA1));
+                tok = timer_irq_works();
+                spin_unlock_irqrestore(&i8259A_lock, flags);
+                if (tok) {
+                        if (nmi_watchdog == NMI_IO_APIC) {
+                                disable_8259A_irq(0);
+                                setup_nmi();
+                                enable_8259A_irq(0);
+                                check_nmi_watchdog();
+                        }
+                        printk(KERN_INFO "..TIMER: works OK on apic pin0 irq0\n" );
+                        return;
+                }
+                /* failed */
+                timer_ack = saved_timer_ack;
+                clear_IO_APIC_pin(0, 0);
+                io_apic_set_pci_routing ( 0, pin1, 0, 0, 0);
+                printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC Pin 0\n");
+        }
+/* end new stuff for nforce2 */
+#endif
+
 	printk(KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... ");
 	if (pin2 != -1) {
 		printk("\n..... (found pin %d) ...", pin2);

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

[-- Attachment #1: Type: text/plain, Size: 2849 bytes --]

Greetings,
I have updated the apic timer ack delay patch and the io-apic edge patch
although I do not think anyone had any problems with the first release.

These patches may not be required if your bios is sorted out - mine are
not yet.

The apic ack delay no longer adds a small delay to every timer int so I 
believe its performance hit is barely detectable. 

In fact the busier the system gets with irqs- the more likelyhood 
of the delay time expiring naturally and the less the impact of the patch.

It now only delays the ones that would be too quick and most likely
cause a hard lockup. It also reports its existence on boot if used. e.g.

..APIC TIMER ack delay, reload:16701, safe:16691

And if  

#define APIC_DEBUG 0

is set to 1 in 

/usr/src/linux-2.4.23-rd2/include/asm-i386/apic.h

Then you can report at boot ten pre delay apic timer times as well to get a 
feel as to whether the delay had to kick in or not. Note that ten is a very
small sample size but it gives an idea of the timing numbers.

The apic timer counts down from the reload value and interrupts at zero.
The reload value corresponds to the time between HZ at the rate the 
counter counts. Nothing new so far.
e.g. My system is running 1000Hz so the time is 1ms.
The reload value is 16701 so that represents 1ms.

The safe count it calculates for my system is 16691 so any number smaller
than that is expected to be when it is safe to ack the local apic after an apic
timer interrupt. If you want to experiment with the delay time in nano seconds
just change the 600UL. If I set mine to 400UL then my hard lockups return.
Interestingly I can read the local apic timer safely right after the interrupt - 
I just can't safely ack it then. It is as if the C1 disconnect logic within the
processor only screws with the ACK irq cancellation logic and then only for
a short time after an irq.



The io-apic edge patch has been cleaned up a little. If the above debug is on
then it will display the 8259a xtpic interrupt mask. I get hex fb and hex ff 
indicating that the only int enabled on the 8259A xtpic which handles irq 0-7
is the cascade irq 2 which is OK because on the other 8259A irq 8-15 are masked.
This masked xtpic should always be the case. We want the 8259A off during
our test to see if the 8254 timer is connected directly to pin 0 of the ioapic.
The other messages are as per previous version.


Lastly I need sleep (4 am) so they are not yet done as patch files. I have
put them into two text files and bzipped them as an attachment. Anyhow they
are small and insert in the same places as their previous versions.

Thanks Jesse for rediffing the original patches for 2.6 would you like to repeat
the favour with these please?

Again they are still experimental.... so I am hoping Ian, Jesse and others will
put them to the test.

Thanks
Ross Dickson.


[-- Attachment #2: rossfixes-ver2.tar.bz2 --]
[-- Type: application/x-tbz, Size: 1443 bytes --]

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Bob]

Jesse Allen wrote:

>On Thu, Dec 11, 2003 at 06:52:41PM +0100, Ian Kumlien wrote:
>  
>
>>Heh, yeah, the need for disconnect is somewhat dodgy, i haven't read up
>>on th rest.
>>
>Hmm, weird.  I went to go look at the Shuttle motherboard maker's site - maybe so that I can bug them for a bios disconnect option - but I checked for a bios update first.  And sure enough like they read my mind, just posted online today, an update.  Here are the details of fixes:
>
>" Checksum:   8B00H                         Date Code: 12/05/03
>1.Support 0.18 micron AMD Duron (Palomino) CPU.
>2.Add C1 disconnect item."
>
>It's almost as they're reading this list.  This disconnect problem was discovered on the 5th (well the 5th in my timezone).  Perhaps they're aware of this issue...  I'm gonna talk to them.
>
>Jesse
>
A bios update for MSI K7N2 MCP2-T nforce2 board
fixed the crashing BEFORE these patches were developed,
but there was no documentation that would relate or explain.

http://www.msi.com.tw/program/support/bios/bos/spt_bos_detail.php?UID=436&kind=1
http://download.msi.com.tw/support/bos_exe/6570v76.exe

Award 7.6 at the top of the list. Maybe somebody can figure
out what they're doing.

Nvidia X driver for ti4200 agp8 still locks up linux though,
but X nv works fine. agp8 3d may expose the timer issue.

-Bob

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Jesse Allen]

On Thu, Dec 11, 2003 at 06:52:41PM +0100, Ian Kumlien wrote:
> Heh, yeah, the need for disconnect is somewhat dodgy, i haven't read up
> on th rest.
> 


Hmm, weird.  I went to go look at the Shuttle motherboard maker's site - maybe so that I can bug them for a bios disconnect option - but I checked for a bios update first.  And sure enough like they read my mind, just posted online today, an update.  Here are the details of fixes:

" Checksum:   8B00H                         Date Code: 12/05/03
1.Support 0.18 micron AMD Duron (Palomino) CPU.
2.Add C1 disconnect item."

It's almost as they're reading this list.  This disconnect problem was discovered on the 5th (well the 5th in my timezone).  Perhaps they're aware of this issue...  I'm gonna talk to them.

Jesse

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 1456 bytes --]

On Thu, 2003-12-11 at 10:12, Ross Dickson wrote:
> On Thursday 11 December 2003 21:47, Ian Kumlien wrote:
> Thanks Ian
> 
> Also many thanks for pointing out the relevant section to look in with the AMD
> cpu link that you sent - Credit where credit is due (assuming we are both on the
> right track).

Heh, thanks, feels nice to have someone who agrees with you =).

> I had a read and refined your surmisings. I think the 
> problem appears synchronous with the apic timer because of two reasons.
> 1) any apic irq can cause re-connection of the system bus after disconnect.
> 2) the apic timer irq in my examinations has the shortest path to an ack.

http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/24416.pdf
Page 42 and 94 might help as well. I haven't grasped it all or had any
food yet but i hope i'm right =)

> I also had a look back through the athlon cooler and power management 
> postings and web site articles. I was blissfully ignorant of these issues when I
> started and now I wonder what I have stepped into... Yuk

Heh, yeah, the need for disconnect is somewhat dodgy, i haven't read up
on th rest.

> I submitted a support request to AMD, apologies for not cc'ing you, I kept
> the cc's down to just nvidia and the mailing list. If you have not seen it yet
> then it is here

Thanks

> We hope....

Yup...

-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Jesse Allen]

On Thu, Dec 11, 2003 at 06:04:54PM +0100, Maciej W. Rozycki wrote:
> On Thu, 11 Dec 2003, Josh McKinney wrote:
> 
> > Should I try it with just the acpi fixes sent by Maciej or are these
> > just general fixes?
> 
>  They should make (at least some of) the reported problems go away,
> superseding the respective workarounds.
> 

As far as I can tell, your patch _alone_ doesn't prevent the lockup, fix the timer, or nmi_watchdog.  I have attached a dmesg of my current running kernel that includes Ross' io_apic patch, the disconnect quirk patch, your acpi patch, and other minor patches.  ACPI and APIC debugging are on.


Linux version 2.6.0-test11 (jesse@tesore) (gcc version 3.3.2) #2 Thu Dec 11 09:45:15 MST 2003
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000000fff0000 (usable)
 BIOS-e820: 000000000fff0000 - 000000000fff3000 (ACPI NVS)
 BIOS-e820: 000000000fff3000 - 0000000010000000 (ACPI data)
 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
255MB LOWMEM available.
On node 0 totalpages: 65520
  DMA zone: 4096 pages, LIFO batch:1
  Normal zone: 61424 pages, LIFO batch:14
  HighMem zone: 0 pages, LIFO batch:1
DMI 2.2 present.
ACPI: RSDP (v000 Nvidia                                    ) @ 0x000f6f60
ACPI: RSDT (v001 Nvidia AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x0fff3000
ACPI: FADT (v001 Nvidia AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x0fff3040
ACPI: MADT (v001 Nvidia AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x0fff7880
ACPI: DSDT (v001 NVIDIA AWRDACPI 0x00001000 MSFT 0x0100000e) @ 0x00000000
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 6:10 APIC version 16
ACPI: LAPIC_NMI (acpi_id[0x00] polarity[0x1] trigger[0x1] lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] global_irq_base[0x0])
IOAPIC[0]: Assigned apic_id 2
IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, IRQ 0-23
ACPI: INT_SRC_OVR (bus[0] irq[0x0] global_irq[0x2] polarity[0x0] trigger[0x0])
ACPI: INT_SRC_OVR (bus[0] irq[0x9] global_irq[0x9] polarity[0x1] trigger[0x3])
ACPI: INT_SRC_OVR (bus[0] irq[0xe] global_irq[0xe] polarity[0x1] trigger[0x1])
ACPI: INT_SRC_OVR (bus[0] irq[0xf] global_irq[0xf] polarity[0x1] trigger[0x1])
Enabling APIC mode:  Flat.  Using 1 I/O APICs
Using ACPI (MADT) for SMP configuration information
Building zonelist for node : 0
Kernel command line: BOOT_IMAGE=Linux-2.6 ro root=301
Initializing CPU#0
PID hash table entries: 1024 (order 10: 8192 bytes)
Detected 1913.621 MHz processor.
Console: colour VGA+ 80x25
Memory: 256144k/262080k available (1611k kernel code, 5212k reserved, 693k data, 128k init, 0k highmem)
Calibrating delay loop... 3784.70 BogoMIPS
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU:     After generic identify, caps: 0383fbff c1c3fbff 00000000 00000000
CPU:     After vendor identify, caps: 0383fbff c1c3fbff 00000000 00000000
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU:     After all inits, caps: 0383fbff c1c3fbff 00000000 00000020
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: AMD Athlon(tm) XP 2600+ stepping 00
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
enabled ExtINT on CPU#0
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
ENABLING IO-APIC IRQs
init IO_APIC IRQs
 IO-APIC (apicid-pin) 2-0, 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
..TIMER: vector=0x31 pin1=2 pin2=-1
..MP-BIOS bug: 8254 timer not connected to IO-APIC
..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...
IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
..TIMER: works OK on apic pin0 irq0
number of MP IRQ sources: 15.
number of IO-APIC #2 registers: 24.
testing the IO APIC.......................
IO APIC #2......
.... register #00: 02000000
.......    : physical APIC id: 02
.......    : Delivery Type: 0
.......    : LTS          : 0
.... register #01: 00170011
.......     : max redirection entries: 0017
.......     : PRQ implemented: 0
.......     : IO APIC version: 0011
.... register #02: 00000000
.......     : arbitration: 00
.... IRQ redirection table:
 NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
 00 001 01  0    0    0   0   0    1    1    31
 01 001 01  0    0    0   0   0    1    1    39
 02 000 00  0    0    0   0   0    0    0    00
 03 001 01  0    0    0   0   0    1    1    41
 04 001 01  0    0    0   0   0    1    1    49
 05 001 01  0    0    0   0   0    1    1    51
 06 001 01  0    0    0   0   0    1    1    59
 07 001 01  0    0    0   0   0    1    1    61
 08 001 01  0    0    0   0   0    1    1    69
 09 001 01  1    1    0   0   0    1    1    71
 0a 001 01  0    0    0   0   0    1    1    79
 0b 001 01  0    0    0   0   0    1    1    81
 0c 001 01  0    0    0   0   0    1    1    89
 0d 001 01  0    0    0   0   0    1    1    91
 0e 001 01  0    0    0   0   0    1    1    99
 0f 001 01  0    0    0   0   0    1    1    A1
 10 000 00  1    0    0   0   0    0    0    00
 11 000 00  1    0    0   0   0    0    0    00
 12 000 00  1    0    0   0   0    0    0    00
 13 000 00  1    0    0   0   0    0    0    00
 14 000 00  1    0    0   0   0    0    0    00
 15 000 00  1    0    0   0   0    0    0    00
 16 000 00  1    0    0   0   0    0    0    00
 17 000 00  1    0    0   0   0    0    0    00
IRQ to pin mappings:
IRQ0 -> 0:2-> 0:0
IRQ1 -> 0:1
IRQ3 -> 0:3
IRQ4 -> 0:4
IRQ5 -> 0:5
IRQ6 -> 0:6
IRQ7 -> 0:7
IRQ8 -> 0:8
IRQ9 -> 0:9
IRQ10 -> 0:10
IRQ11 -> 0:11
IRQ12 -> 0:12
IRQ13 -> 0:13
IRQ14 -> 0:14
IRQ15 -> 0:15
.................................... done.
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 1912.0861 MHz.
..... host bus clock speed is 332.0671 MHz.
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfb590, last bus=2
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
ACPI: Subsystem revision 20031002
 tbxface-0117 [03] acpi_load_tables      : ACPI Tables successfully acquired
Parsing all Control Methods:........................................................................................................................................................................................................................................................................................
Table [DSDT](id F004) - 761 Objects with 78 Devices 280 Methods 30 Regions
ACPI Namespace successfully loaded at root c0378d3c
IOAPIC[0]: Set PCI routing entry (2-9 -> 0x71 -> IRQ 9 Mode:1 Active:0)
evxfevnt-0093 [04] acpi_enable           : Transition to ACPI mode successful
evgpeblk-0748 [06] ev_create_gpe_block   : GPE 00 to 31 [_GPE] 4 regs at 0000000000004020 on int 9
evgpeblk-0748 [06] ev_create_gpe_block   : GPE 32 to 95 [_GPE] 8 regs at 00000000000044A0 on int 9
Completing Region/Field/Buffer/Package initialization:.................................................................................................
Initialized 30/30 Regions 9/9 Fields 31/31 Buffers 27/27 Packages (769 nodes)
Executing all Device _STA and_INI methods:...............................................................................
79 Devices found containing: 79 _STA, 2 _INI methods
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (00:00)
PCI: Probing PCI hardware (bus 00)
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.HUB0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGPB._PRT]
ACPI: PCI Interrupt Link [LNK1] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNK2] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNK3] (IRQs 3 4 5 6 7 10 11 *12 14 15)
ACPI: PCI Interrupt Link [LNK4] (IRQs 3 4 *5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNK5] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LUBA] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LUBB] (IRQs 3 4 *5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LMAC] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LAPU] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LACI] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LMCI] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LSMB] (IRQs 3 4 5 6 7 10 11 *12 14 15)
ACPI: PCI Interrupt Link [LUB2] (IRQs 3 4 5 6 7 10 11 *12 14 15)
ACPI: PCI Interrupt Link [LFIR] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [L3CM] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LIDE] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [APC1] (IRQs 16)
ACPI: PCI Interrupt Link [APC2] (IRQs 17)
ACPI: PCI Interrupt Link [APC3] (IRQs *18)
ACPI: PCI Interrupt Link [APC4] (IRQs *19)
ACPI: PCI Interrupt Link [APC5] (IRQs 16)
pci_link-0262 [40] acpi_pci_link_get_curr: No IRQ resource found
ACPI: PCI Interrupt Link [APCF] (IRQs 20 21 22)
pci_link-0262 [42] acpi_pci_link_get_curr: No IRQ resource found
ACPI: PCI Interrupt Link [APCG] (IRQs 20 21 22)
pci_link-0262 [44] acpi_pci_link_get_curr: No IRQ resource found
ACPI: PCI Interrupt Link [APCH] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCI] (IRQs 20 21 22)
pci_link-0262 [47] acpi_pci_link_get_curr: No IRQ resource found
ACPI: PCI Interrupt Link [APCJ] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCK] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCS] (IRQs *23)
pci_link-0262 [52] acpi_pci_link_get_curr: No IRQ resource found
ACPI: PCI Interrupt Link [APCL] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCM] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [AP3C] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCZ] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCS] enabled at IRQ 23
IOAPIC[0]: Set PCI routing entry (2-23 -> 0xa9 -> IRQ 23 Mode:1 Active:0)
00:00:01[A] -> 2-23 -> IRQ 23
Pin 2-23 already programmed
ACPI: PCI Interrupt Link [APCF] enabled at IRQ 20
IOAPIC[0]: Set PCI routing entry (2-20 -> 0xb1 -> IRQ 20 Mode:1 Active:0)
00:00:02[A] -> 2-20 -> IRQ 20
ACPI: PCI Interrupt Link [APCG] enabled at IRQ 22
IOAPIC[0]: Set PCI routing entry (2-22 -> 0xb9 -> IRQ 22 Mode:1 Active:0)
00:00:02[B] -> 2-22 -> IRQ 22
ACPI: PCI Interrupt Link [APCL] enabled at IRQ 21
IOAPIC[0]: Set PCI routing entry (2-21 -> 0xc1 -> IRQ 21 Mode:1 Active:0)
00:00:02[C] -> 2-21 -> IRQ 21
ACPI: PCI Interrupt Link [APCH] enabled at IRQ 20
Pin 2-20 already programmed
ACPI: PCI Interrupt Link [APCI] enabled at IRQ 22
Pin 2-22 already programmed
ACPI: PCI Interrupt Link [APCJ] enabled at IRQ 21
Pin 2-21 already programmed
ACPI: PCI Interrupt Link [APCK] enabled at IRQ 20
Pin 2-20 already programmed
ACPI: PCI Interrupt Link [APCM] enabled at IRQ 22
Pin 2-22 already programmed
ACPI: PCI Interrupt Link [APCZ] enabled at IRQ 21
Pin 2-21 already programmed
ACPI: PCI Interrupt Link [APC1] enabled at IRQ 16
IOAPIC[0]: Set PCI routing entry (2-16 -> 0xc9 -> IRQ 16 Mode:1 Active:0)
00:01:08[A] -> 2-16 -> IRQ 16
ACPI: PCI Interrupt Link [APC2] enabled at IRQ 17
IOAPIC[0]: Set PCI routing entry (2-17 -> 0xd1 -> IRQ 17 Mode:1 Active:0)
00:01:08[B] -> 2-17 -> IRQ 17
ACPI: PCI Interrupt Link [APC3] enabled at IRQ 18
IOAPIC[0]: Set PCI routing entry (2-18 -> 0xd9 -> IRQ 18 Mode:1 Active:0)
00:01:08[C] -> 2-18 -> IRQ 18
ACPI: PCI Interrupt Link [APC4] enabled at IRQ 19
IOAPIC[0]: Set PCI routing entry (2-19 -> 0xe1 -> IRQ 19 Mode:1 Active:0)
00:01:08[D] -> 2-19 -> IRQ 19
Pin 2-17 already programmed
Pin 2-18 already programmed
Pin 2-19 already programmed
Pin 2-16 already programmed
Pin 2-18 already programmed
Pin 2-19 already programmed
Pin 2-16 already programmed
Pin 2-17 already programmed
Pin 2-19 already programmed
Pin 2-16 already programmed
Pin 2-17 already programmed
Pin 2-18 already programmed
Pin 2-16 already programmed
Pin 2-17 already programmed
Pin 2-18 already programmed
Pin 2-19 already programmed
Pin 2-19 already programmed
PCI: Using ACPI for IRQ routing
PCI: if you experience problems, try using option 'pci=noacpi' or even 'acpi=off'
Machine check exception polling timer started.
ACPI: Power Button (FF) [PWRF]
ACPI: Sleep Button (CM) [SLPB]
ACPI: Fan [FAN] (on)
ACPI: Processor [CPU0] (supports C1)
ACPI: Thermal Zone [THRM] (38 C)
pty: 256 Unix98 ptys configured
Real Time Clock Driver v1.12
Serial: 8250/16550 driver $Revision: 1.90 $ 8 ports, IRQ sharing disabled
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Using anticipatory io scheduler
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
loop: loaded (max 8 devices)
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
NFORCE2: IDE controller at PCI slot 0000:00:09.0
NFORCE2: chipset revision 162
NFORCE2: not 100% native mode: will probe irqs later
NFORCE2: BIOS didn't set cable bits correctly. Enabling workaround.
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
NFORCE2: 0000:00:09.0 (rev a2) UDMA133 controller
    ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA
    ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:DMA
hda: WDC WD200BB-00DEA0, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hdc: MATSHITA CR-585, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 39102336 sectors (20020 MB) w/2048KiB Cache, CHS=38792/16/63, UDMA(100)
 hda: hda1 hda2 hda3
hdc: ATAPI 24X CD-ROM drive, 128kB Cache, DMA
Uniform CD-ROM driver Revision: 3.12
mice: PS/2 mouse device common for all mice
input: PC Speaker
serio: i8042 AUX port at 0x60,0x64 irq 12
input: AT Translated Set 2 keyboard on isa0060/serio0
serio: i8042 KBD port at 0x60,0x64 irq 1
NET: Registered protocol family 2
IP: routing cache hash table of 2048 buckets, 16Kbytes
TCP: Hash tables configured (established 16384 bind 32768)
NET: Registered protocol family 1
NET: Registered protocol family 17
found reiserfs format "3.6" with standard journal
Reiserfs journal params: device hda1, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
reiserfs: checking transaction log (hda1) for (hda1)
Using r5 hash to sort names
VFS: Mounted root (reiserfs filesystem) readonly.
Freeing unused kernel memory: 128k freed
Adding 377516k swap on /dev/hda2.  Priority:-1 extents:1
found reiserfs format "3.6" with standard journal
Reiserfs journal params: device hda3, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
reiserfs: checking transaction log (hda3) for (hda3)
Using r5 hash to sort names
Linux agpgart interface v0.100 (c) Dave Jones
8139too Fast Ethernet driver 0.9.27
eth0: RealTek RTL8139 at 0xd09a1000, 00:40:c7:77:0a:d5, IRQ 18
eth0:  Identified 8139 chip type 'RTL-8139 rev K'
eth0: link up, 100Mbps, full-duplex, lpa 0x45E1
agpgart: Detected NVIDIA nForce2 chipset
agpgart: Maximum main memory to use for agp memory: 203M
agpgart: AGP aperture is 64M @ 0xe8000000
PCI: Setting latency timer of device 0000:00:06.0 to 64
intel8x0: clocking to 47451
/home/jesse/linux/drivers/usb/core/usb.c: registered new driver hub
ohci_hcd: 2003 Oct 13 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
ohci_hcd: block sizes: ed 64 td 64
ohci_hcd 0000:00:02.0: OHCI Host Controller
PCI: Setting latency timer of device 0000:00:02.0 to 64
ohci_hcd 0000:00:02.0: irq 20, pci mem d0a17000
ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 1
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 3 ports detected
ohci_hcd 0000:00:02.1: OHCI Host Controller
PCI: Setting latency timer of device 0000:00:02.1 to 64
ohci_hcd 0000:00:02.1: irq 22, pci mem d0a20000
ohci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 2
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
ehci_hcd 0000:00:02.2: EHCI Host Controller
PCI: Setting latency timer of device 0000:00:02.2 to 64
ehci_hcd 0000:00:02.2: irq 21, pci mem d0a2e000
ehci_hcd 0000:00:02.2: new USB bus registered, assigned bus number 3
PCI: cache line size of 64 is not supported by device 0000:00:02.2
ehci_hcd 0000:00:02.2: USB 2.0 enabled, EHCI 1.00, driver 2003-Jun-13
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 6 ports detected
parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
parport0: cpp_daisy: aa5500ff(38)
parport0: assign_addrs: aa5500ff(38)
parport0: cpp_daisy: aa5500ff(38)
parport0: assign_addrs: aa5500ff(38)
i2c_adapter i2c-0: nForce2 SMBus adapter at 0x5000
i2c_adapter i2c-1: nForce2 SMBus adapter at 0x5100

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Lenar Lõhmus]

mptable for epox 8rda+
looks weird considering OEM&Product ID:

===============================================================================

MPTable, version 2.0.15 Linux

 looking for EBDA pointer @ 0x040e, NOT found
 searching CMOS 'top of mem' @ 0x0009f400 (637K)
 searching default 'top of mem' @ 0x0009fc00 (639K)
 searching BIOS @ 0x000f0000

 MP FPS found in BIOS @ physical addr: 0x000f5ac0

-------------------------------------------------------------------------------

MP Floating Pointer Structure:

  location:                     BIOS
  physical address:             0x000f5ac0
  signature:                    '_MP_'
  length:                       16 bytes
  version:                      1.1
  checksum:                     0x00
  mode:                         Virtual Wire

-------------------------------------------------------------------------------

MP Config Table Header:

  physical address:             0x0xf0c00
  signature:                    'M'
  base table length:            49091
  version:                      1.14
  checksum:                     0x0c
  OEM ID:                       '3ÉèoÃ
                                      '
  Product ID:                   'ÿÿPOWER ON F'
  OEM table pointer:            0x74636e75
  OEM table size:               28521
  entry count:                  8302
  local APIC address:           0x20202020
  extended table length:        8224
  extended table checksum:      32

-------------------------------------------------------------------------------

MP Config Base Table Entries:

--
MPTABLE HOSED! record type = 32

Lenar

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Maciej W. Rozycki]

On Thu, 11 Dec 2003, Josh McKinney wrote:

> Should I try it with just the acpi fixes sent by Maciej or are these
> just general fixes?

 They should make (at least some of) the reported problems go away,
superseding the respective workarounds.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Josh McKinney]

Trying to get a grasp on the all the fixes floating around.  I have
been running the first "timer" patch, the two liner to mpparse.c, for
about five days until I made it crash with by catting 4 drives to
/dev/null.  It crashed after I turned on disconnect with athcool, so
that may be related, because I could crash it with disconnect off.
Now I am running both of Ross's patches for 2.6 for just 10 hours, but
disconnect is still enabled, so far so good.  

So the consensus seems to be that Ross's timer patch and the
disconnect OR delay ACK patch is the mostly *correct* fix?  As of
right now I am compiling kernels with the disconnect patch and ross's
timer patch, and one with those fixes and Maciej's acpi fixes below.
Should I try it with just the acpi fixes sent by Maciej or are these
just general fixes?

I also tried running mptable, but the output is "hosed".

Thanks
    
On approximately Thu, Dec 11, 2003 at 04:15:28PM +0100, Maciej W. Rozycki wrote:
> On Thu, 11 Dec 2003, Ross Dickson wrote:
> 
> > ACPI: IOAPIC (id[0x02] address[0xfec00000] global_irq_base[0x0])
> > IOAPIC[0]: Assigned apic_id 2
> > IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, IRQ 0-23
> > Bus #0 is ISA
> > Int: type 3, pol 0, trig 0, bus 0, irq 0, 2-0
> 
>  I've browsed the relevant part of the ACPI spec and the above entry is 
> incorrect.  It looks like INTIN0 is now the preferred line for the 8254 
> timer; at least it is the default one when using ACPI tables.  This is a 
> bug in Linux.
> 
> > ACPI: INT_SRC_OVR (bus[0] irq[0x0] global_irq[0x2] polarity[0x0] trigger[0x0])
> > Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-2
> 
>  Now this is an explicit entry stating the 8254 timer is connected to
> INTIN2.  If this is not the case, the BIOS is buggy and the solution is to
> fix it.  I don't consider it possible to be worked around in Linux except
> maybe with a command line option added manually.
> 
> > ACPI: INT_SRC_OVR (bus[0] irq[0x9] global_irq[0x9] polarity[0x1] trigger[0x3])
> > Int: type 0, pol 1, trig 3, bus 0, irq 9, 2-9
> 
>  And yet another explicit entry which has an effect on configuration as
> reported below.
> 
> > init IO_APIC IRQs
> >  IO-APIC (apicid-pin) 2-0, 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
> > ..TIMER: vector=0x31 pin1=2 pin2=-1
> > ..MP-BIOS bug: 8254 timer not connected to IO-APIC pin2
> 
>  As reported above, the BIOS explicitly reports the timer is there.
> 
> > ..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...
> > IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
> > ..TIMER check 8259 ints disabled, imr1:ff, imr2:ff
> > ..TIMER: works OK on apic pin0 irq0
> 
>  And this may be correct if the default ACPI settings reflect the actual 
> wiring of this board (but the BIOS says otherwise).
> 
> > IRQ to pin mappings:
> > IRQ0 -> 0:2-> 0:0
> [...]
> > IRQ9 -> 0:9-> 0:9
> 
>  These two entries are wrong -- the interrupts are set up as if they were
> connected to multiple I/O APIC inputs.  The first entry is a result of 
> your hack, but the second one suggests a bug somewhere.
> 
> > Finally others working with kern 2.6  earlier trialled the following patch which may provide some
> > more clues: 
> > retrieved from:
> > 
> > http://www.kernel.org/pub/linux/kernel/people/bart/2.6.0-test11-bart1/broken-out/nforce2-apic.patch 
> >  
> > [x86] do not wrongly override mp_ExtINT IRQ
> 
>  That's a workaround to the bug in Linux I've mentioned earlier.  The bug
> should be fixed instead.  The ACPI spec doesn't support mixed 
> configurations, so ExtINT is irrelevant.
> 
> > Perhaps someone else could get mptable to run on their machine and send you
> > the result.
> 
>  I wanted it to compare with the ACPI table and possibly to treat as a
> reference for a workaround.  Since you have no valid MP-table, there's
> nothing to do.
> 
>  Here's a patch that fixes a few bugs I've spotted browsing through our
> ACPI code.  Please try it and report the result.  I don't have a system
> with ACPI available, so I cannot verify the changes at all.
> 
>  The same bugs are present in 2.4 and I have a corresponding patch
> available if some wants to test the changes with that version.
> 
>   Maciej
> 
> -- 
> +  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
> +--------------------------------------------------------------+
> +        e-mail: macro@ds2.pg.gda.pl, PGP key available        +
> 
> patch-mips-2.6.0-test11-20031209-acpi-irq0-1
> diff -up --recursive --new-file linux-mips-2.6.0-test11-20031209.macro/arch/i386/kernel/mpparse.c linux-mips-2.6.0-test11-20031209/arch/i386/kernel/mpparse.c
> --- linux-mips-2.6.0-test11-20031209.macro/arch/i386/kernel/mpparse.c	2003-11-25 04:57:01.000000000 +0000
> +++ linux-mips-2.6.0-test11-20031209/arch/i386/kernel/mpparse.c	2003-12-11 09:43:26.000000000 +0000
> @@ -940,7 +940,7 @@ void __init mp_override_legacy_irq (
>  	 *      erroneously sets the trigger to level, resulting in a HUGE 
>  	 *      increase of timer interrupts!
>  	 */
> -	if ((bus_irq == 0) && (global_irq == 2) && (trigger == 3))
> +	if ((bus_irq == 0) && (trigger == 3))
>  		trigger = 1;
>  
>  	intsrc.mpc_type = MP_INTSRC;
> @@ -961,7 +961,7 @@ void __init mp_override_legacy_irq (
>  	 * Otherwise create a new entry (e.g. global_irq == 2).
>  	 */
>  	for (i = 0; i < mp_irq_entries; i++) {
> -		if ((mp_irqs[i].mpc_dstapic == intsrc.mpc_dstapic) 
> +		if ((mp_irqs[i].mpc_srcbus == intsrc.mpc_srcbus) 
>  			&& (mp_irqs[i].mpc_srcbusirq == intsrc.mpc_srcbusirq)) {
>  			mp_irqs[i] = intsrc;
>  			found = 1;
> @@ -1008,9 +1008,10 @@ void __init mp_config_acpi_legacy_irqs (
>  	 */
>  	for (i = 0; i < 16; i++) {
>  
> -		if (i == 2) continue;			/* Don't connect IRQ2 */
> +		if (i == 2)
> +			continue;			/* Don't connect IRQ2 */
>  
> -		intsrc.mpc_irqtype = i ? mp_INT : mp_ExtINT;   /* 8259A to #0 */
> +		intsrc.mpc_irqtype = mp_INT;
>  		intsrc.mpc_srcbusirq = i;		   /* Identity mapped */
>  		intsrc.mpc_dstirq = i;
>  
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Josh McKinney		     |	Webmaster: http://joshandangie.org
--------------------------------------------------------------------------
                             | They that can give up essential liberty
Linux, the choice       -o)  | to obtain a little temporary safety deserve 
of the GNU generation    /\  | neither liberty or safety. 
                        _\_v |                          -Benjamin Franklin

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Jesse Allen]

On Thu, Dec 11, 2003 at 04:20:58PM +0100, Craig Bradney wrote:
> Not really sure what I'm looking at here but as you guys are showing
> this information I thought it might be helpful for those that can use it
> to have the information run on a Asus A7N8X Deluxe (v2.0 bios 1007) with
> Athlon XP 2600+. 
> 

Unfortunately, it looks as all our MP tables are invalid.  So I don't think we can use them.  I thought mine was especailly weird because of the Product ID seems to be pointing to a "Press Any Key" string which proves that.

Jesse

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Craig Bradney]

Not really sure what I'm looking at here but as you guys are showing
this information I thought it might be helpful for those that can use it
to have the information run on a Asus A7N8X Deluxe (v2.0 bios 1007) with
Athlon XP 2600+. 

===============================================================================

MPTable, version 2.0.15 Linux

-------------------------------------------------------------------------------

MP Floating Pointer Structure:

  location:                     BIOS
  physical address:             0x000f5ce0
  signature:                    '_MP_'
  length:                       16 bytes
  version:                      1.1
  checksum:                     0x00
  mode:                         Virtual Wire

-------------------------------------------------------------------------------

MP Config Table Header:

  physical address:             0x0xf0c00
  signature:                    '
'
  base table length:            65287
  version:                      1.255
  checksum:                     0x04
  OEM ID:                       ''
  Product ID:                   ''
  OEM table pointer:            0x00000704
  OEM table size:               15
  entry count:                  3896
  local APIC address:           0x00070500
  extended table length:        3584
  extended table checksum:      0

-------------------------------------------------------------------------------

MP Config Base Table Entries:

--
Processors:     APIC ID Version State           Family  Model   Step   
Flags
                13       0x 0    AP, usable      3       0       0      
0xff070600
                 0       0xff    BSP, usable     0       12      4      
0x0001
--
MPTABLE HOSED! record type = 53



Craig


On Thu, 2003-12-11 at 15:58, Jesse Allen wrote:
> My mptable output looks pretty weird.  (Product ID "ny Key "?)
> It doesn't even compare to the other two.  I have a shuttle AN35N.
> 
> 
> ===============================================================================
> 
> MPTable, version 2.0.15 Linux
> 
> -------------------------------------------------------------------------------
> 
> MP Floating Pointer Structure:
> 
>   location:			BIOS
>   physical address:		0x000f5650
>   signature:			'_MP_'
>   length:			16 bytes
>   version:			1.1
>   checksum:			0x00
>   mode:				Virtual Wire
> 
> -------------------------------------------------------------------------------
> 
> MP Config Table Header:
> 
>   physical address:		0x0xf0c00
>   signature:			'N   '
>   base table length:		8224
>   version:			1.32
>   checksum:			0x20
>   OEM ID:			'    : '
>   Product ID:			'ny Key '
>   OEM table pointer:		0x2031462d
>   OEM table size:		17152
>   entry count:			29300
>   local APIC address:		0x32462d6c
>   extended table length:	32
>   extended table checksum:	67
> 
> -------------------------------------------------------------------------------
> 
> MP Config Base Table Entries:
> 
> --
> MPTABLE HOSED! record type = 114
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Maciej W. Rozycki]

On Thu, 11 Dec 2003, Ross Dickson wrote:

> ACPI: IOAPIC (id[0x02] address[0xfec00000] global_irq_base[0x0])
> IOAPIC[0]: Assigned apic_id 2
> IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, IRQ 0-23
> Bus #0 is ISA
> Int: type 3, pol 0, trig 0, bus 0, irq 0, 2-0

 I've browsed the relevant part of the ACPI spec and the above entry is 
incorrect.  It looks like INTIN0 is now the preferred line for the 8254 
timer; at least it is the default one when using ACPI tables.  This is a 
bug in Linux.

> ACPI: INT_SRC_OVR (bus[0] irq[0x0] global_irq[0x2] polarity[0x0] trigger[0x0])
> Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-2

 Now this is an explicit entry stating the 8254 timer is connected to
INTIN2.  If this is not the case, the BIOS is buggy and the solution is to
fix it.  I don't consider it possible to be worked around in Linux except
maybe with a command line option added manually.

> ACPI: INT_SRC_OVR (bus[0] irq[0x9] global_irq[0x9] polarity[0x1] trigger[0x3])
> Int: type 0, pol 1, trig 3, bus 0, irq 9, 2-9

 And yet another explicit entry which has an effect on configuration as
reported below.

> init IO_APIC IRQs
>  IO-APIC (apicid-pin) 2-0, 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
> ..TIMER: vector=0x31 pin1=2 pin2=-1
> ..MP-BIOS bug: 8254 timer not connected to IO-APIC pin2

 As reported above, the BIOS explicitly reports the timer is there.

> ..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...
> IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
> ..TIMER check 8259 ints disabled, imr1:ff, imr2:ff
> ..TIMER: works OK on apic pin0 irq0

 And this may be correct if the default ACPI settings reflect the actual 
wiring of this board (but the BIOS says otherwise).

> IRQ to pin mappings:
> IRQ0 -> 0:2-> 0:0
[...]
> IRQ9 -> 0:9-> 0:9

 These two entries are wrong -- the interrupts are set up as if they were
connected to multiple I/O APIC inputs.  The first entry is a result of 
your hack, but the second one suggests a bug somewhere.

> Finally others working with kern 2.6  earlier trialled the following patch which may provide some
> more clues: 
> retrieved from:
> 
> http://www.kernel.org/pub/linux/kernel/people/bart/2.6.0-test11-bart1/broken-out/nforce2-apic.patch 
>  
> [x86] do not wrongly override mp_ExtINT IRQ

 That's a workaround to the bug in Linux I've mentioned earlier.  The bug
should be fixed instead.  The ACPI spec doesn't support mixed 
configurations, so ExtINT is irrelevant.

> Perhaps someone else could get mptable to run on their machine and send you
> the result.

 I wanted it to compare with the ACPI table and possibly to treat as a
reference for a workaround.  Since you have no valid MP-table, there's
nothing to do.

 Here's a patch that fixes a few bugs I've spotted browsing through our
ACPI code.  Please try it and report the result.  I don't have a system
with ACPI available, so I cannot verify the changes at all.

 The same bugs are present in 2.4 and I have a corresponding patch
available if some wants to test the changes with that version.

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

patch-mips-2.6.0-test11-20031209-acpi-irq0-1
diff -up --recursive --new-file linux-mips-2.6.0-test11-20031209.macro/arch/i386/kernel/mpparse.c linux-mips-2.6.0-test11-20031209/arch/i386/kernel/mpparse.c
--- linux-mips-2.6.0-test11-20031209.macro/arch/i386/kernel/mpparse.c	2003-11-25 04:57:01.000000000 +0000
+++ linux-mips-2.6.0-test11-20031209/arch/i386/kernel/mpparse.c	2003-12-11 09:43:26.000000000 +0000
@@ -940,7 +940,7 @@ void __init mp_override_legacy_irq (
 	 *      erroneously sets the trigger to level, resulting in a HUGE 
 	 *      increase of timer interrupts!
 	 */
-	if ((bus_irq == 0) && (global_irq == 2) && (trigger == 3))
+	if ((bus_irq == 0) && (trigger == 3))
 		trigger = 1;
 
 	intsrc.mpc_type = MP_INTSRC;
@@ -961,7 +961,7 @@ void __init mp_override_legacy_irq (
 	 * Otherwise create a new entry (e.g. global_irq == 2).
 	 */
 	for (i = 0; i < mp_irq_entries; i++) {
-		if ((mp_irqs[i].mpc_dstapic == intsrc.mpc_dstapic) 
+		if ((mp_irqs[i].mpc_srcbus == intsrc.mpc_srcbus) 
 			&& (mp_irqs[i].mpc_srcbusirq == intsrc.mpc_srcbusirq)) {
 			mp_irqs[i] = intsrc;
 			found = 1;
@@ -1008,9 +1008,10 @@ void __init mp_config_acpi_legacy_irqs (
 	 */
 	for (i = 0; i < 16; i++) {
 
-		if (i == 2) continue;			/* Don't connect IRQ2 */
+		if (i == 2)
+			continue;			/* Don't connect IRQ2 */
 
-		intsrc.mpc_irqtype = i ? mp_INT : mp_ExtINT;   /* 8259A to #0 */
+		intsrc.mpc_irqtype = mp_INT;
 		intsrc.mpc_srcbusirq = i;		   /* Identity mapped */
 		intsrc.mpc_dstirq = i;
 

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Jesse Allen]

My mptable output looks pretty weird.  (Product ID "ny Key "?)
It doesn't even compare to the other two.  I have a shuttle AN35N.


===============================================================================

MPTable, version 2.0.15 Linux

-------------------------------------------------------------------------------

MP Floating Pointer Structure:

  location:			BIOS
  physical address:		0x000f5650
  signature:			'_MP_'
  length:			16 bytes
  version:			1.1
  checksum:			0x00
  mode:				Virtual Wire

-------------------------------------------------------------------------------

MP Config Table Header:

  physical address:		0x0xf0c00
  signature:			'N   '
  base table length:		8224
  version:			1.32
  checksum:			0x20
  OEM ID:			'    : '
  Product ID:			'ny Key '
  OEM table pointer:		0x2031462d
  OEM table size:		17152
  entry count:			29300
  local APIC address:		0x32462d6c
  extended table length:	32
  extended table checksum:	67

-------------------------------------------------------------------------------

MP Config Base Table Entries:

--
MPTABLE HOSED! record type = 114

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Jesse Allen]

On Wed, Dec 10, 2003 at 11:00:39AM +0100, Mikael Pettersson wrote:
> Please try without this delay but with the disconnect PCI quirk.
> 


OK,  I have tried it without the delay, and with Ross' timer patch.  It will obviously lockup, and nmi_watchdog doesn't work.  Added the disconnect quirk patch, and lockups are gone and nmi_watchdog works.  So there is no difference between the disconnect patch or the ACK delay patch.  Though I found nmi_watchdog does depend on having either the disconnect patch or the delay patch (not an io_apic patch).  You think the disconnect patch is better?  In any event, they both indicate a behavior, and there maybe a better solution to all of it.

Jesse

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 4023 bytes --]

On Thu, 2003-12-11 at 07:55, Ross Dickson wrote:
> albatron:/usr/src/mptable-2.0.15a # ./mptable -verbose
> 
> ===============================================================================
> 
> MPTable, version 2.0.15 Linux
> 
>  looking for EBDA pointer @ 0x040e, found, searching EBDA @ 0x0009fc00
>  searching CMOS 'top of mem' @ 0x0009f800 (638K)
>  searching default 'top of mem' @ 0x0009fc00 (639K)
>  searching BIOS @ 0x000f0000
> 
>  MP FPS found in BIOS @ physical addr: 0x000f50b0
> 
> -------------------------------------------------------------------------------
> 
> MP Floating Pointer Structure:
> 
>   location:                     BIOS
>   physical address:             0x000f50b0
>   signature:                    '_MP_'
>   length:                       16 bytes
>   version:                      1.1
>   checksum:                     0x00
>   mode:                         Virtual Wire
> 
> -------------------------------------------------------------------------------
> 
> MP Config Table Header:
> 
>   physical address:             0x0xf0c00
>   signature:                    '$ml$'
>   base table length:            0
>   version:                      1.6
>   checksum:                     0x00
>   OEM ID:                       'Ä
>                                   ¸§'
> °öProduct ID:                   '(
> m'P
>   OEM table pointer:            0x12d90e22
>   OEM table size:               7964
>   entry count:                  7964
>   local APIC address:           0x1f1c1f1c
>   extended table length:        65284
>   extended table checksum:      255
> 
> -------------------------------------------------------------------------------
> 
> MP Config Base Table Entries:
> 
> --
> MPTABLE HOSED! record type = 55
> albatron:/usr/src/mptable-2.0.15a #
> 

> Perhaps someone else could get mptable to run on their machine and send you
> the result.

mptable dosn't seem to accept it's own options, anyways, heres the
output.

mptable -extra -verbose -pirq
 
===============================================================================
 
MPTable, version 2.0.15 Linux
 
 looking for EBDA pointer @ 0x040e, found, searching EBDA @ 0x0009fc00
 searching CMOS 'top of mem' @ 0x0009f800 (638K)
 searching default 'top of mem' @ 0x0009fc00 (639K)
 searching BIOS @ 0x000f0000
 
 MP FPS found in BIOS @ physical addr: 0x000f5ce0
 
-------------------------------------------------------------------------------
 
MP Floating Pointer Structure:
 
  location:                     BIOS
  physical address:             0x000f5ce0
  signature:                    '_MP_'
  length:                       16 bytes
  version:                      1.1
  checksum:                     0x00
  mode:                         Virtual Wire
 
-------------------------------------------------------------------------------
 
MP Config Table Header:
 
  physical address:             0x0xf0c00
  signature:                    ''
  base table length:            1280
  version:                      1.7
  checksum:                     0x00
  OEM ID:                       ''
  Product ID:                   ''
  OEM table pointer:            0x0000ffff
  OEM table size:               0
  entry count:                  65535
  local APIC address:           0x000000c4
  extended table length:        1
  extended table checksum:      0
 
-------------------------------------------------------------------------------
 
MP Config Base Table Entries:
 
--
Processors:     APIC ID Version State           Family  Model   Step    Flags
                 0       0x 7    BSP, usable     15      15      15      0x1a00c035
                 0       0x 0    AP, unusable    0       0       10      0x78ffff0a
--
MPTABLE HOSED! record type = 15

I couldn't find the source so i used a old RedHat rpm...
(Asus A7N8X-X bios 1007)
 
-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

On Thursday 11 December 2003 21:47, Ian Kumlien wrote:
> On Thu, 2003-12-11 at 07:55, Ross Dickson wrote:
> > albatron:/usr/src/mptable-2.0.15a # ./mptable -verbose
> > 
> > ===============================================================================
> > 
> > MPTable, version 2.0.15 Linux
> > 
> >  looking for EBDA pointer @ 0x040e, found, searching EBDA @ 0x0009fc00
> >  searching CMOS 'top of mem' @ 0x0009f800 (638K)
> >  searching default 'top of mem' @ 0x0009fc00 (639K)
> >  searching BIOS @ 0x000f0000
> > 
> >  MP FPS found in BIOS @ physical addr: 0x000f50b0
> > 
> > -------------------------------------------------------------------------------
> > 
> > MP Floating Pointer Structure:
> > 
> >   location:                     BIOS
> >   physical address:             0x000f50b0
> >   signature:                    '_MP_'
> >   length:                       16 bytes
> >   version:                      1.1
> >   checksum:                     0x00
> >   mode:                         Virtual Wire
> > 
> > -------------------------------------------------------------------------------
> > 
> > MP Config Table Header:
> > 
> >   physical address:             0x0xf0c00
> >   signature:                    '$ml$'
> >   base table length:            0
> >   version:                      1.6
> >   checksum:                     0x00
> >   OEM ID:                       'Ä
> >                                   ¸§'
> > °öProduct ID:                   '(
> > m'P
> >   OEM table pointer:            0x12d90e22
> >   OEM table size:               7964
> >   entry count:                  7964
> >   local APIC address:           0x1f1c1f1c
> >   extended table length:        65284
> >   extended table checksum:      255
> > 
> > -------------------------------------------------------------------------------
> > 
> > MP Config Base Table Entries:
> > 
> > --
> > MPTABLE HOSED! record type = 55
> > albatron:/usr/src/mptable-2.0.15a #
> > 
> 
> > Perhaps someone else could get mptable to run on their machine and send you
> > the result.
> 
> mptable dosn't seem to accept it's own options, anyways, heres the
> output.
> 
> mptable -extra -verbose -pirq
>  
> ===============================================================================
>  
> MPTable, version 2.0.15 Linux
>  
>  looking for EBDA pointer @ 0x040e, found, searching EBDA @ 0x0009fc00
>  searching CMOS 'top of mem' @ 0x0009f800 (638K)
>  searching default 'top of mem' @ 0x0009fc00 (639K)
>  searching BIOS @ 0x000f0000
>  
>  MP FPS found in BIOS @ physical addr: 0x000f5ce0
>  
> -------------------------------------------------------------------------------
>  
> MP Floating Pointer Structure:
>  
>   location:                     BIOS
>   physical address:             0x000f5ce0
>   signature:                    '_MP_'
>   length:                       16 bytes
>   version:                      1.1
>   checksum:                     0x00
>   mode:                         Virtual Wire
>  
> -------------------------------------------------------------------------------
>  
> MP Config Table Header:
>  
>   physical address:             0x0xf0c00
>   signature:                    ''
>   base table length:            1280
>   version:                      1.7
>   checksum:                     0x00
>   OEM ID:                       ''
>   Product ID:                   ''
>   OEM table pointer:            0x0000ffff
>   OEM table size:               0
>   entry count:                  65535
>   local APIC address:           0x000000c4
>   extended table length:        1
>   extended table checksum:      0
>  
> -------------------------------------------------------------------------------
>  
> MP Config Base Table Entries:
>  
> --
> Processors:     APIC ID Version State           Family  Model   Step    Flags
>                  0       0x 7    BSP, usable     15      15      15      0x1a00c035
>                  0       0x 0    AP, unusable    0       0       10      0x78ffff0a
> --
> MPTABLE HOSED! record type = 15
> 
> I couldn't find the source so i used a old RedHat rpm...
> (Asus A7N8X-X bios 1007)
>  
> -- 
> Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
> 

Thanks Ian

Also many thanks for pointing out the relevant section to look in with the AMD
cpu link that you sent - Credit where credit is due (assuming we are both on the
right track).

I had a read and refined your surmisings. I think the 
problem appears synchronous with the apic timer because of two reasons.
1) any apic irq can cause re-connection of the system bus after disconnect.
2) the apic timer irq in my examinations has the shortest path to an ack.

I also had a look back through the athlon cooler and power management 
postings and web site articles. I was blissfully ignorant of these issues when I
started and now I wonder what I have stepped into... Yuk

I submitted a support request to AMD, apologies for not cc'ing you, I kept
the cc's down to just nvidia and the mailing list. If you have not seen it yet
then it is here

http://lkml.org/lkml/2003/12/11/17

We hope....

Regards
Ross

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

On Thursday 11 December 2003 02:06, Maciej W. Rozycki wrote:
> On Wed, 10 Dec 2003, Ross Dickson wrote:
> 
> > Relevant dmesg output from Albatron KM18G Pro ( this is different MOBO (same type) but 
> > this time has a barton core 2500 XP cpu).
> > 
> > enabled ExtINT on CPU#0
> > ESR value before enabling vector: 00000000
> > ESR value after enabling vector: 00000000
> > ENABLING IO-APIC IRQs
> > init IO_APIC IRQs
> >  IO-APIC (apicid-pin) 2-0, 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
> > ..TIMER: vector=0x31 pin1=2 pin2=-1
> > ..MP-BIOS bug: 8254 timer not connected to IO-APIC pin2
> > ..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...
> > IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
> > ..TIMER check 8259 ints disabled, imr1:ff, imr2:ff
> > ..TIMER: works OK on apic pin0 irq0
> > Using local APIC timer interrupts.
> > calibrating APIC timer ...
> > ..... CPU clock speed is 1829.0708 MHz.
> > ..... host bus clock speed is 332.0674 MHz.
> > cpu: 0, clocks: 332674, slice: 166337
> > CPU0<T0:332672,T1:166320,D:15,S:166337,C:332674>
> 
>  Hmm, while this is different from what is documented in the MP Spec, it
> looks like the 8254 IRQ is connected to INTIN0 indeed.  We can handle such
> a setup if the BIOS reports routing correctly.  Since you invoke
> io_apic_set_pci_routing() I assume you use ACPI for IRQ routing
> information.  Can you please rebuild the kernel with APIC_DEBUG set to 1
> in include/asm-i386/apic.h and send me the bootstrap log?  Can you please
> send me the output of a tool called `mptable' as well, so that I can
> compare the results?
> 
>   Maciej
> 
> -- 
> +  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
> +--------------------------------------------------------------+
> +        e-mail: macro@ds2.pg.gda.pl, PGP key available        +
> 
> 
> 
Thanks Maciej,
bootstrap log follows

CPI: RSDT (v001 Nvidia AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x1dff3000
ACPI: FADT (v001 Nvidia AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x1dff3040
ACPI: MADT (v001 Nvidia AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x1dff7980
ACPI: DSDT (v001 NVIDIA AWRDACPI 0x00001000 MSFT 0x0100000e) @ 0x00000000
ACPI: Local APIC address 0xfee00000
Boot CPU = 0
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 Pentium(tm) Pro APIC version 16
    Floating point unit present.
    Machine Exception supported.
    64 bit compare & exchange supported.
    Internal APIC present.
    SEP present.
    MTRR  present.
    PGE  present.
    MCA  present.
    CMOV  present.
    PAT  present.
    PSE  present.
    MMX  present.
    FXSR  present.
    XMM  present.
    Bootup CPU
ACPI: LAPIC_NMI (acpi_id[0x00] polarity[0x1] trigger[0x1] lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] global_irq_base[0x0])
IOAPIC[0]: Assigned apic_id 2
IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, IRQ 0-23
Bus #0 is ISA
Int: type 3, pol 0, trig 0, bus 0, irq 0, 2-0
Int: type 0, pol 0, trig 0, bus 0, irq 1, 2-1
Int: type 0, pol 0, trig 0, bus 0, irq 3, 2-3
Int: type 0, pol 0, trig 0, bus 0, irq 4, 2-4
Int: type 0, pol 0, trig 0, bus 0, irq 5, 2-5
Int: type 0, pol 0, trig 0, bus 0, irq 6, 2-6
Int: type 0, pol 0, trig 0, bus 0, irq 7, 2-7
Int: type 0, pol 0, trig 0, bus 0, irq 8, 2-8
Int: type 0, pol 0, trig 0, bus 0, irq 9, 2-9
Int: type 0, pol 0, trig 0, bus 0, irq 10, 2-10
Int: type 0, pol 0, trig 0, bus 0, irq 11, 2-11
Int: type 0, pol 0, trig 0, bus 0, irq 12, 2-12
Int: type 0, pol 0, trig 0, bus 0, irq 13, 2-13
Int: type 0, pol 0, trig 0, bus 0, irq 14, 2-14
Int: type 0, pol 0, trig 0, bus 0, irq 15, 2-15
ACPI: INT_SRC_OVR (bus[0] irq[0x0] global_irq[0x2] polarity[0x0] trigger[0x0])
Int: type 0, pol 0, trig 0, bus 0, irq 0, 2-2
ACPI: INT_SRC_OVR (bus[0] irq[0x9] global_irq[0x9] polarity[0x1] trigger[0x3])
Int: type 0, pol 1, trig 3, bus 0, irq 9, 2-9
ACPI BALANCE SET
Using ACPI (MADT) for SMP configuration information
Kernel command line: splash=silent root=/dev/hda2 hdc=ide-scsi hdclun=0
ide_setup: hdc=ide-scsi
ide_setup: hdclun=0
mapped APIC to ffffe000 (fee00000)
mapped IOAPIC to ffffd000 (fec00000)
Initializing CPU#0
Detected 1830.076 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 3620.86 BogoMIPS
Memory: 482980k/491456k available (1800k kernel code, 8088k reserved, 622k data, 112k init, 0k highmem)
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode cache hash table entries: 32768 (order: 6, 262144 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer cache hash table entries: 32768 (order: 5, 131072 bytes)
Page-cache hash table entries: 131072 (order: 7, 524288 bytes)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU:     After generic, caps: 0383fbff c1c3fbff 00000000 00000000
CPU:             Common caps: 0383fbff c1c3fbff 00000000 00000000
CPU: AMD Athlon(tm) XP 2500+ stepping 00
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
Getting VERSION: 40010
Getting VERSION: 40010
Getting ID: 0
Getting ID: f000000
Getting LVT0: 700
Getting LVT1: 400
enabled ExtINT on CPU#0
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
ENABLING IO-APIC IRQs
Synchronizing Arb IDs.
init IO_APIC IRQs
 IO-APIC (apicid-pin) 2-0, 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
..TIMER: vector=0x31 pin1=2 pin2=-1
..MP-BIOS bug: 8254 timer not connected to IO-APIC pin2
..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...
IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
..TIMER check 8259 ints disabled, imr1:ff, imr2:ff
..TIMER: works OK on apic pin0 irq0
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 1829.0813 MHz.
..... host bus clock speed is 332.0693 MHz.
cpu: 0, clocks: 332693, slice: 166346
CPU0<T0:332688,T1:166336,D:6,S:166346,C:332693>
mtrr: v1.40 (20010327) Richard Gooch (rgooch@atnf.csiro.au)
mtrr: detected mtrr type: Intel
ACPI: Subsystem revision 20031002
PCI: PCI BIOS revision 2.10 entry at 0xfb4e0, last bus=2
PCI: Using configuration type 1
IOAPIC[0]: Set PCI routing entry (2-9 -> 0x71 -> IRQ 9 Mode:1 Active:0)
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: System [ACPI] (supports S0 S1 S4 S5)
ACPI: PCI Root Bridge [PCI0] (00:00)
PCI: Probing PCI hardware (bus 00)
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.HUB0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGPB._PRT]
ACPI: PCI Interrupt Link [LNK1] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNK2] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNK3] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNK4] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNK5] (IRQs 3 4 *5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LUBA] (IRQs 3 4 *5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LUBB] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LMAC] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LAPU] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LACI] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LMCI] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LSMB] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LUB2] (IRQs 3 4 *5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LFIR] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [L3CM] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LIDE] (IRQs 3 4 5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [APC1] (IRQs 16)
ACPI: PCI Interrupt Link [APC2] (IRQs 17)
ACPI: PCI Interrupt Link [APC3] (IRQs 18)
ACPI: PCI Interrupt Link [APC4] (IRQs 19)
ACPI: PCI Interrupt Link [APC5] (IRQs *16)
ACPI: PCI Interrupt Link [APCF] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCG] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCH] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCI] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCJ] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCK] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCS] (IRQs *23)
ACPI: PCI Interrupt Link [APCL] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCM] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [AP3C] (IRQs 20 21 22)
ACPI: PCI Interrupt Link [APCZ] (IRQs 20 21 22)
PCI: Probing PCI hardware
ACPI: PCI Interrupt Link [APCS] enabled at IRQ 23
IOAPIC[0]: Set PCI routing entry (2-23 -> 0xa9 -> IRQ 23 Mode:1 Active:0)
00:00:01[A] -> 2-23 -> IRQ 23
Pin 2-23 already programmed
ACPI: PCI Interrupt Link [APCF] enabled at IRQ 20
IOAPIC[0]: Set PCI routing entry (2-20 -> 0xb1 -> IRQ 20 Mode:1 Active:0)
00:00:02[A] -> 2-20 -> IRQ 20
ACPI: PCI Interrupt Link [APCG] enabled at IRQ 22
IOAPIC[0]: Set PCI routing entry (2-22 -> 0xb9 -> IRQ 22 Mode:1 Active:0)
00:00:02[B] -> 2-22 -> IRQ 22
ACPI: PCI Interrupt Link [APCL] enabled at IRQ 21
IOAPIC[0]: Set PCI routing entry (2-21 -> 0xc1 -> IRQ 21 Mode:1 Active:0)
00:00:02[C] -> 2-21 -> IRQ 21
ACPI: PCI Interrupt Link [APCH] enabled at IRQ 20
Pin 2-20 already programmed
ACPI: PCI Interrupt Link [APCI] enabled at IRQ 22
Pin 2-22 already programmed
ACPI: PCI Interrupt Link [APCJ] enabled at IRQ 21
Pin 2-21 already programmed
ACPI: PCI Interrupt Link [APCK] enabled at IRQ 20
Pin 2-20 already programmed
ACPI: PCI Interrupt Link [APCM] enabled at IRQ 22
Pin 2-22 already programmed
ACPI: PCI Interrupt Link [APCZ] enabled at IRQ 21
Pin 2-21 already programmed
ACPI: PCI Interrupt Link [APC3] enabled at IRQ 18
IOAPIC[0]: Set PCI routing entry (2-18 -> 0xc9 -> IRQ 18 Mode:1 Active:0)
00:01:06[A] -> 2-18 -> IRQ 18
ACPI: PCI Interrupt Link [APC4] enabled at IRQ 19
IOAPIC[0]: Set PCI routing entry (2-19 -> 0xd1 -> IRQ 19 Mode:1 Active:0)
00:01:06[B] -> 2-19 -> IRQ 19
ACPI: PCI Interrupt Link [APC1] enabled at IRQ 16
IOAPIC[0]: Set PCI routing entry (2-16 -> 0xd9 -> IRQ 16 Mode:1 Active:0)
00:01:06[C] -> 2-16 -> IRQ 16
ACPI: PCI Interrupt Link [APC2] enabled at IRQ 17
IOAPIC[0]: Set PCI routing entry (2-17 -> 0xe1 -> IRQ 17 Mode:1 Active:0)
00:01:06[D] -> 2-17 -> IRQ 17
Pin 2-19 already programmed
Pin 2-16 already programmed
Pin 2-17 already programmed
Pin 2-18 already programmed
Pin 2-16 already programmed
Pin 2-17 already programmed
Pin 2-18 already programmed
Pin 2-19 already programmed
ACPI: PCI Interrupt Link [APC5] enabled at IRQ 16
Pin 2-16 already programmed
number of MP IRQ sources: 15.
number of IO-APIC #2 registers: 24.
testing the IO APIC.......................

IO APIC #2......
.... register #00: 02000000
.......    : physical APIC id: 02
.......    : Delivery Type: 0
.......    : LTS          : 0
.... register #01: 00170011
.......     : max redirection entries: 0017
.......     : PRQ implemented: 0
.......     : IO APIC version: 0011
.... register #02: 00000000
.......     : arbitration: 00
.... IRQ redirection table:
 NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
 00 001 01  0    0    0   0   0    1    1    31
 01 001 01  0    0    0   0   0    1    1    39
 02 000 00  0    0    0   0   0    0    0    00
 03 001 01  0    0    0   0   0    1    1    41
 04 001 01  0    0    0   0   0    1    1    49
 05 001 01  0    0    0   0   0    1    1    51
 06 001 01  0    0    0   0   0    1    1    59
 07 001 01  0    0    0   0   0    1    1    61
 08 001 01  0    0    0   0   0    1    1    69
 09 001 01  0    1    0   0   0    1    1    71
 0a 001 01  0    0    0   0   0    1    1    79
 0b 001 01  0    0    0   0   0    1    1    81
 0c 001 01  0    0    0   0   0    1    1    89
 0d 001 01  0    0    0   0   0    1    1    91
 0e 001 01  0    0    0   0   0    1    1    99
 0f 001 01  0    0    0   0   0    1    1    A1
 10 001 01  1    1    0   0   0    1    1    D9
 11 001 01  1    1    0   0   0    1    1    E1
 12 001 01  1    1    0   0   0    1    1    C9
 13 001 01  1    1    0   0   0    1    1    D1
 14 001 01  1    1    0   0   0    1    1    B1
 15 001 01  1    1    0   0   0    1    1    C1
 16 001 01  1    1    0   0   0    1    1    B9
 17 001 01  1    1    0   0   0    1    1    A9
IRQ to pin mappings:
IRQ0 -> 0:2-> 0:0
IRQ1 -> 0:1
IRQ3 -> 0:3
IRQ4 -> 0:4
IRQ5 -> 0:5
IRQ6 -> 0:6
IRQ7 -> 0:7
IRQ8 -> 0:8
IRQ9 -> 0:9-> 0:9
IRQ10 -> 0:10
IRQ11 -> 0:11
IRQ12 -> 0:12
IRQ13 -> 0:13
IRQ14 -> 0:14
IRQ15 -> 0:15
IRQ16 -> 0:16
IRQ17 -> 0:17
IRQ18 -> 0:18
IRQ19 -> 0:19
IRQ20 -> 0:20
IRQ21 -> 0:21
IRQ22 -> 0:22
IRQ23 -> 0:23
.................................... done.
PCI: Using ACPI for IRQ routing
PCI: if you experience problems, try using option 'pci=noacpi' or even 'acpi=off'

mptable doesn't like my bios
I tried setting bios mp versions to both 1.1 and 1.4

albatron:/usr/src/mptable-2.0.15a # ./mptable -verbose

===============================================================================

MPTable, version 2.0.15 Linux

 looking for EBDA pointer @ 0x040e, found, searching EBDA @ 0x0009fc00
 searching CMOS 'top of mem' @ 0x0009f800 (638K)
 searching default 'top of mem' @ 0x0009fc00 (639K)
 searching BIOS @ 0x000f0000

 MP FPS found in BIOS @ physical addr: 0x000f50b0

-------------------------------------------------------------------------------

MP Floating Pointer Structure:

  location:                     BIOS
  physical address:             0x000f50b0
  signature:                    '_MP_'
  length:                       16 bytes
  version:                      1.1
  checksum:                     0x00
  mode:                         Virtual Wire

-------------------------------------------------------------------------------

MP Config Table Header:

  physical address:             0x0xf0c00
  signature:                    '$ml$'
  base table length:            0
  version:                      1.6
  checksum:                     0x00
  OEM ID:                       'Ä
                                  ¸§'
°öProduct ID:                   '(
m'P
  OEM table pointer:            0x12d90e22
  OEM table size:               7964
  entry count:                  7964
  local APIC address:           0x1f1c1f1c
  extended table length:        65284
  extended table checksum:      255

-------------------------------------------------------------------------------

MP Config Base Table Entries:

--
MPTABLE HOSED! record type = 55
albatron:/usr/src/mptable-2.0.15a #

Finally others working with kern 2.6  earlier trialled the following patch which may provide some
more clues: 
retrieved from:

http://www.kernel.org/pub/linux/kernel/people/bart/2.6.0-test11-bart1/broken-out/nforce2-apic.patch 
 
[x86] do not wrongly override mp_ExtINT IRQ

From: Mathieu <cheuche+lkml@free.fr>.

With this patch timer IRQ0 is correctly set to IO-APIC-edge
(not XT-PIC) on nForce2 boards when using APIC and ACPI.

 arch/i386/kernel/mpparse.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletion(-)

diff -puN arch/i386/kernel/mpparse.c~nforce2-apic arch/i386/kernel/mpparse.c
--- linux-2.6.0-test11/arch/i386/kernel/mpparse.c~nforce2-apic	2003-12-08 00:12:25.782597272 +0100
+++ linux-2.6.0-test11-root/arch/i386/kernel/mpparse.c	2003-12-08 00:12:25.786596664 +0100
@@ -962,7 +962,8 @@ void __init mp_override_legacy_irq (
 	 */
 	for (i = 0; i < mp_irq_entries; i++) {
 		if ((mp_irqs[i].mpc_dstapic == intsrc.mpc_dstapic)
-			&& (mp_irqs[i].mpc_srcbusirq == intsrc.mpc_srcbusirq)) {
+			&& (mp_irqs[i].mpc_srcbusirq == intsrc.mpc_srcbusirq)
+			&& (mp_irqs[i].mpc_irqtype == intsrc.mpc_irqtype)) {
 			mp_irqs[i] = intsrc;
 			found = 1;
 			break;

_

however the results were not completely successful as this posting shows it
routing through the 8259?

http://linux.derkeiler.com/Mailing-Lists/Kernel/2003-12/1303.html

dmesg differences: 

1. 
 after: 
 ..TIMER: vector=0x31 pin1=2 pin2=0 

before: 
 ..TIMER: vector=0x31 pin1=2 pin2=-1 

2. 
 after: 
 ...trying to set up timer (IRQ0) through the 8259A ... 
 ..... (found pin 0) ...works. 
 number of MP IRQ sources: 16. 

before: 
 ...trying to set up timer (IRQ0) through the 8259A ... failed. 
 ...trying to set up timer as Virtual Wire IRQ... failed. 
 ...trying to set up timer as ExtINT IRQ... works. 
 number of MP IRQ sources: 15. 

Perhaps someone else could get mptable to run on their machine and send you
the result.

Regards
Ross 

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Maciej W. Rozycki]

On Wed, 10 Dec 2003, Ross Dickson wrote:

> Relevant dmesg output from Albatron KM18G Pro ( this is different MOBO (same type) but 
> this time has a barton core 2500 XP cpu).
> 
> enabled ExtINT on CPU#0
> ESR value before enabling vector: 00000000
> ESR value after enabling vector: 00000000
> ENABLING IO-APIC IRQs
> init IO_APIC IRQs
>  IO-APIC (apicid-pin) 2-0, 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
> ..TIMER: vector=0x31 pin1=2 pin2=-1
> ..MP-BIOS bug: 8254 timer not connected to IO-APIC pin2
> ..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...
> IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
> ..TIMER check 8259 ints disabled, imr1:ff, imr2:ff
> ..TIMER: works OK on apic pin0 irq0
> Using local APIC timer interrupts.
> calibrating APIC timer ...
> ..... CPU clock speed is 1829.0708 MHz.
> ..... host bus clock speed is 332.0674 MHz.
> cpu: 0, clocks: 332674, slice: 166337
> CPU0<T0:332672,T1:166320,D:15,S:166337,C:332674>

 Hmm, while this is different from what is documented in the MP Spec, it
looks like the 8254 IRQ is connected to INTIN0 indeed.  We can handle such
a setup if the BIOS reports routing correctly.  Since you invoke
io_apic_set_pci_routing() I assume you use ACPI for IRQ routing
information.  Can you please rebuild the kernel with APIC_DEBUG set to 1
in include/asm-i386/apic.h and send me the bootstrap log?  Can you please
send me the output of a tool called `mptable' as well, so that I can
compare the results?

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Mikael Pettersson]

Jesse Allen writes:
 > --- linux/arch/i386/kernel/apic.c	2003-10-25 11:44:59.000000000 -0700
 > +++ linux-jla/arch/i386/kernel/apic.c	2003-12-09 19:07:19.000000000 -0700
 > @@ -1089,6 +1089,16 @@
 >  	 */
 >  	irq_stat[cpu].apic_timer_irqs++;
 >  
 > +#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
 > +
 > +	/*
 > +	 * on 2200XP & nforce2 chipset we need at least 500ns delay here
 > +	 * to stop lockups with udma100 drive. try to scale delay time
 > +	 * with cpu speed. Ross Dickson.
 > +	 */
 > +	ndelay((cpu_khz >> 12)+200 ); /* don't ack too soon or hard lockup */
 > +#endif
 > +
 >  	/*
 >  	 * NOTE! We'd better ACK the irq immediately,
 >  	 * because timer handling can be slow.

This is too much of a kludge. APIC timer ACKing is supposed to be fast.
Please try without this delay but with the disconnect PCI quirk.

If the delay is still needed even when disconnect is disabled, _then_
can discuss how to do the delay properly.

/Mikael

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

On Wednesday 10 December 2003 13:39, Jesse Allen wrote:
> Hi Ross,
> 
> I have rediffed your two patches for vanilla 2.6.0-test11.  Briefly, I tried the apic patch first, and found that there are no lockups so far; well it passed my grep tests and even a kernel compile =).  Then I tried your io_apic patch + apic patch.  With nmi_watchdog=1 "NMI:" in /proc/interrupts increments alot compared to nmi_watchdog=2 before (as much as the timer).  So I believe your two patches are more correct than the other two.  Especially the fact I can run with CPU Disconnect and not lock up just like windows ... for people that have windows (I dont have windows =) plus a probably working nmi_watchdog.
> 
> And for comparison, my setup:
> Shuttle AN35N Ultra v 1.1  (Nforce2 400 ultra), bios updated
> Athlon Barton 2600+ (1.9 Ghz)
> 256 MB PC3200, single stick.
> 
> The patches are included in this mail.  I suppose the next thing to do is get out of nvidia the corresponding information.  And then clean up the patch for inclusion.
> 
> Jesse
> 
> 
> 
Thank Jesse
It is interesting that the lockup problems also occur with a single memory stick,
I have only tried dual sticks.
Regards
Ross.

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

On Wednesday 10 December 2003 20:00, Mikael Pettersson wrote:
> Jesse Allen writes:
>  > --- linux/arch/i386/kernel/apic.c	2003-10-25 11:44:59.000000000 -0700
>  > +++ linux-jla/arch/i386/kernel/apic.c	2003-12-09 19:07:19.000000000 -0700
>  > @@ -1089,6 +1089,16 @@
>  >  	 */
>  >  	irq_stat[cpu].apic_timer_irqs++;
>  >  
>  > +#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
>  > +
>  > +	/*
>  > +	 * on 2200XP & nforce2 chipset we need at least 500ns delay here
>  > +	 * to stop lockups with udma100 drive. try to scale delay time
>  > +	 * with cpu speed. Ross Dickson.
>  > +	 */
>  > +	ndelay((cpu_khz >> 12)+200 ); /* don't ack too soon or hard lockup */
>  > +#endif
>  > +
>  >  	/*
>  >  	 * NOTE! We'd better ACK the irq immediately,
>  >  	 * because timer handling can be slow.
> 
> This is too much of a kludge. APIC timer ACKing is supposed to be fast.
> Please try without this delay but with the disconnect PCI quirk.
> 
> If the delay is still needed even when disconnect is disabled, _then_
> can discuss how to do the delay properly.
> 
> /Mikael
> 
> 
> 

Thanks Mikael, I think the more heads on this problem the better.

I don't like timing kludges either such as this existing one in ide-iops.c
in kernel 2.4.23

	hwif->OUTBSYNC(drive, cmd, IDE_COMMAND_REG);
	/* Drive takes 400nS to respond, we must avoid the IRQ being
	   serviced before that.

	   FIXME: we could skip this delay with care on non shared
	   devices

	   For DMA transfers highpoint have a neat trick we could
	   use. When they take an IRQ they check STS but also that
	   the DMA count is not zero (see hpt's own driver)
	*/
	ndelay(400);
	spin_unlock_irqrestore(&io_request_lock, flags);
}

But does anyone exactly know what nvidia and the bios writer are doing - why the 
cpu-disconnect is an issue for the nforce2 boards?

Is it technically correct in their view to turn off features that some pci or
other device they have made may expect? I wonder about their
ram devices because I note after some more testing that without any 
lockup fixes the lockups were spaced a lot further apart in time when I used 
a pair of KINGMAX 256MB DDR-333 then when I used a pair of SEITEC 256MB 
DDR-400 memory. The cpu used XP2500 has a 333 fsb. Is the ram driver chip core
enforcing the disconnect for a reason?

When using the ack delay, lockups with both memory types ceased - as they may 
also cease with the disconnect patch. So the disconnect cycles seem related 
to the nforce2 ram driver circuitry. (See Ian's take towards end of this email)

The reason why I put the ack delay in only the apic timer servicing path is that
I think it is the only commonly traversed path which acks the apic so quickly. If
we end up stuck with a delay then we could probably make it more accurate by
reading the apic timer withinin the delay and using the counts down from the reload
value because if our irq was already pre delayed then no additional delay would 
be required. I am sure many clever programmers can improve on it - not that we
want it at all.

I note the following comments in 2.2.23 io_apic.c
/*
 * Level triggered interrupts can just be masked,
 * and shutting down and starting up the interrupt
 * is the same as enabling and disabling them -- except
 * with a startup need to return a "was pending" value.
 *
 * Level triggered interrupts are special because we
 * do not touch any IO-APIC register while handling
 * them. We ack the APIC in the end-IRQ handler, not
 * in the start-IRQ-handler. Protection against reentrance
 * from the same interrupt is still provided, both by the
 * generic IRQ layer and by the fact that an unacked local
 * APIC does not accept IRQs.
 */
If I am reading this correctly then PCI interrupts which are level 
triggered are processed with the equivalent of a global (maskable) hardware 
interrupt disable (on a uniprocessor machine) if all hardware interrupts
are routed via the APIC. Chances are that we have more than 500ns irq off
times occurring with these servicing routines especially if several handlers
are chained on the one pirq.

Another clue may have just come to light, does the ack in this routine (io_apic.c)
usually get done within the 500ns or so from its activation? If it does then either the
mask_IO_APIC_irq() has a positive effect on the lockups or alternately the 
problem is synchronous with, or inherent to the apic timer.

/*
 * Once we have recorded IRQ_PENDING already, we can mask the
 * interrupt for real. This prevents IRQ storms from unhandled
 * devices.
 */
static void ack_edge_ioapic_irq(unsigned int irq)
{
	if ((irq_desc[irq].status & (IRQ_PENDING | IRQ_DISABLED))
					== (IRQ_PENDING | IRQ_DISABLED))
		mask_IO_APIC_irq(irq);
	ack_APIC_irq();
}


How slow can timer handling be?
When I was debugging with my CRO on the LPT port and turning a bit on
going into the smp_apic_timer_interrupt() routine and turning the bit off
when exiting I saw times of greater than 0.5ms for the routine to complete.
Thats milliseconds!. I certainly agree with the comment regarding the ack 
immediately and think it means before 0.5ms instead of after 0.5ms 
because 0.5ms is an eternity to have interrupts disabled in a hardware 
interrupt context.
 /*
  * NOTE! We'd better ACK the irq immediately,
  * because timer handling can be slow.
I am not too crazy about having them off for 500ns to 1000ns either but until I 
know for certain that the cpu disconnect issue is a non issue then I will
choose to suffer a time hit, and leave the hardware run as the maker intended.

BIG HINT TO THOSE IN THE KNOW.
If we had the docs from nvidia regarding the unknown pci devices?

00:00.0 Host bridge: nVidia Corporation: Unknown device 01e0 (rev a2)
00:00.1 RAM memory: nVidia Corporation: Unknown device 01eb (rev a2)
00:00.2 RAM memory: nVidia Corporation: Unknown device 01ee (rev a2)
00:00.3 RAM memory: nVidia Corporation: Unknown device 01ed (rev a2)
00:00.4 RAM memory: nVidia Corporation: Unknown device 01ec (rev a2)
00:00.5 RAM memory: nVidia Corporation: Unknown device 01ef (rev a2)

then perhaps the underlying cause would present itself. Then we could properly
deal with the issue because we would know why we should do whatever it
is we should do. If the disconnect should be left on then hopefully we could test
a register somewhere to know when it is safe to ack or not - something like
that.

I think Ian is heading in the right direction with his comments:

On Wednesday 10 December 2003 11:20, Ian Kumlien wrote:
> Hi, again.
> 
> I did some reading on amd's site, and if the disconnect + apic fixed the
> same problem as the ~500ns delay, then it could be as i suspect...
> 
> I suspect that something goes wrong with apic ack when the cpu is
> disconnected and according to the amd docs we could check the
> Northbridge's CLKFWDRST or isn't that avail on the outside?
> (It would be interesting to see if that fixes the problem as well.)
> 
> http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/26237.PDF
> 
> I don't really have the knowledge but it would sure be nicer to fix this
> by checking this than to just disable it. I dunno if there is something
> we could do from within the kernel aswell with the sending of HLT but i
> doubt it.
> 
> Anyways, we need a generalized patch that does better checking on the
> NMI bit (like Ross' patch). 
> 
> PS. Anyone that can point me to northbridge tech docks? and CC
> 
> -- 
> Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
> 

Regards
Ross.

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

On Wednesday 10 December 2003 01:20, Maciej W. Rozycki wrote:
> On Sun, 7 Dec 2003, Ross Dickson wrote:
> 
> > b) I was also disappointed to see I could not have irq0 timer IO-APIC-edge. 
> > So I have fixed it too (tested on both my epox and albatron MOBOs).
> > Firstly I found 8254 connected directly to pin 0 not pin 2 of io-apic.
> > I have modified check_timer() in io_apic.c to trial connect pin and test for it
> > after the existing test for connection to io-apic.
> 
>  I'm pretty sure this part is bogus.  Have you actually verified it either
> by using a hardware probe or at least by investigating documentation you
> really have IRQ 0 routed to the I/O APIC interrupt #0 (INTIN 0)?  If no,
> then you can almost surely see interrupts travelling across the pair of
> 8259A PICS which are connected to the INTIN 0 input of the first I/O APIC
> in every IA32-based PC system providing an I/O APIC seen so far.
> 
> -- 
> +  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
> +--------------------------------------------------------------+
> +        e-mail: macro@ds2.pg.gda.pl, PGP key available        +
> 
> 
> 

Thanks Maciej for your response.

If everyone followed published recommendations then I would agree with
your comments however nvidia? et al?.

I have no appropriate docs so I cannot confirm via a real hardware probe 
so I can only offer a software confirmation.

Background musings:

I was forced to approach the problems using somewhat educated guesses and
with the tools I had at hand. As with most discoveries about black boxes the answer 
comes about by a combination of educated guess, luck and checking the unlikely.
The apic delay (a) came about because the lockup problem went away when I put 
a debugging outb_p() statement flipping bits at the lpt port while I was trying 
to catch the frozen IRQ state info on my CRO. I was pleasantly surprised when 
the lockups ceased so I replaced the outb_p with a delay and trimmed it as 
best I could without docs. I did not change it within the Ack call as I realised 
that all the other normal apic ack paths had considerably more code delay time - 
although could this be a gotcha depending on what code path is in the driver.
What if we had a really fast cpu or is it restricted solely to the timer irq??

Back to your query:

I approached the io-apic edge with the same what if?
I think I got it right but please check my following code to confirm.  I have
since hacked the kernel as follows.

WARNING Following Mods For Debugging Only!

In File i8259.c I needed to get to "cached_irq_mask" 

/*
 * This contains the irq mask for both 8259A irq controllers,
 */
//static unsigned int cached_irq_mask = 0xffff; debug ross
unsigned int cached_irq_mask = 0xffff;


In File io_apic.c I have tried to fully mask the 8259.

/*
 * This code may look a bit paranoid, but it's supposed to cooperate with
 * a wide range of boards and BIOS bugs.  Fortunately only the timer IRQ
 * is so screwy.  Thanks to Brian Perkins for testing/hacking this beast
 * fanatically on his truly buggy board.
 */
// debug ross
extern spinlock_t i8259A_lock;
extern unsigned int cached_irq_mask;

static inline void check_timer(void)
{
....
<snip>
....
#ifdef CONFIG_ACPI_BOOT && CONFIG_X86_UP_IOAPIC
	/* for nforce2 try vector 0 on pin0
	 * Note the io_apic_set_pci_routing call disables the 8259 irq 0
	 * so we must be connected directly to the 8254 timer if this works
	 * Note2: this violates the above comment re Subtle but works!
	 */
	printk(KERN_INFO "..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...\n");
	if ( pin1 != -1 && nr_ioapics ) {
		int result, tok;
		unsigned long flags;
		unsigned int saved_cached_irq_mask;
		unsigned char imr1, imr2;

		int saved_timer_ack = timer_ack;

		// disable all of 8259 irq's
		spin_lock_irqsave(&i8259A_lock, flags);
		saved_cached_irq_mask = cached_irq_mask;
		cached_irq_mask = 0xffff;; // ensure nothing restores 8259 ints
		outb(0xff, 0x21);	/* mask all of 8259A-1 */
		outb(0xff, 0xA1);	/* mask all of 8259A-2 */
		spin_unlock_irqrestore(&i8259A_lock, flags);

		/*
		 * Ok, does IRQ0 through the IOAPIC work?
		 */
		/* next call also disables 8259 irq0 */
		result = io_apic_set_pci_routing ( 0, 0, 0, 0, 0);
		unmask_IO_APIC_irq(0);
		timer_ack = 0 ;

		spin_lock_irqsave(&i8259A_lock, flags);
		imr1 = inb(0x21);
		imr2 = inb(0xA1);
		printk("..TIMER check 8259 ints disabled, imr1:%02x, imr2:%02x\n", imr1, imr2);
		tok = timer_irq_works();
		spin_unlock_irqrestore(&i8259A_lock, flags);

		// restore 8259 mask
		spin_lock_irqsave(&i8259A_lock, flags);
		cached_irq_mask = saved_cached_irq_mask;
		outb( cached_irq_mask & 0xff, 0x21 ); /* restore all of 8259A-1 */
		outb( cached_irq_mask >> 8, 0xA1 ); /* restore all of 8259A-2 */
		spin_unlock_irqrestore(&i8259A_lock, flags);

		/*
		 * Ok, does IRQ0 through the IOAPIC work?
		 */
//		unmask_IO_APIC_irq(0);
//		timer_ack = 0 ;
//		if (timer_irq_works()) {
		if (tok) {
			if (nmi_watchdog == NMI_IO_APIC) {
				disable_8259A_irq(0);
				setup_nmi();
				enable_8259A_irq(0);
				check_nmi_watchdog();
			}
			printk(KERN_INFO "..TIMER: works OK on apic pin0 irq0\n" );
			return;
		}
		/* failed */
		timer_ack = saved_timer_ack;
		clear_IO_APIC_pin(0, 0);
		result = io_apic_set_pci_routing ( 0, pin1, 0, 0, 0);
		printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC Pin 0\n");
	}
#endif
/* end new stuff for nforce2 */

The inner spinlock around timer_irq_works() I think is redundant but I put it there 
for good measure.
Relevant dmesg output from Albatron KM18G Pro ( this is different MOBO (same type) but 
this time has a barton core 2500 XP cpu).

enabled ExtINT on CPU#0
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
ENABLING IO-APIC IRQs
init IO_APIC IRQs
 IO-APIC (apicid-pin) 2-0, 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
..TIMER: vector=0x31 pin1=2 pin2=-1
..MP-BIOS bug: 8254 timer not connected to IO-APIC pin2
..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...
IOAPIC[0]: Set PCI routing entry (2-0 -> 0x31 -> IRQ 0 Mode:0 Active:0)
..TIMER check 8259 ints disabled, imr1:ff, imr2:ff
..TIMER: works OK on apic pin0 irq0
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 1829.0708 MHz.
..... host bus clock speed is 332.0674 MHz.
cpu: 0, clocks: 332674, slice: 166337
CPU0<T0:332672,T1:166320,D:15,S:166337,C:332674>


Please advise if anyone knows of extra registers which may have been added to the
nforce2 8259 core which could allow the interrupts through the masked chip core?
I note that they may exist after reading your email March 21 2002 (irq FosterP4) 
http://www.ussg.iu.edu/hypermail/linux/kernel/0203.2/1213.html

Note that I think it is safe to leave the 8259 irq(0) implicitly disabled on 
failure exit as the code paths following my code patch do it anyway.


Regards
Ross.

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Jesse Allen]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

Hi Ross,

I have rediffed your two patches for vanilla 2.6.0-test11.  Briefly, I tried the apic patch first, and found that there are no lockups so far; well it passed my grep tests and even a kernel compile =).  Then I tried your io_apic patch + apic patch.  With nmi_watchdog=1 "NMI:" in /proc/interrupts increments alot compared to nmi_watchdog=2 before (as much as the timer).  So I believe your two patches are more correct than the other two.  Especially the fact I can run with CPU Disconnect and not lock up just like windows ... for people that have windows (I dont have windows =) plus a probably working nmi_watchdog.

And for comparison, my setup:
Shuttle AN35N Ultra v 1.1  (Nforce2 400 ultra), bios updated
Athlon Barton 2600+ (1.9 Ghz)
256 MB PC3200, single stick.

The patches are included in this mail.  I suppose the next thing to do is get out of nvidia the corresponding information.  And then clean up the patch for inclusion.

Jesse



[-- Attachment #2: nforce2-apic-delay-2.6t11.patch --]
[-- Type: text/plain, Size: 611 bytes --]

--- linux/arch/i386/kernel/apic.c	2003-10-25 11:44:59.000000000 -0700
+++ linux-jla/arch/i386/kernel/apic.c	2003-12-09 19:07:19.000000000 -0700
@@ -1089,6 +1089,16 @@
 	 */
 	irq_stat[cpu].apic_timer_irqs++;
 
+#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
+
+	/*
+	 * on 2200XP & nforce2 chipset we need at least 500ns delay here
+	 * to stop lockups with udma100 drive. try to scale delay time
+	 * with cpu speed. Ross Dickson.
+	 */
+	ndelay((cpu_khz >> 12)+200 ); /* don't ack too soon or hard lockup */
+#endif
+
 	/*
 	 * NOTE! We'd better ACK the irq immediately,
 	 * because timer handling can be slow.

[-- Attachment #3: nforce2-ioapic-timer-2.6t11.patch --]
[-- Type: text/plain, Size: 1564 bytes --]

--- linux/arch/i386/kernel/io_apic.c	2003-10-25 11:43:20.000000000 -0700
+++ linux-jla/arch/i386/kernel/io_apic.c	2003-12-09 19:56:07.000000000 -0700
@@ -2128,6 +2128,41 @@
 		printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n");
 	}
 
+#ifdef CONFIG_ACPI_BOOT && CONFIG_X86_UP_IOAPIC
+	/* for nforce2 try vector 0 on pin0
+	 * Note the io_apic_set_pci_routing call disables the 8259 irq 0
+	 * so we must be connected directly to the 8254 timer if this works
+	 * Note2: this violates the above comment re Subtle but works!
+	 */
+	printk(KERN_INFO "..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...\n");
+	if ( pin1 != -1 && nr_ioapics ) {
+		int saved_timer_ack = timer_ack;
+		/* next call also disables 8259 irq0 */
+		int result = io_apic_set_pci_routing ( 0, 0, 0, 0, 0);
+		/*
+		 * Ok, does IRQ0 through the IOAPIC work?
+		 */
+		unmask_IO_APIC_irq(0);
+		timer_ack = 0 ;
+		if (timer_irq_works()) {
+			if (nmi_watchdog == NMI_IO_APIC) {
+				disable_8259A_irq(0);
+				setup_nmi();
+				enable_8259A_irq(0);
+				check_nmi_watchdog();
+			}
+			printk(KERN_INFO "..TIMER: works OK on apic pin0 irq0\n" );
+			return;
+		}
+		/* failed */
+		timer_ack = saved_timer_ack;
+		clear_IO_APIC_pin(0, 0);
+		result = io_apic_set_pci_routing ( 0, pin1, 0, 0, 0);
+		printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC Pin 0\n");
+	}
+#endif
+/* end new stuff for nforce2 */
+
 	printk(KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... ");
 	if (pin2 != -1) {
 		printk("\n..... (found pin %d) ...", pin2);

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Maciej W. Rozycki]

On Sun, 7 Dec 2003, Ross Dickson wrote:

> b) I was also disappointed to see I could not have irq0 timer IO-APIC-edge. 
> So I have fixed it too (tested on both my epox and albatron MOBOs).
> Firstly I found 8254 connected directly to pin 0 not pin 2 of io-apic.
> I have modified check_timer() in io_apic.c to trial connect pin and test for it
> after the existing test for connection to io-apic.

 I'm pretty sure this part is bogus.  Have you actually verified it either
by using a hardware probe or at least by investigating documentation you
really have IRQ 0 routed to the I/O APIC interrupt #0 (INTIN 0)?  If no,
then you can almost surely see interrupts travelling across the pair of
8259A PICS which are connected to the INTIN 0 input of the first I/O APIC
in every IA32-based PC system providing an I/O APIC seen so far.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 3751 bytes --]

On Mon, 2003-12-08 at 03:07, Ross Dickson wrote:
> On Monday 08 December 2003 05:58, you wrote:
> > > There are three parts to this email.
> > > a) apic mods.
> > > Lockups are due to too fast an apic acknowledge of apic timer int.
> > > Apic hard locked up the system - no nmi debug available.
> > > Fixed it by introducing a delay of at least 500ns into 
> > > smp_apic_timer_interrupt() just prior to ack_APIC_irq().
> > 
> > I find this really odd... It works just fine... 
> > As did disabling whats now active ie:
> > 'Halt Disconnect and Stop Grant Disconnect' bit is enabled.
> > 
> > So it seems like these are the two most important factors, at least from
> > where i stand. Both enabled me to actually use my machine with IO-APIC.
> > (1, disabling Halt Disconnect and Stop Grand Disconnect bit or 2, Add a
> > delay on the irq ack.)
> > Anyone that has any clues?
> 
> I started work on this about 2 weeks ago and have not yet tried the 
> "Halt Disconnect patch and Stop Grand Disconnect bit or 2" patch. 
> My lockups ceased with just the apic time delay. I agree the delay is
> wasteful but the code branches into other servicing routines which I
> did not want to try to rearrange as yet. Given the infrequent nature
> of the lockup in CPU cycles maybe we can get smarter and read the
> timer register and see if enough time has expired to safely ack it. Is
> the Apic read cycle fast like cache ram (assume as fast as bus
> cycles?) or slow like 8259?

It could be something like cpu-disconnect-while-apic-being-hammered
race... Hell i dunno, i can only see some patterns from my own
experiences.

> After all it counts bus cycles doesn't it.  Alternately perhaps there
> is a status bit in the apic somewhere to check against after we ack to
> ensure that it did its job although we would not want to hammer the
> apic with writes it cannot accept? Or maybe it is not too bad for now,
> I note that there is an existing fixed 400ns delay in the IDE command
> routines ide-iops.c ide_execute_command() which we currently tolerate.

A status bit would be preferred since that would fix any races that
might exist. As for ide it would be preferred to have non-static delays
but if it's not possible to solve in another way then, by all means =).

But in this case, this is a specific workaround, and it works without it
if you disable the AMD cooler thing.

> > > b) io-apic mods
> > > So I have fixed it too (tested on both my epox and albatron MOBOs).
> > > Firstly I found 8254 connected directly to pin 0 not pin 2 of io-apic.
> > > I have modified check_timer() in io_apic.c to trial connect pin and 
> > > test for it after the existing test for connection to io-apic.
> > 
> > Good job, i wonder if it could be more generalized and integrated with
> > the rest of the code (i haven't even checked the rest of the code, but
> > this seemed separated).
> > 
> > One thing though, I get a lot more NMI's now than with nmi_watchdog=2...
> > NMI:      85520
> > LOC:      85477
> > 
> > I usually had a 3 figure number by now... but.. =)

> I have not tested the nmi against the b) io-apic mods. We may have a
> vector clash? Perhaps the new apic mpparse.c patch lets the existing
> check_timer() routines work properly?
> I have not yet tried it.

I dunno, i have never used a machine with IO-APIC on so i dunno whats
normal. But it grows a lot faster than nmi_watchdog=2.

> > > c) ide driver mods
> > 
> > Cool.. 
> > 
> > I applied all patches and it survived my grep test so i think it works.

6h 46 mins tells me that this works.
(peak nonworking uptime is still below 2hours)

-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

On Monday 08 December 2003 05:58, you wrote:
> > I have monitored list and know my nforce2 experiences have been
> > common.
> 
> Hell yeah =)
> 
> > When I enabled either apic or io-apic in kern config, lockups came 
> > hard and fast. Particularly bad under hard disk load. Heaps of lost 
> > ints on irq7 in apic and ioapic mode. Lockups disappeared when I 
> > lowered the ide hda udma speed to mode 3 with hdparm so I went looking
> > for answers which now follow.
> 
> Good job =)
Thanks.
> 
> > There are three parts to this email.
> > a) apic mods.
> > Lockups are due to too fast an apic acknowledge of apic timer int.
> > Apic hard locked up the system - no nmi debug available.
> > Fixed it by introducing a delay of at least 500ns into 
> > smp_apic_timer_interrupt() just prior to ack_APIC_irq().
> 
> I find this really odd... It works just fine... 
> As did disabling whats now active ie:
> 'Halt Disconnect and Stop Grant Disconnect' bit is enabled.
> 
> So it seems like these are the two most important factors, at least from
> where i stand. Both enabled me to actually use my machine with IO-APIC.
> (1, disabling Halt Disconnect and Stop Grand Disconnect bit or 2, Add a
> delay on the irq ack.)
> Anyone that has any clues?

I started work on this about 2 weeks ago and have not yet tried the 
"Halt Disconnect patch and Stop Grand Disconnect bit or 2" patch. 
My lockups ceased with just the apic time delay. I agree the delay is wasteful
but the code branches into other servicing routines which I did not want to try to
rearrange as yet. Given the infrequent nature of the lockup in CPU cycles maybe we can get
smarter and read the timer register and see if enough time has expired to safely ack it. Is
the Apic read cycle fast like cache ram (assume as fast as bus cycles?) or slow like 8259?
Doing something like,
=
= loop:	Read Apic timer count
=	if count is xx cycles since rollover then 
=		Ack apic
=	else
=		loop
=
After all it counts bus cycles doesn't it.  Alternately perhaps there is a status bit in the apic 
somewhere to check against after we ack to ensure that it did its job although we would
not want to hammer the apic with writes it cannot accept?
Or maybe it is not too bad for now, I note that there is an existing fixed 400ns delay in the
IDE command routines ide-iops.c ide_execute_command() which we currently tolerate.

> 
> > b) io-apic mods
> > So I have fixed it too (tested on both my epox and albatron MOBOs).
> > Firstly I found 8254 connected directly to pin 0 not pin 2 of io-apic.
> > I have modified check_timer() in io_apic.c to trial connect pin and 
> > test for it after the existing test for connection to io-apic.
> 
> Good job, i wonder if it could be more generalized and integrated with
> the rest of the code (i haven't even checked the rest of the code, but
> this seemed separated).
> 
> One thing though, I get a lot more NMI's now than with nmi_watchdog=2...
> NMI:      85520
> LOC:      85477
> 
> I usually had a 3 figure number by now... but.. =)
I have not tested the nmi against the b) io-apic mods. We may have a vector clash?
Perhaps the new apic mpparse.c patch lets the existing check_timer() routines work properly?
I have not yet tried it.


> 
> > c) ide driver mods
> 
> Cool.. 
> 
> I applied all patches and it survived my grep test so i think it works.
> 
> -- 
> Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
> 

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Jesse Allen]

On Sun, Dec 07, 2003 at 08:58:47PM +0100, Ian Kumlien wrote:
> > There are three parts to this email.
> > a) apic mods.
> > Lockups are due to too fast an apic acknowledge of apic timer int.
> > Apic hard locked up the system - no nmi debug available.
> > Fixed it by introducing a delay of at least 500ns into 
> > smp_apic_timer_interrupt() just prior to ack_APIC_irq().
> > b) io-apic mods
> > So I have fixed it too (tested on both my epox and albatron MOBOs).
> > Firstly I found 8254 connected directly to pin 0 not pin 2 of io-apic.
> > I have modified check_timer() in io_apic.c to trial connect pin and 
> > test for it after the existing test for connection to io-apic.
> 


So do you think Ross has found the connection between all three issues?  (Timer and NMI watchdog, IRQ 7, and CPU disconnect?)

I suppose we should now try these changes other the last two.  From what he's saying, this will fix the lockup too.  Hopefully the patch will be refined =).  I probably won't be able to run this till tomorrow, and after I get it diffed for 2.6.  I've barely got the cpu disconnect patch going today, but haven't tested it because I can only access my nforce2 machine remotely =).  But from everything I'm hearing, the lockups are gone.

Jesse

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 1906 bytes --]

On Sun, 2003-12-07 at 21:59, Jesse Allen wrote:
> On Sun, Dec 07, 2003 at 08:58:47PM +0100, Ian Kumlien wrote:
> > > There are three parts to this email.
> > > a) apic mods.
> > > Lockups are due to too fast an apic acknowledge of apic timer int.
> > > Apic hard locked up the system - no nmi debug available.
> > > Fixed it by introducing a delay of at least 500ns into 
> > > smp_apic_timer_interrupt() just prior to ack_APIC_irq().
> > > b) io-apic mods
> > > So I have fixed it too (tested on both my epox and albatron MOBOs).
> > > Firstly I found 8254 connected directly to pin 0 not pin 2 of io-apic.
> > > I have modified check_timer() in io_apic.c to trial connect pin and 
> > > test for it after the existing test for connection to io-apic. 
> 
> So do you think Ross has found the connection between all three
> issues?  (Timer and NMI watchdog, IRQ 7, and CPU disconnect?)

Since disabling CPU disconnect or adding this delay both seems to fix
the deadlock, i see the rest as cool stuff. 

> I suppose we should now try these changes other the last two.  From
> what he's saying, this will fix the lockup too.  Hopefully the patch
> will be refined =).  I probably won't be able to run this till
> tomorrow, and after I get it diffed for 2.6.  I've barely got the cpu
> disconnect patch going today, but haven't tested it because I can only
> access my nforce2 machine remotely =).  But from everything I'm
> hearing, the lockups are gone.

Yes, at least for me, and my machine really liked to deadlock.

I'm wondering however, could the ACK be moved down in the code? after
some handling? (i have no clue of how the code works, and even less
about how the hw part works, fyi)
But i thought that if it's moved then that might be a generic way to fix
it without adding a blocking delay.

-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 1985 bytes --]

> I have monitored list and know my nforce2 experiences have been
> common.

Hell yeah =)

> When I enabled either apic or io-apic in kern config, lockups came 
> hard and fast. Particularly bad under hard disk load. Heaps of lost 
> ints on irq7 in apic and ioapic mode. Lockups disappeared when I 
> lowered the ide hda udma speed to mode 3 with hdparm so I went looking
> for answers which now follow.

Good job =)

> There are three parts to this email.
> a) apic mods.
> Lockups are due to too fast an apic acknowledge of apic timer int.
> Apic hard locked up the system - no nmi debug available.
> Fixed it by introducing a delay of at least 500ns into 
> smp_apic_timer_interrupt() just prior to ack_APIC_irq().

I find this really odd... It works just fine... 
As did disabling whats now active ie:
'Halt Disconnect and Stop Grant Disconnect' bit is enabled.

So it seems like these are the two most important factors, at least from
where i stand. Both enabled me to actually use my machine with IO-APIC.
(1, disabling Halt Disconnect and Stop Grand Disconnect bit or 2, Add a
delay on the irq ack.)
Anyone that has any clues?

> b) io-apic mods
> So I have fixed it too (tested on both my epox and albatron MOBOs).
> Firstly I found 8254 connected directly to pin 0 not pin 2 of io-apic.
> I have modified check_timer() in io_apic.c to trial connect pin and 
> test for it after the existing test for connection to io-apic.

Good job, i wonder if it could be more generalized and integrated with
the rest of the code (i haven't even checked the rest of the code, but
this seemed separated).

One thing though, I get a lot more NMI's now than with nmi_watchdog=2...
NMI:      85520
LOC:      85477

I usually had a 3 figure number by now... but.. =)

> c) ide driver mods

Cool.. 

I applied all patches and it survived my grep test so i think it works.

-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered

[Ross Dickson]

[-- Attachment #1: Type: text/plain, Size: 5471 bytes --]

Greetings,
I am not subscribed so please cc responses.
I have monitored list and know my nforce2 experiences have been common.
Attached patches are in a single bzip tar ball.

I have Albatron KM18G Pro & Epox 8RGA+ MOBOs both using nforce2 chipsets.
I made up a kernel as follows.
Get std 2.4.22 src
apply patch-2.4.23
apply 2.4.22-low-latency.patch
apply preempt-kernel-rml-2.4.23-pre5-1.patch
apply vhz-j64-2.4.22.patch

One patch fails on inode.c, dispose_list() so I placed conditional_schedule() as follows
=static void dispose_list(struct list_head *head)
={
=	int nr_disposed = 0;
=
=	while (!list_empty(head)) {
=		struct inode *inode;
=		conditional_schedule();

Config for athlon with 1000hz tics, preempt & low-lat on.
Compiled and installed nvnet & nvidia video driver.

Disclaimer: The following information and code patches are not fully tested and may be 
dangerous, also these are the first patches I have made for public consumption so I hope
that their format works.

Note also that the patches are against 2.4.22 even though they were developed
against the heavily patched 2.4.23 mentioned above. The patch code is the same for both
kernels but at different line numbers.

When I enabled either apic or io-apic in kern config, lockups came hard and fast.
Particularly bad under hard disk load. Heaps of lost ints on irq7 in apic and ioapic mode. 
Lockups disappeared when I lowered the ide hda udma speed to mode 3 with hdparm so
I went looking for answers which now follow.

There are three parts to this email.
a) apic mods.
b) io-apic mods
c) ide driver mods

a) Lockups are due to too fast an apic acknowledge of apic timer int.
Apic hard locked up the system - no nmi debug available.
Fixed it by introducing a delay of at least 500ns into smp_apic_timer_interrupt() 
just prior to ack_APIC_irq().
See attached diff file "nforce2-apic.c-2.4.22.patch" for details. 
I have guessed at a suitable cpu speed dependent delay.
Perhaps someone with AMD cpu docs (apic timing specs)  & analyser tools could refine it.

Maybe nforce2 chipset really is very quick accessing ram in dual dimm mode? 
Or AMD 2200XP has a really slow APIC?

--- linux-2.4.22/arch/i386/kernel/apic.c	2003-06-14 00:51:29.000000000 +1000
+++ linux-2.4.22-rd/arch/i386/kernel/apic.c	2003-12-07 18:27:32.000000000 +1000
@@ -1078,6 +1078,15 @@
 	 */
 	apic_timer_irqs[cpu]++;

+#ifdef CONFIG_MK7 && CONFIG_BLK_DEV_AMD74XX
+	/*
+	 * on 2200XP & nforce2 chipset we need at least 500ns delay here
+	 * to stop lockups with udma100 drive. try to scale delay time
+	 * with cpu speed. Ross Dickson.
+	 */
+	ndelay((cpu_khz >> 12)+200 ); /* don't ack too soon or hard lockup */
+#endif
+
 	/*
 	 * NOTE! We'd better ACK the irq immediately,
 	 * because timer handling can be slow.


b) I was also disappointed to see I could not have irq0 timer IO-APIC-edge. 
So I have fixed it too (tested on both my epox and albatron MOBOs).
Firstly I found 8254 connected directly to pin 0 not pin 2 of io-apic.
I have modified check_timer() in io_apic.c to trial connect pin and test for it
after the existing test for connection to io-apic.
See attached diff file nforce2-io-apic.c-2.4.22 for details.

--- linux-2.4.22/arch/i386/kernel/io_apic.c	2003-08-25 21:44:39.000000000 +1000
+++ linux-2.4.22-rd/arch/i386/kernel/io_apic.c	2003-12-07 18:40:40.000000000 +1000
@@ -1614,9 +1614,44 @@
 			return;
 		}
 		clear_IO_APIC_pin(0, pin1);
-		printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC\n");
+		printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC pin%d\n",pin1);
 	}

+#ifdef CONFIG_ACPI_BOOT && CONFIG_X86_UP_IOAPIC
+	/* for nforce2 try vector 0 on pin0
+	 * Note the io_apic_set_pci_routing call disables the 8259 irq 0
+	 * so we must be connected directly to the 8254 timer if this works
+	 * Note2: this violates the above comment re Subtle but works!
+	 */
+	printk(KERN_INFO "..TIMER: Is timer irq0 connected to IOAPIC Pin0? ...\n");
+	if ( pin1 != -1 && nr_ioapics ) {
+		int saved_timer_ack = timer_ack;
+		/* next call also disables 8259 irq0 */
+		int result = io_apic_set_pci_routing ( 0, 0, 0, 0, 0);
+		/*
+		 * Ok, does IRQ0 through the IOAPIC work?
+		 */
+		unmask_IO_APIC_irq(0);
+		timer_ack = 0 ;
+		if (timer_irq_works()) {
+			if (nmi_watchdog == NMI_IO_APIC) {
+				disable_8259A_irq(0);
+				setup_nmi();
+				enable_8259A_irq(0);
+				check_nmi_watchdog();
+			}
+			printk(KERN_INFO "..TIMER: works OK on apic pin0 irq0\n" );
+			return;
+		}
+		/* failed */
+		timer_ack = saved_timer_ack;
+		clear_IO_APIC_pin(0, 0);
+		result = io_apic_set_pci_routing ( 0, pin1, 0, 0, 0);
+		printk(KERN_ERR "..MP-BIOS bug: 8254 timer not connected to IO-APIC Pin 0\n");
+	}
+#endif
+/* end new stuff for nforce2 */
+
 	printk(KERN_INFO "...trying to set up timer (IRQ0) through the 8259A ... ");
 	if (pin2 != -1) {
 		printk("\n..... (found pin %d) ...", pin2);

c) Finally during my fault finding I merged A.Martins patches for the nforce2 IDE driver.
I note that the nforce2 address setup timing bits are different to the AMD ones.
I have assumed the nforce2 address timings apply to nforce and nforce3 chipsets.
I could be wrong so if someone with the nvidia docs could check it please.
I have also not tested it with anything but a WDC ata100 hard drive.
For info see attached patch files (I think pci ids are already in 2.4.23)
nforce2-amd74xx.c-2.4.22.patch, nforce2-amd74xx.h-2.4.22.patch, nforce2-pci_ids.h-2.4.22.patch

Thanks
Ross Dickson

[-- Attachment #2: ross-diffs.tar.bz2 --]
[-- Type: application/x-tbz, Size: 4375 bytes --]