linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Re: Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered
@ 2003-12-07 19:58 Ian Kumlien
  2003-12-07 20:59 ` Jesse Allen
  2003-12-08  2:07 ` Ross Dickson
  0 siblings, 2 replies; 68+ messages in thread
From: Ian Kumlien @ 2003-12-07 19:58 UTC (permalink / raw)
  To: linux-kernel; +Cc: ross

[-- Attachment #1: Type: text/plain, Size: 1985 bytes --]

> I have monitored list and know my nforce2 experiences have been
> common.

Hell yeah =)

> When I enabled either apic or io-apic in kern config, lockups came 
> hard and fast. Particularly bad under hard disk load. Heaps of lost 
> ints on irq7 in apic and ioapic mode. Lockups disappeared when I 
> lowered the ide hda udma speed to mode 3 with hdparm so I went looking
> for answers which now follow.

Good job =)

> There are three parts to this email.
> a) apic mods.
> Lockups are due to too fast an apic acknowledge of apic timer int.
> Apic hard locked up the system - no nmi debug available.
> Fixed it by introducing a delay of at least 500ns into 
> smp_apic_timer_interrupt() just prior to ack_APIC_irq().

I find this really odd... It works just fine... 
As did disabling whats now active ie:
'Halt Disconnect and Stop Grant Disconnect' bit is enabled.

So it seems like these are the two most important factors, at least from
where i stand. Both enabled me to actually use my machine with IO-APIC.
(1, disabling Halt Disconnect and Stop Grand Disconnect bit or 2, Add a
delay on the irq ack.)
Anyone that has any clues?

> b) io-apic mods
> So I have fixed it too (tested on both my epox and albatron MOBOs).
> Firstly I found 8254 connected directly to pin 0 not pin 2 of io-apic.
> I have modified check_timer() in io_apic.c to trial connect pin and 
> test for it after the existing test for connection to io-apic.

Good job, i wonder if it could be more generalized and integrated with
the rest of the code (i haven't even checked the rest of the code, but
this seemed separated).

One thing though, I get a lot more NMI's now than with nmi_watchdog=2...
NMI:      85520
LOC:      85477

I usually had a 3 figure number by now... but.. =)

> c) ide driver mods

Cool.. 

I applied all patches and it survived my grep test so i think it works.

-- 
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

^ permalink raw reply	[flat|nested] 68+ messages in thread
* Re: Catching NForce2 lockup with NMI watchdog
@ 2003-12-17 18:14 Ross Dickson
  2003-12-17 21:41 ` George Anzinger
                   ` (2 more replies)
  0 siblings, 3 replies; 68+ messages in thread
From: Ross Dickson @ 2003-12-17 18:14 UTC (permalink / raw)
  To: Maciej W. Rozycki, george; +Cc: linux-kernel

>On Tue, 16 Dec 2003, George Anzinger wrote: 
 


>> How confusing :( Could you give me some idea how this works? I have tried 
> > disable_irq(0) and, as best as I can tell, it does not do the trick. The 
> > confusion I have is understanding where in the chain of hardware each of these 
> > thing is taking place. 

Here is where to find Intel's MP arch spec Maceij mentions.
I had to find it recently wrt nforce2 issues

http://www.intel.com/design/pentium/datashts/24201606.pdf

Section 3.6.1 Apic Architecture is relevant
particularly
Section 3.6.2.2 Virtual Wire Mode

<snip>
great diagram!

<snip>
> If the above variant does not work, as a last resort, the path for the 
> 8254 timer interrupt is via the 8259 reconfigured back into its usual mode 
> and then LINT0 of the BSP reconfigured for an ExtINTA APIC interrupt. 
> Additionally, since at this point the glue logic has probably already 
> locked up due to the messing done above, a few artiffical sets of double 
> INTA cycles are sent to the system bus using the RTC chip and INTIN8 
> reconfigured temporarily to send ExtINTA APIC interrupts via the 
> inter-APIC bus. 
 
> I do hope a thorough read of the description will make the available 
> variants clear. The I/O APIC input numbers may differ but so far they are 
> almost always as noted above. 
 
>  Maciej 

All good.

I would like to add a footnote to highlight a potential gotcha as I understand it.

To clarify, the xt pic 8259A does not in itself have a transparent mode as would
a logic buffer or inverter. It always needs inta cycles to function. In PIC mode
it is wired to processor pins as per old 8086 and original cpu architecture
provides the inta cycles to it (bypasses apic, apic seems off).

In virtual wire mode with the 8259A output wired either to a local apic pin on cpu
or through the io-apic. In this mode it is the local apic which has to provide the 
inta cycles on the bus back to the 8259A for it to function correctly. 

The delivery mode has to be set to ExtInt for the register associated with the pin
that the 8259A output (int on Maceij diagram) is connected to. This is the only
way to force the apic to deliver the inta cycles to the 8259A and that is how it
appears transparent to the system. Spec says there can only be one source 
register (local apic) or redirection register (ioapic) of mode ExtInt per system
regardless of how many local apic and io-apic pins it (int on Maceij diagram)
is connected to. 

Gotcha: If none are set to ExtInt then the 8259A will hang for lack of IntA 
cycles.

Section 7.5.11 covers it
24319202.pdf available here

http://www.intel.com/design/pentiumii/manuals/243192.htm

Why only one Extint source in virtual wire mode?:

The 8259A in X86 architecture systems needs two inta cycles per interrupt event.
Do not confuse them with the EOI which is software, the inta is purely hardware.
It only works properly with one source causing inta cycles. Docs I have do not
say what happens with more than one source.

How 8259A works in a nutshell (it is more complex in cascade mode).

First the 8259A gets a request from H/ware and if unmasked etc generates its int 
(int on Maceij diagram) out.  8259A then sits there waiting for Inta from cpu 
(PIC mode) or local apic (Virtual wire mode). When the inta arrives the 8259A
latches its internal ISR bit and waits for second inta. When second inta arrives
it outputs a vector onto the data bus indicating which ISR bit was set. 

If the request from H/ware is still active when the first inta arrives then we get
the correct vector number.

If it is NOT still exerted then its tough luck and the vector we get 7 for the first
8259A or 15 for the second 8259A and it is too late to try and find out where
the real source was, hence the spurious irq7 messages and corresponding
irq 7 count increase.

It is pretty bad when the apic system that is handling the 8259A in virtual
wire mode cannot get the inta to the 8259A in time while the int request
hardware is still exerting but it happens.

I certainly agree with Marceij's comments that mixed mode of having 8254 PIT
routed via the 8259A was never meant to occur alongside ioapic handling of
the other interrupts. It is very problematic not to mention confusing. 

I do not know how smoothly the apic handles the 8259A if you would be turning
that source on and off frequently.

Regards
Ross Dickson




^ permalink raw reply	[flat|nested] 68+ messages in thread
* Re: Catching NForce2 lockup with NMI watchdog
@ 2003-12-13  3:56 Ross Dickson
  2003-12-15 13:16 ` Maciej W. Rozycki
  0 siblings, 1 reply; 68+ messages in thread
From: Ross Dickson @ 2003-12-13  3:56 UTC (permalink / raw)
  To: george; +Cc: Maciej W. Rozycki, linux-kernel

>Having had cause to try and figure out all this, I vote for the following being 
> included in the source somewhere... 
>-g 

Please consider adding

2c. Alternatively the OUT0 output of the 8254 PIT (IOW the timer source) may be 
directly connected to the INTIN0 input of the first I/O APIC. 

which we have found for nforce2 boards.
ref:

http://linux.derkeiler.com/Mailing-Lists/Kernel/2003-12/2375.html

Ross Dickson


>bill davidsen wrote: 
> > In article <Pine.LNX.4.55.0312101421540.31543@jurand.ds.pg.gda.pl>, 
> > Maciej W. Rozycki <macro@ds2.pg.gda.pl> wrote: 
> > 
> > | The I/O APIC NMI watchdog utilizes the property of being transparent to a 
> > | single IRQ source of a specially reconfigured 8259A PIC (the master one in 
> > | the IA32 PC architecture). There are more prerequisites that have to be 
> > | met and all indeed are for a 100% compatible PC as specified by the 
> > | Intel's Multiprocessor Specification. 
> > | 
> > | 1. The INT output of the master 8259A PIC has to be connected to the LINT0 
> > | (or LINTIN0; the name varies by implementations) inputs of all local APICs 
> > | in the system. 
> > | 
> > | 2a. The OUT0 output of the 8254 PIT (IOW the timer source) has to be 
> > | directly connected to the INTIN2 input of the first I/O APIC. 
> > | 
> > | 2b. Alternatively the INT output of the master 8259A PIC has to be 
> > | connected to the INTIN0 input of the first I/O APIC. 
> > | 
> > | 3. There must be no glue logic that would change logical properties of the 
> > | signal between the INT output of the master 8259A PIC and the respective 
> > | APIC interrupt inputs. 
> > | 
> > | In practice, assuming the MP IRQ routing information provided the BIOS has 
> > | been correct (which is not always the case), prerequisites #1 and #2 have 
> > | been met so far, but #3 has proved to be occasionally problematic. 
> > 
> > In practice many system seem to take a good bit of guessing and testing. 
> > I have an old P-II which only works with acpi=force and nmi_watchdog=2, 
> > for instance. 
> > 
> > It would be nice if there were a program which could poke at the 
> > hardware and suggest options which might work, as in eliminating the 
> > ones which can be determined not to work. Absent that trial and error 
> > rule, unfortunately. 
 

^ permalink raw reply	[flat|nested] 68+ messages in thread
* RE: Catching NForce2 lockup with NMI watchdog
@ 2003-12-05 22:41 b
  0 siblings, 0 replies; 68+ messages in thread
From: b @ 2003-12-05 22:41 UTC (permalink / raw)
  To: mfedyk; +Cc: linux-kernel

 >Everyone with this problem should turn on the nmi_watchdog, as
 >someone may
 >have the right circumstances to produce an oops where the
 >others didn't.
 >
 >I say that you're not serious about getting this fixed unless
 >you're going
 >to do all of:

To quote Allen Martin:

 >NVIDIA doesn't provide a windows driver to setup APIC interrupts.
 >
 >APIC functionality is exported through the ACPI methods and MP
 >table in the system BIOS which the motherboard vendors supply.

 >Likely the root of the problem has to do with the way the Linux
 >kernel is using the ACPI methods to setup the interrupts which
 >is different from win 9x/2k/XP.  I can help track this down,
 >unfortunately so far I've been unable to reproduce the hangs
 >on any of the boards I have.

and

 >> Do you know whether the nforce2's with apic support the timer
 >> (IRQ 0) in
 >> IO-APIC mode?  To me, it seems like a bug:
 >> "Dec  4 20:13:11 tesore kernel: ..MP-BIOS bug: 8254 timer not
 >> connected to
 >> IO-APIC"
 >> (This message originates in arch/i386/kernel/io_apic.c)
 >>
 >
 >Yes, Win 9x/2k/XP use the system timer on irq0 and have no problem.  I
 >haven't looked at this yet.
 >

Is it not possible that Linux could be made to handling this hardware
correctly?


 >
 > o turn on nmi_watchdog
 > o try the patches posted[1]
 > o contact nvidia or your motherboard manufacturer saying you
 >need linux
 >   support, and return the board if they don't. (phone, fax,
 >email, or even
 >   local office if there is one)
 >
 >I bought a VIA board to avoid the problems I expected from the
 >nforce, and I
 >needed a system (server) that would *work* now.
 >
 >[1] If you're worried about your file system, just boot the
 >patched kernel in
 >single mode, and that will mount all of your file systems
 >read-only so there
 >will be little chance of corruption.
 >
 >Mike


^ permalink raw reply	[flat|nested] 68+ messages in thread
* RE: Catching NForce2 lockup with NMI watchdog
@ 2003-12-05 20:56 Allen Martin
  0 siblings, 0 replies; 68+ messages in thread
From: Allen Martin @ 2003-12-05 20:56 UTC (permalink / raw)
  To: 'Jesse Allen'; +Cc: linux-kernel

> -----Original Message-----
> From: Jesse Allen [mailto:the3dfxdude@hotmail.com] 
> Sent: Friday, December 05, 2003 12:36 PM
>
> Do you know whether the nforce2's with apic support the timer 
> (IRQ 0) in 
> IO-APIC mode?  To me, it seems like a bug:
> "Dec  4 20:13:11 tesore kernel: ..MP-BIOS bug: 8254 timer not 
> connected to 
> IO-APIC"
> (This message originates in arch/i386/kernel/io_apic.c)
> 

Yes, Win 9x/2k/XP use the system timer on irq0 and have no problem.  I
haven't looked at this yet.

-Allen

^ permalink raw reply	[flat|nested] 68+ messages in thread
* RE: Catching NForce2 lockup with NMI watchdog
@ 2003-12-05 19:11 Allen Martin
  2003-12-05 20:18 ` cheuche+lkml
                   ` (2 more replies)
  0 siblings, 3 replies; 68+ messages in thread
From: Allen Martin @ 2003-12-05 19:11 UTC (permalink / raw)
  To: 'Mikael Pettersson', Josh McKinney; +Cc: linux-kernel

> -----Original Message-----
> From: Mikael Pettersson [mailto:mikpe@csd.uu.se] 
> Sent: Friday, December 05, 2003 4:15 AM
>
>  > So does this confirm that the lockups with nforce2 
> chipsets and apic
>  > is actually a hardware problem after all? 
> 
> Confirm with very high probability. There may be quirks in nVidia's
> chipset that we (unlike their Windoze drivers) don't know about.
> 
> Ask nVidia for detailed chipset documentation. Then maybe we 
> can fix this.

NVIDIA doesn't provide a windows driver to setup APIC interrupts.  APIC
functionality is exported through the ACPI methods and MP table in the
system BIOS which the motherboard vendors supply.

Likely the root of the problem has to do with the way the Linux kernel is
using the ACPI methods to setup the interrupts which is different from win
9x/2k/XP.  I can help track this down, unfortunately so far I've been unable
to reproduce the hangs on any of the boards I have.

-Allen

^ permalink raw reply	[flat|nested] 68+ messages in thread
* Catching NForce2 lockup with NMI watchdog
@ 2003-12-05  4:54 Jesse Allen
  2003-12-05  7:40 ` Mikael Pettersson
  0 siblings, 1 reply; 68+ messages in thread
From: Jesse Allen @ 2003-12-05  4:54 UTC (permalink / raw)
  To: linux-kernel

Hi,

I have a NForce2 board and can easily reproduce a lockup with grep on an IDE 
hard disk at UDMA 100.  The lockup occurs when both Local APIC + IO-APIC are 
enabled.  It was suggested to me to use NMI watchdog to catch it.  However, the 
NMI watchdog doesn't seem to work.

When I set the kernel parameter "nmi_watchdog=1" I get this message in 
/var/log/syslog:
Dec  4 20:10:30 tesore kernel: ..MP-BIOS bug: 8254 timer not connected to 
IO-APIC
Dec  4 20:10:30 tesore kernel: timer doesn't work through the IO-APIC - 
disabling NMI Watchdog!

"nmi_watchdog=2" seems to work at first, In /var/log/messages:
Dec  4 20:13:11 tesore kernel: testing NMI watchdog ... OK.
but it still locks up.

I have the complete logs when running with nmi_watchdog, kernel config, and more
here:
http://www.chez.com/alors/nforce-lockup-logs.tar.gz

If you have any ideas please give them =)

Jesse

^ permalink raw reply	[flat|nested] 68+ messages in thread

end of thread, other threads:[~2003-12-19 15:35 UTC | newest]

Thread overview: 68+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-12-07 19:58 Fixes for nforce2 hard lockup, apic, io-apic, udma133 covered Ian Kumlien
2003-12-07 20:59 ` Jesse Allen
2003-12-07 20:56   ` Ian Kumlien
2003-12-08  2:07 ` Ross Dickson
2003-12-08  2:23   ` Ian Kumlien
2003-12-09 18:12   ` Catching NForce2 lockup with NMI watchdog Ian Kumlien
2003-12-09 22:04     ` Craig Bradney
2003-12-09 23:13       ` Ian Kumlien
2003-12-10  1:20       ` NForce2 and AMD, disconnect Ian Kumlien
2003-12-10  6:14       ` Catching NForce2 lockup with NMI watchdog Bob
2003-12-10  7:51         ` Craig Bradney
  -- strict thread matches above, loose matches on Subject: below --
2003-12-17 18:14 Ross Dickson
2003-12-17 21:41 ` George Anzinger
2003-12-17 21:48 ` George Anzinger
2003-12-18  1:30   ` Ross Dickson
2003-12-18 14:32     ` Maciej W. Rozycki
2003-12-19  4:17       ` Ross Dickson
2003-12-19 15:35         ` Maciej W. Rozycki
2003-12-18 14:04 ` Maciej W. Rozycki
2003-12-18 14:22   ` Craig Bradney
2003-12-19  5:38     ` Ross Dickson
2003-12-19 10:36       ` Craig Bradney
2003-12-19  4:06   ` Ross Dickson
2003-12-19 15:33     ` Maciej W. Rozycki
2003-12-13  3:56 Ross Dickson
2003-12-15 13:16 ` Maciej W. Rozycki
2003-12-05 22:41 b
2003-12-05 20:56 Allen Martin
2003-12-05 19:11 Allen Martin
2003-12-05 20:18 ` cheuche+lkml
2003-12-05 20:34   ` Prakash K. Cheemplavam
2003-12-05 21:02     ` Mike Fedyk
2003-12-05 20:55   ` Jesse Allen
2003-12-06  3:20   ` Jesse Allen
2003-12-05 20:36 ` Jesse Allen
2003-12-05 22:55 ` Mike Fedyk
2003-12-05 23:11   ` Craig Bradney
2003-12-05  4:54 Jesse Allen
2003-12-05  7:40 ` Mikael Pettersson
2003-12-05  8:33   ` Josh McKinney
2003-12-05 12:14     ` Mikael Pettersson
2003-12-05 14:19       ` Craig Bradney
2003-12-05 17:05         ` Craig Bradney
2003-12-05 18:11         ` Josh McKinney
2003-12-05  8:58   ` Mike Fedyk
2003-12-05 12:06     ` Mikael Pettersson
2003-12-08  2:20     ` Bob
2003-12-09 14:21       ` Maciej W. Rozycki
2003-12-09 16:35         ` Bob
2003-12-10 13:41           ` Maciej W. Rozycki
2003-12-12 16:01             ` bill davidsen
2003-12-12 16:47               ` Maciej W. Rozycki
2003-12-12 16:57                 ` Richard B. Johnson
2003-12-12 17:21                   ` Maciej W. Rozycki
2003-12-13  5:16                 ` Bill Davidsen
2003-12-15 13:23                   ` Maciej W. Rozycki
2003-12-12 22:27               ` George Anzinger
2003-12-15 13:13                 ` Maciej W. Rozycki
2003-12-15 21:42                   ` George Anzinger
2003-12-16 13:37                     ` Maciej W. Rozycki
2003-12-16 13:57                       ` Richard B. Johnson
2003-12-16 15:47                         ` Maciej W. Rozycki
2003-12-16 16:44                           ` Richard B. Johnson
2003-12-16 16:50                             ` Maciej W. Rozycki
2003-12-16 17:26                       ` George Anzinger
2003-12-16 20:54                         ` Maciej W. Rozycki
2003-12-16 21:53                           ` George Anzinger
2003-12-17 14:03                             ` Maciej W. Rozycki

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).