Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Karsten Wiese]

Hi Ingo,

suffering (not really ;-) double-rated IO-APIC level-interrupts I found the following patch as a solution:

--- arch/i386/kernel/io_apic.c~	2005-06-28 19:07:49.000000000 +0200
+++ arch/i386/kernel/io_apic.c	2005-06-28 19:07:49.000000000 +0200
@@ -1959,6 +1959,7 @@
 static void mask_and_ack_level_ioapic_irq(unsigned int irq)
 {
 	move_irq(irq);
+	mask_IO_APIC_irq(irq);
 	ack_APIC_irq();
 }
 
--- kernel/irq/handle.c~	2005-06-28 19:19:32.000000000 +0200
+++ kernel/irq/handle.c	2005-06-28 19:19:32.000000000 +0200
@@ -214,7 +214,6 @@
 	 * hardirq redirection to the irqd process context:
 	 */
 	if (redirect_hardirq(desc)) {
-		desc->handler->disable(irq);
 		goto out_no_end;
 	}

it works here on a PREEMPT_RT 2.6.12-RT-V0.7.50-29 base.
Level-interrupts are sanely rated again.
Didn't check, if the patch breaks XT-PIC mode, which works ok without the patch.
Mainline shows the same effect here (VIA K8T800 UP), didn't dig there yet.

thanks,
Karsten

	

	
		
___________________________________________________________ 
Gesendet von Yahoo! Mail - Jetzt mit 1GB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* Karsten Wiese <annabellesgarden@yahoo.de> wrote:

> Hi Ingo,
> 
> suffering (not really ;-) double-rated IO-APIC level-interrupts I 
> found the following patch as a solution:

thanks. I've applied your patch but also tweaked this area a bit, to 
make the i8259A PIC work too. I've uploaded the -31 patch with these 
fixes included.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Karsten Wiese <annabellesgarden@yahoo.de> wrote:
> 
> > Hi Ingo,
> > 
> > suffering (not really ;-) double-rated IO-APIC level-interrupts I 
> > found the following patch as a solution:
> 
> thanks. I've applied your patch but also tweaked this area a bit, to 
> make the i8259A PIC work too. I've uploaded the -31 patch with these 
> fixes included.

make that -50-32, had a leftover hack in io_apic.c.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Karsten Wiese]

Am Dienstag, 28. Juni 2005 22:30 schrieb Ingo Molnar:
> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > 
> > * Karsten Wiese <annabellesgarden@yahoo.de> wrote:
> > 
> > > Hi Ingo,
> > > 
> > > suffering (not really ;-) double-rated IO-APIC level-interrupts I 
> > > found the following patch as a solution:
> > 
> > thanks. I've applied your patch but also tweaked this area a bit, to 
> > make the i8259A PIC work too. I've uploaded the -31 patch with these 
> > fixes included.
> 
> make that -50-32, had a leftover hack in io_apic.c.
> 
looked at -50-33 now and wonder why is mask_IO_APIC_irq() called twice
from  __do_IRQ()?
given a threaded interrupt:
__do_IRQ() calls desc->handler->ack(irq).
ack points to mask_and_ack_level_ioapic_irq(), which calls mask_IO_APIC_irq(irq).
some lines later in __do_IRQ() desc->handler->disable(irq) is called.
disable points to  mask_IO_APIC_irq(), now being called a 2nd time.
I think this 2nd call isn't necessary.
Is there a difference between masking an interrupt line and disabling it?
What am I missing?

Back at 2.6.12-rc5-RT-48-16 mask_and_ack_level_ioapic_irq() also contained the mask_IO_APIC_irq(irq)
call and level interrupt-rates where fine.
Some versions later it vanished there. Why was that?

Karsten



  

	

	
		
___________________________________________________________ 
Gesendet von Yahoo! Mail - Jetzt mit 1GB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* Karsten Wiese <annabellesgarden@yahoo.de> wrote:

> looked at -50-33 now and wonder why is mask_IO_APIC_irq() called twice 
> from __do_IRQ()? given a threaded interrupt: __do_IRQ() calls 
> desc->handler->ack(irq). ack points to 
> mask_and_ack_level_ioapic_irq(), which calls mask_IO_APIC_irq(irq).  
> some lines later in __do_IRQ() desc->handler->disable(irq) is called.  
> disable points to mask_IO_APIC_irq(), now being called a 2nd time. I 
> think this 2nd call isn't necessary. Is there a difference between 
> masking an interrupt line and disabling it? What am I missing?

you are not missing anything - but i found no easy way for the time 
being to get rid of the second masking.

> Back at 2.6.12-rc5-RT-48-16 mask_and_ack_level_ioapic_irq() also 
> contained the mask_IO_APIC_irq(irq) call and level interrupt-rates 
> where fine. Some versions later it vanished there. Why was that?

i reorganized how redirection is being done, and i've implemented 
auto-ACK for the i8259A, to reduce IRQ handling costs. One goal was to 
avoid the masking of the interrupt line for the timer interrupt on 
i8259A - but i think i'm going to revert that, it's causing too many 
problems all around.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> [...] but i think i'm going to revert that, it's causing too many 
> problems all around.

reverted it and this enabled the removal of the extra ->disable() you 
noticed - this should further speed up the IOAPIC code. These changes 
are in the -50-34 kernel i just uploaded.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[William Weston]

On Wed, 29 Jun 2005, Ingo Molnar wrote:

> > [...] but i think i'm going to revert that, it's causing too many 
> > problems all around.
> 
> reverted it and this enabled the removal of the extra ->disable() you 
> noticed - this should further speed up the IOAPIC code. These changes 
> are in the -50-34 kernel i just uploaded.

-50-34 fixed the wakeup latency regression I was seeing on my Athlon box
with -50-33, and seems a bit more responsive than -50-25.  Max wakeup
latency is back down to 14us (from 39us), even while running JACK (xrun
free) and two instances of burnK7.  Overall system response is probably
the best I've seen with the RT kernels ;-}

--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* William Weston <weston@sysex.net> wrote:

> On Wed, 29 Jun 2005, Ingo Molnar wrote:
> 
> > > [...] but i think i'm going to revert that, it's causing too many 
> > > problems all around.
> > 
> > reverted it and this enabled the removal of the extra ->disable() you 
> > noticed - this should further speed up the IOAPIC code. These changes 
> > are in the -50-34 kernel i just uploaded.
> 
> -50-34 fixed the wakeup latency regression I was seeing on my Athlon 
> box with -50-33, and seems a bit more responsive than -50-25.  Max 
> wakeup latency is back down to 14us (from 39us), even while running 
> JACK (xrun free) and two instances of burnK7.  Overall system response 
> is probably the best I've seen with the RT kernels ;-}

great! The SMP box running BurnP6 is another system, right? Could you 
sum up the remaining regressions you are seeing under -RT? (the 
latency.c warning is one, what others are remaining?)

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Karsten Wiese]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Am Mittwoch, 29. Juni 2005 11:15 schrieb William Weston:
> On Wed, 29 Jun 2005, Ingo Molnar wrote:
> 
> > > [...] but i think i'm going to revert that, it's causing too many 
> > > problems all around.
> > 
> > reverted it and this enabled the removal of the extra ->disable() you 
> > noticed - this should further speed up the IOAPIC code. These changes 
> > are in the -50-34 kernel i just uploaded.
> 
> -50-34 fixed the wakeup latency regression I was seeing on my Athlon box
> with -50-33, and seems a bit more responsive than -50-25.  Max wakeup
> latency is back down to 14us (from 39us), even while running JACK (xrun
> free) and two instances of burnK7.  Overall system response is probably
> the best I've seen with the RT kernels ;-}
> 
attached patch for io_apic.c lets
1. gcc 3.4.3 optimize io_apic access a little better.
2. CONFIG_X86_UP_IOAPIC_FAST work here.
   Didn't check, if it really speeds up things.

Karsten




[-- Attachment #2: io_apic-RT-50-35.diff --]
[-- Type: text/x-diff, Size: 3593 bytes --]

--- arch/i386/kernel/io_apic.c-50-33	2005-06-29 16:27:40.000000000 +0200
+++ arch/i386/kernel/io_apic.c	2005-06-29 16:27:40.000000000 +0200
@@ -138,14 +138,22 @@
 /*
  * Cache register values:
  */
-static unsigned int io_apic_cache[MAX_IO_APICS][MAX_IOAPIC_CACHE]
+static struct {
+	unsigned int reg;
+	unsigned int val[MAX_IOAPIC_CACHE];
+} io_apic_cache[MAX_IO_APICS]
 		____cacheline_aligned_in_smp;
 #endif
 
 inline unsigned int __raw_io_apic_read(unsigned int apic, unsigned int reg)
 {
-	*IO_APIC_BASE(apic) = reg;
-	return *(IO_APIC_BASE(apic)+4);
+	volatile unsigned int *io_apic;
+	io_apic = IO_APIC_BASE(apic);
+#ifdef IOAPIC_CACHE
+	io_apic_cache[apic].reg = reg;
+#endif
+	io_apic[0] = reg;
+	return io_apic[4];
 }
 
 unsigned int raw_io_apic_read(unsigned int apic, unsigned int reg)
@@ -153,7 +161,7 @@
 	unsigned int val = __raw_io_apic_read(apic, reg);
 
 #ifdef IOAPIC_CACHE
-	io_apic_cache[apic][reg] = val;
+	io_apic_cache[apic].val[reg] = val;
 #endif
 	return val;
 }
@@ -172,14 +180,17 @@
 		}
 		return __raw_io_apic_read(apic, reg);
 	}
-	if (io_apic_cache[apic][reg] && !sis_apic_bug)
-		return io_apic_cache[apic][reg];
+	if (io_apic_cache[apic].val[reg] && !sis_apic_bug) {
+		io_apic_cache[apic].reg = -1;
+		return io_apic_cache[apic].val[reg];
+	}
 #endif
 	return raw_io_apic_read(apic, reg);
 }
 
 void io_apic_write(unsigned int apic, unsigned int reg, unsigned int val)
 {
+	volatile unsigned int *io_apic;
 #ifdef IOAPIC_CACHE
 	if (unlikely(reg >= MAX_IOAPIC_CACHE)) {
 		static int once = 1;
@@ -191,10 +202,14 @@
 			dump_stack();
 		}
 	} else
-		io_apic_cache[apic][reg] = val;
+		io_apic_cache[apic].val[reg] = val;
 #endif
-	*IO_APIC_BASE(apic) = reg;
-	*(IO_APIC_BASE(apic)+4) = val;
+	io_apic = IO_APIC_BASE(apic);
+#ifdef IOAPIC_CACHE
+	io_apic_cache[apic].reg = reg;
+#endif
+	io_apic[0] = reg;
+	io_apic[4] = val;
 }
 
 /*
@@ -214,34 +229,42 @@
  */
 void io_apic_modify(unsigned int apic, unsigned int reg, unsigned int val)
 {
+	volatile unsigned int *io_apic;
 #ifdef IOAPIC_CACHE
-	io_apic_cache[apic][reg] = val;
+	io_apic_cache[apic].val[reg] = val;
 #endif
-	if (unlikely(sis_apic_bug))
-		*IO_APIC_BASE(apic) = reg;
-	*(IO_APIC_BASE(apic)+4) = val;
+	io_apic = IO_APIC_BASE(apic);
+#ifdef IOAPIC_CACHE
+	if (io_apic_cache[apic].reg != reg || sis_apic_bug) {
+		io_apic_cache[apic].reg = reg;
+#else
+	if (unlikely(sis_apic_bug)) {
+#endif
+		io_apic[0] = reg;
+	}
+	io_apic[4] = val;
 #ifndef IOAPIC_POSTFLUSH
 	if (unlikely(sis_apic_bug))
 #endif
 		/*
 		 * Force POST flush by reading:
  		 */
-		val = *(IO_APIC_BASE(apic)+4);
+		val = io_apic[4];
 }
 
 static void __modify_IO_APIC_irq (unsigned int irq, unsigned long enable, unsigned long disable)
 {
 	struct irq_pin_list *entry = irq_2_pin + irq;
-	unsigned int pin, reg;
+	unsigned int pin, val;
 
 	for (;;) {
 		pin = entry->pin;
 		if (pin == -1)
 			break;
-		reg = io_apic_read(entry->apic, 0x10 + pin*2);
-		reg &= ~disable;
-		reg |= enable;
-		io_apic_modify(entry->apic, 0x10 + pin*2, reg);
+		val = io_apic_read(entry->apic, 0x10 + pin*2);
+		val &= ~disable;
+		val |= enable;
+		io_apic_modify(entry->apic, 0x10 + pin*2, val);
 		if (!entry->next)
 			break;
 		entry = irq_2_pin + entry->next;
@@ -306,9 +329,13 @@
 {
 	int apic, pin;
 
-	for (apic = 0; apic < nr_ioapics; apic++)
+	for (apic = 0; apic < nr_ioapics; apic++) {
+#ifdef IOAPIC_CACHE
+		io_apic_cache[apic].reg = -1;
+#endif
 		for (pin = 0; pin < nr_ioapic_registers[apic]; pin++)
 			clear_IO_APIC_pin(apic, pin);
+	}
 }
 
 static void set_ioapic_affinity_irq(unsigned int irq, cpumask_t cpumask)

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* Karsten Wiese <annabellesgarden@yahoo.de> wrote:

> attached patch for io_apic.c lets
> 1. gcc 3.4.3 optimize io_apic access a little better.
> 2. CONFIG_X86_UP_IOAPIC_FAST work here.
>    Didn't check, if it really speeds up things.

which change made CONFIG_X86_UP_IOAPIC_FAST work on your box? It seems 
you've changed the per-register frontside read-cache to something else - 
was that on purpose?

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Karsten Wiese]

Am Mittwoch, 29. Juni 2005 21:38 schrieb Ingo Molnar:
> 
> * Karsten Wiese <annabellesgarden@yahoo.de> wrote:
> 
> > attached patch for io_apic.c lets
> > 1. gcc 3.4.3 optimize io_apic access a little better.
> > 2. CONFIG_X86_UP_IOAPIC_FAST work here.
> >    Didn't check, if it really speeds up things.
> 
> which change made CONFIG_X86_UP_IOAPIC_FAST work on your box? It seems 
> you've changed the per-register frontside read-cache to something else - 
> was that on purpose?
> 
CONFIG_X86_UP_IOAPIC_FAST started working here, when I made io_apic_modify()
look like that:

void io_apic_modify(unsigned int apic, unsigned int reg, unsigned int val)
{
#ifdef IOAPIC_CACHE
	io_apic_cache[apic][reg] = val;
#endif
//	if (unlikely(sis_apic_bug))
commented this ^^ out 

		*IO_APIC_BASE(apic) = reg;
	*(IO_APIC_BASE(apic)+4) = val;
#ifndef IOAPIC_POSTFLUSH
	if (unlikely(sis_apic_bug))
#endif
		/*
		 * Force POST flush by reading:
 		 */
		val = *(IO_APIC_BASE(apic)+4);
}

This change does it, 'cause when we read a cached value instead of from the ioapic,
the ioapic's address register isn't set and thus the following write in io_apic_modify
might not scratch the right ioapic register.

On top of the above the patch adds
	unsigned int reg;
to io_apic_cache.
with that "reg" struct member a mark -1 can be set, when we read from cache,
or the reg-number, when we read from the ioapic.
Then by comparing parameter reg with io_apic_cache[apic].reg
the patched io_apic_modify() knows, if it has to set the address register or not.
The register caching in the patch should be the same as before,
only the cache changed from a 2 dimensional array
to an array holding structs, which contain an array:
io_apic_cache[apic][reg] should be equivalent to io_apic_cache[apic].val[reg], no?

Karsten




	

	
		
___________________________________________________________ 
Gesendet von Yahoo! Mail - Jetzt mit 1GB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[William Weston]

On Wed, 29 Jun 2005, Ingo Molnar wrote:

> great! The SMP box running BurnP6 is another system, right? Could you 
> sum up the remaining regressions you are seeing under -RT? (the 
> latency.c warning is one, what others are remaining?)

Right.  I'm doing the bulk of my RT testing on three machines:

Home:  UP  Athlon  2GHz    VIA KT400   desktop, audio & MIDI dev
Work:  UP  Xeon    1.8GHz  Intel 845   video encoding & mcast streaming
Work:  SMT Xeon/HT 3.2GHz  Intel 865   desktop, video streaming testing

Here's the max wakeup latencies I'm seeing on each machine with -50-35:

Athlon   15us 
Xeon     22us
Xeon/HT  290us

In all fairness, the Xeon/HT machine is still running with the debug and 
trace options enabled, while the other two have all debug except wakeup 
latency timing disabled.

On UP, everything looks good.  No JACK xruns, even while running burnK7.  
Streaming with VLC from a Hauppauge card to UDP/multicast is almost
flawless (usually takes at least a day before any UDP packets are sent out
too late).  I only saw one warning in my logs on the Athlon box for the
whole -50 series, and that was in -50-15.  The non-HT Xeon box hasn't seen
any bug warnings since plist_init() on -48-33.

On SMT, everything works well under normal desktop load (X, wmaker, 10
dockapps, 10-20 xterms, and firefox).  VLC runs very smoothly when I'm not
stress-testing.  System responsiveness is much better than the 
vanilla kernels.  Wakeup latencies have decreased since the -48 series
(generally <100us, spiking to <300us now instead of generally <200us
spiking to <1000us).

When it comes to stress testing, I'm still finding conditions where CPU
hungry processes (like burnP6) can make the box completely unresponsive.  
So far, I've narrowed it down to the following conditions:  X is running
(xorg-6.8.2), along with two burnP6 instances, and X apps that are
actively updating the screen (like VLC or a collection of wmaker
dockapps).  If one of these conditions is absent, there's no meltdown.

I've been toying with a script to fire up two instances of burnP6, grab
some traces with your trace-it code, and kill off those burnP6 processes.  
The script generally takes the form:

#!/bin/bash
trace-it > trace1.out
sync
burn &
usleep 100000
trace-it > trace2.out
sync
burn &
usleep 100000
trace-it > trace2.out
sync
killall burn

I copied 'burnP6' to 'burn' so that the process name shows up in the 
traces.  Every time system response disappears, I never get any traces 
after the second burnP6 fires up.  Most of the time, I can still move the 
mouse pointer for a few minutes (with serio lost synchro warnings on the 
serial console) before it goes dead, too.  Keyboard is almost always lost.

The results of two tests (one resulting in a meltdown) are located at:

http://sysex.net/testing/2.6.12-RT-V0.7.50-35

The 'test' directory has some traces with X and burnP6 (no VLC or
dockapps)  running.  These traces have chunks of dead time ranging from
roughly 500 to 900 us.

The 'crash' directory has a trace with X, burnP6, and VLC running, and a
bug warning.  As soon as the second burnP6 instance started up, the system
showed no signs of being alive other than the mouse synchro warnings on
the serial console.

Overall, I would have to say that RT_PREEMPT is nearly ready for
primetime, especially for audio, MIDI, and video.  SMT needs some help
when it comes to doing anything that requires X.  I'm not sure about true
SMP, since I can't find a true SMP system to play with right now.

Great Work, Ingo!


Best Regards,
--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Karsten Wiese]

[-- Attachment #1: Type: text/plain, Size: 3389 bytes --]

Am Donnerstag, 30. Juni 2005 01:36 schrieb Karsten Wiese:
> Am Mittwoch, 29. Juni 2005 21:38 schrieb Ingo Molnar:
> > 
> > * Karsten Wiese <annabellesgarden@yahoo.de> wrote:
> > 
> > > attached patch for io_apic.c lets
> > > 1. gcc 3.4.3 optimize io_apic access a little better.
> > > 2. CONFIG_X86_UP_IOAPIC_FAST work here.
> > >    Didn't check, if it really speeds up things.
> > 
> > which change made CONFIG_X86_UP_IOAPIC_FAST work on your box? It seems 
> > you've changed the per-register frontside read-cache to something else - 
> > was that on purpose?
> > 
> CONFIG_X86_UP_IOAPIC_FAST started working here, when I made io_apic_modify()
> look like that:
> 
> void io_apic_modify(unsigned int apic, unsigned int reg, unsigned int val)
> {
> #ifdef IOAPIC_CACHE
> 	io_apic_cache[apic][reg] = val;
> #endif
> //	if (unlikely(sis_apic_bug))
> commented this ^^ out 
> 
> 		*IO_APIC_BASE(apic) = reg;
> 	*(IO_APIC_BASE(apic)+4) = val;
> #ifndef IOAPIC_POSTFLUSH
> 	if (unlikely(sis_apic_bug))
> #endif
> 		/*
> 		 * Force POST flush by reading:
>  		 */
> 		val = *(IO_APIC_BASE(apic)+4);
> }
> 
> This change does it, 'cause when we read a cached value instead of from the ioapic,
> the ioapic's address register isn't set and thus the following write in io_apic_modify
> might not scratch the right ioapic register.
> 
> On top of the above the patch adds
> 	unsigned int reg;
> to io_apic_cache.
> with that "reg" struct member a mark -1 can be set, when we read from cache,
> or the reg-number, when we read from the ioapic.
> Then by comparing parameter reg with io_apic_cache[apic].reg
> the patched io_apic_modify() knows, if it has to set the address register or not.
> The register caching in the patch should be the same as before,
> only the cache changed from a 2 dimensional array
> to an array holding structs, which contain an array:
> io_apic_cache[apic][reg] should be equivalent to io_apic_cache[apic].val[reg], no?
> 
Here come some numbers to back up the usefullness of CONFIG_X86_UP_IOAPIC_FAST.
(and to show that my patch actually works ;-))
All measurement where taken on an UP Athlon64 2Ghz running 32bit 2.6.12-RT-50-35 PREEMPT_RT
on a K8T800 mobo.

I measured the tsc-cycles spent in __do_IRQ() plus those used for interruptcontroller clean-up
after the irq-thread finished the handler.

Did that with 3 Setups:
 XT-PIC
 IO-APIC uncached
 IO-APIC cached
 IO-APIC cached + cpu-cache thrashing application hogging the cpu 

Characteristic values found:
 XT-PIC
  irq#       | cycles/irq
  -----------|------------
   0 timer   | 18733
   5 VIA8237 | 19089
  11 eth0    | 22676
  14 ide0    | 23057

 IO-APIC uncached
  irq#       | cycles/irq
  -----------|------------
   0 timer   |  4711
  22 VIA8237 | 32107
  16 eth0    | 32680
  14 ide0    | 13404

 IO-APIC cached
  irq#       | cycles/irq
  -----------|------------
   0 timer   |  5004
  22 VIA8237 |  1484
  16 eth0    |  1598
  14 ide0    |   905

 IO-APIC cached + cpu-cache thrashing application hogging the cpu 
  irq#       | cycles/irq
  -----------|------------
   0 timer   |  6344
  22 VIA8237 |  2512
  16 eth0    |  2774
  14 ide0    |  1723

Clear winner is IO-APIC cached.

The patch used + full /proc/interrupt logs are attached.
The patch differs from the previously sent one by precalculation 
of the ioapic's virtual address and the cycle-measurement code.

Cheers,
Karsten









[-- Attachment #2: ints_io_apic_cached_fullmemload --]
[-- Type: text/plain, Size: 810 bytes --]

           CPU0       
  0:    7390558    IO-APIC-edge      0/  558  6344 3540429, timer
  1:      14651    IO-APIC-edge      0/ 4651  1583 7364358, i8042
  8:         12    IO-APIC-edge      0/   12  1462 17550, rtc
  9:          0   IO-APIC-level      0/    0     0 0, acpi
 12:     120985    IO-APIC-edge      0/  985  1251 1232698, i8042
 14:     117323    IO-APIC-edge      0/ 7321  1723 12621315, ide0
 15:     328390    IO-APIC-edge      0/ 8388   619 5194067, ide1
 16:     458342   IO-APIC-level      0/ 8342  2774 23146219, eth0, nvidia
 21:          0   IO-APIC-level      0/    0     0 0, ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb4, uhci_hcd:usb5
 22:       7826   IO-APIC-level      0/ 7826  2512 19665441, VIA8237
NMI:          0 
LOC:    7390775 
ERR:          0
MIS:          0

[-- Attachment #3: ints_xt-pic_noload --]
[-- Type: text/plain, Size: 869 bytes --]

           CPU0       
  0:     622269          XT-PIC      0/ 2269 18733 42507219, timer
  1:        886          XT-PIC      0/  886 19145 16962916, i8042
  2:          0          XT-PIC      0/    0     0 0, cascade
  3:          0          XT-PIC      0/    0     0 0, uhci_hcd:usb4, uhci_hcd:usb5
  5:       6381          XT-PIC      0/ 6381 19089 121807578, VIA8237, ehci_hcd:usb1
  8:          1          XT-PIC      0/    1 23697 23697, rtc
  9:          0          XT-PIC      0/    0     0 0, acpi
 11:       1086          XT-PIC      0/ 1086 22676 24626747, uhci_hcd:usb2, uhci_hcd:usb3, eth0
 12:         93          XT-PIC      0/   93 22848 2124875, i8042
 14:       3393          XT-PIC      0/ 3392 23057 78211017, ide0
 15:       5410          XT-PIC      0/ 5409 22818 123423742, ide1
NMI:          0 
LOC:     622240 
ERR:          0
MIS:          0

[-- Attachment #4: ints_io_apic_cached_noload --]
[-- Type: text/plain, Size: 806 bytes --]

           CPU0       
  0:    7520343    IO-APIC-edge      0/  343  5004 1716578, timer
  1:      15028    IO-APIC-edge      0/ 5028  1569 7890363, i8042
  8:         12    IO-APIC-edge      0/   12  1462 17550, rtc
  9:          0   IO-APIC-level      0/    0     0 0, acpi
 12:     122185    IO-APIC-edge      0/ 2185   998 2181980, i8042
 14:     133059    IO-APIC-edge      0/ 3057   905 2769148, ide0
 15:     334204    IO-APIC-edge      0/ 4202   515 2166602, ide1
 16:     466139   IO-APIC-level      0/ 6139  1598 9813868, eth0, nvidia
 21:          0   IO-APIC-level      0/    0     0 0, ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb4, uhci_hcd:usb5
 22:      10619   IO-APIC-level      0/  619  1484 919075, VIA8237
NMI:          0 
LOC:    7520565 
ERR:          0
MIS:          0

[-- Attachment #5: ints_io_apic_noload --]
[-- Type: text/plain, Size: 801 bytes --]

           CPU0       
  0:     260074    IO-APIC-edge      0/   74  4711 348679, timer
  1:        596    IO-APIC-edge      0/  596 11293 6731173, i8042
  8:          1    IO-APIC-edge      0/    1 22686 22686, rtc
  9:          0   IO-APIC-level      0/    0     0 0, acpi
 12:         93    IO-APIC-edge      0/   93 11070 1029547, i8042
 14:       3305    IO-APIC-edge      0/ 3303 13404 44275707, ide0
 15:       2135    IO-APIC-edge      0/ 2133 10937 23329953, ide1
 16:        298   IO-APIC-level      0/  298 32680 9738917, eth0
 21:          0   IO-APIC-level      0/    0     0 0, ehci_hcd:usb1, uhci_hcd:usb2, uhci_hcd:usb3, uhci_hcd:usb4, uhci_hcd:usb5
 22:       1294   IO-APIC-level      0/ 1294 32107 41547375, VIA8237
NMI:          0 
LOC:     260001 
ERR:          0
MIS:          0

[-- Attachment #6: io_apic-RT-50-35.2.diff --]
[-- Type: text/x-diff, Size: 10320 bytes --]

diff -ru linux-2.6.12-RT-50-35/arch/i386/kernel/io_apic.c linux-2.6.12-RT/arch/i386/kernel/io_apic.c
--- linux-2.6.12-RT-50-35/arch/i386/kernel/io_apic.c	2005-06-30 16:38:19.000000000 +0200
+++ linux-2.6.12-RT/arch/i386/kernel/io_apic.c	2005-06-29 20:01:10.000000000 +0200
@@ -138,14 +138,24 @@
 /*
  * Cache register values:
  */
-static unsigned int io_apic_cache[MAX_IO_APICS][MAX_IOAPIC_CACHE]
+static struct {
+	unsigned int reg;
+	unsigned int val[MAX_IOAPIC_CACHE];
+} io_apic_cache[MAX_IO_APICS]
 		____cacheline_aligned_in_smp;
 #endif
 
-inline unsigned int __raw_io_apic_read(unsigned int apic, unsigned int reg)
+volatile unsigned int *io_apic_base[MAX_IO_APICS];
+
+static inline unsigned int __raw_io_apic_read(unsigned int apic, unsigned int reg)
 {
-	*IO_APIC_BASE(apic) = reg;
-	return *(IO_APIC_BASE(apic)+4);
+	volatile unsigned int *io_apic;
+#ifdef IOAPIC_CACHE
+	io_apic_cache[apic].reg = reg;
+#endif
+	io_apic = io_apic_base[apic];
+	io_apic[0] = reg;
+	return io_apic[4];
 }
 
 unsigned int raw_io_apic_read(unsigned int apic, unsigned int reg)
@@ -153,7 +163,7 @@
 	unsigned int val = __raw_io_apic_read(apic, reg);
 
 #ifdef IOAPIC_CACHE
-	io_apic_cache[apic][reg] = val;
+	io_apic_cache[apic].val[reg] = val;
 #endif
 	return val;
 }
@@ -172,14 +182,17 @@
 		}
 		return __raw_io_apic_read(apic, reg);
 	}
-	if (io_apic_cache[apic][reg] && !sis_apic_bug)
-		return io_apic_cache[apic][reg];
+	if (io_apic_cache[apic].val[reg] && !sis_apic_bug) {
+		io_apic_cache[apic].reg = -1;
+		return io_apic_cache[apic].val[reg];
+	}
 #endif
 	return raw_io_apic_read(apic, reg);
 }
 
 void io_apic_write(unsigned int apic, unsigned int reg, unsigned int val)
 {
+	volatile unsigned int *io_apic;
 #ifdef IOAPIC_CACHE
 	if (unlikely(reg >= MAX_IOAPIC_CACHE)) {
 		static int once = 1;
@@ -191,10 +204,14 @@
 			dump_stack();
 		}
 	} else
-		io_apic_cache[apic][reg] = val;
+		io_apic_cache[apic].val[reg] = val;
 #endif
-	*IO_APIC_BASE(apic) = reg;
-	*(IO_APIC_BASE(apic)+4) = val;
+	io_apic = io_apic_base[apic];
+#ifdef IOAPIC_CACHE
+	io_apic_cache[apic].reg = reg;
+#endif
+	io_apic[0] = reg;
+	io_apic[4] = val;
 }
 
 /*
@@ -214,34 +231,42 @@
  */
 void io_apic_modify(unsigned int apic, unsigned int reg, unsigned int val)
 {
+	volatile unsigned int *io_apic;
 #ifdef IOAPIC_CACHE
-	io_apic_cache[apic][reg] = val;
+	io_apic_cache[apic].val[reg] = val;
 #endif
-	if (unlikely(sis_apic_bug))
-		*IO_APIC_BASE(apic) = reg;
-	*(IO_APIC_BASE(apic)+4) = val;
+	io_apic = io_apic_base[apic];
+#ifdef IOAPIC_CACHE
+	if (io_apic_cache[apic].reg != reg || sis_apic_bug) {
+		io_apic_cache[apic].reg = reg;
+#else
+	if (unlikely(sis_apic_bug)) {
+#endif
+		io_apic[0] = reg;
+	}
+	io_apic[4] = val;
 #ifndef IOAPIC_POSTFLUSH
 	if (unlikely(sis_apic_bug))
 #endif
 		/*
 		 * Force POST flush by reading:
  		 */
-		val = *(IO_APIC_BASE(apic)+4);
+		val = io_apic[4];
 }
 
 static void __modify_IO_APIC_irq (unsigned int irq, unsigned long enable, unsigned long disable)
 {
 	struct irq_pin_list *entry = irq_2_pin + irq;
-	unsigned int pin, reg;
+	unsigned int pin, val;
 
 	for (;;) {
 		pin = entry->pin;
 		if (pin == -1)
 			break;
-		reg = io_apic_read(entry->apic, 0x10 + pin*2);
-		reg &= ~disable;
-		reg |= enable;
-		io_apic_modify(entry->apic, 0x10 + pin*2, reg);
+		val = io_apic_read(entry->apic, 0x10 + pin*2);
+		val &= ~disable;
+		val |= enable;
+		io_apic_modify(entry->apic, 0x10 + pin*2, val);
 		if (!entry->next)
 			break;
 		entry = irq_2_pin + entry->next;
@@ -249,13 +274,13 @@
 }
 
 /* mask = 1 */
-static void __mask_IO_APIC_irq (unsigned int irq)
+static inline void __mask_IO_APIC_irq (unsigned int irq)
 {
 	__modify_IO_APIC_irq(irq, 0x00010000, 0);
 }
 
 /* mask = 0 */
-static void __unmask_IO_APIC_irq (unsigned int irq)
+static inline void __unmask_IO_APIC_irq (unsigned int irq)
 {
 	__modify_IO_APIC_irq(irq, 0, 0x00010000);
 }
@@ -306,9 +331,13 @@
 {
 	int apic, pin;
 
-	for (apic = 0; apic < nr_ioapics; apic++)
+	for (apic = 0; apic < nr_ioapics; apic++) {
+#ifdef IOAPIC_CACHE
+		io_apic_cache[apic].reg = -1;
+#endif
 		for (pin = 0; pin < nr_ioapic_registers[apic]; pin++)
 			clear_IO_APIC_pin(apic, pin);
+	}
 }
 
 static void set_ioapic_affinity_irq(unsigned int irq, cpumask_t cpumask)
Nur in linux-2.6.12-RT/arch/i386/kernel: io_apic.c~.
Nur in linux-2.6.12-RT/arch/i386/kernel: io_apic.c-50-33.
diff -ru linux-2.6.12-RT-50-35/arch/i386/kernel/irq.c linux-2.6.12-RT/arch/i386/kernel/irq.c
--- linux-2.6.12-RT-50-35/arch/i386/kernel/irq.c	2005-06-30 16:38:19.000000000 +0200
+++ linux-2.6.12-RT/arch/i386/kernel/irq.c	2005-06-30 16:11:18.000000000 +0200
@@ -217,6 +217,8 @@
 	}
 
 	if (i < NR_IRQS) {
+		unsigned int irq_count;
+		cycles_t cycles;
 		spin_lock_irqsave(&irq_desc[i].lock, flags);
 		action = irq_desc[i].action;
 		if (!action)
@@ -230,12 +232,16 @@
 				seq_printf(p, "%10u ", kstat_cpu(j).irqs[i]);
 #endif
 		seq_printf(p, " %14s", irq_desc[i].handler->typename);
-		seq_printf(p, "  %s", action->name);
 
-		for (action=action->next; action; action = action->next)
-			seq_printf(p, ", %s", action->name);
-		seq_printf(p, "  %d/%d", irq_desc[i].irqs_unhandled, irq_desc[i].irq_count);
+		irq_count = irq_desc[i].irq_count;
+		cycles =  irq_desc[i].cycles;
+		if (irq_count) {
+			do_div(cycles, irq_count);
+		}
+		seq_printf(p, "  %5d/%5d %5lld %lld", irq_desc[i].irqs_unhandled, irq_count, cycles, irq_desc[i].cycles);
 
+		for (; action; action = action->next)
+			seq_printf(p, ", %s", action->name);
 		seq_putc(p, '\n');
 skip:
 		spin_unlock_irqrestore(&irq_desc[i].lock, flags);
Nur in linux-2.6.12-RT/arch/i386/kernel: irq.c~.
diff -ru linux-2.6.12-RT-50-35/arch/i386/kernel/mpparse.c linux-2.6.12-RT/arch/i386/kernel/mpparse.c
--- linux-2.6.12-RT-50-35/arch/i386/kernel/mpparse.c	2005-06-30 16:38:00.000000000 +0200
+++ linux-2.6.12-RT/arch/i386/kernel/mpparse.c	2005-06-29 20:30:50.000000000 +0200
@@ -263,6 +263,7 @@
 		return;
 	}
 	mp_ioapics[nr_ioapics] = *m;
+	io_apic_base[nr_ioapics] = IO_APIC_BASE(nr_ioapics);
 	nr_ioapics++;
 }
 
@@ -914,6 +915,7 @@
 	mp_ioapics[idx].mpc_apicaddr = address;
 
 	set_fixmap_nocache(FIX_IO_APIC_BASE_0 + idx, address);
+	io_apic_base[idx] = IO_APIC_BASE(idx);
 	mp_ioapics[idx].mpc_apicid = io_apic_get_unique_id(idx, id);
 	mp_ioapics[idx].mpc_apicver = io_apic_get_version(idx);
 	
Nur in linux-2.6.12-RT/arch/i386/kernel: mpparse.c~.
Nur in linux-2.6.12-RT/arch/i386/kernel: mpparse.c-RT-50-35.
Nur in linux-2.6.12-RT: cscope.files.
Nur in linux-2.6.12-RT: cscope.files~.
Nur in linux-2.6.12-RT: cscope.out.
Nur in linux-2.6.12-RT/Documentation: realtime-lsm.txt.
diff -ru linux-2.6.12-RT-50-35/include/asm-i386/io_apic.h linux-2.6.12-RT/include/asm-i386/io_apic.h
--- linux-2.6.12-RT-50-35/include/asm-i386/io_apic.h	2005-06-30 16:38:20.000000000 +0200
+++ linux-2.6.12-RT/include/asm-i386/io_apic.h	2005-06-29 19:59:17.000000000 +0200
@@ -155,6 +155,8 @@
 /* MP IRQ source entries */
 extern struct mpc_config_intsrc mp_irqs[MAX_IRQ_SOURCES];
 
+extern volatile unsigned int *io_apic_base[MAX_IO_APICS];
+
 /* non-0 if default (table-less) MP configuration */
 extern int mpc_default_type;
 
Nur in linux-2.6.12-RT/include/asm-i386: io_apic.h~.
diff -ru linux-2.6.12-RT-50-35/include/linux/irq.h linux-2.6.12-RT/include/linux/irq.h
--- linux-2.6.12-RT-50-35/include/linux/irq.h	2005-06-30 16:38:20.000000000 +0200
+++ linux-2.6.12-RT/include/linux/irq.h	2005-06-30 13:24:46.000000000 +0200
@@ -79,7 +79,7 @@
 	unsigned int irqs_unhandled;
 	struct task_struct *thread;
 	wait_queue_head_t wait_for_handler;
-	cycles_t timestamp;
+	cycles_t cycles;
 	raw_spinlock_t lock;
 } ____cacheline_aligned irq_desc_t;
 
Nur in linux-2.6.12-RT/include/linux: irq.h~.
Nur in linux-2.6.12-RT/include: sound.
Nur in linux-2.6.12-RT/kernel: cscope.files.
Nur in linux-2.6.12-RT/kernel: cscope.out.
diff -ru linux-2.6.12-RT-50-35/kernel/irq/handle.c linux-2.6.12-RT/kernel/irq/handle.c
--- linux-2.6.12-RT-50-35/kernel/irq/handle.c	2005-06-30 16:38:20.000000000 +0200
+++ linux-2.6.12-RT/kernel/irq/handle.c	2005-06-30 13:36:11.000000000 +0200
@@ -143,10 +143,10 @@
 	return retval;
 }
 
-cycles_t irq_timestamp(unsigned int irq)
-{
-	return irq_desc[irq].timestamp;
-}
+/* cycles_t irq_timestamp(unsigned int irq) */
+/* { */
+/* 	return irq_desc[irq].timestamp; */
+/* } */
 
 /*
  * do_IRQ handles all normal device IRQ's (the special
@@ -160,7 +160,7 @@
 	unsigned int status;
 #ifdef CONFIG_PREEMPT_RT
 	unsigned long flags;
-
+	cycles_t timestamp;
 	/*
 	 * Disable the soft-irq-flag:
 	 */
@@ -178,7 +178,8 @@
 		desc->handler->end(irq);
 		return 1;
 	}
-	desc->timestamp = get_cycles();
+
+	timestamp = get_cycles();
 
 	spin_lock(&desc->lock);
 	desc->handler->ack(irq);
@@ -256,6 +257,7 @@
 	local_irq_restore(flags);
 	/* re-disable interrupts because callers expect irqs off: */
 //	raw_local_irq_disable();
+	desc->cycles += get_cycles() - timestamp;
 #endif
 	return 1;
 }
Nur in linux-2.6.12-RT/kernel/irq: handle.c~.
diff -ru linux-2.6.12-RT-50-35/kernel/irq/manage.c linux-2.6.12-RT/kernel/irq/manage.c
--- linux-2.6.12-RT-50-35/kernel/irq/manage.c	2005-06-30 16:38:20.000000000 +0200
+++ linux-2.6.12-RT/kernel/irq/manage.c	2005-06-30 13:39:35.000000000 +0200
@@ -412,6 +412,7 @@
 	spin_lock_irq(&desc->lock);
 
 	if (desc->status & IRQ_INPROGRESS) {
+		cycles_t timestamp;
 		action = desc->action;
 		for (;;) {
 			irqreturn_t action_ret = 0;
@@ -434,9 +435,11 @@
 		 * The ->end() handler has to deal with interrupts which got
 		 * disabled while the handler was running.
 		 */
+		timestamp = get_cycles();
 		desc->handler->end(irq);
 		if (!(desc->status & IRQ_DISABLED))
 			desc->handler->enable(irq);
+		desc->cycles += get_cycles() - timestamp;
 	}
 	spin_unlock_irq(&desc->lock);
 
Nur in linux-2.6.12-RT/kernel/irq: manage.c~.
diff -ru linux-2.6.12-RT-50-35/kernel/irq/spurious.c linux-2.6.12-RT/kernel/irq/spurious.c
--- linux-2.6.12-RT-50-35/kernel/irq/spurious.c	2005-06-30 16:38:20.000000000 +0200
+++ linux-2.6.12-RT/kernel/irq/spurious.c	2005-06-30 15:41:54.000000000 +0200
@@ -74,11 +74,12 @@
 	}
 
 	desc->irq_count++;
-	if (desc->irq_count < 100000)
+	if (desc->irq_count < 10000)
 		return;
 
 	desc->irq_count = 0;
-	if (desc->irqs_unhandled > 99900) {
+	desc->cycles = 0;
+	if (desc->irqs_unhandled > 9990) {
 		/*
 		 * The interrupt is stuck
 		 */

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-37

[William Weston]

Hi Ingo,

-50-37 wouldn't compile out of the box on my debug config.
Here's a couple minor cleanups:

--- linux-2.6.12-RT-V0.7.50-37.orig/include/linux/preempt.h	2005-06-30 12:36:37.000000000 -0700
+++ linux-2.6.12-RT-V0.7.50-37/include/linux/preempt.h	2005-06-30 12:23:38.000000000 -0700
@@ -8,6 +8,7 @@
 
 #include <linux/config.h>
 #include <linux/linkage.h>
+#include <linux/thread_info.h>
 
 #if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_CRITICAL_TIMING)
   extern void notrace add_preempt_count(unsigned int val);
--- linux-2.6.12-RT-V0.7.50-37.orig/kernel/rt.c	2005-06-30 12:36:37.000000000 -0700
+++ linux-2.6.12-RT-V0.7.50-37/kernel/rt.c	2005-06-30 12:30:55.000000000 -0700
@@ -450,7 +450,7 @@
 #ifdef CONFIG_DEBUG_PREEMPT
 	if (task->lock_count) {
 		static int once = 1;
-		if (once > 0)
+		if (once > 0) {
 			once--;
 			printk("BUG: nonzero lock count %d at exit time?\n",
 				task->lock_count);
@@ -925,7 +925,7 @@
 static int
 capture_lock(struct rt_mutex_waiter *waiter, struct task_struct *task)
 {
-	struct thread_info *ti = current_thread_info();
+	//struct thread_info *ti = current_thread_info();
 	struct rt_mutex *lock = waiter->lock;
 	unsigned long flags;
 	int ret = 0;
@@ -1411,7 +1411,7 @@
  */
 static void __up_mutex(struct rt_mutex *lock, int save_state __EIP_DECL__)
 {
-	struct thread_info *ti = current_thread_info();
+	//struct thread_info *ti = current_thread_info();
 	struct task_struct *old_owner, *new_owner;
 	struct rt_mutex_waiter *w;
 	unsigned long flags;


FYI, a diff against include/linux/compile.h also made it into the the 
-50-37 patch.


--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-37

[Ingo Molnar]

* William Weston <weston@sysex.net> wrote:

> Hi Ingo,
> 
> -50-37 wouldn't compile out of the box on my debug config.
> Here's a couple minor cleanups:

thanks, applied. except these:

> -	struct thread_info *ti = current_thread_info();
> +	//struct thread_info *ti = current_thread_info();

> -	struct thread_info *ti = current_thread_info();
> +	//struct thread_info *ti = current_thread_info();

they are needed in the nondebug build - i'll fix this up.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-38

[Michal Schmidt]

[-- Attachment #1: Type: text/plain, Size: 98 bytes --]

Here are some more fixes for -RT-2.6.12-final-V0.7.50-38 needed to 
compile it on x86_64.

Michal

[-- Attachment #2: rt-amd64-PER_CPU_LOCKED.diff --]
[-- Type: text/plain, Size: 1203 bytes --]

diff -Nurp -X dontdiff.new linux-RT/arch/x86_64/mm/init.c linux-RT.mich/arch/x86_64/mm/init.c
--- linux-RT/arch/x86_64/mm/init.c	2005-06-30 21:22:33.000000000 +0200
+++ linux-RT.mich/arch/x86_64/mm/init.c	2005-06-30 22:28:16.000000000 +0200
@@ -47,7 +47,7 @@ extern int swiotlb;
 
 extern char _stext[];
 
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU_LOCKED(struct mmu_gather, mmu_gathers);
 
 /*
  * NOTE: pagetable_init alloc all the fixmap pagetables contiguous on the
diff -Nurp -X dontdiff.new linux-RT/include/asm-x86_64/percpu.h linux-RT.mich/include/asm-x86_64/percpu.h
--- linux-RT/include/asm-x86_64/percpu.h	2005-06-30 21:44:37.000000000 +0200
+++ linux-RT.mich/include/asm-x86_64/percpu.h	2005-06-30 22:26:10.000000000 +0200
@@ -63,6 +63,9 @@ extern void setup_per_cpu_areas(void);
 #endif	/* SMP */
 
 #define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
+#define DECLARE_PER_CPU_LOCKED(type, name) \
+	extern spinlock_t per_cpu_lock__##name##_locked; \
+	extern __typeof__(type) per_cpu__##name##_locked
 
 #define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
 #define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-38

[Ingo Molnar]

* Michal Schmidt <xschmi00@stud.feec.vutbr.cz> wrote:

> Here are some more fixes for -RT-2.6.12-final-V0.7.50-38 needed to 
> compile it on x86_64.

thanks - i've put them into the -39 patch.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-37

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * William Weston <weston@sysex.net> wrote:
> 
> > Hi Ingo,
> > 
> > -50-37 wouldn't compile out of the box on my debug config.
> > Here's a couple minor cleanups:
> 
> thanks, applied. [...]

i've uploaded -39 with your fixes and other fixes - could you check that 
it compiles cleanly for you now?

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* Karsten Wiese <annabellesgarden@yahoo.de> wrote:

> Here come some numbers to back up the usefullness of 
> CONFIG_X86_UP_IOAPIC_FAST. (and to show that my patch actually works 
> ;-)) All measurement where taken on an UP Athlon64 2Ghz running 32bit 
> 2.6.12-RT-50-35 PREEMPT_RT on a K8T800 mobo.

thanks - the numbers are pretty convincing. I've applied most of your 
patch (except the instrumentation bits), and it seems to work quite well 
- one of my testsystems that had interrupt storms before can now run 
IOAPIC_FAST. (i also enabled the option to be selectable for SMP kernels 
too. If things work out fine we can make it default-on.) I've uploaded 
the -50-39 patch with these changes included.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-37

[Gene Heskett]

On Thursday 30 June 2005 15:52, Ingo Molnar wrote:
>* William Weston <weston@sysex.net> wrote:
>> Hi Ingo,
>>
>> -50-37 wouldn't compile out of the box on my debug config.
>> Here's a couple minor cleanups:
>
>thanks, applied. except these:
>> - struct thread_info *ti = current_thread_info();
>> + //struct thread_info *ti = current_thread_info();
>>
>> - struct thread_info *ti = current_thread_info();
>> + //struct thread_info *ti = current_thread_info();
>
>they are needed in the nondebug build - i'll fix this up.
>
> Ingo
>-
Humm, I can't get the -37 to apply, something about the wrong -p0/-p1 
option to patch, but neither one finds the file to patch.  PEBKAC? or 
what Ingo?

>To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-37

[Gene Heskett]

On Thursday 30 June 2005 16:48, Ingo Molnar wrote:
>* Ingo Molnar <mingo@elte.hu> wrote:
>> * William Weston <weston@sysex.net> wrote:
>> > Hi Ingo,
>> >
>> > -50-37 wouldn't compile out of the box on my debug config.
>> > Here's a couple minor cleanups:
>>
>> thanks, applied. [...]
>
>i've uploaded -39 with your fixes and other fixes - could you check
> that it compiles cleanly for you now?
>
> Ingo

I'll go try this one after dinner, we been w/o power here for about 3 
hours.  The only machine that rebooted ok was the one NOT on a ups, 
but running the LVM (FC4 box).  Interesting.

>-
>To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.35% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Karsten Wiese]

Am Donnerstag, 30. Juni 2005 22:50 schrieb Ingo Molnar:
> 
> * Karsten Wiese <annabellesgarden@yahoo.de> wrote:
> 
> > Here come some numbers to back up the usefullness of 
> > CONFIG_X86_UP_IOAPIC_FAST. (and to show that my patch actually works 
> > ;-)) All measurement where taken on an UP Athlon64 2Ghz running 32bit 
> > 2.6.12-RT-50-35 PREEMPT_RT on a K8T800 mobo.
> 
> thanks - the numbers are pretty convincing. I've applied most of your 
> patch (except the instrumentation bits), and it seems to work quite well 
> - one of my testsystems that had interrupt storms before can now run 
> IOAPIC_FAST. (i also enabled the option to be selectable for SMP kernels 
> too. If things work out fine we can make it default-on.) I've uploaded 
> the -50-39 patch with these changes included.
> 
Ooops, you didn't apply this part of the patch:

<snip>
--- linux-2.6.12-RT-50-35/arch/i386/kernel/mpparse.c	2005-06-30 16:38:00.000000000 +0200
+++ linux-2.6.12-RT/arch/i386/kernel/mpparse.c	2005-06-29 20:30:50.000000000 +0200
@@ -263,6 +263,7 @@
 		return;
 	}
 	mp_ioapics[nr_ioapics] = *m;
+	io_apic_base[nr_ioapics] = IO_APIC_BASE(nr_ioapics);
 	nr_ioapics++;
 }
 
@@ -914,6 +915,7 @@
 	mp_ioapics[idx].mpc_apicaddr = address;
 
 	set_fixmap_nocache(FIX_IO_APIC_BASE_0 + idx, address);
+	io_apic_base[idx] = IO_APIC_BASE(idx);
 	mp_ioapics[idx].mpc_apicid = io_apic_get_unique_id(idx, id);
 	mp_ioapics[idx].mpc_apicver = io_apic_get_version(idx);
</snip>

It provides the precalculation of the ioapics's virtual address.
Or does IO_APIC_BASE() compile to an indexed lookup?
 then io_apic_base[] wouldn't be needed...

Karsten

	

	
		
___________________________________________________________ 
Gesendet von Yahoo! Mail - Jetzt mit 1GB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[William Weston]

On Fri, 1 Jul 2005, Karsten Wiese wrote:
> 
> <snip>
> --- linux-2.6.12-RT-50-35/arch/i386/kernel/mpparse.c	2005-06-30 16:38:00.000000000 +0200
> +++ linux-2.6.12-RT/arch/i386/kernel/mpparse.c	2005-06-29 20:30:50.000000000 +0200
> @@ -263,6 +263,7 @@
>  		return;
>  	}
>  	mp_ioapics[nr_ioapics] = *m;
> +	io_apic_base[nr_ioapics] = IO_APIC_BASE(nr_ioapics);
>  	nr_ioapics++;
>  }
>  
> @@ -914,6 +915,7 @@
>  	mp_ioapics[idx].mpc_apicaddr = address;
>  
>  	set_fixmap_nocache(FIX_IO_APIC_BASE_0 + idx, address);
> +	io_apic_base[idx] = IO_APIC_BASE(idx);
>  	mp_ioapics[idx].mpc_apicid = io_apic_get_unique_id(idx, id);
>  	mp_ioapics[idx].mpc_apicver = io_apic_get_version(idx);
> </snip>
> 
> It provides the precalculation of the ioapics's virtual address.
> Or does IO_APIC_BASE() compile to an indexed lookup?
>  then io_apic_base[] wouldn't be needed...

Thanks, Karsten.  This patch was needed to get my Xeon/HT (i865) to boot 
-50-39.  Next I'll see if IOAPIC_FAST will work.  I'll keep you posted.

--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Karsten Wiese]

Am Freitag, 1. Juli 2005 00:27 schrieb Karsten Wiese:
> Am Donnerstag, 30. Juni 2005 22:50 schrieb Ingo Molnar:
> > 
> > * Karsten Wiese <annabellesgarden@yahoo.de> wrote:
> > 
> > > Here come some numbers to back up the usefullness of 
> > > CONFIG_X86_UP_IOAPIC_FAST. (and to show that my patch actually works 
> > > ;-)) All measurement where taken on an UP Athlon64 2Ghz running 32bit 
> > > 2.6.12-RT-50-35 PREEMPT_RT on a K8T800 mobo.
> > 
> > thanks - the numbers are pretty convincing. I've applied most of your 
> > patch (except the instrumentation bits), and it seems to work quite well 
> > - one of my testsystems that had interrupt storms before can now run 
> > IOAPIC_FAST. (i also enabled the option to be selectable for SMP kernels 
> > too. If things work out fine we can make it default-on.) I've uploaded 
> > the -50-39 patch with these changes included.
> > 
> Ooops, you didn't apply this part of the patch:
> 
> <snip>
> --- linux-2.6.12-RT-50-35/arch/i386/kernel/mpparse.c	2005-06-30 16:38:00.000000000 +0200
> +++ linux-2.6.12-RT/arch/i386/kernel/mpparse.c	2005-06-29 20:30:50.000000000 +0200
> @@ -263,6 +263,7 @@
>  		return;
>  	}
>  	mp_ioapics[nr_ioapics] = *m;
> +	io_apic_base[nr_ioapics] = IO_APIC_BASE(nr_ioapics);
>  	nr_ioapics++;
>  }
>  
> @@ -914,6 +915,7 @@
>  	mp_ioapics[idx].mpc_apicaddr = address;
>  
>  	set_fixmap_nocache(FIX_IO_APIC_BASE_0 + idx, address);
> +	io_apic_base[idx] = IO_APIC_BASE(idx);
>  	mp_ioapics[idx].mpc_apicid = io_apic_get_unique_id(idx, id);
>  	mp_ioapics[idx].mpc_apicver = io_apic_get_version(idx);
> </snip>
> 
> It provides the precalculation of the ioapics's virtual address.
> Or does IO_APIC_BASE() compile to an indexed lookup?
>  then io_apic_base[] wouldn't be needed...
> 
It is needed (only for optimization):
There are ioapics that have baseaddresses not on a page boundary.
Another idea: to merge the virtual ioapic base addresses into the io_apic_cache struct to use 
the cpu-cache more efficiently ;-) 

Karsten

	

	
		
___________________________________________________________ 
Gesendet von Yahoo! Mail - Jetzt mit 1GB Speicher kostenlos - Hier anmelden: http://mail.yahoo.de

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[William Weston]

On Thu, 30 Jun 2005, William Weston wrote:
> On Fri, 1 Jul 2005, Karsten Wiese wrote:
> > --- linux-2.6.12-RT-50-35/arch/i386/kernel/mpparse.c	2005-06-30 16:38:00.000000000 +0200
> > +++ linux-2.6.12-RT/arch/i386/kernel/mpparse.c	2005-06-29 20:30:50.000000000 +0200
> > @@ -263,6 +263,7 @@
> >  		return;
> >  	}
> >  	mp_ioapics[nr_ioapics] = *m;
> > +	io_apic_base[nr_ioapics] = IO_APIC_BASE(nr_ioapics);
> >  	nr_ioapics++;
> >  }
> >  
> > @@ -914,6 +915,7 @@
> >  	mp_ioapics[idx].mpc_apicaddr = address;
> >  
> >  	set_fixmap_nocache(FIX_IO_APIC_BASE_0 + idx, address);
> > +	io_apic_base[idx] = IO_APIC_BASE(idx);
> >  	mp_ioapics[idx].mpc_apicid = io_apic_get_unique_id(idx, id);
> >  	mp_ioapics[idx].mpc_apicver = io_apic_get_version(idx);
> 
> Thanks, Karsten.  This patch was needed to get my Xeon/HT (i865) to boot 
> -50-39.  Next I'll see if IOAPIC_FAST will work.  I'll keep you posted.

After applying this patch to -50-39, IOAPIC_FAST now works on both the 
Xeon/HT and the non-HT Xeon boxes (i865 and i845 respectively).  Wakeup 
latencies are looking better, especially on the Xeon/HT (max of 53us so 
far, as compared to 183us with -50-37 without IOAPIC_FAST).

I'll test this out on the Athlon box (which hasn't done well with 
IOAPIC_FAST previously) when I get home.

--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* Karsten Wiese <annabellesgarden@yahoo.de> wrote:

> Ooops, you didn't apply this part of the patch:

oops - i had it in my tree (so all my tests passed), but it escaped the 
-39 patch. I've uploaded -50-40 with this included too.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Chuck Harding]

On Fri, 1 Jul 2005, Ingo Molnar wrote:

>
> * Karsten Wiese <annabellesgarden@yahoo.de> wrote:
>
>> Ooops, you didn't apply this part of the patch:
>
> oops - i had it in my tree (so all my tests passed), but it escaped the
> -39 patch. I've uploaded -50-40 with this included too.
>
> 	Ingo
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Well, the -50-42 still has the problem with sox locking up too tightly to
kill. Just saw that the sox processes are all in state 'D', FWIW.

-- 
Charles D. (Chuck) Harding <charding@llnl.gov>  Voice: 925-423-8879
Senior Computer Associate         ICCD            Fax: 925-423-6961
Lawrence Livermore National Laboratory      Computation Directorate
Livermore, CA USA  http://www.llnl.gov  GPG Public Key ID: B9EB6601
------------------ http://tinyurl.com/5w5ey -----------------------
-- Bad breath is better than no breath. --

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[William Weston]

On Fri, 1 Jul 2005, Ingo Molnar wrote:

> oops - i had it in my tree (so all my tests passed), but it escaped the 
> -39 patch. I've uploaded -50-40 with this included too.

FWIW, I'm still seeing the SMT scheduling? meltdown issues with -50-42.  
Running two instances of 'dd if=/dev/zero of=/dev/null bs=65536' instead 
of 'burnP6' results in the same behavior.  Here's a quick recap:

- Start (or login to ) X.
- Start an X app that constantly updates the screen, like wmcube, or vlc.
- Run 'burnP6' or 'dd if=/dev/zero of=/dev/null bs=65536'.
- Run trace-it.  Trace completes without any troubles.
- Run another 'burnP6' or 'dd if=/dev/zero of=/dev/null bs=65536'.

At this point, most of the system is unresponsive:

- Most X apps are frozen (even top in its own xterm).
- Mouse lost synchro serio warnings show up on serial console.
- Serial console is otherwise unresponsive (no SysRq).
- X server quits responding to keyboard input.
- Kbd input makes mouse temporarily unresponsive (for .1 to >5 secs).
- Mouse immediately after kbd triggers more 'mouse lost synchro' messages.
- Networking is lost (box won't respond to pings).
- Any script automating starting burnP6 or dd and then trace-it hangs.

A few things are left working (but not enough to get the system back):

- Mouse pointer (movements are chunky) and window focus.
- Mouse scroll wheel can still scroll xterms and switch workspaces.
- SysRq-B

All of this behavior is 100% reproduceable on recent realtime SMP/SMT
kernels, and is _not_ seen with vanilla SMP/SMT, Fedora SMP, or realtime
UP kernels.  I'll be running realtime UP kernels on this Xeon/HT box for a
while unless someone gives me reason to enable SMT again.

You can tell me to shut up now.... at least until I decide to hunt down
another corner case ;-}


Best Regards,
--William Weston

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Lee Revell]

On Fri, 2005-07-01 at 18:46 -0700, William Weston wrote:
> FWIW, I'm still seeing the SMT scheduling? meltdown issues with
> -50-42.  
> Running two instances of 'dd if=/dev/zero of=/dev/null bs=65536'
> instead 
> of 'burnP6' results in the same behavior.  Here's a quick recap:
> 
> - Start (or login to ) X.
> - Start an X app that constantly updates the screen, like wmcube, or
> vlc. 

Which video driver is X using?  What nice value is the X server running
at?

Does adding:

Option "NoAccel"

to the Device section of your X config file make any difference?

(on most systems X is the only thing besides the kernel that can access
hardware directly, which can cause problems)

Lee

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* William Weston <weston@sysex.net> wrote:

> FWIW, I'm still seeing the SMT scheduling? meltdown issues with 
> -50-42.  Running two instances of 'dd if=/dev/zero of=/dev/null 
> bs=65536' instead of 'burnP6' results in the same behavior.  Here's a 
> quick recap:
> 
> - Start (or login to ) X.
> - Start an X app that constantly updates the screen, like wmcube, or vlc.
> - Run 'burnP6' or 'dd if=/dev/zero of=/dev/null bs=65536'.
> - Run trace-it.  Trace completes without any troubles.
> - Run another 'burnP6' or 'dd if=/dev/zero of=/dev/null bs=65536'.
> 
> At this point, most of the system is unresponsive:
> 
> - Most X apps are frozen (even top in its own xterm).
> - Mouse lost synchro serio warnings show up on serial console.
> - Serial console is otherwise unresponsive (no SysRq).
> - X server quits responding to keyboard input.
> - Kbd input makes mouse temporarily unresponsive (for .1 to >5 secs).
> - Mouse immediately after kbd triggers more 'mouse lost synchro' messages.
> - Networking is lost (box won't respond to pings).
> - Any script automating starting burnP6 or dd and then trace-it hangs.
> 
> A few things are left working (but not enough to get the system back):
> 
> - Mouse pointer (movements are chunky) and window focus.
> - Mouse scroll wheel can still scroll xterms and switch workspaces.
> - SysRq-B

hm, i can reproduce a variant of this, by starting enough 'dd' tasks.  
(it needed more than two on a 2-way/4-way HT testbox though) Indeed 
everything seems to be starved, but SysRq still worked so i was able to 
SysRq-kIll all tasks and thus the system recovered.

i'm debugging this now.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > A few things are left working (but not enough to get the system back):
> > 
> > - Mouse pointer (movements are chunky) and window focus.
> > - Mouse scroll wheel can still scroll xterms and switch workspaces.
> > - SysRq-B
> 
> hm, i can reproduce a variant of this, by starting enough 'dd' tasks.  
> (it needed more than two on a 2-way/4-way HT testbox though) Indeed 
> everything seems to be starved, but SysRq still worked so i was able 
> to SysRq-kIll all tasks and thus the system recovered.
> 
> i'm debugging this now.

ok, found a bug that could explain the situation: mutex sleeps+wakeups 
were incorrectly credited as 'interactive sleep' periods, causing the dd 
processes to be boosted incorrectly. The dd processes created a workload 
in which they blocked each other in such a pattern that they got boosted 
periodically, starving pretty much every other task.

the fix is significant and affects alot of workloads, and should further 
improve interactivity in noticeable ways. I'm not 100% sure it solves 
all the starvation problems (e.g. how could normal-prio dd tasks starve 
the SCHED_FIFO irq threads that drove SysRq?), but the results so far 
look promising.

i've uploaded the -50-45 patch, can you under this kernel trigger a 
'meltdown' on your SMT box?

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[William Weston]

On Fri, 1 Jul 2005, Lee Revell wrote:

> On Fri, 2005-07-01 at 18:46 -0700, William Weston wrote:
> > FWIW, I'm still seeing the SMT scheduling? meltdown issues with
> > -50-42.  
> > Running two instances of 'dd if=/dev/zero of=/dev/null bs=65536'
> > instead 
> > of 'burnP6' results in the same behavior.  Here's a quick recap:
> > 
> > - Start (or login to ) X.
> > - Start an X app that constantly updates the screen, like wmcube, or
> > vlc. 
> 
> Which video driver is X using?  What nice value is the X server running
> at?

Hardware is Intel 82865G (integrated) with DRM i915 1.1.0 20040405 and 
xorg-3.8.2 i810 driver, running at nice 0, priority 15.  Should I bump the 
priority up?  To realtime?

> Does adding:
> 
> Option "NoAccel"
> 
> to the Device section of your X config file make any difference?

Disabling the dri and drm modules didn't help.  I'll turn on NoAccel when 
I'm back in the office on Tuesday.

> (on most systems X is the only thing besides the kernel that can access
> hardware directly, which can cause problems)

So would running X through the framebuffer device be the way to go for 
stability under realtime?  It's been a couple years since I've used fb.

--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* William Weston <weston@sysex.net> wrote:

> > Which video driver is X using?  What nice value is the X server running
> > at?
> 
> Hardware is Intel 82865G (integrated) with DRM i915 1.1.0 20040405 and 
> xorg-3.8.2 i810 driver, running at nice 0, priority 15.  Should I bump 
> the priority up?  To realtime?

i'd first suggest to try the latest kernel, before changing your X 
config - i think the bug might have been fixed meanwhile.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[William Weston]

On Sun, 3 Jul 2005, Ingo Molnar wrote:
> 
> i've uploaded the -50-45 patch, can you under this kernel trigger a 
> 'meltdown' on your SMT box?

Hi Ingo,

Here's the results of trying out everything from -50-45 through -51-01 on
the SMT box:

-50-47 looks better.  4x burnP6 + wmcube doesn't bring the box to a halt
anymore.  Running VLC along with a kernel compile brought all running
processes to a halt.  SysRq-I couldn't recover the system (unless I just
wasn't patient enough), but SysRq-S + SysRq-U + SysRq-B was able to sync,
unmount, and reboot.  -50-45 and -50-46 seemed to have similar behavior.

-50-48 won't boot:

softirq-timer/0/4[CPU#0]: BUG in __up_mutex at kernel/rt.c:1485
 [<c0103ec7>] dump_stack+0x23/0x25 (20)
 [<c0121580>] __WARN_ON+0x66/0x86 (52)
 [<c013bd49>] __up_mutex+0x14e/0x304 (40)
 [<c013c4a3>] up_mutex+0x39/0xe3 (28)
 [<c0131aca>] delayed_work_timer_fn+0x3a/0x41 (24)
 [<c012a7f6>] run_timer_softirq+0x208/0x46d (60)
 [<c0126833>] ksoftirqd+0xee/0x1ad (36)
 [<c0136786>] kthread+0xb6/0xba (44)
 [<c01010c5>] kernel_thread_helper+0x5/0xb (554565660)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c013efaa>] .... add_preempt_count+0x1c/0x1e
.....[<c012153b>] ..   ( <= __WARN_ON+0x21/0x86)
.. [<c013efaa>] .... add_preempt_count+0x1c/0x1e
.....[<c013ffc4>] ..   ( <= print_traces+0x1b/0x52)

------------------------------
| showing all locks held by: |  (softirq-timer/0/4 [def10030,  98]):
------------------------------

ACPI: Processor [CPU1] (supports 8 throttling states)
ACPI: Processor [CPU2] (supports 8 throttling states)
khelper/163[CPU#1]: BUG in set_new_owner at kernel/rt.c:822
NMI watchdog detected lockup on CPU#1 (50000/50000)

Pid: 163, comm:              khelper
EIP: 0060:[<c013b7c4>] CPU: 1
EIP is at __down_trylock+0x88/0x369
 EFLAGS: 00000082    Not tainted  (2.6.12-RT-V0.7.50-48-debug)
EAX: c03da258 EBX: c03da244 ECX: c03da250 EDX: c03da268
ESI: c03da268 EDI: 00000000 EBP: dec69d1c DS: 007b ES: 007b
CR0: 8005003b CR2: 00000000 CR3: 004f8000 CR4: 000006d0
 [<c01010b6>] show_regs+0x164/0x16e (36)
 [<c0112f94>] nmi_watchdog_tick+0x189/0x2b7 (52)
 [<c0104e80>] default_do_nmi+0x7a/0x18a (52)
 [<c0104ff4>] do_nmi+0x5d/0x6f (24)
 [<c0103b52>] nmi_stack_correct+0x1d/0x22 (92)
 [<c013c84f>] rt_down_trylock+0x29/0x179 (36)
 [<c0120e42>] vprintk+0x22e/0x2c3 (116)
 [<c0120c12>] printk+0x20/0x22 (20)
 [<c012157b>] __WARN_ON+0x61/0x86 (52)
 [<c037173b>] __down_mutex+0x4e2/0x689 (108)
 [<c037258b>] _spin_lock_irqsave+0x24/0x89 (32)
 [<c0129ccf>] lock_timer_base+0x1f/0x3a (20)
 [<c0129fa5>] try_to_del_timer_sync+0x1f/0x5f (28)
 [<c012a00f>] del_timer_sync+0x2a/0xb1 (44)
 [<c0135d06>] exit_itimers+0x117/0x1a9 (56)
 [<c012b66a>] __exit_signal+0xfd/0x1c8 (32)
 [<c012210e>] release_task+0x81/0x16a (40)
 [<c01232a8>] exit_notify+0x64f/0x972 (60)
 [<c0123816>] do_exit+0x24b/0x40d (44)
 [<c013178b>] wait_for_helper+0x0/0xb0 (24)
 [<c01010c5>] kernel_thread_helper+0x5/0xb (557408284)
---------------------------
| preempt count: 00010004 ]
| 4-level deep critical section nesting:
----------------------------------------
.. [<c013efaa>] .... add_preempt_count+0x1c/0x1e
.....[<c012153b>] ..   ( <= __WARN_ON+0x21/0x86)
.. [<c013efaa>] .... add_preempt_count+0x1c/0x1e
.....[<c0120c3a>] ..   ( <= vprintk+0x26/0x2c3)
.. [<c013efaa>] .... add_preempt_count+0x1c/0x1e
.....[<c0112f5f>] ..   ( <= nmi_watchdog_tick+0x154/0x2b7)
.. [<c013efaa>] .... add_preempt_count+0x1c/0x1e
.....[<c013ffc4>] ..   ( <= print_traces+0x1b/0x52)

------------------------------
| showing all locks held by: |  (khelper/163 [dec67930, 115]):
------------------------------

NMI watchdog detected lockup on CPU#0 (50000/50000)

Pid: 22, comm:              khelper
EIP: 0060:[<c0371315>] CPU: 0
EIP is at __down_mutex+0xbc/0x689
 EFLAGS: 00000082    Not tainted  (2.6.12-RT-V0.7.50-48-debug)
EAX: c14cde94 EBX: 00000000 ECX: 00000000 EDX: c14cdde8
ESI: c14cde80 EDI: c14cdea4 EBP: c14cde04 DS: 007b ES: 007b
CR0: 8005003b CR2: fffdf000 CR3: 004f8000 CR4: 000006d0
 [<c01010b6>] show_regs+0x164/0x16e (36)
 [<c0112f94>] nmi_watchdog_tick+0x189/0x2b7 (52)
 [<c0104e80>] default_do_nmi+0x7a/0x18a (52)
 [<c0104ff4>] do_nmi+0x5d/0x6f (24)
 [<c0103b52>] nmi_stack_correct+0x1d/0x22 (160)
 [<c03724cc>] _spin_lock_irq+0x24/0x7b (28)
 [<c03701aa>] wait_for_completion+0xa6/0xdb (48)
 [<c011f7a4>] do_fork+0xf1/0x1e2 (132)
 [<c0101151>] kernel_thread+0x86/0x8c (96)
 [<c01318a1>] __call_usermodehelper+0x66/0x68 (24)
 [<c0131d2f>] worker_thread+0x1a1/0x266 (108)
 [<c0136786>] kthread+0xb6/0xba (44)
 [<c01010c5>] kernel_thread_helper+0x5/0xb (1051926556)
---------------------------
| preempt count: 00010002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c013efaa>] .... add_preempt_count+0x1c/0x1e
.....[<c0112f5f>] ..   ( <= nmi_watchdog_tick+0x154/0x2b7)
.. [<c013efaa>] .... add_preempt_count+0x1c/0x1e
.....[<c013ffc4>] ..   ( <= print_traces+0x1b/0x52)

------------------------------
| showing all locks held by: |  (khelper/22 [c14c32f0, 114]):
------------------------------


-50-49 through -50-51 won't compile on my SMT debug config:

	arch/i386/kernel/vm86.c:701: error: syntax error before ',' token

-51-01 won't boot:

softirq-timer/1/13[CPU#1]: BUG in ____up_mutex at kernel/rt.c:1302
 [<c0103ea7>] dump_stack+0x23/0x25 (20)
 [<c0121410>] __WARN_ON+0x66/0x86 (52)
 [<c013ca44>] up_mutex+0x160/0x3b6 (36)
 [<c013195a>] delayed_work_timer_fn+0x3a/0x41 (24)
 [<c012a686>] run_timer_softirq+0x208/0x46d (60)
 [<c01266c3>] ksoftirqd+0xee/0x1ad (36)
 [<c0136616>] kthread+0xb6/0xba (44)
 [<c01010c5>] kernel_thread_helper+0x5/0xb (1052131356)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c01402ca>] .... add_preempt_count+0x1c/0x1e
.....[<c01213cb>] ..   ( <= __WARN_ON+0x21/0x86)
.. [<c01402ca>] .... add_preempt_count+0x1c/0x1e
.....[<c01412e4>] ..   ( <= print_traces+0x1b/0x52)

------------------------------
| showing all locks held by: |  (softirq-timer/1/13 [c1490670,  98]):
------------------------------

ACPI: Processor [CPU1] (supports 8 throttling states)
ACPI: Processor [CPU2] (supports 8 throttling states)
khelper/22[CPU#1]: BUG in set_new_owner at kernel/rt.c:869
NMI watchdog detected lockup on CPU#1 (50000/50000)


BTW, -50-44 through -50-51 wouldn't compile on the UP Athlon box.  -50-43
appears to be rock-solid under the UP non-debug config.  After hammering
on it all weekend, Maximum wakeup latency was 14us (from running JACK, 
phasex, burnK7, dd, and switching from X to text console all at once).


--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Ingo Molnar]

* William Weston <weston@sysex.net> wrote:

> Here's the results of trying out everything from -50-45 through -51-01 on
> the SMT box:

thanks for the extensive testing!

> -51-01 won't boot:
> 
> softirq-timer/1/13[CPU#1]: BUG in ____up_mutex at kernel/rt.c:1302

i have digged out an older HT-box .config of yours and have reproduced 
an assert quite similar to the one above. Found one bug in that area: 
the assert (conditional on RT_DEADLOCK_DETECT) was done a bit too early, 
i have fixed this in my tree and have uploaded -51-02.

> BTW, -50-44 through -50-51 wouldn't compile on the UP Athlon box.  

do the 51-02 (and later) kernels work on the UP Athlon box?

> -50-43 appears to be rock-solid under the UP non-debug config.  After 
> hammering on it all weekend, Maximum wakeup latency was 14us (from 
> running JACK, phasex, burnK7, dd, and switching from X to text console 
> all at once).

great!

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[William Weston]

On Wed, 6 Jul 2005, Ingo Molnar wrote:

> i have digged out an older HT-box .config of yours and have reproduced 
> an assert quite similar to the one above. Found one bug in that area: 
> the assert (conditional on RT_DEADLOCK_DETECT) was done a bit too early, 
> i have fixed this in my tree and have uploaded -51-02.

On -51-06, the vlc+dd test works ok (video is a bit choppy after the third
dd instance).  With the vlc+burnP6 test, the keyboard is lost first (for
everything except SysRq), then the serial console (for everything except
SysRq), then the mouse.  SysRq-I hangs X, but brings the serial console
back for a while.  At this point, I should be able to get more debug info 
than previously.

FWIW, I got this stack dump with SysRq-K:

SysRq : SAK
IRQ 4/307[CPU#0]: BUG in set_palette at drivers/char/vt.c:2918
 [<c0103ea7>] dump_stack+0x23/0x25 (20)
 [<c0121550>] __WARN_ON+0x66/0x86 (52)
 [<c02975df>] set_palette+0x5a/0x5c (24)
 [<c02980a8>] __handle_sysrq+0x9a/0x139 (48)
 [<c029817f>] handle_sysrq+0x38/0x3a (24)
 [<c02b40bf>] receive_chars+0x250/0x2ec (60)
 [<c02b4474>] serial8250_nterrupt+0xea/0x101 (44)
 [<c014b85b>] handle_IRQ_event+0x71/0xef (52)
 [<c014bfcb>] do_hardirq+0x53/0xda (40)
 [<c014c128>] do_irqd+0xd6/0x1d4 (40)
 [<c0136736>] kthread+0xb6/0xba (44)
 [<c01010c1>] kernel_thread_helper+0x5/0xb (556269596)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c01406ba>] .... add_preempt_count+0x1c/0x1e
.....[<c012150b>] ..   ( <= __WARN_ON+0x21/0x86)
.. [<c01406ba>] .... add_preempt_count+0x1c/0x1e
.....[<c01416d4>] ..   ( <= print_traces+0x1b/0x52)

------------------------------
| showing all locks held by: |  (IRQ 4/307 [ded7d930,  57]):
-----------------------------

#001:             [c087b2d0] {&irq_lists[i].lock}
... acquired at:  serial8250_interrupt+0x16/0x101

#002:             [c087ed40] {&port->lock}
... acquired at:  serial8250_interrupt+0x62/0x101

#003:             [c04bf380] {sysrq_key_table_lock}
... acquired at:  __handle_sysrq+0x2a/0x139

SysRq : Terminate All Tasks
SysRq : Kill All Tasks
SysRq : Emergency Sync
Emergency Sync complete
SysRq : Emergency Remount R/O
Emergency Remount complete
SysRq : SAK
IRQ 1/298[CPU#1]: BUG in set_palette at drivers/char/vt.c:2918
 [<c0103ea7>] dump_stack+0x23/0x25 (20)
 [<c0121550>] __WARN_ON+0x66/0x86 (52)
 [<c02975df>] set_palette+0x5a/0x5c (24)
 [<c02980a8>] __handle_sysrq+0x9a/0x139 (48)
 [<c029817f>] handle_sysrq+0x38/0x3a (24)
 [<c02928d3>] kbd_event+0xb9/0xdc (40)
 [<c02f2f06>] input_event+0xe6/0x478 (44)
 [<c02f6c45>] atkbd_report_key+0x3c/0xd4 (32)
 [<c02f6eb8>] atkbd_interrupt+0x1db/0x65f (56)
 [<c02adb6a>] serio_interrupt+0x49/0x8f (40)
 [<c02ae8a6>] i8042_interrupt+0x12a/0x238 (56)
 [<c014b85b>] handle_IRQ_event+0x71/0xef (52)
 [<c014bfcb>] do_hardirq+0x53/0xda (40)
 [<c014c128>] do_irqd+0xd6/0x1d4 (40)
 [<c0136736>] kthread+0xb6/0xba (44)
 [<c01010c1>] kernel_thread_helper+0x5/0xb (557096988)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c01406ba>] .... add_preempt_count+0x1c/0x1e
.....[<c012150b>] ..   ( <= __WARN_ON+0x21/0x86)
.. [<c01406ba>] .... add_preempt_count+0x1c/0x1e
.....[<c01416d4>] ..   ( <= print_traces+0x1b/0x52)

------------------------------
| showing all locks held by: |  (IRQ 1/298 [c14c2cb0,  56]):
------------------------------

#001:             [dec8404c] {&serio->lock}
... acquired at:  serio_interrupt+0x1f/0x8f

#002:             [c04bf380] {sysrq_key_table_lock}
... acquired at:  __handle_sysrq+0x2a/0x139

SysRq : Resetting


> > BTW, -50-44 through -50-51 wouldn't compile on the UP Athlon box.  

Correction:  There were a couple that wouldn't compile in there (-50-49
was one of them).  -50-48 and -51-01 on the Athlon give me a lot of these
with debugging enabled:

Jul  6 01:09:08 astarte kernel: delayed_work_timer_fn+0x1d/0x20softirq-timer/0/3[CPU#0]: BUG in ____up_mutex at kernel/rt.c:1302
Jul  6 01:09:08 astarte kernel:  [<c0104313>] dump_stack+0x23/0x30 (20)
Jul  6 01:09:08 astarte kernel:  [<c0120857>] __WARN_ON+0x77/0xc0 (56)
Jul  6 01:09:08 astarte kernel:  [<c013de4c>] rt_up+0x30c/0x470 (40)
Jul  6 01:09:08 astarte kernel:  [<c01201d2>] release_console_sem+0xc2/0x140 (36)
Jul  6 01:09:08 astarte kernel:  [<c011ff10>] vprintk+0x180/0x2a0 (120)
Jul  6 01:09:08 astarte kernel:  [<c011fd8d>] printk+0x1d/0x20 (16)
Jul  6 01:09:08 astarte kernel:  [<c01461d3>] __print_symbol+0x93/0x110 (400)
Jul  6 01:09:08 astarte kernel:  [<c01041e9>] show_trace+0x79/0xe0 (36)
Jul  6 01:09:08 astarte kernel:  [<c0104313>] dump_stack+0x23/0x30 (20)
Jul  6 01:09:08 astarte kernel:  [<c0120857>] __WARN_ON+0x77/0xc0 (56)
Jul  6 01:09:08 astarte kernel:  [<c013c989>] up_mutex+0x1f9/0x470 (36)
Jul  6 01:09:08 astarte kernel:  [<c0130ecd>] delayed_work_timer_fn+0x1d/0x20 (16)
Jul  6 01:09:08 astarte kernel:  [<c0129948>] run_timer_softirq+0x1f8/0x420 (56)
Jul  6 01:09:08 astarte kernel:  [<c012577f>] ksoftirqd+0xcf/0x170 (28)
Jul  6 01:09:08 astarte kernel:  [<c0135df6>] kthread+0xa6/0xe0 (48)
Jul  6 01:09:08 astarte kernel:  [<c01013e9>] kernel_thread_helper+0x5/0xc (537083932)
Jul  6 01:09:08 astarte kernel: ---------------------------
Jul  6 01:09:08 astarte kernel: | preempt count: 00000004 ]
Jul  6 01:09:08 astarte kernel: | 4-level deep critical section nesting:
Jul  6 01:09:08 astarte kernel: ----------------------------------------
Jul  6 01:09:08 astarte kernel: .. [<c014040c>] .... add_preempt_count+0x1c/0x20
Jul  6 01:09:08 astarte kernel: .....[<c012080f>] ..   ( <= __WARN_ON+0x2f/0xc0)
Jul  6 01:09:08 astarte kernel: .. [<c013db78>] .... rt_up+0x38/0x470
Jul  6 01:09:08 astarte kernel: .....[<c01201d2>] ..   ( <= release_console_sem+0xc2/0x140)
Jul  6 01:09:08 astarte kernel: .. [<c014040c>] .... add_preempt_count+0x1c/0x20
Jul  6 01:09:08 astarte kernel: .....[<c012080f>] ..   ( <= __WARN_ON+0x2f/0xc0)
Jul  6 01:09:08 astarte kernel: .. [<c014040c>] .... add_preempt_count+0x1c/0x20
Jul  6 01:09:08 astarte kernel: .....[<c014133b>] ..   ( <= print_traces+0x1b/0x60)
Jul  6 01:09:08 astarte kernel: 
Jul  6 01:09:09 astarte kernel: ------------------------------
Jul  6 01:09:09 astarte kernel: | showing all locks held by: |  (softirq-timer/0/3 [dffc2c90,  98]):
Jul  6 01:09:09 astarte kernel: ------------------------------

> do the 51-02 (and later) kernels work on the UP Athlon box?

I'll try out -51-06 (or later) on the UP Athlon when I get home.


--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[William Weston]

On Sun, 3 Jul 2005, Ingo Molnar wrote:

> ok, found a bug that could explain the situation: mutex sleeps+wakeups 
> were incorrectly credited as 'interactive sleep' periods, causing the dd 
> processes to be boosted incorrectly. The dd processes created a workload 
> in which they blocked each other in such a pattern that they got boosted 
> periodically, starving pretty much every other task.
> 
> the fix is significant and affects alot of workloads, and should further 
> improve interactivity in noticeable ways. I'm not 100% sure it solves 
> all the starvation problems (e.g. how could normal-prio dd tasks starve 
> the SCHED_FIFO irq threads that drove SysRq?), but the results so far 
> look promising.
> 
> i've uploaded the -50-45 patch, can you under this kernel trigger a 
> 'meltdown' on your SMT box?

Still looking into this issue on -51-06.  Found something really odd:  
SCHED_NORMAL tasks will start to inherit the priority value of some other
SCHED_FIFO task.  If JACK is started at a given SCHED_FIFO priority, X and
all of its children will inherit the same priority value after login.  
Other random processes will inherit this, too -- sometimes init...

SCHED_NORMAL tasks suddenly inheriting priority values in the range
normally reserved for SCHED_FIFO could explain at least part of the
meltdown I've been seeing.  Any thoughts?


Cheers,
--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* William Weston <weston@sysex.net> wrote:

> Still looking into this issue on -51-06.  Found something really odd: 
> SCHED_NORMAL tasks will start to inherit the priority value of some 
> other SCHED_FIFO task.  If JACK is started at a given SCHED_FIFO 
> priority, X and all of its children will inherit the same priority 
> value after login.  Other random processes will inherit this, too -- 
> sometimes init...
> 
> SCHED_NORMAL tasks suddenly inheriting priority values in the range 
> normally reserved for SCHED_FIFO could explain at least part of the 
> meltdown I've been seeing.  Any thoughts?

is this inheritance perpetual? It is normal for some tasks to be boosted 
momentarily, but if the condition remains even after jackd has exited, 
it's clearly an anomaly. (lets call it "RT priority leakage".) Priority 
leakage on SMP was fixed recently, but there could be other bugs 
remaining.

There's priority-leakage debugging code in the -RT kernel, which is 
activated if you enable CONFIG_DEBUG_PREEMPT. This debugging code, if 
triggered, should produce 'BUG: comm/123 leaked RT prio 123 (123)?" type 
of messages. But ... due to a bug it would not print out anything but 
would lock up hard if it detects the condition (and if 
RT_DEADLOCK_DETECT is enabled). I have fixed this reporting bug in the 
-51-08 kernel.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Steven Rostedt]

Hi Ingo,

I've just downloaded 51-09 and tried running it here on a normal intel
pentium4 box here at my customers site.  It included some hotplug PCI
modules (I don't know why since it's doesn't have hotplug devices) and I
got some init_MUTEX_LOCKED bugs on them.  Below you will find the patch (I
assume that compat_semaphore is still used).

Anyway, I also want to let you know that the e100 does not work.  It's
detected, but it wont bring up DHCP, and when I manually configued it, it
just froze (the process not the machine). But when I did a sysrq-t, the
machine froze up after it completed with some RT yeilding bug. Here's what
was last to spit out:

NETDEV WATCHDOG: eth0: transmit timed out
BUG: events/0:10 RT task yield()-ing!
 [<c01042ef>] dump_stack+0x1f/0x30 (20)
 [<c02e8b1a>] yield+0x5a/0x60 (20)
 [<c029bec5>] dev_deactivate+0x65/0x70 (20)
 [<c0296ae8>] linkwatch_run_queue+0xf8/0x110 (40)
 [<c0296b31>] linkwatch_event+0x31/0x70 (16)
 [<c013081b>] worker_thread+0x17b/0x230 (124)
 [<c01353de>] kthread+0xae/0xc0 (48)
 [<c0101445>] kernel_thread_helper+0x5/0x10 (814882844)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c014007a>] .... add_preempt_count+0x1a/0x20
.....[<c0140fcd>] ..   ( <= print_traces+0x1d/0x60)

------------------------------
| showing all locks held by: |  (events/0/10 [cf6ca030,  98]):
------------------------------

#001:             [c0390684] {rtnl_sem.lock}
... acquired at:  linkwatch_event+0x2c/0x70


This was right after the trace and all else froze. Anyway, I'll debug it
tomorrow since I'm about to go home for the day. And here's the patch for
the hotplug stuff (sorry about the -p0).

Index: drivers/pci/hotplug/ibmphp_hpc.c
===================================================================
--- drivers/pci/hotplug/ibmphp_hpc.c	(revision 237)
+++ drivers/pci/hotplug/ibmphp_hpc.c	(working copy)
@@ -104,7 +104,7 @@
 static struct semaphore sem_hpcaccess;	// lock access to HPC
 static struct semaphore semOperations;	// lock all operations and
 					// access to data structures
-static struct semaphore sem_exit;	// make sure polling thread goes away
+static struct compat_semaphore sem_exit;	// make sure polling thread goes away
 //----------------------------------------------------------------------------
 // local function prototypes
 //----------------------------------------------------------------------------
Index: drivers/pci/hotplug/cpqphp_ctrl.c
===================================================================
--- drivers/pci/hotplug/cpqphp_ctrl.c	(revision 237)
+++ drivers/pci/hotplug/cpqphp_ctrl.c	(working copy)
@@ -45,8 +45,8 @@
 			u8 behind_bridge, struct resource_lists *resources);
 static void interrupt_event_handler(struct controller *ctrl);

-static struct semaphore event_semaphore;	/* mutex for process loop (up if something to process) */
-static struct semaphore event_exit;		/* guard ensure thread has exited before calling it quits */
+static struct compat_semaphore event_semaphore;	/* mutex for process loop (up if something to process) */
+static struct compat_semaphore event_exit;		/* guard ensure thread has exited before calling it quits */
 static int event_finished;
 static unsigned long pushbutton_pending;	/* = 0 */

Index: drivers/pci/hotplug/shpchp_ctrl.c
===================================================================
--- drivers/pci/hotplug/shpchp_ctrl.c	(revision 237)
+++ drivers/pci/hotplug/shpchp_ctrl.c	(working copy)
@@ -47,8 +47,8 @@
 	u8 behind_bridge, struct resource_lists *resources, u8 bridge_bus, u8 bridge_dev);
 static void interrupt_event_handler(struct controller *ctrl);

-static struct semaphore event_semaphore;	/* mutex for process loop (up if something to process) */
-static struct semaphore event_exit;		/* guard ensure thread has exited before calling it quits */
+static struct compat_semaphore event_semaphore;	/* mutex for process loop (up if something to process) */
+static struct compat_semaphore event_exit;		/* guard ensure thread has exited before calling it quits */
 static int event_finished;
 static unsigned long pushbutton_pending;	/* = 0 */

Index: drivers/pci/hotplug/cpci_hotplug_core.c
===================================================================
--- drivers/pci/hotplug/cpci_hotplug_core.c	(revision 237)
+++ drivers/pci/hotplug/cpci_hotplug_core.c	(working copy)
@@ -60,8 +60,8 @@
 static atomic_t extracting;
 int cpci_debug;
 static struct cpci_hp_controller *controller;
-static struct semaphore event_semaphore;	/* mutex for process loop (up if something to process) */
-static struct semaphore thread_exit;		/* guard ensure thread has exited before calling it quits */
+static struct compat_semaphore event_semaphore;	/* mutex for process loop (up if something to process) */
+static struct compat_semaphore thread_exit;		/* guard ensure thread has exited before calling it quits */
 static int thread_finished = 1;

 static int enable_slot(struct hotplug_slot *slot);
Index: drivers/pci/hotplug/pciehp_ctrl.c
===================================================================
--- drivers/pci/hotplug/pciehp_ctrl.c	(revision 237)
+++ drivers/pci/hotplug/pciehp_ctrl.c	(working copy)
@@ -48,8 +48,8 @@
 	u8 behind_bridge, struct resource_lists *resources, u8 bridge_bus, u8 bridge_dev);
 static void interrupt_event_handler(struct controller *ctrl);

-static struct semaphore event_semaphore;	/* mutex for process loop (up if something to process) */
-static struct semaphore event_exit;		/* guard ensure thread has exited before calling it quits */
+static struct compat_semaphore event_semaphore;	/* mutex for process loop (up if something to process) */
+static struct compat_semaphore event_exit;		/* guard ensure thread has exited before calling it quits */
 static int event_finished;
 static unsigned long pushbutton_pending;	/* = 0 */
 static unsigned long surprise_rm_pending;	/* = 0 */

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> Hi Ingo,
> 
> I've just downloaded 51-09 and tried running it here on a normal intel 
> pentium4 box here at my customers site.  It included some hotplug PCI 
> modules (I don't know why since it's doesn't have hotplug devices) and 
> I got some init_MUTEX_LOCKED bugs on them.  Below you will find the 
> patch (I assume that compat_semaphore is still used).

thanks, i've applied it, and it's now in 51-11.

> Anyway, I also want to let you know that the e100 does not work.  It's 
> detected, but it wont bring up DHCP, and when I manually configued it, 
> it just froze (the process not the machine). But when I did a sysrq-t, 
> the machine froze up after it completed with some RT yeilding bug. 
> Here's what was last to spit out:

is PCI_MSI enabled by any chance? That is known to break level-triggered 
IOAPIC irqs and devices.

> NETDEV WATCHDOG: eth0: transmit timed out
> BUG: events/0:10 RT task yield()-ing!

i have replaced this yield() with msleep(1) in -51-11, but that's just 
fixing the symptom.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Steven Rostedt]

Darn subversion! I just started a massive commit, and I can't leave work
till it's done. So you still got me here ;-)

On Thu, 7 Jul 2005, Ingo Molnar wrote:
>
> > Anyway, I also want to let you know that the e100 does not work.  It's
> > detected, but it wont bring up DHCP, and when I manually configued it,
> > it just froze (the process not the machine). But when I did a sysrq-t,
> > the machine froze up after it completed with some RT yeilding bug.
> > Here's what was last to spit out:
>
> is PCI_MSI enabled by any chance? That is known to break level-triggered
> IOAPIC irqs and devices.
>

As a matter of fact it is...   I'll turn it off now and try it out.
If the commit is still going, I'll get you a response about the result.

-- Steve

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Steven Rostedt]

On Thu, 7 Jul 2005, Steven Rostedt wrote:
>
> Darn subversion! I just started a massive commit, and I can't leave work
> till it's done. So you still got me here ;-)

That commit is still going.  I can see why subversion was not used for
kernel development.


>
> On Thu, 7 Jul 2005, Ingo Molnar wrote:
> >
> > > Anyway, I also want to let you know that the e100 does not work.  It's
> > > detected, but it wont bring up DHCP, and when I manually configued it,
> > > it just froze (the process not the machine). But when I did a sysrq-t,
> > > the machine froze up after it completed with some RT yeilding bug.
> > > Here's what was last to spit out:
> >
> > is PCI_MSI enabled by any chance? That is known to break level-triggered
> > IOAPIC irqs and devices.
> >
>
> As a matter of fact it is...   I'll turn it off now and try it out.
> If the commit is still going, I'll get you a response about the result.
>

It did the trick.  I got a network. But I also got a hell of a lot of
'enqueued dead tasks'. But stupid me forgot to turn on capture in minicom,
and haven't been able to reproduce the problem. I rebooted the machine
which blew away all evidence of what occured, and it's now fine. I'll
reboot a few more times to see if I can get it to break again.

-- Steve

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> It did the trick.  I got a network. But I also got a hell of a lot of 
> 'enqueued dead tasks'. But stupid me forgot to turn on capture in 
> minicom, and haven't been able to reproduce the problem. I rebooted 
> the machine which blew away all evidence of what occured, and it's now 
> fine. I'll reboot a few more times to see if I can get it to break 
> again.

minicom has a scrollback feature (Alt-B), does that have the oops in 
history perhaps? It can cache a couple of bootups typically.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> > > is PCI_MSI enabled by any chance? That is known to break level-triggered
> > > IOAPIC irqs and devices.
> >
> > As a matter of fact it is...   I'll turn it off now and try it out.
> > If the commit is still going, I'll get you a response about the result.
> 
> It did the trick.  I got a network. [...]

ok, i found the PCI_MSI irq handling bug - the fix is in -51-12.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-06

[William Weston]

On Thu, 7 Jul 2005, Ingo Molnar wrote:

> > Still looking into this issue on -51-06.  Found something really odd: 
> > SCHED_NORMAL tasks will start to inherit the priority value of some 
> > other SCHED_FIFO task.  If JACK is started at a given SCHED_FIFO 
> > priority, X and all of its children will inherit the same priority 
> > value after login.  Other random processes will inherit this, too -- 
> > sometimes init...
> > 
> > SCHED_NORMAL tasks suddenly inheriting priority values in the range 
> > normally reserved for SCHED_FIFO could explain at least part of the 
> > meltdown I've been seeing.  Any thoughts?
> 
> is this inheritance perpetual? It is normal for some tasks to be boosted 
> momentarily, but if the condition remains even after jackd has exited, 
> it's clearly an anomaly. (lets call it "RT priority leakage".) Priority 
> leakage on SMP was fixed recently, but there could be other bugs 
> remaining.

Yes, this leakage perpetual.  That's what struck me as odd.

> There's priority-leakage debugging code in the -RT kernel, which is 
> activated if you enable CONFIG_DEBUG_PREEMPT. This debugging code, if 
> triggered, should produce 'BUG: comm/123 leaked RT prio 123 (123)?" type 
> of messages. But ... due to a bug it would not print out anything but 
> would lock up hard if it detects the condition (and if 
> RT_DEADLOCK_DETECT is enabled). I have fixed this reporting bug in the 
> -51-08 kernel.

-51-08 is building now.  I'll check out the latest, too.

--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-08

[William Weston]

On Wed, 6 Jul 2005, Ingo Molnar wrote:

> do the 51-02 (and later) kernels work on the UP Athlon box?

Hi Ingo,

-51-06 and -51-08 are looking stable on the UP Athlon box with the
following patch which causes edge triggered hardirqs to be masked when
pending _and/or_ disabled (instead of both pending _and_ disabled) and
backs out a change from -50-44 that prevents pending edge triggered irqs
from being unmasked:

--- linux.orig/arch/i386/kernel/io_apic.c	2005-07-06 11:08:39.000000000 -0700
+++ linux/arch/i386/kernel/io_apic.c	2005-07-07 03:40:37.000000000 -0700
@@ -1949,8 +1949,7 @@
 static void ack_edge_ioapic_irq(unsigned int irq)
 {
 	move_irq(irq);
-	if ((irq_desc[irq].status & (IRQ_PENDING | IRQ_DISABLED))
-					== (IRQ_PENDING | IRQ_DISABLED))
+	if (irq_desc[irq].status & (IRQ_PENDING | IRQ_DISABLED))
 		mask_IO_APIC_irq(irq);
 	ack_APIC_irq();
 }
@@ -1997,8 +1996,6 @@
 	unsigned long v;
 	int i;
 
-	if (!(irq_desc[irq].status & IRQ_DISABLED))
-		unmask_IO_APIC_irq(irq);
 /*
  * It appears there is an erratum which affects at least version 0x11
  * of I/O APIC (that's the 82093AA and cores integrated into various
--- linux.orig/kernel/irq/manage.c	2005-07-06 11:08:40.000000000 -0700
+++ linux/kernel/irq/manage.c	2005-07-07 01:04:38.000000000 -0700
@@ -434,6 +434,8 @@
 		 * The ->end() handler has to deal with interrupts which got
 		 * disabled while the handler was running.
 		 */
+		if (!(desc->status & IRQ_DISABLED))
+			desc->handler->enable(irq);
 		desc->handler->end(irq);
 	}
 	spin_unlock_irq(&desc->lock);

The first chunk is based on my understanding of the comments in io_apic.c,
and is not necessary to keep the UP Athlon box stable.  Is my logic OK 
here, or am I missing something really important?

Without the last two chunks of this patch, the UP Athlon box locks up hard
as soon as jackd is started up.

Best Regards,
--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-08

[Ingo Molnar]

* William Weston <weston@sysex.net> wrote:

> Without the last two chunks of this patch, the UP Athlon box locks up 
> hard as soon as jackd is started up.

hm, do you have CONFIG_PCI_MSI enabled by any chance?

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-08

[William Weston]

On Thu, 7 Jul 2005, Ingo Molnar wrote:

> > Without the last two chunks of this patch, the UP Athlon box locks up 
> > hard as soon as jackd is started up.
> 
> hm, do you have CONFIG_PCI_MSI enabled by any chance?

I've never enabled CONFIG_PCI_MSI.  What's your experience when it comes 
to stability and performance?

--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-08

[Ingo Molnar]

* William Weston <weston@sysex.net> wrote:

> On Thu, 7 Jul 2005, Ingo Molnar wrote:
> 
> > > Without the last two chunks of this patch, the UP Athlon box locks up 
> > > hard as soon as jackd is started up.
> > 
> > hm, do you have CONFIG_PCI_MSI enabled by any chance?
> 
> I've never enabled CONFIG_PCI_MSI.  What's your experience when it 
> comes to stability and performance?

it's really not needed unless you have PCI-X hardware, but it sometimes 
gets enabled and it had IRQ handling bugs under PREEMPT_HARDIRQS (and 
PREEMPT_RT), which i fixed in the -51-12 patch. But if you didnt have it 
enabled then there's no IO-APIC change affecting your config.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[William Weston]

On Thu, 7 Jul 2005, Ingo Molnar wrote:

> is this inheritance perpetual? It is normal for some tasks to be boosted 
> momentarily, but if the condition remains even after jackd has exited, 
> it's clearly an anomaly. (lets call it "RT priority leakage".) Priority 
> leakage on SMP was fixed recently, but there could be other bugs 
> remaining.
> 
> There's priority-leakage debugging code in the -RT kernel, which is 
> activated if you enable CONFIG_DEBUG_PREEMPT. This debugging code, if 
> triggered, should produce 'BUG: comm/123 leaked RT prio 123 (123)?" type 
> of messages. But ... due to a bug it would not print out anything but 
> would lock up hard if it detects the condition (and if 
> RT_DEADLOCK_DETECT is enabled). I have fixed this reporting bug in the 
> -51-08 kernel.

-51-12 is still exhibhiting the RT priority leakage, but isn't producing 
any BUG messages.

--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Chuck Harding]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1378 bytes --]

On Thu, 7 Jul 2005, Ingo Molnar wrote:

>
> * Steven Rostedt <rostedt@goodmis.org> wrote:
>
>>>> is PCI_MSI enabled by any chance? That is known to break level-triggered
>>>> IOAPIC irqs and devices.
>>>
>>> As a matter of fact it is...   I'll turn it off now and try it out.
>>> If the commit is still going, I'll get you a response about the result.
>>
>> It did the trick.  I got a network. [...]
>
> ok, i found the PCI_MSI irq handling bug - the fix is in -51-12.
>
> 	Ingo

Which still exhibits the lockup of sox. I built a 2.6.12 vanilla kernel
using the same .config as I used for 51-12 and the failure did not happen.
just the process of booting up causes later invocations of sox to lockup
in the D state. If I don't login to X and just run from a VT I can get
it to lockup by running something like:
$ for ((i=0;i<20;i++))
> do
> echo -ne "\r$i  "
> (play /usr/share/sounds/KDE_Beep_ShortBeep.wav &)
> done

after about 14 or so iterations of the loop and thereafter no more sound
can be played.

(my .config is attached)

-- 
Charles D. (Chuck) Harding <charding@llnl.gov>  Voice: 925-423-8879
Senior Computer Associate         ICCD            Fax: 925-423-6961
Lawrence Livermore National Laboratory      Computation Directorate
Livermore, CA USA  http://www.llnl.gov  GPG Public Key ID: B9EB6601
------------------ http://tinyurl.com/5w5ey -----------------------

[-- Attachment #2: config file for 51-12 --]
[-- Type: TEXT/PLAIN, Size: 32813 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.12-RT-V0.7.51-12
# Thu Jul  7 12:25:34 2005
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32

#
# General setup
#
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_SYSCTL=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
CONFIG_MPENTIUM4=y
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
CONFIG_X86_GENERIC=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
# CONFIG_SMP is not set
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_BKL=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_ASM_SEMAPHORES=y
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_IOAPIC_FAST=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
# CONFIG_X86_MCE_NONFATAL is not set
CONFIG_X86_MCE_P4THERMAL=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
CONFIG_MICROCODE=m
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m

#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_HIGHMEM=y
CONFIG_HIGHPTE=y
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_EFI is not set
CONFIG_HAVE_DEC_LOCK=y
CONFIG_REGPARM=y
CONFIG_SECCOMP=y

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SOFTWARE_SUSPEND is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SLEEP_PROC_FS=y
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=y
CONFIG_ACPI_FAN=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_IBM is not set
# CONFIG_ACPI_TOSHIBA is not set
CONFIG_ACPI_BLACKLIST_YEAR=2001
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set

#
# APM (Advanced Power Management) BIOS Support
#
# CONFIG_APM is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=y
# CONFIG_CPU_FREQ_STAT_DETAILS is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=m
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m

#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=y
# CONFIG_X86_POWERNOW_K6 is not set
# CONFIG_X86_POWERNOW_K7 is not set
# CONFIG_X86_POWERNOW_K8 is not set
# CONFIG_X86_GX_SUSPMOD is not set
CONFIG_X86_SPEEDSTEP_CENTRINO=y
CONFIG_X86_SPEEDSTEP_CENTRINO_ACPI=y
CONFIG_X86_SPEEDSTEP_CENTRINO_TABLE=y
CONFIG_X86_SPEEDSTEP_ICH=y
CONFIG_X86_SPEEDSTEP_SMI=y
CONFIG_X86_P4_CLOCKMOD=y
# CONFIG_X86_CPUFREQ_NFORCE2 is not set
# CONFIG_X86_LONGRUN is not set
# CONFIG_X86_LONGHAUL is not set

#
# shared options
#
# CONFIG_X86_ACPI_CPUFREQ_PROC_INTF is not set
CONFIG_X86_SPEEDSTEP_LIB=y
# CONFIG_X86_SPEEDSTEP_RELAXED_CAP_CHECK is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
# CONFIG_PCIEPORTBUS is not set
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY_PROC=y
# CONFIG_PCI_NAMES is not set
# CONFIG_PCI_DEBUG is not set
CONFIG_ISA_DMA_API=y
# CONFIG_ISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set

#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
# CONFIG_DEBUG_DRIVER is not set

#
# Memory Technology Devices (MTD)
#
CONFIG_MTD=m
# CONFIG_MTD_DEBUG is not set
# CONFIG_MTD_CONCAT is not set
# CONFIG_MTD_PARTITIONS is not set

#
# User Modules And Translation Layers
#
# CONFIG_MTD_CHAR is not set
# CONFIG_MTD_BLOCK is not set
# CONFIG_MTD_BLOCK_RO is not set
# CONFIG_FTL is not set
# CONFIG_NFTL is not set
# CONFIG_INFTL is not set

#
# RAM/ROM/Flash chip drivers
#
# CONFIG_MTD_CFI is not set
# CONFIG_MTD_JEDECPROBE is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
# CONFIG_MTD_RAM is not set
# CONFIG_MTD_ROM is not set
# CONFIG_MTD_ABSENT is not set

#
# Mapping drivers for chip access
#
# CONFIG_MTD_COMPLEX_MAPPINGS is not set

#
# Self-contained MTD device drivers
#
# CONFIG_MTD_PMC551 is not set
# CONFIG_MTD_SLRAM is not set
# CONFIG_MTD_PHRAM is not set
# CONFIG_MTD_MTDRAM is not set
# CONFIG_MTD_BLKMTD is not set
# CONFIG_MTD_BLOCK2MTD is not set

#
# Disk-On-Chip Device Drivers
#
# CONFIG_MTD_DOC2000 is not set
# CONFIG_MTD_DOC2001 is not set
# CONFIG_MTD_DOC2001PLUS is not set

#
# NAND Flash Device Drivers
#
# CONFIG_MTD_NAND is not set

#
# Parallel port support
#
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_1284=y

#
# Plug and Play support
#
# CONFIG_PNP is not set

#
# Block devices
#
# CONFIG_BLK_DEV_FD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_LBD=y
# CONFIG_CDROM_PKTCDVD is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_ATA_OVER_ETH is not set

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
CONFIG_BLK_DEV_IDEFLOPPY=y
# CONFIG_IDE_TASK_IOCTL is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
# CONFIG_BLK_DEV_CMD640 is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
CONFIG_BLK_DEV_PIIX=y
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
# CONFIG_SCSI is not set

#
# Multi-device support (RAID and LVM)
#
# CONFIG_MD is not set

#
# Fusion MPT device support
#

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Networking support
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
CONFIG_IP_ROUTE_MULTIPATH=y
# CONFIG_IP_ROUTE_MULTIPATH_CACHED is not set
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_TUNNEL=m
CONFIG_IP_TCPDIAG=y
# CONFIG_IP_TCPDIAG_IPV6 is not set

#
# IP: Virtual Server Configuration
#
CONFIG_IP_VS=m
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IPV6=m
CONFIG_IPV6_PRIVACY=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
CONFIG_INET6_TUNNEL=m
CONFIG_IPV6_TUNNEL=m
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_BRIDGE_NETFILTER=y

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
CONFIG_IP_NF_CT_ACCT=y
# CONFIG_IP_NF_CONNTRACK_MARK is not set
CONFIG_IP_NF_CT_PROTO_SCTP=m
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_AMANDA=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_IPRANGE=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_PHYSDEV=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_REALM=m
CONFIG_IP_NF_MATCH_SCTP=m
CONFIG_IP_NF_MATCH_COMMENT=m
# CONFIG_IP_NF_MATCH_HASHLIMIT is not set
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_TARGET_NOTRACK=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

#
# IPv6: Netfilter Configuration (EXPERIMENTAL)
#
# CONFIG_IP6_NF_QUEUE is not set
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_LIMIT=m
CONFIG_IP6_NF_MATCH_MAC=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_MULTIPORT=m
CONFIG_IP6_NF_MATCH_OWNER=m
CONFIG_IP6_NF_MATCH_MARK=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_AHESP=m
CONFIG_IP6_NF_MATCH_LENGTH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_PHYSDEV=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_TARGET_MARK=m
CONFIG_IP6_NF_RAW=m

#
# Bridge: Netfilter Configuration
#
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
# CONFIG_BRIDGE_EBT_ULOG is not set
CONFIG_XFRM=y
CONFIG_XFRM_USER=y

#
# SCTP Configuration (EXPERIMENTAL)
#
CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
# CONFIG_ATM_MPOA is not set
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_BRIDGE=m
CONFIG_VLAN_8021Q=m
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
CONFIG_NET_DIVERT=y
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set

#
# QoS and/or fair queueing
#
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CLK_JIFFIES=y
# CONFIG_NET_SCH_CLK_GETTIMEOFDAY is not set
# CONFIG_NET_SCH_CLK_CPU is not set
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
# CONFIG_NET_CLS_BASIC is not set
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_NET_CLS_IND=y
# CONFIG_CLS_U32_MARK is not set
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
# CONFIG_NET_EMATCH is not set
# CONFIG_NET_CLS_ACT is not set
CONFIG_NET_CLS_POLICE=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=m
CONFIG_TYPHOON=m

#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
# CONFIG_NET_PCI is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set

#
# ATM drivers
#
# CONFIG_ATM_TCP is not set
# CONFIG_ATM_LANAI is not set
# CONFIG_ATM_ENI is not set
# CONFIG_ATM_FIRESTREAM is not set
# CONFIG_ATM_ZATM is not set
# CONFIG_ATM_NICSTAR is not set
# CONFIG_ATM_IDT77252 is not set
# CONFIG_ATM_AMBASSADOR is not set
# CONFIG_ATM_HORIZON is not set
# CONFIG_ATM_IA is not set
# CONFIG_ATM_FORE200E_MAYBE is not set
# CONFIG_ATM_HE is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_INPUT_MOUSE=y
# CONFIG_MOUSE_PS2 is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_VSXXXAA is not set
CONFIG_INPUT_JOYSTICK=y
# CONFIG_JOYSTICK_ANALOG is not set
# CONFIG_JOYSTICK_A3D is not set
# CONFIG_JOYSTICK_ADI is not set
# CONFIG_JOYSTICK_COBRA is not set
# CONFIG_JOYSTICK_GF2K is not set
# CONFIG_JOYSTICK_GRIP is not set
# CONFIG_JOYSTICK_GRIP_MP is not set
# CONFIG_JOYSTICK_GUILLEMOT is not set
# CONFIG_JOYSTICK_INTERACT is not set
# CONFIG_JOYSTICK_SIDEWINDER is not set
# CONFIG_JOYSTICK_TMDC is not set
# CONFIG_JOYSTICK_IFORCE is not set
# CONFIG_JOYSTICK_WARRIOR is not set
# CONFIG_JOYSTICK_MAGELLAN is not set
# CONFIG_JOYSTICK_SPACEORB is not set
# CONFIG_JOYSTICK_SPACEBALL is not set
# CONFIG_JOYSTICK_STINGER is not set
# CONFIG_JOYSTICK_TWIDJOY is not set
# CONFIG_JOYSTICK_DB9 is not set
# CONFIG_JOYSTICK_GAMECON is not set
# CONFIG_JOYSTICK_TURBOGRAFX is not set
# CONFIG_JOYSTICK_JOYDUMP is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
CONFIG_INPUT_UINPUT=m

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
# CONFIG_SERIAL_8250_ACPI is not set
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
# CONFIG_SERIAL_8250_MANY_PORTS is not set
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
CONFIG_SERIAL_8250_MULTIPORT=y
CONFIG_SERIAL_8250_RSA=y

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
# CONFIG_PRINTER is not set
# CONFIG_PPDEV is not set
# CONFIG_TIPAR is not set

#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
# CONFIG_HW_RANDOM is not set
CONFIG_NVRAM=m
CONFIG_RTC=y
CONFIG_RTC_HISTOGRAM=y
CONFIG_BLOCKER=y
CONFIG_LPPTEST=m
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
CONFIG_AGP=y
# CONFIG_AGP_ALI is not set
CONFIG_AGP_ATI=y
# CONFIG_AGP_AMD is not set
# CONFIG_AGP_AMD64 is not set
CONFIG_AGP_INTEL=m
# CONFIG_AGP_NVIDIA is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_SWORKS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_AGP_EFFICEON is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
CONFIG_DRM_I810=m
CONFIG_DRM_I830=m
CONFIG_DRM_I915=m
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_MWAVE is not set
CONFIG_RAW_DRIVER=y
CONFIG_HPET=y
# CONFIG_HPET_RTC_IRQ is not set
CONFIG_HPET_MMAP=y
CONFIG_MAX_RAW_DEVS=8192
CONFIG_HANGCHECK_TIMER=y

#
# TPM devices
#
# CONFIG_TCG_TPM is not set

#
# I2C support
#
# CONFIG_I2C is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia devices
#
CONFIG_VIDEO_DEV=m

#
# Video For Linux
#

#
# Video Adapters
#
# CONFIG_VIDEO_BWQCAM is not set
# CONFIG_VIDEO_CQCAM is not set
# CONFIG_VIDEO_W9966 is not set
# CONFIG_VIDEO_CPIA is not set
# CONFIG_VIDEO_STRADIS is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_DPC is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set

#
# Radio Adapters
#
# CONFIG_RADIO_GEMTEK_PCI is not set
# CONFIG_RADIO_MAXIRADIO is not set
# CONFIG_RADIO_MAESTRO is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
# CONFIG_FB is not set
CONFIG_VIDEO_SELECT=y

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y

#
# Sound
#
CONFIG_SOUND=m

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=m
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set

#
# Generic devices
#
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
# CONFIG_SND_SERIAL_U16550 is not set
CONFIG_SND_MPU401=m

#
# PCI devices
#
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_ALI5451=m
CONFIG_SND_ATIIXP=m
CONFIG_SND_ATIIXP_MODEM=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
CONFIG_SND_CS4281=m
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_CA0106=m
CONFIG_SND_KORG1212=m
CONFIG_SND_MIXART=m
CONFIG_SND_NM256=m
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
CONFIG_SND_HDSP=m
CONFIG_SND_TRIDENT=m
CONFIG_SND_YMFPCI=m
CONFIG_SND_ALS4000=m
CONFIG_SND_CMIPCI=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
CONFIG_SND_MAESTRO3=m
CONFIG_SND_FM801=m
CONFIG_SND_FM801_TEA575X=m
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
CONFIG_SND_VX222=m
CONFIG_SND_HDA_INTEL=m

#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set

#
# USB support
#
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
CONFIG_USB_SUSPEND=y
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_SPLIT_ISO=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_BIG_ENDIAN is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_AUDIO is not set
# CONFIG_USB_BLUETOOTH_TTY is not set
CONFIG_USB_MIDI=m
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support' may also be needed; see USB_STORAGE Help for more information
#
# CONFIG_USB_STORAGE is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
CONFIG_USB_HIDINPUT=y
CONFIG_HID_FF=y
CONFIG_HID_PID=y
CONFIG_LOGITECH_FF=y
CONFIG_THRUSTMASTER_FF=y
CONFIG_USB_HIDDEV=y
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_MTOUCH is not set
# CONFIG_USB_EGALAX is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set

#
# USB Multimedia devices
#
# CONFIG_USB_DABUSB is not set
# CONFIG_USB_VICAM is not set
# CONFIG_USB_DSBR is not set
# CONFIG_USB_IBMCAM is not set
# CONFIG_USB_KONICAWC is not set
# CONFIG_USB_OV511 is not set
# CONFIG_USB_SE401 is not set
# CONFIG_USB_SN9C102 is not set
# CONFIG_USB_STV680 is not set
# CONFIG_USB_PWC is not set

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_MON is not set

#
# USB port drivers
#
# CONFIG_USB_USS720 is not set

#
# USB Serial Converter support
#
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGETKIT is not set
# CONFIG_USB_PHIDGETSERVO is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_TEST is not set

#
# USB ATM/DSL drivers
#
# CONFIG_USB_ATM is not set
# CONFIG_USB_SPEEDTOUCH is not set

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# MMC/SD Card support
#
# CONFIG_MMC is not set

#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y

#
# XFS support
#
# CONFIG_XFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_QUOTA=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_DNOTIFY=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
# CONFIG_MSDOS_FS is not set
# CONFIG_VFAT_FS is not set
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
# CONFIG_DEVFS_FS is not set
CONFIG_DEVPTS_FS_XATTR=y
CONFIG_DEVPTS_FS_SECURITY=y
CONFIG_TMPFS=y
CONFIG_TMPFS_XATTR=y
CONFIG_TMPFS_SECURITY=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_RAMFS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_JFFS_FS is not set
# CONFIG_JFFS2_FS is not set
CONFIG_CRAMFS=m
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_RPCSEC_GSS_SPKM3=m
CONFIG_SMB_FS=m
# CONFIG_SMB_NLS_DEFAULT is not set
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_EXPERIMENTAL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m

#
# Profiling support
#
CONFIG_PROFILING=y
CONFIG_OPROFILE=m

#
# Kernel hacking
#
# CONFIG_PRINTK_TIME is not set
# CONFIG_PRINTK_IGNORE_LOGLEVEL is not set
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_LOG_BUF_SHIFT=17
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_IRQ_FLAGS=y
# CONFIG_WAKEUP_TIMING is not set
CONFIG_PREEMPT_TRACE=y
# CONFIG_CRITICAL_PREEMPT_TIMING is not set
# CONFIG_CRITICAL_IRQSOFF_TIMING is not set
# CONFIG_RT_DEADLOCK_DETECT is not set
# CONFIG_DEBUG_RT_LOCKING_MODE is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_FS is not set
# CONFIG_USE_FRAME_POINTER is not set
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_KPROBES is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y

#
# Security options
#
# CONFIG_KEYS is not set
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_CAPABILITIES=y
# CONFIG_SECURITY_ROOTPLUG is not set
# CONFIG_SECURITY_SECLVL is not set
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1

#
# Cryptographic options
#
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_AES_586=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_KHAZAD=m
# CONFIG_CRYPTO_ANUBIS is not set
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_CRC32C=m
# CONFIG_CRYPTO_TEST is not set

#
# Hardware crypto devices
#
# CONFIG_CRYPTO_DEV_PADLOCK is not set

#
# Library routines
#
CONFIG_CRC_CCITT=m
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Ingo Molnar]

* Chuck Harding <charding@llnl.gov> wrote:

> Which still exhibits the lockup of sox. I built a 2.6.12 vanilla kernel
> using the same .config as I used for 51-12 and the failure did not happen.
> just the process of booting up causes later invocations of sox to lockup
> in the D state. If I don't login to X and just run from a VT I can get
> it to lockup by running something like:
> $ for ((i=0;i<20;i++))
> >do
> >echo -ne "\r$i  "
> >(play /usr/share/sounds/KDE_Beep_ShortBeep.wav &)
> >done
> 
> after about 14 or so iterations of the loop and thereafter no more 
> sound can be played.

tried it and cannot reproduce it, so i'll need the full backtrace of all 
tasks in your system, whenever sox gets stuck, via:

  echo t > /proc/sysrq-trigger
  dmesg -s 10000000 > toingo.txt

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Steven Rostedt]

On Thu, 7 Jul 2005, Ingo Molnar wrote:

>
> * Steven Rostedt <rostedt@goodmis.org> wrote:
>
> > It did the trick.  I got a network. But I also got a hell of a lot of
> > 'enqueued dead tasks'. But stupid me forgot to turn on capture in
> > minicom, and haven't been able to reproduce the problem. I rebooted
> > the machine which blew away all evidence of what occured, and it's now
> > fine. I'll reboot a few more times to see if I can get it to break
> > again.
>
> minicom has a scrollback feature (Alt-B), does that have the oops in
> history perhaps? It can cache a couple of bootups typically.
>

Is there also a scrollback for after you log out of minicom and turn off
the machine ;-)

Thanks, I didn't know about that feature of minicom. I should really start
reading the fine manuals.  Unfortunately, the minicom was on my laptop
which I turned off yesterday and only today did I see your note.

To try and reproduce it again, I've added in /etc/rc3.d an S98reboot that
will switch the system to runlevel 6 again, and repeat the process over
and over. All this while connect to minicom and capturing.  Hopefully it
will eventually show the same bug.  If not, maybe it was just a fluke.

-- Steve

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> tried it and cannot reproduce it, so i'll need the full backtrace of 
> all tasks in your system, whenever sox gets stuck, via:

managed to reproduce it via your script - and with RT_DEADLOCK_DETECT 
turned on the circular deadlock was immediately detected (see the trace 
below). It turns out this is an upstream locking bug in the sound 
subsystem, which has been fixed already - i've merged the upstream fix 
to -51-14. Could you check whether sox works for you now?

	Ingo

============================================
[ BUG: circular locking deadlock detected! ]
--------------------------------------------
sox/1828 is deadlocking current task sox/1824


1) sox/1824 is trying to acquire this lock:
 [cf073eac] {(struct semaphore *)(&card->power_lock)}
.. held by:               sox: 1828 [c901f9d0, 119]
... acquired at:          snd_pcm_prepare+0x22/0x6d
... trying at:            snd_pcm_drain+0x187/0x3a2

2) sox/1828 is blocked on this lock:
 [c04e4240] {snd_pcm_link_rwsem.lock}
.. held by:               sox: 1824 [ca3fc9d0, 115]
... acquired at:  snd_pcm_drain+0x37/0x3a2
------------------------------
| showing all locks held by: |  (sox/1824 [ca3fc9d0, 115]):
------------------------------

#001:             [c04e4240] {snd_pcm_link_rwsem.lock}
... acquired at:          snd_pcm_drain+0x37/0x3a2

------------------------------
| showing all locks held by: |  (sox/1828 [c901f9d0, 119]):
------------------------------

#001:             [cf073eac] {(struct semaphore *)(&card->power_lock)}
... acquired at:          snd_pcm_prepare+0x22/0x6d


sox/1828's [blocked] stackdump:

c9038d98 00000082 c05ae8cc c05ae420 0000000a 00000000 c9038dec 00000000 
       c9ac89d0 00011080 926e71c9 00000029 c9ac89d0 c901f9d0 c901faf8 c9038000 
       00000000 c9eef000 c9038db4 c0441321 c013c236 c9038db4 c013e9ab c9038000 
Call Trace:
 [<c0441321>] schedule+0x47/0x124 (28)
 [<c044289c>] __down+0x2f5/0x653 (108)
 [<c04433fc>] rt_down_read+0x26/0x2e (16)
 [<c030273f>] snd_pcm_action_nonatomic+0x21/0x83 (36)
 [<c0303263>] snd_pcm_prepare+0x5a/0x6d (32)
 [<c0305869>] snd_pcm_playback_ioctl1+0x4c/0x2a1 (68)
 [<c0305db9>] snd_pcm_kernel_playback_ioctl+0x36/0x40 (28)
 [<c0312953>] snd_pcm_oss_prepare+0x2a/0x5b (24)
 [<c03129bd>] snd_pcm_oss_make_ready+0x39/0x58 (20)
 [<c0313572>] snd_pcm_oss_sync+0x37/0x2b0 (64)
 [<c017dc2d>] do_ioctl+0x7d/0x8c (36)
 [<c017ddc4>] vfs_ioctl+0x61/0x1c2 (40)
 [<c017df96>] sys_ioctl+0x71/0x7f (40)
 [<c01030e8>] sysenter_past_esp+0x61/0x89 (-4020)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c013e852>] .... add_preempt_count+0x1a/0x1c
.....[<c0440c1f>] ..   ( <= __schedule+0x4f/0x70a)
.. [<c013e852>] .... add_preempt_count+0x1a/0x1c
.....[<c0440cb1>] ..   ( <= __schedule+0xe1/0x70a)

------------------------------
| showing all locks held by: |  (sox/1828 [c901f9d0, 119]):
------------------------------

#001:             [cf073eac] {(struct semaphore *)(&card->power_lock)}
... acquired at:         snd_pcm_prepare+0x22/0x6d


sox/1824's [current] stackdump:

 [<c0104082>] dump_stack+0x1f/0x21 (20)
 [<c0138206>] check_deadlock+0x20f/0x32e (48)
 [<c0138f7c>] task_blocks_on_lock+0x3a/0xe1 (44)
 [<c04427ee>] __down+0x247/0x653 (108)
 [<c013af13>] rt_down+0x29/0x45d (44)
 [<c030352d>] snd_pcm_drain+0x187/0x3a2 (96)
 [<c0305869>] snd_pcm_playback_ioctl1+0x4c/0x2a1 (68)
 [<c0305db9>] snd_pcm_kernel_playback_ioctl+0x36/0x40 (28)
 [<c031361a>] snd_pcm_oss_sync+0xdf/0x2b0 (64)
 [<c0314b25>] snd_pcm_oss_release+0x2b/0xd8 (28)
 [<c016b511>] __fput+0x13d/0x181 (36)
 [<c0169d43>] filp_close+0x4f/0x83 (28)
 [<c0169ddd>] sys_close+0x66/0x81 (32)
 [<c01030e8>] sysenter_past_esp+0x61/0x89 (-4020)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c013e852>] .... add_preempt_count+0x1a/0x1c
.....[<c013f969>] ..   ( <= print_traces+0x1b/0x50)

------------------------------
| showing all locks held by: |  (sox/1824 [ca3fc9d0, 116]):
------------------------------

#001:             [c04e4240] {snd_pcm_link_rwsem.lock}
... acquired at:  snd_pcm_drain+0x37/0x3a2


showing all tasks:
S            init:    1 [cff1c9d0, 116] (not blocked)
S  softirq-high/0:    2 [cff209d0, 110] (not blocked)
S softirq-timer/0:    3 [cff249d0, 105] (not blocked)
S softirq-net-tx/:    4 [cff289d0, 110] (not blocked)
S softirq-net-rx/:    5 [cff2c9d0, 105] (not blocked)
S  softirq-scsi/0:    6 [cff309d0, 110] (not blocked)
S softirq-tasklet:    7 [cff349d0, 105] (not blocked)
S       desched/0:    8 [cff389d0, 105] (not blocked)
S      watchdog/0:    9 [cff3c9d0,   0] (not blocked)
S        events/0:   10 [cff5d9d0,  98] (not blocked)
S         khelper:   11 [cff619d0, 111] (not blocked)
S         kthread:   12 [cfe789d0, 110] (not blocked)
S       kblockd/0:   14 [cfea39d0, 110] (not blocked)
S           khubd:   17 [cfa039d0, 125] (not blocked)
S         pdflush:  133 [cfa0e9d0, 120] (not blocked)
S         pdflush:  134 [cf4929d0, 115] (not blocked)
S           aio/0:  136 [cf4a29d0, 112] (not blocked)
S         kswapd0:  135 [cf4969d0, 125] (not blocked)
S           IRQ 8:  207 [cf54e9d0,  50] (not blocked)
S          IRQ 12:  217 [cf6c49d0,  51] (not blocked)
S         kseriod:  214 [cf69f9d0, 115] (not blocked)
S          IRQ 14:  255 [cf6d89d0,  52] (not blocked)
S          IRQ 15:  257 [cf7149d0,  53] (not blocked)
S           IRQ 1:  279 [cf0849d0,  54] (not blocked)
S          IRQ 17:  297 [cf0b29d0,  55] (not blocked)
S       kjournald:  336 [cf16e9d0, 115] (not blocked)
S           udevd: 1055 [ceb7c9d0, 110] (not blocked)
S          IRQ 16: 1230 [ce1199d0,  56] (not blocked)
S         syslogd: 1284 [ceba39d0, 117] (not blocked)
S           klogd: 1286 [cf0d39d0, 116] (not blocked)
S            sshd: 1306 [ce7759d0, 116] (not blocked)
S             gpm: 1316 [ce7fc9d0, 115] (not blocked)
S           crond: 1404 [ce4b89d0, 118] (not blocked)
S             xfs: 1428 [ce75e9d0, 118] (not blocked)
S             atd: 1443 [ce4949d0, 119] (not blocked)
S     dbus-daemon: 1451 [ce9dc9d0, 116] (not blocked)
S     dbus-daemon: 1452 [ce1a59d0, 121] (not blocked)
S        mingetty: 1532 [cda6f9d0, 118] (not blocked)
S        mingetty: 1533 [caaf29d0, 116] (not blocked)
S        mingetty: 1534 [ce9af9d0, 119] (not blocked)
S        mingetty: 1548 [caa979d0, 120] (not blocked)
S        mingetty: 1549 [ca2d89d0, 121] (not blocked)
S        mingetty: 1582 [c9e449d0, 120] (not blocked)
S            sshd: 1605 [ca3149d0, 116] (not blocked)
S            sshd: 1701 [c9e2b9d0, 116] (not blocked)
S            bash: 1722 [ca2729d0, 116] (not blocked)
S              su: 1792 [ceb1c9d0, 117] (not blocked)
S            bash: 1793 [c9e019d0, 116] (not blocked)
R            bash: 1809 [ca3d89d0, 119] (not blocked)
S            play: 1811 [ca8269d0, 117] (not blocked)
S            play: 1813 [c9e159d0, 118] (not blocked)
R            play: 1815 [ca2ea9d0, 118] (not blocked)
R            play: 1817 [c9ac89d0, 118] (not blocked)
S            play: 1819 [caa319d0, 122] (not blocked)
S            play: 1821 [ca08a9d0, 123] (not blocked)
D             sox: 1824 [ca3fc9d0, 116] (not blocked)
D             sox: 1828 [c901f9d0, 119] blocked on: [c04e4240] {snd_pcm_link_rwsem.lock}
.. held by:               sox: 1824 [ca3fc9d0, 116]
... acquired at:  snd_pcm_drain+0x37/0x3a2
R             sox: 1832 [ca0d79d0, 125] (not blocked)
D            bash: 1833 [c9c389d0, 120] blocked on: [c04cb004] {kernel_sem.lock}
.. held by:              play: 1817 [c9ac89d0, 118]
... acquired at:  lock_kernel+0x25/0x38
S             sox: 1835 [c96669d0, 118] blocked on: [c9efbcd0] {(struct semaphore *)(&tty->atomic_write)}
.. held by:              bash: 1809 [ca3d89d0, 119]
... acquired at:  tty_write+0xb6/0x20d
D            play: 1837 [c9b749d0, 120] blocked on: [c04cb004] {kernel_sem.lock}
.. held by:              play: 1817 [c9ac89d0, 118]
... acquired at:  lock_kernel+0x25/0x38
R            bash: 1839 [c9b819d0, 121] (not blocked)

---------------------------
| showing all locks held: |
---------------------------

#001:             [c07aaa80] {(struct semaphore *)(&hwif->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x93/0x162

#002:             [c07aa6ac] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#003:             [c07aa87c] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#004:             [c07ab0c0] {(struct semaphore *)(&hwif->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x93/0x162

#005:             [c07aacec] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#006:             [c07aaebc] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#007:             [c07ab700] {(struct semaphore *)(&hwif->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x93/0x162

#008:             [c07ab32c] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#009:             [c07ab4fc] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#010:             [c07abd40] {(struct semaphore *)(&hwif->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x93/0x162

#011:             [c07ab96c] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#012:             [c07abb3c] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#013:             [c07ac380] {(struct semaphore *)(&hwif->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x93/0x162

#014:             [c07abfac] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#015:             [c07ac17c] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#016:             [c07ac9c0] {(struct semaphore *)(&hwif->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x93/0x162

#017:             [c07ac5ec] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#018:             [c07ac7bc] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#019:             [c07ad000] {(struct semaphore *)(&hwif->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x93/0x162

#020:             [c07acc2c] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#021:             [c07acdfc] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#022:             [c07ad640] {(struct semaphore *)(&hwif->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x93/0x162

#023:             [c07ad26c] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#024:             [c07ad43c] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#025:             [c07adc80] {(struct semaphore *)(&hwif->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x93/0x162

#026:             [c07ad8ac] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#027:             [c07ada7c] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#028:             [c07ae2c0] {(struct semaphore *)(&hwif->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x93/0x162

#029:             [c07adeec] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#030:             [c07ae0bc] {(struct semaphore *)(&drive->gendev_rel_sem)}
.. held by:              init:    1 [cff1c9d0, 116]
... acquired at:  init_hwif_data+0x151/0x162

#031:             [ceb0bc98] {(struct semaphore *)(&tty->atomic_read)}
.. held by:          mingetty: 1532 [cda6f9d0, 118]
... acquired at:  read_chan+0x3d2/0x632

#032:             [c9ac9c98] {(struct semaphore *)(&tty->atomic_read)}
.. held by:          mingetty: 1533 [caaf29d0, 116]
... acquired at:  read_chan+0x3d2/0x632

#033:             [cba8cc98] {(struct semaphore *)(&tty->atomic_read)}
.. held by:          mingetty: 1534 [ce9af9d0, 119]
... acquired at:  read_chan+0x3d2/0x632

#034:             [c9ed3c98] {(struct semaphore *)(&tty->atomic_read)}
.. held by:          mingetty: 1548 [caa979d0, 120]
... acquired at:  read_chan+0x3d2/0x632

#035:             [c9f02c98] {(struct semaphore *)(&tty->atomic_read)}
.. held by:          mingetty: 1582 [c9e449d0, 120]
... acquired at:  read_chan+0x3d2/0x632

#036:             [cba50c98] {(struct semaphore *)(&tty->atomic_read)}
.. held by:          mingetty: 1549 [ca2d89d0, 121]
... acquired at:  read_chan+0x3d2/0x632

#037:             [c04e4240] {snd_pcm_link_rwsem.lock}
.. held by:               sox: 1824 [ca3fc9d0, 117]
... acquired at:  snd_pcm_drain+0x37/0x3a2

#038:             [cf073eac] {(struct semaphore *)(&card->power_lock)}
.. held by:               sox: 1828 [c901f9d0, 119]
... acquired at:  snd_pcm_prepare+0x22/0x6d

#039:             [c9efbcd0] {(struct semaphore *)(&tty->atomic_write)}
.. held by:              bash: 1809 [ca3d89d0, 119]
... acquired at:  tty_write+0xb6/0x20d

#040:             [c04cb004] {kernel_sem.lock}
.. held by:              play: 1817 [c9ac89d0, 118]
... acquired at:  lock_kernel+0x25/0x38

#041:             [c0565a00] {tasklist_lock}
.. held by:               sox: 1824 [ca3fc9d0, 117]
... acquired at:  _read_trylock+0x13/0x15

=============================================

[ turning off deadlock detection. Please report this trace. ]

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Ingo Molnar]

* William Weston <weston@sysex.net> wrote:

> -51-12 is still exhibhiting the RT priority leakage, but isn't 
> producing any BUG messages.

could you check whether the priority leakage happens if you disable SMP?  
(if you can reproduce it easily)

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-08

[Ingo Molnar]

* William Weston <weston@sysex.net> wrote:

> -51-06 and -51-08 are looking stable on the UP Athlon box with the 
> following patch which causes edge triggered hardirqs to be masked when 
> pending _and/or_ disabled (instead of both pending _and_ disabled) and 
> backs out a change from -50-44 that prevents pending edge triggered 
> irqs from being unmasked:

ok, you are right, the edge case was mishandled - but i think it was 
already mishandled upstream, we just never (or rarely) triggered it.  
I've reworked this area based on your patch, could you check -51-15, 
does it work for you?

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-08

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > -51-06 and -51-08 are looking stable on the UP Athlon box with the 
> > following patch which causes edge triggered hardirqs to be masked when 
> > pending _and/or_ disabled (instead of both pending _and_ disabled) and 
> > backs out a change from -50-44 that prevents pending edge triggered 
> > irqs from being unmasked:
> 
> ok, you are right, the edge case was mishandled - but i think it was 
> already mishandled upstream, we just never (or rarely) triggered it.  
> I've reworked this area based on your patch, could you check -51-15, 
> does it work for you?

if it doesnt work, could you disable CONFIG_X86_IOAPIC_FAST?

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-08

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > > -51-06 and -51-08 are looking stable on the UP Athlon box with the 
> > > following patch which causes edge triggered hardirqs to be masked when 
> > > pending _and/or_ disabled (instead of both pending _and_ disabled) and 
> > > backs out a change from -50-44 that prevents pending edge triggered 
> > > irqs from being unmasked:
> > 
> > ok, you are right, the edge case was mishandled - but i think it was 
> > already mishandled upstream, we just never (or rarely) triggered it.  
> > I've reworked this area based on your patch, could you check -51-15, 
> > does it work for you?
> 
> if it doesnt work, could you disable CONFIG_X86_IOAPIC_FAST?

or rather, please try -51-16. One of my testsystems produced weird 
interrupt storms, which were caused by the IOAPIC_POSTFLUSH 
optimization. In -51-16 i've turned that optimization off, globally.  
Maybe this explains some of the other lockups reported.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Steven Rostedt]

On Fri, 8 Jul 2005, Steven Rostedt wrote:

>
> To try and reproduce it again, I've added in /etc/rc3.d an S98reboot that
> will switch the system to runlevel 6 again, and repeat the process over
> and over. All this while connect to minicom and capturing.  Hopefully it
> will eventually show the same bug.  If not, maybe it was just a fluke.
>

OK, I just rebooted the machine a 100 times, and not one error.  Maybe it
was just because the system failed on an earlier boot?  I don't know, but
I can't spend any more time on it.  I'll keep my minicom's active at all
times now.

-- Steve

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> > To try and reproduce it again, I've added in /etc/rc3.d an S98reboot that
> > will switch the system to runlevel 6 again, and repeat the process over
> > and over. All this while connect to minicom and capturing.  Hopefully it
> > will eventually show the same bug.  If not, maybe it was just a fluke.
> 
> OK, I just rebooted the machine a 100 times, and not one error.  Maybe 
> it was just because the system failed on an earlier boot?  I don't 
> know, but I can't spend any more time on it.  I'll keep my minicom's 
> active at all times now.

hm, the "dead task enqueued!" is a real message that signals some real 
nastiness. It should not occur under any circumstance - nor have i seen 
it during the past couple of months. So lets keep an eye on it, but if 
100 reboots didnt trigger it there's not much we can do but wait and 
see.

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Chuck Harding]

On Fri, 8 Jul 2005, Ingo Molnar wrote:

>
> * Ingo Molnar <mingo@elte.hu> wrote:
>
>> tried it and cannot reproduce it, so i'll need the full backtrace of
>> all tasks in your system, whenever sox gets stuck, via:
>
> managed to reproduce it via your script - and with RT_DEADLOCK_DETECT
> turned on the circular deadlock was immediately detected (see the trace
> below). It turns out this is an upstream locking bug in the sound
> subsystem, which has been fixed already - i've merged the upstream fix
> to -51-14. Could you check whether sox works for you now?
>
> 	Ingo
>

That did it. I can now hammer the sound system and sox won't hang anymore.
Thanks!!

-- 
Charles D. (Chuck) Harding <charding@llnl.gov>  Voice: 925-423-8879
Senior Computer Associate         ICCD            Fax: 925-423-6961
Lawrence Livermore National Laboratory      Computation Directorate
Livermore, CA USA  http://www.llnl.gov  GPG Public Key ID: B9EB6601
------------------ http://tinyurl.com/5w5ey -----------------------
-- You've certainly got smooth skin - between the wrinkles, that is. --

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[William Weston]

On Fri, 8 Jul 2005, Ingo Molnar wrote:

> could you check whether the priority leakage happens if you disable SMP?  
> (if you can reproduce it easily)

No priority leakages have been seen with UP configs on any of the 
machines I've been testing.

The leakage is not hard to reproduce under SMT:  start up jackd from a
text vc with an rt prio of 60 (or some unique prio above the IRQ threads),
then restart X and login.  After several minutes, X and all of its
children will be running at whatever prio jackd was started at (but still
SCHED_NORMAL).  Eventually, init, a handful of SCHED_NORMAL kernel
threads, and other random processes are all running at the same priority.  
When reset to default priority with chrt or schedtool, these processes
eventually revert back to the leaked priority level.  When jackd is
stopped, the priorities stay in their elevated state.  If jackd is not 
started before logging in to X, then the priority of one of the SCHED_FF 
kernel threads is leaked to other processes in the same manner.

--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-17

[William Weston]

On Fri, 8 Jul 2005, Ingo Molnar wrote:

> > > ok, you are right, the edge case was mishandled - but i think it was 
> > > already mishandled upstream, we just never (or rarely) triggered it.  
> > > I've reworked this area based on your patch, could you check -51-15, 
> > > does it work for you?

I figured it would take a little reworking, especially with your recent
MSI changes.  This was just a 'works for me' patch.  Your end_irq() is
cleaner, IMHO.

> > if it doesnt work, could you disable CONFIG_X86_IOAPIC_FAST?
> 
> or rather, please try -51-16. One of my testsystems produced weird 
> interrupt storms, which were caused by the IOAPIC_POSTFLUSH 
> optimization. In -51-16 i've turned that optimization off, globally.  
> Maybe this explains some of the other lockups reported.

OK.  Running -51-17 on the Athlon box.  CONFIG_X86_IOAPIC_FAST is still
enabled.  No lockups.  No jack xruns.  Max wakeup-latency is 27us (so far)  
on the debug config.  Nothing out of the ordinary logged except for the
new stack-footprint maximums during boot.

Any need to go back and check -51-16 as well?  Or -51-17 on a non-debug 
config?

--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-45

[Bill Davidsen]

Ingo Molnar wrote:
> * Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> 
>>It did the trick.  I got a network. But I also got a hell of a lot of 
>>'enqueued dead tasks'. But stupid me forgot to turn on capture in 
>>minicom, and haven't been able to reproduce the problem. I rebooted 
>>the machine which blew away all evidence of what occured, and it's now 
>>fine. I'll reboot a few more times to see if I can get it to break 
>>again.
> 
> 
> minicom has a scrollback feature (Alt-B), does that have the oops in 
> history perhaps? It can cache a couple of bootups typically.

Although the kermit protocol is not much used anymore, the kermit 
program is still a capable terminal interface, and includes logging. For 
those who don't like telnet or minicom, here's another program to dislike.
-- 
bill davidsen <davidsen@tmr.com>
   CTO TMR Associates, Inc
   Doing interesting things with small computers since 1979

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-17

[William Weston]

Hi Ingo,

More SMT weirdness.  Latency traces aren't looking quite right on -51-17 
with my SMT debug config.

Started jackd, immediately after boot (before logging in to X), then:

[root@manzanita ~]# cat /proc/latency_trace
cat/2149[CPU#1]: BUG in update_out_trace at kernel/latency.c:698
 [<c01041a3>] dump_stack+0x23/0x30 (20)
 [<c012317b>] __WARN_ON+0x6b/0x90 (52)
 [<c0142237>] update_out_trace+0x467/0x4e0 (100)
 [<c0142590>] l_start+0x2e0/0x310 (72)
 [<c019415b>] seq_read+0x7b/0x2d0 (60)
 [<c0170f74>] vfs_read+0xd4/0x140 (36)
 [<c0171250>] sys_read+0x50/0x80 (44)
 [<c01031b0>] sysenter_past_esp+0x61/0x89 (-8116)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c01434dc>] .... add_preempt_count+0x1c/0x20
.....[<c0123137>] ..   ( <= __WARN_ON+0x27/0x90)
.. [<c01434dc>] .... add_preempt_count+0x1c/0x20
.....[<c014454d>] ..   ( <= print_traces+0x1d/0x60)

------------------------------
| showing all locks held by: |  (cat/2149 [dda0e030,  39]):
------------------------------

#001:             [db76afa4] {(struct semaphore *)(&p->sem)}
... acquired at:               seq_read+0x2b/0x2d0

#002:             [c05fe424] {out_mutex.lock}
... acquired at:               l_start+0x1b/0x310

#003:             [c06d4404] {max_mutex.lock}
... acquired at:               l_start+0x2db/0x310

CPU0: 0000003e0d590d0c (0000003e0d593ba8) ... #50 (0000003e0d610158) 0000003e0d611a78
CPU1: 0000003e0d6122c8 (0000003e0d6125b0) ... #4 (0000003e0d612dac) 0000003e0d6136bc
CPU0 entries: 50
first stamp: 0000003e0d6122c8
 last stamp: 0000003e0d6122c8
preemption latency trace v1.1.4 on 2.6.12-RT-V0.7.51-17-debug
--------------------------------------------------------------------
 latency: 159 us, #54/54, CPU#0 | (M:rt VP:0, KP:1, SP:1 HP:1 #P:2)
    -----------------
    | task: softirq-timer/0-4 (uid:0 nice:-10 policy:0 rt_prio:0)
    -----------------

                 _------=> CPU#
                / _-----=> irqs-off
               | / _----=> need-resched
               || / _---=> hardirq/softirq
               ||| / _--=> preempt-depth
               |||| /
               |||||     delay
   cmd     pid ||||| time  |   caller
      \   /    |||||   \   |   /


vim:ft=help


After logging into X:

<15:26:38 root@manzanita ~># cat /proc/latency_trace 
preemption latency trace v1.1.4 on 2.6.12-RT-V0.7.51-17-debug
--------------------------------------------------------------------
 latency: 202 us, #102/102, CPU#0 | (M:rt VP:0, KP:1, SP:1 HP:1 #P:2)
    -----------------
    | task: jackd-1929 (uid:500 nice:0 policy:1 rt_prio:60)
    -----------------

                 _------=> CPU#            
                / _-----=> irqs-off        
               | / _----=> need-resched    
               || / _---=> hardirq/softirq 
               ||| / _--=> preempt-depth   
               |||| /                      
               |||||     delay             
   cmd     pid ||||| time  |   caller      
      \   /    |||||   \   |   /           
   <...>-1817  1....    0us : set_palette (redraw_screen)
   IRQ 5-724   0Dnh2    0us : find_next_bit (pull_rt_tasks)
   <...>-1817  1....    0us : is_console_locked (set_palette)
   IRQ 5-724   0Dnh2    0us : dependent_sleeper (__schedule)
   <...>-1817  1....    0us : vgacon_set_palette (set_palette)
   IRQ 5-724   0Dnh2    1us : _raw_spin_unlock (dependent_sleeper)
   <...>-1817  1....    1us : vga_set_palette (vgacon_set_palette)
   IRQ 5-724   0Dnh1    1us : preempt_schedule (dependent_sleeper)
   IRQ 5-724   0Dnh1    1us : _raw_spin_lock (dependent_sleeper)
   IRQ 5-724   0Dnh2    2us : find_next_bit (dependent_sleeper)
   IRQ 5-724   0Dnh2    2us : _raw_spin_lock (dependent_sleeper)
   IRQ 5-724   0Dnh3    2us : find_next_bit (dependent_sleeper)
   IRQ 5-724   0Dnh3    3us : find_next_bit (dependent_sleeper)
   IRQ 5-724   0Dnh3    3us : _raw_spin_unlock (dependent_sleeper)
   IRQ 5-724   0Dnh2    3us : preempt_schedule (dependent_sleeper)
   IRQ 5-724   0Dnh2    3us : find_next_bit (dependent_sleeper)
   IRQ 5-724   0Dnh2    4us : trace_array (__schedule)
   IRQ 5-724   0Dnh2    4us : trace_array <<...>-1929> (27 27)
   IRQ 5-724   0Dnh2    5us : trace_array <IRQ 5-724> (31 31)
   IRQ 5-724   0Dnh2    6us : trace_array (__schedule)
   <...>-1929  0Dnh2    8us+: __switch_to (__schedule)
   <...>-1929  0Dnh2   11us : __schedule <IRQ 5-724> (31 27)
   <...>-1929  0Dnh2   11us : finish_task_switch (__schedule)
   <...>-1929  0Dnh2   13us : smp_send_reschedule_allbutself (finish_task_switch)
   <...>-1929  0Dnh2   13us : __bitmap_weight (smp_send_reschedule_allbutself)
   <...>-1929  0Dnh2   13us : __send_IPI_shortcut (smp_send_reschedule_allbutself)
   <...>-1929  0Dnh2   13us : _raw_spin_unlock (finish_task_switch)
   <...>-1929  0Dnh1   15us : trace_stop_sched_switched (finish_task_switch)
   <...>-1929  0Dnh1   15us : _raw_spin_lock (trace_stop_sched_switched)
   <...>-1929  0Dnh2   15us : trace_stop_sched_switched <<...>-1929> (27 0)
   <...>-1817  1Dnh.   15us : smp_reschedule_interrupt (c026c17a 0 0)
   <...>-1929  0Dnh3   16us : _raw_spin_unlock (trace_stop_sched_switched)
   <...>-1817  1Dnh.   16us : preempt_schedule_irq (need_resched)
   <...>-1929  0Dnh2   16us!: trace_stop_sched_switched (finish_task_switch)
   IRQ 5-724   0Dnh2    0us : find_next_bit (pull_rt_tasks)
   IRQ 5-724   0Dnh2    0us : dependent_sleeper (__schedule)
   IRQ 5-724   0Dnh2    1us : _raw_spin_unlock (dependent_sleeper)
   IRQ 5-724   0Dnh1    1us : preempt_schedule (dependent_sleeper)
   IRQ 5-724   0Dnh1    1us : _raw_spin_lock (dependent_sleeper)
   IRQ 5-724   0Dnh2    2us : find_next_bit (dependent_sleeper)
   IRQ 5-724   0Dnh2    2us : _raw_spin_lock (dependent_sleeper)
   IRQ 5-724   0Dnh3    2us : find_next_bit (dependent_sleeper)
   IRQ 5-724   0Dnh3    3us : find_next_bit (dependent_sleeper)
   IRQ 5-724   0Dnh3    3us : _raw_spin_unlock (dependent_sleeper)
   IRQ 5-724   0Dnh2    3us : preempt_schedule (dependent_sleeper)
   IRQ 5-724   0Dnh2    3us : find_next_bit (dependent_sleeper)
   IRQ 5-724   0Dnh2    4us : trace_array (__schedule)
   IRQ 5-724   0Dnh2    4us : trace_array <<...>-1929> (27 27)
   IRQ 5-724   0Dnh2    5us : trace_array <IRQ 5-724> (31 31)
   IRQ 5-724   0Dnh2    6us : trace_array (__schedule)
   <...>-1929  0Dnh2    8us+: __switch_to (__schedule)
   <...>-1929  0Dnh2   11us : __schedule <IRQ 5-724> (31 27)
   <...>-1929  0Dnh2   11us : finish_task_switch (__schedule)
   <...>-1929  0Dnh2   13us : smp_send_reschedule_allbutself (finish_task_switch)
   <...>-1929  0Dnh2   13us : __bitmap_weight (smp_send_reschedule_allbutself)
   <...>-1929  0Dnh2   13us : __send_IPI_shortcut (smp_send_reschedule_allbutself)
   <...>-1929  0Dnh2   13us : _raw_spin_unlock (finish_task_switch)
   <...>-1929  0Dnh1   15us : trace_stop_sched_switched (finish_task_switch)
   <...>-1929  0Dnh1   15us : _raw_spin_lock (trace_stop_sched_switched)
   <...>-1929  0Dnh2   15us : trace_stop_sched_switched <<...>-1929> (27 0)
   <...>-1929  0Dnh3   16us : _raw_spin_unlock (trace_stop_sched_switched)
   <...>-1929  0Dnh2   16us!: trace_stop_sched_switched (finish_task_switch)


vim:ft=help


--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Peter Zijlstra]

On Fri, 2005-07-08 at 13:12 -0700, William Weston wrote:
> On Fri, 8 Jul 2005, Ingo Molnar wrote:
> 
> > could you check whether the priority leakage happens if you disable SMP?  
> > (if you can reproduce it easily)
> 
> No priority leakages have been seen with UP configs on any of the 
> machines I've been testing.
> 
> The leakage is not hard to reproduce under SMT:  start up jackd from a
> text vc with an rt prio of 60 (or some unique prio above the IRQ threads),
> then restart X and login.  After several minutes, X and all of its
> children will be running at whatever prio jackd was started at (but still
> SCHED_NORMAL).  Eventually, init, a handful of SCHED_NORMAL kernel
> threads, and other random processes are all running at the same priority.  
> When reset to default priority with chrt or schedtool, these processes
> eventually revert back to the leaked priority level.  When jackd is
> stopped, the priorities stay in their elevated state.  If jackd is not 
> started before logging in to X, then the priority of one of the SCHED_FF 
> kernel threads is leaked to other processes in the same manner.
> 

I can reproduce priority leakage on my SMP system; any userspace process
chrt'ed up and a lot will follow. This makes the system very
unresponsive when doing a make -j5. Verified on 51-{6,18,23}.

-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Peter Zijlstra]

On Sat, 2005-07-09 at 23:24 +0200, Peter Zijlstra wrote:
> On Fri, 2005-07-08 at 13:12 -0700, William Weston wrote:
> > On Fri, 8 Jul 2005, Ingo Molnar wrote:
> > 
> > > could you check whether the priority leakage happens if you disable SMP?  
> > > (if you can reproduce it easily)
> > 
> > No priority leakages have been seen with UP configs on any of the 
> > machines I've been testing.
> > 
> > The leakage is not hard to reproduce under SMT:  start up jackd from a
> > text vc with an rt prio of 60 (or some unique prio above the IRQ threads),
> > then restart X and login.  After several minutes, X and all of its
> > children will be running at whatever prio jackd was started at (but still
> > SCHED_NORMAL).  Eventually, init, a handful of SCHED_NORMAL kernel
> > threads, and other random processes are all running at the same priority.  
> > When reset to default priority with chrt or schedtool, these processes
> > eventually revert back to the leaked priority level.  When jackd is
> > stopped, the priorities stay in their elevated state.  If jackd is not 
> > started before logging in to X, then the priority of one of the SCHED_FF 
> > kernel threads is leaked to other processes in the same manner.
> > 
> 
> I can reproduce priority leakage on my SMP system; any userspace process
> chrt'ed up and a lot will follow. This makes the system very
> unresponsive when doing a make -j5. Verified on 51-{6,18,23}.
> 

The following patch seems to fix it for me, YMMV.

--- kernel/sched.c~     2005-07-08 10:27:59.000000000 +0200
+++ kernel/sched.c      2005-07-10 13:00:42.000000000 +0200
@@ -780,7 +780,8 @@ static void recalc_task_prio(task_t *p,
                }
        }

-       p->prio = p->normal_prio = effective_prio(p);
+       p->prio = effective_prio(p);
+       p->normal_prio = unlikely(rt_prio(p->normal_prio)) ? p->prio : __effective_prio(p);
 }

 /*
@@ -1415,7 +1416,8 @@ void fastcall wake_up_new_task(task_t *
        p->sleep_avg = JIFFIES_TO_NS(CURRENT_BONUS(p) *
                CHILD_PENALTY / 100 * MAX_SLEEP_AVG / MAX_BONUS);

-       p->prio = p->normal_prio = effective_prio(p);
+       p->prio = effective_prio(p);
+       p->normal_prio = unlikely(rt_prio(p->normal_prio)) ? p->prio : __effective_prio(p);

        if (likely(cpu == this_cpu)) {
                if (!(clone_flags & CLONE_VM)) {
@@ -1427,7 +1429,8 @@ void fastcall wake_up_new_task(task_t *
                        if (unlikely(!current->array))
                                __activate_task(p, rq);
                        else {
-                               p->prio = p->normal_prio = current->prio;
+                               p->prio = current->prio;
+                               p->normal_prio = current->normal_prio;
                                __activate_task_after(p, current, rq);
                        }
                        set_need_resched();
@@ -2744,7 +2747,8 @@ void scheduler_tick(void)
        if (!--p->time_slice) {
                dequeue_task(p, rq->active);
                set_tsk_need_resched(p);
-               p->prio = p->normal_prio = effective_prio(p);
+               p->prio = effective_prio(p);
+               p->normal_prio = unlikely(rt_prio(p->normal_prio)) ? p->prio : __effective_prio(p);
                p->time_slice = task_timeslice(p);
                p->first_time_slice = 0;



-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> > I can reproduce priority leakage on my SMP system; any userspace process
> > chrt'ed up and a lot will follow. This makes the system very
> > unresponsive when doing a make -j5. Verified on 51-{6,18,23}.
> > 
> 
> The following patch seems to fix it for me, YMMV.
> 
> --- kernel/sched.c~     2005-07-08 10:27:59.000000000 +0200
> +++ kernel/sched.c      2005-07-10 13:00:42.000000000 +0200
> @@ -780,7 +780,8 @@ static void recalc_task_prio(task_t *p,
>                 }
>         }
> 
> -       p->prio = p->normal_prio = effective_prio(p);
> +       p->prio = effective_prio(p);
> +       p->normal_prio = unlikely(rt_prio(p->normal_prio)) ? p->prio : __effective_prio(p);
>  }

ahh, indeed, this code did not take boosting into account. Good catch!  
I'm wondering why this only showed up on SMP. I've fixed it a bit 
differently in my tree, by making the roles of the various priority 
fields and functions more obvious, see the delta patch below.  I've also 
released the -51-23 patch with these changes included. Does this fix 
priority leakage on your SMP system?

	Ingo

Index: linux/include/linux/sched.h
===================================================================
--- linux.orig/include/linux/sched.h
+++ linux/include/linux/sched.h
@@ -1035,7 +1035,7 @@ extern int idle_cpu(int cpu);
 extern int sched_setscheduler(struct task_struct *, int, struct sched_param *);
 extern task_t *idle_task(int cpu);
 extern void mutex_setprio(task_t *p, int prio);
-extern int mutex_getprio(task_t *p);
+extern int normal_prio(task_t *p);
 
 void yield(void);
 void __yield(void);
Index: linux/kernel/rt.c
===================================================================
--- linux.orig/kernel/rt.c
+++ linux/kernel/rt.c
@@ -720,7 +720,7 @@ static void pi_setprio(struct rt_mutex *
 
 #ifdef CONFIG_RT_DEADLOCK_DETECT
 	pi_prio++;
-	if (p->policy != SCHED_NORMAL && prio > mutex_getprio(p)) {
+	if (p->policy != SCHED_NORMAL && prio > normal_prio(p)) {
 		TRACE_OFF();
 
 		printk("huh? (%d->%d??)\n", p->prio, prio);
Index: linux/kernel/sched.c
===================================================================
--- linux.orig/kernel/sched.c
+++ linux/kernel/sched.c
@@ -645,7 +645,7 @@ static inline void enqueue_task_head(str
 }
 
 /*
- * effective_prio - return the priority that is based on the static
+ * __normal_prio - return the priority that is based on the static
  * priority but is modified by bonuses/penalties.
  *
  * We scale the actual sleep average [0 .... MAX_SLEEP_AVG]
@@ -659,7 +659,7 @@ static inline void enqueue_task_head(str
  * Both properties are important to certain workloads.
  */
 
-static inline int __effective_prio(task_t *p)
+static inline int __normal_prio(task_t *p)
 {
 	int bonus, prio;
 
@@ -670,57 +670,53 @@ static inline int __effective_prio(task_
 		prio = MAX_RT_PRIO;
 	if (prio > MAX_PRIO-1)
 		prio = MAX_PRIO-1;
-	return prio;
-}
 
-static int effective_prio(task_t *p)
-{
-	if (rt_task(p))
-		return p->prio;
-	return __effective_prio(p);
-}
-
-static inline void trace_start_sched_wakeup(task_t *p, runqueue_t *rq)
-{
-	if (TASK_PREEMPTS_CURR(p, rq) && (p != rq->curr))
-		__trace_start_sched_wakeup(p);
+	return prio;
 }
 
 /*
- * __activate_task - move a task to the runqueue.
+ * Calculate the expected normal priority: i.e. priority
+ * without taking RT-inheritance into account. Might be
+ * boosted by interactivity modifiers. Changes upon fork,
+ * setprio syscalls, and whenever the interactivity
+ * estimator recalculates.
  */
-static inline void __activate_task(task_t *p, runqueue_t *rq)
+inline int normal_prio(task_t *p)
 {
-	trace_special_pid(p->pid, p->prio, rq->nr_running);
-	enqueue_task(p, rq->active);
-	rq->nr_running++;
+	int prio;
+
+	if (p->policy != SCHED_NORMAL)
+		prio = MAX_RT_PRIO-1 - p->rt_priority;
+	else
+		prio = __normal_prio(p);
+
+	trace_special_pid(p->pid, p->prio, prio);
+	return prio;
 }
 
 /*
- * __activate_task_after - move a task to the runqueue,
- *                         to execute after a specific task.
+ * Calculate the current priority, i.e. the priority
+ * taken into account by the scheduler. This value might
+ * be boosted by RT tasks, or might be boosted by
+ * interactivity modifiers. Will be RT if the task got
+ * RT-boosted. If not then it returns p->normal_prio.
  */
-static inline
-void __activate_task_after(task_t *p, task_t *parent, runqueue_t *rq)
+static void __recalc_task_prio(task_t *p)
 {
-	// FIXME: to head rather?
-	list_add_tail(&p->run_list, &parent->run_list);
-	p->array = parent->array;
-	p->array->nr_active++;
-	rq->nr_running++;
-	inc_rt_tasks(p, rq);
+	p->normal_prio = normal_prio(p);
+	/*
+	 * If we are RT tasks or we were boosted to RT priority,
+	 * keep the priority unchanged. Otherwise, update priority
+	 * to the normal priority:
+	 */
+	if (!rt_prio(p->prio))
+		p->prio = p->normal_prio;
 }
 
 /*
- * __activate_idle_task - move idle task to the _front_ of runqueue.
+ * Recalculate p->normal_prio and p->prio after having slept,
+ * updating the sleep-average too:
  */
-static inline void __activate_idle_task(task_t *p, runqueue_t *rq)
-{
-	enqueue_task_head(p, rq->active);
-	rq->nr_running++;
-	WARN_ON(rt_task(p));
-}
-
 static void recalc_task_prio(task_t *p, unsigned long long now)
 {
 	/* Caller must always ensure 'now >= p->timestamp' */
@@ -780,7 +776,48 @@ static void recalc_task_prio(task_t *p, 
 		}
 	}
 
-	p->prio = p->normal_prio = effective_prio(p);
+	__recalc_task_prio(p);
+}
+
+static inline void trace_start_sched_wakeup(task_t *p, runqueue_t *rq)
+{
+	if (TASK_PREEMPTS_CURR(p, rq) && (p != rq->curr))
+		__trace_start_sched_wakeup(p);
+}
+
+/*
+ * __activate_task - move a task to the runqueue.
+ */
+static inline void __activate_task(task_t *p, runqueue_t *rq)
+{
+	trace_special_pid(p->pid, p->prio, rq->nr_running);
+	enqueue_task(p, rq->active);
+	rq->nr_running++;
+}
+
+/*
+ * __activate_task_after - move a task to the runqueue,
+ *                         to execute after a specific task.
+ */
+static inline
+void __activate_task_after(task_t *p, task_t *parent, runqueue_t *rq)
+{
+	// FIXME: to head rather?
+	list_add_tail(&p->run_list, &parent->run_list);
+	p->array = parent->array;
+	p->array->nr_active++;
+	rq->nr_running++;
+	inc_rt_tasks(p, rq);
+}
+
+/*
+ * __activate_idle_task - move idle task to the _front_ of runqueue.
+ */
+static inline void __activate_idle_task(task_t *p, runqueue_t *rq)
+{
+	enqueue_task_head(p, rq->active);
+	rq->nr_running++;
+	WARN_ON(rt_task(p));
 }
 
 /*
@@ -1415,7 +1452,7 @@ void fastcall wake_up_new_task(task_t * 
 	p->sleep_avg = JIFFIES_TO_NS(CURRENT_BONUS(p) *
 		CHILD_PENALTY / 100 * MAX_SLEEP_AVG / MAX_BONUS);
 
-	p->prio = p->normal_prio = effective_prio(p);
+	__recalc_task_prio(p);
 
 	if (likely(cpu == this_cpu)) {
 		if (!(clone_flags & CLONE_VM)) {
@@ -1427,7 +1464,8 @@ void fastcall wake_up_new_task(task_t * 
 			if (unlikely(!current->array))
 				__activate_task(p, rq);
 			else {
-				p->prio = p->normal_prio = current->prio;
+				p->prio = current->prio;
+				p->normal_prio = current->normal_prio;
 				__activate_task_after(p, current, rq);
 			}
 			set_need_resched();
@@ -2730,6 +2768,10 @@ void scheduler_tick(void)
 		/*
 		 * RR tasks need a special form of timeslice management.
 		 * FIFO tasks have no timeslices.
+		 *
+		 * On PREEMPT_RT, boosted tasks will also get into this
+		 * branch and wont get their timeslice decreased until
+		 * they have done their work.
 		 */
 		if ((p->policy == SCHED_RR) && !--p->time_slice) {
 			p->time_slice = task_timeslice(p);
@@ -2744,7 +2786,7 @@ void scheduler_tick(void)
 	if (!--p->time_slice) {
 		dequeue_task(p, rq->active);
 		set_tsk_need_resched(p);
-		p->prio = p->normal_prio = effective_prio(p);
+		__recalc_task_prio(p);
 		p->time_slice = task_timeslice(p);
 		p->first_time_slice = 0;
 
@@ -3682,7 +3724,7 @@ void set_user_nice(task_t *p, long nice)
 	new_prio = NICE_TO_PRIO(nice);
 	delta = new_prio - old_prio;
 	p->static_prio = NICE_TO_PRIO(nice);
-	p->prio += delta;
+	__recalc_task_prio(p);
 
 	if (array) {
 		enqueue_task(p, array);
@@ -3712,18 +3754,6 @@ int can_nice(const task_t *p, const int 
 		capable(CAP_SYS_NICE));
 }
 
-int mutex_getprio(task_t *p)
-{
-	int prio;
-
-	if (p->policy != SCHED_NORMAL)
-		prio = MAX_RT_PRIO-1 - p->rt_priority;
-	else
-		prio = __effective_prio(p);
-	trace_special_pid(p->pid, p->prio, prio);
-	return prio;
-}
-
 /*
  * Used by the PREEMPT_RT code to implement
  * priority inheritance logic:
@@ -3880,10 +3910,7 @@ static void __setscheduler(struct task_s
 	BUG_ON(p->array);
 	p->policy = policy;
 	p->rt_priority = prio;
-	if (policy != SCHED_NORMAL)
-		p->prio = p->normal_prio = MAX_RT_PRIO-1 - p->rt_priority;
-	else
-		p->prio = p->normal_prio = p->static_prio;
+	__recalc_task_prio(p);
 }
 
 /**

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Peter Zijlstra]

On Sun, 2005-07-10 at 17:10 +0200, Ingo Molnar wrote:
> * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> 
> > > I can reproduce priority leakage on my SMP system; any userspace process
> > > chrt'ed up and a lot will follow. This makes the system very
> > > unresponsive when doing a make -j5. Verified on 51-{6,18,23}.
> > > 
> > 
> > The following patch seems to fix it for me, YMMV.
> > 
> > --- kernel/sched.c~     2005-07-08 10:27:59.000000000 +0200
> > +++ kernel/sched.c      2005-07-10 13:00:42.000000000 +0200
> > @@ -780,7 +780,8 @@ static void recalc_task_prio(task_t *p,
> >                 }
> >         }
> > 
> > -       p->prio = p->normal_prio = effective_prio(p);
> > +       p->prio = effective_prio(p);
> > +       p->normal_prio = unlikely(rt_prio(p->normal_prio)) ? p->prio : __effective_prio(p);
> >  }
> 
> ahh, indeed, this code did not take boosting into account. Good catch!  
> I'm wondering why this only showed up on SMP. 

I was thinking because of the agressive RT rebalance code this codepath
is more exercised on SMP systems.

> I've fixed it a bit 
> differently in my tree, by making the roles of the various priority 
> fields and functions more obvious, see the delta patch below.  

Yes, much nicer :-)

> I've also 
> released the -51-23 patch with these changes included. Does this fix 
> priority leakage on your SMP system?
> 

-51-24 right? I'll give it a spin.

-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Peter Zijlstra]

On Sun, 2005-07-10 at 17:43 +0200, Peter Zijlstra wrote:

> > I've also 
> > released the -51-23 patch with these changes included. Does this fix 
> > priority leakage on your SMP system?
> > 
> 
> -51-24 right? I'll give it a spin.
> 

a quick test seems to indicate it is indeed solved.

-- 
Peter Zijlstra <a.p.zijlstra@chello.nl>

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Sun, 2005-07-10 at 17:43 +0200, Peter Zijlstra wrote:
> 
> > > I've also 
> > > released the -51-23 patch with these changes included. Does this fix 
> > > priority leakage on your SMP system?
> > > 
> > 
> > -51-24 right? I'll give it a spin.
> > 
> 
> a quick test seems to indicate it is indeed solved.

great!

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.51-12

[William Weston]

On Sun, 10 Jul 2005, Peter Zijlstra wrote:

> On Sun, 2005-07-10 at 17:43 +0200, Peter Zijlstra wrote:
> 
> > > I've also 
> > > released the -51-23 patch with these changes included. Does this fix 
> > > priority leakage on your SMP system?
> > > 
> > 
> > -51-24 right? I'll give it a spin.
> > 
> 
> a quick test seems to indicate it is indeed solved.

Running -51-26.  Priority leakage on the SMT box is gone.  This indeed 
fixes the scheduler meltdown issue I was dealing with.  Great work, guys!

--ww

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Lee Revell]

On Mon, 2005-07-04 at 13:16 +0200, Ingo Molnar wrote:
> i'd first suggest to try the latest kernel, before changing your X 
> config - i think the bug might have been fixed meanwhile.

I have found that heavy network traffic really kills the interactive
performance.  In the top excerpt below, gtk-gnutella is generating about
320KB/sec of traffic.

These priorities do not look right:

   14 root     -50  -5     0    0 S  0.0  0.0   0:00.00    0    0    0    0 [IRQ 9]                                        irqd     
  686 root     -49  -5     0    0 S  0.0  0.0  87:58.18    0    0    0    0 [IRQ 8]                                        irqd     
  689 root     -48  -5     0    0 S  0.0  0.0   0:00.00    0    0    0    0 [IRQ 7]                                        irqd     
  694 root     -47  -5     0    0 S  0.0  0.0   0:00.00    0    0    0    0 [IRQ 12]                                       irqd     
  714 root     -46  -5     0    0 S  2.5  0.0   5:25.33    0    0    0    0 [IRQ 15]                                       irqd     
  731 root     -45  -5     0    0 S  0.0  0.0   0:04.74    0    0    0    0 [IRQ 1]                                        irqd     
    3 root     -44 -10     0    0 S  0.0  0.0   1:42.55    0    0    0    0 [softirq-timer/0]                              ksoftirqd
    5 root     -44 -10     0    0 S  2.5  0.0   5:06.49    0    0    0    0 [softirq-net-rx/]                              ksoftirqd
 1541 root     -44  -5     0    0 S  0.0  0.0   9:39.03    0    0    0    0 [IRQ 10]                                       irqd     
 5650 rlrevell -44   0  2804 1616 S 14.8  0.4   5:52.83 1188   52  784    1 /usr/lib/gamin/gam_server                      stext    
30313 rlrevell -44   0 24572  19m S 99.9  4.4  68:15.42 4520 1800  12m   31 gtk-gnutella                                   stext    
 2285 root     -43  -5     0    0 S  0.0  0.0   3:47.34    0    0    0    0 [IRQ 11]                                       irqd     
 2807 root     -42  -5     0    0 S  0.0  0.0   0:00.00    0    0    0    0 [IRQ 14]                                       irqd     
    9 root      -2  -5     0    0 S  0.0  0.0   0:16.05    0    0    0    0 [events/0]                                     worker_th
    4 root       5 -10     0    0 S  0.0  0.0   0:25.95    0    0    0    0 [softirq-net-tx/]                              ksoftirqd
    7 root       5 -10     0    0 S  0.0  0.0   0:01.22    0    0    0    0 [softirq-tasklet]                              ksoftirqd
    8 root       5 -10     0    0 S  0.0  0.0   0:01.30    0    0    0    0 [desched/0]                                    desched_t
28988 root       5 -10  106m  30m S  4.9  6.9 242:59.04  76m 1556  32m    3 /usr/X11R6/bin/X :0 -audit 0 -auth /var/lib/gd stext    
    2 root      10 -10     0    0 S  0.0  0.0   0:00.00    0    0    0    0 [softirq-high/0]                               ksoftirqd
    6 root      10 -10     0    0 S  0.0  0.0   0:00.00    0    0    0    0 [softirq-scsi/0]

Why do gtk-gnutella and gam_server get a higher priority than the disk
(14) and network (11) IRQs?  And shouldn't the softirq threads run at
lower priority than all the hardirq threads?

Looking at the top output I'd expect the machine to be crawling, and
that's what it does.

Lee

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Lee Revell]

On Tue, 2005-07-12 at 21:18 -0400, Lee Revell wrote:
> On Mon, 2005-07-04 at 13:16 +0200, Ingo Molnar wrote:
> > i'd first suggest to try the latest kernel, before changing your X 
> > config - i think the bug might have been fixed meanwhile.
> 
> I have found that heavy network traffic really kills the interactive
> performance.  In the top excerpt below, gtk-gnutella is generating about
> 320KB/sec of traffic.
> 
> These priorities do not look right:

Never mind, I just rediscovered the RT priority leakage bug.

Lee

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Ingo Molnar]

* Lee Revell <rlrevell@joe-job.com> wrote:

> > I have found that heavy network traffic really kills the interactive
> > performance.  In the top excerpt below, gtk-gnutella is generating about
> > 320KB/sec of traffic.
> > 
> > These priorities do not look right:
> 
> Never mind, I just rediscovered the RT priority leakage bug.

ok. And is it fixed by recent kernels?

	Ingo

Re: Real-Time Preemption, -RT-2.6.12-final-V0.7.50-24

[Lee Revell]

On Wed, 2005-07-13 at 08:35 +0200, Ingo Molnar wrote:
> * Lee Revell <rlrevell@joe-job.com> wrote:
> 
> > > I have found that heavy network traffic really kills the interactive
> > > performance.  In the top excerpt below, gtk-gnutella is generating about
> > > 320KB/sec of traffic.
> > > 
> > > These priorities do not look right:
> > 
> > Never mind, I just rediscovered the RT priority leakage bug.
> 
> ok. And is it fixed by recent kernels?
> 

Yes, I have not seen the problem yet with V0.7.51-28.

Lee