network driver disabled interrupts in PREEMPT_RT

network driver disabled interrupts in PREEMPT_RT

[Kristian Benoit]

Hi,
I got lots of these messages when accessing the net running
2.6.12-rc6-RT-V0.7.48-25 :

"network driver disabled interrupts: tg3_start_xmit+0x0/0x629 [tg3]"

it seem to come from net/sched/sch_generic.c.

As of the response time I get, it does not seem normal.

Kristian

Re: network driver disabled interrupts in PREEMPT_RT

[Ingo Molnar]

* Kristian Benoit <kbenoit@opersys.com> wrote:

> Hi,
> I got lots of these messages when accessing the net running
> 2.6.12-rc6-RT-V0.7.48-25 :
> 
> "network driver disabled interrupts: tg3_start_xmit+0x0/0x629 [tg3]"
> 
> it seem to come from net/sched/sch_generic.c.

does the patch below fix it?

	Ingo

--- linux/drivers/net/tg3.c.orig
+++ linux/drivers/net/tg3.c
@@ -3242,9 +3242,9 @@ static int tg3_start_xmit(struct sk_buff
 	 * So we really do need to disable interrupts when taking
 	 * tx_lock here.
 	 */
-	local_irq_save(flags);
+	local_irq_save_nort(flags);
 	if (!spin_trylock(&tp->tx_lock)) { 
-		local_irq_restore(flags);
+		local_irq_restore_nort(flags);
 		return NETDEV_TX_LOCKED; 
 	} 
 

Re: network driver disabled interrupts in PREEMPT_RT

[Christoph Hellwig]

On Mon, Jun 13, 2005 at 08:56:42PM +0200, Ingo Molnar wrote:
> 
> * Kristian Benoit <kbenoit@opersys.com> wrote:
> 
> > Hi,
> > I got lots of these messages when accessing the net running
> > 2.6.12-rc6-RT-V0.7.48-25 :
> > 
> > "network driver disabled interrupts: tg3_start_xmit+0x0/0x629 [tg3]"
> > 
> > it seem to come from net/sched/sch_generic.c.
> 
> does the patch below fix it?

Wouldn't it be much more useful to add spin_trylock_irq?

Re: network driver disabled interrupts in PREEMPT_RT

[Ingo Molnar]

* Christoph Hellwig <hch@infradead.org> wrote:

> On Mon, Jun 13, 2005 at 08:56:42PM +0200, Ingo Molnar wrote:
> > 
> > * Kristian Benoit <kbenoit@opersys.com> wrote:
> > 
> > > Hi,
> > > I got lots of these messages when accessing the net running
> > > 2.6.12-rc6-RT-V0.7.48-25 :
> > > 
> > > "network driver disabled interrupts: tg3_start_xmit+0x0/0x629 [tg3]"
> > > 
> > > it seem to come from net/sched/sch_generic.c.
> > 
> > does the patch below fix it?
> 
> Wouldn't it be much more useful to add spin_trylock_irq?

you are right, a spin_trylock_irqsave() call would be much cleaner.

i think i should explain why you are seeing such hacks in the PREEMPT_RT 
patch: generally we are trying to achieve near-100% driver API 
compatibility (that of course is also one of the requirements for any of 
this to hit mainline), so i'm marking any deviations with _nort or _rt 
postfixes (depending on where the change in semantics is needed). I 
consider them temporary hacks, so i'm usually trying to keep the 
original form of the code, so that i can go back later and fix it.

I had these hacks in tg3.c for some time, and i thought i could drop 
them now that local_irq_*() uses the soft IRQ-flag - but i was wrong.  
One example of the _rt/_nort marking process is e.g. the earlier RCU API 
related grossness, which went away with Paul's PREEMPT_RCU aproach and 
now PREEMPT_RT is fully compatible with the RCU API. Fortunately, 
assymetric local_irq_* + spin_lock_* uses (which are perfectly legal!)  
are relatively rare. Extending the spinlock APIs and converting all 
upstream code would be a good approach to solve this problem, as it's 
also a cleanup. (It would probably also make static lock analysis 
easier.)

	Ingo

Re: network driver disabled interrupts in PREEMPT_RT

[Herbert Xu]

Christoph Hellwig <hch@infradead.org> wrote:
> On Mon, Jun 13, 2005 at 08:56:42PM +0200, Ingo Molnar wrote:
>> 
>> * Kristian Benoit <kbenoit@opersys.com> wrote:
>> 
>> > Hi,
>> > I got lots of these messages when accessing the net running
>> > 2.6.12-rc6-RT-V0.7.48-25 :
>> > 
>> > "network driver disabled interrupts: tg3_start_xmit+0x0/0x629 [tg3]"
>> > 
>> > it seem to come from net/sched/sch_generic.c.
>> 
>> does the patch below fix it?
> 
> Wouldn't it be much more useful to add spin_trylock_irq?

Dave has a patch pending that should remove these IRQ disabling calls.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: network driver disabled interrupts in PREEMPT_RT

[Kristian Benoit]

On Mon, 2005-06-13 at 20:56 +0200, Ingo Molnar wrote:
> does the patch below fix it?

It does!
thank you, the data I'm collecting make much more sens now !!

Kristian

Re: network driver disabled interrupts in PREEMPT_RT

[Serge Noiraud]

Le lun 13/06/2005 à 20:56, Ingo Molnar a écrit :
> * Kristian Benoit <kbenoit@opersys.com> wrote:
> 
> > Hi,
> > I got lots of these messages when accessing the net running
> > 2.6.12-rc6-RT-V0.7.48-25 :
> > 
> > "network driver disabled interrupts: tg3_start_xmit+0x0/0x629 [tg3]"
> > 
> > it seem to come from net/sched/sch_generic.c.
> 
> does the patch below fix it?
> 
> 	Ingo
I have the same problem with an e1000 card for 2.6.12-rc6-RT-V0.7.48-32 :
#dmesg
...
network driver disabled interrupts: e1000_xmit_frame+0x0/0xbc0 [e1000]
...

Re: network driver disabled interrupts in PREEMPT_RT

[Ingo Molnar]

* Serge Noiraud <serge.noiraud@bull.net> wrote:

> Le lun 13/06/2005 à 20:56, Ingo Molnar a écrit :
> > * Kristian Benoit <kbenoit@opersys.com> wrote:
> > 
> > > Hi,
> > > I got lots of these messages when accessing the net running
> > > 2.6.12-rc6-RT-V0.7.48-25 :
> > > 
> > > "network driver disabled interrupts: tg3_start_xmit+0x0/0x629 [tg3]"
> > > 
> > > it seem to come from net/sched/sch_generic.c.
> > 
> > does the patch below fix it?
> > 
> > 	Ingo
> I have the same problem with an e1000 card for 2.6.12-rc6-RT-V0.7.48-32 :
> #dmesg
> ...
> network driver disabled interrupts: e1000_xmit_frame+0x0/0xbc0 [e1000]
> ...

does -48-33 fix it for you?

	Ingo

Re: network driver disabled interrupts in PREEMPT_RT

[Serge Noiraud]

Le mer 15/06/2005 à 15:05, Ingo Molnar a écrit :
> * Serge Noiraud <serge.noiraud@bull.net> wrote:
> 
> > Le lun 13/06/2005 à 20:56, Ingo Molnar a écrit :
> > > * Kristian Benoit <kbenoit@opersys.com> wrote:
> > > 
> > > > Hi,
> > > > I got lots of these messages when accessing the net running
> > > > 2.6.12-rc6-RT-V0.7.48-25 :
> > > > 
> > > > "network driver disabled interrupts: tg3_start_xmit+0x0/0x629 [tg3]"
> > > > 
> > > > it seem to come from net/sched/sch_generic.c.
> > > 
> > > does the patch below fix it?
> > > 
> > > 	Ingo
> > I have the same problem with an e1000 card for 2.6.12-rc6-RT-V0.7.48-32 :
> > #dmesg
> > ...
> > network driver disabled interrupts: e1000_xmit_frame+0x0/0xbc0 [e1000]
> > ...
> 
> does -48-33 fix it for you?
> 
> 	Ingo
No and I have the tg3_start_xmit too. So the first problem is not solved.
I have the following :
network driver disabled interrupts: tg3_start_xmit+0x0/0x620 [tg3]

Re: network driver disabled interrupts in PREEMPT_RT

[Ingo Molnar]

* Serge Noiraud <serge.noiraud@bull.net> wrote:

> > > I have the same problem with an e1000 card for 2.6.12-rc6-RT-V0.7.48-32 :
> > > #dmesg
> > > ...
> > > network driver disabled interrupts: e1000_xmit_frame+0x0/0xbc0 [e1000]
> > > ...
> > 
> > does -48-33 fix it for you?
> > 
> > 	Ingo

> No and I have the tg3_start_xmit too. So the first problem is not 
> solved. I have the following : network driver disabled interrupts: 
> tg3_start_xmit+0x0/0x620 [tg3]

do -50-06 (or later) kernels solve this for you?

	Ingo

Re: network driver disabled interrupts in PREEMPT_RT

[Serge Noiraud]

Le mar 21/06/2005 à 15:18, Ingo Molnar a écrit :
> * Serge Noiraud <serge.noiraud@bull.net> wrote:
> 
> > > > I have the same problem with an e1000 card for 2.6.12-rc6-RT-V0.7.48-32 :
> > > > #dmesg
> > > > ...
> > > > network driver disabled interrupts: e1000_xmit_frame+0x0/0xbc0 [e1000]
> > > > ...
> > > 
> > > does -48-33 fix it for you?
> > > 
> > > 	Ingo
> 
> > No and I have the tg3_start_xmit too. So the first problem is not 
> > solved. I have the following : network driver disabled interrupts: 
> > tg3_start_xmit+0x0/0x620 [tg3]
> 
> do -50-06 (or later) kernels solve this for you?
> 
> 	Ingo
sorry for the delay. I was out of my office for 2 weeks.
The last RT version I tried was 48-36. it worked.

I'm testing 50-51 wihout succes.
One DEFINE_SPINLOCK is not translated in DEFINE_RAW_SPINLOCK in the
following file :
  CC      arch/i386/kernel/vm86.o
arch/i386/kernel/vm86.c:701: error: syntax error before ',' token
make[4]: *** [arch/i386/kernel/vm86.o] Erreur 1
make[3]: *** [arch/i386/kernel] Erreur 2

is it correct ?
I think I had not this error with 50-47.