2.6.15-rt12 bugs

2.6.15-rt12 bugs

[Michal Piotrowski]

Hi,

I have noticed some bugs in latest 2.6.15-rt12, here are some of them:
http://www.stardust.webpages.pl/files/rt/2.6.15-rt12/rt-dmesg
Here is my config:
http://www.stardust.webpages.pl/files/rt/2.6.15-rt12/rt-config

Regards,
Michal Piotrowski

Re: 2.6.15-rt12 bugs

[Steven Rostedt]

On Mon, 2006-01-23 at 14:21 +0100, Michal Piotrowski wrote:
> Hi,
> 
> I have noticed some bugs in latest 2.6.15-rt12, here are some of them:
> http://www.stardust.webpages.pl/files/rt/2.6.15-rt12/rt-dmesg

The only bug I see in this is:

BUG: kstopmachine:338 task might have lost a preemption check!
 [<c0103b1b>] dump_stack+0x1b/0x1f (20)
 [<c0118736>] preempt_enable_no_resched+0x4a/0x50 (20)
 [<c014224d>] restart_machine+0x1a/0x1d (12)
 [<c0142274>] do_stop+0x24/0x65 (20)
 [<c012d6de>] kthread+0x7b/0xa9 (36)
 [<c01010c5>] kernel_thread_helper+0x5/0xb (135438364)


Which I'll go and take a look at.  I need to turn on
CONFIG_DEBUG_PREEMPT on my SMP machine.


> Here is my config:
> http://www.stardust.webpages.pl/files/rt/2.6.15-rt12/rt-config

You might want to turn off:

CONFIG_DEBUG_STACKOVERFLOW

This dumps out the biggest stack usage.  Those dumps are caused by this,
and are not bugs.  It just shows you what's using the most stack, that's
all.  I find it quite annoying, and just turn it off.

-- Steve

[PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

[Steven Rostedt]

Here Ingo,

The kstop_machine has a legitimate preempt_enable_no_resched.

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux-2.6.15-rt12/kernel/stop_machine.c
===================================================================
--- linux-2.6.15-rt12.orig/kernel/stop_machine.c	2006-01-23 09:15:37.000000000 -0500
+++ linux-2.6.15-rt12/kernel/stop_machine.c	2006-01-23 19:58:26.000000000 -0500
@@ -134,7 +134,7 @@
 {
 	stopmachine_set_state(STOPMACHINE_EXIT);
 	raw_local_irq_enable();
-	preempt_enable_no_resched();
+	__preempt_enable_no_resched();
 }
 
 struct stop_machine_data

Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> The kstop_machine has a legitimate preempt_enable_no_resched.

thanks, applied.

	Ingo

Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

[Michal Piotrowski]

Hi,

On 24/01/06, Steven Rostedt <rostedt@goodmis.org> wrote:
> Here Ingo,
>
> The kstop_machine has a legitimate preempt_enable_no_resched.
>
> -- Steve
>
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
>
> Index: linux-2.6.15-rt12/kernel/stop_machine.c
> ===================================================================
> --- linux-2.6.15-rt12.orig/kernel/stop_machine.c        2006-01-23 09:15:37.000000000 -0500
> +++ linux-2.6.15-rt12/kernel/stop_machine.c     2006-01-23 19:58:26.000000000 -0500
> @@ -134,7 +134,7 @@
>  {
>         stopmachine_set_state(STOPMACHINE_EXIT);
>         raw_local_irq_enable();
> -       preempt_enable_no_resched();
> +       __preempt_enable_no_resched();
>  }
>
>  struct stop_machine_data
>
>
>

Great thanks!

Now 2.6.15-rt14 works fine (after disabling CONFIG_DEBUG_STACKOVERFLOW).

I have noticed only one warning
WARNING: softirq-tasklet/8 changed soft IRQ-flags.
 [<c0103b1b>] dump_stack+0x1b/0x1f (20)
 [<c0135218>] illegal_API_call+0x41/0x46 (20)
 [<c0135267>] local_irq_disable+0x1d/0x1f (8)
  [<f886dbd5>] skge_extirq+0x117/0x138 [skge] (32)
 [<c01202d5>] tasklet_action+0x8a/0xf1 (24)
  [<c01205c1>] ksoftirqd+0x10f/0x19e (32)
  [<c012d7ca>] kthread+0x7b/0xa9 (36)
 [<c01010c5>] kernel_thread_helper+0x5/0xb (1047875612)
---------------------------
| preempt count: 00000000 ]
| 0-level deep critical section nesting:
----------------------------------------

------------------------------
| showing all locks held by: |  (softirq-tasklet/8 [c18a97d0,  98]):
------------------------------

And problems while loading ipv6 module
Running ntpdate to synchronize clockCould not allocate 4 bytes percpu data
modprobe: FATAL: Error inserting ipv6
(/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
memory

Could not allocate 4 bytes percpu data
modprobe: FATAL: Error inserting ipv6
(/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
memory

Here is my dmesg:
http://www.stardust.webpages.pl/files/rt/2.6.15-rt14/rt-dmesg
Here is my config:
http://www.stardust.webpages.pl/files/rt/2.6.15-rt14/rt-config

Regards,
Michal Piotrowski

Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

[Steven Rostedt]

On Tue, 2006-01-24 at 14:33 +0100, Michal Piotrowski wrote:

> Great thanks!
> 
> Now 2.6.15-rt14 works fine (after disabling CONFIG_DEBUG_STACKOVERFLOW).
> 
> I have noticed only one warning
> WARNING: softirq-tasklet/8 changed soft IRQ-flags.
>  [<c0103b1b>] dump_stack+0x1b/0x1f (20)
>  [<c0135218>] illegal_API_call+0x41/0x46 (20)
>  [<c0135267>] local_irq_disable+0x1d/0x1f (8)
>   [<f886dbd5>] skge_extirq+0x117/0x138 [skge] (32)
>  [<c01202d5>] tasklet_action+0x8a/0xf1 (24)
>   [<c01205c1>] ksoftirqd+0x10f/0x19e (32)
>   [<c012d7ca>] kthread+0x7b/0xa9 (36)
>  [<c01010c5>] kernel_thread_helper+0x5/0xb (1047875612)
> ---------------------------
> | preempt count: 00000000 ]
> | 0-level deep critical section nesting:
> ----------------------------------------
> 
> ------------------------------
> | showing all locks held by: |  (softirq-tasklet/8 [c18a97d0,  98]):
> ------------------------------

Yeah, I've seen this too. Since it's just a warning, it's been push to
the back of the TODO list.

> 
> And problems while loading ipv6 module
> Running ntpdate to synchronize clockCould not allocate 4 bytes percpu data
> modprobe: FATAL: Error inserting ipv6
> (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> memory
> 
> Could not allocate 4 bytes percpu data
> modprobe: FATAL: Error inserting ipv6
> (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> memory

Is this new with the -rt14? or has this happened before. If it has
happened before, then could you tell us when it started.

Thanks,

-- Steve

Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

[Michal Piotrowski]

Hi,

On 24/01/06, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Tue, 2006-01-24 at 14:33 +0100, Michal Piotrowski wrote:
[snip]
> > And problems while loading ipv6 module
> > Running ntpdate to synchronize clockCould not allocate 4 bytes percpu data
> > modprobe: FATAL: Error inserting ipv6
> > (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> > memory
> >
> > Could not allocate 4 bytes percpu data
> > modprobe: FATAL: Error inserting ipv6
> > (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> > memory
>
> Is this new with the -rt14? or has this happened before. If it has
> happened before, then could you tell us when it started.

It's very hard to track down, because earlier versions of -rt ware too
buggy for me and most of them doesn't compile/boot.

Regards,
Michal Piotrowski

Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

[Steven Rostedt]

On Tue, 2006-01-24 at 16:37 +0100, Michal Piotrowski wrote:
> Hi,
> 
> On 24/01/06, Steven Rostedt <rostedt@goodmis.org> wrote:
> > On Tue, 2006-01-24 at 14:33 +0100, Michal Piotrowski wrote:
> [snip]
> > > And problems while loading ipv6 module
> > > Running ntpdate to synchronize clockCould not allocate 4 bytes percpu data
> > > modprobe: FATAL: Error inserting ipv6
> > > (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> > > memory
> > >
> > > Could not allocate 4 bytes percpu data
> > > modprobe: FATAL: Error inserting ipv6
> > > (/lib/modules/2.6.15-rt14/kernel/net/ipv6/ipv6.ko): Cannot allocate
> > > memory
> >
> > Is this new with the -rt14? or has this happened before. If it has
> > happened before, then could you tell us when it started.
> 
> It's very hard to track down, because earlier versions of -rt ware too
> buggy for me and most of them doesn't compile/boot.

The 2.6.14-rtX series was pretty stable. How early did you go back?

-- Steve

Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

[Michal Piotrowski]

On 24/01/06, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Tue, 2006-01-24 at 16:37 +0100, Michal Piotrowski wrote:
[snip]
> > It's very hard to track down, because earlier versions of -rt ware too
> > buggy for me and most of them doesn't compile/boot.
>
> The 2.6.14-rtX series was pretty stable. How early did you go back?

To 2.6.15-rt1 - it doesn't compile (ipv6 module). 2.6.15-rt2 gives me
wonderful series of oops/warnings/badness on boot ;).

I will try 2.6.14-rtX.

Regards,
Michal Piotrowski

Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

[Steven Rostedt]

On Tue, 2006-01-24 at 22:00 +0100, Michal Piotrowski wrote:
> On 24/01/06, Steven Rostedt <rostedt@goodmis.org> wrote:
> > On Tue, 2006-01-24 at 16:37 +0100, Michal Piotrowski wrote:
> [snip]
> > > It's very hard to track down, because earlier versions of -rt ware too
> > > buggy for me and most of them doesn't compile/boot.
> >
> > The 2.6.14-rtX series was pretty stable. How early did you go back?
> 
> To 2.6.15-rt1 - it doesn't compile (ipv6 module). 2.6.15-rt2 gives me
> wonderful series of oops/warnings/badness on boot ;).

Ingo was busy at the time getting true mutexes into the kernel.  So he
was pushing the -rt stuff out before testing.  So I created a tree to
handle this:

These are patches against Ingo's -rt patches, that should help make some
of your problems work.

http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt1-sr2
http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt2-sr3
http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt3-sr1
http://home.stny.rr.com/rostedt/patches/patch-2.6.15-rt4-sr2

-- Steve

Re: [PATCH RT] kstopmachine has legit preempt_enable_no_resched (was: 2.6.15-rt12 bugs)

[Michal Piotrowski]

Hi,

On 24/01/06, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Tue, 2006-01-24 at 22:00 +0100, Michal Piotrowski wrote:
> > To 2.6.15-rt1 - it doesn't compile (ipv6 module). 2.6.15-rt2 gives me
> > wonderful series of oops/warnings/badness on boot ;).
>
> Ingo was busy at the time getting true mutexes into the kernel.  So he
> was pushing the -rt stuff out before testing.  So I created a tree to
> handle this:
>
> These are patches against Ingo's -rt patches, that should help make some
> of your problems work.
>
> http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt1-sr2
> http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt2-sr3
> http://home.stny.rr.com/rostedt/patches/archive/patch-2.6.15-rt3-sr1
> http://home.stny.rr.com/rostedt/patches/patch-2.6.15-rt4-sr2
>
> -- Steve
>
>
>

Sorry for late answer, but I have exams.

Unfortunately these patches doesn't solve my problems. I have tried
many rt-patches, but everything below 2.6.15-rt8 doesn't work
(compilation or boot problems). Is there any other way to track down
this bug? (any magic ipv6 debugging option? :)

Regards,
Michal Piotrowski