Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19

Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19

[David S. Miller]

   From: Mike Smith <mikesmith1@btconnect.com>
   Date: Tue, 1 Jul 2003 10:52:58 +0000

   please CC me, I'm not subscribed. I'm getting occasional oopses
   from my Linux box -- one every week or two -- and I finally managed
   to capture one (see below -- it's transcribed by hand). This is a
   RH7.2 based box, running kernel 2.4.19. I'm using iptables NAT
   to share my ADSL line. In case it's relevant, I'm using Johan
   Verrept's speedtch driver for my USB-attached SpeedTouch ADSL
   modem. 

I believe 2.4.21 has a fix for this, but the netfilter
experts will know better.

Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19

[Martin Josefsson]

On Tue, 2003-07-01 at 11:47, David S. Miller wrote:

> I believe 2.4.21 has a fix for this, but the netfilter
> experts will know better.

(Dave, it's netfilter-devel@lists.netfilter.org)
(to the rest, the original report can be found on linux-net)

It looks like list-corruption.

ip_conntrack_core.c:665
byipsproto[ipsprotohash].next points into space.

I can't see anything obvious right now, my eyelids are very heavy so
I'll continue looking tomorrow.

There's a small possibility that this is caused by the modify-after-free
bug that's fixed in 2.4.21 (but unfortunately introduced some
refcounting bugs, patches are on the way to Dave, they can be found in
netfilter patch-o-matic as well).

-- 
/Martin

Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19

[Mike Smith]

On Tuesday 01 Jul 2003 21:05 pm, Martin Josefsson wrote:
> On Tue, 2003-07-01 at 11:47, David S. Miller wrote:
> > I believe 2.4.21 has a fix for this, but the netfilter
> > experts will know better.
>
> (Dave, it's netfilter-devel@lists.netfilter.org)
> (to the rest, the original report can be found on linux-net)
>
> It looks like list-corruption.
>
> ip_conntrack_core.c:665
> byipsproto[ipsprotohash].next points into space.
>
> I can't see anything obvious right now, my eyelids are very heavy so
> I'll continue looking tomorrow.
>
> There's a small possibility that this is caused by the modify-after-free
> bug that's fixed in 2.4.21 (but unfortunately introduced some
> refcounting bugs, patches are on the way to Dave, they can be found in
> netfilter patch-o-matic as well).

Hi,

this has happened again. Hand-transcribed oops below. Is this of interest?
I'm happy to accept debug patches etc. Or shall I just move to 2.4.21?

Thanks.

Oops: 0000
CPU:    0
EIP:    0010:[<c0247b2d>]   Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00210206
eax: c88a1fb8   ebx: c2a19cf0   ecx: 00200216   edx: c98a3068
esi: c2a19c90   edi: 0000017b   ebp: c02dbcfc   esp: c02dbcf0
ds: 0018   es: 0018   ss:0018
Process swapper (pid: 0, stackpage=c02db000)
Stack: c2a19bc0 c2a19c90 00000002 c02dbd20 c024625d c2a19bc0 c2a19c90 00000000
       00000002 c02dbd70 c031f0c0 c021e440 c02dbd48 c0213ea7 00000004 c02dbda0
       00000000 c5787000 c021e440 00000000 c5787000 00000004 c02dbd90 c021417d
Call Trace:    [<c024625d>] [<c021e440>] [<c0213ea7>] [<c021e440>] 
[<c021417d>]
  [<c021e440>] [<c021e409>] [<c021e440>] [<c0213ea7>] [<c021bbb6>] 
[<c021417d>]
  [<c02141b3>] [<c021bb27>] [<c021bb80>] [<c021aec3>] [<c021417d>] 
[<c02141b3>]
  [<c020a91a>] [<c021abe5>] [<c021ad50>] [<c0119b20>] [<c0119b31>] 
[<c020e90a>]
  [<c011a6ab>] [<c010a13c>] [<c0106fb0>] [<c010c5b8>] [<c0106fb0>] 
[<c0106fd6>]
  [<c0107062>] [<c0105000>]
Code: 8b 02 89 58 04 89 46 60 89 53 04 89 1a 5b 5e 5f 5d c3 90 55

>>EIP; c0247b2d <replace_in_hashes+7d/90>   <=====
Trace; c024625d <ip_nat_fn+13d/1a0>
Trace; c021e440 <ip_finish_output2+0/d0>
Trace; c0213ea7 <nf_iterate+27/90>
Trace; c021e440 <ip_finish_output2+0/d0>
Trace; c021417d <nf_hook_slow+9d/130>
Trace; c021e440 <ip_finish_output2+0/d0>
Trace; c021e409 <ip_finish_output+f9/110>
Trace; c021e440 <ip_finish_output2+0/d0>
Trace; c0213ea7 <nf_iterate+27/90>
Trace; c021bbb6 <ip_forward_finish+36/70>
Trace; c021417d <nf_hook_slow+9d/130>
Trace; c02141b3 <nf_hook_slow+d3/130>
Trace; c021bb27 <ip_forward+1b7/210>
Trace; c021bb80 <ip_forward_finish+0/70>
Trace; c021aec3 <ip_rcv_finish+173/1a0>
Trace; c021417d <nf_hook_slow+9d/130>
Trace; c02141b3 <nf_hook_slow+d3/130>
Trace; c020a91a <alloc_skb+da/1a0>
Trace; c021abe5 <ip_rcv+325/360>
Trace; c021ad50 <ip_rcv_finish+0/1a0>
Trace; c0119b20 <it_real_fn+0/60>
Trace; c0119b31 <it_real_fn+11/60>
Trace; c020e90a <net_rx_action+13a/210>
Trace; c011a6ab <do_softirq+5b/b0>
Trace; c010a13c <do_IRQ+ac/c0>
Trace; c0106fb0 <default_idle+0/30>
Trace; c010c5b8 <call_do_IRQ+5/d>
Trace; c0106fb0 <default_idle+0/30>
Trace; c0106fd6 <default_idle+26/30>
Trace; c0107062 <cpu_idle+52/70>
Trace; c0105000 <_stext+0/0>
Code;  c0247b2d <replace_in_hashes+7d/90>
00000000 <_EIP>:
Code;  c0247b2d <replace_in_hashes+7d/90>   <=====
   0:   8b 02                     mov    (%edx),%eax   <=====
Code;  c0247b2f <replace_in_hashes+7f/90>
   2:   89 58 04                  mov    %ebx,0x4(%eax)
Code;  c0247b32 <replace_in_hashes+82/90>
   5:   89 46 60                  mov    %eax,0x60(%esi)
Code;  c0247b35 <replace_in_hashes+85/90>
   8:   89 53 04                  mov    %edx,0x4(%ebx)
Code;  c0247b38 <replace_in_hashes+88/90>
   b:   89 1a                     mov    %ebx,(%edx)
Code;  c0247b3a <replace_in_hashes+8a/90>
   d:   5b                        pop    %ebx
Code;  c0247b3b <replace_in_hashes+8b/90>
   e:   5e                        pop    %esi
Code;  c0247b3c <replace_in_hashes+8c/90>
   f:   5f                        pop    %edi
Code;  c0247b3d <replace_in_hashes+8d/90>
  10:   5d                        pop    %ebp
Code;  c0247b3e <replace_in_hashes+8e/90>
  11:   c3                        ret    
Code;  c0247b3f <replace_in_hashes+8f/90>
  12:   90                        nop    
Code;  c0247b40 <place_in_hashes+0/80>
  13:   55                        push   %ebp

 <0>Kernel panic: Aiee, killing interrupt handler!

Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19

[Mike Smith]

On Tuesday 01 Jul 2003 21:05 pm, Martin Josefsson wrote:
> On Tue, 2003-07-01 at 11:47, David S. Miller wrote:
> > I believe 2.4.21 has a fix for this, but the netfilter
> > experts will know better.
>
> (Dave, it's netfilter-devel@lists.netfilter.org)
> (to the rest, the original report can be found on linux-net)
>
> It looks like list-corruption.
>
> ip_conntrack_core.c:665
> byipsproto[ipsprotohash].next points into space.
>
> I can't see anything obvious right now, my eyelids are very heavy so
> I'll continue looking tomorrow.
>
> There's a small possibility that this is caused by the modify-after-free
> bug that's fixed in 2.4.21 (but unfortunately introduced some
> refcounting bugs, patches are on the way to Dave, they can be found in
> netfilter patch-o-matic as well).

Hi,

this has happened again. Hand-transcribed oops below. Is this of interest?
I'm happy to accept debug patches etc. Or shall I just move to 2.4.21?

Thanks.

Oops: 0000
CPU:    0
EIP:    0010:[<c0247b2d>]   Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00210206
eax: c88a1fb8   ebx: c2a19cf0   ecx: 00200216   edx: c98a3068
esi: c2a19c90   edi: 0000017b   ebp: c02dbcfc   esp: c02dbcf0
ds: 0018   es: 0018   ss:0018
Process swapper (pid: 0, stackpage=c02db000)
Stack: c2a19bc0 c2a19c90 00000002 c02dbd20 c024625d c2a19bc0 c2a19c90 00000000
       00000002 c02dbd70 c031f0c0 c021e440 c02dbd48 c0213ea7 00000004 c02dbda0
       00000000 c5787000 c021e440 00000000 c5787000 00000004 c02dbd90 c021417d
Call Trace:    [<c024625d>] [<c021e440>] [<c0213ea7>] [<c021e440>] 
[<c021417d>]
  [<c021e440>] [<c021e409>] [<c021e440>] [<c0213ea7>] [<c021bbb6>] 
[<c021417d>]
  [<c02141b3>] [<c021bb27>] [<c021bb80>] [<c021aec3>] [<c021417d>] 
[<c02141b3>]
  [<c020a91a>] [<c021abe5>] [<c021ad50>] [<c0119b20>] [<c0119b31>] 
[<c020e90a>]
  [<c011a6ab>] [<c010a13c>] [<c0106fb0>] [<c010c5b8>] [<c0106fb0>] 
[<c0106fd6>]
  [<c0107062>] [<c0105000>]
Code: 8b 02 89 58 04 89 46 60 89 53 04 89 1a 5b 5e 5f 5d c3 90 55

>>EIP; c0247b2d <replace_in_hashes+7d/90>   <=====
Trace; c024625d <ip_nat_fn+13d/1a0>
Trace; c021e440 <ip_finish_output2+0/d0>
Trace; c0213ea7 <nf_iterate+27/90>
Trace; c021e440 <ip_finish_output2+0/d0>
Trace; c021417d <nf_hook_slow+9d/130>
Trace; c021e440 <ip_finish_output2+0/d0>
Trace; c021e409 <ip_finish_output+f9/110>
Trace; c021e440 <ip_finish_output2+0/d0>
Trace; c0213ea7 <nf_iterate+27/90>
Trace; c021bbb6 <ip_forward_finish+36/70>
Trace; c021417d <nf_hook_slow+9d/130>
Trace; c02141b3 <nf_hook_slow+d3/130>
Trace; c021bb27 <ip_forward+1b7/210>
Trace; c021bb80 <ip_forward_finish+0/70>
Trace; c021aec3 <ip_rcv_finish+173/1a0>
Trace; c021417d <nf_hook_slow+9d/130>
Trace; c02141b3 <nf_hook_slow+d3/130>
Trace; c020a91a <alloc_skb+da/1a0>
Trace; c021abe5 <ip_rcv+325/360>
Trace; c021ad50 <ip_rcv_finish+0/1a0>
Trace; c0119b20 <it_real_fn+0/60>
Trace; c0119b31 <it_real_fn+11/60>
Trace; c020e90a <net_rx_action+13a/210>
Trace; c011a6ab <do_softirq+5b/b0>
Trace; c010a13c <do_IRQ+ac/c0>
Trace; c0106fb0 <default_idle+0/30>
Trace; c010c5b8 <call_do_IRQ+5/d>
Trace; c0106fb0 <default_idle+0/30>
Trace; c0106fd6 <default_idle+26/30>
Trace; c0107062 <cpu_idle+52/70>
Trace; c0105000 <_stext+0/0>
Code;  c0247b2d <replace_in_hashes+7d/90>
00000000 <_EIP>:
Code;  c0247b2d <replace_in_hashes+7d/90>   <=====
   0:   8b 02                     mov    (%edx),%eax   <=====
Code;  c0247b2f <replace_in_hashes+7f/90>
   2:   89 58 04                  mov    %ebx,0x4(%eax)
Code;  c0247b32 <replace_in_hashes+82/90>
   5:   89 46 60                  mov    %eax,0x60(%esi)
Code;  c0247b35 <replace_in_hashes+85/90>
   8:   89 53 04                  mov    %edx,0x4(%ebx)
Code;  c0247b38 <replace_in_hashes+88/90>
   b:   89 1a                     mov    %ebx,(%edx)
Code;  c0247b3a <replace_in_hashes+8a/90>
   d:   5b                        pop    %ebx
Code;  c0247b3b <replace_in_hashes+8b/90>
   e:   5e                        pop    %esi
Code;  c0247b3c <replace_in_hashes+8c/90>
   f:   5f                        pop    %edi
Code;  c0247b3d <replace_in_hashes+8d/90>
  10:   5d                        pop    %ebp
Code;  c0247b3e <replace_in_hashes+8e/90>
  11:   c3                        ret    
Code;  c0247b3f <replace_in_hashes+8f/90>
  12:   90                        nop    
Code;  c0247b40 <place_in_hashes+0/80>
  13:   55                        push   %ebp

 <0>Kernel panic: Aiee, killing interrupt handler!

Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19

[Martin Josefsson]

On Mon, 21 Jul 2003, Mike Smith wrote:

> > There's a small possibility that this is caused by the modify-after-free
> > bug that's fixed in 2.4.21 (but unfortunately introduced some
> > refcounting bugs, patches are on the way to Dave, they can be found in
> > netfilter patch-o-matic as well).
>
> Hi,
>
> this has happened again. Hand-transcribed oops below. Is this of interest?
> I'm happy to accept debug patches etc. Or shall I just move to 2.4.21?

I'll look over it again later tonight, in the mean time could you switch
to 2.4.21? And apply everything from submitted and pending in
patch-o-matic from cvs (cvs instructions can be found on
http://netfilter.org). Any eventual debugging-patches will be against that
kernel and those patches.

/Martin

Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19

[Mike Smith]

On Monday 21 Jul 2003 11:22 am, Martin Josefsson wrote:
> I'll look over it again later tonight, in the mean time could you switch
> to 2.4.21? And apply everything from submitted and pending in
> patch-o-matic from cvs (cvs instructions can be found on
> http://netfilter.org). Any eventual debugging-patches will be against that
> kernel and those patches.
>
> /Martin

Martin,

thanks for taking an interest in this. I've now switched to 2.4.21.
I'll wait for a bit before taking the patch-o-matic plunge,
(to double-check that 'vanilla' 2.4.21 is stable).

--Mike

Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19

[Mike Smith]

On Monday 21 Jul 2003 11:22 am, Martin Josefsson wrote:
> I'll look over it again later tonight, in the mean time could you switch
> to 2.4.21? And apply everything from submitted and pending in
> patch-o-matic from cvs (cvs instructions can be found on
> http://netfilter.org). Any eventual debugging-patches will be against that
> kernel and those patches.
>
> /Martin

Martin,

thanks for taking an interest in this. I've now switched to 2.4.21.
I'll wait for a bit before taking the patch-o-matic plunge,
(to double-check that 'vanilla' 2.4.21 is stable).

--Mike

Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19 (and 2.4.21)

[Martin Josefsson]

On Tue, 22 Jul 2003, Mike Smith wrote:

> On Monday 21 Jul 2003 11:22 am, Martin Josefsson wrote:
> > I'll look over it again later tonight, in the mean time could you switch
> > to 2.4.21? And apply everything from submitted and pending in
> > patch-o-matic from cvs (cvs instructions can be found on
> > http://netfilter.org). Any eventual debugging-patches will be against that
> > kernel and those patches.
> >
> > /Martin
>
> Martin,
>
> vanilla 2.4.21 oopsed last night. Hand-transcribed oops below. I want to
> apply all the cvs fixes but am a bit short of clues. Is it pom-20030107
> that I want? (it is advertised as 2.4.18 thru 2.4.20...) or do I need to
> do a 'cvs update -d -P' ? Sorry to be so clueless but it's the first
> time I've had to do stuff like this. Help!

You don't want that pom release, it's very old.
You want to get pom from cvs. You have instructions on netfilter.org

cvs -d :pserver:cvs@pserver.netfilter.org:/cvspublic login
(password cvs)

cvs -d :pserver:cvs@pserver.netfilter.org:/cvspublic co netfilter/userspace netfilter/patch-o-matic
(you need the userspace directory since there's a patch that tries to
patch that as well)

cd netfilter/patch-o-matic
KERNEL_DIR=/usr/src/yourkernel ./runme --batch pending

then recompile and reboot.

It would be helpful if you could give me all your rules. If you don't want
the whole world to see them you can send them to me privately. And if you
could describe the type of traffic that goes through the router...

That way I might be able to reproduce the bug.

UP/SMP? any other patches applied? Which modules do you have loaded?
Or do you have the iptables stuff compiled into your kernel? If so, which
things did you select? ...
(You probably wrote some of this stuff in your first mail but I'm too lazy
to find it :)

/Martin

Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19 (and 2.4.21)

[Mike Smith]

On Monday 21 Jul 2003 11:22 am, Martin Josefsson wrote:
> I'll look over it again later tonight, in the mean time could you switch
> to 2.4.21? And apply everything from submitted and pending in
> patch-o-matic from cvs (cvs instructions can be found on
> http://netfilter.org). Any eventual debugging-patches will be against that
> kernel and those patches.
>
> /Martin

Martin,

vanilla 2.4.21 oopsed last night. Hand-transcribed oops below. I want to 
apply all the cvs fixes but am a bit short of clues. Is it pom-20030107
that I want? (it is advertised as 2.4.18 thru 2.4.20...) or do I need to
do a 'cvs update -d -P' ? Sorry to be so clueless but it's the first
time I've had to do stuff like this. Help!

Oops: 0000
CPU:    0
EIP:    0010:[<c0248d99>]   Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00210292
eax: c88a1fb8   ebx: c217d8f8   ecx: 00200292   edx: c98a3448
esi: c217d958   edi: 000000ea   ebp: c02edca8   esp: c02edc9c
ds: 0018   es: 0018   ss:0018
Process swapper (pid: 0, stackpage=c02ed000)
Stack: c217d860 c217d8f8 c02edd50 c02edcd0 c0247d7e c217d860 c217d8f8 00000002
       00000000 00000002 c02edd20 c032d8c0 c021ef60 c02edcf8 c02146e7 00000004
       c02edd50 00000000 c724f200 c021ef60 00000000 c724f200 00000004 c02edd40
Call Trace:    [<c0247d7e>] [<c021ef60>] [<c02146e7>] [<c021ef60>] 
[<c02149bd>]
  [<c021ef60>] [<c021ef29>] [<c021ef60>] [<c02146e7>] [<c021c696>] 
[<c02149bd>]
  [<c02149f3>] [<c021c607>] [<c021c660>] [<c021b743>] [<c02149bd>] 
[<c02149f3>]
  [<c021b465>] [<c021b5d0>] [<c020ec88>] [<c020ae1a>] [<c020efc1>] 
[<c020f05d>]
  [<c01b3f7f>] [<c020f170>] [<c011afab>] [<c010a17c>] [<c0106fb0>] 
[<c010c5f8>]
  [<c0106fb0>] [<c0106fd6>] [<c0107062>] [<c0105000>]
Code: 8b 02 89 70 04 89 43 60 89 56 04 89 32 5b 5e 5f 5d c3 90 8d

>>EIP; c0248d99 <replace_in_hashes+99/b0>   <=====
Trace; c0247d7e <ip_nat_fn+16e/1c0>
Trace; c021ef60 <ip_finish_output2+0/d0>
Trace; c02146e7 <nf_iterate+27/90>
Trace; c021ef60 <ip_finish_output2+0/d0>
Trace; c02149bd <nf_hook_slow+9d/130>
Trace; c021ef60 <ip_finish_output2+0/d0>
Trace; c021ef29 <ip_finish_output+f9/110>
Trace; c021ef60 <ip_finish_output2+0/d0>
Trace; c02146e7 <nf_iterate+27/90>
Trace; c021c696 <ip_forward_finish+36/70>
Trace; c02149bd <nf_hook_slow+9d/130>
Trace; c02149f3 <nf_hook_slow+d3/130>
Trace; c021c607 <ip_forward+1b7/210>
Trace; c021c660 <ip_forward_finish+0/70>
Trace; c021b743 <ip_rcv_finish+173/1a0>
Trace; c02149bd <nf_hook_slow+9d/130>
Trace; c02149f3 <nf_hook_slow+d3/130>
Trace; c021b465 <ip_rcv+325/360>
Trace; c021b5d0 <ip_rcv_finish+0/1a0>
Trace; c020ec88 <netif_rx+78/170>
Trace; c020ae1a <alloc_skb+da/1a0>
Trace; c020efc1 <netif_receive_skb+101/130>
Trace; c020f05d <process_backlog+6d/110>
Trace; c01b3f7f <boomerang_interrupt+12f/3a0>
Trace; c020f170 <net_rx_action+70/100>
Trace; c011afab <do_softirq+5b/b0>
Trace; c010a17c <do_IRQ+ac/c0>
Trace; c0106fb0 <default_idle+0/30>
Trace; c010c5f8 <call_do_IRQ+5/d>
Trace; c0106fb0 <default_idle+0/30>
Trace; c0106fd6 <default_idle+26/30>
Trace; c0107062 <cpu_idle+52/70>
Trace; c0105000 <_stext+0/0>
Code;  c0248d99 <replace_in_hashes+99/b0>
00000000 <_EIP>:
Code;  c0248d99 <replace_in_hashes+99/b0>   <=====
   0:   8b 02                     mov    (%edx),%eax   <=====
Code;  c0248d9b <replace_in_hashes+9b/b0>
   2:   89 70 04                  mov    %esi,0x4(%eax)
Code;  c0248d9e <replace_in_hashes+9e/b0>
   5:   89 43 60                  mov    %eax,0x60(%ebx)
Code;  c0248da1 <replace_in_hashes+a1/b0>
   8:   89 56 04                  mov    %edx,0x4(%esi)
Code;  c0248da4 <replace_in_hashes+a4/b0>
   b:   89 32                     mov    %esi,(%edx)
Code;  c0248da6 <replace_in_hashes+a6/b0>
   d:   5b                        pop    %ebx
Code;  c0248da7 <replace_in_hashes+a7/b0>
   e:   5e                        pop    %esi
Code;  c0248da8 <replace_in_hashes+a8/b0>
   f:   5f                        pop    %edi
Code;  c0248da9 <replace_in_hashes+a9/b0>
  10:   5d                        pop    %ebp
Code;  c0248daa <replace_in_hashes+aa/b0>
  11:   c3                        ret
Code;  c0248dab <replace_in_hashes+ab/b0>
  12:   90                        nop
Code;  c0248dac <replace_in_hashes+ac/b0>
  13:   8d 00                     lea    (%eax),%eax

 <0>Kernel panic: Aiee, killing interrupt handler!

Re: Oops in replace_in_hashes (ip_net_core.c) 2.4.19 (and 2.4.21)

[Mike Smith]

On Monday 21 Jul 2003 11:22 am, Martin Josefsson wrote:
> I'll look over it again later tonight, in the mean time could you switch
> to 2.4.21? And apply everything from submitted and pending in
> patch-o-matic from cvs (cvs instructions can be found on
> http://netfilter.org). Any eventual debugging-patches will be against that
> kernel and those patches.
>
> /Martin

Martin,

vanilla 2.4.21 oopsed last night. Hand-transcribed oops below. I want to 
apply all the cvs fixes but am a bit short of clues. Is it pom-20030107
that I want? (it is advertised as 2.4.18 thru 2.4.20...) or do I need to
do a 'cvs update -d -P' ? Sorry to be so clueless but it's the first
time I've had to do stuff like this. Help!

Oops: 0000
CPU:    0
EIP:    0010:[<c0248d99>]   Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00210292
eax: c88a1fb8   ebx: c217d8f8   ecx: 00200292   edx: c98a3448
esi: c217d958   edi: 000000ea   ebp: c02edca8   esp: c02edc9c
ds: 0018   es: 0018   ss:0018
Process swapper (pid: 0, stackpage=c02ed000)
Stack: c217d860 c217d8f8 c02edd50 c02edcd0 c0247d7e c217d860 c217d8f8 00000002
       00000000 00000002 c02edd20 c032d8c0 c021ef60 c02edcf8 c02146e7 00000004
       c02edd50 00000000 c724f200 c021ef60 00000000 c724f200 00000004 c02edd40
Call Trace:    [<c0247d7e>] [<c021ef60>] [<c02146e7>] [<c021ef60>] 
[<c02149bd>]
  [<c021ef60>] [<c021ef29>] [<c021ef60>] [<c02146e7>] [<c021c696>] 
[<c02149bd>]
  [<c02149f3>] [<c021c607>] [<c021c660>] [<c021b743>] [<c02149bd>] 
[<c02149f3>]
  [<c021b465>] [<c021b5d0>] [<c020ec88>] [<c020ae1a>] [<c020efc1>] 
[<c020f05d>]
  [<c01b3f7f>] [<c020f170>] [<c011afab>] [<c010a17c>] [<c0106fb0>] 
[<c010c5f8>]
  [<c0106fb0>] [<c0106fd6>] [<c0107062>] [<c0105000>]
Code: 8b 02 89 70 04 89 43 60 89 56 04 89 32 5b 5e 5f 5d c3 90 8d

>>EIP; c0248d99 <replace_in_hashes+99/b0>   <=====
Trace; c0247d7e <ip_nat_fn+16e/1c0>
Trace; c021ef60 <ip_finish_output2+0/d0>
Trace; c02146e7 <nf_iterate+27/90>
Trace; c021ef60 <ip_finish_output2+0/d0>
Trace; c02149bd <nf_hook_slow+9d/130>
Trace; c021ef60 <ip_finish_output2+0/d0>
Trace; c021ef29 <ip_finish_output+f9/110>
Trace; c021ef60 <ip_finish_output2+0/d0>
Trace; c02146e7 <nf_iterate+27/90>
Trace; c021c696 <ip_forward_finish+36/70>
Trace; c02149bd <nf_hook_slow+9d/130>
Trace; c02149f3 <nf_hook_slow+d3/130>
Trace; c021c607 <ip_forward+1b7/210>
Trace; c021c660 <ip_forward_finish+0/70>
Trace; c021b743 <ip_rcv_finish+173/1a0>
Trace; c02149bd <nf_hook_slow+9d/130>
Trace; c02149f3 <nf_hook_slow+d3/130>
Trace; c021b465 <ip_rcv+325/360>
Trace; c021b5d0 <ip_rcv_finish+0/1a0>
Trace; c020ec88 <netif_rx+78/170>
Trace; c020ae1a <alloc_skb+da/1a0>
Trace; c020efc1 <netif_receive_skb+101/130>
Trace; c020f05d <process_backlog+6d/110>
Trace; c01b3f7f <boomerang_interrupt+12f/3a0>
Trace; c020f170 <net_rx_action+70/100>
Trace; c011afab <do_softirq+5b/b0>
Trace; c010a17c <do_IRQ+ac/c0>
Trace; c0106fb0 <default_idle+0/30>
Trace; c010c5f8 <call_do_IRQ+5/d>
Trace; c0106fb0 <default_idle+0/30>
Trace; c0106fd6 <default_idle+26/30>
Trace; c0107062 <cpu_idle+52/70>
Trace; c0105000 <_stext+0/0>
Code;  c0248d99 <replace_in_hashes+99/b0>
00000000 <_EIP>:
Code;  c0248d99 <replace_in_hashes+99/b0>   <=====
   0:   8b 02                     mov    (%edx),%eax   <=====
Code;  c0248d9b <replace_in_hashes+9b/b0>
   2:   89 70 04                  mov    %esi,0x4(%eax)
Code;  c0248d9e <replace_in_hashes+9e/b0>
   5:   89 43 60                  mov    %eax,0x60(%ebx)
Code;  c0248da1 <replace_in_hashes+a1/b0>
   8:   89 56 04                  mov    %edx,0x4(%esi)
Code;  c0248da4 <replace_in_hashes+a4/b0>
   b:   89 32                     mov    %esi,(%edx)
Code;  c0248da6 <replace_in_hashes+a6/b0>
   d:   5b                        pop    %ebx
Code;  c0248da7 <replace_in_hashes+a7/b0>
   e:   5e                        pop    %esi
Code;  c0248da8 <replace_in_hashes+a8/b0>
   f:   5f                        pop    %edi
Code;  c0248da9 <replace_in_hashes+a9/b0>
  10:   5d                        pop    %ebp
Code;  c0248daa <replace_in_hashes+aa/b0>
  11:   c3                        ret
Code;  c0248dab <replace_in_hashes+ab/b0>
  12:   90                        nop
Code;  c0248dac <replace_in_hashes+ac/b0>
  13:   8d 00                     lea    (%eax),%eax

 <0>Kernel panic: Aiee, killing interrupt handler!