linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Re: Unable to handle kernel paging request, another 2.6.16.25 server reboots
@ 2006-07-25  9:36 Chuck Ebbert
  0 siblings, 0 replies; 2+ messages in thread
From: Chuck Ebbert @ 2006-07-25  9:36 UTC (permalink / raw)
  To: Jim Klimov; +Cc: linux-kernel, linux-netdev

In-Reply-To: <254599816.20060724120148@2ka.mipt.ru>

On Mon, 24 Jul 2006 12:01:48 +0400, Jim Klimov wrote:

>  I recently wrote about problems with a fileserver rebooting
>  frequently. Another similar server got under NFS load today
>  and rebooted at least twice in the past few hours.
>
>  This server has a similar motherboard (Supermicro X5DP8-G2),
>  dual Xeons@533, two older 3Ware controllers (7506+8506) and
>  a reiserfs v3 archive.
>
>  The server reported last week has two 3Ware 9550 controllers,
>  ext3 archives and primarily a Samba usage.

I decoded your oops.  It's in netfilter:

Unable to handle kernel paging request at virtual address f9445d43
printing eip:
 c0392bba
*pde = 32e59067
Oops: 0000 [#1]
 SMP 
Modules linked in: w83781d hwmon_vid hwmon i2c_isa i2c_core w83627hf_wdt
CPU:    0
EIP:    0060:[<c0392bba>]    Not tainted VLI
EFLAGS: 00010282   (2.6.16.25 #3) 
EIP is at ipt_do_table+0xae/0x385
eax: 00000003   ebx: 00000000   ecx: cbf4b8d8   edx: f944a2c8
esi: e2262940   edi: f9445cf0   ebp: 80000000   esp: f700fac8
ds: 007b   es: 007b   ss: 0068
Process nfsd (pid: 10685, threadinfo=f700e000 task=f36a7ab0)
Stack: f9446b10 00000282 c33e2180 f36cda80 00000000 c047deec f944a2c8 f9418000 
       f7788800 c0530cd4 00000000 cbf4b8d8 00000000 00000003 f700fba0 00000000 
       f700fba0 00000003 c052f0d8 80000000 c03947e7 f7788800 c047dec0 00000000 
Call Trace:
 [<c03947e7>] ipt_local_out_hook+0x72/0x77
 [<c035dfd9>] nf_iterate+0x69/0x83
 [<c036a1ca>] dst_output+0x0/0x7
 [<c036a1ca>] dst_output+0x0/0x7
 [<c035e050>] nf_hook_slow+0x5d/0xea
 [<c036a1ca>] dst_output+0x0/0x7
 [<c0368178>] ip_queue_xmit+0x3d4/0x4f5
 [<c036a1ca>] dst_output+0x0/0x7
 [<c012ce4a>] __rcu_process_callbacks+0x7d/0xc5
 [<c0115d92>] activate_task+0x99/0xa5
 [<c011659c>] try_to_wake_up+0x29c/0x33b
 [<c037d0d1>] tcp_v4_send_check+0x4a/0xdc
 [<c037868d>] tcp_transmit_skb+0x2e6/0x45a
 [<c0379879>] tcp_push_one+0x97/0x104
 [<c036ec4c>] tcp_sendmsg+0x36b/0xb4d
 [<c035dfd9>] nf_iterate+0x69/0x83
 [<c037e653>] tcp_v4_rcv+0x4e6/0x81f
 [<c0389cbe>] inet_sendmsg+0x47/0x5f
 [<c0344768>] sock_sendmsg+0xc9/0xe3
 [<c03649ad>] ip_rcv+0x2bc/0x56f
 [<c034e58a>] netif_receive_skb+0x227/0x2d7
 [<c0348c4c>] __kfree_skb+0x3a/0xc3
 [<c012f3e4>] autoremove_wake_function+0x0/0x43
 [<c0125923>] update_wall_time_one_tick+0x6/0x7e
 [<c01259ce>] update_wall_time+0x8/0x35
 [<c01062ab>] timer_interrupt+0x5b/0x86
 [<c0139975>] handle_IRQ_event+0x26/0x59
 [<c03447b0>] kernel_sendmsg+0x2e/0x3c
 [<c0347a3f>] sock_no_sendpage+0x80/0x9f
 [<c036e8a5>] tcp_sendpage+0x49/0x85
 [<c03a9573>] svc_sendto+0x134/0x250
 [<c034e7ce>] net_rx_action+0x88/0x15f
 [<c0104f82>] do_IRQ+0x1e/0x24
 [<c01035e2>] common_interrupt+0x1a/0x20
 [<c03aa597>] svc_tcp_sendto+0x4d/0x99
 [<c0258eab>] _atomic_dec_and_lock+0x33/0x4c
 [<c03aad1a>] svc_send+0xaa/0xed
 [<c0210abc>] fh_put+0x133/0x17d
 [<c03ac4da>] svcauth_unix_release+0x43/0x45
 [<c021d1fd>] nfs3svc_release_fhandle+0x0/0xe
 [<c03a8b14>] svc_process+0x1b1/0x619
 [<c01183f8>] default_wake_function+0x0/0xc
 [<c020e10d>] nfsd+0x178/0x301
 [<c020df95>] nfsd+0x0/0x301
 [<c01010a1>] kernel_thread_helper+0x5/0xb

   6:   8b 40 10                  mov    0x10(%eax),%eax
   9:   8b 44 86 34               mov    0x34(%esi,%eax,4),%eax
   d:   89 44 24 1c               mov    %eax,0x1c(%esp)
  11:   89 c7                     mov    %eax,%edi
  13:   8b 44 24 34               mov    0x34(%esp),%eax
  17:   8b 54 24 1c               mov    0x1c(%esp),%edx
  1b:   03 7c 86 0c               add    0xc(%esi,%eax,4),%edi
  1f:   03 54 86 20               add    0x20(%esi,%eax,4),%edx
  23:   89 5c 24 10               mov    %ebx,0x10(%esp)
  27:   89 54 24 18               mov    %edx,0x18(%esp)
   0:   0f b6 5f 53               movzbl 0x53(%edi),%ebx   <=====
   4:   89 d8                     mov    %ebx,%eax
   6:   24 08                     and    $0x8,%al
   8:   84 c0                     test   %al,%al
   a:   0f 84 b4 02 00 00         je     2c4 <_EIP+0x2c4>
  10:   8b 47 08                  mov    0x8(%edi),%eax

This is in net/ipv4/netfiler/ip_tables.c::ipt_do_table():

        table_base = (void *)private->entries[smp_processor_id()];
        e = get_entry(table_base, private->hook_entry[hook]);

        /* For return from builtin chain */
        back = get_entry(table_base, private->underflow[hook]);

        do {
                IP_NF_ASSERT(e);
                IP_NF_ASSERT(back);
===>            if (ip_packet_match(ip, indev, outdev, &e->ip, offset)) {

'e' is an invalid pointer. (ip_packet_match() was inlined.)
hook == 3


The call trace seems to show that svc_tcp_sendto() was interrupted by an
IRQ for an incoming packet, or maybe the timer interrupt?

Can you build with CONFIG_FRAME_POINTERS and see if you can get a cleaner
trace?

-- 
Chuck

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Unable to handle kernel paging request, another 2.6.16.25 server reboots
@ 2006-07-24  8:01 Jim Klimov
  0 siblings, 0 replies; 2+ messages in thread
From: Jim Klimov @ 2006-07-24  8:01 UTC (permalink / raw)
  To: linux-kernel

Hello linux-kernel,

  I recently wrote about problems with a fileserver rebooting
  frequently. Another similar server got under NFS load today
  and rebooted at least twice in the past few hours.

  This server has a similar motherboard (Supermicro X5DP8-G2),
  dual Xeons@533, two older 3Ware controllers (7506+8506) and
  a reiserfs v3 archive.

  The server reported last week has two 3Ware 9550 controllers,
  ext3 archives and primarily a Samba usage.

[32262.075038] Unable to handle kernel paging request at virtual address f9445d43
[32262.082673]  printing eip:
[32262.085515] c0392bba
[32262.087822] *pde = 32e59067
[32262.090729] Oops: 0000 [#1]
[32262.093636] SMP 
[32262.095649] Modules linked in: w83781d hwmon_vid hwmon i2c_isa i2c_core w83627hf_wdt
[32262.104138] CPU:    0
[32262.104139] EIP:    0060:[<c0392bba>]    Not tainted VLI
[32262.104140] EFLAGS: 00010282   (2.6.16.25 #3) 
[32262.116474] EIP is at ipt_do_table+0xae/0x385
[32262.121055] eax: 00000003   ebx: 00000000   ecx: cbf4b8d8   edx: f944a2c8
[32262.128063] esi: e2262940   edi: f9445cf0   ebp: 80000000   esp: f700fac8
[32262.135095] ds: 007b   es: 007b   ss: 0068
[32262.139328] Process nfsd (pid: 10685, threadinfo=f700e000 task=f36a7ab0)
[32262.146127] Stack: <0>f9446b10 00000282 c33e2180 f36cda80 00000000 c047deec f944a2c8 f9418000 
[32262.155812]        f7788800 c0530cd4 00000000 cbf4b8d8 00000000 00000003 f700fba0 00000000 
[32262.164948]        f700fba0 00000003 c052f0d8 80000000 c03947e7 f7788800 c047dec0 00000000 
[32262.174128] Call Trace:
[32262.176759]  [<c03947e7>] ipt_local_out_hook+0x72/0x77
[32262.182228]  [<c035dfd9>] nf_iterate+0x69/0x83
[32262.186964]  [<c036a1ca>] dst_output+0x0/0x7
[32262.191502]  [<c036a1ca>] dst_output+0x0/0x7
[32262.196055]  [<c035e050>] nf_hook_slow+0x5d/0xea
[32262.200995]  [<c036a1ca>] dst_output+0x0/0x7
[32262.205528]  [<c0368178>] ip_queue_xmit+0x3d4/0x4f5
[32262.210724]  [<c036a1ca>] dst_output+0x0/0x7
[32262.215276]  [<c012ce4a>] __rcu_process_callbacks+0x7d/0xc5
[32262.221217]  [<c0115d92>] activate_task+0x99/0xa5
[32262.226231]  [<c011659c>] try_to_wake_up+0x29c/0x33b
[32262.231540]  [<c037d0d1>] tcp_v4_send_check+0x4a/0xdc
[32262.236945]  [<c037868d>] tcp_transmit_skb+0x2e6/0x45a
[32262.242419]  [<c0379879>] tcp_push_one+0x97/0x104
[32262.247423]  [<c036ec4c>] tcp_sendmsg+0x36b/0xb4d
[32262.252436]  [<c035dfd9>] nf_iterate+0x69/0x83
[32262.257190]  [<c037e653>] tcp_v4_rcv+0x4e6/0x81f
[32262.262101]  [<c0389cbe>] inet_sendmsg+0x47/0x5f
[32262.267017]  [<c0344768>] sock_sendmsg+0xc9/0xe3
[32262.271942]  [<c03649ad>] ip_rcv+0x2bc/0x56f
[32262.276481]  [<c034e58a>] netif_receive_skb+0x227/0x2d7
[32262.282039]  [<c0348c4c>] __kfree_skb+0x3a/0xc3
[32262.286881]  [<c012f3e4>] autoremove_wake_function+0x0/0x43
[32262.292741]  [<c0125923>] update_wall_time_one_tick+0x6/0x7e
[32262.298772]  [<c01259ce>] update_wall_time+0x8/0x35
[32262.303957]  [<c01062ab>] timer_interrupt+0x5b/0x86
[32262.309124]  [<c0139975>] handle_IRQ_event+0x26/0x59
[32262.314444]  [<c03447b0>] kernel_sendmsg+0x2e/0x3c
[32262.319525]  [<c0347a3f>] sock_no_sendpage+0x80/0x9f
[32262.324772]  [<c036e8a5>] tcp_sendpage+0x49/0x85
[32262.329708]  [<c03a9573>] svc_sendto+0x134/0x250
[32262.334651]  [<c034e7ce>] net_rx_action+0x88/0x15f
[32262.339777]  [<c0104f82>] do_IRQ+0x1e/0x24
[32262.344129]  [<c01035e2>] common_interrupt+0x1a/0x20
[32262.349422]  [<c03aa597>] svc_tcp_sendto+0x4d/0x99
[32262.354495]  [<c0258eab>] _atomic_dec_and_lock+0x33/0x4c
[32262.360121]  [<c03aad1a>] svc_send+0xaa/0xed
[32262.364691]  [<c0210abc>] fh_put+0x133/0x17d
[32262.369239]  [<c03ac4da>] svcauth_unix_release+0x43/0x45
[32262.374885]  [<c021d1fd>] nfs3svc_release_fhandle+0x0/0xe
[32262.380606]  [<c03a8b14>] svc_process+0x1b1/0x619
[32262.386970]  [<c01183f8>] default_wake_function+0x0/0xc
[32262.392485]  [<c020e10d>] nfsd+0x178/0x301
[32262.396891]  [<c020df95>] nfsd+0x0/0x301
[32262.401132]  [<c01010a1>] kernel_thread_helper+0x5/0xb
[32262.406560] Code: ff 21 e0 0f b7 db 8b 40 10 8b 44 86 34 89 44 24 1c
89 c7 8b 44 24 34 8b 54 24 1c 03 7c 86 0c 03 54 86 20 89 5c 24 10 89 54
24 18 <0f> b6 5f 53 89 d8 24 08 84 c0 0f 84 b4 02 00 00 8b 47 08 8b 4c
[32262.429623]  <0>Kernel panic - not syncing: Fatal exception in interrupt
[32262.436516]  <0>Rebooting in 1 seconds..  

-- 
Best regards,
 Jim Klimov                          mailto:klimov@2ka.mipt.ru


^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2006-07-25  9:41 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2006-07-25  9:36 Unable to handle kernel paging request, another 2.6.16.25 server reboots Chuck Ebbert
  -- strict thread matches above, loose matches on Subject: below --
2006-07-24  8:01 Jim Klimov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).