* Re: Unable to handle kernel paging request, another 2.6.16.25 server reboots
@ 2006-07-25 9:36 Chuck Ebbert
0 siblings, 0 replies; 2+ messages in thread
From: Chuck Ebbert @ 2006-07-25 9:36 UTC (permalink / raw)
To: Jim Klimov; +Cc: linux-kernel, linux-netdev
In-Reply-To: <254599816.20060724120148@2ka.mipt.ru>
On Mon, 24 Jul 2006 12:01:48 +0400, Jim Klimov wrote:
> I recently wrote about problems with a fileserver rebooting
> frequently. Another similar server got under NFS load today
> and rebooted at least twice in the past few hours.
>
> This server has a similar motherboard (Supermicro X5DP8-G2),
> dual Xeons@533, two older 3Ware controllers (7506+8506) and
> a reiserfs v3 archive.
>
> The server reported last week has two 3Ware 9550 controllers,
> ext3 archives and primarily a Samba usage.
I decoded your oops. It's in netfilter:
Unable to handle kernel paging request at virtual address f9445d43
printing eip:
c0392bba
*pde = 32e59067
Oops: 0000 [#1]
SMP
Modules linked in: w83781d hwmon_vid hwmon i2c_isa i2c_core w83627hf_wdt
CPU: 0
EIP: 0060:[<c0392bba>] Not tainted VLI
EFLAGS: 00010282 (2.6.16.25 #3)
EIP is at ipt_do_table+0xae/0x385
eax: 00000003 ebx: 00000000 ecx: cbf4b8d8 edx: f944a2c8
esi: e2262940 edi: f9445cf0 ebp: 80000000 esp: f700fac8
ds: 007b es: 007b ss: 0068
Process nfsd (pid: 10685, threadinfo=f700e000 task=f36a7ab0)
Stack: f9446b10 00000282 c33e2180 f36cda80 00000000 c047deec f944a2c8 f9418000
f7788800 c0530cd4 00000000 cbf4b8d8 00000000 00000003 f700fba0 00000000
f700fba0 00000003 c052f0d8 80000000 c03947e7 f7788800 c047dec0 00000000
Call Trace:
[<c03947e7>] ipt_local_out_hook+0x72/0x77
[<c035dfd9>] nf_iterate+0x69/0x83
[<c036a1ca>] dst_output+0x0/0x7
[<c036a1ca>] dst_output+0x0/0x7
[<c035e050>] nf_hook_slow+0x5d/0xea
[<c036a1ca>] dst_output+0x0/0x7
[<c0368178>] ip_queue_xmit+0x3d4/0x4f5
[<c036a1ca>] dst_output+0x0/0x7
[<c012ce4a>] __rcu_process_callbacks+0x7d/0xc5
[<c0115d92>] activate_task+0x99/0xa5
[<c011659c>] try_to_wake_up+0x29c/0x33b
[<c037d0d1>] tcp_v4_send_check+0x4a/0xdc
[<c037868d>] tcp_transmit_skb+0x2e6/0x45a
[<c0379879>] tcp_push_one+0x97/0x104
[<c036ec4c>] tcp_sendmsg+0x36b/0xb4d
[<c035dfd9>] nf_iterate+0x69/0x83
[<c037e653>] tcp_v4_rcv+0x4e6/0x81f
[<c0389cbe>] inet_sendmsg+0x47/0x5f
[<c0344768>] sock_sendmsg+0xc9/0xe3
[<c03649ad>] ip_rcv+0x2bc/0x56f
[<c034e58a>] netif_receive_skb+0x227/0x2d7
[<c0348c4c>] __kfree_skb+0x3a/0xc3
[<c012f3e4>] autoremove_wake_function+0x0/0x43
[<c0125923>] update_wall_time_one_tick+0x6/0x7e
[<c01259ce>] update_wall_time+0x8/0x35
[<c01062ab>] timer_interrupt+0x5b/0x86
[<c0139975>] handle_IRQ_event+0x26/0x59
[<c03447b0>] kernel_sendmsg+0x2e/0x3c
[<c0347a3f>] sock_no_sendpage+0x80/0x9f
[<c036e8a5>] tcp_sendpage+0x49/0x85
[<c03a9573>] svc_sendto+0x134/0x250
[<c034e7ce>] net_rx_action+0x88/0x15f
[<c0104f82>] do_IRQ+0x1e/0x24
[<c01035e2>] common_interrupt+0x1a/0x20
[<c03aa597>] svc_tcp_sendto+0x4d/0x99
[<c0258eab>] _atomic_dec_and_lock+0x33/0x4c
[<c03aad1a>] svc_send+0xaa/0xed
[<c0210abc>] fh_put+0x133/0x17d
[<c03ac4da>] svcauth_unix_release+0x43/0x45
[<c021d1fd>] nfs3svc_release_fhandle+0x0/0xe
[<c03a8b14>] svc_process+0x1b1/0x619
[<c01183f8>] default_wake_function+0x0/0xc
[<c020e10d>] nfsd+0x178/0x301
[<c020df95>] nfsd+0x0/0x301
[<c01010a1>] kernel_thread_helper+0x5/0xb
6: 8b 40 10 mov 0x10(%eax),%eax
9: 8b 44 86 34 mov 0x34(%esi,%eax,4),%eax
d: 89 44 24 1c mov %eax,0x1c(%esp)
11: 89 c7 mov %eax,%edi
13: 8b 44 24 34 mov 0x34(%esp),%eax
17: 8b 54 24 1c mov 0x1c(%esp),%edx
1b: 03 7c 86 0c add 0xc(%esi,%eax,4),%edi
1f: 03 54 86 20 add 0x20(%esi,%eax,4),%edx
23: 89 5c 24 10 mov %ebx,0x10(%esp)
27: 89 54 24 18 mov %edx,0x18(%esp)
0: 0f b6 5f 53 movzbl 0x53(%edi),%ebx <=====
4: 89 d8 mov %ebx,%eax
6: 24 08 and $0x8,%al
8: 84 c0 test %al,%al
a: 0f 84 b4 02 00 00 je 2c4 <_EIP+0x2c4>
10: 8b 47 08 mov 0x8(%edi),%eax
This is in net/ipv4/netfiler/ip_tables.c::ipt_do_table():
table_base = (void *)private->entries[smp_processor_id()];
e = get_entry(table_base, private->hook_entry[hook]);
/* For return from builtin chain */
back = get_entry(table_base, private->underflow[hook]);
do {
IP_NF_ASSERT(e);
IP_NF_ASSERT(back);
===> if (ip_packet_match(ip, indev, outdev, &e->ip, offset)) {
'e' is an invalid pointer. (ip_packet_match() was inlined.)
hook == 3
The call trace seems to show that svc_tcp_sendto() was interrupted by an
IRQ for an incoming packet, or maybe the timer interrupt?
Can you build with CONFIG_FRAME_POINTERS and see if you can get a cleaner
trace?
--
Chuck
^ permalink raw reply [flat|nested] 2+ messages in thread
* Unable to handle kernel paging request, another 2.6.16.25 server reboots
@ 2006-07-24 8:01 Jim Klimov
0 siblings, 0 replies; 2+ messages in thread
From: Jim Klimov @ 2006-07-24 8:01 UTC (permalink / raw)
To: linux-kernel
Hello linux-kernel,
I recently wrote about problems with a fileserver rebooting
frequently. Another similar server got under NFS load today
and rebooted at least twice in the past few hours.
This server has a similar motherboard (Supermicro X5DP8-G2),
dual Xeons@533, two older 3Ware controllers (7506+8506) and
a reiserfs v3 archive.
The server reported last week has two 3Ware 9550 controllers,
ext3 archives and primarily a Samba usage.
[32262.075038] Unable to handle kernel paging request at virtual address f9445d43
[32262.082673] printing eip:
[32262.085515] c0392bba
[32262.087822] *pde = 32e59067
[32262.090729] Oops: 0000 [#1]
[32262.093636] SMP
[32262.095649] Modules linked in: w83781d hwmon_vid hwmon i2c_isa i2c_core w83627hf_wdt
[32262.104138] CPU: 0
[32262.104139] EIP: 0060:[<c0392bba>] Not tainted VLI
[32262.104140] EFLAGS: 00010282 (2.6.16.25 #3)
[32262.116474] EIP is at ipt_do_table+0xae/0x385
[32262.121055] eax: 00000003 ebx: 00000000 ecx: cbf4b8d8 edx: f944a2c8
[32262.128063] esi: e2262940 edi: f9445cf0 ebp: 80000000 esp: f700fac8
[32262.135095] ds: 007b es: 007b ss: 0068
[32262.139328] Process nfsd (pid: 10685, threadinfo=f700e000 task=f36a7ab0)
[32262.146127] Stack: <0>f9446b10 00000282 c33e2180 f36cda80 00000000 c047deec f944a2c8 f9418000
[32262.155812] f7788800 c0530cd4 00000000 cbf4b8d8 00000000 00000003 f700fba0 00000000
[32262.164948] f700fba0 00000003 c052f0d8 80000000 c03947e7 f7788800 c047dec0 00000000
[32262.174128] Call Trace:
[32262.176759] [<c03947e7>] ipt_local_out_hook+0x72/0x77
[32262.182228] [<c035dfd9>] nf_iterate+0x69/0x83
[32262.186964] [<c036a1ca>] dst_output+0x0/0x7
[32262.191502] [<c036a1ca>] dst_output+0x0/0x7
[32262.196055] [<c035e050>] nf_hook_slow+0x5d/0xea
[32262.200995] [<c036a1ca>] dst_output+0x0/0x7
[32262.205528] [<c0368178>] ip_queue_xmit+0x3d4/0x4f5
[32262.210724] [<c036a1ca>] dst_output+0x0/0x7
[32262.215276] [<c012ce4a>] __rcu_process_callbacks+0x7d/0xc5
[32262.221217] [<c0115d92>] activate_task+0x99/0xa5
[32262.226231] [<c011659c>] try_to_wake_up+0x29c/0x33b
[32262.231540] [<c037d0d1>] tcp_v4_send_check+0x4a/0xdc
[32262.236945] [<c037868d>] tcp_transmit_skb+0x2e6/0x45a
[32262.242419] [<c0379879>] tcp_push_one+0x97/0x104
[32262.247423] [<c036ec4c>] tcp_sendmsg+0x36b/0xb4d
[32262.252436] [<c035dfd9>] nf_iterate+0x69/0x83
[32262.257190] [<c037e653>] tcp_v4_rcv+0x4e6/0x81f
[32262.262101] [<c0389cbe>] inet_sendmsg+0x47/0x5f
[32262.267017] [<c0344768>] sock_sendmsg+0xc9/0xe3
[32262.271942] [<c03649ad>] ip_rcv+0x2bc/0x56f
[32262.276481] [<c034e58a>] netif_receive_skb+0x227/0x2d7
[32262.282039] [<c0348c4c>] __kfree_skb+0x3a/0xc3
[32262.286881] [<c012f3e4>] autoremove_wake_function+0x0/0x43
[32262.292741] [<c0125923>] update_wall_time_one_tick+0x6/0x7e
[32262.298772] [<c01259ce>] update_wall_time+0x8/0x35
[32262.303957] [<c01062ab>] timer_interrupt+0x5b/0x86
[32262.309124] [<c0139975>] handle_IRQ_event+0x26/0x59
[32262.314444] [<c03447b0>] kernel_sendmsg+0x2e/0x3c
[32262.319525] [<c0347a3f>] sock_no_sendpage+0x80/0x9f
[32262.324772] [<c036e8a5>] tcp_sendpage+0x49/0x85
[32262.329708] [<c03a9573>] svc_sendto+0x134/0x250
[32262.334651] [<c034e7ce>] net_rx_action+0x88/0x15f
[32262.339777] [<c0104f82>] do_IRQ+0x1e/0x24
[32262.344129] [<c01035e2>] common_interrupt+0x1a/0x20
[32262.349422] [<c03aa597>] svc_tcp_sendto+0x4d/0x99
[32262.354495] [<c0258eab>] _atomic_dec_and_lock+0x33/0x4c
[32262.360121] [<c03aad1a>] svc_send+0xaa/0xed
[32262.364691] [<c0210abc>] fh_put+0x133/0x17d
[32262.369239] [<c03ac4da>] svcauth_unix_release+0x43/0x45
[32262.374885] [<c021d1fd>] nfs3svc_release_fhandle+0x0/0xe
[32262.380606] [<c03a8b14>] svc_process+0x1b1/0x619
[32262.386970] [<c01183f8>] default_wake_function+0x0/0xc
[32262.392485] [<c020e10d>] nfsd+0x178/0x301
[32262.396891] [<c020df95>] nfsd+0x0/0x301
[32262.401132] [<c01010a1>] kernel_thread_helper+0x5/0xb
[32262.406560] Code: ff 21 e0 0f b7 db 8b 40 10 8b 44 86 34 89 44 24 1c
89 c7 8b 44 24 34 8b 54 24 1c 03 7c 86 0c 03 54 86 20 89 5c 24 10 89 54
24 18 <0f> b6 5f 53 89 d8 24 08 84 c0 0f 84 b4 02 00 00 8b 47 08 8b 4c
[32262.429623] <0>Kernel panic - not syncing: Fatal exception in interrupt
[32262.436516] <0>Rebooting in 1 seconds..
--
Best regards,
Jim Klimov mailto:klimov@2ka.mipt.ru
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2006-07-25 9:41 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2006-07-25 9:36 Unable to handle kernel paging request, another 2.6.16.25 server reboots Chuck Ebbert
-- strict thread matches above, loose matches on Subject: below --
2006-07-24 8:01 Jim Klimov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).