* 3.4.0-rc2: skb_put() -> skb_over_panic
@ 2012-04-15 12:24 Alexander Beregalov
2012-04-15 15:16 ` Eric Dumazet
0 siblings, 1 reply; 11+ messages in thread
From: Alexander Beregalov @ 2012-04-15 12:24 UTC (permalink / raw)
To: netdev, Linux Kernel Mailing List
Hi
kernel 3.4.0-rc2-00333-g668ce0a
calltrace is lost on netconsole, sorry.
ethernet is realtek/r8169
It happened already two times with rtorrent, perhaps I can reproduce
it, what else can I provide to you?
skb_over_panic: text:ffffffff8136d919 len:1016 put:815
head:ffff8800afe4d800 data:ffff8800afe4dd6b tail:0x963 end:0x6c0
dev:<NULL>
------------[ cut here ]------------
kernel BUG at net/core/skbuff.c:127!
invalid opcode: 0000 [#1] SMP
CPU 2
Modules linked in:
Pid: 0, comm: swapper/2 Not tainted 3.4.0-rc2-00333-g668ce0a #1
/D525MW
RIP: 0010:[<ffffffff8132414a>] [<ffffffff8132414a>] skb_put+0x7c/0x86
RSP: 0018:ffff8800bed03950 EFLAGS: 00010246
RAX: 0000000000000098 RBX: ffff88002883a580 RCX: 0000000000000015
RDX: 0000000000000001 RSI: 0000000000000046 RDI: ffffffff8162a0b0
RBP: ffff8800bed03970 R08: ffffffff813cd540 R09: ffff8800bed03650
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88002895c380
R13: ffff88002895ca80 R14: ffff88002895c3a8 R15: ffff88002883a660
FS: 0000000000000000(0000) GS:ffff8800bed00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fcbba6e1000 CR3: 0000000001520000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/2 (pid: 0, threadinfo ffff8800bb0a8000, task ffff8800bb06d780)
Stack:
0000000000000963 00000000000006c0 ffffffff814d06e7 ffff88002895c380
ffff8800bed039d0 ffffffff8136d919 ffff88000000032f 000000c98102ebbd
ffff88002883a660 0000019c0000250f ffff88002883a580 ffff88002883a580
Call Trace:
<IRQ>
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: 3.4.0-rc2: skb_put() -> skb_over_panic 2012-04-15 12:24 3.4.0-rc2: skb_put() -> skb_over_panic Alexander Beregalov @ 2012-04-15 15:16 ` Eric Dumazet 2012-04-17 20:15 ` Alexander Beregalov 0 siblings, 1 reply; 11+ messages in thread From: Eric Dumazet @ 2012-04-15 15:16 UTC (permalink / raw) To: Alexander Beregalov; +Cc: netdev, Linux Kernel Mailing List On Sun, 2012-04-15 at 16:24 +0400, Alexander Beregalov wrote: > Hi > > kernel 3.4.0-rc2-00333-g668ce0a > > calltrace is lost on netconsole, sorry. > > ethernet is realtek/r8169 > > It happened already two times with rtorrent, perhaps I can reproduce > it, what else can I provide to you? > > skb_over_panic: text:ffffffff8136d919 len:1016 put:815 > head:ffff8800afe4d800 data:ffff8800afe4dd6b tail:0x963 end:0x6c0 > dev:<NULL> > ------------[ cut here ]------------ > kernel BUG at net/core/skbuff.c:127! > invalid opcode: 0000 [#1] SMP > CPU 2 > Modules linked in: > > Pid: 0, comm: swapper/2 Not tainted 3.4.0-rc2-00333-g668ce0a #1 > /D525MW > RIP: 0010:[<ffffffff8132414a>] [<ffffffff8132414a>] skb_put+0x7c/0x86 > RSP: 0018:ffff8800bed03950 EFLAGS: 00010246 > RAX: 0000000000000098 RBX: ffff88002883a580 RCX: 0000000000000015 > RDX: 0000000000000001 RSI: 0000000000000046 RDI: ffffffff8162a0b0 > RBP: ffff8800bed03970 R08: ffffffff813cd540 R09: ffff8800bed03650 > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88002895c380 > R13: ffff88002895ca80 R14: ffff88002895c3a8 R15: ffff88002883a660 > FS: 0000000000000000(0000) GS:ffff8800bed00000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 00007fcbba6e1000 CR3: 0000000001520000 CR4: 00000000000007e0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process swapper/2 (pid: 0, threadinfo ffff8800bb0a8000, task ffff8800bb06d780) > Stack: > 0000000000000963 00000000000006c0 ffffffff814d06e7 ffff88002895c380 > ffff8800bed039d0 ffffffff8136d919 ffff88000000032f 000000c98102ebbd > ffff88002883a660 0000019c0000250f ffff88002883a580 ffff88002883a580 > Call Trace: > <IRQ> > -- full stack trace needed please. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 3.4.0-rc2: skb_put() -> skb_over_panic 2012-04-15 15:16 ` Eric Dumazet @ 2012-04-17 20:15 ` Alexander Beregalov 2012-04-17 20:45 ` Eric Dumazet 0 siblings, 1 reply; 11+ messages in thread From: Alexander Beregalov @ 2012-04-17 20:15 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev, Linux Kernel Mailing List On 15 April 2012 19:16, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Sun, 2012-04-15 at 16:24 +0400, Alexander Beregalov wrote: >> Hi >> >> kernel 3.4.0-rc2-00333-g668ce0a >> >> calltrace is lost on netconsole, sorry. >> >> ethernet is realtek/r8169 >> >> It happened already two times with rtorrent, perhaps I can reproduce >> it, what else can I provide to you? >> > full stack trace needed please. > This time calltrace is different (I saw few last lines of calltrace on a display, but not enough) and netconsole transmitted complete message, but perhaps it is the same problem. At least 'end' is the same. skb_over_panic: text:ffffffff8136d919 len:1248 put:932 head:ffff8800babcf800 data:ffff8800babcfd10 tail:0x9f0 end:0x6c0 dev:<NULL> ------------[ cut here ]------------ kernel BUG at net/core/skbuff.c:127! invalid opcode: 0000 [#1] SMP CPU 3 Modules linked in: Pid: 1926, comm: rtorrent Not tainted 3.4.0-rc2-00333-g668ce0a #1 /D525MW RIP: 0010:[<ffffffff8132414a>] [<ffffffff8132414a>] skb_put+0x7c/0x86 RSP: 0018:ffff8800bed83d60 EFLAGS: 00010246 RAX: 0000000000000098 RBX: ffff8800b0b244c0 RCX: 000000000000003d RDX: 000000000000000d RSI: 0000000000000046 RDI: ffffffff8162a0b0 RBP: ffff8800bed83d80 R08: 0000000000000001 R09: 0000000000000000 R10: ffff88002ed840c0 R11: 00000000007e28f6 R12: ffff8800b9cb9180 R13: ffff8800b9cb96c0 R14: ffff8800b9cb91a8 R15: ffff8800b0b245a0 FS: 00007fe639e75720(0000) GS:ffff8800bed80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fe63571a000 CR3: 00000000b53fa000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process rtorrent (pid: 1926, threadinfo ffff8800bb0a0000, task ffff8800bb1dddc0) Stack: 00000000000009f0 00000000000006c0 ffffffff814d06e7 ffff8800b9cb9180 ffff8800bed83de0 ffffffff8136d919 ffff8800000003a4 0000013c81536640 ffff8800b0b245a0 000000cc00000006 ffff8800b0b244c0 ffff8800b0b244c0 Call Trace: <IRQ> [<ffffffff8136d919>] tcp_retransmit_skb+0x29a/0x529 [<ffffffff8136eebd>] tcp_retransmit_timer+0x358/0x4e7 [<ffffffff8136f0e8>] tcp_write_timer+0x9c/0x17d [<ffffffff81035bd0>] run_timer_softirq+0x1eb/0x2bf [<ffffffff8136f04c>] ? tcp_retransmit_timer+0x4e7/0x4e7 [<ffffffff810184e9>] ? native_smp_send_reschedule+0x4f/0x51 [<ffffffff8102fb52>] __do_softirq+0xbf/0x17d [<ffffffff810188ec>] ? lapic_next_event+0x18/0x1c [<ffffffff813c05cc>] call_softirq+0x1c/0x30 [<ffffffff810033d6>] do_softirq+0x33/0x69 [<ffffffff8102fdc6>] irq_exit+0x44/0x9c [<ffffffff81018c6f>] smp_apic_timer_interrupt+0x86/0x94 [<ffffffff813bfe47>] apic_timer_interrupt+0x67/0x70 <EOI> [<ffffffff8131d87e>] ? sock_sendmsg+0xe6/0x106 [<ffffffff8135fb7b>] ? tcp_poll+0xaf/0x168 [<ffffffff810ff814>] ? ep_send_events_proc+0x67/0x116 [<ffffffff8131b8ef>] sock_poll+0x15/0x17 [<ffffffff810ff823>] ep_send_events_proc+0x76/0x116 [<ffffffff810ff7ad>] ? ep_read_events_proc+0x99/0x99 [<ffffffff810fff24>] ep_scan_ready_list.clone.6+0x8f/0x16f [<ffffffff81100277>] ep_poll+0x25f/0x2e2 [<ffffffff8131e886>] ? sys_accept4+0x133/0x15f [<ffffffff81100d8e>] sys_epoll_wait+0x90/0xae [<ffffffff813bf2e6>] system_call_fastpath+0x1a/0x1f Code: 8b 57 60 48 89 44 24 10 8b 87 ac 00 00 00 48 89 44 24 08 31 c0 8b bf a8 00 00 00 48 89 3c 24 48 c7 c7 43 07 4d 81 e8 2c 27 09 00 <0f> 0b 89 c0 49 8d 04 00 c9 c3 55 48 89 e5 41 57 41 56 41 55 41 RIP [<ffffffff8132414a>] skb_put+0x7c/0x86 RSP <ffff8800bed83d60> ---[ end trace a721715cd86be064 ]--- ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 3.4.0-rc2: skb_put() -> skb_over_panic 2012-04-17 20:15 ` Alexander Beregalov @ 2012-04-17 20:45 ` Eric Dumazet 2012-04-17 20:47 ` David Miller 2012-04-18 6:37 ` Alexander Beregalov 0 siblings, 2 replies; 11+ messages in thread From: Eric Dumazet @ 2012-04-17 20:45 UTC (permalink / raw) To: Alexander Beregalov; +Cc: netdev, Linux Kernel Mailing List On Wed, 2012-04-18 at 00:15 +0400, Alexander Beregalov wrote: > On 15 April 2012 19:16, Eric Dumazet <eric.dumazet@gmail.com> wrote: > > On Sun, 2012-04-15 at 16:24 +0400, Alexander Beregalov wrote: > >> Hi > >> > >> kernel 3.4.0-rc2-00333-g668ce0a > >> > >> calltrace is lost on netconsole, sorry. > >> > >> ethernet is realtek/r8169 > >> > >> It happened already two times with rtorrent, perhaps I can reproduce > >> it, what else can I provide to you? > >> > > full stack trace needed please. > > > > This time calltrace is different (I saw few last lines of calltrace on > a display, but not enough) and netconsole transmitted complete > message, but perhaps it is the same problem. At least 'end' is the > same. > > > skb_over_panic: text:ffffffff8136d919 len:1248 put:932 > head:ffff8800babcf800 data:ffff8800babcfd10 tail:0x9f0 end:0x6c0 > dev:<NULL> > ------------[ cut here ]------------ > kernel BUG at net/core/skbuff.c:127! > invalid opcode: 0000 [#1] SMP > CPU 3 > Modules linked in: > > Pid: 1926, comm: rtorrent Not tainted 3.4.0-rc2-00333-g668ce0a #1 > /D525MW > RIP: 0010:[<ffffffff8132414a>] [<ffffffff8132414a>] skb_put+0x7c/0x86 > RSP: 0018:ffff8800bed83d60 EFLAGS: 00010246 > RAX: 0000000000000098 RBX: ffff8800b0b244c0 RCX: 000000000000003d > RDX: 000000000000000d RSI: 0000000000000046 RDI: ffffffff8162a0b0 > RBP: ffff8800bed83d80 R08: 0000000000000001 R09: 0000000000000000 > R10: ffff88002ed840c0 R11: 00000000007e28f6 R12: ffff8800b9cb9180 > R13: ffff8800b9cb96c0 R14: ffff8800b9cb91a8 R15: ffff8800b0b245a0 > FS: 00007fe639e75720(0000) GS:ffff8800bed80000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00007fe63571a000 CR3: 00000000b53fa000 CR4: 00000000000007e0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process rtorrent (pid: 1926, threadinfo ffff8800bb0a0000, task ffff8800bb1dddc0) > Stack: > 00000000000009f0 00000000000006c0 ffffffff814d06e7 ffff8800b9cb9180 > ffff8800bed83de0 ffffffff8136d919 ffff8800000003a4 0000013c81536640 > ffff8800b0b245a0 000000cc00000006 ffff8800b0b244c0 ffff8800b0b244c0 > Call Trace: > <IRQ> > [<ffffffff8136d919>] tcp_retransmit_skb+0x29a/0x529 > [<ffffffff8136eebd>] tcp_retransmit_timer+0x358/0x4e7 > [<ffffffff8136f0e8>] tcp_write_timer+0x9c/0x17d > [<ffffffff81035bd0>] run_timer_softirq+0x1eb/0x2bf > [<ffffffff8136f04c>] ? tcp_retransmit_timer+0x4e7/0x4e7 > [<ffffffff810184e9>] ? native_smp_send_reschedule+0x4f/0x51 > [<ffffffff8102fb52>] __do_softirq+0xbf/0x17d > [<ffffffff810188ec>] ? lapic_next_event+0x18/0x1c > [<ffffffff813c05cc>] call_softirq+0x1c/0x30 > [<ffffffff810033d6>] do_softirq+0x33/0x69 > [<ffffffff8102fdc6>] irq_exit+0x44/0x9c > [<ffffffff81018c6f>] smp_apic_timer_interrupt+0x86/0x94 > [<ffffffff813bfe47>] apic_timer_interrupt+0x67/0x70 > <EOI> > [<ffffffff8131d87e>] ? sock_sendmsg+0xe6/0x106 > [<ffffffff8135fb7b>] ? tcp_poll+0xaf/0x168 > [<ffffffff810ff814>] ? ep_send_events_proc+0x67/0x116 > [<ffffffff8131b8ef>] sock_poll+0x15/0x17 > [<ffffffff810ff823>] ep_send_events_proc+0x76/0x116 > [<ffffffff810ff7ad>] ? ep_read_events_proc+0x99/0x99 > [<ffffffff810fff24>] ep_scan_ready_list.clone.6+0x8f/0x16f > [<ffffffff81100277>] ep_poll+0x25f/0x2e2 > [<ffffffff8131e886>] ? sys_accept4+0x133/0x15f > [<ffffffff81100d8e>] sys_epoll_wait+0x90/0xae > [<ffffffff813bf2e6>] system_call_fastpath+0x1a/0x1f > Code: 8b 57 60 48 89 44 24 10 8b 87 ac 00 00 00 48 89 44 24 08 31 c0 > 8b bf a8 00 00 00 48 89 3c 24 48 c7 c7 43 07 4d 81 e8 2c 27 09 00 <0f> > 0b 89 c0 49 8d 04 00 c9 c3 55 48 89 e5 41 57 41 56 41 55 41 > RIP [<ffffffff8132414a>] skb_put+0x7c/0x86 > RSP <ffff8800bed83d60> > ---[ end trace a721715cd86be064 ]--- Thanks a lot, I belive I know where the problem is. Could you check if commit a21d45726acacc963d8baddf74607d9b74e2b723 (tcp: avoid order-1 allocations on wifi and tx path) was in your tree ? ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 3.4.0-rc2: skb_put() -> skb_over_panic 2012-04-17 20:45 ` Eric Dumazet @ 2012-04-17 20:47 ` David Miller 2012-04-17 20:53 ` Eric Dumazet 2012-04-18 6:37 ` Alexander Beregalov 1 sibling, 1 reply; 11+ messages in thread From: David Miller @ 2012-04-17 20:47 UTC (permalink / raw) To: eric.dumazet; +Cc: a.beregalov, netdev, linux-kernel From: Eric Dumazet <eric.dumazet@gmail.com> Date: Tue, 17 Apr 2012 22:45:06 +0200 > Could you check if commit a21d45726acacc963d8baddf74607d9b74e2b723 > (tcp: avoid order-1 allocations on wifi and tx path) > was in your tree ? I was about to say that I think this is the guilty commit too. Good thing I held off the -stable submission of that change for a bit :-) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 3.4.0-rc2: skb_put() -> skb_over_panic 2012-04-17 20:47 ` David Miller @ 2012-04-17 20:53 ` Eric Dumazet 0 siblings, 0 replies; 11+ messages in thread From: Eric Dumazet @ 2012-04-17 20:53 UTC (permalink / raw) To: David Miller; +Cc: a.beregalov, netdev, linux-kernel On Tue, 2012-04-17 at 16:47 -0400, David Miller wrote: > From: Eric Dumazet <eric.dumazet@gmail.com> > Date: Tue, 17 Apr 2012 22:45:06 +0200 > > > Could you check if commit a21d45726acacc963d8baddf74607d9b74e2b723 > > (tcp: avoid order-1 allocations on wifi and tx path) > > was in your tree ? > > I was about to say that I think this is the guilty commit too. > > Good thing I held off the -stable submission of that change > for a bit :-) Fix should be easy I think, but yes you can hold stable submission of course. diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 376b2cf..7ac6423 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -1096,6 +1096,7 @@ static void __pskb_trim_head(struct sk_buff *skb, int len) eat = min_t(int, len, skb_headlen(skb)); if (eat) { __skb_pull(skb, eat); + skb->avail_size -= eat; len -= eat; if (!len) return; ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: 3.4.0-rc2: skb_put() -> skb_over_panic 2012-04-17 20:45 ` Eric Dumazet 2012-04-17 20:47 ` David Miller @ 2012-04-18 6:37 ` Alexander Beregalov 2012-04-18 7:29 ` Eric Dumazet 1 sibling, 1 reply; 11+ messages in thread From: Alexander Beregalov @ 2012-04-18 6:37 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev, Linux Kernel Mailing List On 18 April 2012 00:45, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Wed, 2012-04-18 at 00:15 +0400, Alexander Beregalov wrote: >> On 15 April 2012 19:16, Eric Dumazet <eric.dumazet@gmail.com> wrote: >> > On Sun, 2012-04-15 at 16:24 +0400, Alexander Beregalov wrote: >> >> Hi >> >> >> >> kernel 3.4.0-rc2-00333-g668ce0a >> >> >> >> calltrace is lost on netconsole, sorry. >> >> >> >> ethernet is realtek/r8169 >> >> >> >> It happened already two times with rtorrent, perhaps I can reproduce >> >> it, what else can I provide to you? >> >> >> > full stack trace needed please. >> > >> >> This time calltrace is different (I saw few last lines of calltrace on >> a display, but not enough) and netconsole transmitted complete >> message, but perhaps it is the same problem. At least 'end' is the >> same. >> >> >> skb_over_panic: text:ffffffff8136d919 len:1248 put:932 >> head:ffff8800babcf800 data:ffff8800babcfd10 tail:0x9f0 end:0x6c0 >> dev:<NULL> >> ------------[ cut here ]------------ >> kernel BUG at net/core/skbuff.c:127! >> invalid opcode: 0000 [#1] SMP >> CPU 3 >> Modules linked in: >> >> Pid: 1926, comm: rtorrent Not tainted 3.4.0-rc2-00333-g668ce0a #1 >> /D525MW >> RIP: 0010:[<ffffffff8132414a>] [<ffffffff8132414a>] skb_put+0x7c/0x86 >> RSP: 0018:ffff8800bed83d60 EFLAGS: 00010246 >> RAX: 0000000000000098 RBX: ffff8800b0b244c0 RCX: 000000000000003d >> RDX: 000000000000000d RSI: 0000000000000046 RDI: ffffffff8162a0b0 >> RBP: ffff8800bed83d80 R08: 0000000000000001 R09: 0000000000000000 >> R10: ffff88002ed840c0 R11: 00000000007e28f6 R12: ffff8800b9cb9180 >> R13: ffff8800b9cb96c0 R14: ffff8800b9cb91a8 R15: ffff8800b0b245a0 >> FS: 00007fe639e75720(0000) GS:ffff8800bed80000(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> CR2: 00007fe63571a000 CR3: 00000000b53fa000 CR4: 00000000000007e0 >> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >> Process rtorrent (pid: 1926, threadinfo ffff8800bb0a0000, task ffff8800bb1dddc0) >> Stack: >> 00000000000009f0 00000000000006c0 ffffffff814d06e7 ffff8800b9cb9180 >> ffff8800bed83de0 ffffffff8136d919 ffff8800000003a4 0000013c81536640 >> ffff8800b0b245a0 000000cc00000006 ffff8800b0b244c0 ffff8800b0b244c0 >> Call Trace: >> <IRQ> >> [<ffffffff8136d919>] tcp_retransmit_skb+0x29a/0x529 >> [<ffffffff8136eebd>] tcp_retransmit_timer+0x358/0x4e7 >> [<ffffffff8136f0e8>] tcp_write_timer+0x9c/0x17d >> [<ffffffff81035bd0>] run_timer_softirq+0x1eb/0x2bf >> [<ffffffff8136f04c>] ? tcp_retransmit_timer+0x4e7/0x4e7 >> [<ffffffff810184e9>] ? native_smp_send_reschedule+0x4f/0x51 >> [<ffffffff8102fb52>] __do_softirq+0xbf/0x17d >> [<ffffffff810188ec>] ? lapic_next_event+0x18/0x1c >> [<ffffffff813c05cc>] call_softirq+0x1c/0x30 >> [<ffffffff810033d6>] do_softirq+0x33/0x69 >> [<ffffffff8102fdc6>] irq_exit+0x44/0x9c >> [<ffffffff81018c6f>] smp_apic_timer_interrupt+0x86/0x94 >> [<ffffffff813bfe47>] apic_timer_interrupt+0x67/0x70 >> <EOI> >> [<ffffffff8131d87e>] ? sock_sendmsg+0xe6/0x106 >> [<ffffffff8135fb7b>] ? tcp_poll+0xaf/0x168 >> [<ffffffff810ff814>] ? ep_send_events_proc+0x67/0x116 >> [<ffffffff8131b8ef>] sock_poll+0x15/0x17 >> [<ffffffff810ff823>] ep_send_events_proc+0x76/0x116 >> [<ffffffff810ff7ad>] ? ep_read_events_proc+0x99/0x99 >> [<ffffffff810fff24>] ep_scan_ready_list.clone.6+0x8f/0x16f >> [<ffffffff81100277>] ep_poll+0x25f/0x2e2 >> [<ffffffff8131e886>] ? sys_accept4+0x133/0x15f >> [<ffffffff81100d8e>] sys_epoll_wait+0x90/0xae >> [<ffffffff813bf2e6>] system_call_fastpath+0x1a/0x1f >> Code: 8b 57 60 48 89 44 24 10 8b 87 ac 00 00 00 48 89 44 24 08 31 c0 >> 8b bf a8 00 00 00 48 89 3c 24 48 c7 c7 43 07 4d 81 e8 2c 27 09 00 <0f> >> 0b 89 c0 49 8d 04 00 c9 c3 55 48 89 e5 41 57 41 56 41 55 41 >> RIP [<ffffffff8132414a>] skb_put+0x7c/0x86 >> RSP <ffff8800bed83d60> >> ---[ end trace a721715cd86be064 ]--- > > Thanks a lot, I belive I know where the problem is. > > Could you check if commit a21d45726acacc963d8baddf74607d9b74e2b723 > (tcp: avoid order-1 allocations on wifi and tx path) > was in your tree ? > It was, git show a21d4572 v3.4-rc2..668ce0a shows it ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: 3.4.0-rc2: skb_put() -> skb_over_panic 2012-04-18 6:37 ` Alexander Beregalov @ 2012-04-18 7:29 ` Eric Dumazet 2012-04-18 7:54 ` Alexander Beregalov 2012-04-18 20:14 ` [PATCH] tcp: fix retransmit of partially acked frames Eric Dumazet 0 siblings, 2 replies; 11+ messages in thread From: Eric Dumazet @ 2012-04-18 7:29 UTC (permalink / raw) To: Alexander Beregalov; +Cc: netdev, Linux Kernel Mailing List On Wed, 2012-04-18 at 10:37 +0400, Alexander Beregalov wrote: > On 18 April 2012 00:45, Eric Dumazet <eric.dumazet@gmail.com> wrote: > > On Wed, 2012-04-18 at 00:15 +0400, Alexander Beregalov wrote: > >> On 15 April 2012 19:16, Eric Dumazet <eric.dumazet@gmail.com> wrote: > >> > On Sun, 2012-04-15 at 16:24 +0400, Alexander Beregalov wrote: > >> >> Hi > >> >> > >> >> kernel 3.4.0-rc2-00333-g668ce0a > >> >> > >> >> calltrace is lost on netconsole, sorry. > >> >> > >> >> ethernet is realtek/r8169 > >> >> > >> >> It happened already two times with rtorrent, perhaps I can reproduce > >> >> it, what else can I provide to you? > >> >> > >> > full stack trace needed please. > >> > > >> > >> This time calltrace is different (I saw few last lines of calltrace on > >> a display, but not enough) and netconsole transmitted complete > >> message, but perhaps it is the same problem. At least 'end' is the > >> same. > >> > >> > >> skb_over_panic: text:ffffffff8136d919 len:1248 put:932 > >> head:ffff8800babcf800 data:ffff8800babcfd10 tail:0x9f0 end:0x6c0 > >> dev:<NULL> > >> ------------[ cut here ]------------ > >> kernel BUG at net/core/skbuff.c:127! > >> invalid opcode: 0000 [#1] SMP > >> CPU 3 > >> Modules linked in: > >> > >> Pid: 1926, comm: rtorrent Not tainted 3.4.0-rc2-00333-g668ce0a #1 > >> /D525MW > >> RIP: 0010:[<ffffffff8132414a>] [<ffffffff8132414a>] skb_put+0x7c/0x86 > >> RSP: 0018:ffff8800bed83d60 EFLAGS: 00010246 > >> RAX: 0000000000000098 RBX: ffff8800b0b244c0 RCX: 000000000000003d > >> RDX: 000000000000000d RSI: 0000000000000046 RDI: ffffffff8162a0b0 > >> RBP: ffff8800bed83d80 R08: 0000000000000001 R09: 0000000000000000 > >> R10: ffff88002ed840c0 R11: 00000000007e28f6 R12: ffff8800b9cb9180 > >> R13: ffff8800b9cb96c0 R14: ffff8800b9cb91a8 R15: ffff8800b0b245a0 > >> FS: 00007fe639e75720(0000) GS:ffff8800bed80000(0000) knlGS:0000000000000000 > >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > >> CR2: 00007fe63571a000 CR3: 00000000b53fa000 CR4: 00000000000007e0 > >> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > >> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > >> Process rtorrent (pid: 1926, threadinfo ffff8800bb0a0000, task ffff8800bb1dddc0) > >> Stack: > >> 00000000000009f0 00000000000006c0 ffffffff814d06e7 ffff8800b9cb9180 > >> ffff8800bed83de0 ffffffff8136d919 ffff8800000003a4 0000013c81536640 > >> ffff8800b0b245a0 000000cc00000006 ffff8800b0b244c0 ffff8800b0b244c0 > >> Call Trace: > >> <IRQ> > >> [<ffffffff8136d919>] tcp_retransmit_skb+0x29a/0x529 > >> [<ffffffff8136eebd>] tcp_retransmit_timer+0x358/0x4e7 > >> [<ffffffff8136f0e8>] tcp_write_timer+0x9c/0x17d > >> [<ffffffff81035bd0>] run_timer_softirq+0x1eb/0x2bf > >> [<ffffffff8136f04c>] ? tcp_retransmit_timer+0x4e7/0x4e7 > >> [<ffffffff810184e9>] ? native_smp_send_reschedule+0x4f/0x51 > >> [<ffffffff8102fb52>] __do_softirq+0xbf/0x17d > >> [<ffffffff810188ec>] ? lapic_next_event+0x18/0x1c > >> [<ffffffff813c05cc>] call_softirq+0x1c/0x30 > >> [<ffffffff810033d6>] do_softirq+0x33/0x69 > >> [<ffffffff8102fdc6>] irq_exit+0x44/0x9c > >> [<ffffffff81018c6f>] smp_apic_timer_interrupt+0x86/0x94 > >> [<ffffffff813bfe47>] apic_timer_interrupt+0x67/0x70 > >> <EOI> > >> [<ffffffff8131d87e>] ? sock_sendmsg+0xe6/0x106 > >> [<ffffffff8135fb7b>] ? tcp_poll+0xaf/0x168 > >> [<ffffffff810ff814>] ? ep_send_events_proc+0x67/0x116 > >> [<ffffffff8131b8ef>] sock_poll+0x15/0x17 > >> [<ffffffff810ff823>] ep_send_events_proc+0x76/0x116 > >> [<ffffffff810ff7ad>] ? ep_read_events_proc+0x99/0x99 > >> [<ffffffff810fff24>] ep_scan_ready_list.clone.6+0x8f/0x16f > >> [<ffffffff81100277>] ep_poll+0x25f/0x2e2 > >> [<ffffffff8131e886>] ? sys_accept4+0x133/0x15f > >> [<ffffffff81100d8e>] sys_epoll_wait+0x90/0xae > >> [<ffffffff813bf2e6>] system_call_fastpath+0x1a/0x1f > >> Code: 8b 57 60 48 89 44 24 10 8b 87 ac 00 00 00 48 89 44 24 08 31 c0 > >> 8b bf a8 00 00 00 48 89 3c 24 48 c7 c7 43 07 4d 81 e8 2c 27 09 00 <0f> > >> 0b 89 c0 49 8d 04 00 c9 c3 55 48 89 e5 41 57 41 56 41 55 41 > >> RIP [<ffffffff8132414a>] skb_put+0x7c/0x86 > >> RSP <ffff8800bed83d60> > >> ---[ end trace a721715cd86be064 ]--- > > > > Thanks a lot, I belive I know where the problem is. > > > > Could you check if commit a21d45726acacc963d8baddf74607d9b74e2b723 > > (tcp: avoid order-1 allocations on wifi and tx path) > > was in your tree ? > > > > It was, > git show a21d4572 v3.4-rc2..668ce0a > shows it Thansk for the confirmation. Had you see the patch I sent some hours ago, and can you test it ? If not, I probably can reproduce the problem in my lab. diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 376b2cf..7ac6423 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -1096,6 +1096,7 @@ static void __pskb_trim_head(struct sk_buff *skb, int len) eat = min_t(int, len, skb_headlen(skb)); if (eat) { __skb_pull(skb, eat); + skb->avail_size -= eat; len -= eat; if (!len) return; ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: 3.4.0-rc2: skb_put() -> skb_over_panic 2012-04-18 7:29 ` Eric Dumazet @ 2012-04-18 7:54 ` Alexander Beregalov 2012-04-18 20:14 ` [PATCH] tcp: fix retransmit of partially acked frames Eric Dumazet 1 sibling, 0 replies; 11+ messages in thread From: Alexander Beregalov @ 2012-04-18 7:54 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev, Linux Kernel Mailing List >> > Thanks a lot, I belive I know where the problem is. >> > >> > Could you check if commit a21d45726acacc963d8baddf74607d9b74e2b723 >> > (tcp: avoid order-1 allocations on wifi and tx path) >> > was in your tree ? >> > >> >> It was, >> git show a21d4572 v3.4-rc2..668ce0a >> shows it > > Thansk for the confirmation. > > Had you see the patch I sent some hours ago, and can you test it ? > > If not, I probably can reproduce the problem in my lab. > > diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c > index 376b2cf..7ac6423 100644 > --- a/net/ipv4/tcp_output.c > +++ b/net/ipv4/tcp_output.c > @@ -1096,6 +1096,7 @@ static void __pskb_trim_head(struct sk_buff *skb, int len) > eat = min_t(int, len, skb_headlen(skb)); > if (eat) { > __skb_pull(skb, eat); > + skb->avail_size -= eat; > len -= eat; > if (!len) > return; > > Yes, I am testing it. ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH] tcp: fix retransmit of partially acked frames 2012-04-18 7:29 ` Eric Dumazet 2012-04-18 7:54 ` Alexander Beregalov @ 2012-04-18 20:14 ` Eric Dumazet 2012-04-18 20:54 ` David Miller 1 sibling, 1 reply; 11+ messages in thread From: Eric Dumazet @ 2012-04-18 20:14 UTC (permalink / raw) To: Alexander Beregalov, David Miller Cc: netdev, Linux Kernel Mailing List, Marc MERLIN From: Eric Dumazet <edumazet@google.com> Alexander Beregalov reported skb_over_panic errors and provided stack trace. I occurs commit a21d45726aca (tcp: avoid order-1 allocations on wifi and tx path) added a regression, when a retransmit is done after a partial ACK. tcp_retransmit_skb() tries to aggregate several frames if the first one has enough available room to hold the following ones payload. This is controlled by /proc/sys/net/ipv4/tcp_retrans_collapse tunable (default : enabled) Problem is we must make sure _pskb_trim_head() doesnt fool skb_availroom() when pulling some bytes from skb (this pull is done when receiver ACK part of the frame). Reported-by: Alexander Beregalov <a.beregalov@gmail.com> Cc: Marc MERLIN <marc@merlins.org> Signed-off-by: Eric Dumazet <edumazet@google.com> --- net/ipv4/tcp_output.c | 1 + 1 file changed, 1 insertion(+) diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 376b2cf..7ac6423 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -1096,6 +1096,7 @@ static void __pskb_trim_head(struct sk_buff *skb, int len) eat = min_t(int, len, skb_headlen(skb)); if (eat) { __skb_pull(skb, eat); + skb->avail_size -= eat; len -= eat; if (!len) return; ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH] tcp: fix retransmit of partially acked frames 2012-04-18 20:14 ` [PATCH] tcp: fix retransmit of partially acked frames Eric Dumazet @ 2012-04-18 20:54 ` David Miller 0 siblings, 0 replies; 11+ messages in thread From: David Miller @ 2012-04-18 20:54 UTC (permalink / raw) To: eric.dumazet; +Cc: a.beregalov, netdev, linux-kernel, marc From: Eric Dumazet <eric.dumazet@gmail.com> Date: Wed, 18 Apr 2012 22:14:23 +0200 > From: Eric Dumazet <edumazet@google.com> > > Alexander Beregalov reported skb_over_panic errors and provided stack > trace. > > I occurs commit a21d45726aca (tcp: avoid order-1 allocations on wifi and > tx path) added a regression, when a retransmit is done after a partial > ACK. > > tcp_retransmit_skb() tries to aggregate several frames if the first one > has enough available room to hold the following ones payload. This is > controlled by /proc/sys/net/ipv4/tcp_retrans_collapse tunable (default : > enabled) > > Problem is we must make sure _pskb_trim_head() doesnt fool > skb_availroom() when pulling some bytes from skb (this pull is done when > receiver ACK part of the frame). > > Reported-by: Alexander Beregalov <a.beregalov@gmail.com> > Cc: Marc MERLIN <marc@merlins.org> > Signed-off-by: Eric Dumazet <edumazet@google.com> Applied, thanks. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-04-18 20:54 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-04-15 12:24 3.4.0-rc2: skb_put() -> skb_over_panic Alexander Beregalov 2012-04-15 15:16 ` Eric Dumazet 2012-04-17 20:15 ` Alexander Beregalov 2012-04-17 20:45 ` Eric Dumazet 2012-04-17 20:47 ` David Miller 2012-04-17 20:53 ` Eric Dumazet 2012-04-18 6:37 ` Alexander Beregalov 2012-04-18 7:29 ` Eric Dumazet 2012-04-18 7:54 ` Alexander Beregalov 2012-04-18 20:14 ` [PATCH] tcp: fix retransmit of partially acked frames Eric Dumazet 2012-04-18 20:54 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).