next-20151207 - crash in IPv6 code

next-20151207 - crash in IPv6 code

[Valdis Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 3443 bytes --]

Seen this in 2 boots out of two on next-20151207 when IPV6 networking
was available.  It was stable when no net was available. Also, next-20161127 is OK.
Haven't bisected it yet - this ring any bells?

[   92.231022] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   92.231035] IP: [<ffffffffb48579cb>] nf_ct_frag6_gather+0x81b/0xba0
[   92.231046] PGD 0
[   92.231050] Oops: 0000 [#1] PREEMPT SMP

[   92.231166] Call Trace:
[   92.231170]  <IRQ>
[   92.231196]  [<ffffffffb4856e96>] ipv6_defrag+0x66/0x80
[   92.231206]  [<ffffffffb47547b2>] nf_iterate+0x62/0x80
[   92.231216]  [<ffffffffb475488a>] nf_hook_slow+0xba/0x1b0
[   92.231225]  [<ffffffffb47547d5>] ? nf_hook_slow+0x5/0x1b0
[   92.231235]  [<ffffffffb481444d>] ipv6_rcv+0x83d/0x8d0
[   92.231242]  [<ffffffffb4813c4e>] ? ipv6_rcv+0x3e/0x8d0
[   92.231251]  [<ffffffffb48139a0>] ? ip6_input_finish+0x7e0/0x7e0
[   92.231260]  [<ffffffffb47162ea>] __netif_receive_skb_core+0x60a/0xd70
[   92.231269]  [<ffffffffb4716a70>] __netif_receive_skb+0x20/0x90
[   92.231278]  [<ffffffffb4718c90>] netif_receive_skb_internal+0x70/0x1f0
[   92.231285]  [<ffffffffb4718c45>] ? netif_receive_skb_internal+0x25/0x1f0
[   92.231292]  [<ffffffffb474277b>] ? eth_type_trans+0x11b/0x200
[   92.231300]  [<ffffffffb4718e69>] netif_receive_skb+0x59/0x170
[   92.231308]  [<ffffffffb4949c00>] ieee80211_deliver_skb+0x120/0x180
[   92.231315]  [<ffffffffb494de52>] ieee80211_rx_handlers+0x2762/0x29f0
[   92.231324]  [<ffffffffb46fe300>] ? skb_queue_tail+0x20/0x50
[   92.231335]  [<ffffffffb40c5e78>] ? do_raw_spin_lock+0x148/0x1e0
[   92.231342]  [<ffffffffb40bfb46>] ? trace_hardirqs_on_caller+0x16/0x1b0
[   92.231358]  [<ffffffffb494e32e>] ieee80211_prepare_and_rx_handle+0x24e/0xa80
[   92.231365]  [<ffffffffb494ed9a>] ? ieee80211_rx_napi+0x23a/0xf00
[   92.231373]  [<ffffffffb494f097>] ieee80211_rx_napi+0x537/0xf00
[   92.231380]  [<ffffffffb494ed9a>] ? ieee80211_rx_napi+0x23a/0xf00
[   92.231391]  [<ffffffffb49118a5>] ieee80211_tasklet_handler+0xc5/0xd0
[   92.231401]  [<ffffffffb4066b85>] tasklet_action+0x1d5/0x220
[   92.231409]  [<ffffffffb40672cc>] __do_softirq+0xec/0x5a0
[   92.231417]  [<ffffffffb4067954>] irq_exit+0xd4/0xe0
[   92.231426]  [<ffffffffb49b3afa>] do_IRQ+0x6a/0x120
[   92.231434]  [<ffffffffb49b2089>] common_interrupt+0x89/0x89
[   92.231440]  <EOI>
[   92.231450]  [<ffffffffb465da3c>] ? cpuidle_enter_state+0x1ac/0x410
[   92.231458]  [<ffffffffb40bfced>] ? trace_hardirqs_on+0xd/0x10
[   92.231466]  [<ffffffffb465da47>] ? cpuidle_enter_state+0x1b7/0x410
[   92.231476]  [<ffffffffb465da3c>] ? cpuidle_enter_state+0x1ac/0x410
[   92.231485]  [<ffffffffb465dcd7>] cpuidle_enter+0x17/0x20
[   92.231494]  [<ffffffffb40b4e6d>] cpu_startup_entry+0x48d/0x520
[   92.231503]  [<ffffffffb403c874>] start_secondary+0x154/0x170
[   92.231510] Code: 8b fd ff ff 48 8b 13 48 89 10 49 8b 0e 49 39 ce 0f 84 80 01 00 00 48 8b 11 48 39 d3 0f 84 71 01 00 00 49 39 d6 0f 84 6b 01 00 00 <48
> 8b 0a 48 39 cb 0f 84 59 01 00 00 48 89 ca 49 39 d6 75 ec e9
[   92.231685] RIP  [<ffffffffb48579cb>] nf_ct_frag6_gather+0x81b/0xba0
[   92.231698]  RSP <ffff88022dd03958>
[   92.231704] CR2: 0000000000000000
[   92.231714] ---[ end trace 62089aaf8d90e56a ]---
[   94.678192] Kernel panic - not syncing: Fatal exception in interrupt
[   94.678228] Kernel Offset: 0x33000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)



[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: next-20151207 - crash in IPv6 code

[Florian Westphal]

Valdis Kletnieks <Valdis.Kletnieks@vt.edu> wrote:

[ CC Pablo ]

> Seen this in 2 boots out of two on next-20151207 when IPV6 networking
> was available.  It was stable when no net was available. Also, next-20161127 is OK.
> Haven't bisected it yet - this ring any bells?

Thanks for the report, my fault -- its caused by
029f7f3b8701cc7aca8bdb which is only in Pablos nf-next tree.

This should fix this bug (proper patch w. changelog coming
after more testing):

diff --git a/net/ipv6/netfilter/nf_conntrack_reasm.c b/net/ipv6/netfilter/nf_conntrack_reasm.c
--- a/net/ipv6/netfilter/nf_conntrack_reasm.c
+++ b/net/ipv6/netfilter/nf_conntrack_reasm.c
@@ -441,11 +441,14 @@ nf_ct_frag6_reasm(struct frag_queue *fq, struct sk_buff *prev,  struct net_devic
 			return false;
 
 		fp->next = prev->next;
-		skb_queue_walk(head, iter) {
-			if (iter->next != prev)
-				continue;
-			iter->next = fp;
-			break;
+
+		iter = head;
+		while (iter) {
+			if (iter->next == prev) {
+				iter->next = fp;
+				break;
+			}
+			iter = iter->next;
 		}
 
 		skb_morph(prev, head);

Re: next-20151207 - crash in IPv6 code

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

On Tue, 08 Dec 2015 12:34:09 +0100, Florian Westphal said:
> Valdis Kletnieks <Valdis.Kletnieks@vt.edu> wrote:
>
> [ CC Pablo ]
>
> > Seen this in 2 boots out of two on next-20151207 when IPV6 networking
> > was available.  It was stable when no net was available. Also, next-20161127 is OK.
> > Haven't bisected it yet - this ring any bells?
>
> Thanks for the report, my fault -- its caused by
> 029f7f3b8701cc7aca8bdb which is only in Pablos nf-next tree.
>
> This should fix this bug (proper patch w. changelog coming
> after more testing):
>
> diff --git a/net/ipv6/netfilter/nf_conntrack_reasm.c b/net/ipv6/netfilter/nf_conntrack_reasm.c

Pumped about 100M of IPv6 traffic through, and no problems.

Feel free to stick a Reported-by:/Tested-By: on this patch...

[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]

Re: next-20151207 - crash in IPv6 code

[David Miller]

From: Valdis.Kletnieks@vt.edu
Date: Tue, 08 Dec 2015 16:54:17 -0500

> On Tue, 08 Dec 2015 12:34:09 +0100, Florian Westphal said:
>> Valdis Kletnieks <Valdis.Kletnieks@vt.edu> wrote:
>>
>> [ CC Pablo ]
>>
>> > Seen this in 2 boots out of two on next-20151207 when IPV6 networking
>> > was available.  It was stable when no net was available. Also, next-20161127 is OK.
>> > Haven't bisected it yet - this ring any bells?
>>
>> Thanks for the report, my fault -- its caused by
>> 029f7f3b8701cc7aca8bdb which is only in Pablos nf-next tree.
>>
>> This should fix this bug (proper patch w. changelog coming
>> after more testing):
>>
>> diff --git a/net/ipv6/netfilter/nf_conntrack_reasm.c b/net/ipv6/netfilter/nf_conntrack_reasm.c
> 
> Pumped about 100M of IPv6 traffic through, and no problems.
> 
> Feel free to stick a Reported-by:/Tested-By: on this patch...

Thanks for testing.