BUG: using __this_cpu_read() in preemptible code in ip6_finish_output

BUG: using __this_cpu_read() in preemptible code in ip6_finish_output

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit:    e1427237 macsec: add noinline tag to avoid a frame size wa..
git tree:       net-next
console output: https://syzkaller.appspot.com/x/log.txt?x=15594553200000
kernel config:  https://syzkaller.appspot.com/x/.config?x=8e9bc94c16d346a6
dashboard link: https://syzkaller.appspot.com/bug?extid=51471b4aae195285a4a3
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=156ffb07200000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=14412673200000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+51471b4aae195285a4a3@syzkaller.appspotmail.com

BUG: using __this_cpu_read() in preemptible [00000000] code:  
syz-executor222/7596
caller is dev_recursion_level include/linux/netdevice.h:3052 [inline]
caller is ip6_skb_dst_mtu include/net/ip6_route.h:245 [inline]
caller is ip6_finish_output+0x335/0xdc0 net/ipv6/ip6_output.c:149
CPU: 1 PID: 7596 Comm: syz-executor222 Not tainted 5.1.0-rc2+ #118
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Call Trace:
  __dump_stack lib/dump_stack.c:77 [inline]
  dump_stack+0x172/0x1f0 lib/dump_stack.c:113
  __this_cpu_preempt_check+0x246/0x270 lib/smp_processor_id.c:47
  dev_recursion_level include/linux/netdevice.h:3052 [inline]
  ip6_skb_dst_mtu include/net/ip6_route.h:245 [inline]
  ip6_finish_output+0x335/0xdc0 net/ipv6/ip6_output.c:149
  NF_HOOK_COND include/linux/netfilter.h:278 [inline]
  ip6_output+0x235/0x7f0 net/ipv6/ip6_output.c:171
  dst_output include/net/dst.h:433 [inline]
  NF_HOOK include/linux/netfilter.h:289 [inline]
  NF_HOOK include/linux/netfilter.h:283 [inline]
  ip6_xmit+0xe41/0x20c0 net/ipv6/ip6_output.c:275
  inet6_csk_xmit+0x2fb/0x5d0 net/ipv6/inet6_connection_sock.c:139
  __tcp_transmit_skb+0x1a32/0x3750 net/ipv4/tcp_output.c:1155
  tcp_transmit_skb net/ipv4/tcp_output.c:1171 [inline]
  tcp_send_syn_data net/ipv4/tcp_output.c:3494 [inline]
  tcp_connect+0x1e47/0x4280 net/ipv4/tcp_output.c:3533
  tcp_v6_connect+0x150b/0x20a0 net/ipv6/tcp_ipv6.c:331
  __inet_stream_connect+0x83f/0xea0 net/ipv4/af_inet.c:659
  tcp_sendmsg_fastopen net/ipv4/tcp.c:1155 [inline]
  tcp_sendmsg_locked+0x231f/0x37f0 net/ipv4/tcp.c:1197
  tcp_sendmsg+0x30/0x50 net/ipv4/tcp.c:1431
  inet_sendmsg+0x147/0x5e0 net/ipv4/af_inet.c:802
  sock_sendmsg_nosec net/socket.c:651 [inline]
  sock_sendmsg+0xdd/0x130 net/socket.c:661
  __sys_sendto+0x262/0x380 net/socket.c:1932
  __do_sys_sendto net/socket.c:1944 [inline]
  __se_sys_sendto net/socket.c:1940 [inline]
  __x64_sys_sendto+0xe1/0x1a0 net/socket.c:1940
  do_syscall_64+0x103/0x610 arch/x86/entry/common.c:290
  entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x440189
Code: 18 89 d0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 f8 48 89 f7  
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff  
ff 0f 83 fb 13 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffed7abd1a8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
RAX: ffffffffffffffda RBX: 00000000004002c8 RCX: 0000000000440189
RDX: 0000000000000000 RSI: 0000000000000000 RD


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

Re: BUG: using __this_cpu_read() in preemptible code in ip6_finish_output

[syzbot]

syzbot has bisected this bug to:

commit 97cdcf37b57e3f204be3000b9eab9686f38b4356
Author: Florian Westphal <fw@strlen.de>
Date:   Mon Apr 1 14:42:13 2019 +0000

     net: place xmit recursion in softnet data

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=11c04b0f200000
start commit:   e1427237 macsec: add noinline tag to avoid a frame size wa..
git tree:       net-next
final crash:    https://syzkaller.appspot.com/x/report.txt?x=13c04b0f200000
console output: https://syzkaller.appspot.com/x/log.txt?x=15c04b0f200000
kernel config:  https://syzkaller.appspot.com/x/.config?x=8e9bc94c16d346a6
dashboard link: https://syzkaller.appspot.com/bug?extid=51471b4aae195285a4a3
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=156ffb07200000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=14412673200000

Reported-by: syzbot+51471b4aae195285a4a3@syzkaller.appspotmail.com
Fixes: 97cdcf37b57e ("net: place xmit recursion in softnet data")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

[PATCH net-next] net: use correct this_cpu primitive in dev_recursion_level

[Florian Westphal]

syzbot reports:
BUG: using __this_cpu_read() in preemptible code:
caller is dev_recursion_level include/linux/netdevice.h:3052 [inline]
 __this_cpu_preempt_check+0x246/0x270 lib/smp_processor_id.c:47
 dev_recursion_level include/linux/netdevice.h:3052 [inline]
 ip6_skb_dst_mtu include/net/ip6_route.h:245 [inline]

I erronously downgraded a this_cpu_read to __this_cpu_read when
moving dev_recursion_level() around.

Reported-by: syzbot+51471b4aae195285a4a3@syzkaller.appspotmail.com
Fixes: 97cdcf37b57e ("net: place xmit recursion in softnet data")
Signed-off-by: Florian Westphal <fw@strlen.de>
---
 include/linux/netdevice.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index eb9f05e0863d..521eb869555e 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -3049,7 +3049,7 @@ DECLARE_PER_CPU_ALIGNED(struct softnet_data, softnet_data);
 
 static inline int dev_recursion_level(void)
 {
-	return __this_cpu_read(softnet_data.xmit.recursion);
+	return this_cpu_read(softnet_data.xmit.recursion);
 }
 
 #define XMIT_RECURSION_LIMIT	10
-- 
2.21.0

Re: [PATCH net-next] net: use correct this_cpu primitive in dev_recursion_level

[Eric Dumazet]

On 04/02/2019 11:28 PM, Florian Westphal wrote:
> syzbot reports:
> BUG: using __this_cpu_read() in preemptible code:
> caller is dev_recursion_level include/linux/netdevice.h:3052 [inline]
>  __this_cpu_preempt_check+0x246/0x270 lib/smp_processor_id.c:47
>  dev_recursion_level include/linux/netdevice.h:3052 [inline]
>  ip6_skb_dst_mtu include/net/ip6_route.h:245 [inline]
> 
> I erronously downgraded a this_cpu_read to __this_cpu_read when
> moving dev_recursion_level() around.
> 

Reviewed-by: Eric Dumazet <edumazet@google.com>

Re: [PATCH net-next] net: use correct this_cpu primitive in dev_recursion_level

[David Miller]

From: Florian Westphal <fw@strlen.de>
Date: Wed,  3 Apr 2019 08:28:35 +0200

> syzbot reports:
> BUG: using __this_cpu_read() in preemptible code:
> caller is dev_recursion_level include/linux/netdevice.h:3052 [inline]
>  __this_cpu_preempt_check+0x246/0x270 lib/smp_processor_id.c:47
>  dev_recursion_level include/linux/netdevice.h:3052 [inline]
>  ip6_skb_dst_mtu include/net/ip6_route.h:245 [inline]
> 
> I erronously downgraded a this_cpu_read to __this_cpu_read when
> moving dev_recursion_level() around.
> 
> Reported-by: syzbot+51471b4aae195285a4a3@syzkaller.appspotmail.com
> Fixes: 97cdcf37b57e ("net: place xmit recursion in softnet data")
> Signed-off-by: Florian Westphal <fw@strlen.de>

Applied, thanks Florian.

Re: BUG: using __this_cpu_read() in preemptible code in ip6_finish_output

[Krzysztof Kozlowski]

On Wed, 3 Apr 2019 at 03:14, syzbot
<syzbot+51471b4aae195285a4a3@syzkaller.appspotmail.com> wrote:
>
> syzbot has bisected this bug to:
>
> commit 97cdcf37b57e3f204be3000b9eab9686f38b4356
> Author: Florian Westphal <fw@strlen.de>
> Date:   Mon Apr 1 14:42:13 2019 +0000
>
>      net: place xmit recursion in softnet data

I am seeing this as well on ARMv7 board booted from NFS root (exynos_defconfig):

[ 30.221238] BUG: using __this_cpu_read() in preemptible [00000000]
code: systemd-network/236
[ 30.228576] caller is ip6_output+0x68/0x3e8
[ 30.232578] CPU: 1 PID: 236 Comm: systemd-network Not tainted
5.1.0-rc3-next-20190405 #2
[ 30.240657] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
[ 30.246719] [<c011238c>] (unwind_backtrace) from [<c010df50>]
(show_stack+0x10/0x14)
[ 30.254447] [<c010df50>] (show_stack) from [<c0a87270>] (dump_stack+0x98/0xc4)
[ 30.261638] [<c0a87270>] (dump_stack) from [<c0494a94>]
(__this_cpu_preempt_check+0x124/0x128)
[ 30.270238] [<c0494a94>] (__this_cpu_preempt_check) from [<c08e486c>]
(ip6_output+0x68/0x3e8)
[ 30.278730] [<c08e486c>] (ip6_output) from [<c08e52ac>]
(ip6_send_skb+0x30/0x1d0)
[ 30.286180] [<c08e52ac>] (ip6_send_skb) from [<c0912798>]
(rawv6_sendmsg+0x824/0x9c0)
[ 30.294005] [<c0912798>] (rawv6_sendmsg) from [<c07d8ff8>]
(sock_sendmsg+0x14/0x24)
[ 30.301628] [<c07d8ff8>] (sock_sendmsg) from [<c07d97cc>]
(___sys_sendmsg+0x230/0x244)
[ 30.309514] [<c07d97cc>] (___sys_sendmsg) from [<c07da9f8>]
(__sys_sendmsg+0x50/0x8c)
[ 30.317316] [<c07da9f8>] (__sys_sendmsg) from [<c01011ac>]
(__sys_trace_return+0x0/0x14)
[ 30.325376] Exception stack(0xe79f9fa8 to 0xe79f9ff0)
[ 30.330377] 9fa0: 00000000 b6f3da58 0000000b beeb2a6c 00000000 00000000
[ 30.338560] 9fc0: 00000000 b6f3da58 004015ce 00000128 004b10c8
00000000 020d28f9 00000000
[ 30.346708] 9fe0: 004aeee0 beeb2a40 0047cbdc b6d27684

Full log:
https://krzk.eu/#/builders/22/builds/1055/steps/13/logs/serial0

Best regards,
Krzysztof Kozlowski

Re: BUG: using __this_cpu_read() in preemptible code in ip6_finish_output

[Florian Westphal]

Krzysztof Kozlowski <krzk@kernel.org> wrote:
> On Wed, 3 Apr 2019 at 03:14, syzbot
> <syzbot+51471b4aae195285a4a3@syzkaller.appspotmail.com> wrote:
> >
> > syzbot has bisected this bug to:
> >
> > commit 97cdcf37b57e3f204be3000b9eab9686f38b4356
> > Author: Florian Westphal <fw@strlen.de>
> > Date:   Mon Apr 1 14:42:13 2019 +0000
> >
> >      net: place xmit recursion in softnet data
> 
> I am seeing this as well on ARMv7 board booted from NFS root (exynos_defconfig):

Sorry about this, the fix is now in net-next though:
28b05b92886871bdd ("net: use correct this_cpu primitive in dev_recursion_level")

Re: BUG: using __this_cpu_read() in preemptible code in ip6_finish_output

[Dmitry Vyukov]

On Fri, Apr 5, 2019 at 12:09 PM Florian Westphal <fw@strlen.de> wrote:
>
> Krzysztof Kozlowski <krzk@kernel.org> wrote:
> > On Wed, 3 Apr 2019 at 03:14, syzbot
> > <syzbot+51471b4aae195285a4a3@syzkaller.appspotmail.com> wrote:
> > >
> > > syzbot has bisected this bug to:
> > >
> > > commit 97cdcf37b57e3f204be3000b9eab9686f38b4356
> > > Author: Florian Westphal <fw@strlen.de>
> > > Date:   Mon Apr 1 14:42:13 2019 +0000
> > >
> > >      net: place xmit recursion in softnet data
> >
> > I am seeing this as well on ARMv7 board booted from NFS root (exynos_defconfig):
>
> Sorry about this, the fix is now in net-next though:
> 28b05b92886871bdd ("net: use correct this_cpu primitive in dev_recursion_level")


Krzysztof, just in case, you can see the current bug status on
dashboard as well. E.g. for this one it says:

Fix commit: net: use correct this_cpu primitive in dev_recursion_level