WARNING in __ieee80211_beacon_get

WARNING in __ieee80211_beacon_get

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    456afe01 mptcp: ADD_ADDRs with echo bit are smaller
git tree:       net-next
console output: https://syzkaller.appspot.com/x/log.txt?x=16047c57900000
kernel config:  https://syzkaller.appspot.com/x/.config?x=1e6c5266df853ae
dashboard link: https://syzkaller.appspot.com/bug?extid=18c783c5cf6a781e3e2c
compiler:       gcc (GCC) 10.1.0-syz 20200507
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=11a68fdf900000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=12ef31eb900000

The issue was bisected to:

commit 7dfd8ac327301f302b03072066c66eb32578e940
Author: Ramon Fontes <ramonreisfontes@gmail.com>
Date:   Thu Oct 10 18:13:07 2019 +0000

    mac80211_hwsim: add support for OCB

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=13c463a3900000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=102463a3900000
console output: https://syzkaller.appspot.com/x/log.txt?x=17c463a3900000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+18c783c5cf6a781e3e2c@syzkaller.appspotmail.com
Fixes: 7dfd8ac32730 ("mac80211_hwsim: add support for OCB")

------------[ cut here ]------------
WARNING: CPU: 1 PID: 6900 at net/mac80211/tx.c:4875 __ieee80211_beacon_get+0xb59/0x1aa0 net/mac80211/tx.c:4875
Kernel panic - not syncing: panic_on_warn set ...
CPU: 1 PID: 6900 Comm: syz-executor345 Not tainted 5.9.0-rc6-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x198/0x1fd lib/dump_stack.c:118
 panic+0x382/0x7fb kernel/panic.c:231
 __warn.cold+0x20/0x4b kernel/panic.c:600
 report_bug+0x1bd/0x210 lib/bug.c:198
 handle_bug+0x38/0x90 arch/x86/kernel/traps.c:234
 exc_invalid_op+0x14/0x40 arch/x86/kernel/traps.c:254
 asm_exc_invalid_op+0x12/0x20 arch/x86/include/asm/idtentry.h:536
RIP: 0010:__ieee80211_beacon_get+0xb59/0x1aa0 net/mac80211/tx.c:4875
Code: b8 00 00 00 00 00 fc ff df 48 c1 ea 03 0f b6 04 02 84 c0 74 08 3c 03 0f 8e fe 0c 00 00 41 83 4c 24 28 1a eb 0a e8 a7 15 9b f9 <0f> 0b 45 31 e4 e8 9d 15 9b f9 e8 e8 3e 5b 00 31 ff 89 c3 89 c6 e8
RSP: 0018:ffffc90000da8b40 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8880a96b5e18 RCX: ffffffff87db68e5
RDX: ffff888091824540 RSI: ffffffff87db7209 RDI: 0000000000000005
RBP: 000000000000000b R08: 0000000000000001 R09: ffffc90000da8c88
R10: 0000000000000007 R11: 0000000000000000 R12: 0000000000000000
R13: ffff8880872c0c80 R14: 0000000000000000 R15: ffffc90000da8c88
 ieee80211_beacon_get_tim+0x88/0x910 net/mac80211/tx.c:4939
 ieee80211_beacon_get include/net/mac80211.h:4909 [inline]
 mac80211_hwsim_beacon_tx+0x111/0x910 drivers/net/wireless/mac80211_hwsim.c:1729
 __iterate_interfaces+0x1e5/0x520 net/mac80211/util.c:792
 ieee80211_iterate_active_interfaces_atomic+0x8d/0x170 net/mac80211/util.c:828
 mac80211_hwsim_beacon+0xd5/0x1a0 drivers/net/wireless/mac80211_hwsim.c:1782
 __run_hrtimer kernel/time/hrtimer.c:1524 [inline]
 __hrtimer_run_queues+0x6a9/0xfc0 kernel/time/hrtimer.c:1588
 hrtimer_run_softirq+0x17b/0x360 kernel/time/hrtimer.c:1605
 __do_softirq+0x1f8/0xb23 kernel/softirq.c:298
 asm_call_on_stack+0xf/0x20 arch/x86/entry/entry_64.S:706
 </IRQ>
 __run_on_irqstack arch/x86/include/asm/irq_stack.h:22 [inline]
 run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:48 [inline]
 do_softirq_own_stack+0x9d/0xd0 arch/x86/kernel/irq_64.c:77
 invoke_softirq kernel/softirq.c:393 [inline]
 __irq_exit_rcu kernel/softirq.c:423 [inline]
 irq_exit_rcu+0x235/0x280 kernel/softirq.c:435
 sysvec_apic_timer_interrupt+0x51/0xf0 arch/x86/kernel/apic/apic.c:1091
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:581
RIP: 0010:__sanitizer_cov_trace_pc+0x30/0x60 kernel/kcov.c:197
Code: fe 01 00 65 8b 05 c0 76 8b 7e a9 00 01 ff 00 48 8b 34 24 74 0f f6 c4 01 74 35 8b 82 4c 14 00 00 85 c0 74 2b 8b 82 28 14 00 00 <83> f8 02 75 20 48 8b 8a 30 14 00 00 8b 92 2c 14 00 00 48 8b 01 48
RSP: 0018:ffffc900048072d8 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8880a96b4c00 RCX: ffffffff87dd0f8e
RDX: ffff888091824540 RSI: ffffffff87dd0f61 RDI: ffff8880a96b5558
RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8880872c296f
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff8880a7bbaa20 R14: dffffc0000000000 R15: 0000000000000000
 ieee80211_chanctx_radar_detect+0x1f1/0x3a0 net/mac80211/util.c:4266
 ieee80211_check_combinations+0x3b9/0x880 net/mac80211/util.c:4325
 ieee80211_check_concurrent_iface+0x45b/0x670 net/mac80211/iface.c:309
 ieee80211_runtime_change_iftype net/mac80211/iface.c:1672 [inline]
 ieee80211_if_change_type+0x288/0x620 net/mac80211/iface.c:1712
 ieee80211_change_iface+0x26/0x210 net/mac80211/cfg.c:157
 rdev_change_virtual_intf net/wireless/rdev-ops.h:69 [inline]
 cfg80211_change_iface+0x2ec/0xfe0 net/wireless/util.c:1032
 nl80211_set_interface+0x65c/0x8d0 net/wireless/nl80211.c:3789
 genl_family_rcv_msg_doit+0x228/0x320 net/netlink/genetlink.c:739
 genl_family_rcv_msg net/netlink/genetlink.c:783 [inline]
 genl_rcv_msg+0x328/0x580 net/netlink/genetlink.c:800
 netlink_rcv_skb+0x15a/0x430 net/netlink/af_netlink.c:2489
 genl_rcv+0x24/0x40 net/netlink/genetlink.c:811
 netlink_unicast_kernel net/netlink/af_netlink.c:1304 [inline]
 netlink_unicast+0x533/0x7d0 net/netlink/af_netlink.c:1330
 netlink_sendmsg+0x856/0xd90 net/netlink/af_netlink.c:1919
 sock_sendmsg_nosec net/socket.c:651 [inline]
 sock_sendmsg+0xcf/0x120 net/socket.c:671
 ____sys_sendmsg+0x6e8/0x810 net/socket.c:2353
 ___sys_sendmsg+0xf3/0x170 net/socket.c:2407
 __sys_sendmsg+0xe5/0x1b0 net/socket.c:2440
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x4423d9
Code: e8 ac 00 03 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b 07 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ffcf9989098 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004423d9
RDX: 0000000000000000 RSI: 0000000020000340 RDI: 0000000000000003
RBP: 000000306e616c77 R08: 0000002000000000 R09: 0000002000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000080bde
R13: 0000000000000000 R14: 000000000000000c R15: 0000000000000004
Kernel Offset: disabled
Rebooting in 86400 seconds..


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

Re: WARNING in __ieee80211_beacon_get

[Johannes Berg]

Hi,

Yeah, I know this is old ... still happening though, so I've been
looking at it a bit.

> The issue was bisected to:
> 
> commit 7dfd8ac327301f302b03072066c66eb32578e940
> Author: Ramon Fontes <ramonreisfontes@gmail.com>
> Date:   Thu Oct 10 18:13:07 2019 +0000
> 
>     mac80211_hwsim: add support for OCB

I'm not sure that make sense, FWIW. There isn't even any OCB? The syz
script just contains a channel switch command.

> WARNING: CPU: 1 PID: 6900 at net/mac80211/tx.c:4875 __ieee80211_beacon_get+0xb59/0x1aa0 net/mac80211/tx.c:4875

I also can't reproduce this though.

Is there a chance that somehow there's some scheduling problem and
workers aren't being scheduled, and then ieee80211_csa_finalize_work()
doesn't work between the last pre-CSA and first post-CSA beacon?

I _can_ reproduce this if I just make ieee80211_csa_finalize_work() do
nothing instead of doing what it should, hence the question.

johannes

Re: WARNING in __ieee80211_beacon_get

[Aleksandr Nogikh]

On Wed, Aug 16, 2023 at 10:09 AM Johannes Berg
<johannes@sipsolutions.net> wrote:
>
> Hi,
>
> Yeah, I know this is old ... still happening though, so I've been
> looking at it a bit.
>
> > The issue was bisected to:
> >
> > commit 7dfd8ac327301f302b03072066c66eb32578e940
> > Author: Ramon Fontes <ramonreisfontes@gmail.com>
> > Date:   Thu Oct 10 18:13:07 2019 +0000
> >
> >     mac80211_hwsim: add support for OCB
>
> I'm not sure that make sense, FWIW. There isn't even any OCB? The syz
> script just contains a channel switch command.
>
> > WARNING: CPU: 1 PID: 6900 at net/mac80211/tx.c:4875 __ieee80211_beacon_get+0xb59/0x1aa0 net/mac80211/tx.c:4875
>
> I also can't reproduce this though.

Did you try to run the kernel attached in the assets?
https://github.com/google/syzkaller/blob/master/docs/syzbot_assets.md#run-a-c-reproducer

I've just followed the instructions from there and the C repro did
crash the kernel in ~20 seconds:

[   56.809692][    C1] ------------[ cut here ]------------
[   56.810656][    C1] WARNING: CPU: 1 PID: 5358 at
net/mac80211/tx.c:5011 __ieee80211_beacon_get+0x1495/0x16e0

The disk image I used:
https://storage.googleapis.com/syzbot-assets/c18b40f6d56d/disk-cacc6e22.raw.xz
The C repro:
https://syzkaller.appspot.com/text?tag=ReproC&x=135c0c63a80000

>
> Is there a chance that somehow there's some scheduling problem and
> workers aren't being scheduled, and then ieee80211_csa_finalize_work()
> doesn't work between the last pre-CSA and first post-CSA beacon?
>
> I _can_ reproduce this if I just make ieee80211_csa_finalize_work() do
> nothing instead of doing what it should, hence the question.
>
> johannes
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/4d51d5ec9f1a86b099900934661e1bbdefa269c7.camel%40sipsolutions.net.

Re: WARNING in __ieee80211_beacon_get

[Johannes Berg]

On Wed, 2023-08-16 at 15:39 +0200, Aleksandr Nogikh wrote:
> On Wed, Aug 16, 2023 at 10:09 AM Johannes Berg
> <johannes@sipsolutions.net> wrote:
> > 
> > Hi,
> > 
> > Yeah, I know this is old ... still happening though, so I've been
> > looking at it a bit.
> > 
> > > The issue was bisected to:
> > > 
> > > commit 7dfd8ac327301f302b03072066c66eb32578e940
> > > Author: Ramon Fontes <ramonreisfontes@gmail.com>
> > > Date:   Thu Oct 10 18:13:07 2019 +0000
> > > 
> > >     mac80211_hwsim: add support for OCB
> > 
> > I'm not sure that make sense, FWIW. There isn't even any OCB? The syz
> > script just contains a channel switch command.
> > 
> > > WARNING: CPU: 1 PID: 6900 at net/mac80211/tx.c:4875 __ieee80211_beacon_get+0xb59/0x1aa0 net/mac80211/tx.c:4875
> > 
> > I also can't reproduce this though.
> 
> Did you try to run the kernel attached in the assets?
> https://github.com/google/syzkaller/blob/master/docs/syzbot_assets.md#run-a-c-reproducer
> 
> I've just followed the instructions from there and the C repro did
> crash the kernel in ~20 seconds:
> 
> [   56.809692][    C1] ------------[ cut here ]------------
> [   56.810656][    C1] WARNING: CPU: 1 PID: 5358 at
> net/mac80211/tx.c:5011 __ieee80211_beacon_get+0x1495/0x16e0
> 

To be fair, I didn't, I figured the reproducer was simple enough to just
have a go at it with my own test infra.

Is there an easy way to rebuild the kernel for it?

johannes

Re: WARNING in __ieee80211_beacon_get

[Aleksandr Nogikh]

On Wed, Aug 16, 2023 at 4:01 PM Johannes Berg <johannes@sipsolutions.net> wrote:
>
> On Wed, 2023-08-16 at 15:39 +0200, Aleksandr Nogikh wrote:
> > On Wed, Aug 16, 2023 at 10:09 AM Johannes Berg
> > <johannes@sipsolutions.net> wrote:
> > >
> > > Hi,
> > >
> > > Yeah, I know this is old ... still happening though, so I've been
> > > looking at it a bit.
> > >
> > > > The issue was bisected to:
> > > >
> > > > commit 7dfd8ac327301f302b03072066c66eb32578e940
> > > > Author: Ramon Fontes <ramonreisfontes@gmail.com>
> > > > Date:   Thu Oct 10 18:13:07 2019 +0000
> > > >
> > > >     mac80211_hwsim: add support for OCB
> > >
> > > I'm not sure that make sense, FWIW. There isn't even any OCB? The syz
> > > script just contains a channel switch command.
> > >
> > > > WARNING: CPU: 1 PID: 6900 at net/mac80211/tx.c:4875 __ieee80211_beacon_get+0xb59/0x1aa0 net/mac80211/tx.c:4875
> > >
> > > I also can't reproduce this though.
> >
> > Did you try to run the kernel attached in the assets?
> > https://github.com/google/syzkaller/blob/master/docs/syzbot_assets.md#run-a-c-reproducer
> >
> > I've just followed the instructions from there and the C repro did
> > crash the kernel in ~20 seconds:
> >
> > [   56.809692][    C1] ------------[ cut here ]------------
> > [   56.810656][    C1] WARNING: CPU: 1 PID: 5358 at
> > net/mac80211/tx.c:5011 __ieee80211_beacon_get+0x1495/0x16e0
> >
>
> To be fair, I didn't, I figured the reproducer was simple enough to just
> have a go at it with my own test infra.
>
> Is there an easy way to rebuild the kernel for it?

How does an easy way differ from a difficult one in this case? :)

I've just built the v6.5-rc6 kernel with the
https://syzkaller.appspot.com/text?tag=KernelConfig&x=3e670757e16affb
config and run the C repro mentioned above from root. It crashed the
kernel:

[   78.057333][    C0] ------------[ cut here ]------------
[   78.058289][    C0] WARNING: CPU: 0 PID: 5377 at
net/mac80211/tx.c:5011 __ieee80211_beacon_get+0x1495/0x16e0

-- 
Aleksandr

>
> johannes

Re: WARNING in __ieee80211_beacon_get

[Johannes Berg]

On Wed, 2023-08-16 at 21:56 +0200, Aleksandr Nogikh wrote:
> > 
> > Is there an easy way to rebuild the kernel for it?
> 
> How does an easy way differ from a difficult one in this case? :)
> 

:-)

I was mostly thinking about "how the hell do I get the kernel into the
qemu image" but I realized later I don't even need to since qemu has the
-kernel option to boot the kernel directly.

Then I had to leave for a while so I only got to try it now, and indeed
I can reproduce it with a kernel built/booted that way, so I can make
changes for debugging. In fact, not even sure what else I really need
from the rootfs at that point, but this is still an easy way to
reproduce it.

Sorry for the noise.

johannes

Re: WARNING in __ieee80211_beacon_get

[Johannes Berg]

On Wed, 2023-08-16 at 23:04 +0200, Johannes Berg wrote:
> Then I had to leave for a while so I only got to try it now, and indeed
> I can reproduce it with a kernel built/booted that way, so I can make
> changes for debugging.

But wow, this is complicated, even creating the same interface names in
different network namespaces ...

This is what I got from debug:

[   49.036423][    C0] [ffff88801a9c1d00] wlan1: __ieee80211_beacon_update_cntdwn:5010: counter in ffff88801cd6d800 now 1
[   49.040155][    C0] [ffff88801a9c1d00] wlan1: ieee80211_csa_finish:3589: queue csa_finalize_work

That's fine, what it should be - although I don't see why there are 4ms
between those two lines.

[   49.042665][    C1] [ffff88804ad0ba00] wlan1: __ieee80211_beacon_get:5415

unrelated wlan1 in a different network namespaces (the [pointer] is the
network namespace pointer). I'll skip these for the rest of the log.

[   49.082269][   T11] [ffff88801a9c1d00] wlan1: ieee80211_csa_finalize_work:3732
[   49.084809][   T11] [ffff88801a9c1d00] wlan1: ieee80211_csa_finalize_work:3740
[   49.086646][   T11] [ffff88801a9c1d00] wlan1: ieee80211_csa_finalize_work:3744
[   49.088336][   T11] [ffff88801a9c1d00] wlan1: ieee80211_csa_finalize:3717
[   49.089932][   T11] [ffff88801a9c1d00] wlan1: __ieee80211_csa_finalize:3651
[   49.091642][   T11] [ffff88801a9c1d00] wlan1: __ieee80211_csa_finalize:3670
[   49.093661][   T11] [ffff88801a9c1d00] wlan1: __ieee80211_csa_finalize:3679
[   49.097030][   T11] [ffff88801a9c1d00] wlan1: ieee80211_link_chanctx_reservation_complete:1211: queue csa_finalize_work

That continues running as it should, but ... it took forever! By now,
just to go through a few function calls, it took 57ms?

[   49.130990][   T11] [ffff88801a9c1d00] wlan1: ieee80211_csa_finalize_work:3732
[   49.132892][   T11] [ffff88801a9c1d00] wlan1: ieee80211_csa_finalize_work:3740

and another 33ms to actually start the worker again

[   49.137404][    C0] [ffff88801a9c1d00] wlan1: __ieee80211_beacon_get:5415
[   49.137462][    C1] [ffff88804ad09d00] wlan1: ieee80211_beacon_cntdwn_is_complete:5111
[   49.138897][    C0] [ffff88801a9c1d00] wlan1: __ieee80211_beacon_update_cntdwn:5010: counter in ffff88801cd6d800 now 0
[   49.139480][    C0] ------------[ cut here ]------------
[   49.142567][    C0] WARNING: CPU: 0 PID: 5215 at net/mac80211/tx.c:5013 __ieee80211_beacon_get+0x1604/0x1a10

And the worker doesn't finish fast enough, so we get to 0, warn and
crash.

So what I said before about scheduling still seems like it could be the
case.

I'm not sure what we could do here - we can't delay the beacon, so if
the update work didn't run ... and in general, I think we _do_ want this
reported to see that something is broken, just that maybe with the
single-core setup (two threads) and so much happening in the system, we
don't get this running fast enough?

Maybe we need to make our workqueue high priority? Testing ... no,
doesn't even help.

So not sure. Seems in reality this won't really happen since you have
usually 100ms or so to execute the thing, and only a single (or handful
maybe) interface(s).

johannes