[REGRESSION] wifi: brcmfmac: add firmware vendor info in driver info

[REGRESSION] wifi: brcmfmac: add firmware vendor info in driver info

[Felix]

[-- Attachment #1: Type: text/plain, Size: 1912 bytes --]

Changes made by the mentioned commit lead to Oops when waking up after
suspend to RAM.

When waking up a Dell XPS 13 9350 with BCM4350 wifi card after suspend
to RAM, the kernel hangs with a NULL pointer dereference and Oops. The
issues was introduced by commit
da6d9c8ecd00e20218461007948f2b0a8e7fa242 and only happens when the
brcmfmac module is loaded. This issue is the reason we could not
upgrade to newer kernels than the 6.1 series on the affected machine.


Details:

On a Dell XPS 13 9350 laptop with Broadcom BCM4350 wifi card (according
to lspci), every kernel starting with 6.2 hangs when waking up after
suspend to RAM. The issue persisted as of May 2nd (last tested commit
was 348551ddaf311c76b01cdcbaf61b6fef06a49144). Passing the debug and
no_console_suspend parameters to the kernel show that a NULL pointer
dereference and an Oops happen at wakeup. Please find the kernel
messages readable on the screen at this point attached
(kernel_log_after_suspend.txt).

A bisection was successful and produced the first bad commit
[da6d9c8ecd00e20218461007948f2b0a8e7fa242] wifi: brcmfmac: add firmware
vendor info in driver info
I've attached the bisection log as well (bisect.txt).

Suspend to RAM works on this machine on newer kernels once the brcmfmac
module is unloaded.

Distribution: Arch Linux
Architecture: x86_64
Device: Dell XPS 13 9350 with BCM4350 wifi card
Kernel config used at compilation (created using the Arch Linux default
config and make olddefconfig):
https://gist.githubusercontent.com/maclomhair/e33fa7eece8f8c77e5a88c4eede2f523/raw/429dfb022498c026225865dbb3ab2f75d5030a54/config
Dmesg output after boot:
https://gist.githubusercontent.com/maclomhair/47c75fa759d93800e7fb5e937fabef3e/raw/10750996a1ea9dcb3fa14d9b94e11fedd3abf280/dmesg

#regzbot introduced da6d9c8ecd00e20218461007948f2b0a8e7fa242

Thank you for your time,
Felix



[-- Attachment #2: kernel_log_after_suspend.txt --]
[-- Type: text/plain, Size: 3278 bytes --]

Freezing remaining freezable tasks ... (elapsed 0.000. seconds) done.
ACPI: EC: interrupt blocked
ACPI: PM: Preparing to enter systen sleep state S3
ACPI: EC: event blocked
ACPI: EC: EC stopped
ACPI: PM: Saving platform NVS memory
Disabling non-boot CPUs ...
smpboot: CPU 1 is now offline
smpboot: CPU 2 is now offline
smpboot: CPU 3 is now offline
ACPI: PM: Low-level resume complete
ACPI: EC: EC started
ACPI: PM: Restoring platform NVS memory
Enabling non-boot CPUs ...
x86: Booting SMP configuration:
smpboot: Booting Node 0 Processor 1 APIC Ox2
CPU1 is up
smpboot: Booting Node 0 Processor 2 APIC 0x1
CPU2 is up
smpboot: Booting Node 0 Processor 3 APIC 0x3
CPU3 is up
ACPI: PM: Waking up from system sleep state S3
ACPI: EC: interrupt unblocked
ACPI: EC: event unblocked
xhci hcd 0000:00:14.0: xHC error in resume, USBSTS Ox411, Reinit
usb usb1: root hub lost power or was reset
usb usb2: root hub lost power or was reset
i915 0000:00:02.0: [drm] [ENCODER:94:DDI A/PHY A] is disabled/in DSI node with an ungated DDI clock, gate it
i915 0000:00:02.0: [drm] [ENCODER:102:DDI B/PHY B] is disabled/in DSI mode with an ungated DDI clock, gate it
i915 0000:00:02.0: [drm] [ENCODER:117:DDI C/PHY C1 is disabled/in DSI node with an ungated DDI clock, gate it
nvme nvme0: 4/0/0 default/read/poll queues
BUG: kernel NULL pointer dereference, address: 0000000000000018
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) — not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 3 PID: 576 Comm: kworker/uB:18 Tainted: G         I         6.1.0-rc7-1-git-01766-gda6d9c8ecd00 #20 fa2e32ffad0b39d53b5e52ea9f9e599123b1d112
Hardware name: Dell Inc. XPS 13 9350/0PWNCR, BIDS 1.13.0 02/10/2020
Workgueue: events_unbound async_run_entry_fn
RIP: 0010:brcmf_pcie_probe+Ox16b/0x7a0 [brcmfmac]
Code: 49 89 44 24 48 48 85 c0 Of 84 f4 05 00 00 48 8d 85 d0 00 00 00 48 8b 34 24 48 89 ef 49 c7 44 24 40 e0 39 0d c1 49 89 44 24 10 <49> 8b 45 18 48 89 5e 08 41 89 44 24 38 8b 83 00 04 00 00 4c 89 26
RSP: 0000:ffffbd8041853d78 EFLAGS: 00010286
RAX: ffff97a10154f0d0 RBX: ffff97a105fc2800 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffff97a275f04ff0 RDI: ffff97a10154f000
RBP: ffff97a10154f000 RO8: 0000000000000040 R09: ffff97a102bec0c0
R10: 0000000066000050 R11: ffffffffa6000031 R12: ffff97a1049379c0
R13: 0000000000000000 R14: 0000000000000000 R15: ffff97a10430c1a8
FS:  0000000000000000(0000) GS:ffff97a276d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 00000000080050033
CR2: 0000000000000018 CR3: 0000000177810001 CR4: 00000000003706e0
Call Trace:
 <TASK>
 brcmf_pcie_pm_leave_D3+0xc5/8x1a0 [brcmfmac be3b4cefca451e190fa35be8f00db1bbec293887]
 ? pci_pm_resume+0x5b/0xf0
 ? pci_legacy_resume+0x80/0x80
 dpm_run_callback+0x47/0x150
 device_resume+0xa2/0x1f0
 async_resume+0x1d/0x30
 async_run_entry_fn+0x31/0x130
 process_one_work+0x1c4/0x380
 worker_thread+0x51/0x390
 ? rescuer_thread+0x3b0/0x3b0
 kthread+Oxdb/0x110
 ? kthread_complete_and_exit+0x20/0x20
usb 1-3: reset full-speed USB device number 2 using xhci_hcd706e0000000000004 24 48 89 ef 49 c7 44 24 40 e0 39 0d c1 49 89 44 24 10 <49> 8b 45 18 48 89 5e 08 41 89 44 24 38 8b 83 00 04 00 00 4c 89 26video i8032 serio wmish_cl
usb 1-5: reset high-speed USB device number 3 using xhci_hcd

[-- Attachment #3: bisect.log --]
[-- Type: text/x-log, Size: 2740 bytes --]

# bad: [c9c3395d5e3dcc6daee66c6908354d47bf98cb0c] Linux 6.2
# good: [830b3c68c1fb1e9176028d02ef86f3cf76aa2476] Linux 6.1
git bisect start 'v6.2' 'v6.1'
# good: [1ca06f1c1acecbe02124f14a37cce347b8c1a90c] Merge tag 'xtensa-20221213' of https://github.com/jcmvbkbc/linux-xtensa
git bisect good 1ca06f1c1acecbe02124f14a37cce347b8c1a90c
# bad: [b83a7080d30032cf70832bc2bb04cc342e203b88] Merge tag 'staging-6.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect bad b83a7080d30032cf70832bc2bb04cc342e203b88
# bad: [7e68dd7d07a28faa2e6574dd6b9dbd90cdeaae91] Merge tag 'net-next-6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
git bisect bad 7e68dd7d07a28faa2e6574dd6b9dbd90cdeaae91
# good: [c609d739947894d7370eae4cf04eb2c49e910bcf] Merge tag 'wireless-next-2022-11-18' of git://git.kernel.org/pub/scm/linux/kernel/git/wireless/wireless-next
git bisect good c609d739947894d7370eae4cf04eb2c49e910bcf
# good: [32163491c0c205ffb1596baf9c308dee5338ae94] Merge branch 'r8169-irq-coalesce'
git bisect good 32163491c0c205ffb1596baf9c308dee5338ae94
# good: [b2b509fb5a1e6af1e630a755b32c4658099df70b] Merge tag 'linux-can-next-for-6.2-20221212' of git://git.kernel.org/pub/scm/linux/kernel/git/mkl/linux-can-next
git bisect good b2b509fb5a1e6af1e630a755b32c4658099df70b
# bad: [f9c5cbd52c62f2fa9d2dfbe1af8fb00e5056aefe] btusb: Avoid reset of ISOC endpoint alt settings to zero
git bisect bad f9c5cbd52c62f2fa9d2dfbe1af8fb00e5056aefe
# good: [6b75bd3d036745b9be30917909f03602099adbdb] bpf: Refactor ARG_PTR_TO_DYNPTR checks into process_dynptr_func
git bisect good 6b75bd3d036745b9be30917909f03602099adbdb
# bad: [695c5d3a8055dcb97966496ade51968c9a75f5c6] wifi: rtl8xxxu: fixing IQK failures for rtl8192eu
git bisect bad 695c5d3a8055dcb97966496ade51968c9a75f5c6
# bad: [d592b9f742643f07db53ac34baf6bbd3ce9478dc] wifi: rtw89: stop mac port function when stop_ap()
git bisect bad d592b9f742643f07db53ac34baf6bbd3ce9478dc
# bad: [8041f2bffbf1c0854d9f6675ad01dc6ca72f71dd] wifi: brcmfmac: introduce BRCMFMAC exported symbols namespace
git bisect bad 8041f2bffbf1c0854d9f6675ad01dc6ca72f71dd
# bad: [d6a5c562214f26e442c8ec3ff1e28e16675d1bcf] wifi: brcmfmac: add support for vendor-specific firmware api
git bisect bad d6a5c562214f26e442c8ec3ff1e28e16675d1bcf
# good: [76821aad49cebc69eca86800f41e8133104e91d3] wifi: brcmfmac: add function to unbind device to bus layer api
git bisect good 76821aad49cebc69eca86800f41e8133104e91d3
# bad: [da6d9c8ecd00e20218461007948f2b0a8e7fa242] wifi: brcmfmac: add firmware vendor info in driver info
git bisect bad da6d9c8ecd00e20218461007948f2b0a8e7fa242
# first bad commit: [da6d9c8ecd00e20218461007948f2b0a8e7fa242] wifi: brcmfmac: add firmware vendor info in driver info

Re: [REGRESSION] wifi: brcmfmac: add firmware vendor info in driver info

[Hans de Goede]

Hi Felix,

On 5/10/23 10:25, Felix wrote:
> Changes made by the mentioned commit lead to Oops when waking up after
> suspend to RAM.
> 
> When waking up a Dell XPS 13 9350 with BCM4350 wifi card after suspend
> to RAM, the kernel hangs with a NULL pointer dereference and Oops. The
> issues was introduced by commit
> da6d9c8ecd00e20218461007948f2b0a8e7fa242 and only happens when the
> brcmfmac module is loaded. This issue is the reason we could not
> upgrade to newer kernels than the 6.1 series on the affected machine.
> 
> 
> Details:
> 
> On a Dell XPS 13 9350 laptop with Broadcom BCM4350 wifi card (according
> to lspci), every kernel starting with 6.2 hangs when waking up after
> suspend to RAM. The issue persisted as of May 2nd (last tested commit
> was 348551ddaf311c76b01cdcbaf61b6fef06a49144). Passing the debug and
> no_console_suspend parameters to the kernel show that a NULL pointer
> dereference and an Oops happen at wakeup. Please find the kernel
> messages readable on the screen at this point attached
> (kernel_log_after_suspend.txt).
> 
> A bisection was successful and produced the first bad commit
> [da6d9c8ecd00e20218461007948f2b0a8e7fa242] wifi: brcmfmac: add firmware
> vendor info in driver info
> I've attached the bisection log as well (bisect.txt).
> 
> Suspend to RAM works on this machine on newer kernels once the brcmfmac
> module is unloaded.
> 
> Distribution: Arch Linux
> Architecture: x86_64
> Device: Dell XPS 13 9350 with BCM4350 wifi card
> Kernel config used at compilation (created using the Arch Linux default
> config and make olddefconfig):
> https://gist.githubusercontent.com/maclomhair/e33fa7eece8f8c77e5a88c4eede2f523/raw/429dfb022498c026225865dbb3ab2f75d5030a54/config
> Dmesg output after boot:
> https://gist.githubusercontent.com/maclomhair/47c75fa759d93800e7fb5e937fabef3e/raw/10750996a1ea9dcb3fa14d9b94e11fedd3abf280/dmesg
> 
> #regzbot introduced da6d9c8ecd00e20218461007948f2b0a8e7fa242

Thank you for reporting this and thank you for bisecting it too!

As a result of your excellent bug report I've been able to write
a fix for this. I'm confident enough in the fix that I've submitted
it upstream right away.

But it would still be good if you can test the fix and let us know
if it fixes this issue for you. You should have received a copy
of the patch when I submitted it upstream.

Regards,

Hans

Re: [REGRESSION] wifi: brcmfmac: add firmware vendor info in driver info

[Kalle Valo]

Hans de Goede <hdegoede@redhat.com> writes:

> On 5/10/23 10:25, Felix wrote:
>> Changes made by the mentioned commit lead to Oops when waking up after
>> suspend to RAM.
>> 
>> When waking up a Dell XPS 13 9350 with BCM4350 wifi card after suspend
>> to RAM, the kernel hangs with a NULL pointer dereference and Oops. The
>> issues was introduced by commit
>> da6d9c8ecd00e20218461007948f2b0a8e7fa242 and only happens when the
>> brcmfmac module is loaded. This issue is the reason we could not
>> upgrade to newer kernels than the 6.1 series on the affected machine.
>> 
>> 
>> Details:
>> 
>> On a Dell XPS 13 9350 laptop with Broadcom BCM4350 wifi card (according
>> to lspci), every kernel starting with 6.2 hangs when waking up after
>> suspend to RAM. The issue persisted as of May 2nd (last tested commit
>> was 348551ddaf311c76b01cdcbaf61b6fef06a49144). Passing the debug and
>> no_console_suspend parameters to the kernel show that a NULL pointer
>> dereference and an Oops happen at wakeup. Please find the kernel
>> messages readable on the screen at this point attached
>> (kernel_log_after_suspend.txt).
>> 
>> A bisection was successful and produced the first bad commit
>> [da6d9c8ecd00e20218461007948f2b0a8e7fa242] wifi: brcmfmac: add firmware
>> vendor info in driver info
>> I've attached the bisection log as well (bisect.txt).
>> 
>> Suspend to RAM works on this machine on newer kernels once the brcmfmac
>> module is unloaded.
>> 
>> Distribution: Arch Linux
>> Architecture: x86_64
>> Device: Dell XPS 13 9350 with BCM4350 wifi card
>> Kernel config used at compilation (created using the Arch Linux default
>> config and make olddefconfig):
>> https://gist.githubusercontent.com/maclomhair/e33fa7eece8f8c77e5a88c4eede2f523/raw/429dfb022498c026225865dbb3ab2f75d5030a54/config
>> Dmesg output after boot:
>> https://gist.githubusercontent.com/maclomhair/47c75fa759d93800e7fb5e937fabef3e/raw/10750996a1ea9dcb3fa14d9b94e11fedd3abf280/dmesg
>> 
>> #regzbot introduced da6d9c8ecd00e20218461007948f2b0a8e7fa242
>
> Thank you for reporting this and thank you for bisecting it too!
>
> As a result of your excellent bug report I've been able to write
> a fix for this. I'm confident enough in the fix that I've submitted
> it upstream right away.
>
> But it would still be good if you can test the fix and let us know
> if it fixes this issue for you. You should have received a copy
> of the patch when I submitted it upstream.

For the archives here's the patch:

https://patchwork.kernel.org/project/linux-wireless/patch/20230510100050.27099-1-hdegoede@redhat.com/

-- 
https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

Re: [REGRESSION] wifi: brcmfmac: add firmware vendor info in driver info

[Felix]

On Wed, 2023-05-10 at 15:09 +0300, Kalle Valo wrote:
> Hans de Goede <hdegoede@redhat.com> writes:
> 
> > On 5/10/23 10:25, Felix wrote:
> > > Changes made by the mentioned commit lead to Oops when waking up after
> > > suspend to RAM.
> > > 
> > > When waking up a Dell XPS 13 9350 with BCM4350 wifi card after suspend
> > > to RAM, the kernel hangs with a NULL pointer dereference and Oops. The
> > > issues was introduced by commit
> > > da6d9c8ecd00e20218461007948f2b0a8e7fa242 and only happens when the
> > > brcmfmac module is loaded. This issue is the reason we could not
> > > upgrade to newer kernels than the 6.1 series on the affected machine.
> > > 
> > > 
> > > Details:
> > > 
> > > On a Dell XPS 13 9350 laptop with Broadcom BCM4350 wifi card (according
> > > to lspci), every kernel starting with 6.2 hangs when waking up after
> > > suspend to RAM. The issue persisted as of May 2nd (last tested commit
> > > was 348551ddaf311c76b01cdcbaf61b6fef06a49144). Passing the debug and
> > > no_console_suspend parameters to the kernel show that a NULL pointer
> > > dereference and an Oops happen at wakeup. Please find the kernel
> > > messages readable on the screen at this point attached
> > > (kernel_log_after_suspend.txt).
> > > 
> > > A bisection was successful and produced the first bad commit
> > > [da6d9c8ecd00e20218461007948f2b0a8e7fa242] wifi: brcmfmac: add firmware
> > > vendor info in driver info
> > > I've attached the bisection log as well (bisect.txt).
> > > 
> > > Suspend to RAM works on this machine on newer kernels once the brcmfmac
> > > module is unloaded.
> > > 
> > > Distribution: Arch Linux
> > > Architecture: x86_64
> > > Device: Dell XPS 13 9350 with BCM4350 wifi card
> > > Kernel config used at compilation (created using the Arch Linux default
> > > config and make olddefconfig):
> > > https://gist.githubusercontent.com/maclomhair/e33fa7eece8f8c77e5a88c4eede2f523/raw/429dfb022498c026225865dbb3ab2f75d5030a54/config
> > > Dmesg output after boot:
> > > https://gist.githubusercontent.com/maclomhair/47c75fa759d93800e7fb5e937fabef3e/raw/10750996a1ea9dcb3fa14d9b94e11fedd3abf280/dmesg
> > > 
> > > #regzbot introduced da6d9c8ecd00e20218461007948f2b0a8e7fa242
> > 
> > Thank you for reporting this and thank you for bisecting it too!
> > 
> > As a result of your excellent bug report I've been able to write
> > a fix for this. I'm confident enough in the fix that I've submitted
> > it upstream right away.
> > 
> > But it would still be good if you can test the fix and let us know
> > if it fixes this issue for you. You should have received a copy
> > of the patch when I submitted it upstream.

I have now tested the patch you sent and it indeed fixes the issue for
me! Thank you for your quick response!


> 
> For the archives here's the patch:
> 
> https://patchwork.kernel.org/project/linux-wireless/patch/20230510100050.27099-1-hdegoede@redhat.com/
> 

Re: [REGRESSION] wifi: brcmfmac: add firmware vendor info in driver info

[Nimrod MacIomhair]

On Wed, 2023-05-10 at 14:53 +0200, Felix wrote:
> On Wed, 2023-05-10 at 15:09 +0300, Kalle Valo wrote:
> > Hans de Goede <hdegoede@redhat.com> writes:
> > 
> > > On 5/10/23 10:25, Felix wrote:
> > > > Changes made by the mentioned commit lead to Oops when waking up after
> > > > suspend to RAM.
> > > > 
> > > > When waking up a Dell XPS 13 9350 with BCM4350 wifi card after suspend
> > > > to RAM, the kernel hangs with a NULL pointer dereference and Oops. The
> > > > issues was introduced by commit
> > > > da6d9c8ecd00e20218461007948f2b0a8e7fa242 and only happens when the
> > > > brcmfmac module is loaded. This issue is the reason we could not
> > > > upgrade to newer kernels than the 6.1 series on the affected machine.
> > > > 
> > > > 
> > > > Details:
> > > > 
> > > > On a Dell XPS 13 9350 laptop with Broadcom BCM4350 wifi card (according
> > > > to lspci), every kernel starting with 6.2 hangs when waking up after
> > > > suspend to RAM. The issue persisted as of May 2nd (last tested commit
> > > > was 348551ddaf311c76b01cdcbaf61b6fef06a49144). Passing the debug and
> > > > no_console_suspend parameters to the kernel show that a NULL pointer
> > > > dereference and an Oops happen at wakeup. Please find the kernel
> > > > messages readable on the screen at this point attached
> > > > (kernel_log_after_suspend.txt).
> > > > 
> > > > A bisection was successful and produced the first bad commit
> > > > [da6d9c8ecd00e20218461007948f2b0a8e7fa242] wifi: brcmfmac: add firmware
> > > > vendor info in driver info
> > > > I've attached the bisection log as well (bisect.txt).
> > > > 
> > > > Suspend to RAM works on this machine on newer kernels once the brcmfmac
> > > > module is unloaded.
> > > > 
> > > > Distribution: Arch Linux
> > > > Architecture: x86_64
> > > > Device: Dell XPS 13 9350 with BCM4350 wifi card
> > > > Kernel config used at compilation (created using the Arch Linux default
> > > > config and make olddefconfig):
> > > > https://gist.githubusercontent.com/maclomhair/e33fa7eece8f8c77e5a88c4eede2f523/raw/429dfb022498c026225865dbb3ab2f75d5030a54/config
> > > > Dmesg output after boot:
> > > > https://gist.githubusercontent.com/maclomhair/47c75fa759d93800e7fb5e937fabef3e/raw/10750996a1ea9dcb3fa14d9b94e11fedd3abf280/dmesg
> > > > 
> > > > #regzbot introduced da6d9c8ecd00e20218461007948f2b0a8e7fa242
> > > 
> > > Thank you for reporting this and thank you for bisecting it too!
> > > 
> > > As a result of your excellent bug report I've been able to write
> > > a fix for this. I'm confident enough in the fix that I've submitted
> > > it upstream right away.
> > > 
> > > But it would still be good if you can test the fix and let us know
> > > if it fixes this issue for you. You should have received a copy
> > > of the patch when I submitted it upstream.
> 
> I have now tested the patch you sent and it indeed fixes the issue for
> me! Thank you for your quick response!

I have now also tested the second version of the patch, and it also
fixes the issue for me!

> 
> 
> > 
> > For the archives here's the patch:
> > 
> > https://patchwork.kernel.org/project/linux-wireless/patch/20230510100050.27099-1-hdegoede@redhat.com/
> > 
>