Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Peter Robinson]

Hi Folks,

I'm seeing a regression in the brcmfmac driver which appeared in the
6.1 dev cycle, I didn't pick it up until around rc8 but with deadlines
and travel I've not had a chance to bisect it but wanted to report it
to make people aware. I've seen in on a number of devices with
brcmfmac wifi over SDIO including at least the Raspberry Pi (zero2w,
rpi4, rpi400, rpi3B+) and other devices like  the Rock960, Pinebook
Pro etc.

A quick look at brcmfmac shows arounf 26 changes for 6.1 and it seems
like it's an issue around firmware loading where it looks for a device
specific firmware but doesn't fall back to loading the more generic
one, or tries to load one ending in .txt, but then doesn't load the
NVRAM.

Anyway I wanted to report it as atm I haven't had  time to actually
debug/bisect it more.

a dmesg snippet for the RPi3B+ is:
[   24.541576] brcmfmac: brcmf_fw_alloc_request: using
brcm/brcmfmac43455-sdio for chip BCM4345/6
[   24.551460] usbcore: registered new interface driver brcmfmac
[   24.559669] brcmfmac mmc1:0001:1: Direct firmware load for
brcm/brcmfmac43455-sdio.raspberrypi,3-model-b-plus-Raspberry Pi 3
Model B Plus Rev 1.3.bin failed with error -2
[   24.693853] brcmfmac mmc1:0001:1: Direct firmware load for
brcm/brcmfmac43455-sdio.txt failed with error -2
[   25.717596] brcmfmac: brcmf_sdio_htclk: HT Avail timeout (1000000):
clkctl 0x50

Regards,
Peter

Re: Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Stefan Wahren]

Hi Peter,

could you please provide a "bad case" file list of all firmware files 
for the Raspberry Pi 3 B+ which worked before Linux 6.1, so i can 
reproduce it?

Also OpenSuSE seems to be affected [1].

[1] - https://bugzilla.suse.com/show_bug.cgi?id=1206697

Regards
Stefan

Re: Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Linux kernel regression tracking (#info)]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; all text you find below is based on a few templates
paragraphs you might have encountered already already in similar form.]

On 31.12.22 02:00, Peter Robinson wrote:
> 
> I'm seeing a regression in the brcmfmac driver which appeared in the
> 6.1 dev cycle, I didn't pick it up until around rc8 but with deadlines
> and travel I've not had a chance to bisect it but wanted to report it
> to make people aware. I've seen in on a number of devices with
> brcmfmac wifi over SDIO including at least the Raspberry Pi (zero2w,
> rpi4, rpi400, rpi3B+) and other devices like  the Rock960, Pinebook
> Pro etc.
> [...]

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot introduced v6.0..v6.1 ^
https://bugzilla.suse.com/show_bug.cgi?id=1206697
#regzbot title net: wifi: brcmfmac over SDIO broken on various Raspberry Pi
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Reminder for developers: When fixing the issue, add 'Link:' tags
pointing to the report (see page linked in footer for details).

Ciao, Thorsten
-- 
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr

Annoyed by mails like this? Feel free to send them to /dev/null:
https://linux-regtracking.leemhuis.info/about/#infomails

Re: Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Peter Robinson]

Hi Stefan,

> could you please provide a "bad case" file list of all firmware files
> for the Raspberry Pi 3 B+ which worked before Linux 6.1, so i can
> reproduce it?

What do you mean by the file list? We're running the latest upstream
firmware, we do used compressed firmware but that hasn't regressed
with any of the other firmware, this is very much brcmfmac. With the
same firmware it was working on 6.0.x and reboot into a 6.1+ kernel
and it stops with nothing else changed.

> Also OpenSuSE seems to be affected [1].
>
> [1] - https://bugzilla.suse.com/show_bug.cgi?id=1206697
>
> Regards
> Stefan
>

Re: Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Stefan Wahren]

Hi Peter,

Am 31.12.22 um 15:33 schrieb Peter Robinson:
> Hi Stefan,
>
>> could you please provide a "bad case" file list of all firmware files
>> for the Raspberry Pi 3 B+ which worked before Linux 6.1, so i can
>> reproduce it?
> What do you mean by the file list?

if you look into the firmware directory which files for 
brcmfmac43455-sdio do you have? Just the bin file?

I'm using the rootfs of Raspberry Pi OS which contains:

$ ls brcmfmac43455-sdio*
brcmfmac43455-sdio.bin  brcmfmac43455-sdio.clm_blob brcmfmac43455-sdio.txt

I didn't say please add some files, i only need a test scenario :-)

>   We're running the latest upstream
> firmware, we do used compressed firmware but that hasn't regressed
> with any of the other firmware, this is very much brcmfmac. With the
> same firmware it was working on 6.0.x and reboot into a 6.1+ kernel
> and it stops with nothing else changed.

Yes, i agree this is a regression and yes it's definitely brcmfmac, but 
based on your bug report i wasn't able to reproduce.

Regards

>
>> Also OpenSuSE seems to be affected [1].
>>
>> [1] - https://bugzilla.suse.com/show_bug.cgi?id=1206697
>>
>> Regards
>> Stefan
>>

Re: Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Peter Robinson]

Hi Stefan,

> >> could you please provide a "bad case" file list of all firmware files
> >> for the Raspberry Pi 3 B+ which worked before Linux 6.1, so i can
> >> reproduce it?
> > What do you mean by the file list?
>
> if you look into the firmware directory which files for
> brcmfmac43455-sdio do you have? Just the bin file?
>
> I'm using the rootfs of Raspberry Pi OS which contains:
>
> $ ls brcmfmac43455-sdio*
> brcmfmac43455-sdio.bin  brcmfmac43455-sdio.clm_blob brcmfmac43455-sdio.txt

As shipped in https://mirrors.edge.kernel.org/pub/linux/kernel/firmware/
either the Nov or Dec releases

The kernel supports compressing firmware with: xz -C crc32 "$f"

Regards,
Peter

> I didn't say please add some files, i only need a test scenario :-)
>
> >   We're running the latest upstream
> > firmware, we do used compressed firmware but that hasn't regressed
> > with any of the other firmware, this is very much brcmfmac. With the
> > same firmware it was working on 6.0.x and reboot into a 6.1+ kernel
> > and it stops with nothing else changed.
>
> Yes, i agree this is a regression and yes it's definitely brcmfmac, but
> based on your bug report i wasn't able to reproduce.
>
> Regards
>
> >
> >> Also OpenSuSE seems to be affected [1].
> >>
> >> [1] - https://bugzilla.suse.com/show_bug.cgi?id=1206697
> >>
> >> Regards
> >> Stefan
> >>

Re: Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Stefan Wahren]

Hi Peter,

Am 31.12.22 um 16:58 schrieb Peter Robinson:
> Hi Stefan,
>
>>>> could you please provide a "bad case" file list of all firmware files
>>>> for the Raspberry Pi 3 B+ which worked before Linux 6.1, so i can
>>>> reproduce it?
>>> What do you mean by the file list?
>> if you look into the firmware directory which files for
>> brcmfmac43455-sdio do you have? Just the bin file?
>>
>> I'm using the rootfs of Raspberry Pi OS which contains:
>>
>> $ ls brcmfmac43455-sdio*
>> brcmfmac43455-sdio.bin  brcmfmac43455-sdio.clm_blob brcmfmac43455-sdio.txt

i temporary moved the clm_blob and txt file from the directory and 
switched to Linux v6.0 (multi_v7_defconfig), but the driver doesn't probe:

[   25.943350] brcmfmac: brcmf_fw_alloc_request: using 
brcm/brcmfmac43455-sdio for chip BCM4345/6
[   25.943550] brcmfmac mmc1:0001:1: Direct firmware load for 
brcm/brcmfmac43455-sdio.raspberrypi,3-model-b-plus.bin failed with error -2
[   26.071070] brcmfmac mmc1:0001:1: Direct firmware load for 
brcm/brcmfmac43455-sdio.raspberrypi,3-model-b-plus.txt failed with error -2
[   26.071133] brcmfmac mmc1:0001:1: Direct firmware load for 
brcm/brcmfmac43455-sdio.txt failed with error -2
[   27.078481] brcmfmac: brcmf_sdio_htclk: HT Avail timeout (1000000): 
clkctl 0x50

>> [   25.943350] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
>> [   25.943550] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,3-model-b-plus.bin failed with error -2
>> [   26.071070] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,3-model-b-plus.txt failed with error -2
>> [   26.071133] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.txt failed with error -2
>> [   27.078481] brcmfmac: brcmf_sdio_htclk: HT Avail timeout (1000000): clkctl 0x50
> As shipped in https://mirrors.edge.kernel.org/pub/linux/kernel/firmware/
> either the Nov or Dec releases
>
> The kernel supports compressing firmware with: xz -C crc32 "$f"
>
> Regards,
> Peter
>
>> I didn't say please add some files, i only need a test scenario :-)
>>
>>>    We're running the latest upstream
>>> firmware, we do used compressed firmware but that hasn't regressed
>>> with any of the other firmware, this is very much brcmfmac. With the
>>> same firmware it was working on 6.0.x and reboot into a 6.1+ kernel
>>> and it stops with nothing else changed.
>> Yes, i agree this is a regression and yes it's definitely brcmfmac, but
>> based on your bug report i wasn't able to reproduce.
>>
>> Regards
>>
>>>> Also OpenSuSE seems to be affected [1].
>>>>
>>>> [1] - https://bugzilla.suse.com/show_bug.cgi?id=1206697
>>>>
>>>> Regards
>>>> Stefan
>>>>

Re: Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Stefan Wahren]

Hi Peter,

Am 31.12.22 um 12:11 schrieb Stefan Wahren:
> Hi Peter,
>
> could you please provide a "bad case" file list of all firmware files 
> for the Raspberry Pi 3 B+ which worked before Linux 6.1, so i can 
> reproduce it?
>
> Also OpenSuSE seems to be affected [1].

Ivan send out a fix, which should also work for Fedora [2].

Best regards

[2] - https://lore.kernel.org/all/20230106072746.29516-1-iivanov@suse.de/

>
> [1] - https://bugzilla.suse.com/show_bug.cgi?id=1206697
>
> Regards
> Stefan
>

Re: Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Peter Robinson]

Hi Stefan,

> > could you please provide a "bad case" file list of all firmware files
> > for the Raspberry Pi 3 B+ which worked before Linux 6.1, so i can
> > reproduce it?
> >
> > Also OpenSuSE seems to be affected [1].
>
> Ivan send out a fix, which should also work for Fedora [2].

Yes, he cc:ed me, thanks for heads up, I tested and confirmed it fixed
the issue.

Peter

Re: Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Kalle Valo]

"Linux kernel regression tracking (#info)" <regressions@leemhuis.info>
writes:

> [TLDR: I'm adding this report to the list of tracked Linux kernel
> regressions; all text you find below is based on a few templates
> paragraphs you might have encountered already already in similar form.]
>
> On 31.12.22 02:00, Peter Robinson wrote:
>> 
>> I'm seeing a regression in the brcmfmac driver which appeared in the
>> 6.1 dev cycle, I didn't pick it up until around rc8 but with deadlines
>> and travel I've not had a chance to bisect it but wanted to report it
>> to make people aware. I've seen in on a number of devices with
>> brcmfmac wifi over SDIO including at least the Raspberry Pi (zero2w,
>> rpi4, rpi400, rpi3B+) and other devices like  the Rock960, Pinebook
>> Pro etc.
>> [...]
>
> Thanks for the report. To be sure the issue doesn't fall through the
> cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
> tracking bot:
>
> #regzbot introduced v6.0..v6.1 ^
> https://bugzilla.suse.com/show_bug.cgi?id=1206697
> #regzbot title net: wifi: brcmfmac over SDIO broken on various Raspberry Pi
> #regzbot ignore-activity

This commit should fix the issue:

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=a5a36720c3f650f859f5e9535dd62d06f13f4f3b

-- 
https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

Re: Regression in brcmfmac for 6.1/6.2-rc1 for SDIO devices

[Linux kernel regression tracking (#update)]

On 09.01.23 15:34, Kalle Valo wrote:
> "Linux kernel regression tracking (#info)" <regressions@leemhuis.info>
> writes:
> 
>> [TLDR: I'm adding this report to the list of tracked Linux kernel
>> regressions; all text you find below is based on a few templates
>> paragraphs you might have encountered already already in similar form.]
>>
>> On 31.12.22 02:00, Peter Robinson wrote:
>>>
>>> I'm seeing a regression in the brcmfmac driver which appeared in the
>>> 6.1 dev cycle, I didn't pick it up until around rc8 but with deadlines
>>> and travel I've not had a chance to bisect it but wanted to report it
>>> to make people aware. I've seen in on a number of devices with
>>> brcmfmac wifi over SDIO including at least the Raspberry Pi (zero2w,
>>> rpi4, rpi400, rpi3B+) and other devices like  the Rock960, Pinebook
>>> Pro etc.
>>> [...]
>>
>> Thanks for the report. To be sure the issue doesn't fall through the
>> cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
>> tracking bot:
>>
>> #regzbot introduced v6.0..v6.1 ^
>> https://bugzilla.suse.com/show_bug.cgi?id=1206697
>> #regzbot title net: wifi: brcmfmac over SDIO broken on various Raspberry Pi
>> #regzbot ignore-activity
> 
> This commit should fix the issue:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=a5a36720c3f650f859f5e9535dd62d06f13f4f3b

Great, thx for letting me know!

#regzbot fix: a5a36720c3f65

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.