Re: [EXT] Re: [PATCH v2 net-next 1/2] bnx2x: Utilize firmware 7.13.21.0

[Paul Menzel <pmenzel@molgen.mpg.de>]

Dear Jakub,


Am 15.03.22 um 03:57 schrieb Jakub Kicinski:
> On Mon, 14 Mar 2022 16:07:08 +0100 Paul Menzel wrote:
>>> There might be something more wrong with the patch in the subject: The
>>> usability of the ports from a single card (with older firmware?) now
>>> depends on the order the ports are enabled (first port enabled is
>>> working, second port enabled is not working, driver complaining about a
>>> firmware mismatch).
>>>
>>> In the following examples, the driver was not built-in to the kernel but
>>> loaded from the root filesystem instead, so there is no initramfs
>>> related problem here.
>>>
>>> For the records:
>>>
>>> root@ira:~# dmesg|grep bnx2x
>>> [   18.749871] bnx2x 0000:45:00.0: msix capability found
>>> [   18.766534] bnx2x 0000:45:00.0: part number 394D4342-31373735-31314131-473331
>>> [   18.799198] bnx2x 0000:45:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
>>> [   18.807638] bnx2x 0000:45:00.1: msix capability found
>>> [   18.824509] bnx2x 0000:45:00.1: part number 394D4342-31373735-31314131-473331
>>> [   18.857171] bnx2x 0000:45:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
>>> [   18.865619] bnx2x 0000:46:00.0: msix capability found
>>> [   18.882636] bnx2x 0000:46:00.0: part number 394D4342-31373735-31314131-473331
>>> [   18.915196] bnx2x 0000:46:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
>>> [   18.923636] bnx2x 0000:46:00.1: msix capability found
>>> [   18.940505] bnx2x 0000:46:00.1: part number 394D4342-31373735-31314131-473331
>>> [   18.973167] bnx2x 0000:46:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
>>> [   46.480660] bnx2x 0000:45:00.0 net04: renamed from eth4
>>> [   46.494677] bnx2x 0000:45:00.1 net05: renamed from eth5
>>> [   46.508544] bnx2x 0000:46:00.0 net06: renamed from eth6
>>> [   46.524641] bnx2x 0000:46:00.1 net07: renamed from eth7
>>> root@ira:~# ls /lib/firmware/bnx2x/
>>> bnx2x-e1-6.0.34.0.fw   bnx2x-e1-7.13.1.0.fw   bnx2x-e1-7.8.2.0.fw
>>> bnx2x-e1h-7.12.30.0.fw  bnx2x-e1h-7.8.19.0.fw  bnx2x-e2-7.10.51.0.fw  bnx2x-e2-7.8.17.0.fw
>>> bnx2x-e1-6.2.5.0.fw    bnx2x-e1-7.13.11.0.fw  bnx2x-e1h-6.0.34.0.fw
>>> bnx2x-e1h-7.13.1.0.fw   bnx2x-e1h-7.8.2.0.fw   bnx2x-e2-7.12.30.0.fw  bnx2x-e2-7.8.19.0.fw
>>> bnx2x-e1-6.2.9.0.fw    bnx2x-e1-7.13.15.0.fw  bnx2x-e1h-6.2.5.0.fw
>>> bnx2x-e1h-7.13.11.0.fw  bnx2x-e2-6.0.34.0.fw   bnx2x-e2-7.13.1.0.fw   bnx2x-e2-7.8.2.0.fw
>>> bnx2x-e1-7.0.20.0.fw   bnx2x-e1-7.13.21.0.fw  bnx2x-e1h-6.2.9.0.fw
>>> bnx2x-e1h-7.13.15.0.fw  bnx2x-e2-6.2.5.0.fw    bnx2x-e2-7.13.11.0.fw
>>> bnx2x-e1-7.0.23.0.fw   bnx2x-e1-7.2.16.0.fw   bnx2x-e1h-7.0.20.0.fw
>>> bnx2x-e1h-7.13.21.0.fw  bnx2x-e2-6.2.9.0.fw    bnx2x-e2-7.13.15.0.fw
>>> bnx2x-e1-7.0.29.0.fw   bnx2x-e1-7.2.51.0.fw   bnx2x-e1h-7.0.23.0.fw
>>> bnx2x-e1h-7.2.16.0.fw   bnx2x-e2-7.0.20.0.fw   bnx2x-e2-7.13.21.0.fw
>>> bnx2x-e1-7.10.51.0.fw  bnx2x-e1-7.8.17.0.fw   bnx2x-e1h-7.0.29.0.fw
>>> bnx2x-e1h-7.2.51.0.fw   bnx2x-e2-7.0.23.0.fw   bnx2x-e2-7.2.16.0.fw
>>> bnx2x-e1-7.12.30.0.fw  bnx2x-e1-7.8.19.0.fw   bnx2x-e1h-7.10.51.0.fw
>>> bnx2x-e1h-7.8.17.0.fw   bnx2x-e2-7.0.29.0.fw   bnx2x-e2-7.2.51.0.fw
>>>
>>> Now with v5.10.95, the first kernel of the series which includes
>>> fdcfabd0952d ("bnx2x: Utilize firmware 7.13.21.0") and later:
>>>
>>> root@ira:~# dmesg -w &
>>> [...]
>>> root@ira:~# ip link set net04 up
>>> [   88.504536] bnx2x 0000:45:00.0 net04: using MSI-X  IRQs: sp 47  fp[0] 49 ... fp[7] 56
>>> root@ira:~# ip link set net05 up
>>> [   90.825820] bnx2x: [bnx2x_compare_fw_ver:2380(net05)]bnx2x with FW 120d07 was already loaded which mismatches my 150d07 FW. Aborting
>>> RTNETLINK answers: Device or resource busy
>>> root@ira:~# ip link set net04 down
>>> root@ira:~# ip link set net05 down
>>> root@ira:~# ip link set net05 up
>>> [  114.462448] bnx2x 0000:45:00.1 net05: using MSI-X  IRQs: sp 58  fp[0] 60 ... fp[7] 67
>>> root@ira:~# ip link set net04 up
>>> [  117.247763] bnx2x: [bnx2x_compare_fw_ver:2380(net04)]bnx2x with FW 120d07 was already loaded which mismatches my 150d07 FW. Aborting
>>> RTNETLINK answers: Device or resource busy
>>>
>>> With v5.10.94, both ports work fine:
>>>
>>> root@ira:~# dmesg -w &
>>> [...]
>>> root@ira:~# ip link set net04 up
>>> [  133.126647] bnx2x 0000:45:00.0 net04: using MSI-X  IRQs: sp 47  fp[0] 49 ... fp[7] 56
>>> root@ira:~# ip link set net05 up
>>> [  136.215169] bnx2x 0000:45:00.1 net05: using MSI-X  IRQs: sp 58  fp[0] 60 ... fp[7] 67
>>
>> One additional note, that it’s totally unclear to us, where FW version
>> 120d07 in the error message comes from. It maps to 7.13.18.0, which is
>> nowhere to be found and too new to be on the cards EEPROM, which should
>> be from 2013 or so.
> 
> Hrm, any chance an out-of-tree driver or another OS loaded it?

No, no such chance. The system firmware was not touched in the last five 
years, and there is no other OS.

> Does the dmesg indicate that the host loaded the FW at all?

Unfortunately, no such Linux log messages exist (in the code).

> Looks like upstream went from .15 to .21, .18 was never in the picture.

Yes, that is the strange thing. Is there any chance, that the `.18` is 
in the firmware file somehow, and was forgotten to be updated to `.21`?

```
$ md5sum /lib/firmware/bnx2x/*{15,21}*
4137c813f3ff937f01fddf13c309d972  /lib/firmware/bnx2x/bnx2x-e1-7.13.15.0.fw
075539e1072908a0c86ba352c1130f60  /lib/firmware/bnx2x/bnx2x-e1h-7.13.15.0.fw
e59925dedf7ed95679e629dfeeaf5803  /lib/firmware/bnx2x/bnx2x-e2-7.13.15.0.fw
a470a0d77930ef72ac7b725a5ff447fe  /lib/firmware/bnx2x/bnx2x-e1-7.13.21.0.fw
f02012b118664c0579ab58757f276982  /lib/firmware/bnx2x/bnx2x-e1h-7.13.21.0.fw
47667a7bf156d1e531fde42d800e4a66  /lib/firmware/bnx2x/bnx2x-e2-7.13.21.0.fw
```

> Also, will the revert work for you?

I tested reverting the three patches you also listed on top of 5.10.103, 
and that restored the working behavior. Manish’s patch *[RFC net] bnx2x: 
fix built-in kernel driver load failure* also fixes the issue.


Kind regards,

Paul