* Re: [Intel-wired-lan] [PATCH] ice: wait for EMP reset after firmware flash
2022-04-12 10:27 [PATCH] ice: wait for EMP reset after firmware flash Petr Oros
@ 2022-04-12 13:28 ` Paul Menzel
2022-04-29 11:32 ` [Intel-wired-lan] [PATCH] ice: wait for EMP reset after firmware flash #forregzbot Thorsten Leemhuis
2022-04-12 16:08 ` [Intel-wired-lan] [PATCH] ice: wait for EMP reset after firmware flash Alexander Lobakin
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: Paul Menzel @ 2022-04-12 13:28 UTC (permalink / raw)
To: Petr Oros
Cc: Jacob Keller, ivecera, intel-wired-lan, linux-kernel, kuba,
pabeni, davem, netdev, regressions
[Cc: +Jakob, +regressions@lists.linux.dev]
Dear Petr,
Thank you for your patch for the regression.
#regzbot ^introduced 399e27dbbd9e94
Am 12.04.22 um 12:27 schrieb Petr Oros:
Please mention the time in the commit message summary:
ice: Wait 5 s for EMP reset after firmware flash
> We need to wait for EMP reset after firmware flash.
> Code was extracted from OOT driver and without this wait fw_activate let
Which OOT driver exactly?
> card in inconsistent state recoverable only by second flash/activate
Please reflow for 75 characters per line, and add a dot/period to the
end of sentences.
> Reproducer:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> dmesg after flash:
> [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
> [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
> [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0
> [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
> [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
> [ 55.647348] ice 0000:ca:00.0: PTP init successful
> [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
> [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
> [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
> [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
So, what is the error? `Get PHY capabilities failed status = -5`?
What firmware version did the network card have before and after the update?
> Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
s/don't/doesn’t/
> After patch:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
Did you try anything less than five seconds? i40e uses one second, cf.
commit 9b13bd53134c (i40e: Increase delay to 1 s after global EMP reset).
> Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")
> Signed-off-by: Petr Oros <poros@redhat.com>
> ---
> drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
> index d768925785ca79..90ea2203cdc763 100644
> --- a/drivers/net/ethernet/intel/ice/ice_main.c
> +++ b/drivers/net/ethernet/intel/ice/ice_main.c
> @@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
>
> dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
>
> +#define ICE_EMP_RESET_SLEEP 5000
Please append the unit to the macro name.
> if (reset_type == ICE_RESET_EMPR) {
> /* If an EMP reset has occurred, any previously pending flash
> * update will have completed. We no longer know whether or
> * not the NVM update EMP reset is restricted.
> */
> pf->fw_emp_reset_disabled = false;
> +
> + msleep(ICE_EMP_RESET_SLEEP);
> }
>
> err = ice_init_all_ctrlq(hw);
Kind regards,
Paul
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [Intel-wired-lan] [PATCH] ice: wait for EMP reset after firmware flash #forregzbot
2022-04-12 13:28 ` [Intel-wired-lan] " Paul Menzel
@ 2022-04-29 11:32 ` Thorsten Leemhuis
0 siblings, 0 replies; 9+ messages in thread
From: Thorsten Leemhuis @ 2022-04-29 11:32 UTC (permalink / raw)
To: regressions, Paul Menzel; +Cc: intel-wired-lan, linux-kernel, netdev
TWIMC: this mail is primarily send for documentation purposes and for
regzbot, my Linux kernel regression tracking bot. These mails usually
contain '#forregzbot' in the subject, to make them easy to spot and filter.
#regzbot fixed-by: b537752e6cbf0e
On 12.04.22 15:28, Paul Menzel wrote:
> [Cc: +Jakob, +regressions@lists.linux.dev]
>
>
> Dear Petr,
>
>
> Thank you for your patch for the regression.
>
> #regzbot ^introduced 399e27dbbd9e94
>
>
> Am 12.04.22 um 12:27 schrieb Petr Oros:
>
> Please mention the time in the commit message summary:
>
> ice: Wait 5 s for EMP reset after firmware flash
>
>> We need to wait for EMP reset after firmware flash.
>> Code was extracted from OOT driver and without this wait fw_activate let
>
> Which OOT driver exactly?
>
>> card in inconsistent state recoverable only by second flash/activate
>
> Please reflow for 75 characters per line, and add a dot/period to the
> end of sentences.
>
>> Reproducer:
>> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
>> E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
>> Preparing to flash
>> [fw.mgmt] Erasing
>> [fw.mgmt] Erasing done
>> [fw.mgmt] Flashing 100%
>> [fw.mgmt] Flashing done 100%
>> [fw.undi] Erasing
>> [fw.undi] Erasing done
>> [fw.undi] Flashing 100%
>> [fw.undi] Flashing done 100%
>> [fw.netlist] Erasing
>> [fw.netlist] Erasing done
>> [fw.netlist] Flashing 100%
>> [fw.netlist] Flashing done 100%
>> Activate new firmware by devlink reload
>> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
>> reload_actions_performed:
>> fw_activate
>> [root@host ~]# ip link show ens7f0
>> 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq
>> state DOWN mode DEFAULT group default qlen 1000
>> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
>> altname enp202s0f0
>>
>> dmesg after flash:
>> [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
>> [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status =
>> -5, continuing anyway
>> [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully
>> loaded: ICE OS Default Package version 1.3.28.0
>> [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
>> [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
>> [ 55.647348] ice 0000:ca:00.0: PTP init successful
>> [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max
>> number of TCs supported on this port are 8
>> [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW
>> mode.
>> [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
>> [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe
>> bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable
>> of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
>
> So, what is the error? `Get PHY capabilities failed status = -5`?
>
> What firmware version did the network card have before and after the
> update?
>
>> Reboot don't help, only second flash/activate with OOT or patched
>> driver put card back in consistent state
>
> s/don't/doesn’t/
>
>> After patch:
>> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
>> E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
>> Preparing to flash
>> [fw.mgmt] Erasing
>> [fw.mgmt] Erasing done
>> [fw.mgmt] Flashing 100%
>> [fw.mgmt] Flashing done 100%
>> [fw.undi] Erasing
>> [fw.undi] Erasing done
>> [fw.undi] Flashing 100%
>> [fw.undi] Flashing done 100%
>> [fw.netlist] Erasing
>> [fw.netlist] Erasing done
>> [fw.netlist] Flashing 100%
>> [fw.netlist] Flashing done 100%
>> Activate new firmware by devlink reload
>> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
>> reload_actions_performed:
>> fw_activate
>> [root@host ~]# ip link show ens7f0
>> 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state
>> UP mode DEFAULT group default qlen 1000
>> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
>> altname enp202s0f0
>
> Did you try anything less than five seconds? i40e uses one second, cf.
> commit 9b13bd53134c (i40e: Increase delay to 1 s after global EMP reset).
>
>> Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via
>> devlink reload")
>> Signed-off-by: Petr Oros <poros@redhat.com>
>> ---
>> drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/net/ethernet/intel/ice/ice_main.c
>> b/drivers/net/ethernet/intel/ice/ice_main.c
>> index d768925785ca79..90ea2203cdc763 100644
>> --- a/drivers/net/ethernet/intel/ice/ice_main.c
>> +++ b/drivers/net/ethernet/intel/ice/ice_main.c
>> @@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf,
>> enum ice_reset_req reset_type)
>> dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
>> +#define ICE_EMP_RESET_SLEEP 5000
>
> Please append the unit to the macro name.
>
>> if (reset_type == ICE_RESET_EMPR) {
>> /* If an EMP reset has occurred, any previously pending flash
>> * update will have completed. We no longer know whether or
>> * not the NVM update EMP reset is restricted.
>> */
>> pf->fw_emp_reset_disabled = false;
>> +
>> + msleep(ICE_EMP_RESET_SLEEP);
>> }
>> err = ice_init_all_ctrlq(hw);
>
>
> Kind regards,
>
> Paul
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Intel-wired-lan] [PATCH] ice: wait for EMP reset after firmware flash
2022-04-12 10:27 [PATCH] ice: wait for EMP reset after firmware flash Petr Oros
2022-04-12 13:28 ` [Intel-wired-lan] " Paul Menzel
@ 2022-04-12 16:08 ` Alexander Lobakin
2022-04-12 18:29 ` Jesse Brandeburg
2022-04-12 16:58 ` Jacob Keller
2022-04-13 15:37 ` [PATCH v2] ice: wait 5 s " Petr Oros
3 siblings, 1 reply; 9+ messages in thread
From: Alexander Lobakin @ 2022-04-12 16:08 UTC (permalink / raw)
To: Petr Oros
Cc: Alexander Lobakin, netdev, ivecera, intel-wired-lan,
linux-kernel, kuba, pabeni, davem
From: Petr Oros <poros@redhat.com>
Date: Tue, 12 Apr 2022 12:27:53 +0200
> We need to wait for EMP reset after firmware flash.
> Code was extracted from OOT driver and without this wait fw_activate let
> card in inconsistent state recoverable only by second flash/activate
>
> Reproducer:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> dmesg after flash:
> [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
> [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
> [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0
> [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
> [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
> [ 55.647348] ice 0000:ca:00.0: PTP init successful
> [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
> [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
> [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
> [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
> Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
>
> After patch:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")
> Signed-off-by: Petr Oros <poros@redhat.com>
> ---
> drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
> index d768925785ca79..90ea2203cdc763 100644
> --- a/drivers/net/ethernet/intel/ice/ice_main.c
> +++ b/drivers/net/ethernet/intel/ice/ice_main.c
> @@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
>
> dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
>
> +#define ICE_EMP_RESET_SLEEP 5000
Ooof, 5 sec is a lot! Is there any way to poll the device readiness?
Does it really need the whole 5 sec?
> if (reset_type == ICE_RESET_EMPR) {
> /* If an EMP reset has occurred, any previously pending flash
> * update will have completed. We no longer know whether or
> * not the NVM update EMP reset is restricted.
> */
> pf->fw_emp_reset_disabled = false;
> +
> + msleep(ICE_EMP_RESET_SLEEP);
> }
>
> err = ice_init_all_ctrlq(hw);
> --
> 2.35.1
Thanks,
Al
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [Intel-wired-lan] [PATCH] ice: wait for EMP reset after firmware flash
2022-04-12 16:08 ` [Intel-wired-lan] [PATCH] ice: wait for EMP reset after firmware flash Alexander Lobakin
@ 2022-04-12 18:29 ` Jesse Brandeburg
0 siblings, 0 replies; 9+ messages in thread
From: Jesse Brandeburg @ 2022-04-12 18:29 UTC (permalink / raw)
To: Alexander Lobakin, Petr Oros
Cc: ivecera, netdev, linux-kernel, intel-wired-lan, kuba, pabeni, davem
On 4/12/2022 9:08 AM, Alexander Lobakin wrote:
>> +#define ICE_EMP_RESET_SLEEP 5000
>
> Ooof, 5 sec is a lot! Is there any way to poll the device readiness?
> Does it really need the whole 5 sec?
The problem with resets is usually you don't want to disturb the
hardware with PCIe pokes while it's resetting. In this case, this code
only runs if it was indicated by the update status that the firmware
could internally reset itself to pick up new updates, avoiding a reboot.
Yes the five seconds is less than ideal, but it was the value we found
that offered the most stability / likelihood of reset completing. In
this case the 5 seconds is a tradeoff vs a reboot.
Thank you Ivan for chasing down this bug!
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] ice: wait for EMP reset after firmware flash
2022-04-12 10:27 [PATCH] ice: wait for EMP reset after firmware flash Petr Oros
2022-04-12 13:28 ` [Intel-wired-lan] " Paul Menzel
2022-04-12 16:08 ` [Intel-wired-lan] [PATCH] ice: wait for EMP reset after firmware flash Alexander Lobakin
@ 2022-04-12 16:58 ` Jacob Keller
2022-04-13 15:38 ` Petr Oros
2022-04-13 15:37 ` [PATCH v2] ice: wait 5 s " Petr Oros
3 siblings, 1 reply; 9+ messages in thread
From: Jacob Keller @ 2022-04-12 16:58 UTC (permalink / raw)
To: Petr Oros, netdev
Cc: jesse.brandeburg, anthony.l.nguyen, davem, kuba, pabeni,
intel-wired-lan, linux-kernel, ivecera
On 4/12/2022 3:27 AM, Petr Oros wrote:
> We need to wait for EMP reset after firmware flash.
> Code was extracted from OOT driver and without this wait fw_activate let
> card in inconsistent state recoverable only by second flash/activate
>
> Reproducer:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> dmesg after flash:
> [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
> [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
> [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0
> [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
> [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
> [ 55.647348] ice 0000:ca:00.0: PTP init successful
> [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
> [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
> [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
> [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
> Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
>
> After patch:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
Ahh.. good find. I checked a bunch of places, but didn't check here for
differences. :(
For what its worth, I checked the source history of the out-of-tree
driver this came from. It appears to be a workaround added for fixing a
similar issue.
I haven't been able to dig up the full details yet. It appeares to be a
collision with firmware finalizing recovery after the EMP reset.
Still trying to dig for any more information I can find.
> Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")
> Signed-off-by: Petr Oros <poros@redhat.com>
> ---
> drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
> index d768925785ca79..90ea2203cdc763 100644
> --- a/drivers/net/ethernet/intel/ice/ice_main.c
> +++ b/drivers/net/ethernet/intel/ice/ice_main.c
> @@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
>
> dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
>
> +#define ICE_EMP_RESET_SLEEP 5000
> if (reset_type == ICE_RESET_EMPR) {
> /* If an EMP reset has occurred, any previously pending flash
> * update will have completed. We no longer know whether or
> * not the NVM update EMP reset is restricted.
> */
> pf->fw_emp_reset_disabled = false;
> +
> + msleep(ICE_EMP_RESET_SLEEP);
> }
>
> err = ice_init_all_ctrlq(hw);
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH] ice: wait for EMP reset after firmware flash
2022-04-12 16:58 ` Jacob Keller
@ 2022-04-13 15:38 ` Petr Oros
0 siblings, 0 replies; 9+ messages in thread
From: Petr Oros @ 2022-04-13 15:38 UTC (permalink / raw)
To: Jacob Keller, netdev
Cc: jesse.brandeburg, anthony.l.nguyen, davem, kuba, pabeni,
intel-wired-lan, linux-kernel, ivecera
Jacob Keller píše v Út 12. 04. 2022 v 09:58 -0700:
>
>
> On 4/12/2022 3:27 AM, Petr Oros wrote:
> > We need to wait for EMP reset after firmware flash.
> > Code was extracted from OOT driver and without this wait
> > fw_activate let
> > card in inconsistent state recoverable only by second
> > flash/activate
> >
> > Reproducer:
> > [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
> > E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bi
> > n
> > Preparing to flash
> > [fw.mgmt] Erasing
> > [fw.mgmt] Erasing done
> > [fw.mgmt] Flashing 100%
> > [fw.mgmt] Flashing done 100%
> > [fw.undi] Erasing
> > [fw.undi] Erasing done
> > [fw.undi] Flashing 100%
> > [fw.undi] Flashing done 100%
> > [fw.netlist] Erasing
> > [fw.netlist] Erasing done
> > [fw.netlist] Flashing 100%
> > [fw.netlist] Flashing done 100%
> > Activate new firmware by devlink reload
> > [root@host ~]# devlink dev reload pci/0000:ca:00.0 action
> > fw_activate
> > reload_actions_performed:
> > fw_activate
> > [root@host ~]# ip link show ens7f0
> > 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq
> > state DOWN mode DEFAULT group default qlen 1000
> > link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> > altname enp202s0f0
> >
> > dmesg after flash:
> > [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
> > [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status
> > = -5, continuing anyway
> > [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully
> > loaded: ICE OS Default Package version 1.3.28.0
> > [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
> > [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
> > [ 55.647348] ice 0000:ca:00.0: PTP init successful
> > [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware,
> > max number of TCs supported on this port are 8
> > [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in
> > SW mode.
> > [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the
> > hardware
> > [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe
> > bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0
> > (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
> > Reboot don't help, only second flash/activate with OOT or patched
> > driver put card back in consistent state
> >
> > After patch:
> > [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
> > E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bi
> > n
> > Preparing to flash
> > [fw.mgmt] Erasing
> > [fw.mgmt] Erasing done
> > [fw.mgmt] Flashing 100%
> > [fw.mgmt] Flashing done 100%
> > [fw.undi] Erasing
> > [fw.undi] Erasing done
> > [fw.undi] Flashing 100%
> > [fw.undi] Flashing done 100%
> > [fw.netlist] Erasing
> > [fw.netlist] Erasing done
> > [fw.netlist] Flashing 100%
> > [fw.netlist] Flashing done 100%
> > Activate new firmware by devlink reload
> > [root@host ~]# devlink dev reload pci/0000:ca:00.0 action
> > fw_activate
> > reload_actions_performed:
> > fw_activate
> > [root@host ~]# ip link show ens7f0
> > 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> > state UP mode DEFAULT group default qlen 1000
> > link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> > altname enp202s0f0
> >
>
> Ahh.. good find. I checked a bunch of places, but didn't check here
> for
> differences. :(
>
> For what its worth, I checked the source history of the out-of-tree
> driver this came from. It appears to be a workaround added for fixing
> a
> similar issue.
>
> I haven't been able to dig up the full details yet. It appeares to be
> a
> collision with firmware finalizing recovery after the EMP reset.
>
> Still trying to dig for any more information I can find.
Interesting time frame could be around this commit:
08771bce330036 ("ice: Continue probe on link/PHY errors")
Petr
>
> > Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation
> > via devlink reload")
> > Signed-off-by: Petr Oros <poros@redhat.com>
> > ---
> > drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/net/ethernet/intel/ice/ice_main.c
> > b/drivers/net/ethernet/intel/ice/ice_main.c
> > index d768925785ca79..90ea2203cdc763 100644
> > --- a/drivers/net/ethernet/intel/ice/ice_main.c
> > +++ b/drivers/net/ethernet/intel/ice/ice_main.c
> > @@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf,
> > enum ice_reset_req reset_type)
> >
> > dev_dbg(dev, "rebuilding PF after reset_type=%d\n",
> > reset_type);
> >
> > +#define ICE_EMP_RESET_SLEEP 5000
> > if (reset_type == ICE_RESET_EMPR) {
> > /* If an EMP reset has occurred, any previously
> > pending flash
> > * update will have completed. We no longer know
> > whether or
> > * not the NVM update EMP reset is restricted.
> > */
> > pf->fw_emp_reset_disabled = false;
> > +
> > + msleep(ICE_EMP_RESET_SLEEP);
> > }
> >
> > err = ice_init_all_ctrlq(hw);
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v2] ice: wait 5 s for EMP reset after firmware flash
2022-04-12 10:27 [PATCH] ice: wait for EMP reset after firmware flash Petr Oros
` (2 preceding siblings ...)
2022-04-12 16:58 ` Jacob Keller
@ 2022-04-13 15:37 ` Petr Oros
2022-04-14 12:16 ` [Intel-wired-lan] " G, GurucharanX
3 siblings, 1 reply; 9+ messages in thread
From: Petr Oros @ 2022-04-13 15:37 UTC (permalink / raw)
To: netdev
Cc: jesse.brandeburg, anthony.l.nguyen, davem, kuba, pabeni,
jacob.e.keller, intel-wired-lan, linux-kernel, ivecera, pmenzel,
alexandr.lobakin
We need to wait 5 s for EMP reset after firmware flash. Code was extracted
from OOT driver (ice v1.8.3 downloaded from sourceforge). Without this
wait, fw_activate let card in inconsistent state and recoverable only
by second flash/activate. Flash was tested on these fw's:
From -> To
3.00 -> 3.10/3.20
3.10 -> 3.00/3.20
3.20 -> 3.00/3.10
Reproducer:
[root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
reload_actions_performed:
fw_activate
[root@host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0
dmesg after flash:
[ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
[ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
[ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0
[ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
[ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
[ 55.647348] ice 0000:ca:00.0: PTP init successful
[ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
[ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
[ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
[ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
Reboot doesn’t help, only second flash/activate with OOT or patched
driver put card back in consistent state.
After patch:
[root@host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin
Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
reload_actions_performed:
fw_activate
[root@host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0
v2 changes:
- fixed format issues
- added info about fw and OOT driver versions
- added time in the commit message summary
- appended the unit to the macro name
Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")
Signed-off-by: Petr Oros <poros@redhat.com>
---
drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
index d768925785ca79..38825ed2ecd1de 100644
--- a/drivers/net/ethernet/intel/ice/ice_main.c
+++ b/drivers/net/ethernet/intel/ice/ice_main.c
@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);
+#define ICE_EMP_RESET_SLEEP_MS 5000
if (reset_type == ICE_RESET_EMPR) {
/* If an EMP reset has occurred, any previously pending flash
* update will have completed. We no longer know whether or
* not the NVM update EMP reset is restricted.
*/
pf->fw_emp_reset_disabled = false;
+
+ msleep(ICE_EMP_RESET_SLEEP_MS);
}
err = ice_init_all_ctrlq(hw);
--
2.35.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* RE: [Intel-wired-lan] [PATCH v2] ice: wait 5 s for EMP reset after firmware flash
2022-04-13 15:37 ` [PATCH v2] ice: wait 5 s " Petr Oros
@ 2022-04-14 12:16 ` G, GurucharanX
0 siblings, 0 replies; 9+ messages in thread
From: G, GurucharanX @ 2022-04-14 12:16 UTC (permalink / raw)
To: poros, netdev
Cc: ivecera, pmenzel, intel-wired-lan, linux-kernel, kuba, pabeni, davem
> -----Original Message-----
> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of
> Petr Oros
> Sent: Wednesday, April 13, 2022 9:08 PM
> To: netdev@vger.kernel.org
> Cc: ivecera <ivecera@redhat.com>; pmenzel@molgen.mpg.de; intel-wired-
> lan@lists.osuosl.org; linux-kernel@vger.kernel.org; kuba@kernel.org;
> pabeni@redhat.com; davem@davemloft.net
> Subject: [Intel-wired-lan] [PATCH v2] ice: wait 5 s for EMP reset after
> firmware flash
>
> We need to wait 5 s for EMP reset after firmware flash. Code was extracted
> from OOT driver (ice v1.8.3 downloaded from sourceforge). Without this
> wait, fw_activate let card in inconsistent state and recoverable only by
> second flash/activate. Flash was tested on these fw's:
> From -> To
> 3.00 -> 3.10/3.20
> 3.10 -> 3.00/3.20
> 3.20 -> 3.00/3.10
>
> Reproducer:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
> E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_80
> 00AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq
> state DOWN mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> dmesg after flash:
> [ 55.120788] ice: Copyright (c) 2018, Intel Corporation.
> [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5,
> continuing anyway
> [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE
> OS Default Package version 1.3.28.0
> [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed.
> [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5
> [ 55.647348] ice 0000:ca:00.0: PTP init successful
> [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number
> of TCs supported on this port are 8
> [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
> [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
> [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited
> by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0
> GT/s PCIe x16 link)
> Reboot doesn’t help, only second flash/activate with OOT or patched driver
> put card back in consistent state.
>
> After patch:
> [root@host ~]# devlink dev flash pci/0000:ca:00.0 file
> E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_80
> 00AD7B.bin
> Preparing to flash
> [fw.mgmt] Erasing
> [fw.mgmt] Erasing done
> [fw.mgmt] Flashing 100%
> [fw.mgmt] Flashing done 100%
> [fw.undi] Erasing
> [fw.undi] Erasing done
> [fw.undi] Flashing 100%
> [fw.undi] Flashing done 100%
> [fw.netlist] Erasing
> [fw.netlist] Erasing done
> [fw.netlist] Flashing 100%
> [fw.netlist] Flashing done 100%
> Activate new firmware by devlink reload
> [root@host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activate
> reload_actions_performed:
> fw_activate
> [root@host ~]# ip link show ens7f0
> 19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq
> state UP mode DEFAULT group default qlen 1000
> link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
> altname enp202s0f0
>
> v2 changes:
> - fixed format issues
> - added info about fw and OOT driver versions
> - added time in the commit message summary
> - appended the unit to the macro name
>
> Fixes: 399e27dbbd9e94 ("ice: support immediate firmware activation via
> devlink reload")
> Signed-off-by: Petr Oros <poros@redhat.com>
> ---
> drivers/net/ethernet/intel/ice/ice_main.c | 3 +++
> 1 file changed, 3 insertions(+)
>
Tested-by: Gurucharan <gurucharanx.g@intel.com> (A Contingent worker at Intel)
^ permalink raw reply [flat|nested] 9+ messages in thread