[RFC/PATCH] mwifiex: Driver - Firmware Glitches

[RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Florian Achleitner]

Hi,

we are using a Marvell SD8787-based module by Lesswire, which is connected to 
an imx28 through sdio. We are testing with the most recent kernel from
wireless-drivers-next and Firmware 14.66.35.p52.

Frequently, we see 'glitches' between driver and Firmware. The most recent 
driver answers most of them with an attempt to reset the module, reload the 
firmware, and start over. 

However, this only works for us, because our sdio interface triggers a 
hardware reset for the wireless module whenever it probes the device. Without 
this hardware reset (sdio reset only), the module fails to restart.

Is the necessity of frequent hardware resets a commonly known issue with this 
chip/firmware? Anybody else experiencing these?

Currently, we see three different scenarios. One of them is currently not 
answered by  reset. Refer to the upcoming patch.

(1)  mwifiex_cmd_timeout_func: Timeout cmd .. Ok, after reset.
(2)  Firmware wakeup failed.. Ok, after reset.
(3) DNLD_CMD: host to card failed. No reset triggered. See patch.

I can provide detailed logs, if you want.

Regards,
Florian

[PATCH] mwifiex: Trigger a card reset on "DNLD_CMD: host to card failed".

[Florian Achleitner]

Whenever this problem occurs, the driver can not continue. Thus,
trigger a card reset to restart the module firmware. Other firmware
communication issues are resolved by this last resort.
Also dump registers, to eventually allow some diagnostics.

Signed-off-by: Florian Achleitner <flo@fopen.at>
---

This works well at least for us.

 drivers/net/wireless/mwifiex/cmdevt.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/net/wireless/mwifiex/cmdevt.c b/drivers/net/wireless/mwifiex/cmdevt.c
index c5a14ff..410f4f2 100644
--- a/drivers/net/wireless/mwifiex/cmdevt.c
+++ b/drivers/net/wireless/mwifiex/cmdevt.c
@@ -233,6 +233,12 @@ static int mwifiex_dnld_cmd_to_fw(struct mwifiex_private *priv,
                spin_unlock_irqrestore(&adapter->mwifiex_cmd_lock, flags);
 
                adapter->dbg.num_cmd_host_to_card_failure++;
+
+               if (adapter->if_ops.fw_dump)
+                       adapter->if_ops.fw_dump(adapter);
+
+               if (adapter->if_ops.card_reset)
+                       adapter->if_ops.card_reset(adapter);
                return -1;
        }
 
-- 
2.1.0

RE: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Amitkumar Karwar]

Hi Florian,

> 
> Hi,
> 
> we are using a Marvell SD8787-based module by Lesswire, which is
> connected to an imx28 through sdio. We are testing with the most recent
> kernel from wireless-drivers-next and Firmware 14.66.35.p52.
> 
> Frequently, we see 'glitches' between driver and Firmware. The most
> recent driver answers most of them with an attempt to reset the module,
> reload the firmware, and start over.
> 
> However, this only works for us, because our sdio interface triggers a
> hardware reset for the wireless module whenever it probes the device.
> Without this hardware reset (sdio reset only), the module fails to
> restart.

We haven't seen this behavior yet. Could you please share complete log?

> 
> Is the necessity of frequent hardware resets a commonly known issue with
> this chip/firmware? Anybody else experiencing these?
> 
> Currently, we see three different scenarios. One of them is currently
> not answered by  reset. Refer to the upcoming patch.
> 
> (1)  mwifiex_cmd_timeout_func: Timeout cmd .. Ok, after reset.
> (2)  Firmware wakeup failed.. Ok, after reset.

This is ok. Reset at these places helps us recover if a corner case firmware bug is hit. 

> (3) DNLD_CMD: host to card failed. No reset triggered. See patch.
> 
> I can provide detailed logs, if you want.

"DNLD_CMD: host to card failed" error is expected during heavy Tx traffic if data write port is not available. Traffic will be resumed once the port is available.
Please share detailed logs. It would help us find root cause for the problem.

Regards,
Amitkumar

Re: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[James Cameron]

On Thu, Apr 16, 2015 at 11:10:02AM +0200, Florian Achleitner wrote:
> Is the necessity of frequent hardware resets a commonly known issue
> with this chip/firmware? Anybody else experiencing these?

Yes, but how frequent?

> Currently, we see three different scenarios. One of them is
> currently not answered by reset. Refer to the upcoming patch.
> 
> (1)  mwifiex_cmd_timeout_func: Timeout cmd .. Ok, after reset.

See this a lot during heavy testing.

> (2)  Firmware wakeup failed.. Ok, after reset.

Never see this.

> (3) DNLD_CMD: host to card failed. No reset triggered. See patch.

Very rarely see this.

However, our experience may not be comparable; we are using 8787 with
a 3.5 kernel, because we haven't the resources to use a later kernel
or get backports working.

Also, we use WOL (wake on lan) heavily; frequent automatic suspends,
with a GPIO wakeup in addition to the SDHCI.

-- 
James Cameron
http://quozl.linux.org.au/

Re: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Florian Achleitner]

On Thursday 16 April 2015 02:30:59 Amitkumar Karwar wrote:
>[...]
> We haven't seen this behavior yet. Could you please share complete log?
Logs follow per scenario.

> > (1)  mwifiex_cmd_timeout_func: Timeout cmd .. Ok, after reset.
[164276.211216] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1) failed: 
-110
[164276.219198] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2) failed: 
-110
[164286.247400] mwifiex_sdio mmc0:0001:1: mwifiex_cmd_timeout_func: Timeout cmd 
id = 0x107, act = 0x0
[164286.256501] mwifiex_sdio mmc0:0001:1: num_data_h2c_failure = 0
[164286.262593] mwifiex_sdio mmc0:0001:1: num_cmd_h2c_failure = 0
[164286.268537] mwifiex_sdio mmc0:0001:1: is_cmd_timedout = 1
[164286.274088] mwifiex_sdio mmc0:0001:1: num_tx_timeout = 0
[164286.279565] mwifiex_sdio mmc0:0001:1: last_cmd_index = 0
[164286.285025] mwifiex_sdio mmc0:0001:1: last_cmd_id: 07 01 07 01 07 01 07 01 
07 01
[164286.292587] mwifiex_sdio mmc0:0001:1: last_cmd_act: 00 00 00 00 00 00 00 00 
00 00
[164286.300245] mwifiex_sdio mmc0:0001:1: last_cmd_resp_index = 4
[164286.306101] mwifiex_sdio mmc0:0001:1: last_cmd_resp_id: 07 81 07 81 07 81 
07 81 07 81
[164286.314097] mwifiex_sdio mmc0:0001:1: last_event_index = 3
[164286.319757] mwifiex_sdio mmc0:0001:1: last_event: 58 00 58 00 58 00 0b 00 
58 00
[164286.327180] mwifiex_sdio mmc0:0001:1: data_sent=0 cmd_sent=1
[164286.333026] mwifiex_sdio mmc0:0001:1: ps_mode=1 ps_state=0
[164286.338698] mwifiex_sdio mmc0:0001:1: cmd timeout
[164286.347116] mwifiex_sdio mmc0:0001:1: === DRIVER INFO DUMP START===
[164286.348307] mwifiex_sdio mmc0:0001:1: SDIO register DUMP START
[164286.364842] mwifiex_sdio mmc0:0001:1: SDIO Func0 (0x0-0x9): 32 02 02 00 03 
02 00 02 03 00
[164286.375346] mwifiex_sdio mmc0:0001:1: SDIO Func1 (0x0-0x9): 00 3f 03 03 01 
00 ff ff 0a 00
[164286.375995] mwifiex_sdio mmc0:0001:1: SDIO Func1: (0x28) 00 (0x30) 00 
(0x34) 07 (0x38) 00 (0x3c) 00
[164286.376881] mwifiex_sdio mmc0:0001:1: SDIO Func1 (0x60-0x6a): dc fe ed 00 
1d 00 00 ec ec 00 20
[164286.671632] mwifiex_sdio mmc0:0001:1: SDIO Func1 (0x60-0x6a): dc fe 0b 00 
3b 00 00 ec ec 00 20
[164286.671788] mwifiex_sdio mmc0:0001:1: SDIO register DUMP END
[164286.672062] mwifiex_sdio mmc0:0001:1: === DRIVER INFO DUMP END===
[164286.672087] mwifiex_sdio: Resetting card...
[164286.755091] mmc0: card 0001 removed
[164287.957529] mxs-mmc 80010000.ssp: Device reset triggered!
[164288.628044] mmc0: new high speed SDIO card at address 0001
[164289.750729] mwifiex_sdio mmc0:0001:1: info: FW download over, size 447384 
bytes
[164289.977539] mwifiex_sdio mmc0:0001:1: WLAN FW is active
[164290.445054] ieee80211 phy9: ignoring F/W country code US
[164290.483702] mwifiex_sdio mmc0:0001:1: driver_version = mwifiex 1.0 
(14.66.9.p96)
[165634.314756] cfg80211: Calling CRDA to update world regulatory domain

> > (2)  Firmware wakeup failed.. Ok, after reset.
[230837.497001] mxs-mmc 80010000.ssp: Device reset triggered!
[230838.189850] mmc0: new high speed SDIO card at address 0001
[230840.415197] mwifiex_sdio mmc0:0001:1: info: FW download over, size 447384 
bytes
[230840.656847] mwifiex_sdio mmc0:0001:1: WLAN FW is active
[230841.111489] ieee80211 phy21: ignoring F/W country code US
[230841.169867] mwifiex_sdio mmc0:0001:1: driver_version = mwifiex 1.0 
(14.66.9.p96) 
[231000.116588] mwifiex_sdio mmc0:0001:1: Firmware wakeup failed
[231000.122550] mwifiex_sdio mmc0:0001:1: failed to get signal information
[231000.129544] mwifiex_sdio: Resetting card...

> > (3) DNLD_CMD: host to card failed. No reset triggered. See patch.
Without patch:
134669.799365] cfg80211: Calling CRDA to update world regulatory domain
[141183.720924] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1) failed: 
-110
[141183.721405] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2) failed: 
-110
[141183.721893] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3) failed: 
-110
[141183.722188] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
[178868.935885] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1) failed: 
-84
[178868.936295] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2) failed: 
-84
[178868.936644] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3) failed: 
-84
[178868.936874] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
[178868.947200] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1) failed: 
-110
[178868.947739] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2) failed: 
-110
[178868.948153] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3) failed: 
-110
[178868.948451] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
[178906.974145] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1) failed: 
-84
[178906.974544] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2) failed: 
-84
[178906.974890] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3) failed: 
-84
[178906.975122] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
[178906.985090] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1) failed: 
-110
[178906.985565] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2) failed: 
-110
[178906.986036] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3) failed: 
-110
[178906.986332] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
[178930.845685] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1) failed: 
-84
[178930.846069] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2) failed: 
-84
[178930.846408] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3) failed: 
-84
[178930.846638] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed

The interface does not work in this state. Note the long times between some 
messages.

With patch:

[50410.924229] cfg80211: Calling CRDA to update world regulatory domain
[69944.419055] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1) failed: 
-110
[69944.419443] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2) failed: 
-110
[69944.419787] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3) failed: 
-110
[69944.420017] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
[69944.420199] mwifiex_sdio mmc0:0001:1: === DRIVER INFO DUMP START===
[69944.421173] mwifiex_sdio mmc0:0001:1: SDIO register DUMP START
[69944.423444] mwifiex_sdio mmc0:0001:1: SDIO Func0 (0x0-0x9): 32 02 00 02 03 
00 00 02 03 00 
[69944.425194] mwifiex_sdio mmc0:0001:1: SDIO Func1 (0x0-0x9): 00 3f 03 00 00 
00 03 fc 0a 00 
[69944.425571] mwifiex_sdio mmc0:0001:1: SDIO Func1: (0x28) 00 (0x30) 08 (0x34) 
07 (0x38) 00 (0x3c) 00 
[69944.426270] mwifiex_sdio mmc0:0001:1: SDIO Func1 (0x60-0x6a): dc fe 08 00 d8 
00 00 80 80 00 20 
[69944.646744] mwifiex_sdio mmc0:0001:1: SDIO Func1 (0x60-0x6a): dc fe 24 00 ef 
00 00 80 80 00 20 
[69944.646823] mwifiex_sdio mmc0:0001:1: SDIO register DUMP END
[69944.649330] mwifiex_sdio mmc0:0001:1: === DRIVER INFO DUMP END===
[69944.649372] mwifiex_sdio: Resetting card...
[69944.719068] mmc0: card 0001 removed
[69946.177879] mxs-mmc 80010000.ssp: Device reset triggered!
[69946.854642] mmc0: new high speed SDIO card at address 0001
[69948.300017] mwifiex_sdio mmc0:0001:1: info: FW download over, size 463240 
bytes

> "DNLD_CMD: host to card failed" error is expected during heavy Tx traffic if
> data write port is not available. Traffic will be resumed once the port is
> available. Please share detailed logs. It would help us find root cause for
> the problem.

In our case it seems to fail to resume somehow.

> 
> Regards,
> Amitkumar

Regards,
Florian

Re: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Florian Achleitner]

On Thursday 16 April 2015 20:19:13 James Cameron wrote:
> On Thu, Apr 16, 2015 at 11:10:02AM +0200, Florian Achleitner wrote:
> > Is the necessity of frequent hardware resets a commonly known issue
> > with this chip/firmware? Anybody else experiencing these?
> 
> Yes, but how frequent?

Depends on the specific hardware. Averages follow.

> 
> > Currently, we see three different scenarios. One of them is
> > currently not answered by reset. Refer to the upcoming patch.
> > 
> > (1)  mwifiex_cmd_timeout_func: Timeout cmd .. Ok, after reset.
> 
> See this a lot during heavy testing.

Also a lot, maybe once a day in low load.

> 
> > (2)  Firmware wakeup failed.. Ok, after reset.
> 
> Never see this.

Once a day.

> 
> > (3) DNLD_CMD: host to card failed. No reset triggered. See patch.
> 
> Very rarely see this.

Also very rarely. Once a week, probably.

> 
> However, our experience may not be comparable; we are using 8787 with
> a 3.5 kernel, because we haven't the resources to use a later kernel
> or get backports working.
> 
> Also, we use WOL (wake on lan) heavily; frequent automatic suspends,
> with a GPIO wakeup in addition to the SDHCI.

We don't use WOL or suspends.

Regards,
Florian

RE: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Avinash Patil]

Hi Florian,


> -----Original Message-----
> From: Florian Achleitner [mailto:flo@fopen.at]
> Sent: Thursday, April 16, 2015 4:13 PM
> To: James Cameron
> Cc: linux-wireless@vger.kernel.org; Amitkumar Karwar; Avinash Patil; Maithili
> Hinge
> Subject: Re: [RFC/PATCH] mwifiex: Driver - Firmware Glitches
> 
> On Thursday 16 April 2015 20:19:13 James Cameron wrote:
> > On Thu, Apr 16, 2015 at 11:10:02AM +0200, Florian Achleitner wrote:
> > > Is the necessity of frequent hardware resets a commonly known issue
> > > with this chip/firmware? Anybody else experiencing these?
> >
> > Yes, but how frequent?
> 
> Depends on the specific hardware. Averages follow.
> 
> >
> > > Currently, we see three different scenarios. One of them is
> > > currently not answered by reset. Refer to the upcoming patch.
> > >
> > > (1)  mwifiex_cmd_timeout_func: Timeout cmd .. Ok, after reset.
> >
> > See this a lot during heavy testing.
> 
> Also a lot, maybe once a day in low load.

Can this be related to SDHCI or MMC host controller?
Could you please collect logs with CONFIG_MMC_DEBUG enbled?

> >
> > > (2)  Firmware wakeup failed.. Ok, after reset.
> >
> > Never see this.
> 
> Once a day.
> 
> >
> > > (3) DNLD_CMD: host to card failed. No reset triggered. See patch.
> >
> > Very rarely see this.
> 
> Also very rarely. Once a week, probably.
> 
> >
> > However, our experience may not be comparable; we are using 8787 with
> > a 3.5 kernel, because we haven't the resources to use a later kernel
> > or get backports working.
> >
> > Also, we use WOL (wake on lan) heavily; frequent automatic suspends,
> > with a GPIO wakeup in addition to the SDHCI.
> 
> We don't use WOL or suspends.
> 
> Regards,
> Florian

RE: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Amitkumar Karwar]

Hi Florian,

> On Thursday 16 April 2015 02:30:59 Amitkumar Karwar wrote:
> >[...]
> > We haven't seen this behavior yet. Could you please share complete
> log?
> Logs follow per scenario.
> 
> > > (1)  mwifiex_cmd_timeout_func: Timeout cmd .. Ok, after reset.
> [164276.211216] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1)
> failed:
> -110
> [164276.219198] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2)
> failed:
> -110
> [164286.247400] mwifiex_sdio mmc0:0001:1: mwifiex_cmd_timeout_func:
> Timeout cmd id = 0x107, act = 0x0 [164286.256501] mwifiex_sdio
> mmc0:0001:1: num_data_h2c_failure = 0 [164286.262593] mwifiex_sdio
> mmc0:0001:1: num_cmd_h2c_failure = 0 [164286.268537] mwifiex_sdio
> mmc0:0001:1: is_cmd_timedout = 1 [164286.274088] mwifiex_sdio
> mmc0:0001:1: num_tx_timeout = 0 [164286.279565] mwifiex_sdio
> mmc0:0001:1: last_cmd_index = 0 [164286.285025] mwifiex_sdio
> mmc0:0001:1: last_cmd_id: 07 01 07 01 07 01 07 01
> 07 01
> [164286.292587] mwifiex_sdio mmc0:0001:1: last_cmd_act: 00 00 00 00 00
> 00 00 00
> 00 00
> [164286.300245] mwifiex_sdio mmc0:0001:1: last_cmd_resp_index = 4
> [164286.306101] mwifiex_sdio mmc0:0001:1: last_cmd_resp_id: 07 81 07 81
> 07 81
> 07 81 07 81
> [164286.314097] mwifiex_sdio mmc0:0001:1: last_event_index = 3
> [164286.319757] mwifiex_sdio mmc0:0001:1: last_event: 58 00 58 00 58 00
> 0b 00
> 58 00
> [164286.327180] mwifiex_sdio mmc0:0001:1: data_sent=0 cmd_sent=1
> [164286.333026] mwifiex_sdio mmc0:0001:1: ps_mode=1 ps_state=0
> [164286.338698] mwifiex_sdio mmc0:0001:1: cmd timeout [164286.347116]
> mwifiex_sdio mmc0:0001:1: === DRIVER INFO DUMP START=== [164286.348307]
> mwifiex_sdio mmc0:0001:1: SDIO register DUMP START [164286.364842]
> mwifiex_sdio mmc0:0001:1: SDIO Func0 (0x0-0x9): 32 02 02 00 03
> 02 00 02 03 00
> [164286.375346] mwifiex_sdio mmc0:0001:1: SDIO Func1 (0x0-0x9): 00 3f 03
> 03 01
> 00 ff ff 0a 00
> [164286.375995] mwifiex_sdio mmc0:0001:1: SDIO Func1: (0x28) 00 (0x30)
> 00
> (0x34) 07 (0x38) 00 (0x3c) 00
> [164286.376881] mwifiex_sdio mmc0:0001:1: SDIO Func1 (0x60-0x6a): dc fe
> ed 00 1d 00 00 ec ec 00 20 [164286.671632] mwifiex_sdio mmc0:0001:1:
> SDIO Func1 (0x60-0x6a): dc fe 0b 00 3b 00 00 ec ec 00 20 [164286.671788]
> mwifiex_sdio mmc0:0001:1: SDIO register DUMP END [164286.672062]
> mwifiex_sdio mmc0:0001:1: === DRIVER INFO DUMP END=== [164286.672087]
> mwifiex_sdio: Resetting card...
> [164286.755091] mmc0: card 0001 removed
> [164287.957529] mxs-mmc 80010000.ssp: Device reset triggered!
> [164288.628044] mmc0: new high speed SDIO card at address 0001
> [164289.750729] mwifiex_sdio mmc0:0001:1: info: FW download over, size
> 447384 bytes [164289.977539] mwifiex_sdio mmc0:0001:1: WLAN FW is active
> [164290.445054] ieee80211 phy9: ignoring F/W country code US
> [164290.483702] mwifiex_sdio mmc0:0001:1: driver_version = mwifiex 1.0
> (14.66.9.p96)
> [165634.314756] cfg80211: Calling CRDA to update world regulatory domain
> 
> > > (2)  Firmware wakeup failed.. Ok, after reset.
> [230837.497001] mxs-mmc 80010000.ssp: Device reset triggered!
> [230838.189850] mmc0: new high speed SDIO card at address 0001
> [230840.415197] mwifiex_sdio mmc0:0001:1: info: FW download over, size
> 447384 bytes [230840.656847] mwifiex_sdio mmc0:0001:1: WLAN FW is active
> [230841.111489] ieee80211 phy21: ignoring F/W country code US
> [230841.169867] mwifiex_sdio mmc0:0001:1: driver_version = mwifiex 1.0
> (14.66.9.p96)
> [231000.116588] mwifiex_sdio mmc0:0001:1: Firmware wakeup failed
> [231000.122550] mwifiex_sdio mmc0:0001:1: failed to get signal
> information [231000.129544] mwifiex_sdio: Resetting card...
> 
> > > (3) DNLD_CMD: host to card failed. No reset triggered. See patch.
> Without patch:
> 134669.799365] cfg80211: Calling CRDA to update world regulatory domain
> [141183.720924] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1)
> failed:
> -110
> [141183.721405] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2)
> failed:
> -110
> [141183.721893] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3)
> failed:
> -110
> [141183.722188] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
> [178868.935885] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1)
> failed:
> -84
> [178868.936295] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2)
> failed:
> -84
> [178868.936644] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3)
> failed:
> -84
> [178868.936874] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
> [178868.947200] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1)
> failed:
> -110
> [178868.947739] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2)
> failed:
> -110
> [178868.948153] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3)
> failed:
> -110
> [178868.948451] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
> [178906.974145] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1)
> failed:
> -84
> [178906.974544] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2)
> failed:
> -84
> [178906.974890] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3)
> failed:
> -84
> [178906.975122] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
> [178906.985090] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1)
> failed:
> -110
> [178906.985565] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2)
> failed:
> -110
> [178906.986036] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3)
> failed:
> -110
> [178906.986332] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
> [178930.845685] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (1)
> failed:
> -84
> [178930.846069] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (2)
> failed:
> -84
> [178930.846408] mwifiex_sdio mmc0:0001:1: host_to_card, write iomem (3)
> failed:
> -84
> [178930.846638] mwifiex_sdio mmc0:0001:1: DNLD_CMD: host to card failed
> 
> The interface does not work in this state. Note the long times between
> some messages.


"host_to_card, write iomem (1) failed: -110" error indicates that MMC subsystem's sdio_writesb() API failed with timeout error.

WLAN driver or FW has nothing to do with this. This seems to be your MMC host controller/hardware problem. Adding a recovering mechanism in WLAN driver doesn't look correct here. We should rather need to root cause the problem.
Our "DNLD_CMD: host to card failed" error message is displayed for other failures also. As explained earlier, if it's write data port not available problem. It will automatically resolve after sometimes. Triggering reset would be unnecessary for that case.

Regards,
Amitkumar

Re: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Florian Achleitner]

Hi Avanish, Amitkumar,

Thx for your ideas to check the mmc interface.

On Thursday 16 April 2015 04:21:50 Amitkumar Karwar wrote:
> "host_to_card, write iomem (1) failed: -110" error indicates that MMC
> subsystem's sdio_writesb() API failed with timeout error.
> 
> WLAN driver or FW has nothing to do with this. This seems to be your MMC
> host controller/hardware problem. Adding a recovering mechanism in WLAN
> driver doesn't look correct here. We should rather need to root cause the
> problem. Our "DNLD_CMD: host to card failed" error message is displayed for
> other failures also. As explained earlier, if it's write data port not
> available problem. It will automatically resolve after sometimes.
> Triggering reset would be unnecessary for that case.

I will try to figure out whether the root cause is the MMC interface.
Probably in this case, the reset actually recovers from a bug in the mmc 
driver.

> 
> Regards,
> Amitkumar

Regards,
Florian

Re: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Florian Achleitner]

On Thursday 16 April 2015 15:19:27 Florian Achleitner wrote:
> I will try to figure out whether the root cause is the MMC interface.
> Probably in this case, the reset actually recovers from a bug in the mmc 
> driver.

Anyways, we have a virtually identical hardware with a Marvell SD8688 based 
wireless module, which uses the libertas driver. Running kernel 3.10.41 we 
have no problems for 50 days. 

Running the same kernel with mwifiex with the SD8788 we see a freeze which 
requires a hw-reset within about a day. Similar to the most recent kernel.

Are there known sdio issues with SD8788?
Will try to get useful information from the mmc system..

Regards,
Florian

RE: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Amitkumar Karwar]

Hi Florian,

> From: Florian Achleitner [mailto:achleitner.florian@fronius.com]
> Sent: Friday, April 17, 2015 2:00 PM
> To: Amitkumar Karwar
> Cc: Avinash Patil; linux-wireless@vger.kernel.org; Maithili Hinge
> Subject: Re: [RFC/PATCH] mwifiex: Driver - Firmware Glitches
> 
> On Thursday 16 April 2015 15:19:27 Florian Achleitner wrote:
> > I will try to figure out whether the root cause is the MMC interface.
> > Probably in this case, the reset actually recovers from a bug in the
> > mmc driver.
> 
> Anyways, we have a virtually identical hardware with a Marvell SD8688
> based wireless module, which uses the libertas driver. Running kernel
> 3.10.41 we have no problems for 50 days.
> 
> Running the same kernel with mwifiex with the SD8788 we see a freeze
> which requires a hw-reset within about a day. Similar to the most recent
> kernel.
> 
> Are there known sdio issues with SD8788?
> Will try to get useful information from the mmc system..

Known firmware issues are command timeout and wakeup failure. They are not specific to SDIO. As Avinash mentioned earlier, we need to check CONFIG_MMC_DEBUG enbled dmesg logs for "write iomem (1) failed: -110" issue.

Regards,
Amitkumar

Re: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Florian Achleitner]

On Friday 17 April 2015 02:45:02 Amitkumar Karwar wrote:
> Known firmware issues are command timeout and wakeup failure. They are not
> specific to SDIO. As Avinash mentioned earlier, we need to check
> CONFIG_MMC_DEBUG enbled dmesg logs for "write iomem (1) failed: -110"
> issue.

Thx.
A kernel with CONFIG_MMC_DEBUG is  running since 6 days on 4 devices, but 
nothing valuable captured yet. Unfortunately, it looks like the debugging code 
modifies the timing, such that the failure doesn't occur.. 

> 
> Regards,
> Amitkumar

Regards, 
Flo

Re: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[James Cameron]

On Thu, Apr 23, 2015 at 10:33:02AM +0200, Florian Achleitner wrote:
> On Friday 17 April 2015 02:45:02 Amitkumar Karwar wrote:
> > Known firmware issues are command timeout and wakeup failure. They
> > are not specific to SDIO. As Avinash mentioned earlier, we need to
> > check CONFIG_MMC_DEBUG enbled dmesg logs for "write iomem (1)
> > failed: -110" issue.
> 
> Thx.
> A kernel with CONFIG_MMC_DEBUG is running since 6 days on 4 devices,
> but nothing valuable captured yet. Unfortunately, it looks like the
> debugging code modifies the timing, such that the failure doesn't
> occur..

Yes, I've seen something similar last year, when I began to add
instrumentation code the problems became less reproducible.

So in my situation either there was a race condition with the
firmware, or a signal integrity problem on the SDIO.

I wasn't able to go into full signal integrity work at the time.

I also have rather hazy memory of which particular problem I was
working on.  ;-)

Perhaps you might review the effects of CONFIG_MMC_DEBUG and remove
some of it?

-- 
James Cameron
http://quozl.linux.org.au/

RE: [RFC/PATCH] mwifiex: Driver - Firmware Glitches

[Avinash Patil]

Hi Florian,

> -----Original Message-----
> From: quozl@laptop.org [mailto:quozl@laptop.org]
> Sent: Thursday, April 23, 2015 2:13 PM
> To: Florian Achleitner
> Cc: Amitkumar Karwar; Avinash Patil; linux-wireless@vger.kernel.org; Maithili
> Hinge
> Subject: Re: [RFC/PATCH] mwifiex: Driver - Firmware Glitches
> 
> On Thu, Apr 23, 2015 at 10:33:02AM +0200, Florian Achleitner wrote:
> > On Friday 17 April 2015 02:45:02 Amitkumar Karwar wrote:
> > > Known firmware issues are command timeout and wakeup failure. They
> > > are not specific to SDIO. As Avinash mentioned earlier, we need to
> > > check CONFIG_MMC_DEBUG enbled dmesg logs for "write iomem (1)
> > > failed: -110" issue.
> >
> > Thx.
> > A kernel with CONFIG_MMC_DEBUG is running since 6 days on 4 devices,
> > but nothing valuable captured yet. Unfortunately, it looks like the
> > debugging code modifies the timing, such that the failure doesn't
> > occur..
> 
> Yes, I've seen something similar last year, when I began to add
> instrumentation code the problems became less reproducible.
> 
> So in my situation either there was a race condition with the firmware, or a
> signal integrity problem on the SDIO.
> 
> I wasn't able to go into full signal integrity work at the time.
> 
> I also have rather hazy memory of which particular problem I was working on.
> ;-)
> 
> Perhaps you might review the effects of CONFIG_MMC_DEBUG and remove some of
> it?

As James suggested could you please check if reducing MMC_DEBUG frequency and printks only at error conditions helps?

> --
> James Cameron
> http://quozl.linux.org.au/

Re: [PATCH] mwifiex: Trigger a card reset on "DNLD_CMD: host to card failed".

[Kalle Valo]

Florian Achleitner <flo@fopen.at> writes:

> Whenever this problem occurs, the driver can not continue. Thus,
> trigger a card reset to restart the module firmware. Other firmware
> communication issues are resolved by this last resort.
> Also dump registers, to eventually allow some diagnostics.
>
> Signed-off-by: Florian Achleitner <flo@fopen.at>

Doesn't apply to wireless-drivers-next.git. Can you rebase, please?

-- 
Kalle Valo

Re: [PATCH] mwifiex: Trigger a card reset on "DNLD_CMD: host to card failed".

[Florian Achleitner]

On Saturday 09 May 2015 16:35:03 Kalle Valo wrote:
> Florian Achleitner <flo@fopen.at> writes:
> > Whenever this problem occurs, the driver can not continue. Thus,
> > trigger a card reset to restart the module firmware. Other firmware
> > communication issues are resolved by this last resort.
> > Also dump registers, to eventually allow some diagnostics.
> > 
> > Signed-off-by: Florian Achleitner <flo@fopen.at>
> 
> Doesn't apply to wireless-drivers-next.git. Can you rebase, please?

Please drop my patch! We agreed that it likely tries to fix a problem actually 
caused in MMC system.

Florian

Re: [PATCH] mwifiex: Trigger a card reset on "DNLD_CMD: host to card failed".

[Kalle Valo]

Florian Achleitner <flo@fopen.at> writes:

> On Saturday 09 May 2015 16:35:03 Kalle Valo wrote:
>> Florian Achleitner <flo@fopen.at> writes:
>> > Whenever this problem occurs, the driver can not continue. Thus,
>> > trigger a card reset to restart the module firmware. Other firmware
>> > communication issues are resolved by this last resort.
>> > Also dump registers, to eventually allow some diagnostics.
>> > 
>> > Signed-off-by: Florian Achleitner <flo@fopen.at>
>> 
>> Doesn't apply to wireless-drivers-next.git. Can you rebase, please?
>
> Please drop my patch! We agreed that it likely tries to fix a problem actually 
> caused in MMC system.

Sorry, I had missed that as patchwork didn't contain anything. In the
future it's a good idea to reply to the patch and say something like
"Kalle, please drop this because of foo". That way I see it in
patchwork:

https://patchwork.kernel.org/patch/6224961/

-- 
Kalle Valo