Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

[Ellie Reeves]

Hello,
I noticed the Marvell Mwifiex wireless driver ended up broken since 
kernel 4.13.0, and it is still the case with 4.14.0 on the Marvell 
ESPRESSOBin, soled by Globalscale Technology. I bought the official 
wireless card from them at the same time as the board. However since 
kernel 4.13.0, wifi as provided via mwifiex_pcie module stopped working. 
I don't have much info on the userspace side, as all that happens is the 
wireless interface is never brought up and stays in 'NO-CARRIER'  state 
according to 'ip a'. However on kernel side, I hope I have some info 
that are more useful ! :) This was logged via systemd-journald with 
debug turned on in mwifiex driver. I am willing to try anything to get 
this kernel working, as it's a lts kernel contrary to 4.12.12 on which 
I'm forced to run for now.

https://gist.githubusercontent.com/mk-fg/27c2505fd3bffcf0af87777aaafb71fe/raw/fda70ff9f8eaab248a104e03922323cf2f4754bf/tmp.txt

I hope these info will be useful in debugging. I'm new to all these 
kernel stuff, so I may have posted to the wrong place or in a wrong way, 
if so, please let me know.

Thanks in advance

-- 
Ellie Reeves

RE: [EXT] Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

[Ganapathi Bhat]

SGkgRWxpZSwNCg0KPiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+IEhlbGxvLA0KPiBJIG5vdGljZWQgdGhlIE1h
cnZlbGwgTXdpZmlleCB3aXJlbGVzcyBkcml2ZXIgZW5kZWQgdXAgYnJva2VuIHNpbmNlIGtlcm5l
bA0KPiA0LjEzLjAsIGFuZCBpdCBpcyBzdGlsbCB0aGUgY2FzZSB3aXRoIDQuMTQuMCBvbiB0aGUg
TWFydmVsbCBFU1BSRVNTT0Jpbiwgc29sZWQgYnkNCj4gR2xvYmFsc2NhbGUgVGVjaG5vbG9neS4g
SSBib3VnaHQgdGhlIG9mZmljaWFsIHdpcmVsZXNzIGNhcmQgZnJvbSB0aGVtIGF0IHRoZQ0KPiBz
YW1lIHRpbWUgYXMgdGhlIGJvYXJkLiBIb3dldmVyIHNpbmNlIGtlcm5lbCA0LjEzLjAsIHdpZmkg
YXMgcHJvdmlkZWQgdmlhDQo+IG13aWZpZXhfcGNpZSBtb2R1bGUgc3RvcHBlZCB3b3JraW5nLg0K
PiBJIGRvbid0IGhhdmUgbXVjaCBpbmZvIG9uIHRoZSB1c2Vyc3BhY2Ugc2lkZSwgYXMgYWxsIHRo
YXQgaGFwcGVucyBpcyB0aGUNCj4gd2lyZWxlc3MgaW50ZXJmYWNlIGlzIG5ldmVyIGJyb3VnaHQg
dXAgYW5kIHN0YXlzIGluICdOTy1DQVJSSUVSJz8gc3RhdGUNCj4gYWNjb3JkaW5nIHRvICdpcCBh
Jy4gSG93ZXZlciBvbiBrZXJuZWwgc2lkZSwgSSBob3BlIEkgaGF2ZSBzb21lIGluZm8gdGhhdCBh
cmUNCj4gbW9yZSB1c2VmdWwgISA6KSBUaGlzIHdhcyBsb2dnZWQgdmlhIHN5c3RlbWQtam91cm5h
bGQgd2l0aCBkZWJ1ZyB0dXJuZWQgb24NCj4gaW4gbXdpZmlleCBkcml2ZXIuIEkgYW0gd2lsbGlu
ZyB0byB0cnkgYW55dGhpbmcgdG8gZ2V0IHRoaXMga2VybmVsIHdvcmtpbmcsIGFzIGl0J3MNCj4g
YSBsdHMga2VybmVsIGNvbnRyYXJ5IHRvIDQuMTIuMTIgb24gd2hpY2ggSSdtIGZvcmNlZCB0byBy
dW4gZm9yIG5vdy4NCj4NCj4gaHR0cHM6Ly9naXN0LmdpdGh1YnVzZXJjb250ZW50LmNvbS9tay0N
Cj4gZmcvMjdjMjUwNWZkM2JmZmNmMGFmODc3NzdhYWFmYjcxZmUvcmF3L2ZkYTcwZmY5ZjhlYWFi
MjQ4YTEwNGUwMzkyMjMyDQo+IDNjZjJmNDc1NGJmL3RtcC50eHQNCj4NCj4gSSBob3BlIHRoZXNl
IGluZm8gd2lsbCBiZSB1c2VmdWwgaW4gZGVidWdnaW5nLiBJJ20gbmV3IHRvIGFsbCB0aGVzZSBr
ZXJuZWwNCj4gc3R1ZmYsIHNvIEkgbWF5IGhhdmUgcG9zdGVkIHRvIHRoZSB3cm9uZyBwbGFjZSBv
ciBpbiBhIHdyb25nIHdheSwgaWYgc28sDQo+IHBsZWFzZSBsZXQgbWUga25vdy4NCj4NCj4gVGhh
bmtzIGluIGFkdmFuY2UNCj4NCg0KVGhhbmtzIGZvciB0aGUgbm90aWZpY2F0aW9uIGFuZCBsb2dz
LiAgV2UgaGF2ZSBhIGZpeCBmb3IgdGhpcyBpc3N1ZSwgd2hpY2ggd2lsbCBiZSBzZW5kIHVwc3Ry
ZWFtIHNvb24uDQoNCj4gLS0NCj4gRWxsaWUgUmVldmVzDQoNCg0KUmVnYXJkcywNCkdhbmFwYXRo
aQ0K

Re: [EXT] Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

[Ellie Reeves]

Hi Ganapathi,
thank you a lot for the quick answer :). Would it be possible for you to 
send me the patch to fix this, so I could patch the kernel myself ? I'd 
really appreciate that, and also would like to know a bit more about the 
issue, yes, I'm curious. :) I like learning things.

-- 
Ellie Reeves

Re: [EXT] Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

[Ellie Reeves]

Hello,
is the recent patch for mwifiex the one Ganapathi was talking about, to 
fix mwifiex on espressobin ? If so, I tried it (in v1 I admit), without 
no real change. The interface is no longer shown as 'no-carrier' but 
there is still the timeout, and all the messages about fw being in bad 
state. :/ Sorry if I confused the patch with something unrelated to the 
espressobin.

-- 
Ellie Reeves

RE: [EXT] Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

[Ganapathi Bhat]

SGkgRWxsaWUsDQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogRWxsaWUg
UmVldmVzIFttYWlsdG86ZWxsaWVyZXZ2ZXNAZ21haWwuY29tXQ0KPiBTZW50OiBXZWRuZXNkYXks
IE5vdmVtYmVyIDIyLCAyMDE3IDM6NDMgUE0NCj4gVG86IEdhbmFwYXRoaSBCaGF0DQo+IENjOiBs
aW51eC13aXJlbGVzc0B2Z2VyLmtlcm5lbC5vcmc7IGFtaXRrYXJ3YXJAZ21haWwuY29tOyBOaXNo
YW50DQo+IFNhcm11a2FkYW07IFhpbm1pbmcgSHUNCj4gU3ViamVjdDogUmU6IFtFWFRdIE1hcnZl
bGwgTXdpZmlleCB3aXJlbGVzcyBkcml2ZXI6IG13aWZpZXhfcGNpZSBicm9rZW4gb24NCj4gTWFy
dmVsbCBFU1BSRVNTT0Jpbg0KPg0KPiBIZWxsbywNCj4gaXMgdGhlIHJlY2VudCBwYXRjaCBmb3Ig
bXdpZmlleCB0aGUgb25lIEdhbmFwYXRoaSB3YXMgdGFsa2luZyBhYm91dCwgdG8gZml4DQo+IG13
aWZpZXggb24gZXNwcmVzc29iaW4gPw0KDQpZZXMNCg0KPiBJZiBzbywgSSB0cmllZCBpdCAoaW4g
djEgSSBhZG1pdCksIHdpdGhvdXQgbm8gcmVhbA0KPiBjaGFuZ2UuDQpDYW4geW91IGp1c3Qgc2hh
cmUgdGhlIGRtc2cgbG9ncz8gQWxzbyBsZXQgdXMga25vdyBpZiB0aGVyZSBhcmUgYW55IHNwZWNp
ZmljIHN0ZXBzIHlvdSBmb2xsb3cuDQoNCj4gVGhlIGludGVyZmFjZSBpcyBubyBsb25nZXIgc2hv
d24gYXMgJ25vLWNhcnJpZXInIGJ1dCB0aGVyZSBpcyBzdGlsbCB0aGUNCj4gdGltZW91dCwgYW5k
IGFsbCB0aGUgbWVzc2FnZXMgYWJvdXQgZncgYmVpbmcgaW4gYmFkIHN0YXRlLiA6LyBTb3JyeSBp
ZiBJDQo+IGNvbmZ1c2VkIHRoZSBwYXRjaCB3aXRoIHNvbWV0aGluZyB1bnJlbGF0ZWQgdG8gdGhl
IGVzcHJlc3NvYmluLg0KPg0KPiAtLQ0KPiBFbGxpZSBSZWV2ZXMNCg0KVGhhbmtzLA0KR2FuYXBh
dGhpDQo=

Re: [EXT] Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

[Ellie Reeves]

Hello,
thanks for the quick answer. To answer Ganapathi's question, all I have 
to do in order to trigger the issue is to:

1. Upgrade kernel as usual.
2. Reboot and see the wifi going insane.

If I want to get it working properly I have to:

1. Downgrade kernel to 4.12.12.
2. Make sure initramfs and the uImage for u-boot have been generated for 
the 4.12.12 kernel again (my distrubution has a script that takes care 
of this).
3. Perform a "shutdown now". A simple soft reboot, and even pressing the 
reset button on the ESPRESSObin isn't enough to completely reset the 
state, if you reboot this way you will get the "fw is already running !" 
message and it will not work. You have to unplug and then replug the 
board. After that kernel 4.12.12 will boot and wifi will work.

Here is the dmesg output:

https://paste.xogium.me/view/4dc3bebd
-- 

Ellie Reeves

Re: [EXT] Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

[Ellie Reeves]

Hi again,
I tried the few patch that made it to this list without any real 
success... I suppose the issue isn't that easy to reproduce on your side 
so... Shall I try a git bisect, building only that module and see if I 
can narrow it down a bit ? I'm no expert with git or git bisect for that 
matter, but I'm guessing building the module will be less time consuming 
than building the entire kernel... Unless you want me to do something 
else to help in debugging. I want to help as much as I can to solve 
this, heh, even despite me annoying my father because his wifi access 
point goes poof. :)
Thanks,

-- 
Ellie Reeves

Re: [EXT] Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

[Ellie Reeves]

Hi,
just a short message to mention I booted the 4.14 kernel and grabbed 
what was in /sys/class/devcoredump/devcd1/data into a file. I'm not sure 
if it could be useful, but did it anyways... Let me know if it's useful 
and I'll post the binary somewhere.
Thanks

-------- Message original --------
> Hi again,
> I tried the few patch that made it to this list without any real 
> success... I suppose the issue isn't that easy to reproduce on your 
> side so... Shall I try a git bisect, building only that module and see 
> if I can narrow it down a bit ? I'm no expert with git or git bisect 
> for that matter, but I'm guessing building the module will be less 
> time consuming than building the entire kernel... Unless you want me 
> to do something else to help in debugging. I want to help as much as I 
> can to solve this, heh, even despite me annoying my father because his 
> wifi access point goes poof. :)
> Thanks,
>

-- 
Ellie Reeves

RE: [EXT] Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

[Ganapathi Bhat]

SGkgRWxsaWUNCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBFbGxpZSBS
ZWV2ZXMgW21haWx0bzplbGxpZXJldnZlc0BnbWFpbC5jb21dDQo+IFNlbnQ6IEZyaWRheSwgRGVj
ZW1iZXIgMDEsIDIwMTcgOTowNyBQTQ0KPiBUbzogR2FuYXBhdGhpIEJoYXQNCj4gQ2M6IGxpbnV4
LXdpcmVsZXNzQHZnZXIua2VybmVsLm9yZzsgYW1pdGthcndhckBnbWFpbC5jb207IE5pc2hhbnQN
Cj4gU2FybXVrYWRhbTsgWGlubWluZyBIdQ0KPiBTdWJqZWN0OiBSZTogW0VYVF0gTWFydmVsbCBN
d2lmaWV4IHdpcmVsZXNzIGRyaXZlcjogbXdpZmlleF9wY2llIGJyb2tlbiBvbg0KPiBNYXJ2ZWxs
IEVTUFJFU1NPQmluDQo+DQo+IEhpLA0KPiBqdXN0IGEgc2hvcnQgbWVzc2FnZSB0byBtZW50aW9u
IEkgYm9vdGVkIHRoZSA0LjE0IGtlcm5lbCBhbmQgZ3JhYmJlZCB3aGF0DQo+IHdhcyBpbiAvc3lz
L2NsYXNzL2RldmNvcmVkdW1wL2RldmNkMS9kYXRhIGludG8gYSBmaWxlLiBJJ20gbm90IHN1cmUg
aWYgaXQNCj4gY291bGQgYmUgdXNlZnVsLCBidXQgZGlkIGl0IGFueXdheXMuLi4gTGV0IG1lIGtu
b3cgaWYgaXQncyB1c2VmdWwgYW5kIEknbGwgcG9zdA0KPiB0aGUgYmluYXJ5IHNvbWV3aGVyZS4N
Cj4gVGhhbmtzDQpZb3UgY2FuIHNoYXJlIHRoYXQuICBBZGRpdGlvbmFsbHksIGNhbiB5b3Ugc2hh
cmUgdGhlIGRtZXNnIGxvZ3Mgd2l0aCBleHRlbnNpdmUgbG9nIGxldmVsLg0KWW91IGNhbiB1c2Ug
YmVsb3cgY29tbWFuZCB0byBzZXQgdGhlIHNhbWU6DQplY2hvICAgIDB4ZmZmZmZmZmYgICAgICA+
ICAgICAgIC9zeXMva2VybmVsL2RlYnVnL213aWZpZXgvbWxhbjAvZGVidWdfbWFzaw0KPg0KPiAt
LS0tLS0tLSBNZXNzYWdlIG9yaWdpbmFsIC0tLS0tLS0tDQo+ID4gSGkgYWdhaW4sDQo+ID4gSSB0
cmllZCB0aGUgZmV3IHBhdGNoIHRoYXQgbWFkZSBpdCB0byB0aGlzIGxpc3Qgd2l0aG91dCBhbnkg
cmVhbA0KPiA+IHN1Y2Nlc3MuLi4gSSBzdXBwb3NlIHRoZSBpc3N1ZSBpc24ndCB0aGF0IGVhc3kg
dG8gcmVwcm9kdWNlIG9uIHlvdXINCj4gPiBzaWRlIHNvLi4uIFNoYWxsIEkgdHJ5IGEgZ2l0IGJp
c2VjdCwgYnVpbGRpbmcgb25seSB0aGF0IG1vZHVsZSBhbmQgc2VlDQo+ID4gaWYgSSBjYW4gbmFy
cm93IGl0IGRvd24gYSBiaXQgPyBJJ20gbm8gZXhwZXJ0IHdpdGggZ2l0IG9yIGdpdCBiaXNlY3QN
Cj4gPiBmb3IgdGhhdCBtYXR0ZXIsIGJ1dCBJJ20gZ3Vlc3NpbmcgYnVpbGRpbmcgdGhlIG1vZHVs
ZSB3aWxsIGJlIGxlc3MNCj4gPiB0aW1lIGNvbnN1bWluZyB0aGFuIGJ1aWxkaW5nIHRoZSBlbnRp
cmUga2VybmVsLi4uIFVubGVzcyB5b3Ugd2FudCBtZQ0KPiA+IHRvIGRvIHNvbWV0aGluZyBlbHNl
IHRvIGhlbHAgaW4gZGVidWdnaW5nLiBJIHdhbnQgdG8gaGVscCBhcyBtdWNoIGFzIEkNCj4gPiBj
YW4gdG8gc29sdmUgdGhpcywgaGVoLCBldmVuIGRlc3BpdGUgbWUgYW5ub3lpbmcgbXkgZmF0aGVy
IGJlY2F1c2UgaGlzDQo+ID4gd2lmaSBhY2Nlc3MgcG9pbnQgZ29lcyBwb29mLiA6KSBUaGFua3Ms
DQo+ID4NCj4NCj4gLS0NCj4gRWxsaWUgUmVldmVzDQoNCg==

Re: [EXT] Marvell Mwifiex wireless driver: mwifiex_pcie broken on Marvell ESPRESSOBin

[Ellie Reeves]

Hi all,
here is the firmware dump... It looks more like a binary blob to me, but 
I guess there are some tools to inspect it:
https://cloud.xogium.me/s/wDf9bOWZU3cngw8/download

(dirrect link). As for the dmesg output, the first logs I posted were 
exactly this, all kernel messages filtered from journald, and nothing 
serious seems to have changed. I will redo it as soon as I have a chance 
and post it. I cannot use dmesg because there are so much info generated 
the buffer is incomplete, hence the use of journalctl -b -k. Doesn't 
really help that the poor board is having a load average of 5 while 
doing this, but I will try to capture a new debug log.

Hope this helps

-------- Message original --------
> Hi Ellie
>
>> -----Original Message-----
>> From: Ellie Reeves [mailto:ellierevves@gmail.com]
>> Sent: Friday, December 01, 2017 9:07 PM
>> To: Ganapathi Bhat
>> Cc: linux-wireless@vger.kernel.org; amitkarwar@gmail.com; Nishant
>> Sarmukadam; Xinming Hu
>> Subject: Re: [EXT] Marvell Mwifiex wireless driver: mwifiex_pcie broken on
>> Marvell ESPRESSOBin
>>
>> Hi,
>> just a short message to mention I booted the 4.14 kernel and grabbed what
>> was in /sys/class/devcoredump/devcd1/data into a file. I'm not sure if it
>> could be useful, but did it anyways... Let me know if it's useful and I'll post
>> the binary somewhere.
>> Thanks
> You can share that.  Additionally, can you share the dmesg logs with extensive log level.
> You can use below command to set the same:
> echo    0xffffffff      >       /sys/kernel/debug/mwifiex/mlan0/debug_mask
>> -------- Message original --------
>>> Hi again,
>>> I tried the few patch that made it to this list without any real
>>> success... I suppose the issue isn't that easy to reproduce on your
>>> side so... Shall I try a git bisect, building only that module and see
>>> if I can narrow it down a bit ? I'm no expert with git or git bisect
>>> for that matter, but I'm guessing building the module will be less
>>> time consuming than building the entire kernel... Unless you want me
>>> to do something else to help in debugging. I want to help as much as I
>>> can to solve this, heh, even despite me annoying my father because his
>>> wifi access point goes poof. :) Thanks,
>>>
>> --
>> Ellie Reeves

-- 
Ellie Reeves