Firmware crash w/ 10.2 diagnosis?

Firmware crash w/ 10.2 diagnosis?

[Denton Gentry]

Can anyone tell me what this firmware crash means?
I don't have steps to reproduce it, it seems to happen randomly. The
clients are mostly idle.

[Fri Sep  5 11:37:08 2014] ath10k: firmware crashed!
[Fri Sep  5 11:37:08 2014] ath10k: hardware name qca988x hw2.0 version
0x4100016c
[Fri Sep  5 11:37:08 2014] ath10k: firmware version: 10.2-00082-4-2
[Fri Sep  5 11:37:08 2014] ath10k: target register Dump Location: 0x0040AA34
[Fri Sep  5 11:37:08 2014] ath10k: target Register Dump
[Fri Sep  5 11:37:08 2014] ath10k: [00]: 0x4100016C 0x00000000
0x00949FF9 0x803044BE
[Fri Sep  5 11:37:08 2014] ath10k: [04]: 0x00949FF9 0x00060730
0x00000015 0x00000002
[Fri Sep  5 11:37:08 2014] ath10k: [08]: 0x00425BD0 0x803044BE
0x00000000 0x00000000
[Fri Sep  5 11:37:08 2014] ath10k: [12]: 0x00000009 0xFFFFFFFF
0x00958360 0x0095836B
[Fri Sep  5 11:37:08 2014] ath10k: [16]: 0x00958080 0x009BD000
0x00000000 0x00000000
[Fri Sep  5 11:37:08 2014] ath10k: [20]: 0x40949FF9 0x0040AB44
0x00000000 0x0043AC74
[Fri Sep  5 11:37:08 2014] ath10k: [24]: 0x8099519C 0x0040ABA4
0x00000F40 0xC0949FF9
[Fri Sep  5 11:37:08 2014] ath10k: [28]: 0x8099861D 0x0040ABC4
0x00424B2C 0x00439E64
[Fri Sep  5 11:37:08 2014] ath10k: [32]: 0x80998802 0x0040AD14
0x00000000 0x0042DD18
[Fri Sep  5 11:37:08 2014] ath10k: [36]: 0x80998704 0x0040AD44
0x0042DD18 0x00000002
[Fri Sep  5 11:37:08 2014] ath10k: [40]: 0x80998830 0x0040AD74
0x00439EC4 0x0042DD18
[Fri Sep  5 11:37:08 2014] ath10k: [44]: 0x809970B2 0x0040AD94
0x00439EC4 0x00000002
[Fri Sep  5 11:37:08 2014] ath10k: [48]: 0x80998EF5 0x0040ADB4
0x0041FA54 0x00425760
[Fri Sep  5 11:37:08 2014] ath10k: [52]: 0x809BCF0D 0x0040AEE4
0x004245FC 0x00000002
[Fri Sep  5 11:37:08 2014] ath10k: [56]: 0x80940F18 0x0040AF14
0x00000005 0x004039E4
[Fri Sep  5 11:37:13 2014] ath10k: failed to synchronise monitor vdev 1: -110
[Fri Sep  5 11:37:13 2014] ath10k: failed to stop monitor vdev: -110
[Fri Sep  5 11:37:16 2014] ath10k: failed to request wmi monitor vdev
1 removal: -11
[Fri Sep  5 11:37:16 2014] ath10k: failed to delete monitor vdev: -11
[Fri Sep  5 11:37:19 2014] ath10k_wmi_vdev_start_restart: dtim_period=0
[Fri Sep  5 11:37:19 2014] ath10k: peer 88:dc:96:XX:XX:XX on vdev 0
already exists
[Fri Sep  5 11:37:19 2014] ath10k: failed to add peer
88:dc:96:XX:XX:XX for vdev 0 when changing bssid: -22
[Fri Sep  5 11:37:19 2014] ath10k_wmi_vdev_start_restart: dtim_period=0
[Fri Sep  5 11:37:20 2014] ath10k: device successfully recovered

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Firmware crash w/ 10.2 diagnosis?

[Michal Kazior]

On 5 September 2014 22:56, Denton Gentry <denton.gentry@gmail.com> wrote:
> Can anyone tell me what this firmware crash means?
> I don't have steps to reproduce it, it seems to happen randomly. The
> clients are mostly idle.
>
> [Fri Sep  5 11:37:08 2014] ath10k: firmware crashed!
> [Fri Sep  5 11:37:08 2014] ath10k: hardware name qca988x hw2.0 version
> 0x4100016c
> [Fri Sep  5 11:37:08 2014] ath10k: firmware version: 10.2-00082-4-2
> [Fri Sep  5 11:37:08 2014] ath10k: target register Dump Location: 0x0040AA34
> [Fri Sep  5 11:37:08 2014] ath10k: target Register Dump
> [Fri Sep  5 11:37:08 2014] ath10k: [00]: 0x4100016C 0x00000000
> 0x00949FF9 0x803044BE
> [Fri Sep  5 11:37:08 2014] ath10k: [04]: 0x00949FF9 0x00060730
> 0x00000015 0x00000002
> [Fri Sep  5 11:37:08 2014] ath10k: [08]: 0x00425BD0 0x803044BE
> 0x00000000 0x00000000
> [Fri Sep  5 11:37:08 2014] ath10k: [12]: 0x00000009 0xFFFFFFFF
> 0x00958360 0x0095836B
> [Fri Sep  5 11:37:08 2014] ath10k: [16]: 0x00958080 0x009BD000
> 0x00000000 0x00000000
> [Fri Sep  5 11:37:08 2014] ath10k: [20]: 0x40949FF9 0x0040AB44
> 0x00000000 0x0043AC74
> [Fri Sep  5 11:37:08 2014] ath10k: [24]: 0x8099519C 0x0040ABA4
> 0x00000F40 0xC0949FF9
> [Fri Sep  5 11:37:08 2014] ath10k: [28]: 0x8099861D 0x0040ABC4
> 0x00424B2C 0x00439E64
> [Fri Sep  5 11:37:08 2014] ath10k: [32]: 0x80998802 0x0040AD14
> 0x00000000 0x0042DD18
> [Fri Sep  5 11:37:08 2014] ath10k: [36]: 0x80998704 0x0040AD44
> 0x0042DD18 0x00000002
> [Fri Sep  5 11:37:08 2014] ath10k: [40]: 0x80998830 0x0040AD74
> 0x00439EC4 0x0042DD18
> [Fri Sep  5 11:37:08 2014] ath10k: [44]: 0x809970B2 0x0040AD94
> 0x00439EC4 0x00000002
> [Fri Sep  5 11:37:08 2014] ath10k: [48]: 0x80998EF5 0x0040ADB4
> 0x0041FA54 0x00425760
> [Fri Sep  5 11:37:08 2014] ath10k: [52]: 0x809BCF0D 0x0040AEE4
> 0x004245FC 0x00000002
> [Fri Sep  5 11:37:08 2014] ath10k: [56]: 0x80940F18 0x0040AF14
> 0x00000005 0x004039E4
> [Fri Sep  5 11:37:13 2014] ath10k: failed to synchronise monitor vdev 1: -110
> [Fri Sep  5 11:37:13 2014] ath10k: failed to stop monitor vdev: -110
> [Fri Sep  5 11:37:16 2014] ath10k: failed to request wmi monitor vdev
> 1 removal: -11
> [Fri Sep  5 11:37:16 2014] ath10k: failed to delete monitor vdev: -11
> [Fri Sep  5 11:37:19 2014] ath10k_wmi_vdev_start_restart: dtim_period=0
> [Fri Sep  5 11:37:19 2014] ath10k: peer 88:dc:96:XX:XX:XX on vdev 0
> already exists
> [Fri Sep  5 11:37:19 2014] ath10k: failed to add peer
> 88:dc:96:XX:XX:XX for vdev 0 when changing bssid: -22
> [Fri Sep  5 11:37:19 2014] ath10k_wmi_vdev_start_restart: dtim_period=0

Hmm, you seem to be using some custom patches on ath10k. It seems you
were running a station interface and it roamed but for some reason it
tried to stop monitor vdev as well?

Can you share more details on what your setup is? What the custom patches are?


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Firmware crash w/ 10.2 diagnosis?

[Denton Gentry]

The ath10k is operating in AP mode.

Its using a base ath10k driver from a few months ago:

       git fetch git://github.com/kvalo/ath.git ath-next
       git checkout v3.15-rc1-237-gd9bc4b9

We've pulled in selected changes since. Notably:

warm reset
http://lists.infradead.org/pipermail/ath10k/2014-May/001992.html
http://lists.infradead.org/pipermail/ath10k/2014-May/001993.html

RX Reordering
http://lists.infradead.org/pipermail/ath10k/2014-July/002637.html
http://lists.infradead.org/pipermail/ath10k/2014-July/002621.html
http://lists.infradead.org/pipermail/ath10k/2014-July/002746.html
http://lists.infradead.org/pipermail/ath10k/2014-July/002784.html

Firmware 10.2
https://github.com/kvalo/ath/commit/24c88f7807fb7c723690474d0a5d3441468185d9

More recently, we pulled in TX credit and flushing changes
http://lists.infradead.org/pipermail/ath10k/2014-April/001765.html
http://lists.infradead.org/pipermail/ath10k/2014-April/001709.html
http://lists.infradead.org/pipermail/ath10k/2014-April/001706.html
http://lists.infradead.org/pipermail/ath10k/2014-April/001708.html


The data from the firmware shows something about a monitor vdev being
removed? hostapd does create a monitor interface, though I don't see
an indication that it was being removed at the time of the crash.
hostapd had exited and restarted well before though, is it possible
that removal of a monitor interface some time ago might be
contributing?

On Sun, Sep 7, 2014 at 11:28 PM, Michal Kazior <michal.kazior@tieto.com> wrote:
> On 5 September 2014 22:56, Denton Gentry <denton.gentry@gmail.com> wrote:
>> Can anyone tell me what this firmware crash means?
>> I don't have steps to reproduce it, it seems to happen randomly. The
>> clients are mostly idle.
>>
>> [Fri Sep  5 11:37:08 2014] ath10k: firmware crashed!
>> [Fri Sep  5 11:37:08 2014] ath10k: hardware name qca988x hw2.0 version
>> 0x4100016c
>> [Fri Sep  5 11:37:08 2014] ath10k: firmware version: 10.2-00082-4-2
>> [Fri Sep  5 11:37:08 2014] ath10k: target register Dump Location: 0x0040AA34
>> [Fri Sep  5 11:37:08 2014] ath10k: target Register Dump
>> [Fri Sep  5 11:37:08 2014] ath10k: [00]: 0x4100016C 0x00000000
>> 0x00949FF9 0x803044BE
>> [Fri Sep  5 11:37:08 2014] ath10k: [04]: 0x00949FF9 0x00060730
>> 0x00000015 0x00000002
>> [Fri Sep  5 11:37:08 2014] ath10k: [08]: 0x00425BD0 0x803044BE
>> 0x00000000 0x00000000
>> [Fri Sep  5 11:37:08 2014] ath10k: [12]: 0x00000009 0xFFFFFFFF
>> 0x00958360 0x0095836B
>> [Fri Sep  5 11:37:08 2014] ath10k: [16]: 0x00958080 0x009BD000
>> 0x00000000 0x00000000
>> [Fri Sep  5 11:37:08 2014] ath10k: [20]: 0x40949FF9 0x0040AB44
>> 0x00000000 0x0043AC74
>> [Fri Sep  5 11:37:08 2014] ath10k: [24]: 0x8099519C 0x0040ABA4
>> 0x00000F40 0xC0949FF9
>> [Fri Sep  5 11:37:08 2014] ath10k: [28]: 0x8099861D 0x0040ABC4
>> 0x00424B2C 0x00439E64
>> [Fri Sep  5 11:37:08 2014] ath10k: [32]: 0x80998802 0x0040AD14
>> 0x00000000 0x0042DD18
>> [Fri Sep  5 11:37:08 2014] ath10k: [36]: 0x80998704 0x0040AD44
>> 0x0042DD18 0x00000002
>> [Fri Sep  5 11:37:08 2014] ath10k: [40]: 0x80998830 0x0040AD74
>> 0x00439EC4 0x0042DD18
>> [Fri Sep  5 11:37:08 2014] ath10k: [44]: 0x809970B2 0x0040AD94
>> 0x00439EC4 0x00000002
>> [Fri Sep  5 11:37:08 2014] ath10k: [48]: 0x80998EF5 0x0040ADB4
>> 0x0041FA54 0x00425760
>> [Fri Sep  5 11:37:08 2014] ath10k: [52]: 0x809BCF0D 0x0040AEE4
>> 0x004245FC 0x00000002
>> [Fri Sep  5 11:37:08 2014] ath10k: [56]: 0x80940F18 0x0040AF14
>> 0x00000005 0x004039E4
>> [Fri Sep  5 11:37:13 2014] ath10k: failed to synchronise monitor vdev 1: -110
>> [Fri Sep  5 11:37:13 2014] ath10k: failed to stop monitor vdev: -110
>> [Fri Sep  5 11:37:16 2014] ath10k: failed to request wmi monitor vdev
>> 1 removal: -11
>> [Fri Sep  5 11:37:16 2014] ath10k: failed to delete monitor vdev: -11
>> [Fri Sep  5 11:37:19 2014] ath10k_wmi_vdev_start_restart: dtim_period=0
>> [Fri Sep  5 11:37:19 2014] ath10k: peer 88:dc:96:XX:XX:XX on vdev 0
>> already exists
>> [Fri Sep  5 11:37:19 2014] ath10k: failed to add peer
>> 88:dc:96:XX:XX:XX for vdev 0 when changing bssid: -22
>> [Fri Sep  5 11:37:19 2014] ath10k_wmi_vdev_start_restart: dtim_period=0
>
> Hmm, you seem to be using some custom patches on ath10k. It seems you
> were running a station interface and it roamed but for some reason it
> tried to stop monitor vdev as well?
>
> Can you share more details on what your setup is? What the custom patches are?
>
>
> Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Firmware crash w/ 10.2 diagnosis?

[Kalle Valo]

Denton Gentry <denton.gentry@gmail.com> writes:

> The ath10k is operating in AP mode.
>
> Its using a base ath10k driver from a few months ago:
>
>        git fetch git://github.com/kvalo/ath.git ath-next
>        git checkout v3.15-rc1-237-gd9bc4b9

What is the kernel version? Do you use backports?

> The data from the firmware shows something about a monitor vdev being
> removed? hostapd does create a monitor interface, though I don't see
> an indication that it was being removed at the time of the crash.
> hostapd had exited and restarted well before though, is it possible
> that removal of a monitor interface some time ago might be
> contributing?

So are you still using the old cooked monitor interface for management
frames? I strongly recommend switching to the new way of transmitting
and receiving management frames through nl80211 as then hostapd will not
a create monitor interface. This is what most people use and is most
extensively tested.

Commit 6e3e939f3b1bf8534b32ad09ff199d88800835a0 ('net: add wireless TX
status socket option') adds this capability to the kernel, first
released in Linux 3.3. There might be some dependencies in case you are
porting it to an older kernel, so be extra careful.

Please note that updating backports is not enough for SO_WIFI_STATUS
status, you need to update the kernel itself.

-- 
Kalle Valo

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Firmware crash w/ 10.2 diagnosis?

[Michal Kazior]

On 9 September 2014 14:08, Kalle Valo <kvalo@qca.qualcomm.com> wrote:
> Denton Gentry <denton.gentry@gmail.com> writes:
>
>> The ath10k is operating in AP mode.
>>
>> Its using a base ath10k driver from a few months ago:
>>
>>        git fetch git://github.com/kvalo/ath.git ath-next
>>        git checkout v3.15-rc1-237-gd9bc4b9
>
> What is the kernel version? Do you use backports?
>
>> The data from the firmware shows something about a monitor vdev being
>> removed? hostapd does create a monitor interface, though I don't see
>> an indication that it was being removed at the time of the crash.
>> hostapd had exited and restarted well before though, is it possible
>> that removal of a monitor interface some time ago might be
>> contributing?
>
> So are you still using the old cooked monitor interface for management
> frames? I strongly recommend switching to the new way of transmitting
> and receiving management frames through nl80211 as then hostapd will not
> a create monitor interface. This is what most people use and is most
> extensively tested.

I don't think he's using cooked monitor. Instead his AP interface was
in a bridge which means it was promiscuous which means ath10k used
monitor vdev. That's perfectly normal.

Warnings I was concerned about are actually a result of recovery
procedure itself and do not seem to be related to the crash root
cause.


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Firmware crash w/ 10.2 diagnosis?

[Denton Gentry]

I'm using backports, and the kernel version is 3.2.
hostapd says it is indeed using the cooked monitor:

    monitor mode needed: 1 (poll_cmd=1 data_tx_status=0)

I'll look at updating the kernel, if not to 3.16 then at least to
something recent enough to stop using the monitor interface from
hostapd.

On Tue, Sep 9, 2014 at 5:48 AM, Michal Kazior <michal.kazior@tieto.com> wrote:
> On 9 September 2014 14:08, Kalle Valo <kvalo@qca.qualcomm.com> wrote:
>> Denton Gentry <denton.gentry@gmail.com> writes:
>>
>>> The ath10k is operating in AP mode.
>>>
>>> Its using a base ath10k driver from a few months ago:
>>>
>>>        git fetch git://github.com/kvalo/ath.git ath-next
>>>        git checkout v3.15-rc1-237-gd9bc4b9
>>
>> What is the kernel version? Do you use backports?
>>
>>> The data from the firmware shows something about a monitor vdev being
>>> removed? hostapd does create a monitor interface, though I don't see
>>> an indication that it was being removed at the time of the crash.
>>> hostapd had exited and restarted well before though, is it possible
>>> that removal of a monitor interface some time ago might be
>>> contributing?
>>
>> So are you still using the old cooked monitor interface for management
>> frames? I strongly recommend switching to the new way of transmitting
>> and receiving management frames through nl80211 as then hostapd will not
>> a create monitor interface. This is what most people use and is most
>> extensively tested.
>
> I don't think he's using cooked monitor. Instead his AP interface was
> in a bridge which means it was promiscuous which means ath10k used
> monitor vdev. That's perfectly normal.
>
> Warnings I was concerned about are actually a result of recovery
> procedure itself and do not seem to be related to the crash root
> cause.
>
>
> Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: Firmware crash w/ 10.2 diagnosis?

[Kalle Valo]

Denton Gentry <denton.gentry@gmail.com> writes:

> I'm using backports, and the kernel version is 3.2.
> hostapd says it is indeed using the cooked monitor:
>
>     monitor mode needed: 1 (poll_cmd=1 data_tx_status=0)
>
> I'll look at updating the kernel, if not to 3.16 then at least to
> something recent enough to stop using the monitor interface from
> hostapd.

Good. But if you use bridging the monitor vdev is created anyway. So
most likely switching to nl80211 management frame interface is not going
to workaround the firmware bug.

-- 
Kalle Valo

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k