[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Joe Qiao]

Dear all,

I'm working on the wireless AP issue with ath9k driver now. I always can see the following error log, and the wireless will stop broadcast or client cannot connect to AP.

[212767.228324] ath: phy0: Failed to stop TX DMA, queues=0x004!
[213045.584267] ath: phy0: Failed to stop TX DMA, queues=0x004!
[213102.806419] ath: phy0: Failed to stop TX DMA, queues=0x005!
[213121.954144] ath: phy0: Failed to stop TX DMA, queues=0x001!
[213215.547168] ath: phy0: Failed to stop TX DMA, queues=0x001!

Previously, I used the old driver v3.6.8 (stable branch). But even I update driver to the latest version v3.13, it still cannot solve the problem.

Below is some info from my device:

ieee80211 phy0: Atheros AR9300 Rev:3

Linux version 3.0.34

Loading modules backported from Linux version v3.13-rc8-0-g7e22e91
Backport generated by backports.git v3.13-rc8-1-0-gae71bd3

Would you please give some suggestions with your rich experience?

Much appreciate your help!

Thanks,
-Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20140122/111553f0/attachment-0001.htm 

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Sujith Manoharan]

Joe Qiao wrote:
> I?m working on the wireless AP issue with ath9k driver now. I always can see
> the following error log, and the wireless will stop broadcast or client cannot
> connect to AP.
> 
> [212767.228324] ath: phy0: Failed to stop TX DMA, queues=0x004!
> [213045.584267] ath: phy0: Failed to stop TX DMA, queues=0x004!
> [213102.806419] ath: phy0: Failed to stop TX DMA, queues=0x005!
> [213121.954144] ath: phy0: Failed to stop TX DMA, queues=0x001!
> [213215.547168] ath: phy0: Failed to stop TX DMA, queues=0x001!

Is this issue consistently reproducible in your setup ? What board/platform ?

Sujith

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Joe Qiao]

On powerpc. 

Reproduce rarely, but always can see.

My step is that,
a). Running hostapd, stop it, and start wpa_supplicant, 
b). call SIOCSIWSCAN via ioctl to scan and call SIOCGIWSCAN to get scanning result.
c). stop wpa_supplicant and restart hostapd.

Thanks,
-Joe

-----Original Message-----
From: Sujith Manoharan [mailto:sujith at msujith.org] 
Sent: Wednesday, January 22, 2014 4:29 PM
To: Joe Qiao
Cc: ath9k-devel at lists.ath9k.org
Subject: Re: [ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

Joe Qiao wrote:
> I?m working on the wireless AP issue with ath9k driver now. I always 
> can see the following error log, and the wireless will stop broadcast 
> or client cannot connect to AP.
> 
> [212767.228324] ath: phy0: Failed to stop TX DMA, queues=0x004!
> [213045.584267] ath: phy0: Failed to stop TX DMA, queues=0x004!
> [213102.806419] ath: phy0: Failed to stop TX DMA, queues=0x005!
> [213121.954144] ath: phy0: Failed to stop TX DMA, queues=0x001!
> [213215.547168] ath: phy0: Failed to stop TX DMA, queues=0x001!

Is this issue consistently reproducible in your setup ? What board/platform ?

Sujith

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Sujith Manoharan]

Joe Qiao wrote:
> On powerpc. 
> 
> Reproduce rarely, but always can see.
> 
> My step is that,
> a). Running hostapd, stop it, and start wpa_supplicant, 
> b). call SIOCSIWSCAN via ioctl to scan and call SIOCGIWSCAN to get scanning result.
> c). stop wpa_supplicant and restart hostapd.

Can you use the latest backports release and post the kernel log when the
issue happens ?

https://www.kernel.org/pub/linux/kernel/projects/backports/2014/01/17/

The ath9k-debug config can be used in backports:
make defconfig-ath9k-debug && make && sudo make install

Load the driver with debugging enabled: modprobe athk debug=0x7cb

Sujith

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Joe Qiao]

Sure. Will do and let you know the test result soon.

Thanks,
-Joe

-----Original Message-----
From: Sujith Manoharan [mailto:sujith at msujith.org] 
Sent: Thursday, January 23, 2014 4:29 PM
To: Joe Qiao
Cc: ath9k-devel at lists.ath9k.org
Subject: RE: [ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

Joe Qiao wrote:
> On powerpc. 
> 
> Reproduce rarely, but always can see.
> 
> My step is that,
> a). Running hostapd, stop it, and start wpa_supplicant, b). call 
> SIOCSIWSCAN via ioctl to scan and call SIOCGIWSCAN to get scanning result.
> c). stop wpa_supplicant and restart hostapd.

Can you use the latest backports release and post the kernel log when the issue happens ?

https://www.kernel.org/pub/linux/kernel/projects/backports/2014/01/17/

The ath9k-debug config can be used in backports:
make defconfig-ath9k-debug && make && sudo make install

Load the driver with debugging enabled: modprobe athk debug=0x7cb

Sujith

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Joe Qiao]

Hi Sujith,

The version of 2014/01/17/ works stable in recent 24 hours. It prints lot of debug log, but cannot see the error log again.

Do you think the build of 2014/01/17/ include the fix? Or which stable build can be used in future?

Thanks,
-Joe

-----Original Message-----
From: ath9k-devel-bounces@lists.ath9k.org [mailto:ath9k-devel-bounces at lists.ath9k.org] On Behalf Of Joe Qiao
Sent: Thursday, January 23, 2014 4:55 PM
To: Sujith Manoharan
Cc: ath9k-devel at lists.ath9k.org; Bryan Phillippe
Subject: Re: [ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

Sure. Will do and let you know the test result soon.

Thanks,
-Joe

-----Original Message-----
From: Sujith Manoharan [mailto:sujith at msujith.org]
Sent: Thursday, January 23, 2014 4:29 PM
To: Joe Qiao
Cc: ath9k-devel at lists.ath9k.org
Subject: RE: [ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

Joe Qiao wrote:
> On powerpc. 
> 
> Reproduce rarely, but always can see.
> 
> My step is that,
> a). Running hostapd, stop it, and start wpa_supplicant, b). call 
> SIOCSIWSCAN via ioctl to scan and call SIOCGIWSCAN to get scanning result.
> c). stop wpa_supplicant and restart hostapd.

Can you use the latest backports release and post the kernel log when the issue happens ?

https://www.kernel.org/pub/linux/kernel/projects/backports/2014/01/17/

The ath9k-debug config can be used in backports:
make defconfig-ath9k-debug && make && sudo make install

Load the driver with debugging enabled: modprobe athk debug=0x7cb

Sujith
_______________________________________________
ath9k-devel mailing list
ath9k-devel at lists.ath9k.org
https://lists.ath9k.org/mailman/listinfo/ath9k-devel

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Sujith Manoharan]

Joe Qiao wrote:
> The version of 2014/01/17/ works stable in recent 24 hours. It prints lot of
> debug log, but cannot see the error log again.
> 
> Do you think the build of 2014/01/17/ include the fix? Or which stable build
> can be used in future?

The 2014-01-17 is the latest backports release, so it should be stable.
Just to verify, please load the driver normally, without the debug option
to make sure that the issue doesn't happen.

Sujith

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Joe Qiao]

Sounds great. Thank you, Sujith.

Let me verify the driver without debug option again.

Thanks,
-Joe

-----Original Message-----
From: Sujith Manoharan [mailto:sujith at msujith.org] 
Sent: 2014?1?24? PM 11:07
To: Joe Qiao
Cc: ath9k-devel at lists.ath9k.org; Bryan Phillippe
Subject: RE: [ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

Joe Qiao wrote:
> The version of 2014/01/17/ works stable in recent 24 hours. It prints 
> lot of debug log, but cannot see the error log again.
> 
> Do you think the build of 2014/01/17/ include the fix? Or which stable 
> build can be used in future?

The 2014-01-17 is the latest backports release, so it should be stable.
Just to verify, please load the driver normally, without the debug option to make sure that the issue doesn't happen.

Sujith

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Joe Qiao]

Hi Sujith,

After several days stress test for the ath9k driver 2014-01-17 and 2014-01-31, I still can see the below error log. But less than before (v3.13) I think, sometimes 1 log in one or two days, sometime no error log.

[155943.839310] ath: phy0: Failed to stop TX DMA, queues=0x001!

However, the strange is that, the error log only can see in the driver without debug option. When I run the driver which enable debug option, test always running for 3~4 days, 100MB+ log file record in a day, I can't find similar error log "Failed to stop TX DMA".

Do you have any suggestion?

Thanks,
-Joe

-----Original Message-----
From: ath9k-devel-bounces@lists.ath9k.org [mailto:ath9k-devel-bounces at lists.ath9k.org] On Behalf Of Joe Qiao
Sent: Sunday, January 26, 2014 7:24 AM
To: Sujith Manoharan
Cc: ath9k-devel at lists.ath9k.org; Bryan Phillippe
Subject: Re: [ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

Sounds great. Thank you, Sujith.

Let me verify the driver without debug option again.

Thanks,
-Joe

-----Original Message-----
From: Sujith Manoharan [mailto:sujith at msujith.org]
Sent: 2014?1?24? PM 11:07
To: Joe Qiao
Cc: ath9k-devel at lists.ath9k.org; Bryan Phillippe
Subject: RE: [ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

Joe Qiao wrote:
> The version of 2014/01/17/ works stable in recent 24 hours. It prints 
> lot of debug log, but cannot see the error log again.
> 
> Do you think the build of 2014/01/17/ include the fix? Or which stable 
> build can be used in future?

The 2014-01-17 is the latest backports release, so it should be stable.
Just to verify, please load the driver normally, without the debug option to make sure that the issue doesn't happen.

Sujith
_______________________________________________
ath9k-devel mailing list
ath9k-devel at lists.ath9k.org
https://lists.ath9k.org/mailman/listinfo/ath9k-devel

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Sujith Manoharan]

Joe Qiao wrote:
> After several days stress test for the ath9k driver 2014-01-17 and 2014-01-31,
> I still can see the below error log. But less than before (v3.13) I think,
> sometimes 1 log in one or two days, sometime no error log.
> 
> [155943.839310] ath: phy0: Failed to stop TX DMA, queues=0x001!
> 
> However, the strange is that, the error log only can see in the driver without
> debug option. When I run the driver which enable debug option, test always
> running for 3~4 days, 100MB+ log file record in a day, I can't find similar
> error log "Failed to stop TX DMA".

Hard to debug things when we don't have debug log information. ;)
Can you lower the debug level and use "debug=0x200" ? This will give
us at least some information when/if the issue happens.

Sujth

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Joe Qiao]

Hi Sujith,

I got a duplication when use "debug=0x200", debug log is attached. The line 372 shows the error log.

[ 1809.636327] ath: phy0: Failed to stop TX DMA, queues=0x001!

Thanks,
-Joe

-----Original Message-----
From: Sujith Manoharan [mailto:sujith at msujith.org] 
Sent: Wednesday, February 12, 2014 11:21 PM
To: Joe Qiao
Cc: ath9k-devel at lists.ath9k.org; Bryan Phillippe
Subject: RE: [ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

Joe Qiao wrote:
> After several days stress test for the ath9k driver 2014-01-17 and 
> 2014-01-31, I still can see the below error log. But less than before 
> (v3.13) I think, sometimes 1 log in one or two days, sometime no error log.
> 
> [155943.839310] ath: phy0: Failed to stop TX DMA, queues=0x001!
> 
> However, the strange is that, the error log only can see in the driver 
> without debug option. When I run the driver which enable debug option, 
> test always running for 3~4 days, 100MB+ log file record in a day, I 
> can't find similar error log "Failed to stop TX DMA".

Hard to debug things when we don't have debug log information. ;) Can you lower the debug level and use "debug=0x200" ? This will give us at least some information when/if the issue happens.

Sujth
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: log-stopTXDMA.TXT
Url: http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20140213/c3946149/attachment-0001.txt 

[ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

[Joe Qiao]

Hi Sujith,

I'm investigating driver code in these days, but it seems no big progress. I hope can find a workaround to reset everything in driver (hw queue ...) , then if see the error case, we can recover it except reload driver module. Do you have any suggestion?

BTW: on my side, it is easy to reproduce the problem with "debug=0x200" (running hostapd and terminate it, start wpa_supplicant for scanning, kill it after scan and restart hostapd. Repeat and repeat...).

Highly appreciate your help! Looking forward your feedback!

Thanks,
-Joe

-----Original Message-----
From: ath9k-devel-bounces@lists.ath9k.org [mailto:ath9k-devel-bounces at lists.ath9k.org] On Behalf Of Joe Qiao
Sent: Thursday, February 13, 2014 11:01 AM
To: Sujith Manoharan
Cc: ath9k-devel at lists.ath9k.org; Bryan Phillippe
Subject: Re: [ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

Hi Sujith,

I got a duplication when use "debug=0x200", debug log is attached. The line 372 shows the error log.

[ 1809.636327] ath: phy0: Failed to stop TX DMA, queues=0x001!

Thanks,
-Joe

-----Original Message-----
From: Sujith Manoharan [mailto:sujith at msujith.org]
Sent: Wednesday, February 12, 2014 11:21 PM
To: Joe Qiao
Cc: ath9k-devel at lists.ath9k.org; Bryan Phillippe
Subject: RE: [ath9k-devel] How to solve the problem of "Failed to stop TX DMA"

Joe Qiao wrote:
> After several days stress test for the ath9k driver 2014-01-17 and 
> 2014-01-31, I still can see the below error log. But less than before
> (v3.13) I think, sometimes 1 log in one or two days, sometime no error log.
> 
> [155943.839310] ath: phy0: Failed to stop TX DMA, queues=0x001!
> 
> However, the strange is that, the error log only can see in the driver 
> without debug option. When I run the driver which enable debug option, 
> test always running for 3~4 days, 100MB+ log file record in a day, I 
> can't find similar error log "Failed to stop TX DMA".

Hard to debug things when we don't have debug log information. ;) Can you lower the debug level and use "debug=0x200" ? This will give us at least some information when/if the issue happens.

Sujth