Performance regression between Madwifi/net80211 and ath5k/mac80211

Performance regression between Madwifi/net80211 and ath5k/mac80211

[jpo]

Hello all,
there seems to be a significant performance loss when moving from Madwifi 
to ath5k. For an embedded project we have been using Madwifi with Linux kernel
2.6.23. We have tested ath5k for some time and by now it seems to be stable
enough for production use. Unfortunately it doesn't match Madwifi's performance.
Running iperf (both, UDP and TCP) between two machines equipped with an AR5413
with the Fedora 15 kernel and the stock Fedora 15 wireless subsystem shows a
performance loss of about 10% relative to 2.6.23 and Madwifi-0.9.4.
The tests where done under close to ideal lab conditions (cards connected with
an antenna cable) running an IBSS network on channel 40 (5200MHz).

Can anybody confirm this observation?
Is there an explanation?
Are there some tuning knobs I could try to improve the performance with
mac80211/ath5k?
Is the problem outside the wireless subsystem (Maybe it's the TCP/IP stack or
something Fedora specific)?

Thanks in advance.
  Joerg

Re: Performance regression between Madwifi/net80211 and ath5k/mac80211

[Adrian Chadd]

Have you eliminated the kernel version as being the problem?


Adrian

On 22 June 2011 04:09, jpo <pommnitz@yahoo.com> wrote:
> Hello all,
> there seems to be a significant performance loss when moving from Madwifi
> to ath5k. For an embedded project we have been using Madwifi with Linux kernel
> 2.6.23. We have tested ath5k for some time and by now it seems to be stable
> enough for production use. Unfortunately it doesn't match Madwifi's performance.
> Running iperf (both, UDP and TCP) between two machines equipped with an AR5413
> with the Fedora 15 kernel and the stock Fedora 15 wireless subsystem shows a
> performance loss of about 10% relative to 2.6.23 and Madwifi-0.9.4.
> The tests where done under close to ideal lab conditions (cards connected with
> an antenna cable) running an IBSS network on channel 40 (5200MHz).
>
> Can anybody confirm this observation?
> Is there an explanation?
> Are there some tuning knobs I could try to improve the performance with
> mac80211/ath5k?
> Is the problem outside the wireless subsystem (Maybe it's the TCP/IP stack or
> something Fedora specific)?
>
> Thanks in advance.
>  Joerg
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Performance regression between Madwifi/net80211 and ath5k/mac80211

[Joerg Pommnitz]

Adrian,
thanks for taking an interest. Today I built the current Madwifi driver for
Fedora Kernel 2.6.38.8-32.fc15.i686. The ath5k problem seems to be at the
tx side. If I run the iperf server on the ath5k node and the iperf client
at the madwifi node (identical hardware config), I consistently get a 
significant
speedup compared to ath5k-ath5k. The only difference is, which module is
blacklisted.

I have experimented with the RTS and fragmentation threshold setting without
success. 

Now I'm somewhat lost. The problem is either ath5k or mac80211. 

-- 
Regards 
Joerg 




----- Orifinal Mail ----
> Von: Adrian Chadd <adrian@freebsd.org>
> An: jpo <pommnitz@yahoo.com>
> CC: linux-wireless@vger.kernel.org
> Gesendet: Mittwoch, den 22. Juni 2011, 4:17:05 Uhr
> Betreff: Re: Performance regression between Madwifi/net80211 and 
ath5k/mac80211
> 
> Have you eliminated the kernel version as being the  problem?
> 
> 
> Adrian

Re: Performance regression between Madwifi/net80211 and ath5k/mac80211

[Johannes Berg]

On Wed, 2011-06-22 at 06:41 -0700, Joerg Pommnitz wrote:
> Adrian,
> thanks for taking an interest. Today I built the current Madwifi driver for
> Fedora Kernel 2.6.38.8-32.fc15.i686. The ath5k problem seems to be at the
> tx side. If I run the iperf server on the ath5k node and the iperf client
> at the madwifi node (identical hardware config), I consistently get a 
> significant
> speedup compared to ath5k-ath5k. The only difference is, which module is
> blacklisted.
> 
> I have experimented with the RTS and fragmentation threshold setting without
> success. 
> 
> Now I'm somewhat lost. The problem is either ath5k or mac80211. 

Do you see different CPU usage? Does it max out? If so can you profile
it?

johannes

Re: Performance regression between Madwifi/net80211 and ath5k/mac80211

[Joerg Pommnitz]

Johannes,
I'm glad that you take an interest. Here are my observations:

To put some meat to my claims about the performance difference, I'm
providing the summary lines from the TCP iperf server side:

         [ ID]  Interval      Transfer     Bandwidth
madwifi: [  4]  0.0-10.5 sec  30.9 MBytes  24.7 Mbits/sec
ath5k  : [  4]  0.0-10.8 sec  26.0 MBytes  20.2 Mbits/sec

Note the significant difference in the bandwidth column.

To answer your actual question: No, there doesn't seem to be a
difference in CPU usage as shown by top. If anything, madwifi 
seems to use a little bit more:

madwifi: 85,5% idle, 10% si
ath5k  : 89% idle,   10% si

The CPU is:

processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 13
model name    : Intel(R) Celeron(R) M processor         1.00GHz
stepping    : 8
cpu MHz        : 999.954
cache size    : 512 KB
fdiv_bug    : no
hlt_bug        : no
f00f_bug    : no
coma_bug    : no
fpu        : yes
fpu_exception    : yes
cpuid level    : 2
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov clflush 
dts acpi mmx fxsr sse sse2 ss tm pbe nx up bts
bogomips    : 1999.90
clflush size    : 64
cache_alignment    : 64
address sizes    : 32 bits physical, 32 bits virtual
power management:


Anything else I can do?

-- 
Thanks in advance and kind regards 
Joerg 




----- Original Mail ----
> Von: Johannes Berg <johannes@sipsolutions.net>
> An: Joerg Pommnitz <pommnitz@yahoo.com>
> CC: Adrian Chadd <adrian@freebsd.org>; linux-wireless@vger.kernel.org; 
>ath5k-devel@lists.ath5k.org
> Gesendet: Mittwoch, den 22. Juni 2011, 15:46:56 Uhr
> Betreff: Re: Performance regression between Madwifi/net80211 and 
ath5k/mac80211
>
> Do you see different CPU usage? Does it max  out? If so can you profile
> it?
> 
> johannes
> 
> 

Re: Performance regression between Madwifi/net80211 and ath5k/mac80211

[Johannes Berg]

On Wed, 2011-06-22 at 07:41 -0700, Joerg Pommnitz wrote:

> To answer your actual question: No, there doesn't seem to be a
> difference in CPU usage as shown by top. If anything, madwifi 
> seems to use a little bit more:
> 
> madwifi: 85,5% idle, 10% si
> ath5k  : 89% idle,   10% si

That seems insignificant with so much idle.

> The CPU is:
> 
> processor    : 0
> vendor_id    : GenuineIntel
> cpu family    : 6
> model        : 13
> model name    : Intel(R) Celeron(R) M processor         1.00GHz

So it should be a fairly capable system.

I guess you'll need more help from the HW people here.

johannes

AW: Performance regression between Madwifi/net80211 and ath5k/mac80211

[Joerg Pommnitz]

> I guess you'll  need more help from the HW people here.

Unless the delay is somewhere in the generic mac80211 stack.

The older Atheros hardware is unique, because it has not just
two hardware drivers but also two 802.11 stacks.

It would be nice if somebody could do a similar test with two
different 802.11 stacks and drivers. Comparing the performance
between libertas and libertas_tf might provide an interesting
data point.

 -- 
Regards 
Joerg 

Re: Performance regression between Madwifi/net80211 and ath5k/mac80211

[Joerg Pommnitz]

> I guess you'll  need more help from the HW people here.

Bob, Bruno, Felix, Luis, Nick: Do you read?

 -- 
Regards 
Joerg 

Re: Performance regression between Madwifi/net80211 and ath5k/mac80211

[Luis R. Rodriguez]

On Wed, Jun 22, 2011 at 8:07 AM, Joerg Pommnitz <pommnitz@yahoo.com> wrote:
>> I guess you'll  need more help from the HW people here.
>
> Bob, Bruno, Felix, Luis, Nick: Do you read?

I read but I do not have time to work on this, my focus is on newer
hardware and other things. Only thing I can recommend is to stay with
the latest drivers [1] for your metrics, and if you choose a stable
release instead of bleeding edge pick the latest stable [2]. You may
want to try Oprofile to see where ath5k spends its time doing most
work.

[1] http://wireless.kernel.org/en/users/Download
[2] http://wireless.kernel.org/en/users/Download/stable

  Luis

Re: [ath5k-devel] Performance regression between Madwifi/net80211 and ath5k/mac80211

[Felix Fietkau]

On 2011-06-22 5:07 PM, Joerg Pommnitz wrote:
>>  I guess you'll  need more help from the HW people here.
>
> Bob, Bruno, Felix, Luis, Nick: Do you read?
What mac80211/ath5k version did you use? If you're using the in-kernel 
ath5k from 2.6.38, then I'd suggest testing with a recent 
compat-wireless version. I made some bug fixes and performance 
improvements in April - and I think those went into 2.6.39.

- Felix

Re: [ath5k-devel] Performance regression between Madwifi/net80211 and ath5k/mac80211

[Joerg Pommnitz]

Felix,
thanks for taking an interest in this.

To answer your questions:
Originally I ran the stock Fedora 15 wireless modules from kernel 
2.6.38.8-32.fc15.i686.
Today I reran the test with the wireless modules from compat-wireless-3.0-rc4-1.
Unfortunately there doesn't seem to be any difference. Attached are the summary 
lines
from a 100secon TCP iperf run:

ath5k from kernel 2.6.38.8-32.fc15:   [  4]  0.0-100.7 sec   265 MBytes  22.1 
Mbits/sec
ath5k from compat-wireless-3.0-rc4-1: [  4]  0.0-100.9 sec   268 MBytes  22.2 
Mbits/sec
madwifi-0.9.4-r4144-20110602:         [  4]  0.0-100.4 sec   306 MBytes  25.6 
Mbits/sec

There is obviously little performance change from 2.6.38 to the current 3.0 
version
of ath5k. And there is a significant performance loss relative to Madwifi.

The iperf tests were run in 802.11a (channel 40) in IBSS mode. Similar 
performance
differences were previously observed in AP mode as well.


-- 
Thanks and kind regards 
Joerg 




----- Original Mail ----
> Von: Felix Fietkau <nbd@openwrt.org>
> An: Joerg Pommnitz <pommnitz@yahoo.com>
> CC: linux-wireless@vger.kernel.org; ath5k-devel@lists.ath5k.org
> Gesendet: Sonntag, den 26. Juni 2011, 16:45:29 Uhr
> Betreff: Re: [ath5k-devel] Performance regression between Madwifi/net80211 and 
>ath5k/mac80211
> 
> On 2011-06-22 5:07 PM, Joerg Pommnitz wrote:
> >>  I guess  you'll  need more help from the HW people here.
> > 
> > Bob,  Bruno, Felix, Luis, Nick: Do you read?
> What mac80211/ath5k version did you  use? If you're using the in-kernel ath5k 
>from 2.6.38, 
>
> then I'd suggest testing  with a recent compat-wireless version. I made some 
>bug fixes and
> performance  improvements in April - and I think those went into 2.6.39.
> 
> - Felix
> 

Re: [ath5k-devel] Performance regression between Madwifi/net80211 and ath5k/mac80211

[Bob Copeland]

On Wed, Jun 22, 2011 at 11:07 AM, Joerg Pommnitz <pommnitz@yahoo.com> wrote:
>> I guess you'll  need more help from the HW people here.
>
> Bob, Bruno, Felix, Luis, Nick: Do you read?

I guess I would be interested to know if there are any obvious
differences in the streams. E.g. taking a monitor mode capture
near the receiver should show whether the received power is similar,
whether the # of retransmissions is significantly higher in ath5k
vs madwifi, whether packets are sent at the same bitrates, if there
are any pauses where ath5k doesn't seem to be doing anything, etc.

Would it be possible to do a few captures with your setup?
-- 
Bob Copeland %% www.bobcopeland.com

Re: [ath5k-devel] Performance regression between Madwifi/net80211 and ath5k/mac80211

[Joerg Pommnitz]

Bob,
real world work interfered back in June but now I have a few more data points.
I made a cross test with ath5k on Fedora 2.6.40 (e.g. 3.0) and Madwifi-0.9.4 on
2.6.23. The interesting observation is, that the ath5k bottleneck seems to be
on the transmission end, e.g if the iperf client is attached to ath5k, the
performance is consistently lower.

Here is a table:
ath5k-ath5k     = iperf server attached to ath5k, iperf client attached to ath5k
madwifi-ath5k   = iperf server attached to madwifi, iperf client attached to ath5k
ath5k-madwifi   = iperf server attached to ath5k, iperf client attached to madwifi
madwifi-madwifi = iperf server attached to madwifi, iperf client attached to madwifi

Note that on iperf the server is the traffic sink (and the measurement point) and
the client is the traffic source. Measured traffic was UDP as generated by
iperf -c <ip of server> -u -b 54M -t 15

The test setup was as followed:
* 59db attenuation between the two boxes (calibrated HF cable)
* Frequency 5200MHz (eg. 802.11a channel 40)
* txpower 10dBm on both sides
* ANI disabled on Madwifi

*SLOW*      *SLOW*          *FAST*          *FAST*
ath5k-ath5k madwifi-ath5k   ath5k-madwifi   madwifi-madwifi
27553680    27553680        7426440         33998160
28082880    28135800        24866520        33851160
28212240    28212240        32281200        33615960
28323960    28112280        33033840        34027560
28300440    28259280        32363520        34303920
28088760    28276920        34739040        34021680
28112280    28212240        34874280        34415640
28247520    28088760        34298040        33886440
28247520    28235760        34004040        34292160
28176960    28141680        34944840        34409760
27930000    28224000        35015400        34174560
28206360    28159320        35085960        34403880
28135800    28082880        33574800        33951120
28135800    28041720        34986000        34045200
28147560    27965280        35133000        34174560

I will run with tcpdump on a monitoring interface for the combination
ath5k sender and madwifi sender shortly. Where should I capture the trace?

-- 
Regards
Joerg


----- Ursprüngliche Message -----
> Von: Bob Copeland <me@bobcopeland.com>
> An: Joerg Pommnitz <pommnitz@yahoo.com>
> Cc: linux-wireless@vger.kernel.org; ath5k-devel@lists.ath5k.org
> Gesendet: 13:02 Donnerstag, 30.Juni 2011
> Betreff: Re: [ath5k-devel] Performance regression between Madwifi/net80211 and ath5k/mac80211
>
> On Wed, Jun 22, 2011 at 11:07 AM, Joerg Pommnitz <pommnitz@yahoo.com>
> wrote:
>>>  I guess you'll  need more help from the HW people here.
>>
>>  Bob, Bruno, Felix, Luis, Nick: Do you read?
>
> I guess I would be interested to know if there are any obvious
> differences in the streams. E.g. taking a monitor mode capture
> near the receiver should show whether the received power is similar,
> whether the # of retransmissions is significantly higher in ath5k
> vs madwifi, whether packets are sent at the same bitrates, if there
> are any pauses where ath5k doesn't seem to be doing anything, etc.
>
> Would it be possible to do a few captures with your setup?
> --
> Bob Copeland %% www.bobcopeland.com
>

Re: [ath5k-devel] Performance regression between Madwifi/net80211 and ath5k/mac80211

[Adrian Chadd]

.. does madwifi have that net80211 "aggressive mode" by default, where
it overrides the best-effort WME queue parameters to allow for
bursting?

I see exactly that difference in FreeBSD (33mbit vs 22mbit) when I
disable that aggressive mode code.

Thanks,


Adrian

Re: [ath5k-devel] Performance regression between Madwifi/net80211 and ath5k/mac80211

[Pavel Roskin]

On Wed, 19 Oct 2011 19:33:53 +0800
Adrian Chadd <adrian@freebsd.org> wrote:

> .. does madwifi have that net80211 "aggressive mode" by default, where
> it overrides the best-effort WME queue parameters to allow for
> bursting?

I tried madwifi-0.9.4 snapshot on a CM9 card, and I got:

# iwpriv ath0 get_abolt
ath0      get_abolt:218  

218 is 0xda, which means that the following capabilities are enabled:

IEEE80211_ABOLT_TURBO_PRIME
IEEE80211_ABOLT_FAST_FRAME
IEEE80211_ABOLT_BURST
IEEE80211_ABOLT_XR
IEEE80211_ABOLT_AR

And those are not enabled:

IEEE80211_ABOLT_TURBO_G
IEEE80211_ABOLT_COMPRESSION
IEEE80211_ABOLT_WME_ELE

I would just unset all capabilities and retry.

> I see exactly that difference in FreeBSD (33mbit vs 22mbit) when I
> disable that aggressive mode code.

-- 
Regards,
Pavel Roskin

Re: [ath5k-devel] Performance regression between Madwifi/net80211 and ath5k/mac80211

[Sam Leffler]

On Wed, Oct 19, 2011 at 8:55 AM, Pavel Roskin <proski@gnu.org> wrote:
> On Wed, 19 Oct 2011 19:33:53 +0800
> Adrian Chadd <adrian@freebsd.org> wrote:
>
>> .. does madwifi have that net80211 "aggressive mode" by default, where
>> it overrides the best-effort WME queue parameters to allow for
>> bursting?
>
> I tried madwifi-0.9.4 snapshot on a CM9 card, and I got:
>
> # iwpriv ath0 get_abolt
> ath0      get_abolt:218
>
> 218 is 0xda, which means that the following capabilities are enabled:
>
> IEEE80211_ABOLT_TURBO_PRIME
> IEEE80211_ABOLT_FAST_FRAME
> IEEE80211_ABOLT_BURST
> IEEE80211_ABOLT_XR
> IEEE80211_ABOLT_AR
>
> And those are not enabled:
>
> IEEE80211_ABOLT_TURBO_G
> IEEE80211_ABOLT_COMPRESSION
> IEEE80211_ABOLT_WME_ELE
>
> I would just unset all capabilities and retry.
>
>> I see exactly that difference in FreeBSD (33mbit vs 22mbit) when I
>> disable that aggressive mode code.

Bursting will get you to ~28 and FF's are likely the reset (no dynamic
turbo on freebsd).  It would be nice to have control in linux to
explicitly turn on/off bursting but the usual way people do it is to
negotiate wmm and force the txop limit.

BTW whenever you see a performance difference the first thing to look
at is packets/sec.  I've not found any linux tools that give me this a
la athstats/wlanstats on freebsd.

-Sam