No reorder buffer?

No reorder buffer?

[Denton Gentry]

I'm bridging ath10k to Ethernet, gathering pcaps on another ath10k
system in monitor mode and on the Ethernet system which is receiving
the packets. I see packets being delivered out of order to the
Ethernet system. So far as I can tell, any packet which has to be
re-sent on the Wifi link will be delivered out of order to Ethernet.

I believe this happens because there is no reorder buffer.
ieee80211_rx_reorder_ampdu returns because tid_agg_rx[tid] is NULL.

I believe there is no reorder buffer because
ieee80211_process_addba_request is not being called.
ieee80211_process_addba_request would allocate tid_agg_rx[tid].

I believe ieee80211_process_addba_request is not being called because
the firmware processes the AddBA and does not pass it up. This is the
part I am least sure about.


Can anyone confirm or deny that this is correct: does the firmware
process the AddBA, but not have a reorder buffer in the firmware?


This is noticeable mostly because of Windows. The Windows TCP stack
seems to react more strongly to packet reordering (inferred via SACKs,
I guess), and slows down.

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Denton Gentry]

Following up: to confirm or disprove whether the lack of a reorder
buffer is a factor in abnormally low throughput from Windows STAs, I
hacked sta_info_alloc() to allocate sta->ampdu_mlme.tid_rx[0] directly
instead of allocating it when processing an AddBA from the STA.

Throughput from a Windows system running iperf went up from ~45 Mbps
to 230 Mbps. A pcap collected by the Ethernet system at the other end
of the test no longer shows frames delivered out of order.

I suspect a proper fix would involve having the firmware pass the
AddBA up to the host.


On Tue, Jun 24, 2014 at 7:47 AM, Denton Gentry <denton.gentry@gmail.com> wrote:
> I'm bridging ath10k to Ethernet, gathering pcaps on another ath10k
> system in monitor mode and on the Ethernet system which is receiving
> the packets. I see packets being delivered out of order to the
> Ethernet system. So far as I can tell, any packet which has to be
> re-sent on the Wifi link will be delivered out of order to Ethernet.
>
> I believe this happens because there is no reorder buffer.
> ieee80211_rx_reorder_ampdu returns because tid_agg_rx[tid] is NULL.
>
> I believe there is no reorder buffer because
> ieee80211_process_addba_request is not being called.
> ieee80211_process_addba_request would allocate tid_agg_rx[tid].
>
> I believe ieee80211_process_addba_request is not being called because
> the firmware processes the AddBA and does not pass it up. This is the
> part I am least sure about.
>
>
> Can anyone confirm or deny that this is correct: does the firmware
> process the AddBA, but not have a reorder buffer in the firmware?
>
>
> This is noticeable mostly because of Windows. The Windows TCP stack
> seems to react more strongly to packet reordering (inferred via SACKs,
> I guess), and slows down.

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Dave Taht]

On Wed, Jun 25, 2014 at 10:10 PM, Denton Gentry <denton.gentry@gmail.com> wrote:
> Following up: to confirm or disprove whether the lack of a reorder
> buffer is a factor in abnormally low throughput from Windows STAs, I
> hacked sta_info_alloc() to allocate sta->ampdu_mlme.tid_rx[0] directly
> instead of allocating it when processing an AddBA from the STA.

the proof of concept patch would be nice.

> Throughput from a Windows system running iperf went up from ~45 Mbps
> to 230 Mbps. A pcap collected by the Ethernet system at the other end
> of the test no longer shows frames delivered out of order.

What windows version?

What is the effect on ios and a modern linux? (before/after fix)

What does it do to latency and how does it effect multiple stations
latency and throughput?

> I suspect a proper fix would involve having the firmware pass the
> AddBA up to the host.
>
>
> On Tue, Jun 24, 2014 at 7:47 AM, Denton Gentry <denton.gentry@gmail.com> wrote:
>> I'm bridging ath10k to Ethernet, gathering pcaps on another ath10k
>> system in monitor mode and on the Ethernet system which is receiving
>> the packets. I see packets being delivered out of order to the
>> Ethernet system. So far as I can tell, any packet which has to be
>> re-sent on the Wifi link will be delivered out of order to Ethernet.
>>
>> I believe this happens because there is no reorder buffer.
>> ieee80211_rx_reorder_ampdu returns because tid_agg_rx[tid] is NULL.
>>
>> I believe there is no reorder buffer because
>> ieee80211_process_addba_request is not being called.
>> ieee80211_process_addba_request would allocate tid_agg_rx[tid].
>>
>> I believe ieee80211_process_addba_request is not being called because
>> the firmware processes the AddBA and does not pass it up. This is the
>> part I am least sure about.
>>
>>
>> Can anyone confirm or deny that this is correct: does the firmware
>> process the AddBA, but not have a reorder buffer in the firmware?
>>
>>
>> This is noticeable mostly because of Windows. The Windows TCP stack
>> seems to react more strongly to packet reordering (inferred via SACKs,
>> I guess), and slows down.
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k



-- 
Dave Täht

NSFW: https://w2.eff.org/Censorship/Internet_censorship_bills/russell_0296_indecent.article

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Adrian Chadd]

... hm. Well, in normal operation the firmware is supposed to be doing
reordering. But I don't think ath10k runs the MAC in normal ethernet
pipe mode, right? So a lot of the firmware / hardware offload bits
don't work.

Can someone verify for me which mode the MAC is in? I forget what was done.



-a


On 24 June 2014 07:47, Denton Gentry <denton.gentry@gmail.com> wrote:
> I'm bridging ath10k to Ethernet, gathering pcaps on another ath10k
> system in monitor mode and on the Ethernet system which is receiving
> the packets. I see packets being delivered out of order to the
> Ethernet system. So far as I can tell, any packet which has to be
> re-sent on the Wifi link will be delivered out of order to Ethernet.
>
> I believe this happens because there is no reorder buffer.
> ieee80211_rx_reorder_ampdu returns because tid_agg_rx[tid] is NULL.
>
> I believe there is no reorder buffer because
> ieee80211_process_addba_request is not being called.
> ieee80211_process_addba_request would allocate tid_agg_rx[tid].
>
> I believe ieee80211_process_addba_request is not being called because
> the firmware processes the AddBA and does not pass it up. This is the
> part I am least sure about.
>
>
> Can anyone confirm or deny that this is correct: does the firmware
> process the AddBA, but not have a reorder buffer in the firmware?
>
>
> This is noticeable mostly because of Windows. The Windows TCP stack
> seems to react more strongly to packet reordering (inferred via SACKs,
> I guess), and slows down.
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Adrian Chadd]

.. and to confirm, the initial design of this is that yes, the
firmware is _supposed_ to be doing all of the RX reordering and the TX
aggregation/retransmission/BAR handling and powersave. As far as the
host stack is concerned, the design was supposed to offload as much of
the data pipeline processing as possible, handling only 802.11
negotiation, encryption setup/rekeying, etc.

But ath10k did change that because, and I'm fuzzy on the details,
there's various functionality that gets hidden from you when you're
running in the full offload mode.

So yes, if the MAC isn't configured in ethernet offload mode - if it's
in one of the monitor-ish wifi modes - then the NIC and firmware
aren't doing reordering and you'll have to do it on the host.

hth,

-a

On 24 June 2014 07:47, Denton Gentry <denton.gentry@gmail.com> wrote:
> I'm bridging ath10k to Ethernet, gathering pcaps on another ath10k
> system in monitor mode and on the Ethernet system which is receiving
> the packets. I see packets being delivered out of order to the
> Ethernet system. So far as I can tell, any packet which has to be
> re-sent on the Wifi link will be delivered out of order to Ethernet.
>
> I believe this happens because there is no reorder buffer.
> ieee80211_rx_reorder_ampdu returns because tid_agg_rx[tid] is NULL.
>
> I believe there is no reorder buffer because
> ieee80211_process_addba_request is not being called.
> ieee80211_process_addba_request would allocate tid_agg_rx[tid].
>
> I believe ieee80211_process_addba_request is not being called because
> the firmware processes the AddBA and does not pass it up. This is the
> part I am least sure about.
>
>
> Can anyone confirm or deny that this is correct: does the firmware
> process the AddBA, but not have a reorder buffer in the firmware?
>
>
> This is noticeable mostly because of Windows. The Windows TCP stack
> seems to react more strongly to packet reordering (inferred via SACKs,
> I guess), and slows down.
>
> _______________________________________________
> ath10k mailing list
> ath10k@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/ath10k

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Michal Kazior]

On 26 June 2014 09:38, Adrian Chadd <adrian@freebsd.org> wrote:
> ... hm. Well, in normal operation the firmware is supposed to be doing
> reordering. But I don't think ath10k runs the MAC in normal ethernet
> pipe mode, right? So a lot of the firmware / hardware offload bits
> don't work.
>
> Can someone verify for me which mode the MAC is in? I forget what was done.

ath10k runs in Native Wifi Rx decap mode if that's what you're
interested in? But I wouldn't expect it to behave any different than
802.3 decap mode other than a different frame headers.

Maybe re-ordering is disabled implicitly when a monitor vdev is
running in firmware (e.g. when interface is in promisc mode/ in a
bridge)?


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Adrian Chadd]

On 26 June 2014 00:47, Michal Kazior <michal.kazior@tieto.com> wrote:
> On 26 June 2014 09:38, Adrian Chadd <adrian@freebsd.org> wrote:
>> ... hm. Well, in normal operation the firmware is supposed to be doing
>> reordering. But I don't think ath10k runs the MAC in normal ethernet
>> pipe mode, right? So a lot of the firmware / hardware offload bits
>> don't work.
>>
>> Can someone verify for me which mode the MAC is in? I forget what was done.
>
> ath10k runs in Native Wifi Rx decap mode if that's what you're
> interested in? But I wouldn't expect it to behave any different than
> 802.3 decap mode other than a different frame headers.
>
> Maybe re-ordering is disabled implicitly when a monitor vdev is
> running in firmware (e.g. when interface is in promisc mode/ in a
> bridge)?

Native wifi Rx decap does disable a bunch of hardware offload
features. Ugh. I'd have to go dig up the firmware and QCA9880 chipset
design docs to figure it out again.

Maybe it's time all y'all ath10k developers came out here to San Jose
for a couple of days for a meet, greet, code and learn session. It'd
give me a reason to re-study all this stuff again. :-)



-a

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Yeoh Chun-Yeow]

> Native wifi Rx decap does disable a bunch of hardware offload
> features. Ugh. I'd have to go dig up the firmware and QCA9880 chipset

Can someone share what is the native WiFi Rx Decap? What other modes
that we have in the firmware?

----
Chun-Yeow

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Kalle Valo]

Adrian Chadd <adrian@freebsd.org> writes:

> Maybe it's time all y'all ath10k developers came out here to San Jose
> for a couple of days for a meet, greet, code and learn session.

Good idea, it would be really nice to get all ath10k folks (users +
devs) into one room. There's a wireless summit Dusseldorf in the fall,
that would be one possible place:

http://wireless.kernel.org/en/developers/Summits/D%C3%BCsseldorf-2014

If people are interested about something like this, please let me know
(publically or privately).

-- 
Kalle Valo

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Michal Kazior]

On 24 June 2014 16:47, Denton Gentry <denton.gentry@gmail.com> wrote:
> I'm bridging ath10k to Ethernet, gathering pcaps on another ath10k
> system in monitor mode and on the Ethernet system which is receiving
> the packets. I see packets being delivered out of order to the
> Ethernet system. So far as I can tell, any packet which has to be
> re-sent on the Wifi link will be delivered out of order to Ethernet.

As I've mentioned in my other mail - perhaps reordering is disabled in
firmware under some circumstances.

My suspicion is monitor vdev may be involved. It is created in
firmware when you set ath10k interface into promisc mode (bridging
implies that; most sniffers set promisc mode too unless told to not
to, e.g. tcpdump -p).

Can you somehow verify if you observe the reordering problem without
bridging with ethernet (e.g. run traffic to AP w/ bridging and w/o
bridging) ? If it's too much work then forget it :-)


Michał

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Ben Greear]

On 06/26/2014 02:47 AM, Kalle Valo wrote:
> Adrian Chadd <adrian@freebsd.org> writes:
>
>> Maybe it's time all y'all ath10k developers came out here to San Jose
>> for a couple of days for a meet, greet, code and learn session.
>
> Good idea, it would be really nice to get all ath10k folks (users +
> devs) into one room. There's a wireless summit Dusseldorf in the fall,
> that would be one possible place:
>
> http://wireless.kernel.org/en/developers/Summits/D%C3%BCsseldorf-2014
>
> If people are interested about something like this, please let me know
> (publically or privately).

I'd travel to California for this, but perhaps not all the way to EU.

If the NDA issues can get worked out, we could just trade most of what
we need through non-public email and secured git repositories.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Denton Gentry]

On Thu, Jun 26, 2014 at 3:28 AM, Michal Kazior <michal.kazior@tieto.com> wrote:
> On 24 June 2014 16:47, Denton Gentry <denton.gentry@gmail.com> wrote:
>> I'm bridging ath10k to Ethernet, gathering pcaps on another ath10k
>> system in monitor mode and on the Ethernet system which is receiving
>> the packets. I see packets being delivered out of order to the
>> Ethernet system. So far as I can tell, any packet which has to be
>> re-sent on the Wifi link will be delivered out of order to Ethernet.
>
> As I've mentioned in my other mail - perhaps reordering is disabled in
> firmware under some circumstances.
>
> My suspicion is monitor vdev may be involved. It is created in
> firmware when you set ath10k interface into promisc mode (bridging
> implies that; most sniffers set promisc mode too unless told to not
> to, e.g. tcpdump -p).
>
> Can you somehow verify if you observe the reordering problem without
> bridging with ethernet (e.g. run traffic to AP w/ bridging and w/o
> bridging) ? If it's too much work then forget it :-)

Running an iperf server on my AP, to take bridging out of the picture:
iperf -s -w 1M

Using a Windows iperf client to talk to it:
iperf.exe -c 192.168.254.2 -w 1M -P 1 -i 1 -l 64K -t 10

With the unmodified ath10k (with no reorder buffer) I get 56.7 Mbps.
There is a fair degree of variability second to second, from 41 Mbps
to 76 Mbps.

With the modified driver which always allocates
sta->ampdu_mlme.tid_rx[0] I get 156 Mbps, with a much smaller degree
of variability of 152 - 161 Mbps second to second.

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k

Re: No reorder buffer?

[Avery Pennarun]

On Thu, Jun 26, 2014 at 6:27 PM, Denton Gentry <denton.gentry@gmail.com> wrote:
> On Thu, Jun 26, 2014 at 3:28 AM, Michal Kazior <michal.kazior@tieto.com> wrote:
>> On 24 June 2014 16:47, Denton Gentry <denton.gentry@gmail.com> wrote:
>>> I'm bridging ath10k to Ethernet, gathering pcaps on another ath10k
>>> system in monitor mode and on the Ethernet system which is receiving
>>> the packets. I see packets being delivered out of order to the
>>> Ethernet system. So far as I can tell, any packet which has to be
>>> re-sent on the Wifi link will be delivered out of order to Ethernet.
>>
>> As I've mentioned in my other mail - perhaps reordering is disabled in
>> firmware under some circumstances.
>>
>> My suspicion is monitor vdev may be involved. It is created in
>> firmware when you set ath10k interface into promisc mode (bridging
>> implies that; most sniffers set promisc mode too unless told to not
>> to, e.g. tcpdump -p).
>>
>> Can you somehow verify if you observe the reordering problem without
>> bridging with ethernet (e.g. run traffic to AP w/ bridging and w/o
>> bridging) ? If it's too much work then forget it :-)
>
> Running an iperf server on my AP, to take bridging out of the picture:
> iperf -s -w 1M

Note that our version of hostapd (and possibly all versions of
hostapd) automatically creates a monitor interface alongside the wlan
interface.  From what I've seen in the code, this is entirely for
legacy reasons (??) and it doesn't actually use the monitor interface
anymore for anything.  So we could probably chop out the code that
creates it.  I suspect if we don't, it will be in monitor mode even
when bridging is not active.

For this test, I guess we could also just manually delete the monitor
interface after starting hostapd.  But I'm unclear on whether the
firmware would switch reordering back on in that case.

Or maybe using a separate monitor-mode interface doesn't trigger this
with the main wlan0 interface, but I'm guessing that's not the case.

(Separately, it makes me a bit sad that enabling monitor mode changes
the behaviour of the wifi.  It defeats part of the purpose of monitor
mode.)

_______________________________________________
ath10k mailing list
ath10k@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/ath10k