[ath9k-devel] Has anyone come across tx data / delimiter underruns?

[ath9k-devel] Has anyone come across tx data / delimiter underruns?

[Adrian Chadd]

Hiya,

I have a strange problem with high UDP TX on FreeBSD, at least on the AR5416.

At high (fixed) MCS rates and with RTS disabled (thus aggregates up to
64k are formed) I see high levels of TX delimiter and data underruns
on some platforms.

The required number of delimiters for a non-encrypted stream of UDP
frames (1500 bytes) is 0. The sub-frame length is less than the
minimum frame length (256 bytes), and it's less than the configured
ampdu density (8 microseconds.)

If I artificially limit aggregate size to something lower (say 16k and
32k) I get a corresponding drop in the error rate and thus a rise in
the TX rate.

If I then set the ndelim fields to 100 (ie, 400 bytes of delimiters
between each sub-frame) then all of the problems go away. I can get
around 170mbit UDP TX at MCS14. I can try for higher later on but I
believe I'm hitting the theoretical limit (ie, the Atheros driver
inspired 32 frame max per aggregate, so ~ 48k per aggregate.)

Now, it's quite possible that it's a platform problem rather than a
NIC problem. It's also possible that I could tweak the TX FIFO
pre-fetch and DMA transfer size parameters and get a better performing
setup.

Given all of this - has anyone seen this kind of behaviour before?



Adrian

[ath9k-devel] Has anyone come across tx data / delimiter underruns?

[Peter Stuge]

Adrian Chadd wrote:
> Given all of this - has anyone seen this kind of behaviour before?

There have been numerous reports of performance problems under load
especially with 5416 which always remained uninvestigated and
unexplained - I wouldn't be surprised if it's the same, or at least
a similar, issue.


//Peter

[ath9k-devel] Has anyone come across tx data / delimiter underruns?

[Adrian Chadd]

On 19 February 2013 08:37, Peter Stuge <peter@stuge.se> wrote:
> Adrian Chadd wrote:
>> Given all of this - has anyone seen this kind of behaviour before?
>
> There have been numerous reports of performance problems under load
> especially with 5416 which always remained uninvestigated and
> unexplained - I wouldn't be surprised if it's the same, or at least
> a similar, issue.

Sure. I dislike "magic crap that isn't explained" though.

For instance - I noticed that although I can get significantly higher
TCP throughput on these devices with an AR9280 or later NIC (and I
haven't tried an AR9160 in a laptop yet, I'll do that tomorrow) the
UDP TX performance is just as odd. It boils down to the same thing -
TX delimiter and data underruns. Which make little sense on a PC
device as the PCI(e) and system bus _should_ be uncontested enough. If
I bump the delimiter count up to something silly (ie, 100), things are
fine. That's 400 bytes extra overhead per 1500 byte frame. The
reference drivers don't do this, however, hence why I'm confused why I
need to.

It's this kind of weird problem that makes me want to dust off a
PCI(e) bus protocol analyser and go spelunking.

However, I can't make any assumptions at this point. For all I know
I'm doing something daft like getting the buffer / packet / aggregate
length calculation "slightly wrong" and it's mucking up the behaviour.

At least it receives high traffic rates fine. :-)




Adrian

[ath9k-devel] Has anyone come across tx data / delimiter underruns?

[Adrian Chadd]

Ok, my "I hate magic crap" result:

In summary: it was the cardbus slot fault, not the NIC.

In long(er):

* FreeBSD's HAL didn't initialise some TX FIFO trigger level
parameters right for the AR5416, so it was defaulting to what it did
for the AR5212 (ie, starting at 1 byte.) This was how much the TX FIFO
had DMAed into it before it started firing the buffer over to the
baseband to transmit. I fixed this and I saw a drop (but not zero)
level of underruns.
* The TX FIFO isn't large enough for a full aggregate frame, so if
there's anything that interrupts this process, the TX FIFO can empty
during this process and things will end in a data or delimiter
underrun.
* After having a chat with the MAC team about this, I found a bunch of
issues in the past where either (badly?) configured or (badly?)
designed PCI/PCIe hardware wouldn't "keep up" with the requirements to
sustain TX at the higher MCS rates.
* .. and thus after doing some digging, I found that the one big
difference between my t40p and T400 laptops is that the cardbus slot
has a different idea of what the PCI latency timer value should be.

Once I set the PCI latency timer on my T400 from 0x20 to 0xa8, TX
delimiters under high UDP rates dropped down to zero and I managed to
sustain ~ 200mbit at MCS15. The theoretical maximum is  supposed to be
around 250mbit, so I have a little more digging to go. But seeing
200mbit UDP TX from the AR5416 NIC is quite shiny.

The fix? At least in my case? It's difficult to say - should the
driver poke the cardbus / pci bridge configuration?


Adrian