[RFC] iwlwifi: enable TX AMPDU for some iwldvm

[RFC] iwlwifi: enable TX AMPDU for some iwldvm

[Kevin Locke]

Hi Wireless Developers,

I have a Lenovo ThinkPad T430 (2342-CTO) with an Intel Centrino
Ultimate-N 6300 (8086:4238) wireless card.  With the help of Reventlov
and johill on #linux-wireless, we discovered that enabling TX AMPDU,
by passing module option 11n_disable=8 (IWL_ENABLE_HT_TXAGG) to
iwlwifi, increased TCP throughput significantly:

With an ASUS RT-ACRH13 AP and `iperf3 -s` running on a server with a
1Gbps wired connection, `iperf3 -R -c` on the ThinkPad increased from
63.2 to 104 Mbits/sec (using MCS 15 40MHz short GI).  With a Buffalo
WZR-600DHP running OpenWRT, `iperf3 -R -c` increased from 63.2 to
76.9.

TX AMPDU was disabled by default for all iwldvm cards in 205e2210daa9
because "iwldvm don't handle well TX AMPDU".  However, I have been
using this configuration for >2 weeks without any issues or measurable
change in ping times to the AP.  Are there any other potential
side-effects I can check?

Would it be possible to enable TX AMPDU by default for at least some
of these cards?  If so, what additional information would be required?

Thanks for considering,
Kevin

Re: [linuxwifi] [RFC] iwlwifi: enable TX AMPDU for some iwldvm

[Luciano Coelho]

Hi Kevin,

Johannes (johill) is also in this list. :)

If this feature was explicitly disabled, it certainly means that
something was causing problems with it, so I'd be wary to enable it for
all DVM devices.

I guess we could enable it by default for devices that work fine, but
we would have to run it in real life for a lot longer with a lot more
different APs to be sure it won't cause any problems.

Emmanuel, do you happen to remember what was the issue, so Kevin could
test that specific scenario with this specific NIC?

Kevin, meanwhile you could add the option to your default module
configuration so it would be used by default on your machine.

--
Cheers,
Luca.


On Tue, 2019-03-12 at 11:12 -0600, Kevin Locke wrote:
> Hi Wireless Developers,
> 
> I have a Lenovo ThinkPad T430 (2342-CTO) with an Intel Centrino
> Ultimate-N 6300 (8086:4238) wireless card.  With the help of
> Reventlov
> and johill on #linux-wireless, we discovered that enabling TX AMPDU,
> by passing module option 11n_disable=8 (IWL_ENABLE_HT_TXAGG) to
> iwlwifi, increased TCP throughput significantly:
> 
> With an ASUS RT-ACRH13 AP and `iperf3 -s` running on a server with a
> 1Gbps wired connection, `iperf3 -R -c` on the ThinkPad increased from
> 63.2 to 104 Mbits/sec (using MCS 15 40MHz short GI).  With a Buffalo
> WZR-600DHP running OpenWRT, `iperf3 -R -c` increased from 63.2 to
> 76.9.
> 
> TX AMPDU was disabled by default for all iwldvm cards in 205e2210daa9
> because "iwldvm don't handle well TX AMPDU".  However, I have been
> using this configuration for >2 weeks without any issues or
> measurable
> change in ping times to the AP.  Are there any other potential
> side-effects I can check?
> 
> Would it be possible to enable TX AMPDU by default for at least some
> of these cards?  If so, what additional information would be
> required?
> 
> Thanks for considering,
> Kevin
> -------------------------------------
> linuxwifi@eclists.intel.com
> https://eclists.intel.com/sympa/info/linuxwifi
> Unsubscribe by sending email to sympa@eclists.intel.com with subject
> "Unsubscribe linuxwifi"

Re: [linuxwifi] [RFC] iwlwifi: enable TX AMPDU for some iwldvm

[Grumbach, Emmanuel]

On Tue, 2019-03-12 at 21:38 +0200, Luciano Coelho wrote:
> Hi Kevin,
> 
> Johannes (johill) is also in this list. :)
> 
> If this feature was explicitly disabled, it certainly means that
> something was causing problems with it, so I'd be wary to enable it
> for
> all DVM devices.
> 
> I guess we could enable it by default for devices that work fine, but
> we would have to run it in real life for a lot longer with a lot more
> different APs to be sure it won't cause any problems.
> 
> Emmanuel, do you happen to remember what was the issue, so Kevin
> could
> test that specific scenario with this specific NIC?

long long nights of despair.

We had issues with reclaim path upon BACK. This is of course a firmware
problem...

> 
> Kevin, meanwhile you could add the option to your default module
> configuration so it would be used by default on your machine.
> 
> --
> Cheers,
> Luca.
> 
> 
> On Tue, 2019-03-12 at 11:12 -0600, Kevin Locke wrote:
> > Hi Wireless Developers,
> > 
> > I have a Lenovo ThinkPad T430 (2342-CTO) with an Intel Centrino
> > Ultimate-N 6300 (8086:4238) wireless card.  With the help of
> > Reventlov
> > and johill on #linux-wireless, we discovered that enabling TX
> > AMPDU,
> > by passing module option 11n_disable=8 (IWL_ENABLE_HT_TXAGG) to
> > iwlwifi, increased TCP throughput significantly:
> > 
> > With an ASUS RT-ACRH13 AP and `iperf3 -s` running on a server with
> > a
> > 1Gbps wired connection, `iperf3 -R -c` on the ThinkPad increased
> > from
> > 63.2 to 104 Mbits/sec (using MCS 15 40MHz short GI).  With a
> > Buffalo
> > WZR-600DHP running OpenWRT, `iperf3 -R -c` increased from 63.2 to
> > 76.9.
> > 
> > TX AMPDU was disabled by default for all iwldvm cards in
> > 205e2210daa9
> > because "iwldvm don't handle well TX AMPDU".  However, I have been
> > using this configuration for >2 weeks without any issues or
> > measurable
> > change in ping times to the AP.  Are there any other potential
> > side-effects I can check?
> > 
> > Would it be possible to enable TX AMPDU by default for at least
> > some
> > of these cards?  If so, what additional information would be
> > required?
> > 
> > Thanks for considering,
> > Kevin
> > -------------------------------------
> > linuxwifi@eclists.intel.com
> > https://eclists.intel.com/sympa/info/linuxwifi
> > Unsubscribe by sending email to sympa@eclists.intel.com with
> > subject
> > "Unsubscribe linuxwifi"
> 
> 

Re: [linuxwifi] [RFC] iwlwifi: enable TX AMPDU for some iwldvm

[Kevin Locke]

Thanks Luca and Emmanuel,

On Tue, 2019-03-12 at 19:47 +0000, Grumbach, Emmanuel wrote:
> On Tue, 2019-03-12 at 21:38 +0200, Luciano Coelho wrote:
>> If this feature was explicitly disabled, it certainly means that
>> something was causing problems with it, so I'd be wary to enable it
>> for all DVM devices.
>> 
>> I guess we could enable it by default for devices that work fine, but
>> we would have to run it in real life for a lot longer with a lot more
>> different APs to be sure it won't cause any problems.

Makes sense to me.  I can start testing with available APs.  Let me
know if there is anything specific I should look for or catalog beyond
latency, packet drop rates, and bandwidth.

>> Emmanuel, do you happen to remember what was the issue, so Kevin
>> could test that specific scenario with this specific NIC?
> 
> long long nights of despair.
> 
> We had issues with reclaim path upon BACK. This is of course a firmware
> problem...

Does that suggest the issue may have been fixed by a firmware update?
For reference, I'm currently using "firmware version 9.221.4.1 build
25532" from the firmware-iwlwifi Debian package (version 20190114-1).

If it would be helpful, I could attempt to bisect the firmware
revisions to find the one that fixed it (assuming I can reproduce the
issue with a previous firmware version).

Thanks again,
Kevin

Re: [linuxwifi] [RFC] iwlwifi: enable TX AMPDU for some iwldvm

[Grumbach, Emmanuel]

On Tue, 2019-03-12 at 14:31 -0600, Kevin Locke wrote:
> Thanks Luca and Emmanuel,
> 
> On Tue, 2019-03-12 at 19:47 +0000, Grumbach, Emmanuel wrote:
> > On Tue, 2019-03-12 at 21:38 +0200, Luciano Coelho wrote:
> > > If this feature was explicitly disabled, it certainly means that
> > > something was causing problems with it, so I'd be wary to enable
> > > it
> > > for all DVM devices.
> > > 
> > > I guess we could enable it by default for devices that work fine,
> > > but
> > > we would have to run it in real life for a lot longer with a lot
> > > more
> > > different APs to be sure it won't cause any problems.
> 
> Makes sense to me.  I can start testing with available APs.  Let me
> know if there is anything specific I should look for or catalog
> beyond
> latency, packet drop rates, and bandwidth.
> 
> > > Emmanuel, do you happen to remember what was the issue, so Kevin
> > > could test that specific scenario with this specific NIC?
> > 
> > long long nights of despair.
> > 
> > We had issues with reclaim path upon BACK. This is of course a
> > firmware
> > problem...
> 
> Does that suggest the issue may have been fixed by a firmware update?
> For reference, I'm currently using "firmware version 9.221.4.1 build
> 25532" from the firmware-iwlwifi Debian package (version 20190114-1).
> 
> If it would be helpful, I could attempt to bisect the firmware
> revisions to find the one that fixed it (assuming I can reproduce the
> issue with a previous firmware version).

Well.. Sorry, I wasn't very "technical".
So the problem was really that we stopped getting BACK notifications
from the firmware and that caused a reclaim stall which in turn was
caught by a Tx queue stuck timer firing in the driver.
I was never able to reproduce this. What I can do is to enable A-MPDU
on my old system that has this same device, just to see what happens.
While chasing this bug, I even found another one which bought me a few
moments of fame:

commit d6ee27eb13beab94056e0de52d81220058ca2297
Author: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Date:   Wed Jun 6 09:13:36 2012 +0200

    iwlwifi: don't mess up the SCD when removing a key

and in the commit message of that very commit:

    This doesn't seem to fix the higher queues that get stuck
    from time to time.

There were no new versions of the firmware released since then.
I tried to skim through bugzilla, but couldn't find the bugs I was
handling then.

> 
> Thanks again,
> Kevin

Re: [linuxwifi] [RFC] iwlwifi: enable TX AMPDU for some iwldvm

[Kevin Locke]

On Tue, 2019-03-12 at 20:48 +0000, Grumbach, Emmanuel wrote:
> On Tue, 2019-03-12 at 14:31 -0600, Kevin Locke wrote:
>> On Tue, 2019-03-12 at 19:47 +0000, Grumbach, Emmanuel wrote:
>>> We had issues with reclaim path upon BACK. This is of course a
>>> firmware problem...
>> 
>> Does that suggest the issue may have been fixed by a firmware update?
>> For reference, I'm currently using "firmware version 9.221.4.1 build
>> 25532" from the firmware-iwlwifi Debian package (version 20190114-1).
>> 
>> If it would be helpful, I could attempt to bisect the firmware
>> revisions to find the one that fixed it (assuming I can reproduce the
>> issue with a previous firmware version).
> 
> Well.. Sorry, I wasn't very "technical".
> So the problem was really that we stopped getting BACK notifications
> from the firmware and that caused a reclaim stall which in turn was
> caught by a Tx queue stuck timer firing in the driver.
> I was never able to reproduce this. What I can do is to enable A-MPDU
> on my old system that has this same device, just to see what happens.

Thanks for the additional details and for offering to try it out, that
would be great!

> While chasing this bug, I even found another one which bought me a few
> moments of fame:
> 
> commit d6ee27eb13beab94056e0de52d81220058ca2297
> Author: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
> Date:   Wed Jun 6 09:13:36 2012 +0200
> 
>     iwlwifi: don't mess up the SCD when removing a key
> 
> and in the commit message of that very commit:
> 
>     This doesn't seem to fix the higher queues that get stuck
>     from time to time.
> 
> There were no new versions of the firmware released since then.
> I tried to skim through bugzilla, but couldn't find the bugs I was
> handling then.

Ah.  You are right about the firmware version.  I should have checked.

I see what you mean.  I found several reports for TX queue stuck
issues in Bugzilla.  Perhaps this is one (or one of its many dups):
https://bugzilla.kernel.org/show_bug.cgi?id=56581

Let me know if there is anything I can do to help search or test.

Thanks again,
Kevin

Re: [linuxwifi] [RFC] iwlwifi: enable TX AMPDU for some iwldvm

[Grumbach, Emmanuel]

On Tue, 2019-03-12 at 15:44 -0600, Kevin Locke wrote:
> On Tue, 2019-03-12 at 20:48 +0000, Grumbach, Emmanuel wrote:
> > On Tue, 2019-03-12 at 14:31 -0600, Kevin Locke wrote:
> > > On Tue, 2019-03-12 at 19:47 +0000, Grumbach, Emmanuel wrote:
> > > > We had issues with reclaim path upon BACK. This is of course a
> > > > firmware problem...
> > > 
> > > Does that suggest the issue may have been fixed by a firmware
> > > update?
> > > For reference, I'm currently using "firmware version 9.221.4.1
> > > build
> > > 25532" from the firmware-iwlwifi Debian package (version
> > > 20190114-1).
> > > 
> > > If it would be helpful, I could attempt to bisect the firmware
> > > revisions to find the one that fixed it (assuming I can reproduce
> > > the
> > > issue with a previous firmware version).
> > 
> > Well.. Sorry, I wasn't very "technical".
> > So the problem was really that we stopped getting BACK
> > notifications
> > from the firmware and that caused a reclaim stall which in turn was
> > caught by a Tx queue stuck timer firing in the driver.
> > I was never able to reproduce this. What I can do is to enable A-
> > MPDU
> > on my old system that has this same device, just to see what
> > happens.
> 
> Thanks for the additional details and for offering to try it out,
> that
> would be great!

Just to align on expectations, I don't feel like enable A-MPDU by
default regardless of what will come out of this. People stopped
complaining after we disabled A-MPDU and the very very very few people
that did need more throughput knew how to enable them with the module
parameter. So, I don't plan to re-enable A-MPDU by default.

> 
> > While chasing this bug, I even found another one which bought me a
> > few
> > moments of fame:
> > 
> > commit d6ee27eb13beab94056e0de52d81220058ca2297
> > Author: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
> > Date:   Wed Jun 6 09:13:36 2012 +0200
> > 
> >     iwlwifi: don't mess up the SCD when removing a key
> > 
> > and in the commit message of that very commit:
> > 
> >     This doesn't seem to fix the higher queues that get stuck
> >     from time to time.
> > 
> > There were no new versions of the firmware released since then.
> > I tried to skim through bugzilla, but couldn't find the bugs I was
> > handling then.
> 
> Ah.  You are right about the firmware version.  I should have
> checked.
> 
> I see what you mean.  I found several reports for TX queue stuck
> issues in Bugzilla.  Perhaps this is one (or one of its many dups):
> https://bugzilla.kernel.org/show_bug.cgi?id=56581
> 

And those were very very time consuming. I don't want to go there
again. These are very old devices (they were described as old in 2014
already...).

> Let me know if there is anything I can do to help search or test.
> 
> Thanks again,
> Kevin

Re: [linuxwifi] [RFC] iwlwifi: enable TX AMPDU for some iwldvm

[Kevin Locke]

On Wed, 2019-03-13 at 04:58 +0000, Grumbach, Emmanuel wrote:
> Just to align on expectations, I don't feel like enable A-MPDU by
> default regardless of what will come out of this. People stopped
> complaining after we disabled A-MPDU and the very very very few people
> that did need more throughput knew how to enable them with the module
> parameter. So, I don't plan to re-enable A-MPDU by default.

I understand.  I disagree that very few people would benefit from
nearly doubling TCP throughput or that the fix is known/obvious.
However, if it's not worth the risk and/or effort, or there is no
reliable way to determine which devices/configurations are unaffected,
so be it.  Resources are limited and that's totally understandable.

Let me know if there is anything else I can do to assist.

Thanks,
Kevin