From mboxrd@z Thu Jan 1 00:00:00 1970 From: Roman Yeryomin Subject: Re: OpenWRT wrong adjustment of fq_codel defaults (Was: fq_codel_drop vs a udp flood) Date: Fri, 6 May 2016 21:56:58 +0300 Message-ID: References: <1462125592.5535.194.camel@edumazet-glaptop3.roam.corp.google.com> <865DA393-262D-40B6-A9D3-1B978CD5F6C6@gmail.com> <1462128385.5535.200.camel@edumazet-glaptop3.roam.corp.google.com> <1462136140.5535.219.camel@edumazet-glaptop3.roam.corp.google.com> <1462201620.5535.250.camel@edumazet-glaptop3.roam.corp.google.com> <1462205669.5535.254.camel@edumazet-glaptop3.roam.corp.google.com> <1462464776.13075.18.camel@edumazet-glaptop3.roam.corp.google.com> <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> <20160506114243.4eb4f95e@redhat.com> <20160506144740.210901f5@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: make-wifi-fast@lists.bufferbloat.net, =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= , ath10k , "codel@lists.bufferbloat.net" , "netdev@vger.kernel.org" , Jonathan Morton , OpenWrt Development List , Felix Fietkau To: Jesper Dangaard Brouer Return-path: In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: codel-bounces@lists.bufferbloat.net Sender: "Codel" List-Id: netdev.vger.kernel.org T24gNiBNYXkgMjAxNiBhdCAyMTo0MywgUm9tYW4gWWVyeW9taW4gPGxlcm9pLmxpc3RzQGdtYWls LmNvbT4gd3JvdGU6Cj4gT24gNiBNYXkgMjAxNiBhdCAxNTo0NywgSmVzcGVyIERhbmdhYXJkIEJy b3VlciA8YnJvdWVyQHJlZGhhdC5jb20+IHdyb3RlOgo+Pgo+PiBJJ3ZlIGNyZWF0ZWQgYSBPcGVu V1JUIHRpY2tldFsxXSBvbiB0aGlzIGlzc3VlLCBhcyBpdCBzZWVtcyB0aGF0IHNvbWVvbmVbMl0K Pj4gY2xvc2VkIEZlbGl4J2VzIE9wZW5XUlQgZW1haWwgYWNjb3VudCAoYmFkIGNob2ljZSEgZW1h aWxzIGJvdW5jaW5nKS4KPj4gU291bmRzIGxpa2UgT3BlbldSVCBhbmQgdGhlIExFREUgaHR0cHM6 Ly93d3cubGVkZS1wcm9qZWN0Lm9yZy8gcHJvamVjdAo+PiBpcyBpbiBzb21lIGtpbmQgb2YgY29u ZmxpY3QuCj4+Cj4+IE9wZW5XUlQgdGlja2V0IFsxXSBodHRwczovL2Rldi5vcGVud3J0Lm9yZy90 aWNrZXQvMjIzNDkKPj4KPj4gWzJdIGh0dHA6Ly90aHJlYWQuZ21hbmUub3JnL2dtYW5lLmNvbXAu ZW1iZWRkZWQub3BlbndydC5kZXZlbC80MDI5OC9mb2N1cz00MDMzNQo+Cj4gT0ssIHNvLCBhZnRl ciBwb3J0aW5nIHRoZSBwYXRjaCB0byA0LjEgb3BlbndydCBrZXJuZWwgYW5kIHBsYXlpbmcgYQo+ IGJpdCB3aXRoIGZxX2NvZGVsIGxpbWl0cyBJIHdhcyBhYmxlIHRvIGdldCA0MjBNYnBzIFVEUCBs aWtlIHRoaXM6Cj4gdGMgcWRpc2MgcmVwbGFjZSBkZXYgd2xhbjAgcGFyZW50IDoxIGZxX2NvZGVs IGZsb3dzIDE2IGxpbWl0IDI1NgoKRm9yZ290IHRvIG1lbnRpb24sIEkndmUgcmVkdWNlZCBkcm9w X2JhdGNoX3NpemUgZG93biB0byAzMgoKPiBUaGlzIGlzIGNlcnRhaW5seSBiZXR0ZXIgdGhhbiAz ME1icHMgYnV0IHN0aWxsIG1vcmUgdGhhbiB0d28gdGltZXMKPiBsZXNzIHRoYW4gYmVmb3JlICg5 MDApLgo+IFRDUCBhbHNvIGltcHJvdmVkIGEgbGl0dGxlICg1NTAgdG8gfjU5MCkuCj4KPiBGZWxp eCwgb3RoZXJzLCBkbyB5b3Ugd2FudCB0byBzZWUgdGhlIHBvcnRlZCBwYXRjaCwgbWF5YmUgSSBk aWQgc29tZXRoaW5nIHdyb25nPwo+IERvZXNuJ3QgbG9vayBsaWtlIGl0IHdpbGwgc2F2ZSBhdGgx MGsgZnJvbSBwZXJmb3JtYW5jZSByZWdyZXNzaW9uLgo+Cj4+Cj4+IE9uIEZyaSwgNiBNYXkgMjAx NiAxMTo0Mjo0MyArMDIwMAo+PiBKZXNwZXIgRGFuZ2FhcmQgQnJvdWVyIDxicm91ZXJAcmVkaGF0 LmNvbT4gd3JvdGU6Cj4+Cj4+PiBIaSBGZWxpeCwKPj4+Cj4+PiBUaGlzIGlzIGFuIGltcG9ydGFu dCBmaXggZm9yIE9wZW5XUlQsIHBsZWFzZSByZWFkIQo+Pj4KPj4+IE9wZW5XUlQgY2hhbmdlZCB0 aGUgZGVmYXVsdCBmcV9jb2RlbCBzY2gtPmxpbWl0IGZyb20gMTAyNDAgdG8gMTAyNCwKPj4+IHdp dGhvdXQgYWxzbyBhZGp1c3RpbmcgcS0+Zmxvd3NfY250LiAgRXJpYyBleHBsYWlucyBiZWxvdyB0 aGF0IHlvdSBtdXN0Cj4+PiBhbHNvIGFkanVzdCB0aGUgYnVja2V0cyAocS0+Zmxvd3NfY250KSBm b3IgdGhpcyBub3QgdG8gYnJlYWsuIChKdXN0Cj4+PiBhZGp1c3QgaXQgdG8gMTI4KQo+Pj4KPj4+ IFByb2JsZW1hdGljIE9wZW5XUlQgY29tbWl0IGluIHF1ZXN0aW9uOgo+Pj4gIGh0dHA6Ly9naXQu b3BlbndydC5vcmcvP3A9b3BlbndydC5naXQ7YT1wYXRjaDtoPTEyY2Q2NTc4MDg0ZQo+Pj4gIDEy Y2Q2NTc4MDg0ZSAoImtlcm5lbDogcmV2ZXJ0IGZxX2NvZGVsIHF1YW50dW0gb3ZlcnJpZGUgdG8g cHJldmVudCBpdCBmcm9tIGNhdXNpbmcgdG9vIG11Y2ggY3B1IGxvYWQgd2l0aCBoaWdoZXIgc3Bl ZWQgKCMyMTMyNikiKQo+Pj4KPj4+Cj4+PiBJIGFsc28gaGlnaGx5IHJlY29tbWVuZCB5b3UgY2hl cnJ5LXBpY2sgdGhpcyB2ZXJ5IHJlY2VudCBjb21taXQ6Cj4+PiAgbmV0LW5leHQ6IDlkMTg1NjJh MjI3OCAoImZxX2NvZGVsOiBhZGQgYmF0Y2ggYWJpbGl0eSB0byBmcV9jb2RlbF9kcm9wKCkiKQo+ Pj4gIGh0dHBzOi8vZ2l0Lmtlcm5lbC5vcmcvZGF2ZW0vbmV0LW5leHQvYy85ZDE4NTYyYTIyNwo+ Pj4KPj4+IFRoaXMgc2hvdWxkIGZpeCB2ZXJ5IGhpZ2ggQ1BVIHVzYWdlIGluLWNhc2UgZnFfY29k ZWwgZ29lcyBpbnRvIGRyb3AgbW9kZS4KPj4+IFRoZSBwcm9ibGVtIGlzIHRoYXQgZHJvcCBtb2Rl IHdhcyBjb25zaWRlcmVkIHJhcmUsIGFuZCBpbXBsZW1lbnRhdGlvbgo+Pj4gd2lzZSBpdCB3YXMg Y2hvc2VuIHRvIGJlIG1vcmUgZXhwZW5zaXZlICh0byBzYXZlIGN5Y2xlcyBvbiBub3JtYWwgbW9k ZSkuCj4+PiBVbmZvcnR1bmF0ZWx5IGlzIGl0IGVhc3kgdG8gdHJpZ2dlciB3aXRoIGFuIFVEUCBm bG9vZC4gRHJvcCBtb2RlIGlzCj4+PiBlc3BlY2lhbGx5IGV4cGVuc2l2ZSBmb3Igc21hbGxlciBk ZXZpY2VzLCBhcyBpdCBzY2FucyBhIDRLIGJpZyBhcnJheSwKPj4+IHRodXMgNjQgY2FjaGUgbWlz c2VzIGZvciBzbWFsbCBkZXZpY2VzIQo+Pj4KPj4+IFRoZSBmaXggaXMgdG8gYWxsb3cgZHJvcC1t b2RlIHRvIGJ1bGstZHJvcCBtb3JlIHBhY2tldHMgd2hlbiBlbnRlcmluZwo+Pj4gZHJvcC1tb2Rl IChkZWZhdWx0IDY0IGJ1bGsgZHJvcCkuICBUaGF0IHdheSB3ZSBkb24ndCBzdWRkZW5seQo+Pj4g ZXhwZXJpZW5jZSBhIHNpZ25pZmljYW50bHkgaGlnaGVyIHByb2Nlc3NpbmcgY29zdCBwZXIgcGFj a2V0LCBidXQKPj4+IGluc3RlYWQgY2FuIGFtb3J0aXplIHRoaXMuCj4+Pgo+Pj4gVG8gRXJpYywg c2hvdWxkIHdlIHJlY29tbWVuZCBPcGVuV1JUIHRvIGFkanVzdCBkZWZhdWx0IChtYXgpIDY0IGJ1 bGsKPj4+IGRyb3AsIGdpdmVuIHdlIGFsc28gcmVjb21tZW5kIGJ1Y2tldCBzaXplIHRvIGJlIDEy OCA/ICh0aHVzIHRoZSBhbW91bnQKPj4+IG9mIG1lbW9yeSB0byBzY2FuIGlzIGxlc3MsIGJ1dCB0 aGVpciBDUFUgaXMgYWxzbyBtdWNoIHNtYWxsZXIpLgo+Pj4KPj4+IC0tSmVzcGVyCj4+Pgo+Pj4K Pj4+IE9uIFRodSwgMDUgTWF5IDIwMTYgMTI6MjM6MjcgLTA3MDAgRXJpYyBEdW1hemV0IDxlcmlj LmR1bWF6ZXRAZ21haWwuY29tPiB3cm90ZToKPj4+Cj4+PiA+IE9uIFRodSwgMjAxNi0wNS0wNSBh dCAxOToyNSArMDMwMCwgUm9tYW4gWWVyeW9taW4gd3JvdGU6Cj4+PiA+ID4gT24gNSBNYXkgMjAx NiBhdCAxOToxMiwgRXJpYyBEdW1hemV0IDxlcmljLmR1bWF6ZXRAZ21haWwuY29tPiB3cm90ZToK Pj4+ID4gPiA+IE9uIFRodSwgMjAxNi0wNS0wNSBhdCAxNzo1MyArMDMwMCwgUm9tYW4gWWVyeW9t aW4gd3JvdGU6Cj4+PiA+ID4gPgo+Pj4gPiA+ID4+Cj4+PiA+ID4gPj4gcWRpc2MgZnFfY29kZWwg MDogZGV2IGV0aDAgcm9vdCByZWZjbnQgMiBsaW1pdCAxMDI0cCBmbG93cyAxMDI0Cj4+PiA+ID4g Pj4gcXVhbnR1bSAxNTE0IHRhcmdldCA1LjBtcyBpbnRlcnZhbCAxMDAuMG1zIGVjbgo+Pj4gPiA+ ID4+ICBTZW50IDEyMzA2IGJ5dGVzIDEyOCBwa3QgKGRyb3BwZWQgMCwgb3ZlcmxpbWl0cyAwIHJl cXVldWVzIDApCj4+PiA+ID4gPj4gIGJhY2tsb2cgMGIgMHAgcmVxdWV1ZXMgMAo+Pj4gPiA+ID4+ ICAgbWF4cGFja2V0IDAgZHJvcF9vdmVybGltaXQgMCBuZXdfZmxvd19jb3VudCAwIGVjbl9tYXJr IDAKPj4+ID4gPiA+PiAgIG5ld19mbG93c19sZW4gMCBvbGRfZmxvd3NfbGVuIDAKPj4+ID4gPiA+ Cj4+PiA+ID4gPgo+Pj4gPiA+ID4gTGltaXQgb2YgMTAyNCBwYWNrZXRzIGFuZCAxMDI0IGZsb3dz IGlzIG5vdCB3aXNlIEkgdGhpbmsuCj4+PiA+ID4gPgo+Pj4gPiA+ID4gKElmIGFsbCBidWNrZXRz IGFyZSBpbiB1c2UsIGVhY2ggYnVja2V0IGhhcyBhIHZpcnR1YWwgcXVldWUgb2YgMSBwYWNrZXQs Cj4+PiA+ID4gPiB3aGljaCBpcyBhbG1vc3QgdGhlIHNhbWUgdGhhbiBoYXZpbmcgbm8gcXVldWUg YXQgYWxsKQo+Pj4gPiA+ID4KPj4+ID4gPiA+IEkgc3VnZ2VzdCB0byBoYXZlIGF0IGxlYXN0IDgg cGFja2V0cyBwZXIgYnVja2V0LCB0byBsZXQgQ29kZWwgaGF2ZSBhCj4+PiA+ID4gPiBjaGFuY2Ug dG8gdHJpZ2dlci4KPj4+ID4gPiA+Cj4+PiA+ID4gPiBTbyB5b3UgY291bGQgZWl0aGVyIHJlZHVj ZSBudW1iZXIgb2YgYnVja2V0cyB0byAxMjggKGlmIG1lbW9yeSBpcwo+Pj4gPiA+ID4gdGlnaHQp LCBvciBpbmNyZWFzZSBsaW1pdCB0byA4MTkyLgo+Pj4gPiA+Cj4+PiA+ID4gV2lsbCB0cnksIGJ1 dCB3aGF0IEkndmUgcG9zdGVkIGlzIGRlZmF1bHQsIEkgZGlkbid0IGNoYW5nZS9jb25maWd1cmUg dGhhdC4KPj4+ID4KPj4+ID4gZnFfY29kZWwgaGFzIGEgZGVmYXVsdCBvZiAxMDI0MCBwYWNrZXRz IGFuZCAxMDI0IGJ1Y2tldHMuCj4+PiA+Cj4+PiA+IGh0dHA6Ly9seHIuZnJlZS1lbGVjdHJvbnMu Y29tL3NvdXJjZS9uZXQvc2NoZWQvc2NoX2ZxX2NvZGVsLmMjTDQxMwo+Pj4gPgo+Pj4gPiBJZiBz b21lb25lIGNoYW5nZWQgdGhhdCBpbiB0aGUgbGludXggdmFyaWFudCB5b3UgdXNlLCBoZSBwcm9i YWJseSBzaG91bGQKPj4+ID4gZXhwbGFpbiB0aGUgcmF0aW9uYWxlLgo+Pgo+PiAtLQo+PiBCZXN0 IHJlZ2FyZHMsCj4+ICAgSmVzcGVyIERhbmdhYXJkIEJyb3Vlcgo+PiAgIE1TYy5DUywgUHJpbmNp cGFsIEtlcm5lbCBFbmdpbmVlciBhdCBSZWQgSGF0Cj4+ICAgQXV0aG9yIG9mIGh0dHA6Ly93d3cu aXB0di1hbmFseXplci5vcmcKPj4gICBMaW5rZWRJbjogaHR0cDovL3d3dy5saW5rZWRpbi5jb20v aW4vYnJvdWVyCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f CkNvZGVsIG1haWxpbmcgbGlzdApDb2RlbEBsaXN0cy5idWZmZXJibG9hdC5uZXQKaHR0cHM6Ly9s aXN0cy5idWZmZXJibG9hdC5uZXQvbGlzdGluZm8vY29kZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mail-oi0-x243.google.com ([2607:f8b0:4003:c06::243]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1aykwB-0002PZ-EX for ath10k@lists.infradead.org; Fri, 06 May 2016 18:57:20 +0000 Received: by mail-oi0-x243.google.com with SMTP id i2so19447614oib.3 for ; Fri, 06 May 2016 11:56:59 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: References: <1462125592.5535.194.camel@edumazet-glaptop3.roam.corp.google.com> <865DA393-262D-40B6-A9D3-1B978CD5F6C6@gmail.com> <1462128385.5535.200.camel@edumazet-glaptop3.roam.corp.google.com> <1462136140.5535.219.camel@edumazet-glaptop3.roam.corp.google.com> <1462201620.5535.250.camel@edumazet-glaptop3.roam.corp.google.com> <1462205669.5535.254.camel@edumazet-glaptop3.roam.corp.google.com> <1462464776.13075.18.camel@edumazet-glaptop3.roam.corp.google.com> <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> <20160506114243.4eb4f95e@redhat.com> <20160506144740.210901f5@redhat.com> Date: Fri, 6 May 2016 21:56:58 +0300 Message-ID: Subject: Re: OpenWRT wrong adjustment of fq_codel defaults (Was: [Codel] fq_codel_drop vs a udp flood) From: Roman Yeryomin List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "ath10k" Errors-To: ath10k-bounces+kvalo=adurom.com@lists.infradead.org To: Jesper Dangaard Brouer Cc: make-wifi-fast@lists.bufferbloat.net, =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= , Dave Taht , ath10k , "codel@lists.bufferbloat.net" , "netdev@vger.kernel.org" , Jonathan Morton , OpenWrt Development List , Felix Fietkau On 6 May 2016 at 21:43, Roman Yeryomin wrote: > On 6 May 2016 at 15:47, Jesper Dangaard Brouer wrote: >> >> I've created a OpenWRT ticket[1] on this issue, as it seems that someone[2] >> closed Felix'es OpenWRT email account (bad choice! emails bouncing). >> Sounds like OpenWRT and the LEDE https://www.lede-project.org/ project >> is in some kind of conflict. >> >> OpenWRT ticket [1] https://dev.openwrt.org/ticket/22349 >> >> [2] http://thread.gmane.org/gmane.comp.embedded.openwrt.devel/40298/focus=40335 > > OK, so, after porting the patch to 4.1 openwrt kernel and playing a > bit with fq_codel limits I was able to get 420Mbps UDP like this: > tc qdisc replace dev wlan0 parent :1 fq_codel flows 16 limit 256 Forgot to mention, I've reduced drop_batch_size down to 32 > This is certainly better than 30Mbps but still more than two times > less than before (900). > TCP also improved a little (550 to ~590). > > Felix, others, do you want to see the ported patch, maybe I did something wrong? > Doesn't look like it will save ath10k from performance regression. > >> >> On Fri, 6 May 2016 11:42:43 +0200 >> Jesper Dangaard Brouer wrote: >> >>> Hi Felix, >>> >>> This is an important fix for OpenWRT, please read! >>> >>> OpenWRT changed the default fq_codel sch->limit from 10240 to 1024, >>> without also adjusting q->flows_cnt. Eric explains below that you must >>> also adjust the buckets (q->flows_cnt) for this not to break. (Just >>> adjust it to 128) >>> >>> Problematic OpenWRT commit in question: >>> http://git.openwrt.org/?p=openwrt.git;a=patch;h=12cd6578084e >>> 12cd6578084e ("kernel: revert fq_codel quantum override to prevent it from causing too much cpu load with higher speed (#21326)") >>> >>> >>> I also highly recommend you cherry-pick this very recent commit: >>> net-next: 9d18562a2278 ("fq_codel: add batch ability to fq_codel_drop()") >>> https://git.kernel.org/davem/net-next/c/9d18562a227 >>> >>> This should fix very high CPU usage in-case fq_codel goes into drop mode. >>> The problem is that drop mode was considered rare, and implementation >>> wise it was chosen to be more expensive (to save cycles on normal mode). >>> Unfortunately is it easy to trigger with an UDP flood. Drop mode is >>> especially expensive for smaller devices, as it scans a 4K big array, >>> thus 64 cache misses for small devices! >>> >>> The fix is to allow drop-mode to bulk-drop more packets when entering >>> drop-mode (default 64 bulk drop). That way we don't suddenly >>> experience a significantly higher processing cost per packet, but >>> instead can amortize this. >>> >>> To Eric, should we recommend OpenWRT to adjust default (max) 64 bulk >>> drop, given we also recommend bucket size to be 128 ? (thus the amount >>> of memory to scan is less, but their CPU is also much smaller). >>> >>> --Jesper >>> >>> >>> On Thu, 05 May 2016 12:23:27 -0700 Eric Dumazet wrote: >>> >>> > On Thu, 2016-05-05 at 19:25 +0300, Roman Yeryomin wrote: >>> > > On 5 May 2016 at 19:12, Eric Dumazet wrote: >>> > > > On Thu, 2016-05-05 at 17:53 +0300, Roman Yeryomin wrote: >>> > > > >>> > > >> >>> > > >> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 >>> > > >> quantum 1514 target 5.0ms interval 100.0ms ecn >>> > > >> Sent 12306 bytes 128 pkt (dropped 0, overlimits 0 requeues 0) >>> > > >> backlog 0b 0p requeues 0 >>> > > >> maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 >>> > > >> new_flows_len 0 old_flows_len 0 >>> > > > >>> > > > >>> > > > Limit of 1024 packets and 1024 flows is not wise I think. >>> > > > >>> > > > (If all buckets are in use, each bucket has a virtual queue of 1 packet, >>> > > > which is almost the same than having no queue at all) >>> > > > >>> > > > I suggest to have at least 8 packets per bucket, to let Codel have a >>> > > > chance to trigger. >>> > > > >>> > > > So you could either reduce number of buckets to 128 (if memory is >>> > > > tight), or increase limit to 8192. >>> > > >>> > > Will try, but what I've posted is default, I didn't change/configure that. >>> > >>> > fq_codel has a default of 10240 packets and 1024 buckets. >>> > >>> > http://lxr.free-electrons.com/source/net/sched/sch_fq_codel.c#L413 >>> > >>> > If someone changed that in the linux variant you use, he probably should >>> > explain the rationale. >> >> -- >> Best regards, >> Jesper Dangaard Brouer >> MSc.CS, Principal Kernel Engineer at Red Hat >> Author of http://www.iptv-analyzer.org >> LinkedIn: http://www.linkedin.com/in/brouer _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k