From mboxrd@z Thu Jan 1 00:00:00 1970 From: Roman Yeryomin Subject: Re: OpenWRT wrong adjustment of fq_codel defaults (Was: fq_codel_drop vs a udp flood) Date: Fri, 6 May 2016 21:43:54 +0300 Message-ID: References: <1462125592.5535.194.camel@edumazet-glaptop3.roam.corp.google.com> <865DA393-262D-40B6-A9D3-1B978CD5F6C6@gmail.com> <1462128385.5535.200.camel@edumazet-glaptop3.roam.corp.google.com> <1462136140.5535.219.camel@edumazet-glaptop3.roam.corp.google.com> <1462201620.5535.250.camel@edumazet-glaptop3.roam.corp.google.com> <1462205669.5535.254.camel@edumazet-glaptop3.roam.corp.google.com> <1462464776.13075.18.camel@edumazet-glaptop3.roam.corp.google.com> <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> <20160506114243.4eb4f95e@redhat.com> <20160506144740.210901f5@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: make-wifi-fast@lists.bufferbloat.net, =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= , ath10k , "codel@lists.bufferbloat.net" , "netdev@vger.kernel.org" , Jonathan Morton , OpenWrt Development List , Felix Fietkau To: Jesper Dangaard Brouer Return-path: In-Reply-To: <20160506144740.210901f5@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: codel-bounces@lists.bufferbloat.net Sender: "Codel" List-Id: netdev.vger.kernel.org T24gNiBNYXkgMjAxNiBhdCAxNTo0NywgSmVzcGVyIERhbmdhYXJkIEJyb3VlciA8YnJvdWVyQHJl ZGhhdC5jb20+IHdyb3RlOgo+Cj4gSSd2ZSBjcmVhdGVkIGEgT3BlbldSVCB0aWNrZXRbMV0gb24g dGhpcyBpc3N1ZSwgYXMgaXQgc2VlbXMgdGhhdCBzb21lb25lWzJdCj4gY2xvc2VkIEZlbGl4J2Vz IE9wZW5XUlQgZW1haWwgYWNjb3VudCAoYmFkIGNob2ljZSEgZW1haWxzIGJvdW5jaW5nKS4KPiBT b3VuZHMgbGlrZSBPcGVuV1JUIGFuZCB0aGUgTEVERSBodHRwczovL3d3dy5sZWRlLXByb2plY3Qu b3JnLyBwcm9qZWN0Cj4gaXMgaW4gc29tZSBraW5kIG9mIGNvbmZsaWN0Lgo+Cj4gT3BlbldSVCB0 aWNrZXQgWzFdIGh0dHBzOi8vZGV2Lm9wZW53cnQub3JnL3RpY2tldC8yMjM0OQo+Cj4gWzJdIGh0 dHA6Ly90aHJlYWQuZ21hbmUub3JnL2dtYW5lLmNvbXAuZW1iZWRkZWQub3BlbndydC5kZXZlbC80 MDI5OC9mb2N1cz00MDMzNQoKT0ssIHNvLCBhZnRlciBwb3J0aW5nIHRoZSBwYXRjaCB0byA0LjEg b3BlbndydCBrZXJuZWwgYW5kIHBsYXlpbmcgYQpiaXQgd2l0aCBmcV9jb2RlbCBsaW1pdHMgSSB3 YXMgYWJsZSB0byBnZXQgNDIwTWJwcyBVRFAgbGlrZSB0aGlzOgp0YyBxZGlzYyByZXBsYWNlIGRl diB3bGFuMCBwYXJlbnQgOjEgZnFfY29kZWwgZmxvd3MgMTYgbGltaXQgMjU2CgpUaGlzIGlzIGNl cnRhaW5seSBiZXR0ZXIgdGhhbiAzME1icHMgYnV0IHN0aWxsIG1vcmUgdGhhbiB0d28gdGltZXMK bGVzcyB0aGFuIGJlZm9yZSAoOTAwKS4KVENQIGFsc28gaW1wcm92ZWQgYSBsaXR0bGUgKDU1MCB0 byB+NTkwKS4KCkZlbGl4LCBvdGhlcnMsIGRvIHlvdSB3YW50IHRvIHNlZSB0aGUgcG9ydGVkIHBh dGNoLCBtYXliZSBJIGRpZCBzb21ldGhpbmcgd3Jvbmc/CkRvZXNuJ3QgbG9vayBsaWtlIGl0IHdp bGwgc2F2ZSBhdGgxMGsgZnJvbSBwZXJmb3JtYW5jZSByZWdyZXNzaW9uLgoKPgo+IE9uIEZyaSwg NiBNYXkgMjAxNiAxMTo0Mjo0MyArMDIwMAo+IEplc3BlciBEYW5nYWFyZCBCcm91ZXIgPGJyb3Vl ckByZWRoYXQuY29tPiB3cm90ZToKPgo+PiBIaSBGZWxpeCwKPj4KPj4gVGhpcyBpcyBhbiBpbXBv cnRhbnQgZml4IGZvciBPcGVuV1JULCBwbGVhc2UgcmVhZCEKPj4KPj4gT3BlbldSVCBjaGFuZ2Vk IHRoZSBkZWZhdWx0IGZxX2NvZGVsIHNjaC0+bGltaXQgZnJvbSAxMDI0MCB0byAxMDI0LAo+PiB3 aXRob3V0IGFsc28gYWRqdXN0aW5nIHEtPmZsb3dzX2NudC4gIEVyaWMgZXhwbGFpbnMgYmVsb3cg dGhhdCB5b3UgbXVzdAo+PiBhbHNvIGFkanVzdCB0aGUgYnVja2V0cyAocS0+Zmxvd3NfY250KSBm b3IgdGhpcyBub3QgdG8gYnJlYWsuIChKdXN0Cj4+IGFkanVzdCBpdCB0byAxMjgpCj4+Cj4+IFBy b2JsZW1hdGljIE9wZW5XUlQgY29tbWl0IGluIHF1ZXN0aW9uOgo+PiAgaHR0cDovL2dpdC5vcGVu d3J0Lm9yZy8/cD1vcGVud3J0LmdpdDthPXBhdGNoO2g9MTJjZDY1NzgwODRlCj4+ICAxMmNkNjU3 ODA4NGUgKCJrZXJuZWw6IHJldmVydCBmcV9jb2RlbCBxdWFudHVtIG92ZXJyaWRlIHRvIHByZXZl bnQgaXQgZnJvbSBjYXVzaW5nIHRvbyBtdWNoIGNwdSBsb2FkIHdpdGggaGlnaGVyIHNwZWVkICgj MjEzMjYpIikKPj4KPj4KPj4gSSBhbHNvIGhpZ2hseSByZWNvbW1lbmQgeW91IGNoZXJyeS1waWNr IHRoaXMgdmVyeSByZWNlbnQgY29tbWl0Ogo+PiAgbmV0LW5leHQ6IDlkMTg1NjJhMjI3OCAoImZx X2NvZGVsOiBhZGQgYmF0Y2ggYWJpbGl0eSB0byBmcV9jb2RlbF9kcm9wKCkiKQo+PiAgaHR0cHM6 Ly9naXQua2VybmVsLm9yZy9kYXZlbS9uZXQtbmV4dC9jLzlkMTg1NjJhMjI3Cj4+Cj4+IFRoaXMg c2hvdWxkIGZpeCB2ZXJ5IGhpZ2ggQ1BVIHVzYWdlIGluLWNhc2UgZnFfY29kZWwgZ29lcyBpbnRv IGRyb3AgbW9kZS4KPj4gVGhlIHByb2JsZW0gaXMgdGhhdCBkcm9wIG1vZGUgd2FzIGNvbnNpZGVy ZWQgcmFyZSwgYW5kIGltcGxlbWVudGF0aW9uCj4+IHdpc2UgaXQgd2FzIGNob3NlbiB0byBiZSBt b3JlIGV4cGVuc2l2ZSAodG8gc2F2ZSBjeWNsZXMgb24gbm9ybWFsIG1vZGUpLgo+PiBVbmZvcnR1 bmF0ZWx5IGlzIGl0IGVhc3kgdG8gdHJpZ2dlciB3aXRoIGFuIFVEUCBmbG9vZC4gRHJvcCBtb2Rl IGlzCj4+IGVzcGVjaWFsbHkgZXhwZW5zaXZlIGZvciBzbWFsbGVyIGRldmljZXMsIGFzIGl0IHNj YW5zIGEgNEsgYmlnIGFycmF5LAo+PiB0aHVzIDY0IGNhY2hlIG1pc3NlcyBmb3Igc21hbGwgZGV2 aWNlcyEKPj4KPj4gVGhlIGZpeCBpcyB0byBhbGxvdyBkcm9wLW1vZGUgdG8gYnVsay1kcm9wIG1v cmUgcGFja2V0cyB3aGVuIGVudGVyaW5nCj4+IGRyb3AtbW9kZSAoZGVmYXVsdCA2NCBidWxrIGRy b3ApLiAgVGhhdCB3YXkgd2UgZG9uJ3Qgc3VkZGVubHkKPj4gZXhwZXJpZW5jZSBhIHNpZ25pZmlj YW50bHkgaGlnaGVyIHByb2Nlc3NpbmcgY29zdCBwZXIgcGFja2V0LCBidXQKPj4gaW5zdGVhZCBj YW4gYW1vcnRpemUgdGhpcy4KPj4KPj4gVG8gRXJpYywgc2hvdWxkIHdlIHJlY29tbWVuZCBPcGVu V1JUIHRvIGFkanVzdCBkZWZhdWx0IChtYXgpIDY0IGJ1bGsKPj4gZHJvcCwgZ2l2ZW4gd2UgYWxz byByZWNvbW1lbmQgYnVja2V0IHNpemUgdG8gYmUgMTI4ID8gKHRodXMgdGhlIGFtb3VudAo+PiBv ZiBtZW1vcnkgdG8gc2NhbiBpcyBsZXNzLCBidXQgdGhlaXIgQ1BVIGlzIGFsc28gbXVjaCBzbWFs bGVyKS4KPj4KPj4gLS1KZXNwZXIKPj4KPj4KPj4gT24gVGh1LCAwNSBNYXkgMjAxNiAxMjoyMzoy NyAtMDcwMCBFcmljIER1bWF6ZXQgPGVyaWMuZHVtYXpldEBnbWFpbC5jb20+IHdyb3RlOgo+Pgo+ PiA+IE9uIFRodSwgMjAxNi0wNS0wNSBhdCAxOToyNSArMDMwMCwgUm9tYW4gWWVyeW9taW4gd3Jv dGU6Cj4+ID4gPiBPbiA1IE1heSAyMDE2IGF0IDE5OjEyLCBFcmljIER1bWF6ZXQgPGVyaWMuZHVt YXpldEBnbWFpbC5jb20+IHdyb3RlOgo+PiA+ID4gPiBPbiBUaHUsIDIwMTYtMDUtMDUgYXQgMTc6 NTMgKzAzMDAsIFJvbWFuIFllcnlvbWluIHdyb3RlOgo+PiA+ID4gPgo+PiA+ID4gPj4KPj4gPiA+ ID4+IHFkaXNjIGZxX2NvZGVsIDA6IGRldiBldGgwIHJvb3QgcmVmY250IDIgbGltaXQgMTAyNHAg Zmxvd3MgMTAyNAo+PiA+ID4gPj4gcXVhbnR1bSAxNTE0IHRhcmdldCA1LjBtcyBpbnRlcnZhbCAx MDAuMG1zIGVjbgo+PiA+ID4gPj4gIFNlbnQgMTIzMDYgYnl0ZXMgMTI4IHBrdCAoZHJvcHBlZCAw LCBvdmVybGltaXRzIDAgcmVxdWV1ZXMgMCkKPj4gPiA+ID4+ICBiYWNrbG9nIDBiIDBwIHJlcXVl dWVzIDAKPj4gPiA+ID4+ICAgbWF4cGFja2V0IDAgZHJvcF9vdmVybGltaXQgMCBuZXdfZmxvd19j b3VudCAwIGVjbl9tYXJrIDAKPj4gPiA+ID4+ICAgbmV3X2Zsb3dzX2xlbiAwIG9sZF9mbG93c19s ZW4gMAo+PiA+ID4gPgo+PiA+ID4gPgo+PiA+ID4gPiBMaW1pdCBvZiAxMDI0IHBhY2tldHMgYW5k IDEwMjQgZmxvd3MgaXMgbm90IHdpc2UgSSB0aGluay4KPj4gPiA+ID4KPj4gPiA+ID4gKElmIGFs bCBidWNrZXRzIGFyZSBpbiB1c2UsIGVhY2ggYnVja2V0IGhhcyBhIHZpcnR1YWwgcXVldWUgb2Yg MSBwYWNrZXQsCj4+ID4gPiA+IHdoaWNoIGlzIGFsbW9zdCB0aGUgc2FtZSB0aGFuIGhhdmluZyBu byBxdWV1ZSBhdCBhbGwpCj4+ID4gPiA+Cj4+ID4gPiA+IEkgc3VnZ2VzdCB0byBoYXZlIGF0IGxl YXN0IDggcGFja2V0cyBwZXIgYnVja2V0LCB0byBsZXQgQ29kZWwgaGF2ZSBhCj4+ID4gPiA+IGNo YW5jZSB0byB0cmlnZ2VyLgo+PiA+ID4gPgo+PiA+ID4gPiBTbyB5b3UgY291bGQgZWl0aGVyIHJl ZHVjZSBudW1iZXIgb2YgYnVja2V0cyB0byAxMjggKGlmIG1lbW9yeSBpcwo+PiA+ID4gPiB0aWdo dCksIG9yIGluY3JlYXNlIGxpbWl0IHRvIDgxOTIuCj4+ID4gPgo+PiA+ID4gV2lsbCB0cnksIGJ1 dCB3aGF0IEkndmUgcG9zdGVkIGlzIGRlZmF1bHQsIEkgZGlkbid0IGNoYW5nZS9jb25maWd1cmUg dGhhdC4KPj4gPgo+PiA+IGZxX2NvZGVsIGhhcyBhIGRlZmF1bHQgb2YgMTAyNDAgcGFja2V0cyBh bmQgMTAyNCBidWNrZXRzLgo+PiA+Cj4+ID4gaHR0cDovL2x4ci5mcmVlLWVsZWN0cm9ucy5jb20v c291cmNlL25ldC9zY2hlZC9zY2hfZnFfY29kZWwuYyNMNDEzCj4+ID4KPj4gPiBJZiBzb21lb25l IGNoYW5nZWQgdGhhdCBpbiB0aGUgbGludXggdmFyaWFudCB5b3UgdXNlLCBoZSBwcm9iYWJseSBz aG91bGQKPj4gPiBleHBsYWluIHRoZSByYXRpb25hbGUuCj4KPiAtLQo+IEJlc3QgcmVnYXJkcywK PiAgIEplc3BlciBEYW5nYWFyZCBCcm91ZXIKPiAgIE1TYy5DUywgUHJpbmNpcGFsIEtlcm5lbCBF bmdpbmVlciBhdCBSZWQgSGF0Cj4gICBBdXRob3Igb2YgaHR0cDovL3d3dy5pcHR2LWFuYWx5emVy Lm9yZwo+ICAgTGlua2VkSW46IGh0dHA6Ly93d3cubGlua2VkaW4uY29tL2luL2Jyb3VlcgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpDb2RlbCBtYWlsaW5n IGxpc3QKQ29kZWxAbGlzdHMuYnVmZmVyYmxvYXQubmV0Cmh0dHBzOi8vbGlzdHMuYnVmZmVyYmxv YXQubmV0L2xpc3RpbmZvL2NvZGVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mail-oi0-x241.google.com ([2607:f8b0:4003:c06::241]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1aykjY-00040y-3V for ath10k@lists.infradead.org; Fri, 06 May 2016 18:44:17 +0000 Received: by mail-oi0-x241.google.com with SMTP id f63so19378496oig.0 for ; Fri, 06 May 2016 11:43:55 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20160506144740.210901f5@redhat.com> References: <1462125592.5535.194.camel@edumazet-glaptop3.roam.corp.google.com> <865DA393-262D-40B6-A9D3-1B978CD5F6C6@gmail.com> <1462128385.5535.200.camel@edumazet-glaptop3.roam.corp.google.com> <1462136140.5535.219.camel@edumazet-glaptop3.roam.corp.google.com> <1462201620.5535.250.camel@edumazet-glaptop3.roam.corp.google.com> <1462205669.5535.254.camel@edumazet-glaptop3.roam.corp.google.com> <1462464776.13075.18.camel@edumazet-glaptop3.roam.corp.google.com> <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> <20160506114243.4eb4f95e@redhat.com> <20160506144740.210901f5@redhat.com> Date: Fri, 6 May 2016 21:43:54 +0300 Message-ID: Subject: Re: OpenWRT wrong adjustment of fq_codel defaults (Was: [Codel] fq_codel_drop vs a udp flood) From: Roman Yeryomin List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "ath10k" Errors-To: ath10k-bounces+kvalo=adurom.com@lists.infradead.org To: Jesper Dangaard Brouer Cc: make-wifi-fast@lists.bufferbloat.net, =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= , Dave Taht , ath10k , "codel@lists.bufferbloat.net" , "netdev@vger.kernel.org" , Jonathan Morton , OpenWrt Development List , Felix Fietkau On 6 May 2016 at 15:47, Jesper Dangaard Brouer wrote: > > I've created a OpenWRT ticket[1] on this issue, as it seems that someone[2] > closed Felix'es OpenWRT email account (bad choice! emails bouncing). > Sounds like OpenWRT and the LEDE https://www.lede-project.org/ project > is in some kind of conflict. > > OpenWRT ticket [1] https://dev.openwrt.org/ticket/22349 > > [2] http://thread.gmane.org/gmane.comp.embedded.openwrt.devel/40298/focus=40335 OK, so, after porting the patch to 4.1 openwrt kernel and playing a bit with fq_codel limits I was able to get 420Mbps UDP like this: tc qdisc replace dev wlan0 parent :1 fq_codel flows 16 limit 256 This is certainly better than 30Mbps but still more than two times less than before (900). TCP also improved a little (550 to ~590). Felix, others, do you want to see the ported patch, maybe I did something wrong? Doesn't look like it will save ath10k from performance regression. > > On Fri, 6 May 2016 11:42:43 +0200 > Jesper Dangaard Brouer wrote: > >> Hi Felix, >> >> This is an important fix for OpenWRT, please read! >> >> OpenWRT changed the default fq_codel sch->limit from 10240 to 1024, >> without also adjusting q->flows_cnt. Eric explains below that you must >> also adjust the buckets (q->flows_cnt) for this not to break. (Just >> adjust it to 128) >> >> Problematic OpenWRT commit in question: >> http://git.openwrt.org/?p=openwrt.git;a=patch;h=12cd6578084e >> 12cd6578084e ("kernel: revert fq_codel quantum override to prevent it from causing too much cpu load with higher speed (#21326)") >> >> >> I also highly recommend you cherry-pick this very recent commit: >> net-next: 9d18562a2278 ("fq_codel: add batch ability to fq_codel_drop()") >> https://git.kernel.org/davem/net-next/c/9d18562a227 >> >> This should fix very high CPU usage in-case fq_codel goes into drop mode. >> The problem is that drop mode was considered rare, and implementation >> wise it was chosen to be more expensive (to save cycles on normal mode). >> Unfortunately is it easy to trigger with an UDP flood. Drop mode is >> especially expensive for smaller devices, as it scans a 4K big array, >> thus 64 cache misses for small devices! >> >> The fix is to allow drop-mode to bulk-drop more packets when entering >> drop-mode (default 64 bulk drop). That way we don't suddenly >> experience a significantly higher processing cost per packet, but >> instead can amortize this. >> >> To Eric, should we recommend OpenWRT to adjust default (max) 64 bulk >> drop, given we also recommend bucket size to be 128 ? (thus the amount >> of memory to scan is less, but their CPU is also much smaller). >> >> --Jesper >> >> >> On Thu, 05 May 2016 12:23:27 -0700 Eric Dumazet wrote: >> >> > On Thu, 2016-05-05 at 19:25 +0300, Roman Yeryomin wrote: >> > > On 5 May 2016 at 19:12, Eric Dumazet wrote: >> > > > On Thu, 2016-05-05 at 17:53 +0300, Roman Yeryomin wrote: >> > > > >> > > >> >> > > >> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 >> > > >> quantum 1514 target 5.0ms interval 100.0ms ecn >> > > >> Sent 12306 bytes 128 pkt (dropped 0, overlimits 0 requeues 0) >> > > >> backlog 0b 0p requeues 0 >> > > >> maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 >> > > >> new_flows_len 0 old_flows_len 0 >> > > > >> > > > >> > > > Limit of 1024 packets and 1024 flows is not wise I think. >> > > > >> > > > (If all buckets are in use, each bucket has a virtual queue of 1 packet, >> > > > which is almost the same than having no queue at all) >> > > > >> > > > I suggest to have at least 8 packets per bucket, to let Codel have a >> > > > chance to trigger. >> > > > >> > > > So you could either reduce number of buckets to 128 (if memory is >> > > > tight), or increase limit to 8192. >> > > >> > > Will try, but what I've posted is default, I didn't change/configure that. >> > >> > fq_codel has a default of 10240 packets and 1024 buckets. >> > >> > http://lxr.free-electrons.com/source/net/sched/sch_fq_codel.c#L413 >> > >> > If someone changed that in the linux variant you use, he probably should >> > explain the rationale. > > -- > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > Author of http://www.iptv-analyzer.org > LinkedIn: http://www.linkedin.com/in/brouer _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k