From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jesper Dangaard Brouer Subject: Re: OpenWRT wrong adjustment of fq_codel defaults (Was: fq_codel_drop vs a udp flood) Date: Fri, 6 May 2016 14:47:40 +0200 Message-ID: <20160506144740.210901f5@redhat.com> References: <1462125592.5535.194.camel@edumazet-glaptop3.roam.corp.google.com> <865DA393-262D-40B6-A9D3-1B978CD5F6C6@gmail.com> <1462128385.5535.200.camel@edumazet-glaptop3.roam.corp.google.com> <1462136140.5535.219.camel@edumazet-glaptop3.roam.corp.google.com> <1462201620.5535.250.camel@edumazet-glaptop3.roam.corp.google.com> <1462205669.5535.254.camel@edumazet-glaptop3.roam.corp.google.com> <1462464776.13075.18.camel@edumazet-glaptop3.roam.corp.google.com> <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> <20160506114243.4eb4f95e@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: make-wifi-fast@lists.bufferbloat.net, zajec5@gmail.com, ath10k , "codel@lists.bufferbloat.net" , "netdev@vger.kernel.org" , Jonathan Morton , Roman Yeryomin , openwrt-devel@lists.openwrt.org To: Felix Fietkau , Dave Taht Return-path: In-Reply-To: <20160506114243.4eb4f95e@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: codel-bounces@lists.bufferbloat.net Sender: "Codel" List-Id: netdev.vger.kernel.org CkkndmUgY3JlYXRlZCBhIE9wZW5XUlQgdGlja2V0WzFdIG9uIHRoaXMgaXNzdWUsIGFzIGl0IHNl ZW1zIHRoYXQgc29tZW9uZVsyXQpjbG9zZWQgRmVsaXgnZXMgT3BlbldSVCBlbWFpbCBhY2NvdW50 IChiYWQgY2hvaWNlISBlbWFpbHMgYm91bmNpbmcpLgpTb3VuZHMgbGlrZSBPcGVuV1JUIGFuZCB0 aGUgTEVERSBodHRwczovL3d3dy5sZWRlLXByb2plY3Qub3JnLyBwcm9qZWN0CmlzIGluIHNvbWUg a2luZCBvZiBjb25mbGljdC4KCk9wZW5XUlQgdGlja2V0IFsxXSBodHRwczovL2Rldi5vcGVud3J0 Lm9yZy90aWNrZXQvMjIzNDkKClsyXSBodHRwOi8vdGhyZWFkLmdtYW5lLm9yZy9nbWFuZS5jb21w LmVtYmVkZGVkLm9wZW53cnQuZGV2ZWwvNDAyOTgvZm9jdXM9NDAzMzUKCgpPbiBGcmksIDYgTWF5 IDIwMTYgMTE6NDI6NDMgKzAyMDAKSmVzcGVyIERhbmdhYXJkIEJyb3VlciA8YnJvdWVyQHJlZGhh dC5jb20+IHdyb3RlOgoKPiBIaSBGZWxpeCwKPiAKPiBUaGlzIGlzIGFuIGltcG9ydGFudCBmaXgg Zm9yIE9wZW5XUlQsIHBsZWFzZSByZWFkIQo+IAo+IE9wZW5XUlQgY2hhbmdlZCB0aGUgZGVmYXVs dCBmcV9jb2RlbCBzY2gtPmxpbWl0IGZyb20gMTAyNDAgdG8gMTAyNCwKPiB3aXRob3V0IGFsc28g YWRqdXN0aW5nIHEtPmZsb3dzX2NudC4gIEVyaWMgZXhwbGFpbnMgYmVsb3cgdGhhdCB5b3UgbXVz dAo+IGFsc28gYWRqdXN0IHRoZSBidWNrZXRzIChxLT5mbG93c19jbnQpIGZvciB0aGlzIG5vdCB0 byBicmVhay4gKEp1c3QKPiBhZGp1c3QgaXQgdG8gMTI4KQo+IAo+IFByb2JsZW1hdGljIE9wZW5X UlQgY29tbWl0IGluIHF1ZXN0aW9uOgo+ICBodHRwOi8vZ2l0Lm9wZW53cnQub3JnLz9wPW9wZW53 cnQuZ2l0O2E9cGF0Y2g7aD0xMmNkNjU3ODA4NGUKPiAgMTJjZDY1NzgwODRlICgia2VybmVsOiBy ZXZlcnQgZnFfY29kZWwgcXVhbnR1bSBvdmVycmlkZSB0byBwcmV2ZW50IGl0IGZyb20gY2F1c2lu ZyB0b28gbXVjaCBjcHUgbG9hZCB3aXRoIGhpZ2hlciBzcGVlZCAoIzIxMzI2KSIpCj4gCj4gCj4g SSBhbHNvIGhpZ2hseSByZWNvbW1lbmQgeW91IGNoZXJyeS1waWNrIHRoaXMgdmVyeSByZWNlbnQg Y29tbWl0Ogo+ICBuZXQtbmV4dDogOWQxODU2MmEyMjc4ICgiZnFfY29kZWw6IGFkZCBiYXRjaCBh YmlsaXR5IHRvIGZxX2NvZGVsX2Ryb3AoKSIpCj4gIGh0dHBzOi8vZ2l0Lmtlcm5lbC5vcmcvZGF2 ZW0vbmV0LW5leHQvYy85ZDE4NTYyYTIyNwo+IAo+IFRoaXMgc2hvdWxkIGZpeCB2ZXJ5IGhpZ2gg Q1BVIHVzYWdlIGluLWNhc2UgZnFfY29kZWwgZ29lcyBpbnRvIGRyb3AgbW9kZS4KPiBUaGUgcHJv YmxlbSBpcyB0aGF0IGRyb3AgbW9kZSB3YXMgY29uc2lkZXJlZCByYXJlLCBhbmQgaW1wbGVtZW50 YXRpb24KPiB3aXNlIGl0IHdhcyBjaG9zZW4gdG8gYmUgbW9yZSBleHBlbnNpdmUgKHRvIHNhdmUg Y3ljbGVzIG9uIG5vcm1hbCBtb2RlKS4KPiBVbmZvcnR1bmF0ZWx5IGlzIGl0IGVhc3kgdG8gdHJp Z2dlciB3aXRoIGFuIFVEUCBmbG9vZC4gRHJvcCBtb2RlIGlzCj4gZXNwZWNpYWxseSBleHBlbnNp dmUgZm9yIHNtYWxsZXIgZGV2aWNlcywgYXMgaXQgc2NhbnMgYSA0SyBiaWcgYXJyYXksCj4gdGh1 cyA2NCBjYWNoZSBtaXNzZXMgZm9yIHNtYWxsIGRldmljZXMhCj4gCj4gVGhlIGZpeCBpcyB0byBh bGxvdyBkcm9wLW1vZGUgdG8gYnVsay1kcm9wIG1vcmUgcGFja2V0cyB3aGVuIGVudGVyaW5nCj4g ZHJvcC1tb2RlIChkZWZhdWx0IDY0IGJ1bGsgZHJvcCkuICBUaGF0IHdheSB3ZSBkb24ndCBzdWRk ZW5seQo+IGV4cGVyaWVuY2UgYSBzaWduaWZpY2FudGx5IGhpZ2hlciBwcm9jZXNzaW5nIGNvc3Qg cGVyIHBhY2tldCwgYnV0Cj4gaW5zdGVhZCBjYW4gYW1vcnRpemUgdGhpcy4KPiAKPiBUbyBFcmlj LCBzaG91bGQgd2UgcmVjb21tZW5kIE9wZW5XUlQgdG8gYWRqdXN0IGRlZmF1bHQgKG1heCkgNjQg YnVsawo+IGRyb3AsIGdpdmVuIHdlIGFsc28gcmVjb21tZW5kIGJ1Y2tldCBzaXplIHRvIGJlIDEy OCA/ICh0aHVzIHRoZSBhbW91bnQKPiBvZiBtZW1vcnkgdG8gc2NhbiBpcyBsZXNzLCBidXQgdGhl aXIgQ1BVIGlzIGFsc28gbXVjaCBzbWFsbGVyKS4KPiAKPiAtLUplc3Blcgo+IAo+IAo+IE9uIFRo dSwgMDUgTWF5IDIwMTYgMTI6MjM6MjcgLTA3MDAgRXJpYyBEdW1hemV0IDxlcmljLmR1bWF6ZXRA Z21haWwuY29tPiB3cm90ZToKPiAKPiA+IE9uIFRodSwgMjAxNi0wNS0wNSBhdCAxOToyNSArMDMw MCwgUm9tYW4gWWVyeW9taW4gd3JvdGU6ICAKPiA+ID4gT24gNSBNYXkgMjAxNiBhdCAxOToxMiwg RXJpYyBEdW1hemV0IDxlcmljLmR1bWF6ZXRAZ21haWwuY29tPiB3cm90ZTogICAgCj4gPiA+ID4g T24gVGh1LCAyMDE2LTA1LTA1IGF0IDE3OjUzICswMzAwLCBSb21hbiBZZXJ5b21pbiB3cm90ZToK PiA+ID4gPiAgICAKPiA+ID4gPj4KPiA+ID4gPj4gcWRpc2MgZnFfY29kZWwgMDogZGV2IGV0aDAg cm9vdCByZWZjbnQgMiBsaW1pdCAxMDI0cCBmbG93cyAxMDI0Cj4gPiA+ID4+IHF1YW50dW0gMTUx NCB0YXJnZXQgNS4wbXMgaW50ZXJ2YWwgMTAwLjBtcyBlY24KPiA+ID4gPj4gIFNlbnQgMTIzMDYg Ynl0ZXMgMTI4IHBrdCAoZHJvcHBlZCAwLCBvdmVybGltaXRzIDAgcmVxdWV1ZXMgMCkKPiA+ID4g Pj4gIGJhY2tsb2cgMGIgMHAgcmVxdWV1ZXMgMAo+ID4gPiA+PiAgIG1heHBhY2tldCAwIGRyb3Bf b3ZlcmxpbWl0IDAgbmV3X2Zsb3dfY291bnQgMCBlY25fbWFyayAwCj4gPiA+ID4+ICAgbmV3X2Zs b3dzX2xlbiAwIG9sZF9mbG93c19sZW4gMCAgICAKPiA+ID4gPgo+ID4gPiA+Cj4gPiA+ID4gTGlt aXQgb2YgMTAyNCBwYWNrZXRzIGFuZCAxMDI0IGZsb3dzIGlzIG5vdCB3aXNlIEkgdGhpbmsuCj4g PiA+ID4KPiA+ID4gPiAoSWYgYWxsIGJ1Y2tldHMgYXJlIGluIHVzZSwgZWFjaCBidWNrZXQgaGFz IGEgdmlydHVhbCBxdWV1ZSBvZiAxIHBhY2tldCwKPiA+ID4gPiB3aGljaCBpcyBhbG1vc3QgdGhl IHNhbWUgdGhhbiBoYXZpbmcgbm8gcXVldWUgYXQgYWxsKQo+ID4gPiA+Cj4gPiA+ID4gSSBzdWdn ZXN0IHRvIGhhdmUgYXQgbGVhc3QgOCBwYWNrZXRzIHBlciBidWNrZXQsIHRvIGxldCBDb2RlbCBo YXZlIGEKPiA+ID4gPiBjaGFuY2UgdG8gdHJpZ2dlci4KPiA+ID4gPgo+ID4gPiA+IFNvIHlvdSBj b3VsZCBlaXRoZXIgcmVkdWNlIG51bWJlciBvZiBidWNrZXRzIHRvIDEyOCAoaWYgbWVtb3J5IGlz Cj4gPiA+ID4gdGlnaHQpLCBvciBpbmNyZWFzZSBsaW1pdCB0byA4MTkyLiAgICAKPiA+ID4gCj4g PiA+IFdpbGwgdHJ5LCBidXQgd2hhdCBJJ3ZlIHBvc3RlZCBpcyBkZWZhdWx0LCBJIGRpZG4ndCBj aGFuZ2UvY29uZmlndXJlIHRoYXQuICAgIAo+ID4gCj4gPiBmcV9jb2RlbCBoYXMgYSBkZWZhdWx0 IG9mIDEwMjQwIHBhY2tldHMgYW5kIDEwMjQgYnVja2V0cy4KPiA+IAo+ID4gaHR0cDovL2x4ci5m cmVlLWVsZWN0cm9ucy5jb20vc291cmNlL25ldC9zY2hlZC9zY2hfZnFfY29kZWwuYyNMNDEzCj4g PiAKPiA+IElmIHNvbWVvbmUgY2hhbmdlZCB0aGF0IGluIHRoZSBsaW51eCB2YXJpYW50IHlvdSB1 c2UsIGhlIHByb2JhYmx5IHNob3VsZAo+ID4gZXhwbGFpbiB0aGUgcmF0aW9uYWxlLiAgCgotLSAK QmVzdCByZWdhcmRzLAogIEplc3BlciBEYW5nYWFyZCBCcm91ZXIKICBNU2MuQ1MsIFByaW5jaXBh bCBLZXJuZWwgRW5naW5lZXIgYXQgUmVkIEhhdAogIEF1dGhvciBvZiBodHRwOi8vd3d3LmlwdHYt YW5hbHl6ZXIub3JnCiAgTGlua2VkSW46IGh0dHA6Ly93d3cubGlua2VkaW4uY29tL2luL2Jyb3Vl cgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpDb2RlbCBt YWlsaW5nIGxpc3QKQ29kZWxAbGlzdHMuYnVmZmVyYmxvYXQubmV0Cmh0dHBzOi8vbGlzdHMuYnVm ZmVyYmxvYXQubmV0L2xpc3RpbmZvL2NvZGVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mx1.redhat.com ([209.132.183.28]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1ayfAt-00088g-Sq for ath10k@lists.infradead.org; Fri, 06 May 2016 12:48:08 +0000 Date: Fri, 6 May 2016 14:47:40 +0200 From: Jesper Dangaard Brouer Subject: Re: OpenWRT wrong adjustment of fq_codel defaults (Was: [Codel] fq_codel_drop vs a udp flood) Message-ID: <20160506144740.210901f5@redhat.com> In-Reply-To: <20160506114243.4eb4f95e@redhat.com> References: <1462125592.5535.194.camel@edumazet-glaptop3.roam.corp.google.com> <865DA393-262D-40B6-A9D3-1B978CD5F6C6@gmail.com> <1462128385.5535.200.camel@edumazet-glaptop3.roam.corp.google.com> <1462136140.5535.219.camel@edumazet-glaptop3.roam.corp.google.com> <1462201620.5535.250.camel@edumazet-glaptop3.roam.corp.google.com> <1462205669.5535.254.camel@edumazet-glaptop3.roam.corp.google.com> <1462464776.13075.18.camel@edumazet-glaptop3.roam.corp.google.com> <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> <20160506114243.4eb4f95e@redhat.com> MIME-Version: 1.0 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "ath10k" Errors-To: ath10k-bounces+kvalo=adurom.com@lists.infradead.org To: Felix Fietkau , Dave Taht Cc: make-wifi-fast@lists.bufferbloat.net, zajec5@gmail.com, brouer@redhat.com, ath10k , "codel@lists.bufferbloat.net" , "netdev@vger.kernel.org" , Jonathan Morton , Roman Yeryomin , openwrt-devel@lists.openwrt.org I've created a OpenWRT ticket[1] on this issue, as it seems that someone[2] closed Felix'es OpenWRT email account (bad choice! emails bouncing). Sounds like OpenWRT and the LEDE https://www.lede-project.org/ project is in some kind of conflict. OpenWRT ticket [1] https://dev.openwrt.org/ticket/22349 [2] http://thread.gmane.org/gmane.comp.embedded.openwrt.devel/40298/focus=40335 On Fri, 6 May 2016 11:42:43 +0200 Jesper Dangaard Brouer wrote: > Hi Felix, > > This is an important fix for OpenWRT, please read! > > OpenWRT changed the default fq_codel sch->limit from 10240 to 1024, > without also adjusting q->flows_cnt. Eric explains below that you must > also adjust the buckets (q->flows_cnt) for this not to break. (Just > adjust it to 128) > > Problematic OpenWRT commit in question: > http://git.openwrt.org/?p=openwrt.git;a=patch;h=12cd6578084e > 12cd6578084e ("kernel: revert fq_codel quantum override to prevent it from causing too much cpu load with higher speed (#21326)") > > > I also highly recommend you cherry-pick this very recent commit: > net-next: 9d18562a2278 ("fq_codel: add batch ability to fq_codel_drop()") > https://git.kernel.org/davem/net-next/c/9d18562a227 > > This should fix very high CPU usage in-case fq_codel goes into drop mode. > The problem is that drop mode was considered rare, and implementation > wise it was chosen to be more expensive (to save cycles on normal mode). > Unfortunately is it easy to trigger with an UDP flood. Drop mode is > especially expensive for smaller devices, as it scans a 4K big array, > thus 64 cache misses for small devices! > > The fix is to allow drop-mode to bulk-drop more packets when entering > drop-mode (default 64 bulk drop). That way we don't suddenly > experience a significantly higher processing cost per packet, but > instead can amortize this. > > To Eric, should we recommend OpenWRT to adjust default (max) 64 bulk > drop, given we also recommend bucket size to be 128 ? (thus the amount > of memory to scan is less, but their CPU is also much smaller). > > --Jesper > > > On Thu, 05 May 2016 12:23:27 -0700 Eric Dumazet wrote: > > > On Thu, 2016-05-05 at 19:25 +0300, Roman Yeryomin wrote: > > > On 5 May 2016 at 19:12, Eric Dumazet wrote: > > > > On Thu, 2016-05-05 at 17:53 +0300, Roman Yeryomin wrote: > > > > > > > >> > > > >> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 > > > >> quantum 1514 target 5.0ms interval 100.0ms ecn > > > >> Sent 12306 bytes 128 pkt (dropped 0, overlimits 0 requeues 0) > > > >> backlog 0b 0p requeues 0 > > > >> maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 > > > >> new_flows_len 0 old_flows_len 0 > > > > > > > > > > > > Limit of 1024 packets and 1024 flows is not wise I think. > > > > > > > > (If all buckets are in use, each bucket has a virtual queue of 1 packet, > > > > which is almost the same than having no queue at all) > > > > > > > > I suggest to have at least 8 packets per bucket, to let Codel have a > > > > chance to trigger. > > > > > > > > So you could either reduce number of buckets to 128 (if memory is > > > > tight), or increase limit to 8192. > > > > > > Will try, but what I've posted is default, I didn't change/configure that. > > > > fq_codel has a default of 10240 packets and 1024 buckets. > > > > http://lxr.free-electrons.com/source/net/sched/sch_fq_codel.c#L413 > > > > If someone changed that in the linux variant you use, he probably should > > explain the rationale. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k