From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jesper Dangaard Brouer Subject: OpenWRT wrong adjustment of fq_codel defaults (Was: fq_codel_drop vs a udp flood) Date: Fri, 6 May 2016 11:42:43 +0200 Message-ID: <20160506114243.4eb4f95e@redhat.com> References: <1462125592.5535.194.camel@edumazet-glaptop3.roam.corp.google.com> <865DA393-262D-40B6-A9D3-1B978CD5F6C6@gmail.com> <1462128385.5535.200.camel@edumazet-glaptop3.roam.corp.google.com> <1462136140.5535.219.camel@edumazet-glaptop3.roam.corp.google.com> <1462201620.5535.250.camel@edumazet-glaptop3.roam.corp.google.com> <1462205669.5535.254.camel@edumazet-glaptop3.roam.corp.google.com> <1462464776.13075.18.camel@edumazet-glaptop3.roam.corp.google.com> <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: make-wifi-fast@lists.bufferbloat.net, zajec5@gmail.com, ath10k , "netdev@vger.kernel.org" , "codel@lists.bufferbloat.net" , Jonathan Morton , Roman Yeryomin To: Eric Dumazet , Felix Fietkau , Dave Taht Return-path: In-Reply-To: <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: codel-bounces@lists.bufferbloat.net Sender: "Codel" List-Id: netdev.vger.kernel.org CkhpIEZlbGl4LAoKVGhpcyBpcyBhbiBpbXBvcnRhbnQgZml4IGZvciBPcGVuV1JULCBwbGVhc2Ug cmVhZCEKCk9wZW5XUlQgY2hhbmdlZCB0aGUgZGVmYXVsdCBmcV9jb2RlbCBzY2gtPmxpbWl0IGZy b20gMTAyNDAgdG8gMTAyNCwKd2l0aG91dCBhbHNvIGFkanVzdGluZyBxLT5mbG93c19jbnQuICBF cmljIGV4cGxhaW5zIGJlbG93IHRoYXQgeW91IG11c3QKYWxzbyBhZGp1c3QgdGhlIGJ1Y2tldHMg KHEtPmZsb3dzX2NudCkgZm9yIHRoaXMgbm90IHRvIGJyZWFrLiAoSnVzdAphZGp1c3QgaXQgdG8g MTI4KQoKUHJvYmxlbWF0aWMgT3BlbldSVCBjb21taXQgaW4gcXVlc3Rpb246CiBodHRwOi8vZ2l0 Lm9wZW53cnQub3JnLz9wPW9wZW53cnQuZ2l0O2E9cGF0Y2g7aD0xMmNkNjU3ODA4NGUKIDEyY2Q2 NTc4MDg0ZSAoImtlcm5lbDogcmV2ZXJ0IGZxX2NvZGVsIHF1YW50dW0gb3ZlcnJpZGUgdG8gcHJl dmVudCBpdCBmcm9tIGNhdXNpbmcgdG9vIG11Y2ggY3B1IGxvYWQgd2l0aCBoaWdoZXIgc3BlZWQg KCMyMTMyNikiKQoKCkkgYWxzbyBoaWdobHkgcmVjb21tZW5kIHlvdSBjaGVycnktcGljayB0aGlz IHZlcnkgcmVjZW50IGNvbW1pdDoKIG5ldC1uZXh0OiA5ZDE4NTYyYTIyNzggKCJmcV9jb2RlbDog YWRkIGJhdGNoIGFiaWxpdHkgdG8gZnFfY29kZWxfZHJvcCgpIikKIGh0dHBzOi8vZ2l0Lmtlcm5l bC5vcmcvZGF2ZW0vbmV0LW5leHQvYy85ZDE4NTYyYTIyNwoKVGhpcyBzaG91bGQgZml4IHZlcnkg aGlnaCBDUFUgdXNhZ2UgaW4tY2FzZSBmcV9jb2RlbCBnb2VzIGludG8gZHJvcCBtb2RlLgpUaGUg cHJvYmxlbSBpcyB0aGF0IGRyb3AgbW9kZSB3YXMgY29uc2lkZXJlZCByYXJlLCBhbmQgaW1wbGVt ZW50YXRpb24Kd2lzZSBpdCB3YXMgY2hvc2VuIHRvIGJlIG1vcmUgZXhwZW5zaXZlICh0byBzYXZl IGN5Y2xlcyBvbiBub3JtYWwgbW9kZSkuClVuZm9ydHVuYXRlbHkgaXMgaXQgZWFzeSB0byB0cmln Z2VyIHdpdGggYW4gVURQIGZsb29kLiBEcm9wIG1vZGUgaXMKZXNwZWNpYWxseSBleHBlbnNpdmUg Zm9yIHNtYWxsZXIgZGV2aWNlcywgYXMgaXQgc2NhbnMgYSA0SyBiaWcgYXJyYXksCnRodXMgNjQg Y2FjaGUgbWlzc2VzIGZvciBzbWFsbCBkZXZpY2VzIQoKVGhlIGZpeCBpcyB0byBhbGxvdyBkcm9w LW1vZGUgdG8gYnVsay1kcm9wIG1vcmUgcGFja2V0cyB3aGVuIGVudGVyaW5nCmRyb3AtbW9kZSAo ZGVmYXVsdCA2NCBidWxrIGRyb3ApLiAgVGhhdCB3YXkgd2UgZG9uJ3Qgc3VkZGVubHkKZXhwZXJp ZW5jZSBhIHNpZ25pZmljYW50bHkgaGlnaGVyIHByb2Nlc3NpbmcgY29zdCBwZXIgcGFja2V0LCBi dXQKaW5zdGVhZCBjYW4gYW1vcnRpemUgdGhpcy4KClRvIEVyaWMsIHNob3VsZCB3ZSByZWNvbW1l bmQgT3BlbldSVCB0byBhZGp1c3QgZGVmYXVsdCAobWF4KSA2NCBidWxrCmRyb3AsIGdpdmVuIHdl IGFsc28gcmVjb21tZW5kIGJ1Y2tldCBzaXplIHRvIGJlIDEyOCA/ICh0aHVzIHRoZSBhbW91bnQK b2YgbWVtb3J5IHRvIHNjYW4gaXMgbGVzcywgYnV0IHRoZWlyIENQVSBpcyBhbHNvIG11Y2ggc21h bGxlcikuCgotLUplc3BlcgoKCk9uIFRodSwgMDUgTWF5IDIwMTYgMTI6MjM6MjcgLTA3MDAgRXJp YyBEdW1hemV0IDxlcmljLmR1bWF6ZXRAZ21haWwuY29tPiB3cm90ZToKCj4gT24gVGh1LCAyMDE2 LTA1LTA1IGF0IDE5OjI1ICswMzAwLCBSb21hbiBZZXJ5b21pbiB3cm90ZToKPiA+IE9uIDUgTWF5 IDIwMTYgYXQgMTk6MTIsIEVyaWMgRHVtYXpldCA8ZXJpYy5kdW1hemV0QGdtYWlsLmNvbT4gd3Jv dGU6ICAKPiA+ID4gT24gVGh1LCAyMDE2LTA1LTA1IGF0IDE3OjUzICswMzAwLCBSb21hbiBZZXJ5 b21pbiB3cm90ZToKPiA+ID4gIAo+ID4gPj4KPiA+ID4+IHFkaXNjIGZxX2NvZGVsIDA6IGRldiBl dGgwIHJvb3QgcmVmY250IDIgbGltaXQgMTAyNHAgZmxvd3MgMTAyNAo+ID4gPj4gcXVhbnR1bSAx NTE0IHRhcmdldCA1LjBtcyBpbnRlcnZhbCAxMDAuMG1zIGVjbgo+ID4gPj4gIFNlbnQgMTIzMDYg Ynl0ZXMgMTI4IHBrdCAoZHJvcHBlZCAwLCBvdmVybGltaXRzIDAgcmVxdWV1ZXMgMCkKPiA+ID4+ ICBiYWNrbG9nIDBiIDBwIHJlcXVldWVzIDAKPiA+ID4+ICAgbWF4cGFja2V0IDAgZHJvcF9vdmVy bGltaXQgMCBuZXdfZmxvd19jb3VudCAwIGVjbl9tYXJrIDAKPiA+ID4+ICAgbmV3X2Zsb3dzX2xl biAwIG9sZF9mbG93c19sZW4gMCAgCj4gPiA+Cj4gPiA+Cj4gPiA+IExpbWl0IG9mIDEwMjQgcGFj a2V0cyBhbmQgMTAyNCBmbG93cyBpcyBub3Qgd2lzZSBJIHRoaW5rLgo+ID4gPgo+ID4gPiAoSWYg YWxsIGJ1Y2tldHMgYXJlIGluIHVzZSwgZWFjaCBidWNrZXQgaGFzIGEgdmlydHVhbCBxdWV1ZSBv ZiAxIHBhY2tldCwKPiA+ID4gd2hpY2ggaXMgYWxtb3N0IHRoZSBzYW1lIHRoYW4gaGF2aW5nIG5v IHF1ZXVlIGF0IGFsbCkKPiA+ID4KPiA+ID4gSSBzdWdnZXN0IHRvIGhhdmUgYXQgbGVhc3QgOCBw YWNrZXRzIHBlciBidWNrZXQsIHRvIGxldCBDb2RlbCBoYXZlIGEKPiA+ID4gY2hhbmNlIHRvIHRy aWdnZXIuCj4gPiA+Cj4gPiA+IFNvIHlvdSBjb3VsZCBlaXRoZXIgcmVkdWNlIG51bWJlciBvZiBi dWNrZXRzIHRvIDEyOCAoaWYgbWVtb3J5IGlzCj4gPiA+IHRpZ2h0KSwgb3IgaW5jcmVhc2UgbGlt aXQgdG8gODE5Mi4gIAo+ID4gCj4gPiBXaWxsIHRyeSwgYnV0IHdoYXQgSSd2ZSBwb3N0ZWQgaXMg ZGVmYXVsdCwgSSBkaWRuJ3QgY2hhbmdlL2NvbmZpZ3VyZSB0aGF0LiAgCj4gCj4gZnFfY29kZWwg aGFzIGEgZGVmYXVsdCBvZiAxMDI0MCBwYWNrZXRzIGFuZCAxMDI0IGJ1Y2tldHMuCj4gCj4gaHR0 cDovL2x4ci5mcmVlLWVsZWN0cm9ucy5jb20vc291cmNlL25ldC9zY2hlZC9zY2hfZnFfY29kZWwu YyNMNDEzCj4gCj4gSWYgc29tZW9uZSBjaGFuZ2VkIHRoYXQgaW4gdGhlIGxpbnV4IHZhcmlhbnQg eW91IHVzZSwgaGUgcHJvYmFibHkgc2hvdWxkCj4gZXhwbGFpbiB0aGUgcmF0aW9uYWxlLgoKCi0t IApCZXN0IHJlZ2FyZHMsCiAgSmVzcGVyIERhbmdhYXJkIEJyb3VlcgogIE1TYy5DUywgUHJpbmNp cGFsIEtlcm5lbCBFbmdpbmVlciBhdCBSZWQgSGF0CiAgQXV0aG9yIG9mIGh0dHA6Ly93d3cuaXB0 di1hbmFseXplci5vcmcKICBMaW5rZWRJbjogaHR0cDovL3d3dy5saW5rZWRpbi5jb20vaW4vYnJv dWVyCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCkNvZGVs IG1haWxpbmcgbGlzdApDb2RlbEBsaXN0cy5idWZmZXJibG9hdC5uZXQKaHR0cHM6Ly9saXN0cy5i dWZmZXJibG9hdC5uZXQvbGlzdGluZm8vY29kZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mx1.redhat.com ([209.132.183.28]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1aycHv-0000lU-1k for ath10k@lists.infradead.org; Fri, 06 May 2016 09:43:12 +0000 Date: Fri, 6 May 2016 11:42:43 +0200 From: Jesper Dangaard Brouer Subject: OpenWRT wrong adjustment of fq_codel defaults (Was: [Codel] fq_codel_drop vs a udp flood) Message-ID: <20160506114243.4eb4f95e@redhat.com> In-Reply-To: <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> References: <1462125592.5535.194.camel@edumazet-glaptop3.roam.corp.google.com> <865DA393-262D-40B6-A9D3-1B978CD5F6C6@gmail.com> <1462128385.5535.200.camel@edumazet-glaptop3.roam.corp.google.com> <1462136140.5535.219.camel@edumazet-glaptop3.roam.corp.google.com> <1462201620.5535.250.camel@edumazet-glaptop3.roam.corp.google.com> <1462205669.5535.254.camel@edumazet-glaptop3.roam.corp.google.com> <1462464776.13075.18.camel@edumazet-glaptop3.roam.corp.google.com> <1462476207.13075.20.camel@edumazet-glaptop3.roam.corp.google.com> MIME-Version: 1.0 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "ath10k" Errors-To: ath10k-bounces+kvalo=adurom.com@lists.infradead.org To: Eric Dumazet , Felix Fietkau , Dave Taht Cc: make-wifi-fast@lists.bufferbloat.net, zajec5@gmail.com, ath10k , "netdev@vger.kernel.org" , "codel@lists.bufferbloat.net" , brouer@redhat.com, Jonathan Morton , Roman Yeryomin Hi Felix, This is an important fix for OpenWRT, please read! OpenWRT changed the default fq_codel sch->limit from 10240 to 1024, without also adjusting q->flows_cnt. Eric explains below that you must also adjust the buckets (q->flows_cnt) for this not to break. (Just adjust it to 128) Problematic OpenWRT commit in question: http://git.openwrt.org/?p=openwrt.git;a=patch;h=12cd6578084e 12cd6578084e ("kernel: revert fq_codel quantum override to prevent it from causing too much cpu load with higher speed (#21326)") I also highly recommend you cherry-pick this very recent commit: net-next: 9d18562a2278 ("fq_codel: add batch ability to fq_codel_drop()") https://git.kernel.org/davem/net-next/c/9d18562a227 This should fix very high CPU usage in-case fq_codel goes into drop mode. The problem is that drop mode was considered rare, and implementation wise it was chosen to be more expensive (to save cycles on normal mode). Unfortunately is it easy to trigger with an UDP flood. Drop mode is especially expensive for smaller devices, as it scans a 4K big array, thus 64 cache misses for small devices! The fix is to allow drop-mode to bulk-drop more packets when entering drop-mode (default 64 bulk drop). That way we don't suddenly experience a significantly higher processing cost per packet, but instead can amortize this. To Eric, should we recommend OpenWRT to adjust default (max) 64 bulk drop, given we also recommend bucket size to be 128 ? (thus the amount of memory to scan is less, but their CPU is also much smaller). --Jesper On Thu, 05 May 2016 12:23:27 -0700 Eric Dumazet wrote: > On Thu, 2016-05-05 at 19:25 +0300, Roman Yeryomin wrote: > > On 5 May 2016 at 19:12, Eric Dumazet wrote: > > > On Thu, 2016-05-05 at 17:53 +0300, Roman Yeryomin wrote: > > > > > >> > > >> qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 > > >> quantum 1514 target 5.0ms interval 100.0ms ecn > > >> Sent 12306 bytes 128 pkt (dropped 0, overlimits 0 requeues 0) > > >> backlog 0b 0p requeues 0 > > >> maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0 > > >> new_flows_len 0 old_flows_len 0 > > > > > > > > > Limit of 1024 packets and 1024 flows is not wise I think. > > > > > > (If all buckets are in use, each bucket has a virtual queue of 1 packet, > > > which is almost the same than having no queue at all) > > > > > > I suggest to have at least 8 packets per bucket, to let Codel have a > > > chance to trigger. > > > > > > So you could either reduce number of buckets to 128 (if memory is > > > tight), or increase limit to 8192. > > > > Will try, but what I've posted is default, I didn't change/configure that. > > fq_codel has a default of 10240 packets and 1024 buckets. > > http://lxr.free-electrons.com/source/net/sched/sch_fq_codel.c#L413 > > If someone changed that in the linux variant you use, he probably should > explain the rationale. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k