From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66BFFC3A5A4 for ; Mon, 26 Aug 2019 16:45:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 47410206BA for ; Mon, 26 Aug 2019 16:45:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1733085AbfHZQp5 (ORCPT ); Mon, 26 Aug 2019 12:45:57 -0400 Received: from mx1.redhat.com ([209.132.183.28]:23514 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729441AbfHZQp5 (ORCPT ); Mon, 26 Aug 2019 12:45:57 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D924081127; Mon, 26 Aug 2019 16:45:56 +0000 (UTC) Received: from carbon (ovpn-200-29.brq.redhat.com [10.40.200.29]) by smtp.corp.redhat.com (Postfix) with ESMTP id A7D585D704; Mon, 26 Aug 2019 16:45:43 +0000 (UTC) Date: Mon, 26 Aug 2019 18:45:42 +0200 From: Jesper Dangaard Brouer To: Akshat Kakkar Cc: brouer@redhat.com, Cong Wang , NetFilter , lartc , netdev , Eric Dumazet , Toke =?UTF-8?B?SMO4aWxhbmQtSsO4cmdlbnNlbg==?= , Anton Danilov Subject: Re: Unable to create htb tc classes more than 64K Message-ID: <20190826184542.4d61b609@carbon> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Mon, 26 Aug 2019 16:45:56 +0000 (UTC) Sender: netfilter-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org On Sun, 18 Aug 2019 00:34:33 +0530 Akshat Kakkar wrote: > My goal is not just to make as many classes as possible, but also to > use them to do rate limiting per ip per server. Say, I have a list of > 10000 IPs and more than 100 servers. So simply if I want few IPs to > get speed of says 1Mbps per server but others say speed of 2 Mbps per > server. How can I achieve this without having 10000 x 100 classes. > These numbers can be large than this and hence I am looking for a > generic solution to this. As Eric Dumazet also points out indirectly, you will be creating a huge bottleneck for SMP/multi-core CPUs. As your HTB root qdisc is a serialization point for all egress traffic, that all CPUs will need to take a lock on. It sounds like your use-case is not global rate limiting, but instead the goal is to rate limit customers or services (to something significantly lower than NIC link speed). To get scalability, in this case, you can instead use the MQ qdisc (as Eric also points out). I have an example script here[1], that shows how to setup MQ as root qdisc and add HTB leafs based on how many TX-queue the interface have via /sys/class/net/$DEV/queues/tx-*/ [1] https://github.com/xdp-project/xdp-cpumap-tc/blob/master/bin/tc_mq_htb_setup_example.sh You are not done, yet. For solving the TX-queue locking congestion, the traffic needs to be redirected to the appropriate/correct TX CPUs. This can either be done with RSS (Receive Side Scaling) HW ethtool adjustment (reduce hash to IPs L3 only), or RPS (Receive Packet Steering), or with XDP cpumap redirect. The XDP cpumap redirect feature is implemented with XDP+TC BPF code here[2]. Notice, that XPS can screw with this so there is a XPS disable script here[3]. [2] https://github.com/xdp-project/xdp-cpumap-tc [3] https://github.com/xdp-project/xdp-cpumap-tc/blob/master/bin/xps_setup.sh -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer