From mboxrd@z Thu Jan 1 00:00:00 1970 From: Michael Ma Subject: Re: qdisc spin lock Date: Wed, 20 Apr 2016 14:24:21 -0700 Message-ID: References: <1460125157.6473.434.camel@edumazet-glaptop3.roam.corp.google.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Cc: Cong Wang , Linux Kernel Network Developers To: Eric Dumazet Return-path: Received: from mail-io0-f196.google.com ([209.85.223.196]:35524 "EHLO mail-io0-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751096AbcDTVYW (ORCPT ); Wed, 20 Apr 2016 17:24:22 -0400 Received: by mail-io0-f196.google.com with SMTP id u185so8544623iod.2 for ; Wed, 20 Apr 2016 14:24:22 -0700 (PDT) In-Reply-To: <1460125157.6473.434.camel@edumazet-glaptop3.roam.corp.google.com> Sender: netdev-owner@vger.kernel.org List-ID: 2016-04-08 7:19 GMT-07:00 Eric Dumazet : > On Thu, 2016-03-31 at 16:48 -0700, Michael Ma wrote: >> I didn't really know that multiple qdiscs can be isolated using MQ so >> that each txq can be associated with a particular qdisc. Also we don't >> really have multiple interfaces... >> >> With this MQ solution we'll still need to assign transmit queues to >> different classes by doing some math on the bandwidth limit if I >> understand correctly, which seems to be less convenient compared with >> a solution purely within HTB. >> >> I assume that with this solution I can still share qdisc among >> multiple transmit queues - please let me know if this is not the case. > > Note that this MQ + HTB thing works well, unless you use a bonding > device. (Or you need the MQ+HTB on the slaves, with no way of sharing > tokens between the slaves) Actually MQ+HTB works well for small packets - like flow of 512 byte packets can be throttled by HTB using one txq without being affected by other flows with small packets. However I found using this solution large packets (10k for example) will only achieve very limited bandwidth. In my test I used MQ to assign one txq to a HTB which sets rate at 1Gbit/s, 512 byte packets can achieve the ceiling rate by using 30 threads. But sending 10k packets using 10 threads has only 10 Mbit/s with the same TC configuration. If I increase burst and cburst of HTB to some extreme large value (like 50MB) the ceiling rate can be hit. The strange thing is that I don't see this problem when using HTB as the root. So txq number seems to be a factor here - however it's really hard to understand why would it only affect larger packets. Is this a known issue? Any suggestion on how to investigate the issue further? Profiling shows that the cpu utilization is pretty low. > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=bb1d912323d5dd50e1079e389f4e964be14f0ae3 > > bonding can not really be used as a true MQ device yet. > > I might send a patch to disable this 'bonding feature' if no slave sets > a queue_id. > >