From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46C5BC433DF for ; Sun, 17 May 2020 19:35:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2A2B12065F for ; Sun, 17 May 2020 19:35:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MbPWHsZu" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726693AbgEQTf6 (ORCPT ); Sun, 17 May 2020 15:35:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726269AbgEQTf5 (ORCPT ); Sun, 17 May 2020 15:35:57 -0400 Received: from mail-oo1-xc43.google.com (mail-oo1-xc43.google.com [IPv6:2607:f8b0:4864:20::c43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44017C061A0C for ; Sun, 17 May 2020 12:35:57 -0700 (PDT) Received: by mail-oo1-xc43.google.com with SMTP id s139so1603078oos.1 for ; Sun, 17 May 2020 12:35:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=iv4Y8/o5egzRhCBh31xKdiplT7WXa20FNOa2hJ3YvWI=; b=MbPWHsZuMS0aGofE3MNEaXsWRXlnUuGl5myoKwvfsEX1N049JgGfhYXvsdW/OzF5+o tkYqg+oiHtKxOt9CTjXx4FbLf7kVUDd4VctGFkzgXU4LmDKeCIuW/mtVhhoIzjkvA5pf D1qbCDirX4/RLLyqflOLSB9pnOtpaFUaJP/1mv7FwVj4hh2XaUnppDbzvOjblsLw0xYv 2K6sztiRbV9xfJbQ5t2l2UH41BBlrUSiLNXaKpyUvgAV6CqzsMsucsIRHY6mU/9oOnUx WVJjYt0veaCirYod8oeX9xpJveMIrA4Q6QoC+mGLFo0nX0t+h7eiZ2cYJpbcD+oqf2qH jJqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=iv4Y8/o5egzRhCBh31xKdiplT7WXa20FNOa2hJ3YvWI=; b=Rtct0Ub7XKm+bQyNBkcHs2+GFVXiQTOFiocLdJJ205UNx0K5F17HDAEp4ithlywWkp 72RJaNVdvqzXNI0Wc/SfGvgD5OFm4ti5c9AJIP1EBDAKlVvE+B32U81JDptzBXT1PIo7 BrbbeRnV9VgthcUtkgtWN1m4FQXPk5ELcd2uwUtwP8m6T1qQkXuRaj3S4pmWcq6/laVO WGhiRLLDbI84dJVZ8kPnskQn8ddKN5rcKr7abu7qgMe5bISXfsfFJ2+zkOrFn7SilFlJ pa/Q2s/nIkPj4zBf96UgZW5VdcN/8AdfeGFhT50r6mYIG0jBzsaUjocYWOqqrlJ5q998 Y0dA== X-Gm-Message-State: AOAM530dca0y5/v/VEZKaCC/8K7Xme/MdaBps/WJo+duiMC707uOkbGC pQI+4Uus8NQvZZ6SGd2DWQ2gJV8Fa0SUcjxHYzSsA+tt X-Google-Smtp-Source: ABdhPJyO/CDjBZgQPxTQOt6ddSQVfI7kKiYkO7KG6fWFno2lQ1/MRNItuBFv/lmp6tZnBZjkPI2HHiijMAeAD8Qn0h0= X-Received: by 2002:a4a:e702:: with SMTP id y2mr3139592oou.44.1589744155753; Sun, 17 May 2020 12:35:55 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Cong Wang Date: Sun, 17 May 2020 12:35:44 -0700 Message-ID: Subject: Re: iproute2: tc deletion freezes whole server To: =?UTF-8?Q?V=C3=A1clav_Zindulka?= Cc: Linux Kernel Network Developers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Fri, May 8, 2020 at 6:59 AM V=C3=A1clav Zindulka wrote: > > On Thu, May 7, 2020 at 8:52 PM Cong Wang wrote= : > > > > On Tue, May 5, 2020 at 1:46 AM V=C3=A1clav Zindulka > > wrote: > > > > > > On Mon, May 4, 2020 at 7:46 PM Cong Wang w= rote: > > > > > > > > Sorry for the delay. I lost connection to my dev machine, I am tryi= ng > > > > to setup this on my own laptop. > > > > > > Sorry to hear that. I will gladly give you access to my testing > > > machine where all this nasty stuff happens every time so you can test > > > it in place. You can try everything there and have online results. I > > > can give you access even to the IPMI console so you can switch the > > > kernel during boot easily. I didn't notice this problem until the tim= e > > > of deployment. My prior testing machines were with metallic ethernet > > > ports only so I didn't know about those problems earlier. > > > > Thanks for the offer! No worries, I setup a testing VM on my laptop. > > OK > > > > > > > > > I tried to emulate your test case in my VM, here is the script I us= e: > > > > > > > > =3D=3D=3D=3D > > > > ip li set dev dummy0 up > > > > tc qd add dev dummy0 root handle 1: htb default 1 > > > > for i in `seq 1 1000` > > > > do > > > > tc class add dev dummy0 parent 1:0 classid 1:$i htb rate 1mbit ce= il 1.5mbit > > > > tc qd add dev dummy0 parent 1:$i fq_codel > > > > done > > > > > > > > time tc qd del dev dummy0 root > > > > =3D=3D=3D=3D > > > > > > > > And this is the result: > > > > > > > > Before my patch: > > > > real 0m0.488s > > > > user 0m0.000s > > > > sys 0m0.325s > > > > > > > > After my patch: > > > > real 0m0.180s > > > > user 0m0.000s > > > > sys 0m0.132s > > > > > > My results with your test script. > > > > > > before patch: > > > /usr/bin/time -p tc qdisc del dev enp1s0f0 root > > > real 1.63 > > > user 0.00 > > > sys 1.63 > > > > > > after patch: > > > /usr/bin/time -p tc qdisc del dev enp1s0f0 root > > > real 1.55 > > > user 0.00 > > > sys 1.54 > > > > > > > This is an obvious improvement, so I have no idea why you didn't > > > > catch any difference. > > > > > > We use hfsc instead of htb. I don't know whether it may cause any > > > difference. I can provide you with my test scripts if necessary. > > > > Yeah, you can try to replace the htb with hfsc in my script, > > I didn't spend time to figure out hfsc parameters. > > class add dev dummy0 parent 1:0 classid 1:$i hfsc ls m1 0 d 0 m2 > 13107200 ul m1 0 d 0 m2 13107200 > > but it behaves the same as htb... > > > My point here is, if I can see the difference with merely 1000 > > tc classes, you should see a bigger difference with hundreds > > of thousands classes in your setup. So, I don't know why you > > saw a relatively smaller difference. > > I saw a relatively big difference. It was about 1.5s faster on my huge > setup which is a lot. Yet maybe the problem is caused by something What percentage? IIUC, without patch it took you about 11s, so 1.5s faster means 13% improvement for you? > else? I thought about tx/rx queues. RJ45 ports have up to 4 tx and rx > queues. SFP+ interfaces have much higher limits. 8 or even 64 possible > queues. I've tried to increase the number of queues using ethtool from > 4 to 8 and decreased to 2. But there was no difference. It was about > 1.62 - 1.63 with an unpatched kernel and about 1.55 - 1.58 with your > patches applied. I've tried it for ifb and RJ45 interfaces where it > took about 0.02 - 0.03 with an unpatched kernel and 0.05 with your > patches applied, which is strange, but it may be caused by the fact it > was very fast even before. That is odd. In fact, this is highly related to number of TX queues, because the existing code resets the qdisc's once for each TX queue, so the more TX queues you have, the more resets kernel will do, that is the more time it will take. I plan to address this later on top of the existing patches. Thanks.