netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Václav Zindulka" <vaclav.zindulka@tlapnet.cz>
To: Cong Wang <xiyou.wangcong@gmail.com>
Cc: Linux Kernel Network Developers <netdev@vger.kernel.org>
Subject: Re: iproute2: tc deletion freezes whole server
Date: Tue, 24 Mar 2020 17:27:20 +0100	[thread overview]
Message-ID: <CANxWus9HZhN=K5oFH-qSO43vJ39Yn9YhyviNm5DLkWVnkoSeQQ@mail.gmail.com> (raw)
In-Reply-To: <CAM_iQpUPvcyxoW9=z4pY6rMfeAJNAbh21km4fUTSredm1rP+0Q@mail.gmail.com>

Hello,

Thank you for the reply!

On Mon, Mar 23, 2020 at 7:18 PM Cong Wang <xiyou.wangcong@gmail.com> wrote:
> >
> > perf trace tc qdisc del dev enp1s0f0 root
>
> Can you capture a `perf record` for kernel functions too? We
> need to know where kernel spent time on during this 11s delay.

See perd.data.{ifb0, enp1s0f0} here
https://github.com/zvalcav/tc-kernel/tree/master/20200324. I hope it
is the output you wanted. If you need anything else, let me know.

> > When I call this command on ifb interface or RJ45 interface everything
> > is done within one second.
>
>
> Do they have the same tc configuration and same workload?

Yes, both reproducers are exactly the same, interfaces are configured
in a similar way. I have the most of the offloading turned off for
physical interfaces. Yet metallic interfaces don't cause that big
delay and SFP+ dual port cards do, yet not all of them. Only
difference in reproducers is the interface name. See git repository
above, tc-interface_name-upload/download.txt files. I have altered my
whole setup in daemon I'm working on to change interfaces used. The
only difference is the existence of ingress tc filter rules to
redirect traffic to ifb interfaces in production setup. I don't use tc
filter classification in current setup. I use nftables' ability to
classify traffic. There is no traffic on interfaces except ssh
session. It behaves similar way with and without traffic.

> > My testing setup consists of approx. 18k tc class rules and approx.
> > 13k tc qdisc rules and was altered only with different interface name....
>
> Please share you tc configurations (tc -s -d qd show dev ..., tc
> -s -d filter show dev...).

I've placed whole reproducers into repository. Do you need exports of rules too?

> Also, it would be great if you can provide a minimal reproducer.

I'm afraid that minor reproducer won't cause the problem. This was
happening mostly on servers with large tc rule setups. I was trying to
create small reproducer for nftables developer many times without
success. I can try to create reproducer as small as possible, but it
may still consist of thousands of rules.

  reply	other threads:[~2020-03-24 16:27 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-22 18:06 iproute2: tc deletion freezes whole server Václav Zindulka
2020-03-23 18:17 ` Cong Wang
2020-03-24 16:27   ` Václav Zindulka [this message]
2020-03-24 22:57     ` Cong Wang
2020-03-25 11:27       ` Václav Zindulka
2020-03-25 13:33         ` Václav Zindulka
2020-03-25 17:43         ` Cong Wang
2020-03-25 18:23           ` Václav Zindulka
2020-03-26 14:24             ` Václav Zindulka
2020-03-26 17:07               ` Cong Wang
2020-03-26 17:38                 ` Cong Wang
2020-03-27 10:35                   ` Václav Zindulka
2020-03-28 13:04                     ` Václav Zindulka
2020-03-31  5:59                       ` Cong Wang
2020-03-31 12:46                         ` Václav Zindulka
2020-04-08 20:18                           ` Cong Wang
2020-04-12 20:17                             ` Václav Zindulka
2020-04-13 17:28                               ` Cong Wang
2020-04-15 15:01                                 ` Václav Zindulka
2020-04-30 12:40                                   ` Václav Zindulka
2020-05-04 17:46                                     ` Cong Wang
2020-05-04 20:36                                       ` Cong Wang
2020-05-05  8:46                                         ` Václav Zindulka
2020-05-07 18:52                                           ` Cong Wang
2020-05-08 13:59                                             ` Václav Zindulka
2020-05-17 19:35                                               ` Cong Wang
2020-05-18 14:16                                                 ` Václav Zindulka
2020-05-18 18:22                                                   ` Cong Wang
2020-05-19  8:04                                                     ` Václav Zindulka
2020-05-19 17:56                                                       ` Cong Wang
2020-05-24 10:03                                                         ` Václav Zindulka
2020-05-26 18:37                                                           ` Cong Wang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CANxWus9HZhN=K5oFH-qSO43vJ39Yn9YhyviNm5DLkWVnkoSeQQ@mail.gmail.com' \
    --to=vaclav.zindulka@tlapnet.cz \
    --cc=netdev@vger.kernel.org \
    --cc=xiyou.wangcong@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).