linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Thorsten Leemhuis <regressions@leemhuis.info>
To: "netdev@vger.kernel.org" <netdev@vger.kernel.org>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"regressions@lists.linux.dev" <regressions@lists.linux.dev>
Subject: Re: Intermittent performance regression related to ipset between 5.10 and 5.15 #forregzbot
Date: Fri, 1 Jul 2022 07:18:48 +0200	[thread overview]
Message-ID: <ba461782-a4a7-87f1-bbd4-74fc403d0bb2@leemhuis.info> (raw)
In-Reply-To: <5e56c644-2311-c094-e099-cfe0d574703b@leemhuis.info>

TWIMC: this mail is primarily send for documentation purposes and for
regzbot, my Linux kernel regression tracking bot. These mails usually
contain '#forregzbot' in the subject, to make them easy to spot and filter.

On 16.03.22 10:17, Thorsten Leemhuis wrote:
> [TLDR: I'm adding the regression report below to regzbot, the Linux
> kernel regression tracking bot; all text you find below is compiled from
> a few templates paragraphs you might have encountered already already
> from similar mails.]
> 
> On 16.03.22 00:15, McLean, Patrick wrote:
>> When we upgraded from the 5.10 (5.10.61) series to the 5.15 (5.15.16) series, we encountered an intermittent performance regression that appears to be related to iptables / ipset. This regression was noticed on Kubernetes hosts that run kube-router and experience a high amount of churn to both iptables and ipsets. Specifically, when we run the nftables (iptables-1.8.7 / nftables-1.0.0) iptables wrapper xtables-nft-multi on the 5.15 series kernel, we end up getting extremely laggy response times when iptables attempts to lookup information on the ipsets that are used in the iptables definition. This issue isn’t reproducible on all hosts. However, our experience has been that across a fleet of ~50 hosts we experienced this issue on ~40% of the hosts. When the problem evidences, the time that it takes to run unrestricted iptables list commands like iptables -L or iptables-save gradually increases over the course of about 1 - 2 hours. Growing from less than a second to run, to taking sometimes over 2 minutes to run. After that 2 hour mark it seems to plateau and not grow any longer. Flushing tables or ipsets doesn’t seem to have any affect on the issue. However, rebooting the host does reset the issue. Occasionally, a machine that was evidencing the problem may no longer evidence it after being rebooted.
>>
>> We did try to debug this to find a root cause, but ultimately ran short on time. We were not able to perform a set of bisects to hopefully narrow down the issue as the problem isn’t consistently reproducible. We were able to get some straces where it appears that most of the time is spent on getsockopt() operations. It appears that during iptables operations, it attempts to do some work to resolve the ipsets that are linked to the iptables definitions (perhaps getting the names of the ipsets themselves?). Slowly that getsockopt request takes more and more time on affected hosts. Here is an example strace of the operation in question:
> [...]
> #regzbot ^introduced v5.10..v5.15
> #regzbot title net: netfilter: Intermittent performance regression
> related to ipset
> #regzbot ignore-activity

#regzbot introduced 3976ca101990ca11ddf51f38bec7b86c19d0ca

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

      parent reply	other threads:[~2022-07-01  5:18 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-03-15 23:15 Intermittent performance regression related to ipset between 5.10 and 5.15 McLean, Patrick
2022-03-16  9:17 ` Thorsten Leemhuis
2022-04-11 10:36   ` Thorsten Leemhuis
2022-04-11 11:47     ` Jozsef Kadlecsik
2022-05-04 13:14       ` Thorsten Leemhuis
2022-05-04 19:37         ` U'ren, Aaron
2022-05-30 13:50           ` Thorsten Leemhuis
2022-05-31  7:41             ` Jozsef Kadlecsik
2022-06-20  7:16               ` Thorsten Leemhuis
2022-06-30 14:59                 ` U'ren, Aaron
2022-06-30 18:04                   ` Jakub Kicinski
2022-07-02  0:32                     ` U'ren, Aaron
2022-07-02 17:40                     ` Jozsef Kadlecsik
2022-07-08 20:08                       ` U'ren, Aaron
2022-07-29 20:21                         ` U'ren, Aaron
2022-07-30  1:41                           ` Jakub Kicinski
2022-07-30 10:43                           ` Jozsef Kadlecsik
2022-08-01 23:38                             ` U'ren, Aaron
2022-08-04  6:28                               ` Jozsef Kadlecsik
2022-07-01  5:18   ` Thorsten Leemhuis [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ba461782-a4a7-87f1-bbd4-74fc403d0bb2@leemhuis.info \
    --to=regressions@leemhuis.info \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=regressions@lists.linux.dev \
    --subject='Re: Intermittent performance regression related to ipset between 5.10 and 5.15 #forregzbot' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).