archive mirror
 help / color / mirror / Atom feed
From: Jakub Kicinski <>
To: "U'ren, Aaron" <Aaron.U'>
Cc: Jozsef Kadlecsik <>,
	Thorsten Leemhuis <>,
	"McLean, Patrick" <>,
	Pablo Neira Ayuso <>,
	"Brown, Russell" <>,
	"Rueger, Manuel" <>,
	"" <>,
	"" <>,
	Florian Westphal <>,
	"" <>
Subject: Re: Intermittent performance regression related to ipset between 5.10 and 5.15
Date: Fri, 29 Jul 2022 18:41:01 -0700	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On Fri, 29 Jul 2022 20:21:17 +0000 U'ren, Aaron wrote:
> Jozef / Jakub / Thorsten-
> Thanks for all of your help with this issue. I think that we can close this out now.
> After continuing to dig into this problem some more, I eventually figured out that the problem was caused because of how our userspace tooling was interacting with ipset save / restore and the new (ish) initval option that is included in saves / restores.
> Specifically, kube-router runs an ipset save then processes the saved ipset data, messages it a bit based upon the state from the Kubernetes cluster, and then runs that data back through ipset restore. During this time, we create unique temporary sets based upon unique sets of options and then rotate in the new endpoints into the temporary set and then use swap instructions in order to minimize impact to the data path.
> However, because we were only messaging options that were recognized and important to us, initval was left alone and blindly copied into our option strings for new and temporary sets. This caused initval to be used incorrectly (i.e. the same initval ID was used for multiple sets). I'm not 100% sure about all of the consequences of this, but it seems to have objectively caused some performance issues.
> Additionally, since initval is intentionally unique between sets, this caused us to create many more temporary sets for swapping than was actually necessary. This caused obvious performance issues as restores now contained more instructions than they needed to.
> Reverting the commit removed the issue we saw because it removed the portion of the kernel that generated the initvals which caused ipset save to revert to its previous (5.10 and below) functionality. Additionally, applying your patches also had the same impact because while I believed I was updating our userspace ipset tools in tandem, I found that the headers were actually being copied in from an alternate location and were still using the vanilla headers. This meant that while the kernel was generating initval values, the userspace actually recognized it as IPSET_ATTR_GC values which were then unused.
> This was a very long process to come to such a simple recognition about the ipset save / restore format having been changed. I apologize for the noise.

Thanks for working it out and explaining the root cause :)
I'm probably going to get the syntax wrong, but here goes nothing:

#regzbot invalid: user space mis-configuration

  reply	other threads:[~2022-07-30  1:41 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-03-15 23:15 McLean, Patrick
2022-03-16  9:17 ` Thorsten Leemhuis
2022-04-11 10:36   ` Thorsten Leemhuis
2022-04-11 11:47     ` Jozsef Kadlecsik
2022-05-04 13:14       ` Thorsten Leemhuis
2022-05-04 19:37         ` U'ren, Aaron
2022-05-30 13:50           ` Thorsten Leemhuis
2022-05-31  7:41             ` Jozsef Kadlecsik
2022-06-20  7:16               ` Thorsten Leemhuis
2022-06-30 14:59                 ` U'ren, Aaron
2022-06-30 18:04                   ` Jakub Kicinski
2022-07-02  0:32                     ` U'ren, Aaron
2022-07-02 17:40                     ` Jozsef Kadlecsik
2022-07-08 20:08                       ` U'ren, Aaron
2022-07-29 20:21                         ` U'ren, Aaron
2022-07-30  1:41                           ` Jakub Kicinski [this message]
2022-07-30 10:43                           ` Jozsef Kadlecsik
2022-08-01 23:38                             ` U'ren, Aaron
2022-08-04  6:28                               ` Jozsef Kadlecsik
2022-07-01  5:18   ` Intermittent performance regression related to ipset between 5.10 and 5.15 #forregzbot Thorsten Leemhuis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \
    --cc=Aaron.U' \ \ \ \ \ \ \ \ \ \ \ \
    --subject='Re: Intermittent performance regression related to ipset between 5.10 and 5.15' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).