Re: High cpu usage caused by kernel process when upgraded to linux 5.19.17 or later

From: Steven Rostedt <rostedt@goodmis.org>
To: Vivek Anand <vivekanand754@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>,
	Peter Zijlstra <peterz@infradead.org>,
	Bagas Sanjaya <bagasdotme@gmail.com>,
	Pablo Neira Ayuso <pablo@netfilter.org>,
	Jozsef Kadlecsik <kadlec@netfilter.org>,
	Florian Westphal <fw@strlen.de>,
	"David S. Miller" <davem@davemloft.net>,
	Eric Dumazet <edumazet@google.com>,
	Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
	Lingutla Chandrasekhar <clingutla@codeaurora.org>,
	Frederic Weisbecker <frederic@kernel.org>,
	"J. Avila" <elavila@google.com>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	Thomas Renninger <trenn@suse.com>, Shuah Khan <shuah@kernel.org>,
	Borislav Petkov <bp@alien8.de>,
	Josh Poimboeuf <jpoimboe@kernel.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Linux Regressions <regressions@lists.linux.dev>,
	Linux Netfilter Development <netfilter-devel@vger.kernel.org>,
	Netfilter Core Developers <coreteam@netfilter.org>,
	Linux Networking <netdev@vger.kernel.org>,
	Linux Power Management <linux-pm@vger.kernel.org>,
	x86@kernel.org
Subject: Re: High cpu usage caused by kernel process when upgraded to linux 5.19.17 or later
Date: Thu, 6 Jul 2023 12:33:00 -0400	[thread overview]
Message-ID: <20230706123300.55d6450b@gandalf.local.home> (raw)
In-Reply-To: <CAJnqnX5dHiXe3smKhj6JT9+6FNdgrAR=5_Hm8BSRpVF3uARYUg@mail.gmail.com>

On Mon, 3 Jul 2023 18:23:25 +0530
Vivek Anand <vivekanand754@gmail.com> wrote:

> Hi Thomas,
> 
> Further analyzing, I found that I did set
> "CONFIG_NETFILTER_XT_MATCH_LIMIT=m" in my kernel config earlier which was
> causing high CPU consumption.
> Setting it to "CONFIG_NETFILTER_XT_MATCH_LIMIT=n" resolved the high CPU
> issue.
> 
> Is there any suggestion regarding the use of this config
> "CONFIG_NETFILTER_XT_MATCH_LIMIT" as I'm getting high CPU by setting it to
> "m" ?

That config enables the compiling of: net/netfilter/xt_limit.c

The htable_gc that you reported is defined in: net/netfilter/xt_hashlimit.c

It has:

static void htable_gc(struct work_struct *work)
{
        struct xt_hashlimit_htable *ht;

        ht = container_of(work, struct xt_hashlimit_htable, gc_work.work);

        htable_selective_cleanup(ht, false);

        queue_delayed_work(system_power_efficient_wq,
                           &ht->gc_work, msecs_to_jiffies(ht->cfg.gc_interval));
}

So it queues itself every ht->cfg.gc_interval msecs. That variable seems to
come from some configuration of netfilter, and I think you can see these in:

 find /proc/sys/net -name 'gc_interval'

Perhaps you have it set off to go too much?

-- Steve

> 
> Thanks,
> Vivek
> 
> On Thu, Jun 29, 2023 at 7:48 PM Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> > On Thu, Jun 29 2023 at 12:05, Vivek Anand wrote:
> > > I've tried booting with "spectre_v2=retpoline retbleed=off".
> > > This change didn't work. Still CPU is 100%
> >
> > This does not make sense.
> >
> > retbleed=off has the same effect as CONFIG_X86_IBRS_ENTRY=n.
> >
> > The only difference is that with CONFIG_X86_IBRS_ENTRY=y and
> > retbleed=off there is one extra jump in the low level entry code
> > (syscall, interrupts, exceptions) and one extra jump on the exit side.
> >
> > But those extra jumps are completely irrelevant for the kworker threads.
> >
> > Can you please provide dmesg and the content of the files in
> >
> >  /sys/devices/system/cpu/vulnerabilities/
> >
> > on a kernel booted with "spectre_v2=retpoline retbleed=off" ?
> >
> > Thanks,
> >
> >         tglx
> >