From: Peter Zijlstra <peterz@infradead.org>
To: Bagas Sanjaya <bagasdotme@gmail.com>
Cc: "Steven Rostedt (Google)" <rostedt@goodmis.org>,
Pablo Neira Ayuso <pablo@netfilter.org>,
Jozsef Kadlecsik <kadlec@netfilter.org>,
Florian Westphal <fw@strlen.de>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
Lingutla Chandrasekhar <clingutla@codeaurora.org>,
Frederic Weisbecker <frederic@kernel.org>,
"J. Avila" <elavila@google.com>,
Vivek Anand <vivekanand754@gmail.com>,
"Rafael J. Wysocki" <rafael@kernel.org>,
Thomas Renninger <trenn@suse.com>, Shuah Khan <shuah@kernel.org>,
Borislav Petkov <bp@alien8.de>,
Josh Poimboeuf <jpoimboe@kernel.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Linux Regressions <regressions@lists.linux.dev>,
Linux Netfilter Development <netfilter-devel@vger.kernel.org>,
Netfilter Core Developers <coreteam@netfilter.org>,
Linux Networking <netdev@vger.kernel.org>,
Linux Power Management <linux-pm@vger.kernel.org>,
x86@kernel.org
Subject: Re: Fwd: High cpu usage caused by kernel process when upgraded to linux 5.19.17 or later
Date: Tue, 27 Jun 2023 09:30:35 +0200 [thread overview]
Message-ID: <20230627073035.GV4253@hirez.programming.kicks-ass.net> (raw)
In-Reply-To: <ZJpJkL3dPXxgw6RK@debian.me>
On Tue, Jun 27, 2023 at 09:29:36AM +0700, Bagas Sanjaya wrote:
> On Fri, Jun 23, 2023 at 07:58:51AM +0700, Bagas Sanjaya wrote:
> > Hi,
> >
> > I notice a regression report on Bugzilla [1]. Quoting from it:
> >
> > > kernel process "kworker/events_power_efficient" uses a lot of cpu power (100% on ESXI 6.7, ~30% on ESXI 7.0U3 or later) after upgrading from 5.17.3 to 5.19.17 or later.
> > >
> > > dmesg log:
> > > [ 2430.973102] </TASK>
> > > [ 2430.973131] Sending NMI from CPU 1 to CPUs 0:
> > > [ 2430.973241] NMI backtrace for cpu 0
> > > [ 2430.973247] CPU: 0 PID: 22 Comm: kworker/0:1 Not tainted 6.3.3 #1
> > > [ 2430.973254] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
> > > [ 2430.973258] Workqueue: events_power_efficient htable_gc [xt_hashlimit]
> > > [ 2430.973275] RIP: 0010:preempt_count_sub+0x2e/0xa0
> > > [ 2430.973289] Code: 36 01 85 c9 75 1b 65 8b 15 a7 da f8 5e 89 d1 81 e1 ff ff ff 7f 39 f9 7c 16 81 ff fe 00 00 00 76 3b f7 df 65 01 3d 8a da f8 5e <c3> cc cc cc cc e8 98 aa 25 00 85 c0 74 f2 8b 15 da 71 ed 00 85 d2
> > > [ 2430.973294] RSP: 0018:ffffb15ec00dbe58 EFLAGS: 00000297
> > > [ 2430.973299] RAX: 0000000000000000 RBX: ffffb15ec12ad000 RCX: 0000000000000001
> > > [ 2430.973302] RDX: 0000000080000001 RSI: ffffffffa1c3313b RDI: 00000000ffffffff
> > > [ 2430.973306] RBP: dead000000000122 R08: 0000000000000010 R09: 0000746e65696369
> > > [ 2430.973309] R10: 8080808080808080 R11: 0000000000000018 R12: 0000000000000000
> > > [ 2430.973312] R13: 0000000000001e2b R14: ffffb15ec12ad048 R15: ffff91c279c26a05
> > > [ 2430.973316] FS: 0000000000000000(0000) GS:ffff91c279c00000(0000) knlGS:0000000000000000
> > > [ 2430.973320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 2430.973324] CR2: 000055fc138890e0 CR3: 000000010810e002 CR4: 00000000001706f0
> > > [ 2430.973374] Call Trace:
> > > [ 2430.973388] <TASK>
> > > [ 2430.973390] __local_bh_enable_ip+0x32/0x70
> > > [ 2430.973413] htable_selective_cleanup+0x95/0xc0 [xt_hashlimit]
> > > [ 2430.973428] htable_gc+0xf/0x30 [xt_hashlimit]
> > > [ 2430.973440] process_one_work+0x1d4/0x360
> > > [ 2430.973459] ? process_one_work+0x360/0x360
> > > [ 2430.973467] worker_thread+0x25/0x3b0
> > > [ 2430.973476] ? process_one_work+0x360/0x360
> > > [ 2430.973483] kthread+0xe1/0x110
> > > [ 2430.973499] ? kthread_complete_and_exit+0x20/0x20
> > > [ 2430.973507] ret_from_fork+0x1f/0x30
> > > [ 2430.973526] </TASK>
> >
> > See Bugzilla for the full thread and perf output.
> >
> > Anyway, I'm tracking it in regzbot so that it doesn't fall through
> > cracks unnoticed:
> >
> > #regzbot introduced: v5.17.3..v5.19.17 https://bugzilla.kernel.org/show_bug.cgi?id=217586
> > #regzbot title: kworker/events_power_efficient utilizes full CPU power after kernel upgrade
> >
>
> The reporter had found the culprit (see Bugzilla for more information), thus
> telling regzbot:
>
> #regzbot introduced: 6ad0ad2bf8a67e
> #regzbot title: retbleed reporting causes high cpu utilization due to kworker/events_power_efficient
> #regzbot link: https://lore.kernel.org/all/PH0PR05MB8448A203A909959FAC754B7AAF439@PH0PR05MB8448.namprd05.prod.outlook.com/
> #regzbot link: 6ad0ad2bf8a67e
Is there a problem other than that IBRS is slow and slower still on
shitty virt?
Does booting with: "spectre_v2=retpoline retbleed=off" make it go away?
I can't tell from this. Also, please don't use bugzilla.
next prev parent reply other threads:[~2023-06-27 7:31 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-23 0:58 Fwd: High cpu usage caused by kernel process when upgraded to linux 5.19.17 or later Bagas Sanjaya
2023-06-23 1:26 ` Steven Rostedt
2023-06-27 2:29 ` Fwd: " Bagas Sanjaya
2023-06-27 7:30 ` Peter Zijlstra [this message]
2023-06-27 10:15 ` Bagas Sanjaya
2023-06-27 10:19 ` Peter Zijlstra
[not found] ` <CAJnqnX5rYn65zVQ+SLN4m4ZzM_jOa_RjGhazWO=Fh8ZvdOCadg@mail.gmail.com>
2023-06-29 14:18 ` Thomas Gleixner
[not found] ` <CAJnqnX5dHiXe3smKhj6JT9+6FNdgrAR=5_Hm8BSRpVF3uARYUg@mail.gmail.com>
2023-07-06 16:33 ` Steven Rostedt
2023-10-20 10:55 ` Fwd: " Linux regression tracking #update (Thorsten Leemhuis)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230627073035.GV4253@hirez.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=bagasdotme@gmail.com \
--cc=bp@alien8.de \
--cc=clingutla@codeaurora.org \
--cc=coreteam@netfilter.org \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=elavila@google.com \
--cc=frederic@kernel.org \
--cc=fw@strlen.de \
--cc=jpoimboe@kernel.org \
--cc=kadlec@netfilter.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pm@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=netfilter-devel@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=pablo@netfilter.org \
--cc=rafael@kernel.org \
--cc=regressions@lists.linux.dev \
--cc=rostedt@goodmis.org \
--cc=shuah@kernel.org \
--cc=trenn@suse.com \
--cc=vivekanand754@gmail.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).