From: Mukul Joshi <mukuljoshi2011@gmail.com>
To: linux-rt-users@vger.kernel.org
Subject: ktimersoftd running at 100% CPU usage - 4.9.196-rt131
Date: Wed, 16 Oct 2019 09:43:57 -0400 [thread overview]
Message-ID: <CANaw-r5Kzn0kEUWZpB8F28SYjUm5ZtLxqfD3w9c1+MDfGwN=KA@mail.gmail.com> (raw)
Hi,
I am observing an issue with the latest 4.9 RT patch where sometimes
after reboot, I see ktimersoftd is running at 100% CPU usage. Because
of this, other applications on our box (our own custom applications)
get stuck during their initialization (they are waiting for a
semaphore to be signaled by a thread which never gets scheduled
because of this ktimersoftd).
I am running the latest 4.9.196+RT131 patches on a 2-core ARM platform.
I have also tried with an older 4.9.82-rt61 and I have observed the
same behavior.
Moving to 4.14/4.19 branches is not an option for us right now
unfortunately, so I cannot try this on those branches.
I saw a similar issue being seen with Wireshark (mentioned at
https://lore.kernel.org/patchwork/patch/931504/) and some work related
to that was done and a patch was submitted by Haris:
https://lkml.org/lkml/2018/6/28/581 and
https://lkml.org/lkml/2018/6/28/582
I don't think the patch has made it through into the official release.
I have tried incorporating his patch also but again I see the same issue.
A snippet of the top command shows the following:
top - 06:30:52 up 1:14, 0 users, load average: 7.00, 7.00, 6.99
Tasks: 104 total, 7 running, 97 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.7 us, 13.4 sy, 0.0 ni, 43.6 id, 0.2 wa, 0.0 hi, 38.0 si, 0.0 st
KiB Mem: 1025608 total, 55668 used, 969940 free, 2500 buffers
KiB Swap: 0 total, 0 used, 0 free. 19668 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM
TIME+ COMMAND
5 root -2 0 0 0 0 R
100.0 0.0 74:06.20 ktimersoftd/0
15423 root 20 0 2984 1760 1388 R 6.0
0.2 0:00.02 top
1 root 20 0 5124 3768 2144 S 0.0
0.4 0:06.36 systemd
2 root 20 0 0 0 0 S
0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S
0.0 0.0 0:00.00 ktwork
4 root 20 0 0 0 0 R
0.0 0.0 0:00.05 ksoftirqd/0
6 root 20 0 0 0 0 R
0.0 0.0 0:00.00 kworker/0:0
Another observation is that the rescheduling interrupts on CPU 1 is
very high compared to CPU 0 and increasing at a very high rate.
IPI2: 35231 4341660 Rescheduling interrupts
I am not sure how can I debug this further to figure out whats causing
ktimersoftd to run at 100% CPU.
Can you guys offer any pointers to further debug this issue?
Thanks,
Mukul
reply other threads:[~2019-10-16 13:44 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CANaw-r5Kzn0kEUWZpB8F28SYjUm5ZtLxqfD3w9c1+MDfGwN=KA@mail.gmail.com' \
--to=mukuljoshi2011@gmail.com \
--cc=linux-rt-users@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).