Linux-rt-users archive on lore.kernel.org
 help / color / Atom feed
* ktimersoftd running at 100% CPU usage - 4.9.196-rt131
@ 2019-10-16 13:43 Mukul Joshi
  0 siblings, 0 replies; only message in thread
From: Mukul Joshi @ 2019-10-16 13:43 UTC (permalink / raw)
  To: linux-rt-users

Hi,

I am observing an issue with the latest 4.9 RT patch where sometimes
after reboot, I see ktimersoftd is running at 100% CPU usage. Because
of this, other applications on our box (our own custom applications)
get stuck during their initialization (they are waiting for a
semaphore to be signaled by a thread which never gets scheduled
because of this ktimersoftd).

I am running the latest 4.9.196+RT131 patches on a 2-core ARM platform.
I have also tried with an older 4.9.82-rt61 and I have observed the
same behavior.

Moving to 4.14/4.19 branches is not an option for us right now
unfortunately, so I cannot try this on those branches.

I saw a similar issue being seen with Wireshark (mentioned at
https://lore.kernel.org/patchwork/patch/931504/) and some work related
to that was done and a patch was submitted by Haris:
https://lkml.org/lkml/2018/6/28/581 and
https://lkml.org/lkml/2018/6/28/582

I don't think the patch has made it through into the official release.
I have tried incorporating his patch also but again I see the same issue.

A snippet of the top command shows the following:
top - 06:30:52 up  1:14,  0 users,  load average: 7.00, 7.00, 6.99
Tasks: 104 total,   7 running,  97 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.7 us, 13.4 sy,  0.0 ni, 43.6 id,  0.2 wa,  0.0 hi, 38.0 si,  0.0 st
KiB Mem:   1025608 total,    55668 used,   969940 free,     2500 buffers
KiB Swap:        0 total,        0 used,        0 free.    19668 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR    S    %CPU %MEM
TIME+      COMMAND
    5      root      -2      0       0         0          0       R
  100.0     0.0        74:06.20   ktimersoftd/0
15423  root      20     0    2984     1760   1388    R        6.0
0.2         0:00.02    top
    1       root      20    0    5124     3768    2144   S        0.0
    0.4         0:06.36    systemd
    2       root      20    0       0         0            0      S
    0.0      0.0         0:00.00    kthreadd
    3       root      20    0       0         0             0     S
    0.0      0.0         0:00.00    ktwork
    4       root      20    0       0         0             0     R
    0.0      0.0        0:00.05     ksoftirqd/0
    6       root      20    0       0         0            0      R
    0.0      0.0        0:00.00     kworker/0:0

Another observation is that the rescheduling interrupts on CPU 1 is
very high compared to CPU 0 and increasing at a very high rate.

IPI2:      35231    4341660  Rescheduling interrupts

I am not sure how can I debug this further to figure out whats causing
ktimersoftd to run at 100% CPU.
Can you guys offer any pointers to further debug this issue?

Thanks,
Mukul

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, back to index

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-16 13:43 ktimersoftd running at 100% CPU usage - 4.9.196-rt131 Mukul Joshi

Linux-rt-users archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rt-users/0 linux-rt-users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rt-users linux-rt-users/ https://lore.kernel.org/linux-rt-users \
		linux-rt-users@vger.kernel.org
	public-inbox-index linux-rt-users

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rt-users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git