From: Kegl Rohit <keglrohit@gmail.com>
To: linux-rt-users@vger.kernel.org
Subject: rcu stall caused by rt task with high minor page fault rate
Date: Tue, 30 May 2023 12:51:48 +0200 [thread overview]
Message-ID: <CAMeyCbgGgxpF5Nk=dZXEBS92VxUBR_OcRQqdQdEJAAzqcbP+fQ@mail.gmail.com> (raw)
Hello!
Running 5.10.104-rt63 SMP PREEMPT_RT on dual core imx7d.
Currently I am debugging an rcu stall issue caused by an user-space rt task.
The test procedure to reproduce the issue is:
1. Bootup the system
2. initd starts rt task application (SCHED_FIFO, priority -13 and
affinity set to core0)
3. login via ssh and start e.g. memtester with the maximum amount of
free RAM available
4. memtester locks its memory with mlock successfully
5. After some time the rt task is stuck consuming 100% system time on core0.
6. Kernel produces rcu stall warnings because rcu kthread does not get
any CPU on core0.
Looking at the vm stats of the rt thread shows a minor page fault rate
> 350k/s.
So the process is stuck in memory handling and because of the core
binding the rcu kthread does not get any core0 cpu time and produces
stall warnings.
Reading https://wiki.linuxfoundation.org/realtime/documentation/technical_details/rcu
CONFIG_RCU_BOOST=y, should be the solution for such issues.
But enabling RCU_BOOST did not change anything.
See link above:
> However, bugs can happen, including bugs involving infinite loops in high-priority real-time threads. Debugging these problems is more difficult if the system keeps hanging due to OOM. One way to ease debugging is to build with CONFIG_RCU_BOOST=y,
The main cause for the minor page faults is the missing mlock in the
application.
mlock is always necessary for rt apps.
But for my understanding RCU_BOOST should help here, even if the rt
app is not implemented correctly?
Thanks in advance!
next reply other threads:[~2023-05-30 10:51 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-30 10:51 Kegl Rohit [this message]
2023-08-18 9:42 ` rcu stall caused by rt task with high minor page fault rate Sebastian Andrzej Siewior
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAMeyCbgGgxpF5Nk=dZXEBS92VxUBR_OcRQqdQdEJAAzqcbP+fQ@mail.gmail.com' \
--to=keglrohit@gmail.com \
--cc=linux-rt-users@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).