From: Philippe Gerum <rpm@xenomai.org>
To: Russell Johnson <russell.johnson@kratosdefense.com>
Cc: Bryan Butler <Bryan.Butler@kratosdefense.com>
Cc: "xenomai@lists.linux.dev" <xenomai@lists.linux.dev>
Subject: Re: System hanging when using condition variables
Date: Thu, 22 Sep 2022 16:26:23 +0200 [thread overview]
Message-ID: <87pmfncw9u.fsf@xenomai.org> (raw)
In-Reply-To: <PH1P110MB10500CC96F6D84D2026A6F63E24D9@PH1P110MB1050.NAMP110.PROD.OUTLOOK.COM>
Russell Johnson <russell.johnson@kratosdefense.com> writes:
> [[S/MIME Signed Part:Undecided]]
> Hello,
>
>
>
> I have been trying to debug an issue in our app where the entire system hangs with the following error from the kernel: “kernel:watchdog:
> BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:2:594]”. This happens consistently on every run. I was able to strip down all of the
> relevant code into a simple standalone app that only uses 4 pthreads, 3 EVL events, and 3 EVL mutexes if you would like to be able to
> re-create the issue (I have attached the test file). Is there anything fundamentally flawed in this logic (the same logic worked fine previously
> with STL condition variables and STL mutexes)? It appears that there becomes some kind of deadlock in the kernel due to an EVL event
> and/or EVL mutex. Let me know if there is any more information that I can provide you to help clear up the scenario. I have spent multiple
> weeks tracking this issue with no luck so far.
>
How long does it usually take for the watchdog to trigger with this test
code? I've not been able to reproduce the issue so far after a couple
of hours runtime (kvm/x86 and real hw as well). I'm going to try this on
armv7, armv8 SoCs for good measure.
In the meantime, I may need the .config file for your kernel. Also,
could you enable backtracing on all CPUs upon oops as follows, sending
me the kernel splat this should produce when the watchdog triggers?
# echo 1 > /proc/sys/kernel/oops_all_cpu_backtrace
TIA,
--
Philippe.
next prev parent reply other threads:[~2022-09-22 14:47 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-19 21:38 System hanging when using condition variables Russell Johnson
2022-09-20 6:55 ` Philippe Gerum
2022-09-20 14:31 ` [External] - " Russell Johnson
2022-09-21 14:53 ` Russell Johnson
2022-09-22 14:26 ` Philippe Gerum [this message]
2022-09-22 15:25 ` Bryan Butler
2022-09-23 19:56 ` Bryan Butler
2022-09-24 8:21 ` Philippe Gerum
2022-09-25 14:59 ` Philippe Gerum
2022-09-25 16:32 ` Philippe Gerum
2022-09-26 14:20 ` Bryan Butler
2022-09-27 22:05 ` Russell Johnson
2022-09-27 23:04 ` Russell Johnson
2022-09-28 1:08 ` Bryan Butler
2022-09-28 10:06 ` Philippe Gerum
2022-09-28 10:37 ` Philippe Gerum
2022-09-28 9:59 ` Philippe Gerum
2022-09-28 18:35 ` Russell Johnson
2022-09-29 7:04 ` Philippe Gerum
2022-09-29 18:32 ` Russell Johnson
2022-10-01 4:38 ` Russell Johnson
2022-10-04 15:50 ` Philippe Gerum
2022-10-10 17:04 ` Russell Johnson
2022-10-12 16:11 ` Russell Johnson
2022-10-12 16:24 ` Eric Kuzara
2022-11-03 18:13 ` Philippe Gerum
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87pmfncw9u.fsf@xenomai.org \
--to=rpm@xenomai.org \
--cc=Bryan.Butler@kratosdefense.com \
--cc=russell.johnson@kratosdefense.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.