From: Juergen Gross <jgross@suse.com>
To: minyard@acm.org
Cc: Ingo Molnar <mingo@kernel.org>,
"Paul E. McKenney" <paulmck@kernel.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: Possible reproduction of CSD locking issue
Date: Wed, 26 Jan 2022 15:51:36 +0100 [thread overview]
Message-ID: <70245998-b99e-ebe7-e932-f8a1b46f6ac7@suse.com> (raw)
In-Reply-To: <20220126135639.GS34919@minyard.net>
[-- Attachment #1.1.1: Type: text/plain, Size: 3550 bytes --]
On 26.01.22 14:56, Corey Minyard wrote:
> On Wed, Jan 26, 2022 at 07:08:22AM +0100, Juergen Gross wrote:
>> On 25.01.22 19:27, Corey Minyard wrote:
>>> We have a customer that had been seeing CSD lock issues on a Centos 7
>>> kernel (unfortunately). I couldn't find anything or any kernel changes
>>> that might fix it, so I was consdering it was the CSD locking issue you
>>> have been chasing for a while.
>>
>> Is this on bare metal or in a virtualized environment?
>
> This is bare metal.
>
> I do think I know what happened. Here's my analysis...
>
> csd: Detected non-responsive CSD lock (#1) on CPU#3, waiting 5000000042 ns for CPU#55 flush_tlb_func+0x0/0xb0(0xffff8e0b3e2afbe8).
> csd: CSD lock (#1) unresponsive.
> csd: cnt(0000000): 0000->0000 queue
> csd: cnt(0000001): ffff->0037 idle
>
> The above means that these events weren't seen, I think. We can
> ignore them in any case.
>
> csd: cnt(63d8dd8): 0003->0037 ipi
> csd: cnt(63d8dd9): 0003->0037 ping
> csd: cnt(63d8dda): 0003->ffff pinged
>
> This is a little confusing. The first two lines have to be from
> __smp_call_single_queue_debug. The last line has to be from
> smp_call_function_many. But you never see the pinged from
> __smp_call_single_queue_debug.
Be careful here. For each event-type/cpu combination there is only one
entry saved. It is still possible to see some events more than once,
as each entry holds information of two events (the location _where_
the entry is stored is determining what just happened, and the _data_
stored at this position tells us what happened one event before that).
So any "missing" entry which type/cpu combination is seen further down
in the trace _might_ be just overwritten by the later entry.
I've used this scheme instead of simple trace buffers as on a large
system any buffer would be overwritten multiple times before the CSD
timeout is over. This way at least the trace entries of the cpus
having problems (the sender and/or the receiver) are still available.
After having found a local reproducer for my problem I could enhance
the patches by using small arrays instead of single entries for each
trace entry location, producing better traces with less "missing"
entries. I didn't post those patches, as they were heavily
intertwined with hypervisor patches for producing traces covering
both, kernel and hypervisor events.
>
> csd: cnt(63d8dea): 0035->0037 pinged
>
> The tail of CPU 53 sending an IPI to CPU 55 in
> __smp_call_single_queue_debug.
>
> csd: cnt(63d8deb): ffff->0037 gotipi
> csd: cnt(63d8dec): ffff->0037 handle
> csd: cnt(63d8ded): ffff->0037 dequeue (src CPU 0 == empty)
> csd: cnt(63d8dee): ffff->0037 hdlend (src CPU 0 == early)
>
> CPU 55 is handling the IPI(s) it was sent earlier.
Right. Everything up to here is completely fine and can be ignored.
>
> csd: cnt(63d8e1f): 0003->0037 queue
> csd: cnt(63d8e20): 0003->0037 ipi
> csd: cnt(63d8e21): 0003->0037 ping
>
> In __smp_call_single_queue_debug CPU 3 sends another message to
> CPU 55 and sends an IPI. But there should be a pinged entry
> after this.
>
> csd: cnt(63d8e22): 0003->0037 queue
> csd: cnt(63d8e23): 0003->0037 noipi
This is interesting. Those are 5 consecutive entries without any
missing in between (see the counter values). Could it be that after
the ping there was an interrupt and the code was re-entered for
sending another IPI? This would clearly result in a hang as seen.
Juergen
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
next prev parent reply other threads:[~2022-01-26 14:51 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-25 18:27 Possible reproduction of CSD locking issue Corey Minyard
2022-01-25 18:53 ` Paul E. McKenney
2022-01-25 21:52 ` Corey Minyard
2022-01-25 22:51 ` Paul E. McKenney
2022-01-26 6:08 ` Juergen Gross
2022-01-26 13:56 ` Corey Minyard
2022-01-26 14:12 ` Corey Minyard
2022-01-26 14:51 ` Juergen Gross [this message]
2022-01-26 15:31 ` Corey Minyard
2022-01-26 15:34 ` Juergen Gross
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=70245998-b99e-ebe7-e932-f8a1b46f6ac7@suse.com \
--to=jgross@suse.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=minyard@acm.org \
--cc=paulmck@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).