linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Juergen Gross <jgross@suse.com>
To: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org
Cc: paulmck@kernel.org, mhocko@suse.com,
	Juergen Gross <jgross@suse.com>, Jonathan Corbet <corbet@lwn.net>
Subject: [PATCH 0/3] kernel/smp.c: add more CSD lock debugging
Date: Fri, 26 Feb 2021 12:25:18 +0100	[thread overview]
Message-ID: <20210226112521.8641-1-jgross@suse.com> (raw)

This patch series was created to help catching a rather long standing
problem with smp_call_function_any() and friends.

Very rarely a remote cpu seems not to execute a queued function and
the cpu queueing that function request will wait forever for the
CSD lock to be released by the remote cpu.

This problem has been observed primarily when running as a guest on
top of KVM or Xen, but there are reports of the same pattern for the
bare metal case, too. It seems to exist since about 2 years now, and
there is not much data available.

What is known up to now is that resending an IPI to the remote cpu is
helping.

The patches are adding more debug data being printed in a hang
situation using a kernel with CONFIG_CSD_LOCK_WAIT_DEBUG configured.
Additionally the debug coding can be controlled via a new parameter
in order to make it easier to use such a kernel in a production
environment without too much negative performance impact. Per default
the debugging additions will be switched off and they can be activated
via the new boot parameter:

csdlock_debug=1 will switch on the basic debugging and IPI resend
csdlock_debug=ext will add additional data printed out in a hang
  situation, but this option will have a larger impact on performance.

I hope that the "ext" setting will help to find the root cause of the
problem.

Juergen Gross (3):
  kernel/smp: add boot parameter for controlling CSD lock debugging
  kernel/smp: prepare more CSD lock debugging
  kernel/smp: add more data to CSD lock debugging

 .../admin-guide/kernel-parameters.txt         |  10 +
 kernel/smp.c                                  | 193 +++++++++++++++++-
 2 files changed, 192 insertions(+), 11 deletions(-)

-- 
2.26.2


             reply	other threads:[~2021-02-26 11:26 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-26 11:25 Juergen Gross [this message]
2021-02-26 11:25 ` [PATCH 1/3] kernel/smp: add boot parameter for controlling CSD lock debugging Juergen Gross
2021-02-26 11:25 ` [PATCH 2/3] kernel/smp: prepare more " Juergen Gross
2021-02-26 11:25 ` [PATCH 3/3] kernel/smp: add more data to " Juergen Gross
2021-02-26 16:38   ` Peter Zijlstra
2021-02-26 18:12     ` Paul E. McKenney
2021-02-26 21:05       ` Peter Zijlstra
2021-03-01  0:07         ` Paul E. McKenney
2021-03-01 15:45           ` Peter Zijlstra
2021-03-01 15:53             ` Jürgen Groß
2021-03-01 16:45               ` Paul E. McKenney

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210226112521.8641-1-jgross@suse.com \
    --to=jgross@suse.com \
    --cc=corbet@lwn.net \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mhocko@suse.com \
    --cc=paulmck@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).