From: Daniel Wang <wonderfly@google.com>
To: rostedt@goodmis.org
Cc: Petr Mladek <pmladek@suse.com>,
stable@vger.kernel.org, Alexander.Levin@microsoft.com,
akpm@linux-foundation.org, byungchul.park@lge.com,
dave.hansen@intel.com, hannes@cmpxchg.org, jack@suse.cz,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Mel Gorman <mgorman@suse.de>,
mhocko@kernel.org, pavel@ucw.cz,
penguin-kernel@i-love.sakura.ne.jp, peterz@infradead.org,
tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz,
Cong Wang <xiyou.wangcong@gmail.com>,
Peter Feiner <pfeiner@google.com>
Subject: Re: 4.14 backport request for dbdda842fe96f: "printk: Add console owner and waiter logic to load balance console writes"
Date: Wed, 3 Oct 2018 11:37:56 -0700 [thread overview]
Message-ID: <CAJmjG291w2ZPRiAevSzxGNcuR6vTuqyk6z4SG3xRsbaQh5U3zQ@mail.gmail.com> (raw)
In-Reply-To: <20181003133704.43a58cf5@gandalf.local.home>
[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]
On Wed, Oct 3, 2018 at 10:37 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> Just so I understand correctly. Does the panic hit with and without the
> suggested backport patch? The only difference is that you get the full
> output with the patch and limited output without it?
When `softlockup_panic` is set (which is what my original repro had and
what we use in production), without the backport patch, the expected panic
would hit a seemingly deadlock. So even when the machine is configured
to reboot immediately after the panic (kernel.panic=-1), it just hangs there
with an incomplete backtrace. With your patch, the deadlock doesn't happen
and the machine reboots successfully.
This was and still is the issue this thread is trying to fix. The last
log snippet
was from an "experiment" that I did in order to understand what's really
happening. So far the speculation has been that the panic path was trying
to get a lock held by a backtrace dumping thread, but there is not enough
evidence which thread is holding the lock and how it uses it. So I set
`softlockup_panic` to 0, to get panic out of the equation. Then I saw that one
CPU was indeed holding the console lock, trying to write something out. If
the panic was to hit while it's doing that, we might get a deadlock.
>
> -- Steve
>
--
Best,
Daniel
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4849 bytes --]
next prev parent reply other threads:[~2018-10-03 18:38 UTC|newest]
Thread overview: 52+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-27 19:46 4.14 backport request for dbdda842fe96f: "printk: Add console owner and waiter logic to load balance console writes" Daniel Wang
2018-10-01 19:23 ` Steven Rostedt
2018-10-01 20:13 ` Pavel Machek
2018-10-01 20:21 ` Vlastimil Babka
2018-10-01 20:38 ` Daniel Wang
2018-10-01 20:29 ` Steven Rostedt
2018-10-01 20:37 ` Daniel Wang
2018-10-01 20:40 ` Sasha Levin
2018-10-02 8:42 ` Petr Mladek
2018-10-02 17:21 ` Daniel Wang
2018-10-03 0:15 ` Daniel Wang
2018-10-03 1:23 ` Steven Rostedt
2018-10-03 9:14 ` Petr Mladek
2018-10-03 17:16 ` Daniel Wang
2018-10-03 17:37 ` Steven Rostedt
2018-10-03 18:37 ` Daniel Wang [this message]
2018-10-03 23:37 ` Daniel Wang
2018-10-04 7:44 ` Sergey Senozhatsky
2018-10-04 8:05 ` Sergey Senozhatsky
2018-10-04 8:36 ` Petr Mladek
2018-10-04 8:55 ` Sergey Senozhatsky
2018-10-21 18:09 ` Daniel Wang
2018-10-22 9:32 ` Petr Mladek
2018-10-22 10:09 ` Sergey Senozhatsky
2018-11-01 16:05 ` Daniel Wang
2018-11-09 6:47 ` Sergey Senozhatsky
2018-12-12 1:16 ` Daniel Wang
2018-12-12 5:21 ` Sergey Senozhatsky
2018-12-12 6:08 ` Daniel Wang
2018-12-12 6:28 ` Sergey Senozhatsky
2018-12-12 6:48 ` Sasha Levin
2018-12-12 8:10 ` Sergey Senozhatsky
2018-12-12 13:36 ` Petr Mladek
2018-12-12 13:59 ` Sergey Senozhatsky
2018-12-12 17:43 ` Sasha Levin
2018-12-12 20:11 ` Daniel Wang
2018-12-12 21:43 ` Sasha Levin
2018-12-12 21:49 ` Daniel Wang
2018-12-12 21:52 ` Sasha Levin
2018-12-12 21:56 ` Daniel Wang
2018-12-13 0:40 ` Daniel Wang
2018-12-13 2:27 ` Sergey Senozhatsky
2018-12-13 2:39 ` Daniel Wang
2018-12-13 9:59 ` Petr Mladek
2018-12-13 14:29 ` Sasha Levin
2018-12-13 2:07 ` Sergey Senozhatsky
2018-12-28 0:16 ` Sergey Senozhatsky
2018-12-28 8:27 ` Greg KH
2018-12-28 22:03 ` Daniel Wang
2018-12-30 3:03 ` Sergey Senozhatsky
2018-10-04 7:49 ` Petr Mladek
2018-10-02 8:16 ` Sergey Senozhatsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAJmjG291w2ZPRiAevSzxGNcuR6vTuqyk6z4SG3xRsbaQh5U3zQ@mail.gmail.com \
--to=wonderfly@google.com \
--cc=Alexander.Levin@microsoft.com \
--cc=akpm@linux-foundation.org \
--cc=byungchul.park@lge.com \
--cc=dave.hansen@intel.com \
--cc=hannes@cmpxchg.org \
--cc=jack@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mgorman@suse.de \
--cc=mhocko@kernel.org \
--cc=pavel@ucw.cz \
--cc=penguin-kernel@i-love.sakura.ne.jp \
--cc=peterz@infradead.org \
--cc=pfeiner@google.com \
--cc=pmladek@suse.com \
--cc=rostedt@goodmis.org \
--cc=stable@vger.kernel.org \
--cc=tj@kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=vbabka@suse.cz \
--cc=xiyou.wangcong@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).