From: Christoph Hellwig <hch@infradead.org>
To: Keith Busch <kbusch@kernel.org>
Cc: Meng Wang <meng@hcdatainc.com>,
linux-ext4@vger.kernel.org,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>
Subject: Re: kernel panics when hot removing U.2 nvme disk
Date: Thu, 24 Sep 2020 06:12:30 +0100 [thread overview]
Message-ID: <20200924051230.GA16433@infradead.org> (raw)
In-Reply-To: <20200919014401.GE4030837@dhcp-10-100-145-180.wdl.wdc.com>
On Fri, Sep 18, 2020 at 06:44:01PM -0700, Keith Busch wrote:
> On Fri, Sep 18, 2020 at 11:47:27PM +0000, Meng Wang wrote:
> > Hi,
> > We found kernel panics today when doing test on hot remove U.2 nvme
> > disk. After hot remove the nvme disk (formatted as ext4), the system
> > freezes and all services stuck. Lot of kernel message flushed the
> > syslog, including the CPU soft lockup, ext4 NULL point er dereferece
> > and ib nic transmission timeout. The kernel panics and configuration
> > are shown below. The used kernel is 5.4.0-050400-generic and OS is
> > Ubuntu 16.04. Not sure whether it's a known bug or configuration
> > error. Any advise are welcome.
>
> [cc'ing ext4 mailing list]
>
> The NULL dereference occured before the soft lockup, so I'm guessing the
> Oops'ed process is holding the same lock the removal task wants.
>
> Your kernel is a bit older, so it may be worth verifying if your
> observation still occurs on the current stable or current mainline, but
> the ext4 developers may have a better idea as this doesn't at least
> initially appear specific to nvme.
The problem is the crazy __invalidate_device stuff that calls into
file system eviction from all kinds of super critical block paths.
While I haven't debugged the root cause this kind of thing just causes
problems without really helping anyone. I have a half-finished series
that kills this crap and instead allows the file system (or other
block device user) to pass shutdown and resize callbacks when the
exclusively open a block device. That way the file system driver
can just mark the file system shutdown to prevent any further damage
without all this mess.
next prev parent reply other threads:[~2020-09-24 5:12 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <BYAPR10MB24561C62C45813B7092E346BCB3F0@BYAPR10MB2456.namprd10.prod.outlook.com>
2020-09-19 1:44 ` kernel panics when hot removing U.2 nvme disk Keith Busch
2020-09-23 20:40 ` Meng Wang
2020-09-24 5:12 ` Christoph Hellwig [this message]
2020-09-24 5:47 ` Meng Wang
2020-09-24 6:14 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200924051230.GA16433@infradead.org \
--to=hch@infradead.org \
--cc=kbusch@kernel.org \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=meng@hcdatainc.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).