From: James Puthukattukaran <james.puthukattukaran@oracle.com>
To: Keith Busch <kbusch@kernel.org>
Cc: linux-nvme@lists.infradead.org
Subject: Re: [External] : Re: way to unbind a bad nvme device/controller without powering off system
Date: Thu, 27 Oct 2022 23:14:12 -0400 [thread overview]
Message-ID: <85c20be9-7b67-ffeb-5d80-b1f4b5646b81@oracle.com> (raw)
In-Reply-To: <Y1gVOK8VXhZZbm/C@kbusch-mbp.dhcp.thefacebook.com>
On 10/25/22 12:56, Keith Busch wrote:
> On Mon, Oct 24, 2022 at 08:26:54PM -0600, Keith Busch wrote:
>> On Mon, Oct 24, 2022 at 08:02:33PM -0400, James Puthukattukaran wrote:
>>> On 10/24/22 18:36, Keith Busch wrote:
>>>
>>>>
>>>> Generally, the default timeout is really long. If you have a broken
>>>> controller, it could take several minutes before the driver unblocks
>>>> forward progress to unbind.
>>> One concern is that the reset controller flow attempts to reinitialze the controller and this will cause problems if the controller is bad. Would it make sense to have a sysfs "remove_controller" interface that simply goes through and does a nvme_dev_disable() with the assumption that the controller is dead? Will the nvme_kill_queues() in nvme_dev_disadble() unwedge any potential nvme reset thread that is blocked and thus allow the nvme_remove() flow to complete?
>>> thanks
>>
>> In your log snippet, there's this line:
>>
>> kernel:warning: [10416608.580157] nvme nvme3: I/O 209 QID 1 timeout, disable controller
>>
>> The next action the driver takes after logging that is to drain any
>> outstanding IO through a forced reset, and all subsequent tasks *should*
>> be unblocked after that completes to allow the unbinding, so I don't
>> think adding any new sysfs knobs is going to help if it's not already
>> succeeding.
>>
>> The only other thing that looks odd is that one of your stuck tasks is a
>> user passthrough command, but that should have also been cleared out by
>> the reset. Do you know what command that process is sending? I'll need
>> to double check your kernel version to see if there's anything missing
>> in that driver to ensure the unbinding succeeds.
>
The nvme command is either id-ctrl or id-ns; rather pedestrian
> I think there could be a mismatched queue quiesce state happening, but
> there's some fixes for this in later kernels. Could you possibly try
> something newer, like 6.0-stable, as an experiment?
Can you point me to the patches for this? would it straightforward to backport?
thanks
next prev parent reply other threads:[~2022-10-28 3:14 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-24 21:40 way to unbind a bad nvme device/controller without powering off system James Puthukattukaran
2022-10-24 22:36 ` Keith Busch
2022-10-25 0:02 ` [External] : " James Puthukattukaran
2022-10-25 2:26 ` Keith Busch
2022-10-25 16:56 ` Keith Busch
2022-10-28 3:14 ` James Puthukattukaran [this message]
2022-11-08 19:13 ` James Puthukattukaran
2022-11-08 23:15 ` Keith Busch
2022-11-10 16:51 ` James Puthukattukaran
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=85c20be9-7b67-ffeb-5d80-b1f4b5646b81@oracle.com \
--to=james.puthukattukaran@oracle.com \
--cc=kbusch@kernel.org \
--cc=linux-nvme@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).