From: Sagi Grimberg <sagi@grimberg.me>
To: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
Hannes Reinecke <hare@suse.de>
Cc: Damien Le Moal <Damien.LeMoal@wdc.com>
Subject: Re: bad unlock balance WARNING at nvme/045
Date: Tue, 18 Oct 2022 13:57:41 +0300 [thread overview]
Message-ID: <93e16b0e-4fa0-c0e1-6cc5-dec018db5f5c@grimberg.me> (raw)
In-Reply-To: <20221018080350.zir6xm6dnr4wgb7j@shindev>
> Hello Hannes,
>
> I observed "WARNING: bad unlock balance detected!" at nvme/045 [1]. As the Call
> Trace shows, nvme_auth_reset() has unbalanced mutex lock/unlock.
>
> mutex_lock(&ctrl->dhchap_auth_mutex);
> list_for_each_entry(chap, &ctrl->dhchap_auth_list, entry) {
> mutex_unlock(&ctrl->dhchap_auth_mutex);
> flush_work(&chap->auth_work);
> __nvme_auth_reset(chap);
> }
> mutex_unlock(&ctrl->dhchap_auth_mutex);
>
> I tried to remove the mutex_unlock in the list iteration with a patch [2], but
> it resulted in another "WARNING: possible recursive locking detected" [3]. I'm
> not sure but cause of this WARN could be __nvme_auth_work and
> nvme_dhchap_auth_work in same nvme_wq.
>
> Could you take a look for fix?
I'm looking at the code and I think that the way the concurrent
negotiations and how dhchap_auth_mutex is handled is very fragile,
also why should the per-queue auth_work hold the controller-wide
dhchap_auth_mutex? The only reason I see is because nvme_auth_negotiate
is checking if the chap context is already queued? Why should we
allow that?
I'd suggest to splice dhchap_auth_list, to a local list and then just
flush nvmet_wq in teardown flows. Same for renegotiations/reset flows.
And we should prevent for the double-queuing of chap negotiations to
begin with, instead of handling them (I still don't understand why this
is permitted, but perhaps just return EBUSY in this case?)
next prev parent reply other threads:[~2022-10-18 10:57 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-18 8:03 bad unlock balance WARNING at nvme/045 Shinichiro Kawasaki
2022-10-18 10:57 ` Sagi Grimberg [this message]
2022-10-26 2:13 ` Shinichiro Kawasaki
2022-10-26 6:42 ` Hannes Reinecke
2022-10-26 12:01 ` Shinichiro Kawasaki
2022-10-26 12:38 ` Sagi Grimberg
2022-10-28 13:52 ` Hannes Reinecke
2022-10-26 12:27 ` Sagi Grimberg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=93e16b0e-4fa0-c0e1-6cc5-dec018db5f5c@grimberg.me \
--to=sagi@grimberg.me \
--cc=Damien.LeMoal@wdc.com \
--cc=hare@suse.de \
--cc=linux-nvme@lists.infradead.org \
--cc=shinichiro.kawasaki@wdc.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).