From: Ming Lei <ming.lei@redhat.com>
To: wenxiong@us.ibm.com
Cc: Daniel Wagner <dwagner@suse.de>,
linux-kernel@vger.kernel.org, james.smart@broadcom.com,
wenxiong@us.ibm.com, sagi@grimberg.me
Subject: Re: [PATCH 1/1] block: System crashes when cpu hotplug + bouncing port
Date: Tue, 29 Jun 2021 09:20:27 +0800 [thread overview]
Message-ID: <YNp1Bho5yypHkPfW@T590> (raw)
In-Reply-To: <71d1ce491ed5056bfa921f0e14fa646d@imap.linux.ibm.com>
Hi Wenxiong,
On Mon, Jun 28, 2021 at 01:17:34PM -0500, wenxiong wrote:
>
> >
> > The root cause is that blk-mq doesn't work well on tag allocation from
> > specified hctx(blk_mq_alloc_request_hctx), and blk-mq assumes that any
> > request allocation can't cross hctx inactive/offline, see
> > blk_mq_hctx_notify_offline()
>
> Hi Ming,
>
> I tried to pass online cpu_id(like cpu=8 in my case) to
> blk_mq_alloc_request_hctx(),
> data.hctx = q->queue_hw_ctx[hctx_idx];
> but looks like data.hctx returned with NULL. So system crashed if accessing
> data.hctx later.
>
> blk-mq request allocation can't cross hctx inactive/offline but blk-mq still
> reallocate the hctx for offline cpus(like cpu=4,5,6,7 in my case) in
> blk_mq_realloc_hw_ctxs() and hctx are NULL for online(cpu=8 in my case)cpus.
>
> Below is my understanding for hctxs, please correct me if I am wrong:
>
> Assume a system has two cores with 16 cpus:
> Before doing cpu hot plug events:
> cpu0-cpu7(core 0) : hctx->state is ACTIVE and q->hctx is not NULL.
> cpu8-cpu15(core 1): hctx->state is ACTIVE and q->hctx is not NULL
>
> After doing cpu hot plug events(the second half of each core are offline)
> cpu0-cpu3: online, hctx->state is ACTIVE and q->hctx is not NULL.
> cpu4-cpu7: offline,hctx->state is INACTIVE and q->hctx is not NULL
> cpu8-cpu11: online, hctx->state is ACTIVE but q->hctx = NULL
> cpu12-cpu15:offline, hctx->state is INACTIVE and q->hctx = NULL
>
> So num_online_cpus() is 8 after cpu hotplug events. Either way not working
> for me, no matter I pass 8 online cpus or 4 online/4 offline cpus.
>
> Is this correct? If nvmf pass online cpu ids to blk-mq, why it still
> crashes/fails?
NVMe users have to pass correct hctx_idx to blk_mq_alloc_request_hctx(), but
from the info you provided, they don't provide valid hctx_idx to blk-mq, so
q->queue_hw_ctx[hctx_idx] is NULL and kernel panic.
I believe Daniel's following patch may fix this specific issue if your
controller is FC:
[1] https://lore.kernel.org/linux-nvme/YNXTaUMAFCA84jfZ@T590/T/#t
Thanks,
Ming
next prev parent reply other threads:[~2021-06-29 1:20 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-28 3:14 [PATCH 1/1] block: System crashes when cpu hotplug + bouncing port wenxiong
2021-06-28 9:07 ` Daniel Wagner
2021-06-28 9:59 ` Ming Lei
[not found] ` <71d1ce491ed5056bfa921f0e14fa646d@imap.linux.ibm.com>
2021-06-29 1:20 ` Ming Lei [this message]
[not found] ` <OFE573413D.44652DC5-ON00258703.000DB949-00258703.000EFCD4@ibm.com>
2021-06-29 2:56 ` Ming Lei
[not found] ` <OF8889275F.DC758B38-ON00258703.001297BC-00258703.00143502@ibm.com>
2021-06-29 3:47 ` Ming Lei
2021-06-29 8:25 ` Daniel Wagner
2021-06-29 8:35 ` Daniel Wagner
2021-06-29 9:01 ` Ming Lei
2021-06-29 9:27 ` Daniel Wagner
2021-06-29 9:35 ` Ming Lei
2021-06-29 9:49 ` Daniel Wagner
2021-06-29 10:06 ` Ming Lei
2021-06-29 11:50 ` Daniel Wagner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YNp1Bho5yypHkPfW@T590 \
--to=ming.lei@redhat.com \
--cc=dwagner@suse.de \
--cc=james.smart@broadcom.com \
--cc=linux-kernel@vger.kernel.org \
--cc=sagi@grimberg.me \
--cc=wenxiong@us.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).