From: Tian Lan <tilan7663@gmail.com>
To: ming.lei@redhat.com
Cc: axboe@kernel.dk, linux-block@vger.kernel.org, lkp@intel.com,
llvm@lists.linux.dev, oe-kbuild-all@lists.linux.dev,
tian.lan@twosigma.com, tilan7663@gmail.com
Subject: Re: [PATCH 1/1] blk-mq: fix blk_mq_hw_ctx active request accounting
Date: Sun, 14 May 2023 10:12:54 -0400 [thread overview]
Message-ID: <20230514141254.595099-1-tilan7663@gmail.com> (raw)
In-Reply-To: <ZGDPLEtUiDeIrCyl@ovpn-8-17.pek2.redhat.com>
Hi Ming,
> > kprobe output showing RQF_MQ_INFLIGHT bit is not cleared before
> > __blk_mq_free_request being called.
> RQF_MQ_INFLIGHT won't be cleared when the request is freed normally
> from blk_mq_free_request().
Yes you are correct, maybe I should capture both rq->rq_flags and
rq->state so we know for sure if either of blk_mq_free_request or
__blk_mq_put_driver_tag was being called before hitting __blk_mq_free_request.
> > b'__blk_mq_free_request+0x1 [kernel]'
> > b'bt_iter+0x50 [kernel]'
> > b'blk_mq_queue_tag_busy_iter+0x318 [kernel]'
> > b'blk_mq_timeout_work+0x7c [kernel]'
> > b'process_one_work+0x1c4 [kernel]'
> > b'worker_thread+0x4d [kernel]'
> > b'kthread+0xe6 [kernel]'
> > b'ret_from_fork+0x1f [kernel]'
> If __blk_mq_free_request() is called from timeout, that means this
> request has been freed by blk_mq_free_request() already, so __blk_mq_dec_active_requests
> should have been run.
We are also seeing a different call stack that could also potentially by-pass
__blk_mq_dec_active_requests. Do you think they could be caused by the same
underlying issue.
1976 2000 collectd __blk_mq_free_request rq_flags 0x620c0 in-flight 1
b'__blk_mq_free_request+0x1 [kernel]'
b'bt_iter+0x50 [kernel]'
b'blk_mq_queue_tag_busy_iter+0x318 [kernel]'
b'blk_mq_in_flight+0x35 [kernel]'
b'diskstats_show+0x205 [kernel]'
b'seq_read_iter+0x11f [kernel]'
b'proc_req_read_iter+0x4a [kernel]'
b'vfs_read+0x239 [kernel]'
b'ksys_read+0xb [kernel]'
b'do_syscall_64+0x58 [kernel]'
b'entry_SYSCALL_64_after_hwframe+0x63 [kernel]'
> However, one case is that __blk_mq_dec_active_requests isn't called in
> blk_mq_end_request_batch, so maybe your driver is nvme with multiple
> NSs, so can you try the following patch?
Yes, we are using nvme driver with multiple NSs.
I can test this patch and will update you on the results. I'm just curious
shouldn't the counter be subtracted via __blk_mq_sub_active_requests when
blk_mq_flush_tag_batch is invoked in that case. Then this would result in
double counting, is that correct.
Thanks,
Tian
next prev parent reply other threads:[~2023-05-14 14:12 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-13 14:12 [PATCH 1/1] blk-mq: fix blk_mq_hw_ctx active request accounting Tian Lan
2023-05-13 16:41 ` kernel test robot
2023-05-13 16:41 ` kernel test robot
2023-05-13 19:05 ` Tian Lan
2023-05-13 20:54 ` Simon Horman
2023-05-13 21:15 ` Jens Axboe
2023-05-13 22:11 ` Tian Lan
2023-05-14 1:39 ` Jens Axboe
2023-05-15 12:28 ` Tian Lan
2023-05-13 22:12 ` Tian Lan
2023-05-14 1:52 ` Jens Axboe
2023-05-14 12:20 ` Ming Lei
2023-06-03 22:39 ` [PATCH] " Tian Lan
2023-06-04 0:47 ` Ming Lei
2023-06-03 22:39 ` [PATCH 1/1] " Tian Lan
2023-05-14 14:22 ` Ming Lei
2023-05-14 14:28 ` Tian Lan
2023-05-14 14:53 ` [PATCH] " Tian Lan
2023-05-15 5:54 ` Hannes Reinecke
2023-05-15 12:26 ` Tian Lan
2023-05-25 2:12 ` Ming Lei
2023-06-03 23:20 ` [PATCH 1/1] " Jens Axboe
2023-05-14 12:08 ` Ming Lei
2023-05-14 14:12 ` Tian Lan [this message]
2023-05-14 14:15 ` Ming Lei
2023-05-13 19:11 ` Tian Lan
2023-05-13 16:52 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230514141254.595099-1-tilan7663@gmail.com \
--to=tilan7663@gmail.com \
--cc=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
--cc=lkp@intel.com \
--cc=llvm@lists.linux.dev \
--cc=ming.lei@redhat.com \
--cc=oe-kbuild-all@lists.linux.dev \
--cc=tian.lan@twosigma.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.