From: John Garry <john.garry@huawei.com>
To: Bart Van Assche <bvanassche@acm.org>,
Hannes Reinecke <hare@suse.de>,
"axboe@kernel.dk" <axboe@kernel.dk>,
"jejb@linux.ibm.com" <jejb@linux.ibm.com>,
"martin.petersen@oracle.com" <martin.petersen@oracle.com>
Cc: "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
"ming.lei@redhat.com" <ming.lei@redhat.com>,
"hare@suse.com" <hare@suse.com>,
"chenxiang (M)" <chenxiang66@hisilicon.com>
Subject: Re: [PATCH RFC 3/5] blk-mq: Facilitate a shared tags per tagset
Date: Mon, 18 Nov 2019 10:31:40 +0000 [thread overview]
Message-ID: <1ba51afd-cce5-f7b2-704c-06e00db027bc@huawei.com> (raw)
In-Reply-To: <6b85f172-695c-4757-3794-455b8d55e015@acm.org>
On 15/11/2019 17:57, Bart Van Assche wrote:
> On 11/15/19 2:24 AM, John Garry wrote:
>> Bart Van Assche wrote:
>> > How about sharing tag sets across hardware
>> > queues, e.g. like in the (totally untested) patch below?
>>
>> So this is similar in principle what Ming Lei came up with here:
>> https://lore.kernel.org/linux-block/20190531022801.10003-1-ming.lei@redhat.com/
>>
>> However your implementation looks neater, which is good.
>>
>> My concern with this approach is that we can't differentiate which
>> tags are allocated for which hctx, and sometimes we need to know that.
>>
Hi Bart,
>> An example here was blk_mq_queue_tag_busy_iter(), which iterates the
>> bits for each hctx. This would just be broken by that change, unless
>> we record which bits are associated with each hctx.
>
> I disagree. In bt_iter() I added " && rq->mq_hctx == hctx" such that
> blk_mq_queue_tag_busy_iter() only calls the callback function for
> matching (hctx, rq) pairs.
OK, I see. I assumed that rq->mq_hctx was statically set when we
initially allocate the static requests per hctx; but that doesn’t appear
so - it's set in blk_mq_get_request()->blk_mq_rq_ctx_init().
>
>> Another example was __blk_mq_tag_idle(), which looks problematic.
>
> Please elaborate.
Again, this was for the same reason being that I thought we could not
differentiate which rqs were associated with which hctx.
>
>> For debugfs, when we examine
>> /sys/kernel/debug/block/.../hctxX/tags_bitmap, wouldn't that be the
>> tags for all hctx (hctx0)?
>>
>> For debugging reasons, I would say we want to know which tags are
>> allocated for a specific hctx, as this is tightly related to the
>> requests for that hctx.
>
> That is an open issue in the patch I posted and something that needs to
> be addressed. One way to address this is to change the
> sbitmap_bitmap_show() calls into calls to a function that only shows
> those bits for which rq->mq_hctx == hctx.
Yeah, understood.
>
>>> @@ -341,8 +341,11 @@ void blk_mq_tagset_busy_iter(struct
>>> blk_mq_tag_set *tagset,
>>> int i;
>>>
>>> for (i = 0; i < tagset->nr_hw_queues; i++) {
>>> - if (tagset->tags && tagset->tags[i])
>>> + if (tagset->tags && tagset->tags[i]) {
>>> blk_mq_all_tag_busy_iter(tagset->tags[i], fn, priv);
>>
>> As I mentioned earlier, wouldn't this iterate over all tags for all
>> hctx's, when we just want the tags for hctx[i]?
>>
>> Thanks,
>> John
>>
>> [Not trimming reply for future reference]
>>
>>> + if (tagset->share_tags)
>>> + break;
>>> + }
>>> }
>>> }
>>> EXPORT_SYMBOL(blk_mq_tagset_busy_iter);
>
> Since blk_mq_tagset_busy_iter() loops over all hardware queues all what
> is changed is the order in which requests are examined. I am not aware
> of any block driver that calls blk_mq_tagset_busy_iter() and that
> depends on the order of the requests passed to the callback function.
>
OK, fine.
So, to me, this approach also seems viable then.
I am however not so happy with how we use blk_mq_tag_set.tags[0] for the
shared tags; I would like to use blk_mq_tag_set.shared_tags and make
blk_mq_tag_set.tags[] point at blk_mq_tag_set.shared_tags or maybe not
blk_mq_tag_set.tags[] at all. However maybe that change may be more
intrusive.
And another more real concern is that we miss a check somewhere for
rq->mq_hctx == hctx when examining the bits on the shared tags.
Thanks,
John
next prev parent reply other threads:[~2019-11-18 10:31 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-13 13:36 [PATCH RFC 0/5] blk-mq/scsi: Provide hostwide shared tags for SCSI HBAs John Garry
2019-11-13 13:36 ` [PATCH RFC 1/5] blk-mq: Remove some unused function arguments John Garry
2019-11-13 13:58 ` Hannes Reinecke
2019-11-13 13:36 ` [PATCH RFC 2/5] blk-mq: rename BLK_MQ_F_TAG_SHARED as BLK_MQ_F_TAG_QUEUE_SHARED John Garry
2019-11-13 13:58 ` Hannes Reinecke
2019-11-13 13:36 ` [PATCH RFC 3/5] blk-mq: Facilitate a shared tags per tagset John Garry
2019-11-13 14:06 ` Hannes Reinecke
2019-11-13 14:57 ` John Garry
2019-11-13 15:38 ` Hannes Reinecke
2019-11-13 16:21 ` John Garry
2019-11-13 18:38 ` Hannes Reinecke
2019-11-14 9:41 ` John Garry
2019-11-15 5:30 ` Bart Van Assche
2019-11-15 7:29 ` Hannes Reinecke
2019-11-15 10:24 ` John Garry
2019-11-15 17:57 ` Bart Van Assche
2019-11-18 10:31 ` John Garry [this message]
2019-11-19 9:26 ` John Garry
2019-11-15 7:26 ` Hannes Reinecke
2019-11-15 10:46 ` John Garry
2019-11-13 13:36 ` [PATCH RFC 4/5] scsi: Add template flag 'host_tagset' John Garry
2019-11-13 13:36 ` [PATCH RFC 5/5] scsi: hisi_sas: Switch v3 hw to MQ John Garry
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1ba51afd-cce5-f7b2-704c-06e00db027bc@huawei.com \
--to=john.garry@huawei.com \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=chenxiang66@hisilicon.com \
--cc=hare@suse.com \
--cc=hare@suse.de \
--cc=jejb@linux.ibm.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=ming.lei@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).