linux-block.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Hannes Reinecke <hare@suse.de>
To: John Garry <john.garry@huawei.com>, Ming Lei <ming.lei@redhat.com>
Cc: Jens Axboe <axboe@kernel.dk>, Christoph Hellwig <hch@lst.de>,
	linux-block@vger.kernel.org, Yanhui Ma <yama@redhat.com>,
	Bart Van Assche <bvanassche@acm.org>,
	kashyap.desai@broadcom.com, chenxiang <chenxiang66@hisilicon.com>
Subject: Re: [PATCH] blk-mq: plug request for shared sbitmap
Date: Tue, 18 May 2021 15:22:26 +0200	[thread overview]
Message-ID: <3f61e939-d95a-1dd1-6870-e66795cfc1b1@suse.de> (raw)
In-Reply-To: <7f985b61-fd26-0f90-14ab-d4c857f7851c@huawei.com>

On 5/18/21 2:37 PM, John Garry wrote:
> On 18/05/2021 12:54, Hannes Reinecke wrote:
>>> In yanhui's virt workload(qemu, libaio, dio, high queue depth, single
>>> job), the patch can improve throughput much(>50%) when running
>>> sequential write(dio, libaio, 16 jobs) to XFS. And it is observed that
>>> IO merge is recovered to level of disabling host tags.
>>>
>> I've found a good testcase for this by using null_blk:
>>
>> modprobe null_blk submit_queues=32 queue_mode=2 irqmode=0 bs=4096
>> hw_queue_depth=2048 completion_nsec=1 nr_devices=4 shared_tags=1
>> shared_tag_bitmap=1
>>
>> and using a simple fio job with libaio and rw=read and numjobs=32 will
>> do the trick:
>>
>> [nullb]
>> rw=read
>> size=16g
>> ioengine=libaio
>> direct=1
>> buffered=0
>> group_reporting=1
>> bs=4096
>> numjobs=32
>>
>> (of course, the 'numjobs' and 'submit_queues' parameter would need to be
>> adjusted to your machine).
>> Omitting the 'shared_tag_bitmap' module parameter would yield around 12M
>> iops; adding it would see a rather dramatic drop to 172k iops.
>> With this patchset it's back to the expected iops value.
> 
> I will consider running this test myself, but do you mean that you get
> ~12M with shared_tag_bitmap=1 and this patch?
> 
> I would always expect shared_tag_bitmap=1 to give a drop for null_blk,
> as we move from per-submit queue driver tag to all submit queues sharing
> the same driver tagset.
> 
There is a slight drop (less than 1M iops), but that is about to be
expected; however, as there is a rather high fluctuation in the numbers
I didn't include it.

But nothing as dramatic as without this patchset :-)

> And I am not sure if you are fixing the IO sched from default.
> 
No, I don't; I'm using whatever it's there (ie mq-deadline).

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		        Kernel Storage Architect
hare@suse.de			               +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 Nürnberg
GF: F. Imendörffer, HRB 36809 (AG Nürnberg)

      reply	other threads:[~2021-05-18 13:22 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-14  2:20 [PATCH] blk-mq: plug request for shared sbitmap Ming Lei
2021-05-14 14:59 ` Jens Axboe
2021-05-18  9:44 ` John Garry
2021-05-18 11:16   ` Ming Lei
2021-05-18 11:42     ` John Garry
2021-05-18 12:00       ` Ming Lei
2021-05-18 12:51         ` John Garry
2021-05-18 16:01           ` John Garry
2021-05-19  0:21             ` Ming Lei
2021-05-19  8:41               ` John Garry
2021-05-20  1:23                 ` Ming Lei
2021-05-20  8:21                   ` John Garry
2021-05-18 11:54     ` Hannes Reinecke
2021-05-18 12:37       ` John Garry
2021-05-18 13:22         ` Hannes Reinecke [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3f61e939-d95a-1dd1-6870-e66795cfc1b1@suse.de \
    --to=hare@suse.de \
    --cc=axboe@kernel.dk \
    --cc=bvanassche@acm.org \
    --cc=chenxiang66@hisilicon.com \
    --cc=hch@lst.de \
    --cc=john.garry@huawei.com \
    --cc=kashyap.desai@broadcom.com \
    --cc=linux-block@vger.kernel.org \
    --cc=ming.lei@redhat.com \
    --cc=yama@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).