From: Hannes Reinecke <hare@suse.de>
To: Jens Axboe <axboe@fb.com>,
linux-kernel@vger.kernel.org, linux-block@vger.kernel.org
Cc: osandov@osandov.com, bart.vanassche@sandisk.com
Subject: Re: [PATCHSET v6] blk-mq scheduling framework
Date: Fri, 13 Jan 2017 13:10:07 +0100 [thread overview]
Message-ID: <cab72b22-d672-3c68-ec4a-6e3cb65f93ab@suse.de> (raw)
In-Reply-To: <5850a4ad-74a1-8d30-e865-fbd35679e84f@suse.de>
On 01/13/2017 12:04 PM, Hannes Reinecke wrote:
> On 01/13/2017 09:15 AM, Hannes Reinecke wrote:
>> On 01/11/2017 10:39 PM, Jens Axboe wrote:
>>> Another year, another posting of this patchset. The previous posting
>>> was here:
>>>
>>> https://www.spinics.net/lists/kernel/msg2406106.html
>>>
>>> (yes, I've skipped v5, it was fixes on top of v4, not the rework).
>>>
>>> I've reworked bits of this to get rid of the shadow requests, thanks
>>> to Bart for the inspiration. The missing piece, for me, was the fact
>>> that we have the tags->rqs[] indirection array already. I've done this
>>> somewhat differently, though, by having the internal scheduler tag
>>> map be allocated/torn down when an IO scheduler is attached or
>>> detached. This also means that when we run without a scheduler, we
>>> don't have to do double tag allocations, it'll work like before.
>>>
>>> The patchset applies on top of 4.10-rc3, or can be pulled here:
>>>
>>> git://git.kernel.dk/linux-block blk-mq-sched.6
>>>
>> Well ... something's wrong here on my machine:
>>
[ .. ]
Turns out that selecting CONFIG_DEFAULT_MQ_DEADLINE is the culprit;
switching to CONFIG_DEFAULT_MQ_NONE and selecting mq-deadline after
booting manually makes the problem go away.
So there is a race condition during device init and switching the I/O
scheduler.
But the results from using mq-deadline are promising; the performance
drop I've seen on older hardware seems to be resolved:
mq iosched:
seq read : io=13383MB, bw=228349KB/s, iops=57087
rand read : io=12876MB, bw=219709KB/s, iops=54927
seq write: io=14532MB, bw=247987KB/s, iops=61996
rand write: io=13779MB, bw=235127KB/s, iops=58781
mq default:
seq read : io=13056MB, bw=222588KB/s, iops=55647
rand read : io=12908MB, bw=220069KB/s, iops=55017
seq write: io=13986MB, bw=238444KB/s, iops=59611
rand write: io=13733MB, bw=234128KB/s, iops=58532
sq default:
seq read : io=10240MB, bw=194787KB/s, iops=48696
rand read : io=10240MB, bw=191374KB/s, iops=47843
seq write: io=10240MB, bw=245333KB/s, iops=61333
rand write: io=10240MB, bw=228239KB/s, iops=57059
measured on mpt2sas with SSD devices.
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N�rnberg
GF: F. Imend�rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N�rnberg)
WARNING: multiple messages have this Message-ID (diff)
From: Hannes Reinecke <hare@suse.de>
To: Jens Axboe <axboe@fb.com>,
linux-kernel@vger.kernel.org, linux-block@vger.kernel.org
Cc: osandov@osandov.com, bart.vanassche@sandisk.com
Subject: Re: [PATCHSET v6] blk-mq scheduling framework
Date: Fri, 13 Jan 2017 13:10:07 +0100 [thread overview]
Message-ID: <cab72b22-d672-3c68-ec4a-6e3cb65f93ab@suse.de> (raw)
In-Reply-To: <5850a4ad-74a1-8d30-e865-fbd35679e84f@suse.de>
On 01/13/2017 12:04 PM, Hannes Reinecke wrote:
> On 01/13/2017 09:15 AM, Hannes Reinecke wrote:
>> On 01/11/2017 10:39 PM, Jens Axboe wrote:
>>> Another year, another posting of this patchset. The previous posting
>>> was here:
>>>
>>> https://www.spinics.net/lists/kernel/msg2406106.html
>>>
>>> (yes, I've skipped v5, it was fixes on top of v4, not the rework).
>>>
>>> I've reworked bits of this to get rid of the shadow requests, thanks
>>> to Bart for the inspiration. The missing piece, for me, was the fact
>>> that we have the tags->rqs[] indirection array already. I've done this
>>> somewhat differently, though, by having the internal scheduler tag
>>> map be allocated/torn down when an IO scheduler is attached or
>>> detached. This also means that when we run without a scheduler, we
>>> don't have to do double tag allocations, it'll work like before.
>>>
>>> The patchset applies on top of 4.10-rc3, or can be pulled here:
>>>
>>> git://git.kernel.dk/linux-block blk-mq-sched.6
>>>
>> Well ... something's wrong here on my machine:
>>
[ .. ]
Turns out that selecting CONFIG_DEFAULT_MQ_DEADLINE is the culprit;
switching to CONFIG_DEFAULT_MQ_NONE and selecting mq-deadline after
booting manually makes the problem go away.
So there is a race condition during device init and switching the I/O
scheduler.
But the results from using mq-deadline are promising; the performance
drop I've seen on older hardware seems to be resolved:
mq iosched:
seq read : io=13383MB, bw=228349KB/s, iops=57087
rand read : io=12876MB, bw=219709KB/s, iops=54927
seq write: io=14532MB, bw=247987KB/s, iops=61996
rand write: io=13779MB, bw=235127KB/s, iops=58781
mq default:
seq read : io=13056MB, bw=222588KB/s, iops=55647
rand read : io=12908MB, bw=220069KB/s, iops=55017
seq write: io=13986MB, bw=238444KB/s, iops=59611
rand write: io=13733MB, bw=234128KB/s, iops=58532
sq default:
seq read : io=10240MB, bw=194787KB/s, iops=48696
rand read : io=10240MB, bw=191374KB/s, iops=47843
seq write: io=10240MB, bw=245333KB/s, iops=61333
rand write: io=10240MB, bw=228239KB/s, iops=57059
measured on mpt2sas with SSD devices.
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
next prev parent reply other threads:[~2017-01-13 12:10 UTC|newest]
Thread overview: 93+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-11 21:39 [PATCHSET v6] blk-mq scheduling framework Jens Axboe
2017-01-11 21:39 ` [PATCH 01/10] block: move existing elevator ops to union Jens Axboe
2017-01-12 10:15 ` Johannes Thumshirn
2017-01-12 10:15 ` Johannes Thumshirn
2017-01-12 21:17 ` Bart Van Assche
2017-01-12 21:17 ` Bart Van Assche
2017-01-13 8:34 ` Christoph Hellwig
2017-01-13 15:00 ` Jens Axboe
2017-01-11 21:39 ` [PATCH 02/10] blk-mq: make mq_ops a const pointer Jens Axboe
2017-01-12 10:14 ` Johannes Thumshirn
2017-01-12 10:14 ` Johannes Thumshirn
2017-01-13 8:16 ` Christoph Hellwig
2017-01-11 21:39 ` [PATCH 03/10] block: move rq_ioc() to blk.h Jens Axboe
2017-01-12 10:14 ` Johannes Thumshirn
2017-01-12 10:14 ` Johannes Thumshirn
2017-01-12 21:18 ` Bart Van Assche
2017-01-12 21:18 ` Bart Van Assche
2017-01-13 8:33 ` Christoph Hellwig
2017-01-11 21:39 ` [PATCH 04/10] blk-mq: un-export blk_mq_free_hctx_request() Jens Axboe
2017-01-12 10:13 ` Johannes Thumshirn
2017-01-12 10:13 ` Johannes Thumshirn
2017-01-12 21:18 ` Bart Van Assche
2017-01-12 21:18 ` Bart Van Assche
2017-01-13 8:16 ` Christoph Hellwig
2017-01-11 21:39 ` [PATCH 05/10] blk-mq: export some helpers we need to the scheduling framework Jens Axboe
2017-01-12 10:17 ` Johannes Thumshirn
2017-01-12 10:17 ` Johannes Thumshirn
2017-01-12 21:20 ` Bart Van Assche
2017-01-12 21:20 ` Bart Van Assche
2017-01-13 8:17 ` Christoph Hellwig
2017-01-13 15:01 ` Jens Axboe
2017-01-11 21:39 ` [PATCH 06/10] blk-mq-tag: cleanup the normal/reserved tag allocation Jens Axboe
2017-01-12 21:22 ` Bart Van Assche
2017-01-12 21:22 ` Bart Van Assche
2017-01-12 22:07 ` Jens Axboe
2017-01-13 8:30 ` Christoph Hellwig
2017-01-13 15:06 ` Jens Axboe
2017-01-11 21:40 ` [PATCH 07/10] blk-mq: abstract out helpers for allocating/freeing tag maps Jens Axboe
2017-01-12 21:29 ` Bart Van Assche
2017-01-12 21:29 ` Bart Van Assche
2017-01-12 21:54 ` Jens Axboe
2017-01-13 8:25 ` Johannes Thumshirn
2017-01-13 8:25 ` Johannes Thumshirn
2017-01-11 21:40 ` [PATCH 08/10] blk-mq-sched: add framework for MQ capable IO schedulers Jens Axboe
2017-01-12 21:45 ` Bart Van Assche
2017-01-12 21:45 ` Bart Van Assche
2017-01-12 21:59 ` Jens Axboe
2017-01-13 11:15 ` Hannes Reinecke
2017-01-13 11:15 ` Hannes Reinecke
2017-01-13 16:39 ` Bart Van Assche
2017-01-13 16:39 ` Bart Van Assche
2017-01-13 16:41 ` Omar Sandoval
2017-01-13 17:43 ` Hannes Reinecke
2017-01-13 17:43 ` Hannes Reinecke
2017-01-11 21:40 ` [PATCH 09/10] mq-deadline: add blk-mq adaptation of the deadline IO scheduler Jens Axboe
2017-01-12 21:53 ` Bart Van Assche
2017-01-12 21:53 ` Bart Van Assche
2017-01-11 21:40 ` [PATCH 10/10] blk-mq-sched: allow setting of default " Jens Axboe
2017-01-12 21:54 ` Bart Van Assche
2017-01-12 21:54 ` Bart Van Assche
2017-01-12 21:16 ` [PATCHSET v6] blk-mq scheduling framework Bart Van Assche
2017-01-12 21:16 ` Bart Van Assche
2017-01-13 8:15 ` Hannes Reinecke
2017-01-13 8:15 ` Hannes Reinecke
2017-01-13 11:04 ` Hannes Reinecke
2017-01-13 11:04 ` Hannes Reinecke
2017-01-13 12:10 ` Hannes Reinecke [this message]
2017-01-13 12:10 ` Hannes Reinecke
2017-01-13 15:05 ` Jens Axboe
2017-01-13 15:03 ` Jens Axboe
2017-01-13 15:23 ` Jens Axboe
2017-01-13 15:23 ` Jens Axboe
2017-01-13 15:33 ` Hannes Reinecke
2017-01-13 15:33 ` Hannes Reinecke
2017-01-13 15:34 ` Jens Axboe
2017-01-13 15:34 ` Jens Axboe
2017-01-13 15:59 ` Hannes Reinecke
2017-01-13 15:59 ` Hannes Reinecke
2017-01-13 16:00 ` Jens Axboe
2017-01-13 16:00 ` Jens Axboe
2017-01-13 16:02 ` Jens Axboe
2017-01-13 21:45 ` Jens Axboe
2017-01-16 8:11 ` Hannes Reinecke
2017-01-16 8:11 ` Hannes Reinecke
2017-01-16 15:12 ` Jens Axboe
2017-01-16 15:16 ` Jens Axboe
2017-01-16 15:47 ` Jens Axboe
2017-01-13 10:09 ` Hannes Reinecke
2017-01-13 10:09 ` Hannes Reinecke
2017-01-15 10:12 ` Paolo Valente
2017-01-15 10:12 ` Paolo Valente
2017-01-15 15:55 ` Jens Axboe
2017-01-15 15:55 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cab72b22-d672-3c68-ec4a-6e3cb65f93ab@suse.de \
--to=hare@suse.de \
--cc=axboe@fb.com \
--cc=bart.vanassche@sandisk.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=osandov@osandov.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.