* [patch 3/3]cfq-iosched: don't idle if a deep seek queue is slow
@ 2010-11-08 2:07 Shaohua Li
2010-11-08 14:20 ` Vivek Goyal
0 siblings, 1 reply; 6+ messages in thread
From: Shaohua Li @ 2010-11-08 2:07 UTC (permalink / raw)
To: lkml; +Cc: Jens Axboe, vgoyal, czoccolo
If a deep seek queue slowly deliver requests but disk is much faster, idle
for the queue just wastes disk throughput. If the queue delevers all requests
before half its slice is used, the patch disable idle for it.
In my test, application delivers 32 requests one time, the disk can accept
128 requests at maxium and disk is fast. without the patch, the throughput
is just around 30m/s, while with it, the speed is about 80m/s. The disk is
a SSD, but is detected as a rotational disk. I can configure it as SSD, but
I thought the deep seek queue logic should be fixed too, for example,
considering a fast raid.
Signed-off-by: Shaohua Li <shaohua.li@intel.com>
---
block/cfq-iosched.c | 11 +++++++++++
1 file changed, 11 insertions(+)
Index: linux/block/cfq-iosched.c
===================================================================
--- linux.orig/block/cfq-iosched.c 2010-11-08 08:43:51.000000000 +0800
+++ linux/block/cfq-iosched.c 2010-11-08 08:49:52.000000000 +0800
@@ -2293,6 +2293,17 @@ static struct cfq_queue *cfq_select_queu
goto keep_queue;
}
+ /*
+ * This is a deep seek queue, but the device is much faster than
+ * the queue can deliver, don't idle
+ **/
+ if (CFQQ_SEEKY(cfqq) && cfq_cfqq_idle_window(cfqq) &&
+ (cfq_cfqq_slice_new(cfqq) ||
+ (cfqq->slice_end - jiffies > jiffies - cfqq->slice_start))) {
+ cfq_clear_cfqq_deep(cfqq);
+ cfq_clear_cfqq_idle_window(cfqq);
+ }
+
if (cfqq->dispatched && cfq_should_idle(cfqd, cfqq)) {
cfqq = NULL;
goto keep_queue;
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [patch 3/3]cfq-iosched: don't idle if a deep seek queue is slow
2010-11-08 2:07 [patch 3/3]cfq-iosched: don't idle if a deep seek queue is slow Shaohua Li
@ 2010-11-08 14:20 ` Vivek Goyal
2010-11-08 15:06 ` Jens Axboe
0 siblings, 1 reply; 6+ messages in thread
From: Vivek Goyal @ 2010-11-08 14:20 UTC (permalink / raw)
To: Shaohua Li; +Cc: lkml, Jens Axboe, czoccolo
On Mon, Nov 08, 2010 at 10:07:25AM +0800, Shaohua Li wrote:
> If a deep seek queue slowly deliver requests but disk is much faster, idle
> for the queue just wastes disk throughput. If the queue delevers all requests
> before half its slice is used, the patch disable idle for it.
> In my test, application delivers 32 requests one time, the disk can accept
> 128 requests at maxium and disk is fast. without the patch, the throughput
> is just around 30m/s, while with it, the speed is about 80m/s. The disk is
> a SSD, but is detected as a rotational disk. I can configure it as SSD, but
> I thought the deep seek queue logic should be fixed too, for example,
> considering a fast raid.
>
Hi Shaohua,
So looks like you are trying to cut down queue idling in the case when
device is fast and idling hurts. That's a noble goal, just that detetction
of this condition only for deep queues does not seem to cover lots of
cases. Manually one can set slice_idle=0 to handle this situation.
What about if you have lots of sequential queues (not deep) and they all
will still idle.
Secondly, what if driver is just buffering lots of requests in its device
queue and not necessarily device is processing the reuqests faster.
So I think it is a good idea to cut down on idling if we can find that
underlying device is fast and idling on queue might hurt more. But
discovering this only using deep queues does not sound very appleaing to
me. This is help only a particular workload which is driving deep queues.
So if there was a generic mechanism to tackle this, that would be much
better.
Vivek
> Signed-off-by: Shaohua Li <shaohua.li@intel.com>
>
> ---
> block/cfq-iosched.c | 11 +++++++++++
> 1 file changed, 11 insertions(+)
>
> Index: linux/block/cfq-iosched.c
> ===================================================================
> --- linux.orig/block/cfq-iosched.c 2010-11-08 08:43:51.000000000 +0800
> +++ linux/block/cfq-iosched.c 2010-11-08 08:49:52.000000000 +0800
> @@ -2293,6 +2293,17 @@ static struct cfq_queue *cfq_select_queu
> goto keep_queue;
> }
>
> + /*
> + * This is a deep seek queue, but the device is much faster than
> + * the queue can deliver, don't idle
> + **/
> + if (CFQQ_SEEKY(cfqq) && cfq_cfqq_idle_window(cfqq) &&
> + (cfq_cfqq_slice_new(cfqq) ||
> + (cfqq->slice_end - jiffies > jiffies - cfqq->slice_start))) {
> + cfq_clear_cfqq_deep(cfqq);
> + cfq_clear_cfqq_idle_window(cfqq);
> + }
> +
> if (cfqq->dispatched && cfq_should_idle(cfqd, cfqq)) {
> cfqq = NULL;
> goto keep_queue;
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [patch 3/3]cfq-iosched: don't idle if a deep seek queue is slow
2010-11-08 14:20 ` Vivek Goyal
@ 2010-11-08 15:06 ` Jens Axboe
2010-11-09 1:36 ` Shaohua Li
0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2010-11-08 15:06 UTC (permalink / raw)
To: Vivek Goyal; +Cc: Shaohua Li, lkml, czoccolo
On 2010-11-08 15:20, Vivek Goyal wrote:
> On Mon, Nov 08, 2010 at 10:07:25AM +0800, Shaohua Li wrote:
>> If a deep seek queue slowly deliver requests but disk is much faster, idle
>> for the queue just wastes disk throughput. If the queue delevers all requests
>> before half its slice is used, the patch disable idle for it.
>> In my test, application delivers 32 requests one time, the disk can accept
>> 128 requests at maxium and disk is fast. without the patch, the throughput
>> is just around 30m/s, while with it, the speed is about 80m/s. The disk is
>> a SSD, but is detected as a rotational disk. I can configure it as SSD, but
>> I thought the deep seek queue logic should be fixed too, for example,
>> considering a fast raid.
>>
>
> Hi Shaohua,
>
> So looks like you are trying to cut down queue idling in the case when
> device is fast and idling hurts. That's a noble goal, just that detetction
> of this condition only for deep queues does not seem to cover lots of
> cases. Manually one can set slice_idle=0 to handle this situation.
>
> What about if you have lots of sequential queues (not deep) and they all
> will still idle.
>
> Secondly, what if driver is just buffering lots of requests in its device
> queue and not necessarily device is processing the reuqests faster.
That is not a valid concern, a driver should never extract more than it
can process (pretty much) immediately.
> So I think it is a good idea to cut down on idling if we can find that
> underlying device is fast and idling on queue might hurt more. But
> discovering this only using deep queues does not sound very appleaing to
> me. This is help only a particular workload which is driving deep queues.
> So if there was a generic mechanism to tackle this, that would be much
> better.
Agree, we could use better metrics for this.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [patch 3/3]cfq-iosched: don't idle if a deep seek queue is slow
2010-11-08 15:06 ` Jens Axboe
@ 2010-11-09 1:36 ` Shaohua Li
2010-11-09 2:28 ` Vivek Goyal
0 siblings, 1 reply; 6+ messages in thread
From: Shaohua Li @ 2010-11-09 1:36 UTC (permalink / raw)
To: Jens Axboe; +Cc: Vivek Goyal, lkml, czoccolo
On Mon, 2010-11-08 at 23:06 +0800, Jens Axboe wrote:
> On 2010-11-08 15:20, Vivek Goyal wrote:
> > On Mon, Nov 08, 2010 at 10:07:25AM +0800, Shaohua Li wrote:
> >> If a deep seek queue slowly deliver requests but disk is much faster, idle
> >> for the queue just wastes disk throughput. If the queue delevers all requests
> >> before half its slice is used, the patch disable idle for it.
> >> In my test, application delivers 32 requests one time, the disk can accept
> >> 128 requests at maxium and disk is fast. without the patch, the throughput
> >> is just around 30m/s, while with it, the speed is about 80m/s. The disk is
> >> a SSD, but is detected as a rotational disk. I can configure it as SSD, but
> >> I thought the deep seek queue logic should be fixed too, for example,
> >> considering a fast raid.
> >>
> >
> > Hi Shaohua,
> >
> > So looks like you are trying to cut down queue idling in the case when
> > device is fast and idling hurts. That's a noble goal, just that detetction
> > of this condition only for deep queues does not seem to cover lots of
> > cases. Manually one can set slice_idle=0 to handle this situation.
> >
> > What about if you have lots of sequential queues (not deep) and they all
> > will still idle.
> >
> > Secondly, what if driver is just buffering lots of requests in its device
> > queue and not necessarily device is processing the reuqests faster.
>
> That is not a valid concern, a driver should never extract more than it
> can process (pretty much) immediately.
>
> > So I think it is a good idea to cut down on idling if we can find that
> > underlying device is fast and idling on queue might hurt more. But
> > discovering this only using deep queues does not sound very appleaing to
> > me. This is help only a particular workload which is driving deep queues.
> > So if there was a generic mechanism to tackle this, that would be much
> > better.
>
> Agree, we could use better metrics for this.
Agree we'd better have a better method to measure device speed, but this
seems not easy. Even in a fast device, a request might take long time to
finish when NCQ is enabled. Before we have generic mechanism, we still
need fix some particular cases.
Thanks,
Shaohua
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [patch 3/3]cfq-iosched: don't idle if a deep seek queue is slow
2010-11-09 1:36 ` Shaohua Li
@ 2010-11-09 2:28 ` Vivek Goyal
2010-11-09 2:31 ` Shaohua Li
0 siblings, 1 reply; 6+ messages in thread
From: Vivek Goyal @ 2010-11-09 2:28 UTC (permalink / raw)
To: Shaohua Li; +Cc: Jens Axboe, lkml, czoccolo
On Tue, Nov 09, 2010 at 09:36:42AM +0800, Shaohua Li wrote:
> On Mon, 2010-11-08 at 23:06 +0800, Jens Axboe wrote:
> > On 2010-11-08 15:20, Vivek Goyal wrote:
> > > On Mon, Nov 08, 2010 at 10:07:25AM +0800, Shaohua Li wrote:
> > >> If a deep seek queue slowly deliver requests but disk is much faster, idle
> > >> for the queue just wastes disk throughput. If the queue delevers all requests
> > >> before half its slice is used, the patch disable idle for it.
> > >> In my test, application delivers 32 requests one time, the disk can accept
> > >> 128 requests at maxium and disk is fast. without the patch, the throughput
> > >> is just around 30m/s, while with it, the speed is about 80m/s. The disk is
> > >> a SSD, but is detected as a rotational disk. I can configure it as SSD, but
> > >> I thought the deep seek queue logic should be fixed too, for example,
> > >> considering a fast raid.
> > >>
> > >
> > > Hi Shaohua,
> > >
> > > So looks like you are trying to cut down queue idling in the case when
> > > device is fast and idling hurts. That's a noble goal, just that detetction
> > > of this condition only for deep queues does not seem to cover lots of
> > > cases. Manually one can set slice_idle=0 to handle this situation.
> > >
> > > What about if you have lots of sequential queues (not deep) and they all
> > > will still idle.
> > >
> > > Secondly, what if driver is just buffering lots of requests in its device
> > > queue and not necessarily device is processing the reuqests faster.
> >
> > That is not a valid concern, a driver should never extract more than it
> > can process (pretty much) immediately.
> >
> > > So I think it is a good idea to cut down on idling if we can find that
> > > underlying device is fast and idling on queue might hurt more. But
> > > discovering this only using deep queues does not sound very appleaing to
> > > me. This is help only a particular workload which is driving deep queues.
> > > So if there was a generic mechanism to tackle this, that would be much
> > > better.
> >
> > Agree, we could use better metrics for this.
> Agree we'd better have a better method to measure device speed, but this
> seems not easy. Even in a fast device, a request might take long time to
> finish when NCQ is enabled. Before we have generic mechanism, we still
> need fix some particular cases.
Do you have a real workload for this case or it is just one of the synthetic
workload simulated using fio?
Vivek
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [patch 3/3]cfq-iosched: don't idle if a deep seek queue is slow
2010-11-09 2:28 ` Vivek Goyal
@ 2010-11-09 2:31 ` Shaohua Li
0 siblings, 0 replies; 6+ messages in thread
From: Shaohua Li @ 2010-11-09 2:31 UTC (permalink / raw)
To: Vivek Goyal; +Cc: Jens Axboe, lkml, czoccolo
On Tue, 2010-11-09 at 10:28 +0800, Vivek Goyal wrote:
> On Tue, Nov 09, 2010 at 09:36:42AM +0800, Shaohua Li wrote:
> > On Mon, 2010-11-08 at 23:06 +0800, Jens Axboe wrote:
> > > On 2010-11-08 15:20, Vivek Goyal wrote:
> > > > On Mon, Nov 08, 2010 at 10:07:25AM +0800, Shaohua Li wrote:
> > > >> If a deep seek queue slowly deliver requests but disk is much faster, idle
> > > >> for the queue just wastes disk throughput. If the queue delevers all requests
> > > >> before half its slice is used, the patch disable idle for it.
> > > >> In my test, application delivers 32 requests one time, the disk can accept
> > > >> 128 requests at maxium and disk is fast. without the patch, the throughput
> > > >> is just around 30m/s, while with it, the speed is about 80m/s. The disk is
> > > >> a SSD, but is detected as a rotational disk. I can configure it as SSD, but
> > > >> I thought the deep seek queue logic should be fixed too, for example,
> > > >> considering a fast raid.
> > > >>
> > > >
> > > > Hi Shaohua,
> > > >
> > > > So looks like you are trying to cut down queue idling in the case when
> > > > device is fast and idling hurts. That's a noble goal, just that detetction
> > > > of this condition only for deep queues does not seem to cover lots of
> > > > cases. Manually one can set slice_idle=0 to handle this situation.
> > > >
> > > > What about if you have lots of sequential queues (not deep) and they all
> > > > will still idle.
> > > >
> > > > Secondly, what if driver is just buffering lots of requests in its device
> > > > queue and not necessarily device is processing the reuqests faster.
> > >
> > > That is not a valid concern, a driver should never extract more than it
> > > can process (pretty much) immediately.
> > >
> > > > So I think it is a good idea to cut down on idling if we can find that
> > > > underlying device is fast and idling on queue might hurt more. But
> > > > discovering this only using deep queues does not sound very appleaing to
> > > > me. This is help only a particular workload which is driving deep queues.
> > > > So if there was a generic mechanism to tackle this, that would be much
> > > > better.
> > >
> > > Agree, we could use better metrics for this.
> > Agree we'd better have a better method to measure device speed, but this
> > seems not easy. Even in a fast device, a request might take long time to
> > finish when NCQ is enabled. Before we have generic mechanism, we still
> > need fix some particular cases.
>
> Do you have a real workload for this case or it is just one of the synthetic
> workload simulated using fio?
No, no real workload. We do a lot of fio tests with different scripts,
this is from one of our tests.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-11-09 2:31 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-11-08 2:07 [patch 3/3]cfq-iosched: don't idle if a deep seek queue is slow Shaohua Li
2010-11-08 14:20 ` Vivek Goyal
2010-11-08 15:06 ` Jens Axboe
2010-11-09 1:36 ` Shaohua Li
2010-11-09 2:28 ` Vivek Goyal
2010-11-09 2:31 ` Shaohua Li
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).