[SHED][IO-SHED] Are we missing the big picture?

[SHED][IO-SHED] Are we missing the big picture?

[Ian Kumlien]

Hi all,

I have been following the sheduler and interactivity discussions closely
but via the marc.theaimsgroup.com archive, So i might be behind etc...
=P

[Note: sorry if i sound like mr.know-it-all etc, just trying to get a
point across]

Anyways, i think that the AS discussions that i have seen has missed
some points. Getting the processes priority in AS is one thing, but fist
of all i think there should be a stand off layer. Let me explain:

I liked Jens Axobe's 'CBQ' alike implementation (based on the idea of
Andrea A. (afair i have the names right) since it does the most
important thing... which is *nothing* when there is no load (ie, pass
trough).

AS might be/is the best damn io sheduler for loaded machines but when
there is no load, it's overhead. So in my opinion there should be
something that first warrants the usage of AS before it's actually
engaged.

And, if it's only engaged during high load, additions like basing the
requests priority on the process/tasks priority would make total sense,
adding the 'wakeup on wait' or what it was would also make total
sense... But how many of your machines uses the disk 100% of the time?
(in the real world... )

I don't know how 'CBQ' was implemented but any 'we are under load now'
trigger would do it for me.

Please see to it that my CC is included in any discussions =)

PS. Or was it a version of SFQ? in that case s/CBQ/SFQ/g
DS.

-- 
Ian Kumlien <pomac@vapor.com>

Re: [SHED][IO-SHED] Are we missing the big picture?

[Nick Piggin]

Ian Kumlien wrote:

>Hi all,
>
>I have been following the sheduler and interactivity discussions closely
>but via the marc.theaimsgroup.com archive, So i might be behind etc...
>=P
>
>[Note: sorry if i sound like mr.know-it-all etc, just trying to get a
>point across]
>
>Anyways, i think that the AS discussions that i have seen has missed
>some points. Getting the processes priority in AS is one thing, but fist
>of all i think there should be a stand off layer. Let me explain:
>
>I liked Jens Axobe's 'CBQ' alike implementation (based on the idea of
>Andrea A. (afair i have the names right) since it does the most
>important thing... which is *nothing* when there is no load (ie, pass
>trough).
>
>AS might be/is the best damn io sheduler for loaded machines but when
>there is no load, it's overhead. So in my opinion there should be
>something that first warrants the usage of AS before it's actually
>engaged.
>
>And, if it's only engaged during high load, additions like basing the
>requests priority on the process/tasks priority would make total sense,
>adding the 'wakeup on wait' or what it was would also make total
>sense... But how many of your machines uses the disk 100% of the time?
>(in the real world... )
>
>I don't know how 'CBQ' was implemented but any 'we are under load now'
>trigger would do it for me.
>
>Please see to it that my CC is included in any discussions =)
>
>PS. Or was it a version of SFQ? in that case s/CBQ/SFQ/g
>

To start with its CFQ. Also could you clarify what you mean by
load and what you mean by CFQ doing nothing, and why AS is overhead
in the no load case. I can't really follow what you are saying.

Re: [SHED][IO-SHED] Are we missing the big picture?

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 2452 bytes --]

On Fri, 2003-08-01 at 08:27, Nick Piggin wrote:
> Ian Kumlien wrote:
> 
> >Hi all,
> >
> >I have been following the sheduler and interactivity discussions closely
> >but via the marc.theaimsgroup.com archive, So i might be behind etc...
> >=P
> >
> >[Note: sorry if i sound like mr.know-it-all etc, just trying to get a
> >point across]
> >
> >Anyways, i think that the AS discussions that i have seen has missed
> >some points. Getting the processes priority in AS is one thing, but fist
> >of all i think there should be a stand off layer. Let me explain:
> >
> >I liked Jens Axobe's 'CBQ' alike implementation (based on the idea of
> >Andrea A. (afair i have the names right) since it does the most
> >important thing... which is *nothing* when there is no load (ie, pass
> >trough).
> >
> >AS might be/is the best damn io sheduler for loaded machines but when
> >there is no load, it's overhead. So in my opinion there should be
> >something that first warrants the usage of AS before it's actually
> >engaged.
> >
> >And, if it's only engaged during high load, additions like basing the
> >requests priority on the process/tasks priority would make total sense,
> >adding the 'wakeup on wait' or what it was would also make total
> >sense... But how many of your machines uses the disk 100% of the time?
> >(in the real world... )
> >
> >I don't know how 'CBQ' was implemented but any 'we are under load now'
> >trigger would do it for me.
> >
> >Please see to it that my CC is included in any discussions =)
> >
> >PS. Or was it a version of SFQ? in that case s/CBQ/SFQ/g
> >
> 
> To start with its CFQ. Also could you clarify what you mean by
> load and what you mean by CFQ doing nothing, and why AS is overhead
> in the no load case. I can't really follow what you are saying.

CFQ passes the req's on directly until there is enough load... In the
load case it builds queues. Just like SFQ (but sfq can drop packets
afair).

This way, we wouldn't have the initial
'can-we-merge-this-with-other-data-coming' delay when not needed.

If as could be attached to the 'queue build up' then AS would only be
doing what it's good at, throughput and minimizing head movements.

Also patches that move prioritized data (data for processes with high
pri) would fit right in there since you'd only be doing it during actual
load.

(note: load as in disk load)
-- 
Ian Kumlien <pomac@vapor.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [SHED][IO-SHED] Are we missing the big picture?

[Nick Piggin]

Ian Kumlien wrote:

>On Fri, 2003-08-01 at 08:27, Nick Piggin wrote:
>
>>Ian Kumlien wrote:
>>
>>
>>>Hi all,
>>>
>>>I have been following the sheduler and interactivity discussions closely
>>>but via the marc.theaimsgroup.com archive, So i might be behind etc...
>>>=P
>>>
>>>[Note: sorry if i sound like mr.know-it-all etc, just trying to get a
>>>point across]
>>>
>>>Anyways, i think that the AS discussions that i have seen has missed
>>>some points. Getting the processes priority in AS is one thing, but fist
>>>of all i think there should be a stand off layer. Let me explain:
>>>
>>>I liked Jens Axobe's 'CBQ' alike implementation (based on the idea of
>>>Andrea A. (afair i have the names right) since it does the most
>>>important thing... which is *nothing* when there is no load (ie, pass
>>>trough).
>>>
>>>AS might be/is the best damn io sheduler for loaded machines but when
>>>there is no load, it's overhead. So in my opinion there should be
>>>something that first warrants the usage of AS before it's actually
>>>engaged.
>>>
>>>And, if it's only engaged during high load, additions like basing the
>>>requests priority on the process/tasks priority would make total sense,
>>>adding the 'wakeup on wait' or what it was would also make total
>>>sense... But how many of your machines uses the disk 100% of the time?
>>>(in the real world... )
>>>
>>>I don't know how 'CBQ' was implemented but any 'we are under load now'
>>>trigger would do it for me.
>>>
>>>Please see to it that my CC is included in any discussions =)
>>>
>>>PS. Or was it a version of SFQ? in that case s/CBQ/SFQ/g
>>>
>>>
>>To start with its CFQ. Also could you clarify what you mean by
>>load and what you mean by CFQ doing nothing, and why AS is overhead
>>in the no load case. I can't really follow what you are saying.
>>
>
>CFQ passes the req's on directly until there is enough load... In the
>load case it builds queues. Just like SFQ (but sfq can drop packets
>afair).
>

What do you mean? CFQ merges and sorts requests, and it services
each process in a round robin manner. I don't have the CFQ code
at hand, but I don't think it does anything different in the
"load" case.

>
>This way, we wouldn't have the initial
>'can-we-merge-this-with-other-data-coming' delay when not needed.
>

No that is what queue plugging can be used for.

>
>If as could be attached to the 'queue build up' then AS would only be
>doing what it's good at, throughput and minimizing head movements.
>

No, AS does only try to do what it is good at. As complex as it
is, its meant to be almost as simple as possible.

>
>Also patches that move prioritized data (data for processes with high
>pri) would fit right in there since you'd only be doing it during actual
>load.
>
>(note: load as in disk load)
>

I'm sorry but I'm having trouble working out what you are trying to say.
The disk gets its work in the form of a request. Linux keeps a queue of
outstanding requests for each disk. The IO schedulers are a layer between
the disk (driver) and the request queue. They get to choose the next
request that goes to the disk. AS is only set apart because it sometimes
chooses not to send a request at all even if there are some available.

Now what do you mean by disk load? And actual load? I can imagine that
if there are no requests you would say there is no load on the disk. If
there is 1 request there must be some actual load? Maybe you mean more
than 1 process with outstanding requests?

Re: [SHED][IO-SHED] Are we missing the big picture?

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 3850 bytes --]

On Sat, 2003-08-02 at 03:49, Nick Piggin wrote:
> Ian Kumlien wrote:
> 
> >On Fri, 2003-08-01 at 08:27, Nick Piggin wrote:
> >
> >>To start with its CFQ. Also could you clarify what you mean by
> >>load and what you mean by CFQ doing nothing, and why AS is overhead
> >>in the no load case. I can't really follow what you are saying.

> >CFQ passes the req's on directly until there is enough load... In the
> >load case it builds queues. Just like SFQ (but sfq can drop packets
> >afair).
> >

> What do you mean? CFQ merges and sorts requests, and it services
> each process in a round robin manner. I don't have the CFQ code
> at hand, but I don't think it does anything different in the
> "load" case.

Yes, i thought it worked in a different manner, as Jens Axboe pointed
out. I was under the impression that it worked differently.

> >This way, we wouldn't have the initial
> >'can-we-merge-this-with-other-data-coming' delay when not needed.

> No that is what queue plugging can be used for.

Queue plugging?

> >If as could be attached to the 'queue build up' then AS would only be
> >doing what it's good at, throughput and minimizing head movements.

> No, AS does only try to do what it is good at. As complex as it
> is, its meant to be almost as simple as possible.

Oh? from what i've gathered from my somewhat on and off readings of lkml
i thought that 1, AS added a delay to gather additional requests prior
to sending the request to the disk. 2, (as i found out now) it also
waits before moving the head.

Thats what i wanted to by pass in the "hey, i'm just loading my 10 bytes
config file, and nothing else is happening" scenario.

> >Also patches that move prioritized data (data for processes with high
> >pri) would fit right in there since you'd only be doing it during actual
> >load.

> I'm sorry but I'm having trouble working out what you are trying to say.
> The disk gets its work in the form of a request. Linux keeps a queue of
> outstanding requests for each disk. The IO schedulers are a layer between
> the disk (driver) and the request queue. They get to choose the next
> request that goes to the disk. AS is only set apart because it sometimes
> chooses not to send a request at all even if there are some available.

Which sounds odd... 

> Now what do you mean by disk load? And actual load? I can imagine that
> if there are no requests you would say there is no load on the disk. If
> there is 1 request there must be some actual load? Maybe you mean more
> than 1 process with outstanding requests?

Lets say that when the request queue has reached a certain length, the
disk is loaded.

I also have to say that i have no clue of the kernels real inner
workings esp not the io layer. I have only played with QoS, and i
started when it was undocumented.. =).

Lets say that we could use a CBQ queue with the right bw limits. Now,
lets say that when 70% of that bw is filled, we use a diff approach to
treating the data... We might drop new syns since we might be paranoid
about getting DoSed (stupid example, i know).

Now, Translate this in to the IO part. When we have a request backlog
for X entries during Y 'timeunits' we use a diff way to access the disk.
And since we know that when this mode is triggered, we have a backlog of
requests that we can just feed AS and thus make as do it's best.
Lower latency on small things and better throughput, what could be wrong
about that?

[Side note: In my understanding of SFQ it does 'nothing' to data that
can be sent directly, but queues when it can't (due to load)]

[Also: load is just a very fuzzy term for something that might be highly
complex. ie, the right value for the job in determining how the request
should be handled.]

-- 
Ian Kumlien <pomac@vapor.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [SHED][IO-SHED] Are we missing the big picture?

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 3448 bytes --]

On Fri, 2003-08-01 at 14:32, Jens Axboe wrote:
> On Fri, Aug 01 2003, Ian Kumlien wrote:
> > On Fri, 2003-08-01 at 11:00, Ihar "Philips" Filipau wrote:
> > >    Am I right - judging from your posting - that we finally reached 
> > > moment than Linux will have network-like queueing disciplines for disks 
> > > and CPUs?
> > 
> > CPU's? well, we do have a nice sheduler but i wouldn't say network-like
> > queuing.
> > 
> > >    In any way, CPU/disk throughput just another types of limited resource.
> > >    It would be nice to be able to manage it - who gets more, who gets 
> > > less. CPU/disk schedulers by manageability are far behind network.
> > >    IMHO must have for servers.
> > 
> > Yes, but, thats not what I'm saying.
> > 
> > CFQ as it apparently was called, builds a queue list when the disk is
> 
> I coined that phrase as a variant of SFQ, with C being Complete.

Yeah, i remember that now, but late at night your mind can be playing
tricks on you =P

> > under load. So you get really fast data access if there is no load, and
> > a common queue when there is load. The common queue is the bad thing
> > about CFQ, imagine putting AS there instead... This would mean fast data
> > access on unloaded systems, better throughput on loaded systems and
> > prioritization features could hook right in since AS would only be used
> > during load. IE, you can add all kinds of patches that only matters
> > during heavy load.
> 
> I dunno where you get this from, but you seem to have a misguide picture
> of how io schedulers work in Linux. AS works like deadline, but adds
> anticipation. This means you try to anticipate whether a process will
> issue a nearby read soon, and if so stall the disk head. deadline itself
> has a single queue for merging/sorting, and a single queue as a deadline
> fifo (for each data direction, read and write).

Hummm, from what i gathered from reading this list it has a 'standoff
period' (like you said after getting data but also) before it initiates
a read... And afair, CFQ did just about 'nothing' when the disk wasn't
under 'load'.

I also thought that deadline didn't merge and sort on the same level as
AS.

> Where CFQ differs is that it maintains such a backend system for each
> "class" (user, could be a task grouping of some sort too), with a small
> front end (class independent scheduler is used in some contexts) to
> select which class we do IO from. The old design just round-robined
> between all "busy" tasks, with some heuristics to minimize seeks.

Ahhh

> So for a single task, deadline and CFQ works the same basically. AS
> differs because of the anticipation of course.

Yes, but... 
I was more referring to the stalls in AS (the waits). If one could
eliminate them and just enable AS when the disk is loaded... Then we
could add all the 'bonus on wait' and or process priorities being
honored etc, atm, it just feels like overhead, it might be small but
still.

And, if sfq would build up one queue and feed it to AS, don't you think
there would be a gain? Esp in small reads when not loaded and scattered
small reads... Aswell as during heavy load?

> > > > I liked Jens Axobe's 'CBQ' alike implementation (based on the idea of
> > > > Andrea A. (afair i have the names right) since it does the most
> 
> Nope, it's Axboe :)

Argh!
/me fires himself =)

-- 
Ian Kumlien <pomac@vapor.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [SHED][IO-SHED] Are we missing the big picture?

[Jens Axboe]

On Fri, Aug 01 2003, Ian Kumlien wrote:
> On Fri, 2003-08-01 at 11:00, Ihar "Philips" Filipau wrote:
> >    Am I right - judging from your posting - that we finally reached 
> > moment than Linux will have network-like queueing disciplines for disks 
> > and CPUs?
> 
> CPU's? well, we do have a nice sheduler but i wouldn't say network-like
> queuing.
> 
> >    In any way, CPU/disk throughput just another types of limited resource.
> >    It would be nice to be able to manage it - who gets more, who gets 
> > less. CPU/disk schedulers by manageability are far behind network.
> >    IMHO must have for servers.
> 
> Yes, but, thats not what I'm saying.
> 
> CFQ as it apparently was called, builds a queue list when the disk is

I coined that phrase as a variant of SFQ, with C being Complete.

> under load. So you get really fast data access if there is no load, and
> a common queue when there is load. The common queue is the bad thing
> about CFQ, imagine putting AS there instead... This would mean fast data
> access on unloaded systems, better throughput on loaded systems and
> prioritization features could hook right in since AS would only be used
> during load. IE, you can add all kinds of patches that only matters
> during heavy load.

I dunno where you get this from, but you seem to have a misguide picture
of how io schedulers work in Linux. AS works like deadline, but adds
anticipation. This means you try to anticipate whether a process will
issue a nearby read soon, and if so stall the disk head. deadline itself
has a single queue for merging/sorting, and a single queue as a deadline
fifo (for each data direction, read and write).

Where CFQ differs is that it maintains such a backend system for each
"class" (user, could be a task grouping of some sort too), with a small
front end (class independent scheduler is used in some contexts) to
select which class we do IO from. The old design just round-robined
between all "busy" tasks, with some heuristics to minimize seeks.

So for a single task, deadline and CFQ works the same basically. AS
differs because of the anticipation of course.

> > > I liked Jens Axobe's 'CBQ' alike implementation (based on the idea of
> > > Andrea A. (afair i have the names right) since it does the most

Nope, it's Axboe :)

-- 
Jens Axboe

Re: [SHED][IO-SHED] Are we missing the big picture?

[Ian Kumlien]

[-- Attachment #1: Type: text/plain, Size: 2809 bytes --]

On Fri, 2003-08-01 at 11:00, Ihar "Philips" Filipau wrote:
>    Am I right - judging from your posting - that we finally reached 
> moment than Linux will have network-like queueing disciplines for disks 
> and CPUs?

CPU's? well, we do have a nice sheduler but i wouldn't say network-like
queuing.

>    In any way, CPU/disk throughput just another types of limited resource.
>    It would be nice to be able to manage it - who gets more, who gets 
> less. CPU/disk schedulers by manageability are far behind network.
>    IMHO must have for servers.

Yes, but, thats not what I'm saying.

CFQ as it apparently was called, builds a queue list when the disk is
under load. So you get really fast data access if there is no load, and
a common queue when there is load. The common queue is the bad thing
about CFQ, imagine putting AS there instead... This would mean fast data
access on unloaded systems, better throughput on loaded systems and
prioritization features could hook right in since AS would only be used
during load. IE, you can add all kinds of patches that only matters
during heavy load.

(note: load is only diskload)

> Ian Kumlien wrote:
> > Hi all,
> > 
> > I have been following the sheduler and interactivity discussions closely
> > but via the marc.theaimsgroup.com archive, So i might be behind etc...
> > =P
> > 
> > [Note: sorry if i sound like mr.know-it-all etc, just trying to get a
> > point across]
> > 
> > Anyways, i think that the AS discussions that i have seen has missed
> > some points. Getting the processes priority in AS is one thing, but fist
> > of all i think there should be a stand off layer. Let me explain:
> > 
> > I liked Jens Axobe's 'CBQ' alike implementation (based on the idea of
> > Andrea A. (afair i have the names right) since it does the most
> > important thing... which is *nothing* when there is no load (ie, pass
> > trough).
> > 
> > AS might be/is the best damn io sheduler for loaded machines but when
> > there is no load, it's overhead. So in my opinion there should be
> > something that first warrants the usage of AS before it's actually
> > engaged.
> > 
> > And, if it's only engaged during high load, additions like basing the
> > requests priority on the process/tasks priority would make total sense,
> > adding the 'wakeup on wait' or what it was would also make total
> > sense... But how many of your machines uses the disk 100% of the time?
> > (in the real world... )
> > 
> > I don't know how 'CBQ' was implemented but any 'we are under load now'
> > trigger would do it for me.
> > 
> > Please see to it that my CC is included in any discussions =)
> > 
> > PS. Or was it a version of SFQ? in that case s/CBQ/SFQ/g
> > DS.
> > 
-- 
Ian Kumlien <pomac@vapor.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [SHED][IO-SHED] Are we missing the big picture?

[Ihar "Philips" Filipau]

   Am I right - judging from your posting - that we finally reached 
moment than Linux will have network-like queueing disciplines for disks 
and CPUs?

   In any way, CPU/disk throughput just another types of limited resource.
   It would be nice to be able to manage it - who gets more, who gets 
less. CPU/disk schedulers by manageability are far behind network.
   IMHO must have for servers.

Ian Kumlien wrote:
> Hi all,
> 
> I have been following the sheduler and interactivity discussions closely
> but via the marc.theaimsgroup.com archive, So i might be behind etc...
> =P
> 
> [Note: sorry if i sound like mr.know-it-all etc, just trying to get a
> point across]
> 
> Anyways, i think that the AS discussions that i have seen has missed
> some points. Getting the processes priority in AS is one thing, but fist
> of all i think there should be a stand off layer. Let me explain:
> 
> I liked Jens Axobe's 'CBQ' alike implementation (based on the idea of
> Andrea A. (afair i have the names right) since it does the most
> important thing... which is *nothing* when there is no load (ie, pass
> trough).
> 
> AS might be/is the best damn io sheduler for loaded machines but when
> there is no load, it's overhead. So in my opinion there should be
> something that first warrants the usage of AS before it's actually
> engaged.
> 
> And, if it's only engaged during high load, additions like basing the
> requests priority on the process/tasks priority would make total sense,
> adding the 'wakeup on wait' or what it was would also make total
> sense... But how many of your machines uses the disk 100% of the time?
> (in the real world... )
> 
> I don't know how 'CBQ' was implemented but any 'we are under load now'
> trigger would do it for me.
> 
> Please see to it that my CC is included in any discussions =)
> 
> PS. Or was it a version of SFQ? in that case s/CBQ/SFQ/g
> DS.
>