From: Ingo Molnar <mingo@elte.hu>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Zach Brown <zach.brown@oracle.com>,
linux-kernel@vger.kernel.org, linux-aio@kvack.org,
Suparna Bhattacharya <suparna@in.ibm.com>,
Benjamin LaHaise <bcrl@kvack.org>
Subject: Re: [PATCH 2 of 4] Introduce i386 fibril scheduling
Date: Fri, 2 Feb 2007 11:49:00 +0100 [thread overview]
Message-ID: <20070202104900.GA13941@elte.hu> (raw)
In-Reply-To: <Pine.LNX.4.64.0702011154110.3632@woody.linux-foundation.org>
* Linus Torvalds <torvalds@linux-foundation.org> wrote:
> So stop blathering about scheduling costs, RT kernels and interrupts.
> Interrupts generally happen a few thousand times a second. This is
> soemthing you want to do a *million* times a second, without any IO
> happening at all except for when it has to.
we might be talking past each other.
i never suggested every aio op should create/destroy a kernel thread!
My only suggestion was to have a couple of transparent kernel threads
(not fibrils) attached to a user context that does asynchronous
syscalls! Those kernel threads would be 'switched in' if the current
user-space thread blocks - so instead of having to 'create' any of them
- the fast path would be to /switch/ them to under the current
user-space, so that user-space processing can continue under that other
thread!
That means that in the 'switch kernel context' fastpath it simply needs
to copy the blocked threads' user-space ptregs (~64 bytes) to its own
kernel stack, and then it can do a return-from-syscall without
user-space noticing the switch! Never would we really see the cost of
kernel thread creation. We would never see that cost in the fully cached
case (no other thread is needed then), nor would we see it in the
blocking-IO case, due to pooling. (there are some other details related
to things like the FPU context, but you get the idea.)
Let me quote Zach's reply to my suggestions:
| It'd certainly be doable to throw together a credible attempt to
| service "asys" system call submission with full-on kernel threads.
| That seems like reasonable due diligence to me. If full-on threads
| are almost as cheap, great. If fibrils are so much cheaper that they
| seem to warrant investing in, great.
that's all i wanted to see being considered!
Please ignore my points about scheduling costs - i only talked about
them at length because the only fundamental difference between kernel
threads and fibrils is their /scheduling/ properties. /Not/ the
setup/teardown costs - those are not relevant /precisely/ because they
can be pooled and because they happen relatively rarely, compared to the
cached case. The 'switch to the blocked thread's ptregs' operation also
involves a context-switch under this design. That's why i was talking
about scheduling so much: the /only/ true difference between fibrils and
kernel threads is their /scheduling/.
I believe this is the point where your argument fails:
> - setup/teardown costs. Both memory and CPU. This is where the current
> threads simply don't work. The setup cost of doing a clone/exit is
> actually much higher than the cost of doing the whole operation,
> most of the time.
you are comparing apples to oranges - i never said we should
create/destroy a kernel thread for every async op. That would be insane!
what we need to support asynchronous system-calls is the ability to pick
up an /already created/ kernel thread from a pool of per-task kernel
threads and to switch it to under the current user-space and return to
the user-space stack with that new kernel thread running. (The other,
blocked kernel thread stays blocked and is returned into the pool of
'pending' AIO kernel threads.) And this only needs to happen in the
'cachemiss' case anyway. In the 'cached' case no other kernel thread
would be involved at all, the current one just falls straight through
the system-call.
my argument is that the whole notion of cutting this at the kernel stack
and thread info level and making fibrils in essence a separate
scheduling entitity is wrong, wrong, wrong. Why not use plain kernel
threads for this?
[ finally, i think you totally ignored my main argument, state machines.
The networking stack is a full and very nice state machine. It's
kicked from user-space, and zillions of small contexts (sockets) are
living on without any of the originating tasks having to be involved.
So i'm still holding to the fundamental notion that within the kernel
this form of AIO is a nice but /secondary/ mechanism. If a subsystem
is able to pull it off, it can implement asynchronity via a state
machine - and it will outperform any thread based AIO. Or not. We'll
see. For something like the VFS i doubt we'll see (and i doubt we
/want/ to see) a 'native' state-machine implementation.
this is btw. quite close to the Tux model of doing asynchronous block
IO and asynchronous VFS events such as asynchronous open(). Tux uses a
pool of kernel threads to pass blocking work to, while not holding up
the 'main' thread. But the main Tux speedup comes from having a native
state machine for all the networking IO. ]
Ingo
next prev parent reply other threads:[~2007-02-02 10:50 UTC|newest]
Thread overview: 153+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-01-30 20:39 [PATCH 0 of 4] Generic AIO by scheduling stacks Zach Brown
2007-01-30 20:39 ` [PATCH 1 of 4] Introduce per_call_chain() Zach Brown
2007-01-30 20:39 ` [PATCH 2 of 4] Introduce i386 fibril scheduling Zach Brown
2007-02-01 8:36 ` Ingo Molnar
2007-02-01 13:02 ` Ingo Molnar
2007-02-01 13:19 ` Christoph Hellwig
2007-02-01 13:52 ` Ingo Molnar
2007-02-01 17:13 ` Mark Lord
2007-02-01 18:02 ` Ingo Molnar
2007-02-02 13:23 ` Andi Kleen
2007-02-01 21:52 ` Zach Brown
2007-02-01 22:23 ` Benjamin LaHaise
2007-02-01 22:37 ` Zach Brown
2007-02-02 13:22 ` Andi Kleen
2007-02-01 20:07 ` Linus Torvalds
2007-02-02 10:49 ` Ingo Molnar [this message]
2007-02-02 15:56 ` Linus Torvalds
2007-02-02 19:59 ` Alan
2007-02-02 20:14 ` Linus Torvalds
2007-02-02 20:58 ` Davide Libenzi
2007-02-02 21:09 ` Linus Torvalds
2007-02-02 21:30 ` Alan
2007-02-02 21:30 ` Linus Torvalds
2007-02-02 22:42 ` Ingo Molnar
2007-02-02 23:01 ` Linus Torvalds
2007-02-02 23:17 ` Linus Torvalds
2007-02-03 0:04 ` Alan
2007-02-03 0:23 ` bert hubert
2007-02-02 22:48 ` Alan
2007-02-05 16:44 ` Zach Brown
2007-02-02 22:21 ` Ingo Molnar
2007-02-02 22:49 ` Linus Torvalds
2007-02-02 23:55 ` Ingo Molnar
2007-02-03 0:56 ` Linus Torvalds
2007-02-03 7:15 ` Suparna Bhattacharya
2007-02-03 8:23 ` Ingo Molnar
2007-02-03 9:25 ` Matt Mackall
2007-02-03 10:03 ` Ingo Molnar
2007-02-05 17:44 ` Zach Brown
2007-02-05 19:26 ` Davide Libenzi
2007-02-05 19:41 ` Zach Brown
2007-02-05 20:10 ` Davide Libenzi
2007-02-05 20:21 ` Zach Brown
2007-02-05 20:42 ` Linus Torvalds
2007-02-05 20:39 ` Linus Torvalds
2007-02-05 21:09 ` Davide Libenzi
2007-02-05 21:31 ` Kent Overstreet
2007-02-06 20:25 ` Davide Libenzi
2007-02-06 20:46 ` Linus Torvalds
2007-02-06 21:16 ` David Miller
2007-02-06 21:28 ` Linus Torvalds
2007-02-06 21:31 ` David Miller
2007-02-06 21:46 ` Eric Dumazet
2007-02-06 21:50 ` Linus Torvalds
2007-02-06 22:28 ` Zach Brown
2007-02-06 22:45 ` Kent Overstreet
2007-02-06 23:04 ` Linus Torvalds
2007-02-07 1:22 ` Kent Overstreet
2007-02-06 23:23 ` Davide Libenzi
2007-02-06 23:39 ` Joel Becker
2007-02-06 23:56 ` Davide Libenzi
2007-02-07 0:06 ` Joel Becker
2007-02-07 0:23 ` Davide Libenzi
2007-02-07 0:44 ` Joel Becker
2007-02-07 1:15 ` Davide Libenzi
2007-02-07 1:24 ` Kent Overstreet
2007-02-07 1:30 ` Joel Becker
2007-02-07 6:16 ` Michael K. Edwards
2007-02-07 9:17 ` Michael K. Edwards
2007-02-07 9:37 ` Michael K. Edwards
2007-02-06 0:32 ` Davide Libenzi
2007-02-05 21:21 ` Zach Brown
2007-02-02 23:37 ` Davide Libenzi
2007-02-03 0:02 ` Davide Libenzi
2007-02-05 17:12 ` Zach Brown
2007-02-05 18:24 ` Davide Libenzi
2007-02-05 21:44 ` David Miller
2007-02-06 0:15 ` Davide Libenzi
2007-02-05 21:36 ` bert hubert
2007-02-05 21:57 ` Linus Torvalds
2007-02-05 22:07 ` bert hubert
2007-02-05 22:15 ` Zach Brown
2007-02-05 22:34 ` Davide Libenzi
2007-02-06 0:27 ` Scot McKinley
2007-02-06 0:48 ` David Miller
2007-02-06 0:48 ` Joel Becker
2007-02-05 17:02 ` Zach Brown
2007-02-05 18:52 ` Davide Libenzi
2007-02-05 19:20 ` Zach Brown
2007-02-05 19:38 ` Davide Libenzi
2007-02-04 5:12 ` Davide Libenzi
2007-02-05 17:54 ` Zach Brown
2007-01-30 20:39 ` [PATCH 3 of 4] Teach paths to wake a specific void * target instead of a whole task_struct Zach Brown
2007-01-30 20:39 ` [PATCH 4 of 4] Introduce aio system call submission and completion system calls Zach Brown
2007-01-31 8:58 ` Andi Kleen
2007-01-31 17:15 ` Zach Brown
2007-01-31 17:21 ` Andi Kleen
2007-01-31 19:23 ` Zach Brown
2007-02-01 11:13 ` Suparna Bhattacharya
2007-02-01 19:50 ` Trond Myklebust
2007-02-02 7:19 ` Suparna Bhattacharya
2007-02-02 7:45 ` Andi Kleen
2007-02-01 22:18 ` Zach Brown
2007-02-02 3:35 ` Suparna Bhattacharya
2007-02-01 20:26 ` bert hubert
2007-02-01 21:29 ` Zach Brown
2007-02-02 7:12 ` bert hubert
2007-02-04 5:12 ` Davide Libenzi
2007-01-30 21:58 ` [PATCH 0 of 4] Generic AIO by scheduling stacks Linus Torvalds
2007-01-30 22:23 ` Linus Torvalds
2007-01-30 22:53 ` Zach Brown
2007-01-30 22:40 ` Zach Brown
2007-01-30 22:53 ` Linus Torvalds
2007-01-30 23:45 ` Zach Brown
2007-01-31 2:07 ` Benjamin Herrenschmidt
2007-01-31 2:04 ` Benjamin Herrenschmidt
2007-01-31 2:46 ` Linus Torvalds
2007-01-31 3:02 ` Linus Torvalds
2007-01-31 10:50 ` Xavier Bestel
2007-01-31 19:28 ` Zach Brown
2007-01-31 17:59 ` Zach Brown
2007-01-31 5:16 ` Benjamin Herrenschmidt
2007-01-31 5:36 ` Nick Piggin
2007-01-31 5:51 ` Nick Piggin
2007-01-31 6:06 ` Linus Torvalds
2007-01-31 8:43 ` Ingo Molnar
2007-01-31 20:13 ` Joel Becker
2007-01-31 18:20 ` Zach Brown
2007-01-31 17:47 ` Zach Brown
2007-01-31 17:38 ` Zach Brown
2007-01-31 17:51 ` Benjamin LaHaise
2007-01-31 19:25 ` Zach Brown
2007-01-31 20:05 ` Benjamin LaHaise
2007-01-31 20:41 ` Zach Brown
2007-02-04 5:13 ` Davide Libenzi
2007-02-04 20:00 ` Davide Libenzi
2007-02-09 22:33 ` Linus Torvalds
2007-02-09 23:11 ` Davide Libenzi
2007-02-09 23:35 ` Linus Torvalds
2007-02-10 18:45 ` Davide Libenzi
2007-02-10 19:01 ` Linus Torvalds
2007-02-10 19:35 ` Linus Torvalds
2007-02-10 20:59 ` Davide Libenzi
2007-02-10 0:04 ` Eric Dumazet
2007-02-10 0:12 ` Linus Torvalds
2007-02-10 0:34 ` Alan
2007-02-10 10:47 ` bert hubert
2007-02-10 18:19 ` Davide Libenzi
2007-02-11 0:56 ` David Miller
2007-02-11 2:49 ` Linus Torvalds
2007-02-14 16:42 ` James Antill
2007-02-03 14:05 [PATCH 2 of 4] Introduce i386 fibril scheduling linux
2007-02-06 13:43 Al Boldi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20070202104900.GA13941@elte.hu \
--to=mingo@elte.hu \
--cc=bcrl@kvack.org \
--cc=linux-aio@kvack.org \
--cc=linux-kernel@vger.kernel.org \
--cc=suparna@in.ibm.com \
--cc=torvalds@linux-foundation.org \
--cc=zach.brown@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.