From: Zach Brown <zab@redhat.com>
To: Kent Overstreet <koverstreet@google.com>
Cc: linux-bcache@vger.kernel.org, linux-kernel@vger.kernel.org,
dm-devel@redhat.com, tytso@mit.edu
Subject: Re: [PATCH 5/5] aio: Refactor aio_read_evt, use cmxchg(), fix bug
Date: Tue, 9 Oct 2012 15:47:03 -0700 [thread overview]
Message-ID: <20121009224703.GT26187@lenny.home.zabbo.net> (raw)
In-Reply-To: <20121009212724.GD29494@google.com>
> If libaio is the only thing in userspace looking at the ringbuffer, and
> if I'm looking at the latest libaio code this shouldn't break
> anything...
We can't assume that libaio is the only thing in userspace using the
mapped buffer -- as scary a thought as that is :).
If we wanted to change the behaviour of the ring we'd give userspace a
way to indicate that it accepts the new semantics. I'm not immediately
sure how best to do that -- haven't given it much thought.
> Sticking the head and tail pointers on the same cacheline is one of my
> main complaints. If you could find that comment I'd be interested in
> reading it, though.
Yeah, that was on my list.. Heh, found it!
/*
* This structure defines the head of a ring of events that is mapped into user
* space. It isn't used anymore because it has a number of design flaws. I'll
* mention them here in the hopes that people read this when considering a
* design for a ring shared between user and kernel space.
*
* - The head and tail pointers are in the same cacheline. This introduces
* false sharing between producers and consumers which could otherwise operate
* independent of one-another, presuming that the producers know ahead of time
* that room has been reserved for them.
*
* - The head and tail semantics are unconventional. They're always less than
* the number of events in the ring and their meaning is reversed from the
* usual construct that one sees in, for example, ethernet hardware where the
* ring is a power of two and the indexes wrap but are masked when used to
* dereference elements.
*
* - Because of the weird head and tail semantics one can't distinguish from
* completely empty and full rings. To work around this the ring is actually
* created with more events than the aio user asked for and that number is
* accounted for separately. The number of free elements in the ring is
* greater than the number of operations that the kernel aio submission will
* allow.
*
* - Synchronizing head and tail updates between the kernel and user space
* requires a u32 cmpxchg. futexes have infrastructure for this which requires
* working with user addresses. Trying to nicely re-use the futex code leads
* to awkward code in aio which sometimes wants to work through kmap()
* addresses and other times want to work through the mmaped user space
* pointers.
*
* - The head and tail pointers are restricted in value to being less than the
* number of elements in the ring. This means that a cmpxchg can see a index
* which has wrapped and confuse it for the index that it read before trying
* cmpxchg. This can end up sending a duplicated previous event instead of a
* currently queued event in the worst case.
*
* - User space doesn't explicitly indicate that it wants to work with the ring
* from user space. Even if user space never touches the ring the kernel still
* has pay the cost of the potential sharing with user space when inserting
* events.
*
* - The kernel magically maps it into the address space. Users can't put it
* in whatever memory they like, like shared mem or hugetlb pages or tmpfs
* pages that they want to sendfile out of.
*
* - The ring is allocated out of unswappable kernel memory. It would be
* better to have the ring allocated in userspace and given to the kernel. In
* the common case the pages will be present and the overhead is minimal and
* complexity is kept out of the kernel. In the worst case of memory pressure
* the kernel has fewer pinned pages tying its hands.
*/
The submission and completion interface that I threw together for the
acall experiments took all this into account:
http://lwn.net/Articles/316806/
Man, this all brings back memories :).
- z
next prev parent reply other threads:[~2012-10-09 22:47 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-10-09 6:39 [PATCH 1/5] aio: Kill return value of aio_complete() Kent Overstreet
2012-10-09 6:39 ` [PATCH 2/5] aio: kiocb_cancel() Kent Overstreet
2012-10-09 18:26 ` Zach Brown
2012-10-09 21:37 ` Kent Overstreet
2012-10-10 11:03 ` Theodore Ts'o
2012-10-10 21:20 ` Zach Brown
2012-10-10 23:21 ` Theodore Ts'o
2012-10-11 2:41 ` Kent Overstreet
2012-10-09 6:39 ` [PATCH 3/5] aio: Rewrite refcounting Kent Overstreet
2012-10-09 18:27 ` Zach Brown
2012-10-09 22:21 ` Kent Overstreet
2012-10-09 22:35 ` Zach Brown
2012-10-10 0:17 ` Kent Overstreet
2012-10-09 6:39 ` [PATCH 4/5] aio: vmap ringbuffer Kent Overstreet
2012-10-09 18:29 ` Zach Brown
2012-10-09 21:31 ` Kent Overstreet
2012-10-09 22:32 ` Zach Brown
2012-10-09 22:44 ` Kent Overstreet
2012-10-09 22:58 ` Zach Brown
2012-10-10 0:16 ` Kent Overstreet
2012-10-10 0:36 ` Zach Brown
2012-10-10 1:09 ` Kent Overstreet
2012-10-09 6:39 ` [PATCH 5/5] aio: Refactor aio_read_evt, use cmxchg(), fix bug Kent Overstreet
2012-10-09 18:37 ` Zach Brown
2012-10-09 21:27 ` Kent Overstreet
2012-10-09 22:47 ` Zach Brown [this message]
2012-10-09 22:55 ` Kent Overstreet
2012-10-09 23:10 ` Zach Brown
2012-10-10 0:06 ` Kent Overstreet
2012-10-10 0:26 ` Zach Brown
2012-10-10 0:47 ` Kent Overstreet
2012-10-10 21:43 ` Zach Brown
2012-10-11 2:51 ` Kent Overstreet
2012-10-11 16:43 ` Zach Brown
2012-10-09 18:25 ` [PATCH 1/5] aio: Kill return value of aio_complete() Zach Brown
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121009224703.GT26187@lenny.home.zabbo.net \
--to=zab@redhat.com \
--cc=dm-devel@redhat.com \
--cc=koverstreet@google.com \
--cc=linux-bcache@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).