From: Peter Zijlstra <peterz@infradead.org>
To: Roman Penyaev <rpenyaev@suse.de>
Cc: azat@libevent.org, rpenyaev@suse.de, akpm@linux-foundation.org,
viro@zeniv.linux.org.uk, torvalds@linux-foundation.org,
linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3 00/13] epoll: support pollable epoll from userspace
Date: Fri, 31 May 2019 18:33:12 +0200 [thread overview]
Message-ID: <20190531163312.GW2650@hirez.programming.kicks-ass.net> (raw)
In-Reply-To: <20190516085810.31077-1-rpenyaev@suse.de>
On Thu, May 16, 2019 at 10:57:57AM +0200, Roman Penyaev wrote:
> When new event comes for some epoll item kernel does the following:
>
> struct epoll_uitem *uitem;
>
> /* Each item has a bit (index in user items array), discussed later */
> uitem = user_header->items[epi->bit];
>
> if (!atomic_fetch_or(uitem->ready_events, pollflags)) {
> i = atomic_add(&ep->user_header->tail, 1);
So this is where you increment tail
>
> item_idx = &user_index[i & index_mask];
>
> /* Signal with a bit, user spins on index expecting value > 0 */
> *item_idx = idx + 1;
IUC, this is where you write the idx into shared memory, which is
_after_ tail has already been incremented.
> }
>
> Important thing here is that ring can't infinitely grow and corrupt other
> elements, because kernel always checks that item was marked as ready, so
> userspace has to clear ready_events field.
>
> On userside events the following code should be used in order to consume
> events:
>
> tail = READ_ONCE(header->tail);
> for (i = 0; header->head != tail; header->head++) {
> item_idx_ptr = &index[idx & indeces_mask];
>
> /*
> * Spin here till we see valid index
> */
> while (!(idx = __atomic_load_n(item_idx_ptr, __ATOMIC_ACQUIRE)))
> ;
Which you then try and fix up by busy waiting for @idx to become !0 ?!
Why not write the idx first, then increment the ->tail, such that when
we see ->tail, we already know idx must be correct?
>
> item = &header->items[idx - 1];
>
> /*
> * Mark index as invalid, that is for userspace only, kernel does not care
> * and will refill this pointer only when observes that event is cleared,
> * which happens below.
> */
> *item_idx_ptr = 0;
That avoids this store too.
>
> /*
> * Fetch data first, if event is cleared by the kernel we drop the data
> * returning false.
> */
> event->data = item->event.data;
> event->events = __atomic_exchange_n(&item->ready_events, 0,
> __ATOMIC_RELEASE);
>
> }
Aside from that, you have to READ/WRITE_ONCE() on ->head, to avoid
load/store tearing.
That would give something like:
kernel:
slot = atomic_fetch_inc(&ep->slot);
item_idx = &user_index[slot & idx_mask];
WRITE_ONCE(*item_idx, idx);
smp_store_release(&ep->user_header->tail, slot);
userspace:
tail = smp_load_acquire(&header->tail);
for (head = READ_ONCE(header->head); head != tail; head++) {
idx = READ_ONCE(index[head & idx_mask]);
itemp = &header->items[idx];
...
}
smp_store_release(&header->head, head);
next prev parent reply other threads:[~2019-05-31 16:33 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-16 8:57 [PATCH v3 00/13] epoll: support pollable epoll from userspace Roman Penyaev
2019-05-16 8:57 ` [PATCH v3 01/13] epoll: move private helpers from a header to the source Roman Penyaev
2019-05-16 8:57 ` [PATCH v3 02/13] epoll: introduce user structures for polling from userspace Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 03/13] epoll: allocate user header and user events ring " Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 04/13] epoll: some sanity flags checks for epoll syscalls " Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 05/13] epoll: offload polling to a work in case of epfd polled " Roman Penyaev
2019-05-21 7:51 ` Eric Wong
2019-05-22 12:54 ` Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 06/13] epoll: introduce helpers for adding/removing events to uring Roman Penyaev
2019-05-31 9:55 ` Peter Zijlstra
2019-05-31 11:24 ` Roman Penyaev
2019-05-31 13:11 ` Peter Zijlstra
2019-05-31 9:56 ` Peter Zijlstra
2019-05-31 11:15 ` Roman Penyaev
2019-05-31 12:53 ` Peter Zijlstra
2019-05-31 14:28 ` Roman Penyaev
2019-05-31 16:53 ` Peter Zijlstra
2019-05-31 12:56 ` Peter Zijlstra
2019-05-31 14:21 ` Roman Penyaev
2019-05-31 16:51 ` Peter Zijlstra
2019-05-31 18:58 ` Roman Penyaev
2019-06-03 9:09 ` Peter Zijlstra
2019-06-03 10:02 ` Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 07/13] epoll: call ep_add_event_to_uring() from ep_poll_callback() Roman Penyaev
2019-05-31 9:56 ` Peter Zijlstra
2019-05-31 11:22 ` Roman Penyaev
2019-05-31 13:05 ` Peter Zijlstra
2019-05-31 15:05 ` Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 08/13] epoll: support polling from userspace for ep_insert() Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 09/13] epoll: support polling from userspace for ep_remove() Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 10/13] epoll: support polling from userspace for ep_modify() Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 11/13] epoll: support polling from userspace for ep_poll() Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 12/13] epoll: support mapping for epfd when polled from userspace Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 13/13] epoll: implement epoll_create2() syscall Roman Penyaev
2019-05-16 10:03 ` Arnd Bergmann
2019-05-16 10:20 ` Roman Penyaev
2019-05-16 10:57 ` Arnd Bergmann
2019-05-22 2:33 ` Andrew Morton
2019-05-22 9:11 ` Roman Penyaev
2019-05-22 11:14 ` Arnd Bergmann
2019-05-22 18:36 ` Andrew Morton
2019-05-31 9:55 ` [PATCH v3 00/13] epoll: support pollable epoll from userspace Peter Zijlstra
2019-05-31 14:48 ` Jens Axboe
2019-05-31 16:02 ` Roman Penyaev
2019-05-31 16:54 ` Jens Axboe
2019-05-31 19:45 ` Roman Penyaev
2019-05-31 21:09 ` Jens Axboe
2019-06-05 6:17 ` Roman Penyaev
2019-05-31 16:33 ` Peter Zijlstra [this message]
2019-05-31 18:50 ` Roman Penyaev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190531163312.GW2650@hirez.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=akpm@linux-foundation.org \
--cc=azat@libevent.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=rpenyaev@suse.de \
--cc=torvalds@linux-foundation.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).