From: Roman Penyaev <rpenyaev@suse.de>
To: unlisted-recipients:; (no To-header on input)
Cc: Roman Penyaev <rpenyaev@suse.de>,
Andrew Morton <akpm@linux-foundation.org>,
Al Viro <viro@zeniv.linux.org.uk>,
Linus Torvalds <torvalds@linux-foundation.org>,
linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH v4 11/14] epoll: support polling from userspace for ep_poll()
Date: Tue, 11 Jun 2019 16:54:55 +0200 [thread overview]
Message-ID: <20190611145458.9540-12-rpenyaev@suse.de> (raw)
In-Reply-To: <20190611145458.9540-1-rpenyaev@suse.de>
Rule of thumb for epfd polled from userspace is simple: epfd has
events if ->head != ->tail, no traversing of each item is performed.
Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
fs/eventpoll.c | 69 ++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 59 insertions(+), 10 deletions(-)
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index 9f0d48eb360e..e42ddf580556 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -460,6 +460,13 @@ static inline bool ep_polled_by_user(struct eventpoll *ep)
return !!ep->user_header;
}
+static inline bool ep_uring_events_available(struct eventpoll *ep)
+{
+ return ep_polled_by_user(ep) &&
+ READ_ONCE(ep->user_header->head) !=
+ READ_ONCE(ep->user_header->tail);
+}
+
/**
* ep_events_available - Checks if ready events might be available.
*
@@ -471,7 +478,8 @@ static inline bool ep_polled_by_user(struct eventpoll *ep)
static inline int ep_events_available(struct eventpoll *ep)
{
return !list_empty_careful(&ep->rdllist) ||
- READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR;
+ READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR ||
+ ep_uring_events_available(ep);
}
#ifdef CONFIG_NET_RX_BUSY_POLL
@@ -1309,7 +1317,7 @@ static void ep_ptable_queue_proc(struct file *file, wait_queue_head_t *whead,
static __poll_t ep_item_poll(const struct epitem *epi, poll_table *pt,
int depth)
{
- struct eventpoll *ep;
+ struct eventpoll *ep, *tep;
bool locked;
pt->_key = epi->event.events;
@@ -1318,6 +1326,26 @@ static __poll_t ep_item_poll(const struct epitem *epi, poll_table *pt,
ep = epi->ffd.file->private_data;
poll_wait(epi->ffd.file, &ep->poll_wait, pt);
+
+ tep = epi->ffd.file->private_data;
+ if (ep_polled_by_user(tep)) {
+ /*
+ * The behaviour differs comparing to full scan of ready
+ * list for original epoll. If descriptor is pollable
+ * from userspace we don't do scan of all ready user items:
+ * firstly because we can't do reverse search of epi by
+ * uitem bit, secondly this is simply waste of time for
+ * edge triggered descriptors (user code should be prepared
+ * to deal with EAGAIN returned from read() or write() on
+ * inserted file descriptor) and thirdly once event is put
+ * into user index ring do not touch it from kernel, what
+ * we do is mark it as EPOLLREMOVED on ep_remove() and
+ * that's it.
+ */
+ return ep_uring_events_available(tep) ?
+ EPOLLIN | EPOLLRDNORM : 0;
+ }
+
locked = pt && (pt->_qproc == ep_ptable_queue_proc);
return ep_scan_ready_list(epi->ffd.file->private_data,
@@ -1360,6 +1388,12 @@ static __poll_t ep_eventpoll_poll(struct file *file, poll_table *wait)
/* Insert inside our poll wait queue */
poll_wait(file, &ep->poll_wait, wait);
+ if (ep_polled_by_user(ep)) {
+ /* Please read detailed comments inside ep_item_poll() */
+ return ep_uring_events_available(ep) ?
+ EPOLLIN | EPOLLRDNORM : 0;
+ }
+
/*
* Proceed to find out if wanted events are really available inside
* the ready list.
@@ -2419,6 +2453,8 @@ static int ep_send_events(struct eventpoll *ep,
{
struct ep_send_events_data esed;
+ WARN_ON(ep_polled_by_user(ep));
+
esed.maxevents = maxevents;
esed.events = events;
@@ -2465,6 +2501,12 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
lockdep_assert_irqs_enabled();
+ if (ep_polled_by_user(ep)) {
+ if (ep_uring_events_available(ep))
+ /* Firstly all events from ring have to be consumed */
+ return -ESTALE;
+ }
+
if (timeout > 0) {
struct timespec64 end_time = ep_set_mstimeout(timeout);
@@ -2553,14 +2595,21 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
__set_current_state(TASK_RUNNING);
send_events:
- /*
- * Try to transfer events to user space. In case we get 0 events and
- * there's still timeout left over, we go trying again in search of
- * more luck.
- */
- if (!res && eavail &&
- !(res = ep_send_events(ep, events, maxevents)) && !timed_out)
- goto fetch_events;
+ if (!res && eavail) {
+ if (!ep_polled_by_user(ep)) {
+ /*
+ * Try to transfer events to user space. In case we get
+ * 0 events and there's still timeout left over, we go
+ * trying again in search of more luck.
+ */
+ res = ep_send_events(ep, events, maxevents);
+ if (!res && !timed_out)
+ goto fetch_events;
+ } else {
+ /* User has to deal with the ring himself */
+ res = -ESTALE;
+ }
+ }
if (waiter) {
spin_lock_irq(&ep->wq.lock);
--
2.21.0
next prev parent reply other threads:[~2019-06-11 14:55 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-11 14:54 [PATCH v4 00/14] epoll: support pollable epoll from userspace Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 01/14] epoll: move private helpers from a header to the source Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 02/14] epoll: introduce user structures for polling from userspace Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 03/14] epoll: allocate user header and user events ring " Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 04/14] epoll: some sanity flags checks for epoll syscalls " Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 05/14] epoll: offload polling to a work in case of epfd polled " Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 06/14] epoll: introduce helpers for adding/removing events to uring Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 07/14] epoll: call ep_add_event_to_uring() from ep_poll_callback() Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 08/14] epoll: support polling from userspace for ep_insert() Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 09/14] epoll: support polling from userspace for ep_remove() Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 10/14] epoll: support polling from userspace for ep_modify() Roman Penyaev
2019-06-11 14:54 ` Roman Penyaev [this message]
2019-06-11 14:54 ` [PATCH v4 12/14] epoll: support mapping for epfd when polled from userspace Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 13/14] epoll: implement epoll_create2() syscall Roman Penyaev
2019-06-11 14:54 ` [PATCH v4 14/14] kselftest: add uepoll-test which tests polling from userspace Roman Penyaev
2019-07-26 23:22 ` [PATCH v4 00/14] epoll: support pollable epoll " Andrew Morton
2019-07-27 17:16 ` Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190611145458.9540-12-rpenyaev@suse.de \
--to=rpenyaev@suse.de \
--cc=akpm@linux-foundation.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).