From: Roman Penyaev <rpenyaev@suse.de>
To: unlisted-recipients:; (no To-header on input)
Cc: Azat Khuzhin <azat@libevent.org>,
Roman Penyaev <rpenyaev@suse.de>,
Andrew Morton <akpm@linux-foundation.org>,
Al Viro <viro@zeniv.linux.org.uk>,
Linus Torvalds <torvalds@linux-foundation.org>,
linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH v3 08/13] epoll: support polling from userspace for ep_insert()
Date: Thu, 16 May 2019 10:58:05 +0200 [thread overview]
Message-ID: <20190516085810.31077-9-rpenyaev@suse.de> (raw)
In-Reply-To: <20190516085810.31077-1-rpenyaev@suse.de>
When epfd is polled by userspace and new item is inserted new bit
should be get from a bitmap and then user item is set accordingly.
Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index 55612da9651e..f1ffccfcca67 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -865,6 +865,23 @@ static void epi_rcu_free(struct rcu_head *head)
kmem_cache_free(epi_cache, epi);
}
+static inline int ep_get_bit(struct eventpoll *ep)
+{
+ bool was_set;
+ int bit;
+
+ lockdep_assert_held(&ep->mtx);
+
+ bit = find_first_zero_bit(ep->items_bm, ep->max_items_nr);
+ if (bit >= ep->max_items_nr)
+ return -ENOSPC;
+
+ was_set = test_and_set_bit(bit, ep->items_bm);
+ WARN_ON(was_set);
+
+ return bit;
+}
+
#define atomic_set_unless_zero(ptr, flags) \
({ \
typeof(ptr) _ptr = (ptr); \
@@ -1875,6 +1892,7 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event,
struct epitem *epi;
struct ep_pqueue epq;
+ lockdep_assert_held(&ep->mtx);
lockdep_assert_irqs_enabled();
user_watches = atomic_long_read(&ep->user->epoll_watches);
@@ -1901,6 +1919,29 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event,
RCU_INIT_POINTER(epi->ws, NULL);
}
+ if (ep_polled_by_user(ep)) {
+ struct epoll_uitem *uitem;
+ int bit;
+
+ bit = ep_get_bit(ep);
+ if (unlikely(bit < 0)) {
+ error = bit;
+ goto error_get_bit;
+ }
+ epi->bit = bit;
+
+ /*
+ * Now fill-in user item. Do not touch ready_events, since
+ * it can be EPOLLREMOVED (has been set by previous user
+ * item), thus user index entry can be not yet consumed
+ * by userspace. See ep_remove_user_item() and
+ * ep_add_event_to_uring() for details.
+ */
+ uitem = &ep->user_header->items[epi->bit];
+ uitem->events = event->events;
+ uitem->data = event->data;
+ }
+
/* Initialize the poll table using the queue callback */
epq.epi = epi;
init_poll_funcptr(&epq.pt, ep_ptable_queue_proc);
@@ -1945,16 +1986,23 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event,
/* record NAPI ID of new item if present */
ep_set_busy_poll_napi_id(epi);
- /* If the file is already "ready" we drop it inside the ready list */
- if (revents && !ep_is_linked(epi)) {
- list_add_tail(&epi->rdllink, &ep->rdllist);
- ep_pm_stay_awake(epi);
+ if (revents) {
+ bool added = false;
- /* Notify waiting tasks that events are available */
- if (waitqueue_active(&ep->wq))
- wake_up(&ep->wq);
- if (waitqueue_active(&ep->poll_wait))
- pwake++;
+ if (ep_polled_by_user(ep)) {
+ added = ep_add_event_to_uring(epi, revents);
+ } else if (!ep_is_linked(epi)) {
+ list_add_tail(&epi->rdllink, &ep->rdllist);
+ ep_pm_stay_awake(epi);
+ added = true;
+ }
+ if (added) {
+ /* Notify waiting tasks that events are available */
+ if (waitqueue_active(&ep->wq))
+ wake_up(&ep->wq);
+ if (waitqueue_active(&ep->poll_wait))
+ pwake++;
+ }
}
write_unlock_irq(&ep->lock);
@@ -1983,11 +2031,16 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event,
* list, since that is used/cleaned only inside a section bound by "mtx".
* And ep_insert() is called with "mtx" held.
*/
- write_lock_irq(&ep->lock);
- if (ep_is_linked(epi))
- list_del_init(&epi->rdllink);
- write_unlock_irq(&ep->lock);
+ if (ep_polled_by_user(ep)) {
+ ep_remove_user_item(epi);
+ } else {
+ write_lock_irq(&ep->lock);
+ if (ep_is_linked(epi))
+ list_del_init(&epi->rdllink);
+ write_unlock_irq(&ep->lock);
+ }
+error_get_bit:
wakeup_source_unregister(ep_wakeup_source(epi));
error_create_wakeup_source:
--
2.21.0
next prev parent reply other threads:[~2019-05-16 8:58 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-16 8:57 [PATCH v3 00/13] epoll: support pollable epoll from userspace Roman Penyaev
2019-05-16 8:57 ` [PATCH v3 01/13] epoll: move private helpers from a header to the source Roman Penyaev
2019-05-16 8:57 ` [PATCH v3 02/13] epoll: introduce user structures for polling from userspace Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 03/13] epoll: allocate user header and user events ring " Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 04/13] epoll: some sanity flags checks for epoll syscalls " Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 05/13] epoll: offload polling to a work in case of epfd polled " Roman Penyaev
2019-05-21 7:51 ` Eric Wong
2019-05-22 12:54 ` Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 06/13] epoll: introduce helpers for adding/removing events to uring Roman Penyaev
2019-05-31 9:55 ` Peter Zijlstra
2019-05-31 11:24 ` Roman Penyaev
2019-05-31 13:11 ` Peter Zijlstra
2019-05-31 9:56 ` Peter Zijlstra
2019-05-31 11:15 ` Roman Penyaev
2019-05-31 12:53 ` Peter Zijlstra
2019-05-31 14:28 ` Roman Penyaev
2019-05-31 16:53 ` Peter Zijlstra
2019-05-31 12:56 ` Peter Zijlstra
2019-05-31 14:21 ` Roman Penyaev
2019-05-31 16:51 ` Peter Zijlstra
2019-05-31 18:58 ` Roman Penyaev
2019-06-03 9:09 ` Peter Zijlstra
2019-06-03 10:02 ` Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 07/13] epoll: call ep_add_event_to_uring() from ep_poll_callback() Roman Penyaev
2019-05-31 9:56 ` Peter Zijlstra
2019-05-31 11:22 ` Roman Penyaev
2019-05-31 13:05 ` Peter Zijlstra
2019-05-31 15:05 ` Roman Penyaev
2019-05-16 8:58 ` Roman Penyaev [this message]
2019-05-16 8:58 ` [PATCH v3 09/13] epoll: support polling from userspace for ep_remove() Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 10/13] epoll: support polling from userspace for ep_modify() Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 11/13] epoll: support polling from userspace for ep_poll() Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 12/13] epoll: support mapping for epfd when polled from userspace Roman Penyaev
2019-05-16 8:58 ` [PATCH v3 13/13] epoll: implement epoll_create2() syscall Roman Penyaev
2019-05-16 10:03 ` Arnd Bergmann
2019-05-16 10:20 ` Roman Penyaev
2019-05-16 10:57 ` Arnd Bergmann
2019-05-22 2:33 ` Andrew Morton
2019-05-22 9:11 ` Roman Penyaev
2019-05-22 11:14 ` Arnd Bergmann
2019-05-22 18:36 ` Andrew Morton
2019-05-31 9:55 ` [PATCH v3 00/13] epoll: support pollable epoll from userspace Peter Zijlstra
2019-05-31 14:48 ` Jens Axboe
2019-05-31 16:02 ` Roman Penyaev
2019-05-31 16:54 ` Jens Axboe
2019-05-31 19:45 ` Roman Penyaev
2019-05-31 21:09 ` Jens Axboe
2019-06-05 6:17 ` Roman Penyaev
2019-05-31 16:33 ` Peter Zijlstra
2019-05-31 18:50 ` Roman Penyaev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190516085810.31077-9-rpenyaev@suse.de \
--to=rpenyaev@suse.de \
--cc=akpm@linux-foundation.org \
--cc=azat@libevent.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).