linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Roman Penyaev <rpenyaev@suse.de>
To: unlisted-recipients:; (no To-header on input)
Cc: Roman Penyaev <rpenyaev@suse.de>,
	Andrew Morton <akpm@linux-foundation.org>,
	Al Viro <viro@zeniv.linux.org.uk>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH v5 07/14] epoll: call ep_add_event_to_uring() from ep_poll_callback()
Date: Mon, 24 Jun 2019 16:41:44 +0200	[thread overview]
Message-ID: <20190624144151.22688-8-rpenyaev@suse.de> (raw)
In-Reply-To: <20190624144151.22688-1-rpenyaev@suse.de>

Each ep_poll_callback() is called when fd calls wakeup() on epfd.
So account new event in user ring.

The tricky part here is EPOLLONESHOT.  Since we are lockless we
have to be deal with ep_poll_callbacks() called in paralle, thus
use cmpxchg to clear public event bits and filter out concurrent
call from another cpu.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
---
 fs/eventpoll.c | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index 3b1f6a210247..cc4612e28e03 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -1565,6 +1565,29 @@ struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd,
 }
 #endif /* CONFIG_CHECKPOINT_RESTORE */
 
+/**
+ * Atomically clear public event bits and return %true if the old value has
+ * public event bits set.
+ */
+static inline bool ep_clear_public_event_bits(struct epitem *epi)
+{
+	__poll_t old, flags;
+
+	/*
+	 * Here we race with ourselves and with ep_modify(), which can
+	 * change the event bits.  In order not to override events updated
+	 * by ep_modify() we have to do cmpxchg.
+	 */
+
+	old = READ_ONCE(epi->event.events);
+	do {
+		flags = old;
+	} while ((old = cmpxchg(&epi->event.events, flags,
+				flags & EP_PRIVATE_BITS)) != flags);
+
+	return flags & ~EP_PRIVATE_BITS;
+}
+
 /**
  * Adds a new entry to the tail of the list in a lockless way, i.e.
  * multiple CPUs are allowed to call this function concurrently.
@@ -1684,6 +1707,20 @@ static int ep_poll_callback(struct epitem *epi, __poll_t pollflags)
 	if (pollflags && !(pollflags & epi->event.events))
 		goto out_unlock;
 
+	if (ep_polled_by_user(ep)) {
+		/*
+		 * For polled descriptor from user we have to disable events on
+		 * callback path in case of one-shot.
+		 */
+		if ((epi->event.events & EPOLLONESHOT) &&
+		    !ep_clear_public_event_bits(epi))
+			/* Race is lost, another callback has cleared events */
+			goto out_unlock;
+
+		ep_add_event_to_uring(epi, pollflags);
+		goto wakeup;
+	}
+
 	/*
 	 * If we are transferring events to userspace, we can hold no locks
 	 * (because we're accessing user memory, and because of linux f_op->poll()
@@ -1703,6 +1740,7 @@ static int ep_poll_callback(struct epitem *epi, __poll_t pollflags)
 		ep_pm_stay_awake_rcu(epi);
 	}
 
+wakeup:
 	/*
 	 * Wake up ( if active ) both the eventpoll wait list and the ->poll()
 	 * wait list.
-- 
2.21.0


  parent reply	other threads:[~2019-06-24 14:42 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-24 14:41 [PATCH v5 00/14] epoll: support pollable epoll from userspace Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 01/14] epoll: move private helpers from a header to the source Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 02/14] epoll: introduce user structures for polling from userspace Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 03/14] epoll: allocate user header and user events ring " Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 04/14] epoll: some sanity flags checks for epoll syscalls " Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 05/14] epoll: offload polling to a work in case of epfd polled " Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 06/14] epoll: introduce helpers for adding/removing events to uring Roman Penyaev
2019-06-24 14:41 ` Roman Penyaev [this message]
2019-06-24 14:41 ` [PATCH v5 08/14] epoll: support polling from userspace for ep_insert() Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 09/14] epoll: support polling from userspace for ep_remove() Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 10/14] epoll: support polling from userspace for ep_modify() Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 11/14] epoll: support polling from userspace for ep_poll() Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 12/14] epoll: support mapping for epfd when polled from userspace Roman Penyaev
2019-06-24 14:41 ` [PATCH v5 13/14] epoll: implement epoll_create2() syscall Roman Penyaev
2019-06-24 16:14   ` Arnd Bergmann
2019-06-24 16:25     ` Roman Penyaev
2019-06-25 18:29   ` Eric Wong
2019-06-24 14:41 ` [PATCH v5 14/14] kselftest: add uepoll-test which tests polling from userspace Roman Penyaev
2019-06-24 20:38 ` [PATCH v5 00/14] epoll: support pollable epoll " Linus Torvalds
2019-06-25 11:19   ` Roman Penyaev
2019-06-25  0:24 ` Eric Wong
2019-06-25 11:07   ` Roman Penyaev

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190624144151.22688-8-rpenyaev@suse.de \
    --to=rpenyaev@suse.de \
    --cc=akpm@linux-foundation.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=torvalds@linux-foundation.org \
    --cc=viro@zeniv.linux.org.uk \
    --subject='Re: [PATCH v5 07/14] epoll: call ep_add_event_to_uring() from ep_poll_callback()' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).