LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Roman Penyaev <rpenyaev@suse.de>
To: unlisted-recipients:; (no To-header on input)
Cc: Azat Khuzhin <azat@libevent.org>,
	Roman Penyaev <rpenyaev@suse.de>,
	Andrew Morton <akpm@linux-foundation.org>,
	Al Viro <viro@zeniv.linux.org.uk>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH v3 07/13] epoll: call ep_add_event_to_uring() from ep_poll_callback()
Date: Thu, 16 May 2019 10:58:04 +0200
Message-ID: <20190516085810.31077-8-rpenyaev@suse.de> (raw)
In-Reply-To: <20190516085810.31077-1-rpenyaev@suse.de>

Each ep_poll_callback() is called when fd calls wakeup() on epfd.
So account new event in user ring.

The tricky part here is EPOLLONESHOT.  Since we are lockless we
have to be deal with ep_poll_callbacks() called in paralle, thus
use cmpxchg to clear public event bits and filter out concurrent
call from another cpu.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: linux-fsdevel@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index 2f551c005640..55612da9651e 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -1407,6 +1407,29 @@ struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd,
 }
 #endif /* CONFIG_CHECKPOINT_RESTORE */
 
+/**
+ * Atomically clear public event bits and return %true if the old value has
+ * public event bits set.
+ */
+static inline bool ep_clear_public_event_bits(struct epitem *epi)
+{
+	__poll_t old, flags;
+
+	/*
+	 * Here we race with ourselves and with ep_modify(), which can
+	 * change the event bits.  In order not to override events updated
+	 * by ep_modify() we have to do cmpxchg.
+	 */
+
+	old = epi->event.events;
+	do {
+		flags = old;
+	} while ((old = cmpxchg(&epi->event.events, flags,
+				flags & EP_PRIVATE_BITS)) != flags);
+
+	return flags & ~EP_PRIVATE_BITS;
+}
+
 /**
  * Adds a new entry to the tail of the list in a lockless way, i.e.
  * multiple CPUs are allowed to call this function concurrently.
@@ -1526,6 +1549,20 @@ static int ep_poll_callback(struct epitem *epi, __poll_t pollflags)
 	if (pollflags && !(pollflags & epi->event.events))
 		goto out_unlock;
 
+	if (ep_polled_by_user(ep)) {
+		/*
+		 * For polled descriptor from user we have to disable events on
+		 * callback path in case of one-shot.
+		 */
+		if ((epi->event.events & EPOLLONESHOT) &&
+		    !ep_clear_public_event_bits(epi))
+			/* Race is lost, another callback has cleared events */
+			goto out_unlock;
+
+		ep_add_event_to_uring(epi, pollflags);
+		goto wakeup;
+	}
+
 	/*
 	 * If we are transferring events to userspace, we can hold no locks
 	 * (because we're accessing user memory, and because of linux f_op->poll()
@@ -1545,6 +1582,7 @@ static int ep_poll_callback(struct epitem *epi, __poll_t pollflags)
 		ep_pm_stay_awake_rcu(epi);
 	}
 
+wakeup:
 	/*
 	 * Wake up ( if active ) both the eventpoll wait list and the ->poll()
 	 * wait list.
-- 
2.21.0


  parent reply index

Thread overview: 50+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-05-16  8:57 [PATCH v3 00/13] epoll: support pollable epoll from userspace Roman Penyaev
2019-05-16  8:57 ` [PATCH v3 01/13] epoll: move private helpers from a header to the source Roman Penyaev
2019-05-16  8:57 ` [PATCH v3 02/13] epoll: introduce user structures for polling from userspace Roman Penyaev
2019-05-16  8:58 ` [PATCH v3 03/13] epoll: allocate user header and user events ring " Roman Penyaev
2019-05-16  8:58 ` [PATCH v3 04/13] epoll: some sanity flags checks for epoll syscalls " Roman Penyaev
2019-05-16  8:58 ` [PATCH v3 05/13] epoll: offload polling to a work in case of epfd polled " Roman Penyaev
2019-05-21  7:51   ` Eric Wong
2019-05-22 12:54     ` Roman Penyaev
2019-05-16  8:58 ` [PATCH v3 06/13] epoll: introduce helpers for adding/removing events to uring Roman Penyaev
2019-05-31  9:55   ` Peter Zijlstra
2019-05-31 11:24     ` Roman Penyaev
2019-05-31 13:11       ` Peter Zijlstra
2019-05-31  9:56   ` Peter Zijlstra
2019-05-31 11:15     ` Roman Penyaev
2019-05-31 12:53       ` Peter Zijlstra
2019-05-31 14:28         ` Roman Penyaev
2019-05-31 16:53           ` Peter Zijlstra
2019-05-31 12:56       ` Peter Zijlstra
2019-05-31 14:21         ` Roman Penyaev
2019-05-31 16:51           ` Peter Zijlstra
2019-05-31 18:58             ` Roman Penyaev
2019-06-03  9:09               ` Peter Zijlstra
2019-06-03 10:02                 ` Roman Penyaev
2019-05-16  8:58 ` Roman Penyaev [this message]
2019-05-31  9:56   ` [PATCH v3 07/13] epoll: call ep_add_event_to_uring() from ep_poll_callback() Peter Zijlstra
2019-05-31 11:22     ` Roman Penyaev
2019-05-31 13:05       ` Peter Zijlstra
2019-05-31 15:05         ` Roman Penyaev
2019-05-16  8:58 ` [PATCH v3 08/13] epoll: support polling from userspace for ep_insert() Roman Penyaev
2019-05-16  8:58 ` [PATCH v3 09/13] epoll: support polling from userspace for ep_remove() Roman Penyaev
2019-05-16  8:58 ` [PATCH v3 10/13] epoll: support polling from userspace for ep_modify() Roman Penyaev
2019-05-16  8:58 ` [PATCH v3 11/13] epoll: support polling from userspace for ep_poll() Roman Penyaev
2019-05-16  8:58 ` [PATCH v3 12/13] epoll: support mapping for epfd when polled from userspace Roman Penyaev
2019-05-16  8:58 ` [PATCH v3 13/13] epoll: implement epoll_create2() syscall Roman Penyaev
2019-05-16 10:03   ` Arnd Bergmann
2019-05-16 10:20     ` Roman Penyaev
2019-05-16 10:57       ` Arnd Bergmann
2019-05-22  2:33       ` Andrew Morton
2019-05-22  9:11         ` Roman Penyaev
2019-05-22 11:14         ` Arnd Bergmann
2019-05-22 18:36           ` Andrew Morton
2019-05-31  9:55 ` [PATCH v3 00/13] epoll: support pollable epoll from userspace Peter Zijlstra
2019-05-31 14:48 ` Jens Axboe
2019-05-31 16:02   ` Roman Penyaev
2019-05-31 16:54     ` Jens Axboe
2019-05-31 19:45       ` Roman Penyaev
2019-05-31 21:09         ` Jens Axboe
2019-06-05  6:17           ` Roman Penyaev
2019-05-31 16:33 ` Peter Zijlstra
2019-05-31 18:50   ` Roman Penyaev

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190516085810.31077-8-rpenyaev@suse.de \
    --to=rpenyaev@suse.de \
    --cc=akpm@linux-foundation.org \
    --cc=azat@libevent.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=torvalds@linux-foundation.org \
    --cc=viro@zeniv.linux.org.uk \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git