All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH net v4] net: fix race between napi kthread mode and busy poll
@ 2021-03-16 22:36 Wei Wang
  2021-03-17 18:58 ` Jakub Kicinski
                   ` (2 more replies)
  0 siblings, 3 replies; 25+ messages in thread
From: Wei Wang @ 2021-03-16 22:36 UTC (permalink / raw)
  To: David S . Miller, Jakub Kicinski, netdev
  Cc: Martin Zaharinov, Alexander Duyck, Eric Dumazet, Paolo Abeni,
	Hannes Frederic Sowa

Currently, napi_thread_wait() checks for NAPI_STATE_SCHED bit to
determine if the kthread owns this napi and could call napi->poll() on
it. However, if socket busy poll is enabled, it is possible that the
busy poll thread grabs this SCHED bit (after the previous napi->poll()
invokes napi_complete_done() and clears SCHED bit) and tries to poll
on the same napi. napi_disable() could grab the SCHED bit as well.
This patch tries to fix this race by adding a new bit
NAPI_STATE_SCHED_THREADED in napi->state. This bit gets set in
____napi_schedule() if the threaded mode is enabled, and gets cleared
in napi_complete_done(), and we only poll the napi in kthread if this
bit is set. This helps distinguish the ownership of the napi between
kthread and other scenarios and fixes the race issue.

Fixes: 29863d41bb6e ("net: implement threaded-able napi poll loop support")
Reported-by: Martin Zaharinov <micron10@gmail.com>
Suggested-by: Jakub Kicinski <kuba@kernel.org>
Signed-off-by: Wei Wang <weiwan@google.com>
Cc: Alexander Duyck <alexanderduyck@fb.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Paolo Abeni <pabeni@redhat.com>
Cc: Hannes Frederic Sowa <hannes@stressinduktion.org>
---
Change since v3:
  - Add READ_ONCE() for thread->state and add comments in
    ____napi_schedule().
  
 include/linux/netdevice.h |  2 ++
 net/core/dev.c            | 19 ++++++++++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 5b67ea89d5f2..87a5d186faff 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -360,6 +360,7 @@ enum {
 	NAPI_STATE_IN_BUSY_POLL,	/* sk_busy_loop() owns this NAPI */
 	NAPI_STATE_PREFER_BUSY_POLL,	/* prefer busy-polling over softirq processing*/
 	NAPI_STATE_THREADED,		/* The poll is performed inside its own thread*/
+	NAPI_STATE_SCHED_THREADED,	/* Napi is currently scheduled in threaded mode */
 };
 
 enum {
@@ -372,6 +373,7 @@ enum {
 	NAPIF_STATE_IN_BUSY_POLL	= BIT(NAPI_STATE_IN_BUSY_POLL),
 	NAPIF_STATE_PREFER_BUSY_POLL	= BIT(NAPI_STATE_PREFER_BUSY_POLL),
 	NAPIF_STATE_THREADED		= BIT(NAPI_STATE_THREADED),
+	NAPIF_STATE_SCHED_THREADED	= BIT(NAPI_STATE_SCHED_THREADED),
 };
 
 enum gro_result {
diff --git a/net/core/dev.c b/net/core/dev.c
index 6c5967e80132..d3195a95f30e 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -4294,6 +4294,13 @@ static inline void ____napi_schedule(struct softnet_data *sd,
 		 */
 		thread = READ_ONCE(napi->thread);
 		if (thread) {
+			/* Avoid doing set_bit() if the thread is in
+			 * INTERRUPTIBLE state, cause napi_thread_wait()
+			 * makes sure to proceed with napi polling
+			 * if the thread is explicitly woken from here.
+			 */
+			if (READ_ONCE(thread->state) != TASK_INTERRUPTIBLE)
+				set_bit(NAPI_STATE_SCHED_THREADED, &napi->state);
 			wake_up_process(thread);
 			return;
 		}
@@ -6486,6 +6493,7 @@ bool napi_complete_done(struct napi_struct *n, int work_done)
 		WARN_ON_ONCE(!(val & NAPIF_STATE_SCHED));
 
 		new = val & ~(NAPIF_STATE_MISSED | NAPIF_STATE_SCHED |
+			      NAPIF_STATE_SCHED_THREADED |
 			      NAPIF_STATE_PREFER_BUSY_POLL);
 
 		/* If STATE_MISSED was set, leave STATE_SCHED set,
@@ -6968,16 +6976,25 @@ static int napi_poll(struct napi_struct *n, struct list_head *repoll)
 
 static int napi_thread_wait(struct napi_struct *napi)
 {
+	bool woken = false;
+
 	set_current_state(TASK_INTERRUPTIBLE);
 
 	while (!kthread_should_stop() && !napi_disable_pending(napi)) {
-		if (test_bit(NAPI_STATE_SCHED, &napi->state)) {
+		/* Testing SCHED_THREADED bit here to make sure the current
+		 * kthread owns this napi and could poll on this napi.
+		 * Testing SCHED bit is not enough because SCHED bit might be
+		 * set by some other busy poll thread or by napi_disable().
+		 */
+		if (test_bit(NAPI_STATE_SCHED_THREADED, &napi->state) || woken) {
 			WARN_ON(!list_empty(&napi->poll_list));
 			__set_current_state(TASK_RUNNING);
 			return 0;
 		}
 
 		schedule();
+		/* woken being true indicates this thread owns this napi. */
+		woken = true;
 		set_current_state(TASK_INTERRUPTIBLE);
 	}
 	__set_current_state(TASK_RUNNING);
-- 
2.31.0.rc2.261.g7f71774620-goog


^ permalink raw reply related	[flat|nested] 25+ messages in thread

end of thread, other threads:[~2021-09-24 16:57 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-16 22:36 [PATCH net v4] net: fix race between napi kthread mode and busy poll Wei Wang
2021-03-17 18:58 ` Jakub Kicinski
2021-03-17 21:50 ` patchwork-bot+netdevbpf
2021-03-20  8:45 ` Martin Zaharinov
2021-03-20  9:55   ` Eric Dumazet
2021-03-20 10:31     ` Martin Zaharinov
2021-03-30  9:25     ` Martin Zaharinov
2021-03-30 13:39       ` Eric Dumazet
2021-04-10 11:22         ` Bug Report Napi GRO ixgbe Martin Zaharinov
2021-04-12  8:36           ` Paolo Abeni
2021-04-26  8:31             ` Martin Zaharinov
2021-05-08 10:48               ` Bug Report Napi kthread rcd Martin Zaharinov
2021-05-09 10:40             ` Bug Report Napi GRO ixgbe Martin Zaharinov
2021-09-09 11:18         ` [PATCH net v4] net: fix race between napi kthread mode and busy poll Martin Zaharinov
2021-09-10  0:30           ` Wei Wang
2021-09-10  1:57             ` Martin Zaharinov
2021-09-15 14:22             ` Martin Zaharinov
2021-09-15 15:45               ` Wei Wang
2021-09-15 20:57                 ` Martin Zaharinov
2021-09-22 14:12                 ` Martin Zaharinov
2021-09-23 20:31                   ` Martin Zaharinov
2021-09-24  0:54                     ` Wei Wang
2021-09-24  6:18                       ` Martin Zaharinov
2021-09-24 16:42                         ` Wei Wang
     [not found]                           ` <CALidq=V5O6oco+JRWbdKZ4pUXzZOoaUJCu_yCh55M_ccA_6QYQ@mail.gmail.com>
2021-09-24 16:57                             ` Wei Wang

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.