Possible FIFO race in lock_sock()

Possible FIFO race in lock_sock()

[Stephen Hemminger]

Doing a review to understand socket locking, found a race by inspection
but don't have a test to reproduce the problem.

It appears lock_sock() basically reinvents a semaphore in order to have
FIFO wakeup and allow test semantics for the bottom half.  The problem
is that when socket is locked, the wakeup is guaranteed FIFO.  It is
possible for another requester to sneak in the window between when owner
is cleared and the next queued requester is woken up.

Don't know what this impacts, perhaps out of order data on a pipe?

Scenario:
	Multiple requesters (A, B, C, D) and new requester N
	
	Assume A gets socket lock and is owner. 
	B, C, D are on the wait queue.
	A release_lock which ends up waking up B
	Before B runs and acquires socket lock:
	   N requests socket lock and sees owner is NULL so it grabs it.

The patch just checks the waitq before proceeding with the fast path.

Other alternatives:
1. Ignore it we aren't guaranteeing FIFO anyway.
	- then why bother with waitq when spin lock will do.
2. Replace socket_lock with a semaphore
	- with changes to BH to get same semantics
3. Implement a better FIFO spin lock

Comments?

diff -Nru a/include/net/sock.h b/include/net/sock.h
--- a/include/net/sock.h	Fri Feb 28 13:24:43 2003
+++ b/include/net/sock.h	Fri Feb 28 13:24:43 2003
@@ -356,7 +356,7 @@
 #define lock_sock(__sk) \
 do {	might_sleep(); \
 	spin_lock_bh(&((__sk)->lock.slock)); \
-	if ((__sk)->lock.owner != NULL) \
+	if ((__sk)->lock.owner != NULL || waitqueue_active(&((__sk)->lock.wq))) \
 		__lock_sock(__sk); \
 	(__sk)->lock.owner = (void *)1; \
 	spin_unlock_bh(&((__sk)->lock.slock)); \

Re: Possible FIFO race in lock_sock()

[David S. Miller]

   From: Stephen Hemminger <shemminger@osdl.org>
   Date: 28 Feb 2003 13:25:50 -0800
   
   Don't know what this impacts, perhaps out of order data on a pipe?

There's a race to get the lock, who cares?

If multiple people are playing with the socket, the exact pieces of
data they happen to get is random unless the application does
it's own locking.

Re: Possible FIFO race in lock_sock()

[Steve Hocking]

> Doing a review to understand socket locking, found a race by inspection
> but don't have a test to reproduce the problem.
> 
> It appears lock_sock() basically reinvents a semaphore in order to have
> FIFO wakeup and allow test semantics for the bottom half.  The problem
> is that when socket is locked, the wakeup is guaranteed FIFO.  It is
> possible for another requester to sneak in the window between when owner
> is cleared and the next queued requester is woken up.
> 
> Don't know what this impacts, perhaps out of order data on a pipe?
> 
> Scenario:
> 	Multiple requesters (A, B, C, D) and new requester N
> 	
> 	Assume A gets socket lock and is owner. 
> 	B, C, D are on the wait queue.
> 	A release_lock which ends up waking up B
> 	Before B runs and acquires socket lock:
> 	   N requests socket lock and sees owner is NULL so it grabs it.
> 
> The patch just checks the waitq before proceeding with the fast path.
> 
> Other alternatives:
> 1. Ignore it we aren't guaranteeing FIFO anyway.
> 	- then why bother with waitq when spin lock will do.
> 2. Replace socket_lock with a semaphore
> 	- with changes to BH to get same semantics
> 3. Implement a better FIFO spin lock

We may've seen a problem relating to this in the 2.4.xx series of kernels. It 
presents itself within various comms libs (MPI, PVM) as an invalid message and 
occasionally with data being sent to the wrong process on a node. It's very 
intermittent, and seems to occur less as the speed of the machine in question 
increases. We have some 1400 dual CPU nodes, a mixture of P3 and P4 boxes, 
with jobs spanning up to a hundred nodes, so it tends to pop out rather more 
frequently than most people would see it.


	Stephen
-- 
  The views expressed above are not those of PGS Tensor.

    "We've heard that a million monkeys at a million keyboards could produce
     the Complete Works of Shakespeare; now, thanks to the Internet, we know
     this is not true."            Robert Wilensky, University of California