[PATCH net-next] packet: fix warnings in rollover lock contention

[PATCH net-next] packet: fix warnings in rollover lock contention

[Willem de Bruijn]

From: Willem de Bruijn <willemb@google.com>

Avoid two xchg calls whose return values were unused, causing this
warning on some architectures:

    warning: value computed is not used [-Wunused-value]
    #define xchg(ptr,x) ((__typeof__(*(ptr)))\
        __xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))
                       ^
    net/packet/af_packet.c:1314:3: note: in expansion of macro 'xchg'
    xchg(&po->pressure, !has_room);

The relevant variable is a hint to avoid lock contention. It is
allowed to be imprecise (race).

Still, when rewriting this, also convert to use explicit atomic ops
and remove a race by switching to atomic_cmpxchg. A rerun of the
experiment from the original patch did not show this to cause
significant cache line contention. Another non-atomic conditional
clear remains in packet_poll, and is safe.

Fixes: 2ccdbaa6d55b ("packet: rollover lock contention avoidance")

Signed-off-by: Willem de Bruijn <willemb@google.com>
---
 net/packet/af_packet.c | 12 ++++++------
 net/packet/internal.h  |  2 +-
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 31d5856..ac1a589 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1310,8 +1310,7 @@ static int packet_rcv_has_room(struct packet_sock *po, struct sk_buff *skb)
 	}
 
 	has_room = ret == ROOM_NORMAL;
-	if (po->pressure == has_room)
-		xchg(&po->pressure, !has_room);
+	if (atomic_cmpxchg(&po->pressure, has_room, !has_room)) {}
 
 	return ret;
 }
@@ -1409,7 +1408,7 @@ static unsigned int fanout_demux_rollover(struct packet_fanout *f,
 	i = j = min_t(int, po->rollover->sock, num - 1);
 	do {
 		po_next = pkt_sk(f->arr[i]);
-		if (po_next != po && !po_next->pressure &&
+		if (po_next != po && !atomic_read(&po_next->pressure) &&
 		    packet_rcv_has_room(po_next, skb) == ROOM_NORMAL) {
 			if (i != j)
 				po->rollover->sock = i;
@@ -3045,7 +3044,7 @@ static int packet_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
 	if (skb == NULL)
 		goto out;
 
-	if (pkt_sk(sk)->pressure)
+	if (atomic_read(&pkt_sk(sk)->pressure))
 		packet_rcv_has_room(pkt_sk(sk), NULL);
 
 	if (pkt_sk(sk)->has_vnet_hdr) {
@@ -3813,8 +3812,9 @@ static unsigned int packet_poll(struct file *file, struct socket *sock,
 			TP_STATUS_KERNEL))
 			mask |= POLLIN | POLLRDNORM;
 	}
-	if (po->pressure && __packet_rcv_has_room(po, NULL) == ROOM_NORMAL)
-		xchg(&po->pressure, 0);
+	if (atomic_read(&po->pressure) &&
+	    __packet_rcv_has_room(po, NULL) == ROOM_NORMAL)
+		atomic_set(&po->pressure, 0);
 	spin_unlock_bh(&sk->sk_receive_queue.lock);
 	spin_lock_bh(&sk->sk_write_queue.lock);
 	if (po->tx_ring.pg_vec) {
diff --git a/net/packet/internal.h b/net/packet/internal.h
index c035d26..f96cf54 100644
--- a/net/packet/internal.h
+++ b/net/packet/internal.h
@@ -110,7 +110,7 @@ struct packet_sock {
 				auxdata:1,
 				origdev:1,
 				has_vnet_hdr:1;
-	int			pressure;
+	atomic_t		pressure;
 	int			ifindex;	/* bound device		*/
 	__be16			num;
 	struct packet_rollover	*rollover;
-- 
2.2.0.rc0.207.ga3a616c

Re: [PATCH net-next] packet: fix warnings in rollover lock contention

[Eric Dumazet]

On Thu, 2015-05-14 at 10:42 -0400, Willem de Bruijn wrote:
> From: Willem de Bruijn <willemb@google.com>
> 
> Avoid two xchg calls whose return values were unused, causing this
> warning on some architectures:
> 
>     warning: value computed is not used [-Wunused-value]
>     #define xchg(ptr,x) ((__typeof__(*(ptr)))\
>         __xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))
>                        ^
>     net/packet/af_packet.c:1314:3: note: in expansion of macro 'xchg'
>     xchg(&po->pressure, !has_room);
> 
> The relevant variable is a hint to avoid lock contention. It is
> allowed to be imprecise (race).
> 
> Still, when rewriting this, also convert to use explicit atomic ops
> and remove a race by switching to atomic_cmpxchg. A rerun of the
> experiment from the original patch did not show this to cause
> significant cache line contention. Another non-atomic conditional
> clear remains in packet_poll, and is safe.
> 
> Fixes: 2ccdbaa6d55b ("packet: rollover lock contention avoidance")
> 
> Signed-off-by: Willem de Bruijn <willemb@google.com>
> ---
>  net/packet/af_packet.c | 12 ++++++------
>  net/packet/internal.h  |  2 +-
>  2 files changed, 7 insertions(+), 7 deletions(-)
> 
> diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
> index 31d5856..ac1a589 100644
> --- a/net/packet/af_packet.c
> +++ b/net/packet/af_packet.c
> @@ -1310,8 +1310,7 @@ static int packet_rcv_has_room(struct packet_sock *po, struct sk_buff *skb)
>  	}
>  
>  	has_room = ret == ROOM_NORMAL;
> -	if (po->pressure == has_room)
> -		xchg(&po->pressure, !has_room);
> +	if (atomic_cmpxchg(&po->pressure, has_room, !has_room)) {}
>  

This makes no sense to me.

I thought you wanted to avoid dirtying the cache line.

No atomic op can help the race here.

Re: [PATCH net-next] packet: fix warnings in rollover lock contention

[Willem de Bruijn]

>> diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
>> index 31d5856..ac1a589 100644
>> --- a/net/packet/af_packet.c
>> +++ b/net/packet/af_packet.c
>> @@ -1310,8 +1310,7 @@ static int packet_rcv_has_room(struct packet_sock *po, struct sk_buff *skb)
>>       }
>>
>>       has_room = ret == ROOM_NORMAL;
>> -     if (po->pressure == has_room)
>> -             xchg(&po->pressure, !has_room);
>> +     if (atomic_cmpxchg(&po->pressure, has_room, !has_room)) {}
>>
>
> This makes no sense to me.
>
> I thought you wanted to avoid dirtying the cache line.
> No atomic op can help the race here.

I principally want to avoid the lock contention on sk_receive_queue.lock,
which is held for a lot longer while probing frames. But yes, I'd prefer to
avoid the cacheline contention as well.

The alternative is to keep the race and just replace the xchg with a
straight assignment.

Re: [PATCH net-next] packet: fix warnings in rollover lock contention

[Eric Dumazet]

On Thu, 2015-05-14 at 11:53 -0400, Willem de Bruijn wrote:

> I principally want to avoid the lock contention on sk_receive_queue.lock,
> which is held for a lot longer while probing frames. But yes, I'd prefer to
> avoid the cacheline contention as well.
> 
> The alternative is to keep the race and just replace the xchg with a
> straight assignment.

Please describe the race. It seems quite innocent at first look.

Clearly putting xchg() gives a false sense of security in this context.

Atomic ops should be reserved for cases we cannot avoid them,
not to give false hopes ;)

Re: [PATCH net-next] packet: fix warnings in rollover lock contention

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 14 May 2015 09:24:46 -0700

> On Thu, 2015-05-14 at 11:53 -0400, Willem de Bruijn wrote:
> 
>> I principally want to avoid the lock contention on sk_receive_queue.lock,
>> which is held for a lot longer while probing frames. But yes, I'd prefer to
>> avoid the cacheline contention as well.
>> 
>> The alternative is to keep the race and just replace the xchg with a
>> straight assignment.
> 
> Please describe the race. It seems quite innocent at first look.
> 
> Clearly putting xchg() gives a false sense of security in this context.
> 
> Atomic ops should be reserved for cases we cannot avoid them,
> not to give false hopes ;)

Basically, ->pressure seems to exist merely to optimize the scanner
in fanout_demux_rollover().  It makes it so that we don't check
sockets we already know lack space.

It is set (in an unlocked context) by packet_rcv_has_room() calls
which calculate that the socket lacks space.

It is cleared either in non-tpacket recvmsg() or poll(), the latter
of which holds the socket receive queue spinlock.

This kind of variable and conditional locking is crummy, at best.

Since non-tpacket recvmsg already has to hold the receive queue lock
to pull out the SKB (via skb_recv_datagram()), there is no value to
the conditional locking done by packet_rcv_has_room().

Just take the receive queue lock always, and then you can guarantee
that all ->pressure updates occur under that lock.

Tests can be done asynchronously without locking in the
fanout_demux_rollover() code, and that's fine.  It's a heuristic
after all.

Like this:

diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 31d5856..0947895 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1301,17 +1301,14 @@ static int packet_rcv_has_room(struct packet_sock *po, struct sk_buff *skb)
 	int ret;
 	bool has_room;
 
-	if (po->prot_hook.func == tpacket_rcv) {
-		spin_lock(&po->sk.sk_receive_queue.lock);
-		ret = __packet_rcv_has_room(po, skb);
-		spin_unlock(&po->sk.sk_receive_queue.lock);
-	} else {
-		ret = __packet_rcv_has_room(po, skb);
-	}
+	spin_lock(&po->sk.sk_receive_queue.lock);
 
+	ret = __packet_rcv_has_room(po, skb);
 	has_room = ret == ROOM_NORMAL;
 	if (po->pressure == has_room)
-		xchg(&po->pressure, !has_room);
+		po->pressure = !has_room;
+
+	spin_unlock(&po->sk.sk_receive_queue.lock);
 
 	return ret;
 }
@@ -3814,7 +3811,7 @@ static unsigned int packet_poll(struct file *file, struct socket *sock,
 			mask |= POLLIN | POLLRDNORM;
 	}
 	if (po->pressure && __packet_rcv_has_room(po, NULL) == ROOM_NORMAL)
-		xchg(&po->pressure, 0);
+		po->pressure = 0;
 	spin_unlock_bh(&sk->sk_receive_queue.lock);
 	spin_lock_bh(&sk->sk_write_queue.lock);
 	if (po->tx_ring.pg_vec) {

Re: [PATCH net-next] packet: fix warnings in rollover lock contention

[Willem de Bruijn]

On Thu, May 14, 2015 at 12:59 PM, David Miller <davem@davemloft.net> wrote:
> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Thu, 14 May 2015 09:24:46 -0700
>
>> On Thu, 2015-05-14 at 11:53 -0400, Willem de Bruijn wrote:
>>
>>> I principally want to avoid the lock contention on sk_receive_queue.lock,
>>> which is held for a lot longer while probing frames. But yes, I'd prefer to
>>> avoid the cacheline contention as well.
>>>
>>> The alternative is to keep the race and just replace the xchg with a
>>> straight assignment.
>>
>> Please describe the race. It seems quite innocent at first look.

It is. David described it well.

>> Clearly putting xchg() gives a false sense of security in this context.

Agreed.

>> Atomic ops should be reserved for cases we cannot avoid them,
>> not to give false hopes ;)
>
> Basically, ->pressure seems to exist merely to optimize the scanner
> in fanout_demux_rollover().  It makes it so that we don't check
> sockets we already know lack space.
>
> It is set (in an unlocked context) by packet_rcv_has_room() calls
> which calculate that the socket lacks space.
>
> It is cleared either in non-tpacket recvmsg() or poll(), the latter
> of which holds the socket receive queue spinlock.
>
> This kind of variable and conditional locking is crummy, at best.
>
> Since non-tpacket recvmsg already has to hold the receive queue lock
> to pull out the SKB (via skb_recv_datagram()), there is no value to
> the conditional locking done by packet_rcv_has_room().

Good point. I hadn't thought of that.

> Just take the receive queue lock always, and then you can guarantee
> that all ->pressure updates occur under that lock.
>
> Tests can be done asynchronously without locking in the
> fanout_demux_rollover() code, and that's fine.  It's a heuristic
> after all.
>
> Like this:

This looks great, thanks. I can submit it, but it is essentially your fix.

> diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
> index 31d5856..0947895 100644
> --- a/net/packet/af_packet.c
> +++ b/net/packet/af_packet.c
> @@ -1301,17 +1301,14 @@ static int packet_rcv_has_room(struct packet_sock *po, struct sk_buff *skb)
>         int ret;
>         bool has_room;
>
> -       if (po->prot_hook.func == tpacket_rcv) {
> -               spin_lock(&po->sk.sk_receive_queue.lock);
> -               ret = __packet_rcv_has_room(po, skb);
> -               spin_unlock(&po->sk.sk_receive_queue.lock);
> -       } else {
> -               ret = __packet_rcv_has_room(po, skb);
> -       }
> +       spin_lock(&po->sk.sk_receive_queue.lock);
>
> +       ret = __packet_rcv_has_room(po, skb);
>         has_room = ret == ROOM_NORMAL;
>         if (po->pressure == has_room)
> -               xchg(&po->pressure, !has_room);
> +               po->pressure = !has_room;
> +
> +       spin_unlock(&po->sk.sk_receive_queue.lock);
>
>         return ret;
>  }
> @@ -3814,7 +3811,7 @@ static unsigned int packet_poll(struct file *file, struct socket *sock,
>                         mask |= POLLIN | POLLRDNORM;
>         }
>         if (po->pressure && __packet_rcv_has_room(po, NULL) == ROOM_NORMAL)
> -               xchg(&po->pressure, 0);
> +               po->pressure = 0;
>         spin_unlock_bh(&sk->sk_receive_queue.lock);
>         spin_lock_bh(&sk->sk_write_queue.lock);
>         if (po->tx_ring.pg_vec) {

Re: [PATCH net-next] packet: fix warnings in rollover lock contention

[David Miller]

From: Willem de Bruijn <willemb@google.com>
Date: Thu, 14 May 2015 14:35:57 -0400

> This looks great, thanks. I can submit it, but it is essentially your fix.

Just give me your Acked-by: and I'll apply my patch then.

Thanks.

Re: [PATCH net-next] packet: fix warnings in rollover lock contention

[Willem de Bruijn]

On Thu, May 14, 2015 at 2:46 PM, David Miller <davem@davemloft.net> wrote:
> From: Willem de Bruijn <willemb@google.com>
> Date: Thu, 14 May 2015 14:35:57 -0400
>
>> This looks great, thanks. I can submit it, but it is essentially your fix.
>
> Just give me your Acked-by: and I'll apply my patch then.

Actually, I just observed a soft lockup in testing. I guess we would
need spin_lock_irqsave. I'll verify and send it.