* [PATCH net-next] packet: fix warnings in rollover lock contention
@ 2015-05-14 14:42 Willem de Bruijn
2015-05-14 15:33 ` Eric Dumazet
0 siblings, 1 reply; 8+ messages in thread
From: Willem de Bruijn @ 2015-05-14 14:42 UTC (permalink / raw)
To: netdev; +Cc: davem, Willem de Bruijn
From: Willem de Bruijn <willemb@google.com>
Avoid two xchg calls whose return values were unused, causing this
warning on some architectures:
warning: value computed is not used [-Wunused-value]
#define xchg(ptr,x) ((__typeof__(*(ptr)))\
__xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))
^
net/packet/af_packet.c:1314:3: note: in expansion of macro 'xchg'
xchg(&po->pressure, !has_room);
The relevant variable is a hint to avoid lock contention. It is
allowed to be imprecise (race).
Still, when rewriting this, also convert to use explicit atomic ops
and remove a race by switching to atomic_cmpxchg. A rerun of the
experiment from the original patch did not show this to cause
significant cache line contention. Another non-atomic conditional
clear remains in packet_poll, and is safe.
Fixes: 2ccdbaa6d55b ("packet: rollover lock contention avoidance")
Signed-off-by: Willem de Bruijn <willemb@google.com>
---
net/packet/af_packet.c | 12 ++++++------
net/packet/internal.h | 2 +-
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 31d5856..ac1a589 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1310,8 +1310,7 @@ static int packet_rcv_has_room(struct packet_sock *po, struct sk_buff *skb)
}
has_room = ret == ROOM_NORMAL;
- if (po->pressure == has_room)
- xchg(&po->pressure, !has_room);
+ if (atomic_cmpxchg(&po->pressure, has_room, !has_room)) {}
return ret;
}
@@ -1409,7 +1408,7 @@ static unsigned int fanout_demux_rollover(struct packet_fanout *f,
i = j = min_t(int, po->rollover->sock, num - 1);
do {
po_next = pkt_sk(f->arr[i]);
- if (po_next != po && !po_next->pressure &&
+ if (po_next != po && !atomic_read(&po_next->pressure) &&
packet_rcv_has_room(po_next, skb) == ROOM_NORMAL) {
if (i != j)
po->rollover->sock = i;
@@ -3045,7 +3044,7 @@ static int packet_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
if (skb == NULL)
goto out;
- if (pkt_sk(sk)->pressure)
+ if (atomic_read(&pkt_sk(sk)->pressure))
packet_rcv_has_room(pkt_sk(sk), NULL);
if (pkt_sk(sk)->has_vnet_hdr) {
@@ -3813,8 +3812,9 @@ static unsigned int packet_poll(struct file *file, struct socket *sock,
TP_STATUS_KERNEL))
mask |= POLLIN | POLLRDNORM;
}
- if (po->pressure && __packet_rcv_has_room(po, NULL) == ROOM_NORMAL)
- xchg(&po->pressure, 0);
+ if (atomic_read(&po->pressure) &&
+ __packet_rcv_has_room(po, NULL) == ROOM_NORMAL)
+ atomic_set(&po->pressure, 0);
spin_unlock_bh(&sk->sk_receive_queue.lock);
spin_lock_bh(&sk->sk_write_queue.lock);
if (po->tx_ring.pg_vec) {
diff --git a/net/packet/internal.h b/net/packet/internal.h
index c035d26..f96cf54 100644
--- a/net/packet/internal.h
+++ b/net/packet/internal.h
@@ -110,7 +110,7 @@ struct packet_sock {
auxdata:1,
origdev:1,
has_vnet_hdr:1;
- int pressure;
+ atomic_t pressure;
int ifindex; /* bound device */
__be16 num;
struct packet_rollover *rollover;
--
2.2.0.rc0.207.ga3a616c
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH net-next] packet: fix warnings in rollover lock contention
2015-05-14 14:42 [PATCH net-next] packet: fix warnings in rollover lock contention Willem de Bruijn
@ 2015-05-14 15:33 ` Eric Dumazet
2015-05-14 15:53 ` Willem de Bruijn
0 siblings, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2015-05-14 15:33 UTC (permalink / raw)
To: Willem de Bruijn; +Cc: netdev, davem
On Thu, 2015-05-14 at 10:42 -0400, Willem de Bruijn wrote:
> From: Willem de Bruijn <willemb@google.com>
>
> Avoid two xchg calls whose return values were unused, causing this
> warning on some architectures:
>
> warning: value computed is not used [-Wunused-value]
> #define xchg(ptr,x) ((__typeof__(*(ptr)))\
> __xchg((unsigned long)(x),(ptr),sizeof(*(ptr))))
> ^
> net/packet/af_packet.c:1314:3: note: in expansion of macro 'xchg'
> xchg(&po->pressure, !has_room);
>
> The relevant variable is a hint to avoid lock contention. It is
> allowed to be imprecise (race).
>
> Still, when rewriting this, also convert to use explicit atomic ops
> and remove a race by switching to atomic_cmpxchg. A rerun of the
> experiment from the original patch did not show this to cause
> significant cache line contention. Another non-atomic conditional
> clear remains in packet_poll, and is safe.
>
> Fixes: 2ccdbaa6d55b ("packet: rollover lock contention avoidance")
>
> Signed-off-by: Willem de Bruijn <willemb@google.com>
> ---
> net/packet/af_packet.c | 12 ++++++------
> net/packet/internal.h | 2 +-
> 2 files changed, 7 insertions(+), 7 deletions(-)
>
> diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
> index 31d5856..ac1a589 100644
> --- a/net/packet/af_packet.c
> +++ b/net/packet/af_packet.c
> @@ -1310,8 +1310,7 @@ static int packet_rcv_has_room(struct packet_sock *po, struct sk_buff *skb)
> }
>
> has_room = ret == ROOM_NORMAL;
> - if (po->pressure == has_room)
> - xchg(&po->pressure, !has_room);
> + if (atomic_cmpxchg(&po->pressure, has_room, !has_room)) {}
>
This makes no sense to me.
I thought you wanted to avoid dirtying the cache line.
No atomic op can help the race here.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next] packet: fix warnings in rollover lock contention
2015-05-14 15:33 ` Eric Dumazet
@ 2015-05-14 15:53 ` Willem de Bruijn
2015-05-14 16:24 ` Eric Dumazet
0 siblings, 1 reply; 8+ messages in thread
From: Willem de Bruijn @ 2015-05-14 15:53 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Network Development, David Miller
>> diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
>> index 31d5856..ac1a589 100644
>> --- a/net/packet/af_packet.c
>> +++ b/net/packet/af_packet.c
>> @@ -1310,8 +1310,7 @@ static int packet_rcv_has_room(struct packet_sock *po, struct sk_buff *skb)
>> }
>>
>> has_room = ret == ROOM_NORMAL;
>> - if (po->pressure == has_room)
>> - xchg(&po->pressure, !has_room);
>> + if (atomic_cmpxchg(&po->pressure, has_room, !has_room)) {}
>>
>
> This makes no sense to me.
>
> I thought you wanted to avoid dirtying the cache line.
> No atomic op can help the race here.
I principally want to avoid the lock contention on sk_receive_queue.lock,
which is held for a lot longer while probing frames. But yes, I'd prefer to
avoid the cacheline contention as well.
The alternative is to keep the race and just replace the xchg with a
straight assignment.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next] packet: fix warnings in rollover lock contention
2015-05-14 15:53 ` Willem de Bruijn
@ 2015-05-14 16:24 ` Eric Dumazet
2015-05-14 16:59 ` David Miller
0 siblings, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2015-05-14 16:24 UTC (permalink / raw)
To: Willem de Bruijn; +Cc: Network Development, David Miller
On Thu, 2015-05-14 at 11:53 -0400, Willem de Bruijn wrote:
> I principally want to avoid the lock contention on sk_receive_queue.lock,
> which is held for a lot longer while probing frames. But yes, I'd prefer to
> avoid the cacheline contention as well.
>
> The alternative is to keep the race and just replace the xchg with a
> straight assignment.
Please describe the race. It seems quite innocent at first look.
Clearly putting xchg() gives a false sense of security in this context.
Atomic ops should be reserved for cases we cannot avoid them,
not to give false hopes ;)
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next] packet: fix warnings in rollover lock contention
2015-05-14 16:24 ` Eric Dumazet
@ 2015-05-14 16:59 ` David Miller
2015-05-14 18:35 ` Willem de Bruijn
0 siblings, 1 reply; 8+ messages in thread
From: David Miller @ 2015-05-14 16:59 UTC (permalink / raw)
To: eric.dumazet; +Cc: willemb, netdev
From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 14 May 2015 09:24:46 -0700
> On Thu, 2015-05-14 at 11:53 -0400, Willem de Bruijn wrote:
>
>> I principally want to avoid the lock contention on sk_receive_queue.lock,
>> which is held for a lot longer while probing frames. But yes, I'd prefer to
>> avoid the cacheline contention as well.
>>
>> The alternative is to keep the race and just replace the xchg with a
>> straight assignment.
>
> Please describe the race. It seems quite innocent at first look.
>
> Clearly putting xchg() gives a false sense of security in this context.
>
> Atomic ops should be reserved for cases we cannot avoid them,
> not to give false hopes ;)
Basically, ->pressure seems to exist merely to optimize the scanner
in fanout_demux_rollover(). It makes it so that we don't check
sockets we already know lack space.
It is set (in an unlocked context) by packet_rcv_has_room() calls
which calculate that the socket lacks space.
It is cleared either in non-tpacket recvmsg() or poll(), the latter
of which holds the socket receive queue spinlock.
This kind of variable and conditional locking is crummy, at best.
Since non-tpacket recvmsg already has to hold the receive queue lock
to pull out the SKB (via skb_recv_datagram()), there is no value to
the conditional locking done by packet_rcv_has_room().
Just take the receive queue lock always, and then you can guarantee
that all ->pressure updates occur under that lock.
Tests can be done asynchronously without locking in the
fanout_demux_rollover() code, and that's fine. It's a heuristic
after all.
Like this:
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 31d5856..0947895 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -1301,17 +1301,14 @@ static int packet_rcv_has_room(struct packet_sock *po, struct sk_buff *skb)
int ret;
bool has_room;
- if (po->prot_hook.func == tpacket_rcv) {
- spin_lock(&po->sk.sk_receive_queue.lock);
- ret = __packet_rcv_has_room(po, skb);
- spin_unlock(&po->sk.sk_receive_queue.lock);
- } else {
- ret = __packet_rcv_has_room(po, skb);
- }
+ spin_lock(&po->sk.sk_receive_queue.lock);
+ ret = __packet_rcv_has_room(po, skb);
has_room = ret == ROOM_NORMAL;
if (po->pressure == has_room)
- xchg(&po->pressure, !has_room);
+ po->pressure = !has_room;
+
+ spin_unlock(&po->sk.sk_receive_queue.lock);
return ret;
}
@@ -3814,7 +3811,7 @@ static unsigned int packet_poll(struct file *file, struct socket *sock,
mask |= POLLIN | POLLRDNORM;
}
if (po->pressure && __packet_rcv_has_room(po, NULL) == ROOM_NORMAL)
- xchg(&po->pressure, 0);
+ po->pressure = 0;
spin_unlock_bh(&sk->sk_receive_queue.lock);
spin_lock_bh(&sk->sk_write_queue.lock);
if (po->tx_ring.pg_vec) {
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH net-next] packet: fix warnings in rollover lock contention
2015-05-14 16:59 ` David Miller
@ 2015-05-14 18:35 ` Willem de Bruijn
2015-05-14 18:46 ` David Miller
0 siblings, 1 reply; 8+ messages in thread
From: Willem de Bruijn @ 2015-05-14 18:35 UTC (permalink / raw)
To: David Miller; +Cc: Eric Dumazet, Network Development
On Thu, May 14, 2015 at 12:59 PM, David Miller <davem@davemloft.net> wrote:
> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Thu, 14 May 2015 09:24:46 -0700
>
>> On Thu, 2015-05-14 at 11:53 -0400, Willem de Bruijn wrote:
>>
>>> I principally want to avoid the lock contention on sk_receive_queue.lock,
>>> which is held for a lot longer while probing frames. But yes, I'd prefer to
>>> avoid the cacheline contention as well.
>>>
>>> The alternative is to keep the race and just replace the xchg with a
>>> straight assignment.
>>
>> Please describe the race. It seems quite innocent at first look.
It is. David described it well.
>> Clearly putting xchg() gives a false sense of security in this context.
Agreed.
>> Atomic ops should be reserved for cases we cannot avoid them,
>> not to give false hopes ;)
>
> Basically, ->pressure seems to exist merely to optimize the scanner
> in fanout_demux_rollover(). It makes it so that we don't check
> sockets we already know lack space.
>
> It is set (in an unlocked context) by packet_rcv_has_room() calls
> which calculate that the socket lacks space.
>
> It is cleared either in non-tpacket recvmsg() or poll(), the latter
> of which holds the socket receive queue spinlock.
>
> This kind of variable and conditional locking is crummy, at best.
>
> Since non-tpacket recvmsg already has to hold the receive queue lock
> to pull out the SKB (via skb_recv_datagram()), there is no value to
> the conditional locking done by packet_rcv_has_room().
Good point. I hadn't thought of that.
> Just take the receive queue lock always, and then you can guarantee
> that all ->pressure updates occur under that lock.
>
> Tests can be done asynchronously without locking in the
> fanout_demux_rollover() code, and that's fine. It's a heuristic
> after all.
>
> Like this:
This looks great, thanks. I can submit it, but it is essentially your fix.
> diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
> index 31d5856..0947895 100644
> --- a/net/packet/af_packet.c
> +++ b/net/packet/af_packet.c
> @@ -1301,17 +1301,14 @@ static int packet_rcv_has_room(struct packet_sock *po, struct sk_buff *skb)
> int ret;
> bool has_room;
>
> - if (po->prot_hook.func == tpacket_rcv) {
> - spin_lock(&po->sk.sk_receive_queue.lock);
> - ret = __packet_rcv_has_room(po, skb);
> - spin_unlock(&po->sk.sk_receive_queue.lock);
> - } else {
> - ret = __packet_rcv_has_room(po, skb);
> - }
> + spin_lock(&po->sk.sk_receive_queue.lock);
>
> + ret = __packet_rcv_has_room(po, skb);
> has_room = ret == ROOM_NORMAL;
> if (po->pressure == has_room)
> - xchg(&po->pressure, !has_room);
> + po->pressure = !has_room;
> +
> + spin_unlock(&po->sk.sk_receive_queue.lock);
>
> return ret;
> }
> @@ -3814,7 +3811,7 @@ static unsigned int packet_poll(struct file *file, struct socket *sock,
> mask |= POLLIN | POLLRDNORM;
> }
> if (po->pressure && __packet_rcv_has_room(po, NULL) == ROOM_NORMAL)
> - xchg(&po->pressure, 0);
> + po->pressure = 0;
> spin_unlock_bh(&sk->sk_receive_queue.lock);
> spin_lock_bh(&sk->sk_write_queue.lock);
> if (po->tx_ring.pg_vec) {
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next] packet: fix warnings in rollover lock contention
2015-05-14 18:35 ` Willem de Bruijn
@ 2015-05-14 18:46 ` David Miller
2015-05-14 18:59 ` Willem de Bruijn
0 siblings, 1 reply; 8+ messages in thread
From: David Miller @ 2015-05-14 18:46 UTC (permalink / raw)
To: willemb; +Cc: eric.dumazet, netdev
From: Willem de Bruijn <willemb@google.com>
Date: Thu, 14 May 2015 14:35:57 -0400
> This looks great, thanks. I can submit it, but it is essentially your fix.
Just give me your Acked-by: and I'll apply my patch then.
Thanks.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH net-next] packet: fix warnings in rollover lock contention
2015-05-14 18:46 ` David Miller
@ 2015-05-14 18:59 ` Willem de Bruijn
0 siblings, 0 replies; 8+ messages in thread
From: Willem de Bruijn @ 2015-05-14 18:59 UTC (permalink / raw)
To: David Miller; +Cc: Eric Dumazet, Network Development
On Thu, May 14, 2015 at 2:46 PM, David Miller <davem@davemloft.net> wrote:
> From: Willem de Bruijn <willemb@google.com>
> Date: Thu, 14 May 2015 14:35:57 -0400
>
>> This looks great, thanks. I can submit it, but it is essentially your fix.
>
> Just give me your Acked-by: and I'll apply my patch then.
Actually, I just observed a soft lockup in testing. I guess we would
need spin_lock_irqsave. I'll verify and send it.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2015-05-14 19:00 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-05-14 14:42 [PATCH net-next] packet: fix warnings in rollover lock contention Willem de Bruijn
2015-05-14 15:33 ` Eric Dumazet
2015-05-14 15:53 ` Willem de Bruijn
2015-05-14 16:24 ` Eric Dumazet
2015-05-14 16:59 ` David Miller
2015-05-14 18:35 ` Willem de Bruijn
2015-05-14 18:46 ` David Miller
2015-05-14 18:59 ` Willem de Bruijn
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.