* [PATHC bpf v2] tcp_bpf: Fix one concurrency problem in the tcp_bpf_send_verdict function
@ 2021-10-12 5:20 Liu Jian
2021-10-26 14:26 ` John Fastabend
0 siblings, 1 reply; 2+ messages in thread
From: Liu Jian @ 2021-10-12 5:20 UTC (permalink / raw)
To: john.fastabend, daniel, jakub, lmb, edumazet, davem, yoshfuji,
dsahern, kuba, ast, andrii, kafai, songliubraving, yhs, kpsingh,
netdev, bpf
Cc: liujian56
With two Msgs, msgA and msgB and a user doing nonblocking sendmsg calls (or
multiple cores) on a single socket 'sk' we could get the following flow.
msgA, sk msgB, sk
----------- ---------------
tcp_bpf_sendmsg()
lock(sk)
psock = sk->psock
tcp_bpf_sendmsg()
lock(sk) ... blocking
tcp_bpf_send_verdict
if (psock->eval == NONE)
psock->eval = sk_psock_msg_verdict
..
< handle SK_REDIRECT case >
release_sock(sk) < lock dropped so grab here >
ret = tcp_bpf_sendmsg_redir
psock = sk->psock
tcp_bpf_send_verdict
lock_sock(sk) ... blocking on B
if (psock->eval == NONE) <- boom.
psock->eval will have msgA state
The problem here is we dropped the lock on msgA and grabbed it with msgB.
Now we have old state in psock and importantly psock->eval has not been
cleared. So msgB will run whatever action was done on A and the verdict
program may never see it.
Fixes: 604326b41a6fb ("bpf, sockmap: convert to generic sk_msg interface")
Signed-off-by: Liu Jian <liujian56@huawei.com>
---
v2: change commit message, and add the fixes tag
net/ipv4/tcp_bpf.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c
index d3e9386b493e..9d068153c316 100644
--- a/net/ipv4/tcp_bpf.c
+++ b/net/ipv4/tcp_bpf.c
@@ -232,6 +232,7 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,
bool cork = false, enospc = sk_msg_full(msg);
struct sock *sk_redir;
u32 tosend, delta = 0;
+ u32 eval = __SK_NONE;
int ret;
more_data:
@@ -275,13 +276,24 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,
case __SK_REDIRECT:
sk_redir = psock->sk_redir;
sk_msg_apply_bytes(psock, tosend);
+ if (!psock->apply_bytes) {
+ /* Clean up before releasing the sock lock. */
+ eval = psock->eval;
+ psock->eval = __SK_NONE;
+ psock->sk_redir = NULL;
+ }
if (psock->cork) {
cork = true;
psock->cork = NULL;
}
sk_msg_return(sk, msg, tosend);
release_sock(sk);
+
ret = tcp_bpf_sendmsg_redir(sk_redir, msg, tosend, flags);
+
+ if (eval == __SK_REDIRECT)
+ sock_put(sk_redir);
+
lock_sock(sk);
if (unlikely(ret < 0)) {
int free = sk_msg_free_nocharge(sk, msg);
--
2.17.1
^ permalink raw reply related [flat|nested] 2+ messages in thread
* RE: [PATHC bpf v2] tcp_bpf: Fix one concurrency problem in the tcp_bpf_send_verdict function
2021-10-12 5:20 [PATHC bpf v2] tcp_bpf: Fix one concurrency problem in the tcp_bpf_send_verdict function Liu Jian
@ 2021-10-26 14:26 ` John Fastabend
0 siblings, 0 replies; 2+ messages in thread
From: John Fastabend @ 2021-10-26 14:26 UTC (permalink / raw)
To: Liu Jian, john.fastabend, daniel, jakub, lmb, edumazet, davem,
yoshfuji, dsahern, kuba, ast, andrii, kafai, songliubraving, yhs,
kpsingh, netdev, bpf
Cc: liujian56
Liu Jian wrote:
> With two Msgs, msgA and msgB and a user doing nonblocking sendmsg calls (or
> multiple cores) on a single socket 'sk' we could get the following flow.
>
> msgA, sk msgB, sk
> ----------- ---------------
> tcp_bpf_sendmsg()
> lock(sk)
> psock = sk->psock
> tcp_bpf_sendmsg()
> lock(sk) ... blocking
> tcp_bpf_send_verdict
> if (psock->eval == NONE)
> psock->eval = sk_psock_msg_verdict
> ..
> < handle SK_REDIRECT case >
> release_sock(sk) < lock dropped so grab here >
> ret = tcp_bpf_sendmsg_redir
> psock = sk->psock
> tcp_bpf_send_verdict
> lock_sock(sk) ... blocking on B
> if (psock->eval == NONE) <- boom.
> psock->eval will have msgA state
>
> The problem here is we dropped the lock on msgA and grabbed it with msgB.
> Now we have old state in psock and importantly psock->eval has not been
> cleared. So msgB will run whatever action was done on A and the verdict
> program may never see it.
>
> Fixes: 604326b41a6fb ("bpf, sockmap: convert to generic sk_msg interface")
> Signed-off-by: Liu Jian <liujian56@huawei.com>
Yep thanks for digging into this. Nice catch. And commit looks good now.
Acked-by: John Fastabend <john.fastabend@gmail.com>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2021-10-26 14:26 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-12 5:20 [PATHC bpf v2] tcp_bpf: Fix one concurrency problem in the tcp_bpf_send_verdict function Liu Jian
2021-10-26 14:26 ` John Fastabend
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).