From: Eric Dumazet <edumazet@google.com>
To: Jason Xing <kerneljasonxing@gmail.com>
Cc: davem@davemloft.net, yoshfuji@linux-ipv6.org, dsahern@kernel.org,
kuba@kernel.org--cc, pabeni@redhat.com, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org, Jason Xing <kernelxing@tencent.com>
Subject: Re: [PATCH net] tcp: avoid the lookup process failing to get sk in ehash table
Date: Sat, 14 Jan 2023 10:45:23 +0100 [thread overview]
Message-ID: <CANn89iKQjN1YiHqBTV3+zDYo0G11p-6=p7C-1GvFCp8Y=r4nvQ@mail.gmail.com> (raw)
In-Reply-To: <20230112065336.41034-1-kerneljasonxing@gmail.com>
On Thu, Jan 12, 2023 at 7:54 AM Jason Xing <kerneljasonxing@gmail.com> wrote:
>
> From: Jason Xing <kernelxing@tencent.com>
>
> While one cpu is working on looking up the right socket from ehash
> table, another cpu is done deleting the request socket and is about
> to add (or is adding) the big socket from the table. It means that
> we could miss both of them, even though it has little chance.
>
> Let me draw a call trace map of the server side.
> CPU 0 CPU 1
> ----- -----
> tcp_v4_rcv() syn_recv_sock()
> inet_ehash_insert()
> -> sk_nulls_del_node_init_rcu(osk)
> __inet_lookup_established()
> -> __sk_nulls_add_node_rcu(sk, list)
>
> Notice that the CPU 0 is receiving the data after the final ack
> during 3-way shakehands and CPU 1 is still handling the final ack.
>
> Why could this be a real problem?
> This case is happening only when the final ack and the first data
> receiving by different CPUs. Then the server receiving data with
> ACK flag tries to search one proper established socket from ehash
> table, but apparently it fails as my map shows above. After that,
> the server fetches a listener socket and then sends a RST because
> it finds a ACK flag in the skb (data), which obeys RST definition
> in RFC 793.
>
> Many thanks to Eric for great help from beginning to end.
>
> Fixes: 5e0724d027f0 ("tcp/dccp: fix hashdance race for passive sessions")
> Signed-off-by: Jason Xing <kernelxing@tencent.com>
> ---
> net/ipv4/inet_hashtables.c | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
> index 24a38b56fab9..18f88cb4efcb 100644
> --- a/net/ipv4/inet_hashtables.c
> +++ b/net/ipv4/inet_hashtables.c
> @@ -650,7 +650,16 @@ bool inet_ehash_insert(struct sock *sk, struct sock *osk, bool *found_dup_sk)
> spin_lock(lock);
> if (osk) {
> WARN_ON_ONCE(sk->sk_hash != osk->sk_hash);
> + if (sk_hashed(osk))
> + /* Before deleting the node, we insert a new one to make
> + * sure that the look-up=sk process would not miss either
> + * of them and that at least one node would exist in ehash
> + * table all the time. Otherwise there's a tiny chance
> + * that lookup process could find nothing in ehash table.
> + */
> + __sk_nulls_add_node_rcu(sk, list);
In our private email exchange, I suggested to insert sk at the _tail_
of the hash bucket.
Inserting it at the _head_ would still leave a race condition, because
a concurrent reader might
have already started the bucket traversal, and would not see 'sk'.
Thanks.
> ret = sk_nulls_del_node_init_rcu(osk);
> + goto unlock;
> } else if (found_dup_sk) {
> *found_dup_sk = inet_ehash_lookup_by_sk(sk, list);
> if (*found_dup_sk)
> @@ -660,6 +669,7 @@ bool inet_ehash_insert(struct sock *sk, struct sock *osk, bool *found_dup_sk)
> if (ret)
> __sk_nulls_add_node_rcu(sk, list);
>
> +unlock:
> spin_unlock(lock);
>
> return ret;
> --
> 2.37.3
>
next prev parent reply other threads:[~2023-01-14 9:45 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-12 6:53 [PATCH net] tcp: avoid the lookup process failing to get sk in ehash table Jason Xing
2023-01-14 2:14 ` Jason Xing
2023-01-14 9:26 ` Eric Dumazet
2023-01-14 9:48 ` Jason Xing
2023-01-14 9:45 ` Eric Dumazet [this message]
2023-01-14 12:05 ` Jason Xing
2023-01-14 12:30 ` Eric Dumazet
2023-01-14 12:48 ` Jason Xing
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CANn89iKQjN1YiHqBTV3+zDYo0G11p-6=p7C-1GvFCp8Y=r4nvQ@mail.gmail.com' \
--to=edumazet@google.com \
--cc=davem@davemloft.net \
--cc=dsahern@kernel.org \
--cc=kerneljasonxing@gmail.com \
--cc=kernelxing@tencent.com \
--cc=kuba@kernel.org--cc \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=yoshfuji@linux-ipv6.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).