From: Jason Xing <kerneljasonxing@gmail.com>
To: Eric Dumazet <edumazet@google.com>
Cc: David Miller <davem@davemloft.net>,
Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>,
Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
netdev <netdev@vger.kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
liweishi <liweishi@kuaishou.com>,
Shujin Li <lishujin@kuaishou.com>
Subject: Re: [PATCH] tcp: fix TCP socks unreleased in BBR mode
Date: Wed, 3 Jun 2020 14:32:29 +0800 [thread overview]
Message-ID: <CAL+tcoAQGZRaGJy1a1R=Ut=3+WnvCO7UNLq9J_juTtZJmTnC9g@mail.gmail.com> (raw)
In-Reply-To: <CANn89iJfLM2Hz69d9qOZoRKwzzCCpgVRZ1zbTTbg4vGvSAEZ-w@mail.gmail.com>
On Wed, Jun 3, 2020 at 1:44 PM Eric Dumazet <edumazet@google.com> wrote:
>
> On Tue, Jun 2, 2020 at 10:05 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
> >
> > Hi Eric,
> >
> > I'm still trying to understand what you're saying before. Would this
> > be better as following:
> > 1) discard the tcp_internal_pacing() function.
> > 2) remove where the tcp_internal_pacing() is called in the
> > __tcp_transmit_skb() function.
> >
> > If we do so, we could avoid 'too late to give up pacing'. Meanwhile,
> > should we introduce the tcp_wstamp_ns socket field as commit
> > (864e5c090749) does?
> >
>
> Please do not top-post on netdev mailing list.
>
>
> I basically suggested double-checking which point in TCP could end up
> calling tcp_internal_pacing()
> while the timer was already armed.
>
> I guess this is mtu probing.
Thanks for suggestions. I will recheck the point.
>
> Please try the following patch : If we still have another bug, a
> WARNING should give us a stack trace.
>
Agreed. I will apply this part of code and test it, then get back some
information here.
If it runs well as we expect, I decide to send this patch as v2 for
4.19 linux kernel.
Jason
>
> diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
> index cc4ba42052c21b206850594db6751810d8fc72b4..8f4081b228486305222767d4d118b9b6ed0ffda3
> 100644
> --- a/net/ipv4/tcp_output.c
> +++ b/net/ipv4/tcp_output.c
> @@ -977,12 +977,26 @@ static void tcp_internal_pacing(struct sock *sk,
> const struct sk_buff *skb)
>
> len_ns = (u64)skb->len * NSEC_PER_SEC;
> do_div(len_ns, rate);
> +
> + /* If hrtimer is already armed, then our caller has not properly
> + * used tcp_pacing_check().
> + */
> + if (unlikely(hrtimer_is_queued(&tcp_sk(sk)->pacing_timer))) {
> + WARN_ON_ONCE(1);
> + return;
> + }
> hrtimer_start(&tcp_sk(sk)->pacing_timer,
> ktime_add_ns(ktime_get(), len_ns),
> HRTIMER_MODE_ABS_PINNED_SOFT);
> sock_hold(sk);
> }
>
> +static bool tcp_pacing_check(const struct sock *sk)
> +{
> + return tcp_needs_internal_pacing(sk) &&
> + hrtimer_is_queued(&tcp_sk(sk)->pacing_timer);
> +}
> +
> static void tcp_update_skb_after_send(struct tcp_sock *tp, struct sk_buff *skb)
> {
> skb->skb_mstamp = tp->tcp_mstamp;
> @@ -2117,6 +2131,9 @@ static int tcp_mtu_probe(struct sock *sk)
> if (!tcp_can_coalesce_send_queue_head(sk, probe_size))
> return -1;
>
> + if (tcp_pacing_check(sk))
> + return -1;
> +
> /* We're allowed to probe. Build it now. */
> nskb = sk_stream_alloc_skb(sk, probe_size, GFP_ATOMIC, false);
> if (!nskb)
> @@ -2190,11 +2207,6 @@ static int tcp_mtu_probe(struct sock *sk)
> return -1;
> }
>
> -static bool tcp_pacing_check(const struct sock *sk)
> -{
> - return tcp_needs_internal_pacing(sk) &&
> - hrtimer_is_queued(&tcp_sk(sk)->pacing_timer);
> -}
>
> /* TCP Small Queues :
> * Control number of packets in qdisc/devices to two packets / or ~1 ms.
>
>
>
> > Thanks,
> > Jason
> >
> > On Wed, Jun 3, 2020 at 10:44 AM Eric Dumazet <edumazet@google.com> wrote:
> > >
> > > On Tue, Jun 2, 2020 at 7:42 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
> > > >
> > > > I agree with you. The upstream has already dropped and optimized this
> > > > part (commit 864e5c090749), so it would not happen like that. However
> > > > the old kernels like LTS still have the problem which causes
> > > > large-scale crashes on our thousands of machines after running for a
> > > > long while. I will send the fix to the correct tree soon :)
> > >
> > > If you run BBR at scale (thousands of machines), you probably should
> > > use sch_fq instead of internal pacing,
> > > just saying ;)
> > >
> > >
> > > >
> > > > Thanks again,
> > > > Jason
> > > >
> > > > On Wed, Jun 3, 2020 at 10:29 AM Eric Dumazet <edumazet@google.com> wrote:
> > > > >
> > > > > On Tue, Jun 2, 2020 at 6:53 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
> > > > > >
> > > > > > Hi Eric,
> > > > > >
> > > > > > I'm sorry that I didn't write enough clearly. We're running the
> > > > > > pristine 4.19.125 linux kernel (the latest LTS version) and have been
> > > > > > haunted by such an issue. This patch is high-important, I think. So
> > > > > > I'm going to resend this email with the [patch 4.19] on the headline
> > > > > > and cc Greg.
> > > > >
> > > > > Yes, please always give for which tree a patch is meant for.
> > > > >
> > > > > Problem is that your patch is not correct.
> > > > > In these old kernels, tcp_internal_pacing() is called _after_ the
> > > > > packet has been sent.
> > > > > It is too late to 'give up pacing'
> > > > >
> > > > > The packet should not have been sent if the pacing timer is queued
> > > > > (otherwise this means we do not respect pacing)
> > > > >
> > > > > So the bug should be caught earlier. check where tcp_pacing_check()
> > > > > calls are missing.
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > > On Tue, Jun 2, 2020 at 9:05 PM Eric Dumazet <edumazet@google.com> wrote:
> > > > > > >
> > > > > > > On Tue, Jun 2, 2020 at 1:05 AM <kerneljasonxing@gmail.com> wrote:
> > > > > > > >
> > > > > > > > From: Jason Xing <kerneljasonxing@gmail.com>
> > > > > > > >
> > > > > > > > TCP socks cannot be released because of the sock_hold() increasing the
> > > > > > > > sk_refcnt in the manner of tcp_internal_pacing() when RTO happens.
> > > > > > > > Therefore, this situation could increase the slab memory and then trigger
> > > > > > > > the OOM if the machine has beening running for a long time. This issue,
> > > > > > > > however, can happen on some machine only running a few days.
> > > > > > > >
> > > > > > > > We add one exception case to avoid unneeded use of sock_hold if the
> > > > > > > > pacing_timer is enqueued.
> > > > > > > >
> > > > > > > > Reproduce procedure:
> > > > > > > > 0) cat /proc/slabinfo | grep TCP
> > > > > > > > 1) switch net.ipv4.tcp_congestion_control to bbr
> > > > > > > > 2) using wrk tool something like that to send packages
> > > > > > > > 3) using tc to increase the delay in the dev to simulate the busy case.
> > > > > > > > 4) cat /proc/slabinfo | grep TCP
> > > > > > > > 5) kill the wrk command and observe the number of objects and slabs in TCP.
> > > > > > > > 6) at last, you could notice that the number would not decrease.
> > > > > > > >
> > > > > > > > Signed-off-by: Jason Xing <kerneljasonxing@gmail.com>
> > > > > > > > Signed-off-by: liweishi <liweishi@kuaishou.com>
> > > > > > > > Signed-off-by: Shujin Li <lishujin@kuaishou.com>
> > > > > > > > ---
> > > > > > > > net/ipv4/tcp_output.c | 3 ++-
> > > > > > > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > > > > > > >
> > > > > > > > diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
> > > > > > > > index cc4ba42..5cf63d9 100644
> > > > > > > > --- a/net/ipv4/tcp_output.c
> > > > > > > > +++ b/net/ipv4/tcp_output.c
> > > > > > > > @@ -969,7 +969,8 @@ static void tcp_internal_pacing(struct sock *sk, const struct sk_buff *skb)
> > > > > > > > u64 len_ns;
> > > > > > > > u32 rate;
> > > > > > > >
> > > > > > > > - if (!tcp_needs_internal_pacing(sk))
> > > > > > > > + if (!tcp_needs_internal_pacing(sk) ||
> > > > > > > > + hrtimer_is_queued(&tcp_sk(sk)->pacing_timer))
> > > > > > > > return;
> > > > > > > > rate = sk->sk_pacing_rate;
> > > > > > > > if (!rate || rate == ~0U)
> > > > > > > > --
> > > > > > > > 1.8.3.1
> > > > > > > >
> > > > > > >
> > > > > > > Hi Jason.
> > > > > > >
> > > > > > > Please do not send patches that do not apply to current upstream trees.
> > > > > > >
> > > > > > > Instead, backport to your kernels the needed fixes.
> > > > > > >
> > > > > > > I suspect that you are not using a pristine linux kernel, but some
> > > > > > > heavily modified one and something went wrong in your backports.
> > > > > > > Do not ask us to spend time finding what went wrong.
> > > > > > >
> > > > > > > Thank you.
next prev parent reply other threads:[~2020-06-03 6:33 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-02 8:04 [PATCH] tcp: fix TCP socks unreleased in BBR mode kerneljasonxing
2020-06-02 13:05 ` Eric Dumazet
2020-06-03 1:53 ` Jason Xing
2020-06-03 2:14 ` David Miller
2020-06-03 2:29 ` Eric Dumazet
2020-06-03 2:41 ` Jason Xing
2020-06-03 2:43 ` Eric Dumazet
2020-06-03 2:48 ` Jason Xing
2020-06-03 5:05 ` Jason Xing
2020-06-03 5:44 ` Eric Dumazet
2020-06-03 6:32 ` Jason Xing [this message]
2020-06-03 12:02 ` Neal Cardwell
2020-06-03 13:51 ` Jason Xing
2020-06-03 13:54 ` Eric Dumazet
2020-06-03 14:07 ` Neal Cardwell
2020-06-04 8:40 ` Jason Xing
2020-06-04 9:00 ` [PATCH v2 4.19] " kerneljasonxing
2020-06-04 13:10 ` Eric Dumazet
2020-06-04 13:47 ` Jason Xing
2020-08-11 10:37 ` Jason Xing
2020-08-11 15:32 ` Eric Dumazet
2021-04-30 10:51 ` Jason Xing
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAL+tcoAQGZRaGJy1a1R=Ut=3+WnvCO7UNLq9J_juTtZJmTnC9g@mail.gmail.com' \
--to=kerneljasonxing@gmail.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=kuznet@ms2.inr.ac.ru \
--cc=linux-kernel@vger.kernel.org \
--cc=lishujin@kuaishou.com \
--cc=liweishi@kuaishou.com \
--cc=netdev@vger.kernel.org \
--cc=yoshfuji@linux-ipv6.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).