From: Eric Dumazet <edumazet@google.com>
To: Jakub Kicinski <kuba@kernel.org>
Cc: Eric Dumazet <eric.dumazet@gmail.com>,
"David S . Miller" <davem@davemloft.net>,
netdev <netdev@vger.kernel.org>,
Soheil Hassas Yeganeh <soheil@google.com>,
Neal Cardwell <ncardwell@google.com>,
Arjun Roy <arjunroy@google.com>
Subject: Re: [PATCH net-next 17/20] tcp: defer skb freeing after socket lock is released
Date: Tue, 16 Nov 2021 08:46:37 -0800 [thread overview]
Message-ID: <CANn89iJb7s-JoCCfn=eoxZ_tX_2RaeEPZKO1aHyHtgHxLXsd2A@mail.gmail.com> (raw)
In-Reply-To: <20211116072735.68c104ee@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
On Tue, Nov 16, 2021 at 7:27 AM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Tue, 16 Nov 2021 07:22:02 -0800 Eric Dumazet wrote:
> > Here is the perf top profile on cpu used by user thread doing the
> > recvmsg(), at 96 Gbit/s
> >
> > We no longer see skb freeing related costs, but we still see costs of
> > having to process the backlog.
> >
> > 81.06% [kernel] [k] copy_user_enhanced_fast_string
> > 2.50% [kernel] [k] __skb_datagram_iter
> > 2.25% [kernel] [k] _copy_to_iter
> > 1.45% [kernel] [k] tcp_recvmsg_locked
> > 1.39% [kernel] [k] tcp_rcv_established
>
> Huh, somehow I assumed your 4k MTU numbers were with zero-copy :o
>
> Out of curiosity - what's the softirq load with 4k? Do you have an
> idea what the load is on the CPU consuming the data vs the softirq
> processing with 1500B ?
On my testing host,
4K MTU : processing ~2,600.000 packets per second in GRO and other parts
use about 60% of the core in BH.
(Some of this cost comes from a clang issue, and the csum_partial() one
I was working on last week)
NIC RX interrupts are firing about 25,000 times per second in this setup.
1500 MTU : processing ~ 5,800,000 packets per second uses one core in
BH (and also one core in recvmsg()),
We stay in NAPI mode (no IRQ rearming)
(That was with a TCP_STREAM run sustaining 70Gbit)
BH numbers also depend on IRQ coalescing parameters.
next prev parent reply other threads:[~2021-11-16 16:46 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-15 19:02 [PATCH net-next 00/20] tcp: optimizations for linux-5.17 Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 01/20] tcp: minor optimization in tcp_add_backlog() Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 02/20] tcp: remove dead code in __tcp_v6_send_check() Eric Dumazet
2021-11-16 2:48 ` David Ahern
2021-11-16 2:57 ` Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 03/20] tcp: small optimization in tcp_v6_send_check() Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 04/20] net: use sk_is_tcp() in more places Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 05/20] net: remove sk_route_forced_caps Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 06/20] net: remove sk_route_nocaps Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 07/20] ipv6: shrink struct ipcm6_cookie Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 08/20] net: shrink struct sock by 8 bytes Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 09/20] net: forward_alloc_get depends on CONFIG_MPTCP Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 10/20] net: cache align tcp_memory_allocated, tcp_sockets_allocated Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 11/20] tcp: small optimization in tcp recvmsg() Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 12/20] tcp: add RETPOLINE mitigation to sk_backlog_rcv Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 13/20] tcp: annotate data-races on tp->segs_in and tp->data_segs_in Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 14/20] tcp: annotate races around tp->urg_data Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 15/20] tcp: tp->urg_data is unlikely to be set Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 16/20] tcp: avoid indirect calls to sock_rfree Eric Dumazet
2021-11-15 19:16 ` Arjun Roy
2021-11-15 19:02 ` [PATCH net-next 17/20] tcp: defer skb freeing after socket lock is released Eric Dumazet
2021-11-16 14:27 ` Jakub Kicinski
2021-11-16 15:05 ` Eric Dumazet
2021-11-16 15:20 ` Jakub Kicinski
2021-11-16 15:22 ` Eric Dumazet
2021-11-16 15:27 ` Jakub Kicinski
2021-11-16 16:46 ` Eric Dumazet [this message]
2021-11-16 18:18 ` Jakub Kicinski
2021-11-16 20:45 ` David Ahern
2021-11-16 21:35 ` Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 18/20] tcp: check local var (timeo) before socket fields in one test Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 19/20] tcp: do not call tcp_cleanup_rbuf() if we have a backlog Eric Dumazet
2021-11-15 19:02 ` [PATCH net-next 20/20] net: move early demux fields close to sk_refcnt Eric Dumazet
2021-11-15 20:37 ` [PATCH net-next 00/20] tcp: optimizations for linux-5.17 Soheil Hassas Yeganeh
2021-11-15 21:40 ` Paolo Abeni
2021-11-15 21:47 ` Eric Dumazet
2021-11-16 2:06 ` Eric Dumazet
2021-11-16 4:01 ` Arjun Roy
2021-11-16 13:32 ` David Miller
2021-11-16 15:06 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CANn89iJb7s-JoCCfn=eoxZ_tX_2RaeEPZKO1aHyHtgHxLXsd2A@mail.gmail.com' \
--to=edumazet@google.com \
--cc=arjunroy@google.com \
--cc=davem@davemloft.net \
--cc=eric.dumazet@gmail.com \
--cc=kuba@kernel.org \
--cc=ncardwell@google.com \
--cc=netdev@vger.kernel.org \
--cc=soheil@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.