All of lore.kernel.org
 help / color / mirror / Atom feed
From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
To: "Samudrala, Sridhar" <sridhar.samudrala@intel.com>
Cc: "Björn Töpel" <bjorn.topel@gmail.com>,
	"Toke Høiland-Jørgensen" <toke@redhat.com>,
	"Jakub Kicinski" <jakub.kicinski@netronome.com>,
	Netdev <netdev@vger.kernel.org>,
	intel-wired-lan <intel-wired-lan@lists.osuosl.org>,
	"Herbert, Tom" <tom.herbert@intel.com>,
	"Fijalkowski, Maciej" <maciej.fijalkowski@intel.com>,
	"bpf@vger.kernel.org" <bpf@vger.kernel.org>,
	"Björn Töpel" <bjorn.topel@intel.com>,
	"Karlsson, Magnus" <magnus.karlsson@intel.com>
Subject: Re: [Intel-wired-lan] FW: [PATCH bpf-next 2/4] xsk: allow AF_XDP sockets to receive packets directly from a queue
Date: Wed, 23 Oct 2019 10:42:00 -0700	[thread overview]
Message-ID: <CAADnVQKwnMChzeGaC66A99cHn5szB4hPZaGXq8JAhd8sjrdGeA@mail.gmail.com> (raw)
In-Reply-To: <8956a643-0163-5345-34fa-3566762a2b7d@intel.com>

On Tue, Oct 22, 2019 at 12:06 PM Samudrala, Sridhar
<sridhar.samudrala@intel.com> wrote:
>
> OK. Here is another data point that shows the perf report with the same test but CPU mitigations
> turned OFF. Here bpf_prog overhead goes down from almost (10.18 + 4.51)% to (3.23 + 1.44%).
>
>    21.40%  ksoftirqd/28     [i40e]                     [k] i40e_clean_rx_irq_zc
>    14.13%  xdpsock          [i40e]                     [k] i40e_clean_rx_irq_zc
>     8.33%  ksoftirqd/28     [kernel.vmlinux]           [k] xsk_rcv
>     6.09%  ksoftirqd/28     [kernel.vmlinux]           [k] xdp_do_redirect
>     5.19%  xdpsock          xdpsock                    [.] main
>     3.48%  ksoftirqd/28     [kernel.vmlinux]           [k] bpf_xdp_redirect_map
>     3.23%  ksoftirqd/28     bpf_prog_3c8251c7e0fef8db  [k] bpf_prog_3c8251c7e0fef8db
>
> So a major component of the bpf_prog overhead seems to be due to the CPU vulnerability mitigations.

I feel that it's an incorrect conclusion because JIT is not doing
any retpolines (because there are no indirect calls in bpf).
There should be no difference in bpf_prog runtime with or without mitigations.
Also you're running root, so no spectre mitigations either.

This 3% seems like a lot for a function that does few loads that should
hit d-cache and one direct call.
Please investigate why you're seeing this 10% cpu cost when mitigations are on.
perf report/annotate is the best.
Also please double check that you're using the latest perf.
Since bpf performance analysis was greatly improved several versions ago.
I don't think old perf will be showing bogus numbers, but better to
run the latest.

> The other component is the bpf_xdp_redirect_map() codepath.
>
> Let me know if it helps to collect any other data that should further help with the perf analysis.
>

WARNING: multiple messages have this Message-ID (diff)
From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
To: intel-wired-lan@osuosl.org
Subject: [Intel-wired-lan] FW: [PATCH bpf-next 2/4] xsk: allow AF_XDP sockets to receive packets directly from a queue
Date: Wed, 23 Oct 2019 10:42:00 -0700	[thread overview]
Message-ID: <CAADnVQKwnMChzeGaC66A99cHn5szB4hPZaGXq8JAhd8sjrdGeA@mail.gmail.com> (raw)
In-Reply-To: <8956a643-0163-5345-34fa-3566762a2b7d@intel.com>

On Tue, Oct 22, 2019 at 12:06 PM Samudrala, Sridhar
<sridhar.samudrala@intel.com> wrote:
>
> OK. Here is another data point that shows the perf report with the same test but CPU mitigations
> turned OFF. Here bpf_prog overhead goes down from almost (10.18 + 4.51)% to (3.23 + 1.44%).
>
>    21.40%  ksoftirqd/28     [i40e]                     [k] i40e_clean_rx_irq_zc
>    14.13%  xdpsock          [i40e]                     [k] i40e_clean_rx_irq_zc
>     8.33%  ksoftirqd/28     [kernel.vmlinux]           [k] xsk_rcv
>     6.09%  ksoftirqd/28     [kernel.vmlinux]           [k] xdp_do_redirect
>     5.19%  xdpsock          xdpsock                    [.] main
>     3.48%  ksoftirqd/28     [kernel.vmlinux]           [k] bpf_xdp_redirect_map
>     3.23%  ksoftirqd/28     bpf_prog_3c8251c7e0fef8db  [k] bpf_prog_3c8251c7e0fef8db
>
> So a major component of the bpf_prog overhead seems to be due to the CPU vulnerability mitigations.

I feel that it's an incorrect conclusion because JIT is not doing
any retpolines (because there are no indirect calls in bpf).
There should be no difference in bpf_prog runtime with or without mitigations.
Also you're running root, so no spectre mitigations either.

This 3% seems like a lot for a function that does few loads that should
hit d-cache and one direct call.
Please investigate why you're seeing this 10% cpu cost when mitigations are on.
perf report/annotate is the best.
Also please double check that you're using the latest perf.
Since bpf performance analysis was greatly improved several versions ago.
I don't think old perf will be showing bogus numbers, but better to
run the latest.

> The other component is the bpf_xdp_redirect_map() codepath.
>
> Let me know if it helps to collect any other data that should further help with the perf analysis.
>

  reply	other threads:[~2019-10-23 17:42 UTC|newest]

Thread overview: 84+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-08  6:16 [PATCH bpf-next 0/4] Enable direct receive on AF_XDP sockets Sridhar Samudrala
2019-10-08  6:16 ` [Intel-wired-lan] " Sridhar Samudrala
2019-10-08  6:16 ` [PATCH bpf-next 1/4] bpf: introduce bpf_get_prog_id and bpf_set_prog_id helper functions Sridhar Samudrala
2019-10-08  6:16   ` [Intel-wired-lan] " Sridhar Samudrala
2019-10-08  6:16 ` [PATCH bpf-next 2/4] xsk: allow AF_XDP sockets to receive packets directly from a queue Sridhar Samudrala
2019-10-08  6:16   ` [Intel-wired-lan] " Sridhar Samudrala
2019-10-08  6:58   ` Toke Høiland-Jørgensen
2019-10-08  6:58     ` [Intel-wired-lan] " Toke =?unknown-8bit?q?H=C3=B8iland-J=C3=B8rgensen?=
2019-10-08  8:47     ` Björn Töpel
2019-10-08  8:47       ` [Intel-wired-lan] " =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=
2019-10-08  8:48       ` Björn Töpel
2019-10-08  8:48         ` [Intel-wired-lan] " =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=
2019-10-08  9:04       ` Toke Høiland-Jørgensen
2019-10-08  9:04         ` [Intel-wired-lan] " Toke =?unknown-8bit?q?H=C3=B8iland-J=C3=B8rgensen?=
2019-10-08  8:05   ` Björn Töpel
2019-10-08  8:05     ` [Intel-wired-lan] " =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=
2019-10-09 16:32     ` Samudrala, Sridhar
2019-10-09 16:32       ` [Intel-wired-lan] " Samudrala, Sridhar
2019-10-09  1:20   ` Alexei Starovoitov
2019-10-09  1:20     ` [Intel-wired-lan] " Alexei Starovoitov
     [not found]     ` <3ED8E928C4210A4289A677D2FEB48235140134CE@fmsmsx111.amr.corp.intel.com>
2019-10-09 16:53       ` FW: " Samudrala, Sridhar
2019-10-09 16:53         ` [Intel-wired-lan] " Samudrala, Sridhar
2019-10-09 17:17         ` Alexei Starovoitov
2019-10-09 17:17           ` [Intel-wired-lan] " Alexei Starovoitov
2019-10-09 19:12           ` Samudrala, Sridhar
2019-10-09 19:12             ` [Intel-wired-lan] " Samudrala, Sridhar
2019-10-10  1:06             ` Alexei Starovoitov
2019-10-10  1:06               ` [Intel-wired-lan] " Alexei Starovoitov
2019-10-18 18:40               ` Samudrala, Sridhar
2019-10-18 18:40                 ` [Intel-wired-lan] " Samudrala, Sridhar
2019-10-18 19:22                 ` Toke Høiland-Jørgensen
2019-10-18 19:22                   ` [Intel-wired-lan] " Toke =?unknown-8bit?q?H=C3=B8iland-J=C3=B8rgensen?=
2019-10-19  0:14                 ` Alexei Starovoitov
2019-10-19  0:14                   ` [Intel-wired-lan] " Alexei Starovoitov
2019-10-19  0:45                   ` Samudrala, Sridhar
2019-10-19  0:45                     ` [Intel-wired-lan] " Samudrala, Sridhar
2019-10-19  2:25                     ` Alexei Starovoitov
2019-10-19  2:25                       ` [Intel-wired-lan] " Alexei Starovoitov
2019-10-20 10:14                       ` Toke Høiland-Jørgensen
2019-10-20 10:14                         ` [Intel-wired-lan] " Toke =?unknown-8bit?q?H=C3=B8iland-J=C3=B8rgensen?=
2019-10-20 17:12                         ` Björn Töpel
2019-10-20 17:12                           ` =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=
2019-10-21 20:10                           ` Samudrala, Sridhar
2019-10-21 20:10                             ` Samudrala, Sridhar
2019-10-21 22:34                             ` Alexei Starovoitov
2019-10-21 22:34                               ` Alexei Starovoitov
2019-10-22 19:06                               ` Samudrala, Sridhar
2019-10-22 19:06                                 ` Samudrala, Sridhar
2019-10-23 17:42                                 ` Alexei Starovoitov [this message]
2019-10-23 17:42                                   ` Alexei Starovoitov
2019-10-24 18:12                                   ` Samudrala, Sridhar
2019-10-24 18:12                                     ` Samudrala, Sridhar
2019-10-25  7:42                                     ` Björn Töpel
2019-10-25  7:42                                       ` =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=
2019-10-31 22:38                                       ` Samudrala, Sridhar
2019-10-31 22:38                                         ` Samudrala, Sridhar
2019-10-31 23:15                                         ` Alexei Starovoitov
2019-10-31 23:15                                           ` Alexei Starovoitov
2019-11-01  0:21                                         ` Jakub Kicinski
2019-11-01  0:21                                           ` Jakub Kicinski
2019-11-01 18:31                                           ` Samudrala, Sridhar
2019-11-01 18:31                                             ` Samudrala, Sridhar
2019-11-04  2:08                                           ` dan
2019-11-04  2:08                                             ` dan
2019-10-25  9:07                                   ` Björn Töpel
2019-10-25  9:07                                     ` =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=
2019-10-08  6:16 ` [PATCH bpf-next 3/4] libbpf: handle AF_XDP sockets created with XDP_DIRECT bind flag Sridhar Samudrala
2019-10-08  6:16   ` [Intel-wired-lan] " Sridhar Samudrala
2019-10-08  8:05   ` Björn Töpel
2019-10-08  8:05     ` [Intel-wired-lan] " =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=
2019-10-08  6:16 ` [PATCH bpf-next 4/4] xdpsock: add an option to create AF_XDP sockets in XDP_DIRECT mode Sridhar Samudrala
2019-10-08  6:16   ` [Intel-wired-lan] " Sridhar Samudrala
2019-10-08  8:05   ` Björn Töpel
2019-10-08  8:05     ` [Intel-wired-lan] " =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=
2019-10-08  8:05 ` [PATCH bpf-next 0/4] Enable direct receive on AF_XDP sockets Björn Töpel
2019-10-08  8:05   ` [Intel-wired-lan] " =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?=
2019-10-09 16:19   ` Samudrala, Sridhar
2019-10-09 16:19     ` [Intel-wired-lan] " Samudrala, Sridhar
2019-10-09  0:49 ` Jakub Kicinski
2019-10-09  0:49   ` [Intel-wired-lan] " Jakub Kicinski
2019-10-09  6:29   ` Samudrala, Sridhar
2019-10-09  6:29     ` [Intel-wired-lan] " Samudrala, Sridhar
2019-10-09 16:53     ` Jakub Kicinski
2019-10-09 16:53       ` [Intel-wired-lan] " Jakub Kicinski

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAADnVQKwnMChzeGaC66A99cHn5szB4hPZaGXq8JAhd8sjrdGeA@mail.gmail.com \
    --to=alexei.starovoitov@gmail.com \
    --cc=bjorn.topel@gmail.com \
    --cc=bjorn.topel@intel.com \
    --cc=bpf@vger.kernel.org \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=jakub.kicinski@netronome.com \
    --cc=maciej.fijalkowski@intel.com \
    --cc=magnus.karlsson@intel.com \
    --cc=netdev@vger.kernel.org \
    --cc=sridhar.samudrala@intel.com \
    --cc=toke@redhat.com \
    --cc=tom.herbert@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.