From: Lorenz Bauer <lmb@cloudflare.com>
To: bjorn@kernel.org, Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>
Cc: kernel-team <kernel-team@cloudflare.com>, bpf <bpf@vger.kernel.org>
Subject: Concurrent BPF_PROG_TEST_RUN for XDP contend on dispatcher mutex
Date: Thu, 26 Aug 2021 12:08:53 +0100 [thread overview]
Message-ID: <CACAyw9_y4QumOW35qpgTbLsJ532uGq-kVW-VESJzGyiZkypnvw@mail.gmail.com> (raw)
Hi,
One of the tests for our XDP-based load balancer has gotten quite
slow, so I dug in. Roughly, it simulates 1m distinct packets arriving
at the load balancer by calling BPF_PROG_TEST_RUN a million times.
distribution_test.go:40: 1000000 iterations
distribution_test.go:99: Coefficient of variation: 0.52%
--- PASS: TestLoadBalancerDistribution (0.00s)
--- PASS: TestLoadBalancerDistribution/32_endpoints (22.04s)
You can see that the test takes 20s. Running the same test with slight
variations in three threads results in this:
distribution_test.go:40: 1000000 iterations
=== CONT TestLoadBalancerDistribution/32_endpoints
distribution_test.go:99: Coefficient of variation: 0.60%
=== CONT TestLoadBalancerDistribution/64_endpoints
distribution_test.go:99: Coefficient of variation: 0.82%
=== CONT TestLoadBalancerDistribution/128_endpoints
distribution_test.go:99: Coefficient of variation: 1.24%
--- PASS: TestLoadBalancerDistribution (0.00s)
--- PASS: TestLoadBalancerDistribution/32_endpoints (55.61s)
--- PASS: TestLoadBalancerDistribution/64_endpoints (55.61s)
--- PASS: TestLoadBalancerDistribution/128_endpoints (55.61s)
It's pretty clear that something is serialising the threads. Digging
around in perf reveals that the culprit is bpf_prog_change_xdp called
from bpf_prog_test_run_xdp. The call was added in f23c4b3924d2 ("bpf:
Start using the BPF dispatcher in BPF_TEST_RUN").
Is there something we can do about this? Maybe only call into the
dispatcher when repeat > 1?
Best
Lorenz
--
Lorenz Bauer | Systems Engineer
6th Floor, County Hall/The Riverside Building, SE1 7PB, UK
www.cloudflare.com
next reply other threads:[~2021-08-26 11:09 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-26 11:08 Lorenz Bauer [this message]
2021-08-26 19:39 ` Concurrent BPF_PROG_TEST_RUN for XDP contend on dispatcher mutex Alexei Starovoitov
2021-08-27 8:18 ` Lorenz Bauer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CACAyw9_y4QumOW35qpgTbLsJ532uGq-kVW-VESJzGyiZkypnvw@mail.gmail.com \
--to=lmb@cloudflare.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bjorn@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=kernel-team@cloudflare.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.