From: "Hao Xiang ." <hao.xiang@bytedance.com>
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc: "Ho-Ren (Jack) Chuang" <horenchuang@bytedance.com>,
Alexei Starovoitov <ast@kernel.org>, Hao Luo <haoluo@google.com>,
Jiri Olsa <jolsa@kernel.org>, Jiri Olsa <olsajiri@gmail.com>,
Andrii Nakryiko <andrii@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
John Fastabend <john.fastabend@gmail.com>,
Martin KaFai Lau <martin.lau@linux.dev>,
Song Liu <song@kernel.org>, Yonghong Song <yhs@fb.com>,
KP Singh <kpsingh@kernel.org>,
Stanislav Fomichev <sdf@google.com>,
Quentin Monnet <quentin@isovalent.com>,
Mykola Lysenko <mykolal@fb.com>, Shuah Khan <shuah@kernel.org>,
Nathan Chancellor <nathan@kernel.org>,
Nick Desaulniers <ndesaulniers@google.com>,
Tom Rix <trix@redhat.com>, Joanne Koong <joannelkoong@gmail.com>,
Kui-Feng Lee <kuifeng@fb.com>,
Lorenzo Bianconi <lorenzo@kernel.org>,
Maxim Mikityanskiy <maximmi@nvidia.com>,
Punit Agrawal <punit.agrawal@bytedance.com>,
Yifei Ma <yifeima@bytedance.com>,
Xiaoning Ding <xiaoning.ding@bytedance.com>,
bpf <bpf@vger.kernel.org>, Ho-Ren Chuang <horenc@vt.edu>,
LKML <linux-kernel@vger.kernel.org>,
"open list:KERNEL SELFTEST FRAMEWORK"
<linux-kselftest@vger.kernel.org>,
clang-built-linux <llvm@lists.linux.dev>
Subject: Re: [External] Re: [PATCH bpf-next v1 0/4] Add BPF htab map's used size for monitoring
Date: Mon, 28 Nov 2022 15:03:05 -0800 [thread overview]
Message-ID: <CAAYibXgiCOOEY9NvLXbY4ve7pH8xWrZjnczrj6SHy3x_TtOU1g@mail.gmail.com> (raw)
In-Reply-To: <CAAYibXiHRg3C3tk=wbiHdaUgJD6AfJf7gX3w0gTJ2nsJ=DnY4g@mail.gmail.com>
Hi Alexei, we can use the existing switch bpf_stats_enabled around the
added overhead. The switch is turned off by default so I believe there
will be no extra overhead once we do that. Can you please have a
second thought on this?
On Mon, Nov 7, 2022 at 4:30 PM Hao Xiang . <hao.xiang@bytedance.com> wrote:
>
> Hi Alexei,
>
> We understand the concern on added performance overhead. We had some
> discussion about this while working on the patch and decided to give
> it a try (my bad).
>
> Adding some more context. We are leveraging the BPF_OBJ_GET_INFO_BY_FD
> syscall to trace CPU usage per prog and memory usage per map. We would
> like to use this patch to add an interface for map types to return its
> internal "count". For instance, we are thinking of having the below
> map types to report the "count" and those won't add overhead to the
> hot path.
> 1. ringbuf to return its "count" by calculating the distance between
> producer_pos and consumer_pos
> 2. queue and stack to return its "count" from the head's position
> 3. dev map hash to returns its "count" from items
>
> There are other map types, similar to the hashtab pre-allocation case,
> will introduce overhead in the hot path in order to count the stats. I
> think we can find alternative solutions for those (eg, iterate the map
> and count, count only if bpf_stats_enabled switch is on, etc). There
> are cases where this can't be done at the application level because
> applications don't see the internal stats in order to do the right
> counting.
>
> We can remove the counting for the pre-allocated case in this patch.
> Please let us know what you think.
>
> Thanks, Hao
>
> On Sat, Nov 5, 2022 at 9:20 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Fri, Nov 4, 2022 at 7:52 PM Ho-Ren (Jack) Chuang
> > <horenchuang@bytedance.com> wrote:
> > >
> > > Hello everyone,
> > >
> > > We have prepared patches to address an issue from a previous discussion.
> > > The previous discussion email thread is here: https://lore.kernel.org/all/CAADnVQLBt0snxv4bKwg1WKQ9wDFbaDCtZ03v1-LjOTYtsKPckQ@mail.gmail.com/
> >
> > Rephrasing what was said earlier.
> > We're not keeping the count of elements in a preallocated hash map
> > and we are not going to add one.
> > The bpf prog needs to do the accounting on its own if it needs
> > this kind of statistics.
> > Keeping the count for non-prealloc is already significant performance
> > overhead. We don't trade performance for stats.
prev parent reply other threads:[~2022-11-28 23:03 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-05 2:51 [PATCH bpf-next v1 0/4] Add BPF htab map's used size for monitoring Ho-Ren (Jack) Chuang
2022-11-05 2:51 ` [PATCH bpf-next v1 1/4] bpf: Support reporting " Ho-Ren (Jack) Chuang
2022-11-05 2:51 ` [PATCH bpf-next v1 2/4] bpftool: Add tools support to show BPF htab map's used size Ho-Ren (Jack) Chuang
2022-11-05 2:51 ` [PATCH bpf-next v1 3/4] samples/bpf: Add concurrency testing for " Ho-Ren (Jack) Chuang
2022-11-05 2:51 ` [PATCH bpf-next v1 4/4] selftests/bpf: Add unit tests " Ho-Ren (Jack) Chuang
2022-11-05 16:20 ` [PATCH bpf-next v1 0/4] Add BPF htab map's used size for monitoring Alexei Starovoitov
2022-11-08 0:30 ` [External] " Hao Xiang .
2022-11-28 23:03 ` Hao Xiang . [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAAYibXgiCOOEY9NvLXbY4ve7pH8xWrZjnczrj6SHy3x_TtOU1g@mail.gmail.com \
--to=hao.xiang@bytedance.com \
--cc=alexei.starovoitov@gmail.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=haoluo@google.com \
--cc=horenc@vt.edu \
--cc=horenchuang@bytedance.com \
--cc=joannelkoong@gmail.com \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=kpsingh@kernel.org \
--cc=kuifeng@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=llvm@lists.linux.dev \
--cc=lorenzo@kernel.org \
--cc=martin.lau@linux.dev \
--cc=maximmi@nvidia.com \
--cc=mykolal@fb.com \
--cc=nathan@kernel.org \
--cc=ndesaulniers@google.com \
--cc=olsajiri@gmail.com \
--cc=punit.agrawal@bytedance.com \
--cc=quentin@isovalent.com \
--cc=sdf@google.com \
--cc=shuah@kernel.org \
--cc=song@kernel.org \
--cc=trix@redhat.com \
--cc=xiaoning.ding@bytedance.com \
--cc=yhs@fb.com \
--cc=yifeima@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).