From: Martin KaFai Lau <kafai@fb.com>
To: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Joanne Koong <joannekoong@fb.com>, bpf <bpf@vger.kernel.org>,
Kernel Team <Kernel-team@fb.com>
Subject: Re: [PATCH v3 bpf-next 1/5] bpf: Add bloom filter map implementation
Date: Wed, 22 Sep 2021 15:08:44 -0700 [thread overview]
Message-ID: <20210922220844.ihzoapwytaz2o7nn@kafai-mbp.dhcp.thefacebook.com> (raw)
In-Reply-To: <CAEf4BzYi3VXdMctKVFsDqG+_nDTSGooJ2sSkF1FuKkqDKqc82g@mail.gmail.com>
On Wed, Sep 22, 2021 at 01:52:12PM -0700, Andrii Nakryiko wrote:
> > Agree that a generic hash helper is in general useful. It may be
> > useful in hashing the skb also. The bpf prog only implementation could
> > have more flexibility in configuring roundup to pow2 or not, how to hash,
> > how many hashes, nr of bits ...etc. In the mean time, the bpf prog and
>
> Exactly. If I know better how many bits I need, I'll have to reverse
> engineer kernel's heuristic to provide such max_entries values to
> arrive at the desired amount of memory that Bloom filter will be
> using.
Good point. I don't think it needs to guess. The formula is stable
and publicly known also. The formula comment from kernel/bpf/bloom_filter.c
should be moved to the include/uapi/linux/bpf.h.
> > user space need to co-ordinate more and worry about more things,
> > e.g. how to reuse a bloom filter with different nr_hashes,
> > nr_bits, handle synchronization...etc.
>
> Please see my RFC ([0]). I don't think there is much to coordinate. It
> could be purely BPF-side code, or BPF + user-space initialization
> code, depending on the need. It's a simple and beautiful algorithm,
> which BPF is powerful enough to implement customly and easily.
>
> [0] https://lore.kernel.org/bpf/20210922203224.912809-1-andrii@kernel.org/T/#t
In practice, the bloom filter will be populated only once by the userspace.
The future update will be done by map-in-map to replace the whole bloom filter.
May be with more max_entries with more nr_hashes. May be fewer
max_entries with fewer nr_hashes.
Currently, the continuous running bpf prog using this bloom filter does
not need to worry about any change in the newer bloom filter
configure/setup.
I wonder how that may look like in the custom bpf bloom filter in the
bench prog for the map-in-map usage.
>
> >
> > It is useful to have a default implementation in the kernel
> > for some useful maps like this one that works for most
> > common cases and the bpf user can just use it as get-and-go
> > like all other common bpf maps do.
>
> I disagree with the premise that Bloom filter is a common and
> generally useful data structure, tbh. It has its nice niche
> applications, but its semantics isn't applicable generally, which is
> why I hesitate to claim that this should live in kernel.
I don't agree the application is nice niche. I have encountered this
many times when bumping into networking usecase discussion and not
necessary limited to security usage also. Yes, it is not a link-list
like data structure but its usage is very common.
next prev parent reply other threads:[~2021-09-22 22:08 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-21 21:02 [PATCH v3 bpf-next 0/5] Implement bloom filter map Joanne Koong
2021-09-21 21:02 ` [PATCH v3 bpf-next 1/5] bpf: Add bloom filter map implementation Joanne Koong
2021-09-21 23:44 ` Andrii Nakryiko
2021-09-22 19:06 ` Joanne Koong
2021-09-22 19:38 ` Martin KaFai Lau
2021-09-22 20:52 ` Andrii Nakryiko
2021-09-22 22:08 ` Martin KaFai Lau [this message]
2021-09-22 23:07 ` Andrii Nakryiko
2021-09-23 1:28 ` Martin KaFai Lau
2021-09-23 18:42 ` Andrii Nakryiko
2021-09-23 19:42 ` Martin KaFai Lau
2021-09-23 20:30 ` Alexei Starovoitov
2021-09-23 21:12 ` Andrii Nakryiko
2021-09-23 22:28 ` Joanne Koong
2021-09-23 23:46 ` Martin KaFai Lau
2021-09-24 2:23 ` Andrii Nakryiko
2021-09-24 16:32 ` Joanne Koong
2021-09-24 23:12 ` Andrii Nakryiko
2021-09-27 16:41 ` Alexei Starovoitov
2021-09-27 21:14 ` Andrii Nakryiko
2021-09-27 23:51 ` Alexei Starovoitov
2021-09-28 0:36 ` Andrii Nakryiko
2021-09-28 16:21 ` Alexei Starovoitov
[not found] ` <aa967ed2-a958-f995-3a09-bbd6b6e775d4@fb.com>
2021-09-28 23:54 ` Andrii Nakryiko
2021-09-29 1:54 ` Joanne Koong
2021-09-29 0:14 ` Andrii Nakryiko
2021-09-29 3:17 ` Alexei Starovoitov
2021-09-29 3:38 ` Joanne Koong
2021-09-28 1:09 ` Martin KaFai Lau
2021-09-22 20:44 ` Andrii Nakryiko
2021-09-21 21:02 ` [PATCH v3 bpf-next 2/5] libbpf: Allow the number of hashes in bloom filter maps to be configurable Joanne Koong
2021-09-21 22:24 ` Joanne Koong
2021-09-22 23:14 ` Andrii Nakryiko
2021-09-21 23:57 ` Andrii Nakryiko
2021-09-21 21:02 ` [PATCH v3 bpf-next 3/5] selftests/bpf: Add bloom filter map test cases Joanne Koong
2021-09-21 21:02 ` [PATCH v3 bpf-next 4/5] bpf/benchs: Add benchmark test for bloom filter maps Joanne Koong
2021-09-21 21:02 ` [PATCH v3 bpf-next 5/5] bpf/benchs: Add benchmarks for comparing hashmap lookups with vs. without bloom filter Joanne Koong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210922220844.ihzoapwytaz2o7nn@kafai-mbp.dhcp.thefacebook.com \
--to=kafai@fb.com \
--cc=Kernel-team@fb.com \
--cc=andrii.nakryiko@gmail.com \
--cc=bpf@vger.kernel.org \
--cc=joannekoong@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).