From: Cong Wang <xiyou.wangcong@gmail.com>
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc: Linux Kernel Network Developers <netdev@vger.kernel.org>,
bpf <bpf@vger.kernel.org>,
Xiongchun Duan <duanxiongchun@bytedance.com>,
Dongdong Wang <wangdongdong.6@bytedance.com>,
Muchun Song <songmuchun@bytedance.com>,
Cong Wang <cong.wang@bytedance.com>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>,
Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
Yonghong Song <yhs@fb.com>,
Pedro Tammela <pctammela@mojatatu.com>,
Jamal Hadi Salim <jhs@mojatatu.com>, Joe Stringer <joe@cilium.io>
Subject: Re: [RFC Patch bpf-next] bpf: introduce bpf timer
Date: Sat, 8 May 2021 22:37:42 -0700 [thread overview]
Message-ID: <CAM_iQpWBrxuT=Y3CbhxYpE5a+QSk-O=Vj4euegggXAAKTHRBqw@mail.gmail.com> (raw)
In-Reply-To: <CAADnVQK9BgguVorziWgpMktLHuPCgEaKa4fz-KCfhcZtT46teQ@mail.gmail.com>
On Tue, Apr 27, 2021 at 11:34 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Tue, Apr 27, 2021 at 9:36 AM Cong Wang <xiyou.wangcong@gmail.com> wrote:
> >
> > If we enforce this ownership, in case of conntrack the owner would be
> > the program which sees the connection first, which is pretty much
> > unpredictable. For example, if the ingress program sees a connection
> > first, it installs a timer for this connection, but the traffic is
> > bidirectional,
> > hence egress program needs this connection and its timer too, we
> > should not remove this timer when the ingress program is freed.
>
> Sure. That's trivially achieved with pinning.
If users forget to do so, their ebpf program would crash the kernel,
right? But ebpf programs should never crash the kernel, right?
> One can have an ingress prog that tailcalls into another prog
> that arms the timer with one of its subprogs.
> Egress prog can tailcall into the same prog as well.
> The ingress and egress progs can be replaced one by one
> or removed both together and middle prog can stay alive
> if it's pinned in bpffs or held alive by FD.
This looks necessarily complex. Look at the overhead of using
a timer properly here:
1. pin timer callback program
2. a program to install timer
3. a program array contains the above program
4. a tail call into the above program array
Why not design a simpler solution?
>
> > From another point of view: maps and programs are both first-class
> > resources in eBPF, a timer is stored in a map and associated with a
> > program, so it is naturally a first-class resource too.
>
> Not really. The timer abstraction is about data. It invokes the callback.
> That callback is a part of the program. The lifetime of the timer object
> and lifetime of the callback can be different.
> Obviously the timer logic need to make sure that callback text is alive
> when the timer is armed.
Only if the callback could reference struct bpf_prog... And even if it
could, how about users forgetting to do so? ebpf verifier has to reject
such cases.
> Combining timer and callback concepts creates a messy abstraction.
> In the normal kernel code one can have a timer in any kernel data
> structure and callback in the kernel text or in the kernel module.
> The code needs to make sure that the module won't go away while
> the timer is armed. Same thing with bpf progs. The progs are safe
> kernel modules. The timers are independent objects.
Kernel modules can take reference count of its own module very
easily, plus there is no verifier for kernel modules. I don't understand
why you want to make ebpf programs as close to kernel modules as
possible in this case.
>
> > >
> > > > >
> > > > > Also if your colleagues have something to share they should be
> > > > > posting to the mailing list. Right now you're acting as a broken phone
> > > > > passing info back and forth and the knowledge gets lost.
> > > > > Please ask your colleagues to participate online.
> > > >
> > > > They are already in CC from the very beginning. And our use case is
> > > > public, it is Cilium conntrack:
> > > > https://github.com/cilium/cilium/blob/master/bpf/lib/conntrack.h
> > > >
> > > > The entries of the code are:
> > > > https://github.com/cilium/cilium/blob/master/bpf/bpf_lxc.c
> > > >
> > > > The maps for conntrack are:
> > > > https://github.com/cilium/cilium/blob/master/bpf/lib/conntrack_map.h
> > >
> > > If that's the only goal then kernel timers are not needed.
> > > cilium conntrack works well as-is.
> >
> > We don't go back to why user-space cleanup is inefficient again,
> > do we? ;)
>
> I remain unconvinced that cilium conntrack _needs_ timer apis.
> It works fine in production and I don't hear any complaints
> from cilium users. So 'user space cleanup inefficiencies' is
> very subjective and cannot be the reason to add timer apis.
I am pretty sure I showed the original report to you when I sent
timeout hashmap patch, in case you forgot here it is again:
https://github.com/cilium/cilium/issues/5048
and let me quote the original report here:
"The current implementation (as of v1.2) for managing the contents of
the datapath connection tracking map leaves something to be desired:
Once per minute, the userspace cilium-agent makes a series of calls to
the bpf() syscall to fetch all of the entries in the map to determine
whether they should be deleted. For each entry in the map, 2-3 calls
must be made: One to fetch the next key, one to fetch the value, and
perhaps one to delete the entry. The maximum size of the map is 1
million entries, and if the current count approaches this size then
the garbage collection goroutine may spend a significant number of CPU
cycles iterating and deleting elements from the conntrack map."
(Adding Joe in Cc too.)
Thanks.
next prev parent reply other threads:[~2021-05-09 5:38 UTC|newest]
Thread overview: 77+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-01 4:26 [RFC Patch bpf-next] bpf: introduce bpf timer Cong Wang
2021-04-01 6:38 ` Song Liu
2021-04-01 17:28 ` Cong Wang
2021-04-01 20:17 ` Song Liu
2021-04-02 17:34 ` Cong Wang
2021-04-02 17:57 ` Song Liu
2021-04-02 19:08 ` Cong Wang
2021-04-02 19:43 ` Song Liu
2021-04-02 20:57 ` Cong Wang
2021-04-02 23:31 ` Song Liu
2021-04-05 23:49 ` Cong Wang
2021-04-06 1:07 ` Song Liu
2021-04-06 1:24 ` Cong Wang
2021-04-06 6:17 ` Song Liu
2021-04-06 16:48 ` Cong Wang
2021-04-06 23:36 ` Song Liu
2021-04-08 22:45 ` Cong Wang
2021-04-02 19:28 ` Alexei Starovoitov
2021-04-02 21:24 ` Cong Wang
2021-04-02 23:45 ` Alexei Starovoitov
2021-04-06 0:36 ` Cong Wang
2021-04-12 23:01 ` Alexei Starovoitov
2021-04-15 4:02 ` Cong Wang
2021-04-15 4:25 ` Alexei Starovoitov
2021-04-15 15:51 ` Cong Wang
2021-04-26 23:00 ` Cong Wang
2021-04-26 23:05 ` Alexei Starovoitov
2021-04-26 23:37 ` Cong Wang
2021-04-27 2:01 ` Alexei Starovoitov
2021-04-27 11:52 ` Jamal Hadi Salim
2021-04-27 16:36 ` Cong Wang
2021-04-27 18:33 ` Alexei Starovoitov
2021-05-09 5:37 ` Cong Wang [this message]
2021-05-10 20:55 ` Jamal Hadi Salim
2021-05-11 21:29 ` Cong Wang
2021-05-12 22:56 ` Jamal Hadi Salim
2021-05-11 5:05 ` Joe Stringer
2021-05-11 21:08 ` Cong Wang
2021-05-12 22:43 ` Jamal Hadi Salim
2021-05-13 18:45 ` Jamal Hadi Salim
2021-05-14 2:53 ` Cong Wang
2021-08-11 21:03 ` Joe Stringer
2021-05-20 18:55 [RFC PATCH bpf-next] bpf: Introduce bpf_timer Alexei Starovoitov
2021-05-21 14:38 ` Alexei Starovoitov
2021-05-21 21:37 ` Cong Wang
2021-05-23 16:01 ` Alexei Starovoitov
2021-05-24 8:45 ` Lorenz Bauer
2021-05-25 3:16 ` Cong Wang
2021-05-25 4:59 ` Cong Wang
2021-05-25 18:21 ` Alexei Starovoitov
2021-05-25 19:35 ` Jamal Hadi Salim
2021-05-25 19:57 ` Alexei Starovoitov
2021-05-25 21:09 ` Jamal Hadi Salim
2021-05-25 22:08 ` Alexei Starovoitov
2021-05-26 15:34 ` Jamal Hadi Salim
2021-05-26 16:58 ` Alexei Starovoitov
2021-05-26 18:25 ` Jamal Hadi Salim
2021-05-30 6:36 ` Cong Wang
2021-06-02 2:00 ` Alexei Starovoitov
2021-06-02 8:48 ` Toke Høiland-Jørgensen
2021-06-02 17:54 ` Martin KaFai Lau
2021-06-02 18:13 ` Kumar Kartikeya Dwivedi
2021-06-02 18:26 ` Alexei Starovoitov
2021-06-02 18:30 ` Kumar Kartikeya Dwivedi
2021-06-02 18:46 ` John Fastabend
2021-05-23 11:48 ` Toke Høiland-Jørgensen
2021-05-23 15:58 ` Alexei Starovoitov
2021-05-24 8:42 ` Lorenz Bauer
2021-05-24 14:48 ` Alexei Starovoitov
2021-05-24 17:33 ` Alexei Starovoitov
2021-05-24 18:39 ` Toke Høiland-Jørgensen
2021-05-24 18:38 ` Toke Høiland-Jørgensen
2021-05-24 11:49 ` Lorenz Bauer
2021-05-24 14:56 ` Alexei Starovoitov
2021-05-24 19:13 ` Andrii Nakryiko
2021-05-25 5:22 ` Cong Wang
2021-05-25 19:47 ` Andrii Nakryiko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAM_iQpWBrxuT=Y3CbhxYpE5a+QSk-O=Vj4euegggXAAKTHRBqw@mail.gmail.com' \
--to=xiyou.wangcong@gmail.com \
--cc=alexei.starovoitov@gmail.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=cong.wang@bytedance.com \
--cc=daniel@iogearbox.net \
--cc=duanxiongchun@bytedance.com \
--cc=jhs@mojatatu.com \
--cc=joe@cilium.io \
--cc=kafai@fb.com \
--cc=netdev@vger.kernel.org \
--cc=pctammela@mojatatu.com \
--cc=songliubraving@fb.com \
--cc=songmuchun@bytedance.com \
--cc=wangdongdong.6@bytedance.com \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).