* Are BPF tail calls only supposed to work with pinned maps?
@ 2019-09-26 11:23 Toke Høiland-Jørgensen
2019-09-26 12:53 ` Daniel Borkmann
0 siblings, 1 reply; 5+ messages in thread
From: Toke Høiland-Jørgensen @ 2019-09-26 11:23 UTC (permalink / raw)
To: Daniel Borkmann; +Cc: netdev, bpf
Hi Daniel
While working on a prototype of the XDP chain call feature, I ran into
some strange behaviour with tail calls: If I create a userspace program
that loads two XDP programs, one of which tail calls the other, the tail
call map would appear to be empty even though the userspace program
populates it as part of the program loading.
I eventually tracked this down to this commit:
c9da161c6517 ("bpf: fix clearing on persistent program array maps")
Which clears PROG_ARRAY maps whenever the last uref to it disappears
(which it does when my loader exits after attaching the XDP program).
This effectively means that tail calls only work if the PROG_ARRAY map
is pinned (or the process creating it keeps running). And as far as I
can tell, the inner_map reference in bpf_map_fd_get_ptr() doesn't bump
the uref either, so presumably if one were to create a map-in-map
construct with tail call pointer in the inner map(s), each inner map
would also need to be pinned (haven't tested this case)?
Is this really how things are supposed to work? From an XDP use case PoV
this seems somewhat surprising...
Or am I missing something obvious here?
-Toke
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Are BPF tail calls only supposed to work with pinned maps? 2019-09-26 11:23 Are BPF tail calls only supposed to work with pinned maps? Toke Høiland-Jørgensen @ 2019-09-26 12:53 ` Daniel Borkmann 2019-09-26 13:12 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 5+ messages in thread From: Daniel Borkmann @ 2019-09-26 12:53 UTC (permalink / raw) To: Toke Høiland-Jørgensen; +Cc: netdev, bpf Hi Toke, On Thu, Sep 26, 2019 at 01:23:38PM +0200, Toke Høiland-Jørgensen wrote: [...] > While working on a prototype of the XDP chain call feature, I ran into > some strange behaviour with tail calls: If I create a userspace program > that loads two XDP programs, one of which tail calls the other, the tail > call map would appear to be empty even though the userspace program > populates it as part of the program loading. > > I eventually tracked this down to this commit: > c9da161c6517 ("bpf: fix clearing on persistent program array maps") Correct. > Which clears PROG_ARRAY maps whenever the last uref to it disappears > (which it does when my loader exits after attaching the XDP program). > > This effectively means that tail calls only work if the PROG_ARRAY map > is pinned (or the process creating it keeps running). And as far as I > can tell, the inner_map reference in bpf_map_fd_get_ptr() doesn't bump > the uref either, so presumably if one were to create a map-in-map > construct with tail call pointer in the inner map(s), each inner map > would also need to be pinned (haven't tested this case)? There is no map in map support for tail calls today. > Is this really how things are supposed to work? From an XDP use case PoV > this seems somewhat surprising... > > Or am I missing something obvious here? The way it was done like this back then was in order to break up cyclic dependencies as otherwise the programs and maps involved would never get freed as they reference themselves and live on in the kernel forever consuming potentially large amount of resources, so orchestration tools like Cilium typically just pin the maps in bpf fs (like most other maps it uses and accesses from agent side) in order to up/downgrade the agent while keeping BPF datapath intact. Thanks, Daniel ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Are BPF tail calls only supposed to work with pinned maps? 2019-09-26 12:53 ` Daniel Borkmann @ 2019-09-26 13:12 ` Toke Høiland-Jørgensen 2019-09-26 18:14 ` Daniel Borkmann 0 siblings, 1 reply; 5+ messages in thread From: Toke Høiland-Jørgensen @ 2019-09-26 13:12 UTC (permalink / raw) To: Daniel Borkmann; +Cc: netdev, bpf Daniel Borkmann <daniel@iogearbox.net> writes: > Hi Toke, > > On Thu, Sep 26, 2019 at 01:23:38PM +0200, Toke Høiland-Jørgensen wrote: > [...] >> While working on a prototype of the XDP chain call feature, I ran into >> some strange behaviour with tail calls: If I create a userspace program >> that loads two XDP programs, one of which tail calls the other, the tail >> call map would appear to be empty even though the userspace program >> populates it as part of the program loading. >> >> I eventually tracked this down to this commit: >> c9da161c6517 ("bpf: fix clearing on persistent program array maps") > > Correct. > >> Which clears PROG_ARRAY maps whenever the last uref to it disappears >> (which it does when my loader exits after attaching the XDP program). >> >> This effectively means that tail calls only work if the PROG_ARRAY map >> is pinned (or the process creating it keeps running). And as far as I >> can tell, the inner_map reference in bpf_map_fd_get_ptr() doesn't bump >> the uref either, so presumably if one were to create a map-in-map >> construct with tail call pointer in the inner map(s), each inner map >> would also need to be pinned (haven't tested this case)? > > There is no map in map support for tail calls today. Not directly, but can't a program do: tail_call_map = bpf_map_lookup(outer_map, key); bpf_tail_call(tail_call_map, idx); >> Is this really how things are supposed to work? From an XDP use case PoV >> this seems somewhat surprising... >> >> Or am I missing something obvious here? > > The way it was done like this back then was in order to break up cyclic > dependencies as otherwise the programs and maps involved would never get > freed as they reference themselves and live on in the kernel forever > consuming potentially large amount of resources, so orchestration tools > like Cilium typically just pin the maps in bpf fs (like most other maps > it uses and accesses from agent side) in order to up/downgrade the agent > while keeping BPF datapath intact. Right. I can see how the cyclic reference thing gets thorny otherwise. However, the behaviour was somewhat surprising to me; is it documented anywhere? I think I'll probably end up creating a new map type for chaining programs anyway, so this is not a huge show-stopper for me; but it had me scratching my head for a while there... ;) -Toke ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Are BPF tail calls only supposed to work with pinned maps? 2019-09-26 13:12 ` Toke Høiland-Jørgensen @ 2019-09-26 18:14 ` Daniel Borkmann 2019-09-27 7:27 ` Toke Høiland-Jørgensen 0 siblings, 1 reply; 5+ messages in thread From: Daniel Borkmann @ 2019-09-26 18:14 UTC (permalink / raw) To: Toke Høiland-Jørgensen; +Cc: netdev, bpf On Thu, Sep 26, 2019 at 03:12:30PM +0200, Toke Høiland-Jørgensen wrote: > Daniel Borkmann <daniel@iogearbox.net> writes: > > On Thu, Sep 26, 2019 at 01:23:38PM +0200, Toke Høiland-Jørgensen wrote: > > [...] > >> While working on a prototype of the XDP chain call feature, I ran into > >> some strange behaviour with tail calls: If I create a userspace program > >> that loads two XDP programs, one of which tail calls the other, the tail > >> call map would appear to be empty even though the userspace program > >> populates it as part of the program loading. > >> > >> I eventually tracked this down to this commit: > >> c9da161c6517 ("bpf: fix clearing on persistent program array maps") > > > > Correct. > > > >> Which clears PROG_ARRAY maps whenever the last uref to it disappears > >> (which it does when my loader exits after attaching the XDP program). > >> > >> This effectively means that tail calls only work if the PROG_ARRAY map > >> is pinned (or the process creating it keeps running). And as far as I > >> can tell, the inner_map reference in bpf_map_fd_get_ptr() doesn't bump > >> the uref either, so presumably if one were to create a map-in-map > >> construct with tail call pointer in the inner map(s), each inner map > >> would also need to be pinned (haven't tested this case)? > > > > There is no map in map support for tail calls today. > > Not directly, but can't a program do: > > tail_call_map = bpf_map_lookup(outer_map, key); > bpf_tail_call(tail_call_map, idx); Nope, that is what I meant, bpf_map_meta_alloc() will bail out in that case. > >> Is this really how things are supposed to work? From an XDP use case PoV > >> this seems somewhat surprising... > >> > >> Or am I missing something obvious here? > > > > The way it was done like this back then was in order to break up cyclic > > dependencies as otherwise the programs and maps involved would never get > > freed as they reference themselves and live on in the kernel forever > > consuming potentially large amount of resources, so orchestration tools > > like Cilium typically just pin the maps in bpf fs (like most other maps > > it uses and accesses from agent side) in order to up/downgrade the agent > > while keeping BPF datapath intact. > > Right. I can see how the cyclic reference thing gets thorny otherwise. > However, the behaviour was somewhat surprising to me; is it documented > anywhere? Haven't updated the BPF guide in a while [0], I don't think I documented this detail back then, so right now only in the git log. Improvements to the reference guide definitely welcome. Thanks, Daniel [0] https://cilium.readthedocs.io/en/latest/bpf/ https://github.com/cilium/cilium/blob/master/Documentation/bpf.rst ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Are BPF tail calls only supposed to work with pinned maps? 2019-09-26 18:14 ` Daniel Borkmann @ 2019-09-27 7:27 ` Toke Høiland-Jørgensen 0 siblings, 0 replies; 5+ messages in thread From: Toke Høiland-Jørgensen @ 2019-09-27 7:27 UTC (permalink / raw) To: Daniel Borkmann; +Cc: netdev, bpf Daniel Borkmann <daniel@iogearbox.net> writes: > On Thu, Sep 26, 2019 at 03:12:30PM +0200, Toke Høiland-Jørgensen wrote: >> Daniel Borkmann <daniel@iogearbox.net> writes: >> > On Thu, Sep 26, 2019 at 01:23:38PM +0200, Toke Høiland-Jørgensen wrote: >> > [...] >> >> While working on a prototype of the XDP chain call feature, I ran into >> >> some strange behaviour with tail calls: If I create a userspace program >> >> that loads two XDP programs, one of which tail calls the other, the tail >> >> call map would appear to be empty even though the userspace program >> >> populates it as part of the program loading. >> >> >> >> I eventually tracked this down to this commit: >> >> c9da161c6517 ("bpf: fix clearing on persistent program array maps") >> > >> > Correct. >> > >> >> Which clears PROG_ARRAY maps whenever the last uref to it disappears >> >> (which it does when my loader exits after attaching the XDP program). >> >> >> >> This effectively means that tail calls only work if the PROG_ARRAY map >> >> is pinned (or the process creating it keeps running). And as far as I >> >> can tell, the inner_map reference in bpf_map_fd_get_ptr() doesn't bump >> >> the uref either, so presumably if one were to create a map-in-map >> >> construct with tail call pointer in the inner map(s), each inner map >> >> would also need to be pinned (haven't tested this case)? >> > >> > There is no map in map support for tail calls today. >> >> Not directly, but can't a program do: >> >> tail_call_map = bpf_map_lookup(outer_map, key); >> bpf_tail_call(tail_call_map, idx); > > Nope, that is what I meant, bpf_map_meta_alloc() will bail out in that > case. Oohhh, right. Seems I reversed that if statement in my head. Silly me, thanks for clarifying! >> >> Is this really how things are supposed to work? From an XDP use case PoV >> >> this seems somewhat surprising... >> >> >> >> Or am I missing something obvious here? >> > >> > The way it was done like this back then was in order to break up cyclic >> > dependencies as otherwise the programs and maps involved would never get >> > freed as they reference themselves and live on in the kernel forever >> > consuming potentially large amount of resources, so orchestration tools >> > like Cilium typically just pin the maps in bpf fs (like most other maps >> > it uses and accesses from agent side) in order to up/downgrade the agent >> > while keeping BPF datapath intact. >> >> Right. I can see how the cyclic reference thing gets thorny otherwise. >> However, the behaviour was somewhat surprising to me; is it documented >> anywhere? > > Haven't updated the BPF guide in a while [0], I don't think I > documented this detail back then, so right now only in the git log. > Improvements to the reference guide definitely welcome. Gotcha. I guess we should add something about tail calls (and chain calls once we get them) to the XDP tutorial as well... -Toke ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-09-27 7:27 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-09-26 11:23 Are BPF tail calls only supposed to work with pinned maps? Toke Høiland-Jørgensen 2019-09-26 12:53 ` Daniel Borkmann 2019-09-26 13:12 ` Toke Høiland-Jørgensen 2019-09-26 18:14 ` Daniel Borkmann 2019-09-27 7:27 ` Toke Høiland-Jørgensen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).