Re: libbpf/bpftool inconsistent handling og .data and .bss ?

[Andrii Nakryiko <andrii.nakryiko@gmail.com>]

On Sat, Oct 10, 2020 at 3:49 PM Luigi Rizzo <lrizzo@google.com> wrote:
>
> Coming back to .bss handling:
>
> On Wed, Oct 7, 2020 at 11:29 PM Luigi Rizzo <lrizzo@google.com> wrote:
> >
> > On Wed, Oct 7, 2020 at 10:40 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Wed, Oct 7, 2020 at 1:31 PM Luigi Rizzo <lrizzo@google.com> wrote:
> > > >
> > > > TL;DR; there seems to be a compiler bug with clang-10 and -O2
> > > > when struct are in .data -- details below.
> > > >
> > > > On Wed, Oct 7, 2020 at 8:35 PM Andrii Nakryiko
> > > > <andrii.nakryiko@gmail.com> wrote:
> > > > >
> > > > > On Wed, Oct 7, 2020 at 9:03 AM Luigi Rizzo <rizzo@iet.unipi.it> wrote:
> > > > ...
> > > > > > 2. .bss overrides from userspace are not seen in bpf at runtime
> ...
> > > > >
> > > > > This is quite surprising, given we have explicit selftests validating
> > > > > that all this works. And it seems to work. Please check
> > > > > prog_tests/skeleton.c and progs/test_skeleton.c. Can you try running
> > > > > it and confirm that it works in your setup?
> > > >
> > > > Ah, this was non intuitive but obvious in hindsight:
> > > >
> > > > .bss is zeroed by the kernel after load(), and since my program
> > > > changed the value before foo_bpf__load() , the memory was overwritten
> > > > with 0s. I could confirm this by printing the value after load.
> > > >
> > > > If I update obj->data-><something> after __load(),
> > > > or even after __attach() given that userspace mmaps .bss and .data,
> > > > everything works as expected both for scalars and structs.
> > >
> > > Check prog_tests/skeleton.c again, it sets .data, .bss, and .rodata
> > > before the load. And checks that those values are preserved after
> > > load. So .bss, if you initialize it manually, shouldn't zero-out what
> > > you set.
>
> strace reveals that the .bss is initially created as anonymous memory:
>
>   mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1,
> 0) = 0x7fd074a5f000
>   write(2, "after open bss is at 0x7fd074a5f"..., 36after open bss is
> at 0x7fd074a5f000) = 36
>
> and then remapped after the map has been created:
>   bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4,
> value_size=144,  max_entries=1, map_flags=0x400 /* BPF_F_??? */,
> inner_map_fd=0, map_name="hstats_b.bss", map_ifindex=0, ...}, 120) = 6
>   ...
>   mmap(0x7fd074a5f000, 4096, PROT_READ|PROT_WRITE,
> MAP_SHARED|MAP_FIXED, 6, 0) = 0x7fd074a5f000
>
> so the original content is gone.

not exactly, all of .bss, .rodata, .data and .kconfig are first
mmap()'ed as anonymous memory. I've modified test_skeleton.c to
increase .bss size to 8192 bytes size to distinguish it from other
maps:

1. mmap() anonymous memory (just allocating memory that would keep
initial values that you set with skel->bss->my_var = 123):

mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1,
0) = 0x7fb3b406f000

2. use that anonymous memory with initialized variables to update map
contents during bpf_object's load:

bpf(BPF_MAP_UPDATE_ELEM, {map_fd=7, key=0x7ffdab521d50,
value=0x7fb3b406f000, flags=BPF_ANY}, 120) = 0

3. now re-mmap() it with MAP_FIXED, so the same memory address is
pointing to ARRAY's content in the kernel. Because of
BPF_MAP_UPDATE_ELEM above, contents should be identical:

mmap(0x7fb3b406f000, 8192, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED,
7, 0) = 0x7fb3b406f000

4. later we are done with it:

munmap(0x7fb3b406f000, 8192)            = 0

>
> cheers
> luigi