* can't get BTF: type .rodata.cst32: not found @ 2022-01-25 17:38 Vincent Li 2022-01-25 17:52 ` Vincent Li 0 siblings, 1 reply; 21+ messages in thread From: Vincent Li @ 2022-01-25 17:38 UTC (permalink / raw) To: bpf Hi While developing Cilium VTEP integration feature https://github.com/cilium/cilium/pull/17370, I found a strange issue that seems related to BTF and probably caused by my specific implementation, the issue is described in https://github.com/cilium/cilium/issues/18616, I don't know much about BTF and not sure if my implementation is seriously flawed or just some implementation bug or maybe not compatible with BTF. Strangely, the issue appears related to number of VTEPs I use, no problem with 1 or 2 VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF experts are appreciated :-). Thanks Vincent ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-01-25 17:38 can't get BTF: type .rodata.cst32: not found Vincent Li @ 2022-01-25 17:52 ` Vincent Li 2022-01-25 20:32 ` Vincent Li 0 siblings, 1 reply; 21+ messages in thread From: Vincent Li @ 2022-01-25 17:52 UTC (permalink / raw) To: bpf this is macro I suspected in my implementation that could cause issue with BTF #define ENABLE_VTEP 1 #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, 0x2048a90a, } #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, 0x5eaaed93fdf2, 0x5faaed93fdf2, } #define VTEP_NUMS 4 On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > > Hi > > While developing Cilium VTEP integration feature > https://github.com/cilium/cilium/pull/17370, I found a strange issue > that seems related to BTF and probably caused by my specific > implementation, the issue is described in > https://github.com/cilium/cilium/issues/18616, I don't know much about > BTF and not sure if my implementation is seriously flawed or just some > implementation bug or maybe not compatible with BTF. Strangely, the > issue appears related to number of VTEPs I use, no problem with 1 or 2 > VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF > experts are appreciated :-). > > Thanks > > Vincent ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-01-25 17:52 ` Vincent Li @ 2022-01-25 20:32 ` Vincent Li 2022-01-28 1:50 ` Yonghong Song 0 siblings, 1 reply; 21+ messages in thread From: Vincent Li @ 2022-01-25 20:32 UTC (permalink / raw) To: bpf On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > > this is macro I suspected in my implementation that could cause issue with BTF > > #define ENABLE_VTEP 1 > #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > 0x2048a90a, } > #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, > 0x5eaaed93fdf2, 0x5faaed93fdf2, } > #define VTEP_NUMS 4 > > On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > > > > Hi > > > > While developing Cilium VTEP integration feature > > https://github.com/cilium/cilium/pull/17370, I found a strange issue > > that seems related to BTF and probably caused by my specific > > implementation, the issue is described in > > https://github.com/cilium/cilium/issues/18616, I don't know much about > > BTF and not sure if my implementation is seriously flawed or just some > > implementation bug or maybe not compatible with BTF. Strangely, the > > issue appears related to number of VTEPs I use, no problem with 1 or 2 > > VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF > > experts are appreciated :-). > > > > Thanks > > > > Vincent Sorry for previous top post it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o differently and added " [21] .rodata.cst32 PROGBITS 0000000000000000 00011e68" when following macro exceeded 2 members #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, 0x2048a90a, } no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT member <=2. any reason why compiler would do that? ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-01-25 20:32 ` Vincent Li @ 2022-01-28 1:50 ` Yonghong Song 2022-01-28 18:27 ` Vincent Li 0 siblings, 1 reply; 21+ messages in thread From: Yonghong Song @ 2022-01-28 1:50 UTC (permalink / raw) To: Vincent Li, bpf On 1/25/22 12:32 PM, Vincent Li wrote: > On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >> >> this is macro I suspected in my implementation that could cause issue with BTF >> >> #define ENABLE_VTEP 1 >> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >> 0x2048a90a, } >> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, >> 0x5eaaed93fdf2, 0x5faaed93fdf2, } >> #define VTEP_NUMS 4 >> >> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>> >>> Hi >>> >>> While developing Cilium VTEP integration feature >>> https://github.com/cilium/cilium/pull/17370, I found a strange issue >>> that seems related to BTF and probably caused by my specific >>> implementation, the issue is described in >>> https://github.com/cilium/cilium/issues/18616, I don't know much about >>> BTF and not sure if my implementation is seriously flawed or just some >>> implementation bug or maybe not compatible with BTF. Strangely, the >>> issue appears related to number of VTEPs I use, no problem with 1 or 2 >>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF >>> experts are appreciated :-). >>> >>> Thanks >>> >>> Vincent > > Sorry for previous top post > > it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o > differently and added " [21] .rodata.cst32 PROGBITS > 0000000000000000 00011e68" when following macro exceeded 2 members > > #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > 0x2048a90a, } > > no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT > member <=2. any reason why compiler would do that? Regarding to why compiler generates .rodata.cst32, the reason is you have some 32-byte constants which needs to be saved somewhere. For example, $ cat t.c struct t { long c[2]; int d[4]; }; struct t g; int test() { struct t tmp = {.c = {1, 2}, .d = {3, 4}}; g = tmp; return 0; } $ clang -target bpf -O2 -c t.c $ llvm-readelf -S t.o ... [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 20 AM 0 0 8 ... In the above code, if you change the struct size, say from 32 bytes to 40 bytes, the rodata.cst32 will go away. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-01-28 1:50 ` Yonghong Song @ 2022-01-28 18:27 ` Vincent Li 2022-02-01 18:07 ` Vincent Li 0 siblings, 1 reply; 21+ messages in thread From: Vincent Li @ 2022-01-28 18:27 UTC (permalink / raw) To: Yonghong Song; +Cc: bpf On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: > > > > On 1/25/22 12:32 PM, Vincent Li wrote: > > On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >> > >> this is macro I suspected in my implementation that could cause issue with BTF > >> > >> #define ENABLE_VTEP 1 > >> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > >> 0x2048a90a, } > >> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, > >> 0x5eaaed93fdf2, 0x5faaed93fdf2, } > >> #define VTEP_NUMS 4 > >> > >> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>> > >>> Hi > >>> > >>> While developing Cilium VTEP integration feature > >>> https://github.com/cilium/cilium/pull/17370, I found a strange issue > >>> that seems related to BTF and probably caused by my specific > >>> implementation, the issue is described in > >>> https://github.com/cilium/cilium/issues/18616, I don't know much about > >>> BTF and not sure if my implementation is seriously flawed or just some > >>> implementation bug or maybe not compatible with BTF. Strangely, the > >>> issue appears related to number of VTEPs I use, no problem with 1 or 2 > >>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF > >>> experts are appreciated :-). > >>> > >>> Thanks > >>> > >>> Vincent > > > > Sorry for previous top post > > > > it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o > > differently and added " [21] .rodata.cst32 PROGBITS > > 0000000000000000 00011e68" when following macro exceeded 2 members > > > > #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > > 0x2048a90a, } > > > > no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT > > member <=2. any reason why compiler would do that? > > Regarding to why compiler generates .rodata.cst32, the reason is > you have some 32-byte constants which needs to be saved somewhere. > For example, > > $ cat t.c > struct t { > long c[2]; > int d[4]; > }; > struct t g; > int test() > { > struct t tmp = {.c = {1, 2}, .d = {3, 4}}; > g = tmp; > return 0; > } > > $ clang -target bpf -O2 -c t.c > $ llvm-readelf -S t.o > ... > [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 > 20 AM 0 0 8 > ... > > In the above code, if you change the struct size, say from 32 bytes to > 40 bytes, the rodata.cst32 will go away. Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then Vincent ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-01-28 18:27 ` Vincent Li @ 2022-02-01 18:07 ` Vincent Li 2022-02-02 7:17 ` Yonghong Song 0 siblings, 1 reply; 21+ messages in thread From: Vincent Li @ 2022-02-01 18:07 UTC (permalink / raw) To: Yonghong Song; +Cc: bpf On Fri, Jan 28, 2022 at 10:27 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > > On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: > > > > > > > > On 1/25/22 12:32 PM, Vincent Li wrote: > > > On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > > >> > > >> this is macro I suspected in my implementation that could cause issue with BTF > > >> > > >> #define ENABLE_VTEP 1 > > >> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > > >> 0x2048a90a, } > > >> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, > > >> 0x5eaaed93fdf2, 0x5faaed93fdf2, } > > >> #define VTEP_NUMS 4 > > >> > > >> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > > >>> > > >>> Hi > > >>> > > >>> While developing Cilium VTEP integration feature > > >>> https://github.com/cilium/cilium/pull/17370, I found a strange issue > > >>> that seems related to BTF and probably caused by my specific > > >>> implementation, the issue is described in > > >>> https://github.com/cilium/cilium/issues/18616, I don't know much about > > >>> BTF and not sure if my implementation is seriously flawed or just some > > >>> implementation bug or maybe not compatible with BTF. Strangely, the > > >>> issue appears related to number of VTEPs I use, no problem with 1 or 2 > > >>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF > > >>> experts are appreciated :-). > > >>> > > >>> Thanks > > >>> > > >>> Vincent > > > > > > Sorry for previous top post > > > > > > it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o > > > differently and added " [21] .rodata.cst32 PROGBITS > > > 0000000000000000 00011e68" when following macro exceeded 2 members > > > > > > #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > > > 0x2048a90a, } > > > > > > no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT > > > member <=2. any reason why compiler would do that? > > > > Regarding to why compiler generates .rodata.cst32, the reason is > > you have some 32-byte constants which needs to be saved somewhere. > > For example, > > > > $ cat t.c > > struct t { > > long c[2]; > > int d[4]; > > }; > > struct t g; > > int test() > > { > > struct t tmp = {.c = {1, 2}, .d = {3, 4}}; > > g = tmp; > > return 0; > > } > > > > $ clang -target bpf -O2 -c t.c > > $ llvm-readelf -S t.o > > ... > > [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 > > 20 AM 0 0 8 > > ... > > > > In the above code, if you change the struct size, say from 32 bytes to > > 40 bytes, the rodata.cst32 will go away. > > Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then Hi Yonghong, Here is a follow-up question, it looks cilium/ebpf parse vmlinux and stores BTF type info in btf.Spec.namedTypes, but the elf object file provided by user may have section like rodata.cst32 generated by compiler that does not have accompanying BTF type info stored in btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be guaranteed to have every BTF type info from application/user provided elf object file ? I guess there is no guarantee. Vincent ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-01 18:07 ` Vincent Li @ 2022-02-02 7:17 ` Yonghong Song 2022-02-02 13:47 ` Timo Beckers 0 siblings, 1 reply; 21+ messages in thread From: Yonghong Song @ 2022-02-02 7:17 UTC (permalink / raw) To: Vincent Li; +Cc: bpf On 2/1/22 10:07 AM, Vincent Li wrote: > On Fri, Jan 28, 2022 at 10:27 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >> >> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: >>> >>> >>> >>> On 1/25/22 12:32 PM, Vincent Li wrote: >>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>> >>>>> this is macro I suspected in my implementation that could cause issue with BTF >>>>> >>>>> #define ENABLE_VTEP 1 >>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>> 0x2048a90a, } >>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, >>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } >>>>> #define VTEP_NUMS 4 >>>>> >>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>> >>>>>> Hi >>>>>> >>>>>> While developing Cilium VTEP integration feature >>>>>> https://github.com/cilium/cilium/pull/17370, I found a strange issue >>>>>> that seems related to BTF and probably caused by my specific >>>>>> implementation, the issue is described in >>>>>> https://github.com/cilium/cilium/issues/18616, I don't know much about >>>>>> BTF and not sure if my implementation is seriously flawed or just some >>>>>> implementation bug or maybe not compatible with BTF. Strangely, the >>>>>> issue appears related to number of VTEPs I use, no problem with 1 or 2 >>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF >>>>>> experts are appreciated :-). >>>>>> >>>>>> Thanks >>>>>> >>>>>> Vincent >>>> >>>> Sorry for previous top post >>>> >>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o >>>> differently and added " [21] .rodata.cst32 PROGBITS >>>> 0000000000000000 00011e68" when following macro exceeded 2 members >>>> >>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>> 0x2048a90a, } >>>> >>>> no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT >>>> member <=2. any reason why compiler would do that? >>> >>> Regarding to why compiler generates .rodata.cst32, the reason is >>> you have some 32-byte constants which needs to be saved somewhere. >>> For example, >>> >>> $ cat t.c >>> struct t { >>> long c[2]; >>> int d[4]; >>> }; >>> struct t g; >>> int test() >>> { >>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; >>> g = tmp; >>> return 0; >>> } >>> >>> $ clang -target bpf -O2 -c t.c >>> $ llvm-readelf -S t.o >>> ... >>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 >>> 20 AM 0 0 8 >>> ... >>> >>> In the above code, if you change the struct size, say from 32 bytes to >>> 40 bytes, the rodata.cst32 will go away. >> >> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then > > Hi Yonghong, > > Here is a follow-up question, it looks cilium/ebpf parse vmlinux and > stores BTF type info in btf.Spec.namedTypes, but the elf object file > provided by user may have section like rodata.cst32 generated by > compiler that does not have accompanying BTF type info stored in > btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be > guaranteed to have every BTF type info from application/user provided > elf object file ? I guess there is no guarantee. vmlinux holds kernel types. rodata.cst32 holds data. If the type of rodata.cst32 needs to be emitted, the type will be encoded in bpf program BTF. Did you actually find an issue with .rodata.cst32 section? Such a section is typically generated by the compiler for initial data inside the function and llvm bpf backend tries to inline the values through a bunch of load instructions. So even you see .rodata.cst32, typically you can safely ignore it. > > Vincent ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-02 7:17 ` Yonghong Song @ 2022-02-02 13:47 ` Timo Beckers 2022-02-02 19:55 ` Andrii Nakryiko 2022-02-03 2:11 ` Yonghong Song 0 siblings, 2 replies; 21+ messages in thread From: Timo Beckers @ 2022-02-02 13:47 UTC (permalink / raw) To: Yonghong Song, Vincent Li; +Cc: bpf On 2/2/22 08:17, Yonghong Song wrote: > > > On 2/1/22 10:07 AM, Vincent Li wrote: >> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>> >>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: >>>> >>>> >>>> >>>> On 1/25/22 12:32 PM, Vincent Li wrote: >>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>> >>>>>> this is macro I suspected in my implementation that could cause issue with BTF >>>>>> >>>>>> #define ENABLE_VTEP 1 >>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>>> 0x2048a90a, } >>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, >>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } >>>>>> #define VTEP_NUMS 4 >>>>>> >>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>>> >>>>>>> Hi >>>>>>> >>>>>>> While developing Cilium VTEP integration feature >>>>>>> https://github.com/cilium/cilium/pull/17370, I found a strange issue >>>>>>> that seems related to BTF and probably caused by my specific >>>>>>> implementation, the issue is described in >>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know much about >>>>>>> BTF and not sure if my implementation is seriously flawed or just some >>>>>>> implementation bug or maybe not compatible with BTF. Strangely, the >>>>>>> issue appears related to number of VTEPs I use, no problem with 1 or 2 >>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF >>>>>>> experts are appreciated :-). >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Vincent >>>>> >>>>> Sorry for previous top post >>>>> >>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o >>>>> differently and added " [21] .rodata.cst32 PROGBITS >>>>> 0000000000000000 00011e68" when following macro exceeded 2 members >>>>> >>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>> 0x2048a90a, } >>>>> >>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT >>>>> member <=2. any reason why compiler would do that? >>>> >>>> Regarding to why compiler generates .rodata.cst32, the reason is >>>> you have some 32-byte constants which needs to be saved somewhere. >>>> For example, >>>> >>>> $ cat t.c >>>> struct t { >>>> long c[2]; >>>> int d[4]; >>>> }; >>>> struct t g; >>>> int test() >>>> { >>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; >>>> g = tmp; >>>> return 0; >>>> } >>>> >>>> $ clang -target bpf -O2 -c t.c >>>> $ llvm-readelf -S t.o >>>> ... >>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 >>>> 20 AM 0 0 8 >>>> ... >>>> >>>> In the above code, if you change the struct size, say from 32 bytes to >>>> 40 bytes, the rodata.cst32 will go away. >>> >>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then >> >> Hi Yonghong, >> >> Here is a follow-up question, it looks cilium/ebpf parse vmlinux and >> stores BTF type info in btf.Spec.namedTypes, but the elf object file >> provided by user may have section like rodata.cst32 generated by >> compiler that does not have accompanying BTF type info stored in >> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be >> guaranteed to have every BTF type info from application/user provided >> elf object file ? I guess there is no guarantee. > > vmlinux holds kernel types. rodata.cst32 holds data. If the type of > rodata.cst32 needs to be emitted, the type will be encoded in bpf > program BTF. > > Did you actually find an issue with .rodata.cst32 section? Such a > section is typically generated by the compiler for initial data > inside the function and llvm bpf backend tries to inline the > values through a bunch of load instructions. So even you see > .rodata.cst32, typically you can safely ignore it. > >> >> Vincent > Hi Yonghong, Thanks for the reproducer. Couldn't figure out what to do with .rodata.cst32, since there are no symbols and no BTF info for that section. The values found in .rodata.cst32 are indeed inlined in the bytecode as you mentioned, so it seems like we can ignore it. Why does the compiler emit these sections? cilium/ebpf assumed up until now that all sections starting with '.rodata' are datasecs and must be loaded into the kernel, which of course needs accompanying BTF. What other .rodata.* should we expect? Thanks, Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-02 13:47 ` Timo Beckers @ 2022-02-02 19:55 ` Andrii Nakryiko 2022-02-03 2:11 ` Yonghong Song 1 sibling, 0 replies; 21+ messages in thread From: Andrii Nakryiko @ 2022-02-02 19:55 UTC (permalink / raw) To: Timo Beckers; +Cc: Yonghong Song, Vincent Li, bpf On Wed, Feb 2, 2022 at 9:59 AM Timo Beckers <timo@incline.eu> wrote: > > On 2/2/22 08:17, Yonghong Song wrote: > > > > > > On 2/1/22 10:07 AM, Vincent Li wrote: > >> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>> > >>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: > >>>> > >>>> > >>>> > >>>> On 1/25/22 12:32 PM, Vincent Li wrote: > >>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>>>>> > >>>>>> this is macro I suspected in my implementation that could cause issue with BTF > >>>>>> > >>>>>> #define ENABLE_VTEP 1 > >>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > >>>>>> 0x2048a90a, } > >>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, > >>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } > >>>>>> #define VTEP_NUMS 4 > >>>>>> > >>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>>>>>> > >>>>>>> Hi > >>>>>>> > >>>>>>> While developing Cilium VTEP integration feature > >>>>>>> https://github.com/cilium/cilium/pull/17370, I found a strange issue > >>>>>>> that seems related to BTF and probably caused by my specific > >>>>>>> implementation, the issue is described in > >>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know much about > >>>>>>> BTF and not sure if my implementation is seriously flawed or just some > >>>>>>> implementation bug or maybe not compatible with BTF. Strangely, the > >>>>>>> issue appears related to number of VTEPs I use, no problem with 1 or 2 > >>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF > >>>>>>> experts are appreciated :-). > >>>>>>> > >>>>>>> Thanks > >>>>>>> > >>>>>>> Vincent > >>>>> > >>>>> Sorry for previous top post > >>>>> > >>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o > >>>>> differently and added " [21] .rodata.cst32 PROGBITS > >>>>> 0000000000000000 00011e68" when following macro exceeded 2 members > >>>>> > >>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > >>>>> 0x2048a90a, } > >>>>> > >>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT > >>>>> member <=2. any reason why compiler would do that? > >>>> > >>>> Regarding to why compiler generates .rodata.cst32, the reason is > >>>> you have some 32-byte constants which needs to be saved somewhere. > >>>> For example, > >>>> > >>>> $ cat t.c > >>>> struct t { > >>>> long c[2]; > >>>> int d[4]; > >>>> }; > >>>> struct t g; > >>>> int test() > >>>> { > >>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; > >>>> g = tmp; > >>>> return 0; > >>>> } > >>>> > >>>> $ clang -target bpf -O2 -c t.c > >>>> $ llvm-readelf -S t.o > >>>> ... > >>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 > >>>> 20 AM 0 0 8 > >>>> ... > >>>> > >>>> In the above code, if you change the struct size, say from 32 bytes to > >>>> 40 bytes, the rodata.cst32 will go away. > >>> > >>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then > >> > >> Hi Yonghong, > >> > >> Here is a follow-up question, it looks cilium/ebpf parse vmlinux and > >> stores BTF type info in btf.Spec.namedTypes, but the elf object file > >> provided by user may have section like rodata.cst32 generated by > >> compiler that does not have accompanying BTF type info stored in > >> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be > >> guaranteed to have every BTF type info from application/user provided > >> elf object file ? I guess there is no guarantee. > > > > vmlinux holds kernel types. rodata.cst32 holds data. If the type of > > rodata.cst32 needs to be emitted, the type will be encoded in bpf > > program BTF. > > > > Did you actually find an issue with .rodata.cst32 section? Such a > > section is typically generated by the compiler for initial data > > inside the function and llvm bpf backend tries to inline the > > values through a bunch of load instructions. So even you see > > .rodata.cst32, typically you can safely ignore it. > > > >> > >> Vincent > > > > Hi Yonghong, > > Thanks for the reproducer. Couldn't figure out what to do with .rodata.cst32, > since there are no symbols and no BTF info for that section. > > The values found in .rodata.cst32 are indeed inlined in the bytecode as you > mentioned, so it seems like we can ignore it. > > Why does the compiler emit these sections? cilium/ebpf assumed up until now > that all sections starting with '.rodata' are datasecs and must be loaded into > the kernel, which of course needs accompanying BTF. It doesn't need an accompanying BTF. libbpf loads all .rodata.* sections (unless BPF program code doesn't reference it at all, which is mostly for backwards compatibility with old kernels that don't support global variables) and it all works good. > > What other .rodata.* should we expect? > I'd suggest instead of whitelisting all possible .rodata.* section names just allow any such data section. So that users can have their own custom .rodata (and .data) sections as well. > Thanks, > > Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-02 13:47 ` Timo Beckers 2022-02-02 19:55 ` Andrii Nakryiko @ 2022-02-03 2:11 ` Yonghong Song 2022-02-04 11:11 ` Timo Beckers 1 sibling, 1 reply; 21+ messages in thread From: Yonghong Song @ 2022-02-03 2:11 UTC (permalink / raw) To: Timo Beckers, Vincent Li; +Cc: bpf On 2/2/22 5:47 AM, Timo Beckers wrote: > On 2/2/22 08:17, Yonghong Song wrote: >> >> >> On 2/1/22 10:07 AM, Vincent Li wrote: >>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>> >>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: >>>>> >>>>> >>>>> >>>>> On 1/25/22 12:32 PM, Vincent Li wrote: >>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>>> >>>>>>> this is macro I suspected in my implementation that could cause issue with BTF >>>>>>> >>>>>>> #define ENABLE_VTEP 1 >>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>>>> 0x2048a90a, } >>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, >>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } >>>>>>> #define VTEP_NUMS 4 >>>>>>> >>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> While developing Cilium VTEP integration feature >>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a strange issue >>>>>>>> that seems related to BTF and probably caused by my specific >>>>>>>> implementation, the issue is described in >>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know much about >>>>>>>> BTF and not sure if my implementation is seriously flawed or just some >>>>>>>> implementation bug or maybe not compatible with BTF. Strangely, the >>>>>>>> issue appears related to number of VTEPs I use, no problem with 1 or 2 >>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF >>>>>>>> experts are appreciated :-). >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Vincent >>>>>> >>>>>> Sorry for previous top post >>>>>> >>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o >>>>>> differently and added " [21] .rodata.cst32 PROGBITS >>>>>> 0000000000000000 00011e68" when following macro exceeded 2 members >>>>>> >>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>>> 0x2048a90a, } >>>>>> >>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT >>>>>> member <=2. any reason why compiler would do that? >>>>> >>>>> Regarding to why compiler generates .rodata.cst32, the reason is >>>>> you have some 32-byte constants which needs to be saved somewhere. >>>>> For example, >>>>> >>>>> $ cat t.c >>>>> struct t { >>>>> long c[2]; >>>>> int d[4]; >>>>> }; >>>>> struct t g; >>>>> int test() >>>>> { >>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; >>>>> g = tmp; >>>>> return 0; >>>>> } >>>>> >>>>> $ clang -target bpf -O2 -c t.c >>>>> $ llvm-readelf -S t.o >>>>> ... >>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 >>>>> 20 AM 0 0 8 >>>>> ... >>>>> >>>>> In the above code, if you change the struct size, say from 32 bytes to >>>>> 40 bytes, the rodata.cst32 will go away. >>>> >>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then >>> >>> Hi Yonghong, >>> >>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux and >>> stores BTF type info in btf.Spec.namedTypes, but the elf object file >>> provided by user may have section like rodata.cst32 generated by >>> compiler that does not have accompanying BTF type info stored in >>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be >>> guaranteed to have every BTF type info from application/user provided >>> elf object file ? I guess there is no guarantee. >> >> vmlinux holds kernel types. rodata.cst32 holds data. If the type of >> rodata.cst32 needs to be emitted, the type will be encoded in bpf >> program BTF. >> >> Did you actually find an issue with .rodata.cst32 section? Such a >> section is typically generated by the compiler for initial data >> inside the function and llvm bpf backend tries to inline the >> values through a bunch of load instructions. So even you see >> .rodata.cst32, typically you can safely ignore it. >> >>> >>> Vincent >> > > Hi Yonghong, > > Thanks for the reproducer. Couldn't figure out what to do with .rodata.cst32, > since there are no symbols and no BTF info for that section. > > The values found in .rodata.cst32 are indeed inlined in the bytecode as you > mentioned, so it seems like we can ignore it. > > Why does the compiler emit these sections? cilium/ebpf assumed up until now > that all sections starting with '.rodata' are datasecs and must be loaded into > the kernel, which of course needs accompanying BTF. The clang frontend emits these .rodata.* sections. In early days, kernel doesn't support global data so llvm bpf backend implements an optimization to inline these values. But llvm bpf backend didn't completely remove them as the backend doesn't have a global view whether these .rodata.* are being used in other places or not. Now, llvm bpf backend has better infrastructure and we probably can implement an IR pass to detect all uses of .rodata.*, inline these uses, and remove the .rodata.* global variable. You can check relocation section of the program text. If the .rodata.* section is referenced, you should preserve it. Otherwise, you can ignore that .rodata.* section. > > What other .rodata.* should we expect? Glancing through llvm code, you may see .rodata.{4,8,16,32}, .rodata.str*. > > Thanks, > > Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-03 2:11 ` Yonghong Song @ 2022-02-04 11:11 ` Timo Beckers 2022-02-04 18:04 ` Yonghong Song 0 siblings, 1 reply; 21+ messages in thread From: Timo Beckers @ 2022-02-04 11:11 UTC (permalink / raw) To: Yonghong Song, Vincent Li; +Cc: bpf On 2/3/22 03:11, Yonghong Song wrote: > > > On 2/2/22 5:47 AM, Timo Beckers wrote: >> On 2/2/22 08:17, Yonghong Song wrote: >>> >>> >>> On 2/1/22 10:07 AM, Vincent Li wrote: >>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>> >>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: >>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>>>> >>>>>>>> this is macro I suspected in my implementation that could cause issue with BTF >>>>>>>> >>>>>>>> #define ENABLE_VTEP 1 >>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>>>>> 0x2048a90a, } >>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, >>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } >>>>>>>> #define VTEP_NUMS 4 >>>>>>>> >>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> While developing Cilium VTEP integration feature >>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a strange issue >>>>>>>>> that seems related to BTF and probably caused by my specific >>>>>>>>> implementation, the issue is described in >>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know much about >>>>>>>>> BTF and not sure if my implementation is seriously flawed or just some >>>>>>>>> implementation bug or maybe not compatible with BTF. Strangely, the >>>>>>>>> issue appears related to number of VTEPs I use, no problem with 1 or 2 >>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF >>>>>>>>> experts are appreciated :-). >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Vincent >>>>>>> >>>>>>> Sorry for previous top post >>>>>>> >>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o >>>>>>> differently and added " [21] .rodata.cst32 PROGBITS >>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 members >>>>>>> >>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>>>> 0x2048a90a, } >>>>>>> >>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT >>>>>>> member <=2. any reason why compiler would do that? >>>>>> >>>>>> Regarding to why compiler generates .rodata.cst32, the reason is >>>>>> you have some 32-byte constants which needs to be saved somewhere. >>>>>> For example, >>>>>> >>>>>> $ cat t.c >>>>>> struct t { >>>>>> long c[2]; >>>>>> int d[4]; >>>>>> }; >>>>>> struct t g; >>>>>> int test() >>>>>> { >>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; >>>>>> g = tmp; >>>>>> return 0; >>>>>> } >>>>>> >>>>>> $ clang -target bpf -O2 -c t.c >>>>>> $ llvm-readelf -S t.o >>>>>> ... >>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 >>>>>> 20 AM 0 0 8 >>>>>> ... >>>>>> >>>>>> In the above code, if you change the struct size, say from 32 bytes to >>>>>> 40 bytes, the rodata.cst32 will go away. >>>>> >>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then >>>> >>>> Hi Yonghong, >>>> >>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux and >>>> stores BTF type info in btf.Spec.namedTypes, but the elf object file >>>> provided by user may have section like rodata.cst32 generated by >>>> compiler that does not have accompanying BTF type info stored in >>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be >>>> guaranteed to have every BTF type info from application/user provided >>>> elf object file ? I guess there is no guarantee. >>> >>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of >>> rodata.cst32 needs to be emitted, the type will be encoded in bpf >>> program BTF. >>> >>> Did you actually find an issue with .rodata.cst32 section? Such a >>> section is typically generated by the compiler for initial data >>> inside the function and llvm bpf backend tries to inline the >>> values through a bunch of load instructions. So even you see >>> .rodata.cst32, typically you can safely ignore it. >>> >>>> >>>> Vincent >>> >> >> Hi Yonghong, >> >> Thanks for the reproducer. Couldn't figure out what to do with .rodata.cst32, >> since there are no symbols and no BTF info for that section. >> >> The values found in .rodata.cst32 are indeed inlined in the bytecode as you >> mentioned, so it seems like we can ignore it. >> >> Why does the compiler emit these sections? cilium/ebpf assumed up until now >> that all sections starting with '.rodata' are datasecs and must be loaded into >> the kernel, which of course needs accompanying BTF. > > The clang frontend emits these .rodata.* sections. In early days, kernel > doesn't support global data so llvm bpf backend implements an > optimization to inline these values. But llvm bpf backend didn't completely remove them as the backend doesn't have a global view > whether these .rodata.* are being used in other places or not. > > Now, llvm bpf backend has better infrastructure and we probably can > implement an IR pass to detect all uses of .rodata.*, inline these > uses, and remove the .rodata.* global variable. > > You can check relocation section of the program text. If the .rodata.* > section is referenced, you should preserve it. Otherwise, you can > ignore that .rodata.* section. > >> >> What other .rodata.* should we expect? > > Glancing through llvm code, you may see .rodata.{4,8,16,32}, > .rodata.str*. > >> >> Thanks, >> >> Timo Thanks for the replies all, very insightful. We were already doing things mostly right wrt. .rodata.*, but found a few subtle bugs walking through the code again. I've gotten a hold of the ELF Vincent was trying to load, and I saw a few things that I found unusual. In his case, the values in cst32 are not inlined. Instead, this ELF has a .Lconstinit symbol pointing at the start of .rodata.cst32, and it's an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly strict and requires STT_OBJECTs to be global (for supporting non-static consts). --- ~ llvm-readelf -ar bpf_lxc.o Symbol table '.symtab' contains 606 entries: Num: Value Size Type Bind Vis Ndx Name 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 .Lconstinit Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: Offset Info Type Symbol's Value Symbol's Name 0000000000007300 0000000200000001 R_BPF_64_64 0000000000000000 .Lconstinit --- --- ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o warning: failed to compute relocation: R_BPF_64_64, Invalid data was encountered while parsing the file ... <2 more of these> ... Disassembly of section 2/7: 00000000000072f8 <LBB1_476>: 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0 ll 0000000000007300: R_BPF_64_64 .Lconstinit 3682: 0f 82 00 00 00 00 00 00 r2 += r8 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 Disassembly of section .rodata.cst32: 0000000000000000 <.Lconstinit>: 0: 82 36 4c 98 2e 56 00 00 <unknown> 1: 82 36 4c 98 2e 55 00 00 <unknown> --- This symbol doesn't exist in the program. Worth noting is that the code that accesses this static data sits within a subscope, but not sure what the effect of this would be. Vincent, maybe try removing the enclosing {} to see if that changes anything? --- static __always_inline int foo(struct __ctx_buff *ctx, ... <snip> ... { int i; for (i = 0; i < VTEP_NUMS; i) { if (tunnel_endpoint == VTEP_ENDPOINT[i]) { vtep_mac = VTEP_MAC[i]; break; } } } --- Is this perhaps something that needs to be addressed in the compiler? Thanks again, Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-04 11:11 ` Timo Beckers @ 2022-02-04 18:04 ` Yonghong Song 2022-02-04 19:39 ` Vincent Li 0 siblings, 1 reply; 21+ messages in thread From: Yonghong Song @ 2022-02-04 18:04 UTC (permalink / raw) To: Timo Beckers, Vincent Li; +Cc: bpf On 2/4/22 3:11 AM, Timo Beckers wrote: > On 2/3/22 03:11, Yonghong Song wrote: >> >> >> On 2/2/22 5:47 AM, Timo Beckers wrote: >>> On 2/2/22 08:17, Yonghong Song wrote: >>>> >>>> >>>> On 2/1/22 10:07 AM, Vincent Li wrote: >>>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>> >>>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: >>>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>>>>> >>>>>>>>> this is macro I suspected in my implementation that could cause issue with BTF >>>>>>>>> >>>>>>>>> #define ENABLE_VTEP 1 >>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>>>>>> 0x2048a90a, } >>>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, >>>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } >>>>>>>>> #define VTEP_NUMS 4 >>>>>>>>> >>>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> While developing Cilium VTEP integration feature >>>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a strange issue >>>>>>>>>> that seems related to BTF and probably caused by my specific >>>>>>>>>> implementation, the issue is described in >>>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know much about >>>>>>>>>> BTF and not sure if my implementation is seriously flawed or just some >>>>>>>>>> implementation bug or maybe not compatible with BTF. Strangely, the >>>>>>>>>> issue appears related to number of VTEPs I use, no problem with 1 or 2 >>>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF >>>>>>>>>> experts are appreciated :-). >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> Vincent >>>>>>>> >>>>>>>> Sorry for previous top post >>>>>>>> >>>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o >>>>>>>> differently and added " [21] .rodata.cst32 PROGBITS >>>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 members >>>>>>>> >>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>>>>> 0x2048a90a, } >>>>>>>> >>>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT >>>>>>>> member <=2. any reason why compiler would do that? >>>>>>> >>>>>>> Regarding to why compiler generates .rodata.cst32, the reason is >>>>>>> you have some 32-byte constants which needs to be saved somewhere. >>>>>>> For example, >>>>>>> >>>>>>> $ cat t.c >>>>>>> struct t { >>>>>>> long c[2]; >>>>>>> int d[4]; >>>>>>> }; >>>>>>> struct t g; >>>>>>> int test() >>>>>>> { >>>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; >>>>>>> g = tmp; >>>>>>> return 0; >>>>>>> } >>>>>>> >>>>>>> $ clang -target bpf -O2 -c t.c >>>>>>> $ llvm-readelf -S t.o >>>>>>> ... >>>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 >>>>>>> 20 AM 0 0 8 >>>>>>> ... >>>>>>> >>>>>>> In the above code, if you change the struct size, say from 32 bytes to >>>>>>> 40 bytes, the rodata.cst32 will go away. >>>>>> >>>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then >>>>> >>>>> Hi Yonghong, >>>>> >>>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux and >>>>> stores BTF type info in btf.Spec.namedTypes, but the elf object file >>>>> provided by user may have section like rodata.cst32 generated by >>>>> compiler that does not have accompanying BTF type info stored in >>>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be >>>>> guaranteed to have every BTF type info from application/user provided >>>>> elf object file ? I guess there is no guarantee. >>>> >>>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of >>>> rodata.cst32 needs to be emitted, the type will be encoded in bpf >>>> program BTF. >>>> >>>> Did you actually find an issue with .rodata.cst32 section? Such a >>>> section is typically generated by the compiler for initial data >>>> inside the function and llvm bpf backend tries to inline the >>>> values through a bunch of load instructions. So even you see >>>> .rodata.cst32, typically you can safely ignore it. >>>> >>>>> >>>>> Vincent >>>> >>> >>> Hi Yonghong, >>> >>> Thanks for the reproducer. Couldn't figure out what to do with .rodata.cst32, >>> since there are no symbols and no BTF info for that section. >>> >>> The values found in .rodata.cst32 are indeed inlined in the bytecode as you >>> mentioned, so it seems like we can ignore it. >>> >>> Why does the compiler emit these sections? cilium/ebpf assumed up until now >>> that all sections starting with '.rodata' are datasecs and must be loaded into >>> the kernel, which of course needs accompanying BTF. >> >> The clang frontend emits these .rodata.* sections. In early days, kernel >> doesn't support global data so llvm bpf backend implements an >> optimization to inline these values. But llvm bpf backend didn't completely remove them as the backend doesn't have a global view >> whether these .rodata.* are being used in other places or not. >> >> Now, llvm bpf backend has better infrastructure and we probably can >> implement an IR pass to detect all uses of .rodata.*, inline these >> uses, and remove the .rodata.* global variable. >> >> You can check relocation section of the program text. If the .rodata.* >> section is referenced, you should preserve it. Otherwise, you can >> ignore that .rodata.* section. >> >>> >>> What other .rodata.* should we expect? >> >> Glancing through llvm code, you may see .rodata.{4,8,16,32}, >> .rodata.str*. >> >>> >>> Thanks, >>> >>> Timo > > Thanks for the replies all, very insightful. We were already doing things mostly > right wrt. .rodata.*, but found a few subtle bugs walking through the code again. > > I've gotten a hold of the ELF Vincent was trying to load, and I saw a few things > that I found unusual. In his case, the values in cst32 are not inlined. Instead, > this ELF has a .Lconstinit symbol pointing at the start of .rodata.cst32, and it's > an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly strict and requires > STT_OBJECTs to be global (for supporting non-static consts). There are two ways to resolve the issue. First, extend the loader support to handle STB_LOCAL as well. Or Second, change the code like struct t v = {1, 5, 29, ...}; to struct t v; __builtin_memset(&v, 0, sizeof(struct t)); v.field1 = ...; v.field2 = ...; > > --- > ~ llvm-readelf -ar bpf_lxc.o > > Symbol table '.symtab' contains 606 entries: > Num: Value Size Type Bind Vis Ndx Name > 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 .Lconstinit > > Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: > Offset Info Type Symbol's Value Symbol's Name > 0000000000007300 0000000200000001 R_BPF_64_64 0000000000000000 .Lconstinit > --- > > --- > ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o > warning: failed to compute relocation: R_BPF_64_64, Invalid data was encountered while parsing the file > ... <2 more of these> ... > > Disassembly of section 2/7: > > 00000000000072f8 <LBB1_476>: > 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 > 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0 ll > 0000000000007300: R_BPF_64_64 .Lconstinit > 3682: 0f 82 00 00 00 00 00 00 r2 += r8 > 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) > 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 > > Disassembly of section .rodata.cst32: > > 0000000000000000 <.Lconstinit>: > 0: 82 36 4c 98 2e 56 00 00 <unknown> > 1: 82 36 4c 98 2e 55 00 00 <unknown> > --- > > > This symbol doesn't exist in the program. Worth noting is that the code that accesses > this static data sits within a subscope, but not sure what the effect of this would be. > > Vincent, maybe try removing the enclosing {} to see if that changes anything? > > --- > static __always_inline int foo(struct __ctx_buff *ctx, > > ... <snip> ... > > { > int i; > > for (i = 0; i < VTEP_NUMS; i) { > if (tunnel_endpoint == VTEP_ENDPOINT[i]) { > vtep_mac = VTEP_MAC[i]; > break; > } > } > } > --- > > Is this perhaps something that needs to be addressed in the compiler? If you can give a reproducible test (with .c or .i file), I can take a look at what is missing in llvm compiler and improve it. > > > Thanks again, > > Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-04 18:04 ` Yonghong Song @ 2022-02-04 19:39 ` Vincent Li 2022-02-04 21:22 ` Yonghong Song 0 siblings, 1 reply; 21+ messages in thread From: Vincent Li @ 2022-02-04 19:39 UTC (permalink / raw) To: Yonghong Song; +Cc: Timo Beckers, bpf On Fri, Feb 4, 2022 at 10:04 AM Yonghong Song <yhs@fb.com> wrote: > > > > On 2/4/22 3:11 AM, Timo Beckers wrote: > > On 2/3/22 03:11, Yonghong Song wrote: > >> > >> > >> On 2/2/22 5:47 AM, Timo Beckers wrote: > >>> On 2/2/22 08:17, Yonghong Song wrote: > >>>> > >>>> > >>>> On 2/1/22 10:07 AM, Vincent Li wrote: > >>>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>>>>> > >>>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: > >>>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>>>>>>>> > >>>>>>>>> this is macro I suspected in my implementation that could cause issue with BTF > >>>>>>>>> > >>>>>>>>> #define ENABLE_VTEP 1 > >>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > >>>>>>>>> 0x2048a90a, } > >>>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, > >>>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } > >>>>>>>>> #define VTEP_NUMS 4 > >>>>>>>>> > >>>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi > >>>>>>>>>> > >>>>>>>>>> While developing Cilium VTEP integration feature > >>>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a strange issue > >>>>>>>>>> that seems related to BTF and probably caused by my specific > >>>>>>>>>> implementation, the issue is described in > >>>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know much about > >>>>>>>>>> BTF and not sure if my implementation is seriously flawed or just some > >>>>>>>>>> implementation bug or maybe not compatible with BTF. Strangely, the > >>>>>>>>>> issue appears related to number of VTEPs I use, no problem with 1 or 2 > >>>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF > >>>>>>>>>> experts are appreciated :-). > >>>>>>>>>> > >>>>>>>>>> Thanks > >>>>>>>>>> > >>>>>>>>>> Vincent > >>>>>>>> > >>>>>>>> Sorry for previous top post > >>>>>>>> > >>>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o > >>>>>>>> differently and added " [21] .rodata.cst32 PROGBITS > >>>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 members > >>>>>>>> > >>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > >>>>>>>> 0x2048a90a, } > >>>>>>>> > >>>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT > >>>>>>>> member <=2. any reason why compiler would do that? > >>>>>>> > >>>>>>> Regarding to why compiler generates .rodata.cst32, the reason is > >>>>>>> you have some 32-byte constants which needs to be saved somewhere. > >>>>>>> For example, > >>>>>>> > >>>>>>> $ cat t.c > >>>>>>> struct t { > >>>>>>> long c[2]; > >>>>>>> int d[4]; > >>>>>>> }; > >>>>>>> struct t g; > >>>>>>> int test() > >>>>>>> { > >>>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; > >>>>>>> g = tmp; > >>>>>>> return 0; > >>>>>>> } > >>>>>>> > >>>>>>> $ clang -target bpf -O2 -c t.c > >>>>>>> $ llvm-readelf -S t.o > >>>>>>> ... > >>>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 > >>>>>>> 20 AM 0 0 8 > >>>>>>> ... > >>>>>>> > >>>>>>> In the above code, if you change the struct size, say from 32 bytes to > >>>>>>> 40 bytes, the rodata.cst32 will go away. > >>>>>> > >>>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then > >>>>> > >>>>> Hi Yonghong, > >>>>> > >>>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux and > >>>>> stores BTF type info in btf.Spec.namedTypes, but the elf object file > >>>>> provided by user may have section like rodata.cst32 generated by > >>>>> compiler that does not have accompanying BTF type info stored in > >>>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be > >>>>> guaranteed to have every BTF type info from application/user provided > >>>>> elf object file ? I guess there is no guarantee. > >>>> > >>>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of > >>>> rodata.cst32 needs to be emitted, the type will be encoded in bpf > >>>> program BTF. > >>>> > >>>> Did you actually find an issue with .rodata.cst32 section? Such a > >>>> section is typically generated by the compiler for initial data > >>>> inside the function and llvm bpf backend tries to inline the > >>>> values through a bunch of load instructions. So even you see > >>>> .rodata.cst32, typically you can safely ignore it. > >>>> > >>>>> > >>>>> Vincent > >>>> > >>> > >>> Hi Yonghong, > >>> > >>> Thanks for the reproducer. Couldn't figure out what to do with .rodata.cst32, > >>> since there are no symbols and no BTF info for that section. > >>> > >>> The values found in .rodata.cst32 are indeed inlined in the bytecode as you > >>> mentioned, so it seems like we can ignore it. > >>> > >>> Why does the compiler emit these sections? cilium/ebpf assumed up until now > >>> that all sections starting with '.rodata' are datasecs and must be loaded into > >>> the kernel, which of course needs accompanying BTF. > >> > >> The clang frontend emits these .rodata.* sections. In early days, kernel > >> doesn't support global data so llvm bpf backend implements an > >> optimization to inline these values. But llvm bpf backend didn't completely remove them as the backend doesn't have a global view > >> whether these .rodata.* are being used in other places or not. > >> > >> Now, llvm bpf backend has better infrastructure and we probably can > >> implement an IR pass to detect all uses of .rodata.*, inline these > >> uses, and remove the .rodata.* global variable. > >> > >> You can check relocation section of the program text. If the .rodata.* > >> section is referenced, you should preserve it. Otherwise, you can > >> ignore that .rodata.* section. > >> > >>> > >>> What other .rodata.* should we expect? > >> > >> Glancing through llvm code, you may see .rodata.{4,8,16,32}, > >> .rodata.str*. > >> > >>> > >>> Thanks, > >>> > >>> Timo > > > > Thanks for the replies all, very insightful. We were already doing things mostly > > right wrt. .rodata.*, but found a few subtle bugs walking through the code again. > > > > I've gotten a hold of the ELF Vincent was trying to load, and I saw a few things > > that I found unusual. In his case, the values in cst32 are not inlined. Instead, > > this ELF has a .Lconstinit symbol pointing at the start of .rodata.cst32, and it's > > an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly strict and requires > > STT_OBJECTs to be global (for supporting non-static consts). > > There are two ways to resolve the issue. First, extend the loader > support to handle STB_LOCAL as well. Or Second, change the code like > struct t v = {1, 5, 29, ...}; > to > struct t v; > __builtin_memset(&v, 0, sizeof(struct t)); > v.field1 = ...; > v.field2 = ...; > > > > > > --- > > ~ llvm-readelf -ar bpf_lxc.o > > > > Symbol table '.symtab' contains 606 entries: > > Num: Value Size Type Bind Vis Ndx Name > > 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 .Lconstinit > > > > Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: > > Offset Info Type Symbol's Value Symbol's Name > > 0000000000007300 0000000200000001 R_BPF_64_64 0000000000000000 .Lconstinit > > --- > > > > --- > > ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o > > warning: failed to compute relocation: R_BPF_64_64, Invalid data was encountered while parsing the file > > ... <2 more of these> ... > > > > Disassembly of section 2/7: > > > > 00000000000072f8 <LBB1_476>: > > 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 > > 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0 ll > > 0000000000007300: R_BPF_64_64 .Lconstinit > > 3682: 0f 82 00 00 00 00 00 00 r2 += r8 > > 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) > > 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 > > > > Disassembly of section .rodata.cst32: > > > > 0000000000000000 <.Lconstinit>: > > 0: 82 36 4c 98 2e 56 00 00 <unknown> > > 1: 82 36 4c 98 2e 55 00 00 <unknown> > > --- > > > > > > This symbol doesn't exist in the program. Worth noting is that the code that accesses > > this static data sits within a subscope, but not sure what the effect of this would be. > > > > Vincent, maybe try removing the enclosing {} to see if that changes anything? > > > > --- > > static __always_inline int foo(struct __ctx_buff *ctx, > > > > ... <snip> ... > > > > { > > int i; > > > > for (i = 0; i < VTEP_NUMS; i) { > > if (tunnel_endpoint == VTEP_ENDPOINT[i]) { > > vtep_mac = VTEP_MAC[i]; > > break; > > } > > } > > } > > --- > > > > Is this perhaps something that needs to be addressed in the compiler? > > If you can give a reproducible test (with .c or .i file), I can take a > look at what is missing in llvm compiler and improve it. > not sure if it would help, here is my step to generate the bpf_lxc.o object file with the .rodata.cst32 git clone https://github.com/f5devcentral/cilium.git cd cilium; git checkout vli-vxlan; KERNEL=54 make -C bpf llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf/bpf_lxc.o > > > > > > Thanks again, > > > > Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-04 19:39 ` Vincent Li @ 2022-02-04 21:22 ` Yonghong Song 2022-02-07 21:53 ` Andrii Nakryiko 2022-02-08 6:47 ` Yonghong Song 0 siblings, 2 replies; 21+ messages in thread From: Yonghong Song @ 2022-02-04 21:22 UTC (permalink / raw) To: Vincent Li; +Cc: Timo Beckers, bpf On 2/4/22 11:39 AM, Vincent Li wrote: > On Fri, Feb 4, 2022 at 10:04 AM Yonghong Song <yhs@fb.com> wrote: >> >> >> >> On 2/4/22 3:11 AM, Timo Beckers wrote: >>> On 2/3/22 03:11, Yonghong Song wrote: >>>> >>>> >>>> On 2/2/22 5:47 AM, Timo Beckers wrote: >>>>> On 2/2/22 08:17, Yonghong Song wrote: >>>>>> >>>>>> >>>>>> On 2/1/22 10:07 AM, Vincent Li wrote: >>>>>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>>>> >>>>>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: >>>>>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> this is macro I suspected in my implementation that could cause issue with BTF >>>>>>>>>>> >>>>>>>>>>> #define ENABLE_VTEP 1 >>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>>>>>>>> 0x2048a90a, } >>>>>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, >>>>>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } >>>>>>>>>>> #define VTEP_NUMS 4 >>>>>>>>>>> >>>>>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi >>>>>>>>>>>> >>>>>>>>>>>> While developing Cilium VTEP integration feature >>>>>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a strange issue >>>>>>>>>>>> that seems related to BTF and probably caused by my specific >>>>>>>>>>>> implementation, the issue is described in >>>>>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know much about >>>>>>>>>>>> BTF and not sure if my implementation is seriously flawed or just some >>>>>>>>>>>> implementation bug or maybe not compatible with BTF. Strangely, the >>>>>>>>>>>> issue appears related to number of VTEPs I use, no problem with 1 or 2 >>>>>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF >>>>>>>>>>>> experts are appreciated :-). >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> Vincent >>>>>>>>>> >>>>>>>>>> Sorry for previous top post >>>>>>>>>> >>>>>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o >>>>>>>>>> differently and added " [21] .rodata.cst32 PROGBITS >>>>>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 members >>>>>>>>>> >>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, >>>>>>>>>> 0x2048a90a, } >>>>>>>>>> >>>>>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT >>>>>>>>>> member <=2. any reason why compiler would do that? >>>>>>>>> >>>>>>>>> Regarding to why compiler generates .rodata.cst32, the reason is >>>>>>>>> you have some 32-byte constants which needs to be saved somewhere. >>>>>>>>> For example, >>>>>>>>> >>>>>>>>> $ cat t.c >>>>>>>>> struct t { >>>>>>>>> long c[2]; >>>>>>>>> int d[4]; >>>>>>>>> }; >>>>>>>>> struct t g; >>>>>>>>> int test() >>>>>>>>> { >>>>>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; >>>>>>>>> g = tmp; >>>>>>>>> return 0; >>>>>>>>> } >>>>>>>>> >>>>>>>>> $ clang -target bpf -O2 -c t.c >>>>>>>>> $ llvm-readelf -S t.o >>>>>>>>> ... >>>>>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 >>>>>>>>> 20 AM 0 0 8 >>>>>>>>> ... >>>>>>>>> >>>>>>>>> In the above code, if you change the struct size, say from 32 bytes to >>>>>>>>> 40 bytes, the rodata.cst32 will go away. >>>>>>>> >>>>>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then >>>>>>> >>>>>>> Hi Yonghong, >>>>>>> >>>>>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux and >>>>>>> stores BTF type info in btf.Spec.namedTypes, but the elf object file >>>>>>> provided by user may have section like rodata.cst32 generated by >>>>>>> compiler that does not have accompanying BTF type info stored in >>>>>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be >>>>>>> guaranteed to have every BTF type info from application/user provided >>>>>>> elf object file ? I guess there is no guarantee. >>>>>> >>>>>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of >>>>>> rodata.cst32 needs to be emitted, the type will be encoded in bpf >>>>>> program BTF. >>>>>> >>>>>> Did you actually find an issue with .rodata.cst32 section? Such a >>>>>> section is typically generated by the compiler for initial data >>>>>> inside the function and llvm bpf backend tries to inline the >>>>>> values through a bunch of load instructions. So even you see >>>>>> .rodata.cst32, typically you can safely ignore it. >>>>>> >>>>>>> >>>>>>> Vincent >>>>>> >>>>> >>>>> Hi Yonghong, >>>>> >>>>> Thanks for the reproducer. Couldn't figure out what to do with .rodata.cst32, >>>>> since there are no symbols and no BTF info for that section. >>>>> >>>>> The values found in .rodata.cst32 are indeed inlined in the bytecode as you >>>>> mentioned, so it seems like we can ignore it. >>>>> >>>>> Why does the compiler emit these sections? cilium/ebpf assumed up until now >>>>> that all sections starting with '.rodata' are datasecs and must be loaded into >>>>> the kernel, which of course needs accompanying BTF. >>>> >>>> The clang frontend emits these .rodata.* sections. In early days, kernel >>>> doesn't support global data so llvm bpf backend implements an >>>> optimization to inline these values. But llvm bpf backend didn't completely remove them as the backend doesn't have a global view >>>> whether these .rodata.* are being used in other places or not. >>>> >>>> Now, llvm bpf backend has better infrastructure and we probably can >>>> implement an IR pass to detect all uses of .rodata.*, inline these >>>> uses, and remove the .rodata.* global variable. >>>> >>>> You can check relocation section of the program text. If the .rodata.* >>>> section is referenced, you should preserve it. Otherwise, you can >>>> ignore that .rodata.* section. >>>> >>>>> >>>>> What other .rodata.* should we expect? >>>> >>>> Glancing through llvm code, you may see .rodata.{4,8,16,32}, >>>> .rodata.str*. >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Timo >>> >>> Thanks for the replies all, very insightful. We were already doing things mostly >>> right wrt. .rodata.*, but found a few subtle bugs walking through the code again. >>> >>> I've gotten a hold of the ELF Vincent was trying to load, and I saw a few things >>> that I found unusual. In his case, the values in cst32 are not inlined. Instead, >>> this ELF has a .Lconstinit symbol pointing at the start of .rodata.cst32, and it's >>> an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly strict and requires >>> STT_OBJECTs to be global (for supporting non-static consts). >> >> There are two ways to resolve the issue. First, extend the loader >> support to handle STB_LOCAL as well. Or Second, change the code like >> struct t v = {1, 5, 29, ...}; >> to >> struct t v; >> __builtin_memset(&v, 0, sizeof(struct t)); >> v.field1 = ...; >> v.field2 = ...; >> >> >>> >>> --- >>> ~ llvm-readelf -ar bpf_lxc.o >>> >>> Symbol table '.symtab' contains 606 entries: >>> Num: Value Size Type Bind Vis Ndx Name >>> 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 .Lconstinit >>> >>> Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: >>> Offset Info Type Symbol's Value Symbol's Name >>> 0000000000007300 0000000200000001 R_BPF_64_64 0000000000000000 .Lconstinit >>> --- >>> >>> --- >>> ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o >>> warning: failed to compute relocation: R_BPF_64_64, Invalid data was encountered while parsing the file >>> ... <2 more of these> ... >>> >>> Disassembly of section 2/7: >>> >>> 00000000000072f8 <LBB1_476>: >>> 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 >>> 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0 ll >>> 0000000000007300: R_BPF_64_64 .Lconstinit >>> 3682: 0f 82 00 00 00 00 00 00 r2 += r8 >>> 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) >>> 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 >>> >>> Disassembly of section .rodata.cst32: >>> >>> 0000000000000000 <.Lconstinit>: >>> 0: 82 36 4c 98 2e 56 00 00 <unknown> >>> 1: 82 36 4c 98 2e 55 00 00 <unknown> >>> --- >>> >>> >>> This symbol doesn't exist in the program. Worth noting is that the code that accesses >>> this static data sits within a subscope, but not sure what the effect of this would be. >>> >>> Vincent, maybe try removing the enclosing {} to see if that changes anything? >>> >>> --- >>> static __always_inline int foo(struct __ctx_buff *ctx, >>> >>> ... <snip> ... >>> >>> { >>> int i; >>> >>> for (i = 0; i < VTEP_NUMS; i) { >>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { >>> vtep_mac = VTEP_MAC[i]; >>> break; >>> } >>> } >>> } >>> --- >>> >>> Is this perhaps something that needs to be addressed in the compiler? >> >> If you can give a reproducible test (with .c or .i file), I can take a >> look at what is missing in llvm compiler and improve it. >> > > not sure if it would help, here is my step to generate the bpf_lxc.o > object file with the .rodata.cst32 > > git clone https://github.com/f5devcentral/cilium.git > cd cilium; git checkout vli-vxlan; KERNEL=54 make -C bpf > llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf/bpf_lxc.o Thanks. I can reproduce the issue now. Will take a look and get back to you as soon as I got any concrete results. > >>> >>> >>> Thanks again, >>> >>> Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-04 21:22 ` Yonghong Song @ 2022-02-07 21:53 ` Andrii Nakryiko 2022-02-08 6:47 ` Yonghong Song 1 sibling, 0 replies; 21+ messages in thread From: Andrii Nakryiko @ 2022-02-07 21:53 UTC (permalink / raw) To: Yonghong Song; +Cc: Vincent Li, Timo Beckers, bpf On Sat, Feb 5, 2022 at 2:38 AM Yonghong Song <yhs@fb.com> wrote: > > > > On 2/4/22 11:39 AM, Vincent Li wrote: > > On Fri, Feb 4, 2022 at 10:04 AM Yonghong Song <yhs@fb.com> wrote: > >> > >> > >> > >> On 2/4/22 3:11 AM, Timo Beckers wrote: > >>> On 2/3/22 03:11, Yonghong Song wrote: > >>>> > >>>> > >>>> On 2/2/22 5:47 AM, Timo Beckers wrote: > >>>>> On 2/2/22 08:17, Yonghong Song wrote: > >>>>>> > >>>>>> > >>>>>> On 2/1/22 10:07 AM, Vincent Li wrote: > >>>>>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>>>>>>> > >>>>>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: > >>>>>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>> this is macro I suspected in my implementation that could cause issue with BTF > >>>>>>>>>>> > >>>>>>>>>>> #define ENABLE_VTEP 1 > >>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > >>>>>>>>>>> 0x2048a90a, } > >>>>>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, > >>>>>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } > >>>>>>>>>>> #define VTEP_NUMS 4 > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Hi > >>>>>>>>>>>> > >>>>>>>>>>>> While developing Cilium VTEP integration feature > >>>>>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a strange issue > >>>>>>>>>>>> that seems related to BTF and probably caused by my specific > >>>>>>>>>>>> implementation, the issue is described in > >>>>>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know much about > >>>>>>>>>>>> BTF and not sure if my implementation is seriously flawed or just some > >>>>>>>>>>>> implementation bug or maybe not compatible with BTF. Strangely, the > >>>>>>>>>>>> issue appears related to number of VTEPs I use, no problem with 1 or 2 > >>>>>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance from BTF > >>>>>>>>>>>> experts are appreciated :-). > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks > >>>>>>>>>>>> > >>>>>>>>>>>> Vincent > >>>>>>>>>> > >>>>>>>>>> Sorry for previous top post > >>>>>>>>>> > >>>>>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o > >>>>>>>>>> differently and added " [21] .rodata.cst32 PROGBITS > >>>>>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 members > >>>>>>>>>> > >>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, 0x1f48a90a, > >>>>>>>>>> 0x2048a90a, } > >>>>>>>>>> > >>>>>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above VTEP_ENDPOINT > >>>>>>>>>> member <=2. any reason why compiler would do that? > >>>>>>>>> > >>>>>>>>> Regarding to why compiler generates .rodata.cst32, the reason is > >>>>>>>>> you have some 32-byte constants which needs to be saved somewhere. > >>>>>>>>> For example, > >>>>>>>>> > >>>>>>>>> $ cat t.c > >>>>>>>>> struct t { > >>>>>>>>> long c[2]; > >>>>>>>>> int d[4]; > >>>>>>>>> }; > >>>>>>>>> struct t g; > >>>>>>>>> int test() > >>>>>>>>> { > >>>>>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; > >>>>>>>>> g = tmp; > >>>>>>>>> return 0; > >>>>>>>>> } > >>>>>>>>> > >>>>>>>>> $ clang -target bpf -O2 -c t.c > >>>>>>>>> $ llvm-readelf -S t.o > >>>>>>>>> ... > >>>>>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 0000a8 000020 > >>>>>>>>> 20 AM 0 0 8 > >>>>>>>>> ... > >>>>>>>>> > >>>>>>>>> In the above code, if you change the struct size, say from 32 bytes to > >>>>>>>>> 40 bytes, the rodata.cst32 will go away. > >>>>>>>> > >>>>>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize rodata.cst32 then > >>>>>>> > >>>>>>> Hi Yonghong, > >>>>>>> > >>>>>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux and > >>>>>>> stores BTF type info in btf.Spec.namedTypes, but the elf object file > >>>>>>> provided by user may have section like rodata.cst32 generated by > >>>>>>> compiler that does not have accompanying BTF type info stored in > >>>>>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be > >>>>>>> guaranteed to have every BTF type info from application/user provided > >>>>>>> elf object file ? I guess there is no guarantee. > >>>>>> > >>>>>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of > >>>>>> rodata.cst32 needs to be emitted, the type will be encoded in bpf > >>>>>> program BTF. > >>>>>> > >>>>>> Did you actually find an issue with .rodata.cst32 section? Such a > >>>>>> section is typically generated by the compiler for initial data > >>>>>> inside the function and llvm bpf backend tries to inline the > >>>>>> values through a bunch of load instructions. So even you see > >>>>>> .rodata.cst32, typically you can safely ignore it. > >>>>>> > >>>>>>> > >>>>>>> Vincent > >>>>>> > >>>>> > >>>>> Hi Yonghong, > >>>>> > >>>>> Thanks for the reproducer. Couldn't figure out what to do with .rodata.cst32, > >>>>> since there are no symbols and no BTF info for that section. > >>>>> > >>>>> The values found in .rodata.cst32 are indeed inlined in the bytecode as you > >>>>> mentioned, so it seems like we can ignore it. > >>>>> > >>>>> Why does the compiler emit these sections? cilium/ebpf assumed up until now > >>>>> that all sections starting with '.rodata' are datasecs and must be loaded into > >>>>> the kernel, which of course needs accompanying BTF. > >>>> > >>>> The clang frontend emits these .rodata.* sections. In early days, kernel > >>>> doesn't support global data so llvm bpf backend implements an > >>>> optimization to inline these values. But llvm bpf backend didn't completely remove them as the backend doesn't have a global view > >>>> whether these .rodata.* are being used in other places or not. > >>>> > >>>> Now, llvm bpf backend has better infrastructure and we probably can > >>>> implement an IR pass to detect all uses of .rodata.*, inline these > >>>> uses, and remove the .rodata.* global variable. > >>>> > >>>> You can check relocation section of the program text. If the .rodata.* > >>>> section is referenced, you should preserve it. Otherwise, you can > >>>> ignore that .rodata.* section. > >>>> > >>>>> > >>>>> What other .rodata.* should we expect? > >>>> > >>>> Glancing through llvm code, you may see .rodata.{4,8,16,32}, > >>>> .rodata.str*. > >>>> > >>>>> > >>>>> Thanks, > >>>>> > >>>>> Timo > >>> > >>> Thanks for the replies all, very insightful. We were already doing things mostly > >>> right wrt. .rodata.*, but found a few subtle bugs walking through the code again. > >>> > >>> I've gotten a hold of the ELF Vincent was trying to load, and I saw a few things > >>> that I found unusual. In his case, the values in cst32 are not inlined. Instead, > >>> this ELF has a .Lconstinit symbol pointing at the start of .rodata.cst32, and it's > >>> an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly strict and requires > >>> STT_OBJECTs to be global (for supporting non-static consts). > >> > >> There are two ways to resolve the issue. First, extend the loader > >> support to handle STB_LOCAL as well. Or Second, change the code like > >> struct t v = {1, 5, 29, ...}; > >> to > >> struct t v; > >> __builtin_memset(&v, 0, sizeof(struct t)); > >> v.field1 = ...; > >> v.field2 = ...; > >> > >> > >>> > >>> --- > >>> ~ llvm-readelf -ar bpf_lxc.o > >>> > >>> Symbol table '.symtab' contains 606 entries: > >>> Num: Value Size Type Bind Vis Ndx Name > >>> 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 .Lconstinit > >>> > >>> Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: > >>> Offset Info Type Symbol's Value Symbol's Name > >>> 0000000000007300 0000000200000001 R_BPF_64_64 0000000000000000 .Lconstinit > >>> --- > >>> > >>> --- > >>> ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o > >>> warning: failed to compute relocation: R_BPF_64_64, Invalid data was encountered while parsing the file > >>> ... <2 more of these> ... > >>> > >>> Disassembly of section 2/7: > >>> > >>> 00000000000072f8 <LBB1_476>: > >>> 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 > >>> 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0 ll > >>> 0000000000007300: R_BPF_64_64 .Lconstinit > >>> 3682: 0f 82 00 00 00 00 00 00 r2 += r8 > >>> 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) > >>> 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 > >>> > >>> Disassembly of section .rodata.cst32: > >>> > >>> 0000000000000000 <.Lconstinit>: > >>> 0: 82 36 4c 98 2e 56 00 00 <unknown> > >>> 1: 82 36 4c 98 2e 55 00 00 <unknown> > >>> --- > >>> > >>> > >>> This symbol doesn't exist in the program. Worth noting is that the code that accesses > >>> this static data sits within a subscope, but not sure what the effect of this would be. > >>> > >>> Vincent, maybe try removing the enclosing {} to see if that changes anything? > >>> > >>> --- > >>> static __always_inline int foo(struct __ctx_buff *ctx, > >>> > >>> ... <snip> ... > >>> > >>> { > >>> int i; > >>> > >>> for (i = 0; i < VTEP_NUMS; i) { > >>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { > >>> vtep_mac = VTEP_MAC[i]; > >>> break; > >>> } > >>> } > >>> } > >>> --- > >>> > >>> Is this perhaps something that needs to be addressed in the compiler? > >> > >> If you can give a reproducible test (with .c or .i file), I can take a > >> look at what is missing in llvm compiler and improve it. > >> > > > > not sure if it would help, here is my step to generate the bpf_lxc.o > > object file with the .rodata.cst32 > > > > git clone https://github.com/f5devcentral/cilium.git > > cd cilium; git checkout vli-vxlan; KERNEL=54 make -C bpf > > llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf/bpf_lxc.o > > Thanks. I can reproduce the issue now. Will take a look > and get back to you as soon as I got any concrete results. > All the emails are arriving heavily out of order, it's confusing to follow. Does this affect cilium/ebpf library or libbpf can't handle this as well? > > > >>> > >>> > >>> Thanks again, > >>> > >>> Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-04 21:22 ` Yonghong Song 2022-02-07 21:53 ` Andrii Nakryiko @ 2022-02-08 6:47 ` Yonghong Song 2022-02-08 18:28 ` Vincent Li 1 sibling, 1 reply; 21+ messages in thread From: Yonghong Song @ 2022-02-08 6:47 UTC (permalink / raw) To: Vincent Li; +Cc: Timo Beckers, bpf On 2/4/22 1:22 PM, Yonghong Song wrote: > > > On 2/4/22 11:39 AM, Vincent Li wrote: >> On Fri, Feb 4, 2022 at 10:04 AM Yonghong Song <yhs@fb.com> wrote: >>> >>> >>> >>> On 2/4/22 3:11 AM, Timo Beckers wrote: >>>> On 2/3/22 03:11, Yonghong Song wrote: >>>>> >>>>> >>>>> On 2/2/22 5:47 AM, Timo Beckers wrote: >>>>>> On 2/2/22 08:17, Yonghong Song wrote: >>>>>>> >>>>>>> >>>>>>> On 2/1/22 10:07 AM, Vincent Li wrote: >>>>>>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li >>>>>>>> <vincent.mc.li@gmail.com> wrote: >>>>>>>>> >>>>>>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: >>>>>>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li >>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> this is macro I suspected in my implementation that could >>>>>>>>>>>> cause issue with BTF >>>>>>>>>>>> >>>>>>>>>>>> #define ENABLE_VTEP 1 >>>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, >>>>>>>>>>>> 0x1f48a90a, >>>>>>>>>>>> 0x2048a90a, } >>>>>>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, >>>>>>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } >>>>>>>>>>>> #define VTEP_NUMS 4 >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li >>>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi >>>>>>>>>>>>> >>>>>>>>>>>>> While developing Cilium VTEP integration feature >>>>>>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a >>>>>>>>>>>>> strange issue >>>>>>>>>>>>> that seems related to BTF and probably caused by my specific >>>>>>>>>>>>> implementation, the issue is described in >>>>>>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know >>>>>>>>>>>>> much about >>>>>>>>>>>>> BTF and not sure if my implementation is seriously flawed >>>>>>>>>>>>> or just some >>>>>>>>>>>>> implementation bug or maybe not compatible with BTF. >>>>>>>>>>>>> Strangely, the >>>>>>>>>>>>> issue appears related to number of VTEPs I use, no problem >>>>>>>>>>>>> with 1 or 2 >>>>>>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance >>>>>>>>>>>>> from BTF >>>>>>>>>>>>> experts are appreciated :-). >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> Vincent >>>>>>>>>>> >>>>>>>>>>> Sorry for previous top post >>>>>>>>>>> >>>>>>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o >>>>>>>>>>> differently and added " [21] .rodata.cst32 PROGBITS >>>>>>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 >>>>>>>>>>> members >>>>>>>>>>> >>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, >>>>>>>>>>> 0x1f48a90a, >>>>>>>>>>> 0x2048a90a, } >>>>>>>>>>> >>>>>>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above >>>>>>>>>>> VTEP_ENDPOINT >>>>>>>>>>> member <=2. any reason why compiler would do that? >>>>>>>>>> >>>>>>>>>> Regarding to why compiler generates .rodata.cst32, the reason is >>>>>>>>>> you have some 32-byte constants which needs to be saved >>>>>>>>>> somewhere. >>>>>>>>>> For example, >>>>>>>>>> >>>>>>>>>> $ cat t.c >>>>>>>>>> struct t { >>>>>>>>>> long c[2]; >>>>>>>>>> int d[4]; >>>>>>>>>> }; >>>>>>>>>> struct t g; >>>>>>>>>> int test() >>>>>>>>>> { >>>>>>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; >>>>>>>>>> g = tmp; >>>>>>>>>> return 0; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> $ clang -target bpf -O2 -c t.c >>>>>>>>>> $ llvm-readelf -S t.o >>>>>>>>>> ... >>>>>>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 >>>>>>>>>> 0000a8 000020 >>>>>>>>>> 20 AM 0 0 8 >>>>>>>>>> ... >>>>>>>>>> >>>>>>>>>> In the above code, if you change the struct size, say from 32 >>>>>>>>>> bytes to >>>>>>>>>> 40 bytes, the rodata.cst32 will go away. >>>>>>>>> >>>>>>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize >>>>>>>>> rodata.cst32 then >>>>>>>> >>>>>>>> Hi Yonghong, >>>>>>>> >>>>>>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux >>>>>>>> and >>>>>>>> stores BTF type info in btf.Spec.namedTypes, but the elf object >>>>>>>> file >>>>>>>> provided by user may have section like rodata.cst32 generated by >>>>>>>> compiler that does not have accompanying BTF type info stored in >>>>>>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be >>>>>>>> guaranteed to have every BTF type info from application/user >>>>>>>> provided >>>>>>>> elf object file ? I guess there is no guarantee. >>>>>>> >>>>>>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of >>>>>>> rodata.cst32 needs to be emitted, the type will be encoded in bpf >>>>>>> program BTF. >>>>>>> >>>>>>> Did you actually find an issue with .rodata.cst32 section? Such a >>>>>>> section is typically generated by the compiler for initial data >>>>>>> inside the function and llvm bpf backend tries to inline the >>>>>>> values through a bunch of load instructions. So even you see >>>>>>> .rodata.cst32, typically you can safely ignore it. >>>>>>> >>>>>>>> >>>>>>>> Vincent >>>>>>> >>>>>> >>>>>> Hi Yonghong, >>>>>> >>>>>> Thanks for the reproducer. Couldn't figure out what to do with >>>>>> .rodata.cst32, >>>>>> since there are no symbols and no BTF info for that section. >>>>>> >>>>>> The values found in .rodata.cst32 are indeed inlined in the >>>>>> bytecode as you >>>>>> mentioned, so it seems like we can ignore it. >>>>>> >>>>>> Why does the compiler emit these sections? cilium/ebpf assumed up >>>>>> until now >>>>>> that all sections starting with '.rodata' are datasecs and must be >>>>>> loaded into >>>>>> the kernel, which of course needs accompanying BTF. >>>>> >>>>> The clang frontend emits these .rodata.* sections. In early days, >>>>> kernel >>>>> doesn't support global data so llvm bpf backend implements an >>>>> optimization to inline these values. But llvm bpf backend didn't >>>>> completely remove them as the backend doesn't have a global view >>>>> whether these .rodata.* are being used in other places or not. >>>>> >>>>> Now, llvm bpf backend has better infrastructure and we probably can >>>>> implement an IR pass to detect all uses of .rodata.*, inline these >>>>> uses, and remove the .rodata.* global variable. >>>>> >>>>> You can check relocation section of the program text. If the .rodata.* >>>>> section is referenced, you should preserve it. Otherwise, you can >>>>> ignore that .rodata.* section. >>>>> >>>>>> >>>>>> What other .rodata.* should we expect? >>>>> >>>>> Glancing through llvm code, you may see .rodata.{4,8,16,32}, >>>>> .rodata.str*. >>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Timo >>>> >>>> Thanks for the replies all, very insightful. We were already doing >>>> things mostly >>>> right wrt. .rodata.*, but found a few subtle bugs walking through >>>> the code again. >>>> >>>> I've gotten a hold of the ELF Vincent was trying to load, and I saw >>>> a few things >>>> that I found unusual. In his case, the values in cst32 are not >>>> inlined. Instead, >>>> this ELF has a .Lconstinit symbol pointing at the start of >>>> .rodata.cst32, and it's >>>> an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly >>>> strict and requires >>>> STT_OBJECTs to be global (for supporting non-static consts). >>> >>> There are two ways to resolve the issue. First, extend the loader >>> support to handle STB_LOCAL as well. Or Second, change the code like >>> struct t v = {1, 5, 29, ...}; >>> to >>> struct t v; >>> __builtin_memset(&v, 0, sizeof(struct t)); >>> v.field1 = ...; >>> v.field2 = ...; >>> >>> >>>> >>>> --- >>>> ~ llvm-readelf -ar bpf_lxc.o >>>> >>>> Symbol table '.symtab' contains 606 entries: >>>> Num: Value Size Type Bind Vis Ndx Name >>>> 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 >>>> .Lconstinit >>>> >>>> Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: >>>> Offset Info Type >>>> Symbol's Value Symbol's Name >>>> 0000000000007300 0000000200000001 R_BPF_64_64 >>>> 0000000000000000 .Lconstinit >>>> --- >>>> >>>> --- >>>> ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o >>>> warning: failed to compute relocation: R_BPF_64_64, Invalid data was >>>> encountered while parsing the file >>>> ... <2 more of these> ... >>>> >>>> Disassembly of section 2/7: >>>> >>>> 00000000000072f8 <LBB1_476>: >>>> 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 >>>> 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 >>>> = 0 ll >>>> 0000000000007300: R_BPF_64_64 .Lconstinit >>>> 3682: 0f 82 00 00 00 00 00 00 r2 += r8 >>>> 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) >>>> 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 >>>> >>>> Disassembly of section .rodata.cst32: >>>> >>>> 0000000000000000 <.Lconstinit>: >>>> 0: 82 36 4c 98 2e 56 00 00 <unknown> >>>> 1: 82 36 4c 98 2e 55 00 00 <unknown> >>>> --- >>>> >>>> >>>> This symbol doesn't exist in the program. Worth noting is that the >>>> code that accesses >>>> this static data sits within a subscope, but not sure what the >>>> effect of this would be. >>>> >>>> Vincent, maybe try removing the enclosing {} to see if that changes >>>> anything? >>>> >>>> --- >>>> static __always_inline int foo(struct __ctx_buff *ctx, >>>> >>>> ... <snip> ... >>>> >>>> { >>>> int i; >>>> >>>> for (i = 0; i < VTEP_NUMS; i) { >>>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { >>>> vtep_mac = VTEP_MAC[i]; >>>> break; >>>> } >>>> } >>>> } >>>> --- >>>> >>>> Is this perhaps something that needs to be addressed in the compiler? >>> >>> If you can give a reproducible test (with .c or .i file), I can take a >>> look at what is missing in llvm compiler and improve it. >>> >> >> not sure if it would help, here is my step to generate the bpf_lxc.o >> object file with the .rodata.cst32 >> >> git clone https://github.com/f5devcentral/cilium.git >> cd cilium; git checkout vli-vxlan; KERNEL=54 make -C bpf >> llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf/bpf_lxc.o > > Thanks. I can reproduce the issue now. Will take a look > and get back to you as soon as I got any concrete results. Okay, I found the reason. For the code, for (i = 0; i < VTEP_NUMS; i++) { if (tunnel_endpoint == VTEP_ENDPOINT[i]) { vtep_mac = VTEP_MAC[i]; break; } } The compiler transformed to something like i = 0; if (tunnerl_endpoint == VTEP_ENDPOINT[0]) goto end; i = 1; if (tunnerl_endpoint == VTEP_ENDPOINT[1]) goto end; i = 2; if (tunnerl_endpoint == VTEP_ENDPOINT[2]) goto end; i = 3; if (tunnerl_endpoint == VTEP_ENDPOINT[3]) goto end; end: vtep_mac = VTEP_MAC[i]; The compiler cannot inline VTEP_MAC[i] since 'i' is not a constant. Hence later we have a memory load from a non-global .rodata section. As I mentioned earlier, there are two options to fix the issue. First is for cilium to track and handle non-global .rodata sections. And the second you can apply the below code change, diff --git a/bpf/node_config.h b/bpf/node_config.h index 9783e44548..b80dd2b27b 100644 --- a/bpf/node_config.h +++ b/bpf/node_config.h @@ -176,15 +176,15 @@ DEFINE_IPV6(HOST_IP, 0xbe, 0xef, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0xa, 0x #endif #ifdef ENABLE_VTEP -#define VTEP_ENDPOINT (__u32[]){0xeb48a90a, 0xec48a90a, 0xed48a90a, 0xee48a90a, } +#define VTEP_NUMS 4 +__u32 VTEP_ENDPOINT[VTEP_NUMS] = {0xeb48a90a, 0xec48a90a, 0xed48a90a, 0xee48a90a}; /* HEX representation of VTEP IP * 10.169.72.235, 10.169.72.236, 10.169.72.237, 10.169.72.238 */ -#define VTEP_MAC (__u64[]){0x562e984c3682, 0x552e984c3682, 0x542e984c3682, 0x532e984c3682} +__u64 VTEP_MAC[VTEP_NUMS] = {0x562e984c3682, 0x552e984c3682, 0x542e984c3682, 0x532e984c3682}; /* VTEP MAC address * 82:36:4c:89:2e:56, 82:36:4c:89:2e:55, 82:36:4c:89:2e:54, 82:36:4c:89:2e:53 */ -#define VTEP_NUMS 4 #endif /* It appears that we can support around the below number of prefixes in an > >> >>>> >>>> >>>> Thanks again, >>>> >>>> Timo ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-08 6:47 ` Yonghong Song @ 2022-02-08 18:28 ` Vincent Li 2022-02-08 18:48 ` Vincent Li 0 siblings, 1 reply; 21+ messages in thread From: Vincent Li @ 2022-02-08 18:28 UTC (permalink / raw) To: Yonghong Song; +Cc: Timo Beckers, bpf On Mon, Feb 7, 2022 at 10:47 PM Yonghong Song <yhs@fb.com> wrote: > > > > On 2/4/22 1:22 PM, Yonghong Song wrote: > > > > > > On 2/4/22 11:39 AM, Vincent Li wrote: > >> On Fri, Feb 4, 2022 at 10:04 AM Yonghong Song <yhs@fb.com> wrote: > >>> > >>> > >>> > >>> On 2/4/22 3:11 AM, Timo Beckers wrote: > >>>> On 2/3/22 03:11, Yonghong Song wrote: > >>>>> > >>>>> > >>>>> On 2/2/22 5:47 AM, Timo Beckers wrote: > >>>>>> On 2/2/22 08:17, Yonghong Song wrote: > >>>>>>> > >>>>>>> > >>>>>>> On 2/1/22 10:07 AM, Vincent Li wrote: > >>>>>>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li > >>>>>>>> <vincent.mc.li@gmail.com> wrote: > >>>>>>>>> > >>>>>>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: > >>>>>>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li > >>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> this is macro I suspected in my implementation that could > >>>>>>>>>>>> cause issue with BTF > >>>>>>>>>>>> > >>>>>>>>>>>> #define ENABLE_VTEP 1 > >>>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, > >>>>>>>>>>>> 0x1f48a90a, > >>>>>>>>>>>> 0x2048a90a, } > >>>>>>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, > >>>>>>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } > >>>>>>>>>>>> #define VTEP_NUMS 4 > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li > >>>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi > >>>>>>>>>>>>> > >>>>>>>>>>>>> While developing Cilium VTEP integration feature > >>>>>>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a > >>>>>>>>>>>>> strange issue > >>>>>>>>>>>>> that seems related to BTF and probably caused by my specific > >>>>>>>>>>>>> implementation, the issue is described in > >>>>>>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know > >>>>>>>>>>>>> much about > >>>>>>>>>>>>> BTF and not sure if my implementation is seriously flawed > >>>>>>>>>>>>> or just some > >>>>>>>>>>>>> implementation bug or maybe not compatible with BTF. > >>>>>>>>>>>>> Strangely, the > >>>>>>>>>>>>> issue appears related to number of VTEPs I use, no problem > >>>>>>>>>>>>> with 1 or 2 > >>>>>>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance > >>>>>>>>>>>>> from BTF > >>>>>>>>>>>>> experts are appreciated :-). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks > >>>>>>>>>>>>> > >>>>>>>>>>>>> Vincent > >>>>>>>>>>> > >>>>>>>>>>> Sorry for previous top post > >>>>>>>>>>> > >>>>>>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o > >>>>>>>>>>> differently and added " [21] .rodata.cst32 PROGBITS > >>>>>>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 > >>>>>>>>>>> members > >>>>>>>>>>> > >>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, > >>>>>>>>>>> 0x1f48a90a, > >>>>>>>>>>> 0x2048a90a, } > >>>>>>>>>>> > >>>>>>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above > >>>>>>>>>>> VTEP_ENDPOINT > >>>>>>>>>>> member <=2. any reason why compiler would do that? > >>>>>>>>>> > >>>>>>>>>> Regarding to why compiler generates .rodata.cst32, the reason is > >>>>>>>>>> you have some 32-byte constants which needs to be saved > >>>>>>>>>> somewhere. > >>>>>>>>>> For example, > >>>>>>>>>> > >>>>>>>>>> $ cat t.c > >>>>>>>>>> struct t { > >>>>>>>>>> long c[2]; > >>>>>>>>>> int d[4]; > >>>>>>>>>> }; > >>>>>>>>>> struct t g; > >>>>>>>>>> int test() > >>>>>>>>>> { > >>>>>>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; > >>>>>>>>>> g = tmp; > >>>>>>>>>> return 0; > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> $ clang -target bpf -O2 -c t.c > >>>>>>>>>> $ llvm-readelf -S t.o > >>>>>>>>>> ... > >>>>>>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 > >>>>>>>>>> 0000a8 000020 > >>>>>>>>>> 20 AM 0 0 8 > >>>>>>>>>> ... > >>>>>>>>>> > >>>>>>>>>> In the above code, if you change the struct size, say from 32 > >>>>>>>>>> bytes to > >>>>>>>>>> 40 bytes, the rodata.cst32 will go away. > >>>>>>>>> > >>>>>>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize > >>>>>>>>> rodata.cst32 then > >>>>>>>> > >>>>>>>> Hi Yonghong, > >>>>>>>> > >>>>>>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux > >>>>>>>> and > >>>>>>>> stores BTF type info in btf.Spec.namedTypes, but the elf object > >>>>>>>> file > >>>>>>>> provided by user may have section like rodata.cst32 generated by > >>>>>>>> compiler that does not have accompanying BTF type info stored in > >>>>>>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be > >>>>>>>> guaranteed to have every BTF type info from application/user > >>>>>>>> provided > >>>>>>>> elf object file ? I guess there is no guarantee. > >>>>>>> > >>>>>>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of > >>>>>>> rodata.cst32 needs to be emitted, the type will be encoded in bpf > >>>>>>> program BTF. > >>>>>>> > >>>>>>> Did you actually find an issue with .rodata.cst32 section? Such a > >>>>>>> section is typically generated by the compiler for initial data > >>>>>>> inside the function and llvm bpf backend tries to inline the > >>>>>>> values through a bunch of load instructions. So even you see > >>>>>>> .rodata.cst32, typically you can safely ignore it. > >>>>>>> > >>>>>>>> > >>>>>>>> Vincent > >>>>>>> > >>>>>> > >>>>>> Hi Yonghong, > >>>>>> > >>>>>> Thanks for the reproducer. Couldn't figure out what to do with > >>>>>> .rodata.cst32, > >>>>>> since there are no symbols and no BTF info for that section. > >>>>>> > >>>>>> The values found in .rodata.cst32 are indeed inlined in the > >>>>>> bytecode as you > >>>>>> mentioned, so it seems like we can ignore it. > >>>>>> > >>>>>> Why does the compiler emit these sections? cilium/ebpf assumed up > >>>>>> until now > >>>>>> that all sections starting with '.rodata' are datasecs and must be > >>>>>> loaded into > >>>>>> the kernel, which of course needs accompanying BTF. > >>>>> > >>>>> The clang frontend emits these .rodata.* sections. In early days, > >>>>> kernel > >>>>> doesn't support global data so llvm bpf backend implements an > >>>>> optimization to inline these values. But llvm bpf backend didn't > >>>>> completely remove them as the backend doesn't have a global view > >>>>> whether these .rodata.* are being used in other places or not. > >>>>> > >>>>> Now, llvm bpf backend has better infrastructure and we probably can > >>>>> implement an IR pass to detect all uses of .rodata.*, inline these > >>>>> uses, and remove the .rodata.* global variable. > >>>>> > >>>>> You can check relocation section of the program text. If the .rodata.* > >>>>> section is referenced, you should preserve it. Otherwise, you can > >>>>> ignore that .rodata.* section. > >>>>> > >>>>>> > >>>>>> What other .rodata.* should we expect? > >>>>> > >>>>> Glancing through llvm code, you may see .rodata.{4,8,16,32}, > >>>>> .rodata.str*. > >>>>> > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> Timo > >>>> > >>>> Thanks for the replies all, very insightful. We were already doing > >>>> things mostly > >>>> right wrt. .rodata.*, but found a few subtle bugs walking through > >>>> the code again. > >>>> > >>>> I've gotten a hold of the ELF Vincent was trying to load, and I saw > >>>> a few things > >>>> that I found unusual. In his case, the values in cst32 are not > >>>> inlined. Instead, > >>>> this ELF has a .Lconstinit symbol pointing at the start of > >>>> .rodata.cst32, and it's > >>>> an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly > >>>> strict and requires > >>>> STT_OBJECTs to be global (for supporting non-static consts). > >>> > >>> There are two ways to resolve the issue. First, extend the loader > >>> support to handle STB_LOCAL as well. Or Second, change the code like > >>> struct t v = {1, 5, 29, ...}; > >>> to > >>> struct t v; > >>> __builtin_memset(&v, 0, sizeof(struct t)); > >>> v.field1 = ...; > >>> v.field2 = ...; > >>> > >>> > >>>> > >>>> --- > >>>> ~ llvm-readelf -ar bpf_lxc.o > >>>> > >>>> Symbol table '.symtab' contains 606 entries: > >>>> Num: Value Size Type Bind Vis Ndx Name > >>>> 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 > >>>> .Lconstinit > >>>> > >>>> Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: > >>>> Offset Info Type > >>>> Symbol's Value Symbol's Name > >>>> 0000000000007300 0000000200000001 R_BPF_64_64 > >>>> 0000000000000000 .Lconstinit > >>>> --- > >>>> > >>>> --- > >>>> ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o > >>>> warning: failed to compute relocation: R_BPF_64_64, Invalid data was > >>>> encountered while parsing the file > >>>> ... <2 more of these> ... > >>>> > >>>> Disassembly of section 2/7: > >>>> > >>>> 00000000000072f8 <LBB1_476>: > >>>> 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 > >>>> 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 > >>>> = 0 ll > >>>> 0000000000007300: R_BPF_64_64 .Lconstinit > >>>> 3682: 0f 82 00 00 00 00 00 00 r2 += r8 > >>>> 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) > >>>> 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 > >>>> > >>>> Disassembly of section .rodata.cst32: > >>>> > >>>> 0000000000000000 <.Lconstinit>: > >>>> 0: 82 36 4c 98 2e 56 00 00 <unknown> > >>>> 1: 82 36 4c 98 2e 55 00 00 <unknown> > >>>> --- > >>>> > >>>> > >>>> This symbol doesn't exist in the program. Worth noting is that the > >>>> code that accesses > >>>> this static data sits within a subscope, but not sure what the > >>>> effect of this would be. > >>>> > >>>> Vincent, maybe try removing the enclosing {} to see if that changes > >>>> anything? > >>>> > >>>> --- > >>>> static __always_inline int foo(struct __ctx_buff *ctx, > >>>> > >>>> ... <snip> ... > >>>> > >>>> { > >>>> int i; > >>>> > >>>> for (i = 0; i < VTEP_NUMS; i) { > >>>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { > >>>> vtep_mac = VTEP_MAC[i]; > >>>> break; > >>>> } > >>>> } > >>>> } > >>>> --- > >>>> > >>>> Is this perhaps something that needs to be addressed in the compiler? > >>> > >>> If you can give a reproducible test (with .c or .i file), I can take a > >>> look at what is missing in llvm compiler and improve it. > >>> > >> > >> not sure if it would help, here is my step to generate the bpf_lxc.o > >> object file with the .rodata.cst32 > >> > >> git clone https://github.com/f5devcentral/cilium.git > >> cd cilium; git checkout vli-vxlan; KERNEL=54 make -C bpf > >> llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf/bpf_lxc.o > > > > Thanks. I can reproduce the issue now. Will take a look > > and get back to you as soon as I got any concrete results. > > Okay, I found the reason. > > For the code, > > for (i = 0; i < VTEP_NUMS; i++) { > if (tunnel_endpoint == VTEP_ENDPOINT[i]) { > vtep_mac = VTEP_MAC[i]; > break; > } > } > > The compiler transformed to something like > > i = 0; if (tunnerl_endpoint == VTEP_ENDPOINT[0]) goto end; > i = 1; if (tunnerl_endpoint == VTEP_ENDPOINT[1]) goto end; > i = 2; if (tunnerl_endpoint == VTEP_ENDPOINT[2]) goto end; > i = 3; if (tunnerl_endpoint == VTEP_ENDPOINT[3]) goto end; > > end: > vtep_mac = VTEP_MAC[i]; > > The compiler cannot inline VTEP_MAC[i] since 'i' is not > a constant. Hence later we have a memory load from > a non-global .rodata section. > > As I mentioned earlier, there are two options to fix the issue. > First is for cilium to track and handle non-global .rodata > sections. And the second you can apply the below code change, > > diff --git a/bpf/node_config.h b/bpf/node_config.h > index 9783e44548..b80dd2b27b 100644 > --- a/bpf/node_config.h > +++ b/bpf/node_config.h > @@ -176,15 +176,15 @@ DEFINE_IPV6(HOST_IP, 0xbe, 0xef, 0x0, 0x0, 0x0, > 0x0, 0x0, 0x1, 0x0, 0x0, 0xa, 0x > #endif > > #ifdef ENABLE_VTEP > -#define VTEP_ENDPOINT (__u32[]){0xeb48a90a, 0xec48a90a, 0xed48a90a, > 0xee48a90a, } > +#define VTEP_NUMS 4 > +__u32 VTEP_ENDPOINT[VTEP_NUMS] = {0xeb48a90a, 0xec48a90a, 0xed48a90a, > 0xee48a90a}; > /* HEX representation of VTEP IP > * 10.169.72.235, 10.169.72.236, 10.169.72.237, 10.169.72.238 > */ > -#define VTEP_MAC (__u64[]){0x562e984c3682, 0x552e984c3682, > 0x542e984c3682, 0x532e984c3682} > +__u64 VTEP_MAC[VTEP_NUMS] = {0x562e984c3682, 0x552e984c3682, > 0x542e984c3682, 0x532e984c3682}; > /* VTEP MAC address > * 82:36:4c:89:2e:56, 82:36:4c:89:2e:55, 82:36:4c:89:2e:54, > 82:36:4c:89:2e:53 > */ > -#define VTEP_NUMS 4 > #endif > Thank you Yonghong for the suggestion, the original code is kind of hack from me to work around some issue :). now we decided to abandon above code and use BPF hash map for VTEP lookup in Cilium to avoid this issue and to be more flexible, so it is up to cilium/ebpf to decide to address the rodata or not. > /* It appears that we can support around the below number of prefixes > in an > > > > >> > >>>> > >>>> > >>>> Thanks again, > >>>> > >>>> Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-08 18:28 ` Vincent Li @ 2022-02-08 18:48 ` Vincent Li 2022-02-09 19:34 ` Yonghong Song 0 siblings, 1 reply; 21+ messages in thread From: Vincent Li @ 2022-02-08 18:48 UTC (permalink / raw) To: Yonghong Song; +Cc: Timo Beckers, bpf On Tue, Feb 8, 2022 at 10:28 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > > On Mon, Feb 7, 2022 at 10:47 PM Yonghong Song <yhs@fb.com> wrote: > > > > > > > > On 2/4/22 1:22 PM, Yonghong Song wrote: > > > > > > > > > On 2/4/22 11:39 AM, Vincent Li wrote: > > >> On Fri, Feb 4, 2022 at 10:04 AM Yonghong Song <yhs@fb.com> wrote: > > >>> > > >>> > > >>> > > >>> On 2/4/22 3:11 AM, Timo Beckers wrote: > > >>>> On 2/3/22 03:11, Yonghong Song wrote: > > >>>>> > > >>>>> > > >>>>> On 2/2/22 5:47 AM, Timo Beckers wrote: > > >>>>>> On 2/2/22 08:17, Yonghong Song wrote: > > >>>>>>> > > >>>>>>> > > >>>>>>> On 2/1/22 10:07 AM, Vincent Li wrote: > > >>>>>>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li > > >>>>>>>> <vincent.mc.li@gmail.com> wrote: > > >>>>>>>>> > > >>>>>>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: > > >>>>>>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li > > >>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>> this is macro I suspected in my implementation that could > > >>>>>>>>>>>> cause issue with BTF > > >>>>>>>>>>>> > > >>>>>>>>>>>> #define ENABLE_VTEP 1 > > >>>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, > > >>>>>>>>>>>> 0x1f48a90a, > > >>>>>>>>>>>> 0x2048a90a, } > > >>>>>>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, > > >>>>>>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } > > >>>>>>>>>>>> #define VTEP_NUMS 4 > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li > > >>>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Hi > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> While developing Cilium VTEP integration feature > > >>>>>>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a > > >>>>>>>>>>>>> strange issue > > >>>>>>>>>>>>> that seems related to BTF and probably caused by my specific > > >>>>>>>>>>>>> implementation, the issue is described in > > >>>>>>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know > > >>>>>>>>>>>>> much about > > >>>>>>>>>>>>> BTF and not sure if my implementation is seriously flawed > > >>>>>>>>>>>>> or just some > > >>>>>>>>>>>>> implementation bug or maybe not compatible with BTF. > > >>>>>>>>>>>>> Strangely, the > > >>>>>>>>>>>>> issue appears related to number of VTEPs I use, no problem > > >>>>>>>>>>>>> with 1 or 2 > > >>>>>>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance > > >>>>>>>>>>>>> from BTF > > >>>>>>>>>>>>> experts are appreciated :-). > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Vincent > > >>>>>>>>>>> > > >>>>>>>>>>> Sorry for previous top post > > >>>>>>>>>>> > > >>>>>>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o > > >>>>>>>>>>> differently and added " [21] .rodata.cst32 PROGBITS > > >>>>>>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 > > >>>>>>>>>>> members > > >>>>>>>>>>> > > >>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, > > >>>>>>>>>>> 0x1f48a90a, > > >>>>>>>>>>> 0x2048a90a, } > > >>>>>>>>>>> > > >>>>>>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above > > >>>>>>>>>>> VTEP_ENDPOINT > > >>>>>>>>>>> member <=2. any reason why compiler would do that? > > >>>>>>>>>> > > >>>>>>>>>> Regarding to why compiler generates .rodata.cst32, the reason is > > >>>>>>>>>> you have some 32-byte constants which needs to be saved > > >>>>>>>>>> somewhere. > > >>>>>>>>>> For example, > > >>>>>>>>>> > > >>>>>>>>>> $ cat t.c > > >>>>>>>>>> struct t { > > >>>>>>>>>> long c[2]; > > >>>>>>>>>> int d[4]; > > >>>>>>>>>> }; > > >>>>>>>>>> struct t g; > > >>>>>>>>>> int test() > > >>>>>>>>>> { > > >>>>>>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; > > >>>>>>>>>> g = tmp; > > >>>>>>>>>> return 0; > > >>>>>>>>>> } > > >>>>>>>>>> > > >>>>>>>>>> $ clang -target bpf -O2 -c t.c > > >>>>>>>>>> $ llvm-readelf -S t.o > > >>>>>>>>>> ... > > >>>>>>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 > > >>>>>>>>>> 0000a8 000020 > > >>>>>>>>>> 20 AM 0 0 8 > > >>>>>>>>>> ... > > >>>>>>>>>> > > >>>>>>>>>> In the above code, if you change the struct size, say from 32 > > >>>>>>>>>> bytes to > > >>>>>>>>>> 40 bytes, the rodata.cst32 will go away. > > >>>>>>>>> > > >>>>>>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize > > >>>>>>>>> rodata.cst32 then > > >>>>>>>> > > >>>>>>>> Hi Yonghong, > > >>>>>>>> > > >>>>>>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux > > >>>>>>>> and > > >>>>>>>> stores BTF type info in btf.Spec.namedTypes, but the elf object > > >>>>>>>> file > > >>>>>>>> provided by user may have section like rodata.cst32 generated by > > >>>>>>>> compiler that does not have accompanying BTF type info stored in > > >>>>>>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be > > >>>>>>>> guaranteed to have every BTF type info from application/user > > >>>>>>>> provided > > >>>>>>>> elf object file ? I guess there is no guarantee. > > >>>>>>> > > >>>>>>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of > > >>>>>>> rodata.cst32 needs to be emitted, the type will be encoded in bpf > > >>>>>>> program BTF. > > >>>>>>> > > >>>>>>> Did you actually find an issue with .rodata.cst32 section? Such a > > >>>>>>> section is typically generated by the compiler for initial data > > >>>>>>> inside the function and llvm bpf backend tries to inline the > > >>>>>>> values through a bunch of load instructions. So even you see > > >>>>>>> .rodata.cst32, typically you can safely ignore it. > > >>>>>>> > > >>>>>>>> > > >>>>>>>> Vincent > > >>>>>>> > > >>>>>> > > >>>>>> Hi Yonghong, > > >>>>>> > > >>>>>> Thanks for the reproducer. Couldn't figure out what to do with > > >>>>>> .rodata.cst32, > > >>>>>> since there are no symbols and no BTF info for that section. > > >>>>>> > > >>>>>> The values found in .rodata.cst32 are indeed inlined in the > > >>>>>> bytecode as you > > >>>>>> mentioned, so it seems like we can ignore it. > > >>>>>> > > >>>>>> Why does the compiler emit these sections? cilium/ebpf assumed up > > >>>>>> until now > > >>>>>> that all sections starting with '.rodata' are datasecs and must be > > >>>>>> loaded into > > >>>>>> the kernel, which of course needs accompanying BTF. > > >>>>> > > >>>>> The clang frontend emits these .rodata.* sections. In early days, > > >>>>> kernel > > >>>>> doesn't support global data so llvm bpf backend implements an > > >>>>> optimization to inline these values. But llvm bpf backend didn't > > >>>>> completely remove them as the backend doesn't have a global view > > >>>>> whether these .rodata.* are being used in other places or not. > > >>>>> > > >>>>> Now, llvm bpf backend has better infrastructure and we probably can > > >>>>> implement an IR pass to detect all uses of .rodata.*, inline these > > >>>>> uses, and remove the .rodata.* global variable. > > >>>>> > > >>>>> You can check relocation section of the program text. If the .rodata.* > > >>>>> section is referenced, you should preserve it. Otherwise, you can > > >>>>> ignore that .rodata.* section. > > >>>>> > > >>>>>> > > >>>>>> What other .rodata.* should we expect? > > >>>>> > > >>>>> Glancing through llvm code, you may see .rodata.{4,8,16,32}, > > >>>>> .rodata.str*. > > >>>>> > > >>>>>> > > >>>>>> Thanks, > > >>>>>> > > >>>>>> Timo > > >>>> > > >>>> Thanks for the replies all, very insightful. We were already doing > > >>>> things mostly > > >>>> right wrt. .rodata.*, but found a few subtle bugs walking through > > >>>> the code again. > > >>>> > > >>>> I've gotten a hold of the ELF Vincent was trying to load, and I saw > > >>>> a few things > > >>>> that I found unusual. In his case, the values in cst32 are not > > >>>> inlined. Instead, > > >>>> this ELF has a .Lconstinit symbol pointing at the start of > > >>>> .rodata.cst32, and it's > > >>>> an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly > > >>>> strict and requires > > >>>> STT_OBJECTs to be global (for supporting non-static consts). > > >>> > > >>> There are two ways to resolve the issue. First, extend the loader > > >>> support to handle STB_LOCAL as well. Or Second, change the code like > > >>> struct t v = {1, 5, 29, ...}; > > >>> to > > >>> struct t v; > > >>> __builtin_memset(&v, 0, sizeof(struct t)); > > >>> v.field1 = ...; > > >>> v.field2 = ...; > > >>> > > >>> > > >>>> > > >>>> --- > > >>>> ~ llvm-readelf -ar bpf_lxc.o > > >>>> > > >>>> Symbol table '.symtab' contains 606 entries: > > >>>> Num: Value Size Type Bind Vis Ndx Name > > >>>> 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 > > >>>> .Lconstinit > > >>>> > > >>>> Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: > > >>>> Offset Info Type > > >>>> Symbol's Value Symbol's Name > > >>>> 0000000000007300 0000000200000001 R_BPF_64_64 > > >>>> 0000000000000000 .Lconstinit > > >>>> --- > > >>>> > > >>>> --- > > >>>> ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o > > >>>> warning: failed to compute relocation: R_BPF_64_64, Invalid data was > > >>>> encountered while parsing the file > > >>>> ... <2 more of these> ... > > >>>> > > >>>> Disassembly of section 2/7: > > >>>> > > >>>> 00000000000072f8 <LBB1_476>: > > >>>> 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 > > >>>> 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 > > >>>> = 0 ll > > >>>> 0000000000007300: R_BPF_64_64 .Lconstinit > > >>>> 3682: 0f 82 00 00 00 00 00 00 r2 += r8 > > >>>> 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) > > >>>> 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 > > >>>> > > >>>> Disassembly of section .rodata.cst32: > > >>>> > > >>>> 0000000000000000 <.Lconstinit>: > > >>>> 0: 82 36 4c 98 2e 56 00 00 <unknown> > > >>>> 1: 82 36 4c 98 2e 55 00 00 <unknown> > > >>>> --- > > >>>> > > >>>> > > >>>> This symbol doesn't exist in the program. Worth noting is that the > > >>>> code that accesses > > >>>> this static data sits within a subscope, but not sure what the > > >>>> effect of this would be. > > >>>> > > >>>> Vincent, maybe try removing the enclosing {} to see if that changes > > >>>> anything? > > >>>> > > >>>> --- > > >>>> static __always_inline int foo(struct __ctx_buff *ctx, > > >>>> > > >>>> ... <snip> ... > > >>>> > > >>>> { > > >>>> int i; > > >>>> > > >>>> for (i = 0; i < VTEP_NUMS; i) { > > >>>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { > > >>>> vtep_mac = VTEP_MAC[i]; > > >>>> break; > > >>>> } > > >>>> } > > >>>> } > > >>>> --- > > >>>> > > >>>> Is this perhaps something that needs to be addressed in the compiler? > > >>> > > >>> If you can give a reproducible test (with .c or .i file), I can take a > > >>> look at what is missing in llvm compiler and improve it. > > >>> > > >> > > >> not sure if it would help, here is my step to generate the bpf_lxc.o > > >> object file with the .rodata.cst32 > > >> > > >> git clone https://github.com/f5devcentral/cilium.git > > >> cd cilium; git checkout vli-vxlan; KERNEL=54 make -C bpf > > >> llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf/bpf_lxc.o > > > > > > Thanks. I can reproduce the issue now. Will take a look > > > and get back to you as soon as I got any concrete results. > > > > Okay, I found the reason. > > > > For the code, > > > > for (i = 0; i < VTEP_NUMS; i++) { > > if (tunnel_endpoint == VTEP_ENDPOINT[i]) { > > vtep_mac = VTEP_MAC[i]; > > break; > > } > > } > > > > The compiler transformed to something like > > > > i = 0; if (tunnerl_endpoint == VTEP_ENDPOINT[0]) goto end; > > i = 1; if (tunnerl_endpoint == VTEP_ENDPOINT[1]) goto end; > > i = 2; if (tunnerl_endpoint == VTEP_ENDPOINT[2]) goto end; > > i = 3; if (tunnerl_endpoint == VTEP_ENDPOINT[3]) goto end; > > > > end: > > vtep_mac = VTEP_MAC[i]; > > > > The compiler cannot inline VTEP_MAC[i] since 'i' is not > > a constant. Hence later we have a memory load from > > a non-global .rodata section. > > > > As I mentioned earlier, there are two options to fix the issue. > > First is for cilium to track and handle non-global .rodata > > sections. And the second you can apply the below code change, > > > > diff --git a/bpf/node_config.h b/bpf/node_config.h > > index 9783e44548..b80dd2b27b 100644 > > --- a/bpf/node_config.h > > +++ b/bpf/node_config.h > > @@ -176,15 +176,15 @@ DEFINE_IPV6(HOST_IP, 0xbe, 0xef, 0x0, 0x0, 0x0, > > 0x0, 0x0, 0x1, 0x0, 0x0, 0xa, 0x > > #endif > > > > #ifdef ENABLE_VTEP > > -#define VTEP_ENDPOINT (__u32[]){0xeb48a90a, 0xec48a90a, 0xed48a90a, > > 0xee48a90a, } > > +#define VTEP_NUMS 4 > > +__u32 VTEP_ENDPOINT[VTEP_NUMS] = {0xeb48a90a, 0xec48a90a, 0xed48a90a, > > 0xee48a90a}; > > /* HEX representation of VTEP IP > > * 10.169.72.235, 10.169.72.236, 10.169.72.237, 10.169.72.238 > > */ > > -#define VTEP_MAC (__u64[]){0x562e984c3682, 0x552e984c3682, > > 0x542e984c3682, 0x532e984c3682} > > +__u64 VTEP_MAC[VTEP_NUMS] = {0x562e984c3682, 0x552e984c3682, > > 0x542e984c3682, 0x532e984c3682}; > > /* VTEP MAC address > > * 82:36:4c:89:2e:56, 82:36:4c:89:2e:55, 82:36:4c:89:2e:54, > > 82:36:4c:89:2e:53 > > */ > > -#define VTEP_NUMS 4 > > #endif > > > I may misunderstand you, I thought your suggestion would stop compiler generating .rodata.cst32, but it appears the compiler still generated .rodata.cst32 after applying your changes readelf -e bpf/bpf_lxc.o | grep 'rodata' [51] .rodata.cst32 PROGBITS 0000000000000000 00045f48 > Thank you Yonghong for the suggestion, the original code is kind of > hack from me to work around some issue :). now we decided to abandon > above code and use BPF hash map for VTEP lookup in Cilium to avoid > this issue and to be more flexible, so it is up to cilium/ebpf to > decide to address the rodata or not. > > > /* It appears that we can support around the below number of prefixes > > in an > > > > > > > >> > > >>>> > > >>>> > > >>>> Thanks again, > > >>>> > > >>>> Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-08 18:48 ` Vincent Li @ 2022-02-09 19:34 ` Yonghong Song 2022-02-09 19:53 ` Vincent Li 0 siblings, 1 reply; 21+ messages in thread From: Yonghong Song @ 2022-02-09 19:34 UTC (permalink / raw) To: Vincent Li; +Cc: Timo Beckers, bpf On 2/8/22 10:48 AM, Vincent Li wrote: > On Tue, Feb 8, 2022 at 10:28 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >> >> On Mon, Feb 7, 2022 at 10:47 PM Yonghong Song <yhs@fb.com> wrote: >>> >>> >>> >>> On 2/4/22 1:22 PM, Yonghong Song wrote: >>>> >>>> >>>> On 2/4/22 11:39 AM, Vincent Li wrote: >>>>> On Fri, Feb 4, 2022 at 10:04 AM Yonghong Song <yhs@fb.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On 2/4/22 3:11 AM, Timo Beckers wrote: >>>>>>> On 2/3/22 03:11, Yonghong Song wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 2/2/22 5:47 AM, Timo Beckers wrote: >>>>>>>>> On 2/2/22 08:17, Yonghong Song wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2/1/22 10:07 AM, Vincent Li wrote: >>>>>>>>>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li >>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: >>>>>>>>>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li >>>>>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> this is macro I suspected in my implementation that could >>>>>>>>>>>>>>> cause issue with BTF >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> #define ENABLE_VTEP 1 >>>>>>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, >>>>>>>>>>>>>>> 0x1f48a90a, >>>>>>>>>>>>>>> 0x2048a90a, } >>>>>>>>>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, >>>>>>>>>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } >>>>>>>>>>>>>>> #define VTEP_NUMS 4 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li >>>>>>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> While developing Cilium VTEP integration feature >>>>>>>>>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a >>>>>>>>>>>>>>>> strange issue >>>>>>>>>>>>>>>> that seems related to BTF and probably caused by my specific >>>>>>>>>>>>>>>> implementation, the issue is described in >>>>>>>>>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know >>>>>>>>>>>>>>>> much about >>>>>>>>>>>>>>>> BTF and not sure if my implementation is seriously flawed >>>>>>>>>>>>>>>> or just some >>>>>>>>>>>>>>>> implementation bug or maybe not compatible with BTF. >>>>>>>>>>>>>>>> Strangely, the >>>>>>>>>>>>>>>> issue appears related to number of VTEPs I use, no problem >>>>>>>>>>>>>>>> with 1 or 2 >>>>>>>>>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance >>>>>>>>>>>>>>>> from BTF >>>>>>>>>>>>>>>> experts are appreciated :-). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Vincent >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry for previous top post >>>>>>>>>>>>>> >>>>>>>>>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o >>>>>>>>>>>>>> differently and added " [21] .rodata.cst32 PROGBITS >>>>>>>>>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 >>>>>>>>>>>>>> members >>>>>>>>>>>>>> >>>>>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, >>>>>>>>>>>>>> 0x1f48a90a, >>>>>>>>>>>>>> 0x2048a90a, } >>>>>>>>>>>>>> >>>>>>>>>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above >>>>>>>>>>>>>> VTEP_ENDPOINT >>>>>>>>>>>>>> member <=2. any reason why compiler would do that? >>>>>>>>>>>>> >>>>>>>>>>>>> Regarding to why compiler generates .rodata.cst32, the reason is >>>>>>>>>>>>> you have some 32-byte constants which needs to be saved >>>>>>>>>>>>> somewhere. >>>>>>>>>>>>> For example, >>>>>>>>>>>>> >>>>>>>>>>>>> $ cat t.c >>>>>>>>>>>>> struct t { >>>>>>>>>>>>> long c[2]; >>>>>>>>>>>>> int d[4]; >>>>>>>>>>>>> }; >>>>>>>>>>>>> struct t g; >>>>>>>>>>>>> int test() >>>>>>>>>>>>> { >>>>>>>>>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; >>>>>>>>>>>>> g = tmp; >>>>>>>>>>>>> return 0; >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> $ clang -target bpf -O2 -c t.c >>>>>>>>>>>>> $ llvm-readelf -S t.o >>>>>>>>>>>>> ... >>>>>>>>>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 >>>>>>>>>>>>> 0000a8 000020 >>>>>>>>>>>>> 20 AM 0 0 8 >>>>>>>>>>>>> ... >>>>>>>>>>>>> >>>>>>>>>>>>> In the above code, if you change the struct size, say from 32 >>>>>>>>>>>>> bytes to >>>>>>>>>>>>> 40 bytes, the rodata.cst32 will go away. >>>>>>>>>>>> >>>>>>>>>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize >>>>>>>>>>>> rodata.cst32 then >>>>>>>>>>> >>>>>>>>>>> Hi Yonghong, >>>>>>>>>>> >>>>>>>>>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux >>>>>>>>>>> and >>>>>>>>>>> stores BTF type info in btf.Spec.namedTypes, but the elf object >>>>>>>>>>> file >>>>>>>>>>> provided by user may have section like rodata.cst32 generated by >>>>>>>>>>> compiler that does not have accompanying BTF type info stored in >>>>>>>>>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be >>>>>>>>>>> guaranteed to have every BTF type info from application/user >>>>>>>>>>> provided >>>>>>>>>>> elf object file ? I guess there is no guarantee. >>>>>>>>>> >>>>>>>>>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of >>>>>>>>>> rodata.cst32 needs to be emitted, the type will be encoded in bpf >>>>>>>>>> program BTF. >>>>>>>>>> >>>>>>>>>> Did you actually find an issue with .rodata.cst32 section? Such a >>>>>>>>>> section is typically generated by the compiler for initial data >>>>>>>>>> inside the function and llvm bpf backend tries to inline the >>>>>>>>>> values through a bunch of load instructions. So even you see >>>>>>>>>> .rodata.cst32, typically you can safely ignore it. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Vincent >>>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Yonghong, >>>>>>>>> >>>>>>>>> Thanks for the reproducer. Couldn't figure out what to do with >>>>>>>>> .rodata.cst32, >>>>>>>>> since there are no symbols and no BTF info for that section. >>>>>>>>> >>>>>>>>> The values found in .rodata.cst32 are indeed inlined in the >>>>>>>>> bytecode as you >>>>>>>>> mentioned, so it seems like we can ignore it. >>>>>>>>> >>>>>>>>> Why does the compiler emit these sections? cilium/ebpf assumed up >>>>>>>>> until now >>>>>>>>> that all sections starting with '.rodata' are datasecs and must be >>>>>>>>> loaded into >>>>>>>>> the kernel, which of course needs accompanying BTF. >>>>>>>> >>>>>>>> The clang frontend emits these .rodata.* sections. In early days, >>>>>>>> kernel >>>>>>>> doesn't support global data so llvm bpf backend implements an >>>>>>>> optimization to inline these values. But llvm bpf backend didn't >>>>>>>> completely remove them as the backend doesn't have a global view >>>>>>>> whether these .rodata.* are being used in other places or not. >>>>>>>> >>>>>>>> Now, llvm bpf backend has better infrastructure and we probably can >>>>>>>> implement an IR pass to detect all uses of .rodata.*, inline these >>>>>>>> uses, and remove the .rodata.* global variable. >>>>>>>> >>>>>>>> You can check relocation section of the program text. If the .rodata.* >>>>>>>> section is referenced, you should preserve it. Otherwise, you can >>>>>>>> ignore that .rodata.* section. >>>>>>>> >>>>>>>>> >>>>>>>>> What other .rodata.* should we expect? >>>>>>>> >>>>>>>> Glancing through llvm code, you may see .rodata.{4,8,16,32}, >>>>>>>> .rodata.str*. >>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Timo >>>>>>> >>>>>>> Thanks for the replies all, very insightful. We were already doing >>>>>>> things mostly >>>>>>> right wrt. .rodata.*, but found a few subtle bugs walking through >>>>>>> the code again. >>>>>>> >>>>>>> I've gotten a hold of the ELF Vincent was trying to load, and I saw >>>>>>> a few things >>>>>>> that I found unusual. In his case, the values in cst32 are not >>>>>>> inlined. Instead, >>>>>>> this ELF has a .Lconstinit symbol pointing at the start of >>>>>>> .rodata.cst32, and it's >>>>>>> an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly >>>>>>> strict and requires >>>>>>> STT_OBJECTs to be global (for supporting non-static consts). >>>>>> >>>>>> There are two ways to resolve the issue. First, extend the loader >>>>>> support to handle STB_LOCAL as well. Or Second, change the code like >>>>>> struct t v = {1, 5, 29, ...}; >>>>>> to >>>>>> struct t v; >>>>>> __builtin_memset(&v, 0, sizeof(struct t)); >>>>>> v.field1 = ...; >>>>>> v.field2 = ...; >>>>>> >>>>>> >>>>>>> >>>>>>> --- >>>>>>> ~ llvm-readelf -ar bpf_lxc.o >>>>>>> >>>>>>> Symbol table '.symtab' contains 606 entries: >>>>>>> Num: Value Size Type Bind Vis Ndx Name >>>>>>> 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 >>>>>>> .Lconstinit >>>>>>> >>>>>>> Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: >>>>>>> Offset Info Type >>>>>>> Symbol's Value Symbol's Name >>>>>>> 0000000000007300 0000000200000001 R_BPF_64_64 >>>>>>> 0000000000000000 .Lconstinit >>>>>>> --- >>>>>>> >>>>>>> --- >>>>>>> ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o >>>>>>> warning: failed to compute relocation: R_BPF_64_64, Invalid data was >>>>>>> encountered while parsing the file >>>>>>> ... <2 more of these> ... >>>>>>> >>>>>>> Disassembly of section 2/7: >>>>>>> >>>>>>> 00000000000072f8 <LBB1_476>: >>>>>>> 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 >>>>>>> 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 >>>>>>> = 0 ll >>>>>>> 0000000000007300: R_BPF_64_64 .Lconstinit >>>>>>> 3682: 0f 82 00 00 00 00 00 00 r2 += r8 >>>>>>> 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) >>>>>>> 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 >>>>>>> >>>>>>> Disassembly of section .rodata.cst32: >>>>>>> >>>>>>> 0000000000000000 <.Lconstinit>: >>>>>>> 0: 82 36 4c 98 2e 56 00 00 <unknown> >>>>>>> 1: 82 36 4c 98 2e 55 00 00 <unknown> >>>>>>> --- >>>>>>> >>>>>>> >>>>>>> This symbol doesn't exist in the program. Worth noting is that the >>>>>>> code that accesses >>>>>>> this static data sits within a subscope, but not sure what the >>>>>>> effect of this would be. >>>>>>> >>>>>>> Vincent, maybe try removing the enclosing {} to see if that changes >>>>>>> anything? >>>>>>> >>>>>>> --- >>>>>>> static __always_inline int foo(struct __ctx_buff *ctx, >>>>>>> >>>>>>> ... <snip> ... >>>>>>> >>>>>>> { >>>>>>> int i; >>>>>>> >>>>>>> for (i = 0; i < VTEP_NUMS; i) { >>>>>>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { >>>>>>> vtep_mac = VTEP_MAC[i]; >>>>>>> break; >>>>>>> } >>>>>>> } >>>>>>> } >>>>>>> --- >>>>>>> >>>>>>> Is this perhaps something that needs to be addressed in the compiler? >>>>>> >>>>>> If you can give a reproducible test (with .c or .i file), I can take a >>>>>> look at what is missing in llvm compiler and improve it. >>>>>> >>>>> >>>>> not sure if it would help, here is my step to generate the bpf_lxc.o >>>>> object file with the .rodata.cst32 >>>>> >>>>> git clone https://github.com/f5devcentral/cilium.git >>>>> cd cilium; git checkout vli-vxlan; KERNEL=54 make -C bpf >>>>> llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf/bpf_lxc.o >>>> >>>> Thanks. I can reproduce the issue now. Will take a look >>>> and get back to you as soon as I got any concrete results. >>> >>> Okay, I found the reason. >>> >>> For the code, >>> >>> for (i = 0; i < VTEP_NUMS; i++) { >>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { >>> vtep_mac = VTEP_MAC[i]; >>> break; >>> } >>> } >>> >>> The compiler transformed to something like >>> >>> i = 0; if (tunnerl_endpoint == VTEP_ENDPOINT[0]) goto end; >>> i = 1; if (tunnerl_endpoint == VTEP_ENDPOINT[1]) goto end; >>> i = 2; if (tunnerl_endpoint == VTEP_ENDPOINT[2]) goto end; >>> i = 3; if (tunnerl_endpoint == VTEP_ENDPOINT[3]) goto end; >>> >>> end: >>> vtep_mac = VTEP_MAC[i]; >>> >>> The compiler cannot inline VTEP_MAC[i] since 'i' is not >>> a constant. Hence later we have a memory load from >>> a non-global .rodata section. >>> >>> As I mentioned earlier, there are two options to fix the issue. >>> First is for cilium to track and handle non-global .rodata >>> sections. And the second you can apply the below code change, >>> >>> diff --git a/bpf/node_config.h b/bpf/node_config.h >>> index 9783e44548..b80dd2b27b 100644 >>> --- a/bpf/node_config.h >>> +++ b/bpf/node_config.h >>> @@ -176,15 +176,15 @@ DEFINE_IPV6(HOST_IP, 0xbe, 0xef, 0x0, 0x0, 0x0, >>> 0x0, 0x0, 0x1, 0x0, 0x0, 0xa, 0x >>> #endif >>> >>> #ifdef ENABLE_VTEP >>> -#define VTEP_ENDPOINT (__u32[]){0xeb48a90a, 0xec48a90a, 0xed48a90a, >>> 0xee48a90a, } >>> +#define VTEP_NUMS 4 >>> +__u32 VTEP_ENDPOINT[VTEP_NUMS] = {0xeb48a90a, 0xec48a90a, 0xed48a90a, >>> 0xee48a90a}; >>> /* HEX representation of VTEP IP >>> * 10.169.72.235, 10.169.72.236, 10.169.72.237, 10.169.72.238 >>> */ >>> -#define VTEP_MAC (__u64[]){0x562e984c3682, 0x552e984c3682, >>> 0x542e984c3682, 0x532e984c3682} >>> +__u64 VTEP_MAC[VTEP_NUMS] = {0x562e984c3682, 0x552e984c3682, >>> 0x542e984c3682, 0x532e984c3682}; >>> /* VTEP MAC address >>> * 82:36:4c:89:2e:56, 82:36:4c:89:2e:55, 82:36:4c:89:2e:54, >>> 82:36:4c:89:2e:53 >>> */ >>> -#define VTEP_NUMS 4 >>> #endif >>> >> > > I may misunderstand you, I thought your suggestion would stop compiler > generating .rodata.cst32, but it appears > the compiler still generated .rodata.cst32 after applying your changes > > readelf -e bpf/bpf_lxc.o | grep 'rodata' > > [51] .rodata.cst32 PROGBITS 0000000000000000 00045f48 In my case, I didn't have rodata section. compiled with `KERNEL=54 make -C bpf` clang is latest upstream: clang --version clang version 15.0.0 (https://github.com/llvm/llvm-project.git 4f97aa7e1d70f1b259e5fddd85de05235b01b192) But I think any recent compiler should have the same result. readelf -e bpf/bpf_lxc.o | grep 'rodata' <no output> > >> Thank you Yonghong for the suggestion, the original code is kind of >> hack from me to work around some issue :). now we decided to abandon >> above code and use BPF hash map for VTEP lookup in Cilium to avoid >> this issue and to be more flexible, so it is up to cilium/ebpf to >> decide to address the rodata or not. >> >>> /* It appears that we can support around the below number of prefixes >>> in an >>> >>>> >>>>> >>>>>>> >>>>>>> >>>>>>> Thanks again, >>>>>>> >>>>>>> Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-09 19:34 ` Yonghong Song @ 2022-02-09 19:53 ` Vincent Li 2022-05-06 14:11 ` Timo Beckers 0 siblings, 1 reply; 21+ messages in thread From: Vincent Li @ 2022-02-09 19:53 UTC (permalink / raw) To: Yonghong Song; +Cc: Timo Beckers, bpf On Wed, Feb 9, 2022 at 11:34 AM Yonghong Song <yhs@fb.com> wrote: > > > > On 2/8/22 10:48 AM, Vincent Li wrote: > > On Tue, Feb 8, 2022 at 10:28 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > >> > >> On Mon, Feb 7, 2022 at 10:47 PM Yonghong Song <yhs@fb.com> wrote: > >>> > >>> > >>> > >>> On 2/4/22 1:22 PM, Yonghong Song wrote: > >>>> > >>>> > >>>> On 2/4/22 11:39 AM, Vincent Li wrote: > >>>>> On Fri, Feb 4, 2022 at 10:04 AM Yonghong Song <yhs@fb.com> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> On 2/4/22 3:11 AM, Timo Beckers wrote: > >>>>>>> On 2/3/22 03:11, Yonghong Song wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> On 2/2/22 5:47 AM, Timo Beckers wrote: > >>>>>>>>> On 2/2/22 08:17, Yonghong Song wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On 2/1/22 10:07 AM, Vincent Li wrote: > >>>>>>>>>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li > >>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: > >>>>>>>>>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li > >>>>>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> this is macro I suspected in my implementation that could > >>>>>>>>>>>>>>> cause issue with BTF > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> #define ENABLE_VTEP 1 > >>>>>>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, > >>>>>>>>>>>>>>> 0x1f48a90a, > >>>>>>>>>>>>>>> 0x2048a90a, } > >>>>>>>>>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, > >>>>>>>>>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } > >>>>>>>>>>>>>>> #define VTEP_NUMS 4 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li > >>>>>>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> While developing Cilium VTEP integration feature > >>>>>>>>>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a > >>>>>>>>>>>>>>>> strange issue > >>>>>>>>>>>>>>>> that seems related to BTF and probably caused by my specific > >>>>>>>>>>>>>>>> implementation, the issue is described in > >>>>>>>>>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know > >>>>>>>>>>>>>>>> much about > >>>>>>>>>>>>>>>> BTF and not sure if my implementation is seriously flawed > >>>>>>>>>>>>>>>> or just some > >>>>>>>>>>>>>>>> implementation bug or maybe not compatible with BTF. > >>>>>>>>>>>>>>>> Strangely, the > >>>>>>>>>>>>>>>> issue appears related to number of VTEPs I use, no problem > >>>>>>>>>>>>>>>> with 1 or 2 > >>>>>>>>>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance > >>>>>>>>>>>>>>>> from BTF > >>>>>>>>>>>>>>>> experts are appreciated :-). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Vincent > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Sorry for previous top post > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o > >>>>>>>>>>>>>> differently and added " [21] .rodata.cst32 PROGBITS > >>>>>>>>>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 > >>>>>>>>>>>>>> members > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, > >>>>>>>>>>>>>> 0x1f48a90a, > >>>>>>>>>>>>>> 0x2048a90a, } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above > >>>>>>>>>>>>>> VTEP_ENDPOINT > >>>>>>>>>>>>>> member <=2. any reason why compiler would do that? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Regarding to why compiler generates .rodata.cst32, the reason is > >>>>>>>>>>>>> you have some 32-byte constants which needs to be saved > >>>>>>>>>>>>> somewhere. > >>>>>>>>>>>>> For example, > >>>>>>>>>>>>> > >>>>>>>>>>>>> $ cat t.c > >>>>>>>>>>>>> struct t { > >>>>>>>>>>>>> long c[2]; > >>>>>>>>>>>>> int d[4]; > >>>>>>>>>>>>> }; > >>>>>>>>>>>>> struct t g; > >>>>>>>>>>>>> int test() > >>>>>>>>>>>>> { > >>>>>>>>>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; > >>>>>>>>>>>>> g = tmp; > >>>>>>>>>>>>> return 0; > >>>>>>>>>>>>> } > >>>>>>>>>>>>> > >>>>>>>>>>>>> $ clang -target bpf -O2 -c t.c > >>>>>>>>>>>>> $ llvm-readelf -S t.o > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 > >>>>>>>>>>>>> 0000a8 000020 > >>>>>>>>>>>>> 20 AM 0 0 8 > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> > >>>>>>>>>>>>> In the above code, if you change the struct size, say from 32 > >>>>>>>>>>>>> bytes to > >>>>>>>>>>>>> 40 bytes, the rodata.cst32 will go away. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize > >>>>>>>>>>>> rodata.cst32 then > >>>>>>>>>>> > >>>>>>>>>>> Hi Yonghong, > >>>>>>>>>>> > >>>>>>>>>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux > >>>>>>>>>>> and > >>>>>>>>>>> stores BTF type info in btf.Spec.namedTypes, but the elf object > >>>>>>>>>>> file > >>>>>>>>>>> provided by user may have section like rodata.cst32 generated by > >>>>>>>>>>> compiler that does not have accompanying BTF type info stored in > >>>>>>>>>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be > >>>>>>>>>>> guaranteed to have every BTF type info from application/user > >>>>>>>>>>> provided > >>>>>>>>>>> elf object file ? I guess there is no guarantee. > >>>>>>>>>> > >>>>>>>>>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of > >>>>>>>>>> rodata.cst32 needs to be emitted, the type will be encoded in bpf > >>>>>>>>>> program BTF. > >>>>>>>>>> > >>>>>>>>>> Did you actually find an issue with .rodata.cst32 section? Such a > >>>>>>>>>> section is typically generated by the compiler for initial data > >>>>>>>>>> inside the function and llvm bpf backend tries to inline the > >>>>>>>>>> values through a bunch of load instructions. So even you see > >>>>>>>>>> .rodata.cst32, typically you can safely ignore it. > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Vincent > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Hi Yonghong, > >>>>>>>>> > >>>>>>>>> Thanks for the reproducer. Couldn't figure out what to do with > >>>>>>>>> .rodata.cst32, > >>>>>>>>> since there are no symbols and no BTF info for that section. > >>>>>>>>> > >>>>>>>>> The values found in .rodata.cst32 are indeed inlined in the > >>>>>>>>> bytecode as you > >>>>>>>>> mentioned, so it seems like we can ignore it. > >>>>>>>>> > >>>>>>>>> Why does the compiler emit these sections? cilium/ebpf assumed up > >>>>>>>>> until now > >>>>>>>>> that all sections starting with '.rodata' are datasecs and must be > >>>>>>>>> loaded into > >>>>>>>>> the kernel, which of course needs accompanying BTF. > >>>>>>>> > >>>>>>>> The clang frontend emits these .rodata.* sections. In early days, > >>>>>>>> kernel > >>>>>>>> doesn't support global data so llvm bpf backend implements an > >>>>>>>> optimization to inline these values. But llvm bpf backend didn't > >>>>>>>> completely remove them as the backend doesn't have a global view > >>>>>>>> whether these .rodata.* are being used in other places or not. > >>>>>>>> > >>>>>>>> Now, llvm bpf backend has better infrastructure and we probably can > >>>>>>>> implement an IR pass to detect all uses of .rodata.*, inline these > >>>>>>>> uses, and remove the .rodata.* global variable. > >>>>>>>> > >>>>>>>> You can check relocation section of the program text. If the .rodata.* > >>>>>>>> section is referenced, you should preserve it. Otherwise, you can > >>>>>>>> ignore that .rodata.* section. > >>>>>>>> > >>>>>>>>> > >>>>>>>>> What other .rodata.* should we expect? > >>>>>>>> > >>>>>>>> Glancing through llvm code, you may see .rodata.{4,8,16,32}, > >>>>>>>> .rodata.str*. > >>>>>>>> > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> > >>>>>>>>> Timo > >>>>>>> > >>>>>>> Thanks for the replies all, very insightful. We were already doing > >>>>>>> things mostly > >>>>>>> right wrt. .rodata.*, but found a few subtle bugs walking through > >>>>>>> the code again. > >>>>>>> > >>>>>>> I've gotten a hold of the ELF Vincent was trying to load, and I saw > >>>>>>> a few things > >>>>>>> that I found unusual. In his case, the values in cst32 are not > >>>>>>> inlined. Instead, > >>>>>>> this ELF has a .Lconstinit symbol pointing at the start of > >>>>>>> .rodata.cst32, and it's > >>>>>>> an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly > >>>>>>> strict and requires > >>>>>>> STT_OBJECTs to be global (for supporting non-static consts). > >>>>>> > >>>>>> There are two ways to resolve the issue. First, extend the loader > >>>>>> support to handle STB_LOCAL as well. Or Second, change the code like > >>>>>> struct t v = {1, 5, 29, ...}; > >>>>>> to > >>>>>> struct t v; > >>>>>> __builtin_memset(&v, 0, sizeof(struct t)); > >>>>>> v.field1 = ...; > >>>>>> v.field2 = ...; > >>>>>> > >>>>>> > >>>>>>> > >>>>>>> --- > >>>>>>> ~ llvm-readelf -ar bpf_lxc.o > >>>>>>> > >>>>>>> Symbol table '.symtab' contains 606 entries: > >>>>>>> Num: Value Size Type Bind Vis Ndx Name > >>>>>>> 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 > >>>>>>> .Lconstinit > >>>>>>> > >>>>>>> Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: > >>>>>>> Offset Info Type > >>>>>>> Symbol's Value Symbol's Name > >>>>>>> 0000000000007300 0000000200000001 R_BPF_64_64 > >>>>>>> 0000000000000000 .Lconstinit > >>>>>>> --- > >>>>>>> > >>>>>>> --- > >>>>>>> ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o > >>>>>>> warning: failed to compute relocation: R_BPF_64_64, Invalid data was > >>>>>>> encountered while parsing the file > >>>>>>> ... <2 more of these> ... > >>>>>>> > >>>>>>> Disassembly of section 2/7: > >>>>>>> > >>>>>>> 00000000000072f8 <LBB1_476>: > >>>>>>> 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 > >>>>>>> 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 > >>>>>>> = 0 ll > >>>>>>> 0000000000007300: R_BPF_64_64 .Lconstinit > >>>>>>> 3682: 0f 82 00 00 00 00 00 00 r2 += r8 > >>>>>>> 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) > >>>>>>> 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 > >>>>>>> > >>>>>>> Disassembly of section .rodata.cst32: > >>>>>>> > >>>>>>> 0000000000000000 <.Lconstinit>: > >>>>>>> 0: 82 36 4c 98 2e 56 00 00 <unknown> > >>>>>>> 1: 82 36 4c 98 2e 55 00 00 <unknown> > >>>>>>> --- > >>>>>>> > >>>>>>> > >>>>>>> This symbol doesn't exist in the program. Worth noting is that the > >>>>>>> code that accesses > >>>>>>> this static data sits within a subscope, but not sure what the > >>>>>>> effect of this would be. > >>>>>>> > >>>>>>> Vincent, maybe try removing the enclosing {} to see if that changes > >>>>>>> anything? > >>>>>>> > >>>>>>> --- > >>>>>>> static __always_inline int foo(struct __ctx_buff *ctx, > >>>>>>> > >>>>>>> ... <snip> ... > >>>>>>> > >>>>>>> { > >>>>>>> int i; > >>>>>>> > >>>>>>> for (i = 0; i < VTEP_NUMS; i) { > >>>>>>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { > >>>>>>> vtep_mac = VTEP_MAC[i]; > >>>>>>> break; > >>>>>>> } > >>>>>>> } > >>>>>>> } > >>>>>>> --- > >>>>>>> > >>>>>>> Is this perhaps something that needs to be addressed in the compiler? > >>>>>> > >>>>>> If you can give a reproducible test (with .c or .i file), I can take a > >>>>>> look at what is missing in llvm compiler and improve it. > >>>>>> > >>>>> > >>>>> not sure if it would help, here is my step to generate the bpf_lxc.o > >>>>> object file with the .rodata.cst32 > >>>>> > >>>>> git clone https://github.com/f5devcentral/cilium.git > >>>>> cd cilium; git checkout vli-vxlan; KERNEL=54 make -C bpf > >>>>> llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf/bpf_lxc.o > >>>> > >>>> Thanks. I can reproduce the issue now. Will take a look > >>>> and get back to you as soon as I got any concrete results. > >>> > >>> Okay, I found the reason. > >>> > >>> For the code, > >>> > >>> for (i = 0; i < VTEP_NUMS; i++) { > >>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { > >>> vtep_mac = VTEP_MAC[i]; > >>> break; > >>> } > >>> } > >>> > >>> The compiler transformed to something like > >>> > >>> i = 0; if (tunnerl_endpoint == VTEP_ENDPOINT[0]) goto end; > >>> i = 1; if (tunnerl_endpoint == VTEP_ENDPOINT[1]) goto end; > >>> i = 2; if (tunnerl_endpoint == VTEP_ENDPOINT[2]) goto end; > >>> i = 3; if (tunnerl_endpoint == VTEP_ENDPOINT[3]) goto end; > >>> > >>> end: > >>> vtep_mac = VTEP_MAC[i]; > >>> > >>> The compiler cannot inline VTEP_MAC[i] since 'i' is not > >>> a constant. Hence later we have a memory load from > >>> a non-global .rodata section. > >>> > >>> As I mentioned earlier, there are two options to fix the issue. > >>> First is for cilium to track and handle non-global .rodata > >>> sections. And the second you can apply the below code change, > >>> > >>> diff --git a/bpf/node_config.h b/bpf/node_config.h > >>> index 9783e44548..b80dd2b27b 100644 > >>> --- a/bpf/node_config.h > >>> +++ b/bpf/node_config.h > >>> @@ -176,15 +176,15 @@ DEFINE_IPV6(HOST_IP, 0xbe, 0xef, 0x0, 0x0, 0x0, > >>> 0x0, 0x0, 0x1, 0x0, 0x0, 0xa, 0x > >>> #endif > >>> > >>> #ifdef ENABLE_VTEP > >>> -#define VTEP_ENDPOINT (__u32[]){0xeb48a90a, 0xec48a90a, 0xed48a90a, > >>> 0xee48a90a, } > >>> +#define VTEP_NUMS 4 > >>> +__u32 VTEP_ENDPOINT[VTEP_NUMS] = {0xeb48a90a, 0xec48a90a, 0xed48a90a, > >>> 0xee48a90a}; > >>> /* HEX representation of VTEP IP > >>> * 10.169.72.235, 10.169.72.236, 10.169.72.237, 10.169.72.238 > >>> */ > >>> -#define VTEP_MAC (__u64[]){0x562e984c3682, 0x552e984c3682, > >>> 0x542e984c3682, 0x532e984c3682} > >>> +__u64 VTEP_MAC[VTEP_NUMS] = {0x562e984c3682, 0x552e984c3682, > >>> 0x542e984c3682, 0x532e984c3682}; > >>> /* VTEP MAC address > >>> * 82:36:4c:89:2e:56, 82:36:4c:89:2e:55, 82:36:4c:89:2e:54, > >>> 82:36:4c:89:2e:53 > >>> */ > >>> -#define VTEP_NUMS 4 > >>> #endif > >>> > >> > > > > I may misunderstand you, I thought your suggestion would stop compiler > > generating .rodata.cst32, but it appears > > the compiler still generated .rodata.cst32 after applying your changes > > > > readelf -e bpf/bpf_lxc.o | grep 'rodata' > > > > [51] .rodata.cst32 PROGBITS 0000000000000000 00045f48 > > In my case, I didn't have rodata section. > compiled with `KERNEL=54 make -C bpf` > clang is latest upstream: > clang --version > clang version 15.0.0 (https://github.com/llvm/llvm-project.git > 4f97aa7e1d70f1b259e5fddd85de05235b01b192) > But I think any recent compiler should have the same result. > readelf -e bpf/bpf_lxc.o | grep 'rodata' > <no output> > I see, thanks for checking again, my clang is 12.0.1, I guess it is not recent enough :). > > > >> Thank you Yonghong for the suggestion, the original code is kind of > >> hack from me to work around some issue :). now we decided to abandon > >> above code and use BPF hash map for VTEP lookup in Cilium to avoid > >> this issue and to be more flexible, so it is up to cilium/ebpf to > >> decide to address the rodata or not. > >> > >>> /* It appears that we can support around the below number of prefixes > >>> in an > >>> > >>>> > >>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thanks again, > >>>>>>> > >>>>>>> Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: can't get BTF: type .rodata.cst32: not found 2022-02-09 19:53 ` Vincent Li @ 2022-05-06 14:11 ` Timo Beckers 0 siblings, 0 replies; 21+ messages in thread From: Timo Beckers @ 2022-05-06 14:11 UTC (permalink / raw) To: Vincent Li, Yonghong Song; +Cc: bpf On 2/9/22 20:53, Vincent Li wrote: > On Wed, Feb 9, 2022 at 11:34 AM Yonghong Song <yhs@fb.com> wrote: >> >> >> >> On 2/8/22 10:48 AM, Vincent Li wrote: >>> On Tue, Feb 8, 2022 at 10:28 AM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>> >>>> On Mon, Feb 7, 2022 at 10:47 PM Yonghong Song <yhs@fb.com> wrote: >>>>> >>>>> >>>>> >>>>> On 2/4/22 1:22 PM, Yonghong Song wrote: >>>>>> >>>>>> >>>>>> On 2/4/22 11:39 AM, Vincent Li wrote: >>>>>>> On Fri, Feb 4, 2022 at 10:04 AM Yonghong Song <yhs@fb.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 2/4/22 3:11 AM, Timo Beckers wrote: >>>>>>>>> On 2/3/22 03:11, Yonghong Song wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2/2/22 5:47 AM, Timo Beckers wrote: >>>>>>>>>>> On 2/2/22 08:17, Yonghong Song wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 2/1/22 10:07 AM, Vincent Li wrote: >>>>>>>>>>>>> On Fri, Jan 28, 2022 at 10:27 AM Vincent Li >>>>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Jan 27, 2022 at 5:50 PM Yonghong Song <yhs@fb.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 1/25/22 12:32 PM, Vincent Li wrote: >>>>>>>>>>>>>>>> On Tue, Jan 25, 2022 at 9:52 AM Vincent Li >>>>>>>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> this is macro I suspected in my implementation that could >>>>>>>>>>>>>>>>> cause issue with BTF >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> #define ENABLE_VTEP 1 >>>>>>>>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, >>>>>>>>>>>>>>>>> 0x1f48a90a, >>>>>>>>>>>>>>>>> 0x2048a90a, } >>>>>>>>>>>>>>>>> #define VTEP_MAC (__u64[]){0x562e984c3682, 0x582e984c3682, >>>>>>>>>>>>>>>>> 0x5eaaed93fdf2, 0x5faaed93fdf2, } >>>>>>>>>>>>>>>>> #define VTEP_NUMS 4 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Jan 25, 2022 at 9:38 AM Vincent Li >>>>>>>>>>>>>>>>> <vincent.mc.li@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> While developing Cilium VTEP integration feature >>>>>>>>>>>>>>>>>> https://github.com/cilium/cilium/pull/17370, I found a >>>>>>>>>>>>>>>>>> strange issue >>>>>>>>>>>>>>>>>> that seems related to BTF and probably caused by my specific >>>>>>>>>>>>>>>>>> implementation, the issue is described in >>>>>>>>>>>>>>>>>> https://github.com/cilium/cilium/issues/18616, I don't know >>>>>>>>>>>>>>>>>> much about >>>>>>>>>>>>>>>>>> BTF and not sure if my implementation is seriously flawed >>>>>>>>>>>>>>>>>> or just some >>>>>>>>>>>>>>>>>> implementation bug or maybe not compatible with BTF. >>>>>>>>>>>>>>>>>> Strangely, the >>>>>>>>>>>>>>>>>> issue appears related to number of VTEPs I use, no problem >>>>>>>>>>>>>>>>>> with 1 or 2 >>>>>>>>>>>>>>>>>> VTEP, 3, 4 VTEPs will have problem with BTF, any guidance >>>>>>>>>>>>>>>>>> from BTF >>>>>>>>>>>>>>>>>> experts are appreciated :-). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Vincent >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sorry for previous top post >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> it looks the compiler compiles the cilium bpf_lxc.c to bpf_lxc.o >>>>>>>>>>>>>>>> differently and added " [21] .rodata.cst32 PROGBITS >>>>>>>>>>>>>>>> 0000000000000000 00011e68" when following macro exceeded 2 >>>>>>>>>>>>>>>> members >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> #define VTEP_ENDPOINT (__u32[]){0xec48a90a, 0xee48a90a, >>>>>>>>>>>>>>>> 0x1f48a90a, >>>>>>>>>>>>>>>> 0x2048a90a, } >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> no ".rodata.cst32" compiled in bpf_lxc.o when above >>>>>>>>>>>>>>>> VTEP_ENDPOINT >>>>>>>>>>>>>>>> member <=2. any reason why compiler would do that? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regarding to why compiler generates .rodata.cst32, the reason is >>>>>>>>>>>>>>> you have some 32-byte constants which needs to be saved >>>>>>>>>>>>>>> somewhere. >>>>>>>>>>>>>>> For example, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> $ cat t.c >>>>>>>>>>>>>>> struct t { >>>>>>>>>>>>>>> long c[2]; >>>>>>>>>>>>>>> int d[4]; >>>>>>>>>>>>>>> }; >>>>>>>>>>>>>>> struct t g; >>>>>>>>>>>>>>> int test() >>>>>>>>>>>>>>> { >>>>>>>>>>>>>>> struct t tmp = {.c = {1, 2}, .d = {3, 4}}; >>>>>>>>>>>>>>> g = tmp; >>>>>>>>>>>>>>> return 0; >>>>>>>>>>>>>>> } >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> $ clang -target bpf -O2 -c t.c >>>>>>>>>>>>>>> $ llvm-readelf -S t.o >>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>> [ 4] .rodata.cst32 PROGBITS 0000000000000000 >>>>>>>>>>>>>>> 0000a8 000020 >>>>>>>>>>>>>>> 20 AM 0 0 8 >>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In the above code, if you change the struct size, say from 32 >>>>>>>>>>>>>>> bytes to >>>>>>>>>>>>>>> 40 bytes, the rodata.cst32 will go away. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks Yonghong! I guess it is cilium/ebpf needs to recognize >>>>>>>>>>>>>> rodata.cst32 then >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Yonghong, >>>>>>>>>>>>> >>>>>>>>>>>>> Here is a follow-up question, it looks cilium/ebpf parse vmlinux >>>>>>>>>>>>> and >>>>>>>>>>>>> stores BTF type info in btf.Spec.namedTypes, but the elf object >>>>>>>>>>>>> file >>>>>>>>>>>>> provided by user may have section like rodata.cst32 generated by >>>>>>>>>>>>> compiler that does not have accompanying BTF type info stored in >>>>>>>>>>>>> btf.Spec.NamedTypes for the rodata.cst32, how vmlinux can be >>>>>>>>>>>>> guaranteed to have every BTF type info from application/user >>>>>>>>>>>>> provided >>>>>>>>>>>>> elf object file ? I guess there is no guarantee. >>>>>>>>>>>> >>>>>>>>>>>> vmlinux holds kernel types. rodata.cst32 holds data. If the type of >>>>>>>>>>>> rodata.cst32 needs to be emitted, the type will be encoded in bpf >>>>>>>>>>>> program BTF. >>>>>>>>>>>> >>>>>>>>>>>> Did you actually find an issue with .rodata.cst32 section? Such a >>>>>>>>>>>> section is typically generated by the compiler for initial data >>>>>>>>>>>> inside the function and llvm bpf backend tries to inline the >>>>>>>>>>>> values through a bunch of load instructions. So even you see >>>>>>>>>>>> .rodata.cst32, typically you can safely ignore it. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Vincent >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi Yonghong, >>>>>>>>>>> >>>>>>>>>>> Thanks for the reproducer. Couldn't figure out what to do with >>>>>>>>>>> .rodata.cst32, >>>>>>>>>>> since there are no symbols and no BTF info for that section. >>>>>>>>>>> >>>>>>>>>>> The values found in .rodata.cst32 are indeed inlined in the >>>>>>>>>>> bytecode as you >>>>>>>>>>> mentioned, so it seems like we can ignore it. >>>>>>>>>>> >>>>>>>>>>> Why does the compiler emit these sections? cilium/ebpf assumed up >>>>>>>>>>> until now >>>>>>>>>>> that all sections starting with '.rodata' are datasecs and must be >>>>>>>>>>> loaded into >>>>>>>>>>> the kernel, which of course needs accompanying BTF. >>>>>>>>>> >>>>>>>>>> The clang frontend emits these .rodata.* sections. In early days, >>>>>>>>>> kernel >>>>>>>>>> doesn't support global data so llvm bpf backend implements an >>>>>>>>>> optimization to inline these values. But llvm bpf backend didn't >>>>>>>>>> completely remove them as the backend doesn't have a global view >>>>>>>>>> whether these .rodata.* are being used in other places or not. >>>>>>>>>> >>>>>>>>>> Now, llvm bpf backend has better infrastructure and we probably can >>>>>>>>>> implement an IR pass to detect all uses of .rodata.*, inline these >>>>>>>>>> uses, and remove the .rodata.* global variable. >>>>>>>>>> >>>>>>>>>> You can check relocation section of the program text. If the .rodata.* >>>>>>>>>> section is referenced, you should preserve it. Otherwise, you can >>>>>>>>>> ignore that .rodata.* section. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> What other .rodata.* should we expect? >>>>>>>>>> >>>>>>>>>> Glancing through llvm code, you may see .rodata.{4,8,16,32}, >>>>>>>>>> .rodata.str*. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Timo >>>>>>>>> >>>>>>>>> Thanks for the replies all, very insightful. We were already doing >>>>>>>>> things mostly >>>>>>>>> right wrt. .rodata.*, but found a few subtle bugs walking through >>>>>>>>> the code again. >>>>>>>>> >>>>>>>>> I've gotten a hold of the ELF Vincent was trying to load, and I saw >>>>>>>>> a few things >>>>>>>>> that I found unusual. In his case, the values in cst32 are not >>>>>>>>> inlined. Instead, >>>>>>>>> this ELF has a .Lconstinit symbol pointing at the start of >>>>>>>>> .rodata.cst32, and it's >>>>>>>>> an STT_OBJECT with STB_LOCAL. Our relocation handler is fairly >>>>>>>>> strict and requires >>>>>>>>> STT_OBJECTs to be global (for supporting non-static consts). >>>>>>>> >>>>>>>> There are two ways to resolve the issue. First, extend the loader >>>>>>>> support to handle STB_LOCAL as well. Or Second, change the code like >>>>>>>> struct t v = {1, 5, 29, ...}; >>>>>>>> to >>>>>>>> struct t v; >>>>>>>> __builtin_memset(&v, 0, sizeof(struct t)); >>>>>>>> v.field1 = ...; >>>>>>>> v.field2 = ...; >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> --- >>>>>>>>> ~ llvm-readelf -ar bpf_lxc.o >>>>>>>>> >>>>>>>>> Symbol table '.symtab' contains 606 entries: >>>>>>>>> Num: Value Size Type Bind Vis Ndx Name >>>>>>>>> 2: 0000000000000000 32 OBJECT LOCAL DEFAULT 21 >>>>>>>>> .Lconstinit >>>>>>>>> >>>>>>>>> Relocation section '.rel2/7' at offset 0x6bdf0 contains 173 entries: >>>>>>>>> Offset Info Type >>>>>>>>> Symbol's Value Symbol's Name >>>>>>>>> 0000000000007300 0000000200000001 R_BPF_64_64 >>>>>>>>> 0000000000000000 .Lconstinit >>>>>>>>> --- >>>>>>>>> >>>>>>>>> --- >>>>>>>>> ~ llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf_lxc.o >>>>>>>>> warning: failed to compute relocation: R_BPF_64_64, Invalid data was >>>>>>>>> encountered while parsing the file >>>>>>>>> ... <2 more of these> ... >>>>>>>>> >>>>>>>>> Disassembly of section 2/7: >>>>>>>>> >>>>>>>>> 00000000000072f8 <LBB1_476>: >>>>>>>>> 3679: 67 08 00 00 03 00 00 00 r8 <<= 3 >>>>>>>>> 3680: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 >>>>>>>>> = 0 ll >>>>>>>>> 0000000000007300: R_BPF_64_64 .Lconstinit >>>>>>>>> 3682: 0f 82 00 00 00 00 00 00 r2 += r8 >>>>>>>>> 3683: 79 22 00 00 00 00 00 00 r2 = *(u64 *)(r2 + 0) >>>>>>>>> 3684: 7b 2a 58 ff 00 00 00 00 *(u64 *)(r10 - 168) = r2 >>>>>>>>> >>>>>>>>> Disassembly of section .rodata.cst32: >>>>>>>>> >>>>>>>>> 0000000000000000 <.Lconstinit>: >>>>>>>>> 0: 82 36 4c 98 2e 56 00 00 <unknown> >>>>>>>>> 1: 82 36 4c 98 2e 55 00 00 <unknown> >>>>>>>>> --- >>>>>>>>> >>>>>>>>> >>>>>>>>> This symbol doesn't exist in the program. Worth noting is that the >>>>>>>>> code that accesses >>>>>>>>> this static data sits within a subscope, but not sure what the >>>>>>>>> effect of this would be. >>>>>>>>> >>>>>>>>> Vincent, maybe try removing the enclosing {} to see if that changes >>>>>>>>> anything? >>>>>>>>> >>>>>>>>> --- >>>>>>>>> static __always_inline int foo(struct __ctx_buff *ctx, >>>>>>>>> >>>>>>>>> ... <snip> ... >>>>>>>>> >>>>>>>>> { >>>>>>>>> int i; >>>>>>>>> >>>>>>>>> for (i = 0; i < VTEP_NUMS; i) { >>>>>>>>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { >>>>>>>>> vtep_mac = VTEP_MAC[i]; >>>>>>>>> break; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> } >>>>>>>>> --- >>>>>>>>> >>>>>>>>> Is this perhaps something that needs to be addressed in the compiler? >>>>>>>> >>>>>>>> If you can give a reproducible test (with .c or .i file), I can take a >>>>>>>> look at what is missing in llvm compiler and improve it. >>>>>>>> >>>>>>> >>>>>>> not sure if it would help, here is my step to generate the bpf_lxc.o >>>>>>> object file with the .rodata.cst32 >>>>>>> >>>>>>> git clone https://github.com/f5devcentral/cilium.git >>>>>>> cd cilium; git checkout vli-vxlan; KERNEL=54 make -C bpf >>>>>>> llvm-objdump -S -r -j 2/7 -j .rodata.cst32 bpf/bpf_lxc.o >>>>>> >>>>>> Thanks. I can reproduce the issue now. Will take a look >>>>>> and get back to you as soon as I got any concrete results. >>>>> >>>>> Okay, I found the reason. >>>>> >>>>> For the code, >>>>> >>>>> for (i = 0; i < VTEP_NUMS; i++) { >>>>> if (tunnel_endpoint == VTEP_ENDPOINT[i]) { >>>>> vtep_mac = VTEP_MAC[i]; >>>>> break; >>>>> } >>>>> } >>>>> >>>>> The compiler transformed to something like >>>>> >>>>> i = 0; if (tunnerl_endpoint == VTEP_ENDPOINT[0]) goto end; >>>>> i = 1; if (tunnerl_endpoint == VTEP_ENDPOINT[1]) goto end; >>>>> i = 2; if (tunnerl_endpoint == VTEP_ENDPOINT[2]) goto end; >>>>> i = 3; if (tunnerl_endpoint == VTEP_ENDPOINT[3]) goto end; >>>>> >>>>> end: >>>>> vtep_mac = VTEP_MAC[i]; >>>>> >>>>> The compiler cannot inline VTEP_MAC[i] since 'i' is not >>>>> a constant. Hence later we have a memory load from >>>>> a non-global .rodata section. >>>>> >>>>> As I mentioned earlier, there are two options to fix the issue. >>>>> First is for cilium to track and handle non-global .rodata >>>>> sections. And the second you can apply the below code change, >>>>> >>>>> diff --git a/bpf/node_config.h b/bpf/node_config.h >>>>> index 9783e44548..b80dd2b27b 100644 >>>>> --- a/bpf/node_config.h >>>>> +++ b/bpf/node_config.h >>>>> @@ -176,15 +176,15 @@ DEFINE_IPV6(HOST_IP, 0xbe, 0xef, 0x0, 0x0, 0x0, >>>>> 0x0, 0x0, 0x1, 0x0, 0x0, 0xa, 0x >>>>> #endif >>>>> >>>>> #ifdef ENABLE_VTEP >>>>> -#define VTEP_ENDPOINT (__u32[]){0xeb48a90a, 0xec48a90a, 0xed48a90a, >>>>> 0xee48a90a, } >>>>> +#define VTEP_NUMS 4 >>>>> +__u32 VTEP_ENDPOINT[VTEP_NUMS] = {0xeb48a90a, 0xec48a90a, 0xed48a90a, >>>>> 0xee48a90a}; >>>>> /* HEX representation of VTEP IP >>>>> * 10.169.72.235, 10.169.72.236, 10.169.72.237, 10.169.72.238 >>>>> */ >>>>> -#define VTEP_MAC (__u64[]){0x562e984c3682, 0x552e984c3682, >>>>> 0x542e984c3682, 0x532e984c3682} >>>>> +__u64 VTEP_MAC[VTEP_NUMS] = {0x562e984c3682, 0x552e984c3682, >>>>> 0x542e984c3682, 0x532e984c3682}; >>>>> /* VTEP MAC address >>>>> * 82:36:4c:89:2e:56, 82:36:4c:89:2e:55, 82:36:4c:89:2e:54, >>>>> 82:36:4c:89:2e:53 >>>>> */ >>>>> -#define VTEP_NUMS 4 >>>>> #endif >>>>> >>>> >>> >>> I may misunderstand you, I thought your suggestion would stop compiler >>> generating .rodata.cst32, but it appears >>> the compiler still generated .rodata.cst32 after applying your changes >>> >>> readelf -e bpf/bpf_lxc.o | grep 'rodata' >>> >>> [51] .rodata.cst32 PROGBITS 0000000000000000 00045f48 >> >> In my case, I didn't have rodata section. >> compiled with `KERNEL=54 make -C bpf` >> clang is latest upstream: >> clang --version >> clang version 15.0.0 (https://github.com/llvm/llvm-project.git >> 4f97aa7e1d70f1b259e5fddd85de05235b01b192) >> But I think any recent compiler should have the same result. >> readelf -e bpf/bpf_lxc.o | grep 'rodata' >> <no output> >> > > I see, thanks for checking again, my clang is 12.0.1, I guess it is > not recent enough :). > >>> >>>> Thank you Yonghong for the suggestion, the original code is kind of >>>> hack from me to work around some issue :). now we decided to abandon >>>> above code and use BPF hash map for VTEP lookup in Cilium to avoid >>>> this issue and to be more flexible, so it is up to cilium/ebpf to >>>> decide to address the rodata or not. >>>> >>>>> /* It appears that we can support around the below number of prefixes >>>>> in an >>>>> >>>>>> >>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks again, >>>>>>>>> >>>>>>>>> Timo Hi all, Sorry to dig up this thread. I looked into this again last week, this time with the help of Dylan Reimerink. We came up with a minimal repro: --- #include <linux/bpf.h> #define values (__u32[]){0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7} int test(int ctx) { __u32 out = 0; int i; for (i = 0; i < 2; i++) { if (ctx == values[i]) { out = values[i]; break; } } return out; } --- I'm compiling with 13.0.1. - Emits a .rodata.cst32 section, but only if there are exactly 8 elements in 'values'. - Same result if 'values' is an array of 4 u64's, so the fact that it's related to initializing 32-bit values somewhat confuses me. - When changing 'values' to 7 or 9 entries, there's a .rodata BTF datasec of size=0 vlen=0 instead of a .rodata.cst32. - If a .rodata.cst32 is emitted, it doesn't have a corresponding BTF datasec at all. - The 'ctx == values[i]' comparisons are inlined, but 'out = values[i]' results in a datasec read. Unfortunately, the resulting ELF does not contain a .Lconstinit symbol pointing at the start of .rodata.cst32, but it can still be found in the IR: --- .Ltmp11: r0 <<= 2 r1 = .Lconstinit.1 ll r1 += r0 r0 = *(u32 *)(r1 + 0) --- In the initial repro Vincent posted, this symbol made it into the ELF, but that's with an older LLVM. These are my takeaways for the loader side, please correct me if I'm wrong: - As you've mentioned in this thread, it's not predictable which section 'values' will end up in, so userspace always needs to look for constants/globals in .(ro)data*, not just .(ro)data. - A BTF datasec may not be present at all for ELF data sections that only contain anonymous vars. - A BTF datasec may only partially describe a data section; e.g. some bytes in the section represent vars, but some bytes can be anonymous and are omitted from BTF info. Then, what aboue accepting relo's against local symbols (like .Lconstinit)? I don't know why we currently reject them, but it seems to be a good canary for subtle compiler bugs. Maybe that's something that can be fixed in LLVM. Thank you, Timo ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2022-05-06 14:11 UTC | newest] Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-01-25 17:38 can't get BTF: type .rodata.cst32: not found Vincent Li 2022-01-25 17:52 ` Vincent Li 2022-01-25 20:32 ` Vincent Li 2022-01-28 1:50 ` Yonghong Song 2022-01-28 18:27 ` Vincent Li 2022-02-01 18:07 ` Vincent Li 2022-02-02 7:17 ` Yonghong Song 2022-02-02 13:47 ` Timo Beckers 2022-02-02 19:55 ` Andrii Nakryiko 2022-02-03 2:11 ` Yonghong Song 2022-02-04 11:11 ` Timo Beckers 2022-02-04 18:04 ` Yonghong Song 2022-02-04 19:39 ` Vincent Li 2022-02-04 21:22 ` Yonghong Song 2022-02-07 21:53 ` Andrii Nakryiko 2022-02-08 6:47 ` Yonghong Song 2022-02-08 18:28 ` Vincent Li 2022-02-08 18:48 ` Vincent Li 2022-02-09 19:34 ` Yonghong Song 2022-02-09 19:53 ` Vincent Li 2022-05-06 14:11 ` Timo Beckers
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).