* selftest/bpf/test_verifier_log fails on v5.11-rc5 @ 2021-01-25 20:51 Paul Moore 2021-01-25 22:42 ` Alexei Starovoitov 0 siblings, 1 reply; 13+ messages in thread From: Paul Moore @ 2021-01-25 20:51 UTC (permalink / raw) To: bpf Hello all, My apologies if this has already been reported, but I didn't see anything obvious with a quick search through the archives. I have a test program that behaves very similar to the existing selftest/bpf/test_verifier_log test that has started failing this week with v5.11-rc5; it ran without problem last week on v5.11-rc4. Is this a known problem with a fix already, or is this something new? % uname -r 5.11.0-0.rc5.134.fc34.x86_64 % pwd /.../linux/tools/testing/selftests/bpf % git log --oneline | head -n 1 6ee1d745b7c9 Linux 5.11-rc5 % make test_verifier_log ... BINARY test_verifier_log % ./test_verifier_log Test log_level 0... Test log_size < 128... Test log_buff = NULL... Test oversized buffer... ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13 -- paul moore www.paul-moore.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-01-25 20:51 selftest/bpf/test_verifier_log fails on v5.11-rc5 Paul Moore @ 2021-01-25 22:42 ` Alexei Starovoitov 2021-01-29 22:15 ` Paul Moore 0 siblings, 1 reply; 13+ messages in thread From: Alexei Starovoitov @ 2021-01-25 22:42 UTC (permalink / raw) To: Paul Moore; +Cc: bpf On Mon, Jan 25, 2021 at 12:54 PM Paul Moore <paul@paul-moore.com> wrote: > > Hello all, > > My apologies if this has already been reported, but I didn't see > anything obvious with a quick search through the archives. I have a > test program that behaves very similar to the existing > selftest/bpf/test_verifier_log test that has started failing this week > with v5.11-rc5; it ran without problem last week on v5.11-rc4. Is > this a known problem with a fix already, or is this something new? > > % uname -r > 5.11.0-0.rc5.134.fc34.x86_64 > % pwd > /.../linux/tools/testing/selftests/bpf > % git log --oneline | head -n 1 > 6ee1d745b7c9 Linux 5.11-rc5 > % make test_verifier_log > ... > BINARY test_verifier_log > % ./test_verifier_log > Test log_level 0... > Test log_size < 128... > Test log_buff = NULL... > Test oversized buffer... > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13 Thanks for reporting. bpf and bpf-next don't have this issue. Not sure what changed. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-01-25 22:42 ` Alexei Starovoitov @ 2021-01-29 22:15 ` Paul Moore 2021-01-30 3:13 ` Alexei Starovoitov 0 siblings, 1 reply; 13+ messages in thread From: Paul Moore @ 2021-01-29 22:15 UTC (permalink / raw) To: Alexei Starovoitov; +Cc: bpf, Ondrej Mosnacek On Mon, Jan 25, 2021 at 5:42 PM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > On Mon, Jan 25, 2021 at 12:54 PM Paul Moore <paul@paul-moore.com> wrote: > > > > Hello all, > > > > My apologies if this has already been reported, but I didn't see > > anything obvious with a quick search through the archives. I have a > > test program that behaves very similar to the existing > > selftest/bpf/test_verifier_log test that has started failing this week > > with v5.11-rc5; it ran without problem last week on v5.11-rc4. Is > > this a known problem with a fix already, or is this something new? > > > > % uname -r > > 5.11.0-0.rc5.134.fc34.x86_64 > > % pwd > > /.../linux/tools/testing/selftests/bpf > > % git log --oneline | head -n 1 > > 6ee1d745b7c9 Linux 5.11-rc5 > > % make test_verifier_log > > ... > > BINARY test_verifier_log > > % ./test_verifier_log > > Test log_level 0... > > Test log_size < 128... > > Test log_buff = NULL... > > Test oversized buffer... > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13 > > Thanks for reporting. > bpf and bpf-next don't have this issue. Not sure what changed. I haven't had a chance to look into this any further, but Ondrej Mosnacek (CC'd) found the following today: "So I was trying to debug this further and I think I've identified what triggers the problem. It seems that the BTF debuginfo generation became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4 and -rc5. It also seems to depend on a recent (Fedora Rawhide) version of some component of the build system (GCC, probably), because the problem disappeared when I tried to build the "bad" kernel in F33 buildroot instead of Rawhide." -- paul moore www.paul-moore.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-01-29 22:15 ` Paul Moore @ 2021-01-30 3:13 ` Alexei Starovoitov 2021-01-30 20:48 ` Jiri Olsa 0 siblings, 1 reply; 13+ messages in thread From: Alexei Starovoitov @ 2021-01-30 3:13 UTC (permalink / raw) To: Paul Moore, Jiri Olsa, Arnaldo Carvalho de Melo, Andrii Nakryiko Cc: bpf, Ondrej Mosnacek On Fri, Jan 29, 2021 at 2:15 PM Paul Moore <paul@paul-moore.com> wrote: > > On Mon, Jan 25, 2021 at 5:42 PM Alexei Starovoitov > <alexei.starovoitov@gmail.com> wrote: > > On Mon, Jan 25, 2021 at 12:54 PM Paul Moore <paul@paul-moore.com> wrote: > > > > > > Hello all, > > > > > > My apologies if this has already been reported, but I didn't see > > > anything obvious with a quick search through the archives. I have a > > > test program that behaves very similar to the existing > > > selftest/bpf/test_verifier_log test that has started failing this week > > > with v5.11-rc5; it ran without problem last week on v5.11-rc4. Is > > > this a known problem with a fix already, or is this something new? > > > > > > % uname -r > > > 5.11.0-0.rc5.134.fc34.x86_64 > > > % pwd > > > /.../linux/tools/testing/selftests/bpf > > > % git log --oneline | head -n 1 > > > 6ee1d745b7c9 Linux 5.11-rc5 > > > % make test_verifier_log > > > ... > > > BINARY test_verifier_log > > > % ./test_verifier_log > > > Test log_level 0... > > > Test log_size < 128... > > > Test log_buff = NULL... > > > Test oversized buffer... > > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13 > > > > Thanks for reporting. > > bpf and bpf-next don't have this issue. Not sure what changed. > > I haven't had a chance to look into this any further, but Ondrej > Mosnacek (CC'd) found the following today: > > "So I was trying to debug this further and I think I've identified what > triggers the problem. It seems that the BTF debuginfo generation > became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4 > and -rc5. It also seems to depend on a recent (Fedora Rawhide) version > of some component of the build system (GCC, probably), because the > problem disappeared when I tried to build the "bad" kernel in F33 > buildroot instead of Rawhide." I see. There were fixes for dwarf and btf, but I lost the track. I believe it was a combination of gcc bug that was worked around in pahole. Arnaldo, Jiri, Andrii, what is the status? Did all fixes land in pahole? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-01-30 3:13 ` Alexei Starovoitov @ 2021-01-30 20:48 ` Jiri Olsa 2021-01-31 21:36 ` Jiri Olsa 0 siblings, 1 reply; 13+ messages in thread From: Jiri Olsa @ 2021-01-30 20:48 UTC (permalink / raw) To: Alexei Starovoitov Cc: Paul Moore, Jiri Olsa, Arnaldo Carvalho de Melo, Andrii Nakryiko, bpf, Ondrej Mosnacek On Fri, Jan 29, 2021 at 07:13:21PM -0800, Alexei Starovoitov wrote: > On Fri, Jan 29, 2021 at 2:15 PM Paul Moore <paul@paul-moore.com> wrote: > > > > On Mon, Jan 25, 2021 at 5:42 PM Alexei Starovoitov > > <alexei.starovoitov@gmail.com> wrote: > > > On Mon, Jan 25, 2021 at 12:54 PM Paul Moore <paul@paul-moore.com> wrote: > > > > > > > > Hello all, > > > > > > > > My apologies if this has already been reported, but I didn't see > > > > anything obvious with a quick search through the archives. I have a > > > > test program that behaves very similar to the existing > > > > selftest/bpf/test_verifier_log test that has started failing this week > > > > with v5.11-rc5; it ran without problem last week on v5.11-rc4. Is > > > > this a known problem with a fix already, or is this something new? > > > > > > > > % uname -r > > > > 5.11.0-0.rc5.134.fc34.x86_64 > > > > % pwd > > > > /.../linux/tools/testing/selftests/bpf > > > > % git log --oneline | head -n 1 > > > > 6ee1d745b7c9 Linux 5.11-rc5 > > > > % make test_verifier_log > > > > ... > > > > BINARY test_verifier_log > > > > % ./test_verifier_log > > > > Test log_level 0... > > > > Test log_size < 128... > > > > Test log_buff = NULL... > > > > Test oversized buffer... > > > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13 > > > > > > Thanks for reporting. > > > bpf and bpf-next don't have this issue. Not sure what changed. > > > > I haven't had a chance to look into this any further, but Ondrej > > Mosnacek (CC'd) found the following today: > > > > "So I was trying to debug this further and I think I've identified what > > triggers the problem. It seems that the BTF debuginfo generation > > became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4 > > and -rc5. It also seems to depend on a recent (Fedora Rawhide) version > > of some component of the build system (GCC, probably), because the > > problem disappeared when I tried to build the "bad" kernel in F33 > > buildroot instead of Rawhide." > > I see. There were fixes for dwarf and btf, but I lost the track. > I believe it was a combination of gcc bug that was worked around in pahole. > Arnaldo, Jiri, Andrii, > what is the status? Did all fixes land in pahole? I checked on rawhide and besides many pahole warnings, the resulted BTF data have many duplications in core structs BTFIDS vmlinux WARN: multiple IDs found for 'task_struct': 132, 1247 - using 132 WARN: multiple IDs found for 'file': 440, 1349 - using 440 WARN: multiple IDs found for 'inode': 698, 1645 - using 698 WARN: multiple IDs found for 'path': 729, 1672 - using 729 WARN: multiple IDs found for 'task_struct': 132, 2984 - using 132 WARN: multiple IDs found for 'task_struct': 132, 3043 - using 132 WARN: multiple IDs found for 'file': 440, 3085 - using 440 WARN: multiple IDs found for 'seq_file': 1469, 3125 - using 1469 WARN: multiple IDs found for 'inode': 698, 3336 - using 698 WARN: multiple IDs found for 'path': 729, 3366 - using 729 WARN: multiple IDs found for 'task_struct': 132, 5337 - using 132 WARN: multiple IDs found for 'inode': 698, 5360 - using 698 WARN: multiple IDs found for 'path': 729, 5388 - using 729 WARN: multiple IDs found for 'file': 440, 5412 - using 440 WARN: multiple IDs found for 'seq_file': 1469, 5639 - using 1469 WARN: multiple IDs found for 'task_struct': 132, 6243 - using 132 ... # gcc --version gcc (GCC) 11.0.0 20210123 (Red Hat 11.0.0-0) I'm guessing there are some DWARF changes that screwed BTF generation.. I'll check it's not covered by the fix I posted recently, but I think Arnaldo is now fixing some related stuff.. Arnaldo, maybe you are seeing same errors? I uploaded the build log from linking part to: http://people.redhat.com/~jolsa/build.out.gz jirka ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-01-30 20:48 ` Jiri Olsa @ 2021-01-31 21:36 ` Jiri Olsa 2021-02-01 12:25 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 13+ messages in thread From: Jiri Olsa @ 2021-01-31 21:36 UTC (permalink / raw) To: Alexei Starovoitov, Arnaldo Carvalho de Melo Cc: Paul Moore, Jiri Olsa, Andrii Nakryiko, bpf, Ondrej Mosnacek On Sat, Jan 30, 2021 at 09:48:13PM +0100, Jiri Olsa wrote: SNIP > > > > > % uname -r > > > > > 5.11.0-0.rc5.134.fc34.x86_64 > > > > > % pwd > > > > > /.../linux/tools/testing/selftests/bpf > > > > > % git log --oneline | head -n 1 > > > > > 6ee1d745b7c9 Linux 5.11-rc5 > > > > > % make test_verifier_log > > > > > ... > > > > > BINARY test_verifier_log > > > > > % ./test_verifier_log > > > > > Test log_level 0... > > > > > Test log_size < 128... > > > > > Test log_buff = NULL... > > > > > Test oversized buffer... > > > > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13 > > > > > > > > Thanks for reporting. > > > > bpf and bpf-next don't have this issue. Not sure what changed. > > > > > > I haven't had a chance to look into this any further, but Ondrej > > > Mosnacek (CC'd) found the following today: > > > > > > "So I was trying to debug this further and I think I've identified what > > > triggers the problem. It seems that the BTF debuginfo generation > > > became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4 > > > and -rc5. It also seems to depend on a recent (Fedora Rawhide) version > > > of some component of the build system (GCC, probably), because the > > > problem disappeared when I tried to build the "bad" kernel in F33 > > > buildroot instead of Rawhide." > > > > I see. There were fixes for dwarf and btf, but I lost the track. > > I believe it was a combination of gcc bug that was worked around in pahole. > > Arnaldo, Jiri, Andrii, > > what is the status? Did all fixes land in pahole? > > I checked on rawhide and besides many pahole warnings, > the resulted BTF data have many duplications in core structs > > BTFIDS vmlinux > WARN: multiple IDs found for 'task_struct': 132, 1247 - using 132 > WARN: multiple IDs found for 'file': 440, 1349 - using 440 > WARN: multiple IDs found for 'inode': 698, 1645 - using 698 > WARN: multiple IDs found for 'path': 729, 1672 - using 729 > WARN: multiple IDs found for 'task_struct': 132, 2984 - using 132 > WARN: multiple IDs found for 'task_struct': 132, 3043 - using 132 > WARN: multiple IDs found for 'file': 440, 3085 - using 440 > WARN: multiple IDs found for 'seq_file': 1469, 3125 - using 1469 > WARN: multiple IDs found for 'inode': 698, 3336 - using 698 > WARN: multiple IDs found for 'path': 729, 3366 - using 729 > WARN: multiple IDs found for 'task_struct': 132, 5337 - using 132 > WARN: multiple IDs found for 'inode': 698, 5360 - using 698 > WARN: multiple IDs found for 'path': 729, 5388 - using 729 > WARN: multiple IDs found for 'file': 440, 5412 - using 440 > WARN: multiple IDs found for 'seq_file': 1469, 5639 - using 1469 > WARN: multiple IDs found for 'task_struct': 132, 6243 - using 132 > ... > > # gcc --version > gcc (GCC) 11.0.0 20210123 (Red Hat 11.0.0-0) > > I'm guessing there are some DWARF changes that screwed BTF > generation.. I'll check > > it's not covered by the fix I posted recently, but I think > Arnaldo is now fixing some related stuff.. Arnaldo, maybe > you are seeing same errors? with Arnaldo's fixes I see less struct duplications, but still there's some > > I uploaded the build log from linking part to: > http://people.redhat.com/~jolsa/build.out.gz however looks like we don't handle DW_FORM_implicit_const when counting the byte offset.. it was used for some struct members in my vmlinux, so we got zero for byte offset and that created another unique struct with patch below I no longer see any struct duplication, also test_verifier_log is working for me, but I could not reproduce the error before I'll post full dwarves patch after some more testing also I wonder we could somehow use btf_check_all_metas from kernel after we build BTF data, that'd help to catch this earlier/easier ;-) I'll check on this jirka --- diff --git a/dwarf_loader.c b/dwarf_loader.c index ac22c1b..e2981a4 100644 --- a/dwarf_loader.c +++ b/dwarf_loader.c @@ -296,6 +296,7 @@ static Dwarf_Off __attr_offset(Dwarf_Attribute *attr) Dwarf_Block block; switch (dwarf_whatform(attr)) { + case DW_FORM_implicit_const: case DW_FORM_data1: case DW_FORM_data2: case DW_FORM_data4: ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-01-31 21:36 ` Jiri Olsa @ 2021-02-01 12:25 ` Arnaldo Carvalho de Melo 2021-02-01 14:50 ` Jiri Olsa 0 siblings, 1 reply; 13+ messages in thread From: Arnaldo Carvalho de Melo @ 2021-02-01 12:25 UTC (permalink / raw) To: Jiri Olsa Cc: Alexei Starovoitov, Paul Moore, Jiri Olsa, Andrii Nakryiko, bpf, Ondrej Mosnacek Em Sun, Jan 31, 2021 at 10:36:41PM +0100, Jiri Olsa escreveu: > On Sat, Jan 30, 2021 at 09:48:13PM +0100, Jiri Olsa wrote: > > SNIP > > > > > > > % uname -r > > > > > > 5.11.0-0.rc5.134.fc34.x86_64 > > > > > > % pwd > > > > > > /.../linux/tools/testing/selftests/bpf > > > > > > % git log --oneline | head -n 1 > > > > > > 6ee1d745b7c9 Linux 5.11-rc5 > > > > > > % make test_verifier_log > > > > > > ... > > > > > > BINARY test_verifier_log > > > > > > % ./test_verifier_log > > > > > > Test log_level 0... > > > > > > Test log_size < 128... > > > > > > Test log_buff = NULL... > > > > > > Test oversized buffer... > > > > > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13 > > > > > > > > > > Thanks for reporting. > > > > > bpf and bpf-next don't have this issue. Not sure what changed. > > > > > > > > I haven't had a chance to look into this any further, but Ondrej > > > > Mosnacek (CC'd) found the following today: > > > > > > > > "So I was trying to debug this further and I think I've identified what > > > > triggers the problem. It seems that the BTF debuginfo generation > > > > became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4 > > > > and -rc5. It also seems to depend on a recent (Fedora Rawhide) version > > > > of some component of the build system (GCC, probably), because the > > > > problem disappeared when I tried to build the "bad" kernel in F33 > > > > buildroot instead of Rawhide." > > > > > > I see. There were fixes for dwarf and btf, but I lost the track. > > > I believe it was a combination of gcc bug that was worked around in pahole. > > > Arnaldo, Jiri, Andrii, > > > what is the status? Did all fixes land in pahole? > > > > I checked on rawhide and besides many pahole warnings, > > the resulted BTF data have many duplications in core structs > > > > BTFIDS vmlinux > > WARN: multiple IDs found for 'task_struct': 132, 1247 - using 132 > > WARN: multiple IDs found for 'file': 440, 1349 - using 440 > > WARN: multiple IDs found for 'inode': 698, 1645 - using 698 > > WARN: multiple IDs found for 'path': 729, 1672 - using 729 > > WARN: multiple IDs found for 'task_struct': 132, 2984 - using 132 > > WARN: multiple IDs found for 'task_struct': 132, 3043 - using 132 > > WARN: multiple IDs found for 'file': 440, 3085 - using 440 > > WARN: multiple IDs found for 'seq_file': 1469, 3125 - using 1469 > > WARN: multiple IDs found for 'inode': 698, 3336 - using 698 > > WARN: multiple IDs found for 'path': 729, 3366 - using 729 > > WARN: multiple IDs found for 'task_struct': 132, 5337 - using 132 > > WARN: multiple IDs found for 'inode': 698, 5360 - using 698 > > WARN: multiple IDs found for 'path': 729, 5388 - using 729 > > WARN: multiple IDs found for 'file': 440, 5412 - using 440 > > WARN: multiple IDs found for 'seq_file': 1469, 5639 - using 1469 > > WARN: multiple IDs found for 'task_struct': 132, 6243 - using 132 > > ... > > > > # gcc --version > > gcc (GCC) 11.0.0 20210123 (Red Hat 11.0.0-0) > > > > I'm guessing there are some DWARF changes that screwed BTF > > generation.. I'll check > > > > it's not covered by the fix I posted recently, but I think > > Arnaldo is now fixing some related stuff.. Arnaldo, maybe > > you are seeing same errors? > > with Arnaldo's fixes I see less struct duplications, > but still there's some > > > > > I uploaded the build log from linking part to: > > http://people.redhat.com/~jolsa/build.out.gz > > however looks like we don't handle DW_FORM_implicit_const > when counting the byte offset.. it was used for some struct > members in my vmlinux, so we got zero for byte offset and > that created another unique struct > > with patch below I no longer see any struct duplication, > also test_verifier_log is working for me, but I could > not reproduce the error before > > I'll post full dwarves patch after some more testing > > also I wonder we could somehow use btf_check_all_metas > from kernel after we build BTF data, that'd help to catch > this earlier/easier ;-) I'll check on this Seems like a good idea indeed :-) I'm applying the patch below with your Signed-off-by, etc, ok? - Arnaldo > jirka > > > --- > diff --git a/dwarf_loader.c b/dwarf_loader.c > index ac22c1b..e2981a4 100644 > --- a/dwarf_loader.c > +++ b/dwarf_loader.c > @@ -296,6 +296,7 @@ static Dwarf_Off __attr_offset(Dwarf_Attribute *attr) > Dwarf_Block block; > > switch (dwarf_whatform(attr)) { > + case DW_FORM_implicit_const: > case DW_FORM_data1: > case DW_FORM_data2: > case DW_FORM_data4: > -- - Arnaldo ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-02-01 12:25 ` Arnaldo Carvalho de Melo @ 2021-02-01 14:50 ` Jiri Olsa 2021-02-01 20:23 ` Jiri Olsa 0 siblings, 1 reply; 13+ messages in thread From: Jiri Olsa @ 2021-02-01 14:50 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Alexei Starovoitov, Paul Moore, Jiri Olsa, Andrii Nakryiko, bpf, Ondrej Mosnacek On Mon, Feb 01, 2021 at 09:25:32AM -0300, Arnaldo Carvalho de Melo wrote: > Em Sun, Jan 31, 2021 at 10:36:41PM +0100, Jiri Olsa escreveu: > > On Sat, Jan 30, 2021 at 09:48:13PM +0100, Jiri Olsa wrote: > > > > SNIP > > > > > > > > > % uname -r > > > > > > > 5.11.0-0.rc5.134.fc34.x86_64 > > > > > > > % pwd > > > > > > > /.../linux/tools/testing/selftests/bpf > > > > > > > % git log --oneline | head -n 1 > > > > > > > 6ee1d745b7c9 Linux 5.11-rc5 > > > > > > > % make test_verifier_log > > > > > > > ... > > > > > > > BINARY test_verifier_log > > > > > > > % ./test_verifier_log > > > > > > > Test log_level 0... > > > > > > > Test log_size < 128... > > > > > > > Test log_buff = NULL... > > > > > > > Test oversized buffer... > > > > > > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13 > > > > > > > > > > > > Thanks for reporting. > > > > > > bpf and bpf-next don't have this issue. Not sure what changed. > > > > > > > > > > I haven't had a chance to look into this any further, but Ondrej > > > > > Mosnacek (CC'd) found the following today: > > > > > > > > > > "So I was trying to debug this further and I think I've identified what > > > > > triggers the problem. It seems that the BTF debuginfo generation > > > > > became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4 > > > > > and -rc5. It also seems to depend on a recent (Fedora Rawhide) version > > > > > of some component of the build system (GCC, probably), because the > > > > > problem disappeared when I tried to build the "bad" kernel in F33 > > > > > buildroot instead of Rawhide." > > > > > > > > I see. There were fixes for dwarf and btf, but I lost the track. > > > > I believe it was a combination of gcc bug that was worked around in pahole. > > > > Arnaldo, Jiri, Andrii, > > > > what is the status? Did all fixes land in pahole? > > > > > > I checked on rawhide and besides many pahole warnings, > > > the resulted BTF data have many duplications in core structs > > > > > > BTFIDS vmlinux > > > WARN: multiple IDs found for 'task_struct': 132, 1247 - using 132 > > > WARN: multiple IDs found for 'file': 440, 1349 - using 440 > > > WARN: multiple IDs found for 'inode': 698, 1645 - using 698 > > > WARN: multiple IDs found for 'path': 729, 1672 - using 729 > > > WARN: multiple IDs found for 'task_struct': 132, 2984 - using 132 > > > WARN: multiple IDs found for 'task_struct': 132, 3043 - using 132 > > > WARN: multiple IDs found for 'file': 440, 3085 - using 440 > > > WARN: multiple IDs found for 'seq_file': 1469, 3125 - using 1469 > > > WARN: multiple IDs found for 'inode': 698, 3336 - using 698 > > > WARN: multiple IDs found for 'path': 729, 3366 - using 729 > > > WARN: multiple IDs found for 'task_struct': 132, 5337 - using 132 > > > WARN: multiple IDs found for 'inode': 698, 5360 - using 698 > > > WARN: multiple IDs found for 'path': 729, 5388 - using 729 > > > WARN: multiple IDs found for 'file': 440, 5412 - using 440 > > > WARN: multiple IDs found for 'seq_file': 1469, 5639 - using 1469 > > > WARN: multiple IDs found for 'task_struct': 132, 6243 - using 132 > > > ... > > > > > > # gcc --version > > > gcc (GCC) 11.0.0 20210123 (Red Hat 11.0.0-0) > > > > > > I'm guessing there are some DWARF changes that screwed BTF > > > generation.. I'll check > > > > > > it's not covered by the fix I posted recently, but I think > > > Arnaldo is now fixing some related stuff.. Arnaldo, maybe > > > you are seeing same errors? > > > > with Arnaldo's fixes I see less struct duplications, > > but still there's some > > > > > > > > I uploaded the build log from linking part to: > > > http://people.redhat.com/~jolsa/build.out.gz > > > > however looks like we don't handle DW_FORM_implicit_const > > when counting the byte offset.. it was used for some struct > > members in my vmlinux, so we got zero for byte offset and > > that created another unique struct > > > > with patch below I no longer see any struct duplication, > > also test_verifier_log is working for me, but I could > > not reproduce the error before > > > > I'll post full dwarves patch after some more testing > > > > also I wonder we could somehow use btf_check_all_metas > > from kernel after we build BTF data, that'd help to catch > > this earlier/easier ;-) I'll check on this > > Seems like a good idea indeed :-) > > I'm applying the patch below with your Signed-off-by, etc, ok? ok, thanks jirka ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-02-01 14:50 ` Jiri Olsa @ 2021-02-01 20:23 ` Jiri Olsa 2021-02-01 22:43 ` Ondrej Mosnacek 0 siblings, 1 reply; 13+ messages in thread From: Jiri Olsa @ 2021-02-01 20:23 UTC (permalink / raw) To: Paul Moore, Ondrej Mosnacek Cc: Arnaldo Carvalho de Melo, Alexei Starovoitov, Jiri Olsa, Andrii Nakryiko, bpf On Mon, Feb 01, 2021 at 03:50:22PM +0100, Jiri Olsa wrote: SNIP > > > > > > with Arnaldo's fixes I see less struct duplications, > > > but still there's some > > > > > > > > > > > I uploaded the build log from linking part to: > > > > http://people.redhat.com/~jolsa/build.out.gz > > > > > > however looks like we don't handle DW_FORM_implicit_const > > > when counting the byte offset.. it was used for some struct > > > members in my vmlinux, so we got zero for byte offset and > > > that created another unique struct > > > > > > with patch below I no longer see any struct duplication, > > > also test_verifier_log is working for me, but I could > > > not reproduce the error before > > > > > > I'll post full dwarves patch after some more testing > > > > > > also I wonder we could somehow use btf_check_all_metas > > > from kernel after we build BTF data, that'd help to catch > > > this earlier/easier ;-) I'll check on this > > > > Seems like a good idea indeed :-) > > > > I'm applying the patch below with your Signed-off-by, etc, ok? > > ok, thanks Paul, Ondrej, I put all the recent fixes and made a scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=61049457 if you have a chance to test and check your issue was resolved, that'd be great thanks, jirka ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-02-01 20:23 ` Jiri Olsa @ 2021-02-01 22:43 ` Ondrej Mosnacek 2021-02-01 23:28 ` Paul Moore 2021-02-02 12:43 ` Arnaldo Carvalho de Melo 0 siblings, 2 replies; 13+ messages in thread From: Ondrej Mosnacek @ 2021-02-01 22:43 UTC (permalink / raw) To: Jiri Olsa Cc: Paul Moore, Arnaldo Carvalho de Melo, Alexei Starovoitov, Jiri Olsa, Andrii Nakryiko, bpf On Mon, Feb 1, 2021 at 9:23 PM Jiri Olsa <jolsa@redhat.com> wrote: > On Mon, Feb 01, 2021 at 03:50:22PM +0100, Jiri Olsa wrote: > > SNIP > > > > > > > > > with Arnaldo's fixes I see less struct duplications, > > > > but still there's some > > > > > > > > > > > > > > I uploaded the build log from linking part to: > > > > > http://people.redhat.com/~jolsa/build.out.gz > > > > > > > > however looks like we don't handle DW_FORM_implicit_const > > > > when counting the byte offset.. it was used for some struct > > > > members in my vmlinux, so we got zero for byte offset and > > > > that created another unique struct > > > > > > > > with patch below I no longer see any struct duplication, > > > > also test_verifier_log is working for me, but I could > > > > not reproduce the error before > > > > > > > > I'll post full dwarves patch after some more testing > > > > > > > > also I wonder we could somehow use btf_check_all_metas > > > > from kernel after we build BTF data, that'd help to catch > > > > this earlier/easier ;-) I'll check on this > > > > > > Seems like a good idea indeed :-) > > > > > > I'm applying the patch below with your Signed-off-by, etc, ok? > > > > ok, thanks > > Paul, Ondrej, > > I put all the recent fixes and made a scratch build: > https://koji.fedoraproject.org/koji/taskinfo?taskID=61049457 > > if you have a chance to test and check your issue was resolved, > that'd be great I just built the current master branch of dwarves (d783117162c0, which includes Jirka's patch) [1] in COPR [2] and then rebuilt the kernel with it [3]. With the new dwarves, the issue seems to be fixed - /sys/kernel/btf/vmlinux is back to ~4MB and the selinux-testsuite BPF subtest passes. Thanks everyone for getting to the bottom of this! Hoping to see an updated dwarves in rawhide soon ;) [1] https://github.com/acmel/dwarves/ [2] https://copr.fedorainfracloud.org/coprs/omos/kernel-btf-test/build/1930103/ [3] https://copr.fedorainfracloud.org/coprs/omos/kernel-btf-test/build/1930104/ -- Ondrej Mosnacek Software Engineer, Platform Security - SELinux kernel Red Hat, Inc. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-02-01 22:43 ` Ondrej Mosnacek @ 2021-02-01 23:28 ` Paul Moore 2021-02-02 12:43 ` Arnaldo Carvalho de Melo 1 sibling, 0 replies; 13+ messages in thread From: Paul Moore @ 2021-02-01 23:28 UTC (permalink / raw) To: Ondrej Mosnacek Cc: Jiri Olsa, Arnaldo Carvalho de Melo, Alexei Starovoitov, Jiri Olsa, Andrii Nakryiko, bpf On Mon, Feb 1, 2021 at 5:43 PM Ondrej Mosnacek <omosnace@redhat.com> wrote: > On Mon, Feb 1, 2021 at 9:23 PM Jiri Olsa <jolsa@redhat.com> wrote: ... > > Paul, Ondrej, > > > > I put all the recent fixes and made a scratch build: > > https://koji.fedoraproject.org/koji/taskinfo?taskID=61049457 > > > > if you have a chance to test and check your issue was resolved, > > that'd be great > > I just built the current master branch of dwarves (d783117162c0, which > includes Jirka's patch) [1] in COPR [2] and then rebuilt the kernel > with it [3]. With the new dwarves, the issue seems to be fixed - > /sys/kernel/btf/vmlinux is back to ~4MB and the selinux-testsuite BPF > subtest passes. > > Thanks everyone for getting to the bottom of this! Hoping to see an > updated dwarves in rawhide soon ;) Yes, thanks! I've updated my test systems and I'm building a x86_64 and aarch64 kernel now to test; based on Ondrej's reports I'm sure it will work just fine, but I'll report back if I run into any issues. -- paul moore www.paul-moore.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-02-01 22:43 ` Ondrej Mosnacek 2021-02-01 23:28 ` Paul Moore @ 2021-02-02 12:43 ` Arnaldo Carvalho de Melo 2021-02-02 15:24 ` Paul Moore 1 sibling, 1 reply; 13+ messages in thread From: Arnaldo Carvalho de Melo @ 2021-02-02 12:43 UTC (permalink / raw) To: Ondrej Mosnacek Cc: Jiri Olsa, Paul Moore, Alexei Starovoitov, Jiri Olsa, Andrii Nakryiko, bpf Em Mon, Feb 01, 2021 at 11:43:26PM +0100, Ondrej Mosnacek escreveu: > On Mon, Feb 1, 2021 at 9:23 PM Jiri Olsa <jolsa@redhat.com> wrote: > > On Mon, Feb 01, 2021 at 03:50:22PM +0100, Jiri Olsa wrote: > > Paul, Ondrej, > > I put all the recent fixes and made a scratch build: > > https://koji.fedoraproject.org/koji/taskinfo?taskID=61049457 > > if you have a chance to test and check your issue was resolved, > > that'd be great > I just built the current master branch of dwarves (d783117162c0, which > includes Jirka's patch) [1] in COPR [2] and then rebuilt the kernel > with it [3]. With the new dwarves, the issue seems to be fixed - > /sys/kernel/btf/vmlinux is back to ~4MB and the selinux-testsuite BPF > subtest passes. > Thanks everyone for getting to the bottom of this! Hoping to see an > updated dwarves in rawhide soon ;) Thanks a lot! I've updated a f33 system to rawhide to test all this, fixed up some extra warnings wrt mallinfo(), strndup() error path handling/potential buffer overflow issue and will add a conditional define for DW_FORM_implicit_const found in the libbpf CI tests that Andrii pointed out to me, then go and tag 1.20 and do the rawhide/fedora package update dance. - Arnaldo > [1] https://github.com/acmel/dwarves/ > [2] https://copr.fedorainfracloud.org/coprs/omos/kernel-btf-test/build/1930103/ > [3] https://copr.fedorainfracloud.org/coprs/omos/kernel-btf-test/build/1930104/ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: selftest/bpf/test_verifier_log fails on v5.11-rc5 2021-02-02 12:43 ` Arnaldo Carvalho de Melo @ 2021-02-02 15:24 ` Paul Moore 0 siblings, 0 replies; 13+ messages in thread From: Paul Moore @ 2021-02-02 15:24 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Ondrej Mosnacek, Jiri Olsa, Alexei Starovoitov, Jiri Olsa, Andrii Nakryiko, bpf On Tue, Feb 2, 2021 at 7:43 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote: > I've updated a f33 system to rawhide to test all this, fixed up some > extra warnings wrt mallinfo(), strndup() error path handling/potential > buffer overflow issue and will add a conditional define for > DW_FORM_implicit_const found in the libbpf CI tests that Andrii pointed > out to me, then go and tag 1.20 and do the rawhide/fedora package update > dance. Thanks for taking care of this. FWIW, it looks like both my x86_64 and aarch64 test runs with your Rawhide scratch build went through without a problem last night. -- paul moore www.paul-moore.com ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2021-02-02 15:26 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-25 20:51 selftest/bpf/test_verifier_log fails on v5.11-rc5 Paul Moore 2021-01-25 22:42 ` Alexei Starovoitov 2021-01-29 22:15 ` Paul Moore 2021-01-30 3:13 ` Alexei Starovoitov 2021-01-30 20:48 ` Jiri Olsa 2021-01-31 21:36 ` Jiri Olsa 2021-02-01 12:25 ` Arnaldo Carvalho de Melo 2021-02-01 14:50 ` Jiri Olsa 2021-02-01 20:23 ` Jiri Olsa 2021-02-01 22:43 ` Ondrej Mosnacek 2021-02-01 23:28 ` Paul Moore 2021-02-02 12:43 ` Arnaldo Carvalho de Melo 2021-02-02 15:24 ` Paul Moore
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.