selftest/bpf/test_verifier_log fails on v5.11-rc5

selftest/bpf/test_verifier_log fails on v5.11-rc5

[Paul Moore]

Hello all,

My apologies if this has already been reported, but I didn't see
anything obvious with a quick search through the archives.  I have a
test program that behaves very similar to the existing
selftest/bpf/test_verifier_log test that has started failing this week
with v5.11-rc5; it ran without problem last week on v5.11-rc4.  Is
this a known problem with a fix already, or is this something new?

% uname -r
5.11.0-0.rc5.134.fc34.x86_64
% pwd
/.../linux/tools/testing/selftests/bpf
% git log --oneline | head -n 1
6ee1d745b7c9 Linux 5.11-rc5
% make test_verifier_log
  ...
  BINARY   test_verifier_log
% ./test_verifier_log
Test log_level 0...
Test log_size < 128...
Test log_buff = NULL...
Test oversized buffer...
ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13

-- 
paul moore
www.paul-moore.com

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Alexei Starovoitov]

On Mon, Jan 25, 2021 at 12:54 PM Paul Moore <paul@paul-moore.com> wrote:
>
> Hello all,
>
> My apologies if this has already been reported, but I didn't see
> anything obvious with a quick search through the archives.  I have a
> test program that behaves very similar to the existing
> selftest/bpf/test_verifier_log test that has started failing this week
> with v5.11-rc5; it ran without problem last week on v5.11-rc4.  Is
> this a known problem with a fix already, or is this something new?
>
> % uname -r
> 5.11.0-0.rc5.134.fc34.x86_64
> % pwd
> /.../linux/tools/testing/selftests/bpf
> % git log --oneline | head -n 1
> 6ee1d745b7c9 Linux 5.11-rc5
> % make test_verifier_log
>   ...
>   BINARY   test_verifier_log
> % ./test_verifier_log
> Test log_level 0...
> Test log_size < 128...
> Test log_buff = NULL...
> Test oversized buffer...
> ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13

Thanks for reporting.
bpf and bpf-next don't have this issue. Not sure what changed.

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Paul Moore]

On Mon, Jan 25, 2021 at 5:42 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
> On Mon, Jan 25, 2021 at 12:54 PM Paul Moore <paul@paul-moore.com> wrote:
> >
> > Hello all,
> >
> > My apologies if this has already been reported, but I didn't see
> > anything obvious with a quick search through the archives.  I have a
> > test program that behaves very similar to the existing
> > selftest/bpf/test_verifier_log test that has started failing this week
> > with v5.11-rc5; it ran without problem last week on v5.11-rc4.  Is
> > this a known problem with a fix already, or is this something new?
> >
> > % uname -r
> > 5.11.0-0.rc5.134.fc34.x86_64
> > % pwd
> > /.../linux/tools/testing/selftests/bpf
> > % git log --oneline | head -n 1
> > 6ee1d745b7c9 Linux 5.11-rc5
> > % make test_verifier_log
> >   ...
> >   BINARY   test_verifier_log
> > % ./test_verifier_log
> > Test log_level 0...
> > Test log_size < 128...
> > Test log_buff = NULL...
> > Test oversized buffer...
> > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13
>
> Thanks for reporting.
> bpf and bpf-next don't have this issue. Not sure what changed.

I haven't had a chance to look into this any further, but Ondrej
Mosnacek (CC'd) found the following today:

"So I was trying to debug this further and I think I've identified what
triggers the problem. It seems that the BTF debuginfo generation
became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4
and -rc5. It also seems to depend on a recent (Fedora Rawhide) version
of some component of the build system (GCC, probably), because the
problem disappeared when I tried to build the "bad" kernel in F33
buildroot instead of Rawhide."

-- 
paul moore
www.paul-moore.com

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Alexei Starovoitov]

On Fri, Jan 29, 2021 at 2:15 PM Paul Moore <paul@paul-moore.com> wrote:
>
> On Mon, Jan 25, 2021 at 5:42 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> > On Mon, Jan 25, 2021 at 12:54 PM Paul Moore <paul@paul-moore.com> wrote:
> > >
> > > Hello all,
> > >
> > > My apologies if this has already been reported, but I didn't see
> > > anything obvious with a quick search through the archives.  I have a
> > > test program that behaves very similar to the existing
> > > selftest/bpf/test_verifier_log test that has started failing this week
> > > with v5.11-rc5; it ran without problem last week on v5.11-rc4.  Is
> > > this a known problem with a fix already, or is this something new?
> > >
> > > % uname -r
> > > 5.11.0-0.rc5.134.fc34.x86_64
> > > % pwd
> > > /.../linux/tools/testing/selftests/bpf
> > > % git log --oneline | head -n 1
> > > 6ee1d745b7c9 Linux 5.11-rc5
> > > % make test_verifier_log
> > >   ...
> > >   BINARY   test_verifier_log
> > > % ./test_verifier_log
> > > Test log_level 0...
> > > Test log_size < 128...
> > > Test log_buff = NULL...
> > > Test oversized buffer...
> > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13
> >
> > Thanks for reporting.
> > bpf and bpf-next don't have this issue. Not sure what changed.
>
> I haven't had a chance to look into this any further, but Ondrej
> Mosnacek (CC'd) found the following today:
>
> "So I was trying to debug this further and I think I've identified what
> triggers the problem. It seems that the BTF debuginfo generation
> became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4
> and -rc5. It also seems to depend on a recent (Fedora Rawhide) version
> of some component of the build system (GCC, probably), because the
> problem disappeared when I tried to build the "bad" kernel in F33
> buildroot instead of Rawhide."

I see. There were fixes for dwarf and btf, but I lost the track.
I believe it was a combination of gcc bug that was worked around in pahole.
Arnaldo, Jiri, Andrii,
what is the status? Did all fixes land in pahole?

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Jiri Olsa]

On Fri, Jan 29, 2021 at 07:13:21PM -0800, Alexei Starovoitov wrote:
> On Fri, Jan 29, 2021 at 2:15 PM Paul Moore <paul@paul-moore.com> wrote:
> >
> > On Mon, Jan 25, 2021 at 5:42 PM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> > > On Mon, Jan 25, 2021 at 12:54 PM Paul Moore <paul@paul-moore.com> wrote:
> > > >
> > > > Hello all,
> > > >
> > > > My apologies if this has already been reported, but I didn't see
> > > > anything obvious with a quick search through the archives.  I have a
> > > > test program that behaves very similar to the existing
> > > > selftest/bpf/test_verifier_log test that has started failing this week
> > > > with v5.11-rc5; it ran without problem last week on v5.11-rc4.  Is
> > > > this a known problem with a fix already, or is this something new?
> > > >
> > > > % uname -r
> > > > 5.11.0-0.rc5.134.fc34.x86_64
> > > > % pwd
> > > > /.../linux/tools/testing/selftests/bpf
> > > > % git log --oneline | head -n 1
> > > > 6ee1d745b7c9 Linux 5.11-rc5
> > > > % make test_verifier_log
> > > >   ...
> > > >   BINARY   test_verifier_log
> > > > % ./test_verifier_log
> > > > Test log_level 0...
> > > > Test log_size < 128...
> > > > Test log_buff = NULL...
> > > > Test oversized buffer...
> > > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13
> > >
> > > Thanks for reporting.
> > > bpf and bpf-next don't have this issue. Not sure what changed.
> >
> > I haven't had a chance to look into this any further, but Ondrej
> > Mosnacek (CC'd) found the following today:
> >
> > "So I was trying to debug this further and I think I've identified what
> > triggers the problem. It seems that the BTF debuginfo generation
> > became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4
> > and -rc5. It also seems to depend on a recent (Fedora Rawhide) version
> > of some component of the build system (GCC, probably), because the
> > problem disappeared when I tried to build the "bad" kernel in F33
> > buildroot instead of Rawhide."
> 
> I see. There were fixes for dwarf and btf, but I lost the track.
> I believe it was a combination of gcc bug that was worked around in pahole.
> Arnaldo, Jiri, Andrii,
> what is the status? Did all fixes land in pahole?

I checked on rawhide and besides many pahole warnings,
the resulted BTF data have many duplications in core structs

	  BTFIDS  vmlinux
	WARN: multiple IDs found for 'task_struct': 132, 1247 - using 132
	WARN: multiple IDs found for 'file': 440, 1349 - using 440
	WARN: multiple IDs found for 'inode': 698, 1645 - using 698
	WARN: multiple IDs found for 'path': 729, 1672 - using 729
	WARN: multiple IDs found for 'task_struct': 132, 2984 - using 132
	WARN: multiple IDs found for 'task_struct': 132, 3043 - using 132
	WARN: multiple IDs found for 'file': 440, 3085 - using 440
	WARN: multiple IDs found for 'seq_file': 1469, 3125 - using 1469
	WARN: multiple IDs found for 'inode': 698, 3336 - using 698
	WARN: multiple IDs found for 'path': 729, 3366 - using 729
	WARN: multiple IDs found for 'task_struct': 132, 5337 - using 132
	WARN: multiple IDs found for 'inode': 698, 5360 - using 698
	WARN: multiple IDs found for 'path': 729, 5388 - using 729
	WARN: multiple IDs found for 'file': 440, 5412 - using 440
	WARN: multiple IDs found for 'seq_file': 1469, 5639 - using 1469
	WARN: multiple IDs found for 'task_struct': 132, 6243 - using 132
	...

	# gcc --version
	gcc (GCC) 11.0.0 20210123 (Red Hat 11.0.0-0)

I'm guessing there are some DWARF changes that screwed BTF
generation.. I'll check

it's not covered by the fix I posted recently, but I think
Arnaldo is now fixing some related stuff.. Arnaldo, maybe
you are seeing same errors?

I uploaded the build log from linking part to:
  http://people.redhat.com/~jolsa/build.out.gz

jirka

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Jiri Olsa]

On Sat, Jan 30, 2021 at 09:48:13PM +0100, Jiri Olsa wrote:

SNIP

> > > > > % uname -r
> > > > > 5.11.0-0.rc5.134.fc34.x86_64
> > > > > % pwd
> > > > > /.../linux/tools/testing/selftests/bpf
> > > > > % git log --oneline | head -n 1
> > > > > 6ee1d745b7c9 Linux 5.11-rc5
> > > > > % make test_verifier_log
> > > > >   ...
> > > > >   BINARY   test_verifier_log
> > > > > % ./test_verifier_log
> > > > > Test log_level 0...
> > > > > Test log_size < 128...
> > > > > Test log_buff = NULL...
> > > > > Test oversized buffer...
> > > > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13
> > > >
> > > > Thanks for reporting.
> > > > bpf and bpf-next don't have this issue. Not sure what changed.
> > >
> > > I haven't had a chance to look into this any further, but Ondrej
> > > Mosnacek (CC'd) found the following today:
> > >
> > > "So I was trying to debug this further and I think I've identified what
> > > triggers the problem. It seems that the BTF debuginfo generation
> > > became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4
> > > and -rc5. It also seems to depend on a recent (Fedora Rawhide) version
> > > of some component of the build system (GCC, probably), because the
> > > problem disappeared when I tried to build the "bad" kernel in F33
> > > buildroot instead of Rawhide."
> > 
> > I see. There were fixes for dwarf and btf, but I lost the track.
> > I believe it was a combination of gcc bug that was worked around in pahole.
> > Arnaldo, Jiri, Andrii,
> > what is the status? Did all fixes land in pahole?
> 
> I checked on rawhide and besides many pahole warnings,
> the resulted BTF data have many duplications in core structs
> 
> 	  BTFIDS  vmlinux
> 	WARN: multiple IDs found for 'task_struct': 132, 1247 - using 132
> 	WARN: multiple IDs found for 'file': 440, 1349 - using 440
> 	WARN: multiple IDs found for 'inode': 698, 1645 - using 698
> 	WARN: multiple IDs found for 'path': 729, 1672 - using 729
> 	WARN: multiple IDs found for 'task_struct': 132, 2984 - using 132
> 	WARN: multiple IDs found for 'task_struct': 132, 3043 - using 132
> 	WARN: multiple IDs found for 'file': 440, 3085 - using 440
> 	WARN: multiple IDs found for 'seq_file': 1469, 3125 - using 1469
> 	WARN: multiple IDs found for 'inode': 698, 3336 - using 698
> 	WARN: multiple IDs found for 'path': 729, 3366 - using 729
> 	WARN: multiple IDs found for 'task_struct': 132, 5337 - using 132
> 	WARN: multiple IDs found for 'inode': 698, 5360 - using 698
> 	WARN: multiple IDs found for 'path': 729, 5388 - using 729
> 	WARN: multiple IDs found for 'file': 440, 5412 - using 440
> 	WARN: multiple IDs found for 'seq_file': 1469, 5639 - using 1469
> 	WARN: multiple IDs found for 'task_struct': 132, 6243 - using 132
> 	...
> 
> 	# gcc --version
> 	gcc (GCC) 11.0.0 20210123 (Red Hat 11.0.0-0)
> 
> I'm guessing there are some DWARF changes that screwed BTF
> generation.. I'll check
> 
> it's not covered by the fix I posted recently, but I think
> Arnaldo is now fixing some related stuff.. Arnaldo, maybe
> you are seeing same errors?

with Arnaldo's fixes I see less struct duplications,
but still there's some

> 
> I uploaded the build log from linking part to:
>   http://people.redhat.com/~jolsa/build.out.gz

however looks like we don't handle DW_FORM_implicit_const
when counting the byte offset.. it was used for some struct
members in my vmlinux, so we got zero for byte offset and
that created another unique struct

with patch below I no longer see any struct duplication,
also test_verifier_log is working for me, but I could
not reproduce the error before

I'll post full dwarves patch after some more testing

also I wonder we could somehow use btf_check_all_metas
from kernel after we build BTF data, that'd help to catch
this earlier/easier ;-) I'll check on this

jirka


---
diff --git a/dwarf_loader.c b/dwarf_loader.c
index ac22c1b..e2981a4 100644
--- a/dwarf_loader.c
+++ b/dwarf_loader.c
@@ -296,6 +296,7 @@ static Dwarf_Off __attr_offset(Dwarf_Attribute *attr)
 	Dwarf_Block block;
 
 	switch (dwarf_whatform(attr)) {
+	case DW_FORM_implicit_const:
 	case DW_FORM_data1:
 	case DW_FORM_data2:
 	case DW_FORM_data4:

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Arnaldo Carvalho de Melo]

Em Sun, Jan 31, 2021 at 10:36:41PM +0100, Jiri Olsa escreveu:
> On Sat, Jan 30, 2021 at 09:48:13PM +0100, Jiri Olsa wrote:
> 
> SNIP
> 
> > > > > > % uname -r
> > > > > > 5.11.0-0.rc5.134.fc34.x86_64
> > > > > > % pwd
> > > > > > /.../linux/tools/testing/selftests/bpf
> > > > > > % git log --oneline | head -n 1
> > > > > > 6ee1d745b7c9 Linux 5.11-rc5
> > > > > > % make test_verifier_log
> > > > > >   ...
> > > > > >   BINARY   test_verifier_log
> > > > > > % ./test_verifier_log
> > > > > > Test log_level 0...
> > > > > > Test log_size < 128...
> > > > > > Test log_buff = NULL...
> > > > > > Test oversized buffer...
> > > > > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13
> > > > >
> > > > > Thanks for reporting.
> > > > > bpf and bpf-next don't have this issue. Not sure what changed.
> > > >
> > > > I haven't had a chance to look into this any further, but Ondrej
> > > > Mosnacek (CC'd) found the following today:
> > > >
> > > > "So I was trying to debug this further and I think I've identified what
> > > > triggers the problem. It seems that the BTF debuginfo generation
> > > > became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4
> > > > and -rc5. It also seems to depend on a recent (Fedora Rawhide) version
> > > > of some component of the build system (GCC, probably), because the
> > > > problem disappeared when I tried to build the "bad" kernel in F33
> > > > buildroot instead of Rawhide."
> > > 
> > > I see. There were fixes for dwarf and btf, but I lost the track.
> > > I believe it was a combination of gcc bug that was worked around in pahole.
> > > Arnaldo, Jiri, Andrii,
> > > what is the status? Did all fixes land in pahole?
> > 
> > I checked on rawhide and besides many pahole warnings,
> > the resulted BTF data have many duplications in core structs
> > 
> > 	  BTFIDS  vmlinux
> > 	WARN: multiple IDs found for 'task_struct': 132, 1247 - using 132
> > 	WARN: multiple IDs found for 'file': 440, 1349 - using 440
> > 	WARN: multiple IDs found for 'inode': 698, 1645 - using 698
> > 	WARN: multiple IDs found for 'path': 729, 1672 - using 729
> > 	WARN: multiple IDs found for 'task_struct': 132, 2984 - using 132
> > 	WARN: multiple IDs found for 'task_struct': 132, 3043 - using 132
> > 	WARN: multiple IDs found for 'file': 440, 3085 - using 440
> > 	WARN: multiple IDs found for 'seq_file': 1469, 3125 - using 1469
> > 	WARN: multiple IDs found for 'inode': 698, 3336 - using 698
> > 	WARN: multiple IDs found for 'path': 729, 3366 - using 729
> > 	WARN: multiple IDs found for 'task_struct': 132, 5337 - using 132
> > 	WARN: multiple IDs found for 'inode': 698, 5360 - using 698
> > 	WARN: multiple IDs found for 'path': 729, 5388 - using 729
> > 	WARN: multiple IDs found for 'file': 440, 5412 - using 440
> > 	WARN: multiple IDs found for 'seq_file': 1469, 5639 - using 1469
> > 	WARN: multiple IDs found for 'task_struct': 132, 6243 - using 132
> > 	...
> > 
> > 	# gcc --version
> > 	gcc (GCC) 11.0.0 20210123 (Red Hat 11.0.0-0)
> > 
> > I'm guessing there are some DWARF changes that screwed BTF
> > generation.. I'll check
> > 
> > it's not covered by the fix I posted recently, but I think
> > Arnaldo is now fixing some related stuff.. Arnaldo, maybe
> > you are seeing same errors?
> 
> with Arnaldo's fixes I see less struct duplications,
> but still there's some
> 
> > 
> > I uploaded the build log from linking part to:
> >   http://people.redhat.com/~jolsa/build.out.gz
> 
> however looks like we don't handle DW_FORM_implicit_const
> when counting the byte offset.. it was used for some struct
> members in my vmlinux, so we got zero for byte offset and
> that created another unique struct
> 
> with patch below I no longer see any struct duplication,
> also test_verifier_log is working for me, but I could
> not reproduce the error before
> 
> I'll post full dwarves patch after some more testing
> 
> also I wonder we could somehow use btf_check_all_metas
> from kernel after we build BTF data, that'd help to catch
> this earlier/easier ;-) I'll check on this

Seems like a good idea indeed :-)

I'm applying the patch below with your Signed-off-by, etc, ok?

- Arnaldo
 
> jirka
> 
> 
> ---
> diff --git a/dwarf_loader.c b/dwarf_loader.c
> index ac22c1b..e2981a4 100644
> --- a/dwarf_loader.c
> +++ b/dwarf_loader.c
> @@ -296,6 +296,7 @@ static Dwarf_Off __attr_offset(Dwarf_Attribute *attr)
>  	Dwarf_Block block;
>  
>  	switch (dwarf_whatform(attr)) {
> +	case DW_FORM_implicit_const:
>  	case DW_FORM_data1:
>  	case DW_FORM_data2:
>  	case DW_FORM_data4:
> 

-- 

- Arnaldo

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Jiri Olsa]

On Mon, Feb 01, 2021 at 09:25:32AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Sun, Jan 31, 2021 at 10:36:41PM +0100, Jiri Olsa escreveu:
> > On Sat, Jan 30, 2021 at 09:48:13PM +0100, Jiri Olsa wrote:
> > 
> > SNIP
> > 
> > > > > > > % uname -r
> > > > > > > 5.11.0-0.rc5.134.fc34.x86_64
> > > > > > > % pwd
> > > > > > > /.../linux/tools/testing/selftests/bpf
> > > > > > > % git log --oneline | head -n 1
> > > > > > > 6ee1d745b7c9 Linux 5.11-rc5
> > > > > > > % make test_verifier_log
> > > > > > >   ...
> > > > > > >   BINARY   test_verifier_log
> > > > > > > % ./test_verifier_log
> > > > > > > Test log_level 0...
> > > > > > > Test log_size < 128...
> > > > > > > Test log_buff = NULL...
> > > > > > > Test oversized buffer...
> > > > > > > ERROR: Program load returned: ret:-1/errno:22, expected ret:-1/errno:13
> > > > > >
> > > > > > Thanks for reporting.
> > > > > > bpf and bpf-next don't have this issue. Not sure what changed.
> > > > >
> > > > > I haven't had a chance to look into this any further, but Ondrej
> > > > > Mosnacek (CC'd) found the following today:
> > > > >
> > > > > "So I was trying to debug this further and I think I've identified what
> > > > > triggers the problem. It seems that the BTF debuginfo generation
> > > > > became broken with CONFIG_DEBUG_INFO_DWARF4=n somewhere between -rc4
> > > > > and -rc5. It also seems to depend on a recent (Fedora Rawhide) version
> > > > > of some component of the build system (GCC, probably), because the
> > > > > problem disappeared when I tried to build the "bad" kernel in F33
> > > > > buildroot instead of Rawhide."
> > > > 
> > > > I see. There were fixes for dwarf and btf, but I lost the track.
> > > > I believe it was a combination of gcc bug that was worked around in pahole.
> > > > Arnaldo, Jiri, Andrii,
> > > > what is the status? Did all fixes land in pahole?
> > > 
> > > I checked on rawhide and besides many pahole warnings,
> > > the resulted BTF data have many duplications in core structs
> > > 
> > > 	  BTFIDS  vmlinux
> > > 	WARN: multiple IDs found for 'task_struct': 132, 1247 - using 132
> > > 	WARN: multiple IDs found for 'file': 440, 1349 - using 440
> > > 	WARN: multiple IDs found for 'inode': 698, 1645 - using 698
> > > 	WARN: multiple IDs found for 'path': 729, 1672 - using 729
> > > 	WARN: multiple IDs found for 'task_struct': 132, 2984 - using 132
> > > 	WARN: multiple IDs found for 'task_struct': 132, 3043 - using 132
> > > 	WARN: multiple IDs found for 'file': 440, 3085 - using 440
> > > 	WARN: multiple IDs found for 'seq_file': 1469, 3125 - using 1469
> > > 	WARN: multiple IDs found for 'inode': 698, 3336 - using 698
> > > 	WARN: multiple IDs found for 'path': 729, 3366 - using 729
> > > 	WARN: multiple IDs found for 'task_struct': 132, 5337 - using 132
> > > 	WARN: multiple IDs found for 'inode': 698, 5360 - using 698
> > > 	WARN: multiple IDs found for 'path': 729, 5388 - using 729
> > > 	WARN: multiple IDs found for 'file': 440, 5412 - using 440
> > > 	WARN: multiple IDs found for 'seq_file': 1469, 5639 - using 1469
> > > 	WARN: multiple IDs found for 'task_struct': 132, 6243 - using 132
> > > 	...
> > > 
> > > 	# gcc --version
> > > 	gcc (GCC) 11.0.0 20210123 (Red Hat 11.0.0-0)
> > > 
> > > I'm guessing there are some DWARF changes that screwed BTF
> > > generation.. I'll check
> > > 
> > > it's not covered by the fix I posted recently, but I think
> > > Arnaldo is now fixing some related stuff.. Arnaldo, maybe
> > > you are seeing same errors?
> > 
> > with Arnaldo's fixes I see less struct duplications,
> > but still there's some
> > 
> > > 
> > > I uploaded the build log from linking part to:
> > >   http://people.redhat.com/~jolsa/build.out.gz
> > 
> > however looks like we don't handle DW_FORM_implicit_const
> > when counting the byte offset.. it was used for some struct
> > members in my vmlinux, so we got zero for byte offset and
> > that created another unique struct
> > 
> > with patch below I no longer see any struct duplication,
> > also test_verifier_log is working for me, but I could
> > not reproduce the error before
> > 
> > I'll post full dwarves patch after some more testing
> > 
> > also I wonder we could somehow use btf_check_all_metas
> > from kernel after we build BTF data, that'd help to catch
> > this earlier/easier ;-) I'll check on this
> 
> Seems like a good idea indeed :-)
> 
> I'm applying the patch below with your Signed-off-by, etc, ok?

ok, thanks

jirka

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Jiri Olsa]

On Mon, Feb 01, 2021 at 03:50:22PM +0100, Jiri Olsa wrote:

SNIP

> > > 
> > > with Arnaldo's fixes I see less struct duplications,
> > > but still there's some
> > > 
> > > > 
> > > > I uploaded the build log from linking part to:
> > > >   http://people.redhat.com/~jolsa/build.out.gz
> > > 
> > > however looks like we don't handle DW_FORM_implicit_const
> > > when counting the byte offset.. it was used for some struct
> > > members in my vmlinux, so we got zero for byte offset and
> > > that created another unique struct
> > > 
> > > with patch below I no longer see any struct duplication,
> > > also test_verifier_log is working for me, but I could
> > > not reproduce the error before
> > > 
> > > I'll post full dwarves patch after some more testing
> > > 
> > > also I wonder we could somehow use btf_check_all_metas
> > > from kernel after we build BTF data, that'd help to catch
> > > this earlier/easier ;-) I'll check on this
> > 
> > Seems like a good idea indeed :-)
> > 
> > I'm applying the patch below with your Signed-off-by, etc, ok?
> 
> ok, thanks

Paul, Ondrej,

I put all the recent fixes and made a scratch build:
  https://koji.fedoraproject.org/koji/taskinfo?taskID=61049457

if you have a chance to test and check your issue was resolved,
that'd be great

thanks,
jirka

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Ondrej Mosnacek]

On Mon, Feb 1, 2021 at 9:23 PM Jiri Olsa <jolsa@redhat.com> wrote:
> On Mon, Feb 01, 2021 at 03:50:22PM +0100, Jiri Olsa wrote:
>
> SNIP
>
> > > >
> > > > with Arnaldo's fixes I see less struct duplications,
> > > > but still there's some
> > > >
> > > > >
> > > > > I uploaded the build log from linking part to:
> > > > >   http://people.redhat.com/~jolsa/build.out.gz
> > > >
> > > > however looks like we don't handle DW_FORM_implicit_const
> > > > when counting the byte offset.. it was used for some struct
> > > > members in my vmlinux, so we got zero for byte offset and
> > > > that created another unique struct
> > > >
> > > > with patch below I no longer see any struct duplication,
> > > > also test_verifier_log is working for me, but I could
> > > > not reproduce the error before
> > > >
> > > > I'll post full dwarves patch after some more testing
> > > >
> > > > also I wonder we could somehow use btf_check_all_metas
> > > > from kernel after we build BTF data, that'd help to catch
> > > > this earlier/easier ;-) I'll check on this
> > >
> > > Seems like a good idea indeed :-)
> > >
> > > I'm applying the patch below with your Signed-off-by, etc, ok?
> >
> > ok, thanks
>
> Paul, Ondrej,
>
> I put all the recent fixes and made a scratch build:
>   https://koji.fedoraproject.org/koji/taskinfo?taskID=61049457
>
> if you have a chance to test and check your issue was resolved,
> that'd be great

I just built the current master branch of dwarves (d783117162c0, which
includes Jirka's patch) [1] in COPR [2] and then rebuilt the kernel
with it [3]. With the new dwarves, the issue seems to be fixed -
/sys/kernel/btf/vmlinux is back to ~4MB and the selinux-testsuite BPF
subtest passes.

Thanks everyone for getting to the bottom of this! Hoping to see an
updated dwarves in rawhide soon ;)

[1] https://github.com/acmel/dwarves/
[2] https://copr.fedorainfracloud.org/coprs/omos/kernel-btf-test/build/1930103/
[3] https://copr.fedorainfracloud.org/coprs/omos/kernel-btf-test/build/1930104/

--
Ondrej Mosnacek
Software Engineer, Platform Security - SELinux kernel
Red Hat, Inc.

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Paul Moore]

On Mon, Feb 1, 2021 at 5:43 PM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> On Mon, Feb 1, 2021 at 9:23 PM Jiri Olsa <jolsa@redhat.com> wrote:

...

> > Paul, Ondrej,
> >
> > I put all the recent fixes and made a scratch build:
> >   https://koji.fedoraproject.org/koji/taskinfo?taskID=61049457
> >
> > if you have a chance to test and check your issue was resolved,
> > that'd be great
>
> I just built the current master branch of dwarves (d783117162c0, which
> includes Jirka's patch) [1] in COPR [2] and then rebuilt the kernel
> with it [3]. With the new dwarves, the issue seems to be fixed -
> /sys/kernel/btf/vmlinux is back to ~4MB and the selinux-testsuite BPF
> subtest passes.
>
> Thanks everyone for getting to the bottom of this! Hoping to see an
> updated dwarves in rawhide soon ;)

Yes, thanks!

I've updated my test systems and I'm building a x86_64 and aarch64
kernel now to test; based on Ondrej's reports I'm sure it will work
just fine, but I'll report back if I run into any issues.

-- 
paul moore
www.paul-moore.com

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Arnaldo Carvalho de Melo]

Em Mon, Feb 01, 2021 at 11:43:26PM +0100, Ondrej Mosnacek escreveu:
> On Mon, Feb 1, 2021 at 9:23 PM Jiri Olsa <jolsa@redhat.com> wrote:
> > On Mon, Feb 01, 2021 at 03:50:22PM +0100, Jiri Olsa wrote:
> > Paul, Ondrej,

> > I put all the recent fixes and made a scratch build:
> >   https://koji.fedoraproject.org/koji/taskinfo?taskID=61049457

> > if you have a chance to test and check your issue was resolved,
> > that'd be great

> I just built the current master branch of dwarves (d783117162c0, which
> includes Jirka's patch) [1] in COPR [2] and then rebuilt the kernel
> with it [3]. With the new dwarves, the issue seems to be fixed -
> /sys/kernel/btf/vmlinux is back to ~4MB and the selinux-testsuite BPF
> subtest passes.

> Thanks everyone for getting to the bottom of this! Hoping to see an
> updated dwarves in rawhide soon ;)

Thanks a lot!

I've updated a f33 system to rawhide to test all this, fixed up some
extra warnings wrt mallinfo(), strndup() error path handling/potential
buffer overflow issue and will add a conditional define for
DW_FORM_implicit_const found in the libbpf CI tests that Andrii pointed
out to me, then go and tag 1.20 and do the rawhide/fedora package update
dance.

- Arnaldo
 
> [1] https://github.com/acmel/dwarves/
> [2] https://copr.fedorainfracloud.org/coprs/omos/kernel-btf-test/build/1930103/
> [3] https://copr.fedorainfracloud.org/coprs/omos/kernel-btf-test/build/1930104/

Re: selftest/bpf/test_verifier_log fails on v5.11-rc5

[Paul Moore]

On Tue, Feb 2, 2021 at 7:43 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> I've updated a f33 system to rawhide to test all this, fixed up some
> extra warnings wrt mallinfo(), strndup() error path handling/potential
> buffer overflow issue and will add a conditional define for
> DW_FORM_implicit_const found in the libbpf CI tests that Andrii pointed
> out to me, then go and tag 1.20 and do the rawhide/fedora package update
> dance.

Thanks for taking care of this.  FWIW, it looks like both my x86_64
and aarch64 test runs with your Rawhide scratch build went through
without a problem last night.

-- 
paul moore
www.paul-moore.com