perf test case 40 LLVM search and compile runs endless (eBPF linkage issue)

perf test case 40 LLVM search and compile runs endless (eBPF linkage issue)

[Thomas Richter]

On s390 the test case 40 'LLVM search and compile' runs for ever:

  # date; timeout 30s perf test -F 40; date
  Mon May 23 12:35:48 PM CEST 2022
   40: LLVM search and compile   :
   40.1: Basic BPF llvm compile  :
  Mon May 23 12:36:18 PM CEST 2022
  #

Investigation by Ilya Leoshkevich revealed:
On s390 we hit an endless loop where bpf_object__next_program and bpf_program__next 
infinitely call each other. We tracked this down to ld.so preferring the perf's 
bpf_object__next_program weak copy over the libbpf's strong one.

This is introduced with commit 2a4898fc264a ("perf tools: Add more weak libbpf functions")

Also see discussion in 
https://lore.kernel.org/lkml/YY%2FAR2Dcaws0BAE9@kernel.org/

How can this be fixed asap? Thanks a lot.
-- 
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294

Re: perf test case 40 LLVM search and compile runs endless (eBPF linkage issue)

[Jiri Olsa]

On Mon, May 23, 2022 at 12:52:06PM +0200, Thomas Richter wrote:
> On s390 the test case 40 'LLVM search and compile' runs for ever:
> 
>   # date; timeout 30s perf test -F 40; date
>   Mon May 23 12:35:48 PM CEST 2022
>    40: LLVM search and compile   :
>    40.1: Basic BPF llvm compile  :
>   Mon May 23 12:36:18 PM CEST 2022
>   #
> 
> Investigation by Ilya Leoshkevich revealed:
> On s390 we hit an endless loop where bpf_object__next_program and bpf_program__next 
> infinitely call each other. We tracked this down to ld.so preferring the perf's 
> bpf_object__next_program weak copy over the libbpf's strong one.
> 
> This is introduced with commit 2a4898fc264a ("perf tools: Add more weak libbpf functions")
> 
> Also see discussion in 
> https://lore.kernel.org/lkml/YY%2FAR2Dcaws0BAE9@kernel.org/
> 
> How can this be fixed asap? Thanks a lot.

I was just suggesting to remove that:
  https://lore.kernel.org/bpf/Yos8hq3NmBwemoJw@krava/T/#mda8abccbf475b99d25abe5498a5df1af2da82ee8

it's no longer needed and can cause problem apparently

jirka

Re: perf test case 40 LLVM search and compile runs endless (eBPF linkage issue)

[Jiri Olsa]

On Mon, May 23, 2022 at 01:09:51PM +0200, Jiri Olsa wrote:
> On Mon, May 23, 2022 at 12:52:06PM +0200, Thomas Richter wrote:
> > On s390 the test case 40 'LLVM search and compile' runs for ever:
> > 
> >   # date; timeout 30s perf test -F 40; date
> >   Mon May 23 12:35:48 PM CEST 2022
> >    40: LLVM search and compile   :
> >    40.1: Basic BPF llvm compile  :
> >   Mon May 23 12:36:18 PM CEST 2022
> >   #
> > 
> > Investigation by Ilya Leoshkevich revealed:
> > On s390 we hit an endless loop where bpf_object__next_program and bpf_program__next 
> > infinitely call each other. We tracked this down to ld.so preferring the perf's 
> > bpf_object__next_program weak copy over the libbpf's strong one.
> > 
> > This is introduced with commit 2a4898fc264a ("perf tools: Add more weak libbpf functions")
> > 
> > Also see discussion in 
> > https://lore.kernel.org/lkml/YY%2FAR2Dcaws0BAE9@kernel.org/
> > 
> > How can this be fixed asap? Thanks a lot.
> 
> I was just suggesting to remove that:
>   https://lore.kernel.org/bpf/Yos8hq3NmBwemoJw@krava/T/#mda8abccbf475b99d25abe5498a5df1af2da82ee8
> 
> it's no longer needed and can cause problem apparently

after discussing with Arnaldo we decided to go for another fix,
could you please try this branch:

  git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  perf/weak

thanks,
jirka

Re: perf test case 40 LLVM search and compile runs endless (eBPF linkage issue)

[Arnaldo Carvalho de Melo]

Em Tue, May 24, 2022 at 02:10:17PM +0200, Jiri Olsa escreveu:
> On Mon, May 23, 2022 at 01:09:51PM +0200, Jiri Olsa wrote:
> > On Mon, May 23, 2022 at 12:52:06PM +0200, Thomas Richter wrote:
> > > On s390 the test case 40 'LLVM search and compile' runs for ever:
> > > 
> > >   # date; timeout 30s perf test -F 40; date
> > >   Mon May 23 12:35:48 PM CEST 2022
> > >    40: LLVM search and compile   :
> > >    40.1: Basic BPF llvm compile  :
> > >   Mon May 23 12:36:18 PM CEST 2022
> > >   #
> > > 
> > > Investigation by Ilya Leoshkevich revealed:
> > > On s390 we hit an endless loop where bpf_object__next_program and bpf_program__next 
> > > infinitely call each other. We tracked this down to ld.so preferring the perf's 
> > > bpf_object__next_program weak copy over the libbpf's strong one.
> > > 
> > > This is introduced with commit 2a4898fc264a ("perf tools: Add more weak libbpf functions")
> > > 
> > > Also see discussion in 
> > > https://lore.kernel.org/lkml/YY%2FAR2Dcaws0BAE9@kernel.org/
> > > 
> > > How can this be fixed asap? Thanks a lot.
> > 
> > I was just suggesting to remove that:
> >   https://lore.kernel.org/bpf/Yos8hq3NmBwemoJw@krava/T/#mda8abccbf475b99d25abe5498a5df1af2da82ee8
> > 
> > it's no longer needed and can cause problem apparently
> 
> after discussing with Arnaldo we decided to go for another fix,
> could you please try this branch:
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
>   perf/weak

I'm adding comments, testing and applying that series in my local tree,
so far so good, will push to tmp.perf/core later.

Please let us know if you guys looked at the patches in Jiri's perf/weak
branch and if you are ok with it so that I can add
Tested-by/acked-by/reviewed-by, which are always more than welcome.

Thanks,

- Arnaldo

Re: perf test case 40 LLVM search and compile runs endless (eBPF linkage issue)

[Thomas Richter]

On 5/24/22 16:33, Arnaldo Carvalho de Melo wrote:
> Em Tue, May 24, 2022 at 02:10:17PM +0200, Jiri Olsa escreveu:
>> On Mon, May 23, 2022 at 01:09:51PM +0200, Jiri Olsa wrote:
>>> On Mon, May 23, 2022 at 12:52:06PM +0200, Thomas Richter wrote:
>>>> On s390 the test case 40 'LLVM search and compile' runs for ever:
>>>>
>>>>   # date; timeout 30s perf test -F 40; date
>>>>   Mon May 23 12:35:48 PM CEST 2022
>>>>    40: LLVM search and compile   :
>>>>    40.1: Basic BPF llvm compile  :
>>>>   Mon May 23 12:36:18 PM CEST 2022
>>>>   #
>>>>
>>>> Investigation by Ilya Leoshkevich revealed:
>>>> On s390 we hit an endless loop where bpf_object__next_program and bpf_program__next 
>>>> infinitely call each other. We tracked this down to ld.so preferring the perf's 
>>>> bpf_object__next_program weak copy over the libbpf's strong one.
>>>>
>>>> This is introduced with commit 2a4898fc264a ("perf tools: Add more weak libbpf functions")
>>>>
>>>> Also see discussion in 
>>>> https://lore.kernel.org/lkml/YY%2FAR2Dcaws0BAE9@kernel.org/
>>>>
>>>> How can this be fixed asap? Thanks a lot.
>>>
>>> I was just suggesting to remove that:
>>>   https://lore.kernel.org/bpf/Yos8hq3NmBwemoJw@krava/T/#mda8abccbf475b99d25abe5498a5df1af2da82ee8
>>>
>>> it's no longer needed and can cause problem apparently
>>
>> after discussing with Arnaldo we decided to go for another fix,
>> could you please try this branch:
>>
>>   git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
>>   perf/weak
> 
> I'm adding comments, testing and applying that series in my local tree,
> so far so good, will push to tmp.perf/core later.
> 
> Please let us know if you guys looked at the patches in Jiri's perf/weak
> branch and if you are ok with it so that I can add
> Tested-by/acked-by/reviewed-by, which are always more than welcome.
> 
> Thanks,
> 
> - Arnaldo

I have taken the repo and branch perf/weak and run it through our build process.
It worked fine and you have my Tested-by:

Thanks

-- 
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294