perf BPF/elfdep intermitent build problems

perf BPF/elfdep intermitent build problems

[Arnaldo Carvalho de Melo]

Have you guys ever saw this?

[acme@zoo linux]$ alias m
alias m='perf stat -e cycles make O=/tmp/build/perf -C tools/perf
install-bin'
[acme @zoo linux]$ m
make: Entering directory '/home/git/linux/tools/perf'
  BUILD:   Doing 'make -j4' parallel build
make[2]: Nothing to be done for 'fixdep'.
  CC       /tmp/build/perf/util/machine.o
  LD       /tmp/build/perf/util/libperf-in.o
No libelf found
BPF API too old
Makefile:191: recipe for target 'elfdep' failed
make[2]: *** [elfdep] Error 255
make[2]: *** Waiting for unfinished jobs....
Makefile:194: recipe for target 'bpfdep' failed
make[2]: *** [bpfdep] Error 255
Makefile.perf:434: recipe for target '/tmp/build/perf/libbpf.a' failed
make[1]: *** [/tmp/build/perf/libbpf.a] Error 2
make[1]: *** Waiting for unfinished jobs....
  LD       /tmp/build/perf/libperf-in.o
Makefile:87: recipe for target 'install-bin' failed
make: *** [install-bin] Error 2
make: Leaving directory '/home/git/linux/tools/perf'

 Performance counter stats for 'make O=/tmp/build/perf -C tools/perf
install-bin':

    10,265,920,369      cycles                                                      

       3.002799545 seconds time elapsed

[acme@zoo linux]$ 


If I go and try doing a clean on the libbpf dir I see another problem:

[acme@zoo linux]$ cd tools/lib/bpf
[acme@zoo bpf]$ make clean

Auto-detecting system features:
...                        libelf: [ OFF ]
...                           bpf: [ OFF ]

  CLEAN    libbpf
  CLEAN    core-gen
[acme@zoo bpf]$

No need for feature detection when doing a plain 'make clean', right?

But then, after cleaning it again, and entering tools/perf to do a 'make
clean' as well, it does the feature detection at a second 'make clean'
and this time it shoes [ on ] for both libelf and bpf, and this time
I manage to build perf.

This is all intermitent, will eventually try to drill down if you don't
fix it first

- Arnaldo

Re: perf BPF/elfdep intermitent build problems

[Wangnan (F)]

On 2015/11/19 6:44, Arnaldo Carvalho de Melo wrote:
> Have you guys ever saw this?
>
> [acme@zoo linux]$ alias m
> alias m='perf stat -e cycles make O=/tmp/build/perf -C tools/perf
> install-bin'
> [acme @zoo linux]$ m
> make: Entering directory '/home/git/linux/tools/perf'
>    BUILD:   Doing 'make -j4' parallel build
> make[2]: Nothing to be done for 'fixdep'.
>    CC       /tmp/build/perf/util/machine.o
>    LD       /tmp/build/perf/util/libperf-in.o
> No libelf found
> BPF API too old

This error message means the kernel header in rootfs is too old,
causes tools/build/feature/test-bpf.c failed to be built.

> Makefile:191: recipe for target 'elfdep' failed
> make[2]: *** [elfdep] Error 255
> make[2]: *** Waiting for unfinished jobs....
> Makefile:194: recipe for target 'bpfdep' failed
> make[2]: *** [bpfdep] Error 255
> Makefile.perf:434: recipe for target '/tmp/build/perf/libbpf.a' failed
> make[1]: *** [/tmp/build/perf/libbpf.a] Error 2
> make[1]: *** Waiting for unfinished jobs....
>    LD       /tmp/build/perf/libperf-in.o
> Makefile:87: recipe for target 'install-bin' failed
> make: *** [install-bin] Error 2
> make: Leaving directory '/home/git/linux/tools/perf'
>
>   Performance counter stats for 'make O=/tmp/build/perf -C tools/perf
> install-bin':
>
>      10,265,920,369      cycles
>
>         3.002799545 seconds time elapsed
>
> [acme@zoo linux]$
>
>
> If I go and try doing a clean on the libbpf dir I see another problem:
>
> [acme@zoo linux]$ cd tools/lib/bpf
> [acme@zoo bpf]$ make clean
>
> Auto-detecting system features:
> ...                        libelf: [ OFF ]
> ...                           bpf: [ OFF ]
>
>    CLEAN    libbpf
>    CLEAN    core-gen
> [acme@zoo bpf]$
>
> No need for feature detection when doing a plain 'make clean', right?
>
> But then, after cleaning it again, and entering tools/perf to do a 'make
> clean' as well, it does the feature detection at a second 'make clean'
> and this time it shoes [ on ] for both libelf and bpf, and this time
> I manage to build perf.
>
> This is all intermitent, will eventually try to drill down if you don't
> fix it first

First let me try to trigger it in my environment...

Thank you.

> - Arnaldo

Re: perf BPF/elfdep intermitent build problems

[Wangnan (F)]

On 2015/11/19 9:24, Wangnan (F) wrote:
>
>
> On 2015/11/19 6:44, Arnaldo Carvalho de Melo wrote:
>> Have you guys ever saw this?
>>
>> [acme@zoo linux]$ alias m
>> alias m='perf stat -e cycles make O=/tmp/build/perf -C tools/perf
>> install-bin'
>> [acme @zoo linux]$ m
>> make: Entering directory '/home/git/linux/tools/perf'
>>    BUILD:   Doing 'make -j4' parallel build
>> make[2]: Nothing to be done for 'fixdep'.
>>    CC       /tmp/build/perf/util/machine.o
>>    LD       /tmp/build/perf/util/libperf-in.o
>> No libelf found
>> BPF API too old
>
> This error message means the kernel header in rootfs is too old,
> causes tools/build/feature/test-bpf.c failed to be built.

Strange... even in feature detection it should check BPF API through
kernel source where you are using to build perf...

I guess you are not using NFS or share it with other people?

>
>> Makefile:191: recipe for target 'elfdep' failed
>> make[2]: *** [elfdep] Error 255
>> make[2]: *** Waiting for unfinished jobs....
>> Makefile:194: recipe for target 'bpfdep' failed
>> make[2]: *** [bpfdep] Error 255
>> Makefile.perf:434: recipe for target '/tmp/build/perf/libbpf.a' failed
>> make[1]: *** [/tmp/build/perf/libbpf.a] Error 2
>> make[1]: *** Waiting for unfinished jobs....
>>    LD       /tmp/build/perf/libperf-in.o
>> Makefile:87: recipe for target 'install-bin' failed
>> make: *** [install-bin] Error 2
>> make: Leaving directory '/home/git/linux/tools/perf'
>>
>>   Performance counter stats for 'make O=/tmp/build/perf -C tools/perf
>> install-bin':
>>
>>      10,265,920,369      cycles
>>
>>         3.002799545 seconds time elapsed
>>
>> [acme@zoo linux]$
>>
>>
>> If I go and try doing a clean on the libbpf dir I see another problem:
>>
>> [acme@zoo linux]$ cd tools/lib/bpf
>> [acme@zoo bpf]$ make clean
>>
>> Auto-detecting system features:
>> ...                        libelf: [ OFF ]
>> ...                           bpf: [ OFF ]
>>
>>    CLEAN    libbpf
>>    CLEAN    core-gen
>> [acme@zoo bpf]$
>>
>> No need for feature detection when doing a plain 'make clean', right?
>>

Right. It should be

$ cd tools/lib/bpf
$ make clean
   CLEAN    libbpf
   CLEAN    core-gen
$


After cleaning can you trigger this problem again?

>> But then, after cleaning it again, and entering tools/perf to do a 'make
>> clean' as well, it does the feature detection at a second 'make clean'
>> and this time it shoes [ on ] for both libelf and bpf, and this time
>> I manage to build perf.
>>
>> This is all intermitent, will eventually try to drill down if you don't
>> fix it first
>
> First let me try to trigger it in my environment...
>
> Thank you.
>
>> - Arnaldo
>
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: perf BPF/elfdep intermitent build problems

[Wangnan (F)]

On 2015/11/19 10:21, Wangnan (F) wrote:
>
>
> On 2015/11/19 9:24, Wangnan (F) wrote:
>>
>>
>> On 2015/11/19 6:44, Arnaldo Carvalho de Melo wrote:
>>> Have you guys ever saw this?
>>>
>>> [acme@zoo linux]$ alias m
>>> alias m='perf stat -e cycles make O=/tmp/build/perf -C tools/perf
>>> install-bin'
>>> [acme @zoo linux]$ m
>>> make: Entering directory '/home/git/linux/tools/perf'
>>>    BUILD:   Doing 'make -j4' parallel build
>>> make[2]: Nothing to be done for 'fixdep'.
>>>    CC       /tmp/build/perf/util/machine.o
>>>    LD       /tmp/build/perf/util/libperf-in.o
>>> No libelf found
>>> BPF API too old
>>
>> This error message means the kernel header in rootfs is too old,
>> causes tools/build/feature/test-bpf.c failed to be built.
>
> Strange... even in feature detection it should check BPF API through
> kernel source where you are using to build perf...
>
> I guess you are not using NFS or share it with other people?
>
>>
>>> Makefile:191: recipe for target 'elfdep' failed
>>> make[2]: *** [elfdep] Error 255
>>> make[2]: *** Waiting for unfinished jobs....
>>> Makefile:194: recipe for target 'bpfdep' failed
>>> make[2]: *** [bpfdep] Error 255
>>> Makefile.perf:434: recipe for target '/tmp/build/perf/libbpf.a' failed
>>> make[1]: *** [/tmp/build/perf/libbpf.a] Error 2
>>> make[1]: *** Waiting for unfinished jobs....
>>>    LD       /tmp/build/perf/libperf-in.o
>>> Makefile:87: recipe for target 'install-bin' failed
>>> make: *** [install-bin] Error 2
>>> make: Leaving directory '/home/git/linux/tools/perf'
>>>
>>>   Performance counter stats for 'make O=/tmp/build/perf -C tools/perf
>>> install-bin':
>>>
>>>      10,265,920,369      cycles
>>>
>>>         3.002799545 seconds time elapsed
>>>
>>> [acme@zoo linux]$
>>>
>>>
>>> If I go and try doing a clean on the libbpf dir I see another problem:
>>>
>>> [acme@zoo linux]$ cd tools/lib/bpf
>>> [acme@zoo bpf]$ make clean
>>>
>>> Auto-detecting system features:
>>> ...                        libelf: [ OFF ]
>>> ...                           bpf: [ OFF ]
>>>
>>>    CLEAN    libbpf
>>>    CLEAN    core-gen
>>> [acme@zoo bpf]$
>>>
>>> No need for feature detection when doing a plain 'make clean', right?
>>>
>
> Right. It should be
>
> $ cd tools/lib/bpf
> $ make clean
>   CLEAN    libbpf
>   CLEAN    core-gen
> $
>
>
> After cleaning can you trigger this problem again?
>

Tiggered part of your problem:

$ make

Auto-detecting system features:
...                        libelf: [ on  ]
...                           bpf: [ on  ]

   CC       libbpf.o
   CC       bpf.o
   LD       libbpf-in.o
   LINK     libbpf.a
   LINK     libbpf.so
$ make clean
   CLEAN    libbpf
   CLEAN    core-gen
$ make clean

Auto-detecting system features:
...                        libelf: [ on  ]
...                           bpf: [ on  ]

   CLEAN    libbpf
   CLEAN    core-gen

Feature check would be triggered the second time we make clean.

Will look into it.

Thank you.

Re: perf BPF/elfdep intermitent build problems

[Wangnan (F)]

On 2015/11/19 10:29, Wangnan (F) wrote:
>
>
> On 2015/11/19 10:21, Wangnan (F) wrote:
>>

[SNIP]
>> Right. It should be
>>
>> $ cd tools/lib/bpf
>> $ make clean
>>   CLEAN    libbpf
>>   CLEAN    core-gen
>> $
>>
>>
>> After cleaning can you trigger this problem again?
>>
>
> Tiggered part of your problem:
>
> $ make
>
> Auto-detecting system features:
> ...                        libelf: [ on  ]
> ...                           bpf: [ on  ]
>
>   CC       libbpf.o
>   CC       bpf.o
>   LD       libbpf-in.o
>   LINK     libbpf.a
>   LINK     libbpf.so
> $ make clean
>   CLEAN    libbpf
>   CLEAN    core-gen
> $ make clean
>
> Auto-detecting system features:
> ...                        libelf: [ on  ]
> ...                           bpf: [ on  ]
>
>   CLEAN    libbpf
>   CLEAN    core-gen
>
> Feature check would be triggered the second time we make clean.
>
> Will look into it.
>

I understand this part of the problem.

There's no way for tools/build/Makefile.feature knowing whether
we are doing 'make clean' or 'make'. Therefore, even for
'make clean' those feature tests are mandatory to take action.

I posted a patch for it. Please see:

http://lkml.kernel.org/g/1447922706-99905-1-git-send-email-wangnan0@huawei.com

Thank you.

Re: perf BPF/elfdep intermitent build problems

[Wangnan (F)]

On 2015/11/19 6:44, Arnaldo Carvalho de Melo wrote:
> Have you guys ever saw this?
>
> [acme@zoo linux]$ alias m
> alias m='perf stat -e cycles make O=/tmp/build/perf -C tools/perf
> install-bin'
> [acme @zoo linux]$ m
> make: Entering directory '/home/git/linux/tools/perf'
>    BUILD:   Doing 'make -j4' parallel build
> make[2]: Nothing to be done for 'fixdep'.
>    CC       /tmp/build/perf/util/machine.o
>    LD       /tmp/build/perf/util/libperf-in.o

At the head of 'make' output I don't see

Auto-detecting system features:
...

So I guess your /tmp/build/perf is not an empty directory? Therefore
there might be some garbage in that directory. At least a FEATURE-DUMP
should be there, and that file should match perf's second pass feature
check, and in perf's checks libelf and libpf are both passed, or libbpf
won't be built.

> No libelf found
> BPF API too old

But when building libbpf, these two feature checks failed. Also, the
result is same to the garbage left in /tmp/build/perf/FEATURE-DUMP.libbpf,
or feature check result matrix should be output.

So there are something blocks lib/bpf's feature checks but not
affect perf itselves feature checks.

> Makefile:191: recipe for target 'elfdep' failed
> make[2]: *** [elfdep] Error 255
> make[2]: *** Waiting for unfinished jobs....
> Makefile:194: recipe for target 'bpfdep' failed
> make[2]: *** [bpfdep] Error 255
> Makefile.perf:434: recipe for target '/tmp/build/perf/libbpf.a' failed
> make[1]: *** [/tmp/build/perf/libbpf.a] Error 2
> make[1]: *** Waiting for unfinished jobs....
>    LD       /tmp/build/perf/libperf-in.o
> Makefile:87: recipe for target 'install-bin' failed
> make: *** [install-bin] Error 2
> make: Leaving directory '/home/git/linux/tools/perf'
>
>   Performance counter stats for 'make O=/tmp/build/perf -C tools/perf
> install-bin':
>
>      10,265,920,369      cycles
>
>         3.002799545 seconds time elapsed
>
> [acme@zoo linux]$
>
>
> If I go and try doing a clean on the libbpf dir I see another problem:
>
> [acme@zoo linux]$ cd tools/lib/bpf
> [acme@zoo bpf]$ make clean
>
> Auto-detecting system features:
> ...                        libelf: [ OFF ]
> ...                           bpf: [ OFF ]
>
>    CLEAN    libbpf
>    CLEAN    core-gen
> [acme@zoo bpf]$

Could you please provid the content of following files:

/tmp/build/perf/feature/test-libelf*make.output
/tmp/build/perf/feature/test-bpf.make.output

and those outputs in your perf source tree after you
do this 'perf clean'? If feature check fail these files should
tell us the reason.

> No need for feature detection when doing a plain 'make clean', right?

Please have a look at my patch "tools lib bpf: Don't feature check
when cleaning".

> But then, after cleaning it again, and entering tools/perf to do a 'make
> clean' as well, it does the feature detection at a second 'make clean'
> and this time it shoes [ on ] for both libelf and bpf, and this time
> I manage to build perf.

So the one blocks libbpf's feature checking is removed.
It inside perf source directory, but affects build dir...

Still no clue.

Jiri, can you find something?

Thank you.

Re: perf BPF/elfdep intermitent build problems

[Arnaldo Carvalho de Melo]

Em Thu, Nov 19, 2015 at 10:29:07AM +0800, Wangnan (F) escreveu:
> On 2015/11/19 10:21, Wangnan (F) wrote:
> >On 2015/11/19 9:24, Wangnan (F) wrote:
> >>On 2015/11/19 6:44, Arnaldo Carvalho de Melo wrote:
> >>>Have you guys ever saw this?
> >>>
> >>>[acme@zoo linux]$ alias m
> >>>alias m='perf stat -e cycles make O=/tmp/build/perf -C tools/perf
> >>>install-bin'
> >>>[acme @zoo linux]$ m
> >>>make: Entering directory '/home/git/linux/tools/perf'
> >>>   BUILD:   Doing 'make -j4' parallel build
> >>>make[2]: Nothing to be done for 'fixdep'.
> >>>   CC       /tmp/build/perf/util/machine.o
> >>>   LD       /tmp/build/perf/util/libperf-in.o
> >>>No libelf found
> >>>BPF API too old
> >>
> >>This error message means the kernel header in rootfs is too old,
> >>causes tools/build/feature/test-bpf.c failed to be built.
> >
> >Strange... even in feature detection it should check BPF API through
> >kernel source where you are using to build perf...
> >
> >I guess you are not using NFS or share it with other people?

Some build problem just happened and your question came to my mind,
yeah, sometimes I NFS mount it from a RHEL6 (and other distros) to do
'make -C tools/perf build-test' to check it works with those distros,
and I forgot to do a:

  make -C tools/ clean

I guess I'll have to try to make build-test support O=... So that
polluting the source tree doesn't causes problems like this.

Will try even to make it turn the source tree read only so that we spot 
when O= is not being honoured...

- Arnaldo