Re: [perf] perf probe fails sometimes on 4.9

Re: [perf] perf probe fails sometimes on 4.9

[Pintu Kumar]

Hi All,

'perf probe' is failing sometimes on 4.9.20 with AMD-64.
# perf probe --add schedule
schedule is out of .text, skip it.
  Error: Failed to add events.

If any one have come across this problem please let me know the cause.

Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
However, I manually copied the vmlinux file to /boot/ directory, but
still it does not work.

I checked upstream patches until 4.15 but could not find any clue.
Please let me know if there is any fixes available for this.


Thank You!
Regards,
Pintu


On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> Hi,
>
> ** Changed the subject now, since these issues are related to general
> perf commands.
>
> Following are the issues:
>
> 1) perf probe --add schedule - FAILED
> output:
> schedule is out of .text, skip it.
>   Error: Failed to add events.
>
> what is the issue here?
> Sometimes it pass and sometimes it fails...
> Similar is the case of 'perf inject' as well.
>
> 2) perf test - 1 FAILURE
> 37.1: Test basic BPF filtering                               : FAILED!
> 37.2: Test BPF prologue generation                           : Skip
> 37.3: Test BPF relocation checker                            : Skip
>
> bpf: config program 'func=SyS_epoll_wait'
> symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> bpf: config 'func=SyS_epoll_wait' is ok
> Looking at the vmlinux_path (8 entries long)
> Using /boot/vmlinux for symbols
> Could not open debuginfo. Try to use symbols.
> SyS_epoll_wait is out of .text, skip it.
> bpf_probe: failed to convert perf probe eventsFailed to add events
> selected by BPF
> test child finished with -1
> ---- end ----
> Test BPF filter subtest 0: FAILED!
>
> Looks like both 1,2 are related.
> Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> to /boot/ folder.
>
> -------
> Some more info:
>
> Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> unset env: KBUILD_OPTS
> include option is set to  -nostdinc -isystem
> /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> -I./include -I./arch/x86/include/uapi -I./include/uapi
> -I./include/generated/uapi -include ./include/linux/kconfig.h
> set env: NR_CPUS=8
> set env: LINUX_VERSION_CODE=0x40914
> set env: CLANG_EXEC=/usr/bin/clang
> set env: CLANG_OPTIONS=-xc
> set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> -I./include -I./arch/x86/include/uapi -I./include/uapi
> -I./include/generated/uapi -include ./include/linux/kconfig.h
> set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>
>
> If you have any clue about these failure please help me.
>
>
> Thanks,
> Pintu
>
>
> On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>> Hi,
>>
>> Thanks for your help.
>> Yes it was a sub version issue.
>>
>> Earlier, while building the kernel I left the SUBLEVEL as blank.
>>
>> make -j8 bindeb-pkg SUBLEVEL=20
>>
>> After passing the correct sublevel now the test is working.
>>
>> But still following are failing:
>>
>> 16: Try 'import perf' in python, checking link problems      : FAILED!
>> 37.2: Test BPF prologue generation                           : FAILED!
>>
>>
>> This is the error I get:
>>
>> 16: Try 'import perf' in python, checking link problems      :
>> --- start ---
>> test child forked, pid 7637
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> ImportError: No module named perf
>> test child finished with -1
>> ---- end ----
>> Try 'import perf' in python, checking link problems: FAILED!
>>
>>
>> Looking at the vmlinux_path (8 entries long)
>> symsrc__init: cannot get elf header.
>> Failed to find the path for kernel: Invalid ELF file
>> bpf_probe: failed to convert perf probe eventsFailed to add events
>> selected by BPF
>> test child finished with -1
>> ---- end ----
>> Test BPF filter subtest 1: FAILED!
>>
>>
>>
>> Thanks,
>> Pintu
>>
>>
>> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
>>>
>>>
>>> On 2018/1/23 20:37, Pintu Kumar wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
>>>> kernel version 4.9.20.
>>>> I have installed several others packages including: clang, llvm
>>>>
>>>> But, when I run 'perf test' I get some FAILURE.
>>>> Specially, 'perf test LLVM' is failing.
>>>>
>>>> Please check the below error logs:
>>>>
>>>> ----------------------------------------------------
>>>> # perf test LLVM
>>>> 35: Test LLVM searching and compiling                        :
>>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
>>>> 35.2: Test kbuild searching                                  : Skip
>>>> 35.3: Compile source for BPF prologue generation test        : Skip
>>>> 35.4: Compile source for BPF relocation test                 : Skip
>>>> ----------------------------------------------------
>>>>
>>>> When I run with -v I get this:
>>>>
>>>> -----------------------------------------------------
>>>> # perf test -v LLVM
>>>> 35: Test LLVM searching and compiling                        :
>>>> 35.1: Basic BPF llvm compiling test                          :
>>>> --- start ---
>>>> test child forked, pid 3304
>>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>>>> WARNING:        unable to get correct kernel building directory.
>>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
>>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
>>>>          detection.
>>>>
>>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>>>>
>>>>
>>>> int _version SEC("version") = LINUX_VERSION_CODE;
>>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
>>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
>>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
>>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
>>>> -
>>>> libbpf: 2129190-4a0 doesn't provide kernel version
>>>> Failed to parse test case 'Basic BPF llvm compiling test'
>>>> test child finished with -1
>>>> ---- end ----
>>>> Test LLVM searching and compiling subtest 0: FAILED!
>>>> -----------------------------------------------------
>>>>
>>>>
>>>> Basic BPF LLVM compiling test is failing.
>>>> It reports that bpf could not able to read the kernel version from uname.
>>>> Is it because of this that 'perf test LLVM' is failing ?
>>>>
>>>> My uname says:
>>>> 4.9-xxxx-amd-x86-64
>>>
>>>
>>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
>>> tools/perf/fetch_kernel_version . If your distro uses another method to
>>> report
>>> kernel version number (from your uname string, sublevel can't be
>>> determined),
>>> please help us improve that function like what we have done for ubuntu
>>> (checking /proc/version_signature).
>>>
>>>
>>>
>>>> My kernel version is: 4.9.20 (from Makefile)
>>>>
>>>> So, I dont think there is any problem with kernel version.
>>>>
>>>> If any body have come across this problem please let me know how to
>>>> resolve this issue.
>>>>
>>>>
>>>>
>>>> Thank You!
>>>>
>>>> Regards,
>>>> Pintu
>>>
>>>
>>>

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

[perf] perf probe fails sometimes on 4.9

[Pintu Kumar]

Hi All,

'perf probe' is failing sometimes on 4.9.20 with AMD-64.
# perf probe --add schedule
schedule is out of .text, skip it.
  Error: Failed to add events.

If any one have come across this problem please let me know the cause.

Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
However, I manually copied the vmlinux file to /boot/ directory, but
still it does not work.

I checked upstream patches until 4.15 but could not find any clue.
Please let me know if there is any fixes available for this.


Thank You!
Regards,
Pintu


On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> Hi,
>
> ** Changed the subject now, since these issues are related to general
> perf commands.
>
> Following are the issues:
>
> 1) perf probe --add schedule - FAILED
> output:
> schedule is out of .text, skip it.
>   Error: Failed to add events.
>
> what is the issue here?
> Sometimes it pass and sometimes it fails...
> Similar is the case of 'perf inject' as well.
>
> 2) perf test - 1 FAILURE
> 37.1: Test basic BPF filtering                               : FAILED!
> 37.2: Test BPF prologue generation                           : Skip
> 37.3: Test BPF relocation checker                            : Skip
>
> bpf: config program 'func=SyS_epoll_wait'
> symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> bpf: config 'func=SyS_epoll_wait' is ok
> Looking at the vmlinux_path (8 entries long)
> Using /boot/vmlinux for symbols
> Could not open debuginfo. Try to use symbols.
> SyS_epoll_wait is out of .text, skip it.
> bpf_probe: failed to convert perf probe eventsFailed to add events
> selected by BPF
> test child finished with -1
> ---- end ----
> Test BPF filter subtest 0: FAILED!
>
> Looks like both 1,2 are related.
> Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> to /boot/ folder.
>
> -------
> Some more info:
>
> Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> unset env: KBUILD_OPTS
> include option is set to  -nostdinc -isystem
> /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> -I./include -I./arch/x86/include/uapi -I./include/uapi
> -I./include/generated/uapi -include ./include/linux/kconfig.h
> set env: NR_CPUS=8
> set env: LINUX_VERSION_CODE=0x40914
> set env: CLANG_EXEC=/usr/bin/clang
> set env: CLANG_OPTIONS=-xc
> set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> -I./include -I./arch/x86/include/uapi -I./include/uapi
> -I./include/generated/uapi -include ./include/linux/kconfig.h
> set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>
>
> If you have any clue about these failure please help me.
>
>
> Thanks,
> Pintu
>
>
> On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>> Hi,
>>
>> Thanks for your help.
>> Yes it was a sub version issue.
>>
>> Earlier, while building the kernel I left the SUBLEVEL as blank.
>>
>> make -j8 bindeb-pkg SUBLEVEL=20
>>
>> After passing the correct sublevel now the test is working.
>>
>> But still following are failing:
>>
>> 16: Try 'import perf' in python, checking link problems      : FAILED!
>> 37.2: Test BPF prologue generation                           : FAILED!
>>
>>
>> This is the error I get:
>>
>> 16: Try 'import perf' in python, checking link problems      :
>> --- start ---
>> test child forked, pid 7637
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> ImportError: No module named perf
>> test child finished with -1
>> ---- end ----
>> Try 'import perf' in python, checking link problems: FAILED!
>>
>>
>> Looking at the vmlinux_path (8 entries long)
>> symsrc__init: cannot get elf header.
>> Failed to find the path for kernel: Invalid ELF file
>> bpf_probe: failed to convert perf probe eventsFailed to add events
>> selected by BPF
>> test child finished with -1
>> ---- end ----
>> Test BPF filter subtest 1: FAILED!
>>
>>
>>
>> Thanks,
>> Pintu
>>
>>
>> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
>>>
>>>
>>> On 2018/1/23 20:37, Pintu Kumar wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
>>>> kernel version 4.9.20.
>>>> I have installed several others packages including: clang, llvm
>>>>
>>>> But, when I run 'perf test' I get some FAILURE.
>>>> Specially, 'perf test LLVM' is failing.
>>>>
>>>> Please check the below error logs:
>>>>
>>>> ----------------------------------------------------
>>>> # perf test LLVM
>>>> 35: Test LLVM searching and compiling                        :
>>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
>>>> 35.2: Test kbuild searching                                  : Skip
>>>> 35.3: Compile source for BPF prologue generation test        : Skip
>>>> 35.4: Compile source for BPF relocation test                 : Skip
>>>> ----------------------------------------------------
>>>>
>>>> When I run with -v I get this:
>>>>
>>>> -----------------------------------------------------
>>>> # perf test -v LLVM
>>>> 35: Test LLVM searching and compiling                        :
>>>> 35.1: Basic BPF llvm compiling test                          :
>>>> --- start ---
>>>> test child forked, pid 3304
>>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>>>> WARNING:        unable to get correct kernel building directory.
>>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
>>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
>>>>          detection.
>>>>
>>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>>>>
>>>>
>>>> int _version SEC("version") = LINUX_VERSION_CODE;
>>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
>>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
>>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
>>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
>>>> -
>>>> libbpf: 2129190-4a0 doesn't provide kernel version
>>>> Failed to parse test case 'Basic BPF llvm compiling test'
>>>> test child finished with -1
>>>> ---- end ----
>>>> Test LLVM searching and compiling subtest 0: FAILED!
>>>> -----------------------------------------------------
>>>>
>>>>
>>>> Basic BPF LLVM compiling test is failing.
>>>> It reports that bpf could not able to read the kernel version from uname.
>>>> Is it because of this that 'perf test LLVM' is failing ?
>>>>
>>>> My uname says:
>>>> 4.9-xxxx-amd-x86-64
>>>
>>>
>>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
>>> tools/perf/fetch_kernel_version . If your distro uses another method to
>>> report
>>> kernel version number (from your uname string, sublevel can't be
>>> determined),
>>> please help us improve that function like what we have done for ubuntu
>>> (checking /proc/version_signature).
>>>
>>>
>>>
>>>> My kernel version is: 4.9.20 (from Makefile)
>>>>
>>>> So, I dont think there is any problem with kernel version.
>>>>
>>>> If any body have come across this problem please let me know how to
>>>> resolve this issue.
>>>>
>>>>
>>>>
>>>> Thank You!
>>>>
>>>> Regards,
>>>> Pintu
>>>
>>>
>>>

Re: [perf] perf probe fails sometimes on 4.9

[Masami Hiramatsu]

On Mon, 29 Jan 2018 13:40:34 +0530
Pintu Kumar <pintu.ping@gmail.com> wrote:

> Hi All,
> 
> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
> # perf probe --add schedule
> schedule is out of .text, skip it.
>   Error: Failed to add events.
> 
> If any one have come across this problem please let me know the cause.

Hi Pintu,

Could you run it with --vv?

> 
> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?

Without it, you can not probe source-level probe nor trace local variable.

> However, I manually copied the vmlinux file to /boot/ directory, but
> still it does not work.

That doesn't work.
CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
Without that option, debuginfo is not generated with vmlinux.

> 
> I checked upstream patches until 4.15 but could not find any clue.
> Please let me know if there is any fixes available for this.

Could you also ensure that you run perf by root user?

Thank you,


> 
> 
> Thank You!
> Regards,
> Pintu
> 
> 
> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > Hi,
> >
> > ** Changed the subject now, since these issues are related to general
> > perf commands.
> >
> > Following are the issues:
> >
> > 1) perf probe --add schedule - FAILED
> > output:
> > schedule is out of .text, skip it.
> >   Error: Failed to add events.
> >
> > what is the issue here?
> > Sometimes it pass and sometimes it fails...
> > Similar is the case of 'perf inject' as well.
> >
> > 2) perf test - 1 FAILURE
> > 37.1: Test basic BPF filtering                               : FAILED!
> > 37.2: Test BPF prologue generation                           : Skip
> > 37.3: Test BPF relocation checker                            : Skip
> >
> > bpf: config program 'func=SyS_epoll_wait'
> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> > bpf: config 'func=SyS_epoll_wait' is ok
> > Looking at the vmlinux_path (8 entries long)
> > Using /boot/vmlinux for symbols
> > Could not open debuginfo. Try to use symbols.
> > SyS_epoll_wait is out of .text, skip it.
> > bpf_probe: failed to convert perf probe eventsFailed to add events
> > selected by BPF
> > test child finished with -1
> > ---- end ----
> > Test BPF filter subtest 0: FAILED!
> >
> > Looks like both 1,2 are related.
> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> > to /boot/ folder.
> >
> > -------
> > Some more info:
> >
> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > unset env: KBUILD_OPTS
> > include option is set to  -nostdinc -isystem
> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > set env: NR_CPUS=8
> > set env: LINUX_VERSION_CODE=0x40914
> > set env: CLANG_EXEC=/usr/bin/clang
> > set env: CLANG_OPTIONS=-xc
> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> >
> >
> > If you have any clue about these failure please help me.
> >
> >
> > Thanks,
> > Pintu
> >
> >
> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> >> Hi,
> >>
> >> Thanks for your help.
> >> Yes it was a sub version issue.
> >>
> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
> >>
> >> make -j8 bindeb-pkg SUBLEVEL=20
> >>
> >> After passing the correct sublevel now the test is working.
> >>
> >> But still following are failing:
> >>
> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
> >> 37.2: Test BPF prologue generation                           : FAILED!
> >>
> >>
> >> This is the error I get:
> >>
> >> 16: Try 'import perf' in python, checking link problems      :
> >> --- start ---
> >> test child forked, pid 7637
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 1, in <module>
> >> ImportError: No module named perf
> >> test child finished with -1
> >> ---- end ----
> >> Try 'import perf' in python, checking link problems: FAILED!
> >>
> >>
> >> Looking at the vmlinux_path (8 entries long)
> >> symsrc__init: cannot get elf header.
> >> Failed to find the path for kernel: Invalid ELF file
> >> bpf_probe: failed to convert perf probe eventsFailed to add events
> >> selected by BPF
> >> test child finished with -1
> >> ---- end ----
> >> Test BPF filter subtest 1: FAILED!
> >>
> >>
> >>
> >> Thanks,
> >> Pintu
> >>
> >>
> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
> >>>
> >>>
> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
> >>>> kernel version 4.9.20.
> >>>> I have installed several others packages including: clang, llvm
> >>>>
> >>>> But, when I run 'perf test' I get some FAILURE.
> >>>> Specially, 'perf test LLVM' is failing.
> >>>>
> >>>> Please check the below error logs:
> >>>>
> >>>> ----------------------------------------------------
> >>>> # perf test LLVM
> >>>> 35: Test LLVM searching and compiling                        :
> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
> >>>> 35.2: Test kbuild searching                                  : Skip
> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
> >>>> 35.4: Compile source for BPF relocation test                 : Skip
> >>>> ----------------------------------------------------
> >>>>
> >>>> When I run with -v I get this:
> >>>>
> >>>> -----------------------------------------------------
> >>>> # perf test -v LLVM
> >>>> 35: Test LLVM searching and compiling                        :
> >>>> 35.1: Basic BPF llvm compiling test                          :
> >>>> --- start ---
> >>>> test child forked, pid 3304
> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >>>> WARNING:        unable to get correct kernel building directory.
> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
> >>>>          detection.
> >>>>
> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >>>>
> >>>>
> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
> >>>> -
> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
> >>>> test child finished with -1
> >>>> ---- end ----
> >>>> Test LLVM searching and compiling subtest 0: FAILED!
> >>>> -----------------------------------------------------
> >>>>
> >>>>
> >>>> Basic BPF LLVM compiling test is failing.
> >>>> It reports that bpf could not able to read the kernel version from uname.
> >>>> Is it because of this that 'perf test LLVM' is failing ?
> >>>>
> >>>> My uname says:
> >>>> 4.9-xxxx-amd-x86-64
> >>>
> >>>
> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
> >>> report
> >>> kernel version number (from your uname string, sublevel can't be
> >>> determined),
> >>> please help us improve that function like what we have done for ubuntu
> >>> (checking /proc/version_signature).
> >>>
> >>>
> >>>
> >>>> My kernel version is: 4.9.20 (from Makefile)
> >>>>
> >>>> So, I dont think there is any problem with kernel version.
> >>>>
> >>>> If any body have come across this problem please let me know how to
> >>>> resolve this issue.
> >>>>
> >>>>
> >>>>
> >>>> Thank You!
> >>>>
> >>>> Regards,
> >>>> Pintu
> >>>
> >>>
> >>>


-- 
Masami Hiramatsu <mhiramat@kernel.org>

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

[perf] perf probe fails sometimes on 4.9

[Masami Hiramatsu]

On Mon, 29 Jan 2018 13:40:34 +0530
Pintu Kumar <pintu.ping@gmail.com> wrote:

> Hi All,
> 
> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
> # perf probe --add schedule
> schedule is out of .text, skip it.
>   Error: Failed to add events.
> 
> If any one have come across this problem please let me know the cause.

Hi Pintu,

Could you run it with --vv?

> 
> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?

Without it, you can not probe source-level probe nor trace local variable.

> However, I manually copied the vmlinux file to /boot/ directory, but
> still it does not work.

That doesn't work.
CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
Without that option, debuginfo is not generated with vmlinux.

> 
> I checked upstream patches until 4.15 but could not find any clue.
> Please let me know if there is any fixes available for this.

Could you also ensure that you run perf by root user?

Thank you,


> 
> 
> Thank You!
> Regards,
> Pintu
> 
> 
> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > Hi,
> >
> > ** Changed the subject now, since these issues are related to general
> > perf commands.
> >
> > Following are the issues:
> >
> > 1) perf probe --add schedule - FAILED
> > output:
> > schedule is out of .text, skip it.
> >   Error: Failed to add events.
> >
> > what is the issue here?
> > Sometimes it pass and sometimes it fails...
> > Similar is the case of 'perf inject' as well.
> >
> > 2) perf test - 1 FAILURE
> > 37.1: Test basic BPF filtering                               : FAILED!
> > 37.2: Test BPF prologue generation                           : Skip
> > 37.3: Test BPF relocation checker                            : Skip
> >
> > bpf: config program 'func=SyS_epoll_wait'
> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> > bpf: config 'func=SyS_epoll_wait' is ok
> > Looking at the vmlinux_path (8 entries long)
> > Using /boot/vmlinux for symbols
> > Could not open debuginfo. Try to use symbols.
> > SyS_epoll_wait is out of .text, skip it.
> > bpf_probe: failed to convert perf probe eventsFailed to add events
> > selected by BPF
> > test child finished with -1
> > ---- end ----
> > Test BPF filter subtest 0: FAILED!
> >
> > Looks like both 1,2 are related.
> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> > to /boot/ folder.
> >
> > -------
> > Some more info:
> >
> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > unset env: KBUILD_OPTS
> > include option is set to  -nostdinc -isystem
> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > set env: NR_CPUS=8
> > set env: LINUX_VERSION_CODE=0x40914
> > set env: CLANG_EXEC=/usr/bin/clang
> > set env: CLANG_OPTIONS=-xc
> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> >
> >
> > If you have any clue about these failure please help me.
> >
> >
> > Thanks,
> > Pintu
> >
> >
> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> >> Hi,
> >>
> >> Thanks for your help.
> >> Yes it was a sub version issue.
> >>
> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
> >>
> >> make -j8 bindeb-pkg SUBLEVEL=20
> >>
> >> After passing the correct sublevel now the test is working.
> >>
> >> But still following are failing:
> >>
> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
> >> 37.2: Test BPF prologue generation                           : FAILED!
> >>
> >>
> >> This is the error I get:
> >>
> >> 16: Try 'import perf' in python, checking link problems      :
> >> --- start ---
> >> test child forked, pid 7637
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 1, in <module>
> >> ImportError: No module named perf
> >> test child finished with -1
> >> ---- end ----
> >> Try 'import perf' in python, checking link problems: FAILED!
> >>
> >>
> >> Looking at the vmlinux_path (8 entries long)
> >> symsrc__init: cannot get elf header.
> >> Failed to find the path for kernel: Invalid ELF file
> >> bpf_probe: failed to convert perf probe eventsFailed to add events
> >> selected by BPF
> >> test child finished with -1
> >> ---- end ----
> >> Test BPF filter subtest 1: FAILED!
> >>
> >>
> >>
> >> Thanks,
> >> Pintu
> >>
> >>
> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
> >>>
> >>>
> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
> >>>> kernel version 4.9.20.
> >>>> I have installed several others packages including: clang, llvm
> >>>>
> >>>> But, when I run 'perf test' I get some FAILURE.
> >>>> Specially, 'perf test LLVM' is failing.
> >>>>
> >>>> Please check the below error logs:
> >>>>
> >>>> ----------------------------------------------------
> >>>> # perf test LLVM
> >>>> 35: Test LLVM searching and compiling                        :
> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
> >>>> 35.2: Test kbuild searching                                  : Skip
> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
> >>>> 35.4: Compile source for BPF relocation test                 : Skip
> >>>> ----------------------------------------------------
> >>>>
> >>>> When I run with -v I get this:
> >>>>
> >>>> -----------------------------------------------------
> >>>> # perf test -v LLVM
> >>>> 35: Test LLVM searching and compiling                        :
> >>>> 35.1: Basic BPF llvm compiling test                          :
> >>>> --- start ---
> >>>> test child forked, pid 3304
> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >>>> WARNING:        unable to get correct kernel building directory.
> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
> >>>>          detection.
> >>>>
> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >>>>
> >>>>
> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
> >>>> -
> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
> >>>> test child finished with -1
> >>>> ---- end ----
> >>>> Test LLVM searching and compiling subtest 0: FAILED!
> >>>> -----------------------------------------------------
> >>>>
> >>>>
> >>>> Basic BPF LLVM compiling test is failing.
> >>>> It reports that bpf could not able to read the kernel version from uname.
> >>>> Is it because of this that 'perf test LLVM' is failing ?
> >>>>
> >>>> My uname says:
> >>>> 4.9-xxxx-amd-x86-64
> >>>
> >>>
> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
> >>> report
> >>> kernel version number (from your uname string, sublevel can't be
> >>> determined),
> >>> please help us improve that function like what we have done for ubuntu
> >>> (checking /proc/version_signature).
> >>>
> >>>
> >>>
> >>>> My kernel version is: 4.9.20 (from Makefile)
> >>>>
> >>>> So, I dont think there is any problem with kernel version.
> >>>>
> >>>> If any body have come across this problem please let me know how to
> >>>> resolve this issue.
> >>>>
> >>>>
> >>>>
> >>>> Thank You!
> >>>>
> >>>> Regards,
> >>>> Pintu
> >>>
> >>>
> >>>


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [perf] perf probe fails sometimes on 4.9

[Pintu Kumar]

Dear Masami,

Thank you so much for your reply.
Please find some of my answers inline.


On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> On Mon, 29 Jan 2018 13:40:34 +0530
> Pintu Kumar <pintu.ping@gmail.com> wrote:
>
>> Hi All,
>>
>> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
>> # perf probe --add schedule
>> schedule is out of .text, skip it.
>>   Error: Failed to add events.
>>
>> If any one have come across this problem please let me know the cause.
>
> Hi Pintu,
>
> Could you run it with --vv?
>
Ok, I will send verbose output by tomorrow.

>>
>> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
>
> Without it, you can not probe source-level probe nor trace local variable.
>

Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
However, I will try to manually include "-g" option during compilation.

>> However, I manually copied the vmlinux file to /boot/ directory, but
>> still it does not work.
>
> That doesn't work.
> CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
> Without that option, debuginfo is not generated with vmlinux.
>
>>
>> I checked upstream patches until 4.15 but could not find any clue.
>> Please let me know if there is any fixes available for this.
>
> Could you also ensure that you run perf by root user?
>

Yes I am running with root user.

My concern is sometimes it works but sometimes it fails.
I still needs to figure out, in which condition it works and which
condition it fails.
Usually I noticed that in fresh reboot case it works.


> Thank you,
>
>
>>
>>
>> Thank You!
>> Regards,
>> Pintu
>>
>>
>> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>> > Hi,
>> >
>> > ** Changed the subject now, since these issues are related to general
>> > perf commands.
>> >
>> > Following are the issues:
>> >
>> > 1) perf probe --add schedule - FAILED
>> > output:
>> > schedule is out of .text, skip it.
>> >   Error: Failed to add events.
>> >
>> > what is the issue here?
>> > Sometimes it pass and sometimes it fails...
>> > Similar is the case of 'perf inject' as well.
>> >
>> > 2) perf test - 1 FAILURE
>> > 37.1: Test basic BPF filtering                               : FAILED!
>> > 37.2: Test BPF prologue generation                           : Skip
>> > 37.3: Test BPF relocation checker                            : Skip
>> >
>> > bpf: config program 'func=SyS_epoll_wait'
>> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
>> > bpf: config 'func=SyS_epoll_wait' is ok
>> > Looking at the vmlinux_path (8 entries long)
>> > Using /boot/vmlinux for symbols
>> > Could not open debuginfo. Try to use symbols.
>> > SyS_epoll_wait is out of .text, skip it.
>> > bpf_probe: failed to convert perf probe eventsFailed to add events
>> > selected by BPF
>> > test child finished with -1
>> > ---- end ----
>> > Test BPF filter subtest 0: FAILED!
>> >
>> > Looks like both 1,2 are related.
>> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
>> > to /boot/ folder.
>> >
>> > -------
>> > Some more info:
>> >
>> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
>> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> > unset env: KBUILD_OPTS
>> > include option is set to  -nostdinc -isystem
>> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> > set env: NR_CPUS=8
>> > set env: LINUX_VERSION_CODE=0x40914
>> > set env: CLANG_EXEC=/usr/bin/clang
>> > set env: CLANG_OPTIONS=-xc
>> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
>> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> >
>> >
>> > If you have any clue about these failure please help me.
>> >
>> >
>> > Thanks,
>> > Pintu
>> >
>> >
>> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>> >> Hi,
>> >>
>> >> Thanks for your help.
>> >> Yes it was a sub version issue.
>> >>
>> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
>> >>
>> >> make -j8 bindeb-pkg SUBLEVEL=20
>> >>
>> >> After passing the correct sublevel now the test is working.
>> >>
>> >> But still following are failing:
>> >>
>> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
>> >> 37.2: Test BPF prologue generation                           : FAILED!
>> >>
>> >>
>> >> This is the error I get:
>> >>
>> >> 16: Try 'import perf' in python, checking link problems      :
>> >> --- start ---
>> >> test child forked, pid 7637
>> >> Traceback (most recent call last):
>> >>   File "<stdin>", line 1, in <module>
>> >> ImportError: No module named perf
>> >> test child finished with -1
>> >> ---- end ----
>> >> Try 'import perf' in python, checking link problems: FAILED!
>> >>
>> >>
>> >> Looking at the vmlinux_path (8 entries long)
>> >> symsrc__init: cannot get elf header.
>> >> Failed to find the path for kernel: Invalid ELF file
>> >> bpf_probe: failed to convert perf probe eventsFailed to add events
>> >> selected by BPF
>> >> test child finished with -1
>> >> ---- end ----
>> >> Test BPF filter subtest 1: FAILED!
>> >>
>> >>
>> >>
>> >> Thanks,
>> >> Pintu
>> >>
>> >>
>> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
>> >>>
>> >>>
>> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
>> >>>>
>> >>>> Hi All,
>> >>>>
>> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
>> >>>> kernel version 4.9.20.
>> >>>> I have installed several others packages including: clang, llvm
>> >>>>
>> >>>> But, when I run 'perf test' I get some FAILURE.
>> >>>> Specially, 'perf test LLVM' is failing.
>> >>>>
>> >>>> Please check the below error logs:
>> >>>>
>> >>>> ----------------------------------------------------
>> >>>> # perf test LLVM
>> >>>> 35: Test LLVM searching and compiling                        :
>> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
>> >>>> 35.2: Test kbuild searching                                  : Skip
>> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
>> >>>> 35.4: Compile source for BPF relocation test                 : Skip
>> >>>> ----------------------------------------------------
>> >>>>
>> >>>> When I run with -v I get this:
>> >>>>
>> >>>> -----------------------------------------------------
>> >>>> # perf test -v LLVM
>> >>>> 35: Test LLVM searching and compiling                        :
>> >>>> 35.1: Basic BPF llvm compiling test                          :
>> >>>> --- start ---
>> >>>> test child forked, pid 3304
>> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> >>>> WARNING:        unable to get correct kernel building directory.
>> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
>> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
>> >>>>          detection.
>> >>>>
>> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> >>>>
>> >>>>
>> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
>> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
>> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
>> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
>> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
>> >>>> -
>> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
>> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
>> >>>> test child finished with -1
>> >>>> ---- end ----
>> >>>> Test LLVM searching and compiling subtest 0: FAILED!
>> >>>> -----------------------------------------------------
>> >>>>
>> >>>>
>> >>>> Basic BPF LLVM compiling test is failing.
>> >>>> It reports that bpf could not able to read the kernel version from uname.
>> >>>> Is it because of this that 'perf test LLVM' is failing ?
>> >>>>
>> >>>> My uname says:
>> >>>> 4.9-xxxx-amd-x86-64
>> >>>
>> >>>
>> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
>> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
>> >>> report
>> >>> kernel version number (from your uname string, sublevel can't be
>> >>> determined),
>> >>> please help us improve that function like what we have done for ubuntu
>> >>> (checking /proc/version_signature).
>> >>>
>> >>>
>> >>>
>> >>>> My kernel version is: 4.9.20 (from Makefile)
>> >>>>
>> >>>> So, I dont think there is any problem with kernel version.
>> >>>>
>> >>>> If any body have come across this problem please let me know how to
>> >>>> resolve this issue.
>> >>>>
>> >>>>
>> >>>>
>> >>>> Thank You!
>> >>>>
>> >>>> Regards,
>> >>>> Pintu
>> >>>
>> >>>
>> >>>
>
>
> --
> Masami Hiramatsu <mhiramat@kernel.org>

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

[perf] perf probe fails sometimes on 4.9

[Pintu Kumar]

Dear Masami,

Thank you so much for your reply.
Please find some of my answers inline.


On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> On Mon, 29 Jan 2018 13:40:34 +0530
> Pintu Kumar <pintu.ping@gmail.com> wrote:
>
>> Hi All,
>>
>> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
>> # perf probe --add schedule
>> schedule is out of .text, skip it.
>>   Error: Failed to add events.
>>
>> If any one have come across this problem please let me know the cause.
>
> Hi Pintu,
>
> Could you run it with --vv?
>
Ok, I will send verbose output by tomorrow.

>>
>> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
>
> Without it, you can not probe source-level probe nor trace local variable.
>

Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
However, I will try to manually include "-g" option during compilation.

>> However, I manually copied the vmlinux file to /boot/ directory, but
>> still it does not work.
>
> That doesn't work.
> CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
> Without that option, debuginfo is not generated with vmlinux.
>
>>
>> I checked upstream patches until 4.15 but could not find any clue.
>> Please let me know if there is any fixes available for this.
>
> Could you also ensure that you run perf by root user?
>

Yes I am running with root user.

My concern is sometimes it works but sometimes it fails.
I still needs to figure out, in which condition it works and which
condition it fails.
Usually I noticed that in fresh reboot case it works.


> Thank you,
>
>
>>
>>
>> Thank You!
>> Regards,
>> Pintu
>>
>>
>> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>> > Hi,
>> >
>> > ** Changed the subject now, since these issues are related to general
>> > perf commands.
>> >
>> > Following are the issues:
>> >
>> > 1) perf probe --add schedule - FAILED
>> > output:
>> > schedule is out of .text, skip it.
>> >   Error: Failed to add events.
>> >
>> > what is the issue here?
>> > Sometimes it pass and sometimes it fails...
>> > Similar is the case of 'perf inject' as well.
>> >
>> > 2) perf test - 1 FAILURE
>> > 37.1: Test basic BPF filtering                               : FAILED!
>> > 37.2: Test BPF prologue generation                           : Skip
>> > 37.3: Test BPF relocation checker                            : Skip
>> >
>> > bpf: config program 'func=SyS_epoll_wait'
>> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
>> > bpf: config 'func=SyS_epoll_wait' is ok
>> > Looking at the vmlinux_path (8 entries long)
>> > Using /boot/vmlinux for symbols
>> > Could not open debuginfo. Try to use symbols.
>> > SyS_epoll_wait is out of .text, skip it.
>> > bpf_probe: failed to convert perf probe eventsFailed to add events
>> > selected by BPF
>> > test child finished with -1
>> > ---- end ----
>> > Test BPF filter subtest 0: FAILED!
>> >
>> > Looks like both 1,2 are related.
>> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
>> > to /boot/ folder.
>> >
>> > -------
>> > Some more info:
>> >
>> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
>> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> > unset env: KBUILD_OPTS
>> > include option is set to  -nostdinc -isystem
>> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> > set env: NR_CPUS=8
>> > set env: LINUX_VERSION_CODE=0x40914
>> > set env: CLANG_EXEC=/usr/bin/clang
>> > set env: CLANG_OPTIONS=-xc
>> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
>> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> >
>> >
>> > If you have any clue about these failure please help me.
>> >
>> >
>> > Thanks,
>> > Pintu
>> >
>> >
>> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>> >> Hi,
>> >>
>> >> Thanks for your help.
>> >> Yes it was a sub version issue.
>> >>
>> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
>> >>
>> >> make -j8 bindeb-pkg SUBLEVEL=20
>> >>
>> >> After passing the correct sublevel now the test is working.
>> >>
>> >> But still following are failing:
>> >>
>> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
>> >> 37.2: Test BPF prologue generation                           : FAILED!
>> >>
>> >>
>> >> This is the error I get:
>> >>
>> >> 16: Try 'import perf' in python, checking link problems      :
>> >> --- start ---
>> >> test child forked, pid 7637
>> >> Traceback (most recent call last):
>> >>   File "<stdin>", line 1, in <module>
>> >> ImportError: No module named perf
>> >> test child finished with -1
>> >> ---- end ----
>> >> Try 'import perf' in python, checking link problems: FAILED!
>> >>
>> >>
>> >> Looking at the vmlinux_path (8 entries long)
>> >> symsrc__init: cannot get elf header.
>> >> Failed to find the path for kernel: Invalid ELF file
>> >> bpf_probe: failed to convert perf probe eventsFailed to add events
>> >> selected by BPF
>> >> test child finished with -1
>> >> ---- end ----
>> >> Test BPF filter subtest 1: FAILED!
>> >>
>> >>
>> >>
>> >> Thanks,
>> >> Pintu
>> >>
>> >>
>> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
>> >>>
>> >>>
>> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
>> >>>>
>> >>>> Hi All,
>> >>>>
>> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
>> >>>> kernel version 4.9.20.
>> >>>> I have installed several others packages including: clang, llvm
>> >>>>
>> >>>> But, when I run 'perf test' I get some FAILURE.
>> >>>> Specially, 'perf test LLVM' is failing.
>> >>>>
>> >>>> Please check the below error logs:
>> >>>>
>> >>>> ----------------------------------------------------
>> >>>> # perf test LLVM
>> >>>> 35: Test LLVM searching and compiling                        :
>> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
>> >>>> 35.2: Test kbuild searching                                  : Skip
>> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
>> >>>> 35.4: Compile source for BPF relocation test                 : Skip
>> >>>> ----------------------------------------------------
>> >>>>
>> >>>> When I run with -v I get this:
>> >>>>
>> >>>> -----------------------------------------------------
>> >>>> # perf test -v LLVM
>> >>>> 35: Test LLVM searching and compiling                        :
>> >>>> 35.1: Basic BPF llvm compiling test                          :
>> >>>> --- start ---
>> >>>> test child forked, pid 3304
>> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> >>>> WARNING:        unable to get correct kernel building directory.
>> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
>> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
>> >>>>          detection.
>> >>>>
>> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> >>>>
>> >>>>
>> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
>> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
>> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
>> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
>> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
>> >>>> -
>> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
>> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
>> >>>> test child finished with -1
>> >>>> ---- end ----
>> >>>> Test LLVM searching and compiling subtest 0: FAILED!
>> >>>> -----------------------------------------------------
>> >>>>
>> >>>>
>> >>>> Basic BPF LLVM compiling test is failing.
>> >>>> It reports that bpf could not able to read the kernel version from uname.
>> >>>> Is it because of this that 'perf test LLVM' is failing ?
>> >>>>
>> >>>> My uname says:
>> >>>> 4.9-xxxx-amd-x86-64
>> >>>
>> >>>
>> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
>> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
>> >>> report
>> >>> kernel version number (from your uname string, sublevel can't be
>> >>> determined),
>> >>> please help us improve that function like what we have done for ubuntu
>> >>> (checking /proc/version_signature).
>> >>>
>> >>>
>> >>>
>> >>>> My kernel version is: 4.9.20 (from Makefile)
>> >>>>
>> >>>> So, I dont think there is any problem with kernel version.
>> >>>>
>> >>>> If any body have come across this problem please let me know how to
>> >>>> resolve this issue.
>> >>>>
>> >>>>
>> >>>>
>> >>>> Thank You!
>> >>>>
>> >>>> Regards,
>> >>>> Pintu
>> >>>
>> >>>
>> >>>
>
>
> --
> Masami Hiramatsu <mhiramat@kernel.org>

Re: [perf] perf probe fails sometimes on 4.9

[Masami Hiramatsu]

On Mon, 29 Jan 2018 22:00:52 +0530
Pintu Kumar <pintu.ping@gmail.com> wrote:

> Dear Masami,
> 
> Thank you so much for your reply.
> Please find some of my answers inline.
> 
> 
> On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> > On Mon, 29 Jan 2018 13:40:34 +0530
> > Pintu Kumar <pintu.ping@gmail.com> wrote:
> >
> >> Hi All,
> >>
> >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
> >> # perf probe --add schedule
> >> schedule is out of .text, skip it.
> >>   Error: Failed to add events.
> >>
> >> If any one have come across this problem please let me know the cause.
> >
> > Hi Pintu,
> >
> > Could you run it with --vv?
> >
> Ok, I will send verbose output by tomorrow.
> 
> >>
> >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
> >
> > Without it, you can not probe source-level probe nor trace local variable.
> >
> 
> Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
> However, I will try to manually include "-g" option during compilation.
> 
> >> However, I manually copied the vmlinux file to /boot/ directory, but
> >> still it does not work.
> >
> > That doesn't work.
> > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
> > Without that option, debuginfo is not generated with vmlinux.
> >
> >>
> >> I checked upstream patches until 4.15 but could not find any clue.
> >> Please let me know if there is any fixes available for this.
> >
> > Could you also ensure that you run perf by root user?
> >
> 
> Yes I am running with root user.
> 
> My concern is sometimes it works but sometimes it fails.

What I thought was that your kernel enables kptr_strict(but maybe not.)
Can you also try to find "schedule" and "_etext" functions in
/proc/kallsyms?

> I still needs to figure out, in which condition it works and which
> condition it fails.

Yeah, it is important.

> Usually I noticed that in fresh reboot case it works.

So, after a while, it doesn't work again? If so, it sounds like a daemon
process changes settings in background.

Thank you,

> 
> 
> > Thank you,
> >
> >
> >>
> >>
> >> Thank You!
> >> Regards,
> >> Pintu
> >>
> >>
> >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > ** Changed the subject now, since these issues are related to general
> >> > perf commands.
> >> >
> >> > Following are the issues:
> >> >
> >> > 1) perf probe --add schedule - FAILED
> >> > output:
> >> > schedule is out of .text, skip it.
> >> >   Error: Failed to add events.
> >> >
> >> > what is the issue here?
> >> > Sometimes it pass and sometimes it fails...
> >> > Similar is the case of 'perf inject' as well.
> >> >
> >> > 2) perf test - 1 FAILURE
> >> > 37.1: Test basic BPF filtering                               : FAILED!
> >> > 37.2: Test BPF prologue generation                           : Skip
> >> > 37.3: Test BPF relocation checker                            : Skip
> >> >
> >> > bpf: config program 'func=SyS_epoll_wait'
> >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> >> > bpf: config 'func=SyS_epoll_wait' is ok
> >> > Looking at the vmlinux_path (8 entries long)
> >> > Using /boot/vmlinux for symbols
> >> > Could not open debuginfo. Try to use symbols.
> >> > SyS_epoll_wait is out of .text, skip it.
> >> > bpf_probe: failed to convert perf probe eventsFailed to add events
> >> > selected by BPF
> >> > test child finished with -1
> >> > ---- end ----
> >> > Test BPF filter subtest 0: FAILED!
> >> >
> >> > Looks like both 1,2 are related.
> >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> >> > to /boot/ folder.
> >> >
> >> > -------
> >> > Some more info:
> >> >
> >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> >> > unset env: KBUILD_OPTS
> >> > include option is set to  -nostdinc -isystem
> >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> >> > set env: NR_CPUS=8
> >> > set env: LINUX_VERSION_CODE=0x40914
> >> > set env: CLANG_EXEC=/usr/bin/clang
> >> > set env: CLANG_OPTIONS=-xc
> >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> >> >
> >> >
> >> > If you have any clue about these failure please help me.
> >> >
> >> >
> >> > Thanks,
> >> > Pintu
> >> >
> >> >
> >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> >> >> Hi,
> >> >>
> >> >> Thanks for your help.
> >> >> Yes it was a sub version issue.
> >> >>
> >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
> >> >>
> >> >> make -j8 bindeb-pkg SUBLEVEL=20
> >> >>
> >> >> After passing the correct sublevel now the test is working.
> >> >>
> >> >> But still following are failing:
> >> >>
> >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
> >> >> 37.2: Test BPF prologue generation                           : FAILED!
> >> >>
> >> >>
> >> >> This is the error I get:
> >> >>
> >> >> 16: Try 'import perf' in python, checking link problems      :
> >> >> --- start ---
> >> >> test child forked, pid 7637
> >> >> Traceback (most recent call last):
> >> >>   File "<stdin>", line 1, in <module>
> >> >> ImportError: No module named perf
> >> >> test child finished with -1
> >> >> ---- end ----
> >> >> Try 'import perf' in python, checking link problems: FAILED!
> >> >>
> >> >>
> >> >> Looking at the vmlinux_path (8 entries long)
> >> >> symsrc__init: cannot get elf header.
> >> >> Failed to find the path for kernel: Invalid ELF file
> >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
> >> >> selected by BPF
> >> >> test child finished with -1
> >> >> ---- end ----
> >> >> Test BPF filter subtest 1: FAILED!
> >> >>
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Pintu
> >> >>
> >> >>
> >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
> >> >>>
> >> >>>
> >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
> >> >>>>
> >> >>>> Hi All,
> >> >>>>
> >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
> >> >>>> kernel version 4.9.20.
> >> >>>> I have installed several others packages including: clang, llvm
> >> >>>>
> >> >>>> But, when I run 'perf test' I get some FAILURE.
> >> >>>> Specially, 'perf test LLVM' is failing.
> >> >>>>
> >> >>>> Please check the below error logs:
> >> >>>>
> >> >>>> ----------------------------------------------------
> >> >>>> # perf test LLVM
> >> >>>> 35: Test LLVM searching and compiling                        :
> >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
> >> >>>> 35.2: Test kbuild searching                                  : Skip
> >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
> >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
> >> >>>> ----------------------------------------------------
> >> >>>>
> >> >>>> When I run with -v I get this:
> >> >>>>
> >> >>>> -----------------------------------------------------
> >> >>>> # perf test -v LLVM
> >> >>>> 35: Test LLVM searching and compiling                        :
> >> >>>> 35.1: Basic BPF llvm compiling test                          :
> >> >>>> --- start ---
> >> >>>> test child forked, pid 3304
> >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >> >>>> WARNING:        unable to get correct kernel building directory.
> >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
> >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
> >> >>>>          detection.
> >> >>>>
> >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >> >>>>
> >> >>>>
> >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
> >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
> >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
> >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
> >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
> >> >>>> -
> >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
> >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
> >> >>>> test child finished with -1
> >> >>>> ---- end ----
> >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
> >> >>>> -----------------------------------------------------
> >> >>>>
> >> >>>>
> >> >>>> Basic BPF LLVM compiling test is failing.
> >> >>>> It reports that bpf could not able to read the kernel version from uname.
> >> >>>> Is it because of this that 'perf test LLVM' is failing ?
> >> >>>>
> >> >>>> My uname says:
> >> >>>> 4.9-xxxx-amd-x86-64
> >> >>>
> >> >>>
> >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
> >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
> >> >>> report
> >> >>> kernel version number (from your uname string, sublevel can't be
> >> >>> determined),
> >> >>> please help us improve that function like what we have done for ubuntu
> >> >>> (checking /proc/version_signature).
> >> >>>
> >> >>>
> >> >>>
> >> >>>> My kernel version is: 4.9.20 (from Makefile)
> >> >>>>
> >> >>>> So, I dont think there is any problem with kernel version.
> >> >>>>
> >> >>>> If any body have come across this problem please let me know how to
> >> >>>> resolve this issue.
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> Thank You!
> >> >>>>
> >> >>>> Regards,
> >> >>>> Pintu
> >> >>>
> >> >>>
> >> >>>
> >
> >
> > --
> > Masami Hiramatsu <mhiramat@kernel.org>


-- 
Masami Hiramatsu <mhiramat@kernel.org>

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

[perf] perf probe fails sometimes on 4.9

[Masami Hiramatsu]

On Mon, 29 Jan 2018 22:00:52 +0530
Pintu Kumar <pintu.ping@gmail.com> wrote:

> Dear Masami,
> 
> Thank you so much for your reply.
> Please find some of my answers inline.
> 
> 
> On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> > On Mon, 29 Jan 2018 13:40:34 +0530
> > Pintu Kumar <pintu.ping@gmail.com> wrote:
> >
> >> Hi All,
> >>
> >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
> >> # perf probe --add schedule
> >> schedule is out of .text, skip it.
> >>   Error: Failed to add events.
> >>
> >> If any one have come across this problem please let me know the cause.
> >
> > Hi Pintu,
> >
> > Could you run it with --vv?
> >
> Ok, I will send verbose output by tomorrow.
> 
> >>
> >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
> >
> > Without it, you can not probe source-level probe nor trace local variable.
> >
> 
> Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
> However, I will try to manually include "-g" option during compilation.
> 
> >> However, I manually copied the vmlinux file to /boot/ directory, but
> >> still it does not work.
> >
> > That doesn't work.
> > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
> > Without that option, debuginfo is not generated with vmlinux.
> >
> >>
> >> I checked upstream patches until 4.15 but could not find any clue.
> >> Please let me know if there is any fixes available for this.
> >
> > Could you also ensure that you run perf by root user?
> >
> 
> Yes I am running with root user.
> 
> My concern is sometimes it works but sometimes it fails.

What I thought was that your kernel enables kptr_strict(but maybe not.)
Can you also try to find "schedule" and "_etext" functions in
/proc/kallsyms?

> I still needs to figure out, in which condition it works and which
> condition it fails.

Yeah, it is important.

> Usually I noticed that in fresh reboot case it works.

So, after a while, it doesn't work again? If so, it sounds like a daemon
process changes settings in background.

Thank you,

> 
> 
> > Thank you,
> >
> >
> >>
> >>
> >> Thank You!
> >> Regards,
> >> Pintu
> >>
> >>
> >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > ** Changed the subject now, since these issues are related to general
> >> > perf commands.
> >> >
> >> > Following are the issues:
> >> >
> >> > 1) perf probe --add schedule - FAILED
> >> > output:
> >> > schedule is out of .text, skip it.
> >> >   Error: Failed to add events.
> >> >
> >> > what is the issue here?
> >> > Sometimes it pass and sometimes it fails...
> >> > Similar is the case of 'perf inject' as well.
> >> >
> >> > 2) perf test - 1 FAILURE
> >> > 37.1: Test basic BPF filtering                               : FAILED!
> >> > 37.2: Test BPF prologue generation                           : Skip
> >> > 37.3: Test BPF relocation checker                            : Skip
> >> >
> >> > bpf: config program 'func=SyS_epoll_wait'
> >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> >> > bpf: config 'func=SyS_epoll_wait' is ok
> >> > Looking at the vmlinux_path (8 entries long)
> >> > Using /boot/vmlinux for symbols
> >> > Could not open debuginfo. Try to use symbols.
> >> > SyS_epoll_wait is out of .text, skip it.
> >> > bpf_probe: failed to convert perf probe eventsFailed to add events
> >> > selected by BPF
> >> > test child finished with -1
> >> > ---- end ----
> >> > Test BPF filter subtest 0: FAILED!
> >> >
> >> > Looks like both 1,2 are related.
> >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> >> > to /boot/ folder.
> >> >
> >> > -------
> >> > Some more info:
> >> >
> >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> >> > unset env: KBUILD_OPTS
> >> > include option is set to  -nostdinc -isystem
> >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> >> > set env: NR_CPUS=8
> >> > set env: LINUX_VERSION_CODE=0x40914
> >> > set env: CLANG_EXEC=/usr/bin/clang
> >> > set env: CLANG_OPTIONS=-xc
> >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> >> >
> >> >
> >> > If you have any clue about these failure please help me.
> >> >
> >> >
> >> > Thanks,
> >> > Pintu
> >> >
> >> >
> >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> >> >> Hi,
> >> >>
> >> >> Thanks for your help.
> >> >> Yes it was a sub version issue.
> >> >>
> >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
> >> >>
> >> >> make -j8 bindeb-pkg SUBLEVEL=20
> >> >>
> >> >> After passing the correct sublevel now the test is working.
> >> >>
> >> >> But still following are failing:
> >> >>
> >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
> >> >> 37.2: Test BPF prologue generation                           : FAILED!
> >> >>
> >> >>
> >> >> This is the error I get:
> >> >>
> >> >> 16: Try 'import perf' in python, checking link problems      :
> >> >> --- start ---
> >> >> test child forked, pid 7637
> >> >> Traceback (most recent call last):
> >> >>   File "<stdin>", line 1, in <module>
> >> >> ImportError: No module named perf
> >> >> test child finished with -1
> >> >> ---- end ----
> >> >> Try 'import perf' in python, checking link problems: FAILED!
> >> >>
> >> >>
> >> >> Looking at the vmlinux_path (8 entries long)
> >> >> symsrc__init: cannot get elf header.
> >> >> Failed to find the path for kernel: Invalid ELF file
> >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
> >> >> selected by BPF
> >> >> test child finished with -1
> >> >> ---- end ----
> >> >> Test BPF filter subtest 1: FAILED!
> >> >>
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Pintu
> >> >>
> >> >>
> >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
> >> >>>
> >> >>>
> >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
> >> >>>>
> >> >>>> Hi All,
> >> >>>>
> >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
> >> >>>> kernel version 4.9.20.
> >> >>>> I have installed several others packages including: clang, llvm
> >> >>>>
> >> >>>> But, when I run 'perf test' I get some FAILURE.
> >> >>>> Specially, 'perf test LLVM' is failing.
> >> >>>>
> >> >>>> Please check the below error logs:
> >> >>>>
> >> >>>> ----------------------------------------------------
> >> >>>> # perf test LLVM
> >> >>>> 35: Test LLVM searching and compiling                        :
> >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
> >> >>>> 35.2: Test kbuild searching                                  : Skip
> >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
> >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
> >> >>>> ----------------------------------------------------
> >> >>>>
> >> >>>> When I run with -v I get this:
> >> >>>>
> >> >>>> -----------------------------------------------------
> >> >>>> # perf test -v LLVM
> >> >>>> 35: Test LLVM searching and compiling                        :
> >> >>>> 35.1: Basic BPF llvm compiling test                          :
> >> >>>> --- start ---
> >> >>>> test child forked, pid 3304
> >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >> >>>> WARNING:        unable to get correct kernel building directory.
> >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
> >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
> >> >>>>          detection.
> >> >>>>
> >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >> >>>>
> >> >>>>
> >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
> >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
> >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
> >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
> >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
> >> >>>> -
> >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
> >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
> >> >>>> test child finished with -1
> >> >>>> ---- end ----
> >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
> >> >>>> -----------------------------------------------------
> >> >>>>
> >> >>>>
> >> >>>> Basic BPF LLVM compiling test is failing.
> >> >>>> It reports that bpf could not able to read the kernel version from uname.
> >> >>>> Is it because of this that 'perf test LLVM' is failing ?
> >> >>>>
> >> >>>> My uname says:
> >> >>>> 4.9-xxxx-amd-x86-64
> >> >>>
> >> >>>
> >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
> >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
> >> >>> report
> >> >>> kernel version number (from your uname string, sublevel can't be
> >> >>> determined),
> >> >>> please help us improve that function like what we have done for ubuntu
> >> >>> (checking /proc/version_signature).
> >> >>>
> >> >>>
> >> >>>
> >> >>>> My kernel version is: 4.9.20 (from Makefile)
> >> >>>>
> >> >>>> So, I dont think there is any problem with kernel version.
> >> >>>>
> >> >>>> If any body have come across this problem please let me know how to
> >> >>>> resolve this issue.
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> Thank You!
> >> >>>>
> >> >>>> Regards,
> >> >>>> Pintu
> >> >>>
> >> >>>
> >> >>>
> >
> >
> > --
> > Masami Hiramatsu <mhiramat@kernel.org>


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [perf] perf probe fails sometimes on 4.9

[Pintu Kumar]

On Tue, Jan 30, 2018 at 11:13 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>
> On Mon, 29 Jan 2018 22:00:52 +0530
> Pintu Kumar <pintu.ping@gmail.com> wrote:
>
> > Dear Masami,
> >
> > Thank you so much for your reply.
> > Please find some of my answers inline.
> >
> >
> > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> > > On Mon, 29 Jan 2018 13:40:34 +0530
> > > Pintu Kumar <pintu.ping@gmail.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
> > >> # perf probe --add schedule
> > >> schedule is out of .text, skip it.
> > >>   Error: Failed to add events.
> > >>
> > >> If any one have come across this problem please let me know the cause.
> > >
> > > Hi Pintu,
> > >
> > > Could you run it with --vv?
> > >
> > Ok, I will send verbose output by tomorrow.
> >
> > >>
> > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
> > >
> > > Without it, you can not probe source-level probe nor trace local variable.
> > >
> >
> > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
> > However, I will try to manually include "-g" option during compilation.
> >
> > >> However, I manually copied the vmlinux file to /boot/ directory, but
> > >> still it does not work.
> > >
> > > That doesn't work.
> > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
> > > Without that option, debuginfo is not generated with vmlinux.
> > >
> > >>
> > >> I checked upstream patches until 4.15 but could not find any clue.
> > >> Please let me know if there is any fixes available for this.
> > >
> > > Could you also ensure that you run perf by root user?
> > >
> >
> > Yes I am running with root user.
> >
> > My concern is sometimes it works but sometimes it fails.
>
> What I thought was that your kernel enables kptr_strict(but maybe not.)
> Can you also try to find "schedule" and "_etext" functions in
> /proc/kallsyms?
>

Yes these functions are present in kallsyms.

# cat /proc/kallsyms | grep -w schedule ; cat /proc/kallsyms | grep -w _etext
ffffffff829328e0 T schedule
ffffffff8293e751 T _etext


> > I still needs to figure out, in which condition it works and which
> > condition it fails.
>
> Yeah, it is important.
>
> > Usually I noticed that in fresh reboot case it works.
>
> So, after a while, it doesn't work again? If so, it sounds like a daemon
> process changes settings in background.
>

Fortunately or unfortunately, now it is passing every time.
Still not sure about the cause.
If it fails again, I will collect info with -v and report again.
Any specific info that I need to check when it fails ?

Thank You,
Pintu


> Thank you,
>
> >
> >
> > > Thank you,
> > >
> > >
> > >>
> > >>
> > >> Thank You!
> > >> Regards,
> > >> Pintu
> > >>
> > >>
> > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > >> > Hi,
> > >> >
> > >> > ** Changed the subject now, since these issues are related to general
> > >> > perf commands.
> > >> >
> > >> > Following are the issues:
> > >> >
> > >> > 1) perf probe --add schedule - FAILED
> > >> > output:
> > >> > schedule is out of .text, skip it.
> > >> >   Error: Failed to add events.
> > >> >
> > >> > what is the issue here?
> > >> > Sometimes it pass and sometimes it fails...
> > >> > Similar is the case of 'perf inject' as well.
> > >> >
> > >> > 2) perf test - 1 FAILURE
> > >> > 37.1: Test basic BPF filtering                               : FAILED!
> > >> > 37.2: Test BPF prologue generation                           : Skip
> > >> > 37.3: Test BPF relocation checker                            : Skip
> > >> >
> > >> > bpf: config program 'func=SyS_epoll_wait'
> > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> > >> > bpf: config 'func=SyS_epoll_wait' is ok
> > >> > Looking at the vmlinux_path (8 entries long)
> > >> > Using /boot/vmlinux for symbols
> > >> > Could not open debuginfo. Try to use symbols.
> > >> > SyS_epoll_wait is out of .text, skip it.
> > >> > bpf_probe: failed to convert perf probe eventsFailed to add events
> > >> > selected by BPF
> > >> > test child finished with -1
> > >> > ---- end ----
> > >> > Test BPF filter subtest 0: FAILED!
> > >> >
> > >> > Looks like both 1,2 are related.
> > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> > >> > to /boot/ folder.
> > >> >
> > >> > -------
> > >> > Some more info:
> > >> >
> > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > >> > unset env: KBUILD_OPTS
> > >> > include option is set to  -nostdinc -isystem
> > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > >> > set env: NR_CPUS=8
> > >> > set env: LINUX_VERSION_CODE=0x40914
> > >> > set env: CLANG_EXEC=/usr/bin/clang
> > >> > set env: CLANG_OPTIONS=-xc
> > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > >> >
> > >> >
> > >> > If you have any clue about these failure please help me.
> > >> >
> > >> >
> > >> > Thanks,
> > >> > Pintu
> > >> >
> > >> >
> > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > >> >> Hi,
> > >> >>
> > >> >> Thanks for your help.
> > >> >> Yes it was a sub version issue.
> > >> >>
> > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
> > >> >>
> > >> >> make -j8 bindeb-pkg SUBLEVEL=20
> > >> >>
> > >> >> After passing the correct sublevel now the test is working.
> > >> >>
> > >> >> But still following are failing:
> > >> >>
> > >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
> > >> >> 37.2: Test BPF prologue generation                           : FAILED!
> > >> >>
> > >> >>
> > >> >> This is the error I get:
> > >> >>
> > >> >> 16: Try 'import perf' in python, checking link problems      :
> > >> >> --- start ---
> > >> >> test child forked, pid 7637
> > >> >> Traceback (most recent call last):
> > >> >>   File "<stdin>", line 1, in <module>
> > >> >> ImportError: No module named perf
> > >> >> test child finished with -1
> > >> >> ---- end ----
> > >> >> Try 'import perf' in python, checking link problems: FAILED!
> > >> >>
> > >> >>
> > >> >> Looking at the vmlinux_path (8 entries long)
> > >> >> symsrc__init: cannot get elf header.
> > >> >> Failed to find the path for kernel: Invalid ELF file
> > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
> > >> >> selected by BPF
> > >> >> test child finished with -1
> > >> >> ---- end ----
> > >> >> Test BPF filter subtest 1: FAILED!
> > >> >>
> > >> >>
> > >> >>
> > >> >> Thanks,
> > >> >> Pintu
> > >> >>
> > >> >>
> > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
> > >> >>>
> > >> >>>
> > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
> > >> >>>>
> > >> >>>> Hi All,
> > >> >>>>
> > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
> > >> >>>> kernel version 4.9.20.
> > >> >>>> I have installed several others packages including: clang, llvm
> > >> >>>>
> > >> >>>> But, when I run 'perf test' I get some FAILURE.
> > >> >>>> Specially, 'perf test LLVM' is failing.
> > >> >>>>
> > >> >>>> Please check the below error logs:
> > >> >>>>
> > >> >>>> ----------------------------------------------------
> > >> >>>> # perf test LLVM
> > >> >>>> 35: Test LLVM searching and compiling                        :
> > >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
> > >> >>>> 35.2: Test kbuild searching                                  : Skip
> > >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
> > >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
> > >> >>>> ----------------------------------------------------
> > >> >>>>
> > >> >>>> When I run with -v I get this:
> > >> >>>>
> > >> >>>> -----------------------------------------------------
> > >> >>>> # perf test -v LLVM
> > >> >>>> 35: Test LLVM searching and compiling                        :
> > >> >>>> 35.1: Basic BPF llvm compiling test                          :
> > >> >>>> --- start ---
> > >> >>>> test child forked, pid 3304
> > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> > >> >>>> WARNING:        unable to get correct kernel building directory.
> > >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
> > >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
> > >> >>>>          detection.
> > >> >>>>
> > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> > >> >>>>
> > >> >>>>
> > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
> > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
> > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
> > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
> > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
> > >> >>>> -
> > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
> > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
> > >> >>>> test child finished with -1
> > >> >>>> ---- end ----
> > >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
> > >> >>>> -----------------------------------------------------
> > >> >>>>
> > >> >>>>
> > >> >>>> Basic BPF LLVM compiling test is failing.
> > >> >>>> It reports that bpf could not able to read the kernel version from uname.
> > >> >>>> Is it because of this that 'perf test LLVM' is failing ?
> > >> >>>>
> > >> >>>> My uname says:
> > >> >>>> 4.9-xxxx-amd-x86-64
> > >> >>>
> > >> >>>
> > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
> > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
> > >> >>> report
> > >> >>> kernel version number (from your uname string, sublevel can't be
> > >> >>> determined),
> > >> >>> please help us improve that function like what we have done for ubuntu
> > >> >>> (checking /proc/version_signature).
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>> My kernel version is: 4.9.20 (from Makefile)
> > >> >>>>
> > >> >>>> So, I dont think there is any problem with kernel version.
> > >> >>>>
> > >> >>>> If any body have come across this problem please let me know how to
> > >> >>>> resolve this issue.
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>> Thank You!
> > >> >>>>
> > >> >>>> Regards,
> > >> >>>> Pintu
> > >> >>>
> > >> >>>
> > >> >>>
> > >
> > >
> > > --
> > > Masami Hiramatsu <mhiramat@kernel.org>
>
>
> --
> Masami Hiramatsu <mhiramat@kernel.org>

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

[perf] perf probe fails sometimes on 4.9

[Pintu Kumar]

On Tue, Jan 30, 2018 at 11:13 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>
> On Mon, 29 Jan 2018 22:00:52 +0530
> Pintu Kumar <pintu.ping@gmail.com> wrote:
>
> > Dear Masami,
> >
> > Thank you so much for your reply.
> > Please find some of my answers inline.
> >
> >
> > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> > > On Mon, 29 Jan 2018 13:40:34 +0530
> > > Pintu Kumar <pintu.ping@gmail.com> wrote:
> > >
> > >> Hi All,
> > >>
> > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
> > >> # perf probe --add schedule
> > >> schedule is out of .text, skip it.
> > >>   Error: Failed to add events.
> > >>
> > >> If any one have come across this problem please let me know the cause.
> > >
> > > Hi Pintu,
> > >
> > > Could you run it with --vv?
> > >
> > Ok, I will send verbose output by tomorrow.
> >
> > >>
> > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
> > >
> > > Without it, you can not probe source-level probe nor trace local variable.
> > >
> >
> > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
> > However, I will try to manually include "-g" option during compilation.
> >
> > >> However, I manually copied the vmlinux file to /boot/ directory, but
> > >> still it does not work.
> > >
> > > That doesn't work.
> > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
> > > Without that option, debuginfo is not generated with vmlinux.
> > >
> > >>
> > >> I checked upstream patches until 4.15 but could not find any clue.
> > >> Please let me know if there is any fixes available for this.
> > >
> > > Could you also ensure that you run perf by root user?
> > >
> >
> > Yes I am running with root user.
> >
> > My concern is sometimes it works but sometimes it fails.
>
> What I thought was that your kernel enables kptr_strict(but maybe not.)
> Can you also try to find "schedule" and "_etext" functions in
> /proc/kallsyms?
>

Yes these functions are present in kallsyms.

# cat /proc/kallsyms | grep -w schedule ; cat /proc/kallsyms | grep -w _etext
ffffffff829328e0 T schedule
ffffffff8293e751 T _etext


> > I still needs to figure out, in which condition it works and which
> > condition it fails.
>
> Yeah, it is important.
>
> > Usually I noticed that in fresh reboot case it works.
>
> So, after a while, it doesn't work again? If so, it sounds like a daemon
> process changes settings in background.
>

Fortunately or unfortunately, now it is passing every time.
Still not sure about the cause.
If it fails again, I will collect info with -v and report again.
Any specific info that I need to check when it fails ?

Thank You,
Pintu


> Thank you,
>
> >
> >
> > > Thank you,
> > >
> > >
> > >>
> > >>
> > >> Thank You!
> > >> Regards,
> > >> Pintu
> > >>
> > >>
> > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > >> > Hi,
> > >> >
> > >> > ** Changed the subject now, since these issues are related to general
> > >> > perf commands.
> > >> >
> > >> > Following are the issues:
> > >> >
> > >> > 1) perf probe --add schedule - FAILED
> > >> > output:
> > >> > schedule is out of .text, skip it.
> > >> >   Error: Failed to add events.
> > >> >
> > >> > what is the issue here?
> > >> > Sometimes it pass and sometimes it fails...
> > >> > Similar is the case of 'perf inject' as well.
> > >> >
> > >> > 2) perf test - 1 FAILURE
> > >> > 37.1: Test basic BPF filtering                               : FAILED!
> > >> > 37.2: Test BPF prologue generation                           : Skip
> > >> > 37.3: Test BPF relocation checker                            : Skip
> > >> >
> > >> > bpf: config program 'func=SyS_epoll_wait'
> > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> > >> > bpf: config 'func=SyS_epoll_wait' is ok
> > >> > Looking at the vmlinux_path (8 entries long)
> > >> > Using /boot/vmlinux for symbols
> > >> > Could not open debuginfo. Try to use symbols.
> > >> > SyS_epoll_wait is out of .text, skip it.
> > >> > bpf_probe: failed to convert perf probe eventsFailed to add events
> > >> > selected by BPF
> > >> > test child finished with -1
> > >> > ---- end ----
> > >> > Test BPF filter subtest 0: FAILED!
> > >> >
> > >> > Looks like both 1,2 are related.
> > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> > >> > to /boot/ folder.
> > >> >
> > >> > -------
> > >> > Some more info:
> > >> >
> > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > >> > unset env: KBUILD_OPTS
> > >> > include option is set to  -nostdinc -isystem
> > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > >> > set env: NR_CPUS=8
> > >> > set env: LINUX_VERSION_CODE=0x40914
> > >> > set env: CLANG_EXEC=/usr/bin/clang
> > >> > set env: CLANG_OPTIONS=-xc
> > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > >> >
> > >> >
> > >> > If you have any clue about these failure please help me.
> > >> >
> > >> >
> > >> > Thanks,
> > >> > Pintu
> > >> >
> > >> >
> > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > >> >> Hi,
> > >> >>
> > >> >> Thanks for your help.
> > >> >> Yes it was a sub version issue.
> > >> >>
> > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
> > >> >>
> > >> >> make -j8 bindeb-pkg SUBLEVEL=20
> > >> >>
> > >> >> After passing the correct sublevel now the test is working.
> > >> >>
> > >> >> But still following are failing:
> > >> >>
> > >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
> > >> >> 37.2: Test BPF prologue generation                           : FAILED!
> > >> >>
> > >> >>
> > >> >> This is the error I get:
> > >> >>
> > >> >> 16: Try 'import perf' in python, checking link problems      :
> > >> >> --- start ---
> > >> >> test child forked, pid 7637
> > >> >> Traceback (most recent call last):
> > >> >>   File "<stdin>", line 1, in <module>
> > >> >> ImportError: No module named perf
> > >> >> test child finished with -1
> > >> >> ---- end ----
> > >> >> Try 'import perf' in python, checking link problems: FAILED!
> > >> >>
> > >> >>
> > >> >> Looking at the vmlinux_path (8 entries long)
> > >> >> symsrc__init: cannot get elf header.
> > >> >> Failed to find the path for kernel: Invalid ELF file
> > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
> > >> >> selected by BPF
> > >> >> test child finished with -1
> > >> >> ---- end ----
> > >> >> Test BPF filter subtest 1: FAILED!
> > >> >>
> > >> >>
> > >> >>
> > >> >> Thanks,
> > >> >> Pintu
> > >> >>
> > >> >>
> > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
> > >> >>>
> > >> >>>
> > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
> > >> >>>>
> > >> >>>> Hi All,
> > >> >>>>
> > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
> > >> >>>> kernel version 4.9.20.
> > >> >>>> I have installed several others packages including: clang, llvm
> > >> >>>>
> > >> >>>> But, when I run 'perf test' I get some FAILURE.
> > >> >>>> Specially, 'perf test LLVM' is failing.
> > >> >>>>
> > >> >>>> Please check the below error logs:
> > >> >>>>
> > >> >>>> ----------------------------------------------------
> > >> >>>> # perf test LLVM
> > >> >>>> 35: Test LLVM searching and compiling                        :
> > >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
> > >> >>>> 35.2: Test kbuild searching                                  : Skip
> > >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
> > >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
> > >> >>>> ----------------------------------------------------
> > >> >>>>
> > >> >>>> When I run with -v I get this:
> > >> >>>>
> > >> >>>> -----------------------------------------------------
> > >> >>>> # perf test -v LLVM
> > >> >>>> 35: Test LLVM searching and compiling                        :
> > >> >>>> 35.1: Basic BPF llvm compiling test                          :
> > >> >>>> --- start ---
> > >> >>>> test child forked, pid 3304
> > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> > >> >>>> WARNING:        unable to get correct kernel building directory.
> > >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
> > >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
> > >> >>>>          detection.
> > >> >>>>
> > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> > >> >>>>
> > >> >>>>
> > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
> > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
> > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
> > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
> > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
> > >> >>>> -
> > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
> > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
> > >> >>>> test child finished with -1
> > >> >>>> ---- end ----
> > >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
> > >> >>>> -----------------------------------------------------
> > >> >>>>
> > >> >>>>
> > >> >>>> Basic BPF LLVM compiling test is failing.
> > >> >>>> It reports that bpf could not able to read the kernel version from uname.
> > >> >>>> Is it because of this that 'perf test LLVM' is failing ?
> > >> >>>>
> > >> >>>> My uname says:
> > >> >>>> 4.9-xxxx-amd-x86-64
> > >> >>>
> > >> >>>
> > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
> > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
> > >> >>> report
> > >> >>> kernel version number (from your uname string, sublevel can't be
> > >> >>> determined),
> > >> >>> please help us improve that function like what we have done for ubuntu
> > >> >>> (checking /proc/version_signature).
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>> My kernel version is: 4.9.20 (from Makefile)
> > >> >>>>
> > >> >>>> So, I dont think there is any problem with kernel version.
> > >> >>>>
> > >> >>>> If any body have come across this problem please let me know how to
> > >> >>>> resolve this issue.
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>> Thank You!
> > >> >>>>
> > >> >>>> Regards,
> > >> >>>> Pintu
> > >> >>>
> > >> >>>
> > >> >>>
> > >
> > >
> > > --
> > > Masami Hiramatsu <mhiramat@kernel.org>
>
>
> --
> Masami Hiramatsu <mhiramat@kernel.org>

Re: [perf] perf probe fails sometimes on 4.9

[Masami Hiramatsu]

On Tue, 30 Jan 2018 19:20:36 +0530
Pintu Kumar <pintu.ping@gmail.com> wrote:

> On Tue, Jan 30, 2018 at 11:13 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> >
> > On Mon, 29 Jan 2018 22:00:52 +0530
> > Pintu Kumar <pintu.ping@gmail.com> wrote:
> >
> > > Dear Masami,
> > >
> > > Thank you so much for your reply.
> > > Please find some of my answers inline.
> > >
> > >
> > > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> > > > On Mon, 29 Jan 2018 13:40:34 +0530
> > > > Pintu Kumar <pintu.ping@gmail.com> wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
> > > >> # perf probe --add schedule
> > > >> schedule is out of .text, skip it.
> > > >>   Error: Failed to add events.
> > > >>
> > > >> If any one have come across this problem please let me know the cause.
> > > >
> > > > Hi Pintu,
> > > >
> > > > Could you run it with --vv?
> > > >
> > > Ok, I will send verbose output by tomorrow.
> > >
> > > >>
> > > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
> > > >
> > > > Without it, you can not probe source-level probe nor trace local variable.
> > > >
> > >
> > > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
> > > However, I will try to manually include "-g" option during compilation.
> > >
> > > >> However, I manually copied the vmlinux file to /boot/ directory, but
> > > >> still it does not work.
> > > >
> > > > That doesn't work.
> > > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
> > > > Without that option, debuginfo is not generated with vmlinux.
> > > >
> > > >>
> > > >> I checked upstream patches until 4.15 but could not find any clue.
> > > >> Please let me know if there is any fixes available for this.
> > > >
> > > > Could you also ensure that you run perf by root user?
> > > >
> > >
> > > Yes I am running with root user.
> > >
> > > My concern is sometimes it works but sometimes it fails.
> >
> > What I thought was that your kernel enables kptr_strict(but maybe not.)
> > Can you also try to find "schedule" and "_etext" functions in
> > /proc/kallsyms?
> >
> 
> Yes these functions are present in kallsyms.
> 
> # cat /proc/kallsyms | grep -w schedule ; cat /proc/kallsyms | grep -w _etext
> ffffffff829328e0 T schedule
> ffffffff8293e751 T _etext
> 
> 
> > > I still needs to figure out, in which condition it works and which
> > > condition it fails.
> >
> > Yeah, it is important.
> >
> > > Usually I noticed that in fresh reboot case it works.
> >
> > So, after a while, it doesn't work again? If so, it sounds like a daemon
> > process changes settings in background.
> >
> 
> Fortunately or unfortunately, now it is passing every time.
> Still not sure about the cause.
> If it fails again, I will collect info with -v and report again.
> Any specific info that I need to check when it fails ?

OK, if you hit it again, please run perf-probe with -vvv option
(-v is enough usually, if it is unknown bug, -vvv will be better)

Thank you,


> 
> Thank You,
> Pintu
> 
> 
> > Thank you,
> >
> > >
> > >
> > > > Thank you,
> > > >
> > > >
> > > >>
> > > >>
> > > >> Thank You!
> > > >> Regards,
> > > >> Pintu
> > > >>
> > > >>
> > > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > > >> > Hi,
> > > >> >
> > > >> > ** Changed the subject now, since these issues are related to general
> > > >> > perf commands.
> > > >> >
> > > >> > Following are the issues:
> > > >> >
> > > >> > 1) perf probe --add schedule - FAILED
> > > >> > output:
> > > >> > schedule is out of .text, skip it.
> > > >> >   Error: Failed to add events.
> > > >> >
> > > >> > what is the issue here?
> > > >> > Sometimes it pass and sometimes it fails...
> > > >> > Similar is the case of 'perf inject' as well.
> > > >> >
> > > >> > 2) perf test - 1 FAILURE
> > > >> > 37.1: Test basic BPF filtering                               : FAILED!
> > > >> > 37.2: Test BPF prologue generation                           : Skip
> > > >> > 37.3: Test BPF relocation checker                            : Skip
> > > >> >
> > > >> > bpf: config program 'func=SyS_epoll_wait'
> > > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> > > >> > bpf: config 'func=SyS_epoll_wait' is ok
> > > >> > Looking at the vmlinux_path (8 entries long)
> > > >> > Using /boot/vmlinux for symbols
> > > >> > Could not open debuginfo. Try to use symbols.
> > > >> > SyS_epoll_wait is out of .text, skip it.
> > > >> > bpf_probe: failed to convert perf probe eventsFailed to add events
> > > >> > selected by BPF
> > > >> > test child finished with -1
> > > >> > ---- end ----
> > > >> > Test BPF filter subtest 0: FAILED!
> > > >> >
> > > >> > Looks like both 1,2 are related.
> > > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> > > >> > to /boot/ folder.
> > > >> >
> > > >> > -------
> > > >> > Some more info:
> > > >> >
> > > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> > > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > > >> > unset env: KBUILD_OPTS
> > > >> > include option is set to  -nostdinc -isystem
> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > > >> > set env: NR_CPUS=8
> > > >> > set env: LINUX_VERSION_CODE=0x40914
> > > >> > set env: CLANG_EXEC=/usr/bin/clang
> > > >> > set env: CLANG_OPTIONS=-xc
> > > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > > >> >
> > > >> >
> > > >> > If you have any clue about these failure please help me.
> > > >> >
> > > >> >
> > > >> > Thanks,
> > > >> > Pintu
> > > >> >
> > > >> >
> > > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > > >> >> Hi,
> > > >> >>
> > > >> >> Thanks for your help.
> > > >> >> Yes it was a sub version issue.
> > > >> >>
> > > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
> > > >> >>
> > > >> >> make -j8 bindeb-pkg SUBLEVEL=20
> > > >> >>
> > > >> >> After passing the correct sublevel now the test is working.
> > > >> >>
> > > >> >> But still following are failing:
> > > >> >>
> > > >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
> > > >> >> 37.2: Test BPF prologue generation                           : FAILED!
> > > >> >>
> > > >> >>
> > > >> >> This is the error I get:
> > > >> >>
> > > >> >> 16: Try 'import perf' in python, checking link problems      :
> > > >> >> --- start ---
> > > >> >> test child forked, pid 7637
> > > >> >> Traceback (most recent call last):
> > > >> >>   File "<stdin>", line 1, in <module>
> > > >> >> ImportError: No module named perf
> > > >> >> test child finished with -1
> > > >> >> ---- end ----
> > > >> >> Try 'import perf' in python, checking link problems: FAILED!
> > > >> >>
> > > >> >>
> > > >> >> Looking at the vmlinux_path (8 entries long)
> > > >> >> symsrc__init: cannot get elf header.
> > > >> >> Failed to find the path for kernel: Invalid ELF file
> > > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
> > > >> >> selected by BPF
> > > >> >> test child finished with -1
> > > >> >> ---- end ----
> > > >> >> Test BPF filter subtest 1: FAILED!
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> Thanks,
> > > >> >> Pintu
> > > >> >>
> > > >> >>
> > > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
> > > >> >>>
> > > >> >>>
> > > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
> > > >> >>>>
> > > >> >>>> Hi All,
> > > >> >>>>
> > > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
> > > >> >>>> kernel version 4.9.20.
> > > >> >>>> I have installed several others packages including: clang, llvm
> > > >> >>>>
> > > >> >>>> But, when I run 'perf test' I get some FAILURE.
> > > >> >>>> Specially, 'perf test LLVM' is failing.
> > > >> >>>>
> > > >> >>>> Please check the below error logs:
> > > >> >>>>
> > > >> >>>> ----------------------------------------------------
> > > >> >>>> # perf test LLVM
> > > >> >>>> 35: Test LLVM searching and compiling                        :
> > > >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
> > > >> >>>> 35.2: Test kbuild searching                                  : Skip
> > > >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
> > > >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
> > > >> >>>> ----------------------------------------------------
> > > >> >>>>
> > > >> >>>> When I run with -v I get this:
> > > >> >>>>
> > > >> >>>> -----------------------------------------------------
> > > >> >>>> # perf test -v LLVM
> > > >> >>>> 35: Test LLVM searching and compiling                        :
> > > >> >>>> 35.1: Basic BPF llvm compiling test                          :
> > > >> >>>> --- start ---
> > > >> >>>> test child forked, pid 3304
> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> > > >> >>>> WARNING:        unable to get correct kernel building directory.
> > > >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
> > > >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
> > > >> >>>>          detection.
> > > >> >>>>
> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
> > > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
> > > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
> > > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
> > > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
> > > >> >>>> -
> > > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
> > > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
> > > >> >>>> test child finished with -1
> > > >> >>>> ---- end ----
> > > >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
> > > >> >>>> -----------------------------------------------------
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> Basic BPF LLVM compiling test is failing.
> > > >> >>>> It reports that bpf could not able to read the kernel version from uname.
> > > >> >>>> Is it because of this that 'perf test LLVM' is failing ?
> > > >> >>>>
> > > >> >>>> My uname says:
> > > >> >>>> 4.9-xxxx-amd-x86-64
> > > >> >>>
> > > >> >>>
> > > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
> > > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
> > > >> >>> report
> > > >> >>> kernel version number (from your uname string, sublevel can't be
> > > >> >>> determined),
> > > >> >>> please help us improve that function like what we have done for ubuntu
> > > >> >>> (checking /proc/version_signature).
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>>> My kernel version is: 4.9.20 (from Makefile)
> > > >> >>>>
> > > >> >>>> So, I dont think there is any problem with kernel version.
> > > >> >>>>
> > > >> >>>> If any body have come across this problem please let me know how to
> > > >> >>>> resolve this issue.
> > > >> >>>>
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> Thank You!
> > > >> >>>>
> > > >> >>>> Regards,
> > > >> >>>> Pintu
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >
> > > >
> > > > --
> > > > Masami Hiramatsu <mhiramat@kernel.org>
> >
> >
> > --
> > Masami Hiramatsu <mhiramat@kernel.org>


-- 
Masami Hiramatsu <mhiramat@kernel.org>

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

[perf] perf probe fails sometimes on 4.9

[Masami Hiramatsu]

On Tue, 30 Jan 2018 19:20:36 +0530
Pintu Kumar <pintu.ping@gmail.com> wrote:

> On Tue, Jan 30, 2018 at 11:13 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> >
> > On Mon, 29 Jan 2018 22:00:52 +0530
> > Pintu Kumar <pintu.ping@gmail.com> wrote:
> >
> > > Dear Masami,
> > >
> > > Thank you so much for your reply.
> > > Please find some of my answers inline.
> > >
> > >
> > > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> > > > On Mon, 29 Jan 2018 13:40:34 +0530
> > > > Pintu Kumar <pintu.ping@gmail.com> wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
> > > >> # perf probe --add schedule
> > > >> schedule is out of .text, skip it.
> > > >>   Error: Failed to add events.
> > > >>
> > > >> If any one have come across this problem please let me know the cause.
> > > >
> > > > Hi Pintu,
> > > >
> > > > Could you run it with --vv?
> > > >
> > > Ok, I will send verbose output by tomorrow.
> > >
> > > >>
> > > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
> > > >
> > > > Without it, you can not probe source-level probe nor trace local variable.
> > > >
> > >
> > > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
> > > However, I will try to manually include "-g" option during compilation.
> > >
> > > >> However, I manually copied the vmlinux file to /boot/ directory, but
> > > >> still it does not work.
> > > >
> > > > That doesn't work.
> > > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
> > > > Without that option, debuginfo is not generated with vmlinux.
> > > >
> > > >>
> > > >> I checked upstream patches until 4.15 but could not find any clue.
> > > >> Please let me know if there is any fixes available for this.
> > > >
> > > > Could you also ensure that you run perf by root user?
> > > >
> > >
> > > Yes I am running with root user.
> > >
> > > My concern is sometimes it works but sometimes it fails.
> >
> > What I thought was that your kernel enables kptr_strict(but maybe not.)
> > Can you also try to find "schedule" and "_etext" functions in
> > /proc/kallsyms?
> >
> 
> Yes these functions are present in kallsyms.
> 
> # cat /proc/kallsyms | grep -w schedule ; cat /proc/kallsyms | grep -w _etext
> ffffffff829328e0 T schedule
> ffffffff8293e751 T _etext
> 
> 
> > > I still needs to figure out, in which condition it works and which
> > > condition it fails.
> >
> > Yeah, it is important.
> >
> > > Usually I noticed that in fresh reboot case it works.
> >
> > So, after a while, it doesn't work again? If so, it sounds like a daemon
> > process changes settings in background.
> >
> 
> Fortunately or unfortunately, now it is passing every time.
> Still not sure about the cause.
> If it fails again, I will collect info with -v and report again.
> Any specific info that I need to check when it fails ?

OK, if you hit it again, please run perf-probe with -vvv option
(-v is enough usually, if it is unknown bug, -vvv will be better)

Thank you,


> 
> Thank You,
> Pintu
> 
> 
> > Thank you,
> >
> > >
> > >
> > > > Thank you,
> > > >
> > > >
> > > >>
> > > >>
> > > >> Thank You!
> > > >> Regards,
> > > >> Pintu
> > > >>
> > > >>
> > > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > > >> > Hi,
> > > >> >
> > > >> > ** Changed the subject now, since these issues are related to general
> > > >> > perf commands.
> > > >> >
> > > >> > Following are the issues:
> > > >> >
> > > >> > 1) perf probe --add schedule - FAILED
> > > >> > output:
> > > >> > schedule is out of .text, skip it.
> > > >> >   Error: Failed to add events.
> > > >> >
> > > >> > what is the issue here?
> > > >> > Sometimes it pass and sometimes it fails...
> > > >> > Similar is the case of 'perf inject' as well.
> > > >> >
> > > >> > 2) perf test - 1 FAILURE
> > > >> > 37.1: Test basic BPF filtering                               : FAILED!
> > > >> > 37.2: Test BPF prologue generation                           : Skip
> > > >> > 37.3: Test BPF relocation checker                            : Skip
> > > >> >
> > > >> > bpf: config program 'func=SyS_epoll_wait'
> > > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> > > >> > bpf: config 'func=SyS_epoll_wait' is ok
> > > >> > Looking at the vmlinux_path (8 entries long)
> > > >> > Using /boot/vmlinux for symbols
> > > >> > Could not open debuginfo. Try to use symbols.
> > > >> > SyS_epoll_wait is out of .text, skip it.
> > > >> > bpf_probe: failed to convert perf probe eventsFailed to add events
> > > >> > selected by BPF
> > > >> > test child finished with -1
> > > >> > ---- end ----
> > > >> > Test BPF filter subtest 0: FAILED!
> > > >> >
> > > >> > Looks like both 1,2 are related.
> > > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> > > >> > to /boot/ folder.
> > > >> >
> > > >> > -------
> > > >> > Some more info:
> > > >> >
> > > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> > > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > > >> > unset env: KBUILD_OPTS
> > > >> > include option is set to  -nostdinc -isystem
> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > > >> > set env: NR_CPUS=8
> > > >> > set env: LINUX_VERSION_CODE=0x40914
> > > >> > set env: CLANG_EXEC=/usr/bin/clang
> > > >> > set env: CLANG_OPTIONS=-xc
> > > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> > > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> > > >> >
> > > >> >
> > > >> > If you have any clue about these failure please help me.
> > > >> >
> > > >> >
> > > >> > Thanks,
> > > >> > Pintu
> > > >> >
> > > >> >
> > > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> > > >> >> Hi,
> > > >> >>
> > > >> >> Thanks for your help.
> > > >> >> Yes it was a sub version issue.
> > > >> >>
> > > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
> > > >> >>
> > > >> >> make -j8 bindeb-pkg SUBLEVEL=20
> > > >> >>
> > > >> >> After passing the correct sublevel now the test is working.
> > > >> >>
> > > >> >> But still following are failing:
> > > >> >>
> > > >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
> > > >> >> 37.2: Test BPF prologue generation                           : FAILED!
> > > >> >>
> > > >> >>
> > > >> >> This is the error I get:
> > > >> >>
> > > >> >> 16: Try 'import perf' in python, checking link problems      :
> > > >> >> --- start ---
> > > >> >> test child forked, pid 7637
> > > >> >> Traceback (most recent call last):
> > > >> >>   File "<stdin>", line 1, in <module>
> > > >> >> ImportError: No module named perf
> > > >> >> test child finished with -1
> > > >> >> ---- end ----
> > > >> >> Try 'import perf' in python, checking link problems: FAILED!
> > > >> >>
> > > >> >>
> > > >> >> Looking at the vmlinux_path (8 entries long)
> > > >> >> symsrc__init: cannot get elf header.
> > > >> >> Failed to find the path for kernel: Invalid ELF file
> > > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
> > > >> >> selected by BPF
> > > >> >> test child finished with -1
> > > >> >> ---- end ----
> > > >> >> Test BPF filter subtest 1: FAILED!
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> Thanks,
> > > >> >> Pintu
> > > >> >>
> > > >> >>
> > > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
> > > >> >>>
> > > >> >>>
> > > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
> > > >> >>>>
> > > >> >>>> Hi All,
> > > >> >>>>
> > > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
> > > >> >>>> kernel version 4.9.20.
> > > >> >>>> I have installed several others packages including: clang, llvm
> > > >> >>>>
> > > >> >>>> But, when I run 'perf test' I get some FAILURE.
> > > >> >>>> Specially, 'perf test LLVM' is failing.
> > > >> >>>>
> > > >> >>>> Please check the below error logs:
> > > >> >>>>
> > > >> >>>> ----------------------------------------------------
> > > >> >>>> # perf test LLVM
> > > >> >>>> 35: Test LLVM searching and compiling                        :
> > > >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
> > > >> >>>> 35.2: Test kbuild searching                                  : Skip
> > > >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
> > > >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
> > > >> >>>> ----------------------------------------------------
> > > >> >>>>
> > > >> >>>> When I run with -v I get this:
> > > >> >>>>
> > > >> >>>> -----------------------------------------------------
> > > >> >>>> # perf test -v LLVM
> > > >> >>>> 35: Test LLVM searching and compiling                        :
> > > >> >>>> 35.1: Basic BPF llvm compiling test                          :
> > > >> >>>> --- start ---
> > > >> >>>> test child forked, pid 3304
> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> > > >> >>>> WARNING:        unable to get correct kernel building directory.
> > > >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
> > > >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
> > > >> >>>>          detection.
> > > >> >>>>
> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
> > > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
> > > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
> > > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
> > > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
> > > >> >>>> -
> > > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
> > > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
> > > >> >>>> test child finished with -1
> > > >> >>>> ---- end ----
> > > >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
> > > >> >>>> -----------------------------------------------------
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> Basic BPF LLVM compiling test is failing.
> > > >> >>>> It reports that bpf could not able to read the kernel version from uname.
> > > >> >>>> Is it because of this that 'perf test LLVM' is failing ?
> > > >> >>>>
> > > >> >>>> My uname says:
> > > >> >>>> 4.9-xxxx-amd-x86-64
> > > >> >>>
> > > >> >>>
> > > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
> > > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
> > > >> >>> report
> > > >> >>> kernel version number (from your uname string, sublevel can't be
> > > >> >>> determined),
> > > >> >>> please help us improve that function like what we have done for ubuntu
> > > >> >>> (checking /proc/version_signature).
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>>> My kernel version is: 4.9.20 (from Makefile)
> > > >> >>>>
> > > >> >>>> So, I dont think there is any problem with kernel version.
> > > >> >>>>
> > > >> >>>> If any body have come across this problem please let me know how to
> > > >> >>>> resolve this issue.
> > > >> >>>>
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> Thank You!
> > > >> >>>>
> > > >> >>>> Regards,
> > > >> >>>> Pintu
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >
> > > >
> > > > --
> > > > Masami Hiramatsu <mhiramat@kernel.org>
> >
> >
> > --
> > Masami Hiramatsu <mhiramat@kernel.org>


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [perf] perf probe fails sometimes on 4.9

[Pintu Kumar]

Dear Masami,

Now I am stuck again with 'perf test' failure on 4.9

# perf --version
perf version 4.9.20-<product name>

# perf test
16: Try 'import perf' in python, checking link problems      : FAILED!
37.2: Test BPF prologue generation                           : FAILED!

If you have any clue about these failure please hep me.

Here are the verbose output:
-----------------------------------------------------------------------------
1) # perf test python -v
16: Try 'import perf' in python, checking link problems      :
--- start ---
test child forked, pid 24562
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named perf
test child finished with -1
---- end ----
Try 'import perf' in python, checking link problems: FAILED!
------------------------------------------------------------------------------

2) # perf test BPF -v
-------------------------------------------------------------------------------
.....
bpf: config 'func=null_lseek file->f_mode offset orig' is ok
Looking at the vmlinux_path (8 entries long)
symsrc__init: cannot get elf header.
Failed to find the path for kernel: Invalid ELF file
bpf_probe: failed to convert perf probe eventsFailed to add events
selected by BPF
test child finished with -1
---- end ----
Test BPF filter subtest 1: FAILED!

-------------------------------------------------------------------------------


Thanks,
Pintu


On Wed, Jan 31, 2018 at 9:01 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> On Tue, 30 Jan 2018 19:20:36 +0530
> Pintu Kumar <pintu.ping@gmail.com> wrote:
>
>> On Tue, Jan 30, 2018 at 11:13 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>> >
>> > On Mon, 29 Jan 2018 22:00:52 +0530
>> > Pintu Kumar <pintu.ping@gmail.com> wrote:
>> >
>> > > Dear Masami,
>> > >
>> > > Thank you so much for your reply.
>> > > Please find some of my answers inline.
>> > >
>> > >
>> > > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>> > > > On Mon, 29 Jan 2018 13:40:34 +0530
>> > > > Pintu Kumar <pintu.ping@gmail.com> wrote:
>> > > >
>> > > >> Hi All,
>> > > >>
>> > > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
>> > > >> # perf probe --add schedule
>> > > >> schedule is out of .text, skip it.
>> > > >>   Error: Failed to add events.
>> > > >>
>> > > >> If any one have come across this problem please let me know the cause.
>> > > >
>> > > > Hi Pintu,
>> > > >
>> > > > Could you run it with --vv?
>> > > >
>> > > Ok, I will send verbose output by tomorrow.
>> > >
>> > > >>
>> > > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
>> > > >
>> > > > Without it, you can not probe source-level probe nor trace local variable.
>> > > >
>> > >
>> > > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
>> > > However, I will try to manually include "-g" option during compilation.
>> > >
>> > > >> However, I manually copied the vmlinux file to /boot/ directory, but
>> > > >> still it does not work.
>> > > >
>> > > > That doesn't work.
>> > > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
>> > > > Without that option, debuginfo is not generated with vmlinux.
>> > > >
>> > > >>
>> > > >> I checked upstream patches until 4.15 but could not find any clue.
>> > > >> Please let me know if there is any fixes available for this.
>> > > >
>> > > > Could you also ensure that you run perf by root user?
>> > > >
>> > >
>> > > Yes I am running with root user.
>> > >
>> > > My concern is sometimes it works but sometimes it fails.
>> >
>> > What I thought was that your kernel enables kptr_strict(but maybe not.)
>> > Can you also try to find "schedule" and "_etext" functions in
>> > /proc/kallsyms?
>> >
>>
>> Yes these functions are present in kallsyms.
>>
>> # cat /proc/kallsyms | grep -w schedule ; cat /proc/kallsyms | grep -w _etext
>> ffffffff829328e0 T schedule
>> ffffffff8293e751 T _etext
>>
>>
>> > > I still needs to figure out, in which condition it works and which
>> > > condition it fails.
>> >
>> > Yeah, it is important.
>> >
>> > > Usually I noticed that in fresh reboot case it works.
>> >
>> > So, after a while, it doesn't work again? If so, it sounds like a daemon
>> > process changes settings in background.
>> >
>>
>> Fortunately or unfortunately, now it is passing every time.
>> Still not sure about the cause.
>> If it fails again, I will collect info with -v and report again.
>> Any specific info that I need to check when it fails ?
>
> OK, if you hit it again, please run perf-probe with -vvv option
> (-v is enough usually, if it is unknown bug, -vvv will be better)
>
> Thank you,
>
>
>>
>> Thank You,
>> Pintu
>>
>>
>> > Thank you,
>> >
>> > >
>> > >
>> > > > Thank you,
>> > > >
>> > > >
>> > > >>
>> > > >>
>> > > >> Thank You!
>> > > >> Regards,
>> > > >> Pintu
>> > > >>
>> > > >>
>> > > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>> > > >> > Hi,
>> > > >> >
>> > > >> > ** Changed the subject now, since these issues are related to general
>> > > >> > perf commands.
>> > > >> >
>> > > >> > Following are the issues:
>> > > >> >
>> > > >> > 1) perf probe --add schedule - FAILED
>> > > >> > output:
>> > > >> > schedule is out of .text, skip it.
>> > > >> >   Error: Failed to add events.
>> > > >> >
>> > > >> > what is the issue here?
>> > > >> > Sometimes it pass and sometimes it fails...
>> > > >> > Similar is the case of 'perf inject' as well.
>> > > >> >
>> > > >> > 2) perf test - 1 FAILURE
>> > > >> > 37.1: Test basic BPF filtering                               : FAILED!
>> > > >> > 37.2: Test BPF prologue generation                           : Skip
>> > > >> > 37.3: Test BPF relocation checker                            : Skip
>> > > >> >
>> > > >> > bpf: config program 'func=SyS_epoll_wait'
>> > > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
>> > > >> > bpf: config 'func=SyS_epoll_wait' is ok
>> > > >> > Looking at the vmlinux_path (8 entries long)
>> > > >> > Using /boot/vmlinux for symbols
>> > > >> > Could not open debuginfo. Try to use symbols.
>> > > >> > SyS_epoll_wait is out of .text, skip it.
>> > > >> > bpf_probe: failed to convert perf probe eventsFailed to add events
>> > > >> > selected by BPF
>> > > >> > test child finished with -1
>> > > >> > ---- end ----
>> > > >> > Test BPF filter subtest 0: FAILED!
>> > > >> >
>> > > >> > Looks like both 1,2 are related.
>> > > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
>> > > >> > to /boot/ folder.
>> > > >> >
>> > > >> > -------
>> > > >> > Some more info:
>> > > >> >
>> > > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
>> > > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> > > >> > unset env: KBUILD_OPTS
>> > > >> > include option is set to  -nostdinc -isystem
>> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> > > >> > set env: NR_CPUS=8
>> > > >> > set env: LINUX_VERSION_CODE=0x40914
>> > > >> > set env: CLANG_EXEC=/usr/bin/clang
>> > > >> > set env: CLANG_OPTIONS=-xc
>> > > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
>> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> > > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> > > >> >
>> > > >> >
>> > > >> > If you have any clue about these failure please help me.
>> > > >> >
>> > > >> >
>> > > >> > Thanks,
>> > > >> > Pintu
>> > > >> >
>> > > >> >
>> > > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>> > > >> >> Hi,
>> > > >> >>
>> > > >> >> Thanks for your help.
>> > > >> >> Yes it was a sub version issue.
>> > > >> >>
>> > > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
>> > > >> >>
>> > > >> >> make -j8 bindeb-pkg SUBLEVEL=20
>> > > >> >>
>> > > >> >> After passing the correct sublevel now the test is working.
>> > > >> >>
>> > > >> >> But still following are failing:
>> > > >> >>
>> > > >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
>> > > >> >> 37.2: Test BPF prologue generation                           : FAILED!
>> > > >> >>
>> > > >> >>
>> > > >> >> This is the error I get:
>> > > >> >>
>> > > >> >> 16: Try 'import perf' in python, checking link problems      :
>> > > >> >> --- start ---
>> > > >> >> test child forked, pid 7637
>> > > >> >> Traceback (most recent call last):
>> > > >> >>   File "<stdin>", line 1, in <module>
>> > > >> >> ImportError: No module named perf
>> > > >> >> test child finished with -1
>> > > >> >> ---- end ----
>> > > >> >> Try 'import perf' in python, checking link problems: FAILED!
>> > > >> >>
>> > > >> >>
>> > > >> >> Looking at the vmlinux_path (8 entries long)
>> > > >> >> symsrc__init: cannot get elf header.
>> > > >> >> Failed to find the path for kernel: Invalid ELF file
>> > > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
>> > > >> >> selected by BPF
>> > > >> >> test child finished with -1
>> > > >> >> ---- end ----
>> > > >> >> Test BPF filter subtest 1: FAILED!
>> > > >> >>
>> > > >> >>
>> > > >> >>
>> > > >> >> Thanks,
>> > > >> >> Pintu
>> > > >> >>
>> > > >> >>
>> > > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
>> > > >> >>>
>> > > >> >>>
>> > > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
>> > > >> >>>>
>> > > >> >>>> Hi All,
>> > > >> >>>>
>> > > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
>> > > >> >>>> kernel version 4.9.20.
>> > > >> >>>> I have installed several others packages including: clang, llvm
>> > > >> >>>>
>> > > >> >>>> But, when I run 'perf test' I get some FAILURE.
>> > > >> >>>> Specially, 'perf test LLVM' is failing.
>> > > >> >>>>
>> > > >> >>>> Please check the below error logs:
>> > > >> >>>>
>> > > >> >>>> ----------------------------------------------------
>> > > >> >>>> # perf test LLVM
>> > > >> >>>> 35: Test LLVM searching and compiling                        :
>> > > >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
>> > > >> >>>> 35.2: Test kbuild searching                                  : Skip
>> > > >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
>> > > >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
>> > > >> >>>> ----------------------------------------------------
>> > > >> >>>>
>> > > >> >>>> When I run with -v I get this:
>> > > >> >>>>
>> > > >> >>>> -----------------------------------------------------
>> > > >> >>>> # perf test -v LLVM
>> > > >> >>>> 35: Test LLVM searching and compiling                        :
>> > > >> >>>> 35.1: Basic BPF llvm compiling test                          :
>> > > >> >>>> --- start ---
>> > > >> >>>> test child forked, pid 3304
>> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> > > >> >>>> WARNING:        unable to get correct kernel building directory.
>> > > >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
>> > > >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
>> > > >> >>>>          detection.
>> > > >> >>>>
>> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
>> > > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
>> > > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
>> > > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
>> > > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
>> > > >> >>>> -
>> > > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
>> > > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
>> > > >> >>>> test child finished with -1
>> > > >> >>>> ---- end ----
>> > > >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
>> > > >> >>>> -----------------------------------------------------
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>> Basic BPF LLVM compiling test is failing.
>> > > >> >>>> It reports that bpf could not able to read the kernel version from uname.
>> > > >> >>>> Is it because of this that 'perf test LLVM' is failing ?
>> > > >> >>>>
>> > > >> >>>> My uname says:
>> > > >> >>>> 4.9-xxxx-amd-x86-64
>> > > >> >>>
>> > > >> >>>
>> > > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
>> > > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
>> > > >> >>> report
>> > > >> >>> kernel version number (from your uname string, sublevel can't be
>> > > >> >>> determined),
>> > > >> >>> please help us improve that function like what we have done for ubuntu
>> > > >> >>> (checking /proc/version_signature).
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>> My kernel version is: 4.9.20 (from Makefile)
>> > > >> >>>>
>> > > >> >>>> So, I dont think there is any problem with kernel version.
>> > > >> >>>>
>> > > >> >>>> If any body have come across this problem please let me know how to
>> > > >> >>>> resolve this issue.
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>> Thank You!
>> > > >> >>>>
>> > > >> >>>> Regards,
>> > > >> >>>> Pintu
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>
>> > > >
>> > > >
>> > > > --
>> > > > Masami Hiramatsu <mhiramat@kernel.org>
>> >
>> >
>> > --
>> > Masami Hiramatsu <mhiramat@kernel.org>
>
>
> --
> Masami Hiramatsu <mhiramat@kernel.org>

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

[perf] perf probe fails sometimes on 4.9

[Pintu Kumar]

Dear Masami,

Now I am stuck again with 'perf test' failure on 4.9

# perf --version
perf version 4.9.20-<product name>

# perf test
16: Try 'import perf' in python, checking link problems      : FAILED!
37.2: Test BPF prologue generation                           : FAILED!

If you have any clue about these failure please hep me.

Here are the verbose output:
-----------------------------------------------------------------------------
1) # perf test python -v
16: Try 'import perf' in python, checking link problems      :
--- start ---
test child forked, pid 24562
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named perf
test child finished with -1
---- end ----
Try 'import perf' in python, checking link problems: FAILED!
------------------------------------------------------------------------------

2) # perf test BPF -v
-------------------------------------------------------------------------------
.....
bpf: config 'func=null_lseek file->f_mode offset orig' is ok
Looking at the vmlinux_path (8 entries long)
symsrc__init: cannot get elf header.
Failed to find the path for kernel: Invalid ELF file
bpf_probe: failed to convert perf probe eventsFailed to add events
selected by BPF
test child finished with -1
---- end ----
Test BPF filter subtest 1: FAILED!

-------------------------------------------------------------------------------


Thanks,
Pintu


On Wed, Jan 31, 2018 at 9:01 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
> On Tue, 30 Jan 2018 19:20:36 +0530
> Pintu Kumar <pintu.ping@gmail.com> wrote:
>
>> On Tue, Jan 30, 2018 at 11:13 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>> >
>> > On Mon, 29 Jan 2018 22:00:52 +0530
>> > Pintu Kumar <pintu.ping@gmail.com> wrote:
>> >
>> > > Dear Masami,
>> > >
>> > > Thank you so much for your reply.
>> > > Please find some of my answers inline.
>> > >
>> > >
>> > > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>> > > > On Mon, 29 Jan 2018 13:40:34 +0530
>> > > > Pintu Kumar <pintu.ping@gmail.com> wrote:
>> > > >
>> > > >> Hi All,
>> > > >>
>> > > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
>> > > >> # perf probe --add schedule
>> > > >> schedule is out of .text, skip it.
>> > > >>   Error: Failed to add events.
>> > > >>
>> > > >> If any one have come across this problem please let me know the cause.
>> > > >
>> > > > Hi Pintu,
>> > > >
>> > > > Could you run it with --vv?
>> > > >
>> > > Ok, I will send verbose output by tomorrow.
>> > >
>> > > >>
>> > > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
>> > > >
>> > > > Without it, you can not probe source-level probe nor trace local variable.
>> > > >
>> > >
>> > > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
>> > > However, I will try to manually include "-g" option during compilation.
>> > >
>> > > >> However, I manually copied the vmlinux file to /boot/ directory, but
>> > > >> still it does not work.
>> > > >
>> > > > That doesn't work.
>> > > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
>> > > > Without that option, debuginfo is not generated with vmlinux.
>> > > >
>> > > >>
>> > > >> I checked upstream patches until 4.15 but could not find any clue.
>> > > >> Please let me know if there is any fixes available for this.
>> > > >
>> > > > Could you also ensure that you run perf by root user?
>> > > >
>> > >
>> > > Yes I am running with root user.
>> > >
>> > > My concern is sometimes it works but sometimes it fails.
>> >
>> > What I thought was that your kernel enables kptr_strict(but maybe not.)
>> > Can you also try to find "schedule" and "_etext" functions in
>> > /proc/kallsyms?
>> >
>>
>> Yes these functions are present in kallsyms.
>>
>> # cat /proc/kallsyms | grep -w schedule ; cat /proc/kallsyms | grep -w _etext
>> ffffffff829328e0 T schedule
>> ffffffff8293e751 T _etext
>>
>>
>> > > I still needs to figure out, in which condition it works and which
>> > > condition it fails.
>> >
>> > Yeah, it is important.
>> >
>> > > Usually I noticed that in fresh reboot case it works.
>> >
>> > So, after a while, it doesn't work again? If so, it sounds like a daemon
>> > process changes settings in background.
>> >
>>
>> Fortunately or unfortunately, now it is passing every time.
>> Still not sure about the cause.
>> If it fails again, I will collect info with -v and report again.
>> Any specific info that I need to check when it fails ?
>
> OK, if you hit it again, please run perf-probe with -vvv option
> (-v is enough usually, if it is unknown bug, -vvv will be better)
>
> Thank you,
>
>
>>
>> Thank You,
>> Pintu
>>
>>
>> > Thank you,
>> >
>> > >
>> > >
>> > > > Thank you,
>> > > >
>> > > >
>> > > >>
>> > > >>
>> > > >> Thank You!
>> > > >> Regards,
>> > > >> Pintu
>> > > >>
>> > > >>
>> > > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>> > > >> > Hi,
>> > > >> >
>> > > >> > ** Changed the subject now, since these issues are related to general
>> > > >> > perf commands.
>> > > >> >
>> > > >> > Following are the issues:
>> > > >> >
>> > > >> > 1) perf probe --add schedule - FAILED
>> > > >> > output:
>> > > >> > schedule is out of .text, skip it.
>> > > >> >   Error: Failed to add events.
>> > > >> >
>> > > >> > what is the issue here?
>> > > >> > Sometimes it pass and sometimes it fails...
>> > > >> > Similar is the case of 'perf inject' as well.
>> > > >> >
>> > > >> > 2) perf test - 1 FAILURE
>> > > >> > 37.1: Test basic BPF filtering                               : FAILED!
>> > > >> > 37.2: Test BPF prologue generation                           : Skip
>> > > >> > 37.3: Test BPF relocation checker                            : Skip
>> > > >> >
>> > > >> > bpf: config program 'func=SyS_epoll_wait'
>> > > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
>> > > >> > bpf: config 'func=SyS_epoll_wait' is ok
>> > > >> > Looking at the vmlinux_path (8 entries long)
>> > > >> > Using /boot/vmlinux for symbols
>> > > >> > Could not open debuginfo. Try to use symbols.
>> > > >> > SyS_epoll_wait is out of .text, skip it.
>> > > >> > bpf_probe: failed to convert perf probe eventsFailed to add events
>> > > >> > selected by BPF
>> > > >> > test child finished with -1
>> > > >> > ---- end ----
>> > > >> > Test BPF filter subtest 0: FAILED!
>> > > >> >
>> > > >> > Looks like both 1,2 are related.
>> > > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
>> > > >> > to /boot/ folder.
>> > > >> >
>> > > >> > -------
>> > > >> > Some more info:
>> > > >> >
>> > > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
>> > > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> > > >> > unset env: KBUILD_OPTS
>> > > >> > include option is set to  -nostdinc -isystem
>> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> > > >> > set env: NR_CPUS=8
>> > > >> > set env: LINUX_VERSION_CODE=0x40914
>> > > >> > set env: CLANG_EXEC=/usr/bin/clang
>> > > >> > set env: CLANG_OPTIONS=-xc
>> > > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
>> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> > > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> > > >> >
>> > > >> >
>> > > >> > If you have any clue about these failure please help me.
>> > > >> >
>> > > >> >
>> > > >> > Thanks,
>> > > >> > Pintu
>> > > >> >
>> > > >> >
>> > > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>> > > >> >> Hi,
>> > > >> >>
>> > > >> >> Thanks for your help.
>> > > >> >> Yes it was a sub version issue.
>> > > >> >>
>> > > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
>> > > >> >>
>> > > >> >> make -j8 bindeb-pkg SUBLEVEL=20
>> > > >> >>
>> > > >> >> After passing the correct sublevel now the test is working.
>> > > >> >>
>> > > >> >> But still following are failing:
>> > > >> >>
>> > > >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
>> > > >> >> 37.2: Test BPF prologue generation                           : FAILED!
>> > > >> >>
>> > > >> >>
>> > > >> >> This is the error I get:
>> > > >> >>
>> > > >> >> 16: Try 'import perf' in python, checking link problems      :
>> > > >> >> --- start ---
>> > > >> >> test child forked, pid 7637
>> > > >> >> Traceback (most recent call last):
>> > > >> >>   File "<stdin>", line 1, in <module>
>> > > >> >> ImportError: No module named perf
>> > > >> >> test child finished with -1
>> > > >> >> ---- end ----
>> > > >> >> Try 'import perf' in python, checking link problems: FAILED!
>> > > >> >>
>> > > >> >>
>> > > >> >> Looking at the vmlinux_path (8 entries long)
>> > > >> >> symsrc__init: cannot get elf header.
>> > > >> >> Failed to find the path for kernel: Invalid ELF file
>> > > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
>> > > >> >> selected by BPF
>> > > >> >> test child finished with -1
>> > > >> >> ---- end ----
>> > > >> >> Test BPF filter subtest 1: FAILED!
>> > > >> >>
>> > > >> >>
>> > > >> >>
>> > > >> >> Thanks,
>> > > >> >> Pintu
>> > > >> >>
>> > > >> >>
>> > > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
>> > > >> >>>
>> > > >> >>>
>> > > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
>> > > >> >>>>
>> > > >> >>>> Hi All,
>> > > >> >>>>
>> > > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
>> > > >> >>>> kernel version 4.9.20.
>> > > >> >>>> I have installed several others packages including: clang, llvm
>> > > >> >>>>
>> > > >> >>>> But, when I run 'perf test' I get some FAILURE.
>> > > >> >>>> Specially, 'perf test LLVM' is failing.
>> > > >> >>>>
>> > > >> >>>> Please check the below error logs:
>> > > >> >>>>
>> > > >> >>>> ----------------------------------------------------
>> > > >> >>>> # perf test LLVM
>> > > >> >>>> 35: Test LLVM searching and compiling                        :
>> > > >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
>> > > >> >>>> 35.2: Test kbuild searching                                  : Skip
>> > > >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
>> > > >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
>> > > >> >>>> ----------------------------------------------------
>> > > >> >>>>
>> > > >> >>>> When I run with -v I get this:
>> > > >> >>>>
>> > > >> >>>> -----------------------------------------------------
>> > > >> >>>> # perf test -v LLVM
>> > > >> >>>> 35: Test LLVM searching and compiling                        :
>> > > >> >>>> 35.1: Basic BPF llvm compiling test                          :
>> > > >> >>>> --- start ---
>> > > >> >>>> test child forked, pid 3304
>> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> > > >> >>>> WARNING:        unable to get correct kernel building directory.
>> > > >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
>> > > >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
>> > > >> >>>>          detection.
>> > > >> >>>>
>> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
>> > > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
>> > > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
>> > > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
>> > > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
>> > > >> >>>> -
>> > > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
>> > > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
>> > > >> >>>> test child finished with -1
>> > > >> >>>> ---- end ----
>> > > >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
>> > > >> >>>> -----------------------------------------------------
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>> Basic BPF LLVM compiling test is failing.
>> > > >> >>>> It reports that bpf could not able to read the kernel version from uname.
>> > > >> >>>> Is it because of this that 'perf test LLVM' is failing ?
>> > > >> >>>>
>> > > >> >>>> My uname says:
>> > > >> >>>> 4.9-xxxx-amd-x86-64
>> > > >> >>>
>> > > >> >>>
>> > > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
>> > > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
>> > > >> >>> report
>> > > >> >>> kernel version number (from your uname string, sublevel can't be
>> > > >> >>> determined),
>> > > >> >>> please help us improve that function like what we have done for ubuntu
>> > > >> >>> (checking /proc/version_signature).
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>> My kernel version is: 4.9.20 (from Makefile)
>> > > >> >>>>
>> > > >> >>>> So, I dont think there is any problem with kernel version.
>> > > >> >>>>
>> > > >> >>>> If any body have come across this problem please let me know how to
>> > > >> >>>> resolve this issue.
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>> Thank You!
>> > > >> >>>>
>> > > >> >>>> Regards,
>> > > >> >>>> Pintu
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>
>> > > >
>> > > >
>> > > > --
>> > > > Masami Hiramatsu <mhiramat@kernel.org>
>> >
>> >
>> > --
>> > Masami Hiramatsu <mhiramat@kernel.org>
>
>
> --
> Masami Hiramatsu <mhiramat@kernel.org>

Re: [perf] perf probe fails sometimes on 4.9

[Pintu Kumar]

Hi,

After enabling DEBUG_INFO in kernel I still get this error for BPF test.
Please help.

# perf test BPF -v
.....
Looking at the vmlinux_path (8 entries long)
Using /usr/lib/debug/boot/vmlinux-4.9.00-xxxx-amd-x86-64-00071-gd94c220-dirty
for symbols
Open Debuginfo file:
/usr/lib/debug/boot/vmlinux-4.9.00-xxxx-amd-x86-64-00071-gd94c220-dirty
Try to find probe point from debuginfo.
Matched function: null_lseek [105be32]
Probe point found: null_lseek+0
Searching 'file' variable in context.
Converting variable file into trace event.
converting f_mode in file
file(type:file) has no member f_mode.
An error occurred in debuginfo analysis (-22).
bpf_probe: failed to convert perf probe eventsFailed to add events
selected by BPF
test child finished with -1
---- end ----
Test BPF filter subtest 1: FAILED!



On Thu, Feb 1, 2018 at 10:50 AM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> Dear Masami,
>
> Now I am stuck again with 'perf test' failure on 4.9
>
> # perf --version
> perf version 4.9.20-<product name>
>
> # perf test
> 16: Try 'import perf' in python, checking link problems      : FAILED!
> 37.2: Test BPF prologue generation                           : FAILED!
>
> If you have any clue about these failure please hep me.
>
> Here are the verbose output:
> -----------------------------------------------------------------------------
> 1) # perf test python -v
> 16: Try 'import perf' in python, checking link problems      :
> --- start ---
> test child forked, pid 24562
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ImportError: No module named perf
> test child finished with -1
> ---- end ----
> Try 'import perf' in python, checking link problems: FAILED!
> ------------------------------------------------------------------------------
>
> 2) # perf test BPF -v
> -------------------------------------------------------------------------------
> .....
> bpf: config 'func=null_lseek file->f_mode offset orig' is ok
> Looking at the vmlinux_path (8 entries long)
> symsrc__init: cannot get elf header.
> Failed to find the path for kernel: Invalid ELF file
> bpf_probe: failed to convert perf probe eventsFailed to add events
> selected by BPF
> test child finished with -1
> ---- end ----
> Test BPF filter subtest 1: FAILED!
>
> -------------------------------------------------------------------------------
>
>
> Thanks,
> Pintu
>
>
> On Wed, Jan 31, 2018 at 9:01 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>> On Tue, 30 Jan 2018 19:20:36 +0530
>> Pintu Kumar <pintu.ping@gmail.com> wrote:
>>
>>> On Tue, Jan 30, 2018 at 11:13 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>>> >
>>> > On Mon, 29 Jan 2018 22:00:52 +0530
>>> > Pintu Kumar <pintu.ping@gmail.com> wrote:
>>> >
>>> > > Dear Masami,
>>> > >
>>> > > Thank you so much for your reply.
>>> > > Please find some of my answers inline.
>>> > >
>>> > >
>>> > > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>>> > > > On Mon, 29 Jan 2018 13:40:34 +0530
>>> > > > Pintu Kumar <pintu.ping@gmail.com> wrote:
>>> > > >
>>> > > >> Hi All,
>>> > > >>
>>> > > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
>>> > > >> # perf probe --add schedule
>>> > > >> schedule is out of .text, skip it.
>>> > > >>   Error: Failed to add events.
>>> > > >>
>>> > > >> If any one have come across this problem please let me know the cause.
>>> > > >
>>> > > > Hi Pintu,
>>> > > >
>>> > > > Could you run it with --vv?
>>> > > >
>>> > > Ok, I will send verbose output by tomorrow.
>>> > >
>>> > > >>
>>> > > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
>>> > > >
>>> > > > Without it, you can not probe source-level probe nor trace local variable.
>>> > > >
>>> > >
>>> > > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
>>> > > However, I will try to manually include "-g" option during compilation.
>>> > >
>>> > > >> However, I manually copied the vmlinux file to /boot/ directory, but
>>> > > >> still it does not work.
>>> > > >
>>> > > > That doesn't work.
>>> > > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
>>> > > > Without that option, debuginfo is not generated with vmlinux.
>>> > > >
>>> > > >>
>>> > > >> I checked upstream patches until 4.15 but could not find any clue.
>>> > > >> Please let me know if there is any fixes available for this.
>>> > > >
>>> > > > Could you also ensure that you run perf by root user?
>>> > > >
>>> > >
>>> > > Yes I am running with root user.
>>> > >
>>> > > My concern is sometimes it works but sometimes it fails.
>>> >
>>> > What I thought was that your kernel enables kptr_strict(but maybe not.)
>>> > Can you also try to find "schedule" and "_etext" functions in
>>> > /proc/kallsyms?
>>> >
>>>
>>> Yes these functions are present in kallsyms.
>>>
>>> # cat /proc/kallsyms | grep -w schedule ; cat /proc/kallsyms | grep -w _etext
>>> ffffffff829328e0 T schedule
>>> ffffffff8293e751 T _etext
>>>
>>>
>>> > > I still needs to figure out, in which condition it works and which
>>> > > condition it fails.
>>> >
>>> > Yeah, it is important.
>>> >
>>> > > Usually I noticed that in fresh reboot case it works.
>>> >
>>> > So, after a while, it doesn't work again? If so, it sounds like a daemon
>>> > process changes settings in background.
>>> >
>>>
>>> Fortunately or unfortunately, now it is passing every time.
>>> Still not sure about the cause.
>>> If it fails again, I will collect info with -v and report again.
>>> Any specific info that I need to check when it fails ?
>>
>> OK, if you hit it again, please run perf-probe with -vvv option
>> (-v is enough usually, if it is unknown bug, -vvv will be better)
>>
>> Thank you,
>>
>>
>>>
>>> Thank You,
>>> Pintu
>>>
>>>
>>> > Thank you,
>>> >
>>> > >
>>> > >
>>> > > > Thank you,
>>> > > >
>>> > > >
>>> > > >>
>>> > > >>
>>> > > >> Thank You!
>>> > > >> Regards,
>>> > > >> Pintu
>>> > > >>
>>> > > >>
>>> > > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>>> > > >> > Hi,
>>> > > >> >
>>> > > >> > ** Changed the subject now, since these issues are related to general
>>> > > >> > perf commands.
>>> > > >> >
>>> > > >> > Following are the issues:
>>> > > >> >
>>> > > >> > 1) perf probe --add schedule - FAILED
>>> > > >> > output:
>>> > > >> > schedule is out of .text, skip it.
>>> > > >> >   Error: Failed to add events.
>>> > > >> >
>>> > > >> > what is the issue here?
>>> > > >> > Sometimes it pass and sometimes it fails...
>>> > > >> > Similar is the case of 'perf inject' as well.
>>> > > >> >
>>> > > >> > 2) perf test - 1 FAILURE
>>> > > >> > 37.1: Test basic BPF filtering                               : FAILED!
>>> > > >> > 37.2: Test BPF prologue generation                           : Skip
>>> > > >> > 37.3: Test BPF relocation checker                            : Skip
>>> > > >> >
>>> > > >> > bpf: config program 'func=SyS_epoll_wait'
>>> > > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
>>> > > >> > bpf: config 'func=SyS_epoll_wait' is ok
>>> > > >> > Looking at the vmlinux_path (8 entries long)
>>> > > >> > Using /boot/vmlinux for symbols
>>> > > >> > Could not open debuginfo. Try to use symbols.
>>> > > >> > SyS_epoll_wait is out of .text, skip it.
>>> > > >> > bpf_probe: failed to convert perf probe eventsFailed to add events
>>> > > >> > selected by BPF
>>> > > >> > test child finished with -1
>>> > > >> > ---- end ----
>>> > > >> > Test BPF filter subtest 0: FAILED!
>>> > > >> >
>>> > > >> > Looks like both 1,2 are related.
>>> > > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
>>> > > >> > to /boot/ folder.
>>> > > >> >
>>> > > >> > -------
>>> > > >> > Some more info:
>>> > > >> >
>>> > > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
>>> > > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>>> > > >> > unset env: KBUILD_OPTS
>>> > > >> > include option is set to  -nostdinc -isystem
>>> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>>> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>>> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>>> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>>> > > >> > set env: NR_CPUS=8
>>> > > >> > set env: LINUX_VERSION_CODE=0x40914
>>> > > >> > set env: CLANG_EXEC=/usr/bin/clang
>>> > > >> > set env: CLANG_OPTIONS=-xc
>>> > > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
>>> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>>> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>>> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>>> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>>> > > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>>> > > >> >
>>> > > >> >
>>> > > >> > If you have any clue about these failure please help me.
>>> > > >> >
>>> > > >> >
>>> > > >> > Thanks,
>>> > > >> > Pintu
>>> > > >> >
>>> > > >> >
>>> > > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>>> > > >> >> Hi,
>>> > > >> >>
>>> > > >> >> Thanks for your help.
>>> > > >> >> Yes it was a sub version issue.
>>> > > >> >>
>>> > > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
>>> > > >> >>
>>> > > >> >> make -j8 bindeb-pkg SUBLEVEL=20
>>> > > >> >>
>>> > > >> >> After passing the correct sublevel now the test is working.
>>> > > >> >>
>>> > > >> >> But still following are failing:
>>> > > >> >>
>>> > > >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
>>> > > >> >> 37.2: Test BPF prologue generation                           : FAILED!
>>> > > >> >>
>>> > > >> >>
>>> > > >> >> This is the error I get:
>>> > > >> >>
>>> > > >> >> 16: Try 'import perf' in python, checking link problems      :
>>> > > >> >> --- start ---
>>> > > >> >> test child forked, pid 7637
>>> > > >> >> Traceback (most recent call last):
>>> > > >> >>   File "<stdin>", line 1, in <module>
>>> > > >> >> ImportError: No module named perf
>>> > > >> >> test child finished with -1
>>> > > >> >> ---- end ----
>>> > > >> >> Try 'import perf' in python, checking link problems: FAILED!
>>> > > >> >>
>>> > > >> >>
>>> > > >> >> Looking at the vmlinux_path (8 entries long)
>>> > > >> >> symsrc__init: cannot get elf header.
>>> > > >> >> Failed to find the path for kernel: Invalid ELF file
>>> > > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
>>> > > >> >> selected by BPF
>>> > > >> >> test child finished with -1
>>> > > >> >> ---- end ----
>>> > > >> >> Test BPF filter subtest 1: FAILED!
>>> > > >> >>
>>> > > >> >>
>>> > > >> >>
>>> > > >> >> Thanks,
>>> > > >> >> Pintu
>>> > > >> >>
>>> > > >> >>
>>> > > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
>>> > > >> >>>>
>>> > > >> >>>> Hi All,
>>> > > >> >>>>
>>> > > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
>>> > > >> >>>> kernel version 4.9.20.
>>> > > >> >>>> I have installed several others packages including: clang, llvm
>>> > > >> >>>>
>>> > > >> >>>> But, when I run 'perf test' I get some FAILURE.
>>> > > >> >>>> Specially, 'perf test LLVM' is failing.
>>> > > >> >>>>
>>> > > >> >>>> Please check the below error logs:
>>> > > >> >>>>
>>> > > >> >>>> ----------------------------------------------------
>>> > > >> >>>> # perf test LLVM
>>> > > >> >>>> 35: Test LLVM searching and compiling                        :
>>> > > >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
>>> > > >> >>>> 35.2: Test kbuild searching                                  : Skip
>>> > > >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
>>> > > >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
>>> > > >> >>>> ----------------------------------------------------
>>> > > >> >>>>
>>> > > >> >>>> When I run with -v I get this:
>>> > > >> >>>>
>>> > > >> >>>> -----------------------------------------------------
>>> > > >> >>>> # perf test -v LLVM
>>> > > >> >>>> 35: Test LLVM searching and compiling                        :
>>> > > >> >>>> 35.1: Basic BPF llvm compiling test                          :
>>> > > >> >>>> --- start ---
>>> > > >> >>>> test child forked, pid 3304
>>> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>>> > > >> >>>> WARNING:        unable to get correct kernel building directory.
>>> > > >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
>>> > > >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
>>> > > >> >>>>          detection.
>>> > > >> >>>>
>>> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
>>> > > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
>>> > > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
>>> > > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
>>> > > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
>>> > > >> >>>> -
>>> > > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
>>> > > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
>>> > > >> >>>> test child finished with -1
>>> > > >> >>>> ---- end ----
>>> > > >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
>>> > > >> >>>> -----------------------------------------------------
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>> Basic BPF LLVM compiling test is failing.
>>> > > >> >>>> It reports that bpf could not able to read the kernel version from uname.
>>> > > >> >>>> Is it because of this that 'perf test LLVM' is failing ?
>>> > > >> >>>>
>>> > > >> >>>> My uname says:
>>> > > >> >>>> 4.9-xxxx-amd-x86-64
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
>>> > > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
>>> > > >> >>> report
>>> > > >> >>> kernel version number (from your uname string, sublevel can't be
>>> > > >> >>> determined),
>>> > > >> >>> please help us improve that function like what we have done for ubuntu
>>> > > >> >>> (checking /proc/version_signature).
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>>> My kernel version is: 4.9.20 (from Makefile)
>>> > > >> >>>>
>>> > > >> >>>> So, I dont think there is any problem with kernel version.
>>> > > >> >>>>
>>> > > >> >>>> If any body have come across this problem please let me know how to
>>> > > >> >>>> resolve this issue.
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>> Thank You!
>>> > > >> >>>>
>>> > > >> >>>> Regards,
>>> > > >> >>>> Pintu
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>>
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Masami Hiramatsu <mhiramat@kernel.org>
>>> >
>>> >
>>> > --
>>> > Masami Hiramatsu <mhiramat@kernel.org>
>>
>>
>> --
>> Masami Hiramatsu <mhiramat@kernel.org>

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

[perf] perf probe fails sometimes on 4.9

[Pintu Kumar]

Hi,

After enabling DEBUG_INFO in kernel I still get this error for BPF test.
Please help.

# perf test BPF -v
.....
Looking at the vmlinux_path (8 entries long)
Using /usr/lib/debug/boot/vmlinux-4.9.00-xxxx-amd-x86-64-00071-gd94c220-dirty
for symbols
Open Debuginfo file:
/usr/lib/debug/boot/vmlinux-4.9.00-xxxx-amd-x86-64-00071-gd94c220-dirty
Try to find probe point from debuginfo.
Matched function: null_lseek [105be32]
Probe point found: null_lseek+0
Searching 'file' variable in context.
Converting variable file into trace event.
converting f_mode in file
file(type:file) has no member f_mode.
An error occurred in debuginfo analysis (-22).
bpf_probe: failed to convert perf probe eventsFailed to add events
selected by BPF
test child finished with -1
---- end ----
Test BPF filter subtest 1: FAILED!



On Thu, Feb 1, 2018 at 10:50 AM, Pintu Kumar <pintu.ping@gmail.com> wrote:
> Dear Masami,
>
> Now I am stuck again with 'perf test' failure on 4.9
>
> # perf --version
> perf version 4.9.20-<product name>
>
> # perf test
> 16: Try 'import perf' in python, checking link problems      : FAILED!
> 37.2: Test BPF prologue generation                           : FAILED!
>
> If you have any clue about these failure please hep me.
>
> Here are the verbose output:
> -----------------------------------------------------------------------------
> 1) # perf test python -v
> 16: Try 'import perf' in python, checking link problems      :
> --- start ---
> test child forked, pid 24562
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ImportError: No module named perf
> test child finished with -1
> ---- end ----
> Try 'import perf' in python, checking link problems: FAILED!
> ------------------------------------------------------------------------------
>
> 2) # perf test BPF -v
> -------------------------------------------------------------------------------
> .....
> bpf: config 'func=null_lseek file->f_mode offset orig' is ok
> Looking at the vmlinux_path (8 entries long)
> symsrc__init: cannot get elf header.
> Failed to find the path for kernel: Invalid ELF file
> bpf_probe: failed to convert perf probe eventsFailed to add events
> selected by BPF
> test child finished with -1
> ---- end ----
> Test BPF filter subtest 1: FAILED!
>
> -------------------------------------------------------------------------------
>
>
> Thanks,
> Pintu
>
>
> On Wed, Jan 31, 2018 at 9:01 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>> On Tue, 30 Jan 2018 19:20:36 +0530
>> Pintu Kumar <pintu.ping@gmail.com> wrote:
>>
>>> On Tue, Jan 30, 2018 at 11:13 AM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>>> >
>>> > On Mon, 29 Jan 2018 22:00:52 +0530
>>> > Pintu Kumar <pintu.ping@gmail.com> wrote:
>>> >
>>> > > Dear Masami,
>>> > >
>>> > > Thank you so much for your reply.
>>> > > Please find some of my answers inline.
>>> > >
>>> > >
>>> > > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@kernel.org> wrote:
>>> > > > On Mon, 29 Jan 2018 13:40:34 +0530
>>> > > > Pintu Kumar <pintu.ping@gmail.com> wrote:
>>> > > >
>>> > > >> Hi All,
>>> > > >>
>>> > > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
>>> > > >> # perf probe --add schedule
>>> > > >> schedule is out of .text, skip it.
>>> > > >>   Error: Failed to add events.
>>> > > >>
>>> > > >> If any one have come across this problem please let me know the cause.
>>> > > >
>>> > > > Hi Pintu,
>>> > > >
>>> > > > Could you run it with --vv?
>>> > > >
>>> > > Ok, I will send verbose output by tomorrow.
>>> > >
>>> > > >>
>>> > > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
>>> > > >
>>> > > > Without it, you can not probe source-level probe nor trace local variable.
>>> > > >
>>> > >
>>> > > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
>>> > > However, I will try to manually include "-g" option during compilation.
>>> > >
>>> > > >> However, I manually copied the vmlinux file to /boot/ directory, but
>>> > > >> still it does not work.
>>> > > >
>>> > > > That doesn't work.
>>> > > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
>>> > > > Without that option, debuginfo is not generated with vmlinux.
>>> > > >
>>> > > >>
>>> > > >> I checked upstream patches until 4.15 but could not find any clue.
>>> > > >> Please let me know if there is any fixes available for this.
>>> > > >
>>> > > > Could you also ensure that you run perf by root user?
>>> > > >
>>> > >
>>> > > Yes I am running with root user.
>>> > >
>>> > > My concern is sometimes it works but sometimes it fails.
>>> >
>>> > What I thought was that your kernel enables kptr_strict(but maybe not.)
>>> > Can you also try to find "schedule" and "_etext" functions in
>>> > /proc/kallsyms?
>>> >
>>>
>>> Yes these functions are present in kallsyms.
>>>
>>> # cat /proc/kallsyms | grep -w schedule ; cat /proc/kallsyms | grep -w _etext
>>> ffffffff829328e0 T schedule
>>> ffffffff8293e751 T _etext
>>>
>>>
>>> > > I still needs to figure out, in which condition it works and which
>>> > > condition it fails.
>>> >
>>> > Yeah, it is important.
>>> >
>>> > > Usually I noticed that in fresh reboot case it works.
>>> >
>>> > So, after a while, it doesn't work again? If so, it sounds like a daemon
>>> > process changes settings in background.
>>> >
>>>
>>> Fortunately or unfortunately, now it is passing every time.
>>> Still not sure about the cause.
>>> If it fails again, I will collect info with -v and report again.
>>> Any specific info that I need to check when it fails ?
>>
>> OK, if you hit it again, please run perf-probe with -vvv option
>> (-v is enough usually, if it is unknown bug, -vvv will be better)
>>
>> Thank you,
>>
>>
>>>
>>> Thank You,
>>> Pintu
>>>
>>>
>>> > Thank you,
>>> >
>>> > >
>>> > >
>>> > > > Thank you,
>>> > > >
>>> > > >
>>> > > >>
>>> > > >>
>>> > > >> Thank You!
>>> > > >> Regards,
>>> > > >> Pintu
>>> > > >>
>>> > > >>
>>> > > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>>> > > >> > Hi,
>>> > > >> >
>>> > > >> > ** Changed the subject now, since these issues are related to general
>>> > > >> > perf commands.
>>> > > >> >
>>> > > >> > Following are the issues:
>>> > > >> >
>>> > > >> > 1) perf probe --add schedule - FAILED
>>> > > >> > output:
>>> > > >> > schedule is out of .text, skip it.
>>> > > >> >   Error: Failed to add events.
>>> > > >> >
>>> > > >> > what is the issue here?
>>> > > >> > Sometimes it pass and sometimes it fails...
>>> > > >> > Similar is the case of 'perf inject' as well.
>>> > > >> >
>>> > > >> > 2) perf test - 1 FAILURE
>>> > > >> > 37.1: Test basic BPF filtering                               : FAILED!
>>> > > >> > 37.2: Test BPF prologue generation                           : Skip
>>> > > >> > 37.3: Test BPF relocation checker                            : Skip
>>> > > >> >
>>> > > >> > bpf: config program 'func=SyS_epoll_wait'
>>> > > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
>>> > > >> > bpf: config 'func=SyS_epoll_wait' is ok
>>> > > >> > Looking at the vmlinux_path (8 entries long)
>>> > > >> > Using /boot/vmlinux for symbols
>>> > > >> > Could not open debuginfo. Try to use symbols.
>>> > > >> > SyS_epoll_wait is out of .text, skip it.
>>> > > >> > bpf_probe: failed to convert perf probe eventsFailed to add events
>>> > > >> > selected by BPF
>>> > > >> > test child finished with -1
>>> > > >> > ---- end ----
>>> > > >> > Test BPF filter subtest 0: FAILED!
>>> > > >> >
>>> > > >> > Looks like both 1,2 are related.
>>> > > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
>>> > > >> > to /boot/ folder.
>>> > > >> >
>>> > > >> > -------
>>> > > >> > Some more info:
>>> > > >> >
>>> > > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
>>> > > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>>> > > >> > unset env: KBUILD_OPTS
>>> > > >> > include option is set to  -nostdinc -isystem
>>> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>>> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>>> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>>> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>>> > > >> > set env: NR_CPUS=8
>>> > > >> > set env: LINUX_VERSION_CODE=0x40914
>>> > > >> > set env: CLANG_EXEC=/usr/bin/clang
>>> > > >> > set env: CLANG_OPTIONS=-xc
>>> > > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
>>> > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>>> > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>>> > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>>> > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>>> > > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>>> > > >> >
>>> > > >> >
>>> > > >> > If you have any clue about these failure please help me.
>>> > > >> >
>>> > > >> >
>>> > > >> > Thanks,
>>> > > >> > Pintu
>>> > > >> >
>>> > > >> >
>>> > > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@gmail.com> wrote:
>>> > > >> >> Hi,
>>> > > >> >>
>>> > > >> >> Thanks for your help.
>>> > > >> >> Yes it was a sub version issue.
>>> > > >> >>
>>> > > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
>>> > > >> >>
>>> > > >> >> make -j8 bindeb-pkg SUBLEVEL=20
>>> > > >> >>
>>> > > >> >> After passing the correct sublevel now the test is working.
>>> > > >> >>
>>> > > >> >> But still following are failing:
>>> > > >> >>
>>> > > >> >> 16: Try 'import perf' in python, checking link problems      : FAILED!
>>> > > >> >> 37.2: Test BPF prologue generation                           : FAILED!
>>> > > >> >>
>>> > > >> >>
>>> > > >> >> This is the error I get:
>>> > > >> >>
>>> > > >> >> 16: Try 'import perf' in python, checking link problems      :
>>> > > >> >> --- start ---
>>> > > >> >> test child forked, pid 7637
>>> > > >> >> Traceback (most recent call last):
>>> > > >> >>   File "<stdin>", line 1, in <module>
>>> > > >> >> ImportError: No module named perf
>>> > > >> >> test child finished with -1
>>> > > >> >> ---- end ----
>>> > > >> >> Try 'import perf' in python, checking link problems: FAILED!
>>> > > >> >>
>>> > > >> >>
>>> > > >> >> Looking at the vmlinux_path (8 entries long)
>>> > > >> >> symsrc__init: cannot get elf header.
>>> > > >> >> Failed to find the path for kernel: Invalid ELF file
>>> > > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
>>> > > >> >> selected by BPF
>>> > > >> >> test child finished with -1
>>> > > >> >> ---- end ----
>>> > > >> >> Test BPF filter subtest 1: FAILED!
>>> > > >> >>
>>> > > >> >>
>>> > > >> >>
>>> > > >> >> Thanks,
>>> > > >> >> Pintu
>>> > > >> >>
>>> > > >> >>
>>> > > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@huawei.com> wrote:
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
>>> > > >> >>>>
>>> > > >> >>>> Hi All,
>>> > > >> >>>>
>>> > > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
>>> > > >> >>>> kernel version 4.9.20.
>>> > > >> >>>> I have installed several others packages including: clang, llvm
>>> > > >> >>>>
>>> > > >> >>>> But, when I run 'perf test' I get some FAILURE.
>>> > > >> >>>> Specially, 'perf test LLVM' is failing.
>>> > > >> >>>>
>>> > > >> >>>> Please check the below error logs:
>>> > > >> >>>>
>>> > > >> >>>> ----------------------------------------------------
>>> > > >> >>>> # perf test LLVM
>>> > > >> >>>> 35: Test LLVM searching and compiling                        :
>>> > > >> >>>> 35.1: Basic BPF llvm compiling test                          : FAILED!
>>> > > >> >>>> 35.2: Test kbuild searching                                  : Skip
>>> > > >> >>>> 35.3: Compile source for BPF prologue generation test        : Skip
>>> > > >> >>>> 35.4: Compile source for BPF relocation test                 : Skip
>>> > > >> >>>> ----------------------------------------------------
>>> > > >> >>>>
>>> > > >> >>>> When I run with -v I get this:
>>> > > >> >>>>
>>> > > >> >>>> -----------------------------------------------------
>>> > > >> >>>> # perf test -v LLVM
>>> > > >> >>>> 35: Test LLVM searching and compiling                        :
>>> > > >> >>>> 35.1: Basic BPF llvm compiling test                          :
>>> > > >> >>>> --- start ---
>>> > > >> >>>> test child forked, pid 3304
>>> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>>> > > >> >>>> WARNING:        unable to get correct kernel building directory.
>>> > > >> >>>> Hint:   Set correct kbuild directory using 'kbuild-dir' option in [llvm]
>>> > > >> >>>>          section of ~/.perfconfig or set it to "" to suppress kbuild
>>> > > >> >>>>          detection.
>>> > > >> >>>>
>>> > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
>>> > > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
>>> > > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
>>> > > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
>>> > > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
>>> > > >> >>>> -
>>> > > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
>>> > > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
>>> > > >> >>>> test child finished with -1
>>> > > >> >>>> ---- end ----
>>> > > >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
>>> > > >> >>>> -----------------------------------------------------
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>> Basic BPF LLVM compiling test is failing.
>>> > > >> >>>> It reports that bpf could not able to read the kernel version from uname.
>>> > > >> >>>> Is it because of this that 'perf test LLVM' is failing ?
>>> > > >> >>>>
>>> > > >> >>>> My uname says:
>>> > > >> >>>> 4.9-xxxx-amd-x86-64
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
>>> > > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
>>> > > >> >>> report
>>> > > >> >>> kernel version number (from your uname string, sublevel can't be
>>> > > >> >>> determined),
>>> > > >> >>> please help us improve that function like what we have done for ubuntu
>>> > > >> >>> (checking /proc/version_signature).
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>>> My kernel version is: 4.9.20 (from Makefile)
>>> > > >> >>>>
>>> > > >> >>>> So, I dont think there is any problem with kernel version.
>>> > > >> >>>>
>>> > > >> >>>> If any body have come across this problem please let me know how to
>>> > > >> >>>> resolve this issue.
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>> Thank You!
>>> > > >> >>>>
>>> > > >> >>>> Regards,
>>> > > >> >>>> Pintu
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>>
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Masami Hiramatsu <mhiramat@kernel.org>
>>> >
>>> >
>>> > --
>>> > Masami Hiramatsu <mhiramat@kernel.org>
>>
>>
>> --
>> Masami Hiramatsu <mhiramat@kernel.org>