[PATCH 0/6] perf tools: Improve 'make build-test'

[PATCH 0/6] perf tools: Improve 'make build-test'

[Wang Nan]

There are some bugs and inconveniences in 'make build-test', causes
people with strong motivation to avoid writing to source directory
and people don't have plenty of time skip build-test before sending
patches.

Patch 1 - 3 fixes some existing bug. Patch 4 speedup the test.
Patch 5 and 6 makes build-test obey 'O' option passes from cmdline
of the first 'make'.

With these 6 patches I make build-test success in my environment
fully success the first time.

Wang Nan (6):
  perf tools: Add -lutil in python lib list for broken python-config
  perf tools: Add missing headers in perf's MANIFEST
  perf tools: Fix phony build target for build-test
  perf tools: Set parallel making options build-test
  perf tools: Pass O option to Makefile.perf in build-test
  perf tools: Test correct path of perf in build-test

 tools/perf/MANIFEST        |  2 ++
 tools/perf/config/Makefile |  2 +-
 tools/perf/tests/make      | 56 +++++++++++++++++++++++++++++++++-------------
 3 files changed, 43 insertions(+), 17 deletions(-)

-- 
1.8.3.4

[PATCH 1/6] perf tools: Add -lutil in python lib list for broken python-config

[Wang Nan]

On some system the perf-config is broken, causes link failure like this:

 /usr/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_forkpty':
 /opt/wangnan/yocto-build/tmp-eglibc/work/x86_64-oe-linux/python/2.7.3-r0.3.1/Python-2.7.3/./Modules/posixmodule.c:3816: undefined reference to `forkpty'
 /usr/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_openpty':
 /opt/wangnan/yocto-build/tmp-eglibc/work/x86_64-oe-linux/python/2.7.3-r0.3.1/Python-2.7.3/./Modules/posixmodule.c:3756: undefined reference to `openpty'
 collect2: error: ld returned 1 exit status
make[1]: *** [/home/wangnan/kernel-hydrogen/tools/perf/out/perf] Error 1
make: *** [all] Error 2

 $ python-config --libs
 -lpthread -ldl -lpthread -lutil -lm -lpython2.7

In this case a '-lutil' should be appended to -lpython2.7.

(I know we have --start-group and --end-group. I can see them in
command line of collect2 by strace. However it doesn't work. Seems
I have a broken environment?)

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/config/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/config/Makefile b/tools/perf/config/Makefile
index 18b2f96..603ddb4 100644
--- a/tools/perf/config/Makefile
+++ b/tools/perf/config/Makefile
@@ -493,7 +493,7 @@ else
 
       PYTHON_EMBED_LDOPTS := $(shell $(PYTHON_CONFIG_SQ) --ldflags 2>/dev/null)
       PYTHON_EMBED_LDFLAGS := $(call strip-libs,$(PYTHON_EMBED_LDOPTS))
-      PYTHON_EMBED_LIBADD := $(call grep-libs,$(PYTHON_EMBED_LDOPTS))
+      PYTHON_EMBED_LIBADD := $(call grep-libs,$(PYTHON_EMBED_LDOPTS)) -lutil
       PYTHON_EMBED_CCOPTS := $(shell $(PYTHON_CONFIG_SQ) --cflags 2>/dev/null)
       FLAGS_PYTHON_EMBED := $(PYTHON_EMBED_CCOPTS) $(PYTHON_EMBED_LDOPTS)
 
-- 
1.8.3.4

[PATCH 2/6] perf tools: Add missing headers in perf's MANIFEST

[Wang Nan]

These losted headers are found in arm64 cross building:
failed to build perf using tarball generated using

 $ make perf-targz-src-pkg

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/MANIFEST | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/perf/MANIFEST b/tools/perf/MANIFEST
index ce3932e..c0053af 100644
--- a/tools/perf/MANIFEST
+++ b/tools/perf/MANIFEST
@@ -1,6 +1,7 @@
 tools/perf
 tools/arch/alpha/include/asm/barrier.h
 tools/arch/arm/include/asm/barrier.h
+tools/arch/arm64/include/asm/barrier.h
 tools/arch/ia64/include/asm/barrier.h
 tools/arch/mips/include/asm/barrier.h
 tools/arch/powerpc/include/asm/barrier.h
@@ -30,6 +31,7 @@ tools/lib/util/find_next_bit.c
 tools/include/asm/atomic.h
 tools/include/asm/barrier.h
 tools/include/asm/bug.h
+tools/include/asm-generic/atomic-gcc.h
 tools/include/asm-generic/barrier.h
 tools/include/asm-generic/bitops/arch_hweight.h
 tools/include/asm-generic/bitops/atomic.h
-- 
1.8.3.4

[PATCH 3/6] perf tools: Fix phony build target for build-test

[Wang Nan]

make_kernelsrc and make_kernelsrc_tools are skiped if a previous
build-test is done, because 'make build-test' creates two files with
same names. To avoid this, they should be included in .PHONY list.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/make | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/perf/tests/make b/tools/perf/tests/make
index c1fbb8e..130be7c 100644
--- a/tools/perf/tests/make
+++ b/tools/perf/tests/make
@@ -280,5 +280,5 @@ all: $(run) $(run_O) tarpkg make_kernelsrc make_kernelsrc_tools
 out: $(run_O)
 	@echo OK
 
-.PHONY: all $(run) $(run_O) tarpkg clean
+.PHONY: all $(run) $(run_O) tarpkg clean make_kernelsrc make_kernelsrc_tools
 endif # ifndef MK
-- 
1.8.3.4

[PATCH 4/6] perf tools: Set parallel making options build-test

[Wang Nan]

'make build-test' is painful because of time consuming. In a full test,
all test cases are built twice with tools/perf/Makefile and
tools/perf/Makefile.perf. 'Makefile' automatically computes parallel
options for make, but 'Makefile.perf' not, so all test cases is built
with one job. It is very slow.

This patch adds '-j' options to Makefile.perf testing. It computes
parallel building options like what tools/perf/Makefile does, and pass
'-j' option to Makefile.perf test.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/make | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/tools/perf/tests/make b/tools/perf/tests/make
index 130be7c..bd9c61a 100644
--- a/tools/perf/tests/make
+++ b/tools/perf/tests/make
@@ -3,7 +3,7 @@ ifeq ($(MAKECMDGOALS),)
 # no target specified, trigger the whole suite
 all:
 	@echo "Testing Makefile";      $(MAKE) -sf tests/make MK=Makefile
-	@echo "Testing Makefile.perf"; $(MAKE) -sf tests/make MK=Makefile.perf
+	@echo "Testing Makefile.perf"; $(MAKE) -sf tests/make MK=Makefile.perf SET_PARALLEL=1
 else
 # run only specific test over 'Makefile'
 %:
@@ -12,6 +12,15 @@ endif
 else
 PERF := .
 
+PARALLEL_OPT=
+ifeq ($(SET_PARALLEL),1)
+  cores := $(shell (getconf _NPROCESSORS_ONLN || egrep -c '^processor|^CPU[0-9]' /proc/cpuinfo) 2>/dev/null)
+  ifeq ($(cores),0)
+    cores := 1
+  endif
+  PARALLEL_OPT="-j$(cores)"
+endif
+
 include config/Makefile.arch
 
 # FIXME looks like x86 is the only arch running tests ;-)
@@ -238,7 +247,7 @@ clean := @(cd $(PERF); make -s -f $(MK) clean >/dev/null)
 $(run):
 	$(call clean)
 	@TMP_DEST=$$(mktemp -d); \
-	cmd="cd $(PERF) && make -f $(MK) DESTDIR=$$TMP_DEST $($@)"; \
+	cmd="cd $(PERF) && make -f $(MK) $(PARALLEL_OPT) DESTDIR=$$TMP_DEST $($@)"; \
 	echo "- $@: $$cmd" && echo $$cmd > $@ && \
 	( eval $$cmd ) >> $@ 2>&1; \
 	echo "  test: $(call test,$@)" >> $@ 2>&1; \
@@ -249,7 +258,7 @@ $(run_O):
 	$(call clean)
 	@TMP_O=$$(mktemp -d); \
 	TMP_DEST=$$(mktemp -d); \
-	cmd="cd $(PERF) && make -f $(MK) O=$$TMP_O DESTDIR=$$TMP_DEST $($(patsubst %_O,%,$@))"; \
+	cmd="cd $(PERF) && make -f $(MK) $(PARALLEL_OPT) O=$$TMP_O DESTDIR=$$TMP_DEST $($(patsubst %_O,%,$@))"; \
 	echo "- $@: $$cmd" && echo $$cmd > $@ && \
 	( eval $$cmd ) >> $@ 2>&1 && \
 	echo "  test: $(call test_O,$@)" >> $@ 2>&1; \
@@ -263,15 +272,15 @@ tarpkg:
 	rm -f $@
 
 make_kernelsrc:
-	@echo "- make -C <kernelsrc> tools/perf"
+	@echo "- make -C <kernelsrc> $(PARALLEL_OPT) tools/perf"
 	$(call clean); \
-	(make -C ../.. tools/perf) > $@ 2>&1 && \
+	(make -C ../.. $(PARALLEL_OPT) tools/perf) > $@ 2>&1 && \
 	test -x perf && rm -f $@ || (cat $@ ; false)
 
 make_kernelsrc_tools:
-	@echo "- make -C <kernelsrc>/tools perf"
+	@echo "- make -C <kernelsrc>/tools $(PARALLEL_OPT) perf"
 	$(call clean); \
-	(make -C ../../tools perf) > $@ 2>&1 && \
+	(make -C ../../tools $(PARALLEL_OPT) perf) > $@ 2>&1 && \
 	test -x perf && rm -f $@ || (cat $@ ; false)
 
 all: $(run) $(run_O) tarpkg make_kernelsrc make_kernelsrc_tools
-- 
1.8.3.4

[PATCH 5/6] perf tools: Pass O option to Makefile.perf in build-test

[Wang Nan]

Doesn't like tools/perf/Makefile, tools/perf/Makefile.perf obey 'O'
option when it is passed through cmdline only, because of code in
tools/scripts/Makefile.include:

 ifneq ($(O),)
 ifeq ($(origin O), command line)
 	...
 	ABSOLUTE_O := $(shell cd $(O) ; pwd)
 	OUTPUT := $(ABSOLUTE_O)/$(if $(subdir),$(subdir)/)
 endif
 endif

This patch passes 'O' to Makefile.perf through cmdline explicitly
to make it follow O variable during build-test.

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/make | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/tools/perf/tests/make b/tools/perf/tests/make
index bd9c61a..a32615a3 100644
--- a/tools/perf/tests/make
+++ b/tools/perf/tests/make
@@ -3,7 +3,7 @@ ifeq ($(MAKECMDGOALS),)
 # no target specified, trigger the whole suite
 all:
 	@echo "Testing Makefile";      $(MAKE) -sf tests/make MK=Makefile
-	@echo "Testing Makefile.perf"; $(MAKE) -sf tests/make MK=Makefile.perf SET_PARALLEL=1
+	@echo "Testing Makefile.perf"; $(MAKE) -sf tests/make MK=Makefile.perf SET_PARALLEL=1 SET_O=1
 else
 # run only specific test over 'Makefile'
 %:
@@ -11,6 +11,14 @@ else
 endif
 else
 PERF := .
+O_OPT :=
+
+ifneq ($(O),)
+  FULL_O := $(shell readlink -f $(O) || echo $(O))
+  ifeq ($(SET_O),1)
+    O_OPT := 'O=$(FULL_O)'
+  endif
+endif
 
 PARALLEL_OPT=
 ifeq ($(SET_PARALLEL),1)
@@ -247,7 +255,7 @@ clean := @(cd $(PERF); make -s -f $(MK) clean >/dev/null)
 $(run):
 	$(call clean)
 	@TMP_DEST=$$(mktemp -d); \
-	cmd="cd $(PERF) && make -f $(MK) $(PARALLEL_OPT) DESTDIR=$$TMP_DEST $($@)"; \
+	cmd="cd $(PERF) && make -f $(MK) $(PARALLEL_OPT) $(O_OPT) DESTDIR=$$TMP_DEST $($@)"; \
 	echo "- $@: $$cmd" && echo $$cmd > $@ && \
 	( eval $$cmd ) >> $@ 2>&1; \
 	echo "  test: $(call test,$@)" >> $@ 2>&1; \
-- 
1.8.3.4

[PATCH 6/6] perf tools: Test correct path of perf in build-test

[Wang Nan]

If an 'O' is passed to 'make build-test', many 'test -x' and 'test -f'
will fail because perf resides in a different directory. Fix this by
computing PERF_OUT according to 'O' and test correct output files.
For make_kernelsrc and make_kernelsrc_tools, set KBUILD_OUTPUT_DIR
instead because the path is different from others ($(O)/perf vs
 $(O)/tools/perf).

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
---
 tools/perf/tests/make | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/tools/perf/tests/make b/tools/perf/tests/make
index a32615a3..0f5afcb 100644
--- a/tools/perf/tests/make
+++ b/tools/perf/tests/make
@@ -11,10 +11,12 @@ else
 endif
 else
 PERF := .
+PERF_OUT := $(PERF)
 O_OPT :=
 
 ifneq ($(O),)
   FULL_O := $(shell readlink -f $(O) || echo $(O))
+  PERF_OUT := $(FULL_O)
   ifeq ($(SET_O),1)
     O_OPT := 'O=$(FULL_O)'
   endif
@@ -159,11 +161,11 @@ test_make_doc    := $(test_ok)
 test_make_help_O := $(test_ok)
 test_make_doc_O  := $(test_ok)
 
-test_make_python_perf_so := test -f $(PERF)/python/perf.so
+test_make_python_perf_so := test -f $(PERF_OUT)/python/perf.so
 
-test_make_perf_o           := test -f $(PERF)/perf.o
-test_make_util_map_o       := test -f $(PERF)/util/map.o
-test_make_util_pmu_bison_o := test -f $(PERF)/util/pmu-bison.o
+test_make_perf_o           := test -f $(PERF_OUT)/perf.o
+test_make_util_map_o       := test -f $(PERF_OUT)/util/map.o
+test_make_util_pmu_bison_o := test -f $(PERF_OUT)/util/pmu-bison.o
 
 define test_dest_files
   for file in $(1); do				\
@@ -230,7 +232,7 @@ test_make_perf_o_O            := test -f $$TMP_O/perf.o
 test_make_util_map_o_O        := test -f $$TMP_O/util/map.o
 test_make_util_pmu_bison_o_O := test -f $$TMP_O/util/pmu-bison.o
 
-test_default = test -x $(PERF)/perf
+test_default = test -x $(PERF_OUT)/perf
 test = $(if $(test_$1),$(test_$1),$(test_default))
 
 test_default_O = test -x $$TMP_O/perf
@@ -250,7 +252,7 @@ endif
 
 MAKEFLAGS := --no-print-directory
 
-clean := @(cd $(PERF); make -s -f $(MK) clean >/dev/null)
+clean := @(cd $(PERF); make -s -f $(MK) O=$(PERF_OUT) clean >/dev/null; make -s -f $(MK) clean >/dev/null)
 
 $(run):
 	$(call clean)
@@ -279,17 +281,22 @@ tarpkg:
 	( eval $$cmd ) >> $@ 2>&1 && \
 	rm -f $@
 
+KBUILD_OUTPUT_DIR := ../..
+ifneq ($(O),)
+  KBUILD_OUTPUT_DIR := $(O)
+endif
+
 make_kernelsrc:
 	@echo "- make -C <kernelsrc> $(PARALLEL_OPT) tools/perf"
 	$(call clean); \
 	(make -C ../.. $(PARALLEL_OPT) tools/perf) > $@ 2>&1 && \
-	test -x perf && rm -f $@ || (cat $@ ; false)
+	test -x $(KBUILD_OUTPUT_DIR)/tools/perf && rm -f $@ || (cat $@ ; false)
 
 make_kernelsrc_tools:
 	@echo "- make -C <kernelsrc>/tools $(PARALLEL_OPT) perf"
 	$(call clean); \
 	(make -C ../../tools $(PARALLEL_OPT) perf) > $@ 2>&1 && \
-	test -x perf && rm -f $@ || (cat $@ ; false)
+	test -x $(KBUILD_OUTPUT_DIR)/tools/perf && rm -f $@ || (cat $@ ; false)
 
 all: $(run) $(run_O) tarpkg make_kernelsrc make_kernelsrc_tools
 	@echo OK
-- 
1.8.3.4

Re: [PATCH 0/6] perf tools: Improve 'make build-test'

[Arnaldo Carvalho de Melo]

Em Fri, Jan 08, 2016 at 02:23:55PM +0000, Wang Nan escreveu:
> There are some bugs and inconveniences in 'make build-test', causes
> people with strong motivation to avoid writing to source directory
> and people don't have plenty of time skip build-test before sending
> patches.
> 
> Patch 1 - 3 fixes some existing bug. Patch 4 speedup the test.
> Patch 5 and 6 makes build-test obey 'O' option passes from cmdline
> of the first 'make'.
> 
> With these 6 patches I make build-test success in my environment
> fully success the first time.

Hey, I owe you some beer or other stuff you like first time we meet, now
waiting for Jiri to chime in...

Thanks!

- Arnaldo
 
> Wang Nan (6):
>   perf tools: Add -lutil in python lib list for broken python-config
>   perf tools: Add missing headers in perf's MANIFEST
>   perf tools: Fix phony build target for build-test
>   perf tools: Set parallel making options build-test
>   perf tools: Pass O option to Makefile.perf in build-test
>   perf tools: Test correct path of perf in build-test
> 
>  tools/perf/MANIFEST        |  2 ++
>  tools/perf/config/Makefile |  2 +-
>  tools/perf/tests/make      | 56 +++++++++++++++++++++++++++++++++-------------
>  3 files changed, 43 insertions(+), 17 deletions(-)
> 
> -- 
> 1.8.3.4

[tip:perf/core] perf tools: Add missing headers in perf' s MANIFEST

[tip-bot for Wang Nan]

Commit-ID:  2d7c03e6b0c604decae33b0ce03e69b79b2a39a1
Gitweb:     http://git.kernel.org/tip/2d7c03e6b0c604decae33b0ce03e69b79b2a39a1
Author:     Wang Nan <wangnan0@huawei.com>
AuthorDate: Fri, 8 Jan 2016 14:23:57 +0000
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Fri, 8 Jan 2016 12:48:40 -0300

perf tools: Add missing headers in perf's MANIFEST

These lost headers are found in arm64 cross buildings, failing to build
perf using tarballs generated using:

 $ make perf-targz-src-pkg

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1452263041-225488-3-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/MANIFEST | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/perf/MANIFEST b/tools/perf/MANIFEST
index b3db8df..ddf922f 100644
--- a/tools/perf/MANIFEST
+++ b/tools/perf/MANIFEST
@@ -1,6 +1,7 @@
 tools/perf
 tools/arch/alpha/include/asm/barrier.h
 tools/arch/arm/include/asm/barrier.h
+tools/arch/arm64/include/asm/barrier.h
 tools/arch/ia64/include/asm/barrier.h
 tools/arch/mips/include/asm/barrier.h
 tools/arch/powerpc/include/asm/barrier.h
@@ -30,6 +31,7 @@ tools/lib/find_bit.c
 tools/include/asm/atomic.h
 tools/include/asm/barrier.h
 tools/include/asm/bug.h
+tools/include/asm-generic/atomic-gcc.h
 tools/include/asm-generic/barrier.h
 tools/include/asm-generic/bitops/arch_hweight.h
 tools/include/asm-generic/bitops/atomic.h

Re: [PATCH 0/6] perf tools: Improve 'make build-test'

[Wangnan (F)]

Hi Jiri,

I think if we can fold the feature checking in build-test then
it can be even faster. We need only 2 feature check result (for
normal building and static building), but in a full build-test
we run 138 rounds feature checking. Would you please give some
help on it?

Now I have a 160 cores server for building perf, currently busy
time and idle time are 5:3 (result from top. I use 1s interval.
Each test case takes about 8s,  in about 5 seconds I can see
full of 'cc1' in perf top result). A full test-build still require
more than 0.3 hours (8s * 34 cases * 2 (with and without O) *
2 (Makefile and Makefile.perf) = 1088 seconds).

Thank you.



On 2016/1/8 22:32, Arnaldo Carvalho de Melo wrote:
> Em Fri, Jan 08, 2016 at 02:23:55PM +0000, Wang Nan escreveu:
>> There are some bugs and inconveniences in 'make build-test', causes
>> people with strong motivation to avoid writing to source directory
>> and people don't have plenty of time skip build-test before sending
>> patches.
>>
>> Patch 1 - 3 fixes some existing bug. Patch 4 speedup the test.
>> Patch 5 and 6 makes build-test obey 'O' option passes from cmdline
>> of the first 'make'.
>>
>> With these 6 patches I make build-test success in my environment
>> fully success the first time.
> Hey, I owe you some beer or other stuff you like first time we meet, now
> waiting for Jiri to chime in...
>
> Thanks!
>
> - Arnaldo
>   
>> Wang Nan (6):
>>    perf tools: Add -lutil in python lib list for broken python-config
>>    perf tools: Add missing headers in perf's MANIFEST
>>    perf tools: Fix phony build target for build-test
>>    perf tools: Set parallel making options build-test
>>    perf tools: Pass O option to Makefile.perf in build-test
>>    perf tools: Test correct path of perf in build-test
>>
>>   tools/perf/MANIFEST        |  2 ++
>>   tools/perf/config/Makefile |  2 +-
>>   tools/perf/tests/make      | 56 +++++++++++++++++++++++++++++++++-------------
>>   3 files changed, 43 insertions(+), 17 deletions(-)
>>
>> -- 
>> 1.8.3.4

Re: [PATCH 0/6] perf tools: Improve 'make build-test'

[Jiri Olsa]

On Mon, Jan 11, 2016 at 10:55:50AM +0800, Wangnan (F) wrote:
> Hi Jiri,
> 
> I think if we can fold the feature checking in build-test then
> it can be even faster. We need only 2 feature check result (for
> normal building and static building), but in a full build-test
> we run 138 rounds feature checking. Would you please give some
> help on it?

I think we can add separate target that will skip the feature
detection, given that there's extra call before generating
FEATURE-DUMP file.. I'll check on this

> 
> Now I have a 160 cores server for building perf, currently busy
> time and idle time are 5:3 (result from top. I use 1s interval.
> Each test case takes about 8s,  in about 5 seconds I can see
> full of 'cc1' in perf top result). A full test-build still require
> more than 0.3 hours (8s * 34 cases * 2 (with and without O) *
> 2 (Makefile and Makefile.perf) = 1088 seconds).

nice

thanks,
jirka

Re: [PATCH 1/6] perf tools: Add -lutil in python lib list for broken python-config

[Jiri Olsa]

On Fri, Jan 08, 2016 at 02:23:56PM +0000, Wang Nan wrote:
> On some system the perf-config is broken, causes link failure like this:
> 
>  /usr/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_forkpty':
>  /opt/wangnan/yocto-build/tmp-eglibc/work/x86_64-oe-linux/python/2.7.3-r0.3.1/Python-2.7.3/./Modules/posixmodule.c:3816: undefined reference to `forkpty'
>  /usr/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_openpty':
>  /opt/wangnan/yocto-build/tmp-eglibc/work/x86_64-oe-linux/python/2.7.3-r0.3.1/Python-2.7.3/./Modules/posixmodule.c:3756: undefined reference to `openpty'
>  collect2: error: ld returned 1 exit status
> make[1]: *** [/home/wangnan/kernel-hydrogen/tools/perf/out/perf] Error 1
> make: *** [all] Error 2
> 
>  $ python-config --libs
>  -lpthread -ldl -lpthread -lutil -lm -lpython2.7

so your 'python-config --libs' shows -lutil and you still get the build failure?

after your fix I can see double -lutil in PYTHON_EMBED_LIBADD,
which is probably not a problem but could be ommited I think

> 
> In this case a '-lutil' should be appended to -lpython2.7.
> 
> (I know we have --start-group and --end-group. I can see them in
> command line of collect2 by strace. However it doesn't work. Seems
> I have a broken environment?)

hum, how is this related to start/end -group options?

thanks,
jirka

Re: [PATCH 3/6] perf tools: Fix phony build target for build-test

[Jiri Olsa]

On Fri, Jan 08, 2016 at 02:23:58PM +0000, Wang Nan wrote:
> make_kernelsrc and make_kernelsrc_tools are skiped if a previous
> build-test is done, because 'make build-test' creates two files with
> same names. To avoid this, they should be included in .PHONY list.
> 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Namhyung Kim <namhyung@kernel.org>

Acked-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

> ---
>  tools/perf/tests/make | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tools/perf/tests/make b/tools/perf/tests/make
> index c1fbb8e..130be7c 100644
> --- a/tools/perf/tests/make
> +++ b/tools/perf/tests/make
> @@ -280,5 +280,5 @@ all: $(run) $(run_O) tarpkg make_kernelsrc make_kernelsrc_tools
>  out: $(run_O)
>  	@echo OK
>  
> -.PHONY: all $(run) $(run_O) tarpkg clean
> +.PHONY: all $(run) $(run_O) tarpkg clean make_kernelsrc make_kernelsrc_tools
>  endif # ifndef MK
> -- 
> 1.8.3.4
> 

Re: [PATCH 4/6] perf tools: Set parallel making options build-test

[Jiri Olsa]

On Fri, Jan 08, 2016 at 02:23:59PM +0000, Wang Nan wrote:
> 'make build-test' is painful because of time consuming. In a full test,
> all test cases are built twice with tools/perf/Makefile and
> tools/perf/Makefile.perf. 'Makefile' automatically computes parallel
> options for make, but 'Makefile.perf' not, so all test cases is built
> with one job. It is very slow.
> 
> This patch adds '-j' options to Makefile.perf testing. It computes
> parallel building options like what tools/perf/Makefile does, and pass
> '-j' option to Makefile.perf test.
> 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/tests/make | 23 ++++++++++++++++-------
>  1 file changed, 16 insertions(+), 7 deletions(-)
> 
> diff --git a/tools/perf/tests/make b/tools/perf/tests/make
> index 130be7c..bd9c61a 100644
> --- a/tools/perf/tests/make
> +++ b/tools/perf/tests/make
> @@ -3,7 +3,7 @@ ifeq ($(MAKECMDGOALS),)
>  # no target specified, trigger the whole suite
>  all:
>  	@echo "Testing Makefile";      $(MAKE) -sf tests/make MK=Makefile
> -	@echo "Testing Makefile.perf"; $(MAKE) -sf tests/make MK=Makefile.perf
> +	@echo "Testing Makefile.perf"; $(MAKE) -sf tests/make MK=Makefile.perf SET_PARALLEL=1
>  else
>  # run only specific test over 'Makefile'
>  %:
> @@ -12,6 +12,15 @@ endif
>  else
>  PERF := .
>  
> +PARALLEL_OPT=
> +ifeq ($(SET_PARALLEL),1)
> +  cores := $(shell (getconf _NPROCESSORS_ONLN || egrep -c '^processor|^CPU[0-9]' /proc/cpuinfo) 2>/dev/null)
> +  ifeq ($(cores),0)
> +    cores := 1
> +  endif
> +  PARALLEL_OPT="-j$(cores)"
> +endif

we could share this with Makefile, but there's probably no point to librarize single line ATM

Acked-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

[PATCH] perf tools: Add missing sources in perf's MANIFEST

[Jiri Olsa]

On Fri, Jan 08, 2016 at 02:23:57PM +0000, Wang Nan wrote:
> These losted headers are found in arm64 cross building:
> failed to build perf using tarball generated using
> 
>  $ make perf-targz-src-pkg
> 
> Signed-off-by: Wang Nan <wangnan0@huawei.com>
> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> Cc: Jiri Olsa <jolsa@kernel.org>
> Cc: Namhyung Kim <namhyung@kernel.org>
> ---
>  tools/perf/MANIFEST | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/tools/perf/MANIFEST b/tools/perf/MANIFEST
> index ce3932e..c0053af 100644
> --- a/tools/perf/MANIFEST
> +++ b/tools/perf/MANIFEST
> @@ -1,6 +1,7 @@
>  tools/perf
>  tools/arch/alpha/include/asm/barrier.h
>  tools/arch/arm/include/asm/barrier.h
> +tools/arch/arm64/include/asm/barrier.h
>  tools/arch/ia64/include/asm/barrier.h
>  tools/arch/mips/include/asm/barrier.h
>  tools/arch/powerpc/include/asm/barrier.h
> @@ -30,6 +31,7 @@ tools/lib/util/find_next_bit.c
>  tools/include/asm/atomic.h
>  tools/include/asm/barrier.h
>  tools/include/asm/bug.h
> +tools/include/asm-generic/atomic-gcc.h
>  tools/include/asm-generic/barrier.h
>  tools/include/asm-generic/bitops/arch_hweight.h
>  tools/include/asm-generic/bitops/atomic.h

tarpkg test just failed for me due to missing bitmap sources
I needed to add.. seems related to:
  915b0882c310 tools lib: Move bitmap.[ch] from tools/perf/ to tools/{lib,include}/


thanks,
jirka


---
Adding missing bitmap.[ch] sources to the MINIFEST file.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
diff --git a/tools/perf/MANIFEST b/tools/perf/MANIFEST
index ddf922f93aa1..2e1fa2357528 100644
--- a/tools/perf/MANIFEST
+++ b/tools/perf/MANIFEST
@@ -28,6 +28,7 @@ tools/lib/string.c
 tools/lib/symbol/kallsyms.c
 tools/lib/symbol/kallsyms.h
 tools/lib/find_bit.c
+tools/lib/bitmap.c
 tools/include/asm/atomic.h
 tools/include/asm/barrier.h
 tools/include/asm/bug.h
@@ -57,6 +58,7 @@ tools/include/linux/rbtree_augmented.h
 tools/include/linux/string.h
 tools/include/linux/types.h
 tools/include/linux/err.h
+tools/include/linux/bitmap.h
 include/asm-generic/bitops/arch_hweight.h
 include/asm-generic/bitops/const_hweight.h
 include/asm-generic/bitops/fls64.h

Re: [PATCH 1/6] perf tools: Add -lutil in python lib list for broken python-config

[Wangnan (F)]

On 2016/1/11 17:28, Jiri Olsa wrote:
> On Fri, Jan 08, 2016 at 02:23:56PM +0000, Wang Nan wrote:
>> On some system the perf-config is broken, causes link failure like this:
>>
>>   /usr/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_forkpty':
>>   /opt/wangnan/yocto-build/tmp-eglibc/work/x86_64-oe-linux/python/2.7.3-r0.3.1/Python-2.7.3/./Modules/posixmodule.c:3816: undefined reference to `forkpty'
>>   /usr/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_openpty':
>>   /opt/wangnan/yocto-build/tmp-eglibc/work/x86_64-oe-linux/python/2.7.3-r0.3.1/Python-2.7.3/./Modules/posixmodule.c:3756: undefined reference to `openpty'
>>   collect2: error: ld returned 1 exit status
>> make[1]: *** [/home/wangnan/kernel-hydrogen/tools/perf/out/perf] Error 1
>> make: *** [all] Error 2
>>
>>   $ python-config --libs
>>   -lpthread -ldl -lpthread -lutil -lm -lpython2.7
> so your 'python-config --libs' shows -lutil and you still get the build failure?
>
> after your fix I can see double -lutil in PYTHON_EMBED_LIBADD,
> which is probably not a problem but could be ommited I think

In my python-config:

  # python-config
-lpthread -ldl -lpthread -lutil -lm -lpython2.7

There are multiple -lpthread already. In fact multiple '-l' options is not
a problem at all. Please see below.

>> In this case a '-lutil' should be appended to -lpython2.7.
>>
>> (I know we have --start-group and --end-group. I can see them in
>> command line of collect2 by strace. However it doesn't work. Seems
>> I have a broken environment?)
> hum, how is this related to start/end -group options?

Please check man page of ld:

<quote>
--start-group archives --end-group
     The archives should be a list of archive files. They may be either 
explicit file names, or -l options.

     The specified archives are searched repeatedly until no new 
undefined references are created. Normally, an archive is searched only 
once in the order that it is specified on the command line. If a symbol 
in that archive is needed to resolve an undefined symbol referred to by 
an object in an archive that appears later on the command line, the 
linker would not be able to resolve that reference. By grouping the 
archives, they all be searched repeatedly until all possible references 
are resolved.

     Using this option has a significant performance cost. It is best to 
use it only when there are unavoidable circular references between two 
or more archives.
</quote>

Without --start/end-group, if we provide -l in this order:

  ... -la -lb -lc ...

and libb.a requires a symbol in liba.a, link would fail. The simplest way
to avoid this is to add another -la after -lb:

  ... -la -lb -la -lc ...

In my environment it works as if we don't provide -Wl--start-group and 
-Wl,--end-group.
I don't know why.

Thank you.

Re: [PATCH] perf tools: Add missing sources in perf's MANIFEST

[Wangnan (F)]

On 2016/1/11 17:43, Jiri Olsa wrote:
> On Fri, Jan 08, 2016 at 02:23:57PM +0000, Wang Nan wrote:
>> These losted headers are found in arm64 cross building:
>> failed to build perf using tarball generated using
>>
>>   $ make perf-targz-src-pkg
>>
>> Signed-off-by: Wang Nan <wangnan0@huawei.com>
>> Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
>> Cc: Jiri Olsa <jolsa@kernel.org>
>> Cc: Namhyung Kim <namhyung@kernel.org>
>> ---
>>   tools/perf/MANIFEST | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/tools/perf/MANIFEST b/tools/perf/MANIFEST
>> index ce3932e..c0053af 100644
>> --- a/tools/perf/MANIFEST
>> +++ b/tools/perf/MANIFEST
>> @@ -1,6 +1,7 @@
>>   tools/perf
>>   tools/arch/alpha/include/asm/barrier.h
>>   tools/arch/arm/include/asm/barrier.h
>> +tools/arch/arm64/include/asm/barrier.h
>>   tools/arch/ia64/include/asm/barrier.h
>>   tools/arch/mips/include/asm/barrier.h
>>   tools/arch/powerpc/include/asm/barrier.h
>> @@ -30,6 +31,7 @@ tools/lib/util/find_next_bit.c
>>   tools/include/asm/atomic.h
>>   tools/include/asm/barrier.h
>>   tools/include/asm/bug.h
>> +tools/include/asm-generic/atomic-gcc.h
>>   tools/include/asm-generic/barrier.h
>>   tools/include/asm-generic/bitops/arch_hweight.h
>>   tools/include/asm-generic/bitops/atomic.h
> tarpkg test just failed for me due to missing bitmap sources
> I needed to add.. seems related to:
>    915b0882c310 tools lib: Move bitmap.[ch] from tools/perf/ to tools/{lib,include}/
>

I also saw this:

http://lkml.kernel.org/g/56931E12.8030506@huawei.com

tarpkg works for me when I posted this patchset at Friday, but
fail after I return back today...

Thank you.

Re: [PATCH 1/6] perf tools: Add -lutil in python lib list for broken python-config

[Wangnan (F)]

On 2016/1/11 17:44, Wangnan (F) wrote:
>
>
> On 2016/1/11 17:28, Jiri Olsa wrote:
>> On Fri, Jan 08, 2016 at 02:23:56PM +0000, Wang Nan wrote:
>>> On some system the perf-config is broken, causes link failure like 
>>> this:
>>>
>>>   /usr/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In 
>>> function `posix_forkpty':
>>> /opt/wangnan/yocto-build/tmp-eglibc/work/x86_64-oe-linux/python/2.7.3-r0.3.1/Python-2.7.3/./Modules/posixmodule.c:3816: 
>>> undefined reference to `forkpty'
>>>   /usr/lib64/python2.7/config/libpython2.7.a(posixmodule.o): In 
>>> function `posix_openpty':
>>> /opt/wangnan/yocto-build/tmp-eglibc/work/x86_64-oe-linux/python/2.7.3-r0.3.1/Python-2.7.3/./Modules/posixmodule.c:3756: 
>>> undefined reference to `openpty'
>>>   collect2: error: ld returned 1 exit status
>>> make[1]: *** [/home/wangnan/kernel-hydrogen/tools/perf/out/perf] 
>>> Error 1
>>> make: *** [all] Error 2
>>>
>>>   $ python-config --libs
>>>   -lpthread -ldl -lpthread -lutil -lm -lpython2.7
>> so your 'python-config --libs' shows -lutil and you still get the 
>> build failure?
>>
>> after your fix I can see double -lutil in PYTHON_EMBED_LIBADD,
>> which is probably not a problem but could be ommited I think
>
> In my python-config:
>
>  # python-config
> -lpthread -ldl -lpthread -lutil -lm -lpython2.7
>
> There are multiple -lpthread already. In fact multiple '-l' options is 
> not
> a problem at all. Please see below.
>
>>> In this case a '-lutil' should be appended to -lpython2.7.
>>>
>>> (I know we have --start-group and --end-group. I can see them in
>>> command line of collect2 by strace. However it doesn't work. Seems
>>> I have a broken environment?)
>> hum, how is this related to start/end -group options?
>
> Please check man page of ld:
>
> <quote>
> --start-group archives --end-group
>     The archives should be a list of archive files. They may be either 
> explicit file names, or -l options.
>
>     The specified archives are searched repeatedly until no new 
> undefined references are created. Normally, an archive is searched 
> only once in the order that it is specified on the command line. If a 
> symbol in that archive is needed to resolve an undefined symbol 
> referred to by an object in an archive that appears later on the 
> command line, the linker would not be able to resolve that reference. 
> By grouping the archives, they all be searched repeatedly until all 
> possible references are resolved.
>
>     Using this option has a significant performance cost. It is best 
> to use it only when there are unavoidable circular references between 
> two or more archives.
> </quote>
>
> Without --start/end-group, if we provide -l in this order:
>
>  ... -la -lb -lc ...
>
> and libb.a requires a symbol in liba.a, link would fail. The simplest way
> to avoid this is to add another -la after -lb:
>
>  ... -la -lb -la -lc ...
>
> In my environment it works as if we don't provide -Wl,--start-group 
> and -Wl,--end-group.
> I don't know why.
>

I mean the python case before I add -lutil after python libs. 
-Wl,--start-group and -Wl,--end-group
seems not work for me. They are issued, I confirm this in strace output.

Thank you.

> Thank you.

Re: [PATCH] perf tools: Add missing sources in perf's MANIFEST

[Arnaldo Carvalho de Melo]

Em Mon, Jan 11, 2016 at 05:46:10PM +0800, Wangnan (F) escreveu:
> On 2016/1/11 17:43, Jiri Olsa wrote:
> >On Fri, Jan 08, 2016 at 02:23:57PM +0000, Wang Nan wrote:
> >>These losted headers are found in arm64 cross building:
> >>failed to build perf using tarball generated using
> >>
> >>  $ make perf-targz-src-pkg
> >>
> >>Signed-off-by: Wang Nan <wangnan0@huawei.com>
> >>Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
> >>Cc: Jiri Olsa <jolsa@kernel.org>
> >>Cc: Namhyung Kim <namhyung@kernel.org>
> >>---
> >>  tools/perf/MANIFEST | 2 ++
> >>  1 file changed, 2 insertions(+)
> >>
> >>diff --git a/tools/perf/MANIFEST b/tools/perf/MANIFEST
> >>index ce3932e..c0053af 100644
> >>--- a/tools/perf/MANIFEST
> >>+++ b/tools/perf/MANIFEST
> >>@@ -1,6 +1,7 @@
> >>  tools/perf
> >>  tools/arch/alpha/include/asm/barrier.h
> >>  tools/arch/arm/include/asm/barrier.h
> >>+tools/arch/arm64/include/asm/barrier.h
> >>  tools/arch/ia64/include/asm/barrier.h
> >>  tools/arch/mips/include/asm/barrier.h
> >>  tools/arch/powerpc/include/asm/barrier.h
> >>@@ -30,6 +31,7 @@ tools/lib/util/find_next_bit.c
> >>  tools/include/asm/atomic.h
> >>  tools/include/asm/barrier.h
> >>  tools/include/asm/bug.h
> >>+tools/include/asm-generic/atomic-gcc.h
> >>  tools/include/asm-generic/barrier.h
> >>  tools/include/asm-generic/bitops/arch_hweight.h
> >>  tools/include/asm-generic/bitops/atomic.h
> >tarpkg test just failed for me due to missing bitmap sources
> >I needed to add.. seems related to:
> >   915b0882c310 tools lib: Move bitmap.[ch] from tools/perf/ to tools/{lib,include}/
 
> I also saw this:
 
> http://lkml.kernel.org/g/56931E12.8030506@huawei.com
 
> tarpkg works for me when I posted this patchset at Friday, but
> fail after I return back today...

Right, my turn, we all make mistakes ;-\

Anyway, I'm looking at git commit hooks to try and get this checked
automatically for future renames or when new files gets added that are
not listed in the MANIFEST file...

Thank you guys for spotting this! :-)

- Arnaldo

Re: [PATCH 1/6] perf tools: Add -lutil in python lib list for broken python-config

[Jiri Olsa]

On Mon, Jan 11, 2016 at 05:50:21PM +0800, Wangnan (F) wrote:

SNIP

> >
> >    Using this option has a significant performance cost. It is best to
> >use it only when there are unavoidable circular references between two or
> >more archives.
> ></quote>
> >
> >Without --start/end-group, if we provide -l in this order:
> >
> > ... -la -lb -lc ...
> >
> >and libb.a requires a symbol in liba.a, link would fail. The simplest way
> >to avoid this is to add another -la after -lb:
> >
> > ... -la -lb -la -lc ...
> >
> >In my environment it works as if we don't provide -Wl,--start-group and
> >-Wl,--end-group.
> >I don't know why.
> >
> 
> I mean the python case before I add -lutil after python libs.
> -Wl,--start-group and -Wl,--end-group
> seems not work for me. They are issued, I confirm this in strace output.

ok, got it.. I'm ok with the patch, there's no harm with extra lib,
however I think you should report this to gcc folks

also please resubmit the patch with the above comment,
to make it clear why it's there

thanks,
jirka