[PATCH] Enable ptest support for lttng.

[PATCH] Enable ptest support for lttng.

[Stefan Seefeld]

This patch adds the necessary glue to bind the LTTng test suite to the OE ptest logic.

The patch consists of three parts:

1) The ptest enabling bits, including a new install-ptest target and a run-ptest script.
2) A patch (also filed upstream) to execute the tests also in the 'installcheck' target,
   instead of only the 'check' target.
3) Some test output post-processing to obey the ptest format rules.

Stefan Seefeld (1):
  Enable ptest support for lttng.

 meta/recipes-kernel/lttng/lttng-tools/run-ptest    |  4 ++
 .../recipes-kernel/lttng/lttng-tools/runtest.patch | 52 ++++++++++++++++++++++
 meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb     |  8 +++-
 3 files changed, 62 insertions(+), 2 deletions(-)
 create mode 100755 meta/recipes-kernel/lttng/lttng-tools/run-ptest
 create mode 100644 meta/recipes-kernel/lttng/lttng-tools/runtest.patch

-- 
1.8.3.1

[PATCH] Enable ptest support for lttng.

[Stefan Seefeld]

Signed-off-by: Stefan Seefeld <stefan_seefeld@mentor.com>
---
 meta/recipes-kernel/lttng/lttng-tools/run-ptest    |  4 ++
 .../recipes-kernel/lttng/lttng-tools/runtest.patch | 52 ++++++++++++++++++++++
 meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb     |  8 +++-
 3 files changed, 62 insertions(+), 2 deletions(-)
 create mode 100755 meta/recipes-kernel/lttng/lttng-tools/run-ptest
 create mode 100644 meta/recipes-kernel/lttng/lttng-tools/runtest.patch

diff --git a/meta/recipes-kernel/lttng/lttng-tools/run-ptest b/meta/recipes-kernel/lttng/lttng-tools/run-ptest
new file mode 100755
index 0000000..e758815
--- /dev/null
+++ b/meta/recipes-kernel/lttng/lttng-tools/run-ptest
@@ -0,0 +1,4 @@
+#!/bin/sh
+
+make -C tests installcheck
+
diff --git a/meta/recipes-kernel/lttng/lttng-tools/runtest.patch b/meta/recipes-kernel/lttng/lttng-tools/runtest.patch
new file mode 100644
index 0000000..545a5dc
--- /dev/null
+++ b/meta/recipes-kernel/lttng/lttng-tools/runtest.patch
@@ -0,0 +1,52 @@
+diff --git a/Makefile.am b/Makefile.am
+index 584f59b..c2bcabd 100644
+--- a/Makefile.am
++++ b/Makefile.am
+@@ -12,3 +12,9 @@ dist_doc_DATA = LICENSE \
+ dist_noinst_DATA = CodingStyle
+ 
+ EXTRA_DIST = extras/lttng-bash_completion gpl-2.0.txt lgpl-2.1.txt
++
++install-ptest:
++	cp -r $(srcdir)/tests $(DESTDIR)
++	for m in $$(find $(DESTDIR)/tests -name Makefile); do \
++          sed -i -e 's|^Makefile:|_Makefile:|' $$m; \
++        done
+diff --git a/tests/Makefile.am b/tests/Makefile.am
+index 6d5b00d..3774f9d 100644
+--- a/tests/Makefile.am
++++ b/tests/Makefile.am
+@@ -1,14 +1,17 @@
+ SUBDIRS = utils regression unit stress
+ 
+-if USE_PYTHON
+-check-am:
++installcheck-am:
+ 	./run.sh unit_tests
+ 	./run.sh fast_regression
++if USE_PYTHON
+ 	./run.sh with_bindings_regression
+-else
++endif
++
+ check-am:
+ 	./run.sh unit_tests
+ 	./run.sh fast_regression
++if USE_PYTHON
++	./run.sh with_bindings_regression
+ endif
+ 
+ dist_noinst_SCRIPTS = run.sh unit_tests fast_regression long_regression root_regression with_bindings_regression
+diff --git a/tests/run.sh b/tests/run.sh
+index c6c50fd..6455359 100755
+--- a/tests/run.sh
++++ b/tests/run.sh
+@@ -19,4 +19,7 @@
+ 
+ [ -z "$1" ] && echo "Error: No testlist. Please specify a testlist to run." && exit 1
+ 
+-prove --merge --exec '' - < $1
++prove --merge -v --exec '' - < $1 | sed \
++  -e 's|^ok \(.*\)|PASS: \1|' \
++  -e 's|^not ok \(.*\)|FAIL: \1|' \
++  | egrep -h 'PASS|FAIL'
diff --git a/meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb b/meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb
index 2e0ee15..4322027 100644
--- a/meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb
+++ b/meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb
@@ -10,15 +10,19 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=01d7fc4496aacf37d90df90b90b0cac1 \
                     file://lgpl-2.1.txt;md5=0f0d71500e6a57fd24d825f33242b9ca"
 
 DEPENDS = "liburcu popt lttng-ust"
+RDEPENDS_${PN}-ptest += "make"
 
 SRCREV = "c9dc1289e040c542f96fbfd558267786816d5703"
 PV = "v2.3.0"
 
-SRC_URI = "git://git.lttng.org/lttng-tools.git"
+SRC_URI = "git://git.lttng.org/lttng-tools.git \
+           file://runtest.patch \
+           file://run-ptest \
+	  "
 
 S = "${WORKDIR}/git"
 
-inherit autotools
+inherit autotools ptest
 
 export KERNELDIR="${STAGING_KERNEL_DIR}"
 
-- 
1.8.3.1

Re: [PATCH] Enable ptest support for lttng.

[Saul Wold]

The usual format for the commit summary is as follows:

lttng: Enable ptest support

Please see the Commit/Patch message guideline for details: 
http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines

Thanks
	Sau!

On 09/11/2013 06:09 PM, Stefan Seefeld wrote:
> Signed-off-by: Stefan Seefeld <stefan_seefeld@mentor.com>
> ---
>   meta/recipes-kernel/lttng/lttng-tools/run-ptest    |  4 ++
>   .../recipes-kernel/lttng/lttng-tools/runtest.patch | 52 ++++++++++++++++++++++
>   meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb     |  8 +++-
>   3 files changed, 62 insertions(+), 2 deletions(-)
>   create mode 100755 meta/recipes-kernel/lttng/lttng-tools/run-ptest
>   create mode 100644 meta/recipes-kernel/lttng/lttng-tools/runtest.patch
>
> diff --git a/meta/recipes-kernel/lttng/lttng-tools/run-ptest b/meta/recipes-kernel/lttng/lttng-tools/run-ptest
> new file mode 100755
> index 0000000..e758815
> --- /dev/null
> +++ b/meta/recipes-kernel/lttng/lttng-tools/run-ptest
> @@ -0,0 +1,4 @@
> +#!/bin/sh
> +
> +make -C tests installcheck
> +
> diff --git a/meta/recipes-kernel/lttng/lttng-tools/runtest.patch b/meta/recipes-kernel/lttng/lttng-tools/runtest.patch
> new file mode 100644
> index 0000000..545a5dc
> --- /dev/null
> +++ b/meta/recipes-kernel/lttng/lttng-tools/runtest.patch
> @@ -0,0 +1,52 @@
> +diff --git a/Makefile.am b/Makefile.am
> +index 584f59b..c2bcabd 100644
> +--- a/Makefile.am
> ++++ b/Makefile.am
> +@@ -12,3 +12,9 @@ dist_doc_DATA = LICENSE \
> + dist_noinst_DATA = CodingStyle
> +
> + EXTRA_DIST = extras/lttng-bash_completion gpl-2.0.txt lgpl-2.1.txt
> ++
> ++install-ptest:
> ++	cp -r $(srcdir)/tests $(DESTDIR)
> ++	for m in $$(find $(DESTDIR)/tests -name Makefile); do \
> ++          sed -i -e 's|^Makefile:|_Makefile:|' $$m; \
> ++        done
> +diff --git a/tests/Makefile.am b/tests/Makefile.am
> +index 6d5b00d..3774f9d 100644
> +--- a/tests/Makefile.am
> ++++ b/tests/Makefile.am
> +@@ -1,14 +1,17 @@
> + SUBDIRS = utils regression unit stress
> +
> +-if USE_PYTHON
> +-check-am:
> ++installcheck-am:
> + 	./run.sh unit_tests
> + 	./run.sh fast_regression
> ++if USE_PYTHON
> + 	./run.sh with_bindings_regression
> +-else
> ++endif
> ++
> + check-am:
> + 	./run.sh unit_tests
> + 	./run.sh fast_regression
> ++if USE_PYTHON
> ++	./run.sh with_bindings_regression
> + endif
> +
> + dist_noinst_SCRIPTS = run.sh unit_tests fast_regression long_regression root_regression with_bindings_regression
> +diff --git a/tests/run.sh b/tests/run.sh
> +index c6c50fd..6455359 100755
> +--- a/tests/run.sh
> ++++ b/tests/run.sh
> +@@ -19,4 +19,7 @@
> +
> + [ -z "$1" ] && echo "Error: No testlist. Please specify a testlist to run." && exit 1
> +
> +-prove --merge --exec '' - < $1
> ++prove --merge -v --exec '' - < $1 | sed \
> ++  -e 's|^ok \(.*\)|PASS: \1|' \
> ++  -e 's|^not ok \(.*\)|FAIL: \1|' \
> ++  | egrep -h 'PASS|FAIL'
> diff --git a/meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb b/meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb
> index 2e0ee15..4322027 100644
> --- a/meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb
> +++ b/meta/recipes-kernel/lttng/lttng-tools_2.3.0.bb
> @@ -10,15 +10,19 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=01d7fc4496aacf37d90df90b90b0cac1 \
>                       file://lgpl-2.1.txt;md5=0f0d71500e6a57fd24d825f33242b9ca"
>
>   DEPENDS = "liburcu popt lttng-ust"
> +RDEPENDS_${PN}-ptest += "make"
>
>   SRCREV = "c9dc1289e040c542f96fbfd558267786816d5703"
>   PV = "v2.3.0"
>
> -SRC_URI = "git://git.lttng.org/lttng-tools.git"
> +SRC_URI = "git://git.lttng.org/lttng-tools.git \
> +           file://runtest.patch \
> +           file://run-ptest \
> +	  "
>
>   S = "${WORKDIR}/git"
>
> -inherit autotools
> +inherit autotools ptest
>
>   export KERNELDIR="${STAGING_KERNEL_DIR}"
>
>

Re: [PATCH] Enable ptest support for lttng.

[Stefan Seefeld]

On 2013-09-12 13:13, Saul Wold wrote:
> 
> The usual format for the commit summary is as follows:
> 
> lttng: Enable ptest support
> 
> Please see the Commit/Patch message guideline for details:
> http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines

Thanks, I'll follow that guideline for my next patch(es).

	Stefan


-- 

      ...ich hab' noch einen Koffer in Berlin...

Re: [PATCH] Enable ptest support for lttng.

[Christian Babeux]

Hi Stefan,

> +
> +make -C tests installcheck
> +

A majority of the tests will probably fail this way. As of now, most
tests assumes that they are being runned in the build tree and expect
binaries to be at specific paths. We will have to fix this upstream if
you want to integrate our testsuite with ptest.

> ++prove --merge -v --exec '' - < $1 | sed \
> ++  -e 's|^ok \(.*\)|PASS: \1|' \
> ++  -e 's|^not ok \(.*\)|FAIL: \1|' \
> ++  | egrep -h 'PASS|FAIL'

This is sadly needed because we are using TAP as our standardized test
output format across the multiples lttng codebases.

Thanks,

Christian Babeux
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH] Enable ptest support for lttng.

[Stefan Seefeld]

On 2013-09-12 14:12, Christian Babeux wrote:
> Hi Stefan,
>
>> +
>> +make -C tests installcheck
>> +
> A majority of the tests will probably fail this way. As of now, most
> tests assumes that they are being runned in the build tree and expect
> binaries to be at specific paths. We will have to fix this upstream if
> you want to integrate our testsuite with ptest.

That's right, in my tests only the "unit_tests" suite ran without
issues. The "fast_regressions" suite had many more failures than when
run as part of the 'check' target. (See my mail to the lttng list for more.)

Still, I think it's good to include these tests. If anything they may
just provide more motivation to address the underlying issue. :-)
>> ++prove --merge -v --exec '' - < $1 | sed \
>> ++  -e 's|^ok \(.*\)|PASS: \1|' \
>> ++  -e 's|^not ok \(.*\)|FAIL: \1|' \
>> ++  | egrep -h 'PASS|FAIL'
> This is sadly needed because we are using TAP as our standardized test
> output format across the multiples lttng codebases.

Do you think it would be worthwhile submitting this patch upstream ?
(The output format of prove is not the easiest to understand, and even
with the above some information may be lost. A more robust fix would
probably include a dedicated TAP formatter plugin
(http://perldoc.perl.org/TAP/Formatter/Console.html) that produces
ptest-compatible output.

    Stefan

-- 

      ...ich hab' noch einen Koffer in Berlin...

Re: [PATCH] Enable ptest support for lttng.

[Stefan Seefeld]

On 2013-09-12 14:12, Christian Babeux wrote:
> Hi Stefan,
>
>> +
>> +make -C tests installcheck
>> +
> A majority of the tests will probably fail this way. As of now, most
> tests assumes that they are being runned in the build tree and expect
> binaries to be at specific paths. We will have to fix this upstream if
> you want to integrate our testsuite with ptest.

That's right, in my tests only the "unit_tests" suite ran without
issues. The "fast_regressions" suite had many more failures than when
run as part of the 'check' target. (See my mail to the lttng list for more.)

Still, I think it's good to include these tests. If anything they may
just provide more motivation to address the underlying issue. :-)
>> ++prove --merge -v --exec '' - < $1 | sed \
>> ++  -e 's|^ok \(.*\)|PASS: \1|' \
>> ++  -e 's|^not ok \(.*\)|FAIL: \1|' \
>> ++  | egrep -h 'PASS|FAIL'
> This is sadly needed because we are using TAP as our standardized test
> output format across the multiples lttng codebases.

Do you think it would be worthwhile submitting this patch upstream ?
(The output format of prove is not the easiest to understand, and even
with the above some information may be lost. A more robust fix would
probably include a dedicated TAP formatter plugin
(http://perldoc.perl.org/TAP/Formatter/Console.html) that produces
ptest-compatible output.

    Stefan

-- 

      ...ich hab' noch einen Koffer in Berlin...

Re: [PATCH] Enable ptest support for lttng.

[Christian Babeux]

> Still, I think it's good to include these tests. If anything they may
> just provide more motivation to address the underlying issue. :-)

Absolutely! I'll start a thread on lttng-dev to see how we can deal
with the current testsuite limitations.

>>> ++prove --merge -v --exec '' - < $1 | sed \
>>> ++  -e 's|^ok \(.*\)|PASS: \1|' \
>>> ++  -e 's|^not ok \(.*\)|FAIL: \1|' \
>>> ++  | egrep -h 'PASS|FAIL'
>> This is sadly needed because we are using TAP as our standardized test
>> output format across the multiples lttng codebases.
>
> Do you think it would be worthwhile submitting this patch upstream ?

This is pretty much OpenEmbedded specific and I don't think it is of
much use outside of ptest support.

> (The output format of prove is not the easiest to understand, and even
> with the above some information may be lost. A more robust fix would
> probably include a dedicated TAP formatter plugin
> (http://perldoc.perl.org/TAP/Formatter/Console.html) that produces
> ptest-compatible output.

Yes that would be a cleaner way of dealing with it. We would only need
to feed the appropriate 'testlist' file to this script and get proper
conversion of TAP to Automake-style test output. I'm wondering if such
a script would be useful for other projects/recipes using TAP tests
within oe/yocto?

Thanks,

Christian Babeux
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH] Enable ptest support for lttng.

[Christian Babeux]

> Still, I think it's good to include these tests. If anything they may
> just provide more motivation to address the underlying issue. :-)

Absolutely! I'll start a thread on lttng-dev to see how we can deal
with the current testsuite limitations.

>>> ++prove --merge -v --exec '' - < $1 | sed \
>>> ++  -e 's|^ok \(.*\)|PASS: \1|' \
>>> ++  -e 's|^not ok \(.*\)|FAIL: \1|' \
>>> ++  | egrep -h 'PASS|FAIL'
>> This is sadly needed because we are using TAP as our standardized test
>> output format across the multiples lttng codebases.
>
> Do you think it would be worthwhile submitting this patch upstream ?

This is pretty much OpenEmbedded specific and I don't think it is of
much use outside of ptest support.

> (The output format of prove is not the easiest to understand, and even
> with the above some information may be lost. A more robust fix would
> probably include a dedicated TAP formatter plugin
> (http://perldoc.perl.org/TAP/Formatter/Console.html) that produces
> ptest-compatible output.

Yes that would be a cleaner way of dealing with it. We would only need
to feed the appropriate 'testlist' file to this script and get proper
conversion of TAP to Automake-style test output. I'm wondering if such
a script would be useful for other projects/recipes using TAP tests
within oe/yocto?

Thanks,

Christian Babeux
EfficiOS Inc.
http://www.efficios.com

Re: [PATCH] Enable ptest support for lttng.

[Björn Stenberg]

Stefan Seefeld wrote:
> +RDEPENDS_${PN}-ptest += "make"

I just noticed that the lttng-tools test suite is hardcoded for bash, so that is another dependency.

-- 
Björn