[PATCH 0/5] VSP-Tests: Add suspend resume tests and helpers

[PATCH 0/5] VSP-Tests: Add suspend resume tests and helpers

[Kieran Bingham]

From: Kieran Bingham <kieran.bingham@ideasonboard.com>

Provide two tests for suspend/resume cycles. One will verify the VSP1 is
functional with a test before and after a suspend cycle. The other will
maintain an active pipeline which must succeed despite a suspend resume cycle
occuring in the middle of the test.

Along side these tests are a couple of helpers that I have locally and thought
they might be useful to others, so I'm posting for review. A test suite runner
simplifies executing all vsp-unit tests, and provides the ability to easily
repeat the test suite (for example to run an overnight longevity test).

'bin2png.sh' is a wrapper around the existing tools that helps convert the test
files generated by VSP-Tests into png files for easy viewing.

Finally, I have extended 'logger.sh' to also log to the FTrace buffer. As I have
been making greater use of ftrace for register write capture, and driver flow -
this addition helps separate multiple tests from the ftrace kernelshark view.

Kieran Bingham (5):
  scripts: Test suite runner
  scripts: Provide bin2png.sh helper
  logger: Log to the FTrace buffer if tracing is enabled
  tests: Test suspend/resume on idle pipelines
  tests: Test suspend/resume on active pipelines

 scripts/Makefile            |  2 +-
 scripts/bin2png.sh          | 34 ++++++++++++++++
 scripts/logger.sh           | 13 ++++++-
 scripts/vsp-tests.sh        | 46 ++++++++++++++++++++++
 tests/vsp-unit-test-0019.sh | 84 ++++++++++++++++++++++++++++++++++++++++
 tests/vsp-unit-test-0020.sh | 94 +++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 271 insertions(+), 2 deletions(-)
 create mode 100755 scripts/bin2png.sh
 create mode 100755 scripts/vsp-tests.sh
 create mode 100755 tests/vsp-unit-test-0019.sh
 create mode 100755 tests/vsp-unit-test-0020.sh

-- 
2.7.4

[PATCH 1/5] scripts: Test suite runner

[Kieran Bingham]

From: Kieran Bingham <kieran@bingham.xyz>

Provide a utility script to execute all vsp unit tests, as well
as the option to execute multiple iterations of the suite.

Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
---
 scripts/vsp-tests.sh | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 scripts/vsp-tests.sh

diff --git a/scripts/vsp-tests.sh b/scripts/vsp-tests.sh
new file mode 100755
index 000000000000..e5ffa0ec4236
--- /dev/null
+++ b/scripts/vsp-tests.sh
@@ -0,0 +1,46 @@
+#!/bin/sh
+
+##
+## VSP Tests runner
+##
+## Automatically execute all vsp-unit tests
+## Move test failure results to a specific folder for
+## the running kernel version
+##
+## An argument can be provided to specify the number of
+## iterations to perform
+##
+## usage:
+##  ./vsp-tests.sh <n>
+##
+##   n: Number of iterations to execute test suite
+##
+
+KERNEL_VERSION=`uname -r`
+
+run_test() {
+	echo $1
+	./$1
+
+	if [ $(ls *.bin 2>/dev/null | wc -l) != 0 ];
+	then
+		RESULTS_DIR=$KERNEL_VERSION/test-$1/$2/
+
+		echo "Moving *.bin to test-$1/$2/";
+		mkdir -p $RESULTS_DIR;
+		echo "mv *.bin $RESULTS_DIR";
+		mv *.bin $RESULTS_DIR;
+	fi;
+}
+
+run_suite() {
+	echo "Test loop $1"
+
+	for test in vsp-unit-test*.sh; do
+		run_test $test $1;
+	done;
+}
+
+for loop in `seq 1 1 $1`; do
+	run_suite $loop
+done;
-- 
2.7.4

[PATCH 2/5] scripts: Provide bin2png.sh helper

[Kieran Bingham]

From: Kieran Bingham <kieran@bingham.xyz>

Identify the size and format from the test output filename, and pass
to raw2rgbpnm for conversion to a PNM file.

>From there we can convert easily to a PNG output file.

Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
---
 scripts/Makefile   |  2 +-
 scripts/bin2png.sh | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100755 scripts/bin2png.sh

diff --git a/scripts/Makefile b/scripts/Makefile
index 8c452f4c54ce..6586b2989aed 100644
--- a/scripts/Makefile
+++ b/scripts/Makefile
@@ -1,4 +1,4 @@
-SCRIPTS=logger.sh vsp-lib.sh
+SCRIPTS=$(wildcard *.sh)
 
 all:
 
diff --git a/scripts/bin2png.sh b/scripts/bin2png.sh
new file mode 100755
index 000000000000..527c5546514d
--- /dev/null
+++ b/scripts/bin2png.sh
@@ -0,0 +1,34 @@
+#!/bin/sh
+
+FILE="$1"
+
+fmt=$(echo $FILE | sed -e 's/.*-\([[:alnum:]]*\)-\([0-9]*x[0-9]*\).*.bin/\1/')
+size=$(echo $FILE | sed -e 's/.*-\([[:alnum:]]*\)-\([0-9]*x[0-9]*\).*.bin/\2/')
+
+case $fmt in
+	yuv410m|yvu410m|yuv411m|yuv420m|yvu420m|yuv422m|yvu422m|yuv444m|yvu444m)
+		fmt=`echo $fmt | tr '[:lower:]' '[:upper:]'`
+		fmt=`echo $fmt | tr 'M' 'P'`
+		;;
+	nv12m|nv21m|nv16m|nv61m)
+		fmt=`echo $fmt | tr '[:lower:]' '[:upper:]'`
+		fmt=`echo $fmt | tr -d 'M'`
+		;;
+	argb555|xrgb555)
+		fmt=RGB555X
+		;;
+	argb32|xrgb32)
+		fmt=RGB32
+		;;
+	abgr32|xbgr32)
+		fmt=BGR32
+		;;
+	*)
+		fmt=`echo $fmt | tr '[:lower:]' '[:upper:]'`
+		;;
+esac
+
+# Only run pnmtopng if the conversion to PNM succeeds
+raw2rgbpnm -s $size -f $fmt $FILE $FILE.pnm && \
+	pnmtopng $FILE.pnm > $FILE.png
+rm $FILE.pnm
-- 
2.7.4

[PATCH 3/5] logger: Log to the FTrace buffer if tracing is enabled

[Kieran Bingham]

From: Kieran Bingham <kieran@bingham.xyz>

Extend the logger such that it will detect the tracing system, and also
append print statement to this ring buffer.

This provides the relevant logging output interspersed in the ftrace
logs for an effective solution to identifying the actions that caused
the traces to occur

Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
---
 scripts/logger.sh | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/scripts/logger.sh b/scripts/logger.sh
index 8123f0c9f6e3..8412b0ba9a08 100755
--- a/scripts/logger.sh
+++ b/scripts/logger.sh
@@ -6,6 +6,17 @@ now() {
 
 label=${1:+ [$1]}
 
+TRACE_MARKER=/sys/kernel/debug/tracing/trace_marker
+if [ -e $TRACE_MARKER ]; then
+	extra_log_files=$TRACE_MARKER
+fi
+
 while read line ; do
-	echo "$(now)$label $line"
+	newline="$(now)$label $line"
+
+	echo "$newline"
+
+	for f in $extra_log_files; do
+		echo "$newline" >> $f;
+	done;
 done
-- 
2.7.4

[PATCH 4/5] tests: Test suspend/resume on idle pipelines

[Kieran Bingham]

From: Kieran Bingham <kieran.bingham@ideasonboard.com>

Provide a test to verify the hardware is functional both before and
after entering a suspend / resume cycle. Make use of the
/sys/power/pm_test functionality provided by CONFIG_PM_DEBUG to perform
the testing

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 tests/vsp-unit-test-0019.sh | 77 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)
 create mode 100755 tests/vsp-unit-test-0019.sh

diff --git a/tests/vsp-unit-test-0019.sh b/tests/vsp-unit-test-0019.sh
new file mode 100755
index 000000000000..e7b94996c1aa
--- /dev/null
+++ b/tests/vsp-unit-test-0019.sh
@@ -0,0 +1,77 @@
+#!/bin/sh
+
+#
+# Test power-management suspend/resume whilst pipelines are idle
+#
+# Utilise the basic RPF->WPF packing test case as a measure that the hardware
+# is operable while we perform test suspend and resume, and verify that it is
+# still operable after resume.
+#
+# Format iteration loops are maintained, even with only one format so that this
+# test can be easily extended to try further formats if needed in the future.
+#
+
+source vsp-lib.sh
+
+features="rpf.0 wpf.0"
+formats="RGB24"
+
+# These can be extracted from /sys/power/pm_test
+suspend_modes="freezer devices platform processors core"
+
+test_wpf_packing() {
+	test_start "Verify WPF packing in $format before/after suspend:$mode"
+
+	pipe_configure rpf-wpf 0 0
+	format_configure rpf-wpf 0 0 ARGB32 1024x768 $format
+
+	vsp_runner rpf.0 &
+	vsp_runner wpf.0
+
+	local result=$(compare_frames)
+
+	test_complete $result
+}
+
+test_hw_pipe() {
+	local format
+
+	for format in $formats ; do
+		test_wpf_packing $format
+	done
+}
+
+test_suspend_resume() {
+	local test_results=0
+
+	# Test the hardware each side of suspend resume
+	test_hw_pipe
+
+	# Set the test mode
+	echo $mode > /sys/power/pm_test
+
+	# Comence suspend
+	# The pm_test framework will automatically resume after 5 seconds
+	echo mem > /sys/power/state
+
+	# Verify the hardware is still operational
+	test_hw_pipe
+}
+
+test_main() {
+	local mode;
+
+	# Check for pm-suspend test option
+	if [ ! -e /sys/power/pm_test ] ; then
+		echo "$0: Suspend Resume testing requires CONFIG_PM_DEBUG"
+		test_complete skip
+		return
+	fi;
+
+	for mode in $suspend_modes ; do
+		test_suspend_resume $mode
+	done;
+}
+
+test_init $0 "$features"
+test_run
-- 
2.7.4

[PATCH 5/5] tests: Test suspend/resume on active pipelines

[Kieran Bingham]

From: Kieran Bingham <kieran.bingham@ideasonboard.com>

Provide a test to verify the hardware completes a functional test whilst
performing a suspend resume cycle in parallel. Make use of the
/sys/power/pm_test functionality provided by CONFIG_PM_DEBUG to perform
the testing

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 tests/vsp-unit-test-0020.sh | 86 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)
 create mode 100755 tests/vsp-unit-test-0020.sh

diff --git a/tests/vsp-unit-test-0020.sh b/tests/vsp-unit-test-0020.sh
new file mode 100755
index 000000000000..5838295139b2
--- /dev/null
+++ b/tests/vsp-unit-test-0020.sh
@@ -0,0 +1,86 @@
+#!/bin/sh
+
+#
+# Test power-management suspend/resume whilst pipelines are active
+#
+# Utilise the basic RPF->WPF packing test case as a measure that the hardware
+# is operable while we perform test suspend and resume, and verify that it is
+# still successful even with a suspend resume cycle in the middle of the test.
+#
+# Format iteration loops are maintained, even with only one format so that this
+# test can be easily extended to try further formats if needed in the future.
+#
+
+source vsp-lib.sh
+
+features="rpf.0 wpf.0"
+formats="RGB24"
+
+# These can be extracted from /sys/power/pm_test
+suspend_modes="freezer devices platform processors core"
+
+# This extended function performs the same
+# as it's non-extended name-sake - but runs the pipeline
+# for 300 frames. The suspend action occurs between frame #150~#200
+test_extended_wpf_packing() {
+	test_start "Verify WPF packing in $format during suspend:$mode"
+
+	pipe_configure rpf-wpf 0 0
+	format_configure rpf-wpf 0 0 ARGB32 1024x768 $format
+
+	vsp_runner rpf.0 --count=300 &
+	vsp_runner wpf.0 --count=300 --skip=297
+
+	local result=$(compare_frames)
+
+	test_complete $result
+}
+
+test_hw_pipe() {
+	local format
+
+	for format in $formats ; do
+		test_extended_wpf_packing $format
+	done
+}
+
+test_suspend_resume() {
+	local test_results=0
+	local test_pid
+
+	# Test the hardware each side of suspend resume
+	test_hw_pipe &
+	test_pid=$!
+
+	# Make sure the pipeline has time to start
+	sleep 1
+
+	# Set the test mode
+	echo $mode > /sys/power/pm_test
+
+	# Comence suspend
+	# The pm_test framework will automatically resume after 5 seconds
+	echo mem > /sys/power/state
+
+	# Wait for the pipeline to complete
+	wait $test_pid
+}
+
+test_main() {
+	local mode;
+	local suspend_test_failures
+
+	# Check for pm-suspend test option
+	if [ ! -e /sys/power/pm_test ] ; then
+		echo "$0: Suspend Resume testing requires CONFIG_PM_DEBUG"
+		test_complete skip
+		return
+	fi;
+
+	for mode in $suspend_modes ; do
+		test_suspend_resume $mode
+	done;
+}
+
+test_init $0 "$features"
+test_run
-- 
2.7.4

Re: [PATCH 5/5] tests: Test suspend/resume on active pipelines

[Laurent Pinchart]

Hi Kieran,

Thank you for the patch.

On Friday 25 Nov 2016 13:59:16 Kieran Bingham wrote:
> From: Kieran Bingham <kieran.bingham@ideasonboard.com>
> 
> Provide a test to verify the hardware completes a functional test whilst
> performing a suspend resume cycle in parallel. Make use of the
> /sys/power/pm_test functionality provided by CONFIG_PM_DEBUG to perform
> the testing
> 
> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> ---
>  tests/vsp-unit-test-0020.sh | 86 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 86 insertions(+)
>  create mode 100755 tests/vsp-unit-test-0020.sh
> 
> diff --git a/tests/vsp-unit-test-0020.sh b/tests/vsp-unit-test-0020.sh
> new file mode 100755
> index 000000000000..5838295139b2
> --- /dev/null
> +++ b/tests/vsp-unit-test-0020.sh
> @@ -0,0 +1,86 @@
> +#!/bin/sh
> +
> +#
> +# Test power-management suspend/resume whilst pipelines are active
> +#
> +# Utilise the basic RPF->WPF packing test case as a measure that the
> hardware
> +# is operable while we perform test suspend and resume, and verify that it
> is
> +# still successful even with a suspend resume cycle in the middle of the
> test.
> +#
> +# Format iteration loops are maintained, even with only one format so that
> this
> +# test can be easily extended to try further formats if needed in the
> future.
> +#

Given that this test will suspend during the first iteration only I don't 
think it's very useful to keep the loop.

> +
> +source vsp-lib.sh
> +
> +features="rpf.0 wpf.0"
> +formats="RGB24"
> +
> +# These can be extracted from /sys/power/pm_test
> +suspend_modes="freezer devices platform processors core"

How about extracting them from that file then ? :-) Or maybe just verifying 
they are available before trying to use them ?

> +# This extended function performs the same
> +# as it's non-extended name-sake - but runs the pipeline
> +# for 300 frames. The suspend action occurs between frame #150~#200
> +test_extended_wpf_packing() {
> +	test_start "Verify WPF packing in $format during suspend:$mode"
> +
> +	pipe_configure rpf-wpf 0 0
> +	format_configure rpf-wpf 0 0 ARGB32 1024x768 $format
> +
> +	vsp_runner rpf.0 --count=300 &
> +	vsp_runner wpf.0 --count=300 --skip=297
> +
> +	local result=$(compare_frames)
> +
> +	test_complete $result
> +}
> +
> +test_hw_pipe() {
> +	local format
> +
> +	for format in $formats ; do
> +		test_extended_wpf_packing $format
> +	done
> +}
> +
> +test_suspend_resume() {
> +	local test_results=0

This variable doesn't seem to be used.

> +	local test_pid
> +
> +	# Test the hardware each side of suspend resume
> +	test_hw_pipe &
> +	test_pid=$!
> +
> +	# Make sure the pipeline has time to start
> +	sleep 1
> +
> +	# Set the test mode
> +	echo $mode > /sys/power/pm_test
> +
> +	# Comence suspend

Commence ?

> +	# The pm_test framework will automatically resume after 5 seconds
> +	echo mem > /sys/power/state
> +
> +	# Wait for the pipeline to complete
> +	wait $test_pid

It would be nice to add a timeout here. Maybe something like the following 
(untested) ?

	(sleep 30 ; kill $test_pid) &
	kill_pid=$!
	wait $test_pid
	kill $kill_pid

test_complete should be called here based on both whether the frames compared 
successfully and whether the test timed out.

> +}
> +
> +test_main() {
> +	local mode;

No need for the trailing ;.

> +	local suspend_test_failures

This variable doesn't seem to be used.

> +
> +	# Check for pm-suspend test option
> +	if [ ! -e /sys/power/pm_test ] ; then
> +		echo "$0: Suspend Resume testing requires CONFIG_PM_DEBUG"
> +		test_complete skip
> +		return
> +	fi;
> +
> +	for mode in $suspend_modes ; do
> +		test_suspend_resume $mode
> +	done;
> +}
> +
> +test_init $0 "$features"
> +test_run

-- 
Regards,

Laurent Pinchart

Re: [PATCH 4/5] tests: Test suspend/resume on idle pipelines

[Laurent Pinchart]

Hi Kieran,

Thank you for the patch.

On Friday 25 Nov 2016 13:59:15 Kieran Bingham wrote:
> From: Kieran Bingham <kieran.bingham@ideasonboard.com>
> 
> Provide a test to verify the hardware is functional both before and
> after entering a suspend / resume cycle. Make use of the
> /sys/power/pm_test functionality provided by CONFIG_PM_DEBUG to perform
> the testing
> 
> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> ---
>  tests/vsp-unit-test-0019.sh | 77 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 77 insertions(+)
>  create mode 100755 tests/vsp-unit-test-0019.sh
> 
> diff --git a/tests/vsp-unit-test-0019.sh b/tests/vsp-unit-test-0019.sh
> new file mode 100755
> index 000000000000..e7b94996c1aa
> --- /dev/null
> +++ b/tests/vsp-unit-test-0019.sh
> @@ -0,0 +1,77 @@
> +#!/bin/sh
> +
> +#
> +# Test power-management suspend/resume whilst pipelines are idle
> +#
> +# Utilise the basic RPF->WPF packing test case as a measure that the
> hardware
> +# is operable while we perform test suspend and resume, and verify that it
> is
> +# still operable after resume.
> +#
> +# Format iteration loops are maintained, even with only one format so that
> this
> +# test can be easily extended to try further formats if needed in the
> future.
> +#

I don't think testing multiple formats is needed, but I would like the 
test_wpf_packing() function to be called twice, to verify that stop -> start 
works fine after resume. You could specify two formats to achieve that (with a 
comment explaining why at least two formats should be specified).

> +source vsp-lib.sh
> +
> +features="rpf.0 wpf.0"
> +formats="RGB24"
> +
> +# These can be extracted from /sys/power/pm_test
> +suspend_modes="freezer devices platform processors core"
> +
> +test_wpf_packing() {
> +	test_start "Verify WPF packing in $format before/after suspend:$mode"
> +
> +	pipe_configure rpf-wpf 0 0
> +	format_configure rpf-wpf 0 0 ARGB32 1024x768 $format
> +
> +	vsp_runner rpf.0 &
> +	vsp_runner wpf.0
> +
> +	local result=$(compare_frames)
> +
> +	test_complete $result
> +}
> +
> +test_hw_pipe() {
> +	local format
> +
> +	for format in $formats ; do
> +		test_wpf_packing $format
> +	done
> +}
> +
> +test_suspend_resume() {
> +	local test_results=0

This variable is unused.

> +
> +	# Test the hardware each side of suspend resume
> +	test_hw_pipe
> +
> +	# Set the test mode
> +	echo $mode > /sys/power/pm_test
> +
> +	# Comence suspend
> +	# The pm_test framework will automatically resume after 5 seconds
> +	echo mem > /sys/power/state
> +
> +	# Verify the hardware is still operational
> +	test_hw_pipe

Given that the goal is to test proper operation after resume, it would be 
better to call test_start and test_complete in this function, based on whether 
all the tests passed or not.

> +}
> +
> +test_main() {
> +	local mode;

No need for the trailing ;.

> +
> +	# Check for pm-suspend test option
> +	if [ ! -e /sys/power/pm_test ] ; then
> +		echo "$0: Suspend Resume testing requires CONFIG_PM_DEBUG"
> +		test_complete skip
> +		return
> +	fi;

Ditto.

> +
> +	for mode in $suspend_modes ; do
> +		test_suspend_resume $mode
> +	done;

Ditto.

> +}
> +
> +test_init $0 "$features"
> +test_run

-- 
Regards,

Laurent Pinchart

Re: [PATCH 3/5] logger: Log to the FTrace buffer if tracing is enabled

[Laurent Pinchart]

Hi Kieran,

Thank you for the patch.

On Friday 25 Nov 2016 13:59:14 Kieran Bingham wrote:
> From: Kieran Bingham <kieran@bingham.xyz>
> 
> Extend the logger such that it will detect the tracing system, and also
> append print statement to this ring buffer.
> 
> This provides the relevant logging output interspersed in the ftrace
> logs for an effective solution to identifying the actions that caused
> the traces to occur
> 
> Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
> ---
>  scripts/logger.sh | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/logger.sh b/scripts/logger.sh
> index 8123f0c9f6e3..8412b0ba9a08 100755
> --- a/scripts/logger.sh
> +++ b/scripts/logger.sh
> @@ -6,6 +6,17 @@ now() {
> 
>  label=${1:+ [$1]}
> 
> +TRACE_MARKER=/sys/kernel/debug/tracing/trace_marker
> +if [ -e $TRACE_MARKER ]; then
> +	extra_log_files=$TRACE_MARKER
> +fi
> +
>  while read line ; do
> -	echo "$(now)$label $line"
> +	newline="$(now)$label $line"
> +
> +	echo "$newline"
> +
> +	for f in $extra_log_files; do
> +		echo "$newline" >> $f;

As the tracer adds a timestamp, should you echo "$label $line" only here ?

> +	done;
>  done

-- 
Regards,

Laurent Pinchart

Re: [PATCH 2/5] scripts: Provide bin2png.sh helper

[Laurent Pinchart]

Hi Kieran,

Thank you for the patch.

On Friday 25 Nov 2016 13:59:13 Kieran Bingham wrote:
> From: Kieran Bingham <kieran@bingham.xyz>
> 
> Identify the size and format from the test output filename, and pass
> to raw2rgbpnm for conversion to a PNM file.
> 
> From there we can convert easily to a PNG output file.
> 
> Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
> ---
>  scripts/Makefile   |  2 +-
>  scripts/bin2png.sh | 34 ++++++++++++++++++++++++++++++++++
>  2 files changed, 35 insertions(+), 1 deletion(-)
>  create mode 100755 scripts/bin2png.sh
> 
> diff --git a/scripts/Makefile b/scripts/Makefile
> index 8c452f4c54ce..6586b2989aed 100644
> --- a/scripts/Makefile
> +++ b/scripts/Makefile
> @@ -1,4 +1,4 @@
> -SCRIPTS=logger.sh vsp-lib.sh
> +SCRIPTS=$(wildcard *.sh)
> 
>  all:
> 
> diff --git a/scripts/bin2png.sh b/scripts/bin2png.sh
> new file mode 100755
> index 000000000000..527c5546514d
> --- /dev/null
> +++ b/scripts/bin2png.sh
> @@ -0,0 +1,34 @@
> +#!/bin/sh
> +
> +FILE="$1"
> +
> +fmt=$(echo $FILE | sed -e
> 's/.*-\([[:alnum:]]*\)-\([0-9]*x[0-9]*\).*.bin/\1/')
> +size=$(echo $FILE |
> sed -e 's/.*-\([[:alnum:]]*\)-\([0-9]*x[0-9]*\).*.bin/\2/')
> +
> +case $fmt in
> +	yuv410m|yvu410m|yuv411m|yuv420m|yvu420m|yuv422m|yvu422m|yuv444m|
yvu444m)
> +		fmt=`echo $fmt | tr '[:lower:]' '[:upper:]'`
> +		fmt=`echo $fmt | tr 'M' 'P'`
> +		;;
> +	nv12m|nv21m|nv16m|nv61m)
> +		fmt=`echo $fmt | tr '[:lower:]' '[:upper:]'`
> +		fmt=`echo $fmt | tr -d 'M'`
> +		;;
> +	argb555|xrgb555)
> +		fmt=RGB555X
> +		;;
> +	argb32|xrgb32)
> +		fmt=RGB32
> +		;;
> +	abgr32|xbgr32)
> +		fmt=BGR32
> +		;;
> +	*)
> +		fmt=`echo $fmt | tr '[:lower:]' '[:upper:]'`
> +		;;
> +esac
> +
> +# Only run pnmtopng if the conversion to PNM succeeds
> +raw2rgbpnm -s $size -f $fmt $FILE $FILE.pnm && \

How about stripping the .bin suffix to avoid calling the output file *.bin.png 
?

> +	pnmtopng $FILE.pnm > $FILE.png

We already have a dependency on ImageMagick (although on the target side, not 
the host side). Shoould we use convert instead of pnmtopng ?

> +rm $FILE.pnm

-- 
Regards,

Laurent Pinchart

Re: [PATCH 2/5] scripts: Provide bin2png.sh helper

[Kieran Bingham]

Hi Laurent,

Thanks for the review :D

On 25/11/16 17:55, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Friday 25 Nov 2016 13:59:13 Kieran Bingham wrote:
>> From: Kieran Bingham <kieran@bingham.xyz>
>>
>> Identify the size and format from the test output filename, and pass
>> to raw2rgbpnm for conversion to a PNM file.
>>
>> From there we can convert easily to a PNG output file.
>>
>> Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
>> ---
>>  scripts/Makefile   |  2 +-
>>  scripts/bin2png.sh | 34 ++++++++++++++++++++++++++++++++++
>>  2 files changed, 35 insertions(+), 1 deletion(-)
>>  create mode 100755 scripts/bin2png.sh
>>
>> diff --git a/scripts/Makefile b/scripts/Makefile
>> index 8c452f4c54ce..6586b2989aed 100644
>> --- a/scripts/Makefile
>> +++ b/scripts/Makefile
>> @@ -1,4 +1,4 @@
>> -SCRIPTS=logger.sh vsp-lib.sh
>> +SCRIPTS=$(wildcard *.sh)
>>
>>  all:
>>
>> diff --git a/scripts/bin2png.sh b/scripts/bin2png.sh
>> new file mode 100755
>> index 000000000000..527c5546514d
>> --- /dev/null
>> +++ b/scripts/bin2png.sh
>> @@ -0,0 +1,34 @@
>> +#!/bin/sh
>> +
>> +FILE="$1"
>> +
>> +fmt=$(echo $FILE | sed -e
>> 's/.*-\([[:alnum:]]*\)-\([0-9]*x[0-9]*\).*.bin/\1/')
>> +size=$(echo $FILE |
>> sed -e 's/.*-\([[:alnum:]]*\)-\([0-9]*x[0-9]*\).*.bin/\2/')
>> +
>> +case $fmt in
>> +	yuv410m|yvu410m|yuv411m|yuv420m|yvu420m|yuv422m|yvu422m|yuv444m|
> yvu444m)
>> +		fmt=`echo $fmt | tr '[:lower:]' '[:upper:]'`
>> +		fmt=`echo $fmt | tr 'M' 'P'`
>> +		;;
>> +	nv12m|nv21m|nv16m|nv61m)
>> +		fmt=`echo $fmt | tr '[:lower:]' '[:upper:]'`
>> +		fmt=`echo $fmt | tr -d 'M'`
>> +		;;
>> +	argb555|xrgb555)
>> +		fmt=RGB555X
>> +		;;
>> +	argb32|xrgb32)
>> +		fmt=RGB32
>> +		;;
>> +	abgr32|xbgr32)
>> +		fmt=BGR32
>> +		;;
>> +	*)
>> +		fmt=`echo $fmt | tr '[:lower:]' '[:upper:]'`
>> +		;;
>> +esac
>> +
>> +# Only run pnmtopng if the conversion to PNM succeeds
>> +raw2rgbpnm -s $size -f $fmt $FILE $FILE.pnm && \
> 
> How about stripping the .bin suffix to avoid calling the output file *.bin.png 
> ?

Yeah, we can probably manage this :D

>> +	pnmtopng $FILE.pnm > $FILE.png
> 
> We already have a dependency on ImageMagick (although on the target side, not 
> the host side). Shoould we use convert instead of pnmtopng ?

Ahh - excellent plan. Actually I'd been using this script on the host
only, using convert makes it easy to run this on the target as well.

That means in fact that I can convert 'bin' files from tmpfs storage and
output small png's without hammering the network on NFS usage.

Sounds like a win to me !

> 
>> +rm $FILE.pnm
> 

Re: [PATCH 3/5] logger: Log to the FTrace buffer if tracing is enabled

[Kieran Bingham]

Hi Laurent,

On 25/11/16 17:40, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Friday 25 Nov 2016 13:59:14 Kieran Bingham wrote:
>> From: Kieran Bingham <kieran@bingham.xyz>
>>
>> Extend the logger such that it will detect the tracing system, and also
>> append print statement to this ring buffer.
>>
>> This provides the relevant logging output interspersed in the ftrace
>> logs for an effective solution to identifying the actions that caused
>> the traces to occur
>>
>> Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
>> ---
>>  scripts/logger.sh | 13 ++++++++++++-
>>  1 file changed, 12 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/logger.sh b/scripts/logger.sh
>> index 8123f0c9f6e3..8412b0ba9a08 100755
>> --- a/scripts/logger.sh
>> +++ b/scripts/logger.sh
>> @@ -6,6 +6,17 @@ now() {
>>
>>  label=${1:+ [$1]}
>>
>> +TRACE_MARKER=/sys/kernel/debug/tracing/trace_marker
>> +if [ -e $TRACE_MARKER ]; then
>> +	extra_log_files=$TRACE_MARKER
>> +fi
>> +
>>  while read line ; do
>> -	echo "$(now)$label $line"
>> +	newline="$(now)$label $line"
>> +
>> +	echo "$newline"
>> +
>> +	for f in $extra_log_files; do
>> +		echo "$newline" >> $f;
> 
> As the tracer adds a timestamp, should you echo "$label $line" only here ?

Hrm, yes it is possibly a little bit redundant...

My only argument would be that it will be a 'different' timestamp to the
one logged by logger.sh.

Inspection of a recent log shows a difference of around 40-50 ms, which
will be the latency between capturing the time in $(now) and passing the
buffer to the kernel.

At this level though, the logger.sh is already susceptible to scheduler
jitter anyway, so I'm not too worried about 40 ms. But anyone reading
the logs will have to be aware of that extra latency.


> 
>> +	done;
>>  done
> 

Re: [PATCH 4/5] tests: Test suspend/resume on idle pipelines

[Kieran Bingham]

On 25/11/16 17:13, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Friday 25 Nov 2016 13:59:15 Kieran Bingham wrote:
>> From: Kieran Bingham <kieran.bingham@ideasonboard.com>
>>
>> Provide a test to verify the hardware is functional both before and
>> after entering a suspend / resume cycle. Make use of the
>> /sys/power/pm_test functionality provided by CONFIG_PM_DEBUG to perform
>> the testing
>>
>> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>> ---
>>  tests/vsp-unit-test-0019.sh | 77 ++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 77 insertions(+)
>>  create mode 100755 tests/vsp-unit-test-0019.sh
>>
>> diff --git a/tests/vsp-unit-test-0019.sh b/tests/vsp-unit-test-0019.sh
>> new file mode 100755
>> index 000000000000..e7b94996c1aa
>> --- /dev/null
>> +++ b/tests/vsp-unit-test-0019.sh
>> @@ -0,0 +1,77 @@
>> +#!/bin/sh
>> +
>> +#
>> +# Test power-management suspend/resume whilst pipelines are idle
>> +#
>> +# Utilise the basic RPF->WPF packing test case as a measure that the
>> hardware
>> +# is operable while we perform test suspend and resume, and verify that it
>> is
>> +# still operable after resume.
>> +#
>> +# Format iteration loops are maintained, even with only one format so that
>> this
>> +# test can be easily extended to try further formats if needed in the
>> future.
>> +#
> 
> I don't think testing multiple formats is needed, but I would like the 
> test_wpf_packing() function to be called twice, to verify that stop -> start 
> works fine after resume. You could specify two formats to achieve that (with a 
> comment explaining why at least two formats should be specified).
> 

That sounds like a good plan.

>> +source vsp-lib.sh
>> +
>> +features="rpf.0 wpf.0"
>> +formats="RGB24"
>> +
>> +# These can be extracted from /sys/power/pm_test
>> +suspend_modes="freezer devices platform processors core"
>> +
>> +test_wpf_packing() {
>> +	test_start "Verify WPF packing in $format before/after suspend:$mode"
>> +
>> +	pipe_configure rpf-wpf 0 0
>> +	format_configure rpf-wpf 0 0 ARGB32 1024x768 $format
>> +
>> +	vsp_runner rpf.0 &
>> +	vsp_runner wpf.0
>> +
>> +	local result=$(compare_frames)
>> +
>> +	test_complete $result
>> +}
>> +
>> +test_hw_pipe() {
>> +	local format
>> +
>> +	for format in $formats ; do
>> +		test_wpf_packing $format
>> +	done
>> +}
>> +
>> +test_suspend_resume() {
>> +	local test_results=0
> 
> This variable is unused.

Bah, I missed some. I thought I'd cleaned these out before posting. :D

>> +
>> +	# Test the hardware each side of suspend resume
>> +	test_hw_pipe
>> +
>> +	# Set the test mode
>> +	echo $mode > /sys/power/pm_test
>> +
>> +	# Comence suspend
>> +	# The pm_test framework will automatically resume after 5 seconds
>> +	echo mem > /sys/power/state
>> +
>> +	# Verify the hardware is still operational
>> +	test_hw_pipe
> 
> Given that the goal is to test proper operation after resume, it would be 
> better to call test_start and test_complete in this function, based on whether 
> all the tests passed or not.

Yes, I did go down this route - but stalled. Mainly in getting sensible
return values from the $(compare_frames) which can be tracked through
the functions.

I think it looked like I'd have to string match against, pass, skip,
fail etc ... without a nice easy return value I can add to an integer.

This can be re-examined.

>> +}
>> +
>> +test_main() {
>> +	local mode;
> 
> No need for the trailing ;.

I think my hands are preprogrammed in 'c' coding style. Tough to get out
of sometimes :D

>> +
>> +	# Check for pm-suspend test option
>> +	if [ ! -e /sys/power/pm_test ] ; then
>> +		echo "$0: Suspend Resume testing requires CONFIG_PM_DEBUG"
>> +		test_complete skip
>> +		return
>> +	fi;
> 
> Ditto.

Ack.

> 
>> +
>> +	for mode in $suspend_modes ; do
>> +		test_suspend_resume $mode
>> +	done;
> 
> Ditto.
> 

Ack.


>> +}
>> +
>> +test_init $0 "$features"
>> +test_run
> 

Re: [PATCH 1/5] scripts: Test suite runner

[Laurent Pinchart]

Hi Kieran,

Thank you for the patch.

On Friday 25 Nov 2016 13:59:12 Kieran Bingham wrote:
> From: Kieran Bingham <kieran@bingham.xyz>
> 
> Provide a utility script to execute all vsp unit tests, as well
> as the option to execute multiple iterations of the suite.
> 
> Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
> ---
>  scripts/vsp-tests.sh | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
>  create mode 100755 scripts/vsp-tests.sh
> 
> diff --git a/scripts/vsp-tests.sh b/scripts/vsp-tests.sh
> new file mode 100755
> index 000000000000..e5ffa0ec4236
> --- /dev/null
> +++ b/scripts/vsp-tests.sh
> @@ -0,0 +1,46 @@
> +#!/bin/sh
> +
> +##
> +## VSP Tests runner
> +##
> +## Automatically execute all vsp-unit tests
> +## Move test failure results to a specific folder for
> +## the running kernel version
> +##
> +## An argument can be provided to specify the number of
> +## iterations to perform
> +##
> +## usage:
> +##  ./vsp-tests.sh <n>
> +##
> +##   n: Number of iterations to execute test suite
> +##
> +
> +KERNEL_VERSION=`uname -r`
> +
> +run_test() {
> +	echo $1
> +	./$1
> +
> +	if [ $(ls *.bin 2>/dev/null | wc -l) != 0 ];
> +	then
> +		RESULTS_DIR=$KERNEL_VERSION/test-$1/$2/
> +
> +		echo "Moving *.bin to test-$1/$2/";

You can remove the ; at the end of the lines.

> +		mkdir -p $RESULTS_DIR;
> +		echo "mv *.bin $RESULTS_DIR";

Do we really need this message ?

> +		mv *.bin $RESULTS_DIR;
> +	fi;
> +}
> +
> +run_suite() {
> +	echo "Test loop $1"
> +
> +	for test in vsp-unit-test*.sh; do
> +		run_test $test $1;
> +	done;
> +}
> +
> +for loop in `seq 1 1 $1`; do
> +	run_suite $loop
> +done;

-- 
Regards,

Laurent Pinchart

Re: [PATCH 3/5] logger: Log to the FTrace buffer if tracing is enabled

[Laurent Pinchart]

Hi Kieran,

On Friday 25 Nov 2016 18:10:10 Kieran Bingham wrote:
> On 25/11/16 17:40, Laurent Pinchart wrote:
> > On Friday 25 Nov 2016 13:59:14 Kieran Bingham wrote:
> >> From: Kieran Bingham <kieran@bingham.xyz>
> >> 
> >> Extend the logger such that it will detect the tracing system, and also
> >> append print statement to this ring buffer.
> >> 
> >> This provides the relevant logging output interspersed in the ftrace
> >> logs for an effective solution to identifying the actions that caused
> >> the traces to occur
> >> 
> >> Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
> >> ---
> >> 
> >>  scripts/logger.sh | 13 ++++++++++++-
> >>  1 file changed, 12 insertions(+), 1 deletion(-)
> >> 
> >> diff --git a/scripts/logger.sh b/scripts/logger.sh
> >> index 8123f0c9f6e3..8412b0ba9a08 100755
> >> --- a/scripts/logger.sh
> >> +++ b/scripts/logger.sh
> >> @@ -6,6 +6,17 @@ now() {
> >> 
> >>  label=${1:+ [$1]}
> >> 
> >> +TRACE_MARKER=/sys/kernel/debug/tracing/trace_marker
> >> +if [ -e $TRACE_MARKER ]; then
> >> +	extra_log_files=$TRACE_MARKER
> >> +fi
> >> +
> >> 
> >>  while read line ; do
> >> 
> >> -	echo "$(now)$label $line"
> >> +	newline="$(now)$label $line"
> >> +
> >> +	echo "$newline"
> >> +
> >> +	for f in $extra_log_files; do
> >> +		echo "$newline" >> $f;
> > 
> > As the tracer adds a timestamp, should you echo "$label $line" only here ?
> 
> Hrm, yes it is possibly a little bit redundant...
> 
> My only argument would be that it will be a 'different' timestamp to the
> one logged by logger.sh.
> 
> Inspection of a recent log shows a difference of around 40-50 ms, which
> will be the latency between capturing the time in $(now) and passing the
> buffer to the kernel.
> 
> At this level though, the logger.sh is already susceptible to scheduler
> jitter anyway, so I'm not too worried about 40 ms. But anyone reading
> the logs will have to be aware of that extra latency.

If you think we should keep the timestamp I'm fine with that.

> >> +	done;
> >> 
> >>  done

-- 
Regards,

Laurent Pinchart

Re: [PATCH 5/5] tests: Test suspend/resume on active pipelines

[Kieran Bingham]

Hi Laurent,

On 25/11/16 17:10, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Friday 25 Nov 2016 13:59:16 Kieran Bingham wrote:
>> From: Kieran Bingham <kieran.bingham@ideasonboard.com>
>>
>> Provide a test to verify the hardware completes a functional test whilst
>> performing a suspend resume cycle in parallel. Make use of the
>> /sys/power/pm_test functionality provided by CONFIG_PM_DEBUG to perform
>> the testing
>>
>> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>> ---
>>  tests/vsp-unit-test-0020.sh | 86 ++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 86 insertions(+)
>>  create mode 100755 tests/vsp-unit-test-0020.sh
>>
>> diff --git a/tests/vsp-unit-test-0020.sh b/tests/vsp-unit-test-0020.sh
>> new file mode 100755
>> index 000000000000..5838295139b2
>> --- /dev/null
>> +++ b/tests/vsp-unit-test-0020.sh
>> @@ -0,0 +1,86 @@
>> +#!/bin/sh
>> +
>> +#
>> +# Test power-management suspend/resume whilst pipelines are active
>> +#
>> +# Utilise the basic RPF->WPF packing test case as a measure that the
>> hardware
>> +# is operable while we perform test suspend and resume, and verify that it
>> is
>> +# still successful even with a suspend resume cycle in the middle of the
>> test.
>> +#
>> +# Format iteration loops are maintained, even with only one format so that
>> this
>> +# test can be easily extended to try further formats if needed in the
>> future.
>> +#
> 
> Given that this test will suspend during the first iteration only I don't 
> think it's very useful to keep the loop.
> 

Agreed. Can you tell this test started it's life as
vsp-unit-test-0019.sh ? :D

>> +
>> +source vsp-lib.sh
>> +
>> +features="rpf.0 wpf.0"
>> +formats="RGB24"
>> +
>> +# These can be extracted from /sys/power/pm_test
>> +suspend_modes="freezer devices platform processors core"
> 
> How about extracting them from that file then ? :-) Or maybe just verifying 
> they are available before trying to use them ?

I thought about that - and looked at it.

Annoyingly - /sys/power/pm_test contains none which would need to be
filtered out, and also the active mode is surrounded by square brackets,
which would also need filtering.

# cat /sys/power/pm_test
none core processors platform [devices] freezer

They are also in the reverse order of that which I wanted the tests to run.

All solvable, but I hadn't bothered as I thought it was possibly
overkill. Of course extracting automatically does future proof us
against different modes and changing mode-names - so it does sound like
a valuable thing to do.

>> +# This extended function performs the same
>> +# as it's non-extended name-sake - but runs the pipeline
>> +# for 300 frames. The suspend action occurs between frame #150~#200
>> +test_extended_wpf_packing() {
>> +	test_start "Verify WPF packing in $format during suspend:$mode"
>> +
>> +	pipe_configure rpf-wpf 0 0
>> +	format_configure rpf-wpf 0 0 ARGB32 1024x768 $format
>> +
>> +	vsp_runner rpf.0 --count=300 &
>> +	vsp_runner wpf.0 --count=300 --skip=297
>> +
>> +	local result=$(compare_frames)
>> +
>> +	test_complete $result
>> +}
>> +
>> +test_hw_pipe() {
>> +	local format
>> +
>> +	for format in $formats ; do
>> +		test_extended_wpf_packing $format
>> +	done
>> +}
>> +
>> +test_suspend_resume() {
>> +	local test_results=0
> 
> This variable doesn't seem to be used.
> 

Another left over from when I was experimenting to get the return values
before I was blocked by $(compare_frames) returning a string.


>> +	local test_pid
>> +
>> +	# Test the hardware each side of suspend resume
>> +	test_hw_pipe &
>> +	test_pid=$!
>> +
>> +	# Make sure the pipeline has time to start
>> +	sleep 1
>> +
>> +	# Set the test mode
>> +	echo $mode > /sys/power/pm_test
>> +
>> +	# Comence suspend
> 
> Commence ?

Yes
Where can I get a terminal emulator with spell-check highlighting :D

> 
>> +	# The pm_test framework will automatically resume after 5 seconds
>> +	echo mem > /sys/power/state
>> +
>> +	# Wait for the pipeline to complete
>> +	wait $test_pid
> 
> It would be nice to add a timeout here. Maybe something like the following 
> (untested) ?
> 
> 	(sleep 30 ; kill $test_pid) &
> 	kill_pid=$!
> 	wait $test_pid
> 	kill $kill_pid
> 
> test_complete should be called here based on both whether the frames compared 
> successfully and whether the test timed out.
> 

I agree that the test should have a timeout, and result determined, but
scripting this in shell ... feels ugh :S

Its a shame we don't have gnu-timeout
 - http://man7.org/linux/man-pages/man1/timeout.1.html(man 1 timeout)





>> +}
>> +
>> +test_main() {
>> +	local mode;
> 
> No need for the trailing ;.
> 

Ack.

>> +	local suspend_test_failures
> 
> This variable doesn't seem to be used.

Hrm .. maybe I forgot to do a fresh git-format-patch after I fixed these
up :D

> 
>> +
>> +	# Check for pm-suspend test option
>> +	if [ ! -e /sys/power/pm_test ] ; then
>> +		echo "$0: Suspend Resume testing requires CONFIG_PM_DEBUG"
>> +		test_complete skip
>> +		return
>> +	fi;
>> +
>> +	for mode in $suspend_modes ; do
>> +		test_suspend_resume $mode
>> +	done;
>> +}
>> +
>> +test_init $0 "$features"
>> +test_run
> 

Re: [PATCH 1/5] scripts: Test suite runner

[Kieran Bingham]

On 25/11/16 18:21, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Friday 25 Nov 2016 13:59:12 Kieran Bingham wrote:
>> From: Kieran Bingham <kieran@bingham.xyz>
>>
>> Provide a utility script to execute all vsp unit tests, as well
>> as the option to execute multiple iterations of the suite.
>>
>> Signed-off-by: Kieran Bingham <kieran@bingham.xyz>
>> ---
>>  scripts/vsp-tests.sh | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 46 insertions(+)
>>  create mode 100755 scripts/vsp-tests.sh
>>
>> diff --git a/scripts/vsp-tests.sh b/scripts/vsp-tests.sh
>> new file mode 100755
>> index 000000000000..e5ffa0ec4236
>> --- /dev/null
>> +++ b/scripts/vsp-tests.sh
>> @@ -0,0 +1,46 @@
>> +#!/bin/sh
>> +
>> +##
>> +## VSP Tests runner
>> +##
>> +## Automatically execute all vsp-unit tests
>> +## Move test failure results to a specific folder for
>> +## the running kernel version
>> +##
>> +## An argument can be provided to specify the number of
>> +## iterations to perform
>> +##
>> +## usage:
>> +##  ./vsp-tests.sh <n>
>> +##
>> +##   n: Number of iterations to execute test suite
>> +##
>> +
>> +KERNEL_VERSION=`uname -r`
>> +
>> +run_test() {
>> +	echo $1
>> +	./$1
>> +
>> +	if [ $(ls *.bin 2>/dev/null | wc -l) != 0 ];
>> +	then
>> +		RESULTS_DIR=$KERNEL_VERSION/test-$1/$2/
>> +
>> +		echo "Moving *.bin to test-$1/$2/";
> 
> You can remove the ; at the end of the lines.
> 
>> +		mkdir -p $RESULTS_DIR;
>> +		echo "mv *.bin $RESULTS_DIR";
> 
> Do we really need this message ?

We certainly don't need two :D.

I'll combine the best bits of them both :D
echo "Moving test failures to $RESULTS_DIR";

And in fact, now that bin2png.sh works on the target - this could also
generate the PNG files immediately which would be useful.

>> +		mv *.bin $RESULTS_DIR;
>> +	fi;
>> +}
>> +
>> +run_suite() {
>> +	echo "Test loop $1"
>> +
>> +	for test in vsp-unit-test*.sh; do
>> +		run_test $test $1;
>> +	done;
>> +}
>> +
>> +for loop in `seq 1 1 $1`; do
>> +	run_suite $loop
>> +done;
> 

-- 
Regards

Kieran Bingham

Re: [PATCH 5/5] tests: Test suspend/resume on active pipelines

[Laurent Pinchart]

Hi Kieran,

On Friday 25 Nov 2016 18:40:34 Kieran Bingham wrote:
> On 25/11/16 17:10, Laurent Pinchart wrote:
> > On Friday 25 Nov 2016 13:59:16 Kieran Bingham wrote:
> >> From: Kieran Bingham <kieran.bingham@ideasonboard.com>
> >> 
> >> Provide a test to verify the hardware completes a functional test whilst
> >> performing a suspend resume cycle in parallel. Make use of the
> >> /sys/power/pm_test functionality provided by CONFIG_PM_DEBUG to perform
> >> the testing
> >> 
> >> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> >> ---
> >> 
> >>  tests/vsp-unit-test-0020.sh | 86 +++++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 86 insertions(+)
> >>  create mode 100755 tests/vsp-unit-test-0020.sh
> >> 
> >> diff --git a/tests/vsp-unit-test-0020.sh b/tests/vsp-unit-test-0020.sh
> >> new file mode 100755
> >> index 000000000000..5838295139b2
> >> --- /dev/null
> >> +++ b/tests/vsp-unit-test-0020.sh
> >> @@ -0,0 +1,86 @@
> >> +#!/bin/sh
> >> +
> >> +#
> >> +# Test power-management suspend/resume whilst pipelines are active
> >> +#
> >> +# Utilise the basic RPF->WPF packing test case as a measure that the
> >> hardware
> >> +# is operable while we perform test suspend and resume, and verify that
> >> it is
> >> +# still successful even with a suspend resume cycle in the middle of the
> >> test.
> >> +#
> >> +# Format iteration loops are maintained, even with only one format so
> >> that this
> >> +# test can be easily extended to try further formats if needed in the
> >> future.
> >> +#
> > 
> > Given that this test will suspend during the first iteration only I don't
> > think it's very useful to keep the loop.
> 
> Agreed. Can you tell this test started it's life as
> vsp-unit-test-0019.sh ? :D
> 
> >> +
> >> +source vsp-lib.sh
> >> +
> >> +features="rpf.0 wpf.0"
> >> +formats="RGB24"
> >> +
> >> +# These can be extracted from /sys/power/pm_test
> >> +suspend_modes="freezer devices platform processors core"
> > 
> > How about extracting them from that file then ? :-) Or maybe just
> > verifying they are available before trying to use them ?
> 
> I thought about that - and looked at it.
> 
> Annoyingly - /sys/power/pm_test contains none which would need to be
> filtered out, and also the active mode is surrounded by square brackets,
> which would also need filtering.
> 
> # cat /sys/power/pm_test
> none core processors platform [devices] freezer
> 
> They are also in the reverse order of that which I wanted the tests to run.
> 
> All solvable, but I hadn't bothered as I thought it was possibly
> overkill. Of course extracting automatically does future proof us
> against different modes and changing mode-names - so it does sound like
> a valuable thing to do.

You're right that it might be overkill, it's up to you. If you want to keep 
the list of modes here to select the order, you could do a 

grep -q $mode /sys/power/pm_test || continue

(or something similar possibly with a message being printed to inform that the 
test is skipped)

Even that might be overkill though.

> >> +# This extended function performs the same
> >> +# as it's non-extended name-sake - but runs the pipeline
> >> +# for 300 frames. The suspend action occurs between frame #150~#200
> >> +test_extended_wpf_packing() {
> >> +	test_start "Verify WPF packing in $format during suspend:$mode"
> >> +
> >> +	pipe_configure rpf-wpf 0 0
> >> +	format_configure rpf-wpf 0 0 ARGB32 1024x768 $format
> >> +
> >> +	vsp_runner rpf.0 --count=300 &
> >> +	vsp_runner wpf.0 --count=300 --skip=297
> >> +
> >> +	local result=$(compare_frames)
> >> +
> >> +	test_complete $result
> >> +}
> >> +
> >> +test_hw_pipe() {
> >> +	local format
> >> +
> >> +	for format in $formats ; do
> >> +		test_extended_wpf_packing $format
> >> +	done
> >> +}
> >> +
> >> +test_suspend_resume() {
> >> +	local test_results=0
> > 
> > This variable doesn't seem to be used.
> 
> Another left over from when I was experimenting to get the return values
> before I was blocked by $(compare_frames) returning a string.
> 
> >> +	local test_pid
> >> +
> >> +	# Test the hardware each side of suspend resume
> >> +	test_hw_pipe &
> >> +	test_pid=$!
> >> +
> >> +	# Make sure the pipeline has time to start
> >> +	sleep 1
> >> +
> >> +	# Set the test mode
> >> +	echo $mode > /sys/power/pm_test
> >> +
> >> +	# Comence suspend
> > 
> > Commence ?
> 
> Yes
> Where can I get a terminal emulator with spell-check highlighting :D
> 
> >> +	# The pm_test framework will automatically resume after 5 seconds
> >> +	echo mem > /sys/power/state
> >> +
> >> +	# Wait for the pipeline to complete
> >> +	wait $test_pid
> > 
> > It would be nice to add a timeout here. Maybe something like the following
> > (untested) ?
> > 
> > 	(sleep 30 ; kill $test_pid) &
> > 	kill_pid=$!
> > 	wait $test_pid
> > 	kill $kill_pid
> > 
> > test_complete should be called here based on both whether the frames
> > compared successfully and whether the test timed out.
> 
> I agree that the test should have a timeout, and result determined, but
> scripting this in shell ... feels ugh :S
> 
> Its a shame we don't have gnu-timeout
>  - http://man7.org/linux/man-pages/man1/timeout.1.html(man 1 timeout)

Would the busybox version of timeout work ? If so we could make that a 
requirement.

> >> +}
> >> +
> >> +test_main() {
> >> +	local mode;
> > 
> > No need for the trailing ;.
> 
> Ack.
> 
> >> +	local suspend_test_failures
> > 
> > This variable doesn't seem to be used.
> 
> Hrm .. maybe I forgot to do a fresh git-format-patch after I fixed these
> up :D
> 
> >> +
> >> +	# Check for pm-suspend test option
> >> +	if [ ! -e /sys/power/pm_test ] ; then
> >> +		echo "$0: Suspend Resume testing requires CONFIG_PM_DEBUG"
> >> +		test_complete skip
> >> +		return
> >> +	fi;
> >> +
> >> +	for mode in $suspend_modes ; do
> >> +		test_suspend_resume $mode
> >> +	done;
> >> +}
> >> +
> >> +test_init $0 "$features"
> >> +test_run

-- 
Regards,

Laurent Pinchart