linux-renesas-soc.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [VSP-Tests PATCH 0/7] Reset controls and unloved patches
@ 2018-12-04 15:51 Kieran Bingham
  2018-12-04 15:51 ` [VSP-Tests PATCH 1/7] tests: add pseudo platform test Kieran Bingham
                   ` (6 more replies)
  0 siblings, 7 replies; 24+ messages in thread
From: Kieran Bingham @ 2018-12-04 15:51 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc, Kieran Bingham

Update the VSP-Test suite library to use yavta's (new) --reset-controls
feature to ensure each test starts with a clean environment.

This prevents tests being affected by previous settings to controls such
as rotate or flip.

Also, the series reposts other unloved patches which have gone either
un-reviewed or are still not yet integrated.


Kieran Bingham (7):
  tests: add pseudo platform test
  tests: Provide copy test to validate 1xN streams
  vsp-lib: Provide command line argument parsing
  vsp-lib: trivial: Fix spelling of Reference
  vsp-lib: Allow forcing pixel perfect comparisons
  vsp-lib: Support late queuing of buffers in yavta
  vsp-lib: Reset controls to defaults on each test run

 scripts/vsp-lib.sh          | 62 +++++++++++++++++++++++++++++++++++--
 tests/vsp-unit-test-0000.sh | 20 ++++++++++++
 tests/vsp-unit-test-0025.sh | 45 +++++++++++++++++++++++++++
 3 files changed, 125 insertions(+), 2 deletions(-)
 create mode 100755 tests/vsp-unit-test-0000.sh
 create mode 100755 tests/vsp-unit-test-0025.sh

-- 
2.17.1

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [VSP-Tests PATCH 1/7] tests: add pseudo platform test
  2018-12-04 15:51 [VSP-Tests PATCH 0/7] Reset controls and unloved patches Kieran Bingham
@ 2018-12-04 15:51 ` Kieran Bingham
  2019-02-17  2:41   ` Laurent Pinchart
  2018-12-04 15:51 ` [VSP-Tests PATCH 2/7] tests: Provide copy test to validate 1xN streams Kieran Bingham
                   ` (5 subsequent siblings)
  6 siblings, 1 reply; 24+ messages in thread
From: Kieran Bingham @ 2018-12-04 15:51 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc, Kieran Bingham

Provide an initial test which can run as part of the test suite.
This test will report the platform and kernel version, along with
the identified paths of required utilities.

This will aid in ensuring that required tools are available on a
running platform - and report the kernel and platform details in
any test suite output for clarification of results.

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 tests/vsp-unit-test-0000.sh | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
 create mode 100755 tests/vsp-unit-test-0000.sh

diff --git a/tests/vsp-unit-test-0000.sh b/tests/vsp-unit-test-0000.sh
new file mode 100755
index 000000000000..144cfc677b32
--- /dev/null
+++ b/tests/vsp-unit-test-0000.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+
+# Report testing conditions
+
+model=`cat /sys/firmware/devicetree/base/model`
+
+echo "Test Conditions:"
+
+function check_all() {
+	echo "  Platform: " "$model"
+	echo "  Kernel release: " `uname -r`
+	echo "  convert: " `which convert`
+	echo "  compare: " `which compare`
+	echo "  killall: " `which killall`
+	echo "  raw2rgbpnm: " `which raw2rgbpnm`
+	echo "  stress: " `which stress`
+	echo "  yavta: " `which yavta`
+}
+
+check_all | column -ts ":"
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [VSP-Tests PATCH 2/7] tests: Provide copy test to validate 1xN streams
  2018-12-04 15:51 [VSP-Tests PATCH 0/7] Reset controls and unloved patches Kieran Bingham
  2018-12-04 15:51 ` [VSP-Tests PATCH 1/7] tests: add pseudo platform test Kieran Bingham
@ 2018-12-04 15:51 ` Kieran Bingham
  2019-02-17  2:47   ` Laurent Pinchart
  2018-12-04 15:51 ` [VSP-Tests PATCH 3/7] vsp-lib: Provide command line argument parsing Kieran Bingham
                   ` (4 subsequent siblings)
  6 siblings, 1 reply; 24+ messages in thread
From: Kieran Bingham @ 2018-12-04 15:51 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc, Kieran Bingham

Validate that a 1xN stream can be read through the RPF and written
through the WPF.

The test framework does not currently support processing images where
the stride does not match the output width - so the testing is currently
limited to testing only the vertical direction in this aspect.

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 tests/vsp-unit-test-0025.sh | 45 +++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
 create mode 100755 tests/vsp-unit-test-0025.sh

diff --git a/tests/vsp-unit-test-0025.sh b/tests/vsp-unit-test-0025.sh
new file mode 100755
index 000000000000..57a1fac6e369
--- /dev/null
+++ b/tests/vsp-unit-test-0025.sh
@@ -0,0 +1,45 @@
+#!/bin/sh
+
+#
+# Test pipelines which have a single pixel dimension. Use a RPF -> WPF
+# pipeline with identical input and output formats to generate our output.
+#
+
+. ./vsp-lib.sh
+
+features="rpf.0 uds wpf.0"
+formats="RGB24 ARGB32"
+
+# Input is directly copied to the output. No change in format or size.
+test_copy() {
+	local format=$1
+	local insize=$2
+
+	test_start "copying $insize in $format"
+
+	pipe_configure rpf-wpf 0 0
+	format_configure rpf-wpf 0 0 $format $insize $format
+
+	vsp_runner rpf.0 &
+	vsp_runner wpf.0
+
+	local result=$(compare_frames)
+
+	test_complete $result
+}
+
+test_main() {
+	local format
+
+	for format in $formats ; do
+		test_copy $format 1024x768
+		test_copy $format 128x128
+		test_copy $format 128x1
+
+		# Skipped : Test framework does not yet support strides != width
+		#test_copy $format 1x128
+	done
+}
+
+test_init $0 "$features"
+test_run
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [VSP-Tests PATCH 3/7] vsp-lib: Provide command line argument parsing
  2018-12-04 15:51 [VSP-Tests PATCH 0/7] Reset controls and unloved patches Kieran Bingham
  2018-12-04 15:51 ` [VSP-Tests PATCH 1/7] tests: add pseudo platform test Kieran Bingham
  2018-12-04 15:51 ` [VSP-Tests PATCH 2/7] tests: Provide copy test to validate 1xN streams Kieran Bingham
@ 2018-12-04 15:51 ` Kieran Bingham
  2019-02-17  2:48   ` Laurent Pinchart
  2018-12-04 15:51 ` [VSP-Tests PATCH 4/7] vsp-lib: trivial: Fix spelling of Reference Kieran Bingham
                   ` (3 subsequent siblings)
  6 siblings, 1 reply; 24+ messages in thread
From: Kieran Bingham @ 2018-12-04 15:51 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc, Kieran Bingham

Extend the vsp-lib to support command line parsing for all tests.  The
arguments parsed here should be common to all tests, and initially
provide shell level verbose debug output, and the option to easily keep
frames output by the VSP1.

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 scripts/vsp-lib.sh | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
index 0f3992a7827e..56969606382f 100755
--- a/scripts/vsp-lib.sh
+++ b/scripts/vsp-lib.sh
@@ -1094,3 +1094,37 @@ test_complete() {
 test_run() {
 	test_main | ./logger.sh error >> $logfile
 }
+
+# ------------------------------------------------------------------------------
+# Common argument parsing
+#
+# non-recognised arguments are restored, to allow tests to implement their own
+# parsing if necessary.
+
+POSITIONAL=()
+while [[ $# -gt 0 ]]
+do
+case $1 in
+	-x|--debug)
+		set -x;
+		shift
+		;;
+	-k|--keep-frames)
+		export VSP_KEEP_FRAMES=1
+		shift
+		;;
+	-h|--help)
+		echo "$(basename $0): VSP Test library"
+		echo "  -x|--debug          enable shell debug"
+		echo "  -k|--keep-frames    keep generated and captured frames"
+		echo "  -h|--help           this help"
+		exit
+		shift
+		;;
+	*)    # unknown option
+		POSITIONAL+=("$1") # save it in an array for later
+		shift # past argument
+		;;
+esac
+done
+set -- "${POSITIONAL[@]}" # restore positional parameters
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [VSP-Tests PATCH 4/7] vsp-lib: trivial: Fix spelling of Reference
  2018-12-04 15:51 [VSP-Tests PATCH 0/7] Reset controls and unloved patches Kieran Bingham
                   ` (2 preceding siblings ...)
  2018-12-04 15:51 ` [VSP-Tests PATCH 3/7] vsp-lib: Provide command line argument parsing Kieran Bingham
@ 2018-12-04 15:51 ` Kieran Bingham
  2019-02-17  2:42   ` Laurent Pinchart
  2018-12-04 15:51 ` [VSP-Tests PATCH 5/7] vsp-lib: Allow forcing pixel perfect comparisons Kieran Bingham
                   ` (2 subsequent siblings)
  6 siblings, 1 reply; 24+ messages in thread
From: Kieran Bingham @ 2018-12-04 15:51 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc, Kieran Bingham

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 scripts/vsp-lib.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
index 56969606382f..cf15a045ea8c 100755
--- a/scripts/vsp-lib.sh
+++ b/scripts/vsp-lib.sh
@@ -95,7 +95,7 @@ vsp1_set_control() {
 }
 
 # -----------------------------------------------------------------------------
-# Referance frame generation
+# Reference frame generation
 #
 
 reference_frame() {
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [VSP-Tests PATCH 5/7] vsp-lib: Allow forcing pixel perfect comparisons
  2018-12-04 15:51 [VSP-Tests PATCH 0/7] Reset controls and unloved patches Kieran Bingham
                   ` (3 preceding siblings ...)
  2018-12-04 15:51 ` [VSP-Tests PATCH 4/7] vsp-lib: trivial: Fix spelling of Reference Kieran Bingham
@ 2018-12-04 15:51 ` Kieran Bingham
  2019-02-17 11:59   ` Laurent Pinchart
  2018-12-04 15:51 ` [VSP-Tests PATCH 6/7] vsp-lib: Support late queuing of buffers in yavta Kieran Bingham
  2018-12-04 15:51 ` [VSP-Tests PATCH 7/7] vsp-lib: Reset controls to defaults on each test run Kieran Bingham
  6 siblings, 1 reply; 24+ messages in thread
From: Kieran Bingham @ 2018-12-04 15:51 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc, Kieran Bingham

Provide a means for the tester to request pixel perfect matches on tests.
This can be either through setting the environment variable VSP_PIXEL_PERFECT, or
by passing either '-p' or '--pixel-perfect' on the test command line.

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 scripts/vsp-lib.sh | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
index cf15a045ea8c..9140254c7459 100755
--- a/scripts/vsp-lib.sh
+++ b/scripts/vsp-lib.sh
@@ -287,7 +287,11 @@ compare_frames() {
 	params=${params//)/_}
 	params=$pipe-$in_fmt-$out_fmt-$size$params
 
-	if [ x$__vsp_pixel_perfect != xtrue ] ; then
+	# The system can hint when pixel-perfection is not supported,
+	# however the user can override to force this requirement with
+	# VSP_PIXEL_PERFECT=1 in the environment or by passing -p, or
+	# --pixel-perfect on the commandline.
+	if [ x$__vsp_pixel_perfect != xtrue -a x$VSP_PIXEL_PERFECT != x1 ] ; then
 		method=fuzzy
 	fi
 
@@ -1113,10 +1117,15 @@ case $1 in
 		export VSP_KEEP_FRAMES=1
 		shift
 		;;
+	-p|--pixel-perfect)
+		export VSP_PIXEL_PERFECT=1
+		shift
+		;;
 	-h|--help)
 		echo "$(basename $0): VSP Test library"
 		echo "  -x|--debug          enable shell debug"
 		echo "  -k|--keep-frames    keep generated and captured frames"
+		echo "  -p|--pixel-perfect  frames must match with pixel perfection"
 		echo "  -h|--help           this help"
 		exit
 		shift
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [VSP-Tests PATCH 6/7] vsp-lib: Support late queuing of buffers in yavta
  2018-12-04 15:51 [VSP-Tests PATCH 0/7] Reset controls and unloved patches Kieran Bingham
                   ` (4 preceding siblings ...)
  2018-12-04 15:51 ` [VSP-Tests PATCH 5/7] vsp-lib: Allow forcing pixel perfect comparisons Kieran Bingham
@ 2018-12-04 15:51 ` Kieran Bingham
  2019-02-17 12:00   ` Laurent Pinchart
  2018-12-04 15:51 ` [VSP-Tests PATCH 7/7] vsp-lib: Reset controls to defaults on each test run Kieran Bingham
  6 siblings, 1 reply; 24+ messages in thread
From: Kieran Bingham @ 2018-12-04 15:51 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc, Kieran Bingham

Provide the pass-through option of --queue-late to vsp-runner, to request
that yavta will queue frames after the stream has started.

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 scripts/vsp-lib.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
index 9140254c7459..3d2792707d24 100755
--- a/scripts/vsp-lib.sh
+++ b/scripts/vsp-lib.sh
@@ -898,6 +898,7 @@ vsp_runner() {
 	local count=10
 	local pause=
 	local skip=7
+	local queue_late=
 
 	for option in $* ; do
 		case $option in
@@ -916,6 +917,8 @@ vsp_runner() {
 		--skip=*)
 			skip=${option/--skip=/}
 			;;
+		--queue-late)
+			queue_late=queue-late
 
 		*)
 			return 1
@@ -959,6 +962,7 @@ vsp_runner() {
 
 	$yavta -c$count -n $buffers ${format:+-f $format} ${size:+-s $size} \
 		${skip:+--skip $skip} ${file:+--file=$file} ${pause:+-p$pause} \
+		${queue_late:+--$queue_late} \
 		$videodev | ./logger.sh $entity >> $logfile
 }
 
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [VSP-Tests PATCH 7/7] vsp-lib: Reset controls to defaults on each test run
  2018-12-04 15:51 [VSP-Tests PATCH 0/7] Reset controls and unloved patches Kieran Bingham
                   ` (5 preceding siblings ...)
  2018-12-04 15:51 ` [VSP-Tests PATCH 6/7] vsp-lib: Support late queuing of buffers in yavta Kieran Bingham
@ 2018-12-04 15:51 ` Kieran Bingham
  2019-02-17 12:01   ` Laurent Pinchart
  6 siblings, 1 reply; 24+ messages in thread
From: Kieran Bingham @ 2018-12-04 15:51 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc, Kieran Bingham

Some of our tests set flipping and rotation controls, and the VSP cell
can be used again by later tests. If these controls are not reset, then
that operation is applied to later tests incorrectly causing that test
to fail.

In an ideal world, tests should clean up after themselves, and leave the
system in a known state. However the world is not ideal and we would not
be able to guarantee any previous system state before a test was run
anyway. Therefore it is more effective to reset state at the beginning
of a test.

To repair this - reset all control values to their defaults at the start
of every test during test_init()

Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
---
 scripts/vsp-lib.sh | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
index 3d2792707d24..33442816f208 100755
--- a/scripts/vsp-lib.sh
+++ b/scripts/vsp-lib.sh
@@ -94,6 +94,14 @@ vsp1_set_control() {
 	$yavta --no-query -w "$control $value" $subdev | ./logger.sh "$entity" >> $logfile
 }
 
+vsp1_reset_controls() {
+	local entity=$1
+	local subdev=$(vsp1_entity_subdev $entity)
+
+	echo "Resetting controls on $subdev" | ./logger.sh "$entity" >> $logfile
+	$yavta --no-query --reset-controls $subdev | ./logger.sh "$entity" >> $logfile
+}
+
 # -----------------------------------------------------------------------------
 # Reference frame generation
 #
@@ -1082,6 +1090,9 @@ test_init() {
 	dev=$(vsp1_device $mdev)
 	echo "Using device $mdev ($dev)" | ./logger.sh config >> $logfile
 
+	# Reset any rotation or flipping controls
+	vsp1_reset_controls wpf.0
+
 	vsp_runner=./vsp-runner.sh
 }
 
-- 
2.17.1

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 1/7] tests: add pseudo platform test
  2018-12-04 15:51 ` [VSP-Tests PATCH 1/7] tests: add pseudo platform test Kieran Bingham
@ 2019-02-17  2:41   ` Laurent Pinchart
  2019-02-17 20:41     ` Kieran Bingham
  0 siblings, 1 reply; 24+ messages in thread
From: Laurent Pinchart @ 2019-02-17  2:41 UTC (permalink / raw)
  To: Kieran Bingham; +Cc: Kieran Bingham, linux-renesas-soc

Hi Kieran,

Thank you for the patch.

On Tue, Dec 04, 2018 at 03:51:40PM +0000, Kieran Bingham wrote:
> Provide an initial test which can run as part of the test suite.
> This test will report the platform and kernel version, along with
> the identified paths of required utilities.
> 
> This will aid in ensuring that required tools are available on a
> running platform - and report the kernel and platform details in
> any test suite output for clarification of results.
> 
> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> ---
>  tests/vsp-unit-test-0000.sh | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
>  create mode 100755 tests/vsp-unit-test-0000.sh
> 
> diff --git a/tests/vsp-unit-test-0000.sh b/tests/vsp-unit-test-0000.sh
> new file mode 100755
> index 000000000000..144cfc677b32
> --- /dev/null
> +++ b/tests/vsp-unit-test-0000.sh
> @@ -0,0 +1,20 @@
> +#!/bin/sh
> +
> +# Report testing conditions
> +
> +model=`cat /sys/firmware/devicetree/base/model`

Please use $(...) instead of `...`, it's more readable.

> +
> +echo "Test Conditions:"
> +
> +function check_all() {
> +	echo "  Platform: " "$model"
> +	echo "  Kernel release: " `uname -r`
> +	echo "  convert: " `which convert`
> +	echo "  compare: " `which compare`
> +	echo "  killall: " `which killall`
> +	echo "  raw2rgbpnm: " `which raw2rgbpnm`
> +	echo "  stress: " `which stress`
> +	echo "  yavta: " `which yavta`
> +}
> +
> +check_all | column -ts ":"

Could we remove the dependency on the column tool ? I don't have it in
my buildroot environment :-) I could of course add it, but in this case
I think it's easy enough to align the columns manually.

With these fixed,

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 4/7] vsp-lib: trivial: Fix spelling of Reference
  2018-12-04 15:51 ` [VSP-Tests PATCH 4/7] vsp-lib: trivial: Fix spelling of Reference Kieran Bingham
@ 2019-02-17  2:42   ` Laurent Pinchart
  2019-02-17 20:48     ` Kieran Bingham
  0 siblings, 1 reply; 24+ messages in thread
From: Laurent Pinchart @ 2019-02-17  2:42 UTC (permalink / raw)
  To: Kieran Bingham; +Cc: Kieran Bingham, linux-renesas-soc

Hi Kieran,

Thank you for the patch.

On Tue, Dec 04, 2018 at 03:51:43PM +0000, Kieran Bingham wrote:
> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  scripts/vsp-lib.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
> index 56969606382f..cf15a045ea8c 100755
> --- a/scripts/vsp-lib.sh
> +++ b/scripts/vsp-lib.sh
> @@ -95,7 +95,7 @@ vsp1_set_control() {
>  }
>  
>  # -----------------------------------------------------------------------------
> -# Referance frame generation
> +# Reference frame generation
>  #
>  
>  reference_frame() {

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 2/7] tests: Provide copy test to validate 1xN streams
  2018-12-04 15:51 ` [VSP-Tests PATCH 2/7] tests: Provide copy test to validate 1xN streams Kieran Bingham
@ 2019-02-17  2:47   ` Laurent Pinchart
  2019-02-17 20:43     ` Kieran Bingham
  0 siblings, 1 reply; 24+ messages in thread
From: Laurent Pinchart @ 2019-02-17  2:47 UTC (permalink / raw)
  To: Kieran Bingham; +Cc: Kieran Bingham, linux-renesas-soc

Hi Kieran,

Thank you for the patch.

On Tue, Dec 04, 2018 at 03:51:41PM +0000, Kieran Bingham wrote:
> Validate that a 1xN stream can be read through the RPF and written
> through the WPF.
> 
> The test framework does not currently support processing images where
> the stride does not match the output width - so the testing is currently
> limited to testing only the vertical direction in this aspect.
> 
> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> ---
>  tests/vsp-unit-test-0025.sh | 45 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 45 insertions(+)
>  create mode 100755 tests/vsp-unit-test-0025.sh
> 
> diff --git a/tests/vsp-unit-test-0025.sh b/tests/vsp-unit-test-0025.sh
> new file mode 100755
> index 000000000000..57a1fac6e369
> --- /dev/null
> +++ b/tests/vsp-unit-test-0025.sh
> @@ -0,0 +1,45 @@
> +#!/bin/sh
> +
> +#
> +# Test pipelines which have a single pixel dimension. Use a RPF -> WPF
> +# pipeline with identical input and output formats to generate our output.
> +#
> +
> +. ./vsp-lib.sh
> +
> +features="rpf.0 uds wpf.0"
> +formats="RGB24 ARGB32"
> +
> +# Input is directly copied to the output. No change in format or size.
> +test_copy() {
> +	local format=$1
> +	local insize=$2
> +
> +	test_start "copying $insize in $format"
> +
> +	pipe_configure rpf-wpf 0 0
> +	format_configure rpf-wpf 0 0 $format $insize $format
> +
> +	vsp_runner rpf.0 &
> +	vsp_runner wpf.0
> +
> +	local result=$(compare_frames)
> +
> +	test_complete $result
> +}
> +
> +test_main() {
> +	local format
> +
> +	for format in $formats ; do
> +		test_copy $format 1024x768
> +		test_copy $format 128x128
> +		test_copy $format 128x1

If the purpose is to test Nx1 and 1xN, do we need the first two
resolutions ? Shouldn't the test_start message describe the exact
purpose ?

> +
> +		# Skipped : Test framework does not yet support strides != width
> +		#test_copy $format 1x128
> +	done
> +}
> +
> +test_init $0 "$features"
> +test_run

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 3/7] vsp-lib: Provide command line argument parsing
  2018-12-04 15:51 ` [VSP-Tests PATCH 3/7] vsp-lib: Provide command line argument parsing Kieran Bingham
@ 2019-02-17  2:48   ` Laurent Pinchart
  2019-02-17 20:47     ` Kieran Bingham
  0 siblings, 1 reply; 24+ messages in thread
From: Laurent Pinchart @ 2019-02-17  2:48 UTC (permalink / raw)
  To: Kieran Bingham; +Cc: Kieran Bingham, linux-renesas-soc

Hi Kieran,

Thank you for the patch.

On Tue, Dec 04, 2018 at 03:51:42PM +0000, Kieran Bingham wrote:
> Extend the vsp-lib to support command line parsing for all tests.  The
> arguments parsed here should be common to all tests, and initially
> provide shell level verbose debug output, and the option to easily keep
> frames output by the VSP1.
> 
> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> ---
>  scripts/vsp-lib.sh | 34 ++++++++++++++++++++++++++++++++++
>  1 file changed, 34 insertions(+)
> 
> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
> index 0f3992a7827e..56969606382f 100755
> --- a/scripts/vsp-lib.sh
> +++ b/scripts/vsp-lib.sh
> @@ -1094,3 +1094,37 @@ test_complete() {
>  test_run() {
>  	test_main | ./logger.sh error >> $logfile
>  }
> +
> +# ------------------------------------------------------------------------------
> +# Common argument parsing
> +#
> +# non-recognised arguments are restored, to allow tests to implement their own
> +# parsing if necessary.
> +
> +POSITIONAL=()

This work in bash only :-( Any chance to implement a solution that would
work in ash too ?

> +while [[ $# -gt 0 ]]
> +do
> +case $1 in
> +	-x|--debug)
> +		set -x;
> +		shift
> +		;;
> +	-k|--keep-frames)
> +		export VSP_KEEP_FRAMES=1
> +		shift
> +		;;
> +	-h|--help)
> +		echo "$(basename $0): VSP Test library"
> +		echo "  -x|--debug          enable shell debug"
> +		echo "  -k|--keep-frames    keep generated and captured frames"
> +		echo "  -h|--help           this help"
> +		exit
> +		shift
> +		;;
> +	*)    # unknown option
> +		POSITIONAL+=("$1") # save it in an array for later
> +		shift # past argument
> +		;;
> +esac
> +done
> +set -- "${POSITIONAL[@]}" # restore positional parameters

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 5/7] vsp-lib: Allow forcing pixel perfect comparisons
  2018-12-04 15:51 ` [VSP-Tests PATCH 5/7] vsp-lib: Allow forcing pixel perfect comparisons Kieran Bingham
@ 2019-02-17 11:59   ` Laurent Pinchart
  2019-02-17 20:51     ` Kieran Bingham
  0 siblings, 1 reply; 24+ messages in thread
From: Laurent Pinchart @ 2019-02-17 11:59 UTC (permalink / raw)
  To: Kieran Bingham; +Cc: Kieran Bingham, linux-renesas-soc

Hi Kieran,

Thank you for the patch.

On Tue, Dec 04, 2018 at 03:51:44PM +0000, Kieran Bingham wrote:
> Provide a means for the tester to request pixel perfect matches on tests.
> This can be either through setting the environment variable VSP_PIXEL_PERFECT, or
> by passing either '-p' or '--pixel-perfect' on the test command line.

What's the use case for this ?

> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> ---
>  scripts/vsp-lib.sh | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
> index cf15a045ea8c..9140254c7459 100755
> --- a/scripts/vsp-lib.sh
> +++ b/scripts/vsp-lib.sh
> @@ -287,7 +287,11 @@ compare_frames() {
>  	params=${params//)/_}
>  	params=$pipe-$in_fmt-$out_fmt-$size$params
>  
> -	if [ x$__vsp_pixel_perfect != xtrue ] ; then
> +	# The system can hint when pixel-perfection is not supported,
> +	# however the user can override to force this requirement with
> +	# VSP_PIXEL_PERFECT=1 in the environment or by passing -p, or
> +	# --pixel-perfect on the commandline.
> +	if [ x$__vsp_pixel_perfect != xtrue -a x$VSP_PIXEL_PERFECT != x1 ] ; then
>  		method=fuzzy
>  	fi
>  
> @@ -1113,10 +1117,15 @@ case $1 in
>  		export VSP_KEEP_FRAMES=1
>  		shift
>  		;;
> +	-p|--pixel-perfect)
> +		export VSP_PIXEL_PERFECT=1
> +		shift
> +		;;
>  	-h|--help)
>  		echo "$(basename $0): VSP Test library"
>  		echo "  -x|--debug          enable shell debug"
>  		echo "  -k|--keep-frames    keep generated and captured frames"
> +		echo "  -p|--pixel-perfect  frames must match with pixel perfection"
>  		echo "  -h|--help           this help"
>  		exit
>  		shift

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 6/7] vsp-lib: Support late queuing of buffers in yavta
  2018-12-04 15:51 ` [VSP-Tests PATCH 6/7] vsp-lib: Support late queuing of buffers in yavta Kieran Bingham
@ 2019-02-17 12:00   ` Laurent Pinchart
  2019-02-17 20:57     ` Kieran Bingham
  0 siblings, 1 reply; 24+ messages in thread
From: Laurent Pinchart @ 2019-02-17 12:00 UTC (permalink / raw)
  To: Kieran Bingham; +Cc: Kieran Bingham, linux-renesas-soc

Hi Kieran,

Thank you for the patch.

On Tue, Dec 04, 2018 at 03:51:45PM +0000, Kieran Bingham wrote:
> Provide the pass-through option of --queue-late to vsp-runner, to request
> that yavta will queue frames after the stream has started.

As with 5/7, what is the use case for this ?

> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> ---
>  scripts/vsp-lib.sh | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
> index 9140254c7459..3d2792707d24 100755
> --- a/scripts/vsp-lib.sh
> +++ b/scripts/vsp-lib.sh
> @@ -898,6 +898,7 @@ vsp_runner() {
>  	local count=10
>  	local pause=
>  	local skip=7
> +	local queue_late=
>  
>  	for option in $* ; do
>  		case $option in
> @@ -916,6 +917,8 @@ vsp_runner() {
>  		--skip=*)
>  			skip=${option/--skip=/}
>  			;;
> +		--queue-late)
> +			queue_late=queue-late
>  
>  		*)
>  			return 1
> @@ -959,6 +962,7 @@ vsp_runner() {
>  
>  	$yavta -c$count -n $buffers ${format:+-f $format} ${size:+-s $size} \
>  		${skip:+--skip $skip} ${file:+--file=$file} ${pause:+-p$pause} \
> +		${queue_late:+--$queue_late} \
>  		$videodev | ./logger.sh $entity >> $logfile
>  }
>  

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 7/7] vsp-lib: Reset controls to defaults on each test run
  2018-12-04 15:51 ` [VSP-Tests PATCH 7/7] vsp-lib: Reset controls to defaults on each test run Kieran Bingham
@ 2019-02-17 12:01   ` Laurent Pinchart
  2019-02-17 21:00     ` Kieran Bingham
  0 siblings, 1 reply; 24+ messages in thread
From: Laurent Pinchart @ 2019-02-17 12:01 UTC (permalink / raw)
  To: Kieran Bingham; +Cc: Kieran Bingham, linux-renesas-soc

Hi Kieran,

Thank you for the patch.

On Tue, Dec 04, 2018 at 03:51:46PM +0000, Kieran Bingham wrote:
> Some of our tests set flipping and rotation controls, and the VSP cell

cell ?

> can be used again by later tests. If these controls are not reset, then
> that operation is applied to later tests incorrectly causing that test

s/that test/those tests/ ?

> to fail.
> 
> In an ideal world, tests should clean up after themselves, and leave the
> system in a known state. However the world is not ideal and we would not
> be able to guarantee any previous system state before a test was run
> anyway. Therefore it is more effective to reset state at the beginning
> of a test.
> 
> To repair this - reset all control values to their defaults at the start
> of every test during test_init()
> 
> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  scripts/vsp-lib.sh | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
> index 3d2792707d24..33442816f208 100755
> --- a/scripts/vsp-lib.sh
> +++ b/scripts/vsp-lib.sh
> @@ -94,6 +94,14 @@ vsp1_set_control() {
>  	$yavta --no-query -w "$control $value" $subdev | ./logger.sh "$entity" >> $logfile
>  }
>  
> +vsp1_reset_controls() {
> +	local entity=$1
> +	local subdev=$(vsp1_entity_subdev $entity)
> +
> +	echo "Resetting controls on $subdev" | ./logger.sh "$entity" >> $logfile
> +	$yavta --no-query --reset-controls $subdev | ./logger.sh "$entity" >> $logfile
> +}
> +
>  # -----------------------------------------------------------------------------
>  # Reference frame generation
>  #
> @@ -1082,6 +1090,9 @@ test_init() {
>  	dev=$(vsp1_device $mdev)
>  	echo "Using device $mdev ($dev)" | ./logger.sh config >> $logfile
>  
> +	# Reset any rotation or flipping controls
> +	vsp1_reset_controls wpf.0
> +
>  	vsp_runner=./vsp-runner.sh
>  }
>  

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 1/7] tests: add pseudo platform test
  2019-02-17  2:41   ` Laurent Pinchart
@ 2019-02-17 20:41     ` Kieran Bingham
  0 siblings, 0 replies; 24+ messages in thread
From: Kieran Bingham @ 2019-02-17 20:41 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc

Hi Laurent,

On 17/02/2019 02:41, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Tue, Dec 04, 2018 at 03:51:40PM +0000, Kieran Bingham wrote:
>> Provide an initial test which can run as part of the test suite.
>> This test will report the platform and kernel version, along with
>> the identified paths of required utilities.
>>
>> This will aid in ensuring that required tools are available on a
>> running platform - and report the kernel and platform details in
>> any test suite output for clarification of results.
>>
>> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>> ---
>>  tests/vsp-unit-test-0000.sh | 20 ++++++++++++++++++++
>>  1 file changed, 20 insertions(+)
>>  create mode 100755 tests/vsp-unit-test-0000.sh
>>
>> diff --git a/tests/vsp-unit-test-0000.sh b/tests/vsp-unit-test-0000.sh
>> new file mode 100755
>> index 000000000000..144cfc677b32
>> --- /dev/null
>> +++ b/tests/vsp-unit-test-0000.sh
>> @@ -0,0 +1,20 @@
>> +#!/bin/sh
>> +
>> +# Report testing conditions
>> +
>> +model=`cat /sys/firmware/devicetree/base/model`
> 
> Please use $(...) instead of `...`, it's more readable.

Sure.

> 
>> +
>> +echo "Test Conditions:"
>> +
>> +function check_all() {
>> +	echo "  Platform: " "$model"
>> +	echo "  Kernel release: " `uname -r`
>> +	echo "  convert: " `which convert`
>> +	echo "  compare: " `which compare`
>> +	echo "  killall: " `which killall`
>> +	echo "  raw2rgbpnm: " `which raw2rgbpnm`
>> +	echo "  stress: " `which stress`
>> +	echo "  yavta: " `which yavta`
>> +}
>> +
>> +check_all | column -ts ":"
> 
> Could we remove the dependency on the column tool ? I don't have it in
> my buildroot environment :-) I could of course add it, but in this case
> I think it's easy enough to align the columns manually.

Ahh yes - good point. A few spaces aught to do it :)

and s/`.*`/$(.*)/ here too :)

> 
> With these fixed,
> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 

-- 
Regards
--
Kieran

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 2/7] tests: Provide copy test to validate 1xN streams
  2019-02-17  2:47   ` Laurent Pinchart
@ 2019-02-17 20:43     ` Kieran Bingham
  0 siblings, 0 replies; 24+ messages in thread
From: Kieran Bingham @ 2019-02-17 20:43 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc

Hi Laurent,

On 17/02/2019 02:47, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Tue, Dec 04, 2018 at 03:51:41PM +0000, Kieran Bingham wrote:
>> Validate that a 1xN stream can be read through the RPF and written
>> through the WPF.
>>
>> The test framework does not currently support processing images where
>> the stride does not match the output width - so the testing is currently
>> limited to testing only the vertical direction in this aspect.
>>
>> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>> ---
>>  tests/vsp-unit-test-0025.sh | 45 +++++++++++++++++++++++++++++++++++++
>>  1 file changed, 45 insertions(+)
>>  create mode 100755 tests/vsp-unit-test-0025.sh
>>
>> diff --git a/tests/vsp-unit-test-0025.sh b/tests/vsp-unit-test-0025.sh
>> new file mode 100755
>> index 000000000000..57a1fac6e369
>> --- /dev/null
>> +++ b/tests/vsp-unit-test-0025.sh
>> @@ -0,0 +1,45 @@
>> +#!/bin/sh
>> +
>> +#
>> +# Test pipelines which have a single pixel dimension. Use a RPF -> WPF
>> +# pipeline with identical input and output formats to generate our output.
>> +#
>> +
>> +. ./vsp-lib.sh
>> +
>> +features="rpf.0 uds wpf.0"
>> +formats="RGB24 ARGB32"
>> +
>> +# Input is directly copied to the output. No change in format or size.
>> +test_copy() {
>> +	local format=$1
>> +	local insize=$2
>> +
>> +	test_start "copying $insize in $format"
>> +
>> +	pipe_configure rpf-wpf 0 0
>> +	format_configure rpf-wpf 0 0 $format $insize $format
>> +
>> +	vsp_runner rpf.0 &
>> +	vsp_runner wpf.0
>> +
>> +	local result=$(compare_frames)
>> +
>> +	test_complete $result
>> +}
>> +
>> +test_main() {
>> +	local format
>> +
>> +	for format in $formats ; do
>> +		test_copy $format 1024x768
>> +		test_copy $format 128x128
>> +		test_copy $format 128x1
> 
> If the purpose is to test Nx1 and 1xN, do we need the first two
> resolutions ? Shouldn't the test_start message describe the exact
> purpose ?

Hrm ... It's been too long. I don't recall if the first two were a
copy/paste left-over or if they had more meaning.

They don't look like they have much meaning now though - so they can be
removed. I'm all for speeding up the full test-suite :-)


> 
>> +
>> +		# Skipped : Test framework does not yet support strides != width
>> +		#test_copy $format 1x128
>> +	done
>> +}
>> +
>> +test_init $0 "$features"
>> +test_run
> 

-- 
Regards
--
Kieran

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 3/7] vsp-lib: Provide command line argument parsing
  2019-02-17  2:48   ` Laurent Pinchart
@ 2019-02-17 20:47     ` Kieran Bingham
  0 siblings, 0 replies; 24+ messages in thread
From: Kieran Bingham @ 2019-02-17 20:47 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc

Hi Laurent,

On 17/02/2019 02:48, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Tue, Dec 04, 2018 at 03:51:42PM +0000, Kieran Bingham wrote:
>> Extend the vsp-lib to support command line parsing for all tests.  The
>> arguments parsed here should be common to all tests, and initially
>> provide shell level verbose debug output, and the option to easily keep
>> frames output by the VSP1.
>>
>> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>> ---
>>  scripts/vsp-lib.sh | 34 ++++++++++++++++++++++++++++++++++
>>  1 file changed, 34 insertions(+)
>>
>> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
>> index 0f3992a7827e..56969606382f 100755
>> --- a/scripts/vsp-lib.sh
>> +++ b/scripts/vsp-lib.sh
>> @@ -1094,3 +1094,37 @@ test_complete() {
>>  test_run() {
>>  	test_main | ./logger.sh error >> $logfile
>>  }
>> +
>> +# ------------------------------------------------------------------------------
>> +# Common argument parsing
>> +#
>> +# non-recognised arguments are restored, to allow tests to implement their own
>> +# parsing if necessary.
>> +
>> +POSITIONAL=()
> 
> This work in bash only :-( Any chance to implement a solution that would
> work in ash too ?
> 

Hrm ... I'll have to look into that. Or drop the 'feature'. This was
nice because it let me parse arguments I could handle here, and just
pass everything else through.


>> +while [[ $# -gt 0 ]]
>> +do
>> +case $1 in
>> +	-x|--debug)
>> +		set -x;
>> +		shift
>> +		;;
>> +	-k|--keep-frames)
>> +		export VSP_KEEP_FRAMES=1
>> +		shift
>> +		;;
>> +	-h|--help)
>> +		echo "$(basename $0): VSP Test library"
>> +		echo "  -x|--debug          enable shell debug"
>> +		echo "  -k|--keep-frames    keep generated and captured frames"
>> +		echo "  -h|--help           this help"
>> +		exit
>> +		shift
>> +		;;
>> +	*)    # unknown option
>> +		POSITIONAL+=("$1") # save it in an array for later
>> +		shift # past argument
>> +		;;
>> +esac
>> +done
>> +set -- "${POSITIONAL[@]}" # restore positional parameters
> 

-- 
Regards
--
Kieran

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 4/7] vsp-lib: trivial: Fix spelling of Reference
  2019-02-17  2:42   ` Laurent Pinchart
@ 2019-02-17 20:48     ` Kieran Bingham
  0 siblings, 0 replies; 24+ messages in thread
From: Kieran Bingham @ 2019-02-17 20:48 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc

Hi Laurent,

On 17/02/2019 02:42, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Tue, Dec 04, 2018 at 03:51:43PM +0000, Kieran Bingham wrote:
>> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 

Thanks, This has no dependency on the rest of the series. Feel free to
apply early if you wish.

--
Kieran


>> ---
>>  scripts/vsp-lib.sh | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
>> index 56969606382f..cf15a045ea8c 100755
>> --- a/scripts/vsp-lib.sh
>> +++ b/scripts/vsp-lib.sh
>> @@ -95,7 +95,7 @@ vsp1_set_control() {
>>  }
>>  
>>  # -----------------------------------------------------------------------------
>> -# Referance frame generation
>> +# Reference frame generation
>>  #
>>  
>>  reference_frame() {
> 

-- 
Regards
--
Kieran

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 5/7] vsp-lib: Allow forcing pixel perfect comparisons
  2019-02-17 11:59   ` Laurent Pinchart
@ 2019-02-17 20:51     ` Kieran Bingham
  2019-02-19 16:35       ` Laurent Pinchart
  0 siblings, 1 reply; 24+ messages in thread
From: Kieran Bingham @ 2019-02-17 20:51 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc

Hi Laurent,

On 17/02/2019 11:59, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Tue, Dec 04, 2018 at 03:51:44PM +0000, Kieran Bingham wrote:
>> Provide a means for the tester to request pixel perfect matches on tests.
>> This can be either through setting the environment variable VSP_PIXEL_PERFECT, or
>> by passing either '-p' or '--pixel-perfect' on the test command line.
> 
> What's the use case for this ?

Some of the tests (i.e. scaling) provide 'fuzzy comparisons' to allow "a
close match" to pass.

It can be desirable to 'turn that off' so that you can generate a set of
frames (using the '-k' option) to diff the expected against the actual
frames.

This helps while improving the system without making changes to the
specific tests.

It could be called 'disable fuzzy matching' if you like?


> 
>> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>> ---
>>  scripts/vsp-lib.sh | 11 ++++++++++-
>>  1 file changed, 10 insertions(+), 1 deletion(-)
>>
>> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
>> index cf15a045ea8c..9140254c7459 100755
>> --- a/scripts/vsp-lib.sh
>> +++ b/scripts/vsp-lib.sh
>> @@ -287,7 +287,11 @@ compare_frames() {
>>  	params=${params//)/_}
>>  	params=$pipe-$in_fmt-$out_fmt-$size$params
>>  
>> -	if [ x$__vsp_pixel_perfect != xtrue ] ; then
>> +	# The system can hint when pixel-perfection is not supported,
>> +	# however the user can override to force this requirement with
>> +	# VSP_PIXEL_PERFECT=1 in the environment or by passing -p, or
>> +	# --pixel-perfect on the commandline.
>> +	if [ x$__vsp_pixel_perfect != xtrue -a x$VSP_PIXEL_PERFECT != x1 ] ; then
>>  		method=fuzzy
>>  	fi
>>  
>> @@ -1113,10 +1117,15 @@ case $1 in
>>  		export VSP_KEEP_FRAMES=1
>>  		shift
>>  		;;
>> +	-p|--pixel-perfect)
>> +		export VSP_PIXEL_PERFECT=1
>> +		shift
>> +		;;
>>  	-h|--help)
>>  		echo "$(basename $0): VSP Test library"
>>  		echo "  -x|--debug          enable shell debug"
>>  		echo "  -k|--keep-frames    keep generated and captured frames"
>> +		echo "  -p|--pixel-perfect  frames must match with pixel perfection"
>>  		echo "  -h|--help           this help"
>>  		exit
>>  		shift
> 

-- 
Regards
--
Kieran

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 6/7] vsp-lib: Support late queuing of buffers in yavta
  2019-02-17 12:00   ` Laurent Pinchart
@ 2019-02-17 20:57     ` Kieran Bingham
  2019-02-19 16:37       ` Laurent Pinchart
  0 siblings, 1 reply; 24+ messages in thread
From: Kieran Bingham @ 2019-02-17 20:57 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc

Hi Laurent,

On 17/02/2019 12:00, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Tue, Dec 04, 2018 at 03:51:45PM +0000, Kieran Bingham wrote:
>> Provide the pass-through option of --queue-late to vsp-runner, to request
>> that yavta will queue frames after the stream has started.
> 
> As with 5/7, what is the use case for this ?

I'm sure it had a very good use case at the time.
Please ask 2 months ago Kieran ... Current Kieran has no idea :)

And then let me borrow your time machine - I have some good uses for it :D

I remember asking  you about this option in fact, but I can't remember
what I was testing specifically, or why I have added this here.

Maybe I was using it to investigate something ... It might perhaps be
useful to add a test which uses this to make sure the VSP1 is happy with
late queued frames?

>> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
>> ---
>>  scripts/vsp-lib.sh | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
>> index 9140254c7459..3d2792707d24 100755
>> --- a/scripts/vsp-lib.sh
>> +++ b/scripts/vsp-lib.sh
>> @@ -898,6 +898,7 @@ vsp_runner() {
>>  	local count=10
>>  	local pause=
>>  	local skip=7
>> +	local queue_late=
>>  
>>  	for option in $* ; do
>>  		case $option in
>> @@ -916,6 +917,8 @@ vsp_runner() {
>>  		--skip=*)
>>  			skip=${option/--skip=/}
>>  			;;
>> +		--queue-late)
>> +			queue_late=queue-late

erm ... missing ";;"?

>>  
>>  		*)
>>  			return 1
>> @@ -959,6 +962,7 @@ vsp_runner() {
>>  
>>  	$yavta -c$count -n $buffers ${format:+-f $format} ${size:+-s $size} \
>>  		${skip:+--skip $skip} ${file:+--file=$file} ${pause:+-p$pause} \
>> +		${queue_late:+--$queue_late} \
>>  		$videodev | ./logger.sh $entity >> $logfile
>>  }
>>  
> 

-- 
Regards
--
Kieran

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 7/7] vsp-lib: Reset controls to defaults on each test run
  2019-02-17 12:01   ` Laurent Pinchart
@ 2019-02-17 21:00     ` Kieran Bingham
  0 siblings, 0 replies; 24+ messages in thread
From: Kieran Bingham @ 2019-02-17 21:00 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: Kieran Bingham, linux-renesas-soc

Hi Laurent,

On 17/02/2019 12:01, Laurent Pinchart wrote:
> Hi Kieran,
> 
> Thank you for the patch.
> 
> On Tue, Dec 04, 2018 at 03:51:46PM +0000, Kieran Bingham wrote:
>> Some of our tests set flipping and rotation controls, and the VSP cell
> 
> cell ?

Hrm ... Block, Module, Instance ?
What is your preferred name for a particular instance of the HW entity ?

I think VSP instance might be the right choice.

> 
>> can be used again by later tests. If these controls are not reset, then
>> that operation is applied to later tests incorrectly causing that test
> 
> s/that test/those tests/ ?

Yes, agreed.


> 
>> to fail.
>>
>> In an ideal world, tests should clean up after themselves, and leave the
>> system in a known state. However the world is not ideal and we would not
>> be able to guarantee any previous system state before a test was run
>> anyway. Therefore it is more effective to reset state at the beginning
>> of a test.
>>
>> To repair this - reset all control values to their defaults at the start
>> of every test during test_init()
>>
>> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> 
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
>> ---
>>  scripts/vsp-lib.sh | 11 +++++++++++
>>  1 file changed, 11 insertions(+)
>>
>> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
>> index 3d2792707d24..33442816f208 100755
>> --- a/scripts/vsp-lib.sh
>> +++ b/scripts/vsp-lib.sh
>> @@ -94,6 +94,14 @@ vsp1_set_control() {
>>  	$yavta --no-query -w "$control $value" $subdev | ./logger.sh "$entity" >> $logfile
>>  }
>>  
>> +vsp1_reset_controls() {
>> +	local entity=$1
>> +	local subdev=$(vsp1_entity_subdev $entity)
>> +
>> +	echo "Resetting controls on $subdev" | ./logger.sh "$entity" >> $logfile
>> +	$yavta --no-query --reset-controls $subdev | ./logger.sh "$entity" >> $logfile
>> +}
>> +
>>  # -----------------------------------------------------------------------------
>>  # Reference frame generation
>>  #
>> @@ -1082,6 +1090,9 @@ test_init() {
>>  	dev=$(vsp1_device $mdev)
>>  	echo "Using device $mdev ($dev)" | ./logger.sh config >> $logfile
>>  
>> +	# Reset any rotation or flipping controls
>> +	vsp1_reset_controls wpf.0
>> +
>>  	vsp_runner=./vsp-runner.sh
>>  }
>>  
> 

-- 
Regards
--
Kieran

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 5/7] vsp-lib: Allow forcing pixel perfect comparisons
  2019-02-17 20:51     ` Kieran Bingham
@ 2019-02-19 16:35       ` Laurent Pinchart
  0 siblings, 0 replies; 24+ messages in thread
From: Laurent Pinchart @ 2019-02-19 16:35 UTC (permalink / raw)
  To: Kieran Bingham; +Cc: Kieran Bingham, linux-renesas-soc

Hi Kieran,

On Sun, Feb 17, 2019 at 08:51:52PM +0000, Kieran Bingham wrote:
> On 17/02/2019 11:59, Laurent Pinchart wrote:
> > On Tue, Dec 04, 2018 at 03:51:44PM +0000, Kieran Bingham wrote:
> >> Provide a means for the tester to request pixel perfect matches on tests.
> >> This can be either through setting the environment variable VSP_PIXEL_PERFECT, or
> >> by passing either '-p' or '--pixel-perfect' on the test command line.
> > 
> > What's the use case for this ?
> 
> Some of the tests (i.e. scaling) provide 'fuzzy comparisons' to allow "a
> close match" to pass.
> 
> It can be desirable to 'turn that off' so that you can generate a set of
> frames (using the '-k' option) to diff the expected against the actual
> frames.

Ideally we should improve the reference frame generation tool to
generate pixel-perfect frames :-) In the meantime I agree this could be
useful, but I think we should then integrate this better by marking a
test run as reference, saving frames to a good location, and then
reusing them for the subsequent runs. It's a bit too manual in its
proposed form for an automated test suite :-)

> This helps while improving the system without making changes to the
> specific tests.
> 
> It could be called 'disable fuzzy matching' if you like?
> 
> >> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> >> ---
> >>  scripts/vsp-lib.sh | 11 ++++++++++-
> >>  1 file changed, 10 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
> >> index cf15a045ea8c..9140254c7459 100755
> >> --- a/scripts/vsp-lib.sh
> >> +++ b/scripts/vsp-lib.sh
> >> @@ -287,7 +287,11 @@ compare_frames() {
> >>  	params=${params//)/_}
> >>  	params=$pipe-$in_fmt-$out_fmt-$size$params
> >>  
> >> -	if [ x$__vsp_pixel_perfect != xtrue ] ; then
> >> +	# The system can hint when pixel-perfection is not supported,
> >> +	# however the user can override to force this requirement with
> >> +	# VSP_PIXEL_PERFECT=1 in the environment or by passing -p, or
> >> +	# --pixel-perfect on the commandline.
> >> +	if [ x$__vsp_pixel_perfect != xtrue -a x$VSP_PIXEL_PERFECT != x1 ] ; then
> >>  		method=fuzzy
> >>  	fi
> >>  
> >> @@ -1113,10 +1117,15 @@ case $1 in
> >>  		export VSP_KEEP_FRAMES=1
> >>  		shift
> >>  		;;
> >> +	-p|--pixel-perfect)
> >> +		export VSP_PIXEL_PERFECT=1
> >> +		shift
> >> +		;;
> >>  	-h|--help)
> >>  		echo "$(basename $0): VSP Test library"
> >>  		echo "  -x|--debug          enable shell debug"
> >>  		echo "  -k|--keep-frames    keep generated and captured frames"
> >> +		echo "  -p|--pixel-perfect  frames must match with pixel perfection"
> >>  		echo "  -h|--help           this help"
> >>  		exit
> >>  		shift

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [VSP-Tests PATCH 6/7] vsp-lib: Support late queuing of buffers in yavta
  2019-02-17 20:57     ` Kieran Bingham
@ 2019-02-19 16:37       ` Laurent Pinchart
  0 siblings, 0 replies; 24+ messages in thread
From: Laurent Pinchart @ 2019-02-19 16:37 UTC (permalink / raw)
  To: Kieran Bingham; +Cc: Kieran Bingham, linux-renesas-soc

Hi Kieran,

On Sun, Feb 17, 2019 at 08:57:51PM +0000, Kieran Bingham wrote:
> On 17/02/2019 12:00, Laurent Pinchart wrote:
> > On Tue, Dec 04, 2018 at 03:51:45PM +0000, Kieran Bingham wrote:
> >> Provide the pass-through option of --queue-late to vsp-runner, to request
> >> that yavta will queue frames after the stream has started.
> > 
> > As with 5/7, what is the use case for this ?
> 
> I'm sure it had a very good use case at the time.
> Please ask 2 months ago Kieran ... Current Kieran has no idea :)
> 
> And then let me borrow your time machine - I have some good uses for it :D
> 
> I remember asking  you about this option in fact, but I can't remember
> what I was testing specifically, or why I have added this here.
> 
> Maybe I was using it to investigate something ... It might perhaps be
> useful to add a test which uses this to make sure the VSP1 is happy with
> late queued frames?

At least part of this is handled internally in videobuf2 without being
exposed to the VSP1 driver. If the exposed part is considered to be
possibly problematic then I would indeed prefer to add a specific test
to cover this.

> >> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> >> ---
> >>  scripts/vsp-lib.sh | 4 ++++
> >>  1 file changed, 4 insertions(+)
> >>
> >> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
> >> index 9140254c7459..3d2792707d24 100755
> >> --- a/scripts/vsp-lib.sh
> >> +++ b/scripts/vsp-lib.sh
> >> @@ -898,6 +898,7 @@ vsp_runner() {
> >>  	local count=10
> >>  	local pause=
> >>  	local skip=7
> >> +	local queue_late=
> >>  
> >>  	for option in $* ; do
> >>  		case $option in
> >> @@ -916,6 +917,8 @@ vsp_runner() {
> >>  		--skip=*)
> >>  			skip=${option/--skip=/}
> >>  			;;
> >> +		--queue-late)
> >> +			queue_late=queue-late
> 
> erm ... missing ";;"?
> 
> >>  
> >>  		*)
> >>  			return 1
> >> @@ -959,6 +962,7 @@ vsp_runner() {
> >>  
> >>  	$yavta -c$count -n $buffers ${format:+-f $format} ${size:+-s $size} \
> >>  		${skip:+--skip $skip} ${file:+--file=$file} ${pause:+-p$pause} \
> >> +		${queue_late:+--$queue_late} \
> >>  		$videodev | ./logger.sh $entity >> $logfile
> >>  }
> >>  

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 24+ messages in thread

end of thread, other threads:[~2019-02-19 16:37 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-04 15:51 [VSP-Tests PATCH 0/7] Reset controls and unloved patches Kieran Bingham
2018-12-04 15:51 ` [VSP-Tests PATCH 1/7] tests: add pseudo platform test Kieran Bingham
2019-02-17  2:41   ` Laurent Pinchart
2019-02-17 20:41     ` Kieran Bingham
2018-12-04 15:51 ` [VSP-Tests PATCH 2/7] tests: Provide copy test to validate 1xN streams Kieran Bingham
2019-02-17  2:47   ` Laurent Pinchart
2019-02-17 20:43     ` Kieran Bingham
2018-12-04 15:51 ` [VSP-Tests PATCH 3/7] vsp-lib: Provide command line argument parsing Kieran Bingham
2019-02-17  2:48   ` Laurent Pinchart
2019-02-17 20:47     ` Kieran Bingham
2018-12-04 15:51 ` [VSP-Tests PATCH 4/7] vsp-lib: trivial: Fix spelling of Reference Kieran Bingham
2019-02-17  2:42   ` Laurent Pinchart
2019-02-17 20:48     ` Kieran Bingham
2018-12-04 15:51 ` [VSP-Tests PATCH 5/7] vsp-lib: Allow forcing pixel perfect comparisons Kieran Bingham
2019-02-17 11:59   ` Laurent Pinchart
2019-02-17 20:51     ` Kieran Bingham
2019-02-19 16:35       ` Laurent Pinchart
2018-12-04 15:51 ` [VSP-Tests PATCH 6/7] vsp-lib: Support late queuing of buffers in yavta Kieran Bingham
2019-02-17 12:00   ` Laurent Pinchart
2019-02-17 20:57     ` Kieran Bingham
2019-02-19 16:37       ` Laurent Pinchart
2018-12-04 15:51 ` [VSP-Tests PATCH 7/7] vsp-lib: Reset controls to defaults on each test run Kieran Bingham
2019-02-17 12:01   ` Laurent Pinchart
2019-02-17 21:00     ` Kieran Bingham

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).