Re: [VSP-Tests PATCH 5/7] vsp-lib: Allow forcing pixel perfect comparisons

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
To: Kieran Bingham <kieran.bingham@ideasonboard.com>
Cc: Kieran Bingham <kieran.bingham+renesas@ideasonboard.com>,
	linux-renesas-soc@vger.kernel.org
Subject: Re: [VSP-Tests PATCH 5/7] vsp-lib: Allow forcing pixel perfect comparisons
Date: Tue, 19 Feb 2019 18:35:37 +0200	[thread overview]
Message-ID: <20190219163537.GD3526@pendragon.ideasonboard.com> (raw)
In-Reply-To: <c69a16fc-98d5-8fe9-0c1d-7b06995b4ced@ideasonboard.com>

Hi Kieran,

On Sun, Feb 17, 2019 at 08:51:52PM +0000, Kieran Bingham wrote:
> On 17/02/2019 11:59, Laurent Pinchart wrote:
> > On Tue, Dec 04, 2018 at 03:51:44PM +0000, Kieran Bingham wrote:
> >> Provide a means for the tester to request pixel perfect matches on tests.
> >> This can be either through setting the environment variable VSP_PIXEL_PERFECT, or
> >> by passing either '-p' or '--pixel-perfect' on the test command line.
> > 
> > What's the use case for this ?
> 
> Some of the tests (i.e. scaling) provide 'fuzzy comparisons' to allow "a
> close match" to pass.
> 
> It can be desirable to 'turn that off' so that you can generate a set of
> frames (using the '-k' option) to diff the expected against the actual
> frames.

Ideally we should improve the reference frame generation tool to
generate pixel-perfect frames :-) In the meantime I agree this could be
useful, but I think we should then integrate this better by marking a
test run as reference, saving frames to a good location, and then
reusing them for the subsequent runs. It's a bit too manual in its
proposed form for an automated test suite :-)

> This helps while improving the system without making changes to the
> specific tests.
> 
> It could be called 'disable fuzzy matching' if you like?
> 
> >> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
> >> ---
> >>  scripts/vsp-lib.sh | 11 ++++++++++-
> >>  1 file changed, 10 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/scripts/vsp-lib.sh b/scripts/vsp-lib.sh
> >> index cf15a045ea8c..9140254c7459 100755
> >> --- a/scripts/vsp-lib.sh
> >> +++ b/scripts/vsp-lib.sh
> >> @@ -287,7 +287,11 @@ compare_frames() {
> >>  	params=${params//)/_}
> >>  	params=$pipe-$in_fmt-$out_fmt-$size$params
> >>  
> >> -	if [ x$__vsp_pixel_perfect != xtrue ] ; then
> >> +	# The system can hint when pixel-perfection is not supported,
> >> +	# however the user can override to force this requirement with
> >> +	# VSP_PIXEL_PERFECT=1 in the environment or by passing -p, or
> >> +	# --pixel-perfect on the commandline.
> >> +	if [ x$__vsp_pixel_perfect != xtrue -a x$VSP_PIXEL_PERFECT != x1 ] ; then
> >>  		method=fuzzy
> >>  	fi
> >>  
> >> @@ -1113,10 +1117,15 @@ case $1 in
> >>  		export VSP_KEEP_FRAMES=1
> >>  		shift
> >>  		;;
> >> +	-p|--pixel-perfect)
> >> +		export VSP_PIXEL_PERFECT=1
> >> +		shift
> >> +		;;
> >>  	-h|--help)
> >>  		echo "$(basename $0): VSP Test library"
> >>  		echo "  -x|--debug          enable shell debug"
> >>  		echo "  -k|--keep-frames    keep generated and captured frames"
> >> +		echo "  -p|--pixel-perfect  frames must match with pixel perfection"
> >>  		echo "  -h|--help           this help"
> >>  		exit
> >>  		shift

-- 
Regards,

Laurent Pinchart