[PATCH 0/2] testsuite: add new tags: check-output-{match,returns}

[PATCH 0/2] testsuite: add new tags: check-output-{match,returns}

[Luc Van Oostenryck]

This two patches should help to write IR-related testcases by
allowing an easy way to check that every occurence of a given
instruction has a specific operand or that all functions
have the same specified return value.

Luc Van Oostenryck (2):
  testsuite: add a new tag: check-output-match
  testsuite: add a new tag: check-output-returns

 Documentation/test-suite.rst | 13 +++++++++
 validation/test-suite        | 56 ++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)


base-commit: 49c98aa3ed1b315ed2f4fbe44271ecd5bdd9cbc7
-- 
2.29.2

[PATCH 1/2] testsuite: add a new tag: check-output-match

[Luc Van Oostenryck]

The current tags check-output-contains/excludes/pattern are
quite powerful, universal, but they often need 'complex' regular
expressions with escaping which make them not so nice to read.

For testing IR results, a very common pattern is:
	this instruction must have this (kind of) operand.

So, make a new tag for this. It does nothing than can't be done
with done with the previous ones, on  the contrary, but is much
simpler to use:
	check-output-match(instruction): operand

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 Documentation/test-suite.rst |  8 ++++++++
 validation/test-suite        | 29 +++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)

diff --git a/Documentation/test-suite.rst b/Documentation/test-suite.rst
index 333106ee138a..4ff2db2f8777 100644
--- a/Documentation/test-suite.rst
+++ b/Documentation/test-suite.rst
@@ -88,6 +88,14 @@ Tag's syntax
 	of the number of times the pattern should occur in the output.
 	If *min* or *max* is ``-`` the corresponding check is ignored.
 
+``check-output-match(``\ *start*\ ``):`` *pattern*
+
+	Check that in the output (stdout) all lines starting with the
+	first pattern also contains the second pattern. This should be
+	reserved for matching IR instructions since the '.$size' suffix
+	is ignored in the first pattern but is expected to be followed
+	by a space character.
+
 Using test-suite
 ================
 
diff --git a/validation/test-suite b/validation/test-suite
index f7d992dc7c8c..1f229439d699 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -77,6 +77,7 @@ get_tag_value()
 	check_output_contains=0
 	check_output_excludes=0
 	check_output_pattern=0
+	check_output_match=0
 	check_arch_ignore=""
 	check_arch_only=""
 	check_assert=""
@@ -100,6 +101,7 @@ get_tag_value()
 		check-output-contains:)	check_output_contains=1 ;;
 		check-output-excludes:)	check_output_excludes=1 ;;
 		check-output-pattern)	check_output_pattern=1 ;;
+		check-output-match)	check_output_match=1 ;;
 		check-arch-ignore:)	arch=$(uname -m)
 					check_arch_ignore="$val" ;;
 		check-arch-only:)	arch=$(uname -m)
@@ -204,6 +206,26 @@ minmax_patterns()
 	return $?
 }
 
+##
+match_patterns()
+{
+	ifile="$1"
+	patt="$2"
+	ofile="$3"
+	grep "$patt" "$ifile" | sed -e "s/^.*$patt(\(.*\)): *\(.*\)$/\1 \2/" | \
+	while read ins pat; do
+		echo "ins: $ins"
+		echo "pat: $pat"
+		grep -s "^	$ins\\.*[0-9]* " "$ofile" | grep -v -s -q "$pat"
+		if [ "$?" -ne 1 ]; then
+			error "	IR doesn't match '$pat'"
+			return 1
+		fi
+	done
+
+	return $?
+}
+
 ##
 # arg_file(filename) - checks if filename exists
 arg_file()
@@ -395,6 +417,13 @@ do_test()
 			test_failed=1
 		fi
 	fi
+	if [ $check_output_match -eq 1 ]; then
+		# verify the 'check-output-match($insn): $patt' tags
+		match_patterns "$file" 'check-output-match' $file.output.got
+		if [ "$?" -ne "0" ]; then
+			test_failed=1
+		fi
+	fi
 
 	if [ "$must_fail" -eq "1" ]; then
 		if [ "$test_failed" -eq "1" ]; then
-- 
2.29.2

[PATCH 2/2] testsuite: add a new tag: check-output-returns

[Luc Van Oostenryck]

The current tags check-output-contains/excludes/pattern are
quite powerful and the new check-output-match is easy to use
but it can be even simpler. Indeed, a lot of IR simplifications/
canonicalizations can be tested by checking that the expression
to be tested is equivalent to another one. This is less precise
than some more specific tests but:
* it's a big advantage because it's less sensitive to 'noise'
  like the exact number used by the pseudos or to the results
  of some new simplifications or canonicalizations
* very often, this equivalence is what really matters and not
  the exact transformation.

So, add a new utra-simple-to-use tag: just ask that all functions
of the tests return the same specified value (usually 1):
	check-output-returns: <value>

Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
---
 Documentation/test-suite.rst |  5 +++++
 validation/test-suite        | 27 +++++++++++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/Documentation/test-suite.rst b/Documentation/test-suite.rst
index 4ff2db2f8777..3181e109a104 100644
--- a/Documentation/test-suite.rst
+++ b/Documentation/test-suite.rst
@@ -96,6 +96,11 @@ Tag's syntax
 	is ignored in the first pattern but is expected to be followed
 	by a space character.
 
+``check-output-returns:`` *value*
+
+	Check that in the output (stdout) all IR return instructions
+	have the given value.
+
 Using test-suite
 ================
 
diff --git a/validation/test-suite b/validation/test-suite
index 1f229439d699..2307e4e80d75 100755
--- a/validation/test-suite
+++ b/validation/test-suite
@@ -78,6 +78,7 @@ get_tag_value()
 	check_output_excludes=0
 	check_output_pattern=0
 	check_output_match=0
+	check_output_returns=0
 	check_arch_ignore=""
 	check_arch_only=""
 	check_assert=""
@@ -102,6 +103,7 @@ get_tag_value()
 		check-output-excludes:)	check_output_excludes=1 ;;
 		check-output-pattern)	check_output_pattern=1 ;;
 		check-output-match)	check_output_match=1 ;;
+		check-output-returns:)	check_output_returns=1 ;;
 		check-arch-ignore:)	arch=$(uname -m)
 					check_arch_ignore="$val" ;;
 		check-arch-only:)	arch=$(uname -m)
@@ -226,6 +228,24 @@ match_patterns()
 	return $?
 }
 
+##
+return_patterns()
+{
+	ifile="$1"
+	patt="$2"
+	ofile="$3"
+	grep "$patt:" "$ifile" | sed -e "s/^.*$patt: *\(.*\)$/\1/" | \
+	while read ret; do
+		grep -s "^	ret\\.[0-9]" "$ofile" | grep -v -s -q "[ \$]${ret}\$"
+		if [ "$?" -ne 1 ]; then
+			error "	Return doesn't match '$ret'"
+			return 1
+		fi
+	done
+
+	return $?
+}
+
 ##
 # arg_file(filename) - checks if filename exists
 arg_file()
@@ -424,6 +444,13 @@ do_test()
 			test_failed=1
 		fi
 	fi
+	if [ $check_output_returns -eq 1 ]; then
+		# verify the 'check-output-return: $value' tags
+		return_patterns "$file" 'check-output-returns' $file.output.got
+		if [ "$?" -ne "0" ]; then
+			test_failed=1
+		fi
+	fi
 
 	if [ "$must_fail" -eq "1" ]; then
 		if [ "$test_failed" -eq "1" ]; then
-- 
2.29.2

Re: [PATCH 1/2] testsuite: add a new tag: check-output-match

[Ramsay Jones]

On 31/10/2020 20:04, Luc Van Oostenryck wrote:
> The current tags check-output-contains/excludes/pattern are
> quite powerful, universal, but they often need 'complex' regular
> expressions with escaping which make them not so nice to read.
> 
> For testing IR results, a very common pattern is:
> 	this instruction must have this (kind of) operand.
> 
> So, make a new tag for this. It does nothing than can't be done
> with done with the previous ones, on  the contrary, but is much
> simpler to use:
> 	check-output-match(instruction): operand
> 
> Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
> ---
>  Documentation/test-suite.rst |  8 ++++++++
>  validation/test-suite        | 29 +++++++++++++++++++++++++++++
>  2 files changed, 37 insertions(+)
> 
> diff --git a/Documentation/test-suite.rst b/Documentation/test-suite.rst
> index 333106ee138a..4ff2db2f8777 100644
> --- a/Documentation/test-suite.rst
> +++ b/Documentation/test-suite.rst
> @@ -88,6 +88,14 @@ Tag's syntax
>  	of the number of times the pattern should occur in the output.
>  	If *min* or *max* is ``-`` the corresponding check is ignored.
>  
> +``check-output-match(``\ *start*\ ``):`` *pattern*
> +
> +	Check that in the output (stdout) all lines starting with the
> +	first pattern also contains the second pattern. This should be
> +	reserved for matching IR instructions since the '.$size' suffix
> +	is ignored in the first pattern but is expected to be followed
> +	by a space character.
> +
>  Using test-suite
>  ================
>  
> diff --git a/validation/test-suite b/validation/test-suite
> index f7d992dc7c8c..1f229439d699 100755
> --- a/validation/test-suite
> +++ b/validation/test-suite
> @@ -77,6 +77,7 @@ get_tag_value()
>  	check_output_contains=0
>  	check_output_excludes=0
>  	check_output_pattern=0
> +	check_output_match=0
>  	check_arch_ignore=""
>  	check_arch_only=""
>  	check_assert=""
> @@ -100,6 +101,7 @@ get_tag_value()
>  		check-output-contains:)	check_output_contains=1 ;;
>  		check-output-excludes:)	check_output_excludes=1 ;;
>  		check-output-pattern)	check_output_pattern=1 ;;
> +		check-output-match)	check_output_match=1 ;;
>  		check-arch-ignore:)	arch=$(uname -m)
>  					check_arch_ignore="$val" ;;
>  		check-arch-only:)	arch=$(uname -m)
> @@ -204,6 +206,26 @@ minmax_patterns()
>  	return $?
>  }
>  
> +##
> +match_patterns()
> +{
> +	ifile="$1"
> +	patt="$2"
> +	ofile="$3"
> +	grep "$patt" "$ifile" | sed -e "s/^.*$patt(\(.*\)): *\(.*\)$/\1 \2/" | \
> +	while read ins pat; do
> +		echo "ins: $ins"
> +		echo "pat: $pat"

Left-over debug?

ATB,
Ramsay Jones

> +		grep -s "^	$ins\\.*[0-9]* " "$ofile" | grep -v -s -q "$pat"
> +		if [ "$?" -ne 1 ]; then
> +			error "	IR doesn't match '$pat'"
> +			return 1
> +		fi
> +	done
> +
> +	return $?
> +}
> +
>  ##
>  # arg_file(filename) - checks if filename exists
>  arg_file()
> @@ -395,6 +417,13 @@ do_test()
>  			test_failed=1
>  		fi
>  	fi
> +	if [ $check_output_match -eq 1 ]; then
> +		# verify the 'check-output-match($insn): $patt' tags
> +		match_patterns "$file" 'check-output-match' $file.output.got
> +		if [ "$?" -ne "0" ]; then
> +			test_failed=1
> +		fi
> +	fi
>  
>  	if [ "$must_fail" -eq "1" ]; then
>  		if [ "$test_failed" -eq "1" ]; then
> 

Re: [PATCH 1/2] testsuite: add a new tag: check-output-match

[Luc Van Oostenryck]

On Sat, Oct 31, 2020 at 08:50:37PM +0000, Ramsay Jones wrote:
> On 31/10/2020 20:04, Luc Van Oostenryck wrote:
> > +	grep "$patt" "$ifile" | sed -e "s/^.*$patt(\(.*\)): *\(.*\)$/\1 \2/" | \
> > +	while read ins pat; do
> > +		echo "ins: $ins"
> > +		echo "pat: $pat"
> 
> Left-over debug?

Grnn, yes. Thank you for noticing this!

-- Luc