[PATCH 0/7] fstests: add environments for tests

[PATCH 0/7] fstests: add environments for tests

[Jan Ťulák]

This is a completely rewritten version of what I proposed some time ago. TLDR
summary at first:

  - Allows to separate preparation of the environment (full fs, damaged fs,
    lots of files...) from a test itself. So multiple tests can be run
    in exactly the same conditions...
  - ... and a single test can be run in multiple environments.
  - It do not affects existing tests.
  - Two environments and tests to demonstrate its function.

Long version:

the goal of this patch set is to allow a single test to be run in different
sitations, for example on empty filesystem, full fs, or damaged fs. it provides
an interface for scripts that can prepare the requested environment and takes
care of starting the test for each one.

The changes in existing infrastructure are small. Most of the code is a new
functionality which has to be invoked by a user or a test.

How it works: Most of the it is done in the test itself: The test sets
"$ENV_NAME" variable and calls "_environment_require $TEST_DIR". Where and when
the test do this is in hands of its author. When using the ./new script for
creating a skeleton, everything is prepared, so the author only lists desired
environments.

The test skeleton already contains code for for iterating over the environments
and running each of them. Of course, if only one environment is given, then the
test will run only once as before.

Changes in ./check script are for allowing the user to somehow limit which
environments are used through two new options: -ex and -eo, for exclude/include
some environments. However, this only affects the tests that comply with it. So
it won't select what tests to run or not. It only exports the limited list to a
test, which is then free to do whatever it want. (The autogenerated skeleton is
modified w.r.t. this.)

Already existing tests are not affected by this new functionality at all. Also,
this gives the author of a test the control over when and how the environments
should be called. But if he does not wish to be bothered with it at all, he can
ignore it. The change from what was before is small, only few more lines around
"put your code here".

Unlike in my previous patch set, I decided to not implement some persistency
between different tests using the same environment. With the current code, it
is possible to achieve it in the tests itself.

To show how it works, I modified two performance tests (an important motivation
for these patches) to work with the new environment functionality.


I hope this version is in better shape. :-)

Cheers,
Jan

Jan Ťulák (7):
  fstests: environments - add support for environments into the check
    script
  fstests: environments - new functions in common/rc for environment
    support
  fstests: environments - add option for skipping output diff from a
    test
  fstests: environments - add performance tests (as example of usage)
  fstests: environments - create the environment directory
  fstests: environments - add envs to the skeleton in ./new script and
    README
  fstests: environments - enable performance tests in ./check

 README                        |  12 +++-
 check                         | 142 ++++++++++++++++++++++++++++++++++++++---
 common/environments           |  64 +++++++++++++++++++
 common/rc                     | 143 ++++++++++++++++++++++++++++++++++++++++++
 environments/empty-files      |  77 +++++++++++++++++++++++
 environments/overlayfs        | 136 +++++++++++++++++++++++++++++++++++++++
 new                           |  37 ++++++++++-
 tests/performance/001-readdir | 123 ++++++++++++++++++++++++++++++++++++
 tests/performance/002-sqlite  | 124 ++++++++++++++++++++++++++++++++++++
 tests/performance/Makefile    |  21 +++++++
 tests/performance/group       |   7 +++
 11 files changed, 875 insertions(+), 11 deletions(-)
 create mode 100644 common/environments
 create mode 100644 environments/empty-files
 create mode 100644 environments/overlayfs
 create mode 100755 tests/performance/001-readdir
 create mode 100755 tests/performance/002-sqlite
 create mode 100644 tests/performance/Makefile
 create mode 100644 tests/performance/group

-- 
2.1.0

[PATCH 1/7] fstests: environments - add support for environments into the check script

[Jan Ťulák]

This extends the ./check script for environments support. That includes
arguments parsing for new options "-eo" and "-ex" and few functions
for loading list of available environments from "environments" directory.

The last two hunks in the patch deals with a situation when a user limits
available environments, so a test has no environment it can run in.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 check | 105 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 102 insertions(+), 3 deletions(-)

diff --git a/check b/check
index 4fa96ed..1a691e9 100755
--- a/check
+++ b/check
@@ -60,6 +60,7 @@ fi
 
 SRC_GROUPS="generic shared"
 export SRC_DIR="tests"
+export ENV_DIR="environments"
 
 usage()
 {
@@ -80,6 +81,8 @@ check options
 testlist options
     -g group[,group...]	include tests from these groups
     -x group[,group...]	exclude tests from these groups
+    -eo environment[,environment...]	test only in these environments
+    -ex environment[,environment...]	exclude these environments
     -X file		exclude individual tests
     -E external_file	exclude individual tests
     [testlist]		include tests matching names in testlist
@@ -87,6 +90,85 @@ testlist options
 	    exit 0
 }
 
+
+
+# Find all environments with setup files in given directory.
+get_environments()
+{
+	local env_list=""
+
+	# ommit "none", this one always has to exists implicitly
+	# and we want it add later
+	env_list=$(ls $ENV_DIR|grep -v "none")
+
+	echo "none $env_list"
+}
+
+# Check if all tests passed in first argument exists in a list passed
+# as a second argument.
+environments_test_existence()
+{
+	local specified  existing
+	specified="$1"
+	existing="$2"
+
+	nonexisting=$(_get_lists_difference "$specified" "$existing")
+	if [ "$nonexisting" != "" ];then
+		echo "Unknown environment(s) were passed"\
+			" as an argument: $nonexisting"
+		exit 1
+	fi
+}
+
+# find which environments are required and which are excluded and export
+# them in "$list_env"
+#
+export_filtered_environments()
+{
+	active_tests="$*"
+	sorted_tests=""
+	sorted_envs=""
+	include_implicit=false
+	existing_environments=$(get_environments)
+
+	required_environments="${ENVIRONMENT_LIST/,/ }"
+	excluded_environments="${XENVIRONMENT_LIST/,/ }"
+
+	# test for nonexisting required
+	environments_test_existence \
+		"$required_environments $excluded_environments" \
+		"$existing_environments"
+
+	# filter environments based on -ex or -eo arguments
+	if [ "$required_environments" = "" ];then
+		# we have no explicit list of envs, so include all
+		required_environments="$existing_environments"
+		include_implicit=true
+
+	elif [ $(echo "$required_environments" |\
+		grep -cw "none") -gt 0 ]
+	then
+		# If there is an explicit list, but "none" is listed there,
+		# include implicit "none" too.
+		# Otherwise "none" is ignored.
+		include_implicit=true
+	fi
+
+	# remove any excluded environment from the list
+	for xenv in $excluded_environments; do
+		required_environments=$(echo "$required_environments" |\
+			sed "s/\b$xenv\b//g")
+
+		# do not include implicit none if explicitly blocked
+		if [ "$xenv" = "none" ];then
+			include_implicit=false
+		fi
+	done
+
+	export list_env="$required_environments"
+
+}
+
 get_group_list()
 {
 	grp=$1
@@ -193,6 +275,8 @@ _prepare_test_list()
 	list=`sort -n $tmp.list | uniq`
 	rm -f $tmp.list $tmp.tmp $tmp.grep
 
+	export_filtered_environments $list
+
 	if $randomize
 	then
 		list=`echo $list | awk -f randomize.awk`
@@ -216,6 +300,14 @@ while [ $# -gt 0 ]; do
 		XGROUP_LIST="$XGROUP_LIST $xgroup"
 		;;
 
+	-eo)	environment=$2 ; shift ;
+		ENVIRONMENT_LIST="$ENVIRONMENT_LIST $environment"
+		;;
+
+	-ex)	xenvironment=$2 ; shift ;
+		XENVIRONMENT_LIST="$XENVIRONMENT_LIST $xenvironment"
+		;;
+
 	-X)	xfile=$2; shift ;
 		for d in $SRC_GROUPS $FSTYP; do
 			[ -f $SRC_DIR/$d/$xfile ] || continue
@@ -580,7 +672,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 		else
 			echo -n "	"	# prettier output with timestamps.
 		fi
-		rm -f core $seqres.notrun
+		rm -f core $seqres.notrun $seqres.noenvironment
 
 		start=`_wallclock`
 		$timestamp && echo -n "	["`date "+%T"`"]"
@@ -609,8 +701,15 @@ for section in $HOST_OPTIONS_SECTIONS; do
 
 		if [ -f $seqres.notrun ]
 		then
-		    $timestamp || echo -n " [not run] "
-		    $timestamp && echo " [not run]" && echo -n "	$seqnum -- "
+		    # tell user if the test wasn't run because
+		    # he forbid all environments this test supports
+		    # (by -eo/-ex argruments)
+		    nr="not run"
+		    if [ -f $seqres.noenvironment ];then
+		        nr="no environment available"
+		    fi
+		    $timestamp || echo -n " [$nr] "
+		    $timestamp && echo " [$nr]" && echo -n "	$seqnum -- "
 		    cat $seqres.notrun
 		    notrun="$notrun $seqnum"
 		else
-- 
2.1.0

[PATCH 2/7] fstests: environments - new functions in common/rc for environment support

[Jan Ťulák]

Adds few new functions into common/rc script. Two generic, for working
with lists, and three for manipulaction with the environments,
to be called from tests.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 common/rc | 127 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 127 insertions(+)

diff --git a/common/rc b/common/rc
index c5db0dd..e090ecd 100644
--- a/common/rc
+++ b/common/rc
@@ -50,6 +50,41 @@ _math()
 	echo "$result"
 }
 
+# Get symetric difference of two lists ($1 - $2).
+# Can work with any list of single-worded values.
+#
+_get_lists_difference()
+{
+	local a="$1"
+	local b="$2"
+	local difference=""
+
+	#a=$(echo "$a" | sed -e "s/\b_//g")
+	#b=$(echo "$b" | sed -e "s/\b_//g")
+	for item in $a;do
+        if [ $(echo $b|grep -cw $item) -eq 0 ];then
+			difference="$difference $item"
+		fi
+	done
+	echo $difference
+}
+
+# Get intersect of two lists. Can work with any list of single-worded values.
+#
+_get_lists_intersect()
+{
+	local a="$1"
+	local b="$2"
+	local intersect=""
+
+	for item in $a;do
+        if [ $(echo $b|grep -cwE "_?$item") -gt 0 ];then
+			intersect="$intersect $item"
+		fi
+	done
+	echo $intersect
+}
+
 dd()
 {
    if [ "$HOSTOS" == "Linux" ]
@@ -1074,6 +1109,98 @@ _supported_os()
     _notrun "not suitable for this OS: $HOSTOS"
 }
 
+
+# filter given supported environments by user input
+#
+_filter_environments()
+{
+	supported="$1"
+	# list_env is exported in check script
+	env="$(_get_lists_intersect "$supported" "$list_env")"
+    if [ "$env" = "" ];then
+        touch $seqres.noenvironment
+        touch $seqres.notrun
+    else
+        echo "$env"
+    fi
+}
+
+# Require an environment and make sure it is prepared - call this in a test.
+# First argument is path in which to prepare the env.
+# Optional second argument will overwrite $ENV_NAME variable for selecting
+# an environment.
+_environment_require()
+{
+	target="$1"
+	env="$2"
+	if [ "$env" = "" ]; then
+		env=$ENV_NAME
+	fi
+
+	if [ "$env" = "" ];then
+		1>&2 echo "You must export \$env "\
+			"before calling _environment_require!"
+		exit 1
+	fi
+	if [ "$target" = "" ];then
+		1>&2 echo "You must provide a target for _environment_require!"\
+			" Aborting."
+		exit 1
+	fi
+
+	if [ "$env" != "none" ];then
+		if [ ! -f "./environments/$env" ];then
+			1>&2 echo "Environment $env don't exists!"
+		fi
+		prepare_method="prepare"
+		bash ./environments/$env $prepare_method $target
+		sts=$?
+		if [ "$sts" -ne 0 ]; then
+			echo "       [skipped]"
+			1>&2 echo "Failed to prepare environment $env "\
+				" (will skip the test)!"
+			1>&2 echo ""
+			exit 1
+		fi
+	fi
+}
+
+# clean after an environment
+#
+# First argument is path in which to clean the env.
+# Optional second argument will overwrite $ENV_NAME variable for selecting
+# an environment.
+_environment_clean()
+{
+	target="$1"
+	env="$2"
+	if [ "$env" = "" ]; then
+		env=$ENV_NAME
+	fi
+
+	if [ "$env" = "" ];then
+		1>&2 echo "You must export \$env "\
+			"before calling _environment_clean!"
+		exit 1
+	fi
+	if [ "$target" = "" ];then
+		1>&2 echo "You must provide a target for _environment_clean!"\
+			" Aborting."
+		exit 1
+	fi
+	# Clean environment if there was any.
+	if [ "$env" != "none" ];then
+		bash ./environments/$env clean $target
+		sts=$?
+		if [ "$sts" != "0" ];then
+			1>&2 echo "An error happened when "\
+				"cleaning environment $env!"
+			1>&2 echo ""
+		fi
+	fi
+
+}
+
 # this test needs a scratch partition - check we're ok & unmount it
 # No post-test check of the device is required. e.g. the test intentionally
 # finishes the test with the filesystem in a corrupt state
-- 
2.1.0

[PATCH 3/7] fstests: environments - add option for skipping output diff from a test

[Jan Ťulák]

The performance tests has no "right" output for diff.
Thus, this patch allows tests to opt-out of output diffs and to
have their full output always saved into the results directory.

Changes are both in check script and common/rc.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 check     | 20 ++++++++++++++++++--
 common/rc | 16 ++++++++++++++++
 2 files changed, 34 insertions(+), 2 deletions(-)

diff --git a/check b/check
index 1a691e9..c746d71 100755
--- a/check
+++ b/check
@@ -713,12 +713,21 @@ for section in $HOST_OPTIONS_SECTIONS; do
 		    cat $seqres.notrun
 		    notrun="$notrun $seqnum"
 		else
+		    # Check for diff skip.
+		    # Some tests, like performance ones, has no "right" output,
+		    # so there is no reason to diff it. It is saved instead.
+		    skip_diff=false
+		    if [ -f $seqres.no_output_check ];then
+		        skip_diff=true
+		        rm $seqres.no_output_check
+		    fi
+
 		    if [ $sts -ne 0 ]
 		    then
 			echo -n " [failed, exit status $sts]"
 			err=true
 		    fi
-		    if [ ! -f $seq.out ]
+		    if [ ! -f $seq.out -a $skip_diff = false ]
 		    then
 			echo " - no qualified output"
 			err=true
@@ -727,7 +736,8 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			# coreutils 8.16+ changed quote formats in error messages from
 			# `foo' to 'foo'. Filter old versions to match the new version.
 			sed -i "s/\`/\'/g" $tmp.out
-			if diff $seq.out $tmp.out >/dev/null 2>&1
+			if (test $skip_diff = true) \
+				|| diff $seq.out $tmp.out >/dev/null 2>&1
 			then
 			    if $err
 			    then
@@ -736,6 +746,12 @@ for section in $HOST_OPTIONS_SECTIONS; do
 				echo "$seqnum `expr $stop - $start`" >>$tmp.time
 				echo -n " `expr $stop - $start`s"
 			    fi
+			    # if diff was skipped, we want to have the output saved
+			    if test $skip_diff = true
+			    then
+			        mv $tmp.out $seqres.out
+			        echo -n " Output saved per request of the test."
+			    fi
 			    echo ""
 			else
 			    echo " - output mismatch (see $seqres.out.bad)"
diff --git a/common/rc b/common/rc
index e090ecd..4498810 100644
--- a/common/rc
+++ b/common/rc
@@ -85,6 +85,22 @@ _get_lists_intersect()
 	echo $intersect
 }
 
+# Tell the check script that it should not check output
+# of a test, but should always save it.
+#
+_disable_output_check()
+{
+	touch $seqres.no_output_check
+}
+
+# Tell the check script that it should check output
+# of a test. Revert the effect of "_disable_output_check()" call.
+#
+_enable_output_check()
+{
+	rm $seqres.no_output_check
+}
+
 dd()
 {
    if [ "$HOSTOS" == "Linux" ]
-- 
2.1.0

[PATCH 4/7] fstests: environments - add performance tests (as example of usage)

[Jan Ťulák]

This patch adds new test category for performance tests, which utilises
the new environment architecture.

There are two simple tests. Currently, both are configured to be quick,
running about 20 seconds, so you can easily try them.

- 001 is for benchmarking readdir speed using xfs_io -c "readdir".
  In this test, an environment is used to generate NUM of empty files in
  a directory. Runtime is
- 002 is a simulation of SQLite, using fio for the testing. It uses no environment
  to show how the modified skeleton from ./new script works by default.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 tests/performance/001-readdir | 123 +++++++++++++++++++++++++++++++++++++++++
 tests/performance/002-sqlite  | 124 ++++++++++++++++++++++++++++++++++++++++++
 tests/performance/Makefile    |  21 +++++++
 tests/performance/group       |   7 +++
 4 files changed, 275 insertions(+)
 create mode 100755 tests/performance/001-readdir
 create mode 100755 tests/performance/002-sqlite
 create mode 100644 tests/performance/Makefile
 create mode 100644 tests/performance/group

diff --git a/tests/performance/001-readdir b/tests/performance/001-readdir
new file mode 100755
index 0000000..f415213
--- /dev/null
+++ b/tests/performance/001-readdir
@@ -0,0 +1,123 @@
+#! /bin/bash
+# FS QA Test No. 001
+#
+# Performance test.
+# Readdir speed for various amounts of files in a directory.
+# The test is run on TEST_DIR as well as in overlayfs above TEST_DIR.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2015 RedHat.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+# List of environments this test can use for preparing files for the testing.
+# They are independent of each other (multiple entries = multiple runs).
+# If "none" is given, the test can be run without any environment,
+# ommiting it signifies that the test won't run without any of them.
+#
+# Example: supported_environments="none empty_files full_partition"
+#
+# If you want to combine multiple environments at once, you have to
+# call them yourself: _environment_require PATH ENVIRONMENT_NAME
+
+supported_environments="none overlayfs"
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os IRIX Linux
+_require_test
+_disable_output_check
+
+run_test(){
+	# PUT TEST BELOW
+
+	if ! type xfs_io >/dev/null 2>&1; then 
+		_fail "This test requires xfs_io in PATH."
+	fi
+
+	export ENV_MERGED="$TEST_DIR/merged"
+	mkdir -p "$ENV_MERGED"
+	_environment_require $TEST_DIR
+
+	echo ""
+	echo "## environment $ENV_NAME"
+
+	numbers="1000 5000 10000"
+	for n in $numbers;do
+		echo ""
+		export ENV_FILES_NUMBER=$n
+		_environment_require "$ENV_MERGED" "empty-files"
+		er=$?
+		if [ $er -ne 0 ];then
+			continue
+		fi
+
+
+		verbose=true
+
+		# check for xfs_io tool
+		which xfs_io >/dev/null 2>&1
+		if [ $? -ne 0 ];then
+			_fatal "xfs_io tool has to be installed in PATH!"
+		fi
+		echo "# MEASURED_DATA"
+		xfs_io -c "readdir" "$TEST_DIR" |paste -sd ", " -
+		_environment_clean "$ENV_MERGED" "empty-files"
+	done
+	_environment_clean $TEST_DIR
+	rmdir "$ENV_MERGED"
+
+	# PUT TEST ABOVE
+}
+
+
+# Filter the supported_environments by user input (-eo/-ex arguments)
+# and run the test for all environments in the intersect of
+# the supported and user-allowed environments.
+for environment in $(_filter_environments "$supported_environments")
+do
+	export ENV_NAME="$environment"
+	run_test
+done
+
+# optional stuff if your test has verbose output to help resolve problems
+#echo
+#echo "If failure, check $seqres.full (this) and $seqres.full.ok (reference)"
+
+# success, all done
+status=0
+exit
diff --git a/tests/performance/002-sqlite b/tests/performance/002-sqlite
new file mode 100755
index 0000000..a278124
--- /dev/null
+++ b/tests/performance/002-sqlite
@@ -0,0 +1,124 @@
+#! /bin/bash
+# FS QA Test No. 002
+#
+# Performance test for simulating SQLite.
+#
+# Requires FIO.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2015 YOUR NAME HERE.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+# List of environments this test can use for preparing files for the testing.
+# They are independent of each other (multiple entries = multiple runs).
+# If "none" is given, the test can be run without any environment,
+# ommiting it signifies that the test won't run without any of them.
+#
+# Example: supported_environments="none empty_files full_partition"
+#
+# If you want to combine multiple environments at once, you have to
+# call them yourself: _environment_require PATH ENVIRONMENT_NAME
+
+supported_environments="none"
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	rm database-sqlite.prerun.txt
+    cd /
+    rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os IRIX Linux
+_require_test
+_disable_output_check
+
+FIO_BIN=fio
+RUNTIME=20
+
+cat <<EOF > $tmp.sqlite.fio
+[global]
+rw=randrw
+rwmixread=80
+size=1G
+directory=$TEST_DIR
+fadvise_hint=0
+direct=1
+blocksize=8k
+numjobs=\$ncpus*2
+time_based
+runtime=$RUNTIME
+ioengine=libaio
+norandommap=1
+exec_prerun=echo 3 > /proc/sys/vm/drop_caches
+group_reporting
+
+[database-sqlite]
+nrfiles=1
+numjobs=1
+ioengine=sync
+lockfile=exclusive
+
+EOF
+
+run_test(){
+	_environment_require $TEST_DIR
+	# PUT TEST BELOW
+
+	if ! type $FIO_BIN >/dev/null 2>&1; then 
+		_fail "Can't run without $FIO_BIN installed."
+	fi
+
+	$FIO_BIN --append-terse $tmp.sqlite.fio
+
+	# PUT TEST ABOVE
+	 _environment_clean $TEST_DIR
+}
+
+
+# Filter the supported_environments by user input (-eo/-ex arguments)
+# and run the test for all environments in the intersect of
+# the supported and user-allowed environments.
+for environment in $(_filter_environments "$supported_environments")
+do
+	export ENV_NAME="$environment"
+    run_test
+done
+
+# optional stuff if your test has verbose output to help resolve problems
+#echo
+#echo "If failure, check $seqres.full (this) and $seqres.full.ok (reference)"
+
+# success, all done
+status=0
+exit
diff --git a/tests/performance/Makefile b/tests/performance/Makefile
new file mode 100644
index 0000000..5a50b07
--- /dev/null
+++ b/tests/performance/Makefile
@@ -0,0 +1,21 @@
+#
+# Copyright (c) 2003-2005 Silicon Graphics, Inc.  All Rights Reserved.
+#
+
+TOPDIR = ../..
+include $(TOPDIR)/include/builddefs
+
+PERFORMANCE_DIR = performance
+TARGET_DIR = $(PKG_LIB_DIR)/$(TESTS_DIR)/$(PERFORMANCE_DIR)
+
+include $(BUILDRULES)
+
+install:
+	$(INSTALL) -m 755 -d $(TARGET_DIR)
+	$(INSTALL) -m 755 [0-9]?? $(TARGET_DIR)
+	$(INSTALL) -m 644 group $(TARGET_DIR)
+	$(INSTALL) -m 644 environment $(TARGET_DIR)
+	$(INSTALL) -m 644 [0-9]??.* $(TARGET_DIR)
+
+# Nothing.
+install-dev install-lib:
diff --git a/tests/performance/group b/tests/performance/group
new file mode 100644
index 0000000..25d4059
--- /dev/null
+++ b/tests/performance/group
@@ -0,0 +1,7 @@
+# QA groups control file
+# Defines test groups and nominal group owners
+# - do not start group names with a digit
+# - comment line before each group is "new" description
+#
+001-readdir readdir
+002-sqlite fio db-simulation
-- 
2.1.0

[PATCH 5/7] fstests: environments - create the environment directory

[Jan Ťulák]

Add the environment directory and two environments used by
performance/001-readdir test.

 - empty-files checks ENV_FILES_NUMBER environment variable to know how many files it should create and creates them
 - overlayfs creates and mounts an overlayfs into a directory.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 common/environments      |  64 ++++++++++++++++++++++
 environments/empty-files |  77 +++++++++++++++++++++++++++
 environments/overlayfs   | 136 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 277 insertions(+)
 create mode 100644 common/environments
 create mode 100644 environments/empty-files
 create mode 100644 environments/overlayfs

diff --git a/common/environments b/common/environments
new file mode 100644
index 0000000..a1aa3b5
--- /dev/null
+++ b/common/environments
@@ -0,0 +1,64 @@
+##/bin/bash
+#-----------------------------------------------------------------------
+#  Copyright (c) 2015 Red Hat, Inc.  All Rights Reserved.
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation; either version 2 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software
+#  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#  USA
+#
+#-----------------------------------------------------------------------
+
+# see _usage()
+TARGET="$2"
+
+_usage()
+{
+	echo -e "Usage:\n"
+	"$0 OPTION TARGET_DIR\n"
+	"Where OPTION is one (and only one) of the following:\n"
+	"	prepare, clean"
+}
+
+_init_env()
+{
+	action="$1"
+
+	# arguments...
+	if [ $# -ne 2 ];then
+		_usage
+		return 1
+	fi
+
+	while [ $# -gt 0 ]; do
+		case "$action" in
+			prepare)
+				#echo "# ENVIROMENT $THIS_ENV PREPARE START"
+				touch "$TARGET/.env-prepared-$THIS_ENV"
+				prepare "$2"
+				#echo "# ENVIROMENT $THIS_ENV PREPARE END"
+				shift ;;
+			clean)
+				# clean only if the environment is there
+				if [ -f "$TARGET/.env-prepared-$THIS_ENV" ]; then
+					clean "$2";
+					rm "$TARGET/.env-prepared-$THIS_ENV"
+				fi
+				shift ;;
+			*) usage ; return 1 ;;
+		esac
+		shift
+	done
+
+	return 0
+}
+
diff --git a/environments/empty-files b/environments/empty-files
new file mode 100644
index 0000000..0afa875
--- /dev/null
+++ b/environments/empty-files
@@ -0,0 +1,77 @@
+#!/bin/bash
+# FS QA Environment setup
+#
+# Create directories with lots of empty files. Name the dirs with
+# amount of the files inside. No whitheouts.
+#
+#-----------------------------------------------------------------------
+# Copyright 2015 (C) Red Hat, Inc., Jan Tulak <jtulak@redhat.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# arguments: prepare, clean
+# env variables: ENV_FILES_NUMBER
+THIS_ENV="$(basename "$BASH_SOURCE")"
+. $here/common/environments
+
+# --------------------------------------
+# Put here any custom functions or whatever you need
+
+# default value
+if [ "" = "$ENV_FILES_NUMBER" ];then
+	ENV_FILES_NUMBER=1000
+fi
+
+NR_FILES=$ENV_FILES_NUMBER
+
+
+
+
+
+make_dir(){
+	local path=$1
+	local num=$2
+	local i=0
+	# make files
+	time while [ "$i" -lt "$num" ]; do
+		touch "$path/$THIS_ENV-file-$i"
+		let i+=1
+	done
+}
+
+# --------------------------------------
+# The prepare() and clean() functions are mandatory
+# and are called automaticaly.
+# Use them as your entry points.
+
+
+# This function will prepare the environment.
+prepare()
+{
+	echo "Creating $NR_FILES files in $TARGET."
+	make_dir "$TARGET" $NR_FILES
+}
+
+# This function should clean files created by prepare()
+clean()
+{
+	rm "$TARGET/$THIS_ENV-"*
+}
+
+
+# This is needed for really running this script.
+_init_env "$@"
+exit $?
diff --git a/environments/overlayfs b/environments/overlayfs
new file mode 100644
index 0000000..4e823ff
--- /dev/null
+++ b/environments/overlayfs
@@ -0,0 +1,136 @@
+#!/bin/bash
+# FS QA Environment setup
+#
+# Prepare overlayfs in given directory and mount it into
+# $ENV_MERGED
+#
+# You can specify directories for use by exporting
+# $ENV_LOWERDIR, $ENV_UPPERDIR, $ENV_WORKDIR
+#
+# Note, if you pass your own pathes for overlayfs to use
+# as upper/lower/workdir, then they are not cleaned!
+#
+#-----------------------------------------------------------------------
+# Copyright 2015 (C) Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# arguments: prepare, clean
+# env variables: ENV_FILES_NUMBER
+THIS_ENV="$(basename "$BASH_SOURCE")"
+. $here/common/environments
+
+
+# --------------------------------------
+# Put here any custom functions or whatever you need
+
+if [ "" = "$ENV_MERGED" ];then
+	2>&1 echo "Can't run $THIS_ENV environment without ENV_MERGED variable set."
+	exit 1
+elif [ ! -d "$ENV_MERGED" ]; then
+	2>&1 echo "Provided ENV_MERGED does not exists!"
+	exit 1
+fi
+
+# default values
+if [ "" = "$ENV_LOWERDIR" ]; then
+	ENV_LOWERDIR="$TARGET/"
+	env_lower_default="yes"
+elif [ ! -d "$ENV_LOWERDIR" ]; then
+	2>&1 echo "Provided ENV_LOWERDIR does not exists: " $ENV_LOWERDIR
+	exit 1
+fi
+if [ "" = "$ENV_UPPERDIR" ]; then
+	ENV_UPPERDIR="$TARGET/upperdir"
+	env_upper_default="yes"
+elif [ ! -d "$ENV_UPPERDIR" ]; then
+	2>&1 echo "Provided ENV_UPPERDIR does not exists: " $ENV_UPPERDIR
+	exit 1
+fi
+if [ "" = "$ENV_WORKDIR" ];then
+	ENV_WORKDIR="$TARGET/workdir"
+	env_work_default="yes"
+elif [ ! -d "$ENV_WORKDIR" ]; then
+	2>&1 echo "Provided ENV_WORKDIR does not exists: " $ENV_WORKDIR
+	exit 1
+fi
+
+create_default_dirs()
+{
+	# lowerdir is by default set to TARGET, so don't create it
+
+	if [ "$env_upper_default" = "yes" ]; then
+		mkdir -p "$ENV_UPPERDIR"
+	fi
+	if [ "$env_work_default" = "yes" ]; then
+		mkdir -p "$ENV_WORKDIR"
+	fi
+}
+
+clean_default_dirs()
+{
+	if [ "$env_upper_default" = "yes" ]; then
+		rm -rf "$ENV_UPPERDIR"
+	fi
+	if [ "$env_work_default" = "yes" ]; then
+		rm -rf "$ENV_WORKDIR"
+	fi
+}
+
+
+mount_overlayfs()
+{
+	# We make lowerdir the root dir, because at least for the test dev,
+	# it stands a chance of having content, making overlayfs a bit more
+	# interesting
+	mount -t overlay none \
+		-o lowerdir="$ENV_LOWERDIR" \
+		-o upperdir="$ENV_UPPERDIR" \
+		-o workdir="$ENV_WORKDIR" \
+		"$ENV_MERGED"
+}
+
+umount_overlayfs()
+{
+	umount "$ENV_MERGED"
+}
+
+
+
+# --------------------------------------
+# The prepare() and clean() functions are mandatory
+# and are called automaticaly.
+# Use them as your entry points.
+
+
+# This function will prepare the environment.
+prepare()
+{
+	create_default_dirs
+	mount_overlayfs
+}
+
+# This function should clean files created by prepare()
+clean()
+{
+	umount_overlayfs
+	clean_default_dirs
+}
+
+
+# This is needed for really running this script.
+_init_env "$@"
+exit $?
-- 
2.1.0

[PATCH 6/7] fstests: environments - add envs to the skeleton in ./new script and README

[Jan Ťulák]

New section in README file to shortly describe the environments.

Adds support for environments into the skeleton generated by the ./new script,
so new tests can easily opt-in into using the environments.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 README | 12 +++++++++++-
 new    | 37 +++++++++++++++++++++++++++++++++++--
 2 files changed, 46 insertions(+), 3 deletions(-)

diff --git a/README b/README
index f8a878c..6d4f8d7 100644
--- a/README
+++ b/README
@@ -194,7 +194,17 @@ Test script environment:
 				  and the content of the NIS database
 				  if it is likely to be present)
 
-     5. General recommendations, usage conventions, etc.:
+     5. For setting up files and state of the filesystem before a test,
+	you can use environments. These scripts allows both sharing of the
+	set-up between tests, and running a single test in different
+	conditions (like empty and full filesystem).
+	In new tests, you can simply add them to a list at the beginning
+	of the test script (see "environments" directory for all available
+	environments). The skeleton created by ./new script already contains
+	everything necessary.
+	
+
+     6. General recommendations, usage conventions, etc.:
 	- When the content of the password or group file is
 	  required, get it using the _cat_passwd and _cat_group
 	  functions, to ensure NIS information is included if NIS
diff --git a/new b/new
index c734bdc..a6eb113 100755
--- a/new
+++ b/new
@@ -181,6 +181,18 @@ cat <<End-of-File >$tdir/$id
 #-----------------------------------------------------------------------
 #
 
+# List of environments this test can use for preparing files for the testing.
+# They are independent of each other (multiple entries = multiple runs).
+# If "none" is given, the test can be run without any environment,
+# ommiting it signifies that the test won't run without any of them.
+#
+# Example: supported_environments="none empty_files full_partition"
+#
+# If you want to combine multiple environments at once, you have to
+# call them yourself: _environment_require PATH ENVIRONMENT_NAME
+
+supported_environments="none"
+
 seq=\`basename \$0\`
 seqres=\$RESULT_DIR/\$seq
 echo "QA output created by \$seq"
@@ -207,8 +219,29 @@ _supported_fs generic
 _supported_os IRIX Linux
 _require_test
 
-# if error
-exit
+run_test(){
+    # If you don't set any environment at the beginning,
+    # this will do nothing.
+    _environment_require \$TEST_DIR
+
+    # PUT TEST BELOW
+
+    # if error
+    exit
+
+    # PUT TEST ABOVE
+    _environment_clean \$TEST_DIR
+}
+
+
+# Filter the supported_environments by user input (-eo/-ex arguments)
+# and run the test for all environments in the intersect of
+# the supported and user-allowed environments.
+for environment in \$(_filter_environments "\$supported_environments")
+do
+    export ENV_NAME="\$environment"
+    run_test
+done
 
 # optional stuff if your test has verbose output to help resolve problems
 #echo
-- 
2.1.0

[PATCH 7/7] fstests: environments - enable performance tests in ./check

[Jan Ťulák]

Because performance tests can take longer and are usually not needed,
they won't run together with regression tests, but has to be enabled
using -performance flag. With it, only perf tests will run.

Signed-off-by: Jan Ťulák <jtulak@redhat.com>
---
 check | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/check b/check
index c746d71..a80f69b 100755
--- a/check
+++ b/check
@@ -36,6 +36,7 @@ have_test_arg=false
 randomize=false
 export here=`pwd`
 xfile=""
+performance=false
 
 # start the initialisation work now
 iam=check
@@ -70,6 +71,7 @@ check options
     -nfs                test NFS
     -cifs               test CIFS
     -tmpfs              test TMPFS
+    -performance        performance tests
     -l			line mode diff
     -udiff		show unified diff (default)
     -n			show me, do not run tests
@@ -172,8 +174,11 @@ export_filtered_environments()
 get_group_list()
 {
 	grp=$1
-
-	for d in $SRC_GROUPS $FSTYP; do
+	tests="$SRC_GROUPS $FSTYP"
+	if $performance; then
+		tests="$SRC_GROUPS"
+	fi
+	for d in $tests; do
 		l=$(sed -n < $SRC_DIR/$d/group \
 			-e 's/#.*//' \
 			-e 's/$/ /' \
@@ -188,7 +193,11 @@ get_group_list()
 get_all_tests()
 {
 	touch $tmp.list
-	for d in $SRC_GROUPS $FSTYP; do
+	tests="$SRC_GROUPS $FSTYP"
+	if $performance; then
+		tests="$SRC_GROUPS"
+	fi
+	for d in $tests; do
 		ls $SRC_DIR/$d/* | \
 			grep -v "\..*" | \
 			grep "^$SRC_DIR/$d/$VALID_TEST_NAME"| \
@@ -291,6 +300,8 @@ while [ $# -gt 0 ]; do
 	-nfs)	FSTYP=nfs ;;
 	-cifs)	FSTYP=cifs ;;
 	-tmpfs)	FSTYP=tmpfs ;;
+	-performance)	SRC_GROUPS="performance";
+					performance=true;;
 
 	-g)	group=$2 ; shift ;
 		GROUP_LIST="$GROUP_LIST $group"
-- 
2.1.0