[PATCH 00/10] xfstests: Stress tests improments v5

[PATCH 00/10] xfstests: Stress tests improments v5

[Dmitry Monakhov]

1   add fio requirement V2
2   add configurable load factors
3   hardcode fops for determinable fsstests runs
4   fsstress add replace file operation
5   allow fsstress to use load factor where appropriate
6   move run_check to common.rc
7   add  fallocate/truncate vs AIO/DIO stress test
8   add  fallocate/punch_hole vs AIO/DIO stress test
9   add defragmentation stress tests for ext4
10  add disk failure simulation test

Known issues which can be triggered this test case:
287'th over EXT4 result in https://gist.github.com/dmonakhov/4994589
298'th (8'th patch) over XFS result in https://gist.github.com/dmonakhov/4994465
300'th and 301'th over EXT4 result in https://gist.github.com/dmonakhov/4770294
303'th over XFS result in https://gist.github.com/dmonakhov/4951456

Changes since V4
Rebase on top of current dev tree.


Changes since V2
Mostly code style cleanups accodring to Dave's comments
- add _scale_fsstress_args function
- move run_check function to common.rc
- Add more comments
- split combined e4defrag testcase in to disicated tescases

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 01/10] xfstests: add fio requirement V2

[Dmitry Monakhov]

FIO is very flexible io generator, I would call it IO swiss knife.
Currently we have tonns of hardcoded application which reproduces
some predefined scenario. This approach has obvious dissadvantages
1) Lack of flexability: one written it is hard to modify it in future
2) Code base is large, many routines written again and again

At the same time add new fio based tast is just add simle INI file.
This greatly simplify code review. I do beleve that some day we will
replace most of hardcoded io binaries with fio.

One who is planning to run $FIO_PROG should first check that system
contains appropriate version which is able to handle jobfile
for example:  _require_fio 286-job.fio

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 common.config |    1 +
 common.rc     |   15 +++++++++++++++
 2 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/common.config b/common.config
index 827c718..7681812 100644
--- a/common.config
+++ b/common.config
@@ -159,6 +159,7 @@ export KILLALL_PROG="`set_prog_path killall`"
 export INDENT_PROG="`set_prog_path indent`"
 export XFS_COPY_PROG="`set_prog_path xfs_copy`"
 export FSTRIM_PROG="`set_prog_path fstrim`"
+export FIO_PROG="`set_prog_path fio`"
 
 # Generate a comparable xfsprogs version number in the form of
 # major * 10000 + minor * 100 + release
diff --git a/common.rc b/common.rc
index 2e8581e..14de47b 100644
--- a/common.rc
+++ b/common.rc
@@ -1789,6 +1789,21 @@ _require_freeze()
 	[ $result -eq 0 ] || _notrun "$FSTYP does not support freezing"
 }
 
+# Check that fio is present, and it is able to execute given jobfile
+_require_fio()
+{
+	job=$1
+
+	_require_command $FIO_PROG
+	if [ -z "$1" ]; then
+		return 1;
+	fi
+	
+	$FIO_PROG --warnings-fatal --showcmd $job >/dev/null 2>&1
+	[ $? -eq 0 ] || _notrun "$FIO_PROG too old"
+}
+
+
 # arg 1 is dev to remove and is output of the below eg.
 # ls -l /sys/class/block/sdd | rev | cut -d "/" -f 3 | rev
 _devmgt_remove()
-- 
1.7.1

[PATCH 01/10] xfstests: add fio requirement V2

[Dmitry Monakhov]

FIO is very flexible io generator, I would call it IO swiss knife.
Currently we have tonns of hardcoded application which reproduces
some predefined scenario. This approach has obvious dissadvantages
1) Lack of flexability: one written it is hard to modify it in future
2) Code base is large, many routines written again and again

At the same time add new fio based tast is just add simle INI file.
This greatly simplify code review. I do beleve that some day we will
replace most of hardcoded io binaries with fio.

One who is planning to run $FIO_PROG should first check that system
contains appropriate version which is able to handle jobfile
for example:  _require_fio 286-job.fio

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 common.config |    1 +
 common.rc     |   15 +++++++++++++++
 2 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/common.config b/common.config
index 827c718..7681812 100644
--- a/common.config
+++ b/common.config
@@ -159,6 +159,7 @@ export KILLALL_PROG="`set_prog_path killall`"
 export INDENT_PROG="`set_prog_path indent`"
 export XFS_COPY_PROG="`set_prog_path xfs_copy`"
 export FSTRIM_PROG="`set_prog_path fstrim`"
+export FIO_PROG="`set_prog_path fio`"
 
 # Generate a comparable xfsprogs version number in the form of
 # major * 10000 + minor * 100 + release
diff --git a/common.rc b/common.rc
index 2e8581e..14de47b 100644
--- a/common.rc
+++ b/common.rc
@@ -1789,6 +1789,21 @@ _require_freeze()
 	[ $result -eq 0 ] || _notrun "$FSTYP does not support freezing"
 }
 
+# Check that fio is present, and it is able to execute given jobfile
+_require_fio()
+{
+	job=$1
+
+	_require_command $FIO_PROG
+	if [ -z "$1" ]; then
+		return 1;
+	fi
+	
+	$FIO_PROG --warnings-fatal --showcmd $job >/dev/null 2>&1
+	[ $? -eq 0 ] || _notrun "$FIO_PROG too old"
+}
+
+
 # arg 1 is dev to remove and is output of the below eg.
 # ls -l /sys/class/block/sdd | rev | cut -d "/" -f 3 | rev
 _devmgt_remove()
-- 
1.7.1

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 01/10] xfstests: add fio requirement V2

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> FIO is very flexible io generator, I would call it IO swiss knife.
> Currently we have tonns of hardcoded application which reproduces
> some predefined scenario. This approach has obvious dissadvantages
> 1) Lack of flexability: one written it is hard to modify it in future
> 2) Code base is large, many routines written again and again
>
> At the same time add new fio based tast is just add simle INI file.
> This greatly simplify code review. I do beleve that some day we will
> replace most of hardcoded io binaries with fio.
>
> One who is planning to run $FIO_PROG should first check that system
> contains appropriate version which is able to handle jobfile
> for example:  _require_fio 286-job.fio
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Minor typos in the description, but looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

Re: [PATCH 01/10] xfstests: add fio requirement V2

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> FIO is very flexible io generator, I would call it IO swiss knife.
> Currently we have tonns of hardcoded application which reproduces
> some predefined scenario. This approach has obvious dissadvantages
> 1) Lack of flexability: one written it is hard to modify it in future
> 2) Code base is large, many routines written again and again
>
> At the same time add new fio based tast is just add simle INI file.
> This greatly simplify code review. I do beleve that some day we will
> replace most of hardcoded io binaries with fio.
>
> One who is planning to run $FIO_PROG should first check that system
> contains appropriate version which is able to handle jobfile
> for example:  _require_fio 286-job.fio
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Minor typos in the description, but looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 02/10] xfstest: add configurable load factors

Most stress test has probable behaviour, the longer test run the
larger corner cases will be cover. It is reasonable to allow
user to provide some sort of system load factor.
This patch introduce two global variables
LOAD_FACTOR: Usually means factor number of running tasks
TIME_FACTOR: Usually means factor of run time, or number of operations
If not speficied both variables defined to 1, so original behaviour
preserved.

TODO: Change all stress tests to use this variables

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 common.config |    2 ++
 group         |    7 ++++++-
 2 files changed, 8 insertions(+), 1 deletions(-)

diff --git a/common.config b/common.config
index 7681812..8efc7f8 100644
--- a/common.config
+++ b/common.config
@@ -73,6 +73,8 @@ export HOST_OPTIONS=${HOST_OPTIONS:=local.config}
 export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"}
 export BENCH_PASSES=${BENCH_PASSES:=5}
 export XFS_MKFS_OPTIONS=${XFS_MKFS_OPTIONS:=-bsize=4096}
+export TIME_FACTOR=${TIME_FACTOR:=1}
+export LOAD_FACTOR=${LOAD_FACTOR:=1}
 
 export PWD=`pwd`
 #export DEBUG=${DEBUG:=...} # arbitrary CFLAGS really.
diff --git a/group b/group
index eb4f375..5504557 100644
--- a/group
+++ b/group
@@ -113,7 +113,12 @@ dangerous
 # on current systems
 deprecated
 
-#
+# Stress test with probable behaviour, the longer test run the
+# larger corner cases will be covered. Configurable variables:
+# LOAD_FACTOR: Usually means factor number of running tasks
+# TIME_FACTOR: Usually means factor of run time, or number of operations
+stress
+
 # test-group association ... one line per test
 #
 001 rw dir udf auto quick
-- 
1.7.1

[PATCH 02/10] xfstest: add configurable load factors

[Dmitry Monakhov]

Most stress test has probable behaviour, the longer test run the
larger corner cases will be cover. It is reasonable to allow
user to provide some sort of system load factor.
This patch introduce two global variables
LOAD_FACTOR: Usually means factor number of running tasks
TIME_FACTOR: Usually means factor of run time, or number of operations
If not speficied both variables defined to 1, so original behaviour
preserved.

TODO: Change all stress tests to use this variables

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 common.config |    2 ++
 group         |    7 ++++++-
 2 files changed, 8 insertions(+), 1 deletions(-)

diff --git a/common.config b/common.config
index 7681812..8efc7f8 100644
--- a/common.config
+++ b/common.config
@@ -73,6 +73,8 @@ export HOST_OPTIONS=${HOST_OPTIONS:=local.config}
 export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"}
 export BENCH_PASSES=${BENCH_PASSES:=5}
 export XFS_MKFS_OPTIONS=${XFS_MKFS_OPTIONS:=-bsize=4096}
+export TIME_FACTOR=${TIME_FACTOR:=1}
+export LOAD_FACTOR=${LOAD_FACTOR:=1}
 
 export PWD=`pwd`
 #export DEBUG=${DEBUG:=...} # arbitrary CFLAGS really.
diff --git a/group b/group
index eb4f375..5504557 100644
--- a/group
+++ b/group
@@ -113,7 +113,12 @@ dangerous
 # on current systems
 deprecated
 
-#
+# Stress test with probable behaviour, the longer test run the
+# larger corner cases will be covered. Configurable variables:
+# LOAD_FACTOR: Usually means factor number of running tasks
+# TIME_FACTOR: Usually means factor of run time, or number of operations
+stress
+
 # test-group association ... one line per test
 #
 001 rw dir udf auto quick
-- 
1.7.1

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 02/10] xfstest: add configurable load factors

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Most stress test has probable behauviour, the longer test run the
> larger corner cases will be cover. It is reasonable to allow
> user to provide some sort of system load factor.
> This patch introduce two global variables
> LOAD_FACTOR: Usually means factor number of running tasks
> TIME_FACTOR: Usually means factor of run time, or number of operations
> If not speficied both variables defined to 1, so original behaviour
> preserved.
>
> TODO: Change all stress tests to use this variables
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Nice addition, looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 03/10] xfstests: hardcode fops for determinable fsstests runs

[Dmitry Monakhov]

106,107 and 117 are frozen tests which use known seed, it is
reasonable to explicitly hardcode file operations in order to avoid
implicit changes caused by future changes in fsstress.

NOTE: options genereted like follows: fsstress -S c $ORIG_ARGS

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 106 |   33 +++++++++++++++++++++++++++++++--
 107 |   32 +++++++++++++++++++++++++++++---
 2 files changed, 60 insertions(+), 5 deletions(-)

diff --git a/106 b/106
index 8278691..b351fe5 100755
--- a/106
+++ b/106
@@ -61,8 +61,37 @@ _require_prjquota $SCRATCH_DEV
 
 # initial populate
 target=$SCRATCH_MNT/target
-$FSSTRESS_PROG -s 0xdeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
-$FSSTRESS_PROG -s 0xdeed -m8 -z -p4 -n1000 -fsetxattr=500 -fchown=500 -d $target
+$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
+-f allocsp=1 \
+-f chown=3 \
+-f creat=4 \
+-f dwrite=4 \
+-f fallocate=1 \
+-f fdatasync=1 \
+-f fiemap=1 \
+-f freesp=1 \
+-f fsync=1 \
+-f link=1 \
+-f mkdir=2 \
+-f mknod=2 \
+-f punch=1 \
+-f rename=2 \
+-f resvsp=1 \
+-f rmdir=1 \
+-f setxattr=1 \
+-f symlink=2 \
+-f sync=1 \
+-f truncate=2 \
+-f unlink=1 \
+-f unresvsp=1 \
+-f write=4 \
+-d $target
+
+$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
+-f chown=500 \
+-f setxattr=500 \
+-d $target
+
 
 # also use space, to be able to go over/under limits easily
 uid=255
diff --git a/107 b/107
index 74403e6..eaf7e6a 100755
--- a/107
+++ b/107
@@ -78,9 +78,35 @@ echo "6:$target" | tee -a $seq.full > $tmp.projects
 
 echo "### populate filesystem"
 mkdir $target		|| exit
-FSSTRESS_AVOID="$FSSTRESS_AVOID -fmknod=0 -fsymlink=0"
-$FSSTRESS_PROG -s 0xfeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
-$FSSTRESS_PROG -s 0xbabe -m8 -z -p4 -n500 -fsetxattr=250 -fchown=250 -d $target
+$FSSTRESS_PROG -z -s 65261 -m 8 -n 1000 -p 4 \
+-f allocsp=1 \
+-f chown=3 \
+-f creat=4 \
+-f dwrite=4 \
+-f fallocate=1 \
+-f fdatasync=1 \
+-f fiemap=1 \
+-f freesp=1 \
+-f fsync=1 \
+-f link=1 \
+-f mkdir=2 \
+-f punch=1 \
+-f rename=2 \
+-f resvsp=1 \
+-f rmdir=1 \
+-f setxattr=1 \
+-f sync=1 \
+-f truncate=2 \
+-f unlink=1 \
+-f unresvsp=1 \
+-f write=4 \
+-d $target
+
+$FSSTRESS_PROG -z -s 47806 -m 8 -n 500 -p 4 \
+-f chown=250 \
+-f setxattr=250 \
+-d $target
+
 
 QARGS="-x -D $tmp.projects -P /dev/null $SCRATCH_MNT"
 
-- 
1.7.1

[PATCH 03/10] xfstests: hardcode fops for determinable fsstests runs

[Dmitry Monakhov]

106,107 and 117 are frozen tests which use known seed, it is
reasonable to explicitly hardcode file operations in order to avoid
implicit changes caused by future changes in fsstress.

NOTE: options genereted like follows: fsstress -S c $ORIG_ARGS

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 106 |   33 +++++++++++++++++++++++++++++++--
 107 |   32 +++++++++++++++++++++++++++++---
 2 files changed, 60 insertions(+), 5 deletions(-)

diff --git a/106 b/106
index 8278691..b351fe5 100755
--- a/106
+++ b/106
@@ -61,8 +61,37 @@ _require_prjquota $SCRATCH_DEV
 
 # initial populate
 target=$SCRATCH_MNT/target
-$FSSTRESS_PROG -s 0xdeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
-$FSSTRESS_PROG -s 0xdeed -m8 -z -p4 -n1000 -fsetxattr=500 -fchown=500 -d $target
+$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
+-f allocsp=1 \
+-f chown=3 \
+-f creat=4 \
+-f dwrite=4 \
+-f fallocate=1 \
+-f fdatasync=1 \
+-f fiemap=1 \
+-f freesp=1 \
+-f fsync=1 \
+-f link=1 \
+-f mkdir=2 \
+-f mknod=2 \
+-f punch=1 \
+-f rename=2 \
+-f resvsp=1 \
+-f rmdir=1 \
+-f setxattr=1 \
+-f symlink=2 \
+-f sync=1 \
+-f truncate=2 \
+-f unlink=1 \
+-f unresvsp=1 \
+-f write=4 \
+-d $target
+
+$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
+-f chown=500 \
+-f setxattr=500 \
+-d $target
+
 
 # also use space, to be able to go over/under limits easily
 uid=255
diff --git a/107 b/107
index 74403e6..eaf7e6a 100755
--- a/107
+++ b/107
@@ -78,9 +78,35 @@ echo "6:$target" | tee -a $seq.full > $tmp.projects
 
 echo "### populate filesystem"
 mkdir $target		|| exit
-FSSTRESS_AVOID="$FSSTRESS_AVOID -fmknod=0 -fsymlink=0"
-$FSSTRESS_PROG -s 0xfeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
-$FSSTRESS_PROG -s 0xbabe -m8 -z -p4 -n500 -fsetxattr=250 -fchown=250 -d $target
+$FSSTRESS_PROG -z -s 65261 -m 8 -n 1000 -p 4 \
+-f allocsp=1 \
+-f chown=3 \
+-f creat=4 \
+-f dwrite=4 \
+-f fallocate=1 \
+-f fdatasync=1 \
+-f fiemap=1 \
+-f freesp=1 \
+-f fsync=1 \
+-f link=1 \
+-f mkdir=2 \
+-f punch=1 \
+-f rename=2 \
+-f resvsp=1 \
+-f rmdir=1 \
+-f setxattr=1 \
+-f sync=1 \
+-f truncate=2 \
+-f unlink=1 \
+-f unresvsp=1 \
+-f write=4 \
+-d $target
+
+$FSSTRESS_PROG -z -s 47806 -m 8 -n 500 -p 4 \
+-f chown=250 \
+-f setxattr=250 \
+-d $target
+
 
 QARGS="-x -D $tmp.projects -P /dev/null $SCRATCH_MNT"
 
-- 
1.7.1

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 03/10] xfstests: hardcode fops for determinable fsstests runs

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> 106,107 and 117 are frozen tests which use known seed, it is
> reasonable to explicitly hardcode file operations in order to avoid
> implicit changes caused by future changes in fsstress.
>
> NOTE: options genereted like follows: fsstress -S c $ORIG_ARGS
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

106 and 107 are not valid for ext4. They failed the same before and 
after this patch with xfs. Because 106 and 107 are not in the auto group 
and 117 is in the auto group and passes for both xfs and ext4,
I will say this looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>


> ---
>   106 |   33 +++++++++++++++++++++++++++++++--
>   107 |   32 +++++++++++++++++++++++++++++---
>   2 files changed, 60 insertions(+), 5 deletions(-)
>
> diff --git a/106 b/106
> index 8278691..b351fe5 100755
> --- a/106
> +++ b/106
> @@ -61,8 +61,37 @@ _require_prjquota $SCRATCH_DEV
>
>   # initial populate
>   target=$SCRATCH_MNT/target
> -$FSSTRESS_PROG -s 0xdeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
> -$FSSTRESS_PROG -s 0xdeed -m8 -z -p4 -n1000 -fsetxattr=500 -fchown=500 -d $target
> +$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
> +-f allocsp=1 \
> +-f chown=3 \
> +-f creat=4 \
> +-f dwrite=4 \
> +-f fallocate=1 \
> +-f fdatasync=1 \
> +-f fiemap=1 \
> +-f freesp=1 \
> +-f fsync=1 \
> +-f link=1 \
> +-f mkdir=2 \
> +-f mknod=2 \
> +-f punch=1 \
> +-f rename=2 \
> +-f resvsp=1 \
> +-f rmdir=1 \
> +-f setxattr=1 \
> +-f symlink=2 \
> +-f sync=1 \
> +-f truncate=2 \
> +-f unlink=1 \
> +-f unresvsp=1 \
> +-f write=4 \
> +-d $target
> +
> +$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
> +-f chown=500 \
> +-f setxattr=500 \
> +-d $target
> +
>
>   # also use space, to be able to go over/under limits easily
>   uid=255
> diff --git a/107 b/107
> index 74403e6..eaf7e6a 100755
> --- a/107
> +++ b/107
> @@ -78,9 +78,35 @@ echo "6:$target" | tee -a $seq.full > $tmp.projects
>
>   echo "### populate filesystem"
>   mkdir $target		|| exit
> -FSSTRESS_AVOID="$FSSTRESS_AVOID -fmknod=0 -fsymlink=0"
> -$FSSTRESS_PROG -s 0xfeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
> -$FSSTRESS_PROG -s 0xbabe -m8 -z -p4 -n500 -fsetxattr=250 -fchown=250 -d $target
> +$FSSTRESS_PROG -z -s 65261 -m 8 -n 1000 -p 4 \
> +-f allocsp=1 \
> +-f chown=3 \
> +-f creat=4 \
> +-f dwrite=4 \
> +-f fallocate=1 \
> +-f fdatasync=1 \
> +-f fiemap=1 \
> +-f freesp=1 \
> +-f fsync=1 \
> +-f link=1 \
> +-f mkdir=2 \
> +-f punch=1 \
> +-f rename=2 \
> +-f resvsp=1 \
> +-f rmdir=1 \
> +-f setxattr=1 \
> +-f sync=1 \
> +-f truncate=2 \
> +-f unlink=1 \
> +-f unresvsp=1 \
> +-f write=4 \
> +-d $target
> +
> +$FSSTRESS_PROG -z -s 47806 -m 8 -n 500 -p 4 \
> +-f chown=250 \
> +-f setxattr=250 \
> +-d $target
> +
>
>   QARGS="-x -D $tmp.projects -P /dev/null $SCRATCH_MNT"
>
>

Re: [PATCH 03/10] xfstests: hardcode fops for determinable fsstests runs

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> 106,107 and 117 are frozen tests which use known seed, it is
> reasonable to explicitly hardcode file operations in order to avoid
> implicit changes caused by future changes in fsstress.
>
> NOTE: options genereted like follows: fsstress -S c $ORIG_ARGS
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

106 and 107 are not valid for ext4. They failed the same before and 
after this patch with xfs. Because 106 and 107 are not in the auto group 
and 117 is in the auto group and passes for both xfs and ext4,
I will say this looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>


> ---
>   106 |   33 +++++++++++++++++++++++++++++++--
>   107 |   32 +++++++++++++++++++++++++++++---
>   2 files changed, 60 insertions(+), 5 deletions(-)
>
> diff --git a/106 b/106
> index 8278691..b351fe5 100755
> --- a/106
> +++ b/106
> @@ -61,8 +61,37 @@ _require_prjquota $SCRATCH_DEV
>
>   # initial populate
>   target=$SCRATCH_MNT/target
> -$FSSTRESS_PROG -s 0xdeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
> -$FSSTRESS_PROG -s 0xdeed -m8 -z -p4 -n1000 -fsetxattr=500 -fchown=500 -d $target
> +$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
> +-f allocsp=1 \
> +-f chown=3 \
> +-f creat=4 \
> +-f dwrite=4 \
> +-f fallocate=1 \
> +-f fdatasync=1 \
> +-f fiemap=1 \
> +-f freesp=1 \
> +-f fsync=1 \
> +-f link=1 \
> +-f mkdir=2 \
> +-f mknod=2 \
> +-f punch=1 \
> +-f rename=2 \
> +-f resvsp=1 \
> +-f rmdir=1 \
> +-f setxattr=1 \
> +-f symlink=2 \
> +-f sync=1 \
> +-f truncate=2 \
> +-f unlink=1 \
> +-f unresvsp=1 \
> +-f write=4 \
> +-d $target
> +
> +$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
> +-f chown=500 \
> +-f setxattr=500 \
> +-d $target
> +
>
>   # also use space, to be able to go over/under limits easily
>   uid=255
> diff --git a/107 b/107
> index 74403e6..eaf7e6a 100755
> --- a/107
> +++ b/107
> @@ -78,9 +78,35 @@ echo "6:$target" | tee -a $seq.full > $tmp.projects
>
>   echo "### populate filesystem"
>   mkdir $target		|| exit
> -FSSTRESS_AVOID="$FSSTRESS_AVOID -fmknod=0 -fsymlink=0"
> -$FSSTRESS_PROG -s 0xfeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
> -$FSSTRESS_PROG -s 0xbabe -m8 -z -p4 -n500 -fsetxattr=250 -fchown=250 -d $target
> +$FSSTRESS_PROG -z -s 65261 -m 8 -n 1000 -p 4 \
> +-f allocsp=1 \
> +-f chown=3 \
> +-f creat=4 \
> +-f dwrite=4 \
> +-f fallocate=1 \
> +-f fdatasync=1 \
> +-f fiemap=1 \
> +-f freesp=1 \
> +-f fsync=1 \
> +-f link=1 \
> +-f mkdir=2 \
> +-f punch=1 \
> +-f rename=2 \
> +-f resvsp=1 \
> +-f rmdir=1 \
> +-f setxattr=1 \
> +-f sync=1 \
> +-f truncate=2 \
> +-f unlink=1 \
> +-f unresvsp=1 \
> +-f write=4 \
> +-d $target
> +
> +$FSSTRESS_PROG -z -s 47806 -m 8 -n 500 -p 4 \
> +-f chown=250 \
> +-f setxattr=250 \
> +-d $target
> +
>
>   QARGS="-x -D $tmp.projects -P /dev/null $SCRATCH_MNT"
>
>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 04/10] xfstets: fsstress add replace file operation

[Dmitry Monakhov]

The most common usecase for rename(2) syscall is an atomic replacement
of existing file with newer version. But rename_f() rename some existing
filename to newly generated (non existing) filename. As result the most
important usecase is not covered.
Since rename_f() is already exist in fsstress and it has known behavior,
some tests already depends on that behaviour, let's add another operation
(replace_f) which invoke rename(2) for two existing entries.

OUT_OF_COMMIT_DISCUSSION:
Off course replace_f() break naming convention where fun_name == syscall_f(),
but this is the only way I see to introduce new feature and not break
other tests. May be it is reasonable to call it rename2_f() ?

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 ltp/fsstress.c |   47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 47 insertions(+), 0 deletions(-)

diff --git a/ltp/fsstress.c b/ltp/fsstress.c
index b4cfb25..85ff72a 100644
--- a/ltp/fsstress.c
+++ b/ltp/fsstress.c
@@ -80,6 +80,7 @@ typedef enum {
 	OP_READ,
 	OP_READLINK,
 	OP_RENAME,
+	OP_REPLACE,
 	OP_RESVSP,
 	OP_RMDIR,
 	OP_SETATTR,
@@ -165,6 +166,7 @@ void	punch_f(int, long);
 void	read_f(int, long);
 void	readlink_f(int, long);
 void	rename_f(int, long);
+void	replace_f(int, long);
 void	resvsp_f(int, long);
 void	rmdir_f(int, long);
 void	setattr_f(int, long);
@@ -202,6 +204,7 @@ opdesc_t	ops[] = {
 	{ OP_READ, "read", read_f, 1, 0 },
 	{ OP_READLINK, "readlink", readlink_f, 1, 0 },
 	{ OP_RENAME, "rename", rename_f, 2, 1 },
+	{ OP_REPLACE, "replace", replace_f, 2, 1 },
 	{ OP_RESVSP, "resvsp", resvsp_f, 1, 1 },
 	{ OP_RMDIR, "rmdir", rmdir_f, 1, 1 },
 	{ OP_SETATTR, "setattr", setattr_f, 0, 1 },
@@ -2680,6 +2683,50 @@ rename_f(int opno, long r)
 }
 
 void
+replace_f(int opno, long r)
+{
+	int		e;
+	pathname_t	src_f, dst_f;
+	fent_t		*src_fep, *dst_fep;
+	flist_t		*src_flp, *dst_flp;
+	int		v;
+	int		v1;
+
+	/* get an existing path for the source of the rename */
+	init_pathname(&src_f);
+	if (!get_fname(FT_ANYm, r, &src_f, &src_flp, &src_fep, &v)) {
+		if (v)
+			printf("%d/%d: replace - no filename\n", procid, opno);
+		free_pathname(&src_f);
+		return;
+	}
+	/* get an existing path for the destination of the rename */
+	init_pathname(&dst_f);
+	if (!get_fname(1 << (src_flp - flist), rand(), &dst_f, &dst_flp, &dst_fep, &v1)) {
+		if (v1)
+			printf("%d/%d: replace - no filename\n", procid, opno);
+		free_pathname(&dst_f);
+		return;
+	}
+	
+	v |= v1;
+	e = rename_path(&src_f, &dst_f) < 0 ? errno : 0;
+	check_cwd();
+	if (e == 0 && src_fep->id != dst_fep->id ) {
+		del_from_flist(src_flp - flist, src_fep - src_flp->fents);
+	}
+	if (v) {
+		printf("%d/%d: replace %s with %s %d\n", procid, opno, dst_f.path,
+			src_f.path, e);
+		if (e == 0 && src_fep->id != dst_fep->id)
+			printf("%d/%d: replace del entry: id=%d,parent=%d\n",
+				procid, opno, src_fep->id, src_fep->parent);
+	}
+	free_pathname(&src_f);
+	free_pathname(&dst_f);
+}
+
+void
 resvsp_f(int opno, long r)
 {
 	int		e;
-- 
1.7.1

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 04/10] xfstets: fsstress add replace file operation

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> The most common usecase for rename(2) syscall is an atomic replacement
> of existing file with newer version. But rename_f() rename some existing
> filename to newly generated (non existing) filename. As result the most
> important usecase is not covered.

Good catch.

> Since rename_f() is already exist in fsstress and it has known behavior,
> some tests already depends on that behaviour, let's add another operation
> (replace_f) which invoke rename(2) for two existing entries.

>
> OUT_OF_COMMIT_DISCUSSION:
> Off course replace_f() break naming convention where fun_name == syscall_f(),
> but this is the only way I see to introduce new feature and not break
> other tests. May be it is reasonable to call it rename2_f() ?
>

I think this possible exposes a bug which was not exposed by before when 
running for example test 076 and test 083 on both ext4 and xfs.

Suggest this new function is called rename2_() so that we don't change 
the existing known tests.

Regards
--Rich

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 05/10] xfstest: allow fsstress to use load factor where appropriate

[Dmitry Monakhov]

1) Add _scale_fsstress_args function which transform argumets according
   to load factors
2) Let all non deterministic fsstress tests to use scaled arguments

I've able to trigger OOPS on xfs see:https://gist.github.com/dmonakhov/4762653

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 017       |    3 ++-
 068       |    4 ++--
 070       |    5 +++--
 076       |    5 +++--
 083       |    4 ++--
 087       |    3 ++-
 104       |    4 +++-
 114       |    4 ++--
 167       |    5 +++--
 232       |    4 ++--
 232.out   |    1 -
 233       |    6 +++---
 233.out   |    1 -
 269       |    2 +-
 270       |    2 +-
 common.rc |   14 ++++++++++++++
 group     |   26 +++++++++++++-------------
 17 files changed, 56 insertions(+), 37 deletions(-)

diff --git a/017 b/017
index 9ca0e72..7db4667 100755
--- a/017
+++ b/017
@@ -67,7 +67,8 @@ echo "*** test"
 for l in 0 1 2 3 4
 do
         echo "    *** test $l"
-        $FSSTRESS_PROG -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID >>$seq.full
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID`
+        $FSSTRESS_PROG  $FSSTRESS_ARGS >>$seq.full
 
         _scratch_mount -o remount,ro \
             || _fail "remount ro failed"
diff --git a/068 b/068
index a641e2f..cbfea95 100755
--- a/068
+++ b/068
@@ -82,8 +82,8 @@ touch $tmp.running
       do
       # We do both read & write IO - not only is this more realistic,
       # but it also potentially tests atime updates
-      $FSSTRESS_PROG -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID \
-	  > /dev/null 2>&1
+      FSSTRESS_ARGS=`_scale_fsstress_args -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID`
+      $FSSTRESS_PROG $FSSTRESS_ARGS > /dev/null 2>&1
     done
 
     rm -r $STRESS_DIR/*
diff --git a/070 b/070
index f48c33c..334cce7 100755
--- a/070
+++ b/070
@@ -52,7 +52,7 @@ _require_attrs
 
 _setup_testdir
 
-$FSSTRESS_PROG \
+FSSTRESS_ARGS=`_scale_fsstress_args \
 	-d $testdir/fsstress \
 	-f allocsp=0 \
 	-f freesp=0 \
@@ -62,7 +62,8 @@ $FSSTRESS_PROG \
 	-f unresvsp=0 \
 	-f attr_set=100 \
 	-f attr_remove=100 \
-        -p 1 -n 10000 -S c >$seq.full 2>&1
+        -p 1 -n 10000 -S c`
+$FSSTRESS_PROG $FSSTRESS_ARGS >$seq.full 2>&1
 
 status=$?
 exit
diff --git a/076 b/076
index e472b26..793b869 100755
--- a/076
+++ b/076
@@ -74,8 +74,9 @@ echo "*** test concurrent block/fs access"
 cat $SCRATCH_DEV >/dev/null &
 pid=$!
 
-$FSSTRESS_PROG -d $SCRATCH_MNT -p 2 -n 2000 $FSSTRESS_AVOID >>$seq.full
-
+FSSTRESS_ARGS=`_scale_fsstress_args -p 2 -n 2000 $FSSTRESS_AVOID`
+echo "run fsstress with args: $FSSTRESS_ARGS" >>$seq.full
+$FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full
 _lets_get_pidst
 _check_scratch_fs
 
diff --git a/083 b/083
index e0670b9..f5349a9 100755
--- a/083
+++ b/083
@@ -84,8 +84,8 @@ workout()
 		|| _fail "mount failed"
 
 	# -w ensures that the only ops are ones which cause write I/O
-	$FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID \
-		>>$seq.full
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full
 	_check_scratch_fs
 }
 
diff --git a/087 b/087
index 48e5eaa..e7be9a0 100755
--- a/087
+++ b/087
@@ -47,7 +47,8 @@ _do_meta()
     param="-p 4 -z -f rmdir=10 -f link=10 -f creat=10 -f mkdir=10 \
            -f rename=30 -f stat=30 -f unlink=30 -f truncate=20"
     _echofull "calling fsstress $param -m8 -n $count"
-    if ! $FSSTRESS_PROG $param $FSSTRESS_AVOID -m 8 -n $count -d $out >>$seq.full 2>&1
+    FSSTRESS_ARGS=`_scale_fsstress_args $param $FSSTRESS_AVOID -m 8 -n $count -d $out`
+    if ! $FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full 2>&1
     then
 	_echofull "fsstress failed"
     fi
diff --git a/104 b/104
index 14f2669..8db6d88 100755
--- a/104
+++ b/104
@@ -64,7 +64,9 @@ _stress_scratch()
 	procs=3
 	nops=1000
 	# -w ensures that the only ops are ones which cause write I/O
-	$FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID > /dev/null &
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
+	    -n $nops $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seq.full &
 }
 
 # real QA test starts here
diff --git a/114 b/114
index 7679222..edce0f4 100755
--- a/114
+++ b/114
@@ -246,11 +246,11 @@ _test_fsstress()
 
 	out=$SCRATCH_MNT/fsstress.$$
 	count=1000
-	args="-z \
+	args=`_scale_fsstress_args -z \
 -f rmdir=10 -f link=10 -f creat=10 \
 -f mkdir=10 -f rename=30 -f unlink=10 \
 -f symlink=10 \
--n $count -d $out -p 3"
+-n $count -d $out -p 3`
 
 	echo "fsstress $args" | sed -e "s#$out#outdir#"
 	if ! $FSSTRESS_PROG $args | _filter_num
diff --git a/167 b/167
index ccb6c2a..5fb95e8 100755
--- a/167
+++ b/167
@@ -44,8 +44,9 @@ workout()
 {
 	procs=100
 	nops=15000
-	$FSSTRESS_PROG -d $SCRATCH_MNT -p $procs -n $nops $FSSTRESS_AVOID \
-		>>$seq.full &
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p $procs -n $nops \
+	    $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seq.full &
 	sleep 2
 }
 
diff --git a/232 b/232
index 2795da7..d915d42 100755
--- a/232
+++ b/232
@@ -54,9 +54,9 @@ _fsstress()
 
 	out=$SCRATCH_MNT/fsstress.$$
 	count=2000
-	args="-n $count -d $out -p 7"
+	args=`_scale_fsstress_args -d $out -n $count -p 7`
 
-	echo "fsstress $args" | tee -a $here/$seq.full | sed -e "s#$out#outdir#"
+	echo "fsstress $args" >> tee -a $here/$seq.full
 	if ! $FSSTRESS_PROG $args | tee -a $here/$seq.full | _filter_num
 	then
 		echo "    fsstress $args returned $?"
diff --git a/232.out b/232.out
index ef82a89..5da53d4 100644
--- a/232.out
+++ b/232.out
@@ -2,7 +2,6 @@ QA output created by 232
 
 Testing fsstress
 
-fsstress -n 2000 -d outdir -p 7
 seed = S
 Comparing user usage
 Comparing group usage
diff --git a/233 b/233
index 28e6ac7..649de51 100755
--- a/233
+++ b/233
@@ -58,12 +58,12 @@ _fsstress()
 
 	out=$SCRATCH_MNT/fsstress.$$
 	count=5000
-	args="-z \
+	args=`_scale_fsstress_args -z \
 -f rmdir=20 -f link=10 -f creat=10 -f mkdir=10 -f unlink=20 -f symlink=10 \
 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 \
--n $count -d $out -p 7"
+-n $count -d $out -p 7`
 
-	echo "fsstress $args" | tee -a $here/$seq.full | sed -e "s#$out#outdir#"
+	echo "fsstress $args" >> tee -a $here/$seq.full
 	if ! su $qa_user -c "$FSSTRESS_PROG $args" | tee -a $here/$seq.full | _filter_num
 	then
 		echo "    fsstress $args returned $?"
diff --git a/233.out b/233.out
index fa36ca3..91c1a30 100644
--- a/233.out
+++ b/233.out
@@ -2,7 +2,6 @@ QA output created by 233
 
 Testing fsstress
 
-fsstress -z -f rmdir=20 -f link=10 -f creat=10 -f mkdir=10 -f unlink=20 -f symlink=10 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 -n 5000 -d outdir -p 7
 seed = S
 Comparing user usage
 Comparing group usage
diff --git a/269 b/269
index 7e13ed9..7d63b87 100755
--- a/269
+++ b/269
@@ -45,7 +45,7 @@ _workout()
 	num_iterations=10
 	enospc_time=2
 	out=$SCRATCH_MNT/fsstress.$$
-	args="-p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
+	args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
 	echo "fsstress $args" >> $here/$seq.full
 	$FSSTRESS_PROG $args > /dev/null 2>&1 &
 	pid=$!
diff --git a/270 b/270
index b9ada27..b753923 100755
--- a/270
+++ b/270
@@ -48,7 +48,7 @@ _workout()
 	num_iterations=10
 	enospc_time=2
 	out=$SCRATCH_MNT/fsstress.$$
-	args="-p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
+	args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
 	echo "fsstress $args" >> $here/$seq.full
 	# Grant chown capability 
 	cp $FSSTRESS_PROG  $tmp.fsstress.bin
diff --git a/common.rc b/common.rc
index 14de47b..3635a45 100644
--- a/common.rc
+++ b/common.rc
@@ -1841,6 +1841,20 @@ _test_batched_discard()
 	$FSTRIM_PROG ${1} &>/dev/null
 }
 
+_scale_fsstress_args()
+{
+    args=""                                                                                                         
+    while [ $# -gt 0 ]; do
+        case "$1" in
+            -n) args="$args $1 $(($2 * $TIME_FACTOR))"; shift ;;
+            -p) args="$args $1 $(($2 * $LOAD_FACTOR))"; shift ;;
+            *) args="$args $1" ;;
+        esac                                                                                                    
+        shift
+    done
+    echo $args                                                                                            
+}
+  
 ################################################################################
 
 if [ "$iam" != new -a "$iam" != bench ]
diff --git a/group b/group
index 5504557..1fb2feb 100644
--- a/group
+++ b/group
@@ -137,7 +137,7 @@ stress
 014 rw udf auto quick
 015 other auto quick
 016 rw auto quick
-017 mount auto quick
+017 mount auto quick stress
 018 deprecated # log logprint v2log
 019 mkfs auto quick
 020 metadata attr udf auto quick
@@ -188,26 +188,26 @@ stress
 065 dump auto
 066 dump ioctl auto quick
 067 acl attr auto quick
-068 other auto freeze dangerous
+068 other auto freeze dangerous stress
 069 rw udf auto quick
-070 attr udf auto quick
+070 attr udf auto quick stress
 071 rw auto
 072 rw auto prealloc quick
 073 copy auto
 074 rw udf auto
 075 rw udf auto quick
-076 metadata rw udf auto quick
+076 metadata rw udf auto quick stress
 077 acl attr auto enospc
 078 growfs auto quick
 079 acl attr ioctl metadata auto quick
 080 rw ioctl
 081 deprecated # log logprint quota
 082 deprecated # log logprint v2log
-083 rw auto
+083 rw auto enospc stress
 084 ioctl rw auto
 085 log auto quick
 086 log v2log auto
-087 log v2log auto quota
+087 log v2log auto quota stress
 088 perms auto quick
 089 metadata auto
 090 rw auto
@@ -224,7 +224,7 @@ stress
 101 udf
 102 udf
 103 metadata dir ioctl auto quick
-104 growfs ioctl prealloc auto
+104 growfs ioctl prealloc auto stress
 105 acl auto quick
 106 quota
 107 quota
@@ -234,7 +234,7 @@ stress
 111 ioctl
 112 rw aio auto quick
 113 rw aio auto quick
-114 parent attr
+114 parent attr stress
 115 parent attr
 116 quota auto quick
 117 attr auto quick
@@ -287,7 +287,7 @@ stress
 164 rw pattern auto prealloc quick
 165 rw pattern auto prealloc quick
 166 rw metadata auto quick
-167 rw metadata auto
+167 rw metadata auto stress
 168 dmapi auto
 169 rw metadata auto quick
 170 rw filestreams auto quick
@@ -355,8 +355,8 @@ stress
 229 auto rw
 230 auto quota quick
 231 auto quota
-232 auto quota
-233 auto quota
+232 auto quota stress
+233 auto quota stress
 234 auto quota
 235 auto quota quick
 236 auto quick metadata
@@ -392,8 +392,8 @@ stress
 266 dump ioctl auto quick
 267 dump ioctl tape
 268 dump ioctl tape
-269 auto rw prealloc ioctl enospc
-270 auto quota rw prealloc ioctl enospc
+269 auto rw prealloc ioctl enospc stress
+270 auto quota rw prealloc ioctl enospc stress
 271 auto rw quick
 272 auto enospc rw
 273 auto rw
-- 
1.7.1

[PATCH 05/10] xfstest: allow fsstress to use load factor where appropriate

[Dmitry Monakhov]

1) Add _scale_fsstress_args function which transform argumets according
   to load factors
2) Let all non deterministic fsstress tests to use scaled arguments

I've able to trigger OOPS on xfs see:https://gist.github.com/dmonakhov/4762653

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 017       |    3 ++-
 068       |    4 ++--
 070       |    5 +++--
 076       |    5 +++--
 083       |    4 ++--
 087       |    3 ++-
 104       |    4 +++-
 114       |    4 ++--
 167       |    5 +++--
 232       |    4 ++--
 232.out   |    1 -
 233       |    6 +++---
 233.out   |    1 -
 269       |    2 +-
 270       |    2 +-
 common.rc |   14 ++++++++++++++
 group     |   26 +++++++++++++-------------
 17 files changed, 56 insertions(+), 37 deletions(-)

diff --git a/017 b/017
index 9ca0e72..7db4667 100755
--- a/017
+++ b/017
@@ -67,7 +67,8 @@ echo "*** test"
 for l in 0 1 2 3 4
 do
         echo "    *** test $l"
-        $FSSTRESS_PROG -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID >>$seq.full
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID`
+        $FSSTRESS_PROG  $FSSTRESS_ARGS >>$seq.full
 
         _scratch_mount -o remount,ro \
             || _fail "remount ro failed"
diff --git a/068 b/068
index a641e2f..cbfea95 100755
--- a/068
+++ b/068
@@ -82,8 +82,8 @@ touch $tmp.running
       do
       # We do both read & write IO - not only is this more realistic,
       # but it also potentially tests atime updates
-      $FSSTRESS_PROG -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID \
-	  > /dev/null 2>&1
+      FSSTRESS_ARGS=`_scale_fsstress_args -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID`
+      $FSSTRESS_PROG $FSSTRESS_ARGS > /dev/null 2>&1
     done
 
     rm -r $STRESS_DIR/*
diff --git a/070 b/070
index f48c33c..334cce7 100755
--- a/070
+++ b/070
@@ -52,7 +52,7 @@ _require_attrs
 
 _setup_testdir
 
-$FSSTRESS_PROG \
+FSSTRESS_ARGS=`_scale_fsstress_args \
 	-d $testdir/fsstress \
 	-f allocsp=0 \
 	-f freesp=0 \
@@ -62,7 +62,8 @@ $FSSTRESS_PROG \
 	-f unresvsp=0 \
 	-f attr_set=100 \
 	-f attr_remove=100 \
-        -p 1 -n 10000 -S c >$seq.full 2>&1
+        -p 1 -n 10000 -S c`
+$FSSTRESS_PROG $FSSTRESS_ARGS >$seq.full 2>&1
 
 status=$?
 exit
diff --git a/076 b/076
index e472b26..793b869 100755
--- a/076
+++ b/076
@@ -74,8 +74,9 @@ echo "*** test concurrent block/fs access"
 cat $SCRATCH_DEV >/dev/null &
 pid=$!
 
-$FSSTRESS_PROG -d $SCRATCH_MNT -p 2 -n 2000 $FSSTRESS_AVOID >>$seq.full
-
+FSSTRESS_ARGS=`_scale_fsstress_args -p 2 -n 2000 $FSSTRESS_AVOID`
+echo "run fsstress with args: $FSSTRESS_ARGS" >>$seq.full
+$FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full
 _lets_get_pidst
 _check_scratch_fs
 
diff --git a/083 b/083
index e0670b9..f5349a9 100755
--- a/083
+++ b/083
@@ -84,8 +84,8 @@ workout()
 		|| _fail "mount failed"
 
 	# -w ensures that the only ops are ones which cause write I/O
-	$FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID \
-		>>$seq.full
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full
 	_check_scratch_fs
 }
 
diff --git a/087 b/087
index 48e5eaa..e7be9a0 100755
--- a/087
+++ b/087
@@ -47,7 +47,8 @@ _do_meta()
     param="-p 4 -z -f rmdir=10 -f link=10 -f creat=10 -f mkdir=10 \
            -f rename=30 -f stat=30 -f unlink=30 -f truncate=20"
     _echofull "calling fsstress $param -m8 -n $count"
-    if ! $FSSTRESS_PROG $param $FSSTRESS_AVOID -m 8 -n $count -d $out >>$seq.full 2>&1
+    FSSTRESS_ARGS=`_scale_fsstress_args $param $FSSTRESS_AVOID -m 8 -n $count -d $out`
+    if ! $FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full 2>&1
     then
 	_echofull "fsstress failed"
     fi
diff --git a/104 b/104
index 14f2669..8db6d88 100755
--- a/104
+++ b/104
@@ -64,7 +64,9 @@ _stress_scratch()
 	procs=3
 	nops=1000
 	# -w ensures that the only ops are ones which cause write I/O
-	$FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID > /dev/null &
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
+	    -n $nops $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seq.full &
 }
 
 # real QA test starts here
diff --git a/114 b/114
index 7679222..edce0f4 100755
--- a/114
+++ b/114
@@ -246,11 +246,11 @@ _test_fsstress()
 
 	out=$SCRATCH_MNT/fsstress.$$
 	count=1000
-	args="-z \
+	args=`_scale_fsstress_args -z \
 -f rmdir=10 -f link=10 -f creat=10 \
 -f mkdir=10 -f rename=30 -f unlink=10 \
 -f symlink=10 \
--n $count -d $out -p 3"
+-n $count -d $out -p 3`
 
 	echo "fsstress $args" | sed -e "s#$out#outdir#"
 	if ! $FSSTRESS_PROG $args | _filter_num
diff --git a/167 b/167
index ccb6c2a..5fb95e8 100755
--- a/167
+++ b/167
@@ -44,8 +44,9 @@ workout()
 {
 	procs=100
 	nops=15000
-	$FSSTRESS_PROG -d $SCRATCH_MNT -p $procs -n $nops $FSSTRESS_AVOID \
-		>>$seq.full &
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p $procs -n $nops \
+	    $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seq.full &
 	sleep 2
 }
 
diff --git a/232 b/232
index 2795da7..d915d42 100755
--- a/232
+++ b/232
@@ -54,9 +54,9 @@ _fsstress()
 
 	out=$SCRATCH_MNT/fsstress.$$
 	count=2000
-	args="-n $count -d $out -p 7"
+	args=`_scale_fsstress_args -d $out -n $count -p 7`
 
-	echo "fsstress $args" | tee -a $here/$seq.full | sed -e "s#$out#outdir#"
+	echo "fsstress $args" >> tee -a $here/$seq.full
 	if ! $FSSTRESS_PROG $args | tee -a $here/$seq.full | _filter_num
 	then
 		echo "    fsstress $args returned $?"
diff --git a/232.out b/232.out
index ef82a89..5da53d4 100644
--- a/232.out
+++ b/232.out
@@ -2,7 +2,6 @@ QA output created by 232
 
 Testing fsstress
 
-fsstress -n 2000 -d outdir -p 7
 seed = S
 Comparing user usage
 Comparing group usage
diff --git a/233 b/233
index 28e6ac7..649de51 100755
--- a/233
+++ b/233
@@ -58,12 +58,12 @@ _fsstress()
 
 	out=$SCRATCH_MNT/fsstress.$$
 	count=5000
-	args="-z \
+	args=`_scale_fsstress_args -z \
 -f rmdir=20 -f link=10 -f creat=10 -f mkdir=10 -f unlink=20 -f symlink=10 \
 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 \
--n $count -d $out -p 7"
+-n $count -d $out -p 7`
 
-	echo "fsstress $args" | tee -a $here/$seq.full | sed -e "s#$out#outdir#"
+	echo "fsstress $args" >> tee -a $here/$seq.full
 	if ! su $qa_user -c "$FSSTRESS_PROG $args" | tee -a $here/$seq.full | _filter_num
 	then
 		echo "    fsstress $args returned $?"
diff --git a/233.out b/233.out
index fa36ca3..91c1a30 100644
--- a/233.out
+++ b/233.out
@@ -2,7 +2,6 @@ QA output created by 233
 
 Testing fsstress
 
-fsstress -z -f rmdir=20 -f link=10 -f creat=10 -f mkdir=10 -f unlink=20 -f symlink=10 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 -n 5000 -d outdir -p 7
 seed = S
 Comparing user usage
 Comparing group usage
diff --git a/269 b/269
index 7e13ed9..7d63b87 100755
--- a/269
+++ b/269
@@ -45,7 +45,7 @@ _workout()
 	num_iterations=10
 	enospc_time=2
 	out=$SCRATCH_MNT/fsstress.$$
-	args="-p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
+	args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
 	echo "fsstress $args" >> $here/$seq.full
 	$FSSTRESS_PROG $args > /dev/null 2>&1 &
 	pid=$!
diff --git a/270 b/270
index b9ada27..b753923 100755
--- a/270
+++ b/270
@@ -48,7 +48,7 @@ _workout()
 	num_iterations=10
 	enospc_time=2
 	out=$SCRATCH_MNT/fsstress.$$
-	args="-p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
+	args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
 	echo "fsstress $args" >> $here/$seq.full
 	# Grant chown capability 
 	cp $FSSTRESS_PROG  $tmp.fsstress.bin
diff --git a/common.rc b/common.rc
index 14de47b..3635a45 100644
--- a/common.rc
+++ b/common.rc
@@ -1841,6 +1841,20 @@ _test_batched_discard()
 	$FSTRIM_PROG ${1} &>/dev/null
 }
 
+_scale_fsstress_args()
+{
+    args=""                                                                                                         
+    while [ $# -gt 0 ]; do
+        case "$1" in
+            -n) args="$args $1 $(($2 * $TIME_FACTOR))"; shift ;;
+            -p) args="$args $1 $(($2 * $LOAD_FACTOR))"; shift ;;
+            *) args="$args $1" ;;
+        esac                                                                                                    
+        shift
+    done
+    echo $args                                                                                            
+}
+  
 ################################################################################
 
 if [ "$iam" != new -a "$iam" != bench ]
diff --git a/group b/group
index 5504557..1fb2feb 100644
--- a/group
+++ b/group
@@ -137,7 +137,7 @@ stress
 014 rw udf auto quick
 015 other auto quick
 016 rw auto quick
-017 mount auto quick
+017 mount auto quick stress
 018 deprecated # log logprint v2log
 019 mkfs auto quick
 020 metadata attr udf auto quick
@@ -188,26 +188,26 @@ stress
 065 dump auto
 066 dump ioctl auto quick
 067 acl attr auto quick
-068 other auto freeze dangerous
+068 other auto freeze dangerous stress
 069 rw udf auto quick
-070 attr udf auto quick
+070 attr udf auto quick stress
 071 rw auto
 072 rw auto prealloc quick
 073 copy auto
 074 rw udf auto
 075 rw udf auto quick
-076 metadata rw udf auto quick
+076 metadata rw udf auto quick stress
 077 acl attr auto enospc
 078 growfs auto quick
 079 acl attr ioctl metadata auto quick
 080 rw ioctl
 081 deprecated # log logprint quota
 082 deprecated # log logprint v2log
-083 rw auto
+083 rw auto enospc stress
 084 ioctl rw auto
 085 log auto quick
 086 log v2log auto
-087 log v2log auto quota
+087 log v2log auto quota stress
 088 perms auto quick
 089 metadata auto
 090 rw auto
@@ -224,7 +224,7 @@ stress
 101 udf
 102 udf
 103 metadata dir ioctl auto quick
-104 growfs ioctl prealloc auto
+104 growfs ioctl prealloc auto stress
 105 acl auto quick
 106 quota
 107 quota
@@ -234,7 +234,7 @@ stress
 111 ioctl
 112 rw aio auto quick
 113 rw aio auto quick
-114 parent attr
+114 parent attr stress
 115 parent attr
 116 quota auto quick
 117 attr auto quick
@@ -287,7 +287,7 @@ stress
 164 rw pattern auto prealloc quick
 165 rw pattern auto prealloc quick
 166 rw metadata auto quick
-167 rw metadata auto
+167 rw metadata auto stress
 168 dmapi auto
 169 rw metadata auto quick
 170 rw filestreams auto quick
@@ -355,8 +355,8 @@ stress
 229 auto rw
 230 auto quota quick
 231 auto quota
-232 auto quota
-233 auto quota
+232 auto quota stress
+233 auto quota stress
 234 auto quota
 235 auto quota quick
 236 auto quick metadata
@@ -392,8 +392,8 @@ stress
 266 dump ioctl auto quick
 267 dump ioctl tape
 268 dump ioctl tape
-269 auto rw prealloc ioctl enospc
-270 auto quota rw prealloc ioctl enospc
+269 auto rw prealloc ioctl enospc stress
+270 auto quota rw prealloc ioctl enospc stress
 271 auto rw quick
 272 auto enospc rw
 273 auto rw
-- 
1.7.1

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 05/10] xfstest: allow fsstress to use load factor where appropriate

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> 1) Add _scale_fsstress_args function which transform argumets according
>     to load factors
> 2) Let all non deterministic fsstress tests to use scaled arguments
>
> I've able to trigger OOPS on xfs see:https://gist.github.com/dmonakhov/4762653
>

I did not see any OOPS.  Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 06/10] xfstest: move run_check to common.rc

[Dmitry Monakhov]

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 276       |    6 ------
 common.rc |    8 +++++++-
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/276 b/276
index 082f943..c1ce8ac 100755
--- a/276
+++ b/276
@@ -165,12 +165,6 @@ _btrfs_inspect_check()
 	return $ret
 }
 
-run_check()
-{
-	echo "# $@" >> $seq.full 2>&1
-	"$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
-}
-
 workout()
 {
 	fsz=$1
diff --git a/common.rc b/common.rc
index 3635a45..e7ae9bf 100644
--- a/common.rc
+++ b/common.rc
@@ -1854,7 +1854,13 @@ _scale_fsstress_args()
     done
     echo $args                                                                                            
 }
-  
+
+run_check()
+{
+	echo "# $@" >> $seq.full 2>&1
+	"$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
+}
+
 ################################################################################
 
 if [ "$iam" != new -a "$iam" != bench ]
-- 
1.7.1

[PATCH 06/10] xfstest: move run_check to common.rc

[Dmitry Monakhov]

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 276       |    6 ------
 common.rc |    8 +++++++-
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/276 b/276
index 082f943..c1ce8ac 100755
--- a/276
+++ b/276
@@ -165,12 +165,6 @@ _btrfs_inspect_check()
 	return $ret
 }
 
-run_check()
-{
-	echo "# $@" >> $seq.full 2>&1
-	"$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
-}
-
 workout()
 {
 	fsz=$1
diff --git a/common.rc b/common.rc
index 3635a45..e7ae9bf 100644
--- a/common.rc
+++ b/common.rc
@@ -1854,7 +1854,13 @@ _scale_fsstress_args()
     done
     echo $args                                                                                            
 }
-  
+
+run_check()
+{
+	echo "# $@" >> $seq.full 2>&1
+	"$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
+}
+
 ################################################################################
 
 if [ "$iam" != new -a "$iam" != bench ]
-- 
1.7.1

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 06/10] xfstest: move run_check to common.rc

[Rich Johnston]

Looks good, will add the following simple description when this is committed

Move run_check to common.rc

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 07/10] xfstest: add fallocate/truncate vs AIO/DIO stress test

[Dmitry Monakhov]

Run  DIO, fallocate and truncate threads on a common file in parallel.
If race exist old dio request may rewrite blocks after it was allocated
to another file, we will catch that by verifying blocks content.

this patch known to catch deadlock for ext4
http://lists.openwall.net/linux-ext4/2012/09/06/3

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 297     |  157 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 297.out |    5 ++
 group   |    1 +
 3 files changed, 163 insertions(+), 0 deletions(-)
 create mode 100755 297
 create mode 100644 297.out

diff --git a/297 b/297
new file mode 100755
index 0000000..83bc572
--- /dev/null
+++ b/297
@@ -0,0 +1,157 @@
+#! /bin/bash
+# FSQA Test No. 297
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity and fallocate/truncate simultaneously
+# Test will operate on huge sparsed files so ENOSPC is expected.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+iodepth=128*${LOAD_FACTOR}
+continue_on_error=write
+ignore_error=,ENOSPC
+error_dump=0
+create_on_open=1
+fallocate=none
+exitall=1
+
+## Perform direct aio, to files which may be truncated
+## by external task
+[direct_aio]
+direct=1
+buffered=0
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=100*${TIME_FACTOR}
+time_based
+
+# Perform direct aio and verify data
+# This test case should check use-after-free issues
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=1
+bs=4k
+rw=randrw
+filename=aio-dio-verifier
+
+# Perform buffered aio and verify data
+# This test case should check use-after-free issues
+[buffered-aio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=0
+buffered=1
+bs=4k
+rw=randrw
+filename=buffered-aio-verifier
+EOF
+
+_require_fio $tmp-$seq.fio
+
+_workout()
+{
+	echo ""
+	echo "Run fio with random aio-dio pattern"
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio &
+	pid=$!
+	echo "Start fallocate/truncate loop"
+	
+	for ((i=0; ; i++))
+	do
+	    for ((k=1; k <= NUM_JOBS; k++))
+	    do
+		fallocate -l $FILE_SIZE $SCRATCH_MNT/direct_aio.$k.0 \
+		    	>> $seq.full 2>&1
+	    done
+	    for ((k=1; k <= NUM_JOBS; k++))
+	    do
+		truncate -s 0 $SCRATCH_MNT/direct_aio.$k.0
+	    done
+	    # Following like will check that pid is still run.
+	    # Once fio exit we can stop fallocate/truncate loop
+	    kill -0 $pid > /dev/null 2>&1 || break
+	done
+	wait $pid
+}
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/297.out b/297.out
new file mode 100644
index 0000000..435fac7
--- /dev/null
+++ b/297.out
@@ -0,0 +1,5 @@
+QA output created by 297
+
+Run fio with random aio-dio pattern
+
+Start fallocate/truncate loop
diff --git a/group b/group
index 1fb2feb..2783286 100644
--- a/group
+++ b/group
@@ -420,3 +420,4 @@ stress
 294 auto quick
 295 auto logprint quick
 296 dump auto quick
+297 auto aio enospc rw stress
\ No newline at end of file
-- 
1.7.1

[PATCH 07/10] xfstest: add fallocate/truncate vs AIO/DIO stress test

[Dmitry Monakhov]

Run  DIO, fallocate and truncate threads on a common file in parallel.
If race exist old dio request may rewrite blocks after it was allocated
to another file, we will catch that by verifying blocks content.

this patch known to catch deadlock for ext4
http://lists.openwall.net/linux-ext4/2012/09/06/3

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 297     |  157 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 297.out |    5 ++
 group   |    1 +
 3 files changed, 163 insertions(+), 0 deletions(-)
 create mode 100755 297
 create mode 100644 297.out

diff --git a/297 b/297
new file mode 100755
index 0000000..83bc572
--- /dev/null
+++ b/297
@@ -0,0 +1,157 @@
+#! /bin/bash
+# FSQA Test No. 297
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity and fallocate/truncate simultaneously
+# Test will operate on huge sparsed files so ENOSPC is expected.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+iodepth=128*${LOAD_FACTOR}
+continue_on_error=write
+ignore_error=,ENOSPC
+error_dump=0
+create_on_open=1
+fallocate=none
+exitall=1
+
+## Perform direct aio, to files which may be truncated
+## by external task
+[direct_aio]
+direct=1
+buffered=0
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=100*${TIME_FACTOR}
+time_based
+
+# Perform direct aio and verify data
+# This test case should check use-after-free issues
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=1
+bs=4k
+rw=randrw
+filename=aio-dio-verifier
+
+# Perform buffered aio and verify data
+# This test case should check use-after-free issues
+[buffered-aio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=0
+buffered=1
+bs=4k
+rw=randrw
+filename=buffered-aio-verifier
+EOF
+
+_require_fio $tmp-$seq.fio
+
+_workout()
+{
+	echo ""
+	echo "Run fio with random aio-dio pattern"
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio &
+	pid=$!
+	echo "Start fallocate/truncate loop"
+	
+	for ((i=0; ; i++))
+	do
+	    for ((k=1; k <= NUM_JOBS; k++))
+	    do
+		fallocate -l $FILE_SIZE $SCRATCH_MNT/direct_aio.$k.0 \
+		    	>> $seq.full 2>&1
+	    done
+	    for ((k=1; k <= NUM_JOBS; k++))
+	    do
+		truncate -s 0 $SCRATCH_MNT/direct_aio.$k.0
+	    done
+	    # Following like will check that pid is still run.
+	    # Once fio exit we can stop fallocate/truncate loop
+	    kill -0 $pid > /dev/null 2>&1 || break
+	done
+	wait $pid
+}
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/297.out b/297.out
new file mode 100644
index 0000000..435fac7
--- /dev/null
+++ b/297.out
@@ -0,0 +1,5 @@
+QA output created by 297
+
+Run fio with random aio-dio pattern
+
+Start fallocate/truncate loop
diff --git a/group b/group
index 1fb2feb..2783286 100644
--- a/group
+++ b/group
@@ -420,3 +420,4 @@ stress
 294 auto quick
 295 auto logprint quick
 296 dump auto quick
+297 auto aio enospc rw stress
\ No newline at end of file
-- 
1.7.1

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 07/10] xfstest: add fallocate/truncate vs AIO/DIO stress test

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Run  DIO, fallocate and truncate threads on a common file in parallel.
> If race exist old dio request may rewrite blocks after it was allocated
> to another file, we will catch that by verifying blocks content.
>
> this patch known to catch deadlock for ext4
> http://lists.openwall.net/linux-ext4/2012/09/06/3
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

Re: [PATCH 07/10] xfstest: add fallocate/truncate vs AIO/DIO stress test

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Run  DIO, fallocate and truncate threads on a common file in parallel.
> If race exist old dio request may rewrite blocks after it was allocated
> to another file, we will catch that by verifying blocks content.
>
> this patch known to catch deadlock for ext4
> http://lists.openwall.net/linux-ext4/2012/09/06/3
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 08/10] xfstest: add fallocate/punch_hole vs AIO/DIO stress test

[Dmitry Monakhov]

Run  random AIO/DIO activity (fio's job:direct_aio_raicer)
random fallocate activity(fio's job:falloc_raicer)
and random punch_hole activity(punch_hole_raicer) on a common
file in parallel. If race exist old dio request may rewrite
punched block after it was allocated to another file, we will
catch that by verifier fio's job: "aio-dio-verifier".

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 298     |  153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 298.out |    4 ++
 group   |    3 +-
 3 files changed, 159 insertions(+), 1 deletions(-)
 create mode 100755 298
 create mode 100644 298.out

diff --git a/298 b/298
new file mode 100755
index 0000000..3264b8e
--- /dev/null
+++ b/298
@@ -0,0 +1,153 @@
+#! /bin/bash
+# FSQA Test No. 298
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity and fallocate/punch_hole simultaneously
+# Test will operate on huge sparsed file so ENOSPC is expected.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+if [ $((BLK_DEV_SIZE)) -gt 1048576 ]
+then
+    BLK_DEV_SIZE=1048576
+fi 
+FS_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test fio activity
+# Run  DIO, fallocate and punch_hole threads on a single in parallel 
+#
+# If race exist old dio request may rewrite punched block after it was
+# allocated to another file, we will catch that by verifying blocks content
+#
+[global]
+directory=${SCRATCH_MNT}
+filesize=${FS_SIZE}
+size=999G
+continue_on_error=write
+ignore_error=,ENOSPC
+error_dump=0
+
+create_on_open=1
+fallocate=none
+exitall=1
+
+## Perform direct aio, to files which may be truncated
+## by external task
+[direct_aio_raicer]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR} 
+bs=128k 
+direct=1
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=100*${TIME_FACTOR}
+time_based
+filename=racer
+
+# Run falloc and punch_hole threads in parallel
+# After activity file will be highly fragmented
+[falloc_raicer]
+ioengine=falloc
+runtime=100*${TIME_FACTOR}
+iodepth=1
+bssplit=128k/80:512k/10:32k/10
+rw=randwrite
+numjobs=1
+filename=racer
+
+[punch_hole_raicer]
+ioengine=falloc
+runtime=100*${TIME_FACTOR}
+bs=4k
+time_based=10
+rw=randtrim
+numjobs=2
+filename=racer
+time_based
+
+# Verifier thread continiously write to newly allcated blocks
+# and veryfy written content
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=1
+bs=4k
+rw=randwrite
+filename=aio-dio-verifier
+EOF
+
+_workout()
+{
+	echo ""
+	echo "Run fio with random aio-dio pattern"
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs_sized $FS_SIZE >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/298.out b/298.out
new file mode 100644
index 0000000..faedc77
--- /dev/null
+++ b/298.out
@@ -0,0 +1,4 @@
+QA output created by 298
+
+Run fio with random aio-dio pattern
+
diff --git a/group b/group
index 2783286..ff893ac 100644
--- a/group
+++ b/group
@@ -420,4 +420,5 @@ stress
 294 auto quick
 295 auto logprint quick
 296 dump auto quick
-297 auto aio enospc rw stress
\ No newline at end of file
+297 auto aio enospc rw stress
+298 auto aio enospc preallocrw stress
\ No newline at end of file
-- 
1.7.1

[PATCH 08/10] xfstest: add fallocate/punch_hole vs AIO/DIO stress test

[Dmitry Monakhov]

Run  random AIO/DIO activity (fio's job:direct_aio_raicer)
random fallocate activity(fio's job:falloc_raicer)
and random punch_hole activity(punch_hole_raicer) on a common
file in parallel. If race exist old dio request may rewrite
punched block after it was allocated to another file, we will
catch that by verifier fio's job: "aio-dio-verifier".

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 298     |  153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 298.out |    4 ++
 group   |    3 +-
 3 files changed, 159 insertions(+), 1 deletions(-)
 create mode 100755 298
 create mode 100644 298.out

diff --git a/298 b/298
new file mode 100755
index 0000000..3264b8e
--- /dev/null
+++ b/298
@@ -0,0 +1,153 @@
+#! /bin/bash
+# FSQA Test No. 298
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity and fallocate/punch_hole simultaneously
+# Test will operate on huge sparsed file so ENOSPC is expected.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+if [ $((BLK_DEV_SIZE)) -gt 1048576 ]
+then
+    BLK_DEV_SIZE=1048576
+fi 
+FS_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test fio activity
+# Run  DIO, fallocate and punch_hole threads on a single in parallel 
+#
+# If race exist old dio request may rewrite punched block after it was
+# allocated to another file, we will catch that by verifying blocks content
+#
+[global]
+directory=${SCRATCH_MNT}
+filesize=${FS_SIZE}
+size=999G
+continue_on_error=write
+ignore_error=,ENOSPC
+error_dump=0
+
+create_on_open=1
+fallocate=none
+exitall=1
+
+## Perform direct aio, to files which may be truncated
+## by external task
+[direct_aio_raicer]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR} 
+bs=128k 
+direct=1
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=100*${TIME_FACTOR}
+time_based
+filename=racer
+
+# Run falloc and punch_hole threads in parallel
+# After activity file will be highly fragmented
+[falloc_raicer]
+ioengine=falloc
+runtime=100*${TIME_FACTOR}
+iodepth=1
+bssplit=128k/80:512k/10:32k/10
+rw=randwrite
+numjobs=1
+filename=racer
+
+[punch_hole_raicer]
+ioengine=falloc
+runtime=100*${TIME_FACTOR}
+bs=4k
+time_based=10
+rw=randtrim
+numjobs=2
+filename=racer
+time_based
+
+# Verifier thread continiously write to newly allcated blocks
+# and veryfy written content
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=1
+bs=4k
+rw=randwrite
+filename=aio-dio-verifier
+EOF
+
+_workout()
+{
+	echo ""
+	echo "Run fio with random aio-dio pattern"
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs_sized $FS_SIZE >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/298.out b/298.out
new file mode 100644
index 0000000..faedc77
--- /dev/null
+++ b/298.out
@@ -0,0 +1,4 @@
+QA output created by 298
+
+Run fio with random aio-dio pattern
+
diff --git a/group b/group
index 2783286..ff893ac 100644
--- a/group
+++ b/group
@@ -420,4 +420,5 @@ stress
 294 auto quick
 295 auto logprint quick
 296 dump auto quick
-297 auto aio enospc rw stress
\ No newline at end of file
+297 auto aio enospc rw stress
+298 auto aio enospc preallocrw stress
\ No newline at end of file
-- 
1.7.1

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 08/10] xfstest: add fallocate/punch_hole vs AIO/DIO stress test

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Run  random AIO/DIO activity (fio's job:direct_aio_raicer)
> random fallocate activity(fio's job:falloc_raicer)
> and random punch_hole activity(punch_hole_raicer) on a common
> file in parallel. If race exist old dio request may rewrite
> punched block after it was allocated to another file, we will
> catch that by verifier fio's job: "aio-dio-verifier".
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

Re: [PATCH 08/10] xfstest: add fallocate/punch_hole vs AIO/DIO stress test

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Run  random AIO/DIO activity (fio's job:direct_aio_raicer)
> random fallocate activity(fio's job:falloc_raicer)
> and random punch_hole activity(punch_hole_raicer) on a common
> file in parallel. If race exist old dio request may rewrite
> punched block after it was allocated to another file, we will
> catch that by verifier fio's job: "aio-dio-verifier".
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>

Looks good.

Reviewed-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Dmitry Monakhov]

Perform various regression tests for ext4defrag  subsystem

 299'th Test1: Defragment file while other task does direct AIO
 300'th Test2: Perform defragmentation on file under buffered AIO
 	while third task does direct AIO to donor file
 301'th Test3: Two defrag tasks use common donor file.
 302'th Test4: Stress defragmentation. Several threads pefrorm
 	fragmentation at random position use inplace=1 will
	allocate and free blocks inside defrag event improve
	load pressure.

This tests are considered dengerous because 300'th and 301'th are known
to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 299           |  126 +++++++++++++++++++++++++++++++++++++++++++++++
 299.out       |    4 ++
 300           |  143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 300.out       |    4 ++
 301           |  153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 301.out       |    4 ++
 302           |  130 ++++++++++++++++++++++++++++++++++++++++++++++++
 302.out       |    4 ++
 common.defrag |    4 +-
 group         |    6 ++-
 10 files changed, 575 insertions(+), 3 deletions(-)
 create mode 100755 299
 create mode 100644 299.out
 create mode 100755 300
 create mode 100644 300.out
 create mode 100755 301
 create mode 100644 301.out
 create mode 100755 302
 create mode 100644 302.out

diff --git a/299 b/299
new file mode 100755
index 0000000..4a7a98c
--- /dev/null
+++ b/299
@@ -0,0 +1,126 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Defragment file while other task does direct io
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC 
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test1
+# Defragment file while other task does direct io
+
+# Continious sequential defrag activity
+[defrag-4k]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test1.def
+filename=test1
+inplace=0
+rw=write
+numjobs=${NUM_JOBS}
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Verifier
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=randwrite
+filename=test1
+runtime=30*${TIME_FACTOR}
+time_based
+EOF
+
+_workout()
+{
+	echo ""
+	echo " Start defragment activity "
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs  >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
diff --git a/299.out b/299.out
new file mode 100644
index 0000000..c2b0b4c
--- /dev/null
+++ b/299.out
@@ -0,0 +1,4 @@
+QA output created by 299
+
+ Start defragment activity 
+
diff --git a/300 b/300
new file mode 100755
index 0000000..ff24765
--- /dev/null
+++ b/300
@@ -0,0 +1,143 @@
+#! /bin/bash
+# FSQA Test No. 300
+#
+# Ext4 defragmentation stress test
+# Perform defragmentation on file under buffered io
+# while third task does direct io to donor file
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC 
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+##########################################
+# Test2 
+# Perform defragmentation on file under buffered io
+# while third task does direct io to donor file
+#
+# Continious sequential defrag activity
+[defrag-4k]
+stonewall
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test2.def
+filename=test2
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Run DIO/AIO for donor file
+[donor-file-fuzzer]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=${LOAD_FACTOR}
+verify=0
+direct=1
+bs=64k
+rw=randwrite
+filename=test2.def
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Verifier thread
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+buffered=1
+bs=64k
+rw=randrw
+filename=test2
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+	echo ""
+	echo " Start defragment activity "
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs  >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/300.out b/300.out
new file mode 100644
index 0000000..926d985
--- /dev/null
+++ b/300.out
@@ -0,0 +1,4 @@
+QA output created by 300
+
+ Start defragment activity 
+
diff --git a/301 b/301
new file mode 100755
index 0000000..7700c36
--- /dev/null
+++ b/301
@@ -0,0 +1,153 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Two defrag tasks use common donor file
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 3 files (one donor file and two test files)
+# Reserve space for 4 files in order to avoid ENOSPC
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (3+1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test3
+# Two defrag tasks use common donor file
+[defrag-1]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test3.def
+filename=test31
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[defrag-2]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test3.def
+filename=test32
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-dio-verifier-1]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=write
+filename=test31
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-buffer-verifier-2]
+ioengine=libaio 
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+buffered=1
+bs=64k
+rw=randrw
+filename=test32
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+	echo ""
+	echo " Start defragment activity "
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs  >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/301.out b/301.out
new file mode 100644
index 0000000..961c3bf
--- /dev/null
+++ b/301.out
@@ -0,0 +1,4 @@
+QA output created by 301
+
+ Start defragment activity 
+
diff --git a/302 b/302
new file mode 100755
index 0000000..3bd82a4
--- /dev/null
+++ b/302
@@ -0,0 +1,130 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Several threads pefrorm defragmentatin at random position
+# using 'inplace' mode (allocate and free blocks inside defrag event)
+# which significantly improve load pressure on block allocator.
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC 
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test4
+# Stress test defragmentation engine
+# Several threads pefrorm defragmentatin at random position
+# use inplace=1 will allocate and free blocks inside defrag event
+# which highly increase defragmentation
+[defrag-fuzzer]
+ioengine=e4defrag
+iodepth=1
+bs=8k
+donorname=test4.def
+filename=test4
+inplace=1
+rw=randwrite
+numjobs=4*${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128
+iomem_align=4k
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=write
+filename=test4
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+	echo ""
+	echo " Start defragment activity "
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs  >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/302.out b/302.out
new file mode 100644
index 0000000..a80ab81
--- /dev/null
+++ b/302.out
@@ -0,0 +1,4 @@
+QA output created by 302
+
+ Start defragment activity 
+
diff --git a/common.defrag b/common.defrag
index ea6c14c..84b1e65 100644
--- a/common.defrag
+++ b/common.defrag
@@ -24,10 +24,10 @@ _require_defrag()
 {
     case "$FSTYP" in
     xfs)
-        DEFRAG_PROG=/usr/sbin/xfs_fsr
+        DEFRAG_PROG="`set_prog_path xfs_fsr`"
 	;;
     ext4|ext4dev)
-        DEFRAG_PROG=/usr/bin/e4defrag
+        DEFRAG_PROG="`set_prog_path e4defrag`"
 	;;
     btrfs)
 	DEFRAG_PROG="$BTRFS_UTIL_PROG filesystem defragment"
diff --git a/group b/group
index ff893ac..2fe406b 100644
--- a/group
+++ b/group
@@ -421,4 +421,8 @@ stress
 295 auto logprint quick
 296 dump auto quick
 297 auto aio enospc rw stress
-298 auto aio enospc preallocrw stress
\ No newline at end of file
+298 auto aio enospc preallocrw stress
+299 aio dangerous ioctl rw stress
+300 aio dangerous ioctl rw stress
+301 aio dangerous ioctl rw stress
+302 aio dangerous ioctl rw stress
\ No newline at end of file
-- 
1.7.1

[PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Dmitry Monakhov]

Perform various regression tests for ext4defrag  subsystem

 299'th Test1: Defragment file while other task does direct AIO
 300'th Test2: Perform defragmentation on file under buffered AIO
 	while third task does direct AIO to donor file
 301'th Test3: Two defrag tasks use common donor file.
 302'th Test4: Stress defragmentation. Several threads pefrorm
 	fragmentation at random position use inplace=1 will
	allocate and free blocks inside defrag event improve
	load pressure.

This tests are considered dengerous because 300'th and 301'th are known
to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 299           |  126 +++++++++++++++++++++++++++++++++++++++++++++++
 299.out       |    4 ++
 300           |  143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 300.out       |    4 ++
 301           |  153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 301.out       |    4 ++
 302           |  130 ++++++++++++++++++++++++++++++++++++++++++++++++
 302.out       |    4 ++
 common.defrag |    4 +-
 group         |    6 ++-
 10 files changed, 575 insertions(+), 3 deletions(-)
 create mode 100755 299
 create mode 100644 299.out
 create mode 100755 300
 create mode 100644 300.out
 create mode 100755 301
 create mode 100644 301.out
 create mode 100755 302
 create mode 100644 302.out

diff --git a/299 b/299
new file mode 100755
index 0000000..4a7a98c
--- /dev/null
+++ b/299
@@ -0,0 +1,126 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Defragment file while other task does direct io
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC 
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test1
+# Defragment file while other task does direct io
+
+# Continious sequential defrag activity
+[defrag-4k]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test1.def
+filename=test1
+inplace=0
+rw=write
+numjobs=${NUM_JOBS}
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Verifier
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=randwrite
+filename=test1
+runtime=30*${TIME_FACTOR}
+time_based
+EOF
+
+_workout()
+{
+	echo ""
+	echo " Start defragment activity "
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs  >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
diff --git a/299.out b/299.out
new file mode 100644
index 0000000..c2b0b4c
--- /dev/null
+++ b/299.out
@@ -0,0 +1,4 @@
+QA output created by 299
+
+ Start defragment activity 
+
diff --git a/300 b/300
new file mode 100755
index 0000000..ff24765
--- /dev/null
+++ b/300
@@ -0,0 +1,143 @@
+#! /bin/bash
+# FSQA Test No. 300
+#
+# Ext4 defragmentation stress test
+# Perform defragmentation on file under buffered io
+# while third task does direct io to donor file
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC 
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+##########################################
+# Test2 
+# Perform defragmentation on file under buffered io
+# while third task does direct io to donor file
+#
+# Continious sequential defrag activity
+[defrag-4k]
+stonewall
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test2.def
+filename=test2
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Run DIO/AIO for donor file
+[donor-file-fuzzer]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=${LOAD_FACTOR}
+verify=0
+direct=1
+bs=64k
+rw=randwrite
+filename=test2.def
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Verifier thread
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+buffered=1
+bs=64k
+rw=randrw
+filename=test2
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+	echo ""
+	echo " Start defragment activity "
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs  >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/300.out b/300.out
new file mode 100644
index 0000000..926d985
--- /dev/null
+++ b/300.out
@@ -0,0 +1,4 @@
+QA output created by 300
+
+ Start defragment activity 
+
diff --git a/301 b/301
new file mode 100755
index 0000000..7700c36
--- /dev/null
+++ b/301
@@ -0,0 +1,153 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Two defrag tasks use common donor file
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 3 files (one donor file and two test files)
+# Reserve space for 4 files in order to avoid ENOSPC
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (3+1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test3
+# Two defrag tasks use common donor file
+[defrag-1]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test3.def
+filename=test31
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[defrag-2]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test3.def
+filename=test32
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-dio-verifier-1]
+ioengine=libaio 
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=write
+filename=test31
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-buffer-verifier-2]
+ioengine=libaio 
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+buffered=1
+bs=64k
+rw=randrw
+filename=test32
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+	echo ""
+	echo " Start defragment activity "
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs  >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/301.out b/301.out
new file mode 100644
index 0000000..961c3bf
--- /dev/null
+++ b/301.out
@@ -0,0 +1,4 @@
+QA output created by 301
+
+ Start defragment activity 
+
diff --git a/302 b/302
new file mode 100755
index 0000000..3bd82a4
--- /dev/null
+++ b/302
@@ -0,0 +1,130 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Several threads pefrorm defragmentatin at random position
+# using 'inplace' mode (allocate and free blocks inside defrag event)
+# which significantly improve load pressure on block allocator.
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC 
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test4
+# Stress test defragmentation engine
+# Several threads pefrorm defragmentatin at random position
+# use inplace=1 will allocate and free blocks inside defrag event
+# which highly increase defragmentation
+[defrag-fuzzer]
+ioengine=e4defrag
+iodepth=1
+bs=8k
+donorname=test4.def
+filename=test4
+inplace=1
+rw=randwrite
+numjobs=4*${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-dio-verifier]
+ioengine=libaio 
+iodepth=128
+iomem_align=4k
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=write
+filename=test4
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+	echo ""
+	echo " Start defragment activity "
+	echo ""
+	cat $tmp-$seq.fio >>  $seq.full
+	run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs  >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	umount $SCRATCH_DEV 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/302.out b/302.out
new file mode 100644
index 0000000..a80ab81
--- /dev/null
+++ b/302.out
@@ -0,0 +1,4 @@
+QA output created by 302
+
+ Start defragment activity 
+
diff --git a/common.defrag b/common.defrag
index ea6c14c..84b1e65 100644
--- a/common.defrag
+++ b/common.defrag
@@ -24,10 +24,10 @@ _require_defrag()
 {
     case "$FSTYP" in
     xfs)
-        DEFRAG_PROG=/usr/sbin/xfs_fsr
+        DEFRAG_PROG="`set_prog_path xfs_fsr`"
 	;;
     ext4|ext4dev)
-        DEFRAG_PROG=/usr/bin/e4defrag
+        DEFRAG_PROG="`set_prog_path e4defrag`"
 	;;
     btrfs)
 	DEFRAG_PROG="$BTRFS_UTIL_PROG filesystem defragment"
diff --git a/group b/group
index ff893ac..2fe406b 100644
--- a/group
+++ b/group
@@ -421,4 +421,8 @@ stress
 295 auto logprint quick
 296 dump auto quick
 297 auto aio enospc rw stress
-298 auto aio enospc preallocrw stress
\ No newline at end of file
+298 auto aio enospc preallocrw stress
+299 aio dangerous ioctl rw stress
+300 aio dangerous ioctl rw stress
+301 aio dangerous ioctl rw stress
+302 aio dangerous ioctl rw stress
\ No newline at end of file
-- 
1.7.1

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Perform various regression tests for ext4defrag  subsystem
>
>   299'th Test1: Defragment file while other task does direct AIO
>   300'th Test2: Perform defragmentation on file under buffered AIO
>   	while third task does direct AIO to donor file
>   301'th Test3: Two defrag tasks use common donor file.
>   302'th Test4: Stress defragmentation. Several threads pefrorm
typo 	                                                  perform

>   	fragmentation at random position use inplace=1 will
> 	allocate and free blocks inside defrag event improve
> 	load pressure.
>
> This tests are considered dengerous because 300'th and 301'th are known
typo                        dangerous

> to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> ---
>   299           |  126 +++++++++++++++++++++++++++++++++++++++++++++++
>   299.out       |    4 ++
>   300           |  143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>   300.out       |    4 ++
>   301           |  153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   301.out       |    4 ++
>   302           |  130 ++++++++++++++++++++++++++++++++++++++++++++++++
>   302.out       |    4 ++
>   common.defrag |    4 +-
>   group         |    6 ++-
>   10 files changed, 575 insertions(+), 3 deletions(-)
>   create mode 100755 299
>   create mode 100644 299.out
>   create mode 100755 300
>   create mode 100644 300.out
>   create mode 100755 301
>   create mode 100644 301.out
>   create mode 100755 302
>   create mode 100644 302.out
>
> diff --git a/299 b/299
> new file mode 100755
> index 0000000..4a7a98c
> --- /dev/null
> +++ b/299
> @@ -0,0 +1,126 @@
> +#! /bin/bash
> +# FSQA Test No. 299
> +#
> +# Ext4 defragmentation stress test
> +# Defragment file while other task does direct io
> +#-----------------------------------------------------------------------
> +# (c) 2013 Dmitry Monakhov
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#
> +#-----------------------------------------------------------------------
> +#
> +# creator
> +owner=dmonakhov@openvz.org
> +
> +seq=`basename $0`
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common.rc
> +. ./common.filter
> +. ./common.defrag
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs ext4
> +_need_to_be_root
> +_require_scratch
> +_require_defrag
> +
I think you meant to set NUM_JOBS here like you did in [Patch 08/10], 
something like:

+NUM_JOBS=$((4*LOAD_FACTOR))

otherwise you get

###	 [not run] /usr/local/bin/fio too old

> +BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
> +# We need space for 2 files (test file, and donor one)
> +# reserve 30% in order to avoid ENOSPC
> +FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
> +
> +cat >$tmp-$seq.fio <<EOF
> +# Common e4defrag regression tests
> +[global]
> +ioengine=ioe_e4defrag
> +iodepth=1
> +directory=${SCRATCH_MNT}
> +filesize=${FILE_SIZE}
> +size=999G
> +buffered=0
> +fadvise_hint=0
> +group_reporting
> +
> +#################################
> +# Test1
> +# Defragment file while other task does direct io
> +
> +# Continious sequential defrag activity
> +[defrag-4k]
> +ioengine=e4defrag
> +iodepth=1
> +bs=128k
> +donorname=test1.def
> +filename=test1
> +inplace=0
> +rw=write
> +numjobs=${NUM_JOBS}
> +runtime=30*${TIME_FACTOR}
> +time_based
> +
> +# Verifier
> +[aio-dio-verifier]
> +ioengine=libaio
> +iodepth=128*${LOAD_FACTOR}
> +numjobs=1
> +verify=crc32c-intel
> +verify_fatal=1
> +verify_dump=1
> +verify_backlog=1024
> +verify_async=1
> +verifysort=1
> +direct=1
> +bs=64k
> +rw=randwrite
> +filename=test1
> +runtime=30*${TIME_FACTOR}
> +time_based
> +EOF
> +
> +_workout()
> +{
> +	echo ""
> +	echo " Start defragment activity "
git does not like trailing whitespace   ^
(same for the other 3 tests)
> +	echo ""
> +	cat $tmp-$seq.fio >>  $seq.full
> +	run_check $FIO_PROG $tmp-$seq.fio
> +}
> +
> +_require_fio $tmp-$seq.fio
> +
> +_scratch_mkfs  >> $seq.full 2>&1
> +_scratch_mount
> +
> +if ! _workout; then
> +	umount $SCRATCH_DEV 2>/dev/null
> +	exit
> +fi
> +
> +if ! _scratch_unmount; then
> +	echo "failed to umount"
> +	status=1
> +	exit
> +fi
> +_check_scratch_fs
> +status=$?
> +exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
> diff --git a/299.out b/299.out
> new file mode 100644
> index 0000000..c2b0b4c
> --- /dev/null
> +++ b/299.out
> @@ -0,0 +1,4 @@
> +QA output created by 299
> +
> + Start defragment activity
remove trailing whitespace   ^
(same for the other 3 tests)

Let me know if you agree with these changes and I will make them at 
commit time.

Thanks
--Rich

Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Perform various regression tests for ext4defrag  subsystem
>
>   299'th Test1: Defragment file while other task does direct AIO
>   300'th Test2: Perform defragmentation on file under buffered AIO
>   	while third task does direct AIO to donor file
>   301'th Test3: Two defrag tasks use common donor file.
>   302'th Test4: Stress defragmentation. Several threads pefrorm
typo 	                                                  perform

>   	fragmentation at random position use inplace=1 will
> 	allocate and free blocks inside defrag event improve
> 	load pressure.
>
> This tests are considered dengerous because 300'th and 301'th are known
typo                        dangerous

> to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> ---
>   299           |  126 +++++++++++++++++++++++++++++++++++++++++++++++
>   299.out       |    4 ++
>   300           |  143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>   300.out       |    4 ++
>   301           |  153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   301.out       |    4 ++
>   302           |  130 ++++++++++++++++++++++++++++++++++++++++++++++++
>   302.out       |    4 ++
>   common.defrag |    4 +-
>   group         |    6 ++-
>   10 files changed, 575 insertions(+), 3 deletions(-)
>   create mode 100755 299
>   create mode 100644 299.out
>   create mode 100755 300
>   create mode 100644 300.out
>   create mode 100755 301
>   create mode 100644 301.out
>   create mode 100755 302
>   create mode 100644 302.out
>
> diff --git a/299 b/299
> new file mode 100755
> index 0000000..4a7a98c
> --- /dev/null
> +++ b/299
> @@ -0,0 +1,126 @@
> +#! /bin/bash
> +# FSQA Test No. 299
> +#
> +# Ext4 defragmentation stress test
> +# Defragment file while other task does direct io
> +#-----------------------------------------------------------------------
> +# (c) 2013 Dmitry Monakhov
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#
> +#-----------------------------------------------------------------------
> +#
> +# creator
> +owner=dmonakhov@openvz.org
> +
> +seq=`basename $0`
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common.rc
> +. ./common.filter
> +. ./common.defrag
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs ext4
> +_need_to_be_root
> +_require_scratch
> +_require_defrag
> +
I think you meant to set NUM_JOBS here like you did in [Patch 08/10], 
something like:

+NUM_JOBS=$((4*LOAD_FACTOR))

otherwise you get

###	 [not run] /usr/local/bin/fio too old

> +BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
> +# We need space for 2 files (test file, and donor one)
> +# reserve 30% in order to avoid ENOSPC
> +FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
> +
> +cat >$tmp-$seq.fio <<EOF
> +# Common e4defrag regression tests
> +[global]
> +ioengine=ioe_e4defrag
> +iodepth=1
> +directory=${SCRATCH_MNT}
> +filesize=${FILE_SIZE}
> +size=999G
> +buffered=0
> +fadvise_hint=0
> +group_reporting
> +
> +#################################
> +# Test1
> +# Defragment file while other task does direct io
> +
> +# Continious sequential defrag activity
> +[defrag-4k]
> +ioengine=e4defrag
> +iodepth=1
> +bs=128k
> +donorname=test1.def
> +filename=test1
> +inplace=0
> +rw=write
> +numjobs=${NUM_JOBS}
> +runtime=30*${TIME_FACTOR}
> +time_based
> +
> +# Verifier
> +[aio-dio-verifier]
> +ioengine=libaio
> +iodepth=128*${LOAD_FACTOR}
> +numjobs=1
> +verify=crc32c-intel
> +verify_fatal=1
> +verify_dump=1
> +verify_backlog=1024
> +verify_async=1
> +verifysort=1
> +direct=1
> +bs=64k
> +rw=randwrite
> +filename=test1
> +runtime=30*${TIME_FACTOR}
> +time_based
> +EOF
> +
> +_workout()
> +{
> +	echo ""
> +	echo " Start defragment activity "
git does not like trailing whitespace   ^
(same for the other 3 tests)
> +	echo ""
> +	cat $tmp-$seq.fio >>  $seq.full
> +	run_check $FIO_PROG $tmp-$seq.fio
> +}
> +
> +_require_fio $tmp-$seq.fio
> +
> +_scratch_mkfs  >> $seq.full 2>&1
> +_scratch_mount
> +
> +if ! _workout; then
> +	umount $SCRATCH_DEV 2>/dev/null
> +	exit
> +fi
> +
> +if ! _scratch_unmount; then
> +	echo "failed to umount"
> +	status=1
> +	exit
> +fi
> +_check_scratch_fs
> +status=$?
> +exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
> diff --git a/299.out b/299.out
> new file mode 100644
> index 0000000..c2b0b4c
> --- /dev/null
> +++ b/299.out
> @@ -0,0 +1,4 @@
> +QA output created by 299
> +
> + Start defragment activity
remove trailing whitespace   ^
(same for the other 3 tests)

Let me know if you agree with these changes and I will make them at 
commit time.

Thanks
--Rich

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Dmitry Monakhov]

On Fri, 1 Mar 2013 13:23:50 -0600, Rich Johnston <rjohnston@sgi.com> wrote:
> On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> > Perform various regression tests for ext4defrag  subsystem
> >
> >   299'th Test1: Defragment file while other task does direct AIO
> >   300'th Test2: Perform defragmentation on file under buffered AIO
> >   	while third task does direct AIO to donor file
> >   301'th Test3: Two defrag tasks use common donor file.
> >   302'th Test4: Stress defragmentation. Several threads pefrorm
> typo 	                                                  perform
> 
> >   	fragmentation at random position use inplace=1 will
> > 	allocate and free blocks inside defrag event improve
> > 	load pressure.
> >
> > This tests are considered dengerous because 300'th and 301'th are known
> typo                        dangerous
> 
> > to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
> >
> > Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> > ---
> >   299           |  126 +++++++++++++++++++++++++++++++++++++++++++++++
> >   299.out       |    4 ++
> >   300           |  143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >   300.out       |    4 ++
> >   301           |  153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >   301.out       |    4 ++
> >   302           |  130 ++++++++++++++++++++++++++++++++++++++++++++++++
> >   302.out       |    4 ++
> >   common.defrag |    4 +-
> >   group         |    6 ++-
> >   10 files changed, 575 insertions(+), 3 deletions(-)
> >   create mode 100755 299
> >   create mode 100644 299.out
> >   create mode 100755 300
> >   create mode 100644 300.out
> >   create mode 100755 301
> >   create mode 100644 301.out
> >   create mode 100755 302
> >   create mode 100644 302.out
> >
> > diff --git a/299 b/299
> > new file mode 100755
> > index 0000000..4a7a98c
> > --- /dev/null
> > +++ b/299
> > @@ -0,0 +1,126 @@
> > +#! /bin/bash
> > +# FSQA Test No. 299
> > +#
> > +# Ext4 defragmentation stress test
> > +# Defragment file while other task does direct io
> > +#-----------------------------------------------------------------------
> > +# (c) 2013 Dmitry Monakhov
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > +#
> > +#-----------------------------------------------------------------------
> > +#
> > +# creator
> > +owner=dmonakhov@openvz.org
> > +
> > +seq=`basename $0`
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1	# failure is the default!
> > +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> > +
> > +# get standard environment, filters and checks
> > +. ./common.rc
> > +. ./common.filter
> > +. ./common.defrag
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> > +_supported_fs ext4
> > +_need_to_be_root
> > +_require_scratch
> > +_require_defrag
> > +
> I think you meant to set NUM_JOBS here like you did in [Patch 08/10], 
> something like:
> 
> +NUM_JOBS=$((4*LOAD_FACTOR))
Indeed. I've forget to update patch.
> 
> otherwise you get
> 
> ###	 [not run] /usr/local/bin/fio too old
> 
> > +BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
> > +# We need space for 2 files (test file, and donor one)
> > +# reserve 30% in order to avoid ENOSPC
> > +FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
> > +
> > +cat >$tmp-$seq.fio <<EOF
> > +# Common e4defrag regression tests
> > +[global]
> > +ioengine=ioe_e4defrag
> > +iodepth=1
> > +directory=${SCRATCH_MNT}
> > +filesize=${FILE_SIZE}
> > +size=999G
> > +buffered=0
> > +fadvise_hint=0
> > +group_reporting
> > +
> > +#################################
> > +# Test1
> > +# Defragment file while other task does direct io
> > +
> > +# Continious sequential defrag activity
> > +[defrag-4k]
> > +ioengine=e4defrag
> > +iodepth=1
> > +bs=128k
> > +donorname=test1.def
> > +filename=test1
> > +inplace=0
> > +rw=write
> > +numjobs=${NUM_JOBS}
> > +runtime=30*${TIME_FACTOR}
> > +time_based
> > +
> > +# Verifier
> > +[aio-dio-verifier]
> > +ioengine=libaio
> > +iodepth=128*${LOAD_FACTOR}
> > +numjobs=1
> > +verify=crc32c-intel
> > +verify_fatal=1
> > +verify_dump=1
> > +verify_backlog=1024
> > +verify_async=1
> > +verifysort=1
> > +direct=1
> > +bs=64k
> > +rw=randwrite
> > +filename=test1
> > +runtime=30*${TIME_FACTOR}
> > +time_based
> > +EOF
> > +
> > +_workout()
> > +{
> > +	echo ""
> > +	echo " Start defragment activity "
> git does not like trailing whitespace   ^
> (same for the other 3 tests)
> > +	echo ""
> > +	cat $tmp-$seq.fio >>  $seq.full
> > +	run_check $FIO_PROG $tmp-$seq.fio
> > +}
> > +
> > +_require_fio $tmp-$seq.fio
> > +
> > +_scratch_mkfs  >> $seq.full 2>&1
> > +_scratch_mount
> > +
> > +if ! _workout; then
> > +	umount $SCRATCH_DEV 2>/dev/null
> > +	exit
> > +fi
> > +
> > +if ! _scratch_unmount; then
> > +	echo "failed to umount"
> > +	status=1
> > +	exit
> > +fi
> > +_check_scratch_fs
> > +status=$?
> > +exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
> > diff --git a/299.out b/299.out
> > new file mode 100644
> > index 0000000..c2b0b4c
> > --- /dev/null
> > +++ b/299.out
> > @@ -0,0 +1,4 @@
> > +QA output created by 299
> > +
> > + Start defragment activity
> remove trailing whitespace   ^
> (same for the other 3 tests)
> 
> Let me know if you agree with these changes and I will make them at 
> commit time.
Yes, please do.
> 
> Thanks
> --Rich
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Dmitry Monakhov]

On Fri, 1 Mar 2013 13:23:50 -0600, Rich Johnston <rjohnston@sgi.com> wrote:
> On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> > Perform various regression tests for ext4defrag  subsystem
> >
> >   299'th Test1: Defragment file while other task does direct AIO
> >   300'th Test2: Perform defragmentation on file under buffered AIO
> >   	while third task does direct AIO to donor file
> >   301'th Test3: Two defrag tasks use common donor file.
> >   302'th Test4: Stress defragmentation. Several threads pefrorm
> typo 	                                                  perform
> 
> >   	fragmentation at random position use inplace=1 will
> > 	allocate and free blocks inside defrag event improve
> > 	load pressure.
> >
> > This tests are considered dengerous because 300'th and 301'th are known
> typo                        dangerous
> 
> > to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
> >
> > Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> > ---
> >   299           |  126 +++++++++++++++++++++++++++++++++++++++++++++++
> >   299.out       |    4 ++
> >   300           |  143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >   300.out       |    4 ++
> >   301           |  153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >   301.out       |    4 ++
> >   302           |  130 ++++++++++++++++++++++++++++++++++++++++++++++++
> >   302.out       |    4 ++
> >   common.defrag |    4 +-
> >   group         |    6 ++-
> >   10 files changed, 575 insertions(+), 3 deletions(-)
> >   create mode 100755 299
> >   create mode 100644 299.out
> >   create mode 100755 300
> >   create mode 100644 300.out
> >   create mode 100755 301
> >   create mode 100644 301.out
> >   create mode 100755 302
> >   create mode 100644 302.out
> >
> > diff --git a/299 b/299
> > new file mode 100755
> > index 0000000..4a7a98c
> > --- /dev/null
> > +++ b/299
> > @@ -0,0 +1,126 @@
> > +#! /bin/bash
> > +# FSQA Test No. 299
> > +#
> > +# Ext4 defragmentation stress test
> > +# Defragment file while other task does direct io
> > +#-----------------------------------------------------------------------
> > +# (c) 2013 Dmitry Monakhov
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > +#
> > +#-----------------------------------------------------------------------
> > +#
> > +# creator
> > +owner=dmonakhov@openvz.org
> > +
> > +seq=`basename $0`
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1	# failure is the default!
> > +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> > +
> > +# get standard environment, filters and checks
> > +. ./common.rc
> > +. ./common.filter
> > +. ./common.defrag
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> > +_supported_fs ext4
> > +_need_to_be_root
> > +_require_scratch
> > +_require_defrag
> > +
> I think you meant to set NUM_JOBS here like you did in [Patch 08/10], 
> something like:
> 
> +NUM_JOBS=$((4*LOAD_FACTOR))
Indeed. I've forget to update patch.
> 
> otherwise you get
> 
> ###	 [not run] /usr/local/bin/fio too old
> 
> > +BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
> > +# We need space for 2 files (test file, and donor one)
> > +# reserve 30% in order to avoid ENOSPC
> > +FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
> > +
> > +cat >$tmp-$seq.fio <<EOF
> > +# Common e4defrag regression tests
> > +[global]
> > +ioengine=ioe_e4defrag
> > +iodepth=1
> > +directory=${SCRATCH_MNT}
> > +filesize=${FILE_SIZE}
> > +size=999G
> > +buffered=0
> > +fadvise_hint=0
> > +group_reporting
> > +
> > +#################################
> > +# Test1
> > +# Defragment file while other task does direct io
> > +
> > +# Continious sequential defrag activity
> > +[defrag-4k]
> > +ioengine=e4defrag
> > +iodepth=1
> > +bs=128k
> > +donorname=test1.def
> > +filename=test1
> > +inplace=0
> > +rw=write
> > +numjobs=${NUM_JOBS}
> > +runtime=30*${TIME_FACTOR}
> > +time_based
> > +
> > +# Verifier
> > +[aio-dio-verifier]
> > +ioengine=libaio
> > +iodepth=128*${LOAD_FACTOR}
> > +numjobs=1
> > +verify=crc32c-intel
> > +verify_fatal=1
> > +verify_dump=1
> > +verify_backlog=1024
> > +verify_async=1
> > +verifysort=1
> > +direct=1
> > +bs=64k
> > +rw=randwrite
> > +filename=test1
> > +runtime=30*${TIME_FACTOR}
> > +time_based
> > +EOF
> > +
> > +_workout()
> > +{
> > +	echo ""
> > +	echo " Start defragment activity "
> git does not like trailing whitespace   ^
> (same for the other 3 tests)
> > +	echo ""
> > +	cat $tmp-$seq.fio >>  $seq.full
> > +	run_check $FIO_PROG $tmp-$seq.fio
> > +}
> > +
> > +_require_fio $tmp-$seq.fio
> > +
> > +_scratch_mkfs  >> $seq.full 2>&1
> > +_scratch_mount
> > +
> > +if ! _workout; then
> > +	umount $SCRATCH_DEV 2>/dev/null
> > +	exit
> > +fi
> > +
> > +if ! _scratch_unmount; then
> > +	echo "failed to umount"
> > +	status=1
> > +	exit
> > +fi
> > +_check_scratch_fs
> > +status=$?
> > +exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
> > diff --git a/299.out b/299.out
> > new file mode 100644
> > index 0000000..c2b0b4c
> > --- /dev/null
> > +++ b/299.out
> > @@ -0,0 +1,4 @@
> > +QA output created by 299
> > +
> > + Start defragment activity
> remove trailing whitespace   ^
> (same for the other 3 tests)
> 
> Let me know if you agree with these changes and I will make them at 
> commit time.
Yes, please do.
> 
> Thanks
> --Rich
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Rich Johnston]

On 03/01/2013 07:30 PM, Dmitry Monakhov wrote:
> On Fri, 1 Mar 2013 13:23:50 -0600, Rich Johnston <rjohnston@sgi.com> wrote:
>> On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:

>>
>> Let me know if you agree with these changes and I will make them at
>> commit time.
> Yes, please do.
>>
>> Thanks
>> --Rich

commit a3d92a6cbdf9e98d8e256974b50b025d1d4575ec
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date:   Wed Feb 20 10:42:14 2013 +0000

     xfstest: add defragmentation stress tests for ext4

     Perform various regression tests for ext4defrag subsystem

      301'th Test1: Defragment file while other task does direct AIO
      302'th Test2: Perform defragmentation on file under buffered AIO
         while third task does direct AIO to donor file
      303'th Test3: Two defrag tasks use common donor file.
      304'th Test4: Stress defragmentation. Several threads perform
         fragmentation at random position use inplace=1 will
         allocate and free blocks inside defrag event improve
         load pressure.

     This tests are considered dangerous because 302'th and 303'th are known
     to trigger OOPS on recent kernels 
see:https://gist.github.com/dmonakhov/4770
294

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Zheng Liu]

On Wed, Feb 20, 2013 at 02:42:14PM +0400, Dmitry Monakhov wrote:
> Perform various regression tests for ext4defrag  subsystem
> 
>  299'th Test1: Defragment file while other task does direct AIO

Hi Dmitry,

I couldn't run this testcase in my sandbox.  I got the following error
message.

FSTYP         -- ext4
PLATFORM      -- Linux/x86_64 lz-desktop 3.8.0
MKFS_OPTIONS  -- /dev/sda2
MOUNT_OPTIONS -- -o acl,user_xattr /dev/sda2 /mnt/sda2

299      [not run] /usr/local/bin/fio too old
Not run: 299
Passed all 0 tests

My fio version is fio-2.0.14-23-g9c63.  It seems that there are some
errors in fio config file.

Regards,
                                                - Zheng

Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Zheng Liu]

On Wed, Feb 20, 2013 at 02:42:14PM +0400, Dmitry Monakhov wrote:
> Perform various regression tests for ext4defrag  subsystem
> 
>  299'th Test1: Defragment file while other task does direct AIO

Hi Dmitry,

I couldn't run this testcase in my sandbox.  I got the following error
message.

FSTYP         -- ext4
PLATFORM      -- Linux/x86_64 lz-desktop 3.8.0
MKFS_OPTIONS  -- /dev/sda2
MOUNT_OPTIONS -- -o acl,user_xattr /dev/sda2 /mnt/sda2

299      [not run] /usr/local/bin/fio too old
Not run: 299
Passed all 0 tests

My fio version is fio-2.0.14-23-g9c63.  It seems that there are some
errors in fio config file.

Regards,
                                                - Zheng

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Zheng Liu]

On Mon, Mar 04, 2013 at 12:43:42AM +0800, Zheng Liu wrote:
> On Wed, Feb 20, 2013 at 02:42:14PM +0400, Dmitry Monakhov wrote:
> > Perform various regression tests for ext4defrag  subsystem
> > 
> >  299'th Test1: Defragment file while other task does direct AIO
> 
> Hi Dmitry,
> 
> I couldn't run this testcase in my sandbox.  I got the following error
> message.
> 
> FSTYP         -- ext4
> PLATFORM      -- Linux/x86_64 lz-desktop 3.8.0
> MKFS_OPTIONS  -- /dev/sda2
> MOUNT_OPTIONS -- -o acl,user_xattr /dev/sda2 /mnt/sda2
> 
> 299      [not run] /usr/local/bin/fio too old
> Not run: 299
> Passed all 0 tests
> 
> My fio version is fio-2.0.14-23-g9c63.  It seems that there are some
> errors in fio config file.

I just notice Rich's reply.  After adding 'NUM_JOBS=$((4*LOAD_FACTOR))',
the testcase can run.  Sorry for the noisy.

Regards,
                                                - Zheng

Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4

[Zheng Liu]

On Mon, Mar 04, 2013 at 12:43:42AM +0800, Zheng Liu wrote:
> On Wed, Feb 20, 2013 at 02:42:14PM +0400, Dmitry Monakhov wrote:
> > Perform various regression tests for ext4defrag  subsystem
> > 
> >  299'th Test1: Defragment file while other task does direct AIO
> 
> Hi Dmitry,
> 
> I couldn't run this testcase in my sandbox.  I got the following error
> message.
> 
> FSTYP         -- ext4
> PLATFORM      -- Linux/x86_64 lz-desktop 3.8.0
> MKFS_OPTIONS  -- /dev/sda2
> MOUNT_OPTIONS -- -o acl,user_xattr /dev/sda2 /mnt/sda2
> 
> 299      [not run] /usr/local/bin/fio too old
> Not run: 299
> Passed all 0 tests
> 
> My fio version is fio-2.0.14-23-g9c63.  It seems that there are some
> errors in fio config file.

I just notice Rich's reply.  After adding 'NUM_JOBS=$((4*LOAD_FACTOR))',
the testcase can run.  Sorry for the noisy.

Regards,
                                                - Zheng

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[PATCH 10/10] xfstests: add disk failure simulation test

[Dmitry Monakhov]

There are many situations where disk may fail for example
1) brutal usb dongle unplug
2) iscsi (or any other netbdev) failure due to network issues
In this situation filesystem which use this blockdevice is
expected to fail(force RO remount, abort, etc) but whole system
should still be operational. In other words:
1) Kernel should not panic
2) Memory should not leak
3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
   for affected filesystem
4) It should be possible to umount broken filesystem

Later when disk becomes available again we expect(only for journaled filesystems):
5) It will be possible to mount filesystem w/o explicit fsck (in order to caught
   issues like https://patchwork.kernel.org/patch/1983981/)
6) Filesystem should be operational
7) After mount/umount has being done all errors should be fixed so fsck should
   not spot any issues.

This test use fault enjection (CONFIG_FAIL_MAKE_REQUEST=y config option )
which force all new IO requests to fail for a given device. Xfs already has
XFS_IOC_GOINGDOWN ioctl which provides similar behaviour, but it is fsspeciffic
and it does it in an easy way because it perform freeze_bdev() before actual
shotdown.

Test run fsstress in background and then force disk failure.
Once disk failed it check that (1)-(4) is true.
Then makes disk available again and check that (5)-(7) is also true

BE CAREFUL!! test known to cause memory corruption for XFS
see: https://gist.github.com/dmonakhov/4953045

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 303           |  186 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 303.out       |    9 +++
 common.config |    1 +
 common.rc     |   13 ++++
 group         |    3 +-
 5 files changed, 211 insertions(+), 1 deletions(-)
 create mode 100755 303
 create mode 100644 303.out

diff --git a/303 b/303
new file mode 100755
index 0000000..01bc5f1
--- /dev/null
+++ b/303
@@ -0,0 +1,186 @@
+#! /bin/bash
+# FSQA Test No. 303
+#
+# Run fsstress and fio(dio/aio and mmap) and simulate disk failure
+# check filesystem consistency at the end.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+_supported_fs ext3 ext4 xfs btrfs reiserfs
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+_require_fail_make_request
+
+# TODO: Function are common enough to be moved to common.blkdev
+SCRATCH_REAL_DEV=`readlink -f $SCRATCH_DEV`
+SCRATCH_BDEV=`basename $SCRATCH_REAL_DEV`
+
+allow_fail_make_request()
+{
+    echo "Allow global fail_make_request feature"
+    echo 100 > $DEBUGFS_MNT/fail_make_request/probability
+    echo 9999999 > $DEBUGFS_MNT/fail_make_request/times
+    echo 0 >  /sys/kernel/debug/fail_make_request/verbose
+}
+
+disallow_fail_make_request()
+{
+    echo "Disallow global fail_make_request feature"
+    echo 0 > $DEBUGFS_MNT/fail_make_request/probability
+    echo 0 > $DEBUGFS_MNT/fail_make_request/times
+}
+
+start_fail_scratch_dev()
+{
+    echo "Force SCRATCH_DEV device failure"
+    echo " echo 1 > /sys/block/$SCRATCH_BDEV/make-it-fail" >> $here/$seq.full
+    echo 1 > /sys/block/$SCRATCH_BDEV/make-it-fail
+
+}
+
+stop_fail_scratch_dev()
+{
+    echo "Make SCRATCH_DEV device operable again"
+    echo " echo 0 > /sys/block/$SCRATCH_BDEV/make-it-fail" >> $here/$seq.full
+    echo 0 > /sys/block/$SCRATCH_BDEV/make-it-fail
+
+}
+
+_cleanup()
+{
+    poweron_scratch_dev
+    disallow_fail_make_request
+}
+trap "_cleanup; exit \$status" 1 2 3 15
+
+RUN_TIME=$((20+10*$TIME_FACTOR))
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test's fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=4k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=9999T
+continue_on_error=write
+ignore_error=EIO,ENOSPC:EIO
+error_dump=0
+
+[stress_dio_aio_activity]
+create_on_open=1
+fallocate=none
+iodepth=128*${LOAD_FACTOR}
+direct=1
+buffered=0
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=40+${RUN_TIME}
+time_based
+
+[stress_mmap_activity]
+ioengine=mmap
+create_on_open=0
+fallocate=1
+fdatasync=40960
+filesize=8M
+size=9999T
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=40+${RUN_TIME}
+time_based
+
+EOF
+
+_require_fio $tmp-$seq.fio
+
+# Disable all sync operations to get higher load
+FSSTRESS_AVOID="$FSSTRESS_AVOID -ffsync=0 -fsync=0 -ffdatasync=0 -f setattr=1"
+
+_workout()
+{
+	out=$SCRATCH_MNT/fsstress.$$
+	args=`_scale_fsstress_args -p 1 -n999999999 -f setattr=0 $FSSTRESS_AVOID -d $out`
+	echo ""
+	echo "Start fsstress.."
+	echo ""
+	echo "fsstress $args" >> $here/$seq.full
+	$FSSTRESS_PROG $args > /dev/null 2>&1 &
+	fs_pid=$!
+	echo "Start fio.."
+	cat $tmp-$seq.fio >>  $seq.full
+	$FIO_PROG $tmp-$seq.fio >> $here/$seq.full 2>&1 &
+	fio_pid=$!
+
+	# Let's it work for awhile, and force device failure
+	sleep $RUN_TIME
+	start_fail_scratch_dev
+	# After device turns in to failed state filesystem may yet not know about
+	# that so buffered write(2) may succeed, but any integrity operations
+	# such as (sync, fsync, fdatasync, direct-io) should fail.
+	dd if=/dev/zero of=$SCRATCH_MNT/touch_failed_filesystem count=1 bs=4k conv=fsync \
+	    >> $here/$seq.full 2>&1 && \
+	    _fail "failed: still able to perform integrity fsync on $SCRATCH_MNT"
+
+	kill $fs_pid
+	wait $fs_pid
+	wait $fio_pid
+
+	# We expect that broken FS still can be umounted
+	run_check umount $SCRATCH_DEV
+	# Once filesystem was umounted no one is able to write to block device
+	# It is now safe to bring device back to normal state
+	stop_fail_scratch_dev
+
+	# In order to check that filesystem is able to recover journal on mount(2)
+	# perform mount/umount, after that all errors should be fixed
+	run_check _scratch_mount
+	run_check _scratch_unmount
+	_check_scratch_fs
+}
+
+# real QA test starts here
+
+_scratch_mkfs >> $here/$seq.full 2>&1 || _fail "mkfs failed"
+_scratch_mount || _fail "mount failed"
+allow_fail_make_request
+_workout
+status=$?
+disallow_fail_make_request
+exit
diff --git a/303.out b/303.out
new file mode 100644
index 0000000..a936471
--- /dev/null
+++ b/303.out
@@ -0,0 +1,9 @@
+QA output created by 303
+Allow global fail_make_request feature
+
+Start fsstress..
+
+Start fio..
+Force SCRATCH_DEV device failure
+Make SCRATCH_DEV device operable again
+Disallow global fail_make_request feature
diff --git a/common.config b/common.config
index 8efc7f8..8643c0b 100644
--- a/common.config
+++ b/common.config
@@ -75,6 +75,7 @@ export BENCH_PASSES=${BENCH_PASSES:=5}
 export XFS_MKFS_OPTIONS=${XFS_MKFS_OPTIONS:=-bsize=4096}
 export TIME_FACTOR=${TIME_FACTOR:=1}
 export LOAD_FACTOR=${LOAD_FACTOR:=1}
+export DEBUGFS_MNT=${DEBUGFS_MNT:="/sys/kernel/debug"}
 
 export PWD=`pwd`
 #export DEBUG=${DEBUG:=...} # arbitrary CFLAGS really.
diff --git a/common.rc b/common.rc
index e7ae9bf..b795c22 100644
--- a/common.rc
+++ b/common.rc
@@ -1049,6 +1049,19 @@ _require_sparse_files()
     esac
 }
 
+_require_debugfs()
+{
+    #boot_params always present in debugfs
+    [ -d "$DEBUGFS_MNT/boot_params" ] || _notrun "Debugfs not mounted"
+}
+
+_require_fail_make_request()
+{
+    [ -f "$DEBUGFS_MNT/fail_make_request/probability" ] \
+	|| _notrun "$DEBUGFS_MNT/fail_make_request \
+ not found. Seems that CONFIG_FAIL_MAKE_REQUEST kernel config option not enabled"
+}
+
 # check that a FS on a device is mounted
 # if so, return mount point
 #
diff --git a/group b/group
index 2fe406b..20b7a9c 100644
--- a/group
+++ b/group
@@ -425,4 +425,5 @@ stress
 299 aio dangerous ioctl rw stress
 300 aio dangerous ioctl rw stress
 301 aio dangerous ioctl rw stress
-302 aio dangerous ioctl rw stress
\ No newline at end of file
+302 aio dangerous ioctl rw stress
+303 aio dangerous enospc rw stress
\ No newline at end of file
-- 
1.7.1

[PATCH 10/10] xfstests: add disk failure simulation test

[Dmitry Monakhov]

There are many situations where disk may fail for example
1) brutal usb dongle unplug
2) iscsi (or any other netbdev) failure due to network issues
In this situation filesystem which use this blockdevice is
expected to fail(force RO remount, abort, etc) but whole system
should still be operational. In other words:
1) Kernel should not panic
2) Memory should not leak
3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
   for affected filesystem
4) It should be possible to umount broken filesystem

Later when disk becomes available again we expect(only for journaled filesystems):
5) It will be possible to mount filesystem w/o explicit fsck (in order to caught
   issues like https://patchwork.kernel.org/patch/1983981/)
6) Filesystem should be operational
7) After mount/umount has being done all errors should be fixed so fsck should
   not spot any issues.

This test use fault enjection (CONFIG_FAIL_MAKE_REQUEST=y config option )
which force all new IO requests to fail for a given device. Xfs already has
XFS_IOC_GOINGDOWN ioctl which provides similar behaviour, but it is fsspeciffic
and it does it in an easy way because it perform freeze_bdev() before actual
shotdown.

Test run fsstress in background and then force disk failure.
Once disk failed it check that (1)-(4) is true.
Then makes disk available again and check that (5)-(7) is also true

BE CAREFUL!! test known to cause memory corruption for XFS
see: https://gist.github.com/dmonakhov/4953045

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 303           |  186 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 303.out       |    9 +++
 common.config |    1 +
 common.rc     |   13 ++++
 group         |    3 +-
 5 files changed, 211 insertions(+), 1 deletions(-)
 create mode 100755 303
 create mode 100644 303.out

diff --git a/303 b/303
new file mode 100755
index 0000000..01bc5f1
--- /dev/null
+++ b/303
@@ -0,0 +1,186 @@
+#! /bin/bash
+# FSQA Test No. 303
+#
+# Run fsstress and fio(dio/aio and mmap) and simulate disk failure
+# check filesystem consistency at the end.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+_supported_fs ext3 ext4 xfs btrfs reiserfs
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+_require_fail_make_request
+
+# TODO: Function are common enough to be moved to common.blkdev
+SCRATCH_REAL_DEV=`readlink -f $SCRATCH_DEV`
+SCRATCH_BDEV=`basename $SCRATCH_REAL_DEV`
+
+allow_fail_make_request()
+{
+    echo "Allow global fail_make_request feature"
+    echo 100 > $DEBUGFS_MNT/fail_make_request/probability
+    echo 9999999 > $DEBUGFS_MNT/fail_make_request/times
+    echo 0 >  /sys/kernel/debug/fail_make_request/verbose
+}
+
+disallow_fail_make_request()
+{
+    echo "Disallow global fail_make_request feature"
+    echo 0 > $DEBUGFS_MNT/fail_make_request/probability
+    echo 0 > $DEBUGFS_MNT/fail_make_request/times
+}
+
+start_fail_scratch_dev()
+{
+    echo "Force SCRATCH_DEV device failure"
+    echo " echo 1 > /sys/block/$SCRATCH_BDEV/make-it-fail" >> $here/$seq.full
+    echo 1 > /sys/block/$SCRATCH_BDEV/make-it-fail
+
+}
+
+stop_fail_scratch_dev()
+{
+    echo "Make SCRATCH_DEV device operable again"
+    echo " echo 0 > /sys/block/$SCRATCH_BDEV/make-it-fail" >> $here/$seq.full
+    echo 0 > /sys/block/$SCRATCH_BDEV/make-it-fail
+
+}
+
+_cleanup()
+{
+    poweron_scratch_dev
+    disallow_fail_make_request
+}
+trap "_cleanup; exit \$status" 1 2 3 15
+
+RUN_TIME=$((20+10*$TIME_FACTOR))
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test's fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=4k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=9999T
+continue_on_error=write
+ignore_error=EIO,ENOSPC:EIO
+error_dump=0
+
+[stress_dio_aio_activity]
+create_on_open=1
+fallocate=none
+iodepth=128*${LOAD_FACTOR}
+direct=1
+buffered=0
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=40+${RUN_TIME}
+time_based
+
+[stress_mmap_activity]
+ioengine=mmap
+create_on_open=0
+fallocate=1
+fdatasync=40960
+filesize=8M
+size=9999T
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=40+${RUN_TIME}
+time_based
+
+EOF
+
+_require_fio $tmp-$seq.fio
+
+# Disable all sync operations to get higher load
+FSSTRESS_AVOID="$FSSTRESS_AVOID -ffsync=0 -fsync=0 -ffdatasync=0 -f setattr=1"
+
+_workout()
+{
+	out=$SCRATCH_MNT/fsstress.$$
+	args=`_scale_fsstress_args -p 1 -n999999999 -f setattr=0 $FSSTRESS_AVOID -d $out`
+	echo ""
+	echo "Start fsstress.."
+	echo ""
+	echo "fsstress $args" >> $here/$seq.full
+	$FSSTRESS_PROG $args > /dev/null 2>&1 &
+	fs_pid=$!
+	echo "Start fio.."
+	cat $tmp-$seq.fio >>  $seq.full
+	$FIO_PROG $tmp-$seq.fio >> $here/$seq.full 2>&1 &
+	fio_pid=$!
+
+	# Let's it work for awhile, and force device failure
+	sleep $RUN_TIME
+	start_fail_scratch_dev
+	# After device turns in to failed state filesystem may yet not know about
+	# that so buffered write(2) may succeed, but any integrity operations
+	# such as (sync, fsync, fdatasync, direct-io) should fail.
+	dd if=/dev/zero of=$SCRATCH_MNT/touch_failed_filesystem count=1 bs=4k conv=fsync \
+	    >> $here/$seq.full 2>&1 && \
+	    _fail "failed: still able to perform integrity fsync on $SCRATCH_MNT"
+
+	kill $fs_pid
+	wait $fs_pid
+	wait $fio_pid
+
+	# We expect that broken FS still can be umounted
+	run_check umount $SCRATCH_DEV
+	# Once filesystem was umounted no one is able to write to block device
+	# It is now safe to bring device back to normal state
+	stop_fail_scratch_dev
+
+	# In order to check that filesystem is able to recover journal on mount(2)
+	# perform mount/umount, after that all errors should be fixed
+	run_check _scratch_mount
+	run_check _scratch_unmount
+	_check_scratch_fs
+}
+
+# real QA test starts here
+
+_scratch_mkfs >> $here/$seq.full 2>&1 || _fail "mkfs failed"
+_scratch_mount || _fail "mount failed"
+allow_fail_make_request
+_workout
+status=$?
+disallow_fail_make_request
+exit
diff --git a/303.out b/303.out
new file mode 100644
index 0000000..a936471
--- /dev/null
+++ b/303.out
@@ -0,0 +1,9 @@
+QA output created by 303
+Allow global fail_make_request feature
+
+Start fsstress..
+
+Start fio..
+Force SCRATCH_DEV device failure
+Make SCRATCH_DEV device operable again
+Disallow global fail_make_request feature
diff --git a/common.config b/common.config
index 8efc7f8..8643c0b 100644
--- a/common.config
+++ b/common.config
@@ -75,6 +75,7 @@ export BENCH_PASSES=${BENCH_PASSES:=5}
 export XFS_MKFS_OPTIONS=${XFS_MKFS_OPTIONS:=-bsize=4096}
 export TIME_FACTOR=${TIME_FACTOR:=1}
 export LOAD_FACTOR=${LOAD_FACTOR:=1}
+export DEBUGFS_MNT=${DEBUGFS_MNT:="/sys/kernel/debug"}
 
 export PWD=`pwd`
 #export DEBUG=${DEBUG:=...} # arbitrary CFLAGS really.
diff --git a/common.rc b/common.rc
index e7ae9bf..b795c22 100644
--- a/common.rc
+++ b/common.rc
@@ -1049,6 +1049,19 @@ _require_sparse_files()
     esac
 }
 
+_require_debugfs()
+{
+    #boot_params always present in debugfs
+    [ -d "$DEBUGFS_MNT/boot_params" ] || _notrun "Debugfs not mounted"
+}
+
+_require_fail_make_request()
+{
+    [ -f "$DEBUGFS_MNT/fail_make_request/probability" ] \
+	|| _notrun "$DEBUGFS_MNT/fail_make_request \
+ not found. Seems that CONFIG_FAIL_MAKE_REQUEST kernel config option not enabled"
+}
+
 # check that a FS on a device is mounted
 # if so, return mount point
 #
diff --git a/group b/group
index 2fe406b..20b7a9c 100644
--- a/group
+++ b/group
@@ -425,4 +425,5 @@ stress
 299 aio dangerous ioctl rw stress
 300 aio dangerous ioctl rw stress
 301 aio dangerous ioctl rw stress
-302 aio dangerous ioctl rw stress
\ No newline at end of file
+302 aio dangerous ioctl rw stress
+303 aio dangerous enospc rw stress
\ No newline at end of file
-- 
1.7.1

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 10/10] xfstests: add disk failure simulation test

[Rich Johnston]

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> There are many situations where disk may fail for example
> 1) brutal usb dongle unplug
> 2) iscsi (or any other netbdev) failure due to network issues
> In this situation filesystem which use this blockdevice is
> expected to fail(force RO remount, abort, etc) but whole system
> should still be operational. In other words:
> 1) Kernel should not panic
> 2) Memory should not leak
> 3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
>     for affected filesystem
> 4) It should be possible to umount broken filesystem
>
> Later when disk becomes available again we expect(only for journaled filesystems):
> 5) It will be possible to mount filesystem w/o explicit fsck (in order to caught

typo                                     s/caught/catch/g

>     issues like https://patchwork.kernel.org/patch/1983981/)
> 6) Filesystem should be operational
> 7) After mount/umount has being done all errors should be fixed so fsck should
>     not spot any issues.
>
> This test use fault enjection (CONFIG_FAIL_MAKE_REQUEST=y config option )
   May want to mention all the kernel config options required.
i.e. CONFIG_FAULT_INJECTION=y ... are there others?
CONFIG_FAULT_INJECTION_DEBUG_FS=y ???

> which force all new IO requests to fail for a given device. Xfs already has
   to force

> XFS_IOC_GOINGDOWN ioctl which provides similar behaviour, but it is fs speciffic

typos s/behaviour/behavior/g  s/speciffic/specific
                                         > and it does it in an easy way 
because it perform freeze_bdev() before actual
> shotdown.
typo s/shotdown/shutdown/g

>
> Test run fsstress in background and then force disk failure.
> Once disk failed it check that (1)-(4) is true.
   Once the disk fails, check that (1)-(4) are true.

> Then makes disk available again and check that (5)-(7) is also true
        make the disk ...                                 are
>
> BE CAREFUL!! test known to cause memory corruption for XFS
> see: https://gist.github.com/dmonakhov/4953045
>




_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 10/10] xfstests: add disk failure simulation test

[Dmitry Monakhov]

On Fri, 1 Mar 2013 14:11:15 -0600, Rich Johnston <rjohnston@sgi.com> wrote:
> On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> > There are many situations where disk may fail for example
> > 1) brutal usb dongle unplug
> > 2) iscsi (or any other netbdev) failure due to network issues
> > In this situation filesystem which use this blockdevice is
> > expected to fail(force RO remount, abort, etc) but whole system
> > should still be operational. In other words:
> > 1) Kernel should not panic
> > 2) Memory should not leak
> > 3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
> >     for affected filesystem
> > 4) It should be possible to umount broken filesystem
> >
> > Later when disk becomes available again we expect(only for journaled filesystems):
> > 5) It will be possible to mount filesystem w/o explicit fsck (in order to caught
> 
> typo                                     s/caught/catch/g
> 
> >     issues like https://patchwork.kernel.org/patch/1983981/)
> > 6) Filesystem should be operational
> > 7) After mount/umount has being done all errors should be fixed so fsck should
> >     not spot any issues.
> >
> > This test use fault enjection (CONFIG_FAIL_MAKE_REQUEST=y config option )
>    May want to mention all the kernel config options required.
> i.e. CONFIG_FAULT_INJECTION=y ... are there others?
> CONFIG_FAULT_INJECTION_DEBUG_FS=y ???
Yes, all three options are required. 
> 
> > which force all new IO requests to fail for a given device. Xfs already has
>    to force
> 
> > XFS_IOC_GOINGDOWN ioctl which provides similar behaviour, but it is fs speciffic
> 
> typos s/behaviour/behavior/g  s/speciffic/specific
>                                          > and it does it in an easy way 
> because it perform freeze_bdev() before actual
> > shotdown.
> typo s/shotdown/shutdown/g
Agree with your diagnosis. My gramma is bad and I've forget to call spell check
before submission. Should I resend this one or you fix it manually
on commit time?
> 
> >
> > Test run fsstress in background and then force disk failure.
> > Once disk failed it check that (1)-(4) is true.
>    Once the disk fails, check that (1)-(4) are true.
> 
> > Then makes disk available again and check that (5)-(7) is also true
>         make the disk ...                                 are
> >
> > BE CAREFUL!! test known to cause memory corruption for XFS
> > see: https://gist.github.com/dmonakhov/4953045
> >
> 
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 10/10] xfstests: add disk failure simulation test

[Rich Johnston]

On 03/01/2013 07:49 PM, Dmitry Monakhov wrote:
> On Fri, 1 Mar 2013 14:11:15 -0600, Rich Johnston <rjohnston@sgi.com> wrote:
>> On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
>>> There are many situations where disk may fail for example
>>> 1) brutal usb dongle unplug
>>> 2) iscsi (or any other netbdev) failure due to network issues
>>> In this situation filesystem which use this blockdevice is
>>> expected to fail(force RO remount, abort, etc) but whole system
>>> should still be operational. In other words:
>>> 1) Kernel should not panic
>>> 2) Memory should not leak
>>> 3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
>>>      for affected filesystem
>>> 4) It should be possible to umount broken filesystem
>>>
>>> Later when disk becomes available again we expect(only for journaled filesystems):
>>> 5) It will be possible to mount filesystem w/o explicit fsck (in order to caught
>>
>> typo                                     s/caught/catch/g
>>
>>>      issues like https://patchwork.kernel.org/patch/1983981/)
>>> 6) Filesystem should be operational
>>> 7) After mount/umount has being done all errors should be fixed so fsck should
>>>      not spot any issues.
>>>
>>> This test use fault enjection (CONFIG_FAIL_MAKE_REQUEST=y config option )
>>     May want to mention all the kernel config options required.
>> i.e. CONFIG_FAULT_INJECTION=y ... are there others?
>> CONFIG_FAULT_INJECTION_DEBUG_FS=y ???
> Yes, all three options are required.
>>
>>> which force all new IO requests to fail for a given device. Xfs already has
>>     to force
>>
>>> XFS_IOC_GOINGDOWN ioctl which provides similar behaviour, but it is fs speciffic
>>
>> typos s/behaviour/behavior/g  s/speciffic/specific
>>                                           > and it does it in an easy way
>> because it perform freeze_bdev() before actual
>>> shotdown.
>> typo s/shotdown/shutdown/g
> Agree with your diagnosis. My gramma is bad and I've forget to call spell check
> before submission. Should I resend this one or you fix it manually
> on commit time?

No worries, I'm sure your English is much better than any of my attempts 
to write in your native tongue. ;)
No need to resend, glad to take care of these minor changes at commit time.

commit 02e57e1e3a42856dca9061ff943ba72fa7be8469
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date:   Wed Feb 20 10:42:15 2013 +0000

     xfstests: add disk failure simulation test

     There are many situations where disk may fail for example
     1) brutal usb dongle unplug
     2) iscsi (or any other netbdev) failure due to network issues
     In this situation filesystem which use this blockdevice is
     expected to fail(force RO remount, abort, etc) but whole system
     should still be operational. In other words:
     1) Kernel should not panic
     2) Memory should not leak
     3) Data integrity operations (sync,fsync,fdatasync, directio) 
should fail
        for affected filesystem
     4) It should be possible to umount broken filesystem

     Later when disk becomes available again we expect(only for 
journaled filesystems):
     5) It will be possible to mount filesystem w/o explicit fsck (in 
order to catch
        issues like https://patchwork.kernel.org/patch/1983981/)
     6) Filesystem should be operational
     7) After mount/umount has being done all errors should be fixed so 
fsck should
        not spot any issues.

     This test use fault injection (CONFIG_FAULT_INJECTION=y,
     CONFIG_FAIL_MAKE_REQUEST=y and CONFIG_FAULT_INJECTION_DEBUG_FS=y config
     options) to force all new IO requests to fail for a given device. Xfs
     already has XFS_IOC_GOINGDOWN ioctl which provides similar 
behavior, but it
     is fs specific and it does it in an easy way because it performs 
freeze_bdev()
     before actual shutdown.

     Test run fsstress in background and then force disk failure.
     Once disk failed it check that (1)-(4) is true.
     Then makes disk available again and check that (5)-(7) is also true

     BE CAREFUL!! test known to cause memory corruption for XFS
     see: https://gist.github.com/dmonakhov/4953045



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 00/10] xfstests: Stress tests improments v5

[Theodore Ts'o]

Hi, what's the status of this patch series?  It looks like Dmitry has
responded to all of Dave's concerns --- is that correct?

Is there anything else we need to do before these patches get merged
into the xfstests git tree?

Thanks!!

						- Ted

Re: [PATCH 00/10] xfstests: Stress tests improments v5

[Theodore Ts'o]

Hi, what's the status of this patch series?  It looks like Dmitry has
responded to all of Dave's concerns --- is that correct?

Is there anything else we need to do before these patches get merged
into the xfstests git tree?

Thanks!!

						- Ted

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 00/10] xfstests: Stress tests improments v5

[Ben Myers]

Hey Ted,

On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> Hi, what's the status of this patch series?  It looks like Dmitry has
> responded to all of Dave's concerns --- is that correct?
> 
> Is there anything else we need to do before these patches get merged
> into the xfstests git tree?

I think we just need a review to see if Dave's concerns have been addressed and
then we're ready to go.

Regards,
	Ben

Re: [PATCH 00/10] xfstests: Stress tests improments v5

[Ben Myers]

Hey Ted,

On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> Hi, what's the status of this patch series?  It looks like Dmitry has
> responded to all of Dave's concerns --- is that correct?
> 
> Is there anything else we need to do before these patches get merged
> into the xfstests git tree?

I think we just need a review to see if Dave's concerns have been addressed and
then we're ready to go.

Regards,
	Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 00/10] xfstests: Stress tests improments v5

[Dave Chinner]

On Mon, Feb 25, 2013 at 03:28:11PM -0600, Ben Myers wrote:
> Hey Ted,
> 
> On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> > Hi, what's the status of this patch series?  It looks like Dmitry has
> > responded to all of Dave's concerns --- is that correct?
> > 
> > Is there anything else we need to do before these patches get merged
> > into the xfstests git tree?
> 
> I think we just need a review to see if Dave's concerns have been addressed and
> then we're ready to go.

That doesn't stop anyone else from reviewing it. I've got my hands
full right now, and doing stuff like reviewing patches that
nobody else seems to want to look at is something that I'm doing
when I have a little spare time.

I've already reviewed al the patches and the changes have been made,
so all my final review is going to be is validating the patches
apply and work as advertised, so anyone really can do that. I
couldn't do that last time around because the patches didn't apply
to my tree. Since then I haven't had a slot available to go back and
look at it.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH 00/10] xfstests: Stress tests improments v5

[Dave Chinner]

On Mon, Feb 25, 2013 at 03:28:11PM -0600, Ben Myers wrote:
> Hey Ted,
> 
> On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> > Hi, what's the status of this patch series?  It looks like Dmitry has
> > responded to all of Dave's concerns --- is that correct?
> > 
> > Is there anything else we need to do before these patches get merged
> > into the xfstests git tree?
> 
> I think we just need a review to see if Dave's concerns have been addressed and
> then we're ready to go.

That doesn't stop anyone else from reviewing it. I've got my hands
full right now, and doing stuff like reviewing patches that
nobody else seems to want to look at is something that I'm doing
when I have a little spare time.

I've already reviewed al the patches and the changes have been made,
so all my final review is going to be is validating the patches
apply and work as advertised, so anyone really can do that. I
couldn't do that last time around because the patches didn't apply
to my tree. Since then I haven't had a slot available to go back and
look at it.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 00/10] xfstests: Stress tests improments v5

[Ben Myers]

Hi,

On Tue, Feb 26, 2013 at 09:09:41AM +1100, Dave Chinner wrote:
> On Mon, Feb 25, 2013 at 03:28:11PM -0600, Ben Myers wrote:
> > On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> > > Hi, what's the status of this patch series?  It looks like Dmitry has
> > > responded to all of Dave's concerns --- is that correct?
> > > 
> > > Is there anything else we need to do before these patches get merged
> > > into the xfstests git tree?
> > 
> > I think we just need a review to see if Dave's concerns have been addressed and
> > then we're ready to go.
> 
> That doesn't stop anyone else from reviewing it.

Yes.  Anyone is welcome to extend a Reviewed-by.  ;)

-Ben

Re: [PATCH 00/10] xfstests: Stress tests improments v5

[Ben Myers]

Hi,

On Tue, Feb 26, 2013 at 09:09:41AM +1100, Dave Chinner wrote:
> On Mon, Feb 25, 2013 at 03:28:11PM -0600, Ben Myers wrote:
> > On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> > > Hi, what's the status of this patch series?  It looks like Dmitry has
> > > responded to all of Dave's concerns --- is that correct?
> > > 
> > > Is there anything else we need to do before these patches get merged
> > > into the xfstests git tree?
> > 
> > I think we just need a review to see if Dave's concerns have been addressed and
> > then we're ready to go.
> 
> That doesn't stop anyone else from reviewing it.

Yes.  Anyone is welcome to extend a Reviewed-by.  ;)

-Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH 00/10] xfstests: Stress tests improments v5

[Rich Johnston]

Committed patches 1-3 and 5-8.

Waiting for responses back on patches 4, 9, and 10.

Thanks
--Rich

On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> 1   add fio requirement V2
> 2   add configurable load factors
> 3   hardcode fops for determinable fsstests runs
> 4   fsstress add replace file operation
> 5   allow fsstress to use load factor where appropriate
> 6   move run_check to common.rc
> 7   add  fallocate/truncate vs AIO/DIO stress test
> 8   add  fallocate/punch_hole vs AIO/DIO stress test
> 9   add defragmentation stress tests for ext4
> 10  add disk failure simulation test
>


commit 892125a53aac8e82dec99fe4e824c39513d4a17a
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date:   Wed Feb 20 10:42:13 2013 +0000

     xfstest: add fallocate/punch_hole vs AIO/DIO stress test

     Run random AIO/DIO activity (fio's job:direct_aio_raicer)
     random fallocate activity(fio's job:falloc_raicer)
     and random punch_hole activity(punch_hole_raicer) on a common
     file in parallel. If a race exists, old dio request may rewrite
     punched block after it was allocated to another file, we will
     catch that by verifier fio's job: "aio-dio-verifier".

     Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
     Reviewed-by: Rich Johnston <rjohnston@sgi.com>
     Signed-off-by: Rich Johnston <rjohnston@sgi.com>

commit 0f88dc26abf55cee39ede490da08ed0d2960cdb2
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date:   Wed Feb 20 10:42:12 2013 +0000

     xfstest: add fallocate/truncate vs AIO/DIO stress test

     Run DIO, fallocate and truncate threads on a common file in parallel.
     If a race exists, the old dio request may rewrite blocks after it was
     allocated to another file, we will catch that by verifying blocks 
content.

     this patch known to catch deadlock for ext4
     http://lists.openwall.net/linux-ext4/2012/09/06/3

     Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
     Reviewed-by: Rich Johnston <rjohnston@sgi.com>
     Signed-off-by: Rich Johnston <rjohnston@sgi.com>

commit bb949015a8d3fcf4a5c105b1edd74f27b848a806
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date:   Wed Feb 20 10:42:11 2013 +0000

     xfstest: move run_check to common.rc

     Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
     Reviewed-by: Rich Johnston <rjohnston@sgi.com>
     Signed-off-by: Rich Johnston <rjohnston@sgi.com>

     Move run_check to common.rc.
commit b84aade2f49f309c44e6c0b029586c9dcb7b1c22
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date:   Wed Feb 20 10:42:10 2013 +0000

     xfstest: allow fsstress to use load factor where appropriate

     1) Add _scale_fsstress_args function which transform arguments 
according
        to load factors
     2) Let all non deterministic fsstress tests to use scaled arguments

     Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
     Reviewed-by: Rich Johnston <rjohnston@sgi.com>
     Signed-off-by: Rich Johnston <rjohnston@sgi.com>

commit d14981fa536ac0106dbe9b3d5088a5a50be49f55
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date:   Wed Feb 20 10:42:08 2013 +0000

     xfstests: hardcode fops for determinable fsstests runs

     106,107 and 117 are frozen tests which use known seed, it is
     reasonable to explicitly hardcode file operations in order to avoid
     implicit changes caused by future changes in fsstress.

     NOTE: options generated like follows: fsstress -S c $ORIG_ARGS

     Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
     Reviewed-by: Rich Johnston <rjohnston@sgi.com>
     Signed-off-by: Rich Johnston <rjohnston@sgi.com>

commit 66a98478e3cffecfbd62d3a124b626ddda447c57
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date:   Wed Feb 20 10:42:07 2013 +0000

     xfstest: add configurable load factors

     Most stress test has probable behavior, the longer test run the
     larger corner cases will be cover. It is reasonable to allow
     user to provide some sort of system load factor.
     This patch introduce two global variables
     LOAD_FACTOR: Usually means factor number of running tasks
     TIME_FACTOR: Usually means factor of run time, or number of operations
     If not specified both variables defined to 1, so original behavior
     preserved.
     TODO: Change all stress tests to use this variables

     Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
     Reviewed-by: Rich Johnston <rjohnston@sgi.com>
     Signed-off-by: Rich Johnston <rjohnston@sgi.com>

commit b3db6021fc2df4e99c9e098612446f4c86e70766
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date:   Wed Feb 20 10:42:06 2013 +0000

     xfstests: add fio requirement V2

     FIO is very flexible io generator, I would call it IO swiss knife.
     Currently we have tons of hardcoded application which reproduces
     some predefined scenario. This approach has obvious disadvantages
     1) Lack of flexibility: one written it is hard to modify it in future
     2) Code base is large, many routines written again and again

     At the same time add new fio based test, just add simple INI file.
     This greatly simplifies code review. I do believe that some day we will
     replace most of hardcoded io binaries with fio.

     One who is planning to run $FIO_PROG should first check that system
     contains appropriate version which is able to handle jobfile
     for example:  _require_fio 286-job.fio

     Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
     Reviewed-by: Rich Johnston <rjohnston@sgi.com>
     Signed-off-by: Rich Johnston <rjohnston@sgi.com>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs