* [PATCH 00/10] xfstests: Stress tests improments v5
@ 2013-02-20 10:42 Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
` (11 more replies)
0 siblings, 12 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
1 add fio requirement V2
2 add configurable load factors
3 hardcode fops for determinable fsstests runs
4 fsstress add replace file operation
5 allow fsstress to use load factor where appropriate
6 move run_check to common.rc
7 add fallocate/truncate vs AIO/DIO stress test
8 add fallocate/punch_hole vs AIO/DIO stress test
9 add defragmentation stress tests for ext4
10 add disk failure simulation test
Known issues which can be triggered this test case:
287'th over EXT4 result in https://gist.github.com/dmonakhov/4994589
298'th (8'th patch) over XFS result in https://gist.github.com/dmonakhov/4994465
300'th and 301'th over EXT4 result in https://gist.github.com/dmonakhov/4770294
303'th over XFS result in https://gist.github.com/dmonakhov/4951456
Changes since V4
Rebase on top of current dev tree.
Changes since V2
Mostly code style cleanups accodring to Dave's comments
- add _scale_fsstress_args function
- move run_check function to common.rc
- Add more comments
- split combined e4defrag testcase in to disicated tescases
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* [PATCH 01/10] xfstests: add fio requirement V2
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
@ 2013-02-20 10:42 ` Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
` (10 subsequent siblings)
11 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-ext4, linux-fsdevel, dchinner, Dmitry Monakhov
FIO is very flexible io generator, I would call it IO swiss knife.
Currently we have tonns of hardcoded application which reproduces
some predefined scenario. This approach has obvious dissadvantages
1) Lack of flexability: one written it is hard to modify it in future
2) Code base is large, many routines written again and again
At the same time add new fio based tast is just add simle INI file.
This greatly simplify code review. I do beleve that some day we will
replace most of hardcoded io binaries with fio.
One who is planning to run $FIO_PROG should first check that system
contains appropriate version which is able to handle jobfile
for example: _require_fio 286-job.fio
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
common.config | 1 +
common.rc | 15 +++++++++++++++
2 files changed, 16 insertions(+), 0 deletions(-)
diff --git a/common.config b/common.config
index 827c718..7681812 100644
--- a/common.config
+++ b/common.config
@@ -159,6 +159,7 @@ export KILLALL_PROG="`set_prog_path killall`"
export INDENT_PROG="`set_prog_path indent`"
export XFS_COPY_PROG="`set_prog_path xfs_copy`"
export FSTRIM_PROG="`set_prog_path fstrim`"
+export FIO_PROG="`set_prog_path fio`"
# Generate a comparable xfsprogs version number in the form of
# major * 10000 + minor * 100 + release
diff --git a/common.rc b/common.rc
index 2e8581e..14de47b 100644
--- a/common.rc
+++ b/common.rc
@@ -1789,6 +1789,21 @@ _require_freeze()
[ $result -eq 0 ] || _notrun "$FSTYP does not support freezing"
}
+# Check that fio is present, and it is able to execute given jobfile
+_require_fio()
+{
+ job=$1
+
+ _require_command $FIO_PROG
+ if [ -z "$1" ]; then
+ return 1;
+ fi
+
+ $FIO_PROG --warnings-fatal --showcmd $job >/dev/null 2>&1
+ [ $? -eq 0 ] || _notrun "$FIO_PROG too old"
+}
+
+
# arg 1 is dev to remove and is output of the below eg.
# ls -l /sys/class/block/sdd | rev | cut -d "/" -f 3 | rev
_devmgt_remove()
--
1.7.1
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 01/10] xfstests: add fio requirement V2
@ 2013-02-20 10:42 ` Dmitry Monakhov
0 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
FIO is very flexible io generator, I would call it IO swiss knife.
Currently we have tonns of hardcoded application which reproduces
some predefined scenario. This approach has obvious dissadvantages
1) Lack of flexability: one written it is hard to modify it in future
2) Code base is large, many routines written again and again
At the same time add new fio based tast is just add simle INI file.
This greatly simplify code review. I do beleve that some day we will
replace most of hardcoded io binaries with fio.
One who is planning to run $FIO_PROG should first check that system
contains appropriate version which is able to handle jobfile
for example: _require_fio 286-job.fio
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
common.config | 1 +
common.rc | 15 +++++++++++++++
2 files changed, 16 insertions(+), 0 deletions(-)
diff --git a/common.config b/common.config
index 827c718..7681812 100644
--- a/common.config
+++ b/common.config
@@ -159,6 +159,7 @@ export KILLALL_PROG="`set_prog_path killall`"
export INDENT_PROG="`set_prog_path indent`"
export XFS_COPY_PROG="`set_prog_path xfs_copy`"
export FSTRIM_PROG="`set_prog_path fstrim`"
+export FIO_PROG="`set_prog_path fio`"
# Generate a comparable xfsprogs version number in the form of
# major * 10000 + minor * 100 + release
diff --git a/common.rc b/common.rc
index 2e8581e..14de47b 100644
--- a/common.rc
+++ b/common.rc
@@ -1789,6 +1789,21 @@ _require_freeze()
[ $result -eq 0 ] || _notrun "$FSTYP does not support freezing"
}
+# Check that fio is present, and it is able to execute given jobfile
+_require_fio()
+{
+ job=$1
+
+ _require_command $FIO_PROG
+ if [ -z "$1" ]; then
+ return 1;
+ fi
+
+ $FIO_PROG --warnings-fatal --showcmd $job >/dev/null 2>&1
+ [ $? -eq 0 ] || _notrun "$FIO_PROG too old"
+}
+
+
# arg 1 is dev to remove and is output of the below eg.
# ls -l /sys/class/block/sdd | rev | cut -d "/" -f 3 | rev
_devmgt_remove()
--
1.7.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 02/10] xfstest: add configurable load factors
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
@ 2013-02-20 10:42 ` Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
` (10 subsequent siblings)
11 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-ext4, linux-fsdevel, dchinner, Dmitry Monakhov
Most stress test has probable behaviour, the longer test run the
larger corner cases will be cover. It is reasonable to allow
user to provide some sort of system load factor.
This patch introduce two global variables
LOAD_FACTOR: Usually means factor number of running tasks
TIME_FACTOR: Usually means factor of run time, or number of operations
If not speficied both variables defined to 1, so original behaviour
preserved.
TODO: Change all stress tests to use this variables
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
common.config | 2 ++
group | 7 ++++++-
2 files changed, 8 insertions(+), 1 deletions(-)
diff --git a/common.config b/common.config
index 7681812..8efc7f8 100644
--- a/common.config
+++ b/common.config
@@ -73,6 +73,8 @@ export HOST_OPTIONS=${HOST_OPTIONS:=local.config}
export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"}
export BENCH_PASSES=${BENCH_PASSES:=5}
export XFS_MKFS_OPTIONS=${XFS_MKFS_OPTIONS:=-bsize=4096}
+export TIME_FACTOR=${TIME_FACTOR:=1}
+export LOAD_FACTOR=${LOAD_FACTOR:=1}
export PWD=`pwd`
#export DEBUG=${DEBUG:=...} # arbitrary CFLAGS really.
diff --git a/group b/group
index eb4f375..5504557 100644
--- a/group
+++ b/group
@@ -113,7 +113,12 @@ dangerous
# on current systems
deprecated
-#
+# Stress test with probable behaviour, the longer test run the
+# larger corner cases will be covered. Configurable variables:
+# LOAD_FACTOR: Usually means factor number of running tasks
+# TIME_FACTOR: Usually means factor of run time, or number of operations
+stress
+
# test-group association ... one line per test
#
001 rw dir udf auto quick
--
1.7.1
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 02/10] xfstest: add configurable load factors
@ 2013-02-20 10:42 ` Dmitry Monakhov
0 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
Most stress test has probable behaviour, the longer test run the
larger corner cases will be cover. It is reasonable to allow
user to provide some sort of system load factor.
This patch introduce two global variables
LOAD_FACTOR: Usually means factor number of running tasks
TIME_FACTOR: Usually means factor of run time, or number of operations
If not speficied both variables defined to 1, so original behaviour
preserved.
TODO: Change all stress tests to use this variables
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
common.config | 2 ++
group | 7 ++++++-
2 files changed, 8 insertions(+), 1 deletions(-)
diff --git a/common.config b/common.config
index 7681812..8efc7f8 100644
--- a/common.config
+++ b/common.config
@@ -73,6 +73,8 @@ export HOST_OPTIONS=${HOST_OPTIONS:=local.config}
export CHECK_OPTIONS=${CHECK_OPTIONS:="-g auto"}
export BENCH_PASSES=${BENCH_PASSES:=5}
export XFS_MKFS_OPTIONS=${XFS_MKFS_OPTIONS:=-bsize=4096}
+export TIME_FACTOR=${TIME_FACTOR:=1}
+export LOAD_FACTOR=${LOAD_FACTOR:=1}
export PWD=`pwd`
#export DEBUG=${DEBUG:=...} # arbitrary CFLAGS really.
diff --git a/group b/group
index eb4f375..5504557 100644
--- a/group
+++ b/group
@@ -113,7 +113,12 @@ dangerous
# on current systems
deprecated
-#
+# Stress test with probable behaviour, the longer test run the
+# larger corner cases will be covered. Configurable variables:
+# LOAD_FACTOR: Usually means factor number of running tasks
+# TIME_FACTOR: Usually means factor of run time, or number of operations
+stress
+
# test-group association ... one line per test
#
001 rw dir udf auto quick
--
1.7.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 03/10] xfstests: hardcode fops for determinable fsstests runs
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
@ 2013-02-20 10:42 ` Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
` (10 subsequent siblings)
11 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-ext4, linux-fsdevel, dchinner, Dmitry Monakhov
106,107 and 117 are frozen tests which use known seed, it is
reasonable to explicitly hardcode file operations in order to avoid
implicit changes caused by future changes in fsstress.
NOTE: options genereted like follows: fsstress -S c $ORIG_ARGS
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
106 | 33 +++++++++++++++++++++++++++++++--
107 | 32 +++++++++++++++++++++++++++++---
2 files changed, 60 insertions(+), 5 deletions(-)
diff --git a/106 b/106
index 8278691..b351fe5 100755
--- a/106
+++ b/106
@@ -61,8 +61,37 @@ _require_prjquota $SCRATCH_DEV
# initial populate
target=$SCRATCH_MNT/target
-$FSSTRESS_PROG -s 0xdeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
-$FSSTRESS_PROG -s 0xdeed -m8 -z -p4 -n1000 -fsetxattr=500 -fchown=500 -d $target
+$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
+-f allocsp=1 \
+-f chown=3 \
+-f creat=4 \
+-f dwrite=4 \
+-f fallocate=1 \
+-f fdatasync=1 \
+-f fiemap=1 \
+-f freesp=1 \
+-f fsync=1 \
+-f link=1 \
+-f mkdir=2 \
+-f mknod=2 \
+-f punch=1 \
+-f rename=2 \
+-f resvsp=1 \
+-f rmdir=1 \
+-f setxattr=1 \
+-f symlink=2 \
+-f sync=1 \
+-f truncate=2 \
+-f unlink=1 \
+-f unresvsp=1 \
+-f write=4 \
+-d $target
+
+$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
+-f chown=500 \
+-f setxattr=500 \
+-d $target
+
# also use space, to be able to go over/under limits easily
uid=255
diff --git a/107 b/107
index 74403e6..eaf7e6a 100755
--- a/107
+++ b/107
@@ -78,9 +78,35 @@ echo "6:$target" | tee -a $seq.full > $tmp.projects
echo "### populate filesystem"
mkdir $target || exit
-FSSTRESS_AVOID="$FSSTRESS_AVOID -fmknod=0 -fsymlink=0"
-$FSSTRESS_PROG -s 0xfeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
-$FSSTRESS_PROG -s 0xbabe -m8 -z -p4 -n500 -fsetxattr=250 -fchown=250 -d $target
+$FSSTRESS_PROG -z -s 65261 -m 8 -n 1000 -p 4 \
+-f allocsp=1 \
+-f chown=3 \
+-f creat=4 \
+-f dwrite=4 \
+-f fallocate=1 \
+-f fdatasync=1 \
+-f fiemap=1 \
+-f freesp=1 \
+-f fsync=1 \
+-f link=1 \
+-f mkdir=2 \
+-f punch=1 \
+-f rename=2 \
+-f resvsp=1 \
+-f rmdir=1 \
+-f setxattr=1 \
+-f sync=1 \
+-f truncate=2 \
+-f unlink=1 \
+-f unresvsp=1 \
+-f write=4 \
+-d $target
+
+$FSSTRESS_PROG -z -s 47806 -m 8 -n 500 -p 4 \
+-f chown=250 \
+-f setxattr=250 \
+-d $target
+
QARGS="-x -D $tmp.projects -P /dev/null $SCRATCH_MNT"
--
1.7.1
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 03/10] xfstests: hardcode fops for determinable fsstests runs
@ 2013-02-20 10:42 ` Dmitry Monakhov
0 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
106,107 and 117 are frozen tests which use known seed, it is
reasonable to explicitly hardcode file operations in order to avoid
implicit changes caused by future changes in fsstress.
NOTE: options genereted like follows: fsstress -S c $ORIG_ARGS
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
106 | 33 +++++++++++++++++++++++++++++++--
107 | 32 +++++++++++++++++++++++++++++---
2 files changed, 60 insertions(+), 5 deletions(-)
diff --git a/106 b/106
index 8278691..b351fe5 100755
--- a/106
+++ b/106
@@ -61,8 +61,37 @@ _require_prjquota $SCRATCH_DEV
# initial populate
target=$SCRATCH_MNT/target
-$FSSTRESS_PROG -s 0xdeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
-$FSSTRESS_PROG -s 0xdeed -m8 -z -p4 -n1000 -fsetxattr=500 -fchown=500 -d $target
+$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
+-f allocsp=1 \
+-f chown=3 \
+-f creat=4 \
+-f dwrite=4 \
+-f fallocate=1 \
+-f fdatasync=1 \
+-f fiemap=1 \
+-f freesp=1 \
+-f fsync=1 \
+-f link=1 \
+-f mkdir=2 \
+-f mknod=2 \
+-f punch=1 \
+-f rename=2 \
+-f resvsp=1 \
+-f rmdir=1 \
+-f setxattr=1 \
+-f symlink=2 \
+-f sync=1 \
+-f truncate=2 \
+-f unlink=1 \
+-f unresvsp=1 \
+-f write=4 \
+-d $target
+
+$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
+-f chown=500 \
+-f setxattr=500 \
+-d $target
+
# also use space, to be able to go over/under limits easily
uid=255
diff --git a/107 b/107
index 74403e6..eaf7e6a 100755
--- a/107
+++ b/107
@@ -78,9 +78,35 @@ echo "6:$target" | tee -a $seq.full > $tmp.projects
echo "### populate filesystem"
mkdir $target || exit
-FSSTRESS_AVOID="$FSSTRESS_AVOID -fmknod=0 -fsymlink=0"
-$FSSTRESS_PROG -s 0xfeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
-$FSSTRESS_PROG -s 0xbabe -m8 -z -p4 -n500 -fsetxattr=250 -fchown=250 -d $target
+$FSSTRESS_PROG -z -s 65261 -m 8 -n 1000 -p 4 \
+-f allocsp=1 \
+-f chown=3 \
+-f creat=4 \
+-f dwrite=4 \
+-f fallocate=1 \
+-f fdatasync=1 \
+-f fiemap=1 \
+-f freesp=1 \
+-f fsync=1 \
+-f link=1 \
+-f mkdir=2 \
+-f punch=1 \
+-f rename=2 \
+-f resvsp=1 \
+-f rmdir=1 \
+-f setxattr=1 \
+-f sync=1 \
+-f truncate=2 \
+-f unlink=1 \
+-f unresvsp=1 \
+-f write=4 \
+-d $target
+
+$FSSTRESS_PROG -z -s 47806 -m 8 -n 500 -p 4 \
+-f chown=250 \
+-f setxattr=250 \
+-d $target
+
QARGS="-x -D $tmp.projects -P /dev/null $SCRATCH_MNT"
--
1.7.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 04/10] xfstets: fsstress add replace file operation
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
` (2 preceding siblings ...)
2013-02-20 10:42 ` Dmitry Monakhov
@ 2013-02-20 10:42 ` Dmitry Monakhov
2013-03-01 15:43 ` Rich Johnston
2013-02-20 10:42 ` Dmitry Monakhov
` (7 subsequent siblings)
11 siblings, 1 reply; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
The most common usecase for rename(2) syscall is an atomic replacement
of existing file with newer version. But rename_f() rename some existing
filename to newly generated (non existing) filename. As result the most
important usecase is not covered.
Since rename_f() is already exist in fsstress and it has known behavior,
some tests already depends on that behaviour, let's add another operation
(replace_f) which invoke rename(2) for two existing entries.
OUT_OF_COMMIT_DISCUSSION:
Off course replace_f() break naming convention where fun_name == syscall_f(),
but this is the only way I see to introduce new feature and not break
other tests. May be it is reasonable to call it rename2_f() ?
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
ltp/fsstress.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 47 insertions(+), 0 deletions(-)
diff --git a/ltp/fsstress.c b/ltp/fsstress.c
index b4cfb25..85ff72a 100644
--- a/ltp/fsstress.c
+++ b/ltp/fsstress.c
@@ -80,6 +80,7 @@ typedef enum {
OP_READ,
OP_READLINK,
OP_RENAME,
+ OP_REPLACE,
OP_RESVSP,
OP_RMDIR,
OP_SETATTR,
@@ -165,6 +166,7 @@ void punch_f(int, long);
void read_f(int, long);
void readlink_f(int, long);
void rename_f(int, long);
+void replace_f(int, long);
void resvsp_f(int, long);
void rmdir_f(int, long);
void setattr_f(int, long);
@@ -202,6 +204,7 @@ opdesc_t ops[] = {
{ OP_READ, "read", read_f, 1, 0 },
{ OP_READLINK, "readlink", readlink_f, 1, 0 },
{ OP_RENAME, "rename", rename_f, 2, 1 },
+ { OP_REPLACE, "replace", replace_f, 2, 1 },
{ OP_RESVSP, "resvsp", resvsp_f, 1, 1 },
{ OP_RMDIR, "rmdir", rmdir_f, 1, 1 },
{ OP_SETATTR, "setattr", setattr_f, 0, 1 },
@@ -2680,6 +2683,50 @@ rename_f(int opno, long r)
}
void
+replace_f(int opno, long r)
+{
+ int e;
+ pathname_t src_f, dst_f;
+ fent_t *src_fep, *dst_fep;
+ flist_t *src_flp, *dst_flp;
+ int v;
+ int v1;
+
+ /* get an existing path for the source of the rename */
+ init_pathname(&src_f);
+ if (!get_fname(FT_ANYm, r, &src_f, &src_flp, &src_fep, &v)) {
+ if (v)
+ printf("%d/%d: replace - no filename\n", procid, opno);
+ free_pathname(&src_f);
+ return;
+ }
+ /* get an existing path for the destination of the rename */
+ init_pathname(&dst_f);
+ if (!get_fname(1 << (src_flp - flist), rand(), &dst_f, &dst_flp, &dst_fep, &v1)) {
+ if (v1)
+ printf("%d/%d: replace - no filename\n", procid, opno);
+ free_pathname(&dst_f);
+ return;
+ }
+
+ v |= v1;
+ e = rename_path(&src_f, &dst_f) < 0 ? errno : 0;
+ check_cwd();
+ if (e == 0 && src_fep->id != dst_fep->id ) {
+ del_from_flist(src_flp - flist, src_fep - src_flp->fents);
+ }
+ if (v) {
+ printf("%d/%d: replace %s with %s %d\n", procid, opno, dst_f.path,
+ src_f.path, e);
+ if (e == 0 && src_fep->id != dst_fep->id)
+ printf("%d/%d: replace del entry: id=%d,parent=%d\n",
+ procid, opno, src_fep->id, src_fep->parent);
+ }
+ free_pathname(&src_f);
+ free_pathname(&dst_f);
+}
+
+void
resvsp_f(int opno, long r)
{
int e;
--
1.7.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 05/10] xfstest: allow fsstress to use load factor where appropriate
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
@ 2013-02-20 10:42 ` Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
` (10 subsequent siblings)
11 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-ext4, linux-fsdevel, dchinner, Dmitry Monakhov
1) Add _scale_fsstress_args function which transform argumets according
to load factors
2) Let all non deterministic fsstress tests to use scaled arguments
I've able to trigger OOPS on xfs see:https://gist.github.com/dmonakhov/4762653
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
017 | 3 ++-
068 | 4 ++--
070 | 5 +++--
076 | 5 +++--
083 | 4 ++--
087 | 3 ++-
104 | 4 +++-
114 | 4 ++--
167 | 5 +++--
232 | 4 ++--
232.out | 1 -
233 | 6 +++---
233.out | 1 -
269 | 2 +-
270 | 2 +-
common.rc | 14 ++++++++++++++
group | 26 +++++++++++++-------------
17 files changed, 56 insertions(+), 37 deletions(-)
diff --git a/017 b/017
index 9ca0e72..7db4667 100755
--- a/017
+++ b/017
@@ -67,7 +67,8 @@ echo "*** test"
for l in 0 1 2 3 4
do
echo " *** test $l"
- $FSSTRESS_PROG -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID >>$seq.full
+ FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID`
+ $FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full
_scratch_mount -o remount,ro \
|| _fail "remount ro failed"
diff --git a/068 b/068
index a641e2f..cbfea95 100755
--- a/068
+++ b/068
@@ -82,8 +82,8 @@ touch $tmp.running
do
# We do both read & write IO - not only is this more realistic,
# but it also potentially tests atime updates
- $FSSTRESS_PROG -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID \
- > /dev/null 2>&1
+ FSSTRESS_ARGS=`_scale_fsstress_args -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID`
+ $FSSTRESS_PROG $FSSTRESS_ARGS > /dev/null 2>&1
done
rm -r $STRESS_DIR/*
diff --git a/070 b/070
index f48c33c..334cce7 100755
--- a/070
+++ b/070
@@ -52,7 +52,7 @@ _require_attrs
_setup_testdir
-$FSSTRESS_PROG \
+FSSTRESS_ARGS=`_scale_fsstress_args \
-d $testdir/fsstress \
-f allocsp=0 \
-f freesp=0 \
@@ -62,7 +62,8 @@ $FSSTRESS_PROG \
-f unresvsp=0 \
-f attr_set=100 \
-f attr_remove=100 \
- -p 1 -n 10000 -S c >$seq.full 2>&1
+ -p 1 -n 10000 -S c`
+$FSSTRESS_PROG $FSSTRESS_ARGS >$seq.full 2>&1
status=$?
exit
diff --git a/076 b/076
index e472b26..793b869 100755
--- a/076
+++ b/076
@@ -74,8 +74,9 @@ echo "*** test concurrent block/fs access"
cat $SCRATCH_DEV >/dev/null &
pid=$!
-$FSSTRESS_PROG -d $SCRATCH_MNT -p 2 -n 2000 $FSSTRESS_AVOID >>$seq.full
-
+FSSTRESS_ARGS=`_scale_fsstress_args -p 2 -n 2000 $FSSTRESS_AVOID`
+echo "run fsstress with args: $FSSTRESS_ARGS" >>$seq.full
+$FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full
_lets_get_pidst
_check_scratch_fs
diff --git a/083 b/083
index e0670b9..f5349a9 100755
--- a/083
+++ b/083
@@ -84,8 +84,8 @@ workout()
|| _fail "mount failed"
# -w ensures that the only ops are ones which cause write I/O
- $FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID \
- >>$seq.full
+ FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID`
+ $FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full
_check_scratch_fs
}
diff --git a/087 b/087
index 48e5eaa..e7be9a0 100755
--- a/087
+++ b/087
@@ -47,7 +47,8 @@ _do_meta()
param="-p 4 -z -f rmdir=10 -f link=10 -f creat=10 -f mkdir=10 \
-f rename=30 -f stat=30 -f unlink=30 -f truncate=20"
_echofull "calling fsstress $param -m8 -n $count"
- if ! $FSSTRESS_PROG $param $FSSTRESS_AVOID -m 8 -n $count -d $out >>$seq.full 2>&1
+ FSSTRESS_ARGS=`_scale_fsstress_args $param $FSSTRESS_AVOID -m 8 -n $count -d $out`
+ if ! $FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full 2>&1
then
_echofull "fsstress failed"
fi
diff --git a/104 b/104
index 14f2669..8db6d88 100755
--- a/104
+++ b/104
@@ -64,7 +64,9 @@ _stress_scratch()
procs=3
nops=1000
# -w ensures that the only ops are ones which cause write I/O
- $FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID > /dev/null &
+ FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
+ -n $nops $FSSTRESS_AVOID`
+ $FSSTRESS_PROG $FSSTRESS_ARGS >> $seq.full &
}
# real QA test starts here
diff --git a/114 b/114
index 7679222..edce0f4 100755
--- a/114
+++ b/114
@@ -246,11 +246,11 @@ _test_fsstress()
out=$SCRATCH_MNT/fsstress.$$
count=1000
- args="-z \
+ args=`_scale_fsstress_args -z \
-f rmdir=10 -f link=10 -f creat=10 \
-f mkdir=10 -f rename=30 -f unlink=10 \
-f symlink=10 \
--n $count -d $out -p 3"
+-n $count -d $out -p 3`
echo "fsstress $args" | sed -e "s#$out#outdir#"
if ! $FSSTRESS_PROG $args | _filter_num
diff --git a/167 b/167
index ccb6c2a..5fb95e8 100755
--- a/167
+++ b/167
@@ -44,8 +44,9 @@ workout()
{
procs=100
nops=15000
- $FSSTRESS_PROG -d $SCRATCH_MNT -p $procs -n $nops $FSSTRESS_AVOID \
- >>$seq.full &
+ FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p $procs -n $nops \
+ $FSSTRESS_AVOID`
+ $FSSTRESS_PROG $FSSTRESS_ARGS >> $seq.full &
sleep 2
}
diff --git a/232 b/232
index 2795da7..d915d42 100755
--- a/232
+++ b/232
@@ -54,9 +54,9 @@ _fsstress()
out=$SCRATCH_MNT/fsstress.$$
count=2000
- args="-n $count -d $out -p 7"
+ args=`_scale_fsstress_args -d $out -n $count -p 7`
- echo "fsstress $args" | tee -a $here/$seq.full | sed -e "s#$out#outdir#"
+ echo "fsstress $args" >> tee -a $here/$seq.full
if ! $FSSTRESS_PROG $args | tee -a $here/$seq.full | _filter_num
then
echo " fsstress $args returned $?"
diff --git a/232.out b/232.out
index ef82a89..5da53d4 100644
--- a/232.out
+++ b/232.out
@@ -2,7 +2,6 @@ QA output created by 232
Testing fsstress
-fsstress -n 2000 -d outdir -p 7
seed = S
Comparing user usage
Comparing group usage
diff --git a/233 b/233
index 28e6ac7..649de51 100755
--- a/233
+++ b/233
@@ -58,12 +58,12 @@ _fsstress()
out=$SCRATCH_MNT/fsstress.$$
count=5000
- args="-z \
+ args=`_scale_fsstress_args -z \
-f rmdir=20 -f link=10 -f creat=10 -f mkdir=10 -f unlink=20 -f symlink=10 \
-f rename=10 -f fsync=2 -f write=15 -f dwrite=15 \
--n $count -d $out -p 7"
+-n $count -d $out -p 7`
- echo "fsstress $args" | tee -a $here/$seq.full | sed -e "s#$out#outdir#"
+ echo "fsstress $args" >> tee -a $here/$seq.full
if ! su $qa_user -c "$FSSTRESS_PROG $args" | tee -a $here/$seq.full | _filter_num
then
echo " fsstress $args returned $?"
diff --git a/233.out b/233.out
index fa36ca3..91c1a30 100644
--- a/233.out
+++ b/233.out
@@ -2,7 +2,6 @@ QA output created by 233
Testing fsstress
-fsstress -z -f rmdir=20 -f link=10 -f creat=10 -f mkdir=10 -f unlink=20 -f symlink=10 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 -n 5000 -d outdir -p 7
seed = S
Comparing user usage
Comparing group usage
diff --git a/269 b/269
index 7e13ed9..7d63b87 100755
--- a/269
+++ b/269
@@ -45,7 +45,7 @@ _workout()
num_iterations=10
enospc_time=2
out=$SCRATCH_MNT/fsstress.$$
- args="-p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
+ args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
echo "fsstress $args" >> $here/$seq.full
$FSSTRESS_PROG $args > /dev/null 2>&1 &
pid=$!
diff --git a/270 b/270
index b9ada27..b753923 100755
--- a/270
+++ b/270
@@ -48,7 +48,7 @@ _workout()
num_iterations=10
enospc_time=2
out=$SCRATCH_MNT/fsstress.$$
- args="-p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
+ args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
echo "fsstress $args" >> $here/$seq.full
# Grant chown capability
cp $FSSTRESS_PROG $tmp.fsstress.bin
diff --git a/common.rc b/common.rc
index 14de47b..3635a45 100644
--- a/common.rc
+++ b/common.rc
@@ -1841,6 +1841,20 @@ _test_batched_discard()
$FSTRIM_PROG ${1} &>/dev/null
}
+_scale_fsstress_args()
+{
+ args=""
+ while [ $# -gt 0 ]; do
+ case "$1" in
+ -n) args="$args $1 $(($2 * $TIME_FACTOR))"; shift ;;
+ -p) args="$args $1 $(($2 * $LOAD_FACTOR))"; shift ;;
+ *) args="$args $1" ;;
+ esac
+ shift
+ done
+ echo $args
+}
+
################################################################################
if [ "$iam" != new -a "$iam" != bench ]
diff --git a/group b/group
index 5504557..1fb2feb 100644
--- a/group
+++ b/group
@@ -137,7 +137,7 @@ stress
014 rw udf auto quick
015 other auto quick
016 rw auto quick
-017 mount auto quick
+017 mount auto quick stress
018 deprecated # log logprint v2log
019 mkfs auto quick
020 metadata attr udf auto quick
@@ -188,26 +188,26 @@ stress
065 dump auto
066 dump ioctl auto quick
067 acl attr auto quick
-068 other auto freeze dangerous
+068 other auto freeze dangerous stress
069 rw udf auto quick
-070 attr udf auto quick
+070 attr udf auto quick stress
071 rw auto
072 rw auto prealloc quick
073 copy auto
074 rw udf auto
075 rw udf auto quick
-076 metadata rw udf auto quick
+076 metadata rw udf auto quick stress
077 acl attr auto enospc
078 growfs auto quick
079 acl attr ioctl metadata auto quick
080 rw ioctl
081 deprecated # log logprint quota
082 deprecated # log logprint v2log
-083 rw auto
+083 rw auto enospc stress
084 ioctl rw auto
085 log auto quick
086 log v2log auto
-087 log v2log auto quota
+087 log v2log auto quota stress
088 perms auto quick
089 metadata auto
090 rw auto
@@ -224,7 +224,7 @@ stress
101 udf
102 udf
103 metadata dir ioctl auto quick
-104 growfs ioctl prealloc auto
+104 growfs ioctl prealloc auto stress
105 acl auto quick
106 quota
107 quota
@@ -234,7 +234,7 @@ stress
111 ioctl
112 rw aio auto quick
113 rw aio auto quick
-114 parent attr
+114 parent attr stress
115 parent attr
116 quota auto quick
117 attr auto quick
@@ -287,7 +287,7 @@ stress
164 rw pattern auto prealloc quick
165 rw pattern auto prealloc quick
166 rw metadata auto quick
-167 rw metadata auto
+167 rw metadata auto stress
168 dmapi auto
169 rw metadata auto quick
170 rw filestreams auto quick
@@ -355,8 +355,8 @@ stress
229 auto rw
230 auto quota quick
231 auto quota
-232 auto quota
-233 auto quota
+232 auto quota stress
+233 auto quota stress
234 auto quota
235 auto quota quick
236 auto quick metadata
@@ -392,8 +392,8 @@ stress
266 dump ioctl auto quick
267 dump ioctl tape
268 dump ioctl tape
-269 auto rw prealloc ioctl enospc
-270 auto quota rw prealloc ioctl enospc
+269 auto rw prealloc ioctl enospc stress
+270 auto quota rw prealloc ioctl enospc stress
271 auto rw quick
272 auto enospc rw
273 auto rw
--
1.7.1
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 05/10] xfstest: allow fsstress to use load factor where appropriate
@ 2013-02-20 10:42 ` Dmitry Monakhov
0 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
1) Add _scale_fsstress_args function which transform argumets according
to load factors
2) Let all non deterministic fsstress tests to use scaled arguments
I've able to trigger OOPS on xfs see:https://gist.github.com/dmonakhov/4762653
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
017 | 3 ++-
068 | 4 ++--
070 | 5 +++--
076 | 5 +++--
083 | 4 ++--
087 | 3 ++-
104 | 4 +++-
114 | 4 ++--
167 | 5 +++--
232 | 4 ++--
232.out | 1 -
233 | 6 +++---
233.out | 1 -
269 | 2 +-
270 | 2 +-
common.rc | 14 ++++++++++++++
group | 26 +++++++++++++-------------
17 files changed, 56 insertions(+), 37 deletions(-)
diff --git a/017 b/017
index 9ca0e72..7db4667 100755
--- a/017
+++ b/017
@@ -67,7 +67,8 @@ echo "*** test"
for l in 0 1 2 3 4
do
echo " *** test $l"
- $FSSTRESS_PROG -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID >>$seq.full
+ FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID`
+ $FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full
_scratch_mount -o remount,ro \
|| _fail "remount ro failed"
diff --git a/068 b/068
index a641e2f..cbfea95 100755
--- a/068
+++ b/068
@@ -82,8 +82,8 @@ touch $tmp.running
do
# We do both read & write IO - not only is this more realistic,
# but it also potentially tests atime updates
- $FSSTRESS_PROG -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID \
- > /dev/null 2>&1
+ FSSTRESS_ARGS=`_scale_fsstress_args -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID`
+ $FSSTRESS_PROG $FSSTRESS_ARGS > /dev/null 2>&1
done
rm -r $STRESS_DIR/*
diff --git a/070 b/070
index f48c33c..334cce7 100755
--- a/070
+++ b/070
@@ -52,7 +52,7 @@ _require_attrs
_setup_testdir
-$FSSTRESS_PROG \
+FSSTRESS_ARGS=`_scale_fsstress_args \
-d $testdir/fsstress \
-f allocsp=0 \
-f freesp=0 \
@@ -62,7 +62,8 @@ $FSSTRESS_PROG \
-f unresvsp=0 \
-f attr_set=100 \
-f attr_remove=100 \
- -p 1 -n 10000 -S c >$seq.full 2>&1
+ -p 1 -n 10000 -S c`
+$FSSTRESS_PROG $FSSTRESS_ARGS >$seq.full 2>&1
status=$?
exit
diff --git a/076 b/076
index e472b26..793b869 100755
--- a/076
+++ b/076
@@ -74,8 +74,9 @@ echo "*** test concurrent block/fs access"
cat $SCRATCH_DEV >/dev/null &
pid=$!
-$FSSTRESS_PROG -d $SCRATCH_MNT -p 2 -n 2000 $FSSTRESS_AVOID >>$seq.full
-
+FSSTRESS_ARGS=`_scale_fsstress_args -p 2 -n 2000 $FSSTRESS_AVOID`
+echo "run fsstress with args: $FSSTRESS_ARGS" >>$seq.full
+$FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full
_lets_get_pidst
_check_scratch_fs
diff --git a/083 b/083
index e0670b9..f5349a9 100755
--- a/083
+++ b/083
@@ -84,8 +84,8 @@ workout()
|| _fail "mount failed"
# -w ensures that the only ops are ones which cause write I/O
- $FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID \
- >>$seq.full
+ FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID`
+ $FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full
_check_scratch_fs
}
diff --git a/087 b/087
index 48e5eaa..e7be9a0 100755
--- a/087
+++ b/087
@@ -47,7 +47,8 @@ _do_meta()
param="-p 4 -z -f rmdir=10 -f link=10 -f creat=10 -f mkdir=10 \
-f rename=30 -f stat=30 -f unlink=30 -f truncate=20"
_echofull "calling fsstress $param -m8 -n $count"
- if ! $FSSTRESS_PROG $param $FSSTRESS_AVOID -m 8 -n $count -d $out >>$seq.full 2>&1
+ FSSTRESS_ARGS=`_scale_fsstress_args $param $FSSTRESS_AVOID -m 8 -n $count -d $out`
+ if ! $FSSTRESS_PROG $FSSTRESS_ARGS >>$seq.full 2>&1
then
_echofull "fsstress failed"
fi
diff --git a/104 b/104
index 14f2669..8db6d88 100755
--- a/104
+++ b/104
@@ -64,7 +64,9 @@ _stress_scratch()
procs=3
nops=1000
# -w ensures that the only ops are ones which cause write I/O
- $FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID > /dev/null &
+ FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
+ -n $nops $FSSTRESS_AVOID`
+ $FSSTRESS_PROG $FSSTRESS_ARGS >> $seq.full &
}
# real QA test starts here
diff --git a/114 b/114
index 7679222..edce0f4 100755
--- a/114
+++ b/114
@@ -246,11 +246,11 @@ _test_fsstress()
out=$SCRATCH_MNT/fsstress.$$
count=1000
- args="-z \
+ args=`_scale_fsstress_args -z \
-f rmdir=10 -f link=10 -f creat=10 \
-f mkdir=10 -f rename=30 -f unlink=10 \
-f symlink=10 \
--n $count -d $out -p 3"
+-n $count -d $out -p 3`
echo "fsstress $args" | sed -e "s#$out#outdir#"
if ! $FSSTRESS_PROG $args | _filter_num
diff --git a/167 b/167
index ccb6c2a..5fb95e8 100755
--- a/167
+++ b/167
@@ -44,8 +44,9 @@ workout()
{
procs=100
nops=15000
- $FSSTRESS_PROG -d $SCRATCH_MNT -p $procs -n $nops $FSSTRESS_AVOID \
- >>$seq.full &
+ FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p $procs -n $nops \
+ $FSSTRESS_AVOID`
+ $FSSTRESS_PROG $FSSTRESS_ARGS >> $seq.full &
sleep 2
}
diff --git a/232 b/232
index 2795da7..d915d42 100755
--- a/232
+++ b/232
@@ -54,9 +54,9 @@ _fsstress()
out=$SCRATCH_MNT/fsstress.$$
count=2000
- args="-n $count -d $out -p 7"
+ args=`_scale_fsstress_args -d $out -n $count -p 7`
- echo "fsstress $args" | tee -a $here/$seq.full | sed -e "s#$out#outdir#"
+ echo "fsstress $args" >> tee -a $here/$seq.full
if ! $FSSTRESS_PROG $args | tee -a $here/$seq.full | _filter_num
then
echo " fsstress $args returned $?"
diff --git a/232.out b/232.out
index ef82a89..5da53d4 100644
--- a/232.out
+++ b/232.out
@@ -2,7 +2,6 @@ QA output created by 232
Testing fsstress
-fsstress -n 2000 -d outdir -p 7
seed = S
Comparing user usage
Comparing group usage
diff --git a/233 b/233
index 28e6ac7..649de51 100755
--- a/233
+++ b/233
@@ -58,12 +58,12 @@ _fsstress()
out=$SCRATCH_MNT/fsstress.$$
count=5000
- args="-z \
+ args=`_scale_fsstress_args -z \
-f rmdir=20 -f link=10 -f creat=10 -f mkdir=10 -f unlink=20 -f symlink=10 \
-f rename=10 -f fsync=2 -f write=15 -f dwrite=15 \
--n $count -d $out -p 7"
+-n $count -d $out -p 7`
- echo "fsstress $args" | tee -a $here/$seq.full | sed -e "s#$out#outdir#"
+ echo "fsstress $args" >> tee -a $here/$seq.full
if ! su $qa_user -c "$FSSTRESS_PROG $args" | tee -a $here/$seq.full | _filter_num
then
echo " fsstress $args returned $?"
diff --git a/233.out b/233.out
index fa36ca3..91c1a30 100644
--- a/233.out
+++ b/233.out
@@ -2,7 +2,6 @@ QA output created by 233
Testing fsstress
-fsstress -z -f rmdir=20 -f link=10 -f creat=10 -f mkdir=10 -f unlink=20 -f symlink=10 -f rename=10 -f fsync=2 -f write=15 -f dwrite=15 -n 5000 -d outdir -p 7
seed = S
Comparing user usage
Comparing group usage
diff --git a/269 b/269
index 7e13ed9..7d63b87 100755
--- a/269
+++ b/269
@@ -45,7 +45,7 @@ _workout()
num_iterations=10
enospc_time=2
out=$SCRATCH_MNT/fsstress.$$
- args="-p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
+ args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
echo "fsstress $args" >> $here/$seq.full
$FSSTRESS_PROG $args > /dev/null 2>&1 &
pid=$!
diff --git a/270 b/270
index b9ada27..b753923 100755
--- a/270
+++ b/270
@@ -48,7 +48,7 @@ _workout()
num_iterations=10
enospc_time=2
out=$SCRATCH_MNT/fsstress.$$
- args="-p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out"
+ args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
echo "fsstress $args" >> $here/$seq.full
# Grant chown capability
cp $FSSTRESS_PROG $tmp.fsstress.bin
diff --git a/common.rc b/common.rc
index 14de47b..3635a45 100644
--- a/common.rc
+++ b/common.rc
@@ -1841,6 +1841,20 @@ _test_batched_discard()
$FSTRIM_PROG ${1} &>/dev/null
}
+_scale_fsstress_args()
+{
+ args=""
+ while [ $# -gt 0 ]; do
+ case "$1" in
+ -n) args="$args $1 $(($2 * $TIME_FACTOR))"; shift ;;
+ -p) args="$args $1 $(($2 * $LOAD_FACTOR))"; shift ;;
+ *) args="$args $1" ;;
+ esac
+ shift
+ done
+ echo $args
+}
+
################################################################################
if [ "$iam" != new -a "$iam" != bench ]
diff --git a/group b/group
index 5504557..1fb2feb 100644
--- a/group
+++ b/group
@@ -137,7 +137,7 @@ stress
014 rw udf auto quick
015 other auto quick
016 rw auto quick
-017 mount auto quick
+017 mount auto quick stress
018 deprecated # log logprint v2log
019 mkfs auto quick
020 metadata attr udf auto quick
@@ -188,26 +188,26 @@ stress
065 dump auto
066 dump ioctl auto quick
067 acl attr auto quick
-068 other auto freeze dangerous
+068 other auto freeze dangerous stress
069 rw udf auto quick
-070 attr udf auto quick
+070 attr udf auto quick stress
071 rw auto
072 rw auto prealloc quick
073 copy auto
074 rw udf auto
075 rw udf auto quick
-076 metadata rw udf auto quick
+076 metadata rw udf auto quick stress
077 acl attr auto enospc
078 growfs auto quick
079 acl attr ioctl metadata auto quick
080 rw ioctl
081 deprecated # log logprint quota
082 deprecated # log logprint v2log
-083 rw auto
+083 rw auto enospc stress
084 ioctl rw auto
085 log auto quick
086 log v2log auto
-087 log v2log auto quota
+087 log v2log auto quota stress
088 perms auto quick
089 metadata auto
090 rw auto
@@ -224,7 +224,7 @@ stress
101 udf
102 udf
103 metadata dir ioctl auto quick
-104 growfs ioctl prealloc auto
+104 growfs ioctl prealloc auto stress
105 acl auto quick
106 quota
107 quota
@@ -234,7 +234,7 @@ stress
111 ioctl
112 rw aio auto quick
113 rw aio auto quick
-114 parent attr
+114 parent attr stress
115 parent attr
116 quota auto quick
117 attr auto quick
@@ -287,7 +287,7 @@ stress
164 rw pattern auto prealloc quick
165 rw pattern auto prealloc quick
166 rw metadata auto quick
-167 rw metadata auto
+167 rw metadata auto stress
168 dmapi auto
169 rw metadata auto quick
170 rw filestreams auto quick
@@ -355,8 +355,8 @@ stress
229 auto rw
230 auto quota quick
231 auto quota
-232 auto quota
-233 auto quota
+232 auto quota stress
+233 auto quota stress
234 auto quota
235 auto quota quick
236 auto quick metadata
@@ -392,8 +392,8 @@ stress
266 dump ioctl auto quick
267 dump ioctl tape
268 dump ioctl tape
-269 auto rw prealloc ioctl enospc
-270 auto quota rw prealloc ioctl enospc
+269 auto rw prealloc ioctl enospc stress
+270 auto quota rw prealloc ioctl enospc stress
271 auto rw quick
272 auto enospc rw
273 auto rw
--
1.7.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 06/10] xfstest: move run_check to common.rc
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
@ 2013-02-20 10:42 ` Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
` (10 subsequent siblings)
11 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-ext4, linux-fsdevel, dchinner, Dmitry Monakhov
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
276 | 6 ------
common.rc | 8 +++++++-
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/276 b/276
index 082f943..c1ce8ac 100755
--- a/276
+++ b/276
@@ -165,12 +165,6 @@ _btrfs_inspect_check()
return $ret
}
-run_check()
-{
- echo "# $@" >> $seq.full 2>&1
- "$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
-}
-
workout()
{
fsz=$1
diff --git a/common.rc b/common.rc
index 3635a45..e7ae9bf 100644
--- a/common.rc
+++ b/common.rc
@@ -1854,7 +1854,13 @@ _scale_fsstress_args()
done
echo $args
}
-
+
+run_check()
+{
+ echo "# $@" >> $seq.full 2>&1
+ "$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
+}
+
################################################################################
if [ "$iam" != new -a "$iam" != bench ]
--
1.7.1
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 06/10] xfstest: move run_check to common.rc
@ 2013-02-20 10:42 ` Dmitry Monakhov
0 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
276 | 6 ------
common.rc | 8 +++++++-
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/276 b/276
index 082f943..c1ce8ac 100755
--- a/276
+++ b/276
@@ -165,12 +165,6 @@ _btrfs_inspect_check()
return $ret
}
-run_check()
-{
- echo "# $@" >> $seq.full 2>&1
- "$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
-}
-
workout()
{
fsz=$1
diff --git a/common.rc b/common.rc
index 3635a45..e7ae9bf 100644
--- a/common.rc
+++ b/common.rc
@@ -1854,7 +1854,13 @@ _scale_fsstress_args()
done
echo $args
}
-
+
+run_check()
+{
+ echo "# $@" >> $seq.full 2>&1
+ "$@" >> $seq.full 2>&1 || _fail "failed: '$@'"
+}
+
################################################################################
if [ "$iam" != new -a "$iam" != bench ]
--
1.7.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 07/10] xfstest: add fallocate/truncate vs AIO/DIO stress test
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
@ 2013-02-20 10:42 ` Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
` (10 subsequent siblings)
11 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-ext4, linux-fsdevel, dchinner, Dmitry Monakhov
Run DIO, fallocate and truncate threads on a common file in parallel.
If race exist old dio request may rewrite blocks after it was allocated
to another file, we will catch that by verifying blocks content.
this patch known to catch deadlock for ext4
http://lists.openwall.net/linux-ext4/2012/09/06/3
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
297 | 157 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
297.out | 5 ++
group | 1 +
3 files changed, 163 insertions(+), 0 deletions(-)
create mode 100755 297
create mode 100644 297.out
diff --git a/297 b/297
new file mode 100755
index 0000000..83bc572
--- /dev/null
+++ b/297
@@ -0,0 +1,157 @@
+#! /bin/bash
+# FSQA Test No. 297
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity and fallocate/truncate simultaneously
+# Test will operate on huge sparsed files so ENOSPC is expected.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+iodepth=128*${LOAD_FACTOR}
+continue_on_error=write
+ignore_error=,ENOSPC
+error_dump=0
+create_on_open=1
+fallocate=none
+exitall=1
+
+## Perform direct aio, to files which may be truncated
+## by external task
+[direct_aio]
+direct=1
+buffered=0
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=100*${TIME_FACTOR}
+time_based
+
+# Perform direct aio and verify data
+# This test case should check use-after-free issues
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=1
+bs=4k
+rw=randrw
+filename=aio-dio-verifier
+
+# Perform buffered aio and verify data
+# This test case should check use-after-free issues
+[buffered-aio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=0
+buffered=1
+bs=4k
+rw=randrw
+filename=buffered-aio-verifier
+EOF
+
+_require_fio $tmp-$seq.fio
+
+_workout()
+{
+ echo ""
+ echo "Run fio with random aio-dio pattern"
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio &
+ pid=$!
+ echo "Start fallocate/truncate loop"
+
+ for ((i=0; ; i++))
+ do
+ for ((k=1; k <= NUM_JOBS; k++))
+ do
+ fallocate -l $FILE_SIZE $SCRATCH_MNT/direct_aio.$k.0 \
+ >> $seq.full 2>&1
+ done
+ for ((k=1; k <= NUM_JOBS; k++))
+ do
+ truncate -s 0 $SCRATCH_MNT/direct_aio.$k.0
+ done
+ # Following like will check that pid is still run.
+ # Once fio exit we can stop fallocate/truncate loop
+ kill -0 $pid > /dev/null 2>&1 || break
+ done
+ wait $pid
+}
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/297.out b/297.out
new file mode 100644
index 0000000..435fac7
--- /dev/null
+++ b/297.out
@@ -0,0 +1,5 @@
+QA output created by 297
+
+Run fio with random aio-dio pattern
+
+Start fallocate/truncate loop
diff --git a/group b/group
index 1fb2feb..2783286 100644
--- a/group
+++ b/group
@@ -420,3 +420,4 @@ stress
294 auto quick
295 auto logprint quick
296 dump auto quick
+297 auto aio enospc rw stress
\ No newline at end of file
--
1.7.1
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 07/10] xfstest: add fallocate/truncate vs AIO/DIO stress test
@ 2013-02-20 10:42 ` Dmitry Monakhov
0 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
Run DIO, fallocate and truncate threads on a common file in parallel.
If race exist old dio request may rewrite blocks after it was allocated
to another file, we will catch that by verifying blocks content.
this patch known to catch deadlock for ext4
http://lists.openwall.net/linux-ext4/2012/09/06/3
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
297 | 157 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
297.out | 5 ++
group | 1 +
3 files changed, 163 insertions(+), 0 deletions(-)
create mode 100755 297
create mode 100644 297.out
diff --git a/297 b/297
new file mode 100755
index 0000000..83bc572
--- /dev/null
+++ b/297
@@ -0,0 +1,157 @@
+#! /bin/bash
+# FSQA Test No. 297
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity and fallocate/truncate simultaneously
+# Test will operate on huge sparsed files so ENOSPC is expected.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+iodepth=128*${LOAD_FACTOR}
+continue_on_error=write
+ignore_error=,ENOSPC
+error_dump=0
+create_on_open=1
+fallocate=none
+exitall=1
+
+## Perform direct aio, to files which may be truncated
+## by external task
+[direct_aio]
+direct=1
+buffered=0
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=100*${TIME_FACTOR}
+time_based
+
+# Perform direct aio and verify data
+# This test case should check use-after-free issues
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=1
+bs=4k
+rw=randrw
+filename=aio-dio-verifier
+
+# Perform buffered aio and verify data
+# This test case should check use-after-free issues
+[buffered-aio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=0
+buffered=1
+bs=4k
+rw=randrw
+filename=buffered-aio-verifier
+EOF
+
+_require_fio $tmp-$seq.fio
+
+_workout()
+{
+ echo ""
+ echo "Run fio with random aio-dio pattern"
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio &
+ pid=$!
+ echo "Start fallocate/truncate loop"
+
+ for ((i=0; ; i++))
+ do
+ for ((k=1; k <= NUM_JOBS; k++))
+ do
+ fallocate -l $FILE_SIZE $SCRATCH_MNT/direct_aio.$k.0 \
+ >> $seq.full 2>&1
+ done
+ for ((k=1; k <= NUM_JOBS; k++))
+ do
+ truncate -s 0 $SCRATCH_MNT/direct_aio.$k.0
+ done
+ # Following like will check that pid is still run.
+ # Once fio exit we can stop fallocate/truncate loop
+ kill -0 $pid > /dev/null 2>&1 || break
+ done
+ wait $pid
+}
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/297.out b/297.out
new file mode 100644
index 0000000..435fac7
--- /dev/null
+++ b/297.out
@@ -0,0 +1,5 @@
+QA output created by 297
+
+Run fio with random aio-dio pattern
+
+Start fallocate/truncate loop
diff --git a/group b/group
index 1fb2feb..2783286 100644
--- a/group
+++ b/group
@@ -420,3 +420,4 @@ stress
294 auto quick
295 auto logprint quick
296 dump auto quick
+297 auto aio enospc rw stress
\ No newline at end of file
--
1.7.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 08/10] xfstest: add fallocate/punch_hole vs AIO/DIO stress test
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
@ 2013-02-20 10:42 ` Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
` (10 subsequent siblings)
11 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-ext4, linux-fsdevel, dchinner, Dmitry Monakhov
Run random AIO/DIO activity (fio's job:direct_aio_raicer)
random fallocate activity(fio's job:falloc_raicer)
and random punch_hole activity(punch_hole_raicer) on a common
file in parallel. If race exist old dio request may rewrite
punched block after it was allocated to another file, we will
catch that by verifier fio's job: "aio-dio-verifier".
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
298 | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
298.out | 4 ++
group | 3 +-
3 files changed, 159 insertions(+), 1 deletions(-)
create mode 100755 298
create mode 100644 298.out
diff --git a/298 b/298
new file mode 100755
index 0000000..3264b8e
--- /dev/null
+++ b/298
@@ -0,0 +1,153 @@
+#! /bin/bash
+# FSQA Test No. 298
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity and fallocate/punch_hole simultaneously
+# Test will operate on huge sparsed file so ENOSPC is expected.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+if [ $((BLK_DEV_SIZE)) -gt 1048576 ]
+then
+ BLK_DEV_SIZE=1048576
+fi
+FS_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test fio activity
+# Run DIO, fallocate and punch_hole threads on a single in parallel
+#
+# If race exist old dio request may rewrite punched block after it was
+# allocated to another file, we will catch that by verifying blocks content
+#
+[global]
+directory=${SCRATCH_MNT}
+filesize=${FS_SIZE}
+size=999G
+continue_on_error=write
+ignore_error=,ENOSPC
+error_dump=0
+
+create_on_open=1
+fallocate=none
+exitall=1
+
+## Perform direct aio, to files which may be truncated
+## by external task
+[direct_aio_raicer]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+bs=128k
+direct=1
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=100*${TIME_FACTOR}
+time_based
+filename=racer
+
+# Run falloc and punch_hole threads in parallel
+# After activity file will be highly fragmented
+[falloc_raicer]
+ioengine=falloc
+runtime=100*${TIME_FACTOR}
+iodepth=1
+bssplit=128k/80:512k/10:32k/10
+rw=randwrite
+numjobs=1
+filename=racer
+
+[punch_hole_raicer]
+ioengine=falloc
+runtime=100*${TIME_FACTOR}
+bs=4k
+time_based=10
+rw=randtrim
+numjobs=2
+filename=racer
+time_based
+
+# Verifier thread continiously write to newly allcated blocks
+# and veryfy written content
+[aio-dio-verifier]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=1
+bs=4k
+rw=randwrite
+filename=aio-dio-verifier
+EOF
+
+_workout()
+{
+ echo ""
+ echo "Run fio with random aio-dio pattern"
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs_sized $FS_SIZE >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/298.out b/298.out
new file mode 100644
index 0000000..faedc77
--- /dev/null
+++ b/298.out
@@ -0,0 +1,4 @@
+QA output created by 298
+
+Run fio with random aio-dio pattern
+
diff --git a/group b/group
index 2783286..ff893ac 100644
--- a/group
+++ b/group
@@ -420,4 +420,5 @@ stress
294 auto quick
295 auto logprint quick
296 dump auto quick
-297 auto aio enospc rw stress
\ No newline at end of file
+297 auto aio enospc rw stress
+298 auto aio enospc preallocrw stress
\ No newline at end of file
--
1.7.1
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 08/10] xfstest: add fallocate/punch_hole vs AIO/DIO stress test
@ 2013-02-20 10:42 ` Dmitry Monakhov
0 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
Run random AIO/DIO activity (fio's job:direct_aio_raicer)
random fallocate activity(fio's job:falloc_raicer)
and random punch_hole activity(punch_hole_raicer) on a common
file in parallel. If race exist old dio request may rewrite
punched block after it was allocated to another file, we will
catch that by verifier fio's job: "aio-dio-verifier".
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
298 | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
298.out | 4 ++
group | 3 +-
3 files changed, 159 insertions(+), 1 deletions(-)
create mode 100755 298
create mode 100644 298.out
diff --git a/298 b/298
new file mode 100755
index 0000000..3264b8e
--- /dev/null
+++ b/298
@@ -0,0 +1,153 @@
+#! /bin/bash
+# FSQA Test No. 298
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity and fallocate/punch_hole simultaneously
+# Test will operate on huge sparsed file so ENOSPC is expected.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+if [ $((BLK_DEV_SIZE)) -gt 1048576 ]
+then
+ BLK_DEV_SIZE=1048576
+fi
+FS_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test fio activity
+# Run DIO, fallocate and punch_hole threads on a single in parallel
+#
+# If race exist old dio request may rewrite punched block after it was
+# allocated to another file, we will catch that by verifying blocks content
+#
+[global]
+directory=${SCRATCH_MNT}
+filesize=${FS_SIZE}
+size=999G
+continue_on_error=write
+ignore_error=,ENOSPC
+error_dump=0
+
+create_on_open=1
+fallocate=none
+exitall=1
+
+## Perform direct aio, to files which may be truncated
+## by external task
+[direct_aio_raicer]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+bs=128k
+direct=1
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=100*${TIME_FACTOR}
+time_based
+filename=racer
+
+# Run falloc and punch_hole threads in parallel
+# After activity file will be highly fragmented
+[falloc_raicer]
+ioengine=falloc
+runtime=100*${TIME_FACTOR}
+iodepth=1
+bssplit=128k/80:512k/10:32k/10
+rw=randwrite
+numjobs=1
+filename=racer
+
+[punch_hole_raicer]
+ioengine=falloc
+runtime=100*${TIME_FACTOR}
+bs=4k
+time_based=10
+rw=randtrim
+numjobs=2
+filename=racer
+time_based
+
+# Verifier thread continiously write to newly allcated blocks
+# and veryfy written content
+[aio-dio-verifier]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+verifysort=1
+direct=1
+bs=4k
+rw=randwrite
+filename=aio-dio-verifier
+EOF
+
+_workout()
+{
+ echo ""
+ echo "Run fio with random aio-dio pattern"
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs_sized $FS_SIZE >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/298.out b/298.out
new file mode 100644
index 0000000..faedc77
--- /dev/null
+++ b/298.out
@@ -0,0 +1,4 @@
+QA output created by 298
+
+Run fio with random aio-dio pattern
+
diff --git a/group b/group
index 2783286..ff893ac 100644
--- a/group
+++ b/group
@@ -420,4 +420,5 @@ stress
294 auto quick
295 auto logprint quick
296 dump auto quick
-297 auto aio enospc rw stress
\ No newline at end of file
+297 auto aio enospc rw stress
+298 auto aio enospc preallocrw stress
\ No newline at end of file
--
1.7.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
@ 2013-02-20 10:42 ` Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
` (10 subsequent siblings)
11 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-ext4, linux-fsdevel, dchinner, Dmitry Monakhov
Perform various regression tests for ext4defrag subsystem
299'th Test1: Defragment file while other task does direct AIO
300'th Test2: Perform defragmentation on file under buffered AIO
while third task does direct AIO to donor file
301'th Test3: Two defrag tasks use common donor file.
302'th Test4: Stress defragmentation. Several threads pefrorm
fragmentation at random position use inplace=1 will
allocate and free blocks inside defrag event improve
load pressure.
This tests are considered dengerous because 300'th and 301'th are known
to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
299 | 126 +++++++++++++++++++++++++++++++++++++++++++++++
299.out | 4 ++
300 | 143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
300.out | 4 ++
301 | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
301.out | 4 ++
302 | 130 ++++++++++++++++++++++++++++++++++++++++++++++++
302.out | 4 ++
common.defrag | 4 +-
group | 6 ++-
10 files changed, 575 insertions(+), 3 deletions(-)
create mode 100755 299
create mode 100644 299.out
create mode 100755 300
create mode 100644 300.out
create mode 100755 301
create mode 100644 301.out
create mode 100755 302
create mode 100644 302.out
diff --git a/299 b/299
new file mode 100755
index 0000000..4a7a98c
--- /dev/null
+++ b/299
@@ -0,0 +1,126 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Defragment file while other task does direct io
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test1
+# Defragment file while other task does direct io
+
+# Continious sequential defrag activity
+[defrag-4k]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test1.def
+filename=test1
+inplace=0
+rw=write
+numjobs=${NUM_JOBS}
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Verifier
+[aio-dio-verifier]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=randwrite
+filename=test1
+runtime=30*${TIME_FACTOR}
+time_based
+EOF
+
+_workout()
+{
+ echo ""
+ echo " Start defragment activity "
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
diff --git a/299.out b/299.out
new file mode 100644
index 0000000..c2b0b4c
--- /dev/null
+++ b/299.out
@@ -0,0 +1,4 @@
+QA output created by 299
+
+ Start defragment activity
+
diff --git a/300 b/300
new file mode 100755
index 0000000..ff24765
--- /dev/null
+++ b/300
@@ -0,0 +1,143 @@
+#! /bin/bash
+# FSQA Test No. 300
+#
+# Ext4 defragmentation stress test
+# Perform defragmentation on file under buffered io
+# while third task does direct io to donor file
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+##########################################
+# Test2
+# Perform defragmentation on file under buffered io
+# while third task does direct io to donor file
+#
+# Continious sequential defrag activity
+[defrag-4k]
+stonewall
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test2.def
+filename=test2
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Run DIO/AIO for donor file
+[donor-file-fuzzer]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+numjobs=${LOAD_FACTOR}
+verify=0
+direct=1
+bs=64k
+rw=randwrite
+filename=test2.def
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Verifier thread
+[aio-dio-verifier]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+buffered=1
+bs=64k
+rw=randrw
+filename=test2
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+ echo ""
+ echo " Start defragment activity "
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/300.out b/300.out
new file mode 100644
index 0000000..926d985
--- /dev/null
+++ b/300.out
@@ -0,0 +1,4 @@
+QA output created by 300
+
+ Start defragment activity
+
diff --git a/301 b/301
new file mode 100755
index 0000000..7700c36
--- /dev/null
+++ b/301
@@ -0,0 +1,153 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Two defrag tasks use common donor file
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 3 files (one donor file and two test files)
+# Reserve space for 4 files in order to avoid ENOSPC
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (3+1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test3
+# Two defrag tasks use common donor file
+[defrag-1]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test3.def
+filename=test31
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[defrag-2]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test3.def
+filename=test32
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-dio-verifier-1]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=write
+filename=test31
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-buffer-verifier-2]
+ioengine=libaio
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+buffered=1
+bs=64k
+rw=randrw
+filename=test32
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+ echo ""
+ echo " Start defragment activity "
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/301.out b/301.out
new file mode 100644
index 0000000..961c3bf
--- /dev/null
+++ b/301.out
@@ -0,0 +1,4 @@
+QA output created by 301
+
+ Start defragment activity
+
diff --git a/302 b/302
new file mode 100755
index 0000000..3bd82a4
--- /dev/null
+++ b/302
@@ -0,0 +1,130 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Several threads pefrorm defragmentatin at random position
+# using 'inplace' mode (allocate and free blocks inside defrag event)
+# which significantly improve load pressure on block allocator.
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test4
+# Stress test defragmentation engine
+# Several threads pefrorm defragmentatin at random position
+# use inplace=1 will allocate and free blocks inside defrag event
+# which highly increase defragmentation
+[defrag-fuzzer]
+ioengine=e4defrag
+iodepth=1
+bs=8k
+donorname=test4.def
+filename=test4
+inplace=1
+rw=randwrite
+numjobs=4*${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-dio-verifier]
+ioengine=libaio
+iodepth=128
+iomem_align=4k
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=write
+filename=test4
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+ echo ""
+ echo " Start defragment activity "
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/302.out b/302.out
new file mode 100644
index 0000000..a80ab81
--- /dev/null
+++ b/302.out
@@ -0,0 +1,4 @@
+QA output created by 302
+
+ Start defragment activity
+
diff --git a/common.defrag b/common.defrag
index ea6c14c..84b1e65 100644
--- a/common.defrag
+++ b/common.defrag
@@ -24,10 +24,10 @@ _require_defrag()
{
case "$FSTYP" in
xfs)
- DEFRAG_PROG=/usr/sbin/xfs_fsr
+ DEFRAG_PROG="`set_prog_path xfs_fsr`"
;;
ext4|ext4dev)
- DEFRAG_PROG=/usr/bin/e4defrag
+ DEFRAG_PROG="`set_prog_path e4defrag`"
;;
btrfs)
DEFRAG_PROG="$BTRFS_UTIL_PROG filesystem defragment"
diff --git a/group b/group
index ff893ac..2fe406b 100644
--- a/group
+++ b/group
@@ -421,4 +421,8 @@ stress
295 auto logprint quick
296 dump auto quick
297 auto aio enospc rw stress
-298 auto aio enospc preallocrw stress
\ No newline at end of file
+298 auto aio enospc preallocrw stress
+299 aio dangerous ioctl rw stress
+300 aio dangerous ioctl rw stress
+301 aio dangerous ioctl rw stress
+302 aio dangerous ioctl rw stress
\ No newline at end of file
--
1.7.1
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
@ 2013-02-20 10:42 ` Dmitry Monakhov
0 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
Perform various regression tests for ext4defrag subsystem
299'th Test1: Defragment file while other task does direct AIO
300'th Test2: Perform defragmentation on file under buffered AIO
while third task does direct AIO to donor file
301'th Test3: Two defrag tasks use common donor file.
302'th Test4: Stress defragmentation. Several threads pefrorm
fragmentation at random position use inplace=1 will
allocate and free blocks inside defrag event improve
load pressure.
This tests are considered dengerous because 300'th and 301'th are known
to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
299 | 126 +++++++++++++++++++++++++++++++++++++++++++++++
299.out | 4 ++
300 | 143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
300.out | 4 ++
301 | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
301.out | 4 ++
302 | 130 ++++++++++++++++++++++++++++++++++++++++++++++++
302.out | 4 ++
common.defrag | 4 +-
group | 6 ++-
10 files changed, 575 insertions(+), 3 deletions(-)
create mode 100755 299
create mode 100644 299.out
create mode 100755 300
create mode 100644 300.out
create mode 100755 301
create mode 100644 301.out
create mode 100755 302
create mode 100644 302.out
diff --git a/299 b/299
new file mode 100755
index 0000000..4a7a98c
--- /dev/null
+++ b/299
@@ -0,0 +1,126 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Defragment file while other task does direct io
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test1
+# Defragment file while other task does direct io
+
+# Continious sequential defrag activity
+[defrag-4k]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test1.def
+filename=test1
+inplace=0
+rw=write
+numjobs=${NUM_JOBS}
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Verifier
+[aio-dio-verifier]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=randwrite
+filename=test1
+runtime=30*${TIME_FACTOR}
+time_based
+EOF
+
+_workout()
+{
+ echo ""
+ echo " Start defragment activity "
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
diff --git a/299.out b/299.out
new file mode 100644
index 0000000..c2b0b4c
--- /dev/null
+++ b/299.out
@@ -0,0 +1,4 @@
+QA output created by 299
+
+ Start defragment activity
+
diff --git a/300 b/300
new file mode 100755
index 0000000..ff24765
--- /dev/null
+++ b/300
@@ -0,0 +1,143 @@
+#! /bin/bash
+# FSQA Test No. 300
+#
+# Ext4 defragmentation stress test
+# Perform defragmentation on file under buffered io
+# while third task does direct io to donor file
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+##########################################
+# Test2
+# Perform defragmentation on file under buffered io
+# while third task does direct io to donor file
+#
+# Continious sequential defrag activity
+[defrag-4k]
+stonewall
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test2.def
+filename=test2
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Run DIO/AIO for donor file
+[donor-file-fuzzer]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+numjobs=${LOAD_FACTOR}
+verify=0
+direct=1
+bs=64k
+rw=randwrite
+filename=test2.def
+runtime=30*${TIME_FACTOR}
+time_based
+
+# Verifier thread
+[aio-dio-verifier]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+buffered=1
+bs=64k
+rw=randrw
+filename=test2
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+ echo ""
+ echo " Start defragment activity "
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/300.out b/300.out
new file mode 100644
index 0000000..926d985
--- /dev/null
+++ b/300.out
@@ -0,0 +1,4 @@
+QA output created by 300
+
+ Start defragment activity
+
diff --git a/301 b/301
new file mode 100755
index 0000000..7700c36
--- /dev/null
+++ b/301
@@ -0,0 +1,153 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Two defrag tasks use common donor file
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 3 files (one donor file and two test files)
+# Reserve space for 4 files in order to avoid ENOSPC
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (3+1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test3
+# Two defrag tasks use common donor file
+[defrag-1]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test3.def
+filename=test31
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[defrag-2]
+ioengine=e4defrag
+iodepth=1
+bs=128k
+donorname=test3.def
+filename=test32
+inplace=0
+rw=write
+numjobs=${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-dio-verifier-1]
+ioengine=libaio
+iodepth=128*${LOAD_FACTOR}
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=write
+filename=test31
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-buffer-verifier-2]
+ioengine=libaio
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+buffered=1
+bs=64k
+rw=randrw
+filename=test32
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+ echo ""
+ echo " Start defragment activity "
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/301.out b/301.out
new file mode 100644
index 0000000..961c3bf
--- /dev/null
+++ b/301.out
@@ -0,0 +1,4 @@
+QA output created by 301
+
+ Start defragment activity
+
diff --git a/302 b/302
new file mode 100755
index 0000000..3bd82a4
--- /dev/null
+++ b/302
@@ -0,0 +1,130 @@
+#! /bin/bash
+# FSQA Test No. 299
+#
+# Ext4 defragmentation stress test
+# Several threads pefrorm defragmentatin at random position
+# using 'inplace' mode (allocate and free blocks inside defrag event)
+# which significantly improve load pressure on block allocator.
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.defrag
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ext4
+_need_to_be_root
+_require_scratch
+_require_defrag
+
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+# We need space for 2 files (test file, and donor one)
+# reserve 30% in order to avoid ENOSPC
+FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
+
+cat >$tmp-$seq.fio <<EOF
+# Common e4defrag regression tests
+[global]
+ioengine=ioe_e4defrag
+iodepth=1
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=999G
+buffered=0
+fadvise_hint=0
+group_reporting
+
+#################################
+# Test4
+# Stress test defragmentation engine
+# Several threads pefrorm defragmentatin at random position
+# use inplace=1 will allocate and free blocks inside defrag event
+# which highly increase defragmentation
+[defrag-fuzzer]
+ioengine=e4defrag
+iodepth=1
+bs=8k
+donorname=test4.def
+filename=test4
+inplace=1
+rw=randwrite
+numjobs=4*${LOAD_FACTOR}
+runtime=30*${TIME_FACTOR}
+time_based
+
+[aio-dio-verifier]
+ioengine=libaio
+iodepth=128
+iomem_align=4k
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=1
+verifysort=1
+direct=1
+bs=64k
+rw=write
+filename=test4
+runtime=30*${TIME_FACTOR}
+time_based
+
+EOF
+
+_workout()
+{
+ echo ""
+ echo " Start defragment activity "
+ echo ""
+ cat $tmp-$seq.fio >> $seq.full
+ run_check $FIO_PROG $tmp-$seq.fio
+}
+
+_require_fio $tmp-$seq.fio
+
+_scratch_mkfs >> $seq.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+ umount $SCRATCH_DEV 2>/dev/null
+ exit
+fi
+
+if ! _scratch_unmount; then
+ echo "failed to umount"
+ status=1
+ exit
+fi
+_check_scratch_fs
+status=$?
+exit
diff --git a/302.out b/302.out
new file mode 100644
index 0000000..a80ab81
--- /dev/null
+++ b/302.out
@@ -0,0 +1,4 @@
+QA output created by 302
+
+ Start defragment activity
+
diff --git a/common.defrag b/common.defrag
index ea6c14c..84b1e65 100644
--- a/common.defrag
+++ b/common.defrag
@@ -24,10 +24,10 @@ _require_defrag()
{
case "$FSTYP" in
xfs)
- DEFRAG_PROG=/usr/sbin/xfs_fsr
+ DEFRAG_PROG="`set_prog_path xfs_fsr`"
;;
ext4|ext4dev)
- DEFRAG_PROG=/usr/bin/e4defrag
+ DEFRAG_PROG="`set_prog_path e4defrag`"
;;
btrfs)
DEFRAG_PROG="$BTRFS_UTIL_PROG filesystem defragment"
diff --git a/group b/group
index ff893ac..2fe406b 100644
--- a/group
+++ b/group
@@ -421,4 +421,8 @@ stress
295 auto logprint quick
296 dump auto quick
297 auto aio enospc rw stress
-298 auto aio enospc preallocrw stress
\ No newline at end of file
+298 auto aio enospc preallocrw stress
+299 aio dangerous ioctl rw stress
+300 aio dangerous ioctl rw stress
+301 aio dangerous ioctl rw stress
+302 aio dangerous ioctl rw stress
\ No newline at end of file
--
1.7.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 10/10] xfstests: add disk failure simulation test
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
@ 2013-02-20 10:42 ` Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
` (10 subsequent siblings)
11 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-ext4, linux-fsdevel, dchinner, Dmitry Monakhov
There are many situations where disk may fail for example
1) brutal usb dongle unplug
2) iscsi (or any other netbdev) failure due to network issues
In this situation filesystem which use this blockdevice is
expected to fail(force RO remount, abort, etc) but whole system
should still be operational. In other words:
1) Kernel should not panic
2) Memory should not leak
3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
for affected filesystem
4) It should be possible to umount broken filesystem
Later when disk becomes available again we expect(only for journaled filesystems):
5) It will be possible to mount filesystem w/o explicit fsck (in order to caught
issues like https://patchwork.kernel.org/patch/1983981/)
6) Filesystem should be operational
7) After mount/umount has being done all errors should be fixed so fsck should
not spot any issues.
This test use fault enjection (CONFIG_FAIL_MAKE_REQUEST=y config option )
which force all new IO requests to fail for a given device. Xfs already has
XFS_IOC_GOINGDOWN ioctl which provides similar behaviour, but it is fsspeciffic
and it does it in an easy way because it perform freeze_bdev() before actual
shotdown.
Test run fsstress in background and then force disk failure.
Once disk failed it check that (1)-(4) is true.
Then makes disk available again and check that (5)-(7) is also true
BE CAREFUL!! test known to cause memory corruption for XFS
see: https://gist.github.com/dmonakhov/4953045
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
303 | 186 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
303.out | 9 +++
common.config | 1 +
common.rc | 13 ++++
group | 3 +-
5 files changed, 211 insertions(+), 1 deletions(-)
create mode 100755 303
create mode 100644 303.out
diff --git a/303 b/303
new file mode 100755
index 0000000..01bc5f1
--- /dev/null
+++ b/303
@@ -0,0 +1,186 @@
+#! /bin/bash
+# FSQA Test No. 303
+#
+# Run fsstress and fio(dio/aio and mmap) and simulate disk failure
+# check filesystem consistency at the end.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+_supported_fs ext3 ext4 xfs btrfs reiserfs
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+_require_fail_make_request
+
+# TODO: Function are common enough to be moved to common.blkdev
+SCRATCH_REAL_DEV=`readlink -f $SCRATCH_DEV`
+SCRATCH_BDEV=`basename $SCRATCH_REAL_DEV`
+
+allow_fail_make_request()
+{
+ echo "Allow global fail_make_request feature"
+ echo 100 > $DEBUGFS_MNT/fail_make_request/probability
+ echo 9999999 > $DEBUGFS_MNT/fail_make_request/times
+ echo 0 > /sys/kernel/debug/fail_make_request/verbose
+}
+
+disallow_fail_make_request()
+{
+ echo "Disallow global fail_make_request feature"
+ echo 0 > $DEBUGFS_MNT/fail_make_request/probability
+ echo 0 > $DEBUGFS_MNT/fail_make_request/times
+}
+
+start_fail_scratch_dev()
+{
+ echo "Force SCRATCH_DEV device failure"
+ echo " echo 1 > /sys/block/$SCRATCH_BDEV/make-it-fail" >> $here/$seq.full
+ echo 1 > /sys/block/$SCRATCH_BDEV/make-it-fail
+
+}
+
+stop_fail_scratch_dev()
+{
+ echo "Make SCRATCH_DEV device operable again"
+ echo " echo 0 > /sys/block/$SCRATCH_BDEV/make-it-fail" >> $here/$seq.full
+ echo 0 > /sys/block/$SCRATCH_BDEV/make-it-fail
+
+}
+
+_cleanup()
+{
+ poweron_scratch_dev
+ disallow_fail_make_request
+}
+trap "_cleanup; exit \$status" 1 2 3 15
+
+RUN_TIME=$((20+10*$TIME_FACTOR))
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test's fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=4k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=9999T
+continue_on_error=write
+ignore_error=EIO,ENOSPC:EIO
+error_dump=0
+
+[stress_dio_aio_activity]
+create_on_open=1
+fallocate=none
+iodepth=128*${LOAD_FACTOR}
+direct=1
+buffered=0
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=40+${RUN_TIME}
+time_based
+
+[stress_mmap_activity]
+ioengine=mmap
+create_on_open=0
+fallocate=1
+fdatasync=40960
+filesize=8M
+size=9999T
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=40+${RUN_TIME}
+time_based
+
+EOF
+
+_require_fio $tmp-$seq.fio
+
+# Disable all sync operations to get higher load
+FSSTRESS_AVOID="$FSSTRESS_AVOID -ffsync=0 -fsync=0 -ffdatasync=0 -f setattr=1"
+
+_workout()
+{
+ out=$SCRATCH_MNT/fsstress.$$
+ args=`_scale_fsstress_args -p 1 -n999999999 -f setattr=0 $FSSTRESS_AVOID -d $out`
+ echo ""
+ echo "Start fsstress.."
+ echo ""
+ echo "fsstress $args" >> $here/$seq.full
+ $FSSTRESS_PROG $args > /dev/null 2>&1 &
+ fs_pid=$!
+ echo "Start fio.."
+ cat $tmp-$seq.fio >> $seq.full
+ $FIO_PROG $tmp-$seq.fio >> $here/$seq.full 2>&1 &
+ fio_pid=$!
+
+ # Let's it work for awhile, and force device failure
+ sleep $RUN_TIME
+ start_fail_scratch_dev
+ # After device turns in to failed state filesystem may yet not know about
+ # that so buffered write(2) may succeed, but any integrity operations
+ # such as (sync, fsync, fdatasync, direct-io) should fail.
+ dd if=/dev/zero of=$SCRATCH_MNT/touch_failed_filesystem count=1 bs=4k conv=fsync \
+ >> $here/$seq.full 2>&1 && \
+ _fail "failed: still able to perform integrity fsync on $SCRATCH_MNT"
+
+ kill $fs_pid
+ wait $fs_pid
+ wait $fio_pid
+
+ # We expect that broken FS still can be umounted
+ run_check umount $SCRATCH_DEV
+ # Once filesystem was umounted no one is able to write to block device
+ # It is now safe to bring device back to normal state
+ stop_fail_scratch_dev
+
+ # In order to check that filesystem is able to recover journal on mount(2)
+ # perform mount/umount, after that all errors should be fixed
+ run_check _scratch_mount
+ run_check _scratch_unmount
+ _check_scratch_fs
+}
+
+# real QA test starts here
+
+_scratch_mkfs >> $here/$seq.full 2>&1 || _fail "mkfs failed"
+_scratch_mount || _fail "mount failed"
+allow_fail_make_request
+_workout
+status=$?
+disallow_fail_make_request
+exit
diff --git a/303.out b/303.out
new file mode 100644
index 0000000..a936471
--- /dev/null
+++ b/303.out
@@ -0,0 +1,9 @@
+QA output created by 303
+Allow global fail_make_request feature
+
+Start fsstress..
+
+Start fio..
+Force SCRATCH_DEV device failure
+Make SCRATCH_DEV device operable again
+Disallow global fail_make_request feature
diff --git a/common.config b/common.config
index 8efc7f8..8643c0b 100644
--- a/common.config
+++ b/common.config
@@ -75,6 +75,7 @@ export BENCH_PASSES=${BENCH_PASSES:=5}
export XFS_MKFS_OPTIONS=${XFS_MKFS_OPTIONS:=-bsize=4096}
export TIME_FACTOR=${TIME_FACTOR:=1}
export LOAD_FACTOR=${LOAD_FACTOR:=1}
+export DEBUGFS_MNT=${DEBUGFS_MNT:="/sys/kernel/debug"}
export PWD=`pwd`
#export DEBUG=${DEBUG:=...} # arbitrary CFLAGS really.
diff --git a/common.rc b/common.rc
index e7ae9bf..b795c22 100644
--- a/common.rc
+++ b/common.rc
@@ -1049,6 +1049,19 @@ _require_sparse_files()
esac
}
+_require_debugfs()
+{
+ #boot_params always present in debugfs
+ [ -d "$DEBUGFS_MNT/boot_params" ] || _notrun "Debugfs not mounted"
+}
+
+_require_fail_make_request()
+{
+ [ -f "$DEBUGFS_MNT/fail_make_request/probability" ] \
+ || _notrun "$DEBUGFS_MNT/fail_make_request \
+ not found. Seems that CONFIG_FAIL_MAKE_REQUEST kernel config option not enabled"
+}
+
# check that a FS on a device is mounted
# if so, return mount point
#
diff --git a/group b/group
index 2fe406b..20b7a9c 100644
--- a/group
+++ b/group
@@ -425,4 +425,5 @@ stress
299 aio dangerous ioctl rw stress
300 aio dangerous ioctl rw stress
301 aio dangerous ioctl rw stress
-302 aio dangerous ioctl rw stress
\ No newline at end of file
+302 aio dangerous ioctl rw stress
+303 aio dangerous enospc rw stress
\ No newline at end of file
--
1.7.1
^ permalink raw reply related [flat|nested] 53+ messages in thread
* [PATCH 10/10] xfstests: add disk failure simulation test
@ 2013-02-20 10:42 ` Dmitry Monakhov
0 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-02-20 10:42 UTC (permalink / raw)
To: xfs; +Cc: linux-fsdevel, Dmitry Monakhov, linux-ext4, dchinner
There are many situations where disk may fail for example
1) brutal usb dongle unplug
2) iscsi (or any other netbdev) failure due to network issues
In this situation filesystem which use this blockdevice is
expected to fail(force RO remount, abort, etc) but whole system
should still be operational. In other words:
1) Kernel should not panic
2) Memory should not leak
3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
for affected filesystem
4) It should be possible to umount broken filesystem
Later when disk becomes available again we expect(only for journaled filesystems):
5) It will be possible to mount filesystem w/o explicit fsck (in order to caught
issues like https://patchwork.kernel.org/patch/1983981/)
6) Filesystem should be operational
7) After mount/umount has being done all errors should be fixed so fsck should
not spot any issues.
This test use fault enjection (CONFIG_FAIL_MAKE_REQUEST=y config option )
which force all new IO requests to fail for a given device. Xfs already has
XFS_IOC_GOINGDOWN ioctl which provides similar behaviour, but it is fsspeciffic
and it does it in an easy way because it perform freeze_bdev() before actual
shotdown.
Test run fsstress in background and then force disk failure.
Once disk failed it check that (1)-(4) is true.
Then makes disk available again and check that (5)-(7) is also true
BE CAREFUL!! test known to cause memory corruption for XFS
see: https://gist.github.com/dmonakhov/4953045
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
303 | 186 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
303.out | 9 +++
common.config | 1 +
common.rc | 13 ++++
group | 3 +-
5 files changed, 211 insertions(+), 1 deletions(-)
create mode 100755 303
create mode 100644 303.out
diff --git a/303 b/303
new file mode 100755
index 0000000..01bc5f1
--- /dev/null
+++ b/303
@@ -0,0 +1,186 @@
+#! /bin/bash
+# FSQA Test No. 303
+#
+# Run fsstress and fio(dio/aio and mmap) and simulate disk failure
+# check filesystem consistency at the end.
+#
+#-----------------------------------------------------------------------
+# (c) 2013 Dmitry Monakhov
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#-----------------------------------------------------------------------
+#
+# creator
+owner=dmonakhov@openvz.org
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+_supported_fs ext3 ext4 xfs btrfs reiserfs
+_supported_os Linux
+_need_to_be_root
+_require_scratch
+_require_fail_make_request
+
+# TODO: Function are common enough to be moved to common.blkdev
+SCRATCH_REAL_DEV=`readlink -f $SCRATCH_DEV`
+SCRATCH_BDEV=`basename $SCRATCH_REAL_DEV`
+
+allow_fail_make_request()
+{
+ echo "Allow global fail_make_request feature"
+ echo 100 > $DEBUGFS_MNT/fail_make_request/probability
+ echo 9999999 > $DEBUGFS_MNT/fail_make_request/times
+ echo 0 > /sys/kernel/debug/fail_make_request/verbose
+}
+
+disallow_fail_make_request()
+{
+ echo "Disallow global fail_make_request feature"
+ echo 0 > $DEBUGFS_MNT/fail_make_request/probability
+ echo 0 > $DEBUGFS_MNT/fail_make_request/times
+}
+
+start_fail_scratch_dev()
+{
+ echo "Force SCRATCH_DEV device failure"
+ echo " echo 1 > /sys/block/$SCRATCH_BDEV/make-it-fail" >> $here/$seq.full
+ echo 1 > /sys/block/$SCRATCH_BDEV/make-it-fail
+
+}
+
+stop_fail_scratch_dev()
+{
+ echo "Make SCRATCH_DEV device operable again"
+ echo " echo 0 > /sys/block/$SCRATCH_BDEV/make-it-fail" >> $here/$seq.full
+ echo 0 > /sys/block/$SCRATCH_BDEV/make-it-fail
+
+}
+
+_cleanup()
+{
+ poweron_scratch_dev
+ disallow_fail_make_request
+}
+trap "_cleanup; exit \$status" 1 2 3 15
+
+RUN_TIME=$((20+10*$TIME_FACTOR))
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$((BLK_DEV_SIZE * 512))
+
+cat >$tmp-$seq.fio <<EOF
+###########
+# $seq test's fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=4k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=9999T
+continue_on_error=write
+ignore_error=EIO,ENOSPC:EIO
+error_dump=0
+
+[stress_dio_aio_activity]
+create_on_open=1
+fallocate=none
+iodepth=128*${LOAD_FACTOR}
+direct=1
+buffered=0
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=40+${RUN_TIME}
+time_based
+
+[stress_mmap_activity]
+ioengine=mmap
+create_on_open=0
+fallocate=1
+fdatasync=40960
+filesize=8M
+size=9999T
+numjobs=${NUM_JOBS}
+rw=randwrite
+runtime=40+${RUN_TIME}
+time_based
+
+EOF
+
+_require_fio $tmp-$seq.fio
+
+# Disable all sync operations to get higher load
+FSSTRESS_AVOID="$FSSTRESS_AVOID -ffsync=0 -fsync=0 -ffdatasync=0 -f setattr=1"
+
+_workout()
+{
+ out=$SCRATCH_MNT/fsstress.$$
+ args=`_scale_fsstress_args -p 1 -n999999999 -f setattr=0 $FSSTRESS_AVOID -d $out`
+ echo ""
+ echo "Start fsstress.."
+ echo ""
+ echo "fsstress $args" >> $here/$seq.full
+ $FSSTRESS_PROG $args > /dev/null 2>&1 &
+ fs_pid=$!
+ echo "Start fio.."
+ cat $tmp-$seq.fio >> $seq.full
+ $FIO_PROG $tmp-$seq.fio >> $here/$seq.full 2>&1 &
+ fio_pid=$!
+
+ # Let's it work for awhile, and force device failure
+ sleep $RUN_TIME
+ start_fail_scratch_dev
+ # After device turns in to failed state filesystem may yet not know about
+ # that so buffered write(2) may succeed, but any integrity operations
+ # such as (sync, fsync, fdatasync, direct-io) should fail.
+ dd if=/dev/zero of=$SCRATCH_MNT/touch_failed_filesystem count=1 bs=4k conv=fsync \
+ >> $here/$seq.full 2>&1 && \
+ _fail "failed: still able to perform integrity fsync on $SCRATCH_MNT"
+
+ kill $fs_pid
+ wait $fs_pid
+ wait $fio_pid
+
+ # We expect that broken FS still can be umounted
+ run_check umount $SCRATCH_DEV
+ # Once filesystem was umounted no one is able to write to block device
+ # It is now safe to bring device back to normal state
+ stop_fail_scratch_dev
+
+ # In order to check that filesystem is able to recover journal on mount(2)
+ # perform mount/umount, after that all errors should be fixed
+ run_check _scratch_mount
+ run_check _scratch_unmount
+ _check_scratch_fs
+}
+
+# real QA test starts here
+
+_scratch_mkfs >> $here/$seq.full 2>&1 || _fail "mkfs failed"
+_scratch_mount || _fail "mount failed"
+allow_fail_make_request
+_workout
+status=$?
+disallow_fail_make_request
+exit
diff --git a/303.out b/303.out
new file mode 100644
index 0000000..a936471
--- /dev/null
+++ b/303.out
@@ -0,0 +1,9 @@
+QA output created by 303
+Allow global fail_make_request feature
+
+Start fsstress..
+
+Start fio..
+Force SCRATCH_DEV device failure
+Make SCRATCH_DEV device operable again
+Disallow global fail_make_request feature
diff --git a/common.config b/common.config
index 8efc7f8..8643c0b 100644
--- a/common.config
+++ b/common.config
@@ -75,6 +75,7 @@ export BENCH_PASSES=${BENCH_PASSES:=5}
export XFS_MKFS_OPTIONS=${XFS_MKFS_OPTIONS:=-bsize=4096}
export TIME_FACTOR=${TIME_FACTOR:=1}
export LOAD_FACTOR=${LOAD_FACTOR:=1}
+export DEBUGFS_MNT=${DEBUGFS_MNT:="/sys/kernel/debug"}
export PWD=`pwd`
#export DEBUG=${DEBUG:=...} # arbitrary CFLAGS really.
diff --git a/common.rc b/common.rc
index e7ae9bf..b795c22 100644
--- a/common.rc
+++ b/common.rc
@@ -1049,6 +1049,19 @@ _require_sparse_files()
esac
}
+_require_debugfs()
+{
+ #boot_params always present in debugfs
+ [ -d "$DEBUGFS_MNT/boot_params" ] || _notrun "Debugfs not mounted"
+}
+
+_require_fail_make_request()
+{
+ [ -f "$DEBUGFS_MNT/fail_make_request/probability" ] \
+ || _notrun "$DEBUGFS_MNT/fail_make_request \
+ not found. Seems that CONFIG_FAIL_MAKE_REQUEST kernel config option not enabled"
+}
+
# check that a FS on a device is mounted
# if so, return mount point
#
diff --git a/group b/group
index 2fe406b..20b7a9c 100644
--- a/group
+++ b/group
@@ -425,4 +425,5 @@ stress
299 aio dangerous ioctl rw stress
300 aio dangerous ioctl rw stress
301 aio dangerous ioctl rw stress
-302 aio dangerous ioctl rw stress
\ No newline at end of file
+302 aio dangerous ioctl rw stress
+303 aio dangerous enospc rw stress
\ No newline at end of file
--
1.7.1
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply related [flat|nested] 53+ messages in thread
* Re: [PATCH 00/10] xfstests: Stress tests improments v5
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
@ 2013-02-25 15:45 ` Theodore Ts'o
2013-02-20 10:42 ` Dmitry Monakhov
` (10 subsequent siblings)
11 siblings, 0 replies; 53+ messages in thread
From: Theodore Ts'o @ 2013-02-25 15:45 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: xfs, linux-ext4, linux-fsdevel, dchinner
Hi, what's the status of this patch series? It looks like Dmitry has
responded to all of Dave's concerns --- is that correct?
Is there anything else we need to do before these patches get merged
into the xfstests git tree?
Thanks!!
- Ted
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 00/10] xfstests: Stress tests improments v5
@ 2013-02-25 15:45 ` Theodore Ts'o
0 siblings, 0 replies; 53+ messages in thread
From: Theodore Ts'o @ 2013-02-25 15:45 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
Hi, what's the status of this patch series? It looks like Dmitry has
responded to all of Dave's concerns --- is that correct?
Is there anything else we need to do before these patches get merged
into the xfstests git tree?
Thanks!!
- Ted
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 00/10] xfstests: Stress tests improments v5
2013-02-25 15:45 ` Theodore Ts'o
@ 2013-02-25 21:28 ` Ben Myers
-1 siblings, 0 replies; 53+ messages in thread
From: Ben Myers @ 2013-02-25 21:28 UTC (permalink / raw)
To: Theodore Ts'o
Cc: Dmitry Monakhov, linux-fsdevel, linux-ext4, dchinner, xfs
Hey Ted,
On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> Hi, what's the status of this patch series? It looks like Dmitry has
> responded to all of Dave's concerns --- is that correct?
>
> Is there anything else we need to do before these patches get merged
> into the xfstests git tree?
I think we just need a review to see if Dave's concerns have been addressed and
then we're ready to go.
Regards,
Ben
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 00/10] xfstests: Stress tests improments v5
@ 2013-02-25 21:28 ` Ben Myers
0 siblings, 0 replies; 53+ messages in thread
From: Ben Myers @ 2013-02-25 21:28 UTC (permalink / raw)
To: Theodore Ts'o
Cc: linux-fsdevel, linux-ext4, xfs, Dmitry Monakhov, dchinner
Hey Ted,
On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> Hi, what's the status of this patch series? It looks like Dmitry has
> responded to all of Dave's concerns --- is that correct?
>
> Is there anything else we need to do before these patches get merged
> into the xfstests git tree?
I think we just need a review to see if Dave's concerns have been addressed and
then we're ready to go.
Regards,
Ben
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 00/10] xfstests: Stress tests improments v5
2013-02-25 21:28 ` Ben Myers
@ 2013-02-25 22:09 ` Dave Chinner
-1 siblings, 0 replies; 53+ messages in thread
From: Dave Chinner @ 2013-02-25 22:09 UTC (permalink / raw)
To: Ben Myers
Cc: Theodore Ts'o, linux-fsdevel, linux-ext4, xfs,
Dmitry Monakhov, dchinner
On Mon, Feb 25, 2013 at 03:28:11PM -0600, Ben Myers wrote:
> Hey Ted,
>
> On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> > Hi, what's the status of this patch series? It looks like Dmitry has
> > responded to all of Dave's concerns --- is that correct?
> >
> > Is there anything else we need to do before these patches get merged
> > into the xfstests git tree?
>
> I think we just need a review to see if Dave's concerns have been addressed and
> then we're ready to go.
That doesn't stop anyone else from reviewing it. I've got my hands
full right now, and doing stuff like reviewing patches that
nobody else seems to want to look at is something that I'm doing
when I have a little spare time.
I've already reviewed al the patches and the changes have been made,
so all my final review is going to be is validating the patches
apply and work as advertised, so anyone really can do that. I
couldn't do that last time around because the patches didn't apply
to my tree. Since then I haven't had a slot available to go back and
look at it.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 00/10] xfstests: Stress tests improments v5
@ 2013-02-25 22:09 ` Dave Chinner
0 siblings, 0 replies; 53+ messages in thread
From: Dave Chinner @ 2013-02-25 22:09 UTC (permalink / raw)
To: Ben Myers
Cc: Theodore Ts'o, xfs, Dmitry Monakhov, dchinner, linux-fsdevel,
linux-ext4
On Mon, Feb 25, 2013 at 03:28:11PM -0600, Ben Myers wrote:
> Hey Ted,
>
> On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> > Hi, what's the status of this patch series? It looks like Dmitry has
> > responded to all of Dave's concerns --- is that correct?
> >
> > Is there anything else we need to do before these patches get merged
> > into the xfstests git tree?
>
> I think we just need a review to see if Dave's concerns have been addressed and
> then we're ready to go.
That doesn't stop anyone else from reviewing it. I've got my hands
full right now, and doing stuff like reviewing patches that
nobody else seems to want to look at is something that I'm doing
when I have a little spare time.
I've already reviewed al the patches and the changes have been made,
so all my final review is going to be is validating the patches
apply and work as advertised, so anyone really can do that. I
couldn't do that last time around because the patches didn't apply
to my tree. Since then I haven't had a slot available to go back and
look at it.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 00/10] xfstests: Stress tests improments v5
2013-02-25 22:09 ` Dave Chinner
@ 2013-02-25 23:00 ` Ben Myers
-1 siblings, 0 replies; 53+ messages in thread
From: Ben Myers @ 2013-02-25 23:00 UTC (permalink / raw)
To: Dave Chinner
Cc: Theodore Ts'o, linux-fsdevel, linux-ext4, xfs,
Dmitry Monakhov, dchinner
Hi,
On Tue, Feb 26, 2013 at 09:09:41AM +1100, Dave Chinner wrote:
> On Mon, Feb 25, 2013 at 03:28:11PM -0600, Ben Myers wrote:
> > On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> > > Hi, what's the status of this patch series? It looks like Dmitry has
> > > responded to all of Dave's concerns --- is that correct?
> > >
> > > Is there anything else we need to do before these patches get merged
> > > into the xfstests git tree?
> >
> > I think we just need a review to see if Dave's concerns have been addressed and
> > then we're ready to go.
>
> That doesn't stop anyone else from reviewing it.
Yes. Anyone is welcome to extend a Reviewed-by. ;)
-Ben
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 00/10] xfstests: Stress tests improments v5
@ 2013-02-25 23:00 ` Ben Myers
0 siblings, 0 replies; 53+ messages in thread
From: Ben Myers @ 2013-02-25 23:00 UTC (permalink / raw)
To: Dave Chinner
Cc: Theodore Ts'o, xfs, Dmitry Monakhov, dchinner, linux-fsdevel,
linux-ext4
Hi,
On Tue, Feb 26, 2013 at 09:09:41AM +1100, Dave Chinner wrote:
> On Mon, Feb 25, 2013 at 03:28:11PM -0600, Ben Myers wrote:
> > On Mon, Feb 25, 2013 at 10:45:12AM -0500, Theodore Ts'o wrote:
> > > Hi, what's the status of this patch series? It looks like Dmitry has
> > > responded to all of Dave's concerns --- is that correct?
> > >
> > > Is there anything else we need to do before these patches get merged
> > > into the xfstests git tree?
> >
> > I think we just need a review to see if Dave's concerns have been addressed and
> > then we're ready to go.
>
> That doesn't stop anyone else from reviewing it.
Yes. Anyone is welcome to extend a Reviewed-by. ;)
-Ben
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 01/10] xfstests: add fio requirement V2
2013-02-20 10:42 ` Dmitry Monakhov
@ 2013-03-01 15:43 ` Rich Johnston
-1 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 15:43 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: xfs, linux-fsdevel, linux-ext4, dchinner
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> FIO is very flexible io generator, I would call it IO swiss knife.
> Currently we have tonns of hardcoded application which reproduces
> some predefined scenario. This approach has obvious dissadvantages
> 1) Lack of flexability: one written it is hard to modify it in future
> 2) Code base is large, many routines written again and again
>
> At the same time add new fio based tast is just add simle INI file.
> This greatly simplify code review. I do beleve that some day we will
> replace most of hardcoded io binaries with fio.
>
> One who is planning to run $FIO_PROG should first check that system
> contains appropriate version which is able to handle jobfile
> for example: _require_fio 286-job.fio
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Minor typos in the description, but looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 01/10] xfstests: add fio requirement V2
@ 2013-03-01 15:43 ` Rich Johnston
0 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 15:43 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> FIO is very flexible io generator, I would call it IO swiss knife.
> Currently we have tonns of hardcoded application which reproduces
> some predefined scenario. This approach has obvious dissadvantages
> 1) Lack of flexability: one written it is hard to modify it in future
> 2) Code base is large, many routines written again and again
>
> At the same time add new fio based tast is just add simle INI file.
> This greatly simplify code review. I do beleve that some day we will
> replace most of hardcoded io binaries with fio.
>
> One who is planning to run $FIO_PROG should first check that system
> contains appropriate version which is able to handle jobfile
> for example: _require_fio 286-job.fio
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Minor typos in the description, but looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 02/10] xfstest: add configurable load factors
2013-02-20 10:42 ` Dmitry Monakhov
(?)
@ 2013-03-01 15:43 ` Rich Johnston
-1 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 15:43 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Most stress test has probable behauviour, the longer test run the
> larger corner cases will be cover. It is reasonable to allow
> user to provide some sort of system load factor.
> This patch introduce two global variables
> LOAD_FACTOR: Usually means factor number of running tasks
> TIME_FACTOR: Usually means factor of run time, or number of operations
> If not speficied both variables defined to 1, so original behaviour
> preserved.
>
> TODO: Change all stress tests to use this variables
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Nice addition, looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 03/10] xfstests: hardcode fops for determinable fsstests runs
2013-02-20 10:42 ` Dmitry Monakhov
@ 2013-03-01 15:43 ` Rich Johnston
-1 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 15:43 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: xfs, linux-fsdevel, linux-ext4, dchinner
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> 106,107 and 117 are frozen tests which use known seed, it is
> reasonable to explicitly hardcode file operations in order to avoid
> implicit changes caused by future changes in fsstress.
>
> NOTE: options genereted like follows: fsstress -S c $ORIG_ARGS
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
106 and 107 are not valid for ext4. They failed the same before and
after this patch with xfs. Because 106 and 107 are not in the auto group
and 117 is in the auto group and passes for both xfs and ext4,
I will say this looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
> ---
> 106 | 33 +++++++++++++++++++++++++++++++--
> 107 | 32 +++++++++++++++++++++++++++++---
> 2 files changed, 60 insertions(+), 5 deletions(-)
>
> diff --git a/106 b/106
> index 8278691..b351fe5 100755
> --- a/106
> +++ b/106
> @@ -61,8 +61,37 @@ _require_prjquota $SCRATCH_DEV
>
> # initial populate
> target=$SCRATCH_MNT/target
> -$FSSTRESS_PROG -s 0xdeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
> -$FSSTRESS_PROG -s 0xdeed -m8 -z -p4 -n1000 -fsetxattr=500 -fchown=500 -d $target
> +$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
> +-f allocsp=1 \
> +-f chown=3 \
> +-f creat=4 \
> +-f dwrite=4 \
> +-f fallocate=1 \
> +-f fdatasync=1 \
> +-f fiemap=1 \
> +-f freesp=1 \
> +-f fsync=1 \
> +-f link=1 \
> +-f mkdir=2 \
> +-f mknod=2 \
> +-f punch=1 \
> +-f rename=2 \
> +-f resvsp=1 \
> +-f rmdir=1 \
> +-f setxattr=1 \
> +-f symlink=2 \
> +-f sync=1 \
> +-f truncate=2 \
> +-f unlink=1 \
> +-f unresvsp=1 \
> +-f write=4 \
> +-d $target
> +
> +$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
> +-f chown=500 \
> +-f setxattr=500 \
> +-d $target
> +
>
> # also use space, to be able to go over/under limits easily
> uid=255
> diff --git a/107 b/107
> index 74403e6..eaf7e6a 100755
> --- a/107
> +++ b/107
> @@ -78,9 +78,35 @@ echo "6:$target" | tee -a $seq.full > $tmp.projects
>
> echo "### populate filesystem"
> mkdir $target || exit
> -FSSTRESS_AVOID="$FSSTRESS_AVOID -fmknod=0 -fsymlink=0"
> -$FSSTRESS_PROG -s 0xfeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
> -$FSSTRESS_PROG -s 0xbabe -m8 -z -p4 -n500 -fsetxattr=250 -fchown=250 -d $target
> +$FSSTRESS_PROG -z -s 65261 -m 8 -n 1000 -p 4 \
> +-f allocsp=1 \
> +-f chown=3 \
> +-f creat=4 \
> +-f dwrite=4 \
> +-f fallocate=1 \
> +-f fdatasync=1 \
> +-f fiemap=1 \
> +-f freesp=1 \
> +-f fsync=1 \
> +-f link=1 \
> +-f mkdir=2 \
> +-f punch=1 \
> +-f rename=2 \
> +-f resvsp=1 \
> +-f rmdir=1 \
> +-f setxattr=1 \
> +-f sync=1 \
> +-f truncate=2 \
> +-f unlink=1 \
> +-f unresvsp=1 \
> +-f write=4 \
> +-d $target
> +
> +$FSSTRESS_PROG -z -s 47806 -m 8 -n 500 -p 4 \
> +-f chown=250 \
> +-f setxattr=250 \
> +-d $target
> +
>
> QARGS="-x -D $tmp.projects -P /dev/null $SCRATCH_MNT"
>
>
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 03/10] xfstests: hardcode fops for determinable fsstests runs
@ 2013-03-01 15:43 ` Rich Johnston
0 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 15:43 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> 106,107 and 117 are frozen tests which use known seed, it is
> reasonable to explicitly hardcode file operations in order to avoid
> implicit changes caused by future changes in fsstress.
>
> NOTE: options genereted like follows: fsstress -S c $ORIG_ARGS
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
106 and 107 are not valid for ext4. They failed the same before and
after this patch with xfs. Because 106 and 107 are not in the auto group
and 117 is in the auto group and passes for both xfs and ext4,
I will say this looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
> ---
> 106 | 33 +++++++++++++++++++++++++++++++--
> 107 | 32 +++++++++++++++++++++++++++++---
> 2 files changed, 60 insertions(+), 5 deletions(-)
>
> diff --git a/106 b/106
> index 8278691..b351fe5 100755
> --- a/106
> +++ b/106
> @@ -61,8 +61,37 @@ _require_prjquota $SCRATCH_DEV
>
> # initial populate
> target=$SCRATCH_MNT/target
> -$FSSTRESS_PROG -s 0xdeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
> -$FSSTRESS_PROG -s 0xdeed -m8 -z -p4 -n1000 -fsetxattr=500 -fchown=500 -d $target
> +$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
> +-f allocsp=1 \
> +-f chown=3 \
> +-f creat=4 \
> +-f dwrite=4 \
> +-f fallocate=1 \
> +-f fdatasync=1 \
> +-f fiemap=1 \
> +-f freesp=1 \
> +-f fsync=1 \
> +-f link=1 \
> +-f mkdir=2 \
> +-f mknod=2 \
> +-f punch=1 \
> +-f rename=2 \
> +-f resvsp=1 \
> +-f rmdir=1 \
> +-f setxattr=1 \
> +-f symlink=2 \
> +-f sync=1 \
> +-f truncate=2 \
> +-f unlink=1 \
> +-f unresvsp=1 \
> +-f write=4 \
> +-d $target
> +
> +$FSSTRESS_PROG -z -s 57069 -m 8 -n 1000 -p 4 \
> +-f chown=500 \
> +-f setxattr=500 \
> +-d $target
> +
>
> # also use space, to be able to go over/under limits easily
> uid=255
> diff --git a/107 b/107
> index 74403e6..eaf7e6a 100755
> --- a/107
> +++ b/107
> @@ -78,9 +78,35 @@ echo "6:$target" | tee -a $seq.full > $tmp.projects
>
> echo "### populate filesystem"
> mkdir $target || exit
> -FSSTRESS_AVOID="$FSSTRESS_AVOID -fmknod=0 -fsymlink=0"
> -$FSSTRESS_PROG -s 0xfeed -m8 -w -p4 -n1000 $FSSTRESS_AVOID -d $target
> -$FSSTRESS_PROG -s 0xbabe -m8 -z -p4 -n500 -fsetxattr=250 -fchown=250 -d $target
> +$FSSTRESS_PROG -z -s 65261 -m 8 -n 1000 -p 4 \
> +-f allocsp=1 \
> +-f chown=3 \
> +-f creat=4 \
> +-f dwrite=4 \
> +-f fallocate=1 \
> +-f fdatasync=1 \
> +-f fiemap=1 \
> +-f freesp=1 \
> +-f fsync=1 \
> +-f link=1 \
> +-f mkdir=2 \
> +-f punch=1 \
> +-f rename=2 \
> +-f resvsp=1 \
> +-f rmdir=1 \
> +-f setxattr=1 \
> +-f sync=1 \
> +-f truncate=2 \
> +-f unlink=1 \
> +-f unresvsp=1 \
> +-f write=4 \
> +-d $target
> +
> +$FSSTRESS_PROG -z -s 47806 -m 8 -n 500 -p 4 \
> +-f chown=250 \
> +-f setxattr=250 \
> +-d $target
> +
>
> QARGS="-x -D $tmp.projects -P /dev/null $SCRATCH_MNT"
>
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 04/10] xfstets: fsstress add replace file operation
2013-02-20 10:42 ` [PATCH 04/10] xfstets: fsstress add replace file operation Dmitry Monakhov
@ 2013-03-01 15:43 ` Rich Johnston
0 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 15:43 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> The most common usecase for rename(2) syscall is an atomic replacement
> of existing file with newer version. But rename_f() rename some existing
> filename to newly generated (non existing) filename. As result the most
> important usecase is not covered.
Good catch.
> Since rename_f() is already exist in fsstress and it has known behavior,
> some tests already depends on that behaviour, let's add another operation
> (replace_f) which invoke rename(2) for two existing entries.
>
> OUT_OF_COMMIT_DISCUSSION:
> Off course replace_f() break naming convention where fun_name == syscall_f(),
> but this is the only way I see to introduce new feature and not break
> other tests. May be it is reasonable to call it rename2_f() ?
>
I think this possible exposes a bug which was not exposed by before when
running for example test 076 and test 083 on both ext4 and xfs.
Suggest this new function is called rename2_() so that we don't change
the existing known tests.
Regards
--Rich
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 05/10] xfstest: allow fsstress to use load factor where appropriate
2013-02-20 10:42 ` Dmitry Monakhov
(?)
@ 2013-03-01 15:43 ` Rich Johnston
-1 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 15:43 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> 1) Add _scale_fsstress_args function which transform argumets according
> to load factors
> 2) Let all non deterministic fsstress tests to use scaled arguments
>
> I've able to trigger OOPS on xfs see:https://gist.github.com/dmonakhov/4762653
>
I did not see any OOPS. Looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 06/10] xfstest: move run_check to common.rc
2013-02-20 10:42 ` Dmitry Monakhov
(?)
@ 2013-03-01 15:44 ` Rich Johnston
-1 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 15:44 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
Looks good, will add the following simple description when this is committed
Move run_check to common.rc
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 07/10] xfstest: add fallocate/truncate vs AIO/DIO stress test
2013-02-20 10:42 ` Dmitry Monakhov
@ 2013-03-01 17:49 ` Rich Johnston
-1 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 17:49 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: xfs, linux-fsdevel, linux-ext4, dchinner
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Run DIO, fallocate and truncate threads on a common file in parallel.
> If race exist old dio request may rewrite blocks after it was allocated
> to another file, we will catch that by verifying blocks content.
>
> this patch known to catch deadlock for ext4
> http://lists.openwall.net/linux-ext4/2012/09/06/3
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 07/10] xfstest: add fallocate/truncate vs AIO/DIO stress test
@ 2013-03-01 17:49 ` Rich Johnston
0 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 17:49 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Run DIO, fallocate and truncate threads on a common file in parallel.
> If race exist old dio request may rewrite blocks after it was allocated
> to another file, we will catch that by verifying blocks content.
>
> this patch known to catch deadlock for ext4
> http://lists.openwall.net/linux-ext4/2012/09/06/3
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 08/10] xfstest: add fallocate/punch_hole vs AIO/DIO stress test
2013-02-20 10:42 ` Dmitry Monakhov
@ 2013-03-01 17:50 ` Rich Johnston
-1 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 17:50 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: xfs, linux-fsdevel, linux-ext4, dchinner
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Run random AIO/DIO activity (fio's job:direct_aio_raicer)
> random fallocate activity(fio's job:falloc_raicer)
> and random punch_hole activity(punch_hole_raicer) on a common
> file in parallel. If race exist old dio request may rewrite
> punched block after it was allocated to another file, we will
> catch that by verifier fio's job: "aio-dio-verifier".
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 08/10] xfstest: add fallocate/punch_hole vs AIO/DIO stress test
@ 2013-03-01 17:50 ` Rich Johnston
0 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 17:50 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Run random AIO/DIO activity (fio's job:direct_aio_raicer)
> random fallocate activity(fio's job:falloc_raicer)
> and random punch_hole activity(punch_hole_raicer) on a common
> file in parallel. If race exist old dio request may rewrite
> punched block after it was allocated to another file, we will
> catch that by verifier fio's job: "aio-dio-verifier".
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Looks good.
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
2013-02-20 10:42 ` Dmitry Monakhov
@ 2013-03-01 19:23 ` Rich Johnston
-1 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 19:23 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: xfs, linux-fsdevel, linux-ext4, dchinner
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Perform various regression tests for ext4defrag subsystem
>
> 299'th Test1: Defragment file while other task does direct AIO
> 300'th Test2: Perform defragmentation on file under buffered AIO
> while third task does direct AIO to donor file
> 301'th Test3: Two defrag tasks use common donor file.
> 302'th Test4: Stress defragmentation. Several threads pefrorm
typo perform
> fragmentation at random position use inplace=1 will
> allocate and free blocks inside defrag event improve
> load pressure.
>
> This tests are considered dengerous because 300'th and 301'th are known
typo dangerous
> to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> ---
> 299 | 126 +++++++++++++++++++++++++++++++++++++++++++++++
> 299.out | 4 ++
> 300 | 143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> 300.out | 4 ++
> 301 | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 301.out | 4 ++
> 302 | 130 ++++++++++++++++++++++++++++++++++++++++++++++++
> 302.out | 4 ++
> common.defrag | 4 +-
> group | 6 ++-
> 10 files changed, 575 insertions(+), 3 deletions(-)
> create mode 100755 299
> create mode 100644 299.out
> create mode 100755 300
> create mode 100644 300.out
> create mode 100755 301
> create mode 100644 301.out
> create mode 100755 302
> create mode 100644 302.out
>
> diff --git a/299 b/299
> new file mode 100755
> index 0000000..4a7a98c
> --- /dev/null
> +++ b/299
> @@ -0,0 +1,126 @@
> +#! /bin/bash
> +# FSQA Test No. 299
> +#
> +# Ext4 defragmentation stress test
> +# Defragment file while other task does direct io
> +#-----------------------------------------------------------------------
> +# (c) 2013 Dmitry Monakhov
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#
> +#-----------------------------------------------------------------------
> +#
> +# creator
> +owner=dmonakhov@openvz.org
> +
> +seq=`basename $0`
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common.rc
> +. ./common.filter
> +. ./common.defrag
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs ext4
> +_need_to_be_root
> +_require_scratch
> +_require_defrag
> +
I think you meant to set NUM_JOBS here like you did in [Patch 08/10],
something like:
+NUM_JOBS=$((4*LOAD_FACTOR))
otherwise you get
### [not run] /usr/local/bin/fio too old
> +BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
> +# We need space for 2 files (test file, and donor one)
> +# reserve 30% in order to avoid ENOSPC
> +FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
> +
> +cat >$tmp-$seq.fio <<EOF
> +# Common e4defrag regression tests
> +[global]
> +ioengine=ioe_e4defrag
> +iodepth=1
> +directory=${SCRATCH_MNT}
> +filesize=${FILE_SIZE}
> +size=999G
> +buffered=0
> +fadvise_hint=0
> +group_reporting
> +
> +#################################
> +# Test1
> +# Defragment file while other task does direct io
> +
> +# Continious sequential defrag activity
> +[defrag-4k]
> +ioengine=e4defrag
> +iodepth=1
> +bs=128k
> +donorname=test1.def
> +filename=test1
> +inplace=0
> +rw=write
> +numjobs=${NUM_JOBS}
> +runtime=30*${TIME_FACTOR}
> +time_based
> +
> +# Verifier
> +[aio-dio-verifier]
> +ioengine=libaio
> +iodepth=128*${LOAD_FACTOR}
> +numjobs=1
> +verify=crc32c-intel
> +verify_fatal=1
> +verify_dump=1
> +verify_backlog=1024
> +verify_async=1
> +verifysort=1
> +direct=1
> +bs=64k
> +rw=randwrite
> +filename=test1
> +runtime=30*${TIME_FACTOR}
> +time_based
> +EOF
> +
> +_workout()
> +{
> + echo ""
> + echo " Start defragment activity "
git does not like trailing whitespace ^
(same for the other 3 tests)
> + echo ""
> + cat $tmp-$seq.fio >> $seq.full
> + run_check $FIO_PROG $tmp-$seq.fio
> +}
> +
> +_require_fio $tmp-$seq.fio
> +
> +_scratch_mkfs >> $seq.full 2>&1
> +_scratch_mount
> +
> +if ! _workout; then
> + umount $SCRATCH_DEV 2>/dev/null
> + exit
> +fi
> +
> +if ! _scratch_unmount; then
> + echo "failed to umount"
> + status=1
> + exit
> +fi
> +_check_scratch_fs
> +status=$?
> +exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
> diff --git a/299.out b/299.out
> new file mode 100644
> index 0000000..c2b0b4c
> --- /dev/null
> +++ b/299.out
> @@ -0,0 +1,4 @@
> +QA output created by 299
> +
> + Start defragment activity
remove trailing whitespace ^
(same for the other 3 tests)
Let me know if you agree with these changes and I will make them at
commit time.
Thanks
--Rich
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
@ 2013-03-01 19:23 ` Rich Johnston
0 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 19:23 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> Perform various regression tests for ext4defrag subsystem
>
> 299'th Test1: Defragment file while other task does direct AIO
> 300'th Test2: Perform defragmentation on file under buffered AIO
> while third task does direct AIO to donor file
> 301'th Test3: Two defrag tasks use common donor file.
> 302'th Test4: Stress defragmentation. Several threads pefrorm
typo perform
> fragmentation at random position use inplace=1 will
> allocate and free blocks inside defrag event improve
> load pressure.
>
> This tests are considered dengerous because 300'th and 301'th are known
typo dangerous
> to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
>
> Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> ---
> 299 | 126 +++++++++++++++++++++++++++++++++++++++++++++++
> 299.out | 4 ++
> 300 | 143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> 300.out | 4 ++
> 301 | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 301.out | 4 ++
> 302 | 130 ++++++++++++++++++++++++++++++++++++++++++++++++
> 302.out | 4 ++
> common.defrag | 4 +-
> group | 6 ++-
> 10 files changed, 575 insertions(+), 3 deletions(-)
> create mode 100755 299
> create mode 100644 299.out
> create mode 100755 300
> create mode 100644 300.out
> create mode 100755 301
> create mode 100644 301.out
> create mode 100755 302
> create mode 100644 302.out
>
> diff --git a/299 b/299
> new file mode 100755
> index 0000000..4a7a98c
> --- /dev/null
> +++ b/299
> @@ -0,0 +1,126 @@
> +#! /bin/bash
> +# FSQA Test No. 299
> +#
> +# Ext4 defragmentation stress test
> +# Defragment file while other task does direct io
> +#-----------------------------------------------------------------------
> +# (c) 2013 Dmitry Monakhov
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#
> +#-----------------------------------------------------------------------
> +#
> +# creator
> +owner=dmonakhov@openvz.org
> +
> +seq=`basename $0`
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common.rc
> +. ./common.filter
> +. ./common.defrag
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs ext4
> +_need_to_be_root
> +_require_scratch
> +_require_defrag
> +
I think you meant to set NUM_JOBS here like you did in [Patch 08/10],
something like:
+NUM_JOBS=$((4*LOAD_FACTOR))
otherwise you get
### [not run] /usr/local/bin/fio too old
> +BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
> +# We need space for 2 files (test file, and donor one)
> +# reserve 30% in order to avoid ENOSPC
> +FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
> +
> +cat >$tmp-$seq.fio <<EOF
> +# Common e4defrag regression tests
> +[global]
> +ioengine=ioe_e4defrag
> +iodepth=1
> +directory=${SCRATCH_MNT}
> +filesize=${FILE_SIZE}
> +size=999G
> +buffered=0
> +fadvise_hint=0
> +group_reporting
> +
> +#################################
> +# Test1
> +# Defragment file while other task does direct io
> +
> +# Continious sequential defrag activity
> +[defrag-4k]
> +ioengine=e4defrag
> +iodepth=1
> +bs=128k
> +donorname=test1.def
> +filename=test1
> +inplace=0
> +rw=write
> +numjobs=${NUM_JOBS}
> +runtime=30*${TIME_FACTOR}
> +time_based
> +
> +# Verifier
> +[aio-dio-verifier]
> +ioengine=libaio
> +iodepth=128*${LOAD_FACTOR}
> +numjobs=1
> +verify=crc32c-intel
> +verify_fatal=1
> +verify_dump=1
> +verify_backlog=1024
> +verify_async=1
> +verifysort=1
> +direct=1
> +bs=64k
> +rw=randwrite
> +filename=test1
> +runtime=30*${TIME_FACTOR}
> +time_based
> +EOF
> +
> +_workout()
> +{
> + echo ""
> + echo " Start defragment activity "
git does not like trailing whitespace ^
(same for the other 3 tests)
> + echo ""
> + cat $tmp-$seq.fio >> $seq.full
> + run_check $FIO_PROG $tmp-$seq.fio
> +}
> +
> +_require_fio $tmp-$seq.fio
> +
> +_scratch_mkfs >> $seq.full 2>&1
> +_scratch_mount
> +
> +if ! _workout; then
> + umount $SCRATCH_DEV 2>/dev/null
> + exit
> +fi
> +
> +if ! _scratch_unmount; then
> + echo "failed to umount"
> + status=1
> + exit
> +fi
> +_check_scratch_fs
> +status=$?
> +exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
> diff --git a/299.out b/299.out
> new file mode 100644
> index 0000000..c2b0b4c
> --- /dev/null
> +++ b/299.out
> @@ -0,0 +1,4 @@
> +QA output created by 299
> +
> + Start defragment activity
remove trailing whitespace ^
(same for the other 3 tests)
Let me know if you agree with these changes and I will make them at
commit time.
Thanks
--Rich
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 10/10] xfstests: add disk failure simulation test
2013-02-20 10:42 ` Dmitry Monakhov
(?)
@ 2013-03-01 20:11 ` Rich Johnston
2013-03-02 1:49 ` Dmitry Monakhov
-1 siblings, 1 reply; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 20:11 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> There are many situations where disk may fail for example
> 1) brutal usb dongle unplug
> 2) iscsi (or any other netbdev) failure due to network issues
> In this situation filesystem which use this blockdevice is
> expected to fail(force RO remount, abort, etc) but whole system
> should still be operational. In other words:
> 1) Kernel should not panic
> 2) Memory should not leak
> 3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
> for affected filesystem
> 4) It should be possible to umount broken filesystem
>
> Later when disk becomes available again we expect(only for journaled filesystems):
> 5) It will be possible to mount filesystem w/o explicit fsck (in order to caught
typo s/caught/catch/g
> issues like https://patchwork.kernel.org/patch/1983981/)
> 6) Filesystem should be operational
> 7) After mount/umount has being done all errors should be fixed so fsck should
> not spot any issues.
>
> This test use fault enjection (CONFIG_FAIL_MAKE_REQUEST=y config option )
May want to mention all the kernel config options required.
i.e. CONFIG_FAULT_INJECTION=y ... are there others?
CONFIG_FAULT_INJECTION_DEBUG_FS=y ???
> which force all new IO requests to fail for a given device. Xfs already has
to force
> XFS_IOC_GOINGDOWN ioctl which provides similar behaviour, but it is fs speciffic
typos s/behaviour/behavior/g s/speciffic/specific
> and it does it in an easy way
because it perform freeze_bdev() before actual
> shotdown.
typo s/shotdown/shutdown/g
>
> Test run fsstress in background and then force disk failure.
> Once disk failed it check that (1)-(4) is true.
Once the disk fails, check that (1)-(4) are true.
> Then makes disk available again and check that (5)-(7) is also true
make the disk ... are
>
> BE CAREFUL!! test known to cause memory corruption for XFS
> see: https://gist.github.com/dmonakhov/4953045
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 00/10] xfstests: Stress tests improments v5
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
` (10 preceding siblings ...)
2013-02-25 15:45 ` Theodore Ts'o
@ 2013-03-01 20:23 ` Rich Johnston
11 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-01 20:23 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
Committed patches 1-3 and 5-8.
Waiting for responses back on patches 4, 9, and 10.
Thanks
--Rich
On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> 1 add fio requirement V2
> 2 add configurable load factors
> 3 hardcode fops for determinable fsstests runs
> 4 fsstress add replace file operation
> 5 allow fsstress to use load factor where appropriate
> 6 move run_check to common.rc
> 7 add fallocate/truncate vs AIO/DIO stress test
> 8 add fallocate/punch_hole vs AIO/DIO stress test
> 9 add defragmentation stress tests for ext4
> 10 add disk failure simulation test
>
commit 892125a53aac8e82dec99fe4e824c39513d4a17a
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date: Wed Feb 20 10:42:13 2013 +0000
xfstest: add fallocate/punch_hole vs AIO/DIO stress test
Run random AIO/DIO activity (fio's job:direct_aio_raicer)
random fallocate activity(fio's job:falloc_raicer)
and random punch_hole activity(punch_hole_raicer) on a common
file in parallel. If a race exists, old dio request may rewrite
punched block after it was allocated to another file, we will
catch that by verifier fio's job: "aio-dio-verifier".
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
commit 0f88dc26abf55cee39ede490da08ed0d2960cdb2
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date: Wed Feb 20 10:42:12 2013 +0000
xfstest: add fallocate/truncate vs AIO/DIO stress test
Run DIO, fallocate and truncate threads on a common file in parallel.
If a race exists, the old dio request may rewrite blocks after it was
allocated to another file, we will catch that by verifying blocks
content.
this patch known to catch deadlock for ext4
http://lists.openwall.net/linux-ext4/2012/09/06/3
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
commit bb949015a8d3fcf4a5c105b1edd74f27b848a806
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date: Wed Feb 20 10:42:11 2013 +0000
xfstest: move run_check to common.rc
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
Move run_check to common.rc.
commit b84aade2f49f309c44e6c0b029586c9dcb7b1c22
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date: Wed Feb 20 10:42:10 2013 +0000
xfstest: allow fsstress to use load factor where appropriate
1) Add _scale_fsstress_args function which transform arguments
according
to load factors
2) Let all non deterministic fsstress tests to use scaled arguments
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
commit d14981fa536ac0106dbe9b3d5088a5a50be49f55
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date: Wed Feb 20 10:42:08 2013 +0000
xfstests: hardcode fops for determinable fsstests runs
106,107 and 117 are frozen tests which use known seed, it is
reasonable to explicitly hardcode file operations in order to avoid
implicit changes caused by future changes in fsstress.
NOTE: options generated like follows: fsstress -S c $ORIG_ARGS
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
commit 66a98478e3cffecfbd62d3a124b626ddda447c57
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date: Wed Feb 20 10:42:07 2013 +0000
xfstest: add configurable load factors
Most stress test has probable behavior, the longer test run the
larger corner cases will be cover. It is reasonable to allow
user to provide some sort of system load factor.
This patch introduce two global variables
LOAD_FACTOR: Usually means factor number of running tasks
TIME_FACTOR: Usually means factor of run time, or number of operations
If not specified both variables defined to 1, so original behavior
preserved.
TODO: Change all stress tests to use this variables
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
commit b3db6021fc2df4e99c9e098612446f4c86e70766
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date: Wed Feb 20 10:42:06 2013 +0000
xfstests: add fio requirement V2
FIO is very flexible io generator, I would call it IO swiss knife.
Currently we have tons of hardcoded application which reproduces
some predefined scenario. This approach has obvious disadvantages
1) Lack of flexibility: one written it is hard to modify it in future
2) Code base is large, many routines written again and again
At the same time add new fio based test, just add simple INI file.
This greatly simplifies code review. I do believe that some day we will
replace most of hardcoded io binaries with fio.
One who is planning to run $FIO_PROG should first check that system
contains appropriate version which is able to handle jobfile
for example: _require_fio 286-job.fio
Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
Reviewed-by: Rich Johnston <rjohnston@sgi.com>
Signed-off-by: Rich Johnston <rjohnston@sgi.com>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
2013-03-01 19:23 ` Rich Johnston
@ 2013-03-02 1:30 ` Dmitry Monakhov
-1 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-03-02 1:30 UTC (permalink / raw)
To: Rich Johnston; +Cc: xfs, linux-fsdevel, linux-ext4, dchinner
On Fri, 1 Mar 2013 13:23:50 -0600, Rich Johnston <rjohnston@sgi.com> wrote:
> On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> > Perform various regression tests for ext4defrag subsystem
> >
> > 299'th Test1: Defragment file while other task does direct AIO
> > 300'th Test2: Perform defragmentation on file under buffered AIO
> > while third task does direct AIO to donor file
> > 301'th Test3: Two defrag tasks use common donor file.
> > 302'th Test4: Stress defragmentation. Several threads pefrorm
> typo perform
>
> > fragmentation at random position use inplace=1 will
> > allocate and free blocks inside defrag event improve
> > load pressure.
> >
> > This tests are considered dengerous because 300'th and 301'th are known
> typo dangerous
>
> > to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
> >
> > Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> > ---
> > 299 | 126 +++++++++++++++++++++++++++++++++++++++++++++++
> > 299.out | 4 ++
> > 300 | 143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 300.out | 4 ++
> > 301 | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 301.out | 4 ++
> > 302 | 130 ++++++++++++++++++++++++++++++++++++++++++++++++
> > 302.out | 4 ++
> > common.defrag | 4 +-
> > group | 6 ++-
> > 10 files changed, 575 insertions(+), 3 deletions(-)
> > create mode 100755 299
> > create mode 100644 299.out
> > create mode 100755 300
> > create mode 100644 300.out
> > create mode 100755 301
> > create mode 100644 301.out
> > create mode 100755 302
> > create mode 100644 302.out
> >
> > diff --git a/299 b/299
> > new file mode 100755
> > index 0000000..4a7a98c
> > --- /dev/null
> > +++ b/299
> > @@ -0,0 +1,126 @@
> > +#! /bin/bash
> > +# FSQA Test No. 299
> > +#
> > +# Ext4 defragmentation stress test
> > +# Defragment file while other task does direct io
> > +#-----------------------------------------------------------------------
> > +# (c) 2013 Dmitry Monakhov
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> > +#
> > +#-----------------------------------------------------------------------
> > +#
> > +# creator
> > +owner=dmonakhov@openvz.org
> > +
> > +seq=`basename $0`
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1 # failure is the default!
> > +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> > +
> > +# get standard environment, filters and checks
> > +. ./common.rc
> > +. ./common.filter
> > +. ./common.defrag
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> > +_supported_fs ext4
> > +_need_to_be_root
> > +_require_scratch
> > +_require_defrag
> > +
> I think you meant to set NUM_JOBS here like you did in [Patch 08/10],
> something like:
>
> +NUM_JOBS=$((4*LOAD_FACTOR))
Indeed. I've forget to update patch.
>
> otherwise you get
>
> ### [not run] /usr/local/bin/fio too old
>
> > +BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
> > +# We need space for 2 files (test file, and donor one)
> > +# reserve 30% in order to avoid ENOSPC
> > +FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
> > +
> > +cat >$tmp-$seq.fio <<EOF
> > +# Common e4defrag regression tests
> > +[global]
> > +ioengine=ioe_e4defrag
> > +iodepth=1
> > +directory=${SCRATCH_MNT}
> > +filesize=${FILE_SIZE}
> > +size=999G
> > +buffered=0
> > +fadvise_hint=0
> > +group_reporting
> > +
> > +#################################
> > +# Test1
> > +# Defragment file while other task does direct io
> > +
> > +# Continious sequential defrag activity
> > +[defrag-4k]
> > +ioengine=e4defrag
> > +iodepth=1
> > +bs=128k
> > +donorname=test1.def
> > +filename=test1
> > +inplace=0
> > +rw=write
> > +numjobs=${NUM_JOBS}
> > +runtime=30*${TIME_FACTOR}
> > +time_based
> > +
> > +# Verifier
> > +[aio-dio-verifier]
> > +ioengine=libaio
> > +iodepth=128*${LOAD_FACTOR}
> > +numjobs=1
> > +verify=crc32c-intel
> > +verify_fatal=1
> > +verify_dump=1
> > +verify_backlog=1024
> > +verify_async=1
> > +verifysort=1
> > +direct=1
> > +bs=64k
> > +rw=randwrite
> > +filename=test1
> > +runtime=30*${TIME_FACTOR}
> > +time_based
> > +EOF
> > +
> > +_workout()
> > +{
> > + echo ""
> > + echo " Start defragment activity "
> git does not like trailing whitespace ^
> (same for the other 3 tests)
> > + echo ""
> > + cat $tmp-$seq.fio >> $seq.full
> > + run_check $FIO_PROG $tmp-$seq.fio
> > +}
> > +
> > +_require_fio $tmp-$seq.fio
> > +
> > +_scratch_mkfs >> $seq.full 2>&1
> > +_scratch_mount
> > +
> > +if ! _workout; then
> > + umount $SCRATCH_DEV 2>/dev/null
> > + exit
> > +fi
> > +
> > +if ! _scratch_unmount; then
> > + echo "failed to umount"
> > + status=1
> > + exit
> > +fi
> > +_check_scratch_fs
> > +status=$?
> > +exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
> > diff --git a/299.out b/299.out
> > new file mode 100644
> > index 0000000..c2b0b4c
> > --- /dev/null
> > +++ b/299.out
> > @@ -0,0 +1,4 @@
> > +QA output created by 299
> > +
> > + Start defragment activity
> remove trailing whitespace ^
> (same for the other 3 tests)
>
> Let me know if you agree with these changes and I will make them at
> commit time.
Yes, please do.
>
> Thanks
> --Rich
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
@ 2013-03-02 1:30 ` Dmitry Monakhov
0 siblings, 0 replies; 53+ messages in thread
From: Dmitry Monakhov @ 2013-03-02 1:30 UTC (permalink / raw)
To: Rich Johnston; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On Fri, 1 Mar 2013 13:23:50 -0600, Rich Johnston <rjohnston@sgi.com> wrote:
> On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> > Perform various regression tests for ext4defrag subsystem
> >
> > 299'th Test1: Defragment file while other task does direct AIO
> > 300'th Test2: Perform defragmentation on file under buffered AIO
> > while third task does direct AIO to donor file
> > 301'th Test3: Two defrag tasks use common donor file.
> > 302'th Test4: Stress defragmentation. Several threads pefrorm
> typo perform
>
> > fragmentation at random position use inplace=1 will
> > allocate and free blocks inside defrag event improve
> > load pressure.
> >
> > This tests are considered dengerous because 300'th and 301'th are known
> typo dangerous
>
> > to trigger OOPS on recent kernels see:https://gist.github.com/dmonakhov/4770294
> >
> > Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
> > ---
> > 299 | 126 +++++++++++++++++++++++++++++++++++++++++++++++
> > 299.out | 4 ++
> > 300 | 143 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 300.out | 4 ++
> > 301 | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > 301.out | 4 ++
> > 302 | 130 ++++++++++++++++++++++++++++++++++++++++++++++++
> > 302.out | 4 ++
> > common.defrag | 4 +-
> > group | 6 ++-
> > 10 files changed, 575 insertions(+), 3 deletions(-)
> > create mode 100755 299
> > create mode 100644 299.out
> > create mode 100755 300
> > create mode 100644 300.out
> > create mode 100755 301
> > create mode 100644 301.out
> > create mode 100755 302
> > create mode 100644 302.out
> >
> > diff --git a/299 b/299
> > new file mode 100755
> > index 0000000..4a7a98c
> > --- /dev/null
> > +++ b/299
> > @@ -0,0 +1,126 @@
> > +#! /bin/bash
> > +# FSQA Test No. 299
> > +#
> > +# Ext4 defragmentation stress test
> > +# Defragment file while other task does direct io
> > +#-----------------------------------------------------------------------
> > +# (c) 2013 Dmitry Monakhov
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> > +#
> > +#-----------------------------------------------------------------------
> > +#
> > +# creator
> > +owner=dmonakhov@openvz.org
> > +
> > +seq=`basename $0`
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1 # failure is the default!
> > +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> > +
> > +# get standard environment, filters and checks
> > +. ./common.rc
> > +. ./common.filter
> > +. ./common.defrag
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> > +_supported_fs ext4
> > +_need_to_be_root
> > +_require_scratch
> > +_require_defrag
> > +
> I think you meant to set NUM_JOBS here like you did in [Patch 08/10],
> something like:
>
> +NUM_JOBS=$((4*LOAD_FACTOR))
Indeed. I've forget to update patch.
>
> otherwise you get
>
> ### [not run] /usr/local/bin/fio too old
>
> > +BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
> > +# We need space for 2 files (test file, and donor one)
> > +# reserve 30% in order to avoid ENOSPC
> > +FILE_SIZE=$((BLK_DEV_SIZE * (512 / (2 + 1))))
> > +
> > +cat >$tmp-$seq.fio <<EOF
> > +# Common e4defrag regression tests
> > +[global]
> > +ioengine=ioe_e4defrag
> > +iodepth=1
> > +directory=${SCRATCH_MNT}
> > +filesize=${FILE_SIZE}
> > +size=999G
> > +buffered=0
> > +fadvise_hint=0
> > +group_reporting
> > +
> > +#################################
> > +# Test1
> > +# Defragment file while other task does direct io
> > +
> > +# Continious sequential defrag activity
> > +[defrag-4k]
> > +ioengine=e4defrag
> > +iodepth=1
> > +bs=128k
> > +donorname=test1.def
> > +filename=test1
> > +inplace=0
> > +rw=write
> > +numjobs=${NUM_JOBS}
> > +runtime=30*${TIME_FACTOR}
> > +time_based
> > +
> > +# Verifier
> > +[aio-dio-verifier]
> > +ioengine=libaio
> > +iodepth=128*${LOAD_FACTOR}
> > +numjobs=1
> > +verify=crc32c-intel
> > +verify_fatal=1
> > +verify_dump=1
> > +verify_backlog=1024
> > +verify_async=1
> > +verifysort=1
> > +direct=1
> > +bs=64k
> > +rw=randwrite
> > +filename=test1
> > +runtime=30*${TIME_FACTOR}
> > +time_based
> > +EOF
> > +
> > +_workout()
> > +{
> > + echo ""
> > + echo " Start defragment activity "
> git does not like trailing whitespace ^
> (same for the other 3 tests)
> > + echo ""
> > + cat $tmp-$seq.fio >> $seq.full
> > + run_check $FIO_PROG $tmp-$seq.fio
> > +}
> > +
> > +_require_fio $tmp-$seq.fio
> > +
> > +_scratch_mkfs >> $seq.full 2>&1
> > +_scratch_mount
> > +
> > +if ! _workout; then
> > + umount $SCRATCH_DEV 2>/dev/null
> > + exit
> > +fi
> > +
> > +if ! _scratch_unmount; then
> > + echo "failed to umount"
> > + status=1
> > + exit
> > +fi
> > +_check_scratch_fs
> > +status=$?
> > +exit diff --git a/299.out b/299.out new file mode 100644 index 0000000..b215a3f
> > diff --git a/299.out b/299.out
> > new file mode 100644
> > index 0000000..c2b0b4c
> > --- /dev/null
> > +++ b/299.out
> > @@ -0,0 +1,4 @@
> > +QA output created by 299
> > +
> > + Start defragment activity
> remove trailing whitespace ^
> (same for the other 3 tests)
>
> Let me know if you agree with these changes and I will make them at
> commit time.
Yes, please do.
>
> Thanks
> --Rich
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 10/10] xfstests: add disk failure simulation test
2013-03-01 20:11 ` Rich Johnston
@ 2013-03-02 1:49 ` Dmitry Monakhov
2013-03-04 23:44 ` Rich Johnston
0 siblings, 1 reply; 53+ messages in thread
From: Dmitry Monakhov @ 2013-03-02 1:49 UTC (permalink / raw)
To: Rich Johnston; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On Fri, 1 Mar 2013 14:11:15 -0600, Rich Johnston <rjohnston@sgi.com> wrote:
> On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
> > There are many situations where disk may fail for example
> > 1) brutal usb dongle unplug
> > 2) iscsi (or any other netbdev) failure due to network issues
> > In this situation filesystem which use this blockdevice is
> > expected to fail(force RO remount, abort, etc) but whole system
> > should still be operational. In other words:
> > 1) Kernel should not panic
> > 2) Memory should not leak
> > 3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
> > for affected filesystem
> > 4) It should be possible to umount broken filesystem
> >
> > Later when disk becomes available again we expect(only for journaled filesystems):
> > 5) It will be possible to mount filesystem w/o explicit fsck (in order to caught
>
> typo s/caught/catch/g
>
> > issues like https://patchwork.kernel.org/patch/1983981/)
> > 6) Filesystem should be operational
> > 7) After mount/umount has being done all errors should be fixed so fsck should
> > not spot any issues.
> >
> > This test use fault enjection (CONFIG_FAIL_MAKE_REQUEST=y config option )
> May want to mention all the kernel config options required.
> i.e. CONFIG_FAULT_INJECTION=y ... are there others?
> CONFIG_FAULT_INJECTION_DEBUG_FS=y ???
Yes, all three options are required.
>
> > which force all new IO requests to fail for a given device. Xfs already has
> to force
>
> > XFS_IOC_GOINGDOWN ioctl which provides similar behaviour, but it is fs speciffic
>
> typos s/behaviour/behavior/g s/speciffic/specific
> > and it does it in an easy way
> because it perform freeze_bdev() before actual
> > shotdown.
> typo s/shotdown/shutdown/g
Agree with your diagnosis. My gramma is bad and I've forget to call spell check
before submission. Should I resend this one or you fix it manually
on commit time?
>
> >
> > Test run fsstress in background and then force disk failure.
> > Once disk failed it check that (1)-(4) is true.
> Once the disk fails, check that (1)-(4) are true.
>
> > Then makes disk available again and check that (5)-(7) is also true
> make the disk ... are
> >
> > BE CAREFUL!! test known to cause memory corruption for XFS
> > see: https://gist.github.com/dmonakhov/4953045
> >
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
2013-02-20 10:42 ` Dmitry Monakhov
@ 2013-03-03 16:43 ` Zheng Liu
-1 siblings, 0 replies; 53+ messages in thread
From: Zheng Liu @ 2013-03-03 16:43 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: xfs, linux-fsdevel, linux-ext4, dchinner
On Wed, Feb 20, 2013 at 02:42:14PM +0400, Dmitry Monakhov wrote:
> Perform various regression tests for ext4defrag subsystem
>
> 299'th Test1: Defragment file while other task does direct AIO
Hi Dmitry,
I couldn't run this testcase in my sandbox. I got the following error
message.
FSTYP -- ext4
PLATFORM -- Linux/x86_64 lz-desktop 3.8.0
MKFS_OPTIONS -- /dev/sda2
MOUNT_OPTIONS -- -o acl,user_xattr /dev/sda2 /mnt/sda2
299 [not run] /usr/local/bin/fio too old
Not run: 299
Passed all 0 tests
My fio version is fio-2.0.14-23-g9c63. It seems that there are some
errors in fio config file.
Regards,
- Zheng
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
@ 2013-03-03 16:43 ` Zheng Liu
0 siblings, 0 replies; 53+ messages in thread
From: Zheng Liu @ 2013-03-03 16:43 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On Wed, Feb 20, 2013 at 02:42:14PM +0400, Dmitry Monakhov wrote:
> Perform various regression tests for ext4defrag subsystem
>
> 299'th Test1: Defragment file while other task does direct AIO
Hi Dmitry,
I couldn't run this testcase in my sandbox. I got the following error
message.
FSTYP -- ext4
PLATFORM -- Linux/x86_64 lz-desktop 3.8.0
MKFS_OPTIONS -- /dev/sda2
MOUNT_OPTIONS -- -o acl,user_xattr /dev/sda2 /mnt/sda2
299 [not run] /usr/local/bin/fio too old
Not run: 299
Passed all 0 tests
My fio version is fio-2.0.14-23-g9c63. It seems that there are some
errors in fio config file.
Regards,
- Zheng
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
2013-03-03 16:43 ` Zheng Liu
@ 2013-03-03 17:23 ` Zheng Liu
-1 siblings, 0 replies; 53+ messages in thread
From: Zheng Liu @ 2013-03-03 17:23 UTC (permalink / raw)
To: Dmitry Monakhov, xfs, linux-fsdevel, linux-ext4, dchinner
On Mon, Mar 04, 2013 at 12:43:42AM +0800, Zheng Liu wrote:
> On Wed, Feb 20, 2013 at 02:42:14PM +0400, Dmitry Monakhov wrote:
> > Perform various regression tests for ext4defrag subsystem
> >
> > 299'th Test1: Defragment file while other task does direct AIO
>
> Hi Dmitry,
>
> I couldn't run this testcase in my sandbox. I got the following error
> message.
>
> FSTYP -- ext4
> PLATFORM -- Linux/x86_64 lz-desktop 3.8.0
> MKFS_OPTIONS -- /dev/sda2
> MOUNT_OPTIONS -- -o acl,user_xattr /dev/sda2 /mnt/sda2
>
> 299 [not run] /usr/local/bin/fio too old
> Not run: 299
> Passed all 0 tests
>
> My fio version is fio-2.0.14-23-g9c63. It seems that there are some
> errors in fio config file.
I just notice Rich's reply. After adding 'NUM_JOBS=$((4*LOAD_FACTOR))',
the testcase can run. Sorry for the noisy.
Regards,
- Zheng
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
@ 2013-03-03 17:23 ` Zheng Liu
0 siblings, 0 replies; 53+ messages in thread
From: Zheng Liu @ 2013-03-03 17:23 UTC (permalink / raw)
To: Dmitry Monakhov, xfs, linux-fsdevel, linux-ext4, dchinner
On Mon, Mar 04, 2013 at 12:43:42AM +0800, Zheng Liu wrote:
> On Wed, Feb 20, 2013 at 02:42:14PM +0400, Dmitry Monakhov wrote:
> > Perform various regression tests for ext4defrag subsystem
> >
> > 299'th Test1: Defragment file while other task does direct AIO
>
> Hi Dmitry,
>
> I couldn't run this testcase in my sandbox. I got the following error
> message.
>
> FSTYP -- ext4
> PLATFORM -- Linux/x86_64 lz-desktop 3.8.0
> MKFS_OPTIONS -- /dev/sda2
> MOUNT_OPTIONS -- -o acl,user_xattr /dev/sda2 /mnt/sda2
>
> 299 [not run] /usr/local/bin/fio too old
> Not run: 299
> Passed all 0 tests
>
> My fio version is fio-2.0.14-23-g9c63. It seems that there are some
> errors in fio config file.
I just notice Rich's reply. After adding 'NUM_JOBS=$((4*LOAD_FACTOR))',
the testcase can run. Sorry for the noisy.
Regards,
- Zheng
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 09/10] xfstest: add defragmentation stress tests for ext4
2013-03-02 1:30 ` Dmitry Monakhov
(?)
@ 2013-03-04 23:19 ` Rich Johnston
-1 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-04 23:19 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 03/01/2013 07:30 PM, Dmitry Monakhov wrote:
> On Fri, 1 Mar 2013 13:23:50 -0600, Rich Johnston <rjohnston@sgi.com> wrote:
>> On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
>>
>> Let me know if you agree with these changes and I will make them at
>> commit time.
> Yes, please do.
>>
>> Thanks
>> --Rich
commit a3d92a6cbdf9e98d8e256974b50b025d1d4575ec
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date: Wed Feb 20 10:42:14 2013 +0000
xfstest: add defragmentation stress tests for ext4
Perform various regression tests for ext4defrag subsystem
301'th Test1: Defragment file while other task does direct AIO
302'th Test2: Perform defragmentation on file under buffered AIO
while third task does direct AIO to donor file
303'th Test3: Two defrag tasks use common donor file.
304'th Test4: Stress defragmentation. Several threads perform
fragmentation at random position use inplace=1 will
allocate and free blocks inside defrag event improve
load pressure.
This tests are considered dangerous because 302'th and 303'th are known
to trigger OOPS on recent kernels
see:https://gist.github.com/dmonakhov/4770
294
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 10/10] xfstests: add disk failure simulation test
2013-03-02 1:49 ` Dmitry Monakhov
@ 2013-03-04 23:44 ` Rich Johnston
0 siblings, 0 replies; 53+ messages in thread
From: Rich Johnston @ 2013-03-04 23:44 UTC (permalink / raw)
To: Dmitry Monakhov; +Cc: linux-fsdevel, linux-ext4, dchinner, xfs
On 03/01/2013 07:49 PM, Dmitry Monakhov wrote:
> On Fri, 1 Mar 2013 14:11:15 -0600, Rich Johnston <rjohnston@sgi.com> wrote:
>> On 02/20/2013 04:42 AM, Dmitry Monakhov wrote:
>>> There are many situations where disk may fail for example
>>> 1) brutal usb dongle unplug
>>> 2) iscsi (or any other netbdev) failure due to network issues
>>> In this situation filesystem which use this blockdevice is
>>> expected to fail(force RO remount, abort, etc) but whole system
>>> should still be operational. In other words:
>>> 1) Kernel should not panic
>>> 2) Memory should not leak
>>> 3) Data integrity operations (sync,fsync,fdatasync, directio) should fail
>>> for affected filesystem
>>> 4) It should be possible to umount broken filesystem
>>>
>>> Later when disk becomes available again we expect(only for journaled filesystems):
>>> 5) It will be possible to mount filesystem w/o explicit fsck (in order to caught
>>
>> typo s/caught/catch/g
>>
>>> issues like https://patchwork.kernel.org/patch/1983981/)
>>> 6) Filesystem should be operational
>>> 7) After mount/umount has being done all errors should be fixed so fsck should
>>> not spot any issues.
>>>
>>> This test use fault enjection (CONFIG_FAIL_MAKE_REQUEST=y config option )
>> May want to mention all the kernel config options required.
>> i.e. CONFIG_FAULT_INJECTION=y ... are there others?
>> CONFIG_FAULT_INJECTION_DEBUG_FS=y ???
> Yes, all three options are required.
>>
>>> which force all new IO requests to fail for a given device. Xfs already has
>> to force
>>
>>> XFS_IOC_GOINGDOWN ioctl which provides similar behaviour, but it is fs speciffic
>>
>> typos s/behaviour/behavior/g s/speciffic/specific
>> > and it does it in an easy way
>> because it perform freeze_bdev() before actual
>>> shotdown.
>> typo s/shotdown/shutdown/g
> Agree with your diagnosis. My gramma is bad and I've forget to call spell check
> before submission. Should I resend this one or you fix it manually
> on commit time?
No worries, I'm sure your English is much better than any of my attempts
to write in your native tongue. ;)
No need to resend, glad to take care of these minor changes at commit time.
commit 02e57e1e3a42856dca9061ff943ba72fa7be8469
Author: Dmitry Monakhov <dmonakhov@openvz.org>
Date: Wed Feb 20 10:42:15 2013 +0000
xfstests: add disk failure simulation test
There are many situations where disk may fail for example
1) brutal usb dongle unplug
2) iscsi (or any other netbdev) failure due to network issues
In this situation filesystem which use this blockdevice is
expected to fail(force RO remount, abort, etc) but whole system
should still be operational. In other words:
1) Kernel should not panic
2) Memory should not leak
3) Data integrity operations (sync,fsync,fdatasync, directio)
should fail
for affected filesystem
4) It should be possible to umount broken filesystem
Later when disk becomes available again we expect(only for
journaled filesystems):
5) It will be possible to mount filesystem w/o explicit fsck (in
order to catch
issues like https://patchwork.kernel.org/patch/1983981/)
6) Filesystem should be operational
7) After mount/umount has being done all errors should be fixed so
fsck should
not spot any issues.
This test use fault injection (CONFIG_FAULT_INJECTION=y,
CONFIG_FAIL_MAKE_REQUEST=y and CONFIG_FAULT_INJECTION_DEBUG_FS=y config
options) to force all new IO requests to fail for a given device. Xfs
already has XFS_IOC_GOINGDOWN ioctl which provides similar
behavior, but it
is fs specific and it does it in an easy way because it performs
freeze_bdev()
before actual shutdown.
Test run fsstress in background and then force disk failure.
Once disk failed it check that (1)-(4) is true.
Then makes disk available again and check that (5)-(7) is also true
BE CAREFUL!! test known to cause memory corruption for XFS
see: https://gist.github.com/dmonakhov/4953045
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 53+ messages in thread
end of thread, other threads:[~2013-03-04 23:44 UTC | newest]
Thread overview: 53+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-02-20 10:42 [PATCH 00/10] xfstests: Stress tests improments v5 Dmitry Monakhov
2013-02-20 10:42 ` [PATCH 01/10] xfstests: add fio requirement V2 Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
2013-03-01 15:43 ` Rich Johnston
2013-03-01 15:43 ` Rich Johnston
2013-02-20 10:42 ` [PATCH 02/10] xfstest: add configurable load factors Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
2013-03-01 15:43 ` Rich Johnston
2013-02-20 10:42 ` [PATCH 03/10] xfstests: hardcode fops for determinable fsstests runs Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
2013-03-01 15:43 ` Rich Johnston
2013-03-01 15:43 ` Rich Johnston
2013-02-20 10:42 ` [PATCH 04/10] xfstets: fsstress add replace file operation Dmitry Monakhov
2013-03-01 15:43 ` Rich Johnston
2013-02-20 10:42 ` [PATCH 05/10] xfstest: allow fsstress to use load factor where appropriate Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
2013-03-01 15:43 ` Rich Johnston
2013-02-20 10:42 ` [PATCH 06/10] xfstest: move run_check to common.rc Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
2013-03-01 15:44 ` Rich Johnston
2013-02-20 10:42 ` [PATCH 07/10] xfstest: add fallocate/truncate vs AIO/DIO stress test Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
2013-03-01 17:49 ` Rich Johnston
2013-03-01 17:49 ` Rich Johnston
2013-02-20 10:42 ` [PATCH 08/10] xfstest: add fallocate/punch_hole " Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
2013-03-01 17:50 ` Rich Johnston
2013-03-01 17:50 ` Rich Johnston
2013-02-20 10:42 ` [PATCH 09/10] xfstest: add defragmentation stress tests for ext4 Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
2013-03-01 19:23 ` Rich Johnston
2013-03-01 19:23 ` Rich Johnston
2013-03-02 1:30 ` Dmitry Monakhov
2013-03-02 1:30 ` Dmitry Monakhov
2013-03-04 23:19 ` Rich Johnston
2013-03-03 16:43 ` Zheng Liu
2013-03-03 16:43 ` Zheng Liu
2013-03-03 17:23 ` Zheng Liu
2013-03-03 17:23 ` Zheng Liu
2013-02-20 10:42 ` [PATCH 10/10] xfstests: add disk failure simulation test Dmitry Monakhov
2013-02-20 10:42 ` Dmitry Monakhov
2013-03-01 20:11 ` Rich Johnston
2013-03-02 1:49 ` Dmitry Monakhov
2013-03-04 23:44 ` Rich Johnston
2013-02-25 15:45 ` [PATCH 00/10] xfstests: Stress tests improments v5 Theodore Ts'o
2013-02-25 15:45 ` Theodore Ts'o
2013-02-25 21:28 ` Ben Myers
2013-02-25 21:28 ` Ben Myers
2013-02-25 22:09 ` Dave Chinner
2013-02-25 22:09 ` Dave Chinner
2013-02-25 23:00 ` Ben Myers
2013-02-25 23:00 ` Ben Myers
2013-03-01 20:23 ` Rich Johnston
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.