[PATCH] update 068 to reproduce an unfreeze hanging up problem

[PATCH] update 068 to reproduce an unfreeze hanging up problem

[Masayoshi MIZUMA]

update 068 to reproduce an unfreeze hanging up problem which is unfreeze
function, thaw_super(), sometimes hangs up if flush kernel thread does
writeback to the same filesystem concurrently.
The problem occurs on ext4 and ext3. They are reported at
ext4:
http://marc.info/?l=linux-ext4&m=132339590004560&w=2
ext3:
http://marc.info/?l=linux-ext4&m=131536612113658&w=2

This test runs freeze/unfreeze under heavy load. If the problem is
reproduced, this test will hang up because "xfs_freeze -u" hangs up...

Signed-off-by: Masayoshi MIZUMA <m.mizuma@jp.fujitsu.com>
---
 068     |   17 +++----
 068.out |  160 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 167 insertions(+), 10 deletions(-)

diff --git a/068 b/068
index 5d0053d..3b3597d 100755
--- a/068
+++ b/068
@@ -32,7 +32,7 @@ here=`pwd`
 tmp=/tmp/$$
 status=0	# success is the default!
 
-ITERATIONS=10
+ITERATIONS=50
 
 _cleanup()
 {
@@ -51,7 +51,7 @@ trap "_cleanup" 0 1 2 3 15
 . ./common.filter
 
 # real QA test starts here
-_supported_fs xfs
+_supported_fs ext3 ext4 xfs
 _supported_os Linux IRIX
 
 _require_scratch
@@ -62,7 +62,7 @@ rm -f $seq.full
 umount $SCRATCH_DEV >/dev/null 2>&1
 echo "*** MKFS ***"                         >>$seq.full
 echo ""                                     >>$seq.full
-_scratch_mkfs_xfs                           >>$seq.full 2>&1 \
+_scratch_mkfs                               >>$seq.full 2>&1 \
     || _fail "mkfs failed"
 _scratch_mount                              >>$seq.full 2>&1 \
     || _fail "mount failed"
@@ -75,12 +75,11 @@ touch $tmp.running
     STRESS_DIR="$SCRATCH_MNT/fsstress_test_dir"
     mkdir "$STRESS_DIR"
 
-    procs=2
-    nops=200
+    procs=100
+    nops=1000
     while [ -f "$tmp.running" ]
-      do
-      #	-w ensures that the only ops are ones which cause write I/O
-      $FSSTRESS_PROG -d $STRESS_DIR -w -p $procs -n $nops $FSSTRESS_AVOID \
+    do
+      $FSSTRESS_PROG -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID \
 	  > /dev/null 2>&1
     done
 
@@ -99,13 +98,11 @@ do
 	xfs_freeze -f "$SCRATCH_MNT" | tee -a $seq.full
 	[ $? != 0 ] && echo xfs_freeze -f "$SCRATCH_MNT" failed | \
 	    tee -a $seq.full
-	sleep 2
 
 	echo "*** thawing  \$SCRATCH_MNT" | tee -a $seq.full
 	xfs_freeze -u "$SCRATCH_MNT" | tee -a $seq.full
 	[ $? != 0 ] && echo xfs_freeze -u "$SCRATCH_MNT" failed | \
 	    tee -a $seq.full
-	sleep 2
 
 	echo  | tee -a $seq.full
 	let i=$i+1
diff --git a/068.out b/068.out
index 363d0e9..11eb58d 100644
--- a/068.out
+++ b/068.out
@@ -41,3 +41,163 @@ QA output created by 068
 *** freezing $SCRATCH_MNT
 *** thawing  $SCRATCH_MNT
 
+*** iteration: 10
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 11
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 12
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 13
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 14
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 15
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 16
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 17
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 18
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 19
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 20
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 21
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 22
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 23
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 24
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 25
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 26
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 27
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 28
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 29
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 30
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 31
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 32
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 33
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 34
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 35
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 36
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 37
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 38
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 39
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 40
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 41
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 42
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 43
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 44
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 45
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 46
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 47
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 48
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
+*** iteration: 49
+*** freezing $SCRATCH_MNT
+*** thawing  $SCRATCH_MNT
+
-- 
1.7.1

Thanks,
Masayoshi Mizuma

Re: [PATCH] update 068 to reproduce an unfreeze hanging up problem

[Dave Chinner]

On Tue, Dec 13, 2011 at 09:42:46AM +0900, Masayoshi MIZUMA wrote:
> update 068 to reproduce an unfreeze hanging up problem which is unfreeze
> function, thaw_super(), sometimes hangs up if flush kernel thread does
> writeback to the same filesystem concurrently.

That's not exactly what I asked to be done when I reviewed the
original patch -  I asked you to "make 068 generic" because it
already exercises freeze/thaw under a stressful workload. What I
expected was a change to "supported_fs" and the scratch mkfs
command so it will run on all filesystems.

test 068 will catch problems like the one your specific test
catches, but maybe not every time. Test 068 will catch problems your
test case won't, though - it's a trade-off between having lots of
tests that are similar but slightly different (difficult to
maintain, increases runtime, etc), and having one test that
exercises the functionality in a simple manner likely to detect
problems.

Test 068 is likely to detect problems because it:

	a) allows fsstress to try to do stuff while the filesystem
	is frozen by waiting a short time before thawing, hence load
	processes can get stuck either during the freeze of once the
	freeze is complete. Without that window, we are much less
	likely to test opeations on a frozen filesystem.

	b) allows more dirty data/metadata to build up between
	thaw/freeze commands, rather than running them as quickly as
	possible. This means freeze has more work to do, extenting
	the different phases of the freeze, making it more likely we
	have processes hit in different phases and hence test
	different parts of the freeze process.

IOWs, test 068 gives good coverage across most aspects of
freezing/thawing filesystems under load - and a lot of that woul dbe
lost by changing the test to mimic the ext4 specific test case you
have. It will still be able to trigger the problem you are trying to
expose, but it also has a much better chance of triggering problems
at different points of the freeze/thaw lifecycle that your specific
test....

> The problem occurs on ext4 and ext3. They are reported at
> ext4:
> http://marc.info/?l=linux-ext4&m=132339590004560&w=2
> ext3:
> http://marc.info/?l=linux-ext4&m=131536612113658&w=2
> 
> This test runs freeze/unfreeze under heavy load. If the problem is
> reproduced, this test will hang up because "xfs_freeze -u" hangs up...

> -ITERATIONS=10
> +ITERATIONS=50

....

> -    procs=2
> -    nops=200
> +    procs=100
> +    nops=1000

>      while [ -f "$tmp.running" ]
> -      do
> -      #	-w ensures that the only ops are ones which cause write I/O
> -      $FSSTRESS_PROG -d $STRESS_DIR -w -p $procs -n $nops $FSSTRESS_AVOID \
> +    do
> +      $FSSTRESS_PROG -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID \
>  	  > /dev/null 2>&1

And this is one of those cases - it is the write operations
that are the ones that cause trouble for freeze/thaw, so changing
the test to use read operations simply reduces the stress that is
being put on the filesytem freeze...

Also, you don't need lots of processes and ops to keep the filesystems
busy while freeze/thaw cycles are going on - if fsstress completes,
it simply gets started again. Hence it doesn't need to be configured
to run for a long time by ramping up processes and opcount. Yes, the
proc count could probably be increased a bit to increase the freeze
load, but i don't think that will improve the test all that much...

> @@ -99,13 +98,11 @@ do
>  	xfs_freeze -f "$SCRATCH_MNT" | tee -a $seq.full
>  	[ $? != 0 ] && echo xfs_freeze -f "$SCRATCH_MNT" failed | \
>  	    tee -a $seq.full
> -	sleep 2

And this simulates typical freeze/do something/thaw cycles. It also
allows fsstress to execute operations while the filesytem is frozen
and potentially try to grab things like the superblock lock because
fsstress issued a sync() system call. Dropping the sleep makes the
test less likely to find problems....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH] update 068 to reproduce an unfreeze hanging up problem

[Masayoshi MIZUMA]

(2011/12/13 15:32), Dave Chinner wrote:

> On Tue, Dec 13, 2011 at 09:42:46AM +0900, Masayoshi MIZUMA wrote:
> > update 068 to reproduce an unfreeze hanging up problem which is unfreeze
> > function, thaw_super(), sometimes hangs up if flush kernel thread does
> > writeback to the same filesystem concurrently.
> 
> That's not exactly what I asked to be done when I reviewed the
> original patch -  I asked you to "make 068 generic" because it
> already exercises freeze/thaw under a stressful workload. What I
> expected was a change to "supported_fs" and the scratch mkfs
> command so it will run on all filesystems.
> 
> test 068 will catch problems like the one your specific test
> catches, but maybe not every time. Test 068 will catch problems your
> test case won't, though - it's a trade-off between having lots of
> tests that are similar but slightly different (difficult to
> maintain, increases runtime, etc), and having one test that
> exercises the functionality in a simple manner likely to detect
> problems.

Thank you for your explanation about the policy and I understand it.

(snip)
 
> > @@ -99,13 +98,11 @@ do
> >  	xfs_freeze -f "$SCRATCH_MNT" | tee -a $seq.full
> >  	[ $? != 0 ] && echo xfs_freeze -f "$SCRATCH_MNT" failed | \
> >  	    tee -a $seq.full
> > -	sleep 2
> 
> And this simulates typical freeze/do something/thaw cycles. It also
> allows fsstress to execute operations while the filesytem is frozen
> and potentially try to grab things like the superblock lock because
> fsstress issued a sync() system call. Dropping the sleep makes the
> test less likely to find problems....

I tried to reproduce the problem not dropping the sleep, but the problem was
not reproduced... Therefore, I dropped it and the problem was reproduced.

However, as you mentioned, the problem is a timing proglem, so the
my reproduction might be just by chance. Dropping sleep may increase
the possibility of the reproduction, but not every time, so the change
is not good. That is same for the arguments of fsstress which I changed.

OK, I update 068 just to run other filesystem, ext3, ext4 and btrfs which
I confirmed xfs_freeze works on.
(xfs_freeze may work on the other filesystems which have freeze_fs/unfreeze_fs
 super_operations but I don't confirm...)

The patch is below.
-----------------------------------------------------------
update 068 to run other filesystems, ext3, ext4 and btrfs because
xfs_freeze works on the filesystems.

---
 068 |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/068 b/068
index 5d0053d..6f08f18 100755
--- a/068
+++ b/068
@@ -51,7 +51,7 @@ trap "_cleanup" 0 1 2 3 15
 . ./common.filter
 
 # real QA test starts here
-_supported_fs xfs
+_supported_fs btrfs ext3 ext4 xfs
 _supported_os Linux IRIX
 
 _require_scratch
@@ -62,7 +62,7 @@ rm -f $seq.full
 umount $SCRATCH_DEV >/dev/null 2>&1
 echo "*** MKFS ***"                         >>$seq.full
 echo ""                                     >>$seq.full
-_scratch_mkfs_xfs                           >>$seq.full 2>&1 \
+_scratch_mkfs                               >>$seq.full 2>&1 \
     || _fail "mkfs failed"
 _scratch_mount                              >>$seq.full 2>&1 \
     || _fail "mount failed"
-- 
1.7.1

Thanks,
Masayoshi Mizuma


> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Re: [PATCH] update 068 to reproduce an unfreeze hanging up problem

[Christoph Hellwig]

On Wed, Dec 14, 2011 at 11:22:10AM +0900, Masayoshi MIZUMA wrote:
> > test 068 will catch problems like the one your specific test
> > catches, but maybe not every time. Test 068 will catch problems your
> > test case won't, though - it's a trade-off between having lots of
> > tests that are similar but slightly different (difficult to
> > maintain, increases runtime, etc), and having one test that
> > exercises the functionality in a simple manner likely to detect
> > problems.
> 
> Thank you for your explanation about the policy and I understand it.


> I tried to reproduce the problem not dropping the sleep, but the problem was
> not reproduced... Therefore, I dropped it and the problem was reproduced.
> 
> However, as you mentioned, the problem is a timing proglem, so the
> my reproduction might be just by chance. Dropping sleep may increase
> the possibility of the reproduction, but not every time, so the change
> is not good. That is same for the arguments of fsstress which I changed.
> 
> OK, I update 068 just to run other filesystem, ext3, ext4 and btrfs which
> I confirmed xfs_freeze works on.
> (xfs_freeze may work on the other filesystems which have freeze_fs/unfreeze_fs
>  super_operations but I don't confirm...)
> 
> The patch is below.

Given that MIZUMAs patch reproduces a real life issue I think adding
his original patch in addition to this change would be a good idea.

Dave, do you have a strong opinion against that?

Re: 068: run on more filesystems

[Christoph Hellwig]

Thanks, applied.