[xfstest PATCH V2 0/2] overlay/031: add two tests of whiteout check in origin dir

[xfstest PATCH V2 0/2] overlay/031: add two tests of whiteout check in origin dir

[zhangyi (F)]

Add two tests of whiteout check in origin dir, cover other
two situations:

1) Lower origined dir have whiteouts;
2) Both upper and lower origined dirs have whiteouts (although
this case is pass now, still add this to cover all three kinds
of situations).

V2:
factor out _create_whiteout() helper,
test whiteout in the middle layer,
fix test failure with inodes index feature.

zhangyi (F) (2):
  overlay/031: factor out _create_whiteout() helper
  overlay/031: add tests of whiteouts in lowerdir and both dirs

 tests/overlay/031 | 84 ++++++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 64 insertions(+), 20 deletions(-)

-- 
2.5.0

[xfstest PATCH V2 1/2] overlay/031: factor out _create_whiteout() helper

[zhangyi (F)]

Factor out helper for create an leftover whiteout,
this is common to repeat test of whiteout check in
different underlaying directories.

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
---
 tests/overlay/031 | 38 +++++++++++++++++++++++---------------
 1 file changed, 23 insertions(+), 15 deletions(-)

diff --git a/tests/overlay/031 b/tests/overlay/031
index 674c11f..cb29ee5 100755
--- a/tests/overlay/031
+++ b/tests/overlay/031
@@ -38,6 +38,20 @@ _cleanup()
 	rm -f $tmp.*
 }
 
+# create test directory and test file, mount overlayfs and remove
+# testfile to create a whiteout in upper dir.
+_create_whiteout()
+{
+	mkdir -p $1/testdir
+	touch $1/testdir/$4
+
+	_overlay_mount_dirs $1 $2 $3 $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
+
+	rm -f $SCRATCH_MNT/testdir/$4
+
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
 # get standard environment, filters and checks
 . ./common/rc
 . ./common/filter
@@ -52,26 +66,20 @@ _require_scratch
 # remove all files from previous runs
 _scratch_mkfs
 
-# create test directory and test file
-lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER
-upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
-workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
-mkdir -p $lowerdir/testdir
-touch $lowerdir/testdir/a
-
-# mount overlayfs and remove testfile to create a whiteout
-_scratch_mount
+# create test directorys and whiteout
+lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+upperdir=$OVL_BASE_SCRATCH_MNT/upper
+workdir=$OVL_BASE_SCRATCH_MNT/workdir
+testfile=a
+mkdir -p $lowerdir $upperdir $workdir
 
-rm -f $SCRATCH_MNT/testdir/a
-
-# unmount overlayfs but not base fs
-$UMOUNT_PROG $SCRATCH_MNT
+_create_whiteout $lowerdir $upperdir $workdir $testfile
 
 # clean up the lower directory and mount overlay again,
 # whiteout will expose.
 rm -rf $lowerdir/testdir
 
-_scratch_mount
+_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
 
 ls $SCRATCH_MNT/testdir
 
@@ -84,7 +92,7 @@ rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
 $UMOUNT_PROG $SCRATCH_MNT
 touch $lowerdir/testdir
 
-_scratch_mount
+_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
 
 # try to remove test dir from overlay dir, trigger ovl_remove_and_whiteout,
 # it will not clean up the dir and lead to residue.
-- 
2.5.0

[xfstest PATCH V2 2/2] overlay/031: add tests of whiteouts in lowerdir and both dirs

[zhangyi (F)]

In overlay/031, it only cover the test case of whiteouts in
origined upper dir. This patch add two cases cover the other
two situations:

1) Lower origined dir have whiteouts;
2) Both upper and lower origined dirs have whiteouts (although
this case is pass now, still add this to cover all three kinds
of situations).

Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
---
 tests/overlay/031 | 58 ++++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 47 insertions(+), 11 deletions(-)

diff --git a/tests/overlay/031 b/tests/overlay/031
index cb29ee5..b7bad3a 100755
--- a/tests/overlay/031
+++ b/tests/overlay/031
@@ -1,9 +1,10 @@
 #! /bin/bash
 # FS QA Test 031
 #
-# The unmerged and impure upper directories may contain invalid
-# whiteouts when we change lowerdir(e.g. clean up dir) and remount
-# overlay. This may lead to whiteouts exposure and rmdir failure.
+# The unmerged and origined directories may contain invalid
+# whiteouts when we change underlaying dir (e.g. clean up lowerdir)
+# and remount overlay. This may lead to whiteouts exposure and
+# rmdir failure.
 #
 #-----------------------------------------------------------------------
 # Copyright (c) 2017 Huawei.  All Rights Reserved.
@@ -67,19 +68,21 @@ _require_scratch
 _scratch_mkfs
 
 # create test directorys and whiteout
-lowerdir=$OVL_BASE_SCRATCH_MNT/lower
+lowerdir1=$OVL_BASE_SCRATCH_MNT/lower1
+lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
 upperdir=$OVL_BASE_SCRATCH_MNT/upper
 workdir=$OVL_BASE_SCRATCH_MNT/workdir
-testfile=a
-mkdir -p $lowerdir $upperdir $workdir
+testfile1=a
+testfile2=b
+mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir
 
-_create_whiteout $lowerdir $upperdir $workdir $testfile
+_create_whiteout $lowerdir1 $upperdir $workdir $testfile1
 
 # clean up the lower directory and mount overlay again,
 # whiteout will expose.
-rm -rf $lowerdir/testdir
+rm -rf $lowerdir1/testdir
 
-_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
+_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
 
 ls $SCRATCH_MNT/testdir
 
@@ -90,15 +93,48 @@ rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
 # umount overlay again, create a new file with the same name and
 # mount overlay again.
 $UMOUNT_PROG $SCRATCH_MNT
-touch $lowerdir/testdir
+touch $lowerdir1/testdir
 
-_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
+_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
 
 # try to remove test dir from overlay dir, trigger ovl_remove_and_whiteout,
 # it will not clean up the dir and lead to residue.
 rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
 ls $workdir/work
 
+$UMOUNT_PROG $SCRATCH_MNT
+
+# let lower dir have invalid whiteouts, repeat ls and rmdir test again.
+rm -rf $lowerdir1/testdir
+rm -rf $upperdir/testdir
+
+_create_whiteout $lowerdir2 $lowerdir1 $workdir $testfile1
+
+rm -rf $lowerdir2/testdir
+
+_overlay_mount_dirs "$lowerdir1:$lowerdir2" $upperdir $workdir \
+		     $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
+
+ls $SCRATCH_MNT/testdir
+rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
+
+$UMOUNT_PROG $SCRATCH_MNT
+
+# let lower dir and upper dir both have invalid whiteouts, repeat ls and rmdir again.
+rm -rf $lowerdir1/testdir
+rm -rf $upperdir/testdir
+
+_create_whiteout $lowerdir1 $upperdir $workdir $testfile1
+
+rm -rf $lowerdir1/testdir
+
+_create_whiteout $lowerdir2 $lowerdir1 $workdir $testfile2
+
+_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
+
+ls $SCRATCH_MNT/testdir
+rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
+
 # success, all done
 echo "Silence is golden"
 status=0
-- 
2.5.0

Re: [xfstest PATCH V2 2/2] overlay/031: add tests of whiteouts in lowerdir and both dirs

[Amir Goldstein]

On Fri, Jun 30, 2017 at 6:20 AM, zhangyi (F) <yi.zhang@huawei.com> wrote:
> In overlay/031, it only cover the test case of whiteouts in
> origined upper dir. This patch add two cases cover the other
> two situations:
>
> 1) Lower origined dir have whiteouts;
> 2) Both upper and lower origined dirs have whiteouts (although
> this case is pass now, still add this to cover all three kinds
> of situations).
>
> Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>

Reviewed-by: Amir Goldstein <amir73il@gmail.com>

Except one issue below.

> ---
>  tests/overlay/031 | 58 ++++++++++++++++++++++++++++++++++++++++++++-----------
>  1 file changed, 47 insertions(+), 11 deletions(-)
>
> diff --git a/tests/overlay/031 b/tests/overlay/031
> index cb29ee5..b7bad3a 100755
> --- a/tests/overlay/031
> +++ b/tests/overlay/031
> @@ -1,9 +1,10 @@
>  #! /bin/bash
>  # FS QA Test 031
>  #
> -# The unmerged and impure upper directories may contain invalid
> -# whiteouts when we change lowerdir(e.g. clean up dir) and remount
> -# overlay. This may lead to whiteouts exposure and rmdir failure.
> +# The unmerged and origined directories may contain invalid
> +# whiteouts when we change underlaying dir (e.g. clean up lowerdir)
> +# and remount overlay. This may lead to whiteouts exposure and
> +# rmdir failure.
>  #
>  #-----------------------------------------------------------------------
>  # Copyright (c) 2017 Huawei.  All Rights Reserved.
> @@ -67,19 +68,21 @@ _require_scratch
>  _scratch_mkfs
>
>  # create test directorys and whiteout
> -lowerdir=$OVL_BASE_SCRATCH_MNT/lower
> +lowerdir1=$OVL_BASE_SCRATCH_MNT/lower1
> +lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
>  upperdir=$OVL_BASE_SCRATCH_MNT/upper
>  workdir=$OVL_BASE_SCRATCH_MNT/workdir
> -testfile=a
> -mkdir -p $lowerdir $upperdir $workdir
> +testfile1=a
> +testfile2=b
> +mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir
>
> -_create_whiteout $lowerdir $upperdir $workdir $testfile
> +_create_whiteout $lowerdir1 $upperdir $workdir $testfile1
>
>  # clean up the lower directory and mount overlay again,
>  # whiteout will expose.
> -rm -rf $lowerdir/testdir
> +rm -rf $lowerdir1/testdir
>
> -_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>
>  ls $SCRATCH_MNT/testdir
>
> @@ -90,15 +93,48 @@ rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>  # umount overlay again, create a new file with the same name and
>  # mount overlay again.
>  $UMOUNT_PROG $SCRATCH_MNT
> -touch $lowerdir/testdir
> +touch $lowerdir1/testdir
>
> -_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>
>  # try to remove test dir from overlay dir, trigger ovl_remove_and_whiteout,
>  # it will not clean up the dir and lead to residue.
>  rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>  ls $workdir/work
>
> +$UMOUNT_PROG $SCRATCH_MNT
> +
> +# let lower dir have invalid whiteouts, repeat ls and rmdir test again.
> +rm -rf $lowerdir1/testdir
> +rm -rf $upperdir/testdir
> +
> +_create_whiteout $lowerdir2 $lowerdir1 $workdir $testfile1
> +
> +rm -rf $lowerdir2/testdir
> +
> +_overlay_mount_dirs "$lowerdir1:$lowerdir2" $upperdir $workdir \
> +                    $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
> +
> +ls $SCRATCH_MNT/testdir
> +rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
> +
> +$UMOUNT_PROG $SCRATCH_MNT
> +
> +# let lower dir and upper dir both have invalid whiteouts, repeat ls and rmdir again.
> +rm -rf $lowerdir1/testdir
> +rm -rf $upperdir/testdir
> +
> +_create_whiteout $lowerdir1 $upperdir $workdir $testfile1
> +
> +rm -rf $lowerdir1/testdir
> +
> +_create_whiteout $lowerdir2 $lowerdir1 $workdir $testfile2
> +
> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
> +

Forgot to rm -rf $lowerdir2/testdir ?

Re: [xfstest PATCH V2 1/2] overlay/031: factor out _create_whiteout() helper

[Amir Goldstein]

On Fri, Jun 30, 2017 at 6:20 AM, zhangyi (F) <yi.zhang@huawei.com> wrote:
> Factor out helper for create an leftover whiteout,
> this is common to repeat test of whiteout check in
> different underlaying directories.
>
> Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>

Reviewed-by: Amir Goldstein <amir73il@gmail.com>

> ---
>  tests/overlay/031 | 38 +++++++++++++++++++++++---------------
>  1 file changed, 23 insertions(+), 15 deletions(-)
>
> diff --git a/tests/overlay/031 b/tests/overlay/031
> index 674c11f..cb29ee5 100755
> --- a/tests/overlay/031
> +++ b/tests/overlay/031
> @@ -38,6 +38,20 @@ _cleanup()
>         rm -f $tmp.*
>  }
>
> +# create test directory and test file, mount overlayfs and remove
> +# testfile to create a whiteout in upper dir.
> +_create_whiteout()
> +{
> +       mkdir -p $1/testdir
> +       touch $1/testdir/$4
> +
> +       _overlay_mount_dirs $1 $2 $3 $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
> +
> +       rm -f $SCRATCH_MNT/testdir/$4
> +
> +       $UMOUNT_PROG $SCRATCH_MNT
> +}
> +
>  # get standard environment, filters and checks
>  . ./common/rc
>  . ./common/filter
> @@ -52,26 +66,20 @@ _require_scratch
>  # remove all files from previous runs
>  _scratch_mkfs
>
> -# create test directory and test file
> -lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER
> -upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
> -workdir=$OVL_BASE_SCRATCH_MNT/$OVL_WORK
> -mkdir -p $lowerdir/testdir
> -touch $lowerdir/testdir/a
> -
> -# mount overlayfs and remove testfile to create a whiteout
> -_scratch_mount
> +# create test directorys and whiteout
> +lowerdir=$OVL_BASE_SCRATCH_MNT/lower
> +upperdir=$OVL_BASE_SCRATCH_MNT/upper
> +workdir=$OVL_BASE_SCRATCH_MNT/workdir
> +testfile=a
> +mkdir -p $lowerdir $upperdir $workdir
>
> -rm -f $SCRATCH_MNT/testdir/a
> -
> -# unmount overlayfs but not base fs
> -$UMOUNT_PROG $SCRATCH_MNT
> +_create_whiteout $lowerdir $upperdir $workdir $testfile
>
>  # clean up the lower directory and mount overlay again,
>  # whiteout will expose.
>  rm -rf $lowerdir/testdir
>
> -_scratch_mount
> +_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>
>  ls $SCRATCH_MNT/testdir
>
> @@ -84,7 +92,7 @@ rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>  $UMOUNT_PROG $SCRATCH_MNT
>  touch $lowerdir/testdir
>
> -_scratch_mount
> +_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>
>  # try to remove test dir from overlay dir, trigger ovl_remove_and_whiteout,
>  # it will not clean up the dir and lead to residue.
> --
> 2.5.0
>

Re: [xfstest PATCH V2 2/2] overlay/031: add tests of whiteouts in lowerdir and both dirs

[zhangyi (F)]

On 2017/6/30 22:28, Amir Goldstein wrote:
> On Fri, Jun 30, 2017 at 6:20 AM, zhangyi (F) <yi.zhang@huawei.com> wrote:
>> In overlay/031, it only cover the test case of whiteouts in
>> origined upper dir. This patch add two cases cover the other
>> two situations:
>>
>> 1) Lower origined dir have whiteouts;
>> 2) Both upper and lower origined dirs have whiteouts (although
>> this case is pass now, still add this to cover all three kinds
>> of situations).
>>
>> Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
> 
> Reviewed-by: Amir Goldstein <amir73il@gmail.com>
> 
> Except one issue below.
> 
>> ---
>>  tests/overlay/031 | 58 ++++++++++++++++++++++++++++++++++++++++++++-----------
>>  1 file changed, 47 insertions(+), 11 deletions(-)
>>
>> diff --git a/tests/overlay/031 b/tests/overlay/031
>> index cb29ee5..b7bad3a 100755
>> --- a/tests/overlay/031
>> +++ b/tests/overlay/031
>> @@ -1,9 +1,10 @@
>>  #! /bin/bash
>>  # FS QA Test 031
>>  #
>> -# The unmerged and impure upper directories may contain invalid
>> -# whiteouts when we change lowerdir(e.g. clean up dir) and remount
>> -# overlay. This may lead to whiteouts exposure and rmdir failure.
>> +# The unmerged and origined directories may contain invalid
>> +# whiteouts when we change underlaying dir (e.g. clean up lowerdir)
>> +# and remount overlay. This may lead to whiteouts exposure and
>> +# rmdir failure.
>>  #
>>  #-----------------------------------------------------------------------
>>  # Copyright (c) 2017 Huawei.  All Rights Reserved.
>> @@ -67,19 +68,21 @@ _require_scratch
>>  _scratch_mkfs
>>
>>  # create test directorys and whiteout
>> -lowerdir=$OVL_BASE_SCRATCH_MNT/lower
>> +lowerdir1=$OVL_BASE_SCRATCH_MNT/lower1
>> +lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
>>  upperdir=$OVL_BASE_SCRATCH_MNT/upper
>>  workdir=$OVL_BASE_SCRATCH_MNT/workdir
>> -testfile=a
>> -mkdir -p $lowerdir $upperdir $workdir
>> +testfile1=a
>> +testfile2=b
>> +mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir
>>
>> -_create_whiteout $lowerdir $upperdir $workdir $testfile
>> +_create_whiteout $lowerdir1 $upperdir $workdir $testfile1
>>
>>  # clean up the lower directory and mount overlay again,
>>  # whiteout will expose.
>> -rm -rf $lowerdir/testdir
>> +rm -rf $lowerdir1/testdir
>>
>> -_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>
>>  ls $SCRATCH_MNT/testdir
>>
>> @@ -90,15 +93,48 @@ rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>>  # umount overlay again, create a new file with the same name and
>>  # mount overlay again.
>>  $UMOUNT_PROG $SCRATCH_MNT
>> -touch $lowerdir/testdir
>> +touch $lowerdir1/testdir
>>
>> -_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>
>>  # try to remove test dir from overlay dir, trigger ovl_remove_and_whiteout,
>>  # it will not clean up the dir and lead to residue.
>>  rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>>  ls $workdir/work
>>
>> +$UMOUNT_PROG $SCRATCH_MNT
>> +
>> +# let lower dir have invalid whiteouts, repeat ls and rmdir test again.
>> +rm -rf $lowerdir1/testdir
>> +rm -rf $upperdir/testdir
>> +
>> +_create_whiteout $lowerdir2 $lowerdir1 $workdir $testfile1
>> +
>> +rm -rf $lowerdir2/testdir
>> +
>> +_overlay_mount_dirs "$lowerdir1:$lowerdir2" $upperdir $workdir \
>> +                    $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>> +
>> +ls $SCRATCH_MNT/testdir
>> +rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>> +
>> +$UMOUNT_PROG $SCRATCH_MNT
>> +
>> +# let lower dir and upper dir both have invalid whiteouts, repeat ls and rmdir again.
>> +rm -rf $lowerdir1/testdir
>> +rm -rf $upperdir/testdir
>> +
>> +_create_whiteout $lowerdir1 $upperdir $workdir $testfile1
>> +
>> +rm -rf $lowerdir1/testdir
>> +
>> +_create_whiteout $lowerdir2 $lowerdir1 $workdir $testfile2
>> +
>> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>> +
> 
> Forgot to rm -rf $lowerdir2/testdir ?
> 

Lowerdir2 is no longer used for the final test, so I think there is no difference
whether we removed lowerdir2/testdir or not.(I miss something?)

Thanks,
ZhangYi.

Re: [xfstest PATCH V2 2/2] overlay/031: add tests of whiteouts in lowerdir and both dirs

[Amir Goldstein]

On Mon, Jul 3, 2017 at 4:42 AM, zhangyi (F) <yi.zhang@huawei.com> wrote:
> On 2017/6/30 22:28, Amir Goldstein wrote:
>> On Fri, Jun 30, 2017 at 6:20 AM, zhangyi (F) <yi.zhang@huawei.com> wrote:
>>> In overlay/031, it only cover the test case of whiteouts in
>>> origined upper dir. This patch add two cases cover the other
>>> two situations:
>>>
>>> 1) Lower origined dir have whiteouts;
>>> 2) Both upper and lower origined dirs have whiteouts (although
>>> this case is pass now, still add this to cover all three kinds
>>> of situations).
>>>
>>> Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
>>
>> Reviewed-by: Amir Goldstein <amir73il@gmail.com>
>>
>> Except one issue below.
>>
>>> ---
>>>  tests/overlay/031 | 58 ++++++++++++++++++++++++++++++++++++++++++++-----------
>>>  1 file changed, 47 insertions(+), 11 deletions(-)
>>>
>>> diff --git a/tests/overlay/031 b/tests/overlay/031
>>> index cb29ee5..b7bad3a 100755
>>> --- a/tests/overlay/031
>>> +++ b/tests/overlay/031
>>> @@ -1,9 +1,10 @@
>>>  #! /bin/bash
>>>  # FS QA Test 031
>>>  #
>>> -# The unmerged and impure upper directories may contain invalid
>>> -# whiteouts when we change lowerdir(e.g. clean up dir) and remount
>>> -# overlay. This may lead to whiteouts exposure and rmdir failure.
>>> +# The unmerged and origined directories may contain invalid
>>> +# whiteouts when we change underlaying dir (e.g. clean up lowerdir)
>>> +# and remount overlay. This may lead to whiteouts exposure and
>>> +# rmdir failure.
>>>  #
>>>  #-----------------------------------------------------------------------
>>>  # Copyright (c) 2017 Huawei.  All Rights Reserved.
>>> @@ -67,19 +68,21 @@ _require_scratch
>>>  _scratch_mkfs
>>>
>>>  # create test directorys and whiteout
>>> -lowerdir=$OVL_BASE_SCRATCH_MNT/lower
>>> +lowerdir1=$OVL_BASE_SCRATCH_MNT/lower1
>>> +lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
>>>  upperdir=$OVL_BASE_SCRATCH_MNT/upper
>>>  workdir=$OVL_BASE_SCRATCH_MNT/workdir
>>> -testfile=a
>>> -mkdir -p $lowerdir $upperdir $workdir
>>> +testfile1=a
>>> +testfile2=b
>>> +mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir
>>>
>>> -_create_whiteout $lowerdir $upperdir $workdir $testfile
>>> +_create_whiteout $lowerdir1 $upperdir $workdir $testfile1
>>>
>>>  # clean up the lower directory and mount overlay again,
>>>  # whiteout will expose.
>>> -rm -rf $lowerdir/testdir
>>> +rm -rf $lowerdir1/testdir
>>>
>>> -_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>>
>>>  ls $SCRATCH_MNT/testdir
>>>
>>> @@ -90,15 +93,48 @@ rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>>>  # umount overlay again, create a new file with the same name and
>>>  # mount overlay again.
>>>  $UMOUNT_PROG $SCRATCH_MNT
>>> -touch $lowerdir/testdir
>>> +touch $lowerdir1/testdir
>>>
>>> -_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>>
>>>  # try to remove test dir from overlay dir, trigger ovl_remove_and_whiteout,
>>>  # it will not clean up the dir and lead to residue.
>>>  rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>>>  ls $workdir/work
>>>
>>> +$UMOUNT_PROG $SCRATCH_MNT
>>> +
>>> +# let lower dir have invalid whiteouts, repeat ls and rmdir test again.
>>> +rm -rf $lowerdir1/testdir
>>> +rm -rf $upperdir/testdir
>>> +
>>> +_create_whiteout $lowerdir2 $lowerdir1 $workdir $testfile1
>>> +
>>> +rm -rf $lowerdir2/testdir
>>> +
>>> +_overlay_mount_dirs "$lowerdir1:$lowerdir2" $upperdir $workdir \
>>> +                    $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>> +
>>> +ls $SCRATCH_MNT/testdir
>>> +rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>>> +
>>> +$UMOUNT_PROG $SCRATCH_MNT
>>> +
>>> +# let lower dir and upper dir both have invalid whiteouts, repeat ls and rmdir again.
>>> +rm -rf $lowerdir1/testdir
>>> +rm -rf $upperdir/testdir
>>> +
>>> +_create_whiteout $lowerdir1 $upperdir $workdir $testfile1
>>> +
>>> +rm -rf $lowerdir1/testdir
>>> +
>>> +_create_whiteout $lowerdir2 $lowerdir1 $workdir $testfile2
>>> +
>>> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>> +
>>
>> Forgot to rm -rf $lowerdir2/testdir ?
>>
>
> Lowerdir2 is no longer used for the final test, so I think there is no difference
> whether we removed lowerdir2/testdir or not.(I miss something?)
>

Right. I tested with my overlayfs-devel branch and found a different issue
with the 'whiteouts in both upper and lower' test case.
When you run the test you see the warning:
overlayfs: failed to verify origin (lower1/testdir, ino=16777474, err=-116)

This warning is generated by commit ("ovl: verify origin of merge dir lower")
and means that the test does not follow the lower dir at all.

The solution is simple:

rm -f $lowerdir1/testdir/$testfile1 instead of rm -rf $lowerdir1/testdir
to keep the original lower testdir dir intact, instead of recreating the
dir with the second _create_whiteout call.

Cheers,
Amir.

Re: [xfstest PATCH V2 2/2] overlay/031: add tests of whiteouts in lowerdir and both dirs

[zhangyi (F)]

On 2017/7/3 14:41, Amir Goldstein wrote:
> On Mon, Jul 3, 2017 at 4:42 AM, zhangyi (F) <yi.zhang@huawei.com> wrote:
>> On 2017/6/30 22:28, Amir Goldstein wrote:
>>> On Fri, Jun 30, 2017 at 6:20 AM, zhangyi (F) <yi.zhang@huawei.com> wrote:
>>>> In overlay/031, it only cover the test case of whiteouts in
>>>> origined upper dir. This patch add two cases cover the other
>>>> two situations:
>>>>
>>>> 1) Lower origined dir have whiteouts;
>>>> 2) Both upper and lower origined dirs have whiteouts (although
>>>> this case is pass now, still add this to cover all three kinds
>>>> of situations).
>>>>
>>>> Signed-off-by: zhangyi (F) <yi.zhang@huawei.com>
>>>
>>> Reviewed-by: Amir Goldstein <amir73il@gmail.com>
>>>
>>> Except one issue below.
>>>
>>>> ---
>>>>  tests/overlay/031 | 58 ++++++++++++++++++++++++++++++++++++++++++++-----------
>>>>  1 file changed, 47 insertions(+), 11 deletions(-)
>>>>
>>>> diff --git a/tests/overlay/031 b/tests/overlay/031
>>>> index cb29ee5..b7bad3a 100755
>>>> --- a/tests/overlay/031
>>>> +++ b/tests/overlay/031
>>>> @@ -1,9 +1,10 @@
>>>>  #! /bin/bash
>>>>  # FS QA Test 031
>>>>  #
>>>> -# The unmerged and impure upper directories may contain invalid
>>>> -# whiteouts when we change lowerdir(e.g. clean up dir) and remount
>>>> -# overlay. This may lead to whiteouts exposure and rmdir failure.
>>>> +# The unmerged and origined directories may contain invalid
>>>> +# whiteouts when we change underlaying dir (e.g. clean up lowerdir)
>>>> +# and remount overlay. This may lead to whiteouts exposure and
>>>> +# rmdir failure.
>>>>  #
>>>>  #-----------------------------------------------------------------------
>>>>  # Copyright (c) 2017 Huawei.  All Rights Reserved.
>>>> @@ -67,19 +68,21 @@ _require_scratch
>>>>  _scratch_mkfs
>>>>
>>>>  # create test directorys and whiteout
>>>> -lowerdir=$OVL_BASE_SCRATCH_MNT/lower
>>>> +lowerdir1=$OVL_BASE_SCRATCH_MNT/lower1
>>>> +lowerdir2=$OVL_BASE_SCRATCH_MNT/lower2
>>>>  upperdir=$OVL_BASE_SCRATCH_MNT/upper
>>>>  workdir=$OVL_BASE_SCRATCH_MNT/workdir
>>>> -testfile=a
>>>> -mkdir -p $lowerdir $upperdir $workdir
>>>> +testfile1=a
>>>> +testfile2=b
>>>> +mkdir -p $lowerdir1 $lowerdir2 $upperdir $workdir
>>>>
>>>> -_create_whiteout $lowerdir $upperdir $workdir $testfile
>>>> +_create_whiteout $lowerdir1 $upperdir $workdir $testfile1
>>>>
>>>>  # clean up the lower directory and mount overlay again,
>>>>  # whiteout will expose.
>>>> -rm -rf $lowerdir/testdir
>>>> +rm -rf $lowerdir1/testdir
>>>>
>>>> -_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>>> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>>>
>>>>  ls $SCRATCH_MNT/testdir
>>>>
>>>> @@ -90,15 +93,48 @@ rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>>>>  # umount overlay again, create a new file with the same name and
>>>>  # mount overlay again.
>>>>  $UMOUNT_PROG $SCRATCH_MNT
>>>> -touch $lowerdir/testdir
>>>> +touch $lowerdir1/testdir
>>>>
>>>> -_overlay_mount_dirs $lowerdir $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>>> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>>>
>>>>  # try to remove test dir from overlay dir, trigger ovl_remove_and_whiteout,
>>>>  # it will not clean up the dir and lead to residue.
>>>>  rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>>>>  ls $workdir/work
>>>>
>>>> +$UMOUNT_PROG $SCRATCH_MNT
>>>> +
>>>> +# let lower dir have invalid whiteouts, repeat ls and rmdir test again.
>>>> +rm -rf $lowerdir1/testdir
>>>> +rm -rf $upperdir/testdir
>>>> +
>>>> +_create_whiteout $lowerdir2 $lowerdir1 $workdir $testfile1
>>>> +
>>>> +rm -rf $lowerdir2/testdir
>>>> +
>>>> +_overlay_mount_dirs "$lowerdir1:$lowerdir2" $upperdir $workdir \
>>>> +                    $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>>> +
>>>> +ls $SCRATCH_MNT/testdir
>>>> +rm -rf $SCRATCH_MNT/testdir 2>&1 | _filter_scratch
>>>> +
>>>> +$UMOUNT_PROG $SCRATCH_MNT
>>>> +
>>>> +# let lower dir and upper dir both have invalid whiteouts, repeat ls and rmdir again.
>>>> +rm -rf $lowerdir1/testdir
>>>> +rm -rf $upperdir/testdir
>>>> +
>>>> +_create_whiteout $lowerdir1 $upperdir $workdir $testfile1
>>>> +
>>>> +rm -rf $lowerdir1/testdir
>>>> +
>>>> +_create_whiteout $lowerdir2 $lowerdir1 $workdir $testfile2
>>>> +
>>>> +_overlay_mount_dirs $lowerdir1 $upperdir $workdir $OVL_BASE_SCRATCH_MNT $SCRATCH_MNT
>>>> +
>>>
>>> Forgot to rm -rf $lowerdir2/testdir ?
>>>
>>
>> Lowerdir2 is no longer used for the final test, so I think there is no difference
>> whether we removed lowerdir2/testdir or not.(I miss something?)
>>
> 
> Right. I tested with my overlayfs-devel branch and found a different issue
> with the 'whiteouts in both upper and lower' test case.
> When you run the test you see the warning:
> overlayfs: failed to verify origin (lower1/testdir, ino=16777474, err=-116)
> 
> This warning is generated by commit ("ovl: verify origin of merge dir lower")
> and means that the test does not follow the lower dir at all.
> 
> The solution is simple:
> 
> rm -f $lowerdir1/testdir/$testfile1 instead of rm -rf $lowerdir1/testdir
> to keep the original lower testdir dir intact, instead of recreating the
> dir with the second _create_whiteout call.
> 

Yes, indeed, will fix it, thanks a lot.

Thanks,
ZhangYi.