[PATCH 0/2] Another overlay test for online inconsistency detection

[PATCH 0/2] Another overlay test for online inconsistency detection

[Amir Goldstein]

Eryu,

Originally, I wrote overlay/049 to test online inconcsistency detection
feature, which I wrote as a sub feature of NFS export, so the initial
version of the test required and eanbled NFS export.

Then, at some point, the test started to pass without NFS export enabled,
which was a by-product of another fix for inotify I did around the same
time. So the test got merged without requiring NFS export, but with said
fix commit on next branch.

On a recent discussion with Vivek, I realized that there was another
"by-product" inconsistency detection introduced by the same fix commit
and not covered by overlay/049.

The variant overlay/059 checks for this case of malformed multiply
referenced origin files. I wrote the test to support upcoming 'metacopy'
feature by Vivek, so it requires and enables redirect_dir, which is
harmless otherwise.

I tested overlay/059 on several old kernels. The results are:
- notrun on kernel < v4.10, before "redirect_dir" feature
- pass on kernels < v4.12, before "origin" was introduced
- fail on kernels < v4.16, before the fix commit
- pass on newer kernels

Amir Goldstein (2):
  overlay/049: update fix commit id
  overlay: test multiple origin references to the same lower file

 tests/overlay/049     |   4 +-
 tests/overlay/059     | 105 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/059.out |   2 +
 tests/overlay/group   |   1 +
 4 files changed, 110 insertions(+), 2 deletions(-)
 create mode 100755 tests/overlay/059
 create mode 100644 tests/overlay/059.out

-- 
2.7.4

[PATCH 1/2] overlay/049: update fix commit id

[Amir Goldstein]

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/049 | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/overlay/049 b/tests/overlay/049
index 1dafc46..376acf5 100755
--- a/tests/overlay/049
+++ b/tests/overlay/049
@@ -10,8 +10,8 @@
 # This test checks that overlayfs detects and fails lookup of a multiply
 # redirected dir.
 #
-# The check for multiply redirected dir was implemented by kernel titled
-# "ovl: hash directory inodes for fsnotify"
+# The check for multiply redirected dir was a by-product of kernel
+# commit 31747eda41ef ("ovl: hash directory inodes for fsnotify")
 #
 #-----------------------------------------------------------------------
 # Copyright (C) 2018 CTERA Networks. All Rights Reserved.
-- 
2.7.4

[PATCH 2/2] overlay: test multiple origin references to the same lower file

[Amir Goldstein]

Multiple origin references to the same lower file from upper files that
are not hardlinks will falsely return the same st_ino/st_dev for two
different overlay files and will cause 'diff' to falsely report that
content of files is the same when it is not.

This test checks that overlayfs detects and fails lookup of a multiply
referenced origin.

The check for multiply referenced origin was a by-product of kernel
commit 31747eda41ef ("ovl: hash directory inodes for fsnotify")

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/overlay/059     | 105 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/overlay/059.out |   2 +
 tests/overlay/group   |   1 +
 3 files changed, 108 insertions(+)
 create mode 100755 tests/overlay/059
 create mode 100644 tests/overlay/059.out

diff --git a/tests/overlay/059 b/tests/overlay/059
new file mode 100755
index 0000000..5ac2b1b
--- /dev/null
+++ b/tests/overlay/059
@@ -0,0 +1,105 @@
+#! /bin/bash
+# FS QA Test 059
+#
+# Test multiple origin references to the same lower file.
+#
+# Multiple origin references to the same lower file from upper files that
+# are not hardlinks will falsely return the same st_ino/st_dev for two
+# different overlay files and will cause 'diff' to falsely report that
+# content of files is the same when it is not.
+#
+# This test checks that overlayfs detects and fails lookup of a multiply
+# referenced origin.
+#
+# The check for multiply referenced origin was a by-product of kernel
+# commit 31747eda41ef ("ovl: hash directory inodes for fsnotify")
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2018 CTERA Networks. All Rights Reserved.
+# Author: Amir Goldstein <amir73il@gmail.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# Rename lower file to create copy up with origin xattr
+create_origin_ref()
+{
+	local ref=$1
+
+	touch $lowerdir/origin
+
+	# Enabling redirect_dir may seem irrelevant to rename of non-dir,
+	# but with upcoming 'metacopy' feature, redirects will be set also
+	# on non-dir and may also create multiple redirects.
+	_scratch_mount -o redirect_dir=on
+	mv $SCRATCH_MNT/origin $SCRATCH_MNT/$ref
+
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs overlay
+_supported_os Linux
+_require_scratch_nocheck
+_require_scratch_feature redirect_dir
+
+# remove all files from previous runs
+_scratch_mkfs
+
+upperdir=$OVL_BASE_SCRATCH_MNT/$OVL_UPPER
+lowerdir=$OVL_BASE_SCRATCH_MNT/$OVL_LOWER
+
+# Create copied up file with origin xattr
+create_origin_ref ref1
+# Duplicate the copied up file
+cp -a $upperdir/ref1 $upperdir/ref2
+
+# Diverge the content of the two copies of file
+# and the content of two copies of redirected dir
+echo right >> $upperdir/ref1
+echo wrong >> $upperdir/ref2
+
+_scratch_mount -o redirect_dir=on
+
+# If both copies of file use the same st_dev/st_ino in overlay
+# diff won't detect that their content differs
+diff -q $SCRATCH_MNT/ref1 $SCRATCH_MNT/ref2 &>/dev/null && \
+	echo "diff on duplicated upper files doesn't know right from wrong!"
+
+# success, all done
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/overlay/059.out b/tests/overlay/059.out
new file mode 100644
index 0000000..36ebeea
--- /dev/null
+++ b/tests/overlay/059.out
@@ -0,0 +1,2 @@
+QA output created by 059
+Silence is golden
diff --git a/tests/overlay/group b/tests/overlay/group
index 359ed42..40445a0 100644
--- a/tests/overlay/group
+++ b/tests/overlay/group
@@ -61,3 +61,4 @@
 056 auto quick fsck
 057 auto quick redirect
 058 auto quick exportfs
+059 auto quick copyup
-- 
2.7.4

Re: [PATCH 0/2] Another overlay test for online inconsistency detection

[Eryu Guan]

On Fri, May 04, 2018 at 12:41:29PM +0300, Amir Goldstein wrote:
> Eryu,
> 
> Originally, I wrote overlay/049 to test online inconcsistency detection
> feature, which I wrote as a sub feature of NFS export, so the initial
> version of the test required and eanbled NFS export.
> 
> Then, at some point, the test started to pass without NFS export enabled,
> which was a by-product of another fix for inotify I did around the same
> time. So the test got merged without requiring NFS export, but with said
> fix commit on next branch.
> 
> On a recent discussion with Vivek, I realized that there was another
> "by-product" inconsistency detection introduced by the same fix commit
> and not covered by overlay/049.
> 
> The variant overlay/059 checks for this case of malformed multiply
> referenced origin files. I wrote the test to support upcoming 'metacopy'
> feature by Vivek, so it requires and enables redirect_dir, which is
> harmless otherwise.
> 
> I tested overlay/059 on several old kernels. The results are:
> - notrun on kernel < v4.10, before "redirect_dir" feature
> - pass on kernels < v4.12, before "origin" was introduced
> - fail on kernels < v4.16, before the fix commit
> - pass on newer kernels

Thanks a lot for the detailed background information and test status!

BTW, I noticed that you still send patches to address eguan@redhat.com,
I'm not using thaa address anymore, please send to guaneryu@gmail.com
instead :)

Thanks,
Eryu

> 
> Amir Goldstein (2):
>   overlay/049: update fix commit id
>   overlay: test multiple origin references to the same lower file
> 
>  tests/overlay/049     |   4 +-
>  tests/overlay/059     | 105 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/overlay/059.out |   2 +
>  tests/overlay/group   |   1 +
>  4 files changed, 110 insertions(+), 2 deletions(-)
>  create mode 100755 tests/overlay/059
>  create mode 100644 tests/overlay/059.out
> 
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-unionfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] Another overlay test for online inconsistency detection

[Amir Goldstein]

On Wed, May 9, 2018 at 6:39 PM, Eryu Guan <guaneryu@gmail.com> wrote:
> On Fri, May 04, 2018 at 12:41:29PM +0300, Amir Goldstein wrote:
>> Eryu,
>>
>> Originally, I wrote overlay/049 to test online inconcsistency detection
>> feature, which I wrote as a sub feature of NFS export, so the initial
>> version of the test required and eanbled NFS export.
>>
>> Then, at some point, the test started to pass without NFS export enabled,
>> which was a by-product of another fix for inotify I did around the same
>> time. So the test got merged without requiring NFS export, but with said
>> fix commit on next branch.
>>
>> On a recent discussion with Vivek, I realized that there was another
>> "by-product" inconsistency detection introduced by the same fix commit
>> and not covered by overlay/049.
>>
>> The variant overlay/059 checks for this case of malformed multiply
>> referenced origin files. I wrote the test to support upcoming 'metacopy'
>> feature by Vivek, so it requires and enables redirect_dir, which is
>> harmless otherwise.
>>
>> I tested overlay/059 on several old kernels. The results are:
>> - notrun on kernel < v4.10, before "redirect_dir" feature
>> - pass on kernels < v4.12, before "origin" was introduced
>> - fail on kernels < v4.16, before the fix commit
>> - pass on newer kernels
>
> Thanks a lot for the detailed background information and test status!
>
> BTW, I noticed that you still send patches to address eguan@redhat.com,
> I'm not using thaa address anymore, please send to guaneryu@gmail.com
> instead :)
>

Right. Forgot to update my scripts.
Should be good now.

Thanks,
Amir.