* [PATCH v7 0/1] xfstests: add log attribute replay test
@ 2022-02-23 3:37 Catherine Hoang
2022-02-23 3:37 ` [PATCH v7 1/1] xfstests: Add Log Attribute Replay test Catherine Hoang
0 siblings, 1 reply; 15+ messages in thread
From: Catherine Hoang @ 2022-02-23 3:37 UTC (permalink / raw)
To: linux-xfs, fstests
Hi all,
This patch aims to provide better test coverage for Allison’s delayed
attribute set. These tests aim to cover cases where attributes are added,
removed, and overwritten in each format (shortform, leaf, node). Error
inject is used to replay these operations from the log.
The following cases are covered in this test:
- empty, add/remove inline attr (64 bytes)
- empty, add/remove internal attr (1kB)
- empty, add/remove remote attr (64kB)
- inline, add/remove inline attr (64 bytes)
- inline, add/remove internal attr (1kB)
- inline, add/remove remote attr (64kB)
- extent, add/remove internal attr (1kB)
- extent, add multiple internal attr (inject error on split operation)
- extent, add multiple internal attr (inject error on transition to node)
- extent, add/remove remote attr (64kB)
- btree, add/remove multiple internal (1kB)
- btree, add/remove remote attr (64kB)
- overwrite shortform attr
- overwrite leaf attr
- overwrite node attr
Running these tests require the corresponding kernel and xfsprogs changes
that add the new error tags da_leaf_split and attr_leaf_to_node.
v6->v7:
- Rename larp_leaf_to_node to attr_leaf_to_node
- Check that the larp knob is writable
- Cleanup files that were created during testing
Suggestions and feedback are appreciated!
Catherine
Allison Henderson (1):
xfstests: Add Log Attribute Replay test
tests/xfs/543 | 176 ++++++++++++++++++++++++++++++++++++++++++++++
tests/xfs/543.out | 149 +++++++++++++++++++++++++++++++++++++++
2 files changed, 325 insertions(+)
create mode 100755 tests/xfs/543
create mode 100644 tests/xfs/543.out
--
2.25.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v7 1/1] xfstests: Add Log Attribute Replay test
2022-02-23 3:37 [PATCH v7 0/1] xfstests: add log attribute replay test Catherine Hoang
@ 2022-02-23 3:37 ` Catherine Hoang
2022-03-01 20:46 ` Allison Henderson
` (2 more replies)
0 siblings, 3 replies; 15+ messages in thread
From: Catherine Hoang @ 2022-02-23 3:37 UTC (permalink / raw)
To: linux-xfs, fstests
From: Allison Henderson <allison.henderson@oracle.com>
This patch adds tests to exercise the log attribute error
inject and log replay. These tests aim to cover cases where attributes
are added, removed, and overwritten in each format (shortform, leaf,
node). Error inject is used to replay these operations from the log.
Signed-off-by: Allison Henderson <allison.henderson@oracle.com>
Signed-off-by: Catherine Hoang <catherine.hoang@oracle.com>
---
tests/xfs/543 | 176 ++++++++++++++++++++++++++++++++++++++++++++++
tests/xfs/543.out | 149 +++++++++++++++++++++++++++++++++++++++
2 files changed, 325 insertions(+)
create mode 100755 tests/xfs/543
create mode 100644 tests/xfs/543.out
diff --git a/tests/xfs/543 b/tests/xfs/543
new file mode 100755
index 00000000..06f16f21
--- /dev/null
+++ b/tests/xfs/543
@@ -0,0 +1,176 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022, Oracle and/or its affiliates. All Rights Reserved.
+#
+# FS QA Test 543
+#
+# Log attribute replay test
+#
+. ./common/preamble
+_begin_fstest auto quick attr
+
+# get standard environment, filters and checks
+. ./common/filter
+. ./common/attr
+. ./common/inject
+
+_cleanup()
+{
+ rm -rf $tmp.* $testdir
+ test -w /sys/fs/xfs/debug/larp && \
+ echo 0 > /sys/fs/xfs/debug/larp
+}
+
+test_attr_replay()
+{
+ testfile=$testdir/$1
+ attr_name=$2
+ attr_value=$3
+ flag=$4
+ error_tag=$5
+
+ # Inject error
+ _scratch_inject_error $error_tag
+
+ # Set attribute
+ echo "$attr_value" | ${ATTR_PROG} -$flag "$attr_name" $testfile 2>&1 | \
+ _filter_scratch
+
+ # FS should be shut down, touch will fail
+ touch $testfile 2>&1 | _filter_scratch
+
+ # Remount to replay log
+ _scratch_remount_dump_log >> $seqres.full
+
+ # FS should be online, touch should succeed
+ touch $testfile
+
+ # Verify attr recovery
+ { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 | _filter_scratch
+
+ echo ""
+}
+
+create_test_file()
+{
+ filename=$testdir/$1
+ count=$2
+ attr_value=$3
+
+ touch $filename
+
+ for i in `seq $count`
+ do
+ $ATTR_PROG -s "attr_name$i" -V $attr_value $filename >> \
+ $seqres.full
+ done
+}
+
+# real QA test starts here
+_supported_fs xfs
+
+_require_scratch
+_require_attrs
+_require_xfs_io_error_injection "larp"
+_require_xfs_io_error_injection "da_leaf_split"
+_require_xfs_io_error_injection "attr_leaf_to_node"
+_require_xfs_sysfs debug/larp
+test -w /sys/fs/xfs/debug/larp || _notrun "larp knob not writable"
+
+# turn on log attributes
+echo 1 > /sys/fs/xfs/debug/larp
+
+attr16="0123456789ABCDEF"
+attr64="$attr16$attr16$attr16$attr16"
+attr256="$attr64$attr64$attr64$attr64"
+attr1k="$attr256$attr256$attr256$attr256"
+attr4k="$attr1k$attr1k$attr1k$attr1k"
+attr8k="$attr4k$attr4k"
+attr16k="$attr8k$attr8k"
+attr32k="$attr16k$attr16k"
+attr64k="$attr32k$attr32k"
+
+echo "*** mkfs"
+_scratch_mkfs >/dev/null
+
+echo "*** mount FS"
+_scratch_mount
+
+testdir=$SCRATCH_MNT/testdir
+mkdir $testdir
+
+# empty, inline
+create_test_file empty_file1 0
+test_attr_replay empty_file1 "attr_name" $attr64 "s" "larp"
+test_attr_replay empty_file1 "attr_name" $attr64 "r" "larp"
+
+# empty, internal
+create_test_file empty_file2 0
+test_attr_replay empty_file2 "attr_name" $attr1k "s" "larp"
+test_attr_replay empty_file2 "attr_name" $attr1k "r" "larp"
+
+# empty, remote
+create_test_file empty_file3 0
+test_attr_replay empty_file3 "attr_name" $attr64k "s" "larp"
+test_attr_replay empty_file3 "attr_name" $attr64k "r" "larp"
+
+# inline, inline
+create_test_file inline_file1 1 $attr16
+test_attr_replay inline_file1 "attr_name2" $attr64 "s" "larp"
+test_attr_replay inline_file1 "attr_name2" $attr64 "r" "larp"
+
+# inline, internal
+create_test_file inline_file2 1 $attr16
+test_attr_replay inline_file2 "attr_name2" $attr1k "s" "larp"
+test_attr_replay inline_file2 "attr_name2" $attr1k "r" "larp"
+
+# inline, remote
+create_test_file inline_file3 1 $attr16
+test_attr_replay inline_file3 "attr_name2" $attr64k "s" "larp"
+test_attr_replay inline_file3 "attr_name2" $attr64k "r" "larp"
+
+# extent, internal
+create_test_file extent_file1 1 $attr1k
+test_attr_replay extent_file1 "attr_name2" $attr1k "s" "larp"
+test_attr_replay extent_file1 "attr_name2" $attr1k "r" "larp"
+
+# extent, inject error on split
+create_test_file extent_file2 3 $attr1k
+test_attr_replay extent_file2 "attr_name4" $attr1k "s" "da_leaf_split"
+
+# extent, inject error on fork transition
+create_test_file extent_file3 3 $attr1k
+test_attr_replay extent_file3 "attr_name4" $attr1k "s" "attr_leaf_to_node"
+
+# extent, remote
+create_test_file extent_file4 1 $attr1k
+test_attr_replay extent_file4 "attr_name2" $attr64k "s" "larp"
+test_attr_replay extent_file4 "attr_name2" $attr64k "r" "larp"
+
+# remote, internal
+create_test_file remote_file1 1 $attr64k
+test_attr_replay remote_file1 "attr_name2" $attr1k "s" "larp"
+test_attr_replay remote_file1 "attr_name2" $attr1k "r" "larp"
+
+# remote, remote
+create_test_file remote_file2 1 $attr64k
+test_attr_replay remote_file2 "attr_name2" $attr64k "s" "larp"
+test_attr_replay remote_file2 "attr_name2" $attr64k "r" "larp"
+
+# replace shortform
+create_test_file sf_file 2 $attr64
+test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
+
+# replace leaf
+create_test_file leaf_file 2 $attr1k
+test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
+
+# replace node
+create_test_file node_file 1 $attr64k
+$ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
+ >> $seqres.full
+test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
+
+echo "*** done"
+status=0
+exit
diff --git a/tests/xfs/543.out b/tests/xfs/543.out
new file mode 100644
index 00000000..1c74e795
--- /dev/null
+++ b/tests/xfs/543.out
@@ -0,0 +1,149 @@
+QA output created by 543
+*** mkfs
+*** mount FS
+attr_set: Input/output error
+Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
+touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
+21d850f99c43cc13abbe34838a8a3c8a -
+
+attr_remove: Input/output error
+Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file1
+touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
+attr_get: No data available
+Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file1
+d41d8cd98f00b204e9800998ecf8427e -
+
+attr_set: Input/output error
+Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
+touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
+2ff89c2935debc431745ec791be5421a -
+
+attr_remove: Input/output error
+Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file2
+touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
+attr_get: No data available
+Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file2
+d41d8cd98f00b204e9800998ecf8427e -
+
+attr_set: Input/output error
+Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
+touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
+5d24b314242c52176c98ac4bd685da8b -
+
+attr_remove: Input/output error
+Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file3
+touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
+attr_get: No data available
+Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file3
+d41d8cd98f00b204e9800998ecf8427e -
+
+attr_set: Input/output error
+Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file1
+touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
+5a7b559a70d8e92b4f3c6f7158eead08 -
+
+attr_remove: Input/output error
+Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file1
+touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
+attr_get: No data available
+Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file1
+d41d8cd98f00b204e9800998ecf8427e -
+
+attr_set: Input/output error
+Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file2
+touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
+5717d5e66c70be6bdb00ecbaca0b7749 -
+
+attr_remove: Input/output error
+Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file2
+touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
+attr_get: No data available
+Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file2
+d41d8cd98f00b204e9800998ecf8427e -
+
+attr_set: Input/output error
+Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file3
+touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
+5c929964efd1b243aa8cceb6524f4810 -
+
+attr_remove: Input/output error
+Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file3
+touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
+attr_get: No data available
+Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file3
+d41d8cd98f00b204e9800998ecf8427e -
+
+attr_set: Input/output error
+Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file1
+touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
+51ccb5cdfc9082060f0f94a8a108fea0 -
+
+attr_remove: Input/output error
+Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file1
+touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
+attr_get: No data available
+Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file1
+d41d8cd98f00b204e9800998ecf8427e -
+
+attr_set: Input/output error
+Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file2
+touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2': Input/output error
+8d530bbe852d8bca83b131d5b3e497f5 -
+
+attr_set: Input/output error
+Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file3
+touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3': Input/output error
+5d77c4d3831a35bcbbd6e7677119ce9a -
+
+attr_set: Input/output error
+Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file4
+touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
+6707ec2431e4dbea20e17da0816520bb -
+
+attr_remove: Input/output error
+Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file4
+touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
+attr_get: No data available
+Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file4
+d41d8cd98f00b204e9800998ecf8427e -
+
+attr_set: Input/output error
+Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file1
+touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
+767ebca3e4a6d24170857364f2bf2a3c -
+
+attr_remove: Input/output error
+Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file1
+touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
+attr_get: No data available
+Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file1
+d41d8cd98f00b204e9800998ecf8427e -
+
+attr_set: Input/output error
+Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file2
+touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
+fd84ddec89237e6d34a1703639efaebf -
+
+attr_remove: Input/output error
+Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file2
+touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
+attr_get: No data available
+Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file2
+d41d8cd98f00b204e9800998ecf8427e -
+
+attr_set: Input/output error
+Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
+touch: cannot touch 'SCRATCH_MNT/testdir/sf_file': Input/output error
+34aaa49662bafb46c76e377454685071 -
+
+attr_set: Input/output error
+Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
+touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file': Input/output error
+664e95ec28830ffb367c0950026e0d21 -
+
+attr_set: Input/output error
+Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
+touch: cannot touch 'SCRATCH_MNT/testdir/node_file': Input/output error
+bb37a78ce26472eeb711e3559933db42 -
+
+*** done
--
2.25.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v7 1/1] xfstests: Add Log Attribute Replay test
2022-02-23 3:37 ` [PATCH v7 1/1] xfstests: Add Log Attribute Replay test Catherine Hoang
@ 2022-03-01 20:46 ` Allison Henderson
2022-05-05 23:54 ` Dave Chinner
2022-05-06 7:51 ` [PATCH] xfs/larp: Make test failures debuggable Dave Chinner
2 siblings, 0 replies; 15+ messages in thread
From: Allison Henderson @ 2022-03-01 20:46 UTC (permalink / raw)
To: Catherine Hoang, linux-xfs, fstests
On 2/22/22 8:37 PM, Catherine Hoang wrote:
> From: Allison Henderson <allison.henderson@oracle.com>
>
> This patch adds tests to exercise the log attribute error
> inject and log replay. These tests aim to cover cases where attributes
> are added, removed, and overwritten in each format (shortform, leaf,
> node). Error inject is used to replay these operations from the log.
>
> Signed-off-by: Allison Henderson <allison.henderson@oracle.com>
> Signed-off-by: Catherine Hoang <catherine.hoang@oracle.com>
This looks good to me, though I can't give an rvb as a co-author. We'll
need to get the kernel/user space features merged before the test can
run though. It seems Darrick has found a bug in generic/467, so we'll
need to get that weeded out before the rest can follow. We'll chat more
in our meeting today.
In the meantime, I encourgage reviews for the testcase since we will
eventually need it.
Thanks all!
Allison
> ---
> tests/xfs/543 | 176 ++++++++++++++++++++++++++++++++++++++++++++++
> tests/xfs/543.out | 149 +++++++++++++++++++++++++++++++++++++++
> 2 files changed, 325 insertions(+)
> create mode 100755 tests/xfs/543
> create mode 100644 tests/xfs/543.out
>
> diff --git a/tests/xfs/543 b/tests/xfs/543
> new file mode 100755
> index 00000000..06f16f21
> --- /dev/null
> +++ b/tests/xfs/543
> @@ -0,0 +1,176 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2022, Oracle and/or its affiliates. All Rights Reserved.
> +#
> +# FS QA Test 543
> +#
> +# Log attribute replay test
> +#
> +. ./common/preamble
> +_begin_fstest auto quick attr
> +
> +# get standard environment, filters and checks
> +. ./common/filter
> +. ./common/attr
> +. ./common/inject
> +
> +_cleanup()
> +{
> + rm -rf $tmp.* $testdir
> + test -w /sys/fs/xfs/debug/larp && \
> + echo 0 > /sys/fs/xfs/debug/larp
> +}
> +
> +test_attr_replay()
> +{
> + testfile=$testdir/$1
> + attr_name=$2
> + attr_value=$3
> + flag=$4
> + error_tag=$5
> +
> + # Inject error
> + _scratch_inject_error $error_tag
> +
> + # Set attribute
> + echo "$attr_value" | ${ATTR_PROG} -$flag "$attr_name" $testfile 2>&1 | \
> + _filter_scratch
> +
> + # FS should be shut down, touch will fail
> + touch $testfile 2>&1 | _filter_scratch
> +
> + # Remount to replay log
> + _scratch_remount_dump_log >> $seqres.full
> +
> + # FS should be online, touch should succeed
> + touch $testfile
> +
> + # Verify attr recovery
> + { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 | _filter_scratch
> +
> + echo ""
> +}
> +
> +create_test_file()
> +{
> + filename=$testdir/$1
> + count=$2
> + attr_value=$3
> +
> + touch $filename
> +
> + for i in `seq $count`
> + do
> + $ATTR_PROG -s "attr_name$i" -V $attr_value $filename >> \
> + $seqres.full
> + done
> +}
> +
> +# real QA test starts here
> +_supported_fs xfs
> +
> +_require_scratch
> +_require_attrs
> +_require_xfs_io_error_injection "larp"
> +_require_xfs_io_error_injection "da_leaf_split"
> +_require_xfs_io_error_injection "attr_leaf_to_node"
> +_require_xfs_sysfs debug/larp
> +test -w /sys/fs/xfs/debug/larp || _notrun "larp knob not writable"
> +
> +# turn on log attributes
> +echo 1 > /sys/fs/xfs/debug/larp
> +
> +attr16="0123456789ABCDEF"
> +attr64="$attr16$attr16$attr16$attr16"
> +attr256="$attr64$attr64$attr64$attr64"
> +attr1k="$attr256$attr256$attr256$attr256"
> +attr4k="$attr1k$attr1k$attr1k$attr1k"
> +attr8k="$attr4k$attr4k"
> +attr16k="$attr8k$attr8k"
> +attr32k="$attr16k$attr16k"
> +attr64k="$attr32k$attr32k"
> +
> +echo "*** mkfs"
> +_scratch_mkfs >/dev/null
> +
> +echo "*** mount FS"
> +_scratch_mount
> +
> +testdir=$SCRATCH_MNT/testdir
> +mkdir $testdir
> +
> +# empty, inline
> +create_test_file empty_file1 0
> +test_attr_replay empty_file1 "attr_name" $attr64 "s" "larp"
> +test_attr_replay empty_file1 "attr_name" $attr64 "r" "larp"
> +
> +# empty, internal
> +create_test_file empty_file2 0
> +test_attr_replay empty_file2 "attr_name" $attr1k "s" "larp"
> +test_attr_replay empty_file2 "attr_name" $attr1k "r" "larp"
> +
> +# empty, remote
> +create_test_file empty_file3 0
> +test_attr_replay empty_file3 "attr_name" $attr64k "s" "larp"
> +test_attr_replay empty_file3 "attr_name" $attr64k "r" "larp"
> +
> +# inline, inline
> +create_test_file inline_file1 1 $attr16
> +test_attr_replay inline_file1 "attr_name2" $attr64 "s" "larp"
> +test_attr_replay inline_file1 "attr_name2" $attr64 "r" "larp"
> +
> +# inline, internal
> +create_test_file inline_file2 1 $attr16
> +test_attr_replay inline_file2 "attr_name2" $attr1k "s" "larp"
> +test_attr_replay inline_file2 "attr_name2" $attr1k "r" "larp"
> +
> +# inline, remote
> +create_test_file inline_file3 1 $attr16
> +test_attr_replay inline_file3 "attr_name2" $attr64k "s" "larp"
> +test_attr_replay inline_file3 "attr_name2" $attr64k "r" "larp"
> +
> +# extent, internal
> +create_test_file extent_file1 1 $attr1k
> +test_attr_replay extent_file1 "attr_name2" $attr1k "s" "larp"
> +test_attr_replay extent_file1 "attr_name2" $attr1k "r" "larp"
> +
> +# extent, inject error on split
> +create_test_file extent_file2 3 $attr1k
> +test_attr_replay extent_file2 "attr_name4" $attr1k "s" "da_leaf_split"
> +
> +# extent, inject error on fork transition
> +create_test_file extent_file3 3 $attr1k
> +test_attr_replay extent_file3 "attr_name4" $attr1k "s" "attr_leaf_to_node"
> +
> +# extent, remote
> +create_test_file extent_file4 1 $attr1k
> +test_attr_replay extent_file4 "attr_name2" $attr64k "s" "larp"
> +test_attr_replay extent_file4 "attr_name2" $attr64k "r" "larp"
> +
> +# remote, internal
> +create_test_file remote_file1 1 $attr64k
> +test_attr_replay remote_file1 "attr_name2" $attr1k "s" "larp"
> +test_attr_replay remote_file1 "attr_name2" $attr1k "r" "larp"
> +
> +# remote, remote
> +create_test_file remote_file2 1 $attr64k
> +test_attr_replay remote_file2 "attr_name2" $attr64k "s" "larp"
> +test_attr_replay remote_file2 "attr_name2" $attr64k "r" "larp"
> +
> +# replace shortform
> +create_test_file sf_file 2 $attr64
> +test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
> +
> +# replace leaf
> +create_test_file leaf_file 2 $attr1k
> +test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
> +
> +# replace node
> +create_test_file node_file 1 $attr64k
> +$ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
> + >> $seqres.full
> +test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
> +
> +echo "*** done"
> +status=0
> +exit
> diff --git a/tests/xfs/543.out b/tests/xfs/543.out
> new file mode 100644
> index 00000000..1c74e795
> --- /dev/null
> +++ b/tests/xfs/543.out
> @@ -0,0 +1,149 @@
> +QA output created by 543
> +*** mkfs
> +*** mount FS
> +attr_set: Input/output error
> +Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> +touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> +21d850f99c43cc13abbe34838a8a3c8a -
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file1
> +touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> +attr_get: No data available
> +Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file1
> +d41d8cd98f00b204e9800998ecf8427e -
> +
> +attr_set: Input/output error
> +Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
> +touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> +2ff89c2935debc431745ec791be5421a -
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file2
> +touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> +attr_get: No data available
> +Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file2
> +d41d8cd98f00b204e9800998ecf8427e -
> +
> +attr_set: Input/output error
> +Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
> +touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> +5d24b314242c52176c98ac4bd685da8b -
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file3
> +touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> +attr_get: No data available
> +Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file3
> +d41d8cd98f00b204e9800998ecf8427e -
> +
> +attr_set: Input/output error
> +Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> +touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> +5a7b559a70d8e92b4f3c6f7158eead08 -
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> +touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> +attr_get: No data available
> +Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> +d41d8cd98f00b204e9800998ecf8427e -
> +
> +attr_set: Input/output error
> +Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> +touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> +5717d5e66c70be6bdb00ecbaca0b7749 -
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> +touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> +attr_get: No data available
> +Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> +d41d8cd98f00b204e9800998ecf8427e -
> +
> +attr_set: Input/output error
> +Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> +touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> +5c929964efd1b243aa8cceb6524f4810 -
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> +touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> +attr_get: No data available
> +Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> +d41d8cd98f00b204e9800998ecf8427e -
> +
> +attr_set: Input/output error
> +Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> +touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> +51ccb5cdfc9082060f0f94a8a108fea0 -
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> +touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> +attr_get: No data available
> +Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> +d41d8cd98f00b204e9800998ecf8427e -
> +
> +attr_set: Input/output error
> +Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file2
> +touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2': Input/output error
> +8d530bbe852d8bca83b131d5b3e497f5 -
> +
> +attr_set: Input/output error
> +Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file3
> +touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3': Input/output error
> +5d77c4d3831a35bcbbd6e7677119ce9a -
> +
> +attr_set: Input/output error
> +Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> +touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> +6707ec2431e4dbea20e17da0816520bb -
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> +touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> +attr_get: No data available
> +Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> +d41d8cd98f00b204e9800998ecf8427e -
> +
> +attr_set: Input/output error
> +Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> +touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> +767ebca3e4a6d24170857364f2bf2a3c -
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> +touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> +attr_get: No data available
> +Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> +d41d8cd98f00b204e9800998ecf8427e -
> +
> +attr_set: Input/output error
> +Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> +touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> +fd84ddec89237e6d34a1703639efaebf -
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> +touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> +attr_get: No data available
> +Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> +d41d8cd98f00b204e9800998ecf8427e -
> +
> +attr_set: Input/output error
> +Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
> +touch: cannot touch 'SCRATCH_MNT/testdir/sf_file': Input/output error
> +34aaa49662bafb46c76e377454685071 -
> +
> +attr_set: Input/output error
> +Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
> +touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file': Input/output error
> +664e95ec28830ffb367c0950026e0d21 -
> +
> +attr_set: Input/output error
> +Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
> +touch: cannot touch 'SCRATCH_MNT/testdir/node_file': Input/output error
> +bb37a78ce26472eeb711e3559933db42 -
> +
> +*** done
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v7 1/1] xfstests: Add Log Attribute Replay test
2022-02-23 3:37 ` [PATCH v7 1/1] xfstests: Add Log Attribute Replay test Catherine Hoang
2022-03-01 20:46 ` Allison Henderson
@ 2022-05-05 23:54 ` Dave Chinner
2022-05-06 7:51 ` [PATCH] xfs/larp: Make test failures debuggable Dave Chinner
2 siblings, 0 replies; 15+ messages in thread
From: Dave Chinner @ 2022-05-05 23:54 UTC (permalink / raw)
To: Catherine Hoang; +Cc: linux-xfs, fstests
On Wed, Feb 23, 2022 at 03:37:51AM +0000, Catherine Hoang wrote:
> From: Allison Henderson <allison.henderson@oracle.com>
>
> This patch adds tests to exercise the log attribute error
> inject and log replay. These tests aim to cover cases where attributes
> are added, removed, and overwritten in each format (shortform, leaf,
> node). Error inject is used to replay these operations from the log.
>
> Signed-off-by: Allison Henderson <allison.henderson@oracle.com>
> Signed-off-by: Catherine Hoang <catherine.hoang@oracle.com>
> ---
> tests/xfs/543 | 176 ++++++++++++++++++++++++++++++++++++++++++++++
> tests/xfs/543.out | 149 +++++++++++++++++++++++++++++++++++++++
> 2 files changed, 325 insertions(+)
> create mode 100755 tests/xfs/543
> create mode 100644 tests/xfs/543.out
>
> diff --git a/tests/xfs/543 b/tests/xfs/543
> new file mode 100755
> index 00000000..06f16f21
> --- /dev/null
> +++ b/tests/xfs/543
> @@ -0,0 +1,176 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2022, Oracle and/or its affiliates. All Rights Reserved.
> +#
> +# FS QA Test 543
> +#
> +# Log attribute replay test
> +#
> +. ./common/preamble
> +_begin_fstest auto quick attr
> +
> +# get standard environment, filters and checks
> +. ./common/filter
> +. ./common/attr
> +. ./common/inject
> +
> +_cleanup()
> +{
> + rm -rf $tmp.* $testdir
> + test -w /sys/fs/xfs/debug/larp && \
> + echo 0 > /sys/fs/xfs/debug/larp
This is problematic - I set larp=1 before I start running fstests
so all tests exercise the LARP paths. This will turn off LARP
unconditionally, so after this runs nothing will exercise the LARP
paths, even if that's what I want the tests to do.
Also, this does not replicate what the generic _cleanup() function
does.
Also, no need to remove $testdir - it's on the scratch device.
Also, don't call things "testdir" because "test directory" has
specific meaning - i.e. the *test device mount* - and this is too
easy to confuse when reading the test code.
> +}
> +
> +test_attr_replay()
> +{
> + testfile=$testdir/$1
> + attr_name=$2
> + attr_value=$3
> + flag=$4
> + error_tag=$5
> +
> + # Inject error
> + _scratch_inject_error $error_tag
> +
> + # Set attribute
> + echo "$attr_value" | ${ATTR_PROG} -$flag "$attr_name" $testfile 2>&1 | \
> + _filter_scratch
> +
> + # FS should be shut down, touch will fail
> + touch $testfile 2>&1 | _filter_scratch
> +
> + # Remount to replay log
> + _scratch_remount_dump_log >> $seqres.full
> +
> + # FS should be online, touch should succeed
> + touch $testfile
> +
> + # Verify attr recovery
> + { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 | _filter_scratch
> +
> + echo ""
> +}
> +
> +create_test_file()
> +{
> + filename=$testdir/$1
> + count=$2
> + attr_value=$3
> +
> + touch $filename
> +
> + for i in `seq $count`
> + do
> + $ATTR_PROG -s "attr_name$i" -V $attr_value $filename >> \
> + $seqres.full
> + done
> +}
> +
> +# real QA test starts here
> +_supported_fs xfs
> +
> +_require_scratch
> +_require_attrs
> +_require_xfs_io_error_injection "larp"
> +_require_xfs_io_error_injection "da_leaf_split"
> +_require_xfs_io_error_injection "attr_leaf_to_node"
> +_require_xfs_sysfs debug/larp
These go first, before any code.
> +test -w /sys/fs/xfs/debug/larp || _notrun "larp knob not writable"
When is the sysfs not writeable?
> +# turn on log attributes
> +echo 1 > /sys/fs/xfs/debug/larp
Needs to store the previous value so it can be restored at cleanup.
> +attr16="0123456789ABCDEF"
> +attr64="$attr16$attr16$attr16$attr16"
> +attr256="$attr64$attr64$attr64$attr64"
> +attr1k="$attr256$attr256$attr256$attr256"
> +attr4k="$attr1k$attr1k$attr1k$attr1k"
> +attr8k="$attr4k$attr4k"
> +attr16k="$attr8k$attr8k"
> +attr32k="$attr16k$attr16k"
> +attr64k="$attr32k$attr32k"
> +
> +echo "*** mkfs"
> +_scratch_mkfs >/dev/null
> +
> +echo "*** mount FS"
> +_scratch_mount
There's no need for echo lines like this in the golden output - it's
obvious what failed from the diff...
> +testdir=$SCRATCH_MNT/testdir
> +mkdir $testdir
Why not just use $SCRATCH_MNT directly?
> +
> +# empty, inline
> +create_test_file empty_file1 0
> +test_attr_replay empty_file1 "attr_name" $attr64 "s" "larp"
> +test_attr_replay empty_file1 "attr_name" $attr64 "r" "larp"
> +
> +# empty, internal
> +create_test_file empty_file2 0
> +test_attr_replay empty_file2 "attr_name" $attr1k "s" "larp"
> +test_attr_replay empty_file2 "attr_name" $attr1k "r" "larp"
> +
> +# empty, remote
> +create_test_file empty_file3 0
> +test_attr_replay empty_file3 "attr_name" $attr64k "s" "larp"
> +test_attr_replay empty_file3 "attr_name" $attr64k "r" "larp"
single attr insert/remove. OK.
> +
> +# inline, inline
> +create_test_file inline_file1 1 $attr16
> +test_attr_replay inline_file1 "attr_name2" $attr64 "s" "larp"
> +test_attr_replay inline_file1 "attr_name2" $attr64 "r" "larp"
...
Insert/remove of a second attr and the format
conversions they cause. OK.
Doesn't check that the first xattr is retained and uncorrupted
anywhere.
> +# replace shortform
> +create_test_file sf_file 2 $attr64
> +test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
This only replaces with same size. We also need coverage
of replace with smaller size and larger size, as well as
replace causing sf -> leaf and sf -> remote attr format conversions.
> +# replace leaf
> +create_test_file leaf_file 2 $attr1k
> +test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
Need replace causing leaf -> sf (smaller) and well as leaf -> remote
attr (larger). Also leaf w/ remote attr -> sf (much smaller!).
> +# replace node
> +create_test_file node_file 1 $attr64k
> +$ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
> + >> $seqres.full
> +test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
Need node -> leaf, node -> sf and node w/ remote attr -> sf
> --- /dev/null
> +++ b/tests/xfs/543.out
> @@ -0,0 +1,149 @@
> +QA output created by 543
> +*** mkfs
> +*** mount FS
> +attr_set: Input/output error
> +Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> +touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
We don't really need to capture the error injection errors, all we
care about is that the recovered md5sum matches the expected md5sum:
> +21d850f99c43cc13abbe34838a8a3c8a -
That's the recovered md5sum, what was the expected md5sum of the
original attr value that we stored? i.e. how do we validate taht we
got the correct attr value?
> +
> +attr_remove: Input/output error
> +Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file1
> +touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> +attr_get: No data available
> +Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file1
> +d41d8cd98f00b204e9800998ecf8427e -
Why do we have md5sum output for an attr that does not exist? We
should be capturing the ENODATA error here, as that is the correct
response after recovery of a remove operation.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH] xfs/larp: Make test failures debuggable
2022-02-23 3:37 ` [PATCH v7 1/1] xfstests: Add Log Attribute Replay test Catherine Hoang
2022-03-01 20:46 ` Allison Henderson
2022-05-05 23:54 ` Dave Chinner
@ 2022-05-06 7:51 ` Dave Chinner
2022-05-06 16:14 ` Darrick J. Wong
` (2 more replies)
2 siblings, 3 replies; 15+ messages in thread
From: Dave Chinner @ 2022-05-06 7:51 UTC (permalink / raw)
To: Catherine Hoang; +Cc: linux-xfs, fstests
From: Dave Chinner <dchinner@redhat.com>
Md5sum output for attributes created combined program output and
attribute values. This adds variable path names to the md5sum, so
there's no way to tell if the md5sum is actually correct for the
given attribute value that is returned as it's not constant from
test to test. Hence we can't actually say that the output is correct
because we can't reproduce exactly what we are hashing easily.
Indeed, the last attr test in series (node w/ replace) had an
invalid md5sum. The attr value being produced after recovery was
correct, but the md5sum output match was failing. Golden output
appears to be wrong.
Fix this issue by seperately dumping all the attributes on the inode
via a list operation to indicate their size, then dump the value of
the test attribute directly to md5sum. This means the md5sum for
the attributes using the same fixed values are all identical, so
it's easy to tell if the md5sum for a given test is correct. We also
check that all attributes that should be present after recovery are
still there (e.g. checks recovery didn't trash innocent bystanders).
Further, the attribute replace tests replace an attribute with an
identical value, hence there is no way to tell if recovery has
resulted in the original being left behind or the new attribute
being fully recovered because both have the same name and value.
When replacing the attribute value, use a different sized value so
it is trivial to determine that we've recovered the new attribute
value correctly.
Also, the test runs on the scratch device - there is no need to
remove the testdir in _cleanup. Doing so prevents post-mortem
failure analysis because it burns the dead, broken corpse to ash and
leaves no way of to determine cause of death.
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
Hi Catherine,
These are all the mods I needed to make to be able to understand the
test failures I was getting as I debugged the new LARP recovery
algorithm I've written. You'll need to massage the test number in
this patch to apply it on top of your patch.
I haven't added any new test cases yet, nor have I done anything to
manage the larp sysfs knob, but we'll need to do those in the near
future.
Zorro, can you consider merging this test in the near future? We're
right at the point of merging the upstream kernel code and so really
need to start growing the test coverage of this feature, and this
test should simply not-run on kernels that don't have the feature
enabled....
Cheers,
Dave.
---
tests/xfs/600 | 20 +++++-----
tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++------------------
2 files changed, 85 insertions(+), 44 deletions(-)
diff --git a/tests/xfs/600 b/tests/xfs/600
index 252cdf27..84704646 100755
--- a/tests/xfs/600
+++ b/tests/xfs/600
@@ -16,7 +16,7 @@ _begin_fstest auto quick attr
_cleanup()
{
- rm -rf $tmp.* $testdir
+ rm -rf $tmp.*
test -w /sys/fs/xfs/debug/larp && \
echo 0 > /sys/fs/xfs/debug/larp
}
@@ -46,7 +46,9 @@ test_attr_replay()
touch $testfile
# Verify attr recovery
- { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 | _filter_scratch
+ $ATTR_PROG -l $testfile | _filter_scratch
+ echo -n "$attr_name: "
+ $ATTR_PROG -q -g $attr_name $testfile | md5sum;
echo ""
}
@@ -157,19 +159,19 @@ create_test_file remote_file2 1 $attr64k
test_attr_replay remote_file2 "attr_name2" $attr64k "s" "larp"
test_attr_replay remote_file2 "attr_name2" $attr64k "r" "larp"
-# replace shortform
+# replace shortform with different value
create_test_file sf_file 2 $attr64
-test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
+test_attr_replay sf_file "attr_name2" $attr16 "s" "larp"
-# replace leaf
-create_test_file leaf_file 2 $attr1k
-test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
+# replace leaf with different value
+create_test_file leaf_file 3 $attr1k
+test_attr_replay leaf_file "attr_name2" $attr256 "s" "larp"
-# replace node
+# replace node with a different value
create_test_file node_file 1 $attr64k
$ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
>> $seqres.full
-test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
+test_attr_replay node_file "attr_name2" $attr256 "s" "larp"
echo "*** done"
status=0
diff --git a/tests/xfs/600.out b/tests/xfs/600.out
index 96b1d7d9..fe25ea3e 100644
--- a/tests/xfs/600.out
+++ b/tests/xfs/600.out
@@ -4,146 +4,185 @@ QA output created by 600
attr_set: Input/output error
Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
-21d850f99c43cc13abbe34838a8a3c8a -
+Attribute "attr_name" has a 65 byte value for SCRATCH_MNT/testdir/empty_file1
+attr_name: cfbe2a33be4601d2b655d099a18378fc -
attr_remove: Input/output error
Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file1
touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
-attr_get: No data available
-Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file1
+attr_name: attr_get: No data available
+Could not get "attr_name" for /mnt/scratch/testdir/empty_file1
d41d8cd98f00b204e9800998ecf8427e -
attr_set: Input/output error
Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
-2ff89c2935debc431745ec791be5421a -
+Attribute "attr_name" has a 1025 byte value for SCRATCH_MNT/testdir/empty_file2
+attr_name: 9fd415c49d67afc4b78fad4055a3a376 -
attr_remove: Input/output error
Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file2
touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
-attr_get: No data available
-Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file2
+attr_name: attr_get: No data available
+Could not get "attr_name" for /mnt/scratch/testdir/empty_file2
d41d8cd98f00b204e9800998ecf8427e -
attr_set: Input/output error
Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
-5d24b314242c52176c98ac4bd685da8b -
+Attribute "attr_name" has a 65536 byte value for SCRATCH_MNT/testdir/empty_file3
+attr_name: 7f6fd1b6d872108bd44bd143cbcdfa19 -
attr_remove: Input/output error
Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file3
touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
-attr_get: No data available
-Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file3
+attr_name: attr_get: No data available
+Could not get "attr_name" for /mnt/scratch/testdir/empty_file3
d41d8cd98f00b204e9800998ecf8427e -
attr_set: Input/output error
Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file1
touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
-5a7b559a70d8e92b4f3c6f7158eead08 -
+Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
+Attribute "attr_name2" has a 65 byte value for SCRATCH_MNT/testdir/inline_file1
+attr_name2: cfbe2a33be4601d2b655d099a18378fc -
attr_remove: Input/output error
Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file1
touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
-attr_get: No data available
-Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file1
+Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
+attr_name2: attr_get: No data available
+Could not get "attr_name2" for /mnt/scratch/testdir/inline_file1
d41d8cd98f00b204e9800998ecf8427e -
attr_set: Input/output error
Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file2
touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
-5717d5e66c70be6bdb00ecbaca0b7749 -
+Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/inline_file2
+Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
+attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
attr_remove: Input/output error
Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file2
touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
-attr_get: No data available
-Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file2
+Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
+attr_name2: attr_get: No data available
+Could not get "attr_name2" for /mnt/scratch/testdir/inline_file2
d41d8cd98f00b204e9800998ecf8427e -
attr_set: Input/output error
Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file3
touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
-5c929964efd1b243aa8cceb6524f4810 -
+Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/inline_file3
+Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
+attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
attr_remove: Input/output error
Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file3
touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
-attr_get: No data available
-Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file3
+Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
+attr_name2: attr_get: No data available
+Could not get "attr_name2" for /mnt/scratch/testdir/inline_file3
d41d8cd98f00b204e9800998ecf8427e -
attr_set: Input/output error
Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file1
touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
-51ccb5cdfc9082060f0f94a8a108fea0 -
+Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file1
+Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
+attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
attr_remove: Input/output error
Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file1
touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
-attr_get: No data available
-Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file1
+Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
+attr_name2: attr_get: No data available
+Could not get "attr_name2" for /mnt/scratch/testdir/extent_file1
d41d8cd98f00b204e9800998ecf8427e -
attr_set: Input/output error
Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file2
touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2': Input/output error
-8d530bbe852d8bca83b131d5b3e497f5 -
+Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file2
+Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
+Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
+Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
+attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
attr_set: Input/output error
Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file3
touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3': Input/output error
-5d77c4d3831a35bcbbd6e7677119ce9a -
+Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file3
+Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
+Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
+Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
+attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
attr_set: Input/output error
Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file4
touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
-6707ec2431e4dbea20e17da0816520bb -
+Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/extent_file4
+Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
+attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
attr_remove: Input/output error
Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file4
touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
-attr_get: No data available
-Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file4
+Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
+attr_name2: attr_get: No data available
+Could not get "attr_name2" for /mnt/scratch/testdir/extent_file4
d41d8cd98f00b204e9800998ecf8427e -
attr_set: Input/output error
Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file1
touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
-767ebca3e4a6d24170857364f2bf2a3c -
+Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/remote_file1
+Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
+attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
attr_remove: Input/output error
Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file1
touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
-attr_get: No data available
-Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file1
+Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
+attr_name2: attr_get: No data available
+Could not get "attr_name2" for /mnt/scratch/testdir/remote_file1
d41d8cd98f00b204e9800998ecf8427e -
attr_set: Input/output error
Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file2
touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
-fd84ddec89237e6d34a1703639efaebf -
+Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
+Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
+attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
attr_remove: Input/output error
Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file2
touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
-attr_get: No data available
-Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file2
+Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
+attr_name2: attr_get: No data available
+Could not get "attr_name2" for /mnt/scratch/testdir/remote_file2
d41d8cd98f00b204e9800998ecf8427e -
attr_set: Input/output error
Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
touch: cannot touch 'SCRATCH_MNT/testdir/sf_file': Input/output error
-34aaa49662bafb46c76e377454685071 -
+Attribute "attr_name1" has a 64 byte value for SCRATCH_MNT/testdir/sf_file
+Attribute "attr_name2" has a 17 byte value for SCRATCH_MNT/testdir/sf_file
+attr_name2: 9a6eb1bc9da3c66a9b495dfe2fe8a756 -
attr_set: Input/output error
Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file': Input/output error
-664e95ec28830ffb367c0950026e0d21 -
+Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/leaf_file
+Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
+Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
+attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
attr_set: Input/output error
Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
touch: cannot touch 'SCRATCH_MNT/testdir/node_file': Input/output error
-bb37a78ce26472eeb711e3559933db42 -
+Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/node_file
+Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/node_file
+attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
*** done
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH] xfs/larp: Make test failures debuggable
2022-05-06 7:51 ` [PATCH] xfs/larp: Make test failures debuggable Dave Chinner
@ 2022-05-06 16:14 ` Darrick J. Wong
2022-05-06 16:40 ` Zorro Lang
2022-05-06 22:16 ` Dave Chinner
2022-05-06 16:28 ` Zorro Lang
2022-05-06 17:58 ` Catherine Hoang
2 siblings, 2 replies; 15+ messages in thread
From: Darrick J. Wong @ 2022-05-06 16:14 UTC (permalink / raw)
To: Dave Chinner; +Cc: Catherine Hoang, linux-xfs, fstests
On Fri, May 06, 2022 at 05:51:41PM +1000, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
>
> Md5sum output for attributes created combined program output and
> attribute values. This adds variable path names to the md5sum, so
> there's no way to tell if the md5sum is actually correct for the
> given attribute value that is returned as it's not constant from
> test to test. Hence we can't actually say that the output is correct
> because we can't reproduce exactly what we are hashing easily.
>
> Indeed, the last attr test in series (node w/ replace) had an
> invalid md5sum. The attr value being produced after recovery was
> correct, but the md5sum output match was failing. Golden output
> appears to be wrong.
>
> Fix this issue by seperately dumping all the attributes on the inode
> via a list operation to indicate their size, then dump the value of
> the test attribute directly to md5sum. This means the md5sum for
> the attributes using the same fixed values are all identical, so
> it's easy to tell if the md5sum for a given test is correct. We also
> check that all attributes that should be present after recovery are
> still there (e.g. checks recovery didn't trash innocent bystanders).
>
> Further, the attribute replace tests replace an attribute with an
> identical value, hence there is no way to tell if recovery has
> resulted in the original being left behind or the new attribute
> being fully recovered because both have the same name and value.
> When replacing the attribute value, use a different sized value so
> it is trivial to determine that we've recovered the new attribute
> value correctly.
>
> Also, the test runs on the scratch device - there is no need to
> remove the testdir in _cleanup. Doing so prevents post-mortem
> failure analysis because it burns the dead, broken corpse to ash and
> leaves no way of to determine cause of death.
>
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
>
> Hi Catherine,
>
> These are all the mods I needed to make to be able to understand the
> test failures I was getting as I debugged the new LARP recovery
> algorithm I've written. You'll need to massage the test number in
> this patch to apply it on top of your patch.
>
> I haven't added any new test cases yet, nor have I done anything to
> manage the larp sysfs knob, but we'll need to do those in the near
> future.
>
> Zorro, can you consider merging this test in the near future? We're
> right at the point of merging the upstream kernel code and so really
> need to start growing the test coverage of this feature, and this
> test should simply not-run on kernels that don't have the feature
> enabled....
>
> Cheers,
>
> Dave.
> ---
>
> tests/xfs/600 | 20 +++++-----
> tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++------------------
> 2 files changed, 85 insertions(+), 44 deletions(-)
>
> diff --git a/tests/xfs/600 b/tests/xfs/600
> index 252cdf27..84704646 100755
> --- a/tests/xfs/600
> +++ b/tests/xfs/600
> @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
>
> _cleanup()
> {
> - rm -rf $tmp.* $testdir
> + rm -rf $tmp.*
> test -w /sys/fs/xfs/debug/larp && \
> echo 0 > /sys/fs/xfs/debug/larp
Blergh, this ^^^^^^^^^ is going to need fixing too.
Please save the old value, then write it back in the _cleanup function.
<slightly ot rant>
These sysfs knobs are a pain because they all reset to defaults if
xfs.ko gets cycled. I know, I know, at least Dave and Ted don't do
modular kernels and so never see this happen, but I do. I bet Dave also
hasn't ever run xfs/434 or xfs/436, which might be why I'm the only one
seeing dquot leaks with 5.19-next.
(I might be able to lift the xfs-as-module requirement if "echo 1 >
/sys/kernel/slab/*/validate" does what I think it might, since all we
want to do is look for slab leaks, and those tests rmmod/modprobe as a
brute force way of making the slab debug code check for leaks.)
In case anyone's wondering, a solution to the knobs getting unset after
an rmmod/modprobe cycle is to add a file
/etc/udev/rules.d/99-fstester.rules containing:
ACTION=="add|change", SUBSYSTEM=="module", DEVPATH=="/module/${module}", RUN+="/bin/sh -c \"echo ${value} > ${knob}\""
which should be enough to keep LARP turned on.
<end rant>
Anyway, since this is a proposed test, I say that with this applied and
the debug knob bits fixed, the whole thing is
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
--D
> }
> @@ -46,7 +46,9 @@ test_attr_replay()
> touch $testfile
>
> # Verify attr recovery
> - { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 | _filter_scratch
> + $ATTR_PROG -l $testfile | _filter_scratch
> + echo -n "$attr_name: "
> + $ATTR_PROG -q -g $attr_name $testfile | md5sum;
>
> echo ""
> }
> @@ -157,19 +159,19 @@ create_test_file remote_file2 1 $attr64k
> test_attr_replay remote_file2 "attr_name2" $attr64k "s" "larp"
> test_attr_replay remote_file2 "attr_name2" $attr64k "r" "larp"
>
> -# replace shortform
> +# replace shortform with different value
> create_test_file sf_file 2 $attr64
> -test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
> +test_attr_replay sf_file "attr_name2" $attr16 "s" "larp"
>
> -# replace leaf
> -create_test_file leaf_file 2 $attr1k
> -test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
> +# replace leaf with different value
> +create_test_file leaf_file 3 $attr1k
> +test_attr_replay leaf_file "attr_name2" $attr256 "s" "larp"
>
> -# replace node
> +# replace node with a different value
> create_test_file node_file 1 $attr64k
> $ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
> >> $seqres.full
> -test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
> +test_attr_replay node_file "attr_name2" $attr256 "s" "larp"
>
> echo "*** done"
> status=0
> diff --git a/tests/xfs/600.out b/tests/xfs/600.out
> index 96b1d7d9..fe25ea3e 100644
> --- a/tests/xfs/600.out
> +++ b/tests/xfs/600.out
> @@ -4,146 +4,185 @@ QA output created by 600
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> -21d850f99c43cc13abbe34838a8a3c8a -
> +Attribute "attr_name" has a 65 byte value for SCRATCH_MNT/testdir/empty_file1
> +attr_name: cfbe2a33be4601d2b655d099a18378fc -
>
> attr_remove: Input/output error
> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file1
> +attr_name: attr_get: No data available
> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> -2ff89c2935debc431745ec791be5421a -
> +Attribute "attr_name" has a 1025 byte value for SCRATCH_MNT/testdir/empty_file2
> +attr_name: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> -attr_get: No data available
> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file2
> +attr_name: attr_get: No data available
> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file2
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> -5d24b314242c52176c98ac4bd685da8b -
> +Attribute "attr_name" has a 65536 byte value for SCRATCH_MNT/testdir/empty_file3
> +attr_name: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> -attr_get: No data available
> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file3
> +attr_name: attr_get: No data available
> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file3
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> -5a7b559a70d8e92b4f3c6f7158eead08 -
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
> +Attribute "attr_name2" has a 65 byte value for SCRATCH_MNT/testdir/inline_file1
> +attr_name2: cfbe2a33be4601d2b655d099a18378fc -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> -5717d5e66c70be6bdb00ecbaca0b7749 -
> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/inline_file2
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file2
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> -5c929964efd1b243aa8cceb6524f4810 -
> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/inline_file3
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file3
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> -51ccb5cdfc9082060f0f94a8a108fea0 -
> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file1
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2': Input/output error
> -8d530bbe852d8bca83b131d5b3e497f5 -
> +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file2
> +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_set: Input/output error
> Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3': Input/output error
> -5d77c4d3831a35bcbbd6e7677119ce9a -
> +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file3
> +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> -6707ec2431e4dbea20e17da0816520bb -
> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/extent_file4
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file4
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> -767ebca3e4a6d24170857364f2bf2a3c -
> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/remote_file1
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> -fd84ddec89237e6d34a1703639efaebf -
> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file2
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
> touch: cannot touch 'SCRATCH_MNT/testdir/sf_file': Input/output error
> -34aaa49662bafb46c76e377454685071 -
> +Attribute "attr_name1" has a 64 byte value for SCRATCH_MNT/testdir/sf_file
> +Attribute "attr_name2" has a 17 byte value for SCRATCH_MNT/testdir/sf_file
> +attr_name2: 9a6eb1bc9da3c66a9b495dfe2fe8a756 -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
> touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file': Input/output error
> -664e95ec28830ffb367c0950026e0d21 -
> +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/leaf_file
> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
> +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
> touch: cannot touch 'SCRATCH_MNT/testdir/node_file': Input/output error
> -bb37a78ce26472eeb711e3559933db42 -
> +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/node_file
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/node_file
> +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
>
> *** done
> --
> Dave Chinner
> david@fromorbit.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] xfs/larp: Make test failures debuggable
2022-05-06 7:51 ` [PATCH] xfs/larp: Make test failures debuggable Dave Chinner
2022-05-06 16:14 ` Darrick J. Wong
@ 2022-05-06 16:28 ` Zorro Lang
2022-05-06 17:58 ` Catherine Hoang
2 siblings, 0 replies; 15+ messages in thread
From: Zorro Lang @ 2022-05-06 16:28 UTC (permalink / raw)
To: Dave Chinner; +Cc: Catherine Hoang, linux-xfs, fstests
On Fri, May 06, 2022 at 05:51:41PM +1000, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
>
> Md5sum output for attributes created combined program output and
> attribute values. This adds variable path names to the md5sum, so
> there's no way to tell if the md5sum is actually correct for the
> given attribute value that is returned as it's not constant from
> test to test. Hence we can't actually say that the output is correct
> because we can't reproduce exactly what we are hashing easily.
>
> Indeed, the last attr test in series (node w/ replace) had an
> invalid md5sum. The attr value being produced after recovery was
> correct, but the md5sum output match was failing. Golden output
> appears to be wrong.
>
> Fix this issue by seperately dumping all the attributes on the inode
> via a list operation to indicate their size, then dump the value of
> the test attribute directly to md5sum. This means the md5sum for
> the attributes using the same fixed values are all identical, so
> it's easy to tell if the md5sum for a given test is correct. We also
> check that all attributes that should be present after recovery are
> still there (e.g. checks recovery didn't trash innocent bystanders).
>
> Further, the attribute replace tests replace an attribute with an
> identical value, hence there is no way to tell if recovery has
> resulted in the original being left behind or the new attribute
> being fully recovered because both have the same name and value.
> When replacing the attribute value, use a different sized value so
> it is trivial to determine that we've recovered the new attribute
> value correctly.
>
> Also, the test runs on the scratch device - there is no need to
> remove the testdir in _cleanup. Doing so prevents post-mortem
> failure analysis because it burns the dead, broken corpse to ash and
> leaves no way of to determine cause of death.
>
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
>
> Hi Catherine,
>
> These are all the mods I needed to make to be able to understand the
> test failures I was getting as I debugged the new LARP recovery
> algorithm I've written. You'll need to massage the test number in
> this patch to apply it on top of your patch.
>
> I haven't added any new test cases yet, nor have I done anything to
> manage the larp sysfs knob, but we'll need to do those in the near
> future.
>
> Zorro, can you consider merging this test in the near future? We're
> right at the point of merging the upstream kernel code and so really
> need to start growing the test coverage of this feature, and this
> test should simply not-run on kernels that don't have the feature
> enabled....
Sure, due to LSF meeting, we have few of reviewed patches this week.
I'll merge Catherine's "[PATCH v7 1/1] xfstests: Add Log Attribute Replay
test"(with your RVB) into fstests for-next branch at first, then this
patch ensued. I'll push it this weekend, after my basic testing pass. If
you'd like to combine this patch with Catherine's patch into one patch,
or you have more review points, feel free to tell me.
Thanks,
Zorro
>
> Cheers,
>
> Dave.
> ---
>
> tests/xfs/600 | 20 +++++-----
> tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++------------------
> 2 files changed, 85 insertions(+), 44 deletions(-)
>
> diff --git a/tests/xfs/600 b/tests/xfs/600
> index 252cdf27..84704646 100755
> --- a/tests/xfs/600
> +++ b/tests/xfs/600
> @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
>
> _cleanup()
> {
> - rm -rf $tmp.* $testdir
> + rm -rf $tmp.*
> test -w /sys/fs/xfs/debug/larp && \
> echo 0 > /sys/fs/xfs/debug/larp
> }
> @@ -46,7 +46,9 @@ test_attr_replay()
> touch $testfile
>
> # Verify attr recovery
> - { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 | _filter_scratch
> + $ATTR_PROG -l $testfile | _filter_scratch
> + echo -n "$attr_name: "
> + $ATTR_PROG -q -g $attr_name $testfile | md5sum;
>
> echo ""
> }
> @@ -157,19 +159,19 @@ create_test_file remote_file2 1 $attr64k
> test_attr_replay remote_file2 "attr_name2" $attr64k "s" "larp"
> test_attr_replay remote_file2 "attr_name2" $attr64k "r" "larp"
>
> -# replace shortform
> +# replace shortform with different value
> create_test_file sf_file 2 $attr64
> -test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
> +test_attr_replay sf_file "attr_name2" $attr16 "s" "larp"
>
> -# replace leaf
> -create_test_file leaf_file 2 $attr1k
> -test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
> +# replace leaf with different value
> +create_test_file leaf_file 3 $attr1k
> +test_attr_replay leaf_file "attr_name2" $attr256 "s" "larp"
>
> -# replace node
> +# replace node with a different value
> create_test_file node_file 1 $attr64k
> $ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
> >> $seqres.full
> -test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
> +test_attr_replay node_file "attr_name2" $attr256 "s" "larp"
>
> echo "*** done"
> status=0
> diff --git a/tests/xfs/600.out b/tests/xfs/600.out
> index 96b1d7d9..fe25ea3e 100644
> --- a/tests/xfs/600.out
> +++ b/tests/xfs/600.out
> @@ -4,146 +4,185 @@ QA output created by 600
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> -21d850f99c43cc13abbe34838a8a3c8a -
> +Attribute "attr_name" has a 65 byte value for SCRATCH_MNT/testdir/empty_file1
> +attr_name: cfbe2a33be4601d2b655d099a18378fc -
>
> attr_remove: Input/output error
> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file1
> +attr_name: attr_get: No data available
> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> -2ff89c2935debc431745ec791be5421a -
> +Attribute "attr_name" has a 1025 byte value for SCRATCH_MNT/testdir/empty_file2
> +attr_name: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> -attr_get: No data available
> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file2
> +attr_name: attr_get: No data available
> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file2
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> -5d24b314242c52176c98ac4bd685da8b -
> +Attribute "attr_name" has a 65536 byte value for SCRATCH_MNT/testdir/empty_file3
> +attr_name: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> -attr_get: No data available
> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file3
> +attr_name: attr_get: No data available
> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file3
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> -5a7b559a70d8e92b4f3c6f7158eead08 -
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
> +Attribute "attr_name2" has a 65 byte value for SCRATCH_MNT/testdir/inline_file1
> +attr_name2: cfbe2a33be4601d2b655d099a18378fc -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> -5717d5e66c70be6bdb00ecbaca0b7749 -
> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/inline_file2
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file2
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> -5c929964efd1b243aa8cceb6524f4810 -
> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/inline_file3
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file3
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> -51ccb5cdfc9082060f0f94a8a108fea0 -
> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file1
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2': Input/output error
> -8d530bbe852d8bca83b131d5b3e497f5 -
> +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file2
> +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_set: Input/output error
> Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3': Input/output error
> -5d77c4d3831a35bcbbd6e7677119ce9a -
> +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file3
> +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> -6707ec2431e4dbea20e17da0816520bb -
> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/extent_file4
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file4
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> -767ebca3e4a6d24170857364f2bf2a3c -
> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/remote_file1
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> -fd84ddec89237e6d34a1703639efaebf -
> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file2
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
> touch: cannot touch 'SCRATCH_MNT/testdir/sf_file': Input/output error
> -34aaa49662bafb46c76e377454685071 -
> +Attribute "attr_name1" has a 64 byte value for SCRATCH_MNT/testdir/sf_file
> +Attribute "attr_name2" has a 17 byte value for SCRATCH_MNT/testdir/sf_file
> +attr_name2: 9a6eb1bc9da3c66a9b495dfe2fe8a756 -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
> touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file': Input/output error
> -664e95ec28830ffb367c0950026e0d21 -
> +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/leaf_file
> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
> +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
> touch: cannot touch 'SCRATCH_MNT/testdir/node_file': Input/output error
> -bb37a78ce26472eeb711e3559933db42 -
> +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/node_file
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/node_file
> +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
>
> *** done
> --
> Dave Chinner
> david@fromorbit.com
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] xfs/larp: Make test failures debuggable
2022-05-06 16:14 ` Darrick J. Wong
@ 2022-05-06 16:40 ` Zorro Lang
2022-05-06 18:08 ` Catherine Hoang
2022-05-06 22:16 ` Dave Chinner
1 sibling, 1 reply; 15+ messages in thread
From: Zorro Lang @ 2022-05-06 16:40 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: Dave Chinner, Catherine Hoang, linux-xfs, fstests
On Fri, May 06, 2022 at 09:14:42AM -0700, Darrick J. Wong wrote:
> On Fri, May 06, 2022 at 05:51:41PM +1000, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@redhat.com>
> >
> > Md5sum output for attributes created combined program output and
> > attribute values. This adds variable path names to the md5sum, so
> > there's no way to tell if the md5sum is actually correct for the
> > given attribute value that is returned as it's not constant from
> > test to test. Hence we can't actually say that the output is correct
> > because we can't reproduce exactly what we are hashing easily.
> >
> > Indeed, the last attr test in series (node w/ replace) had an
> > invalid md5sum. The attr value being produced after recovery was
> > correct, but the md5sum output match was failing. Golden output
> > appears to be wrong.
> >
> > Fix this issue by seperately dumping all the attributes on the inode
> > via a list operation to indicate their size, then dump the value of
> > the test attribute directly to md5sum. This means the md5sum for
> > the attributes using the same fixed values are all identical, so
> > it's easy to tell if the md5sum for a given test is correct. We also
> > check that all attributes that should be present after recovery are
> > still there (e.g. checks recovery didn't trash innocent bystanders).
> >
> > Further, the attribute replace tests replace an attribute with an
> > identical value, hence there is no way to tell if recovery has
> > resulted in the original being left behind or the new attribute
> > being fully recovered because both have the same name and value.
> > When replacing the attribute value, use a different sized value so
> > it is trivial to determine that we've recovered the new attribute
> > value correctly.
> >
> > Also, the test runs on the scratch device - there is no need to
> > remove the testdir in _cleanup. Doing so prevents post-mortem
> > failure analysis because it burns the dead, broken corpse to ash and
> > leaves no way of to determine cause of death.
> >
> > Signed-off-by: Dave Chinner <dchinner@redhat.com>
> > ---
> >
> > Hi Catherine,
> >
> > These are all the mods I needed to make to be able to understand the
> > test failures I was getting as I debugged the new LARP recovery
> > algorithm I've written. You'll need to massage the test number in
> > this patch to apply it on top of your patch.
> >
> > I haven't added any new test cases yet, nor have I done anything to
> > manage the larp sysfs knob, but we'll need to do those in the near
> > future.
> >
> > Zorro, can you consider merging this test in the near future? We're
> > right at the point of merging the upstream kernel code and so really
> > need to start growing the test coverage of this feature, and this
> > test should simply not-run on kernels that don't have the feature
> > enabled....
> >
> > Cheers,
> >
> > Dave.
> > ---
> >
> > tests/xfs/600 | 20 +++++-----
> > tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++------------------
> > 2 files changed, 85 insertions(+), 44 deletions(-)
> >
> > diff --git a/tests/xfs/600 b/tests/xfs/600
> > index 252cdf27..84704646 100755
> > --- a/tests/xfs/600
> > +++ b/tests/xfs/600
> > @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
> >
> > _cleanup()
> > {
> > - rm -rf $tmp.* $testdir
> > + rm -rf $tmp.*
> > test -w /sys/fs/xfs/debug/larp && \
> > echo 0 > /sys/fs/xfs/debug/larp
>
> Blergh, this ^^^^^^^^^ is going to need fixing too.
>
> Please save the old value, then write it back in the _cleanup function.
Ok, I'm going to do that when I merge it, if Catherine wouldn't like to do
more changes in a V8 patch. If this case still need more changes, please tell
me in time, and then it might have to wait the fstests release after next, if
too late.
Thanks,
Zorro
>
> <slightly ot rant>
>
> These sysfs knobs are a pain because they all reset to defaults if
> xfs.ko gets cycled. I know, I know, at least Dave and Ted don't do
> modular kernels and so never see this happen, but I do. I bet Dave also
> hasn't ever run xfs/434 or xfs/436, which might be why I'm the only one
> seeing dquot leaks with 5.19-next.
>
> (I might be able to lift the xfs-as-module requirement if "echo 1 >
> /sys/kernel/slab/*/validate" does what I think it might, since all we
> want to do is look for slab leaks, and those tests rmmod/modprobe as a
> brute force way of making the slab debug code check for leaks.)
>
> In case anyone's wondering, a solution to the knobs getting unset after
> an rmmod/modprobe cycle is to add a file
> /etc/udev/rules.d/99-fstester.rules containing:
>
> ACTION=="add|change", SUBSYSTEM=="module", DEVPATH=="/module/${module}", RUN+="/bin/sh -c \"echo ${value} > ${knob}\""
>
> which should be enough to keep LARP turned on.
>
> <end rant>
>
> Anyway, since this is a proposed test, I say that with this applied and
> the debug knob bits fixed, the whole thing is
> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
>
> --D
>
> > }
> > @@ -46,7 +46,9 @@ test_attr_replay()
> > touch $testfile
> >
> > # Verify attr recovery
> > - { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 | _filter_scratch
> > + $ATTR_PROG -l $testfile | _filter_scratch
> > + echo -n "$attr_name: "
> > + $ATTR_PROG -q -g $attr_name $testfile | md5sum;
> >
> > echo ""
> > }
> > @@ -157,19 +159,19 @@ create_test_file remote_file2 1 $attr64k
> > test_attr_replay remote_file2 "attr_name2" $attr64k "s" "larp"
> > test_attr_replay remote_file2 "attr_name2" $attr64k "r" "larp"
> >
> > -# replace shortform
> > +# replace shortform with different value
> > create_test_file sf_file 2 $attr64
> > -test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
> > +test_attr_replay sf_file "attr_name2" $attr16 "s" "larp"
> >
> > -# replace leaf
> > -create_test_file leaf_file 2 $attr1k
> > -test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
> > +# replace leaf with different value
> > +create_test_file leaf_file 3 $attr1k
> > +test_attr_replay leaf_file "attr_name2" $attr256 "s" "larp"
> >
> > -# replace node
> > +# replace node with a different value
> > create_test_file node_file 1 $attr64k
> > $ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
> > >> $seqres.full
> > -test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
> > +test_attr_replay node_file "attr_name2" $attr256 "s" "larp"
> >
> > echo "*** done"
> > status=0
> > diff --git a/tests/xfs/600.out b/tests/xfs/600.out
> > index 96b1d7d9..fe25ea3e 100644
> > --- a/tests/xfs/600.out
> > +++ b/tests/xfs/600.out
> > @@ -4,146 +4,185 @@ QA output created by 600
> > attr_set: Input/output error
> > Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> > -21d850f99c43cc13abbe34838a8a3c8a -
> > +Attribute "attr_name" has a 65 byte value for SCRATCH_MNT/testdir/empty_file1
> > +attr_name: cfbe2a33be4601d2b655d099a18378fc -
> >
> > attr_remove: Input/output error
> > Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file1
> > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> > -attr_get: No data available
> > -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file1
> > +attr_name: attr_get: No data available
> > +Could not get "attr_name" for /mnt/scratch/testdir/empty_file1
> > d41d8cd98f00b204e9800998ecf8427e -
> >
> > attr_set: Input/output error
> > Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
> > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> > -2ff89c2935debc431745ec791be5421a -
> > +Attribute "attr_name" has a 1025 byte value for SCRATCH_MNT/testdir/empty_file2
> > +attr_name: 9fd415c49d67afc4b78fad4055a3a376 -
> >
> > attr_remove: Input/output error
> > Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file2
> > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> > -attr_get: No data available
> > -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file2
> > +attr_name: attr_get: No data available
> > +Could not get "attr_name" for /mnt/scratch/testdir/empty_file2
> > d41d8cd98f00b204e9800998ecf8427e -
> >
> > attr_set: Input/output error
> > Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
> > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> > -5d24b314242c52176c98ac4bd685da8b -
> > +Attribute "attr_name" has a 65536 byte value for SCRATCH_MNT/testdir/empty_file3
> > +attr_name: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> >
> > attr_remove: Input/output error
> > Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file3
> > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> > -attr_get: No data available
> > -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file3
> > +attr_name: attr_get: No data available
> > +Could not get "attr_name" for /mnt/scratch/testdir/empty_file3
> > d41d8cd98f00b204e9800998ecf8427e -
> >
> > attr_set: Input/output error
> > Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> > -5a7b559a70d8e92b4f3c6f7158eead08 -
> > +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
> > +Attribute "attr_name2" has a 65 byte value for SCRATCH_MNT/testdir/inline_file1
> > +attr_name2: cfbe2a33be4601d2b655d099a18378fc -
> >
> > attr_remove: Input/output error
> > Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> > -attr_get: No data available
> > -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> > +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
> > +attr_name2: attr_get: No data available
> > +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file1
> > d41d8cd98f00b204e9800998ecf8427e -
> >
> > attr_set: Input/output error
> > Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> > -5717d5e66c70be6bdb00ecbaca0b7749 -
> > +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/inline_file2
> > +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
> > +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
> >
> > attr_remove: Input/output error
> > Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> > -attr_get: No data available
> > -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> > +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
> > +attr_name2: attr_get: No data available
> > +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file2
> > d41d8cd98f00b204e9800998ecf8427e -
> >
> > attr_set: Input/output error
> > Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> > -5c929964efd1b243aa8cceb6524f4810 -
> > +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/inline_file3
> > +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
> > +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> >
> > attr_remove: Input/output error
> > Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> > -attr_get: No data available
> > -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> > +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
> > +attr_name2: attr_get: No data available
> > +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file3
> > d41d8cd98f00b204e9800998ecf8427e -
> >
> > attr_set: Input/output error
> > Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> > -51ccb5cdfc9082060f0f94a8a108fea0 -
> > +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file1
> > +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
> > +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
> >
> > attr_remove: Input/output error
> > Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> > -attr_get: No data available
> > -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> > +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
> > +attr_name2: attr_get: No data available
> > +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file1
> > d41d8cd98f00b204e9800998ecf8427e -
> >
> > attr_set: Input/output error
> > Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file2
> > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2': Input/output error
> > -8d530bbe852d8bca83b131d5b3e497f5 -
> > +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file2
> > +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> > +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> > +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> > +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
> >
> > attr_set: Input/output error
> > Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file3
> > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3': Input/output error
> > -5d77c4d3831a35bcbbd6e7677119ce9a -
> > +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file3
> > +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> > +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> > +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> > +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
> >
> > attr_set: Input/output error
> > Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> > -6707ec2431e4dbea20e17da0816520bb -
> > +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/extent_file4
> > +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
> > +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> >
> > attr_remove: Input/output error
> > Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> > -attr_get: No data available
> > -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> > +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
> > +attr_name2: attr_get: No data available
> > +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file4
> > d41d8cd98f00b204e9800998ecf8427e -
> >
> > attr_set: Input/output error
> > Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> > -767ebca3e4a6d24170857364f2bf2a3c -
> > +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/remote_file1
> > +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
> > +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
> >
> > attr_remove: Input/output error
> > Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> > -attr_get: No data available
> > -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> > +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
> > +attr_name2: attr_get: No data available
> > +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file1
> > d41d8cd98f00b204e9800998ecf8427e -
> >
> > attr_set: Input/output error
> > Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> > -fd84ddec89237e6d34a1703639efaebf -
> > +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> > +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> > +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> >
> > attr_remove: Input/output error
> > Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> > -attr_get: No data available
> > -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> > +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> > +attr_name2: attr_get: No data available
> > +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file2
> > d41d8cd98f00b204e9800998ecf8427e -
> >
> > attr_set: Input/output error
> > Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
> > touch: cannot touch 'SCRATCH_MNT/testdir/sf_file': Input/output error
> > -34aaa49662bafb46c76e377454685071 -
> > +Attribute "attr_name1" has a 64 byte value for SCRATCH_MNT/testdir/sf_file
> > +Attribute "attr_name2" has a 17 byte value for SCRATCH_MNT/testdir/sf_file
> > +attr_name2: 9a6eb1bc9da3c66a9b495dfe2fe8a756 -
> >
> > attr_set: Input/output error
> > Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
> > touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file': Input/output error
> > -664e95ec28830ffb367c0950026e0d21 -
> > +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/leaf_file
> > +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
> > +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
> > +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
> >
> > attr_set: Input/output error
> > Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
> > touch: cannot touch 'SCRATCH_MNT/testdir/node_file': Input/output error
> > -bb37a78ce26472eeb711e3559933db42 -
> > +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/node_file
> > +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/node_file
> > +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
> >
> > *** done
> > --
> > Dave Chinner
> > david@fromorbit.com
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH] xfs/larp: Make test failures debuggable
2022-05-06 7:51 ` [PATCH] xfs/larp: Make test failures debuggable Dave Chinner
2022-05-06 16:14 ` Darrick J. Wong
2022-05-06 16:28 ` Zorro Lang
@ 2022-05-06 17:58 ` Catherine Hoang
2 siblings, 0 replies; 15+ messages in thread
From: Catherine Hoang @ 2022-05-06 17:58 UTC (permalink / raw)
To: Dave Chinner; +Cc: linux-xfs, fstests
> On May 6, 2022, at 12:51 AM, Dave Chinner <david@fromorbit.com> wrote:
>
> From: Dave Chinner <dchinner@redhat.com>
>
> Md5sum output for attributes created combined program output and
> attribute values. This adds variable path names to the md5sum, so
> there's no way to tell if the md5sum is actually correct for the
> given attribute value that is returned as it's not constant from
> test to test. Hence we can't actually say that the output is correct
> because we can't reproduce exactly what we are hashing easily.
>
> Indeed, the last attr test in series (node w/ replace) had an
> invalid md5sum. The attr value being produced after recovery was
> correct, but the md5sum output match was failing. Golden output
> appears to be wrong.
>
> Fix this issue by seperately dumping all the attributes on the inode
> via a list operation to indicate their size, then dump the value of
> the test attribute directly to md5sum. This means the md5sum for
> the attributes using the same fixed values are all identical, so
> it's easy to tell if the md5sum for a given test is correct. We also
> check that all attributes that should be present after recovery are
> still there (e.g. checks recovery didn't trash innocent bystanders).
>
> Further, the attribute replace tests replace an attribute with an
> identical value, hence there is no way to tell if recovery has
> resulted in the original being left behind or the new attribute
> being fully recovered because both have the same name and value.
> When replacing the attribute value, use a different sized value so
> it is trivial to determine that we've recovered the new attribute
> value correctly.
>
> Also, the test runs on the scratch device - there is no need to
> remove the testdir in _cleanup. Doing so prevents post-mortem
> failure analysis because it burns the dead, broken corpse to ash and
> leaves no way of to determine cause of death.
>
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
>
> Hi Catherine,
>
> These are all the mods I needed to make to be able to understand the
> test failures I was getting as I debugged the new LARP recovery
> algorithm I've written. You'll need to massage the test number in
> this patch to apply it on top of your patch.
>
> I haven't added any new test cases yet, nor have I done anything to
> manage the larp sysfs knob, but we'll need to do those in the near
> future.
>
> Zorro, can you consider merging this test in the near future? We're
> right at the point of merging the upstream kernel code and so really
> need to start growing the test coverage of this feature, and this
> test should simply not-run on kernels that don't have the feature
> enabled....
>
> Cheers,
>
> Dave.
Looks good, thanks for the fixes
Reviewed-by: Catherine Hoang <catherine.hoang@oracle.com>
> ---
>
> tests/xfs/600 | 20 +++++-----
> tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++------------------
> 2 files changed, 85 insertions(+), 44 deletions(-)
>
> diff --git a/tests/xfs/600 b/tests/xfs/600
> index 252cdf27..84704646 100755
> --- a/tests/xfs/600
> +++ b/tests/xfs/600
> @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
>
> _cleanup()
> {
> - rm -rf $tmp.* $testdir
> + rm -rf $tmp.*
> test -w /sys/fs/xfs/debug/larp && \
> echo 0 > /sys/fs/xfs/debug/larp
> }
> @@ -46,7 +46,9 @@ test_attr_replay()
> touch $testfile
>
> # Verify attr recovery
> - { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 | _filter_scratch
> + $ATTR_PROG -l $testfile | _filter_scratch
> + echo -n "$attr_name: "
> + $ATTR_PROG -q -g $attr_name $testfile | md5sum;
>
> echo ""
> }
> @@ -157,19 +159,19 @@ create_test_file remote_file2 1 $attr64k
> test_attr_replay remote_file2 "attr_name2" $attr64k "s" "larp"
> test_attr_replay remote_file2 "attr_name2" $attr64k "r" "larp"
>
> -# replace shortform
> +# replace shortform with different value
> create_test_file sf_file 2 $attr64
> -test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
> +test_attr_replay sf_file "attr_name2" $attr16 "s" "larp"
>
> -# replace leaf
> -create_test_file leaf_file 2 $attr1k
> -test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
> +# replace leaf with different value
> +create_test_file leaf_file 3 $attr1k
> +test_attr_replay leaf_file "attr_name2" $attr256 "s" "larp"
>
> -# replace node
> +# replace node with a different value
> create_test_file node_file 1 $attr64k
> $ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
> >> $seqres.full
> -test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
> +test_attr_replay node_file "attr_name2" $attr256 "s" "larp"
>
> echo "*** done"
> status=0
> diff --git a/tests/xfs/600.out b/tests/xfs/600.out
> index 96b1d7d9..fe25ea3e 100644
> --- a/tests/xfs/600.out
> +++ b/tests/xfs/600.out
> @@ -4,146 +4,185 @@ QA output created by 600
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> -21d850f99c43cc13abbe34838a8a3c8a -
> +Attribute "attr_name" has a 65 byte value for SCRATCH_MNT/testdir/empty_file1
> +attr_name: cfbe2a33be4601d2b655d099a18378fc -
>
> attr_remove: Input/output error
> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file1
> +attr_name: attr_get: No data available
> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> -2ff89c2935debc431745ec791be5421a -
> +Attribute "attr_name" has a 1025 byte value for SCRATCH_MNT/testdir/empty_file2
> +attr_name: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> -attr_get: No data available
> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file2
> +attr_name: attr_get: No data available
> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file2
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> -5d24b314242c52176c98ac4bd685da8b -
> +Attribute "attr_name" has a 65536 byte value for SCRATCH_MNT/testdir/empty_file3
> +attr_name: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> -attr_get: No data available
> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file3
> +attr_name: attr_get: No data available
> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file3
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> -5a7b559a70d8e92b4f3c6f7158eead08 -
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
> +Attribute "attr_name2" has a 65 byte value for SCRATCH_MNT/testdir/inline_file1
> +attr_name2: cfbe2a33be4601d2b655d099a18378fc -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> -5717d5e66c70be6bdb00ecbaca0b7749 -
> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/inline_file2
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file2
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> -5c929964efd1b243aa8cceb6524f4810 -
> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/inline_file3
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file3
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> -51ccb5cdfc9082060f0f94a8a108fea0 -
> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file1
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2': Input/output error
> -8d530bbe852d8bca83b131d5b3e497f5 -
> +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file2
> +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_set: Input/output error
> Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file3
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3': Input/output error
> -5d77c4d3831a35bcbbd6e7677119ce9a -
> +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file3
> +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> -6707ec2431e4dbea20e17da0816520bb -
> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/extent_file4
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file4
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> -767ebca3e4a6d24170857364f2bf2a3c -
> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/remote_file1
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file1
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> -fd84ddec89237e6d34a1703639efaebf -
> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>
> attr_remove: Input/output error
> Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> -attr_get: No data available
> -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> +attr_name2: attr_get: No data available
> +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file2
> d41d8cd98f00b204e9800998ecf8427e -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
> touch: cannot touch 'SCRATCH_MNT/testdir/sf_file': Input/output error
> -34aaa49662bafb46c76e377454685071 -
> +Attribute "attr_name1" has a 64 byte value for SCRATCH_MNT/testdir/sf_file
> +Attribute "attr_name2" has a 17 byte value for SCRATCH_MNT/testdir/sf_file
> +attr_name2: 9a6eb1bc9da3c66a9b495dfe2fe8a756 -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
> touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file': Input/output error
> -664e95ec28830ffb367c0950026e0d21 -
> +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/leaf_file
> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
> +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
>
> attr_set: Input/output error
> Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
> touch: cannot touch 'SCRATCH_MNT/testdir/node_file': Input/output error
> -bb37a78ce26472eeb711e3559933db42 -
> +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/node_file
> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/node_file
> +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
>
> *** done
> --
> Dave Chinner
> david@fromorbit.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] xfs/larp: Make test failures debuggable
2022-05-06 16:40 ` Zorro Lang
@ 2022-05-06 18:08 ` Catherine Hoang
2022-05-06 20:02 ` Zorro Lang
0 siblings, 1 reply; 15+ messages in thread
From: Catherine Hoang @ 2022-05-06 18:08 UTC (permalink / raw)
To: Zorro Lang; +Cc: Darrick J. Wong, Dave Chinner, linux-xfs, fstests
> On May 6, 2022, at 9:40 AM, Zorro Lang <zlang@redhat.com> wrote:
>
> On Fri, May 06, 2022 at 09:14:42AM -0700, Darrick J. Wong wrote:
>> On Fri, May 06, 2022 at 05:51:41PM +1000, Dave Chinner wrote:
>>> From: Dave Chinner <dchinner@redhat.com>
>>>
>>> Md5sum output for attributes created combined program output and
>>> attribute values. This adds variable path names to the md5sum, so
>>> there's no way to tell if the md5sum is actually correct for the
>>> given attribute value that is returned as it's not constant from
>>> test to test. Hence we can't actually say that the output is correct
>>> because we can't reproduce exactly what we are hashing easily.
>>>
>>> Indeed, the last attr test in series (node w/ replace) had an
>>> invalid md5sum. The attr value being produced after recovery was
>>> correct, but the md5sum output match was failing. Golden output
>>> appears to be wrong.
>>>
>>> Fix this issue by seperately dumping all the attributes on the inode
>>> via a list operation to indicate their size, then dump the value of
>>> the test attribute directly to md5sum. This means the md5sum for
>>> the attributes using the same fixed values are all identical, so
>>> it's easy to tell if the md5sum for a given test is correct. We also
>>> check that all attributes that should be present after recovery are
>>> still there (e.g. checks recovery didn't trash innocent bystanders).
>>>
>>> Further, the attribute replace tests replace an attribute with an
>>> identical value, hence there is no way to tell if recovery has
>>> resulted in the original being left behind or the new attribute
>>> being fully recovered because both have the same name and value.
>>> When replacing the attribute value, use a different sized value so
>>> it is trivial to determine that we've recovered the new attribute
>>> value correctly.
>>>
>>> Also, the test runs on the scratch device - there is no need to
>>> remove the testdir in _cleanup. Doing so prevents post-mortem
>>> failure analysis because it burns the dead, broken corpse to ash and
>>> leaves no way of to determine cause of death.
>>>
>>> Signed-off-by: Dave Chinner <dchinner@redhat.com>
>>> ---
>>>
>>> Hi Catherine,
>>>
>>> These are all the mods I needed to make to be able to understand the
>>> test failures I was getting as I debugged the new LARP recovery
>>> algorithm I've written. You'll need to massage the test number in
>>> this patch to apply it on top of your patch.
>>>
>>> I haven't added any new test cases yet, nor have I done anything to
>>> manage the larp sysfs knob, but we'll need to do those in the near
>>> future.
>>>
>>> Zorro, can you consider merging this test in the near future? We're
>>> right at the point of merging the upstream kernel code and so really
>>> need to start growing the test coverage of this feature, and this
>>> test should simply not-run on kernels that don't have the feature
>>> enabled....
>>>
>>> Cheers,
>>>
>>> Dave.
>>> ---
>>>
>>> tests/xfs/600 | 20 +++++-----
>>> tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++------------------
>>> 2 files changed, 85 insertions(+), 44 deletions(-)
>>>
>>> diff --git a/tests/xfs/600 b/tests/xfs/600
>>> index 252cdf27..84704646 100755
>>> --- a/tests/xfs/600
>>> +++ b/tests/xfs/600
>>> @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
>>>
>>> _cleanup()
>>> {
>>> - rm -rf $tmp.* $testdir
>>> + rm -rf $tmp.*
>>> test -w /sys/fs/xfs/debug/larp && \
>>> echo 0 > /sys/fs/xfs/debug/larp
>>
>> Blergh, this ^^^^^^^^^ is going to need fixing too.
>>
>> Please save the old value, then write it back in the _cleanup function.
>
> Ok, I'm going to do that when I merge it, if Catherine wouldn't like to do
> more changes in a V8 patch. If this case still need more changes, please tell
> me in time, and then it might have to wait the fstests release after next, if
> too late.
>
> Thanks,
> Zorro
Based on Dave’s feedback, it looks like the patch will need a few more
changes before it’s ready.
Catherine
>
>>
>> <slightly ot rant>
>>
>> These sysfs knobs are a pain because they all reset to defaults if
>> xfs.ko gets cycled. I know, I know, at least Dave and Ted don't do
>> modular kernels and so never see this happen, but I do. I bet Dave also
>> hasn't ever run xfs/434 or xfs/436, which might be why I'm the only one
>> seeing dquot leaks with 5.19-next.
>>
>> (I might be able to lift the xfs-as-module requirement if "echo 1 >
>> /sys/kernel/slab/*/validate" does what I think it might, since all we
>> want to do is look for slab leaks, and those tests rmmod/modprobe as a
>> brute force way of making the slab debug code check for leaks.)
>>
>> In case anyone's wondering, a solution to the knobs getting unset after
>> an rmmod/modprobe cycle is to add a file
>> /etc/udev/rules.d/99-fstester.rules containing:
>>
>> ACTION=="add|change", SUBSYSTEM=="module", DEVPATH=="/module/${module}", RUN+="/bin/sh -c \"echo ${value} > ${knob}\""
>>
>> which should be enough to keep LARP turned on.
>>
>> <end rant>
>>
>> Anyway, since this is a proposed test, I say that with this applied and
>> the debug knob bits fixed, the whole thing is
>> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
>>
>> --D
>>
>>> }
>>> @@ -46,7 +46,9 @@ test_attr_replay()
>>> touch $testfile
>>>
>>> # Verify attr recovery
>>> - { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 | _filter_scratch
>>> + $ATTR_PROG -l $testfile | _filter_scratch
>>> + echo -n "$attr_name: "
>>> + $ATTR_PROG -q -g $attr_name $testfile | md5sum;
>>>
>>> echo ""
>>> }
>>> @@ -157,19 +159,19 @@ create_test_file remote_file2 1 $attr64k
>>> test_attr_replay remote_file2 "attr_name2" $attr64k "s" "larp"
>>> test_attr_replay remote_file2 "attr_name2" $attr64k "r" "larp"
>>>
>>> -# replace shortform
>>> +# replace shortform with different value
>>> create_test_file sf_file 2 $attr64
>>> -test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
>>> +test_attr_replay sf_file "attr_name2" $attr16 "s" "larp"
>>>
>>> -# replace leaf
>>> -create_test_file leaf_file 2 $attr1k
>>> -test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
>>> +# replace leaf with different value
>>> +create_test_file leaf_file 3 $attr1k
>>> +test_attr_replay leaf_file "attr_name2" $attr256 "s" "larp"
>>>
>>> -# replace node
>>> +# replace node with a different value
>>> create_test_file node_file 1 $attr64k
>>> $ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
>>> >> $seqres.full
>>> -test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
>>> +test_attr_replay node_file "attr_name2" $attr256 "s" "larp"
>>>
>>> echo "*** done"
>>> status=0
>>> diff --git a/tests/xfs/600.out b/tests/xfs/600.out
>>> index 96b1d7d9..fe25ea3e 100644
>>> --- a/tests/xfs/600.out
>>> +++ b/tests/xfs/600.out
>>> @@ -4,146 +4,185 @@ QA output created by 600
>>> attr_set: Input/output error
>>> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
>>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
>>> -21d850f99c43cc13abbe34838a8a3c8a -
>>> +Attribute "attr_name" has a 65 byte value for SCRATCH_MNT/testdir/empty_file1
>>> +attr_name: cfbe2a33be4601d2b655d099a18378fc -
>>>
>>> attr_remove: Input/output error
>>> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file1
>>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
>>> -attr_get: No data available
>>> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file1
>>> +attr_name: attr_get: No data available
>>> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file1
>>> d41d8cd98f00b204e9800998ecf8427e -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
>>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
>>> -2ff89c2935debc431745ec791be5421a -
>>> +Attribute "attr_name" has a 1025 byte value for SCRATCH_MNT/testdir/empty_file2
>>> +attr_name: 9fd415c49d67afc4b78fad4055a3a376 -
>>>
>>> attr_remove: Input/output error
>>> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file2
>>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
>>> -attr_get: No data available
>>> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file2
>>> +attr_name: attr_get: No data available
>>> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file2
>>> d41d8cd98f00b204e9800998ecf8427e -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
>>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
>>> -5d24b314242c52176c98ac4bd685da8b -
>>> +Attribute "attr_name" has a 65536 byte value for SCRATCH_MNT/testdir/empty_file3
>>> +attr_name: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>>>
>>> attr_remove: Input/output error
>>> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file3
>>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
>>> -attr_get: No data available
>>> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file3
>>> +attr_name: attr_get: No data available
>>> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file3
>>> d41d8cd98f00b204e9800998ecf8427e -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file1
>>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
>>> -5a7b559a70d8e92b4f3c6f7158eead08 -
>>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
>>> +Attribute "attr_name2" has a 65 byte value for SCRATCH_MNT/testdir/inline_file1
>>> +attr_name2: cfbe2a33be4601d2b655d099a18378fc -
>>>
>>> attr_remove: Input/output error
>>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file1
>>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
>>> -attr_get: No data available
>>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file1
>>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
>>> +attr_name2: attr_get: No data available
>>> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file1
>>> d41d8cd98f00b204e9800998ecf8427e -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file2
>>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
>>> -5717d5e66c70be6bdb00ecbaca0b7749 -
>>> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/inline_file2
>>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
>>> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>>>
>>> attr_remove: Input/output error
>>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file2
>>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
>>> -attr_get: No data available
>>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file2
>>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
>>> +attr_name2: attr_get: No data available
>>> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file2
>>> d41d8cd98f00b204e9800998ecf8427e -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file3
>>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
>>> -5c929964efd1b243aa8cceb6524f4810 -
>>> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/inline_file3
>>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
>>> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>>>
>>> attr_remove: Input/output error
>>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file3
>>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
>>> -attr_get: No data available
>>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file3
>>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
>>> +attr_name2: attr_get: No data available
>>> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file3
>>> d41d8cd98f00b204e9800998ecf8427e -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file1
>>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
>>> -51ccb5cdfc9082060f0f94a8a108fea0 -
>>> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file1
>>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
>>> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>>>
>>> attr_remove: Input/output error
>>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file1
>>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
>>> -attr_get: No data available
>>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file1
>>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
>>> +attr_name2: attr_get: No data available
>>> +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file1
>>> d41d8cd98f00b204e9800998ecf8427e -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file2
>>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2': Input/output error
>>> -8d530bbe852d8bca83b131d5b3e497f5 -
>>> +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file2
>>> +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
>>> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
>>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
>>> +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file3
>>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3': Input/output error
>>> -5d77c4d3831a35bcbbd6e7677119ce9a -
>>> +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file3
>>> +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
>>> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
>>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
>>> +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file4
>>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
>>> -6707ec2431e4dbea20e17da0816520bb -
>>> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/extent_file4
>>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
>>> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>>>
>>> attr_remove: Input/output error
>>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file4
>>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
>>> -attr_get: No data available
>>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file4
>>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
>>> +attr_name2: attr_get: No data available
>>> +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file4
>>> d41d8cd98f00b204e9800998ecf8427e -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file1
>>> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
>>> -767ebca3e4a6d24170857364f2bf2a3c -
>>> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/remote_file1
>>> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
>>> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
>>>
>>> attr_remove: Input/output error
>>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file1
>>> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
>>> -attr_get: No data available
>>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file1
>>> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
>>> +attr_name2: attr_get: No data available
>>> +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file1
>>> d41d8cd98f00b204e9800998ecf8427e -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file2
>>> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
>>> -fd84ddec89237e6d34a1703639efaebf -
>>> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
>>> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
>>> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
>>>
>>> attr_remove: Input/output error
>>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file2
>>> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
>>> -attr_get: No data available
>>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file2
>>> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
>>> +attr_name2: attr_get: No data available
>>> +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file2
>>> d41d8cd98f00b204e9800998ecf8427e -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
>>> touch: cannot touch 'SCRATCH_MNT/testdir/sf_file': Input/output error
>>> -34aaa49662bafb46c76e377454685071 -
>>> +Attribute "attr_name1" has a 64 byte value for SCRATCH_MNT/testdir/sf_file
>>> +Attribute "attr_name2" has a 17 byte value for SCRATCH_MNT/testdir/sf_file
>>> +attr_name2: 9a6eb1bc9da3c66a9b495dfe2fe8a756 -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
>>> touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file': Input/output error
>>> -664e95ec28830ffb367c0950026e0d21 -
>>> +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/leaf_file
>>> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
>>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
>>> +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
>>>
>>> attr_set: Input/output error
>>> Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
>>> touch: cannot touch 'SCRATCH_MNT/testdir/node_file': Input/output error
>>> -bb37a78ce26472eeb711e3559933db42 -
>>> +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/node_file
>>> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/node_file
>>> +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
>>>
>>> *** done
>>> --
>>> Dave Chinner
>>> david@fromorbit.com
>>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] xfs/larp: Make test failures debuggable
2022-05-06 18:08 ` Catherine Hoang
@ 2022-05-06 20:02 ` Zorro Lang
2022-05-06 21:48 ` Dave Chinner
2022-05-06 21:55 ` Alli
0 siblings, 2 replies; 15+ messages in thread
From: Zorro Lang @ 2022-05-06 20:02 UTC (permalink / raw)
To: Catherine Hoang; +Cc: Darrick J. Wong, Dave Chinner, linux-xfs, fstests
On Fri, May 06, 2022 at 06:08:08PM +0000, Catherine Hoang wrote:
> > On May 6, 2022, at 9:40 AM, Zorro Lang <zlang@redhat.com> wrote:
> >
> > On Fri, May 06, 2022 at 09:14:42AM -0700, Darrick J. Wong wrote:
> >> On Fri, May 06, 2022 at 05:51:41PM +1000, Dave Chinner wrote:
> >>> From: Dave Chinner <dchinner@redhat.com>
> >>>
> >>> Md5sum output for attributes created combined program output and
> >>> attribute values. This adds variable path names to the md5sum, so
> >>> there's no way to tell if the md5sum is actually correct for the
> >>> given attribute value that is returned as it's not constant from
> >>> test to test. Hence we can't actually say that the output is correct
> >>> because we can't reproduce exactly what we are hashing easily.
> >>>
> >>> Indeed, the last attr test in series (node w/ replace) had an
> >>> invalid md5sum. The attr value being produced after recovery was
> >>> correct, but the md5sum output match was failing. Golden output
> >>> appears to be wrong.
> >>>
> >>> Fix this issue by seperately dumping all the attributes on the inode
> >>> via a list operation to indicate their size, then dump the value of
> >>> the test attribute directly to md5sum. This means the md5sum for
> >>> the attributes using the same fixed values are all identical, so
> >>> it's easy to tell if the md5sum for a given test is correct. We also
> >>> check that all attributes that should be present after recovery are
> >>> still there (e.g. checks recovery didn't trash innocent bystanders).
> >>>
> >>> Further, the attribute replace tests replace an attribute with an
> >>> identical value, hence there is no way to tell if recovery has
> >>> resulted in the original being left behind or the new attribute
> >>> being fully recovered because both have the same name and value.
> >>> When replacing the attribute value, use a different sized value so
> >>> it is trivial to determine that we've recovered the new attribute
> >>> value correctly.
> >>>
> >>> Also, the test runs on the scratch device - there is no need to
> >>> remove the testdir in _cleanup. Doing so prevents post-mortem
> >>> failure analysis because it burns the dead, broken corpse to ash and
> >>> leaves no way of to determine cause of death.
> >>>
> >>> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> >>> ---
> >>>
> >>> Hi Catherine,
> >>>
> >>> These are all the mods I needed to make to be able to understand the
> >>> test failures I was getting as I debugged the new LARP recovery
> >>> algorithm I've written. You'll need to massage the test number in
> >>> this patch to apply it on top of your patch.
> >>>
> >>> I haven't added any new test cases yet, nor have I done anything to
> >>> manage the larp sysfs knob, but we'll need to do those in the near
> >>> future.
> >>>
> >>> Zorro, can you consider merging this test in the near future? We're
> >>> right at the point of merging the upstream kernel code and so really
> >>> need to start growing the test coverage of this feature, and this
> >>> test should simply not-run on kernels that don't have the feature
> >>> enabled....
> >>>
> >>> Cheers,
> >>>
> >>> Dave.
> >>> ---
> >>>
> >>> tests/xfs/600 | 20 +++++-----
> >>> tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++------------------
> >>> 2 files changed, 85 insertions(+), 44 deletions(-)
> >>>
> >>> diff --git a/tests/xfs/600 b/tests/xfs/600
> >>> index 252cdf27..84704646 100755
> >>> --- a/tests/xfs/600
> >>> +++ b/tests/xfs/600
> >>> @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
> >>>
> >>> _cleanup()
> >>> {
> >>> - rm -rf $tmp.* $testdir
> >>> + rm -rf $tmp.*
> >>> test -w /sys/fs/xfs/debug/larp && \
> >>> echo 0 > /sys/fs/xfs/debug/larp
> >>
> >> Blergh, this ^^^^^^^^^ is going to need fixing too.
> >>
> >> Please save the old value, then write it back in the _cleanup function.
> >
> > Ok, I'm going to do that when I merge it, if Catherine wouldn't like to do
> > more changes in a V8 patch. If this case still need more changes, please tell
> > me in time, and then it might have to wait the fstests release after next, if
> > too late.
> >
> > Thanks,
> > Zorro
>
> Based on Dave’s feedback, it looks like the patch will need a few more
> changes before it’s ready.
Great, that would be really helpful if you'd like to rebase this patch to fstests
for-next branch. And how about combine Dave's patch with your patch? I don't think
it's worth merging two seperated patches for one signle new case. What does Dave think?
I just merged below two patchset[1] into xfs-5.18-fixes-1, then tried to test this case
(with you and Dave's patch). But it always failed as [2]. Please make sure it works
as expected with those kernel patches still reviewing, I can't be sure I have the
testing environment to guarantee that. Or I'd like to push this case after I see
its target kernel patchset be merged by xfs-linux at first.
Thanks,
Zorro
[1]
[PATCH v29 00/15] xfs: Log Attribute Replay
[PATCH 00/10 v2] xfs: LARP - clean up xfs_attr_set_iter state machine
[2] out.bad output:
*** mkfs
*** mount FS
attr_set: Input/output error
Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
xfs_logprint: unknown log operation type (1246)
Bad data in log
mount: /mnt/scratch: mount(2) system call failed: Structure needs cleaning.
>
> Catherine
> >
> >>
> >> <slightly ot rant>
> >>
> >> These sysfs knobs are a pain because they all reset to defaults if
> >> xfs.ko gets cycled. I know, I know, at least Dave and Ted don't do
> >> modular kernels and so never see this happen, but I do. I bet Dave also
> >> hasn't ever run xfs/434 or xfs/436, which might be why I'm the only one
> >> seeing dquot leaks with 5.19-next.
> >>
> >> (I might be able to lift the xfs-as-module requirement if "echo 1 >
> >> /sys/kernel/slab/*/validate" does what I think it might, since all we
> >> want to do is look for slab leaks, and those tests rmmod/modprobe as a
> >> brute force way of making the slab debug code check for leaks.)
> >>
> >> In case anyone's wondering, a solution to the knobs getting unset after
> >> an rmmod/modprobe cycle is to add a file
> >> /etc/udev/rules.d/99-fstester.rules containing:
> >>
> >> ACTION=="add|change", SUBSYSTEM=="module", DEVPATH=="/module/${module}", RUN+="/bin/sh -c \"echo ${value} > ${knob}\""
> >>
> >> which should be enough to keep LARP turned on.
> >>
> >> <end rant>
> >>
> >> Anyway, since this is a proposed test, I say that with this applied and
> >> the debug knob bits fixed, the whole thing is
> >> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> >>
> >> --D
> >>
> >>> }
> >>> @@ -46,7 +46,9 @@ test_attr_replay()
> >>> touch $testfile
> >>>
> >>> # Verify attr recovery
> >>> - { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 | _filter_scratch
> >>> + $ATTR_PROG -l $testfile | _filter_scratch
> >>> + echo -n "$attr_name: "
> >>> + $ATTR_PROG -q -g $attr_name $testfile | md5sum;
> >>>
> >>> echo ""
> >>> }
> >>> @@ -157,19 +159,19 @@ create_test_file remote_file2 1 $attr64k
> >>> test_attr_replay remote_file2 "attr_name2" $attr64k "s" "larp"
> >>> test_attr_replay remote_file2 "attr_name2" $attr64k "r" "larp"
> >>>
> >>> -# replace shortform
> >>> +# replace shortform with different value
> >>> create_test_file sf_file 2 $attr64
> >>> -test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
> >>> +test_attr_replay sf_file "attr_name2" $attr16 "s" "larp"
> >>>
> >>> -# replace leaf
> >>> -create_test_file leaf_file 2 $attr1k
> >>> -test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
> >>> +# replace leaf with different value
> >>> +create_test_file leaf_file 3 $attr1k
> >>> +test_attr_replay leaf_file "attr_name2" $attr256 "s" "larp"
> >>>
> >>> -# replace node
> >>> +# replace node with a different value
> >>> create_test_file node_file 1 $attr64k
> >>> $ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
> >>> >> $seqres.full
> >>> -test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
> >>> +test_attr_replay node_file "attr_name2" $attr256 "s" "larp"
> >>>
> >>> echo "*** done"
> >>> status=0
> >>> diff --git a/tests/xfs/600.out b/tests/xfs/600.out
> >>> index 96b1d7d9..fe25ea3e 100644
> >>> --- a/tests/xfs/600.out
> >>> +++ b/tests/xfs/600.out
> >>> @@ -4,146 +4,185 @@ QA output created by 600
> >>> attr_set: Input/output error
> >>> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> >>> -21d850f99c43cc13abbe34838a8a3c8a -
> >>> +Attribute "attr_name" has a 65 byte value for SCRATCH_MNT/testdir/empty_file1
> >>> +attr_name: cfbe2a33be4601d2b655d099a18378fc -
> >>>
> >>> attr_remove: Input/output error
> >>> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file1
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output error
> >>> -attr_get: No data available
> >>> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file1
> >>> +attr_name: attr_get: No data available
> >>> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file1
> >>> d41d8cd98f00b204e9800998ecf8427e -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> >>> -2ff89c2935debc431745ec791be5421a -
> >>> +Attribute "attr_name" has a 1025 byte value for SCRATCH_MNT/testdir/empty_file2
> >>> +attr_name: 9fd415c49d67afc4b78fad4055a3a376 -
> >>>
> >>> attr_remove: Input/output error
> >>> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file2
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2': Input/output error
> >>> -attr_get: No data available
> >>> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file2
> >>> +attr_name: attr_get: No data available
> >>> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file2
> >>> d41d8cd98f00b204e9800998ecf8427e -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> >>> -5d24b314242c52176c98ac4bd685da8b -
> >>> +Attribute "attr_name" has a 65536 byte value for SCRATCH_MNT/testdir/empty_file3
> >>> +attr_name: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> >>>
> >>> attr_remove: Input/output error
> >>> Could not remove "attr_name" for SCRATCH_MNT/testdir/empty_file3
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3': Input/output error
> >>> -attr_get: No data available
> >>> -Could not get "attr_name" for SCRATCH_MNT/testdir/empty_file3
> >>> +attr_name: attr_get: No data available
> >>> +Could not get "attr_name" for /mnt/scratch/testdir/empty_file3
> >>> d41d8cd98f00b204e9800998ecf8427e -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> >>> -5a7b559a70d8e92b4f3c6f7158eead08 -
> >>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
> >>> +Attribute "attr_name2" has a 65 byte value for SCRATCH_MNT/testdir/inline_file1
> >>> +attr_name2: cfbe2a33be4601d2b655d099a18378fc -
> >>>
> >>> attr_remove: Input/output error
> >>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1': Input/output error
> >>> -attr_get: No data available
> >>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file1
> >>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file1
> >>> +attr_name2: attr_get: No data available
> >>> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file1
> >>> d41d8cd98f00b204e9800998ecf8427e -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> >>> -5717d5e66c70be6bdb00ecbaca0b7749 -
> >>> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/inline_file2
> >>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
> >>> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
> >>>
> >>> attr_remove: Input/output error
> >>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2': Input/output error
> >>> -attr_get: No data available
> >>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file2
> >>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file2
> >>> +attr_name2: attr_get: No data available
> >>> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file2
> >>> d41d8cd98f00b204e9800998ecf8427e -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> >>> -5c929964efd1b243aa8cceb6524f4810 -
> >>> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/inline_file3
> >>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
> >>> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> >>>
> >>> attr_remove: Input/output error
> >>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3': Input/output error
> >>> -attr_get: No data available
> >>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/inline_file3
> >>> +Attribute "attr_name1" has a 16 byte value for SCRATCH_MNT/testdir/inline_file3
> >>> +attr_name2: attr_get: No data available
> >>> +Could not get "attr_name2" for /mnt/scratch/testdir/inline_file3
> >>> d41d8cd98f00b204e9800998ecf8427e -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> >>> -51ccb5cdfc9082060f0f94a8a108fea0 -
> >>> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file1
> >>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
> >>> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
> >>>
> >>> attr_remove: Input/output error
> >>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1': Input/output error
> >>> -attr_get: No data available
> >>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file1
> >>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file1
> >>> +attr_name2: attr_get: No data available
> >>> +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file1
> >>> d41d8cd98f00b204e9800998ecf8427e -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file2
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2': Input/output error
> >>> -8d530bbe852d8bca83b131d5b3e497f5 -
> >>> +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file2
> >>> +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> >>> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> >>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file2
> >>> +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name4" for SCRATCH_MNT/testdir/extent_file3
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3': Input/output error
> >>> -5d77c4d3831a35bcbbd6e7677119ce9a -
> >>> +Attribute "attr_name4" has a 1025 byte value for SCRATCH_MNT/testdir/extent_file3
> >>> +Attribute "attr_name2" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> >>> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> >>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file3
> >>> +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> >>> -6707ec2431e4dbea20e17da0816520bb -
> >>> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/extent_file4
> >>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
> >>> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> >>>
> >>> attr_remove: Input/output error
> >>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4': Input/output error
> >>> -attr_get: No data available
> >>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/extent_file4
> >>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/extent_file4
> >>> +attr_name2: attr_get: No data available
> >>> +Could not get "attr_name2" for /mnt/scratch/testdir/extent_file4
> >>> d41d8cd98f00b204e9800998ecf8427e -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> >>> -767ebca3e4a6d24170857364f2bf2a3c -
> >>> +Attribute "attr_name2" has a 1025 byte value for SCRATCH_MNT/testdir/remote_file1
> >>> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
> >>> +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
> >>>
> >>> attr_remove: Input/output error
> >>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1': Input/output error
> >>> -attr_get: No data available
> >>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file1
> >>> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file1
> >>> +attr_name2: attr_get: No data available
> >>> +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file1
> >>> d41d8cd98f00b204e9800998ecf8427e -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> >>> -fd84ddec89237e6d34a1703639efaebf -
> >>> +Attribute "attr_name2" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> >>> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> >>> +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> >>>
> >>> attr_remove: Input/output error
> >>> Could not remove "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2': Input/output error
> >>> -attr_get: No data available
> >>> -Could not get "attr_name2" for SCRATCH_MNT/testdir/remote_file2
> >>> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/remote_file2
> >>> +attr_name2: attr_get: No data available
> >>> +Could not get "attr_name2" for /mnt/scratch/testdir/remote_file2
> >>> d41d8cd98f00b204e9800998ecf8427e -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/sf_file': Input/output error
> >>> -34aaa49662bafb46c76e377454685071 -
> >>> +Attribute "attr_name1" has a 64 byte value for SCRATCH_MNT/testdir/sf_file
> >>> +Attribute "attr_name2" has a 17 byte value for SCRATCH_MNT/testdir/sf_file
> >>> +attr_name2: 9a6eb1bc9da3c66a9b495dfe2fe8a756 -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file': Input/output error
> >>> -664e95ec28830ffb367c0950026e0d21 -
> >>> +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/leaf_file
> >>> +Attribute "attr_name3" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
> >>> +Attribute "attr_name1" has a 1024 byte value for SCRATCH_MNT/testdir/leaf_file
> >>> +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
> >>>
> >>> attr_set: Input/output error
> >>> Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
> >>> touch: cannot touch 'SCRATCH_MNT/testdir/node_file': Input/output error
> >>> -bb37a78ce26472eeb711e3559933db42 -
> >>> +Attribute "attr_name2" has a 257 byte value for SCRATCH_MNT/testdir/node_file
> >>> +Attribute "attr_name1" has a 65536 byte value for SCRATCH_MNT/testdir/node_file
> >>> +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
> >>>
> >>> *** done
> >>> --
> >>> Dave Chinner
> >>> david@fromorbit.com
> >>
> >
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] xfs/larp: Make test failures debuggable
2022-05-06 20:02 ` Zorro Lang
@ 2022-05-06 21:48 ` Dave Chinner
2022-05-07 5:34 ` Zorro Lang
2022-05-06 21:55 ` Alli
1 sibling, 1 reply; 15+ messages in thread
From: Dave Chinner @ 2022-05-06 21:48 UTC (permalink / raw)
To: Catherine Hoang, Darrick J. Wong, linux-xfs, fstests
On Sat, May 07, 2022 at 04:02:12AM +0800, Zorro Lang wrote:
> On Fri, May 06, 2022 at 06:08:08PM +0000, Catherine Hoang wrote:
> > > On May 6, 2022, at 9:40 AM, Zorro Lang <zlang@redhat.com> wrote:
> > >
> > > On Fri, May 06, 2022 at 09:14:42AM -0700, Darrick J. Wong wrote:
> > >> On Fri, May 06, 2022 at 05:51:41PM +1000, Dave Chinner wrote:
> > >>> From: Dave Chinner <dchinner@redhat.com>
> > >>>
> > >>> Md5sum output for attributes created combined program output and
> > >>> attribute values. This adds variable path names to the md5sum, so
> > >>> there's no way to tell if the md5sum is actually correct for the
> > >>> given attribute value that is returned as it's not constant from
> > >>> test to test. Hence we can't actually say that the output is correct
> > >>> because we can't reproduce exactly what we are hashing easily.
> > >>>
> > >>> Indeed, the last attr test in series (node w/ replace) had an
> > >>> invalid md5sum. The attr value being produced after recovery was
> > >>> correct, but the md5sum output match was failing. Golden output
> > >>> appears to be wrong.
> > >>>
> > >>> Fix this issue by seperately dumping all the attributes on the inode
> > >>> via a list operation to indicate their size, then dump the value of
> > >>> the test attribute directly to md5sum. This means the md5sum for
> > >>> the attributes using the same fixed values are all identical, so
> > >>> it's easy to tell if the md5sum for a given test is correct. We also
> > >>> check that all attributes that should be present after recovery are
> > >>> still there (e.g. checks recovery didn't trash innocent bystanders).
> > >>>
> > >>> Further, the attribute replace tests replace an attribute with an
> > >>> identical value, hence there is no way to tell if recovery has
> > >>> resulted in the original being left behind or the new attribute
> > >>> being fully recovered because both have the same name and value.
> > >>> When replacing the attribute value, use a different sized value so
> > >>> it is trivial to determine that we've recovered the new attribute
> > >>> value correctly.
> > >>>
> > >>> Also, the test runs on the scratch device - there is no need to
> > >>> remove the testdir in _cleanup. Doing so prevents post-mortem
> > >>> failure analysis because it burns the dead, broken corpse to ash and
> > >>> leaves no way of to determine cause of death.
> > >>>
> > >>> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> > >>> ---
> > >>>
> > >>> Hi Catherine,
> > >>>
> > >>> These are all the mods I needed to make to be able to understand the
> > >>> test failures I was getting as I debugged the new LARP recovery
> > >>> algorithm I've written. You'll need to massage the test number in
> > >>> this patch to apply it on top of your patch.
> > >>>
> > >>> I haven't added any new test cases yet, nor have I done anything to
> > >>> manage the larp sysfs knob, but we'll need to do those in the near
> > >>> future.
> > >>>
> > >>> Zorro, can you consider merging this test in the near future? We're
> > >>> right at the point of merging the upstream kernel code and so really
> > >>> need to start growing the test coverage of this feature, and this
> > >>> test should simply not-run on kernels that don't have the feature
> > >>> enabled....
> > >>>
> > >>> Cheers,
> > >>>
> > >>> Dave.
> > >>> ---
> > >>>
> > >>> tests/xfs/600 | 20 +++++-----
> > >>> tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++------------------
> > >>> 2 files changed, 85 insertions(+), 44 deletions(-)
> > >>>
> > >>> diff --git a/tests/xfs/600 b/tests/xfs/600
> > >>> index 252cdf27..84704646 100755
> > >>> --- a/tests/xfs/600
> > >>> +++ b/tests/xfs/600
> > >>> @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
> > >>>
> > >>> _cleanup()
> > >>> {
> > >>> - rm -rf $tmp.* $testdir
> > >>> + rm -rf $tmp.*
> > >>> test -w /sys/fs/xfs/debug/larp && \
> > >>> echo 0 > /sys/fs/xfs/debug/larp
> > >>
> > >> Blergh, this ^^^^^^^^^ is going to need fixing too.
Yes, I did point that out.
> > >>
> > >> Please save the old value, then write it back in the _cleanup function.
> > >
> > > Ok, I'm going to do that when I merge it,
No, please don't. I don't want random changes added to the test on
commit right now as I'm still actively working on it. I've said
twice now that this needs fixing (3 if you count this mention) and
that the test coverage also needs improving. If someone is still
working on the tests, then why make more work for everyone by making
unnecessary, unreviewed changes on commit?
> > > if Catherine wouldn't like to do
> > > more changes in a V8 patch. If this case still need more changes, please tell
> > > me in time, and then it might have to wait the fstests release after next, if
> > > too late.
> > >
> > > Thanks,
> > > Zorro
> >
> > Based on Dave’s feedback, it looks like the patch will need a few more
> > changes before it’s ready.
That doesn't mean it can't be merged. It is a pain for mulitple
people to collaborate on a test that isn't merged because the test
number is not set in stone and chosing numbers always conflicts with
other tests under development. Getting the test merged early makes
knocking all the problems out of the test (and the code it is
testing!) much, much easier.
> Great, that would be really helpful if you'd like to rebase this patch to fstests
> for-next branch. And how about combine Dave's patch with your patch? I don't think
> it's worth merging two seperated patches for one signle new case. What does Dave think?
>
> I just merged below two patchset[1] into xfs-5.18-fixes-1, then tried to test this case
> (with you and Dave's patch). But it always failed as [2].
You built a broken kernel as it has none of the dependencies and bug
fixes that had already been committed to for-next for the new
functionality to work correctly. I posted a V3 patchset last night
and a published a git branch with all the kernel changes that you
can use for testing if you want.
> Please make sure it works
> as expected with those kernel patches still reviewing,
I did - the failures you see are expected from what you were
testing. i.e. the test ran just fine, the kernel code you were
running is buggy and you didn't update xfsprogs so logprint
supported the new log types, hence the test (correctly) reported
failures.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] xfs/larp: Make test failures debuggable
2022-05-06 20:02 ` Zorro Lang
2022-05-06 21:48 ` Dave Chinner
@ 2022-05-06 21:55 ` Alli
1 sibling, 0 replies; 15+ messages in thread
From: Alli @ 2022-05-06 21:55 UTC (permalink / raw)
To: Zorro Lang, Catherine Hoang
Cc: Darrick J. Wong, Dave Chinner, linux-xfs, fstests
On Sat, 2022-05-07 at 04:02 +0800, Zorro Lang wrote:
> On Fri, May 06, 2022 at 06:08:08PM +0000, Catherine Hoang wrote:
> > > On May 6, 2022, at 9:40 AM, Zorro Lang <zlang@redhat.com> wrote:
> > >
> > > On Fri, May 06, 2022 at 09:14:42AM -0700, Darrick J. Wong wrote:
> > > > On Fri, May 06, 2022 at 05:51:41PM +1000, Dave Chinner wrote:
> > > > > From: Dave Chinner <dchinner@redhat.com>
> > > > >
> > > > > Md5sum output for attributes created combined program output
> > > > > and
> > > > > attribute values. This adds variable path names to the
> > > > > md5sum, so
> > > > > there's no way to tell if the md5sum is actually correct for
> > > > > the
> > > > > given attribute value that is returned as it's not constant
> > > > > from
> > > > > test to test. Hence we can't actually say that the output is
> > > > > correct
> > > > > because we can't reproduce exactly what we are hashing
> > > > > easily.
> > > > >
> > > > > Indeed, the last attr test in series (node w/ replace) had an
> > > > > invalid md5sum. The attr value being produced after recovery
> > > > > was
> > > > > correct, but the md5sum output match was failing. Golden
> > > > > output
> > > > > appears to be wrong.
> > > > >
> > > > > Fix this issue by seperately dumping all the attributes on
> > > > > the inode
> > > > > via a list operation to indicate their size, then dump the
> > > > > value of
> > > > > the test attribute directly to md5sum. This means the md5sum
> > > > > for
> > > > > the attributes using the same fixed values are all identical,
> > > > > so
> > > > > it's easy to tell if the md5sum for a given test is correct.
> > > > > We also
> > > > > check that all attributes that should be present after
> > > > > recovery are
> > > > > still there (e.g. checks recovery didn't trash innocent
> > > > > bystanders).
> > > > >
> > > > > Further, the attribute replace tests replace an attribute
> > > > > with an
> > > > > identical value, hence there is no way to tell if recovery
> > > > > has
> > > > > resulted in the original being left behind or the new
> > > > > attribute
> > > > > being fully recovered because both have the same name and
> > > > > value.
> > > > > When replacing the attribute value, use a different sized
> > > > > value so
> > > > > it is trivial to determine that we've recovered the new
> > > > > attribute
> > > > > value correctly.
> > > > >
> > > > > Also, the test runs on the scratch device - there is no need
> > > > > to
> > > > > remove the testdir in _cleanup. Doing so prevents post-mortem
> > > > > failure analysis because it burns the dead, broken corpse to
> > > > > ash and
> > > > > leaves no way of to determine cause of death.
> > > > >
> > > > > Signed-off-by: Dave Chinner <dchinner@redhat.com>
> > > > > ---
> > > > >
> > > > > Hi Catherine,
> > > > >
> > > > > These are all the mods I needed to make to be able to
> > > > > understand the
> > > > > test failures I was getting as I debugged the new LARP
> > > > > recovery
> > > > > algorithm I've written. You'll need to massage the test
> > > > > number in
> > > > > this patch to apply it on top of your patch.
> > > > >
> > > > > I haven't added any new test cases yet, nor have I done
> > > > > anything to
> > > > > manage the larp sysfs knob, but we'll need to do those in the
> > > > > near
> > > > > future.
> > > > >
> > > > > Zorro, can you consider merging this test in the near
> > > > > future? We're
> > > > > right at the point of merging the upstream kernel code and so
> > > > > really
> > > > > need to start growing the test coverage of this feature, and
> > > > > this
> > > > > test should simply not-run on kernels that don't have the
> > > > > feature
> > > > > enabled....
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Dave.
> > > > > ---
> > > > >
> > > > > tests/xfs/600 | 20 +++++-----
> > > > > tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++-
> > > > > -----------------
> > > > > 2 files changed, 85 insertions(+), 44 deletions(-)
> > > > >
> > > > > diff --git a/tests/xfs/600 b/tests/xfs/600
> > > > > index 252cdf27..84704646 100755
> > > > > --- a/tests/xfs/600
> > > > > +++ b/tests/xfs/600
> > > > > @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
> > > > >
> > > > > _cleanup()
> > > > > {
> > > > > - rm -rf $tmp.* $testdir
> > > > > + rm -rf $tmp.*
> > > > > test -w /sys/fs/xfs/debug/larp && \
> > > > > echo 0 > /sys/fs/xfs/debug/larp
> > > >
> > > > Blergh, this ^^^^^^^^^ is going to need fixing too.
> > > >
> > > > Please save the old value, then write it back in the _cleanup
> > > > function.
> > >
> > > Ok, I'm going to do that when I merge it, if Catherine wouldn't
> > > like to do
> > > more changes in a V8 patch. If this case still need more changes,
> > > please tell
> > > me in time, and then it might have to wait the fstests release
> > > after next, if
> > > too late.
> > >
> > > Thanks,
> > > Zorro
> >
> > Based on Dave’s feedback, it looks like the patch will need a few
> > more
> > changes before it’s ready.
>
> Great, that would be really helpful if you'd like to rebase this
> patch to fstests
> for-next branch. And how about combine Dave's patch with your patch?
> I don't think
> it's worth merging two seperated patches for one signle new case.
> What does Dave think?
>
> I just merged below two patchset[1] into xfs-5.18-fixes-1, then tried
> to test this case
> (with you and Dave's patch). But it always failed as [2]. Please make
> sure it works
> as expected with those kernel patches still reviewing, I can't be
> sure I have the
> testing environment to guarantee that. Or I'd like to push this case
> after I see
> its target kernel patchset be merged by xfs-linux at first.
>
> Thanks,
> Zorro
>
> [1]
> [PATCH v29 00/15] xfs: Log Attribute Replay
> [PATCH 00/10 v2] xfs: LARP - clean up xfs_attr_set_iter state machine
>
>
> [2] out.bad output:
> *** mkfs
> *** mount FS
> attr_set: Input/output error
> Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1': Input/output
> error
> xfs_logprint: unknown log operation type (1246)
> Bad data in log
> mount: /mnt/scratch: mount(2) system call failed: Structure needs
> cleaning.
I think that happens when you dont have the corresponding xfsprogs
updates. The userspace log print does not recognize the new log item.
Assuming you are running with an unpatched xfsprogs, I would have
thought the lack of error injects might have stopped the test from
running though?
Allison
> > Catherine
> > > > <slightly ot rant>
> > > >
> > > > These sysfs knobs are a pain because they all reset to defaults
> > > > if
> > > > xfs.ko gets cycled. I know, I know, at least Dave and Ted
> > > > don't do
> > > > modular kernels and so never see this happen, but I do. I bet
> > > > Dave also
> > > > hasn't ever run xfs/434 or xfs/436, which might be why I'm the
> > > > only one
> > > > seeing dquot leaks with 5.19-next.
> > > >
> > > > (I might be able to lift the xfs-as-module requirement if "echo
> > > > 1 >
> > > > /sys/kernel/slab/*/validate" does what I think it might, since
> > > > all we
> > > > want to do is look for slab leaks, and those tests
> > > > rmmod/modprobe as a
> > > > brute force way of making the slab debug code check for leaks.)
> > > >
> > > > In case anyone's wondering, a solution to the knobs getting
> > > > unset after
> > > > an rmmod/modprobe cycle is to add a file
> > > > /etc/udev/rules.d/99-fstester.rules containing:
> > > >
> > > > ACTION=="add|change", SUBSYSTEM=="module",
> > > > DEVPATH=="/module/${module}", RUN+="/bin/sh -c \"echo ${value}
> > > > > ${knob}\""
> > > >
> > > > which should be enough to keep LARP turned on.
> > > >
> > > > <end rant>
> > > >
> > > > Anyway, since this is a proposed test, I say that with this
> > > > applied and
> > > > the debug knob bits fixed, the whole thing is
> > > > Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> > > >
> > > > --D
> > > >
> > > > > }
> > > > > @@ -46,7 +46,9 @@ test_attr_replay()
> > > > > touch $testfile
> > > > >
> > > > > # Verify attr recovery
> > > > > - { $ATTR_PROG -g $attr_name $testfile | md5sum; } 2>&1 |
> > > > > _filter_scratch
> > > > > + $ATTR_PROG -l $testfile | _filter_scratch
> > > > > + echo -n "$attr_name: "
> > > > > + $ATTR_PROG -q -g $attr_name $testfile | md5sum;
> > > > >
> > > > > echo ""
> > > > > }
> > > > > @@ -157,19 +159,19 @@ create_test_file remote_file2 1
> > > > > $attr64k
> > > > > test_attr_replay remote_file2 "attr_name2" $attr64k "s"
> > > > > "larp"
> > > > > test_attr_replay remote_file2 "attr_name2" $attr64k "r"
> > > > > "larp"
> > > > >
> > > > > -# replace shortform
> > > > > +# replace shortform with different value
> > > > > create_test_file sf_file 2 $attr64
> > > > > -test_attr_replay sf_file "attr_name2" $attr64 "s" "larp"
> > > > > +test_attr_replay sf_file "attr_name2" $attr16 "s" "larp"
> > > > >
> > > > > -# replace leaf
> > > > > -create_test_file leaf_file 2 $attr1k
> > > > > -test_attr_replay leaf_file "attr_name2" $attr1k "s" "larp"
> > > > > +# replace leaf with different value
> > > > > +create_test_file leaf_file 3 $attr1k
> > > > > +test_attr_replay leaf_file "attr_name2" $attr256 "s" "larp"
> > > > >
> > > > > -# replace node
> > > > > +# replace node with a different value
> > > > > create_test_file node_file 1 $attr64k
> > > > > $ATTR_PROG -s "attr_name2" -V $attr1k $testdir/node_file \
> > > > > >> $seqres.full
> > > > > -test_attr_replay node_file "attr_name2" $attr1k "s" "larp"
> > > > > +test_attr_replay node_file "attr_name2" $attr256 "s" "larp"
> > > > >
> > > > > echo "*** done"
> > > > > status=0
> > > > > diff --git a/tests/xfs/600.out b/tests/xfs/600.out
> > > > > index 96b1d7d9..fe25ea3e 100644
> > > > > --- a/tests/xfs/600.out
> > > > > +++ b/tests/xfs/600.out
> > > > > @@ -4,146 +4,185 @@ QA output created by 600
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1':
> > > > > Input/output error
> > > > > -21d850f99c43cc13abbe34838a8a3c8a -
> > > > > +Attribute "attr_name" has a 65 byte value for
> > > > > SCRATCH_MNT/testdir/empty_file1
> > > > > +attr_name: cfbe2a33be4601d2b655d099a18378fc -
> > > > >
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file1':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file1
> > > > > +attr_name: attr_get: No data available
> > > > > +Could not get "attr_name" for
> > > > > /mnt/scratch/testdir/empty_file1
> > > > > d41d8cd98f00b204e9800998ecf8427e -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2':
> > > > > Input/output error
> > > > > -2ff89c2935debc431745ec791be5421a -
> > > > > +Attribute "attr_name" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/empty_file2
> > > > > +attr_name: 9fd415c49d67afc4b78fad4055a3a376 -
> > > > >
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file2':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file2
> > > > > +attr_name: attr_get: No data available
> > > > > +Could not get "attr_name" for
> > > > > /mnt/scratch/testdir/empty_file2
> > > > > d41d8cd98f00b204e9800998ecf8427e -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name" for SCRATCH_MNT/testdir/empty_file3
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3':
> > > > > Input/output error
> > > > > -5d24b314242c52176c98ac4bd685da8b -
> > > > > +Attribute "attr_name" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/empty_file3
> > > > > +attr_name: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> > > > >
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file3
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/empty_file3':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name" for
> > > > > SCRATCH_MNT/testdir/empty_file3
> > > > > +attr_name: attr_get: No data available
> > > > > +Could not get "attr_name" for
> > > > > /mnt/scratch/testdir/empty_file3
> > > > > d41d8cd98f00b204e9800998ecf8427e -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1':
> > > > > Input/output error
> > > > > -5a7b559a70d8e92b4f3c6f7158eead08 -
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > +Attribute "attr_name2" has a 65 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > +attr_name2: cfbe2a33be4601d2b655d099a18378fc -
> > > > >
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file1':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file1
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/inline_file1
> > > > > d41d8cd98f00b204e9800998ecf8427e -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2':
> > > > > Input/output error
> > > > > -5717d5e66c70be6bdb00ecbaca0b7749 -
> > > > > +Attribute "attr_name2" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
> > > > >
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file2':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file2
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/inline_file2
> > > > > d41d8cd98f00b204e9800998ecf8427e -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3':
> > > > > Input/output error
> > > > > -5c929964efd1b243aa8cceb6524f4810 -
> > > > > +Attribute "attr_name2" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> > > > >
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/inline_file3':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > +Attribute "attr_name1" has a 16 byte value for
> > > > > SCRATCH_MNT/testdir/inline_file3
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/inline_file3
> > > > > d41d8cd98f00b204e9800998ecf8427e -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1':
> > > > > Input/output error
> > > > > -51ccb5cdfc9082060f0f94a8a108fea0 -
> > > > > +Attribute "attr_name2" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
> > > > >
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file1':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file1
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/extent_file1
> > > > > d41d8cd98f00b204e9800998ecf8427e -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name4" for
> > > > > SCRATCH_MNT/testdir/extent_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file2':
> > > > > Input/output error
> > > > > -8d530bbe852d8bca83b131d5b3e497f5 -
> > > > > +Attribute "attr_name4" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file2
> > > > > +Attribute "attr_name2" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file2
> > > > > +Attribute "attr_name3" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file2
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file2
> > > > > +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name4" for
> > > > > SCRATCH_MNT/testdir/extent_file3
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file3':
> > > > > Input/output error
> > > > > -5d77c4d3831a35bcbbd6e7677119ce9a -
> > > > > +Attribute "attr_name4" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file3
> > > > > +Attribute "attr_name2" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file3
> > > > > +Attribute "attr_name3" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file3
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file3
> > > > > +attr_name4: 9fd415c49d67afc4b78fad4055a3a376 -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4':
> > > > > Input/output error
> > > > > -6707ec2431e4dbea20e17da0816520bb -
> > > > > +Attribute "attr_name2" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> > > > >
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/extent_file4':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/extent_file4
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/extent_file4
> > > > > d41d8cd98f00b204e9800998ecf8427e -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1':
> > > > > Input/output error
> > > > > -767ebca3e4a6d24170857364f2bf2a3c -
> > > > > +Attribute "attr_name2" has a 1025 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > +Attribute "attr_name1" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > +attr_name2: 9fd415c49d67afc4b78fad4055a3a376 -
> > > > >
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file1':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > +Attribute "attr_name1" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file1
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/remote_file1
> > > > > d41d8cd98f00b204e9800998ecf8427e -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2':
> > > > > Input/output error
> > > > > -fd84ddec89237e6d34a1703639efaebf -
> > > > > +Attribute "attr_name2" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > +Attribute "attr_name1" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > +attr_name2: 7f6fd1b6d872108bd44bd143cbcdfa19 -
> > > > >
> > > > > attr_remove: Input/output error
> > > > > Could not remove "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/remote_file2':
> > > > > Input/output error
> > > > > -attr_get: No data available
> > > > > -Could not get "attr_name2" for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > +Attribute "attr_name1" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/remote_file2
> > > > > +attr_name2: attr_get: No data available
> > > > > +Could not get "attr_name2" for
> > > > > /mnt/scratch/testdir/remote_file2
> > > > > d41d8cd98f00b204e9800998ecf8427e -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for SCRATCH_MNT/testdir/sf_file
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/sf_file':
> > > > > Input/output error
> > > > > -34aaa49662bafb46c76e377454685071 -
> > > > > +Attribute "attr_name1" has a 64 byte value for
> > > > > SCRATCH_MNT/testdir/sf_file
> > > > > +Attribute "attr_name2" has a 17 byte value for
> > > > > SCRATCH_MNT/testdir/sf_file
> > > > > +attr_name2: 9a6eb1bc9da3c66a9b495dfe2fe8a756 -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for SCRATCH_MNT/testdir/leaf_file
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/leaf_file':
> > > > > Input/output error
> > > > > -664e95ec28830ffb367c0950026e0d21 -
> > > > > +Attribute "attr_name2" has a 257 byte value for
> > > > > SCRATCH_MNT/testdir/leaf_file
> > > > > +Attribute "attr_name3" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/leaf_file
> > > > > +Attribute "attr_name1" has a 1024 byte value for
> > > > > SCRATCH_MNT/testdir/leaf_file
> > > > > +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
> > > > >
> > > > > attr_set: Input/output error
> > > > > Could not set "attr_name2" for SCRATCH_MNT/testdir/node_file
> > > > > touch: cannot touch 'SCRATCH_MNT/testdir/node_file':
> > > > > Input/output error
> > > > > -bb37a78ce26472eeb711e3559933db42 -
> > > > > +Attribute "attr_name2" has a 257 byte value for
> > > > > SCRATCH_MNT/testdir/node_file
> > > > > +Attribute "attr_name1" has a 65536 byte value for
> > > > > SCRATCH_MNT/testdir/node_file
> > > > > +attr_name2: f4ea5799d72a0a9bf2d56a685c9cba7a -
> > > > >
> > > > > *** done
> > > > > --
> > > > > Dave Chinner
> > > > > david@fromorbit.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] xfs/larp: Make test failures debuggable
2022-05-06 16:14 ` Darrick J. Wong
2022-05-06 16:40 ` Zorro Lang
@ 2022-05-06 22:16 ` Dave Chinner
1 sibling, 0 replies; 15+ messages in thread
From: Dave Chinner @ 2022-05-06 22:16 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: Catherine Hoang, linux-xfs, fstests
On Fri, May 06, 2022 at 09:14:42AM -0700, Darrick J. Wong wrote:
> On Fri, May 06, 2022 at 05:51:41PM +1000, Dave Chinner wrote:
> > Hi Catherine,
> >
> > These are all the mods I needed to make to be able to understand the
> > test failures I was getting as I debugged the new LARP recovery
> > algorithm I've written. You'll need to massage the test number in
> > this patch to apply it on top of your patch.
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
> > I haven't added any new test cases yet, nor have I done anything to
> > manage the larp sysfs knob, but we'll need to do those in the near
> > future.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > Zorro, can you consider merging this test in the near future? We're
> > right at the point of merging the upstream kernel code and so really
> > need to start growing the test coverage of this feature, and this
> > test should simply not-run on kernels that don't have the feature
> > enabled....
> >
> > Cheers,
> >
> > Dave.
> > ---
> >
> > tests/xfs/600 | 20 +++++-----
> > tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++------------------
> > 2 files changed, 85 insertions(+), 44 deletions(-)
> >
> > diff --git a/tests/xfs/600 b/tests/xfs/600
> > index 252cdf27..84704646 100755
> > --- a/tests/xfs/600
> > +++ b/tests/xfs/600
> > @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
> >
> > _cleanup()
> > {
> > - rm -rf $tmp.* $testdir
> > + rm -rf $tmp.*
> > test -w /sys/fs/xfs/debug/larp && \
> > echo 0 > /sys/fs/xfs/debug/larp
>
> Blergh, this ^^^^^^^^^ is going to need fixing too.
>
> Please save the old value, then write it back in the _cleanup function.
Yes, I know about this. I haven't done anything about it yet because
I haven't decided yet whether it should stay or be removed. Getting
the test and the code it tests working has been my focus so far.
I have a wrapper script that sets the knob for the entire of the
fstests run (i.e. so all attr and recovery tests exercise the LARP
functionality) and so I'm not sure that this test should twiddle the
knob at all. i.e. right now it would be more useful to me for the
test to check if the knob is not set to just _not_run the test,
rather than turn it on and off and all that...
Indeed, we probably shouldn't be forcing everyone to test a brand
new experimental feature that might be unstable and cause the test
run to halt prematurely (e.g. because it crashes).
> <slightly ot rant>
>
> These sysfs knobs are a pain because they all reset to defaults if
> xfs.ko gets cycled. I know, I know, at least Dave and Ted don't do
> modular kernels and so never see this happen, but I do.
sysfs knobs get reset by reboots to load a new kernel, so the impact
is identical - I have to reset the knob every time I load a new test
kernel, just like you have to reset it every time you load a new
module.
> I bet Dave also
> hasn't ever run xfs/434 or xfs/436, which might be why I'm the only one
> seeing dquot leaks with 5.19-next.
Yup, there are a bunch of tests that I never run because:
- scsi_debug requires modular kernel
- I always build with fatal XFS ASSERTs, so the tests that want
non-fatal asserts don't run
- xfs/43[4-6] require a modular kernel
- xfs/490 requires CONFIG_XFS_DEBUG be turned off (I *never* do
that)
- I generally don't have source code on my test machines, so
api-violations.sh won't run (xfs/437)
And there's probably a few more in that list as well.
IMO, any test that depends on a specific debug config or test
environment setup that aren't (or can't be) consistently used by the
entire developer base should be discouraged precisely because it
means different developers end up with different code coverage and
might miss problems they really should know about.
> (I might be able to lift the xfs-as-module requirement if "echo 1 >
> /sys/kernel/slab/*/validate" does what I think it might, since all we
> want to do is look for slab leaks, and those tests rmmod/modprobe as a
> brute force way of making the slab debug code check for leaks.)
Doesn't running fstests with KASAN enabled find those slab leaks?
> In case anyone's wondering, a solution to the knobs getting unset after
> an rmmod/modprobe cycle is to add a file
> /etc/udev/rules.d/99-fstester.rules containing:
>
> ACTION=="add|change", SUBSYSTEM=="module", DEVPATH=="/module/${module}", RUN+="/bin/sh -c \"echo ${value} > ${knob}\""
>
> which should be enough to keep LARP turned on.
I don't want it turned on unconditionally - I have to test both the
existing code path and the new LARP code path, so I'm constantly
toggling the value between test runs.
> Anyway, since this is a proposed test, I say that with this applied and
> the debug knob bits fixed, the whole thing is
> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
Thanks!
-Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH] xfs/larp: Make test failures debuggable
2022-05-06 21:48 ` Dave Chinner
@ 2022-05-07 5:34 ` Zorro Lang
0 siblings, 0 replies; 15+ messages in thread
From: Zorro Lang @ 2022-05-07 5:34 UTC (permalink / raw)
To: Dave Chinner; +Cc: Catherine Hoang, Darrick J. Wong, linux-xfs, fstests
On Sat, May 07, 2022 at 07:48:59AM +1000, Dave Chinner wrote:
> On Sat, May 07, 2022 at 04:02:12AM +0800, Zorro Lang wrote:
> > On Fri, May 06, 2022 at 06:08:08PM +0000, Catherine Hoang wrote:
> > > > On May 6, 2022, at 9:40 AM, Zorro Lang <zlang@redhat.com> wrote:
> > > >
> > > > On Fri, May 06, 2022 at 09:14:42AM -0700, Darrick J. Wong wrote:
> > > >> On Fri, May 06, 2022 at 05:51:41PM +1000, Dave Chinner wrote:
> > > >>> From: Dave Chinner <dchinner@redhat.com>
> > > >>>
> > > >>> Md5sum output for attributes created combined program output and
> > > >>> attribute values. This adds variable path names to the md5sum, so
> > > >>> there's no way to tell if the md5sum is actually correct for the
> > > >>> given attribute value that is returned as it's not constant from
> > > >>> test to test. Hence we can't actually say that the output is correct
> > > >>> because we can't reproduce exactly what we are hashing easily.
> > > >>>
> > > >>> Indeed, the last attr test in series (node w/ replace) had an
> > > >>> invalid md5sum. The attr value being produced after recovery was
> > > >>> correct, but the md5sum output match was failing. Golden output
> > > >>> appears to be wrong.
> > > >>>
> > > >>> Fix this issue by seperately dumping all the attributes on the inode
> > > >>> via a list operation to indicate their size, then dump the value of
> > > >>> the test attribute directly to md5sum. This means the md5sum for
> > > >>> the attributes using the same fixed values are all identical, so
> > > >>> it's easy to tell if the md5sum for a given test is correct. We also
> > > >>> check that all attributes that should be present after recovery are
> > > >>> still there (e.g. checks recovery didn't trash innocent bystanders).
> > > >>>
> > > >>> Further, the attribute replace tests replace an attribute with an
> > > >>> identical value, hence there is no way to tell if recovery has
> > > >>> resulted in the original being left behind or the new attribute
> > > >>> being fully recovered because both have the same name and value.
> > > >>> When replacing the attribute value, use a different sized value so
> > > >>> it is trivial to determine that we've recovered the new attribute
> > > >>> value correctly.
> > > >>>
> > > >>> Also, the test runs on the scratch device - there is no need to
> > > >>> remove the testdir in _cleanup. Doing so prevents post-mortem
> > > >>> failure analysis because it burns the dead, broken corpse to ash and
> > > >>> leaves no way of to determine cause of death.
> > > >>>
> > > >>> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> > > >>> ---
> > > >>>
> > > >>> Hi Catherine,
> > > >>>
> > > >>> These are all the mods I needed to make to be able to understand the
> > > >>> test failures I was getting as I debugged the new LARP recovery
> > > >>> algorithm I've written. You'll need to massage the test number in
> > > >>> this patch to apply it on top of your patch.
> > > >>>
> > > >>> I haven't added any new test cases yet, nor have I done anything to
> > > >>> manage the larp sysfs knob, but we'll need to do those in the near
> > > >>> future.
> > > >>>
> > > >>> Zorro, can you consider merging this test in the near future? We're
> > > >>> right at the point of merging the upstream kernel code and so really
> > > >>> need to start growing the test coverage of this feature, and this
> > > >>> test should simply not-run on kernels that don't have the feature
> > > >>> enabled....
> > > >>>
> > > >>> Cheers,
> > > >>>
> > > >>> Dave.
> > > >>> ---
> > > >>>
> > > >>> tests/xfs/600 | 20 +++++-----
> > > >>> tests/xfs/600.out | 109 ++++++++++++++++++++++++++++++++++++------------------
> > > >>> 2 files changed, 85 insertions(+), 44 deletions(-)
> > > >>>
> > > >>> diff --git a/tests/xfs/600 b/tests/xfs/600
> > > >>> index 252cdf27..84704646 100755
> > > >>> --- a/tests/xfs/600
> > > >>> +++ b/tests/xfs/600
> > > >>> @@ -16,7 +16,7 @@ _begin_fstest auto quick attr
> > > >>>
> > > >>> _cleanup()
> > > >>> {
> > > >>> - rm -rf $tmp.* $testdir
> > > >>> + rm -rf $tmp.*
> > > >>> test -w /sys/fs/xfs/debug/larp && \
> > > >>> echo 0 > /sys/fs/xfs/debug/larp
> > > >>
> > > >> Blergh, this ^^^^^^^^^ is going to need fixing too.
>
> Yes, I did point that out.
>
> > > >>
> > > >> Please save the old value, then write it back in the _cleanup function.
> > > >
> > > > Ok, I'm going to do that when I merge it,
>
> No, please don't. I don't want random changes added to the test on
> commit right now as I'm still actively working on it. I've said
> twice now that this needs fixing (3 if you count this mention) and
> that the test coverage also needs improving. If someone is still
> working on the tests, then why make more work for everyone by making
> unnecessary, unreviewed changes on commit?
>
> > > > if Catherine wouldn't like to do
> > > > more changes in a V8 patch. If this case still need more changes, please tell
> > > > me in time, and then it might have to wait the fstests release after next, if
> > > > too late.
> > > >
> > > > Thanks,
> > > > Zorro
> > >
> > > Based on Dave’s feedback, it looks like the patch will need a few more
> > > changes before it’s ready.
>
> That doesn't mean it can't be merged. It is a pain for mulitple
> people to collaborate on a test that isn't merged because the test
> number is not set in stone and chosing numbers always conflicts with
> other tests under development. Getting the test merged early makes
> knocking all the problems out of the test (and the code it is
> testing!) much, much easier.
Oh, I thought you're hurry to hope this test be merged, so I tried to help it to
catch the fstests release of this Sunday. If it's not such hurry, let's back to
the regular review workflow, I'll merge this test case *immediately* when I saw
a clear RVB on it from you. Then it'll catch the latest fstests release after
that (not guarantee this weekend). Hope that help.
Thanks,
Zorro
>
> > Great, that would be really helpful if you'd like to rebase this patch to fstests
> > for-next branch. And how about combine Dave's patch with your patch? I don't think
> > it's worth merging two seperated patches for one signle new case. What does Dave think?
> >
> > I just merged below two patchset[1] into xfs-5.18-fixes-1, then tried to test this case
> > (with you and Dave's patch). But it always failed as [2].
>
> You built a broken kernel as it has none of the dependencies and bug
> fixes that had already been committed to for-next for the new
> functionality to work correctly. I posted a V3 patchset last night
> and a published a git branch with all the kernel changes that you
> can use for testing if you want.
>
> > Please make sure it works
> > as expected with those kernel patches still reviewing,
>
> I did - the failures you see are expected from what you were
> testing. i.e. the test ran just fine, the kernel code you were
> running is buggy and you didn't update xfsprogs so logprint
> supported the new log types, hence the test (correctly) reported
> failures.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2022-05-07 5:34 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-23 3:37 [PATCH v7 0/1] xfstests: add log attribute replay test Catherine Hoang
2022-02-23 3:37 ` [PATCH v7 1/1] xfstests: Add Log Attribute Replay test Catherine Hoang
2022-03-01 20:46 ` Allison Henderson
2022-05-05 23:54 ` Dave Chinner
2022-05-06 7:51 ` [PATCH] xfs/larp: Make test failures debuggable Dave Chinner
2022-05-06 16:14 ` Darrick J. Wong
2022-05-06 16:40 ` Zorro Lang
2022-05-06 18:08 ` Catherine Hoang
2022-05-06 20:02 ` Zorro Lang
2022-05-06 21:48 ` Dave Chinner
2022-05-07 5:34 ` Zorro Lang
2022-05-06 21:55 ` Alli
2022-05-06 22:16 ` Dave Chinner
2022-05-06 16:28 ` Zorro Lang
2022-05-06 17:58 ` Catherine Hoang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.