* [NYE DELUGE 2/4] xfs: online repair in its entirety @ 2022-12-30 21:14 Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress Darrick J. Wong ` (17 more replies) 0 siblings, 18 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 21:14 UTC (permalink / raw) To: Dave Chinner, Allison Henderson, Chandan Babu R, Catherine Hoang, djwong Cc: xfs, greg.marsden, shirley.ma, konrad.wilk, linux-fsdevel, Matthew Wilcox, tpkelly, smahar, Christoph Hellwig, fstests, Zorro Lang, Carlos Maiolino Hi everyone, As I've mentioned several times throughout 2022, I would like to merge the online fsck feature in time for the 2023 LTS kernel. This is the second part of that effort. This deluge contains all of the online repair kernel code, a significant amount of restructuring of how repairs work in the userspace driver program, and a ton of fstests updates to provide automated fuzz testing and stress testing of forced repairs. Within the kernel section, the major pieces are the use of tmpfs files to provide pageable kernel memory for staging repair information; lightweight hooks into the main xfs filesystem for scrub via jump labels; coordinated inode scans for live index construction; and the atomic file mapping swap feature. Changes to the userspace driver program fall into two main categories: restructuring how repairs are scheduled so that they're tracked by inode or AG; establishing data dependency chains so that we scan and repair things in the correct order; and reworking the systemd background services to be more secure, enable periodic media scans, and provide some semblance of fs corruption reporting. The fstests changes are a substantial reworking of the fuzzing code to fit the testing described in the design documentation; adding stress testing of online repairs vs. fsstress; and functional tests for all the new features that ride in with online repair. For this review, I would like people to focus the following: - Are the major subsystems sufficiently documented that you could figure out what the code does? - Do you see any problems that are severe enough to cause long term support hassles? (e.g. bad API design, writing weird metadata to disk) - Can you spot mis-interactions between the subsystems? - What were my blind spots in devising this feature? - Are there missing pieces that you'd like to help build? - Can I just merge all of this? The one thing that is /not/ in scope for this review are requests for more refactoring of existing subsystems. While there are usually valid arguments for performing such cleanups, those are separate tasks to be prioritized separately. I will get to them after merging online fsck, because revising existing subsystems generally involves rebasing work in this patchset, which means the affected patches need re-reviewing. Unless it's absolutely necessary, this just creates more work for everybody. I've been running daily online **repairs** of every computer I own for the last eight months. All modifications so far have been to optimize data structures (holes in the xattr structures, excessively large rmap btrees, and bugs in quota resource counter updates). So far, no damage has resulted from these operations. All issues observed in that time have been corrected in this submission. Fuzz and stress testing of online repairs have been running well for a year now. As of this writing, online repair can fix slightly more things than offline repair, and the fsstress+repair long soak test has passed 100 million repairs with zero problems observed. (For comparison, the long soak fsx test recently passed 92 billion file operations, so online fsck has a ways to go...) As a warning, the patches will likely take several days to trickle in. While everyone else looks at this, I plan to prototype directory tree reconstruction with Allison's parent pointers v27 patchset. Having a user of that functionality is (I think) the last major hurdle to ensuring that parent pointers are a good fit for the problems that need solving, which in turn is the last requirement for merging that feature. --D ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata Darrick J. Wong ` (4 more replies) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] xfs: force rebuilding of metadata Darrick J. Wong ` (16 subsequent siblings) 17 siblings, 5 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, Now that we've created a whole loop control infrastructure to ensure that we can race fsstress and xfs_scrub's rmapbt scanning code, expand the testing to scrubs of every other type of space and file metadata. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-rtsummary fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-rtsummary --- common/fuzzy | 166 +++++++++++++++++++++++++++- common/quota | 64 +++++++++++ configure.ac | 5 + doc/group-names.txt | 1 include/builddefs.in | 4 + m4/package_libcdev.m4 | 47 ++++++++ m4/package_xfslibs.m4 | 16 +++ src/Makefile | 10 ++ src/xfsfind.c | 290 +++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/357 | 2 tests/xfs/782 | 37 ++++++ tests/xfs/782.out | 2 tests/xfs/783 | 37 ++++++ tests/xfs/783.out | 2 tests/xfs/784 | 37 ++++++ tests/xfs/784.out | 2 tests/xfs/785 | 37 ++++++ tests/xfs/785.out | 2 tests/xfs/786 | 38 ++++++ tests/xfs/786.out | 2 tests/xfs/787 | 38 ++++++ tests/xfs/787.out | 2 tests/xfs/788 | 38 ++++++ tests/xfs/788.out | 2 tests/xfs/789 | 39 +++++++ tests/xfs/789.out | 2 tests/xfs/790 | 39 +++++++ tests/xfs/790.out | 2 tests/xfs/791 | 40 +++++++ tests/xfs/791.out | 2 tests/xfs/792 | 38 ++++++ tests/xfs/792.out | 2 tests/xfs/793 | 37 ++++++ tests/xfs/793.out | 2 tests/xfs/794 | 39 +++++++ tests/xfs/794.out | 2 tests/xfs/795 | 39 +++++++ tests/xfs/795.out | 2 tests/xfs/796 | 37 ++++++ tests/xfs/796.out | 2 tests/xfs/797 | 40 +++++++ tests/xfs/797.out | 2 tests/xfs/798 | 44 +++++++ tests/xfs/798.out | 2 tests/xfs/799 | 38 ++++++ tests/xfs/799.out | 2 tests/xfs/800 | 40 +++++++ tests/xfs/800.out | 2 tests/xfs/801 | 47 ++++++++ tests/xfs/801.out | 2 tests/xfs/802 | 40 +++++++ tests/xfs/802.out | 2 tests/xfs/803 | 40 +++++++ tests/xfs/803.out | 2 tests/xfs/804 | 40 +++++++ tests/xfs/804.out | 2 tests/xfs/805 | 38 ++++++ tests/xfs/805.out | 2 tests/xfs/826 | 38 ++++++ tests/xfs/826.out | 2 tests/xfs/827 | 39 +++++++ tests/xfs/827.out | 2 62 files changed, 1662 insertions(+), 9 deletions(-) create mode 100644 src/xfsfind.c create mode 100755 tests/xfs/782 create mode 100644 tests/xfs/782.out create mode 100755 tests/xfs/783 create mode 100644 tests/xfs/783.out create mode 100755 tests/xfs/784 create mode 100644 tests/xfs/784.out create mode 100755 tests/xfs/785 create mode 100644 tests/xfs/785.out create mode 100755 tests/xfs/786 create mode 100644 tests/xfs/786.out create mode 100755 tests/xfs/787 create mode 100644 tests/xfs/787.out create mode 100755 tests/xfs/788 create mode 100644 tests/xfs/788.out create mode 100755 tests/xfs/789 create mode 100644 tests/xfs/789.out create mode 100755 tests/xfs/790 create mode 100644 tests/xfs/790.out create mode 100755 tests/xfs/791 create mode 100644 tests/xfs/791.out create mode 100755 tests/xfs/792 create mode 100644 tests/xfs/792.out create mode 100755 tests/xfs/793 create mode 100644 tests/xfs/793.out create mode 100755 tests/xfs/794 create mode 100644 tests/xfs/794.out create mode 100755 tests/xfs/795 create mode 100644 tests/xfs/795.out create mode 100755 tests/xfs/796 create mode 100644 tests/xfs/796.out create mode 100755 tests/xfs/797 create mode 100644 tests/xfs/797.out create mode 100755 tests/xfs/798 create mode 100644 tests/xfs/798.out create mode 100755 tests/xfs/799 create mode 100644 tests/xfs/799.out create mode 100755 tests/xfs/800 create mode 100644 tests/xfs/800.out create mode 100755 tests/xfs/801 create mode 100644 tests/xfs/801.out create mode 100755 tests/xfs/802 create mode 100644 tests/xfs/802.out create mode 100755 tests/xfs/803 create mode 100644 tests/xfs/803.out create mode 100755 tests/xfs/804 create mode 100644 tests/xfs/804.out create mode 100755 tests/xfs/805 create mode 100644 tests/xfs/805.out create mode 100755 tests/xfs/826 create mode 100644 tests/xfs/826.out create mode 100755 tests/xfs/827 create mode 100644 tests/xfs/827.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-05 13:04 ` Zorro Lang 2023-02-07 17:02 ` [PATCH v24.1 " Darrick J. Wong 2022-12-30 22:19 ` [PATCH 5/5] xfs: race fsstress with online scrubbers for file metadata Darrick J. Wong ` (3 subsequent siblings) 4 siblings, 2 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> For each XFS_SCRUB_TYPE_* that looks at AG or filesystem metadata, create a test that runs that scrubber in the foreground and fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/quota | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++ doc/group-names.txt | 1 + tests/xfs/782 | 37 +++++++++++++++++++++++++++++ tests/xfs/782.out | 2 ++ tests/xfs/783 | 37 +++++++++++++++++++++++++++++ tests/xfs/783.out | 2 ++ tests/xfs/784 | 37 +++++++++++++++++++++++++++++ tests/xfs/784.out | 2 ++ tests/xfs/785 | 37 +++++++++++++++++++++++++++++ tests/xfs/785.out | 2 ++ tests/xfs/786 | 38 ++++++++++++++++++++++++++++++ tests/xfs/786.out | 2 ++ tests/xfs/787 | 38 ++++++++++++++++++++++++++++++ tests/xfs/787.out | 2 ++ tests/xfs/788 | 38 ++++++++++++++++++++++++++++++ tests/xfs/788.out | 2 ++ tests/xfs/789 | 39 +++++++++++++++++++++++++++++++ tests/xfs/789.out | 2 ++ tests/xfs/790 | 39 +++++++++++++++++++++++++++++++ tests/xfs/790.out | 2 ++ tests/xfs/791 | 40 ++++++++++++++++++++++++++++++++ tests/xfs/791.out | 2 ++ tests/xfs/798 | 44 +++++++++++++++++++++++++++++++++++ tests/xfs/798.out | 2 ++ tests/xfs/800 | 40 ++++++++++++++++++++++++++++++++ tests/xfs/800.out | 2 ++ tests/xfs/801 | 47 +++++++++++++++++++++++++++++++++++++ tests/xfs/801.out | 2 ++ tests/xfs/802 | 40 ++++++++++++++++++++++++++++++++ tests/xfs/802.out | 2 ++ tests/xfs/803 | 40 ++++++++++++++++++++++++++++++++ tests/xfs/803.out | 2 ++ tests/xfs/804 | 40 ++++++++++++++++++++++++++++++++ tests/xfs/804.out | 2 ++ tests/xfs/805 | 38 ++++++++++++++++++++++++++++++ tests/xfs/805.out | 2 ++ 36 files changed, 768 insertions(+) create mode 100755 tests/xfs/782 create mode 100644 tests/xfs/782.out create mode 100755 tests/xfs/783 create mode 100644 tests/xfs/783.out create mode 100755 tests/xfs/784 create mode 100644 tests/xfs/784.out create mode 100755 tests/xfs/785 create mode 100644 tests/xfs/785.out create mode 100755 tests/xfs/786 create mode 100644 tests/xfs/786.out create mode 100755 tests/xfs/787 create mode 100644 tests/xfs/787.out create mode 100755 tests/xfs/788 create mode 100644 tests/xfs/788.out create mode 100755 tests/xfs/789 create mode 100644 tests/xfs/789.out create mode 100755 tests/xfs/790 create mode 100644 tests/xfs/790.out create mode 100755 tests/xfs/791 create mode 100644 tests/xfs/791.out create mode 100755 tests/xfs/798 create mode 100644 tests/xfs/798.out create mode 100755 tests/xfs/800 create mode 100644 tests/xfs/800.out create mode 100755 tests/xfs/801 create mode 100644 tests/xfs/801.out create mode 100755 tests/xfs/802 create mode 100644 tests/xfs/802.out create mode 100755 tests/xfs/803 create mode 100644 tests/xfs/803.out create mode 100755 tests/xfs/804 create mode 100644 tests/xfs/804.out create mode 100755 tests/xfs/805 create mode 100644 tests/xfs/805.out diff --git a/common/quota b/common/quota index 24251d092a..96b8d04424 100644 --- a/common/quota +++ b/common/quota @@ -53,6 +53,70 @@ _require_xfs_quota() [ -n "$XFS_QUOTA_PROG" ] || _notrun "XFS quota user tools not installed" } +# Check that a mounted fs has a particular type of quota accounting turned on. +# +# The first argument must be the data device of a mounted fs. It must not be +# the actual mountpath. +# +# The second argument is the quota type ('usrquota', 'grpquota', 'prjquota', +# 'any', or 'all'). +_xfs_quota_acct_enabled() +{ + local dev="$1" + local qtype="$2" + local f_args=() + local any= + + case "$qtype" in + "usrquota"|"uquota") f_args=("-U");; + "grpquota"|"gquota") f_args=("-G");; + "prjquota"|"pquota") f_args=("-P");; + "all") f_args=("-U" "-G" "-P");; + "any") f_args=("-U" "-G" "-P"); any=1;; + *) echo "$qtype: Unknown quota type."; return 1;; + esac + + if [ "$any" = "1" ]; then + for arg in "$f_args"; do + $here/src/feature "$arg" "$dev" && return 0 + done + return 1 + fi + + $here/src/feature "${f_args[@]}" "$dev" +} + +# Require that a mounted fs has a particular type of quota turned on. This +# takes the same arguments as _xfs_quota_acct_enabled. If the third argument is +# '-u' (or is empty and dev is $SCRATCH_DEV) the fs will be unmounted on +# failure. +_require_xfs_quota_acct_enabled() +{ + local dev="$1" + local qtype="$2" + local umount="$3" + local fsname="$dev" + + _xfs_quota_acct_enabled "$dev" "$qtype" "$qmode" && return 0 + + if [ -z "$umount" ] && [ "$dev" = "$SCRATCH_DEV" ]; then + umount="-u" + fi + test "$umount" = "-u" && umount "$dev" &>/dev/null + + case "$dev" in + "$TEST_DEV") fsname="test";; + "$SCRATCH_DEV") fsname="scratch";; + esac + + case "$qtype" in + "any") qtype="any quotas";; + "all") qtype="all quotas";; + esac + + _notrun "$qtype: accounting not enabled on $fsname filesystem." +} + # # checks that xfs_quota can operate on foreign (non-xfs) filesystems # Skips check on xfs filesystems, old xfs_quota is fine there. diff --git a/doc/group-names.txt b/doc/group-names.txt index ac219e05b3..771ce937ae 100644 --- a/doc/group-names.txt +++ b/doc/group-names.txt @@ -35,6 +35,7 @@ dangerous_fuzzers fuzzers that can crash your computer dangerous_norepair fuzzers to evaluate kernel metadata verifiers dangerous_online_repair fuzzers to evaluate xfs_scrub online repair dangerous_fsstress_repair race fsstress and xfs_scrub online repair +dangerous_fsstress_scrub race fsstress and xfs_scrub checking dangerous_repair fuzzers to evaluate xfs_repair offline repair dangerous_scrub fuzzers to evaluate xfs_scrub checking data data loss checkers diff --git a/tests/xfs/782 b/tests/xfs/782 new file mode 100755 index 0000000000..4801eda4bd --- /dev/null +++ b/tests/xfs/782 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 782 +# +# Race fsstress and superblock scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub sb %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/782.out b/tests/xfs/782.out new file mode 100644 index 0000000000..6e378f0e53 --- /dev/null +++ b/tests/xfs/782.out @@ -0,0 +1,2 @@ +QA output created by 782 +Silence is golden diff --git a/tests/xfs/783 b/tests/xfs/783 new file mode 100755 index 0000000000..379a9369e5 --- /dev/null +++ b/tests/xfs/783 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 783 +# +# Race fsstress and AGF scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub agf %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/783.out b/tests/xfs/783.out new file mode 100644 index 0000000000..2522395956 --- /dev/null +++ b/tests/xfs/783.out @@ -0,0 +1,2 @@ +QA output created by 783 +Silence is golden diff --git a/tests/xfs/784 b/tests/xfs/784 new file mode 100755 index 0000000000..2b89361c36 --- /dev/null +++ b/tests/xfs/784 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 784 +# +# Race fsstress and AGFL scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub agfl %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/784.out b/tests/xfs/784.out new file mode 100644 index 0000000000..48d9b24dd0 --- /dev/null +++ b/tests/xfs/784.out @@ -0,0 +1,2 @@ +QA output created by 784 +Silence is golden diff --git a/tests/xfs/785 b/tests/xfs/785 new file mode 100755 index 0000000000..34a13b058d --- /dev/null +++ b/tests/xfs/785 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 785 +# +# Race fsstress and AGI scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub agi %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/785.out b/tests/xfs/785.out new file mode 100644 index 0000000000..6ecb0c61b3 --- /dev/null +++ b/tests/xfs/785.out @@ -0,0 +1,2 @@ +QA output created by 785 +Silence is golden diff --git a/tests/xfs/786 b/tests/xfs/786 new file mode 100755 index 0000000000..157200ea8c --- /dev/null +++ b/tests/xfs/786 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 786 +# +# Race fsstress and freespace by block btree scrub for a while to see if we +# crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub bnobt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/786.out b/tests/xfs/786.out new file mode 100644 index 0000000000..ccb9167df9 --- /dev/null +++ b/tests/xfs/786.out @@ -0,0 +1,2 @@ +QA output created by 786 +Silence is golden diff --git a/tests/xfs/787 b/tests/xfs/787 new file mode 100755 index 0000000000..91eaf5a7af --- /dev/null +++ b/tests/xfs/787 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 787 +# +# Race fsstress and free space by length btree scrub for a while to see if we +# crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub cntbt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/787.out b/tests/xfs/787.out new file mode 100644 index 0000000000..fa7f038120 --- /dev/null +++ b/tests/xfs/787.out @@ -0,0 +1,2 @@ +QA output created by 787 +Silence is golden diff --git a/tests/xfs/788 b/tests/xfs/788 new file mode 100755 index 0000000000..f1369e5309 --- /dev/null +++ b/tests/xfs/788 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 788 +# +# Race fsstress and inode btree scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -x 'dir' -s "scrub inobt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/788.out b/tests/xfs/788.out new file mode 100644 index 0000000000..5ddd661113 --- /dev/null +++ b/tests/xfs/788.out @@ -0,0 +1,2 @@ +QA output created by 788 +Silence is golden diff --git a/tests/xfs/789 b/tests/xfs/789 new file mode 100755 index 0000000000..550ff2c690 --- /dev/null +++ b/tests/xfs/789 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 789 +# +# Race fsstress and free inode btree scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" finobt +_scratch_xfs_stress_scrub -x 'dir' -s "scrub finobt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/789.out b/tests/xfs/789.out new file mode 100644 index 0000000000..da88fc99cb --- /dev/null +++ b/tests/xfs/789.out @@ -0,0 +1,2 @@ +QA output created by 789 +Silence is golden diff --git a/tests/xfs/790 b/tests/xfs/790 new file mode 100755 index 0000000000..c4e5779ef7 --- /dev/null +++ b/tests/xfs/790 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 790 +# +# Race fsstress and reverse mapping btree scrub for a while to see if we crash +# or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" rmapbt +_scratch_xfs_stress_scrub -s "scrub rmapbt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/790.out b/tests/xfs/790.out new file mode 100644 index 0000000000..7102c590f0 --- /dev/null +++ b/tests/xfs/790.out @@ -0,0 +1,2 @@ +QA output created by 790 +Silence is golden diff --git a/tests/xfs/791 b/tests/xfs/791 new file mode 100755 index 0000000000..6939d910c9 --- /dev/null +++ b/tests/xfs/791 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 791 +# +# Race fsstress and reference count btree scrub for a while to see if we crash +# or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" reflink +_scratch_xfs_stress_scrub -s "scrub refcountbt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/791.out b/tests/xfs/791.out new file mode 100644 index 0000000000..758905371d --- /dev/null +++ b/tests/xfs/791.out @@ -0,0 +1,2 @@ +QA output created by 791 +Silence is golden diff --git a/tests/xfs/798 b/tests/xfs/798 new file mode 100755 index 0000000000..c5bdfad50a --- /dev/null +++ b/tests/xfs/798 @@ -0,0 +1,44 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 798 +# +# Race fsstress and fscounter scrub on the realtime device for a while to see +# if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime + +# Force all files to be allocated on the realtime device +_xfs_force_bdev realtime $SCRATCH_MNT + +_scratch_xfs_stress_scrub -s 'scrub fscounters' + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/798.out b/tests/xfs/798.out new file mode 100644 index 0000000000..216d6e93f4 --- /dev/null +++ b/tests/xfs/798.out @@ -0,0 +1,2 @@ +QA output created by 798 +Silence is golden diff --git a/tests/xfs/800 b/tests/xfs/800 new file mode 100755 index 0000000000..cbcfb5f5a6 --- /dev/null +++ b/tests/xfs/800 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 800 +# +# Race fsstress and realtime bitmap scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime +_scratch_xfs_stress_scrub -s "scrub rtbitmap" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/800.out b/tests/xfs/800.out new file mode 100644 index 0000000000..bdfaa2cecd --- /dev/null +++ b/tests/xfs/800.out @@ -0,0 +1,2 @@ +QA output created by 800 +Silence is golden diff --git a/tests/xfs/801 b/tests/xfs/801 new file mode 100755 index 0000000000..a51fab523b --- /dev/null +++ b/tests/xfs/801 @@ -0,0 +1,47 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 801 +# +# Race fsstress and realtime summary scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime + +# XXX the realtime summary scrubber isn't currently implemented upstream. +# Don't bother trying to test it on those kernels +$XFS_IO_PROG -c 'scrub rtsummary' -c 'scrub rtsummary' "$SCRATCH_MNT" 2>&1 | \ + grep -q 'Scan was not complete' && \ + _notrun "rtsummary scrub is incomplete" + +_scratch_xfs_stress_scrub -s "scrub rtsummary" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/801.out b/tests/xfs/801.out new file mode 100644 index 0000000000..39481b38e2 --- /dev/null +++ b/tests/xfs/801.out @@ -0,0 +1,2 @@ +QA output created by 801 +Silence is golden diff --git a/tests/xfs/802 b/tests/xfs/802 new file mode 100755 index 0000000000..1f3b83882e --- /dev/null +++ b/tests/xfs/802 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 802 +# +# Race fsstress and user quota scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" usrquota +_scratch_xfs_stress_scrub -s "scrub usrquota" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/802.out b/tests/xfs/802.out new file mode 100644 index 0000000000..a69c05391f --- /dev/null +++ b/tests/xfs/802.out @@ -0,0 +1,2 @@ +QA output created by 802 +Silence is golden diff --git a/tests/xfs/803 b/tests/xfs/803 new file mode 100755 index 0000000000..b2bb85672d --- /dev/null +++ b/tests/xfs/803 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 803 +# +# Race fsstress and group quota scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" grpquota +_scratch_xfs_stress_scrub -s "scrub grpquota" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/803.out b/tests/xfs/803.out new file mode 100644 index 0000000000..38ba741d0f --- /dev/null +++ b/tests/xfs/803.out @@ -0,0 +1,2 @@ +QA output created by 803 +Silence is golden diff --git a/tests/xfs/804 b/tests/xfs/804 new file mode 100755 index 0000000000..129724eb11 --- /dev/null +++ b/tests/xfs/804 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 804 +# +# Race fsstress and project quota scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" prjquota +_scratch_xfs_stress_scrub -s "scrub prjquota" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/804.out b/tests/xfs/804.out new file mode 100644 index 0000000000..5e0cb437e7 --- /dev/null +++ b/tests/xfs/804.out @@ -0,0 +1,2 @@ +QA output created by 804 +Silence is golden diff --git a/tests/xfs/805 b/tests/xfs/805 new file mode 100755 index 0000000000..aca9b9cdf4 --- /dev/null +++ b/tests/xfs/805 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 805 +# +# Race fsstress and summary counters scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub fscounters" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/805.out b/tests/xfs/805.out new file mode 100644 index 0000000000..ac324c5874 --- /dev/null +++ b/tests/xfs/805.out @@ -0,0 +1,2 @@ +QA output created by 805 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata 2022-12-30 22:19 ` [PATCH 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata Darrick J. Wong @ 2023-02-05 13:04 ` Zorro Lang 2023-02-07 16:58 ` Darrick J. Wong 2023-02-07 17:02 ` [PATCH v24.1 " Darrick J. Wong 1 sibling, 1 reply; 107+ messages in thread From: Zorro Lang @ 2023-02-05 13:04 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:06PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > For each XFS_SCRUB_TYPE_* that looks at AG or filesystem metadata, > create a test that runs that scrubber in the foreground and fsstress in > the background. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- > common/quota | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++ > doc/group-names.txt | 1 + [snip] > diff --git a/doc/group-names.txt b/doc/group-names.txt > index ac219e05b3..771ce937ae 100644 > --- a/doc/group-names.txt > +++ b/doc/group-names.txt > @@ -35,6 +35,7 @@ dangerous_fuzzers fuzzers that can crash your computer > dangerous_norepair fuzzers to evaluate kernel metadata verifiers > dangerous_online_repair fuzzers to evaluate xfs_scrub online repair > dangerous_fsstress_repair race fsstress and xfs_scrub online repair > +dangerous_fsstress_scrub race fsstress and xfs_scrub checking We've added this group name, so this patch will hit conflict. But I think I can use `git am --3way ...` to apply this patch forcibly :) Thanks, Zorro > dangerous_repair fuzzers to evaluate xfs_repair offline repair > dangerous_scrub fuzzers to evaluate xfs_scrub checking > data data loss checkers > diff --git a/tests/xfs/782 b/tests/xfs/782 > new file mode 100755 > index 0000000000..4801eda4bd > --- /dev/null > +++ b/tests/xfs/782 > @@ -0,0 +1,37 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 782 > +# > +# Race fsstress and superblock scrub for a while to see if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub sb %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/782.out b/tests/xfs/782.out > new file mode 100644 > index 0000000000..6e378f0e53 > --- /dev/null > +++ b/tests/xfs/782.out > @@ -0,0 +1,2 @@ > +QA output created by 782 > +Silence is golden > diff --git a/tests/xfs/783 b/tests/xfs/783 > new file mode 100755 > index 0000000000..379a9369e5 > --- /dev/null > +++ b/tests/xfs/783 > @@ -0,0 +1,37 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 783 > +# > +# Race fsstress and AGF scrub for a while to see if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub agf %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/783.out b/tests/xfs/783.out > new file mode 100644 > index 0000000000..2522395956 > --- /dev/null > +++ b/tests/xfs/783.out > @@ -0,0 +1,2 @@ > +QA output created by 783 > +Silence is golden > diff --git a/tests/xfs/784 b/tests/xfs/784 > new file mode 100755 > index 0000000000..2b89361c36 > --- /dev/null > +++ b/tests/xfs/784 > @@ -0,0 +1,37 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 784 > +# > +# Race fsstress and AGFL scrub for a while to see if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub agfl %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/784.out b/tests/xfs/784.out > new file mode 100644 > index 0000000000..48d9b24dd0 > --- /dev/null > +++ b/tests/xfs/784.out > @@ -0,0 +1,2 @@ > +QA output created by 784 > +Silence is golden > diff --git a/tests/xfs/785 b/tests/xfs/785 > new file mode 100755 > index 0000000000..34a13b058d > --- /dev/null > +++ b/tests/xfs/785 > @@ -0,0 +1,37 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 785 > +# > +# Race fsstress and AGI scrub for a while to see if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub agi %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/785.out b/tests/xfs/785.out > new file mode 100644 > index 0000000000..6ecb0c61b3 > --- /dev/null > +++ b/tests/xfs/785.out > @@ -0,0 +1,2 @@ > +QA output created by 785 > +Silence is golden > diff --git a/tests/xfs/786 b/tests/xfs/786 > new file mode 100755 > index 0000000000..157200ea8c > --- /dev/null > +++ b/tests/xfs/786 > @@ -0,0 +1,38 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 786 > +# > +# Race fsstress and freespace by block btree scrub for a while to see if we > +# crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub bnobt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/786.out b/tests/xfs/786.out > new file mode 100644 > index 0000000000..ccb9167df9 > --- /dev/null > +++ b/tests/xfs/786.out > @@ -0,0 +1,2 @@ > +QA output created by 786 > +Silence is golden > diff --git a/tests/xfs/787 b/tests/xfs/787 > new file mode 100755 > index 0000000000..91eaf5a7af > --- /dev/null > +++ b/tests/xfs/787 > @@ -0,0 +1,38 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 787 > +# > +# Race fsstress and free space by length btree scrub for a while to see if we > +# crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub cntbt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/787.out b/tests/xfs/787.out > new file mode 100644 > index 0000000000..fa7f038120 > --- /dev/null > +++ b/tests/xfs/787.out > @@ -0,0 +1,2 @@ > +QA output created by 787 > +Silence is golden > diff --git a/tests/xfs/788 b/tests/xfs/788 > new file mode 100755 > index 0000000000..f1369e5309 > --- /dev/null > +++ b/tests/xfs/788 > @@ -0,0 +1,38 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 788 > +# > +# Race fsstress and inode btree scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -x 'dir' -s "scrub inobt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/788.out b/tests/xfs/788.out > new file mode 100644 > index 0000000000..5ddd661113 > --- /dev/null > +++ b/tests/xfs/788.out > @@ -0,0 +1,2 @@ > +QA output created by 788 > +Silence is golden > diff --git a/tests/xfs/789 b/tests/xfs/789 > new file mode 100755 > index 0000000000..550ff2c690 > --- /dev/null > +++ b/tests/xfs/789 > @@ -0,0 +1,39 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 789 > +# > +# Race fsstress and free inode btree scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" finobt > +_scratch_xfs_stress_scrub -x 'dir' -s "scrub finobt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/789.out b/tests/xfs/789.out > new file mode 100644 > index 0000000000..da88fc99cb > --- /dev/null > +++ b/tests/xfs/789.out > @@ -0,0 +1,2 @@ > +QA output created by 789 > +Silence is golden > diff --git a/tests/xfs/790 b/tests/xfs/790 > new file mode 100755 > index 0000000000..c4e5779ef7 > --- /dev/null > +++ b/tests/xfs/790 > @@ -0,0 +1,39 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 790 > +# > +# Race fsstress and reverse mapping btree scrub for a while to see if we crash > +# or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" rmapbt > +_scratch_xfs_stress_scrub -s "scrub rmapbt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/790.out b/tests/xfs/790.out > new file mode 100644 > index 0000000000..7102c590f0 > --- /dev/null > +++ b/tests/xfs/790.out > @@ -0,0 +1,2 @@ > +QA output created by 790 > +Silence is golden > diff --git a/tests/xfs/791 b/tests/xfs/791 > new file mode 100755 > index 0000000000..6939d910c9 > --- /dev/null > +++ b/tests/xfs/791 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 791 > +# > +# Race fsstress and reference count btree scrub for a while to see if we crash > +# or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/reflink > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" reflink > +_scratch_xfs_stress_scrub -s "scrub refcountbt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/791.out b/tests/xfs/791.out > new file mode 100644 > index 0000000000..758905371d > --- /dev/null > +++ b/tests/xfs/791.out > @@ -0,0 +1,2 @@ > +QA output created by 791 > +Silence is golden > diff --git a/tests/xfs/798 b/tests/xfs/798 > new file mode 100755 > index 0000000000..c5bdfad50a > --- /dev/null > +++ b/tests/xfs/798 > @@ -0,0 +1,44 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# Copyright (c) 2022 Oracle. All Rights Reserved. > +# > +# FS QA Test No. 798 > +# > +# Race fsstress and fscounter scrub on the realtime device for a while to see > +# if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_realtime > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" realtime > + > +# Force all files to be allocated on the realtime device > +_xfs_force_bdev realtime $SCRATCH_MNT > + > +_scratch_xfs_stress_scrub -s 'scrub fscounters' > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/798.out b/tests/xfs/798.out > new file mode 100644 > index 0000000000..216d6e93f4 > --- /dev/null > +++ b/tests/xfs/798.out > @@ -0,0 +1,2 @@ > +QA output created by 798 > +Silence is golden > diff --git a/tests/xfs/800 b/tests/xfs/800 > new file mode 100755 > index 0000000000..cbcfb5f5a6 > --- /dev/null > +++ b/tests/xfs/800 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 800 > +# > +# Race fsstress and realtime bitmap scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_realtime > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" realtime > +_scratch_xfs_stress_scrub -s "scrub rtbitmap" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/800.out b/tests/xfs/800.out > new file mode 100644 > index 0000000000..bdfaa2cecd > --- /dev/null > +++ b/tests/xfs/800.out > @@ -0,0 +1,2 @@ > +QA output created by 800 > +Silence is golden > diff --git a/tests/xfs/801 b/tests/xfs/801 > new file mode 100755 > index 0000000000..a51fab523b > --- /dev/null > +++ b/tests/xfs/801 > @@ -0,0 +1,47 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 801 > +# > +# Race fsstress and realtime summary scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_realtime > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" realtime > + > +# XXX the realtime summary scrubber isn't currently implemented upstream. > +# Don't bother trying to test it on those kernels > +$XFS_IO_PROG -c 'scrub rtsummary' -c 'scrub rtsummary' "$SCRATCH_MNT" 2>&1 | \ > + grep -q 'Scan was not complete' && \ > + _notrun "rtsummary scrub is incomplete" > + > +_scratch_xfs_stress_scrub -s "scrub rtsummary" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/801.out b/tests/xfs/801.out > new file mode 100644 > index 0000000000..39481b38e2 > --- /dev/null > +++ b/tests/xfs/801.out > @@ -0,0 +1,2 @@ > +QA output created by 801 > +Silence is golden > diff --git a/tests/xfs/802 b/tests/xfs/802 > new file mode 100755 > index 0000000000..1f3b83882e > --- /dev/null > +++ b/tests/xfs/802 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 802 > +# > +# Race fsstress and user quota scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" usrquota > +_scratch_xfs_stress_scrub -s "scrub usrquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/802.out b/tests/xfs/802.out > new file mode 100644 > index 0000000000..a69c05391f > --- /dev/null > +++ b/tests/xfs/802.out > @@ -0,0 +1,2 @@ > +QA output created by 802 > +Silence is golden > diff --git a/tests/xfs/803 b/tests/xfs/803 > new file mode 100755 > index 0000000000..b2bb85672d > --- /dev/null > +++ b/tests/xfs/803 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 803 > +# > +# Race fsstress and group quota scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" grpquota > +_scratch_xfs_stress_scrub -s "scrub grpquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/803.out b/tests/xfs/803.out > new file mode 100644 > index 0000000000..38ba741d0f > --- /dev/null > +++ b/tests/xfs/803.out > @@ -0,0 +1,2 @@ > +QA output created by 803 > +Silence is golden > diff --git a/tests/xfs/804 b/tests/xfs/804 > new file mode 100755 > index 0000000000..129724eb11 > --- /dev/null > +++ b/tests/xfs/804 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 804 > +# > +# Race fsstress and project quota scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" prjquota > +_scratch_xfs_stress_scrub -s "scrub prjquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/804.out b/tests/xfs/804.out > new file mode 100644 > index 0000000000..5e0cb437e7 > --- /dev/null > +++ b/tests/xfs/804.out > @@ -0,0 +1,2 @@ > +QA output created by 804 > +Silence is golden > diff --git a/tests/xfs/805 b/tests/xfs/805 > new file mode 100755 > index 0000000000..aca9b9cdf4 > --- /dev/null > +++ b/tests/xfs/805 > @@ -0,0 +1,38 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 805 > +# > +# Race fsstress and summary counters scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub fscounters" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/805.out b/tests/xfs/805.out > new file mode 100644 > index 0000000000..ac324c5874 > --- /dev/null > +++ b/tests/xfs/805.out > @@ -0,0 +1,2 @@ > +QA output created by 805 > +Silence is golden > ^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata 2023-02-05 13:04 ` Zorro Lang @ 2023-02-07 16:58 ` Darrick J. Wong 0 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2023-02-07 16:58 UTC (permalink / raw) To: Zorro Lang; +Cc: linux-xfs, fstests On Sun, Feb 05, 2023 at 09:04:12PM +0800, Zorro Lang wrote: > On Fri, Dec 30, 2022 at 02:19:06PM -0800, Darrick J. Wong wrote: > > From: Darrick J. Wong <djwong@kernel.org> > > > > For each XFS_SCRUB_TYPE_* that looks at AG or filesystem metadata, > > create a test that runs that scrubber in the foreground and fsstress in > > the background. > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > --- > > common/quota | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++ > > doc/group-names.txt | 1 + > > [snip] > > > diff --git a/doc/group-names.txt b/doc/group-names.txt > > index ac219e05b3..771ce937ae 100644 > > --- a/doc/group-names.txt > > +++ b/doc/group-names.txt > > @@ -35,6 +35,7 @@ dangerous_fuzzers fuzzers that can crash your computer > > dangerous_norepair fuzzers to evaluate kernel metadata verifiers > > dangerous_online_repair fuzzers to evaluate xfs_scrub online repair > > dangerous_fsstress_repair race fsstress and xfs_scrub online repair > > +dangerous_fsstress_scrub race fsstress and xfs_scrub checking > > We've added this group name, so this patch will hit conflict. But I think I > can use `git am --3way ...` to apply this patch forcibly :) Or I'll just repost this patch, since I already/always rebase everything. --D > Thanks, > Zorro > > > dangerous_repair fuzzers to evaluate xfs_repair offline repair > > dangerous_scrub fuzzers to evaluate xfs_scrub checking > > data data loss checkers > > diff --git a/tests/xfs/782 b/tests/xfs/782 > > new file mode 100755 > > index 0000000000..4801eda4bd > > --- /dev/null > > +++ b/tests/xfs/782 > > @@ -0,0 +1,37 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 782 > > +# > > +# Race fsstress and superblock scrub for a while to see if we crash or livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_scratch_xfs_stress_scrub -s "scrub sb %agno%" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/782.out b/tests/xfs/782.out > > new file mode 100644 > > index 0000000000..6e378f0e53 > > --- /dev/null > > +++ b/tests/xfs/782.out > > @@ -0,0 +1,2 @@ > > +QA output created by 782 > > +Silence is golden > > diff --git a/tests/xfs/783 b/tests/xfs/783 > > new file mode 100755 > > index 0000000000..379a9369e5 > > --- /dev/null > > +++ b/tests/xfs/783 > > @@ -0,0 +1,37 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 783 > > +# > > +# Race fsstress and AGF scrub for a while to see if we crash or livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_scratch_xfs_stress_scrub -s "scrub agf %agno%" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/783.out b/tests/xfs/783.out > > new file mode 100644 > > index 0000000000..2522395956 > > --- /dev/null > > +++ b/tests/xfs/783.out > > @@ -0,0 +1,2 @@ > > +QA output created by 783 > > +Silence is golden > > diff --git a/tests/xfs/784 b/tests/xfs/784 > > new file mode 100755 > > index 0000000000..2b89361c36 > > --- /dev/null > > +++ b/tests/xfs/784 > > @@ -0,0 +1,37 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 784 > > +# > > +# Race fsstress and AGFL scrub for a while to see if we crash or livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_scratch_xfs_stress_scrub -s "scrub agfl %agno%" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/784.out b/tests/xfs/784.out > > new file mode 100644 > > index 0000000000..48d9b24dd0 > > --- /dev/null > > +++ b/tests/xfs/784.out > > @@ -0,0 +1,2 @@ > > +QA output created by 784 > > +Silence is golden > > diff --git a/tests/xfs/785 b/tests/xfs/785 > > new file mode 100755 > > index 0000000000..34a13b058d > > --- /dev/null > > +++ b/tests/xfs/785 > > @@ -0,0 +1,37 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 785 > > +# > > +# Race fsstress and AGI scrub for a while to see if we crash or livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_scratch_xfs_stress_scrub -s "scrub agi %agno%" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/785.out b/tests/xfs/785.out > > new file mode 100644 > > index 0000000000..6ecb0c61b3 > > --- /dev/null > > +++ b/tests/xfs/785.out > > @@ -0,0 +1,2 @@ > > +QA output created by 785 > > +Silence is golden > > diff --git a/tests/xfs/786 b/tests/xfs/786 > > new file mode 100755 > > index 0000000000..157200ea8c > > --- /dev/null > > +++ b/tests/xfs/786 > > @@ -0,0 +1,38 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 786 > > +# > > +# Race fsstress and freespace by block btree scrub for a while to see if we > > +# crash or livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_scratch_xfs_stress_scrub -s "scrub bnobt %agno%" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/786.out b/tests/xfs/786.out > > new file mode 100644 > > index 0000000000..ccb9167df9 > > --- /dev/null > > +++ b/tests/xfs/786.out > > @@ -0,0 +1,2 @@ > > +QA output created by 786 > > +Silence is golden > > diff --git a/tests/xfs/787 b/tests/xfs/787 > > new file mode 100755 > > index 0000000000..91eaf5a7af > > --- /dev/null > > +++ b/tests/xfs/787 > > @@ -0,0 +1,38 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 787 > > +# > > +# Race fsstress and free space by length btree scrub for a while to see if we > > +# crash or livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_scratch_xfs_stress_scrub -s "scrub cntbt %agno%" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/787.out b/tests/xfs/787.out > > new file mode 100644 > > index 0000000000..fa7f038120 > > --- /dev/null > > +++ b/tests/xfs/787.out > > @@ -0,0 +1,2 @@ > > +QA output created by 787 > > +Silence is golden > > diff --git a/tests/xfs/788 b/tests/xfs/788 > > new file mode 100755 > > index 0000000000..f1369e5309 > > --- /dev/null > > +++ b/tests/xfs/788 > > @@ -0,0 +1,38 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 788 > > +# > > +# Race fsstress and inode btree scrub for a while to see if we crash or > > +# livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_scratch_xfs_stress_scrub -x 'dir' -s "scrub inobt %agno%" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/788.out b/tests/xfs/788.out > > new file mode 100644 > > index 0000000000..5ddd661113 > > --- /dev/null > > +++ b/tests/xfs/788.out > > @@ -0,0 +1,2 @@ > > +QA output created by 788 > > +Silence is golden > > diff --git a/tests/xfs/789 b/tests/xfs/789 > > new file mode 100755 > > index 0000000000..550ff2c690 > > --- /dev/null > > +++ b/tests/xfs/789 > > @@ -0,0 +1,39 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 789 > > +# > > +# Race fsstress and free inode btree scrub for a while to see if we crash or > > +# livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_require_xfs_has_feature "$SCRATCH_MNT" finobt > > +_scratch_xfs_stress_scrub -x 'dir' -s "scrub finobt %agno%" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/789.out b/tests/xfs/789.out > > new file mode 100644 > > index 0000000000..da88fc99cb > > --- /dev/null > > +++ b/tests/xfs/789.out > > @@ -0,0 +1,2 @@ > > +QA output created by 789 > > +Silence is golden > > diff --git a/tests/xfs/790 b/tests/xfs/790 > > new file mode 100755 > > index 0000000000..c4e5779ef7 > > --- /dev/null > > +++ b/tests/xfs/790 > > @@ -0,0 +1,39 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 790 > > +# > > +# Race fsstress and reverse mapping btree scrub for a while to see if we crash > > +# or livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_require_xfs_has_feature "$SCRATCH_MNT" rmapbt > > +_scratch_xfs_stress_scrub -s "scrub rmapbt %agno%" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/790.out b/tests/xfs/790.out > > new file mode 100644 > > index 0000000000..7102c590f0 > > --- /dev/null > > +++ b/tests/xfs/790.out > > @@ -0,0 +1,2 @@ > > +QA output created by 790 > > +Silence is golden > > diff --git a/tests/xfs/791 b/tests/xfs/791 > > new file mode 100755 > > index 0000000000..6939d910c9 > > --- /dev/null > > +++ b/tests/xfs/791 > > @@ -0,0 +1,40 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 791 > > +# > > +# Race fsstress and reference count btree scrub for a while to see if we crash > > +# or livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > +. ./common/reflink > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_require_xfs_has_feature "$SCRATCH_MNT" reflink > > +_scratch_xfs_stress_scrub -s "scrub refcountbt %agno%" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/791.out b/tests/xfs/791.out > > new file mode 100644 > > index 0000000000..758905371d > > --- /dev/null > > +++ b/tests/xfs/791.out > > @@ -0,0 +1,2 @@ > > +QA output created by 791 > > +Silence is golden > > diff --git a/tests/xfs/798 b/tests/xfs/798 > > new file mode 100755 > > index 0000000000..c5bdfad50a > > --- /dev/null > > +++ b/tests/xfs/798 > > @@ -0,0 +1,44 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0-or-later > > +# Copyright (c) 2022 Oracle. All Rights Reserved. > > +# > > +# FS QA Test No. 798 > > +# > > +# Race fsstress and fscounter scrub on the realtime device for a while to see > > +# if we crash or livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_realtime > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_require_xfs_has_feature "$SCRATCH_MNT" realtime > > + > > +# Force all files to be allocated on the realtime device > > +_xfs_force_bdev realtime $SCRATCH_MNT > > + > > +_scratch_xfs_stress_scrub -s 'scrub fscounters' > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/798.out b/tests/xfs/798.out > > new file mode 100644 > > index 0000000000..216d6e93f4 > > --- /dev/null > > +++ b/tests/xfs/798.out > > @@ -0,0 +1,2 @@ > > +QA output created by 798 > > +Silence is golden > > diff --git a/tests/xfs/800 b/tests/xfs/800 > > new file mode 100755 > > index 0000000000..cbcfb5f5a6 > > --- /dev/null > > +++ b/tests/xfs/800 > > @@ -0,0 +1,40 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 800 > > +# > > +# Race fsstress and realtime bitmap scrub for a while to see if we crash or > > +# livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_realtime > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_require_xfs_has_feature "$SCRATCH_MNT" realtime > > +_scratch_xfs_stress_scrub -s "scrub rtbitmap" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/800.out b/tests/xfs/800.out > > new file mode 100644 > > index 0000000000..bdfaa2cecd > > --- /dev/null > > +++ b/tests/xfs/800.out > > @@ -0,0 +1,2 @@ > > +QA output created by 800 > > +Silence is golden > > diff --git a/tests/xfs/801 b/tests/xfs/801 > > new file mode 100755 > > index 0000000000..a51fab523b > > --- /dev/null > > +++ b/tests/xfs/801 > > @@ -0,0 +1,47 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 801 > > +# > > +# Race fsstress and realtime summary scrub for a while to see if we crash or > > +# livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_realtime > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_require_xfs_has_feature "$SCRATCH_MNT" realtime > > + > > +# XXX the realtime summary scrubber isn't currently implemented upstream. > > +# Don't bother trying to test it on those kernels > > +$XFS_IO_PROG -c 'scrub rtsummary' -c 'scrub rtsummary' "$SCRATCH_MNT" 2>&1 | \ > > + grep -q 'Scan was not complete' && \ > > + _notrun "rtsummary scrub is incomplete" > > + > > +_scratch_xfs_stress_scrub -s "scrub rtsummary" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/801.out b/tests/xfs/801.out > > new file mode 100644 > > index 0000000000..39481b38e2 > > --- /dev/null > > +++ b/tests/xfs/801.out > > @@ -0,0 +1,2 @@ > > +QA output created by 801 > > +Silence is golden > > diff --git a/tests/xfs/802 b/tests/xfs/802 > > new file mode 100755 > > index 0000000000..1f3b83882e > > --- /dev/null > > +++ b/tests/xfs/802 > > @@ -0,0 +1,40 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 802 > > +# > > +# Race fsstress and user quota scrub for a while to see if we crash or > > +# livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > +. ./common/quota > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" usrquota > > +_scratch_xfs_stress_scrub -s "scrub usrquota" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/802.out b/tests/xfs/802.out > > new file mode 100644 > > index 0000000000..a69c05391f > > --- /dev/null > > +++ b/tests/xfs/802.out > > @@ -0,0 +1,2 @@ > > +QA output created by 802 > > +Silence is golden > > diff --git a/tests/xfs/803 b/tests/xfs/803 > > new file mode 100755 > > index 0000000000..b2bb85672d > > --- /dev/null > > +++ b/tests/xfs/803 > > @@ -0,0 +1,40 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 803 > > +# > > +# Race fsstress and group quota scrub for a while to see if we crash or > > +# livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > +. ./common/quota > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" grpquota > > +_scratch_xfs_stress_scrub -s "scrub grpquota" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/803.out b/tests/xfs/803.out > > new file mode 100644 > > index 0000000000..38ba741d0f > > --- /dev/null > > +++ b/tests/xfs/803.out > > @@ -0,0 +1,2 @@ > > +QA output created by 803 > > +Silence is golden > > diff --git a/tests/xfs/804 b/tests/xfs/804 > > new file mode 100755 > > index 0000000000..129724eb11 > > --- /dev/null > > +++ b/tests/xfs/804 > > @@ -0,0 +1,40 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 804 > > +# > > +# Race fsstress and project quota scrub for a while to see if we crash or > > +# livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > +. ./common/quota > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" prjquota > > +_scratch_xfs_stress_scrub -s "scrub prjquota" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/804.out b/tests/xfs/804.out > > new file mode 100644 > > index 0000000000..5e0cb437e7 > > --- /dev/null > > +++ b/tests/xfs/804.out > > @@ -0,0 +1,2 @@ > > +QA output created by 804 > > +Silence is golden > > diff --git a/tests/xfs/805 b/tests/xfs/805 > > new file mode 100755 > > index 0000000000..aca9b9cdf4 > > --- /dev/null > > +++ b/tests/xfs/805 > > @@ -0,0 +1,38 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0 > > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > > +# > > +# FS QA Test No. 805 > > +# > > +# Race fsstress and summary counters scrub for a while to see if we crash or > > +# livelock. > > +# > > +. ./common/preamble > > +_begin_fstest scrub dangerous_fsstress_scrub > > + > > +_cleanup() { > > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > > + cd / > > + rm -r -f $tmp.* > > +} > > +_register_cleanup "_cleanup" BUS > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/fuzzy > > +. ./common/inject > > +. ./common/xfs > > + > > +# real QA test starts here > > +_supported_fs xfs > > +_require_scratch > > +_require_xfs_stress_scrub > > + > > +_scratch_mkfs > "$seqres.full" 2>&1 > > +_scratch_mount > > +_scratch_xfs_stress_scrub -s "scrub fscounters" > > + > > +# success, all done > > +echo Silence is golden > > +status=0 > > +exit > > diff --git a/tests/xfs/805.out b/tests/xfs/805.out > > new file mode 100644 > > index 0000000000..ac324c5874 > > --- /dev/null > > +++ b/tests/xfs/805.out > > @@ -0,0 +1,2 @@ > > +QA output created by 805 > > +Silence is golden > > > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v24.1 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata 2022-12-30 22:19 ` [PATCH 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata Darrick J. Wong 2023-02-05 13:04 ` Zorro Lang @ 2023-02-07 17:02 ` Darrick J. Wong 2023-02-07 18:45 ` Zorro Lang 1 sibling, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2023-02-07 17:02 UTC (permalink / raw) To: zlang; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> For each XFS_SCRUB_TYPE_* that looks at AG or filesystem metadata, create a test that runs that scrubber in the foreground and fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- v24.1: rebase against newer upstream to get rid of merge conflicts --- common/quota | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/782 | 37 +++++++++++++++++++++++++++++++ tests/xfs/782.out | 2 ++ tests/xfs/783 | 37 +++++++++++++++++++++++++++++++ tests/xfs/783.out | 2 ++ tests/xfs/784 | 37 +++++++++++++++++++++++++++++++ tests/xfs/784.out | 2 ++ tests/xfs/785 | 37 +++++++++++++++++++++++++++++++ tests/xfs/785.out | 2 ++ tests/xfs/786 | 38 +++++++++++++++++++++++++++++++ tests/xfs/786.out | 2 ++ tests/xfs/787 | 38 +++++++++++++++++++++++++++++++ tests/xfs/787.out | 2 ++ tests/xfs/788 | 38 +++++++++++++++++++++++++++++++ tests/xfs/788.out | 2 ++ tests/xfs/789 | 39 ++++++++++++++++++++++++++++++++ tests/xfs/789.out | 2 ++ tests/xfs/790 | 39 ++++++++++++++++++++++++++++++++ tests/xfs/790.out | 2 ++ tests/xfs/791 | 40 +++++++++++++++++++++++++++++++++ tests/xfs/791.out | 2 ++ tests/xfs/798 | 44 ++++++++++++++++++++++++++++++++++++ tests/xfs/798.out | 2 ++ tests/xfs/800 | 40 +++++++++++++++++++++++++++++++++ tests/xfs/800.out | 2 ++ tests/xfs/801 | 47 +++++++++++++++++++++++++++++++++++++++ tests/xfs/801.out | 2 ++ tests/xfs/802 | 40 +++++++++++++++++++++++++++++++++ tests/xfs/802.out | 2 ++ tests/xfs/803 | 40 +++++++++++++++++++++++++++++++++ tests/xfs/803.out | 2 ++ tests/xfs/804 | 40 +++++++++++++++++++++++++++++++++ tests/xfs/804.out | 2 ++ tests/xfs/805 | 38 +++++++++++++++++++++++++++++++ tests/xfs/805.out | 2 ++ 35 files changed, 767 insertions(+) create mode 100755 tests/xfs/782 create mode 100644 tests/xfs/782.out create mode 100755 tests/xfs/783 create mode 100644 tests/xfs/783.out create mode 100755 tests/xfs/784 create mode 100644 tests/xfs/784.out create mode 100755 tests/xfs/785 create mode 100644 tests/xfs/785.out create mode 100755 tests/xfs/786 create mode 100644 tests/xfs/786.out create mode 100755 tests/xfs/787 create mode 100644 tests/xfs/787.out create mode 100755 tests/xfs/788 create mode 100644 tests/xfs/788.out create mode 100755 tests/xfs/789 create mode 100644 tests/xfs/789.out create mode 100755 tests/xfs/790 create mode 100644 tests/xfs/790.out create mode 100755 tests/xfs/791 create mode 100644 tests/xfs/791.out create mode 100755 tests/xfs/798 create mode 100644 tests/xfs/798.out create mode 100755 tests/xfs/800 create mode 100644 tests/xfs/800.out create mode 100755 tests/xfs/801 create mode 100644 tests/xfs/801.out create mode 100755 tests/xfs/802 create mode 100644 tests/xfs/802.out create mode 100755 tests/xfs/803 create mode 100644 tests/xfs/803.out create mode 100755 tests/xfs/804 create mode 100644 tests/xfs/804.out create mode 100755 tests/xfs/805 create mode 100644 tests/xfs/805.out diff --git a/common/quota b/common/quota index 24251d092a..96b8d04424 100644 --- a/common/quota +++ b/common/quota @@ -53,6 +53,70 @@ _require_xfs_quota() [ -n "$XFS_QUOTA_PROG" ] || _notrun "XFS quota user tools not installed" } +# Check that a mounted fs has a particular type of quota accounting turned on. +# +# The first argument must be the data device of a mounted fs. It must not be +# the actual mountpath. +# +# The second argument is the quota type ('usrquota', 'grpquota', 'prjquota', +# 'any', or 'all'). +_xfs_quota_acct_enabled() +{ + local dev="$1" + local qtype="$2" + local f_args=() + local any= + + case "$qtype" in + "usrquota"|"uquota") f_args=("-U");; + "grpquota"|"gquota") f_args=("-G");; + "prjquota"|"pquota") f_args=("-P");; + "all") f_args=("-U" "-G" "-P");; + "any") f_args=("-U" "-G" "-P"); any=1;; + *) echo "$qtype: Unknown quota type."; return 1;; + esac + + if [ "$any" = "1" ]; then + for arg in "$f_args"; do + $here/src/feature "$arg" "$dev" && return 0 + done + return 1 + fi + + $here/src/feature "${f_args[@]}" "$dev" +} + +# Require that a mounted fs has a particular type of quota turned on. This +# takes the same arguments as _xfs_quota_acct_enabled. If the third argument is +# '-u' (or is empty and dev is $SCRATCH_DEV) the fs will be unmounted on +# failure. +_require_xfs_quota_acct_enabled() +{ + local dev="$1" + local qtype="$2" + local umount="$3" + local fsname="$dev" + + _xfs_quota_acct_enabled "$dev" "$qtype" "$qmode" && return 0 + + if [ -z "$umount" ] && [ "$dev" = "$SCRATCH_DEV" ]; then + umount="-u" + fi + test "$umount" = "-u" && umount "$dev" &>/dev/null + + case "$dev" in + "$TEST_DEV") fsname="test";; + "$SCRATCH_DEV") fsname="scratch";; + esac + + case "$qtype" in + "any") qtype="any quotas";; + "all") qtype="all quotas";; + esac + + _notrun "$qtype: accounting not enabled on $fsname filesystem." +} + # # checks that xfs_quota can operate on foreign (non-xfs) filesystems # Skips check on xfs filesystems, old xfs_quota is fine there. diff --git a/tests/xfs/782 b/tests/xfs/782 new file mode 100755 index 0000000000..4801eda4bd --- /dev/null +++ b/tests/xfs/782 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 782 +# +# Race fsstress and superblock scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub sb %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/782.out b/tests/xfs/782.out new file mode 100644 index 0000000000..6e378f0e53 --- /dev/null +++ b/tests/xfs/782.out @@ -0,0 +1,2 @@ +QA output created by 782 +Silence is golden diff --git a/tests/xfs/783 b/tests/xfs/783 new file mode 100755 index 0000000000..379a9369e5 --- /dev/null +++ b/tests/xfs/783 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 783 +# +# Race fsstress and AGF scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub agf %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/783.out b/tests/xfs/783.out new file mode 100644 index 0000000000..2522395956 --- /dev/null +++ b/tests/xfs/783.out @@ -0,0 +1,2 @@ +QA output created by 783 +Silence is golden diff --git a/tests/xfs/784 b/tests/xfs/784 new file mode 100755 index 0000000000..2b89361c36 --- /dev/null +++ b/tests/xfs/784 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 784 +# +# Race fsstress and AGFL scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub agfl %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/784.out b/tests/xfs/784.out new file mode 100644 index 0000000000..48d9b24dd0 --- /dev/null +++ b/tests/xfs/784.out @@ -0,0 +1,2 @@ +QA output created by 784 +Silence is golden diff --git a/tests/xfs/785 b/tests/xfs/785 new file mode 100755 index 0000000000..34a13b058d --- /dev/null +++ b/tests/xfs/785 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 785 +# +# Race fsstress and AGI scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub agi %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/785.out b/tests/xfs/785.out new file mode 100644 index 0000000000..6ecb0c61b3 --- /dev/null +++ b/tests/xfs/785.out @@ -0,0 +1,2 @@ +QA output created by 785 +Silence is golden diff --git a/tests/xfs/786 b/tests/xfs/786 new file mode 100755 index 0000000000..157200ea8c --- /dev/null +++ b/tests/xfs/786 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 786 +# +# Race fsstress and freespace by block btree scrub for a while to see if we +# crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub bnobt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/786.out b/tests/xfs/786.out new file mode 100644 index 0000000000..ccb9167df9 --- /dev/null +++ b/tests/xfs/786.out @@ -0,0 +1,2 @@ +QA output created by 786 +Silence is golden diff --git a/tests/xfs/787 b/tests/xfs/787 new file mode 100755 index 0000000000..91eaf5a7af --- /dev/null +++ b/tests/xfs/787 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 787 +# +# Race fsstress and free space by length btree scrub for a while to see if we +# crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub cntbt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/787.out b/tests/xfs/787.out new file mode 100644 index 0000000000..fa7f038120 --- /dev/null +++ b/tests/xfs/787.out @@ -0,0 +1,2 @@ +QA output created by 787 +Silence is golden diff --git a/tests/xfs/788 b/tests/xfs/788 new file mode 100755 index 0000000000..f1369e5309 --- /dev/null +++ b/tests/xfs/788 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 788 +# +# Race fsstress and inode btree scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -x 'dir' -s "scrub inobt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/788.out b/tests/xfs/788.out new file mode 100644 index 0000000000..5ddd661113 --- /dev/null +++ b/tests/xfs/788.out @@ -0,0 +1,2 @@ +QA output created by 788 +Silence is golden diff --git a/tests/xfs/789 b/tests/xfs/789 new file mode 100755 index 0000000000..550ff2c690 --- /dev/null +++ b/tests/xfs/789 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 789 +# +# Race fsstress and free inode btree scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" finobt +_scratch_xfs_stress_scrub -x 'dir' -s "scrub finobt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/789.out b/tests/xfs/789.out new file mode 100644 index 0000000000..da88fc99cb --- /dev/null +++ b/tests/xfs/789.out @@ -0,0 +1,2 @@ +QA output created by 789 +Silence is golden diff --git a/tests/xfs/790 b/tests/xfs/790 new file mode 100755 index 0000000000..c4e5779ef7 --- /dev/null +++ b/tests/xfs/790 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 790 +# +# Race fsstress and reverse mapping btree scrub for a while to see if we crash +# or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" rmapbt +_scratch_xfs_stress_scrub -s "scrub rmapbt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/790.out b/tests/xfs/790.out new file mode 100644 index 0000000000..7102c590f0 --- /dev/null +++ b/tests/xfs/790.out @@ -0,0 +1,2 @@ +QA output created by 790 +Silence is golden diff --git a/tests/xfs/791 b/tests/xfs/791 new file mode 100755 index 0000000000..6939d910c9 --- /dev/null +++ b/tests/xfs/791 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 791 +# +# Race fsstress and reference count btree scrub for a while to see if we crash +# or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" reflink +_scratch_xfs_stress_scrub -s "scrub refcountbt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/791.out b/tests/xfs/791.out new file mode 100644 index 0000000000..758905371d --- /dev/null +++ b/tests/xfs/791.out @@ -0,0 +1,2 @@ +QA output created by 791 +Silence is golden diff --git a/tests/xfs/798 b/tests/xfs/798 new file mode 100755 index 0000000000..c5bdfad50a --- /dev/null +++ b/tests/xfs/798 @@ -0,0 +1,44 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 798 +# +# Race fsstress and fscounter scrub on the realtime device for a while to see +# if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime + +# Force all files to be allocated on the realtime device +_xfs_force_bdev realtime $SCRATCH_MNT + +_scratch_xfs_stress_scrub -s 'scrub fscounters' + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/798.out b/tests/xfs/798.out new file mode 100644 index 0000000000..216d6e93f4 --- /dev/null +++ b/tests/xfs/798.out @@ -0,0 +1,2 @@ +QA output created by 798 +Silence is golden diff --git a/tests/xfs/800 b/tests/xfs/800 new file mode 100755 index 0000000000..cbcfb5f5a6 --- /dev/null +++ b/tests/xfs/800 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 800 +# +# Race fsstress and realtime bitmap scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime +_scratch_xfs_stress_scrub -s "scrub rtbitmap" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/800.out b/tests/xfs/800.out new file mode 100644 index 0000000000..bdfaa2cecd --- /dev/null +++ b/tests/xfs/800.out @@ -0,0 +1,2 @@ +QA output created by 800 +Silence is golden diff --git a/tests/xfs/801 b/tests/xfs/801 new file mode 100755 index 0000000000..a51fab523b --- /dev/null +++ b/tests/xfs/801 @@ -0,0 +1,47 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 801 +# +# Race fsstress and realtime summary scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime + +# XXX the realtime summary scrubber isn't currently implemented upstream. +# Don't bother trying to test it on those kernels +$XFS_IO_PROG -c 'scrub rtsummary' -c 'scrub rtsummary' "$SCRATCH_MNT" 2>&1 | \ + grep -q 'Scan was not complete' && \ + _notrun "rtsummary scrub is incomplete" + +_scratch_xfs_stress_scrub -s "scrub rtsummary" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/801.out b/tests/xfs/801.out new file mode 100644 index 0000000000..39481b38e2 --- /dev/null +++ b/tests/xfs/801.out @@ -0,0 +1,2 @@ +QA output created by 801 +Silence is golden diff --git a/tests/xfs/802 b/tests/xfs/802 new file mode 100755 index 0000000000..1f3b83882e --- /dev/null +++ b/tests/xfs/802 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 802 +# +# Race fsstress and user quota scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" usrquota +_scratch_xfs_stress_scrub -s "scrub usrquota" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/802.out b/tests/xfs/802.out new file mode 100644 index 0000000000..a69c05391f --- /dev/null +++ b/tests/xfs/802.out @@ -0,0 +1,2 @@ +QA output created by 802 +Silence is golden diff --git a/tests/xfs/803 b/tests/xfs/803 new file mode 100755 index 0000000000..b2bb85672d --- /dev/null +++ b/tests/xfs/803 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 803 +# +# Race fsstress and group quota scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" grpquota +_scratch_xfs_stress_scrub -s "scrub grpquota" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/803.out b/tests/xfs/803.out new file mode 100644 index 0000000000..38ba741d0f --- /dev/null +++ b/tests/xfs/803.out @@ -0,0 +1,2 @@ +QA output created by 803 +Silence is golden diff --git a/tests/xfs/804 b/tests/xfs/804 new file mode 100755 index 0000000000..129724eb11 --- /dev/null +++ b/tests/xfs/804 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 804 +# +# Race fsstress and project quota scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" prjquota +_scratch_xfs_stress_scrub -s "scrub prjquota" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/804.out b/tests/xfs/804.out new file mode 100644 index 0000000000..5e0cb437e7 --- /dev/null +++ b/tests/xfs/804.out @@ -0,0 +1,2 @@ +QA output created by 804 +Silence is golden diff --git a/tests/xfs/805 b/tests/xfs/805 new file mode 100755 index 0000000000..aca9b9cdf4 --- /dev/null +++ b/tests/xfs/805 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 805 +# +# Race fsstress and summary counters scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub fscounters" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/805.out b/tests/xfs/805.out new file mode 100644 index 0000000000..ac324c5874 --- /dev/null +++ b/tests/xfs/805.out @@ -0,0 +1,2 @@ +QA output created by 805 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH v24.1 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata 2023-02-07 17:02 ` [PATCH v24.1 " Darrick J. Wong @ 2023-02-07 18:45 ` Zorro Lang 0 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-07 18:45 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Tue, Feb 07, 2023 at 09:02:12AM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > For each XFS_SCRUB_TYPE_* that looks at AG or filesystem metadata, > create a test that runs that scrubber in the foreground and fsstress in > the background. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- > v24.1: rebase against newer upstream to get rid of merge conflicts > --- This version is good to me. Reviewed-by: Zorro Lang <zlang@redhat.com> > common/quota | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > tests/xfs/782 | 37 +++++++++++++++++++++++++++++++ > tests/xfs/782.out | 2 ++ > tests/xfs/783 | 37 +++++++++++++++++++++++++++++++ > tests/xfs/783.out | 2 ++ > tests/xfs/784 | 37 +++++++++++++++++++++++++++++++ > tests/xfs/784.out | 2 ++ > tests/xfs/785 | 37 +++++++++++++++++++++++++++++++ > tests/xfs/785.out | 2 ++ > tests/xfs/786 | 38 +++++++++++++++++++++++++++++++ > tests/xfs/786.out | 2 ++ > tests/xfs/787 | 38 +++++++++++++++++++++++++++++++ > tests/xfs/787.out | 2 ++ > tests/xfs/788 | 38 +++++++++++++++++++++++++++++++ > tests/xfs/788.out | 2 ++ > tests/xfs/789 | 39 ++++++++++++++++++++++++++++++++ > tests/xfs/789.out | 2 ++ > tests/xfs/790 | 39 ++++++++++++++++++++++++++++++++ > tests/xfs/790.out | 2 ++ > tests/xfs/791 | 40 +++++++++++++++++++++++++++++++++ > tests/xfs/791.out | 2 ++ > tests/xfs/798 | 44 ++++++++++++++++++++++++++++++++++++ > tests/xfs/798.out | 2 ++ > tests/xfs/800 | 40 +++++++++++++++++++++++++++++++++ > tests/xfs/800.out | 2 ++ > tests/xfs/801 | 47 +++++++++++++++++++++++++++++++++++++++ > tests/xfs/801.out | 2 ++ > tests/xfs/802 | 40 +++++++++++++++++++++++++++++++++ > tests/xfs/802.out | 2 ++ > tests/xfs/803 | 40 +++++++++++++++++++++++++++++++++ > tests/xfs/803.out | 2 ++ > tests/xfs/804 | 40 +++++++++++++++++++++++++++++++++ > tests/xfs/804.out | 2 ++ > tests/xfs/805 | 38 +++++++++++++++++++++++++++++++ > tests/xfs/805.out | 2 ++ > 35 files changed, 767 insertions(+) > create mode 100755 tests/xfs/782 > create mode 100644 tests/xfs/782.out > create mode 100755 tests/xfs/783 > create mode 100644 tests/xfs/783.out > create mode 100755 tests/xfs/784 > create mode 100644 tests/xfs/784.out > create mode 100755 tests/xfs/785 > create mode 100644 tests/xfs/785.out > create mode 100755 tests/xfs/786 > create mode 100644 tests/xfs/786.out > create mode 100755 tests/xfs/787 > create mode 100644 tests/xfs/787.out > create mode 100755 tests/xfs/788 > create mode 100644 tests/xfs/788.out > create mode 100755 tests/xfs/789 > create mode 100644 tests/xfs/789.out > create mode 100755 tests/xfs/790 > create mode 100644 tests/xfs/790.out > create mode 100755 tests/xfs/791 > create mode 100644 tests/xfs/791.out > create mode 100755 tests/xfs/798 > create mode 100644 tests/xfs/798.out > create mode 100755 tests/xfs/800 > create mode 100644 tests/xfs/800.out > create mode 100755 tests/xfs/801 > create mode 100644 tests/xfs/801.out > create mode 100755 tests/xfs/802 > create mode 100644 tests/xfs/802.out > create mode 100755 tests/xfs/803 > create mode 100644 tests/xfs/803.out > create mode 100755 tests/xfs/804 > create mode 100644 tests/xfs/804.out > create mode 100755 tests/xfs/805 > create mode 100644 tests/xfs/805.out > > diff --git a/common/quota b/common/quota > index 24251d092a..96b8d04424 100644 > --- a/common/quota > +++ b/common/quota > @@ -53,6 +53,70 @@ _require_xfs_quota() > [ -n "$XFS_QUOTA_PROG" ] || _notrun "XFS quota user tools not installed" > } > > +# Check that a mounted fs has a particular type of quota accounting turned on. > +# > +# The first argument must be the data device of a mounted fs. It must not be > +# the actual mountpath. > +# > +# The second argument is the quota type ('usrquota', 'grpquota', 'prjquota', > +# 'any', or 'all'). > +_xfs_quota_acct_enabled() > +{ > + local dev="$1" > + local qtype="$2" > + local f_args=() > + local any= > + > + case "$qtype" in > + "usrquota"|"uquota") f_args=("-U");; > + "grpquota"|"gquota") f_args=("-G");; > + "prjquota"|"pquota") f_args=("-P");; > + "all") f_args=("-U" "-G" "-P");; > + "any") f_args=("-U" "-G" "-P"); any=1;; > + *) echo "$qtype: Unknown quota type."; return 1;; > + esac > + > + if [ "$any" = "1" ]; then > + for arg in "$f_args"; do > + $here/src/feature "$arg" "$dev" && return 0 > + done > + return 1 > + fi > + > + $here/src/feature "${f_args[@]}" "$dev" > +} > + > +# Require that a mounted fs has a particular type of quota turned on. This > +# takes the same arguments as _xfs_quota_acct_enabled. If the third argument is > +# '-u' (or is empty and dev is $SCRATCH_DEV) the fs will be unmounted on > +# failure. > +_require_xfs_quota_acct_enabled() > +{ > + local dev="$1" > + local qtype="$2" > + local umount="$3" > + local fsname="$dev" > + > + _xfs_quota_acct_enabled "$dev" "$qtype" "$qmode" && return 0 > + > + if [ -z "$umount" ] && [ "$dev" = "$SCRATCH_DEV" ]; then > + umount="-u" > + fi > + test "$umount" = "-u" && umount "$dev" &>/dev/null > + > + case "$dev" in > + "$TEST_DEV") fsname="test";; > + "$SCRATCH_DEV") fsname="scratch";; > + esac > + > + case "$qtype" in > + "any") qtype="any quotas";; > + "all") qtype="all quotas";; > + esac > + > + _notrun "$qtype: accounting not enabled on $fsname filesystem." > +} > + > # > # checks that xfs_quota can operate on foreign (non-xfs) filesystems > # Skips check on xfs filesystems, old xfs_quota is fine there. > diff --git a/tests/xfs/782 b/tests/xfs/782 > new file mode 100755 > index 0000000000..4801eda4bd > --- /dev/null > +++ b/tests/xfs/782 > @@ -0,0 +1,37 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 782 > +# > +# Race fsstress and superblock scrub for a while to see if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub sb %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/782.out b/tests/xfs/782.out > new file mode 100644 > index 0000000000..6e378f0e53 > --- /dev/null > +++ b/tests/xfs/782.out > @@ -0,0 +1,2 @@ > +QA output created by 782 > +Silence is golden > diff --git a/tests/xfs/783 b/tests/xfs/783 > new file mode 100755 > index 0000000000..379a9369e5 > --- /dev/null > +++ b/tests/xfs/783 > @@ -0,0 +1,37 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 783 > +# > +# Race fsstress and AGF scrub for a while to see if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub agf %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/783.out b/tests/xfs/783.out > new file mode 100644 > index 0000000000..2522395956 > --- /dev/null > +++ b/tests/xfs/783.out > @@ -0,0 +1,2 @@ > +QA output created by 783 > +Silence is golden > diff --git a/tests/xfs/784 b/tests/xfs/784 > new file mode 100755 > index 0000000000..2b89361c36 > --- /dev/null > +++ b/tests/xfs/784 > @@ -0,0 +1,37 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 784 > +# > +# Race fsstress and AGFL scrub for a while to see if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub agfl %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/784.out b/tests/xfs/784.out > new file mode 100644 > index 0000000000..48d9b24dd0 > --- /dev/null > +++ b/tests/xfs/784.out > @@ -0,0 +1,2 @@ > +QA output created by 784 > +Silence is golden > diff --git a/tests/xfs/785 b/tests/xfs/785 > new file mode 100755 > index 0000000000..34a13b058d > --- /dev/null > +++ b/tests/xfs/785 > @@ -0,0 +1,37 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 785 > +# > +# Race fsstress and AGI scrub for a while to see if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub agi %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/785.out b/tests/xfs/785.out > new file mode 100644 > index 0000000000..6ecb0c61b3 > --- /dev/null > +++ b/tests/xfs/785.out > @@ -0,0 +1,2 @@ > +QA output created by 785 > +Silence is golden > diff --git a/tests/xfs/786 b/tests/xfs/786 > new file mode 100755 > index 0000000000..157200ea8c > --- /dev/null > +++ b/tests/xfs/786 > @@ -0,0 +1,38 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 786 > +# > +# Race fsstress and freespace by block btree scrub for a while to see if we > +# crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub bnobt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/786.out b/tests/xfs/786.out > new file mode 100644 > index 0000000000..ccb9167df9 > --- /dev/null > +++ b/tests/xfs/786.out > @@ -0,0 +1,2 @@ > +QA output created by 786 > +Silence is golden > diff --git a/tests/xfs/787 b/tests/xfs/787 > new file mode 100755 > index 0000000000..91eaf5a7af > --- /dev/null > +++ b/tests/xfs/787 > @@ -0,0 +1,38 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 787 > +# > +# Race fsstress and free space by length btree scrub for a while to see if we > +# crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub cntbt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/787.out b/tests/xfs/787.out > new file mode 100644 > index 0000000000..fa7f038120 > --- /dev/null > +++ b/tests/xfs/787.out > @@ -0,0 +1,2 @@ > +QA output created by 787 > +Silence is golden > diff --git a/tests/xfs/788 b/tests/xfs/788 > new file mode 100755 > index 0000000000..f1369e5309 > --- /dev/null > +++ b/tests/xfs/788 > @@ -0,0 +1,38 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 788 > +# > +# Race fsstress and inode btree scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -x 'dir' -s "scrub inobt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/788.out b/tests/xfs/788.out > new file mode 100644 > index 0000000000..5ddd661113 > --- /dev/null > +++ b/tests/xfs/788.out > @@ -0,0 +1,2 @@ > +QA output created by 788 > +Silence is golden > diff --git a/tests/xfs/789 b/tests/xfs/789 > new file mode 100755 > index 0000000000..550ff2c690 > --- /dev/null > +++ b/tests/xfs/789 > @@ -0,0 +1,39 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 789 > +# > +# Race fsstress and free inode btree scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" finobt > +_scratch_xfs_stress_scrub -x 'dir' -s "scrub finobt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/789.out b/tests/xfs/789.out > new file mode 100644 > index 0000000000..da88fc99cb > --- /dev/null > +++ b/tests/xfs/789.out > @@ -0,0 +1,2 @@ > +QA output created by 789 > +Silence is golden > diff --git a/tests/xfs/790 b/tests/xfs/790 > new file mode 100755 > index 0000000000..c4e5779ef7 > --- /dev/null > +++ b/tests/xfs/790 > @@ -0,0 +1,39 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 790 > +# > +# Race fsstress and reverse mapping btree scrub for a while to see if we crash > +# or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" rmapbt > +_scratch_xfs_stress_scrub -s "scrub rmapbt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/790.out b/tests/xfs/790.out > new file mode 100644 > index 0000000000..7102c590f0 > --- /dev/null > +++ b/tests/xfs/790.out > @@ -0,0 +1,2 @@ > +QA output created by 790 > +Silence is golden > diff --git a/tests/xfs/791 b/tests/xfs/791 > new file mode 100755 > index 0000000000..6939d910c9 > --- /dev/null > +++ b/tests/xfs/791 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 791 > +# > +# Race fsstress and reference count btree scrub for a while to see if we crash > +# or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/reflink > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" reflink > +_scratch_xfs_stress_scrub -s "scrub refcountbt %agno%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/791.out b/tests/xfs/791.out > new file mode 100644 > index 0000000000..758905371d > --- /dev/null > +++ b/tests/xfs/791.out > @@ -0,0 +1,2 @@ > +QA output created by 791 > +Silence is golden > diff --git a/tests/xfs/798 b/tests/xfs/798 > new file mode 100755 > index 0000000000..c5bdfad50a > --- /dev/null > +++ b/tests/xfs/798 > @@ -0,0 +1,44 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# Copyright (c) 2022 Oracle. All Rights Reserved. > +# > +# FS QA Test No. 798 > +# > +# Race fsstress and fscounter scrub on the realtime device for a while to see > +# if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_realtime > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" realtime > + > +# Force all files to be allocated on the realtime device > +_xfs_force_bdev realtime $SCRATCH_MNT > + > +_scratch_xfs_stress_scrub -s 'scrub fscounters' > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/798.out b/tests/xfs/798.out > new file mode 100644 > index 0000000000..216d6e93f4 > --- /dev/null > +++ b/tests/xfs/798.out > @@ -0,0 +1,2 @@ > +QA output created by 798 > +Silence is golden > diff --git a/tests/xfs/800 b/tests/xfs/800 > new file mode 100755 > index 0000000000..cbcfb5f5a6 > --- /dev/null > +++ b/tests/xfs/800 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 800 > +# > +# Race fsstress and realtime bitmap scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_realtime > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" realtime > +_scratch_xfs_stress_scrub -s "scrub rtbitmap" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/800.out b/tests/xfs/800.out > new file mode 100644 > index 0000000000..bdfaa2cecd > --- /dev/null > +++ b/tests/xfs/800.out > @@ -0,0 +1,2 @@ > +QA output created by 800 > +Silence is golden > diff --git a/tests/xfs/801 b/tests/xfs/801 > new file mode 100755 > index 0000000000..a51fab523b > --- /dev/null > +++ b/tests/xfs/801 > @@ -0,0 +1,47 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 801 > +# > +# Race fsstress and realtime summary scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_realtime > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_has_feature "$SCRATCH_MNT" realtime > + > +# XXX the realtime summary scrubber isn't currently implemented upstream. > +# Don't bother trying to test it on those kernels > +$XFS_IO_PROG -c 'scrub rtsummary' -c 'scrub rtsummary' "$SCRATCH_MNT" 2>&1 | \ > + grep -q 'Scan was not complete' && \ > + _notrun "rtsummary scrub is incomplete" > + > +_scratch_xfs_stress_scrub -s "scrub rtsummary" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/801.out b/tests/xfs/801.out > new file mode 100644 > index 0000000000..39481b38e2 > --- /dev/null > +++ b/tests/xfs/801.out > @@ -0,0 +1,2 @@ > +QA output created by 801 > +Silence is golden > diff --git a/tests/xfs/802 b/tests/xfs/802 > new file mode 100755 > index 0000000000..1f3b83882e > --- /dev/null > +++ b/tests/xfs/802 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 802 > +# > +# Race fsstress and user quota scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" usrquota > +_scratch_xfs_stress_scrub -s "scrub usrquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/802.out b/tests/xfs/802.out > new file mode 100644 > index 0000000000..a69c05391f > --- /dev/null > +++ b/tests/xfs/802.out > @@ -0,0 +1,2 @@ > +QA output created by 802 > +Silence is golden > diff --git a/tests/xfs/803 b/tests/xfs/803 > new file mode 100755 > index 0000000000..b2bb85672d > --- /dev/null > +++ b/tests/xfs/803 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 803 > +# > +# Race fsstress and group quota scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" grpquota > +_scratch_xfs_stress_scrub -s "scrub grpquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/803.out b/tests/xfs/803.out > new file mode 100644 > index 0000000000..38ba741d0f > --- /dev/null > +++ b/tests/xfs/803.out > @@ -0,0 +1,2 @@ > +QA output created by 803 > +Silence is golden > diff --git a/tests/xfs/804 b/tests/xfs/804 > new file mode 100755 > index 0000000000..129724eb11 > --- /dev/null > +++ b/tests/xfs/804 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 804 > +# > +# Race fsstress and project quota scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" prjquota > +_scratch_xfs_stress_scrub -s "scrub prjquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/804.out b/tests/xfs/804.out > new file mode 100644 > index 0000000000..5e0cb437e7 > --- /dev/null > +++ b/tests/xfs/804.out > @@ -0,0 +1,2 @@ > +QA output created by 804 > +Silence is golden > diff --git a/tests/xfs/805 b/tests/xfs/805 > new file mode 100755 > index 0000000000..aca9b9cdf4 > --- /dev/null > +++ b/tests/xfs/805 > @@ -0,0 +1,38 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 805 > +# > +# Race fsstress and summary counters scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -s "scrub fscounters" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/805.out b/tests/xfs/805.out > new file mode 100644 > index 0000000000..ac324c5874 > --- /dev/null > +++ b/tests/xfs/805.out > @@ -0,0 +1,2 @@ > +QA output created by 805 > +Silence is golden > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 5/5] xfs: race fsstress with online scrubbers for file metadata 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 3/5] fuzzy: add a custom xfs find utility for scrub stress tests Darrick J. Wong ` (2 subsequent siblings) 4 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> For each XFS_SCRUB_TYPE_* that looks at file metadata, create a test that runs that scrubber in the foreground and fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++--- tests/xfs/792 | 38 +++++++++++++++++++++++ tests/xfs/792.out | 2 + tests/xfs/793 | 37 ++++++++++++++++++++++ tests/xfs/793.out | 2 + tests/xfs/794 | 39 +++++++++++++++++++++++ tests/xfs/794.out | 2 + tests/xfs/795 | 39 +++++++++++++++++++++++ tests/xfs/795.out | 2 + tests/xfs/796 | 37 ++++++++++++++++++++++ tests/xfs/796.out | 2 + tests/xfs/797 | 40 ++++++++++++++++++++++++ tests/xfs/797.out | 2 + tests/xfs/799 | 38 +++++++++++++++++++++++ tests/xfs/799.out | 2 + tests/xfs/826 | 38 +++++++++++++++++++++++ tests/xfs/826.out | 2 + tests/xfs/827 | 39 +++++++++++++++++++++++ tests/xfs/827.out | 2 + 19 files changed, 447 insertions(+), 5 deletions(-) create mode 100755 tests/xfs/792 create mode 100644 tests/xfs/792.out create mode 100755 tests/xfs/793 create mode 100644 tests/xfs/793.out create mode 100755 tests/xfs/794 create mode 100644 tests/xfs/794.out create mode 100755 tests/xfs/795 create mode 100644 tests/xfs/795.out create mode 100755 tests/xfs/796 create mode 100644 tests/xfs/796.out create mode 100755 tests/xfs/797 create mode 100644 tests/xfs/797.out create mode 100755 tests/xfs/799 create mode 100644 tests/xfs/799.out create mode 100755 tests/xfs/826 create mode 100644 tests/xfs/826.out create mode 100755 tests/xfs/827 create mode 100644 tests/xfs/827.out diff --git a/common/fuzzy b/common/fuzzy index c4a5bc9261..f7f660bc31 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -330,12 +330,20 @@ __stress_freeze_filter_output() { # Filter scrub output so that we don't tarnish the golden output if the fs is # too busy to scrub. Note: Tests should _notrun if the scrub type is not -# supported. +# supported. Callers can provide extra strings to filter out as function +# arguments. __stress_scrub_filter_output() { + local extra_args=() + + for arg in "$@"; do + extra_args+=(-e "/${arg}/d") + done + _filter_scratch | \ sed -e '/Device or resource busy/d' \ -e '/Optimization possible/d' \ - -e '/No space left on device/d' + -e '/No space left on device/d' \ + "${extra_args[@]}" } # Decide if the scratch filesystem is still alive. @@ -401,13 +409,34 @@ __stress_one_scrub_loop() { fi done + local extra_filters=() + local target_cmd=(echo "$scrub_tgt") + case "$scrub_tgt" in + "%file%"|"%datafile%"|"%attrfile%") + extra_filters+=('No such file or directory' 'No such device or address') + target_cmd=(find "$SCRATCH_MNT" -print) + ;; + "%dir%") + extra_filters+=('No such file or directory' 'Not a directory') + target_cmd=(find "$SCRATCH_MNT" -type d -print) + ;; + "%regfile%"|"%cowfile%") + extra_filters+=('No such file or directory') + target_cmd=(find "$SCRATCH_MNT" -type f -print) + ;; + esac + while __stress_scrub_running "$scrub_startat" "$runningfile"; do sleep 1 done while __stress_scrub_running "$end" "$runningfile"; do - $XFS_IO_PROG -x "${xfs_io_args[@]}" "$scrub_tgt" 2>&1 | \ - __stress_scrub_filter_output + readarray -t fnames < <("${target_cmd[@]}" 2>/dev/null) + for fname in "${fnames[@]}"; do + $XFS_IO_PROG -x "${xfs_io_args[@]}" "$fname" 2>&1 | \ + __stress_scrub_filter_output "${extra_filters[@]}" + __stress_scrub_running "$end" "$runningfile" || break + done done } @@ -585,6 +614,22 @@ __stress_scrub_fsstress_loop() { "default") # No new arguments ;; + "symlink") + focus+=('-z') + + # Only create, read, and delete symbolic links + focus+=('-f' 'symlink=4') + focus+=('-f' 'readlink=10') + focus+=('-f' 'unlink=1') + ;; + "mknod") + focus+=('-z') + + # Only create and delete special files + focus+=('-f' 'mknod=4') + focus+=('-f' 'getdents=100') + focus+=('-f' 'unlink=1') + ;; *) echo "$stress_tgt: Unrecognized stress target, using defaults." ;; @@ -715,9 +760,31 @@ __stress_scrub_check_commands() { local scrub_tgt="$1" shift + local cooked_tgt="$scrub_tgt" + case "$scrub_tgt" in + "%file%"|"%dir%") + cooked_tgt="$SCRATCH_MNT" + ;; + "%regfile%"|"%datafile%") + cooked_tgt="$SCRATCH_MNT/testfile" + echo test > "$cooked_tgt" + ;; + "%attrfile%") + cooked_tgt="$SCRATCH_MNT/testfile" + $XFS_IO_PROG -f -c 'pwrite -S 0x58 0 64k' "$cooked_tgt" &>/dev/null + attr -s attrname "$cooked_tgt" < "$cooked_tgt" &>/dev/null + ;; + "%cowfile%") + cooked_tgt="$SCRATCH_MNT/testfile" + $XFS_IO_PROG -f -c 'pwrite -S 0x58 0 128k' "$cooked_tgt" &>/dev/null + _cp_reflink "$cooked_tgt" "$cooked_tgt.1" + $XFS_IO_PROG -f -c 'pwrite -S 0x58 0 1' "$cooked_tgt.1" &>/dev/null + ;; + esac + for arg in "$@"; do local cooked_arg="$(echo "$arg" | sed -e "s/%agno%/0/g")" - testio=`$XFS_IO_PROG -x -c "$cooked_arg" $scrub_tgt 2>&1` + testio=`$XFS_IO_PROG -x -c "$cooked_arg" "$cooked_tgt" 2>&1` echo $testio | grep -q "Unknown type" && \ _notrun "xfs_io scrub subcommand support is missing" echo $testio | grep -q "Inappropriate ioctl" && \ @@ -749,6 +816,16 @@ __stress_scrub_check_commands() { # -S Pass this option to xfs_scrub. If zero -S options are specified, # xfs_scrub will not be run. To select repair mode, pass '-k' or '-v'. # -t Run online scrub against this file; $SCRATCH_MNT is the default. +# Special values are as follows: +# +# %file% all files +# %regfile% regular files +# %dir% direct +# %datafile% regular files with data blocks +# %attrfile% regular files with xattr blocks +# %cowfile% regular files with shared blocks +# +# File selection races with fsstress, so the selection is best-effort. # -w Delay the start of the scrub/repair loop by this number of seconds. # Defaults to no delay unless XFS_SCRUB_STRESS_DELAY is set. This value # will be clamped to ten seconds before the end time. @@ -758,6 +835,8 @@ __stress_scrub_check_commands() { # 'xattr': Grow extended attributes in a small tree. # 'default': Run fsstress with default arguments. # 'writeonly': Only perform fs updates, no reads. +# 'symlink': Only create symbolic links. +# 'mknod': Only create special files. # # The default is 'default' unless XFS_SCRUB_STRESS_TARGET is set. # -X Run this program to exercise the filesystem. Currently supported diff --git a/tests/xfs/792 b/tests/xfs/792 new file mode 100755 index 0000000000..0806e87909 --- /dev/null +++ b/tests/xfs/792 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 792 +# +# Race fsstress and inode record scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub inode" -t "%file%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/792.out b/tests/xfs/792.out new file mode 100644 index 0000000000..c9b5ef3a7c --- /dev/null +++ b/tests/xfs/792.out @@ -0,0 +1,2 @@ +QA output created by 792 +Silence is golden diff --git a/tests/xfs/793 b/tests/xfs/793 new file mode 100755 index 0000000000..41be82d621 --- /dev/null +++ b/tests/xfs/793 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 793 +# +# Race fsstress and data fork scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub bmapbtd" -t "%datafile%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/793.out b/tests/xfs/793.out new file mode 100644 index 0000000000..e8a17d4ecb --- /dev/null +++ b/tests/xfs/793.out @@ -0,0 +1,2 @@ +QA output created by 793 +Silence is golden diff --git a/tests/xfs/794 b/tests/xfs/794 new file mode 100755 index 0000000000..8f4835dbc9 --- /dev/null +++ b/tests/xfs/794 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 794 +# +# Race fsstress and attr fork scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/attr + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_attrs +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -x 'xattr' -s "scrub bmapbta" -t "%attrfile%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/794.out b/tests/xfs/794.out new file mode 100644 index 0000000000..bc999c055c --- /dev/null +++ b/tests/xfs/794.out @@ -0,0 +1,2 @@ +QA output created by 794 +Silence is golden diff --git a/tests/xfs/795 b/tests/xfs/795 new file mode 100755 index 0000000000..ec065bafdd --- /dev/null +++ b/tests/xfs/795 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 795 +# +# Race fsstress and cow fork scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" reflink +_scratch_xfs_stress_scrub -s "scrub bmapbtc" -t "%cowfile%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/795.out b/tests/xfs/795.out new file mode 100644 index 0000000000..cb357003dd --- /dev/null +++ b/tests/xfs/795.out @@ -0,0 +1,2 @@ +QA output created by 795 +Silence is golden diff --git a/tests/xfs/796 b/tests/xfs/796 new file mode 100755 index 0000000000..d337701264 --- /dev/null +++ b/tests/xfs/796 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 796 +# +# Race fsstress and directory scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -x 'dir' -s "scrub directory" -t "%dir%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/796.out b/tests/xfs/796.out new file mode 100644 index 0000000000..374e3774a2 --- /dev/null +++ b/tests/xfs/796.out @@ -0,0 +1,2 @@ +QA output created by 796 +Silence is golden diff --git a/tests/xfs/797 b/tests/xfs/797 new file mode 100755 index 0000000000..c68b43be7a --- /dev/null +++ b/tests/xfs/797 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 797 +# +# Race fsstress and extended attributes scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/attr + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_attrs +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -x 'xattr' -s "scrub xattr" -t "%attrfile%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/797.out b/tests/xfs/797.out new file mode 100644 index 0000000000..6b64f4bf21 --- /dev/null +++ b/tests/xfs/797.out @@ -0,0 +1,2 @@ +QA output created by 797 +Silence is golden diff --git a/tests/xfs/799 b/tests/xfs/799 new file mode 100755 index 0000000000..84007ea9c0 --- /dev/null +++ b/tests/xfs/799 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 799 +# +# Race fsstress and parent pointers scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -s "scrub parent" -t "%dir%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/799.out b/tests/xfs/799.out new file mode 100644 index 0000000000..f3fd9fa2a0 --- /dev/null +++ b/tests/xfs/799.out @@ -0,0 +1,2 @@ +QA output created by 799 +Silence is golden diff --git a/tests/xfs/826 b/tests/xfs/826 new file mode 100755 index 0000000000..7660270571 --- /dev/null +++ b/tests/xfs/826 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 826 +# +# Race fsstress and symlink scrub for a while to see if we crash or livelock. +# We can't open symlink files directly for scrubbing, so we use xfs_scrub(8). +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +XFS_SCRUB_PHASE=3 _scratch_xfs_stress_scrub -x 'symlink' -S '-n' + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/826.out b/tests/xfs/826.out new file mode 100644 index 0000000000..93fae86b82 --- /dev/null +++ b/tests/xfs/826.out @@ -0,0 +1,2 @@ +QA output created by 826 +Silence is golden diff --git a/tests/xfs/827 b/tests/xfs/827 new file mode 100755 index 0000000000..55ec01d1e6 --- /dev/null +++ b/tests/xfs/827 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 827 +# +# Race fsstress and special file scrub for a while to see if we crash or +# livelock. We can't open special files directly for scrubbing, so we use +# xfs_scrub(8). +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +XFS_SCRUB_PHASE=3 _scratch_xfs_stress_scrub -x 'mknod' -S '-n' + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/827.out b/tests/xfs/827.out new file mode 100644 index 0000000000..65f29d949d --- /dev/null +++ b/tests/xfs/827.out @@ -0,0 +1,2 @@ +QA output created by 827 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 3/5] fuzzy: add a custom xfs find utility for scrub stress tests 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata Darrick J. Wong 2022-12-30 22:19 ` [PATCH 5/5] xfs: race fsstress with online scrubbers for file metadata Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-05 12:57 ` Zorro Lang 2023-02-07 17:01 ` [PATCH v24.1 " Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/5] xfs/357: switch fuzzing to agi 1 Darrick J. Wong 2022-12-30 22:19 ` [PATCH 4/5] fuzzy: allow xfs scrub stress tests to pick preconfigured fsstress configs Darrick J. Wong 4 siblings, 2 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create a new find(1) like utility that doesn't crash on directory tree changes (like find does due to bugs in its loop detector) and actually implements the custom xfs attribute predicates that we need for scrub stress tests. This program will be needed for a future patch where we add stress tests for scrub and repair of file metadata. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- configure.ac | 5 + include/builddefs.in | 4 + m4/package_libcdev.m4 | 47 ++++++++ m4/package_xfslibs.m4 | 16 +++ src/Makefile | 10 ++ src/xfsfind.c | 290 +++++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 372 insertions(+) create mode 100644 src/xfsfind.c diff --git a/configure.ac b/configure.ac index cbf8377988..e92bd6b26d 100644 --- a/configure.ac +++ b/configure.ac @@ -66,6 +66,11 @@ AC_PACKAGE_WANT_LINUX_FS_H AC_PACKAGE_WANT_LIBBTRFSUTIL AC_HAVE_COPY_FILE_RANGE +AC_HAVE_SEEK_DATA +AC_HAVE_BMV_OF_SHARED +AC_HAVE_NFTW +AC_HAVE_RLIMIT_NOFILE + AC_CHECK_FUNCS([renameat2]) AC_CHECK_FUNCS([reallocarray]) AC_CHECK_TYPES([struct mount_attr], [], [], [[#include <linux/mount.h>]]) diff --git a/include/builddefs.in b/include/builddefs.in index 6641209f81..dab10c968f 100644 --- a/include/builddefs.in +++ b/include/builddefs.in @@ -68,6 +68,10 @@ HAVE_FIEMAP = @have_fiemap@ HAVE_FALLOCATE = @have_fallocate@ HAVE_COPY_FILE_RANGE = @have_copy_file_range@ HAVE_LIBBTRFSUTIL = @have_libbtrfsutil@ +HAVE_SEEK_DATA = @have_seek_data@ +HAVE_NFTW = @have_nftw@ +HAVE_BMV_OF_SHARED = @have_bmv_of_shared@ +HAVE_RLIMIT_NOFILE = @have_rlimit_nofile@ GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 index 5c76c0f73e..e1b381c16f 100644 --- a/m4/package_libcdev.m4 +++ b/m4/package_libcdev.m4 @@ -110,3 +110,50 @@ AC_DEFUN([AC_HAVE_COPY_FILE_RANGE], AC_SUBST(have_copy_file_range) ]) +# Check if we have SEEK_DATA +AC_DEFUN([AC_HAVE_SEEK_DATA], + [ AC_MSG_CHECKING([for SEEK_DATA]) + AC_TRY_LINK([ +#define _GNU_SOURCE +#include <sys/types.h> +#include <unistd.h> + ], [ + lseek(-1, 0, SEEK_DATA); + ], have_seek_data=yes + AC_MSG_RESULT(yes), + AC_MSG_RESULT(no)) + AC_SUBST(have_seek_data) + ]) + +# Check if we have nftw +AC_DEFUN([AC_HAVE_NFTW], + [ AC_MSG_CHECKING([for nftw]) + AC_TRY_LINK([ +#define _GNU_SOURCE +#include <stddef.h> +#include <ftw.h> + ], [ + nftw("/", (int (*)(const char *, const struct stat *, int, struct FTW *))1, 0, 0); + ], have_nftw=yes + AC_MSG_RESULT(yes), + AC_MSG_RESULT(no)) + AC_SUBST(have_nftw) + ]) + +# Check if we have RLIMIT_NOFILE +AC_DEFUN([AC_HAVE_RLIMIT_NOFILE], + [ AC_MSG_CHECKING([for RLIMIT_NOFILE]) + AC_TRY_LINK([ +#define _GNU_SOURCE +#include <sys/time.h> +#include <sys/resource.h> + ], [ + struct rlimit rlimit; + + rlimit.rlim_cur = 0; + getrlimit(RLIMIT_NOFILE, &rlimit); + ], have_rlimit_nofile=yes + AC_MSG_RESULT(yes), + AC_MSG_RESULT(no)) + AC_SUBST(have_rlimit_nofile) + ]) diff --git a/m4/package_xfslibs.m4 b/m4/package_xfslibs.m4 index 0746cd1dc5..479f30a29b 100644 --- a/m4/package_xfslibs.m4 +++ b/m4/package_xfslibs.m4 @@ -104,3 +104,19 @@ AC_DEFUN([AC_PACKAGE_NEED_XFSCTL_MACRO], exit 1 ]) ]) + +# Check if we have BMV_OF_SHARED from the GETBMAPX ioctl +AC_DEFUN([AC_HAVE_BMV_OF_SHARED], + [ AC_MSG_CHECKING([for BMV_OF_SHARED]) + AC_TRY_LINK([ +#define _GNU_SOURCE +#include <xfs/xfs.h> + ], [ + struct getbmapx obj; + ioctl(-1, XFS_IOC_GETBMAPX, &obj); + obj.bmv_oflags |= BMV_OF_SHARED; + ], have_bmv_of_shared=yes + AC_MSG_RESULT(yes), + AC_MSG_RESULT(no)) + AC_SUBST(have_bmv_of_shared) + ]) diff --git a/src/Makefile b/src/Makefile index afdf6b30c5..7807ca89a5 100644 --- a/src/Makefile +++ b/src/Makefile @@ -83,6 +83,16 @@ ifeq ($(HAVE_LIBCAP), true) LLDLIBS += -lcap endif +ifeq ($(HAVE_SEEK_DATA), yes) + ifeq ($(HAVE_NFTW), yes) + ifeq ($(HAVE_BMV_OF_SHARED), yes) + ifeq ($(HAVE_RLIMIT_NOFILE), yes) + TARGETS += xfsfind + endif + endif + endif +endif + CFILES = $(TARGETS:=.c) LDIRT = $(TARGETS) fssum diff --git a/src/xfsfind.c b/src/xfsfind.c new file mode 100644 index 0000000000..6b0a93e793 --- /dev/null +++ b/src/xfsfind.c @@ -0,0 +1,290 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * find(1) but with special predicates for finding XFS attributes. + * Copyright (C) 2022 Oracle. + */ +#include <sys/time.h> +#include <sys/resource.h> +#include <sys/types.h> +#include <stdio.h> +#include <unistd.h> +#include <ftw.h> +#include <linux/fs.h> +#include <xfs/xfs.h> + +#include "global.h" + +static int want_anyfile; +static int want_datafile; +static int want_attrfile; +static int want_dir; +static int want_regfile; +static int want_sharedfile; +static int report_errors = 1; + +static int +check_datafile( + const char *path, + int fd) +{ + off_t off; + + off = lseek(fd, 0, SEEK_DATA); + if (off >= 0) + return 1; + + if (errno == ENXIO) + return 0; + + if (report_errors) + perror(path); + + return -1; +} + +static int +check_attrfile( + const char *path, + int fd) +{ + struct fsxattr fsx; + int ret; + + ret = ioctl(fd, XFS_IOC_FSGETXATTR, &fsx); + if (ret) { + if (report_errors) + perror(path); + return -1; + } + + if (want_attrfile && (fsx.fsx_xflags & XFS_XFLAG_HASATTR)) + return 1; + return 0; +} + +#define BMAP_NR 33 +static struct getbmapx bmaps[BMAP_NR]; + +static int +check_sharedfile( + const char *path, + int fd) +{ + struct getbmapx *key = &bmaps[0]; + unsigned int i; + int ret; + + memset(key, 0, sizeof(struct getbmapx)); + key->bmv_length = ULLONG_MAX; + /* no holes and don't flush dirty pages */ + key->bmv_iflags = BMV_IF_DELALLOC | BMV_IF_NO_HOLES; + key->bmv_count = BMAP_NR; + + while ((ret = ioctl(fd, XFS_IOC_GETBMAPX, bmaps)) == 0) { + struct getbmapx *p = &bmaps[1]; + xfs_off_t new_off; + + for (i = 0; i < key->bmv_entries; i++, p++) { + if (p->bmv_oflags & BMV_OF_SHARED) + return 1; + } + + if (key->bmv_entries == 0) + break; + p = key + key->bmv_entries; + if (p->bmv_oflags & BMV_OF_LAST) + return 0; + + new_off = p->bmv_offset + p->bmv_length; + key->bmv_length -= new_off - key->bmv_offset; + key->bmv_offset = new_off; + } + if (ret < 0) { + if (report_errors) + perror(path); + return -1; + } + + return 0; +} + +static void +print_help( + const char *name) +{ + printf("Usage: %s [OPTIONS] path\n", name); + printf("\n"); + printf("Print all file paths matching any of the given predicates.\n"); + printf("\n"); + printf("-a Match files with xattrs.\n"); + printf("-b Match files with data blocks.\n"); + printf("-d Match directories.\n"); + printf("-q Ignore errors while walking directory tree.\n"); + printf("-r Match regular files.\n"); + printf("-s Match files with shared blocks.\n"); + printf("\n"); + printf("If no matching options are given, match all files found.\n"); +} + +static int +visit( + const char *path, + const struct stat *sb, + int typeflag, + struct FTW *ftwbuf) +{ + int printme = 1; + int fd = -1; + int retval = FTW_CONTINUE; + + if (want_anyfile) + goto out; + if (want_regfile && typeflag == FTW_F) + goto out; + if (want_dir && typeflag == FTW_D) + goto out; + + /* + * We can only open directories and files; screen out everything else. + * Note that nftw lies and reports FTW_F for device files, so check the + * statbuf mode too. + */ + if (typeflag != FTW_F && typeflag != FTW_D) { + printme = 0; + goto out; + } + + if (!S_ISREG(sb->st_mode) && !S_ISDIR(sb->st_mode)) { + printme = 0; + goto out; + } + + fd = open(path, O_RDONLY); + if (fd < 0) { + if (report_errors) { + perror(path); + return FTW_STOP; + } + + return FTW_CONTINUE; + } + + if (want_datafile && typeflag == FTW_F) { + int ret = check_datafile(path, fd); + if (ret < 0 && report_errors) { + printme = 0; + retval = FTW_STOP; + goto out_fd; + } + + if (ret == 1) + goto out_fd; + } + + if (want_attrfile) { + int ret = check_attrfile(path, fd); + if (ret < 0 && report_errors) { + printme = 0; + retval = FTW_STOP; + goto out_fd; + } + + if (ret == 1) + goto out_fd; + } + + if (want_sharedfile) { + int ret = check_sharedfile(path, fd); + if (ret < 0 && report_errors) { + printme = 0; + retval = FTW_STOP; + goto out_fd; + } + + if (ret == 1) + goto out_fd; + } + + printme = 0; +out_fd: + close(fd); +out: + if (printme) + printf("%s\n", path); + return retval; +} + +static void +handle_sigabrt( + int signal, + siginfo_t *info, + void *ucontext) +{ + fprintf(stderr, "Signal %u, exiting.\n", signal); + exit(2); +} + +int +main( + int argc, + char *argv[]) +{ + struct rlimit rlimit; + struct sigaction abrt = { + .sa_sigaction = handle_sigabrt, + .sa_flags = SA_SIGINFO, + }; + int c; + int ret; + + while ((c = getopt(argc, argv, "abdqrs")) >= 0) { + switch (c) { + case 'a': want_attrfile = 1; break; + case 'b': want_datafile = 1; break; + case 'd': want_dir = 1; break; + case 'q': report_errors = 0; break; + case 'r': want_regfile = 1; break; + case 's': want_sharedfile = 1; break; + default: + print_help(argv[0]); + return 1; + } + } + + ret = getrlimit(RLIMIT_NOFILE, &rlimit); + if (ret) { + perror("RLIMIT_NOFILE"); + return 1; + } + + if (!want_attrfile && !want_datafile && !want_dir && !want_regfile && + !want_sharedfile) + want_anyfile = 1; + + /* + * nftw is known to abort() if a directory it is walking disappears out + * from under it. Handle this with grace if the caller wants us to run + * quietly. + */ + if (!report_errors) { + ret = sigaction(SIGABRT, &abrt, NULL); + if (ret) { + perror("SIGABRT handler"); + return 1; + } + } + + for (c = optind; c < argc; c++) { + ret = nftw(argv[c], visit, rlimit.rlim_cur - 5, + FTW_ACTIONRETVAL | FTW_CHDIR | FTW_MOUNT | + FTW_PHYS); + if (ret && report_errors) { + perror(argv[c]); + break; + } + } + + if (ret) + return 1; + return 0; +} ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 3/5] fuzzy: add a custom xfs find utility for scrub stress tests 2022-12-30 22:19 ` [PATCH 3/5] fuzzy: add a custom xfs find utility for scrub stress tests Darrick J. Wong @ 2023-02-05 12:57 ` Zorro Lang 2023-02-07 16:57 ` Darrick J. Wong 2023-02-07 17:01 ` [PATCH v24.1 " Darrick J. Wong 1 sibling, 1 reply; 107+ messages in thread From: Zorro Lang @ 2023-02-05 12:57 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:06PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Create a new find(1) like utility that doesn't crash on directory tree > changes (like find does due to bugs in its loop detector) and actually > implements the custom xfs attribute predicates that we need for scrub > stress tests. This program will be needed for a future patch where we > add stress tests for scrub and repair of file metadata. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- > configure.ac | 5 + > include/builddefs.in | 4 + > m4/package_libcdev.m4 | 47 ++++++++ > m4/package_xfslibs.m4 | 16 +++ > src/Makefile | 10 ++ > src/xfsfind.c | 290 +++++++++++++++++++++++++++++++++++++++++++++++++ > 6 files changed, 372 insertions(+) > create mode 100644 src/xfsfind.c > > > diff --git a/configure.ac b/configure.ac > index cbf8377988..e92bd6b26d 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -66,6 +66,11 @@ AC_PACKAGE_WANT_LINUX_FS_H > AC_PACKAGE_WANT_LIBBTRFSUTIL > > AC_HAVE_COPY_FILE_RANGE > +AC_HAVE_SEEK_DATA > +AC_HAVE_BMV_OF_SHARED > +AC_HAVE_NFTW > +AC_HAVE_RLIMIT_NOFILE > + > AC_CHECK_FUNCS([renameat2]) > AC_CHECK_FUNCS([reallocarray]) > AC_CHECK_TYPES([struct mount_attr], [], [], [[#include <linux/mount.h>]]) > diff --git a/include/builddefs.in b/include/builddefs.in > index 6641209f81..dab10c968f 100644 > --- a/include/builddefs.in > +++ b/include/builddefs.in > @@ -68,6 +68,10 @@ HAVE_FIEMAP = @have_fiemap@ > HAVE_FALLOCATE = @have_fallocate@ > HAVE_COPY_FILE_RANGE = @have_copy_file_range@ > HAVE_LIBBTRFSUTIL = @have_libbtrfsutil@ > +HAVE_SEEK_DATA = @have_seek_data@ > +HAVE_NFTW = @have_nftw@ > +HAVE_BMV_OF_SHARED = @have_bmv_of_shared@ > +HAVE_RLIMIT_NOFILE = @have_rlimit_nofile@ > > GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall > > diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 > index 5c76c0f73e..e1b381c16f 100644 > --- a/m4/package_libcdev.m4 > +++ b/m4/package_libcdev.m4 > @@ -110,3 +110,50 @@ AC_DEFUN([AC_HAVE_COPY_FILE_RANGE], > AC_SUBST(have_copy_file_range) > ]) > > +# Check if we have SEEK_DATA > +AC_DEFUN([AC_HAVE_SEEK_DATA], > + [ AC_MSG_CHECKING([for SEEK_DATA]) > + AC_TRY_LINK([ The AC_TRY_LINK is obsolete by autoconf, refer to: https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Obsolete-Macros.html So as the suggestion of above link, we'd better to replace: Macro: AC_TRY_LINK (includes, function-body, [action-if-true], [action-if-false]) with: AC_LINK_IFELSE( [AC_LANG_PROGRAM([[includes]], [[function-body]])], [action-if-true], [action-if-false]) For example (hope it's right:) diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 index e1b381c1..7f1767a4 100644 --- a/m4/package_libcdev.m4 +++ b/m4/package_libcdev.m4 @@ -113,13 +113,13 @@ AC_DEFUN([AC_HAVE_COPY_FILE_RANGE], # Check if we have SEEK_DATA AC_DEFUN([AC_HAVE_SEEK_DATA], [ AC_MSG_CHECKING([for SEEK_DATA]) - AC_TRY_LINK([ + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ #define _GNU_SOURCE #include <sys/types.h> #include <unistd.h> - ], [ + ]], [[ lseek(-1, 0, SEEK_DATA); - ], have_seek_data=yes + ]])], have_seek_data=yes AC_MSG_RESULT(yes), AC_MSG_RESULT(no)) AC_SUBST(have_seek_data) > +#define _GNU_SOURCE > +#include <sys/types.h> > +#include <unistd.h> > + ], [ > + lseek(-1, 0, SEEK_DATA); > + ], have_seek_data=yes > + AC_MSG_RESULT(yes), > + AC_MSG_RESULT(no)) > + AC_SUBST(have_seek_data) > + ]) > + > +# Check if we have nftw > +AC_DEFUN([AC_HAVE_NFTW], > + [ AC_MSG_CHECKING([for nftw]) > + AC_TRY_LINK([ Same as above > +#define _GNU_SOURCE > +#include <stddef.h> > +#include <ftw.h> > + ], [ > + nftw("/", (int (*)(const char *, const struct stat *, int, struct FTW *))1, 0, 0); > + ], have_nftw=yes > + AC_MSG_RESULT(yes), > + AC_MSG_RESULT(no)) > + AC_SUBST(have_nftw) > + ]) > + > +# Check if we have RLIMIT_NOFILE > +AC_DEFUN([AC_HAVE_RLIMIT_NOFILE], > + [ AC_MSG_CHECKING([for RLIMIT_NOFILE]) > + AC_TRY_LINK([ Same as above > +#define _GNU_SOURCE > +#include <sys/time.h> > +#include <sys/resource.h> > + ], [ > + struct rlimit rlimit; > + > + rlimit.rlim_cur = 0; > + getrlimit(RLIMIT_NOFILE, &rlimit); > + ], have_rlimit_nofile=yes > + AC_MSG_RESULT(yes), > + AC_MSG_RESULT(no)) > + AC_SUBST(have_rlimit_nofile) > + ]) > diff --git a/m4/package_xfslibs.m4 b/m4/package_xfslibs.m4 > index 0746cd1dc5..479f30a29b 100644 > --- a/m4/package_xfslibs.m4 > +++ b/m4/package_xfslibs.m4 > @@ -104,3 +104,19 @@ AC_DEFUN([AC_PACKAGE_NEED_XFSCTL_MACRO], > exit 1 > ]) > ]) > + > +# Check if we have BMV_OF_SHARED from the GETBMAPX ioctl > +AC_DEFUN([AC_HAVE_BMV_OF_SHARED], > + [ AC_MSG_CHECKING([for BMV_OF_SHARED]) > + AC_TRY_LINK([ Same as above Thanks, Zorro > +#define _GNU_SOURCE > +#include <xfs/xfs.h> > + ], [ > + struct getbmapx obj; > + ioctl(-1, XFS_IOC_GETBMAPX, &obj); > + obj.bmv_oflags |= BMV_OF_SHARED; > + ], have_bmv_of_shared=yes > + AC_MSG_RESULT(yes), > + AC_MSG_RESULT(no)) > + AC_SUBST(have_bmv_of_shared) > + ]) > diff --git a/src/Makefile b/src/Makefile > index afdf6b30c5..7807ca89a5 100644 > --- a/src/Makefile > +++ b/src/Makefile > @@ -83,6 +83,16 @@ ifeq ($(HAVE_LIBCAP), true) > LLDLIBS += -lcap > endif > > +ifeq ($(HAVE_SEEK_DATA), yes) > + ifeq ($(HAVE_NFTW), yes) > + ifeq ($(HAVE_BMV_OF_SHARED), yes) > + ifeq ($(HAVE_RLIMIT_NOFILE), yes) > + TARGETS += xfsfind > + endif > + endif > + endif > +endif > + > CFILES = $(TARGETS:=.c) > LDIRT = $(TARGETS) fssum > > diff --git a/src/xfsfind.c b/src/xfsfind.c > new file mode 100644 > index 0000000000..6b0a93e793 > --- /dev/null > +++ b/src/xfsfind.c > @@ -0,0 +1,290 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * find(1) but with special predicates for finding XFS attributes. > + * Copyright (C) 2022 Oracle. > + */ > +#include <sys/time.h> > +#include <sys/resource.h> > +#include <sys/types.h> > +#include <stdio.h> > +#include <unistd.h> > +#include <ftw.h> > +#include <linux/fs.h> > +#include <xfs/xfs.h> > + > +#include "global.h" > + > +static int want_anyfile; > +static int want_datafile; > +static int want_attrfile; > +static int want_dir; > +static int want_regfile; > +static int want_sharedfile; > +static int report_errors = 1; > + > +static int > +check_datafile( > + const char *path, > + int fd) > +{ > + off_t off; > + > + off = lseek(fd, 0, SEEK_DATA); > + if (off >= 0) > + return 1; > + > + if (errno == ENXIO) > + return 0; > + > + if (report_errors) > + perror(path); > + > + return -1; > +} > + > +static int > +check_attrfile( > + const char *path, > + int fd) > +{ > + struct fsxattr fsx; > + int ret; > + > + ret = ioctl(fd, XFS_IOC_FSGETXATTR, &fsx); > + if (ret) { > + if (report_errors) > + perror(path); > + return -1; > + } > + > + if (want_attrfile && (fsx.fsx_xflags & XFS_XFLAG_HASATTR)) > + return 1; > + return 0; > +} > + > +#define BMAP_NR 33 > +static struct getbmapx bmaps[BMAP_NR]; > + > +static int > +check_sharedfile( > + const char *path, > + int fd) > +{ > + struct getbmapx *key = &bmaps[0]; > + unsigned int i; > + int ret; > + > + memset(key, 0, sizeof(struct getbmapx)); > + key->bmv_length = ULLONG_MAX; > + /* no holes and don't flush dirty pages */ > + key->bmv_iflags = BMV_IF_DELALLOC | BMV_IF_NO_HOLES; > + key->bmv_count = BMAP_NR; > + > + while ((ret = ioctl(fd, XFS_IOC_GETBMAPX, bmaps)) == 0) { > + struct getbmapx *p = &bmaps[1]; > + xfs_off_t new_off; > + > + for (i = 0; i < key->bmv_entries; i++, p++) { > + if (p->bmv_oflags & BMV_OF_SHARED) > + return 1; > + } > + > + if (key->bmv_entries == 0) > + break; > + p = key + key->bmv_entries; > + if (p->bmv_oflags & BMV_OF_LAST) > + return 0; > + > + new_off = p->bmv_offset + p->bmv_length; > + key->bmv_length -= new_off - key->bmv_offset; > + key->bmv_offset = new_off; > + } > + if (ret < 0) { > + if (report_errors) > + perror(path); > + return -1; > + } > + > + return 0; > +} > + > +static void > +print_help( > + const char *name) > +{ > + printf("Usage: %s [OPTIONS] path\n", name); > + printf("\n"); > + printf("Print all file paths matching any of the given predicates.\n"); > + printf("\n"); > + printf("-a Match files with xattrs.\n"); > + printf("-b Match files with data blocks.\n"); > + printf("-d Match directories.\n"); > + printf("-q Ignore errors while walking directory tree.\n"); > + printf("-r Match regular files.\n"); > + printf("-s Match files with shared blocks.\n"); > + printf("\n"); > + printf("If no matching options are given, match all files found.\n"); > +} > + > +static int > +visit( > + const char *path, > + const struct stat *sb, > + int typeflag, > + struct FTW *ftwbuf) > +{ > + int printme = 1; > + int fd = -1; > + int retval = FTW_CONTINUE; > + > + if (want_anyfile) > + goto out; > + if (want_regfile && typeflag == FTW_F) > + goto out; > + if (want_dir && typeflag == FTW_D) > + goto out; > + > + /* > + * We can only open directories and files; screen out everything else. > + * Note that nftw lies and reports FTW_F for device files, so check the > + * statbuf mode too. > + */ > + if (typeflag != FTW_F && typeflag != FTW_D) { > + printme = 0; > + goto out; > + } > + > + if (!S_ISREG(sb->st_mode) && !S_ISDIR(sb->st_mode)) { > + printme = 0; > + goto out; > + } > + > + fd = open(path, O_RDONLY); > + if (fd < 0) { > + if (report_errors) { > + perror(path); > + return FTW_STOP; > + } > + > + return FTW_CONTINUE; > + } > + > + if (want_datafile && typeflag == FTW_F) { > + int ret = check_datafile(path, fd); > + if (ret < 0 && report_errors) { > + printme = 0; > + retval = FTW_STOP; > + goto out_fd; > + } > + > + if (ret == 1) > + goto out_fd; > + } > + > + if (want_attrfile) { > + int ret = check_attrfile(path, fd); > + if (ret < 0 && report_errors) { > + printme = 0; > + retval = FTW_STOP; > + goto out_fd; > + } > + > + if (ret == 1) > + goto out_fd; > + } > + > + if (want_sharedfile) { > + int ret = check_sharedfile(path, fd); > + if (ret < 0 && report_errors) { > + printme = 0; > + retval = FTW_STOP; > + goto out_fd; > + } > + > + if (ret == 1) > + goto out_fd; > + } > + > + printme = 0; > +out_fd: > + close(fd); > +out: > + if (printme) > + printf("%s\n", path); > + return retval; > +} > + > +static void > +handle_sigabrt( > + int signal, > + siginfo_t *info, > + void *ucontext) > +{ > + fprintf(stderr, "Signal %u, exiting.\n", signal); > + exit(2); > +} > + > +int > +main( > + int argc, > + char *argv[]) > +{ > + struct rlimit rlimit; > + struct sigaction abrt = { > + .sa_sigaction = handle_sigabrt, > + .sa_flags = SA_SIGINFO, > + }; > + int c; > + int ret; > + > + while ((c = getopt(argc, argv, "abdqrs")) >= 0) { > + switch (c) { > + case 'a': want_attrfile = 1; break; > + case 'b': want_datafile = 1; break; > + case 'd': want_dir = 1; break; > + case 'q': report_errors = 0; break; > + case 'r': want_regfile = 1; break; > + case 's': want_sharedfile = 1; break; > + default: > + print_help(argv[0]); > + return 1; > + } > + } > + > + ret = getrlimit(RLIMIT_NOFILE, &rlimit); > + if (ret) { > + perror("RLIMIT_NOFILE"); > + return 1; > + } > + > + if (!want_attrfile && !want_datafile && !want_dir && !want_regfile && > + !want_sharedfile) > + want_anyfile = 1; > + > + /* > + * nftw is known to abort() if a directory it is walking disappears out > + * from under it. Handle this with grace if the caller wants us to run > + * quietly. > + */ > + if (!report_errors) { > + ret = sigaction(SIGABRT, &abrt, NULL); > + if (ret) { > + perror("SIGABRT handler"); > + return 1; > + } > + } > + > + for (c = optind; c < argc; c++) { > + ret = nftw(argv[c], visit, rlimit.rlim_cur - 5, > + FTW_ACTIONRETVAL | FTW_CHDIR | FTW_MOUNT | > + FTW_PHYS); > + if (ret && report_errors) { > + perror(argv[c]); > + break; > + } > + } > + > + if (ret) > + return 1; > + return 0; > +} > ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 3/5] fuzzy: add a custom xfs find utility for scrub stress tests 2023-02-05 12:57 ` Zorro Lang @ 2023-02-07 16:57 ` Darrick J. Wong 0 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2023-02-07 16:57 UTC (permalink / raw) To: Zorro Lang; +Cc: linux-xfs, fstests On Sun, Feb 05, 2023 at 08:57:47PM +0800, Zorro Lang wrote: > On Fri, Dec 30, 2022 at 02:19:06PM -0800, Darrick J. Wong wrote: > > From: Darrick J. Wong <djwong@kernel.org> > > > > Create a new find(1) like utility that doesn't crash on directory tree > > changes (like find does due to bugs in its loop detector) and actually > > implements the custom xfs attribute predicates that we need for scrub > > stress tests. This program will be needed for a future patch where we > > add stress tests for scrub and repair of file metadata. > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > --- > > configure.ac | 5 + > > include/builddefs.in | 4 + > > m4/package_libcdev.m4 | 47 ++++++++ > > m4/package_xfslibs.m4 | 16 +++ > > src/Makefile | 10 ++ > > src/xfsfind.c | 290 +++++++++++++++++++++++++++++++++++++++++++++++++ > > 6 files changed, 372 insertions(+) > > create mode 100644 src/xfsfind.c > > > > > > diff --git a/configure.ac b/configure.ac > > index cbf8377988..e92bd6b26d 100644 > > --- a/configure.ac > > +++ b/configure.ac > > @@ -66,6 +66,11 @@ AC_PACKAGE_WANT_LINUX_FS_H > > AC_PACKAGE_WANT_LIBBTRFSUTIL > > > > AC_HAVE_COPY_FILE_RANGE > > +AC_HAVE_SEEK_DATA > > +AC_HAVE_BMV_OF_SHARED > > +AC_HAVE_NFTW > > +AC_HAVE_RLIMIT_NOFILE > > + > > AC_CHECK_FUNCS([renameat2]) > > AC_CHECK_FUNCS([reallocarray]) > > AC_CHECK_TYPES([struct mount_attr], [], [], [[#include <linux/mount.h>]]) > > diff --git a/include/builddefs.in b/include/builddefs.in > > index 6641209f81..dab10c968f 100644 > > --- a/include/builddefs.in > > +++ b/include/builddefs.in > > @@ -68,6 +68,10 @@ HAVE_FIEMAP = @have_fiemap@ > > HAVE_FALLOCATE = @have_fallocate@ > > HAVE_COPY_FILE_RANGE = @have_copy_file_range@ > > HAVE_LIBBTRFSUTIL = @have_libbtrfsutil@ > > +HAVE_SEEK_DATA = @have_seek_data@ > > +HAVE_NFTW = @have_nftw@ > > +HAVE_BMV_OF_SHARED = @have_bmv_of_shared@ > > +HAVE_RLIMIT_NOFILE = @have_rlimit_nofile@ > > > > GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall > > > > diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 > > index 5c76c0f73e..e1b381c16f 100644 > > --- a/m4/package_libcdev.m4 > > +++ b/m4/package_libcdev.m4 > > @@ -110,3 +110,50 @@ AC_DEFUN([AC_HAVE_COPY_FILE_RANGE], > > AC_SUBST(have_copy_file_range) > > ]) > > > > +# Check if we have SEEK_DATA > > +AC_DEFUN([AC_HAVE_SEEK_DATA], > > + [ AC_MSG_CHECKING([for SEEK_DATA]) > > + AC_TRY_LINK([ > > The AC_TRY_LINK is obsolete by autoconf, refer to: > https://www.gnu.org/software/autoconf/manual/autoconf-2.69/html_node/Obsolete-Macros.html > > So as the suggestion of above link, we'd better to replace: > Macro: AC_TRY_LINK (includes, function-body, [action-if-true], [action-if-false]) > with: > AC_LINK_IFELSE( > [AC_LANG_PROGRAM([[includes]], > [[function-body]])], > [action-if-true], > [action-if-false]) > > For example (hope it's right:) Yeah... this patch was written so long ago I wasn't even aware of the deprecations. I've run autoupdate to fix the problems and will repost. --D > diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 > index e1b381c1..7f1767a4 100644 > --- a/m4/package_libcdev.m4 > +++ b/m4/package_libcdev.m4 > @@ -113,13 +113,13 @@ AC_DEFUN([AC_HAVE_COPY_FILE_RANGE], > # Check if we have SEEK_DATA > AC_DEFUN([AC_HAVE_SEEK_DATA], > [ AC_MSG_CHECKING([for SEEK_DATA]) > - AC_TRY_LINK([ > + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ > #define _GNU_SOURCE > #include <sys/types.h> > #include <unistd.h> > - ], [ > + ]], [[ > lseek(-1, 0, SEEK_DATA); > - ], have_seek_data=yes > + ]])], have_seek_data=yes > AC_MSG_RESULT(yes), > AC_MSG_RESULT(no)) > AC_SUBST(have_seek_data) > > > +#define _GNU_SOURCE > > +#include <sys/types.h> > > +#include <unistd.h> > > + ], [ > > + lseek(-1, 0, SEEK_DATA); > > + ], have_seek_data=yes > > + AC_MSG_RESULT(yes), > > + AC_MSG_RESULT(no)) > > + AC_SUBST(have_seek_data) > > + ]) > > + > > +# Check if we have nftw > > +AC_DEFUN([AC_HAVE_NFTW], > > + [ AC_MSG_CHECKING([for nftw]) > > + AC_TRY_LINK([ > > Same as above > > > +#define _GNU_SOURCE > > +#include <stddef.h> > > +#include <ftw.h> > > + ], [ > > + nftw("/", (int (*)(const char *, const struct stat *, int, struct FTW *))1, 0, 0); > > + ], have_nftw=yes > > + AC_MSG_RESULT(yes), > > + AC_MSG_RESULT(no)) > > + AC_SUBST(have_nftw) > > + ]) > > + > > +# Check if we have RLIMIT_NOFILE > > +AC_DEFUN([AC_HAVE_RLIMIT_NOFILE], > > + [ AC_MSG_CHECKING([for RLIMIT_NOFILE]) > > + AC_TRY_LINK([ > > Same as above > > > +#define _GNU_SOURCE > > +#include <sys/time.h> > > +#include <sys/resource.h> > > + ], [ > > + struct rlimit rlimit; > > + > > + rlimit.rlim_cur = 0; > > + getrlimit(RLIMIT_NOFILE, &rlimit); > > + ], have_rlimit_nofile=yes > > + AC_MSG_RESULT(yes), > > + AC_MSG_RESULT(no)) > > + AC_SUBST(have_rlimit_nofile) > > + ]) > > diff --git a/m4/package_xfslibs.m4 b/m4/package_xfslibs.m4 > > index 0746cd1dc5..479f30a29b 100644 > > --- a/m4/package_xfslibs.m4 > > +++ b/m4/package_xfslibs.m4 > > @@ -104,3 +104,19 @@ AC_DEFUN([AC_PACKAGE_NEED_XFSCTL_MACRO], > > exit 1 > > ]) > > ]) > > + > > +# Check if we have BMV_OF_SHARED from the GETBMAPX ioctl > > +AC_DEFUN([AC_HAVE_BMV_OF_SHARED], > > + [ AC_MSG_CHECKING([for BMV_OF_SHARED]) > > + AC_TRY_LINK([ > > Same as above > > Thanks, > Zorro > > > +#define _GNU_SOURCE > > +#include <xfs/xfs.h> > > + ], [ > > + struct getbmapx obj; > > + ioctl(-1, XFS_IOC_GETBMAPX, &obj); > > + obj.bmv_oflags |= BMV_OF_SHARED; > > + ], have_bmv_of_shared=yes > > + AC_MSG_RESULT(yes), > > + AC_MSG_RESULT(no)) > > + AC_SUBST(have_bmv_of_shared) > > + ]) > > diff --git a/src/Makefile b/src/Makefile > > index afdf6b30c5..7807ca89a5 100644 > > --- a/src/Makefile > > +++ b/src/Makefile > > @@ -83,6 +83,16 @@ ifeq ($(HAVE_LIBCAP), true) > > LLDLIBS += -lcap > > endif > > > > +ifeq ($(HAVE_SEEK_DATA), yes) > > + ifeq ($(HAVE_NFTW), yes) > > + ifeq ($(HAVE_BMV_OF_SHARED), yes) > > + ifeq ($(HAVE_RLIMIT_NOFILE), yes) > > + TARGETS += xfsfind > > + endif > > + endif > > + endif > > +endif > > + > > CFILES = $(TARGETS:=.c) > > LDIRT = $(TARGETS) fssum > > > > diff --git a/src/xfsfind.c b/src/xfsfind.c > > new file mode 100644 > > index 0000000000..6b0a93e793 > > --- /dev/null > > +++ b/src/xfsfind.c > > @@ -0,0 +1,290 @@ > > +// SPDX-License-Identifier: GPL-2.0 > > +/* > > + * find(1) but with special predicates for finding XFS attributes. > > + * Copyright (C) 2022 Oracle. > > + */ > > +#include <sys/time.h> > > +#include <sys/resource.h> > > +#include <sys/types.h> > > +#include <stdio.h> > > +#include <unistd.h> > > +#include <ftw.h> > > +#include <linux/fs.h> > > +#include <xfs/xfs.h> > > + > > +#include "global.h" > > + > > +static int want_anyfile; > > +static int want_datafile; > > +static int want_attrfile; > > +static int want_dir; > > +static int want_regfile; > > +static int want_sharedfile; > > +static int report_errors = 1; > > + > > +static int > > +check_datafile( > > + const char *path, > > + int fd) > > +{ > > + off_t off; > > + > > + off = lseek(fd, 0, SEEK_DATA); > > + if (off >= 0) > > + return 1; > > + > > + if (errno == ENXIO) > > + return 0; > > + > > + if (report_errors) > > + perror(path); > > + > > + return -1; > > +} > > + > > +static int > > +check_attrfile( > > + const char *path, > > + int fd) > > +{ > > + struct fsxattr fsx; > > + int ret; > > + > > + ret = ioctl(fd, XFS_IOC_FSGETXATTR, &fsx); > > + if (ret) { > > + if (report_errors) > > + perror(path); > > + return -1; > > + } > > + > > + if (want_attrfile && (fsx.fsx_xflags & XFS_XFLAG_HASATTR)) > > + return 1; > > + return 0; > > +} > > + > > +#define BMAP_NR 33 > > +static struct getbmapx bmaps[BMAP_NR]; > > + > > +static int > > +check_sharedfile( > > + const char *path, > > + int fd) > > +{ > > + struct getbmapx *key = &bmaps[0]; > > + unsigned int i; > > + int ret; > > + > > + memset(key, 0, sizeof(struct getbmapx)); > > + key->bmv_length = ULLONG_MAX; > > + /* no holes and don't flush dirty pages */ > > + key->bmv_iflags = BMV_IF_DELALLOC | BMV_IF_NO_HOLES; > > + key->bmv_count = BMAP_NR; > > + > > + while ((ret = ioctl(fd, XFS_IOC_GETBMAPX, bmaps)) == 0) { > > + struct getbmapx *p = &bmaps[1]; > > + xfs_off_t new_off; > > + > > + for (i = 0; i < key->bmv_entries; i++, p++) { > > + if (p->bmv_oflags & BMV_OF_SHARED) > > + return 1; > > + } > > + > > + if (key->bmv_entries == 0) > > + break; > > + p = key + key->bmv_entries; > > + if (p->bmv_oflags & BMV_OF_LAST) > > + return 0; > > + > > + new_off = p->bmv_offset + p->bmv_length; > > + key->bmv_length -= new_off - key->bmv_offset; > > + key->bmv_offset = new_off; > > + } > > + if (ret < 0) { > > + if (report_errors) > > + perror(path); > > + return -1; > > + } > > + > > + return 0; > > +} > > + > > +static void > > +print_help( > > + const char *name) > > +{ > > + printf("Usage: %s [OPTIONS] path\n", name); > > + printf("\n"); > > + printf("Print all file paths matching any of the given predicates.\n"); > > + printf("\n"); > > + printf("-a Match files with xattrs.\n"); > > + printf("-b Match files with data blocks.\n"); > > + printf("-d Match directories.\n"); > > + printf("-q Ignore errors while walking directory tree.\n"); > > + printf("-r Match regular files.\n"); > > + printf("-s Match files with shared blocks.\n"); > > + printf("\n"); > > + printf("If no matching options are given, match all files found.\n"); > > +} > > + > > +static int > > +visit( > > + const char *path, > > + const struct stat *sb, > > + int typeflag, > > + struct FTW *ftwbuf) > > +{ > > + int printme = 1; > > + int fd = -1; > > + int retval = FTW_CONTINUE; > > + > > + if (want_anyfile) > > + goto out; > > + if (want_regfile && typeflag == FTW_F) > > + goto out; > > + if (want_dir && typeflag == FTW_D) > > + goto out; > > + > > + /* > > + * We can only open directories and files; screen out everything else. > > + * Note that nftw lies and reports FTW_F for device files, so check the > > + * statbuf mode too. > > + */ > > + if (typeflag != FTW_F && typeflag != FTW_D) { > > + printme = 0; > > + goto out; > > + } > > + > > + if (!S_ISREG(sb->st_mode) && !S_ISDIR(sb->st_mode)) { > > + printme = 0; > > + goto out; > > + } > > + > > + fd = open(path, O_RDONLY); > > + if (fd < 0) { > > + if (report_errors) { > > + perror(path); > > + return FTW_STOP; > > + } > > + > > + return FTW_CONTINUE; > > + } > > + > > + if (want_datafile && typeflag == FTW_F) { > > + int ret = check_datafile(path, fd); > > + if (ret < 0 && report_errors) { > > + printme = 0; > > + retval = FTW_STOP; > > + goto out_fd; > > + } > > + > > + if (ret == 1) > > + goto out_fd; > > + } > > + > > + if (want_attrfile) { > > + int ret = check_attrfile(path, fd); > > + if (ret < 0 && report_errors) { > > + printme = 0; > > + retval = FTW_STOP; > > + goto out_fd; > > + } > > + > > + if (ret == 1) > > + goto out_fd; > > + } > > + > > + if (want_sharedfile) { > > + int ret = check_sharedfile(path, fd); > > + if (ret < 0 && report_errors) { > > + printme = 0; > > + retval = FTW_STOP; > > + goto out_fd; > > + } > > + > > + if (ret == 1) > > + goto out_fd; > > + } > > + > > + printme = 0; > > +out_fd: > > + close(fd); > > +out: > > + if (printme) > > + printf("%s\n", path); > > + return retval; > > +} > > + > > +static void > > +handle_sigabrt( > > + int signal, > > + siginfo_t *info, > > + void *ucontext) > > +{ > > + fprintf(stderr, "Signal %u, exiting.\n", signal); > > + exit(2); > > +} > > + > > +int > > +main( > > + int argc, > > + char *argv[]) > > +{ > > + struct rlimit rlimit; > > + struct sigaction abrt = { > > + .sa_sigaction = handle_sigabrt, > > + .sa_flags = SA_SIGINFO, > > + }; > > + int c; > > + int ret; > > + > > + while ((c = getopt(argc, argv, "abdqrs")) >= 0) { > > + switch (c) { > > + case 'a': want_attrfile = 1; break; > > + case 'b': want_datafile = 1; break; > > + case 'd': want_dir = 1; break; > > + case 'q': report_errors = 0; break; > > + case 'r': want_regfile = 1; break; > > + case 's': want_sharedfile = 1; break; > > + default: > > + print_help(argv[0]); > > + return 1; > > + } > > + } > > + > > + ret = getrlimit(RLIMIT_NOFILE, &rlimit); > > + if (ret) { > > + perror("RLIMIT_NOFILE"); > > + return 1; > > + } > > + > > + if (!want_attrfile && !want_datafile && !want_dir && !want_regfile && > > + !want_sharedfile) > > + want_anyfile = 1; > > + > > + /* > > + * nftw is known to abort() if a directory it is walking disappears out > > + * from under it. Handle this with grace if the caller wants us to run > > + * quietly. > > + */ > > + if (!report_errors) { > > + ret = sigaction(SIGABRT, &abrt, NULL); > > + if (ret) { > > + perror("SIGABRT handler"); > > + return 1; > > + } > > + } > > + > > + for (c = optind; c < argc; c++) { > > + ret = nftw(argv[c], visit, rlimit.rlim_cur - 5, > > + FTW_ACTIONRETVAL | FTW_CHDIR | FTW_MOUNT | > > + FTW_PHYS); > > + if (ret && report_errors) { > > + perror(argv[c]); > > + break; > > + } > > + } > > + > > + if (ret) > > + return 1; > > + return 0; > > +} > > > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH v24.1 3/5] fuzzy: add a custom xfs find utility for scrub stress tests 2022-12-30 22:19 ` [PATCH 3/5] fuzzy: add a custom xfs find utility for scrub stress tests Darrick J. Wong 2023-02-05 12:57 ` Zorro Lang @ 2023-02-07 17:01 ` Darrick J. Wong 2023-02-07 18:42 ` Zorro Lang 1 sibling, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2023-02-07 17:01 UTC (permalink / raw) To: zlang; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create a new find(1) like utility that doesn't crash on directory tree changes (like find does due to bugs in its loop detector) and actually implements the custom xfs attribute predicates that we need for scrub stress tests. This program will be needed for a future patch where we add stress tests for scrub and repair of file metadata. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- v24.1: apply some autoupdate love to the m4 macros --- configure.ac | 5 + include/builddefs.in | 4 + m4/package_libcdev.m4 | 44 +++++++ m4/package_xfslibs.m4 | 15 +++ src/Makefile | 10 ++ src/xfsfind.c | 290 +++++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 368 insertions(+) create mode 100644 src/xfsfind.c diff --git a/configure.ac b/configure.ac index cbf8377988..e92bd6b26d 100644 --- a/configure.ac +++ b/configure.ac @@ -66,6 +66,11 @@ AC_PACKAGE_WANT_LINUX_FS_H AC_PACKAGE_WANT_LIBBTRFSUTIL AC_HAVE_COPY_FILE_RANGE +AC_HAVE_SEEK_DATA +AC_HAVE_BMV_OF_SHARED +AC_HAVE_NFTW +AC_HAVE_RLIMIT_NOFILE + AC_CHECK_FUNCS([renameat2]) AC_CHECK_FUNCS([reallocarray]) AC_CHECK_TYPES([struct mount_attr], [], [], [[#include <linux/mount.h>]]) diff --git a/include/builddefs.in b/include/builddefs.in index 6641209f81..dab10c968f 100644 --- a/include/builddefs.in +++ b/include/builddefs.in @@ -68,6 +68,10 @@ HAVE_FIEMAP = @have_fiemap@ HAVE_FALLOCATE = @have_fallocate@ HAVE_COPY_FILE_RANGE = @have_copy_file_range@ HAVE_LIBBTRFSUTIL = @have_libbtrfsutil@ +HAVE_SEEK_DATA = @have_seek_data@ +HAVE_NFTW = @have_nftw@ +HAVE_BMV_OF_SHARED = @have_bmv_of_shared@ +HAVE_RLIMIT_NOFILE = @have_rlimit_nofile@ GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 index 5c76c0f73e..98572aecd9 100644 --- a/m4/package_libcdev.m4 +++ b/m4/package_libcdev.m4 @@ -110,3 +110,47 @@ AC_DEFUN([AC_HAVE_COPY_FILE_RANGE], AC_SUBST(have_copy_file_range) ]) +# Check if we have SEEK_DATA +AC_DEFUN([AC_HAVE_SEEK_DATA], + [ AC_MSG_CHECKING([for SEEK_DATA]) + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ +#define _GNU_SOURCE +#include <sys/types.h> +#include <unistd.h> + ]], [[ + lseek(-1, 0, SEEK_DATA); + ]])],[have_seek_data=yes + AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) + AC_SUBST(have_seek_data) + ]) + +# Check if we have nftw +AC_DEFUN([AC_HAVE_NFTW], + [ AC_MSG_CHECKING([for nftw]) + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ +#define _GNU_SOURCE +#include <stddef.h> +#include <ftw.h> + ]], [[ + nftw("/", (int (*)(const char *, const struct stat *, int, struct FTW *))1, 0, 0); + ]])],[have_nftw=yes + AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) + AC_SUBST(have_nftw) + ]) + +# Check if we have RLIMIT_NOFILE +AC_DEFUN([AC_HAVE_RLIMIT_NOFILE], + [ AC_MSG_CHECKING([for RLIMIT_NOFILE]) + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ +#define _GNU_SOURCE +#include <sys/time.h> +#include <sys/resource.h> + ]], [[ + struct rlimit rlimit; + + rlimit.rlim_cur = 0; + getrlimit(RLIMIT_NOFILE, &rlimit); + ]])],[have_rlimit_nofile=yes + AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) + AC_SUBST(have_rlimit_nofile) + ]) diff --git a/m4/package_xfslibs.m4 b/m4/package_xfslibs.m4 index 0746cd1dc5..8ef58cc064 100644 --- a/m4/package_xfslibs.m4 +++ b/m4/package_xfslibs.m4 @@ -104,3 +104,18 @@ AC_DEFUN([AC_PACKAGE_NEED_XFSCTL_MACRO], exit 1 ]) ]) + +# Check if we have BMV_OF_SHARED from the GETBMAPX ioctl +AC_DEFUN([AC_HAVE_BMV_OF_SHARED], + [ AC_MSG_CHECKING([for BMV_OF_SHARED]) + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ +#define _GNU_SOURCE +#include <xfs/xfs.h> + ]], [[ + struct getbmapx obj; + ioctl(-1, XFS_IOC_GETBMAPX, &obj); + obj.bmv_oflags |= BMV_OF_SHARED; + ]])],[have_bmv_of_shared=yes + AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) + AC_SUBST(have_bmv_of_shared) + ]) diff --git a/src/Makefile b/src/Makefile index f270015ce8..a574f7bd03 100644 --- a/src/Makefile +++ b/src/Makefile @@ -83,6 +83,16 @@ ifeq ($(HAVE_LIBCAP), true) LLDLIBS += -lcap endif +ifeq ($(HAVE_SEEK_DATA), yes) + ifeq ($(HAVE_NFTW), yes) + ifeq ($(HAVE_BMV_OF_SHARED), yes) + ifeq ($(HAVE_RLIMIT_NOFILE), yes) + TARGETS += xfsfind + endif + endif + endif +endif + CFILES = $(TARGETS:=.c) LDIRT = $(TARGETS) fssum diff --git a/src/xfsfind.c b/src/xfsfind.c new file mode 100644 index 0000000000..6b0a93e793 --- /dev/null +++ b/src/xfsfind.c @@ -0,0 +1,290 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * find(1) but with special predicates for finding XFS attributes. + * Copyright (C) 2022 Oracle. + */ +#include <sys/time.h> +#include <sys/resource.h> +#include <sys/types.h> +#include <stdio.h> +#include <unistd.h> +#include <ftw.h> +#include <linux/fs.h> +#include <xfs/xfs.h> + +#include "global.h" + +static int want_anyfile; +static int want_datafile; +static int want_attrfile; +static int want_dir; +static int want_regfile; +static int want_sharedfile; +static int report_errors = 1; + +static int +check_datafile( + const char *path, + int fd) +{ + off_t off; + + off = lseek(fd, 0, SEEK_DATA); + if (off >= 0) + return 1; + + if (errno == ENXIO) + return 0; + + if (report_errors) + perror(path); + + return -1; +} + +static int +check_attrfile( + const char *path, + int fd) +{ + struct fsxattr fsx; + int ret; + + ret = ioctl(fd, XFS_IOC_FSGETXATTR, &fsx); + if (ret) { + if (report_errors) + perror(path); + return -1; + } + + if (want_attrfile && (fsx.fsx_xflags & XFS_XFLAG_HASATTR)) + return 1; + return 0; +} + +#define BMAP_NR 33 +static struct getbmapx bmaps[BMAP_NR]; + +static int +check_sharedfile( + const char *path, + int fd) +{ + struct getbmapx *key = &bmaps[0]; + unsigned int i; + int ret; + + memset(key, 0, sizeof(struct getbmapx)); + key->bmv_length = ULLONG_MAX; + /* no holes and don't flush dirty pages */ + key->bmv_iflags = BMV_IF_DELALLOC | BMV_IF_NO_HOLES; + key->bmv_count = BMAP_NR; + + while ((ret = ioctl(fd, XFS_IOC_GETBMAPX, bmaps)) == 0) { + struct getbmapx *p = &bmaps[1]; + xfs_off_t new_off; + + for (i = 0; i < key->bmv_entries; i++, p++) { + if (p->bmv_oflags & BMV_OF_SHARED) + return 1; + } + + if (key->bmv_entries == 0) + break; + p = key + key->bmv_entries; + if (p->bmv_oflags & BMV_OF_LAST) + return 0; + + new_off = p->bmv_offset + p->bmv_length; + key->bmv_length -= new_off - key->bmv_offset; + key->bmv_offset = new_off; + } + if (ret < 0) { + if (report_errors) + perror(path); + return -1; + } + + return 0; +} + +static void +print_help( + const char *name) +{ + printf("Usage: %s [OPTIONS] path\n", name); + printf("\n"); + printf("Print all file paths matching any of the given predicates.\n"); + printf("\n"); + printf("-a Match files with xattrs.\n"); + printf("-b Match files with data blocks.\n"); + printf("-d Match directories.\n"); + printf("-q Ignore errors while walking directory tree.\n"); + printf("-r Match regular files.\n"); + printf("-s Match files with shared blocks.\n"); + printf("\n"); + printf("If no matching options are given, match all files found.\n"); +} + +static int +visit( + const char *path, + const struct stat *sb, + int typeflag, + struct FTW *ftwbuf) +{ + int printme = 1; + int fd = -1; + int retval = FTW_CONTINUE; + + if (want_anyfile) + goto out; + if (want_regfile && typeflag == FTW_F) + goto out; + if (want_dir && typeflag == FTW_D) + goto out; + + /* + * We can only open directories and files; screen out everything else. + * Note that nftw lies and reports FTW_F for device files, so check the + * statbuf mode too. + */ + if (typeflag != FTW_F && typeflag != FTW_D) { + printme = 0; + goto out; + } + + if (!S_ISREG(sb->st_mode) && !S_ISDIR(sb->st_mode)) { + printme = 0; + goto out; + } + + fd = open(path, O_RDONLY); + if (fd < 0) { + if (report_errors) { + perror(path); + return FTW_STOP; + } + + return FTW_CONTINUE; + } + + if (want_datafile && typeflag == FTW_F) { + int ret = check_datafile(path, fd); + if (ret < 0 && report_errors) { + printme = 0; + retval = FTW_STOP; + goto out_fd; + } + + if (ret == 1) + goto out_fd; + } + + if (want_attrfile) { + int ret = check_attrfile(path, fd); + if (ret < 0 && report_errors) { + printme = 0; + retval = FTW_STOP; + goto out_fd; + } + + if (ret == 1) + goto out_fd; + } + + if (want_sharedfile) { + int ret = check_sharedfile(path, fd); + if (ret < 0 && report_errors) { + printme = 0; + retval = FTW_STOP; + goto out_fd; + } + + if (ret == 1) + goto out_fd; + } + + printme = 0; +out_fd: + close(fd); +out: + if (printme) + printf("%s\n", path); + return retval; +} + +static void +handle_sigabrt( + int signal, + siginfo_t *info, + void *ucontext) +{ + fprintf(stderr, "Signal %u, exiting.\n", signal); + exit(2); +} + +int +main( + int argc, + char *argv[]) +{ + struct rlimit rlimit; + struct sigaction abrt = { + .sa_sigaction = handle_sigabrt, + .sa_flags = SA_SIGINFO, + }; + int c; + int ret; + + while ((c = getopt(argc, argv, "abdqrs")) >= 0) { + switch (c) { + case 'a': want_attrfile = 1; break; + case 'b': want_datafile = 1; break; + case 'd': want_dir = 1; break; + case 'q': report_errors = 0; break; + case 'r': want_regfile = 1; break; + case 's': want_sharedfile = 1; break; + default: + print_help(argv[0]); + return 1; + } + } + + ret = getrlimit(RLIMIT_NOFILE, &rlimit); + if (ret) { + perror("RLIMIT_NOFILE"); + return 1; + } + + if (!want_attrfile && !want_datafile && !want_dir && !want_regfile && + !want_sharedfile) + want_anyfile = 1; + + /* + * nftw is known to abort() if a directory it is walking disappears out + * from under it. Handle this with grace if the caller wants us to run + * quietly. + */ + if (!report_errors) { + ret = sigaction(SIGABRT, &abrt, NULL); + if (ret) { + perror("SIGABRT handler"); + return 1; + } + } + + for (c = optind; c < argc; c++) { + ret = nftw(argv[c], visit, rlimit.rlim_cur - 5, + FTW_ACTIONRETVAL | FTW_CHDIR | FTW_MOUNT | + FTW_PHYS); + if (ret && report_errors) { + perror(argv[c]); + break; + } + } + + if (ret) + return 1; + return 0; +} ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH v24.1 3/5] fuzzy: add a custom xfs find utility for scrub stress tests 2023-02-07 17:01 ` [PATCH v24.1 " Darrick J. Wong @ 2023-02-07 18:42 ` Zorro Lang 0 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-07 18:42 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Tue, Feb 07, 2023 at 09:01:41AM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Create a new find(1) like utility that doesn't crash on directory tree > changes (like find does due to bugs in its loop detector) and actually > implements the custom xfs attribute predicates that we need for scrub > stress tests. This program will be needed for a future patch where we > add stress tests for scrub and repair of file metadata. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- > v24.1: apply some autoupdate love to the m4 macros > --- This version looks better to me, I'll merge it with a little change on .gitignore file to filter src/xfsfind. Reviewed-by: Zorro Lang <zlang@redhat.com> > configure.ac | 5 + > include/builddefs.in | 4 + > m4/package_libcdev.m4 | 44 +++++++ > m4/package_xfslibs.m4 | 15 +++ > src/Makefile | 10 ++ > src/xfsfind.c | 290 +++++++++++++++++++++++++++++++++++++++++++++++++ > 6 files changed, 368 insertions(+) > create mode 100644 src/xfsfind.c > > diff --git a/configure.ac b/configure.ac > index cbf8377988..e92bd6b26d 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -66,6 +66,11 @@ AC_PACKAGE_WANT_LINUX_FS_H > AC_PACKAGE_WANT_LIBBTRFSUTIL > > AC_HAVE_COPY_FILE_RANGE > +AC_HAVE_SEEK_DATA > +AC_HAVE_BMV_OF_SHARED > +AC_HAVE_NFTW > +AC_HAVE_RLIMIT_NOFILE > + > AC_CHECK_FUNCS([renameat2]) > AC_CHECK_FUNCS([reallocarray]) > AC_CHECK_TYPES([struct mount_attr], [], [], [[#include <linux/mount.h>]]) > diff --git a/include/builddefs.in b/include/builddefs.in > index 6641209f81..dab10c968f 100644 > --- a/include/builddefs.in > +++ b/include/builddefs.in > @@ -68,6 +68,10 @@ HAVE_FIEMAP = @have_fiemap@ > HAVE_FALLOCATE = @have_fallocate@ > HAVE_COPY_FILE_RANGE = @have_copy_file_range@ > HAVE_LIBBTRFSUTIL = @have_libbtrfsutil@ > +HAVE_SEEK_DATA = @have_seek_data@ > +HAVE_NFTW = @have_nftw@ > +HAVE_BMV_OF_SHARED = @have_bmv_of_shared@ > +HAVE_RLIMIT_NOFILE = @have_rlimit_nofile@ > > GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall > > diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 > index 5c76c0f73e..98572aecd9 100644 > --- a/m4/package_libcdev.m4 > +++ b/m4/package_libcdev.m4 > @@ -110,3 +110,47 @@ AC_DEFUN([AC_HAVE_COPY_FILE_RANGE], > AC_SUBST(have_copy_file_range) > ]) > > +# Check if we have SEEK_DATA > +AC_DEFUN([AC_HAVE_SEEK_DATA], > + [ AC_MSG_CHECKING([for SEEK_DATA]) > + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ > +#define _GNU_SOURCE > +#include <sys/types.h> > +#include <unistd.h> > + ]], [[ > + lseek(-1, 0, SEEK_DATA); > + ]])],[have_seek_data=yes > + AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) > + AC_SUBST(have_seek_data) > + ]) > + > +# Check if we have nftw > +AC_DEFUN([AC_HAVE_NFTW], > + [ AC_MSG_CHECKING([for nftw]) > + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ > +#define _GNU_SOURCE > +#include <stddef.h> > +#include <ftw.h> > + ]], [[ > + nftw("/", (int (*)(const char *, const struct stat *, int, struct FTW *))1, 0, 0); > + ]])],[have_nftw=yes > + AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) > + AC_SUBST(have_nftw) > + ]) > + > +# Check if we have RLIMIT_NOFILE > +AC_DEFUN([AC_HAVE_RLIMIT_NOFILE], > + [ AC_MSG_CHECKING([for RLIMIT_NOFILE]) > + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ > +#define _GNU_SOURCE > +#include <sys/time.h> > +#include <sys/resource.h> > + ]], [[ > + struct rlimit rlimit; > + > + rlimit.rlim_cur = 0; > + getrlimit(RLIMIT_NOFILE, &rlimit); > + ]])],[have_rlimit_nofile=yes > + AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) > + AC_SUBST(have_rlimit_nofile) > + ]) > diff --git a/m4/package_xfslibs.m4 b/m4/package_xfslibs.m4 > index 0746cd1dc5..8ef58cc064 100644 > --- a/m4/package_xfslibs.m4 > +++ b/m4/package_xfslibs.m4 > @@ -104,3 +104,18 @@ AC_DEFUN([AC_PACKAGE_NEED_XFSCTL_MACRO], > exit 1 > ]) > ]) > + > +# Check if we have BMV_OF_SHARED from the GETBMAPX ioctl > +AC_DEFUN([AC_HAVE_BMV_OF_SHARED], > + [ AC_MSG_CHECKING([for BMV_OF_SHARED]) > + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ > +#define _GNU_SOURCE > +#include <xfs/xfs.h> > + ]], [[ > + struct getbmapx obj; > + ioctl(-1, XFS_IOC_GETBMAPX, &obj); > + obj.bmv_oflags |= BMV_OF_SHARED; > + ]])],[have_bmv_of_shared=yes > + AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) > + AC_SUBST(have_bmv_of_shared) > + ]) > diff --git a/src/Makefile b/src/Makefile > index f270015ce8..a574f7bd03 100644 > --- a/src/Makefile > +++ b/src/Makefile > @@ -83,6 +83,16 @@ ifeq ($(HAVE_LIBCAP), true) > LLDLIBS += -lcap > endif > > +ifeq ($(HAVE_SEEK_DATA), yes) > + ifeq ($(HAVE_NFTW), yes) > + ifeq ($(HAVE_BMV_OF_SHARED), yes) > + ifeq ($(HAVE_RLIMIT_NOFILE), yes) > + TARGETS += xfsfind > + endif > + endif > + endif > +endif > + > CFILES = $(TARGETS:=.c) > LDIRT = $(TARGETS) fssum > > diff --git a/src/xfsfind.c b/src/xfsfind.c > new file mode 100644 > index 0000000000..6b0a93e793 > --- /dev/null > +++ b/src/xfsfind.c > @@ -0,0 +1,290 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * find(1) but with special predicates for finding XFS attributes. > + * Copyright (C) 2022 Oracle. > + */ > +#include <sys/time.h> > +#include <sys/resource.h> > +#include <sys/types.h> > +#include <stdio.h> > +#include <unistd.h> > +#include <ftw.h> > +#include <linux/fs.h> > +#include <xfs/xfs.h> > + > +#include "global.h" > + > +static int want_anyfile; > +static int want_datafile; > +static int want_attrfile; > +static int want_dir; > +static int want_regfile; > +static int want_sharedfile; > +static int report_errors = 1; > + > +static int > +check_datafile( > + const char *path, > + int fd) > +{ > + off_t off; > + > + off = lseek(fd, 0, SEEK_DATA); > + if (off >= 0) > + return 1; > + > + if (errno == ENXIO) > + return 0; > + > + if (report_errors) > + perror(path); > + > + return -1; > +} > + > +static int > +check_attrfile( > + const char *path, > + int fd) > +{ > + struct fsxattr fsx; > + int ret; > + > + ret = ioctl(fd, XFS_IOC_FSGETXATTR, &fsx); > + if (ret) { > + if (report_errors) > + perror(path); > + return -1; > + } > + > + if (want_attrfile && (fsx.fsx_xflags & XFS_XFLAG_HASATTR)) > + return 1; > + return 0; > +} > + > +#define BMAP_NR 33 > +static struct getbmapx bmaps[BMAP_NR]; > + > +static int > +check_sharedfile( > + const char *path, > + int fd) > +{ > + struct getbmapx *key = &bmaps[0]; > + unsigned int i; > + int ret; > + > + memset(key, 0, sizeof(struct getbmapx)); > + key->bmv_length = ULLONG_MAX; > + /* no holes and don't flush dirty pages */ > + key->bmv_iflags = BMV_IF_DELALLOC | BMV_IF_NO_HOLES; > + key->bmv_count = BMAP_NR; > + > + while ((ret = ioctl(fd, XFS_IOC_GETBMAPX, bmaps)) == 0) { > + struct getbmapx *p = &bmaps[1]; > + xfs_off_t new_off; > + > + for (i = 0; i < key->bmv_entries; i++, p++) { > + if (p->bmv_oflags & BMV_OF_SHARED) > + return 1; > + } > + > + if (key->bmv_entries == 0) > + break; > + p = key + key->bmv_entries; > + if (p->bmv_oflags & BMV_OF_LAST) > + return 0; > + > + new_off = p->bmv_offset + p->bmv_length; > + key->bmv_length -= new_off - key->bmv_offset; > + key->bmv_offset = new_off; > + } > + if (ret < 0) { > + if (report_errors) > + perror(path); > + return -1; > + } > + > + return 0; > +} > + > +static void > +print_help( > + const char *name) > +{ > + printf("Usage: %s [OPTIONS] path\n", name); > + printf("\n"); > + printf("Print all file paths matching any of the given predicates.\n"); > + printf("\n"); > + printf("-a Match files with xattrs.\n"); > + printf("-b Match files with data blocks.\n"); > + printf("-d Match directories.\n"); > + printf("-q Ignore errors while walking directory tree.\n"); > + printf("-r Match regular files.\n"); > + printf("-s Match files with shared blocks.\n"); > + printf("\n"); > + printf("If no matching options are given, match all files found.\n"); > +} > + > +static int > +visit( > + const char *path, > + const struct stat *sb, > + int typeflag, > + struct FTW *ftwbuf) > +{ > + int printme = 1; > + int fd = -1; > + int retval = FTW_CONTINUE; > + > + if (want_anyfile) > + goto out; > + if (want_regfile && typeflag == FTW_F) > + goto out; > + if (want_dir && typeflag == FTW_D) > + goto out; > + > + /* > + * We can only open directories and files; screen out everything else. > + * Note that nftw lies and reports FTW_F for device files, so check the > + * statbuf mode too. > + */ > + if (typeflag != FTW_F && typeflag != FTW_D) { > + printme = 0; > + goto out; > + } > + > + if (!S_ISREG(sb->st_mode) && !S_ISDIR(sb->st_mode)) { > + printme = 0; > + goto out; > + } > + > + fd = open(path, O_RDONLY); > + if (fd < 0) { > + if (report_errors) { > + perror(path); > + return FTW_STOP; > + } > + > + return FTW_CONTINUE; > + } > + > + if (want_datafile && typeflag == FTW_F) { > + int ret = check_datafile(path, fd); > + if (ret < 0 && report_errors) { > + printme = 0; > + retval = FTW_STOP; > + goto out_fd; > + } > + > + if (ret == 1) > + goto out_fd; > + } > + > + if (want_attrfile) { > + int ret = check_attrfile(path, fd); > + if (ret < 0 && report_errors) { > + printme = 0; > + retval = FTW_STOP; > + goto out_fd; > + } > + > + if (ret == 1) > + goto out_fd; > + } > + > + if (want_sharedfile) { > + int ret = check_sharedfile(path, fd); > + if (ret < 0 && report_errors) { > + printme = 0; > + retval = FTW_STOP; > + goto out_fd; > + } > + > + if (ret == 1) > + goto out_fd; > + } > + > + printme = 0; > +out_fd: > + close(fd); > +out: > + if (printme) > + printf("%s\n", path); > + return retval; > +} > + > +static void > +handle_sigabrt( > + int signal, > + siginfo_t *info, > + void *ucontext) > +{ > + fprintf(stderr, "Signal %u, exiting.\n", signal); > + exit(2); > +} > + > +int > +main( > + int argc, > + char *argv[]) > +{ > + struct rlimit rlimit; > + struct sigaction abrt = { > + .sa_sigaction = handle_sigabrt, > + .sa_flags = SA_SIGINFO, > + }; > + int c; > + int ret; > + > + while ((c = getopt(argc, argv, "abdqrs")) >= 0) { > + switch (c) { > + case 'a': want_attrfile = 1; break; > + case 'b': want_datafile = 1; break; > + case 'd': want_dir = 1; break; > + case 'q': report_errors = 0; break; > + case 'r': want_regfile = 1; break; > + case 's': want_sharedfile = 1; break; > + default: > + print_help(argv[0]); > + return 1; > + } > + } > + > + ret = getrlimit(RLIMIT_NOFILE, &rlimit); > + if (ret) { > + perror("RLIMIT_NOFILE"); > + return 1; > + } > + > + if (!want_attrfile && !want_datafile && !want_dir && !want_regfile && > + !want_sharedfile) > + want_anyfile = 1; > + > + /* > + * nftw is known to abort() if a directory it is walking disappears out > + * from under it. Handle this with grace if the caller wants us to run > + * quietly. > + */ > + if (!report_errors) { > + ret = sigaction(SIGABRT, &abrt, NULL); > + if (ret) { > + perror("SIGABRT handler"); > + return 1; > + } > + } > + > + for (c = optind; c < argc; c++) { > + ret = nftw(argv[c], visit, rlimit.rlim_cur - 5, > + FTW_ACTIONRETVAL | FTW_CHDIR | FTW_MOUNT | > + FTW_PHYS); > + if (ret && report_errors) { > + perror(argv[c]); > + break; > + } > + } > + > + if (ret) > + return 1; > + return 0; > +} > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/5] xfs/357: switch fuzzing to agi 1 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 3/5] fuzzy: add a custom xfs find utility for scrub stress tests Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-07 18:46 ` Zorro Lang 2022-12-30 22:19 ` [PATCH 4/5] fuzzy: allow xfs scrub stress tests to pick preconfigured fsstress configs Darrick J. Wong 4 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Since we now require a working AGI 0 to mount the system, fuzz AGI 1 instead. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/357 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/xfs/357 b/tests/xfs/357 index 8a2c920ef4..25af8624db 100755 --- a/tests/xfs/357 +++ b/tests/xfs/357 @@ -25,7 +25,7 @@ echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 echo "Fuzz AGI" -_scratch_xfs_fuzz_metadata '' 'online' 'agi 0' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' 'agi 1' >> $seqres.full echo "Done fuzzing AGI" # success, all done ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 1/5] xfs/357: switch fuzzing to agi 1 2022-12-30 22:19 ` [PATCH 1/5] xfs/357: switch fuzzing to agi 1 Darrick J. Wong @ 2023-02-07 18:46 ` Zorro Lang 0 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-07 18:46 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:06PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Since we now require a working AGI 0 to mount the system, fuzz AGI 1 > instead. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- Looks good to me, Reviewed-by: Zorro Lang <zlang@redhat.com> > tests/xfs/357 | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > > diff --git a/tests/xfs/357 b/tests/xfs/357 > index 8a2c920ef4..25af8624db 100755 > --- a/tests/xfs/357 > +++ b/tests/xfs/357 > @@ -25,7 +25,7 @@ echo "Format and populate" > _scratch_populate_cached nofill > $seqres.full 2>&1 > > echo "Fuzz AGI" > -_scratch_xfs_fuzz_metadata '' 'online' 'agi 0' >> $seqres.full > +_scratch_xfs_fuzz_metadata '' 'online' 'agi 1' >> $seqres.full > echo "Done fuzzing AGI" > > # success, all done > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 4/5] fuzzy: allow xfs scrub stress tests to pick preconfigured fsstress configs 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 1/5] xfs/357: switch fuzzing to agi 1 Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-07 18:48 ` Zorro Lang 4 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Make it so that xfs_scrub stress tests can select what kind of fsstress operations they want to run. This will make it easier for, say, directory scrubbers to configure fsstress to exercise directory tree changes while skipping file data updates, because those are irrelevant. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 74 insertions(+), 3 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index e39f787e78..c4a5bc9261 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -466,6 +466,7 @@ __stress_scrub_fsx_loop() { local end="$1" local runningfile="$2" local remount_period="$3" + local stress_tgt="$4" # ignored local focus=(-q -X) # quiet, validate file contents # As of November 2022, 2 million fsx ops should be enough to keep @@ -528,10 +529,70 @@ __stress_scrub_fsstress_loop() { local end="$1" local runningfile="$2" local remount_period="$3" + local stress_tgt="$4" + local focus=() + + case "$stress_tgt" in + "dir") + focus+=('-z') + + # Create a directory tree rapidly + for op in creat link mkdir mknod symlink; do + focus+=('-f' "${op}=8") + done + focus+=('-f' 'rmdir=2' '-f' 'unlink=8') + + # Rename half as often + for op in rename rnoreplace rexchange; do + focus+=('-f' "${op}=4") + done + + # Read and sync occasionally + for op in getdents stat fsync; do + focus+=('-f' "${op}=1") + done + ;; + "xattr") + focus+=('-z') + + # Create a directory tree slowly + for op in creat ; do + focus+=('-f' "${op}=2") + done + for op in unlink rmdir; do + focus+=('-f' "${op}=1") + done + + # Create xattrs rapidly + for op in attr_set setfattr; do + focus+=('-f' "${op}=80") + done + + # Remove xattrs 1/4 as quickly + for op in attr_remove removefattr; do + focus+=('-f' "${op}=20") + done + + # Read and sync occasionally + for op in listfattr getfattr fsync; do + focus+=('-f' "${op}=10") + done + ;; + "writeonly") + # Only do things that cause filesystem writes + focus+=('-w') + ;; + "default") + # No new arguments + ;; + *) + echo "$stress_tgt: Unrecognized stress target, using defaults." + ;; + esac # As of March 2022, 2 million fsstress ops should be enough to keep # any filesystem busy for a couple of hours. - local args=$(_scale_fsstress_args -p 4 -d $SCRATCH_MNT -n 2000000 $FSSTRESS_AVOID) + local args=$(_scale_fsstress_args -p 4 -d $SCRATCH_MNT -n 2000000 "${focus[@]}" $FSSTRESS_AVOID) echo "Running $FSSTRESS_PROG $args" >> $seqres.full if [ -n "$remount_period" ]; then @@ -691,6 +752,14 @@ __stress_scrub_check_commands() { # -w Delay the start of the scrub/repair loop by this number of seconds. # Defaults to no delay unless XFS_SCRUB_STRESS_DELAY is set. This value # will be clamped to ten seconds before the end time. +# -x Focus on this type of fsstress operation. Possible values: +# +# 'dir': Grow the directory trees as much as possible. +# 'xattr': Grow extended attributes in a small tree. +# 'default': Run fsstress with default arguments. +# 'writeonly': Only perform fs updates, no reads. +# +# The default is 'default' unless XFS_SCRUB_STRESS_TARGET is set. # -X Run this program to exercise the filesystem. Currently supported # options are 'fsx' and 'fsstress'. The default is 'fsstress'. _scratch_xfs_stress_scrub() { @@ -703,6 +772,7 @@ _scratch_xfs_stress_scrub() { local exerciser="fsstress" local io_args=() local remount_period="${XFS_SCRUB_STRESS_REMOUNT_PERIOD}" + local stress_tgt="${XFS_SCRUB_STRESS_TARGET:-default}" __SCRUB_STRESS_FREEZE_PID="" __SCRUB_STRESS_REMOUNT_LOOP="" @@ -710,7 +780,7 @@ _scratch_xfs_stress_scrub() { touch "$runningfile" OPTIND=1 - while getopts "fi:r:s:S:t:w:X:" c; do + while getopts "fi:r:s:S:t:w:x:X:" c; do case "$c" in f) freeze=yes;; i) io_args+=("$OPTARG");; @@ -719,6 +789,7 @@ _scratch_xfs_stress_scrub() { S) xfs_scrub_args+=("$OPTARG");; t) scrub_tgt="$OPTARG";; w) scrub_delay="$OPTARG";; + x) stress_tgt="$OPTARG";; X) exerciser="$OPTARG";; *) return 1; ;; esac @@ -757,7 +828,7 @@ _scratch_xfs_stress_scrub() { fi "__stress_scrub_${exerciser}_loop" "$end" "$runningfile" \ - "$remount_period" & + "$remount_period" "$stress_tgt" & if [ -n "$freeze" ]; then __stress_scrub_freeze_loop "$end" "$runningfile" & ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 4/5] fuzzy: allow xfs scrub stress tests to pick preconfigured fsstress configs 2022-12-30 22:19 ` [PATCH 4/5] fuzzy: allow xfs scrub stress tests to pick preconfigured fsstress configs Darrick J. Wong @ 2023-02-07 18:48 ` Zorro Lang 0 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-07 18:48 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:06PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Make it so that xfs_scrub stress tests can select what kind of fsstress > operations they want to run. This will make it easier for, say, > directory scrubbers to configure fsstress to exercise directory tree > changes while skipping file data updates, because those are irrelevant. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- Reviewed-by: Zorro Lang <zlang@redhat.com> > common/fuzzy | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- > 1 file changed, 74 insertions(+), 3 deletions(-) > > > diff --git a/common/fuzzy b/common/fuzzy > index e39f787e78..c4a5bc9261 100644 > --- a/common/fuzzy > +++ b/common/fuzzy > @@ -466,6 +466,7 @@ __stress_scrub_fsx_loop() { > local end="$1" > local runningfile="$2" > local remount_period="$3" > + local stress_tgt="$4" # ignored > local focus=(-q -X) # quiet, validate file contents > > # As of November 2022, 2 million fsx ops should be enough to keep > @@ -528,10 +529,70 @@ __stress_scrub_fsstress_loop() { > local end="$1" > local runningfile="$2" > local remount_period="$3" > + local stress_tgt="$4" > + local focus=() > + > + case "$stress_tgt" in > + "dir") > + focus+=('-z') > + > + # Create a directory tree rapidly > + for op in creat link mkdir mknod symlink; do > + focus+=('-f' "${op}=8") > + done > + focus+=('-f' 'rmdir=2' '-f' 'unlink=8') > + > + # Rename half as often > + for op in rename rnoreplace rexchange; do > + focus+=('-f' "${op}=4") > + done > + > + # Read and sync occasionally > + for op in getdents stat fsync; do > + focus+=('-f' "${op}=1") > + done > + ;; > + "xattr") > + focus+=('-z') > + > + # Create a directory tree slowly > + for op in creat ; do > + focus+=('-f' "${op}=2") > + done > + for op in unlink rmdir; do > + focus+=('-f' "${op}=1") > + done > + > + # Create xattrs rapidly > + for op in attr_set setfattr; do > + focus+=('-f' "${op}=80") > + done > + > + # Remove xattrs 1/4 as quickly > + for op in attr_remove removefattr; do > + focus+=('-f' "${op}=20") > + done > + > + # Read and sync occasionally > + for op in listfattr getfattr fsync; do > + focus+=('-f' "${op}=10") > + done > + ;; > + "writeonly") > + # Only do things that cause filesystem writes > + focus+=('-w') > + ;; > + "default") > + # No new arguments > + ;; > + *) > + echo "$stress_tgt: Unrecognized stress target, using defaults." > + ;; > + esac > > # As of March 2022, 2 million fsstress ops should be enough to keep > # any filesystem busy for a couple of hours. > - local args=$(_scale_fsstress_args -p 4 -d $SCRATCH_MNT -n 2000000 $FSSTRESS_AVOID) > + local args=$(_scale_fsstress_args -p 4 -d $SCRATCH_MNT -n 2000000 "${focus[@]}" $FSSTRESS_AVOID) > echo "Running $FSSTRESS_PROG $args" >> $seqres.full > > if [ -n "$remount_period" ]; then > @@ -691,6 +752,14 @@ __stress_scrub_check_commands() { > # -w Delay the start of the scrub/repair loop by this number of seconds. > # Defaults to no delay unless XFS_SCRUB_STRESS_DELAY is set. This value > # will be clamped to ten seconds before the end time. > +# -x Focus on this type of fsstress operation. Possible values: > +# > +# 'dir': Grow the directory trees as much as possible. > +# 'xattr': Grow extended attributes in a small tree. > +# 'default': Run fsstress with default arguments. > +# 'writeonly': Only perform fs updates, no reads. > +# > +# The default is 'default' unless XFS_SCRUB_STRESS_TARGET is set. > # -X Run this program to exercise the filesystem. Currently supported > # options are 'fsx' and 'fsstress'. The default is 'fsstress'. > _scratch_xfs_stress_scrub() { > @@ -703,6 +772,7 @@ _scratch_xfs_stress_scrub() { > local exerciser="fsstress" > local io_args=() > local remount_period="${XFS_SCRUB_STRESS_REMOUNT_PERIOD}" > + local stress_tgt="${XFS_SCRUB_STRESS_TARGET:-default}" > > __SCRUB_STRESS_FREEZE_PID="" > __SCRUB_STRESS_REMOUNT_LOOP="" > @@ -710,7 +780,7 @@ _scratch_xfs_stress_scrub() { > touch "$runningfile" > > OPTIND=1 > - while getopts "fi:r:s:S:t:w:X:" c; do > + while getopts "fi:r:s:S:t:w:x:X:" c; do > case "$c" in > f) freeze=yes;; > i) io_args+=("$OPTARG");; > @@ -719,6 +789,7 @@ _scratch_xfs_stress_scrub() { > S) xfs_scrub_args+=("$OPTARG");; > t) scrub_tgt="$OPTARG";; > w) scrub_delay="$OPTARG";; > + x) stress_tgt="$OPTARG";; > X) exerciser="$OPTARG";; > *) return 1; ;; > esac > @@ -757,7 +828,7 @@ _scratch_xfs_stress_scrub() { > fi > > "__stress_scrub_${exerciser}_loop" "$end" "$runningfile" \ > - "$remount_period" & > + "$remount_period" "$stress_tgt" & > > if [ -n "$freeze" ]; then > __stress_scrub_freeze_loop "$end" "$runningfile" & > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/1] xfs: force rebuilding of metadata 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] fuzzy: use FORCE_REBUILD over injecting force_repair Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Darrick J. Wong ` (15 subsequent siblings) 17 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, This patchset adds a new IFLAG to the scrub ioctl so that userspace can force a rebuild of an otherwise consistent piece of metadata. This will eventually enable the use of online repair to relocate metadata during a filesystem reorganization (e.g. shrink). For now, it facilitates stress testing of online repair without needing the debugging knobs to be enabled. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-force-rebuild xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-force-rebuild fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-force-rebuild --- common/fuzzy | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/1] fuzzy: use FORCE_REBUILD over injecting force_repair 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] xfs: force rebuilding of metadata Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-14 8:00 ` Zorro Lang 0 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> For stress testing online repair, try to use the FORCE_REBUILD ioctl flag over the error injection knobs whenever possible because the knobs are very noisy and are not always available. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index f7f660bc31..14f7fdf03c 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -398,6 +398,9 @@ __stress_one_scrub_loop() { local xfs_io_args=() for arg in "$@"; do + if [ -n "$SCRUBSTRESS_USE_FORCE_REBUILD" ]; then + arg="$(echo "$arg" | sed -e 's/^repair/repair -R/g')" + fi if echo "$arg" | grep -q -w '%agno%'; then # Substitute the AG number for ((agno = 0; agno < agcount; agno++)); do @@ -695,13 +698,21 @@ _require_xfs_stress_scrub() { _notrun 'xfs scrub stress test requires common/filter' } +# Make sure that we can force repairs either by error injection or passing +# FORCE_REBUILD via ioctl. +__require_xfs_stress_force_rebuild() { + local output="$($XFS_IO_PROG -x -c 'repair -R probe' $SCRATCH_MNT 2>&1)" + test -z "$output" && return + _require_xfs_io_error_injection "force_repair" +} + # Make sure we have everything we need to run stress and online repair _require_xfs_stress_online_repair() { _require_xfs_stress_scrub _require_xfs_io_command "repair" command -v _require_xfs_io_error_injection &>/dev/null || \ _notrun 'xfs repair stress test requires common/inject' - _require_xfs_io_error_injection "force_repair" + __require_xfs_stress_force_rebuild _require_freeze } @@ -783,7 +794,11 @@ __stress_scrub_check_commands() { esac for arg in "$@"; do - local cooked_arg="$(echo "$arg" | sed -e "s/%agno%/0/g")" + local cooked_arg="$arg" + if [ -n "$SCRUBSTRESS_USE_FORCE_REBUILD" ]; then + cooked_arg="$(echo "$cooked_arg" | sed -e 's/^repair/repair -R/g')" + fi + cooked_arg="$(echo "$cooked_arg" | sed -e "s/%agno%/0/g")" testio=`$XFS_IO_PROG -x -c "$cooked_arg" "$cooked_tgt" 2>&1` echo $testio | grep -q "Unknown type" && \ _notrun "xfs_io scrub subcommand support is missing" @@ -943,10 +958,23 @@ _scratch_xfs_stress_scrub() { echo "Loop finished at $(date)" >> $seqres.full } +# Decide if we're going to force repairs either by error injection or passing +# FORCE_REBUILD via ioctl. +__scratch_xfs_stress_setup_force_rebuild() { + local output="$($XFS_IO_PROG -x -c 'repair -R probe' $SCRATCH_MNT 2>&1)" + + if [ -z "$output" ]; then + export SCRUBSTRESS_USE_FORCE_REBUILD=1 + return + fi + + $XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT +} + # Start online repair, freeze, and fsstress in background looping processes, # and wait for 30*TIME_FACTOR seconds to see if the filesystem goes down. # Same requirements and arguments as _scratch_xfs_stress_scrub. _scratch_xfs_stress_online_repair() { - $XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT + __scratch_xfs_stress_setup_force_rebuild XFS_SCRUB_FORCE_REPAIR=1 _scratch_xfs_stress_scrub "$@" } ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 1/1] fuzzy: use FORCE_REBUILD over injecting force_repair 2022-12-30 22:19 ` [PATCH 1/1] fuzzy: use FORCE_REBUILD over injecting force_repair Darrick J. Wong @ 2023-02-14 8:00 ` Zorro Lang 2023-02-14 18:18 ` Darrick J. Wong 0 siblings, 1 reply; 107+ messages in thread From: Zorro Lang @ 2023-02-14 8:00 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:09PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > For stress testing online repair, try to use the FORCE_REBUILD ioctl > flag over the error injection knobs whenever possible because the knobs > are very noisy and are not always available. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- > common/fuzzy | 34 +++++++++++++++++++++++++++++++--- > 1 file changed, 31 insertions(+), 3 deletions(-) > > > diff --git a/common/fuzzy b/common/fuzzy > index f7f660bc31..14f7fdf03c 100644 > --- a/common/fuzzy > +++ b/common/fuzzy > @@ -398,6 +398,9 @@ __stress_one_scrub_loop() { > > local xfs_io_args=() > for arg in "$@"; do > + if [ -n "$SCRUBSTRESS_USE_FORCE_REBUILD" ]; then > + arg="$(echo "$arg" | sed -e 's/^repair/repair -R/g')" > + fi > if echo "$arg" | grep -q -w '%agno%'; then > # Substitute the AG number > for ((agno = 0; agno < agcount; agno++)); do > @@ -695,13 +698,21 @@ _require_xfs_stress_scrub() { > _notrun 'xfs scrub stress test requires common/filter' > } > > +# Make sure that we can force repairs either by error injection or passing > +# FORCE_REBUILD via ioctl. > +__require_xfs_stress_force_rebuild() { > + local output="$($XFS_IO_PROG -x -c 'repair -R probe' $SCRATCH_MNT 2>&1)" > + test -z "$output" && return > + _require_xfs_io_error_injection "force_repair" > +} > + > # Make sure we have everything we need to run stress and online repair > _require_xfs_stress_online_repair() { > _require_xfs_stress_scrub > _require_xfs_io_command "repair" > command -v _require_xfs_io_error_injection &>/dev/null || \ > _notrun 'xfs repair stress test requires common/inject' > - _require_xfs_io_error_injection "force_repair" > + __require_xfs_stress_force_rebuild > _require_freeze > } > > @@ -783,7 +794,11 @@ __stress_scrub_check_commands() { > esac > > for arg in "$@"; do > - local cooked_arg="$(echo "$arg" | sed -e "s/%agno%/0/g")" > + local cooked_arg="$arg" > + if [ -n "$SCRUBSTRESS_USE_FORCE_REBUILD" ]; then > + cooked_arg="$(echo "$cooked_arg" | sed -e 's/^repair/repair -R/g')" > + fi > + cooked_arg="$(echo "$cooked_arg" | sed -e "s/%agno%/0/g")" > testio=`$XFS_IO_PROG -x -c "$cooked_arg" "$cooked_tgt" 2>&1` > echo $testio | grep -q "Unknown type" && \ > _notrun "xfs_io scrub subcommand support is missing" > @@ -943,10 +958,23 @@ _scratch_xfs_stress_scrub() { > echo "Loop finished at $(date)" >> $seqres.full > } > > +# Decide if we're going to force repairs either by error injection or passing > +# FORCE_REBUILD via ioctl. > +__scratch_xfs_stress_setup_force_rebuild() { > + local output="$($XFS_IO_PROG -x -c 'repair -R probe' $SCRATCH_MNT 2>&1)" > + > + if [ -z "$output" ]; then > + export SCRUBSTRESS_USE_FORCE_REBUILD=1 Do you need to use this parameter ^^ in another child process? Is the "export" necessary? Thanks, Zorro > + return > + fi > + > + $XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT > +} > + > # Start online repair, freeze, and fsstress in background looping processes, > # and wait for 30*TIME_FACTOR seconds to see if the filesystem goes down. > # Same requirements and arguments as _scratch_xfs_stress_scrub. > _scratch_xfs_stress_online_repair() { > - $XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT > + __scratch_xfs_stress_setup_force_rebuild > XFS_SCRUB_FORCE_REPAIR=1 _scratch_xfs_stress_scrub "$@" > } > ^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH 1/1] fuzzy: use FORCE_REBUILD over injecting force_repair 2023-02-14 8:00 ` Zorro Lang @ 2023-02-14 18:18 ` Darrick J. Wong 2023-02-16 14:57 ` Zorro Lang 0 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2023-02-14 18:18 UTC (permalink / raw) To: Zorro Lang; +Cc: linux-xfs, fstests On Tue, Feb 14, 2023 at 04:00:07PM +0800, Zorro Lang wrote: > On Fri, Dec 30, 2022 at 02:19:09PM -0800, Darrick J. Wong wrote: > > From: Darrick J. Wong <djwong@kernel.org> > > > > For stress testing online repair, try to use the FORCE_REBUILD ioctl > > flag over the error injection knobs whenever possible because the knobs > > are very noisy and are not always available. > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > --- > > common/fuzzy | 34 +++++++++++++++++++++++++++++++--- > > 1 file changed, 31 insertions(+), 3 deletions(-) > > > > > > diff --git a/common/fuzzy b/common/fuzzy > > index f7f660bc31..14f7fdf03c 100644 > > --- a/common/fuzzy > > +++ b/common/fuzzy > > @@ -398,6 +398,9 @@ __stress_one_scrub_loop() { > > > > local xfs_io_args=() > > for arg in "$@"; do > > + if [ -n "$SCRUBSTRESS_USE_FORCE_REBUILD" ]; then > > + arg="$(echo "$arg" | sed -e 's/^repair/repair -R/g')" > > + fi > > if echo "$arg" | grep -q -w '%agno%'; then > > # Substitute the AG number > > for ((agno = 0; agno < agcount; agno++)); do > > @@ -695,13 +698,21 @@ _require_xfs_stress_scrub() { > > _notrun 'xfs scrub stress test requires common/filter' > > } > > > > +# Make sure that we can force repairs either by error injection or passing > > +# FORCE_REBUILD via ioctl. > > +__require_xfs_stress_force_rebuild() { > > + local output="$($XFS_IO_PROG -x -c 'repair -R probe' $SCRATCH_MNT 2>&1)" > > + test -z "$output" && return > > + _require_xfs_io_error_injection "force_repair" > > +} > > + > > # Make sure we have everything we need to run stress and online repair > > _require_xfs_stress_online_repair() { > > _require_xfs_stress_scrub > > _require_xfs_io_command "repair" > > command -v _require_xfs_io_error_injection &>/dev/null || \ > > _notrun 'xfs repair stress test requires common/inject' > > - _require_xfs_io_error_injection "force_repair" > > + __require_xfs_stress_force_rebuild > > _require_freeze > > } > > > > @@ -783,7 +794,11 @@ __stress_scrub_check_commands() { > > esac > > > > for arg in "$@"; do > > - local cooked_arg="$(echo "$arg" | sed -e "s/%agno%/0/g")" > > + local cooked_arg="$arg" > > + if [ -n "$SCRUBSTRESS_USE_FORCE_REBUILD" ]; then > > + cooked_arg="$(echo "$cooked_arg" | sed -e 's/^repair/repair -R/g')" > > + fi > > + cooked_arg="$(echo "$cooked_arg" | sed -e "s/%agno%/0/g")" > > testio=`$XFS_IO_PROG -x -c "$cooked_arg" "$cooked_tgt" 2>&1` > > echo $testio | grep -q "Unknown type" && \ > > _notrun "xfs_io scrub subcommand support is missing" > > @@ -943,10 +958,23 @@ _scratch_xfs_stress_scrub() { > > echo "Loop finished at $(date)" >> $seqres.full > > } > > > > +# Decide if we're going to force repairs either by error injection or passing > > +# FORCE_REBUILD via ioctl. > > +__scratch_xfs_stress_setup_force_rebuild() { > > + local output="$($XFS_IO_PROG -x -c 'repair -R probe' $SCRATCH_MNT 2>&1)" > > + > > + if [ -z "$output" ]; then > > + export SCRUBSTRESS_USE_FORCE_REBUILD=1 > > Do you need to use this parameter ^^ in another child process? Is the "export" > necessary? Nope, it's not required. --D > Thanks, > Zorro > > > + return > > + fi > > + > > + $XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT > > +} > > + > > # Start online repair, freeze, and fsstress in background looping processes, > > # and wait for 30*TIME_FACTOR seconds to see if the filesystem goes down. > > # Same requirements and arguments as _scratch_xfs_stress_scrub. > > _scratch_xfs_stress_online_repair() { > > - $XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT > > + __scratch_xfs_stress_setup_force_rebuild > > XFS_SCRUB_FORCE_REPAIR=1 _scratch_xfs_stress_scrub "$@" > > } > > > ^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH 1/1] fuzzy: use FORCE_REBUILD over injecting force_repair 2023-02-14 18:18 ` Darrick J. Wong @ 2023-02-16 14:57 ` Zorro Lang 0 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-16 14:57 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Tue, Feb 14, 2023 at 10:18:20AM -0800, Darrick J. Wong wrote: > On Tue, Feb 14, 2023 at 04:00:07PM +0800, Zorro Lang wrote: > > On Fri, Dec 30, 2022 at 02:19:09PM -0800, Darrick J. Wong wrote: > > > From: Darrick J. Wong <djwong@kernel.org> > > > > > > For stress testing online repair, try to use the FORCE_REBUILD ioctl > > > flag over the error injection knobs whenever possible because the knobs > > > are very noisy and are not always available. > > > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > > --- > > > common/fuzzy | 34 +++++++++++++++++++++++++++++++--- > > > 1 file changed, 31 insertions(+), 3 deletions(-) > > > > > > > > > diff --git a/common/fuzzy b/common/fuzzy > > > index f7f660bc31..14f7fdf03c 100644 > > > --- a/common/fuzzy > > > +++ b/common/fuzzy > > > @@ -398,6 +398,9 @@ __stress_one_scrub_loop() { > > > > > > local xfs_io_args=() > > > for arg in "$@"; do > > > + if [ -n "$SCRUBSTRESS_USE_FORCE_REBUILD" ]; then > > > + arg="$(echo "$arg" | sed -e 's/^repair/repair -R/g')" > > > + fi > > > if echo "$arg" | grep -q -w '%agno%'; then > > > # Substitute the AG number > > > for ((agno = 0; agno < agcount; agno++)); do > > > @@ -695,13 +698,21 @@ _require_xfs_stress_scrub() { > > > _notrun 'xfs scrub stress test requires common/filter' > > > } > > > > > > +# Make sure that we can force repairs either by error injection or passing > > > +# FORCE_REBUILD via ioctl. > > > +__require_xfs_stress_force_rebuild() { > > > + local output="$($XFS_IO_PROG -x -c 'repair -R probe' $SCRATCH_MNT 2>&1)" > > > + test -z "$output" && return > > > + _require_xfs_io_error_injection "force_repair" > > > +} > > > + > > > # Make sure we have everything we need to run stress and online repair > > > _require_xfs_stress_online_repair() { > > > _require_xfs_stress_scrub > > > _require_xfs_io_command "repair" > > > command -v _require_xfs_io_error_injection &>/dev/null || \ > > > _notrun 'xfs repair stress test requires common/inject' > > > - _require_xfs_io_error_injection "force_repair" > > > + __require_xfs_stress_force_rebuild > > > _require_freeze > > > } > > > > > > @@ -783,7 +794,11 @@ __stress_scrub_check_commands() { > > > esac > > > > > > for arg in "$@"; do > > > - local cooked_arg="$(echo "$arg" | sed -e "s/%agno%/0/g")" > > > + local cooked_arg="$arg" > > > + if [ -n "$SCRUBSTRESS_USE_FORCE_REBUILD" ]; then > > > + cooked_arg="$(echo "$cooked_arg" | sed -e 's/^repair/repair -R/g')" > > > + fi > > > + cooked_arg="$(echo "$cooked_arg" | sed -e "s/%agno%/0/g")" > > > testio=`$XFS_IO_PROG -x -c "$cooked_arg" "$cooked_tgt" 2>&1` > > > echo $testio | grep -q "Unknown type" && \ > > > _notrun "xfs_io scrub subcommand support is missing" > > > @@ -943,10 +958,23 @@ _scratch_xfs_stress_scrub() { > > > echo "Loop finished at $(date)" >> $seqres.full > > > } > > > > > > +# Decide if we're going to force repairs either by error injection or passing > > > +# FORCE_REBUILD via ioctl. > > > +__scratch_xfs_stress_setup_force_rebuild() { > > > + local output="$($XFS_IO_PROG -x -c 'repair -R probe' $SCRATCH_MNT 2>&1)" > > > + > > > + if [ -z "$output" ]; then > > > + export SCRUBSTRESS_USE_FORCE_REBUILD=1 > > > > Do you need to use this parameter ^^ in another child process? Is the "export" > > necessary? > > Nope, it's not required. OK, others looks good to me, will merge this patch without that "export". Reviewed-by: Zorro Lang <zlang@redhat.com> > > --D > > > Thanks, > > Zorro > > > > > + return > > > + fi > > > + > > > + $XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT > > > +} > > > + > > > # Start online repair, freeze, and fsstress in background looping processes, > > > # and wait for 30*TIME_FACTOR seconds to see if the filesystem goes down. > > > # Same requirements and arguments as _scratch_xfs_stress_scrub. > > > _scratch_xfs_stress_online_repair() { > > > - $XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT > > > + __scratch_xfs_stress_setup_force_rebuild > > > XFS_SCRUB_FORCE_REPAIR=1 _scratch_xfs_stress_scrub "$@" > > > } > > > > > > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/2] fstests: online repair of AG btrees 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] xfs: force rebuilding of metadata Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] xfs: stress test ag repair functions Darrick J. Wong ` (2 more replies) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of inodes Darrick J. Wong ` (14 subsequent siblings) 17 siblings, 3 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, Now that we've spent a lot of time reworking common code in online fsck, we're ready to start rebuilding the AG space btrees. This series implements repair functions for the free space, inode, and refcount btrees. Rebuilding the reverse mapping btree is much more intense and is left for a subsequent patchset. The fstests counterpart of this patchset implements stress testing of repair. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-ag-btrees fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-ag-btrees --- README | 3 ++ common/fuzzy | 39 +++++++++++++++++++-------- common/rc | 2 + common/xfs | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/725 | 37 +++++++++++++++++++++++++ tests/xfs/725.out | 2 + tests/xfs/726 | 37 +++++++++++++++++++++++++ tests/xfs/726.out | 2 + tests/xfs/727 | 38 ++++++++++++++++++++++++++ tests/xfs/727.out | 2 + tests/xfs/728 | 37 +++++++++++++++++++++++++ tests/xfs/728.out | 2 + tests/xfs/729 | 37 +++++++++++++++++++++++++ tests/xfs/729.out | 2 + tests/xfs/730 | 37 +++++++++++++++++++++++++ tests/xfs/730.out | 2 + tests/xfs/731 | 37 +++++++++++++++++++++++++ tests/xfs/731.out | 2 + 18 files changed, 382 insertions(+), 13 deletions(-) create mode 100755 tests/xfs/725 create mode 100644 tests/xfs/725.out create mode 100755 tests/xfs/726 create mode 100644 tests/xfs/726.out create mode 100755 tests/xfs/727 create mode 100644 tests/xfs/727.out create mode 100755 tests/xfs/728 create mode 100644 tests/xfs/728.out create mode 100755 tests/xfs/729 create mode 100644 tests/xfs/729.out create mode 100755 tests/xfs/730 create mode 100644 tests/xfs/730.out create mode 100755 tests/xfs/731 create mode 100644 tests/xfs/731.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 2/2] xfs: stress test ag repair functions 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] xfs: test rebuilding the entire filesystem with online fsck Darrick J. Wong 2023-02-18 6:06 ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Zorro Lang 2 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Race fsstress and various AG repair functions. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 38 ++++++++++++++++++++++++++------------ tests/xfs/725 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/725.out | 2 ++ tests/xfs/726 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/726.out | 2 ++ tests/xfs/727 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/727.out | 2 ++ tests/xfs/728 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/728.out | 2 ++ tests/xfs/729 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/729.out | 2 ++ tests/xfs/730 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/730.out | 2 ++ tests/xfs/731 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/731.out | 2 ++ 15 files changed, 300 insertions(+), 12 deletions(-) create mode 100755 tests/xfs/725 create mode 100644 tests/xfs/725.out create mode 100755 tests/xfs/726 create mode 100644 tests/xfs/726.out create mode 100755 tests/xfs/727 create mode 100644 tests/xfs/727.out create mode 100755 tests/xfs/728 create mode 100644 tests/xfs/728.out create mode 100755 tests/xfs/729 create mode 100644 tests/xfs/729.out create mode 100755 tests/xfs/730 create mode 100644 tests/xfs/730.out create mode 100755 tests/xfs/731 create mode 100644 tests/xfs/731.out diff --git a/common/fuzzy b/common/fuzzy index d8de55250d..d4177c3136 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -393,7 +393,8 @@ __stress_one_scrub_loop() { local runningfile="$2" local scrub_tgt="$3" local scrub_startat="$4" - shift; shift; shift; shift + local start_agno="$5" + shift; shift; shift; shift; shift local agcount="$(_xfs_mount_agcount $SCRATCH_MNT)" local xfs_io_args=() @@ -403,7 +404,7 @@ __stress_one_scrub_loop() { fi if echo "$arg" | grep -q -w '%agno%'; then # Substitute the AG number - for ((agno = 0; agno < agcount; agno++)); do + for ((agno = start_agno; agno < agcount; agno++)); do local ag_arg="$(echo "$arg" | sed -e "s|%agno%|$agno|g")" xfs_io_args+=('-c' "$ag_arg") done @@ -413,28 +414,34 @@ __stress_one_scrub_loop() { done local extra_filters=() - local target_cmd=(echo "$scrub_tgt") case "$scrub_tgt" in "%file%"|"%datafile%"|"%attrfile%") extra_filters+=('No such file or directory' 'No such device or address') - target_cmd=(find "$SCRATCH_MNT" -print) ;; "%dir%") extra_filters+=('No such file or directory' 'Not a directory') - target_cmd=(find "$SCRATCH_MNT" -type d -print) ;; "%regfile%"|"%cowfile%") extra_filters+=('No such file or directory') - target_cmd=(find "$SCRATCH_MNT" -type f -print) ;; esac + local target_cmd=(echo "$scrub_tgt") + case "$scrub_tgt" in + "%file%") target_cmd=($here/src/xfsfind -q "$SCRATCH_MNT");; + "%attrfile%") target_cmd=($here/src/xfsfind -qa "$SCRATCH_MNT");; + "%datafile%") target_cmd=($here/src/xfsfind -qb "$SCRATCH_MNT");; + "%dir%") target_cmd=($here/src/xfsfind -qd "$SCRATCH_MNT");; + "%regfile%") target_cmd=($here/src/xfsfind -qr "$SCRATCH_MNT");; + "%cowfile%") target_cmd=($here/src/xfsfind -qs "$SCRATCH_MNT");; + esac + while __stress_scrub_running "$scrub_startat" "$runningfile"; do sleep 1 done while __stress_scrub_running "$end" "$runningfile"; do - readarray -t fnames < <("${target_cmd[@]}" 2>/dev/null) + readarray -t fnames < <("${target_cmd[@]}" 2>> $seqres.full) for fname in "${fnames[@]}"; do $XFS_IO_PROG -x "${xfs_io_args[@]}" "$fname" 2>&1 | \ __stress_scrub_filter_output "${extra_filters[@]}" @@ -692,6 +699,7 @@ __stress_scrub_fsstress_loop() { # Make sure we have everything we need to run stress and scrub _require_xfs_stress_scrub() { _require_xfs_io_command "scrub" + _require_test_program "xfsfind" _require_command "$KILLALL_PROG" killall _require_freeze command -v _filter_scratch &>/dev/null || \ @@ -769,7 +777,8 @@ _scratch_xfs_stress_scrub_cleanup() { # filesystem before we start running them in a loop. __stress_scrub_check_commands() { local scrub_tgt="$1" - shift + local start_agno="$2" + shift; shift local cooked_tgt="$scrub_tgt" case "$scrub_tgt" in @@ -798,7 +807,7 @@ __stress_scrub_check_commands() { if [ -n "$SCRUBSTRESS_USE_FORCE_REBUILD" ]; then cooked_arg="$(echo "$cooked_arg" | sed -e 's/^repair/repair -R/g')" fi - cooked_arg="$(echo "$cooked_arg" | sed -e "s/%agno%/0/g")" + cooked_arg="$(echo "$cooked_arg" | sed -e "s/%agno%/$start_agno/g")" testio=`$XFS_IO_PROG -x -c "$cooked_arg" "$cooked_tgt" 2>&1` echo $testio | grep -q "Unknown type" && \ _notrun "xfs_io scrub subcommand support is missing" @@ -817,6 +826,7 @@ __stress_scrub_check_commands() { # # Various options include: # +# -a For %agno% substitution, start with this AG instead of AG 0. # -f Run a freeze/thaw loop while we're doing other things. Defaults to # disabled, unless XFS_SCRUB_STRESS_FREEZE is set. # -i Pass this command to xfs_io to exercise something that is not scrub @@ -867,6 +877,7 @@ _scratch_xfs_stress_scrub() { local io_args=() local remount_period="${XFS_SCRUB_STRESS_REMOUNT_PERIOD}" local stress_tgt="${XFS_SCRUB_STRESS_TARGET:-default}" + local start_agno=0 __SCRUB_STRESS_FREEZE_PID="" __SCRUB_STRESS_REMOUNT_LOOP="" @@ -874,8 +885,9 @@ _scratch_xfs_stress_scrub() { touch "$runningfile" OPTIND=1 - while getopts "fi:r:s:S:t:w:x:X:" c; do + while getopts "a:fi:r:s:S:t:w:x:X:" c; do case "$c" in + a) start_agno="$OPTARG";; f) freeze=yes;; i) io_args+=("$OPTARG");; r) remount_period="$OPTARG";; @@ -889,7 +901,8 @@ _scratch_xfs_stress_scrub() { esac done - __stress_scrub_check_commands "$scrub_tgt" "${one_scrub_args[@]}" + __stress_scrub_check_commands "$scrub_tgt" "$start_agno" \ + "${one_scrub_args[@]}" if ! command -v "__stress_scrub_${exerciser}_loop" &>/dev/null; then echo "${exerciser}: Unknown fs exercise program." @@ -936,7 +949,8 @@ _scratch_xfs_stress_scrub() { if [ "${#one_scrub_args[@]}" -gt 0 ]; then __stress_one_scrub_loop "$end" "$runningfile" "$scrub_tgt" \ - "$scrub_startat" "${one_scrub_args[@]}" & + "$scrub_startat" "$start_agno" \ + "${one_scrub_args[@]}" & fi if [ "${#xfs_scrub_args[@]}" -gt 0 ]; then diff --git a/tests/xfs/725 b/tests/xfs/725 new file mode 100755 index 0000000000..8466b4a77f --- /dev/null +++ b/tests/xfs/725 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 725 +# +# Race fsstress and bnobt repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -s "repair bnobt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/725.out b/tests/xfs/725.out new file mode 100644 index 0000000000..128709eb38 --- /dev/null +++ b/tests/xfs/725.out @@ -0,0 +1,2 @@ +QA output created by 725 +Silence is golden diff --git a/tests/xfs/726 b/tests/xfs/726 new file mode 100755 index 0000000000..4f34c69ba4 --- /dev/null +++ b/tests/xfs/726 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 725 +# +# Race fsstress and inobt repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -s "repair inobt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/726.out b/tests/xfs/726.out new file mode 100644 index 0000000000..40767062d2 --- /dev/null +++ b/tests/xfs/726.out @@ -0,0 +1,2 @@ +QA output created by 726 +Silence is golden diff --git a/tests/xfs/727 b/tests/xfs/727 new file mode 100755 index 0000000000..d16bb3ece2 --- /dev/null +++ b/tests/xfs/727 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 725 +# +# Race fsstress and refcountbt repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" reflink +_scratch_xfs_stress_online_repair -s "repair refcountbt %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/727.out b/tests/xfs/727.out new file mode 100644 index 0000000000..2de2b4b2ce --- /dev/null +++ b/tests/xfs/727.out @@ -0,0 +1,2 @@ +QA output created by 727 +Silence is golden diff --git a/tests/xfs/728 b/tests/xfs/728 new file mode 100755 index 0000000000..f0dd536d49 --- /dev/null +++ b/tests/xfs/728 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 728 +# +# Race fsstress and superblock repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -a 1 -s "repair sb %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/728.out b/tests/xfs/728.out new file mode 100644 index 0000000000..ab39f45fe5 --- /dev/null +++ b/tests/xfs/728.out @@ -0,0 +1,2 @@ +QA output created by 728 +Silence is golden diff --git a/tests/xfs/729 b/tests/xfs/729 new file mode 100755 index 0000000000..85d53b5f0b --- /dev/null +++ b/tests/xfs/729 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 729 +# +# Race fsstress and agf repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -s "repair agf %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/729.out b/tests/xfs/729.out new file mode 100644 index 0000000000..0f175ae2f9 --- /dev/null +++ b/tests/xfs/729.out @@ -0,0 +1,2 @@ +QA output created by 729 +Silence is golden diff --git a/tests/xfs/730 b/tests/xfs/730 new file mode 100755 index 0000000000..a452016bb1 --- /dev/null +++ b/tests/xfs/730 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 730 +# +# Race fsstress and agfl repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -s "repair agfl %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/730.out b/tests/xfs/730.out new file mode 100644 index 0000000000..50c3c832f0 --- /dev/null +++ b/tests/xfs/730.out @@ -0,0 +1,2 @@ +QA output created by 730 +Silence is golden diff --git a/tests/xfs/731 b/tests/xfs/731 new file mode 100755 index 0000000000..7d0492a10d --- /dev/null +++ b/tests/xfs/731 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 731 +# +# Race fsstress and agi repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -s "repair agi %agno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/731.out b/tests/xfs/731.out new file mode 100644 index 0000000000..93b1b2692d --- /dev/null +++ b/tests/xfs/731.out @@ -0,0 +1,2 @@ +QA output created by 731 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 1/2] xfs: test rebuilding the entire filesystem with online fsck 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] xfs: stress test ag repair functions Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-18 6:06 ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Zorro Lang 2 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add a new knob, TEST_XFS_SCRUB_REBUILD, that makes it so that we use xfs_scrub to rebuild the ondisk metadata after every test. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- README | 3 ++ common/fuzzy | 1 + common/rc | 2 +- common/xfs | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 82 insertions(+), 1 deletion(-) diff --git a/README b/README index 4c4f22f853..744317625f 100644 --- a/README +++ b/README @@ -191,6 +191,9 @@ Extra XFS specification: to check the filesystem. As of August 2021, xfs_repair finds all filesystem corruptions found by xfs_check, and more, which means that xfs_check is no longer run by default. + - Set TEST_XFS_SCRUB_REBUILD=1 to have _check_xfs_filesystem run xfs_scrub in + "force_repair" mode to rebuild the filesystem; and xfs_repair -n to check + the results of the rebuilding. - xfs_scrub, if present, will always check the test and scratch filesystems if they are still online at the end of the test. It is no longer necessary to set TEST_XFS_SCRUB. diff --git a/common/fuzzy b/common/fuzzy index 14f7fdf03c..d8de55250d 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -975,6 +975,7 @@ __scratch_xfs_stress_setup_force_rebuild() { # and wait for 30*TIME_FACTOR seconds to see if the filesystem goes down. # Same requirements and arguments as _scratch_xfs_stress_scrub. _scratch_xfs_stress_online_repair() { + touch "$RESULT_DIR/.skip_orebuild" # no need to test online rebuild __scratch_xfs_stress_setup_force_rebuild XFS_SCRUB_FORCE_REPAIR=1 _scratch_xfs_stress_scrub "$@" } diff --git a/common/rc b/common/rc index 23530413ec..a1b65f0a7f 100644 --- a/common/rc +++ b/common/rc @@ -1685,7 +1685,7 @@ _require_scratch_nocheck() exit 1 fi fi - rm -f ${RESULT_DIR}/require_scratch + rm -f ${RESULT_DIR}/require_scratch "$RESULT_DIR/.skip_orebuild" } # we need the scratch device and it needs to not be an lvm device diff --git a/common/xfs b/common/xfs index 436569ba28..804047557b 100644 --- a/common/xfs +++ b/common/xfs @@ -692,6 +692,8 @@ _scratch_xfs_mdrestore() # run xfs_check and friends on a FS. _check_xfs_filesystem() { + local can_scrub= + if [ $# -ne 3 ]; then echo "Usage: _check_xfs_filesystem device <logdev>|none <rtdev>|none" 1>&2 exit 1 @@ -726,6 +728,8 @@ _check_xfs_filesystem() # Run online scrub if we can. mntpt="$(_is_dev_mounted $device)" if [ -n "$mntpt" ] && _supports_xfs_scrub "$mntpt" "$device"; then + can_scrub=1 + # Tests can create a scenario in which a call to syncfs() issued # at the end of the execution of the test script would return an # error code. xfs_scrub internally calls syncfs() before @@ -842,6 +846,79 @@ _check_xfs_filesystem() _mount_or_remount_rw "$extra_mount_options" $device $mountpoint fi + # If desired, test the online metadata rebuilding behavior if the + # filesystem was mounted when this function was called. + if [ -n "$TEST_XFS_SCRUB_REBUILD" ] && [ -n "$can_scrub" ] && [ ! -e "$RESULT_DIR/.skip_orebuild" ]; then + orebuild_ok=1 + + # Walk the entire directory tree to load directory blocks into + # memory and populate the dentry cache, which can speed up the + # repairs considerably when the directory tree is very large. + find $mntpt &>/dev/null & + + XFS_SCRUB_FORCE_REPAIR=1 "$XFS_SCRUB_PROG" -v -d $mntpt > $tmp.scrub 2>&1 + if [ $? -ne 0 ]; then + if grep -q 'No space left on device' $tmp.scrub; then + # It's not an error if the fs does not have + # enough space to complete a repair. We will + # check everything, though. + echo "*** XFS_SCRUB_FORCE_REPAIR=1 xfs_scrub -v -d ran out of space ***" >> $seqres.full + cat $tmp.scrub >> $seqres.full + echo "*** end xfs_scrub output" >> $seqres.full + else + _log_err "_check_xfs_filesystem: filesystem on $device failed scrub orebuild" + echo "*** XFS_SCRUB_FORCE_REPAIR=1 xfs_scrub -v -d output ***" >> $seqres.full + cat $tmp.scrub >> $seqres.full + echo "*** end xfs_scrub output" >> $seqres.full + ok=0 + orebuild_ok=0 + fi + fi + rm -f $tmp.scrub + + # Clear force_repair because xfs_scrub could have set it + $XFS_IO_PROG -x -c 'inject noerror' "$mntpt" >> $seqres.full + + "$XFS_SCRUB_PROG" -v -d -n $mntpt > $tmp.scrub 2>&1 + if [ $? -ne 0 ]; then + _log_err "_check_xfs_filesystem: filesystem on $device failed scrub orebuild recheck" + echo "*** xfs_scrub -v -d -n output ***" >> $seqres.full + cat $tmp.scrub >> $seqres.full + echo "*** end xfs_scrub output" >> $seqres.full + ok=0 + orebuild_ok=0 + fi + rm -f $tmp.scrub + + mountpoint=`_umount_or_remount_ro $device` + + $XFS_REPAIR_PROG -n $extra_options $extra_log_options $extra_rt_options $device >$tmp.repair 2>&1 + if [ $? -ne 0 ]; then + _log_err "_check_xfs_filesystem: filesystem on $device is inconsistent (orebuild-reverify)" + echo "*** xfs_repair -n output ***" >>$seqres.full + cat $tmp.repair >>$seqres.full + echo "*** end xfs_repair output" >>$seqres.full + + ok=0 + orebuild_ok=0 + fi + rm -f $tmp.repair + + if [ $ok -eq 0 ]; then + echo "*** mount output ***" >>$seqres.full + _mount >>$seqres.full + echo "*** end mount output" >>$seqres.full + elif [ "$type" = "xfs" ]; then + _mount_or_remount_rw "$extra_mount_options" $device $mountpoint + fi + + if [ "$orebuild_ok" -ne 1 ] && [ "$DUMP_CORRUPT_FS" = "1" ]; then + local flatdev="$(basename "$device")" + _xfs_metadump "$seqres.$flatdev.orebuild.md" "$device" \ + "$logdev" compress >> $seqres.full + fi + fi + if [ $ok -eq 0 ]; then status=1 if [ "$iam" != "check" ]; then ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCHSET v24.0 0/2] fstests: online repair of AG btrees 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] xfs: stress test ag repair functions Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] xfs: test rebuilding the entire filesystem with online fsck Darrick J. Wong @ 2023-02-18 6:06 ` Zorro Lang 2 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-18 6:06 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:12PM -0800, Darrick J. Wong wrote: > Hi all, > > Now that we've spent a lot of time reworking common code in online fsck, > we're ready to start rebuilding the AG space btrees. This series > implements repair functions for the free space, inode, and refcount > btrees. Rebuilding the reverse mapping btree is much more intense and > is left for a subsequent patchset. The fstests counterpart of this > patchset implements stress testing of repair. > > If you're going to start using this mess, you probably ought to just > pull from my git trees, which are linked below. > > This is an extraordinary way to destroy everything. Enjoy! > Comments and questions are, as always, welcome. > > --D LGTM, Reviewed-by: Zorro Lang <zlang@redhat.com> > > kernel git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-ag-btrees > > xfsprogs git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-ag-btrees > > fstests git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-ag-btrees > --- > README | 3 ++ > common/fuzzy | 39 +++++++++++++++++++-------- > common/rc | 2 + > common/xfs | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > tests/xfs/725 | 37 +++++++++++++++++++++++++ > tests/xfs/725.out | 2 + > tests/xfs/726 | 37 +++++++++++++++++++++++++ > tests/xfs/726.out | 2 + > tests/xfs/727 | 38 ++++++++++++++++++++++++++ > tests/xfs/727.out | 2 + > tests/xfs/728 | 37 +++++++++++++++++++++++++ > tests/xfs/728.out | 2 + > tests/xfs/729 | 37 +++++++++++++++++++++++++ > tests/xfs/729.out | 2 + > tests/xfs/730 | 37 +++++++++++++++++++++++++ > tests/xfs/730.out | 2 + > tests/xfs/731 | 37 +++++++++++++++++++++++++ > tests/xfs/731.out | 2 + > 18 files changed, 382 insertions(+), 13 deletions(-) > create mode 100755 tests/xfs/725 > create mode 100644 tests/xfs/725.out > create mode 100755 tests/xfs/726 > create mode 100644 tests/xfs/726.out > create mode 100755 tests/xfs/727 > create mode 100644 tests/xfs/727.out > create mode 100755 tests/xfs/728 > create mode 100644 tests/xfs/728.out > create mode 100755 tests/xfs/729 > create mode 100644 tests/xfs/729.out > create mode 100755 tests/xfs/730 > create mode 100644 tests/xfs/730.out > create mode 100755 tests/xfs/731 > create mode 100644 tests/xfs/731.out > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/1] fstests: online repair of inodes 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online repair for inode record metadata Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Darrick J. Wong ` (13 subsequent siblings) 17 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, In this series, online repair gains the ability to repair inode records. To do this, we must repair the ondisk inode and fork information enough to pass the iget verifiers and hence make the inode igettable again. Once that's done, we can perform higher level repairs on the incore inode. The fstests counterpart of this patchset implements stress testing of repair. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-inodes xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-inodes fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-inodes --- tests/xfs/806 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/806.out | 2 ++ 2 files changed, 40 insertions(+) create mode 100755 tests/xfs/806 create mode 100644 tests/xfs/806.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/1] xfs: race fsstress with online repair for inode record metadata 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of inodes Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-18 6:07 ` Zorro Lang 0 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create a test that runs the inode record repairer in the foreground and fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/806 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/806.out | 2 ++ 2 files changed, 40 insertions(+) create mode 100755 tests/xfs/806 create mode 100644 tests/xfs/806.out diff --git a/tests/xfs/806 b/tests/xfs/806 new file mode 100755 index 0000000000..e07f9f9141 --- /dev/null +++ b/tests/xfs/806 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 806 +# +# Race fsstress and inode record repair for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -s "repair inode" -t "%file%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/806.out b/tests/xfs/806.out new file mode 100644 index 0000000000..463bd7f008 --- /dev/null +++ b/tests/xfs/806.out @@ -0,0 +1,2 @@ +QA output created by 806 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 1/1] xfs: race fsstress with online repair for inode record metadata 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online repair for inode record metadata Darrick J. Wong @ 2023-02-18 6:07 ` Zorro Lang 0 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-18 6:07 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:15PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Create a test that runs the inode record repairer in the foreground and > fsstress in the background. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- LGTM, Reviewed-by: Zorro Lang <zlang@redhat.com> > tests/xfs/806 | 38 ++++++++++++++++++++++++++++++++++++++ > tests/xfs/806.out | 2 ++ > 2 files changed, 40 insertions(+) > create mode 100755 tests/xfs/806 > create mode 100644 tests/xfs/806.out > > > diff --git a/tests/xfs/806 b/tests/xfs/806 > new file mode 100755 > index 0000000000..e07f9f9141 > --- /dev/null > +++ b/tests/xfs/806 > @@ -0,0 +1,38 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 806 > +# > +# Race fsstress and inode record repair for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest online_repair dangerous_fsstress_repair > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_online_repair > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_online_repair -s "repair inode" -t "%file%" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/806.out b/tests/xfs/806.out > new file mode 100644 > index 0000000000..463bd7f008 > --- /dev/null > +++ b/tests/xfs/806.out > @@ -0,0 +1,2 @@ > +QA output created by 806 > +Silence is golden > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of inodes Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 4/4] xfs: race fsstress with online repair for special file metadata Darrick J. Wong ` (4 more replies) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of quota and counters Darrick J. Wong ` (12 subsequent siblings) 17 siblings, 5 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, In this series, online repair gains the ability to rebuild data and attr fork mappings from the reverse mapping information. It is at this point where we reintroduce the ability to reap file extents. Repair of CoW forks is a little different -- on disk, CoW staging extents are owned by the refcount btree and cannot be mapped back to individual files. Hence we can only detect staging extents that don't quite look right (missing reverse mappings, shared staging extents) and replace them with fresh allocations. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-file-mappings xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-file-mappings fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-file-mappings --- tests/xfs/746 | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/746.out | 2 + tests/xfs/807 | 37 +++++++++++++++++++++++ tests/xfs/807.out | 2 + tests/xfs/808 | 39 ++++++++++++++++++++++++ tests/xfs/808.out | 2 + tests/xfs/828 | 38 ++++++++++++++++++++++++ tests/xfs/828.out | 2 + tests/xfs/829 | 39 ++++++++++++++++++++++++ tests/xfs/829.out | 2 + tests/xfs/840 | 72 +++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/840.out | 3 ++ tests/xfs/846 | 39 ++++++++++++++++++++++++ tests/xfs/846.out | 2 + 14 files changed, 364 insertions(+) create mode 100755 tests/xfs/746 create mode 100644 tests/xfs/746.out create mode 100755 tests/xfs/807 create mode 100644 tests/xfs/807.out create mode 100755 tests/xfs/808 create mode 100644 tests/xfs/808.out create mode 100755 tests/xfs/828 create mode 100644 tests/xfs/828.out create mode 100755 tests/xfs/829 create mode 100644 tests/xfs/829.out create mode 100755 tests/xfs/840 create mode 100644 tests/xfs/840.out create mode 100755 tests/xfs/846 create mode 100644 tests/xfs/846.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 4/4] xfs: race fsstress with online repair for special file metadata 2022-12-30 22:19 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/4] xfs: test rebuilding xattrs when the data fork is btree format Darrick J. Wong ` (3 subsequent siblings) 4 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> For each XFS_SCRUB_TYPE_* that looks at symbolic link and special file metadata, create a test that runs that repairer in the foreground and fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/828 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/828.out | 2 ++ tests/xfs/829 | 39 +++++++++++++++++++++++++++++++++++++++ tests/xfs/829.out | 2 ++ 4 files changed, 81 insertions(+) create mode 100755 tests/xfs/828 create mode 100644 tests/xfs/828.out create mode 100755 tests/xfs/829 create mode 100644 tests/xfs/829.out diff --git a/tests/xfs/828 b/tests/xfs/828 new file mode 100755 index 0000000000..99020e9b3c --- /dev/null +++ b/tests/xfs/828 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 828 +# +# Race fsstress and symlink repair for a while to see if we crash or livelock. +# We can't open special files directly for scrubbing, so we use xfs_scrub(8). +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +XFS_SCRUB_PHASE=3 _scratch_xfs_stress_online_repair -x 'symlink' -S '-k' + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/828.out b/tests/xfs/828.out new file mode 100644 index 0000000000..d0291290a9 --- /dev/null +++ b/tests/xfs/828.out @@ -0,0 +1,2 @@ +QA output created by 828 +Silence is golden diff --git a/tests/xfs/829 b/tests/xfs/829 new file mode 100755 index 0000000000..7451f66069 --- /dev/null +++ b/tests/xfs/829 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 827 +# +# Race fsstress and special file repair for a while to see if we crash or +# livelock. We can't open special files directly for scrubbing, so we use +# xfs_scrub(8). +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +XFS_SCRUB_PHASE=3 _scratch_xfs_stress_online_repair -x 'mknod' -S '-k' + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/829.out b/tests/xfs/829.out new file mode 100644 index 0000000000..8a43a15204 --- /dev/null +++ b/tests/xfs/829.out @@ -0,0 +1,2 @@ +QA output created by 829 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 1/4] xfs: test rebuilding xattrs when the data fork is btree format 2022-12-30 22:19 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Darrick J. Wong 2022-12-30 22:19 ` [PATCH 4/4] xfs: race fsstress with online repair for special file metadata Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 3/4] xfs: ensure that online file data fork repairs don't hit EDQUOT Darrick J. Wong ` (2 subsequent siblings) 4 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Make sure we handle the case of rebuilding extended attributes properly when the data fork is in btree format and we therefore cannot zap the attr fork. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/746 | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/746.out | 2 + 2 files changed, 87 insertions(+) create mode 100755 tests/xfs/746 create mode 100644 tests/xfs/746.out diff --git a/tests/xfs/746 b/tests/xfs/746 new file mode 100755 index 0000000000..5853259e84 --- /dev/null +++ b/tests/xfs/746 @@ -0,0 +1,85 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 746 +# +# Make sure online repair can handle rebuilding xattrs when the data fork is +# in btree format and we cannot just zap the attr fork. + +. ./common/preamble +_begin_fstest auto quick online_repair + +# Override the default cleanup function. +_cleanup() +{ + cd / +} + +# Import common functions. +. ./common/inject +. ./common/filter + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_io_error_injection "force_repair" +_require_xfs_io_command "falloc" +_require_xfs_io_command "repair" +_require_test_program "punch-alternating" + +_scratch_mkfs > $tmp.mkfs +_scratch_mount + +_supports_xfs_scrub $SCRATCH_MNT $SCRATCH_DEV || _notrun "Scrub not supported" + +# Force data device extents so that we can create a file with the exact bmbt +# that we need regardless of rt configuration. +_xfs_force_bdev data $SCRATCH_MNT + +file=$SCRATCH_MNT/moofile +touch $file + +# Create some xattrs so that we have to rebuild them. +$XFS_IO_PROG -f -c 'pwrite -S 0x58 0 76' $file.txt >> $seqres.full +$SETFATTR_PROG -n user.SGI_BCL_FILE -v "$(cat $file.txt)" $file + +$SETFATTR_PROG -n user.crtime_usec -v 12345678 $file + +blksz=$(_get_file_block_size $SCRATCH_MNT) +ino=$(stat -c '%i' $file) + +# Figure out how many extents we need to have to create a data fork that's in +# btree format. +umount $SCRATCH_MNT +di_forkoff=$(_scratch_xfs_db -c "inode $ino" -c "p core.forkoff" | \ + awk '{print $3}') +_scratch_xfs_db -c "inode $ino" -c "p" >> $seqres.full +_scratch_mount + +# Create a data fork in btree format +min_ext_for_btree=$((di_forkoff * 8 / 16)) +$XFS_IO_PROG -c "falloc 0 $(( (min_ext_for_btree + 1) * 2 * blksz))" $file +$here/src/punch-alternating $file + +# Make sure the data fork is in btree format. +umount $SCRATCH_MNT +_scratch_xfs_db -c "inode $ino" -c "p core.format" | grep -q "btree" || \ + echo "data fork not in btree format?" +echo "about to start test" >> $seqres.full +_scratch_xfs_db -c "inode $ino" -c "p" >> $seqres.full +_scratch_mount + +# Force repair the xattr fork +_scratch_inject_error force_repair +$XFS_IO_PROG -x -c 'repair xattr' $file 2>&1 | tee $tmp.repair.log +grep -q 'Operation not supported' $tmp.repair.log && \ + _notrun "online xattr repair not supported" + +# If online repair did it correctly, the filesystem won't be corrupt. Let the +# post-test check do its thing. + +# success, all done +echo "Silence is golden." +status=0 +exit diff --git a/tests/xfs/746.out b/tests/xfs/746.out new file mode 100644 index 0000000000..365485b0b3 --- /dev/null +++ b/tests/xfs/746.out @@ -0,0 +1,2 @@ +QA output created by 746 +Silence is golden. ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 3/4] xfs: ensure that online file data fork repairs don't hit EDQUOT 2022-12-30 22:19 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Darrick J. Wong 2022-12-30 22:19 ` [PATCH 4/4] xfs: race fsstress with online repair for special file metadata Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/4] xfs: test rebuilding xattrs when the data fork is btree format Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/4] xfs: race fsstress with online repair for inode and fork metadata Darrick J. Wong 2023-02-18 6:07 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Zorro Lang 4 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add a test to ensure that the sysadmin doesn't get EDQUOT if they try to repair file data fork metadata when we've already exceeded a quota limit somewhere. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/840 | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/840.out | 3 ++ 2 files changed, 75 insertions(+) create mode 100755 tests/xfs/840 create mode 100644 tests/xfs/840.out diff --git a/tests/xfs/840 b/tests/xfs/840 new file mode 100755 index 0000000000..fff41c5b8a --- /dev/null +++ b/tests/xfs/840 @@ -0,0 +1,72 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test 840 +# +# Ensure that the sysadmin won't hit EDQUOT while repairing file data forks +# even if the file's quota limits have been exceeded. This tests the quota +# reservation handling inside the bmap btree rebuilding code. +# +. ./common/preamble +_begin_fstest online_repair + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/quota + +# real QA test starts here + +# Modify as appropriate. +_supported_fs xfs +_require_xfs_io_command "falloc" +_require_quota +_require_user +_require_test_program "punch-alternating" +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_qmount_option usrquota +_qmount + +blocksize=$(_get_block_size $SCRATCH_MNT) +alloc_unit=$(_get_file_block_size $SCRATCH_MNT) + +# Make sure we can actually repair a data fork +scratchfile=$SCRATCH_MNT/file +touch $scratchfile +$XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT +__stress_scrub_check_commands "$scratchfile" "" 'repair bmapbtd' + +# Compute the number of extent records needed to guarantee btree format, +# assuming 16 bytes for each ondisk extent record +bmbt_records=$(( (blocksize / 16) * 5 / 4 )) +total_size=$(( bmbt_records * 2 * alloc_unit )) + +# Create a file with a data fork in bmap btree format +$XFS_IO_PROG -c "falloc 0 $total_size" $scratchfile >> $seqres.full +$here/src/punch-alternating $scratchfile + +# Set a low quota hardlimit for an unprivileged uid and chown the file to it +echo "set up quota" >> $seqres.full +$XFS_QUOTA_PROG -x -c "limit -u bhard=$((alloc_unit * 2)) $qa_user" $SCRATCH_MNT +chown $qa_user $scratchfile +$XFS_QUOTA_PROG -x -c 'report -u' $SCRATCH_MNT >> $seqres.full + +# Rebuild the data fork +echo "repairs" >> $seqres.full +$XFS_IO_PROG -x -c 'inject force_repair' -c 'repair bmapbtd' $scratchfile +$XFS_QUOTA_PROG -x -c 'report -u' $SCRATCH_MNT >> $seqres.full + +# Fail at appending the file as qa_user to ensure quota enforcement works +echo "fail quota" >> $seqres.full +su - "$qa_user" -c "$XFS_IO_PROG -c 'pwrite 10g 1' $scratchfile" >> $seqres.full +$XFS_QUOTA_PROG -x -c 'report -u' $SCRATCH_MNT >> $seqres.full + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/840.out b/tests/xfs/840.out new file mode 100644 index 0000000000..8c32ec12bb --- /dev/null +++ b/tests/xfs/840.out @@ -0,0 +1,3 @@ +QA output created by 840 +pwrite: Disk quota exceeded +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 2/4] xfs: race fsstress with online repair for inode and fork metadata 2022-12-30 22:19 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 3/4] xfs: ensure that online file data fork repairs don't hit EDQUOT Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-18 6:07 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Zorro Lang 4 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> For each XFS_SCRUB_TYPE_* that looks at inode and data/attr/cow fork metadata, create a test that runs that repairer in the foreground and fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/807 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/807.out | 2 ++ tests/xfs/808 | 39 +++++++++++++++++++++++++++++++++++++++ tests/xfs/808.out | 2 ++ tests/xfs/846 | 39 +++++++++++++++++++++++++++++++++++++++ tests/xfs/846.out | 2 ++ 6 files changed, 121 insertions(+) create mode 100755 tests/xfs/807 create mode 100644 tests/xfs/807.out create mode 100755 tests/xfs/808 create mode 100644 tests/xfs/808.out create mode 100755 tests/xfs/846 create mode 100644 tests/xfs/846.out diff --git a/tests/xfs/807 b/tests/xfs/807 new file mode 100755 index 0000000000..e32a37057d --- /dev/null +++ b/tests/xfs/807 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 807 +# +# Race fsstress and data fork repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -s "repair bmapbtd" -t "%datafile%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/807.out b/tests/xfs/807.out new file mode 100644 index 0000000000..3752a5f715 --- /dev/null +++ b/tests/xfs/807.out @@ -0,0 +1,2 @@ +QA output created by 807 +Silence is golden diff --git a/tests/xfs/808 b/tests/xfs/808 new file mode 100755 index 0000000000..378b606427 --- /dev/null +++ b/tests/xfs/808 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 808 +# +# Race fsstress and attr fork repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/attr + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_attrs +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -x 'xattr' -s "repair bmapbta" -t "%attrfile%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/808.out b/tests/xfs/808.out new file mode 100644 index 0000000000..8825342849 --- /dev/null +++ b/tests/xfs/808.out @@ -0,0 +1,2 @@ +QA output created by 808 +Silence is golden diff --git a/tests/xfs/846 b/tests/xfs/846 new file mode 100755 index 0000000000..8388a22730 --- /dev/null +++ b/tests/xfs/846 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 846 +# +# Race fsstress and CoW fork repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" reflink +_scratch_xfs_stress_online_repair -s "repair bmapbtc" -t "%cowfile%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/846.out b/tests/xfs/846.out new file mode 100644 index 0000000000..c88a3035c3 --- /dev/null +++ b/tests/xfs/846.out @@ -0,0 +1,2 @@ +QA output created by 846 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings 2022-12-30 22:19 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 2/4] xfs: race fsstress with online repair for inode and fork metadata Darrick J. Wong @ 2023-02-18 6:07 ` Zorro Lang 4 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-18 6:07 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:18PM -0800, Darrick J. Wong wrote: > Hi all, > > In this series, online repair gains the ability to rebuild data and attr > fork mappings from the reverse mapping information. It is at this point > where we reintroduce the ability to reap file extents. > > Repair of CoW forks is a little different -- on disk, CoW staging > extents are owned by the refcount btree and cannot be mapped back to > individual files. Hence we can only detect staging extents that don't > quite look right (missing reverse mappings, shared staging extents) and > replace them with fresh allocations. > > If you're going to start using this mess, you probably ought to just > pull from my git trees, which are linked below. > > This is an extraordinary way to destroy everything. Enjoy! > Comments and questions are, as always, welcome. > > --D Ack, will merge this patchset Reviewed-by: Zorro Lang <zlang@redhat.com> > > kernel git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-file-mappings > > xfsprogs git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-file-mappings > > fstests git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-file-mappings > --- > tests/xfs/746 | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > tests/xfs/746.out | 2 + > tests/xfs/807 | 37 +++++++++++++++++++++++ > tests/xfs/807.out | 2 + > tests/xfs/808 | 39 ++++++++++++++++++++++++ > tests/xfs/808.out | 2 + > tests/xfs/828 | 38 ++++++++++++++++++++++++ > tests/xfs/828.out | 2 + > tests/xfs/829 | 39 ++++++++++++++++++++++++ > tests/xfs/829.out | 2 + > tests/xfs/840 | 72 +++++++++++++++++++++++++++++++++++++++++++++ > tests/xfs/840.out | 3 ++ > tests/xfs/846 | 39 ++++++++++++++++++++++++ > tests/xfs/846.out | 2 + > 14 files changed, 364 insertions(+) > create mode 100755 tests/xfs/746 > create mode 100644 tests/xfs/746.out > create mode 100755 tests/xfs/807 > create mode 100644 tests/xfs/807.out > create mode 100755 tests/xfs/808 > create mode 100644 tests/xfs/808.out > create mode 100755 tests/xfs/828 > create mode 100644 tests/xfs/828.out > create mode 100755 tests/xfs/829 > create mode 100644 tests/xfs/829.out > create mode 100755 tests/xfs/840 > create mode 100644 tests/xfs/840.out > create mode 100755 tests/xfs/846 > create mode 100644 tests/xfs/846.out > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/1] fstests: online repair of quota and counters 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (4 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online scrub and repair for quota metadata Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of quota counters Darrick J. Wong ` (11 subsequent siblings) 17 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, XFS stores quota records and free space bitmap information in files. Add the necessary infrastructure to enable repairing metadata inodes and their forks, and then make it so that we can repair the file metadata for the rtbitmap. Repairing the bitmap contents (and the summary file) is left for subsequent patchsets. We also add the ability to repair file metadata the quota files. As part of these repairs, we also reinitialize the ondisk dquot records as necessary to get the incore dquots working. We can also correct obviously bad dquot record attributes, but we leave checking the resource usage counts for the next patchsets. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quota xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-quota fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-quota --- tests/xfs/809 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/809.out | 2 ++ tests/xfs/810 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/810.out | 2 ++ tests/xfs/811 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/811.out | 2 ++ 6 files changed, 126 insertions(+) create mode 100755 tests/xfs/809 create mode 100644 tests/xfs/809.out create mode 100755 tests/xfs/810 create mode 100644 tests/xfs/810.out create mode 100755 tests/xfs/811 create mode 100644 tests/xfs/811.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/1] xfs: race fsstress with online scrub and repair for quota metadata 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of quota and counters Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-18 6:10 ` Zorro Lang 2023-02-18 6:12 ` Zorro Lang 0 siblings, 2 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create tests to race fsstress with dquot repair while running fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/809 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/809.out | 2 ++ tests/xfs/810 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/810.out | 2 ++ tests/xfs/811 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/811.out | 2 ++ 6 files changed, 126 insertions(+) create mode 100755 tests/xfs/809 create mode 100644 tests/xfs/809.out create mode 100755 tests/xfs/810 create mode 100644 tests/xfs/810.out create mode 100755 tests/xfs/811 create mode 100644 tests/xfs/811.out diff --git a/tests/xfs/809 b/tests/xfs/809 new file mode 100755 index 0000000000..35ac02ff85 --- /dev/null +++ b/tests/xfs/809 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 809 +# +# Race fsstress and user quota repair for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" usrquota +_scratch_xfs_stress_online_repair -s "repair usrquota" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/809.out b/tests/xfs/809.out new file mode 100644 index 0000000000..e90865ca8f --- /dev/null +++ b/tests/xfs/809.out @@ -0,0 +1,2 @@ +QA output created by 809 +Silence is golden diff --git a/tests/xfs/810 b/tests/xfs/810 new file mode 100755 index 0000000000..7387910504 --- /dev/null +++ b/tests/xfs/810 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 810 +# +# Race fsstress and group quota repair for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" grpquota +_scratch_xfs_stress_online_repair -s "repair grpquota" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/810.out b/tests/xfs/810.out new file mode 100644 index 0000000000..90f12fdd21 --- /dev/null +++ b/tests/xfs/810.out @@ -0,0 +1,2 @@ +QA output created by 810 +Silence is golden diff --git a/tests/xfs/811 b/tests/xfs/811 new file mode 100755 index 0000000000..1e13940b46 --- /dev/null +++ b/tests/xfs/811 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 811 +# +# Race fsstress and project quota repair for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" prjquota +_scratch_xfs_stress_online_repair -s "repair prjquota" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/811.out b/tests/xfs/811.out new file mode 100644 index 0000000000..cf30f69bdc --- /dev/null +++ b/tests/xfs/811.out @@ -0,0 +1,2 @@ +QA output created by 811 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 1/1] xfs: race fsstress with online scrub and repair for quota metadata 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online scrub and repair for quota metadata Darrick J. Wong @ 2023-02-18 6:10 ` Zorro Lang 2023-02-18 6:12 ` Zorro Lang 1 sibling, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-18 6:10 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:21PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Create tests to race fsstress with dquot repair while running fsstress > in the background. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- Reviewed-by: Zorro Lang <zlang@redhat.com> > tests/xfs/809 | 40 ++++++++++++++++++++++++++++++++++++++++ > tests/xfs/809.out | 2 ++ > tests/xfs/810 | 40 ++++++++++++++++++++++++++++++++++++++++ > tests/xfs/810.out | 2 ++ > tests/xfs/811 | 40 ++++++++++++++++++++++++++++++++++++++++ > tests/xfs/811.out | 2 ++ > 6 files changed, 126 insertions(+) > create mode 100755 tests/xfs/809 > create mode 100644 tests/xfs/809.out > create mode 100755 tests/xfs/810 > create mode 100644 tests/xfs/810.out > create mode 100755 tests/xfs/811 > create mode 100644 tests/xfs/811.out > > > diff --git a/tests/xfs/809 b/tests/xfs/809 > new file mode 100755 > index 0000000000..35ac02ff85 > --- /dev/null > +++ b/tests/xfs/809 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 809 > +# > +# Race fsstress and user quota repair for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest online_repair dangerous_fsstress_repair > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_online_repair > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" usrquota > +_scratch_xfs_stress_online_repair -s "repair usrquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/809.out b/tests/xfs/809.out > new file mode 100644 > index 0000000000..e90865ca8f > --- /dev/null > +++ b/tests/xfs/809.out > @@ -0,0 +1,2 @@ > +QA output created by 809 > +Silence is golden > diff --git a/tests/xfs/810 b/tests/xfs/810 > new file mode 100755 > index 0000000000..7387910504 > --- /dev/null > +++ b/tests/xfs/810 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 810 > +# > +# Race fsstress and group quota repair for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest online_repair dangerous_fsstress_repair > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_online_repair > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" grpquota > +_scratch_xfs_stress_online_repair -s "repair grpquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/810.out b/tests/xfs/810.out > new file mode 100644 > index 0000000000..90f12fdd21 > --- /dev/null > +++ b/tests/xfs/810.out > @@ -0,0 +1,2 @@ > +QA output created by 810 > +Silence is golden > diff --git a/tests/xfs/811 b/tests/xfs/811 > new file mode 100755 > index 0000000000..1e13940b46 > --- /dev/null > +++ b/tests/xfs/811 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 811 > +# > +# Race fsstress and project quota repair for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest online_repair dangerous_fsstress_repair > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_online_repair > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" prjquota > +_scratch_xfs_stress_online_repair -s "repair prjquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/811.out b/tests/xfs/811.out > new file mode 100644 > index 0000000000..cf30f69bdc > --- /dev/null > +++ b/tests/xfs/811.out > @@ -0,0 +1,2 @@ > +QA output created by 811 > +Silence is golden > ^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH 1/1] xfs: race fsstress with online scrub and repair for quota metadata 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online scrub and repair for quota metadata Darrick J. Wong 2023-02-18 6:10 ` Zorro Lang @ 2023-02-18 6:12 ` Zorro Lang 1 sibling, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-18 6:12 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:21PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Create tests to race fsstress with dquot repair while running fsstress > in the background. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- Reviewed-by: Zorro Lang <zlang@redhat.com> > tests/xfs/809 | 40 ++++++++++++++++++++++++++++++++++++++++ > tests/xfs/809.out | 2 ++ > tests/xfs/810 | 40 ++++++++++++++++++++++++++++++++++++++++ > tests/xfs/810.out | 2 ++ > tests/xfs/811 | 40 ++++++++++++++++++++++++++++++++++++++++ > tests/xfs/811.out | 2 ++ > 6 files changed, 126 insertions(+) > create mode 100755 tests/xfs/809 > create mode 100644 tests/xfs/809.out > create mode 100755 tests/xfs/810 > create mode 100644 tests/xfs/810.out > create mode 100755 tests/xfs/811 > create mode 100644 tests/xfs/811.out > > > diff --git a/tests/xfs/809 b/tests/xfs/809 > new file mode 100755 > index 0000000000..35ac02ff85 > --- /dev/null > +++ b/tests/xfs/809 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 809 > +# > +# Race fsstress and user quota repair for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest online_repair dangerous_fsstress_repair > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_online_repair > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" usrquota > +_scratch_xfs_stress_online_repair -s "repair usrquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/809.out b/tests/xfs/809.out > new file mode 100644 > index 0000000000..e90865ca8f > --- /dev/null > +++ b/tests/xfs/809.out > @@ -0,0 +1,2 @@ > +QA output created by 809 > +Silence is golden > diff --git a/tests/xfs/810 b/tests/xfs/810 > new file mode 100755 > index 0000000000..7387910504 > --- /dev/null > +++ b/tests/xfs/810 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 810 > +# > +# Race fsstress and group quota repair for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest online_repair dangerous_fsstress_repair > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_online_repair > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" grpquota > +_scratch_xfs_stress_online_repair -s "repair grpquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/810.out b/tests/xfs/810.out > new file mode 100644 > index 0000000000..90f12fdd21 > --- /dev/null > +++ b/tests/xfs/810.out > @@ -0,0 +1,2 @@ > +QA output created by 810 > +Silence is golden > diff --git a/tests/xfs/811 b/tests/xfs/811 > new file mode 100755 > index 0000000000..1e13940b46 > --- /dev/null > +++ b/tests/xfs/811 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 811 > +# > +# Race fsstress and project quota repair for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest online_repair dangerous_fsstress_repair > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_online_repair > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" prjquota > +_scratch_xfs_stress_online_repair -s "repair prjquota" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/811.out b/tests/xfs/811.out > new file mode 100644 > index 0000000000..cf30f69bdc > --- /dev/null > +++ b/tests/xfs/811.out > @@ -0,0 +1,2 @@ > +QA output created by 811 > +Silence is golden > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/1] fstests: online repair of quota counters 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (5 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of quota and counters Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online scrub and repair for quotacheck Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of file link counts Darrick J. Wong ` (10 subsequent siblings) 17 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, This series uses the inode scanner and live update hook functionality introduced in the last patchset to implement quotacheck on a live filesystem. The quotacheck scrubber builds an incore copy of the dquot resource usage counters and compares it to the live dquots to report discrepancies. If the user chooses to repair the quota counters, the repair function visits each incore dquot to update the counts from the live information. The live update hooks are key to keeping the incore copy up to date. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-quotacheck xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-quotacheck fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-quotacheck --- tests/xfs/715 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/715.out | 2 ++ tests/xfs/812 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/812.out | 2 ++ 4 files changed, 84 insertions(+) create mode 100755 tests/xfs/715 create mode 100644 tests/xfs/715.out create mode 100755 tests/xfs/812 create mode 100644 tests/xfs/812.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/1] xfs: race fsstress with online scrub and repair for quotacheck 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of quota counters Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-18 6:12 ` Zorro Lang 0 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create tests to race fsstress with quota count check and repair while running fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/715 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/715.out | 2 ++ tests/xfs/812 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/812.out | 2 ++ 4 files changed, 84 insertions(+) create mode 100755 tests/xfs/715 create mode 100644 tests/xfs/715.out create mode 100755 tests/xfs/812 create mode 100644 tests/xfs/812.out diff --git a/tests/xfs/715 b/tests/xfs/715 new file mode 100755 index 0000000000..eca979b297 --- /dev/null +++ b/tests/xfs/715 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 715 +# +# Race fsstress and quotacheck repair for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" any +_scratch_xfs_stress_online_repair -s "repair quotacheck" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/715.out b/tests/xfs/715.out new file mode 100644 index 0000000000..b5947d898b --- /dev/null +++ b/tests/xfs/715.out @@ -0,0 +1,2 @@ +QA output created by 715 +Silence is golden diff --git a/tests/xfs/812 b/tests/xfs/812 new file mode 100755 index 0000000000..f84494e392 --- /dev/null +++ b/tests/xfs/812 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 812 +# +# Race fsstress and quotacheck scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" any +_scratch_xfs_stress_scrub -s "scrub quotacheck" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/812.out b/tests/xfs/812.out new file mode 100644 index 0000000000..d8dbb15dc7 --- /dev/null +++ b/tests/xfs/812.out @@ -0,0 +1,2 @@ +QA output created by 812 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 1/1] xfs: race fsstress with online scrub and repair for quotacheck 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online scrub and repair for quotacheck Darrick J. Wong @ 2023-02-18 6:12 ` Zorro Lang 0 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-18 6:12 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:24PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Create tests to race fsstress with quota count check and repair while > running fsstress in the background. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- Reviewed-by: Zorro Lang <zlang@redhat.com> > tests/xfs/715 | 40 ++++++++++++++++++++++++++++++++++++++++ > tests/xfs/715.out | 2 ++ > tests/xfs/812 | 40 ++++++++++++++++++++++++++++++++++++++++ > tests/xfs/812.out | 2 ++ > 4 files changed, 84 insertions(+) > create mode 100755 tests/xfs/715 > create mode 100644 tests/xfs/715.out > create mode 100755 tests/xfs/812 > create mode 100644 tests/xfs/812.out > > > diff --git a/tests/xfs/715 b/tests/xfs/715 > new file mode 100755 > index 0000000000..eca979b297 > --- /dev/null > +++ b/tests/xfs/715 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 715 > +# > +# Race fsstress and quotacheck repair for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest online_repair dangerous_fsstress_repair > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_online_repair > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" any > +_scratch_xfs_stress_online_repair -s "repair quotacheck" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/715.out b/tests/xfs/715.out > new file mode 100644 > index 0000000000..b5947d898b > --- /dev/null > +++ b/tests/xfs/715.out > @@ -0,0 +1,2 @@ > +QA output created by 715 > +Silence is golden > diff --git a/tests/xfs/812 b/tests/xfs/812 > new file mode 100755 > index 0000000000..f84494e392 > --- /dev/null > +++ b/tests/xfs/812 > @@ -0,0 +1,40 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. > +# > +# FS QA Test No. 812 > +# > +# Race fsstress and quotacheck scrub for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > +. ./common/quota > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_require_xfs_quota_acct_enabled "$SCRATCH_DEV" any > +_scratch_xfs_stress_scrub -s "scrub quotacheck" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/812.out b/tests/xfs/812.out > new file mode 100644 > index 0000000000..d8dbb15dc7 > --- /dev/null > +++ b/tests/xfs/812.out > @@ -0,0 +1,2 @@ > +QA output created by 812 > +Silence is golden > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/1] fstests: online repair of file link counts 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (6 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of quota counters Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with inode link count check and repair Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair for fs summary counters Darrick J. Wong ` (9 subsequent siblings) 17 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, Now that we've created the infrastructure to perform live scans of every file in the filesystem and the necessary hook infrastructure to observe live updates, use it to scan directories to compute the correct link counts for files in the filesystem, and reset those link counts. This patchset creates a tailored readdir implementation for scrub because the regular version has to cycle ILOCKs to copy information to userspace. We can't cycle the ILOCK during the nlink scan and we don't need all the other VFS support code (maintaining a readdir cursor and translating XFS structures to VFS structures and back) so it was easier to duplicate the code. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=scrub-nlinks xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-nlinks fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-nlinks --- tests/xfs/772 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/772.out | 2 ++ tests/xfs/820 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/820.out | 2 ++ 4 files changed, 79 insertions(+) create mode 100755 tests/xfs/772 create mode 100644 tests/xfs/772.out create mode 100755 tests/xfs/820 create mode 100644 tests/xfs/820.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/1] xfs: race fsstress with inode link count check and repair 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of file link counts Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-18 6:13 ` Zorro Lang 0 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Race fsstress with inode link count checking and repair. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/772 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/772.out | 2 ++ tests/xfs/820 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/820.out | 2 ++ 4 files changed, 79 insertions(+) create mode 100755 tests/xfs/772 create mode 100644 tests/xfs/772.out create mode 100755 tests/xfs/820 create mode 100644 tests/xfs/820.out diff --git a/tests/xfs/772 b/tests/xfs/772 new file mode 100755 index 0000000000..a00c2796c5 --- /dev/null +++ b/tests/xfs/772 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 772 +# +# Race fsstress and inode link count repair for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -x "dir" -s "repair nlinks" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/772.out b/tests/xfs/772.out new file mode 100644 index 0000000000..98c1396896 --- /dev/null +++ b/tests/xfs/772.out @@ -0,0 +1,2 @@ +QA output created by 772 +Silence is golden diff --git a/tests/xfs/820 b/tests/xfs/820 new file mode 100755 index 0000000000..58a5d4cc91 --- /dev/null +++ b/tests/xfs/820 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 820 +# +# Race fsstress and nlinks scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_scrub -x "dir" -s "scrub nlinks" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/820.out b/tests/xfs/820.out new file mode 100644 index 0000000000..29ab2e2d8c --- /dev/null +++ b/tests/xfs/820.out @@ -0,0 +1,2 @@ +QA output created by 820 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 1/1] xfs: race fsstress with inode link count check and repair 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with inode link count check and repair Darrick J. Wong @ 2023-02-18 6:13 ` Zorro Lang 0 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-18 6:13 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:27PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Race fsstress with inode link count checking and repair. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- Reviewed-by: Zorro Lang <zlang@redhat.com> > tests/xfs/772 | 38 ++++++++++++++++++++++++++++++++++++++ > tests/xfs/772.out | 2 ++ > tests/xfs/820 | 37 +++++++++++++++++++++++++++++++++++++ > tests/xfs/820.out | 2 ++ > 4 files changed, 79 insertions(+) > create mode 100755 tests/xfs/772 > create mode 100644 tests/xfs/772.out > create mode 100755 tests/xfs/820 > create mode 100644 tests/xfs/820.out > > > diff --git a/tests/xfs/772 b/tests/xfs/772 > new file mode 100755 > index 0000000000..a00c2796c5 > --- /dev/null > +++ b/tests/xfs/772 > @@ -0,0 +1,38 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# Copyright (c) 2022 Oracle. All Rights Reserved. > +# > +# FS QA Test No. 772 > +# > +# Race fsstress and inode link count repair for a while to see if we crash or > +# livelock. > +# > +. ./common/preamble > +_begin_fstest online_repair dangerous_fsstress_repair > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_online_repair > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_online_repair -x "dir" -s "repair nlinks" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/772.out b/tests/xfs/772.out > new file mode 100644 > index 0000000000..98c1396896 > --- /dev/null > +++ b/tests/xfs/772.out > @@ -0,0 +1,2 @@ > +QA output created by 772 > +Silence is golden > diff --git a/tests/xfs/820 b/tests/xfs/820 > new file mode 100755 > index 0000000000..58a5d4cc91 > --- /dev/null > +++ b/tests/xfs/820 > @@ -0,0 +1,37 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# Copyright (c) 2022 Oracle. All Rights Reserved. > +# > +# FS QA Test No. 820 > +# > +# Race fsstress and nlinks scrub for a while to see if we crash or livelock. > +# > +. ./common/preamble > +_begin_fstest scrub dangerous_fsstress_scrub > + > +_cleanup() { > + _scratch_xfs_stress_scrub_cleanup &> /dev/null > + cd / > + rm -r -f $tmp.* > +} > +_register_cleanup "_cleanup" BUS > + > +# Import common functions. > +. ./common/filter > +. ./common/fuzzy > +. ./common/inject > +. ./common/xfs > + > +# real QA test starts here > +_supported_fs xfs > +_require_scratch > +_require_xfs_stress_scrub > + > +_scratch_mkfs > "$seqres.full" 2>&1 > +_scratch_mount > +_scratch_xfs_stress_scrub -x "dir" -s "scrub nlinks" > + > +# success, all done > +echo Silence is golden > +status=0 > +exit > diff --git a/tests/xfs/820.out b/tests/xfs/820.out > new file mode 100644 > index 0000000000..29ab2e2d8c > --- /dev/null > +++ b/tests/xfs/820.out > @@ -0,0 +1,2 @@ > +QA output created by 820 > +Silence is golden > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/2] fstests: online repair for fs summary counters 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (7 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of file link counts Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] xfs: test fs summary counter online repair Darrick J. Wong ` (2 more replies) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of rmap btrees Darrick J. Wong ` (8 subsequent siblings) 17 siblings, 3 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, A longstanding deficiency in the online fs summary counter scrubbing code is that it hasn't any means to quiesce the incore percpu counters while it's running. There is no way to coordinate with other threads are reserving or freeing free space simultaneously, which leads to false error reports. Right now, if the discrepancy is large, we just sort of shrug and bail out with an incomplete flag, but this is lame. For repair activity, we actually /do/ need to stabilize the counters to get an accurate reading and install it in the percpu counter. To improve the former and enable the latter, allow the fscounters online fsck code to perform an exclusive mini-freeze on the filesystem. The exclusivity prevents userspace from thawing while we're running, and the mini-freeze means that we don't wait for the log to quiesce, which will make both speedier. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-fscounters fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-fscounters --- tests/xfs/713 | 36 ++++++++++++++++++++++++++++++++++++ tests/xfs/713.out | 4 ++++ tests/xfs/714 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/714.out | 2 ++ tests/xfs/762 | 46 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/762.out | 2 ++ 6 files changed, 131 insertions(+) create mode 100755 tests/xfs/713 create mode 100644 tests/xfs/713.out create mode 100755 tests/xfs/714 create mode 100644 tests/xfs/714.out create mode 100755 tests/xfs/762 create mode 100644 tests/xfs/762.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/2] xfs: test fs summary counter online repair 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair for fs summary counters Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] xfs: race fsstress with online repair for summary counters Darrick J. Wong 2023-02-18 6:14 ` [PATCHSET v24.0 0/2] fstests: online repair for fs " Zorro Lang 2 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Fuzz the fs summary counters in the primary super and see if online repair can fix them. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/713 | 36 ++++++++++++++++++++++++++++++++++++ tests/xfs/713.out | 4 ++++ 2 files changed, 40 insertions(+) create mode 100755 tests/xfs/713 create mode 100644 tests/xfs/713.out diff --git a/tests/xfs/713 b/tests/xfs/713 new file mode 100755 index 0000000000..7ac6d1458f --- /dev/null +++ b/tests/xfs/713 @@ -0,0 +1,36 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 713 +# +# Populate a XFS filesystem and fuzz every fscounter field. +# Use xfs_scrub to fix the corruption. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz fscounters" +test -z "$SCRATCH_XFS_LIST_METADATA_FIELDS" && + SCRATCH_XFS_LIST_METADATA_FIELDS='icount,ifree,fdblocks' +export SCRATCH_XFS_LIST_METADATA_FIELDS +_scratch_xfs_fuzz_metadata '' 'online' 'sb 0' >> $seqres.full +echo "Done fuzzing fscounters" + +# success, all done +status=0 +exit diff --git a/tests/xfs/713.out b/tests/xfs/713.out new file mode 100644 index 0000000000..6dd322f4ca --- /dev/null +++ b/tests/xfs/713.out @@ -0,0 +1,4 @@ +QA output created by 713 +Format and populate +Fuzz fscounters +Done fuzzing fscounters ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 2/2] xfs: race fsstress with online repair for summary counters 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair for fs summary counters Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] xfs: test fs summary counter online repair Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-18 6:14 ` [PATCHSET v24.0 0/2] fstests: online repair for fs " Zorro Lang 2 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create tests to race fsstress with fs summary counter repair while running fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/714 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/714.out | 2 ++ tests/xfs/762 | 46 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/762.out | 2 ++ 4 files changed, 91 insertions(+) create mode 100755 tests/xfs/714 create mode 100644 tests/xfs/714.out create mode 100755 tests/xfs/762 create mode 100644 tests/xfs/762.out diff --git a/tests/xfs/714 b/tests/xfs/714 new file mode 100755 index 0000000000..c1b6cd919a --- /dev/null +++ b/tests/xfs/714 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 714 +# +# Race fsstress and fscounter repair for a while to see if we crash or livelock. +# Summary counter repair requires us to freeze the filesystem to stop all +# filesystem activity, so we can't have userspace wandering in and thawing it. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +# Override the default cleanup function. +_cleanup() +{ + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -rf $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -s "repair fscounters" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/714.out b/tests/xfs/714.out new file mode 100644 index 0000000000..f5ce748b71 --- /dev/null +++ b/tests/xfs/714.out @@ -0,0 +1,2 @@ +QA output created by 714 +Silence is golden diff --git a/tests/xfs/762 b/tests/xfs/762 new file mode 100755 index 0000000000..0f70b632a3 --- /dev/null +++ b/tests/xfs/762 @@ -0,0 +1,46 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 762 +# +# Race fsstress and fscounter repair on the realtime device for a while to see +# if we crash or livelock. Summary counter repair requires us to freeze the +# filesystem to stop all filesystem activity, so we can't have userspace +# wandering in and thawing it. +# +. ./common/preamble +_begin_fstest auto quick rw scrub realtime + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_realtime +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime + +# Force all files to be allocated on the realtime device +_xfs_force_bdev realtime $SCRATCH_MNT +_scratch_xfs_stress_online_repair -s 'scrub fscounters' -s "repair fscounters" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/762.out b/tests/xfs/762.out new file mode 100644 index 0000000000..fbaeb29706 --- /dev/null +++ b/tests/xfs/762.out @@ -0,0 +1,2 @@ +QA output created by 762 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCHSET v24.0 0/2] fstests: online repair for fs summary counters 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair for fs summary counters Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] xfs: test fs summary counter online repair Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] xfs: race fsstress with online repair for summary counters Darrick J. Wong @ 2023-02-18 6:14 ` Zorro Lang 2 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-18 6:14 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:30PM -0800, Darrick J. Wong wrote: > Hi all, > > A longstanding deficiency in the online fs summary counter scrubbing > code is that it hasn't any means to quiesce the incore percpu counters > while it's running. There is no way to coordinate with other threads > are reserving or freeing free space simultaneously, which leads to false > error reports. Right now, if the discrepancy is large, we just sort of > shrug and bail out with an incomplete flag, but this is lame. > > For repair activity, we actually /do/ need to stabilize the counters to > get an accurate reading and install it in the percpu counter. To > improve the former and enable the latter, allow the fscounters online > fsck code to perform an exclusive mini-freeze on the filesystem. The > exclusivity prevents userspace from thawing while we're running, and the > mini-freeze means that we don't wait for the log to quiesce, which will > make both speedier. > > If you're going to start using this mess, you probably ought to just > pull from my git trees, which are linked below. > > This is an extraordinary way to destroy everything. Enjoy! > Comments and questions are, as always, welcome. > > --D This patchset looks good to me, Reviewed-by: Zorro Lang <zlang@redhat.com> > > kernel git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-fscounters > > fstests git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-fscounters > --- > tests/xfs/713 | 36 ++++++++++++++++++++++++++++++++++++ > tests/xfs/713.out | 4 ++++ > tests/xfs/714 | 41 +++++++++++++++++++++++++++++++++++++++++ > tests/xfs/714.out | 2 ++ > tests/xfs/762 | 46 ++++++++++++++++++++++++++++++++++++++++++++++ > tests/xfs/762.out | 2 ++ > 6 files changed, 131 insertions(+) > create mode 100755 tests/xfs/713 > create mode 100644 tests/xfs/713.out > create mode 100755 tests/xfs/714 > create mode 100644 tests/xfs/714.out > create mode 100755 tests/xfs/762 > create mode 100644 tests/xfs/762.out > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/1] fstests: online repair of rmap btrees 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (8 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair for fs summary counters Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs/422: don't freeze while racing rmap repair and fsstress Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Darrick J. Wong ` (7 subsequent siblings) 17 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, We have now constructed the four tools that we need to scan the filesystem looking for reverse mappings: an inode scanner, hooks to receive live updates from other writer threads, the ability to construct btrees in memory, and a btree bulk loader. This series glues those three together, enabling us to scan the filesystem for mappings and keep it up to date while other writers run, and then commit the new btree to disk atomically. To reduce the size of each patch, the functionality is left disabled until the end of the series and broken up into three patches: one to create the mechanics of scanning the filesystem, a second to transition to in-memory btrees, and a third to set up the live hooks. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rmap-btree xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-rmap-btree fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-rmap-btree --- tests/xfs/422 | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/1] xfs/422: don't freeze while racing rmap repair and fsstress 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of rmap btrees Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-18 6:15 ` Zorro Lang 0 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Since we're moving away from freezing the filesystem for rmap repair, remove the freeze/thaw race from this test to make it more interesting. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/422 | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tests/xfs/422 b/tests/xfs/422 index 995f612166..339f12976a 100755 --- a/tests/xfs/422 +++ b/tests/xfs/422 @@ -5,8 +5,6 @@ # FS QA Test No. 422 # # Race fsstress and rmapbt repair for a while to see if we crash or livelock. -# rmapbt repair requires us to freeze the filesystem to stop all filesystem -# activity, so we can't have userspace wandering in and thawing it. # . ./common/preamble _begin_fstest online_repair dangerous_fsstress_repair freeze @@ -31,7 +29,7 @@ _require_xfs_stress_online_repair _scratch_mkfs > "$seqres.full" 2>&1 _scratch_mount _require_xfs_has_feature "$SCRATCH_MNT" rmapbt -_scratch_xfs_stress_online_repair -f -s "repair rmapbt %agno%" +_scratch_xfs_stress_online_repair -s "repair rmapbt %agno%" # success, all done echo Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 1/1] xfs/422: don't freeze while racing rmap repair and fsstress 2022-12-30 22:19 ` [PATCH 1/1] xfs/422: don't freeze while racing rmap repair and fsstress Darrick J. Wong @ 2023-02-18 6:15 ` Zorro Lang 0 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-18 6:15 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests, guan On Fri, Dec 30, 2022 at 02:19:33PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Since we're moving away from freezing the filesystem for rmap repair, > remove the freeze/thaw race from this test to make it more interesting. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- Looks good to me, Reviewed-by: Zorro Lang <zlang@redhat.com> > tests/xfs/422 | 4 +--- > 1 file changed, 1 insertion(+), 3 deletions(-) > > > diff --git a/tests/xfs/422 b/tests/xfs/422 > index 995f612166..339f12976a 100755 > --- a/tests/xfs/422 > +++ b/tests/xfs/422 > @@ -5,8 +5,6 @@ > # FS QA Test No. 422 > # > # Race fsstress and rmapbt repair for a while to see if we crash or livelock. > -# rmapbt repair requires us to freeze the filesystem to stop all filesystem > -# activity, so we can't have userspace wandering in and thawing it. > # > . ./common/preamble > _begin_fstest online_repair dangerous_fsstress_repair freeze > @@ -31,7 +29,7 @@ _require_xfs_stress_online_repair > _scratch_mkfs > "$seqres.full" 2>&1 > _scratch_mount > _require_xfs_has_feature "$SCRATCH_MNT" rmapbt > -_scratch_xfs_stress_online_repair -f -s "repair rmapbt %agno%" > +_scratch_xfs_stress_online_repair -s "repair rmapbt %agno%" > > # success, all done > echo Silence is golden > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (9 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of rmap btrees Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] populate: take a snapshot of the filesystem if creation fails Darrick J. Wong ` (2 more replies) 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (6 subsequent siblings) 17 siblings, 3 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, Before we start on an intense patchset of improving the XFS fuzz testing framework, let's fix a couple of bugs in the code that creates sample filesystems with all types of metadata. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-populate-problems --- common/populate | 70 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 46 insertions(+), 24 deletions(-) ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/2] populate: take a snapshot of the filesystem if creation fails 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] populate: fix some weirdness in __populate_check_xfs_agbtree_height Darrick J. Wong 2023-02-18 6:16 ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Zorro Lang 2 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> There have been a few bug reports filed about people not being able to use the filesystem metadata population code to create filesystems with all types of metadata on them. Right now this is super-annoying to debug because we don't capture a metadump for easy debugging. Fix that. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/populate | 59 +++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 40 insertions(+), 19 deletions(-) diff --git a/common/populate b/common/populate index 44b4af1667..e4090a29d3 100644 --- a/common/populate +++ b/common/populate @@ -40,6 +40,27 @@ __populate_create_file() { $XFS_IO_PROG -f -c "pwrite -S 0x62 -W -b 1m 0 $sz" "${fname}" } +# Fail the test if we failed to create some kind of filesystem metadata. +# Create a metadata dump of the failed filesystem so that we can analyze +# how things went rong. +__populate_fail() { + local flatdev="$(basename "$SCRATCH_DEV")" + local metadump="$seqres.$flatdev.populate.md" + + case "$FSTYP" in + xfs) + _scratch_unmount + _scratch_xfs_metadump "$metadump" + ;; + ext4) + _scratch_unmount + _ext4_metadump "${SCRATCH_DEV}" "$metadump" + ;; + esac + + _fail "$@" +} + # Punch out every other hole in this file, if it exists. # # The goal here is to force the creation of a large number of metadata records @@ -501,7 +522,7 @@ __populate_check_xfs_dformat() { format="$2" fmt="$(_scratch_xfs_db -c "inode ${inode}" -c 'p core.format' | sed -e 's/^.*(\([a-z]*\)).*$/\1/g')" - test "${format}" = "${fmt}" || _fail "failed to create ino ${inode} dformat expected ${format} saw ${fmt}" + test "${format}" = "${fmt}" || __populate_fail "failed to create ino ${inode} dformat expected ${format} saw ${fmt}" } # Check attr fork format of XFS file @@ -510,7 +531,7 @@ __populate_check_xfs_aformat() { format="$2" fmt="$(_scratch_xfs_db -c "inode ${inode}" -c 'p core.aformat' | sed -e 's/^.*(\([a-z]*\)).*$/\1/g')" - test "${format}" = "${fmt}" || _fail "failed to create ino ${inode} aformat expected ${format} saw ${fmt}" + test "${format}" = "${fmt}" || __populate_fail "failed to create ino ${inode} aformat expected ${format} saw ${fmt}" } # Check structure of XFS directory @@ -529,21 +550,21 @@ __populate_check_xfs_dir() { case "${dtype}" in "shortform"|"inline"|"local") - (test "${datab}" -eq 0 && test "${leafb}" -eq 0 && test "${freeb}" -eq 0) || _fail "failed to create ${dtype} dir ino ${inode} datab ${datab} leafb ${leafb} freeb ${freeb}" + (test "${datab}" -eq 0 && test "${leafb}" -eq 0 && test "${freeb}" -eq 0) || __populate_fail "failed to create ${dtype} dir ino ${inode} datab ${datab} leafb ${leafb} freeb ${freeb}" ;; "block") - (test "${datab}" -eq 1 && test "${leafb}" -eq 0 && test "${freeb}" -eq 0) || _fail "failed to create ${dtype} dir ino ${inode} datab ${datab} leafb ${leafb} freeb ${freeb}" + (test "${datab}" -eq 1 && test "${leafb}" -eq 0 && test "${freeb}" -eq 0) || __populate_fail "failed to create ${dtype} dir ino ${inode} datab ${datab} leafb ${leafb} freeb ${freeb}" ;; "leaf") - (test "${datab}" -eq 1 && test "${leafb}" -eq 1 && test "${freeb}" -eq 0) || _fail "failed to create ${dtype} dir ino ${inode} datab ${datab} leafb ${leafb} freeb ${freeb}" + (test "${datab}" -eq 1 && test "${leafb}" -eq 1 && test "${freeb}" -eq 0) || __populate_fail "failed to create ${dtype} dir ino ${inode} datab ${datab} leafb ${leafb} freeb ${freeb}" ;; "leafn") _scratch_xfs_db -x -c "inode ${inode}" -c "dblock ${leaf_lblk}" -c "p lhdr.info.hdr.magic" | grep -q '0x3dff' && return _scratch_xfs_db -x -c "inode ${inode}" -c "dblock ${leaf_lblk}" -c "p lhdr.info.magic" | grep -q '0xd2ff' && return - _fail "failed to create ${dtype} dir ino ${inode} datab ${datab} leafb ${leafb} freeb ${freeb}" + __populate_fail "failed to create ${dtype} dir ino ${inode} datab ${datab} leafb ${leafb} freeb ${freeb}" ;; "node"|"btree") - (test "${datab}" -eq 1 && test "${leafb}" -eq 1 && test "${freeb}" -eq 1) || _fail "failed to create ${dtype} dir ino ${inode} datab ${datab} leafb ${leafb} freeb ${freeb}" + (test "${datab}" -eq 1 && test "${leafb}" -eq 1 && test "${freeb}" -eq 1) || __populate_fail "failed to create ${dtype} dir ino ${inode} datab ${datab} leafb ${leafb} freeb ${freeb}" ;; *) _fail "Unknown directory type ${dtype}" @@ -563,13 +584,13 @@ __populate_check_xfs_attr() { case "${atype}" in "shortform"|"inline"|"local") - (test "${datab}" -eq 0 && test "${leafb}" -eq 0) || _fail "failed to create ${atype} attr ino ${inode} datab ${datab} leafb ${leafb}" + (test "${datab}" -eq 0 && test "${leafb}" -eq 0) || __populate_fail "failed to create ${atype} attr ino ${inode} datab ${datab} leafb ${leafb}" ;; "leaf") - (test "${datab}" -eq 1 && test "${leafb}" -eq 0) || _fail "failed to create ${atype} attr ino ${inode} datab ${datab} leafb ${leafb}" + (test "${datab}" -eq 1 && test "${leafb}" -eq 0) || __populate_fail "failed to create ${atype} attr ino ${inode} datab ${datab} leafb ${leafb}" ;; "node"|"btree") - (test "${datab}" -eq 1 && test "${leafb}" -eq 1) || _fail "failed to create ${atype} attr ino ${inode} datab ${datab} leafb ${leafb}" + (test "${datab}" -eq 1 && test "${leafb}" -eq 1) || __populate_fail "failed to create ${atype} attr ino ${inode} datab ${datab} leafb ${leafb}" ;; *) _fail "Unknown attribute type ${atype}" @@ -605,7 +626,7 @@ __populate_check_xfs_agbtree_height() { return 100 fi done - test $? -eq 100 || _fail "Failed to create ${bt_type} of sufficient height!" + test $? -eq 100 || __populate_fail "Failed to create ${bt_type} of sufficient height!" return 1 } @@ -678,13 +699,13 @@ __populate_check_ext4_dformat() { case "${format}" in "blockmap") - test "${extents}" -eq 0 || _fail "failed to create ino ${inode} with blockmap" + test "${extents}" -eq 0 || __populate_fail "failed to create ino ${inode} with blockmap" ;; "extent"|"extents") - test "${extents}" -eq 1 || _fail "failed to create ino ${inode} with extents" + test "${extents}" -eq 1 || __populate_fail "failed to create ino ${inode} with extents" ;; "etree") - (test "${extents}" -eq 1 && test "${etree}" -eq 1) || _fail "failed to create ino ${inode} with extent tree" + (test "${extents}" -eq 1 && test "${etree}" -eq 1) || __populate_fail "failed to create ino ${inode} with extent tree" ;; *) _fail "Unknown dformat ${format}" @@ -702,10 +723,10 @@ __populate_check_ext4_aformat() { case "${format}" in "local"|"inline") - test "${ablock}" -eq 0 || _fail "failed to create inode ${inode} with ${format} xattr" + test "${ablock}" -eq 0 || __populate_fail "failed to create inode ${inode} with ${format} xattr" ;; "block") - test "${extents}" -eq 1 || _fail "failed to create inode ${inode} with ${format} xattr" + test "${extents}" -eq 1 || __populate_fail "failed to create inode ${inode} with ${format} xattr" ;; *) _fail "Unknown aformat ${format}" @@ -726,13 +747,13 @@ __populate_check_ext4_dir() { case "${dtype}" in "inline") - (test "${inline}" -eq 1 && test "${htree}" -eq 0) || _fail "failed to create ${dtype} dir ino ${inode} htree ${htree} inline ${inline}" + (test "${inline}" -eq 1 && test "${htree}" -eq 0) || __populate_fail "failed to create ${dtype} dir ino ${inode} htree ${htree} inline ${inline}" ;; "block") - (test "${inline}" -eq 0 && test "${htree}" -eq 0) || _fail "failed to create ${dtype} dir ino ${inode} htree ${htree} inline ${inline}" + (test "${inline}" -eq 0 && test "${htree}" -eq 0) || __populate_fail "failed to create ${dtype} dir ino ${inode} htree ${htree} inline ${inline}" ;; "htree") - (test "${inline}" -eq 0 && test "${htree}" -eq 1) || _fail "failed to create ${dtype} dir ino ${inode} htree ${htree} inline ${inline}" + (test "${inline}" -eq 0 && test "${htree}" -eq 1) || __populate_fail "failed to create ${dtype} dir ino ${inode} htree ${htree} inline ${inline}" ;; *) _fail "Unknown directory type ${dtype}" ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 2/2] populate: fix some weirdness in __populate_check_xfs_agbtree_height 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] populate: take a snapshot of the filesystem if creation fails Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-18 6:16 ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Zorro Lang 2 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Use a for loop to scan the AGs, and make all the variables local like you'd expect them to be. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/populate | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/common/populate b/common/populate index e4090a29d3..29ea637ecb 100644 --- a/common/populate +++ b/common/populate @@ -599,8 +599,8 @@ __populate_check_xfs_attr() { # Check that there's at least one per-AG btree with multiple levels __populate_check_xfs_agbtree_height() { - bt_type="$1" - nr_ags=$(_scratch_xfs_db -c 'sb 0' -c 'p agcount' | awk '{print $3}') + local bt_type="$1" + local agcount=$(_scratch_xfs_db -c 'sb 0' -c 'p agcount' | awk '{print $3}') case "${bt_type}" in "bno"|"cnt"|"rmap"|"refcnt") @@ -620,13 +620,14 @@ __populate_check_xfs_agbtree_height() { ;; esac - seq 0 $((nr_ags - 1)) | while read ag; do - bt_level=$(_scratch_xfs_db -c "${hdr} ${ag}" -c "p ${bt_prefix}level" | awk '{print $3}') + for ((agno = 0; agno < agcount; agno++)); do + bt_level=$(_scratch_xfs_db -c "${hdr} ${agno}" -c "p ${bt_prefix}level" | awk '{print $3}') + # "level" is really the btree height if [ "${bt_level}" -gt 1 ]; then - return 100 + return 0 fi done - test $? -eq 100 || __populate_fail "Failed to create ${bt_type} of sufficient height!" + __populate_fail "Failed to create ${bt_type} of sufficient height!" return 1 } ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] populate: take a snapshot of the filesystem if creation fails Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] populate: fix some weirdness in __populate_check_xfs_agbtree_height Darrick J. Wong @ 2023-02-18 6:16 ` Zorro Lang 2 siblings, 0 replies; 107+ messages in thread From: Zorro Lang @ 2023-02-18 6:16 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:36PM -0800, Darrick J. Wong wrote: > Hi all, > > Before we start on an intense patchset of improving the XFS fuzz testing > framework, let's fix a couple of bugs in the code that creates sample > filesystems with all types of metadata. > > If you're going to start using this mess, you probably ought to just > pull from my git trees, which are linked below. > > This is an extraordinary way to destroy everything. Enjoy! > Comments and questions are, as always, welcome. > > --D Ack, Reviewed-by: Zorro Lang <zlang@redhat.com> > > fstests git tree: > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fix-populate-problems > --- > common/populate | 70 ++++++++++++++++++++++++++++++++++++------------------- > 1 file changed, 46 insertions(+), 24 deletions(-) > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (10 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 01/24] fuzzy: disable per-field random fuzzing by default Darrick J. Wong ` (23 more replies) 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: strengthen fuzz testing Darrick J. Wong ` (5 subsequent siblings) 17 siblings, 24 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, There are a ton of improvements to the XFS fuzzing code in this update. We start by disabling by default two parts of the fuzz testing that don't lead to predictable golden output: fuzzing with the 'random' verb, and fuzzing the 'LSN' field. Next, refactor the inner fuzzing loop so that each of the four repair strategies are broken out into separate functions, as well as the post-repair attempts to modify the filesystem. This splitting makes it much easier to fix some longstanding problems with the fuzzer logic. We also revise the strategies a bit so that they more accurately reflect the intended usage patterns of the repair programs. Then, strengthen other parts of the fuzzing -- we make the post-repair modification exercises a bit more strenuous by running fsstress on the repaired fs; adding an evaluation of xfs_check vs. xfs_repair; and making it possible to check the xfs health reporting system. The xfs_check changes were key to disabling it in fstests in 2021. Finally, improve the array handling of the xfs fuzz tests so that we actually know about array indices as a first class concept, instead of the current mucking around we do with regular expressions. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fuzzer-improvements --- common/fuzzy | 513 ++++++++++++++++++++++++++++++++++++++++++++------------- common/xfs | 80 +++++++++ tests/xfs/354 | 7 + tests/xfs/355 | 7 + tests/xfs/358 | 5 - tests/xfs/359 | 5 - tests/xfs/360 | 5 - tests/xfs/361 | 5 - tests/xfs/362 | 5 - tests/xfs/363 | 5 - tests/xfs/364 | 5 - tests/xfs/365 | 5 - tests/xfs/366 | 5 - tests/xfs/367 | 5 - tests/xfs/368 | 5 - tests/xfs/369 | 5 - tests/xfs/370 | 5 - tests/xfs/371 | 5 - tests/xfs/372 | 5 - tests/xfs/373 | 7 + tests/xfs/410 | 5 - tests/xfs/411 | 5 - tests/xfs/455 | 7 + tests/xfs/457 | 5 - tests/xfs/458 | 5 - tests/xfs/459 | 5 - tests/xfs/460 | 5 - tests/xfs/461 | 5 - tests/xfs/462 | 5 - tests/xfs/463 | 5 - tests/xfs/464 | 7 + tests/xfs/483 | 5 - 32 files changed, 606 insertions(+), 147 deletions(-) ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 01/24] fuzzy: disable per-field random fuzzing by default 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 02/24] fuzzy: disable timstamp " Darrick J. Wong ` (22 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Don't run the random fuzzer by default so that we can try to stabilize the output somewhat. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/common/fuzzy b/common/fuzzy index d4177c3136..cd6e2a0e08 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -284,7 +284,8 @@ _scratch_xfs_list_fuzz_verbs() { return; fi _scratch_xfs_db -x -c 'sb 0' -c 'fuzz' | grep '^Fuzz commands:' | \ - sed -e 's/[,.]//g' -e 's/Fuzz commands: //g' -e 's/ /\n/g' + sed -e 's/[,.]//g' -e 's/Fuzz commands: //g' -e 's/ /\n/g' | \ + grep -v '^random$' } # Fuzz some of the fields of some piece of metadata ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 02/24] fuzzy: disable timstamp fuzzing by default 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong 2022-12-30 22:19 ` [PATCH 01/24] fuzzy: disable per-field random fuzzing by default Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 07/24] fuzzy: don't fuzz xattr namespace flags and values Darrick J. Wong ` (21 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Don't fuzz timestamps since all bit patterns are valid and XFS itself does not perform any validation on them. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/common/fuzzy b/common/fuzzy index cd6e2a0e08..2798c257a0 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -73,6 +73,14 @@ __filter_xfs_db_keys() { -e '/pad/d' } +# Filter out metadata fields that are completely controlled by userspace +# or are arbitrary bit sequences. In other words, fields where the filesystem +# does no validation. +__filter_unvalidated_xfs_db_fields() { + sed -e '/\.sec/d' \ + -e '/\.nsec/d' +} + # Filter the xfs_db print command's field debug information # into field name and type. __filter_xfs_db_print_fields() { @@ -91,7 +99,7 @@ __filter_xfs_db_print_fields() { else echo "${fuzzkey}" fi - done | grep -E "${filter}" + done | grep -E "${filter}" | __filter_unvalidated_xfs_db_fields } # Navigate to some part of the filesystem and print the field info. ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 07/24] fuzzy: don't fuzz xattr namespace flags and values 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong 2022-12-30 22:19 ` [PATCH 01/24] fuzzy: disable per-field random fuzzing by default Darrick J. Wong 2022-12-30 22:19 ` [PATCH 02/24] fuzzy: disable timstamp " Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 05/24] fuzzy: don't fuzz inode generation numbers Darrick J. Wong ` (20 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Extended attribute namespace flags are controlled by userspace, and there is no validation imposed on the values. Don't bother fuzzing either of these things. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/common/fuzzy b/common/fuzzy index 6f5083041a..866901931e 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -92,7 +92,13 @@ __filter_unvalidated_xfs_db_fields() { -e '/^core.noatime/d' \ -e '/^core.nodump/d' \ -e '/^core.nodefrag/d' \ - -e '/^v3.dax/d' + -e '/^v3.dax/d' \ + -e '/^nvlist.*value/d' \ + -e '/^entries.*root/d' \ + -e '/^entries.*secure/d' \ + -e '/^a.sfattr.list.*value/d' \ + -e '/^a.sfattr.list.*root/d' \ + -e '/^a.sfattr.list.*secure/d' } # Filter the xfs_db print command's field debug information ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 05/24] fuzzy: don't fuzz inode generation numbers 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 07/24] fuzzy: don't fuzz xattr namespace flags and values Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 03/24] fuzzy: don't fuzz the log sequence number Darrick J. Wong ` (19 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The inode generation number is a randomly selected 32-bit integer that isn't itself validated anywhere. No need to fuzz that. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/common/fuzzy b/common/fuzzy index d183052a14..0d7e60a011 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -83,7 +83,8 @@ __filter_unvalidated_xfs_db_fields() { -e '/\.lsn/d' \ -e '/^core.flushiter/d' \ -e '/^core.dmevmask/d' \ - -e '/^core.dmstate/d' + -e '/^core.dmstate/d' \ + -e '/^core.gen/d' } # Filter the xfs_db print command's field debug information ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 03/24] fuzzy: don't fuzz the log sequence number 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 05/24] fuzzy: don't fuzz inode generation numbers Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 06/24] fuzzy: don't fuzz user-controllable inode flags Darrick J. Wong ` (18 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Don't bother filtering log sequence numbers since xfs_db doesn't have the ability to tell us the range of LSNs that would actually cause validation failures. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/common/fuzzy b/common/fuzzy index 2798c257a0..677e655d68 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -78,7 +78,9 @@ __filter_xfs_db_keys() { # does no validation. __filter_unvalidated_xfs_db_fields() { sed -e '/\.sec/d' \ - -e '/\.nsec/d' + -e '/\.nsec/d' \ + -e '/^lsn$/d' \ + -e '/\.lsn/d' } # Filter the xfs_db print command's field debug information ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 06/24] fuzzy: don't fuzz user-controllable inode flags 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (4 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 03/24] fuzzy: don't fuzz the log sequence number Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 04/24] fuzzy: don't fuzz obsolete inode fields Darrick J. Wong ` (17 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Don't fuzz the inode flags that are controlled by userspace and don't actually have any other effects on the ondisk metadata. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/common/fuzzy b/common/fuzzy index 0d7e60a011..6f5083041a 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -84,7 +84,15 @@ __filter_unvalidated_xfs_db_fields() { -e '/^core.flushiter/d' \ -e '/^core.dmevmask/d' \ -e '/^core.dmstate/d' \ - -e '/^core.gen/d' + -e '/^core.gen/d' \ + -e '/^core.prealloc/d' \ + -e '/^core.immutable/d' \ + -e '/^core.append/d' \ + -e '/^core.sync/d' \ + -e '/^core.noatime/d' \ + -e '/^core.nodump/d' \ + -e '/^core.nodefrag/d' \ + -e '/^v3.dax/d' } # Filter the xfs_db print command's field debug information ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 04/24] fuzzy: don't fuzz obsolete inode fields 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (5 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 06/24] fuzzy: don't fuzz user-controllable inode flags Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 12/24] common/fuzzy: fix some problems with the offline repair strategy Darrick J. Wong ` (16 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> We don't really care about inode fields were used in V4 (deprecated) or DMAPI (unsupported) so don't bother fuzzing them. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/common/fuzzy b/common/fuzzy index 677e655d68..d183052a14 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -80,7 +80,10 @@ __filter_unvalidated_xfs_db_fields() { sed -e '/\.sec/d' \ -e '/\.nsec/d' \ -e '/^lsn$/d' \ - -e '/\.lsn/d' + -e '/\.lsn/d' \ + -e '/^core.flushiter/d' \ + -e '/^core.dmevmask/d' \ + -e '/^core.dmstate/d' } # Filter the xfs_db print command's field debug information ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 12/24] common/fuzzy: fix some problems with the offline repair strategy 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (6 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 04/24] fuzzy: don't fuzz obsolete inode fields Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 11/24] common/fuzzy: fix some problems with the online " Darrick J. Wong ` (15 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> While auditing the fuzz tester code, I noticed there were numerous problems with the offline repair strategy -- the stages of the strategy are not consistently logged to the kernel log, we don't actually check that repair -n finds the fuzzed field, and since this is an offline test, we don't need or want to mount or try to run the online scrubber. Also, disable prefetch to reduce the chance of an OOM kill. Rework the error messages to make reading the golden output easier. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 31 ++++++++++--------------------- 1 file changed, 10 insertions(+), 21 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index 8b52d289d1..07f597627c 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -246,37 +246,26 @@ __scratch_xfs_fuzz_field_online() { __scratch_xfs_fuzz_field_offline() { local fuzz_action="$1" - # Mount or else we can't do anything offline - echo "+ Mount filesystem to try offline repair" - _try_scratch_mount 2>&1 - res=$? - if [ $res -ne 0 ]; then - (>&2 echo "mount failed ($res) with ${fuzz_action}.") - return 0 - fi - - # Make sure online scrub will catch whatever we fuzzed - echo "++ Online scrub" - _scratch_scrub -n -a 1 -e continue 2>&1 + # Make sure offline scrub will catch whatever we fuzzed + __fuzz_notify "+ Detect fuzzed field (offline)" + _scratch_xfs_repair -P -n 2>&1 res=$? test $res -eq 0 && \ - (>&2 echo "scrub didn't fail with ${fuzz_action}.") - - __scratch_xfs_fuzz_unmount + (>&2 echo "${fuzz_action}: offline scrub didn't fail.") # Repair the filesystem offline - echo "+ Try to repair the filesystem offline" - _repair_scratch_fs 2>&1 + __fuzz_notify "+ Try to repair the filesystem (offline)" + _repair_scratch_fs -P 2>&1 res=$? test $res -ne 0 && \ - (>&2 echo "offline repair failed ($res) with ${fuzz_action}.") + (>&2 echo "${fuzz_action}: offline repair failed ($res).") # See if repair finds a clean fs - echo "+ Make sure error is gone (offline)" - _scratch_xfs_repair -n 2>&1 + __fuzz_notify "+ Make sure error is gone (offline)" + _scratch_xfs_repair -P -n 2>&1 res=$? test $res -ne 0 && \ - (>&2 echo "offline re-scrub failed ($res) with ${fuzz_action}.") + (>&2 echo "${fuzz_action}: offline re-scrub failed ($res).") return 0 } ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 11/24] common/fuzzy: fix some problems with the online repair strategy 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (7 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 12/24] common/fuzzy: fix some problems with the offline repair strategy Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 08/24] common/fuzzy: split out each repair strategy into a separate helper Darrick J. Wong ` (14 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> While auditing the fuzz tester code, I noticed there were numerous problems with the online repair strategy -- the stages of the strategy are not consistently logged to the kernel log, some of the error messages don't identify /which/ scrubber we're calling, and we don't actually re-run online scrub after a repair to make sure that it's verification is ok. Disable xfs_repair prefetch to reduce the chances of an OOM kill, and abort the fuzz test if we can't mount. We also reorganize the error messages to make reading the golden output easier. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index e90f414d34..8b52d289d1 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -201,36 +201,43 @@ __scratch_xfs_fuzz_field_online() { local fuzz_action="$1" # Mount or else we can't do anything online - echo "+ Mount filesystem to try online repair" + __fuzz_notify "+ Mount filesystem to try online repair" _try_scratch_mount 2>&1 res=$? if [ $res -ne 0 ]; then - (>&2 echo "mount failed ($res) with ${fuzz_action}.") - return 0 + (>&2 echo "${fuzz_action}: mount failed ($res).") + return 1 fi # Make sure online scrub will catch whatever we fuzzed - echo "++ Online scrub" + __fuzz_notify "++ Detect fuzzed field (online)" _scratch_scrub -n -a 1 -e continue 2>&1 res=$? test $res -eq 0 && \ - (>&2 echo "scrub didn't fail with ${fuzz_action}.") + (>&2 echo "${fuzz_action}: online scrub didn't fail.") # Try fixing the filesystem online - __fuzz_notify "++ Try to repair filesystem online" + __fuzz_notify "++ Try to repair filesystem (online)" _scratch_scrub 2>&1 res=$? test $res -ne 0 && \ - (>&2 echo "online repair failed ($res) with ${fuzz_action}.") + (>&2 echo "${fuzz_action}: online repair failed ($res).") + + # Online scrub should pass now + __fuzz_notify "++ Make sure error is gone (online)" + _scratch_scrub -n -a 1 -e continue 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "${fuzz_action}: online re-scrub failed ($res).") __scratch_xfs_fuzz_unmount # Offline scrub should pass now - echo "+ Make sure error is gone (offline)" - _scratch_xfs_repair -n 2>&1 + __fuzz_notify "+ Make sure error is gone (offline)" + _scratch_xfs_repair -P -n 2>&1 res=$? test $res -ne 0 && \ - (>&2 echo "offline re-scrub failed ($res) with ${fuzz_action}.") + (>&2 echo "${fuzz_action}: offline re-scrub failed ($res).") return 0 } ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 08/24] common/fuzzy: split out each repair strategy into a separate helper 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (8 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 11/24] common/fuzzy: fix some problems with the online " Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 10/24] common/fuzzy: hoist the post-repair fs modification step Darrick J. Wong ` (13 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Refactor __scratch_xfs_fuzz_field_test to split out each repair strategy into a separate helper function. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 212 +++++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 166 insertions(+), 46 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index 866901931e..ef42336fa6 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -194,6 +194,148 @@ __fuzz_notify() { test -w /dev/ttyprintk && echo "$@" >> /dev/ttyprintk } +# Perform the online repair part of a fuzz test. +__scratch_xfs_fuzz_field_online() { + local fuzz_action="$1" + + # Mount or else we can't do anything online + echo "+ Mount filesystem to try online repair" + _try_scratch_mount 2>&1 + res=$? + if [ $res -ne 0 ]; then + (>&2 echo "mount failed ($res) with ${fuzz_action}.") + return 0 + fi + + # Make sure online scrub will catch whatever we fuzzed + echo "++ Online scrub" + _scratch_scrub -n -a 1 -e continue 2>&1 + res=$? + test $res -eq 0 && \ + (>&2 echo "scrub didn't fail with ${fuzz_action}.") + + # Try fixing the filesystem online + __fuzz_notify "++ Try to repair filesystem online" + _scratch_scrub 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "online repair failed ($res) with ${fuzz_action}.") + + __scratch_xfs_fuzz_unmount + + # Offline scrub should pass now + echo "+ Make sure error is gone (offline)" + _scratch_xfs_repair -n 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "offline re-scrub failed ($res) with ${fuzz_action}.") + + return 0 +} + +# Perform the offline repair part of a fuzz test. +__scratch_xfs_fuzz_field_offline() { + local fuzz_action="$1" + + # Mount or else we can't do anything offline + echo "+ Mount filesystem to try offline repair" + _try_scratch_mount 2>&1 + res=$? + if [ $res -ne 0 ]; then + (>&2 echo "mount failed ($res) with ${fuzz_action}.") + return 0 + fi + + # Make sure online scrub will catch whatever we fuzzed + echo "++ Online scrub" + _scratch_scrub -n -a 1 -e continue 2>&1 + res=$? + test $res -eq 0 && \ + (>&2 echo "scrub didn't fail with ${fuzz_action}.") + + __scratch_xfs_fuzz_unmount + + # Repair the filesystem offline + echo "+ Try to repair the filesystem offline" + _repair_scratch_fs 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "offline repair failed ($res) with ${fuzz_action}.") + + # See if repair finds a clean fs + echo "+ Make sure error is gone (offline)" + _scratch_xfs_repair -n 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "offline re-scrub failed ($res) with ${fuzz_action}.") + + return 0 +} + +# Perform the no-repair part of a fuzz test. +__scratch_xfs_fuzz_field_norepair() { + local fuzz_action="$1" + + # Mount or else we can't do anything in norepair mode + echo "+ Mount filesystem to try no repair" + _try_scratch_mount 2>&1 + res=$? + if [ $res -ne 0 ]; then + (>&2 echo "mount failed ($res) with ${fuzz_action}.") + return 0 + fi + + __scratch_xfs_fuzz_unmount + + return 0 +} + +# Perform the online-then-offline repair part of a fuzz test. +__scratch_xfs_fuzz_field_both() { + local fuzz_action="$1" + + # Mount or else we can't do anything in both repair mode + echo "+ Mount filesystem to try both repairs" + _try_scratch_mount 2>&1 + res=$? + if [ $res -ne 0 ]; then + (>&2 echo "mount failed ($res) with ${fuzz_action}.") + return 0 + fi + + # Make sure online scrub will catch whatever we fuzzed + echo "++ Online scrub" + _scratch_scrub -n -a 1 -e continue 2>&1 + res=$? + test $res -eq 0 && \ + (>&2 echo "online scrub didn't fail with ${fuzz_action}.") + + # Try fixing the filesystem online + __fuzz_notify "++ Try to repair filesystem online" + _scratch_scrub 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "online repair failed ($res) with ${fuzz_action}.") + + __scratch_xfs_fuzz_unmount + + # Repair the filesystem offline? + echo "+ Try to repair the filesystem offline" + _repair_scratch_fs 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "offline repair failed ($res) with ${fuzz_action}.") + + # See if repair finds a clean fs + echo "+ Make sure error is gone (offline)" + _scratch_xfs_repair -n 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "offline re-scrub ($res) with ${fuzz_action}.") + + return 0 +} + # Fuzz one field of some piece of metadata. # First arg is the field name # Second arg is the fuzz verb (ones, zeroes, random, add, sub...) @@ -211,54 +353,32 @@ __scratch_xfs_fuzz_field_test() { res=$? test $res -ne 0 && return - # Try to catch the error with scrub - echo "+ Try to catch the error" - _try_scratch_mount 2>&1 - res=$? - if [ $res -eq 0 ]; then - # Try an online scrub unless we're fuzzing ag 0's sb, - # which scrub doesn't know how to fix. - if [ "${repair}" != "none" ]; then - echo "++ Online scrub" - if [ "$1" != "sb 0" ]; then - _scratch_scrub -n -a 1 -e continue 2>&1 - res=$? - test $res -eq 0 && \ - (>&2 echo "scrub didn't fail with ${field} = ${fuzzverb}.") - fi - fi - - # Try fixing the filesystem online?! - if [ "${repair}" = "online" ] || [ "${repair}" = "both" ]; then - __fuzz_notify "++ Try to repair filesystem online" - _scratch_scrub 2>&1 - res=$? - test $res -ne 0 && \ - (>&2 echo "online repair failed ($res) with ${field} = ${fuzzverb}.") - fi - - __scratch_xfs_fuzz_unmount - elif [ "${repair}" = "online" ] || [ "${repair}" = "both" ]; then - (>&2 echo "mount failed ($res) with ${field} = ${fuzzverb}.") - fi - - # Repair the filesystem offline? - if [ "${repair}" = "offline" ] || [ "${repair}" = "both" ]; then - echo "+ Try to repair the filesystem offline" - _repair_scratch_fs 2>&1 + # Try to catch the error with whatever repair strategy we picked. + # The fs should not be mounted before or after the strategy call. + local fuzz_action="${field} = ${fuzzverb}" + case "${repair}" in + "online") + __scratch_xfs_fuzz_field_online "${fuzz_action}" res=$? - test $res -ne 0 && \ - (>&2 echo "offline repair failed ($res) with ${field} = ${fuzzverb}.") - fi - - # See if repair finds a clean fs - if [ "${repair}" != "none" ]; then - echo "+ Make sure error is gone (offline)" - _scratch_xfs_repair -n 2>&1 + ;; + "offline") + __scratch_xfs_fuzz_field_offline "${fuzz_action}" res=$? - test $res -ne 0 && \ - (>&2 echo "offline re-scrub ($res) with ${field} = ${fuzzverb}.") - fi + ;; + "none") + __scratch_xfs_fuzz_field_norepair "${fuzz_action}" + res=$? + ;; + "both") + __scratch_xfs_fuzz_field_both "${fuzz_action}" + res=$? + ;; + *) + (>&2 echo "unknown repair strategy ${repair}.") + res=2 + ;; + esac + test $res -eq 0 || return $res # See if scrub finds a clean fs echo "+ Make sure error is gone (online)" ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 10/24] common/fuzzy: hoist the post-repair fs modification step 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (9 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 08/24] common/fuzzy: split out each repair strategy into a separate helper Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 09/24] common/fuzzy: add an underline to the full log between sections Darrick J. Wong ` (12 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Hoist the code that tries to modify an fs after repairing our fuzz damage into a separate function, so that we can further simplify the caller. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 77 +++++++++++++++++++++++++++++++++------------------------- 1 file changed, 44 insertions(+), 33 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index 7efa5eeaf7..e90f414d34 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -338,6 +338,47 @@ __scratch_xfs_fuzz_field_both() { return 0 } +# Assess the state of the filesystem after a repair strategy has been run by +# trying to make changes to it. +_scratch_xfs_fuzz_field_modifyfs() { + local fuzz_action="$1" + local repair="$2" + + # Try to mount the filesystem + echo "+ Make sure error is gone (online)" + _try_scratch_mount 2>&1 + res=$? + if [ $res -eq 0 ]; then + # Make sure online scrub says the filesystem is clean now + if [ "${repair}" != "none" ]; then + echo "++ Online scrub" + _scratch_scrub -n -e continue 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "online re-scrub ($res) with ${field} = ${fuzzverb}.") + fi + fi + + # Try modifying the filesystem again + __fuzz_notify "++ Try to write filesystem again" + _scratch_fuzz_modify 100 2>&1 + __scratch_xfs_fuzz_unmount + else + (>&2 echo "re-mount failed ($res) with ${fuzz_action}.") + fi + + # See if repair finds a clean fs + if [ "${repair}" != "none" ]; then + echo "+ Re-check the filesystem (offline)" + _scratch_xfs_repair -n 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "re-repair failed ($res) with ${field} = ${fuzzverb}.") + fi + + return 0 +} + # Fuzz one field of some piece of metadata. # First arg is the field name # Second arg is the fuzz verb (ones, zeroes, random, add, sub...) @@ -381,39 +422,9 @@ __scratch_xfs_fuzz_field_test() { esac test $res -eq 0 || return $res - # See if scrub finds a clean fs - echo "+ Make sure error is gone (online)" - _try_scratch_mount 2>&1 - res=$? - if [ $res -eq 0 ]; then - # Try an online scrub unless we're fuzzing ag 0's sb, - # which scrub doesn't know how to fix. - if [ "${repair}" != "none" ]; then - echo "++ Online scrub" - if [ "$1" != "sb 0" ]; then - _scratch_scrub -n -e continue 2>&1 - res=$? - test $res -ne 0 && \ - (>&2 echo "online re-scrub ($res) with ${field} = ${fuzzverb}.") - fi - fi - - # Try modifying the filesystem again! - __fuzz_notify "++ Try to write filesystem again" - _scratch_fuzz_modify 100 2>&1 - __scratch_xfs_fuzz_unmount - else - (>&2 echo "re-mount failed ($res) with ${field} = ${fuzzverb}.") - fi - - # See if repair finds a clean fs - if [ "${repair}" != "none" ]; then - echo "+ Re-check the filesystem (offline)" - _scratch_xfs_repair -n 2>&1 - res=$? - test $res -ne 0 && \ - (>&2 echo "re-repair failed ($res) with ${field} = ${fuzzverb}.") - fi + # See what happens when we modify the fs + _scratch_xfs_fuzz_field_modifyfs "${fuzz_action}" "${repair}" + return $? } # Make sure we have all the pieces we need for field fuzzing ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 09/24] common/fuzzy: add an underline to the full log between sections 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (10 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 10/24] common/fuzzy: hoist the post-repair fs modification step Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 13/24] common/fuzzy: fix some problems with the no-repair strategy Darrick J. Wong ` (11 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The fuzz scripts use __fuzz_notify in effect to log each step in the fuzz process. Enhance it to print an "underline" to ease readability a bit. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/common/fuzzy b/common/fuzzy index ef42336fa6..7efa5eeaf7 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -190,7 +190,9 @@ __scratch_xfs_fuzz_mdrestore() } __fuzz_notify() { + echo '========================================' echo "$@" + echo '========================================' test -w /dev/ttyprintk && echo "$@" >> /dev/ttyprintk } @@ -348,7 +350,6 @@ __scratch_xfs_fuzz_field_test() { # Set the new field value __fuzz_notify "+ Fuzz ${field} = ${fuzzverb}" - echo "========================" _scratch_xfs_fuzz_metadata_field "${field}" ${fuzzverb} "$@" res=$? test $res -ne 0 && return ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 13/24] common/fuzzy: fix some problems with the no-repair strategy 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (11 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 09/24] common/fuzzy: add an underline to the full log between sections Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 14/24] common/fuzzy: fix some problems with the online-then-offline repair strategy Darrick J. Wong ` (10 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> While auditing the fuzz tester code, I noticed there were numerous problems with the no repair strategy -- the stages of the strategy are not consistently logged to the kernel log, and we don't actually verify that either online or offline scrubs notice the fuzz. Rework the error messages to make reading the golden output easier. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index 07f597627c..16fca67534 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -274,15 +274,29 @@ __scratch_xfs_fuzz_field_offline() { __scratch_xfs_fuzz_field_norepair() { local fuzz_action="$1" + # Make sure offline scrub will catch whatever we fuzzed + __fuzz_notify "+ Detect fuzzed field (offline)" + _scratch_xfs_repair -P -n 2>&1 + res=$? + test $res -eq 0 && \ + (>&2 echo "${fuzz_action}: offline scrub didn't fail.") + # Mount or else we can't do anything in norepair mode - echo "+ Mount filesystem to try no repair" + __fuzz_notify "+ Mount filesystem to try online scan" _try_scratch_mount 2>&1 res=$? if [ $res -ne 0 ]; then - (>&2 echo "mount failed ($res) with ${fuzz_action}.") - return 0 + (>&2 echo "${fuzz_action}: mount failed ($res).") + return 1 fi + # Make sure online scrub will catch whatever we fuzzed + __fuzz_notify "++ Detect fuzzed field (online)" + _scratch_scrub -n -a 1 -e continue 2>&1 + res=$? + test $res -eq 0 && \ + (>&2 echo "${fuzz_action}: online scrub didn't fail.") + __scratch_xfs_fuzz_unmount return 0 ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 14/24] common/fuzzy: fix some problems with the online-then-offline repair strategy 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (12 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 13/24] common/fuzzy: fix some problems with the no-repair strategy Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 16/24] xfs/{35[45],455}: fix bogus corruption errors Darrick J. Wong ` (9 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> While auditing the fuzz tester code, I noticed there were numerous problems with the online-then-offline repair strategy -- the stages of the strategy are not consistently logged to the kernel log, some of the error messages don't identify /which/ scrubber we're calling, we don't do a pre-repair check to make sure we detect the fuzzed fields, and we don't actually re-run online scrub after a repair to make sure that it's ok. Disable xfs_repair prefetch to reduce the possibility of OOM kills. Rework the error messages to make reading the golden output easier. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 80 ++++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 53 insertions(+), 27 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index 16fca67534..a33c230b40 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -306,45 +306,71 @@ __scratch_xfs_fuzz_field_norepair() { __scratch_xfs_fuzz_field_both() { local fuzz_action="$1" + # Make sure offline scrub will catch whatever we fuzzed + __fuzz_notify "+ Detect fuzzed field (offline)" + _scratch_xfs_repair -P -n 2>&1 + res=$? + test $res -eq 0 && \ + (>&2 echo "${fuzz_action}: offline scrub didn't fail.") + # Mount or else we can't do anything in both repair mode - echo "+ Mount filesystem to try both repairs" + __fuzz_notify "+ Mount filesystem to try both repairs" _try_scratch_mount 2>&1 res=$? if [ $res -ne 0 ]; then - (>&2 echo "mount failed ($res) with ${fuzz_action}.") - return 0 + (>&2 echo "${fuzz_action}: mount failed ($res).") + else + # Make sure online scrub will catch whatever we fuzzed + __fuzz_notify "++ Detect fuzzed field (online)" + _scratch_scrub -n -a 1 -e continue 2>&1 + res=$? + test $res -eq 0 && \ + (>&2 echo "${fuzz_action}: online scrub didn't fail.") + + # Try fixing the filesystem online + __fuzz_notify "++ Try to repair filesystem (online)" + _scratch_scrub 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "${fuzz_action}: online repair failed ($res).") + + __scratch_xfs_fuzz_unmount + fi + + # Repair the filesystem offline if online repair failed? + if [ $res -ne 0 ]; then + __fuzz_notify "+ Try to repair the filesystem (offline)" + _repair_scratch_fs -P 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "${fuzz_action}: offline repair failed ($res).") + fi + + # See if repair finds a clean fs + __fuzz_notify "+ Make sure error is gone (offline)" + _scratch_xfs_repair -P -n 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "${fuzz_action}: offline re-scrub failed ($res).") + + # Mount so that we can see what scrub says after we've fixed the fs + __fuzz_notify "+ Re-mount filesystem to re-try online scan" + _try_scratch_mount 2>&1 + res=$? + if [ $res -ne 0 ]; then + (>&2 echo "${fuzz_action}: mount failed ($res).") + return 1 fi - # Make sure online scrub will catch whatever we fuzzed - echo "++ Online scrub" + # Online scrub should pass now + __fuzz_notify "++ Make sure error is gone (online)" _scratch_scrub -n -a 1 -e continue 2>&1 res=$? - test $res -eq 0 && \ - (>&2 echo "online scrub didn't fail with ${fuzz_action}.") - - # Try fixing the filesystem online - __fuzz_notify "++ Try to repair filesystem online" - _scratch_scrub 2>&1 - res=$? test $res -ne 0 && \ - (>&2 echo "online repair failed ($res) with ${fuzz_action}.") + (>&2 echo "${fuzz_action}: online re-scrub failed ($res).") __scratch_xfs_fuzz_unmount - # Repair the filesystem offline? - echo "+ Try to repair the filesystem offline" - _repair_scratch_fs 2>&1 - res=$? - test $res -ne 0 && \ - (>&2 echo "offline repair failed ($res) with ${fuzz_action}.") - - # See if repair finds a clean fs - echo "+ Make sure error is gone (offline)" - _scratch_xfs_repair -n 2>&1 - res=$? - test $res -ne 0 && \ - (>&2 echo "offline re-scrub ($res) with ${fuzz_action}.") - return 0 } ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 16/24] xfs/{35[45],455}: fix bogus corruption errors 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (13 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 14/24] common/fuzzy: fix some problems with the online-then-offline repair strategy Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 21/24] fuzzy: compress coredumps created while fuzzing Darrick J. Wong ` (8 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The AGFL fuzz tests first fuzz the entire block header, and second extract flfirst from the AGF header to start a second round of targeted fuzzing of live bno pointers in the AGFL. However, flfirst (and the AGFL field detection at the start of the second round of fuzzing) are detected after we've already been fuzz testing, which means that the AGFL might be garbage because repair failed or was not called. If this is the case, test will fail because the _scratch_xfs_db -c 'agf 0' -c 'p flfirst' call emits things like this: Fuzz AGFL flfirst Metadata corruption detected at 0x55f4f789fbc0, xfs_agfl block 0x3/0x200 Metadata corruption detected at 0x55b7356e0bc0, xfs_agfl block 0x3/0x200 Done fuzzing AGFL flfirst Fix this by restoring the scratch fs before probing flfirst and starting the second round of fuzzing. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/354 | 7 ++++++- tests/xfs/355 | 7 ++++++- tests/xfs/455 | 7 ++++++- 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/tests/xfs/354 b/tests/xfs/354 index b10ce1d68f..8abf527ea6 100755 --- a/tests/xfs/354 +++ b/tests/xfs/354 @@ -28,8 +28,13 @@ echo "Fuzz AGFL" _scratch_xfs_fuzz_metadata '' 'offline' 'agfl 0' >> $seqres.full echo "Done fuzzing AGFL" -echo "Fuzz AGFL flfirst" +# Restore a correct copy of the filesystem before we start the second round of +# fuzzing. This avoids corruption errors from xfs_db when we probe for flfirst +# in the AGF and later when _scratch_xfs_fuzz_metadata probes the AGFL fields. +__scratch_xfs_fuzz_mdrestore flfirst=$(_scratch_xfs_db -c 'agf 0' -c 'p flfirst' | sed -e 's/flfirst = //g') + +echo "Fuzz AGFL flfirst" SCRATCH_XFS_LIST_METADATA_FIELDS="bno[${flfirst}]" _scratch_xfs_fuzz_metadata '' 'offline' 'agfl 0' >> $seqres.full echo "Done fuzzing AGFL flfirst" diff --git a/tests/xfs/355 b/tests/xfs/355 index 530c9a970a..2d552a591c 100755 --- a/tests/xfs/355 +++ b/tests/xfs/355 @@ -28,8 +28,13 @@ echo "Fuzz AGFL" _scratch_xfs_fuzz_metadata '' 'online' 'agfl 0' >> $seqres.full echo "Done fuzzing AGFL" -echo "Fuzz AGFL flfirst" +# Restore a correct copy of the filesystem before we start the second round of +# fuzzing. This avoids corruption errors from xfs_db when we probe for flfirst +# in the AGF and later when _scratch_xfs_fuzz_metadata probes the AGFL fields. +__scratch_xfs_fuzz_mdrestore flfirst=$(_scratch_xfs_db -c 'agf 0' -c 'p flfirst' | sed -e 's/flfirst = //g') + +echo "Fuzz AGFL flfirst" SCRATCH_XFS_LIST_METADATA_FIELDS="bno[${flfirst}]" _scratch_xfs_fuzz_metadata '' 'online' 'agfl 0' >> $seqres.full echo "Done fuzzing AGFL flfirst" diff --git a/tests/xfs/455 b/tests/xfs/455 index 96820bc3b8..9f06c71fa2 100755 --- a/tests/xfs/455 +++ b/tests/xfs/455 @@ -29,8 +29,13 @@ echo "Fuzz AGFL" _scratch_xfs_fuzz_metadata '' 'none' 'agfl 0' >> $seqres.full echo "Done fuzzing AGFL" -echo "Fuzz AGFL flfirst" +# Restore a correct copy of the filesystem before we start the second round of +# fuzzing. This avoids corruption errors from xfs_db when we probe for flfirst +# in the AGF and later when _scratch_xfs_fuzz_metadata probes the AGFL fields. +__scratch_xfs_fuzz_mdrestore flfirst=$(_scratch_xfs_db -c 'agf 0' -c 'p flfirst' | sed -e 's/flfirst = //g') + +echo "Fuzz AGFL flfirst" SCRATCH_XFS_LIST_METADATA_FIELDS="bno[${flfirst}]" _scratch_xfs_fuzz_metadata '' 'none' 'agfl 0' >> $seqres.full echo "Done fuzzing AGFL flfirst" ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 21/24] fuzzy: compress coredumps created while fuzzing 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (14 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 16/24] xfs/{35[45],455}: fix bogus corruption errors Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 15/24] common/fuzzy: fix some problems with the post-repair fs modification code Darrick J. Wong ` (7 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Compress the coredumps and put them in the results directory. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/common/fuzzy b/common/fuzzy index 939f5e5ef2..7eaf883c0f 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -554,6 +554,14 @@ _scratch_xfs_fuzz_metadata() { echo "${verbs}" | while read fuzzverb; do __scratch_xfs_fuzz_mdrestore __scratch_xfs_fuzz_field_test "${field}" "${fuzzverb}" "${repair}" "$@" + + # Collect compresssed coredumps in the test results + # directory if the sysadmin didn't override the default + # coredump strategy. + for i in core core.*; do + test -f "$i" || continue + _save_coredump "$i" + done done done } ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 15/24] common/fuzzy: fix some problems with the post-repair fs modification code 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (15 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 21/24] fuzzy: compress coredumps created while fuzzing Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 19/24] common/fuzzy: exercise the filesystem a little harder after repairing Darrick J. Wong ` (6 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> While auditing the fuzz tester code, I noticed there were numerous problems with the code that test-drives the filesystem after we've run the repair strategy. Now that we've made sure that the repair strategy checks its own efficacy, we can rearrange this function to try making mods and then re-check the filesystem afterwards. Also, disable xfs_repair prefetch to reduce the likelihood of OOM kills. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 56 +++++++++++++++++++++++++++++++------------------------- 1 file changed, 31 insertions(+), 25 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index a33c230b40..e9a5d67592 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -380,37 +380,43 @@ _scratch_xfs_fuzz_field_modifyfs() { local fuzz_action="$1" local repair="$2" - # Try to mount the filesystem - echo "+ Make sure error is gone (online)" + # Try to mount the filesystem so that we can make changes + __fuzz_notify "+ Mount filesystem to make changes" _try_scratch_mount 2>&1 res=$? - if [ $res -eq 0 ]; then - # Make sure online scrub says the filesystem is clean now - if [ "${repair}" != "none" ]; then - echo "++ Online scrub" - _scratch_scrub -n -e continue 2>&1 - res=$? - test $res -ne 0 && \ - (>&2 echo "online re-scrub ($res) with ${field} = ${fuzzverb}.") - fi - fi + if [ $res -ne 0 ]; then + (>&2 echo "${fuzz_action}: pre-mod mount failed ($res).") + return $res + fi - # Try modifying the filesystem again - __fuzz_notify "++ Try to write filesystem again" - _scratch_fuzz_modify 100 2>&1 + # Try modifying the filesystem again + __fuzz_notify "++ Try to write filesystem again" + _scratch_fuzz_modify 100 2>&1 + + # If we didn't repair anything, there's no point in checking further, + # the fs is still corrupt. + if [ "${repair}" = "none" ]; then __scratch_xfs_fuzz_unmount - else - (>&2 echo "re-mount failed ($res) with ${fuzz_action}.") + return 0 fi - # See if repair finds a clean fs - if [ "${repair}" != "none" ]; then - echo "+ Re-check the filesystem (offline)" - _scratch_xfs_repair -n 2>&1 - res=$? - test $res -ne 0 && \ - (>&2 echo "re-repair failed ($res) with ${field} = ${fuzzverb}.") - fi + # Run an online check to make sure the fs is still ok, unless we + # are running the norepair strategy. + __fuzz_notify "+ Re-check the filesystem (online)" + _scratch_scrub -n -e continue 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "${fuzz_action}: online post-mod scrub failed ($res).") + + __scratch_xfs_fuzz_unmount + + # Run an offline check to make sure the fs is still ok, unless we + # are running the norepair strategy. + __fuzz_notify "+ Re-check the filesystem (offline)" + _scratch_xfs_repair -P -n 2>&1 + res=$? + test $res -ne 0 && \ + (>&2 echo "${fuzz_action}: offline post-mod scrub failed ($res).") return 0 } ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 19/24] common/fuzzy: exercise the filesystem a little harder after repairing 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (16 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 15/24] common/fuzzy: fix some problems with the post-repair fs modification code Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 22/24] fuzzy: report the fuzzing repair strategy in seqres.full Darrick J. Wong ` (5 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Use fsstress to exercise the filesystem a little more strenuously after we've run the fuzzing repair strategy, so that we have a better chance of tripping over corruption problems. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 36 ++++++++++++------------------------ 1 file changed, 12 insertions(+), 24 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index d841d435eb..3de6f43dc6 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -6,30 +6,18 @@ # Modify various files after a fuzzing operation _scratch_fuzz_modify() { - nr="$1" + echo "+++ stressing filesystem" + mkdir -p $SCRATCH_MNT/data + _xfs_force_bdev data $SCRATCH_MNT/data + $FSSTRESS_PROG -n $((TIME_FACTOR * 10000)) -p $((LOAD_FACTOR * 4)) -d $SCRATCH_MNT/data - test -z "${nr}" && nr=50000 - echo "+++ touch ${nr} files" - blk_sz=$(stat -f -c '%s' ${SCRATCH_MNT}) - $XFS_IO_PROG -f -c "pwrite -S 0x63 0 ${blk_sz}" "/tmp/afile" > /dev/null - date="$(date)" - find "${SCRATCH_MNT}/" -type f 2> /dev/null | head -n "${nr}" | while read f; do - # try to remove append, immutable (and even dax) flag if exists - $XFS_IO_PROG -rc 'chattr -x -i -a' "$f" > /dev/null 2>&1 - setfattr -n "user.date" -v "${date}" "$f" - cat "/tmp/afile" >> "$f" - mv "$f" "$f.longer" - done - sync - rm -rf "/tmp/afile" - - echo "+++ create files" - mkdir -p "${SCRATCH_MNT}/test.moo" - $XFS_IO_PROG -f -c 'pwrite -S 0x80 0 65536' "${SCRATCH_MNT}/test.moo/urk" > /dev/null - sync - - echo "+++ remove files" - rm -rf "${SCRATCH_MNT}/test.moo" + if _xfs_has_feature "$SCRATCH_MNT" realtime; then + mkdir -p $SCRATCH_MNT/rt + _xfs_force_bdev realtime $SCRATCH_MNT/rt + $FSSTRESS_PROG -n $((TIME_FACTOR * 10000)) -p $((LOAD_FACTOR * 4)) -d $SCRATCH_MNT/rt + else + echo "+++ xfs realtime not configured" + fi } # Try to access files after fuzzing @@ -429,7 +417,7 @@ _scratch_xfs_fuzz_field_modifyfs() { # Try modifying the filesystem again __fuzz_notify "++ Try to write filesystem again" - _scratch_fuzz_modify 100 2>&1 + _scratch_fuzz_modify 2>&1 # If we didn't repair anything, there's no point in checking further, # the fs is still corrupt. ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 22/24] fuzzy: report the fuzzing repair strategy in seqres.full 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (17 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 19/24] common/fuzzy: exercise the filesystem a little harder after repairing Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 17/24] common/fuzzy: evaluate xfs_check vs xfs_repair Darrick J. Wong ` (4 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Record in the seqres.full file the filesystem repair strategy that we're going to use to detect the fuzzed metadata. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/common/fuzzy b/common/fuzzy index 7eaf883c0f..f34fcadefe 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -540,7 +540,7 @@ _scratch_xfs_fuzz_metadata() { fields="$(_scratch_xfs_list_metadata_fields "${filter}" "$@")" verbs="$(_scratch_xfs_list_fuzz_verbs)" - echo "Fields we propose to fuzz under: $@" + echo "Fields we propose to fuzz with the \"${repair}\" repair strategy: $@" echo $(echo "${fields}") echo "Verbs we propose to fuzz with:" echo $(echo "${verbs}") ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 17/24] common/fuzzy: evaluate xfs_check vs xfs_repair 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (18 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 22/24] fuzzy: report the fuzzing repair strategy in seqres.full Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 20/24] fuzzy: dump metadata state before fuzzing Darrick J. Wong ` (3 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> When fuzzing a filesystem and using the offline repair strategy, compare the outputs of xfs_check against xfs_repair to ensure that the newer xfs_repair catches at least as many things as xfs_check does. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/common/fuzzy b/common/fuzzy index e9a5d67592..cf085f8b28 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -253,6 +253,17 @@ __scratch_xfs_fuzz_field_offline() { test $res -eq 0 && \ (>&2 echo "${fuzz_action}: offline scrub didn't fail.") + # Make sure xfs_repair catches at least as many things as the old + # xfs_check did. + if [ -n "${SCRATCH_XFS_FUZZ_CHECK}" ]; then + __fuzz_notify "+ Detect fuzzed field (xfs_check)" + _scratch_xfs_check 2>&1 + res1=$? + if [ $res1 -ne 0 ] && [ $res -eq 0 ]; then + (>&2 echo "${fuzz_action}: xfs_repair passed but xfs_check failed ($res1).") + fi + fi + # Repair the filesystem offline __fuzz_notify "+ Try to repair the filesystem (offline)" _repair_scratch_fs -P 2>&1 ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 20/24] fuzzy: dump metadata state before fuzzing 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (19 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 17/24] common/fuzzy: evaluate xfs_check vs xfs_repair Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 18/24] common: check xfs health after doing an online scrub Darrick J. Wong ` (2 subsequent siblings) 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> When we start a fuzz test, dump the metadata to stdout so that anyone analyzing a failure can see what was in the (supposedly) good image, and what it turns into after fuzzing. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/common/fuzzy b/common/fuzzy index 3de6f43dc6..939f5e5ef2 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -110,6 +110,16 @@ __filter_xfs_db_print_fields() { done | grep -E "${filter}" | __filter_unvalidated_xfs_db_fields } +# Dump the current contents of a metadata object. +# All arguments are xfs_db commands to locate the metadata. +_scratch_xfs_dump_metadata() { + local cmds=() + for arg in "$@"; do + cmds+=("-c" "${arg}") + done + _scratch_xfs_db "${cmds[@]}" -c print +} + # Navigate to some part of the filesystem and print the field info. # The first argument is an grep filter for the fields # The rest of the arguments are xfs_db commands to locate the metadata. @@ -534,6 +544,8 @@ _scratch_xfs_fuzz_metadata() { echo $(echo "${fields}") echo "Verbs we propose to fuzz with:" echo $(echo "${verbs}") + echo "Current metadata object state:" + _scratch_xfs_dump_metadata "$@" # Always capture full core dumps from crashing tools ulimit -c unlimited ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 18/24] common: check xfs health after doing an online scrub 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (20 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 20/24] fuzzy: dump metadata state before fuzzing Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 24/24] fuzzy: for fuzzing ag btrees, find the path to the AG header Darrick J. Wong 2022-12-30 22:19 ` [PATCH 23/24] xfs: improve metadata array field handling when fuzzing Darrick J. Wong 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> After we've run xfs_scrub -n to perform a check of a mounted filesystem's metadata, we should check the health reporting system to make sure that the results got recorded. Also wire this up to the xfs fuzz testing helpers. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 27 +++++++++++++++++++++++++++ common/xfs | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/common/fuzzy b/common/fuzzy index cf085f8b28..d841d435eb 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -216,6 +216,15 @@ __scratch_xfs_fuzz_field_online() { test $res -eq 0 && \ (>&2 echo "${fuzz_action}: online scrub didn't fail.") + # Does the health status report reflect the corruption? + if [ $res -ne 0 ]; then + __fuzz_notify "++ Detect fuzzed field ill-health report" + _check_xfs_health $SCRATCH_MNT 2>&1 + res=$? + test $res -ne 1 && \ + (>&2 echo "${fuzz_action}: online health check failed ($res).") + fi + # Try fixing the filesystem online __fuzz_notify "++ Try to repair filesystem (online)" _scratch_scrub 2>&1 @@ -308,6 +317,15 @@ __scratch_xfs_fuzz_field_norepair() { test $res -eq 0 && \ (>&2 echo "${fuzz_action}: online scrub didn't fail.") + # Does the health status report reflect the corruption? + if [ $res -ne 0 ]; then + __fuzz_notify "++ Detect fuzzed field ill-health report" + _check_xfs_health $SCRATCH_MNT 2>&1 + res=$? + test $res -ne 1 && \ + (>&2 echo "${fuzz_action}: online health check failed ($res).") + fi + __scratch_xfs_fuzz_unmount return 0 @@ -338,6 +356,15 @@ __scratch_xfs_fuzz_field_both() { test $res -eq 0 && \ (>&2 echo "${fuzz_action}: online scrub didn't fail.") + # Does the health status report reflect the corruption? + if [ $res -ne 0 ]; then + __fuzz_notify "++ Detect fuzzed field ill-health report" + _check_xfs_health $SCRATCH_MNT 2>&1 + res=$? + test $res -ne 1 && \ + (>&2 echo "${fuzz_action}: online health check failed ($res).") + fi + # Try fixing the filesystem online __fuzz_notify "++ Try to repair filesystem (online)" _scratch_scrub 2>&1 diff --git a/common/xfs b/common/xfs index 804047557b..371618dc7b 100644 --- a/common/xfs +++ b/common/xfs @@ -599,6 +599,37 @@ _require_xfs_db_command() _notrun "xfs_db $command support is missing" } +# Check the health of a mounted XFS filesystem. Callers probably want to +# ensure that xfs_scrub has been run first. Returns 1 if unhealthy metadata +# are found or 0 otherwise. +_check_xfs_health() { + local mntpt="$1" + local ret=0 + local t="$tmp.health_helper" + + test -x "$XFS_SPACEMAN_PROG" || return 0 + + $XFS_SPACEMAN_PROG -c 'health -c -q' $mntpt > $t.out 2> $t.err + test $? -ne 0 && ret=1 + + # Don't return error if userspace or kernel don't support health + # reporting. + grep -q 'command.*health.*not found' $t.err && return 0 + grep -q 'Inappropriate ioctl for device' $t.err && return 0 + + # Filter out the "please run scrub" message if nothing's been checked. + sed -e '/Health status has not been/d' -e '/Please run xfs_scrub/d' -i \ + $t.err + + grep -q unhealthy $t.out && ret=1 + test $(wc -l < $t.err) -gt 0 && ret=1 + cat $t.out + cat $t.err 1>&2 + rm -f $t.out $t.err + + return $ret +} + # Does the filesystem mounted from a particular device support scrub? _supports_xfs_scrub() { @@ -750,6 +781,18 @@ _check_xfs_filesystem() ok=0 fi rm -f $tmp.scrub + + # Does the health reporting notice anything? + _check_xfs_health $mntpt > $tmp.health 2>&1 + res=$? + if [ $((res ^ ok)) -eq 0 ]; then + _log_err "_check_xfs_filesystem: filesystem on $device failed health check" + echo "*** xfs_spaceman -c 'health -c -q' output ***" >> $seqres.full + cat $tmp.health >> $seqres.full + echo "*** end xfs_spaceman output" >> $seqres.full + ok=0 + fi + rm -f $tmp.health fi if [ "$type" = "xfs" ]; then ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 24/24] fuzzy: for fuzzing ag btrees, find the path to the AG header 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (21 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 18/24] common: check xfs health after doing an online scrub Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 23/24] xfs: improve metadata array field handling when fuzzing Darrick J. Wong 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The fs population code creates various btrees in /some/ allocation group with at least two levels. These btrees aren't necessarily created in agno 0, so we need to find it programmatically. While we're at it, fix a few of the comments that failed to mention when we're fuzzing interior nodes and not leaves. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/358 | 5 ++++- tests/xfs/359 | 5 ++++- tests/xfs/360 | 5 ++++- tests/xfs/361 | 5 ++++- tests/xfs/362 | 5 ++++- tests/xfs/363 | 5 ++++- tests/xfs/364 | 5 ++++- tests/xfs/365 | 5 ++++- tests/xfs/366 | 5 ++++- tests/xfs/367 | 5 ++++- tests/xfs/368 | 5 ++++- tests/xfs/369 | 5 ++++- tests/xfs/370 | 5 ++++- tests/xfs/371 | 5 ++++- tests/xfs/372 | 5 ++++- tests/xfs/373 | 7 +++++-- tests/xfs/410 | 5 ++++- tests/xfs/411 | 5 ++++- tests/xfs/457 | 5 ++++- tests/xfs/458 | 5 ++++- tests/xfs/459 | 5 ++++- tests/xfs/460 | 5 ++++- tests/xfs/461 | 5 ++++- tests/xfs/462 | 5 ++++- tests/xfs/463 | 5 ++++- tests/xfs/464 | 7 +++++-- tests/xfs/483 | 5 ++++- 28 files changed, 147 insertions(+), 29 deletions(-) diff --git a/common/xfs b/common/xfs index 371618dc7b..610730e5ef 100644 --- a/common/xfs +++ b/common/xfs @@ -1686,3 +1686,40 @@ _xfs_get_inode_core_bytes() echo 96 fi } + +# Find us the path to the AG header containing a per-AG btree with a specific +# height. +_scratch_xfs_find_agbtree_height() { + local bt_type="$1" + local bt_height="$2" + local agcount=$(_xfs_mount_agcount $SCRATCH_DEV) + + case "${bt_type}" in + "bno"|"cnt"|"rmap"|"refcnt") + hdr="agf" + bt_prefix="${bt_type}" + ;; + "ino") + hdr="agi" + bt_prefix="" + ;; + "fino") + hdr="agi" + bt_prefix="free_" + ;; + *) + _fail "Don't know about AG btree ${bt_type}" + ;; + esac + + for ((agno = 0; agno < agcount; agno++)); do + bt_level=$(_scratch_xfs_db -c "${hdr} ${agno}" -c "p ${bt_prefix}level" | awk '{print $3}') + # "level" is really the btree height + if [ "${bt_level}" -eq "${bt_height}" ]; then + echo "${hdr} ${agno}" + return 0 + fi + done + + return 1 +} diff --git a/tests/xfs/358 b/tests/xfs/358 index a00eb6f9cb..92180e5196 100755 --- a/tests/xfs/358 +++ b/tests/xfs/358 @@ -24,8 +24,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'bno' 2)" || \ + _fail "could not find two-level bnobt" + echo "Fuzz bnobt recs" -_scratch_xfs_fuzz_metadata '' 'offline' 'agf 0' 'addr bnoroot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'addr bnoroot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing bnobt recs" # success, all done diff --git a/tests/xfs/359 b/tests/xfs/359 index f0a82db4b8..0498aaccf5 100755 --- a/tests/xfs/359 +++ b/tests/xfs/359 @@ -24,8 +24,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'bno' 2)" || \ + _fail "could not find two-level bnobt" + echo "Fuzz bnobt recs" -_scratch_xfs_fuzz_metadata '' 'online' 'agf 0' 'addr bnoroot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'addr bnoroot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing bnobt recs" # success, all done diff --git a/tests/xfs/360 b/tests/xfs/360 index 3942ffd1b2..c34f455403 100755 --- a/tests/xfs/360 +++ b/tests/xfs/360 @@ -24,8 +24,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'bno' 2)" || \ + _fail "could not find two-level bnobt" + echo "Fuzz bnobt keyptr" -_scratch_xfs_fuzz_metadata '' 'offline' 'agf 0' 'addr bnoroot' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'addr bnoroot' >> $seqres.full echo "Done fuzzing bnobt keyptr" # success, all done diff --git a/tests/xfs/361 b/tests/xfs/361 index b7ee0f6f94..22b1af4ea3 100755 --- a/tests/xfs/361 +++ b/tests/xfs/361 @@ -24,8 +24,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'bno' 2)" || \ + _fail "could not find two-level bnobt" + echo "Fuzz bnobt keyptr" -_scratch_xfs_fuzz_metadata '' 'online' 'agf 0' 'addr bnoroot' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'addr bnoroot' >> $seqres.full echo "Done fuzzing bnobt keyptr" # success, all done diff --git a/tests/xfs/362 b/tests/xfs/362 index f711661b02..51727edc06 100755 --- a/tests/xfs/362 +++ b/tests/xfs/362 @@ -24,8 +24,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'cnt' 2)" || \ + _fail "could not find two-level cntbt" + echo "Fuzz cntbt" -_scratch_xfs_fuzz_metadata '' 'offline' 'agf 0' 'addr cntroot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'addr cntroot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing cntbt" # success, all done diff --git a/tests/xfs/363 b/tests/xfs/363 index 6be9109eca..8a62c1c821 100755 --- a/tests/xfs/363 +++ b/tests/xfs/363 @@ -24,8 +24,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'cnt' 2)" || \ + _fail "could not find two-level cntbt" + echo "Fuzz cntbt" -_scratch_xfs_fuzz_metadata '' 'online' 'agf 0' 'addr cntroot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'addr cntroot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing cntbt" # success, all done diff --git a/tests/xfs/364 b/tests/xfs/364 index fcd18fe686..984ecdafed 100755 --- a/tests/xfs/364 +++ b/tests/xfs/364 @@ -24,8 +24,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'ino' 2)" || \ + _fail "could not find two-level inobt" + echo "Fuzz inobt" -_scratch_xfs_fuzz_metadata '' 'offline' 'agi 0' 'addr root' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'addr root' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing inobt" # success, all done diff --git a/tests/xfs/365 b/tests/xfs/365 index 6f116f9b9c..e4325c35d1 100755 --- a/tests/xfs/365 +++ b/tests/xfs/365 @@ -24,8 +24,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'ino' 2)" || \ + _fail "could not find two-level inobt" + echo "Fuzz inobt" -_scratch_xfs_fuzz_metadata '' 'online' 'agi 1' 'addr root' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'addr root' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing inobt" # success, all done diff --git a/tests/xfs/366 b/tests/xfs/366 index 4c651288c0..8a52d21a0f 100755 --- a/tests/xfs/366 +++ b/tests/xfs/366 @@ -25,8 +25,11 @@ _require_xfs_finobt echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'fino' 2)" || \ + _fail "could not find two-level finobt" + echo "Fuzz finobt" -_scratch_xfs_fuzz_metadata '' 'offline' 'agi 0' 'addr free_root' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'addr free_root' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing finobt" # success, all done diff --git a/tests/xfs/367 b/tests/xfs/367 index c474a9e7d8..d9d07faab2 100755 --- a/tests/xfs/367 +++ b/tests/xfs/367 @@ -25,8 +25,11 @@ _require_xfs_finobt echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'fino' 2)" || \ + _fail "could not find two-level finobt" + echo "Fuzz finobt" -_scratch_xfs_fuzz_metadata '' 'online' 'agi 0' 'addr free_root' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'addr free_root' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing finobt" # success, all done diff --git a/tests/xfs/368 b/tests/xfs/368 index b1c1f97664..83499827c9 100755 --- a/tests/xfs/368 +++ b/tests/xfs/368 @@ -25,8 +25,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rmapbt" + echo "Fuzz rmapbt recs" -_scratch_xfs_fuzz_metadata '' 'offline' 'agf 0' 'addr rmaproot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'addr rmaproot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing rmapbt recs" # success, all done diff --git a/tests/xfs/369 b/tests/xfs/369 index 5e6d8d9be0..3236b50e00 100755 --- a/tests/xfs/369 +++ b/tests/xfs/369 @@ -25,8 +25,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rmapbt" + echo "Fuzz rmapbt recs" -_scratch_xfs_fuzz_metadata '' 'online' 'agf 0' 'addr rmaproot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'addr rmaproot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing rmapbt recs" # success, all done diff --git a/tests/xfs/370 b/tests/xfs/370 index 0a916242e2..891d5e2572 100755 --- a/tests/xfs/370 +++ b/tests/xfs/370 @@ -26,8 +26,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rmapbt" + echo "Fuzz rmapbt keyptr" -_scratch_xfs_fuzz_metadata '' 'offline' 'agf 0' 'addr rmaproot' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'addr rmaproot' >> $seqres.full echo "Done fuzzing rmapbt keyptr" # success, all done diff --git a/tests/xfs/371 b/tests/xfs/371 index a9b914d9f5..f7a336b170 100755 --- a/tests/xfs/371 +++ b/tests/xfs/371 @@ -26,8 +26,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rmapbt" + echo "Fuzz rmapbt keyptr" -_scratch_xfs_fuzz_metadata '' 'online' 'agf 0' 'addr rmaproot' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'addr rmaproot' >> $seqres.full echo "Done fuzzing rmapbt keyptr" # success, all done diff --git a/tests/xfs/372 b/tests/xfs/372 index c39a917500..2250322527 100755 --- a/tests/xfs/372 +++ b/tests/xfs/372 @@ -26,8 +26,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level refcountbt" + echo "Fuzz refcountbt" -_scratch_xfs_fuzz_metadata '' 'offline' 'agf 0' 'addr refcntroot' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'addr refcntroot' >> $seqres.full echo "Done fuzzing refcountbt" # success, all done diff --git a/tests/xfs/373 b/tests/xfs/373 index 324aa9fe7d..e0c20044ec 100755 --- a/tests/xfs/373 +++ b/tests/xfs/373 @@ -4,7 +4,7 @@ # # FS QA Test No. 373 # -# Populate a XFS filesystem and fuzz every refcountbt field. +# Populate a XFS filesystem and fuzz every refcountbt key/pointer field. # Use xfs_scrub to fix the corruption. # . ./common/preamble @@ -26,8 +26,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level refcountbt" + echo "Fuzz refcountbt" -_scratch_xfs_fuzz_metadata '' 'online' 'agf 0' 'addr refcntroot' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'addr refcntroot' >> $seqres.full echo "Done fuzzing refcountbt" # success, all done diff --git a/tests/xfs/410 b/tests/xfs/410 index e98a63ebf5..388ed7d190 100755 --- a/tests/xfs/410 +++ b/tests/xfs/410 @@ -26,8 +26,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level refcountbt" + echo "Fuzz refcountbt" -_scratch_xfs_fuzz_metadata '' 'offline' 'agf 0' 'addr refcntroot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'addr refcntroot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing refcountbt" # success, all done diff --git a/tests/xfs/411 b/tests/xfs/411 index cfe7796102..a9fc25ce7d 100755 --- a/tests/xfs/411 +++ b/tests/xfs/411 @@ -26,8 +26,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level refcountbt" + echo "Fuzz refcountbt" -_scratch_xfs_fuzz_metadata '' 'online' 'agf 0' 'addr refcntroot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'addr refcntroot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing refcountbt" # success, all done diff --git a/tests/xfs/457 b/tests/xfs/457 index 332eeb9837..64cd6b4b82 100755 --- a/tests/xfs/457 +++ b/tests/xfs/457 @@ -25,8 +25,11 @@ _disable_dmesg_check echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'bno' 2)" || \ + _fail "could not find two-level bnobt" + echo "Fuzz bnobt recs" -_scratch_xfs_fuzz_metadata '' 'none' 'agf 0' 'addr bnoroot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'addr bnoroot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing bnobt recs" # success, all done diff --git a/tests/xfs/458 b/tests/xfs/458 index ce03d687ab..8d87ec569f 100755 --- a/tests/xfs/458 +++ b/tests/xfs/458 @@ -25,8 +25,11 @@ _disable_dmesg_check echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'bno' 2)" || \ + _fail "could not find two-level bnobt" + echo "Fuzz bnobt keyptr" -_scratch_xfs_fuzz_metadata '' 'none' 'agf 0' 'addr bnoroot' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'addr bnoroot' >> $seqres.full echo "Done fuzzing bnobt keyptr" # success, all done diff --git a/tests/xfs/459 b/tests/xfs/459 index d166160f87..5989bc1e6e 100755 --- a/tests/xfs/459 +++ b/tests/xfs/459 @@ -25,8 +25,11 @@ _disable_dmesg_check echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'cnt' 2)" || \ + _fail "could not find two-level cntbt" + echo "Fuzz cntbt" -_scratch_xfs_fuzz_metadata '' 'none' 'agf 0' 'addr cntroot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'addr cntroot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing cntbt" # success, all done diff --git a/tests/xfs/460 b/tests/xfs/460 index 0daafa3066..7117477011 100755 --- a/tests/xfs/460 +++ b/tests/xfs/460 @@ -25,8 +25,11 @@ _disable_dmesg_check echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'ino' 2)" || \ + _fail "could not find two-level inobt" + echo "Fuzz inobt" -_scratch_xfs_fuzz_metadata '' 'none' 'agi 1' 'addr root' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'addr root' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing inobt" # success, all done diff --git a/tests/xfs/461 b/tests/xfs/461 index 2d20c69d87..7c1327b052 100755 --- a/tests/xfs/461 +++ b/tests/xfs/461 @@ -26,8 +26,11 @@ _require_xfs_finobt echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'fino' 2)" || \ + _fail "could not find two-level finobt" + echo "Fuzz finobt" -_scratch_xfs_fuzz_metadata '' 'none' 'agi 0' 'addr free_root' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'addr free_root' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing finobt" # success, all done diff --git a/tests/xfs/462 b/tests/xfs/462 index 587facc03c..1ee4d27e92 100755 --- a/tests/xfs/462 +++ b/tests/xfs/462 @@ -26,8 +26,11 @@ _disable_dmesg_check echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rmapbt" + echo "Fuzz rmapbt recs" -_scratch_xfs_fuzz_metadata '' 'none' 'agf 0' 'addr rmaproot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'addr rmaproot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing rmapbt recs" # success, all done diff --git a/tests/xfs/463 b/tests/xfs/463 index 7270f7017a..7dd2d37dea 100755 --- a/tests/xfs/463 +++ b/tests/xfs/463 @@ -26,8 +26,11 @@ _disable_dmesg_check echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rmapbt" + echo "Fuzz rmapbt keyptr" -_scratch_xfs_fuzz_metadata '' 'none' 'agf 0' 'addr rmaproot' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'addr rmaproot' >> $seqres.full echo "Done fuzzing rmapbt keyptr" # success, all done diff --git a/tests/xfs/464 b/tests/xfs/464 index 59d25ae1c0..719901e66d 100755 --- a/tests/xfs/464 +++ b/tests/xfs/464 @@ -4,7 +4,7 @@ # # FS QA Test No. 464 # -# Populate a XFS filesystem and fuzz every refcountbt field. +# Populate a XFS filesystem and fuzz every refcountbt key/pointer field. # Do not fix the filesystem, to test metadata verifiers. . ./common/preamble @@ -27,8 +27,11 @@ _disable_dmesg_check echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level refcountbt" + echo "Fuzz refcountbt" -_scratch_xfs_fuzz_metadata '' 'none' 'agf 0' 'addr refcntroot' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'addr refcntroot' >> $seqres.full echo "Done fuzzing refcountbt" # success, all done diff --git a/tests/xfs/483 b/tests/xfs/483 index d7b0101a82..56670ba178 100755 --- a/tests/xfs/483 +++ b/tests/xfs/483 @@ -27,8 +27,11 @@ _disable_dmesg_check echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_agbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level refcountbt" + echo "Fuzz refcountbt" -_scratch_xfs_fuzz_metadata '' 'none' 'agf 0' 'addr refcntroot' 'addr ptrs[1]' >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'addr refcntroot' 'addr ptrs[1]' >> $seqres.full echo "Done fuzzing refcountbt" # success, all done ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 23/24] xfs: improve metadata array field handling when fuzzing 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong ` (22 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 24/24] fuzzy: for fuzzing ag btrees, find the path to the AG header Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 23 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Currently, we use some gnarly regular expressions to try to constrain the amount of time we spend fuzzing each element of a metadata array. This is pretty inflexible (and buggy) since we limit some arrays (e.g. dir hashes) to the first ten elements and other arrays (e.g. extent mappings) that use compact index ranges to the first one. Replace this whole weird mess with logic that can tease out the array indices, unroll the compact indices if needed, and give the user more flexible control over which array elements get used. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 52 +++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index f34fcadefe..53fe22db69 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -53,12 +53,46 @@ _scratch_scrub() { esac } -# Filter out any keys with an array index >= 10, collapse any array range -# ("[1-195]") to the first item, and ignore padding fields. -__filter_xfs_db_keys() { - sed -e '/\([a-z]*\)\[\([0-9][0-9]\+\)\].*/d' \ - -e 's/\([a-zA-Z0-9_]*\)\[\([0-9]*\)-[0-9]*\]/\1[\2]/g' \ - -e '/pad/d' +# Expand indexed keys (i.e. arrays) into a long format so that we can filter +# the array indices individually, and pass regular keys right through. +# +# For example, "u3.bmx[0-1] = [foo,bar]" is exploded into: +# u3.bmx[0] = [foo,bar] +# u3.bmx[1] = [foo,bar] +# +# Note that we restrict array indices to [0-9] to reduce fuzz runtime. The +# minimum and maximum array indices can be changed by setting the variables +# SCRATCH_XFS_{MIN,MAX}_ARRAY_IDX. +# +# Also filter padding fields. +__explode_xfs_db_fields() { + local min_idx="${SCRATCH_XFS_MIN_ARRAY_IDX}" + local max_idx="${SCRATCH_XFS_MAX_ARRAY_IDX}" + + test -z "${min_idx}" && min_idx=0 + test -z "${max_idx}" && max_idx=9 + test "${max_idx}" = "none" && max_idx=99999 + + grep ' = ' | \ + sed -e 's/^\([.a-zA-Z0-9_]*\)\[\([0-9]*\)-\([0-9]*\)\]\(.*\) = \(.*\)$/\1[%d]\4 \2 \3 = \5/g' \ + -e 's/^\([.a-zA-Z0-9_]*\)\[\([0-9]*\)\]\(.*\) = \(.*\)$/\1[%d]\3 \2 \2 = \4/g' | \ + while read name col1 col2 rest; do + if [[ "${name}" == *pad* ]]; then + continue + fi + + if [ "${col1}" = "=" ]; then + echo "${name} ${col1} ${col2} ${rest}" + continue + fi + + test "${min_idx}" -gt "${col1}" && col1="${min_idx}" + test "${max_idx}" -lt "${col2}" && col2="${max_idx}" + + seq "${col1}" "${col2}" | while read idx; do + printf "${name} %s\n" "${idx}" "${rest}" + done + done } # Filter out metadata fields that are completely controlled by userspace @@ -96,14 +130,14 @@ __filter_xfs_db_print_fields() { if [ -z "${filter}" ] || [ "${filter}" = "nofilter" ]; then filter='^' fi - grep ' = ' | while read key equals value; do - fuzzkey="$(echo "${key}" | __filter_xfs_db_keys)" + __explode_xfs_db_fields | while read key equals value; do + fuzzkey="$(echo "${key}")" if [ -z "${fuzzkey}" ]; then continue elif [[ "${value}" == "["* ]]; then echo "${value}" | sed -e 's/^.//g' -e 's/.$//g' -e 's/,/\n/g' | while read subfield; do echo "${fuzzkey}.${subfield}" - done | __filter_xfs_db_keys + done else echo "${fuzzkey}" fi ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/5] fstests: strengthen fuzz testing 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (11 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/5] fuzzy: test fuzzing directory block mappings Darrick J. Wong ` (4 more replies) 2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong ` (4 subsequent siblings) 17 siblings, 5 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, This patchset fills some gaps in our fuzz testing for XFS. I forgot to include fuzz testing the data fork mappings of directories and xattrs to see how scrub responds to nonsensical file maps, and I omitted tests to fuzz the realtime metadata. Add those. Finally, add a new fuzz testing strategy known as bothrepair. This simulates what system administrators are most likely to do upon receiving a health alert about a pet filesystem -- try to repair it with xfs_scrub, and if that doesn't work, unmount the filesystem and run xfs_repair. Between the two, we ought to be able to fix every possible problem. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=more-fuzz-testing --- common/fuzzy | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++- tests/xfs/1500 | 35 +++++++++++++ tests/xfs/1500.out | 4 + tests/xfs/1501 | 35 +++++++++++++ tests/xfs/1501.out | 4 + tests/xfs/1502 | 45 +++++++++++++++++ tests/xfs/1502.out | 6 ++ tests/xfs/1503 | 35 +++++++++++++ tests/xfs/1503.out | 4 + tests/xfs/1504 | 38 ++++++++++++++ tests/xfs/1504.out | 4 + tests/xfs/1505 | 38 ++++++++++++++ tests/xfs/1505.out | 4 + tests/xfs/1506 | 38 ++++++++++++++ tests/xfs/1506.out | 4 + tests/xfs/1507 | 38 ++++++++++++++ tests/xfs/1507.out | 4 + tests/xfs/1508 | 39 +++++++++++++++ tests/xfs/1508.out | 4 + tests/xfs/1509 | 39 +++++++++++++++ tests/xfs/1509.out | 4 + tests/xfs/1510 | 39 +++++++++++++++ tests/xfs/1510.out | 4 + tests/xfs/1511 | 40 +++++++++++++++ tests/xfs/1511.out | 4 + tests/xfs/1512 | 40 +++++++++++++++ tests/xfs/1512.out | 5 ++ tests/xfs/1513 | 40 +++++++++++++++ tests/xfs/1513.out | 5 ++ tests/xfs/1514 | 40 +++++++++++++++ tests/xfs/1514.out | 5 ++ tests/xfs/1515 | 42 ++++++++++++++++ tests/xfs/1515.out | 5 ++ tests/xfs/1516 | 40 +++++++++++++++ tests/xfs/1516.out | 5 ++ tests/xfs/1517 | 40 +++++++++++++++ tests/xfs/1517.out | 5 ++ tests/xfs/1518 | 40 +++++++++++++++ tests/xfs/1518.out | 5 ++ tests/xfs/1519 | 41 +++++++++++++++ tests/xfs/1519.out | 5 ++ tests/xfs/1520 | 42 ++++++++++++++++ tests/xfs/1520.out | 5 ++ tests/xfs/1521 | 42 ++++++++++++++++ tests/xfs/1521.out | 5 ++ tests/xfs/1522 | 42 ++++++++++++++++ tests/xfs/1522.out | 5 ++ tests/xfs/1523 | 42 ++++++++++++++++ tests/xfs/1523.out | 5 ++ tests/xfs/1524 | 40 +++++++++++++++ tests/xfs/1524.out | 5 ++ tests/xfs/1525 | 40 +++++++++++++++ tests/xfs/1525.out | 5 ++ tests/xfs/1526 | 40 +++++++++++++++ tests/xfs/1526.out | 5 ++ tests/xfs/1527 | 40 +++++++++++++++ tests/xfs/1527.out | 5 ++ tests/xfs/1530 | 40 +++++++++++++++ tests/xfs/1530.out | 4 + tests/xfs/1531 | 40 +++++++++++++++ tests/xfs/1531.out | 5 ++ tests/xfs/1532 | 40 +++++++++++++++ tests/xfs/1532.out | 5 ++ tests/xfs/1533 | 40 +++++++++++++++ tests/xfs/1533.out | 5 ++ tests/xfs/1534 | 38 ++++++++++++++ tests/xfs/1534.out | 4 + tests/xfs/1535 | 38 ++++++++++++++ tests/xfs/1535.out | 4 + tests/xfs/1536 | 38 ++++++++++++++ tests/xfs/1536.out | 4 + tests/xfs/1537 | 41 +++++++++++++++ tests/xfs/1537.out | 5 ++ tests/xfs/1554 | 48 ++++++++++++++++++ tests/xfs/1554.out | 10 ++++ tests/xfs/1555 | 48 ++++++++++++++++++ tests/xfs/1555.out | 10 ++++ tests/xfs/1556 | 48 ++++++++++++++++++ tests/xfs/1556.out | 10 ++++ tests/xfs/1557 | 48 ++++++++++++++++++ tests/xfs/1557.out | 10 ++++ tests/xfs/1558 | 48 ++++++++++++++++++ tests/xfs/1558.out | 10 ++++ tests/xfs/1559 | 48 ++++++++++++++++++ tests/xfs/1559.out | 10 ++++ tests/xfs/1560 | 49 ++++++++++++++++++ tests/xfs/1560.out | 10 ++++ tests/xfs/1561 | 49 ++++++++++++++++++ tests/xfs/1561.out | 10 ++++ tests/xfs/1562 | 41 +++++++++++++++ tests/xfs/1562.out | 4 + tests/xfs/1563 | 41 +++++++++++++++ tests/xfs/1563.out | 4 + tests/xfs/1564 | 41 +++++++++++++++ tests/xfs/1564.out | 4 + tests/xfs/1565 | 41 +++++++++++++++ tests/xfs/1565.out | 4 + tests/xfs/1566 | 42 ++++++++++++++++ tests/xfs/1566.out | 4 + tests/xfs/1567 | 42 ++++++++++++++++ tests/xfs/1567.out | 4 + tests/xfs/1568 | 41 +++++++++++++++ tests/xfs/1568.out | 4 + tests/xfs/1569 | 41 +++++++++++++++ tests/xfs/1569.out | 4 + tests/xfs/1570 | 36 +++++++++++++ tests/xfs/1570.out | 4 + tests/xfs/1571 | 36 +++++++++++++ tests/xfs/1571.out | 4 + tests/xfs/1572 | 38 ++++++++++++++ tests/xfs/1572.out | 4 + tests/xfs/1573 | 37 ++++++++++++++ tests/xfs/1573.out | 4 + 113 files changed, 2717 insertions(+), 4 deletions(-) create mode 100755 tests/xfs/1500 create mode 100644 tests/xfs/1500.out create mode 100755 tests/xfs/1501 create mode 100644 tests/xfs/1501.out create mode 100755 tests/xfs/1502 create mode 100644 tests/xfs/1502.out create mode 100755 tests/xfs/1503 create mode 100644 tests/xfs/1503.out create mode 100755 tests/xfs/1504 create mode 100644 tests/xfs/1504.out create mode 100755 tests/xfs/1505 create mode 100644 tests/xfs/1505.out create mode 100755 tests/xfs/1506 create mode 100644 tests/xfs/1506.out create mode 100755 tests/xfs/1507 create mode 100644 tests/xfs/1507.out create mode 100755 tests/xfs/1508 create mode 100644 tests/xfs/1508.out create mode 100755 tests/xfs/1509 create mode 100644 tests/xfs/1509.out create mode 100755 tests/xfs/1510 create mode 100644 tests/xfs/1510.out create mode 100755 tests/xfs/1511 create mode 100644 tests/xfs/1511.out create mode 100755 tests/xfs/1512 create mode 100644 tests/xfs/1512.out create mode 100755 tests/xfs/1513 create mode 100644 tests/xfs/1513.out create mode 100755 tests/xfs/1514 create mode 100644 tests/xfs/1514.out create mode 100755 tests/xfs/1515 create mode 100644 tests/xfs/1515.out create mode 100755 tests/xfs/1516 create mode 100644 tests/xfs/1516.out create mode 100755 tests/xfs/1517 create mode 100644 tests/xfs/1517.out create mode 100755 tests/xfs/1518 create mode 100644 tests/xfs/1518.out create mode 100755 tests/xfs/1519 create mode 100644 tests/xfs/1519.out create mode 100755 tests/xfs/1520 create mode 100644 tests/xfs/1520.out create mode 100755 tests/xfs/1521 create mode 100644 tests/xfs/1521.out create mode 100755 tests/xfs/1522 create mode 100644 tests/xfs/1522.out create mode 100755 tests/xfs/1523 create mode 100644 tests/xfs/1523.out create mode 100755 tests/xfs/1524 create mode 100644 tests/xfs/1524.out create mode 100755 tests/xfs/1525 create mode 100644 tests/xfs/1525.out create mode 100755 tests/xfs/1526 create mode 100644 tests/xfs/1526.out create mode 100755 tests/xfs/1527 create mode 100644 tests/xfs/1527.out create mode 100755 tests/xfs/1530 create mode 100644 tests/xfs/1530.out create mode 100755 tests/xfs/1531 create mode 100644 tests/xfs/1531.out create mode 100755 tests/xfs/1532 create mode 100644 tests/xfs/1532.out create mode 100755 tests/xfs/1533 create mode 100644 tests/xfs/1533.out create mode 100755 tests/xfs/1534 create mode 100644 tests/xfs/1534.out create mode 100755 tests/xfs/1535 create mode 100644 tests/xfs/1535.out create mode 100755 tests/xfs/1536 create mode 100644 tests/xfs/1536.out create mode 100755 tests/xfs/1537 create mode 100644 tests/xfs/1537.out create mode 100755 tests/xfs/1554 create mode 100644 tests/xfs/1554.out create mode 100755 tests/xfs/1555 create mode 100644 tests/xfs/1555.out create mode 100755 tests/xfs/1556 create mode 100644 tests/xfs/1556.out create mode 100755 tests/xfs/1557 create mode 100644 tests/xfs/1557.out create mode 100755 tests/xfs/1558 create mode 100644 tests/xfs/1558.out create mode 100755 tests/xfs/1559 create mode 100644 tests/xfs/1559.out create mode 100755 tests/xfs/1560 create mode 100644 tests/xfs/1560.out create mode 100755 tests/xfs/1561 create mode 100644 tests/xfs/1561.out create mode 100755 tests/xfs/1562 create mode 100644 tests/xfs/1562.out create mode 100755 tests/xfs/1563 create mode 100644 tests/xfs/1563.out create mode 100755 tests/xfs/1564 create mode 100644 tests/xfs/1564.out create mode 100755 tests/xfs/1565 create mode 100644 tests/xfs/1565.out create mode 100755 tests/xfs/1566 create mode 100644 tests/xfs/1566.out create mode 100755 tests/xfs/1567 create mode 100644 tests/xfs/1567.out create mode 100755 tests/xfs/1568 create mode 100644 tests/xfs/1568.out create mode 100755 tests/xfs/1569 create mode 100644 tests/xfs/1569.out create mode 100755 tests/xfs/1570 create mode 100644 tests/xfs/1570.out create mode 100755 tests/xfs/1571 create mode 100644 tests/xfs/1571.out create mode 100755 tests/xfs/1572 create mode 100755 tests/xfs/1572.out create mode 100755 tests/xfs/1573 create mode 100644 tests/xfs/1573.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/5] fuzzy: test fuzzing directory block mappings 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: strengthen fuzz testing Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/5] fuzzy: test fuzzing xattr " Darrick J. Wong ` (3 subsequent siblings) 4 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Fuzz the block mappings of directories to see what happens. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 16 ++++++++++++++++ tests/xfs/1554 | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1554.out | 10 ++++++++++ tests/xfs/1555 | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1555.out | 10 ++++++++++ tests/xfs/1556 | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1556.out | 10 ++++++++++ 7 files changed, 190 insertions(+) create mode 100755 tests/xfs/1554 create mode 100644 tests/xfs/1554.out create mode 100755 tests/xfs/1555 create mode 100644 tests/xfs/1555.out create mode 100755 tests/xfs/1556 create mode 100644 tests/xfs/1556.out diff --git a/common/fuzzy b/common/fuzzy index 53fe22db69..09f42d9225 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -549,6 +549,22 @@ _require_scratch_xfs_fuzz_fields() _require_xfs_db_command "fuzz" } +# Sets the array SCRATCH_XFS_DIR_FUZZ_TYPES to the list of directory formats +# available for fuzzing. Each list item must match one of the /S_IFDIR.FMT_* +# files created by the fs population code. Users can override this by setting +# SCRATCH_XFS_LIST_FUZZ_DIRTYPE in the environment. BTREE is omitted here +# because that refers to the fork format and does not affect the directory +# structure at all. +_scratch_xfs_set_dir_fuzz_types() { + if [ -n "${SCRATCH_XFS_LIST_FUZZ_DIRTYPE}" ]; then + mapfile -t SCRATCH_XFS_DIR_FUZZ_TYPES < \ + <(echo "${SCRATCH_XFS_LIST_FUZZ_DIRTYPE}" | tr '[ ,]' '[\n\n]') + return + fi + + SCRATCH_XFS_DIR_FUZZ_TYPES=(BLOCK LEAF LEAFN NODE) +} + # Grab the list of available fuzzing verbs _scratch_xfs_list_fuzz_verbs() { if [ -n "${SCRATCH_XFS_LIST_FUZZ_VERBS}" ]; then diff --git a/tests/xfs/1554 b/tests/xfs/1554 new file mode 100755 index 0000000000..b43c705cb9 --- /dev/null +++ b/tests/xfs/1554 @@ -0,0 +1,48 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1554 +# +# Populate a XFS filesystem and fuzz the data mappings of every directory type. +# Use xfs_scrub to fix the corruption. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +_scratch_xfs_set_dir_fuzz_types + +# Now fuzz the block maps of each directory type. +for dirtype in "${SCRATCH_XFS_DIR_FUZZ_TYPES[@]}"; do + echo "Fuzz block map for ${dirtype}" | tee -a $seqres.full + + # Restore a correct copy of the filesystem before we start a round of + # fuzzing. This avoids corruption errors from xfs_db when + # _scratch_xfs_fuzz_metadata probes the directory block fields. + __scratch_xfs_fuzz_mdrestore + + _scratch_mount + inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_${dirtype}) + _scratch_unmount + + _scratch_xfs_fuzz_metadata 'u*.bmx' 'online' "inode ${inum}" >> $seqres.full + echo "Done fuzzing dir map ${dirtype}" +done + +# success, all done +status=0 +exit diff --git a/tests/xfs/1554.out b/tests/xfs/1554.out new file mode 100644 index 0000000000..9985ed1990 --- /dev/null +++ b/tests/xfs/1554.out @@ -0,0 +1,10 @@ +QA output created by 1554 +Format and populate +Fuzz block map for BLOCK +Done fuzzing dir map BLOCK +Fuzz block map for LEAF +Done fuzzing dir map LEAF +Fuzz block map for LEAFN +Done fuzzing dir map LEAFN +Fuzz block map for NODE +Done fuzzing dir map NODE diff --git a/tests/xfs/1555 b/tests/xfs/1555 new file mode 100755 index 0000000000..8dee177a3d --- /dev/null +++ b/tests/xfs/1555 @@ -0,0 +1,48 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1555 +# +# Populate a XFS filesystem and fuzz the data mappings of every directory type. +# Use xfs_repair to fix the corruption. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +_scratch_xfs_set_dir_fuzz_types + +# Now fuzz the block maps of each directory type. +for dirtype in "${SCRATCH_XFS_DIR_FUZZ_TYPES[@]}"; do + echo "Fuzz block map for ${dirtype}" | tee -a $seqres.full + + # Restore a correct copy of the filesystem before we start a round of + # fuzzing. This avoids corruption errors from xfs_db when + # _scratch_xfs_fuzz_metadata probes the directory block fields. + __scratch_xfs_fuzz_mdrestore + + _scratch_mount + inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_${dirtype}) + _scratch_unmount + + _scratch_xfs_fuzz_metadata 'u*.bmx' 'offline' "inode ${inum}" >> $seqres.full + echo "Done fuzzing dir map ${dirtype}" +done + +# success, all done +status=0 +exit diff --git a/tests/xfs/1555.out b/tests/xfs/1555.out new file mode 100644 index 0000000000..38c5714d13 --- /dev/null +++ b/tests/xfs/1555.out @@ -0,0 +1,10 @@ +QA output created by 1555 +Format and populate +Fuzz block map for BLOCK +Done fuzzing dir map BLOCK +Fuzz block map for LEAF +Done fuzzing dir map LEAF +Fuzz block map for LEAFN +Done fuzzing dir map LEAFN +Fuzz block map for NODE +Done fuzzing dir map NODE diff --git a/tests/xfs/1556 b/tests/xfs/1556 new file mode 100755 index 0000000000..54df601c11 --- /dev/null +++ b/tests/xfs/1556 @@ -0,0 +1,48 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1556 +# +# Populate a XFS filesystem and fuzz the data mappings of every directory type. +# Do not fix the filesystem, to test metadata verifiers. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_norepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +_scratch_xfs_set_dir_fuzz_types + +# Now fuzz the block maps of each directory type. +for dirtype in "${SCRATCH_XFS_DIR_FUZZ_TYPES[@]}"; do + echo "Fuzz block map for ${dirtype}" | tee -a $seqres.full + + # Restore a correct copy of the filesystem before we start a round of + # fuzzing. This avoids corruption errors from xfs_db when + # _scratch_xfs_fuzz_metadata probes the directory block fields. + __scratch_xfs_fuzz_mdrestore + + _scratch_mount + inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_${dirtype}) + _scratch_unmount + + _scratch_xfs_fuzz_metadata 'u*.bmx' 'none' "inode ${inum}" >> $seqres.full + echo "Done fuzzing dir map ${dirtype}" +done + +# success, all done +status=0 +exit diff --git a/tests/xfs/1556.out b/tests/xfs/1556.out new file mode 100644 index 0000000000..b947285caa --- /dev/null +++ b/tests/xfs/1556.out @@ -0,0 +1,10 @@ +QA output created by 1556 +Format and populate +Fuzz block map for BLOCK +Done fuzzing dir map BLOCK +Fuzz block map for LEAF +Done fuzzing dir map LEAF +Fuzz block map for LEAFN +Done fuzzing dir map LEAFN +Fuzz block map for NODE +Done fuzzing dir map NODE ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 2/5] fuzzy: test fuzzing xattr block mappings 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: strengthen fuzz testing Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/5] fuzzy: test fuzzing directory block mappings Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 5/5] fuzzy: fuzz test key/pointers of inode btrees Darrick J. Wong ` (2 subsequent siblings) 4 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Fuzz the block mappings of extended attributes to see what happens. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 16 ++++++++++++++++ tests/xfs/1557 | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1557.out | 10 ++++++++++ tests/xfs/1558 | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1558.out | 10 ++++++++++ tests/xfs/1559 | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1559.out | 10 ++++++++++ 7 files changed, 190 insertions(+) create mode 100755 tests/xfs/1557 create mode 100644 tests/xfs/1557.out create mode 100755 tests/xfs/1558 create mode 100644 tests/xfs/1558.out create mode 100755 tests/xfs/1559 create mode 100644 tests/xfs/1559.out diff --git a/common/fuzzy b/common/fuzzy index 09f42d9225..d8eb7d8b72 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -565,6 +565,22 @@ _scratch_xfs_set_dir_fuzz_types() { SCRATCH_XFS_DIR_FUZZ_TYPES=(BLOCK LEAF LEAFN NODE) } +# Sets the array SCRATCH_XFS_XATTR_FUZZ_TYPES to the list of xattr formats +# available for fuzzing. Each list item must match one of the /ATTR.FMT_* +# files created by the fs population code. Users can override this by setting +# SCRATCH_XFS_LIST_FUZZ_XATTRTYPE in the environment. BTREE is omitted here +# because that refers to the fork format and does not affect the extended +# attribute structure at all. +_scratch_xfs_set_xattr_fuzz_types() { + if [ -n "${SCRATCH_XFS_LIST_FUZZ_XATTRTYPE}" ]; then + mapfile -t SCRATCH_XFS_XATTR_FUZZ_TYPES < \ + <(echo "${SCRATCH_XFS_LIST_FUZZ_XATTRTYPE}" | tr '[ ,]' '[\n\n]') + return + fi + + SCRATCH_XFS_XATTR_FUZZ_TYPES=(EXTENTS_REMOTE3K EXTENTS_REMOTE4K LEAF NODE) +} + # Grab the list of available fuzzing verbs _scratch_xfs_list_fuzz_verbs() { if [ -n "${SCRATCH_XFS_LIST_FUZZ_VERBS}" ]; then diff --git a/tests/xfs/1557 b/tests/xfs/1557 new file mode 100755 index 0000000000..afd5d31f62 --- /dev/null +++ b/tests/xfs/1557 @@ -0,0 +1,48 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1555 +# +# Populate a XFS filesystem and fuzz the attr mappings of every xattr type. +# Use xfs_scrub to fix the corruption. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +_scratch_xfs_set_xattr_fuzz_types + +# Now fuzz the block maps of each xattr type. +for attrtype in "${SCRATCH_XFS_XATTR_FUZZ_TYPES[@]}"; do + echo "Fuzz block map for ${attrtype}" | tee -a $seqres.full + + # Restore a correct copy of the filesystem before we start a round of + # fuzzing. This avoids corruption errors from xfs_db when + # _scratch_xfs_fuzz_metadata probes the xattr block fields. + __scratch_xfs_fuzz_mdrestore + + _scratch_mount + inum=$(stat -c '%i' $SCRATCH_MNT/ATTR.FMT_${attrtype}) + _scratch_unmount + + _scratch_xfs_fuzz_metadata 'a*.bmx' 'online' "inode ${inum}" >> $seqres.full + echo "Done fuzzing attr map ${attrtype}" +done + +# success, all done +status=0 +exit diff --git a/tests/xfs/1557.out b/tests/xfs/1557.out new file mode 100644 index 0000000000..e4d92dd6b8 --- /dev/null +++ b/tests/xfs/1557.out @@ -0,0 +1,10 @@ +QA output created by 1557 +Format and populate +Fuzz block map for EXTENTS_REMOTE3K +Done fuzzing attr map EXTENTS_REMOTE3K +Fuzz block map for EXTENTS_REMOTE4K +Done fuzzing attr map EXTENTS_REMOTE4K +Fuzz block map for LEAF +Done fuzzing attr map LEAF +Fuzz block map for NODE +Done fuzzing attr map NODE diff --git a/tests/xfs/1558 b/tests/xfs/1558 new file mode 100755 index 0000000000..0683b06010 --- /dev/null +++ b/tests/xfs/1558 @@ -0,0 +1,48 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1557 +# +# Populate a XFS filesystem and fuzz the attr mappings of every xattr type. +# Use xfs_repair to fix the corruption. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +_scratch_xfs_set_xattr_fuzz_types + +# Now fuzz the block maps of each xattr type. +for attrtype in "${SCRATCH_XFS_XATTR_FUZZ_TYPES[@]}"; do + echo "Fuzz block map for ${attrtype}" | tee -a $seqres.full + + # Restore a correct copy of the filesystem before we start a round of + # fuzzing. This avoids corruption errors from xfs_db when + # _scratch_xfs_fuzz_metadata probes the xattr block fields. + __scratch_xfs_fuzz_mdrestore + + _scratch_mount + inum=$(stat -c '%i' $SCRATCH_MNT/ATTR.FMT_${attrtype}) + _scratch_unmount + + _scratch_xfs_fuzz_metadata 'a*.bmx' 'offline' "inode ${inum}" >> $seqres.full + echo "Done fuzzing attr map ${attrtype}" +done + +# success, all done +status=0 +exit diff --git a/tests/xfs/1558.out b/tests/xfs/1558.out new file mode 100644 index 0000000000..6fed892cb4 --- /dev/null +++ b/tests/xfs/1558.out @@ -0,0 +1,10 @@ +QA output created by 1558 +Format and populate +Fuzz block map for EXTENTS_REMOTE3K +Done fuzzing attr map EXTENTS_REMOTE3K +Fuzz block map for EXTENTS_REMOTE4K +Done fuzzing attr map EXTENTS_REMOTE4K +Fuzz block map for LEAF +Done fuzzing attr map LEAF +Fuzz block map for NODE +Done fuzzing attr map NODE diff --git a/tests/xfs/1559 b/tests/xfs/1559 new file mode 100755 index 0000000000..886c1f2641 --- /dev/null +++ b/tests/xfs/1559 @@ -0,0 +1,48 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1559 +# +# Populate a XFS filesystem and fuzz the attr mappings of every xattr type. +# Do not fix the filesystem, to test metadata verifiers. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_norepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +_scratch_xfs_set_xattr_fuzz_types + +# Now fuzz the block maps of each xattr type. +for attrtype in "${SCRATCH_XFS_XATTR_FUZZ_TYPES[@]}"; do + echo "Fuzz block map for ${attrtype}" | tee -a $seqres.full + + # Restore a correct copy of the filesystem before we start a round of + # fuzzing. This avoids corruption errors from xfs_db when + # _scratch_xfs_fuzz_metadata probes the xattr block fields. + __scratch_xfs_fuzz_mdrestore + + _scratch_mount + inum=$(stat -c '%i' $SCRATCH_MNT/ATTR.FMT_${attrtype}) + _scratch_unmount + + _scratch_xfs_fuzz_metadata 'a*.bmx' 'none' "inode ${inum}" >> $seqres.full + echo "Done fuzzing attr map ${attrtype}" +done + +# success, all done +status=0 +exit diff --git a/tests/xfs/1559.out b/tests/xfs/1559.out new file mode 100644 index 0000000000..19fe4c91df --- /dev/null +++ b/tests/xfs/1559.out @@ -0,0 +1,10 @@ +QA output created by 1559 +Format and populate +Fuzz block map for EXTENTS_REMOTE3K +Done fuzzing attr map EXTENTS_REMOTE3K +Fuzz block map for EXTENTS_REMOTE4K +Done fuzzing attr map EXTENTS_REMOTE4K +Fuzz block map for LEAF +Done fuzzing attr map LEAF +Fuzz block map for NODE +Done fuzzing attr map NODE ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 5/5] fuzzy: fuzz test key/pointers of inode btrees 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: strengthen fuzz testing Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/5] fuzzy: test fuzzing directory block mappings Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/5] fuzzy: test fuzzing xattr " Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 4/5] xfs: fuzz test both repair strategies Darrick J. Wong 2022-12-30 22:19 ` [PATCH 3/5] fuzzy: test fuzzing realtime free space metadata Darrick J. Wong 4 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Test what happens when we fuzz the key/pointer blocks (aka the interior nodes) of the inode btree. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/1570 | 36 ++++++++++++++++++++++++++++++++++++ tests/xfs/1570.out | 4 ++++ tests/xfs/1571 | 36 ++++++++++++++++++++++++++++++++++++ tests/xfs/1571.out | 4 ++++ tests/xfs/1572 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/1572.out | 4 ++++ tests/xfs/1573 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/1573.out | 4 ++++ 8 files changed, 163 insertions(+) create mode 100755 tests/xfs/1570 create mode 100644 tests/xfs/1570.out create mode 100755 tests/xfs/1571 create mode 100644 tests/xfs/1571.out create mode 100755 tests/xfs/1572 create mode 100755 tests/xfs/1572.out create mode 100755 tests/xfs/1573 create mode 100644 tests/xfs/1573.out diff --git a/tests/xfs/1570 b/tests/xfs/1570 new file mode 100755 index 0000000000..c2d144e298 --- /dev/null +++ b/tests/xfs/1570 @@ -0,0 +1,36 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle, Inc. All Rights Reserved. +# +# FS QA Test No. 1570 +# +# Populate a XFS filesystem and fuzz every inobt key/pointer field. +# Use xfs_repair to fix the corruption. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'ino' 2)" || \ + _fail "could not find two-level inobt" + +echo "Fuzz inobt" +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'addr root' >> $seqres.full +echo "Done fuzzing inobt" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1570.out b/tests/xfs/1570.out new file mode 100644 index 0000000000..b3977dca57 --- /dev/null +++ b/tests/xfs/1570.out @@ -0,0 +1,4 @@ +QA output created by 1570 +Format and populate +Fuzz inobt +Done fuzzing inobt diff --git a/tests/xfs/1571 b/tests/xfs/1571 new file mode 100755 index 0000000000..c64b321ff6 --- /dev/null +++ b/tests/xfs/1571 @@ -0,0 +1,36 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle, Inc. All Rights Reserved. +# +# FS QA Test No. 1571 +# +# Populate a XFS filesystem and fuzz every inobt key/pointer field. +# Use xfs_scrub to fix the corruption. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'ino' 2)" || \ + _fail "could not find two-level inobt" + +echo "Fuzz inobt" +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'addr root' >> $seqres.full +echo "Done fuzzing inobt" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1571.out b/tests/xfs/1571.out new file mode 100644 index 0000000000..292e8bdec4 --- /dev/null +++ b/tests/xfs/1571.out @@ -0,0 +1,4 @@ +QA output created by 1571 +Format and populate +Fuzz inobt +Done fuzzing inobt diff --git a/tests/xfs/1572 b/tests/xfs/1572 new file mode 100755 index 0000000000..abcdc2397f --- /dev/null +++ b/tests/xfs/1572 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1572 +# +# Populate a XFS filesystem and fuzz every inobt key/pointer field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'ino' 2)" || \ + _fail "could not find two-level inobt" + +echo "Fuzz inobt" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'addr root' >> $seqres.full +echo "Done fuzzing inobt" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1572.out b/tests/xfs/1572.out new file mode 100755 index 0000000000..8afa3ea075 --- /dev/null +++ b/tests/xfs/1572.out @@ -0,0 +1,4 @@ +QA output created by 1572 +Format and populate +Fuzz inobt +Done fuzzing inobt diff --git a/tests/xfs/1573 b/tests/xfs/1573 new file mode 100755 index 0000000000..7a816e59b8 --- /dev/null +++ b/tests/xfs/1573 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle, Inc. All rights reserved. +# +# FS QA Test No. 1573 +# +# Populate a XFS filesystem and fuzz every inobt key/pointer field. +# Do not fix the filesystem, to test metadata verifiers. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_norepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'ino' 2)" || \ + _fail "could not find two-level inobt" + +echo "Fuzz inobt" +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'addr root' >> $seqres.full +echo "Done fuzzing inobt" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1573.out b/tests/xfs/1573.out new file mode 100644 index 0000000000..cef5aef758 --- /dev/null +++ b/tests/xfs/1573.out @@ -0,0 +1,4 @@ +QA output created by 1573 +Format and populate +Fuzz inobt +Done fuzzing inobt ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 4/5] xfs: fuzz test both repair strategies 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: strengthen fuzz testing Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 5/5] fuzzy: fuzz test key/pointers of inode btrees Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 3/5] fuzzy: test fuzzing realtime free space metadata Darrick J. Wong 4 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add more fuzz tests to examine the effectiveness of online and then offline repair. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/1500 | 35 +++++++++++++++++++++++++++++++++++ tests/xfs/1500.out | 4 ++++ tests/xfs/1501 | 35 +++++++++++++++++++++++++++++++++++ tests/xfs/1501.out | 4 ++++ tests/xfs/1502 | 45 +++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1502.out | 6 ++++++ tests/xfs/1503 | 35 +++++++++++++++++++++++++++++++++++ tests/xfs/1503.out | 4 ++++ tests/xfs/1504 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/1504.out | 4 ++++ tests/xfs/1505 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/1505.out | 4 ++++ tests/xfs/1506 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/1506.out | 4 ++++ tests/xfs/1507 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/1507.out | 4 ++++ tests/xfs/1508 | 39 +++++++++++++++++++++++++++++++++++++++ tests/xfs/1508.out | 4 ++++ tests/xfs/1509 | 39 +++++++++++++++++++++++++++++++++++++++ tests/xfs/1509.out | 4 ++++ tests/xfs/1510 | 39 +++++++++++++++++++++++++++++++++++++++ tests/xfs/1510.out | 4 ++++ tests/xfs/1511 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1511.out | 4 ++++ tests/xfs/1512 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1512.out | 5 +++++ tests/xfs/1513 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1513.out | 5 +++++ tests/xfs/1514 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1514.out | 5 +++++ tests/xfs/1515 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1515.out | 5 +++++ tests/xfs/1516 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1516.out | 5 +++++ tests/xfs/1517 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1517.out | 5 +++++ tests/xfs/1518 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1518.out | 5 +++++ tests/xfs/1519 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/1519.out | 5 +++++ tests/xfs/1520 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1520.out | 5 +++++ tests/xfs/1521 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1521.out | 5 +++++ tests/xfs/1522 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1522.out | 5 +++++ tests/xfs/1523 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1523.out | 5 +++++ tests/xfs/1524 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1524.out | 5 +++++ tests/xfs/1525 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1525.out | 5 +++++ tests/xfs/1526 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1526.out | 5 +++++ tests/xfs/1527 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1527.out | 5 +++++ tests/xfs/1530 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1530.out | 4 ++++ tests/xfs/1531 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1531.out | 5 +++++ tests/xfs/1532 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1532.out | 5 +++++ tests/xfs/1533 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1533.out | 5 +++++ tests/xfs/1534 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/1534.out | 4 ++++ tests/xfs/1535 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/1535.out | 4 ++++ tests/xfs/1536 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/1536.out | 4 ++++ tests/xfs/1537 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/1537.out | 5 +++++ tests/xfs/1560 | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1560.out | 10 ++++++++++ tests/xfs/1561 | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1561.out | 10 ++++++++++ 76 files changed, 1709 insertions(+) create mode 100755 tests/xfs/1500 create mode 100644 tests/xfs/1500.out create mode 100755 tests/xfs/1501 create mode 100644 tests/xfs/1501.out create mode 100755 tests/xfs/1502 create mode 100644 tests/xfs/1502.out create mode 100755 tests/xfs/1503 create mode 100644 tests/xfs/1503.out create mode 100755 tests/xfs/1504 create mode 100644 tests/xfs/1504.out create mode 100755 tests/xfs/1505 create mode 100644 tests/xfs/1505.out create mode 100755 tests/xfs/1506 create mode 100644 tests/xfs/1506.out create mode 100755 tests/xfs/1507 create mode 100644 tests/xfs/1507.out create mode 100755 tests/xfs/1508 create mode 100644 tests/xfs/1508.out create mode 100755 tests/xfs/1509 create mode 100644 tests/xfs/1509.out create mode 100755 tests/xfs/1510 create mode 100644 tests/xfs/1510.out create mode 100755 tests/xfs/1511 create mode 100644 tests/xfs/1511.out create mode 100755 tests/xfs/1512 create mode 100644 tests/xfs/1512.out create mode 100755 tests/xfs/1513 create mode 100644 tests/xfs/1513.out create mode 100755 tests/xfs/1514 create mode 100644 tests/xfs/1514.out create mode 100755 tests/xfs/1515 create mode 100644 tests/xfs/1515.out create mode 100755 tests/xfs/1516 create mode 100644 tests/xfs/1516.out create mode 100755 tests/xfs/1517 create mode 100644 tests/xfs/1517.out create mode 100755 tests/xfs/1518 create mode 100644 tests/xfs/1518.out create mode 100755 tests/xfs/1519 create mode 100644 tests/xfs/1519.out create mode 100755 tests/xfs/1520 create mode 100644 tests/xfs/1520.out create mode 100755 tests/xfs/1521 create mode 100644 tests/xfs/1521.out create mode 100755 tests/xfs/1522 create mode 100644 tests/xfs/1522.out create mode 100755 tests/xfs/1523 create mode 100644 tests/xfs/1523.out create mode 100755 tests/xfs/1524 create mode 100644 tests/xfs/1524.out create mode 100755 tests/xfs/1525 create mode 100644 tests/xfs/1525.out create mode 100755 tests/xfs/1526 create mode 100644 tests/xfs/1526.out create mode 100755 tests/xfs/1527 create mode 100644 tests/xfs/1527.out create mode 100755 tests/xfs/1530 create mode 100644 tests/xfs/1530.out create mode 100755 tests/xfs/1531 create mode 100644 tests/xfs/1531.out create mode 100755 tests/xfs/1532 create mode 100644 tests/xfs/1532.out create mode 100755 tests/xfs/1533 create mode 100644 tests/xfs/1533.out create mode 100755 tests/xfs/1534 create mode 100644 tests/xfs/1534.out create mode 100755 tests/xfs/1535 create mode 100644 tests/xfs/1535.out create mode 100755 tests/xfs/1536 create mode 100644 tests/xfs/1536.out create mode 100755 tests/xfs/1537 create mode 100644 tests/xfs/1537.out create mode 100755 tests/xfs/1560 create mode 100644 tests/xfs/1560.out create mode 100755 tests/xfs/1561 create mode 100644 tests/xfs/1561.out diff --git a/tests/xfs/1500 b/tests/xfs/1500 new file mode 100755 index 0000000000..cae5e05caf --- /dev/null +++ b/tests/xfs/1500 @@ -0,0 +1,35 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1500 +# +# Populate a XFS filesystem and fuzz every superblock field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz superblock" +_scratch_xfs_fuzz_metadata '' 'both' 'sb 1' >> $seqres.full +echo "Done fuzzing superblock" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1500.out b/tests/xfs/1500.out new file mode 100644 index 0000000000..bb485204c7 --- /dev/null +++ b/tests/xfs/1500.out @@ -0,0 +1,4 @@ +QA output created by 1500 +Format and populate +Fuzz superblock +Done fuzzing superblock diff --git a/tests/xfs/1501 b/tests/xfs/1501 new file mode 100755 index 0000000000..867584ebbe --- /dev/null +++ b/tests/xfs/1501 @@ -0,0 +1,35 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1501 +# +# Populate a XFS filesystem and fuzz every AGF field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz AGF" +_scratch_xfs_fuzz_metadata '' 'both' 'agf 0' >> $seqres.full +echo "Done fuzzing AGF" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1501.out b/tests/xfs/1501.out new file mode 100644 index 0000000000..0c8b1be2a5 --- /dev/null +++ b/tests/xfs/1501.out @@ -0,0 +1,4 @@ +QA output created by 1501 +Format and populate +Fuzz AGF +Done fuzzing AGF diff --git a/tests/xfs/1502 b/tests/xfs/1502 new file mode 100755 index 0000000000..04116d6d0e --- /dev/null +++ b/tests/xfs/1502 @@ -0,0 +1,45 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1502 +# +# Populate a XFS filesystem and fuzz every AGFL field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz AGFL" +_scratch_xfs_fuzz_metadata '' 'both' 'agfl 0' >> $seqres.full +echo "Done fuzzing AGFL" + +# Restore a correct copy of the filesystem before we start the second round of +# fuzzing. This avoids corruption errors from xfs_db when we probe for flfirst +# in the AGF and later when _scratch_xfs_fuzz_metadata probes the AGFL fields. +__scratch_xfs_fuzz_mdrestore +flfirst=$(_scratch_xfs_db -c 'agf 0' -c 'p flfirst' | sed -e 's/flfirst = //g') + +echo "Fuzz AGFL flfirst" +SCRATCH_XFS_LIST_METADATA_FIELDS="bno[${flfirst}]" _scratch_xfs_fuzz_metadata '' 'both' 'agfl 0' >> $seqres.full +echo "Done fuzzing AGFL flfirst" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1502.out b/tests/xfs/1502.out new file mode 100644 index 0000000000..99c0facdac --- /dev/null +++ b/tests/xfs/1502.out @@ -0,0 +1,6 @@ +QA output created by 1502 +Format and populate +Fuzz AGFL +Done fuzzing AGFL +Fuzz AGFL flfirst +Done fuzzing AGFL flfirst diff --git a/tests/xfs/1503 b/tests/xfs/1503 new file mode 100755 index 0000000000..8c4b125dfa --- /dev/null +++ b/tests/xfs/1503 @@ -0,0 +1,35 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1503 +# +# Populate a XFS filesystem and fuzz every AGI field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz AGI" +_scratch_xfs_fuzz_metadata '' 'both' 'agi 0' >> $seqres.full +echo "Done fuzzing AGI" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1503.out b/tests/xfs/1503.out new file mode 100644 index 0000000000..0029001317 --- /dev/null +++ b/tests/xfs/1503.out @@ -0,0 +1,4 @@ +QA output created by 1503 +Format and populate +Fuzz AGI +Done fuzzing AGI diff --git a/tests/xfs/1504 b/tests/xfs/1504 new file mode 100755 index 0000000000..e2712e646d --- /dev/null +++ b/tests/xfs/1504 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1504 +# +# Populate a XFS filesystem and fuzz every bnobt field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'bno' 2)" || \ + _fail "could not find two-level bnobt" + +echo "Fuzz bnobt recs" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'addr bnoroot' 'addr ptrs[1]' >> $seqres.full +echo "Done fuzzing bnobt recs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1504.out b/tests/xfs/1504.out new file mode 100644 index 0000000000..2c8162dd31 --- /dev/null +++ b/tests/xfs/1504.out @@ -0,0 +1,4 @@ +QA output created by 1504 +Format and populate +Fuzz bnobt recs +Done fuzzing bnobt recs diff --git a/tests/xfs/1505 b/tests/xfs/1505 new file mode 100755 index 0000000000..dbf850f8b4 --- /dev/null +++ b/tests/xfs/1505 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1505 +# +# Populate a XFS filesystem and fuzz every bnobt key/pointer. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'bno' 2)" || \ + _fail "could not find two-level bnobt" + +echo "Fuzz bnobt keyptr" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'addr bnoroot' >> $seqres.full +echo "Done fuzzing bnobt keyptr" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1505.out b/tests/xfs/1505.out new file mode 100644 index 0000000000..7c326a36dc --- /dev/null +++ b/tests/xfs/1505.out @@ -0,0 +1,4 @@ +QA output created by 1505 +Format and populate +Fuzz bnobt keyptr +Done fuzzing bnobt keyptr diff --git a/tests/xfs/1506 b/tests/xfs/1506 new file mode 100755 index 0000000000..efce4928fa --- /dev/null +++ b/tests/xfs/1506 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1506 +# +# Populate a XFS filesystem and fuzz every cntbt field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'cnt' 2)" || \ + _fail "could not find two-level cntbt" + +echo "Fuzz cntbt" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'addr cntroot' 'addr ptrs[1]' >> $seqres.full +echo "Done fuzzing cntbt" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1506.out b/tests/xfs/1506.out new file mode 100644 index 0000000000..bed44c625b --- /dev/null +++ b/tests/xfs/1506.out @@ -0,0 +1,4 @@ +QA output created by 1506 +Format and populate +Fuzz cntbt +Done fuzzing cntbt diff --git a/tests/xfs/1507 b/tests/xfs/1507 new file mode 100755 index 0000000000..7cece6854d --- /dev/null +++ b/tests/xfs/1507 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1507 +# +# Populate a XFS filesystem and fuzz every inobt field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'ino' 2)" || \ + _fail "could not find two-level inobt" + +echo "Fuzz inobt" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'addr root' 'addr ptrs[1]' >> $seqres.full +echo "Done fuzzing inobt" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1507.out b/tests/xfs/1507.out new file mode 100644 index 0000000000..3607a3a554 --- /dev/null +++ b/tests/xfs/1507.out @@ -0,0 +1,4 @@ +QA output created by 1507 +Format and populate +Fuzz inobt +Done fuzzing inobt diff --git a/tests/xfs/1508 b/tests/xfs/1508 new file mode 100755 index 0000000000..e2b38a9242 --- /dev/null +++ b/tests/xfs/1508 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1508 +# +# Populate a XFS filesystem and fuzz every finobt field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check +_require_xfs_finobt + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'fino' 2)" || \ + _fail "could not find two-level finobt" + +echo "Fuzz finobt" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'addr free_root' 'addr ptrs[1]' >> $seqres.full +echo "Done fuzzing finobt" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1508.out b/tests/xfs/1508.out new file mode 100644 index 0000000000..08b8e704b4 --- /dev/null +++ b/tests/xfs/1508.out @@ -0,0 +1,4 @@ +QA output created by 1508 +Format and populate +Fuzz finobt +Done fuzzing finobt diff --git a/tests/xfs/1509 b/tests/xfs/1509 new file mode 100755 index 0000000000..c90c4e6784 --- /dev/null +++ b/tests/xfs/1509 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1509 +# +# Populate a XFS filesystem and fuzz every rmapbt field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_xfs_scratch_rmapbt +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rmapbt" + +echo "Fuzz rmapbt recs" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'addr rmaproot' 'addr ptrs[1]' >> $seqres.full +echo "Done fuzzing rmapbt recs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1509.out b/tests/xfs/1509.out new file mode 100644 index 0000000000..95a25b005f --- /dev/null +++ b/tests/xfs/1509.out @@ -0,0 +1,4 @@ +QA output created by 1509 +Format and populate +Fuzz rmapbt recs +Done fuzzing rmapbt recs diff --git a/tests/xfs/1510 b/tests/xfs/1510 new file mode 100755 index 0000000000..fd8b994a98 --- /dev/null +++ b/tests/xfs/1510 @@ -0,0 +1,39 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1510 +# +# Populate a XFS filesystem and fuzz every rmapbt key/pointer field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_xfs_scratch_rmapbt +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rmapbt" + +echo "Fuzz rmapbt keyptr" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'addr rmaproot' >> $seqres.full +echo "Done fuzzing rmapbt keyptr" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1510.out b/tests/xfs/1510.out new file mode 100644 index 0000000000..0d5d467e2e --- /dev/null +++ b/tests/xfs/1510.out @@ -0,0 +1,4 @@ +QA output created by 1510 +Format and populate +Fuzz rmapbt keyptr +Done fuzzing rmapbt keyptr diff --git a/tests/xfs/1511 b/tests/xfs/1511 new file mode 100755 index 0000000000..ac945c17ed --- /dev/null +++ b/tests/xfs/1511 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1511 +# +# Populate a XFS filesystem and fuzz every refcountbt key/pointer field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_scratch_reflink +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level refcountbt" + +echo "Fuzz refcountbt" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'addr refcntroot' >> $seqres.full +echo "Done fuzzing refcountbt" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1511.out b/tests/xfs/1511.out new file mode 100644 index 0000000000..0628ebe519 --- /dev/null +++ b/tests/xfs/1511.out @@ -0,0 +1,4 @@ +QA output created by 1511 +Format and populate +Fuzz refcountbt +Done fuzzing refcountbt diff --git a/tests/xfs/1512 b/tests/xfs/1512 new file mode 100755 index 0000000000..9e3b859e1b --- /dev/null +++ b/tests/xfs/1512 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1512 +# +# Populate a XFS filesystem and fuzz every btree-format directory inode field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find btree-format dir inode" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_BTREE) +_scratch_unmount + +echo "Fuzz inode" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" >> $seqres.full +echo "Done fuzzing inode" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1512.out b/tests/xfs/1512.out new file mode 100644 index 0000000000..c28f0fc936 --- /dev/null +++ b/tests/xfs/1512.out @@ -0,0 +1,5 @@ +QA output created by 1512 +Format and populate +Find btree-format dir inode +Fuzz inode +Done fuzzing inode diff --git a/tests/xfs/1513 b/tests/xfs/1513 new file mode 100755 index 0000000000..5b1ed15290 --- /dev/null +++ b/tests/xfs/1513 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1513 +# +# Populate a XFS filesystem and fuzz every extents-format file inode field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find extents-format file inode" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFREG.FMT_EXTENTS) +_scratch_unmount + +echo "Fuzz inode" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" >> $seqres.full +echo "Done fuzzing inode" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1513.out b/tests/xfs/1513.out new file mode 100644 index 0000000000..6ee6f85ac3 --- /dev/null +++ b/tests/xfs/1513.out @@ -0,0 +1,5 @@ +QA output created by 1513 +Format and populate +Find extents-format file inode +Fuzz inode +Done fuzzing inode diff --git a/tests/xfs/1514 b/tests/xfs/1514 new file mode 100755 index 0000000000..8f530466d8 --- /dev/null +++ b/tests/xfs/1514 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1514 +# +# Populate a XFS filesystem and fuzz every btree-format file inode field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find btree-format file inode" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFREG.FMT_BTREE) +_scratch_unmount + +echo "Fuzz inode" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" >> $seqres.full +echo "Done fuzzing inode" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1514.out b/tests/xfs/1514.out new file mode 100644 index 0000000000..9bebc6d1a0 --- /dev/null +++ b/tests/xfs/1514.out @@ -0,0 +1,5 @@ +QA output created by 1514 +Format and populate +Find btree-format file inode +Fuzz inode +Done fuzzing inode diff --git a/tests/xfs/1515 b/tests/xfs/1515 new file mode 100755 index 0000000000..7a67448a40 --- /dev/null +++ b/tests/xfs/1515 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1515 +# +# Populate a XFS filesystem and fuzz every bmbt block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find bmbt block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFREG.FMT_BTREE) +_scratch_unmount + +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "inode ${inum}") + +echo "Fuzz bmbt" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" "addr u${inode_ver}.bmbt.ptrs[1]" >> $seqres.full +echo "Done fuzzing bmbt" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1515.out b/tests/xfs/1515.out new file mode 100644 index 0000000000..239ad87fa9 --- /dev/null +++ b/tests/xfs/1515.out @@ -0,0 +1,5 @@ +QA output created by 1515 +Format and populate +Find bmbt block +Fuzz bmbt +Done fuzzing bmbt diff --git a/tests/xfs/1516 b/tests/xfs/1516 new file mode 100755 index 0000000000..9c89964f85 --- /dev/null +++ b/tests/xfs/1516 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1516 +# +# Populate a XFS filesystem and fuzz every symlink remote block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find symlink remote block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFLNK.FMT_EXTENTS) +_scratch_unmount + +echo "Fuzz symlink remote block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" 'dblock 0' >> $seqres.full +echo "Done fuzzing symlink remote block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1516.out b/tests/xfs/1516.out new file mode 100644 index 0000000000..043db933d2 --- /dev/null +++ b/tests/xfs/1516.out @@ -0,0 +1,5 @@ +QA output created by 1516 +Format and populate +Find symlink remote block +Fuzz symlink remote block +Done fuzzing symlink remote block diff --git a/tests/xfs/1517 b/tests/xfs/1517 new file mode 100755 index 0000000000..e54f081281 --- /dev/null +++ b/tests/xfs/1517 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1517 +# +# Populate a XFS filesystem and fuzz every inline directory inode field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find inline-format dir inode" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_INLINE) +_scratch_unmount + +echo "Fuzz inline-format dir inode" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" >> $seqres.full +echo "Done fuzzing inline-format dir inode" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1517.out b/tests/xfs/1517.out new file mode 100644 index 0000000000..94063af2ab --- /dev/null +++ b/tests/xfs/1517.out @@ -0,0 +1,5 @@ +QA output created by 1517 +Format and populate +Find inline-format dir inode +Fuzz inline-format dir inode +Done fuzzing inline-format dir inode diff --git a/tests/xfs/1518 b/tests/xfs/1518 new file mode 100755 index 0000000000..14b1d74837 --- /dev/null +++ b/tests/xfs/1518 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1518 +# +# Populate a XFS filesystem and fuzz every block-format dir block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find data-format dir block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_BLOCK) +_scratch_unmount + +echo "Fuzz data-format dir block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" 'dblock 0' >> $seqres.full +echo "Done fuzzing data-format dir block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1518.out b/tests/xfs/1518.out new file mode 100644 index 0000000000..a3831196fd --- /dev/null +++ b/tests/xfs/1518.out @@ -0,0 +1,5 @@ +QA output created by 1518 +Format and populate +Find data-format dir block +Fuzz data-format dir block +Done fuzzing data-format dir block diff --git a/tests/xfs/1519 b/tests/xfs/1519 new file mode 100755 index 0000000000..98d719d33f --- /dev/null +++ b/tests/xfs/1519 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1519 +# +# Populate a XFS filesystem and fuzz every data-format dir block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find data-format dir block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_LEAF) +blk_sz=$(_get_block_size $SCRATCH_MNT) +_scratch_unmount + +echo "Fuzz data-format dir block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" "dblock 0" >> $seqres.full +echo "Done fuzzing data-format dir block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1519.out b/tests/xfs/1519.out new file mode 100644 index 0000000000..d1607473d5 --- /dev/null +++ b/tests/xfs/1519.out @@ -0,0 +1,5 @@ +QA output created by 1519 +Format and populate +Find data-format dir block +Fuzz data-format dir block +Done fuzzing data-format dir block diff --git a/tests/xfs/1520 b/tests/xfs/1520 new file mode 100755 index 0000000000..fedee5a52f --- /dev/null +++ b/tests/xfs/1520 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1520 +# +# Populate a XFS filesystem and fuzz every leaf1-format dir block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find leaf1-format dir block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_LEAF) +blk_sz=$(_get_block_size $SCRATCH_MNT) +_scratch_unmount + +leaf_offset=$(( (2 ** 35) / blk_sz)) +echo "Fuzz leaf1-format dir block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" "dblock ${leaf_offset}" >> $seqres.full +echo "Done fuzzing leaf1-format dir block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1520.out b/tests/xfs/1520.out new file mode 100644 index 0000000000..a97bf281f6 --- /dev/null +++ b/tests/xfs/1520.out @@ -0,0 +1,5 @@ +QA output created by 1520 +Format and populate +Find leaf1-format dir block +Fuzz leaf1-format dir block +Done fuzzing leaf1-format dir block diff --git a/tests/xfs/1521 b/tests/xfs/1521 new file mode 100755 index 0000000000..9d9bfcf407 --- /dev/null +++ b/tests/xfs/1521 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1521 +# +# Populate a XFS filesystem and fuzz every leafn-format dir block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find leafn-format dir block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_NODE) +blk_sz=$(_get_block_size $SCRATCH_MNT) +_scratch_unmount + +leaf_offset=$(( ( (2 ** 35) / blk_sz) + 1)) +echo "Fuzz leafn-format dir block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" "dblock ${leaf_offset}" >> $seqres.full +echo "Done fuzzing leafn-format dir block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1521.out b/tests/xfs/1521.out new file mode 100644 index 0000000000..c47f0f08c5 --- /dev/null +++ b/tests/xfs/1521.out @@ -0,0 +1,5 @@ +QA output created by 1521 +Format and populate +Find leafn-format dir block +Fuzz leafn-format dir block +Done fuzzing leafn-format dir block diff --git a/tests/xfs/1522 b/tests/xfs/1522 new file mode 100755 index 0000000000..1ae8eac378 --- /dev/null +++ b/tests/xfs/1522 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1522 +# +# Populate a XFS filesystem and fuzz every node-format dir block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find node-format dir block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_NODE) +blk_sz=$(_get_block_size $SCRATCH_MNT) +_scratch_unmount + +leaf_offset=$(( (2 ** 35) / blk_sz )) +echo "Fuzz node-format dir block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" "dblock ${leaf_offset}" >> $seqres.full +echo "Done fuzzing node-format dir block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1522.out b/tests/xfs/1522.out new file mode 100644 index 0000000000..ae26b05740 --- /dev/null +++ b/tests/xfs/1522.out @@ -0,0 +1,5 @@ +QA output created by 1522 +Format and populate +Find node-format dir block +Fuzz node-format dir block +Done fuzzing node-format dir block diff --git a/tests/xfs/1523 b/tests/xfs/1523 new file mode 100755 index 0000000000..345bc6e57a --- /dev/null +++ b/tests/xfs/1523 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1523 +# +# Populate a XFS filesystem and fuzz every freeindex-format dir block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find freeindex-format dir block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_NODE) +blk_sz=$(_get_block_size $SCRATCH_MNT) +_scratch_unmount + +leaf_offset=$(( (2 ** 36) / blk_sz )) +echo "Fuzz freeindex-format dir block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" "dblock ${leaf_offset}" >> $seqres.full +echo "Done fuzzing freeindex-format dir block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1523.out b/tests/xfs/1523.out new file mode 100644 index 0000000000..b4dd1270ae --- /dev/null +++ b/tests/xfs/1523.out @@ -0,0 +1,5 @@ +QA output created by 1523 +Format and populate +Find freeindex-format dir block +Fuzz freeindex-format dir block +Done fuzzing freeindex-format dir block diff --git a/tests/xfs/1524 b/tests/xfs/1524 new file mode 100755 index 0000000000..6602fac980 --- /dev/null +++ b/tests/xfs/1524 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1524 +# +# Populate a XFS filesystem and fuzz every inline attr inode field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find inline-format attr inode" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/ATTR.FMT_LOCAL) +_scratch_unmount + +echo "Fuzz inline-format attr inode" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" >> $seqres.full +echo "Done fuzzing inline-format attr inode" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1524.out b/tests/xfs/1524.out new file mode 100644 index 0000000000..2afb919ce8 --- /dev/null +++ b/tests/xfs/1524.out @@ -0,0 +1,5 @@ +QA output created by 1524 +Format and populate +Find inline-format attr inode +Fuzz inline-format attr inode +Done fuzzing inline-format attr inode diff --git a/tests/xfs/1525 b/tests/xfs/1525 new file mode 100755 index 0000000000..78ae622739 --- /dev/null +++ b/tests/xfs/1525 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1525 +# +# Populate a XFS filesystem and fuzz every leaf-format attr block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find leaf-format attr block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/ATTR.FMT_LEAF) +_scratch_unmount + +echo "Fuzz leaf-format attr block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" 'ablock 0' >> $seqres.full +echo "Done fuzzing leaf-format attr block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1525.out b/tests/xfs/1525.out new file mode 100644 index 0000000000..dda2601e13 --- /dev/null +++ b/tests/xfs/1525.out @@ -0,0 +1,5 @@ +QA output created by 1525 +Format and populate +Find leaf-format attr block +Fuzz leaf-format attr block +Done fuzzing leaf-format attr block diff --git a/tests/xfs/1526 b/tests/xfs/1526 new file mode 100755 index 0000000000..7efe2fdbfc --- /dev/null +++ b/tests/xfs/1526 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1526 +# +# Populate a XFS filesystem and fuzz every node-format attr block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find node-format attr block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/ATTR.FMT_NODE) +_scratch_unmount + +echo "Fuzz node-format attr block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" "ablock 0" >> $seqres.full +echo "Done fuzzing node-format attr block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1526.out b/tests/xfs/1526.out new file mode 100644 index 0000000000..ef7b4c7a9e --- /dev/null +++ b/tests/xfs/1526.out @@ -0,0 +1,5 @@ +QA output created by 1526 +Format and populate +Find node-format attr block +Fuzz node-format attr block +Done fuzzing node-format attr block diff --git a/tests/xfs/1527 b/tests/xfs/1527 new file mode 100755 index 0000000000..9ebcbe7117 --- /dev/null +++ b/tests/xfs/1527 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1527 +# +# Populate a XFS filesystem and fuzz every external attr block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find external attr block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/ATTR.FMT_EXTENTS_REMOTE3K) +_scratch_unmount + +echo "Fuzz external attr block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" "ablock 1" >> $seqres.full +echo "Done fuzzing external attr block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1527.out b/tests/xfs/1527.out new file mode 100644 index 0000000000..a79038cccf --- /dev/null +++ b/tests/xfs/1527.out @@ -0,0 +1,5 @@ +QA output created by 1527 +Format and populate +Find external attr block +Fuzz external attr block +Done fuzzing external attr block diff --git a/tests/xfs/1530 b/tests/xfs/1530 new file mode 100755 index 0000000000..6225391078 --- /dev/null +++ b/tests/xfs/1530 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1530 +# +# Populate a XFS filesystem and fuzz every refcountbt field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_scratch_reflink +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_agbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level refcountbt" + +echo "Fuzz refcountbt" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'addr refcntroot' 'addr ptrs[1]' >> $seqres.full +echo "Done fuzzing refcountbt" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1530.out b/tests/xfs/1530.out new file mode 100644 index 0000000000..4c2f39053e --- /dev/null +++ b/tests/xfs/1530.out @@ -0,0 +1,4 @@ +QA output created by 1530 +Format and populate +Fuzz refcountbt +Done fuzzing refcountbt diff --git a/tests/xfs/1531 b/tests/xfs/1531 new file mode 100755 index 0000000000..43a446e538 --- /dev/null +++ b/tests/xfs/1531 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1531 +# +# Populate a XFS filesystem and fuzz every btree-format attr inode field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find btree-format attr inode" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/ATTR.FMT_BTREE) +_scratch_unmount + +echo "Fuzz inode" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" >> $seqres.full +echo "Done fuzzing inode" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1531.out b/tests/xfs/1531.out new file mode 100644 index 0000000000..6c4deceaf0 --- /dev/null +++ b/tests/xfs/1531.out @@ -0,0 +1,5 @@ +QA output created by 1531 +Format and populate +Find btree-format attr inode +Fuzz inode +Done fuzzing inode diff --git a/tests/xfs/1532 b/tests/xfs/1532 new file mode 100755 index 0000000000..1aa65139a6 --- /dev/null +++ b/tests/xfs/1532 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1532 +# +# Populate a XFS filesystem and fuzz every blockdev inode field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find blockdev inode" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFBLK) +_scratch_unmount + +echo "Fuzz inode" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" >> $seqres.full +echo "Done fuzzing inode" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1532.out b/tests/xfs/1532.out new file mode 100644 index 0000000000..4bb5defc3f --- /dev/null +++ b/tests/xfs/1532.out @@ -0,0 +1,5 @@ +QA output created by 1532 +Format and populate +Find blockdev inode +Fuzz inode +Done fuzzing inode diff --git a/tests/xfs/1533 b/tests/xfs/1533 new file mode 100755 index 0000000000..e873432a0e --- /dev/null +++ b/tests/xfs/1533 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1533 +# +# Populate a XFS filesystem and fuzz every local-format symlink inode field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find local-format symlink inode" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFLNK.FMT_LOCAL) +_scratch_unmount + +echo "Fuzz inode" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" >> $seqres.full +echo "Done fuzzing inode" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1533.out b/tests/xfs/1533.out new file mode 100644 index 0000000000..198e5266fc --- /dev/null +++ b/tests/xfs/1533.out @@ -0,0 +1,5 @@ +QA output created by 1533 +Format and populate +Find local-format symlink inode +Fuzz inode +Done fuzzing inode diff --git a/tests/xfs/1534 b/tests/xfs/1534 new file mode 100755 index 0000000000..20a0faa56f --- /dev/null +++ b/tests/xfs/1534 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1534 +# +# Populate a XFS filesystem and fuzz every user dquot field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check +_require_quota + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 +echo "${MOUNT_OPTIONS}" | grep -q 'usrquota' || _notrun "user quota disabled" + +echo "Fuzz user 0 dquot" +_scratch_xfs_fuzz_metadata '' 'both' "dquot -u 0" >> $seqres.full +echo "Done fuzzing dquot" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1534.out b/tests/xfs/1534.out new file mode 100644 index 0000000000..0e5484adfc --- /dev/null +++ b/tests/xfs/1534.out @@ -0,0 +1,4 @@ +QA output created by 1534 +Format and populate +Fuzz user 0 dquot +Done fuzzing dquot diff --git a/tests/xfs/1535 b/tests/xfs/1535 new file mode 100755 index 0000000000..c6b268621c --- /dev/null +++ b/tests/xfs/1535 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1535 +# +# Populate a XFS filesystem and fuzz every group dquot field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check +_require_quota + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 +echo "${MOUNT_OPTIONS}" | grep -q 'grpquota' || _notrun "group quota disabled" + +echo "Fuzz group 0 dquot" +_scratch_xfs_fuzz_metadata '' 'both' "dquot -g 0" >> $seqres.full +echo "Done fuzzing dquot" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1535.out b/tests/xfs/1535.out new file mode 100644 index 0000000000..ad317e3673 --- /dev/null +++ b/tests/xfs/1535.out @@ -0,0 +1,4 @@ +QA output created by 1535 +Format and populate +Fuzz group 0 dquot +Done fuzzing dquot diff --git a/tests/xfs/1536 b/tests/xfs/1536 new file mode 100755 index 0000000000..20d054df6f --- /dev/null +++ b/tests/xfs/1536 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1536 +# +# Populate a XFS filesystem and fuzz every project dquot field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/quota + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check +_require_quota + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 +echo "${MOUNT_OPTIONS}" | grep -q 'prjquota' || _notrun "project quota disabled" + +echo "Fuzz project 0 dquot" +_scratch_xfs_fuzz_metadata '' 'both' "dquot -p 0" >> $seqres.full +echo "Done fuzzing dquot" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1536.out b/tests/xfs/1536.out new file mode 100644 index 0000000000..8738d707dd --- /dev/null +++ b/tests/xfs/1536.out @@ -0,0 +1,4 @@ +QA output created by 1536 +Format and populate +Fuzz project 0 dquot +Done fuzzing dquot diff --git a/tests/xfs/1537 b/tests/xfs/1537 new file mode 100755 index 0000000000..21f962dfaf --- /dev/null +++ b/tests/xfs/1537 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1537 +# +# Populate a XFS filesystem and fuzz every single-leafn-format dir block field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Find single-leafn-format dir block" +_scratch_mount +inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_LEAFN) +blk_sz=$(_get_block_size $SCRATCH_MNT) +_scratch_unmount + +leaf_offset=$(( (2 ** 35) / blk_sz )) +echo "Fuzz single-leafn-format dir block" +_scratch_xfs_fuzz_metadata '' 'both' "inode ${inum}" "dblock ${leaf_offset}" >> $seqres.full +echo "Done fuzzing single-leafn-format dir block" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1537.out b/tests/xfs/1537.out new file mode 100644 index 0000000000..f298ecce6b --- /dev/null +++ b/tests/xfs/1537.out @@ -0,0 +1,5 @@ +QA output created by 1537 +Format and populate +Find single-leafn-format dir block +Fuzz single-leafn-format dir block +Done fuzzing single-leafn-format dir block diff --git a/tests/xfs/1560 b/tests/xfs/1560 new file mode 100755 index 0000000000..456f079919 --- /dev/null +++ b/tests/xfs/1560 @@ -0,0 +1,49 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1560 +# +# Populate a XFS filesystem and fuzz the data mappings of every directory type. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +_scratch_xfs_set_dir_fuzz_types + +# Now fuzz the block maps of each directory type. +for dirtype in "${SCRATCH_XFS_DIR_FUZZ_TYPES[@]}"; do + echo "Fuzz block map for ${dirtype}" | tee -a $seqres.full + + # Restore a correct copy of the filesystem before we start a round of + # fuzzing. This avoids corruption errors from xfs_db when + # _scratch_xfs_fuzz_metadata probes the directory block fields. + __scratch_xfs_fuzz_mdrestore + + _scratch_mount + inum=$(stat -c '%i' $SCRATCH_MNT/S_IFDIR.FMT_${dirtype}) + _scratch_unmount + + _scratch_xfs_fuzz_metadata 'u*.bmx' 'both' "inode ${inum}" >> $seqres.full + echo "Done fuzzing dir map ${dirtype}" +done + +# success, all done +status=0 +exit diff --git a/tests/xfs/1560.out b/tests/xfs/1560.out new file mode 100644 index 0000000000..429cd3e0ce --- /dev/null +++ b/tests/xfs/1560.out @@ -0,0 +1,10 @@ +QA output created by 1560 +Format and populate +Fuzz block map for BLOCK +Done fuzzing dir map BLOCK +Fuzz block map for LEAF +Done fuzzing dir map LEAF +Fuzz block map for LEAFN +Done fuzzing dir map LEAFN +Fuzz block map for NODE +Done fuzzing dir map NODE diff --git a/tests/xfs/1561 b/tests/xfs/1561 new file mode 100755 index 0000000000..936e4c264f --- /dev/null +++ b/tests/xfs/1561 @@ -0,0 +1,49 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1561 +# +# Populate a XFS filesystem and fuzz the attr mappings of every xattr type. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. +# +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_scratch_xfs_fuzz_fields + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +_scratch_xfs_set_xattr_fuzz_types + +# Now fuzz the block maps of each xattr type. +for attrtype in "${SCRATCH_XFS_XATTR_FUZZ_TYPES[@]}"; do + echo "Fuzz block map for ${attrtype}" | tee -a $seqres.full + + # Restore a correct copy of the filesystem before we start a round of + # fuzzing. This avoids corruption errors from xfs_db when + # _scratch_xfs_fuzz_metadata probes the xattr block fields. + __scratch_xfs_fuzz_mdrestore + + _scratch_mount + inum=$(stat -c '%i' $SCRATCH_MNT/ATTR.FMT_${attrtype}) + _scratch_unmount + + _scratch_xfs_fuzz_metadata 'a*.bmx' 'both' "inode ${inum}" >> $seqres.full + echo "Done fuzzing attr map ${attrtype}" +done + +# success, all done +status=0 +exit diff --git a/tests/xfs/1561.out b/tests/xfs/1561.out new file mode 100644 index 0000000000..39e5dcba71 --- /dev/null +++ b/tests/xfs/1561.out @@ -0,0 +1,10 @@ +QA output created by 1561 +Format and populate +Fuzz block map for EXTENTS_REMOTE3K +Done fuzzing attr map EXTENTS_REMOTE3K +Fuzz block map for EXTENTS_REMOTE4K +Done fuzzing attr map EXTENTS_REMOTE4K +Fuzz block map for LEAF +Done fuzzing attr map LEAF +Fuzz block map for NODE +Done fuzzing attr map NODE ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 3/5] fuzzy: test fuzzing realtime free space metadata 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: strengthen fuzz testing Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 4/5] xfs: fuzz test both repair strategies Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 4 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Fuzz the contents of the realtime bitmap and summary files to see what happens. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 107 ++++++++++++++++++++++++++++++++++++++++++++++++++-- tests/xfs/1562 | 41 ++++++++++++++++++++ tests/xfs/1562.out | 4 ++ tests/xfs/1563 | 41 ++++++++++++++++++++ tests/xfs/1563.out | 4 ++ tests/xfs/1564 | 41 ++++++++++++++++++++ tests/xfs/1564.out | 4 ++ tests/xfs/1565 | 41 ++++++++++++++++++++ tests/xfs/1565.out | 4 ++ tests/xfs/1566 | 42 ++++++++++++++++++++ tests/xfs/1566.out | 4 ++ tests/xfs/1567 | 42 ++++++++++++++++++++ tests/xfs/1567.out | 4 ++ tests/xfs/1568 | 41 ++++++++++++++++++++ tests/xfs/1568.out | 4 ++ tests/xfs/1569 | 41 ++++++++++++++++++++ tests/xfs/1569.out | 4 ++ 17 files changed, 465 insertions(+), 4 deletions(-) create mode 100755 tests/xfs/1562 create mode 100644 tests/xfs/1562.out create mode 100755 tests/xfs/1563 create mode 100644 tests/xfs/1563.out create mode 100755 tests/xfs/1564 create mode 100644 tests/xfs/1564.out create mode 100755 tests/xfs/1565 create mode 100644 tests/xfs/1565.out create mode 100755 tests/xfs/1566 create mode 100644 tests/xfs/1566.out create mode 100755 tests/xfs/1567 create mode 100644 tests/xfs/1567.out create mode 100755 tests/xfs/1568 create mode 100644 tests/xfs/1568.out create mode 100755 tests/xfs/1569 create mode 100644 tests/xfs/1569.out diff --git a/common/fuzzy b/common/fuzzy index d8eb7d8b72..ef54f2fe2c 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -154,6 +154,12 @@ _scratch_xfs_dump_metadata() { _scratch_xfs_db "${cmds[@]}" -c print } +# Decide from the output of the xfs_db "stack" command if the debugger's io +# cursor is pointed at a block that is an unstructured data format (blob). +__scratch_xfs_detect_blob_from_stack() { + grep -q -E 'inode.*, type (data|rtsummary|rtbitmap)' +} + # Navigate to some part of the filesystem and print the field info. # The first argument is an grep filter for the fields # The rest of the arguments are xfs_db commands to locate the metadata. @@ -169,7 +175,17 @@ _scratch_xfs_list_metadata_fields() { for arg in "$@"; do cmds+=("-c" "${arg}") done - _scratch_xfs_db "${cmds[@]}" -c print | __filter_xfs_db_print_fields "${filter}" + + # Does the path argument point towards something that is an + # unstructured blob? + if _scratch_xfs_db "${cmds[@]}" -c stack 2>/dev/null | \ + __scratch_xfs_detect_blob_from_stack; then + echo "<blob>" + return + fi + + _scratch_xfs_db "${cmds[@]}" -c print | \ + __filter_xfs_db_print_fields "${filter}" } # Fuzz a metadata field @@ -207,6 +223,70 @@ _scratch_xfs_fuzz_metadata_field() { return 0 } +# List the fuzzing verbs available for unstructured blobs +__scratch_xfs_list_blob_fuzz_verbs() { + cat << ENDL +zeroes +ones +firstbit +middlebit +lastbit +random +ENDL +} + +# Fuzz a metadata blob +# The first arg is a blob fuzzing verb +# The rest of the arguments are xfs_db commands to find the metadata. +_scratch_xfs_fuzz_metadata_blob() { + local fuzzverb="$1" + shift + local trashcmd=(blocktrash -z) + + local cmds=() + for arg in "$@"; do + cmds+=("-c" "${arg}") + done + + local bytecount=$(_scratch_xfs_db "${cmds[@]}" -c "stack" | grep 'byte.*length' | awk '{print $5}') + local bitmax=$((bytecount * 8)) + + case "${fuzzverb}" in + "zeroes") + trashcmd+=(-0 -o 0 -x "${bitmax}" -y "${bitmax}");; + "ones") + trashcmd+=(-1 -o 0 -x "${bitmax}" -y "${bitmax}");; + "firstbit") + trashcmd+=(-2 -o 0 -x 1 -y 1);; + "middlebit") + trashcmd+=(-2 -o $((bitmax / 2)) -x 1 -y 1);; + "lastbit") + trashcmd+=(-2 -o "${bitmax}" -x 1 -y 1);; + "random") + trashcmd+=(-3 -o 0 -x "${bitmax}" -y "${bitmax}");; + *) + echo "Unknown blob fuzz verb \"${fuzzverb}\"." + return 1 + ;; + esac + + trashcmd="${trashcmd[@]}" + oldval="$(_scratch_xfs_get_metadata_field "" "$@")" + while true; do + _scratch_xfs_db -x "${cmds[@]}" -c "${trashcmd}" + echo + newval="$(_scratch_xfs_get_metadata_field "" "$@" 2> /dev/null)" + if [ "${fuzzverb}" != "random" ] || [ "${oldval}" != "${newval}" ]; then + break; + fi + done + if [ "${oldval}" = "${newval}" ]; then + echo "Blob already set to new value, skipping test." + return 1 + fi + return 0 +} + # Try to forcibly unmount the scratch fs __scratch_xfs_fuzz_unmount() { @@ -503,7 +583,11 @@ __scratch_xfs_fuzz_field_test() { # Set the new field value __fuzz_notify "+ Fuzz ${field} = ${fuzzverb}" - _scratch_xfs_fuzz_metadata_field "${field}" ${fuzzverb} "$@" + if [ "$field" = "<blob>" ]; then + _scratch_xfs_fuzz_metadata_blob ${fuzzverb} "$@" + else + _scratch_xfs_fuzz_metadata_field "${field}" ${fuzzverb} "$@" + fi res=$? test $res -ne 0 && return @@ -587,7 +671,22 @@ _scratch_xfs_list_fuzz_verbs() { echo "${SCRATCH_XFS_LIST_FUZZ_VERBS}" | tr '[ ,]' '[\n\n]' return; fi - _scratch_xfs_db -x -c 'sb 0' -c 'fuzz' | grep '^Fuzz commands:' | \ + + local cmds=() + for arg in "$@"; do + cmds+=("-c" "${arg}") + done + test "${#cmds[@]}" -eq 0 && cmds=('-c' 'sb 0') + + # Does the path argument point towards something that is an + # unstructured blob? + if _scratch_xfs_db "${cmds[@]}" -c stack 2>/dev/null | \ + __scratch_xfs_detect_blob_from_stack; then + __scratch_xfs_list_blob_fuzz_verbs + return + fi + + _scratch_xfs_db -x "${cmds[@]}" -c 'fuzz' | grep '^Fuzz commands:' | \ sed -e 's/[,.]//g' -e 's/Fuzz commands: //g' -e 's/ /\n/g' | \ grep -v '^random$' } @@ -605,7 +704,7 @@ _scratch_xfs_fuzz_metadata() { shift; shift fields="$(_scratch_xfs_list_metadata_fields "${filter}" "$@")" - verbs="$(_scratch_xfs_list_fuzz_verbs)" + verbs="$(_scratch_xfs_list_fuzz_verbs "$@")" echo "Fields we propose to fuzz with the \"${repair}\" repair strategy: $@" echo $(echo "${fields}") echo "Verbs we propose to fuzz with:" diff --git a/tests/xfs/1562 b/tests/xfs/1562 new file mode 100755 index 0000000000..015209eeb2 --- /dev/null +++ b/tests/xfs/1562 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1562 +# +# Populate a XFS filesystem and fuzz every realtime bitmap field. +# Use xfs_scrub to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz rtbitmap" +is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') +if [ -n "$is_metadir" ]; then + path=('path -m /realtime/0.bitmap') +else + path=('sb' 'addr rbmino') +fi +_scratch_xfs_fuzz_metadata '' 'online' "${path[@]}" 'dblock 0' >> $seqres.full +echo "Done fuzzing rtbitmap" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1562.out b/tests/xfs/1562.out new file mode 100644 index 0000000000..63a3bc7600 --- /dev/null +++ b/tests/xfs/1562.out @@ -0,0 +1,4 @@ +QA output created by 1562 +Format and populate +Fuzz rtbitmap +Done fuzzing rtbitmap diff --git a/tests/xfs/1563 b/tests/xfs/1563 new file mode 100755 index 0000000000..2be0870a3d --- /dev/null +++ b/tests/xfs/1563 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1563 +# +# Populate a XFS filesystem and fuzz every realtime summary field. +# Use xfs_scrub to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz rtsummary" +is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') +if [ -n "$is_metadir" ]; then + path=('path -m /realtime/0.summary') +else + path=('sb' 'addr rsumino') +fi +_scratch_xfs_fuzz_metadata '' 'online' "${path[@]}" 'dblock 0' >> $seqres.full +echo "Done fuzzing rtsummary" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1563.out b/tests/xfs/1563.out new file mode 100644 index 0000000000..e4ca8c3733 --- /dev/null +++ b/tests/xfs/1563.out @@ -0,0 +1,4 @@ +QA output created by 1563 +Format and populate +Fuzz rtsummary +Done fuzzing rtsummary diff --git a/tests/xfs/1564 b/tests/xfs/1564 new file mode 100755 index 0000000000..c0d10ff0e9 --- /dev/null +++ b/tests/xfs/1564 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1564 +# +# Populate a XFS filesystem and fuzz every realtime bitmap field. +# Use xfs_repair to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_repair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz rtbitmap" +is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') +if [ -n "$is_metadir" ]; then + path=('path -m /realtime/0.bitmap') +else + path=('sb' 'addr rbmino') +fi +_scratch_xfs_fuzz_metadata '' 'offline' "${path[@]}" 'dblock 0' >> $seqres.full +echo "Done fuzzing rtbitmap" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1564.out b/tests/xfs/1564.out new file mode 100644 index 0000000000..afede817b8 --- /dev/null +++ b/tests/xfs/1564.out @@ -0,0 +1,4 @@ +QA output created by 1564 +Format and populate +Fuzz rtbitmap +Done fuzzing rtbitmap diff --git a/tests/xfs/1565 b/tests/xfs/1565 new file mode 100755 index 0000000000..6b4186fb3c --- /dev/null +++ b/tests/xfs/1565 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1565 +# +# Populate a XFS filesystem and fuzz every realtime summary field. +# Use xfs_repair to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_repair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz rtsummary" +is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') +if [ -n "$is_metadir" ]; then + path=('path -m /realtime/0.summary') +else + path=('sb' 'addr rsumino') +fi +_scratch_xfs_fuzz_metadata '' 'offline' "${path[@]}" 'dblock 0' >> $seqres.full +echo "Done fuzzing rtsummary" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1565.out b/tests/xfs/1565.out new file mode 100644 index 0000000000..7a8d9d04d6 --- /dev/null +++ b/tests/xfs/1565.out @@ -0,0 +1,4 @@ +QA output created by 1565 +Format and populate +Fuzz rtsummary +Done fuzzing rtsummary diff --git a/tests/xfs/1566 b/tests/xfs/1566 new file mode 100755 index 0000000000..8d0f61ae10 --- /dev/null +++ b/tests/xfs/1566 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1566 +# +# Populate a XFS filesystem and fuzz every realtime bitmap field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_bothrepair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz rtbitmap" +is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') +if [ -n "$is_metadir" ]; then + path=('path -m /realtime/0.bitmap') +else + path=('sb' 'addr rbmino') +fi +_scratch_xfs_fuzz_metadata '' 'both' "${path[@]}" 'dblock 0' >> $seqres.full +echo "Done fuzzing rtbitmap" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1566.out b/tests/xfs/1566.out new file mode 100644 index 0000000000..d50e1d8539 --- /dev/null +++ b/tests/xfs/1566.out @@ -0,0 +1,4 @@ +QA output created by 1566 +Format and populate +Fuzz rtbitmap +Done fuzzing rtbitmap diff --git a/tests/xfs/1567 b/tests/xfs/1567 new file mode 100755 index 0000000000..7dc2012b67 --- /dev/null +++ b/tests/xfs/1567 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1567 +# +# Populate a XFS filesystem and fuzz every realtime summary field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_bothrepair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz rtsummary" +is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') +if [ -n "$is_metadir" ]; then + path=('path -m /realtime/0.summary') +else + path=('sb' 'addr rsumino') +fi +_scratch_xfs_fuzz_metadata '' 'both' "${path[@]}" 'dblock 0' >> $seqres.full +echo "Done fuzzing rtsummary" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1567.out b/tests/xfs/1567.out new file mode 100644 index 0000000000..b88fa0c1b3 --- /dev/null +++ b/tests/xfs/1567.out @@ -0,0 +1,4 @@ +QA output created by 1567 +Format and populate +Fuzz rtsummary +Done fuzzing rtsummary diff --git a/tests/xfs/1568 b/tests/xfs/1568 new file mode 100755 index 0000000000..c80640ef97 --- /dev/null +++ b/tests/xfs/1568 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1568 +# +# Populate a XFS filesystem and fuzz every realtime bitmap field. +# Do not fix the filesystem, to test metadata verifiers. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_norepair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz rtbitmap" +is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') +if [ -n "$is_metadir" ]; then + path=('path -m /realtime/0.bitmap') +else + path=('sb' 'addr rbmino') +fi +_scratch_xfs_fuzz_metadata '' 'none' "${path[@]}" 'dblock 0' >> $seqres.full +echo "Done fuzzing rtbitmap" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1568.out b/tests/xfs/1568.out new file mode 100644 index 0000000000..a80f579662 --- /dev/null +++ b/tests/xfs/1568.out @@ -0,0 +1,4 @@ +QA output created by 1568 +Format and populate +Fuzz rtbitmap +Done fuzzing rtbitmap diff --git a/tests/xfs/1569 b/tests/xfs/1569 new file mode 100755 index 0000000000..e303f08ff5 --- /dev/null +++ b/tests/xfs/1569 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1569 +# +# Populate a XFS filesystem and fuzz every realtime summary field. +# Do not fix the filesystem, to test metadata verifiers. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_norepair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +echo "Fuzz rtsummary" +is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') +if [ -n "$is_metadir" ]; then + path=('path -m /realtime/0.summary') +else + path=('sb' 'addr rsumino') +fi +_scratch_xfs_fuzz_metadata '' 'none' "${path[@]}" 'dblock 0' >> $seqres.full +echo "Done fuzzing rtsummary" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1569.out b/tests/xfs/1569.out new file mode 100644 index 0000000000..d6202cc3af --- /dev/null +++ b/tests/xfs/1569.out @@ -0,0 +1,4 @@ +QA output created by 1569 +Format and populate +Fuzz rtsummary +Done fuzzing rtsummary ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/7] fstests: atomic file updates 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (12 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: strengthen fuzz testing Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/7] xfs/122: fix for swapext log items Darrick J. Wong ` (6 more replies) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of realtime summaries Darrick J. Wong ` (3 subsequent siblings) 17 siblings, 7 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, This series creates a new FIEXCHANGE_RANGE system call to exchange ranges of bytes between two files atomically. This new functionality enables data storage programs to stage and commit file updates such that reader programs will see either the old contents or the new contents in their entirety, with no chance of torn writes. A successful call completion guarantees that the new contents will be seen even if the system fails. The ability to swap extent mappings between files in this manner is critical to supporting online filesystem repair, which is built upon the strategy of constructing a clean copy of a damaged structure and committing the new structure into the metadata file atomically. User programs will be able to update files atomically by opening an O_TMPFILE, reflinking the source file to it, making whatever updates they want to make, and exchange the relevant ranges of the temp file with the original file. If the updates are aligned with the file block size, a new (since v2) flag provides for exchanging only the written areas. Callers can arrange for the update to be rejected if the original file has been changed. The intent behind this new userspace functionality is to enable atomic rewrites of arbitrary parts of individual files. For years, application programmers wanting to ensure the atomicity of a file update had to write the changes to a new file in the same directory, fsync the new file, rename the new file on top of the old filename, and then fsync the directory. People get it wrong all the time, and $fs hacks abound. The reference implementation in XFS creates a new log incompat feature and log intent items to track high level progress of swapping ranges of two files and finish interrupted work if the system goes down. Sample code can be found in the corresponding changes to xfs_io to exercise the use case mentioned above. Note that this function is /not/ the O_DIRECT atomic file writes concept that has also been floating around for years. This RFC is constructed entirely in software, which means that there are no limitations other than the general filesystem limits. As a side note, the original motivation behind the kernel functionality is online repair of file-based metadata. The atomic file swap is implemented as an atomic inode fork swap, which means that we can implement online reconstruction of extended attributes and directories by building a new one in another inode and atomically swap the contents. Subsequent patchsets adapt the online filesystem repair code to use atomic extent swapping. This enables repair functions to construct a clean copy of a directory, xattr information, symbolic links, realtime bitmaps, and realtime summary information in a temporary inode. If this completes successfully, the new contents can be swapped atomically into the inode being repaired. This is essential to avoid making corruption problems worse if the system goes down in the middle of running repair. This patchset also ports the old XFS extent swap ioctl interface to use the new extent swap code. For userspace, this series also includes the userspace pieces needed to test the new functionality, and a sample implementation of atomic file updates. Question: Should we really bother with fsdevel bikeshedding? Most filesystems cannot support this functionality, so we could keep it private to XFS for now. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=atomic-file-updates xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=atomic-file-updates fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=atomic-file-updates xfsdocs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=atomic-file-updates --- common/rc | 24 +++++++ common/reflink | 33 +++++++++ common/xfs | 19 +++++ configure.ac | 1 doc/group-names.txt | 2 + include/builddefs.in | 1 ltp/Makefile | 4 + ltp/fsstress.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++ ltp/fsx.c | 160 +++++++++++++++++++++++++++++++++++++++++++++- m4/package_libcdev.m4 | 21 ++++++ src/fiexchange.h | 101 +++++++++++++++++++++++++++++ src/global.h | 6 ++ tests/generic/1200 | 55 ++++++++++++++++ tests/generic/1200.out | 3 + tests/generic/1201 | 53 +++++++++++++++ tests/generic/1201.out | 4 + tests/generic/1202 | 47 +++++++++++++ tests/generic/1202.out | 2 + tests/generic/1203 | 58 +++++++++++++++++ tests/generic/1203.out | 2 + tests/generic/1204 | 100 +++++++++++++++++++++++++++++ tests/generic/1204.out | 86 +++++++++++++++++++++++++ tests/generic/1205 | 116 +++++++++++++++++++++++++++++++++ tests/generic/1205.out | 90 ++++++++++++++++++++++++++ tests/generic/1206 | 76 ++++++++++++++++++++++ tests/generic/1206.out | 32 +++++++++ tests/generic/1207 | 122 +++++++++++++++++++++++++++++++++++ tests/generic/1207.out | 48 ++++++++++++++ tests/generic/1209 | 101 +++++++++++++++++++++++++++++ tests/generic/1209.out | 33 +++++++++ tests/generic/1210 | 48 ++++++++++++++ tests/generic/1210.out | 6 ++ tests/generic/1211 | 105 ++++++++++++++++++++++++++++++ tests/generic/1211.out | 40 +++++++++++ tests/generic/1212 | 58 +++++++++++++++++ tests/generic/1212.out | 2 + tests/generic/1213 | 126 ++++++++++++++++++++++++++++++++++++ tests/generic/1213.out | 48 ++++++++++++++ tests/generic/1214 | 63 ++++++++++++++++++ tests/generic/1214.out | 2 + tests/generic/1215 | 70 ++++++++++++++++++++ tests/generic/1215.out | 2 + tests/generic/1216 | 53 +++++++++++++++ tests/generic/1216.out | 6 ++ tests/generic/1217 | 56 ++++++++++++++++ tests/generic/1217.out | 4 + tests/generic/1218 | 115 +++++++++++++++++++++++++++++++++ tests/generic/1218.out | 49 ++++++++++++++ tests/generic/1219 | 83 ++++++++++++++++++++++++ tests/generic/1219.out | 17 +++++ tests/xfs/1202 | 59 +++++++++++++++++ tests/xfs/1202.out | 12 +++ tests/xfs/1208 | 62 ++++++++++++++++++ tests/xfs/1208.out | 10 +++ tests/xfs/1211 | 59 +++++++++++++++++ tests/xfs/1211.out | 7 ++ tests/xfs/1212 | 61 +++++++++++++++++ tests/xfs/1212.out | 5 + tests/xfs/122.out | 3 + tests/xfs/537 | 2 - 60 files changed, 2798 insertions(+), 3 deletions(-) create mode 100644 src/fiexchange.h create mode 100755 tests/generic/1200 create mode 100644 tests/generic/1200.out create mode 100755 tests/generic/1201 create mode 100644 tests/generic/1201.out create mode 100755 tests/generic/1202 create mode 100644 tests/generic/1202.out create mode 100755 tests/generic/1203 create mode 100644 tests/generic/1203.out create mode 100755 tests/generic/1204 create mode 100644 tests/generic/1204.out create mode 100755 tests/generic/1205 create mode 100644 tests/generic/1205.out create mode 100755 tests/generic/1206 create mode 100644 tests/generic/1206.out create mode 100755 tests/generic/1207 create mode 100644 tests/generic/1207.out create mode 100755 tests/generic/1209 create mode 100644 tests/generic/1209.out create mode 100755 tests/generic/1210 create mode 100644 tests/generic/1210.out create mode 100755 tests/generic/1211 create mode 100644 tests/generic/1211.out create mode 100755 tests/generic/1212 create mode 100644 tests/generic/1212.out create mode 100755 tests/generic/1213 create mode 100644 tests/generic/1213.out create mode 100755 tests/generic/1214 create mode 100644 tests/generic/1214.out create mode 100755 tests/generic/1215 create mode 100644 tests/generic/1215.out create mode 100755 tests/generic/1216 create mode 100644 tests/generic/1216.out create mode 100755 tests/generic/1217 create mode 100644 tests/generic/1217.out create mode 100755 tests/generic/1218 create mode 100644 tests/generic/1218.out create mode 100755 tests/generic/1219 create mode 100755 tests/generic/1219.out create mode 100755 tests/xfs/1202 create mode 100644 tests/xfs/1202.out create mode 100755 tests/xfs/1208 create mode 100644 tests/xfs/1208.out create mode 100755 tests/xfs/1211 create mode 100644 tests/xfs/1211.out create mode 100755 tests/xfs/1212 create mode 100644 tests/xfs/1212.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/7] xfs/122: fix for swapext log items 2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 5/7] generic: test that file privilege gets dropped with FIEXCHANGE_RANGE Darrick J. Wong ` (5 subsequent siblings) 6 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add entries for the extent swapping log items. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/122.out | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/xfs/122.out b/tests/xfs/122.out index 95e53c5081..21549db7fd 100644 --- a/tests/xfs/122.out +++ b/tests/xfs/122.out @@ -117,6 +117,9 @@ sizeof(struct xfs_rtrmap_root) = 4 sizeof(struct xfs_rud_log_format) = 16 sizeof(struct xfs_rui_log_format) = 16 sizeof(struct xfs_scrub_metadata) = 64 +sizeof(struct xfs_swap_extent) = 64 +sizeof(struct xfs_sxd_log_format) = 16 +sizeof(struct xfs_sxi_log_format) = 80 sizeof(struct xfs_unmount_log_format) = 8 sizeof(xfs_agf_t) = 224 sizeof(xfs_agfl_t) = 36 ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 5/7] generic: test that file privilege gets dropped with FIEXCHANGE_RANGE 2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/7] xfs/122: fix for swapext log items Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 4/7] generic, xfs: test scatter-gather atomic file updates Darrick J. Wong ` (4 subsequent siblings) 6 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Make sure that we clear the suid and sgid bits and capabilities during a FIEXCHANGE_RANGE call just like we would for a regular file write. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/generic/1218 | 115 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/1218.out | 49 ++++++++++++++++++++ tests/generic/1219 | 83 +++++++++++++++++++++++++++++++++++ tests/generic/1219.out | 17 +++++++ 4 files changed, 264 insertions(+) create mode 100755 tests/generic/1218 create mode 100644 tests/generic/1218.out create mode 100755 tests/generic/1219 create mode 100755 tests/generic/1219.out diff --git a/tests/generic/1218 b/tests/generic/1218 new file mode 100755 index 0000000000..e6c170351e --- /dev/null +++ b/tests/generic/1218 @@ -0,0 +1,115 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test 1218 +# +# Functional test for dropping suid and sgid bits as part of an atomic file +# commit. +# +. ./common/preamble +_begin_fstest auto fiexchange swapext quick + +# Override the default cleanup function. +# _cleanup() +# { +# cd / +# rm -r -f $tmp.* +# } + +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs generic +_require_user +_require_xfs_io_command swapext '-v vfs -a' +_require_xfs_io_command startupdate +_require_scratch + +_scratch_mkfs >> $seqres.full +_scratch_mount +_require_congruent_file_oplen $SCRATCH_MNT 1048576 +chmod a+rw $SCRATCH_MNT/ + +setup_testfile() { + rm -f $SCRATCH_MNT/a + _pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full + sync +} + +commit_and_check() { + local user="$1" + + md5sum $SCRATCH_MNT/a | _filter_scratch + stat -c '%a %A %n' $SCRATCH_MNT/a | _filter_scratch + + local cmd="$XFS_IO_PROG -c 'startupdate' -c 'pwrite -S 0x57 0 1m' -c 'commitupdate' $SCRATCH_MNT/a" + if [ -n "$user" ]; then + su - "$user" -c "$cmd" >> $seqres.full + else + $SHELL -c "$cmd" >> $seqres.full + fi + + _scratch_cycle_mount + md5sum $SCRATCH_MNT/a | _filter_scratch + stat -c '%a %A %n' $SCRATCH_MNT/a | _filter_scratch + + # Blank line in output + echo +} + +# Commit to a non-exec file by an unprivileged user clears suid but leaves +# sgid. +echo "Test 1 - qa_user, non-exec file" +setup_testfile +chmod a+rws $SCRATCH_MNT/a +commit_and_check "$qa_user" + +# Commit to a group-exec file by an unprivileged user clears suid and sgid. +echo "Test 2 - qa_user, group-exec file" +setup_testfile +chmod g+x,a+rws $SCRATCH_MNT/a +commit_and_check "$qa_user" + +# Commit to a user-exec file by an unprivileged user clears suid but not sgid. +echo "Test 3 - qa_user, user-exec file" +setup_testfile +chmod u+x,a+rws,g-x $SCRATCH_MNT/a +commit_and_check "$qa_user" + +# Commit to a all-exec file by an unprivileged user clears suid and sgid. +echo "Test 4 - qa_user, all-exec file" +setup_testfile +chmod a+rwxs $SCRATCH_MNT/a +commit_and_check "$qa_user" + +# Commit to a non-exec file by root leaves suid and sgid. +echo "Test 5 - root, non-exec file" +setup_testfile +chmod a+rws $SCRATCH_MNT/a +commit_and_check + +# Commit to a group-exec file by root leaves suid and sgid. +echo "Test 6 - root, group-exec file" +setup_testfile +chmod g+x,a+rws $SCRATCH_MNT/a +commit_and_check + +# Commit to a user-exec file by root leaves suid and sgid. +echo "Test 7 - root, user-exec file" +setup_testfile +chmod u+x,a+rws,g-x $SCRATCH_MNT/a +commit_and_check + +# Commit to a all-exec file by root leaves suid and sgid. +echo "Test 8 - root, all-exec file" +setup_testfile +chmod a+rwxs $SCRATCH_MNT/a +commit_and_check + +# success, all done +status=0 +exit diff --git a/tests/generic/1218.out b/tests/generic/1218.out new file mode 100644 index 0000000000..8f4469aad4 --- /dev/null +++ b/tests/generic/1218.out @@ -0,0 +1,49 @@ +QA output created by 1218 +Test 1 - qa_user, non-exec file +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +6666 -rwSrwSrw- SCRATCH_MNT/a +3784de23efab7a2074c9ec66901e39e5 SCRATCH_MNT/a +666 -rw-rw-rw- SCRATCH_MNT/a + +Test 2 - qa_user, group-exec file +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +6676 -rwSrwsrw- SCRATCH_MNT/a +3784de23efab7a2074c9ec66901e39e5 SCRATCH_MNT/a +676 -rw-rwxrw- SCRATCH_MNT/a + +Test 3 - qa_user, user-exec file +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +6766 -rwsrwSrw- SCRATCH_MNT/a +3784de23efab7a2074c9ec66901e39e5 SCRATCH_MNT/a +766 -rwxrw-rw- SCRATCH_MNT/a + +Test 4 - qa_user, all-exec file +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +6777 -rwsrwsrwx SCRATCH_MNT/a +3784de23efab7a2074c9ec66901e39e5 SCRATCH_MNT/a +777 -rwxrwxrwx SCRATCH_MNT/a + +Test 5 - root, non-exec file +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +6666 -rwSrwSrw- SCRATCH_MNT/a +3784de23efab7a2074c9ec66901e39e5 SCRATCH_MNT/a +6666 -rwSrwSrw- SCRATCH_MNT/a + +Test 6 - root, group-exec file +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +6676 -rwSrwsrw- SCRATCH_MNT/a +3784de23efab7a2074c9ec66901e39e5 SCRATCH_MNT/a +6676 -rwSrwsrw- SCRATCH_MNT/a + +Test 7 - root, user-exec file +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +6766 -rwsrwSrw- SCRATCH_MNT/a +3784de23efab7a2074c9ec66901e39e5 SCRATCH_MNT/a +6766 -rwsrwSrw- SCRATCH_MNT/a + +Test 8 - root, all-exec file +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +6777 -rwsrwsrwx SCRATCH_MNT/a +3784de23efab7a2074c9ec66901e39e5 SCRATCH_MNT/a +6777 -rwsrwsrwx SCRATCH_MNT/a + diff --git a/tests/generic/1219 b/tests/generic/1219 new file mode 100755 index 0000000000..fe20475058 --- /dev/null +++ b/tests/generic/1219 @@ -0,0 +1,83 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test 1219 +# +# Functional test for dropping capability bits as part of an atomic file +# commit. +# +. ./common/preamble +_begin_fstest auto fiexchange swapext quick + +# Override the default cleanup function. +# _cleanup() +# { +# cd / +# rm -r -f $tmp.* +# } + +# Import common functions. +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs generic +_require_user +_require_command "$GETCAP_PROG" getcap +_require_command "$SETCAP_PROG" setcap +_require_xfs_io_command swapext '-v vfs -a' +_require_xfs_io_command startupdate +_require_scratch + +_scratch_mkfs >> $seqres.full +_scratch_mount +_require_congruent_file_oplen $SCRATCH_MNT 1048576 +chmod a+rw $SCRATCH_MNT/ + +setup_testfile() { + rm -f $SCRATCH_MNT/a $SCRATCH_MNT/b + _pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full + _pwrite_byte 0x57 0 1m $SCRATCH_MNT/b >> $seqres.full + chmod a+rw $SCRATCH_MNT/a $SCRATCH_MNT/b + $SETCAP_PROG cap_setgid,cap_setuid+ep $SCRATCH_MNT/a + sync +} + +commit_and_check() { + local user="$1" + + md5sum $SCRATCH_MNT/a | _filter_scratch + stat -c '%a %A %n' $SCRATCH_MNT/a | _filter_scratch + _getcap -v $SCRATCH_MNT/a | _filter_scratch + + local cmd="$XFS_IO_PROG -c 'startupdate' -c 'pwrite -S 0x57 0 1m' -c 'commitupdate' $SCRATCH_MNT/a" + if [ -n "$user" ]; then + su - "$user" -c "$cmd" >> $seqres.full + else + $SHELL -c "$cmd" >> $seqres.full + fi + + _scratch_cycle_mount + md5sum $SCRATCH_MNT/a | _filter_scratch + stat -c '%a %A %n' $SCRATCH_MNT/a | _filter_scratch + _getcap -v $SCRATCH_MNT/a | _filter_scratch + + # Blank line in output + echo +} + +# Commit by an unprivileged user clears capability bits. +echo "Test 1 - qa_user" +setup_testfile +commit_and_check "$qa_user" + +# Commit by root leaves capability bits. +echo "Test 2 - root" +setup_testfile +commit_and_check + +# success, all done +status=0 +exit diff --git a/tests/generic/1219.out b/tests/generic/1219.out new file mode 100755 index 0000000000..a925b4ec4f --- /dev/null +++ b/tests/generic/1219.out @@ -0,0 +1,17 @@ +QA output created by 1219 +Test 1 - qa_user +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +666 -rw-rw-rw- SCRATCH_MNT/a +SCRATCH_MNT/a cap_setgid,cap_setuid=ep +3784de23efab7a2074c9ec66901e39e5 SCRATCH_MNT/a +666 -rw-rw-rw- SCRATCH_MNT/a +SCRATCH_MNT/a + +Test 2 - root +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +666 -rw-rw-rw- SCRATCH_MNT/a +SCRATCH_MNT/a cap_setgid,cap_setuid=ep +3784de23efab7a2074c9ec66901e39e5 SCRATCH_MNT/a +666 -rw-rw-rw- SCRATCH_MNT/a +SCRATCH_MNT/a + ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 4/7] generic, xfs: test scatter-gather atomic file updates 2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/7] xfs/122: fix for swapext log items Darrick J. Wong 2022-12-30 22:19 ` [PATCH 5/7] generic: test that file privilege gets dropped with FIEXCHANGE_RANGE Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 3/7] generic: test new vfs swapext ioctl Darrick J. Wong ` (3 subsequent siblings) 6 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Make sure that FILE_SWAP_RANGE_SKIP_FILE1_HOLES does what we want it to do -- provide a means to implement scatter-gather atomic file writes. That means we create a temporary file, write whatever sparse bits to it that we want, and swap the non-hole parts of the temp file. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 19 +++++++++++++++ tests/generic/1216 | 53 ++++++++++++++++++++++++++++++++++++++++++ tests/generic/1216.out | 6 +++++ tests/generic/1217 | 56 ++++++++++++++++++++++++++++++++++++++++++++ tests/generic/1217.out | 4 +++ tests/xfs/1211 | 59 ++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1211.out | 7 ++++++ tests/xfs/1212 | 61 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1212.out | 5 ++++ 9 files changed, 270 insertions(+) create mode 100755 tests/generic/1216 create mode 100644 tests/generic/1216.out create mode 100755 tests/generic/1217 create mode 100644 tests/generic/1217.out create mode 100755 tests/xfs/1211 create mode 100644 tests/xfs/1211.out create mode 100755 tests/xfs/1212 create mode 100644 tests/xfs/1212.out diff --git a/common/xfs b/common/xfs index 610730e5ef..8b365ad18b 100644 --- a/common/xfs +++ b/common/xfs @@ -1723,3 +1723,22 @@ _scratch_xfs_find_agbtree_height() { return 1 } + +_require_xfs_mkfs_atomicswap() +{ + # atomicswap can be activated on rmap or reflink filesystems. + # reflink is newer (4.9 for reflink vs. 4.8 for rmap) so test that. + _scratch_mkfs_xfs_supported -m reflink=1 >/dev/null 2>&1 || \ + _notrun "mkfs.xfs doesn't have atomicswap dependent features" +} + +_require_xfs_scratch_atomicswap() +{ + _require_xfs_mkfs_atomicswap + _require_scratch + _require_xfs_io_command swapext '-v vfs -a' + _scratch_mkfs -m reflink=1 > /dev/null + _try_scratch_mount || \ + _notrun "atomicswap dependencies not supported by scratch filesystem type: $FSTYP" + _scratch_unmount +} diff --git a/tests/generic/1216 b/tests/generic/1216 new file mode 100755 index 0000000000..802cfe14d4 --- /dev/null +++ b/tests/generic/1216 @@ -0,0 +1,53 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1216 +# +# Test scatter-gather atomic file writes. We create a temporary file, write +# sparsely to it, then use FILE_SWAP_RANGE_SKIP_FILE1_HOLES flag to swap +# atomicallly only the ranges that we wrote. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs -a' +_require_scratch + +_scratch_mkfs >> $seqres.full +_scratch_mount +_require_congruent_file_oplen $SCRATCH_MNT 65536 + +# Create original file +_pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full + +# Create the donor file +_pwrite_byte 0x59 64k 64k $SCRATCH_MNT/b >> $seqres.full +_pwrite_byte 0x57 768k 64k $SCRATCH_MNT/b >> $seqres.full +$XFS_IO_PROG -c 'truncate 1m' $SCRATCH_MNT/b + +md5sum $SCRATCH_MNT/a | _filter_scratch +md5sum $SCRATCH_MNT/b | _filter_scratch + +# Test swapext. -h means skip holes in /b, and -e means operate to EOF +echo swap | tee -a $seqres.full +$XFS_IO_PROG -c "swapext -v vfs -f -u -h -e -a $SCRATCH_MNT/b" $SCRATCH_MNT/a +_scratch_cycle_mount + +md5sum $SCRATCH_MNT/a | _filter_scratch +md5sum $SCRATCH_MNT/b | _filter_scratch + +# success, all done +status=0 +exit diff --git a/tests/generic/1216.out b/tests/generic/1216.out new file mode 100644 index 0000000000..2b49d5cfa8 --- /dev/null +++ b/tests/generic/1216.out @@ -0,0 +1,6 @@ +QA output created by 1216 +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +c9fb827e2e3e579dc2a733ddad486d1d SCRATCH_MNT/b +swap +e9cbfe8489a68efaa5fcf40cf3106118 SCRATCH_MNT/a +faf8ed02a5b0638096a817abcc6c2127 SCRATCH_MNT/b diff --git a/tests/generic/1217 b/tests/generic/1217 new file mode 100755 index 0000000000..78c49751f7 --- /dev/null +++ b/tests/generic/1217 @@ -0,0 +1,56 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1217 +# +# Test scatter-gather atomic file commits. Use the startupdate command to +# create a temporary file, write sparsely to it, then commitupdate -h to +# perform the scattered update. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs -a' +_require_xfs_io_command startupdate '-e' +_require_scratch + +_scratch_mkfs >> $seqres.full +_scratch_mount +_require_congruent_file_oplen $SCRATCH_MNT 65536 + +# Create original file +_pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full +sync +md5sum $SCRATCH_MNT/a | _filter_scratch + +# Test atomic scatter-gather file commits. +echo commit | tee -a $seqres.full +$XFS_IO_PROG \ + -c 'bmap -elpv' \ + -c 'startupdate -e' \ + -c 'truncate 1m' \ + -c 'pwrite -S 0x59 64k 64k' \ + -c 'pwrite -S 0x57 768k 64k' \ + -c 'bmap -elpv' \ + -c 'commitupdate -h -k' \ + -c 'bmap -elpv' \ + $SCRATCH_MNT/a >> $seqres.full +_scratch_cycle_mount + +md5sum $SCRATCH_MNT/a | _filter_scratch + +# success, all done +status=0 +exit diff --git a/tests/generic/1217.out b/tests/generic/1217.out new file mode 100644 index 0000000000..73bf89895d --- /dev/null +++ b/tests/generic/1217.out @@ -0,0 +1,4 @@ +QA output created by 1217 +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +commit +e9cbfe8489a68efaa5fcf40cf3106118 SCRATCH_MNT/a diff --git a/tests/xfs/1211 b/tests/xfs/1211 new file mode 100755 index 0000000000..6297b8ad67 --- /dev/null +++ b/tests/xfs/1211 @@ -0,0 +1,59 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1211 +# +# Test scatter-gather atomic file writes. We create a temporary file, write +# sparsely to it, then use FILE_SWAP_RANGE_SKIP_FILE1_HOLES flag to swap +# atomicallly only the ranges that we wrote. Inject an error so that we can +# test that log recovery finishes the swap. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/inject + +# real QA test starts here +_supported_fs xfs +_require_xfs_io_command swapext '-v vfs -a' +_require_xfs_scratch_atomicswap +_require_xfs_io_error_injection "bmap_finish_one" + +_scratch_mkfs >> $seqres.full +_scratch_mount +_require_congruent_file_oplen $SCRATCH_MNT 65536 + +# Create original file +_pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full + +# Create the donor file +_pwrite_byte 0x59 64k 64k $SCRATCH_MNT/b >> $seqres.full +_pwrite_byte 0x57 768k 64k $SCRATCH_MNT/b >> $seqres.full +$XFS_IO_PROG -c 'truncate 1m' $SCRATCH_MNT/b +sync + +md5sum $SCRATCH_MNT/a | _filter_scratch +md5sum $SCRATCH_MNT/b | _filter_scratch + +# Test swapext. -h means skip holes in /b, and -e means operate to EOF +echo swap | tee -a $seqres.full +$XFS_IO_PROG -x -c 'inject bmap_finish_one' \ + -c "swapext -v vfs -f -u -h -e -a $SCRATCH_MNT/b" $SCRATCH_MNT/a +_scratch_cycle_mount + +md5sum $SCRATCH_MNT/a | _filter_scratch +md5sum $SCRATCH_MNT/b | _filter_scratch + +# success, all done +status=0 +exit diff --git a/tests/xfs/1211.out b/tests/xfs/1211.out new file mode 100644 index 0000000000..19412c21a1 --- /dev/null +++ b/tests/xfs/1211.out @@ -0,0 +1,7 @@ +QA output created by 1211 +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +c9fb827e2e3e579dc2a733ddad486d1d SCRATCH_MNT/b +swap +swapext: Input/output error +e9cbfe8489a68efaa5fcf40cf3106118 SCRATCH_MNT/a +faf8ed02a5b0638096a817abcc6c2127 SCRATCH_MNT/b diff --git a/tests/xfs/1212 b/tests/xfs/1212 new file mode 100755 index 0000000000..d2292d65a2 --- /dev/null +++ b/tests/xfs/1212 @@ -0,0 +1,61 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1212 +# +# Test scatter-gather atomic file commits. Use the startupdate command to +# create a temporary file, write sparsely to it, then commitupdate -h to +# perform the scattered update. Inject an error so that we can test that log +# recovery finishes the swap. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/inject + +# real QA test starts here +_supported_fs xfs +_require_xfs_io_command swapext '-v vfs -a' +_require_xfs_io_command startupdate '-e' +_require_xfs_scratch_atomicswap +_require_xfs_io_error_injection "bmap_finish_one" + +_scratch_mkfs >> $seqres.full +_scratch_mount + +# Create original file +_pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full +sync +md5sum $SCRATCH_MNT/a | _filter_scratch + +# Test atomic scatter-gather file commits. +echo commit | tee -a $seqres.full +$XFS_IO_PROG -x \ + -c 'bmap -elpv' \ + -c 'startupdate -e' \ + -c 'truncate 1m' \ + -c 'pwrite -S 0x59 64k 64k' \ + -c 'pwrite -S 0x57 768k 64k' \ + -c 'sync' \ + -c 'bmap -elpv' \ + -c 'inject bmap_finish_one' \ + -c 'commitupdate -h -k' \ + $SCRATCH_MNT/a >> $seqres.full +_scratch_cycle_mount + +$XFS_IO_PROG -c 'bmap -elpv' $SCRATCH_MNT/a >> $seqres.full +md5sum $SCRATCH_MNT/a | _filter_scratch + +# success, all done +status=0 +exit diff --git a/tests/xfs/1212.out b/tests/xfs/1212.out new file mode 100644 index 0000000000..29718e2024 --- /dev/null +++ b/tests/xfs/1212.out @@ -0,0 +1,5 @@ +QA output created by 1212 +310f146ce52077fcd3308dcbe7632bb2 SCRATCH_MNT/a +commit +committing update: Input/output error +e9cbfe8489a68efaa5fcf40cf3106118 SCRATCH_MNT/a ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 3/7] generic: test new vfs swapext ioctl 2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 4/7] generic, xfs: test scatter-gather atomic file updates Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/7] generic: test old xfs extent swapping ioctl Darrick J. Wong ` (2 subsequent siblings) 6 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Test the new vfs swapext ioctl. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/rc | 13 +++++ common/reflink | 33 +++++++++++++ tests/generic/1203 | 58 ++++++++++++++++++++++ tests/generic/1203.out | 2 + tests/generic/1204 | 100 ++++++++++++++++++++++++++++++++++++++ tests/generic/1204.out | 86 +++++++++++++++++++++++++++++++++ tests/generic/1205 | 116 ++++++++++++++++++++++++++++++++++++++++++++ tests/generic/1205.out | 90 ++++++++++++++++++++++++++++++++++ tests/generic/1206 | 76 +++++++++++++++++++++++++++++ tests/generic/1206.out | 32 ++++++++++++ tests/generic/1207 | 122 ++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/1207.out | 48 ++++++++++++++++++ tests/generic/1209 | 101 ++++++++++++++++++++++++++++++++++++++ tests/generic/1209.out | 33 +++++++++++++ tests/generic/1210 | 48 ++++++++++++++++++ tests/generic/1210.out | 6 ++ tests/generic/1211 | 105 ++++++++++++++++++++++++++++++++++++++++ tests/generic/1211.out | 40 +++++++++++++++ tests/generic/1212 | 58 ++++++++++++++++++++++ tests/generic/1212.out | 2 + tests/generic/1213 | 126 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/1213.out | 48 ++++++++++++++++++ tests/generic/1214 | 63 ++++++++++++++++++++++++ tests/generic/1214.out | 2 + tests/generic/1215 | 70 +++++++++++++++++++++++++++ tests/generic/1215.out | 2 + tests/xfs/1208 | 62 ++++++++++++++++++++++++ tests/xfs/1208.out | 10 ++++ 28 files changed, 1552 insertions(+) create mode 100755 tests/generic/1203 create mode 100644 tests/generic/1203.out create mode 100755 tests/generic/1204 create mode 100644 tests/generic/1204.out create mode 100755 tests/generic/1205 create mode 100644 tests/generic/1205.out create mode 100755 tests/generic/1206 create mode 100644 tests/generic/1206.out create mode 100755 tests/generic/1207 create mode 100644 tests/generic/1207.out create mode 100755 tests/generic/1209 create mode 100644 tests/generic/1209.out create mode 100755 tests/generic/1210 create mode 100644 tests/generic/1210.out create mode 100755 tests/generic/1211 create mode 100644 tests/generic/1211.out create mode 100755 tests/generic/1212 create mode 100644 tests/generic/1212.out create mode 100755 tests/generic/1213 create mode 100644 tests/generic/1213.out create mode 100755 tests/generic/1214 create mode 100644 tests/generic/1214.out create mode 100755 tests/generic/1215 create mode 100644 tests/generic/1215.out create mode 100755 tests/xfs/1208 create mode 100644 tests/xfs/1208.out diff --git a/common/rc b/common/rc index 88ecc1837d..cfe765de2e 100644 --- a/common/rc +++ b/common/rc @@ -2575,6 +2575,17 @@ _require_xfs_io_command() echo $testio | grep -q "Inappropriate ioctl" && \ _notrun "xfs_io $command support is missing" ;; + "startupdate"|"commitupdate"|"cancelupdate") + $XFS_IO_PROG -f -c 'pwrite -S 0x58 0 128k -b 128k' $testfile > /dev/null + testio=$($XFS_IO_PROG -c "startupdate $param" \ + -c 'pwrite -S 0x59 0 192k' \ + -c 'commitupdate' $testfile 2>&1) + echo $testio | grep -q "Inappropriate ioctl" && \ + _notrun "xfs_io $command $param support is missing" + echo $testio | grep -q "Operation not supported" && \ + _notrun "xfs_io $command $param kernel support is missing" + param_checked="$param" + ;; "swapext") $XFS_IO_PROG -f -c 'pwrite -S 0x58 0 128k -b 128k' $testfile > /dev/null $XFS_IO_PROG -f -c 'truncate 128k' $testfile.1 > /dev/null @@ -2583,6 +2594,8 @@ _require_xfs_io_command() _notrun "xfs_io $command $param support is missing" echo $testio | grep -q "Inappropriate ioctl" && \ _notrun "xfs_io $command $param ioctl support is missing" + echo $testio | grep -q "Operation not supported" && \ + _notrun "xfs_io $command $param kernel support is missing" rm -f $testfile.1 param_checked="$param" ;; diff --git a/common/reflink b/common/reflink index 76e9cb7d32..22adc4449b 100644 --- a/common/reflink +++ b/common/reflink @@ -325,6 +325,39 @@ _weave_reflink_regular() { done } +# Create a file of interleaved holes, unwritten blocks, and regular blocks. +_weave_file_rainbow() { + blksz=$1 + nr=$2 + dfile=$3 + + $XFS_IO_PROG -f -c "truncate $((blksz * nr))" $dfile + _pwrite_byte 0x00 0 $((blksz * nr)) $dfile.chk + # 0 blocks are unwritten + seq 1 5 $((nr - 1)) | while read i; do + $XFS_IO_PROG -f -c "falloc $((blksz * i)) $blksz" $dfile + _pwrite_byte 0x00 $((blksz * i)) $blksz $dfile.chk + done + # 1 blocks are holes + seq 2 5 $((nr - 1)) | while read i; do + _pwrite_byte 0x00 $((blksz * i)) $blksz $dfile.chk + done + # 2 blocks are regular + seq 3 5 $((nr - 1)) | while read i; do + _pwrite_byte 0x71 $((blksz * i)) $blksz $dfile + _pwrite_byte 0x71 $((blksz * i)) $blksz $dfile.chk + done + # 3 blocks are holes + seq 2 5 $((nr - 1)) | while read i; do + _pwrite_byte 0x00 $((blksz * i)) $blksz $dfile.chk + done + # 4 blocks are delalloc + seq 4 5 $((nr - 1)) | while read i; do + _pwrite_byte 0x62 $((blksz * i)) $blksz $dfile + _pwrite_byte 0x62 $((blksz * i)) $blksz $dfile.chk + done +} + # Create a file of interleaved holes, unwritten blocks, regular blocks, and # reflinked blocks _weave_reflink_rainbow() { diff --git a/tests/generic/1203 b/tests/generic/1203 new file mode 100755 index 0000000000..890b0b4c86 --- /dev/null +++ b/tests/generic/1203 @@ -0,0 +1,58 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1202 +# +# Make sure that swapext modifies ctime and not mtime of the file. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter + +# real QA test starts here +_require_test_program punch-alternating +_require_xfs_io_command swapext '-v vfs' +_require_test + +dir=$TEST_DIR/test-$seq +mkdir -p $dir + +# Set up initial files +$XFS_IO_PROG -f -c 'pwrite -S 0x58 0 256k -b 1m' $dir/a >> $seqres.full +$here/src/punch-alternating $dir/a +$XFS_IO_PROG -f -c 'pwrite -S 0x59 0 256k -b 1m' $dir/b >> $seqres.full + +# Snapshot the 'a' file before we swap +echo before >> $seqres.full +md5sum $dir/a $dir/b >> $seqres.full +old_mtime="$(echo $(stat -c '%y' $dir/a $dir/b))" +old_ctime="$(echo $(stat -c '%z' $dir/a $dir/b))" +stat -c '%y %Y %z %Z' $dir/a $dir/b >> $seqres.full + +# Now try to swapext +$XFS_IO_PROG -c "swapext $dir/b" $dir/a + +# Snapshot the 'a' file after we swap +echo after >> $seqres.full +md5sum $dir/a $dir/b >> $seqres.full +new_mtime="$(echo $(stat -c '%y' $dir/a $dir/b))" +new_ctime="$(echo $(stat -c '%z' $dir/a $dir/b))" +stat -c '%y %Y %z %Z' $dir/a $dir/b >> $seqres.full + +test "$new_mtime" = "$old_mtime" && echo "mtime: $new_mtime == $old_mtime" +test "$new_ctime" = "$old_ctime" && echo "ctime: $new_ctime == $old_ctime" + +# success, all done +echo Silence is golden. +status=0 +exit diff --git a/tests/generic/1203.out b/tests/generic/1203.out new file mode 100644 index 0000000000..904b25beb4 --- /dev/null +++ b/tests/generic/1203.out @@ -0,0 +1,2 @@ +QA output created by 1203 +Silence is golden. diff --git a/tests/generic/1204 b/tests/generic/1204 new file mode 100755 index 0000000000..5db3878520 --- /dev/null +++ b/tests/generic/1204 @@ -0,0 +1,100 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1204 +# +# Test swapext between ranges of two different files. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/reflink + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs -s 64k -l 64k' +_require_xfs_io_command "falloc" +_require_test + +filesnap() { + echo "$1" + if [ "$2" != "$3" ]; then + md5sum $2 $3 | _filter_test_dir + else + md5sum $2 | _filter_test_dir + fi +} + +test_swapext_once() { + filesnap "$1: before swapext" $dir/$3 $dir/$4 + $XFS_IO_PROG -c "swapext -v vfs $2 $dir/$3" $dir/$4 + filesnap "$1: after swapext" $dir/$3 $dir/$4 + _test_cycle_mount + filesnap "$1: after cycling mount" $dir/$3 $dir/$4 + echo +} + +test_swapext_two() { + # swapext the same range of two files + test_swapext_once "$@: samerange" \ + "-s $((blksz * 3)) -d $((blksz * 3)) -l $((blksz * 5))" b a + + # swapext different ranges of two files + test_swapext_once "$@: diffrange" \ + "-s $((blksz * 37)) -d $((blksz * 47)) -l $((blksz * 7))" b a + + # swapext overlapping ranges of two files + test_swapext_once "$@: overlap" \ + "-s $((blksz * 17)) -d $((blksz * 23)) -l $((blksz * 7))" b a +} + +dir=$TEST_DIR/test-$seq +mkdir -p $dir +blksz=65536 +nrblks=57 +_require_congruent_file_oplen $TEST_DIR $blksz + +# Set up some simple files for a first test. +rm -rf $dir/a $dir/b +_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full +_pwrite_byte 0x59 0 $((blksz * nrblks)) $dir/b >> $seqres.full +test_swapext_two "simple" + +# Make some files that don't end an aligned offset. +rm -rf $dir/a $dir/b +_pwrite_byte 0x58 0 $(( (blksz * nrblks) + 37)) $dir/a >> $seqres.full +_pwrite_byte 0x59 0 $(( (blksz * nrblks) + 37)) $dir/b >> $seqres.full +test_swapext_once "unalignedeof" "" a b + +# Set up some crazy rainbow files +rm -rf $dir/a $dir/b +_weave_file_rainbow $blksz $nrblks $dir/a >> $seqres.full +_weave_file_rainbow $blksz $nrblks $dir/b >> $seqres.full +test_swapext_two "rainbow" + +# Now set up a simple file for testing within the same file +rm -rf $dir/c +$XFS_IO_PROG -f -c "pwrite -S 0x58 0 $((blksz * nrblks))" \ + -c "pwrite -S 0x59 $((blksz * nrblks)) $((blksz * nrblks))" \ + $dir/c >> $seqres.full + +# swapext the same offset into the 'X' and 'Y' regions of the file +test_swapext_once "single: sameXandY" \ + "-s $((blksz * 3)) -d $((blksz * (nrblks + 3))) -l $((blksz * 5))" c c + +# swapext the same offset into the 'X' and 'Y' regions of the file +test_swapext_once "single: overlap" \ + "-s $((blksz * 13)) -d $((blksz * 17)) -l $((blksz * 5))" c c + +# success, all done +status=0 +exit diff --git a/tests/generic/1204.out b/tests/generic/1204.out new file mode 100644 index 0000000000..cdf9f7315d --- /dev/null +++ b/tests/generic/1204.out @@ -0,0 +1,86 @@ +QA output created by 1204 +simple: samerange: before swapext +db85d578204631f2b4eb1e73974253c2 TEST_DIR/test-1204/b +d0425612f15c6071022cf7127620f63d TEST_DIR/test-1204/a +simple: samerange: after swapext +20beef1c9ed7de02e4229c69bd43bd8f TEST_DIR/test-1204/b +e7697fa99d08f7eb76fa3fb963fe916a TEST_DIR/test-1204/a +simple: samerange: after cycling mount +20beef1c9ed7de02e4229c69bd43bd8f TEST_DIR/test-1204/b +e7697fa99d08f7eb76fa3fb963fe916a TEST_DIR/test-1204/a + +simple: diffrange: before swapext +20beef1c9ed7de02e4229c69bd43bd8f TEST_DIR/test-1204/b +e7697fa99d08f7eb76fa3fb963fe916a TEST_DIR/test-1204/a +simple: diffrange: after swapext +cd32ce54c295fcdf571ce7f8220fac56 TEST_DIR/test-1204/b +d9771c5bb6d9db00b9abe65a4410e1a6 TEST_DIR/test-1204/a +simple: diffrange: after cycling mount +cd32ce54c295fcdf571ce7f8220fac56 TEST_DIR/test-1204/b +d9771c5bb6d9db00b9abe65a4410e1a6 TEST_DIR/test-1204/a + +simple: overlap: before swapext +cd32ce54c295fcdf571ce7f8220fac56 TEST_DIR/test-1204/b +d9771c5bb6d9db00b9abe65a4410e1a6 TEST_DIR/test-1204/a +simple: overlap: after swapext +e0fff655f6a08fc2f03ee01e4767060c TEST_DIR/test-1204/b +ec7d764c85e583e305028c9cba5b25b6 TEST_DIR/test-1204/a +simple: overlap: after cycling mount +e0fff655f6a08fc2f03ee01e4767060c TEST_DIR/test-1204/b +ec7d764c85e583e305028c9cba5b25b6 TEST_DIR/test-1204/a + +unalignedeof: before swapext +9f8c731a4f1946ffdda8c33e82417f2d TEST_DIR/test-1204/a +7a5d2ba7508226751c835292e28cd227 TEST_DIR/test-1204/b +unalignedeof: after swapext +7a5d2ba7508226751c835292e28cd227 TEST_DIR/test-1204/a +9f8c731a4f1946ffdda8c33e82417f2d TEST_DIR/test-1204/b +unalignedeof: after cycling mount +7a5d2ba7508226751c835292e28cd227 TEST_DIR/test-1204/a +9f8c731a4f1946ffdda8c33e82417f2d TEST_DIR/test-1204/b + +rainbow: samerange: before swapext +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/a +rainbow: samerange: after swapext +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/a +rainbow: samerange: after cycling mount +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/a + +rainbow: diffrange: before swapext +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/a +rainbow: diffrange: after swapext +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/a +rainbow: diffrange: after cycling mount +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/a + +rainbow: overlap: before swapext +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1204/a +rainbow: overlap: after swapext +6753bc585e3c71d53bfaae11d2ffee99 TEST_DIR/test-1204/b +39597abd4d9d0c9ceac22b77eb00c373 TEST_DIR/test-1204/a +rainbow: overlap: after cycling mount +6753bc585e3c71d53bfaae11d2ffee99 TEST_DIR/test-1204/b +39597abd4d9d0c9ceac22b77eb00c373 TEST_DIR/test-1204/a + +single: sameXandY: before swapext +39e17753fa9e923a3b5928e13775e358 TEST_DIR/test-1204/c +single: sameXandY: after swapext +8262c617070703fb0e2a28d8f05e3112 TEST_DIR/test-1204/c +single: sameXandY: after cycling mount +8262c617070703fb0e2a28d8f05e3112 TEST_DIR/test-1204/c + +single: overlap: before swapext +8262c617070703fb0e2a28d8f05e3112 TEST_DIR/test-1204/c +swapext: Invalid argument +single: overlap: after swapext +8262c617070703fb0e2a28d8f05e3112 TEST_DIR/test-1204/c +single: overlap: after cycling mount +8262c617070703fb0e2a28d8f05e3112 TEST_DIR/test-1204/c + diff --git a/tests/generic/1205 b/tests/generic/1205 new file mode 100755 index 0000000000..dfa5adf0e1 --- /dev/null +++ b/tests/generic/1205 @@ -0,0 +1,116 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1205 +# +# Test swapext between ranges of two different files, when one of the files +# is shared. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/reflink + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs' +_require_xfs_io_command "falloc" +_require_test_reflink + +filesnap() { + echo "$1" + if [ "$2" != "$3" ]; then + md5sum $2 $3 | _filter_test_dir + else + md5sum $2 | _filter_test_dir + fi +} + +test_swapext_once() { + filesnap "$1: before swapext" $dir/$3 $dir/$4 + $XFS_IO_PROG -c "swapext -v vfs $2 $dir/$3" $dir/$4 + filesnap "$1: after swapext" $dir/$3 $dir/$4 + _test_cycle_mount + filesnap "$1: after cycling mount" $dir/$3 $dir/$4 + echo +} + +test_swapext_two() { + # swapext the same range of two files + test_swapext_once "$@: samerange" \ + "-s $((blksz * 3)) -d $((blksz * 3)) -l $((blksz * 5))" b a + + # swapext different ranges of two files + test_swapext_once "$@: diffrange" \ + "-s $((blksz * 37)) -d $((blksz * 47)) -l $((blksz * 7))" b a + + # swapext overlapping ranges of two files + test_swapext_once "$@: overlap" \ + "-s $((blksz * 17)) -d $((blksz * 23)) -l $((blksz * 7))" b a + + # Now let's overwrite a entirely to make sure COW works + echo "overwrite A and B entirely" + md5sum $dir/sharea | _filter_test_dir + $XFS_IO_PROG -c "pwrite -S 0x60 0 $((blksz * nrblks))" $dir/a >> $seqres.full + $XFS_IO_PROG -c "pwrite -S 0x60 0 $((blksz * nrblks))" $dir/b >> $seqres.full + md5sum $dir/sharea | _filter_test_dir + _test_cycle_mount + md5sum $dir/sharea | _filter_test_dir + echo +} + +dir=$TEST_DIR/test-$seq +mkdir -p $dir +blksz=65536 +_require_congruent_file_oplen $TEST_DIR $blksz +nrblks=57 + +# Set up some simple files for a first test. +rm -f $dir/a $dir/b $dir/sharea +_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full +_pwrite_byte 0x59 0 $((blksz * nrblks)) $dir/b >> $seqres.full +_cp_reflink $dir/a $dir/sharea +test_swapext_two "simple" + +# Set up some crazy rainbow files +rm -f $dir/a $dir/b $dir/sharea +_weave_file_rainbow $blksz $nrblks $dir/a >> $seqres.full +_weave_file_rainbow $blksz $nrblks $dir/b >> $seqres.full +_cp_reflink $dir/a $dir/sharea +test_swapext_two "rainbow" + +# Now set up a simple file for testing within the same file +rm -f $dir/c $dir/sharec +$XFS_IO_PROG -f -c "pwrite -S 0x58 0 $((blksz * nrblks))" \ + -c "pwrite -S 0x59 $((blksz * nrblks)) $((blksz * nrblks))" \ + $dir/c >> $seqres.full +_cp_reflink $dir/c $dir/sharec + +# swapext the same offset into the 'X' and 'Y' regions of the file +test_swapext_once "single: sameXandY" \ + "-s $((blksz * 3)) -d $((blksz * (nrblks + 3))) -l $((blksz * 5))" c c + +# swapext the same offset into the 'X' and 'Y' regions of the file +test_swapext_once "single: overlap" \ + "-s $((blksz * 13)) -d $((blksz * 17)) -l $((blksz * 5))" c c + +# Now let's overwrite a entirely to make sure COW works +echo "overwrite C entirely" +md5sum $dir/sharec | _filter_test_dir +$XFS_IO_PROG -c "pwrite -S 0x60 0 $((blksz * nrblks))" $dir/c >> $seqres.full +md5sum $dir/sharec | _filter_test_dir +_test_cycle_mount +md5sum $dir/sharec | _filter_test_dir + +# success, all done +status=0 +exit diff --git a/tests/generic/1205.out b/tests/generic/1205.out new file mode 100644 index 0000000000..9a95742e78 --- /dev/null +++ b/tests/generic/1205.out @@ -0,0 +1,90 @@ +QA output created by 1205 +simple: samerange: before swapext +db85d578204631f2b4eb1e73974253c2 TEST_DIR/test-1205/b +d0425612f15c6071022cf7127620f63d TEST_DIR/test-1205/a +simple: samerange: after swapext +20beef1c9ed7de02e4229c69bd43bd8f TEST_DIR/test-1205/b +e7697fa99d08f7eb76fa3fb963fe916a TEST_DIR/test-1205/a +simple: samerange: after cycling mount +20beef1c9ed7de02e4229c69bd43bd8f TEST_DIR/test-1205/b +e7697fa99d08f7eb76fa3fb963fe916a TEST_DIR/test-1205/a + +simple: diffrange: before swapext +20beef1c9ed7de02e4229c69bd43bd8f TEST_DIR/test-1205/b +e7697fa99d08f7eb76fa3fb963fe916a TEST_DIR/test-1205/a +simple: diffrange: after swapext +cd32ce54c295fcdf571ce7f8220fac56 TEST_DIR/test-1205/b +d9771c5bb6d9db00b9abe65a4410e1a6 TEST_DIR/test-1205/a +simple: diffrange: after cycling mount +cd32ce54c295fcdf571ce7f8220fac56 TEST_DIR/test-1205/b +d9771c5bb6d9db00b9abe65a4410e1a6 TEST_DIR/test-1205/a + +simple: overlap: before swapext +cd32ce54c295fcdf571ce7f8220fac56 TEST_DIR/test-1205/b +d9771c5bb6d9db00b9abe65a4410e1a6 TEST_DIR/test-1205/a +simple: overlap: after swapext +e0fff655f6a08fc2f03ee01e4767060c TEST_DIR/test-1205/b +ec7d764c85e583e305028c9cba5b25b6 TEST_DIR/test-1205/a +simple: overlap: after cycling mount +e0fff655f6a08fc2f03ee01e4767060c TEST_DIR/test-1205/b +ec7d764c85e583e305028c9cba5b25b6 TEST_DIR/test-1205/a + +overwrite A and B entirely +d0425612f15c6071022cf7127620f63d TEST_DIR/test-1205/sharea +d0425612f15c6071022cf7127620f63d TEST_DIR/test-1205/sharea +d0425612f15c6071022cf7127620f63d TEST_DIR/test-1205/sharea + +rainbow: samerange: before swapext +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/a +rainbow: samerange: after swapext +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/a +rainbow: samerange: after cycling mount +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/a + +rainbow: diffrange: before swapext +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/a +rainbow: diffrange: after swapext +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/a +rainbow: diffrange: after cycling mount +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/a + +rainbow: overlap: before swapext +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/b +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/a +rainbow: overlap: after swapext +6753bc585e3c71d53bfaae11d2ffee99 TEST_DIR/test-1205/b +39597abd4d9d0c9ceac22b77eb00c373 TEST_DIR/test-1205/a +rainbow: overlap: after cycling mount +6753bc585e3c71d53bfaae11d2ffee99 TEST_DIR/test-1205/b +39597abd4d9d0c9ceac22b77eb00c373 TEST_DIR/test-1205/a + +overwrite A and B entirely +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/sharea +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/sharea +48b41ee1970eb71064b77181f42634cf TEST_DIR/test-1205/sharea + +single: sameXandY: before swapext +39e17753fa9e923a3b5928e13775e358 TEST_DIR/test-1205/c +single: sameXandY: after swapext +8262c617070703fb0e2a28d8f05e3112 TEST_DIR/test-1205/c +single: sameXandY: after cycling mount +8262c617070703fb0e2a28d8f05e3112 TEST_DIR/test-1205/c + +single: overlap: before swapext +8262c617070703fb0e2a28d8f05e3112 TEST_DIR/test-1205/c +swapext: Invalid argument +single: overlap: after swapext +8262c617070703fb0e2a28d8f05e3112 TEST_DIR/test-1205/c +single: overlap: after cycling mount +8262c617070703fb0e2a28d8f05e3112 TEST_DIR/test-1205/c + +overwrite C entirely +39e17753fa9e923a3b5928e13775e358 TEST_DIR/test-1205/sharec +39e17753fa9e923a3b5928e13775e358 TEST_DIR/test-1205/sharec +39e17753fa9e923a3b5928e13775e358 TEST_DIR/test-1205/sharec diff --git a/tests/generic/1206 b/tests/generic/1206 new file mode 100755 index 0000000000..a4d9dd083e --- /dev/null +++ b/tests/generic/1206 @@ -0,0 +1,76 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1206 +# +# Test swapext between two files of unlike size. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/reflink + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs -s 64k -l 64k' +_require_test + +filesnap() { + echo "$1" + if [ "$2" != "$3" ]; then + md5sum $2 $3 | _filter_test_dir + else + md5sum $2 | _filter_test_dir + fi +} + +test_swapext_once() { + local tag=$1 + local a_len=$2 + local b_len=$3 + local a_off=$4 + local b_off=$5 + local len=$6 + + # len is either a block count or -e to swap to EOF + if [ "$len" != "-e" ]; then + len="-l $((blksz * len))" + fi + + rm -f $dir/a $dir/b + _pwrite_byte 0x58 0 $((blksz * a_len)) $dir/a >> $seqres.full + _pwrite_byte 0x59 0 $((blksz * b_len)) $dir/b >> $seqres.full + filesnap "$tag: before swapext" $dir/a $dir/b + + cmd="swapext -v vfs -s $((blksz * a_off)) -d $((blksz * b_off)) $len $dir/a" + echo "$cmd" >> $seqres.full + $XFS_IO_PROG -c "$cmd" $dir/b + filesnap "$tag: after swapext" $dir/a $dir/b + + _test_cycle_mount + filesnap "$tag: after cycling mount" $dir/a $dir/b + echo +} + +dir=$TEST_DIR/test-$seq +mkdir -p $dir +blksz=65536 + +test_swapext_once "last 5 blocks" 27 37 22 32 5 + +test_swapext_once "whole file to eof" 27 37 0 0 -e + +test_swapext_once "blocks 30-40" 27 37 30 30 10 + +# success, all done +status=0 +exit diff --git a/tests/generic/1206.out b/tests/generic/1206.out new file mode 100644 index 0000000000..82f4185684 --- /dev/null +++ b/tests/generic/1206.out @@ -0,0 +1,32 @@ +QA output created by 1206 +last 5 blocks: before swapext +207ea56e0ccbf50d38fd3a2d842aa170 TEST_DIR/test-1206/a +eb58941d31f5be1e4e22df8c536dd490 TEST_DIR/test-1206/b +last 5 blocks: after swapext +3f34470fe9feb8513d5f3a8538f2c5f3 TEST_DIR/test-1206/a +c3daca7dd9218371cd0dc64f11e4b0bf TEST_DIR/test-1206/b +last 5 blocks: after cycling mount +3f34470fe9feb8513d5f3a8538f2c5f3 TEST_DIR/test-1206/a +c3daca7dd9218371cd0dc64f11e4b0bf TEST_DIR/test-1206/b + +whole file to eof: before swapext +207ea56e0ccbf50d38fd3a2d842aa170 TEST_DIR/test-1206/a +eb58941d31f5be1e4e22df8c536dd490 TEST_DIR/test-1206/b +whole file to eof: after swapext +eb58941d31f5be1e4e22df8c536dd490 TEST_DIR/test-1206/a +207ea56e0ccbf50d38fd3a2d842aa170 TEST_DIR/test-1206/b +whole file to eof: after cycling mount +eb58941d31f5be1e4e22df8c536dd490 TEST_DIR/test-1206/a +207ea56e0ccbf50d38fd3a2d842aa170 TEST_DIR/test-1206/b + +blocks 30-40: before swapext +207ea56e0ccbf50d38fd3a2d842aa170 TEST_DIR/test-1206/a +eb58941d31f5be1e4e22df8c536dd490 TEST_DIR/test-1206/b +swapext: Invalid argument +blocks 30-40: after swapext +207ea56e0ccbf50d38fd3a2d842aa170 TEST_DIR/test-1206/a +eb58941d31f5be1e4e22df8c536dd490 TEST_DIR/test-1206/b +blocks 30-40: after cycling mount +207ea56e0ccbf50d38fd3a2d842aa170 TEST_DIR/test-1206/a +eb58941d31f5be1e4e22df8c536dd490 TEST_DIR/test-1206/b + diff --git a/tests/generic/1207 b/tests/generic/1207 new file mode 100755 index 0000000000..3a69a1ec57 --- /dev/null +++ b/tests/generic/1207 @@ -0,0 +1,122 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1207 +# +# Test atomic file updates when (a) the length is the same; (b) the length +# is different; and (c) someone modifies the original file and we need to +# cancel the update. The file contents are cloned into the staging file, +# and some of the contents are updated. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/reflink + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs' +_require_xfs_io_command startupdate +_require_test_reflink +_require_test + +filesnap() { + echo "$1" + md5sum $2 | _filter_test_dir +} + +mkfile() { + rm -f $dir/a + _pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full + sync +} + +dir=$TEST_DIR/test-$seq +mkdir -p $dir +blksz=65536 +nrblks=64 + +# Use the atomic file update staging prototype in xfs_io to update a file. +mkfile +filesnap "before commit" $dir/a + +$XFS_IO_PROG \ + -c 'startupdate' \ + -c 'pwrite -S 0x60 44k 55k -b 1m' \ + -c 'commitupdate -q' \ + "$dir/a" 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after commit" $dir/a +echo + +# Use the atomic file updates to replace a file with a shorter file. +mkfile +filesnap "before shorten commit" $dir/a + +$XFS_IO_PROG \ + -c 'startupdate' \ + -c 'truncate 55k' \ + -c 'pwrite -S 0x60 0 55k' \ + -c 'commitupdate -q' \ + "$dir/a" 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after shorten commit" $dir/a +echo + +# Use the atomic file updates to replace a file with a longer file. +mkfile +filesnap "before lengthen commit" $dir/a + +$XFS_IO_PROG \ + -c 'startupdate' \ + -c "pwrite -S 0x60 0 $(( (blksz * nrblks) + 37373 ))" \ + -c 'commitupdate -q' \ + "$dir/a" 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after lengthen commit" $dir/a +echo + +# Use the atomic file update staging prototype in xfs_io to cancel updating a +# file. +mkfile +filesnap "before cancel" $dir/a + +$XFS_IO_PROG \ + -c 'startupdate' \ + -c 'pwrite -S 0x60 44k 55k -b 1m' \ + -c 'cancelupdate' \ + "$dir/a" 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after cancel" $dir/a +echo + +# Now try the update but with the A file open separately so that we clobber +# mtime and fail the update. +mkfile +filesnap "before fail commit" $dir/a + +$XFS_IO_PROG \ + -c "open $dir/a" \ + -c 'startupdate' \ + -c 'pwrite -S 0x58 44k 55k -b 1m' \ + -c 'file 0' \ + -c 'close' \ + -c 'pwrite -S 0x61 22k 11k -b 1m' \ + -c 'commitupdate -q' \ + "$dir/a" 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after fail commit" $dir/a +echo + +# success, all done +status=0 +exit diff --git a/tests/generic/1207.out b/tests/generic/1207.out new file mode 100644 index 0000000000..ee33564067 --- /dev/null +++ b/tests/generic/1207.out @@ -0,0 +1,48 @@ +QA output created by 1207 +before commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1207/a +wrote 56320/56320 bytes at offset 45056 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Committed updates to 'TEST_DIR/test-1207/a'. +after commit +bedbd22b58a680219a1225353f6195fa TEST_DIR/test-1207/a + +before shorten commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1207/a +wrote 56320/56320 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Committed updates to 'TEST_DIR/test-1207/a'. +after shorten commit +52353039d89c5f2b76b9003464e5276a TEST_DIR/test-1207/a + +before lengthen commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1207/a +wrote 4231677/4231677 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Committed updates to 'TEST_DIR/test-1207/a'. +after lengthen commit +1839e7c6bf616160dc51b12179db2642 TEST_DIR/test-1207/a + +before cancel +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1207/a +wrote 56320/56320 bytes at offset 45056 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Cancelled updates to 'TEST_DIR/test-1207/a'. +after cancel +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1207/a + +before fail commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1207/a +committing update: Device or resource busy +wrote 56320/56320 bytes at offset 45056 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 56320/56320 bytes at offset 45056 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +[000] TEST_DIR/test-1207/a (xfs,non-sync,non-direct,read-write) + 001 TEST_DIR/test-1207/a (fileupdate) (xfs,non-sync,non-direct,read-write) +[000] TEST_DIR/test-1207/a (fileupdate) (xfs,non-sync,non-direct,read-write) +wrote 11264/11264 bytes at offset 22528 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +after fail commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1207/a + diff --git a/tests/generic/1209 b/tests/generic/1209 new file mode 100755 index 0000000000..6b287654ca --- /dev/null +++ b/tests/generic/1209 @@ -0,0 +1,101 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1209 +# +# Try invalid parameters to see if they fail. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/reflink + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs' +_require_xfs_io_command startupdate +_require_test +_require_scratch +_require_chattr i + +dir=$TEST_DIR/test-$seq +mkdir -p $dir +blksz=65536 +nrblks=64 + +_scratch_mkfs >> $seqres.full +_scratch_mount + +_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full +_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/b >> $seqres.full + +echo Immutable files +$XFS_IO_PROG -c 'chattr +i' -c "swapext $dir/b" $dir/a +$CHATTR_PROG -i $dir/a + +echo Readonly files +$XFS_IO_PROG -r -c "swapext $dir/b" $dir/a + +echo Directories +$XFS_IO_PROG -c "swapext $dir/b" $dir + +echo Unaligned ranges +$XFS_IO_PROG -c "swapext -s 37 -d 61 -l 17 $dir/b" $dir/a + +echo file1 range entirely beyond EOF +$XFS_IO_PROG -c "swapext -s $(( blksz * (nrblks + 500) )) -d 0 -l $blksz $dir/b" $dir/a + +echo file2 range entirely beyond EOF +$XFS_IO_PROG -c "swapext -d $(( blksz * (nrblks + 500) )) -s 0 -l $blksz $dir/b" $dir/a + +echo Both ranges entirely beyond EOF +$XFS_IO_PROG -c "swapext -d $(( blksz * (nrblks + 500) )) -s $(( blksz * (nrblks + 500) )) -l $blksz $dir/b" $dir/a + +echo file1 range crossing EOF +$XFS_IO_PROG -c "swapext -s $(( blksz * (nrblks - 1) )) -d 0 -l $((2 * blksz)) $dir/b" $dir/a + +echo file2 range crossing EOF +$XFS_IO_PROG -c "swapext -d $(( blksz * (nrblks - 1) )) -s 0 -l $((2 * blksz)) $dir/b" $dir/a + +echo Both ranges crossing EOF +$XFS_IO_PROG -c "swapext -d $(( blksz * (nrblks - 1) )) -s $(( blksz * (nrblks - 1) )) -l $((blksz * 2)) $dir/b" $dir/a + +echo file1 unaligned EOF to file2 nowhere near EOF +_pwrite_byte 0x58 $((blksz * nrblks)) 37 $dir/a >> $seqres.full +_pwrite_byte 0x59 $((blksz * nrblks)) 37 $dir/b >> $seqres.full +$XFS_IO_PROG -c "swapext -d 0 -s $(( blksz * nrblks )) -l 37 $dir/b" $dir/a + +echo file2 unaligned EOF to file1 nowhere near EOF +$XFS_IO_PROG -c "swapext -s 0 -d $(( blksz * nrblks )) -l 37 $dir/b" $dir/a + +echo Files of unequal length +_pwrite_byte 0x58 $((blksz * nrblks)) $((blksz * 2)) $dir/a >> $seqres.full +_pwrite_byte 0x59 $((blksz * nrblks)) $blksz $dir/b >> $seqres.full +$XFS_IO_PROG -c "swapext $dir/b" $dir/a + +echo Files on different filesystems +_pwrite_byte 0x58 0 $((blksz * nrblks)) $SCRATCH_MNT/c >> $seqres.full +$XFS_IO_PROG -c "swapext $SCRATCH_MNT/c" $dir/a + +echo Files on different mounts +mkdir -p $SCRATCH_MNT/xyz +mount --bind $dir $SCRATCH_MNT/xyz --bind +_pwrite_byte 0x60 0 $((blksz * (nrblks + 2))) $dir/c >> $seqres.full +$XFS_IO_PROG -c "swapext $SCRATCH_MNT/xyz/c" $dir/a +umount $SCRATCH_MNT/xyz + +echo Swapping a file with itself +$XFS_IO_PROG -c "swapext $dir/a" $dir/a + +# success, all done +status=0 +exit diff --git a/tests/generic/1209.out b/tests/generic/1209.out new file mode 100644 index 0000000000..a051ed97d9 --- /dev/null +++ b/tests/generic/1209.out @@ -0,0 +1,33 @@ +QA output created by 1209 +Immutable files +swapext: Operation not permitted +Readonly files +swapext: Bad file descriptor +Directories +swapext: Is a directory +Unaligned ranges +swapext: Invalid argument +file1 range entirely beyond EOF +swapext: Invalid argument +file2 range entirely beyond EOF +swapext: Invalid argument +Both ranges entirely beyond EOF +swapext: Invalid argument +file1 range crossing EOF +swapext: Invalid argument +file2 range crossing EOF +swapext: Invalid argument +Both ranges crossing EOF +swapext: Invalid argument +file1 unaligned EOF to file2 nowhere near EOF +swapext: Invalid argument +file2 unaligned EOF to file1 nowhere near EOF +swapext: Invalid argument +Files of unequal length +swapext: Bad address +Files on different filesystems +swapext: Invalid cross-device link +Files on different mounts +swapext: Invalid cross-device link +Swapping a file with itself +swapext: Invalid argument diff --git a/tests/generic/1210 b/tests/generic/1210 new file mode 100755 index 0000000000..93a325ad7a --- /dev/null +++ b/tests/generic/1210 @@ -0,0 +1,48 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1210 +# +# Make sure swapext honors RLIMIT_FSIZE. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/reflink + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs' +_require_test + +dir=$TEST_DIR/test-$seq +mkdir -p $dir +blksz=65536 +nrblks=64 + +# Create some 4M files to test swapext +_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full +_pwrite_byte 0x59 0 $((blksz * nrblks)) $dir/b >> $seqres.full +sync +md5sum $dir/a $dir/b | _filter_test_dir + +# Set FSIZE to twice the blocksize (IOWs, 128k) +ulimit -f $(( (blksz * 2) / 512)) +ulimit -a >> $seqres.full + +# Now try to swapext +$XFS_IO_PROG -c "swapext $dir/b" $dir/a +md5sum $dir/a $dir/b | _filter_test_dir + +# success, all done +status=0 +exit diff --git a/tests/generic/1210.out b/tests/generic/1210.out new file mode 100644 index 0000000000..02d41b8838 --- /dev/null +++ b/tests/generic/1210.out @@ -0,0 +1,6 @@ +QA output created by 1210 +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1210/a +901e136269b8d283d311697b7c6dc1f2 TEST_DIR/test-1210/b +swapext: Invalid argument +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1210/a +901e136269b8d283d311697b7c6dc1f2 TEST_DIR/test-1210/b diff --git a/tests/generic/1211 b/tests/generic/1211 new file mode 100755 index 0000000000..f7b8a8d280 --- /dev/null +++ b/tests/generic/1211 @@ -0,0 +1,105 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1211 +# +# Test atomic file replacement when (a) the length is the same; (b) the length +# is different; and (c) someone modifies the original file and we need to +# cancel the update. The staging file is created empty, which implies that the +# caller wants a full file replacement. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs' +_require_xfs_io_command startupdate '-e' +_require_test + +filesnap() { + echo "$1" + md5sum $2 | _filter_test_dir +} + +mkfile() { + rm -f $dir/a + _pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full + sync +} + +dir=$TEST_DIR/test-$seq +mkdir -p $dir +blksz=65536 +nrblks=64 + +# Use the atomic file update staging prototype in xfs_io to update a file. +mkfile +filesnap "before commit" $dir/a + +$XFS_IO_PROG \ + -c 'startupdate -e' \ + -c "pwrite -S 0x60 0 $((blksz * nrblks))" \ + -c 'commitupdate -q' \ + "$dir/a" 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after commit" $dir/a +echo + +# Use the atomic file updates to replace a file with a shorter file. +mkfile +filesnap "before shorten commit" $dir/a + +$XFS_IO_PROG \ + -c 'startupdate -e' \ + -c 'pwrite -S 0x60 0 55k' \ + -c 'commitupdate -q' \ + "$dir/a" 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after shorten commit" $dir/a +echo + +# Use the atomic file updates to replace a file with a longer file. +mkfile +filesnap "before lengthen commit" $dir/a + +$XFS_IO_PROG \ + -c 'startupdate -e' \ + -c "pwrite -S 0x60 0 $(( (blksz * nrblks) + 37373 ))" \ + -c 'commitupdate -q' \ + "$dir/a" 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after lengthen commit" $dir/a +echo + +# Now try the update but with the A file open separately so that we clobber +# mtime and fail the update. +mkfile +filesnap "before fail commit" $dir/a + +$XFS_IO_PROG \ + -c "open $dir/a" \ + -c 'startupdate -e ' \ + -c 'pwrite -S 0x58 44k 55k -b 1m' \ + -c 'file 0' \ + -c 'close' \ + -c 'pwrite -S 0x61 22k 11k -b 1m' \ + -c 'commitupdate -q' \ + "$dir/a" 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after fail commit" $dir/a +echo + +# success, all done +status=0 +exit diff --git a/tests/generic/1211.out b/tests/generic/1211.out new file mode 100644 index 0000000000..a149de198f --- /dev/null +++ b/tests/generic/1211.out @@ -0,0 +1,40 @@ +QA output created by 1211 +before commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1211/a +wrote 4194304/4194304 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Committed updates to 'TEST_DIR/test-1211/a'. +after commit +0558063c531ca7c7864fc5a4923f7144 TEST_DIR/test-1211/a + +before shorten commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1211/a +wrote 56320/56320 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Committed updates to 'TEST_DIR/test-1211/a'. +after shorten commit +52353039d89c5f2b76b9003464e5276a TEST_DIR/test-1211/a + +before lengthen commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1211/a +wrote 4231677/4231677 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Committed updates to 'TEST_DIR/test-1211/a'. +after lengthen commit +1839e7c6bf616160dc51b12179db2642 TEST_DIR/test-1211/a + +before fail commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1211/a +committing update: Device or resource busy +wrote 56320/56320 bytes at offset 45056 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 56320/56320 bytes at offset 45056 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +[000] TEST_DIR/test-1211/a (xfs,non-sync,non-direct,read-write) + 001 TEST_DIR/test-1211/a (fileupdate) (xfs,non-sync,non-direct,read-write) +[000] TEST_DIR/test-1211/a (fileupdate) (xfs,non-sync,non-direct,read-write) +wrote 11264/11264 bytes at offset 22528 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +after fail commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1211/a + diff --git a/tests/generic/1212 b/tests/generic/1212 new file mode 100755 index 0000000000..c6599e3a35 --- /dev/null +++ b/tests/generic/1212 @@ -0,0 +1,58 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1212 +# +# Stress testing with a lot of extents. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs' +_require_test_program punch-alternating +_require_test + +dir=$TEST_DIR/test-$seq +mkdir -p $dir +blksz=$(_get_file_block_size $TEST_DIR) +nrblks=$((LOAD_FACTOR * 100000)) + +_require_fs_space $TEST_DIR $(( (2 * blksz * nrblks) / 1024 )) + +# Create some big swiss cheese files to test swapext with a lot of extents +_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full +$here/src/punch-alternating $dir/a +_pwrite_byte 0x59 0 $((blksz * nrblks)) $dir/b >> $seqres.full +$here/src/punch-alternating -o 1 $dir/b +filefrag -v $dir/a $dir/b >> $seqres.full + +# Now try to swapext +md5_a="$(md5sum < $dir/a)" +md5_b="$(md5sum < $dir/b)" +date >> $seqres.full +$XFS_IO_PROG -c "swapext $dir/b" $dir/a +date >> $seqres.full + +echo "md5_a=$md5_a" >> $seqres.full +echo "md5_b=$md5_b" >> $seqres.full +md5sum $dir/a $dir/b >> $seqres.full + +test "$(md5sum < $dir/b)" = "$md5_a" || echo "file b does not match former a" +test "$(md5sum < $dir/a)" = "$md5_b" || echo "file a does not match former b" + +echo "Silence is golden!" +# success, all done +status=0 +exit diff --git a/tests/generic/1212.out b/tests/generic/1212.out new file mode 100644 index 0000000000..fb775977ca --- /dev/null +++ b/tests/generic/1212.out @@ -0,0 +1,2 @@ +QA output created by 1212 +Silence is golden! diff --git a/tests/generic/1213 b/tests/generic/1213 new file mode 100755 index 0000000000..d466263a9d --- /dev/null +++ b/tests/generic/1213 @@ -0,0 +1,126 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1213 +# +# Test non-root atomic file updates when (a) the file contents are cloned into +# the staging file; and (b) when the staging file is created empty. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/reflink + +# real QA test starts here +_require_xfs_io_command startupdate +_require_test_reflink +_require_test +_require_user + +filesnap() { + echo "$1" + md5sum $2 | _filter_test_dir +} + +mkfile() { + rm -f $dir/a + _pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full + chown $qa_user $dir/a $dir/ + sync +} + +dir=$TEST_DIR/test-$seq +mkdir -p $dir +blksz=65536 +nrblks=64 + +# Use the atomic file update staging prototype in xfs_io to update a file. +mkfile +filesnap "before commit" $dir/a + +cmd="$XFS_IO_PROG \ + -c 'startupdate' \ + -c 'pwrite -S 0x60 44k 55k -b 1m' \ + -c 'commitupdate -q' \ + \"$dir/a\"" +su -s /bin/bash -c "$cmd" $qa_user 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after commit" $dir/a +echo + +# Use the atomic file updates to replace a file with a shorter file. +mkfile +filesnap "before shorten commit" $dir/a + +cmd="$XFS_IO_PROG \ + -c 'startupdate' \ + -c 'truncate 55k' \ + -c 'pwrite -S 0x60 0 55k' \ + -c 'commitupdate -q' \ + \"$dir/a\"" +su -s /bin/bash -c "$cmd" $qa_user 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after shorten commit" $dir/a +echo + +# Use the atomic file updates to replace a file with a longer file. +mkfile +filesnap "before lengthen commit" $dir/a + +cmd="$XFS_IO_PROG \ + -c 'startupdate' \ + -c \"pwrite -S 0x60 0 $(( (blksz * nrblks) + 37373 ))\" \ + -c 'commitupdate -q' \ + \"$dir/a\"" +su -s /bin/bash -c "$cmd" $qa_user 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after lengthen commit" $dir/a +echo + +# Use the atomic file update staging prototype in xfs_io to cancel updating a +# file. +mkfile +filesnap "before cancel" $dir/a + +cmd="$XFS_IO_PROG \ + -c 'startupdate' \ + -c 'pwrite -S 0x60 44k 55k -b 1m' \ + -c 'cancelupdate' \ + \"$dir/a\"" +su -s /bin/bash -c "$cmd" $qa_user 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after cancel" $dir/a +echo + +# Now try the update but with the A file open separately so that we clobber +# mtime and fail the update. +mkfile +filesnap "before fail commit" $dir/a + +cmd="$XFS_IO_PROG \ + -c \"open $dir/a\" \ + -c 'startupdate' \ + -c 'pwrite -S 0x58 44k 55k -b 1m' \ + -c 'file 0' \ + -c 'close' \ + -c 'pwrite -S 0x61 22k 11k -b 1m' \ + -c 'commitupdate -q' \ + \"$dir/a\"" +su -s /bin/bash -c "$cmd" $qa_user 2>&1 | _filter_xfs_io | _filter_test_dir + +filesnap "after fail commit" $dir/a +echo + +# success, all done +status=0 +exit diff --git a/tests/generic/1213.out b/tests/generic/1213.out new file mode 100644 index 0000000000..b5a140560d --- /dev/null +++ b/tests/generic/1213.out @@ -0,0 +1,48 @@ +QA output created by 1213 +before commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1213/a +wrote 56320/56320 bytes at offset 45056 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Committed updates to 'TEST_DIR/test-1213/a'. +after commit +bedbd22b58a680219a1225353f6195fa TEST_DIR/test-1213/a + +before shorten commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1213/a +wrote 56320/56320 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Committed updates to 'TEST_DIR/test-1213/a'. +after shorten commit +52353039d89c5f2b76b9003464e5276a TEST_DIR/test-1213/a + +before lengthen commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1213/a +wrote 4231677/4231677 bytes at offset 0 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Committed updates to 'TEST_DIR/test-1213/a'. +after lengthen commit +1839e7c6bf616160dc51b12179db2642 TEST_DIR/test-1213/a + +before cancel +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1213/a +wrote 56320/56320 bytes at offset 45056 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Cancelled updates to 'TEST_DIR/test-1213/a'. +after cancel +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1213/a + +before fail commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1213/a +committing update: Device or resource busy +wrote 56320/56320 bytes at offset 45056 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 56320/56320 bytes at offset 45056 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +[000] TEST_DIR/test-1213/a (xfs,non-sync,non-direct,read-write) + 001 TEST_DIR/test-1213/a (fileupdate) (xfs,non-sync,non-direct,read-write) +[000] TEST_DIR/test-1213/a (fileupdate) (xfs,non-sync,non-direct,read-write) +wrote 11264/11264 bytes at offset 22528 +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +after fail commit +d712f003e9d467e063cda1baf319b928 TEST_DIR/test-1213/a + diff --git a/tests/generic/1214 b/tests/generic/1214 new file mode 100755 index 0000000000..0cacc57e9c --- /dev/null +++ b/tests/generic/1214 @@ -0,0 +1,63 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1214 +# +# Test swapext with the fsync flag flushes everything to disk before the call +# returns. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/reflink + +# real QA test starts here +_require_test_program "punch-alternating" +_require_xfs_io_command swapext '-v vfs -a' +_require_scratch +_require_scratch_shutdown + +_scratch_mkfs >> $seqres.full +_scratch_mount + +_pwrite_byte 0x58 0 2m $SCRATCH_MNT/a >> $seqres.full +_pwrite_byte 0x59 0 2m $SCRATCH_MNT/b >> $seqres.full +$here/src/punch-alternating $SCRATCH_MNT/a +$here/src/punch-alternating $SCRATCH_MNT/b + +old_a=$(md5sum $SCRATCH_MNT/a | awk '{print $1}') +old_b=$(md5sum $SCRATCH_MNT/b | awk '{print $1}') +echo "md5 a: $old_a md5 b: $old_b" >> $seqres.full + +od -tx1 -Ad -c $SCRATCH_MNT/a > /tmp/a0 +od -tx1 -Ad -c $SCRATCH_MNT/b > /tmp/b0 + +echo swap >> $seqres.full +$XFS_IO_PROG -c "swapext -v vfs -a -e -f -u $SCRATCH_MNT/a" $SCRATCH_MNT/b +_scratch_shutdown +_scratch_cycle_mount + +new_a=$(md5sum $SCRATCH_MNT/a | awk '{print $1}') +new_b=$(md5sum $SCRATCH_MNT/b | awk '{print $1}') +echo "md5 a: $new_a md5 b: $new_b" >> $seqres.full + +test $old_a = $new_b || echo "scratch file B doesn't match old file A" +test $old_b = $new_a || echo "scratch file A doesn't match old file B" + +od -tx1 -Ad -c $SCRATCH_MNT/a > /tmp/a1 +od -tx1 -Ad -c $SCRATCH_MNT/b > /tmp/b1 + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/generic/1214.out b/tests/generic/1214.out new file mode 100644 index 0000000000..a529e42333 --- /dev/null +++ b/tests/generic/1214.out @@ -0,0 +1,2 @@ +QA output created by 1214 +Silence is golden diff --git a/tests/generic/1215 b/tests/generic/1215 new file mode 100755 index 0000000000..28eb8356d8 --- /dev/null +++ b/tests/generic/1215 @@ -0,0 +1,70 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1215 +# +# Test swapext with the dry run flag doesn't change anything. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/reflink + +# real QA test starts here +_require_test_program "punch-alternating" +_require_xfs_io_command swapext '-v vfs' +_require_scratch + +_scratch_mkfs >> $seqres.full +_scratch_mount + +_pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full +_pwrite_byte 0x59 0 2m $SCRATCH_MNT/b >> $seqres.full +$XFS_IO_PROG -c 'truncate 2m' $SCRATCH_MNT/a +$here/src/punch-alternating $SCRATCH_MNT/a +$here/src/punch-alternating $SCRATCH_MNT/b + +old_a=$(md5sum $SCRATCH_MNT/a | awk '{print $1}') +old_b=$(md5sum $SCRATCH_MNT/b | awk '{print $1}') +echo "md5 a: $old_a md5 b: $old_b" >> $seqres.full + +# Test swapext with the -n option, which will do all the input parameter +# checking and return 0 without changing anything. +echo dry run swap >> $seqres.full +$XFS_IO_PROG -c "swapext -v vfs -n -f -u $SCRATCH_MNT/a" $SCRATCH_MNT/b +_scratch_cycle_mount + +new_a=$(md5sum $SCRATCH_MNT/a | awk '{print $1}') +new_b=$(md5sum $SCRATCH_MNT/b | awk '{print $1}') +echo "md5 a: $new_a md5 b: $new_b" >> $seqres.full + +test $old_a = $new_a || echo "scratch file A should not have swapped" +test $old_b = $new_b || echo "scratch file B should not have swapped" + +# Do it again, but without the -n option, to prove that we can actually +# swap the file contents. +echo actual swap >> $seqres.full +$XFS_IO_PROG -c "swapext -v vfs -f -u $SCRATCH_MNT/a" $SCRATCH_MNT/b +_scratch_cycle_mount + +new_a=$(md5sum $SCRATCH_MNT/a | awk '{print $1}') +new_b=$(md5sum $SCRATCH_MNT/b | awk '{print $1}') +echo "md5 a: $new_a md5 b: $new_b" >> $seqres.full + +test $old_a = $new_b || echo "scratch file A should have swapped" +test $old_b = $new_a || echo "scratch file B should have swapped" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/generic/1215.out b/tests/generic/1215.out new file mode 100644 index 0000000000..f68a8e6998 --- /dev/null +++ b/tests/generic/1215.out @@ -0,0 +1,2 @@ +QA output created by 1215 +Silence is golden diff --git a/tests/xfs/1208 b/tests/xfs/1208 new file mode 100755 index 0000000000..3106a4036c --- /dev/null +++ b/tests/xfs/1208 @@ -0,0 +1,62 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1208 +# +# Make sure an atomic swapext actually runs to completion even if we shut +# down the filesystem midway through. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter +. ./common/reflink +. ./common/inject + +# real QA test starts here +_supported_fs xfs +_require_xfs_io_command swapext '-v vfs -a' +_require_test_program "punch-alternating" +_require_xfs_io_command startupdate +_require_xfs_io_error_injection "bmap_finish_one" +_require_test + +filesnap() { + echo "$1" + md5sum $dir/a $dir/b $dir/c | _filter_test_dir +} + +dir=$TEST_DIR/test-$seq +mkdir -p $dir +blksz=65536 +nrblks=137 + +# Create two files to swap, and try to fragment the first file. +rm -f $dir/a +_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full +$here/src/punch-alternating $dir/a +_pwrite_byte 0x59 0 $((blksz * nrblks)) $dir/b >> $seqres.full +_pwrite_byte 0x59 0 $((blksz * nrblks)) $dir/c >> $seqres.full +_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full +sync + +# Inject a bmap error and trigger it via swapext. +filesnap "before commit" +$XFS_IO_PROG -x -c 'inject bmap_finish_one' -c "swapext $dir/b" $dir/a + +# Check the file afterwards. +_test_cycle_mount +filesnap "after commit" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1208.out b/tests/xfs/1208.out new file mode 100644 index 0000000000..b6aa1b6231 --- /dev/null +++ b/tests/xfs/1208.out @@ -0,0 +1,10 @@ +QA output created by 1208 +before commit +c7221b1494117327570a0958b0abca51 TEST_DIR/test-1208/a +30cc2b6b307081e10972317013efb0f3 TEST_DIR/test-1208/b +30cc2b6b307081e10972317013efb0f3 TEST_DIR/test-1208/c +swapext: Input/output error +after commit +30cc2b6b307081e10972317013efb0f3 TEST_DIR/test-1208/a +c7221b1494117327570a0958b0abca51 TEST_DIR/test-1208/b +30cc2b6b307081e10972317013efb0f3 TEST_DIR/test-1208/c ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 2/7] generic: test old xfs extent swapping ioctl 2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 3/7] generic: test new vfs swapext ioctl Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 7/7] fsstress: update for FIEXCHANGE_RANGE Darrick J. Wong 2022-12-30 22:19 ` [PATCH 6/7] fsx: support FIEXCHANGE_RANGE Darrick J. Wong 6 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add some tests to check the operation of the old xfs swapext ioctl. There aren't any xfs-specific pieces in here, so they're in generic/ Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/rc | 11 +++++++++ doc/group-names.txt | 2 ++ tests/generic/1200 | 55 +++++++++++++++++++++++++++++++++++++++++++++ tests/generic/1200.out | 3 ++ tests/generic/1201 | 53 +++++++++++++++++++++++++++++++++++++++++++ tests/generic/1201.out | 4 +++ tests/generic/1202 | 47 ++++++++++++++++++++++++++++++++++++++ tests/generic/1202.out | 2 ++ tests/xfs/1202 | 59 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1202.out | 12 ++++++++++ tests/xfs/537 | 2 +- 11 files changed, 249 insertions(+), 1 deletion(-) create mode 100755 tests/generic/1200 create mode 100644 tests/generic/1200.out create mode 100755 tests/generic/1201 create mode 100644 tests/generic/1201.out create mode 100755 tests/generic/1202 create mode 100644 tests/generic/1202.out create mode 100755 tests/xfs/1202 create mode 100644 tests/xfs/1202.out diff --git a/common/rc b/common/rc index a1b65f0a7f..88ecc1837d 100644 --- a/common/rc +++ b/common/rc @@ -2575,6 +2575,17 @@ _require_xfs_io_command() echo $testio | grep -q "Inappropriate ioctl" && \ _notrun "xfs_io $command support is missing" ;; + "swapext") + $XFS_IO_PROG -f -c 'pwrite -S 0x58 0 128k -b 128k' $testfile > /dev/null + $XFS_IO_PROG -f -c 'truncate 128k' $testfile.1 > /dev/null + testio=`$XFS_IO_PROG -c "$command $param $testfile.1" $testfile 2>&1` + echo $testio | grep -q "bad argument count" && \ + _notrun "xfs_io $command $param support is missing" + echo $testio | grep -q "Inappropriate ioctl" && \ + _notrun "xfs_io $command $param ioctl support is missing" + rm -f $testfile.1 + param_checked="$param" + ;; "utimes" ) testio=`$XFS_IO_PROG -f -c "utimes 0 0 0 0" $testfile 2>&1` ;; diff --git a/doc/group-names.txt b/doc/group-names.txt index 771ce937ae..e88dcc0fdd 100644 --- a/doc/group-names.txt +++ b/doc/group-names.txt @@ -51,6 +51,7 @@ enospc ENOSPC error reporting exportfs file handles fiemap fiemap ioctl filestreams XFS filestreams allocator +fiexchange FIEXCHANGE_RANGE ioctl freeze filesystem freeze tests fsck general fsck tests fsmap FS_IOC_GETFSMAP ioctl @@ -121,6 +122,7 @@ splice splice system call stress fsstress filesystem exerciser subvol btrfs subvolumes swap swap files +swapext XFS_IOC_SWAPEXT ioctl symlink symbolic links tape dump and restore with a tape thin thin provisioning diff --git a/tests/generic/1200 b/tests/generic/1200 new file mode 100755 index 0000000000..58b93e2b6e --- /dev/null +++ b/tests/generic/1200 @@ -0,0 +1,55 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1200 +# +# Can we use swapext to make the quota accounting incorrect? + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext quota + +# Import common functions. +. ./common/filter +. ./common/quota + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs' +_require_user +_require_nobody +_require_quota +_require_xfs_quota +_require_scratch + +# Format filesystem and set up quota limits +_scratch_mkfs > $seqres.full +_qmount_option "usrquota,grpquota" +_qmount >> $seqres.full + +# Set up initial files +$XFS_IO_PROG -f -c 'pwrite -S 0x58 0 256k -b 1m' $SCRATCH_MNT/a >> $seqres.full +chown $qa_user $SCRATCH_MNT/a +$XFS_IO_PROG -f -c 'pwrite -S 0x59 0 64k -b 64k' -c 'truncate 256k' $SCRATCH_MNT/b >> $seqres.full +chown nobody $SCRATCH_MNT/b + +echo before swapext >> $seqres.full +$XFS_QUOTA_PROG -x -c 'report -a' $SCRATCH_MNT >> $seqres.full +stat $SCRATCH_MNT/* >> $seqres.full + +# Now try to swap the extents of the two files. The command is allowed to +# fail with -EINVAL (since that's what the first kernel fix does) or succeed +# (because subsequent rewrites can handle quota). Whatever the outcome, the +# quota usage check at the end should never show a discrepancy. +$XFS_IO_PROG -c "swapext $SCRATCH_MNT/b" $SCRATCH_MNT/a &> $tmp.swap +cat $tmp.swap >> $seqres.full +grep -v 'Invalid argument' $tmp.swap + +echo after swapext >> $seqres.full +$XFS_QUOTA_PROG -x -c 'report -a' $SCRATCH_MNT >> $seqres.full +stat $SCRATCH_MNT/* >> $seqres.full + +_check_quota_usage + +# success, all done +status=0 +exit diff --git a/tests/generic/1200.out b/tests/generic/1200.out new file mode 100644 index 0000000000..b1a6357719 --- /dev/null +++ b/tests/generic/1200.out @@ -0,0 +1,3 @@ +QA output created by 1200 +Comparing user usage +Comparing group usage diff --git a/tests/generic/1201 b/tests/generic/1201 new file mode 100755 index 0000000000..91e3773eaa --- /dev/null +++ b/tests/generic/1201 @@ -0,0 +1,53 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1201 +# +# Can we use swapext to exceed the quota enforcement? + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext quota + +# Import common functions. +. ./common/filter +. ./common/quota + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs' +_require_user +_require_nobody +_require_quota +_require_xfs_quota +_require_scratch + +# Format filesystem and set up quota limits +_scratch_mkfs > $seqres.full +_qmount_option "usrquota,grpquota" +_qmount >> $seqres.full + +# Set up initial files +$XFS_IO_PROG -f -c 'pwrite -S 0x58 0 256k -b 1m' $SCRATCH_MNT/a >> $seqres.full +chown $qa_user $SCRATCH_MNT/a +$XFS_IO_PROG -f -c 'pwrite -S 0x59 0 64k -b 64k' -c 'truncate 256k' $SCRATCH_MNT/b >> $seqres.full +chown nobody $SCRATCH_MNT/b + +# Set up a quota limit +$XFS_QUOTA_PROG -x -c "limit -u bhard=70k nobody" $SCRATCH_MNT + +echo before swapext >> $seqres.full +$XFS_QUOTA_PROG -x -c 'report -a' $SCRATCH_MNT >> $seqres.full +stat $SCRATCH_MNT/* >> $seqres.full + +# Now try to swapext +$XFS_IO_PROG -c "swapext $SCRATCH_MNT/b" $SCRATCH_MNT/a + +echo after swapext >> $seqres.full +$XFS_QUOTA_PROG -x -c 'report -a' $SCRATCH_MNT >> $seqres.full +stat $SCRATCH_MNT/* >> $seqres.full + +_check_quota_usage + +# success, all done +status=0 +exit diff --git a/tests/generic/1201.out b/tests/generic/1201.out new file mode 100644 index 0000000000..9001a59958 --- /dev/null +++ b/tests/generic/1201.out @@ -0,0 +1,4 @@ +QA output created by 1201 +swapext: Disk quota exceeded +Comparing user usage +Comparing group usage diff --git a/tests/generic/1202 b/tests/generic/1202 new file mode 100755 index 0000000000..2afa546f1f --- /dev/null +++ b/tests/generic/1202 @@ -0,0 +1,47 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1202 +# +# Make sure that swapext won't touch a swap file. + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + test -e "$dir/a" && swapoff $dir/a + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter + +# real QA test starts here +_require_xfs_io_command swapext '-v vfs' +_require_test + +dir=$TEST_DIR/test-$seq +mkdir -p $dir + +# Set up a fragmented swapfile and a dummy donor file. +$XFS_IO_PROG -f -c 'pwrite -S 0x58 0 32m -b 1m' -c fsync $dir/a >> $seqres.full +$here/src/punch-alternating $dir/a +$XFS_IO_PROG -f -c 'pwrite -S 0x58 0 32m -b 1m' -c fsync $dir/a >> $seqres.full +$MKSWAP_PROG $dir/a >> $seqres.full + +$XFS_IO_PROG -f -c 'pwrite -S 0x59 0 32m -b 1m' $dir/b >> $seqres.full + +swapon $dir/a || _notrun 'failed to swapon' + +# Now try to swapext. The old code would return EINVAL for swapfiles +# even though everything else in the VFS returns ETXTBSY. +$XFS_IO_PROG -c "swapext $dir/b" $dir/a 2>&1 | \ + sed -e 's/swapext: Invalid argument/swapext: Text file busy/g' + +# success, all done +status=0 +exit diff --git a/tests/generic/1202.out b/tests/generic/1202.out new file mode 100644 index 0000000000..029d65e4d0 --- /dev/null +++ b/tests/generic/1202.out @@ -0,0 +1,2 @@ +QA output created by 1202 +swapext: Text file busy diff --git a/tests/xfs/1202 b/tests/xfs/1202 new file mode 100755 index 0000000000..73e56641af --- /dev/null +++ b/tests/xfs/1202 @@ -0,0 +1,59 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1202 +# +# Simple tests of the old xfs swapext ioctl + +. ./common/preamble +_begin_fstest auto quick fiexchange swapext + +# Override the default cleanup function. +_cleanup() +{ + cd / + rm -r -f $tmp.* $dir +} + +# Import common functions. +. ./common/filter + +# real QA test starts here +_supported_fs xfs +_require_xfs_io_command swapext '-v vfs' +_require_test + +# We can't do any reasonable swapping if the files we're going to create are +# realtime files, the rt extent size is greater than 1 block, and we can't use +# atomic extent swapping to make sure that partially written extents are fully +# swapped. +file_blksz=$(_get_file_block_size $TEST_DIR) +fs_blksz=$(_get_block_size $TEST_DIR) +if (( $file_blksz != $fs_blksz )); then + _xfs_has_feature $TEST_DIR reflink || \ + _notrun "test requires atomic extent swapping for rextsize=$((file_blksz / fs_blksz))" +fi + +dir=$TEST_DIR/test-$seq +mkdir -p $dir + +$XFS_IO_PROG -f -c 'pwrite -S 0x58 0 256k -b 1m' $dir/a >> $seqres.full +$XFS_IO_PROG -f -c 'pwrite -S 0x59 0 256k -b 1m' $dir/b >> $seqres.full +$XFS_IO_PROG -f -c 'pwrite -S 0x60 0 256k -b 1m' $dir/c >> $seqres.full +$XFS_IO_PROG -f -c 'pwrite -S 0x61 0 128k -b 1m' $dir/d >> $seqres.full +md5sum $dir/a $dir/b $dir/c $dir/d | _filter_test_dir + +# Swap two files that are the same length +echo swap +$XFS_IO_PROG -c "swapext $dir/b" $dir/a +md5sum $dir/a $dir/b | _filter_test_dir + +# Try to swap two files that are not the same length +echo fail swap +$XFS_IO_PROG -c "swapext $dir/c" $dir/d +md5sum $dir/c $dir/d | _filter_test_dir + +# success, all done +status=0 +exit diff --git a/tests/xfs/1202.out b/tests/xfs/1202.out new file mode 100644 index 0000000000..23127eb7b7 --- /dev/null +++ b/tests/xfs/1202.out @@ -0,0 +1,12 @@ +QA output created by 1202 +af5c6e2d6c297f3139a4e99df396c072 TEST_DIR/test-1202/a +fba5c83875b2ab054e06a0284346ebdf TEST_DIR/test-1202/b +40c08c6f2aca19bb0d2cf1dbd8bc1b1c TEST_DIR/test-1202/c +81615449a98aaaad8dc179b3bec87f38 TEST_DIR/test-1202/d +swap +fba5c83875b2ab054e06a0284346ebdf TEST_DIR/test-1202/a +af5c6e2d6c297f3139a4e99df396c072 TEST_DIR/test-1202/b +fail swap +swapext: Bad address +40c08c6f2aca19bb0d2cf1dbd8bc1b1c TEST_DIR/test-1202/c +81615449a98aaaad8dc179b3bec87f38 TEST_DIR/test-1202/d diff --git a/tests/xfs/537 b/tests/xfs/537 index 7e11488799..6364db9b5d 100755 --- a/tests/xfs/537 +++ b/tests/xfs/537 @@ -7,7 +7,7 @@ # Verify that XFS does not cause inode fork's extent count to overflow when # swapping forks between files . ./common/preamble -_begin_fstest auto quick collapse +_begin_fstest auto quick collapse swapext # Import common functions. . ./common/filter ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 7/7] fsstress: update for FIEXCHANGE_RANGE 2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong ` (4 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 2/7] generic: test old xfs extent swapping ioctl Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 6/7] fsx: support FIEXCHANGE_RANGE Darrick J. Wong 6 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Teach this stress tool to be able to use the file content exchange ioctl. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- ltp/fsstress.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) diff --git a/ltp/fsstress.c b/ltp/fsstress.c index 10608fb554..0fba3d92a0 100644 --- a/ltp/fsstress.c +++ b/ltp/fsstress.c @@ -143,6 +143,7 @@ typedef enum { OP_URING_WRITE, OP_WRITE, OP_WRITEV, + OP_XCHGRANGE, OP_LAST } opty_t; @@ -272,6 +273,8 @@ void uring_read_f(opnum_t, long); void uring_write_f(opnum_t, long); void write_f(opnum_t, long); void writev_f(opnum_t, long); +void xchgrange_f(opnum_t, long); + char *xattr_flag_to_string(int); struct opdesc ops[OP_LAST] = { @@ -340,6 +343,7 @@ struct opdesc ops[OP_LAST] = { [OP_URING_WRITE] = {"uring_write", uring_write_f, 1, 1 }, [OP_WRITE] = {"write", write_f, 4, 1 }, [OP_WRITEV] = {"writev", writev_f, 4, 1 }, + [OP_XCHGRANGE] = {"xchgrange", xchgrange_f, 4, 1 }, }, *ops_end; flist_t flist[FT_nft] = { @@ -2494,6 +2498,170 @@ chown_f(opnum_t opno, long r) free_pathname(&f); } +/* exchange some arbitrary range of f1 to f2...fn. */ +void +xchgrange_f( + opnum_t opno, + long r) +{ +#ifdef FIEXCHANGE_RANGE + struct file_xchg_range fxr = { 0 }; + static __u64 swap_flags = 0; + struct pathname fpath1; + struct pathname fpath2; + struct stat64 stat1; + struct stat64 stat2; + char inoinfo1[1024]; + char inoinfo2[1024]; + off64_t lr; + off64_t off1; + off64_t off2; + off64_t max_off2; + size_t len; + int v1; + int v2; + int fd1; + int fd2; + int ret; + int tries = 0; + int e; + + /* Load paths */ + init_pathname(&fpath1); + if (!get_fname(FT_REGm, r, &fpath1, NULL, NULL, &v1)) { + if (v1) + printf("%d/%lld: xchgrange read - no filename\n", + procid, opno); + goto out_fpath1; + } + + init_pathname(&fpath2); + if (!get_fname(FT_REGm, random(), &fpath2, NULL, NULL, &v2)) { + if (v2) + printf("%d/%lld: xchgrange write - no filename\n", + procid, opno); + goto out_fpath2; + } + + /* Open files */ + fd1 = open_path(&fpath1, O_RDONLY); + e = fd1 < 0 ? errno : 0; + check_cwd(); + if (fd1 < 0) { + if (v1) + printf("%d/%lld: xchgrange read - open %s failed %d\n", + procid, opno, fpath1.path, e); + goto out_fpath2; + } + + fd2 = open_path(&fpath2, O_WRONLY); + e = fd2 < 0 ? errno : 0; + check_cwd(); + if (fd2 < 0) { + if (v2) + printf("%d/%lld: xchgrange write - open %s failed %d\n", + procid, opno, fpath2.path, e); + goto out_fd1; + } + + /* Get file stats */ + if (fstat64(fd1, &stat1) < 0) { + if (v1) + printf("%d/%lld: xchgrange read - fstat64 %s failed %d\n", + procid, opno, fpath1.path, errno); + goto out_fd2; + } + inode_info(inoinfo1, sizeof(inoinfo1), &stat1, v1); + + if (fstat64(fd2, &stat2) < 0) { + if (v2) + printf("%d/%lld: xchgrange write - fstat64 %s failed %d\n", + procid, opno, fpath2.path, errno); + goto out_fd2; + } + inode_info(inoinfo2, sizeof(inoinfo2), &stat2, v2); + + if (stat1.st_size < (stat1.st_blksize * 2) || + stat2.st_size < (stat2.st_blksize * 2)) { + if (v2) + printf("%d/%lld: xchgrange - files are too small\n", + procid, opno); + goto out_fd2; + } + + /* Never let us swap more than 1/4 of the files. */ + len = (random() % FILELEN_MAX) + 1; + if (len > stat1.st_size / 4) + len = stat1.st_size / 4; + if (len > stat2.st_size / 4) + len = stat2.st_size / 4; + len = rounddown_64(len, stat1.st_blksize); + if (len == 0) + len = stat1.st_blksize; + + /* Calculate offsets */ + lr = ((int64_t)random() << 32) + random(); + if (stat1.st_size == len) + off1 = 0; + else + off1 = (off64_t)(lr % MIN(stat1.st_size - len, MAXFSIZE)); + off1 %= maxfsize; + off1 = rounddown_64(off1, stat1.st_blksize); + + /* + * If srcfile == destfile, randomly generate destination ranges + * until we find one that doesn't overlap the source range. + */ + max_off2 = MIN(stat2.st_size - len, MAXFSIZE); + do { + lr = ((int64_t)random() << 32) + random(); + if (stat2.st_size == len) + off2 = 0; + else + off2 = (off64_t)(lr % max_off2); + off2 %= maxfsize; + off2 = rounddown_64(off2, stat2.st_blksize); + } while (stat1.st_ino == stat2.st_ino && + llabs(off2 - off1) < len && + tries++ < 10); + + /* Swap data blocks */ + fxr.file1_fd = fd1; + fxr.file1_offset = off1; + fxr.length = len; + fxr.file2_offset = off2; + fxr.flags = swap_flags; + +retry: + ret = ioctl(fd2, FIEXCHANGE_RANGE, &fxr); + e = ret < 0 ? errno : 0; + if (e == EOPNOTSUPP && !(swap_flags & FILE_XCHG_RANGE_NONATOMIC)) { + swap_flags = FILE_XCHG_RANGE_NONATOMIC; + fxr.flags |= swap_flags; + goto retry; + } + if (v1 || v2) { + printf("%d/%lld: xchgrange %s%s [%lld,%lld] -> %s%s [%lld,%lld]", + procid, opno, + fpath1.path, inoinfo1, (long long)off1, (long long)len, + fpath2.path, inoinfo2, (long long)off2, (long long)len); + + if (ret < 0) + printf(" error %d", e); + printf("\n"); + } + +out_fd2: + close(fd2); +out_fd1: + close(fd1); +out_fpath2: + free_pathname(&fpath2); +out_fpath1: + free_pathname(&fpath1); +#endif +} + /* reflink some arbitrary range of f1 to f2. */ void clonerange_f( ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 6/7] fsx: support FIEXCHANGE_RANGE 2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong ` (5 preceding siblings ...) 2022-12-30 22:19 ` [PATCH 7/7] fsstress: update for FIEXCHANGE_RANGE Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2023-02-28 1:55 ` Zorro Lang 6 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Upgrade fsx to support exchanging file contents. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- configure.ac | 1 include/builddefs.in | 1 ltp/Makefile | 4 + ltp/fsx.c | 160 ++++++++++++++++++++++++++++++++++++++++++++++++- m4/package_libcdev.m4 | 21 ++++++ src/fiexchange.h | 101 +++++++++++++++++++++++++++++++ src/global.h | 6 ++ 7 files changed, 292 insertions(+), 2 deletions(-) create mode 100644 src/fiexchange.h diff --git a/configure.ac b/configure.ac index e92bd6b26d..4687d8a3c0 100644 --- a/configure.ac +++ b/configure.ac @@ -70,6 +70,7 @@ AC_HAVE_SEEK_DATA AC_HAVE_BMV_OF_SHARED AC_HAVE_NFTW AC_HAVE_RLIMIT_NOFILE +AC_HAVE_FIEXCHANGE AC_CHECK_FUNCS([renameat2]) AC_CHECK_FUNCS([reallocarray]) diff --git a/include/builddefs.in b/include/builddefs.in index dab10c968f..969acf0da2 100644 --- a/include/builddefs.in +++ b/include/builddefs.in @@ -72,6 +72,7 @@ HAVE_SEEK_DATA = @have_seek_data@ HAVE_NFTW = @have_nftw@ HAVE_BMV_OF_SHARED = @have_bmv_of_shared@ HAVE_RLIMIT_NOFILE = @have_rlimit_nofile@ +HAVE_FIEXCHANGE = @have_fiexchange@ GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall diff --git a/ltp/Makefile b/ltp/Makefile index 85f634145c..c2b70d896e 100644 --- a/ltp/Makefile +++ b/ltp/Makefile @@ -36,6 +36,10 @@ ifeq ($(HAVE_COPY_FILE_RANGE),yes) LCFLAGS += -DHAVE_COPY_FILE_RANGE endif +ifeq ($(HAVE_FIEXCHANGE),yes) +LCFLAGS += -DHAVE_FIEXCHANGE +endif + default: depend $(TARGETS) depend: .dep diff --git a/ltp/fsx.c b/ltp/fsx.c index 12c2cc33bf..ee4b8fe45d 100644 --- a/ltp/fsx.c +++ b/ltp/fsx.c @@ -111,6 +111,7 @@ enum { OP_CLONE_RANGE, OP_DEDUPE_RANGE, OP_COPY_RANGE, + OP_EXCHANGE_RANGE, OP_MAX_FULL, /* integrity operations */ @@ -175,6 +176,7 @@ int check_file = 0; /* -X flag enables */ int clone_range_calls = 1; /* -J flag disables */ int dedupe_range_calls = 1; /* -B flag disables */ int copy_range_calls = 1; /* -E flag disables */ +int xchg_range_calls = 1; /* -0 flag disables */ int integrity = 0; /* -i flag */ int fsxgoodfd = 0; int o_direct; /* -Z */ @@ -268,6 +270,7 @@ static const char *op_names[] = { [OP_DEDUPE_RANGE] = "dedupe_range", [OP_COPY_RANGE] = "copy_range", [OP_FSYNC] = "fsync", + [OP_EXCHANGE_RANGE] = "xchg_range", }; static const char *op_name(int operation) @@ -452,6 +455,20 @@ logdump(void) if (overlap) prt("\t******IIII"); break; + case OP_EXCHANGE_RANGE: + prt("XCHG 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x", + lp->args[0], lp->args[0] + lp->args[1] - 1, + lp->args[1], + lp->args[2], lp->args[2] + lp->args[1] - 1); + overlap2 = badoff >= lp->args[2] && + badoff < lp->args[2] + lp->args[1]; + if (overlap && overlap2) + prt("\tXXXX**XXXX"); + else if (overlap) + prt("\tXXXX******"); + else if (overlap2) + prt("\t******XXXX"); + break; case OP_CLONE_RANGE: prt("CLONE 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x", lp->args[0], lp->args[0] + lp->args[1] - 1, @@ -1369,6 +1386,116 @@ do_insert_range(unsigned offset, unsigned length) } #endif +#ifdef FIEXCHANGE_RANGE +static __u64 swap_flags = 0; + +int +test_xchg_range(void) +{ + struct file_xchg_range fsr = { + .file1_fd = fd, + .flags = FILE_XCHG_RANGE_DRY_RUN | swap_flags, + }; + int ret, e; + +retry: + ret = ioctl(fd, FIEXCHANGE_RANGE, &fsr); + e = ret < 0 ? errno : 0; + if (e == EOPNOTSUPP && !(swap_flags & FILE_XCHG_RANGE_NONATOMIC)) { + /* + * If the call fails with atomic mode, try again with non + * atomic mode. + */ + swap_flags = FILE_XCHG_RANGE_NONATOMIC; + fsr.flags |= swap_flags; + goto retry; + } + if (e == EOPNOTSUPP || errno == ENOTTY) { + if (!quiet) + fprintf(stderr, + "main: filesystem does not support " + "exchange range, disabling!\n"); + return 0; + } + + return 1; +} + +void +do_xchg_range(unsigned offset, unsigned length, unsigned dest) +{ + struct file_xchg_range fsr = { + .file1_fd = fd, + .file1_offset = offset, + .file2_offset = dest, + .length = length, + .flags = swap_flags, + }; + void *p; + + if (length == 0) { + if (!quiet && testcalls > simulatedopcount) + prt("skipping zero length exchange range\n"); + log5(OP_EXCHANGE_RANGE, offset, length, dest, FL_SKIPPED); + return; + } + + if ((loff_t)offset >= file_size || (loff_t)dest >= file_size) { + if (!quiet && testcalls > simulatedopcount) + prt("skipping exchange range behind EOF\n"); + log5(OP_EXCHANGE_RANGE, offset, length, dest, FL_SKIPPED); + return; + } + + p = malloc(length); + if (!p) { + if (!quiet && testcalls > simulatedopcount) + prt("skipping exchange range due to ENOMEM\n"); + log5(OP_EXCHANGE_RANGE, offset, length, dest, FL_SKIPPED); + return; + } + + log5(OP_EXCHANGE_RANGE, offset, length, dest, FL_NONE); + + if (testcalls <= simulatedopcount) + goto out_free; + + if ((progressinterval && testcalls % progressinterval == 0) || + (debug && (monitorstart == -1 || monitorend == -1 || + dest <= monitorstart || dest + length <= monitorend))) { + prt("%lu swap\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n", + testcalls, offset, offset+length, length, dest); + } + + if (ioctl(fd, FIEXCHANGE_RANGE, &fsr) == -1) { + prt("exchange range: 0x%x to 0x%x at 0x%x\n", offset, + offset + length, dest); + prterr("do_xchg_range: FIEXCHANGE_RANGE"); + report_failure(161); + goto out_free; + } + + memcpy(p, good_buf + offset, length); + memcpy(good_buf + offset, good_buf + dest, length); + memcpy(good_buf + dest, p, length); +out_free: + free(p); +} + +#else +int +test_xchg_range(void) +{ + return 0; +} + +void +do_xchg_range(unsigned offset, unsigned length, unsigned dest) +{ + return; +} +#endif + #ifdef FICLONERANGE int test_clone_range(void) @@ -1856,6 +1983,7 @@ static int op_args_count(int operation) { switch (operation) { + case OP_EXCHANGE_RANGE: case OP_CLONE_RANGE: case OP_DEDUPE_RANGE: case OP_COPY_RANGE: @@ -2053,6 +2181,9 @@ test(void) case OP_COPY_RANGE: generate_dest_range(true, maxfilelen, &offset, &size, &offset2); break; + case OP_EXCHANGE_RANGE: + generate_dest_range(false, file_size, &offset, &size, &offset2); + break; } have_op: @@ -2096,6 +2227,12 @@ test(void) goto out; } break; + case OP_EXCHANGE_RANGE: + if (!xchg_range_calls) { + log5(op, offset, size, offset2, FL_SKIPPED); + goto out; + } + break; case OP_CLONE_RANGE: if (!clone_range_calls) { log5(op, offset, size, offset2, FL_SKIPPED); @@ -2180,6 +2317,18 @@ test(void) do_insert_range(offset, size); break; + case OP_EXCHANGE_RANGE: + if (size == 0) { + log5(OP_EXCHANGE_RANGE, offset, size, offset2, FL_SKIPPED); + goto out; + } + if (offset2 + size > maxfilelen) { + log5(OP_EXCHANGE_RANGE, offset, size, offset2, FL_SKIPPED); + goto out; + } + + do_xchg_range(offset, size, offset2); + break; case OP_CLONE_RANGE: if (size == 0) { log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED); @@ -2294,6 +2443,9 @@ usage(void) #ifdef HAVE_COPY_FILE_RANGE " -E: Do not use copy range calls\n" #endif +#ifdef FIEXCHANGE_RANGE +" -0: Do not use exchange range calls\n" +#endif " -L: fsxLite - no file creations & no file size changes\n\ -N numops: total # operations to do (default infinity)\n\ -O: use oplen (see -o flag) for every op (default random)\n\ @@ -2608,12 +2760,11 @@ main(int argc, char **argv) page_size = getpagesize(); page_mask = page_size - 1; mmap_mask = page_mask; - setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */ while ((ch = getopt_long(argc, argv, - "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:UWXZ", + "0b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:UWXZ", longopts, NULL)) != EOF) switch (ch) { case 'b': @@ -2747,6 +2898,9 @@ main(int argc, char **argv) case 'I': insert_range_calls = 0; break; + case '0': + xchg_range_calls = 0; + break; case 'J': clone_range_calls = 0; break; @@ -2988,6 +3142,8 @@ main(int argc, char **argv) dedupe_range_calls = test_dedupe_range(); if (copy_range_calls) copy_range_calls = test_copy_range(); + if (xchg_range_calls) + xchg_range_calls = test_xchg_range(); while (numops == -1 || numops--) if (!test()) diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 index e1b381c16f..db663970c2 100644 --- a/m4/package_libcdev.m4 +++ b/m4/package_libcdev.m4 @@ -157,3 +157,24 @@ AC_DEFUN([AC_HAVE_RLIMIT_NOFILE], AC_MSG_RESULT(no)) AC_SUBST(have_rlimit_nofile) ]) + +# +# Check if we have a FIEXCHANGE_RANGE ioctl (Linux) +# +AC_DEFUN([AC_HAVE_FIEXCHANGE], + [ AC_MSG_CHECKING([for FIEXCHANGE_RANGE]) + AC_TRY_LINK([ +#define _GNU_SOURCE +#include <sys/syscall.h> +#include <sys/ioctl.h> +#include <unistd.h> +#include <linux/fs.h> +#include <linux/fiexchange.h> + ], [ + struct file_xchg_range fxr; + ioctl(-1, FIEXCHANGE_RANGE, &fxr); + ], have_fiexchange=yes + AC_MSG_RESULT(yes), + AC_MSG_RESULT(no)) + AC_SUBST(have_fiexchange) + ]) diff --git a/src/fiexchange.h b/src/fiexchange.h new file mode 100644 index 0000000000..29b3ac0ff5 --- /dev/null +++ b/src/fiexchange.h @@ -0,0 +1,101 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later WITH Linux-syscall-note */ +/* + * FIEXCHANGE ioctl definitions, to facilitate exchanging parts of files. + * + * Copyright (C) 2022 Oracle. All Rights Reserved. + * + * Author: Darrick J. Wong <djwong@kernel.org> + */ +#ifndef _LINUX_FIEXCHANGE_H +#define _LINUX_FIEXCHANGE_H + +#include <linux/types.h> + +/* + * Exchange part of file1 with part of the file that this ioctl that is being + * called against (which we'll call file2). Filesystems must be able to + * restart and complete the operation even after the system goes down. + */ +struct file_xchg_range { + __s64 file1_fd; + __s64 file1_offset; /* file1 offset, bytes */ + __s64 file2_offset; /* file2 offset, bytes */ + __s64 length; /* bytes to exchange */ + + __u64 flags; /* see FILE_XCHG_RANGE_* below */ + + /* file2 metadata for optional freshness checks */ + __s64 file2_ino; /* inode number */ + __s64 file2_mtime; /* modification time */ + __s64 file2_ctime; /* change time */ + __s32 file2_mtime_nsec; /* mod time, nsec */ + __s32 file2_ctime_nsec; /* change time, nsec */ + + __u64 pad[6]; /* must be zeroes */ +}; + +/* + * Atomic exchange operations are not required. This relaxes the requirement + * that the filesystem must be able to complete the operation after a crash. + */ +#define FILE_XCHG_RANGE_NONATOMIC (1 << 0) + +/* + * Check that file2's inode number, mtime, and ctime against the values + * provided, and return -EBUSY if there isn't an exact match. + */ +#define FILE_XCHG_RANGE_FILE2_FRESH (1 << 1) + +/* + * Check that the file1's length is equal to file1_offset + length, and that + * file2's length is equal to file2_offset + length. Returns -EDOM if there + * isn't an exact match. + */ +#define FILE_XCHG_RANGE_FULL_FILES (1 << 2) + +/* + * Exchange file data all the way to the ends of both files, and then exchange + * the file sizes. This flag can be used to replace a file's contents with a + * different amount of data. length will be ignored. + */ +#define FILE_XCHG_RANGE_TO_EOF (1 << 3) + +/* Flush all changes in file data and file metadata to disk before returning. */ +#define FILE_XCHG_RANGE_FSYNC (1 << 4) + +/* Dry run; do all the parameter verification but do not change anything. */ +#define FILE_XCHG_RANGE_DRY_RUN (1 << 5) + +/* + * Do not exchange any part of the range where file1's mapping is a hole. This + * can be used to emulate scatter-gather atomic writes with a temp file. + */ +#define FILE_XCHG_RANGE_SKIP_FILE1_HOLES (1 << 6) + +/* + * Commit the contents of file1 into file2 if file2 has the same inode number, + * mtime, and ctime as the arguments provided to the call. The old contents of + * file2 will be moved to file1. + * + * With this flag, all committed information can be retrieved even if the + * system crashes or is rebooted. This includes writing through or flushing a + * disk cache if present. The call blocks until the device reports that the + * commit is complete. + * + * This flag should not be combined with NONATOMIC. It can be combined with + * SKIP_FILE1_HOLES. + */ +#define FILE_XCHG_RANGE_COMMIT (FILE_XCHG_RANGE_FILE2_FRESH | \ + FILE_XCHG_RANGE_FSYNC) + +#define FILE_XCHG_RANGE_ALL_FLAGS (FILE_XCHG_RANGE_NONATOMIC | \ + FILE_XCHG_RANGE_FILE2_FRESH | \ + FILE_XCHG_RANGE_FULL_FILES | \ + FILE_XCHG_RANGE_TO_EOF | \ + FILE_XCHG_RANGE_FSYNC | \ + FILE_XCHG_RANGE_DRY_RUN | \ + FILE_XCHG_RANGE_SKIP_FILE1_HOLES) + +#define FIEXCHANGE_RANGE _IOWR('X', 129, struct file_xchg_range) + +#endif /* _LINUX_FIEXCHANGE_H */ diff --git a/src/global.h b/src/global.h index b44070993c..49570ef117 100644 --- a/src/global.h +++ b/src/global.h @@ -171,6 +171,12 @@ #include <sys/mman.h> #endif +#ifdef HAVE_FIEXCHANGE +# include <linux/fiexchange.h> +#else +# include "fiexchange.h" +#endif + static inline unsigned long long rounddown_64(unsigned long long x, unsigned int y) { ^ permalink raw reply related [flat|nested] 107+ messages in thread
* Re: [PATCH 6/7] fsx: support FIEXCHANGE_RANGE 2022-12-30 22:19 ` [PATCH 6/7] fsx: support FIEXCHANGE_RANGE Darrick J. Wong @ 2023-02-28 1:55 ` Zorro Lang 2023-03-01 2:56 ` Darrick J. Wong 0 siblings, 1 reply; 107+ messages in thread From: Zorro Lang @ 2023-02-28 1:55 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:19:49PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Upgrade fsx to support exchanging file contents. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- Hi Darrick, I've merged most of patches of [NYE DELUGE 2/4], now I'm trying to merge the rest of it this time. This patch will get build warning [1] from autoconf, can you rebase this patch to current for-next branch, and use autoupdate to update the configure.ac and lib/autoconf/general.m4 ? Thanks, Zorro [1] autoconf configure.ac:73: warning: The macro `AC_TRY_LINK' is obsolete. configure.ac:73: You should run autoupdate. ./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from... m4/package_libcdev.m4:161: AC_HAVE_FIEXCHANGE is expanded from... configure.ac:73: the top level ./configure \ --libexecdir=/usr/lib \ --exec_prefix=/var/lib > configure.ac | 1 > include/builddefs.in | 1 > ltp/Makefile | 4 + > ltp/fsx.c | 160 ++++++++++++++++++++++++++++++++++++++++++++++++- > m4/package_libcdev.m4 | 21 ++++++ > src/fiexchange.h | 101 +++++++++++++++++++++++++++++++ > src/global.h | 6 ++ > 7 files changed, 292 insertions(+), 2 deletions(-) > create mode 100644 src/fiexchange.h > > > diff --git a/configure.ac b/configure.ac > index e92bd6b26d..4687d8a3c0 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -70,6 +70,7 @@ AC_HAVE_SEEK_DATA > AC_HAVE_BMV_OF_SHARED > AC_HAVE_NFTW > AC_HAVE_RLIMIT_NOFILE > +AC_HAVE_FIEXCHANGE [snip] > setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */ > > while ((ch = getopt_long(argc, argv, > - "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:UWXZ", > + "0b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:UWXZ", Looks like we nearly used up most of letters for fsx, to avoid some operations. Maybe we can use a single option (e.g. -a means avoid) and suboptions to help that. For example "-a xchg_range,clone_range,dedupe_range" to avoid these 3 operations. Or use long option, e.g. --no-xchg-range, --no-clone-range to replace short ones. What do you think? (Anyway, that's not the problem of this patch) Thanks, Zorro > longopts, NULL)) != EOF) > switch (ch) { > case 'b': > @@ -2747,6 +2898,9 @@ main(int argc, char **argv) > case 'I': > insert_range_calls = 0; > break; > + case '0': > + xchg_range_calls = 0; > + break; > case 'J': > clone_range_calls = 0; > break; > @@ -2988,6 +3142,8 @@ main(int argc, char **argv) > dedupe_range_calls = test_dedupe_range(); > if (copy_range_calls) > copy_range_calls = test_copy_range(); > + if (xchg_range_calls) > + xchg_range_calls = test_xchg_range(); > > while (numops == -1 || numops--) > if (!test()) > diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 > index e1b381c16f..db663970c2 100644 > --- a/m4/package_libcdev.m4 > +++ b/m4/package_libcdev.m4 > @@ -157,3 +157,24 @@ AC_DEFUN([AC_HAVE_RLIMIT_NOFILE], > AC_MSG_RESULT(no)) > AC_SUBST(have_rlimit_nofile) > ]) > + > +# > +# Check if we have a FIEXCHANGE_RANGE ioctl (Linux) > +# > +AC_DEFUN([AC_HAVE_FIEXCHANGE], > + [ AC_MSG_CHECKING([for FIEXCHANGE_RANGE]) > + AC_TRY_LINK([ > +#define _GNU_SOURCE > +#include <sys/syscall.h> > +#include <sys/ioctl.h> > +#include <unistd.h> > +#include <linux/fs.h> > +#include <linux/fiexchange.h> > + ], [ > + struct file_xchg_range fxr; > + ioctl(-1, FIEXCHANGE_RANGE, &fxr); > + ], have_fiexchange=yes > + AC_MSG_RESULT(yes), > + AC_MSG_RESULT(no)) > + AC_SUBST(have_fiexchange) > + ]) > diff --git a/src/fiexchange.h b/src/fiexchange.h > new file mode 100644 > index 0000000000..29b3ac0ff5 > --- /dev/null > +++ b/src/fiexchange.h > @@ -0,0 +1,101 @@ > +/* SPDX-License-Identifier: GPL-2.0-or-later WITH Linux-syscall-note */ > +/* > + * FIEXCHANGE ioctl definitions, to facilitate exchanging parts of files. > + * > + * Copyright (C) 2022 Oracle. All Rights Reserved. > + * > + * Author: Darrick J. Wong <djwong@kernel.org> > + */ > +#ifndef _LINUX_FIEXCHANGE_H > +#define _LINUX_FIEXCHANGE_H > + > +#include <linux/types.h> > + > +/* > + * Exchange part of file1 with part of the file that this ioctl that is being > + * called against (which we'll call file2). Filesystems must be able to > + * restart and complete the operation even after the system goes down. > + */ > +struct file_xchg_range { > + __s64 file1_fd; > + __s64 file1_offset; /* file1 offset, bytes */ > + __s64 file2_offset; /* file2 offset, bytes */ > + __s64 length; /* bytes to exchange */ > + > + __u64 flags; /* see FILE_XCHG_RANGE_* below */ > + > + /* file2 metadata for optional freshness checks */ > + __s64 file2_ino; /* inode number */ > + __s64 file2_mtime; /* modification time */ > + __s64 file2_ctime; /* change time */ > + __s32 file2_mtime_nsec; /* mod time, nsec */ > + __s32 file2_ctime_nsec; /* change time, nsec */ > + > + __u64 pad[6]; /* must be zeroes */ > +}; > + > +/* > + * Atomic exchange operations are not required. This relaxes the requirement > + * that the filesystem must be able to complete the operation after a crash. > + */ > +#define FILE_XCHG_RANGE_NONATOMIC (1 << 0) > + > +/* > + * Check that file2's inode number, mtime, and ctime against the values > + * provided, and return -EBUSY if there isn't an exact match. > + */ > +#define FILE_XCHG_RANGE_FILE2_FRESH (1 << 1) > + > +/* > + * Check that the file1's length is equal to file1_offset + length, and that > + * file2's length is equal to file2_offset + length. Returns -EDOM if there > + * isn't an exact match. > + */ > +#define FILE_XCHG_RANGE_FULL_FILES (1 << 2) > + > +/* > + * Exchange file data all the way to the ends of both files, and then exchange > + * the file sizes. This flag can be used to replace a file's contents with a > + * different amount of data. length will be ignored. > + */ > +#define FILE_XCHG_RANGE_TO_EOF (1 << 3) > + > +/* Flush all changes in file data and file metadata to disk before returning. */ > +#define FILE_XCHG_RANGE_FSYNC (1 << 4) > + > +/* Dry run; do all the parameter verification but do not change anything. */ > +#define FILE_XCHG_RANGE_DRY_RUN (1 << 5) > + > +/* > + * Do not exchange any part of the range where file1's mapping is a hole. This > + * can be used to emulate scatter-gather atomic writes with a temp file. > + */ > +#define FILE_XCHG_RANGE_SKIP_FILE1_HOLES (1 << 6) > + > +/* > + * Commit the contents of file1 into file2 if file2 has the same inode number, > + * mtime, and ctime as the arguments provided to the call. The old contents of > + * file2 will be moved to file1. > + * > + * With this flag, all committed information can be retrieved even if the > + * system crashes or is rebooted. This includes writing through or flushing a > + * disk cache if present. The call blocks until the device reports that the > + * commit is complete. > + * > + * This flag should not be combined with NONATOMIC. It can be combined with > + * SKIP_FILE1_HOLES. > + */ > +#define FILE_XCHG_RANGE_COMMIT (FILE_XCHG_RANGE_FILE2_FRESH | \ > + FILE_XCHG_RANGE_FSYNC) > + > +#define FILE_XCHG_RANGE_ALL_FLAGS (FILE_XCHG_RANGE_NONATOMIC | \ > + FILE_XCHG_RANGE_FILE2_FRESH | \ > + FILE_XCHG_RANGE_FULL_FILES | \ > + FILE_XCHG_RANGE_TO_EOF | \ > + FILE_XCHG_RANGE_FSYNC | \ > + FILE_XCHG_RANGE_DRY_RUN | \ > + FILE_XCHG_RANGE_SKIP_FILE1_HOLES) > + > +#define FIEXCHANGE_RANGE _IOWR('X', 129, struct file_xchg_range) > + > +#endif /* _LINUX_FIEXCHANGE_H */ > diff --git a/src/global.h b/src/global.h > index b44070993c..49570ef117 100644 > --- a/src/global.h > +++ b/src/global.h > @@ -171,6 +171,12 @@ > #include <sys/mman.h> > #endif > > +#ifdef HAVE_FIEXCHANGE > +# include <linux/fiexchange.h> > +#else > +# include "fiexchange.h" > +#endif > + > static inline unsigned long long > rounddown_64(unsigned long long x, unsigned int y) > { > ^ permalink raw reply [flat|nested] 107+ messages in thread
* Re: [PATCH 6/7] fsx: support FIEXCHANGE_RANGE 2023-02-28 1:55 ` Zorro Lang @ 2023-03-01 2:56 ` Darrick J. Wong 0 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2023-03-01 2:56 UTC (permalink / raw) To: Zorro Lang; +Cc: linux-xfs, fstests On Tue, Feb 28, 2023 at 09:55:28AM +0800, Zorro Lang wrote: > On Fri, Dec 30, 2022 at 02:19:49PM -0800, Darrick J. Wong wrote: > > From: Darrick J. Wong <djwong@kernel.org> > > > > Upgrade fsx to support exchanging file contents. > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > --- > > Hi Darrick, > > I've merged most of patches of [NYE DELUGE 2/4], now I'm trying to merge > the rest of it this time. > > This patch will get build warning [1] from autoconf, can you rebase this patch > to current for-next branch, and use autoupdate to update the configure.ac > and lib/autoconf/general.m4 ? Will do. Thanks for merging this all, I really appreciate it! --D > Thanks, > Zorro > > [1] > autoconf > configure.ac:73: warning: The macro `AC_TRY_LINK' is obsolete. > configure.ac:73: You should run autoupdate. > ./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from... > m4/package_libcdev.m4:161: AC_HAVE_FIEXCHANGE is expanded from... > configure.ac:73: the top level > ./configure \ > --libexecdir=/usr/lib \ > --exec_prefix=/var/lib > > > configure.ac | 1 > > include/builddefs.in | 1 > > ltp/Makefile | 4 + > > ltp/fsx.c | 160 ++++++++++++++++++++++++++++++++++++++++++++++++- > > m4/package_libcdev.m4 | 21 ++++++ > > src/fiexchange.h | 101 +++++++++++++++++++++++++++++++ > > src/global.h | 6 ++ > > 7 files changed, 292 insertions(+), 2 deletions(-) > > create mode 100644 src/fiexchange.h > > > > > > diff --git a/configure.ac b/configure.ac > > index e92bd6b26d..4687d8a3c0 100644 > > --- a/configure.ac > > +++ b/configure.ac > > @@ -70,6 +70,7 @@ AC_HAVE_SEEK_DATA > > AC_HAVE_BMV_OF_SHARED > > AC_HAVE_NFTW > > AC_HAVE_RLIMIT_NOFILE > > +AC_HAVE_FIEXCHANGE > > [snip] > > > setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */ > > > > while ((ch = getopt_long(argc, argv, > > - "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:UWXZ", > > + "0b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:UWXZ", > > Looks like we nearly used up most of letters for fsx, to avoid some operations. > > Maybe we can use a single option (e.g. -a means avoid) and suboptions to > help that. For example "-a xchg_range,clone_range,dedupe_range" to avoid > these 3 operations. Or use long option, e.g. --no-xchg-range, --no-clone-range > to replace short ones. > > What do you think? (Anyway, that's not the problem of this patch) > > Thanks, > Zorro > > > longopts, NULL)) != EOF) > > switch (ch) { > > case 'b': > > @@ -2747,6 +2898,9 @@ main(int argc, char **argv) > > case 'I': > > insert_range_calls = 0; > > break; > > + case '0': > > + xchg_range_calls = 0; > > + break; > > case 'J': > > clone_range_calls = 0; > > break; > > @@ -2988,6 +3142,8 @@ main(int argc, char **argv) > > dedupe_range_calls = test_dedupe_range(); > > if (copy_range_calls) > > copy_range_calls = test_copy_range(); > > + if (xchg_range_calls) > > + xchg_range_calls = test_xchg_range(); > > > > while (numops == -1 || numops--) > > if (!test()) > > diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 > > index e1b381c16f..db663970c2 100644 > > --- a/m4/package_libcdev.m4 > > +++ b/m4/package_libcdev.m4 > > @@ -157,3 +157,24 @@ AC_DEFUN([AC_HAVE_RLIMIT_NOFILE], > > AC_MSG_RESULT(no)) > > AC_SUBST(have_rlimit_nofile) > > ]) > > + > > +# > > +# Check if we have a FIEXCHANGE_RANGE ioctl (Linux) > > +# > > +AC_DEFUN([AC_HAVE_FIEXCHANGE], > > + [ AC_MSG_CHECKING([for FIEXCHANGE_RANGE]) > > + AC_TRY_LINK([ > > +#define _GNU_SOURCE > > +#include <sys/syscall.h> > > +#include <sys/ioctl.h> > > +#include <unistd.h> > > +#include <linux/fs.h> > > +#include <linux/fiexchange.h> > > + ], [ > > + struct file_xchg_range fxr; > > + ioctl(-1, FIEXCHANGE_RANGE, &fxr); > > + ], have_fiexchange=yes > > + AC_MSG_RESULT(yes), > > + AC_MSG_RESULT(no)) > > + AC_SUBST(have_fiexchange) > > + ]) > > diff --git a/src/fiexchange.h b/src/fiexchange.h > > new file mode 100644 > > index 0000000000..29b3ac0ff5 > > --- /dev/null > > +++ b/src/fiexchange.h > > @@ -0,0 +1,101 @@ > > +/* SPDX-License-Identifier: GPL-2.0-or-later WITH Linux-syscall-note */ > > +/* > > + * FIEXCHANGE ioctl definitions, to facilitate exchanging parts of files. > > + * > > + * Copyright (C) 2022 Oracle. All Rights Reserved. > > + * > > + * Author: Darrick J. Wong <djwong@kernel.org> > > + */ > > +#ifndef _LINUX_FIEXCHANGE_H > > +#define _LINUX_FIEXCHANGE_H > > + > > +#include <linux/types.h> > > + > > +/* > > + * Exchange part of file1 with part of the file that this ioctl that is being > > + * called against (which we'll call file2). Filesystems must be able to > > + * restart and complete the operation even after the system goes down. > > + */ > > +struct file_xchg_range { > > + __s64 file1_fd; > > + __s64 file1_offset; /* file1 offset, bytes */ > > + __s64 file2_offset; /* file2 offset, bytes */ > > + __s64 length; /* bytes to exchange */ > > + > > + __u64 flags; /* see FILE_XCHG_RANGE_* below */ > > + > > + /* file2 metadata for optional freshness checks */ > > + __s64 file2_ino; /* inode number */ > > + __s64 file2_mtime; /* modification time */ > > + __s64 file2_ctime; /* change time */ > > + __s32 file2_mtime_nsec; /* mod time, nsec */ > > + __s32 file2_ctime_nsec; /* change time, nsec */ > > + > > + __u64 pad[6]; /* must be zeroes */ > > +}; > > + > > +/* > > + * Atomic exchange operations are not required. This relaxes the requirement > > + * that the filesystem must be able to complete the operation after a crash. > > + */ > > +#define FILE_XCHG_RANGE_NONATOMIC (1 << 0) > > + > > +/* > > + * Check that file2's inode number, mtime, and ctime against the values > > + * provided, and return -EBUSY if there isn't an exact match. > > + */ > > +#define FILE_XCHG_RANGE_FILE2_FRESH (1 << 1) > > + > > +/* > > + * Check that the file1's length is equal to file1_offset + length, and that > > + * file2's length is equal to file2_offset + length. Returns -EDOM if there > > + * isn't an exact match. > > + */ > > +#define FILE_XCHG_RANGE_FULL_FILES (1 << 2) > > + > > +/* > > + * Exchange file data all the way to the ends of both files, and then exchange > > + * the file sizes. This flag can be used to replace a file's contents with a > > + * different amount of data. length will be ignored. > > + */ > > +#define FILE_XCHG_RANGE_TO_EOF (1 << 3) > > + > > +/* Flush all changes in file data and file metadata to disk before returning. */ > > +#define FILE_XCHG_RANGE_FSYNC (1 << 4) > > + > > +/* Dry run; do all the parameter verification but do not change anything. */ > > +#define FILE_XCHG_RANGE_DRY_RUN (1 << 5) > > + > > +/* > > + * Do not exchange any part of the range where file1's mapping is a hole. This > > + * can be used to emulate scatter-gather atomic writes with a temp file. > > + */ > > +#define FILE_XCHG_RANGE_SKIP_FILE1_HOLES (1 << 6) > > + > > +/* > > + * Commit the contents of file1 into file2 if file2 has the same inode number, > > + * mtime, and ctime as the arguments provided to the call. The old contents of > > + * file2 will be moved to file1. > > + * > > + * With this flag, all committed information can be retrieved even if the > > + * system crashes or is rebooted. This includes writing through or flushing a > > + * disk cache if present. The call blocks until the device reports that the > > + * commit is complete. > > + * > > + * This flag should not be combined with NONATOMIC. It can be combined with > > + * SKIP_FILE1_HOLES. > > + */ > > +#define FILE_XCHG_RANGE_COMMIT (FILE_XCHG_RANGE_FILE2_FRESH | \ > > + FILE_XCHG_RANGE_FSYNC) > > + > > +#define FILE_XCHG_RANGE_ALL_FLAGS (FILE_XCHG_RANGE_NONATOMIC | \ > > + FILE_XCHG_RANGE_FILE2_FRESH | \ > > + FILE_XCHG_RANGE_FULL_FILES | \ > > + FILE_XCHG_RANGE_TO_EOF | \ > > + FILE_XCHG_RANGE_FSYNC | \ > > + FILE_XCHG_RANGE_DRY_RUN | \ > > + FILE_XCHG_RANGE_SKIP_FILE1_HOLES) > > + > > +#define FIEXCHANGE_RANGE _IOWR('X', 129, struct file_xchg_range) > > + > > +#endif /* _LINUX_FIEXCHANGE_H */ > > diff --git a/src/global.h b/src/global.h > > index b44070993c..49570ef117 100644 > > --- a/src/global.h > > +++ b/src/global.h > > @@ -171,6 +171,12 @@ > > #include <sys/mman.h> > > #endif > > > > +#ifdef HAVE_FIEXCHANGE > > +# include <linux/fiexchange.h> > > +#else > > +# include "fiexchange.h" > > +#endif > > + > > static inline unsigned long long > > rounddown_64(unsigned long long x, unsigned int y) > > { > > > ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/1] fstests: online repair of realtime summaries 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (13 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online repair of realtime summary files Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of extended attributes Darrick J. Wong ` (2 subsequent siblings) 17 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, We now have all the infrastructure we need to repair file metadata. We'll begin with the realtime summary file, because it is the least complex data structure. To support this we need to add three more pieces to the temporary file code from the previous patchset -- preallocating space in the temp file, formatting metadata into that space and writing the blocks to disk, and swapping the fork mappings atomically. After that, the actual reconstruction of the realtime summary information is pretty simple, since we can simply write the incore copy computed by the rtsummary scrubber to the temporary file, swap the contents, and reap the old blocks. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-rtsummary xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-rtsummary fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-rtsummary --- tests/xfs/813 | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/813.out | 2 ++ 2 files changed, 50 insertions(+) create mode 100755 tests/xfs/813 create mode 100644 tests/xfs/813.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/1] xfs: race fsstress with online repair of realtime summary files 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of realtime summaries Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 0 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create tests to race fsstress with rt summary file repair while running fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/813 | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/813.out | 2 ++ 2 files changed, 50 insertions(+) create mode 100755 tests/xfs/813 create mode 100644 tests/xfs/813.out diff --git a/tests/xfs/813 b/tests/xfs/813 new file mode 100755 index 0000000000..5efe923c75 --- /dev/null +++ b/tests/xfs/813 @@ -0,0 +1,48 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 813 +# +# Race fsstress and realtime summary repair for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime +_xfs_force_bdev realtime $SCRATCH_MNT + +# XXX the realtime summary scrubber isn't currently implemented upstream. +# Don't bother trying to fix it on those kernels +$XFS_IO_PROG -c 'scrub rtsummary' -c 'scrub rtsummary' "$SCRATCH_MNT" 2>&1 | \ + grep -q 'Scan was not complete' && \ + _notrun "rtsummary scrub is incomplete" + +_scratch_xfs_stress_online_repair -s "repair rtsummary" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/813.out b/tests/xfs/813.out new file mode 100644 index 0000000000..f0c2a12bea --- /dev/null +++ b/tests/xfs/813.out @@ -0,0 +1,2 @@ +QA output created by 813 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/1] fstests: online repair of extended attributes 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (14 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of realtime summaries Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online repair of extended attribute data Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of directories Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v24.0 0/1] fstests: test automatic scrub optimization by default Darrick J. Wong 17 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, This series employs atomic extent swapping to enable safe reconstruction of extended attribute data attached to a file. Because xattrs do not have any redundant information to draw off of, we can at best salvage as much data as we can and build a new structure. Rebuilding an extended attribute structure consists of these three steps: First, we walk the existing attributes to salvage as many of them as we can, by adding them as new attributes attached to the repair tempfile. We need to add a new xfile-based data structure to hold blobs of arbitrary length to stage the xattr names and values. Second, we write the salvaged attributes to a temporary file, and use atomic extent swaps to exchange the entire attribute fork between the two files. Finally, we reap the old xattr blocks (which are now in the temporary file) as carefully as we can. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-xattrs xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-xattrs fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-xattrs --- tests/xfs/814 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/814.out | 2 ++ 2 files changed, 42 insertions(+) create mode 100755 tests/xfs/814 create mode 100644 tests/xfs/814.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/1] xfs: race fsstress with online repair of extended attribute data 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of extended attributes Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 0 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create tests to race fsstress with extended attribute repair while running fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/814 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/814.out | 2 ++ 2 files changed, 42 insertions(+) create mode 100755 tests/xfs/814 create mode 100644 tests/xfs/814.out diff --git a/tests/xfs/814 b/tests/xfs/814 new file mode 100755 index 0000000000..96abb13691 --- /dev/null +++ b/tests/xfs/814 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 814 +# +# Race fsstress and extended attributes repair for a while to see if we crash +# or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs +. ./common/attr + +# real QA test starts here +_supported_fs xfs +_require_attrs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -x 'xattr' -s "repair xattr" -t "%attrfile%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/814.out b/tests/xfs/814.out new file mode 100644 index 0000000000..95532d3da7 --- /dev/null +++ b/tests/xfs/814.out @@ -0,0 +1,2 @@ +QA output created by 814 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/2] fstests: online repair of directories 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (15 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of extended attributes Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] xfs: ensure that online directory repairs don't hit EDQUOT Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] xfs: race fsstress with online repair of dirs and parent pointers Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v24.0 0/1] fstests: test automatic scrub optimization by default Darrick J. Wong 17 siblings, 2 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, This series employs atomic extent swapping to enable safe reconstruction of directory data. For now, XFS does not support reverse directory links (aka parent pointers), so we can only salvage the dirents of a directory and construct a new structure. Directory repair therefore consists of five main parts: First, we walk the existing directory to salvage as many entries as we can, by adding them as new directory entries to the repair temp dir. Second, we validate the parent pointer found in the directory. If one was not found, we scan the entire filesystem looking for a potential parent. Third, we use atomic extent swaps to exchange the entire data fork between the two directories. Fourth, we reap the old directory blocks as carefully as we can. To wrap up the directory repair code, we need to add to the regular filesystem the ability to free all the data fork blocks in a directory. This does not change anything with normal directories, since they must still unlink and shrink one entry at a time. However, this will facilitate freeing of partially-inactivated temporary directories during log recovery. The second half of this patchset implements repairs for the dotdot entries of directories. For now there is only rudimentary support for this, because there are no directory parent pointers, so the best we can do is scanning the filesystem and the VFS dcache for answers. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=repair-dirs xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=repair-dirs fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=repair-dirs --- tests/xfs/815 | 37 +++++++++++++++++++++++++ tests/xfs/815.out | 2 + tests/xfs/816 | 38 ++++++++++++++++++++++++++ tests/xfs/816.out | 2 + tests/xfs/841 | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/841.out | 3 ++ 6 files changed, 159 insertions(+) create mode 100755 tests/xfs/815 create mode 100644 tests/xfs/815.out create mode 100755 tests/xfs/816 create mode 100644 tests/xfs/816.out create mode 100755 tests/xfs/841 create mode 100644 tests/xfs/841.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/2] xfs: ensure that online directory repairs don't hit EDQUOT 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of directories Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] xfs: race fsstress with online repair of dirs and parent pointers Darrick J. Wong 1 sibling, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add a test to ensure that the sysadmin doesn't get EDQUOT if they try to repair directory metadata when we've already exceeded a quota limit somewhere. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/841 | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/841.out | 3 ++ 2 files changed, 80 insertions(+) create mode 100755 tests/xfs/841 create mode 100644 tests/xfs/841.out diff --git a/tests/xfs/841 b/tests/xfs/841 new file mode 100755 index 0000000000..f743454971 --- /dev/null +++ b/tests/xfs/841 @@ -0,0 +1,77 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test 841 +# +# Ensure that the sysadmin won't hit EDQUOT while repairing file data contents +# even if the file's quota limits have been exceeded. This tests the quota +# reservation handling inside the swapext code used by repair. +# +. ./common/preamble +_begin_fstest online_repair + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/quota + +# real QA test starts here + +# Modify as appropriate. +_supported_fs xfs +_require_quota +_require_user +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_qmount_option usrquota +_qmount + +blocksize=$(_get_block_size $SCRATCH_MNT) +alloc_unit=$(_xfs_get_dir_blocksize $SCRATCH_MNT) + +# Make sure we can actually repair a directory +scratchdir=$SCRATCH_MNT/dir +scratchfile=$SCRATCH_MNT/file +mkdir $scratchdir +touch $scratchfile +$XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT +__stress_scrub_check_commands "$scratchdir" "" 'repair directory' + +# Create a 2-dirblock directory +total_size=$((alloc_unit * 2)) +dirents=$((total_size / 255)) + +for ((i = 0; i < dirents; i++)); do + name=$(printf "x%0254d" $i) + ln $scratchfile $scratchdir/$name +done + +# Set a low quota hardlimit for an unprivileged uid and chown the files to it +echo "set up quota" >> $seqres.full +$XFS_QUOTA_PROG -x -c "limit -u bhard=$alloc_unit $qa_user" $SCRATCH_MNT +chown $qa_user $scratchdir $scratchfile +$XFS_QUOTA_PROG -x -c 'report -u' $SCRATCH_MNT >> $seqres.full + +# Rebuild the directory +echo "repairs" >> $seqres.full +$XFS_IO_PROG -x -c 'inject force_repair' -c 'repair directory' $scratchdir +$XFS_QUOTA_PROG -x -c 'report -u' $SCRATCH_MNT >> $seqres.full + +# Fail at appending the directory as qa_user to ensure quota enforcement works +echo "fail quota" >> $seqres.full +for ((i = 0; i < dirents; i++)); do + name=$(printf "y%0254d" $i) + su - "$qa_user" -c "ln $scratchfile $scratchdir/$name" 2>&1 | \ + _filter_scratch | sed -e 's/y[0-9]*/yXXX/g' + test "${PIPESTATUS[0]}" -ne 0 && break +done +$XFS_QUOTA_PROG -x -c 'report -u' $SCRATCH_MNT >> $seqres.full + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/841.out b/tests/xfs/841.out new file mode 100644 index 0000000000..e8e169111a --- /dev/null +++ b/tests/xfs/841.out @@ -0,0 +1,3 @@ +QA output created by 841 +ln: failed to create hard link 'SCRATCH_MNT/dir/yXXX': Disk quota exceeded +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCH 2/2] xfs: race fsstress with online repair of dirs and parent pointers 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of directories Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] xfs: ensure that online directory repairs don't hit EDQUOT Darrick J. Wong @ 2022-12-30 22:19 ` Darrick J. Wong 1 sibling, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:19 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create tests to race fsstress with directories and directory parent pointer repair while running fsstress in the background. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/815 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/815.out | 2 ++ tests/xfs/816 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/816.out | 2 ++ 4 files changed, 79 insertions(+) create mode 100755 tests/xfs/815 create mode 100644 tests/xfs/815.out create mode 100755 tests/xfs/816 create mode 100644 tests/xfs/816.out diff --git a/tests/xfs/815 b/tests/xfs/815 new file mode 100755 index 0000000000..745afec792 --- /dev/null +++ b/tests/xfs/815 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 815 +# +# Race fsstress and directory repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -x 'dir' -s "repair directory" -t "%dir%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/815.out b/tests/xfs/815.out new file mode 100644 index 0000000000..6ea462f3f7 --- /dev/null +++ b/tests/xfs/815.out @@ -0,0 +1,2 @@ +QA output created by 815 +Silence is golden diff --git a/tests/xfs/816 b/tests/xfs/816 new file mode 100755 index 0000000000..25a79005f8 --- /dev/null +++ b/tests/xfs/816 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 816 +# +# Race fsstress and parent pointers repair for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_scratch_xfs_stress_online_repair -s "repair parent" -t "%dir%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/816.out b/tests/xfs/816.out new file mode 100644 index 0000000000..a9d8f943c8 --- /dev/null +++ b/tests/xfs/816.out @@ -0,0 +1,2 @@ +QA output created by 816 +Silence is golden ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCHSET v24.0 0/1] fstests: test automatic scrub optimization by default 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong ` (16 preceding siblings ...) 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of directories Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/1] xfs: test xfs_scrub dry run, preen, and repair mode Darrick J. Wong 17 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, This final patchset in the online fsck series enables the background service to optimize filesystems by default. This is the first step towards enabling repairs by default. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-optimize-by-default fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-optimize-by-default --- tests/xfs/850 | 105 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/850.out | 32 ++++++++++++++++ 2 files changed, 137 insertions(+) create mode 100755 tests/xfs/850 create mode 100644 tests/xfs/850.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 1/1] xfs: test xfs_scrub dry run, preen, and repair mode 2022-12-30 22:20 ` [PATCHSET v24.0 0/1] fstests: test automatic scrub optimization by default Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 0 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> For each of the three operational modes of xfs_scrub, make sure that we /only/ repair that which we're supposed to. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/850 | 105 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/850.out | 32 ++++++++++++++++ 2 files changed, 137 insertions(+) create mode 100755 tests/xfs/850 create mode 100644 tests/xfs/850.out diff --git a/tests/xfs/850 b/tests/xfs/850 new file mode 100755 index 0000000000..bb46915c89 --- /dev/null +++ b/tests/xfs/850 @@ -0,0 +1,105 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test 850 +# +# Make sure that xfs_scrub dry run, preen, and repair modes only modify the +# things that they're allowed to touch. +# +. ./common/preamble +_begin_fstest auto quick online_repair + +# Import common functions. +. ./common/fuzzy +. ./common/filter + +# real QA test starts here + +# Modify as appropriate. +_supported_fs xfs +_require_scratch_nocheck +_require_scrub +_require_xfs_db_command "fuzz" +_require_xfs_io_command "repair" + +# Make sure we support repair? +output="$($XFS_IO_PROG -x -c 'repair -R probe' $SCRATCH_MNT 2>&1)" +test -z "$output" && _notrun 'kernel does not support repair' + +# Make sure xfs_scrub is new enough to support -p(reen) +$XFS_SCRUB_PROG -p 2>&1 | grep -q 'invalid option' && \ + _notrun 'scrub does not support -p' + +_scratch_mkfs | _filter_mkfs 2>$tmp.mkfs >/dev/null +. $tmp.mkfs + +test $agcount -ge 3 || _notrun 'filesystem must have at least 3 AGs' + +AWK_PROG=' +{ + if ($1 ~ /Optimized:/) + optimized++; + else if ($1 ~ /Repaired:/) + repaired++; + else if ($1 ~ /Corruption:/) + corruption++; +} +END { + printf("corruption: %u optimized: %u repaired: %u\n", + corruption, optimized, repaired); +} +' + +test_scrub() { + local mode="$1" + local scrub_arg="$2" + local db_args=(-x) + + # Fuzz secondary superblocks because this won't cause mount failures + if [[ $mode =~ c ]]; then + db_args+=(-c 'sb 1' -c 'fuzz -d dblocks add') + fi + if [[ $mode =~ o ]]; then + db_args+=(-c 'sb 2' -c 'fuzz -d fname random') + fi + + echo "testing mode? $mode scrub_arg $scrub_arg" + echo "db_args:${db_args[@]}:scrub_arg:$scrub_arg:$mode:" >> $seqres.full + echo "----------------" >> $seqres.full + + _scratch_mkfs >> $seqres.full + + # Make sure there's nothing left to optimize, at least according to + # xfs_scrub. This clears the way for us to make targeted changes to + # the filesystem. + _scratch_mount + _scratch_scrub $scrub_arg >> /dev/null + _scratch_unmount + + # Modify the filesystem as needed to trip up xfs_scrub + _scratch_xfs_db "${db_args[@]}" >> $seqres.full + + # See how many optimizations, repairs, and corruptions it'll report + _scratch_mount + _scratch_scrub $scrub_arg 2>&1 | awk "$AWK_PROG" + test "${PIPESTATUS[0]}" -eq 0 || echo "xfs_scrub returned ${PIPESTATUS[0]}?" + echo + _scratch_unmount +} + +test_scrub 'o' -n # dry run with possible optimizations +test_scrub 'o' -p # preen +test_scrub 'o' # repair + +test_scrub 'co' -n # dry run with corruptions and optimizations +test_scrub 'co' -p # preen +test_scrub 'co' # repair + +test_scrub 'c' -n # dry run with corruptions +test_scrub 'c' -p # preen +test_scrub 'c' # repair + +# success, all done +status=0 +exit diff --git a/tests/xfs/850.out b/tests/xfs/850.out new file mode 100644 index 0000000000..6e43fd919d --- /dev/null +++ b/tests/xfs/850.out @@ -0,0 +1,32 @@ +QA output created by 850 +testing mode? o scrub_arg -n +corruption: 0 optimized: 0 repaired: 0 + +testing mode? o scrub_arg -p +corruption: 0 optimized: 1 repaired: 0 + +testing mode? o scrub_arg +corruption: 0 optimized: 1 repaired: 0 + +testing mode? co scrub_arg -n +corruption: 1 optimized: 0 repaired: 0 +xfs_scrub returned 1? + +testing mode? co scrub_arg -p +corruption: 1 optimized: 0 repaired: 0 +xfs_scrub returned 1? + +testing mode? co scrub_arg +corruption: 0 optimized: 1 repaired: 1 + +testing mode? c scrub_arg -n +corruption: 1 optimized: 0 repaired: 0 +xfs_scrub returned 1? + +testing mode? c scrub_arg -p +corruption: 1 optimized: 0 repaired: 0 +xfs_scrub returned 1? + +testing mode? c scrub_arg +corruption: 0 optimized: 0 repaired: 1 + ^ permalink raw reply related [flat|nested] 107+ messages in thread
* [PATCHSET v24.2 0/7] fstests: atomic file updates @ 2023-03-01 2:59 Darrick J. Wong 2023-03-01 2:59 ` [PATCH 6/7] fsx: support FIEXCHANGE_RANGE Darrick J. Wong 0 siblings, 1 reply; 107+ messages in thread From: Darrick J. Wong @ 2023-03-01 2:59 UTC (permalink / raw) To: djwong, zlang; +Cc: linux-xfs, fstests, guan Hi all, This series creates a new FIEXCHANGE_RANGE system call to exchange ranges of bytes between two files atomically. This new functionality enables data storage programs to stage and commit file updates such that reader programs will see either the old contents or the new contents in their entirety, with no chance of torn writes. A successful call completion guarantees that the new contents will be seen even if the system fails. The ability to swap extent mappings between files in this manner is critical to supporting online filesystem repair, which is built upon the strategy of constructing a clean copy of a damaged structure and committing the new structure into the metadata file atomically. User programs will be able to update files atomically by opening an O_TMPFILE, reflinking the source file to it, making whatever updates they want to make, and exchange the relevant ranges of the temp file with the original file. If the updates are aligned with the file block size, a new (since v2) flag provides for exchanging only the written areas. Callers can arrange for the update to be rejected if the original file has been changed. The intent behind this new userspace functionality is to enable atomic rewrites of arbitrary parts of individual files. For years, application programmers wanting to ensure the atomicity of a file update had to write the changes to a new file in the same directory, fsync the new file, rename the new file on top of the old filename, and then fsync the directory. People get it wrong all the time, and $fs hacks abound. The reference implementation in XFS creates a new log incompat feature and log intent items to track high level progress of swapping ranges of two files and finish interrupted work if the system goes down. Sample code can be found in the corresponding changes to xfs_io to exercise the use case mentioned above. Note that this function is /not/ the O_DIRECT atomic file writes concept that has also been floating around for years. This RFC is constructed entirely in software, which means that there are no limitations other than the general filesystem limits. As a side note, the original motivation behind the kernel functionality is online repair of file-based metadata. The atomic file swap is implemented as an atomic inode fork swap, which means that we can implement online reconstruction of extended attributes and directories by building a new one in another inode and atomically swap the contents. Subsequent patchsets adapt the online filesystem repair code to use atomic extent swapping. This enables repair functions to construct a clean copy of a directory, xattr information, symbolic links, realtime bitmaps, and realtime summary information in a temporary inode. If this completes successfully, the new contents can be swapped atomically into the inode being repaired. This is essential to avoid making corruption problems worse if the system goes down in the middle of running repair. This patchset also ports the old XFS extent swap ioctl interface to use the new extent swap code. For userspace, this series also includes the userspace pieces needed to test the new functionality, and a sample implementation of atomic file updates. Question: Should we really bother with fsdevel bikeshedding? Most filesystems cannot support this functionality, so we could keep it private to XFS for now. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=atomic-file-updates xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=atomic-file-updates fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=atomic-file-updates xfsdocs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=atomic-file-updates --- common/rc | 24 +++++++ common/reflink | 33 +++++++++ common/xfs | 19 +++++ configure.ac | 1 doc/group-names.txt | 2 + include/builddefs.in | 1 ltp/Makefile | 4 + ltp/fsstress.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++ ltp/fsx.c | 160 +++++++++++++++++++++++++++++++++++++++++++++- m4/package_libcdev.m4 | 20 ++++++ src/fiexchange.h | 101 +++++++++++++++++++++++++++++ src/global.h | 6 ++ tests/generic/1200 | 55 ++++++++++++++++ tests/generic/1200.out | 3 + tests/generic/1201 | 53 +++++++++++++++ tests/generic/1201.out | 4 + tests/generic/1202 | 47 +++++++++++++ tests/generic/1202.out | 2 + tests/generic/1203 | 58 +++++++++++++++++ tests/generic/1203.out | 2 + tests/generic/1204 | 100 +++++++++++++++++++++++++++++ tests/generic/1204.out | 86 +++++++++++++++++++++++++ tests/generic/1205 | 116 +++++++++++++++++++++++++++++++++ tests/generic/1205.out | 90 ++++++++++++++++++++++++++ tests/generic/1206 | 76 ++++++++++++++++++++++ tests/generic/1206.out | 32 +++++++++ tests/generic/1207 | 122 +++++++++++++++++++++++++++++++++++ tests/generic/1207.out | 48 ++++++++++++++ tests/generic/1209 | 101 +++++++++++++++++++++++++++++ tests/generic/1209.out | 33 +++++++++ tests/generic/1210 | 48 ++++++++++++++ tests/generic/1210.out | 6 ++ tests/generic/1211 | 105 ++++++++++++++++++++++++++++++ tests/generic/1211.out | 40 +++++++++++ tests/generic/1212 | 58 +++++++++++++++++ tests/generic/1212.out | 2 + tests/generic/1213 | 126 ++++++++++++++++++++++++++++++++++++ tests/generic/1213.out | 48 ++++++++++++++ tests/generic/1214 | 63 ++++++++++++++++++ tests/generic/1214.out | 2 + tests/generic/1215 | 70 ++++++++++++++++++++ tests/generic/1215.out | 2 + tests/generic/1216 | 53 +++++++++++++++ tests/generic/1216.out | 6 ++ tests/generic/1217 | 56 ++++++++++++++++ tests/generic/1217.out | 4 + tests/generic/1218 | 115 +++++++++++++++++++++++++++++++++ tests/generic/1218.out | 49 ++++++++++++++ tests/generic/1219 | 83 ++++++++++++++++++++++++ tests/generic/1219.out | 17 +++++ tests/xfs/1202 | 59 +++++++++++++++++ tests/xfs/1202.out | 12 +++ tests/xfs/1208 | 62 ++++++++++++++++++ tests/xfs/1208.out | 10 +++ tests/xfs/1211 | 59 +++++++++++++++++ tests/xfs/1211.out | 7 ++ tests/xfs/1212 | 61 +++++++++++++++++ tests/xfs/1212.out | 5 + tests/xfs/122.out | 3 + tests/xfs/537 | 2 - 60 files changed, 2797 insertions(+), 3 deletions(-) create mode 100644 src/fiexchange.h create mode 100755 tests/generic/1200 create mode 100644 tests/generic/1200.out create mode 100755 tests/generic/1201 create mode 100644 tests/generic/1201.out create mode 100755 tests/generic/1202 create mode 100644 tests/generic/1202.out create mode 100755 tests/generic/1203 create mode 100644 tests/generic/1203.out create mode 100755 tests/generic/1204 create mode 100644 tests/generic/1204.out create mode 100755 tests/generic/1205 create mode 100644 tests/generic/1205.out create mode 100755 tests/generic/1206 create mode 100644 tests/generic/1206.out create mode 100755 tests/generic/1207 create mode 100644 tests/generic/1207.out create mode 100755 tests/generic/1209 create mode 100644 tests/generic/1209.out create mode 100755 tests/generic/1210 create mode 100644 tests/generic/1210.out create mode 100755 tests/generic/1211 create mode 100644 tests/generic/1211.out create mode 100755 tests/generic/1212 create mode 100644 tests/generic/1212.out create mode 100755 tests/generic/1213 create mode 100644 tests/generic/1213.out create mode 100755 tests/generic/1214 create mode 100644 tests/generic/1214.out create mode 100755 tests/generic/1215 create mode 100644 tests/generic/1215.out create mode 100755 tests/generic/1216 create mode 100644 tests/generic/1216.out create mode 100755 tests/generic/1217 create mode 100644 tests/generic/1217.out create mode 100755 tests/generic/1218 create mode 100644 tests/generic/1218.out create mode 100755 tests/generic/1219 create mode 100755 tests/generic/1219.out create mode 100755 tests/xfs/1202 create mode 100644 tests/xfs/1202.out create mode 100755 tests/xfs/1208 create mode 100644 tests/xfs/1208.out create mode 100755 tests/xfs/1211 create mode 100644 tests/xfs/1211.out create mode 100755 tests/xfs/1212 create mode 100644 tests/xfs/1212.out ^ permalink raw reply [flat|nested] 107+ messages in thread
* [PATCH 6/7] fsx: support FIEXCHANGE_RANGE 2023-03-01 2:59 [PATCHSET v24.2 0/7] fstests: atomic file updates Darrick J. Wong @ 2023-03-01 2:59 ` Darrick J. Wong 0 siblings, 0 replies; 107+ messages in thread From: Darrick J. Wong @ 2023-03-01 2:59 UTC (permalink / raw) To: djwong, zlang; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Upgrade fsx to support exchanging file contents. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- configure.ac | 1 include/builddefs.in | 1 ltp/Makefile | 4 + ltp/fsx.c | 160 ++++++++++++++++++++++++++++++++++++++++++++++++- m4/package_libcdev.m4 | 20 ++++++ src/fiexchange.h | 101 +++++++++++++++++++++++++++++++ src/global.h | 6 ++ 7 files changed, 291 insertions(+), 2 deletions(-) create mode 100644 src/fiexchange.h diff --git a/configure.ac b/configure.ac index e92bd6b26d..4687d8a3c0 100644 --- a/configure.ac +++ b/configure.ac @@ -70,6 +70,7 @@ AC_HAVE_SEEK_DATA AC_HAVE_BMV_OF_SHARED AC_HAVE_NFTW AC_HAVE_RLIMIT_NOFILE +AC_HAVE_FIEXCHANGE AC_CHECK_FUNCS([renameat2]) AC_CHECK_FUNCS([reallocarray]) diff --git a/include/builddefs.in b/include/builddefs.in index dab10c968f..969acf0da2 100644 --- a/include/builddefs.in +++ b/include/builddefs.in @@ -72,6 +72,7 @@ HAVE_SEEK_DATA = @have_seek_data@ HAVE_NFTW = @have_nftw@ HAVE_BMV_OF_SHARED = @have_bmv_of_shared@ HAVE_RLIMIT_NOFILE = @have_rlimit_nofile@ +HAVE_FIEXCHANGE = @have_fiexchange@ GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall diff --git a/ltp/Makefile b/ltp/Makefile index 85f634145c..c2b70d896e 100644 --- a/ltp/Makefile +++ b/ltp/Makefile @@ -36,6 +36,10 @@ ifeq ($(HAVE_COPY_FILE_RANGE),yes) LCFLAGS += -DHAVE_COPY_FILE_RANGE endif +ifeq ($(HAVE_FIEXCHANGE),yes) +LCFLAGS += -DHAVE_FIEXCHANGE +endif + default: depend $(TARGETS) depend: .dep diff --git a/ltp/fsx.c b/ltp/fsx.c index 12c2cc33bf..ee4b8fe45d 100644 --- a/ltp/fsx.c +++ b/ltp/fsx.c @@ -111,6 +111,7 @@ enum { OP_CLONE_RANGE, OP_DEDUPE_RANGE, OP_COPY_RANGE, + OP_EXCHANGE_RANGE, OP_MAX_FULL, /* integrity operations */ @@ -175,6 +176,7 @@ int check_file = 0; /* -X flag enables */ int clone_range_calls = 1; /* -J flag disables */ int dedupe_range_calls = 1; /* -B flag disables */ int copy_range_calls = 1; /* -E flag disables */ +int xchg_range_calls = 1; /* -0 flag disables */ int integrity = 0; /* -i flag */ int fsxgoodfd = 0; int o_direct; /* -Z */ @@ -268,6 +270,7 @@ static const char *op_names[] = { [OP_DEDUPE_RANGE] = "dedupe_range", [OP_COPY_RANGE] = "copy_range", [OP_FSYNC] = "fsync", + [OP_EXCHANGE_RANGE] = "xchg_range", }; static const char *op_name(int operation) @@ -452,6 +455,20 @@ logdump(void) if (overlap) prt("\t******IIII"); break; + case OP_EXCHANGE_RANGE: + prt("XCHG 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x", + lp->args[0], lp->args[0] + lp->args[1] - 1, + lp->args[1], + lp->args[2], lp->args[2] + lp->args[1] - 1); + overlap2 = badoff >= lp->args[2] && + badoff < lp->args[2] + lp->args[1]; + if (overlap && overlap2) + prt("\tXXXX**XXXX"); + else if (overlap) + prt("\tXXXX******"); + else if (overlap2) + prt("\t******XXXX"); + break; case OP_CLONE_RANGE: prt("CLONE 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x", lp->args[0], lp->args[0] + lp->args[1] - 1, @@ -1369,6 +1386,116 @@ do_insert_range(unsigned offset, unsigned length) } #endif +#ifdef FIEXCHANGE_RANGE +static __u64 swap_flags = 0; + +int +test_xchg_range(void) +{ + struct file_xchg_range fsr = { + .file1_fd = fd, + .flags = FILE_XCHG_RANGE_DRY_RUN | swap_flags, + }; + int ret, e; + +retry: + ret = ioctl(fd, FIEXCHANGE_RANGE, &fsr); + e = ret < 0 ? errno : 0; + if (e == EOPNOTSUPP && !(swap_flags & FILE_XCHG_RANGE_NONATOMIC)) { + /* + * If the call fails with atomic mode, try again with non + * atomic mode. + */ + swap_flags = FILE_XCHG_RANGE_NONATOMIC; + fsr.flags |= swap_flags; + goto retry; + } + if (e == EOPNOTSUPP || errno == ENOTTY) { + if (!quiet) + fprintf(stderr, + "main: filesystem does not support " + "exchange range, disabling!\n"); + return 0; + } + + return 1; +} + +void +do_xchg_range(unsigned offset, unsigned length, unsigned dest) +{ + struct file_xchg_range fsr = { + .file1_fd = fd, + .file1_offset = offset, + .file2_offset = dest, + .length = length, + .flags = swap_flags, + }; + void *p; + + if (length == 0) { + if (!quiet && testcalls > simulatedopcount) + prt("skipping zero length exchange range\n"); + log5(OP_EXCHANGE_RANGE, offset, length, dest, FL_SKIPPED); + return; + } + + if ((loff_t)offset >= file_size || (loff_t)dest >= file_size) { + if (!quiet && testcalls > simulatedopcount) + prt("skipping exchange range behind EOF\n"); + log5(OP_EXCHANGE_RANGE, offset, length, dest, FL_SKIPPED); + return; + } + + p = malloc(length); + if (!p) { + if (!quiet && testcalls > simulatedopcount) + prt("skipping exchange range due to ENOMEM\n"); + log5(OP_EXCHANGE_RANGE, offset, length, dest, FL_SKIPPED); + return; + } + + log5(OP_EXCHANGE_RANGE, offset, length, dest, FL_NONE); + + if (testcalls <= simulatedopcount) + goto out_free; + + if ((progressinterval && testcalls % progressinterval == 0) || + (debug && (monitorstart == -1 || monitorend == -1 || + dest <= monitorstart || dest + length <= monitorend))) { + prt("%lu swap\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n", + testcalls, offset, offset+length, length, dest); + } + + if (ioctl(fd, FIEXCHANGE_RANGE, &fsr) == -1) { + prt("exchange range: 0x%x to 0x%x at 0x%x\n", offset, + offset + length, dest); + prterr("do_xchg_range: FIEXCHANGE_RANGE"); + report_failure(161); + goto out_free; + } + + memcpy(p, good_buf + offset, length); + memcpy(good_buf + offset, good_buf + dest, length); + memcpy(good_buf + dest, p, length); +out_free: + free(p); +} + +#else +int +test_xchg_range(void) +{ + return 0; +} + +void +do_xchg_range(unsigned offset, unsigned length, unsigned dest) +{ + return; +} +#endif + #ifdef FICLONERANGE int test_clone_range(void) @@ -1856,6 +1983,7 @@ static int op_args_count(int operation) { switch (operation) { + case OP_EXCHANGE_RANGE: case OP_CLONE_RANGE: case OP_DEDUPE_RANGE: case OP_COPY_RANGE: @@ -2053,6 +2181,9 @@ test(void) case OP_COPY_RANGE: generate_dest_range(true, maxfilelen, &offset, &size, &offset2); break; + case OP_EXCHANGE_RANGE: + generate_dest_range(false, file_size, &offset, &size, &offset2); + break; } have_op: @@ -2096,6 +2227,12 @@ test(void) goto out; } break; + case OP_EXCHANGE_RANGE: + if (!xchg_range_calls) { + log5(op, offset, size, offset2, FL_SKIPPED); + goto out; + } + break; case OP_CLONE_RANGE: if (!clone_range_calls) { log5(op, offset, size, offset2, FL_SKIPPED); @@ -2180,6 +2317,18 @@ test(void) do_insert_range(offset, size); break; + case OP_EXCHANGE_RANGE: + if (size == 0) { + log5(OP_EXCHANGE_RANGE, offset, size, offset2, FL_SKIPPED); + goto out; + } + if (offset2 + size > maxfilelen) { + log5(OP_EXCHANGE_RANGE, offset, size, offset2, FL_SKIPPED); + goto out; + } + + do_xchg_range(offset, size, offset2); + break; case OP_CLONE_RANGE: if (size == 0) { log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED); @@ -2294,6 +2443,9 @@ usage(void) #ifdef HAVE_COPY_FILE_RANGE " -E: Do not use copy range calls\n" #endif +#ifdef FIEXCHANGE_RANGE +" -0: Do not use exchange range calls\n" +#endif " -L: fsxLite - no file creations & no file size changes\n\ -N numops: total # operations to do (default infinity)\n\ -O: use oplen (see -o flag) for every op (default random)\n\ @@ -2608,12 +2760,11 @@ main(int argc, char **argv) page_size = getpagesize(); page_mask = page_size - 1; mmap_mask = page_mask; - setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */ while ((ch = getopt_long(argc, argv, - "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:UWXZ", + "0b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:UWXZ", longopts, NULL)) != EOF) switch (ch) { case 'b': @@ -2747,6 +2898,9 @@ main(int argc, char **argv) case 'I': insert_range_calls = 0; break; + case '0': + xchg_range_calls = 0; + break; case 'J': clone_range_calls = 0; break; @@ -2988,6 +3142,8 @@ main(int argc, char **argv) dedupe_range_calls = test_dedupe_range(); if (copy_range_calls) copy_range_calls = test_copy_range(); + if (xchg_range_calls) + xchg_range_calls = test_xchg_range(); while (numops == -1 || numops--) if (!test()) diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4 index 98572aecd9..b41c087bfb 100644 --- a/m4/package_libcdev.m4 +++ b/m4/package_libcdev.m4 @@ -154,3 +154,23 @@ AC_DEFUN([AC_HAVE_RLIMIT_NOFILE], AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) AC_SUBST(have_rlimit_nofile) ]) + +# +# Check if we have a FIEXCHANGE_RANGE ioctl (Linux) +# +AC_DEFUN([AC_HAVE_FIEXCHANGE], + [ AC_MSG_CHECKING([for FIEXCHANGE_RANGE]) + AC_LINK_IFELSE([AC_LANG_PROGRAM([[ +#define _GNU_SOURCE +#include <sys/syscall.h> +#include <sys/ioctl.h> +#include <unistd.h> +#include <linux/fs.h> +#include <linux/fiexchange.h> + ]], [[ + struct file_xchg_range fxr; + ioctl(-1, FIEXCHANGE_RANGE, &fxr); + ]])],[have_fiexchange=yes + AC_MSG_RESULT(yes)],[AC_MSG_RESULT(no)]) + AC_SUBST(have_fiexchange) + ]) diff --git a/src/fiexchange.h b/src/fiexchange.h new file mode 100644 index 0000000000..29b3ac0ff5 --- /dev/null +++ b/src/fiexchange.h @@ -0,0 +1,101 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later WITH Linux-syscall-note */ +/* + * FIEXCHANGE ioctl definitions, to facilitate exchanging parts of files. + * + * Copyright (C) 2022 Oracle. All Rights Reserved. + * + * Author: Darrick J. Wong <djwong@kernel.org> + */ +#ifndef _LINUX_FIEXCHANGE_H +#define _LINUX_FIEXCHANGE_H + +#include <linux/types.h> + +/* + * Exchange part of file1 with part of the file that this ioctl that is being + * called against (which we'll call file2). Filesystems must be able to + * restart and complete the operation even after the system goes down. + */ +struct file_xchg_range { + __s64 file1_fd; + __s64 file1_offset; /* file1 offset, bytes */ + __s64 file2_offset; /* file2 offset, bytes */ + __s64 length; /* bytes to exchange */ + + __u64 flags; /* see FILE_XCHG_RANGE_* below */ + + /* file2 metadata for optional freshness checks */ + __s64 file2_ino; /* inode number */ + __s64 file2_mtime; /* modification time */ + __s64 file2_ctime; /* change time */ + __s32 file2_mtime_nsec; /* mod time, nsec */ + __s32 file2_ctime_nsec; /* change time, nsec */ + + __u64 pad[6]; /* must be zeroes */ +}; + +/* + * Atomic exchange operations are not required. This relaxes the requirement + * that the filesystem must be able to complete the operation after a crash. + */ +#define FILE_XCHG_RANGE_NONATOMIC (1 << 0) + +/* + * Check that file2's inode number, mtime, and ctime against the values + * provided, and return -EBUSY if there isn't an exact match. + */ +#define FILE_XCHG_RANGE_FILE2_FRESH (1 << 1) + +/* + * Check that the file1's length is equal to file1_offset + length, and that + * file2's length is equal to file2_offset + length. Returns -EDOM if there + * isn't an exact match. + */ +#define FILE_XCHG_RANGE_FULL_FILES (1 << 2) + +/* + * Exchange file data all the way to the ends of both files, and then exchange + * the file sizes. This flag can be used to replace a file's contents with a + * different amount of data. length will be ignored. + */ +#define FILE_XCHG_RANGE_TO_EOF (1 << 3) + +/* Flush all changes in file data and file metadata to disk before returning. */ +#define FILE_XCHG_RANGE_FSYNC (1 << 4) + +/* Dry run; do all the parameter verification but do not change anything. */ +#define FILE_XCHG_RANGE_DRY_RUN (1 << 5) + +/* + * Do not exchange any part of the range where file1's mapping is a hole. This + * can be used to emulate scatter-gather atomic writes with a temp file. + */ +#define FILE_XCHG_RANGE_SKIP_FILE1_HOLES (1 << 6) + +/* + * Commit the contents of file1 into file2 if file2 has the same inode number, + * mtime, and ctime as the arguments provided to the call. The old contents of + * file2 will be moved to file1. + * + * With this flag, all committed information can be retrieved even if the + * system crashes or is rebooted. This includes writing through or flushing a + * disk cache if present. The call blocks until the device reports that the + * commit is complete. + * + * This flag should not be combined with NONATOMIC. It can be combined with + * SKIP_FILE1_HOLES. + */ +#define FILE_XCHG_RANGE_COMMIT (FILE_XCHG_RANGE_FILE2_FRESH | \ + FILE_XCHG_RANGE_FSYNC) + +#define FILE_XCHG_RANGE_ALL_FLAGS (FILE_XCHG_RANGE_NONATOMIC | \ + FILE_XCHG_RANGE_FILE2_FRESH | \ + FILE_XCHG_RANGE_FULL_FILES | \ + FILE_XCHG_RANGE_TO_EOF | \ + FILE_XCHG_RANGE_FSYNC | \ + FILE_XCHG_RANGE_DRY_RUN | \ + FILE_XCHG_RANGE_SKIP_FILE1_HOLES) + +#define FIEXCHANGE_RANGE _IOWR('X', 129, struct file_xchg_range) + +#endif /* _LINUX_FIEXCHANGE_H */ diff --git a/src/global.h b/src/global.h index b44070993c..49570ef117 100644 --- a/src/global.h +++ b/src/global.h @@ -171,6 +171,12 @@ #include <sys/mman.h> #endif +#ifdef HAVE_FIEXCHANGE +# include <linux/fiexchange.h> +#else +# include "fiexchange.h" +#endif + static inline unsigned long long rounddown_64(unsigned long long x, unsigned int y) { ^ permalink raw reply related [flat|nested] 107+ messages in thread
end of thread, other threads:[~2023-03-01 2:59 UTC | newest] Thread overview: 107+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-12-30 21:14 [NYE DELUGE 2/4] xfs: online repair in its entirety Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: race online scrub with fsstress Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/5] xfs: race fsstress with online scrubbers for AG and fs metadata Darrick J. Wong 2023-02-05 13:04 ` Zorro Lang 2023-02-07 16:58 ` Darrick J. Wong 2023-02-07 17:02 ` [PATCH v24.1 " Darrick J. Wong 2023-02-07 18:45 ` Zorro Lang 2022-12-30 22:19 ` [PATCH 5/5] xfs: race fsstress with online scrubbers for file metadata Darrick J. Wong 2022-12-30 22:19 ` [PATCH 3/5] fuzzy: add a custom xfs find utility for scrub stress tests Darrick J. Wong 2023-02-05 12:57 ` Zorro Lang 2023-02-07 16:57 ` Darrick J. Wong 2023-02-07 17:01 ` [PATCH v24.1 " Darrick J. Wong 2023-02-07 18:42 ` Zorro Lang 2022-12-30 22:19 ` [PATCH 1/5] xfs/357: switch fuzzing to agi 1 Darrick J. Wong 2023-02-07 18:46 ` Zorro Lang 2022-12-30 22:19 ` [PATCH 4/5] fuzzy: allow xfs scrub stress tests to pick preconfigured fsstress configs Darrick J. Wong 2023-02-07 18:48 ` Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] xfs: force rebuilding of metadata Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] fuzzy: use FORCE_REBUILD over injecting force_repair Darrick J. Wong 2023-02-14 8:00 ` Zorro Lang 2023-02-14 18:18 ` Darrick J. Wong 2023-02-16 14:57 ` Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] xfs: stress test ag repair functions Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] xfs: test rebuilding the entire filesystem with online fsck Darrick J. Wong 2023-02-18 6:06 ` [PATCHSET v24.0 0/2] fstests: online repair of AG btrees Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of inodes Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online repair for inode record metadata Darrick J. Wong 2023-02-18 6:07 ` Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Darrick J. Wong 2022-12-30 22:19 ` [PATCH 4/4] xfs: race fsstress with online repair for special file metadata Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/4] xfs: test rebuilding xattrs when the data fork is btree format Darrick J. Wong 2022-12-30 22:19 ` [PATCH 3/4] xfs: ensure that online file data fork repairs don't hit EDQUOT Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/4] xfs: race fsstress with online repair for inode and fork metadata Darrick J. Wong 2023-02-18 6:07 ` [PATCHSET v24.0 0/4] fstests: online repair of file fork mappings Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of quota and counters Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online scrub and repair for quota metadata Darrick J. Wong 2023-02-18 6:10 ` Zorro Lang 2023-02-18 6:12 ` Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of quota counters Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online scrub and repair for quotacheck Darrick J. Wong 2023-02-18 6:12 ` Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of file link counts Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with inode link count check and repair Darrick J. Wong 2023-02-18 6:13 ` Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair for fs summary counters Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] xfs: test fs summary counter online repair Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] xfs: race fsstress with online repair for summary counters Darrick J. Wong 2023-02-18 6:14 ` [PATCHSET v24.0 0/2] fstests: online repair for fs " Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of rmap btrees Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs/422: don't freeze while racing rmap repair and fsstress Darrick J. Wong 2023-02-18 6:15 ` Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] populate: take a snapshot of the filesystem if creation fails Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] populate: fix some weirdness in __populate_check_xfs_agbtree_height Darrick J. Wong 2023-02-18 6:16 ` [PATCHSET v24.0 0/2] fstests: fix a few bugs in fs population Zorro Lang 2022-12-30 22:19 ` [PATCHSET v24.0 00/24] fstests: improve xfs fuzzing Darrick J. Wong 2022-12-30 22:19 ` [PATCH 01/24] fuzzy: disable per-field random fuzzing by default Darrick J. Wong 2022-12-30 22:19 ` [PATCH 02/24] fuzzy: disable timstamp " Darrick J. Wong 2022-12-30 22:19 ` [PATCH 07/24] fuzzy: don't fuzz xattr namespace flags and values Darrick J. Wong 2022-12-30 22:19 ` [PATCH 05/24] fuzzy: don't fuzz inode generation numbers Darrick J. Wong 2022-12-30 22:19 ` [PATCH 03/24] fuzzy: don't fuzz the log sequence number Darrick J. Wong 2022-12-30 22:19 ` [PATCH 06/24] fuzzy: don't fuzz user-controllable inode flags Darrick J. Wong 2022-12-30 22:19 ` [PATCH 04/24] fuzzy: don't fuzz obsolete inode fields Darrick J. Wong 2022-12-30 22:19 ` [PATCH 12/24] common/fuzzy: fix some problems with the offline repair strategy Darrick J. Wong 2022-12-30 22:19 ` [PATCH 11/24] common/fuzzy: fix some problems with the online " Darrick J. Wong 2022-12-30 22:19 ` [PATCH 08/24] common/fuzzy: split out each repair strategy into a separate helper Darrick J. Wong 2022-12-30 22:19 ` [PATCH 10/24] common/fuzzy: hoist the post-repair fs modification step Darrick J. Wong 2022-12-30 22:19 ` [PATCH 09/24] common/fuzzy: add an underline to the full log between sections Darrick J. Wong 2022-12-30 22:19 ` [PATCH 13/24] common/fuzzy: fix some problems with the no-repair strategy Darrick J. Wong 2022-12-30 22:19 ` [PATCH 14/24] common/fuzzy: fix some problems with the online-then-offline repair strategy Darrick J. Wong 2022-12-30 22:19 ` [PATCH 16/24] xfs/{35[45],455}: fix bogus corruption errors Darrick J. Wong 2022-12-30 22:19 ` [PATCH 21/24] fuzzy: compress coredumps created while fuzzing Darrick J. Wong 2022-12-30 22:19 ` [PATCH 15/24] common/fuzzy: fix some problems with the post-repair fs modification code Darrick J. Wong 2022-12-30 22:19 ` [PATCH 19/24] common/fuzzy: exercise the filesystem a little harder after repairing Darrick J. Wong 2022-12-30 22:19 ` [PATCH 22/24] fuzzy: report the fuzzing repair strategy in seqres.full Darrick J. Wong 2022-12-30 22:19 ` [PATCH 17/24] common/fuzzy: evaluate xfs_check vs xfs_repair Darrick J. Wong 2022-12-30 22:19 ` [PATCH 20/24] fuzzy: dump metadata state before fuzzing Darrick J. Wong 2022-12-30 22:19 ` [PATCH 18/24] common: check xfs health after doing an online scrub Darrick J. Wong 2022-12-30 22:19 ` [PATCH 24/24] fuzzy: for fuzzing ag btrees, find the path to the AG header Darrick J. Wong 2022-12-30 22:19 ` [PATCH 23/24] xfs: improve metadata array field handling when fuzzing Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/5] fstests: strengthen fuzz testing Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/5] fuzzy: test fuzzing directory block mappings Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/5] fuzzy: test fuzzing xattr " Darrick J. Wong 2022-12-30 22:19 ` [PATCH 5/5] fuzzy: fuzz test key/pointers of inode btrees Darrick J. Wong 2022-12-30 22:19 ` [PATCH 4/5] xfs: fuzz test both repair strategies Darrick J. Wong 2022-12-30 22:19 ` [PATCH 3/5] fuzzy: test fuzzing realtime free space metadata Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/7] fstests: atomic file updates Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/7] xfs/122: fix for swapext log items Darrick J. Wong 2022-12-30 22:19 ` [PATCH 5/7] generic: test that file privilege gets dropped with FIEXCHANGE_RANGE Darrick J. Wong 2022-12-30 22:19 ` [PATCH 4/7] generic, xfs: test scatter-gather atomic file updates Darrick J. Wong 2022-12-30 22:19 ` [PATCH 3/7] generic: test new vfs swapext ioctl Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/7] generic: test old xfs extent swapping ioctl Darrick J. Wong 2022-12-30 22:19 ` [PATCH 7/7] fsstress: update for FIEXCHANGE_RANGE Darrick J. Wong 2022-12-30 22:19 ` [PATCH 6/7] fsx: support FIEXCHANGE_RANGE Darrick J. Wong 2023-02-28 1:55 ` Zorro Lang 2023-03-01 2:56 ` Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of realtime summaries Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online repair of realtime summary files Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/1] fstests: online repair of extended attributes Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/1] xfs: race fsstress with online repair of extended attribute data Darrick J. Wong 2022-12-30 22:19 ` [PATCHSET v24.0 0/2] fstests: online repair of directories Darrick J. Wong 2022-12-30 22:19 ` [PATCH 1/2] xfs: ensure that online directory repairs don't hit EDQUOT Darrick J. Wong 2022-12-30 22:19 ` [PATCH 2/2] xfs: race fsstress with online repair of dirs and parent pointers Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v24.0 0/1] fstests: test automatic scrub optimization by default Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/1] xfs: test xfs_scrub dry run, preen, and repair mode Darrick J. Wong 2023-03-01 2:59 [PATCHSET v24.2 0/7] fstests: atomic file updates Darrick J. Wong 2023-03-01 2:59 ` [PATCH 6/7] fsx: support FIEXCHANGE_RANGE Darrick J. Wong
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).