* [NYE DELUGE 3/4] xfs: modernize the realtime volume @ 2022-12-30 21:14 Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET 0/1] fstests: test upgrading older features Darrick J. Wong ` (7 more replies) 0 siblings, 8 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 21:14 UTC (permalink / raw) To: djwong; +Cc: xfs, fstests Hi all, This third patchset deluge is for the realtime modernization project. There are five main parts to this effort -- adding a metadata directory tree; sharding the realtime volume into allocation groups to reduce metadata lock contention; adding reverse mapping; adding reflink; and adding the one piece needed to make quotas work on realtime. This brings the robustness of the realtime volume up to par with the data volume. Originally, the modernization effort was a side project that was intended to match XFS up to the proliferation of persistent memory. The data device would store metadata on cheap(er) flash storage, and the realtime volume would be used to map persistent memory to files and take advantage of the ability to do PMD-aligned allocations. It's now less clear how much of that will actually happen (CXL?), but the code's finished, reasonably well tested, and ready for review. NOTE: I hacked up metadump to support saving the metadata contents of external logs and realtime devices so that I could run fuzz testing in the least thoughtful way possible. Chandan is working on improving the deployment image story for our customers, and will likely produce something better than my rush job. As a warning, the patches will likely take several days to trickle in. --D ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCHSET 0/1] fstests: test upgrading older features 2022-12-30 21:14 [NYE DELUGE 3/4] xfs: modernize the realtime volume Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/1] xfs: test upgrading old features Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong ` (6 subsequent siblings) 7 siblings, 1 reply; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, Here is a general regression test to make sure that we can invoke the xfs_repair feature to add new features to V5 filesystems without errors. There are already targeted functionality tests for inobtcount and bigtime; this new one exists as a general upgrade exerciser. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=upgrade-older-features fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=upgrade-older-features --- tests/xfs/769 | 248 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/769.out | 2 2 files changed, 250 insertions(+) create mode 100755 tests/xfs/769 create mode 100644 tests/xfs/769.out ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCH 1/1] xfs: test upgrading old features 2022-12-30 22:20 ` [PATCHSET 0/1] fstests: test upgrading older features Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2023-03-06 15:56 ` Zorro Lang 0 siblings, 1 reply; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Test the ability to add older v5 features. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/769 | 248 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/769.out | 2 2 files changed, 250 insertions(+) create mode 100755 tests/xfs/769 create mode 100644 tests/xfs/769.out diff --git a/tests/xfs/769 b/tests/xfs/769 new file mode 100755 index 0000000000..7613048f52 --- /dev/null +++ b/tests/xfs/769 @@ -0,0 +1,248 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 769 +# +# Test upgrading filesystems with new features. +# +. ./common/preamble +_begin_fstest auto mkfs repair + +# Import common functions. +. ./common/filter +. ./common/populate + +# real QA test starts here +_supported_fs xfs + +test -w /dev/ttyprintk || _notrun "test requires writable /dev/ttyprintk" +_require_check_dmesg +_require_scratch_nocheck +_require_scratch_xfs_crc + +# Does repair know how to add a particular feature to a filesystem? +check_repair_upgrade() +{ + $XFS_REPAIR_PROG -c "$1=narf" 2>&1 | \ + grep -q 'unknown option' && return 1 + return 0 +} + +# Are we configured for realtime? +rt_configured() +{ + test "$USE_EXTERNAL" = "yes" && test -n "$SCRATCH_RTDEV" +} + +# Compute the MKFS_OPTIONS string for a particular feature upgrade test +compute_mkfs_options() +{ + local m_opts="" + local caller_options="$MKFS_OPTIONS" + + for feat in "${FEATURES[@]}"; do + local feat_state="${FEATURE_STATE["${feat}"]}" + + if echo "$caller_options" | grep -E -w -q "${feat}=[0-9]*"; then + # Change the caller's options + caller_options="$(echo "$caller_options" | \ + sed -e "s/\([^[:alnum:]]\)${feat}=[0-9]*/\1${feat}=${feat_state}/g")" + else + # Add it to our list of new mkfs flags + m_opts="${feat}=${feat_state},${m_opts}" + fi + done + + test -n "$m_opts" && m_opts=" -m $m_opts" + + echo "$caller_options$m_opts" +} + +# Log the start of an upgrade. +function upgrade_start_message() +{ + local feat="$1" + + echo "Add $feat to filesystem" +} + +# Find dmesg log messages since we started a particular upgrade test +function dmesg_since_feature_upgrade_start() +{ + local feat_logmsg="$(upgrade_start_message "$1")" + + # search the dmesg log of last run of $seqnum for possible failures + # use sed \cregexpc address type, since $seqnum contains "/" + dmesg | \ + tac | \ + sed -ne "0,\#run fstests $seqnum at $date_time#p" | \ + sed -ne "0,\#${feat_logmsg}#p" | \ + tac +} + +# Did the mount fail because this feature is not supported? +function feature_unsupported() +{ + local feat="$1" + + dmesg_since_feature_upgrade_start "$feat" | \ + grep -q 'has unknown.*features' +} + +# Exercise the scratch fs +function scratch_fsstress() +{ + echo moo > $SCRATCH_MNT/sample.txt + $FSSTRESS_PROG -n $((TIME_FACTOR * 1000)) -p $((LOAD_FACTOR * 4)) \ + -d $SCRATCH_MNT/data >> $seqres.full +} + +# Exercise the filesystem a little bit and emit a manifest. +function pre_exercise() +{ + local feat="$1" + + _try_scratch_mount &> $tmp.mount + res=$? + # If the kernel doesn't support the filesystem even after a + # fresh format, skip the rest of the upgrade test quietly. + if [ $res -eq 32 ] && feature_unsupported "$feat"; then + echo "mount failed due to unsupported feature $feat" >> $seqres.full + return 1 + fi + if [ $res -ne 0 ]; then + cat $tmp.mount + echo "mount failed with $res before upgrading to $feat" | \ + tee -a $seqres.full + return 1 + fi + + scratch_fsstress + find $SCRATCH_MNT -type f -print0 | xargs -r -0 md5sum > $tmp.manifest + _scratch_unmount + return 0 +} + +# Check the manifest and exercise the filesystem more +function post_exercise() +{ + local feat="$1" + + _try_scratch_mount &> $tmp.mount + res=$? + # If the kernel doesn't support the filesystem even after a + # fresh format, skip the rest of the upgrade test quietly. + if [ $res -eq 32 ] && feature_unsupported "$feat"; then + echo "mount failed due to unsupported feature $feat" >> $seqres.full + return 1 + fi + if [ $res -ne 0 ]; then + cat $tmp.mount + echo "mount failed with $res after upgrading to $feat" | \ + tee -a $seqres.full + return 1 + fi + + md5sum --quiet -c $tmp.manifest || \ + echo "fs contents ^^^ changed after adding $feat" + + iam="check" _check_scratch_fs || \ + echo "scratch fs check failed after adding $feat" + + # Try to mount the fs in case the check unmounted it + _try_scratch_mount &>> $seqres.full + + scratch_fsstress + + iam="check" _check_scratch_fs || \ + echo "scratch fs check failed after exercising $feat" + + # Try to unmount the fs in case the check didn't + _scratch_unmount &>> $seqres.full + return 0 +} + +# Create a list of fs features in the order that support for them was added +# to the kernel driver. For each feature upgrade test, we enable all the +# features that came before it and none of the ones after, which means we're +# testing incremental migrations. We start each run with a clean fs so that +# errors and unsatisfied requirements (log size, root ino position, etc) in one +# upgrade don't spread failure to the rest of the tests. +FEATURES=() +if rt_configured; then + check_repair_upgrade finobt && FEATURES+=("finobt") + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") + check_repair_upgrade bigtime && FEATURES+=("bigtime") +else + check_repair_upgrade finobt && FEATURES+=("finobt") + check_repair_upgrade rmapbt && FEATURES+=("rmapbt") + check_repair_upgrade reflink && FEATURES+=("reflink") + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") + check_repair_upgrade bigtime && FEATURES+=("bigtime") +fi + +test "${#FEATURES[@]}" -eq 0 && \ + _notrun "xfs_repair does not know how to add V5 features" + +declare -A FEATURE_STATE +for f in "${FEATURES[@]}"; do + FEATURE_STATE["$f"]=0 +done + +for feat in "${FEATURES[@]}"; do + echo "-----------------------" >> $seqres.full + + upgrade_start_message "$feat" | tee -a $seqres.full /dev/ttyprintk > /dev/null + + opts="$(compute_mkfs_options)" + echo "mkfs.xfs $opts" >> $seqres.full + + # Format filesystem + MKFS_OPTIONS="$opts" _scratch_mkfs &>> $seqres.full + res=$? + outcome="mkfs returns $res for $feat upgrade test" + echo "$outcome" >> $seqres.full + if [ $res -ne 0 ]; then + echo "$outcome" + continue + fi + + # Create some files to make things interesting. + pre_exercise "$feat" || break + + # Upgrade the fs + _scratch_xfs_repair -c "${feat}=1" &> $tmp.upgrade + res=$? + cat $tmp.upgrade >> $seqres.full + grep -q "^Adding" $tmp.upgrade || \ + echo "xfs_repair ignored command to add $feat" + + outcome="xfs_repair returns $res while adding $feat" + echo "$outcome" >> $seqres.full + if [ $res -ne 0 ]; then + # Couldn't upgrade filesystem, move on to the next feature. + FEATURE_STATE["$feat"]=1 + continue + fi + + # Make sure repair runs cleanly afterwards + _scratch_xfs_repair -n &>> $seqres.full + res=$? + outcome="xfs_repair -n returns $res after adding $feat" + echo "$outcome" >> $seqres.full + if [ $res -ne 0 ]; then + echo "$outcome" + fi + + # Make sure we can still exercise the filesystem. + post_exercise "$feat" || break + + # Update feature state for next run + FEATURE_STATE["$feat"]=1 +done + +# success, all done +echo Silence is golden. +status=0 +exit diff --git a/tests/xfs/769.out b/tests/xfs/769.out new file mode 100644 index 0000000000..332432db97 --- /dev/null +++ b/tests/xfs/769.out @@ -0,0 +1,2 @@ +QA output created by 769 +Silence is golden. ^ permalink raw reply related [flat|nested] 68+ messages in thread
* Re: [PATCH 1/1] xfs: test upgrading old features 2022-12-30 22:20 ` [PATCH 1/1] xfs: test upgrading old features Darrick J. Wong @ 2023-03-06 15:56 ` Zorro Lang 2023-03-06 16:41 ` Darrick J. Wong 0 siblings, 1 reply; 68+ messages in thread From: Zorro Lang @ 2023-03-06 15:56 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:20:29PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > Test the ability to add older v5 features. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- > tests/xfs/769 | 248 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > tests/xfs/769.out | 2 > 2 files changed, 250 insertions(+) > create mode 100755 tests/xfs/769 > create mode 100644 tests/xfs/769.out > > > diff --git a/tests/xfs/769 b/tests/xfs/769 > new file mode 100755 > index 0000000000..7613048f52 > --- /dev/null > +++ b/tests/xfs/769 > @@ -0,0 +1,248 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# Copyright (c) 2022 Oracle. All Rights Reserved. > +# > +# FS QA Test No. 769 > +# > +# Test upgrading filesystems with new features. > +# > +. ./common/preamble > +_begin_fstest auto mkfs repair > + > +# Import common functions. > +. ./common/filter > +. ./common/populate > + > +# real QA test starts here > +_supported_fs xfs > + > +test -w /dev/ttyprintk || _notrun "test requires writable /dev/ttyprintk" Hi Darrick, I'm not sure why /dev/ttyprintk is necessary. I think sometimes we might not have this driver, but has /dev/kmsg. Can /dev/kmsg be a replacement of ttyprintk ? Thanks, Zorro > +_require_check_dmesg > +_require_scratch_nocheck > +_require_scratch_xfs_crc > + > +# Does repair know how to add a particular feature to a filesystem? > +check_repair_upgrade() > +{ > + $XFS_REPAIR_PROG -c "$1=narf" 2>&1 | \ > + grep -q 'unknown option' && return 1 > + return 0 > +} > + > +# Are we configured for realtime? > +rt_configured() > +{ > + test "$USE_EXTERNAL" = "yes" && test -n "$SCRATCH_RTDEV" > +} > + > +# Compute the MKFS_OPTIONS string for a particular feature upgrade test > +compute_mkfs_options() > +{ > + local m_opts="" > + local caller_options="$MKFS_OPTIONS" > + > + for feat in "${FEATURES[@]}"; do > + local feat_state="${FEATURE_STATE["${feat}"]}" > + > + if echo "$caller_options" | grep -E -w -q "${feat}=[0-9]*"; then > + # Change the caller's options > + caller_options="$(echo "$caller_options" | \ > + sed -e "s/\([^[:alnum:]]\)${feat}=[0-9]*/\1${feat}=${feat_state}/g")" > + else > + # Add it to our list of new mkfs flags > + m_opts="${feat}=${feat_state},${m_opts}" > + fi > + done > + > + test -n "$m_opts" && m_opts=" -m $m_opts" > + > + echo "$caller_options$m_opts" > +} > + > +# Log the start of an upgrade. > +function upgrade_start_message() > +{ > + local feat="$1" > + > + echo "Add $feat to filesystem" > +} > + > +# Find dmesg log messages since we started a particular upgrade test > +function dmesg_since_feature_upgrade_start() > +{ > + local feat_logmsg="$(upgrade_start_message "$1")" > + > + # search the dmesg log of last run of $seqnum for possible failures > + # use sed \cregexpc address type, since $seqnum contains "/" > + dmesg | \ > + tac | \ > + sed -ne "0,\#run fstests $seqnum at $date_time#p" | \ > + sed -ne "0,\#${feat_logmsg}#p" | \ > + tac > +} > + > +# Did the mount fail because this feature is not supported? > +function feature_unsupported() > +{ > + local feat="$1" > + > + dmesg_since_feature_upgrade_start "$feat" | \ > + grep -q 'has unknown.*features' > +} > + > +# Exercise the scratch fs > +function scratch_fsstress() > +{ > + echo moo > $SCRATCH_MNT/sample.txt > + $FSSTRESS_PROG -n $((TIME_FACTOR * 1000)) -p $((LOAD_FACTOR * 4)) \ > + -d $SCRATCH_MNT/data >> $seqres.full > +} > + > +# Exercise the filesystem a little bit and emit a manifest. > +function pre_exercise() > +{ > + local feat="$1" > + > + _try_scratch_mount &> $tmp.mount > + res=$? > + # If the kernel doesn't support the filesystem even after a > + # fresh format, skip the rest of the upgrade test quietly. > + if [ $res -eq 32 ] && feature_unsupported "$feat"; then > + echo "mount failed due to unsupported feature $feat" >> $seqres.full > + return 1 > + fi > + if [ $res -ne 0 ]; then > + cat $tmp.mount > + echo "mount failed with $res before upgrading to $feat" | \ > + tee -a $seqres.full > + return 1 > + fi > + > + scratch_fsstress > + find $SCRATCH_MNT -type f -print0 | xargs -r -0 md5sum > $tmp.manifest > + _scratch_unmount > + return 0 > +} > + > +# Check the manifest and exercise the filesystem more > +function post_exercise() > +{ > + local feat="$1" > + > + _try_scratch_mount &> $tmp.mount > + res=$? > + # If the kernel doesn't support the filesystem even after a > + # fresh format, skip the rest of the upgrade test quietly. > + if [ $res -eq 32 ] && feature_unsupported "$feat"; then > + echo "mount failed due to unsupported feature $feat" >> $seqres.full > + return 1 > + fi > + if [ $res -ne 0 ]; then > + cat $tmp.mount > + echo "mount failed with $res after upgrading to $feat" | \ > + tee -a $seqres.full > + return 1 > + fi > + > + md5sum --quiet -c $tmp.manifest || \ > + echo "fs contents ^^^ changed after adding $feat" > + > + iam="check" _check_scratch_fs || \ > + echo "scratch fs check failed after adding $feat" > + > + # Try to mount the fs in case the check unmounted it > + _try_scratch_mount &>> $seqres.full > + > + scratch_fsstress > + > + iam="check" _check_scratch_fs || \ > + echo "scratch fs check failed after exercising $feat" > + > + # Try to unmount the fs in case the check didn't > + _scratch_unmount &>> $seqres.full > + return 0 > +} > + > +# Create a list of fs features in the order that support for them was added > +# to the kernel driver. For each feature upgrade test, we enable all the > +# features that came before it and none of the ones after, which means we're > +# testing incremental migrations. We start each run with a clean fs so that > +# errors and unsatisfied requirements (log size, root ino position, etc) in one > +# upgrade don't spread failure to the rest of the tests. > +FEATURES=() > +if rt_configured; then > + check_repair_upgrade finobt && FEATURES+=("finobt") > + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") > + check_repair_upgrade bigtime && FEATURES+=("bigtime") > +else > + check_repair_upgrade finobt && FEATURES+=("finobt") > + check_repair_upgrade rmapbt && FEATURES+=("rmapbt") > + check_repair_upgrade reflink && FEATURES+=("reflink") > + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") > + check_repair_upgrade bigtime && FEATURES+=("bigtime") > +fi > + > +test "${#FEATURES[@]}" -eq 0 && \ > + _notrun "xfs_repair does not know how to add V5 features" > + > +declare -A FEATURE_STATE > +for f in "${FEATURES[@]}"; do > + FEATURE_STATE["$f"]=0 > +done > + > +for feat in "${FEATURES[@]}"; do > + echo "-----------------------" >> $seqres.full > + > + upgrade_start_message "$feat" | tee -a $seqres.full /dev/ttyprintk > /dev/null > + > + opts="$(compute_mkfs_options)" > + echo "mkfs.xfs $opts" >> $seqres.full > + > + # Format filesystem > + MKFS_OPTIONS="$opts" _scratch_mkfs &>> $seqres.full > + res=$? > + outcome="mkfs returns $res for $feat upgrade test" > + echo "$outcome" >> $seqres.full > + if [ $res -ne 0 ]; then > + echo "$outcome" > + continue > + fi > + > + # Create some files to make things interesting. > + pre_exercise "$feat" || break > + > + # Upgrade the fs > + _scratch_xfs_repair -c "${feat}=1" &> $tmp.upgrade > + res=$? > + cat $tmp.upgrade >> $seqres.full > + grep -q "^Adding" $tmp.upgrade || \ > + echo "xfs_repair ignored command to add $feat" > + > + outcome="xfs_repair returns $res while adding $feat" > + echo "$outcome" >> $seqres.full > + if [ $res -ne 0 ]; then > + # Couldn't upgrade filesystem, move on to the next feature. > + FEATURE_STATE["$feat"]=1 > + continue > + fi > + > + # Make sure repair runs cleanly afterwards > + _scratch_xfs_repair -n &>> $seqres.full > + res=$? > + outcome="xfs_repair -n returns $res after adding $feat" > + echo "$outcome" >> $seqres.full > + if [ $res -ne 0 ]; then > + echo "$outcome" > + fi > + > + # Make sure we can still exercise the filesystem. > + post_exercise "$feat" || break > + > + # Update feature state for next run > + FEATURE_STATE["$feat"]=1 > +done > + > +# success, all done > +echo Silence is golden. > +status=0 > +exit > diff --git a/tests/xfs/769.out b/tests/xfs/769.out > new file mode 100644 > index 0000000000..332432db97 > --- /dev/null > +++ b/tests/xfs/769.out > @@ -0,0 +1,2 @@ > +QA output created by 769 > +Silence is golden. > ^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 1/1] xfs: test upgrading old features 2023-03-06 15:56 ` Zorro Lang @ 2023-03-06 16:41 ` Darrick J. Wong 2023-03-06 16:54 ` Zorro Lang 0 siblings, 1 reply; 68+ messages in thread From: Darrick J. Wong @ 2023-03-06 16:41 UTC (permalink / raw) To: Zorro Lang; +Cc: linux-xfs, fstests On Mon, Mar 06, 2023 at 11:56:11PM +0800, Zorro Lang wrote: > On Fri, Dec 30, 2022 at 02:20:29PM -0800, Darrick J. Wong wrote: > > From: Darrick J. Wong <djwong@kernel.org> > > > > Test the ability to add older v5 features. > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > --- > > tests/xfs/769 | 248 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > > tests/xfs/769.out | 2 > > 2 files changed, 250 insertions(+) > > create mode 100755 tests/xfs/769 > > create mode 100644 tests/xfs/769.out > > > > > > diff --git a/tests/xfs/769 b/tests/xfs/769 > > new file mode 100755 > > index 0000000000..7613048f52 > > --- /dev/null > > +++ b/tests/xfs/769 > > @@ -0,0 +1,248 @@ > > +#! /bin/bash > > +# SPDX-License-Identifier: GPL-2.0-or-later > > +# Copyright (c) 2022 Oracle. All Rights Reserved. > > +# > > +# FS QA Test No. 769 > > +# > > +# Test upgrading filesystems with new features. > > +# > > +. ./common/preamble > > +_begin_fstest auto mkfs repair > > + > > +# Import common functions. > > +. ./common/filter > > +. ./common/populate > > + > > +# real QA test starts here > > +_supported_fs xfs > > + > > +test -w /dev/ttyprintk || _notrun "test requires writable /dev/ttyprintk" > > Hi Darrick, > > I'm not sure why /dev/ttyprintk is necessary. I think sometimes we might not > have this driver, but has /dev/kmsg. Can /dev/kmsg be a replacement of > ttyprintk ? The kernel logging here is for debugging purposes -- if the upgrade corrupts the fs and the kernel splats, whoever triages the failure can have at least some clue as to where things went wrong. I think the _notrun line should go away though. --D > Thanks, > Zorro > > > > +_require_check_dmesg > > +_require_scratch_nocheck > > +_require_scratch_xfs_crc > > + > > +# Does repair know how to add a particular feature to a filesystem? > > +check_repair_upgrade() > > +{ > > + $XFS_REPAIR_PROG -c "$1=narf" 2>&1 | \ > > + grep -q 'unknown option' && return 1 > > + return 0 > > +} > > + > > +# Are we configured for realtime? > > +rt_configured() > > +{ > > + test "$USE_EXTERNAL" = "yes" && test -n "$SCRATCH_RTDEV" > > +} > > + > > +# Compute the MKFS_OPTIONS string for a particular feature upgrade test > > +compute_mkfs_options() > > +{ > > + local m_opts="" > > + local caller_options="$MKFS_OPTIONS" > > + > > + for feat in "${FEATURES[@]}"; do > > + local feat_state="${FEATURE_STATE["${feat}"]}" > > + > > + if echo "$caller_options" | grep -E -w -q "${feat}=[0-9]*"; then > > + # Change the caller's options > > + caller_options="$(echo "$caller_options" | \ > > + sed -e "s/\([^[:alnum:]]\)${feat}=[0-9]*/\1${feat}=${feat_state}/g")" > > + else > > + # Add it to our list of new mkfs flags > > + m_opts="${feat}=${feat_state},${m_opts}" > > + fi > > + done > > + > > + test -n "$m_opts" && m_opts=" -m $m_opts" > > + > > + echo "$caller_options$m_opts" > > +} > > + > > +# Log the start of an upgrade. > > +function upgrade_start_message() > > +{ > > + local feat="$1" > > + > > + echo "Add $feat to filesystem" > > +} > > + > > +# Find dmesg log messages since we started a particular upgrade test > > +function dmesg_since_feature_upgrade_start() > > +{ > > + local feat_logmsg="$(upgrade_start_message "$1")" > > + > > + # search the dmesg log of last run of $seqnum for possible failures > > + # use sed \cregexpc address type, since $seqnum contains "/" > > + dmesg | \ > > + tac | \ > > + sed -ne "0,\#run fstests $seqnum at $date_time#p" | \ > > + sed -ne "0,\#${feat_logmsg}#p" | \ > > + tac > > +} > > + > > +# Did the mount fail because this feature is not supported? > > +function feature_unsupported() > > +{ > > + local feat="$1" > > + > > + dmesg_since_feature_upgrade_start "$feat" | \ > > + grep -q 'has unknown.*features' > > +} > > + > > +# Exercise the scratch fs > > +function scratch_fsstress() > > +{ > > + echo moo > $SCRATCH_MNT/sample.txt > > + $FSSTRESS_PROG -n $((TIME_FACTOR * 1000)) -p $((LOAD_FACTOR * 4)) \ > > + -d $SCRATCH_MNT/data >> $seqres.full > > +} > > + > > +# Exercise the filesystem a little bit and emit a manifest. > > +function pre_exercise() > > +{ > > + local feat="$1" > > + > > + _try_scratch_mount &> $tmp.mount > > + res=$? > > + # If the kernel doesn't support the filesystem even after a > > + # fresh format, skip the rest of the upgrade test quietly. > > + if [ $res -eq 32 ] && feature_unsupported "$feat"; then > > + echo "mount failed due to unsupported feature $feat" >> $seqres.full > > + return 1 > > + fi > > + if [ $res -ne 0 ]; then > > + cat $tmp.mount > > + echo "mount failed with $res before upgrading to $feat" | \ > > + tee -a $seqres.full > > + return 1 > > + fi > > + > > + scratch_fsstress > > + find $SCRATCH_MNT -type f -print0 | xargs -r -0 md5sum > $tmp.manifest > > + _scratch_unmount > > + return 0 > > +} > > + > > +# Check the manifest and exercise the filesystem more > > +function post_exercise() > > +{ > > + local feat="$1" > > + > > + _try_scratch_mount &> $tmp.mount > > + res=$? > > + # If the kernel doesn't support the filesystem even after a > > + # fresh format, skip the rest of the upgrade test quietly. > > + if [ $res -eq 32 ] && feature_unsupported "$feat"; then > > + echo "mount failed due to unsupported feature $feat" >> $seqres.full > > + return 1 > > + fi > > + if [ $res -ne 0 ]; then > > + cat $tmp.mount > > + echo "mount failed with $res after upgrading to $feat" | \ > > + tee -a $seqres.full > > + return 1 > > + fi > > + > > + md5sum --quiet -c $tmp.manifest || \ > > + echo "fs contents ^^^ changed after adding $feat" > > + > > + iam="check" _check_scratch_fs || \ > > + echo "scratch fs check failed after adding $feat" > > + > > + # Try to mount the fs in case the check unmounted it > > + _try_scratch_mount &>> $seqres.full > > + > > + scratch_fsstress > > + > > + iam="check" _check_scratch_fs || \ > > + echo "scratch fs check failed after exercising $feat" > > + > > + # Try to unmount the fs in case the check didn't > > + _scratch_unmount &>> $seqres.full > > + return 0 > > +} > > + > > +# Create a list of fs features in the order that support for them was added > > +# to the kernel driver. For each feature upgrade test, we enable all the > > +# features that came before it and none of the ones after, which means we're > > +# testing incremental migrations. We start each run with a clean fs so that > > +# errors and unsatisfied requirements (log size, root ino position, etc) in one > > +# upgrade don't spread failure to the rest of the tests. > > +FEATURES=() > > +if rt_configured; then > > + check_repair_upgrade finobt && FEATURES+=("finobt") > > + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") > > + check_repair_upgrade bigtime && FEATURES+=("bigtime") > > +else > > + check_repair_upgrade finobt && FEATURES+=("finobt") > > + check_repair_upgrade rmapbt && FEATURES+=("rmapbt") > > + check_repair_upgrade reflink && FEATURES+=("reflink") > > + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") > > + check_repair_upgrade bigtime && FEATURES+=("bigtime") > > +fi > > + > > +test "${#FEATURES[@]}" -eq 0 && \ > > + _notrun "xfs_repair does not know how to add V5 features" > > + > > +declare -A FEATURE_STATE > > +for f in "${FEATURES[@]}"; do > > + FEATURE_STATE["$f"]=0 > > +done > > + > > +for feat in "${FEATURES[@]}"; do > > + echo "-----------------------" >> $seqres.full > > + > > + upgrade_start_message "$feat" | tee -a $seqres.full /dev/ttyprintk > /dev/null > > + > > + opts="$(compute_mkfs_options)" > > + echo "mkfs.xfs $opts" >> $seqres.full > > + > > + # Format filesystem > > + MKFS_OPTIONS="$opts" _scratch_mkfs &>> $seqres.full > > + res=$? > > + outcome="mkfs returns $res for $feat upgrade test" > > + echo "$outcome" >> $seqres.full > > + if [ $res -ne 0 ]; then > > + echo "$outcome" > > + continue > > + fi > > + > > + # Create some files to make things interesting. > > + pre_exercise "$feat" || break > > + > > + # Upgrade the fs > > + _scratch_xfs_repair -c "${feat}=1" &> $tmp.upgrade > > + res=$? > > + cat $tmp.upgrade >> $seqres.full > > + grep -q "^Adding" $tmp.upgrade || \ > > + echo "xfs_repair ignored command to add $feat" > > + > > + outcome="xfs_repair returns $res while adding $feat" > > + echo "$outcome" >> $seqres.full > > + if [ $res -ne 0 ]; then > > + # Couldn't upgrade filesystem, move on to the next feature. > > + FEATURE_STATE["$feat"]=1 > > + continue > > + fi > > + > > + # Make sure repair runs cleanly afterwards > > + _scratch_xfs_repair -n &>> $seqres.full > > + res=$? > > + outcome="xfs_repair -n returns $res after adding $feat" > > + echo "$outcome" >> $seqres.full > > + if [ $res -ne 0 ]; then > > + echo "$outcome" > > + fi > > + > > + # Make sure we can still exercise the filesystem. > > + post_exercise "$feat" || break > > + > > + # Update feature state for next run > > + FEATURE_STATE["$feat"]=1 > > +done > > + > > +# success, all done > > +echo Silence is golden. > > +status=0 > > +exit > > diff --git a/tests/xfs/769.out b/tests/xfs/769.out > > new file mode 100644 > > index 0000000000..332432db97 > > --- /dev/null > > +++ b/tests/xfs/769.out > > @@ -0,0 +1,2 @@ > > +QA output created by 769 > > +Silence is golden. > > > ^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 1/1] xfs: test upgrading old features 2023-03-06 16:41 ` Darrick J. Wong @ 2023-03-06 16:54 ` Zorro Lang 2023-03-06 23:14 ` Darrick J. Wong 0 siblings, 1 reply; 68+ messages in thread From: Zorro Lang @ 2023-03-06 16:54 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Mon, Mar 06, 2023 at 08:41:55AM -0800, Darrick J. Wong wrote: > On Mon, Mar 06, 2023 at 11:56:11PM +0800, Zorro Lang wrote: > > On Fri, Dec 30, 2022 at 02:20:29PM -0800, Darrick J. Wong wrote: > > > From: Darrick J. Wong <djwong@kernel.org> > > > > > > Test the ability to add older v5 features. > > > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > > --- > > > tests/xfs/769 | 248 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > tests/xfs/769.out | 2 > > > 2 files changed, 250 insertions(+) > > > create mode 100755 tests/xfs/769 > > > create mode 100644 tests/xfs/769.out > > > > > > > > > diff --git a/tests/xfs/769 b/tests/xfs/769 > > > new file mode 100755 > > > index 0000000000..7613048f52 > > > --- /dev/null > > > +++ b/tests/xfs/769 > > > @@ -0,0 +1,248 @@ > > > +#! /bin/bash > > > +# SPDX-License-Identifier: GPL-2.0-or-later > > > +# Copyright (c) 2022 Oracle. All Rights Reserved. > > > +# > > > +# FS QA Test No. 769 > > > +# > > > +# Test upgrading filesystems with new features. > > > +# > > > +. ./common/preamble > > > +_begin_fstest auto mkfs repair > > > + > > > +# Import common functions. > > > +. ./common/filter > > > +. ./common/populate > > > + > > > +# real QA test starts here > > > +_supported_fs xfs > > > + > > > +test -w /dev/ttyprintk || _notrun "test requires writable /dev/ttyprintk" > > > > Hi Darrick, > > > > I'm not sure why /dev/ttyprintk is necessary. I think sometimes we might not > > have this driver, but has /dev/kmsg. Can /dev/kmsg be a replacement of > > ttyprintk ? > > The kernel logging here is for debugging purposes -- if the upgrade > corrupts the fs and the kernel splats, whoever triages the failure can > have at least some clue as to where things went wrong. > > I think the _notrun line should go away though. If only for debugging, better to let this case keep running without ttyprintk. Or it'll _notrun on many systems without ttyprintk driver. There's a log_dmesg() helper in common/rc, the /dev/kmsg might be more often. As you've called *_require_check_dmesg*, it helps to check kmsg file. So how about use it? Or making a helper to write messages to ttyprintk, then fallback to kmsg if ttyprintk is missing ? Thanks, Zorro > > --D > > > Thanks, > > Zorro > > > > > > > +_require_check_dmesg > > > +_require_scratch_nocheck > > > +_require_scratch_xfs_crc > > > + > > > +# Does repair know how to add a particular feature to a filesystem? > > > +check_repair_upgrade() > > > +{ > > > + $XFS_REPAIR_PROG -c "$1=narf" 2>&1 | \ > > > + grep -q 'unknown option' && return 1 > > > + return 0 > > > +} > > > + > > > +# Are we configured for realtime? > > > +rt_configured() > > > +{ > > > + test "$USE_EXTERNAL" = "yes" && test -n "$SCRATCH_RTDEV" > > > +} > > > + > > > +# Compute the MKFS_OPTIONS string for a particular feature upgrade test > > > +compute_mkfs_options() > > > +{ > > > + local m_opts="" > > > + local caller_options="$MKFS_OPTIONS" > > > + > > > + for feat in "${FEATURES[@]}"; do > > > + local feat_state="${FEATURE_STATE["${feat}"]}" > > > + > > > + if echo "$caller_options" | grep -E -w -q "${feat}=[0-9]*"; then > > > + # Change the caller's options > > > + caller_options="$(echo "$caller_options" | \ > > > + sed -e "s/\([^[:alnum:]]\)${feat}=[0-9]*/\1${feat}=${feat_state}/g")" > > > + else > > > + # Add it to our list of new mkfs flags > > > + m_opts="${feat}=${feat_state},${m_opts}" > > > + fi > > > + done > > > + > > > + test -n "$m_opts" && m_opts=" -m $m_opts" > > > + > > > + echo "$caller_options$m_opts" > > > +} > > > + > > > +# Log the start of an upgrade. > > > +function upgrade_start_message() > > > +{ > > > + local feat="$1" > > > + > > > + echo "Add $feat to filesystem" > > > +} > > > + > > > +# Find dmesg log messages since we started a particular upgrade test > > > +function dmesg_since_feature_upgrade_start() > > > +{ > > > + local feat_logmsg="$(upgrade_start_message "$1")" > > > + > > > + # search the dmesg log of last run of $seqnum for possible failures > > > + # use sed \cregexpc address type, since $seqnum contains "/" > > > + dmesg | \ > > > + tac | \ > > > + sed -ne "0,\#run fstests $seqnum at $date_time#p" | \ > > > + sed -ne "0,\#${feat_logmsg}#p" | \ > > > + tac > > > +} > > > + > > > +# Did the mount fail because this feature is not supported? > > > +function feature_unsupported() > > > +{ > > > + local feat="$1" > > > + > > > + dmesg_since_feature_upgrade_start "$feat" | \ > > > + grep -q 'has unknown.*features' > > > +} > > > + > > > +# Exercise the scratch fs > > > +function scratch_fsstress() > > > +{ > > > + echo moo > $SCRATCH_MNT/sample.txt > > > + $FSSTRESS_PROG -n $((TIME_FACTOR * 1000)) -p $((LOAD_FACTOR * 4)) \ > > > + -d $SCRATCH_MNT/data >> $seqres.full > > > +} > > > + > > > +# Exercise the filesystem a little bit and emit a manifest. > > > +function pre_exercise() > > > +{ > > > + local feat="$1" > > > + > > > + _try_scratch_mount &> $tmp.mount > > > + res=$? > > > + # If the kernel doesn't support the filesystem even after a > > > + # fresh format, skip the rest of the upgrade test quietly. > > > + if [ $res -eq 32 ] && feature_unsupported "$feat"; then > > > + echo "mount failed due to unsupported feature $feat" >> $seqres.full > > > + return 1 > > > + fi > > > + if [ $res -ne 0 ]; then > > > + cat $tmp.mount > > > + echo "mount failed with $res before upgrading to $feat" | \ > > > + tee -a $seqres.full > > > + return 1 > > > + fi > > > + > > > + scratch_fsstress > > > + find $SCRATCH_MNT -type f -print0 | xargs -r -0 md5sum > $tmp.manifest > > > + _scratch_unmount > > > + return 0 > > > +} > > > + > > > +# Check the manifest and exercise the filesystem more > > > +function post_exercise() > > > +{ > > > + local feat="$1" > > > + > > > + _try_scratch_mount &> $tmp.mount > > > + res=$? > > > + # If the kernel doesn't support the filesystem even after a > > > + # fresh format, skip the rest of the upgrade test quietly. > > > + if [ $res -eq 32 ] && feature_unsupported "$feat"; then > > > + echo "mount failed due to unsupported feature $feat" >> $seqres.full > > > + return 1 > > > + fi > > > + if [ $res -ne 0 ]; then > > > + cat $tmp.mount > > > + echo "mount failed with $res after upgrading to $feat" | \ > > > + tee -a $seqres.full > > > + return 1 > > > + fi > > > + > > > + md5sum --quiet -c $tmp.manifest || \ > > > + echo "fs contents ^^^ changed after adding $feat" > > > + > > > + iam="check" _check_scratch_fs || \ > > > + echo "scratch fs check failed after adding $feat" > > > + > > > + # Try to mount the fs in case the check unmounted it > > > + _try_scratch_mount &>> $seqres.full > > > + > > > + scratch_fsstress > > > + > > > + iam="check" _check_scratch_fs || \ > > > + echo "scratch fs check failed after exercising $feat" > > > + > > > + # Try to unmount the fs in case the check didn't > > > + _scratch_unmount &>> $seqres.full > > > + return 0 > > > +} > > > + > > > +# Create a list of fs features in the order that support for them was added > > > +# to the kernel driver. For each feature upgrade test, we enable all the > > > +# features that came before it and none of the ones after, which means we're > > > +# testing incremental migrations. We start each run with a clean fs so that > > > +# errors and unsatisfied requirements (log size, root ino position, etc) in one > > > +# upgrade don't spread failure to the rest of the tests. > > > +FEATURES=() > > > +if rt_configured; then > > > + check_repair_upgrade finobt && FEATURES+=("finobt") > > > + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") > > > + check_repair_upgrade bigtime && FEATURES+=("bigtime") > > > +else > > > + check_repair_upgrade finobt && FEATURES+=("finobt") > > > + check_repair_upgrade rmapbt && FEATURES+=("rmapbt") > > > + check_repair_upgrade reflink && FEATURES+=("reflink") > > > + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") > > > + check_repair_upgrade bigtime && FEATURES+=("bigtime") > > > +fi > > > + > > > +test "${#FEATURES[@]}" -eq 0 && \ > > > + _notrun "xfs_repair does not know how to add V5 features" > > > + > > > +declare -A FEATURE_STATE > > > +for f in "${FEATURES[@]}"; do > > > + FEATURE_STATE["$f"]=0 > > > +done > > > + > > > +for feat in "${FEATURES[@]}"; do > > > + echo "-----------------------" >> $seqres.full > > > + > > > + upgrade_start_message "$feat" | tee -a $seqres.full /dev/ttyprintk > /dev/null > > > + > > > + opts="$(compute_mkfs_options)" > > > + echo "mkfs.xfs $opts" >> $seqres.full > > > + > > > + # Format filesystem > > > + MKFS_OPTIONS="$opts" _scratch_mkfs &>> $seqres.full > > > + res=$? > > > + outcome="mkfs returns $res for $feat upgrade test" > > > + echo "$outcome" >> $seqres.full > > > + if [ $res -ne 0 ]; then > > > + echo "$outcome" > > > + continue > > > + fi > > > + > > > + # Create some files to make things interesting. > > > + pre_exercise "$feat" || break > > > + > > > + # Upgrade the fs > > > + _scratch_xfs_repair -c "${feat}=1" &> $tmp.upgrade > > > + res=$? > > > + cat $tmp.upgrade >> $seqres.full > > > + grep -q "^Adding" $tmp.upgrade || \ > > > + echo "xfs_repair ignored command to add $feat" > > > + > > > + outcome="xfs_repair returns $res while adding $feat" > > > + echo "$outcome" >> $seqres.full > > > + if [ $res -ne 0 ]; then > > > + # Couldn't upgrade filesystem, move on to the next feature. > > > + FEATURE_STATE["$feat"]=1 > > > + continue > > > + fi > > > + > > > + # Make sure repair runs cleanly afterwards > > > + _scratch_xfs_repair -n &>> $seqres.full > > > + res=$? > > > + outcome="xfs_repair -n returns $res after adding $feat" > > > + echo "$outcome" >> $seqres.full > > > + if [ $res -ne 0 ]; then > > > + echo "$outcome" > > > + fi > > > + > > > + # Make sure we can still exercise the filesystem. > > > + post_exercise "$feat" || break > > > + > > > + # Update feature state for next run > > > + FEATURE_STATE["$feat"]=1 > > > +done > > > + > > > +# success, all done > > > +echo Silence is golden. > > > +status=0 > > > +exit > > > diff --git a/tests/xfs/769.out b/tests/xfs/769.out > > > new file mode 100644 > > > index 0000000000..332432db97 > > > --- /dev/null > > > +++ b/tests/xfs/769.out > > > @@ -0,0 +1,2 @@ > > > +QA output created by 769 > > > +Silence is golden. > > > > > > ^ permalink raw reply [flat|nested] 68+ messages in thread
* Re: [PATCH 1/1] xfs: test upgrading old features 2023-03-06 16:54 ` Zorro Lang @ 2023-03-06 23:14 ` Darrick J. Wong 0 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2023-03-06 23:14 UTC (permalink / raw) To: Zorro Lang; +Cc: linux-xfs, fstests On Tue, Mar 07, 2023 at 12:54:02AM +0800, Zorro Lang wrote: > On Mon, Mar 06, 2023 at 08:41:55AM -0800, Darrick J. Wong wrote: > > On Mon, Mar 06, 2023 at 11:56:11PM +0800, Zorro Lang wrote: > > > On Fri, Dec 30, 2022 at 02:20:29PM -0800, Darrick J. Wong wrote: > > > > From: Darrick J. Wong <djwong@kernel.org> > > > > > > > > Test the ability to add older v5 features. > > > > > > > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > > > > --- > > > > tests/xfs/769 | 248 +++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > tests/xfs/769.out | 2 > > > > 2 files changed, 250 insertions(+) > > > > create mode 100755 tests/xfs/769 > > > > create mode 100644 tests/xfs/769.out > > > > > > > > > > > > diff --git a/tests/xfs/769 b/tests/xfs/769 > > > > new file mode 100755 > > > > index 0000000000..7613048f52 > > > > --- /dev/null > > > > +++ b/tests/xfs/769 > > > > @@ -0,0 +1,248 @@ > > > > +#! /bin/bash > > > > +# SPDX-License-Identifier: GPL-2.0-or-later > > > > +# Copyright (c) 2022 Oracle. All Rights Reserved. > > > > +# > > > > +# FS QA Test No. 769 > > > > +# > > > > +# Test upgrading filesystems with new features. > > > > +# > > > > +. ./common/preamble > > > > +_begin_fstest auto mkfs repair > > > > + > > > > +# Import common functions. > > > > +. ./common/filter > > > > +. ./common/populate > > > > + > > > > +# real QA test starts here > > > > +_supported_fs xfs > > > > + > > > > +test -w /dev/ttyprintk || _notrun "test requires writable /dev/ttyprintk" > > > > > > Hi Darrick, > > > > > > I'm not sure why /dev/ttyprintk is necessary. I think sometimes we might not > > > have this driver, but has /dev/kmsg. Can /dev/kmsg be a replacement of > > > ttyprintk ? > > > > The kernel logging here is for debugging purposes -- if the upgrade > > corrupts the fs and the kernel splats, whoever triages the failure can > > have at least some clue as to where things went wrong. > > > > I think the _notrun line should go away though. > > If only for debugging, better to let this case keep running without ttyprintk. > Or it'll _notrun on many systems without ttyprintk driver. > > There's a log_dmesg() helper in common/rc, the /dev/kmsg might be more often. > As you've called *_require_check_dmesg*, it helps to check kmsg file. So > how about use it? Or making a helper to write messages to ttyprintk, then > fallback to kmsg if ttyprintk is missing ? Yeah, I can hoist the ttyprintk usage into a helper that emits to ttyprintk or kmsg depending on what's available. --D > Thanks, > Zorro > > > > > --D > > > > > Thanks, > > > Zorro > > > > > > > > > > +_require_check_dmesg > > > > +_require_scratch_nocheck > > > > +_require_scratch_xfs_crc > > > > + > > > > +# Does repair know how to add a particular feature to a filesystem? > > > > +check_repair_upgrade() > > > > +{ > > > > + $XFS_REPAIR_PROG -c "$1=narf" 2>&1 | \ > > > > + grep -q 'unknown option' && return 1 > > > > + return 0 > > > > +} > > > > + > > > > +# Are we configured for realtime? > > > > +rt_configured() > > > > +{ > > > > + test "$USE_EXTERNAL" = "yes" && test -n "$SCRATCH_RTDEV" > > > > +} > > > > + > > > > +# Compute the MKFS_OPTIONS string for a particular feature upgrade test > > > > +compute_mkfs_options() > > > > +{ > > > > + local m_opts="" > > > > + local caller_options="$MKFS_OPTIONS" > > > > + > > > > + for feat in "${FEATURES[@]}"; do > > > > + local feat_state="${FEATURE_STATE["${feat}"]}" > > > > + > > > > + if echo "$caller_options" | grep -E -w -q "${feat}=[0-9]*"; then > > > > + # Change the caller's options > > > > + caller_options="$(echo "$caller_options" | \ > > > > + sed -e "s/\([^[:alnum:]]\)${feat}=[0-9]*/\1${feat}=${feat_state}/g")" > > > > + else > > > > + # Add it to our list of new mkfs flags > > > > + m_opts="${feat}=${feat_state},${m_opts}" > > > > + fi > > > > + done > > > > + > > > > + test -n "$m_opts" && m_opts=" -m $m_opts" > > > > + > > > > + echo "$caller_options$m_opts" > > > > +} > > > > + > > > > +# Log the start of an upgrade. > > > > +function upgrade_start_message() > > > > +{ > > > > + local feat="$1" > > > > + > > > > + echo "Add $feat to filesystem" > > > > +} > > > > + > > > > +# Find dmesg log messages since we started a particular upgrade test > > > > +function dmesg_since_feature_upgrade_start() > > > > +{ > > > > + local feat_logmsg="$(upgrade_start_message "$1")" > > > > + > > > > + # search the dmesg log of last run of $seqnum for possible failures > > > > + # use sed \cregexpc address type, since $seqnum contains "/" > > > > + dmesg | \ > > > > + tac | \ > > > > + sed -ne "0,\#run fstests $seqnum at $date_time#p" | \ > > > > + sed -ne "0,\#${feat_logmsg}#p" | \ > > > > + tac > > > > +} > > > > + > > > > +# Did the mount fail because this feature is not supported? > > > > +function feature_unsupported() > > > > +{ > > > > + local feat="$1" > > > > + > > > > + dmesg_since_feature_upgrade_start "$feat" | \ > > > > + grep -q 'has unknown.*features' > > > > +} > > > > + > > > > +# Exercise the scratch fs > > > > +function scratch_fsstress() > > > > +{ > > > > + echo moo > $SCRATCH_MNT/sample.txt > > > > + $FSSTRESS_PROG -n $((TIME_FACTOR * 1000)) -p $((LOAD_FACTOR * 4)) \ > > > > + -d $SCRATCH_MNT/data >> $seqres.full > > > > +} > > > > + > > > > +# Exercise the filesystem a little bit and emit a manifest. > > > > +function pre_exercise() > > > > +{ > > > > + local feat="$1" > > > > + > > > > + _try_scratch_mount &> $tmp.mount > > > > + res=$? > > > > + # If the kernel doesn't support the filesystem even after a > > > > + # fresh format, skip the rest of the upgrade test quietly. > > > > + if [ $res -eq 32 ] && feature_unsupported "$feat"; then > > > > + echo "mount failed due to unsupported feature $feat" >> $seqres.full > > > > + return 1 > > > > + fi > > > > + if [ $res -ne 0 ]; then > > > > + cat $tmp.mount > > > > + echo "mount failed with $res before upgrading to $feat" | \ > > > > + tee -a $seqres.full > > > > + return 1 > > > > + fi > > > > + > > > > + scratch_fsstress > > > > + find $SCRATCH_MNT -type f -print0 | xargs -r -0 md5sum > $tmp.manifest > > > > + _scratch_unmount > > > > + return 0 > > > > +} > > > > + > > > > +# Check the manifest and exercise the filesystem more > > > > +function post_exercise() > > > > +{ > > > > + local feat="$1" > > > > + > > > > + _try_scratch_mount &> $tmp.mount > > > > + res=$? > > > > + # If the kernel doesn't support the filesystem even after a > > > > + # fresh format, skip the rest of the upgrade test quietly. > > > > + if [ $res -eq 32 ] && feature_unsupported "$feat"; then > > > > + echo "mount failed due to unsupported feature $feat" >> $seqres.full > > > > + return 1 > > > > + fi > > > > + if [ $res -ne 0 ]; then > > > > + cat $tmp.mount > > > > + echo "mount failed with $res after upgrading to $feat" | \ > > > > + tee -a $seqres.full > > > > + return 1 > > > > + fi > > > > + > > > > + md5sum --quiet -c $tmp.manifest || \ > > > > + echo "fs contents ^^^ changed after adding $feat" > > > > + > > > > + iam="check" _check_scratch_fs || \ > > > > + echo "scratch fs check failed after adding $feat" > > > > + > > > > + # Try to mount the fs in case the check unmounted it > > > > + _try_scratch_mount &>> $seqres.full > > > > + > > > > + scratch_fsstress > > > > + > > > > + iam="check" _check_scratch_fs || \ > > > > + echo "scratch fs check failed after exercising $feat" > > > > + > > > > + # Try to unmount the fs in case the check didn't > > > > + _scratch_unmount &>> $seqres.full > > > > + return 0 > > > > +} > > > > + > > > > +# Create a list of fs features in the order that support for them was added > > > > +# to the kernel driver. For each feature upgrade test, we enable all the > > > > +# features that came before it and none of the ones after, which means we're > > > > +# testing incremental migrations. We start each run with a clean fs so that > > > > +# errors and unsatisfied requirements (log size, root ino position, etc) in one > > > > +# upgrade don't spread failure to the rest of the tests. > > > > +FEATURES=() > > > > +if rt_configured; then > > > > + check_repair_upgrade finobt && FEATURES+=("finobt") > > > > + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") > > > > + check_repair_upgrade bigtime && FEATURES+=("bigtime") > > > > +else > > > > + check_repair_upgrade finobt && FEATURES+=("finobt") > > > > + check_repair_upgrade rmapbt && FEATURES+=("rmapbt") > > > > + check_repair_upgrade reflink && FEATURES+=("reflink") > > > > + check_repair_upgrade inobtcount && FEATURES+=("inobtcount") > > > > + check_repair_upgrade bigtime && FEATURES+=("bigtime") > > > > +fi > > > > + > > > > +test "${#FEATURES[@]}" -eq 0 && \ > > > > + _notrun "xfs_repair does not know how to add V5 features" > > > > + > > > > +declare -A FEATURE_STATE > > > > +for f in "${FEATURES[@]}"; do > > > > + FEATURE_STATE["$f"]=0 > > > > +done > > > > + > > > > +for feat in "${FEATURES[@]}"; do > > > > + echo "-----------------------" >> $seqres.full > > > > + > > > > + upgrade_start_message "$feat" | tee -a $seqres.full /dev/ttyprintk > /dev/null > > > > + > > > > + opts="$(compute_mkfs_options)" > > > > + echo "mkfs.xfs $opts" >> $seqres.full > > > > + > > > > + # Format filesystem > > > > + MKFS_OPTIONS="$opts" _scratch_mkfs &>> $seqres.full > > > > + res=$? > > > > + outcome="mkfs returns $res for $feat upgrade test" > > > > + echo "$outcome" >> $seqres.full > > > > + if [ $res -ne 0 ]; then > > > > + echo "$outcome" > > > > + continue > > > > + fi > > > > + > > > > + # Create some files to make things interesting. > > > > + pre_exercise "$feat" || break > > > > + > > > > + # Upgrade the fs > > > > + _scratch_xfs_repair -c "${feat}=1" &> $tmp.upgrade > > > > + res=$? > > > > + cat $tmp.upgrade >> $seqres.full > > > > + grep -q "^Adding" $tmp.upgrade || \ > > > > + echo "xfs_repair ignored command to add $feat" > > > > + > > > > + outcome="xfs_repair returns $res while adding $feat" > > > > + echo "$outcome" >> $seqres.full > > > > + if [ $res -ne 0 ]; then > > > > + # Couldn't upgrade filesystem, move on to the next feature. > > > > + FEATURE_STATE["$feat"]=1 > > > > + continue > > > > + fi > > > > + > > > > + # Make sure repair runs cleanly afterwards > > > > + _scratch_xfs_repair -n &>> $seqres.full > > > > + res=$? > > > > + outcome="xfs_repair -n returns $res after adding $feat" > > > > + echo "$outcome" >> $seqres.full > > > > + if [ $res -ne 0 ]; then > > > > + echo "$outcome" > > > > + fi > > > > + > > > > + # Make sure we can still exercise the filesystem. > > > > + post_exercise "$feat" || break > > > > + > > > > + # Update feature state for next run > > > > + FEATURE_STATE["$feat"]=1 > > > > +done > > > > + > > > > +# success, all done > > > > +echo Silence is golden. > > > > +status=0 > > > > +exit > > > > diff --git a/tests/xfs/769.out b/tests/xfs/769.out > > > > new file mode 100644 > > > > index 0000000000..332432db97 > > > > --- /dev/null > > > > +++ b/tests/xfs/769.out > > > > @@ -0,0 +1,2 @@ > > > > +QA output created by 769 > > > > +Silence is golden. > > > > > > > > > > ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCHSET v1.0 0/9] fstests: test XFS metadata directories 2022-12-30 21:14 [NYE DELUGE 3/4] xfs: modernize the realtime volume Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET 0/1] fstests: test upgrading older features Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/9] xfs/{030,033,178}: forcibly disable metadata directory trees Darrick J. Wong ` (8 more replies) 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: support metadump to external devices Darrick J. Wong ` (5 subsequent siblings) 7 siblings, 9 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, Adjust fstests as needed to support the XFS metadata directory feature, and add some new tests for online fsck and fuzz testing of the ondisk metadata. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=metadir xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=metadir fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=metadir xfsdocs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=metadir --- common/filter | 7 +++- common/repair | 4 ++ common/xfs | 90 +++++++++++++++++++++++++++++++++++++++++++++++++++- tests/xfs/007 | 16 +++++---- tests/xfs/030 | 1 + tests/xfs/033 | 1 + tests/xfs/050 | 1 + tests/xfs/122.out | 1 + tests/xfs/153 | 1 + tests/xfs/1546 | 37 +++++++++++++++++++++ tests/xfs/1546.out | 4 ++ tests/xfs/1547 | 37 +++++++++++++++++++++ tests/xfs/1547.out | 4 ++ tests/xfs/1548 | 37 +++++++++++++++++++++ tests/xfs/1548.out | 4 ++ tests/xfs/1549 | 38 ++++++++++++++++++++++ tests/xfs/1549.out | 4 ++ tests/xfs/1550 | 37 +++++++++++++++++++++ tests/xfs/1550.out | 4 ++ tests/xfs/1551 | 37 +++++++++++++++++++++ tests/xfs/1551.out | 4 ++ tests/xfs/1552 | 37 +++++++++++++++++++++ tests/xfs/1552.out | 4 ++ tests/xfs/1553 | 38 ++++++++++++++++++++++ tests/xfs/1553.out | 4 ++ tests/xfs/1562 | 9 +---- tests/xfs/1563 | 9 +---- tests/xfs/1564 | 9 +---- tests/xfs/1565 | 9 +---- tests/xfs/1566 | 9 +---- tests/xfs/1567 | 9 +---- tests/xfs/1568 | 9 +---- tests/xfs/1569 | 9 +---- tests/xfs/178 | 1 + tests/xfs/206 | 3 +- tests/xfs/299 | 1 + tests/xfs/330 | 6 +++ tests/xfs/509 | 21 +++++++++++- tests/xfs/529 | 5 +-- tests/xfs/530 | 6 +-- tests/xfs/769 | 2 + 41 files changed, 491 insertions(+), 78 deletions(-) create mode 100755 tests/xfs/1546 create mode 100644 tests/xfs/1546.out create mode 100755 tests/xfs/1547 create mode 100644 tests/xfs/1547.out create mode 100755 tests/xfs/1548 create mode 100644 tests/xfs/1548.out create mode 100755 tests/xfs/1549 create mode 100644 tests/xfs/1549.out create mode 100755 tests/xfs/1550 create mode 100644 tests/xfs/1550.out create mode 100755 tests/xfs/1551 create mode 100644 tests/xfs/1551.out create mode 100755 tests/xfs/1552 create mode 100644 tests/xfs/1552.out create mode 100755 tests/xfs/1553 create mode 100644 tests/xfs/1553.out ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCH 3/9] xfs/{030,033,178}: forcibly disable metadata directory trees 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/9] xfs/122: fix metadirino Darrick J. Wong ` (7 subsequent siblings) 8 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The golden output for thests tests encode the xfs_repair output when we fuzz various parts of the filesystem. With metadata directory trees enabled, however, the golden output changes dramatically to reflect reconstruction of the metadata directory tree. To avoid regressions, add a helper to force metadata directories off via MKFS_OPTIONS. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 13 +++++++++++++ tests/xfs/030 | 1 + tests/xfs/033 | 1 + tests/xfs/178 | 1 + 4 files changed, 16 insertions(+) diff --git a/common/xfs b/common/xfs index dafbd1b874..0f69d3eb18 100644 --- a/common/xfs +++ b/common/xfs @@ -1770,3 +1770,16 @@ _scratch_xfs_find_metafile() echo "${selector}" return 0 } + +# Force metadata directories off. +_scratch_xfs_force_no_metadir() +{ + if echo "$MKFS_OPTIONS" | grep -q 'metadir='; then + MKFS_OPTIONS="$(echo "$MKFS_OPTIONS" | sed -e 's/metadir=\([01]\)/metadir=0/g')" + return + fi + + if grep -q 'metadir=' $MKFS_XFS_PROG; then + MKFS_OPTIONS="-m metadir=0 $MKFS_OPTIONS" + fi +} diff --git a/tests/xfs/030 b/tests/xfs/030 index 201a901579..a62ea4fab3 100755 --- a/tests/xfs/030 +++ b/tests/xfs/030 @@ -50,6 +50,7 @@ _supported_fs xfs _require_scratch _require_no_large_scratch_dev +_scratch_xfs_force_no_metadir DSIZE="-dsize=100m,agcount=6" diff --git a/tests/xfs/033 b/tests/xfs/033 index ef5dc4fa36..e886c15082 100755 --- a/tests/xfs/033 +++ b/tests/xfs/033 @@ -53,6 +53,7 @@ _supported_fs xfs _require_scratch _require_no_large_scratch_dev +_scratch_xfs_force_no_metadir # devzero blows away 512byte blocks, so make 512byte inodes (at least) _scratch_mkfs_xfs | _filter_mkfs 2>$tmp.mkfs >/dev/null diff --git a/tests/xfs/178 b/tests/xfs/178 index a65197cde3..72b4d347fd 100755 --- a/tests/xfs/178 +++ b/tests/xfs/178 @@ -45,6 +45,7 @@ _supported_fs xfs # fix filesystem, new mkfs.xfs will be fine. _require_scratch +_scratch_xfs_force_no_metadir _scratch_mkfs_xfs | _filter_mkfs 2>$tmp.mkfs test "${PIPESTATUS[0]}" -eq 0 || _fail "mkfs failed!" ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 1/9] xfs/122: fix metadirino 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/9] xfs/{030,033,178}: forcibly disable metadata directory trees Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 2/9] various: fix finding metadata inode numbers when metadir is enabled Darrick J. Wong ` (6 subsequent siblings) 8 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Fix xfs/122 to work properly with metadirino. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/122.out | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/xfs/122.out b/tests/xfs/122.out index 21549db7fd..eee6c1ee6d 100644 --- a/tests/xfs/122.out +++ b/tests/xfs/122.out @@ -35,6 +35,7 @@ offsetof(xfs_sb_t, sb_logsunit) = 196 offsetof(xfs_sb_t, sb_lsn) = 240 offsetof(xfs_sb_t, sb_magicnum) = 0 offsetof(xfs_sb_t, sb_meta_uuid) = 248 +offsetof(xfs_sb_t, sb_metadirino) = 264 offsetof(xfs_sb_t, sb_pquotino) = 232 offsetof(xfs_sb_t, sb_qflags) = 176 offsetof(xfs_sb_t, sb_rblocks) = 16 ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 2/9] various: fix finding metadata inode numbers when metadir is enabled 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/9] xfs/{030,033,178}: forcibly disable metadata directory trees Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/9] xfs/122: fix metadirino Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2023-03-06 16:41 ` Zorro Lang 2022-12-30 22:20 ` [PATCH 4/9] common/repair: patch up repair sb inode value complaints Darrick J. Wong ` (5 subsequent siblings) 8 siblings, 1 reply; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> There are a number of tests that use xfs_db to examine the contents of metadata inodes to check correct functioning. The logic is scattered everywhere and won't work with metadata directory trees, so make a shared helper to find these inodes. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 32 ++++++++++++++++++++++++++++++-- tests/xfs/007 | 16 +++++++++------- tests/xfs/1562 | 9 ++------- tests/xfs/1563 | 9 ++------- tests/xfs/1564 | 9 ++------- tests/xfs/1565 | 9 ++------- tests/xfs/1566 | 9 ++------- tests/xfs/1567 | 9 ++------- tests/xfs/1568 | 9 ++------- tests/xfs/1569 | 9 ++------- tests/xfs/529 | 5 ++--- tests/xfs/530 | 6 ++---- 12 files changed, 59 insertions(+), 72 deletions(-) diff --git a/common/xfs b/common/xfs index 8b365ad18b..dafbd1b874 100644 --- a/common/xfs +++ b/common/xfs @@ -1396,7 +1396,7 @@ _scratch_get_bmx_prefix() { _scratch_get_iext_count() { - local ino=$1 + local selector=$1 local whichfork=$2 local field="" @@ -1411,7 +1411,7 @@ _scratch_get_iext_count() return 1 esac - _scratch_xfs_get_metadata_field $field "inode $ino" + _scratch_xfs_get_metadata_field $field "$selector" } # @@ -1742,3 +1742,31 @@ _require_xfs_scratch_atomicswap() _notrun "atomicswap dependencies not supported by scratch filesystem type: $FSTYP" _scratch_unmount } + +# Find a metadata file within an xfs filesystem. The sole argument is the +# name of the field within the superblock. +_scratch_xfs_find_metafile() +{ + local metafile="$1" + local selector= + + if ! _check_scratch_xfs_features METADIR > /dev/null; then + sb_field="$(_scratch_xfs_get_sb_field "$metafile")" + if echo "$sb_field" | grep -q -w 'not found'; then + return 1 + fi + selector="inode $sb_field" + else + case "${metafile}" in + "rootino") selector="path /";; + "uquotino") selector="path -m /quota/user";; + "gquotino") selector="path -m /quota/group";; + "pquotino") selector="path -m /quota/project";; + "rbmino") selector="path -m /realtime/bitmap";; + "rsumino") selector="path -m /realtime/summary";; + esac + fi + + echo "${selector}" + return 0 +} diff --git a/tests/xfs/007 b/tests/xfs/007 index 4f864100fd..6d6d828b13 100755 --- a/tests/xfs/007 +++ b/tests/xfs/007 @@ -22,6 +22,11 @@ _require_xfs_quota _scratch_mkfs_xfs | _filter_mkfs > /dev/null 2> $tmp.mkfs . $tmp.mkfs +get_qfile_nblocks() { + local selector="$(_scratch_xfs_find_metafile "$1")" + _scratch_xfs_db -c "$selector" -c "p core.nblocks" +} + do_test() { qino_1=$1 @@ -31,12 +36,9 @@ do_test() echo "*** umount" _scratch_unmount - QINO_1=`_scratch_xfs_get_sb_field $qino_1` - QINO_2=`_scratch_xfs_get_sb_field $qino_2` - echo "*** Usage before quotarm ***" - _scratch_xfs_db -c "inode $QINO_1" -c "p core.nblocks" - _scratch_xfs_db -c "inode $QINO_2" -c "p core.nblocks" + get_qfile_nblocks $qino_1 + get_qfile_nblocks $qino_2 _qmount echo "*** turn off $off_opts quotas" @@ -66,8 +68,8 @@ do_test() _scratch_unmount echo "*** Usage after quotarm ***" - _scratch_xfs_db -c "inode $QINO_1" -c "p core.nblocks" - _scratch_xfs_db -c "inode $QINO_2" -c "p core.nblocks" + get_qfile_nblocks $qino_1 + get_qfile_nblocks $qino_2 } # Test user and group first diff --git a/tests/xfs/1562 b/tests/xfs/1562 index 015209eeb2..1e5b6881ee 100755 --- a/tests/xfs/1562 +++ b/tests/xfs/1562 @@ -27,13 +27,8 @@ echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 echo "Fuzz rtbitmap" -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') -if [ -n "$is_metadir" ]; then - path=('path -m /realtime/0.bitmap') -else - path=('sb' 'addr rbmino') -fi -_scratch_xfs_fuzz_metadata '' 'online' "${path[@]}" 'dblock 0' >> $seqres.full +path="$(_scratch_xfs_find_metafile rbmino)" +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'dblock 0' >> $seqres.full echo "Done fuzzing rtbitmap" # success, all done diff --git a/tests/xfs/1563 b/tests/xfs/1563 index 2be0870a3d..a9da78106d 100755 --- a/tests/xfs/1563 +++ b/tests/xfs/1563 @@ -27,13 +27,8 @@ echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 echo "Fuzz rtsummary" -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') -if [ -n "$is_metadir" ]; then - path=('path -m /realtime/0.summary') -else - path=('sb' 'addr rsumino') -fi -_scratch_xfs_fuzz_metadata '' 'online' "${path[@]}" 'dblock 0' >> $seqres.full +path="$(_scratch_xfs_find_metafile rsumino)" +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'dblock 0' >> $seqres.full echo "Done fuzzing rtsummary" # success, all done diff --git a/tests/xfs/1564 b/tests/xfs/1564 index c0d10ff0e9..4482861d50 100755 --- a/tests/xfs/1564 +++ b/tests/xfs/1564 @@ -27,13 +27,8 @@ echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 echo "Fuzz rtbitmap" -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') -if [ -n "$is_metadir" ]; then - path=('path -m /realtime/0.bitmap') -else - path=('sb' 'addr rbmino') -fi -_scratch_xfs_fuzz_metadata '' 'offline' "${path[@]}" 'dblock 0' >> $seqres.full +path="$(_scratch_xfs_find_metafile rbmino)" +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'dblock 0' >> $seqres.full echo "Done fuzzing rtbitmap" # success, all done diff --git a/tests/xfs/1565 b/tests/xfs/1565 index 6b4186fb3c..c43ccd848e 100755 --- a/tests/xfs/1565 +++ b/tests/xfs/1565 @@ -27,13 +27,8 @@ echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 echo "Fuzz rtsummary" -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') -if [ -n "$is_metadir" ]; then - path=('path -m /realtime/0.summary') -else - path=('sb' 'addr rsumino') -fi -_scratch_xfs_fuzz_metadata '' 'offline' "${path[@]}" 'dblock 0' >> $seqres.full +path="$(_scratch_xfs_find_metafile rsumino)" +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'dblock 0' >> $seqres.full echo "Done fuzzing rtsummary" # success, all done diff --git a/tests/xfs/1566 b/tests/xfs/1566 index 8d0f61ae10..aad4fafb15 100755 --- a/tests/xfs/1566 +++ b/tests/xfs/1566 @@ -28,13 +28,8 @@ echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 echo "Fuzz rtbitmap" -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') -if [ -n "$is_metadir" ]; then - path=('path -m /realtime/0.bitmap') -else - path=('sb' 'addr rbmino') -fi -_scratch_xfs_fuzz_metadata '' 'both' "${path[@]}" 'dblock 0' >> $seqres.full +path="$(_scratch_xfs_find_metafile rbmino)" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'dblock 0' >> $seqres.full echo "Done fuzzing rtbitmap" # success, all done diff --git a/tests/xfs/1567 b/tests/xfs/1567 index 7dc2012b67..ff782fc239 100755 --- a/tests/xfs/1567 +++ b/tests/xfs/1567 @@ -28,13 +28,8 @@ echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 echo "Fuzz rtsummary" -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') -if [ -n "$is_metadir" ]; then - path=('path -m /realtime/0.summary') -else - path=('sb' 'addr rsumino') -fi -_scratch_xfs_fuzz_metadata '' 'both' "${path[@]}" 'dblock 0' >> $seqres.full +path="$(_scratch_xfs_find_metafile rsumino)" +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'dblock 0' >> $seqres.full echo "Done fuzzing rtsummary" # success, all done diff --git a/tests/xfs/1568 b/tests/xfs/1568 index c80640ef97..e2a28df58a 100755 --- a/tests/xfs/1568 +++ b/tests/xfs/1568 @@ -27,13 +27,8 @@ echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 echo "Fuzz rtbitmap" -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') -if [ -n "$is_metadir" ]; then - path=('path -m /realtime/0.bitmap') -else - path=('sb' 'addr rbmino') -fi -_scratch_xfs_fuzz_metadata '' 'none' "${path[@]}" 'dblock 0' >> $seqres.full +path="$(_scratch_xfs_find_metafile rbmino)" +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'dblock 0' >> $seqres.full echo "Done fuzzing rtbitmap" # success, all done diff --git a/tests/xfs/1569 b/tests/xfs/1569 index e303f08ff5..dcb07440e8 100755 --- a/tests/xfs/1569 +++ b/tests/xfs/1569 @@ -27,13 +27,8 @@ echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 echo "Fuzz rtsummary" -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') -if [ -n "$is_metadir" ]; then - path=('path -m /realtime/0.summary') -else - path=('sb' 'addr rsumino') -fi -_scratch_xfs_fuzz_metadata '' 'none' "${path[@]}" 'dblock 0' >> $seqres.full +path="$(_scratch_xfs_find_metafile rsumino)" +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'dblock 0' >> $seqres.full echo "Done fuzzing rtsummary" # success, all done diff --git a/tests/xfs/529 b/tests/xfs/529 index 83d24da0ac..e10af6753b 100755 --- a/tests/xfs/529 +++ b/tests/xfs/529 @@ -159,9 +159,8 @@ done _scratch_unmount >> $seqres.full echo "Verify uquota inode's extent count" -uquotino=$(_scratch_xfs_get_metadata_field 'uquotino' 'sb 0') - -nextents=$(_scratch_get_iext_count $uquotino data || \ +selector="$(_scratch_xfs_find_metafile uquotino)" +nextents=$(_scratch_get_iext_count "$selector" data || \ _fail "Unable to obtain inode fork's extent count") if (( $nextents > 10 )); then echo "Extent count overflow check failed: nextents = $nextents" diff --git a/tests/xfs/530 b/tests/xfs/530 index 56f5e7ebdb..cb8c2e3978 100755 --- a/tests/xfs/530 +++ b/tests/xfs/530 @@ -104,10 +104,8 @@ _scratch_unmount >> $seqres.full echo "Verify rbmino's and rsumino's extent count" for rtino in rbmino rsumino; do - ino=$(_scratch_xfs_get_metadata_field $rtino "sb 0") - echo "$rtino = $ino" >> $seqres.full - - nextents=$(_scratch_get_iext_count $ino data || \ + selector="$(_scratch_xfs_find_metafile "$rtino")" + nextents=$(_scratch_get_iext_count "$selector" data || \ _fail "Unable to obtain inode fork's extent count") if (( $nextents > 10 )); then echo "Extent count overflow check failed: nextents = $nextents" ^ permalink raw reply related [flat|nested] 68+ messages in thread
* Re: [PATCH 2/9] various: fix finding metadata inode numbers when metadir is enabled 2022-12-30 22:20 ` [PATCH 2/9] various: fix finding metadata inode numbers when metadir is enabled Darrick J. Wong @ 2023-03-06 16:41 ` Zorro Lang 0 siblings, 0 replies; 68+ messages in thread From: Zorro Lang @ 2023-03-06 16:41 UTC (permalink / raw) To: Darrick J. Wong; +Cc: linux-xfs, fstests On Fri, Dec 30, 2022 at 02:20:32PM -0800, Darrick J. Wong wrote: > From: Darrick J. Wong <djwong@kernel.org> > > There are a number of tests that use xfs_db to examine the contents of > metadata inodes to check correct functioning. The logic is scattered > everywhere and won't work with metadata directory trees, so make a > shared helper to find these inodes. > > Signed-off-by: Darrick J. Wong <djwong@kernel.org> > --- > common/xfs | 32 ++++++++++++++++++++++++++++++-- > tests/xfs/007 | 16 +++++++++------- > tests/xfs/1562 | 9 ++------- > tests/xfs/1563 | 9 ++------- > tests/xfs/1564 | 9 ++------- > tests/xfs/1565 | 9 ++------- > tests/xfs/1566 | 9 ++------- > tests/xfs/1567 | 9 ++------- > tests/xfs/1568 | 9 ++------- > tests/xfs/1569 | 9 ++------- These case names are temporary names, I've renamed them when I merged them, so this patch need to rebase. Sorry for this trouble :) Thanks, Zorro > tests/xfs/529 | 5 ++--- > tests/xfs/530 | 6 ++---- > 12 files changed, 59 insertions(+), 72 deletions(-) > > > diff --git a/common/xfs b/common/xfs > index 8b365ad18b..dafbd1b874 100644 > --- a/common/xfs > +++ b/common/xfs > @@ -1396,7 +1396,7 @@ _scratch_get_bmx_prefix() { > > _scratch_get_iext_count() > { > - local ino=$1 > + local selector=$1 > local whichfork=$2 > local field="" > > @@ -1411,7 +1411,7 @@ _scratch_get_iext_count() > return 1 > esac > > - _scratch_xfs_get_metadata_field $field "inode $ino" > + _scratch_xfs_get_metadata_field $field "$selector" > } > > # > @@ -1742,3 +1742,31 @@ _require_xfs_scratch_atomicswap() > _notrun "atomicswap dependencies not supported by scratch filesystem type: $FSTYP" > _scratch_unmount > } > + > +# Find a metadata file within an xfs filesystem. The sole argument is the > +# name of the field within the superblock. > +_scratch_xfs_find_metafile() > +{ > + local metafile="$1" > + local selector= > + > + if ! _check_scratch_xfs_features METADIR > /dev/null; then > + sb_field="$(_scratch_xfs_get_sb_field "$metafile")" > + if echo "$sb_field" | grep -q -w 'not found'; then > + return 1 > + fi > + selector="inode $sb_field" > + else > + case "${metafile}" in > + "rootino") selector="path /";; > + "uquotino") selector="path -m /quota/user";; > + "gquotino") selector="path -m /quota/group";; > + "pquotino") selector="path -m /quota/project";; > + "rbmino") selector="path -m /realtime/bitmap";; > + "rsumino") selector="path -m /realtime/summary";; > + esac > + fi > + > + echo "${selector}" > + return 0 > +} > diff --git a/tests/xfs/007 b/tests/xfs/007 > index 4f864100fd..6d6d828b13 100755 > --- a/tests/xfs/007 > +++ b/tests/xfs/007 > @@ -22,6 +22,11 @@ _require_xfs_quota > _scratch_mkfs_xfs | _filter_mkfs > /dev/null 2> $tmp.mkfs > . $tmp.mkfs > > +get_qfile_nblocks() { > + local selector="$(_scratch_xfs_find_metafile "$1")" > + _scratch_xfs_db -c "$selector" -c "p core.nblocks" > +} > + > do_test() > { > qino_1=$1 > @@ -31,12 +36,9 @@ do_test() > echo "*** umount" > _scratch_unmount > > - QINO_1=`_scratch_xfs_get_sb_field $qino_1` > - QINO_2=`_scratch_xfs_get_sb_field $qino_2` > - > echo "*** Usage before quotarm ***" > - _scratch_xfs_db -c "inode $QINO_1" -c "p core.nblocks" > - _scratch_xfs_db -c "inode $QINO_2" -c "p core.nblocks" > + get_qfile_nblocks $qino_1 > + get_qfile_nblocks $qino_2 > > _qmount > echo "*** turn off $off_opts quotas" > @@ -66,8 +68,8 @@ do_test() > _scratch_unmount > > echo "*** Usage after quotarm ***" > - _scratch_xfs_db -c "inode $QINO_1" -c "p core.nblocks" > - _scratch_xfs_db -c "inode $QINO_2" -c "p core.nblocks" > + get_qfile_nblocks $qino_1 > + get_qfile_nblocks $qino_2 > } > > # Test user and group first > diff --git a/tests/xfs/1562 b/tests/xfs/1562 > index 015209eeb2..1e5b6881ee 100755 > --- a/tests/xfs/1562 > +++ b/tests/xfs/1562 > @@ -27,13 +27,8 @@ echo "Format and populate" > _scratch_populate_cached nofill > $seqres.full 2>&1 > > echo "Fuzz rtbitmap" > -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') > -if [ -n "$is_metadir" ]; then > - path=('path -m /realtime/0.bitmap') > -else > - path=('sb' 'addr rbmino') > -fi > -_scratch_xfs_fuzz_metadata '' 'online' "${path[@]}" 'dblock 0' >> $seqres.full > +path="$(_scratch_xfs_find_metafile rbmino)" > +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'dblock 0' >> $seqres.full > echo "Done fuzzing rtbitmap" > > # success, all done > diff --git a/tests/xfs/1563 b/tests/xfs/1563 > index 2be0870a3d..a9da78106d 100755 > --- a/tests/xfs/1563 > +++ b/tests/xfs/1563 > @@ -27,13 +27,8 @@ echo "Format and populate" > _scratch_populate_cached nofill > $seqres.full 2>&1 > > echo "Fuzz rtsummary" > -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') > -if [ -n "$is_metadir" ]; then > - path=('path -m /realtime/0.summary') > -else > - path=('sb' 'addr rsumino') > -fi > -_scratch_xfs_fuzz_metadata '' 'online' "${path[@]}" 'dblock 0' >> $seqres.full > +path="$(_scratch_xfs_find_metafile rsumino)" > +_scratch_xfs_fuzz_metadata '' 'online' "$path" 'dblock 0' >> $seqres.full > echo "Done fuzzing rtsummary" > > # success, all done > diff --git a/tests/xfs/1564 b/tests/xfs/1564 > index c0d10ff0e9..4482861d50 100755 > --- a/tests/xfs/1564 > +++ b/tests/xfs/1564 > @@ -27,13 +27,8 @@ echo "Format and populate" > _scratch_populate_cached nofill > $seqres.full 2>&1 > > echo "Fuzz rtbitmap" > -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') > -if [ -n "$is_metadir" ]; then > - path=('path -m /realtime/0.bitmap') > -else > - path=('sb' 'addr rbmino') > -fi > -_scratch_xfs_fuzz_metadata '' 'offline' "${path[@]}" 'dblock 0' >> $seqres.full > +path="$(_scratch_xfs_find_metafile rbmino)" > +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'dblock 0' >> $seqres.full > echo "Done fuzzing rtbitmap" > > # success, all done > diff --git a/tests/xfs/1565 b/tests/xfs/1565 > index 6b4186fb3c..c43ccd848e 100755 > --- a/tests/xfs/1565 > +++ b/tests/xfs/1565 > @@ -27,13 +27,8 @@ echo "Format and populate" > _scratch_populate_cached nofill > $seqres.full 2>&1 > > echo "Fuzz rtsummary" > -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') > -if [ -n "$is_metadir" ]; then > - path=('path -m /realtime/0.summary') > -else > - path=('sb' 'addr rsumino') > -fi > -_scratch_xfs_fuzz_metadata '' 'offline' "${path[@]}" 'dblock 0' >> $seqres.full > +path="$(_scratch_xfs_find_metafile rsumino)" > +_scratch_xfs_fuzz_metadata '' 'offline' "$path" 'dblock 0' >> $seqres.full > echo "Done fuzzing rtsummary" > > # success, all done > diff --git a/tests/xfs/1566 b/tests/xfs/1566 > index 8d0f61ae10..aad4fafb15 100755 > --- a/tests/xfs/1566 > +++ b/tests/xfs/1566 > @@ -28,13 +28,8 @@ echo "Format and populate" > _scratch_populate_cached nofill > $seqres.full 2>&1 > > echo "Fuzz rtbitmap" > -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') > -if [ -n "$is_metadir" ]; then > - path=('path -m /realtime/0.bitmap') > -else > - path=('sb' 'addr rbmino') > -fi > -_scratch_xfs_fuzz_metadata '' 'both' "${path[@]}" 'dblock 0' >> $seqres.full > +path="$(_scratch_xfs_find_metafile rbmino)" > +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'dblock 0' >> $seqres.full > echo "Done fuzzing rtbitmap" > > # success, all done > diff --git a/tests/xfs/1567 b/tests/xfs/1567 > index 7dc2012b67..ff782fc239 100755 > --- a/tests/xfs/1567 > +++ b/tests/xfs/1567 > @@ -28,13 +28,8 @@ echo "Format and populate" > _scratch_populate_cached nofill > $seqres.full 2>&1 > > echo "Fuzz rtsummary" > -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') > -if [ -n "$is_metadir" ]; then > - path=('path -m /realtime/0.summary') > -else > - path=('sb' 'addr rsumino') > -fi > -_scratch_xfs_fuzz_metadata '' 'both' "${path[@]}" 'dblock 0' >> $seqres.full > +path="$(_scratch_xfs_find_metafile rsumino)" > +_scratch_xfs_fuzz_metadata '' 'both' "$path" 'dblock 0' >> $seqres.full > echo "Done fuzzing rtsummary" > > # success, all done > diff --git a/tests/xfs/1568 b/tests/xfs/1568 > index c80640ef97..e2a28df58a 100755 > --- a/tests/xfs/1568 > +++ b/tests/xfs/1568 > @@ -27,13 +27,8 @@ echo "Format and populate" > _scratch_populate_cached nofill > $seqres.full 2>&1 > > echo "Fuzz rtbitmap" > -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.bitmap') > -if [ -n "$is_metadir" ]; then > - path=('path -m /realtime/0.bitmap') > -else > - path=('sb' 'addr rbmino') > -fi > -_scratch_xfs_fuzz_metadata '' 'none' "${path[@]}" 'dblock 0' >> $seqres.full > +path="$(_scratch_xfs_find_metafile rbmino)" > +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'dblock 0' >> $seqres.full > echo "Done fuzzing rtbitmap" > > # success, all done > diff --git a/tests/xfs/1569 b/tests/xfs/1569 > index e303f08ff5..dcb07440e8 100755 > --- a/tests/xfs/1569 > +++ b/tests/xfs/1569 > @@ -27,13 +27,8 @@ echo "Format and populate" > _scratch_populate_cached nofill > $seqres.full 2>&1 > > echo "Fuzz rtsummary" > -is_metadir=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime/0.summary') > -if [ -n "$is_metadir" ]; then > - path=('path -m /realtime/0.summary') > -else > - path=('sb' 'addr rsumino') > -fi > -_scratch_xfs_fuzz_metadata '' 'none' "${path[@]}" 'dblock 0' >> $seqres.full > +path="$(_scratch_xfs_find_metafile rsumino)" > +_scratch_xfs_fuzz_metadata '' 'none' "$path" 'dblock 0' >> $seqres.full > echo "Done fuzzing rtsummary" > > # success, all done > diff --git a/tests/xfs/529 b/tests/xfs/529 > index 83d24da0ac..e10af6753b 100755 > --- a/tests/xfs/529 > +++ b/tests/xfs/529 > @@ -159,9 +159,8 @@ done > _scratch_unmount >> $seqres.full > > echo "Verify uquota inode's extent count" > -uquotino=$(_scratch_xfs_get_metadata_field 'uquotino' 'sb 0') > - > -nextents=$(_scratch_get_iext_count $uquotino data || \ > +selector="$(_scratch_xfs_find_metafile uquotino)" > +nextents=$(_scratch_get_iext_count "$selector" data || \ > _fail "Unable to obtain inode fork's extent count") > if (( $nextents > 10 )); then > echo "Extent count overflow check failed: nextents = $nextents" > diff --git a/tests/xfs/530 b/tests/xfs/530 > index 56f5e7ebdb..cb8c2e3978 100755 > --- a/tests/xfs/530 > +++ b/tests/xfs/530 > @@ -104,10 +104,8 @@ _scratch_unmount >> $seqres.full > > echo "Verify rbmino's and rsumino's extent count" > for rtino in rbmino rsumino; do > - ino=$(_scratch_xfs_get_metadata_field $rtino "sb 0") > - echo "$rtino = $ino" >> $seqres.full > - > - nextents=$(_scratch_get_iext_count $ino data || \ > + selector="$(_scratch_xfs_find_metafile "$rtino")" > + nextents=$(_scratch_get_iext_count "$selector" data || \ > _fail "Unable to obtain inode fork's extent count") > if (( $nextents > 10 )); then > echo "Extent count overflow check failed: nextents = $nextents" > ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCH 4/9] common/repair: patch up repair sb inode value complaints 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 2/9] various: fix finding metadata inode numbers when metadir is enabled Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 6/9] xfs/{050,144,153,299,330}: update quota reports to leave out metadir files Darrick J. Wong ` (4 subsequent siblings) 8 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Now that we've refactored xfs_repair to be more consistent in how it reports unexpected superblock inode pointer values, we have to fix up the fstests repair filters to emulate the old golden output. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/repair | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/common/repair b/common/repair index 8945d0028c..c3afcfb3e6 100644 --- a/common/repair +++ b/common/repair @@ -28,6 +28,10 @@ _filter_repair() perl -ne ' # for sb /- agno = / && next; # remove each AG line (variable number) +s/realtime bitmap inode pointer/realtime bitmap ino pointer/; +s/sb realtime bitmap inode value/sb realtime bitmap inode/; +s/realtime summary inode pointer/realtime summary ino pointer/; +s/sb realtime summary inode value/sb realtime summary inode/; s/(pointer to) (\d+)/\1 INO/; # Changed inode output in 5.5.0 s/sb root inode value /sb root inode /; ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 6/9] xfs/{050,144,153,299,330}: update quota reports to leave out metadir files 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 4/9] common/repair: patch up repair sb inode value complaints Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 9/9] xfs: create fuzz tests for metadata directories Darrick J. Wong ` (3 subsequent siblings) 8 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Remove the metadata directory tree directories from the quota reporting in these tests so that we don't regress the golden output. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/filter | 7 +++++-- common/xfs | 23 +++++++++++++++++++++++ tests/xfs/050 | 1 + tests/xfs/153 | 1 + tests/xfs/299 | 1 + tests/xfs/330 | 6 +++++- 6 files changed, 36 insertions(+), 3 deletions(-) diff --git a/common/filter b/common/filter index 3e3fea7ea0..49c6859992 100644 --- a/common/filter +++ b/common/filter @@ -618,11 +618,14 @@ _filter_getcap() # Filter user/group/project id numbers out of quota reports, and standardize # the block counts to use filesystem block size. Callers must set the id and -# bsize variables before calling this function. +# bsize variables before calling this function. The qhidden_rootfiles variable +# (by default zero) is the number of root files to filter out of the inode +# count part of the quota report. _filter_quota_report() { test -n "$id" || echo "id must be set" test -n "$bsize" || echo "block size must be set" + test -n "$qhidden_rootfiles" || qhidden_rootfiles=0 tr -s '[:space:]' | \ perl -npe ' @@ -630,7 +633,7 @@ _filter_quota_report() s/^\#0 \d+ /[ROOT] 0 /g; s/6 days/7 days/g' | perl -npe ' - $val = 0; + $val = '"$qhidden_rootfiles"'; if ($ENV{'LARGE_SCRATCH_DEV'}) { $val = $ENV{'NUM_SPACE_FILES'}; } diff --git a/common/xfs b/common/xfs index 0f69d3eb18..99e377631b 100644 --- a/common/xfs +++ b/common/xfs @@ -1783,3 +1783,26 @@ _scratch_xfs_force_no_metadir() MKFS_OPTIONS="-m metadir=0 $MKFS_OPTIONS" fi } + +# Decide if a mount filesystem has metadata directory trees. +_xfs_mount_has_metadir() { + local mount="$1" + + # spaceman (and its info command) predate metadir + test ! -e "$XFS_SPACEMAN_PROG" && return 1 + $XFS_SPACEMAN_PROG -c "info" "$mount" | grep -q 'metadir=1' +} + +# Compute the number of files in the metadata directory tree. +_xfs_calc_metadir_files() { + local mount="$1" + + if ! _xfs_mount_has_metadir "$mount"; then + echo 0 + return + fi + + local regfiles="$($XFS_IO_PROG -c 'bulkstat' "$mount" | grep '^bs_ino' | wc -l)" + local metafiles="$($XFS_IO_PROG -c 'bulkstat -m' "$mount" 2>&1 | grep '^bs_ino' | wc -l)" + echo $((metafiles - regfiles)) +} diff --git a/tests/xfs/050 b/tests/xfs/050 index 2220e47016..64fbaf687d 100755 --- a/tests/xfs/050 +++ b/tests/xfs/050 @@ -34,6 +34,7 @@ _require_xfs_quota _scratch_mkfs >/dev/null 2>&1 _scratch_mount bsize=$(_get_file_block_size $SCRATCH_MNT) +qhidden_rootfiles=$(_xfs_calc_metadir_files $SCRATCH_MNT) _scratch_unmount bsoft=$(( 200 * $bsize )) diff --git a/tests/xfs/153 b/tests/xfs/153 index dbe26b6803..fc64bf734a 100755 --- a/tests/xfs/153 +++ b/tests/xfs/153 @@ -39,6 +39,7 @@ _require_test_program "vfs/mount-idmapped" _scratch_mkfs >/dev/null 2>&1 _scratch_mount bsize=$(_get_file_block_size $SCRATCH_MNT) +qhidden_rootfiles=$(_xfs_calc_metadir_files $SCRATCH_MNT) _scratch_unmount bsoft=$(( 200 * $bsize )) diff --git a/tests/xfs/299 b/tests/xfs/299 index 4b9df3c6aa..2167c492c4 100755 --- a/tests/xfs/299 +++ b/tests/xfs/299 @@ -159,6 +159,7 @@ _qmount_option "uquota,gquota,pquota" _qmount bsize=$(_get_file_block_size $SCRATCH_MNT) +qhidden_rootfiles=$(_xfs_calc_metadir_files $SCRATCH_MNT) bsoft=$(( 100 * $bsize )) bhard=$(( 500 * $bsize )) diff --git a/tests/xfs/330 b/tests/xfs/330 index c6e74e67e8..e919ccc1ca 100755 --- a/tests/xfs/330 +++ b/tests/xfs/330 @@ -26,7 +26,10 @@ _require_nobody do_repquota() { - repquota $SCRATCH_MNT | grep -E '^(fsgqa|root|nobody)' | sort -r + repquota $SCRATCH_MNT | grep -E '^(fsgqa|root|nobody)' | sort -r | \ + perl -npe ' + $val = '"$qhidden_rootfiles"'; + s/(^root\s+--\s+\S+\s+\S+\s+\S+\s+)(\S+)/$1@{[$2 - $val]}/g' } rm -f "$seqres.full" @@ -35,6 +38,7 @@ echo "Format and mount" _scratch_mkfs > "$seqres.full" 2>&1 export MOUNT_OPTIONS="-o usrquota,grpquota $MOUNT_OPTIONS" _scratch_mount >> "$seqres.full" 2>&1 +qhidden_rootfiles=$(_xfs_calc_metadir_files $SCRATCH_MNT) quotacheck -u -g $SCRATCH_MNT 2> /dev/null quotaon $SCRATCH_MNT 2> /dev/null ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 9/9] xfs: create fuzz tests for metadata directories 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong ` (4 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 6/9] xfs/{050,144,153,299,330}: update quota reports to leave out metadir files Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 7/9] xfs/769: add metadir upgrade to test matrix Darrick J. Wong ` (2 subsequent siblings) 8 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create fuzz tests to make sure that all the validation works for metadata directories and subdirectories. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 22 ++++++++++++++++++++++ tests/xfs/1546 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/1546.out | 4 ++++ tests/xfs/1547 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/1547.out | 4 ++++ tests/xfs/1548 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/1548.out | 4 ++++ tests/xfs/1549 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/1549.out | 4 ++++ tests/xfs/1550 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/1550.out | 4 ++++ tests/xfs/1551 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/1551.out | 4 ++++ tests/xfs/1552 | 37 +++++++++++++++++++++++++++++++++++++ tests/xfs/1552.out | 4 ++++ tests/xfs/1553 | 38 ++++++++++++++++++++++++++++++++++++++ tests/xfs/1553.out | 4 ++++ 17 files changed, 352 insertions(+) create mode 100755 tests/xfs/1546 create mode 100644 tests/xfs/1546.out create mode 100755 tests/xfs/1547 create mode 100644 tests/xfs/1547.out create mode 100755 tests/xfs/1548 create mode 100644 tests/xfs/1548.out create mode 100755 tests/xfs/1549 create mode 100644 tests/xfs/1549.out create mode 100755 tests/xfs/1550 create mode 100644 tests/xfs/1550.out create mode 100755 tests/xfs/1551 create mode 100644 tests/xfs/1551.out create mode 100755 tests/xfs/1552 create mode 100644 tests/xfs/1552.out create mode 100755 tests/xfs/1553 create mode 100644 tests/xfs/1553.out diff --git a/common/xfs b/common/xfs index 99e377631b..77af8a6d60 100644 --- a/common/xfs +++ b/common/xfs @@ -1806,3 +1806,25 @@ _xfs_calc_metadir_files() { local metafiles="$($XFS_IO_PROG -c 'bulkstat -m' "$mount" 2>&1 | grep '^bs_ino' | wc -l)" echo $((metafiles - regfiles)) } + +_require_xfs_mkfs_metadir() +{ + _scratch_mkfs_xfs_supported -m metadir=1 >/dev/null 2>&1 || \ + _notrun "mkfs.xfs doesn't have metadir features" +} + +_require_xfs_scratch_metadir() +{ + _require_xfs_mkfs_metadir + _require_scratch + + _scratch_mkfs -m metadir=1 &> /dev/null + _require_scratch_xfs_features METADIR + _try_scratch_mount + res=$? + if [ $res -ne 0 ]; then + _notrun "mounting with metadir not supported by filesystem type: $FSTYP" + else + _scratch_unmount + fi +} diff --git a/tests/xfs/1546 b/tests/xfs/1546 new file mode 100755 index 0000000000..5b48463abe --- /dev/null +++ b/tests/xfs/1546 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1546 +# +# Populate a XFS filesystem and fuzz every metadir root field. +# Use xfs_scrub to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_xfs_scratch_metadir +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /') + +echo "Fuzz metadir root" +_scratch_xfs_fuzz_metadata '' 'online' 'path -m /' >> $seqres.full +echo "Done fuzzing metadir root" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1546.out b/tests/xfs/1546.out new file mode 100644 index 0000000000..b72891a758 --- /dev/null +++ b/tests/xfs/1546.out @@ -0,0 +1,4 @@ +QA output created by 1546 +Format and populate +Fuzz metadir root +Done fuzzing metadir root diff --git a/tests/xfs/1547 b/tests/xfs/1547 new file mode 100755 index 0000000000..ff86bc657e --- /dev/null +++ b/tests/xfs/1547 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1547 +# +# Populate a XFS filesystem and fuzz every metadir root field. +# Use xfs_repair to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_xfs_scratch_metadir +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /') + +echo "Fuzz metadir root" +_scratch_xfs_fuzz_metadata '' 'offline' 'path -m /' >> $seqres.full +echo "Done fuzzing metadir root" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1547.out b/tests/xfs/1547.out new file mode 100644 index 0000000000..983cc01343 --- /dev/null +++ b/tests/xfs/1547.out @@ -0,0 +1,4 @@ +QA output created by 1547 +Format and populate +Fuzz metadir root +Done fuzzing metadir root diff --git a/tests/xfs/1548 b/tests/xfs/1548 new file mode 100755 index 0000000000..1f29dfda3b --- /dev/null +++ b/tests/xfs/1548 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1548 +# +# Populate a XFS filesystem and fuzz every metadir root field. +# Do not fix the filesystem, to test metadata verifiers. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_norepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_xfs_scratch_metadir +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /') + +echo "Fuzz metadir root" +_scratch_xfs_fuzz_metadata '' 'none' 'path -m /' >> $seqres.full +echo "Done fuzzing metadir root" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1548.out b/tests/xfs/1548.out new file mode 100644 index 0000000000..9e395bb059 --- /dev/null +++ b/tests/xfs/1548.out @@ -0,0 +1,4 @@ +QA output created by 1548 +Format and populate +Fuzz metadir root +Done fuzzing metadir root diff --git a/tests/xfs/1549 b/tests/xfs/1549 new file mode 100755 index 0000000000..865023f218 --- /dev/null +++ b/tests/xfs/1549 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1549 +# +# Populate a XFS filesystem and fuzz every metadir root field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_xfs_scratch_metadir +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /') + +echo "Fuzz metadir root" +_scratch_xfs_fuzz_metadata '' 'both' 'path -m /' >> $seqres.full +echo "Done fuzzing metadir root" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1549.out b/tests/xfs/1549.out new file mode 100644 index 0000000000..22b3d215e3 --- /dev/null +++ b/tests/xfs/1549.out @@ -0,0 +1,4 @@ +QA output created by 1549 +Format and populate +Fuzz metadir root +Done fuzzing metadir root diff --git a/tests/xfs/1550 b/tests/xfs/1550 new file mode 100755 index 0000000000..62219e65fc --- /dev/null +++ b/tests/xfs/1550 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1550 +# +# Populate a XFS filesystem and fuzz every metadir subdir field. +# Use xfs_scrub to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_xfs_scratch_metadir +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime') + +echo "Fuzz metadir subdir" +_scratch_xfs_fuzz_metadata '' 'online' 'path -m /realtime' >> $seqres.full +echo "Done fuzzing metadir subdir" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1550.out b/tests/xfs/1550.out new file mode 100644 index 0000000000..7694cd670b --- /dev/null +++ b/tests/xfs/1550.out @@ -0,0 +1,4 @@ +QA output created by 1550 +Format and populate +Fuzz metadir subdir +Done fuzzing metadir subdir diff --git a/tests/xfs/1551 b/tests/xfs/1551 new file mode 100755 index 0000000000..f101529364 --- /dev/null +++ b/tests/xfs/1551 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1551 +# +# Populate a XFS filesystem and fuzz every metadir subdir field. +# Use xfs_repair to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_repair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_xfs_scratch_metadir +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime') + +echo "Fuzz metadir subdir" +_scratch_xfs_fuzz_metadata '' 'offline' 'path -m /realtime' >> $seqres.full +echo "Done fuzzing metadir subdir" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1551.out b/tests/xfs/1551.out new file mode 100644 index 0000000000..4c3360d08b --- /dev/null +++ b/tests/xfs/1551.out @@ -0,0 +1,4 @@ +QA output created by 1551 +Format and populate +Fuzz metadir subdir +Done fuzzing metadir subdir diff --git a/tests/xfs/1552 b/tests/xfs/1552 new file mode 100755 index 0000000000..ab3b89ec40 --- /dev/null +++ b/tests/xfs/1552 @@ -0,0 +1,37 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1552 +# +# Populate a XFS filesystem and fuzz every metadir subdir field. +# Do not fix the filesystem, to test metadata verifiers. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_norepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_xfs_scratch_metadir +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime') + +echo "Fuzz metadir subdir" +_scratch_xfs_fuzz_metadata '' 'none' 'path -m /realtime' >> $seqres.full +echo "Done fuzzing metadir subdir" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1552.out b/tests/xfs/1552.out new file mode 100644 index 0000000000..6636b1b656 --- /dev/null +++ b/tests/xfs/1552.out @@ -0,0 +1,4 @@ +QA output created by 1552 +Format and populate +Fuzz metadir subdir +Done fuzzing metadir subdir diff --git a/tests/xfs/1553 b/tests/xfs/1553 new file mode 100755 index 0000000000..6acbacbe16 --- /dev/null +++ b/tests/xfs/1553 @@ -0,0 +1,38 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1553 +# +# Populate a XFS filesystem and fuzz every metadir subdir field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_xfs_scratch_metadir +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'path -m /realtime') + +echo "Fuzz metadir subdir" +_scratch_xfs_fuzz_metadata '' 'both' 'path -m /realtime' >> $seqres.full +echo "Done fuzzing metadir subdir" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1553.out b/tests/xfs/1553.out new file mode 100644 index 0000000000..0298fcfddb --- /dev/null +++ b/tests/xfs/1553.out @@ -0,0 +1,4 @@ +QA output created by 1553 +Format and populate +Fuzz metadir subdir +Done fuzzing metadir subdir ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 7/9] xfs/769: add metadir upgrade to test matrix 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong ` (5 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 9/9] xfs: create fuzz tests for metadata directories Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 5/9] xfs/206: update for metadata directory support Darrick J. Wong 2022-12-30 22:20 ` [PATCH 8/9] xfs/509: adjust inumbers accounting for metadata directories Darrick J. Wong 8 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add metadata directory trees to the features that this test will try to upgrade. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/769 | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/xfs/769 b/tests/xfs/769 index 7613048f52..624dd2a338 100755 --- a/tests/xfs/769 +++ b/tests/xfs/769 @@ -174,12 +174,14 @@ if rt_configured; then check_repair_upgrade finobt && FEATURES+=("finobt") check_repair_upgrade inobtcount && FEATURES+=("inobtcount") check_repair_upgrade bigtime && FEATURES+=("bigtime") + check_repair_upgrade metadir && FEATURES+=("metadir") else check_repair_upgrade finobt && FEATURES+=("finobt") check_repair_upgrade rmapbt && FEATURES+=("rmapbt") check_repair_upgrade reflink && FEATURES+=("reflink") check_repair_upgrade inobtcount && FEATURES+=("inobtcount") check_repair_upgrade bigtime && FEATURES+=("bigtime") + check_repair_upgrade metadir && FEATURES+=("metadir") fi test "${#FEATURES[@]}" -eq 0 && \ ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 5/9] xfs/206: update for metadata directory support 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong ` (6 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 7/9] xfs/769: add metadir upgrade to test matrix Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 8/9] xfs/509: adjust inumbers accounting for metadata directories Darrick J. Wong 8 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Filter 'metadir=' out of the golden output so that metadata directories don't cause this test to regress. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/206 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/xfs/206 b/tests/xfs/206 index cb346b6dc9..c181d7dd3e 100755 --- a/tests/xfs/206 +++ b/tests/xfs/206 @@ -64,7 +64,8 @@ mkfs_filter() -e "s/\(sunit=\)\([0-9]* blks,\)/\10 blks,/" \ -e "s/, lazy-count=[0-9]//" \ -e "/.*crc=/d" \ - -e "/^Default configuration/d" + -e "/^Default configuration/d" \ + -e "/metadir=.*/d" } # mkfs slightly smaller than that, small log for speed. ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 8/9] xfs/509: adjust inumbers accounting for metadata directories 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong ` (7 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 5/9] xfs/206: update for metadata directory support Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 8 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The INUMBERS ioctl exports data from the inode btree directly -- the number of inodes it reports is taken from ir_freemask and includes all the files in the metadata directory tree. BULKSTAT, on the other hand, only reports non-metadata files. When metadir is enabled, this will (eventually) cause a discrepancy in the inode counts that is large enough to exceed the tolerances, thereby causing a test failure. Correct this by counting the files in the metadata directory and subtracting that from the INUMBERS totals. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/509 | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/tests/xfs/509 b/tests/xfs/509 index d04dfbbfba..b87abef964 100755 --- a/tests/xfs/509 +++ b/tests/xfs/509 @@ -91,13 +91,13 @@ inumbers_count() bstat_versions | while read v_tag v_flag; do echo -n "inumbers all($v_tag): " nr=$(inumbers_fs $SCRATCH_MNT $v_flag) - _within_tolerance "inumbers" $nr $expect $tolerance -v + _within_tolerance "inumbers" $((nr - METADATA_FILES)) $expect $tolerance -v local agcount=$(_xfs_mount_agcount $SCRATCH_MNT) for batchsize in 71 2 1; do echo -n "inumbers $batchsize($v_tag): " nr=$(inumbers_ag $agcount $batchsize $SCRATCH_MNT $v_flag) - _within_tolerance "inumbers" $nr $expect $tolerance -v + _within_tolerance "inumbers" $((nr - METADATA_FILES)) $expect $tolerance -v done done } @@ -143,9 +143,26 @@ _supported_fs xfs DIRCOUNT=8 INOCOUNT=$((2048 / DIRCOUNT)) +# Count everything in the metadata directory tree. +count_metadir_files() { + local metadirs=('/realtime' '/quota') + local db_args=('-f') + + for m in "${metadirs[@]}"; do + db_args+=('-c' "ls -m $m") + done + + local ret=$(_scratch_xfs_db "${db_args[@]}" 2>/dev/null | grep regular | wc -l) + test -z "$ret" && ret=0 + echo $ret +} + _scratch_mkfs "-d agcount=$DIRCOUNT" >> $seqres.full 2>&1 || _fail "mkfs failed" _scratch_mount +METADATA_FILES=$(count_metadir_files) +echo "found $METADATA_FILES metadata files" >> $seqres.full + # Figure out if we have v5 bulkstat/inumbers ioctls. has_v5= bs_root_out="$($XFS_IO_PROG -c 'bulkstat_single root' $SCRATCH_MNT 2>>$seqres.full)" ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCHSET v1.0 0/4] fstests: support metadump to external devices 2022-12-30 21:14 [NYE DELUGE 3/4] xfs: modernize the realtime volume Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET 0/1] fstests: test upgrading older features Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/4] common/ext4: reformat external logs during mdrestore operations Darrick J. Wong ` (3 more replies) 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong ` (4 subsequent siblings) 7 siblings, 4 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, This series modifies fstests to take advantage of the fact that xfs_metadump and xfs_mdrestore now support capturing the contents of an external log in a metadump, and restoring it on the other end. The first part of this series refactors and cleans up the common code a bit, and the rest add the actual support. Once this is merged, we'll be able to cache metadumps of populated filesystems with external log devices, which will enable faster fuzz testing. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=metadump-external-devices fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=metadump-external-devices --- common/ext4 | 17 ++++++++++++- common/fuzzy | 7 +++++ common/populate | 72 ++++++++++++++++++++++++++++--------------------------- common/xfs | 39 ++++++++++++++++++++++++++++-- 4 files changed, 95 insertions(+), 40 deletions(-) ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCH 3/4] common/ext4: reformat external logs during mdrestore operations 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: support metadump to external devices Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 2/4] common/xfs: wipe " Darrick J. Wong ` (2 subsequent siblings) 3 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The e2image file format doesn't support the capture of external log devices, which means that mdrestore ought to reformat the external log to get the restored filesystem to work again. The common/populate code could already do this, so push it to the common ext4 helper. While we're at it, fix the uncareful usage of SCRATCH_LOGDEV in the populate code. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/ext4 | 17 ++++++++++++++++- common/populate | 16 ++-------------- 2 files changed, 18 insertions(+), 15 deletions(-) diff --git a/common/ext4 b/common/ext4 index 3dcbfe17c9..5171b8df68 100644 --- a/common/ext4 +++ b/common/ext4 @@ -134,7 +134,8 @@ _ext4_mdrestore() { local metadump="$1" local device="$2" - shift; shift + local logdev="$3" + shift; shift; shift local options="$@" # If we're configured for compressed dumps and there isn't already an @@ -148,6 +149,20 @@ _ext4_mdrestore() test -r "$metadump" || return 1 $E2IMAGE_PROG $options -r "${metadump}" "${SCRATCH_DEV}" + res=$? + test $res -ne 0 && return $res + + # ext4 cannot e2image external logs, so we have to reformat the log + # device to match the restored fs + if [ "${logdev}" != "none" ]; then + local fsuuid="$($DUMPE2FS_PROG -h "${SCRATCH_DEV}" 2>/dev/null | \ + grep 'Journal UUID:' | \ + sed -e 's/Journal UUID:[[:space:]]*//g')" + $MKFS_EXT4_PROG -O journal_dev "${logdev}" \ + -F -U "${fsuuid}" + res=$? + fi + return $res } # this test requires the ext4 kernel support crc feature on scratch device diff --git a/common/populate b/common/populate index 08c4bdc151..095e771d67 100644 --- a/common/populate +++ b/common/populate @@ -912,20 +912,8 @@ _scratch_populate_restore_cached() { return $? ;; "ext2"|"ext3"|"ext4") - _ext4_mdrestore "${metadump}" "${SCRATCH_DEV}" - ret=$? - test $ret -ne 0 && return $ret - - # ext4 cannot e2image external logs, so we have to reformat - # the scratch device to match the restored fs - if [ -n "${SCRATCH_LOGDEV}" ]; then - local fsuuid="$($DUMPE2FS_PROG -h "${SCRATCH_DEV}" 2>/dev/null | \ - grep 'Journal UUID:' | \ - sed -e 's/Journal UUID:[[:space:]]*//g')" - $MKFS_EXT4_PROG -O journal_dev "${SCRATCH_LOGDEV}" \ - -F -U "${fsuuid}" - fi - return 0 + _ext4_mdrestore "${metadump}" "${SCRATCH_DEV}" "${logdev}" + return $? ;; esac return 1 ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 2/4] common/xfs: wipe external logs during mdrestore operations 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: support metadump to external devices Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/4] common/ext4: reformat external logs during mdrestore operations Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/4] common/populate: refactor caching of metadumps to a helper Darrick J. Wong 2022-12-30 22:20 ` [PATCH 4/4] common/xfs: capture external logs during metadump/mdrestore Darrick J. Wong 3 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The XFS metadump file format doesn't support the capture of external log devices, which means that mdrestore ought to wipe the external log and run xfs_repair to rewrite the log device as needed to get the restored filesystem to work again. The common/populate code could already do this, so push it to the common xfs helper. While we're at it, fix the uncareful usage of SCRATCH_LOGDEV in the populate code. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 7 ++++++- common/populate | 19 ++++++------------- common/xfs | 21 +++++++++++++++++++-- 3 files changed, 31 insertions(+), 16 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index ef54f2fe2c..7034ff8c42 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -297,7 +297,12 @@ __scratch_xfs_fuzz_unmount() __scratch_xfs_fuzz_mdrestore() { __scratch_xfs_fuzz_unmount - _xfs_mdrestore "${POPULATE_METADUMP}" "${SCRATCH_DEV}" || \ + + local logdev=none + [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \ + logdev=$SCRATCH_LOGDEV + + _xfs_mdrestore "${POPULATE_METADUMP}" "${SCRATCH_DEV}" "${logdev}" || \ _fail "${POPULATE_METADUMP}: Could not find metadump to restore?" } diff --git a/common/populate b/common/populate index 8db7acefb6..08c4bdc151 100644 --- a/common/populate +++ b/common/populate @@ -902,21 +902,14 @@ _scratch_populate_cache_tag() { _scratch_populate_restore_cached() { local metadump="$1" + local logdev=none + [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \ + logdev=$SCRATCH_LOGDEV + case "${FSTYP}" in "xfs") - _xfs_mdrestore "${metadump}" "${SCRATCH_DEV}" - res=$? - test $res -ne 0 && return $res - - # Cached images should have been unmounted cleanly, so if - # there's an external log we need to wipe it and run repair to - # format it to match this filesystem. - if [ -n "${SCRATCH_LOGDEV}" ]; then - $WIPEFS_PROG -a "${SCRATCH_LOGDEV}" - _scratch_xfs_repair - res=$? - fi - return $res + _xfs_mdrestore "${metadump}" "${SCRATCH_DEV}" "${logdev}" + return $? ;; "ext2"|"ext3"|"ext4") _ext4_mdrestore "${metadump}" "${SCRATCH_DEV}" diff --git a/common/xfs b/common/xfs index 77af8a6d60..29130fabbc 100644 --- a/common/xfs +++ b/common/xfs @@ -682,7 +682,8 @@ _xfs_metadump() { _xfs_mdrestore() { local metadump="$1" local device="$2" - shift; shift + local logdev="$3" + shift; shift; shift local options="$@" # If we're configured for compressed dumps and there isn't already an @@ -696,6 +697,18 @@ _xfs_mdrestore() { test -r "$metadump" || return 1 $XFS_MDRESTORE_PROG $options "${metadump}" "${device}" + res=$? + test $res -ne 0 && return $res + + # Cached images should have been unmounted cleanly, so if there's an + # external log we need to wipe it and run repair to format it to match + # this filesystem. + if [ "${logdev}" != "none" ]; then + $WIPEFS_PROG -a "${logdev}" + _scratch_xfs_repair + res=$? + fi + return $res } # Snapshot the metadata on the scratch device @@ -717,7 +730,11 @@ _scratch_xfs_mdrestore() local metadump=$1 shift - _xfs_mdrestore "$metadump" "$SCRATCH_DEV" "$@" + local logdev=none + [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \ + logdev=$SCRATCH_LOGDEV + + _xfs_mdrestore "$metadump" "$SCRATCH_DEV" "$logdev" "$@" } # run xfs_check and friends on a FS. ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 1/4] common/populate: refactor caching of metadumps to a helper 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: support metadump to external devices Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/4] common/ext4: reformat external logs during mdrestore operations Darrick J. Wong 2022-12-30 22:20 ` [PATCH 2/4] common/xfs: wipe " Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 4/4] common/xfs: capture external logs during metadump/mdrestore Darrick J. Wong 3 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Hoist out of _scratch_populate_cached all the code that we use to save a metadump of the populated filesystem. We're going to make this more involved for XFS in the next few patches so that we can take advantage of the new support for external devices in metadump/mdrestore. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/populate | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/common/populate b/common/populate index 29ea637ecb..8db7acefb6 100644 --- a/common/populate +++ b/common/populate @@ -938,6 +938,31 @@ _scratch_populate_restore_cached() { return 1 } +# Take a metadump of the scratch filesystem and cache it for later. +_scratch_populate_save_metadump() +{ + local metadump_file="$1" + + case "${FSTYP}" in + "xfs") + local logdev=none + [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \ + logdev=$SCRATCH_LOGDEV + + _xfs_metadump "$metadump_file" "$SCRATCH_DEV" "$logdev" \ + compress + res=$? + ;; + "ext2"|"ext3"|"ext4") + _ext4_metadump "${SCRATCH_DEV}" "${metadump_file}" compress + res=$? + ;; + *) + _fail "Don't know how to save a ${FSTYP} filesystem." + esac + return $res +} + # Populate a scratch FS from scratch or from a cached image. _scratch_populate_cached() { local meta_descr="$(_scratch_populate_cache_tag "$@")" @@ -961,26 +986,20 @@ _scratch_populate_cached() { # Oh well, just create one from scratch _scratch_mkfs - echo "${meta_descr}" > "${populate_metadump_descr}" case "${FSTYP}" in "xfs") _scratch_xfs_populate $@ _scratch_xfs_populate_check - - local logdev=none - [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \ - logdev=$SCRATCH_LOGDEV - - _xfs_metadump "$POPULATE_METADUMP" "$SCRATCH_DEV" "$logdev" \ - compress ;; "ext2"|"ext3"|"ext4") _scratch_ext4_populate $@ _scratch_ext4_populate_check - _ext4_metadump "${SCRATCH_DEV}" "${POPULATE_METADUMP}" compress ;; *) _fail "Don't know how to populate a ${FSTYP} filesystem." ;; esac + + _scratch_populate_save_metadump "${POPULATE_METADUMP}" && \ + echo "${meta_descr}" > "${populate_metadump_descr}" } ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 4/4] common/xfs: capture external logs during metadump/mdrestore 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: support metadump to external devices Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 1/4] common/populate: refactor caching of metadumps to a helper Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 3 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> If xfs_metadump supports the -x switch to capture the contents of external log devices and there is an external log device, add the option to the command line to enable preservation. Similarly, if xfs_mdrestore supports the -l switch and there's an external scratch log, pass the option so that we can restore log contents. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/common/xfs b/common/xfs index 29130fabbc..36e02413db 100644 --- a/common/xfs +++ b/common/xfs @@ -667,9 +667,20 @@ _xfs_metadump() { shift; shift; shift; shift local options="$@" test -z "$options" && options="-a -o" + local metadump_has_dash_x + + # Does metadump support capturing from external devices? + $XFS_METADUMP_PROG --help 2>&1 | grep -q -- '-[a-zA-Z]*[wW]x' && \ + metadump_has_dash_x=1 if [ "$logdev" != "none" ]; then options="$options -l $logdev" + + # Tell metadump to capture the log device + if [ -n "$metadump_has_dash_x" ]; then + options="$options -x" + unset metadump_has_dash_x + fi fi $XFS_METADUMP_PROG $options "$device" "$metadump" @@ -696,6 +707,13 @@ _xfs_mdrestore() { fi test -r "$metadump" || return 1 + # Does mdrestore support restoring to external log devices? If so, + # restore to it, and do not wipe it afterwards. + if [ "$logdev" != "none" ] && $XFS_MDRESTORE_PROG --help 2>&1 | grep -q -- '-l logdev'; then + options="$options -l $logdev" + logdev="none" + fi + $XFS_MDRESTORE_PROG $options "${metadump}" "${device}" res=$? test $res -ne 0 && return $res ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section 2022-12-30 21:14 [NYE DELUGE 3/4] xfs: modernize the realtime volume Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: support metadump to external devices Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 03/12] xfs/206: update mkfs filtering for rt groups feature Darrick J. Wong ` (11 more replies) 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (3 subsequent siblings) 7 siblings, 12 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, Right now, the realtime section uses a single pair of metadata inodes to store the free space information. This presents a scalability problem since every thread trying to allocate or free rt extents have to lock these files. It would be very useful if we could begin to tackle these problems by sharding the realtime section, so create the notion of realtime groups, which are similar to allocation groups on the data section. While we're at it, define a superblock to be stamped into the start of each rt section. This enables utilities such as blkid to identify block devices containing realtime sections, and helpfully avoids the situation where a file extent can cross an rtgroup boundary. The best advantage for rtgroups will become evident later when we get to adding rmap and reflink to the realtime volume, since the geometry constraints are the same for rt groups and AGs. Hence we can reuse all that code directly. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-groups xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=realtime-groups fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-groups --- common/fuzzy | 33 +++++++++++++++---- common/populate | 12 ++++++- common/xfs | 83 ++++++++++++++++++++++++++++++++++++++++------- src/punch-alternating.c | 28 +++++++++++++++- tests/xfs/114 | 4 ++ tests/xfs/122 | 2 + tests/xfs/122.out | 8 +++++ tests/xfs/146 | 2 + tests/xfs/185 | 2 + tests/xfs/187 | 3 +- tests/xfs/206 | 3 +- tests/xfs/271 | 3 +- tests/xfs/341 | 4 +- tests/xfs/449 | 6 +++ tests/xfs/556 | 16 ++++++--- tests/xfs/800 | 2 + tests/xfs/840 | 2 + tests/xfs/841 | 2 + 18 files changed, 176 insertions(+), 39 deletions(-) ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCH 03/12] xfs/206: update mkfs filtering for rt groups feature 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 01/12] xfs/122: update for rtgroups Darrick J. Wong ` (10 subsequent siblings) 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Filter out the new mkfs lines that show the rtgroup information, since this test is heavily dependent on old mkfs output. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/206 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/xfs/206 b/tests/xfs/206 index c181d7dd3e..904d53deb0 100755 --- a/tests/xfs/206 +++ b/tests/xfs/206 @@ -65,7 +65,8 @@ mkfs_filter() -e "s/, lazy-count=[0-9]//" \ -e "/.*crc=/d" \ -e "/^Default configuration/d" \ - -e "/metadir=.*/d" + -e "/metadir=.*/d" \ + -e '/rgcount=/d' } # mkfs slightly smaller than that, small log for speed. ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 01/12] xfs/122: update for rtgroups 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong 2022-12-30 22:20 ` [PATCH 03/12] xfs/206: update mkfs filtering for rt groups feature Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 02/12] punch-alternating: detect xfs realtime files with large allocation units Darrick J. Wong ` (9 subsequent siblings) 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add our new metadata for realtime allocation groups to the ondisk checking. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/122.out | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/xfs/122.out b/tests/xfs/122.out index eee6c1ee6d..01376180cc 100644 --- a/tests/xfs/122.out +++ b/tests/xfs/122.out @@ -44,6 +44,9 @@ offsetof(xfs_sb_t, sb_rbmino) = 64 offsetof(xfs_sb_t, sb_rextents) = 24 offsetof(xfs_sb_t, sb_rextsize) = 80 offsetof(xfs_sb_t, sb_rextslog) = 125 +offsetof(xfs_sb_t, sb_rgblklog) = 280 +offsetof(xfs_sb_t, sb_rgblocks) = 272 +offsetof(xfs_sb_t, sb_rgcount) = 276 offsetof(xfs_sb_t, sb_rootino) = 56 offsetof(xfs_sb_t, sb_rrmapino) = 264 offsetof(xfs_sb_t, sb_rsumino) = 72 @@ -112,9 +115,11 @@ sizeof(struct xfs_refcount_key) = 4 sizeof(struct xfs_refcount_rec) = 12 sizeof(struct xfs_rmap_key) = 20 sizeof(struct xfs_rmap_rec) = 24 +sizeof(struct xfs_rtgroup_geometry) = 128 sizeof(struct xfs_rtrmap_key) = 24 sizeof(struct xfs_rtrmap_rec) = 32 sizeof(struct xfs_rtrmap_root) = 4 +sizeof(struct xfs_rtsb) = 104 sizeof(struct xfs_rud_log_format) = 16 sizeof(struct xfs_rui_log_format) = 16 sizeof(struct xfs_scrub_metadata) = 64 ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 02/12] punch-alternating: detect xfs realtime files with large allocation units 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong 2022-12-30 22:20 ` [PATCH 03/12] xfs/206: update mkfs filtering for rt groups feature Darrick J. Wong 2022-12-30 22:20 ` [PATCH 01/12] xfs/122: update for rtgroups Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 10/12] xfs/27[46],xfs/556: fix tests to deal with rtgroups output in bmap/fsmap commands Darrick J. Wong ` (8 subsequent siblings) 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> For files on the XFS realtime volume, it's possible that the file allocation unit (aka the minimum size we have to punch to deallocate file blocks) could be greater than a single fs block. This utility assumed that it's always possible to punch a single fs block, but for these types of files, all that does is zeroes the page cache. While that's what most *user applications* want, fstests uses punching to fragment file mapping metadata and/or fragment free space, so adapt this test for that purpose by detecting realtime files. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- src/punch-alternating.c | 28 +++++++++++++++++++++++++++- tests/xfs/114 | 4 ++++ tests/xfs/146 | 2 +- tests/xfs/187 | 3 ++- tests/xfs/341 | 4 ++-- 5 files changed, 36 insertions(+), 5 deletions(-) diff --git a/src/punch-alternating.c b/src/punch-alternating.c index 18dd215197..d2bb4b6a22 100644 --- a/src/punch-alternating.c +++ b/src/punch-alternating.c @@ -20,6 +20,28 @@ void usage(char *cmd) exit(1); } +/* Compute the file allocation unit size for an XFS file. */ +static int detect_xfs_alloc_unit(int fd) +{ + struct fsxattr fsx; + struct xfs_fsop_geom fsgeom; + int ret; + + ret = ioctl(fd, XFS_IOC_FSGEOMETRY, &fsgeom); + if (ret) + return -1; + + ret = ioctl(fd, XFS_IOC_FSGETXATTR, &fsx); + if (ret) + return -1; + + ret = fsgeom.blocksize; + if (fsx.fsx_xflags & XFS_XFLAG_REALTIME) + ret *= fsgeom.rtextsize; + + return ret; +} + int main(int argc, char *argv[]) { struct stat s; @@ -82,7 +104,11 @@ int main(int argc, char *argv[]) goto err; sz = s.st_size; - blksz = sf.f_bsize; + c = detect_xfs_alloc_unit(fd); + if (c > 0) + blksz = c; + else + blksz = sf.f_bsize; mode = FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE; for (offset = start_offset * blksz; diff --git a/tests/xfs/114 b/tests/xfs/114 index 0e8a0529ab..7ecb4d217c 100755 --- a/tests/xfs/114 +++ b/tests/xfs/114 @@ -49,6 +49,10 @@ $XFS_IO_PROG -f \ -c "pwrite -S 0x68 -b 1048576 0 $len2" \ $SCRATCH_MNT/f2 >> $seqres.full +# The arguments to punch-alternating must be specified in units of file +# allocation units, so we divide the argument by $file_blksz. We already +# verified that $blksz is congruent with $file_blksz, so the fpunch parameters +# will always align with the file allocation unit. $here/src/punch-alternating -o $((16 * blksz / file_blksz)) \ -s $((blksz / file_blksz)) \ -i $((blksz * 2 / file_blksz)) \ diff --git a/tests/xfs/146 b/tests/xfs/146 index 123bdff59f..c1ef5e7e1b 100755 --- a/tests/xfs/146 +++ b/tests/xfs/146 @@ -68,7 +68,7 @@ _xfs_force_bdev realtime $SCRATCH_MNT # Allocate some stuff at the start, to force the first falloc of the ouch file # to happen somewhere in the middle of the rt volume $XFS_IO_PROG -f -c 'falloc 0 64m' "$SCRATCH_MNT/b" -$here/src/punch-alternating -i $((rextblks * 2)) -s $((rextblks)) "$SCRATCH_MNT/b" +$here/src/punch-alternating "$SCRATCH_MNT/b" avail="$(df -P "$SCRATCH_MNT" | awk 'END {print $4}')"1 toobig="$((avail * 2))" diff --git a/tests/xfs/187 b/tests/xfs/187 index 7c34d8e630..14c3b37670 100755 --- a/tests/xfs/187 +++ b/tests/xfs/187 @@ -132,7 +132,8 @@ $XFS_IO_PROG -f -c "truncate $required_sz" -c "falloc 0 $remap_sz" $SCRATCH_MNT/ # Punch out every other extent of the last two sections, to fragment free space. frag_sz=$((remap_sz * 3)) punch_off=$((bigfile_sz - frag_sz)) -$here/src/punch-alternating $SCRATCH_MNT/bigfile -o $((punch_off / fsbsize)) -i $((rtextsize_blks * 2)) -s $rtextsize_blks +rtextsize_bytes=$((fsbsize * rtextsize_blks)) +$here/src/punch-alternating $SCRATCH_MNT/bigfile -o $((punch_off / rtextsize_bytes)) # Make sure we have some free rtextents. free_rtx=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep statfs.f_bavail | awk '{print $3}') diff --git a/tests/xfs/341 b/tests/xfs/341 index 1f734c9015..7d2842b579 100755 --- a/tests/xfs/341 +++ b/tests/xfs/341 @@ -43,8 +43,8 @@ len=$((blocks * rtextsz)) echo "Create some files" $XFS_IO_PROG -f -R -c "falloc 0 $len" -c "pwrite -S 0x68 -b 1048576 0 $len" $SCRATCH_MNT/f1 >> $seqres.full $XFS_IO_PROG -f -R -c "falloc 0 $len" -c "pwrite -S 0x68 -b 1048576 0 $len" $SCRATCH_MNT/f2 >> $seqres.full -$here/src/punch-alternating -i $((2 * rtextsz_blks)) -s $rtextsz_blks $SCRATCH_MNT/f1 >> "$seqres.full" -$here/src/punch-alternating -i $((2 * rtextsz_blks)) -s $rtextsz_blks $SCRATCH_MNT/f2 >> "$seqres.full" +$here/src/punch-alternating $SCRATCH_MNT/f1 >> "$seqres.full" +$here/src/punch-alternating $SCRATCH_MNT/f2 >> "$seqres.full" echo garbage > $SCRATCH_MNT/f3 ino=$(stat -c '%i' $SCRATCH_MNT/f3) _scratch_unmount ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 10/12] xfs/27[46],xfs/556: fix tests to deal with rtgroups output in bmap/fsmap commands 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 02/12] punch-alternating: detect xfs realtime files with large allocation units Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 07/12] xfs/449: update test to know about xfs_db -R Darrick J. Wong ` (7 subsequent siblings) 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Fix these tests to deal with the xfs_io bmap and fsmap commands printing out realtime group numbers if the feature is enabled. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 4 ++++ tests/xfs/271 | 3 ++- tests/xfs/556 | 16 ++++++++++------ 3 files changed, 16 insertions(+), 7 deletions(-) diff --git a/common/xfs b/common/xfs index ccdcf45d0d..6089a05d0e 100644 --- a/common/xfs +++ b/common/xfs @@ -486,6 +486,10 @@ _xfs_has_feature() feat="rtextents" feat_regex="[1-9][0-9]*" ;; + "rtgroups") + feat="rgcount" + feat_regex="[1-9][0-9]*" + ;; esac local answer="$($XFS_INFO_PROG "$fs" 2>&1 | grep -E -w -c "$feat=$feat_regex")" diff --git a/tests/xfs/271 b/tests/xfs/271 index d67ac4d6c1..74e2c822c1 100755 --- a/tests/xfs/271 +++ b/tests/xfs/271 @@ -31,6 +31,7 @@ _scratch_mkfs > "$seqres.full" 2>&1 _scratch_mount agcount=$(_xfs_mount_agcount $SCRATCH_MNT) +rgcount=$(_xfs_mount_rgcount $SCRATCH_MNT) # mkfs lays out btree root blocks in the order bnobt, cntbt, inobt, finobt, # rmapbt, refcountbt, and then allocates AGFL blocks. Since GETFSMAP has the @@ -48,7 +49,7 @@ cat $TEST_DIR/fsmap >> $seqres.full echo "Check AG header" | tee -a $seqres.full grep 'static fs metadata[[:space:]]*[0-9]*[[:space:]]*(0\.\.' $TEST_DIR/fsmap | tee -a $seqres.full > $TEST_DIR/testout -_within_tolerance "AG header count" $(wc -l < $TEST_DIR/testout) $agcount 0 -v +_within_tolerance "AG header count" $(wc -l < $TEST_DIR/testout) $((agcount + rgcount)) 0 -v echo "Check freesp/rmap btrees" | tee -a $seqres.full grep 'per-AG metadata[[:space:]]*[0-9]*[[:space:]]*([0-9]*\.\.' $TEST_DIR/fsmap | tee -a $seqres.full > $TEST_DIR/testout diff --git a/tests/xfs/556 b/tests/xfs/556 index 66908a5410..72343e8625 100755 --- a/tests/xfs/556 +++ b/tests/xfs/556 @@ -47,16 +47,20 @@ victim=$SCRATCH_MNT/a file_blksz=$(_get_file_block_size $SCRATCH_MNT) $XFS_IO_PROG -f -c "pwrite -S 0x58 0 $((4 * file_blksz))" -c "fsync" $victim >> $seqres.full unset errordev -_xfs_is_realtime_file $victim && errordev="RT" + +awk_len_prog='{print $6}' +if _xfs_is_realtime_file $victim; then + if ! _xfs_has_feature $SCRATCH_MNT rtgroups; then + awk_len_prog='{print $4}' + fi + errordev="RT" +fi bmap_str="$($XFS_IO_PROG -c "bmap -elpv" $victim | grep "^[[:space:]]*0:")" echo "$errordev:$bmap_str" >> $seqres.full phys="$(echo "$bmap_str" | $AWK_PROG '{print $3}')" -if [ "$errordev" = "RT" ]; then - len="$(echo "$bmap_str" | $AWK_PROG '{print $4}')" -else - len="$(echo "$bmap_str" | $AWK_PROG '{print $6}')" -fi +len="$(echo "$bmap_str" | $AWK_PROG "$awk_len_prog")" + fs_blksz=$(_get_block_size $SCRATCH_MNT) echo "file_blksz:$file_blksz:fs_blksz:$fs_blksz" >> $seqres.full kernel_sectors_per_fs_block=$((fs_blksz / 512)) ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 07/12] xfs/449: update test to know about xfs_db -R 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 10/12] xfs/27[46],xfs/556: fix tests to deal with rtgroups output in bmap/fsmap commands Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 11/12] common/xfs: capture realtime devices during metadump/mdrestore Darrick J. Wong ` (6 subsequent siblings) 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The realtime groups feature added a -R flag to xfs_db so that users can pass in the realtime device. Since we've now modified the _scratch_xfs_db to use this facility, we can update the test to do exact comparisons of the xfs_db info command against the mkfs output. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/449 | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/xfs/449 b/tests/xfs/449 index 5374bf2f85..66c443e994 100755 --- a/tests/xfs/449 +++ b/tests/xfs/449 @@ -32,7 +32,11 @@ echo DB >> $seqres.full cat $tmp.dbinfo >> $seqres.full # xfs_db doesn't take a rtdev argument, so it reports "realtime=external". # mkfs does, so make a quick substitution -diff -u <(cat $tmp.mkfs | sed -e 's/realtime =\/.*extsz=/realtime =external extsz=/g') $tmp.dbinfo +if $XFS_DB_PROG --help 2>&1 | grep -q -- '-R rtdev'; then + diff -u $tmp.mkfs $tmp.dbinfo +else + diff -u <(cat $tmp.mkfs | sed -e 's/realtime =\/.*extsz=/realtime =external extsz=/g') $tmp.dbinfo +fi _scratch_mount ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 11/12] common/xfs: capture realtime devices during metadump/mdrestore 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong ` (4 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 07/12] xfs/449: update test to know about xfs_db -R Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 05/12] common: filter rtgroups when we're disabling metadir Darrick J. Wong ` (5 subsequent siblings) 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> If xfs_metadump supports the -x and -R switches to capture the contents of realtime devices and there is a realtime device, add the option to the command line to enable preservation. Similarly, if xfs_mdrestore supports the -R switch and there's an external scratch rtdev, pass the option so that we can restore rtdev contents. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 6 +++++- common/populate | 12 ++++++++++-- common/xfs | 48 +++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 54 insertions(+), 12 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index 7034ff8c42..7f96384402 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -302,7 +302,11 @@ __scratch_xfs_fuzz_mdrestore() [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \ logdev=$SCRATCH_LOGDEV - _xfs_mdrestore "${POPULATE_METADUMP}" "${SCRATCH_DEV}" "${logdev}" || \ + local rtdev=none + [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_RTDEV" ] && \ + rtdev=$SCRATCH_RTDEV + + _xfs_mdrestore "${POPULATE_METADUMP}" "${SCRATCH_DEV}" "${logdev}" "${rtdev}" || \ _fail "${POPULATE_METADUMP}: Could not find metadump to restore?" } diff --git a/common/populate b/common/populate index 095e771d67..c0bbbc3f3b 100644 --- a/common/populate +++ b/common/populate @@ -908,7 +908,11 @@ _scratch_populate_restore_cached() { case "${FSTYP}" in "xfs") - _xfs_mdrestore "${metadump}" "${SCRATCH_DEV}" "${logdev}" + local rtdev=none + [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_RTDEV" ] && \ + rtdev=$SCRATCH_RTDEV + + _xfs_mdrestore "${metadump}" "${SCRATCH_DEV}" "${logdev}" "${rtdev}" return $? ;; "ext2"|"ext3"|"ext4") @@ -930,8 +934,12 @@ _scratch_populate_save_metadump() [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \ logdev=$SCRATCH_LOGDEV + local rtdev=none + [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_RTDEV" ] && \ + rtdev=$SCRATCH_RTDEV + _xfs_metadump "$metadump_file" "$SCRATCH_DEV" "$logdev" \ - compress + "$rtdev" compress res=$? ;; "ext2"|"ext3"|"ext4") diff --git a/common/xfs b/common/xfs index 6089a05d0e..a37284068f 100644 --- a/common/xfs +++ b/common/xfs @@ -679,15 +679,20 @@ _xfs_metadump() { local metadump="$1" local device="$2" local logdev="$3" - local compressopt="$4" - shift; shift; shift; shift + local rtdev="$4" + local compressopt="$5" + shift; shift; shift; shift; shift local options="$@" test -z "$options" && options="-a -o" local metadump_has_dash_x + local metadump_has_dash_R # Does metadump support capturing from external devices? $XFS_METADUMP_PROG --help 2>&1 | grep -q -- '-[a-zA-Z]*[wW]x' && \ metadump_has_dash_x=1 + # Does metadump support capturing realtime devices? + $XFS_METADUMP_PROG --help 2>&1 | grep -q -- '-R rtdev' && \ + metadump_has_dash_R=1 if [ "$logdev" != "none" ]; then options="$options -l $logdev" @@ -699,6 +704,17 @@ _xfs_metadump() { fi fi + # Capture the realtime device, if possible + if [ "$rtdev" != "none" ] && [ -n "$metadump_has_dash_R" ]; then + options="$options -R $rtdev" + + # Tell metadump to capture the rt device + if [ -n "$metadump_has_dash_x" ]; then + options="$options -x" + unset metadump_has_dash_x + fi + fi + $XFS_METADUMP_PROG $options "$device" "$metadump" res=$? [ "$compressopt" = "compress" ] && [ -n "$DUMP_COMPRESSOR" ] && @@ -710,7 +726,8 @@ _xfs_mdrestore() { local metadump="$1" local device="$2" local logdev="$3" - shift; shift; shift + local rtdev="$4" + shift; shift; shift; shift local options="$@" # If we're configured for compressed dumps and there isn't already an @@ -730,6 +747,11 @@ _xfs_mdrestore() { logdev="none" fi + # Does mdrestore support restoring to realtime devices? + if [ "$rtdev" != "none" ] && $XFS_MDRESTORE_PROG --help 2>&1 | grep -q -- '-R rtdev'; then + options="$options -R $rtdev" + fi + $XFS_MDRESTORE_PROG $options "${metadump}" "${device}" res=$? test $res -ne 0 && return $res @@ -750,12 +772,16 @@ _scratch_xfs_metadump() { local metadump=$1 shift + local logdev=none - [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \ logdev=$SCRATCH_LOGDEV - _xfs_metadump "$metadump" "$SCRATCH_DEV" "$logdev" nocompress "$@" + local rtdev=none + [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_RTDEV" ] && \ + rtdev=$SCRATCH_RTDEV + + _xfs_metadump "$metadump" "$SCRATCH_DEV" "$logdev" "$rtdev" nocompress "$@" } # Restore snapshotted metadata on the scratch device @@ -768,7 +794,11 @@ _scratch_xfs_mdrestore() [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \ logdev=$SCRATCH_LOGDEV - _xfs_mdrestore "$metadump" "$SCRATCH_DEV" "$logdev" "$@" + local rtdev=none + [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_RTDEV" ] && \ + rtdev=$SCRATCH_RTDEV + + _xfs_mdrestore "$metadump" "$SCRATCH_DEV" "$logdev" "$rtdev" "$@" } # run xfs_check and friends on a FS. @@ -895,7 +925,7 @@ _check_xfs_filesystem() if [ "$ok" -ne 1 ] && [ "$DUMP_CORRUPT_FS" = "1" ]; then local flatdev="$(basename "$device")" _xfs_metadump "$seqres.$flatdev.check.md" "$device" "$logdev" \ - compress >> $seqres.full + "$rtdev" compress >> $seqres.full fi # Optionally test the index rebuilding behavior. @@ -928,7 +958,7 @@ _check_xfs_filesystem() if [ "$rebuild_ok" -ne 1 ] && [ "$DUMP_CORRUPT_FS" = "1" ]; then local flatdev="$(basename "$device")" _xfs_metadump "$seqres.$flatdev.rebuild.md" "$device" \ - "$logdev" compress >> $seqres.full + "$logdev" "$rtdev" compress >> $seqres.full fi fi @@ -1009,7 +1039,7 @@ _check_xfs_filesystem() if [ "$orebuild_ok" -ne 1 ] && [ "$DUMP_CORRUPT_FS" = "1" ]; then local flatdev="$(basename "$device")" _xfs_metadump "$seqres.$flatdev.orebuild.md" "$device" \ - "$logdev" compress >> $seqres.full + "$logdev" "$rtdev" compress >> $seqres.full fi fi ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 05/12] common: filter rtgroups when we're disabling metadir 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong ` (5 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 11/12] common/xfs: capture realtime devices during metadump/mdrestore Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 08/12] xfs/122: update for rtbitmap headers Darrick J. Wong ` (4 subsequent siblings) 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> If we're forcing a filesystem to be created without the metadir feature, we should forcibly disable rtgroups as well. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/common/xfs b/common/xfs index 0d1e0ec4bc..ccdcf45d0d 100644 --- a/common/xfs +++ b/common/xfs @@ -1821,6 +1821,10 @@ _scratch_xfs_find_metafile() # Force metadata directories off. _scratch_xfs_force_no_metadir() { + if echo "$MKFS_OPTIONS" | grep -q 'rtgroups='; then + MKFS_OPTIONS="$(echo "$MKFS_OPTIONS" | sed -e 's/rtgroups=\([01]\)/rtgroups=0/g')" + fi + if echo "$MKFS_OPTIONS" | grep -q 'metadir='; then MKFS_OPTIONS="$(echo "$MKFS_OPTIONS" | sed -e 's/metadir=\([01]\)/metadir=0/g')" return ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 08/12] xfs/122: update for rtbitmap headers 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong ` (6 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 05/12] common: filter rtgroups when we're disabling metadir Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 06/12] xfs/185: update for rtgroups Darrick J. Wong ` (3 subsequent siblings) 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/122.out | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/xfs/122.out b/tests/xfs/122.out index 01376180cc..336618cf7a 100644 --- a/tests/xfs/122.out +++ b/tests/xfs/122.out @@ -115,6 +115,7 @@ sizeof(struct xfs_refcount_key) = 4 sizeof(struct xfs_refcount_rec) = 12 sizeof(struct xfs_rmap_key) = 20 sizeof(struct xfs_rmap_rec) = 24 +sizeof(struct xfs_rtbuf_blkinfo) = 48 sizeof(struct xfs_rtgroup_geometry) = 128 sizeof(struct xfs_rtrmap_key) = 24 sizeof(struct xfs_rtrmap_rec) = 32 ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 06/12] xfs/185: update for rtgroups 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong ` (7 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 08/12] xfs/122: update for rtbitmap headers Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 04/12] common: pass the realtime device to xfs_db when possible Darrick J. Wong ` (2 subsequent siblings) 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Send the fallocate results to seqres.full, since it doesn't matter if the call fails as long as we get the layout that we wanted. This test already has code to check the layout, so there's no point in failing on random ENOSPC errors. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/185 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/xfs/185 b/tests/xfs/185 index abeb052580..04770fd6c9 100755 --- a/tests/xfs/185 +++ b/tests/xfs/185 @@ -100,7 +100,7 @@ test "$ddbytes" -lt "$((rtbytes + (10 * rtextsize) ))" || \ # easy because fallocate for the first rt file always starts allocating at # physical offset zero. alloc_rtx="$((rtbytes / rtextsize))" -$XFS_IO_PROG -c "falloc 0 $((alloc_rtx * rtextsize))" $rtfile +$XFS_IO_PROG -c "falloc 0 $((alloc_rtx * rtextsize))" $rtfile &>> $seqres.full expected_end="$(( (alloc_rtx * rtextsize - 1) / 512 ))" ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 04/12] common: pass the realtime device to xfs_db when possible 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong ` (8 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 06/12] xfs/185: update for rtgroups Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 09/12] xfs/122: udpate test to pick up rtword/suminfo ondisk unions Darrick J. Wong 2022-12-30 22:20 ` [PATCH 12/12] common/fuzzy: adapt the scrub stress tests to support rtgroups Darrick J. Wong 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Teach xfstests to pass the realtime device to xfs_db when it supports that option. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/common/xfs b/common/xfs index 36e02413db..0d1e0ec4bc 100644 --- a/common/xfs +++ b/common/xfs @@ -281,10 +281,10 @@ _xfs_check() { OPTS=" " DBOPTS=" " - USAGE="Usage: xfs_check [-fsvV] [-l logdev] [-i ino]... [-b bno]... special" + USAGE="Usage: xfs_check [-fsvV] [-l logdev] [-R rtdev] [-i ino]... [-b bno]... special" OPTIND=1 - while getopts "b:fi:l:stvV" c; do + while getopts "b:fi:l:stvVR:" c; do case $c in s) OPTS=$OPTS"-s ";; t) OPTS=$OPTS"-t ";; @@ -296,12 +296,14 @@ _xfs_check() V) $XFS_DB_PROG -p xfs_check -V return $? ;; + R) DBOPTS="$DBOPTS -R $OPTARG";; esac done set -- extra $@ shift $OPTIND case $# in - 1) ${XFS_DB_PROG}${DBOPTS} -F -i -p xfs_check -c "check$OPTS" $1 + 1) echo "${XFS_DB_PROG}${DBOPTS} -F -i -p xfs_check -c check$OPTS $1" >> /dev/ttyprintk + ${XFS_DB_PROG}${DBOPTS} -F -i -p xfs_check -c "check$OPTS" $1 status=$? ;; 2) echo $USAGE 1>&1 @@ -339,6 +341,11 @@ _scratch_xfs_db_options() SCRATCH_OPTIONS="" [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \ SCRATCH_OPTIONS="-l$SCRATCH_LOGDEV" + if [ "$USE_EXTERNAL" = yes ] && [ ! -z "$SCRATCH_RTDEV" ]; then + $XFS_DB_PROG --help 2>&1 | grep -q -- '-R rtdev' || \ + _notrun 'xfs_db does not support rt devices' + SCRATCH_OPTIONS="$SCRATCH_OPTIONS -R$SCRATCH_RTDEV" + fi echo $SCRATCH_OPTIONS $* $SCRATCH_DEV } @@ -403,6 +410,11 @@ _scratch_xfs_check() SCRATCH_OPTIONS="-l $SCRATCH_LOGDEV" [ "$LARGE_SCRATCH_DEV" = yes ] && \ SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -t" + if [ "$USE_EXTERNAL" = yes ] && [ ! -z "$SCRATCH_RTDEV" ]; then + $XFS_DB_PROG --help 2>&1 | grep -q -- '-R rtdev' || \ + _notrun 'xfs_db does not support rt devices' + SCRATCH_OPTIONS="$SCRATCH_OPTIONS -R$SCRATCH_RTDEV" + fi _xfs_check $SCRATCH_OPTIONS $* $SCRATCH_DEV } ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 09/12] xfs/122: udpate test to pick up rtword/suminfo ondisk unions 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong ` (9 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 04/12] common: pass the realtime device to xfs_db when possible Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 12/12] common/fuzzy: adapt the scrub stress tests to support rtgroups Darrick J. Wong 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Update this test to check that the ondisk unions for rt bitmap word and rt summary counts are always the correct size. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/122 | 2 +- tests/xfs/122.out | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/xfs/122 b/tests/xfs/122 index e616f1987d..fe6c1e36e9 100755 --- a/tests/xfs/122 +++ b/tests/xfs/122 @@ -187,7 +187,7 @@ echo 'int main(int argc, char *argv[]) {' >>$cprog # cat /usr/include/xfs/xfs*.h | indent |\ _attribute_filter |\ -grep -E '(} *xfs_.*_t|^struct xfs_[a-z0-9_]*$)' |\ +grep -E '(} *xfs_.*_t|^(union|struct) xfs_[a-z0-9_]*$)' |\ grep -E -v -f $tmp.ignore |\ sed -e 's/^.*}[[:space:]]*//g' -e 's/;.*$//g' -e 's/_t, /_t\n/g' |\ sort | uniq |\ diff --git a/tests/xfs/122.out b/tests/xfs/122.out index 336618cf7a..1379c7b3b5 100644 --- a/tests/xfs/122.out +++ b/tests/xfs/122.out @@ -128,6 +128,8 @@ sizeof(struct xfs_swap_extent) = 64 sizeof(struct xfs_sxd_log_format) = 16 sizeof(struct xfs_sxi_log_format) = 80 sizeof(struct xfs_unmount_log_format) = 8 +sizeof(union xfs_rtword_ondisk) = 4 +sizeof(union xfs_suminfo_ondisk) = 4 sizeof(xfs_agf_t) = 224 sizeof(xfs_agfl_t) = 36 sizeof(xfs_agi_t) = 344 ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 12/12] common/fuzzy: adapt the scrub stress tests to support rtgroups 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong ` (10 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 09/12] xfs/122: udpate test to pick up rtword/suminfo ondisk unions Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 11 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Adapt the scrub stress testing framework to support checking realtime group metadata. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/fuzzy | 27 ++++++++++++++++++++++----- common/xfs | 9 +++++++++ tests/xfs/800 | 2 +- tests/xfs/840 | 2 +- tests/xfs/841 | 2 +- 5 files changed, 34 insertions(+), 8 deletions(-) diff --git a/common/fuzzy b/common/fuzzy index 7f96384402..baba05de0a 100644 --- a/common/fuzzy +++ b/common/fuzzy @@ -817,8 +817,10 @@ __stress_one_scrub_loop() { local scrub_tgt="$3" local scrub_startat="$4" local start_agno="$5" - shift; shift; shift; shift; shift + local start_rgno="$6" + shift; shift; shift; shift; shift; shift local agcount="$(_xfs_mount_agcount $SCRATCH_MNT)" + local rgcount="$(_xfs_mount_rgcount $SCRATCH_MNT)" local xfs_io_args=() for arg in "$@"; do @@ -831,6 +833,12 @@ __stress_one_scrub_loop() { local ag_arg="$(echo "$arg" | sed -e "s|%agno%|$agno|g")" xfs_io_args+=('-c' "$ag_arg") done + elif echo "$arg" | grep -q -w '%rgno%'; then + # Substitute the rtgroup number + for ((rgno = start_rgno; rgno < rgcount; rgno++)); do + local rg_arg="$(echo "$arg" | sed -e "s|%rgno%|$rgno|g")" + xfs_io_args+=('-c' "$rg_arg") + done else xfs_io_args+=('-c' "$arg") fi @@ -1201,7 +1209,9 @@ _scratch_xfs_stress_scrub_cleanup() { __stress_scrub_check_commands() { local scrub_tgt="$1" local start_agno="$2" - shift; shift + local start_rgno="$3" + shift; shift; shift + local rgcount="$(_xfs_mount_rgcount $SCRATCH_MNT)" local cooked_tgt="$scrub_tgt" case "$scrub_tgt" in @@ -1231,6 +1241,10 @@ __stress_scrub_check_commands() { cooked_arg="$(echo "$cooked_arg" | sed -e 's/^repair/repair -R/g')" fi cooked_arg="$(echo "$cooked_arg" | sed -e "s/%agno%/$start_agno/g")" + if echo "$cooked_arg" | grep -q -w '%rgno%'; then + test "$rgcount" -eq 0 && continue + cooked_arg="$(echo "$cooked_arg" | sed -e "s/%rgno%/$start_rgno/g")" + fi testio=`$XFS_IO_PROG -x -c "$cooked_arg" "$cooked_tgt" 2>&1` echo $testio | grep -q "Unknown type" && \ _notrun "xfs_io scrub subcommand support is missing" @@ -1256,6 +1270,7 @@ __stress_scrub_check_commands() { # in a separate loop. If zero -i options are specified, do not run. # Callers must check each of these commands (via _require_xfs_io_command) # before calling here. +# -R For %rgno% substitution, start with this rtgroup instead of rtgroup 0. # -r Run fsstress for this amount of time, then remount the fs ro or rw. # The default is to run fsstress continuously with no remount, unless # XFS_SCRUB_STRESS_REMOUNT_PERIOD is set. @@ -1301,6 +1316,7 @@ _scratch_xfs_stress_scrub() { local remount_period="${XFS_SCRUB_STRESS_REMOUNT_PERIOD}" local stress_tgt="${XFS_SCRUB_STRESS_TARGET:-default}" local start_agno=0 + local start_rgno=0 __SCRUB_STRESS_FREEZE_PID="" __SCRUB_STRESS_REMOUNT_LOOP="" @@ -1308,12 +1324,13 @@ _scratch_xfs_stress_scrub() { touch "$runningfile" OPTIND=1 - while getopts "a:fi:r:s:S:t:w:x:X:" c; do + while getopts "a:fi:r:R:s:S:t:w:x:X:" c; do case "$c" in a) start_agno="$OPTARG";; f) freeze=yes;; i) io_args+=("$OPTARG");; r) remount_period="$OPTARG";; + R) start_rgno="$OPTARG";; s) one_scrub_args+=("$OPTARG");; S) xfs_scrub_args+=("$OPTARG");; t) scrub_tgt="$OPTARG";; @@ -1324,7 +1341,7 @@ _scratch_xfs_stress_scrub() { esac done - __stress_scrub_check_commands "$scrub_tgt" "$start_agno" \ + __stress_scrub_check_commands "$scrub_tgt" "$start_agno" "$start_rgno" \ "${one_scrub_args[@]}" if ! command -v "__stress_scrub_${exerciser}_loop" &>/dev/null; then @@ -1372,7 +1389,7 @@ _scratch_xfs_stress_scrub() { if [ "${#one_scrub_args[@]}" -gt 0 ]; then __stress_one_scrub_loop "$end" "$runningfile" "$scrub_tgt" \ - "$scrub_startat" "$start_agno" \ + "$scrub_startat" "$start_agno" "$start_rgno" \ "${one_scrub_args[@]}" & fi diff --git a/common/xfs b/common/xfs index a37284068f..f451dfb8ae 100644 --- a/common/xfs +++ b/common/xfs @@ -1524,6 +1524,15 @@ _xfs_mount_agcount() $XFS_INFO_PROG "$1" | sed -n "s/^.*agcount=\([[:digit:]]*\).*/\1/p" } +# Find rtgroup count of mounted filesystem +_xfs_mount_rgcount() +{ + local rtgroups="$($XFS_INFO_PROG "$1" | grep rgcount= | sed -e 's/^.*rgcount=\([0-9]*\).*$/\1/g')" + + test -z "$rtgroups" && rtgroups=0 + echo "$rtgroups" +} + # Wipe the superblock of each XFS AGs _try_wipe_scratch_xfs() { diff --git a/tests/xfs/800 b/tests/xfs/800 index cbcfb5f5a6..a2542be5ec 100755 --- a/tests/xfs/800 +++ b/tests/xfs/800 @@ -32,7 +32,7 @@ _require_xfs_stress_scrub _scratch_mkfs > "$seqres.full" 2>&1 _scratch_mount _require_xfs_has_feature "$SCRATCH_MNT" realtime -_scratch_xfs_stress_scrub -s "scrub rtbitmap" +_scratch_xfs_stress_scrub -s "scrub rtbitmap" -s "scrub rgbitmap %rgno%" # success, all done echo Silence is golden diff --git a/tests/xfs/840 b/tests/xfs/840 index fff41c5b8a..b9ed0a55b3 100755 --- a/tests/xfs/840 +++ b/tests/xfs/840 @@ -39,7 +39,7 @@ alloc_unit=$(_get_file_block_size $SCRATCH_MNT) scratchfile=$SCRATCH_MNT/file touch $scratchfile $XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT -__stress_scrub_check_commands "$scratchfile" "" 'repair bmapbtd' +__stress_scrub_check_commands "$scratchfile" "" "" 'repair bmapbtd' # Compute the number of extent records needed to guarantee btree format, # assuming 16 bytes for each ondisk extent record diff --git a/tests/xfs/841 b/tests/xfs/841 index f743454971..5831961f5f 100755 --- a/tests/xfs/841 +++ b/tests/xfs/841 @@ -39,7 +39,7 @@ scratchfile=$SCRATCH_MNT/file mkdir $scratchdir touch $scratchfile $XFS_IO_PROG -x -c 'inject force_repair' $SCRATCH_MNT -__stress_scrub_check_commands "$scratchdir" "" 'repair directory' +__stress_scrub_check_commands "$scratchdir" "" "" 'repair directory' # Create a 2-dirblock directory total_size=$((alloc_unit * 2)) ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap 2022-12-30 21:14 [NYE DELUGE 3/4] xfs: modernize the realtime volume Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 03/13] xfs: race fsstress with realtime rmap btree scrub and repair Darrick J. Wong ` (12 more replies) 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong ` (2 subsequent siblings) 7 siblings, 13 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, Fix a few regressions in fstests when rmap is enabled on the realtime volume. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-rmap xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=realtime-rmap fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-rmap xfsdocs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=realtime-rmap --- common/populate | 36 +++++++++++++++++++++++++++++++++++- common/xfs | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/104 | 1 + tests/xfs/122.out | 5 ++--- tests/xfs/1528 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/1528.out | 4 ++++ tests/xfs/1529 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1529.out | 4 ++++ tests/xfs/272 | 2 +- tests/xfs/276 | 2 +- tests/xfs/277 | 2 +- tests/xfs/291 | 3 ++- tests/xfs/332 | 6 +----- tests/xfs/332.out | 2 -- tests/xfs/333 | 45 --------------------------------------------- tests/xfs/333.out | 6 ------ tests/xfs/337 | 2 +- tests/xfs/338 | 21 ++++++++++++++++----- tests/xfs/339 | 5 +++-- tests/xfs/340 | 15 ++++++++++----- tests/xfs/341 | 12 ++++-------- tests/xfs/341.out | 1 - tests/xfs/342 | 4 ++-- tests/xfs/343 | 2 ++ tests/xfs/406 | 6 ++++-- tests/xfs/407 | 6 ++++-- tests/xfs/408 | 7 +++++-- tests/xfs/409 | 7 +++++-- tests/xfs/443 | 9 +++++---- tests/xfs/481 | 6 ++++-- tests/xfs/482 | 7 +++++-- tests/xfs/769 | 29 ++++++++++++++++++++++++++++- tests/xfs/781 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/781.out | 2 ++ tests/xfs/817 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/817.out | 2 ++ tests/xfs/821 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/821.out | 2 ++ 38 files changed, 414 insertions(+), 107 deletions(-) create mode 100755 tests/xfs/1528 create mode 100644 tests/xfs/1528.out create mode 100755 tests/xfs/1529 create mode 100644 tests/xfs/1529.out delete mode 100755 tests/xfs/333 delete mode 100644 tests/xfs/333.out create mode 100755 tests/xfs/781 create mode 100644 tests/xfs/781.out create mode 100755 tests/xfs/817 create mode 100644 tests/xfs/817.out create mode 100755 tests/xfs/821 create mode 100644 tests/xfs/821.out ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCH 03/13] xfs: race fsstress with realtime rmap btree scrub and repair 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 04/13] xfs/769: add rtrmapbt upgrade to test matrix Darrick J. Wong ` (11 subsequent siblings) 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Race checking and rebuilding realtime rmap btrees with fsstress. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/781 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/781.out | 2 ++ tests/xfs/817 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/817.out | 2 ++ tests/xfs/821 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/821.out | 2 ++ 6 files changed, 132 insertions(+) create mode 100755 tests/xfs/781 create mode 100644 tests/xfs/781.out create mode 100755 tests/xfs/817 create mode 100644 tests/xfs/817.out create mode 100755 tests/xfs/821 create mode 100644 tests/xfs/821.out diff --git a/tests/xfs/781 b/tests/xfs/781 new file mode 100755 index 0000000000..938777952f --- /dev/null +++ b/tests/xfs/781 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 781 +# +# Race fsstress and rtrmapbt repair for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime +_require_xfs_has_feature "$SCRATCH_MNT" rmapbt +_xfs_force_bdev realtime $SCRATCH_MNT + +_scratch_xfs_stress_online_repair -s "repair rtrmapbt %rgno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/781.out b/tests/xfs/781.out new file mode 100644 index 0000000000..e7f74cf644 --- /dev/null +++ b/tests/xfs/781.out @@ -0,0 +1,2 @@ +QA output created by 781 +Silence is golden diff --git a/tests/xfs/817 b/tests/xfs/817 new file mode 100755 index 0000000000..88d0a18e8d --- /dev/null +++ b/tests/xfs/817 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 817 +# +# Race fsstress and rtrmapbt scrub for a while to see if we crash or livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime +_require_xfs_has_feature "$SCRATCH_MNT" rmapbt +_xfs_force_bdev realtime $SCRATCH_MNT + +_scratch_xfs_stress_scrub -s "scrub rtrmapbt %rgno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/817.out b/tests/xfs/817.out new file mode 100644 index 0000000000..86920a4fc6 --- /dev/null +++ b/tests/xfs/817.out @@ -0,0 +1,2 @@ +QA output created by 817 +Silence is golden diff --git a/tests/xfs/821 b/tests/xfs/821 new file mode 100755 index 0000000000..45b999e3b5 --- /dev/null +++ b/tests/xfs/821 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 821 +# +# Race fsstress and realtime bitmap repair for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime +_xfs_force_bdev realtime $SCRATCH_MNT + +_scratch_xfs_stress_online_repair -s "repair rtbitmap" -s "repair rgbitmap %rgno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/821.out b/tests/xfs/821.out new file mode 100644 index 0000000000..17994b8627 --- /dev/null +++ b/tests/xfs/821.out @@ -0,0 +1,2 @@ +QA output created by 821 +Silence is golden ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 04/13] xfs/769: add rtrmapbt upgrade to test matrix 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong 2022-12-30 22:20 ` [PATCH 03/13] xfs: race fsstress with realtime rmap btree scrub and repair Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 05/13] xfs/122: update for rtgroups-based realtime rmap btrees Darrick J. Wong ` (10 subsequent siblings) 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add realtime reverse mapping btrees to the features that this test will try to upgrade. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/769 | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/tests/xfs/769 b/tests/xfs/769 index 624dd2a338..ccc3ea10bc 100755 --- a/tests/xfs/769 +++ b/tests/xfs/769 @@ -35,11 +35,36 @@ rt_configured() test "$USE_EXTERNAL" = "yes" && test -n "$SCRATCH_RTDEV" } +need_rtgroups() +{ + local feat="$1" + + # if realtime isn't configured, we don't need rt groups + rt_configured || return 1 + + # rt rmap btrees require rt groups but rt groups cannot be added to + # an existing filesystem, so we must force it on at mkfs time + test "${FEATURE_STATE["rmapbt"]}" -eq 1 && return 0 + test "$feat" = "rmapbt" && return 0 + + return 1 +} + # Compute the MKFS_OPTIONS string for a particular feature upgrade test compute_mkfs_options() { + local feat="$1" local m_opts="" local caller_options="$MKFS_OPTIONS" + local rtgroups + + need_rtgroups "$feat" && rtgroups=1 + if echo "$caller_options" | grep -q 'rtgroups='; then + test -z "$rtgroups" && rtgroups=0 + caller_options="$(echo "$caller_options" | sed -e 's/rtgroups=*[0-9]*/rtgroups='$rtgroups'/g')" + elif [ -n "$rtgroups" ]; then + caller_options="$caller_options -r rtgroups=$rtgroups" + fi for feat in "${FEATURES[@]}"; do local feat_state="${FEATURE_STATE["${feat}"]}" @@ -171,10 +196,12 @@ function post_exercise() # upgrade don't spread failure to the rest of the tests. FEATURES=() if rt_configured; then + # rmap wasn't added to rt devices until after metadir check_repair_upgrade finobt && FEATURES+=("finobt") check_repair_upgrade inobtcount && FEATURES+=("inobtcount") check_repair_upgrade bigtime && FEATURES+=("bigtime") check_repair_upgrade metadir && FEATURES+=("metadir") + check_repair_upgrade rmapbt && FEATURES+=("rmapbt") else check_repair_upgrade finobt && FEATURES+=("finobt") check_repair_upgrade rmapbt && FEATURES+=("rmapbt") @@ -197,7 +224,7 @@ for feat in "${FEATURES[@]}"; do upgrade_start_message "$feat" | tee -a $seqres.full /dev/ttyprintk > /dev/null - opts="$(compute_mkfs_options)" + opts="$(compute_mkfs_options "$feat")" echo "mkfs.xfs $opts" >> $seqres.full # Format filesystem ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 05/13] xfs/122: update for rtgroups-based realtime rmap btrees 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong 2022-12-30 22:20 ` [PATCH 03/13] xfs: race fsstress with realtime rmap btree scrub and repair Darrick J. Wong 2022-12-30 22:20 ` [PATCH 04/13] xfs/769: add rtrmapbt upgrade to test matrix Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 07/13] xfs/341: update test for rtgroup-based rmap Darrick J. Wong ` (9 subsequent siblings) 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Now that we've redesigned realtime rmap to require that the rt section be sharded into allocation groups of no more than 2^31 blocks, we've reduced the size of the ondisk structures and therefore need to update this test. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/122.out | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/xfs/122.out b/tests/xfs/122.out index 53eff0027e..e0801f9660 100644 --- a/tests/xfs/122.out +++ b/tests/xfs/122.out @@ -116,8 +116,8 @@ sizeof(struct xfs_rmap_key) = 20 sizeof(struct xfs_rmap_rec) = 24 sizeof(struct xfs_rtbuf_blkinfo) = 48 sizeof(struct xfs_rtgroup_geometry) = 128 -sizeof(struct xfs_rtrmap_key) = 24 -sizeof(struct xfs_rtrmap_rec) = 32 +sizeof(struct xfs_rtrmap_key) = 20 +sizeof(struct xfs_rtrmap_rec) = 24 sizeof(struct xfs_rtrmap_root) = 4 sizeof(struct xfs_rtsb) = 104 sizeof(struct xfs_rud_log_format) = 16 ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 07/13] xfs/341: update test for rtgroup-based rmap 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 05/13] xfs/122: update for rtgroups-based realtime rmap btrees Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 06/13] xfs: fix various problems with fsmap detecting the data device Darrick J. Wong ` (8 subsequent siblings) 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Now that we're sharding the realtime volume into multiple allocation groups, update this test to reflect the new reality. The realtime rmap btree record and key sizes have shrunk, and we can't guarantee that a quick file write actually hits the same rt group as the one we fuzzed, so eliminate the file write test since we're really only curious if xfs_repair will fix the problem. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/341 | 10 +++------- tests/xfs/341.out | 1 - 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/tests/xfs/341 b/tests/xfs/341 index 8861e751a9..561054f0bd 100755 --- a/tests/xfs/341 +++ b/tests/xfs/341 @@ -32,10 +32,10 @@ blksz="$(_get_block_size $SCRATCH_MNT)" rtextsz_blks=$((rtextsz / blksz)) # inode core size is at least 176 bytes; btree header is 56 bytes; -# rtrmap record is 32 bytes; and rtrmap key/pointer are 56 bytes. +# rtrmap record is 24 bytes; and rtrmap key/pointer are 48 bytes. i_core_size="$(_xfs_get_inode_core_bytes $SCRATCH_MNT)" -i_ptrs=$(( (isize - i_core_size) / 56 )) -bt_recs=$(( (blksz - 56) / 32 )) +i_ptrs=$(( (isize - i_core_size) / 48 )) +bt_recs=$(( (blksz - 56) / 24 )) blocks=$((i_ptrs * bt_recs + 1)) len=$((blocks * rtextsz)) @@ -57,10 +57,6 @@ _scratch_xfs_db -x -c 'path -m /realtime/0.rmap' \ -c "write u3.rtrmapbt.ptrs[1] $fsbno" -c 'p' >> $seqres.full _scratch_mount -echo "Try to create more files" -$XFS_IO_PROG -f -R -c "pwrite -S 0x68 0 9999" $SCRATCH_MNT/f5 >> $seqres.full 2>&1 -test -e $SCRATCH_MNT/f5 && echo "should not have been able to write f5" - echo "Repair fs" _scratch_unmount 2>&1 | _filter_scratch _repair_scratch_fs >> $seqres.full 2>&1 diff --git a/tests/xfs/341.out b/tests/xfs/341.out index 75a5bc6c61..580d788954 100644 --- a/tests/xfs/341.out +++ b/tests/xfs/341.out @@ -2,6 +2,5 @@ QA output created by 341 Format and mount Create some files Corrupt fs -Try to create more files Repair fs Try to create more files (again) ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 06/13] xfs: fix various problems with fsmap detecting the data device 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 07/13] xfs/341: update test for rtgroup-based rmap Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 01/13] xfs: fix tests that try to access the realtime rmap inode Darrick J. Wong ` (7 subsequent siblings) 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Various tests of realtime rmap functionality assumed that the data device could be picked out from the GETFSMAP output by looking for static fs metadata. This is no longer true, since rtgroups have a static superblock header at the start, so update these tests. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/272 | 2 +- tests/xfs/276 | 2 +- tests/xfs/277 | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/xfs/272 b/tests/xfs/272 index 7ed8b95122..42b4a2edb5 100755 --- a/tests/xfs/272 +++ b/tests/xfs/272 @@ -57,7 +57,7 @@ cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total c done echo "Check device field of FS metadata and regular file" -data_dev=$(grep 'static fs metadata' $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') +data_dev=$(grep 'inode btree' $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') rt_dev=$(grep "${ino}[[:space:]]*[0-9]*\.\.[0-9]*" $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') test "${data_dev}" = "${rt_dev}" || echo "data ${data_dev} realtime ${rt_dev}?" diff --git a/tests/xfs/276 b/tests/xfs/276 index 8cc486752a..a05ca1961d 100755 --- a/tests/xfs/276 +++ b/tests/xfs/276 @@ -61,7 +61,7 @@ cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total c done echo "Check device field of FS metadata and realtime file" -data_dev=$(grep 'static fs metadata' $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') +data_dev=$(grep 'inode btree' $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') rt_dev=$(grep "${ino}[[:space:]]*[0-9]*\.\.[0-9]*[[:space:]]*[0-9]*$" $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') test "${data_dev}" != "${rt_dev}" || echo "data ${data_dev} realtime ${rt_dev}?" diff --git a/tests/xfs/277 b/tests/xfs/277 index 03208ef233..eff54a2a50 100755 --- a/tests/xfs/277 +++ b/tests/xfs/277 @@ -38,7 +38,7 @@ $XFS_IO_PROG -c 'fsmap -v' $SCRATCH_MNT >> $seqres.full $XFS_IO_PROG -c 'fsmap -v' $SCRATCH_MNT | tr '[]()' ' ' > $TEST_DIR/fsmap echo "Check device field of FS metadata and journalling log" -data_dev=$(grep 'static fs metadata' $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') +data_dev=$(grep 'inode btree' $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') journal_dev=$(grep 'journalling log' $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') test "${data_dev}" = "${journal_dev}" || echo "data ${data_dev} journal ${journal_dev}?" ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 01/13] xfs: fix tests that try to access the realtime rmap inode 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (4 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 06/13] xfs: fix various problems with fsmap detecting the data device Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 02/13] fuzz: for fuzzing the rtrmapbt, find the path to the rt rmap btree file Darrick J. Wong ` (6 subsequent siblings) 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The realtime rmap tests were added to fstests a long time ago. Since they were added, we decided to create a metadata file directory structure instead of adding more fields to the superblock. Therefore, fix all the tests that try to access these paths. While we're at it, fix xfs/409 to run the *online* scrub program like it's supposed to. xfs/408 is the fuzzer for xfs_repair testing. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 18 ++++++++++++++++++ tests/xfs/122.out | 1 - tests/xfs/333 | 45 --------------------------------------------- tests/xfs/333.out | 6 ------ tests/xfs/337 | 2 +- tests/xfs/338 | 21 ++++++++++++++++----- tests/xfs/339 | 5 +++-- tests/xfs/340 | 15 ++++++++++----- tests/xfs/341 | 2 +- tests/xfs/342 | 4 ++-- 10 files changed, 51 insertions(+), 68 deletions(-) delete mode 100755 tests/xfs/333 delete mode 100644 tests/xfs/333.out diff --git a/common/xfs b/common/xfs index f451dfb8ae..aea2b678c8 100644 --- a/common/xfs +++ b/common/xfs @@ -1922,3 +1922,21 @@ _require_xfs_scratch_metadir() _scratch_unmount fi } + +# Resolve a metadata directory tree path and return the inode number. +_scratch_metadir_lookup() { + local res="$(_scratch_xfs_db -c "ls -i -m $1")" + test "${PIPESTATUS[0]}" -eq 0 && echo "$res" +} + +# Figure out which directory entry we have to change to update the rtrmap +# inode pointer. Assumes the /realtime directory is a short format dir. +_scratch_find_rt_metadir_entry() { + local sfkey="$(_scratch_xfs_db -c 'path -m /realtime' -c print | \ + grep "\"$1\"" | \ + sed -e 's/.name.*$//g' -e 's/\[/\\[/g' -e 's/\]/\\]/g' )" + test -n "$sfkey" || return 1 + _scratch_xfs_db -c 'path -m /realtime' -c print | \ + grep "${sfkey}.inumber" | awk '{print $1}' + return 0 +} diff --git a/tests/xfs/122.out b/tests/xfs/122.out index 1379c7b3b5..53eff0027e 100644 --- a/tests/xfs/122.out +++ b/tests/xfs/122.out @@ -48,7 +48,6 @@ offsetof(xfs_sb_t, sb_rgblklog) = 280 offsetof(xfs_sb_t, sb_rgblocks) = 272 offsetof(xfs_sb_t, sb_rgcount) = 276 offsetof(xfs_sb_t, sb_rootino) = 56 -offsetof(xfs_sb_t, sb_rrmapino) = 264 offsetof(xfs_sb_t, sb_rsumino) = 72 offsetof(xfs_sb_t, sb_sectlog) = 121 offsetof(xfs_sb_t, sb_sectsize) = 102 diff --git a/tests/xfs/333 b/tests/xfs/333 deleted file mode 100755 index 728c518402..0000000000 --- a/tests/xfs/333 +++ /dev/null @@ -1,45 +0,0 @@ -#! /bin/bash -# SPDX-License-Identifier: GPL-2.0 -# Copyright (c) 2016, Oracle and/or its affiliates. All Rights Reserved. -# -# FS QA Test No. 333 -# -# Set rrmapino to another inode on an non-rt rmap fs and see if repair fixes it. -# -. ./common/preamble -_begin_fstest auto quick rmap realtime - -# Import common functions. -. ./common/filter - -# real QA test starts here -_supported_fs xfs -_require_xfs_scratch_rmapbt -_disable_dmesg_check - -rm -f "$seqres.full" - -unset SCRATCH_RTDEV - -echo "Format and mount" -_scratch_mkfs > "$seqres.full" 2>&1 -rrmapino="$(_scratch_xfs_db -c 'sb 0' -c 'p rrmapino' 2>&1)" -test "${rrmapino}" = "field rrmapino not found" && _notrun "realtime rmapbt not supported" -_scratch_mount - -echo "Create some files" -$XFS_IO_PROG -f -c "pwrite -S 0x68 0 9999" $SCRATCH_MNT/f1 >> $seqres.full -$XFS_IO_PROG -f -c "pwrite -S 0x68 0 9999" $SCRATCH_MNT/f2 >> $seqres.full -echo garbage > $SCRATCH_MNT/f3 -ino=$(stat -c '%i' $SCRATCH_MNT/f3) -_scratch_unmount - -echo "Corrupt fs" -_scratch_xfs_db -x -c 'sb 0' -c "write rrmapino $ino" >> $seqres.full -_try_scratch_mount 2>&1 | _filter_error_mount - -echo "Test done, mount should have failed" - -# success, all done -status=0 -exit diff --git a/tests/xfs/333.out b/tests/xfs/333.out deleted file mode 100644 index b3c698750f..0000000000 --- a/tests/xfs/333.out +++ /dev/null @@ -1,6 +0,0 @@ -QA output created by 333 -Format and mount -Create some files -Corrupt fs -mount: Structure needs cleaning -Test done, mount should have failed diff --git a/tests/xfs/337 b/tests/xfs/337 index f74baae9b0..9ea8587b27 100755 --- a/tests/xfs/337 +++ b/tests/xfs/337 @@ -53,7 +53,7 @@ echo "+ check fs" _scratch_xfs_repair -n >> $seqres.full 2>&1 || echo "xfs_repair should not fail" echo "+ corrupt image" -_scratch_xfs_db -x -c "sb" -c "addr rrmapino" -c "addr u3.rtrmapbt.ptrs[1]" \ +_scratch_xfs_db -x -c "path -m /realtime/0.rmap" -c "addr u3.rtrmapbt.ptrs[1]" \ -c "stack" -c "blocktrash -x 4096 -y 4096 -n 8 -3 -z" \ >> $seqres.full 2>&1 diff --git a/tests/xfs/338 b/tests/xfs/338 index 9f36150c7e..9d41a83ec2 100755 --- a/tests/xfs/338 +++ b/tests/xfs/338 @@ -29,13 +29,24 @@ $XFS_IO_PROG -f -R -c "pwrite -S 0x68 0 9999" $SCRATCH_MNT/f2 >> $seqres.full _scratch_unmount echo "Corrupt fs" -_scratch_xfs_db -x -c 'sb 0' -c 'addr rrmapino' \ - -c 'write core.nlinkv2 0' -c 'write core.mode 0' -c 'sb 0' \ - -c 'write rrmapino 0' >> $seqres.full -_try_scratch_mount >> $seqres.full 2>&1 && echo "mount should have failed" +rtrmap_sfentry="$(_scratch_find_rt_metadir_entry 0.rmap)" +test -n "$rtrmap_sfentry" || _fail "Could not find rtrmap metadir entry?" +_scratch_xfs_db -x -c 'path -m /realtime/0.rmap' \ + -c 'write core.nlinkv2 0' -c 'write core.mode 0' \ + -c 'path -m /realtime' \ + -c "write $rtrmap_sfentry 0" >> $seqres.full +if _try_scratch_mount >> $seqres.full 2>&1; then + echo "mount should have failed" + _scratch_unmount +else + # If the verifiers are working properly, the mount will fail because + # we fuzzed the metadata root directory. This causes loud complaints + # to dmesg, so we want to ignore those. + _disable_dmesg_check +fi echo "Repair fs" -_scratch_unmount 2>&1 | _filter_scratch +_scratch_unmount 2>&1 | _filter_scratch | _filter_ending_dot _repair_scratch_fs >> $seqres.full 2>&1 echo "Try to create more files (again)" diff --git a/tests/xfs/339 b/tests/xfs/339 index 3e0b4d97ab..24a90d0ba3 100755 --- a/tests/xfs/339 +++ b/tests/xfs/339 @@ -31,7 +31,8 @@ ln $SCRATCH_MNT/f3 $SCRATCH_MNT/f4 _scratch_unmount echo "Corrupt fs" -rrmapino=`_scratch_xfs_get_sb_field rrmapino` +rrmapino=$(_scratch_metadir_lookup /realtime/0.rmap) +test -n "$rrmapino" || _fail "Could not find rtrmap inode?" _scratch_xfs_set_metadata_field "u3.sfdir3.list[3].inumber.i4" $rrmapino \ 'sb 0' 'addr rootino' >> $seqres.full _scratch_mount @@ -43,7 +44,7 @@ echo "Try to create more files" $XFS_IO_PROG -f -R -c "pwrite -S 0x68 0 9999" $SCRATCH_MNT/f5 >> $seqres.full 2>&1 echo "Repair fs" -_scratch_unmount 2>&1 | _filter_scratch +_scratch_unmount 2>&1 | _filter_scratch | _filter_ending_dot _repair_scratch_fs >> $seqres.full 2>&1 echo "Try to create more files (again)" diff --git a/tests/xfs/340 b/tests/xfs/340 index 2c0014513e..1236f6520f 100755 --- a/tests/xfs/340 +++ b/tests/xfs/340 @@ -31,16 +31,21 @@ ino=$(stat -c '%i' $SCRATCH_MNT/f3) _scratch_unmount echo "Corrupt fs" -rrmapino=$(_scratch_xfs_get_sb_field rrmapino) -_scratch_xfs_db -x -c "inode $rrmapino" \ +rtrmap_sfentry="$(_scratch_find_rt_metadir_entry 0.rmap)" +test -n "$rtrmap_sfentry" || _fail "Could not find rtrmap metadir entry?" +rrmapino=$(_scratch_metadir_lookup /realtime/0.rmap) +test -n "$rrmapino" || _fail "Could not find rtrmap inode?" +_scratch_xfs_db -x -c "path -m /realtime/0.rmap" \ -c 'write core.format 2' -c 'write core.size 0' \ - -c 'write core.nblocks 0' -c 'sb 0' -c 'addr rootino' \ + -c 'write core.nblocks 0' \ + -c 'sb 0' -c 'addr rootino' \ -c "write u3.sfdir3.list[2].inumber.i4 $rrmapino" \ - -c 'sb 0' -c "write rrmapino $ino" >> $seqres.full + -c 'path -m /realtime' \ + -c "write $rtrmap_sfentry $ino" >> $seqres.full _try_scratch_mount >> $seqres.full 2>&1 && echo "mount should have failed" echo "Repair fs" -_scratch_unmount 2>&1 | _filter_scratch +_scratch_unmount 2>&1 | _filter_scratch | _filter_ending_dot _repair_scratch_fs >> $seqres.full 2>&1 echo "Try to create more files (again)" diff --git a/tests/xfs/341 b/tests/xfs/341 index 7d2842b579..8861e751a9 100755 --- a/tests/xfs/341 +++ b/tests/xfs/341 @@ -53,7 +53,7 @@ echo "Corrupt fs" fsbno=$(_scratch_xfs_db -c "inode $ino" -c 'bmap' | grep 'flag 0' | head -n 1 | \ sed -e 's/^.*startblock \([0-9]*\) .*$/\1/g') -_scratch_xfs_db -x -c 'sb 0' -c 'addr rrmapino' \ +_scratch_xfs_db -x -c 'path -m /realtime/0.rmap' \ -c "write u3.rtrmapbt.ptrs[1] $fsbno" -c 'p' >> $seqres.full _scratch_mount diff --git a/tests/xfs/342 b/tests/xfs/342 index 538c8987ef..f29bd874e9 100755 --- a/tests/xfs/342 +++ b/tests/xfs/342 @@ -47,9 +47,9 @@ ino=$(stat -c '%i' $SCRATCH_MNT/f3) _scratch_unmount echo "Corrupt fs" -_scratch_xfs_db -c 'sb 0' -c 'addr rrmapino' -c 'p u3.rtrmapbt.ptrs[1]' >> $seqres.full +_scratch_xfs_db -c 'path -m /realtime/0.rmap' -c 'p u3.rtrmapbt.ptrs[1]' >> $seqres.full -fsbno=$(_scratch_xfs_db -c 'sb 0' -c 'addr rrmapino' \ +fsbno=$(_scratch_xfs_db -c 'path -m /realtime/0.rmap' \ -c 'p u3.rtrmapbt.ptrs[1]' | sed -e 's/^.*://g') _scratch_xfs_db -x -c "inode $ino" -c "write u3.bmx[0].startblock $fsbno" >> $seqres.full _scratch_mount ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 02/13] fuzz: for fuzzing the rtrmapbt, find the path to the rt rmap btree file 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (5 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 01/13] xfs: fix tests that try to access the realtime rmap inode Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 09/13] xfs: skip tests if formatting small filesystem fails Darrick J. Wong ` (5 subsequent siblings) 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The fs population code creates a realtime rmap btree in /some/ realtime group with at least two levels. This rmapbt file isn't necessarily the one for group 0, so we need to find it programmatically. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 33 +++++++++++++++++++++++++++++++++ tests/xfs/406 | 6 ++++-- tests/xfs/407 | 6 ++++-- tests/xfs/408 | 7 +++++-- tests/xfs/409 | 7 +++++-- tests/xfs/481 | 6 ++++-- tests/xfs/482 | 7 +++++-- 7 files changed, 60 insertions(+), 12 deletions(-) diff --git a/common/xfs b/common/xfs index aea2b678c8..63eff39d47 100644 --- a/common/xfs +++ b/common/xfs @@ -1814,6 +1814,39 @@ _scratch_xfs_find_agbtree_height() { return 1 } +# Find us the path to the inode containing a realtime btree with a specific +# height. +_scratch_xfs_find_rgbtree_height() { + local bt_type="$1" + local bt_height="$2" + local rgcount=$(_xfs_mount_rgcount $SCRATCH_DEV) + local path + local path_format + local bt_prefix + + case "${bt_type}" in + "rmap") + path_format="/realtime/%u.rmap" + bt_prefix="u3.rtrmapbt" + ;; + *) + _fail "Don't know about rt btree ${bt_type}" + ;; + esac + + for ((rgno = 0; rgno < rgcount; rgno++)); do + path="$(printf "${path_format}" "${rgno}")" + bt_level=$(_scratch_xfs_db -c "path -m ${path}" -c "p ${bt_prefix}.level" | awk '{print $3}') + # "level" is the actual level within the btree + if [ "${bt_level}" -eq "$((bt_height - 1))" ]; then + echo "${path}" + return 0 + fi + done + + return 1 +} + _require_xfs_mkfs_atomicswap() { # atomicswap can be activated on rmap or reflink filesystems. diff --git a/tests/xfs/406 b/tests/xfs/406 index 78db18077c..8c5570886b 100755 --- a/tests/xfs/406 +++ b/tests/xfs/406 @@ -26,10 +26,12 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 -inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'sb 0' 'addr rrmapino') +path="$(_scratch_xfs_find_rgbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rtrmapbt" +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "path -m $path") echo "Fuzz rtrmapbt recs" -_scratch_xfs_fuzz_metadata '' 'offline' 'sb 0' 'addr rrmapino' "addr u${inode_ver}.rtrmapbt.ptrs[1]" >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'offline' "path -m $path" "addr u${inode_ver}.rtrmapbt.ptrs[1]" >> $seqres.full echo "Done fuzzing rtrmapbt recs" # success, all done diff --git a/tests/xfs/407 b/tests/xfs/407 index 5a43775b55..2460ea336c 100755 --- a/tests/xfs/407 +++ b/tests/xfs/407 @@ -26,10 +26,12 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 -inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'sb 0' 'addr rrmapino') +path="$(_scratch_xfs_find_rgbtree_height 'rmap' 1)" || \ + _fail "could not find two-level rtrmapbt" +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "path -m $path") echo "Fuzz rtrmapbt recs" -_scratch_xfs_fuzz_metadata '' 'online' 'sb 0' 'addr rrmapino' "addr u${inode_ver}.rtrmapbt.ptrs[1]" >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'online' "path -m $path" "addr u${inode_ver}.rtrmapbt.ptrs[1]" >> $seqres.full echo "Done fuzzing rtrmapbt recs" # success, all done diff --git a/tests/xfs/408 b/tests/xfs/408 index 8049d6bead..3bed3824e8 100755 --- a/tests/xfs/408 +++ b/tests/xfs/408 @@ -4,7 +4,7 @@ # # FS QA Test No. 408 # -# Populate a XFS filesystem and fuzz every rtrmapbt keyptr field. +# Populate a XFS filesystem and fuzz every rtrmapbt key/pointer field. # Use xfs_repair to fix the corruption. # . ./common/preamble @@ -26,8 +26,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_rgbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rtrmapbt" + echo "Fuzz rtrmapbt keyptrs" -_scratch_xfs_fuzz_metadata '(rtrmapbt)' 'offline' 'sb 0' 'addr rrmapino' >> $seqres.full +_scratch_xfs_fuzz_metadata '(rtrmapbt)' 'offline' "path -m $path" >> $seqres.full echo "Done fuzzing rtrmapbt keyptrs" # success, all done diff --git a/tests/xfs/409 b/tests/xfs/409 index adac95fea8..ce66175c6e 100755 --- a/tests/xfs/409 +++ b/tests/xfs/409 @@ -4,7 +4,7 @@ # # FS QA Test No. 409 # -# Populate a XFS filesystem and fuzz every rtrmapbt keyptr field. +# Populate a XFS filesystem and fuzz every rtrmapbt key/pointer field. # Use xfs_scrub to fix the corruption. # . ./common/preamble @@ -26,8 +26,11 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_rgbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rtrmapbt" + echo "Fuzz rtrmapbt keyptrs" -_scratch_xfs_fuzz_metadata '(rtrmapbt)' 'offline' 'sb 0' 'addr rrmapino' >> $seqres.full +_scratch_xfs_fuzz_metadata '(rtrmapbt)' 'online' "path -m $path" >> $seqres.full echo "Done fuzzing rtrmapbt keyptrs" # success, all done diff --git a/tests/xfs/481 b/tests/xfs/481 index 48c7580ccb..d303f2c27d 100755 --- a/tests/xfs/481 +++ b/tests/xfs/481 @@ -27,10 +27,12 @@ _disable_dmesg_check echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 -inode_ver=$(_scratch_xfs_get_metadata_field "core.version" 'sb 0' 'addr rrmapino') +path="$(_scratch_xfs_find_rgbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rtrmapbt" +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "path -m $path") echo "Fuzz rtrmapbt recs" -_scratch_xfs_fuzz_metadata '' 'none' 'sb 0' 'addr rrmapino' "addr u${inode_ver}.rtrmapbt.ptrs[1]" >> $seqres.full +_scratch_xfs_fuzz_metadata '' 'none' "path -m $path" "addr u${inode_ver}.rtrmapbt.ptrs[1]" >> $seqres.full echo "Done fuzzing rtrmapbt recs" # success, all done diff --git a/tests/xfs/482 b/tests/xfs/482 index 0192b94cc0..32a3012154 100755 --- a/tests/xfs/482 +++ b/tests/xfs/482 @@ -4,7 +4,7 @@ # # FS QA Test No. 482 # -# Populate a XFS filesystem and fuzz every rtrmapbt keyptr field. +# Populate a XFS filesystem and fuzz every rtrmapbt key/pointer field. # Do not fix the filesystem, to test metadata verifiers. . ./common/preamble @@ -27,8 +27,11 @@ _disable_dmesg_check echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 +path="$(_scratch_xfs_find_rgbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rtrmapbt" + echo "Fuzz rtrmapbt keyptrs" -_scratch_xfs_fuzz_metadata '(rtrmapbt)' 'offline' 'sb 0' 'addr rrmapino' >> $seqres.full +_scratch_xfs_fuzz_metadata '(rtrmapbt)' 'offline' "path -m $path" >> $seqres.full echo "Done fuzzing rtrmapbt keyptrs" # success, all done ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 09/13] xfs: skip tests if formatting small filesystem fails 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (6 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 02/13] fuzz: for fuzzing the rtrmapbt, find the path to the rt rmap btree file Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 10/13] xfs/443: use file allocation unit, not dbsize Darrick J. Wong ` (4 subsequent siblings) 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> There are a few tests that try to exercise XFS functionality with an unusually small (< 500MB) filesystem. Formatting can fail if the test configuration also specifies a very large realtime device because mkfs hits ENOSPC when allocating the realtime metadata. The test proceeds anyway (which causes an immediate mount failure) so we might as well skip these. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/104 | 1 + tests/xfs/291 | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/xfs/104 b/tests/xfs/104 index d16f46d8e4..c3d1d18a58 100755 --- a/tests/xfs/104 +++ b/tests/xfs/104 @@ -16,6 +16,7 @@ _create_scratch() { echo "*** mkfs" _scratch_mkfs_xfs $@ | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs + test "${PIPESTATUS[0]}" -eq 0 || _notrun "formatting small scratch fs failed" . $tmp.mkfs echo "*** mount" diff --git a/tests/xfs/291 b/tests/xfs/291 index 600dcb2eba..70e5f51cee 100755 --- a/tests/xfs/291 +++ b/tests/xfs/291 @@ -18,7 +18,8 @@ _require_command "$XFS_MDRESTORE_PROG" "xfs_mdrestore" # real QA test starts here _require_scratch logblks=$(_scratch_find_xfs_min_logblocks -n size=16k -d size=133m) -_scratch_mkfs_xfs -n size=16k -l size=${logblks}b -d size=133m >> $seqres.full 2>&1 +_scratch_mkfs_xfs -n size=16k -l size=${logblks}b -d size=133m >> $seqres.full 2>&1 || \ + _notrun "formatting small scratch fs failed" _scratch_mount # First we cause very badly fragmented freespace, then ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 10/13] xfs/443: use file allocation unit, not dbsize 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (7 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 09/13] xfs: skip tests if formatting small filesystem fails Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 12/13] populate: check that we created a realtime rmap btree of the given height Darrick J. Wong ` (3 subsequent siblings) 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> We can only punch in units of file allocation boundaries, so update this test to use that instead of the fs blocksize. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/443 | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/tests/xfs/443 b/tests/xfs/443 index 56828decae..ab3cda59f3 100755 --- a/tests/xfs/443 +++ b/tests/xfs/443 @@ -40,14 +40,15 @@ _scratch_mount file1=$SCRATCH_MNT/file1 file2=$SCRATCH_MNT/file2 +file_blksz=$(_get_file_block_size $SCRATCH_MNT) # The goal is run an extent swap where one of the associated files has the # minimum number of extents to remain in btree format. First, create a couple # files with large enough extent counts (200 or so should be plenty) to ensure # btree format on the largest possible inode size filesystems. -$XFS_IO_PROG -fc "falloc 0 $((400 * dbsize))" $file1 +$XFS_IO_PROG -fc "falloc 0 $((400 * file_blksz))" $file1 $here/src/punch-alternating $file1 -$XFS_IO_PROG -fc "falloc 0 $((400 * dbsize))" $file2 +$XFS_IO_PROG -fc "falloc 0 $((400 * file_blksz))" $file2 $here/src/punch-alternating $file2 # Now run an extent swap at every possible extent count down to 0. Depending on @@ -55,12 +56,12 @@ $here/src/punch-alternating $file2 # btree format. for i in $(seq 1 2 399); do # punch one extent from the tmpfile and swap - $XFS_IO_PROG -c "fpunch $((i * dbsize)) $dbsize" $file2 + $XFS_IO_PROG -c "fpunch $((i * file_blksz)) $file_blksz" $file2 $XFS_IO_PROG -c "swapext $file2" $file1 # punch the same extent from the old fork (now in file2) to resync the # extent counts and repeat - $XFS_IO_PROG -c "fpunch $((i * dbsize)) $dbsize" $file2 + $XFS_IO_PROG -c "fpunch $((i * file_blksz)) $file_blksz" $file2 done # sanity check that no extents are left over ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 12/13] populate: check that we created a realtime rmap btree of the given height 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (8 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 10/13] xfs/443: use file allocation unit, not dbsize Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 13/13] fuzzy: create missing fuzz tests for rt rmap btrees Darrick J. Wong ` (2 subsequent siblings) 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Make sure that we actually create an rt rmap btree of the desired height somewhere in the filesystem. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/populate | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/common/populate b/common/populate index 7d57cd1287..6a05177e6d 100644 --- a/common/populate +++ b/common/populate @@ -631,6 +631,37 @@ __populate_check_xfs_agbtree_height() { return 1 } +# Check that there's at least one rt btree with multiple levels +__populate_check_xfs_rgbtree_height() { + local bt_type="$1" + local rgcount=$(_scratch_xfs_db -c 'sb 0' -c 'p rgcount' | awk '{print $3}') + local path + local path_format + local bt_prefix + + case "${bt_type}" in + "rmap") + path_format="/realtime/%u.rmap" + bt_prefix="u3.rtrmapbt" + ;; + *) + _fail "Don't know about rt btree ${bt_type}" + ;; + esac + + for ((rgno = 0; rgno < rgcount; rgno++)); do + path="$(printf "${path_format}" "${rgno}")" + bt_level=$(_scratch_xfs_db -c "path -m ${path}" -c "p ${bt_prefix}.level" | awk '{print $3}') + # "level" is the actual level within the btree + if [ "${bt_level}" -gt 0 ]; then + return 0 + fi + done + + __populate_fail "Failed to create rt ${bt_type} of sufficient height!" + return 1 +} + # Check that populate created all the types of files we wanted _scratch_xfs_populate_check() { _scratch_mount @@ -654,6 +685,7 @@ _scratch_xfs_populate_check() { is_finobt=$(_xfs_has_feature "$SCRATCH_MNT" finobt -v) is_rmapbt=$(_xfs_has_feature "$SCRATCH_MNT" rmapbt -v) is_reflink=$(_xfs_has_feature "$SCRATCH_MNT" reflink -v) + is_rt="$(_xfs_get_rtextents "$SCRATCH_MNT")" blksz="$(stat -f -c '%s' "${SCRATCH_MNT}")" dblksz="$(_xfs_get_dir_blocksize "$SCRATCH_MNT")" @@ -684,6 +716,8 @@ _scratch_xfs_populate_check() { test $is_finobt -ne 0 && __populate_check_xfs_agbtree_height "fino" test $is_rmapbt -ne 0 && __populate_check_xfs_agbtree_height "rmap" test $is_reflink -ne 0 && __populate_check_xfs_agbtree_height "refcnt" + test $is_rmapbt -ne 0 && test $is_rt -gt 0 && \ + __populate_check_xfs_rgbtree_height "rmap" } # Check data fork format of ext4 file ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 13/13] fuzzy: create missing fuzz tests for rt rmap btrees 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (9 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 12/13] populate: check that we created a realtime rmap btree of the given height Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 11/13] populate: adjust rtrmap calculations for rtgroups Darrick J. Wong 2022-12-30 22:20 ` [PATCH 08/13] xfs/3{43,32}: adapt tests for rt extent size greater than 1 Darrick J. Wong 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Back when I first created the fuzz tests for the realtime rmap btree, I forgot a couple of things. Add tests to fuzz rtrmap btree leaf records, and node keys. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/1528 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/1528.out | 4 ++++ tests/xfs/1529 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1529.out | 4 ++++ tests/xfs/407 | 2 +- 5 files changed, 90 insertions(+), 1 deletion(-) create mode 100755 tests/xfs/1528 create mode 100644 tests/xfs/1528.out create mode 100755 tests/xfs/1529 create mode 100644 tests/xfs/1529.out diff --git a/tests/xfs/1528 b/tests/xfs/1528 new file mode 100755 index 0000000000..b2e1193ebd --- /dev/null +++ b/tests/xfs/1528 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1528 +# +# Populate a XFS filesystem and fuzz every rtrmapbt record field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_xfs_scratch_rmapbt +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_rgbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rtrmapbt" +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "path -m $path") + +echo "Fuzz rtrmapbt recs" +_scratch_xfs_fuzz_metadata '' 'both' "path -m $path" "addr u${inode_ver}.rtrmapbt.ptrs[1]" >> $seqres.full +echo "Done fuzzing rtrmapbt recs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1528.out b/tests/xfs/1528.out new file mode 100644 index 0000000000..b51b640c40 --- /dev/null +++ b/tests/xfs/1528.out @@ -0,0 +1,4 @@ +QA output created by 1528 +Format and populate +Fuzz rtrmapbt recs +Done fuzzing rtrmapbt recs diff --git a/tests/xfs/1529 b/tests/xfs/1529 new file mode 100755 index 0000000000..91a673c049 --- /dev/null +++ b/tests/xfs/1529 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1529 +# +# Populate a XFS filesystem and fuzz every rtrmapbt keyptr field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_xfs_scratch_rmapbt +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_rgbtree_height 'rmap' 2)" || \ + _fail "could not find two-level rtrmapbt" + +echo "Fuzz rtrmapbt keyptrs" +_scratch_xfs_fuzz_metadata '(rtrmapbt)' 'offline' "path -m $path" >> $seqres.full +echo "Done fuzzing rtrmapbt keyptrs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1529.out b/tests/xfs/1529.out new file mode 100644 index 0000000000..808fcc957f --- /dev/null +++ b/tests/xfs/1529.out @@ -0,0 +1,4 @@ +QA output created by 1529 +Format and populate +Fuzz rtrmapbt keyptrs +Done fuzzing rtrmapbt keyptrs diff --git a/tests/xfs/407 b/tests/xfs/407 index 2460ea336c..bd439105e2 100755 --- a/tests/xfs/407 +++ b/tests/xfs/407 @@ -26,7 +26,7 @@ _require_scratch_xfs_fuzz_fields echo "Format and populate" _scratch_populate_cached nofill > $seqres.full 2>&1 -path="$(_scratch_xfs_find_rgbtree_height 'rmap' 1)" || \ +path="$(_scratch_xfs_find_rgbtree_height 'rmap' 2)" || \ _fail "could not find two-level rtrmapbt" inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "path -m $path") ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 11/13] populate: adjust rtrmap calculations for rtgroups 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (10 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 13/13] fuzzy: create missing fuzz tests for rt rmap btrees Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 08/13] xfs/3{43,32}: adapt tests for rt extent size greater than 1 Darrick J. Wong 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Now that we've sharded the realtime volume and created per-group rmap btrees, we need to adjust downward the size of rtrmapbt records since the block counts are now 32-bit instead of 64-bit. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/populate | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/common/populate b/common/populate index c0bbbc3f3b..7d57cd1287 100644 --- a/common/populate +++ b/common/populate @@ -379,7 +379,7 @@ _scratch_xfs_populate() { is_rt="$(_xfs_get_rtextents "$SCRATCH_MNT")" if [ $is_rmapbt -gt 0 ] && [ $is_rt -gt 0 ]; then echo "+ rtrmapbt btree" - nr="$((blksz * 2 / 32))" + nr="$((blksz * 2 / 24))" $XFS_IO_PROG -R -f -c 'truncate 0' "${SCRATCH_MNT}/RTRMAPBT" __populate_create_file $((blksz * nr)) "${SCRATCH_MNT}/RTRMAPBT" fi ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 08/13] xfs/3{43,32}: adapt tests for rt extent size greater than 1 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong ` (11 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 11/13] populate: adjust rtrmap calculations for rtgroups Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 12 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Both of these tests for the realtime volume can fail when the rt extent size is larger than a single block. 332 is a read-write functionality test that encodes md5sum in the output, so we need to skip it if $blksz isn't congruent with the extent size, because the fcollapse call will fail. 343 is a test of the rmap btree, so the fix here is simpler -- make $blksz the file allocation unit, and get rid of the md5sum in the golden output. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/332 | 6 +----- tests/xfs/332.out | 2 -- tests/xfs/343 | 2 ++ 3 files changed, 3 insertions(+), 7 deletions(-) diff --git a/tests/xfs/332 b/tests/xfs/332 index a2d37ee905..c1ac87adcb 100755 --- a/tests/xfs/332 +++ b/tests/xfs/332 @@ -28,7 +28,7 @@ rm -f "$seqres.full" echo "Format and mount" _scratch_mkfs > "$seqres.full" 2>&1 _scratch_mount -blksz=65536 +blksz=$(_get_file_block_size $SCRATCH_MNT) # 65536 blocks=16 len=$((blocks * blksz)) @@ -45,10 +45,6 @@ $XFS_IO_PROG -c "fpunch $blksz $blksz" \ -c "fcollapse $((9 * blksz)) $blksz" \ -c "finsert $((10 * blksz)) $blksz" $SCRATCH_MNT/f1 >> $seqres.full -echo "Check file" -md5sum $SCRATCH_MNT/f1 | _filter_scratch -od -tx1 -Ad -c $SCRATCH_MNT/f1 >> $seqres.full - echo "Unmount" _scratch_unmount diff --git a/tests/xfs/332.out b/tests/xfs/332.out index 9beff7cc37..3a7ca95b40 100644 --- a/tests/xfs/332.out +++ b/tests/xfs/332.out @@ -2,8 +2,6 @@ QA output created by 332 Format and mount Create some files Manipulate file -Check file -e45c5707fcf6817e914ffb6ce37a0ac7 SCRATCH_MNT/f1 Unmount Try a regular fsmap Try a bad fsmap diff --git a/tests/xfs/343 b/tests/xfs/343 index bffcc7d9ac..fe461847ed 100755 --- a/tests/xfs/343 +++ b/tests/xfs/343 @@ -31,6 +31,8 @@ blksz=65536 blocks=16 len=$((blocks * blksz)) +_require_congruent_file_oplen $SCRATCH_MNT $blksz + echo "Create some files" $XFS_IO_PROG -f -R -c "falloc 0 $len" -c "pwrite -S 0x68 -b 1048576 0 $len" $SCRATCH_MNT/f1 >> $seqres.full ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCHSET v1.0 00/10] fstests: reflink on the realtime device 2022-12-30 21:14 [NYE DELUGE 3/4] xfs: modernize the realtime volume Darrick J. Wong ` (4 preceding siblings ...) 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 01/10] xfs/122: update fields for realtime reflink Darrick J. Wong ` (9 more replies) 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: reflink with large realtime extents Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v1.0 0/1] fstests: functional tests for rt quota Darrick J. Wong 7 siblings, 10 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, This patchset enables use of the file data block sharing feature (i.e. reflink) on the realtime device. It follows the same basic sequence as the realtime rmap series -- first a few cleanups; then widening of the API parameters; and introduction of the new btree format and inode fork format. Next comes enabling CoW and remapping for the rt device; new scrub, repair, and health reporting code; and at the end we implement some code to lengthen write requests so that rt extents are always CoWed fully. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-reflink xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=realtime-reflink fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-reflink xfsdocs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-documentation.git/log/?h=realtime-reflink --- common/populate | 26 ++++++++++++++++++--- common/xfs | 15 ++++++++++++ tests/generic/331 | 12 ++++++++- tests/generic/331.out | 2 +- tests/xfs/122.out | 3 ++ tests/xfs/131 | 48 -------------------------------------- tests/xfs/131.out | 5 ---- tests/xfs/1538 | 41 ++++++++++++++++++++++++++++++++ tests/xfs/1538.out | 4 +++ tests/xfs/1539 | 41 ++++++++++++++++++++++++++++++++ tests/xfs/1539.out | 4 +++ tests/xfs/1540 | 41 ++++++++++++++++++++++++++++++++ tests/xfs/1540.out | 4 +++ tests/xfs/1541 | 42 +++++++++++++++++++++++++++++++++ tests/xfs/1541.out | 4 +++ tests/xfs/1542 | 41 ++++++++++++++++++++++++++++++++ tests/xfs/1542.out | 4 +++ tests/xfs/1543 | 40 ++++++++++++++++++++++++++++++++ tests/xfs/1543.out | 4 +++ tests/xfs/1544 | 40 ++++++++++++++++++++++++++++++++ tests/xfs/1544.out | 4 +++ tests/xfs/1545 | 41 ++++++++++++++++++++++++++++++++ tests/xfs/1545.out | 4 +++ tests/xfs/240 | 13 +++++++++- tests/xfs/240.out | 2 +- tests/xfs/243 | 5 ++++ tests/xfs/272 | 40 +++++++++++++++++++++----------- tests/xfs/274 | 62 ++++++++++++++++++++++++++++++++++--------------- tests/xfs/769 | 3 ++ tests/xfs/818 | 43 ++++++++++++++++++++++++++++++++++ tests/xfs/818.out | 2 ++ tests/xfs/819 | 43 ++++++++++++++++++++++++++++++++++ tests/xfs/819.out | 2 ++ 33 files changed, 590 insertions(+), 95 deletions(-) delete mode 100755 tests/xfs/131 delete mode 100644 tests/xfs/131.out create mode 100755 tests/xfs/1538 create mode 100644 tests/xfs/1538.out create mode 100755 tests/xfs/1539 create mode 100644 tests/xfs/1539.out create mode 100755 tests/xfs/1540 create mode 100644 tests/xfs/1540.out create mode 100755 tests/xfs/1541 create mode 100644 tests/xfs/1541.out create mode 100755 tests/xfs/1542 create mode 100644 tests/xfs/1542.out create mode 100755 tests/xfs/1543 create mode 100644 tests/xfs/1543.out create mode 100755 tests/xfs/1544 create mode 100644 tests/xfs/1544.out create mode 100755 tests/xfs/1545 create mode 100644 tests/xfs/1545.out create mode 100755 tests/xfs/818 create mode 100644 tests/xfs/818.out create mode 100755 tests/xfs/819 create mode 100644 tests/xfs/819.out ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCH 01/10] xfs/122: update fields for realtime reflink 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 02/10] common/populate: create realtime refcount btree Darrick J. Wong ` (8 subsequent siblings) 9 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add a few more ondisk structures for realtime reflink. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/122.out | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/xfs/122.out b/tests/xfs/122.out index e0801f9660..3239a655f9 100644 --- a/tests/xfs/122.out +++ b/tests/xfs/122.out @@ -116,6 +116,9 @@ sizeof(struct xfs_rmap_key) = 20 sizeof(struct xfs_rmap_rec) = 24 sizeof(struct xfs_rtbuf_blkinfo) = 48 sizeof(struct xfs_rtgroup_geometry) = 128 +sizeof(struct xfs_rtrefcount_key) = 4 +sizeof(struct xfs_rtrefcount_rec) = 12 +sizeof(struct xfs_rtrefcount_root) = 4 sizeof(struct xfs_rtrmap_key) = 20 sizeof(struct xfs_rtrmap_rec) = 24 sizeof(struct xfs_rtrmap_root) = 4 ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 02/10] common/populate: create realtime refcount btree 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong 2022-12-30 22:20 ` [PATCH 01/10] xfs/122: update fields for realtime reflink Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 03/10] xfs: create fuzz tests for the " Darrick J. Wong ` (7 subsequent siblings) 9 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Populate a realtime refcount btree when we're creating a sample fs. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/populate | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/common/populate b/common/populate index 6a05177e6d..734e31345f 100644 --- a/common/populate +++ b/common/populate @@ -367,16 +367,30 @@ _scratch_xfs_populate() { rm -f "${dir}/${f}" done - # Reverse-mapping btree + is_rt="$(_xfs_get_rtextents "$SCRATCH_MNT")" is_rmapbt="$(_xfs_has_feature "$SCRATCH_MNT" rmapbt -v)" + is_reflink="$(_xfs_has_feature "$SCRATCH_MNT" reflink -v)" + + # Reverse-mapping btree if [ $is_rmapbt -gt 0 ]; then echo "+ rmapbt btree" nr="$((blksz * 2 / 24))" __populate_create_file $((blksz * nr)) "${SCRATCH_MNT}/RMAPBT" fi + # Realtime Reference-count btree comes before the rtrmapbt so that + # the refcount entries are created in rtgroup 0. + if [ $is_reflink -gt 0 ] && [ $is_rt -gt 0 ]; then + echo "+ rtreflink btree" + rt_blksz=$(_xfs_get_rtextsize "$SCRATCH_MNT") + nr="$((rt_blksz * 2 / 12))" + $XFS_IO_PROG -R -f -c 'truncate 0' "${SCRATCH_MNT}/RTREFCOUNTBT" + __populate_create_file $((blksz * nr)) "${SCRATCH_MNT}/RTREFCOUNTBT" + $XFS_IO_PROG -R -f -c 'truncate 0' "${SCRATCH_MNT}/RTREFCOUNTBT2" + cp --reflink=always "${SCRATCH_MNT}/RTREFCOUNTBT" "${SCRATCH_MNT}/RTREFCOUNTBT2" + fi + # Realtime Reverse-mapping btree - is_rt="$(_xfs_get_rtextents "$SCRATCH_MNT")" if [ $is_rmapbt -gt 0 ] && [ $is_rt -gt 0 ]; then echo "+ rtrmapbt btree" nr="$((blksz * 2 / 24))" @@ -385,7 +399,6 @@ _scratch_xfs_populate() { fi # Reference-count btree - is_reflink="$(_xfs_has_feature "$SCRATCH_MNT" reflink -v)" if [ $is_reflink -gt 0 ]; then echo "+ reflink btree" nr="$((blksz * 2 / 12))" @@ -403,6 +416,7 @@ _scratch_xfs_populate() { __populate_fragment_file "${SCRATCH_MNT}/RMAPBT" __populate_fragment_file "${SCRATCH_MNT}/RTRMAPBT" __populate_fragment_file "${SCRATCH_MNT}/REFCOUNTBT" + __populate_fragment_file "${SCRATCH_MNT}/RTREFCOUNTBT" umount "${SCRATCH_MNT}" } @@ -644,6 +658,10 @@ __populate_check_xfs_rgbtree_height() { path_format="/realtime/%u.rmap" bt_prefix="u3.rtrmapbt" ;; + "refcnt") + path_format="/realtime/%u.refcount" + bt_prefix="u3.rtrefcbt" + ;; *) _fail "Don't know about rt btree ${bt_type}" ;; @@ -718,6 +736,8 @@ _scratch_xfs_populate_check() { test $is_reflink -ne 0 && __populate_check_xfs_agbtree_height "refcnt" test $is_rmapbt -ne 0 && test $is_rt -gt 0 && \ __populate_check_xfs_rgbtree_height "rmap" + test $is_reflink -ne 0 && test $is_rt -gt 0 && \ + __populate_check_xfs_rgbtree_height "refcnt" } # Check data fork format of ext4 file ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 03/10] xfs: create fuzz tests for the realtime refcount btree 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong 2022-12-30 22:20 ` [PATCH 01/10] xfs/122: update fields for realtime reflink Darrick J. Wong 2022-12-30 22:20 ` [PATCH 02/10] common/populate: create realtime refcount btree Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 06/10] xfs: race fsstress with realtime refcount btree scrub and repair Darrick J. Wong ` (6 subsequent siblings) 9 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Create fuzz tests for the realtime refcount btree record and key/ptr blocks. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 4 ++++ tests/xfs/1538 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/1538.out | 4 ++++ tests/xfs/1539 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/1539.out | 4 ++++ tests/xfs/1540 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/1540.out | 4 ++++ tests/xfs/1541 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1541.out | 4 ++++ tests/xfs/1542 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/1542.out | 4 ++++ tests/xfs/1543 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1543.out | 4 ++++ tests/xfs/1544 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/xfs/1544.out | 4 ++++ tests/xfs/1545 | 41 +++++++++++++++++++++++++++++++++++++++++ tests/xfs/1545.out | 4 ++++ 17 files changed, 363 insertions(+) create mode 100755 tests/xfs/1538 create mode 100644 tests/xfs/1538.out create mode 100755 tests/xfs/1539 create mode 100644 tests/xfs/1539.out create mode 100755 tests/xfs/1540 create mode 100644 tests/xfs/1540.out create mode 100755 tests/xfs/1541 create mode 100644 tests/xfs/1541.out create mode 100755 tests/xfs/1542 create mode 100644 tests/xfs/1542.out create mode 100755 tests/xfs/1543 create mode 100644 tests/xfs/1543.out create mode 100755 tests/xfs/1544 create mode 100644 tests/xfs/1544.out create mode 100755 tests/xfs/1545 create mode 100644 tests/xfs/1545.out diff --git a/common/xfs b/common/xfs index 63eff39d47..7b7b3a35b5 100644 --- a/common/xfs +++ b/common/xfs @@ -1829,6 +1829,10 @@ _scratch_xfs_find_rgbtree_height() { path_format="/realtime/%u.rmap" bt_prefix="u3.rtrmapbt" ;; + "refcnt") + path_format="/realtime/%u.refcount" + bt_prefix="u3.rtrefcbt" + ;; *) _fail "Don't know about rt btree ${bt_type}" ;; diff --git a/tests/xfs/1538 b/tests/xfs/1538 new file mode 100755 index 0000000000..e62bf49b29 --- /dev/null +++ b/tests/xfs/1538 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1538 +# +# Populate a XFS filesystem and fuzz every rtrefcountbt record field. +# Use xfs_scrub to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_reflink +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_rgbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level rtrefcountbt" +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "path -m $path") + +echo "Fuzz rtrefcountbt recs" +_scratch_xfs_fuzz_metadata '' 'online' "path -m $path" "addr u${inode_ver}.rtrefcbt.ptrs[1]" >> $seqres.full +echo "Done fuzzing rtrefcountbt recs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1538.out b/tests/xfs/1538.out new file mode 100644 index 0000000000..968cfd6ef9 --- /dev/null +++ b/tests/xfs/1538.out @@ -0,0 +1,4 @@ +QA output created by 1538 +Format and populate +Fuzz rtrefcountbt recs +Done fuzzing rtrefcountbt recs diff --git a/tests/xfs/1539 b/tests/xfs/1539 new file mode 100755 index 0000000000..36cef96a91 --- /dev/null +++ b/tests/xfs/1539 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1539 +# +# Populate a XFS filesystem and fuzz every rtrefcountbt record field. +# Use xfs_repair to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_repair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_reflink +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_rgbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level rtrefcountbt" +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "path -m $path") + +echo "Fuzz rtrefcountbt recs" +_scratch_xfs_fuzz_metadata '' 'offline' "path -m $path" "addr u${inode_ver}.rtrefcbt.ptrs[1]" >> $seqres.full +echo "Done fuzzing rtrefcountbt recs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1539.out b/tests/xfs/1539.out new file mode 100644 index 0000000000..aa3a963dc2 --- /dev/null +++ b/tests/xfs/1539.out @@ -0,0 +1,4 @@ +QA output created by 1539 +Format and populate +Fuzz rtrefcountbt recs +Done fuzzing rtrefcountbt recs diff --git a/tests/xfs/1540 b/tests/xfs/1540 new file mode 100755 index 0000000000..fa08d3fb54 --- /dev/null +++ b/tests/xfs/1540 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1540 +# +# Populate a XFS filesystem and fuzz every rtrefcountbt record field. +# Do not fix the filesystem, to test metadata verifiers. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_norepair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_reflink +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_rgbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level rtrefcountbt" +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "path -m $path") + +echo "Fuzz rtrefcountbt recs" +_scratch_xfs_fuzz_metadata '' 'none' "path -m $path" "addr u${inode_ver}.rtrefcbt.ptrs[1]" >> $seqres.full +echo "Done fuzzing rtrefcountbt recs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1540.out b/tests/xfs/1540.out new file mode 100644 index 0000000000..37f3311837 --- /dev/null +++ b/tests/xfs/1540.out @@ -0,0 +1,4 @@ +QA output created by 1540 +Format and populate +Fuzz rtrefcountbt recs +Done fuzzing rtrefcountbt recs diff --git a/tests/xfs/1541 b/tests/xfs/1541 new file mode 100755 index 0000000000..ecf6fdc56c --- /dev/null +++ b/tests/xfs/1541 @@ -0,0 +1,42 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1541 +# +# Populate a XFS filesystem and fuzz every rtrefcountbt record field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_reflink +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_rgbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level rtrefcountbt" +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "path -m $path") + +echo "Fuzz rtrefcountbt recs" +_scratch_xfs_fuzz_metadata '' 'both' "path -m $path" "addr u${inode_ver}.rtrefcbt.ptrs[1]" >> $seqres.full +echo "Done fuzzing rtrefcountbt recs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1541.out b/tests/xfs/1541.out new file mode 100644 index 0000000000..35a9b73471 --- /dev/null +++ b/tests/xfs/1541.out @@ -0,0 +1,4 @@ +QA output created by 1541 +Format and populate +Fuzz rtrefcountbt recs +Done fuzzing rtrefcountbt recs diff --git a/tests/xfs/1542 b/tests/xfs/1542 new file mode 100755 index 0000000000..37ef8a2b4c --- /dev/null +++ b/tests/xfs/1542 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1542 +# +# Populate a XFS filesystem and fuzz every rtrefcountbt key/ptr field. +# Use xfs_scrub to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_reflink +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_rgbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level rtrefcountbt" +inode_ver=$(_scratch_xfs_get_metadata_field "core.version" "path -m $path") + +echo "Fuzz rtrefcountbt keyptrs" +_scratch_xfs_fuzz_metadata '(rtrefcbt)' 'online' "path -m $path" >> $seqres.full +echo "Done fuzzing rtrefcountbt keyptrs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1542.out b/tests/xfs/1542.out new file mode 100644 index 0000000000..55d820b4b1 --- /dev/null +++ b/tests/xfs/1542.out @@ -0,0 +1,4 @@ +QA output created by 1542 +Format and populate +Fuzz rtrefcountbt keyptrs +Done fuzzing rtrefcountbt keyptrs diff --git a/tests/xfs/1543 b/tests/xfs/1543 new file mode 100755 index 0000000000..0acd3203e3 --- /dev/null +++ b/tests/xfs/1543 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1543 +# +# Populate a XFS filesystem and fuzz every rtrefcountbt key/ptr field. +# Use xfs_repair to fix the corruption. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_repair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_reflink +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_rgbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level rtrefcountbt" + +echo "Fuzz rtrefcountbt keyptrs" +_scratch_xfs_fuzz_metadata '(rtrefcbt)' 'offline' "path -m $path" >> $seqres.full +echo "Done fuzzing rtrefcountbt keyptrs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1543.out b/tests/xfs/1543.out new file mode 100644 index 0000000000..e7afa10744 --- /dev/null +++ b/tests/xfs/1543.out @@ -0,0 +1,4 @@ +QA output created by 1543 +Format and populate +Fuzz rtrefcountbt keyptrs +Done fuzzing rtrefcountbt keyptrs diff --git a/tests/xfs/1544 b/tests/xfs/1544 new file mode 100755 index 0000000000..165f96f6a4 --- /dev/null +++ b/tests/xfs/1544 @@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1544 +# +# Populate a XFS filesystem and fuzz every rtrefcountbt key/ptr field. +# Do not fix the filesystem, to test metadata verifiers. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_norepair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_reflink +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_rgbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level rtrefcountbt" + +echo "Fuzz rtrefcountbt keyptrs" +_scratch_xfs_fuzz_metadata '(rtrefcbt)' 'none' "path -m $path" >> $seqres.full +echo "Done fuzzing rtrefcountbt keyptrs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1544.out b/tests/xfs/1544.out new file mode 100644 index 0000000000..b39532c160 --- /dev/null +++ b/tests/xfs/1544.out @@ -0,0 +1,4 @@ +QA output created by 1544 +Format and populate +Fuzz rtrefcountbt keyptrs +Done fuzzing rtrefcountbt keyptrs diff --git a/tests/xfs/1545 b/tests/xfs/1545 new file mode 100755 index 0000000000..a467662f2f --- /dev/null +++ b/tests/xfs/1545 @@ -0,0 +1,41 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 1545 +# +# Populate a XFS filesystem and fuzz every rtrefcountbt key/ptr field. +# Try online repair and, if necessary, offline repair, +# to test the most likely usage pattern. + +. ./common/preamble +_begin_fstest dangerous_fuzzers dangerous_bothrepair realtime + +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/populate +. ./common/fuzzy +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch_reflink +_require_scratch_xfs_fuzz_fields +_disable_dmesg_check + +echo "Format and populate" +_scratch_populate_cached nofill > $seqres.full 2>&1 + +path="$(_scratch_xfs_find_rgbtree_height 'refcnt' 2)" || \ + _fail "could not find two-level rtrefcountbt" + +echo "Fuzz rtrefcountbt keyptrs" +_scratch_xfs_fuzz_metadata '(rtrefcbt)' 'both' "path -m $path" >> $seqres.full +echo "Done fuzzing rtrefcountbt keyptrs" + +# success, all done +status=0 +exit diff --git a/tests/xfs/1545.out b/tests/xfs/1545.out new file mode 100644 index 0000000000..982a0d64df --- /dev/null +++ b/tests/xfs/1545.out @@ -0,0 +1,4 @@ +QA output created by 1545 +Format and populate +Fuzz rtrefcountbt keyptrs +Done fuzzing rtrefcountbt keyptrs ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 06/10] xfs: race fsstress with realtime refcount btree scrub and repair 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 03/10] xfs: create fuzz tests for the " Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 09/10] generic/331,xfs/240: support files that skip delayed allocation Darrick J. Wong ` (5 subsequent siblings) 9 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Race checking and rebuilding realtime refcount btrees with fsstress. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/818 | 43 +++++++++++++++++++++++++++++++++++++++++++ tests/xfs/818.out | 2 ++ tests/xfs/819 | 43 +++++++++++++++++++++++++++++++++++++++++++ tests/xfs/819.out | 2 ++ 4 files changed, 90 insertions(+) create mode 100755 tests/xfs/818 create mode 100644 tests/xfs/818.out create mode 100755 tests/xfs/819 create mode 100644 tests/xfs/819.out diff --git a/tests/xfs/818 b/tests/xfs/818 new file mode 100755 index 0000000000..aabe636750 --- /dev/null +++ b/tests/xfs/818 @@ -0,0 +1,43 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 818 +# +# Race fsstress and rt refcount btree scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest scrub dangerous_fsstress_scrub + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_scrub + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime +_require_xfs_has_feature "$SCRATCH_MNT" reflink +_xfs_force_bdev realtime $SCRATCH_MNT + +_scratch_xfs_stress_scrub -s "scrub rtrefcountbt %rgno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/818.out b/tests/xfs/818.out new file mode 100644 index 0000000000..cb0997862e --- /dev/null +++ b/tests/xfs/818.out @@ -0,0 +1,2 @@ +QA output created by 818 +Silence is golden diff --git a/tests/xfs/819 b/tests/xfs/819 new file mode 100755 index 0000000000..e302ed1fdc --- /dev/null +++ b/tests/xfs/819 @@ -0,0 +1,43 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2022 Oracle. Inc. All Rights Reserved. +# +# FS QA Test No. 819 +# +# Race fsstress and rt refcount btree scrub for a while to see if we crash or +# livelock. +# +. ./common/preamble +_begin_fstest online_repair dangerous_fsstress_repair + +_cleanup() { + _scratch_xfs_stress_scrub_cleanup &> /dev/null + cd / + rm -r -f $tmp.* +} +_register_cleanup "_cleanup" BUS + +# Import common functions. +. ./common/filter +. ./common/fuzzy +. ./common/inject +. ./common/xfs + +# real QA test starts here +_supported_fs xfs +_require_realtime +_require_scratch +_require_xfs_stress_online_repair + +_scratch_mkfs > "$seqres.full" 2>&1 +_scratch_mount +_require_xfs_has_feature "$SCRATCH_MNT" realtime +_require_xfs_has_feature "$SCRATCH_MNT" reflink +_xfs_force_bdev realtime $SCRATCH_MNT + +_scratch_xfs_stress_online_repair -s "repair rtrefcountbt %rgno%" + +# success, all done +echo Silence is golden +status=0 +exit diff --git a/tests/xfs/819.out b/tests/xfs/819.out new file mode 100644 index 0000000000..f5df7622a7 --- /dev/null +++ b/tests/xfs/819.out @@ -0,0 +1,2 @@ +QA output created by 819 +Silence is golden ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 09/10] generic/331,xfs/240: support files that skip delayed allocation 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong ` (3 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 06/10] xfs: race fsstress with realtime refcount btree scrub and repair Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 05/10] xfs/243: don't run when realtime storage is the default Darrick J. Wong ` (4 subsequent siblings) 9 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The goal of this test is to ensure that log recovery finishes a copy on write operation in the event of temporary media errors. It's important that the test observe some sort of IO error once we switch the scratch device to fail all IOs, but regrettably the test encoded the specific behavior of XFS and btrfs when the test was written -- the aio write to the page cache doesn't have to touch the disk and succeeds, and the fdatasync flushes things to disk and hits the IO error. However, this is not how things work on the XFS realtime device. There is no delalloc on realtime, so the aio write allocates an unwritten extent to stage the write. The allocation fails due to EIO, so it's the write call that fails. Therefore, all we need to do is to detect an IO error at any point between the write and the fdatasync call to be satisfied that the test does what we want to do. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/generic/331 | 12 ++++++++++-- tests/generic/331.out | 2 +- tests/xfs/240 | 13 +++++++++++-- tests/xfs/240.out | 2 +- 4 files changed, 23 insertions(+), 6 deletions(-) diff --git a/tests/generic/331 b/tests/generic/331 index 492abedf76..8c665ce4fc 100755 --- a/tests/generic/331 +++ b/tests/generic/331 @@ -59,9 +59,17 @@ echo "CoW and unmount" $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full sync + +# If the filesystem supports delalloc, then the fdatasync will report an IO +# error. If the write goes directly to disk, then aiocp will return nonzero. +unset write_failed _dmerror_load_error_table -$AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full -$XFS_IO_PROG -c "fdatasync" $testdir/file2 +$AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 &>> $seqres.full || \ + write_failed=1 +$XFS_IO_PROG -c "fdatasync" $testdir/file2 2>&1 | grep -q 'Input.output error' && \ + write_failed=1 +test -n $write_failed && echo "write failed" + _dmerror_load_working_table _dmerror_unmount _dmerror_mount diff --git a/tests/generic/331.out b/tests/generic/331.out index adbf841d00..d8ccea704b 100644 --- a/tests/generic/331.out +++ b/tests/generic/331.out @@ -5,7 +5,7 @@ Compare files 1886e67cf8783e89ce6ddc5bb09a3944 SCRATCH_MNT/test-331/file1 1886e67cf8783e89ce6ddc5bb09a3944 SCRATCH_MNT/test-331/file2 CoW and unmount -fdatasync: Input/output error +write failed Compare files 1886e67cf8783e89ce6ddc5bb09a3944 SCRATCH_MNT/test-331/file1 d94b0ab13385aba594411c174b1cc13c SCRATCH_MNT/test-331/file2 diff --git a/tests/xfs/240 b/tests/xfs/240 index a65c270d23..cabe309201 100755 --- a/tests/xfs/240 +++ b/tests/xfs/240 @@ -66,8 +66,17 @@ $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full sync _dmerror_load_error_table -$AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full -$XFS_IO_PROG -c "fdatasync" $testdir/file2 + +# If the filesystem supports delalloc, then the fdatasync will report an IO +# error. If the write goes directly to disk, then aiocp will return nonzero. +unset write_failed +_dmerror_load_error_table +$AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 &>> $seqres.full || \ + write_failed=1 +$XFS_IO_PROG -c "fdatasync" $testdir/file2 2>&1 | grep -q 'Input.output error' && \ + write_failed=1 +test -n $write_failed && echo "write failed" + _dmerror_load_working_table _dmerror_unmount _dmerror_mount diff --git a/tests/xfs/240.out b/tests/xfs/240.out index 1a22e8a389..00bb116e5c 100644 --- a/tests/xfs/240.out +++ b/tests/xfs/240.out @@ -5,7 +5,7 @@ Compare files 1886e67cf8783e89ce6ddc5bb09a3944 SCRATCH_MNT/test-240/file1 1886e67cf8783e89ce6ddc5bb09a3944 SCRATCH_MNT/test-240/file2 CoW and unmount -fdatasync: Input/output error +write failed Compare files 1886e67cf8783e89ce6ddc5bb09a3944 SCRATCH_MNT/test-240/file1 d94b0ab13385aba594411c174b1cc13c SCRATCH_MNT/test-240/file2 ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 05/10] xfs/243: don't run when realtime storage is the default 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong ` (4 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 09/10] generic/331,xfs/240: support files that skip delayed allocation Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 04/10] xfs/27[24]: adapt for checking files on the realtime volume Darrick J. Wong ` (3 subsequent siblings) 9 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Realtime volumes don't support delayed allocation, so don't run this test when the mkfs configuration specifies realtime creation by default. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/243 | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/xfs/243 b/tests/xfs/243 index 514fa35667..dda4a0c223 100755 --- a/tests/xfs/243 +++ b/tests/xfs/243 @@ -38,6 +38,11 @@ echo "Format and mount" _scratch_mkfs > $seqres.full 2>&1 _scratch_mount >> $seqres.full 2>&1 +# XFS does not support delayed allocation on realtime volumes (even for COW) +# so skip this test on those platforms. +$XFS_IO_PROG -c 'stat -v' $SCRATCH_MNT | grep -q "xflags.*rt-inherit" && + _notrun "delalloc not used for CoW on realtime device" + testdir=$SCRATCH_MNT/test-$seq mkdir $testdir ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 04/10] xfs/27[24]: adapt for checking files on the realtime volume 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong ` (5 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 05/10] xfs/243: don't run when realtime storage is the default Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 08/10] xfs/769: add rtreflink upgrade to test matrix Darrick J. Wong ` (2 subsequent siblings) 9 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Adapt both tests to behave properly if the two files being tested are on the realtime volume. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/272 | 40 +++++++++++++++++++++++++------------ tests/xfs/274 | 62 ++++++++++++++++++++++++++++++++++++++++----------------- 2 files changed, 70 insertions(+), 32 deletions(-) diff --git a/tests/xfs/272 b/tests/xfs/272 index 42b4a2edb5..2d7fc57d55 100755 --- a/tests/xfs/272 +++ b/tests/xfs/272 @@ -40,26 +40,40 @@ $here/src/punch-alternating $SCRATCH_MNT/urk >> $seqres.full ino=$(stat -c '%i' $SCRATCH_MNT/urk) echo "Get fsmap" | tee -a $seqres.full -$XFS_IO_PROG -c 'fsmap -v' $SCRATCH_MNT >> $seqres.full $XFS_IO_PROG -c 'fsmap -v' $SCRATCH_MNT | tr '[]()' ' ' > $TEST_DIR/fsmap +cat $TEST_DIR/fsmap >> $seqres.full echo "Get bmap" | tee -a $seqres.full -$XFS_IO_PROG -c 'bmap -v' $SCRATCH_MNT/urk >> $seqres.full $XFS_IO_PROG -c 'bmap -v' $SCRATCH_MNT/urk | grep '^[[:space:]]*[0-9]*:' | grep -v 'hole' | tr '[]()' ' ' > $TEST_DIR/bmap +cat $TEST_DIR/bmap >> $seqres.full echo "Check bmap and fsmap" | tee -a $seqres.full -cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total crap; do - qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total}$" - echo "${qstr}" >> $seqres.full - grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full - found=$(grep -c "${qstr}" $TEST_DIR/fsmap) - test $found -eq 1 || echo "Unexpected output for offset ${offrange}." -done +if $XFS_IO_PROG -c 'stat -v' $SCRATCH_MNT/urk | grep -q realtime; then + # file on rt volume + cat $TEST_DIR/bmap | while read ext offrange colon rtblockrange total crap; do + qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${rtblockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${total}$" + echo "${qstr}" >> $seqres.full + grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full + found=$(grep -c "${qstr}" $TEST_DIR/fsmap) + test $found -eq 1 || echo "Unexpected output for offset ${offrange}." + done -echo "Check device field of FS metadata and regular file" -data_dev=$(grep 'inode btree' $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') -rt_dev=$(grep "${ino}[[:space:]]*[0-9]*\.\.[0-9]*" $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') -test "${data_dev}" = "${rt_dev}" || echo "data ${data_dev} realtime ${rt_dev}?" + echo "Check device field of FS metadata and regular file" +else + # file on data volume + cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total crap; do + qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total}$" + echo "${qstr}" >> $seqres.full + grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full + found=$(grep -c "${qstr}" $TEST_DIR/fsmap) + test $found -eq 1 || echo "Unexpected output for offset ${offrange}." + done + + echo "Check device field of FS metadata and regular file" + data_dev=$(grep 'inode btree' $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') + rt_dev=$(grep "${ino}[[:space:]]*[0-9]*\.\.[0-9]*" $TEST_DIR/fsmap | head -n 1 | awk '{print $2}') + test "${data_dev}" = "${rt_dev}" || echo "data ${data_dev} realtime ${rt_dev}?" +fi # success, all done status=0 diff --git a/tests/xfs/274 b/tests/xfs/274 index dcaea68804..25dd0c3f74 100755 --- a/tests/xfs/274 +++ b/tests/xfs/274 @@ -40,34 +40,58 @@ _cp_reflink $SCRATCH_MNT/f1 $SCRATCH_MNT/f2 ino=$(stat -c '%i' $SCRATCH_MNT/f1) echo "Get fsmap" | tee -a $seqres.full -$XFS_IO_PROG -c 'fsmap -v' $SCRATCH_MNT >> $seqres.full $XFS_IO_PROG -c 'fsmap -v' $SCRATCH_MNT | tr '[]()' ' ' > $TEST_DIR/fsmap +cat $TEST_DIR/fsmap >> $seqres.full echo "Get f1 bmap" | tee -a $seqres.full -$XFS_IO_PROG -c 'bmap -v' $SCRATCH_MNT/f1 >> $seqres.full $XFS_IO_PROG -c 'bmap -v' $SCRATCH_MNT/f1 | grep '^[[:space:]]*[0-9]*:' | grep -v 'hole' | tr '[]()' ' ' > $TEST_DIR/bmap +cat $TEST_DIR/bmap >> $seqres.full -echo "Check f1 bmap and fsmap" | tee -a $seqres.full -cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total crap; do - qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total} 0100000$" - echo "${qstr}" >> $seqres.full - grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full - found=$(grep -c "${qstr}" $TEST_DIR/fsmap) - test $found -eq 1 || echo "Unexpected output for offset ${offrange}." -done +if _xfs_is_realtime_file $SCRATCH_MNT/f1 && ! _xfs_has_feature $SCRATCH_MNT rtgroups; then + # file on rt volume + echo "Check f1 bmap and fsmap" | tee -a $seqres.full + cat $TEST_DIR/bmap | while read ext offrange colon rtblockrange total crap; do + qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${rtblockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${total} 0100000$" + echo "${qstr}" >> $seqres.full + grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full + found=$(grep -c "${qstr}" $TEST_DIR/fsmap) + test $found -eq 1 || echo "Unexpected output for offset ${offrange}." + done +else + # file on data volume + echo "Check f1 bmap and fsmap" | tee -a $seqres.full + cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total crap; do + qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total} 0100000$" + echo "${qstr}" >> $seqres.full + grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full + found=$(grep -c "${qstr}" $TEST_DIR/fsmap) + test $found -eq 1 || echo "Unexpected output for offset ${offrange}." + done +fi echo "Get f2 bmap" | tee -a $seqres.full -$XFS_IO_PROG -c 'bmap -v' $SCRATCH_MNT/f2 >> $seqres.full $XFS_IO_PROG -c 'bmap -v' $SCRATCH_MNT/f2 | grep '^[[:space:]]*[0-9]*:' | grep -v 'hole' | tr '[]()' ' ' > $TEST_DIR/bmap +cat $TEST_DIR/bmap >> $seqres.full -echo "Check f2 bmap and fsmap" | tee -a $seqres.full -cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total crap; do - qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total} 0100000$" - echo "${qstr}" >> $seqres.full - grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full - found=$(grep -c "${qstr}" $TEST_DIR/fsmap) - test $found -eq 1 || echo "Unexpected output for offset ${offrange}." -done +if _xfs_is_realtime_file $SCRATCH_MNT/f2 && ! _xfs_has_feature $SCRATCH_MNT rtgroups; then + echo "Check f2 bmap and fsmap" | tee -a $seqres.full + cat $TEST_DIR/bmap | while read ext offrange colon rtblockrange total crap; do + qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${rtblockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${total} 0100000$" + echo "${qstr}" >> $seqres.full + grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full + found=$(grep -c "${qstr}" $TEST_DIR/fsmap) + test $found -eq 1 || echo "Unexpected output for offset ${offrange}." + done +else + echo "Check f2 bmap and fsmap" | tee -a $seqres.full + cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total crap; do + qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total} 0100000$" + echo "${qstr}" >> $seqres.full + grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full + found=$(grep -c "${qstr}" $TEST_DIR/fsmap) + test $found -eq 1 || echo "Unexpected output for offset ${offrange}." + done +fi # success, all done status=0 ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 08/10] xfs/769: add rtreflink upgrade to test matrix 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong ` (6 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 04/10] xfs/27[24]: adapt for checking files on the realtime volume Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 10/10] common/xfs: fix _xfs_get_file_block_size when rtinherit is set and no rt section Darrick J. Wong 2022-12-30 22:20 ` [PATCH 07/10] xfs: remove xfs/131 now that we allow reflink on realtime volumes Darrick J. Wong 9 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Add realtime reflink to the features that this test will try to upgrade. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/769 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/xfs/769 b/tests/xfs/769 index ccc3ea10bc..72863a6e83 100755 --- a/tests/xfs/769 +++ b/tests/xfs/769 @@ -196,12 +196,13 @@ function post_exercise() # upgrade don't spread failure to the rest of the tests. FEATURES=() if rt_configured; then - # rmap wasn't added to rt devices until after metadir + # reflink & rmap weren't added to rt devices until after metadir check_repair_upgrade finobt && FEATURES+=("finobt") check_repair_upgrade inobtcount && FEATURES+=("inobtcount") check_repair_upgrade bigtime && FEATURES+=("bigtime") check_repair_upgrade metadir && FEATURES+=("metadir") check_repair_upgrade rmapbt && FEATURES+=("rmapbt") + check_repair_upgrade reflink && FEATURES+=("reflink") else check_repair_upgrade finobt && FEATURES+=("finobt") check_repair_upgrade rmapbt && FEATURES+=("rmapbt") ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 10/10] common/xfs: fix _xfs_get_file_block_size when rtinherit is set and no rt section 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong ` (7 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 08/10] xfs/769: add rtreflink upgrade to test matrix Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 07/10] xfs: remove xfs/131 now that we allow reflink on realtime volumes Darrick J. Wong 9 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> It's possible for the sysadmin to set rtinherit on the directory tree even if there isn't a realtime section attached to the filesystem. When this is the case, the realtime flag is /not/ passed to new files, and file data is written to the data device. The file allocation unit for the file is the fs blocksize, and it is not correct to use the rt extent. fstests can be fooled into doing the incorrect thing if test runner puts '-d rtinherit=1 -r extsize=28k' into MKFS_OPTIONS without configuring a realtime device. This causes many tests to do the wrong thing because they think they must operate on units of 28k (and not 4k). Fix this. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/xfs | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/common/xfs b/common/xfs index 7b7b3a35b5..546853247c 100644 --- a/common/xfs +++ b/common/xfs @@ -207,6 +207,8 @@ _xfs_get_file_block_size() { local path="$1" + # If rtinherit or realtime are not set on the path, then all files + # will be created on the data device. if ! ($XFS_IO_PROG -c "stat -v" "$path" 2>&1 | grep -E -q '(rt-inherit|realtime)'); then _get_block_size "$path" return @@ -217,6 +219,15 @@ _xfs_get_file_block_size() while ! $XFS_INFO_PROG "$path" &>/dev/null && [ "$path" != "/" ]; do path="$(dirname "$path")" done + + # If there's no realtime section, the rtinherit and rextsize settings + # are irrelevant -- all files are created on the data device. + if $XFS_INFO_PROG "$path" | grep -q 'realtime =none'; then + _get_block_size "$path" + return + fi + + # Otherwise, report the rt extent size. _xfs_get_rtextsize "$path" } ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 07/10] xfs: remove xfs/131 now that we allow reflink on realtime volumes 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong ` (8 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 10/10] common/xfs: fix _xfs_get_file_block_size when rtinherit is set and no rt section Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 9 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Remove this test, since we now support reflink on the rt volume. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/131 | 48 ------------------------------------------------ tests/xfs/131.out | 5 ----- 2 files changed, 53 deletions(-) delete mode 100755 tests/xfs/131 delete mode 100644 tests/xfs/131.out diff --git a/tests/xfs/131 b/tests/xfs/131 deleted file mode 100755 index 879e2dc6e8..0000000000 --- a/tests/xfs/131 +++ /dev/null @@ -1,48 +0,0 @@ -#! /bin/bash -# SPDX-License-Identifier: GPL-2.0 -# Copyright (c) 2015, Oracle and/or its affiliates. All Rights Reserved. -# -# FS QA Test No. 131 -# -# Ensure that we can't reflink realtime files. -# -. ./common/preamble -_begin_fstest auto quick clone realtime - -# Override the default cleanup function. -_cleanup() -{ - cd / - umount $SCRATCH_MNT > /dev/null 2>&1 - rm -rf $tmp.* $testdir $metadump_file -} - -# Import common functions. -. ./common/filter -. ./common/reflink - -# real QA test starts here -_supported_fs xfs -_require_realtime -_require_scratch_reflink -_require_cp_reflink - -echo "Format and mount scratch device" -_scratch_mkfs >> $seqres.full -_scratch_mount - -testdir=$SCRATCH_MNT/test-$seq -mkdir $testdir - -echo "Create the original file blocks" -blksz=65536 -$XFS_IO_PROG -R -f -c "truncate $blksz" $testdir/file1 - -echo "Reflink every block" -_cp_reflink $testdir/file1 $testdir/file2 2>&1 | _filter_scratch - -test -s $testdir/file2 && _fail "Should not be able to reflink a realtime file." - -# success, all done -status=0 -exit diff --git a/tests/xfs/131.out b/tests/xfs/131.out deleted file mode 100644 index 3c0186f0c7..0000000000 --- a/tests/xfs/131.out +++ /dev/null @@ -1,5 +0,0 @@ -QA output created by 131 -Format and mount scratch device -Create the original file blocks -Reflink every block -cp: failed to clone 'SCRATCH_MNT/test-131/file2' from 'SCRATCH_MNT/test-131/file1': Invalid argument ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCHSET v1.0 0/4] fstests: reflink with large realtime extents 2022-12-30 21:14 [NYE DELUGE 3/4] xfs: modernize the realtime volume Darrick J. Wong ` (5 preceding siblings ...) 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/4] xfs: make sure that CoW will write around when rextsize > 1 Darrick J. Wong ` (3 more replies) 2022-12-30 22:20 ` [PATCHSET v1.0 0/1] fstests: functional tests for rt quota Darrick J. Wong 7 siblings, 4 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, Now that we've landed support for reflink on the realtime device for cases where the rt extent size is the same as the fs block size, enhance the reflink code further to support cases where the rt extent size is a power-of-two multiple of the fs block size. This enables us to do data block sharing (for example) for much larger allocation units by dirtying pagecache around shared extents and expanding writeback to write back shared extents fully. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-reflink-extsize xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=realtime-reflink-extsize fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-reflink-extsize --- common/rc | 23 +++++++ common/reflink | 27 +++++++++ tests/generic/145 | 1 tests/generic/147 | 1 tests/generic/261 | 1 tests/generic/262 | 1 tests/generic/303 | 8 ++- tests/generic/331 | 1 tests/generic/353 | 3 + tests/generic/517 | 1 tests/generic/657 | 1 tests/generic/658 | 1 tests/generic/659 | 1 tests/generic/660 | 1 tests/generic/663 | 1 tests/generic/664 | 1 tests/generic/665 | 1 tests/generic/670 | 1 tests/generic/672 | 1 tests/xfs/1212 | 1 tests/xfs/180 | 1 tests/xfs/182 | 1 tests/xfs/184 | 1 tests/xfs/192 | 1 tests/xfs/200 | 1 tests/xfs/204 | 1 tests/xfs/208 | 1 tests/xfs/315 | 1 tests/xfs/326 | 6 ++ tests/xfs/420 | 3 + tests/xfs/421 | 3 + tests/xfs/919 | 163 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/919.out | 84 +++++++++++++++++++++++++++ 33 files changed, 342 insertions(+), 2 deletions(-) create mode 100755 tests/xfs/919 create mode 100644 tests/xfs/919.out ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCH 1/4] xfs: make sure that CoW will write around when rextsize > 1 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: reflink with large realtime extents Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 2/4] xfs: skip cowextsize hint fragmentation tests on realtime volumes Darrick J. Wong ` (2 subsequent siblings) 3 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Make sure that CoW triggers the intended copy-around behavior when we write a tiny amount to the middle of a large rt extent. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/xfs/919 | 163 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/919.out | 84 +++++++++++++++++++++++++++ 2 files changed, 247 insertions(+) create mode 100755 tests/xfs/919 create mode 100644 tests/xfs/919.out diff --git a/tests/xfs/919 b/tests/xfs/919 new file mode 100755 index 0000000000..45bb42f91f --- /dev/null +++ b/tests/xfs/919 @@ -0,0 +1,163 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 919 +# +# Make sure that copy on write actually does the intended write-around when we +# stage a tiny modification to a large shared realtime extent. We should never +# end up with multiple rt extents mapped to the same region. +# +. ./common/preamble +_begin_fstest auto quick clone realtime + +# Import common functions. +. ./common/filter +. ./common/reflink + +# real QA test starts here +_supported_fs xfs +_require_xfs_io_command "fpunch" +_require_xfs_io_command "fzero" +_require_xfs_io_command "fcollapse" +_require_xfs_io_command "finsert" +_require_xfs_io_command "funshare" +_require_realtime +_require_scratch_reflink + +rtextsz=262144 +filesz=$((rtextsz * 3)) + +echo "Format filesystem and populate" +_scratch_mkfs -m reflink=1 -r extsize=$rtextsz > $seqres.full +_scratch_mount >> $seqres.full + +# Force all our files to be on the realtime device +_xfs_force_bdev realtime $SCRATCH_MNT + +check_file() { + $XFS_IO_PROG -c fsync -c 'bmap -elpv' $1 >> $seqres.full + md5sum $SCRATCH_MNT/a | _filter_scratch + md5sum $1 | _filter_scratch +} + +rtextsz_got=$(_xfs_get_rtextsize "$SCRATCH_MNT") +test $rtextsz_got -eq $rtextsz || \ + _notrun "got rtextsize $rtextsz_got, wanted $rtextsz" + +_pwrite_byte 0x59 0 $filesz $SCRATCH_MNT/a >> $seqres.full +sync +md5sum $SCRATCH_MNT/a | _filter_scratch +$XFS_IO_PROG -c 'bmap -elpv' $SCRATCH_MNT/a >> $seqres.full + +echo "pwrite 1 byte in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/b +_pwrite_byte 0x00 345678 1 $SCRATCH_MNT/b >> $seqres.full +check_file $SCRATCH_MNT/b + +echo "mwrite 1 byte in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/c +$XFS_IO_PROG -c "mmap -rw 0 $filesz" -c "mwrite -S 0x00 345678 1" -c msync $SCRATCH_MNT/c +check_file $SCRATCH_MNT/c + +echo "fzero 1 byte in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/d +$XFS_IO_PROG -c "fzero 345678 1" $SCRATCH_MNT/d +check_file $SCRATCH_MNT/d + +echo "fpunch 1 byte in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/e +$XFS_IO_PROG -c "fpunch 345678 1" $SCRATCH_MNT/e +check_file $SCRATCH_MNT/e + +echo "funshare 1 byte in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/f +$XFS_IO_PROG -c "funshare 345678 1" $SCRATCH_MNT/f +check_file $SCRATCH_MNT/f + +echo "pwrite 1 block in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/g +_pwrite_byte 0x00 327680 65536 $SCRATCH_MNT/g >> $seqres.full +check_file $SCRATCH_MNT/g + +echo "mwrite 1 block in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/h +$XFS_IO_PROG -c "mmap -rw 0 $filesz" -c "mwrite -S 0x00 327680 65536" -c msync $SCRATCH_MNT/h +check_file $SCRATCH_MNT/h + +echo "fzero 1 block in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/i +$XFS_IO_PROG -c "fzero 327680 65536" $SCRATCH_MNT/i +check_file $SCRATCH_MNT/i + +echo "fpunch 1 block in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/j +$XFS_IO_PROG -c "fpunch 327680 65536" $SCRATCH_MNT/j +check_file $SCRATCH_MNT/j + +echo "funshare 1 block in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/k +$XFS_IO_PROG -c "funshare 327680 65536" $SCRATCH_MNT/k +check_file $SCRATCH_MNT/k + +echo "pwrite 1 extent in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/l +_pwrite_byte 0x00 262144 262144 $SCRATCH_MNT/l >> $seqres.full +check_file $SCRATCH_MNT/l + +echo "mwrite 1 extent in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/m +$XFS_IO_PROG -c "mmap -rw 0 $filesz" -c "mwrite -S 0x00 262144 262144" -c msync $SCRATCH_MNT/m +check_file $SCRATCH_MNT/m + +echo "fzero 1 extent in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/n +$XFS_IO_PROG -c "fzero 262144 262144" $SCRATCH_MNT/n +check_file $SCRATCH_MNT/n + +echo "fpunch 1 extent in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/o +$XFS_IO_PROG -c "fpunch 262144 262144" $SCRATCH_MNT/o +check_file $SCRATCH_MNT/o + +echo "funshare 1 extent in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/p +$XFS_IO_PROG -c "funshare 262144 262144" $SCRATCH_MNT/p +check_file $SCRATCH_MNT/p + +echo "fcollapse 1 extent in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/q +$XFS_IO_PROG -c "fcollapse 262144 262144" $SCRATCH_MNT/q +check_file $SCRATCH_MNT/q + +echo "finsert 1 extent in the middle" | tee -a $seqres.full +_cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/r +$XFS_IO_PROG -c "finsert 262144 262144" $SCRATCH_MNT/r +check_file $SCRATCH_MNT/r + +echo "copy unwritten blocks in large rtext" | tee -a $seqres.full +$XFS_IO_PROG -f -c "falloc 0 $filesz" -c 'pwrite -S 0x59 345678 1' $SCRATCH_MNT/s >> $seqres.full +$XFS_IO_PROG -c 'bmap -elpv' $SCRATCH_MNT/s >> $seqres.full +_cp_reflink $SCRATCH_MNT/s $SCRATCH_MNT/t +$XFS_IO_PROG -c 'bmap -elpv' $SCRATCH_MNT/s >> $seqres.full +$XFS_IO_PROG -f -c 'pwrite -S 0x59 1048576 1' $SCRATCH_MNT/s >> $seqres.full +$XFS_IO_PROG -f -c 'pwrite -S 0x59 1048576 1' $SCRATCH_MNT/t >> $seqres.full +check_file $SCRATCH_MNT/s +check_file $SCRATCH_MNT/t + +echo "test writing to shared unwritten extent" | tee -a $seqres.full +$XFS_IO_PROG -c 'bmap -elpv' $SCRATCH_MNT/s >> $seqres.full +_cp_reflink $SCRATCH_MNT/s $SCRATCH_MNT/u +$XFS_IO_PROG -c 'bmap -elpv' $SCRATCH_MNT/s >> $seqres.full +$XFS_IO_PROG -f -c 'pwrite -S 0x59 345678 1' $SCRATCH_MNT/u >> $seqres.full +check_file $SCRATCH_MNT/u + +echo "Remount and recheck" | tee -a $seqres.full +md5sum $SCRATCH_MNT/a | _filter_scratch +for i in b c d e f g h i j k l m n o p q r s t u; do + check_file $SCRATCH_MNT/$i | grep -v SCRATCH_MNT.a +done + +# success, all done +status=0 +exit diff --git a/tests/xfs/919.out b/tests/xfs/919.out new file mode 100644 index 0000000000..6ab3f70d17 --- /dev/null +++ b/tests/xfs/919.out @@ -0,0 +1,84 @@ +QA output created by 919 +Format filesystem and populate +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +pwrite 1 byte in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +ea0b05f13c8cce703accaffe56d59bd3 SCRATCH_MNT/b +mwrite 1 byte in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +ea0b05f13c8cce703accaffe56d59bd3 SCRATCH_MNT/c +fzero 1 byte in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +ea0b05f13c8cce703accaffe56d59bd3 SCRATCH_MNT/d +fpunch 1 byte in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +ea0b05f13c8cce703accaffe56d59bd3 SCRATCH_MNT/e +funshare 1 byte in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/f +pwrite 1 block in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +2a508e23efc80e468efa7004fd8a1839 SCRATCH_MNT/g +mwrite 1 block in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +2a508e23efc80e468efa7004fd8a1839 SCRATCH_MNT/h +fzero 1 block in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +2a508e23efc80e468efa7004fd8a1839 SCRATCH_MNT/i +fpunch 1 block in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +2a508e23efc80e468efa7004fd8a1839 SCRATCH_MNT/j +funshare 1 block in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/k +pwrite 1 extent in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +352abe71b0d40f194b9d701750b0d7f3 SCRATCH_MNT/l +mwrite 1 extent in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +352abe71b0d40f194b9d701750b0d7f3 SCRATCH_MNT/m +fzero 1 extent in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +352abe71b0d40f194b9d701750b0d7f3 SCRATCH_MNT/n +fpunch 1 extent in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +352abe71b0d40f194b9d701750b0d7f3 SCRATCH_MNT/o +funshare 1 extent in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/p +fcollapse 1 extent in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +b0581637c15320958874ef3f082111da SCRATCH_MNT/q +finsert 1 extent in the middle +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +a7359d0c100367c2cd430be334dffbd3 SCRATCH_MNT/r +copy unwritten blocks in large rtext +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +bff4e0a70430429c92d6139065e6949b SCRATCH_MNT/s +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +bff4e0a70430429c92d6139065e6949b SCRATCH_MNT/t +test writing to shared unwritten extent +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +bff4e0a70430429c92d6139065e6949b SCRATCH_MNT/u +Remount and recheck +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/a +ea0b05f13c8cce703accaffe56d59bd3 SCRATCH_MNT/b +ea0b05f13c8cce703accaffe56d59bd3 SCRATCH_MNT/c +ea0b05f13c8cce703accaffe56d59bd3 SCRATCH_MNT/d +ea0b05f13c8cce703accaffe56d59bd3 SCRATCH_MNT/e +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/f +2a508e23efc80e468efa7004fd8a1839 SCRATCH_MNT/g +2a508e23efc80e468efa7004fd8a1839 SCRATCH_MNT/h +2a508e23efc80e468efa7004fd8a1839 SCRATCH_MNT/i +2a508e23efc80e468efa7004fd8a1839 SCRATCH_MNT/j +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/k +352abe71b0d40f194b9d701750b0d7f3 SCRATCH_MNT/l +352abe71b0d40f194b9d701750b0d7f3 SCRATCH_MNT/m +352abe71b0d40f194b9d701750b0d7f3 SCRATCH_MNT/n +352abe71b0d40f194b9d701750b0d7f3 SCRATCH_MNT/o +924a97fdaa2ab30e2768081469e728a7 SCRATCH_MNT/p +b0581637c15320958874ef3f082111da SCRATCH_MNT/q +a7359d0c100367c2cd430be334dffbd3 SCRATCH_MNT/r +bff4e0a70430429c92d6139065e6949b SCRATCH_MNT/s +bff4e0a70430429c92d6139065e6949b SCRATCH_MNT/t +bff4e0a70430429c92d6139065e6949b SCRATCH_MNT/u ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 2/4] xfs: skip cowextsize hint fragmentation tests on realtime volumes 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: reflink with large realtime extents Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/4] xfs: make sure that CoW will write around when rextsize > 1 Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 4/4] generic/303: avoid test failures on weird rt extent sizes Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/4] misc: add more congruent oplen testing Darrick J. Wong 3 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> The XFS CoW extent size hint is ignored on realtime filesystems when the rt extent size set to a unit larger than a single filesystem block because it is assumed that the larger allocation unit is the administrator's sole and mandatory anti-fragmentation strategy. As such, we can skip these tests. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/reflink | 27 +++++++++++++++++++++++++++ tests/xfs/180 | 1 + tests/xfs/182 | 1 + tests/xfs/184 | 1 + tests/xfs/192 | 1 + tests/xfs/200 | 1 + tests/xfs/204 | 1 + tests/xfs/208 | 1 + tests/xfs/315 | 1 + tests/xfs/326 | 6 ++++++ 10 files changed, 41 insertions(+) diff --git a/common/reflink b/common/reflink index 22adc4449b..1082642e4e 100644 --- a/common/reflink +++ b/common/reflink @@ -521,3 +521,30 @@ _sweave_reflink_rainbow_delalloc() { _pwrite_byte 0x62 $((blksz * i)) $blksz $dfile.chk done } + +# Require that the COW extent size hint can actually be used to combat +# fragmentation on the scratch filesystem. This is (so far) true for any +# filesystem except for the ones where the realtime extent size is larger +# than one fs block, for it is assumed that setting a rt extent size is the +# preferred fragmentation avoidance strategy. +_require_scratch_cowextsize_useful() { + local testfile=$SCRATCH_MNT/hascowextsize + local param="${1:-1m}" + + rm -f $testfile + touch $testfile + local before="$($XFS_IO_PROG -c 'cowextsize' $testfile)" + + $XFS_IO_PROG -c "cowextsize $param" $testfile + local after="$($XFS_IO_PROG -c 'cowextsize' $testfile)" + rm -f $testfile + + test "$before" != "$after" || \ + _notrun "setting cowextsize to $param had no effect" + + local fileblocksize=$(_get_file_block_size $SCRATCH_MNT) + local fsblocksize=$(_get_block_size $SCRATCH_MNT) + + test $fsblocksize -eq $fileblocksize || \ + _notrun "XFS does not support cowextsize when rt extsize ($fileblocksize) > 1FSB ($fsblocksize)" +} diff --git a/tests/xfs/180 b/tests/xfs/180 index cfea2020ce..06b4b69d52 100755 --- a/tests/xfs/180 +++ b/tests/xfs/180 @@ -37,6 +37,7 @@ nr=128 filesize=$((blksz * nr)) bufnr=16 bufsize=$((blksz * bufnr)) +_require_scratch_cowextsize_useful $bufsize _require_fs_space $SCRATCH_MNT $((filesize / 1024 * 3 * 5 / 4)) real_blksz=$(_get_block_size $testdir) diff --git a/tests/xfs/182 b/tests/xfs/182 index 511aca6f2d..7c0713b248 100755 --- a/tests/xfs/182 +++ b/tests/xfs/182 @@ -39,6 +39,7 @@ nr=128 filesize=$((blksz * nr)) bufnr=16 bufsize=$((blksz * bufnr)) +_require_scratch_cowextsize_useful $bufsize _require_fs_space $SCRATCH_MNT $((filesize / 1024 * 3 * 5 / 4)) real_blksz=$(_get_block_size $testdir) diff --git a/tests/xfs/184 b/tests/xfs/184 index 3bdd86addf..a0dc2741f5 100755 --- a/tests/xfs/184 +++ b/tests/xfs/184 @@ -39,6 +39,7 @@ nr=128 filesize=$((blksz * nr)) bufnr=16 bufsize=$((blksz * bufnr)) +_require_scratch_cowextsize_useful $bufsize _require_fs_space $SCRATCH_MNT $((filesize / 1024 * 3 * 5 / 4)) real_blksz=$(_get_block_size $testdir) diff --git a/tests/xfs/192 b/tests/xfs/192 index eb577f15fc..daa4fcb1e0 100755 --- a/tests/xfs/192 +++ b/tests/xfs/192 @@ -41,6 +41,7 @@ nr=128 filesize=$((blksz * nr)) bufnr=16 bufsize=$((blksz * bufnr)) +_require_scratch_cowextsize_useful $bufsize _require_fs_space $SCRATCH_MNT $((filesize / 1024 * 3 * 5 / 4)) real_blksz=$(_get_block_size $testdir) diff --git a/tests/xfs/200 b/tests/xfs/200 index b51b9a54f5..8eb54b5755 100755 --- a/tests/xfs/200 +++ b/tests/xfs/200 @@ -41,6 +41,7 @@ nr=128 filesize=$((blksz * nr)) bufnr=16 bufsize=$((blksz * bufnr)) +_require_scratch_cowextsize_useful $bufsize _require_fs_space $SCRATCH_MNT $((filesize / 1024 * 3 * 5 / 4)) real_blksz=$(_get_block_size $testdir) diff --git a/tests/xfs/204 b/tests/xfs/204 index 7d6b79a86d..85b7ed7cd9 100755 --- a/tests/xfs/204 +++ b/tests/xfs/204 @@ -43,6 +43,7 @@ nr=128 filesize=$((blksz * nr)) bufnr=16 bufsize=$((blksz * bufnr)) +_require_scratch_cowextsize_useful $bufsize _require_fs_space $SCRATCH_MNT $((filesize / 1024 * 3 * 5 / 4)) real_blksz=$(_get_block_size $testdir) diff --git a/tests/xfs/208 b/tests/xfs/208 index 9a71b74f6f..3a4a3e4df1 100755 --- a/tests/xfs/208 +++ b/tests/xfs/208 @@ -40,6 +40,7 @@ nr=128 filesize=$((blksz * nr)) bufnr=16 bufsize=$((blksz * bufnr)) +_require_scratch_cowextsize_useful $bufsize _require_fs_space $SCRATCH_MNT $((filesize / 1024 * 3 * 5 / 4)) real_blksz=$(_get_file_block_size $testdir) diff --git a/tests/xfs/315 b/tests/xfs/315 index 9f6b39c8cc..3a618a3680 100755 --- a/tests/xfs/315 +++ b/tests/xfs/315 @@ -38,6 +38,7 @@ echo "Format filesystem" _scratch_mkfs >/dev/null 2>&1 _scratch_mount >> $seqres.full _require_congruent_file_oplen $SCRATCH_MNT $blksz +_require_scratch_cowextsize_useful $sz $XFS_IO_PROG -c "cowextsize $sz" $SCRATCH_MNT diff --git a/tests/xfs/326 b/tests/xfs/326 index ac620fc433..a3fed8b6ac 100755 --- a/tests/xfs/326 +++ b/tests/xfs/326 @@ -55,6 +55,12 @@ $XFS_IO_PROG -c "cowextsize $sz" $SCRATCH_MNT # staging extent for an unshared extent and trips over the injected error. _require_no_xfs_always_cow +# This test uses a very large cowextszhint to manipulate the COW fork to +# contain a large unwritten extent before injecting the error. XFS ignores +# cowextsize when the realtime extent size is greater than 1FSB, so this test +# cannot set up the preconditions for the test. +_require_scratch_cowextsize_useful $sz + echo "Create files" _pwrite_byte 0x66 0 $sz $SCRATCH_MNT/file1 >> $seqres.full _cp_reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file2 ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 4/4] generic/303: avoid test failures on weird rt extent sizes 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: reflink with large realtime extents Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/4] xfs: make sure that CoW will write around when rextsize > 1 Darrick J. Wong 2022-12-30 22:20 ` [PATCH 2/4] xfs: skip cowextsize hint fragmentation tests on realtime volumes Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/4] misc: add more congruent oplen testing Darrick J. Wong 3 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Fix this test to skip the high offset reflink test if (on XFS) the rt extent size isn't congruent with the chosen target offset. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/rc | 23 +++++++++++++++++++++++ tests/generic/303 | 8 +++++++- 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/common/rc b/common/rc index cfe765de2e..3c30a444fe 100644 --- a/common/rc +++ b/common/rc @@ -4488,6 +4488,29 @@ _get_file_block_size() esac } +_test_congruent_file_oplen() +{ + local file="$1" + local alloc_unit=$(_get_file_block_size "$file") + local oplen="$2" + + case $FSTYP in + nfs*|cifs|9p|virtiofs|ceph|glusterfs|overlay|pvfs2) + # Network filesystems don't know about (or tell the client + # about) the underlying file allocation unit and they generally + # pass the file IO request to the underlying filesystem, so we + # don't have anything to check here. + return + ;; + esac + + if [ $alloc_unit -gt $oplen ]; then + return 1 + fi + test $((oplen % alloc_unit)) -eq 0 || return 1 + return 0 +} + # Given a file path and a byte length of a file operation under test, ensure # that the length is an integer multiple of the file's allocation unit size. # In other words, skip the test unless (oplen ≡ alloc_unit mod 0). This is diff --git a/tests/generic/303 b/tests/generic/303 index 95679569e4..ef88d2357b 100755 --- a/tests/generic/303 +++ b/tests/generic/303 @@ -48,7 +48,13 @@ echo "Reflink past maximum file size in dest file (should fail)" _reflink_range $testdir/file1 0 $testdir/file5 4611686018427322368 $len >> $seqres.full echo "Reflink high offset to low offset" -_reflink_range $testdir/file1 $bigoff_64k $testdir/file6 1048576 65535 >> $seqres.full +oplen=1048576 +if _test_congruent_file_oplen $testdir $oplen; then + _reflink_range $testdir/file1 $bigoff_64k $testdir/file6 $oplen 65535 >> $seqres.full +else + # If we can't do the ficlonerange test, fake it in the output file + $XFS_IO_PROG -f -c 'pwrite -S 0x61 1114110 1' $testdir/file6 >> $seqres.full +fi echo "Reflink past source file EOF (should fail)" _reflink_range $testdir/file2 524288 $testdir/file7 0 1048576 >> $seqres.full ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCH 3/4] misc: add more congruent oplen testing 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: reflink with large realtime extents Darrick J. Wong ` (2 preceding siblings ...) 2022-12-30 22:20 ` [PATCH 4/4] generic/303: avoid test failures on weird rt extent sizes Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 3 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Do more checking for file allocation operation op length congruency. This prevents tests from failing with EINVAL when the realtime extent size is something weird like 28k or 1GB. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- tests/generic/145 | 1 + tests/generic/147 | 1 + tests/generic/261 | 1 + tests/generic/262 | 1 + tests/generic/331 | 1 + tests/generic/353 | 3 ++- tests/generic/517 | 1 + tests/generic/657 | 1 + tests/generic/658 | 1 + tests/generic/659 | 1 + tests/generic/660 | 1 + tests/generic/663 | 1 + tests/generic/664 | 1 + tests/generic/665 | 1 + tests/generic/670 | 1 + tests/generic/672 | 1 + tests/xfs/1212 | 1 + tests/xfs/420 | 3 +++ tests/xfs/421 | 3 +++ 19 files changed, 24 insertions(+), 1 deletion(-) diff --git a/tests/generic/145 b/tests/generic/145 index f213f53be8..81fc5f6c2f 100755 --- a/tests/generic/145 +++ b/tests/generic/145 @@ -36,6 +36,7 @@ mkdir $testdir echo "Create the original files" blksz=65536 +_require_congruent_file_oplen $TEST_DIR $blksz _pwrite_byte 0x61 0 $blksz $testdir/file1 >> $seqres.full _pwrite_byte 0x62 $blksz $blksz $testdir/file1 >> $seqres.full _pwrite_byte 0x63 $((blksz * 2)) $blksz $testdir/file1 >> $seqres.full diff --git a/tests/generic/147 b/tests/generic/147 index 113800944b..bb17bb1c0b 100755 --- a/tests/generic/147 +++ b/tests/generic/147 @@ -35,6 +35,7 @@ mkdir $testdir echo "Create the original files" blksz=65536 +_require_congruent_file_oplen $TEST_DIR $blksz _pwrite_byte 0x61 0 $blksz $testdir/file1 >> $seqres.full _pwrite_byte 0x62 $blksz $blksz $testdir/file1 >> $seqres.full _pwrite_byte 0x63 $((blksz * 2)) $blksz $testdir/file1 >> $seqres.full diff --git a/tests/generic/261 b/tests/generic/261 index 93c1c349b1..deb360288e 100755 --- a/tests/generic/261 +++ b/tests/generic/261 @@ -29,6 +29,7 @@ testdir=$SCRATCH_MNT/test-$seq mkdir $testdir blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=5 filesize=$((blksz * nr)) diff --git a/tests/generic/262 b/tests/generic/262 index 46e88f8731..f296e37e02 100755 --- a/tests/generic/262 +++ b/tests/generic/262 @@ -29,6 +29,7 @@ testdir=$SCRATCH_MNT/test-$seq mkdir $testdir blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=4 filesize=$((blksz * nr)) diff --git a/tests/generic/331 b/tests/generic/331 index 8c665ce4fc..9b6801e16f 100755 --- a/tests/generic/331 +++ b/tests/generic/331 @@ -38,6 +38,7 @@ testdir=$SCRATCH_MNT/test-$seq mkdir $testdir blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=640 bufnr=128 filesize=$((blksz * nr)) diff --git a/tests/generic/353 b/tests/generic/353 index 9a1471bd81..94c9ac2273 100755 --- a/tests/generic/353 +++ b/tests/generic/353 @@ -29,7 +29,8 @@ _require_xfs_io_command "fiemap" _scratch_mkfs > /dev/null 2>&1 _scratch_mount -blocksize=64k +blocksize=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blocksize file1="$SCRATCH_MNT/file1" file2="$SCRATCH_MNT/file2" diff --git a/tests/generic/517 b/tests/generic/517 index cf3031ed2d..229358d06b 100755 --- a/tests/generic/517 +++ b/tests/generic/517 @@ -21,6 +21,7 @@ _require_scratch_dedupe _scratch_mkfs >>$seqres.full 2>&1 _scratch_mount +_require_congruent_file_oplen $SCRATCH_MNT 65536 # The first byte with a value of 0xae starts at an offset (512Kb + 100) which is # not a multiple of the block size. diff --git a/tests/generic/657 b/tests/generic/657 index e0fecd544c..9f4673dda3 100755 --- a/tests/generic/657 +++ b/tests/generic/657 @@ -30,6 +30,7 @@ mkdir $testdir echo "Create the original files" blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=64 filesize=$((blksz * nr)) _pwrite_byte 0x61 0 $filesize $testdir/file1 >> $seqres.full diff --git a/tests/generic/658 b/tests/generic/658 index a5cbadaaa5..e9519c25e2 100755 --- a/tests/generic/658 +++ b/tests/generic/658 @@ -31,6 +31,7 @@ mkdir $testdir echo "Create the original files" blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=64 filesize=$((blksz * nr)) _weave_reflink_regular $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full diff --git a/tests/generic/659 b/tests/generic/659 index ccc2d7950d..05436edfab 100755 --- a/tests/generic/659 +++ b/tests/generic/659 @@ -31,6 +31,7 @@ mkdir $testdir echo "Create the original files" blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=64 filesize=$((blksz * nr)) _weave_reflink_unwritten $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full diff --git a/tests/generic/660 b/tests/generic/660 index bc17dc5e59..52b0d1ea9e 100755 --- a/tests/generic/660 +++ b/tests/generic/660 @@ -31,6 +31,7 @@ mkdir $testdir echo "Create the original files" blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=64 filesize=$((blksz * nr)) _weave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full diff --git a/tests/generic/663 b/tests/generic/663 index 658a5b7004..692c77b745 100755 --- a/tests/generic/663 +++ b/tests/generic/663 @@ -32,6 +32,7 @@ mkdir $testdir echo "Create the original files" blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=64 filesize=$((blksz * nr)) _sweave_reflink_regular $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full diff --git a/tests/generic/664 b/tests/generic/664 index 3009101fdc..40fb8c6d92 100755 --- a/tests/generic/664 +++ b/tests/generic/664 @@ -34,6 +34,7 @@ mkdir $testdir echo "Create the original files" blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=64 filesize=$((blksz * nr)) _sweave_reflink_unwritten $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full diff --git a/tests/generic/665 b/tests/generic/665 index 86ba578720..ee511755e6 100755 --- a/tests/generic/665 +++ b/tests/generic/665 @@ -34,6 +34,7 @@ mkdir $testdir echo "Create the original files" blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=64 filesize=$((blksz * nr)) _sweave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full diff --git a/tests/generic/670 b/tests/generic/670 index 67de167405..80f9fe6d4f 100755 --- a/tests/generic/670 +++ b/tests/generic/670 @@ -31,6 +31,7 @@ mkdir $testdir loops=512 nr_loops=$((loops - 1)) blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz echo "Initialize files" echo >> $seqres.full diff --git a/tests/generic/672 b/tests/generic/672 index 9e3a97ec5e..0710a04294 100755 --- a/tests/generic/672 +++ b/tests/generic/672 @@ -30,6 +30,7 @@ mkdir $testdir loops=1024 nr_loops=$((loops - 1)) blksz=65536 +_require_congruent_file_oplen $SCRATCH_MNT $blksz echo "Initialize files" echo >> $seqres.full diff --git a/tests/xfs/1212 b/tests/xfs/1212 index d2292d65a2..655cd14021 100755 --- a/tests/xfs/1212 +++ b/tests/xfs/1212 @@ -32,6 +32,7 @@ _require_xfs_io_error_injection "bmap_finish_one" _scratch_mkfs >> $seqres.full _scratch_mount +_require_congruent_file_oplen $SCRATCH_MNT 65536 # Create original file _pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full diff --git a/tests/xfs/420 b/tests/xfs/420 index d38772c9d9..51f87bc304 100755 --- a/tests/xfs/420 +++ b/tests/xfs/420 @@ -69,6 +69,9 @@ exercise_lseek() { } blksz=65536 +# Golden output encodes SEEK_HOLE/DATA output, which depends on COW only +# happening on $blksz granularity +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=8 filesize=$((blksz * nr)) diff --git a/tests/xfs/421 b/tests/xfs/421 index 027ae47c21..429333e349 100755 --- a/tests/xfs/421 +++ b/tests/xfs/421 @@ -51,6 +51,9 @@ testdir=$SCRATCH_MNT/test-$seq mkdir $testdir blksz=65536 +# Golden output encodes SEEK_HOLE/DATA output, which depends on COW only +# happening on $blksz granularity +_require_congruent_file_oplen $SCRATCH_MNT $blksz nr=8 filesize=$((blksz * nr)) ^ permalink raw reply related [flat|nested] 68+ messages in thread
* [PATCHSET v1.0 0/1] fstests: functional tests for rt quota 2022-12-30 21:14 [NYE DELUGE 3/4] xfs: modernize the realtime volume Darrick J. Wong ` (6 preceding siblings ...) 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: reflink with large realtime extents Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/1] xfs: regression testing of quota on the realtime device Darrick J. Wong 7 siblings, 1 reply; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan Hi all, The sole patch in this series sets up functional testing for quota on the xfs realtime device. If you're going to start using this mess, you probably ought to just pull from my git trees, which are linked below. This is an extraordinary way to destroy everything. Enjoy! Comments and questions are, as always, welcome. --D kernel git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-quotas xfsprogs git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=realtime-quotas fstests git tree: https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-quotas --- common/quota | 30 ++++++++++ tests/xfs/767 | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/767.out | 41 +++++++++++++ 3 files changed, 238 insertions(+) create mode 100755 tests/xfs/767 create mode 100644 tests/xfs/767.out ^ permalink raw reply [flat|nested] 68+ messages in thread
* [PATCH 1/1] xfs: regression testing of quota on the realtime device 2022-12-30 22:20 ` [PATCHSET v1.0 0/1] fstests: functional tests for rt quota Darrick J. Wong @ 2022-12-30 22:20 ` Darrick J. Wong 0 siblings, 0 replies; 68+ messages in thread From: Darrick J. Wong @ 2022-12-30 22:20 UTC (permalink / raw) To: zlang, djwong; +Cc: linux-xfs, fstests, guan From: Darrick J. Wong <djwong@kernel.org> Make sure that quota accounting and enforcement work correctly for realtime volumes on XFS. Signed-off-by: Darrick J. Wong <djwong@kernel.org> --- common/quota | 30 ++++++++++ tests/xfs/767 | 167 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/767.out | 41 +++++++++++++ 3 files changed, 238 insertions(+) create mode 100755 tests/xfs/767 create mode 100644 tests/xfs/767.out diff --git a/common/quota b/common/quota index 96b8d04424..f4c528c836 100644 --- a/common/quota +++ b/common/quota @@ -117,6 +117,36 @@ _require_xfs_quota_acct_enabled() _notrun "$qtype: accounting not enabled on $fsname filesystem." } +# Decide if the mounted filesystem supports realtime quotas. +_require_rtquota() +{ + local dev="$1" + test -z "$dev" && dev="$TEST_DEV" + local rtdev="$2" + test -z "$rtdev" && rtdev="$TEST_RTDEV" + + test "$FSTYP" = "xfs" || \ + _notrun "Realtime quota only supported on xfs" + + [ -n "$XFS_QUOTA_PROG" ] || \ + _notrun "xfs_quota user tool not installed" + + $here/src/feature -q $dev || \ + _notrun "Installed kernel does not support XFS quotas" + + test -b "$rtdev" || \ + _notrun "No realtime device supplied?" + + test "$USE_EXTERNAL" = "yes" || \ + _notrun "Realtime requires USE_EXTERNAL='yes'" + + $here/src/feature -U $dev || \ + $here/src/feature -G $dev || \ + $here/src/feature -P $dev || \ + _notrun "Mounted rt filesystem does not have quotas enabled" + +} + # # checks that xfs_quota can operate on foreign (non-xfs) filesystems # Skips check on xfs filesystems, old xfs_quota is fine there. diff --git a/tests/xfs/767 b/tests/xfs/767 new file mode 100755 index 0000000000..f30321d7fc --- /dev/null +++ b/tests/xfs/767 @@ -0,0 +1,167 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2022 Oracle. All Rights Reserved. +# +# FS QA Test No. 767 +# +# Functional testing for realtime quotas. + +. ./common/preamble +_begin_fstest auto quick quota realtime + +# Import common functions. +. ./common/quota +. ./common/filter + +# real QA test starts here +_supported_fs xfs +_require_test_program "punch-alternating" +_require_scratch +_require_user + +echo "Format filesystem" | tee -a $seqres.full +_scratch_mkfs > $seqres.full +_qmount_option 'usrquota' +_qmount +_require_rtquota $SCRATCH_DEV $SCRATCH_RTDEV + +# Make sure all our files are on the rt device +_xfs_force_bdev realtime $SCRATCH_MNT +chmod a+rwx $SCRATCH_MNT + +# Record rt geometry +bmbt_blksz=$(_get_block_size $SCRATCH_MNT) +file_blksz=$(_get_file_block_size $SCRATCH_MNT) +rextsize=$((file_blksz / bmbt_blksz)) +echo "bmbt_blksz $bmbt_blksz" >> $seqres.full +echo "file_blksz $file_blksz" >> $seqres.full +echo "rextsize $rextsize" >> $seqres.full + +note() { + echo -e "\n$@" | tee -a $seqres.full +} + +# Report on the user's block and rt block usage, soft limit, hard limit, and +# warning count for rt volumes +report_rtusage() { + local user="$1" + local timeout_arg="$2" + local print_timeout=0 + + test -z "$user" && user=$qa_user + test -n "$timeout_arg" && print_timeout=1 + + $XFS_QUOTA_PROG -c "quota -u -r -n -N $user" $SCRATCH_MNT | \ + sed -e 's/ days/_days/g' >> $seqres.full + + $XFS_QUOTA_PROG -c "quota -u -r -n -N $user" $SCRATCH_MNT | \ + sed -e 's/ days/_days/g' | \ + awk -v user=$user -v print_timeout=$print_timeout -v file_blksz=$file_blksz \ + '{printf("%s[real] %d %d %d %d %s\n", user, $2 * 1024 / file_blksz, $3 * 1024 / file_blksz, $4 * 1024 / file_blksz, $5, print_timeout ? $6 : "---");}' +} + +note "Write 128rx to root" +$XFS_IO_PROG -f -c "pwrite 0 $((128 * file_blksz))" $SCRATCH_MNT/file1 > /dev/null +chmod a+r $SCRATCH_MNT/file1 +sync +report_rtusage 0 + +note "Write 64rx to root, 4444, and 5555." +$XFS_IO_PROG -f -c "pwrite 0 $((64 * file_blksz))" $SCRATCH_MNT/file3.5555 > /dev/null +chown 5555 $SCRATCH_MNT/file3.5555 +$XFS_IO_PROG -f -c "pwrite 0 $((64 * file_blksz))" $SCRATCH_MNT/file3.4444 > /dev/null +chown 4444 $SCRATCH_MNT/file3.4444 +$XFS_IO_PROG -f -c "pwrite 0 $((64 * file_blksz))" $SCRATCH_MNT/file3 > /dev/null +sync +report_rtusage 0 +report_rtusage 4444 +report_rtusage 5555 + +note "Move 64rx from root to 5555" +chown 5555 $SCRATCH_MNT/file3 +report_rtusage 0 +report_rtusage 4444 +report_rtusage 5555 + +note "Move 64rx from 5555 to 4444" +chown 4444 $SCRATCH_MNT/file3 +report_rtusage 0 +report_rtusage 4444 +report_rtusage 5555 + +note "Set hard limit of 1024rx and check enforcement" +$XFS_QUOTA_PROG -x -c "limit -u rtbhard=$((1024 * file_blksz)) $qa_user" $SCRATCH_MNT +su $qa_user -c "$XFS_IO_PROG -f -c 'pwrite 0 $((2048 * file_blksz))' $SCRATCH_MNT/file2" +report_rtusage + +note "Set soft limit of 512rx and check timelimit enforcement" +rm -f $SCRATCH_MNT/file2 $SCRATCH_MNT/file2.1 +$XFS_QUOTA_PROG -x -c "limit -u rtbsoft=$((512 * file_blksz)) rtbhard=0 $qa_user" $SCRATCH_MNT +$XFS_QUOTA_PROG -x -c "timer -u -r -d 2" $SCRATCH_MNT +$XFS_QUOTA_PROG -x -c 'state -u' $SCRATCH_MNT >> $seqres.full + +su $qa_user -c "$XFS_IO_PROG -f -c 'pwrite 0 $((512 * file_blksz))' $SCRATCH_MNT/file2" > /dev/null +report_rtusage + +overflow=$(date +%s) +su $qa_user -c "$XFS_IO_PROG -f -c 'pwrite -b $file_blksz 0 $file_blksz' $SCRATCH_MNT/file2.1" > /dev/null +report_rtusage +sleep 1 +echo "Try again after 1s" +su $qa_user -c "$XFS_IO_PROG -f -c 'pwrite -b $file_blksz $file_blksz $file_blksz' $SCRATCH_MNT/file2.1" > /dev/null +report_rtusage +sleep 2 +echo "Try again after 3s" +su $qa_user -c "$XFS_IO_PROG -f -c 'pwrite -b $file_blksz $((2 * file_blksz)) $file_blksz' $SCRATCH_MNT/file2.1" > /dev/null +report_rtusage + +note "Extend time limits and warnings" +rm -f $SCRATCH_MNT/file2 $SCRATCH_MNT/file2.1 +$XFS_QUOTA_PROG -x -c "limit -u rtbsoft=$((512 * file_blksz)) rtbhard=0 $qa_user" $SCRATCH_MNT +$XFS_QUOTA_PROG -x -c "timer -u -r -d 49h" $SCRATCH_MNT +$XFS_QUOTA_PROG -x -c 'state -u' $SCRATCH_MNT >> $seqres.full + +su $qa_user -c "$XFS_IO_PROG -f -c 'pwrite 0 $((512 * file_blksz))' $SCRATCH_MNT/file2" > /dev/null +report_rtusage $qa_user want_timeout + +su $qa_user -c "$XFS_IO_PROG -f -c 'pwrite -b $file_blksz 0 $file_blksz' $SCRATCH_MNT/file2.1" > /dev/null +report_rtusage $qa_user want_timeout + +$XFS_QUOTA_PROG -x -c "timer -u -r 73h $qa_user" $SCRATCH_MNT + +su $qa_user -c "$XFS_IO_PROG -f -c 'pwrite -b $file_blksz $file_blksz $file_blksz' $SCRATCH_MNT/file2.1" > /dev/null +report_rtusage $qa_user want_timeout + +note "Test quota applied to bmbt" + +# Testing quota enforcement for bmbt shape changes is tricky. The block +# reservation will be for enough blocks to handle the maximal btree split. +# This is (approximately) 9 blocks no matter the size of the existing extent +# map structure, so we set the hard limit to one more than this quantity. +# +# However, that means that we need to make a file of at least twice that size +# to ensure that we create enough extent records even in the rextsize==1 case +# where punching doesn't just create unwritten records. +# +# Unfortunately, it's very difficult to predict when exactly the EDQUOT will +# come down, so we just look for the error message. +extent_records=$(( (25 * bmbt_blksz) / 16)) +echo "extent_records $extent_records" >> $seqres.full + +rm -f $SCRATCH_MNT/file2 +$XFS_QUOTA_PROG -x -c "limit -u rtbsoft=0 rtbhard=0 $qa_user" $SCRATCH_MNT +$XFS_QUOTA_PROG -x -c "limit -u bhard=$((bmbt_blksz * 10)) bsoft=0 $qa_user" $SCRATCH_MNT +$XFS_QUOTA_PROG -x -c 'state -u' $SCRATCH_MNT >> $seqres.full +$XFS_IO_PROG -f -c "pwrite -S 0x58 -b 64m 0 $((extent_records * file_blksz))" $SCRATCH_MNT/file2 > /dev/null +sync +chown $qa_user $SCRATCH_MNT/file2 +$here/src/punch-alternating $SCRATCH_MNT/file2 2>&1 | _filter_scratch + +$XFS_QUOTA_PROG -c "quota -u -r -n -N $qa_user" -c "quota -u -b -n -N $qa_user" $SCRATCH_MNT >> $seqres.full +$XFS_IO_PROG -c "bmap -e -l -p -v" $SCRATCH_MNT/file2 >> $seqres.full + +# success, all done +$XFS_QUOTA_PROG -x -c 'report -a -u' -c 'report -a -u -r' $SCRATCH_MNT >> $seqres.full +ls -latr $SCRATCH_MNT >> $seqres.full +status=0 +exit diff --git a/tests/xfs/767.out b/tests/xfs/767.out new file mode 100644 index 0000000000..bff7c0f44c --- /dev/null +++ b/tests/xfs/767.out @@ -0,0 +1,41 @@ +QA output created by 767 +Format filesystem + +Write 128rx to root +0[real] 128 0 0 0 --- + +Write 64rx to root, 4444, and 5555. +0[real] 192 0 0 0 --- +4444[real] 64 0 0 0 --- +5555[real] 64 0 0 0 --- + +Move 64rx from root to 5555 +0[real] 128 0 0 0 --- +4444[real] 64 0 0 0 --- +5555[real] 128 0 0 0 --- + +Move 64rx from 5555 to 4444 +0[real] 128 0 0 0 --- +4444[real] 128 0 0 0 --- +5555[real] 64 0 0 0 --- + +Set hard limit of 1024rx and check enforcement +pwrite: Disk quota exceeded +fsgqa[real] 1024 0 1024 0 --- + +Set soft limit of 512rx and check timelimit enforcement +fsgqa[real] 512 512 0 0 --- +fsgqa[real] 513 512 0 0 --- +Try again after 1s +fsgqa[real] 514 512 0 0 --- +Try again after 3s +pwrite: Disk quota exceeded +fsgqa[real] 514 512 0 0 --- + +Extend time limits and warnings +fsgqa[real] 512 512 0 0 [--------] +fsgqa[real] 513 512 0 0 [2_days] +fsgqa[real] 514 512 0 0 [3_days] + +Test quota applied to bmbt +SCRATCH_MNT/file2: Disk quota exceeded ^ permalink raw reply related [flat|nested] 68+ messages in thread
end of thread, other threads:[~2023-03-06 23:15 UTC | newest] Thread overview: 68+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-12-30 21:14 [NYE DELUGE 3/4] xfs: modernize the realtime volume Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET 0/1] fstests: test upgrading older features Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/1] xfs: test upgrading old features Darrick J. Wong 2023-03-06 15:56 ` Zorro Lang 2023-03-06 16:41 ` Darrick J. Wong 2023-03-06 16:54 ` Zorro Lang 2023-03-06 23:14 ` Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v1.0 0/9] fstests: test XFS metadata directories Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/9] xfs/{030,033,178}: forcibly disable metadata directory trees Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/9] xfs/122: fix metadirino Darrick J. Wong 2022-12-30 22:20 ` [PATCH 2/9] various: fix finding metadata inode numbers when metadir is enabled Darrick J. Wong 2023-03-06 16:41 ` Zorro Lang 2022-12-30 22:20 ` [PATCH 4/9] common/repair: patch up repair sb inode value complaints Darrick J. Wong 2022-12-30 22:20 ` [PATCH 6/9] xfs/{050,144,153,299,330}: update quota reports to leave out metadir files Darrick J. Wong 2022-12-30 22:20 ` [PATCH 9/9] xfs: create fuzz tests for metadata directories Darrick J. Wong 2022-12-30 22:20 ` [PATCH 7/9] xfs/769: add metadir upgrade to test matrix Darrick J. Wong 2022-12-30 22:20 ` [PATCH 5/9] xfs/206: update for metadata directory support Darrick J. Wong 2022-12-30 22:20 ` [PATCH 8/9] xfs/509: adjust inumbers accounting for metadata directories Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: support metadump to external devices Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/4] common/ext4: reformat external logs during mdrestore operations Darrick J. Wong 2022-12-30 22:20 ` [PATCH 2/4] common/xfs: wipe " Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/4] common/populate: refactor caching of metadumps to a helper Darrick J. Wong 2022-12-30 22:20 ` [PATCH 4/4] common/xfs: capture external logs during metadump/mdrestore Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v1.0 00/12] xfsprogs: shard the realtime section Darrick J. Wong 2022-12-30 22:20 ` [PATCH 03/12] xfs/206: update mkfs filtering for rt groups feature Darrick J. Wong 2022-12-30 22:20 ` [PATCH 01/12] xfs/122: update for rtgroups Darrick J. Wong 2022-12-30 22:20 ` [PATCH 02/12] punch-alternating: detect xfs realtime files with large allocation units Darrick J. Wong 2022-12-30 22:20 ` [PATCH 10/12] xfs/27[46],xfs/556: fix tests to deal with rtgroups output in bmap/fsmap commands Darrick J. Wong 2022-12-30 22:20 ` [PATCH 07/12] xfs/449: update test to know about xfs_db -R Darrick J. Wong 2022-12-30 22:20 ` [PATCH 11/12] common/xfs: capture realtime devices during metadump/mdrestore Darrick J. Wong 2022-12-30 22:20 ` [PATCH 05/12] common: filter rtgroups when we're disabling metadir Darrick J. Wong 2022-12-30 22:20 ` [PATCH 08/12] xfs/122: update for rtbitmap headers Darrick J. Wong 2022-12-30 22:20 ` [PATCH 06/12] xfs/185: update for rtgroups Darrick J. Wong 2022-12-30 22:20 ` [PATCH 04/12] common: pass the realtime device to xfs_db when possible Darrick J. Wong 2022-12-30 22:20 ` [PATCH 09/12] xfs/122: udpate test to pick up rtword/suminfo ondisk unions Darrick J. Wong 2022-12-30 22:20 ` [PATCH 12/12] common/fuzzy: adapt the scrub stress tests to support rtgroups Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v1.0 00/13] fstests: fixes for realtime rmap Darrick J. Wong 2022-12-30 22:20 ` [PATCH 03/13] xfs: race fsstress with realtime rmap btree scrub and repair Darrick J. Wong 2022-12-30 22:20 ` [PATCH 04/13] xfs/769: add rtrmapbt upgrade to test matrix Darrick J. Wong 2022-12-30 22:20 ` [PATCH 05/13] xfs/122: update for rtgroups-based realtime rmap btrees Darrick J. Wong 2022-12-30 22:20 ` [PATCH 07/13] xfs/341: update test for rtgroup-based rmap Darrick J. Wong 2022-12-30 22:20 ` [PATCH 06/13] xfs: fix various problems with fsmap detecting the data device Darrick J. Wong 2022-12-30 22:20 ` [PATCH 01/13] xfs: fix tests that try to access the realtime rmap inode Darrick J. Wong 2022-12-30 22:20 ` [PATCH 02/13] fuzz: for fuzzing the rtrmapbt, find the path to the rt rmap btree file Darrick J. Wong 2022-12-30 22:20 ` [PATCH 09/13] xfs: skip tests if formatting small filesystem fails Darrick J. Wong 2022-12-30 22:20 ` [PATCH 10/13] xfs/443: use file allocation unit, not dbsize Darrick J. Wong 2022-12-30 22:20 ` [PATCH 12/13] populate: check that we created a realtime rmap btree of the given height Darrick J. Wong 2022-12-30 22:20 ` [PATCH 13/13] fuzzy: create missing fuzz tests for rt rmap btrees Darrick J. Wong 2022-12-30 22:20 ` [PATCH 11/13] populate: adjust rtrmap calculations for rtgroups Darrick J. Wong 2022-12-30 22:20 ` [PATCH 08/13] xfs/3{43,32}: adapt tests for rt extent size greater than 1 Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v1.0 00/10] fstests: reflink on the realtime device Darrick J. Wong 2022-12-30 22:20 ` [PATCH 01/10] xfs/122: update fields for realtime reflink Darrick J. Wong 2022-12-30 22:20 ` [PATCH 02/10] common/populate: create realtime refcount btree Darrick J. Wong 2022-12-30 22:20 ` [PATCH 03/10] xfs: create fuzz tests for the " Darrick J. Wong 2022-12-30 22:20 ` [PATCH 06/10] xfs: race fsstress with realtime refcount btree scrub and repair Darrick J. Wong 2022-12-30 22:20 ` [PATCH 09/10] generic/331,xfs/240: support files that skip delayed allocation Darrick J. Wong 2022-12-30 22:20 ` [PATCH 05/10] xfs/243: don't run when realtime storage is the default Darrick J. Wong 2022-12-30 22:20 ` [PATCH 04/10] xfs/27[24]: adapt for checking files on the realtime volume Darrick J. Wong 2022-12-30 22:20 ` [PATCH 08/10] xfs/769: add rtreflink upgrade to test matrix Darrick J. Wong 2022-12-30 22:20 ` [PATCH 10/10] common/xfs: fix _xfs_get_file_block_size when rtinherit is set and no rt section Darrick J. Wong 2022-12-30 22:20 ` [PATCH 07/10] xfs: remove xfs/131 now that we allow reflink on realtime volumes Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v1.0 0/4] fstests: reflink with large realtime extents Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/4] xfs: make sure that CoW will write around when rextsize > 1 Darrick J. Wong 2022-12-30 22:20 ` [PATCH 2/4] xfs: skip cowextsize hint fragmentation tests on realtime volumes Darrick J. Wong 2022-12-30 22:20 ` [PATCH 4/4] generic/303: avoid test failures on weird rt extent sizes Darrick J. Wong 2022-12-30 22:20 ` [PATCH 3/4] misc: add more congruent oplen testing Darrick J. Wong 2022-12-30 22:20 ` [PATCHSET v1.0 0/1] fstests: functional tests for rt quota Darrick J. Wong 2022-12-30 22:20 ` [PATCH 1/1] xfs: regression testing of quota on the realtime device Darrick J. Wong
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).