* Re: [PATCH v3] generic: test truncating mixed written/unwritten XFS realtime extent
2019-12-03 23:51 [PATCH v3] generic: test truncating mixed written/unwritten XFS realtime extent Omar Sandoval
@ 2019-12-04 0:12 ` Darrick J. Wong
2019-12-08 15:43 ` Eryu Guan
2019-12-08 17:04 ` Eryu Guan
2 siblings, 0 replies; 4+ messages in thread
From: Darrick J. Wong @ 2019-12-04 0:12 UTC (permalink / raw)
To: Omar Sandoval; +Cc: fstests, kernel-team, linux-xfs
On Tue, Dec 03, 2019 at 03:51:52PM -0800, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
>
> The only XFS-specific part of this test is the setup, so we can make the
> rest a generic test. It's slow, though, as it needs to write 8GB to
> convert a big unwritten extent to written.
>
> Signed-off-by: Omar Sandoval <osandov@fb.com>
Looks good to me,
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
--D
> ---
> Changes from v2 -> v3:
>
> - Use _create_loop_device and _destroy_loop_device instead of losetup
>
> Changes from v1 -> v2:
>
> - If rtdev is not configured, fall back to loop device on test
> filesystem
> - Use XFS_IO_PROG instead of fallocate/sync/dd
> - Use truncate instead of rm
> - Add comments explaining the steps
>
> tests/generic/589 | 100 ++++++++++++++++++++++++++++++++++++++++++
> tests/generic/589.out | 2 +
> tests/generic/group | 1 +
> 3 files changed, 103 insertions(+)
> create mode 100755 tests/generic/589
> create mode 100644 tests/generic/589.out
>
> diff --git a/tests/generic/589 b/tests/generic/589
> new file mode 100755
> index 00000000..aab37bb4
> --- /dev/null
> +++ b/tests/generic/589
> @@ -0,0 +1,100 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2019 Facebook. All Rights Reserved.
> +#
> +# FS QA Test 589
> +#
> +# Test "xfs: fix realtime file data space leak" and "xfs: don't check for AG
> +# deadlock for realtime files in bunmapi". On XFS without the fix, truncate
> +# will hang forever. On other filesystems, this just tests writing into big
> +# fallocates.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> + test -n "$loop" && _destroy_loop_device "$loop"
> + rm -f "$TEST_DIR/$seq"
> +}
> +
> +. ./common/rc
> +. ./common/filter
> +
> +rm -f $seqres.full
> +
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch_nocheck
> +
> +maxextlen=$((0x1fffff))
> +bs=4096
> +rextsize=4
> +filesz=$(((maxextlen + 1) * bs))
> +
> +extra_options=""
> +# If we're testing XFS, set up the realtime device to reproduce the bug.
> +if [[ $FSTYP = xfs ]]; then
> + # If we don't have a realtime device, set up a loop device on the test
> + # filesystem.
> + if [[ $USE_EXTERNAL != yes || -z $SCRATCH_RTDEV ]]; then
> + _require_test
> + loopsz="$((filesz + (1 << 26)))"
> + _require_fs_space "$TEST_DIR" $((loopsz / 1024))
> + $XFS_IO_PROG -c "truncate $loopsz" -f "$TEST_DIR/$seq"
> + loop="$(_create_loop_device "$TEST_DIR/$seq")"
> + USE_EXTERNAL=yes
> + SCRATCH_RTDEV="$loop"
> + fi
> + extra_options="$extra_options -bsize=$bs"
> + extra_options="$extra_options -r extsize=$((bs * rextsize))"
> + extra_options="$extra_options -d agsize=$(((maxextlen + 1) * bs / 2)),rtinherit=1"
> +fi
> +_scratch_mkfs $extra_options >>$seqres.full 2>&1
> +_scratch_mount
> +_require_fs_space "$SCRATCH_MNT" $((filesz / 1024))
> +
> +# Allocate maxextlen + 1 blocks. As long as the allocator does something sane,
> +# we should end up with two extents that look something like:
> +#
> +# u3.bmx[0-1] = [startoff,startblock,blockcount,extentflag]
> +# 0:[0,0,2097148,1]
> +# 1:[2097148,2097148,4,1]
> +#
> +# Extent 0 has blockcount = ALIGN_DOWN(maxextlen, rextsize). Extent 1 is
> +# adjacent and has blockcount = rextsize. Both are unwritten.
> +$XFS_IO_PROG -c "falloc 0 $filesz" -c fsync -f "$SCRATCH_MNT/file"
> +
> +# Write extent 0 + one block of extent 1. Our extents should end up like so:
> +#
> +# u3.bmx[0-1] = [startoff,startblock,blockcount,extentflag]
> +# 0:[0,0,2097149,0]
> +# 1:[2097149,2097149,3,1]
> +#
> +# Extent 0 is written and has blockcount = ALIGN_DOWN(maxextlen, rextsize) + 1,
> +# Extent 1 is adjacent, unwritten, and has blockcount = rextsize - 1 and
> +# startblock % rextsize = 1.
> +#
> +# The -b is just to speed things up (doing GBs of I/O in 4k chunks kind of
> +# sucks).
> +$XFS_IO_PROG -c "pwrite -b 1M -W 0 $(((maxextlen + 2 - rextsize) * bs))" \
> + "$SCRATCH_MNT/file" >> "$seqres.full"
> +
> +# Truncate the extents.
> +$XFS_IO_PROG -c "truncate 0" -c fsync "$SCRATCH_MNT/file"
> +
> +# We need to do this before the loop device gets torn down.
> +_scratch_unmount
> +_check_scratch_fs
> +
> +echo "Silence is golden"
> +status=0
> +exit
> diff --git a/tests/generic/589.out b/tests/generic/589.out
> new file mode 100644
> index 00000000..5ab6ab10
> --- /dev/null
> +++ b/tests/generic/589.out
> @@ -0,0 +1,2 @@
> +QA output created by 589
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index 87d7441c..be6f4a43 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -591,3 +591,4 @@
> 586 auto quick rw prealloc
> 587 auto quick rw prealloc
> 588 auto quick log clone
> +589 auto prealloc preallocrw dangerous
> --
> 2.24.0
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v3] generic: test truncating mixed written/unwritten XFS realtime extent
2019-12-03 23:51 [PATCH v3] generic: test truncating mixed written/unwritten XFS realtime extent Omar Sandoval
2019-12-04 0:12 ` Darrick J. Wong
@ 2019-12-08 15:43 ` Eryu Guan
2019-12-08 17:04 ` Eryu Guan
2 siblings, 0 replies; 4+ messages in thread
From: Eryu Guan @ 2019-12-08 15:43 UTC (permalink / raw)
To: Omar Sandoval; +Cc: fstests, kernel-team, linux-xfs
On Tue, Dec 03, 2019 at 03:51:52PM -0800, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
>
> The only XFS-specific part of this test is the setup, so we can make the
> rest a generic test. It's slow, though, as it needs to write 8GB to
> convert a big unwritten extent to written.
>
> Signed-off-by: Omar Sandoval <osandov@fb.com>
> ---
> Changes from v2 -> v3:
>
> - Use _create_loop_device and _destroy_loop_device instead of losetup
>
> Changes from v1 -> v2:
>
> - If rtdev is not configured, fall back to loop device on test
> filesystem
> - Use XFS_IO_PROG instead of fallocate/sync/dd
> - Use truncate instead of rm
> - Add comments explaining the steps
>
> tests/generic/589 | 100 ++++++++++++++++++++++++++++++++++++++++++
> tests/generic/589.out | 2 +
> tests/generic/group | 1 +
> 3 files changed, 103 insertions(+)
> create mode 100755 tests/generic/589
> create mode 100644 tests/generic/589.out
>
> diff --git a/tests/generic/589 b/tests/generic/589
> new file mode 100755
> index 00000000..aab37bb4
> --- /dev/null
> +++ b/tests/generic/589
> @@ -0,0 +1,100 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2019 Facebook. All Rights Reserved.
> +#
> +# FS QA Test 589
> +#
> +# Test "xfs: fix realtime file data space leak" and "xfs: don't check for AG
> +# deadlock for realtime files in bunmapi". On XFS without the fix, truncate
> +# will hang forever. On other filesystems, this just tests writing into big
> +# fallocates.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> + test -n "$loop" && _destroy_loop_device "$loop"
> + rm -f "$TEST_DIR/$seq"
> +}
> +
> +. ./common/rc
> +. ./common/filter
> +
> +rm -f $seqres.full
> +
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch_nocheck
> +
> +maxextlen=$((0x1fffff))
> +bs=4096
> +rextsize=4
> +filesz=$(((maxextlen + 1) * bs))
> +
> +extra_options=""
> +# If we're testing XFS, set up the realtime device to reproduce the bug.
> +if [[ $FSTYP = xfs ]]; then
> + # If we don't have a realtime device, set up a loop device on the test
> + # filesystem.
> + if [[ $USE_EXTERNAL != yes || -z $SCRATCH_RTDEV ]]; then
> + _require_test
> + loopsz="$((filesz + (1 << 26)))"
> + _require_fs_space "$TEST_DIR" $((loopsz / 1024))
> + $XFS_IO_PROG -c "truncate $loopsz" -f "$TEST_DIR/$seq"
> + loop="$(_create_loop_device "$TEST_DIR/$seq")"
> + USE_EXTERNAL=yes
> + SCRATCH_RTDEV="$loop"
> + fi
> + extra_options="$extra_options -bsize=$bs"
> + extra_options="$extra_options -r extsize=$((bs * rextsize))"
> + extra_options="$extra_options -d agsize=$(((maxextlen + 1) * bs / 2)),rtinherit=1"
reflink is enabled by default since xfsprogs v5.1.0, and this causes
_scratch_mkfs to fail as "reflink not supported with realtime devices",
so I added a check here
if _scratch_mkfs_xfs_supported -m reflink=0 >/dev/null 2>&1; then
extra_options="$extra_options -m reflink=0"
fi
Thanks,
Eryu
> +_scratch_mount
> +_require_fs_space "$SCRATCH_MNT" $((filesz / 1024))
> +
> +# Allocate maxextlen + 1 blocks. As long as the allocator does something sane,
> +# we should end up with two extents that look something like:
> +#
> +# u3.bmx[0-1] = [startoff,startblock,blockcount,extentflag]
> +# 0:[0,0,2097148,1]
> +# 1:[2097148,2097148,4,1]
> +#
> +# Extent 0 has blockcount = ALIGN_DOWN(maxextlen, rextsize). Extent 1 is
> +# adjacent and has blockcount = rextsize. Both are unwritten.
> +$XFS_IO_PROG -c "falloc 0 $filesz" -c fsync -f "$SCRATCH_MNT/file"
> +
> +# Write extent 0 + one block of extent 1. Our extents should end up like so:
> +#
> +# u3.bmx[0-1] = [startoff,startblock,blockcount,extentflag]
> +# 0:[0,0,2097149,0]
> +# 1:[2097149,2097149,3,1]
> +#
> +# Extent 0 is written and has blockcount = ALIGN_DOWN(maxextlen, rextsize) + 1,
> +# Extent 1 is adjacent, unwritten, and has blockcount = rextsize - 1 and
> +# startblock % rextsize = 1.
> +#
> +# The -b is just to speed things up (doing GBs of I/O in 4k chunks kind of
> +# sucks).
> +$XFS_IO_PROG -c "pwrite -b 1M -W 0 $(((maxextlen + 2 - rextsize) * bs))" \
> + "$SCRATCH_MNT/file" >> "$seqres.full"
> +
> +# Truncate the extents.
> +$XFS_IO_PROG -c "truncate 0" -c fsync "$SCRATCH_MNT/file"
> +
> +# We need to do this before the loop device gets torn down.
> +_scratch_unmount
> +_check_scratch_fs
> +
> +echo "Silence is golden"
> +status=0
> +exit
> diff --git a/tests/generic/589.out b/tests/generic/589.out
> new file mode 100644
> index 00000000..5ab6ab10
> --- /dev/null
> +++ b/tests/generic/589.out
> @@ -0,0 +1,2 @@
> +QA output created by 589
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index 87d7441c..be6f4a43 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -591,3 +591,4 @@
> 586 auto quick rw prealloc
> 587 auto quick rw prealloc
> 588 auto quick log clone
> +589 auto prealloc preallocrw dangerous
> --
> 2.24.0
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v3] generic: test truncating mixed written/unwritten XFS realtime extent
2019-12-03 23:51 [PATCH v3] generic: test truncating mixed written/unwritten XFS realtime extent Omar Sandoval
2019-12-04 0:12 ` Darrick J. Wong
2019-12-08 15:43 ` Eryu Guan
@ 2019-12-08 17:04 ` Eryu Guan
2 siblings, 0 replies; 4+ messages in thread
From: Eryu Guan @ 2019-12-08 17:04 UTC (permalink / raw)
To: Omar Sandoval; +Cc: fstests, kernel-team, linux-xfs
On Tue, Dec 03, 2019 at 03:51:52PM -0800, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
>
> The only XFS-specific part of this test is the setup, so we can make the
> rest a generic test. It's slow, though, as it needs to write 8GB to
> convert a big unwritten extent to written.
>
> Signed-off-by: Omar Sandoval <osandov@fb.com>
> ---
> Changes from v2 -> v3:
>
> - Use _create_loop_device and _destroy_loop_device instead of losetup
>
> Changes from v1 -> v2:
>
> - If rtdev is not configured, fall back to loop device on test
> filesystem
> - Use XFS_IO_PROG instead of fallocate/sync/dd
> - Use truncate instead of rm
> - Add comments explaining the steps
>
> tests/generic/589 | 100 ++++++++++++++++++++++++++++++++++++++++++
> tests/generic/589.out | 2 +
> tests/generic/group | 1 +
> 3 files changed, 103 insertions(+)
> create mode 100755 tests/generic/589
> create mode 100644 tests/generic/589.out
>
> diff --git a/tests/generic/589 b/tests/generic/589
> new file mode 100755
> index 00000000..aab37bb4
> --- /dev/null
> +++ b/tests/generic/589
> @@ -0,0 +1,100 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2019 Facebook. All Rights Reserved.
> +#
> +# FS QA Test 589
> +#
> +# Test "xfs: fix realtime file data space leak" and "xfs: don't check for AG
> +# deadlock for realtime files in bunmapi". On XFS without the fix, truncate
> +# will hang forever. On other filesystems, this just tests writing into big
> +# fallocates.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> + test -n "$loop" && _destroy_loop_device "$loop"
> + rm -f "$TEST_DIR/$seq"
> +}
> +
> +. ./common/rc
> +. ./common/filter
> +
> +rm -f $seqres.full
> +
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch_nocheck
> +
> +maxextlen=$((0x1fffff))
> +bs=4096
> +rextsize=4
> +filesz=$(((maxextlen + 1) * bs))
> +
> +extra_options=""
> +# If we're testing XFS, set up the realtime device to reproduce the bug.
> +if [[ $FSTYP = xfs ]]; then
> + # If we don't have a realtime device, set up a loop device on the test
> + # filesystem.
> + if [[ $USE_EXTERNAL != yes || -z $SCRATCH_RTDEV ]]; then
> + _require_test
> + loopsz="$((filesz + (1 << 26)))"
> + _require_fs_space "$TEST_DIR" $((loopsz / 1024))
> + $XFS_IO_PROG -c "truncate $loopsz" -f "$TEST_DIR/$seq"
> + loop="$(_create_loop_device "$TEST_DIR/$seq")"
> + USE_EXTERNAL=yes
> + SCRATCH_RTDEV="$loop"
> + fi
> + extra_options="$extra_options -bsize=$bs"
> + extra_options="$extra_options -r extsize=$((bs * rextsize))"
> + extra_options="$extra_options -d agsize=$(((maxextlen + 1) * bs / 2)),rtinherit=1"
> +fi
> +_scratch_mkfs $extra_options >>$seqres.full 2>&1
> +_scratch_mount
> +_require_fs_space "$SCRATCH_MNT" $((filesz / 1024))
> +
> +# Allocate maxextlen + 1 blocks. As long as the allocator does something sane,
> +# we should end up with two extents that look something like:
> +#
> +# u3.bmx[0-1] = [startoff,startblock,blockcount,extentflag]
> +# 0:[0,0,2097148,1]
> +# 1:[2097148,2097148,4,1]
> +#
> +# Extent 0 has blockcount = ALIGN_DOWN(maxextlen, rextsize). Extent 1 is
> +# adjacent and has blockcount = rextsize. Both are unwritten.
> +$XFS_IO_PROG -c "falloc 0 $filesz" -c fsync -f "$SCRATCH_MNT/file"
> +
> +# Write extent 0 + one block of extent 1. Our extents should end up like so:
> +#
> +# u3.bmx[0-1] = [startoff,startblock,blockcount,extentflag]
> +# 0:[0,0,2097149,0]
> +# 1:[2097149,2097149,3,1]
> +#
> +# Extent 0 is written and has blockcount = ALIGN_DOWN(maxextlen, rextsize) + 1,
> +# Extent 1 is adjacent, unwritten, and has blockcount = rextsize - 1 and
> +# startblock % rextsize = 1.
> +#
> +# The -b is just to speed things up (doing GBs of I/O in 4k chunks kind of
> +# sucks).
> +$XFS_IO_PROG -c "pwrite -b 1M -W 0 $(((maxextlen + 2 - rextsize) * bs))" \
> + "$SCRATCH_MNT/file" >> "$seqres.full"
> +
> +# Truncate the extents.
> +$XFS_IO_PROG -c "truncate 0" -c fsync "$SCRATCH_MNT/file"
> +
> +# We need to do this before the loop device gets torn down.
> +_scratch_unmount
> +_check_scratch_fs
> +
> +echo "Silence is golden"
> +status=0
> +exit
> diff --git a/tests/generic/589.out b/tests/generic/589.out
> new file mode 100644
> index 00000000..5ab6ab10
> --- /dev/null
> +++ b/tests/generic/589.out
> @@ -0,0 +1,2 @@
> +QA output created by 589
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index 87d7441c..be6f4a43 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -591,3 +591,4 @@
> 586 auto quick rw prealloc
> 587 auto quick rw prealloc
> 588 auto quick log clone
> +589 auto prealloc preallocrw dangerous
Also, I noticed that the fixes are already in latest upstream kernel, so
I removed dangerous group as well.
Thanks,
Eryu
> --
> 2.24.0
>
^ permalink raw reply [flat|nested] 4+ messages in thread