[PATCH 0/3] fstests: btrfs validate sysfs fsid and usage

[PATCH 0/3] fstests: btrfs validate sysfs fsid and usage

[Anand Jain]

Patch 1 adds a helper to check if the kernel is latest enough to provide
the fsid from the sysfs.
Patch 2 validates the sysfs fsid against the fsid from the sb
Patch 3 is a test case to make sure if the btrfs filesystem usage is
successful on a sprout filesystem with a seed device missing.

Anand Jain (3):
  common/btrfs: add _require_btrfs_sysfs_fsid helper
  btrfs/248: validate sysfs fsid
  btrfs/249: test btrfs filesystem usage command on missing seed device

 common/btrfs        | 14 ++++++++++
 tests/btrfs/248     | 66 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/248.out |  2 ++
 tests/btrfs/249     | 67 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/249.out |  2 ++
 5 files changed, 151 insertions(+)
 create mode 100755 tests/btrfs/248
 create mode 100644 tests/btrfs/248.out
 create mode 100755 tests/btrfs/249
 create mode 100644 tests/btrfs/249.out

-- 
2.27.0

[PATCH 1/3] common/btrfs: add _require_btrfs_sysfs_fsid helper

[Anand Jain]

It checks if the kernel has the following patch
 btrfs: sysfs add devinfo/fsid to retrieve fsid from the device
that added sysfs interface to get fsid.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 common/btrfs | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/common/btrfs b/common/btrfs
index ac880bddf524..5d938c19b56a 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -445,3 +445,17 @@ _scratch_btrfs_is_zoned()
 	[ `_zone_type ${SCRATCH_DEV}` != "none" ] && return 0
 	return 1
 }
+
+_require_btrfs_sysfs_fsid()
+{
+	local fsid
+
+	fsid=$($BTRFS_UTIL_PROG filesystem show $TEST_DIR |grep uuid: |\
+	       awk '{print $NF}')
+
+	# Check if the kernel has sysfs fsid support.
+	# Following kernel patch adds it:
+	#   btrfs: sysfs add devinfo/fsid to retrieve fsid from the device
+	test -f /sys/fs/btrfs/$fsid/devinfo/1/fsid ||\
+		_notrun "Need btrfs sysfs fsid support"
+}
-- 
2.27.0

[PATCH 2/3] btrfs/248: validate sysfs fsid

[Anand Jain]

Validate if the sysfs fsid is the same as fsid as obtained from the
superblock.

Suggested-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tests/btrfs/248     | 66 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/248.out |  2 ++
 2 files changed, 68 insertions(+)
 create mode 100755 tests/btrfs/248
 create mode 100644 tests/btrfs/248.out

diff --git a/tests/btrfs/248 b/tests/btrfs/248
new file mode 100755
index 000000000000..856cd489d4e4
--- /dev/null
+++ b/tests/btrfs/248
@@ -0,0 +1,66 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Anand Jain.  All Rights Reserved.
+# Copyright (c) 2021 Oracle.  All Rights Reserved.
+#
+# FS QA Test 248
+#
+# Validate if the sysfs devinfo/<devid>/fsid behaves properly
+# Steps:
+#  Create a sprout filesystem (an rw device on top of a seed device)
+#  Read the seed and sprout devices fsid from the superblock
+#  Validate with the sysfs fsid
+
+. ./common/preamble
+_begin_fstest auto quick seed volume
+
+# Import common functions.
+# . ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_test
+_require_scratch_dev_pool 2
+_require_btrfs_forget_or_module_loadable
+_require_command "$BTRFS_TUNE_PROG" btrfstune
+_require_btrfs_command inspect-internal dump-super
+_require_btrfs_sysfs_fsid
+
+_scratch_dev_pool_get 1
+# use the scratch device as a seed device
+seed_dev=$(echo $SCRATCH_DEV_POOL | $AWK_PROG '{ print $1 }')
+
+# use the spare device as a sprout device
+_spare_dev_get
+
+# create seed device
+_scratch_pool_mkfs >> $seqres.full 2>&1
+$BTRFS_TUNE_PROG -S 1 $seed_dev
+_scratch_mount >> $seqres.full 2>&1
+
+# create a sprout device
+$BTRFS_UTIL_PROG device add -f $SPARE_DEV $SCRATCH_MNT >> $seqres.full 2>&1
+_scratch_unmount
+
+# record the fsid of both seed and sprout devices
+seedfsid=$($BTRFS_UTIL_PROG inspect-internal dump-super $seed_dev |grep ^fsid |\
+	   $AWK_PROG '{print $2}')
+sproutfsid=$($BTRFS_UTIL_PROG inspect-internal dump-super $SPARE_DEV |\
+	     grep ^fsid |$AWK_PROG '{print $2}')
+
+# validate it with the fsid as shown in the sysfs
+_mount -o device=$seed_dev $SPARE_DEV $SCRATCH_MNT
+
+cat /sys/fs/btrfs/$sproutfsid/devinfo/2/fsid | grep -v $sproutfsid
+cat /sys/fs/btrfs/$sproutfsid/devinfo/1/fsid | grep -v $seedfsid
+
+_scratch_unmount
+_spare_dev_put
+_scratch_dev_pool_put
+
+echo "Silence is golden"
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/248.out b/tests/btrfs/248.out
new file mode 100644
index 000000000000..58af9173bea3
--- /dev/null
+++ b/tests/btrfs/248.out
@@ -0,0 +1,2 @@
+QA output created by 248
+Silence is golden
-- 
2.27.0

[PATCH 3/3] btrfs/249: test btrfs filesystem usage command on missing seed device

[Anand Jain]

If there is a missing seed device in a sprout, the btrfs filesystem usage
command fails, which is fixed by the following patches:

  btrfs: sysfs add devinfo/fsid to retrieve fsid from the device
  btrfs-progs: read fsid from the sysfs in device_is_seed

Test if it works now after these patches in the kernel and in the
btrfs-progs respectively.

Suggested-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tests/btrfs/249     | 67 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/249.out |  2 ++
 2 files changed, 69 insertions(+)
 create mode 100755 tests/btrfs/249
 create mode 100644 tests/btrfs/249.out

diff --git a/tests/btrfs/249 b/tests/btrfs/249
new file mode 100755
index 000000000000..f8f2f07052c6
--- /dev/null
+++ b/tests/btrfs/249
@@ -0,0 +1,67 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Anand Jain.  All Rights Reserved.
+# Copyright (c) 2021 Oracle.  All Rights Reserved.
+#
+# FS QA Test 249
+#
+# Validate if the command 'btrfs filesystem usage' works with missing seed
+# device
+# Steps:
+#  Create a degraded raid1 seed device
+#  Create a sprout filesystem (an rw device on top of a seed device)
+#  Dump 'btrfs filesystem usage', check it didn't fail
+#
+# Tests btrfs-progs bug fixed by the kernel patch and a btrfs-prog patch
+#   btrfs: sysfs add devinfo/fsid to retrieve fsid from the device
+#   btrfs-progs: read fsid from the sysfs in device_is_seed
+
+. ./common/preamble
+_begin_fstest auto quick seed volume
+
+# Import common functions.
+# . ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch_dev_pool 3
+_require_command "$WIPEFS_PROG" wipefs
+_require_btrfs_forget_or_module_loadable
+
+_scratch_dev_pool_get 2
+# use the scratch devices as seed devices
+seed_dev1=$(echo $SCRATCH_DEV_POOL | $AWK_PROG '{ print $1 }')
+seed_dev2=$(echo $SCRATCH_DEV_POOL | $AWK_PROG '{ print $2 }')
+
+# use the spare device as a sprout device
+_spare_dev_get
+sprout_dev=$SPARE_DEV
+
+# create raid1 seed filesystem
+_scratch_pool_mkfs "-draid1 -mraid1" >> $seqres.full 2>&1
+$BTRFS_TUNE_PROG -S 1 $seed_dev1
+$WIPEFS_PROG -a $seed_dev1 >> $seqres.full 2>&1
+_btrfs_forget_or_module_reload
+_mount -o degraded $seed_dev2 $SCRATCH_MNT >> $seqres.full 2>&1
+
+# create a sprout device
+$BTRFS_UTIL_PROG device add -f $SPARE_DEV $SCRATCH_MNT >> $seqres.full 2>&1
+
+# dump filesystem usage if it fails error goes to the bad.out file
+$BTRFS_UTIL_PROG filesystem usage $SCRATCH_MNT >> $seqres.full
+# also check for the error code
+ret=$?
+if [ $ret != 0 ]; then
+	_fail "FAILED: btrfs filesystem usage, ret $ret"
+fi
+
+_scratch_unmount
+_spare_dev_put
+_scratch_dev_pool_put
+
+echo "Silence is golden"
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/249.out b/tests/btrfs/249.out
new file mode 100644
index 000000000000..b79a5dca8820
--- /dev/null
+++ b/tests/btrfs/249.out
@@ -0,0 +1,2 @@
+QA output created by 249
+Silence is golden
-- 
2.27.0

Re: [PATCH 3/3] btrfs/249: test btrfs filesystem usage command on missing seed device

[Josef Bacik]

On Wed, Oct 20, 2021 at 03:16:44PM +0800, Anand Jain wrote:
> If there is a missing seed device in a sprout, the btrfs filesystem usage
> command fails, which is fixed by the following patches:
> 
>   btrfs: sysfs add devinfo/fsid to retrieve fsid from the device
>   btrfs-progs: read fsid from the sysfs in device_is_seed
> 
> Test if it works now after these patches in the kernel and in the
> btrfs-progs respectively.
> 
> Suggested-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Anand Jain <anand.jain@oracle.com>

Shouldn't this use


as well?  I wish there was a way to detect that btrfs-progs had support for
reading it but I suppose this is a good enough gate.  Maybe add a


> ---
>  tests/btrfs/249     | 67 +++++++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/249.out |  2 ++
>  2 files changed, 69 insertions(+)
>  create mode 100755 tests/btrfs/249
>  create mode 100644 tests/btrfs/249.out
> 
> diff --git a/tests/btrfs/249 b/tests/btrfs/249
> new file mode 100755
> index 000000000000..f8f2f07052c6
> --- /dev/null
> +++ b/tests/btrfs/249
> @@ -0,0 +1,67 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2021 Anand Jain.  All Rights Reserved.
> +# Copyright (c) 2021 Oracle.  All Rights Reserved.
> +#
> +# FS QA Test 249
> +#
> +# Validate if the command 'btrfs filesystem usage' works with missing seed
> +# device
> +# Steps:
> +#  Create a degraded raid1 seed device
> +#  Create a sprout filesystem (an rw device on top of a seed device)
> +#  Dump 'btrfs filesystem usage', check it didn't fail
> +#
> +# Tests btrfs-progs bug fixed by the kernel patch and a btrfs-prog patch
> +#   btrfs: sysfs add devinfo/fsid to retrieve fsid from the device
> +#   btrfs-progs: read fsid from the sysfs in device_is_seed
> +
> +. ./common/preamble
> +_begin_fstest auto quick seed volume
> +
> +# Import common functions.
> +# . ./common/filter
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_require_scratch_dev_pool 3
> +_require_command "$WIPEFS_PROG" wipefs
> +_require_btrfs_forget_or_module_loadable

Need

_require_btrfs_sysfs_fsid

here I think.

> +
> +_scratch_dev_pool_get 2
> +# use the scratch devices as seed devices
> +seed_dev1=$(echo $SCRATCH_DEV_POOL | $AWK_PROG '{ print $1 }')
> +seed_dev2=$(echo $SCRATCH_DEV_POOL | $AWK_PROG '{ print $2 }')
> +
> +# use the spare device as a sprout device
> +_spare_dev_get
> +sprout_dev=$SPARE_DEV
> +
> +# create raid1 seed filesystem
> +_scratch_pool_mkfs "-draid1 -mraid1" >> $seqres.full 2>&1
> +$BTRFS_TUNE_PROG -S 1 $seed_dev1
> +$WIPEFS_PROG -a $seed_dev1 >> $seqres.full 2>&1
> +_btrfs_forget_or_module_reload
> +_mount -o degraded $seed_dev2 $SCRATCH_MNT >> $seqres.full 2>&1
> +
> +# create a sprout device
> +$BTRFS_UTIL_PROG device add -f $SPARE_DEV $SCRATCH_MNT >> $seqres.full 2>&1
> +
> +# dump filesystem usage if it fails error goes to the bad.out file
> +$BTRFS_UTIL_PROG filesystem usage $SCRATCH_MNT >> $seqres.full
> +# also check for the error code
> +ret=$?
> +if [ $ret != 0 ]; then
> +	_fail "FAILED: btrfs filesystem usage, ret $ret"

Can you add "check your btrfs-progs version" here or something?  In case I'm an
idiot and forget to update btrfs-progs on the overnight xfstests boxes?  Thanks,

Josef

Re: [PATCH 3/3] btrfs/249: test btrfs filesystem usage command on missing seed device

[Anand Jain]

On 20/10/2021 21:53, Josef Bacik wrote:
> On Wed, Oct 20, 2021 at 03:16:44PM +0800, Anand Jain wrote:
>> If there is a missing seed device in a sprout, the btrfs filesystem usage
>> command fails, which is fixed by the following patches:
>>
>>    btrfs: sysfs add devinfo/fsid to retrieve fsid from the device
>>    btrfs-progs: read fsid from the sysfs in device_is_seed
>>
>> Test if it works now after these patches in the kernel and in the
>> btrfs-progs respectively.
>>
>> Suggested-by: Josef Bacik <josef@toxicpanda.com>
>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> 
> Shouldn't this use
> 
> 
> as well?  I wish there was a way to detect that btrfs-progs had support for
> reading it but I suppose this is a good enough gate.  Maybe add a

  _require_btrfs_sysfs_fsid ?

The problem is about the nonexistence of the sysfs interface to read 
fsid. Adding it will fail to reproduce the problem.

So if there is no sysfs interface, then btrfs-progs will fail back to 
the read-sb method, which shall expose it to fail.

> 
> 
>> ---
>>   tests/btrfs/249     | 67 +++++++++++++++++++++++++++++++++++++++++++++
>>   tests/btrfs/249.out |  2 ++
>>   2 files changed, 69 insertions(+)
>>   create mode 100755 tests/btrfs/249
>>   create mode 100644 tests/btrfs/249.out
>>
>> diff --git a/tests/btrfs/249 b/tests/btrfs/249
>> new file mode 100755
>> index 000000000000..f8f2f07052c6
>> --- /dev/null
>> +++ b/tests/btrfs/249
>> @@ -0,0 +1,67 @@
>> +#! /bin/bash
>> +# SPDX-License-Identifier: GPL-2.0
>> +# Copyright (c) 2021 Anand Jain.  All Rights Reserved.
>> +# Copyright (c) 2021 Oracle.  All Rights Reserved.
>> +#
>> +# FS QA Test 249
>> +#
>> +# Validate if the command 'btrfs filesystem usage' works with missing seed
>> +# device
>> +# Steps:
>> +#  Create a degraded raid1 seed device
>> +#  Create a sprout filesystem (an rw device on top of a seed device)
>> +#  Dump 'btrfs filesystem usage', check it didn't fail
>> +#
>> +# Tests btrfs-progs bug fixed by the kernel patch and a btrfs-prog patch
>> +#   btrfs: sysfs add devinfo/fsid to retrieve fsid from the device
>> +#   btrfs-progs: read fsid from the sysfs in device_is_seed
>> +
>> +. ./common/preamble
>> +_begin_fstest auto quick seed volume
>> +
>> +# Import common functions.
>> +# . ./common/filter
>> +
>> +# real QA test starts here
>> +
>> +# Modify as appropriate.
>> +_supported_fs btrfs
>> +_require_scratch_dev_pool 3
>> +_require_command "$WIPEFS_PROG" wipefs
>> +_require_btrfs_forget_or_module_loadable
> 
> Need
> 
> _require_btrfs_sysfs_fsid
> 
> here I think.

  As above.

> 
>> +
>> +_scratch_dev_pool_get 2
>> +# use the scratch devices as seed devices
>> +seed_dev1=$(echo $SCRATCH_DEV_POOL | $AWK_PROG '{ print $1 }')
>> +seed_dev2=$(echo $SCRATCH_DEV_POOL | $AWK_PROG '{ print $2 }')
>> +
>> +# use the spare device as a sprout device
>> +_spare_dev_get
>> +sprout_dev=$SPARE_DEV
>> +
>> +# create raid1 seed filesystem
>> +_scratch_pool_mkfs "-draid1 -mraid1" >> $seqres.full 2>&1
>> +$BTRFS_TUNE_PROG -S 1 $seed_dev1
>> +$WIPEFS_PROG -a $seed_dev1 >> $seqres.full 2>&1
>> +_btrfs_forget_or_module_reload
>> +_mount -o degraded $seed_dev2 $SCRATCH_MNT >> $seqres.full 2>&1
>> +
>> +# create a sprout device
>> +$BTRFS_UTIL_PROG device add -f $SPARE_DEV $SCRATCH_MNT >> $seqres.full 2>&1
>> +
>> +# dump filesystem usage if it fails error goes to the bad.out file
>> +$BTRFS_UTIL_PROG filesystem usage $SCRATCH_MNT >> $seqres.full
>> +# also check for the error code
>> +ret=$?
>> +if [ $ret != 0 ]; then
>> +	_fail "FAILED: btrfs filesystem usage, ret $ret"
> 
> Can you add "check your btrfs-progs version" here or something?  In case I'm an
> idiot and forget to update btrfs-progs on the overnight xfstests boxes?  Thanks,
> 

  Sure.

Thanks, Anand


> Josef
> 

Re: [PATCH 3/3] btrfs/249: test btrfs filesystem usage command on missing seed device

[Josef Bacik]

On Wed, Oct 20, 2021 at 10:58:56PM +0800, Anand Jain wrote:
> 
> 
> On 20/10/2021 21:53, Josef Bacik wrote:
> > On Wed, Oct 20, 2021 at 03:16:44PM +0800, Anand Jain wrote:
> > > If there is a missing seed device in a sprout, the btrfs filesystem usage
> > > command fails, which is fixed by the following patches:
> > > 
> > >    btrfs: sysfs add devinfo/fsid to retrieve fsid from the device
> > >    btrfs-progs: read fsid from the sysfs in device_is_seed
> > > 
> > > Test if it works now after these patches in the kernel and in the
> > > btrfs-progs respectively.
> > > 
> > > Suggested-by: Josef Bacik <josef@toxicpanda.com>
> > > Signed-off-by: Anand Jain <anand.jain@oracle.com>
> > 
> > Shouldn't this use
> > 
> > 
> > as well?  I wish there was a way to detect that btrfs-progs had support for
> > reading it but I suppose this is a good enough gate.  Maybe add a
> 
>  _require_btrfs_sysfs_fsid ?
> 
> The problem is about the nonexistence of the sysfs interface to read fsid.
> Adding it will fail to reproduce the problem.
> 
> So if there is no sysfs interface, then btrfs-progs will fail back to the
> read-sb method, which shall expose it to fail.
> 

Sigh sorry, I typed this out and then decided I wanted to comment below, and
forgot to delete this bit.

Yeah sorry I'm thinking about it in terms of the CI testing for us, we'll start
failing this test once its merged without the fix merged, but I guess that's
kinda what we want.  Let's just leave this as it is.  Thanks,

Josef

Re: [PATCH 0/3] fstests: btrfs validate sysfs fsid and usage

[Josef Bacik]

On Wed, Oct 20, 2021 at 03:16:41PM +0800, Anand Jain wrote:
> Patch 1 adds a helper to check if the kernel is latest enough to provide
> the fsid from the sysfs.
> Patch 2 validates the sysfs fsid against the fsid from the sb
> Patch 3 is a test case to make sure if the btrfs filesystem usage is
> successful on a sprout filesystem with a seed device missing.
> 
> Anand Jain (3):
>   common/btrfs: add _require_btrfs_sysfs_fsid helper
>   btrfs/248: validate sysfs fsid
>   btrfs/249: test btrfs filesystem usage command on missing seed device
> 

You can add

Reviewed-by: Josef Bacik <josef@toxicpanda.com>

Thanks,

Josef