[PATCH] xfs: test xfs_metadump for leaked strings

[PATCH] xfs: test xfs_metadump for leaked strings

[Eric Sandeen]

xfs_metadump by default sanitizes the image so that all strings
longer than 5 chars are obfusccated, and all stale data in metadata
blocks (i.e. unused/unwritten data) is zeroed out.  We didn't have
a test for this, though, so this does it.

It patterns 256M of the scratch device, then uses djwong's
populate infrastructure to write all types of metadata,
metadumps & mdrestores it, then looks for either the leaked
pre-pattern or any leaked strings or filenames.

The strings we look for are, unfortunately, a bit ad-hoc based on
what is currently used in the populate routines.

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
---

this exposes a bug in xfs_metadump, sending a fix for that soon


diff --git a/tests/xfs/425 b/tests/xfs/425
new file mode 100755
index 0000000..02cb60f
--- /dev/null
+++ b/tests/xfs/425
@@ -0,0 +1,121 @@
+#! /bin/bash
+# FS QA Test 425
+#
+# Look for stale data leaks in an xfs_metadump
+#
+# If this fails, get the byte offsets of the leaked strings
+# which are found, then on the restored image in $SCRATCH_MNT,
+# do:
+#
+# xfs_db> blockget -n
+# xfs_db> convert byte $BYTE daddr
+# $RESULT
+# xfs_db> daddr $RESULT
+# xfs_db> blockuse -n
+#
+# to see information about the metadata block which contains the
+# leaked string in question
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/populate
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs xfs
+_supported_os Linux
+_require_test
+_require_scratch
+_require_populate_commands
+
+METADUMP_FILE="${TEST_DIR}/${seq}_metadump"
+MDRESTORE_FILE="${TEST_DIR}/${seq}_mdrestore"
+
+echo "Silence is golden"
+
+# Pattern the scratch disk, mkfs, and restore.
+$XFS_IO_PROG -d -c "pwrite 0 256M" $SCRATCH_DEV > $seqres.full 2>&1
+_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_populate nofill >> $seqres.full 2>&1
+
+# populate unmounts the fs for us
+
+_scratch_metadump $METADUMP_FILE 
+xfs_mdrestore $METADUMP_FILE $MDRESTORE_FILE
+
+# Grep for stale data (leaked cd cd pattern) or strings
+# from populate routine
+
+hexdump -C $MDRESTORE_FILE | grep \
+"cd cd cd cd\| \
+41 41 41 41\|\
+42 42 42 42\|\
+43 43 43 43\|\
+44 44 44 44\|\
+61 61 61 61\|\
+62 62 62 62\|\
+63 63 63 63\|\
+64 64 64 64\|\
+dummy\|\
+S_IF\|\
+FMT_\|\
+INLINE\|\
+BLOCK\|\
+LEAF\|\
+NODE\|\
+BTREE\|\
+LOCAL\|\
+EXTENTS\|\
+REMOTE\|\
+ATTR\|\
+SYSTEM\|\
+TRUSTED\|\
+SECURITY\|\
+attrvalfile\|\
+unused\|\
+BNOBT\|\
+RMAPBT\|\
+RTRMAPBT\|\
+REFCOUNTBT" && echo "Leaked data found; see comments in test to debug"
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/group b/tests/xfs/group
index 185487d..d550224 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -422,3 +422,4 @@
 422 dangerous_scrub dangerous_online_repair
 423 dangerous_scrub
 424 auto quick dump
+425 auto,metadata

Re: [PATCH] xfs: test xfs_metadump for leaked strings

[Eryu Guan]

On Wed, Jul 26, 2017 at 03:33:20PM -0500, Eric Sandeen wrote:
> xfs_metadump by default sanitizes the image so that all strings
> longer than 5 chars are obfusccated, and all stale data in metadata
> blocks (i.e. unused/unwritten data) is zeroed out.  We didn't have
> a test for this, though, so this does it.
> 
> It patterns 256M of the scratch device, then uses djwong's
> populate infrastructure to write all types of metadata,
> metadumps & mdrestores it, then looks for either the leaked
> pre-pattern or any leaked strings or filenames.
> 
> The strings we look for are, unfortunately, a bit ad-hoc based on
> what is currently used in the populate routines.

I don't have a better idea to avoid this either..

Some minor issues below.

> 
> Signed-off-by: Eric Sandeen <sandeen@redhat.com>
> ---
> 
> this exposes a bug in xfs_metadump, sending a fix for that soon
> 
> 
> diff --git a/tests/xfs/425 b/tests/xfs/425
> new file mode 100755
> index 0000000..02cb60f
> --- /dev/null
> +++ b/tests/xfs/425
> @@ -0,0 +1,121 @@
> +#! /bin/bash
> +# FS QA Test 425
> +#
> +# Look for stale data leaks in an xfs_metadump
> +#
> +# If this fails, get the byte offsets of the leaked strings
> +# which are found, then on the restored image in $SCRATCH_MNT,
                                                     ^^^^^^^^^^ TEST_DIR?
> +# do:
> +#
> +# xfs_db> blockget -n
> +# xfs_db> convert byte $BYTE daddr
> +# $RESULT
> +# xfs_db> daddr $RESULT
> +# xfs_db> blockuse -n
> +#
> +# to see information about the metadata block which contains the
> +# leaked string in question
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Red Hat, Inc.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/populate
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs xfs
> +_supported_os Linux
> +_require_test
> +_require_scratch
> +_require_populate_commands
> +
> +METADUMP_FILE="${TEST_DIR}/${seq}_metadump"
> +MDRESTORE_FILE="${TEST_DIR}/${seq}_mdrestore"
> +
> +echo "Silence is golden"
> +
> +# Pattern the scratch disk, mkfs, and restore.
> +$XFS_IO_PROG -d -c "pwrite 0 256M" $SCRATCH_DEV > $seqres.full 2>&1
> +_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
> +_scratch_populate nofill >> $seqres.full 2>&1
> +
> +# populate unmounts the fs for us
> +
> +_scratch_metadump $METADUMP_FILE 

Trailing whitespace in above line.

> +xfs_mdrestore $METADUMP_FILE $MDRESTORE_FILE
> +
> +# Grep for stale data (leaked cd cd pattern) or strings
> +# from populate routine
> +
> +hexdump -C $MDRESTORE_FILE | grep \
> +"cd cd cd cd\| \
> +41 41 41 41\|\
> +42 42 42 42\|\
> +43 43 43 43\|\
> +44 44 44 44\|\
> +61 61 61 61\|\
> +62 62 62 62\|\
> +63 63 63 63\|\
> +64 64 64 64\|\
> +dummy\|\
> +S_IF\|\
> +FMT_\|\
> +INLINE\|\
> +BLOCK\|\
> +LEAF\|\
> +NODE\|\
> +BTREE\|\
> +LOCAL\|\
> +EXTENTS\|\
> +REMOTE\|\
> +ATTR\|\
> +SYSTEM\|\
> +TRUSTED\|\
> +SECURITY\|\
> +attrvalfile\|\
> +unused\|\
> +BNOBT\|\
> +RMAPBT\|\
> +RTRMAPBT\|\
> +REFCOUNTBT" && echo "Leaked data found; see comments in test to debug"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/xfs/group b/tests/xfs/group
> index 185487d..d550224 100644
> --- a/tests/xfs/group
> +++ b/tests/xfs/group
> @@ -422,3 +422,4 @@
>  422 dangerous_scrub dangerous_online_repair
>  423 dangerous_scrub
>  424 auto quick dump
> +425 auto,metadata

A single space to separate different groups, not ','. And there's no
425.out file :)

Thanks,
Eryu

Re: [PATCH] xfs: test xfs_metadump for leaked strings

[Eric Sandeen]

On 7/27/17 5:52 AM, Eryu Guan wrote:
> On Wed, Jul 26, 2017 at 03:33:20PM -0500, Eric Sandeen wrote:
>> xfs_metadump by default sanitizes the image so that all strings
>> longer than 5 chars are obfusccated, and all stale data in metadata
>> blocks (i.e. unused/unwritten data) is zeroed out.  We didn't have
>> a test for this, though, so this does it.
>>
>> It patterns 256M of the scratch device, then uses djwong's
>> populate infrastructure to write all types of metadata,
>> metadumps & mdrestores it, then looks for either the leaked
>> pre-pattern or any leaked strings or filenames.
>>
>> The strings we look for are, unfortunately, a bit ad-hoc based on
>> what is currently used in the populate routines.
> 
> I don't have a better idea to avoid this either..
> 
> Some minor issues below.

comment typo: will fix
whitespace: sigh, will fix ;)
groups: a patch to new script to check for this might be nice ;) will fix.
out file: Argh, always forget to add that to git

will resend, thanks.

-Eric

[PATCH V2] xfs: test xfs_metadump for leaked strings

[Eric Sandeen]

xfs_metadump by default sanitizes the image so that all strings
longer than 5 chars are obfusccated, and all stale data in metadata
blocks (i.e. unused/unwritten data) is zeroed out.  We didn't have
a test for this, though, so this does it.

It patterns 256M of the scratch device, then uses djwong's
populate infrastructure to write all types of metadata,
metadumps & mdrestores it, then looks for either the leaked
pre-pattern or any leaked strings or filenames.

The strings we look for are, unfortunately, a bit ad-hoc based on
what is currently used in the populate routines.

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
---

V2: include .out file, minor whitespace & typo edits

diff --git a/common/populate b/common/populate
index 498151f..725cad4 100644
--- a/common/populate
+++ b/common/populate
@@ -198,18 +198,20 @@ _scratch_xfs_populate() {
 	touch ${SCRATCH_MNT}/ATTR.SYSTEM
 	setfacl -m u:root:r ${SCRATCH_MNT}/ATTR.SYSTEM
 
+ATTRVALFILE="${SCRATCH_MNT}/attrvalfile"
+
 	# FMT_EXTENTS with a remote less-than-a-block value
 	echo "+ attr extents with a remote less-than-a-block value"
 	touch "${SCRATCH_MNT}/ATTR.FMT_EXTENTS_REMOTE3K"
-	$XFS_IO_PROG -f -c "pwrite -S 0x43 0 $((blksz - 300))" "${SCRATCH_MNT}/attrvalfile" > /dev/null
-	attr -q -s user.remotebtreeattrname "${SCRATCH_MNT}/ATTR.FMT_EXTENTS_REMOTE3K" < "${SCRATCH_MNT}/attrvalfile"
+	$XFS_IO_PROG -f -c "pwrite -S 0x43 0 $((blksz - 300))" "${ATTRVALFILE}" > /dev/null
+	attr -q -s user.remotebtreeattrname "${SCRATCH_MNT}/ATTR.FMT_EXTENTS_REMOTE3K" < "${ATTRVALFILE}"
 
 	# FMT_EXTENTS with a remote block-size value
 	echo "+ attr extents with a remote one-block value"
 	touch "${SCRATCH_MNT}/ATTR.FMT_EXTENTS_REMOTE4K"
-	$XFS_IO_PROG -f -c "pwrite -S 0x44 0 ${blksz}" "${SCRATCH_MNT}/attrvalfile" > /dev/null
-	attr -q -s user.remotebtreeattrname "${SCRATCH_MNT}/ATTR.FMT_EXTENTS_REMOTE4K" < "${SCRATCH_MNT}/attrvalfile"
-	rm -rf "${SCRATCH_MNT}/attrvalfile"
+	$XFS_IO_PROG -f -c "pwrite -S 0x44 0 ${blksz}" "${ATTRVALFILE}" > /dev/null
+	attr -q -s user.remotebtreeattrname "${SCRATCH_MNT}/ATTR.FMT_EXTENTS_REMOTE4K" < "${ATTRVALFILE}"
+	rm -rf "${ATTRVALFILE}"
 
 	# Make an unused inode
 	echo "+ empty file"
diff --git a/tests/xfs/425 b/tests/xfs/425
new file mode 100755
index 0000000..dd11236
--- /dev/null
+++ b/tests/xfs/425
@@ -0,0 +1,121 @@
+#! /bin/bash
+# FS QA Test 425
+#
+# Look for stale data leaks in an xfs_metadump
+#
+# If this fails, get the byte offset of the leaked strings
+# which are found, then on the restored image in $TEST_DIR,
+# do:
+#
+# xfs_db> blockget -n
+# xfs_db> convert byte $BYTE daddr
+# $RESULT
+# xfs_db> daddr $RESULT
+# xfs_db> blockuse -n
+#
+# to see information about the metadata block which contains the
+# leaked strings
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/populate
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs xfs
+_supported_os Linux
+_require_test
+_require_scratch
+_require_populate_commands
+
+METADUMP_FILE="${TEST_DIR}/${seq}_metadump"
+MDRESTORE_FILE="${TEST_DIR}/${seq}_mdrestore"
+
+echo "Silence is golden"
+
+# Pattern the scratch disk, mkfs, and restore.
+$XFS_IO_PROG -d -c "pwrite 0 256M" $SCRATCH_DEV > $seqres.full 2>&1
+_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_populate nofill >> $seqres.full 2>&1
+
+# populate unmounts the fs for us
+
+_scratch_metadump $METADUMP_FILE
+xfs_mdrestore $METADUMP_FILE $MDRESTORE_FILE
+
+# Grep for stale data (leaked cd cd pattern) or strings
+# from populate routine
+
+hexdump -C $MDRESTORE_FILE | grep \
+"cd cd cd cd\| \
+41 41 41 41\|\
+42 42 42 42\|\
+43 43 43 43\|\
+44 44 44 44\|\
+61 61 61 61\|\
+62 62 62 62\|\
+63 63 63 63\|\
+64 64 64 64\|\
+dummy\|\
+S_IF\|\
+FMT_\|\
+INLINE\|\
+BLOCK\|\
+LEAF\|\
+NODE\|\
+BTREE\|\
+LOCAL\|\
+EXTENTS\|\
+REMOTE\|\
+ATTR\|\
+SYSTEM\|\
+TRUSTED\|\
+SECURITY\|\
+attrvalfile\|\
+unused\|\
+BNOBT\|\
+RMAPBT\|\
+RTRMAPBT\|\
+REFCOUNTBT" && echo "Leaked data found; see comments in test to debug"
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/425.out b/tests/xfs/425.out
new file mode 100644
index 0000000..7a9714b
--- /dev/null
+++ b/tests/xfs/425.out
@@ -0,0 +1,2 @@
+QA output created by 425
+Silence is golden
diff --git a/tests/xfs/group b/tests/xfs/group
index 185487d..d0d26ee 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -422,3 +422,4 @@
 422 dangerous_scrub dangerous_online_repair
 423 dangerous_scrub
 424 auto quick dump
+425 auto metadata

Re: [PATCH V2] xfs: test xfs_metadump for leaked strings

[Eric Sandeen]

On 8/1/17 9:58 PM, Eric Sandeen wrote:
> xfs_metadump by default sanitizes the image so that all strings
> longer than 5 chars are obfusccated, and all stale data in metadata
> blocks (i.e. unused/unwritten data) is zeroed out.  We didn't have
> a test for this, though, so this does it.
> 
> It patterns 256M of the scratch device, then uses djwong's
> populate infrastructure to write all types of metadata,
> metadumps & mdrestores it, then looks for either the leaked
> pre-pattern or any leaked strings or filenames.
> 
> The strings we look for are, unfortunately, a bit ad-hoc based on
> what is currently used in the populate routines.
> 
> Signed-off-by: Eric Sandeen <sandeen@redhat.com>

Sigh, debugging junk snuck into this one, hang on for v3, sorry.  :(

[PATCH Ve] xfs: test xfs_metadump for leaked strings

[Eric Sandeen]

xfs_metadump by default sanitizes the image so that all strings
longer than 5 chars are obfusccated, and all stale data in metadata
blocks (i.e. unused/unwritten data) is zeroed out.  We didn't have
a test for this, though, so this does it.

It patterns 256M of the scratch device, then uses djwong's
populate infrastructure to write all types of metadata,
metadumps & mdrestores it, then looks for either the leaked
pre-pattern or any leaked strings or filenames.

The strings we look for are, unfortunately, a bit ad-hoc based on
what is currently used in the populate routines.

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
---

V2: include .out file, minor whitespace & typo edits
V3: remove populate debug stuff that snuck in

diff --git a/tests/xfs/425 b/tests/xfs/425
new file mode 100755
index 0000000..dd11236
--- /dev/null
+++ b/tests/xfs/425
@@ -0,0 +1,121 @@
+#! /bin/bash
+# FS QA Test 425
+#
+# Look for stale data leaks in an xfs_metadump
+#
+# If this fails, get the byte offset of the leaked strings
+# which are found, then on the restored image in $TEST_DIR,
+# do:
+#
+# xfs_db> blockget -n
+# xfs_db> convert byte $BYTE daddr
+# $RESULT
+# xfs_db> daddr $RESULT
+# xfs_db> blockuse -n
+#
+# to see information about the metadata block which contains the
+# leaked strings
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Red Hat, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/populate
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs xfs
+_supported_os Linux
+_require_test
+_require_scratch
+_require_populate_commands
+
+METADUMP_FILE="${TEST_DIR}/${seq}_metadump"
+MDRESTORE_FILE="${TEST_DIR}/${seq}_mdrestore"
+
+echo "Silence is golden"
+
+# Pattern the scratch disk, mkfs, and restore.
+$XFS_IO_PROG -d -c "pwrite 0 256M" $SCRATCH_DEV > $seqres.full 2>&1
+_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_populate nofill >> $seqres.full 2>&1
+
+# populate unmounts the fs for us
+
+_scratch_metadump $METADUMP_FILE
+xfs_mdrestore $METADUMP_FILE $MDRESTORE_FILE
+
+# Grep for stale data (leaked cd cd pattern) or strings
+# from populate routine
+
+hexdump -C $MDRESTORE_FILE | grep \
+"cd cd cd cd\| \
+41 41 41 41\|\
+42 42 42 42\|\
+43 43 43 43\|\
+44 44 44 44\|\
+61 61 61 61\|\
+62 62 62 62\|\
+63 63 63 63\|\
+64 64 64 64\|\
+dummy\|\
+S_IF\|\
+FMT_\|\
+INLINE\|\
+BLOCK\|\
+LEAF\|\
+NODE\|\
+BTREE\|\
+LOCAL\|\
+EXTENTS\|\
+REMOTE\|\
+ATTR\|\
+SYSTEM\|\
+TRUSTED\|\
+SECURITY\|\
+attrvalfile\|\
+unused\|\
+BNOBT\|\
+RMAPBT\|\
+RTRMAPBT\|\
+REFCOUNTBT" && echo "Leaked data found; see comments in test to debug"
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/425.out b/tests/xfs/425.out
new file mode 100644
index 0000000..7a9714b
--- /dev/null
+++ b/tests/xfs/425.out
@@ -0,0 +1,2 @@
+QA output created by 425
+Silence is golden
diff --git a/tests/xfs/group b/tests/xfs/group
index 185487d..d0d26ee 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -422,3 +422,4 @@
 422 dangerous_scrub dangerous_online_repair
 423 dangerous_scrub
 424 auto quick dump
+425 auto metadata

Re: [PATCH Ve] xfs: test xfs_metadump for leaked strings

[Darrick J. Wong]

On Tue, Aug 01, 2017 at 10:45:06PM -0500, Eric Sandeen wrote:
> xfs_metadump by default sanitizes the image so that all strings
> longer than 5 chars are obfusccated, and all stale data in metadata
> blocks (i.e. unused/unwritten data) is zeroed out.  We didn't have
> a test for this, though, so this does it.
> 
> It patterns 256M of the scratch device, then uses djwong's
> populate infrastructure to write all types of metadata,
> metadumps & mdrestores it, then looks for either the leaked
> pre-pattern or any leaked strings or filenames.
> 
> The strings we look for are, unfortunately, a bit ad-hoc based on
> what is currently used in the populate routines.
> 
> Signed-off-by: Eric Sandeen <sandeen@redhat.com>
> ---
> 
> V2: include .out file, minor whitespace & typo edits
> V3: remove populate debug stuff that snuck in
> 
> diff --git a/tests/xfs/425 b/tests/xfs/425
> new file mode 100755
> index 0000000..dd11236
> --- /dev/null
> +++ b/tests/xfs/425
> @@ -0,0 +1,121 @@
> +#! /bin/bash
> +# FS QA Test 425
> +#
> +# Look for stale data leaks in an xfs_metadump
> +#
> +# If this fails, get the byte offset of the leaked strings
> +# which are found, then on the restored image in $TEST_DIR,
> +# do:
> +#
> +# xfs_db> blockget -n
> +# xfs_db> convert byte $BYTE daddr
> +# $RESULT
> +# xfs_db> daddr $RESULT
> +# xfs_db> blockuse -n
> +#
> +# to see information about the metadata block which contains the
> +# leaked strings
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Red Hat, Inc.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/populate
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs xfs
> +_supported_os Linux
> +_require_test
> +_require_scratch
> +_require_populate_commands
> +
> +METADUMP_FILE="${TEST_DIR}/${seq}_metadump"
> +MDRESTORE_FILE="${TEST_DIR}/${seq}_mdrestore"
> +
> +echo "Silence is golden"
> +
> +# Pattern the scratch disk, mkfs, and restore.
> +$XFS_IO_PROG -d -c "pwrite 0 256M" $SCRATCH_DEV > $seqres.full 2>&1

pwrite -b 1m to speed this up a bit...

> +_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
> +_scratch_populate nofill >> $seqres.full 2>&1
> +
> +# populate unmounts the fs for us
> +
> +_scratch_metadump $METADUMP_FILE
> +xfs_mdrestore $METADUMP_FILE $MDRESTORE_FILE
> +
> +# Grep for stale data (leaked cd cd pattern) or strings
> +# from populate routine
> +
> +hexdump -C $MDRESTORE_FILE | grep \

_require_command /usr/bin/hexdump "hexdump" ?

> +"cd cd cd cd\| \
> +41 41 41 41\|\
> +42 42 42 42\|\
> +43 43 43 43\|\
> +44 44 44 44\|\
> +61 61 61 61\|\
> +62 62 62 62\|\
> +63 63 63 63\|\
> +64 64 64 64\|\
> +dummy\|\
> +S_IF\|\
> +FMT_\|\
> +INLINE\|\
> +BLOCK\|\
> +LEAF\|\
> +NODE\|\
> +BTREE\|\
> +LOCAL\|\
> +EXTENTS\|\
> +REMOTE\|\
> +ATTR\|\
> +SYSTEM\|\
> +TRUSTED\|\
> +SECURITY\|\
> +attrvalfile\|\
> +unused\|\
> +BNOBT\|\
> +RMAPBT\|\
> +RTRMAPBT\|\
> +REFCOUNTBT" && echo "Leaked data found; see comments in test to debug"

/me wonders if this grep pattern ought to be some kind of _filter
function in common/populate?  That way if the xfs populate command adds
more weird strings, the search pattern is right there in the rc file,
not buried in tests/xfs/ somewhere.

OTOH a good counterargument is that this is the only test that cares
about that, so why make it common code?  <sigh>

--D

> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/xfs/425.out b/tests/xfs/425.out
> new file mode 100644
> index 0000000..7a9714b
> --- /dev/null
> +++ b/tests/xfs/425.out
> @@ -0,0 +1,2 @@
> +QA output created by 425
> +Silence is golden
> diff --git a/tests/xfs/group b/tests/xfs/group
> index 185487d..d0d26ee 100644
> --- a/tests/xfs/group
> +++ b/tests/xfs/group
> @@ -422,3 +422,4 @@
>  422 dangerous_scrub dangerous_online_repair
>  423 dangerous_scrub
>  424 auto quick dump
> +425 auto metadata
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH Ve] xfs: test xfs_metadump for leaked strings

[Eric Sandeen]

On 8/2/17 12:29 PM, Darrick J. Wong wrote:
> On Tue, Aug 01, 2017 at 10:45:06PM -0500, Eric Sandeen wrote:
>> xfs_metadump by default sanitizes the image so that all strings
>> longer than 5 chars are obfusccated, and all stale data in metadata
>> blocks (i.e. unused/unwritten data) is zeroed out.  We didn't have
>> a test for this, though, so this does it.
>>
>> It patterns 256M of the scratch device, then uses djwong's
>> populate infrastructure to write all types of metadata,
>> metadumps & mdrestores it, then looks for either the leaked
>> pre-pattern or any leaked strings or filenames.
>>
>> The strings we look for are, unfortunately, a bit ad-hoc based on
>> what is currently used in the populate routines.
>>
>> Signed-off-by: Eric Sandeen <sandeen@redhat.com>
>> ---
>>
>> V2: include .out file, minor whitespace & typo edits
>> V3: remove populate debug stuff that snuck in


>> +echo "Silence is golden"
>> +
>> +# Pattern the scratch disk, mkfs, and restore.
>> +$XFS_IO_PROG -d -c "pwrite 0 256M" $SCRATCH_DEV > $seqres.full 2>&1
> 
> pwrite -b 1m to speed this up a bit...

ok
 
>> +_scratch_mkfs_sized $((256 * 1024 * 1024)) >> $seqres.full 2>&1
>> +_scratch_populate nofill >> $seqres.full 2>&1
>> +
>> +# populate unmounts the fs for us
>> +
>> +_scratch_metadump $METADUMP_FILE
>> +xfs_mdrestore $METADUMP_FILE $MDRESTORE_FILE
>> +
>> +# Grep for stale data (leaked cd cd pattern) or strings
>> +# from populate routine
>> +
>> +hexdump -C $MDRESTORE_FILE | grep \
> 
> _require_command /usr/bin/hexdump "hexdump" ?

sure? other tests don't bother, but ok.

> 
>> +"cd cd cd cd\| \
>> +41 41 41 41\|\
>> +42 42 42 42\|\
>> +43 43 43 43\|\
>> +44 44 44 44\|\
>> +61 61 61 61\|\
>> +62 62 62 62\|\
>> +63 63 63 63\|\
>> +64 64 64 64\|\
>> +dummy\|\
>> +S_IF\|\
>> +FMT_\|\
>> +INLINE\|\
>> +BLOCK\|\
>> +LEAF\|\
>> +NODE\|\
>> +BTREE\|\
>> +LOCAL\|\
>> +EXTENTS\|\
>> +REMOTE\|\
>> +ATTR\|\
>> +SYSTEM\|\
>> +TRUSTED\|\
>> +SECURITY\|\
>> +attrvalfile\|\
>> +unused\|\
>> +BNOBT\|\
>> +RMAPBT\|\
>> +RTRMAPBT\|\
>> +REFCOUNTBT" && echo "Leaked data found; see comments in test to debug"
> 
> /me wonders if this grep pattern ought to be some kind of _filter
> function in common/populate?  That way if the xfs populate command adds
> more weird strings, the search pattern is right there in the rc file,
> not buried in tests/xfs/ somewhere.
> 
> OTOH a good counterargument is that this is the only test that cares
> about that, so why make it common code?  <sigh>

um, I totally agree with both your arguments... :/

i'll put it in populate I guess.

> 
> --D