[PATCH v4 0/4] tests for btrfs fsverity

[PATCH v4 0/4] tests for btrfs fsverity

[Boris Burkov]

This patchset provides tests for fsverity support in btrfs.

It includes modifications for generic tests to pass with btrfs as well
as new tests.

--
v4:
- mark local variables
- get rid of redundant mounts and syncs
- use '_' in function names correctly
- add a test for the EFBIG case
- reduce usage of requires_btrfs_corrupt_block
- handle variable input when corrupting merkle tree
v3: rebase onto xfstests master branch
v2: pass generic tests, add logwrites test

Boris Burkov (4):
  btrfs: test btrfs specific fsverity corruption
  generic/574: corrupt btrfs merkle tree data
  btrfs: test verity orphans with dmlogwrites
  generic: test fs-verity EFBIG scenarios

 common/btrfs          |   5 ++
 common/config         |   1 +
 common/verity         |  25 ++++++
 tests/btrfs/290       | 180 ++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/290.out   |  25 ++++++
 tests/btrfs/291       | 157 ++++++++++++++++++++++++++++++++++++
 tests/btrfs/291.out   |   2 +
 tests/btrfs/group     |   2 +
 tests/generic/574     |   5 ++
 tests/generic/632     |  86 ++++++++++++++++++++
 tests/generic/632.out |   7 ++
 tests/generic/group   |   1 +
 12 files changed, 496 insertions(+)
 create mode 100755 tests/btrfs/290
 create mode 100644 tests/btrfs/290.out
 create mode 100755 tests/btrfs/291
 create mode 100644 tests/btrfs/291.out
 create mode 100755 tests/generic/632
 create mode 100644 tests/generic/632.out

-- 
2.30.2

[PATCH v4 1/4] btrfs: test btrfs specific fsverity corruption

[Boris Burkov]

There are some btrfs specific fsverity scenarios that don't map
neatly onto the tests in generic/574 like holes, inline extents,
and preallocated extents. Cover those in a btrfs specific test.

This test relies on the btrfs implementation of fsverity in the patches
titled:
btrfs: initial fsverity support
btrfs: check verity for reads of inline extents and holes

and on btrfs-corrupt-block for corruption in the patches titled:
btrfs-progs: corrupt generic item data with btrfs-corrupt-block
btrfs-progs: expand corrupt_file_extent in btrfs-corrupt-block

Signed-off-by: Boris Burkov <boris@bur.io>
---
 common/btrfs        |   5 ++
 common/config       |   1 +
 common/verity       |   7 ++
 tests/btrfs/290     | 180 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/290.out |  25 ++++++
 tests/btrfs/group   |   1 +
 6 files changed, 219 insertions(+)
 create mode 100755 tests/btrfs/290
 create mode 100644 tests/btrfs/290.out

diff --git a/common/btrfs b/common/btrfs
index ebe6ce26..bd6e87ce 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -419,3 +419,8 @@ _btrfs_rescan_devices()
 {
 	$BTRFS_UTIL_PROG device scan &> /dev/null
 }
+
+_require_btrfs_corrupt_block()
+{
+	_require_command "$BTRFS_CORRUPT_BLOCK_PROG" btrfs_corrupt_block
+}
diff --git a/common/config b/common/config
index a47e462c..003b2a88 100644
--- a/common/config
+++ b/common/config
@@ -256,6 +256,7 @@ export BTRFS_UTIL_PROG=$(type -P btrfs)
 export BTRFS_SHOW_SUPER_PROG=$(type -P btrfs-show-super)
 export BTRFS_CONVERT_PROG=$(type -P btrfs-convert)
 export BTRFS_TUNE_PROG=$(type -P btrfstune)
+export BTRFS_CORRUPT_BLOCK_PROG=$(type -P btrfs-corrupt-block)
 export XFS_FSR_PROG=$(type -P xfs_fsr)
 export MKFS_NFS_PROG="false"
 export MKFS_CIFS_PROG="false"
diff --git a/common/verity b/common/verity
index 38eea157..d2c1ea24 100644
--- a/common/verity
+++ b/common/verity
@@ -8,6 +8,10 @@ _require_scratch_verity()
 	_require_scratch
 	_require_command "$FSVERITY_PROG" fsverity
 
+	if [ $FSTYP == "btrfs" ]; then
+		_require_command "$BTRFS_CORRUPT_BLOCK_PROG" btrfs_corrupt_block
+	fi
+
 	if ! _scratch_mkfs_verity &>>$seqres.full; then
 		# ext4: need e2fsprogs v1.44.5 or later (but actually v1.45.2+
 		#       is needed for some tests to pass, due to an e2fsck bug)
@@ -147,6 +151,9 @@ _scratch_mkfs_verity()
 	ext4|f2fs)
 		_scratch_mkfs -O verity
 		;;
+	btrfs)
+		_scratch_mkfs
+		;;
 	*)
 		_notrun "No verity support for $FSTYP"
 		;;
diff --git a/tests/btrfs/290 b/tests/btrfs/290
new file mode 100755
index 00000000..26939833
--- /dev/null
+++ b/tests/btrfs/290
@@ -0,0 +1,180 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2021 Facebook, Inc. All Rights Reserved.
+#
+# FS QA Test 290
+#
+# Test btrfs support for fsverity.
+# This test extends the generic fsverity testing by corrupting inline extents,
+# preallocated extents, holes, and the Merkle descriptor in a btrfs-aware way.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/verity
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+_supported_fs btrfs
+_require_scratch_verity
+_require_scratch_nocheck
+_require_odirect
+_require_btrfs_corrupt_block
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+get_ino() {
+	local file=$1
+	stat -c "%i" $file
+}
+
+validate() {
+	local f=$1
+	local sz=$(_get_filesize $f)
+	# buffered io
+	echo $(basename $f)
+	$XFS_IO_PROG -rc "pread -q 0 $sz" $f 2>&1 | _filter_scratch
+	# direct io
+	$XFS_IO_PROG -rdc "pread -q 0 $sz" $f 2>&1 | _filter_scratch
+}
+
+# corrupt the data portion of an inline extent
+corrupt_inline() {
+	local f=$SCRATCH_MNT/inl
+	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 42" $f
+	local ino=$(get_ino $f)
+	_fsv_enable $f
+	_scratch_unmount
+	# inline data starts at disk_bytenr
+	# overwrite the first u64 with random bogus junk
+	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 0 -f disk_bytenr $SCRATCH_DEV > /dev/null 2>&1
+	_scratch_mount
+	validate $f
+}
+
+# preallocate a file, then corrupt it by changing it to a regular file
+corrupt_prealloc_to_reg() {
+	local f=$SCRATCH_MNT/prealloc
+	$XFS_IO_PROG -fc "falloc 0 12k" $f
+	local ino=$(get_ino $f)
+	_fsv_enable $f
+	_scratch_unmount
+	# set extent type from prealloc (2) to reg (1)
+	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 0 -f type -v 1 $SCRATCH_DEV >/dev/null 2>&1
+	_scratch_mount
+	validate $f
+}
+
+# corrupt a regular file by changing the type to preallocated
+corrupt_reg_to_prealloc() {
+	local f=$SCRATCH_MNT/reg
+	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+	local ino=$(get_ino $f)
+	_fsv_enable $f
+	_scratch_unmount
+	# set type from reg (1) to prealloc (2)
+	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 0 -f type -v 2 $SCRATCH_DEV >/dev/null 2>&1
+	_scratch_mount
+	validate $f
+}
+
+# corrupt a file by punching a hole
+corrupt_punch_hole() {
+	local f=$SCRATCH_MNT/punch
+	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+	local ino=$(get_ino $f)
+	# make a new extent in the middle, sync so the writes don't coalesce
+	$XFS_IO_PROG -c sync $SCRATCH_MNT
+	$XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f
+	_fsv_enable $f
+	_scratch_unmount
+	# change disk_bytenr to 0, representing a hole
+	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr -v 0 $SCRATCH_DEV > /dev/null 2>&1
+	_scratch_mount
+	validate $f
+}
+
+# plug hole
+corrupt_plug_hole() {
+	local f=$SCRATCH_MNT/plug
+	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+	local ino=$(get_ino $f)
+	$XFS_IO_PROG -fc "falloc 4k 4k" $f
+	_fsv_enable $f
+	_scratch_unmount
+	# change disk_bytenr to some value, plugging the hole
+	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr -v 13639680 $SCRATCH_DEV > /dev/null 2>&1
+	_scratch_mount
+	validate $f
+}
+
+# corrupt the fsverity descriptor item indiscriminately (causes EINVAL)
+corrupt_verity_descriptor() {
+	local f=$SCRATCH_MNT/desc
+	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+	local ino=$(get_ino $f)
+	_fsv_enable $f
+	_scratch_unmount
+	# key for the descriptor item is <inode, BTRFS_VERITY_DESC_ITEM_KEY, 1>,
+	# 88 is X. So we write 5 Xs to the start of the descriptor
+	$BTRFS_CORRUPT_BLOCK_PROG -r 5 -I $ino,36,1 -v 88 -o 0 -b 5 $SCRATCH_DEV > /dev/null 2>&1
+	_scratch_mount
+	validate $f
+}
+
+# specifically target the root hash in the descriptor (causes EIO)
+corrupt_root_hash() {
+	local f=$SCRATCH_MNT/roothash
+	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+	local ino=$(get_ino $f)
+	_fsv_enable $f
+	_scratch_unmount
+	$BTRFS_CORRUPT_BLOCK_PROG -r 5 -I $ino,36,1 -v 88 -o 16 -b 1 $SCRATCH_DEV > /dev/null 2>&1
+	_scratch_mount
+	validate $f
+}
+
+# corrupt the Merkle tree data itself
+corrupt_merkle_tree() {
+	local f=$SCRATCH_MNT/merkle
+	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
+	local ino=$(get_ino $f)
+	_fsv_enable $f
+	_scratch_unmount
+	# key for the descriptor item is <inode, BTRFS_VERITY_MERKLE_ITEM_KEY, 0>,
+	# 88 is X. So we write 5 Xs to somewhere in the middle of the first
+	# merkle item
+	$BTRFS_CORRUPT_BLOCK_PROG -r 5 -I $ino,37,0 -v 88 -o 100 -b 5 $SCRATCH_DEV > /dev/null 2>&1
+	_scratch_mount
+	validate $f
+}
+
+# real QA test starts here
+_scratch_mkfs >/dev/null
+_scratch_mount
+
+corrupt_inline
+corrupt_prealloc_to_reg
+corrupt_reg_to_prealloc
+corrupt_punch_hole
+corrupt_plug_hole
+corrupt_verity_descriptor
+corrupt_root_hash
+corrupt_merkle_tree
+
+status=0
+exit
diff --git a/tests/btrfs/290.out b/tests/btrfs/290.out
new file mode 100644
index 00000000..056b114b
--- /dev/null
+++ b/tests/btrfs/290.out
@@ -0,0 +1,25 @@
+QA output created by 290
+inl
+pread: Input/output error
+pread: Input/output error
+prealloc
+pread: Input/output error
+pread: Input/output error
+reg
+pread: Input/output error
+pread: Input/output error
+punch
+pread: Input/output error
+pread: Input/output error
+plug
+pread: Input/output error
+pread: Input/output error
+desc
+SCRATCH_MNT/desc: Invalid argument
+SCRATCH_MNT/desc: Invalid argument
+roothash
+pread: Input/output error
+pread: Input/output error
+merkle
+pread: Input/output error
+pread: Input/output error
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 331dd432..13051562 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -238,3 +238,4 @@
 233 auto quick subvolume
 234 auto quick compress rw
 235 auto quick send
+290 auto quick verity
-- 
2.30.2

[PATCH v4 2/4] generic/574: corrupt btrfs merkle tree data

[Boris Burkov]

generic/574 has tests for corrupting the merkle tree data stored by the
filesystem. Since btrfs uses a different scheme for storing this data,
the existing logic for corrupting it doesn't work out of the box. Adapt
it to properly corrupt btrfs merkle items.

This test relies on the btrfs implementation of fsverity in the patches
titled:
btrfs: initial fsverity support
btrfs: check verity for reads of inline extents and holes
btrfs: fallback to buffered io for verity files

A fix for fiemap in the patch titled:
btrfs: return whole extents in fiemap

and on btrfs-corrupt-block for corruption in the patches titled:
btrfs-progs: corrupt generic item data with btrfs-corrupt-block
btrfs-progs: expand corrupt_file_extent in btrfs-corrupt-block

Signed-off-by: Boris Burkov <boris@bur.io>
---
 common/verity     | 18 ++++++++++++++++++
 tests/generic/574 |  5 +++++
 2 files changed, 23 insertions(+)

diff --git a/common/verity b/common/verity
index d2c1ea24..1636e88b 100644
--- a/common/verity
+++ b/common/verity
@@ -315,6 +315,24 @@ _fsv_scratch_corrupt_merkle_tree()
 		(( offset += ($(_get_filesize $file) + 65535) & ~65535 ))
 		_fsv_scratch_corrupt_bytes $file $offset
 		;;
+	btrfs)
+		local ino=$(stat -c '%i' $file)
+		_scratch_unmount
+		local byte=""
+		while read -n 1 byte; do
+			if [ -z $byte ]; then
+				break
+			fi
+			local ascii=$(printf "%d" "'$byte'")
+			# This command will find a Merkle tree item for the inode (-I $ino,37,0)
+			# in the default filesystem tree (-r 5) and corrupt one byte (-b 1) at
+			# $offset (-o $offset) with the ascii representation of the byte we read
+			# (-v $ascii)
+			$BTRFS_CORRUPT_BLOCK_PROG -r 5 -I $ino,37,0 -v $ascii -o $offset -b 1 $SCRATCH_DEV
+			(( offset += 1 ))
+		done
+		_scratch_mount
+		;;
 	*)
 		_fail "_fsv_scratch_corrupt_merkle_tree() unimplemented on $FSTYP"
 		;;
diff --git a/tests/generic/574 b/tests/generic/574
index 1e296618..e4370dae 100755
--- a/tests/generic/574
+++ b/tests/generic/574
@@ -43,6 +43,11 @@ _scratch_mount
 fsv_orig_file=$SCRATCH_MNT/file
 fsv_file=$SCRATCH_MNT/file.fsv
 
+# utility needed for corrupting Merkle data itself in btrfs
+if [ $FSTYP == "btrfs" ]; then
+	_require_command "$BTRFS_CORRUPT_BLOCK_PROG" btrfs_corrupt_block
+fi
+
 setup_zeroed_file()
 {
 	local len=$1
-- 
2.30.2

[PATCH v4 3/4] btrfs: test verity orphans with dmlogwrites

[Boris Burkov]

The behavior of orphans is most interesting across mounts, interrupted
at arbitrary points during fsverity enable. To cover as many such cases
as possible, use dmlogwrites and dmsnapshot as in
log-writes/replay-individual.sh. At each log entry, we want to assert a
somewhat complicated invariant:

If verity has not yet started: an orphan indicates that verity has
started.
If verity has started: mount should handle the orphan and blow away
verity data: expect 0 merkle items after mounting the snapshot dev. If
we can measure the file, verity has finished.
If verity has finished: the orphan should be gone, so mount should not
blow away merkle items. Expect the same number of merkle items before
and after mounting the snapshot dev.

Note that this relies on grepping btrfs inspect-internal dump-tree.
Until btrfs-progs has the ability to print the new Merkle items, they
will show up as UNKNOWN.36/37.

Signed-off-by: Boris Burkov <boris@bur.io>
---
 tests/btrfs/291     | 157 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/291.out |   2 +
 tests/btrfs/group   |   1 +
 3 files changed, 160 insertions(+)
 create mode 100755 tests/btrfs/291
 create mode 100644 tests/btrfs/291.out

diff --git a/tests/btrfs/291 b/tests/btrfs/291
new file mode 100755
index 00000000..eab3a495
--- /dev/null
+++ b/tests/btrfs/291
@@ -0,0 +1,157 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2021 Facebook, Inc. All Rights Reserved.
+#
+# FS QA Test 291
+#
+# Test btrfs consistency after each FUA while enabling verity on a file
+# This test works by following the pattern in log-writes/replay-individual.sh:
+# 1. run a workload (verity + sync) while logging to the log device
+# 2. replay an entry to the replay device
+# 3. snapshot the replay device to the snapshot device
+# 4. run destructive tests on the snapshot device (e.g. mount with orphans)
+# 5. goto 2
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	_log_writes_cleanup &> /dev/null
+	rm -f $tmp.*
+	$LVM_PROG vgremove -f -y $vgname >>$seqres.full 2>&1
+	losetup -d $loop_dev >>$seqres.full 2>&1
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+. ./common/dmlogwrites
+. ./common/verity
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+
+_require_scratch
+_require_log_writes
+_require_dm_target snapshot
+_require_command $LVM_PROG lvm
+_require_scratch_verity
+
+sync_loop() {
+	i=$1
+	[ -z "$i" ] && _fail "sync loop needs a number of iterations"
+	while [ $i -gt 0 ]
+	do
+		$XFS_IO_PROG -c sync $SCRATCH_MNT
+		let i-=1
+	done
+}
+
+dump_tree() {
+	local dev=$1
+	$BTRFS_UTIL_PROG inspect-internal dump-tree $dev
+}
+
+count_item() {
+	local dev=$1
+	local item=$2
+	dump_tree $dev | grep -c $item
+}
+
+_log_writes_init $SCRATCH_DEV
+_log_writes_mkfs
+_log_writes_mount
+
+f=$SCRATCH_MNT/fsv
+MB=$((1024 * 1024))
+$XFS_IO_PROG -fc "pwrite -q 0 $((10 * $MB))" $f
+$XFS_IO_PROG -c sync $SCRATCH_MNT
+sync_loop 10 &
+_fsv_enable $f
+$XFS_IO_PROG -c sync $SCRATCH_MNT
+
+_log_writes_unmount
+_log_writes_remove
+
+$XFS_IO_PROG -fc "pwrite -q -S 0 $((8192 * $MB)) $MB" $tmp.loop-file
+loop_dev=$(losetup -f --show $tmp.loop-file)
+vgname=vg_replay
+lvname=lv_replay
+replay_dev=/dev/mapper/vg_replay-lv_replay
+snapname=lv_snap
+snap_dev=/dev/mapper/vg_replay-$snapname
+
+$LVM_PROG vgcreate -f $vgname $loop_dev >>$seqres.full 2>&1 || _fail "failed to vgcreate $vgname"
+$LVM_PROG lvcreate -L 4G -n $lvname $vgname -y >>$seqres.full 2>&1 || \
+	_fail "failed to lvcreate $lvname"
+$UDEV_SETTLE_PROG >>$seqres.full 2>&1
+
+replay_log_prog=$here/src/log-writes/replay-log
+num_entries=$($replay_log_prog --log $LOGWRITES_DEV --num-entries)
+entry=$($replay_log_prog --log $LOGWRITES_DEV --replay $replay_dev --find --end-mark mkfs | cut -d@ -f1)
+$replay_log_prog --log $LOGWRITES_DEV --replay $replay_dev --limit $entry || \
+	_fail "failed to replay to start entry $entry"
+let entry+=1
+
+# state = 0: verity hasn't started
+# state = 1: verity underway
+# state = 2: verity done
+state=0
+while [ $entry -lt $num_entries ];
+do
+	$replay_log_prog --limit 1 --log $LOGWRITES_DEV --replay $replay_dev --start $entry || \
+		_fail "failed to take replay step at entry: $entry"
+
+	$LVM_PROG lvcreate -s -L 4M -n $snapname $vgname/$lvname >>$seqres.full 2>&1 || \
+		_fail "Failed to create snapshot"
+	$UDEV_SETTLE_PROG >>$seqres.full 2>&1
+
+	orphan=$(count_item $snap_dev ORPHAN)
+	if [ $state -eq 0 ]; then
+		[ $orphan -gt 0 ] && state=1
+	fi
+
+	pre_mount=$(count_item $snap_dev UNKNOWN.3[67])
+	_mount $snap_dev $SCRATCH_MNT || _fail "mount failed at entry $entry"
+	fsverity measure $SCRATCH_MNT/fsv >>$seqres.full 2>&1
+	measured=$?
+	umount $SCRATCH_MNT
+	[ $state -eq 1 ] && [ $measured -eq 0 ] && state=2
+	[ $state -eq 2 ] && ([ $measured -eq 0 ] || _fail "verity done, but measurement failed at entry $entry")
+	post_mount=$(count_item $snap_dev UNKNOWN.3[67])
+
+	echo "entry: $entry, state: $state, orphan: $orphan, pre_mount: $pre_mount, post_mount: $post_mount" >> $seqres.full
+
+	if [ $state -eq 1 ]; then
+		[ $post_mount -eq 0 ] || \
+			_fail "mount failed to clear under-construction merkle items pre: $pre_mount, post: $post_mount at entry $entry";
+	fi
+	if [ $state -eq 2 ]; then
+		[ $pre_mount -gt 0 ] || \
+			_fail "expected to have verity items before mount at entry $entry"
+		[ $pre_mount -eq $post_mount ] || \
+			_fail "mount cleared merkle items after verity was enabled $pre_mount vs $post_mount at entry $entry";
+	fi
+
+	let entry+=1
+	$LVM_PROG lvremove $vgname/$snapname -y >>$seqres.full
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/291.out b/tests/btrfs/291.out
new file mode 100644
index 00000000..04605c70
--- /dev/null
+++ b/tests/btrfs/291.out
@@ -0,0 +1,2 @@
+QA output created by 291
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 13051562..cc5a811e 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -239,3 +239,4 @@
 234 auto quick compress rw
 235 auto quick send
 290 auto quick verity
+291 auto verity
-- 
2.30.2

[PATCH v4 4/4] generic: test fs-verity EFBIG scenarios

[Boris Burkov]

btrfs, ext4, and f2fs cache the Merkle tree past EOF, which restricts
the maximum file size beneath the normal maximum. Test the logic in
those filesystems against files with sizes near the maximum.

To work properly, this does require some understanding of the practical
but not standardized layout of the Merkle tree. This is a bit unpleasant
and could make the test incorrect in the future, if the implementation
changes. On the other hand, it feels quite useful to test this tricky
edge case. It could perhaps be made more generic by adding some ioctls
to let the file system communicate the maximum file size for a verity
file or some information about the storage of the Merkle tree.

Signed-off-by: Boris Burkov <boris@bur.io>
---
 tests/generic/632     | 86 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/632.out |  7 ++++
 tests/generic/group   |  1 +
 3 files changed, 94 insertions(+)
 create mode 100755 tests/generic/632
 create mode 100644 tests/generic/632.out

diff --git a/tests/generic/632 b/tests/generic/632
new file mode 100755
index 00000000..5a5ed576
--- /dev/null
+++ b/tests/generic/632
@@ -0,0 +1,86 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Facebook, Inc.  All Rights Reserved.
+#
+# FS QA Test 632
+#
+# Test some EFBIG scenarios with very large files.
+# To create the files, use pwrite with an offset close to the
+# file system's max file size.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/verity
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs ext4 f2fs
+_require_test
+_require_math
+_require_scratch_verity
+
+_scratch_mkfs_verity &>> $seqres.full
+_scratch_mount
+
+fsv_file=$SCRATCH_MNT/file.fsv
+
+max_sz=$(_get_max_file_size)
+_fsv_scratch_begin_subtest "way too big: fail on first merkle block"
+# have to go back by 4096 from max to not hit the fsverity MAX_DEPTH check.
+$XFS_IO_PROG -fc "pwrite -q $(($max_sz - 4096)) 1" $fsv_file
+_fsv_enable $fsv_file |& _filter_scratch
+
+# The goal of this second test is to make a big enough file that we trip the
+# EFBIG codepath, but not so big that we hit it immediately as soon as we try
+# to write a Merkle leaf. Because of the layout of the Merkle tree that
+# fs-verity uses, this is a bit complicated to compute dynamically.
+
+# The layout of the Merkle tree has the leaf nodes last, but writes them first.
+# To get an interesting overflow, we need the start of L0 to be < MAX but the
+# end of the merkle tree (EOM) to be past MAX. Ideally, the start of L0 is only
+# just smaller than MAX, so that we don't have to write many blocks to blow up.
+
+# 0                        EOF round-to-64k L7L6L5 L4   L3    L2    L1  L0 MAX  EOM
+# |-------------------------|               ||-|--|---|----|-----|------|--|!!!!!|
+
+# Given this structure, we can compute the size of the file that yields the
+# desired properties:
+# sz + 64k + sz/128^8 + sz/128^7 + ... + sz/128^2 < MAX
+# (128^8)sz + (128^8)64k + sz + (128)sz + (128^2)sz + ... + (128^6)sz < (128^8)MAX
+# sz(128^8 + 128^6 + 128^5 + 128^4 + 128^3 + 128^2 + 128 + 1) < (128^8)(MAX - 64k)
+# sz < (128^8/(128^8 + (128^6 + ... 1))(MAX - 64k)
+#
+# Do the actual caclulation with 'bc' and 20 digits of precision.
+set -f
+calc="scale=20; ($max_sz - 65536) * ((128^8) / (1 + 128 + 128^2 + 128^3 + 128^4 + 128^5 + 128^6 + 128^8))"
+sz=$(echo $calc | $BC -q | cut -d. -f1)
+set +f
+
+
+_fsv_scratch_begin_subtest "still too big: fail on first invalid merkle block"
+$XFS_IO_PROG -fc "pwrite -q $sz 1" $fsv_file
+_fsv_enable $fsv_file |& _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/632.out b/tests/generic/632.out
new file mode 100644
index 00000000..59602c24
--- /dev/null
+++ b/tests/generic/632.out
@@ -0,0 +1,7 @@
+QA output created by 632
+
+# way too big: fail on first merkle block
+ERROR: FS_IOC_ENABLE_VERITY failed on 'SCRATCH_MNT/file.fsv': File too large
+
+# still too big: fail on first invalid merkle block
+ERROR: FS_IOC_ENABLE_VERITY failed on 'SCRATCH_MNT/file.fsv': File too large
diff --git a/tests/generic/group b/tests/generic/group
index ab00cc04..76d46e86 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -634,3 +634,4 @@
 629 auto quick rw copy_range
 630 auto quick rw dedupe clone
 631 auto rw overlay rename
+632 auto quick verity
-- 
2.30.2

Re: [PATCH v4 1/4] btrfs: test btrfs specific fsverity corruption

[Eryu Guan]

On Wed, May 05, 2021 at 02:04:43PM -0700, Boris Burkov wrote:
> There are some btrfs specific fsverity scenarios that don't map
> neatly onto the tests in generic/574 like holes, inline extents,
> and preallocated extents. Cover those in a btrfs specific test.
> 
> This test relies on the btrfs implementation of fsverity in the patches
> titled:
> btrfs: initial fsverity support
> btrfs: check verity for reads of inline extents and holes
> 
> and on btrfs-corrupt-block for corruption in the patches titled:
> btrfs-progs: corrupt generic item data with btrfs-corrupt-block
> btrfs-progs: expand corrupt_file_extent in btrfs-corrupt-block
> 
> Signed-off-by: Boris Burkov <boris@bur.io>
> ---
>  common/btrfs        |   5 ++
>  common/config       |   1 +
>  common/verity       |   7 ++
>  tests/btrfs/290     | 180 ++++++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/290.out |  25 ++++++
>  tests/btrfs/group   |   1 +
>  6 files changed, 219 insertions(+)
>  create mode 100755 tests/btrfs/290
>  create mode 100644 tests/btrfs/290.out
> 
> diff --git a/common/btrfs b/common/btrfs
> index ebe6ce26..bd6e87ce 100644
> --- a/common/btrfs
> +++ b/common/btrfs
> @@ -419,3 +419,8 @@ _btrfs_rescan_devices()
>  {
>  	$BTRFS_UTIL_PROG device scan &> /dev/null
>  }
> +
> +_require_btrfs_corrupt_block()
> +{
> +	_require_command "$BTRFS_CORRUPT_BLOCK_PROG" btrfs_corrupt_block

The command should be btrfs-corrupt-block, instead of
btrfs_corrupt_block?

> +}
> diff --git a/common/config b/common/config
> index a47e462c..003b2a88 100644
> --- a/common/config
> +++ b/common/config
> @@ -256,6 +256,7 @@ export BTRFS_UTIL_PROG=$(type -P btrfs)
>  export BTRFS_SHOW_SUPER_PROG=$(type -P btrfs-show-super)
>  export BTRFS_CONVERT_PROG=$(type -P btrfs-convert)
>  export BTRFS_TUNE_PROG=$(type -P btrfstune)
> +export BTRFS_CORRUPT_BLOCK_PROG=$(type -P btrfs-corrupt-block)
>  export XFS_FSR_PROG=$(type -P xfs_fsr)
>  export MKFS_NFS_PROG="false"
>  export MKFS_CIFS_PROG="false"
> diff --git a/common/verity b/common/verity
> index 38eea157..d2c1ea24 100644
> --- a/common/verity
> +++ b/common/verity
> @@ -8,6 +8,10 @@ _require_scratch_verity()
>  	_require_scratch
>  	_require_command "$FSVERITY_PROG" fsverity
>  
> +	if [ $FSTYP == "btrfs" ]; then
> +		_require_command "$BTRFS_CORRUPT_BLOCK_PROG" btrfs_corrupt_block

Use the _require_btrfs_corrupt_block helper?

> +	fi
> +
>  	if ! _scratch_mkfs_verity &>>$seqres.full; then
>  		# ext4: need e2fsprogs v1.44.5 or later (but actually v1.45.2+
>  		#       is needed for some tests to pass, due to an e2fsck bug)
> @@ -147,6 +151,9 @@ _scratch_mkfs_verity()
>  	ext4|f2fs)
>  		_scratch_mkfs -O verity
>  		;;
> +	btrfs)
> +		_scratch_mkfs
> +		;;
>  	*)
>  		_notrun "No verity support for $FSTYP"
>  		;;
> diff --git a/tests/btrfs/290 b/tests/btrfs/290
> new file mode 100755
> index 00000000..26939833
> --- /dev/null
> +++ b/tests/btrfs/290
> @@ -0,0 +1,180 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2021 Facebook, Inc. All Rights Reserved.
> +#
> +# FS QA Test 290
> +#
> +# Test btrfs support for fsverity.
> +# This test extends the generic fsverity testing by corrupting inline extents,
> +# preallocated extents, holes, and the Merkle descriptor in a btrfs-aware way.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/verity
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +_supported_fs btrfs
> +_require_scratch_verity
> +_require_scratch_nocheck
> +_require_odirect
> +_require_btrfs_corrupt_block

This is already checked in _require_scratch_verity.

And we also need '_require_xfs_io_command "falloc"'

Otherwise looks good to me from fstests' point of view.

Thanks,
Eryu

> +
> +_cleanup()
> +{
> +	cd /
> +	rm -f $tmp.*
> +}
> +
> +get_ino() {
> +	local file=$1
> +	stat -c "%i" $file
> +}
> +
> +validate() {
> +	local f=$1
> +	local sz=$(_get_filesize $f)
> +	# buffered io
> +	echo $(basename $f)
> +	$XFS_IO_PROG -rc "pread -q 0 $sz" $f 2>&1 | _filter_scratch
> +	# direct io
> +	$XFS_IO_PROG -rdc "pread -q 0 $sz" $f 2>&1 | _filter_scratch
> +}
> +
> +# corrupt the data portion of an inline extent
> +corrupt_inline() {
> +	local f=$SCRATCH_MNT/inl
> +	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 42" $f
> +	local ino=$(get_ino $f)
> +	_fsv_enable $f
> +	_scratch_unmount
> +	# inline data starts at disk_bytenr
> +	# overwrite the first u64 with random bogus junk
> +	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 0 -f disk_bytenr $SCRATCH_DEV > /dev/null 2>&1
> +	_scratch_mount
> +	validate $f
> +}
> +
> +# preallocate a file, then corrupt it by changing it to a regular file
> +corrupt_prealloc_to_reg() {
> +	local f=$SCRATCH_MNT/prealloc
> +	$XFS_IO_PROG -fc "falloc 0 12k" $f
> +	local ino=$(get_ino $f)
> +	_fsv_enable $f
> +	_scratch_unmount
> +	# set extent type from prealloc (2) to reg (1)
> +	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 0 -f type -v 1 $SCRATCH_DEV >/dev/null 2>&1
> +	_scratch_mount
> +	validate $f
> +}
> +
> +# corrupt a regular file by changing the type to preallocated
> +corrupt_reg_to_prealloc() {
> +	local f=$SCRATCH_MNT/reg
> +	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
> +	local ino=$(get_ino $f)
> +	_fsv_enable $f
> +	_scratch_unmount
> +	# set type from reg (1) to prealloc (2)
> +	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 0 -f type -v 2 $SCRATCH_DEV >/dev/null 2>&1
> +	_scratch_mount
> +	validate $f
> +}
> +
> +# corrupt a file by punching a hole
> +corrupt_punch_hole() {
> +	local f=$SCRATCH_MNT/punch
> +	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
> +	local ino=$(get_ino $f)
> +	# make a new extent in the middle, sync so the writes don't coalesce
> +	$XFS_IO_PROG -c sync $SCRATCH_MNT
> +	$XFS_IO_PROG -fc "pwrite -q -S 0x59 4096 4096" $f
> +	_fsv_enable $f
> +	_scratch_unmount
> +	# change disk_bytenr to 0, representing a hole
> +	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr -v 0 $SCRATCH_DEV > /dev/null 2>&1
> +	_scratch_mount
> +	validate $f
> +}
> +
> +# plug hole
> +corrupt_plug_hole() {
> +	local f=$SCRATCH_MNT/plug
> +	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
> +	local ino=$(get_ino $f)
> +	$XFS_IO_PROG -fc "falloc 4k 4k" $f
> +	_fsv_enable $f
> +	_scratch_unmount
> +	# change disk_bytenr to some value, plugging the hole
> +	$BTRFS_CORRUPT_BLOCK_PROG -i $ino -x 4096 -f disk_bytenr -v 13639680 $SCRATCH_DEV > /dev/null 2>&1
> +	_scratch_mount
> +	validate $f
> +}
> +
> +# corrupt the fsverity descriptor item indiscriminately (causes EINVAL)
> +corrupt_verity_descriptor() {
> +	local f=$SCRATCH_MNT/desc
> +	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
> +	local ino=$(get_ino $f)
> +	_fsv_enable $f
> +	_scratch_unmount
> +	# key for the descriptor item is <inode, BTRFS_VERITY_DESC_ITEM_KEY, 1>,
> +	# 88 is X. So we write 5 Xs to the start of the descriptor
> +	$BTRFS_CORRUPT_BLOCK_PROG -r 5 -I $ino,36,1 -v 88 -o 0 -b 5 $SCRATCH_DEV > /dev/null 2>&1
> +	_scratch_mount
> +	validate $f
> +}
> +
> +# specifically target the root hash in the descriptor (causes EIO)
> +corrupt_root_hash() {
> +	local f=$SCRATCH_MNT/roothash
> +	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
> +	local ino=$(get_ino $f)
> +	_fsv_enable $f
> +	_scratch_unmount
> +	$BTRFS_CORRUPT_BLOCK_PROG -r 5 -I $ino,36,1 -v 88 -o 16 -b 1 $SCRATCH_DEV > /dev/null 2>&1
> +	_scratch_mount
> +	validate $f
> +}
> +
> +# corrupt the Merkle tree data itself
> +corrupt_merkle_tree() {
> +	local f=$SCRATCH_MNT/merkle
> +	$XFS_IO_PROG -fc "pwrite -q -S 0x58 0 12288" $f
> +	local ino=$(get_ino $f)
> +	_fsv_enable $f
> +	_scratch_unmount
> +	# key for the descriptor item is <inode, BTRFS_VERITY_MERKLE_ITEM_KEY, 0>,
> +	# 88 is X. So we write 5 Xs to somewhere in the middle of the first
> +	# merkle item
> +	$BTRFS_CORRUPT_BLOCK_PROG -r 5 -I $ino,37,0 -v 88 -o 100 -b 5 $SCRATCH_DEV > /dev/null 2>&1
> +	_scratch_mount
> +	validate $f
> +}
> +
> +# real QA test starts here
> +_scratch_mkfs >/dev/null
> +_scratch_mount
> +
> +corrupt_inline
> +corrupt_prealloc_to_reg
> +corrupt_reg_to_prealloc
> +corrupt_punch_hole
> +corrupt_plug_hole
> +corrupt_verity_descriptor
> +corrupt_root_hash
> +corrupt_merkle_tree
> +
> +status=0
> +exit
> diff --git a/tests/btrfs/290.out b/tests/btrfs/290.out
> new file mode 100644
> index 00000000..056b114b
> --- /dev/null
> +++ b/tests/btrfs/290.out
> @@ -0,0 +1,25 @@
> +QA output created by 290
> +inl
> +pread: Input/output error
> +pread: Input/output error
> +prealloc
> +pread: Input/output error
> +pread: Input/output error
> +reg
> +pread: Input/output error
> +pread: Input/output error
> +punch
> +pread: Input/output error
> +pread: Input/output error
> +plug
> +pread: Input/output error
> +pread: Input/output error
> +desc
> +SCRATCH_MNT/desc: Invalid argument
> +SCRATCH_MNT/desc: Invalid argument
> +roothash
> +pread: Input/output error
> +pread: Input/output error
> +merkle
> +pread: Input/output error
> +pread: Input/output error
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index 331dd432..13051562 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -238,3 +238,4 @@
>  233 auto quick subvolume
>  234 auto quick compress rw
>  235 auto quick send
> +290 auto quick verity
> -- 
> 2.30.2

Re: [PATCH v4 2/4] generic/574: corrupt btrfs merkle tree data

[Eryu Guan]

On Wed, May 05, 2021 at 02:04:44PM -0700, Boris Burkov wrote:
> generic/574 has tests for corrupting the merkle tree data stored by the
> filesystem. Since btrfs uses a different scheme for storing this data,
> the existing logic for corrupting it doesn't work out of the box. Adapt
> it to properly corrupt btrfs merkle items.
> 
> This test relies on the btrfs implementation of fsverity in the patches
> titled:
> btrfs: initial fsverity support
> btrfs: check verity for reads of inline extents and holes
> btrfs: fallback to buffered io for verity files
> 
> A fix for fiemap in the patch titled:
> btrfs: return whole extents in fiemap
> 
> and on btrfs-corrupt-block for corruption in the patches titled:
> btrfs-progs: corrupt generic item data with btrfs-corrupt-block
> btrfs-progs: expand corrupt_file_extent in btrfs-corrupt-block
> 
> Signed-off-by: Boris Burkov <boris@bur.io>
> ---
>  common/verity     | 18 ++++++++++++++++++
>  tests/generic/574 |  5 +++++
>  2 files changed, 23 insertions(+)
> 
> diff --git a/common/verity b/common/verity
> index d2c1ea24..1636e88b 100644
> --- a/common/verity
> +++ b/common/verity
> @@ -315,6 +315,24 @@ _fsv_scratch_corrupt_merkle_tree()
>  		(( offset += ($(_get_filesize $file) + 65535) & ~65535 ))
>  		_fsv_scratch_corrupt_bytes $file $offset
>  		;;
> +	btrfs)
> +		local ino=$(stat -c '%i' $file)
> +		_scratch_unmount
> +		local byte=""
> +		while read -n 1 byte; do
> +			if [ -z $byte ]; then
> +				break
> +			fi
> +			local ascii=$(printf "%d" "'$byte'")
> +			# This command will find a Merkle tree item for the inode (-I $ino,37,0)
> +			# in the default filesystem tree (-r 5) and corrupt one byte (-b 1) at
> +			# $offset (-o $offset) with the ascii representation of the byte we read
> +			# (-v $ascii)
> +			$BTRFS_CORRUPT_BLOCK_PROG -r 5 -I $ino,37,0 -v $ascii -o $offset -b 1 $SCRATCH_DEV
> +			(( offset += 1 ))
> +		done
> +		_scratch_mount
> +		;;
>  	*)
>  		_fail "_fsv_scratch_corrupt_merkle_tree() unimplemented on $FSTYP"
>  		;;
> diff --git a/tests/generic/574 b/tests/generic/574
> index 1e296618..e4370dae 100755
> --- a/tests/generic/574
> +++ b/tests/generic/574
> @@ -43,6 +43,11 @@ _scratch_mount
>  fsv_orig_file=$SCRATCH_MNT/file
>  fsv_file=$SCRATCH_MNT/file.fsv
>  
> +# utility needed for corrupting Merkle data itself in btrfs
> +if [ $FSTYP == "btrfs" ]; then
> +	_require_command "$BTRFS_CORRUPT_BLOCK_PROG" btrfs_corrupt_block
> +fi

I don't think this is needed, as _require_scratch_verity() already did
this check.

Thanks,
Eryu

> +
>  setup_zeroed_file()
>  {
>  	local len=$1
> -- 
> 2.30.2

Re: [PATCH v4 3/4] btrfs: test verity orphans with dmlogwrites

[Eryu Guan]

On Wed, May 05, 2021 at 02:04:45PM -0700, Boris Burkov wrote:
> The behavior of orphans is most interesting across mounts, interrupted
> at arbitrary points during fsverity enable. To cover as many such cases
> as possible, use dmlogwrites and dmsnapshot as in
> log-writes/replay-individual.sh. At each log entry, we want to assert a
> somewhat complicated invariant:
> 
> If verity has not yet started: an orphan indicates that verity has
> started.
> If verity has started: mount should handle the orphan and blow away
> verity data: expect 0 merkle items after mounting the snapshot dev. If
> we can measure the file, verity has finished.
> If verity has finished: the orphan should be gone, so mount should not
> blow away merkle items. Expect the same number of merkle items before
> and after mounting the snapshot dev.
> 
> Note that this relies on grepping btrfs inspect-internal dump-tree.
> Until btrfs-progs has the ability to print the new Merkle items, they
> will show up as UNKNOWN.36/37.
> 
> Signed-off-by: Boris Burkov <boris@bur.io>
> ---
>  tests/btrfs/291     | 157 ++++++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/291.out |   2 +
>  tests/btrfs/group   |   1 +
>  3 files changed, 160 insertions(+)
>  create mode 100755 tests/btrfs/291
>  create mode 100644 tests/btrfs/291.out
> 
> diff --git a/tests/btrfs/291 b/tests/btrfs/291
> new file mode 100755
> index 00000000..eab3a495
> --- /dev/null
> +++ b/tests/btrfs/291
> @@ -0,0 +1,157 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2021 Facebook, Inc. All Rights Reserved.
> +#
> +# FS QA Test 291
> +#
> +# Test btrfs consistency after each FUA while enabling verity on a file
> +# This test works by following the pattern in log-writes/replay-individual.sh:
> +# 1. run a workload (verity + sync) while logging to the log device
> +# 2. replay an entry to the replay device
> +# 3. snapshot the replay device to the snapshot device
> +# 4. run destructive tests on the snapshot device (e.g. mount with orphans)
> +# 5. goto 2
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +	cd /
> +	_log_writes_cleanup &> /dev/null
> +	rm -f $tmp.*
> +	$LVM_PROG vgremove -f -y $vgname >>$seqres.full 2>&1
> +	losetup -d $loop_dev >>$seqres.full 2>&1
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/attr
> +. ./common/dmlogwrites
> +. ./common/verity
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +_supported_fs btrfs
> +
> +_require_scratch
> +_require_log_writes
> +_require_dm_target snapshot
> +_require_command $LVM_PROG lvm
> +_require_scratch_verity

Also need

_require_btrfs_command inspect-internal dump-tree
_require_test_program "log-writes/replay-log"

> +
> +sync_loop() {
> +	i=$1
> +	[ -z "$i" ] && _fail "sync loop needs a number of iterations"
> +	while [ $i -gt 0 ]
> +	do
> +		$XFS_IO_PROG -c sync $SCRATCH_MNT
> +		let i-=1
> +	done
> +}
> +
> +dump_tree() {
> +	local dev=$1
> +	$BTRFS_UTIL_PROG inspect-internal dump-tree $dev
> +}
> +
> +count_item() {
> +	local dev=$1
> +	local item=$2
> +	dump_tree $dev | grep -c $item
> +}
> +
> +_log_writes_init $SCRATCH_DEV
> +_log_writes_mkfs
> +_log_writes_mount
> +
> +f=$SCRATCH_MNT/fsv
> +MB=$((1024 * 1024))
> +$XFS_IO_PROG -fc "pwrite -q 0 $((10 * $MB))" $f
> +$XFS_IO_PROG -c sync $SCRATCH_MNT
> +sync_loop 10 &
> +_fsv_enable $f
> +$XFS_IO_PROG -c sync $SCRATCH_MNT
> +
> +_log_writes_unmount
> +_log_writes_remove
> +
> +$XFS_IO_PROG -fc "pwrite -q -S 0 $((8192 * $MB)) $MB" $tmp.loop-file

Better to create fs image file for loop dev under $TEST_DIR. Then we
need _require_test as well.

Thanks,
Eryu

> +loop_dev=$(losetup -f --show $tmp.loop-file)
> +vgname=vg_replay
> +lvname=lv_replay
> +replay_dev=/dev/mapper/vg_replay-lv_replay
> +snapname=lv_snap
> +snap_dev=/dev/mapper/vg_replay-$snapname
> +
> +$LVM_PROG vgcreate -f $vgname $loop_dev >>$seqres.full 2>&1 || _fail "failed to vgcreate $vgname"
> +$LVM_PROG lvcreate -L 4G -n $lvname $vgname -y >>$seqres.full 2>&1 || \
> +	_fail "failed to lvcreate $lvname"
> +$UDEV_SETTLE_PROG >>$seqres.full 2>&1
> +
> +replay_log_prog=$here/src/log-writes/replay-log
> +num_entries=$($replay_log_prog --log $LOGWRITES_DEV --num-entries)
> +entry=$($replay_log_prog --log $LOGWRITES_DEV --replay $replay_dev --find --end-mark mkfs | cut -d@ -f1)
> +$replay_log_prog --log $LOGWRITES_DEV --replay $replay_dev --limit $entry || \
> +	_fail "failed to replay to start entry $entry"
> +let entry+=1
> +
> +# state = 0: verity hasn't started
> +# state = 1: verity underway
> +# state = 2: verity done
> +state=0
> +while [ $entry -lt $num_entries ];
> +do
> +	$replay_log_prog --limit 1 --log $LOGWRITES_DEV --replay $replay_dev --start $entry || \
> +		_fail "failed to take replay step at entry: $entry"
> +
> +	$LVM_PROG lvcreate -s -L 4M -n $snapname $vgname/$lvname >>$seqres.full 2>&1 || \
> +		_fail "Failed to create snapshot"
> +	$UDEV_SETTLE_PROG >>$seqres.full 2>&1
> +
> +	orphan=$(count_item $snap_dev ORPHAN)
> +	if [ $state -eq 0 ]; then
> +		[ $orphan -gt 0 ] && state=1
> +	fi
> +
> +	pre_mount=$(count_item $snap_dev UNKNOWN.3[67])
> +	_mount $snap_dev $SCRATCH_MNT || _fail "mount failed at entry $entry"
> +	fsverity measure $SCRATCH_MNT/fsv >>$seqres.full 2>&1
> +	measured=$?
> +	umount $SCRATCH_MNT
> +	[ $state -eq 1 ] && [ $measured -eq 0 ] && state=2
> +	[ $state -eq 2 ] && ([ $measured -eq 0 ] || _fail "verity done, but measurement failed at entry $entry")
> +	post_mount=$(count_item $snap_dev UNKNOWN.3[67])
> +
> +	echo "entry: $entry, state: $state, orphan: $orphan, pre_mount: $pre_mount, post_mount: $post_mount" >> $seqres.full
> +
> +	if [ $state -eq 1 ]; then
> +		[ $post_mount -eq 0 ] || \
> +			_fail "mount failed to clear under-construction merkle items pre: $pre_mount, post: $post_mount at entry $entry";
> +	fi
> +	if [ $state -eq 2 ]; then
> +		[ $pre_mount -gt 0 ] || \
> +			_fail "expected to have verity items before mount at entry $entry"
> +		[ $pre_mount -eq $post_mount ] || \
> +			_fail "mount cleared merkle items after verity was enabled $pre_mount vs $post_mount at entry $entry";
> +	fi
> +
> +	let entry+=1
> +	$LVM_PROG lvremove $vgname/$snapname -y >>$seqres.full
> +done
> +
> +echo "Silence is golden"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/291.out b/tests/btrfs/291.out
> new file mode 100644
> index 00000000..04605c70
> --- /dev/null
> +++ b/tests/btrfs/291.out
> @@ -0,0 +1,2 @@
> +QA output created by 291
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index 13051562..cc5a811e 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -239,3 +239,4 @@
>  234 auto quick compress rw
>  235 auto quick send
>  290 auto quick verity
> +291 auto verity
> -- 
> 2.30.2

Re: [PATCH v4 4/4] generic: test fs-verity EFBIG scenarios

[Eryu Guan]

On Wed, May 05, 2021 at 02:04:46PM -0700, Boris Burkov wrote:
> btrfs, ext4, and f2fs cache the Merkle tree past EOF, which restricts
> the maximum file size beneath the normal maximum. Test the logic in
> those filesystems against files with sizes near the maximum.
> 
> To work properly, this does require some understanding of the practical
> but not standardized layout of the Merkle tree. This is a bit unpleasant
> and could make the test incorrect in the future, if the implementation
> changes. On the other hand, it feels quite useful to test this tricky
> edge case. It could perhaps be made more generic by adding some ioctls
> to let the file system communicate the maximum file size for a verity
> file or some information about the storage of the Merkle tree.
> 
> Signed-off-by: Boris Burkov <boris@bur.io>
> ---
>  tests/generic/632     | 86 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/632.out |  7 ++++
>  tests/generic/group   |  1 +
>  3 files changed, 94 insertions(+)
>  create mode 100755 tests/generic/632
>  create mode 100644 tests/generic/632.out
> 
> diff --git a/tests/generic/632 b/tests/generic/632
> new file mode 100755
> index 00000000..5a5ed576
> --- /dev/null
> +++ b/tests/generic/632
> @@ -0,0 +1,86 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2021 Facebook, Inc.  All Rights Reserved.
> +#
> +# FS QA Test 632
> +#
> +# Test some EFBIG scenarios with very large files.
> +# To create the files, use pwrite with an offset close to the
> +# file system's max file size.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/verity
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs ext4 f2fs

A generic test should '_supported_fs generic', and use proper _require
rules to filter unsupported filesystems.

> +_require_test
> +_require_math
> +_require_scratch_verity
> +
> +_scratch_mkfs_verity &>> $seqres.full
> +_scratch_mount
> +
> +fsv_file=$SCRATCH_MNT/file.fsv
> +
> +max_sz=$(_get_max_file_size)
> +_fsv_scratch_begin_subtest "way too big: fail on first merkle block"
> +# have to go back by 4096 from max to not hit the fsverity MAX_DEPTH check.
> +$XFS_IO_PROG -fc "pwrite -q $(($max_sz - 4096)) 1" $fsv_file
> +_fsv_enable $fsv_file |& _filter_scratch
> +
> +# The goal of this second test is to make a big enough file that we trip the
> +# EFBIG codepath, but not so big that we hit it immediately as soon as we try
> +# to write a Merkle leaf. Because of the layout of the Merkle tree that
> +# fs-verity uses, this is a bit complicated to compute dynamically.
> +
> +# The layout of the Merkle tree has the leaf nodes last, but writes them first.
> +# To get an interesting overflow, we need the start of L0 to be < MAX but the
> +# end of the merkle tree (EOM) to be past MAX. Ideally, the start of L0 is only
> +# just smaller than MAX, so that we don't have to write many blocks to blow up.
> +
> +# 0                        EOF round-to-64k L7L6L5 L4   L3    L2    L1  L0 MAX  EOM
> +# |-------------------------|               ||-|--|---|----|-----|------|--|!!!!!|
> +
> +# Given this structure, we can compute the size of the file that yields the
> +# desired properties:
> +# sz + 64k + sz/128^8 + sz/128^7 + ... + sz/128^2 < MAX
> +# (128^8)sz + (128^8)64k + sz + (128)sz + (128^2)sz + ... + (128^6)sz < (128^8)MAX
> +# sz(128^8 + 128^6 + 128^5 + 128^4 + 128^3 + 128^2 + 128 + 1) < (128^8)(MAX - 64k)
> +# sz < (128^8/(128^8 + (128^6 + ... 1))(MAX - 64k)
> +#
> +# Do the actual caclulation with 'bc' and 20 digits of precision.
> +set -f
> +calc="scale=20; ($max_sz - 65536) * ((128^8) / (1 + 128 + 128^2 + 128^3 + 128^4 + 128^5 + 128^6 + 128^8))"
> +sz=$(echo $calc | $BC -q | cut -d. -f1)
> +set +f

Now sure why set -f is needed here, add some comments about it as well?

Thanks,
Eryu

> +
> +
> +_fsv_scratch_begin_subtest "still too big: fail on first invalid merkle block"
> +$XFS_IO_PROG -fc "pwrite -q $sz 1" $fsv_file
> +_fsv_enable $fsv_file |& _filter_scratch
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/632.out b/tests/generic/632.out
> new file mode 100644
> index 00000000..59602c24
> --- /dev/null
> +++ b/tests/generic/632.out
> @@ -0,0 +1,7 @@
> +QA output created by 632
> +
> +# way too big: fail on first merkle block
> +ERROR: FS_IOC_ENABLE_VERITY failed on 'SCRATCH_MNT/file.fsv': File too large
> +
> +# still too big: fail on first invalid merkle block
> +ERROR: FS_IOC_ENABLE_VERITY failed on 'SCRATCH_MNT/file.fsv': File too large
> diff --git a/tests/generic/group b/tests/generic/group
> index ab00cc04..76d46e86 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -634,3 +634,4 @@
>  629 auto quick rw copy_range
>  630 auto quick rw dedupe clone
>  631 auto rw overlay rename
> +632 auto quick verity
> -- 
> 2.30.2

Re: [PATCH v4 0/4] tests for btrfs fsverity

[Eric Biggers]

On Wed, May 05, 2021 at 02:04:42PM -0700, Boris Burkov wrote:
> This patchset provides tests for fsverity support in btrfs.
> 
> It includes modifications for generic tests to pass with btrfs as well
> as new tests.
> 

Can you please mention which xfstests commit this patchset applies to?

- Eric

Re: [PATCH v4 4/4] generic: test fs-verity EFBIG scenarios

[Eric Biggers]

On Wed, May 05, 2021 at 02:04:46PM -0700, Boris Burkov wrote:
> diff --git a/tests/generic/632 b/tests/generic/632
> new file mode 100755
> index 00000000..5a5ed576
> --- /dev/null
> +++ b/tests/generic/632
> @@ -0,0 +1,86 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2021 Facebook, Inc.  All Rights Reserved.
> +#
> +# FS QA Test 632
> +#
> +# Test some EFBIG scenarios with very large files.
> +# To create the files, use pwrite with an offset close to the
> +# file system's max file size.

Can you please make this comment properly describe the purpose of this test?
As-is it doesn't mention that it is related to fs-verity at all, let alone to
specific filesystems' implementations of fs-verity.

> +max_sz=$(_get_max_file_size)
> +_fsv_scratch_begin_subtest "way too big: fail on first merkle block"
> +# have to go back by 4096 from max to not hit the fsverity MAX_DEPTH check.

What is meant by the "fsverity MAX_DEPTH" check?

> +$XFS_IO_PROG -fc "pwrite -q $(($max_sz - 4096)) 1" $fsv_file
> +_fsv_enable $fsv_file |& _filter_scratch

Using the "truncate" xfs_io command instead of "pwrite" would probably make more
sense here, as the goal is to just create a file of a specific size.

> +
> +# The goal of this second test is to make a big enough file that we trip the
> +# EFBIG codepath, but not so big that we hit it immediately as soon as we try
> +# to write a Merkle leaf. Because of the layout of the Merkle tree that
> +# fs-verity uses, this is a bit complicated to compute dynamically.
> +
> +# The layout of the Merkle tree has the leaf nodes last, but writes them first.
> +# To get an interesting overflow, we need the start of L0 to be < MAX but the
> +# end of the merkle tree (EOM) to be past MAX. Ideally, the start of L0 is only
> +# just smaller than MAX, so that we don't have to write many blocks to blow up.
> +
> +# 0                        EOF round-to-64k L7L6L5 L4   L3    L2    L1  L0 MAX  EOM
> +# |-------------------------|               ||-|--|---|----|-----|------|--|!!!!!|
> +
> +# Given this structure, we can compute the size of the file that yields the
> +# desired properties:
> +# sz + 64k + sz/128^8 + sz/128^7 + ... + sz/128^2 < MAX
> +# (128^8)sz + (128^8)64k + sz + (128)sz + (128^2)sz + ... + (128^6)sz < (128^8)MAX
> +# sz(128^8 + 128^6 + 128^5 + 128^4 + 128^3 + 128^2 + 128 + 1) < (128^8)(MAX - 64k)
> +# sz < (128^8/(128^8 + (128^6 + ... 1))(MAX - 64k)
> +#
> +# Do the actual caclulation with 'bc' and 20 digits of precision.

This calculation isn't completely accurate because it doesn't round the levels
to a block boundary.  Nor does it consider that the 64K is an alignment rather
than a fixed amount added.

But for the test you don't need the absolute largest file whose level 1 doesn't
exceed the limit, but rather just one almost that large.

So it would be okay to add 64K as a fixed amount, along with 4K for every level
on top of the 'sz/128^(level+1)' you already have, to get an over-estimate of
the amount of extra space needed to cache the Merkle tree.

But please make it clear that it's an over-estimate, and hence an under-estimate
of the file size desired for the test.

Also please document that this is all assuming SHA-256 with 4K blocks, and also
that the maximum file size is assumed to fit in 64 bits; hence the consideration
of 8 levels is sufficient.

- Eric

Re: [PATCH v4 4/4] generic: test fs-verity EFBIG scenarios

[Boris Burkov]

On Tue, May 25, 2021 at 01:24:11PM -0700, Eric Biggers wrote:
> On Wed, May 05, 2021 at 02:04:46PM -0700, Boris Burkov wrote:
> > diff --git a/tests/generic/632 b/tests/generic/632
> > new file mode 100755
> > index 00000000..5a5ed576
> > --- /dev/null
> > +++ b/tests/generic/632
> > @@ -0,0 +1,86 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2021 Facebook, Inc.  All Rights Reserved.
> > +#
> > +# FS QA Test 632
> > +#
> > +# Test some EFBIG scenarios with very large files.
> > +# To create the files, use pwrite with an offset close to the
> > +# file system's max file size.
> 
> Can you please make this comment properly describe the purpose of this test?
> As-is it doesn't mention that it is related to fs-verity at all, let alone to
> specific filesystems' implementations of fs-verity.

Sorry for disappearing on this one for a while.

Oops, good point. In addressing your and Eryu's points, I realized that
this isn't really a generic test, since as you say, it assumes the
filesystem's implementation. Further, I think it is plausible for an fs
to cache the Merkle tree pages some other way which wouldn't need to
EFBIG for large files. With that said, I do think it's a useful test of
an edge case I got wrong several times in the btrfs implementation.

I am leaning towards making this a btrfs specific test. Just wanted to
double check with you if you think ext4 and f2fs would benefit from
running this test too..

> 
> > +max_sz=$(_get_max_file_size)
> > +_fsv_scratch_begin_subtest "way too big: fail on first merkle block"
> > +# have to go back by 4096 from max to not hit the fsverity MAX_DEPTH check.
> 
> What is meant by the "fsverity MAX_DEPTH" check?

If you use $max_sz or $max_sz-1 (or anything bigger than $max_sz-4096)
the vfs fsverity code will conclude the tree will exceed MAX_LEVELS. I
got LEVELS and DEPTH mixed up.

> 
> > +$XFS_IO_PROG -fc "pwrite -q $(($max_sz - 4096)) 1" $fsv_file
> > +_fsv_enable $fsv_file |& _filter_scratch
> 
> Using the "truncate" xfs_io command instead of "pwrite" would probably make more
> sense here, as the goal is to just create a file of a specific size.

In my memory, truncate didn't work for btrfs, but it took me a while to
get this to work, so I might have made some silly mistake early on with
truncate. I'll try again to be sure.

> 
> > +
> > +# The goal of this second test is to make a big enough file that we trip the
> > +# EFBIG codepath, but not so big that we hit it immediately as soon as we try
> > +# to write a Merkle leaf. Because of the layout of the Merkle tree that
> > +# fs-verity uses, this is a bit complicated to compute dynamically.
> > +
> > +# The layout of the Merkle tree has the leaf nodes last, but writes them first.
> > +# To get an interesting overflow, we need the start of L0 to be < MAX but the
> > +# end of the merkle tree (EOM) to be past MAX. Ideally, the start of L0 is only
> > +# just smaller than MAX, so that we don't have to write many blocks to blow up.
> > +
> > +# 0                        EOF round-to-64k L7L6L5 L4   L3    L2    L1  L0 MAX  EOM
> > +# |-------------------------|               ||-|--|---|----|-----|------|--|!!!!!|
> > +
> > +# Given this structure, we can compute the size of the file that yields the
> > +# desired properties:
> > +# sz + 64k + sz/128^8 + sz/128^7 + ... + sz/128^2 < MAX
> > +# (128^8)sz + (128^8)64k + sz + (128)sz + (128^2)sz + ... + (128^6)sz < (128^8)MAX
> > +# sz(128^8 + 128^6 + 128^5 + 128^4 + 128^3 + 128^2 + 128 + 1) < (128^8)(MAX - 64k)
> > +# sz < (128^8/(128^8 + (128^6 + ... 1))(MAX - 64k)
> > +#
> > +# Do the actual caclulation with 'bc' and 20 digits of precision.
> 
> This calculation isn't completely accurate because it doesn't round the levels
> to a block boundary.  Nor does it consider that the 64K is an alignment rather
> than a fixed amount added.
> 
> But for the test you don't need the absolute largest file whose level 1 doesn't
> exceed the limit, but rather just one almost that large.
> 
> So it would be okay to add 64K as a fixed amount, along with 4K for every level
> on top of the 'sz/128^(level+1)' you already have, to get an over-estimate of
> the amount of extra space needed to cache the Merkle tree.
> 
> But please make it clear that it's an over-estimate, and hence an under-estimate
> of the file size desired for the test.
> 
> Also please document that this is all assuming SHA-256 with 4K blocks, and also
> that the maximum file size is assumed to fit in 64 bits; hence the consideration
> of 8 levels is sufficient.

Agreed with all of this, will do.

> 
> - Eric

Re: [PATCH v4 4/4] generic: test fs-verity EFBIG scenarios

[Eric Biggers]

On Fri, Sep 10, 2021 at 04:26:42PM -0700, Boris Burkov wrote:
> 
> I am leaning towards making this a btrfs specific test. Just wanted to
> double check with you if you think ext4 and f2fs would benefit from
> running this test too..

It's applicable to ext4 and f2fs too, so it probably should be kept as a generic
test.  Just make sure that filesystems have to "opt in" to it (with a new
function _require_fsverity_max_file_size_limit() in common/verity or something),
since it's probably not going to be applicable to every filesystem that
implements fs-verity.

- Eric