* [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size
@ 2022-12-23 1:05 Eric Biggers
2022-12-23 1:05 ` [PATCH v2 01/10] common/verity: add and use _fsv_can_enable() Eric Biggers
` (9 more replies)
0 siblings, 10 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
This series updates the verity xfstests to eliminate implicit
assumptions that 'merkle_tree_block_size == fs_block_size == page_size
== 4096', and to provide some test coverage for cases where
merkle_tree_block_size differs from fs_block_size and/or page_size. It
doesn't add any new test scripts, but it does update some of the
existing test scripts to test multiple block sizes.
This goes along with my kernel patch series
"fsverity: support for non-4K pages"
(https://lore.kernel.org/linux-fsdevel/20221028224539.171818-1-ebiggers@kernel.org/T/#u).
However, it's not necessary to wait for that kernel patch series to be
applied before applying this xfstests patch series.
Changed since v1:
- Adjusted the output of generic/574, generic/575, and generic/624
slightly to avoid confusion.
Eric Biggers (10):
common/verity: add and use _fsv_can_enable()
common/verity: set FSV_BLOCK_SIZE to an appropriate value
common/verity: use FSV_BLOCK_SIZE by default
common/verity: add _filter_fsverity_digest()
generic/572: support non-4K Merkle tree block size
generic/573: support non-4K Merkle tree block size
generic/577: support non-4K Merkle tree block size
generic/574: test multiple Merkle tree block sizes
generic/624: test multiple Merkle tree block sizes
generic/575: test 1K Merkle tree block size
common/verity | 84 +++++++++++++++-----
tests/generic/572 | 21 ++---
tests/generic/572.out | 10 +--
tests/generic/573 | 8 +-
tests/generic/574 | 177 ++++++++++++++++++++++++++----------------
tests/generic/574.out | 83 ++------------------
tests/generic/575 | 57 ++++++++++----
tests/generic/575.out | 8 +-
tests/generic/577 | 24 +++---
tests/generic/577.out | 10 +--
tests/generic/624 | 119 ++++++++++++++++++++--------
tests/generic/624.out | 15 ++--
12 files changed, 348 insertions(+), 268 deletions(-)
base-commit: e263104046712af5fb5dcc7d289ac3fa5f14b764
--
2.39.0
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 01/10] common/verity: add and use _fsv_can_enable()
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
@ 2022-12-23 1:05 ` Eric Biggers
2022-12-23 1:05 ` [PATCH v2 02/10] common/verity: set FSV_BLOCK_SIZE to an appropriate value Eric Biggers
` (8 subsequent siblings)
9 siblings, 0 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
From: Eric Biggers <ebiggers@google.com>
Replace _fsv_have_hash_algorithm() with a more general function
_fsv_can_enable() which checks whether 'fsverity enable' with the given
parameters works. For now it is just used with --hash-alg or with no
parameters, but soon it will be used with --block-size too.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
common/verity | 17 ++++++-----------
tests/generic/575 | 2 +-
tests/generic/577 | 2 +-
3 files changed, 8 insertions(+), 13 deletions(-)
diff --git a/common/verity b/common/verity
index f98dcb07..1a53a7ea 100644
--- a/common/verity
+++ b/common/verity
@@ -42,13 +42,7 @@ _require_scratch_verity()
# The filesystem may have fs-verity enabled but not actually usable by
# default. E.g., ext4 only supports verity on extent-based files, so it
# doesn't work on ext3-style filesystems. So, try actually using it.
- echo foo > $SCRATCH_MNT/tmpfile
- _disable_fsverity_signatures
- _fsv_enable $SCRATCH_MNT/tmpfile
- local status=$?
- _restore_prev_fsverity_signatures
- rm -f $SCRATCH_MNT/tmpfile
- if (( $status != 0 )); then
+ if ! _fsv_can_enable $SCRATCH_MNT/tmpfile; then
_notrun "$FSTYP verity isn't usable by default with these mkfs options"
fi
@@ -256,15 +250,16 @@ _fsv_create_enable_file()
_fsv_enable "$file" "$@"
}
-_fsv_have_hash_algorithm()
+_fsv_can_enable()
{
- local hash_alg=$1
- local test_file=$2
+ local test_file=$1
+ shift
+ local params=("$@")
_disable_fsverity_signatures
rm -f $test_file
head -c 4096 /dev/zero > $test_file
- _fsv_enable --hash-alg=$hash_alg $test_file &>> $seqres.full
+ _fsv_enable $test_file "${params[@]}" &>> $seqres.full
local status=$?
_restore_prev_fsverity_signatures
rm -f $test_file
diff --git a/tests/generic/575 b/tests/generic/575
index ffa6b61d..0ece8826 100755
--- a/tests/generic/575
+++ b/tests/generic/575
@@ -71,7 +71,7 @@ test_alg()
_fsv_scratch_begin_subtest "Check for expected measurement values ($alg)"
- if ! _fsv_have_hash_algorithm $alg $fsv_file; then
+ if ! _fsv_can_enable $fsv_file --hash-alg=$alg; then
if [ "$alg" = sha256 ]; then
_fail "Something is wrong - sha256 hash should always be available"
fi
diff --git a/tests/generic/577 b/tests/generic/577
index 5f7e0573..85d680df 100755
--- a/tests/generic/577
+++ b/tests/generic/577
@@ -112,7 +112,7 @@ _fsv_enable $fsv_file --signature=$sigfile.salted --salt=abcd
cmp $fsv_file $fsv_orig_file
echo -e "\n# Testing non-default hash algorithm"
-if _fsv_have_hash_algorithm sha512 $fsv_file; then
+if _fsv_can_enable $fsv_file --hash-alg=sha512; then
reset_fsv_file
_fsv_sign $fsv_orig_file $sigfile.sha512 --key=$keyfile \
--cert=$certfile --hash-alg=sha512 > /dev/null
--
2.39.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 02/10] common/verity: set FSV_BLOCK_SIZE to an appropriate value
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
2022-12-23 1:05 ` [PATCH v2 01/10] common/verity: add and use _fsv_can_enable() Eric Biggers
@ 2022-12-23 1:05 ` Eric Biggers
2022-12-23 1:05 ` [PATCH v2 03/10] common/verity: use FSV_BLOCK_SIZE by default Eric Biggers
` (7 subsequent siblings)
9 siblings, 0 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
From: Eric Biggers <ebiggers@google.com>
In order to maximize the chance that the verity tests can actually be
run, FSV_BLOCK_SIZE (the default Merkle tree size for the verity tests)
needs to be min(fs_block_size, page_size), not simply page_size. The
only reason that page_size was okay before was because the kernel only
supported merkle_tree_block_size == fs_block_size == page_size anyway.
But that is changing.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
common/verity | 32 +++++++++++++++++++++++++++-----
1 file changed, 27 insertions(+), 5 deletions(-)
diff --git a/common/verity b/common/verity
index 1a53a7ea..a94ebf8e 100644
--- a/common/verity
+++ b/common/verity
@@ -13,6 +13,11 @@ if [ "$FSTYP" == "btrfs" ]; then
fi
fi
+# Require fs-verity support on the scratch filesystem.
+#
+# FSV_BLOCK_SIZE will be set to a Merkle tree block size that is supported by
+# the filesystem. Other sizes may be supported too, but FSV_BLOCK_SIZE is the
+# only size that is guaranteed to work without any additional checks.
_require_scratch_verity()
{
_require_scratch
@@ -27,7 +32,7 @@ _require_scratch_verity()
# Try to mount the filesystem. If this fails then either the kernel
# isn't aware of fs-verity, or the mkfs options were not compatible with
- # verity (e.g. ext4 with block size != PAGE_SIZE).
+ # verity (e.g. ext4 with block size != PAGE_SIZE on old kernels).
if ! _try_scratch_mount &>>$seqres.full; then
_notrun "kernel is unaware of $FSTYP verity feature," \
"or mkfs options are not compatible with verity"
@@ -39,6 +44,27 @@ _require_scratch_verity()
_notrun "kernel $FSTYP isn't configured with verity support"
fi
+ # Select a default Merkle tree block size for when tests don't
+ # explicitly specify one.
+ #
+ # For consistency reasons, all 'fsverity' subcommands, including
+ # 'fsverity enable', default to 4K Merkle tree blocks. That's generally
+ # not ideal for tests, since it's possible that the filesystem doesn't
+ # support 4K blocks but does support another size. Specifically, the
+ # kernel originally supported only merkle_tree_block_size ==
+ # fs_block_size == page_size, and later it was updated to support
+ # merkle_tree_block_size <= min(fs_block_size, page_size).
+ #
+ # Therefore, we default to merkle_tree_block_size == min(fs_block_size,
+ # page_size). That maximizes the chance of verity actually working.
+ local fs_block_size=$(_get_block_size $SCRATCH_MNT)
+ local page_size=$(get_page_size)
+ if (( fs_block_size <= page_size )); then
+ FSV_BLOCK_SIZE=$fs_block_size
+ else
+ FSV_BLOCK_SIZE=$page_size
+ fi
+
# The filesystem may have fs-verity enabled but not actually usable by
# default. E.g., ext4 only supports verity on extent-based files, so it
# doesn't work on ext3-style filesystems. So, try actually using it.
@@ -47,10 +73,6 @@ _require_scratch_verity()
fi
_scratch_unmount
-
- # Merkle tree block size. Currently all filesystems only support
- # PAGE_SIZE for this. This is also the default for 'fsverity enable'.
- FSV_BLOCK_SIZE=$(get_page_size)
}
# Check for CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y, as well as the userspace
--
2.39.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 03/10] common/verity: use FSV_BLOCK_SIZE by default
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
2022-12-23 1:05 ` [PATCH v2 01/10] common/verity: add and use _fsv_can_enable() Eric Biggers
2022-12-23 1:05 ` [PATCH v2 02/10] common/verity: set FSV_BLOCK_SIZE to an appropriate value Eric Biggers
@ 2022-12-23 1:05 ` Eric Biggers
2022-12-23 1:05 ` [PATCH v2 04/10] common/verity: add _filter_fsverity_digest() Eric Biggers
` (6 subsequent siblings)
9 siblings, 0 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
From: Eric Biggers <ebiggers@google.com>
Make _fsv_enable() and _fsv_sign() default to FSV_BLOCK_SIZE if no block
size is explicitly specified, so that the individual tests don't have to
do this themselves. This overrides the fsverity-utils default of 4096
bytes, or the page size in older versions of fsverity-utils, both of
which may differ from FSV_BLOCK_SIZE.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
common/verity | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/common/verity b/common/verity
index a94ebf8e..1c706b80 100644
--- a/common/verity
+++ b/common/verity
@@ -249,7 +249,13 @@ _fsv_dump_signature()
_fsv_enable()
{
- $FSVERITY_PROG enable "$@"
+ local args=("$@")
+ # If the caller didn't explicitly specify a Merkle tree block size, then
+ # use FSV_BLOCK_SIZE.
+ if ! [[ " $*" =~ " --block-size" ]]; then
+ args+=("--block-size=$FSV_BLOCK_SIZE")
+ fi
+ $FSVERITY_PROG enable "${args[@]}"
}
_fsv_measure()
@@ -259,7 +265,13 @@ _fsv_measure()
_fsv_sign()
{
- $FSVERITY_PROG sign "$@"
+ local args=("$@")
+ # If the caller didn't explicitly specify a Merkle tree block size, then
+ # use FSV_BLOCK_SIZE.
+ if ! [[ " $*" =~ " --block-size" ]]; then
+ args+=("--block-size=$FSV_BLOCK_SIZE")
+ fi
+ $FSVERITY_PROG sign "${args[@]}"
}
# Generate a file, then enable verity on it.
--
2.39.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 04/10] common/verity: add _filter_fsverity_digest()
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
` (2 preceding siblings ...)
2022-12-23 1:05 ` [PATCH v2 03/10] common/verity: use FSV_BLOCK_SIZE by default Eric Biggers
@ 2022-12-23 1:05 ` Eric Biggers
2022-12-23 1:05 ` [PATCH v2 05/10] generic/572: support non-4K Merkle tree block size Eric Biggers
` (5 subsequent siblings)
9 siblings, 0 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
From: Eric Biggers <ebiggers@google.com>
Add a filter that replaces fs-verity digests with a fixed string. This
is needed because the fs-verity digests that some tests print are going
to start depending on the default Merkle tree block size.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
common/verity | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/common/verity b/common/verity
index 1c706b80..b88e839b 100644
--- a/common/verity
+++ b/common/verity
@@ -406,3 +406,11 @@ _require_fsverity_max_file_size_limit()
;;
esac
}
+
+# Replace fs-verity digests, as formatted by the 'fsverity' tool, with <digest>.
+# This function can be used by tests where fs-verity digests depend on the
+# default Merkle tree block size (FSV_BLOCK_SIZE).
+_filter_fsverity_digest()
+{
+ sed -E 's/\b(sha(256|512)):[a-f0-9]{64,}\b/\1:<digest>/'
+}
--
2.39.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 05/10] generic/572: support non-4K Merkle tree block size
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
` (3 preceding siblings ...)
2022-12-23 1:05 ` [PATCH v2 04/10] common/verity: add _filter_fsverity_digest() Eric Biggers
@ 2022-12-23 1:05 ` Eric Biggers
2022-12-23 1:05 ` [PATCH v2 06/10] generic/573: " Eric Biggers
` (4 subsequent siblings)
9 siblings, 0 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
From: Eric Biggers <ebiggers@google.com>
Update generic/572 to not implicitly assume that the Merkle tree block
size being used is 4096 bytes. Also remove an unused function.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
tests/generic/572 | 21 ++++++---------------
tests/generic/572.out | 10 +++-------
2 files changed, 9 insertions(+), 22 deletions(-)
diff --git a/tests/generic/572 b/tests/generic/572
index cded9ac6..d8071a34 100755
--- a/tests/generic/572
+++ b/tests/generic/572
@@ -9,7 +9,7 @@
# - conditions for enabling verity
# - verity files have correct contents and size
# - can't change contents of verity files, but can change metadata
-# - can retrieve a verity file's measurement via FS_IOC_MEASURE_VERITY
+# - can retrieve a verity file's digest via FS_IOC_MEASURE_VERITY
#
. ./common/preamble
_begin_fstest auto quick verity
@@ -48,15 +48,6 @@ verify_data_readable()
md5sum $file > /dev/null
}
-verify_data_unreadable()
-{
- local file=$1
-
- # try both reading just the first data block, and reading until EOF
- head -c $FSV_BLOCK_SIZE $file 2>&1 >/dev/null | filter_output
- md5sum $file |& filter_output
-}
-
_fsv_scratch_begin_subtest "Enabling verity on file with verity already enabled fails with EEXIST"
_fsv_create_enable_file $fsv_file
echo "(trying again)"
@@ -94,7 +85,7 @@ verify_data_readable $fsv_file
_fsv_scratch_begin_subtest "Enabling verity can be interrupted"
dd if=/dev/zero of=$fsv_file bs=1 count=0 seek=$((1 << 34)) status=none
start_time=$(date +%s)
-$FSVERITY_PROG enable $fsv_file &
+$FSVERITY_PROG enable --block-size=$FSV_BLOCK_SIZE $fsv_file &
sleep 0.5
kill %1
wait
@@ -106,7 +97,7 @@ fi
_fsv_scratch_begin_subtest "Enabling verity on file with verity already being enabled fails with EBUSY"
dd if=/dev/zero of=$fsv_file bs=1 count=0 seek=$((1 << 34)) status=none
start_time=$(date +%s)
-$FSVERITY_PROG enable $fsv_file &
+$FSVERITY_PROG enable --block-size=$FSV_BLOCK_SIZE $fsv_file &
sleep 0.5
_fsv_enable $fsv_file |& filter_output
kill %1
@@ -129,7 +120,7 @@ verify_data_readable $fsv_file
_fsv_scratch_begin_subtest "verity file can be measured"
_fsv_create_enable_file $fsv_file >> $seqres.full
-_fsv_measure $fsv_file
+_fsv_measure $fsv_file | _filter_fsverity_digest
_fsv_scratch_begin_subtest "verity file can be renamed"
_fsv_create_enable_file $fsv_file
@@ -170,8 +161,8 @@ verify_data_readable $fsv_file
# Test files <= 1 block in size. These are a bit of a special case since there
# are no hash blocks; the root hash is calculated directly over the data block.
+_fsv_scratch_begin_subtest "verity on small files"
for size in 1 $((FSV_BLOCK_SIZE - 1)) $FSV_BLOCK_SIZE; do
- _fsv_scratch_begin_subtest "verity on $size-byte file"
head -c $size /dev/urandom > $fsv_orig_file
cp $fsv_orig_file $fsv_file
_fsv_enable $fsv_file
@@ -179,7 +170,7 @@ for size in 1 $((FSV_BLOCK_SIZE - 1)) $FSV_BLOCK_SIZE; do
rm -f $fsv_file
done
-_fsv_scratch_begin_subtest "verity on 100M file (multiple levels in hash tree)"
+_fsv_scratch_begin_subtest "verity on 100MB file (multiple levels in hash tree)"
head -c 100000000 /dev/urandom > $fsv_orig_file
cp $fsv_orig_file $fsv_file
_fsv_enable $fsv_file
diff --git a/tests/generic/572.out b/tests/generic/572.out
index ad381629..d703835b 100644
--- a/tests/generic/572.out
+++ b/tests/generic/572.out
@@ -39,7 +39,7 @@ bash: SCRATCH_MNT/file.fsv: Operation not permitted
# verity file can be read
# verity file can be measured
-sha256:be54121da3877f8852c65136d731784f134c4dd9d95071502e80d7be9f99b263
+sha256:<digest>
# verity file can be renamed
@@ -58,16 +58,12 @@ sha256:be54121da3877f8852c65136d731784f134c4dd9d95071502e80d7be9f99b263
# Trying to measure non-verity file fails with ENODATA
ERROR: FS_IOC_MEASURE_VERITY failed on 'SCRATCH_MNT/file.fsv': No data available
-# verity on 1-byte file
+# verity on small files
Files matched
-
-# verity on 4095-byte file
Files matched
-
-# verity on 4096-byte file
Files matched
-# verity on 100M file (multiple levels in hash tree)
+# verity on 100MB file (multiple levels in hash tree)
Files matched
# verity on sparse file
--
2.39.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 06/10] generic/573: support non-4K Merkle tree block size
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
` (4 preceding siblings ...)
2022-12-23 1:05 ` [PATCH v2 05/10] generic/572: support non-4K Merkle tree block size Eric Biggers
@ 2022-12-23 1:05 ` Eric Biggers
2022-12-23 1:05 ` [PATCH v2 07/10] generic/577: " Eric Biggers
` (3 subsequent siblings)
9 siblings, 0 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
From: Eric Biggers <ebiggers@google.com>
Update generic/573 to not implicitly assume that the Merkle tree block
size being used is 4096 bytes.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
tests/generic/573 | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/tests/generic/573 b/tests/generic/573
index 63c0aef5..ca0f27f9 100755
--- a/tests/generic/573
+++ b/tests/generic/573
@@ -36,23 +36,23 @@ fsv_file=$SCRATCH_MNT/file.fsv
_fsv_scratch_begin_subtest "FS_IOC_ENABLE_VERITY doesn't require root"
echo foo > $fsv_file
chmod 666 $fsv_file
-_user_do "$FSVERITY_PROG enable $fsv_file"
+_user_do "$FSVERITY_PROG enable --block-size=$FSV_BLOCK_SIZE $fsv_file"
_fsv_scratch_begin_subtest "FS_IOC_ENABLE_VERITY requires write access"
echo foo > $fsv_file >> $seqres.full
chmod 444 $fsv_file
-_user_do "$FSVERITY_PROG enable $fsv_file" |& _filter_scratch
+_user_do "$FSVERITY_PROG enable --block-size=$FSV_BLOCK_SIZE $fsv_file" |& _filter_scratch
_fsv_scratch_begin_subtest "FS_IOC_ENABLE_VERITY requires !append-only"
echo foo > $fsv_file >> $seqres.full
$CHATTR_PROG +a $fsv_file
-$FSVERITY_PROG enable $fsv_file |& _filter_scratch
+_fsv_enable $fsv_file |& _filter_scratch
$CHATTR_PROG -a $fsv_file
_fsv_scratch_begin_subtest "FS_IOC_ENABLE_VERITY requires !immutable"
echo foo > $fsv_file >> $seqres.full
$CHATTR_PROG +i $fsv_file
-$FSVERITY_PROG enable $fsv_file |& _filter_scratch
+_fsv_enable $fsv_file |& _filter_scratch
$CHATTR_PROG -i $fsv_file
_fsv_scratch_begin_subtest "FS_IOC_MEASURE_VERITY doesn't require root"
--
2.39.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 07/10] generic/577: support non-4K Merkle tree block size
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
` (5 preceding siblings ...)
2022-12-23 1:05 ` [PATCH v2 06/10] generic/573: " Eric Biggers
@ 2022-12-23 1:05 ` Eric Biggers
2022-12-23 1:05 ` [PATCH v2 08/10] generic/574: test multiple Merkle tree block sizes Eric Biggers
` (2 subsequent siblings)
9 siblings, 0 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
From: Eric Biggers <ebiggers@google.com>
Update generic/577 to not implicitly assume that the Merkle tree block
size being used is 4096 bytes.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
tests/generic/577 | 22 ++++++++++++----------
tests/generic/577.out | 10 +++++-----
2 files changed, 17 insertions(+), 15 deletions(-)
diff --git a/tests/generic/577 b/tests/generic/577
index 85d680df..bbbfdb0a 100755
--- a/tests/generic/577
+++ b/tests/generic/577
@@ -38,6 +38,11 @@ sigfile=$tmp.sig
otherfile=$SCRATCH_MNT/otherfile
othersigfile=$tmp.othersig
+sign()
+{
+ _fsv_sign "$@" | _filter_scratch | _filter_fsverity_digest
+}
+
# Setup
echo -e "\n# Generating certificates and private keys"
@@ -57,14 +62,13 @@ _enable_fsverity_signatures
echo -e "\n# Generating file and signing it for fs-verity"
head -c 100000 /dev/zero > $fsv_orig_file
for suffix in '' '.2'; do
- _fsv_sign $fsv_orig_file $sigfile$suffix --key=$keyfile$suffix \
- --cert=$certfile$suffix | _filter_scratch
+ sign $fsv_orig_file $sigfile$suffix --key=$keyfile$suffix \
+ --cert=$certfile$suffix
done
echo -e "\n# Signing a different file for fs-verity"
head -c 100000 /dev/zero | tr '\0' 'X' > $otherfile
-_fsv_sign $otherfile $othersigfile --key=$keyfile --cert=$certfile \
- | _filter_scratch
+sign $otherfile $othersigfile --key=$keyfile --cert=$certfile
# Actual tests
@@ -106,16 +110,15 @@ _fsv_enable $fsv_file --signature=$tmp.malformed_sig |& _filter_scratch
echo -e "\n# Testing salt"
reset_fsv_file
-_fsv_sign $fsv_orig_file $sigfile.salted --key=$keyfile --cert=$certfile \
- --salt=abcd | _filter_scratch
+sign $fsv_orig_file $sigfile.salted --key=$keyfile --cert=$certfile --salt=abcd
_fsv_enable $fsv_file --signature=$sigfile.salted --salt=abcd
cmp $fsv_file $fsv_orig_file
echo -e "\n# Testing non-default hash algorithm"
if _fsv_can_enable $fsv_file --hash-alg=sha512; then
reset_fsv_file
- _fsv_sign $fsv_orig_file $sigfile.sha512 --key=$keyfile \
- --cert=$certfile --hash-alg=sha512 > /dev/null
+ sign $fsv_orig_file $sigfile.sha512 --key=$keyfile --cert=$certfile \
+ --hash-alg=sha512 > /dev/null
_fsv_enable $fsv_file --signature=$sigfile.sha512 --hash-alg=sha512
cmp $fsv_file $fsv_orig_file
fi
@@ -123,8 +126,7 @@ fi
echo -e "\n# Testing empty file"
rm -f $fsv_file
echo -n > $fsv_file
-_fsv_sign $fsv_file $sigfile.emptyfile --key=$keyfile --cert=$certfile | \
- _filter_scratch
+sign $fsv_file $sigfile.emptyfile --key=$keyfile --cert=$certfile
_fsv_enable $fsv_file --signature=$sigfile.emptyfile
# success, all done
diff --git a/tests/generic/577.out b/tests/generic/577.out
index 0ca417c4..4f360d57 100644
--- a/tests/generic/577.out
+++ b/tests/generic/577.out
@@ -9,11 +9,11 @@ QA output created by 577
# Enabling fs.verity.require_signatures
# Generating file and signing it for fs-verity
-Signed file 'SCRATCH_MNT/file' (sha256:ecabbfca4efd69a721be824965da10d27900b109549f96687b35a4d91d810dac)
-Signed file 'SCRATCH_MNT/file' (sha256:ecabbfca4efd69a721be824965da10d27900b109549f96687b35a4d91d810dac)
+Signed file 'SCRATCH_MNT/file' (sha256:<digest>)
+Signed file 'SCRATCH_MNT/file' (sha256:<digest>)
# Signing a different file for fs-verity
-Signed file 'SCRATCH_MNT/otherfile' (sha256:b2a419c5a8c767a78c6275d6729794bf51e52ddf8713e31d12a93d61d961f49f)
+Signed file 'SCRATCH_MNT/otherfile' (sha256:<digest>)
# Enabling verity with valid signature (should succeed)
@@ -33,9 +33,9 @@ ERROR: FS_IOC_ENABLE_VERITY failed on 'SCRATCH_MNT/file.fsv': Key was rejected b
ERROR: FS_IOC_ENABLE_VERITY failed on 'SCRATCH_MNT/file.fsv': Bad message
# Testing salt
-Signed file 'SCRATCH_MNT/file' (sha256:1cb173bcd199133eb80e9ea4f0f741001b9e73227aa8812685156f2bc8ff45f5)
+Signed file 'SCRATCH_MNT/file' (sha256:<digest>)
# Testing non-default hash algorithm
# Testing empty file
-Signed file 'SCRATCH_MNT/file.fsv' (sha256:3d248ca542a24fc62d1c43b916eae5016878e2533c88238480b26128a1f1af95)
+Signed file 'SCRATCH_MNT/file.fsv' (sha256:<digest>)
--
2.39.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 08/10] generic/574: test multiple Merkle tree block sizes
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
` (6 preceding siblings ...)
2022-12-23 1:05 ` [PATCH v2 07/10] generic/577: " Eric Biggers
@ 2022-12-23 1:05 ` Eric Biggers
2022-12-25 12:46 ` Zorro Lang
2022-12-23 1:05 ` [PATCH v2 09/10] generic/624: " Eric Biggers
2022-12-23 1:05 ` [PATCH v2 10/10] generic/575: test 1K Merkle tree block size Eric Biggers
9 siblings, 1 reply; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
From: Eric Biggers <ebiggers@google.com>
Instead of only testing 4K Merkle tree blocks, test FSV_BLOCK_SIZE, and
also other sizes if they happen to be supported. This allows this test
to run in cases where it couldn't before and improves test coverage in
cases where it did run before.
Given that the list of Merkle tree block sizes that will actually work
is not fixed, this required reworking the test to not rely on the .out
file so heavily.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
tests/generic/574 | 177 ++++++++++++++++++++++++++----------------
tests/generic/574.out | 83 ++------------------
2 files changed, 114 insertions(+), 146 deletions(-)
diff --git a/tests/generic/574 b/tests/generic/574
index fd4488c9..8f7923ba 100755
--- a/tests/generic/574
+++ b/tests/generic/574
@@ -37,18 +37,19 @@ fsv_file=$SCRATCH_MNT/file.fsv
setup_zeroed_file()
{
- local len=$1
- local sparse=$2
+ local block_size=$1
+ local file_len=$2
+ local sparse=$3
if $sparse; then
- dd if=/dev/zero of=$fsv_orig_file bs=1 count=0 seek=$len \
+ dd if=/dev/zero of=$fsv_orig_file bs=1 count=0 seek=$file_len \
status=none
else
- head -c $len /dev/zero > $fsv_orig_file
+ head -c $file_len /dev/zero > $fsv_orig_file
fi
cp $fsv_orig_file $fsv_file
- _fsv_enable $fsv_file
- md5sum $fsv_file |& _filter_scratch
+ _fsv_enable $fsv_file --block-size=$block_size
+ cmp $fsv_orig_file $fsv_file
}
filter_sigbus()
@@ -66,63 +67,84 @@ round_up_to_page_boundary()
corruption_test()
{
- local file_len=$1
- local zap_offset=$2
- local zap_len=$3
- local is_merkle_tree=${4:-false} # if true, zap tree instead of data
- local use_sparse_file=${5:-false}
+ local block_size=$1
+ local file_len=$2
+ local zap_offset=$3
+ local zap_len=$4
+ local is_merkle_tree=${5:-false} # if true, zap tree instead of data
+ local use_sparse_file=${6:-false}
local page_aligned_eof=$(round_up_to_page_boundary $file_len)
- local measurement
+
+ local paramstr="block_size=$block_size"
+ paramstr+=", file_len=$file_len"
+ paramstr+=", zap_offset=$zap_offset"
+ paramstr+=", zap_len=$zap_len"
+ paramstr+=", is_merkle_tree=$is_merkle_tree"
+ paramstr+=", use_sparse_file=$use_sparse_file"
if $is_merkle_tree; then
local corrupt_func=_fsv_scratch_corrupt_merkle_tree
else
local corrupt_func=_fsv_scratch_corrupt_bytes
fi
+ local fs_block_size=$(_get_block_size $SCRATCH_MNT)
- local msg="Corruption test:"
- msg+=" file_len=$file_len"
- if $use_sparse_file; then
- msg+=" (sparse)"
- fi
- msg+=" zap_offset=$zap_offset"
- if $is_merkle_tree; then
- msg+=" (in Merkle tree)"
- fi
- msg+=" zap_len=$zap_len"
+ rm -rf "${SCRATCH_MNT:?}"/*
+ setup_zeroed_file $block_size $file_len $use_sparse_file
- _fsv_scratch_begin_subtest "$msg"
- setup_zeroed_file $file_len $use_sparse_file
- cmp $fsv_file $fsv_orig_file
- echo "Corrupting bytes..."
+ # Corrupt part of the file (data or Merkle tree).
head -c $zap_len /dev/zero | tr '\0' X \
| $corrupt_func $fsv_file $zap_offset
- echo "Validating corruption (reading full file)..."
+ # Reading the full file with buffered I/O should fail.
_scratch_cycle_mount
- md5sum $fsv_file |& _filter_scratch
+ if cat $fsv_file >/dev/null 2>$tmp.out; then
+ echo "Unexpectedly was able to read full file ($paramstr)"
+ elif ! grep -q 'Input/output error' $tmp.out; then
+ echo "Wrong error reading full file ($paramstr):"
+ cat $tmp.out
+ fi
- echo "Validating corruption (direct I/O)..."
+ # Reading the full file with direct I/O should fail.
_scratch_cycle_mount
- dd if=$fsv_file bs=$FSV_BLOCK_SIZE iflag=direct status=none \
- of=/dev/null |& _filter_scratch
+ if dd if=$fsv_file bs=$fs_block_size iflag=direct status=none \
+ of=/dev/null 2>$tmp.out
+ then
+ echo "Unexpectedly was able to read full file with DIO ($paramstr)"
+ elif ! grep -q 'Input/output error' $tmp.out; then
+ echo "Wrong error reading full file with DIO ($paramstr):"
+ cat $tmp.out
+ fi
- if (( zap_offset < file_len )) && ! $is_merkle_tree; then
- echo "Validating corruption (reading just corrupted part)..."
- dd if=$fsv_file bs=1 skip=$zap_offset count=$zap_len \
- of=/dev/null status=none |& _filter_scratch
+ # Reading just the corrupted part of the file should fail.
+ if ! $is_merkle_tree; then
+ if dd if=$fsv_file bs=1 skip=$zap_offset count=$zap_len \
+ of=/dev/null status=none 2>$tmp.out; then
+ echo "Unexpectedly was able to read corrupted part ($paramstr)"
+ elif ! grep -q 'Input/output error' $tmp.out; then
+ echo "Wrong error reading corrupted part ($paramstr):"
+ cat $tmp.out
+ fi
fi
- echo "Validating corruption (reading full file via mmap)..."
+ # Reading the full file via mmap should fail.
bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file \
-c 'mmap -r 0 $page_aligned_eof' \
- -c 'mread 0 $file_len'" |& filter_sigbus
+ -c 'mread 0 $file_len'" >/dev/null 2>$tmp.out
+ if ! grep -q 'Bus error' $tmp.out; then
+ echo "Didn't see SIGBUS when reading file via mmap"
+ cat $tmp.out
+ fi
+ # Reading just the corrupted part via mmap should fail.
if ! $is_merkle_tree; then
- echo "Validating corruption (reading just corrupted part via mmap)..."
bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file \
-c 'mmap -r 0 $page_aligned_eof' \
- -c 'mread $zap_offset $zap_len'" |& filter_sigbus
+ -c 'mread $zap_offset $zap_len'" >/dev/null 2>$tmp.out
+ if ! grep -q 'Bus error' $tmp.out; then
+ echo "Didn't see SIGBUS when reading corrupted part via mmap"
+ cat $tmp.out
+ fi
fi
}
@@ -131,18 +153,18 @@ corruption_test()
# return zeros in the last block past EOF, regardless of the contents on
# disk. In the former, corruption should be detected and result in SIGBUS,
# while in the latter we would expect zeros past EOF, but no error.
-corrupt_eof_block_test() {
- local file_len=$1
- local zap_len=$2
+corrupt_eof_block_test()
+{
+ local block_size=$1
+ local file_len=$2
+ local zap_len=$3
local page_aligned_eof=$(round_up_to_page_boundary $file_len)
- _fsv_scratch_begin_subtest "Corruption test: EOF block"
- setup_zeroed_file $file_len false
- cmp $fsv_file $fsv_orig_file
- echo "Corrupting bytes..."
+
+ rm -rf "${SCRATCH_MNT:?}"/*
+ setup_zeroed_file $block_size $file_len false
head -c $zap_len /dev/zero | tr '\0' X \
| _fsv_scratch_corrupt_bytes $fsv_file $file_len
- echo "Reading eof block via mmap into a temporary file..."
bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file \
-c 'mmap -r 0 $page_aligned_eof' \
-c 'mread -v $file_len $zap_len'" \
@@ -153,30 +175,49 @@ corrupt_eof_block_test() {
-c "mmap -r 0 $page_aligned_eof" \
-c "mread -v $file_len $zap_len" >$tmp.eof_zero_read
- echo "Checking for SIGBUS or zeros..."
- grep -q -e '^Bus error$' $tmp.eof_block_read \
- || diff $tmp.eof_block_read $tmp.eof_zero_read \
- && echo "OK"
+ grep -q -e '^Bus error$' $tmp.eof_block_read || \
+ diff $tmp.eof_block_read $tmp.eof_zero_read
}
-# Note: these tests just overwrite some bytes without checking their original
-# values. Therefore, make sure to overwrite at least 5 or so bytes, to make it
-# nearly guaranteed that there will be a change -- even when the test file is
-# encrypted due to the test_dummy_encryption mount option being specified.
-
-corruption_test 131072 0 5
-corruption_test 131072 4091 5
-corruption_test 131072 65536 65536
-corruption_test 131072 131067 5
-
-corrupt_eof_block_test 130999 72
-
-# Merkle tree corruption.
-corruption_test 200000 100 10 true
+test_block_size()
+{
+ local block_size=$1
+
+ # Note: these tests just overwrite some bytes without checking their
+ # original values. Therefore, make sure to overwrite at least 5 or so
+ # bytes, to make it nearly guaranteed that there will be a change --
+ # even when the test file is encrypted due to the test_dummy_encryption
+ # mount option being specified.
+ corruption_test $block_size 131072 0 5
+ corruption_test $block_size 131072 4091 5
+ corruption_test $block_size 131072 65536 65536
+ corruption_test $block_size 131072 131067 5
+
+ corrupt_eof_block_test $block_size 130999 72
+
+ # Merkle tree corruption.
+ corruption_test $block_size 200000 100 10 true
+
+ # Sparse file. Corrupting the Merkle tree should still cause reads to
+ # fail, i.e. the filesystem must verify holes.
+ corruption_test $block_size 200000 100 10 true true
+}
-# Sparse file. Corrupting the Merkle tree should still cause reads to fail,
-# i.e. the filesystem must verify holes.
-corruption_test 200000 100 10 true true
+# Always test FSV_BLOCK_SIZE. Also test some other block sizes if they happen
+# to be supported.
+_fsv_scratch_begin_subtest "Testing block_size=FSV_BLOCK_SIZE"
+test_block_size $FSV_BLOCK_SIZE
+for block_size in 1024 4096 16384 65536; do
+ _fsv_scratch_begin_subtest "Testing block_size=$block_size if supported"
+ if (( block_size == FSV_BLOCK_SIZE )); then
+ continue # Skip redundant test case.
+ fi
+ if ! _fsv_can_enable $fsv_file --block-size=$block_size; then
+ echo "block_size=$block_size is unsupported" >> $seqres.full
+ continue
+ fi
+ test_block_size $block_size
+done
# success, all done
status=0
diff --git a/tests/generic/574.out b/tests/generic/574.out
index d40d1263..3ed57f1c 100644
--- a/tests/generic/574.out
+++ b/tests/generic/574.out
@@ -1,84 +1,11 @@
QA output created by 574
-# Corruption test: file_len=131072 zap_offset=0 zap_len=5
-0dfbe8aa4c20b52e1b8bf3cb6cbdf193 SCRATCH_MNT/file.fsv
-Corrupting bytes...
-Validating corruption (reading full file)...
-md5sum: SCRATCH_MNT/file.fsv: Input/output error
-Validating corruption (direct I/O)...
-dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
-Validating corruption (reading just corrupted part)...
-dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
-Validating corruption (reading full file via mmap)...
-Bus error
-Validating corruption (reading just corrupted part via mmap)...
-Bus error
+# Testing block_size=FSV_BLOCK_SIZE
-# Corruption test: file_len=131072 zap_offset=4091 zap_len=5
-0dfbe8aa4c20b52e1b8bf3cb6cbdf193 SCRATCH_MNT/file.fsv
-Corrupting bytes...
-Validating corruption (reading full file)...
-md5sum: SCRATCH_MNT/file.fsv: Input/output error
-Validating corruption (direct I/O)...
-dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
-Validating corruption (reading just corrupted part)...
-dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
-Validating corruption (reading full file via mmap)...
-Bus error
-Validating corruption (reading just corrupted part via mmap)...
-Bus error
+# Testing block_size=1024 if supported
-# Corruption test: file_len=131072 zap_offset=65536 zap_len=65536
-0dfbe8aa4c20b52e1b8bf3cb6cbdf193 SCRATCH_MNT/file.fsv
-Corrupting bytes...
-Validating corruption (reading full file)...
-md5sum: SCRATCH_MNT/file.fsv: Input/output error
-Validating corruption (direct I/O)...
-dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
-Validating corruption (reading just corrupted part)...
-dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
-Validating corruption (reading full file via mmap)...
-Bus error
-Validating corruption (reading just corrupted part via mmap)...
-Bus error
+# Testing block_size=4096 if supported
-# Corruption test: file_len=131072 zap_offset=131067 zap_len=5
-0dfbe8aa4c20b52e1b8bf3cb6cbdf193 SCRATCH_MNT/file.fsv
-Corrupting bytes...
-Validating corruption (reading full file)...
-md5sum: SCRATCH_MNT/file.fsv: Input/output error
-Validating corruption (direct I/O)...
-dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
-Validating corruption (reading just corrupted part)...
-dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
-Validating corruption (reading full file via mmap)...
-Bus error
-Validating corruption (reading just corrupted part via mmap)...
-Bus error
+# Testing block_size=16384 if supported
-# Corruption test: EOF block
-f5cca0d7fbb8b02bc6118a9954d5d306 SCRATCH_MNT/file.fsv
-Corrupting bytes...
-Reading eof block via mmap into a temporary file...
-Checking for SIGBUS or zeros...
-OK
-
-# Corruption test: file_len=200000 zap_offset=100 (in Merkle tree) zap_len=10
-4a1e4325031b13f933ac4f1db9ecb63f SCRATCH_MNT/file.fsv
-Corrupting bytes...
-Validating corruption (reading full file)...
-md5sum: SCRATCH_MNT/file.fsv: Input/output error
-Validating corruption (direct I/O)...
-dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
-Validating corruption (reading full file via mmap)...
-Bus error
-
-# Corruption test: file_len=200000 (sparse) zap_offset=100 (in Merkle tree) zap_len=10
-4a1e4325031b13f933ac4f1db9ecb63f SCRATCH_MNT/file.fsv
-Corrupting bytes...
-Validating corruption (reading full file)...
-md5sum: SCRATCH_MNT/file.fsv: Input/output error
-Validating corruption (direct I/O)...
-dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
-Validating corruption (reading full file via mmap)...
-Bus error
+# Testing block_size=65536 if supported
--
2.39.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 09/10] generic/624: test multiple Merkle tree block sizes
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
` (7 preceding siblings ...)
2022-12-23 1:05 ` [PATCH v2 08/10] generic/574: test multiple Merkle tree block sizes Eric Biggers
@ 2022-12-23 1:05 ` Eric Biggers
2022-12-23 1:05 ` [PATCH v2 10/10] generic/575: test 1K Merkle tree block size Eric Biggers
9 siblings, 0 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
From: Eric Biggers <ebiggers@google.com>
Instead of only testing 4K Merkle tree blocks, test FSV_BLOCK_SIZE, and
also other sizes if they happen to be supported. This allows this test
to run in cases where it couldn't before and improves test coverage in
cases where it did run before.
This required reworking the test to compute the expected digests using
the 'fsverity digest' command, instead of relying on .out file
comparisons. (There isn't much downside to this, since comparison to
known-good file digests already happens in generic/575. So if both the
kernel and fsverity-utils were to be broken in the same way, generic/575
would detect it. generic/624 serves a different purpose.)
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
common/verity | 11 ++++
tests/generic/624 | 119 ++++++++++++++++++++++++++++++------------
tests/generic/624.out | 15 ++----
3 files changed, 101 insertions(+), 44 deletions(-)
diff --git a/common/verity b/common/verity
index b88e839b..77c257d3 100644
--- a/common/verity
+++ b/common/verity
@@ -263,6 +263,17 @@ _fsv_measure()
$FSVERITY_PROG measure "$@" | awk '{print $1}'
}
+_fsv_digest()
+{
+ local args=("$@")
+ # If the caller didn't explicitly specify a Merkle tree block size, then
+ # use FSV_BLOCK_SIZE.
+ if ! [[ " $*" =~ " --block-size" ]]; then
+ args+=("--block-size=$FSV_BLOCK_SIZE")
+ fi
+ $FSVERITY_PROG digest "${args[@]}" | awk '{print $1}'
+}
+
_fsv_sign()
{
local args=("$@")
diff --git a/tests/generic/624 b/tests/generic/624
index 7c447289..db4b6731 100755
--- a/tests/generic/624
+++ b/tests/generic/624
@@ -24,48 +24,99 @@ _cleanup()
_supported_fs generic
_require_scratch_verity
_disable_fsverity_signatures
-# For the output of this test to always be the same, it has to use a specific
-# Merkle tree block size.
-if [ $FSV_BLOCK_SIZE != 4096 ]; then
- _notrun "4096-byte verity block size not supported on this platform"
-fi
+fsv_orig_file=$SCRATCH_MNT/file
+fsv_file=$SCRATCH_MNT/file.fsv
_scratch_mkfs_verity &>> $seqres.full
_scratch_mount
-
-echo -e "\n# Creating a verity file"
-fsv_file=$SCRATCH_MNT/file
-# Always use the same file contents, so that the output of the test is always
-# the same. Also use a file that is large enough to have multiple Merkle tree
-# levels, so that the test verifies that the blocks are returned in the expected
-# order. A 1 MB file with SHA-256 and a Merkle tree block size of 4096 will
-# have 3 Merkle tree blocks (3*4096 bytes): two at level 0 and one at level 1.
-head -c 1000000 /dev/zero > $fsv_file
-merkle_tree_size=$((3 * FSV_BLOCK_SIZE))
-fsverity_descriptor_size=256
-_fsv_enable $fsv_file --salt=abcd
+_fsv_create_enable_file $fsv_file
_require_fsverity_dump_metadata $fsv_file
-_fsv_measure $fsv_file
-echo -e "\n# Dumping Merkle tree"
-_fsv_dump_merkle_tree $fsv_file | sha256sum
+# Test FS_IOC_READ_VERITY_METADATA on a file that uses the given Merkle tree
+# block size.
+test_block_size()
+{
+ local block_size=$1
+ local digest_size=32 # assuming SHA-256
+ local i
+
+ # Create the file. Make the file size big enough to result in multiple
+ # Merkle tree levels being needed. The following expression computes a
+ # file size that needs 2 blocks at level 0, and thus 1 block at level 1.
+ local file_size=$((block_size * (2 * (block_size / digest_size))))
+ head -c $file_size /dev/zero > $fsv_orig_file
+ local tree_params=("--salt=abcd" "--block-size=$block_size")
+ cp $fsv_orig_file $fsv_file
+ _fsv_enable $fsv_file "${tree_params[@]}"
+
+ # Use the 'fsverity digest' command to compute the expected Merkle tree,
+ # descriptor, and file digest.
+ #
+ # Ideally we'd just hard-code expected values into the .out file and
+ # echo the actual values. That doesn't quite work here, since all these
+ # values depend on the Merkle tree block size, and the Merkle tree block
+ # sizes that are supported (and thus get tested here) vary. Therefore,
+ # we calculate the expected values in userspace with the help of
+ # 'fsverity digest', then do explicit comparisons with them. This works
+ # fine as long as fsverity-utils and the kernel don't get broken in the
+ # same way, in which case generic/575 should detect the problem anyway.
+ local expected_file_digest=$(_fsv_digest $fsv_orig_file \
+ "${tree_params[@]}" \
+ --out-merkle-tree=$tmp.merkle_tree.expected \
+ --out-descriptor=$tmp.descriptor.expected)
+ local merkle_tree_size=$(_get_filesize $tmp.merkle_tree.expected)
+ local descriptor_size=$(_get_filesize $tmp.descriptor.expected)
-echo -e "\n# Dumping Merkle tree (in chunks)"
-# The above test may get the whole tree in one read, so also try reading it in
-# chunks.
-for (( i = 0; i < merkle_tree_size; i += 997 )); do
- _fsv_dump_merkle_tree $fsv_file --offset=$i --length=997
-done | sha256sum
+ # 'fsverity measure' should return the expected file digest.
+ local actual_file_digest=$(_fsv_measure $fsv_file)
+ if [ "$actual_file_digest" != "$expected_file_digest" ]; then
+ echo "Measure returned $actual_file_digest but expected $expected_file_digest"
+ fi
-echo -e "\n# Dumping descriptor"
-# Note that the hash that is printed here should be the same hash that was
-# printed by _fsv_measure above.
-_fsv_dump_descriptor $fsv_file | sha256sum
+ # Test dumping the Merkle tree.
+ _fsv_dump_merkle_tree $fsv_file > $tmp.merkle_tree.actual
+ if ! cmp $tmp.merkle_tree.expected $tmp.merkle_tree.actual; then
+ echo "Dumped Merkle tree didn't match"
+ fi
+
+ # Test dumping the Merkle tree in chunks.
+ for (( i = 0; i < merkle_tree_size; i += 997 )); do
+ _fsv_dump_merkle_tree $fsv_file --offset=$i --length=997
+ done > $tmp.merkle_tree.actual
+ if ! cmp $tmp.merkle_tree.expected $tmp.merkle_tree.actual; then
+ echo "Dumped Merkle tree (in chunks) didn't match"
+ fi
+
+ # Test dumping the descriptor.
+ _fsv_dump_descriptor $fsv_file > $tmp.descriptor.actual
+ if ! cmp $tmp.descriptor.expected $tmp.descriptor.actual; then
+ echo "Dumped descriptor didn't match"
+ fi
+
+ # Test dumping the descriptor in chunks.
+ for (( i = 0; i < descriptor_size; i += 13 )); do
+ _fsv_dump_descriptor $fsv_file --offset=$i --length=13
+ done > $tmp.descriptor.actual
+ if ! cmp $tmp.descriptor.expected $tmp.descriptor.actual; then
+ echo "Dumped descriptor (in chunks) didn't match"
+ fi
+}
-echo -e "\n# Dumping descriptor (in chunks)"
-for (( i = 0; i < fsverity_descriptor_size; i += 13 )); do
- _fsv_dump_descriptor $fsv_file --offset=$i --length=13
-done | sha256sum
+# Always test FSV_BLOCK_SIZE. Also test some other block sizes if they happen
+# to be supported.
+_fsv_scratch_begin_subtest "Testing block_size=FSV_BLOCK_SIZE"
+test_block_size $FSV_BLOCK_SIZE
+for block_size in 1024 4096 16384 65536; do
+ _fsv_scratch_begin_subtest "Testing block_size=$block_size if supported"
+ if (( block_size == FSV_BLOCK_SIZE )); then
+ continue # Skip redundant test case.
+ fi
+ if ! _fsv_can_enable $fsv_file --block-size=$block_size; then
+ echo "block_size=$block_size is unsupported" >> $seqres.full
+ continue
+ fi
+ test_block_size $block_size
+done
# success, all done
status=0
diff --git a/tests/generic/624.out b/tests/generic/624.out
index 912826d3..0ea19ee5 100644
--- a/tests/generic/624.out
+++ b/tests/generic/624.out
@@ -1,16 +1,11 @@
QA output created by 624
-# Creating a verity file
-sha256:11e4f886bf2d70a6ef3a8b6ce8e8c62c9e5d3263208b9f120ae46791f124be73
+# Testing block_size=FSV_BLOCK_SIZE
-# Dumping Merkle tree
-db88cdad554734cd648a1bfbb5be7f86646c54397847aab0b3f42a28829fed17 -
+# Testing block_size=1024 if supported
-# Dumping Merkle tree (in chunks)
-db88cdad554734cd648a1bfbb5be7f86646c54397847aab0b3f42a28829fed17 -
+# Testing block_size=4096 if supported
-# Dumping descriptor
-11e4f886bf2d70a6ef3a8b6ce8e8c62c9e5d3263208b9f120ae46791f124be73 -
+# Testing block_size=16384 if supported
-# Dumping descriptor (in chunks)
-11e4f886bf2d70a6ef3a8b6ce8e8c62c9e5d3263208b9f120ae46791f124be73 -
+# Testing block_size=65536 if supported
--
2.39.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 10/10] generic/575: test 1K Merkle tree block size
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
` (8 preceding siblings ...)
2022-12-23 1:05 ` [PATCH v2 09/10] generic/624: " Eric Biggers
@ 2022-12-23 1:05 ` Eric Biggers
9 siblings, 0 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-23 1:05 UTC (permalink / raw)
To: fstests; +Cc: linux-fscrypt
From: Eric Biggers <ebiggers@google.com>
In addition to 4K, test 1K Merkle tree blocks. Also always run this
test, regardless of FSV_BLOCK_SIZE, but allow skipping tests of
parameters that are unsupported, unless they are the default.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
tests/generic/575 | 57 +++++++++++++++++++++++++++++++------------
tests/generic/575.out | 8 ++++--
2 files changed, 47 insertions(+), 18 deletions(-)
diff --git a/tests/generic/575 b/tests/generic/575
index 0ece8826..344fd2b9 100755
--- a/tests/generic/575
+++ b/tests/generic/575
@@ -4,7 +4,7 @@
#
# FS QA Test generic/575
#
-# Test that fs-verity is using the correct measurement values. This test
+# Test that fs-verity is using the correct file digest values. This test
# verifies that fs-verity is doing its Merkle tree-based hashing correctly,
# i.e. that it hasn't been broken by a change.
#
@@ -26,9 +26,6 @@ _cleanup()
# real QA test starts here
_supported_fs generic
_require_scratch_verity
-if [ $FSV_BLOCK_SIZE != 4096 ]; then
- _notrun "4096-byte verity block size not supported on this platform"
-fi
_disable_fsverity_signatures
_scratch_mkfs_verity &>> $seqres.full
@@ -36,24 +33,42 @@ _scratch_mount
fsv_orig_file=$SCRATCH_MNT/file
fsv_file=$SCRATCH_MNT/file.fsv
+# Try multiple hash algorithms.
algs=(sha256 sha512)
+# Try multiple Merkle tree block sizes.
+block_sizes=(1024 4096)
+
# Try files with 0, 1, and multiple Merkle tree levels.
file_sizes=(0 4096 65536 65536 100000000)
# Try both unsalted and salted, and check that empty salt is the same as no salt
salts=('' '' '' '--salt=' '--salt=f3c93fa6fb828c0e1587e5714ecf6f56')
-# The expected file measurements are here rather than in the expected output
-# file because not all hash algorithms may be available.
-sha256_vals=(
+# The expected file digests are here rather than in the expected output file
+# because the kernel might not support all hash algorithms and block sizes.
+sha256_vals_bsize1024=(
+sha256:f2cca36b9b1b7f07814e4284b10121809133e7cb9c4528c8f6846e85fc624ffa
+sha256:ea08590a4fe9c3d6c9dafe0eedacd9dffff8f24e24f1865ee3af132a495ab087
+sha256:527496288d703686e31092f5cca7e1306b2467f00b247ad01056ee5ec35a4bb9
+sha256:527496288d703686e31092f5cca7e1306b2467f00b247ad01056ee5ec35a4bb9
+sha256:087818b23312acb15dff9ff6e2b4f601406d08bb36013542444cc15248f47016
+)
+sha256_vals_bsize4096=(
sha256:3d248ca542a24fc62d1c43b916eae5016878e2533c88238480b26128a1f1af95
sha256:babc284ee4ffe7f449377fbf6692715b43aec7bc39c094a95878904d34bac97e
sha256:011e3f2b1dc89b75d78cddcc2a1b85cd8a64b2883e5f20f277ae4c0617e0404f
sha256:011e3f2b1dc89b75d78cddcc2a1b85cd8a64b2883e5f20f277ae4c0617e0404f
sha256:9d33cab743468fcbe4edab91a275b30dd543c12dd5e6ce6f2f737f66a1558f06
)
-sha512_vals=(
+sha512_vals_bsize1024=(
+sha512:8451664f25b2ad3f24391280e0c5681cb843389c180baa719f8fdfb063f5ddfa2d1c4433e55e2b6fbb3ba6aa2df8a4f41bf56cb7e0a3b617b6919a42c80f034c
+sha512:ab3c6444ab377bbe54c604c26cad241b146d85dc29727703a0d8134f70a8172fb3fa67d171355106b69cc0a9e7e9debb335f9461b3aba44093914867f7c73233
+sha512:e6a11353c24dd7b4603cb8ffa50a7041dbea7382d4698474ccbc2d8b34f3a83d8bf16df25c64ed31ee27213a8a3cbd001fb1ccde46384c23b81305c2393c1046
+sha512:e6a11353c24dd7b4603cb8ffa50a7041dbea7382d4698474ccbc2d8b34f3a83d8bf16df25c64ed31ee27213a8a3cbd001fb1ccde46384c23b81305c2393c1046
+sha512:517d573bd50d5f3787f5766c2ac60c7af854b0901b69757b4ef8dd70aa6b30fcc10d81629ce923bdd062a01c20fad0f063a081a2f3b0814ac06547b26bedc0d9
+)
+sha512_vals_bsize4096=(
sha512:ccf9e5aea1c2a64efa2f2354a6024b90dffde6bbc017825045dce374474e13d10adb9dadcc6ca8e17a3c075fbd31336e8f266ae6fa93a6c3bed66f9e784e5abf
sha512:928922686c4caf32175f5236a7f964e9925d10a74dc6d8344a8bd08b23c228ff5792573987d7895f628f39c4f4ebe39a7367d7aeb16aaa0cd324ac1d53664e61
sha512:eab7224ce374a0a4babcb2db25e24836247f38b87806ad9be9e5ba4daac2f5b814fc0cbdfd9f1f8499b3c9a6c1b38fe08974cce49883ab4ccd04462fd2f9507f
@@ -61,20 +76,27 @@ sha512:eab7224ce374a0a4babcb2db25e24836247f38b87806ad9be9e5ba4daac2f5b814fc0cbdf
sha512:f7083a38644880d25539488313e9e5b41a4d431a0e383945129ad2c36e3c1d0f28928a424641bb1363c12b6e770578102566acea73baf1ce8ee15336f5ba2446
)
-test_alg()
+test_alg_with_block_size()
{
local alg=$1
- local -n vals=${alg}_vals
+ local block_size=$2
+ local -n vals=${alg}_vals_bsize${block_size}
local i
local file_size
local expected actual salt_arg
- _fsv_scratch_begin_subtest "Check for expected measurement values ($alg)"
+ _fsv_scratch_begin_subtest "Testing alg=$alg, block_size=$block_size if supported"
- if ! _fsv_can_enable $fsv_file --hash-alg=$alg; then
- if [ "$alg" = sha256 ]; then
- _fail "Something is wrong - sha256 hash should always be available"
+ if ! _fsv_can_enable $fsv_file --hash-alg=$alg --block-size=$block_size
+ then
+ # Since this is after _require_scratch_verity, sha256 with
+ # FSV_BLOCK_SIZE must be supported.
+ if [ "$alg" = "sha256" -a "$block_size" = "$FSV_BLOCK_SIZE" ]
+ then
+ _fail "Failed to enable verity with default params"
fi
+ # This combination of parameters is unsupported; skip it.
+ echo "alg=$alg, block_size=$block_size is unsupported" >> $seqres.full
return 0
fi
@@ -85,7 +107,8 @@ test_alg()
head -c $file_size /dev/zero > $fsv_orig_file
cp $fsv_orig_file $fsv_file
- _fsv_enable --hash-alg=$alg $salt_arg $fsv_file
+ _fsv_enable $fsv_file --hash-alg=$alg --block-size=$block_size \
+ $salt_arg
actual=$(_fsv_measure $fsv_file)
if [ "$actual" != "$expected" ]; then
echo "Mismatch: expected $expected, kernel calculated $actual (file_size=$file_size)"
@@ -96,7 +119,9 @@ test_alg()
}
for alg in ${algs[@]}; do
- test_alg $alg
+ for block_size in ${block_sizes[@]}; do
+ test_alg_with_block_size $alg $block_size
+ done
done
# success, all done
diff --git a/tests/generic/575.out b/tests/generic/575.out
index 77bec43e..5ad70773 100644
--- a/tests/generic/575.out
+++ b/tests/generic/575.out
@@ -1,5 +1,9 @@
QA output created by 575
-# Check for expected measurement values (sha256)
+# Testing alg=sha256, block_size=1024 if supported
-# Check for expected measurement values (sha512)
+# Testing alg=sha256, block_size=4096 if supported
+
+# Testing alg=sha512, block_size=1024 if supported
+
+# Testing alg=sha512, block_size=4096 if supported
--
2.39.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v2 08/10] generic/574: test multiple Merkle tree block sizes
2022-12-23 1:05 ` [PATCH v2 08/10] generic/574: test multiple Merkle tree block sizes Eric Biggers
@ 2022-12-25 12:46 ` Zorro Lang
2022-12-26 5:21 ` Eric Biggers
0 siblings, 1 reply; 16+ messages in thread
From: Zorro Lang @ 2022-12-25 12:46 UTC (permalink / raw)
To: Eric Biggers; +Cc: fstests, linux-fscrypt
On Thu, Dec 22, 2022 at 05:05:52PM -0800, Eric Biggers wrote:
> From: Eric Biggers <ebiggers@google.com>
>
> Instead of only testing 4K Merkle tree blocks, test FSV_BLOCK_SIZE, and
> also other sizes if they happen to be supported. This allows this test
> to run in cases where it couldn't before and improves test coverage in
> cases where it did run before.
>
> Given that the list of Merkle tree block sizes that will actually work
> is not fixed, this required reworking the test to not rely on the .out
> file so heavily.
>
> Signed-off-by: Eric Biggers <ebiggers@google.com>
> ---
> tests/generic/574 | 177 ++++++++++++++++++++++++++----------------
> tests/generic/574.out | 83 ++------------------
> 2 files changed, 114 insertions(+), 146 deletions(-)
>
> diff --git a/tests/generic/574 b/tests/generic/574
> index fd4488c9..8f7923ba 100755
> --- a/tests/generic/574
> +++ b/tests/generic/574
> @@ -37,18 +37,19 @@ fsv_file=$SCRATCH_MNT/file.fsv
>
> setup_zeroed_file()
> {
> - local len=$1
> - local sparse=$2
> + local block_size=$1
> + local file_len=$2
> + local sparse=$3
>
> if $sparse; then
> - dd if=/dev/zero of=$fsv_orig_file bs=1 count=0 seek=$len \
> + dd if=/dev/zero of=$fsv_orig_file bs=1 count=0 seek=$file_len \
> status=none
> else
> - head -c $len /dev/zero > $fsv_orig_file
> + head -c $file_len /dev/zero > $fsv_orig_file
> fi
> cp $fsv_orig_file $fsv_file
> - _fsv_enable $fsv_file
> - md5sum $fsv_file |& _filter_scratch
> + _fsv_enable $fsv_file --block-size=$block_size
> + cmp $fsv_orig_file $fsv_file
> }
>
> filter_sigbus()
> @@ -66,63 +67,84 @@ round_up_to_page_boundary()
>
> corruption_test()
> {
> - local file_len=$1
> - local zap_offset=$2
> - local zap_len=$3
> - local is_merkle_tree=${4:-false} # if true, zap tree instead of data
> - local use_sparse_file=${5:-false}
> + local block_size=$1
> + local file_len=$2
> + local zap_offset=$3
> + local zap_len=$4
> + local is_merkle_tree=${5:-false} # if true, zap tree instead of data
> + local use_sparse_file=${6:-false}
> local page_aligned_eof=$(round_up_to_page_boundary $file_len)
> - local measurement
> +
> + local paramstr="block_size=$block_size"
> + paramstr+=", file_len=$file_len"
> + paramstr+=", zap_offset=$zap_offset"
> + paramstr+=", zap_len=$zap_len"
> + paramstr+=", is_merkle_tree=$is_merkle_tree"
> + paramstr+=", use_sparse_file=$use_sparse_file"
>
> if $is_merkle_tree; then
> local corrupt_func=_fsv_scratch_corrupt_merkle_tree
> else
> local corrupt_func=_fsv_scratch_corrupt_bytes
> fi
> + local fs_block_size=$(_get_block_size $SCRATCH_MNT)
>
> - local msg="Corruption test:"
> - msg+=" file_len=$file_len"
> - if $use_sparse_file; then
> - msg+=" (sparse)"
> - fi
> - msg+=" zap_offset=$zap_offset"
> - if $is_merkle_tree; then
> - msg+=" (in Merkle tree)"
> - fi
> - msg+=" zap_len=$zap_len"
> + rm -rf "${SCRATCH_MNT:?}"/*
> + setup_zeroed_file $block_size $file_len $use_sparse_file
>
> - _fsv_scratch_begin_subtest "$msg"
> - setup_zeroed_file $file_len $use_sparse_file
> - cmp $fsv_file $fsv_orig_file
> - echo "Corrupting bytes..."
> + # Corrupt part of the file (data or Merkle tree).
> head -c $zap_len /dev/zero | tr '\0' X \
> | $corrupt_func $fsv_file $zap_offset
>
> - echo "Validating corruption (reading full file)..."
> + # Reading the full file with buffered I/O should fail.
> _scratch_cycle_mount
> - md5sum $fsv_file |& _filter_scratch
> + if cat $fsv_file >/dev/null 2>$tmp.out; then
> + echo "Unexpectedly was able to read full file ($paramstr)"
> + elif ! grep -q 'Input/output error' $tmp.out; then
> + echo "Wrong error reading full file ($paramstr):"
> + cat $tmp.out
> + fi
>
> - echo "Validating corruption (direct I/O)..."
> + # Reading the full file with direct I/O should fail.
> _scratch_cycle_mount
> - dd if=$fsv_file bs=$FSV_BLOCK_SIZE iflag=direct status=none \
> - of=/dev/null |& _filter_scratch
> + if dd if=$fsv_file bs=$fs_block_size iflag=direct status=none \
> + of=/dev/null 2>$tmp.out
> + then
> + echo "Unexpectedly was able to read full file with DIO ($paramstr)"
> + elif ! grep -q 'Input/output error' $tmp.out; then
> + echo "Wrong error reading full file with DIO ($paramstr):"
> + cat $tmp.out
> + fi
>
> - if (( zap_offset < file_len )) && ! $is_merkle_tree; then
> - echo "Validating corruption (reading just corrupted part)..."
> - dd if=$fsv_file bs=1 skip=$zap_offset count=$zap_len \
> - of=/dev/null status=none |& _filter_scratch
> + # Reading just the corrupted part of the file should fail.
> + if ! $is_merkle_tree; then
> + if dd if=$fsv_file bs=1 skip=$zap_offset count=$zap_len \
> + of=/dev/null status=none 2>$tmp.out; then
> + echo "Unexpectedly was able to read corrupted part ($paramstr)"
> + elif ! grep -q 'Input/output error' $tmp.out; then
> + echo "Wrong error reading corrupted part ($paramstr):"
> + cat $tmp.out
> + fi
> fi
>
> - echo "Validating corruption (reading full file via mmap)..."
> + # Reading the full file via mmap should fail.
> bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file \
> -c 'mmap -r 0 $page_aligned_eof' \
> - -c 'mread 0 $file_len'" |& filter_sigbus
> + -c 'mread 0 $file_len'" >/dev/null 2>$tmp.out
> + if ! grep -q 'Bus error' $tmp.out; then
> + echo "Didn't see SIGBUS when reading file via mmap"
Hmm... will sigbus error really be output to stderr? From a testing, the
generic/574 fails as:
# ./check -s simpledev generic/574
SECTION -- simpledev
FSTYP -- ext4
PLATFORM -- Linux/x86_64 xx-xxxxxx-xxx 6.1.0-rc3 #5 SMP PREEMPT_DYNAMIC Tue Nov 1 01:08:52 CST 2022
MKFS_OPTIONS -- -F /dev/sdb
MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 /dev/sdb /mnt/scratch
generic/574 - output mismatch (see /root/git/xfstests/results//simpledev/generic/574.out.bad)
--- tests/generic/574.out 2022-12-25 20:02:41.609104749 +0800
+++ /root/git/xfstests/results//simpledev/generic/574.out.bad 2022-12-25 20:21:57.430719504 +0800
@@ -1,6 +1,32 @@
QA output created by 574
# Testing block_size=FSV_BLOCK_SIZE
+/root/git/xfstests/tests/generic/574: line 69: 1949533 Bus error (core dumped) bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file -c 'mmap -r 0 $page_aligned_eof' -c 'mread 0 $file_len'" > /dev/null 2> $tmp.out
+Didn't see SIGBUS when reading file via mmap
+/root/git/xfstests/tests/generic/574: line 69: 1949544 Bus error (core dumped) bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file -c 'mmap -r 0 $page_aligned_eof' -c 'mread $zap_offset $zap_len'" > /dev/null 2> $tmp.out
+Didn't see SIGBUS when reading corrupted part via mmap
...
(Run 'diff -u /root/git/xfstests/tests/generic/574.out /root/git/xfstests/results//simpledev/generic/574.out.bad' to see the entire diff)
Ran: generic/574
Failures: generic/574
Failed 1 of 1 tests
Thanks,
Zorro
> + cat $tmp.out
> + fi
>
> + # Reading just the corrupted part via mmap should fail.
> if ! $is_merkle_tree; then
> - echo "Validating corruption (reading just corrupted part via mmap)..."
> bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file \
> -c 'mmap -r 0 $page_aligned_eof' \
> - -c 'mread $zap_offset $zap_len'" |& filter_sigbus
> + -c 'mread $zap_offset $zap_len'" >/dev/null 2>$tmp.out
> + if ! grep -q 'Bus error' $tmp.out; then
> + echo "Didn't see SIGBUS when reading corrupted part via mmap"
> + cat $tmp.out
> + fi
> fi
> }
>
> @@ -131,18 +153,18 @@ corruption_test()
> # return zeros in the last block past EOF, regardless of the contents on
> # disk. In the former, corruption should be detected and result in SIGBUS,
> # while in the latter we would expect zeros past EOF, but no error.
> -corrupt_eof_block_test() {
> - local file_len=$1
> - local zap_len=$2
> +corrupt_eof_block_test()
> +{
> + local block_size=$1
> + local file_len=$2
> + local zap_len=$3
> local page_aligned_eof=$(round_up_to_page_boundary $file_len)
> - _fsv_scratch_begin_subtest "Corruption test: EOF block"
> - setup_zeroed_file $file_len false
> - cmp $fsv_file $fsv_orig_file
> - echo "Corrupting bytes..."
> +
> + rm -rf "${SCRATCH_MNT:?}"/*
> + setup_zeroed_file $block_size $file_len false
> head -c $zap_len /dev/zero | tr '\0' X \
> | _fsv_scratch_corrupt_bytes $fsv_file $file_len
>
> - echo "Reading eof block via mmap into a temporary file..."
> bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file \
> -c 'mmap -r 0 $page_aligned_eof' \
> -c 'mread -v $file_len $zap_len'" \
> @@ -153,30 +175,49 @@ corrupt_eof_block_test() {
> -c "mmap -r 0 $page_aligned_eof" \
> -c "mread -v $file_len $zap_len" >$tmp.eof_zero_read
>
> - echo "Checking for SIGBUS or zeros..."
> - grep -q -e '^Bus error$' $tmp.eof_block_read \
> - || diff $tmp.eof_block_read $tmp.eof_zero_read \
> - && echo "OK"
> + grep -q -e '^Bus error$' $tmp.eof_block_read || \
> + diff $tmp.eof_block_read $tmp.eof_zero_read
> }
>
> -# Note: these tests just overwrite some bytes without checking their original
> -# values. Therefore, make sure to overwrite at least 5 or so bytes, to make it
> -# nearly guaranteed that there will be a change -- even when the test file is
> -# encrypted due to the test_dummy_encryption mount option being specified.
> -
> -corruption_test 131072 0 5
> -corruption_test 131072 4091 5
> -corruption_test 131072 65536 65536
> -corruption_test 131072 131067 5
> -
> -corrupt_eof_block_test 130999 72
> -
> -# Merkle tree corruption.
> -corruption_test 200000 100 10 true
> +test_block_size()
> +{
> + local block_size=$1
> +
> + # Note: these tests just overwrite some bytes without checking their
> + # original values. Therefore, make sure to overwrite at least 5 or so
> + # bytes, to make it nearly guaranteed that there will be a change --
> + # even when the test file is encrypted due to the test_dummy_encryption
> + # mount option being specified.
> + corruption_test $block_size 131072 0 5
> + corruption_test $block_size 131072 4091 5
> + corruption_test $block_size 131072 65536 65536
> + corruption_test $block_size 131072 131067 5
> +
> + corrupt_eof_block_test $block_size 130999 72
> +
> + # Merkle tree corruption.
> + corruption_test $block_size 200000 100 10 true
> +
> + # Sparse file. Corrupting the Merkle tree should still cause reads to
> + # fail, i.e. the filesystem must verify holes.
> + corruption_test $block_size 200000 100 10 true true
> +}
>
> -# Sparse file. Corrupting the Merkle tree should still cause reads to fail,
> -# i.e. the filesystem must verify holes.
> -corruption_test 200000 100 10 true true
> +# Always test FSV_BLOCK_SIZE. Also test some other block sizes if they happen
> +# to be supported.
> +_fsv_scratch_begin_subtest "Testing block_size=FSV_BLOCK_SIZE"
> +test_block_size $FSV_BLOCK_SIZE
> +for block_size in 1024 4096 16384 65536; do
> + _fsv_scratch_begin_subtest "Testing block_size=$block_size if supported"
> + if (( block_size == FSV_BLOCK_SIZE )); then
> + continue # Skip redundant test case.
> + fi
> + if ! _fsv_can_enable $fsv_file --block-size=$block_size; then
> + echo "block_size=$block_size is unsupported" >> $seqres.full
> + continue
> + fi
> + test_block_size $block_size
> +done
>
> # success, all done
> status=0
> diff --git a/tests/generic/574.out b/tests/generic/574.out
> index d40d1263..3ed57f1c 100644
> --- a/tests/generic/574.out
> +++ b/tests/generic/574.out
> @@ -1,84 +1,11 @@
> QA output created by 574
>
> -# Corruption test: file_len=131072 zap_offset=0 zap_len=5
> -0dfbe8aa4c20b52e1b8bf3cb6cbdf193 SCRATCH_MNT/file.fsv
> -Corrupting bytes...
> -Validating corruption (reading full file)...
> -md5sum: SCRATCH_MNT/file.fsv: Input/output error
> -Validating corruption (direct I/O)...
> -dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
> -Validating corruption (reading just corrupted part)...
> -dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
> -Validating corruption (reading full file via mmap)...
> -Bus error
> -Validating corruption (reading just corrupted part via mmap)...
> -Bus error
> +# Testing block_size=FSV_BLOCK_SIZE
>
> -# Corruption test: file_len=131072 zap_offset=4091 zap_len=5
> -0dfbe8aa4c20b52e1b8bf3cb6cbdf193 SCRATCH_MNT/file.fsv
> -Corrupting bytes...
> -Validating corruption (reading full file)...
> -md5sum: SCRATCH_MNT/file.fsv: Input/output error
> -Validating corruption (direct I/O)...
> -dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
> -Validating corruption (reading just corrupted part)...
> -dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
> -Validating corruption (reading full file via mmap)...
> -Bus error
> -Validating corruption (reading just corrupted part via mmap)...
> -Bus error
> +# Testing block_size=1024 if supported
>
> -# Corruption test: file_len=131072 zap_offset=65536 zap_len=65536
> -0dfbe8aa4c20b52e1b8bf3cb6cbdf193 SCRATCH_MNT/file.fsv
> -Corrupting bytes...
> -Validating corruption (reading full file)...
> -md5sum: SCRATCH_MNT/file.fsv: Input/output error
> -Validating corruption (direct I/O)...
> -dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
> -Validating corruption (reading just corrupted part)...
> -dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
> -Validating corruption (reading full file via mmap)...
> -Bus error
> -Validating corruption (reading just corrupted part via mmap)...
> -Bus error
> +# Testing block_size=4096 if supported
>
> -# Corruption test: file_len=131072 zap_offset=131067 zap_len=5
> -0dfbe8aa4c20b52e1b8bf3cb6cbdf193 SCRATCH_MNT/file.fsv
> -Corrupting bytes...
> -Validating corruption (reading full file)...
> -md5sum: SCRATCH_MNT/file.fsv: Input/output error
> -Validating corruption (direct I/O)...
> -dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
> -Validating corruption (reading just corrupted part)...
> -dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
> -Validating corruption (reading full file via mmap)...
> -Bus error
> -Validating corruption (reading just corrupted part via mmap)...
> -Bus error
> +# Testing block_size=16384 if supported
>
> -# Corruption test: EOF block
> -f5cca0d7fbb8b02bc6118a9954d5d306 SCRATCH_MNT/file.fsv
> -Corrupting bytes...
> -Reading eof block via mmap into a temporary file...
> -Checking for SIGBUS or zeros...
> -OK
> -
> -# Corruption test: file_len=200000 zap_offset=100 (in Merkle tree) zap_len=10
> -4a1e4325031b13f933ac4f1db9ecb63f SCRATCH_MNT/file.fsv
> -Corrupting bytes...
> -Validating corruption (reading full file)...
> -md5sum: SCRATCH_MNT/file.fsv: Input/output error
> -Validating corruption (direct I/O)...
> -dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
> -Validating corruption (reading full file via mmap)...
> -Bus error
> -
> -# Corruption test: file_len=200000 (sparse) zap_offset=100 (in Merkle tree) zap_len=10
> -4a1e4325031b13f933ac4f1db9ecb63f SCRATCH_MNT/file.fsv
> -Corrupting bytes...
> -Validating corruption (reading full file)...
> -md5sum: SCRATCH_MNT/file.fsv: Input/output error
> -Validating corruption (direct I/O)...
> -dd: error reading 'SCRATCH_MNT/file.fsv': Input/output error
> -Validating corruption (reading full file via mmap)...
> -Bus error
> +# Testing block_size=65536 if supported
> --
> 2.39.0
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 08/10] generic/574: test multiple Merkle tree block sizes
2022-12-25 12:46 ` Zorro Lang
@ 2022-12-26 5:21 ` Eric Biggers
2022-12-28 12:50 ` Theodore Ts'o
2022-12-29 16:32 ` Zorro Lang
0 siblings, 2 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-26 5:21 UTC (permalink / raw)
To: Zorro Lang; +Cc: fstests, linux-fscrypt
On Sun, Dec 25, 2022 at 08:46:00PM +0800, Zorro Lang wrote:
> > + # Reading the full file via mmap should fail.
> > bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file \
> > -c 'mmap -r 0 $page_aligned_eof' \
> > - -c 'mread 0 $file_len'" |& filter_sigbus
> > + -c 'mread 0 $file_len'" >/dev/null 2>$tmp.out
> > + if ! grep -q 'Bus error' $tmp.out; then
> > + echo "Didn't see SIGBUS when reading file via mmap"
>
> Hmm... will sigbus error really be output to stderr? From a testing, the
> generic/574 fails as:
>
> # ./check -s simpledev generic/574
> SECTION -- simpledev
> FSTYP -- ext4
> PLATFORM -- Linux/x86_64 xx-xxxxxx-xxx 6.1.0-rc3 #5 SMP PREEMPT_DYNAMIC Tue Nov 1 01:08:52 CST 2022
> MKFS_OPTIONS -- -F /dev/sdb
> MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 /dev/sdb /mnt/scratch
>
> generic/574 - output mismatch (see /root/git/xfstests/results//simpledev/generic/574.out.bad)
> --- tests/generic/574.out 2022-12-25 20:02:41.609104749 +0800
> +++ /root/git/xfstests/results//simpledev/generic/574.out.bad 2022-12-25 20:21:57.430719504 +0800
> @@ -1,6 +1,32 @@
> QA output created by 574
>
> # Testing block_size=FSV_BLOCK_SIZE
> +/root/git/xfstests/tests/generic/574: line 69: 1949533 Bus error (core dumped) bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file -c 'mmap -r 0 $page_aligned_eof' -c 'mread 0 $file_len'" > /dev/null 2> $tmp.out
> +Didn't see SIGBUS when reading file via mmap
This test passes for me both before and after this patch series.
Both before and after, the way this is supposed to work is that in:
bash -c "trap '' SIGBUS; command_that_exits_with_sigbus"
... bash should print "Bus error" to stderr due to
'command_that_exits_with_sigbus'. That "Bus error" is then redirected. Before
it was redirected to a pipeline; after it is redirected to a file.
I think what's happening is that the version of bash your system has is not
forking before exec'ing 'command_that_exits_with_sigbus'. As a result, "Bus
error" is printed by the *parent* bash process instead, skipping any redirection
in the shell script.
Apparently skipping fork is a real thing in bash, and different versions of bash
have had subtly different conditions for enabling it. So this seems plausible.
Adding an extra command after 'command_that_exits_with_sigbus' should fix this:
bash -c "trap '' SIGBUS; command_that_exits_with_sigbus; true"
The joy of working with a shell scripting system where everyone has different
versions of everything installed...
- Eric
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 08/10] generic/574: test multiple Merkle tree block sizes
2022-12-26 5:21 ` Eric Biggers
@ 2022-12-28 12:50 ` Theodore Ts'o
2022-12-29 16:32 ` Zorro Lang
1 sibling, 0 replies; 16+ messages in thread
From: Theodore Ts'o @ 2022-12-28 12:50 UTC (permalink / raw)
To: Eric Biggers; +Cc: Zorro Lang, fstests, linux-fscrypt
On Sun, Dec 25, 2022 at 09:21:34PM -0800, Eric Biggers wrote:
> The joy of working with a shell scripting system where everyone has different
> versions of everything installed...
And this is why some large companies are still stuck on Bash v3
because the SRE's are afraid that untold number of shell scripts will
break in subtle and entertaining ways, leading to customer outages...
- Ted
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 08/10] generic/574: test multiple Merkle tree block sizes
2022-12-26 5:21 ` Eric Biggers
2022-12-28 12:50 ` Theodore Ts'o
@ 2022-12-29 16:32 ` Zorro Lang
2022-12-29 23:47 ` Eric Biggers
1 sibling, 1 reply; 16+ messages in thread
From: Zorro Lang @ 2022-12-29 16:32 UTC (permalink / raw)
To: Eric Biggers; +Cc: fstests, linux-fscrypt
On Sun, Dec 25, 2022 at 09:21:34PM -0800, Eric Biggers wrote:
> On Sun, Dec 25, 2022 at 08:46:00PM +0800, Zorro Lang wrote:
> > > + # Reading the full file via mmap should fail.
> > > bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file \
> > > -c 'mmap -r 0 $page_aligned_eof' \
> > > - -c 'mread 0 $file_len'" |& filter_sigbus
> > > + -c 'mread 0 $file_len'" >/dev/null 2>$tmp.out
> > > + if ! grep -q 'Bus error' $tmp.out; then
> > > + echo "Didn't see SIGBUS when reading file via mmap"
> >
> > Hmm... will sigbus error really be output to stderr? From a testing, the
> > generic/574 fails as:
> >
> > # ./check -s simpledev generic/574
> > SECTION -- simpledev
> > FSTYP -- ext4
> > PLATFORM -- Linux/x86_64 xx-xxxxxx-xxx 6.1.0-rc3 #5 SMP PREEMPT_DYNAMIC Tue Nov 1 01:08:52 CST 2022
> > MKFS_OPTIONS -- -F /dev/sdb
> > MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:root_t:s0 /dev/sdb /mnt/scratch
> >
> > generic/574 - output mismatch (see /root/git/xfstests/results//simpledev/generic/574.out.bad)
> > --- tests/generic/574.out 2022-12-25 20:02:41.609104749 +0800
> > +++ /root/git/xfstests/results//simpledev/generic/574.out.bad 2022-12-25 20:21:57.430719504 +0800
> > @@ -1,6 +1,32 @@
> > QA output created by 574
> >
> > # Testing block_size=FSV_BLOCK_SIZE
> > +/root/git/xfstests/tests/generic/574: line 69: 1949533 Bus error (core dumped) bash -c "trap '' SIGBUS; $XFS_IO_PROG -r $fsv_file -c 'mmap -r 0 $page_aligned_eof' -c 'mread 0 $file_len'" > /dev/null 2> $tmp.out
> > +Didn't see SIGBUS when reading file via mmap
>
> This test passes for me both before and after this patch series.
>
> Both before and after, the way this is supposed to work is that in:
>
> bash -c "trap '' SIGBUS; command_that_exits_with_sigbus"
>
> ... bash should print "Bus error" to stderr due to
> 'command_that_exits_with_sigbus'. That "Bus error" is then redirected. Before
> it was redirected to a pipeline; after it is redirected to a file.
>
> I think what's happening is that the version of bash your system has is not
> forking before exec'ing 'command_that_exits_with_sigbus'. As a result, "Bus
> error" is printed by the *parent* bash process instead, skipping any redirection
> in the shell script.
>
> Apparently skipping fork is a real thing in bash, and different versions of bash
> have had subtly different conditions for enabling it. So this seems plausible.
>
> Adding an extra command after 'command_that_exits_with_sigbus' should fix this:
>
> bash -c "trap '' SIGBUS; command_that_exits_with_sigbus; true"
Thanks for this explanation, I think you're right!
I'm not sure if it's a bug of bash. If it's not a bug, I think we can do this
change (add a true) to avoid that failure. If it's a bug, hmmm..., I think we'd
better to avoid that failure too, due to we don't test for bash :-/
How about your resend this single patch (by version 2.1), to fix this problem.
Other patches looks good to me, I'd like to merge this patchset this weekend.
Thanks,
Zorro
>
> The joy of working with a shell scripting system where everyone has different
> versions of everything installed...
>
> - Eric
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 08/10] generic/574: test multiple Merkle tree block sizes
2022-12-29 16:32 ` Zorro Lang
@ 2022-12-29 23:47 ` Eric Biggers
0 siblings, 0 replies; 16+ messages in thread
From: Eric Biggers @ 2022-12-29 23:47 UTC (permalink / raw)
To: Zorro Lang; +Cc: fstests, linux-fscrypt
On Fri, Dec 30, 2022 at 12:32:15AM +0800, Zorro Lang wrote:
> >
> > This test passes for me both before and after this patch series.
> >
> > Both before and after, the way this is supposed to work is that in:
> >
> > bash -c "trap '' SIGBUS; command_that_exits_with_sigbus"
> >
> > ... bash should print "Bus error" to stderr due to
> > 'command_that_exits_with_sigbus'. That "Bus error" is then redirected. Before
> > it was redirected to a pipeline; after it is redirected to a file.
> >
> > I think what's happening is that the version of bash your system has is not
> > forking before exec'ing 'command_that_exits_with_sigbus'. As a result, "Bus
> > error" is printed by the *parent* bash process instead, skipping any redirection
> > in the shell script.
> >
> > Apparently skipping fork is a real thing in bash, and different versions of bash
> > have had subtly different conditions for enabling it. So this seems plausible.
> >
> > Adding an extra command after 'command_that_exits_with_sigbus' should fix this:
> >
> > bash -c "trap '' SIGBUS; command_that_exits_with_sigbus; true"
>
> Thanks for this explanation, I think you're right!
>
> I'm not sure if it's a bug of bash. If it's not a bug, I think we can do this
> change (add a true) to avoid that failure. If it's a bug, hmmm..., I think we'd
> better to avoid that failure too, due to we don't test for bash :-/
>
> How about your resend this single patch (by version 2.1), to fix this problem.
> Other patches looks good to me, I'd like to merge this patchset this weekend.
>
> Thanks,
> Zorro
I expect that the bash developers wouldn't consider this to be a bug, since it
could only be fixed by removing the "skip fork" optimization entirely.
Anyway, the workaround I suggested is simple enough.
I just resent the whole series, since it's confusing to have new versions of
patches within existing series versions. But only this one patch changed.
Thanks!
- Eric
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2022-12-29 23:47 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-12-23 1:05 [PATCH v2 00/10] xfstests: update verity tests for non-4K block and page size Eric Biggers
2022-12-23 1:05 ` [PATCH v2 01/10] common/verity: add and use _fsv_can_enable() Eric Biggers
2022-12-23 1:05 ` [PATCH v2 02/10] common/verity: set FSV_BLOCK_SIZE to an appropriate value Eric Biggers
2022-12-23 1:05 ` [PATCH v2 03/10] common/verity: use FSV_BLOCK_SIZE by default Eric Biggers
2022-12-23 1:05 ` [PATCH v2 04/10] common/verity: add _filter_fsverity_digest() Eric Biggers
2022-12-23 1:05 ` [PATCH v2 05/10] generic/572: support non-4K Merkle tree block size Eric Biggers
2022-12-23 1:05 ` [PATCH v2 06/10] generic/573: " Eric Biggers
2022-12-23 1:05 ` [PATCH v2 07/10] generic/577: " Eric Biggers
2022-12-23 1:05 ` [PATCH v2 08/10] generic/574: test multiple Merkle tree block sizes Eric Biggers
2022-12-25 12:46 ` Zorro Lang
2022-12-26 5:21 ` Eric Biggers
2022-12-28 12:50 ` Theodore Ts'o
2022-12-29 16:32 ` Zorro Lang
2022-12-29 23:47 ` Eric Biggers
2022-12-23 1:05 ` [PATCH v2 09/10] generic/624: " Eric Biggers
2022-12-23 1:05 ` [PATCH v2 10/10] generic/575: test 1K Merkle tree block size Eric Biggers
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.