[PATCH 0/5] btrfs-progs: tests: Make 64K page size system happier

[PATCH 0/5] btrfs-progs: tests: Make 64K page size system happier

[Qu Wenruo]

Since I got another rockpro64, finally I could do some tests with
aarch64 64K page size mode. (The first board is working as a NAS for
a while)

Unsurprisingly there are several false test alerts in btrfs-progs
selftests.

Although there is no existing CI service based on 64K page sized system,
we'd better support for 64K page size as it's easier and easier to get
SBC with good enough aarch64 SoC to compile kernel/btrfs-progs and run
various tests on them.

The first patch fix a bug which mkfs can't accept any sector size on 64K
page size system.

The remaining patches enhance test cases to make them work on 64K page
size system (skip those tests unless kernel support subpage sized sector
size)

Qu Wenruo (5):
  btrfs-progs: mkfs: Apply the sectorsize user specified on 64k page
    size system
  btrfs-progs: fsck-tests: Check if current kernel can mount fs with
    specified sector size
  btrfs-progs: mkfs-tests: Skip 010-minimal-size if we can't mount with
    4k sector size
  btrfs-progs: misc-tests: Make test cases work or skipped on 64K page
    size system
  btrfs-progs: convert-tests: Skip tests if kernel doesn't support
    subpage sized sector size

 mkfs/main.c                                   | 12 +++++-
 tests/common                                  | 29 +++++++++++++
 tests/convert-tests/001-ext2-basic/test.sh    |  1 +
 tests/convert-tests/002-ext3-basic/test.sh    |  1 +
 tests/convert-tests/003-ext4-basic/test.sh    |  1 +
 .../004-ext2-backup-superblock-ranges/test.sh |  1 +
 .../005-delete-all-rollback/test.sh           |  1 +
 .../006-large-hole-extent/test.sh             |  2 +
 .../convert-tests/008-readonly-image/test.sh  |  1 +
 .../009-common-inode-flags/test.sh            |  1 +
 .../convert-tests/010-reiserfs-basic/test.sh  |  2 +
 .../011-reiserfs-delete-all-rollback/test.sh  |  1 +
 .../012-reiserfs-large-hole-extent/test.sh    |  1 +
 .../013-reiserfs-common-inode-flags/test.sh   |  1 +
 .../014-reiserfs-tail-handling/test.sh        |  1 +
 .../015-no-rollback-after-balance/test.sh     |  1 +
 .../016-invalid-large-inline-extent/test.sh   |  1 +
 tests/fsck-tests/012-leaf-corruption/test.sh  |  1 +
 .../028-unaligned-super-dev-sizes/test.sh     |  1 +
 .../037-freespacetree-repair/test.sh          |  3 +-
 .../010-convert-delete-ext2-subvol/test.sh    |  5 ++-
 tests/mkfs-tests/010-minimal-size/test.sh     | 41 ++++++++++---------
 22 files changed, 86 insertions(+), 23 deletions(-)

-- 
2.22.0

[PATCH 1/5] btrfs-progs: mkfs: Apply the sectorsize user specified on 64k page size system

[Qu Wenruo]

[BUG]
On aarch64 with 64k page size, mkfs.btrfs -s option doesn't work:
  $ mkfs.btrfs  -s 4096 ~/10G.img  -f
  btrfs-progs v5.1.1
  See http://btrfs.wiki.kernel.org for more information.

  Label:              (null)
  UUID:               c2a09334-aaca-4980-aefa-4b3e27390658
  Node size:          65536
  Sector size:        65536		<< Still 64K, not 4K
  Filesystem size:    10.00GiB
  Block group profiles:
    Data:             single            8.00MiB
    Metadata:         DUP             256.00MiB
    System:           DUP               8.00MiB
  SSD detected:       no
  Incompat features:  extref, skinny-metadata
  Number of devices:  1
  Devices:
     ID        SIZE  PATH
      1    10.00GiB  /home/adam/10G.img

[CAUSE]
This is because we automatically detect sectorsize based on current
system page size, then get the maxium number between user specified -s
parameter and system page size.

It's fine for x86 as it has fixed page size 4K, also the minimium valid
sector size.

But for system like aarch64 or ppc64le, where we can have 64K page size,
and it makes us unable to create a 4k sector sized btrfs.

[FIX]
Only do auto detect when no -s|--sectorsize option is specified.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 mkfs/main.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/mkfs/main.c b/mkfs/main.c
index 8dbec0717b89..26d84e9dafc3 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -817,6 +817,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	char *source_dir = NULL;
 	bool source_dir_set = false;
 	bool shrink_rootdir = false;
+	bool sectorsize_set = false;
 	u64 source_dir_size = 0;
 	u64 min_dev_size;
 	u64 shrink_size;
@@ -906,6 +907,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 				}
 			case 's':
 				sectorsize = parse_size(optarg);
+				sectorsize_set = true;
 				break;
 			case 'b':
 				block_count = parse_size(optarg);
@@ -943,7 +945,15 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 		printf("See %s for more information.\n\n", PACKAGE_URL);
 	}
 
-	sectorsize = max(sectorsize, (u32)sysconf(_SC_PAGESIZE));
+	if (!sectorsize_set)
+		sectorsize = max(sectorsize, (u32)sysconf(_SC_PAGESIZE));
+	if (!is_power_of_2(sectorsize) || sectorsize < 4096 ||
+	    sectorsize > SZ_64K) {
+		error(
+		"invalid sectorsize: %u, expect either 4k, 8k, 16k, 32k or 64k",
+			sectorsize);
+		goto error;
+	}
 	stripesize = sectorsize;
 	saved_optind = optind;
 	dev_cnt = argc - optind;
-- 
2.22.0

[PATCH 2/5] btrfs-progs: fsck-tests: Check if current kernel can mount fs with specified sector size

[Qu Wenruo]

[BUG]
When doing test on platforms with page size other than 4K (e.g aarch64
can use 64K page size, or ppc64le), certain test wills fail like:
      [TEST/fsck]   012-leaf-corruption
  mount: /home/adam/btrfs-progs/tests/mnt: wrong fs type, bad option, bad superblock on /dev/loop4, missing codepage or helper program, or other error.
  find: '/home/adam/btrfs-progs/tests//mnt/lost+found': No such file or directory
  inode 1862 not recovered correctly
  test failed for case 012-leaf-corruption

      [TEST/fsck]   028-unaligned-super-dev-sizes
  failed: mount -t btrfs -o loop ./dev_and_super_mismatch_unaligned.raw.restored /home/adam/btrfs-progs/tests//mnt
  test failed for case 028-unaligned-super-dev-sizes

      [TEST/fsck]   037-freespacetree-repair
  failed: /home/adam/btrfs-progs/mkfs.btrfs -f -n 4k /home/adam/btrfs-progs/tests//test.img
  test failed for case 037-freespacetree-repair

[CAUSE]

For fsck/012 and fsck/028, it's caused by the lack of subpage size sector
size support, thus we require kernel page size to match on-disk sector size:
  BTRFS error (device loop4): sectorsize 4096 not supported yet, only support 65536
  BTRFS error (device loop4): superblock contains fatal errors
  BTRFS error (device loop4): open_ctree failed

For fsck/037, it's mkfs causing the problem as we're using 4k nodesize,
but on 64K page sized system, we will use 64K sectorsize and cause
conflicts.

[FIX]
Considering it's easier and easier to get aarch64 boards with enough
performance (e.g rpi4, rk3399, S922) to compile kernel and run tests,
let's skip such tests before widespread complain comes.

This patch will introduce a new check, check_prereq_mount_with_sectorsize(),
which will test if kernel can mount btrfs with specified sectorsize.
So that even one day we support subpage sized sectorsize, we won't need
to update test case again.

For fsck/037, also specify sector size manually. And since in that case
we still need to mount the fs, also add
check_prereq_mount_with_sectorsize() call.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/common                                  | 29 +++++++++++++++++++
 tests/fsck-tests/012-leaf-corruption/test.sh  |  1 +
 .../028-unaligned-super-dev-sizes/test.sh     |  1 +
 .../037-freespacetree-repair/test.sh          |  3 +-
 4 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/tests/common b/tests/common
index 79a16f1e187d..5ad16b69b61d 100644
--- a/tests/common
+++ b/tests/common
@@ -306,6 +306,35 @@ check_prereq()
 	fi
 }
 
+# Require to mount images with speicified sectorsize
+# This is to make sure we can run this test on different arch
+# (e.g aarch64 with 64K pagesize)
+check_prereq_mount_with_sectorsize()
+{
+	prepare_test_dev 128M
+	check_prereq mkfs.btrfs
+	setup_root_helper
+
+	local sectorsize=$1
+	local loop_opt
+
+	if [[ -b "$TEST_DEV" ]]; then
+		loop_opt=""
+	elif [[ -f "$TEST_DEV" ]]; then
+		loop_opt="-o loop"
+	else
+		_fail "Invalid \$TEST_DEV: $TEST_DEV"
+	fi
+
+	run_check_mkfs_test_dev -f -s $sectorsize
+	$SUDO_HELPER mount -t btrfs $loop_opt "$TEST_DEV" "$TEST_MNT" \
+		&> /dev/null
+	if [ $? -ne 0 ]; then
+		_not_run "kernel doesn't support sectorsize $sectorsize"
+	fi
+	run_check_umount_test_dev
+}
+
 check_global_prereq()
 {
 	which "$1" &> /dev/null
diff --git a/tests/fsck-tests/012-leaf-corruption/test.sh b/tests/fsck-tests/012-leaf-corruption/test.sh
index 68d9f695d4de..d5da1d210f28 100755
--- a/tests/fsck-tests/012-leaf-corruption/test.sh
+++ b/tests/fsck-tests/012-leaf-corruption/test.sh
@@ -107,6 +107,7 @@ check_leaf_corrupt_no_data_ext()
 
 setup_root_helper
 
+check_prereq_mount_with_sectorsize 4096
 generate_leaf_corrupt_no_data_ext test.img
 check_image test.img
 check_leaf_corrupt_no_data_ext test.img
diff --git a/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh b/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh
index 4015df2d8570..49fa35241d04 100755
--- a/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh
+++ b/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh
@@ -6,6 +6,7 @@
 source "$TEST_TOP/common"
 
 check_prereq btrfs
+check_prereq_mount_with_sectorsize 4096
 setup_root_helper
 
 check_all_images
diff --git a/tests/fsck-tests/037-freespacetree-repair/test.sh b/tests/fsck-tests/037-freespacetree-repair/test.sh
index 7f547a33512d..32e6651ac705 100755
--- a/tests/fsck-tests/037-freespacetree-repair/test.sh
+++ b/tests/fsck-tests/037-freespacetree-repair/test.sh
@@ -10,6 +10,7 @@ prepare_test_dev 256M
 
 check_prereq btrfs
 check_prereq mkfs.btrfs
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq grep
 check_global_prereq tail
 check_global_prereq head
@@ -55,7 +56,7 @@ if ! [ -f "/sys/fs/btrfs/features/free_space_tree" ]; then
 	exit
 fi
 
-run_check_mkfs_test_dev -n 4k
+run_check_mkfs_test_dev -s 4k -n 4k
 run_check_mount_test_dev -oclear_cache,space_cache=v2
 
 # create files which will populate the FST
-- 
2.22.0

[PATCH 3/5] btrfs-progs: mkfs-tests: Skip 010-minimal-size if we can't mount with 4k sector size

[Qu Wenruo]

[BUG]
Test case 010-minimal-size fails on aarch64 with 64K page size:
      [TEST/mkfs]   010-minimal-size
  failed: /home/adam/btrfs-progs/mkfs.btrfs -f -n 4k -m single -d single /home/adam/btrfs-progs/tests//test.img
  test failed for case 010-minimal-size
  make: *** [Makefile:361: test-mkfs] Error 1

[CAUSE]
Mkfs.btrfs defaults to page size as sector size. However this test uses
4k, 16k, 32K and 64K as node size. 4K node size will conflict with 64K
sector size.

[FIX]
- Specify sector size 4K manually
  So at least no conflict at mkfs time.

- Skip the test case if kernel can't mount with 4k sector size
  So once we add such support, the test can be automatically re-enabled.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/mkfs-tests/010-minimal-size/test.sh | 41 ++++++++++++-----------
 1 file changed, 21 insertions(+), 20 deletions(-)

diff --git a/tests/mkfs-tests/010-minimal-size/test.sh b/tests/mkfs-tests/010-minimal-size/test.sh
index 8480e4c5ae23..b49fad63e519 100755
--- a/tests/mkfs-tests/010-minimal-size/test.sh
+++ b/tests/mkfs-tests/010-minimal-size/test.sh
@@ -5,6 +5,7 @@ source "$TEST_TOP/common"
 
 check_prereq mkfs.btrfs
 check_prereq btrfs
+check_prereq_mount_with_sectorsize 4096
 
 setup_root_helper
 
@@ -24,20 +25,20 @@ do_test()
 	run_check_umount_test_dev
 }
 
-do_test -n 4k	-m single	-d single
-do_test -n 4k	-m single	-d dup
-do_test -n 4k	-m dup		-d single
-do_test -n 4k	-m dup		-d dup
+do_test -s 4k	-n 4k	-m single	-d single
+do_test -s 4k	-n 4k	-m single	-d dup
+do_test -s 4k	-n 4k	-m dup		-d single
+do_test -s 4k	-n 4k	-m dup		-d dup
 
-do_test -n 8k	-m single	-d single
-do_test -n 8k	-m single	-d dup
-do_test -n 8k	-m dup		-d single
-do_test -n 8k	-m dup		-d dup
+do_test -s 4k	-n 8k	-m single	-d single
+do_test -s 4k	-n 8k	-m single	-d dup
+do_test -s 4k	-n 8k	-m dup		-d single
+do_test -s 4k	-n 8k	-m dup		-d dup
 
-do_test -n 16k	-m single	-d single
-do_test -n 16k	-m single	-d dup
-do_test -n 16k	-m dup		-d single
-do_test -n 16k	-m dup		-d dup
+do_test -s 4k	-n 16k	-m single	-d single
+do_test -s 4k	-n 16k	-m single	-d dup
+do_test -s 4k	-n 16k	-m dup		-d single
+do_test -s 4k	-n 16k	-m dup		-d dup
 
 # Temporary: disable the following tests as they fail inside travis but run
 # fine otherwise. This is probably caused by kernel version, 4.4 fails and 4.14
@@ -52,12 +53,12 @@ if [ "$TRAVIS" = true ]; then
 	exit 0
 fi
 
-do_test -n 32k	-m single	-d single
-do_test -n 32k	-m single	-d dup
-do_test -n 32k	-m dup		-d single
-do_test -n 32k	-m dup		-d dup
+do_test -s 4k	-n 32k	-m single	-d single
+do_test -s 4k	-n 32k	-m single	-d dup
+do_test -s 4k	-n 32k	-m dup		-d single
+do_test -s 4k	-n 32k	-m dup		-d dup
 
-do_test -n 64k	-m single	-d single
-do_test -n 64k	-m single	-d dup
-do_test -n 64k	-m dup		-d single
-do_test -n 64k	-m dup		-d dup
+do_test -s 4k	-n 64k	-m single	-d single
+do_test -s 4k	-n 64k	-m single	-d dup
+do_test -s 4k	-n 64k	-m dup		-d single
+do_test -s 4k	-n 64k	-m dup		-d dup
-- 
2.22.0

[PATCH 4/5] btrfs-progs: misc-tests: Make test cases work or skipped on 64K page size system

[Qu Wenruo]

[BUG]
The following test cases fails on aarch64 64K page size mode:
      [TEST/misc]   010-convert-delete-ext2-subvol
  failed: mount -t btrfs -o loop /home/adam/btrfs-progs/tests//test.img /home/adam/btrfs-progs/tests//mnt
  test failed for case 010-convert-delete-ext2-subvol
  make: *** [Makefile:387: test-misc] Error 1

[CAUSE]
Most of them are caused by the lack of subpage sized sector size
support.
Or the conflicts sector size (64K) and nodesize (smaller than 64K).

[FIX]
Check if the current kernel accepts subpage sized sector size and
manually specify 4K sector size.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/misc-tests/010-convert-delete-ext2-subvol/test.sh | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tests/misc-tests/010-convert-delete-ext2-subvol/test.sh b/tests/misc-tests/010-convert-delete-ext2-subvol/test.sh
index 5f441a7f5eb9..aa741e7aca7f 100755
--- a/tests/misc-tests/010-convert-delete-ext2-subvol/test.sh
+++ b/tests/misc-tests/010-convert-delete-ext2-subvol/test.sh
@@ -7,12 +7,15 @@ source "$TEST_TOP/common"
 
 check_prereq btrfs-convert
 check_prereq btrfs
+check_global_prereq getconf
+
+page_size=$(getconf PAGESIZE)
 
 setup_root_helper
 prepare_test_dev
 
 run_check truncate -s 2G "$TEST_DEV"
-run_check mkfs.ext4 -F "$TEST_DEV"
+run_check mkfs.ext4 -F "$TEST_DEV" -b $page_size
 run_check "$TOP/btrfs-convert" "$TEST_DEV"
 run_check $SUDO_HELPER "$TOP/btrfs" inspect-internal dump-tree "$TEST_DEV"
 run_check_mount_test_dev
-- 
2.22.0

[PATCH 5/5] btrfs-progs: convert-tests: Skip tests if kernel doesn't support subpage sized sector size

[Qu Wenruo]

Most convert tests needs to mount the converted image, and both reiserfs
and ext* uses 4k block size, on 32K page size system we can't mount them
and will cause test failure.

Skip most of convert tests, except 007-unsupported-block-sizes, which
should fail on all systems.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/convert-tests/001-ext2-basic/test.sh                    | 1 +
 tests/convert-tests/002-ext3-basic/test.sh                    | 1 +
 tests/convert-tests/003-ext4-basic/test.sh                    | 1 +
 tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh | 1 +
 tests/convert-tests/005-delete-all-rollback/test.sh           | 1 +
 tests/convert-tests/006-large-hole-extent/test.sh             | 2 ++
 tests/convert-tests/008-readonly-image/test.sh                | 1 +
 tests/convert-tests/009-common-inode-flags/test.sh            | 1 +
 tests/convert-tests/010-reiserfs-basic/test.sh                | 2 ++
 tests/convert-tests/011-reiserfs-delete-all-rollback/test.sh  | 1 +
 tests/convert-tests/012-reiserfs-large-hole-extent/test.sh    | 1 +
 tests/convert-tests/013-reiserfs-common-inode-flags/test.sh   | 1 +
 tests/convert-tests/014-reiserfs-tail-handling/test.sh        | 1 +
 tests/convert-tests/015-no-rollback-after-balance/test.sh     | 1 +
 tests/convert-tests/016-invalid-large-inline-extent/test.sh   | 1 +
 15 files changed, 17 insertions(+)

diff --git a/tests/convert-tests/001-ext2-basic/test.sh b/tests/convert-tests/001-ext2-basic/test.sh
index 74cc74e86fe4..94be68bc6b84 100755
--- a/tests/convert-tests/001-ext2-basic/test.sh
+++ b/tests/convert-tests/001-ext2-basic/test.sh
@@ -4,6 +4,7 @@ source "$TEST_TOP/common"
 source "$TEST_TOP/common.convert"
 
 setup_root_helper
+check_prereq_mount_with_sectorsize 4096
 prepare_test_dev
 check_prereq btrfs-convert
 check_global_prereq mke2fs
diff --git a/tests/convert-tests/002-ext3-basic/test.sh b/tests/convert-tests/002-ext3-basic/test.sh
index f08698976c15..57e83681adb7 100755
--- a/tests/convert-tests/002-ext3-basic/test.sh
+++ b/tests/convert-tests/002-ext3-basic/test.sh
@@ -4,6 +4,7 @@ source "$TEST_TOP/common"
 source "$TEST_TOP/common.convert"
 
 setup_root_helper
+check_prereq_mount_with_sectorsize 4096
 prepare_test_dev
 check_prereq btrfs-convert
 check_global_prereq mke2fs
diff --git a/tests/convert-tests/003-ext4-basic/test.sh b/tests/convert-tests/003-ext4-basic/test.sh
index c5caf67c9300..d20d854fc387 100755
--- a/tests/convert-tests/003-ext4-basic/test.sh
+++ b/tests/convert-tests/003-ext4-basic/test.sh
@@ -4,6 +4,7 @@ source "$TEST_TOP/common"
 source "$TEST_TOP/common.convert"
 
 setup_root_helper
+check_prereq_mount_with_sectorsize 4096
 prepare_test_dev
 check_prereq btrfs-convert
 check_global_prereq mke2fs
diff --git a/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh b/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh
index 857e240837dc..620374e857d1 100755
--- a/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh
+++ b/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh
@@ -14,6 +14,7 @@ source "$TEST_TOP/common"
 
 check_prereq btrfs-convert
 check_prereq btrfs
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq e2fsck
 check_global_prereq xzcat
 
diff --git a/tests/convert-tests/005-delete-all-rollback/test.sh b/tests/convert-tests/005-delete-all-rollback/test.sh
index a5f9d594bfd5..431c642b221d 100755
--- a/tests/convert-tests/005-delete-all-rollback/test.sh
+++ b/tests/convert-tests/005-delete-all-rollback/test.sh
@@ -8,6 +8,7 @@ source "$TEST_TOP/common.convert"
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq mke2fs
 
 # simple wrapper for a convert test
diff --git a/tests/convert-tests/006-large-hole-extent/test.sh b/tests/convert-tests/006-large-hole-extent/test.sh
index a37fcbdc99a6..7a974d74e47f 100755
--- a/tests/convert-tests/006-large-hole-extent/test.sh
+++ b/tests/convert-tests/006-large-hole-extent/test.sh
@@ -11,6 +11,8 @@ source "$TEST_TOP/common.convert"
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
+prepare_test_dev
 check_global_prereq mke2fs
 
 default_mke2fs="mke2fs -t ext4 -b 4096"
diff --git a/tests/convert-tests/008-readonly-image/test.sh b/tests/convert-tests/008-readonly-image/test.sh
index 1a65ea6b3a19..08d87b0d4433 100755
--- a/tests/convert-tests/008-readonly-image/test.sh
+++ b/tests/convert-tests/008-readonly-image/test.sh
@@ -7,6 +7,7 @@ source "$TEST_TOP/common.convert"
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq mke2fs
 
 default_mke2fs="mke2fs -t ext4 -b 4096"
diff --git a/tests/convert-tests/009-common-inode-flags/test.sh b/tests/convert-tests/009-common-inode-flags/test.sh
index 428213bfcb15..22702af338f7 100755
--- a/tests/convert-tests/009-common-inode-flags/test.sh
+++ b/tests/convert-tests/009-common-inode-flags/test.sh
@@ -7,6 +7,7 @@ source "$TEST_TOP/common.convert"
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq mke2fs
 check_global_prereq lsattr
 check_global_prereq chattr
diff --git a/tests/convert-tests/010-reiserfs-basic/test.sh b/tests/convert-tests/010-reiserfs-basic/test.sh
index 73652991cc6b..7f14956fd853 100755
--- a/tests/convert-tests/010-reiserfs-basic/test.sh
+++ b/tests/convert-tests/010-reiserfs-basic/test.sh
@@ -10,6 +10,8 @@ fi
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
+prepare_test_dev
 check_global_prereq mkreiserfs
 
 for feature in '' 'extref' 'skinny-metadata' 'no-holes'; do
diff --git a/tests/convert-tests/011-reiserfs-delete-all-rollback/test.sh b/tests/convert-tests/011-reiserfs-delete-all-rollback/test.sh
index 28877e142483..36b3b3888ddb 100755
--- a/tests/convert-tests/011-reiserfs-delete-all-rollback/test.sh
+++ b/tests/convert-tests/011-reiserfs-delete-all-rollback/test.sh
@@ -11,6 +11,7 @@ fi
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq mkreiserfs
 
 # simple wrapper for a convert test
diff --git a/tests/convert-tests/012-reiserfs-large-hole-extent/test.sh b/tests/convert-tests/012-reiserfs-large-hole-extent/test.sh
index d492779a386b..55d17cd54bf2 100755
--- a/tests/convert-tests/012-reiserfs-large-hole-extent/test.sh
+++ b/tests/convert-tests/012-reiserfs-large-hole-extent/test.sh
@@ -15,6 +15,7 @@ fi
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq mkreiserfs
 
 default_mkfs="mkreiserfs -b 4096"
diff --git a/tests/convert-tests/013-reiserfs-common-inode-flags/test.sh b/tests/convert-tests/013-reiserfs-common-inode-flags/test.sh
index 521e8bd4fba4..cf83b29b4d82 100755
--- a/tests/convert-tests/013-reiserfs-common-inode-flags/test.sh
+++ b/tests/convert-tests/013-reiserfs-common-inode-flags/test.sh
@@ -11,6 +11,7 @@ fi
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq mkreiserfs
 check_global_prereq chattr
 check_global_prereq lsattr
diff --git a/tests/convert-tests/014-reiserfs-tail-handling/test.sh b/tests/convert-tests/014-reiserfs-tail-handling/test.sh
index 3be2ed5b6299..a2757f4674d6 100755
--- a/tests/convert-tests/014-reiserfs-tail-handling/test.sh
+++ b/tests/convert-tests/014-reiserfs-tail-handling/test.sh
@@ -16,6 +16,7 @@ fi
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq md5sum
 check_global_prereq mkreiserfs
 
diff --git a/tests/convert-tests/015-no-rollback-after-balance/test.sh b/tests/convert-tests/015-no-rollback-after-balance/test.sh
index 2f6407f3a5f7..155887879aa2 100755
--- a/tests/convert-tests/015-no-rollback-after-balance/test.sh
+++ b/tests/convert-tests/015-no-rollback-after-balance/test.sh
@@ -8,6 +8,7 @@ source "$TEST_TOP/common.convert"
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq mke2fs
 
 # convert_test_prep_fs() will create large enough file inside the test device,
diff --git a/tests/convert-tests/016-invalid-large-inline-extent/test.sh b/tests/convert-tests/016-invalid-large-inline-extent/test.sh
index f37c7c09d2e7..0308dcea13b3 100755
--- a/tests/convert-tests/016-invalid-large-inline-extent/test.sh
+++ b/tests/convert-tests/016-invalid-large-inline-extent/test.sh
@@ -8,6 +8,7 @@ source "$TEST_TOP/common.convert"
 setup_root_helper
 prepare_test_dev
 check_prereq btrfs-convert
+check_prereq_mount_with_sectorsize 4096
 check_global_prereq mke2fs
 
 convert_test_prep_fs ext4 mke2fs -t ext4 -b 4096
-- 
2.22.0

Re: [PATCH 1/5] btrfs-progs: mkfs: Apply the sectorsize user specified on 64k page size system

[Nikolay Borisov]

On 5.07.19 г. 10:26 ч., Qu Wenruo wrote:
> [BUG]
> On aarch64 with 64k page size, mkfs.btrfs -s option doesn't work:
>   $ mkfs.btrfs  -s 4096 ~/10G.img  -f
>   btrfs-progs v5.1.1
>   See http://btrfs.wiki.kernel.org for more information.
> 
>   Label:              (null)
>   UUID:               c2a09334-aaca-4980-aefa-4b3e27390658
>   Node size:          65536
>   Sector size:        65536		<< Still 64K, not 4K
>   Filesystem size:    10.00GiB
>   Block group profiles:
>     Data:             single            8.00MiB
>     Metadata:         DUP             256.00MiB
>     System:           DUP               8.00MiB
>   SSD detected:       no
>   Incompat features:  extref, skinny-metadata
>   Number of devices:  1
>   Devices:
>      ID        SIZE  PATH
>       1    10.00GiB  /home/adam/10G.img
> 
> [CAUSE]
> This is because we automatically detect sectorsize based on current
> system page size, then get the maxium number between user specified -s
> parameter and system page size.
> 
> It's fine for x86 as it has fixed page size 4K, also the minimium valid
> sector size.
> 
> But for system like aarch64 or ppc64le, where we can have 64K page size,
> and it makes us unable to create a 4k sector sized btrfs.
> 
> [FIX]
> Only do auto detect when no -s|--sectorsize option is specified.
> 
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>  mkfs/main.c | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/mkfs/main.c b/mkfs/main.c
> index 8dbec0717b89..26d84e9dafc3 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -817,6 +817,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>  	char *source_dir = NULL;
>  	bool source_dir_set = false;
>  	bool shrink_rootdir = false;
> +	bool sectorsize_set = false;
>  	u64 source_dir_size = 0;
>  	u64 min_dev_size;
>  	u64 shrink_size;
> @@ -906,6 +907,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>  				}
>  			case 's':
>  				sectorsize = parse_size(optarg);
> +				sectorsize_set = true;
>  				break;
>  			case 'b':
>  				block_count = parse_size(optarg);
> @@ -943,7 +945,15 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>  		printf("See %s for more information.\n\n", PACKAGE_URL);
>  	}
>  
> -	sectorsize = max(sectorsize, (u32)sysconf(_SC_PAGESIZE));
> +	if (!sectorsize_set)
> +		sectorsize = max(sectorsize, (u32)sysconf(_SC_PAGESIZE));

This means it's possible for the user to create a filesystem that is not
mountable on his current machine, due to the presence of the following
check in validate_super:

if (sectorsize != PAGE_SIZE) {
btrfs_err(..)
}

Perhaps the risk is not that big since if someone creates such a
filesystem they will almost instantly realize it won't work and
re-create it properly.



> +	if (!is_power_of_2(sectorsize) || sectorsize < 4096 ||
> +	    sectorsize > SZ_64K) {

nit: Perhaps this check should be modified so that it follows the kernel
style :
if (!is_power_of_2(sectorsize) || sectorsize < 4096 ||
              sectorsize > BTRFS_MAX_METADATA_BLOCKSIZE) {

MAX_METADATA_BLOCKSIZE is defined as 64k but using the same defines
seems more clear to me.


> +		error(
> +		"invalid sectorsize: %u, expect either 4k, 8k, 16k, 32k or 64k",
> +			sectorsize);
> +		goto error;
> +	}
>  	stripesize = sectorsize;
>  	saved_optind = optind;
>  	dev_cnt = argc - optind;
> 

Re: [PATCH 1/5] btrfs-progs: mkfs: Apply the sectorsize user specified on 64k page size system

[Qu Wenruo]

On 2019/7/5 下午3:45, Nikolay Borisov wrote:
>
>
> On 5.07.19 г. 10:26 ч., Qu Wenruo wrote:
>> [BUG]
>> On aarch64 with 64k page size, mkfs.btrfs -s option doesn't work:
>>   $ mkfs.btrfs  -s 4096 ~/10G.img  -f
>>   btrfs-progs v5.1.1
>>   See http://btrfs.wiki.kernel.org for more information.
>>
>>   Label:              (null)
>>   UUID:               c2a09334-aaca-4980-aefa-4b3e27390658
>>   Node size:          65536
>>   Sector size:        65536		<< Still 64K, not 4K
>>   Filesystem size:    10.00GiB
>>   Block group profiles:
>>     Data:             single            8.00MiB
>>     Metadata:         DUP             256.00MiB
>>     System:           DUP               8.00MiB
>>   SSD detected:       no
>>   Incompat features:  extref, skinny-metadata
>>   Number of devices:  1
>>   Devices:
>>      ID        SIZE  PATH
>>       1    10.00GiB  /home/adam/10G.img
>>
>> [CAUSE]
>> This is because we automatically detect sectorsize based on current
>> system page size, then get the maxium number between user specified -s
>> parameter and system page size.
>>
>> It's fine for x86 as it has fixed page size 4K, also the minimium valid
>> sector size.
>>
>> But for system like aarch64 or ppc64le, where we can have 64K page size,
>> and it makes us unable to create a 4k sector sized btrfs.
>>
>> [FIX]
>> Only do auto detect when no -s|--sectorsize option is specified.
>>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>> ---
>>  mkfs/main.c | 12 +++++++++++-
>>  1 file changed, 11 insertions(+), 1 deletion(-)
>>
>> diff --git a/mkfs/main.c b/mkfs/main.c
>> index 8dbec0717b89..26d84e9dafc3 100644
>> --- a/mkfs/main.c
>> +++ b/mkfs/main.c
>> @@ -817,6 +817,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>>  	char *source_dir = NULL;
>>  	bool source_dir_set = false;
>>  	bool shrink_rootdir = false;
>> +	bool sectorsize_set = false;
>>  	u64 source_dir_size = 0;
>>  	u64 min_dev_size;
>>  	u64 shrink_size;
>> @@ -906,6 +907,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>>  				}
>>  			case 's':
>>  				sectorsize = parse_size(optarg);
>> +				sectorsize_set = true;
>>  				break;
>>  			case 'b':
>>  				block_count = parse_size(optarg);
>> @@ -943,7 +945,15 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>>  		printf("See %s for more information.\n\n", PACKAGE_URL);
>>  	}
>>
>> -	sectorsize = max(sectorsize, (u32)sysconf(_SC_PAGESIZE));
>> +	if (!sectorsize_set)
>> +		sectorsize = max(sectorsize, (u32)sysconf(_SC_PAGESIZE));
>
> This means it's possible for the user to create a filesystem that is not
> mountable on his current machine, due to the presence of the following
> check in validate_super:

There is no problem for it as we can also do that on x86:
mkfs.btrfs -f -s 64k -n 64k

>
> if (sectorsize != PAGE_SIZE) {
> btrfs_err(..)
> }
>
> Perhaps the risk is not that big since if someone creates such a
> filesystem they will almost instantly realize it won't work and
> re-create it properly.

And I'd argue any user of -s option should be considered as experienced
user.

>
>
>
>> +	if (!is_power_of_2(sectorsize) || sectorsize < 4096 ||
>> +	    sectorsize > SZ_64K) {
>
> nit: Perhaps this check should be modified so that it follows the kernel
> style :
> if (!is_power_of_2(sectorsize) || sectorsize < 4096 ||
>               sectorsize > BTRFS_MAX_METADATA_BLOCKSIZE) {
>
> MAX_METADATA_BLOCKSIZE is defined as 64k but using the same defines
> seems more clear to me.

That's correct and looks cleaner.

I'll update this patch.

Thanks,
Qu

>
>
>> +		error(
>> +		"invalid sectorsize: %u, expect either 4k, 8k, 16k, 32k or 64k",
>> +			sectorsize);
>> +		goto error;
>> +	}
>>  	stripesize = sectorsize;
>>  	saved_optind = optind;
>>  	dev_cnt = argc - optind;
>>

Re: [PATCH 0/5] btrfs-progs: tests: Make 64K page size system happier

[David Sterba]

On Fri, Jul 05, 2019 at 03:26:46PM +0800, Qu Wenruo wrote:
> Since I got another rockpro64, finally I could do some tests with
> aarch64 64K page size mode. (The first board is working as a NAS for
> a while)
> 
> Unsurprisingly there are several false test alerts in btrfs-progs
> selftests.
> 
> Although there is no existing CI service based on 64K page sized system,
> we'd better support for 64K page size as it's easier and easier to get
> SBC with good enough aarch64 SoC to compile kernel/btrfs-progs and run
> various tests on them.
> 
> The first patch fix a bug which mkfs can't accept any sector size on 64K
> page size system.
> 
> The remaining patches enhance test cases to make them work on 64K page
> size system (skip those tests unless kernel support subpage sized sector
> size)
> 
> Qu Wenruo (5):
>   btrfs-progs: mkfs: Apply the sectorsize user specified on 64k page
>     size system
>   btrfs-progs: fsck-tests: Check if current kernel can mount fs with
>     specified sector size
>   btrfs-progs: mkfs-tests: Skip 010-minimal-size if we can't mount with
>     4k sector size
>   btrfs-progs: misc-tests: Make test cases work or skipped on 64K page
>     size system
>   btrfs-progs: convert-tests: Skip tests if kernel doesn't support
>     subpage sized sector size

The fix is ok, but the test updates and pre-checks do not seem right to
me. The check_preerq helpers are for binaries and rather simple checks
unlike what the mkfs/mount test for 4k sectors does. I'll reply under
the patches with more specific comments.

Re: [PATCH 2/5] btrfs-progs: fsck-tests: Check if current kernel can mount fs with specified sector size

[David Sterba]

On Fri, Jul 05, 2019 at 03:26:48PM +0800, Qu Wenruo wrote:
> [BUG]
> When doing test on platforms with page size other than 4K (e.g aarch64
> can use 64K page size, or ppc64le), certain test wills fail like:
>       [TEST/fsck]   012-leaf-corruption
>   mount: /home/adam/btrfs-progs/tests/mnt: wrong fs type, bad option, bad superblock on /dev/loop4, missing codepage or helper program, or other error.
>   find: '/home/adam/btrfs-progs/tests//mnt/lost+found': No such file or directory
>   inode 1862 not recovered correctly
>   test failed for case 012-leaf-corruption
> 
>       [TEST/fsck]   028-unaligned-super-dev-sizes
>   failed: mount -t btrfs -o loop ./dev_and_super_mismatch_unaligned.raw.restored /home/adam/btrfs-progs/tests//mnt
>   test failed for case 028-unaligned-super-dev-sizes
> 
>       [TEST/fsck]   037-freespacetree-repair
>   failed: /home/adam/btrfs-progs/mkfs.btrfs -f -n 4k /home/adam/btrfs-progs/tests//test.img
>   test failed for case 037-freespacetree-repair
> 
> [CAUSE]
> 
> For fsck/012 and fsck/028, it's caused by the lack of subpage size sector
> size support, thus we require kernel page size to match on-disk sector size:
>   BTRFS error (device loop4): sectorsize 4096 not supported yet, only support 65536
>   BTRFS error (device loop4): superblock contains fatal errors
>   BTRFS error (device loop4): open_ctree failed
> 
> For fsck/037, it's mkfs causing the problem as we're using 4k nodesize,
> but on 64K page sized system, we will use 64K sectorsize and cause
> conflicts.
> 
> [FIX]
> Considering it's easier and easier to get aarch64 boards with enough
> performance (e.g rpi4, rk3399, S922) to compile kernel and run tests,
> let's skip such tests before widespread complain comes.
> 
> This patch will introduce a new check, check_prereq_mount_with_sectorsize(),
> which will test if kernel can mount btrfs with specified sectorsize.
> So that even one day we support subpage sized sectorsize, we won't need
> to update test case again.
> 
> For fsck/037, also specify sector size manually. And since in that case
> we still need to mount the fs, also add
> check_prereq_mount_with_sectorsize() call.
> 
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>  tests/common                                  | 29 +++++++++++++++++++
>  tests/fsck-tests/012-leaf-corruption/test.sh  |  1 +
>  .../028-unaligned-super-dev-sizes/test.sh     |  1 +
>  .../037-freespacetree-repair/test.sh          |  3 +-
>  4 files changed, 33 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/common b/tests/common
> index 79a16f1e187d..5ad16b69b61d 100644
> --- a/tests/common
> +++ b/tests/common
> @@ -306,6 +306,35 @@ check_prereq()
>  	fi
>  }
>  
> +# Require to mount images with speicified sectorsize
> +# This is to make sure we can run this test on different arch
> +# (e.g aarch64 with 64K pagesize)
> +check_prereq_mount_with_sectorsize()
> +{
> +	prepare_test_dev 128M
> +	check_prereq mkfs.btrfs
> +	setup_root_helper
> +
> +	local sectorsize=$1
> +	local loop_opt
> +
> +	if [[ -b "$TEST_DEV" ]]; then
> +		loop_opt=""
> +	elif [[ -f "$TEST_DEV" ]]; then
> +		loop_opt="-o loop"
> +	else
> +		_fail "Invalid \$TEST_DEV: $TEST_DEV"
> +	fi
> +
> +	run_check_mkfs_test_dev -f -s $sectorsize
> +	$SUDO_HELPER mount -t btrfs $loop_opt "$TEST_DEV" "$TEST_MNT" \
> +		&> /dev/null
> +	if [ $? -ne 0 ]; then
> +		_not_run "kernel doesn't support sectorsize $sectorsize"
> +	fi
> +	run_check_umount_test_dev
> +}

I think the reason why each requires the specific 4k size should be
documented there. This helper hides several things, like forcing 128M on
the default test device, mkfs and mount. Those are probably inevitable
to make sure the combination is supported but the helper should be
independent and not possibly interfere with anything the test could use.

The helper could be added as a separate patch, where the updated tests
should document the reasons for using it.

> +
>  check_global_prereq()
>  {
>  	which "$1" &> /dev/null
> diff --git a/tests/fsck-tests/012-leaf-corruption/test.sh b/tests/fsck-tests/012-leaf-corruption/test.sh
> index 68d9f695d4de..d5da1d210f28 100755
> --- a/tests/fsck-tests/012-leaf-corruption/test.sh
> +++ b/tests/fsck-tests/012-leaf-corruption/test.sh
> @@ -107,6 +107,7 @@ check_leaf_corrupt_no_data_ext()
>  
>  setup_root_helper
>  
> +check_prereq_mount_with_sectorsize 4096
>  generate_leaf_corrupt_no_data_ext test.img

So the tests with pre-generated images can't succeed so the test must be
skipped, unless we have images for all valid page size values.

>  check_image test.img
>  check_leaf_corrupt_no_data_ext test.img
> diff --git a/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh b/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh
> index 4015df2d8570..49fa35241d04 100755
> --- a/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh
> +++ b/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh
> @@ -6,6 +6,7 @@
>  source "$TEST_TOP/common"
>  
>  check_prereq btrfs
> +check_prereq_mount_with_sectorsize 4096
>  setup_root_helper
>  
>  check_all_images
> diff --git a/tests/fsck-tests/037-freespacetree-repair/test.sh b/tests/fsck-tests/037-freespacetree-repair/test.sh
> index 7f547a33512d..32e6651ac705 100755
> --- a/tests/fsck-tests/037-freespacetree-repair/test.sh
> +++ b/tests/fsck-tests/037-freespacetree-repair/test.sh
> @@ -10,6 +10,7 @@ prepare_test_dev 256M
>  
>  check_prereq btrfs
>  check_prereq mkfs.btrfs
> +check_prereq_mount_with_sectorsize 4096
>  check_global_prereq grep
>  check_global_prereq tail
>  check_global_prereq head
> @@ -55,7 +56,7 @@ if ! [ -f "/sys/fs/btrfs/features/free_space_tree" ]; then
>  	exit
>  fi
>  
> -run_check_mkfs_test_dev -n 4k
> +run_check_mkfs_test_dev -s 4k -n 4k

Can the test be updated so it always succeeds, ie. giving it a valid
sectorsize/nodesize based on getconf? Besides the hardcoded size values
(for the device and files), I don't see anything that would mandate 4k
sectorsize.

>  run_check_mount_test_dev -oclear_cache,space_cache=v2
>  
>  # create files which will populate the FST
> -- 
> 2.22.0

Re: [PATCH 3/5] btrfs-progs: mkfs-tests: Skip 010-minimal-size if we can't mount with 4k sector size

[David Sterba]

On Fri, Jul 05, 2019 at 03:26:49PM +0800, Qu Wenruo wrote:
> [BUG]
> Test case 010-minimal-size fails on aarch64 with 64K page size:
>       [TEST/mkfs]   010-minimal-size
>   failed: /home/adam/btrfs-progs/mkfs.btrfs -f -n 4k -m single -d single /home/adam/btrfs-progs/tests//test.img
>   test failed for case 010-minimal-size
>   make: *** [Makefile:361: test-mkfs] Error 1
> 
> [CAUSE]
> Mkfs.btrfs defaults to page size as sector size. However this test uses
> 4k, 16k, 32K and 64K as node size. 4K node size will conflict with 64K
> sector size.
> 
> [FIX]
> - Specify sector size 4K manually
>   So at least no conflict at mkfs time.
> 
> - Skip the test case if kernel can't mount with 4k sector size
>   So once we add such support, the test can be automatically re-enabled.
> 
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>  tests/mkfs-tests/010-minimal-size/test.sh | 41 ++++++++++++-----------
>  1 file changed, 21 insertions(+), 20 deletions(-)
> 
> diff --git a/tests/mkfs-tests/010-minimal-size/test.sh b/tests/mkfs-tests/010-minimal-size/test.sh
> index 8480e4c5ae23..b49fad63e519 100755
> --- a/tests/mkfs-tests/010-minimal-size/test.sh
> +++ b/tests/mkfs-tests/010-minimal-size/test.sh
> @@ -5,6 +5,7 @@ source "$TEST_TOP/common"
>  
>  check_prereq mkfs.btrfs
>  check_prereq btrfs
> +check_prereq_mount_with_sectorsize 4096

How about

	if `getconf PAGE_SIZE` != 4096; then
		_not_run "requires 4k pages"
	fi

and be done?

I don't like hardcoding the sectorsize, the effect should be the same
when the test is skipped at the beginning.

When we have the subpage-size functionality, there will be an incompat
bit exported through sysfs that we can check. Until then the cude
pagesize check should do. It is going to be a big change so we'll have
to revisit all tests anyway.

Re: [PATCH 5/5] btrfs-progs: convert-tests: Skip tests if kernel doesn't support subpage sized sector size

[David Sterba]

On Fri, Jul 05, 2019 at 03:26:51PM +0800, Qu Wenruo wrote:
> Most convert tests needs to mount the converted image, and both reiserfs
> and ext* uses 4k block size, on 32K page size system we can't mount them
> and will cause test failure.
> 
> Skip most of convert tests, except 007-unsupported-block-sizes, which
> should fail on all systems.

Ok agreed, I don't see a better way than to skip the tests. Some of them
can be made to work based on the page size and adjusting the
ext4/reiserfs block sizes.

As this requires hw support to verify that test works, I would rather do
that one by one at the expense that non-4k page testing coverage will be
missing.

I saw some tests were using the assumptions of 4k sectorsize for some
test file generation so to avoid silent breakage, reviewing change to
each test should give some guarantees.

Alternatively, tests can be extended to iterate over the block sizes
from 4k to 64k. But then it's again difficult to see which combinations
make sense and must succeed or not. Oh well.

> --- a/tests/convert-tests/006-large-hole-extent/test.sh
> +++ b/tests/convert-tests/006-large-hole-extent/test.sh
> @@ -11,6 +11,8 @@ source "$TEST_TOP/common.convert"
>  setup_root_helper
>  prepare_test_dev
>  check_prereq btrfs-convert
> +check_prereq_mount_with_sectorsize 4096
> +prepare_test_dev

See, this is an example why the pre-checks should be independent,
requiring to call prepare_test_dev again is quite counter-intuitive and
error prone.

>  check_global_prereq mke2fs
>  
>  default_mke2fs="mke2fs -t ext4 -b 4096"

Re: [PATCH 2/5] btrfs-progs: fsck-tests: Check if current kernel can mount fs with specified sector size

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 7060 bytes --]



On 2019/7/23 上午1:07, David Sterba wrote:
> On Fri, Jul 05, 2019 at 03:26:48PM +0800, Qu Wenruo wrote:
>> [BUG]
>> When doing test on platforms with page size other than 4K (e.g aarch64
>> can use 64K page size, or ppc64le), certain test wills fail like:
>>       [TEST/fsck]   012-leaf-corruption
>>   mount: /home/adam/btrfs-progs/tests/mnt: wrong fs type, bad option, bad superblock on /dev/loop4, missing codepage or helper program, or other error.
>>   find: '/home/adam/btrfs-progs/tests//mnt/lost+found': No such file or directory
>>   inode 1862 not recovered correctly
>>   test failed for case 012-leaf-corruption
>>
>>       [TEST/fsck]   028-unaligned-super-dev-sizes
>>   failed: mount -t btrfs -o loop ./dev_and_super_mismatch_unaligned.raw.restored /home/adam/btrfs-progs/tests//mnt
>>   test failed for case 028-unaligned-super-dev-sizes
>>
>>       [TEST/fsck]   037-freespacetree-repair
>>   failed: /home/adam/btrfs-progs/mkfs.btrfs -f -n 4k /home/adam/btrfs-progs/tests//test.img
>>   test failed for case 037-freespacetree-repair
>>
>> [CAUSE]
>>
>> For fsck/012 and fsck/028, it's caused by the lack of subpage size sector
>> size support, thus we require kernel page size to match on-disk sector size:
>>   BTRFS error (device loop4): sectorsize 4096 not supported yet, only support 65536
>>   BTRFS error (device loop4): superblock contains fatal errors
>>   BTRFS error (device loop4): open_ctree failed
>>
>> For fsck/037, it's mkfs causing the problem as we're using 4k nodesize,
>> but on 64K page sized system, we will use 64K sectorsize and cause
>> conflicts.
>>
>> [FIX]
>> Considering it's easier and easier to get aarch64 boards with enough
>> performance (e.g rpi4, rk3399, S922) to compile kernel and run tests,
>> let's skip such tests before widespread complain comes.
>>
>> This patch will introduce a new check, check_prereq_mount_with_sectorsize(),
>> which will test if kernel can mount btrfs with specified sectorsize.
>> So that even one day we support subpage sized sectorsize, we won't need
>> to update test case again.
>>
>> For fsck/037, also specify sector size manually. And since in that case
>> we still need to mount the fs, also add
>> check_prereq_mount_with_sectorsize() call.
>>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>> ---
>>  tests/common                                  | 29 +++++++++++++++++++
>>  tests/fsck-tests/012-leaf-corruption/test.sh  |  1 +
>>  .../028-unaligned-super-dev-sizes/test.sh     |  1 +
>>  .../037-freespacetree-repair/test.sh          |  3 +-
>>  4 files changed, 33 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/common b/tests/common
>> index 79a16f1e187d..5ad16b69b61d 100644
>> --- a/tests/common
>> +++ b/tests/common
>> @@ -306,6 +306,35 @@ check_prereq()
>>  	fi
>>  }
>>  
>> +# Require to mount images with speicified sectorsize
>> +# This is to make sure we can run this test on different arch
>> +# (e.g aarch64 with 64K pagesize)
>> +check_prereq_mount_with_sectorsize()
>> +{
>> +	prepare_test_dev 128M
>> +	check_prereq mkfs.btrfs
>> +	setup_root_helper
>> +
>> +	local sectorsize=$1
>> +	local loop_opt
>> +
>> +	if [[ -b "$TEST_DEV" ]]; then
>> +		loop_opt=""
>> +	elif [[ -f "$TEST_DEV" ]]; then
>> +		loop_opt="-o loop"
>> +	else
>> +		_fail "Invalid \$TEST_DEV: $TEST_DEV"
>> +	fi
>> +
>> +	run_check_mkfs_test_dev -f -s $sectorsize
>> +	$SUDO_HELPER mount -t btrfs $loop_opt "$TEST_DEV" "$TEST_MNT" \
>> +		&> /dev/null
>> +	if [ $? -ne 0 ]; then
>> +		_not_run "kernel doesn't support sectorsize $sectorsize"
>> +	fi
>> +	run_check_umount_test_dev
>> +}
> 
> I think the reason why each requires the specific 4k size should be
> documented there.

I'll move the comment to each test calling this check.

> This helper hides several things, like forcing 128M on
> the default test device, mkfs and mount. Those are probably inevitable
> to make sure the combination is supported but the helper should be
> independent and not possibly interfere with anything the test could use.

The only thing may affect test cases is the "prepare_test_dev 128M" call.
It can be easily avoided by calling this check before real
prepare_test_dev call.

I can update the test cases to co-operate this, and add extra comment
for that.

> 
> The helper could be added as a separate patch, where the updated tests
> should document the reasons for using it.

No problem, and just in case I will also add comment in each test
calling this check to explain why.

> 
>> +
>>  check_global_prereq()
>>  {
>>  	which "$1" &> /dev/null
>> diff --git a/tests/fsck-tests/012-leaf-corruption/test.sh b/tests/fsck-tests/012-leaf-corruption/test.sh
>> index 68d9f695d4de..d5da1d210f28 100755
>> --- a/tests/fsck-tests/012-leaf-corruption/test.sh
>> +++ b/tests/fsck-tests/012-leaf-corruption/test.sh
>> @@ -107,6 +107,7 @@ check_leaf_corrupt_no_data_ext()
>>  
>>  setup_root_helper
>>  
>> +check_prereq_mount_with_sectorsize 4096
>>  generate_leaf_corrupt_no_data_ext test.img
> 
> So the tests with pre-generated images can't succeed so the test must be
> skipped, unless we have images for all valid page size values.

Or after we can mount 4K sector size fs on 64K page size system.

> 
>>  check_image test.img
>>  check_leaf_corrupt_no_data_ext test.img
>> diff --git a/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh b/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh
>> index 4015df2d8570..49fa35241d04 100755
>> --- a/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh
>> +++ b/tests/fsck-tests/028-unaligned-super-dev-sizes/test.sh
>> @@ -6,6 +6,7 @@
>>  source "$TEST_TOP/common"
>>  
>>  check_prereq btrfs
>> +check_prereq_mount_with_sectorsize 4096
>>  setup_root_helper
>>  
>>  check_all_images
>> diff --git a/tests/fsck-tests/037-freespacetree-repair/test.sh b/tests/fsck-tests/037-freespacetree-repair/test.sh
>> index 7f547a33512d..32e6651ac705 100755
>> --- a/tests/fsck-tests/037-freespacetree-repair/test.sh
>> +++ b/tests/fsck-tests/037-freespacetree-repair/test.sh
>> @@ -10,6 +10,7 @@ prepare_test_dev 256M
>>  
>>  check_prereq btrfs
>>  check_prereq mkfs.btrfs
>> +check_prereq_mount_with_sectorsize 4096
>>  check_global_prereq grep
>>  check_global_prereq tail
>>  check_global_prereq head
>> @@ -55,7 +56,7 @@ if ! [ -f "/sys/fs/btrfs/features/free_space_tree" ]; then
>>  	exit
>>  fi
>>  
>> -run_check_mkfs_test_dev -n 4k
>> +run_check_mkfs_test_dev -s 4k -n 4k
> 
> Can the test be updated so it always succeeds, ie. giving it a valid
> sectorsize/nodesize based on getconf? Besides the hardcoded size values
> (for the device and files), I don't see anything that would mandate 4k
> sectorsize.

Sure, I could try to test all valid node size combination based on
system page size.

Thanks,
Qu

> 
>>  run_check_mount_test_dev -oclear_cache,space_cache=v2
>>  
>>  # create files which will populate the FST
>> -- 
>> 2.22.0


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 3/5] btrfs-progs: mkfs-tests: Skip 010-minimal-size if we can't mount with 4k sector size

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2484 bytes --]



On 2019/7/23 上午1:15, David Sterba wrote:
> On Fri, Jul 05, 2019 at 03:26:49PM +0800, Qu Wenruo wrote:
>> [BUG]
>> Test case 010-minimal-size fails on aarch64 with 64K page size:
>>       [TEST/mkfs]   010-minimal-size
>>   failed: /home/adam/btrfs-progs/mkfs.btrfs -f -n 4k -m single -d single /home/adam/btrfs-progs/tests//test.img
>>   test failed for case 010-minimal-size
>>   make: *** [Makefile:361: test-mkfs] Error 1
>>
>> [CAUSE]
>> Mkfs.btrfs defaults to page size as sector size. However this test uses
>> 4k, 16k, 32K and 64K as node size. 4K node size will conflict with 64K
>> sector size.
>>
>> [FIX]
>> - Specify sector size 4K manually
>>   So at least no conflict at mkfs time.
>>
>> - Skip the test case if kernel can't mount with 4k sector size
>>   So once we add such support, the test can be automatically re-enabled.
>>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>> ---
>>  tests/mkfs-tests/010-minimal-size/test.sh | 41 ++++++++++++-----------
>>  1 file changed, 21 insertions(+), 20 deletions(-)
>>
>> diff --git a/tests/mkfs-tests/010-minimal-size/test.sh b/tests/mkfs-tests/010-minimal-size/test.sh
>> index 8480e4c5ae23..b49fad63e519 100755
>> --- a/tests/mkfs-tests/010-minimal-size/test.sh
>> +++ b/tests/mkfs-tests/010-minimal-size/test.sh
>> @@ -5,6 +5,7 @@ source "$TEST_TOP/common"
>>  
>>  check_prereq mkfs.btrfs
>>  check_prereq btrfs
>> +check_prereq_mount_with_sectorsize 4096
> 
> How about
> 
> 	if `getconf PAGE_SIZE` != 4096; then
> 		_not_run "requires 4k pages"
> 	fi
> 
> and be done?

I'd like to make these tests future proof.

Maybe the subpage size support from IBM hasn't been updated for a long
time, but we should have such support eventually.

When that support is merged, with my current check no modification needs
to be done and every test can be run on 64k page size systems.

> 
> I don't like hardcoding the sectorsize, the effect should be the same
> when the test is skipped at the beginning.

So is the patch doing, just skipping the test on 64K system.
I didn't see any difference here except how we skip the tests.

> 
> When we have the subpage-size functionality, there will be an incompat
> bit exported through sysfs that we can check.

Why, there is no change in on-disk format, just a runtime feature.

> Until then the cude
> pagesize check should do. It is going to be a big change so we'll have
> to revisit all tests anyway.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]