[PATCH RFC 0/2] fsx: make fsx perceptive to cluster size

[PATCH RFC 0/2] fsx: make fsx perceptive to cluster size

[Jeffle Xu]

Offset and size should be aligned with cluster_size when inserting or
collapsing range on ext4 with 'bigalloc' enabled. Currently fsx only
align offset/size with block size.

In fact I have no idea which is the best way to fix this isue. On one hand,
fsx should be general and has no knowledge of the underlying filesystem.
Besides the cluster size seems to be stored on ext4_super_block and there's
no easy way to get it. But on the oter hand, quite many tests call fsx
directly, e.g., generic/127, rather than the generic routine run_fsx()
defined in common/rc.


Jeffle Xu (2):
  xfstests: fsx: add support for cluster size
  xfstests: common/rc: add cluster size support for ext4

 common/rc |  9 +++++++++
 ltp/fsx.c | 20 ++++++++++++++------
 2 files changed, 23 insertions(+), 6 deletions(-)

-- 
1.8.3.1

[PATCH RFC 1/2] xfstests: fsx: add support for cluster size

[Jeffle Xu]

The offset and size should be aligned with cluster size when inserting
or collapsing range on ext4 with 'bigalloc' feature enabled. Currently
I can find only ext4 with this limitation.

Since fsx should have no assumption of the underlying filesystem, and
thus add the '-u cluster_size' option. Tests can set this option when
the underlying filesystem is ext4 with bigalloc enabled.

Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com>
---
 ltp/fsx.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/ltp/fsx.c b/ltp/fsx.c
index 9d598a4..5fe5738 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -133,6 +133,7 @@ int	dirpath = 0;			/* -P flag */
 int	fd;				/* fd for our test file */
 
 blksize_t	block_size = 0;
+blksize_t	cluster_size = 0;
 off_t		file_size = 0;
 off_t		biggest = 0;
 long long	testcalls = 0;		/* calls to function "test" */
@@ -2146,8 +2147,8 @@ have_op:
 		break;
 	case OP_COLLAPSE_RANGE:
 		TRIM_OFF_LEN(offset, size, file_size - 1);
-		offset = offset & ~(block_size - 1);
-		size = size & ~(block_size - 1);
+		offset = offset & ~(cluster_size - 1);
+		size = size & ~(cluster_size - 1);
 		if (size == 0) {
 			log4(OP_COLLAPSE_RANGE, offset, size, FL_SKIPPED);
 			goto out;
@@ -2157,8 +2158,8 @@ have_op:
 	case OP_INSERT_RANGE:
 		TRIM_OFF(offset, file_size);
 		TRIM_LEN(file_size, size, maxfilelen);
-		offset = offset & ~(block_size - 1);
-		size = size & ~(block_size - 1);
+		offset = offset & ~(cluster_size - 1);
+		size = size & ~(cluster_size - 1);
 		if (size == 0) {
 			log4(OP_INSERT_RANGE, offset, size, FL_SKIPPED);
 			goto out;
@@ -2231,7 +2232,7 @@ void
 usage(void)
 {
 	fprintf(stdout, "usage: %s",
-		"fsx [-dknqxABEFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+		"fsx [-dknqxABEFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-u csize] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
 	-b opnum: beginning operation number (default 1)\n\
 	-c P: 1 in P chance of file close+open at each op (default infinity)\n\
 	-d: debug output for all operations\n\
@@ -2249,6 +2250,7 @@ usage(void)
 	-r readbdy: 4096 would make reads page aligned (default 1)\n\
 	-s style: 1 gives smaller truncates (default 0)\n\
 	-t truncbdy: 4096 would make truncates page aligned (default 1)\n\
+	-u csize: filesystem specific cluster size that may be used for ops like insert/collapse range\n\
 	-w writebdy: 4096 would make writes page aligned (default 1)\n\
 	-x: preallocate file space before starting, XFS only (default 0)\n\
 	-y synchronize changes to a file\n"
@@ -2485,7 +2487,7 @@ main(int argc, char **argv)
 	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
 
 	while ((ch = getopt_long(argc, argv,
-				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:WXZ",
+				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:u:w:xyABD:EFJKHzCILN:OP:RS:WXZ",
 				 longopts, NULL)) != EOF)
 		switch (ch) {
 		case 'b':
@@ -2579,6 +2581,11 @@ main(int argc, char **argv)
 			if (truncbdy <= 0)
 				usage();
 			break;
+		case 'u':
+			cluster_size = getnum(optarg, &endp);
+			if (cluster_size <= 0)
+				usage();
+			break;
 		case 'w':
 			writebdy = getnum(optarg, &endp);
 			if (writebdy <= 0)
@@ -2720,6 +2727,7 @@ main(int argc, char **argv)
 		exit(91);
 	}
 	block_size = statbuf.st_blksize;
+	cluster_size = cluster_size ? : block_size;
 #ifdef XFS
 	if (prealloc) {
 		xfs_flock64_t	resv = { 0 };
-- 
1.8.3.1

[PATCH RFC 2/2] xfstests: common/rc: add cluster size support for ext4

[Jeffle Xu]

Inserting and collapsing range on ext4 with 'bigalloc' feature will
fail due to the offset and size should be alligned with the cluster
size.

The previous patch has add support for cluster size in fsx. Detect and
pass the cluster size parameter to fsx if the underlying filesystem
is ext4 with bigalloc.

Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com>
---
 common/rc | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/common/rc b/common/rc
index 2000bd9..71dde5f 100644
--- a/common/rc
+++ b/common/rc
@@ -3908,6 +3908,15 @@ run_fsx()
 {
 	echo fsx $@
 	local args=`echo $@ | sed -e "s/ BSIZE / $bsize /g" -e "s/ PSIZE / $psize /g"`
+
+	if [ "$FSTYP" == "ext4" ]; then
+		local cluster_size=$(tune2fs -l $TEST_DEV | grep 'Cluster size' | awk '{print $3}')
+		if [ -n $cluster_size ]; then
+			echo "cluster size: $cluster_size"
+			args="$args -u $cluster_size"
+		fi
+	fi
+
 	set -- $here/ltp/fsx $args $FSX_AVOID $TEST_DIR/junk
 	echo "$@" >>$seqres.full
 	rm -f $TEST_DIR/junk
-- 
1.8.3.1

Re: [PATCH RFC 1/2] xfstests: fsx: add support for cluster size

[Darrick J. Wong]

On Fri, Apr 24, 2020 at 05:33:49PM +0800, Jeffle Xu wrote:
> The offset and size should be aligned with cluster size when inserting
> or collapsing range on ext4 with 'bigalloc' feature enabled. Currently
> I can find only ext4 with this limitation.

ocfs2 also has this magic, um, ability.

As does xfs under certain circumstance (realtime volumes).

> Since fsx should have no assumption of the underlying filesystem, and
> thus add the '-u cluster_size' option. Tests can set this option when
> the underlying filesystem is ext4 with bigalloc enabled.

Do copyrange, clonerange, or deduperange have this problem? ;)

> Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com>
> ---
>  ltp/fsx.c | 20 ++++++++++++++------
>  1 file changed, 14 insertions(+), 6 deletions(-)
> 
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index 9d598a4..5fe5738 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -133,6 +133,7 @@ int	dirpath = 0;			/* -P flag */
>  int	fd;				/* fd for our test file */
>  
>  blksize_t	block_size = 0;
> +blksize_t	cluster_size = 0;
>  off_t		file_size = 0;
>  off_t		biggest = 0;
>  long long	testcalls = 0;		/* calls to function "test" */
> @@ -2146,8 +2147,8 @@ have_op:
>  		break;
>  	case OP_COLLAPSE_RANGE:
>  		TRIM_OFF_LEN(offset, size, file_size - 1);
> -		offset = offset & ~(block_size - 1);
> -		size = size & ~(block_size - 1);
> +		offset = offset & ~(cluster_size - 1);
> +		size = size & ~(cluster_size - 1);
>  		if (size == 0) {
>  			log4(OP_COLLAPSE_RANGE, offset, size, FL_SKIPPED);
>  			goto out;
> @@ -2157,8 +2158,8 @@ have_op:
>  	case OP_INSERT_RANGE:
>  		TRIM_OFF(offset, file_size);
>  		TRIM_LEN(file_size, size, maxfilelen);
> -		offset = offset & ~(block_size - 1);
> -		size = size & ~(block_size - 1);
> +		offset = offset & ~(cluster_size - 1);
> +		size = size & ~(cluster_size - 1);
>  		if (size == 0) {
>  			log4(OP_INSERT_RANGE, offset, size, FL_SKIPPED);
>  			goto out;
> @@ -2231,7 +2232,7 @@ void
>  usage(void)
>  {
>  	fprintf(stdout, "usage: %s",
> -		"fsx [-dknqxABEFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
> +		"fsx [-dknqxABEFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-u csize] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
>  	-b opnum: beginning operation number (default 1)\n\
>  	-c P: 1 in P chance of file close+open at each op (default infinity)\n\
>  	-d: debug output for all operations\n\
> @@ -2249,6 +2250,7 @@ usage(void)
>  	-r readbdy: 4096 would make reads page aligned (default 1)\n\
>  	-s style: 1 gives smaller truncates (default 0)\n\
>  	-t truncbdy: 4096 would make truncates page aligned (default 1)\n\
> +	-u csize: filesystem specific cluster size that may be used for ops like insert/collapse range\n\
>  	-w writebdy: 4096 would make writes page aligned (default 1)\n\
>  	-x: preallocate file space before starting, XFS only (default 0)\n\
>  	-y synchronize changes to a file\n"
> @@ -2485,7 +2487,7 @@ main(int argc, char **argv)
>  	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
>  
>  	while ((ch = getopt_long(argc, argv,
> -				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:WXZ",
> +				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:u:w:xyABD:EFJKHzCILN:OP:RS:WXZ",
>  				 longopts, NULL)) != EOF)
>  		switch (ch) {
>  		case 'b':
> @@ -2579,6 +2581,11 @@ main(int argc, char **argv)
>  			if (truncbdy <= 0)
>  				usage();
>  			break;
> +		case 'u':
> +			cluster_size = getnum(optarg, &endp);
> +			if (cluster_size <= 0)
> +				usage();
> +			break;
>  		case 'w':
>  			writebdy = getnum(optarg, &endp);
>  			if (writebdy <= 0)
> @@ -2720,6 +2727,7 @@ main(int argc, char **argv)
>  		exit(91);
>  	}
>  	block_size = statbuf.st_blksize;
> +	cluster_size = cluster_size ? : block_size;
>  #ifdef XFS
>  	if (prealloc) {
>  		xfs_flock64_t	resv = { 0 };
> -- 
> 1.8.3.1
> 

Re: [PATCH RFC 2/2] xfstests: common/rc: add cluster size support for ext4

[Darrick J. Wong]

On Fri, Apr 24, 2020 at 05:33:50PM +0800, Jeffle Xu wrote:
> Inserting and collapsing range on ext4 with 'bigalloc' feature will
> fail due to the offset and size should be alligned with the cluster
> size.
> 
> The previous patch has add support for cluster size in fsx. Detect and
> pass the cluster size parameter to fsx if the underlying filesystem
> is ext4 with bigalloc.
> 
> Signed-off-by: Jeffle Xu <jefflexu@linux.alibaba.com>
> ---
>  common/rc | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/common/rc b/common/rc
> index 2000bd9..71dde5f 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -3908,6 +3908,15 @@ run_fsx()
>  {
>  	echo fsx $@
>  	local args=`echo $@ | sed -e "s/ BSIZE / $bsize /g" -e "s/ PSIZE / $psize /g"`
> +
> +	if [ "$FSTYP" == "ext4" ]; then
> +		local cluster_size=$(tune2fs -l $TEST_DEV | grep 'Cluster size' | awk '{print $3}')
> +		if [ -n $cluster_size ]; then
> +			echo "cluster size: $cluster_size"
> +			args="$args -u $cluster_size"
> +		fi
> +	fi

Computing the file allocation block size ought to be a separate helper.

I wonder if there's a standard way to report cluster sizes, seeing as
fat, ext4, ocfs2, and xfs can all have minimum space allocation units
that are larger than the base fs block size.

--D

> +
>  	set -- $here/ltp/fsx $args $FSX_AVOID $TEST_DIR/junk
>  	echo "$@" >>$seqres.full
>  	rm -f $TEST_DIR/junk
> -- 
> 1.8.3.1
> 

Re: [PATCH RFC 1/2] xfstests: fsx: add support for cluster size

[JeffleXu]

On 4/28/20 1:29 AM, Darrick J. Wong wrote:
> On Fri, Apr 24, 2020 at 05:33:49PM +0800, Jeffle Xu wrote:
>> The offset and size should be aligned with cluster size when inserting
>> or collapsing range on ext4 with 'bigalloc' feature enabled. Currently
>> I can find only ext4 with this limitation.
> ocfs2 also has this magic, um, ability.
>
> As does xfs under certain circumstance (realtime volumes).
>
>> Since fsx should have no assumption of the underlying filesystem, and
>> thus add the '-u cluster_size' option. Tests can set this option when
>> the underlying filesystem is ext4 with bigalloc enabled.
> Do copyrange, clonerange, or deduperange have this problem? ;)


clonerange and deduperange are not supported in ext4, while copyrange
  and zerorange work as the range has no need to be aligned with cluster size
in these two situations.