[PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress

[PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress

[Darrick J. Wong]

Hi all,

This series adds to fsx and fsstress support for FICLONERANGE,
FIDEDUPERANGE, and copy_file_range.

First, I fix some gcc warnings in fsx.

Then, I teach fsx to read the fsx file after every operation to compare
it to the good buffer.  This made it easier for me to find corruption
problem as soon as they happen, though I'm not sure it really makes
sense to have this enabled by default because of the behavior change
that it makes.

Next come a couple of generic reworks to fsx that we need to support the
new clone/dedupe/copy commands.

Patches 5-6 add clone and dedupe to fsx.

Patches 7-8 add copy_file_range support to fsstress and fsx.

Dave Chinner contributed some cleanups to the fsx patches as the 9th
patch.

The last patch fixes the common/dump tests to disable the new commands
so that the dump/restore tests continue to function exactly as they have
for years.

There are known failures in 4.20-rc3, particularly with copy_file_range,
which hopefully have been fixed by the patch series that Dave Chinner
posted to the xfs list yesterday.  Branch can be downloaded here[1].

--D

[1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fsstress-clone

[PATCH 01/10] fsx: shut up compiler warnings

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Fix unused variables and potential filename overflows in fsx.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index b0157ba3..5601c70c 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -121,7 +121,7 @@ char	*bname;				/* basename of our test file */
 char	*logdev;			/* -i flag */
 char	*logid;				/* -j flag */
 char	dname[1024];			/* -P flag */
-char	goodfile[1024];
+char	goodfile[PATH_MAX];
 int	dirpath = 0;			/* -P flag */
 int	fd;				/* fd for our test file */
 
@@ -185,7 +185,7 @@ const char *replayops = NULL;
 const char *recordops = NULL;
 FILE *	fsxlogf = NULL;
 FILE *	replayopsf = NULL;
-char opsfile[1024];
+char opsfile[PATH_MAX];
 int badoff = -1;
 int closeopen = 0;
 
@@ -541,13 +541,13 @@ mark_log(void)
 void
 dump_fsync_buffer(void)
 {
-	char fname_buffer[1024];
+	char fname_buffer[PATH_MAX];
 	int good_fd;
 
 	if (!good_buf)
 		return;
 
-	snprintf(fname_buffer, 1024, "%s%s.mark%d", dname,
+	snprintf(fname_buffer, sizeof(fname_buffer), "%s%s.mark%d", dname,
 		 bname, mark_nr);
 	good_fd = open(fname_buffer, O_WRONLY|O_CREAT|O_TRUNC, 0666);
 	if (good_fd < 0) {
@@ -1865,7 +1865,7 @@ main(int argc, char **argv)
 {
 	int	i, style, ch;
 	char	*endp, *tmp;
-	char logfile[1024];
+	char logfile[PATH_MAX];
 	struct stat statbuf;
 	int o_flags = O_RDWR|O_CREAT|O_TRUNC;
 

[PATCH 02/10] fsx: always check buffer after each operation

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add a new option to make fsx read the file after each operation and
compare it with the good buffer to try to catch corruptions as soon as
they occur.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |   65 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 56 insertions(+), 9 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index 5601c70c..4b3a5d5a 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -160,7 +160,8 @@ int     punch_hole_calls = 1;           /* -H flag disables */
 int     zero_range_calls = 1;           /* -z flag disables */
 int	collapse_range_calls = 1;	/* -C flag disables */
 int	insert_range_calls = 1;		/* -I flag disables */
-int 	mapped_reads = 1;		/* -R flag disables it */
+int	mapped_reads = 1;		/* -R flag disables it */
+int	check_file = 1;			/* -X flag disables */
 int	integrity = 0;			/* -i flag */
 int	fsxgoodfd = 0;
 int	o_direct;			/* -Z */
@@ -561,7 +562,7 @@ dump_fsync_buffer(void)
 }
 
 void
-check_buffers(unsigned offset, unsigned size)
+check_buffers(char *buf, unsigned offset, unsigned size)
 {
 	unsigned char c, t;
 	unsigned i = 0;
@@ -569,19 +570,19 @@ check_buffers(unsigned offset, unsigned size)
 	unsigned op = 0;
 	unsigned bad = 0;
 
-	if (memcmp(good_buf + offset, temp_buf, size) != 0) {
+	if (memcmp(good_buf + offset, buf, size) != 0) {
 		prt("READ BAD DATA: offset = 0x%x, size = 0x%x, fname = %s\n",
 		    offset, size, fname);
 		prt("OFFSET\tGOOD\tBAD\tRANGE\n");
 		while (size > 0) {
 			c = good_buf[offset];
-			t = temp_buf[i];
+			t = buf[i];
 			if (c != t) {
 			        if (n < 16) {
-					bad = short_at(&temp_buf[i]);
+					bad = short_at(&buf[i]);
 				        prt("0x%05x\t0x%04x\t0x%04x", offset,
 				            short_at(&good_buf[offset]), bad);
-					op = temp_buf[offset & 1 ? i+1 : i];
+					op = buf[offset & 1 ? i+1 : i];
 				        prt("\t0x%05x\n", n);
 					if (op)
 						prt("operation# (mod 256) for "
@@ -725,7 +726,46 @@ doread(unsigned offset, unsigned size)
 			    iret, size);
 		report_failure(141);
 	}
-	check_buffers(offset, size);
+	check_buffers(temp_buf, offset, size);
+}
+
+void
+check_contents(void)
+{
+	static char *check_buf;
+	unsigned offset = 0;
+	unsigned size = file_size;
+	off_t ret;
+	unsigned iret;
+
+	if (!check_buf) {
+		check_buf = (char *) malloc(maxfilelen + writebdy);
+		assert(check_buf != NULL);
+		check_buf = round_ptr_up(check_buf, writebdy, 0);
+		memset(check_buf, '\0', maxfilelen);
+	}
+
+	if (o_direct)
+		size -= size % readbdy;
+	if (size == 0)
+		return;
+
+	ret = lseek(fd, (off_t)offset, SEEK_SET);
+	if (ret == (off_t)-1) {
+		prterr("doread: lseek");
+		report_failure(140);
+	}
+
+	iret = fsxread(fd, check_buf, size, offset);
+	if (iret != size) {
+		if (iret == -1)
+			prterr("check_contents: read");
+		else
+			prt("short check read: 0x%x bytes instead of 0x%x\n",
+			    iret, size);
+		report_failure(141);
+	}
+	check_buffers(check_buf, offset, size);
 }
 
 
@@ -808,7 +848,7 @@ domapread(unsigned offset, unsigned size)
 		report_failure(191);
 	}
 
-	check_buffers(offset, size);
+	check_buffers(temp_buf, offset, size);
 }
 
 
@@ -1624,6 +1664,9 @@ test(void)
 		break;
 	}
 
+	if (check_file)
+		check_contents();
+
 out:
 	if (sizechecks && testcalls > simulatedopcount)
 		check_size();
@@ -1684,6 +1727,7 @@ usage(void)
 	-P: save .fsxlog .fsxops and .fsxgood files in dirpath (default ./)\n\
 	-S seed: for random # generator (default 1) 0 gets timestamp\n\
 	-W: mapped write operations DISabled\n\
+	-X: Do not read file and compare to good buffer after every operation.\n\
         -R: read() system calls only (mapped reads disabled)\n\
         -Z: O_DIRECT (use -R, -W, -r and -w too)\n\
 	--replay-ops opsfile: replay ops from recorded .fsxops file\n\
@@ -1880,7 +1924,7 @@ main(int argc, char **argv)
 	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
 
 	while ((ch = getopt_long(argc, argv,
-				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
+				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
 				 longopts, NULL)) != EOF)
 		switch (ch) {
 		case 'b':
@@ -2044,6 +2088,9 @@ main(int argc, char **argv)
 			if (!quiet)
 				prt("mapped writes DISABLED\n");
 			break;
+		case 'X':
+			check_file = 0;
+			break;
 		case 'Z':
 			o_direct = O_DIRECT;
 			o_flags |= O_DIRECT;

Re: [PATCH 02/10] fsx: always check buffer after each operation

[Darrick J. Wong]

On Tue, Nov 20, 2018 at 01:56:53PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Add a new option to make fsx read the file after each operation and
> compare it with the good buffer to try to catch corruptions as soon as
> they occur.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  ltp/fsx.c |   65 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
>  1 file changed, 56 insertions(+), 9 deletions(-)
> 
> 
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index 5601c70c..4b3a5d5a 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -160,7 +160,8 @@ int     punch_hole_calls = 1;           /* -H flag disables */
>  int     zero_range_calls = 1;           /* -z flag disables */
>  int	collapse_range_calls = 1;	/* -C flag disables */
>  int	insert_range_calls = 1;		/* -I flag disables */
> -int 	mapped_reads = 1;		/* -R flag disables it */
> +int	mapped_reads = 1;		/* -R flag disables it */
> +int	check_file = 1;			/* -X flag disables */

[from irc discussion with dave]

This probably ought to be inverted (i.e. 0 by default, -X sets it to 1)
so that we don't go messing with internal state.

--D

>  int	integrity = 0;			/* -i flag */
>  int	fsxgoodfd = 0;
>  int	o_direct;			/* -Z */
> @@ -561,7 +562,7 @@ dump_fsync_buffer(void)
>  }
>  
>  void
> -check_buffers(unsigned offset, unsigned size)
> +check_buffers(char *buf, unsigned offset, unsigned size)
>  {
>  	unsigned char c, t;
>  	unsigned i = 0;
> @@ -569,19 +570,19 @@ check_buffers(unsigned offset, unsigned size)
>  	unsigned op = 0;
>  	unsigned bad = 0;
>  
> -	if (memcmp(good_buf + offset, temp_buf, size) != 0) {
> +	if (memcmp(good_buf + offset, buf, size) != 0) {
>  		prt("READ BAD DATA: offset = 0x%x, size = 0x%x, fname = %s\n",
>  		    offset, size, fname);
>  		prt("OFFSET\tGOOD\tBAD\tRANGE\n");
>  		while (size > 0) {
>  			c = good_buf[offset];
> -			t = temp_buf[i];
> +			t = buf[i];
>  			if (c != t) {
>  			        if (n < 16) {
> -					bad = short_at(&temp_buf[i]);
> +					bad = short_at(&buf[i]);
>  				        prt("0x%05x\t0x%04x\t0x%04x", offset,
>  				            short_at(&good_buf[offset]), bad);
> -					op = temp_buf[offset & 1 ? i+1 : i];
> +					op = buf[offset & 1 ? i+1 : i];
>  				        prt("\t0x%05x\n", n);
>  					if (op)
>  						prt("operation# (mod 256) for "
> @@ -725,7 +726,46 @@ doread(unsigned offset, unsigned size)
>  			    iret, size);
>  		report_failure(141);
>  	}
> -	check_buffers(offset, size);
> +	check_buffers(temp_buf, offset, size);
> +}
> +
> +void
> +check_contents(void)
> +{
> +	static char *check_buf;
> +	unsigned offset = 0;
> +	unsigned size = file_size;
> +	off_t ret;
> +	unsigned iret;
> +
> +	if (!check_buf) {
> +		check_buf = (char *) malloc(maxfilelen + writebdy);
> +		assert(check_buf != NULL);
> +		check_buf = round_ptr_up(check_buf, writebdy, 0);
> +		memset(check_buf, '\0', maxfilelen);
> +	}
> +
> +	if (o_direct)
> +		size -= size % readbdy;
> +	if (size == 0)
> +		return;
> +
> +	ret = lseek(fd, (off_t)offset, SEEK_SET);
> +	if (ret == (off_t)-1) {
> +		prterr("doread: lseek");
> +		report_failure(140);
> +	}
> +
> +	iret = fsxread(fd, check_buf, size, offset);
> +	if (iret != size) {
> +		if (iret == -1)
> +			prterr("check_contents: read");
> +		else
> +			prt("short check read: 0x%x bytes instead of 0x%x\n",
> +			    iret, size);
> +		report_failure(141);
> +	}
> +	check_buffers(check_buf, offset, size);
>  }
>  
>  
> @@ -808,7 +848,7 @@ domapread(unsigned offset, unsigned size)
>  		report_failure(191);
>  	}
>  
> -	check_buffers(offset, size);
> +	check_buffers(temp_buf, offset, size);
>  }
>  
>  
> @@ -1624,6 +1664,9 @@ test(void)
>  		break;
>  	}
>  
> +	if (check_file)
> +		check_contents();
> +
>  out:
>  	if (sizechecks && testcalls > simulatedopcount)
>  		check_size();
> @@ -1684,6 +1727,7 @@ usage(void)
>  	-P: save .fsxlog .fsxops and .fsxgood files in dirpath (default ./)\n\
>  	-S seed: for random # generator (default 1) 0 gets timestamp\n\
>  	-W: mapped write operations DISabled\n\
> +	-X: Do not read file and compare to good buffer after every operation.\n\
>          -R: read() system calls only (mapped reads disabled)\n\
>          -Z: O_DIRECT (use -R, -W, -r and -w too)\n\
>  	--replay-ops opsfile: replay ops from recorded .fsxops file\n\
> @@ -1880,7 +1924,7 @@ main(int argc, char **argv)
>  	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
>  
>  	while ((ch = getopt_long(argc, argv,
> -				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
> +				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
>  				 longopts, NULL)) != EOF)
>  		switch (ch) {
>  		case 'b':
> @@ -2044,6 +2088,9 @@ main(int argc, char **argv)
>  			if (!quiet)
>  				prt("mapped writes DISABLED\n");
>  			break;
> +		case 'X':
> +			check_file = 0;
> +			break;
>  		case 'Z':
>  			o_direct = O_DIRECT;
>  			o_flags |= O_DIRECT;
> 

Re: [PATCH 02/10] fsx: always check buffer after each operation

[Darrick J. Wong]

On Tue, Nov 20, 2018 at 01:56:53PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Add a new option to make fsx read the file after each operation and
> compare it with the good buffer to try to catch corruptions as soon as
> they occur.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  ltp/fsx.c |   65 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
>  1 file changed, 56 insertions(+), 9 deletions(-)
> 
> 
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index 5601c70c..4b3a5d5a 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -160,7 +160,8 @@ int     punch_hole_calls = 1;           /* -H flag disables */
>  int     zero_range_calls = 1;           /* -z flag disables */
>  int	collapse_range_calls = 1;	/* -C flag disables */
>  int	insert_range_calls = 1;		/* -I flag disables */
> -int 	mapped_reads = 1;		/* -R flag disables it */
> +int	mapped_reads = 1;		/* -R flag disables it */
> +int	check_file = 1;			/* -X flag disables */
>  int	integrity = 0;			/* -i flag */
>  int	fsxgoodfd = 0;
>  int	o_direct;			/* -Z */
> @@ -561,7 +562,7 @@ dump_fsync_buffer(void)
>  }
>  
>  void
> -check_buffers(unsigned offset, unsigned size)
> +check_buffers(char *buf, unsigned offset, unsigned size)
>  {
>  	unsigned char c, t;
>  	unsigned i = 0;
> @@ -569,19 +570,19 @@ check_buffers(unsigned offset, unsigned size)
>  	unsigned op = 0;
>  	unsigned bad = 0;
>  
> -	if (memcmp(good_buf + offset, temp_buf, size) != 0) {
> +	if (memcmp(good_buf + offset, buf, size) != 0) {
>  		prt("READ BAD DATA: offset = 0x%x, size = 0x%x, fname = %s\n",
>  		    offset, size, fname);
>  		prt("OFFSET\tGOOD\tBAD\tRANGE\n");
>  		while (size > 0) {
>  			c = good_buf[offset];
> -			t = temp_buf[i];
> +			t = buf[i];
>  			if (c != t) {
>  			        if (n < 16) {
> -					bad = short_at(&temp_buf[i]);
> +					bad = short_at(&buf[i]);
>  				        prt("0x%05x\t0x%04x\t0x%04x", offset,
>  				            short_at(&good_buf[offset]), bad);
> -					op = temp_buf[offset & 1 ? i+1 : i];
> +					op = buf[offset & 1 ? i+1 : i];
>  				        prt("\t0x%05x\n", n);
>  					if (op)
>  						prt("operation# (mod 256) for "
> @@ -725,7 +726,46 @@ doread(unsigned offset, unsigned size)
>  			    iret, size);
>  		report_failure(141);
>  	}
> -	check_buffers(offset, size);
> +	check_buffers(temp_buf, offset, size);
> +}
> +
> +void
> +check_contents(void)
> +{
> +	static char *check_buf;
> +	unsigned offset = 0;
> +	unsigned size = file_size;
> +	off_t ret;
> +	unsigned iret;
> +
> +	if (!check_buf) {
> +		check_buf = (char *) malloc(maxfilelen + writebdy);
> +		assert(check_buf != NULL);
> +		check_buf = round_ptr_up(check_buf, writebdy, 0);
> +		memset(check_buf, '\0', maxfilelen);
> +	}
> +
> +	if (o_direct)
> +		size -= size % readbdy;
> +	if (size == 0)
> +		return;
> +
> +	ret = lseek(fd, (off_t)offset, SEEK_SET);
> +	if (ret == (off_t)-1) {
> +		prterr("doread: lseek");
> +		report_failure(140);
> +	}
> +
> +	iret = fsxread(fd, check_buf, size, offset);
> +	if (iret != size) {
> +		if (iret == -1)
> +			prterr("check_contents: read");
> +		else
> +			prt("short check read: 0x%x bytes instead of 0x%x\n",
> +			    iret, size);
> +		report_failure(141);
> +	}
> +	check_buffers(check_buf, offset, size);
>  }
>  
>  
> @@ -808,7 +848,7 @@ domapread(unsigned offset, unsigned size)
>  		report_failure(191);
>  	}
>  
> -	check_buffers(offset, size);
> +	check_buffers(temp_buf, offset, size);
>  }
>  
>  
> @@ -1624,6 +1664,9 @@ test(void)
>  		break;
>  	}
>  
> +	if (check_file)

NAK, this needs to be "if (check_file && testcalls > simulatedopcount)"
or else we end up reading the file before we actually made any
modifications to it (e.g. if -S XXX is given).

--D

> +		check_contents();
> +
>  out:
>  	if (sizechecks && testcalls > simulatedopcount)
>  		check_size();
> @@ -1684,6 +1727,7 @@ usage(void)
>  	-P: save .fsxlog .fsxops and .fsxgood files in dirpath (default ./)\n\
>  	-S seed: for random # generator (default 1) 0 gets timestamp\n\
>  	-W: mapped write operations DISabled\n\
> +	-X: Do not read file and compare to good buffer after every operation.\n\
>          -R: read() system calls only (mapped reads disabled)\n\
>          -Z: O_DIRECT (use -R, -W, -r and -w too)\n\
>  	--replay-ops opsfile: replay ops from recorded .fsxops file\n\
> @@ -1880,7 +1924,7 @@ main(int argc, char **argv)
>  	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
>  
>  	while ((ch = getopt_long(argc, argv,
> -				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
> +				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
>  				 longopts, NULL)) != EOF)
>  		switch (ch) {
>  		case 'b':
> @@ -2044,6 +2088,9 @@ main(int argc, char **argv)
>  			if (!quiet)
>  				prt("mapped writes DISABLED\n");
>  			break;
> +		case 'X':
> +			check_file = 0;
> +			break;
>  		case 'Z':
>  			o_direct = O_DIRECT;
>  			o_flags |= O_DIRECT;
> 

[PATCH 03/10] fsx: use an enum to define the operation commands

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Use an enum to define operation codes and the boundaries between
operation classes so that we can add new commands without having to
change a bunch of unrelated #defines.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |   40 +++++++++++++++++++++-------------------
 1 file changed, 21 insertions(+), 19 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index 4b3a5d5a..290b42f6 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -88,25 +88,27 @@ int			logcount = 0;	/* total ops */
  * mode being run.
  */
 
-/* common operations */
-#define	OP_READ		0
-#define OP_WRITE	1
-#define OP_MAPREAD	2
-#define OP_MAPWRITE	3
-#define OP_MAX_LITE	4
-
-/* !lite operations */
-#define OP_TRUNCATE		4
-#define OP_FALLOCATE		5
-#define OP_PUNCH_HOLE		6
-#define OP_ZERO_RANGE		7
-#define OP_COLLAPSE_RANGE	8
-#define OP_INSERT_RANGE	9
-#define OP_MAX_FULL		10
-
-/* integrity operations */
-#define OP_FSYNC		10
-#define OP_MAX_INTEGRITY	11
+enum {
+	/* common operations */
+	OP_READ = 0,
+	OP_WRITE,
+	OP_MAPREAD,
+	OP_MAPWRITE,
+	OP_MAX_LITE,
+
+	/* !lite operations */
+	OP_TRUNCATE = OP_MAX_LITE,
+	OP_FALLOCATE,
+	OP_PUNCH_HOLE,
+	OP_ZERO_RANGE,
+	OP_COLLAPSE_RANGE,
+	OP_INSERT_RANGE,
+	OP_MAX_FULL,
+
+	/* integrity operations */
+	OP_FSYNC = OP_MAX_FULL,
+	OP_MAX_INTEGRITY,
+};
 
 #undef PAGE_SIZE
 #define PAGE_SIZE       getpagesize()

[PATCH 04/10] fsx: add five-argument logging function

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add a five-argument logging function to support new operations.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |   45 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 40 insertions(+), 5 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index 290b42f6..2372964d 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -51,7 +51,8 @@ enum opflags { FL_NONE = 0, FL_SKIPPED = 1, FL_CLOSE_OPEN = 2, FL_KEEP_SIZE = 4
 
 struct log_entry {
 	int	operation;
-	int	args[3];
+	int	nr_args;
+	int	args[4];
 	enum opflags flags;
 };
 
@@ -278,6 +279,27 @@ static int op_code(const char *name)
 	return -1;
 }
 
+void
+log5(int operation, int arg0, int arg1, int arg2, enum opflags flags)
+{
+	struct log_entry *le;
+
+	le = &oplog[logptr];
+	le->operation = operation;
+	if (closeopen)
+		flags |= FL_CLOSE_OPEN;
+	le->args[0] = arg0;
+	le->args[1] = arg1;
+	le->args[2] = arg2;
+	le->args[3] = file_size;
+	le->nr_args = 4;
+	le->flags = flags;
+	logptr++;
+	logcount++;
+	if (logptr >= LOGSIZE)
+		logptr = 0;
+}
+
 void
 log4(int operation, int arg0, int arg1, enum opflags flags)
 {
@@ -290,6 +312,7 @@ log4(int operation, int arg0, int arg1, enum opflags flags)
 	le->args[0] = arg0;
 	le->args[1] = arg1;
 	le->args[2] = file_size;
+	le->nr_args = 3;
 	le->flags = flags;
 	logptr++;
 	logcount++;
@@ -439,11 +462,13 @@ logdump(void)
 			i = 0;
 
 		if (logopsf) {
+			int j;
+
 			if (lp->flags & FL_SKIPPED)
 				fprintf(logopsf, "skip ");
-			fprintf(logopsf, "%s 0x%x 0x%x 0x%x",
-				op_name(lp->operation),
-				lp->args[0], lp->args[1], lp->args[2]);
+			fprintf(logopsf, "%s", op_name(lp->operation));
+			for (j = 0; j < lp->nr_args; j++)
+				fprintf(logopsf, " 0x%x", lp->args[j]);
 			if (lp->flags & FL_KEEP_SIZE)
 				fprintf(logopsf, " keep_size");
 			if (lp->flags & FL_CLOSE_OPEN)
@@ -1413,6 +1438,15 @@ cleanup(int sig)
 	exit(sig);
 }
 
+static int
+op_args_count(int operation)
+{
+	switch (operation) {
+	default:
+		return 3;
+	}
+}
+
 static int
 read_op(struct log_entry *log_entry)
 {
@@ -1445,7 +1479,8 @@ read_op(struct log_entry *log_entry)
 		log_entry->operation = op_code(str);
 		if (log_entry->operation == -1)
 			goto fail;
-		for (i = 0; i < 3; i++) {
+		log_entry->nr_args = op_args_count(log_entry->operation);
+		for (i = 0; i < log_entry->nr_args; i++) {
 			char *end;
 
 			str = strtok(NULL, " \t\n");

[PATCH 05/10] fsx: add clone range

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add support for FICLONERANGE to fsx.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |  133 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 128 insertions(+), 5 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index 2372964d..a1a44b23 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -104,6 +104,7 @@ enum {
 	OP_ZERO_RANGE,
 	OP_COLLAPSE_RANGE,
 	OP_INSERT_RANGE,
+	OP_CLONE_RANGE,
 	OP_MAX_FULL,
 
 	/* integrity operations */
@@ -165,6 +166,7 @@ int	collapse_range_calls = 1;	/* -C flag disables */
 int	insert_range_calls = 1;		/* -I flag disables */
 int	mapped_reads = 1;		/* -R flag disables it */
 int	check_file = 1;			/* -X flag disables */
+int	clone_range_calls = 1;		/* -J flag disables */
 int	integrity = 0;			/* -i flag */
 int	fsxgoodfd = 0;
 int	o_direct;			/* -Z */
@@ -258,6 +260,7 @@ static const char *op_names[] = {
 	[OP_ZERO_RANGE] = "zero_range",
 	[OP_COLLAPSE_RANGE] = "collapse_range",
 	[OP_INSERT_RANGE] = "insert_range",
+	[OP_CLONE_RANGE] = "clone_range",
 	[OP_FSYNC] = "fsync",
 };
 
@@ -320,7 +323,6 @@ log4(int operation, int arg0, int arg1, enum opflags flags)
 		logptr = 0;
 }
 
-
 void
 logdump(void)
 {
@@ -342,7 +344,7 @@ logdump(void)
 		count = LOGSIZE;
 	}
 	for ( ; count > 0; count--) {
-		bool overlap;
+		bool overlap, overlap2;
 		int opnum;
 
 		opnum = i+1 + (logcount/LOGSIZE)*LOGSIZE;
@@ -444,6 +446,20 @@ logdump(void)
 			if (overlap)
 				prt("\t******IIII");
 			break;
+		case OP_CLONE_RANGE:
+			prt("CLONE 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
+			    lp->args[0], lp->args[0] + lp->args[1] - 1,
+			    lp->args[1],
+			    lp->args[2], lp->args[2] + lp->args[1] - 1);
+			overlap2 = badoff >= lp->args[2] &&
+				  badoff < lp->args[2] + lp->args[1];
+			if (overlap && overlap2)
+				prt("\tJJJJ**JJJJ");
+			else if (overlap)
+				prt("\tJJJJ******");
+			else if (overlap2)
+				prt("\t******JJJJ");
+			break;
 		case OP_FSYNC:
 			prt("FSYNC");
 			break;
@@ -1285,6 +1301,73 @@ do_insert_range(unsigned offset, unsigned length)
 }
 #endif
 
+#ifdef FICLONERANGE
+void
+do_clone_range(unsigned offset, unsigned length, unsigned dest)
+{
+	struct file_clone_range	fcr;
+
+	if (length == 0) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping zero length clone range\n");
+		log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	if ((loff_t)offset >= file_size) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping clone range behind EOF\n");
+		log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	log5(OP_CLONE_RANGE, offset, length, dest, FL_NONE);
+
+	if (testcalls <= simulatedopcount)
+		return;
+
+	if ((progressinterval && testcalls % progressinterval == 0) ||
+	    (debug && (monitorstart == -1 || monitorend == -1 ||
+		       dest <= monitorstart || dest + length <= monitorend))) {
+		prt("%lu clone\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
+			testcalls, offset, offset+length, length, dest);
+	}
+
+	fcr.src_fd = fd;
+	fcr.src_offset = offset;
+	fcr.src_length = length;
+	fcr.dest_offset = dest;
+
+	if (ioctl(fd, FICLONERANGE, &fcr) == -1) {
+		if (errno == EOPNOTSUPP || errno == ENOTTY) {
+			if (!quiet && testcalls > simulatedopcount)
+				prt("skipping unsupported clone range\n");
+			logptr--;
+			log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
+			return;
+		}
+
+		prt("clone range: 0x%x to 0x%x at 0x%x\n", offset,
+				offset + length, dest);
+		prterr("do_clone_range: FICLONERANGE");
+		report_failure(161);
+	}
+
+	memcpy(good_buf + dest, good_buf + offset, length);
+	if (dest > file_size)
+		memset(good_buf + file_size, '\0', dest - file_size);
+	if (dest + length > file_size)
+		file_size = dest + length;
+}
+
+#else
+void
+do_clone_range(unsigned offset, unsigned length, unsigned dest)
+{
+	return;
+}
+#endif
+
 #ifdef HAVE_LINUX_FALLOC_H
 /* fallocate is basically a no-op unless extending, then a lot like a truncate */
 void
@@ -1442,6 +1525,8 @@ static int
 op_args_count(int operation)
 {
 	switch (operation) {
+	case OP_CLONE_RANGE:
+		return 4;
 	default:
 		return 3;
 	}
@@ -1450,6 +1535,7 @@ op_args_count(int operation)
 static int
 read_op(struct log_entry *log_entry)
 {
+	char *end;
 	char line[256];
 
 	memset(log_entry, 0, sizeof(*log_entry));
@@ -1515,7 +1601,7 @@ read_op(struct log_entry *log_entry)
 int
 test(void)
 {
-	unsigned long	offset;
+	unsigned long	offset, offset2;
 	unsigned long	size;
 	unsigned long	rv;
 	unsigned long	op;
@@ -1546,6 +1632,7 @@ test(void)
 			op = log_entry.operation;
 			offset = log_entry.args[0];
 			size = log_entry.args[1];
+			offset2 = log_entry.args[2];
 			closeopen = !!(log_entry.flags & FL_CLOSE_OPEN);
 			keep_size = !!(log_entry.flags & FL_KEEP_SIZE);
 			goto have_op;
@@ -1558,6 +1645,7 @@ test(void)
 		closeopen = (rv >> 3) < (1 << 28) / closeprob;
 
 	offset = random();
+	offset2 = 0;
 	size = maxoplen;
 	if (randomoplen)
 		size = random() % (maxoplen + 1);
@@ -1583,6 +1671,17 @@ test(void)
 		if (zero_range_calls && size && keep_size_calls)
 			keep_size = random() % 2;
 		break;
+	case OP_CLONE_RANGE:
+		TRIM_OFF_LEN(offset, size, file_size);
+		offset = offset & ~(block_size - 1);
+		size = size & ~(block_size - 1);
+		do {
+			offset2 = random();
+			TRIM_OFF(offset2, maxfilelen);
+			offset2 = offset2 & ~(block_size - 1);
+		} while (llabs(offset2 - offset) < size ||
+			 offset2 + size > maxfilelen);
+		break;
 	}
 
 have_op:
@@ -1626,6 +1725,12 @@ test(void)
 			goto out;
 		}
 		break;
+	case OP_CLONE_RANGE:
+		if (!clone_range_calls) {
+			log5(op, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		break;
 	}
 
 	switch (op) {
@@ -1692,6 +1797,18 @@ test(void)
 
 		do_insert_range(offset, size);
 		break;
+	case OP_CLONE_RANGE:
+		if (size == 0) {
+			log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		if (offset2 + size > maxfilelen) {
+			log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+
+		do_clone_range(offset, size, offset2);
+		break;
 	case OP_FSYNC:
 		dofsync();
 		break;
@@ -1717,7 +1834,7 @@ void
 usage(void)
 {
 	fprintf(stdout, "usage: %s",
-		"fsx [-dknqxAFLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+		"fsx [-dknqxAFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
 	-b opnum: beginning operation number (default 1)\n\
 	-c P: 1 in P chance of file close+open at each op (default infinity)\n\
 	-d: debug output for all operations\n\
@@ -1758,6 +1875,9 @@ usage(void)
 #ifdef FALLOC_FL_INSERT_RANGE
 "	-I: Do not use insert range calls\n"
 #endif
+#ifdef FICLONERANGE
+"	-J: Do not use clone range calls\n"
+#endif
 "	-L: fsxLite - no file creations & no file size changes\n\
 	-N numops: total # operations to do (default infinity)\n\
 	-O: use oplen (see -o flag) for every op (default random)\n\
@@ -1961,7 +2081,7 @@ main(int argc, char **argv)
 	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
 
 	while ((ch = getopt_long(argc, argv,
-				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
+				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FJKHzCILN:OP:RS:WXZ",
 				 longopts, NULL)) != EOF)
 		switch (ch) {
 		case 'b':
@@ -2091,6 +2211,9 @@ main(int argc, char **argv)
 		case 'I':
 			insert_range_calls = 0;
 			break;
+		case 'J':
+			clone_range_calls = 0;
+			break;
 		case 'L':
 		        lite = 1;
 			o_flags &= ~(O_CREAT|O_TRUNC);

Re: [PATCH 05/10] fsx: add clone range

[Darrick J. Wong]

On Tue, Nov 20, 2018 at 01:57:12PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Add support for FICLONERANGE to fsx.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  ltp/fsx.c |  133 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 128 insertions(+), 5 deletions(-)
> 
> 
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index 2372964d..a1a44b23 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -104,6 +104,7 @@ enum {
>  	OP_ZERO_RANGE,
>  	OP_COLLAPSE_RANGE,
>  	OP_INSERT_RANGE,
> +	OP_CLONE_RANGE,
>  	OP_MAX_FULL,
>  
>  	/* integrity operations */
> @@ -165,6 +166,7 @@ int	collapse_range_calls = 1;	/* -C flag disables */
>  int	insert_range_calls = 1;		/* -I flag disables */
>  int	mapped_reads = 1;		/* -R flag disables it */
>  int	check_file = 1;			/* -X flag disables */
> +int	clone_range_calls = 1;		/* -J flag disables */
>  int	integrity = 0;			/* -i flag */
>  int	fsxgoodfd = 0;
>  int	o_direct;			/* -Z */
> @@ -258,6 +260,7 @@ static const char *op_names[] = {
>  	[OP_ZERO_RANGE] = "zero_range",
>  	[OP_COLLAPSE_RANGE] = "collapse_range",
>  	[OP_INSERT_RANGE] = "insert_range",
> +	[OP_CLONE_RANGE] = "clone_range",
>  	[OP_FSYNC] = "fsync",
>  };
>  
> @@ -320,7 +323,6 @@ log4(int operation, int arg0, int arg1, enum opflags flags)
>  		logptr = 0;
>  }
>  
> -
>  void
>  logdump(void)
>  {
> @@ -342,7 +344,7 @@ logdump(void)
>  		count = LOGSIZE;
>  	}
>  	for ( ; count > 0; count--) {
> -		bool overlap;
> +		bool overlap, overlap2;
>  		int opnum;
>  
>  		opnum = i+1 + (logcount/LOGSIZE)*LOGSIZE;
> @@ -444,6 +446,20 @@ logdump(void)
>  			if (overlap)
>  				prt("\t******IIII");
>  			break;
> +		case OP_CLONE_RANGE:
> +			prt("CLONE 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
> +			    lp->args[0], lp->args[0] + lp->args[1] - 1,
> +			    lp->args[1],
> +			    lp->args[2], lp->args[2] + lp->args[1] - 1);
> +			overlap2 = badoff >= lp->args[2] &&
> +				  badoff < lp->args[2] + lp->args[1];
> +			if (overlap && overlap2)
> +				prt("\tJJJJ**JJJJ");
> +			else if (overlap)
> +				prt("\tJJJJ******");
> +			else if (overlap2)
> +				prt("\t******JJJJ");
> +			break;
>  		case OP_FSYNC:
>  			prt("FSYNC");
>  			break;
> @@ -1285,6 +1301,73 @@ do_insert_range(unsigned offset, unsigned length)
>  }
>  #endif
>  
> +#ifdef FICLONERANGE
> +void
> +do_clone_range(unsigned offset, unsigned length, unsigned dest)
> +{
> +	struct file_clone_range	fcr;
> +
> +	if (length == 0) {
> +		if (!quiet && testcalls > simulatedopcount)
> +			prt("skipping zero length clone range\n");
> +		log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
> +		return;
> +	}
> +
> +	if ((loff_t)offset >= file_size) {
> +		if (!quiet && testcalls > simulatedopcount)
> +			prt("skipping clone range behind EOF\n");
> +		log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
> +		return;
> +	}
> +
> +	log5(OP_CLONE_RANGE, offset, length, dest, FL_NONE);
> +
> +	if (testcalls <= simulatedopcount)
> +		return;
> +
> +	if ((progressinterval && testcalls % progressinterval == 0) ||
> +	    (debug && (monitorstart == -1 || monitorend == -1 ||
> +		       dest <= monitorstart || dest + length <= monitorend))) {
> +		prt("%lu clone\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
> +			testcalls, offset, offset+length, length, dest);
> +	}
> +
> +	fcr.src_fd = fd;
> +	fcr.src_offset = offset;
> +	fcr.src_length = length;
> +	fcr.dest_offset = dest;
> +
> +	if (ioctl(fd, FICLONERANGE, &fcr) == -1) {
> +		if (errno == EOPNOTSUPP || errno == ENOTTY) {
> +			if (!quiet && testcalls > simulatedopcount)
> +				prt("skipping unsupported clone range\n");
> +			logptr--;

NAK, we should just have a test_clone_range that we can use to disable
clone range calls before we start generating calls, because oplog[] is a
rolling buffer and this can cause internal memory corruption if logptr
is 0.

--D

> +			log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
> +			return;
> +		}
> +
> +		prt("clone range: 0x%x to 0x%x at 0x%x\n", offset,
> +				offset + length, dest);
> +		prterr("do_clone_range: FICLONERANGE");
> +		report_failure(161);
> +	}
> +
> +	memcpy(good_buf + dest, good_buf + offset, length);
> +	if (dest > file_size)
> +		memset(good_buf + file_size, '\0', dest - file_size);
> +	if (dest + length > file_size)
> +		file_size = dest + length;
> +}
> +
> +#else
> +void
> +do_clone_range(unsigned offset, unsigned length, unsigned dest)
> +{
> +	return;
> +}
> +#endif
> +
>  #ifdef HAVE_LINUX_FALLOC_H
>  /* fallocate is basically a no-op unless extending, then a lot like a truncate */
>  void
> @@ -1442,6 +1525,8 @@ static int
>  op_args_count(int operation)
>  {
>  	switch (operation) {
> +	case OP_CLONE_RANGE:
> +		return 4;
>  	default:
>  		return 3;
>  	}
> @@ -1450,6 +1535,7 @@ op_args_count(int operation)
>  static int
>  read_op(struct log_entry *log_entry)
>  {
> +	char *end;
>  	char line[256];
>  
>  	memset(log_entry, 0, sizeof(*log_entry));
> @@ -1515,7 +1601,7 @@ read_op(struct log_entry *log_entry)
>  int
>  test(void)
>  {
> -	unsigned long	offset;
> +	unsigned long	offset, offset2;
>  	unsigned long	size;
>  	unsigned long	rv;
>  	unsigned long	op;
> @@ -1546,6 +1632,7 @@ test(void)
>  			op = log_entry.operation;
>  			offset = log_entry.args[0];
>  			size = log_entry.args[1];
> +			offset2 = log_entry.args[2];
>  			closeopen = !!(log_entry.flags & FL_CLOSE_OPEN);
>  			keep_size = !!(log_entry.flags & FL_KEEP_SIZE);
>  			goto have_op;
> @@ -1558,6 +1645,7 @@ test(void)
>  		closeopen = (rv >> 3) < (1 << 28) / closeprob;
>  
>  	offset = random();
> +	offset2 = 0;
>  	size = maxoplen;
>  	if (randomoplen)
>  		size = random() % (maxoplen + 1);
> @@ -1583,6 +1671,17 @@ test(void)
>  		if (zero_range_calls && size && keep_size_calls)
>  			keep_size = random() % 2;
>  		break;
> +	case OP_CLONE_RANGE:
> +		TRIM_OFF_LEN(offset, size, file_size);
> +		offset = offset & ~(block_size - 1);
> +		size = size & ~(block_size - 1);
> +		do {
> +			offset2 = random();
> +			TRIM_OFF(offset2, maxfilelen);
> +			offset2 = offset2 & ~(block_size - 1);
> +		} while (llabs(offset2 - offset) < size ||
> +			 offset2 + size > maxfilelen);
> +		break;
>  	}
>  
>  have_op:
> @@ -1626,6 +1725,12 @@ test(void)
>  			goto out;
>  		}
>  		break;
> +	case OP_CLONE_RANGE:
> +		if (!clone_range_calls) {
> +			log5(op, offset, size, offset2, FL_SKIPPED);
> +			goto out;
> +		}
> +		break;
>  	}
>  
>  	switch (op) {
> @@ -1692,6 +1797,18 @@ test(void)
>  
>  		do_insert_range(offset, size);
>  		break;
> +	case OP_CLONE_RANGE:
> +		if (size == 0) {
> +			log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED);
> +			goto out;
> +		}
> +		if (offset2 + size > maxfilelen) {
> +			log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED);
> +			goto out;
> +		}
> +
> +		do_clone_range(offset, size, offset2);
> +		break;
>  	case OP_FSYNC:
>  		dofsync();
>  		break;
> @@ -1717,7 +1834,7 @@ void
>  usage(void)
>  {
>  	fprintf(stdout, "usage: %s",
> -		"fsx [-dknqxAFLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
> +		"fsx [-dknqxAFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
>  	-b opnum: beginning operation number (default 1)\n\
>  	-c P: 1 in P chance of file close+open at each op (default infinity)\n\
>  	-d: debug output for all operations\n\
> @@ -1758,6 +1875,9 @@ usage(void)
>  #ifdef FALLOC_FL_INSERT_RANGE
>  "	-I: Do not use insert range calls\n"
>  #endif
> +#ifdef FICLONERANGE
> +"	-J: Do not use clone range calls\n"
> +#endif
>  "	-L: fsxLite - no file creations & no file size changes\n\
>  	-N numops: total # operations to do (default infinity)\n\
>  	-O: use oplen (see -o flag) for every op (default random)\n\
> @@ -1961,7 +2081,7 @@ main(int argc, char **argv)
>  	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
>  
>  	while ((ch = getopt_long(argc, argv,
> -				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
> +				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FJKHzCILN:OP:RS:WXZ",
>  				 longopts, NULL)) != EOF)
>  		switch (ch) {
>  		case 'b':
> @@ -2091,6 +2211,9 @@ main(int argc, char **argv)
>  		case 'I':
>  			insert_range_calls = 0;
>  			break;
> +		case 'J':
> +			clone_range_calls = 0;
> +			break;
>  		case 'L':
>  		        lite = 1;
>  			o_flags &= ~(O_CREAT|O_TRUNC);
> 

[PATCH 06/10] fsx: add FIDEDUPERANGE support

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |  141 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 139 insertions(+), 2 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index a1a44b23..6f6263e2 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -105,6 +105,7 @@ enum {
 	OP_COLLAPSE_RANGE,
 	OP_INSERT_RANGE,
 	OP_CLONE_RANGE,
+	OP_DEDUPE_RANGE,
 	OP_MAX_FULL,
 
 	/* integrity operations */
@@ -167,6 +168,7 @@ int	insert_range_calls = 1;		/* -I flag disables */
 int	mapped_reads = 1;		/* -R flag disables it */
 int	check_file = 1;			/* -X flag disables */
 int	clone_range_calls = 1;		/* -J flag disables */
+int	dedupe_range_calls = 1;		/* -B flag disables */
 int	integrity = 0;			/* -i flag */
 int	fsxgoodfd = 0;
 int	o_direct;			/* -Z */
@@ -261,6 +263,7 @@ static const char *op_names[] = {
 	[OP_COLLAPSE_RANGE] = "collapse_range",
 	[OP_INSERT_RANGE] = "insert_range",
 	[OP_CLONE_RANGE] = "clone_range",
+	[OP_DEDUPE_RANGE] = "dedupe_range",
 	[OP_FSYNC] = "fsync",
 };
 
@@ -460,6 +463,20 @@ logdump(void)
 			else if (overlap2)
 				prt("\t******JJJJ");
 			break;
+		case OP_DEDUPE_RANGE:
+			prt("DEDUPE 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
+			    lp->args[0], lp->args[0] + lp->args[1] - 1,
+			    lp->args[1],
+			    lp->args[2], lp->args[2] + lp->args[1] - 1);
+			overlap2 = badoff >= lp->args[2] &&
+				  badoff < lp->args[2] + lp->args[1];
+			if (overlap && overlap2)
+				prt("\tBBBB**BBBB");
+			else if (overlap)
+				prt("\tBBBB******");
+			else if (overlap2)
+				prt("\t******BBBB");
+			break;
 		case OP_FSYNC:
 			prt("FSYNC");
 			break;
@@ -1368,6 +1385,82 @@ do_clone_range(unsigned offset, unsigned length, unsigned dest)
 }
 #endif
 
+#ifdef FIDEDUPERANGE
+void
+do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
+{
+	struct file_dedupe_range *fdr;
+
+	if (length == 0) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping zero length dedupe range\n");
+		log5(OP_DEDUPE_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	if ((loff_t)offset >= file_size) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping dedupe range behind EOF\n");
+		log5(OP_DEDUPE_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	log5(OP_DEDUPE_RANGE, offset, length, dest, FL_NONE);
+
+	if (testcalls <= simulatedopcount)
+		return;
+
+	if ((progressinterval && testcalls % progressinterval == 0) ||
+	    (debug && (monitorstart == -1 || monitorend == -1 ||
+		       dest <= monitorstart || dest + length <= monitorend))) {
+		prt("%lu dedupe\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
+			testcalls, offset, offset+length, length, dest);
+	}
+
+	/* Alloc memory */
+	fdr = malloc(sizeof(struct file_dedupe_range_info) +
+		     sizeof(struct file_dedupe_range));
+	if (!fdr) {
+		prterr("do_dedupe_range: malloc");
+		report_failure(161);
+	}
+	memset(fdr, 0, (sizeof(struct file_dedupe_range_info) +
+			sizeof(struct file_dedupe_range)));
+
+	/* Dedupe data blocks */
+	fdr->src_offset = offset;
+	fdr->src_length = length;
+	fdr->dest_count = 1;
+	fdr->info[0].dest_fd = fd;
+	fdr->info[0].dest_offset = dest;
+
+	if (ioctl(fd, FIDEDUPERANGE, fdr) == -1) {
+		if (errno == EOPNOTSUPP || errno == ENOTTY) {
+			if (!quiet && testcalls > simulatedopcount)
+				prt("skipping unsupported dedupe range\n");
+			logptr--;
+			log5(OP_DEDUPE_RANGE, offset, length, dest, FL_SKIPPED);
+			free(fdr);
+			return;
+		}
+
+		prt("dedupe range: 0x%x to 0x%x at 0x%x\n", offset,
+				offset + length, dest);
+		prterr("do_dedupe_range: FIDEDUPERANGE");
+		report_failure(161);
+	}
+
+	free(fdr);
+}
+
+#else
+void
+do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
+{
+	return;
+}
+#endif
+
 #ifdef HAVE_LINUX_FALLOC_H
 /* fallocate is basically a no-op unless extending, then a lot like a truncate */
 void
@@ -1526,6 +1619,7 @@ op_args_count(int operation)
 {
 	switch (operation) {
 	case OP_CLONE_RANGE:
+	case OP_DEDUPE_RANGE:
 		return 4;
 	default:
 		return 3;
@@ -1682,6 +1776,25 @@ test(void)
 		} while (llabs(offset2 - offset) < size ||
 			 offset2 + size > maxfilelen);
 		break;
+	case OP_DEDUPE_RANGE:
+		{
+			int tries = 0;
+
+			TRIM_OFF_LEN(offset, size, file_size);
+			offset = offset & ~(block_size - 1);
+			size = size & ~(block_size - 1);
+			do {
+				if (tries++ >= 30) {
+					size = 0;
+					break;
+				}
+				offset2 = random();
+				TRIM_OFF(offset2, file_size);
+				offset2 = offset2 & ~(block_size - 1);
+			} while (llabs(offset2 - offset) < size ||
+				 offset2 + size > file_size);
+			break;
+		}
 	}
 
 have_op:
@@ -1731,6 +1844,12 @@ test(void)
 			goto out;
 		}
 		break;
+	case OP_DEDUPE_RANGE:
+		if (!dedupe_range_calls) {
+			log5(op, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		break;
 	}
 
 	switch (op) {
@@ -1809,6 +1928,18 @@ test(void)
 
 		do_clone_range(offset, size, offset2);
 		break;
+	case OP_DEDUPE_RANGE:
+		if (size == 0) {
+			log5(OP_DEDUPE_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		if (offset2 + size > maxfilelen) {
+			log5(OP_DEDUPE_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+
+		do_dedupe_range(offset, size, offset2);
+		break;
 	case OP_FSYNC:
 		dofsync();
 		break;
@@ -1834,7 +1965,7 @@ void
 usage(void)
 {
 	fprintf(stdout, "usage: %s",
-		"fsx [-dknqxAFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+		"fsx [-dknqxABFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
 	-b opnum: beginning operation number (default 1)\n\
 	-c P: 1 in P chance of file close+open at each op (default infinity)\n\
 	-d: debug output for all operations\n\
@@ -1878,6 +2009,9 @@ usage(void)
 #ifdef FICLONERANGE
 "	-J: Do not use clone range calls\n"
 #endif
+#ifdef FIDEDUPERANGE
+"	-B: Do not use dedupe range calls\n"
+#endif
 "	-L: fsxLite - no file creations & no file size changes\n\
 	-N numops: total # operations to do (default infinity)\n\
 	-O: use oplen (see -o flag) for every op (default random)\n\
@@ -2081,7 +2215,7 @@ main(int argc, char **argv)
 	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
 
 	while ((ch = getopt_long(argc, argv,
-				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FJKHzCILN:OP:RS:WXZ",
+				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:FJKHzCILN:OP:RS:WXZ",
 				 longopts, NULL)) != EOF)
 		switch (ch) {
 		case 'b':
@@ -2214,6 +2348,9 @@ main(int argc, char **argv)
 		case 'J':
 			clone_range_calls = 0;
 			break;
+		case 'B':
+			dedupe_range_calls = 0;
+			break;
 		case 'L':
 		        lite = 1;
 			o_flags &= ~(O_CREAT|O_TRUNC);

[PATCH 07/10] fsstress: add copy_file_range support

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Support the copy_file_range syscall.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 configure.ac          |    2 +
 include/builddefs.in  |    1 
 ltp/Makefile          |    4 +
 ltp/fsstress.c        |  133 +++++++++++++++++++++++++++++++++++++++++++++++++
 m4/package_libcdev.m4 |   19 +++++++
 5 files changed, 159 insertions(+)


diff --git a/configure.ac b/configure.ac
index aede4f59..19798824 100644
--- a/configure.ac
+++ b/configure.ac
@@ -68,6 +68,8 @@ AC_PACKAGE_WANT_OPEN_BY_HANDLE_AT
 AC_PACKAGE_WANT_LINUX_PRCTL_H
 AC_PACKAGE_WANT_LINUX_FS_H
 
+AC_HAVE_COPY_FILE_RANGE
+
 AC_CHECK_FUNCS([renameat2])
 
 AC_CONFIG_HEADER(include/config.h)
diff --git a/include/builddefs.in b/include/builddefs.in
index fb8e912b..2605e42d 100644
--- a/include/builddefs.in
+++ b/include/builddefs.in
@@ -67,6 +67,7 @@ HAVE_DMAPI = @have_dmapi@
 HAVE_ATTR_LIST = @have_attr_list@
 HAVE_FIEMAP = @have_fiemap@
 HAVE_FALLOCATE = @have_fallocate@
+HAVE_COPY_FILE_RANGE = @have_copy_file_range@
 
 GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall
 
diff --git a/ltp/Makefile b/ltp/Makefile
index 5f784ecf..e4ca45f4 100644
--- a/ltp/Makefile
+++ b/ltp/Makefile
@@ -28,6 +28,10 @@ ifeq ($(HAVE_FALLOCATE), true)
 LCFLAGS += -DFALLOCATE
 endif
 
+ifeq ($(HAVE_COPY_FILE_RANGE),yes)
+LCFLAGS += -DHAVE_COPY_FILE_RANGE
+endif
+
 default: depend $(TARGETS)
 
 depend: .dep
diff --git a/ltp/fsstress.c b/ltp/fsstress.c
index edbb3f2f..dded0cf7 100644
--- a/ltp/fsstress.c
+++ b/ltp/fsstress.c
@@ -26,6 +26,7 @@
 #include <libaio.h>
 io_context_t	io_ctx;
 #endif
+#include <sys/syscall.h>
 
 #ifndef FS_IOC_GETFLAGS
 #define FS_IOC_GETFLAGS                 _IOR('f', 1, long)
@@ -55,6 +56,7 @@ typedef enum {
 	OP_BULKSTAT1,
 	OP_CHOWN,
 	OP_CLONERANGE,
+	OP_COPYRANGE,
 	OP_CREAT,
 	OP_DEDUPERANGE,
 	OP_DREAD,
@@ -163,6 +165,7 @@ void	bulkstat_f(int, long);
 void	bulkstat1_f(int, long);
 void	chown_f(int, long);
 void	clonerange_f(int, long);
+void	copyrange_f(int, long);
 void	creat_f(int, long);
 void	deduperange_f(int, long);
 void	dread_f(int, long);
@@ -212,6 +215,7 @@ opdesc_t	ops[] = {
 	{ OP_BULKSTAT1, "bulkstat1", bulkstat1_f, 1, 0 },
 	{ OP_CHOWN, "chown", chown_f, 3, 1 },
 	{ OP_CLONERANGE, "clonerange", clonerange_f, 4, 1 },
+	{ OP_COPYRANGE, "copyrange", copyrange_f, 4, 1 },
 	{ OP_CREAT, "creat", creat_f, 4, 1 },
 	{ OP_DEDUPERANGE, "deduperange", deduperange_f, 4, 1},
 	{ OP_DREAD, "dread", dread_f, 4, 0 },
@@ -2329,6 +2333,135 @@ clonerange_f(
 #endif
 }
 
+/* copy some arbitrary range of f1 to f2. */
+void
+copyrange_f(
+	int			opno,
+	long			r)
+{
+#ifdef HAVE_COPY_FILE_RANGE
+	struct pathname		fpath1;
+	struct pathname		fpath2;
+	struct stat64		stat1;
+	struct stat64		stat2;
+	char			inoinfo1[1024];
+	char			inoinfo2[1024];
+	loff_t			lr;
+	loff_t			off1;
+	loff_t			off2;
+	loff_t			max_off2;
+	size_t			len;
+	int			v1;
+	int			v2;
+	int			fd1;
+	int			fd2;
+	int			ret;
+	int			e;
+
+	/* Load paths */
+	init_pathname(&fpath1);
+	if (!get_fname(FT_REGm, r, &fpath1, NULL, NULL, &v1)) {
+		if (v1)
+			printf("%d/%d: copyrange read - no filename\n",
+				procid, opno);
+		goto out_fpath1;
+	}
+
+	init_pathname(&fpath2);
+	if (!get_fname(FT_REGm, random(), &fpath2, NULL, NULL, &v2)) {
+		if (v2)
+			printf("%d/%d: copyrange write - no filename\n",
+				procid, opno);
+		goto out_fpath2;
+	}
+
+	/* Open files */
+	fd1 = open_path(&fpath1, O_RDONLY);
+	e = fd1 < 0 ? errno : 0;
+	check_cwd();
+	if (fd1 < 0) {
+		if (v1)
+			printf("%d/%d: copyrange read - open %s failed %d\n",
+				procid, opno, fpath1.path, e);
+		goto out_fpath2;
+	}
+
+	fd2 = open_path(&fpath2, O_WRONLY);
+	e = fd2 < 0 ? errno : 0;
+	check_cwd();
+	if (fd2 < 0) {
+		if (v2)
+			printf("%d/%d: copyrange write - open %s failed %d\n",
+				procid, opno, fpath2.path, e);
+		goto out_fd1;
+	}
+
+	/* Get file stats */
+	if (fstat64(fd1, &stat1) < 0) {
+		if (v1)
+			printf("%d/%d: copyrange read - fstat64 %s failed %d\n",
+				procid, opno, fpath1.path, errno);
+		goto out_fd2;
+	}
+	inode_info(inoinfo1, sizeof(inoinfo1), &stat1, v1);
+
+	if (fstat64(fd2, &stat2) < 0) {
+		if (v2)
+			printf("%d/%d: copyrange write - fstat64 %s failed %d\n",
+				procid, opno, fpath2.path, errno);
+		goto out_fd2;
+	}
+	inode_info(inoinfo2, sizeof(inoinfo2), &stat2, v2);
+
+	/* Calculate offsets */
+	len = (random() % FILELEN_MAX) + 1;
+	if (len == 0)
+		len = stat1.st_blksize;
+	if (len > stat1.st_size)
+		len = stat1.st_size;
+
+	lr = ((int64_t)random() << 32) + random();
+	if (stat1.st_size == len)
+		off1 = 0;
+	else
+		off1 = (off64_t)(lr % MIN(stat1.st_size - len, MAXFSIZE));
+	off1 %= maxfsize;
+
+	/*
+	 * If srcfile == destfile, randomly generate destination ranges
+	 * until we find one that doesn't overlap the source range.
+	 */
+	max_off2 = MIN(stat2.st_size + (1024ULL * stat2.st_blksize), MAXFSIZE);
+	do {
+		lr = ((int64_t)random() << 32) + random();
+		off2 = (off64_t)(lr % max_off2);
+		off2 %= maxfsize;
+	} while (stat1.st_ino == stat2.st_ino && llabs(off2 - off1) < len);
+
+	ret = copy_file_range(fd1, &off1, fd2, &off2, len, 0);
+	e = ret < 0 ? errno : 0;
+	if (v1 || v2) {
+		printf("%d/%d: copyrange %s%s [%lld,%lld] -> %s%s [%lld,%lld]",
+			procid, opno,
+			fpath1.path, inoinfo1, (long long)off1, (long long)len,
+			fpath2.path, inoinfo2, (long long)off2, (long long)len);
+
+		if (ret < 0)
+			printf(" error %d", e);
+		printf("\n");
+	}
+
+out_fd2:
+	close(fd2);
+out_fd1:
+	close(fd1);
+out_fpath2:
+	free_pathname(&fpath2);
+out_fpath1:
+	free_pathname(&fpath1);
+#endif
+}
+
 /* dedupe some arbitrary range of f1 to f2...fn. */
 void
 deduperange_f(
diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4
index fb123f33..14e67e18 100644
--- a/m4/package_libcdev.m4
+++ b/m4/package_libcdev.m4
@@ -98,3 +98,22 @@ AC_DEFUN([AC_HAVE_GETMNTINFO],
        AC_MSG_RESULT(no))
     AC_SUBST(have_getmntinfo)
   ])
+
+#
+#
+# Check if we have a copy_file_range system call (Linux)
+#
+AC_DEFUN([AC_HAVE_COPY_FILE_RANGE],
+  [ AC_MSG_CHECKING([for copy_file_range])
+    AC_TRY_LINK([
+#define _GNU_SOURCE
+#include <sys/syscall.h>
+#include <unistd.h>
+    ], [
+         syscall(__NR_copy_file_range, 0, 0, 0, 0, 0, 0);
+    ], have_copy_file_range=yes
+       AC_MSG_RESULT(yes),
+       AC_MSG_RESULT(no))
+    AC_SUBST(have_copy_file_range)
+  ])
+

[PATCH 08/10] fsx: add copy_file_range support

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |  122 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 120 insertions(+), 2 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index 6f6263e2..ec3defad 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -34,6 +34,7 @@
 #ifdef AIO
 #include <libaio.h>
 #endif
+#include <sys/syscall.h>
 
 #ifndef MAP_FILE
 # define MAP_FILE 0
@@ -106,6 +107,7 @@ enum {
 	OP_INSERT_RANGE,
 	OP_CLONE_RANGE,
 	OP_DEDUPE_RANGE,
+	OP_COPY_RANGE,
 	OP_MAX_FULL,
 
 	/* integrity operations */
@@ -169,6 +171,7 @@ int	mapped_reads = 1;		/* -R flag disables it */
 int	check_file = 1;			/* -X flag disables */
 int	clone_range_calls = 1;		/* -J flag disables */
 int	dedupe_range_calls = 1;		/* -B flag disables */
+int	copy_range_calls = 1;		/* -E flag disables */
 int	integrity = 0;			/* -i flag */
 int	fsxgoodfd = 0;
 int	o_direct;			/* -Z */
@@ -264,6 +267,7 @@ static const char *op_names[] = {
 	[OP_INSERT_RANGE] = "insert_range",
 	[OP_CLONE_RANGE] = "clone_range",
 	[OP_DEDUPE_RANGE] = "dedupe_range",
+	[OP_COPY_RANGE] = "copy_range",
 	[OP_FSYNC] = "fsync",
 };
 
@@ -477,6 +481,20 @@ logdump(void)
 			else if (overlap2)
 				prt("\t******BBBB");
 			break;
+		case OP_COPY_RANGE:
+			prt("COPY 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
+			    lp->args[0], lp->args[0] + lp->args[1] - 1,
+			    lp->args[1],
+			    lp->args[2], lp->args[2] + lp->args[1] - 1);
+			overlap2 = badoff >= lp->args[2] &&
+				  badoff < lp->args[2] + lp->args[1];
+			if (overlap && overlap2)
+				prt("\tEEEE**EEEE");
+			else if (overlap)
+				prt("\tEEEE******");
+			else if (overlap2)
+				prt("\t******EEEE");
+			break;
 		case OP_FSYNC:
 			prt("FSYNC");
 			break;
@@ -1461,6 +1479,73 @@ do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
 }
 #endif
 
+#ifdef HAVE_COPY_FILE_RANGE
+void
+do_copy_range(unsigned offset, unsigned length, unsigned dest)
+{
+	loff_t o1, o2;
+	ssize_t nr;
+
+	if (length == 0) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping zero length copy range\n");
+		log5(OP_COPY_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	if ((loff_t)offset >= file_size) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping copy range behind EOF\n");
+		log5(OP_COPY_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	log5(OP_COPY_RANGE, offset, length, dest, FL_NONE);
+
+	if (testcalls <= simulatedopcount)
+		return;
+
+	if ((progressinterval && testcalls % progressinterval == 0) ||
+	    (debug && (monitorstart == -1 || monitorend == -1 ||
+		       dest <= monitorstart || dest + length <= monitorend))) {
+		prt("%lu copy\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
+			testcalls, offset, offset+length, length, dest);
+	}
+
+	o1 = offset;
+	o2 = dest;
+
+	nr = copy_file_range(fd, &o1, fd, &o2, length, 0);
+	if (nr == -1) {
+		if (errno == EOPNOTSUPP || errno == ENOTTY) {
+			if (!quiet && testcalls > simulatedopcount)
+				prt("skipping unsupported copy range\n");
+			logptr--;
+			log5(OP_COPY_RANGE, offset, length, dest, FL_SKIPPED);
+			return;
+		}
+
+		prt("copy range: 0x%x to 0x%x at 0x%x\n", offset,
+				offset + length, dest);
+		prterr("do_copy_range:");
+		report_failure(161);
+	}
+
+	memcpy(good_buf + dest, good_buf + offset, nr);
+	if (dest > file_size)
+		memset(good_buf + file_size, '\0', dest - file_size);
+	if (dest + nr > file_size)
+		file_size = dest + nr;
+}
+
+#else
+void
+do_copy_range(unsigned offset, unsigned length, unsigned dest)
+{
+	return;
+}
+#endif
+
 #ifdef HAVE_LINUX_FALLOC_H
 /* fallocate is basically a no-op unless extending, then a lot like a truncate */
 void
@@ -1620,6 +1705,7 @@ op_args_count(int operation)
 	switch (operation) {
 	case OP_CLONE_RANGE:
 	case OP_DEDUPE_RANGE:
+	case OP_COPY_RANGE:
 		return 4;
 	default:
 		return 3;
@@ -1795,6 +1881,14 @@ test(void)
 				 offset2 + size > file_size);
 			break;
 		}
+	case OP_COPY_RANGE:
+		TRIM_OFF_LEN(offset, size, file_size);
+		do {
+			offset2 = random();
+			TRIM_OFF(offset2, maxfilelen);
+		} while (llabs(offset2 - offset) < size ||
+			 offset2 + size > maxfilelen);
+		break;
 	}
 
 have_op:
@@ -1850,6 +1944,12 @@ test(void)
 			goto out;
 		}
 		break;
+	case OP_COPY_RANGE:
+		if (!copy_range_calls) {
+			log5(op, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		break;
 	}
 
 	switch (op) {
@@ -1940,6 +2040,18 @@ test(void)
 
 		do_dedupe_range(offset, size, offset2);
 		break;
+	case OP_COPY_RANGE:
+		if (size == 0) {
+			log5(OP_COPY_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		if (offset2 + size > maxfilelen) {
+			log5(OP_COPY_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+
+		do_copy_range(offset, size, offset2);
+		break;
 	case OP_FSYNC:
 		dofsync();
 		break;
@@ -1965,7 +2077,7 @@ void
 usage(void)
 {
 	fprintf(stdout, "usage: %s",
-		"fsx [-dknqxABFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+		"fsx [-dknqxABEFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
 	-b opnum: beginning operation number (default 1)\n\
 	-c P: 1 in P chance of file close+open at each op (default infinity)\n\
 	-d: debug output for all operations\n\
@@ -2012,6 +2124,9 @@ usage(void)
 #ifdef FIDEDUPERANGE
 "	-B: Do not use dedupe range calls\n"
 #endif
+#ifdef HAVE_COPY_FILE_RANGE
+"	-E: Do not use copy range calls\n"
+#endif
 "	-L: fsxLite - no file creations & no file size changes\n\
 	-N numops: total # operations to do (default infinity)\n\
 	-O: use oplen (see -o flag) for every op (default random)\n\
@@ -2215,7 +2330,7 @@ main(int argc, char **argv)
 	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
 
 	while ((ch = getopt_long(argc, argv,
-				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:FJKHzCILN:OP:RS:WXZ",
+				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:WXZ",
 				 longopts, NULL)) != EOF)
 		switch (ch) {
 		case 'b':
@@ -2351,6 +2466,9 @@ main(int argc, char **argv)
 		case 'B':
 			dedupe_range_calls = 0;
 			break;
+		case 'E':
+			copy_range_calls = 0;
+			break;
 		case 'L':
 		        lite = 1;
 			o_flags &= ~(O_CREAT|O_TRUNC);

[PATCH 09/10] fsx: clean up copy/dedupe file range support.

[Darrick J. Wong]

From: Dave Chinner <dchinner@redhat.com>

copy_file_range() needs to obey read/write constraints otherwise is
blows up when direct IO is used.

FIDEDUPERANGE has a completely screwed up API for error reporting.
The ioctl succeeds even if dedupe fails, so you have to check
every individual dedupe operations for failure. Without this, dedupe
"succeeds" on kernels filesystems that don't even support dedupe...

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |   11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index ec3defad..cea765d8 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -1452,7 +1452,11 @@ do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
 	fdr->info[0].dest_fd = fd;
 	fdr->info[0].dest_offset = dest;
 
-	if (ioctl(fd, FIDEDUPERANGE, fdr) == -1) {
+	if (ioctl(fd, FIDEDUPERANGE, fdr) == -1 ||
+	    fdr->info[0].status < 0) {
+		if (fdr->info[0].status < 0)
+			errno = -fdr->info[0].status;
+
 		if (errno == EOPNOTSUPP || errno == ENOTTY) {
 			if (!quiet && testcalls > simulatedopcount)
 				prt("skipping unsupported dedupe range\n");
@@ -1486,6 +1490,11 @@ do_copy_range(unsigned offset, unsigned length, unsigned dest)
 	loff_t o1, o2;
 	ssize_t nr;
 
+	offset -= offset % readbdy;
+	dest -= dest % writebdy;
+	if (o_direct)
+		length -= length % readbdy;
+
 	if (length == 0) {
 		if (!quiet && testcalls > simulatedopcount)
 			prt("skipping zero length copy range\n");

[PATCH 10/10] common/dump: disable copyrange

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

In _create_dumpdir_stress_num, disable copyrange in fsstress so that we
dump exactly the same set of files and directories no matter how the xfs
is configured.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/dump |    3 +++
 1 file changed, 3 insertions(+)


diff --git a/common/dump b/common/dump
index c45b8a5a..4d1a1607 100644
--- a/common/dump
+++ b/common/dump
@@ -307,6 +307,9 @@ _create_dumpdir_stress_num()
     if $FSSTRESS_PROG | grep -q deduperange; then
         FSSTRESS_AVOID="-f deduperange=0 $FSSTRESS_AVOID"
     fi
+    if $FSSTRESS_PROG | grep -q copyrange; then
+        FSSTRESS_AVOID="-f copyrange=0 $FSSTRESS_AVOID"
+    fi
 
     echo ""
     echo "-----------------------------------------------"

[PATCH 11/10] generic: long fsx soak tests

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add a few tests to implement long soak tests of fsx.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/generic/938     |   65 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/938.out |    2 ++
 tests/generic/939     |   60 +++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/939.out |    2 ++
 tests/generic/group   |    2 ++
 5 files changed, 131 insertions(+)
 create mode 100644 tests/generic/938
 create mode 100644 tests/generic/938.out
 create mode 100644 tests/generic/939
 create mode 100644 tests/generic/939.out

diff --git a/tests/generic/938 b/tests/generic/938
new file mode 100644
index 00000000..a85a6472
--- /dev/null
+++ b/tests/generic/938
@@ -0,0 +1,65 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2018 Oracle.  All Rights Reserved.
+#
+# FS QA Test 938
+#
+# Long-soak directio fsx test
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1        # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+        cd /
+        rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+
+# Run fsx for a million ops or more
+nr_ops=$((1000000 * TIME_FACTOR))
+op_sz=$((128000 * LOAD_FACTOR))
+file_sz=$((600000 * LOAD_FACTOR))
+fsx_file=$TEST_DIR/fsx.$seq
+min_dio_sz=$(_min_dio_alignment)
+
+fsx_args=(-q)
+fsx_args+=(-N $nr_ops)
+fsx_args+=(-p $((nr_ops / 100)))
+fsx_args+=(-o $op_sz)
+fsx_args+=(-l $file_sz)
+fsx_args+=(-r $min_dio_sz)
+fsx_args+=(-t $min_dio_sz)
+fsx_args+=(-w $min_dio_sz)
+fsx_args+=(-Z)
+fsx_args+=($fsx_file)
+
+echo "ltp ${fsx_args[@]} $FSX_ARGS" >> $seqres.full
+
+if ! ltp/fsx "${fsx_args[@]}" $FSX_ARGS > $tmp.fsx 2>&1; then
+	cat $tmp.fsx | tee -a $seqres.full
+fi
+
+# success, all done
+echo Silence is golden
+status=0
+exit
diff --git a/tests/generic/938.out b/tests/generic/938.out
new file mode 100644
index 00000000..11b565ee
--- /dev/null
+++ b/tests/generic/938.out
@@ -0,0 +1,2 @@
+QA output created by 938
+Silence is golden
diff --git a/tests/generic/939 b/tests/generic/939
new file mode 100644
index 00000000..5e341e30
--- /dev/null
+++ b/tests/generic/939
@@ -0,0 +1,60 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2018 Oracle.  All Rights Reserved.
+#
+# FS QA Test 938
+#
+# Long-soak buffered fsx test
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1        # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+        cd /
+        rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+
+# Run fsx for a million ops or more
+nr_ops=$((1000000 * TIME_FACTOR))
+op_sz=$((128000 * LOAD_FACTOR))
+file_sz=$((600000 * LOAD_FACTOR))
+fsx_file=$TEST_DIR/fsx.$seq
+
+fsx_args=(-q)
+fsx_args+=(-N $nr_ops)
+fsx_args+=(-p $((nr_ops / 100)))
+fsx_args+=(-o $op_sz)
+fsx_args+=(-l $file_sz)
+fsx_args+=($fsx_file)
+
+echo "ltp ${fsx_args[@]} $FSX_ARGS" >> $seqres.full
+
+if ! ltp/fsx "${fsx_args[@]}" $FSX_ARGS > $tmp.fsx 2>&1; then
+	cat $tmp.fsx | tee -a $seqres.full
+fi
+
+# success, all done
+echo Silence is golden
+status=0
+exit
diff --git a/tests/generic/939.out b/tests/generic/939.out
new file mode 100644
index 00000000..a86e90c5
--- /dev/null
+++ b/tests/generic/939.out
@@ -0,0 +1,2 @@
+QA output created by 939
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 92330b52..16e5cc17 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -522,3 +522,5 @@
 517 auto quick dedupe clone
 518 auto quick clone
 519 auto quick
+938 soak long_rw
+939 soak long_rw

Re: [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress

[Luis Henriques]

"Darrick J. Wong" <darrick.wong@oracle.com> writes:

> Hi all,
>
> This series adds to fsx and fsstress support for FICLONERANGE,
> FIDEDUPERANGE, and copy_file_range.
>
> First, I fix some gcc warnings in fsx.
>
> Then, I teach fsx to read the fsx file after every operation to compare
> it to the good buffer.  This made it easier for me to find corruption
> problem as soon as they happen, though I'm not sure it really makes
> sense to have this enabled by default because of the behavior change
> that it makes.
>
> Next come a couple of generic reworks to fsx that we need to support the
> new clone/dedupe/copy commands.
>
> Patches 5-6 add clone and dedupe to fsx.
>
> Patches 7-8 add copy_file_range support to fsstress and fsx.

An annoying side-effect of these changes is that I now see a couple of
new generic tests failing on cephfs (for example, generic/075).  That's
because fsx does use copy_file_range with the same fd both as source and
destination.  We currently return -EINVAL in that case (as nfs and cifs
seem to be doing as well btw).

At least for the cephfs, this check could eventually be changed but I
would need to spend some time trying and testing the effects of
offloading object copies on the same file.  Of course that another
(easy!) option would be to simply return -EOPNOTSUPP instead and
fallback to the VFS implementation.

Cheers,
-- 
Luis

>
> Dave Chinner contributed some cleanups to the fsx patches as the 9th
> patch.
>
> The last patch fixes the common/dump tests to disable the new commands
> so that the dump/restore tests continue to function exactly as they have
> for years.
>
> There are known failures in 4.20-rc3, particularly with copy_file_range,
> which hopefully have been fixed by the patch series that Dave Chinner
> posted to the xfs list yesterday.  Branch can be downloaded here[1].
>
> --D
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fsstress-clone

Re: [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress

[Darrick J. Wong]

On Mon, Dec 10, 2018 at 05:30:54PM +0000, Luis Henriques wrote:
> "Darrick J. Wong" <darrick.wong@oracle.com> writes:
> 
> > Hi all,
> >
> > This series adds to fsx and fsstress support for FICLONERANGE,
> > FIDEDUPERANGE, and copy_file_range.
> >
> > First, I fix some gcc warnings in fsx.
> >
> > Then, I teach fsx to read the fsx file after every operation to compare
> > it to the good buffer.  This made it easier for me to find corruption
> > problem as soon as they happen, though I'm not sure it really makes
> > sense to have this enabled by default because of the behavior change
> > that it makes.
> >
> > Next come a couple of generic reworks to fsx that we need to support the
> > new clone/dedupe/copy commands.
> >
> > Patches 5-6 add clone and dedupe to fsx.
> >
> > Patches 7-8 add copy_file_range support to fsstress and fsx.
> 
> An annoying side-effect of these changes is that I now see a couple of
> new generic tests failing on cephfs (for example, generic/075).  That's
> because fsx does use copy_file_range with the same fd both as source and
> destination.  We currently return -EINVAL in that case (as nfs and cifs
> seem to be doing as well btw).
> 
> At least for the cephfs, this check could eventually be changed but I
> would need to spend some time trying and testing the effects of
> offloading object copies on the same file.  Of course that another
> (easy!) option would be to simply return -EOPNOTSUPP instead and
> fallback to the VFS implementation.

I think Dave Chinner's copy_file_range cleanup series fixes that, among
other problems.

--D

> Cheers,
> -- 
> Luis
> 
> >
> > Dave Chinner contributed some cleanups to the fsx patches as the 9th
> > patch.
> >
> > The last patch fixes the common/dump tests to disable the new commands
> > so that the dump/restore tests continue to function exactly as they have
> > for years.
> >
> > There are known failures in 4.20-rc3, particularly with copy_file_range,
> > which hopefully have been fixed by the patch series that Dave Chinner
> > posted to the xfs list yesterday.  Branch can be downloaded here[1].
> >
> > --D
> >
> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fsstress-clone

Re: [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress

[Luis Henriques]

"Darrick J. Wong" <darrick.wong@oracle.com> writes:

> On Mon, Dec 10, 2018 at 05:30:54PM +0000, Luis Henriques wrote:
>> "Darrick J. Wong" <darrick.wong@oracle.com> writes:
>> 
>> > Hi all,
>> >
>> > This series adds to fsx and fsstress support for FICLONERANGE,
>> > FIDEDUPERANGE, and copy_file_range.
>> >
>> > First, I fix some gcc warnings in fsx.
>> >
>> > Then, I teach fsx to read the fsx file after every operation to compare
>> > it to the good buffer.  This made it easier for me to find corruption
>> > problem as soon as they happen, though I'm not sure it really makes
>> > sense to have this enabled by default because of the behavior change
>> > that it makes.
>> >
>> > Next come a couple of generic reworks to fsx that we need to support the
>> > new clone/dedupe/copy commands.
>> >
>> > Patches 5-6 add clone and dedupe to fsx.
>> >
>> > Patches 7-8 add copy_file_range support to fsstress and fsx.
>> 
>> An annoying side-effect of these changes is that I now see a couple of
>> new generic tests failing on cephfs (for example, generic/075).  That's
>> because fsx does use copy_file_range with the same fd both as source and
>> destination.  We currently return -EINVAL in that case (as nfs and cifs
>> seem to be doing as well btw).
>> 
>> At least for the cephfs, this check could eventually be changed but I
>> would need to spend some time trying and testing the effects of
>> offloading object copies on the same file.  Of course that another
>> (easy!) option would be to simply return -EOPNOTSUPP instead and
>> fallback to the VFS implementation.
>
> I think Dave Chinner's copy_file_range cleanup series fixes that, among
> other problems.

No, unfortunately it doesn't because it's not really something that
would make sense to do at the VFS level.  CephFS currently does not
allow to offload a copy where the source and destination are the same
file so it simply returns -EINVAL.

But that could probably be changed with Chinner's patchset, because it
adds some extra checks that make it safe to simply return -EOPNOTSUPP
instead, and fallback to VFS implementation.  I'll follow-up on that in
the copy_file_range thread.

(And sorry for replying to the wrong email thread, I should have sent my
email to the latest version that actually got merged into the fstests.)

Cheers,
-- 
Luis

>> >
>> > Dave Chinner contributed some cleanups to the fsx patches as the 9th
>> > patch.
>> >
>> > The last patch fixes the common/dump tests to disable the new commands
>> > so that the dump/restore tests continue to function exactly as they have
>> > for years.
>> >
>> > There are known failures in 4.20-rc3, particularly with copy_file_range,
>> > which hopefully have been fixed by the patch series that Dave Chinner
>> > posted to the xfs list yesterday.  Branch can be downloaded here[1].
>> >
>> > --D
>> >
>> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fsstress-clone
>