[PATCH v3 00/12] xfstests: add copy/dedupe/clone to fsx/fsstress

[PATCH v3 00/12] xfstests: add copy/dedupe/clone to fsx/fsstress

[Darrick J. Wong]

Hi all,

This series adds to fsx and fsstress support for FICLONERANGE,
FIDEDUPERANGE, and copy_file_range.

The first three patches fix gcc warnings in fsx and fsstress.

Patch 4 adds an -X option to fsx that causes it to re-read the fsx file
after every operation to compare it to the good buffer.  This made it
easier for me to find corruption problem as soon as they happen, though
it slows down fsx and perturbs in-core state considerably.

Patche 5-6 are a couple of generic reworks to fsx that we need to
support the new clone/dedupe/copy commands.

Patches 7-8 add clone and dedupe to fsx.

Patches 9-10 add copy_file_range support to fsstress and fsx.

Patch 11 fixes the common/dump tests to disable the new commands so that
the dump/restore tests continue to function exactly as they have for
years.

Patch 12 contains long-soak fsx tests designed to draw out corruption
problems by running (1 million * TIME_FACTOR) operations.  These will be
running all weekend during the US Thanksgiving holiday, though I hope
we've sorted out all the problems.

There are known failures in XFS on 4.20-rc3, particularly with
copy_file_range and reflink, all of which hopefully have been fixed by
the patch series that Dave Chinner posted to the xfs list yesterday.
A branch with the fstests patches can be downloaded here[1].

--D

[1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fsstress-clone

[PATCH 01/12] fsstress: fix compiler warnings

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Fix all the gcc warnings in fsstress.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsstress.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


diff --git a/ltp/fsstress.c b/ltp/fsstress.c
index edbb3f2f..8d0734fc 100644
--- a/ltp/fsstress.c
+++ b/ltp/fsstress.c
@@ -499,7 +499,7 @@ int main(int argc, char **argv)
 		exit(1);
 	}
 	if (logname) {
-		char path[PATH_MAX];
+		char path[PATH_MAX + NAME_MAX + 1];
 		snprintf(path, sizeof(path), "%s/%s", rpath, logname);
 		if (freopen(path, "a", stdout) == NULL) {
 			perror("freopen logfile failed");
@@ -583,7 +583,7 @@ int main(int argc, char **argv)
 				return 0;
 #endif
 			if (logname) {
-				char path[PATH_MAX];
+				char path[PATH_MAX + NAME_MAX + 2 + 11];
 				snprintf(path, sizeof(path), "%s/%s.%d",
 					 rpath, logname, i);
 				if (freopen(path, "a", stdout) == NULL) {

Re: [PATCH 01/12] fsstress: fix compiler warnings

[Allison Henderson]

On 11/22/18 10:58 AM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Fix all the gcc warnings in fsstress.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>   ltp/fsstress.c |    4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> 
> diff --git a/ltp/fsstress.c b/ltp/fsstress.c
> index edbb3f2f..8d0734fc 100644
> --- a/ltp/fsstress.c
> +++ b/ltp/fsstress.c
> @@ -499,7 +499,7 @@ int main(int argc, char **argv)
>   		exit(1);
>   	}
>   	if (logname) {
> -		char path[PATH_MAX];
> +		char path[PATH_MAX + NAME_MAX + 1];
>   		snprintf(path, sizeof(path), "%s/%s", rpath, logname);
>   		if (freopen(path, "a", stdout) == NULL) {
>   			perror("freopen logfile failed");
> @@ -583,7 +583,7 @@ int main(int argc, char **argv)
>   				return 0;
>   #endif
>   			if (logname) {
> -				char path[PATH_MAX];
> +				char path[PATH_MAX + NAME_MAX + 2 + 11];
>   				snprintf(path, sizeof(path), "%s/%s.%d",
>   					 rpath, logname, i);
>   				if (freopen(path, "a", stdout) == NULL) {
> 

Looks ok.  I'm assuming the 2 and the 11 are for the slash, dot and 
logname?  Maybe just a comment or two in the commit.  Otherwise looks 
good to me, you can add my review.  Thx!  :-)

Reviewed-By: Allison Henderson <allison.henderson@oracle.com>

Allison

[PATCH 02/12] fsstress: check system call return values

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Check the return values of various system calls and blow up if something
went wrong.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsstress.c |   56 ++++++++++++++++++++++++++++++--------------------------
 1 file changed, 30 insertions(+), 26 deletions(-)


diff --git a/ltp/fsstress.c b/ltp/fsstress.c
index 8d0734fc..af5d125f 100644
--- a/ltp/fsstress.c
+++ b/ltp/fsstress.c
@@ -733,7 +733,7 @@ append_pathname(pathname_t *name, char *str)
 	/* attempting to append to a dir a zero length path */
 	if (len && *str == '/' && name->len == 0) {
 		fprintf(stderr, "fsstress: append_pathname failure\n");
-		chdir(homedir);
+		assert(chdir(homedir) == 0);
 		abort();
 		/* NOTREACHED */
 	}
@@ -765,7 +765,7 @@ attr_list_path(pathname_t *name,
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = attr_list_path(&newname, buffer, buffersize, flags, cursor);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -784,7 +784,7 @@ attr_remove_path(pathname_t *name, const char *attrname, int flags)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = attr_remove_path(&newname, attrname, flags);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -805,7 +805,7 @@ attr_set_path(pathname_t *name, const char *attrname, const char *attrvalue,
 	if (chdir(buf) == 0) {
 		rval = attr_set_path(&newname, attrname, attrvalue, valuelength,
 			flags);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -819,7 +819,7 @@ check_cwd(void)
 
 	if (stat64(".", &statbuf) == 0 && statbuf.st_ino == top_ino)
 		return;
-	chdir(homedir);
+	assert(chdir(homedir) == 0);
 	fprintf(stderr, "fsstress: check_cwd failure\n");
 	abort();
 	/* NOTREACHED */
@@ -858,7 +858,7 @@ creat_path(pathname_t *name, mode_t mode)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = creat_path(&newname, mode);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -996,11 +996,15 @@ doproc(void)
 		}
 	}
 errout:
-	chdir("..");
+	assert(chdir("..") == 0);
 	free(homedir);
 	if (cleanup) {
+		int ret;
+
 		sprintf(cmd, "rm -rf %s", buf);
-		system(cmd);
+		ret = system(cmd);
+		if (ret != 0)
+			perror("cleaning up");
 		cleanup_flist();
 	}
 }
@@ -1216,7 +1220,7 @@ lchown_path(pathname_t *name, uid_t owner, gid_t group)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = lchown_path(&newname, owner, group);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1240,7 +1244,7 @@ link_path(pathname_t *name1, pathname_t *name2)
 	if (strcmp(buf1, buf2) == 0) {
 		if (chdir(buf1) == 0) {
 			rval = link_path(&newname1, &newname2);
-			chdir("..");
+			assert(chdir("..") == 0);
 		}
 	} else {
 		if (strcmp(buf1, "..") == 0)
@@ -1260,7 +1264,7 @@ link_path(pathname_t *name1, pathname_t *name2)
 			append_pathname(&newname2, name2->path);
 			if (chdir(buf1) == 0) {
 				rval = link_path(&newname1, &newname2);
-				chdir("..");
+				assert(chdir("..") == 0);
 			}
 		} else {
 			free_pathname(&newname1);
@@ -1268,7 +1272,7 @@ link_path(pathname_t *name1, pathname_t *name2)
 			append_pathname(&newname1, name1->path);
 			if (chdir(buf2) == 0) {
 				rval = link_path(&newname1, &newname2);
-				chdir("..");
+				assert(chdir("..") == 0);
 			}
 		}
 	}
@@ -1290,7 +1294,7 @@ lstat64_path(pathname_t *name, struct stat64 *sbuf)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = lstat64_path(&newname, sbuf);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1326,7 +1330,7 @@ mkdir_path(pathname_t *name, mode_t mode)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = mkdir_path(&newname, mode);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1345,7 +1349,7 @@ mknod_path(pathname_t *name, mode_t mode, dev_t dev)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = mknod_path(&newname, mode, dev);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1386,7 +1390,7 @@ open_path(pathname_t *name, int oflag)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = open_path(&newname, oflag);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1405,7 +1409,7 @@ opendir_path(pathname_t *name)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = opendir_path(&newname);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1446,7 +1450,7 @@ readlink_path(pathname_t *name, char *lbuf, size_t lbufsiz)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = readlink_path(&newname, lbuf, lbufsiz);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1470,7 +1474,7 @@ rename_path(pathname_t *name1, pathname_t *name2)
 	if (strcmp(buf1, buf2) == 0) {
 		if (chdir(buf1) == 0) {
 			rval = rename_path(&newname1, &newname2);
-			chdir("..");
+			assert(chdir("..") == 0);
 		}
 	} else {
 		if (strcmp(buf1, "..") == 0)
@@ -1490,7 +1494,7 @@ rename_path(pathname_t *name1, pathname_t *name2)
 			append_pathname(&newname2, name2->path);
 			if (chdir(buf1) == 0) {
 				rval = rename_path(&newname1, &newname2);
-				chdir("..");
+				assert(chdir("..") == 0);
 			}
 		} else {
 			free_pathname(&newname1);
@@ -1498,7 +1502,7 @@ rename_path(pathname_t *name1, pathname_t *name2)
 			append_pathname(&newname1, name1->path);
 			if (chdir(buf2) == 0) {
 				rval = rename_path(&newname1, &newname2);
-				chdir("..");
+				assert(chdir("..") == 0);
 			}
 		}
 	}
@@ -1520,7 +1524,7 @@ rmdir_path(pathname_t *name)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = rmdir_path(&newname);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1603,7 +1607,7 @@ stat64_path(pathname_t *name, struct stat64 *sbuf)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = stat64_path(&newname, sbuf);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1627,7 +1631,7 @@ symlink_path(const char *name1, pathname_t *name)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = symlink_path(name1, &newname);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1646,7 +1650,7 @@ truncate64_path(pathname_t *name, off64_t length)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = truncate64_path(&newname, length);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;
@@ -1665,7 +1669,7 @@ unlink_path(pathname_t *name)
 	separate_pathname(name, buf, &newname);
 	if (chdir(buf) == 0) {
 		rval = unlink_path(&newname);
-		chdir("..");
+		assert(chdir("..") == 0);
 	}
 	free_pathname(&newname);
 	return rval;

Re: [PATCH 02/12] fsstress: check system call return values

[Allison Henderson]

Looks ok, you can add my review:

Reviewed-By: Allison Henderson <allison.henderson@oracle.com>

Allison

On 11/22/18 10:58 AM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Check the return values of various system calls and blow up if something
> went wrong.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>   ltp/fsstress.c |   56 ++++++++++++++++++++++++++++++--------------------------
>   1 file changed, 30 insertions(+), 26 deletions(-)
> 
> 
> diff --git a/ltp/fsstress.c b/ltp/fsstress.c
> index 8d0734fc..af5d125f 100644
> --- a/ltp/fsstress.c
> +++ b/ltp/fsstress.c
> @@ -733,7 +733,7 @@ append_pathname(pathname_t *name, char *str)
>   	/* attempting to append to a dir a zero length path */
>   	if (len && *str == '/' && name->len == 0) {
>   		fprintf(stderr, "fsstress: append_pathname failure\n");
> -		chdir(homedir);
> +		assert(chdir(homedir) == 0);
>   		abort();
>   		/* NOTREACHED */
>   	}
> @@ -765,7 +765,7 @@ attr_list_path(pathname_t *name,
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = attr_list_path(&newname, buffer, buffersize, flags, cursor);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -784,7 +784,7 @@ attr_remove_path(pathname_t *name, const char *attrname, int flags)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = attr_remove_path(&newname, attrname, flags);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -805,7 +805,7 @@ attr_set_path(pathname_t *name, const char *attrname, const char *attrvalue,
>   	if (chdir(buf) == 0) {
>   		rval = attr_set_path(&newname, attrname, attrvalue, valuelength,
>   			flags);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -819,7 +819,7 @@ check_cwd(void)
>   
>   	if (stat64(".", &statbuf) == 0 && statbuf.st_ino == top_ino)
>   		return;
> -	chdir(homedir);
> +	assert(chdir(homedir) == 0);
>   	fprintf(stderr, "fsstress: check_cwd failure\n");
>   	abort();
>   	/* NOTREACHED */
> @@ -858,7 +858,7 @@ creat_path(pathname_t *name, mode_t mode)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = creat_path(&newname, mode);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -996,11 +996,15 @@ doproc(void)
>   		}
>   	}
>   errout:
> -	chdir("..");
> +	assert(chdir("..") == 0);
>   	free(homedir);
>   	if (cleanup) {
> +		int ret;
> +
>   		sprintf(cmd, "rm -rf %s", buf);
> -		system(cmd);
> +		ret = system(cmd);
> +		if (ret != 0)
> +			perror("cleaning up");
>   		cleanup_flist();
>   	}
>   }
> @@ -1216,7 +1220,7 @@ lchown_path(pathname_t *name, uid_t owner, gid_t group)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = lchown_path(&newname, owner, group);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1240,7 +1244,7 @@ link_path(pathname_t *name1, pathname_t *name2)
>   	if (strcmp(buf1, buf2) == 0) {
>   		if (chdir(buf1) == 0) {
>   			rval = link_path(&newname1, &newname2);
> -			chdir("..");
> +			assert(chdir("..") == 0);
>   		}
>   	} else {
>   		if (strcmp(buf1, "..") == 0)
> @@ -1260,7 +1264,7 @@ link_path(pathname_t *name1, pathname_t *name2)
>   			append_pathname(&newname2, name2->path);
>   			if (chdir(buf1) == 0) {
>   				rval = link_path(&newname1, &newname2);
> -				chdir("..");
> +				assert(chdir("..") == 0);
>   			}
>   		} else {
>   			free_pathname(&newname1);
> @@ -1268,7 +1272,7 @@ link_path(pathname_t *name1, pathname_t *name2)
>   			append_pathname(&newname1, name1->path);
>   			if (chdir(buf2) == 0) {
>   				rval = link_path(&newname1, &newname2);
> -				chdir("..");
> +				assert(chdir("..") == 0);
>   			}
>   		}
>   	}
> @@ -1290,7 +1294,7 @@ lstat64_path(pathname_t *name, struct stat64 *sbuf)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = lstat64_path(&newname, sbuf);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1326,7 +1330,7 @@ mkdir_path(pathname_t *name, mode_t mode)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = mkdir_path(&newname, mode);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1345,7 +1349,7 @@ mknod_path(pathname_t *name, mode_t mode, dev_t dev)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = mknod_path(&newname, mode, dev);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1386,7 +1390,7 @@ open_path(pathname_t *name, int oflag)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = open_path(&newname, oflag);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1405,7 +1409,7 @@ opendir_path(pathname_t *name)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = opendir_path(&newname);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1446,7 +1450,7 @@ readlink_path(pathname_t *name, char *lbuf, size_t lbufsiz)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = readlink_path(&newname, lbuf, lbufsiz);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1470,7 +1474,7 @@ rename_path(pathname_t *name1, pathname_t *name2)
>   	if (strcmp(buf1, buf2) == 0) {
>   		if (chdir(buf1) == 0) {
>   			rval = rename_path(&newname1, &newname2);
> -			chdir("..");
> +			assert(chdir("..") == 0);
>   		}
>   	} else {
>   		if (strcmp(buf1, "..") == 0)
> @@ -1490,7 +1494,7 @@ rename_path(pathname_t *name1, pathname_t *name2)
>   			append_pathname(&newname2, name2->path);
>   			if (chdir(buf1) == 0) {
>   				rval = rename_path(&newname1, &newname2);
> -				chdir("..");
> +				assert(chdir("..") == 0);
>   			}
>   		} else {
>   			free_pathname(&newname1);
> @@ -1498,7 +1502,7 @@ rename_path(pathname_t *name1, pathname_t *name2)
>   			append_pathname(&newname1, name1->path);
>   			if (chdir(buf2) == 0) {
>   				rval = rename_path(&newname1, &newname2);
> -				chdir("..");
> +				assert(chdir("..") == 0);
>   			}
>   		}
>   	}
> @@ -1520,7 +1524,7 @@ rmdir_path(pathname_t *name)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = rmdir_path(&newname);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1603,7 +1607,7 @@ stat64_path(pathname_t *name, struct stat64 *sbuf)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = stat64_path(&newname, sbuf);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1627,7 +1631,7 @@ symlink_path(const char *name1, pathname_t *name)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = symlink_path(name1, &newname);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1646,7 +1650,7 @@ truncate64_path(pathname_t *name, off64_t length)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = truncate64_path(&newname, length);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> @@ -1665,7 +1669,7 @@ unlink_path(pathname_t *name)
>   	separate_pathname(name, buf, &newname);
>   	if (chdir(buf) == 0) {
>   		rval = unlink_path(&newname);
> -		chdir("..");
> +		assert(chdir("..") == 0);
>   	}
>   	free_pathname(&newname);
>   	return rval;
> 

[PATCH 03/12] fsx: shut up compiler warnings

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Fix unused variables and potential filename overflows in fsx.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index b0157ba3..5601c70c 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -121,7 +121,7 @@ char	*bname;				/* basename of our test file */
 char	*logdev;			/* -i flag */
 char	*logid;				/* -j flag */
 char	dname[1024];			/* -P flag */
-char	goodfile[1024];
+char	goodfile[PATH_MAX];
 int	dirpath = 0;			/* -P flag */
 int	fd;				/* fd for our test file */
 
@@ -185,7 +185,7 @@ const char *replayops = NULL;
 const char *recordops = NULL;
 FILE *	fsxlogf = NULL;
 FILE *	replayopsf = NULL;
-char opsfile[1024];
+char opsfile[PATH_MAX];
 int badoff = -1;
 int closeopen = 0;
 
@@ -541,13 +541,13 @@ mark_log(void)
 void
 dump_fsync_buffer(void)
 {
-	char fname_buffer[1024];
+	char fname_buffer[PATH_MAX];
 	int good_fd;
 
 	if (!good_buf)
 		return;
 
-	snprintf(fname_buffer, 1024, "%s%s.mark%d", dname,
+	snprintf(fname_buffer, sizeof(fname_buffer), "%s%s.mark%d", dname,
 		 bname, mark_nr);
 	good_fd = open(fname_buffer, O_WRONLY|O_CREAT|O_TRUNC, 0666);
 	if (good_fd < 0) {
@@ -1865,7 +1865,7 @@ main(int argc, char **argv)
 {
 	int	i, style, ch;
 	char	*endp, *tmp;
-	char logfile[1024];
+	char logfile[PATH_MAX];
 	struct stat statbuf;
 	int o_flags = O_RDWR|O_CREAT|O_TRUNC;
 

Re: [PATCH 03/12] fsx: shut up compiler warnings

[Allison Henderson]

Looks good:

Reviewed-By: Allison Henderson <allison.henderson@oracle.com>

Allison

On 11/22/18 10:58 AM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Fix unused variables and potential filename overflows in fsx.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>   ltp/fsx.c |   10 +++++-----
>   1 file changed, 5 insertions(+), 5 deletions(-)
> 
> 
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index b0157ba3..5601c70c 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -121,7 +121,7 @@ char	*bname;				/* basename of our test file */
>   char	*logdev;			/* -i flag */
>   char	*logid;				/* -j flag */
>   char	dname[1024];			/* -P flag */
> -char	goodfile[1024];
> +char	goodfile[PATH_MAX];
>   int	dirpath = 0;			/* -P flag */
>   int	fd;				/* fd for our test file */
>   
> @@ -185,7 +185,7 @@ const char *replayops = NULL;
>   const char *recordops = NULL;
>   FILE *	fsxlogf = NULL;
>   FILE *	replayopsf = NULL;
> -char opsfile[1024];
> +char opsfile[PATH_MAX];
>   int badoff = -1;
>   int closeopen = 0;
>   
> @@ -541,13 +541,13 @@ mark_log(void)
>   void
>   dump_fsync_buffer(void)
>   {
> -	char fname_buffer[1024];
> +	char fname_buffer[PATH_MAX];
>   	int good_fd;
>   
>   	if (!good_buf)
>   		return;
>   
> -	snprintf(fname_buffer, 1024, "%s%s.mark%d", dname,
> +	snprintf(fname_buffer, sizeof(fname_buffer), "%s%s.mark%d", dname,
>   		 bname, mark_nr);
>   	good_fd = open(fname_buffer, O_WRONLY|O_CREAT|O_TRUNC, 0666);
>   	if (good_fd < 0) {
> @@ -1865,7 +1865,7 @@ main(int argc, char **argv)
>   {
>   	int	i, style, ch;
>   	char	*endp, *tmp;
> -	char logfile[1024];
> +	char logfile[PATH_MAX];
>   	struct stat statbuf;
>   	int o_flags = O_RDWR|O_CREAT|O_TRUNC;
>   
> 

[PATCH 04/12] fsx: always check buffer after each operation

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add a new option to make fsx read the file after each operation and
compare it with the good buffer to try to catch corruptions as soon as
they occur.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |   65 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 56 insertions(+), 9 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index 5601c70c..7fb8b3ab 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -160,7 +160,8 @@ int     punch_hole_calls = 1;           /* -H flag disables */
 int     zero_range_calls = 1;           /* -z flag disables */
 int	collapse_range_calls = 1;	/* -C flag disables */
 int	insert_range_calls = 1;		/* -I flag disables */
-int 	mapped_reads = 1;		/* -R flag disables it */
+int	mapped_reads = 1;		/* -R flag disables it */
+int	check_file = 0;			/* -X flag enables */
 int	integrity = 0;			/* -i flag */
 int	fsxgoodfd = 0;
 int	o_direct;			/* -Z */
@@ -561,7 +562,7 @@ dump_fsync_buffer(void)
 }
 
 void
-check_buffers(unsigned offset, unsigned size)
+check_buffers(char *buf, unsigned offset, unsigned size)
 {
 	unsigned char c, t;
 	unsigned i = 0;
@@ -569,19 +570,19 @@ check_buffers(unsigned offset, unsigned size)
 	unsigned op = 0;
 	unsigned bad = 0;
 
-	if (memcmp(good_buf + offset, temp_buf, size) != 0) {
+	if (memcmp(good_buf + offset, buf, size) != 0) {
 		prt("READ BAD DATA: offset = 0x%x, size = 0x%x, fname = %s\n",
 		    offset, size, fname);
 		prt("OFFSET\tGOOD\tBAD\tRANGE\n");
 		while (size > 0) {
 			c = good_buf[offset];
-			t = temp_buf[i];
+			t = buf[i];
 			if (c != t) {
 			        if (n < 16) {
-					bad = short_at(&temp_buf[i]);
+					bad = short_at(&buf[i]);
 				        prt("0x%05x\t0x%04x\t0x%04x", offset,
 				            short_at(&good_buf[offset]), bad);
-					op = temp_buf[offset & 1 ? i+1 : i];
+					op = buf[offset & 1 ? i+1 : i];
 				        prt("\t0x%05x\n", n);
 					if (op)
 						prt("operation# (mod 256) for "
@@ -725,7 +726,46 @@ doread(unsigned offset, unsigned size)
 			    iret, size);
 		report_failure(141);
 	}
-	check_buffers(offset, size);
+	check_buffers(temp_buf, offset, size);
+}
+
+void
+check_contents(void)
+{
+	static char *check_buf;
+	unsigned offset = 0;
+	unsigned size = file_size;
+	off_t ret;
+	unsigned iret;
+
+	if (!check_buf) {
+		check_buf = (char *) malloc(maxfilelen + writebdy);
+		assert(check_buf != NULL);
+		check_buf = round_ptr_up(check_buf, writebdy, 0);
+		memset(check_buf, '\0', maxfilelen);
+	}
+
+	if (o_direct)
+		size -= size % readbdy;
+	if (size == 0)
+		return;
+
+	ret = lseek(fd, (off_t)offset, SEEK_SET);
+	if (ret == (off_t)-1) {
+		prterr("doread: lseek");
+		report_failure(140);
+	}
+
+	iret = fsxread(fd, check_buf, size, offset);
+	if (iret != size) {
+		if (iret == -1)
+			prterr("check_contents: read");
+		else
+			prt("short check read: 0x%x bytes instead of 0x%x\n",
+			    iret, size);
+		report_failure(141);
+	}
+	check_buffers(check_buf, offset, size);
 }
 
 
@@ -808,7 +848,7 @@ domapread(unsigned offset, unsigned size)
 		report_failure(191);
 	}
 
-	check_buffers(offset, size);
+	check_buffers(temp_buf, offset, size);
 }
 
 
@@ -1624,6 +1664,9 @@ test(void)
 		break;
 	}
 
+	if (check_file && testcalls > simulatedopcount)
+		check_contents();
+
 out:
 	if (sizechecks && testcalls > simulatedopcount)
 		check_size();
@@ -1684,6 +1727,7 @@ usage(void)
 	-P: save .fsxlog .fsxops and .fsxgood files in dirpath (default ./)\n\
 	-S seed: for random # generator (default 1) 0 gets timestamp\n\
 	-W: mapped write operations DISabled\n\
+	-X: Read file and compare to good buffer after every operation.\n\
         -R: read() system calls only (mapped reads disabled)\n\
         -Z: O_DIRECT (use -R, -W, -r and -w too)\n\
 	--replay-ops opsfile: replay ops from recorded .fsxops file\n\
@@ -1880,7 +1924,7 @@ main(int argc, char **argv)
 	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
 
 	while ((ch = getopt_long(argc, argv,
-				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
+				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
 				 longopts, NULL)) != EOF)
 		switch (ch) {
 		case 'b':
@@ -2044,6 +2088,9 @@ main(int argc, char **argv)
 			if (!quiet)
 				prt("mapped writes DISABLED\n");
 			break;
+		case 'X':
+			check_file = 1;
+			break;
 		case 'Z':
 			o_direct = O_DIRECT;
 			o_flags |= O_DIRECT;

Re: [PATCH 04/12] fsx: always check buffer after each operation

[Allison Henderson]

On 11/22/18 10:58 AM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Add a new option to make fsx read the file after each operation and
> compare it with the good buffer to try to catch corruptions as soon as
> they occur.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>   ltp/fsx.c |   65 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
>   1 file changed, 56 insertions(+), 9 deletions(-)
> 
> 
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index 5601c70c..7fb8b3ab 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -160,7 +160,8 @@ int     punch_hole_calls = 1;           /* -H flag disables */
>   int     zero_range_calls = 1;           /* -z flag disables */
>   int	collapse_range_calls = 1;	/* -C flag disables */
>   int	insert_range_calls = 1;		/* -I flag disables */
> -int 	mapped_reads = 1;		/* -R flag disables it */
> +int	mapped_reads = 1;		/* -R flag disables it */
> +int	check_file = 0;			/* -X flag enables */
>   int	integrity = 0;			/* -i flag */
>   int	fsxgoodfd = 0;
>   int	o_direct;			/* -Z */
> @@ -561,7 +562,7 @@ dump_fsync_buffer(void)
>   }
>   
>   void
> -check_buffers(unsigned offset, unsigned size)
> +check_buffers(char *buf, unsigned offset, unsigned size)
>   {
>   	unsigned char c, t;
>   	unsigned i = 0;
> @@ -569,19 +570,19 @@ check_buffers(unsigned offset, unsigned size)
>   	unsigned op = 0;
>   	unsigned bad = 0;
>   
> -	if (memcmp(good_buf + offset, temp_buf, size) != 0) {
> +	if (memcmp(good_buf + offset, buf, size) != 0) {
>   		prt("READ BAD DATA: offset = 0x%x, size = 0x%x, fname = %s\n",
>   		    offset, size, fname);
>   		prt("OFFSET\tGOOD\tBAD\tRANGE\n");
>   		while (size > 0) {
>   			c = good_buf[offset];
> -			t = temp_buf[i];
> +			t = buf[i];
>   			if (c != t) {
>   			        if (n < 16) {
> -					bad = short_at(&temp_buf[i]);
> +					bad = short_at(&buf[i]);
>   				        prt("0x%05x\t0x%04x\t0x%04x", offset,
>   				            short_at(&good_buf[offset]), bad);
> -					op = temp_buf[offset & 1 ? i+1 : i];
> +					op = buf[offset & 1 ? i+1 : i];
>   				        prt("\t0x%05x\n", n);
>   					if (op)
>   						prt("operation# (mod 256) for "
> @@ -725,7 +726,46 @@ doread(unsigned offset, unsigned size)
>   			    iret, size);
>   		report_failure(141);
>   	}
> -	check_buffers(offset, size);
> +	check_buffers(temp_buf, offset, size);
> +}
> +
> +void
> +check_contents(void)
> +{
> +	static char *check_buf;
> +	unsigned offset = 0;
> +	unsigned size = file_size;
> +	off_t ret;
> +	unsigned iret;
> +
> +	if (!check_buf) {

Just following the logic here:
Did you maybe mean to malloc check_buf before checking to see if it's 
null in the condition above?

Allison

> +		check_buf = (char *) malloc(maxfilelen + writebdy);
> +		assert(check_buf != NULL);
> +		check_buf = round_ptr_up(check_buf, writebdy, 0);
> +		memset(check_buf, '\0', maxfilelen);
> +	}
> +
> +	if (o_direct)
> +		size -= size % readbdy;
> +	if (size == 0)
> +		return;
> +
> +	ret = lseek(fd, (off_t)offset, SEEK_SET);
> +	if (ret == (off_t)-1) {
> +		prterr("doread: lseek");
> +		report_failure(140);
> +	}
> +
> +	iret = fsxread(fd, check_buf, size, offset);
> +	if (iret != size) {
> +		if (iret == -1)
> +			prterr("check_contents: read");
> +		else
> +			prt("short check read: 0x%x bytes instead of 0x%x\n",
> +			    iret, size);
> +		report_failure(141);
> +	}
> +	check_buffers(check_buf, offset, size);
>   }
>   
>   
> @@ -808,7 +848,7 @@ domapread(unsigned offset, unsigned size)
>   		report_failure(191);
>   	}
>   
> -	check_buffers(offset, size);
> +	check_buffers(temp_buf, offset, size);
>   }
>   
>   
> @@ -1624,6 +1664,9 @@ test(void)
>   		break;
>   	}
>   
> +	if (check_file && testcalls > simulatedopcount)
> +		check_contents();
> +
>   out:
>   	if (sizechecks && testcalls > simulatedopcount)
>   		check_size();
> @@ -1684,6 +1727,7 @@ usage(void)
>   	-P: save .fsxlog .fsxops and .fsxgood files in dirpath (default ./)\n\
>   	-S seed: for random # generator (default 1) 0 gets timestamp\n\
>   	-W: mapped write operations DISabled\n\
> +	-X: Read file and compare to good buffer after every operation.\n\
>           -R: read() system calls only (mapped reads disabled)\n\
>           -Z: O_DIRECT (use -R, -W, -r and -w too)\n\
>   	--replay-ops opsfile: replay ops from recorded .fsxops file\n\
> @@ -1880,7 +1924,7 @@ main(int argc, char **argv)
>   	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
>   
>   	while ((ch = getopt_long(argc, argv,
> -				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
> +				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
>   				 longopts, NULL)) != EOF)
>   		switch (ch) {
>   		case 'b':
> @@ -2044,6 +2088,9 @@ main(int argc, char **argv)
>   			if (!quiet)
>   				prt("mapped writes DISABLED\n");
>   			break;
> +		case 'X':
> +			check_file = 1;
> +			break;
>   		case 'Z':
>   			o_direct = O_DIRECT;
>   			o_flags |= O_DIRECT;
> 

[PATCH 05/12] fsx: use an enum to define the operation commands

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Use an enum to define operation codes and the boundaries between
operation classes so that we can add new commands without having to
change a bunch of unrelated #defines.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |   40 +++++++++++++++++++++-------------------
 1 file changed, 21 insertions(+), 19 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index 7fb8b3ab..dabad302 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -88,25 +88,27 @@ int			logcount = 0;	/* total ops */
  * mode being run.
  */
 
-/* common operations */
-#define	OP_READ		0
-#define OP_WRITE	1
-#define OP_MAPREAD	2
-#define OP_MAPWRITE	3
-#define OP_MAX_LITE	4
-
-/* !lite operations */
-#define OP_TRUNCATE		4
-#define OP_FALLOCATE		5
-#define OP_PUNCH_HOLE		6
-#define OP_ZERO_RANGE		7
-#define OP_COLLAPSE_RANGE	8
-#define OP_INSERT_RANGE	9
-#define OP_MAX_FULL		10
-
-/* integrity operations */
-#define OP_FSYNC		10
-#define OP_MAX_INTEGRITY	11
+enum {
+	/* common operations */
+	OP_READ = 0,
+	OP_WRITE,
+	OP_MAPREAD,
+	OP_MAPWRITE,
+	OP_MAX_LITE,
+
+	/* !lite operations */
+	OP_TRUNCATE = OP_MAX_LITE,
+	OP_FALLOCATE,
+	OP_PUNCH_HOLE,
+	OP_ZERO_RANGE,
+	OP_COLLAPSE_RANGE,
+	OP_INSERT_RANGE,
+	OP_MAX_FULL,
+
+	/* integrity operations */
+	OP_FSYNC = OP_MAX_FULL,
+	OP_MAX_INTEGRITY,
+};
 
 #undef PAGE_SIZE
 #define PAGE_SIZE       getpagesize()

Re: [PATCH 05/12] fsx: use an enum to define the operation commands

[Allison Henderson]

Looks ok to me

Reviewed-By: Allison Henderson <allison.henderson@oracle.com>

On 11/22/18 10:58 AM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Use an enum to define operation codes and the boundaries between
> operation classes so that we can add new commands without having to
> change a bunch of unrelated #defines.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>   ltp/fsx.c |   40 +++++++++++++++++++++-------------------
>   1 file changed, 21 insertions(+), 19 deletions(-)
> 
> 
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index 7fb8b3ab..dabad302 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -88,25 +88,27 @@ int			logcount = 0;	/* total ops */
>    * mode being run.
>    */
>   
> -/* common operations */
> -#define	OP_READ		0
> -#define OP_WRITE	1
> -#define OP_MAPREAD	2
> -#define OP_MAPWRITE	3
> -#define OP_MAX_LITE	4
> -
> -/* !lite operations */
> -#define OP_TRUNCATE		4
> -#define OP_FALLOCATE		5
> -#define OP_PUNCH_HOLE		6
> -#define OP_ZERO_RANGE		7
> -#define OP_COLLAPSE_RANGE	8
> -#define OP_INSERT_RANGE	9
> -#define OP_MAX_FULL		10
> -
> -/* integrity operations */
> -#define OP_FSYNC		10
> -#define OP_MAX_INTEGRITY	11
> +enum {
> +	/* common operations */
> +	OP_READ = 0,
> +	OP_WRITE,
> +	OP_MAPREAD,
> +	OP_MAPWRITE,
> +	OP_MAX_LITE,
> +
> +	/* !lite operations */
> +	OP_TRUNCATE = OP_MAX_LITE,
> +	OP_FALLOCATE,
> +	OP_PUNCH_HOLE,
> +	OP_ZERO_RANGE,
> +	OP_COLLAPSE_RANGE,
> +	OP_INSERT_RANGE,
> +	OP_MAX_FULL,
> +
> +	/* integrity operations */
> +	OP_FSYNC = OP_MAX_FULL,
> +	OP_MAX_INTEGRITY,
> +};
>   
>   #undef PAGE_SIZE
>   #define PAGE_SIZE       getpagesize()
> 

[PATCH 06/12] fsx: add five-argument logging function

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add a five-argument logging function to support new operations.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |   45 ++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 40 insertions(+), 5 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index dabad302..c525ba13 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -51,7 +51,8 @@ enum opflags { FL_NONE = 0, FL_SKIPPED = 1, FL_CLOSE_OPEN = 2, FL_KEEP_SIZE = 4
 
 struct log_entry {
 	int	operation;
-	int	args[3];
+	int	nr_args;
+	int	args[4];
 	enum opflags flags;
 };
 
@@ -278,6 +279,27 @@ static int op_code(const char *name)
 	return -1;
 }
 
+void
+log5(int operation, int arg0, int arg1, int arg2, enum opflags flags)
+{
+	struct log_entry *le;
+
+	le = &oplog[logptr];
+	le->operation = operation;
+	if (closeopen)
+		flags |= FL_CLOSE_OPEN;
+	le->args[0] = arg0;
+	le->args[1] = arg1;
+	le->args[2] = arg2;
+	le->args[3] = file_size;
+	le->nr_args = 4;
+	le->flags = flags;
+	logptr++;
+	logcount++;
+	if (logptr >= LOGSIZE)
+		logptr = 0;
+}
+
 void
 log4(int operation, int arg0, int arg1, enum opflags flags)
 {
@@ -290,6 +312,7 @@ log4(int operation, int arg0, int arg1, enum opflags flags)
 	le->args[0] = arg0;
 	le->args[1] = arg1;
 	le->args[2] = file_size;
+	le->nr_args = 3;
 	le->flags = flags;
 	logptr++;
 	logcount++;
@@ -439,11 +462,13 @@ logdump(void)
 			i = 0;
 
 		if (logopsf) {
+			int j;
+
 			if (lp->flags & FL_SKIPPED)
 				fprintf(logopsf, "skip ");
-			fprintf(logopsf, "%s 0x%x 0x%x 0x%x",
-				op_name(lp->operation),
-				lp->args[0], lp->args[1], lp->args[2]);
+			fprintf(logopsf, "%s", op_name(lp->operation));
+			for (j = 0; j < lp->nr_args; j++)
+				fprintf(logopsf, " 0x%x", lp->args[j]);
 			if (lp->flags & FL_KEEP_SIZE)
 				fprintf(logopsf, " keep_size");
 			if (lp->flags & FL_CLOSE_OPEN)
@@ -1413,6 +1438,15 @@ cleanup(int sig)
 	exit(sig);
 }
 
+static int
+op_args_count(int operation)
+{
+	switch (operation) {
+	default:
+		return 3;
+	}
+}
+
 static int
 read_op(struct log_entry *log_entry)
 {
@@ -1445,7 +1479,8 @@ read_op(struct log_entry *log_entry)
 		log_entry->operation = op_code(str);
 		if (log_entry->operation == -1)
 			goto fail;
-		for (i = 0; i < 3; i++) {
+		log_entry->nr_args = op_args_count(log_entry->operation);
+		for (i = 0; i < log_entry->nr_args; i++) {
 			char *end;
 
 			str = strtok(NULL, " \t\n");

Re: [PATCH 06/12] fsx: add five-argument logging function

[Allison Henderson]

Ok, you can add my review:

Reviewed-By: Allison Henderson <allison.henderson@oracle.com>

On 11/22/18 10:59 AM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Add a five-argument logging function to support new operations.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>   ltp/fsx.c |   45 ++++++++++++++++++++++++++++++++++++++++-----
>   1 file changed, 40 insertions(+), 5 deletions(-)
> 
> 
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index dabad302..c525ba13 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -51,7 +51,8 @@ enum opflags { FL_NONE = 0, FL_SKIPPED = 1, FL_CLOSE_OPEN = 2, FL_KEEP_SIZE = 4
>   
>   struct log_entry {
>   	int	operation;
> -	int	args[3];
> +	int	nr_args;
> +	int	args[4];
>   	enum opflags flags;
>   };
>   
> @@ -278,6 +279,27 @@ static int op_code(const char *name)
>   	return -1;
>   }
>   
> +void
> +log5(int operation, int arg0, int arg1, int arg2, enum opflags flags)
> +{
> +	struct log_entry *le;
> +
> +	le = &oplog[logptr];
> +	le->operation = operation;
> +	if (closeopen)
> +		flags |= FL_CLOSE_OPEN;
> +	le->args[0] = arg0;
> +	le->args[1] = arg1;
> +	le->args[2] = arg2;
> +	le->args[3] = file_size;
> +	le->nr_args = 4;
> +	le->flags = flags;
> +	logptr++;
> +	logcount++;
> +	if (logptr >= LOGSIZE)
> +		logptr = 0;
> +}
> +
>   void
>   log4(int operation, int arg0, int arg1, enum opflags flags)
>   {
> @@ -290,6 +312,7 @@ log4(int operation, int arg0, int arg1, enum opflags flags)
>   	le->args[0] = arg0;
>   	le->args[1] = arg1;
>   	le->args[2] = file_size;
> +	le->nr_args = 3;
>   	le->flags = flags;
>   	logptr++;
>   	logcount++;
> @@ -439,11 +462,13 @@ logdump(void)
>   			i = 0;
>   
>   		if (logopsf) {
> +			int j;
> +
>   			if (lp->flags & FL_SKIPPED)
>   				fprintf(logopsf, "skip ");
> -			fprintf(logopsf, "%s 0x%x 0x%x 0x%x",
> -				op_name(lp->operation),
> -				lp->args[0], lp->args[1], lp->args[2]);
> +			fprintf(logopsf, "%s", op_name(lp->operation));
> +			for (j = 0; j < lp->nr_args; j++)
> +				fprintf(logopsf, " 0x%x", lp->args[j]);
>   			if (lp->flags & FL_KEEP_SIZE)
>   				fprintf(logopsf, " keep_size");
>   			if (lp->flags & FL_CLOSE_OPEN)
> @@ -1413,6 +1438,15 @@ cleanup(int sig)
>   	exit(sig);
>   }
>   
> +static int
> +op_args_count(int operation)
> +{
> +	switch (operation) {
> +	default:
> +		return 3;
> +	}
> +}
> +
>   static int
>   read_op(struct log_entry *log_entry)
>   {
> @@ -1445,7 +1479,8 @@ read_op(struct log_entry *log_entry)
>   		log_entry->operation = op_code(str);
>   		if (log_entry->operation == -1)
>   			goto fail;
> -		for (i = 0; i < 3; i++) {
> +		log_entry->nr_args = op_args_count(log_entry->operation);
> +		for (i = 0; i < log_entry->nr_args; i++) {
>   			char *end;
>   
>   			str = strtok(NULL, " \t\n");
> 

[PATCH 07/12] fsx: add FICLONERANGE support

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add support for the clone range (i.e. reflink) ioctl to fsx.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |  157 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 152 insertions(+), 5 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index c525ba13..f51d38cf 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -104,6 +104,7 @@ enum {
 	OP_ZERO_RANGE,
 	OP_COLLAPSE_RANGE,
 	OP_INSERT_RANGE,
+	OP_CLONE_RANGE,
 	OP_MAX_FULL,
 
 	/* integrity operations */
@@ -165,6 +166,7 @@ int	collapse_range_calls = 1;	/* -C flag disables */
 int	insert_range_calls = 1;		/* -I flag disables */
 int	mapped_reads = 1;		/* -R flag disables it */
 int	check_file = 0;			/* -X flag enables */
+int	clone_range_calls = 1;		/* -J flag disables */
 int	integrity = 0;			/* -i flag */
 int	fsxgoodfd = 0;
 int	o_direct;			/* -Z */
@@ -258,6 +260,7 @@ static const char *op_names[] = {
 	[OP_ZERO_RANGE] = "zero_range",
 	[OP_COLLAPSE_RANGE] = "collapse_range",
 	[OP_INSERT_RANGE] = "insert_range",
+	[OP_CLONE_RANGE] = "clone_range",
 	[OP_FSYNC] = "fsync",
 };
 
@@ -320,7 +323,6 @@ log4(int operation, int arg0, int arg1, enum opflags flags)
 		logptr = 0;
 }
 
-
 void
 logdump(void)
 {
@@ -342,7 +344,7 @@ logdump(void)
 		count = LOGSIZE;
 	}
 	for ( ; count > 0; count--) {
-		bool overlap;
+		bool overlap, overlap2;
 		int opnum;
 
 		opnum = i+1 + (logcount/LOGSIZE)*LOGSIZE;
@@ -444,6 +446,20 @@ logdump(void)
 			if (overlap)
 				prt("\t******IIII");
 			break;
+		case OP_CLONE_RANGE:
+			prt("CLONE 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
+			    lp->args[0], lp->args[0] + lp->args[1] - 1,
+			    lp->args[1],
+			    lp->args[2], lp->args[2] + lp->args[1] - 1);
+			overlap2 = badoff >= lp->args[2] &&
+				  badoff < lp->args[2] + lp->args[1];
+			if (overlap && overlap2)
+				prt("\tJJJJ**JJJJ");
+			else if (overlap)
+				prt("\tJJJJ******");
+			else if (overlap2)
+				prt("\t******JJJJ");
+			break;
 		case OP_FSYNC:
 			prt("FSYNC");
 			break;
@@ -1285,6 +1301,96 @@ do_insert_range(unsigned offset, unsigned length)
 }
 #endif
 
+#ifdef FICLONERANGE
+int
+test_clone_range(void)
+{
+	struct file_clone_range	fcr = {
+		.src_fd = fd,
+	};
+
+	if (ioctl(fd, FICLONERANGE, &fcr) &&
+	    (errno = EOPNOTSUPP || errno == ENOTTY)) {
+		if (!quiet)
+			fprintf(stderr,
+				"main: filesystem does not support "
+				"clone range, disabling!\n");
+		return 0;
+	}
+
+	return 1;
+}
+
+void
+do_clone_range(unsigned offset, unsigned length, unsigned dest)
+{
+	struct file_clone_range	fcr = {
+		.src_fd = fd,
+		.src_offset = offset,
+		.src_length = length,
+		.dest_offset = dest,
+	};
+
+	if (length == 0) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping zero length clone range\n");
+		log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	if ((loff_t)offset >= file_size) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping clone range behind EOF\n");
+		log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	if (dest + length > biggest) {
+		biggest = dest + length;
+		if (!quiet && testcalls > simulatedopcount)
+			prt("cloning to largest ever: 0x%x\n", dest + length);
+	}
+
+	log5(OP_CLONE_RANGE, offset, length, dest, FL_NONE);
+
+	if (testcalls <= simulatedopcount)
+		return;
+
+	if ((progressinterval && testcalls % progressinterval == 0) ||
+	    (debug && (monitorstart == -1 || monitorend == -1 ||
+		       dest <= monitorstart || dest + length <= monitorend))) {
+		prt("%lu clone\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
+			testcalls, offset, offset+length, length, dest);
+	}
+
+	if (ioctl(fd, FICLONERANGE, &fcr) == -1) {
+		prt("clone range: 0x%x to 0x%x at 0x%x\n", offset,
+				offset + length, dest);
+		prterr("do_clone_range: FICLONERANGE");
+		report_failure(161);
+	}
+
+	memcpy(good_buf + dest, good_buf + offset, length);
+	if (dest > file_size)
+		memset(good_buf + file_size, '\0', dest - file_size);
+	if (dest + length > file_size)
+		file_size = dest + length;
+}
+
+#else
+int
+test_clone_range(void)
+{
+	return 0;
+}
+
+void
+do_clone_range(unsigned offset, unsigned length, unsigned dest)
+{
+	return;
+}
+#endif
+
 #ifdef HAVE_LINUX_FALLOC_H
 /* fallocate is basically a no-op unless extending, then a lot like a truncate */
 void
@@ -1442,6 +1548,8 @@ static int
 op_args_count(int operation)
 {
 	switch (operation) {
+	case OP_CLONE_RANGE:
+		return 4;
 	default:
 		return 3;
 	}
@@ -1515,7 +1623,7 @@ read_op(struct log_entry *log_entry)
 int
 test(void)
 {
-	unsigned long	offset;
+	unsigned long	offset, offset2;
 	unsigned long	size;
 	unsigned long	rv;
 	unsigned long	op;
@@ -1546,6 +1654,7 @@ test(void)
 			op = log_entry.operation;
 			offset = log_entry.args[0];
 			size = log_entry.args[1];
+			offset2 = log_entry.args[2];
 			closeopen = !!(log_entry.flags & FL_CLOSE_OPEN);
 			keep_size = !!(log_entry.flags & FL_KEEP_SIZE);
 			goto have_op;
@@ -1558,6 +1667,7 @@ test(void)
 		closeopen = (rv >> 3) < (1 << 28) / closeprob;
 
 	offset = random();
+	offset2 = 0;
 	size = maxoplen;
 	if (randomoplen)
 		size = random() % (maxoplen + 1);
@@ -1583,6 +1693,17 @@ test(void)
 		if (zero_range_calls && size && keep_size_calls)
 			keep_size = random() % 2;
 		break;
+	case OP_CLONE_RANGE:
+		TRIM_OFF_LEN(offset, size, file_size);
+		offset = offset & ~(block_size - 1);
+		size = size & ~(block_size - 1);
+		do {
+			offset2 = random();
+			TRIM_OFF(offset2, maxfilelen);
+			offset2 = offset2 & ~(block_size - 1);
+		} while (llabs(offset2 - offset) < size ||
+			 offset2 + size > maxfilelen);
+		break;
 	}
 
 have_op:
@@ -1626,6 +1747,12 @@ test(void)
 			goto out;
 		}
 		break;
+	case OP_CLONE_RANGE:
+		if (!clone_range_calls) {
+			log5(op, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		break;
 	}
 
 	switch (op) {
@@ -1692,6 +1819,18 @@ test(void)
 
 		do_insert_range(offset, size);
 		break;
+	case OP_CLONE_RANGE:
+		if (size == 0) {
+			log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		if (offset2 + size > maxfilelen) {
+			log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+
+		do_clone_range(offset, size, offset2);
+		break;
 	case OP_FSYNC:
 		dofsync();
 		break;
@@ -1717,7 +1856,7 @@ void
 usage(void)
 {
 	fprintf(stdout, "usage: %s",
-		"fsx [-dknqxAFLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+		"fsx [-dknqxAFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
 	-b opnum: beginning operation number (default 1)\n\
 	-c P: 1 in P chance of file close+open at each op (default infinity)\n\
 	-d: debug output for all operations\n\
@@ -1758,6 +1897,9 @@ usage(void)
 #ifdef FALLOC_FL_INSERT_RANGE
 "	-I: Do not use insert range calls\n"
 #endif
+#ifdef FICLONERANGE
+"	-J: Do not use clone range calls\n"
+#endif
 "	-L: fsxLite - no file creations & no file size changes\n\
 	-N numops: total # operations to do (default infinity)\n\
 	-O: use oplen (see -o flag) for every op (default random)\n\
@@ -1961,7 +2103,7 @@ main(int argc, char **argv)
 	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
 
 	while ((ch = getopt_long(argc, argv,
-				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
+				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FJKHzCILN:OP:RS:WXZ",
 				 longopts, NULL)) != EOF)
 		switch (ch) {
 		case 'b':
@@ -2091,6 +2233,9 @@ main(int argc, char **argv)
 		case 'I':
 			insert_range_calls = 0;
 			break;
+		case 'J':
+			clone_range_calls = 0;
+			break;
 		case 'L':
 		        lite = 1;
 			o_flags &= ~(O_CREAT|O_TRUNC);
@@ -2309,6 +2454,8 @@ main(int argc, char **argv)
 		collapse_range_calls = test_fallocate(FALLOC_FL_COLLAPSE_RANGE);
 	if (insert_range_calls)
 		insert_range_calls = test_fallocate(FALLOC_FL_INSERT_RANGE);
+	if (clone_range_calls)
+		clone_range_calls = test_clone_range();
 
 	while (numops == -1 || numops--)
 		if (!test())

[PATCH 08/12] fsx: add FIDEDUPERANGE support

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add support for the dedupe range ioctl to fsx.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 ltp/fsx.c |  198 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 196 insertions(+), 2 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index f51d38cf..577c7309 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -105,6 +105,7 @@ enum {
 	OP_COLLAPSE_RANGE,
 	OP_INSERT_RANGE,
 	OP_CLONE_RANGE,
+	OP_DEDUPE_RANGE,
 	OP_MAX_FULL,
 
 	/* integrity operations */
@@ -167,6 +168,7 @@ int	insert_range_calls = 1;		/* -I flag disables */
 int	mapped_reads = 1;		/* -R flag disables it */
 int	check_file = 0;			/* -X flag enables */
 int	clone_range_calls = 1;		/* -J flag disables */
+int	dedupe_range_calls = 1;		/* -B flag disables */
 int	integrity = 0;			/* -i flag */
 int	fsxgoodfd = 0;
 int	o_direct;			/* -Z */
@@ -261,6 +263,7 @@ static const char *op_names[] = {
 	[OP_COLLAPSE_RANGE] = "collapse_range",
 	[OP_INSERT_RANGE] = "insert_range",
 	[OP_CLONE_RANGE] = "clone_range",
+	[OP_DEDUPE_RANGE] = "dedupe_range",
 	[OP_FSYNC] = "fsync",
 };
 
@@ -460,6 +463,20 @@ logdump(void)
 			else if (overlap2)
 				prt("\t******JJJJ");
 			break;
+		case OP_DEDUPE_RANGE:
+			prt("DEDUPE 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
+			    lp->args[0], lp->args[0] + lp->args[1] - 1,
+			    lp->args[1],
+			    lp->args[2], lp->args[2] + lp->args[1] - 1);
+			overlap2 = badoff >= lp->args[2] &&
+				  badoff < lp->args[2] + lp->args[1];
+			if (overlap && overlap2)
+				prt("\tBBBB**BBBB");
+			else if (overlap)
+				prt("\tBBBB******");
+			else if (overlap2)
+				prt("\t******BBBB");
+			break;
 		case OP_FSYNC:
 			prt("FSYNC");
 			break;
@@ -1391,6 +1408,137 @@ do_clone_range(unsigned offset, unsigned length, unsigned dest)
 }
 #endif
 
+#ifdef FIDEDUPERANGE
+int
+test_dedupe_range(void)
+{
+	struct file_dedupe_range *fdr;
+	off_t new_len;
+	int error;
+	int ret = 1;
+
+	/* Alloc memory */
+	fdr = calloc(sizeof(struct file_dedupe_range_info) +
+		     sizeof(struct file_dedupe_range), 1);
+	if (!fdr) {
+		prterr("do_dedupe_range: malloc");
+		report_failure(161);
+	}
+
+	/* Make sure we have at least two blocks */
+	new_len = block_size * 2;
+	if (file_size < new_len && ftruncate(fd, new_len)) {
+		warn("main: ftruncate");
+		exit(132);
+	}
+
+	/* Try to dedupe them */
+	fdr->src_length = block_size;
+	fdr->dest_count = 1;
+	fdr->info[0].dest_fd = fd;
+	fdr->info[0].dest_offset = block_size;
+
+	if (ioctl(fd, FIDEDUPERANGE, fdr))
+		error = errno;
+	else if (fdr->info[0].status < 0)
+		error = -fdr->info[0].status;
+	else
+		error = 0;
+
+	if (error == EOPNOTSUPP || error == ENOTTY) {
+		if (!quiet)
+			fprintf(stderr,
+				"main: filesystem does not support "
+				"dedupe range, disabling!\n");
+		ret = 0;
+	}
+
+	/* Put the file back the way it was. */
+	if (file_size < new_len && ftruncate(fd, file_size)) {
+		warn("main: ftruncate");
+		exit(132);
+	}
+
+	free(fdr);
+	return ret;
+}
+
+void
+do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
+{
+	struct file_dedupe_range *fdr;
+
+	if (length == 0) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping zero length dedupe range\n");
+		log5(OP_DEDUPE_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	if ((loff_t)offset >= file_size) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping dedupe range behind EOF\n");
+		log5(OP_DEDUPE_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	log5(OP_DEDUPE_RANGE, offset, length, dest, FL_NONE);
+
+	if (testcalls <= simulatedopcount)
+		return;
+
+	if ((progressinterval && testcalls % progressinterval == 0) ||
+	    (debug && (monitorstart == -1 || monitorend == -1 ||
+		       dest <= monitorstart || dest + length <= monitorend))) {
+		prt("%lu dedupe\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
+			testcalls, offset, offset+length, length, dest);
+	}
+
+	/* Alloc memory */
+	fdr = calloc(sizeof(struct file_dedupe_range_info) +
+		     sizeof(struct file_dedupe_range), 1);
+	if (!fdr) {
+		prterr("do_dedupe_range: malloc");
+		report_failure(161);
+	}
+
+	/* Dedupe data blocks */
+	fdr->src_offset = offset;
+	fdr->src_length = length;
+	fdr->dest_count = 1;
+	fdr->info[0].dest_fd = fd;
+	fdr->info[0].dest_offset = dest;
+
+	if (ioctl(fd, FIDEDUPERANGE, fdr) == -1) {
+		prt("dedupe range: 0x%x to 0x%x at 0x%x\n", offset,
+				offset + length, dest);
+		prterr("do_dedupe_range(0): FIDEDUPERANGE");
+		report_failure(161);
+	} else if (fdr->info[0].status < 0) {
+		errno = -fdr->info[0].status;
+		prt("dedupe range: 0x%x to 0x%x at 0x%x\n", offset,
+				offset + length, dest);
+		prterr("do_dedupe_range(1): FIDEDUPERANGE");
+		report_failure(161);
+	}
+
+	free(fdr);
+}
+
+#else
+int
+test_dedupe_range(void)
+{
+	return 0;
+}
+
+void
+do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
+{
+	return;
+}
+#endif
+
 #ifdef HAVE_LINUX_FALLOC_H
 /* fallocate is basically a no-op unless extending, then a lot like a truncate */
 void
@@ -1549,6 +1697,7 @@ op_args_count(int operation)
 {
 	switch (operation) {
 	case OP_CLONE_RANGE:
+	case OP_DEDUPE_RANGE:
 		return 4;
 	default:
 		return 3;
@@ -1704,6 +1853,25 @@ test(void)
 		} while (llabs(offset2 - offset) < size ||
 			 offset2 + size > maxfilelen);
 		break;
+	case OP_DEDUPE_RANGE:
+		{
+			int tries = 0;
+
+			TRIM_OFF_LEN(offset, size, file_size);
+			offset = offset & ~(block_size - 1);
+			size = size & ~(block_size - 1);
+			do {
+				if (tries++ >= 30) {
+					size = 0;
+					break;
+				}
+				offset2 = random();
+				TRIM_OFF(offset2, file_size);
+				offset2 = offset2 & ~(block_size - 1);
+			} while (llabs(offset2 - offset) < size ||
+				 offset2 + size > file_size);
+			break;
+		}
 	}
 
 have_op:
@@ -1753,6 +1921,12 @@ test(void)
 			goto out;
 		}
 		break;
+	case OP_DEDUPE_RANGE:
+		if (!dedupe_range_calls) {
+			log5(op, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		break;
 	}
 
 	switch (op) {
@@ -1831,6 +2005,18 @@ test(void)
 
 		do_clone_range(offset, size, offset2);
 		break;
+	case OP_DEDUPE_RANGE:
+		if (size == 0) {
+			log5(OP_DEDUPE_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		if (offset2 + size > maxfilelen) {
+			log5(OP_DEDUPE_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+
+		do_dedupe_range(offset, size, offset2);
+		break;
 	case OP_FSYNC:
 		dofsync();
 		break;
@@ -1856,7 +2042,7 @@ void
 usage(void)
 {
 	fprintf(stdout, "usage: %s",
-		"fsx [-dknqxAFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+		"fsx [-dknqxABFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
 	-b opnum: beginning operation number (default 1)\n\
 	-c P: 1 in P chance of file close+open at each op (default infinity)\n\
 	-d: debug output for all operations\n\
@@ -1900,6 +2086,9 @@ usage(void)
 #ifdef FICLONERANGE
 "	-J: Do not use clone range calls\n"
 #endif
+#ifdef FIDEDUPERANGE
+"	-B: Do not use dedupe range calls\n"
+#endif
 "	-L: fsxLite - no file creations & no file size changes\n\
 	-N numops: total # operations to do (default infinity)\n\
 	-O: use oplen (see -o flag) for every op (default random)\n\
@@ -2103,7 +2292,7 @@ main(int argc, char **argv)
 	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
 
 	while ((ch = getopt_long(argc, argv,
-				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FJKHzCILN:OP:RS:WXZ",
+				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:FJKHzCILN:OP:RS:WXZ",
 				 longopts, NULL)) != EOF)
 		switch (ch) {
 		case 'b':
@@ -2236,6 +2425,9 @@ main(int argc, char **argv)
 		case 'J':
 			clone_range_calls = 0;
 			break;
+		case 'B':
+			dedupe_range_calls = 0;
+			break;
 		case 'L':
 		        lite = 1;
 			o_flags &= ~(O_CREAT|O_TRUNC);
@@ -2456,6 +2648,8 @@ main(int argc, char **argv)
 		insert_range_calls = test_fallocate(FALLOC_FL_INSERT_RANGE);
 	if (clone_range_calls)
 		clone_range_calls = test_clone_range();
+	if (dedupe_range_calls)
+		dedupe_range_calls = test_dedupe_range();
 
 	while (numops == -1 || numops--)
 		if (!test())

[PATCH 09/12] fsstress: add copy_file_range support

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Support the copy_file_range syscall in fsstress.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 configure.ac          |    2 +
 include/builddefs.in  |    1 
 ltp/Makefile          |    4 +
 ltp/fsstress.c        |  133 +++++++++++++++++++++++++++++++++++++++++++++++++
 m4/package_libcdev.m4 |   19 +++++++
 5 files changed, 159 insertions(+)


diff --git a/configure.ac b/configure.ac
index aede4f59..19798824 100644
--- a/configure.ac
+++ b/configure.ac
@@ -68,6 +68,8 @@ AC_PACKAGE_WANT_OPEN_BY_HANDLE_AT
 AC_PACKAGE_WANT_LINUX_PRCTL_H
 AC_PACKAGE_WANT_LINUX_FS_H
 
+AC_HAVE_COPY_FILE_RANGE
+
 AC_CHECK_FUNCS([renameat2])
 
 AC_CONFIG_HEADER(include/config.h)
diff --git a/include/builddefs.in b/include/builddefs.in
index fb8e912b..2605e42d 100644
--- a/include/builddefs.in
+++ b/include/builddefs.in
@@ -67,6 +67,7 @@ HAVE_DMAPI = @have_dmapi@
 HAVE_ATTR_LIST = @have_attr_list@
 HAVE_FIEMAP = @have_fiemap@
 HAVE_FALLOCATE = @have_fallocate@
+HAVE_COPY_FILE_RANGE = @have_copy_file_range@
 
 GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall
 
diff --git a/ltp/Makefile b/ltp/Makefile
index 5f784ecf..e4ca45f4 100644
--- a/ltp/Makefile
+++ b/ltp/Makefile
@@ -28,6 +28,10 @@ ifeq ($(HAVE_FALLOCATE), true)
 LCFLAGS += -DFALLOCATE
 endif
 
+ifeq ($(HAVE_COPY_FILE_RANGE),yes)
+LCFLAGS += -DHAVE_COPY_FILE_RANGE
+endif
+
 default: depend $(TARGETS)
 
 depend: .dep
diff --git a/ltp/fsstress.c b/ltp/fsstress.c
index af5d125f..e0330c95 100644
--- a/ltp/fsstress.c
+++ b/ltp/fsstress.c
@@ -26,6 +26,7 @@
 #include <libaio.h>
 io_context_t	io_ctx;
 #endif
+#include <sys/syscall.h>
 
 #ifndef FS_IOC_GETFLAGS
 #define FS_IOC_GETFLAGS                 _IOR('f', 1, long)
@@ -55,6 +56,7 @@ typedef enum {
 	OP_BULKSTAT1,
 	OP_CHOWN,
 	OP_CLONERANGE,
+	OP_COPYRANGE,
 	OP_CREAT,
 	OP_DEDUPERANGE,
 	OP_DREAD,
@@ -163,6 +165,7 @@ void	bulkstat_f(int, long);
 void	bulkstat1_f(int, long);
 void	chown_f(int, long);
 void	clonerange_f(int, long);
+void	copyrange_f(int, long);
 void	creat_f(int, long);
 void	deduperange_f(int, long);
 void	dread_f(int, long);
@@ -212,6 +215,7 @@ opdesc_t	ops[] = {
 	{ OP_BULKSTAT1, "bulkstat1", bulkstat1_f, 1, 0 },
 	{ OP_CHOWN, "chown", chown_f, 3, 1 },
 	{ OP_CLONERANGE, "clonerange", clonerange_f, 4, 1 },
+	{ OP_COPYRANGE, "copyrange", copyrange_f, 4, 1 },
 	{ OP_CREAT, "creat", creat_f, 4, 1 },
 	{ OP_DEDUPERANGE, "deduperange", deduperange_f, 4, 1},
 	{ OP_DREAD, "dread", dread_f, 4, 0 },
@@ -2333,6 +2337,135 @@ clonerange_f(
 #endif
 }
 
+/* copy some arbitrary range of f1 to f2. */
+void
+copyrange_f(
+	int			opno,
+	long			r)
+{
+#ifdef HAVE_COPY_FILE_RANGE
+	struct pathname		fpath1;
+	struct pathname		fpath2;
+	struct stat64		stat1;
+	struct stat64		stat2;
+	char			inoinfo1[1024];
+	char			inoinfo2[1024];
+	loff_t			lr;
+	loff_t			off1;
+	loff_t			off2;
+	loff_t			max_off2;
+	size_t			len;
+	int			v1;
+	int			v2;
+	int			fd1;
+	int			fd2;
+	int			ret;
+	int			e;
+
+	/* Load paths */
+	init_pathname(&fpath1);
+	if (!get_fname(FT_REGm, r, &fpath1, NULL, NULL, &v1)) {
+		if (v1)
+			printf("%d/%d: copyrange read - no filename\n",
+				procid, opno);
+		goto out_fpath1;
+	}
+
+	init_pathname(&fpath2);
+	if (!get_fname(FT_REGm, random(), &fpath2, NULL, NULL, &v2)) {
+		if (v2)
+			printf("%d/%d: copyrange write - no filename\n",
+				procid, opno);
+		goto out_fpath2;
+	}
+
+	/* Open files */
+	fd1 = open_path(&fpath1, O_RDONLY);
+	e = fd1 < 0 ? errno : 0;
+	check_cwd();
+	if (fd1 < 0) {
+		if (v1)
+			printf("%d/%d: copyrange read - open %s failed %d\n",
+				procid, opno, fpath1.path, e);
+		goto out_fpath2;
+	}
+
+	fd2 = open_path(&fpath2, O_WRONLY);
+	e = fd2 < 0 ? errno : 0;
+	check_cwd();
+	if (fd2 < 0) {
+		if (v2)
+			printf("%d/%d: copyrange write - open %s failed %d\n",
+				procid, opno, fpath2.path, e);
+		goto out_fd1;
+	}
+
+	/* Get file stats */
+	if (fstat64(fd1, &stat1) < 0) {
+		if (v1)
+			printf("%d/%d: copyrange read - fstat64 %s failed %d\n",
+				procid, opno, fpath1.path, errno);
+		goto out_fd2;
+	}
+	inode_info(inoinfo1, sizeof(inoinfo1), &stat1, v1);
+
+	if (fstat64(fd2, &stat2) < 0) {
+		if (v2)
+			printf("%d/%d: copyrange write - fstat64 %s failed %d\n",
+				procid, opno, fpath2.path, errno);
+		goto out_fd2;
+	}
+	inode_info(inoinfo2, sizeof(inoinfo2), &stat2, v2);
+
+	/* Calculate offsets */
+	len = (random() % FILELEN_MAX) + 1;
+	if (len == 0)
+		len = stat1.st_blksize;
+	if (len > stat1.st_size)
+		len = stat1.st_size;
+
+	lr = ((int64_t)random() << 32) + random();
+	if (stat1.st_size == len)
+		off1 = 0;
+	else
+		off1 = (off64_t)(lr % MIN(stat1.st_size - len, MAXFSIZE));
+	off1 %= maxfsize;
+
+	/*
+	 * If srcfile == destfile, randomly generate destination ranges
+	 * until we find one that doesn't overlap the source range.
+	 */
+	max_off2 = MIN(stat2.st_size + (1024ULL * stat2.st_blksize), MAXFSIZE);
+	do {
+		lr = ((int64_t)random() << 32) + random();
+		off2 = (off64_t)(lr % max_off2);
+		off2 %= maxfsize;
+	} while (stat1.st_ino == stat2.st_ino && llabs(off2 - off1) < len);
+
+	ret = copy_file_range(fd1, &off1, fd2, &off2, len, 0);
+	e = ret < 0 ? errno : 0;
+	if (v1 || v2) {
+		printf("%d/%d: copyrange %s%s [%lld,%lld] -> %s%s [%lld,%lld]",
+			procid, opno,
+			fpath1.path, inoinfo1, (long long)off1, (long long)len,
+			fpath2.path, inoinfo2, (long long)off2, (long long)len);
+
+		if (ret < 0)
+			printf(" error %d", e);
+		printf("\n");
+	}
+
+out_fd2:
+	close(fd2);
+out_fd1:
+	close(fd1);
+out_fpath2:
+	free_pathname(&fpath2);
+out_fpath1:
+	free_pathname(&fpath1);
+#endif
+}
+
 /* dedupe some arbitrary range of f1 to f2...fn. */
 void
 deduperange_f(
diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4
index fb123f33..14e67e18 100644
--- a/m4/package_libcdev.m4
+++ b/m4/package_libcdev.m4
@@ -98,3 +98,22 @@ AC_DEFUN([AC_HAVE_GETMNTINFO],
        AC_MSG_RESULT(no))
     AC_SUBST(have_getmntinfo)
   ])
+
+#
+#
+# Check if we have a copy_file_range system call (Linux)
+#
+AC_DEFUN([AC_HAVE_COPY_FILE_RANGE],
+  [ AC_MSG_CHECKING([for copy_file_range])
+    AC_TRY_LINK([
+#define _GNU_SOURCE
+#include <sys/syscall.h>
+#include <unistd.h>
+    ], [
+         syscall(__NR_copy_file_range, 0, 0, 0, 0, 0, 0);
+    ], have_copy_file_range=yes
+       AC_MSG_RESULT(yes),
+       AC_MSG_RESULT(no))
+    AC_SUBST(have_copy_file_range)
+  ])
+

[PATCH 10/12] fsx: add copy_file_range support

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add support for the copy_file_range system call to fsx.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
[dchinner: copy_file_range() needs to obey read/write constraints
otherwise is blows up when direct IO is used]
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 ltp/fsx.c |  150 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 148 insertions(+), 2 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index 577c7309..92e4c30f 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -34,6 +34,7 @@
 #ifdef AIO
 #include <libaio.h>
 #endif
+#include <sys/syscall.h>
 
 #ifndef MAP_FILE
 # define MAP_FILE 0
@@ -106,6 +107,7 @@ enum {
 	OP_INSERT_RANGE,
 	OP_CLONE_RANGE,
 	OP_DEDUPE_RANGE,
+	OP_COPY_RANGE,
 	OP_MAX_FULL,
 
 	/* integrity operations */
@@ -169,6 +171,7 @@ int	mapped_reads = 1;		/* -R flag disables it */
 int	check_file = 0;			/* -X flag enables */
 int	clone_range_calls = 1;		/* -J flag disables */
 int	dedupe_range_calls = 1;		/* -B flag disables */
+int	copy_range_calls = 1;		/* -E flag disables */
 int	integrity = 0;			/* -i flag */
 int	fsxgoodfd = 0;
 int	o_direct;			/* -Z */
@@ -264,6 +267,7 @@ static const char *op_names[] = {
 	[OP_INSERT_RANGE] = "insert_range",
 	[OP_CLONE_RANGE] = "clone_range",
 	[OP_DEDUPE_RANGE] = "dedupe_range",
+	[OP_COPY_RANGE] = "copy_range",
 	[OP_FSYNC] = "fsync",
 };
 
@@ -477,6 +481,20 @@ logdump(void)
 			else if (overlap2)
 				prt("\t******BBBB");
 			break;
+		case OP_COPY_RANGE:
+			prt("COPY 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
+			    lp->args[0], lp->args[0] + lp->args[1] - 1,
+			    lp->args[1],
+			    lp->args[2], lp->args[2] + lp->args[1] - 1);
+			overlap2 = badoff >= lp->args[2] &&
+				  badoff < lp->args[2] + lp->args[1];
+			if (overlap && overlap2)
+				prt("\tEEEE**EEEE");
+			else if (overlap)
+				prt("\tEEEE******");
+			else if (overlap2)
+				prt("\t******EEEE");
+			break;
 		case OP_FSYNC:
 			prt("FSYNC");
 			break;
@@ -1539,6 +1557,99 @@ do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
 }
 #endif
 
+#ifdef HAVE_COPY_FILE_RANGE
+int
+test_copy_range(void)
+{
+	loff_t o1 = 0, o2 = 0;
+
+	if (copy_file_range(fd, &o1, fd, &o2, 0, 0) == -1 &&
+	    (errno == EOPNOTSUPP || errno == ENOTTY)) {
+		if (!quiet)
+			fprintf(stderr,
+				"main: filesystem does not support "
+				"copy range, disabling!\n");
+		return 0;
+	}
+
+	return 1;
+}
+
+void
+do_copy_range(unsigned offset, unsigned length, unsigned dest)
+{
+	loff_t o1, o2;
+	ssize_t nr;
+
+	offset -= offset % readbdy;
+	dest -= dest % writebdy;
+	if (o_direct)
+		length -= length % readbdy;
+
+	if (length == 0) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping zero length copy range\n");
+		log5(OP_COPY_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	if ((loff_t)offset >= file_size) {
+		if (!quiet && testcalls > simulatedopcount)
+			prt("skipping copy range behind EOF\n");
+		log5(OP_COPY_RANGE, offset, length, dest, FL_SKIPPED);
+		return;
+	}
+
+	if (dest + length > biggest) {
+		biggest = dest + length;
+		if (!quiet && testcalls > simulatedopcount)
+			prt("copying to largest ever: 0x%x\n", dest + length);
+	}
+
+	log5(OP_COPY_RANGE, offset, length, dest, FL_NONE);
+
+	if (testcalls <= simulatedopcount)
+		return;
+
+	if ((progressinterval && testcalls % progressinterval == 0) ||
+	    (debug && (monitorstart == -1 || monitorend == -1 ||
+		       dest <= monitorstart || dest + length <= monitorend))) {
+		prt("%lu copy\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
+			testcalls, offset, offset+length, length, dest);
+	}
+
+	o1 = offset;
+	o2 = dest;
+
+	nr = copy_file_range(fd, &o1, fd, &o2, length, 0);
+	if (nr == -1) {
+		prt("copy range: 0x%x to 0x%x at 0x%x\n", offset,
+				offset + length, dest);
+		prterr("do_copy_range:");
+		report_failure(161);
+	}
+
+	memcpy(good_buf + dest, good_buf + offset, nr);
+	if (dest > file_size)
+		memset(good_buf + file_size, '\0', dest - file_size);
+	if (dest + nr > file_size)
+		file_size = dest + nr;
+}
+
+#else
+int
+test_copy_range(void)
+{
+	return 0;
+}
+
+void
+do_copy_range(unsigned offset, unsigned length, unsigned dest)
+{
+	return;
+}
+#endif
+
 #ifdef HAVE_LINUX_FALLOC_H
 /* fallocate is basically a no-op unless extending, then a lot like a truncate */
 void
@@ -1698,6 +1809,7 @@ op_args_count(int operation)
 	switch (operation) {
 	case OP_CLONE_RANGE:
 	case OP_DEDUPE_RANGE:
+	case OP_COPY_RANGE:
 		return 4;
 	default:
 		return 3;
@@ -1872,6 +1984,14 @@ test(void)
 				 offset2 + size > file_size);
 			break;
 		}
+	case OP_COPY_RANGE:
+		TRIM_OFF_LEN(offset, size, file_size);
+		do {
+			offset2 = random();
+			TRIM_OFF(offset2, maxfilelen);
+		} while (llabs(offset2 - offset) < size ||
+			 offset2 + size > maxfilelen);
+		break;
 	}
 
 have_op:
@@ -1927,6 +2047,12 @@ test(void)
 			goto out;
 		}
 		break;
+	case OP_COPY_RANGE:
+		if (!copy_range_calls) {
+			log5(op, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		break;
 	}
 
 	switch (op) {
@@ -2017,6 +2143,18 @@ test(void)
 
 		do_dedupe_range(offset, size, offset2);
 		break;
+	case OP_COPY_RANGE:
+		if (size == 0) {
+			log5(OP_COPY_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+		if (offset2 + size > maxfilelen) {
+			log5(OP_COPY_RANGE, offset, size, offset2, FL_SKIPPED);
+			goto out;
+		}
+
+		do_copy_range(offset, size, offset2);
+		break;
 	case OP_FSYNC:
 		dofsync();
 		break;
@@ -2042,7 +2180,7 @@ void
 usage(void)
 {
 	fprintf(stdout, "usage: %s",
-		"fsx [-dknqxABFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+		"fsx [-dknqxABEFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
 	-b opnum: beginning operation number (default 1)\n\
 	-c P: 1 in P chance of file close+open at each op (default infinity)\n\
 	-d: debug output for all operations\n\
@@ -2089,6 +2227,9 @@ usage(void)
 #ifdef FIDEDUPERANGE
 "	-B: Do not use dedupe range calls\n"
 #endif
+#ifdef HAVE_COPY_FILE_RANGE
+"	-E: Do not use copy range calls\n"
+#endif
 "	-L: fsxLite - no file creations & no file size changes\n\
 	-N numops: total # operations to do (default infinity)\n\
 	-O: use oplen (see -o flag) for every op (default random)\n\
@@ -2292,7 +2433,7 @@ main(int argc, char **argv)
 	setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
 
 	while ((ch = getopt_long(argc, argv,
-				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:FJKHzCILN:OP:RS:WXZ",
+				 "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:WXZ",
 				 longopts, NULL)) != EOF)
 		switch (ch) {
 		case 'b':
@@ -2428,6 +2569,9 @@ main(int argc, char **argv)
 		case 'B':
 			dedupe_range_calls = 0;
 			break;
+		case 'E':
+			copy_range_calls = 0;
+			break;
 		case 'L':
 		        lite = 1;
 			o_flags &= ~(O_CREAT|O_TRUNC);
@@ -2650,6 +2794,8 @@ main(int argc, char **argv)
 		clone_range_calls = test_clone_range();
 	if (dedupe_range_calls)
 		dedupe_range_calls = test_dedupe_range();
+	if (copy_range_calls)
+		copy_range_calls = test_copy_range();
 
 	while (numops == -1 || numops--)
 		if (!test())

[PATCH 11/12] common/dump: disable copyrange

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

In _create_dumpdir_stress_num, disable copyrange in fsstress so that we
dump exactly the same set of files and directories no matter how the xfs
is configured.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/dump |    3 +++
 1 file changed, 3 insertions(+)


diff --git a/common/dump b/common/dump
index c45b8a5a..4d1a1607 100644
--- a/common/dump
+++ b/common/dump
@@ -307,6 +307,9 @@ _create_dumpdir_stress_num()
     if $FSSTRESS_PROG | grep -q deduperange; then
         FSSTRESS_AVOID="-f deduperange=0 $FSSTRESS_AVOID"
     fi
+    if $FSSTRESS_PROG | grep -q copyrange; then
+        FSSTRESS_AVOID="-f copyrange=0 $FSSTRESS_AVOID"
+    fi
 
     echo ""
     echo "-----------------------------------------------"

[PATCH 12/12] generic: long fsx soak tests

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add a few tests to implement long soak tests of fsx.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/generic/938     |   65 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/938.out |    2 ++
 tests/generic/939     |   60 +++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/939.out |    2 ++
 tests/generic/group   |    2 ++
 5 files changed, 131 insertions(+)
 create mode 100644 tests/generic/938
 create mode 100644 tests/generic/938.out
 create mode 100644 tests/generic/939
 create mode 100644 tests/generic/939.out


diff --git a/tests/generic/938 b/tests/generic/938
new file mode 100644
index 00000000..a85a6472
--- /dev/null
+++ b/tests/generic/938
@@ -0,0 +1,65 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2018 Oracle.  All Rights Reserved.
+#
+# FS QA Test 938
+#
+# Long-soak directio fsx test
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1        # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+        cd /
+        rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+
+# Run fsx for a million ops or more
+nr_ops=$((1000000 * TIME_FACTOR))
+op_sz=$((128000 * LOAD_FACTOR))
+file_sz=$((600000 * LOAD_FACTOR))
+fsx_file=$TEST_DIR/fsx.$seq
+min_dio_sz=$(_min_dio_alignment)
+
+fsx_args=(-q)
+fsx_args+=(-N $nr_ops)
+fsx_args+=(-p $((nr_ops / 100)))
+fsx_args+=(-o $op_sz)
+fsx_args+=(-l $file_sz)
+fsx_args+=(-r $min_dio_sz)
+fsx_args+=(-t $min_dio_sz)
+fsx_args+=(-w $min_dio_sz)
+fsx_args+=(-Z)
+fsx_args+=($fsx_file)
+
+echo "ltp ${fsx_args[@]} $FSX_ARGS" >> $seqres.full
+
+if ! ltp/fsx "${fsx_args[@]}" $FSX_ARGS > $tmp.fsx 2>&1; then
+	cat $tmp.fsx | tee -a $seqres.full
+fi
+
+# success, all done
+echo Silence is golden
+status=0
+exit
diff --git a/tests/generic/938.out b/tests/generic/938.out
new file mode 100644
index 00000000..11b565ee
--- /dev/null
+++ b/tests/generic/938.out
@@ -0,0 +1,2 @@
+QA output created by 938
+Silence is golden
diff --git a/tests/generic/939 b/tests/generic/939
new file mode 100644
index 00000000..5e341e30
--- /dev/null
+++ b/tests/generic/939
@@ -0,0 +1,60 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2018 Oracle.  All Rights Reserved.
+#
+# FS QA Test 938
+#
+# Long-soak buffered fsx test
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1        # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+        cd /
+        rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+
+# Run fsx for a million ops or more
+nr_ops=$((1000000 * TIME_FACTOR))
+op_sz=$((128000 * LOAD_FACTOR))
+file_sz=$((600000 * LOAD_FACTOR))
+fsx_file=$TEST_DIR/fsx.$seq
+
+fsx_args=(-q)
+fsx_args+=(-N $nr_ops)
+fsx_args+=(-p $((nr_ops / 100)))
+fsx_args+=(-o $op_sz)
+fsx_args+=(-l $file_sz)
+fsx_args+=($fsx_file)
+
+echo "ltp ${fsx_args[@]} $FSX_ARGS" >> $seqres.full
+
+if ! ltp/fsx "${fsx_args[@]}" $FSX_ARGS > $tmp.fsx 2>&1; then
+	cat $tmp.fsx | tee -a $seqres.full
+fi
+
+# success, all done
+echo Silence is golden
+status=0
+exit
diff --git a/tests/generic/939.out b/tests/generic/939.out
new file mode 100644
index 00000000..a86e90c5
--- /dev/null
+++ b/tests/generic/939.out
@@ -0,0 +1,2 @@
+QA output created by 939
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 92330b52..16e5cc17 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -522,3 +522,5 @@
 517 auto quick dedupe clone
 518 auto quick clone
 519 auto quick
+938 soak long_rw
+939 soak long_rw

Re: [PATCH 12/12] generic: long fsx soak tests

[Eryu Guan]

On Thu, Nov 22, 2018 at 09:59:44AM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Add a few tests to implement long soak tests of fsx.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  tests/generic/938     |   65 +++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/938.out |    2 ++
>  tests/generic/939     |   60 +++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/939.out |    2 ++
>  tests/generic/group   |    2 ++
>  5 files changed, 131 insertions(+)
>  create mode 100644 tests/generic/938

The file mode should be 755.

>  create mode 100644 tests/generic/938.out
>  create mode 100644 tests/generic/939

Same here.

>  create mode 100644 tests/generic/939.out
> 
> 
> diff --git a/tests/generic/938 b/tests/generic/938
> new file mode 100644
> index 00000000..a85a6472
> --- /dev/null
> +++ b/tests/generic/938
> @@ -0,0 +1,65 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2018 Oracle.  All Rights Reserved.
> +#
> +# FS QA Test 938
> +#
> +# Long-soak directio fsx test
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1        # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +        cd /
> +        rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs generic
> +_supported_os Linux
> +_require_test

_require_odirect ?

> +
> +# Run fsx for a million ops or more
> +nr_ops=$((1000000 * TIME_FACTOR))
> +op_sz=$((128000 * LOAD_FACTOR))
> +file_sz=$((600000 * LOAD_FACTOR))
> +fsx_file=$TEST_DIR/fsx.$seq
> +min_dio_sz=$(_min_dio_alignment)
> +
> +fsx_args=(-q)
> +fsx_args+=(-N $nr_ops)
> +fsx_args+=(-p $((nr_ops / 100)))
> +fsx_args+=(-o $op_sz)
> +fsx_args+=(-l $file_sz)
> +fsx_args+=(-r $min_dio_sz)
> +fsx_args+=(-t $min_dio_sz)
> +fsx_args+=(-w $min_dio_sz)
> +fsx_args+=(-Z)
> +fsx_args+=($fsx_file)
> +
> +echo "ltp ${fsx_args[@]} $FSX_ARGS" >> $seqres.full

FSX_ARGS is not a global variable. Perhaps you mean FSX_AVOID?

> +
> +if ! ltp/fsx "${fsx_args[@]}" $FSX_ARGS > $tmp.fsx 2>&1; then
> +	cat $tmp.fsx | tee -a $seqres.full
> +fi

And maybe "run_fsx" from common/rc could do all the trick?

Thanks,
Eryu

> +
> +# success, all done
> +echo Silence is golden
> +status=0
> +exit
> diff --git a/tests/generic/938.out b/tests/generic/938.out
> new file mode 100644
> index 00000000..11b565ee
> --- /dev/null
> +++ b/tests/generic/938.out
> @@ -0,0 +1,2 @@
> +QA output created by 938
> +Silence is golden
> diff --git a/tests/generic/939 b/tests/generic/939
> new file mode 100644
> index 00000000..5e341e30
> --- /dev/null
> +++ b/tests/generic/939
> @@ -0,0 +1,60 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2018 Oracle.  All Rights Reserved.
> +#
> +# FS QA Test 938
> +#
> +# Long-soak buffered fsx test
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1        # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +        cd /
> +        rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs generic
> +_supported_os Linux
> +_require_test
> +
> +# Run fsx for a million ops or more
> +nr_ops=$((1000000 * TIME_FACTOR))
> +op_sz=$((128000 * LOAD_FACTOR))
> +file_sz=$((600000 * LOAD_FACTOR))
> +fsx_file=$TEST_DIR/fsx.$seq
> +
> +fsx_args=(-q)
> +fsx_args+=(-N $nr_ops)
> +fsx_args+=(-p $((nr_ops / 100)))
> +fsx_args+=(-o $op_sz)
> +fsx_args+=(-l $file_sz)
> +fsx_args+=($fsx_file)
> +
> +echo "ltp ${fsx_args[@]} $FSX_ARGS" >> $seqres.full
> +
> +if ! ltp/fsx "${fsx_args[@]}" $FSX_ARGS > $tmp.fsx 2>&1; then
> +	cat $tmp.fsx | tee -a $seqres.full
> +fi
> +
> +# success, all done
> +echo Silence is golden
> +status=0
> +exit
> diff --git a/tests/generic/939.out b/tests/generic/939.out
> new file mode 100644
> index 00000000..a86e90c5
> --- /dev/null
> +++ b/tests/generic/939.out
> @@ -0,0 +1,2 @@
> +QA output created by 939
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index 92330b52..16e5cc17 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -522,3 +522,5 @@
>  517 auto quick dedupe clone
>  518 auto quick clone
>  519 auto quick
> +938 soak long_rw
> +939 soak long_rw
> 

Re: [PATCH 12/12] generic: long fsx soak tests

[Darrick J. Wong]

On Mon, Nov 26, 2018 at 12:27:41AM +0800, Eryu Guan wrote:
> On Thu, Nov 22, 2018 at 09:59:44AM -0800, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Add a few tests to implement long soak tests of fsx.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  tests/generic/938     |   65 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/generic/938.out |    2 ++
> >  tests/generic/939     |   60 +++++++++++++++++++++++++++++++++++++++++++++
> >  tests/generic/939.out |    2 ++
> >  tests/generic/group   |    2 ++
> >  5 files changed, 131 insertions(+)
> >  create mode 100644 tests/generic/938
> 
> The file mode should be 755.

Fixed.

> >  create mode 100644 tests/generic/938.out
> >  create mode 100644 tests/generic/939
> 
> Same here.

Fixed.

> >  create mode 100644 tests/generic/939.out
> > 
> > 
> > diff --git a/tests/generic/938 b/tests/generic/938
> > new file mode 100644
> > index 00000000..a85a6472
> > --- /dev/null
> > +++ b/tests/generic/938
> > @@ -0,0 +1,65 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2018 Oracle.  All Rights Reserved.
> > +#
> > +# FS QA Test 938
> > +#
> > +# Long-soak directio fsx test
> > +#
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1        # failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > +
> > +_cleanup()
> > +{
> > +        cd /
> > +        rm -f $tmp.*
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +# remove previous $seqres.full before test
> > +rm -f $seqres.full
> > +
> > +# real QA test starts here
> > +
> > +# Modify as appropriate.
> > +_supported_fs generic
> > +_supported_os Linux
> > +_require_test
> 
> _require_odirect ?

Fixed.

> > +
> > +# Run fsx for a million ops or more
> > +nr_ops=$((1000000 * TIME_FACTOR))
> > +op_sz=$((128000 * LOAD_FACTOR))
> > +file_sz=$((600000 * LOAD_FACTOR))
> > +fsx_file=$TEST_DIR/fsx.$seq
> > +min_dio_sz=$(_min_dio_alignment)
> > +
> > +fsx_args=(-q)
> > +fsx_args+=(-N $nr_ops)
> > +fsx_args+=(-p $((nr_ops / 100)))
> > +fsx_args+=(-o $op_sz)
> > +fsx_args+=(-l $file_sz)
> > +fsx_args+=(-r $min_dio_sz)
> > +fsx_args+=(-t $min_dio_sz)
> > +fsx_args+=(-w $min_dio_sz)
> > +fsx_args+=(-Z)
> > +fsx_args+=($fsx_file)
> > +
> > +echo "ltp ${fsx_args[@]} $FSX_ARGS" >> $seqres.full
> 
> FSX_ARGS is not a global variable. Perhaps you mean FSX_AVOID?
> 
> > +
> > +if ! ltp/fsx "${fsx_args[@]}" $FSX_ARGS > $tmp.fsx 2>&1; then
> > +	cat $tmp.fsx | tee -a $seqres.full
> > +fi
> 
> And maybe "run_fsx" from common/rc could do all the trick?

Good suggestion, thank you.

--D

> Thanks,
> Eryu
> 
> > +
> > +# success, all done
> > +echo Silence is golden
> > +status=0
> > +exit
> > diff --git a/tests/generic/938.out b/tests/generic/938.out
> > new file mode 100644
> > index 00000000..11b565ee
> > --- /dev/null
> > +++ b/tests/generic/938.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 938
> > +Silence is golden
> > diff --git a/tests/generic/939 b/tests/generic/939
> > new file mode 100644
> > index 00000000..5e341e30
> > --- /dev/null
> > +++ b/tests/generic/939
> > @@ -0,0 +1,60 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2018 Oracle.  All Rights Reserved.
> > +#
> > +# FS QA Test 938
> > +#
> > +# Long-soak buffered fsx test
> > +#
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1        # failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > +
> > +_cleanup()
> > +{
> > +        cd /
> > +        rm -f $tmp.*
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +# remove previous $seqres.full before test
> > +rm -f $seqres.full
> > +
> > +# real QA test starts here
> > +
> > +# Modify as appropriate.
> > +_supported_fs generic
> > +_supported_os Linux
> > +_require_test
> > +
> > +# Run fsx for a million ops or more
> > +nr_ops=$((1000000 * TIME_FACTOR))
> > +op_sz=$((128000 * LOAD_FACTOR))
> > +file_sz=$((600000 * LOAD_FACTOR))
> > +fsx_file=$TEST_DIR/fsx.$seq
> > +
> > +fsx_args=(-q)
> > +fsx_args+=(-N $nr_ops)
> > +fsx_args+=(-p $((nr_ops / 100)))
> > +fsx_args+=(-o $op_sz)
> > +fsx_args+=(-l $file_sz)
> > +fsx_args+=($fsx_file)
> > +
> > +echo "ltp ${fsx_args[@]} $FSX_ARGS" >> $seqres.full
> > +
> > +if ! ltp/fsx "${fsx_args[@]}" $FSX_ARGS > $tmp.fsx 2>&1; then
> > +	cat $tmp.fsx | tee -a $seqres.full
> > +fi
> > +
> > +# success, all done
> > +echo Silence is golden
> > +status=0
> > +exit
> > diff --git a/tests/generic/939.out b/tests/generic/939.out
> > new file mode 100644
> > index 00000000..a86e90c5
> > --- /dev/null
> > +++ b/tests/generic/939.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 939
> > +Silence is golden
> > diff --git a/tests/generic/group b/tests/generic/group
> > index 92330b52..16e5cc17 100644
> > --- a/tests/generic/group
> > +++ b/tests/generic/group
> > @@ -522,3 +522,5 @@
> >  517 auto quick dedupe clone
> >  518 auto quick clone
> >  519 auto quick
> > +938 soak long_rw
> > +939 soak long_rw
> > 

[PATCH v4 00/12] xfstests: add copy/dedupe/clone to fsx/fsstress

[Darrick J. Wong]

Hi all,

This series adds to fsx and fsstress support for FICLONERANGE,
FIDEDUPERANGE, and copy_file_range.  The only differences since v3 have
been to deal with copy_file_range's short copies correctly...ish.

The first three patches fix gcc warnings in fsx and fsstress.

Patch 4 adds an -X option to fsx that causes it to re-read the fsx file
after every operation to compare it to the good buffer.  This made it
easier for me to find corruption problem as soon as they happen, though
it slows down fsx and perturbs in-core state considerably.

Patche 5-6 are a couple of generic reworks to fsx that we need to
support the new clone/dedupe/copy commands.

Patches 7-8 add clone and dedupe to fsx.

Patches 9-10 add copy_file_range support to fsstress and fsx.

Patch 11 fixes the common/dump tests to disable the new commands so that
the dump/restore tests continue to function exactly as they have for
years.

Patch 12 contains long-soak fsx tests designed to draw out corruption
problems by running (1 million * TIME_FACTOR) operations.  These will be
running all weekend during the US Thanksgiving holiday, though I hope
we've sorted out all the problems.

There are known failures in XFS on 4.20-rc5, particularly with
copy_file_range and reflink, all of which hopefully will have been fixed
by the time 4.21 comes out.  Dave Chinner has more fixes in progress to
correct various flaws of copy_file_range.

--D

[PATCH 03/12] fsx: shut up compiler warnings

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Fix unused variables and potential filename overflows in fsx.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-By: Allison Henderson <allison.henderson@oracle.com>
---
 ltp/fsx.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index b0157ba3..5601c70c 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -121,7 +121,7 @@ char	*bname;				/* basename of our test file */
 char	*logdev;			/* -i flag */
 char	*logid;				/* -j flag */
 char	dname[1024];			/* -P flag */
-char	goodfile[1024];
+char	goodfile[PATH_MAX];
 int	dirpath = 0;			/* -P flag */
 int	fd;				/* fd for our test file */
 
@@ -185,7 +185,7 @@ const char *replayops = NULL;
 const char *recordops = NULL;
 FILE *	fsxlogf = NULL;
 FILE *	replayopsf = NULL;
-char opsfile[1024];
+char opsfile[PATH_MAX];
 int badoff = -1;
 int closeopen = 0;
 
@@ -541,13 +541,13 @@ mark_log(void)
 void
 dump_fsync_buffer(void)
 {
-	char fname_buffer[1024];
+	char fname_buffer[PATH_MAX];
 	int good_fd;
 
 	if (!good_buf)
 		return;
 
-	snprintf(fname_buffer, 1024, "%s%s.mark%d", dname,
+	snprintf(fname_buffer, sizeof(fname_buffer), "%s%s.mark%d", dname,
 		 bname, mark_nr);
 	good_fd = open(fname_buffer, O_WRONLY|O_CREAT|O_TRUNC, 0666);
 	if (good_fd < 0) {
@@ -1865,7 +1865,7 @@ main(int argc, char **argv)
 {
 	int	i, style, ch;
 	char	*endp, *tmp;
-	char logfile[1024];
+	char logfile[PATH_MAX];
 	struct stat statbuf;
 	int o_flags = O_RDWR|O_CREAT|O_TRUNC;