* [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress
@ 2018-11-20 21:56 Darrick J. Wong
2018-11-20 21:56 ` [PATCH 01/10] fsx: shut up compiler warnings Darrick J. Wong
` (11 more replies)
0 siblings, 12 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:56 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests
Hi all,
This series adds to fsx and fsstress support for FICLONERANGE,
FIDEDUPERANGE, and copy_file_range.
First, I fix some gcc warnings in fsx.
Then, I teach fsx to read the fsx file after every operation to compare
it to the good buffer. This made it easier for me to find corruption
problem as soon as they happen, though I'm not sure it really makes
sense to have this enabled by default because of the behavior change
that it makes.
Next come a couple of generic reworks to fsx that we need to support the
new clone/dedupe/copy commands.
Patches 5-6 add clone and dedupe to fsx.
Patches 7-8 add copy_file_range support to fsstress and fsx.
Dave Chinner contributed some cleanups to the fsx patches as the 9th
patch.
The last patch fixes the common/dump tests to disable the new commands
so that the dump/restore tests continue to function exactly as they have
for years.
There are known failures in 4.20-rc3, particularly with copy_file_range,
which hopefully have been fixed by the patch series that Dave Chinner
posted to the xfs list yesterday. Branch can be downloaded here[1].
--D
[1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fsstress-clone
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 01/10] fsx: shut up compiler warnings
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
@ 2018-11-20 21:56 ` Darrick J. Wong
2018-11-20 21:56 ` [PATCH 02/10] fsx: always check buffer after each operation Darrick J. Wong
` (10 subsequent siblings)
11 siblings, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:56 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests
From: Darrick J. Wong <darrick.wong@oracle.com>
Fix unused variables and potential filename overflows in fsx.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
ltp/fsx.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index b0157ba3..5601c70c 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -121,7 +121,7 @@ char *bname; /* basename of our test file */
char *logdev; /* -i flag */
char *logid; /* -j flag */
char dname[1024]; /* -P flag */
-char goodfile[1024];
+char goodfile[PATH_MAX];
int dirpath = 0; /* -P flag */
int fd; /* fd for our test file */
@@ -185,7 +185,7 @@ const char *replayops = NULL;
const char *recordops = NULL;
FILE * fsxlogf = NULL;
FILE * replayopsf = NULL;
-char opsfile[1024];
+char opsfile[PATH_MAX];
int badoff = -1;
int closeopen = 0;
@@ -541,13 +541,13 @@ mark_log(void)
void
dump_fsync_buffer(void)
{
- char fname_buffer[1024];
+ char fname_buffer[PATH_MAX];
int good_fd;
if (!good_buf)
return;
- snprintf(fname_buffer, 1024, "%s%s.mark%d", dname,
+ snprintf(fname_buffer, sizeof(fname_buffer), "%s%s.mark%d", dname,
bname, mark_nr);
good_fd = open(fname_buffer, O_WRONLY|O_CREAT|O_TRUNC, 0666);
if (good_fd < 0) {
@@ -1865,7 +1865,7 @@ main(int argc, char **argv)
{
int i, style, ch;
char *endp, *tmp;
- char logfile[1024];
+ char logfile[PATH_MAX];
struct stat statbuf;
int o_flags = O_RDWR|O_CREAT|O_TRUNC;
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 02/10] fsx: always check buffer after each operation
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
2018-11-20 21:56 ` [PATCH 01/10] fsx: shut up compiler warnings Darrick J. Wong
@ 2018-11-20 21:56 ` Darrick J. Wong
2018-11-21 1:18 ` Darrick J. Wong
2018-11-22 2:15 ` Darrick J. Wong
2018-11-20 21:56 ` [PATCH 03/10] fsx: use an enum to define the operation commands Darrick J. Wong
` (9 subsequent siblings)
11 siblings, 2 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:56 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests
From: Darrick J. Wong <darrick.wong@oracle.com>
Add a new option to make fsx read the file after each operation and
compare it with the good buffer to try to catch corruptions as soon as
they occur.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
ltp/fsx.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 56 insertions(+), 9 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index 5601c70c..4b3a5d5a 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -160,7 +160,8 @@ int punch_hole_calls = 1; /* -H flag disables */
int zero_range_calls = 1; /* -z flag disables */
int collapse_range_calls = 1; /* -C flag disables */
int insert_range_calls = 1; /* -I flag disables */
-int mapped_reads = 1; /* -R flag disables it */
+int mapped_reads = 1; /* -R flag disables it */
+int check_file = 1; /* -X flag disables */
int integrity = 0; /* -i flag */
int fsxgoodfd = 0;
int o_direct; /* -Z */
@@ -561,7 +562,7 @@ dump_fsync_buffer(void)
}
void
-check_buffers(unsigned offset, unsigned size)
+check_buffers(char *buf, unsigned offset, unsigned size)
{
unsigned char c, t;
unsigned i = 0;
@@ -569,19 +570,19 @@ check_buffers(unsigned offset, unsigned size)
unsigned op = 0;
unsigned bad = 0;
- if (memcmp(good_buf + offset, temp_buf, size) != 0) {
+ if (memcmp(good_buf + offset, buf, size) != 0) {
prt("READ BAD DATA: offset = 0x%x, size = 0x%x, fname = %s\n",
offset, size, fname);
prt("OFFSET\tGOOD\tBAD\tRANGE\n");
while (size > 0) {
c = good_buf[offset];
- t = temp_buf[i];
+ t = buf[i];
if (c != t) {
if (n < 16) {
- bad = short_at(&temp_buf[i]);
+ bad = short_at(&buf[i]);
prt("0x%05x\t0x%04x\t0x%04x", offset,
short_at(&good_buf[offset]), bad);
- op = temp_buf[offset & 1 ? i+1 : i];
+ op = buf[offset & 1 ? i+1 : i];
prt("\t0x%05x\n", n);
if (op)
prt("operation# (mod 256) for "
@@ -725,7 +726,46 @@ doread(unsigned offset, unsigned size)
iret, size);
report_failure(141);
}
- check_buffers(offset, size);
+ check_buffers(temp_buf, offset, size);
+}
+
+void
+check_contents(void)
+{
+ static char *check_buf;
+ unsigned offset = 0;
+ unsigned size = file_size;
+ off_t ret;
+ unsigned iret;
+
+ if (!check_buf) {
+ check_buf = (char *) malloc(maxfilelen + writebdy);
+ assert(check_buf != NULL);
+ check_buf = round_ptr_up(check_buf, writebdy, 0);
+ memset(check_buf, '\0', maxfilelen);
+ }
+
+ if (o_direct)
+ size -= size % readbdy;
+ if (size == 0)
+ return;
+
+ ret = lseek(fd, (off_t)offset, SEEK_SET);
+ if (ret == (off_t)-1) {
+ prterr("doread: lseek");
+ report_failure(140);
+ }
+
+ iret = fsxread(fd, check_buf, size, offset);
+ if (iret != size) {
+ if (iret == -1)
+ prterr("check_contents: read");
+ else
+ prt("short check read: 0x%x bytes instead of 0x%x\n",
+ iret, size);
+ report_failure(141);
+ }
+ check_buffers(check_buf, offset, size);
}
@@ -808,7 +848,7 @@ domapread(unsigned offset, unsigned size)
report_failure(191);
}
- check_buffers(offset, size);
+ check_buffers(temp_buf, offset, size);
}
@@ -1624,6 +1664,9 @@ test(void)
break;
}
+ if (check_file)
+ check_contents();
+
out:
if (sizechecks && testcalls > simulatedopcount)
check_size();
@@ -1684,6 +1727,7 @@ usage(void)
-P: save .fsxlog .fsxops and .fsxgood files in dirpath (default ./)\n\
-S seed: for random # generator (default 1) 0 gets timestamp\n\
-W: mapped write operations DISabled\n\
+ -X: Do not read file and compare to good buffer after every operation.\n\
-R: read() system calls only (mapped reads disabled)\n\
-Z: O_DIRECT (use -R, -W, -r and -w too)\n\
--replay-ops opsfile: replay ops from recorded .fsxops file\n\
@@ -1880,7 +1924,7 @@ main(int argc, char **argv)
setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
while ((ch = getopt_long(argc, argv,
- "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
+ "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
longopts, NULL)) != EOF)
switch (ch) {
case 'b':
@@ -2044,6 +2088,9 @@ main(int argc, char **argv)
if (!quiet)
prt("mapped writes DISABLED\n");
break;
+ case 'X':
+ check_file = 0;
+ break;
case 'Z':
o_direct = O_DIRECT;
o_flags |= O_DIRECT;
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 03/10] fsx: use an enum to define the operation commands
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
2018-11-20 21:56 ` [PATCH 01/10] fsx: shut up compiler warnings Darrick J. Wong
2018-11-20 21:56 ` [PATCH 02/10] fsx: always check buffer after each operation Darrick J. Wong
@ 2018-11-20 21:56 ` Darrick J. Wong
2018-11-20 21:57 ` [PATCH 04/10] fsx: add five-argument logging function Darrick J. Wong
` (8 subsequent siblings)
11 siblings, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:56 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests
From: Darrick J. Wong <darrick.wong@oracle.com>
Use an enum to define operation codes and the boundaries between
operation classes so that we can add new commands without having to
change a bunch of unrelated #defines.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
ltp/fsx.c | 40 +++++++++++++++++++++-------------------
1 file changed, 21 insertions(+), 19 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index 4b3a5d5a..290b42f6 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -88,25 +88,27 @@ int logcount = 0; /* total ops */
* mode being run.
*/
-/* common operations */
-#define OP_READ 0
-#define OP_WRITE 1
-#define OP_MAPREAD 2
-#define OP_MAPWRITE 3
-#define OP_MAX_LITE 4
-
-/* !lite operations */
-#define OP_TRUNCATE 4
-#define OP_FALLOCATE 5
-#define OP_PUNCH_HOLE 6
-#define OP_ZERO_RANGE 7
-#define OP_COLLAPSE_RANGE 8
-#define OP_INSERT_RANGE 9
-#define OP_MAX_FULL 10
-
-/* integrity operations */
-#define OP_FSYNC 10
-#define OP_MAX_INTEGRITY 11
+enum {
+ /* common operations */
+ OP_READ = 0,
+ OP_WRITE,
+ OP_MAPREAD,
+ OP_MAPWRITE,
+ OP_MAX_LITE,
+
+ /* !lite operations */
+ OP_TRUNCATE = OP_MAX_LITE,
+ OP_FALLOCATE,
+ OP_PUNCH_HOLE,
+ OP_ZERO_RANGE,
+ OP_COLLAPSE_RANGE,
+ OP_INSERT_RANGE,
+ OP_MAX_FULL,
+
+ /* integrity operations */
+ OP_FSYNC = OP_MAX_FULL,
+ OP_MAX_INTEGRITY,
+};
#undef PAGE_SIZE
#define PAGE_SIZE getpagesize()
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 04/10] fsx: add five-argument logging function
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
` (2 preceding siblings ...)
2018-11-20 21:56 ` [PATCH 03/10] fsx: use an enum to define the operation commands Darrick J. Wong
@ 2018-11-20 21:57 ` Darrick J. Wong
2018-11-20 21:57 ` [PATCH 05/10] fsx: add clone range Darrick J. Wong
` (7 subsequent siblings)
11 siblings, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:57 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests
From: Darrick J. Wong <darrick.wong@oracle.com>
Add a five-argument logging function to support new operations.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
ltp/fsx.c | 45 ++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 40 insertions(+), 5 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index 290b42f6..2372964d 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -51,7 +51,8 @@ enum opflags { FL_NONE = 0, FL_SKIPPED = 1, FL_CLOSE_OPEN = 2, FL_KEEP_SIZE = 4
struct log_entry {
int operation;
- int args[3];
+ int nr_args;
+ int args[4];
enum opflags flags;
};
@@ -278,6 +279,27 @@ static int op_code(const char *name)
return -1;
}
+void
+log5(int operation, int arg0, int arg1, int arg2, enum opflags flags)
+{
+ struct log_entry *le;
+
+ le = &oplog[logptr];
+ le->operation = operation;
+ if (closeopen)
+ flags |= FL_CLOSE_OPEN;
+ le->args[0] = arg0;
+ le->args[1] = arg1;
+ le->args[2] = arg2;
+ le->args[3] = file_size;
+ le->nr_args = 4;
+ le->flags = flags;
+ logptr++;
+ logcount++;
+ if (logptr >= LOGSIZE)
+ logptr = 0;
+}
+
void
log4(int operation, int arg0, int arg1, enum opflags flags)
{
@@ -290,6 +312,7 @@ log4(int operation, int arg0, int arg1, enum opflags flags)
le->args[0] = arg0;
le->args[1] = arg1;
le->args[2] = file_size;
+ le->nr_args = 3;
le->flags = flags;
logptr++;
logcount++;
@@ -439,11 +462,13 @@ logdump(void)
i = 0;
if (logopsf) {
+ int j;
+
if (lp->flags & FL_SKIPPED)
fprintf(logopsf, "skip ");
- fprintf(logopsf, "%s 0x%x 0x%x 0x%x",
- op_name(lp->operation),
- lp->args[0], lp->args[1], lp->args[2]);
+ fprintf(logopsf, "%s", op_name(lp->operation));
+ for (j = 0; j < lp->nr_args; j++)
+ fprintf(logopsf, " 0x%x", lp->args[j]);
if (lp->flags & FL_KEEP_SIZE)
fprintf(logopsf, " keep_size");
if (lp->flags & FL_CLOSE_OPEN)
@@ -1413,6 +1438,15 @@ cleanup(int sig)
exit(sig);
}
+static int
+op_args_count(int operation)
+{
+ switch (operation) {
+ default:
+ return 3;
+ }
+}
+
static int
read_op(struct log_entry *log_entry)
{
@@ -1445,7 +1479,8 @@ read_op(struct log_entry *log_entry)
log_entry->operation = op_code(str);
if (log_entry->operation == -1)
goto fail;
- for (i = 0; i < 3; i++) {
+ log_entry->nr_args = op_args_count(log_entry->operation);
+ for (i = 0; i < log_entry->nr_args; i++) {
char *end;
str = strtok(NULL, " \t\n");
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 05/10] fsx: add clone range
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
` (3 preceding siblings ...)
2018-11-20 21:57 ` [PATCH 04/10] fsx: add five-argument logging function Darrick J. Wong
@ 2018-11-20 21:57 ` Darrick J. Wong
2018-11-22 2:14 ` Darrick J. Wong
2018-11-20 21:57 ` [PATCH 06/10] fsx: add FIDEDUPERANGE support Darrick J. Wong
` (6 subsequent siblings)
11 siblings, 1 reply; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:57 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests
From: Darrick J. Wong <darrick.wong@oracle.com>
Add support for FICLONERANGE to fsx.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
ltp/fsx.c | 133 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 128 insertions(+), 5 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index 2372964d..a1a44b23 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -104,6 +104,7 @@ enum {
OP_ZERO_RANGE,
OP_COLLAPSE_RANGE,
OP_INSERT_RANGE,
+ OP_CLONE_RANGE,
OP_MAX_FULL,
/* integrity operations */
@@ -165,6 +166,7 @@ int collapse_range_calls = 1; /* -C flag disables */
int insert_range_calls = 1; /* -I flag disables */
int mapped_reads = 1; /* -R flag disables it */
int check_file = 1; /* -X flag disables */
+int clone_range_calls = 1; /* -J flag disables */
int integrity = 0; /* -i flag */
int fsxgoodfd = 0;
int o_direct; /* -Z */
@@ -258,6 +260,7 @@ static const char *op_names[] = {
[OP_ZERO_RANGE] = "zero_range",
[OP_COLLAPSE_RANGE] = "collapse_range",
[OP_INSERT_RANGE] = "insert_range",
+ [OP_CLONE_RANGE] = "clone_range",
[OP_FSYNC] = "fsync",
};
@@ -320,7 +323,6 @@ log4(int operation, int arg0, int arg1, enum opflags flags)
logptr = 0;
}
-
void
logdump(void)
{
@@ -342,7 +344,7 @@ logdump(void)
count = LOGSIZE;
}
for ( ; count > 0; count--) {
- bool overlap;
+ bool overlap, overlap2;
int opnum;
opnum = i+1 + (logcount/LOGSIZE)*LOGSIZE;
@@ -444,6 +446,20 @@ logdump(void)
if (overlap)
prt("\t******IIII");
break;
+ case OP_CLONE_RANGE:
+ prt("CLONE 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
+ lp->args[0], lp->args[0] + lp->args[1] - 1,
+ lp->args[1],
+ lp->args[2], lp->args[2] + lp->args[1] - 1);
+ overlap2 = badoff >= lp->args[2] &&
+ badoff < lp->args[2] + lp->args[1];
+ if (overlap && overlap2)
+ prt("\tJJJJ**JJJJ");
+ else if (overlap)
+ prt("\tJJJJ******");
+ else if (overlap2)
+ prt("\t******JJJJ");
+ break;
case OP_FSYNC:
prt("FSYNC");
break;
@@ -1285,6 +1301,73 @@ do_insert_range(unsigned offset, unsigned length)
}
#endif
+#ifdef FICLONERANGE
+void
+do_clone_range(unsigned offset, unsigned length, unsigned dest)
+{
+ struct file_clone_range fcr;
+
+ if (length == 0) {
+ if (!quiet && testcalls > simulatedopcount)
+ prt("skipping zero length clone range\n");
+ log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
+ return;
+ }
+
+ if ((loff_t)offset >= file_size) {
+ if (!quiet && testcalls > simulatedopcount)
+ prt("skipping clone range behind EOF\n");
+ log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
+ return;
+ }
+
+ log5(OP_CLONE_RANGE, offset, length, dest, FL_NONE);
+
+ if (testcalls <= simulatedopcount)
+ return;
+
+ if ((progressinterval && testcalls % progressinterval == 0) ||
+ (debug && (monitorstart == -1 || monitorend == -1 ||
+ dest <= monitorstart || dest + length <= monitorend))) {
+ prt("%lu clone\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
+ testcalls, offset, offset+length, length, dest);
+ }
+
+ fcr.src_fd = fd;
+ fcr.src_offset = offset;
+ fcr.src_length = length;
+ fcr.dest_offset = dest;
+
+ if (ioctl(fd, FICLONERANGE, &fcr) == -1) {
+ if (errno == EOPNOTSUPP || errno == ENOTTY) {
+ if (!quiet && testcalls > simulatedopcount)
+ prt("skipping unsupported clone range\n");
+ logptr--;
+ log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
+ return;
+ }
+
+ prt("clone range: 0x%x to 0x%x at 0x%x\n", offset,
+ offset + length, dest);
+ prterr("do_clone_range: FICLONERANGE");
+ report_failure(161);
+ }
+
+ memcpy(good_buf + dest, good_buf + offset, length);
+ if (dest > file_size)
+ memset(good_buf + file_size, '\0', dest - file_size);
+ if (dest + length > file_size)
+ file_size = dest + length;
+}
+
+#else
+void
+do_clone_range(unsigned offset, unsigned length, unsigned dest)
+{
+ return;
+}
+#endif
+
#ifdef HAVE_LINUX_FALLOC_H
/* fallocate is basically a no-op unless extending, then a lot like a truncate */
void
@@ -1442,6 +1525,8 @@ static int
op_args_count(int operation)
{
switch (operation) {
+ case OP_CLONE_RANGE:
+ return 4;
default:
return 3;
}
@@ -1450,6 +1535,7 @@ op_args_count(int operation)
static int
read_op(struct log_entry *log_entry)
{
+ char *end;
char line[256];
memset(log_entry, 0, sizeof(*log_entry));
@@ -1515,7 +1601,7 @@ read_op(struct log_entry *log_entry)
int
test(void)
{
- unsigned long offset;
+ unsigned long offset, offset2;
unsigned long size;
unsigned long rv;
unsigned long op;
@@ -1546,6 +1632,7 @@ test(void)
op = log_entry.operation;
offset = log_entry.args[0];
size = log_entry.args[1];
+ offset2 = log_entry.args[2];
closeopen = !!(log_entry.flags & FL_CLOSE_OPEN);
keep_size = !!(log_entry.flags & FL_KEEP_SIZE);
goto have_op;
@@ -1558,6 +1645,7 @@ test(void)
closeopen = (rv >> 3) < (1 << 28) / closeprob;
offset = random();
+ offset2 = 0;
size = maxoplen;
if (randomoplen)
size = random() % (maxoplen + 1);
@@ -1583,6 +1671,17 @@ test(void)
if (zero_range_calls && size && keep_size_calls)
keep_size = random() % 2;
break;
+ case OP_CLONE_RANGE:
+ TRIM_OFF_LEN(offset, size, file_size);
+ offset = offset & ~(block_size - 1);
+ size = size & ~(block_size - 1);
+ do {
+ offset2 = random();
+ TRIM_OFF(offset2, maxfilelen);
+ offset2 = offset2 & ~(block_size - 1);
+ } while (llabs(offset2 - offset) < size ||
+ offset2 + size > maxfilelen);
+ break;
}
have_op:
@@ -1626,6 +1725,12 @@ test(void)
goto out;
}
break;
+ case OP_CLONE_RANGE:
+ if (!clone_range_calls) {
+ log5(op, offset, size, offset2, FL_SKIPPED);
+ goto out;
+ }
+ break;
}
switch (op) {
@@ -1692,6 +1797,18 @@ test(void)
do_insert_range(offset, size);
break;
+ case OP_CLONE_RANGE:
+ if (size == 0) {
+ log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED);
+ goto out;
+ }
+ if (offset2 + size > maxfilelen) {
+ log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED);
+ goto out;
+ }
+
+ do_clone_range(offset, size, offset2);
+ break;
case OP_FSYNC:
dofsync();
break;
@@ -1717,7 +1834,7 @@ void
usage(void)
{
fprintf(stdout, "usage: %s",
- "fsx [-dknqxAFLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+ "fsx [-dknqxAFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
-b opnum: beginning operation number (default 1)\n\
-c P: 1 in P chance of file close+open at each op (default infinity)\n\
-d: debug output for all operations\n\
@@ -1758,6 +1875,9 @@ usage(void)
#ifdef FALLOC_FL_INSERT_RANGE
" -I: Do not use insert range calls\n"
#endif
+#ifdef FICLONERANGE
+" -J: Do not use clone range calls\n"
+#endif
" -L: fsxLite - no file creations & no file size changes\n\
-N numops: total # operations to do (default infinity)\n\
-O: use oplen (see -o flag) for every op (default random)\n\
@@ -1961,7 +2081,7 @@ main(int argc, char **argv)
setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
while ((ch = getopt_long(argc, argv,
- "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
+ "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FJKHzCILN:OP:RS:WXZ",
longopts, NULL)) != EOF)
switch (ch) {
case 'b':
@@ -2091,6 +2211,9 @@ main(int argc, char **argv)
case 'I':
insert_range_calls = 0;
break;
+ case 'J':
+ clone_range_calls = 0;
+ break;
case 'L':
lite = 1;
o_flags &= ~(O_CREAT|O_TRUNC);
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 06/10] fsx: add FIDEDUPERANGE support
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
` (4 preceding siblings ...)
2018-11-20 21:57 ` [PATCH 05/10] fsx: add clone range Darrick J. Wong
@ 2018-11-20 21:57 ` Darrick J. Wong
2018-11-20 21:57 ` [PATCH 07/10] fsstress: add copy_file_range support Darrick J. Wong
` (5 subsequent siblings)
11 siblings, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:57 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests
From: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
ltp/fsx.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 139 insertions(+), 2 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index a1a44b23..6f6263e2 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -105,6 +105,7 @@ enum {
OP_COLLAPSE_RANGE,
OP_INSERT_RANGE,
OP_CLONE_RANGE,
+ OP_DEDUPE_RANGE,
OP_MAX_FULL,
/* integrity operations */
@@ -167,6 +168,7 @@ int insert_range_calls = 1; /* -I flag disables */
int mapped_reads = 1; /* -R flag disables it */
int check_file = 1; /* -X flag disables */
int clone_range_calls = 1; /* -J flag disables */
+int dedupe_range_calls = 1; /* -B flag disables */
int integrity = 0; /* -i flag */
int fsxgoodfd = 0;
int o_direct; /* -Z */
@@ -261,6 +263,7 @@ static const char *op_names[] = {
[OP_COLLAPSE_RANGE] = "collapse_range",
[OP_INSERT_RANGE] = "insert_range",
[OP_CLONE_RANGE] = "clone_range",
+ [OP_DEDUPE_RANGE] = "dedupe_range",
[OP_FSYNC] = "fsync",
};
@@ -460,6 +463,20 @@ logdump(void)
else if (overlap2)
prt("\t******JJJJ");
break;
+ case OP_DEDUPE_RANGE:
+ prt("DEDUPE 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
+ lp->args[0], lp->args[0] + lp->args[1] - 1,
+ lp->args[1],
+ lp->args[2], lp->args[2] + lp->args[1] - 1);
+ overlap2 = badoff >= lp->args[2] &&
+ badoff < lp->args[2] + lp->args[1];
+ if (overlap && overlap2)
+ prt("\tBBBB**BBBB");
+ else if (overlap)
+ prt("\tBBBB******");
+ else if (overlap2)
+ prt("\t******BBBB");
+ break;
case OP_FSYNC:
prt("FSYNC");
break;
@@ -1368,6 +1385,82 @@ do_clone_range(unsigned offset, unsigned length, unsigned dest)
}
#endif
+#ifdef FIDEDUPERANGE
+void
+do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
+{
+ struct file_dedupe_range *fdr;
+
+ if (length == 0) {
+ if (!quiet && testcalls > simulatedopcount)
+ prt("skipping zero length dedupe range\n");
+ log5(OP_DEDUPE_RANGE, offset, length, dest, FL_SKIPPED);
+ return;
+ }
+
+ if ((loff_t)offset >= file_size) {
+ if (!quiet && testcalls > simulatedopcount)
+ prt("skipping dedupe range behind EOF\n");
+ log5(OP_DEDUPE_RANGE, offset, length, dest, FL_SKIPPED);
+ return;
+ }
+
+ log5(OP_DEDUPE_RANGE, offset, length, dest, FL_NONE);
+
+ if (testcalls <= simulatedopcount)
+ return;
+
+ if ((progressinterval && testcalls % progressinterval == 0) ||
+ (debug && (monitorstart == -1 || monitorend == -1 ||
+ dest <= monitorstart || dest + length <= monitorend))) {
+ prt("%lu dedupe\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
+ testcalls, offset, offset+length, length, dest);
+ }
+
+ /* Alloc memory */
+ fdr = malloc(sizeof(struct file_dedupe_range_info) +
+ sizeof(struct file_dedupe_range));
+ if (!fdr) {
+ prterr("do_dedupe_range: malloc");
+ report_failure(161);
+ }
+ memset(fdr, 0, (sizeof(struct file_dedupe_range_info) +
+ sizeof(struct file_dedupe_range)));
+
+ /* Dedupe data blocks */
+ fdr->src_offset = offset;
+ fdr->src_length = length;
+ fdr->dest_count = 1;
+ fdr->info[0].dest_fd = fd;
+ fdr->info[0].dest_offset = dest;
+
+ if (ioctl(fd, FIDEDUPERANGE, fdr) == -1) {
+ if (errno == EOPNOTSUPP || errno == ENOTTY) {
+ if (!quiet && testcalls > simulatedopcount)
+ prt("skipping unsupported dedupe range\n");
+ logptr--;
+ log5(OP_DEDUPE_RANGE, offset, length, dest, FL_SKIPPED);
+ free(fdr);
+ return;
+ }
+
+ prt("dedupe range: 0x%x to 0x%x at 0x%x\n", offset,
+ offset + length, dest);
+ prterr("do_dedupe_range: FIDEDUPERANGE");
+ report_failure(161);
+ }
+
+ free(fdr);
+}
+
+#else
+void
+do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
+{
+ return;
+}
+#endif
+
#ifdef HAVE_LINUX_FALLOC_H
/* fallocate is basically a no-op unless extending, then a lot like a truncate */
void
@@ -1526,6 +1619,7 @@ op_args_count(int operation)
{
switch (operation) {
case OP_CLONE_RANGE:
+ case OP_DEDUPE_RANGE:
return 4;
default:
return 3;
@@ -1682,6 +1776,25 @@ test(void)
} while (llabs(offset2 - offset) < size ||
offset2 + size > maxfilelen);
break;
+ case OP_DEDUPE_RANGE:
+ {
+ int tries = 0;
+
+ TRIM_OFF_LEN(offset, size, file_size);
+ offset = offset & ~(block_size - 1);
+ size = size & ~(block_size - 1);
+ do {
+ if (tries++ >= 30) {
+ size = 0;
+ break;
+ }
+ offset2 = random();
+ TRIM_OFF(offset2, file_size);
+ offset2 = offset2 & ~(block_size - 1);
+ } while (llabs(offset2 - offset) < size ||
+ offset2 + size > file_size);
+ break;
+ }
}
have_op:
@@ -1731,6 +1844,12 @@ test(void)
goto out;
}
break;
+ case OP_DEDUPE_RANGE:
+ if (!dedupe_range_calls) {
+ log5(op, offset, size, offset2, FL_SKIPPED);
+ goto out;
+ }
+ break;
}
switch (op) {
@@ -1809,6 +1928,18 @@ test(void)
do_clone_range(offset, size, offset2);
break;
+ case OP_DEDUPE_RANGE:
+ if (size == 0) {
+ log5(OP_DEDUPE_RANGE, offset, size, offset2, FL_SKIPPED);
+ goto out;
+ }
+ if (offset2 + size > maxfilelen) {
+ log5(OP_DEDUPE_RANGE, offset, size, offset2, FL_SKIPPED);
+ goto out;
+ }
+
+ do_dedupe_range(offset, size, offset2);
+ break;
case OP_FSYNC:
dofsync();
break;
@@ -1834,7 +1965,7 @@ void
usage(void)
{
fprintf(stdout, "usage: %s",
- "fsx [-dknqxAFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+ "fsx [-dknqxABFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
-b opnum: beginning operation number (default 1)\n\
-c P: 1 in P chance of file close+open at each op (default infinity)\n\
-d: debug output for all operations\n\
@@ -1878,6 +2009,9 @@ usage(void)
#ifdef FICLONERANGE
" -J: Do not use clone range calls\n"
#endif
+#ifdef FIDEDUPERANGE
+" -B: Do not use dedupe range calls\n"
+#endif
" -L: fsxLite - no file creations & no file size changes\n\
-N numops: total # operations to do (default infinity)\n\
-O: use oplen (see -o flag) for every op (default random)\n\
@@ -2081,7 +2215,7 @@ main(int argc, char **argv)
setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
while ((ch = getopt_long(argc, argv,
- "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FJKHzCILN:OP:RS:WXZ",
+ "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:FJKHzCILN:OP:RS:WXZ",
longopts, NULL)) != EOF)
switch (ch) {
case 'b':
@@ -2214,6 +2348,9 @@ main(int argc, char **argv)
case 'J':
clone_range_calls = 0;
break;
+ case 'B':
+ dedupe_range_calls = 0;
+ break;
case 'L':
lite = 1;
o_flags &= ~(O_CREAT|O_TRUNC);
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 07/10] fsstress: add copy_file_range support
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
` (5 preceding siblings ...)
2018-11-20 21:57 ` [PATCH 06/10] fsx: add FIDEDUPERANGE support Darrick J. Wong
@ 2018-11-20 21:57 ` Darrick J. Wong
2018-11-20 21:57 ` [PATCH 08/10] fsx: " Darrick J. Wong
` (4 subsequent siblings)
11 siblings, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:57 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests
From: Darrick J. Wong <darrick.wong@oracle.com>
Support the copy_file_range syscall.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
configure.ac | 2 +
include/builddefs.in | 1
ltp/Makefile | 4 +
ltp/fsstress.c | 133 +++++++++++++++++++++++++++++++++++++++++++++++++
m4/package_libcdev.m4 | 19 +++++++
5 files changed, 159 insertions(+)
diff --git a/configure.ac b/configure.ac
index aede4f59..19798824 100644
--- a/configure.ac
+++ b/configure.ac
@@ -68,6 +68,8 @@ AC_PACKAGE_WANT_OPEN_BY_HANDLE_AT
AC_PACKAGE_WANT_LINUX_PRCTL_H
AC_PACKAGE_WANT_LINUX_FS_H
+AC_HAVE_COPY_FILE_RANGE
+
AC_CHECK_FUNCS([renameat2])
AC_CONFIG_HEADER(include/config.h)
diff --git a/include/builddefs.in b/include/builddefs.in
index fb8e912b..2605e42d 100644
--- a/include/builddefs.in
+++ b/include/builddefs.in
@@ -67,6 +67,7 @@ HAVE_DMAPI = @have_dmapi@
HAVE_ATTR_LIST = @have_attr_list@
HAVE_FIEMAP = @have_fiemap@
HAVE_FALLOCATE = @have_fallocate@
+HAVE_COPY_FILE_RANGE = @have_copy_file_range@
GCCFLAGS = -funsigned-char -fno-strict-aliasing -Wall
diff --git a/ltp/Makefile b/ltp/Makefile
index 5f784ecf..e4ca45f4 100644
--- a/ltp/Makefile
+++ b/ltp/Makefile
@@ -28,6 +28,10 @@ ifeq ($(HAVE_FALLOCATE), true)
LCFLAGS += -DFALLOCATE
endif
+ifeq ($(HAVE_COPY_FILE_RANGE),yes)
+LCFLAGS += -DHAVE_COPY_FILE_RANGE
+endif
+
default: depend $(TARGETS)
depend: .dep
diff --git a/ltp/fsstress.c b/ltp/fsstress.c
index edbb3f2f..dded0cf7 100644
--- a/ltp/fsstress.c
+++ b/ltp/fsstress.c
@@ -26,6 +26,7 @@
#include <libaio.h>
io_context_t io_ctx;
#endif
+#include <sys/syscall.h>
#ifndef FS_IOC_GETFLAGS
#define FS_IOC_GETFLAGS _IOR('f', 1, long)
@@ -55,6 +56,7 @@ typedef enum {
OP_BULKSTAT1,
OP_CHOWN,
OP_CLONERANGE,
+ OP_COPYRANGE,
OP_CREAT,
OP_DEDUPERANGE,
OP_DREAD,
@@ -163,6 +165,7 @@ void bulkstat_f(int, long);
void bulkstat1_f(int, long);
void chown_f(int, long);
void clonerange_f(int, long);
+void copyrange_f(int, long);
void creat_f(int, long);
void deduperange_f(int, long);
void dread_f(int, long);
@@ -212,6 +215,7 @@ opdesc_t ops[] = {
{ OP_BULKSTAT1, "bulkstat1", bulkstat1_f, 1, 0 },
{ OP_CHOWN, "chown", chown_f, 3, 1 },
{ OP_CLONERANGE, "clonerange", clonerange_f, 4, 1 },
+ { OP_COPYRANGE, "copyrange", copyrange_f, 4, 1 },
{ OP_CREAT, "creat", creat_f, 4, 1 },
{ OP_DEDUPERANGE, "deduperange", deduperange_f, 4, 1},
{ OP_DREAD, "dread", dread_f, 4, 0 },
@@ -2329,6 +2333,135 @@ clonerange_f(
#endif
}
+/* copy some arbitrary range of f1 to f2. */
+void
+copyrange_f(
+ int opno,
+ long r)
+{
+#ifdef HAVE_COPY_FILE_RANGE
+ struct pathname fpath1;
+ struct pathname fpath2;
+ struct stat64 stat1;
+ struct stat64 stat2;
+ char inoinfo1[1024];
+ char inoinfo2[1024];
+ loff_t lr;
+ loff_t off1;
+ loff_t off2;
+ loff_t max_off2;
+ size_t len;
+ int v1;
+ int v2;
+ int fd1;
+ int fd2;
+ int ret;
+ int e;
+
+ /* Load paths */
+ init_pathname(&fpath1);
+ if (!get_fname(FT_REGm, r, &fpath1, NULL, NULL, &v1)) {
+ if (v1)
+ printf("%d/%d: copyrange read - no filename\n",
+ procid, opno);
+ goto out_fpath1;
+ }
+
+ init_pathname(&fpath2);
+ if (!get_fname(FT_REGm, random(), &fpath2, NULL, NULL, &v2)) {
+ if (v2)
+ printf("%d/%d: copyrange write - no filename\n",
+ procid, opno);
+ goto out_fpath2;
+ }
+
+ /* Open files */
+ fd1 = open_path(&fpath1, O_RDONLY);
+ e = fd1 < 0 ? errno : 0;
+ check_cwd();
+ if (fd1 < 0) {
+ if (v1)
+ printf("%d/%d: copyrange read - open %s failed %d\n",
+ procid, opno, fpath1.path, e);
+ goto out_fpath2;
+ }
+
+ fd2 = open_path(&fpath2, O_WRONLY);
+ e = fd2 < 0 ? errno : 0;
+ check_cwd();
+ if (fd2 < 0) {
+ if (v2)
+ printf("%d/%d: copyrange write - open %s failed %d\n",
+ procid, opno, fpath2.path, e);
+ goto out_fd1;
+ }
+
+ /* Get file stats */
+ if (fstat64(fd1, &stat1) < 0) {
+ if (v1)
+ printf("%d/%d: copyrange read - fstat64 %s failed %d\n",
+ procid, opno, fpath1.path, errno);
+ goto out_fd2;
+ }
+ inode_info(inoinfo1, sizeof(inoinfo1), &stat1, v1);
+
+ if (fstat64(fd2, &stat2) < 0) {
+ if (v2)
+ printf("%d/%d: copyrange write - fstat64 %s failed %d\n",
+ procid, opno, fpath2.path, errno);
+ goto out_fd2;
+ }
+ inode_info(inoinfo2, sizeof(inoinfo2), &stat2, v2);
+
+ /* Calculate offsets */
+ len = (random() % FILELEN_MAX) + 1;
+ if (len == 0)
+ len = stat1.st_blksize;
+ if (len > stat1.st_size)
+ len = stat1.st_size;
+
+ lr = ((int64_t)random() << 32) + random();
+ if (stat1.st_size == len)
+ off1 = 0;
+ else
+ off1 = (off64_t)(lr % MIN(stat1.st_size - len, MAXFSIZE));
+ off1 %= maxfsize;
+
+ /*
+ * If srcfile == destfile, randomly generate destination ranges
+ * until we find one that doesn't overlap the source range.
+ */
+ max_off2 = MIN(stat2.st_size + (1024ULL * stat2.st_blksize), MAXFSIZE);
+ do {
+ lr = ((int64_t)random() << 32) + random();
+ off2 = (off64_t)(lr % max_off2);
+ off2 %= maxfsize;
+ } while (stat1.st_ino == stat2.st_ino && llabs(off2 - off1) < len);
+
+ ret = copy_file_range(fd1, &off1, fd2, &off2, len, 0);
+ e = ret < 0 ? errno : 0;
+ if (v1 || v2) {
+ printf("%d/%d: copyrange %s%s [%lld,%lld] -> %s%s [%lld,%lld]",
+ procid, opno,
+ fpath1.path, inoinfo1, (long long)off1, (long long)len,
+ fpath2.path, inoinfo2, (long long)off2, (long long)len);
+
+ if (ret < 0)
+ printf(" error %d", e);
+ printf("\n");
+ }
+
+out_fd2:
+ close(fd2);
+out_fd1:
+ close(fd1);
+out_fpath2:
+ free_pathname(&fpath2);
+out_fpath1:
+ free_pathname(&fpath1);
+#endif
+}
+
/* dedupe some arbitrary range of f1 to f2...fn. */
void
deduperange_f(
diff --git a/m4/package_libcdev.m4 b/m4/package_libcdev.m4
index fb123f33..14e67e18 100644
--- a/m4/package_libcdev.m4
+++ b/m4/package_libcdev.m4
@@ -98,3 +98,22 @@ AC_DEFUN([AC_HAVE_GETMNTINFO],
AC_MSG_RESULT(no))
AC_SUBST(have_getmntinfo)
])
+
+#
+#
+# Check if we have a copy_file_range system call (Linux)
+#
+AC_DEFUN([AC_HAVE_COPY_FILE_RANGE],
+ [ AC_MSG_CHECKING([for copy_file_range])
+ AC_TRY_LINK([
+#define _GNU_SOURCE
+#include <sys/syscall.h>
+#include <unistd.h>
+ ], [
+ syscall(__NR_copy_file_range, 0, 0, 0, 0, 0, 0);
+ ], have_copy_file_range=yes
+ AC_MSG_RESULT(yes),
+ AC_MSG_RESULT(no))
+ AC_SUBST(have_copy_file_range)
+ ])
+
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 08/10] fsx: add copy_file_range support
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
` (6 preceding siblings ...)
2018-11-20 21:57 ` [PATCH 07/10] fsstress: add copy_file_range support Darrick J. Wong
@ 2018-11-20 21:57 ` Darrick J. Wong
2018-11-20 21:57 ` [PATCH 09/10] fsx: clean up copy/dedupe file range support Darrick J. Wong
` (3 subsequent siblings)
11 siblings, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:57 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests
From: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
ltp/fsx.c | 122 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 120 insertions(+), 2 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index 6f6263e2..ec3defad 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -34,6 +34,7 @@
#ifdef AIO
#include <libaio.h>
#endif
+#include <sys/syscall.h>
#ifndef MAP_FILE
# define MAP_FILE 0
@@ -106,6 +107,7 @@ enum {
OP_INSERT_RANGE,
OP_CLONE_RANGE,
OP_DEDUPE_RANGE,
+ OP_COPY_RANGE,
OP_MAX_FULL,
/* integrity operations */
@@ -169,6 +171,7 @@ int mapped_reads = 1; /* -R flag disables it */
int check_file = 1; /* -X flag disables */
int clone_range_calls = 1; /* -J flag disables */
int dedupe_range_calls = 1; /* -B flag disables */
+int copy_range_calls = 1; /* -E flag disables */
int integrity = 0; /* -i flag */
int fsxgoodfd = 0;
int o_direct; /* -Z */
@@ -264,6 +267,7 @@ static const char *op_names[] = {
[OP_INSERT_RANGE] = "insert_range",
[OP_CLONE_RANGE] = "clone_range",
[OP_DEDUPE_RANGE] = "dedupe_range",
+ [OP_COPY_RANGE] = "copy_range",
[OP_FSYNC] = "fsync",
};
@@ -477,6 +481,20 @@ logdump(void)
else if (overlap2)
prt("\t******BBBB");
break;
+ case OP_COPY_RANGE:
+ prt("COPY 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
+ lp->args[0], lp->args[0] + lp->args[1] - 1,
+ lp->args[1],
+ lp->args[2], lp->args[2] + lp->args[1] - 1);
+ overlap2 = badoff >= lp->args[2] &&
+ badoff < lp->args[2] + lp->args[1];
+ if (overlap && overlap2)
+ prt("\tEEEE**EEEE");
+ else if (overlap)
+ prt("\tEEEE******");
+ else if (overlap2)
+ prt("\t******EEEE");
+ break;
case OP_FSYNC:
prt("FSYNC");
break;
@@ -1461,6 +1479,73 @@ do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
}
#endif
+#ifdef HAVE_COPY_FILE_RANGE
+void
+do_copy_range(unsigned offset, unsigned length, unsigned dest)
+{
+ loff_t o1, o2;
+ ssize_t nr;
+
+ if (length == 0) {
+ if (!quiet && testcalls > simulatedopcount)
+ prt("skipping zero length copy range\n");
+ log5(OP_COPY_RANGE, offset, length, dest, FL_SKIPPED);
+ return;
+ }
+
+ if ((loff_t)offset >= file_size) {
+ if (!quiet && testcalls > simulatedopcount)
+ prt("skipping copy range behind EOF\n");
+ log5(OP_COPY_RANGE, offset, length, dest, FL_SKIPPED);
+ return;
+ }
+
+ log5(OP_COPY_RANGE, offset, length, dest, FL_NONE);
+
+ if (testcalls <= simulatedopcount)
+ return;
+
+ if ((progressinterval && testcalls % progressinterval == 0) ||
+ (debug && (monitorstart == -1 || monitorend == -1 ||
+ dest <= monitorstart || dest + length <= monitorend))) {
+ prt("%lu copy\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
+ testcalls, offset, offset+length, length, dest);
+ }
+
+ o1 = offset;
+ o2 = dest;
+
+ nr = copy_file_range(fd, &o1, fd, &o2, length, 0);
+ if (nr == -1) {
+ if (errno == EOPNOTSUPP || errno == ENOTTY) {
+ if (!quiet && testcalls > simulatedopcount)
+ prt("skipping unsupported copy range\n");
+ logptr--;
+ log5(OP_COPY_RANGE, offset, length, dest, FL_SKIPPED);
+ return;
+ }
+
+ prt("copy range: 0x%x to 0x%x at 0x%x\n", offset,
+ offset + length, dest);
+ prterr("do_copy_range:");
+ report_failure(161);
+ }
+
+ memcpy(good_buf + dest, good_buf + offset, nr);
+ if (dest > file_size)
+ memset(good_buf + file_size, '\0', dest - file_size);
+ if (dest + nr > file_size)
+ file_size = dest + nr;
+}
+
+#else
+void
+do_copy_range(unsigned offset, unsigned length, unsigned dest)
+{
+ return;
+}
+#endif
+
#ifdef HAVE_LINUX_FALLOC_H
/* fallocate is basically a no-op unless extending, then a lot like a truncate */
void
@@ -1620,6 +1705,7 @@ op_args_count(int operation)
switch (operation) {
case OP_CLONE_RANGE:
case OP_DEDUPE_RANGE:
+ case OP_COPY_RANGE:
return 4;
default:
return 3;
@@ -1795,6 +1881,14 @@ test(void)
offset2 + size > file_size);
break;
}
+ case OP_COPY_RANGE:
+ TRIM_OFF_LEN(offset, size, file_size);
+ do {
+ offset2 = random();
+ TRIM_OFF(offset2, maxfilelen);
+ } while (llabs(offset2 - offset) < size ||
+ offset2 + size > maxfilelen);
+ break;
}
have_op:
@@ -1850,6 +1944,12 @@ test(void)
goto out;
}
break;
+ case OP_COPY_RANGE:
+ if (!copy_range_calls) {
+ log5(op, offset, size, offset2, FL_SKIPPED);
+ goto out;
+ }
+ break;
}
switch (op) {
@@ -1940,6 +2040,18 @@ test(void)
do_dedupe_range(offset, size, offset2);
break;
+ case OP_COPY_RANGE:
+ if (size == 0) {
+ log5(OP_COPY_RANGE, offset, size, offset2, FL_SKIPPED);
+ goto out;
+ }
+ if (offset2 + size > maxfilelen) {
+ log5(OP_COPY_RANGE, offset, size, offset2, FL_SKIPPED);
+ goto out;
+ }
+
+ do_copy_range(offset, size, offset2);
+ break;
case OP_FSYNC:
dofsync();
break;
@@ -1965,7 +2077,7 @@ void
usage(void)
{
fprintf(stdout, "usage: %s",
- "fsx [-dknqxABFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+ "fsx [-dknqxABEFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
-b opnum: beginning operation number (default 1)\n\
-c P: 1 in P chance of file close+open at each op (default infinity)\n\
-d: debug output for all operations\n\
@@ -2012,6 +2124,9 @@ usage(void)
#ifdef FIDEDUPERANGE
" -B: Do not use dedupe range calls\n"
#endif
+#ifdef HAVE_COPY_FILE_RANGE
+" -E: Do not use copy range calls\n"
+#endif
" -L: fsxLite - no file creations & no file size changes\n\
-N numops: total # operations to do (default infinity)\n\
-O: use oplen (see -o flag) for every op (default random)\n\
@@ -2215,7 +2330,7 @@ main(int argc, char **argv)
setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
while ((ch = getopt_long(argc, argv,
- "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:FJKHzCILN:OP:RS:WXZ",
+ "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyABD:EFJKHzCILN:OP:RS:WXZ",
longopts, NULL)) != EOF)
switch (ch) {
case 'b':
@@ -2351,6 +2466,9 @@ main(int argc, char **argv)
case 'B':
dedupe_range_calls = 0;
break;
+ case 'E':
+ copy_range_calls = 0;
+ break;
case 'L':
lite = 1;
o_flags &= ~(O_CREAT|O_TRUNC);
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 09/10] fsx: clean up copy/dedupe file range support.
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
` (7 preceding siblings ...)
2018-11-20 21:57 ` [PATCH 08/10] fsx: " Darrick J. Wong
@ 2018-11-20 21:57 ` Darrick J. Wong
2018-11-20 21:57 ` [PATCH 10/10] common/dump: disable copyrange Darrick J. Wong
` (2 subsequent siblings)
11 siblings, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:57 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests, Dave Chinner
From: Dave Chinner <dchinner@redhat.com>
copy_file_range() needs to obey read/write constraints otherwise is
blows up when direct IO is used.
FIDEDUPERANGE has a completely screwed up API for error reporting.
The ioctl succeeds even if dedupe fails, so you have to check
every individual dedupe operations for failure. Without this, dedupe
"succeeds" on kernels filesystems that don't even support dedupe...
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
ltp/fsx.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index ec3defad..cea765d8 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -1452,7 +1452,11 @@ do_dedupe_range(unsigned offset, unsigned length, unsigned dest)
fdr->info[0].dest_fd = fd;
fdr->info[0].dest_offset = dest;
- if (ioctl(fd, FIDEDUPERANGE, fdr) == -1) {
+ if (ioctl(fd, FIDEDUPERANGE, fdr) == -1 ||
+ fdr->info[0].status < 0) {
+ if (fdr->info[0].status < 0)
+ errno = -fdr->info[0].status;
+
if (errno == EOPNOTSUPP || errno == ENOTTY) {
if (!quiet && testcalls > simulatedopcount)
prt("skipping unsupported dedupe range\n");
@@ -1486,6 +1490,11 @@ do_copy_range(unsigned offset, unsigned length, unsigned dest)
loff_t o1, o2;
ssize_t nr;
+ offset -= offset % readbdy;
+ dest -= dest % writebdy;
+ if (o_direct)
+ length -= length % readbdy;
+
if (length == 0) {
if (!quiet && testcalls > simulatedopcount)
prt("skipping zero length copy range\n");
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 10/10] common/dump: disable copyrange
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
` (8 preceding siblings ...)
2018-11-20 21:57 ` [PATCH 09/10] fsx: clean up copy/dedupe file range support Darrick J. Wong
@ 2018-11-20 21:57 ` Darrick J. Wong
2018-11-21 18:38 ` [PATCH 11/10] generic: long fsx soak tests Darrick J. Wong
2018-12-10 17:30 ` [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Luis Henriques
11 siblings, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-20 21:57 UTC (permalink / raw)
To: guaneryu, darrick.wong; +Cc: linux-xfs, fstests
From: Darrick J. Wong <darrick.wong@oracle.com>
In _create_dumpdir_stress_num, disable copyrange in fsstress so that we
dump exactly the same set of files and directories no matter how the xfs
is configured.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
common/dump | 3 +++
1 file changed, 3 insertions(+)
diff --git a/common/dump b/common/dump
index c45b8a5a..4d1a1607 100644
--- a/common/dump
+++ b/common/dump
@@ -307,6 +307,9 @@ _create_dumpdir_stress_num()
if $FSSTRESS_PROG | grep -q deduperange; then
FSSTRESS_AVOID="-f deduperange=0 $FSSTRESS_AVOID"
fi
+ if $FSSTRESS_PROG | grep -q copyrange; then
+ FSSTRESS_AVOID="-f copyrange=0 $FSSTRESS_AVOID"
+ fi
echo ""
echo "-----------------------------------------------"
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 02/10] fsx: always check buffer after each operation
2018-11-20 21:56 ` [PATCH 02/10] fsx: always check buffer after each operation Darrick J. Wong
@ 2018-11-21 1:18 ` Darrick J. Wong
2018-11-22 2:15 ` Darrick J. Wong
1 sibling, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-21 1:18 UTC (permalink / raw)
To: guaneryu; +Cc: linux-xfs, fstests
On Tue, Nov 20, 2018 at 01:56:53PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
>
> Add a new option to make fsx read the file after each operation and
> compare it with the good buffer to try to catch corruptions as soon as
> they occur.
>
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
> ltp/fsx.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
> 1 file changed, 56 insertions(+), 9 deletions(-)
>
>
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index 5601c70c..4b3a5d5a 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -160,7 +160,8 @@ int punch_hole_calls = 1; /* -H flag disables */
> int zero_range_calls = 1; /* -z flag disables */
> int collapse_range_calls = 1; /* -C flag disables */
> int insert_range_calls = 1; /* -I flag disables */
> -int mapped_reads = 1; /* -R flag disables it */
> +int mapped_reads = 1; /* -R flag disables it */
> +int check_file = 1; /* -X flag disables */
[from irc discussion with dave]
This probably ought to be inverted (i.e. 0 by default, -X sets it to 1)
so that we don't go messing with internal state.
--D
> int integrity = 0; /* -i flag */
> int fsxgoodfd = 0;
> int o_direct; /* -Z */
> @@ -561,7 +562,7 @@ dump_fsync_buffer(void)
> }
>
> void
> -check_buffers(unsigned offset, unsigned size)
> +check_buffers(char *buf, unsigned offset, unsigned size)
> {
> unsigned char c, t;
> unsigned i = 0;
> @@ -569,19 +570,19 @@ check_buffers(unsigned offset, unsigned size)
> unsigned op = 0;
> unsigned bad = 0;
>
> - if (memcmp(good_buf + offset, temp_buf, size) != 0) {
> + if (memcmp(good_buf + offset, buf, size) != 0) {
> prt("READ BAD DATA: offset = 0x%x, size = 0x%x, fname = %s\n",
> offset, size, fname);
> prt("OFFSET\tGOOD\tBAD\tRANGE\n");
> while (size > 0) {
> c = good_buf[offset];
> - t = temp_buf[i];
> + t = buf[i];
> if (c != t) {
> if (n < 16) {
> - bad = short_at(&temp_buf[i]);
> + bad = short_at(&buf[i]);
> prt("0x%05x\t0x%04x\t0x%04x", offset,
> short_at(&good_buf[offset]), bad);
> - op = temp_buf[offset & 1 ? i+1 : i];
> + op = buf[offset & 1 ? i+1 : i];
> prt("\t0x%05x\n", n);
> if (op)
> prt("operation# (mod 256) for "
> @@ -725,7 +726,46 @@ doread(unsigned offset, unsigned size)
> iret, size);
> report_failure(141);
> }
> - check_buffers(offset, size);
> + check_buffers(temp_buf, offset, size);
> +}
> +
> +void
> +check_contents(void)
> +{
> + static char *check_buf;
> + unsigned offset = 0;
> + unsigned size = file_size;
> + off_t ret;
> + unsigned iret;
> +
> + if (!check_buf) {
> + check_buf = (char *) malloc(maxfilelen + writebdy);
> + assert(check_buf != NULL);
> + check_buf = round_ptr_up(check_buf, writebdy, 0);
> + memset(check_buf, '\0', maxfilelen);
> + }
> +
> + if (o_direct)
> + size -= size % readbdy;
> + if (size == 0)
> + return;
> +
> + ret = lseek(fd, (off_t)offset, SEEK_SET);
> + if (ret == (off_t)-1) {
> + prterr("doread: lseek");
> + report_failure(140);
> + }
> +
> + iret = fsxread(fd, check_buf, size, offset);
> + if (iret != size) {
> + if (iret == -1)
> + prterr("check_contents: read");
> + else
> + prt("short check read: 0x%x bytes instead of 0x%x\n",
> + iret, size);
> + report_failure(141);
> + }
> + check_buffers(check_buf, offset, size);
> }
>
>
> @@ -808,7 +848,7 @@ domapread(unsigned offset, unsigned size)
> report_failure(191);
> }
>
> - check_buffers(offset, size);
> + check_buffers(temp_buf, offset, size);
> }
>
>
> @@ -1624,6 +1664,9 @@ test(void)
> break;
> }
>
> + if (check_file)
> + check_contents();
> +
> out:
> if (sizechecks && testcalls > simulatedopcount)
> check_size();
> @@ -1684,6 +1727,7 @@ usage(void)
> -P: save .fsxlog .fsxops and .fsxgood files in dirpath (default ./)\n\
> -S seed: for random # generator (default 1) 0 gets timestamp\n\
> -W: mapped write operations DISabled\n\
> + -X: Do not read file and compare to good buffer after every operation.\n\
> -R: read() system calls only (mapped reads disabled)\n\
> -Z: O_DIRECT (use -R, -W, -r and -w too)\n\
> --replay-ops opsfile: replay ops from recorded .fsxops file\n\
> @@ -1880,7 +1924,7 @@ main(int argc, char **argv)
> setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
>
> while ((ch = getopt_long(argc, argv,
> - "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
> + "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
> longopts, NULL)) != EOF)
> switch (ch) {
> case 'b':
> @@ -2044,6 +2088,9 @@ main(int argc, char **argv)
> if (!quiet)
> prt("mapped writes DISABLED\n");
> break;
> + case 'X':
> + check_file = 0;
> + break;
> case 'Z':
> o_direct = O_DIRECT;
> o_flags |= O_DIRECT;
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 11/10] generic: long fsx soak tests
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
` (9 preceding siblings ...)
2018-11-20 21:57 ` [PATCH 10/10] common/dump: disable copyrange Darrick J. Wong
@ 2018-11-21 18:38 ` Darrick J. Wong
2018-12-10 17:30 ` [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Luis Henriques
11 siblings, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-21 18:38 UTC (permalink / raw)
To: guaneryu; +Cc: linux-xfs, fstests
From: Darrick J. Wong <darrick.wong@oracle.com>
Add a few tests to implement long soak tests of fsx.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
tests/generic/938 | 65 +++++++++++++++++++++++++++++++++++++++++++++++++
tests/generic/938.out | 2 ++
tests/generic/939 | 60 +++++++++++++++++++++++++++++++++++++++++++++
tests/generic/939.out | 2 ++
tests/generic/group | 2 ++
5 files changed, 131 insertions(+)
create mode 100644 tests/generic/938
create mode 100644 tests/generic/938.out
create mode 100644 tests/generic/939
create mode 100644 tests/generic/939.out
diff --git a/tests/generic/938 b/tests/generic/938
new file mode 100644
index 00000000..a85a6472
--- /dev/null
+++ b/tests/generic/938
@@ -0,0 +1,65 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2018 Oracle. All Rights Reserved.
+#
+# FS QA Test 938
+#
+# Long-soak directio fsx test
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+
+# Run fsx for a million ops or more
+nr_ops=$((1000000 * TIME_FACTOR))
+op_sz=$((128000 * LOAD_FACTOR))
+file_sz=$((600000 * LOAD_FACTOR))
+fsx_file=$TEST_DIR/fsx.$seq
+min_dio_sz=$(_min_dio_alignment)
+
+fsx_args=(-q)
+fsx_args+=(-N $nr_ops)
+fsx_args+=(-p $((nr_ops / 100)))
+fsx_args+=(-o $op_sz)
+fsx_args+=(-l $file_sz)
+fsx_args+=(-r $min_dio_sz)
+fsx_args+=(-t $min_dio_sz)
+fsx_args+=(-w $min_dio_sz)
+fsx_args+=(-Z)
+fsx_args+=($fsx_file)
+
+echo "ltp ${fsx_args[@]} $FSX_ARGS" >> $seqres.full
+
+if ! ltp/fsx "${fsx_args[@]}" $FSX_ARGS > $tmp.fsx 2>&1; then
+ cat $tmp.fsx | tee -a $seqres.full
+fi
+
+# success, all done
+echo Silence is golden
+status=0
+exit
diff --git a/tests/generic/938.out b/tests/generic/938.out
new file mode 100644
index 00000000..11b565ee
--- /dev/null
+++ b/tests/generic/938.out
@@ -0,0 +1,2 @@
+QA output created by 938
+Silence is golden
diff --git a/tests/generic/939 b/tests/generic/939
new file mode 100644
index 00000000..5e341e30
--- /dev/null
+++ b/tests/generic/939
@@ -0,0 +1,60 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2018 Oracle. All Rights Reserved.
+#
+# FS QA Test 938
+#
+# Long-soak buffered fsx test
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_test
+
+# Run fsx for a million ops or more
+nr_ops=$((1000000 * TIME_FACTOR))
+op_sz=$((128000 * LOAD_FACTOR))
+file_sz=$((600000 * LOAD_FACTOR))
+fsx_file=$TEST_DIR/fsx.$seq
+
+fsx_args=(-q)
+fsx_args+=(-N $nr_ops)
+fsx_args+=(-p $((nr_ops / 100)))
+fsx_args+=(-o $op_sz)
+fsx_args+=(-l $file_sz)
+fsx_args+=($fsx_file)
+
+echo "ltp ${fsx_args[@]} $FSX_ARGS" >> $seqres.full
+
+if ! ltp/fsx "${fsx_args[@]}" $FSX_ARGS > $tmp.fsx 2>&1; then
+ cat $tmp.fsx | tee -a $seqres.full
+fi
+
+# success, all done
+echo Silence is golden
+status=0
+exit
diff --git a/tests/generic/939.out b/tests/generic/939.out
new file mode 100644
index 00000000..a86e90c5
--- /dev/null
+++ b/tests/generic/939.out
@@ -0,0 +1,2 @@
+QA output created by 939
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 92330b52..16e5cc17 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -522,3 +522,5 @@
517 auto quick dedupe clone
518 auto quick clone
519 auto quick
+938 soak long_rw
+939 soak long_rw
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH 05/10] fsx: add clone range
2018-11-20 21:57 ` [PATCH 05/10] fsx: add clone range Darrick J. Wong
@ 2018-11-22 2:14 ` Darrick J. Wong
0 siblings, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-22 2:14 UTC (permalink / raw)
To: guaneryu; +Cc: linux-xfs, fstests
On Tue, Nov 20, 2018 at 01:57:12PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
>
> Add support for FICLONERANGE to fsx.
>
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
> ltp/fsx.c | 133 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
> 1 file changed, 128 insertions(+), 5 deletions(-)
>
>
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index 2372964d..a1a44b23 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -104,6 +104,7 @@ enum {
> OP_ZERO_RANGE,
> OP_COLLAPSE_RANGE,
> OP_INSERT_RANGE,
> + OP_CLONE_RANGE,
> OP_MAX_FULL,
>
> /* integrity operations */
> @@ -165,6 +166,7 @@ int collapse_range_calls = 1; /* -C flag disables */
> int insert_range_calls = 1; /* -I flag disables */
> int mapped_reads = 1; /* -R flag disables it */
> int check_file = 1; /* -X flag disables */
> +int clone_range_calls = 1; /* -J flag disables */
> int integrity = 0; /* -i flag */
> int fsxgoodfd = 0;
> int o_direct; /* -Z */
> @@ -258,6 +260,7 @@ static const char *op_names[] = {
> [OP_ZERO_RANGE] = "zero_range",
> [OP_COLLAPSE_RANGE] = "collapse_range",
> [OP_INSERT_RANGE] = "insert_range",
> + [OP_CLONE_RANGE] = "clone_range",
> [OP_FSYNC] = "fsync",
> };
>
> @@ -320,7 +323,6 @@ log4(int operation, int arg0, int arg1, enum opflags flags)
> logptr = 0;
> }
>
> -
> void
> logdump(void)
> {
> @@ -342,7 +344,7 @@ logdump(void)
> count = LOGSIZE;
> }
> for ( ; count > 0; count--) {
> - bool overlap;
> + bool overlap, overlap2;
> int opnum;
>
> opnum = i+1 + (logcount/LOGSIZE)*LOGSIZE;
> @@ -444,6 +446,20 @@ logdump(void)
> if (overlap)
> prt("\t******IIII");
> break;
> + case OP_CLONE_RANGE:
> + prt("CLONE 0x%x thru 0x%x\t(0x%x bytes) to 0x%x thru 0x%x",
> + lp->args[0], lp->args[0] + lp->args[1] - 1,
> + lp->args[1],
> + lp->args[2], lp->args[2] + lp->args[1] - 1);
> + overlap2 = badoff >= lp->args[2] &&
> + badoff < lp->args[2] + lp->args[1];
> + if (overlap && overlap2)
> + prt("\tJJJJ**JJJJ");
> + else if (overlap)
> + prt("\tJJJJ******");
> + else if (overlap2)
> + prt("\t******JJJJ");
> + break;
> case OP_FSYNC:
> prt("FSYNC");
> break;
> @@ -1285,6 +1301,73 @@ do_insert_range(unsigned offset, unsigned length)
> }
> #endif
>
> +#ifdef FICLONERANGE
> +void
> +do_clone_range(unsigned offset, unsigned length, unsigned dest)
> +{
> + struct file_clone_range fcr;
> +
> + if (length == 0) {
> + if (!quiet && testcalls > simulatedopcount)
> + prt("skipping zero length clone range\n");
> + log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
> + return;
> + }
> +
> + if ((loff_t)offset >= file_size) {
> + if (!quiet && testcalls > simulatedopcount)
> + prt("skipping clone range behind EOF\n");
> + log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
> + return;
> + }
> +
> + log5(OP_CLONE_RANGE, offset, length, dest, FL_NONE);
> +
> + if (testcalls <= simulatedopcount)
> + return;
> +
> + if ((progressinterval && testcalls % progressinterval == 0) ||
> + (debug && (monitorstart == -1 || monitorend == -1 ||
> + dest <= monitorstart || dest + length <= monitorend))) {
> + prt("%lu clone\tfrom 0x%x to 0x%x, (0x%x bytes) at 0x%x\n",
> + testcalls, offset, offset+length, length, dest);
> + }
> +
> + fcr.src_fd = fd;
> + fcr.src_offset = offset;
> + fcr.src_length = length;
> + fcr.dest_offset = dest;
> +
> + if (ioctl(fd, FICLONERANGE, &fcr) == -1) {
> + if (errno == EOPNOTSUPP || errno == ENOTTY) {
> + if (!quiet && testcalls > simulatedopcount)
> + prt("skipping unsupported clone range\n");
> + logptr--;
NAK, we should just have a test_clone_range that we can use to disable
clone range calls before we start generating calls, because oplog[] is a
rolling buffer and this can cause internal memory corruption if logptr
is 0.
--D
> + log5(OP_CLONE_RANGE, offset, length, dest, FL_SKIPPED);
> + return;
> + }
> +
> + prt("clone range: 0x%x to 0x%x at 0x%x\n", offset,
> + offset + length, dest);
> + prterr("do_clone_range: FICLONERANGE");
> + report_failure(161);
> + }
> +
> + memcpy(good_buf + dest, good_buf + offset, length);
> + if (dest > file_size)
> + memset(good_buf + file_size, '\0', dest - file_size);
> + if (dest + length > file_size)
> + file_size = dest + length;
> +}
> +
> +#else
> +void
> +do_clone_range(unsigned offset, unsigned length, unsigned dest)
> +{
> + return;
> +}
> +#endif
> +
> #ifdef HAVE_LINUX_FALLOC_H
> /* fallocate is basically a no-op unless extending, then a lot like a truncate */
> void
> @@ -1442,6 +1525,8 @@ static int
> op_args_count(int operation)
> {
> switch (operation) {
> + case OP_CLONE_RANGE:
> + return 4;
> default:
> return 3;
> }
> @@ -1450,6 +1535,7 @@ op_args_count(int operation)
> static int
> read_op(struct log_entry *log_entry)
> {
> + char *end;
> char line[256];
>
> memset(log_entry, 0, sizeof(*log_entry));
> @@ -1515,7 +1601,7 @@ read_op(struct log_entry *log_entry)
> int
> test(void)
> {
> - unsigned long offset;
> + unsigned long offset, offset2;
> unsigned long size;
> unsigned long rv;
> unsigned long op;
> @@ -1546,6 +1632,7 @@ test(void)
> op = log_entry.operation;
> offset = log_entry.args[0];
> size = log_entry.args[1];
> + offset2 = log_entry.args[2];
> closeopen = !!(log_entry.flags & FL_CLOSE_OPEN);
> keep_size = !!(log_entry.flags & FL_KEEP_SIZE);
> goto have_op;
> @@ -1558,6 +1645,7 @@ test(void)
> closeopen = (rv >> 3) < (1 << 28) / closeprob;
>
> offset = random();
> + offset2 = 0;
> size = maxoplen;
> if (randomoplen)
> size = random() % (maxoplen + 1);
> @@ -1583,6 +1671,17 @@ test(void)
> if (zero_range_calls && size && keep_size_calls)
> keep_size = random() % 2;
> break;
> + case OP_CLONE_RANGE:
> + TRIM_OFF_LEN(offset, size, file_size);
> + offset = offset & ~(block_size - 1);
> + size = size & ~(block_size - 1);
> + do {
> + offset2 = random();
> + TRIM_OFF(offset2, maxfilelen);
> + offset2 = offset2 & ~(block_size - 1);
> + } while (llabs(offset2 - offset) < size ||
> + offset2 + size > maxfilelen);
> + break;
> }
>
> have_op:
> @@ -1626,6 +1725,12 @@ test(void)
> goto out;
> }
> break;
> + case OP_CLONE_RANGE:
> + if (!clone_range_calls) {
> + log5(op, offset, size, offset2, FL_SKIPPED);
> + goto out;
> + }
> + break;
> }
>
> switch (op) {
> @@ -1692,6 +1797,18 @@ test(void)
>
> do_insert_range(offset, size);
> break;
> + case OP_CLONE_RANGE:
> + if (size == 0) {
> + log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED);
> + goto out;
> + }
> + if (offset2 + size > maxfilelen) {
> + log5(OP_CLONE_RANGE, offset, size, offset2, FL_SKIPPED);
> + goto out;
> + }
> +
> + do_clone_range(offset, size, offset2);
> + break;
> case OP_FSYNC:
> dofsync();
> break;
> @@ -1717,7 +1834,7 @@ void
> usage(void)
> {
> fprintf(stdout, "usage: %s",
> - "fsx [-dknqxAFLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
> + "fsx [-dknqxAFJLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
> -b opnum: beginning operation number (default 1)\n\
> -c P: 1 in P chance of file close+open at each op (default infinity)\n\
> -d: debug output for all operations\n\
> @@ -1758,6 +1875,9 @@ usage(void)
> #ifdef FALLOC_FL_INSERT_RANGE
> " -I: Do not use insert range calls\n"
> #endif
> +#ifdef FICLONERANGE
> +" -J: Do not use clone range calls\n"
> +#endif
> " -L: fsxLite - no file creations & no file size changes\n\
> -N numops: total # operations to do (default infinity)\n\
> -O: use oplen (see -o flag) for every op (default random)\n\
> @@ -1961,7 +2081,7 @@ main(int argc, char **argv)
> setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
>
> while ((ch = getopt_long(argc, argv,
> - "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
> + "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FJKHzCILN:OP:RS:WXZ",
> longopts, NULL)) != EOF)
> switch (ch) {
> case 'b':
> @@ -2091,6 +2211,9 @@ main(int argc, char **argv)
> case 'I':
> insert_range_calls = 0;
> break;
> + case 'J':
> + clone_range_calls = 0;
> + break;
> case 'L':
> lite = 1;
> o_flags &= ~(O_CREAT|O_TRUNC);
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 02/10] fsx: always check buffer after each operation
2018-11-20 21:56 ` [PATCH 02/10] fsx: always check buffer after each operation Darrick J. Wong
2018-11-21 1:18 ` Darrick J. Wong
@ 2018-11-22 2:15 ` Darrick J. Wong
1 sibling, 0 replies; 18+ messages in thread
From: Darrick J. Wong @ 2018-11-22 2:15 UTC (permalink / raw)
To: guaneryu; +Cc: linux-xfs, fstests
On Tue, Nov 20, 2018 at 01:56:53PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
>
> Add a new option to make fsx read the file after each operation and
> compare it with the good buffer to try to catch corruptions as soon as
> they occur.
>
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
> ltp/fsx.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------
> 1 file changed, 56 insertions(+), 9 deletions(-)
>
>
> diff --git a/ltp/fsx.c b/ltp/fsx.c
> index 5601c70c..4b3a5d5a 100644
> --- a/ltp/fsx.c
> +++ b/ltp/fsx.c
> @@ -160,7 +160,8 @@ int punch_hole_calls = 1; /* -H flag disables */
> int zero_range_calls = 1; /* -z flag disables */
> int collapse_range_calls = 1; /* -C flag disables */
> int insert_range_calls = 1; /* -I flag disables */
> -int mapped_reads = 1; /* -R flag disables it */
> +int mapped_reads = 1; /* -R flag disables it */
> +int check_file = 1; /* -X flag disables */
> int integrity = 0; /* -i flag */
> int fsxgoodfd = 0;
> int o_direct; /* -Z */
> @@ -561,7 +562,7 @@ dump_fsync_buffer(void)
> }
>
> void
> -check_buffers(unsigned offset, unsigned size)
> +check_buffers(char *buf, unsigned offset, unsigned size)
> {
> unsigned char c, t;
> unsigned i = 0;
> @@ -569,19 +570,19 @@ check_buffers(unsigned offset, unsigned size)
> unsigned op = 0;
> unsigned bad = 0;
>
> - if (memcmp(good_buf + offset, temp_buf, size) != 0) {
> + if (memcmp(good_buf + offset, buf, size) != 0) {
> prt("READ BAD DATA: offset = 0x%x, size = 0x%x, fname = %s\n",
> offset, size, fname);
> prt("OFFSET\tGOOD\tBAD\tRANGE\n");
> while (size > 0) {
> c = good_buf[offset];
> - t = temp_buf[i];
> + t = buf[i];
> if (c != t) {
> if (n < 16) {
> - bad = short_at(&temp_buf[i]);
> + bad = short_at(&buf[i]);
> prt("0x%05x\t0x%04x\t0x%04x", offset,
> short_at(&good_buf[offset]), bad);
> - op = temp_buf[offset & 1 ? i+1 : i];
> + op = buf[offset & 1 ? i+1 : i];
> prt("\t0x%05x\n", n);
> if (op)
> prt("operation# (mod 256) for "
> @@ -725,7 +726,46 @@ doread(unsigned offset, unsigned size)
> iret, size);
> report_failure(141);
> }
> - check_buffers(offset, size);
> + check_buffers(temp_buf, offset, size);
> +}
> +
> +void
> +check_contents(void)
> +{
> + static char *check_buf;
> + unsigned offset = 0;
> + unsigned size = file_size;
> + off_t ret;
> + unsigned iret;
> +
> + if (!check_buf) {
> + check_buf = (char *) malloc(maxfilelen + writebdy);
> + assert(check_buf != NULL);
> + check_buf = round_ptr_up(check_buf, writebdy, 0);
> + memset(check_buf, '\0', maxfilelen);
> + }
> +
> + if (o_direct)
> + size -= size % readbdy;
> + if (size == 0)
> + return;
> +
> + ret = lseek(fd, (off_t)offset, SEEK_SET);
> + if (ret == (off_t)-1) {
> + prterr("doread: lseek");
> + report_failure(140);
> + }
> +
> + iret = fsxread(fd, check_buf, size, offset);
> + if (iret != size) {
> + if (iret == -1)
> + prterr("check_contents: read");
> + else
> + prt("short check read: 0x%x bytes instead of 0x%x\n",
> + iret, size);
> + report_failure(141);
> + }
> + check_buffers(check_buf, offset, size);
> }
>
>
> @@ -808,7 +848,7 @@ domapread(unsigned offset, unsigned size)
> report_failure(191);
> }
>
> - check_buffers(offset, size);
> + check_buffers(temp_buf, offset, size);
> }
>
>
> @@ -1624,6 +1664,9 @@ test(void)
> break;
> }
>
> + if (check_file)
NAK, this needs to be "if (check_file && testcalls > simulatedopcount)"
or else we end up reading the file before we actually made any
modifications to it (e.g. if -S XXX is given).
--D
> + check_contents();
> +
> out:
> if (sizechecks && testcalls > simulatedopcount)
> check_size();
> @@ -1684,6 +1727,7 @@ usage(void)
> -P: save .fsxlog .fsxops and .fsxgood files in dirpath (default ./)\n\
> -S seed: for random # generator (default 1) 0 gets timestamp\n\
> -W: mapped write operations DISabled\n\
> + -X: Do not read file and compare to good buffer after every operation.\n\
> -R: read() system calls only (mapped reads disabled)\n\
> -Z: O_DIRECT (use -R, -W, -r and -w too)\n\
> --replay-ops opsfile: replay ops from recorded .fsxops file\n\
> @@ -1880,7 +1924,7 @@ main(int argc, char **argv)
> setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
>
> while ((ch = getopt_long(argc, argv,
> - "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
> + "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WXZ",
> longopts, NULL)) != EOF)
> switch (ch) {
> case 'b':
> @@ -2044,6 +2088,9 @@ main(int argc, char **argv)
> if (!quiet)
> prt("mapped writes DISABLED\n");
> break;
> + case 'X':
> + check_file = 0;
> + break;
> case 'Z':
> o_direct = O_DIRECT;
> o_flags |= O_DIRECT;
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
` (10 preceding siblings ...)
2018-11-21 18:38 ` [PATCH 11/10] generic: long fsx soak tests Darrick J. Wong
@ 2018-12-10 17:30 ` Luis Henriques
2018-12-12 4:58 ` Darrick J. Wong
11 siblings, 1 reply; 18+ messages in thread
From: Luis Henriques @ 2018-12-10 17:30 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: guaneryu, linux-xfs, fstests
"Darrick J. Wong" <darrick.wong@oracle.com> writes:
> Hi all,
>
> This series adds to fsx and fsstress support for FICLONERANGE,
> FIDEDUPERANGE, and copy_file_range.
>
> First, I fix some gcc warnings in fsx.
>
> Then, I teach fsx to read the fsx file after every operation to compare
> it to the good buffer. This made it easier for me to find corruption
> problem as soon as they happen, though I'm not sure it really makes
> sense to have this enabled by default because of the behavior change
> that it makes.
>
> Next come a couple of generic reworks to fsx that we need to support the
> new clone/dedupe/copy commands.
>
> Patches 5-6 add clone and dedupe to fsx.
>
> Patches 7-8 add copy_file_range support to fsstress and fsx.
An annoying side-effect of these changes is that I now see a couple of
new generic tests failing on cephfs (for example, generic/075). That's
because fsx does use copy_file_range with the same fd both as source and
destination. We currently return -EINVAL in that case (as nfs and cifs
seem to be doing as well btw).
At least for the cephfs, this check could eventually be changed but I
would need to spend some time trying and testing the effects of
offloading object copies on the same file. Of course that another
(easy!) option would be to simply return -EOPNOTSUPP instead and
fallback to the VFS implementation.
Cheers,
--
Luis
>
> Dave Chinner contributed some cleanups to the fsx patches as the 9th
> patch.
>
> The last patch fixes the common/dump tests to disable the new commands
> so that the dump/restore tests continue to function exactly as they have
> for years.
>
> There are known failures in 4.20-rc3, particularly with copy_file_range,
> which hopefully have been fixed by the patch series that Dave Chinner
> posted to the xfs list yesterday. Branch can be downloaded here[1].
>
> --D
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fsstress-clone
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress
2018-12-10 17:30 ` [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Luis Henriques
@ 2018-12-12 4:58 ` Darrick J. Wong
2018-12-12 11:19 ` Luis Henriques
0 siblings, 1 reply; 18+ messages in thread
From: Darrick J. Wong @ 2018-12-12 4:58 UTC (permalink / raw)
To: Luis Henriques; +Cc: guaneryu, linux-xfs, fstests
On Mon, Dec 10, 2018 at 05:30:54PM +0000, Luis Henriques wrote:
> "Darrick J. Wong" <darrick.wong@oracle.com> writes:
>
> > Hi all,
> >
> > This series adds to fsx and fsstress support for FICLONERANGE,
> > FIDEDUPERANGE, and copy_file_range.
> >
> > First, I fix some gcc warnings in fsx.
> >
> > Then, I teach fsx to read the fsx file after every operation to compare
> > it to the good buffer. This made it easier for me to find corruption
> > problem as soon as they happen, though I'm not sure it really makes
> > sense to have this enabled by default because of the behavior change
> > that it makes.
> >
> > Next come a couple of generic reworks to fsx that we need to support the
> > new clone/dedupe/copy commands.
> >
> > Patches 5-6 add clone and dedupe to fsx.
> >
> > Patches 7-8 add copy_file_range support to fsstress and fsx.
>
> An annoying side-effect of these changes is that I now see a couple of
> new generic tests failing on cephfs (for example, generic/075). That's
> because fsx does use copy_file_range with the same fd both as source and
> destination. We currently return -EINVAL in that case (as nfs and cifs
> seem to be doing as well btw).
>
> At least for the cephfs, this check could eventually be changed but I
> would need to spend some time trying and testing the effects of
> offloading object copies on the same file. Of course that another
> (easy!) option would be to simply return -EOPNOTSUPP instead and
> fallback to the VFS implementation.
I think Dave Chinner's copy_file_range cleanup series fixes that, among
other problems.
--D
> Cheers,
> --
> Luis
>
> >
> > Dave Chinner contributed some cleanups to the fsx patches as the 9th
> > patch.
> >
> > The last patch fixes the common/dump tests to disable the new commands
> > so that the dump/restore tests continue to function exactly as they have
> > for years.
> >
> > There are known failures in 4.20-rc3, particularly with copy_file_range,
> > which hopefully have been fixed by the patch series that Dave Chinner
> > posted to the xfs list yesterday. Branch can be downloaded here[1].
> >
> > --D
> >
> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fsstress-clone
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress
2018-12-12 4:58 ` Darrick J. Wong
@ 2018-12-12 11:19 ` Luis Henriques
0 siblings, 0 replies; 18+ messages in thread
From: Luis Henriques @ 2018-12-12 11:19 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: guaneryu, linux-xfs, fstests
"Darrick J. Wong" <darrick.wong@oracle.com> writes:
> On Mon, Dec 10, 2018 at 05:30:54PM +0000, Luis Henriques wrote:
>> "Darrick J. Wong" <darrick.wong@oracle.com> writes:
>>
>> > Hi all,
>> >
>> > This series adds to fsx and fsstress support for FICLONERANGE,
>> > FIDEDUPERANGE, and copy_file_range.
>> >
>> > First, I fix some gcc warnings in fsx.
>> >
>> > Then, I teach fsx to read the fsx file after every operation to compare
>> > it to the good buffer. This made it easier for me to find corruption
>> > problem as soon as they happen, though I'm not sure it really makes
>> > sense to have this enabled by default because of the behavior change
>> > that it makes.
>> >
>> > Next come a couple of generic reworks to fsx that we need to support the
>> > new clone/dedupe/copy commands.
>> >
>> > Patches 5-6 add clone and dedupe to fsx.
>> >
>> > Patches 7-8 add copy_file_range support to fsstress and fsx.
>>
>> An annoying side-effect of these changes is that I now see a couple of
>> new generic tests failing on cephfs (for example, generic/075). That's
>> because fsx does use copy_file_range with the same fd both as source and
>> destination. We currently return -EINVAL in that case (as nfs and cifs
>> seem to be doing as well btw).
>>
>> At least for the cephfs, this check could eventually be changed but I
>> would need to spend some time trying and testing the effects of
>> offloading object copies on the same file. Of course that another
>> (easy!) option would be to simply return -EOPNOTSUPP instead and
>> fallback to the VFS implementation.
>
> I think Dave Chinner's copy_file_range cleanup series fixes that, among
> other problems.
No, unfortunately it doesn't because it's not really something that
would make sense to do at the VFS level. CephFS currently does not
allow to offload a copy where the source and destination are the same
file so it simply returns -EINVAL.
But that could probably be changed with Chinner's patchset, because it
adds some extra checks that make it safe to simply return -EOPNOTSUPP
instead, and fallback to VFS implementation. I'll follow-up on that in
the copy_file_range thread.
(And sorry for replying to the wrong email thread, I should have sent my
email to the latest version that actually got merged into the fstests.)
Cheers,
--
Luis
>> >
>> > Dave Chinner contributed some cleanups to the fsx patches as the 9th
>> > patch.
>> >
>> > The last patch fixes the common/dump tests to disable the new commands
>> > so that the dump/restore tests continue to function exactly as they have
>> > for years.
>> >
>> > There are known failures in 4.20-rc3, particularly with copy_file_range,
>> > which hopefully have been fixed by the patch series that Dave Chinner
>> > posted to the xfs list yesterday. Branch can be downloaded here[1].
>> >
>> > --D
>> >
>> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/log/?h=fsstress-clone
>
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2018-12-12 11:18 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-20 21:56 [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Darrick J. Wong
2018-11-20 21:56 ` [PATCH 01/10] fsx: shut up compiler warnings Darrick J. Wong
2018-11-20 21:56 ` [PATCH 02/10] fsx: always check buffer after each operation Darrick J. Wong
2018-11-21 1:18 ` Darrick J. Wong
2018-11-22 2:15 ` Darrick J. Wong
2018-11-20 21:56 ` [PATCH 03/10] fsx: use an enum to define the operation commands Darrick J. Wong
2018-11-20 21:57 ` [PATCH 04/10] fsx: add five-argument logging function Darrick J. Wong
2018-11-20 21:57 ` [PATCH 05/10] fsx: add clone range Darrick J. Wong
2018-11-22 2:14 ` Darrick J. Wong
2018-11-20 21:57 ` [PATCH 06/10] fsx: add FIDEDUPERANGE support Darrick J. Wong
2018-11-20 21:57 ` [PATCH 07/10] fsstress: add copy_file_range support Darrick J. Wong
2018-11-20 21:57 ` [PATCH 08/10] fsx: " Darrick J. Wong
2018-11-20 21:57 ` [PATCH 09/10] fsx: clean up copy/dedupe file range support Darrick J. Wong
2018-11-20 21:57 ` [PATCH 10/10] common/dump: disable copyrange Darrick J. Wong
2018-11-21 18:38 ` [PATCH 11/10] generic: long fsx soak tests Darrick J. Wong
2018-12-10 17:30 ` [PATCH v2 00/10] xfstests: add copy/dedupe/clone to fsx/fsstress Luis Henriques
2018-12-12 4:58 ` Darrick J. Wong
2018-12-12 11:19 ` Luis Henriques
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).