* [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes
@ 2017-09-05 19:11 Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 01/13] fsx: add support for integrity check with dm-log-writes target Amir Goldstein
` (12 more replies)
0 siblings, 13 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
Eryu,
This is the 3rd revision of crash consistency patch set.
I addressed all your review commennts on v2 patches.
These patches are originally from Josef Bacik.
I've added some debugging options for running fsx
and replay-log, which I used to analyze the crash consistency
bugs uncovered by the tests.
The original fsx crash consistency test (500) is failing sometimes
on kernel v4.13 for ext4, btrfs and xfs.
A fix for xfs is queued for v4.14, I think there is a fix by Josef
for btrfs queued for v4.14 as well.
A 'quick' and reliable regression test (501) was created for the
ext4 issue.
I created another crash consistency test which runs fsx on cloned
files (502). This test is failing check fs sometimes on kernel v4.13,
at the end of the test, but this is not related to crash consistency.
There is a fix is the works for v4.14.
A 'quick' and reliable regression test (503) was created for this issue.
The new 'clones crash' test (502) is also failing sometimes on xfs_repair
errors like: "fatal error -- illegal state 13 in block map 147", on crash
points, but I havn't been able to narrow down the issue yet.
Amir.
Amir Goldstein (13):
fsx: add support for integrity check with dm-log-writes target
fsx: add optional logid prefix to log messages
fsx: add support for recording operations to a file
fsx: add support for writing constant instead of random data
fsx: add support for keeping existing file
log-writes: add replay-log program to replay dm-log-writes target
replay-log: add validations for corrupt log entries
replay-log: add support for replaying ops in target device sector
fstests: add support for working with dm-log-writes target
fstests: crash consistency fsx test using dm-log-writes
fstests: regression test for ext4 crash consistency bug
fstests: crash consistency fsx test for cloned files
fstests: regression test for xfs leftover CoW extent error
.gitignore | 1 +
README | 2 +
common/dmlogwrites | 94 +++++++++
doc/auxiliary-programs.txt | 8 +
doc/requirement-checking.txt | 20 ++
ltp/fsx.c | 246 ++++++++++++++++++-----
src/Makefile | 2 +-
src/log-writes/Makefile | 23 +++
src/log-writes/SOURCE | 6 +
src/log-writes/log-writes.c | 458 +++++++++++++++++++++++++++++++++++++++++++
src/log-writes/log-writes.h | 81 ++++++++
src/log-writes/replay-log.c | 389 ++++++++++++++++++++++++++++++++++++
tests/generic/500 | 135 +++++++++++++
tests/generic/500.out | 2 +
tests/generic/501 | 80 ++++++++
tests/generic/501.out | 2 +
tests/generic/502 | 142 ++++++++++++++
tests/generic/502.out | 2 +
tests/generic/503 | 69 +++++++
tests/generic/503.out | 2 +
tests/generic/group | 4 +
21 files changed, 1721 insertions(+), 47 deletions(-)
create mode 100644 common/dmlogwrites
create mode 100644 src/log-writes/Makefile
create mode 100644 src/log-writes/SOURCE
create mode 100644 src/log-writes/log-writes.c
create mode 100644 src/log-writes/log-writes.h
create mode 100644 src/log-writes/replay-log.c
create mode 100755 tests/generic/500
create mode 100644 tests/generic/500.out
create mode 100755 tests/generic/501
create mode 100644 tests/generic/501.out
create mode 100755 tests/generic/502
create mode 100644 tests/generic/502.out
create mode 100755 tests/generic/503
create mode 100644 tests/generic/503.out
--
2.7.4
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH v3 01/13] fsx: add support for integrity check with dm-log-writes target
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 02/13] fsx: add optional logid prefix to log messages Amir Goldstein
` (11 subsequent siblings)
12 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
Cherry-picked the relevant fsx bits from commit 70d41e17164b
in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
Quoting from Josef's commit message:
I've rigged up fsx to have an integrity check mode. Basically it works
like it normally works, but when it fsync()'s it marks the log with a
unique mark and dumps it's buffer to a file with the mark in the filename.
I did this with a system() call simply because it was the fastest. I can
link the device-mapper libraries and do it programatically if that would
be preferred, but this works pretty well.
Signed-off-by: Josef Bacik <jbacik@fb.com>
[Amir:]
- Fix some exit codes
- Require -P dirpath for -i logdev
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
ltp/fsx.c | 147 ++++++++++++++++++++++++++++++++++++++++++++++++++++----------
1 file changed, 123 insertions(+), 24 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index ea00ff7..ac466ee 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -67,15 +67,17 @@ int logcount = 0; /* total ops */
* be careful in how we select the different operations. The active operations
* are mapped to numbers as follows:
*
- * lite !lite
- * READ: 0 0
- * WRITE: 1 1
- * MAPREAD: 2 2
- * MAPWRITE: 3 3
- * TRUNCATE: - 4
- * FALLOCATE: - 5
- * PUNCH HOLE: - 6
- * ZERO RANGE: - 7
+ * lite !lite integrity
+ * READ: 0 0 0
+ * WRITE: 1 1 1
+ * MAPREAD: 2 2 2
+ * MAPWRITE: 3 3 3
+ * TRUNCATE: - 4 4
+ * FALLOCATE: - 5 5
+ * PUNCH HOLE: - 6 6
+ * ZERO RANGE: - 7 7
+ * COLLAPSE RANGE: - 8 8
+ * FSYNC: - - 9
*
* When mapped read/writes are disabled, they are simply converted to normal
* reads and writes. When fallocate/fpunch calls are disabled, they are
@@ -102,6 +104,10 @@ int logcount = 0; /* total ops */
#define OP_INSERT_RANGE 9
#define OP_MAX_FULL 10
+/* integrity operations */
+#define OP_FSYNC 10
+#define OP_MAX_INTEGRITY 11
+
#undef PAGE_SIZE
#define PAGE_SIZE getpagesize()
#undef PAGE_MASK
@@ -111,6 +117,10 @@ char *original_buf; /* a pointer to the original data */
char *good_buf; /* a pointer to the correct data */
char *temp_buf; /* a pointer to the current data */
char *fname; /* name of our test file */
+char *bname; /* basename of our test file */
+char *logdev; /* -i flag */
+char dname[1024]; /* -P flag */
+int dirpath = 0; /* -P flag */
int fd; /* fd for our test file */
blksize_t block_size = 0;
@@ -148,9 +158,11 @@ int zero_range_calls = 1; /* -z flag disables */
int collapse_range_calls = 1; /* -C flag disables */
int insert_range_calls = 1; /* -I flag disables */
int mapped_reads = 1; /* -R flag disables it */
+int integrity = 0; /* -i flag */
int fsxgoodfd = 0;
int o_direct; /* -Z */
int aio = 0;
+int mark_nr = 0;
int page_size;
int page_mask;
@@ -234,6 +246,7 @@ static const char *op_names[] = {
[OP_ZERO_RANGE] = "zero_range",
[OP_COLLAPSE_RANGE] = "collapse_range",
[OP_INSERT_RANGE] = "insert_range",
+ [OP_FSYNC] = "fsync",
};
static const char *op_name(int operation)
@@ -397,6 +410,9 @@ logdump(void)
if (overlap)
prt("\t******IIII");
break;
+ case OP_FSYNC:
+ prt("FSYNC");
+ break;
default:
prt("BOGUS LOG ENTRY (operation code = %d)!",
lp->operation);
@@ -500,6 +516,43 @@ report_failure(int status)
*(((unsigned char *)(cp)) + 1)))
void
+mark_log(void)
+{
+ char command[256];
+ int ret;
+
+ snprintf(command, 256, "dmsetup message %s 0 mark %s.mark%d", logdev,
+ bname, mark_nr);
+ ret = system(command);
+ if (ret) {
+ prterr("dmsetup mark failed");
+ exit(211);
+ }
+}
+
+void
+dump_fsync_buffer(void)
+{
+ char fname_buffer[1024];
+ int good_fd;
+
+ if (!good_buf)
+ return;
+
+ snprintf(fname_buffer, 1024, "%s%s.mark%d", dname,
+ bname, mark_nr);
+ good_fd = open(fname_buffer, O_WRONLY|O_CREAT|O_TRUNC, 0666);
+ if (good_fd < 0) {
+ prterr(fname_buffer);
+ exit(212);
+ }
+
+ save_buffer(good_buf, file_size, good_fd);
+ close(good_fd);
+ prt("Dumped fsync buffer to %s\n", fname_buffer + dirpath);
+}
+
+void
check_buffers(unsigned offset, unsigned size)
{
unsigned char c, t;
@@ -1256,6 +1309,25 @@ docloseopen(void)
}
}
+void
+dofsync(void)
+{
+ int ret;
+
+ if (testcalls <= simulatedopcount)
+ return;
+ if (debug)
+ prt("%lu fsync\n", testcalls);
+ log4(OP_FSYNC, 0, 0, 0);
+ ret = fsync(fd);
+ if (ret < 0) {
+ prterr("dofsync");
+ report_failure(210);
+ }
+ mark_log();
+ dump_fsync_buffer();
+ mark_nr++;
+}
#define TRIM_OFF(off, size) \
do { \
@@ -1403,8 +1475,10 @@ test(void)
/* calculate appropriate op to run */
if (lite)
op = rv % OP_MAX_LITE;
- else
+ else if (!integrity)
op = rv % OP_MAX_FULL;
+ else
+ op = rv % OP_MAX_INTEGRITY;
switch(op) {
case OP_TRUNCATE:
@@ -1528,6 +1602,9 @@ have_op:
do_insert_range(offset, size);
break;
+ case OP_FSYNC:
+ dofsync();
+ break;
default:
prterr("test: unknown operation");
report_failure(42);
@@ -1547,11 +1624,12 @@ void
usage(void)
{
fprintf(stdout, "usage: %s",
- "fsx [-dnqxAFLOWZ] [-b opnum] [-c Prob] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+ "fsx [-dnqxAFLOWZ] [-b opnum] [-c Prob] [-i logdev] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
-b opnum: beginning operation number (default 1)\n\
-c P: 1 in P chance of file close+open at each op (default infinity)\n\
-d: debug output for all operations\n\
-f flush and invalidate cache after I/O\n\
+ -i logdev: do integrity testing, logdev is the dm log writes device\n\
-l flen: the upper bound on file size (default 262144)\n\
-m startop:endop: monitor (print debug output) specified byte range (default 0:infinity)\n\
-n: no verifications of file size\n\
@@ -1767,14 +1845,14 @@ int
main(int argc, char **argv)
{
int i, style, ch;
- char *endp;
+ char *endp, *tmp;
char goodfile[1024];
char logfile[1024];
- int dirpath = 0;
struct stat statbuf;
goodfile[0] = 0;
logfile[0] = 0;
+ dname[0] = 0;
page_size = getpagesize();
page_mask = page_size - 1;
@@ -1784,7 +1862,7 @@ main(int argc, char **argv)
setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
while ((ch = getopt_long(argc, argv,
- "b:c:dfl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
+ "b:c:dfi:l:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
longopts, NULL)) != EOF)
switch (ch) {
case 'b':
@@ -1811,6 +1889,14 @@ main(int argc, char **argv)
case 'f':
flush = 1;
break;
+ case 'i':
+ integrity = 1;
+ logdev = strdup(optarg);
+ if (!logdev) {
+ prterr("strdup");
+ exit(101);
+ }
+ break;
case 'l':
maxfilelen = getnum(optarg, &endp);
if (maxfilelen <= 0)
@@ -1908,13 +1994,13 @@ main(int argc, char **argv)
randomoplen = 0;
break;
case 'P':
- strncpy(goodfile, optarg, sizeof(goodfile));
- strcat(goodfile, "/");
- strncpy(logfile, optarg, sizeof(logfile));
- strcat(logfile, "/");
- strncpy(opsfile, optarg, sizeof(logfile));
- strcat(opsfile, "/");
- dirpath = 1;
+ strncpy(dname, optarg, sizeof(dname));
+ strcat(dname, "/");
+ dirpath = strlen(dname);
+
+ strncpy(goodfile, dname, sizeof(goodfile));
+ strncpy(logfile, dname, sizeof(logfile));
+ strncpy(opsfile, dname, sizeof(logfile));
break;
case 'R':
mapped_reads = 0;
@@ -1949,7 +2035,19 @@ main(int argc, char **argv)
argv += optind;
if (argc != 1)
usage();
+
+ if (integrity && !dirpath) {
+ fprintf(stderr, "option -i <logdev> requires -P <dirpath>\n");
+ usage();
+ }
+
fname = argv[0];
+ tmp = strdup(fname);
+ if (!tmp) {
+ prterr("strdup");
+ exit(101);
+ }
+ bname = basename(tmp);
signal(SIGHUP, cleanup);
signal(SIGINT, cleanup);
@@ -1991,21 +2089,21 @@ main(int argc, char **argv)
}
}
#endif
- strncat(goodfile, dirpath ? basename(fname) : fname, 256);
+ strncat(goodfile, dirpath ? bname : fname, 256);
strcat (goodfile, ".fsxgood");
fsxgoodfd = open(goodfile, O_RDWR|O_CREAT|O_TRUNC, 0666);
if (fsxgoodfd < 0) {
prterr(goodfile);
exit(92);
}
- strncat(logfile, dirpath ? basename(fname) : fname, 256);
+ strncat(logfile, dirpath ? bname : fname, 256);
strcat (logfile, ".fsxlog");
fsxlogf = fopen(logfile, "w");
if (fsxlogf == NULL) {
prterr(logfile);
exit(93);
}
- strncat(opsfile, dirpath ? basename(fname) : fname, 256);
+ strncat(opsfile, dirpath ? bname : fname, 256);
strcat(opsfile, ".fsxops");
unlink(opsfile);
@@ -2081,6 +2179,7 @@ main(int argc, char **argv)
if (!test())
break;
+ free(tmp);
if (close(fd)) {
prterr("close");
report_failure(99);
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 02/13] fsx: add optional logid prefix to log messages
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 01/13] fsx: add support for integrity check with dm-log-writes target Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 03/13] fsx: add support for recording operations to a file Amir Goldstein
` (10 subsequent siblings)
12 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
When writing the intermixed output of several fsx processes
to a single log file, it is usefull to prefix logs with a log id.
Use fsx -j <logid> to define the log messages prefix.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
ltp/fsx.c | 28 +++++++++++++++++++---------
1 file changed, 19 insertions(+), 9 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index ac466ee..16684cc 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -132,6 +132,7 @@ unsigned long simulatedopcount = 0; /* -b flag */
int closeprob = 0; /* -c flag */
int debug = 0; /* -d flag */
unsigned long debugstart = 0; /* -D flag */
+int logid = 0; /* -j flag */
int flush = 0; /* -f flag */
int do_fsync = 0; /* -y flag */
unsigned long maxfilelen = 256 * 1024; /* -l flag */
@@ -195,13 +196,16 @@ static void *round_ptr_up(void *ptr, unsigned long align, unsigned long offset)
}
void
-vwarnc(int code, const char *fmt, va_list ap) {
- fprintf(stderr, "fsx: ");
- if (fmt != NULL) {
- vfprintf(stderr, fmt, ap);
- fprintf(stderr, ": ");
- }
- fprintf(stderr, "%s\n", strerror(code));
+vwarnc(int code, const char *fmt, va_list ap)
+{
+ if (logid)
+ fprintf(stderr, "%d: ", logid);
+ fprintf(stderr, "fsx: ");
+ if (fmt != NULL) {
+ vfprintf(stderr, fmt, ap);
+ fprintf(stderr, ": ");
+ }
+ fprintf(stderr, "%s\n", strerror(code));
}
void
@@ -223,6 +227,8 @@ prt(const char *fmt, ...)
va_start(args, fmt);
vsnprintf(buffer, BUF_SIZE, fmt, args);
va_end(args);
+ if (logid)
+ fprintf(stdout, "%d: ", logid);
fprintf(stdout, "%s", buffer);
if (fsxlogf)
fprintf(fsxlogf, "%s", buffer);
@@ -1624,12 +1630,13 @@ void
usage(void)
{
fprintf(stdout, "usage: %s",
- "fsx [-dnqxAFLOWZ] [-b opnum] [-c Prob] [-i logdev] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+ "fsx [-dnqxAFLOWZ] [-b opnum] [-c Prob] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
-b opnum: beginning operation number (default 1)\n\
-c P: 1 in P chance of file close+open at each op (default infinity)\n\
-d: debug output for all operations\n\
-f flush and invalidate cache after I/O\n\
-i logdev: do integrity testing, logdev is the dm log writes device\n\
+ -j logid: prefix logs with this id\n\
-l flen: the upper bound on file size (default 262144)\n\
-m startop:endop: monitor (print debug output) specified byte range (default 0:infinity)\n\
-n: no verifications of file size\n\
@@ -1862,7 +1869,7 @@ main(int argc, char **argv)
setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
while ((ch = getopt_long(argc, argv,
- "b:c:dfi:l:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
+ "b:c:dfi:j:l:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
longopts, NULL)) != EOF)
switch (ch) {
case 'b':
@@ -1897,6 +1904,9 @@ main(int argc, char **argv)
exit(101);
}
break;
+ case 'j':
+ logid = getnum(optarg, &endp);
+ break;
case 'l':
maxfilelen = getnum(optarg, &endp);
if (maxfilelen <= 0)
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 03/13] fsx: add support for recording operations to a file
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 01/13] fsx: add support for integrity check with dm-log-writes target Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 02/13] fsx: add optional logid prefix to log messages Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 04/13] fsx: add support for writing constant instead of random data Amir Goldstein
` (9 subsequent siblings)
12 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
Usually, fsx dumps an .fsxops file on failure with same basename
as work file and possibly under dirctory specified by -P dirpath.
The --record-ops[=opsfile] flag can be use to dump ops file also
on success and to optionally specify the ops file name.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
ltp/fsx.c | 17 +++++++++++++++--
1 file changed, 15 insertions(+), 2 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index 16684cc..85fd50a 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -180,6 +180,7 @@ int aio_rw(int rw, int fd, char *buf, unsigned len, unsigned offset);
#endif
const char *replayops = NULL;
+const char *recordops = NULL;
FILE * fsxlogf = NULL;
FILE * replayopsf = NULL;
char opsfile[1024];
@@ -1677,6 +1678,8 @@ usage(void)
-W: mapped write operations DISabled\n\
-R: read() system calls only (mapped reads disabled)\n\
-Z: O_DIRECT (use -R, -W, -r and -w too)\n\
+ --replay-ops opsfile: replay ops from recorded .fsxops file\n\
+ --record-ops[=opsfile]: dump ops file also on success. optionally specify ops file name\n\
fname: this filename is REQUIRED (no default)\n");
exit(90);
}
@@ -1845,6 +1848,7 @@ __test_fallocate(int mode, const char *mode_str)
static struct option longopts[] = {
{"replay-ops", required_argument, 0, 256},
+ {"record-ops", optional_argument, 0, 255},
{ }
};
@@ -2034,6 +2038,11 @@ main(int argc, char **argv)
case 'Z':
o_direct = O_DIRECT;
break;
+ case 255: /* --record-ops */
+ if (optarg)
+ strncpy(opsfile, optarg, sizeof(opsfile));
+ recordops = opsfile;
+ break;
case 256: /* --replay-ops */
replayops = optarg;
break;
@@ -2113,8 +2122,10 @@ main(int argc, char **argv)
prterr(logfile);
exit(93);
}
- strncat(opsfile, dirpath ? bname : fname, 256);
- strcat(opsfile, ".fsxops");
+ if (!*opsfile) {
+ strncat(opsfile, dirpath ? bname : fname, 256);
+ strcat(opsfile, ".fsxops");
+ }
unlink(opsfile);
if (replayops) {
@@ -2195,6 +2206,8 @@ main(int argc, char **argv)
report_failure(99);
}
prt("All %lu operations completed A-OK!\n", testcalls);
+ if (recordops)
+ logdump();
exit(0);
return 0;
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 04/13] fsx: add support for writing constant instead of random data
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
` (2 preceding siblings ...)
2017-09-05 19:11 ` [PATCH v3 03/13] fsx: add support for recording operations to a file Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 05/13] fsx: add support for keeping existing file Amir Goldstein
` (8 subsequent siblings)
12 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
-g X: write character X instead of random generated data
This is useful to compare holes between good and bad files
because hexdump of good and bad files compacts the contigious
ranges of X and zeroes.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
ltp/fsx.c | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index 85fd50a..3510ace 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -132,6 +132,7 @@ unsigned long simulatedopcount = 0; /* -b flag */
int closeprob = 0; /* -c flag */
int debug = 0; /* -d flag */
unsigned long debugstart = 0; /* -D flag */
+char filldata = 0; /* -g flag */
int logid = 0; /* -j flag */
int flush = 0; /* -f flag */
int do_fsync = 0; /* -y flag */
@@ -814,9 +815,13 @@ void
gendata(char *original_buf, char *good_buf, unsigned offset, unsigned size)
{
while (size--) {
- good_buf[offset] = testcalls % 256;
- if (offset % 2)
- good_buf[offset] += original_buf[offset];
+ if (filldata) {
+ good_buf[offset] = filldata;
+ } else {
+ good_buf[offset] = testcalls % 256;
+ if (offset % 2)
+ good_buf[offset] += original_buf[offset];
+ }
offset++;
}
}
@@ -1631,11 +1636,12 @@ void
usage(void)
{
fprintf(stdout, "usage: %s",
- "fsx [-dnqxAFLOWZ] [-b opnum] [-c Prob] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+ "fsx [-dnqxAFLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
-b opnum: beginning operation number (default 1)\n\
-c P: 1 in P chance of file close+open at each op (default infinity)\n\
-d: debug output for all operations\n\
-f flush and invalidate cache after I/O\n\
+ -g X: write character X instead of random generated data\n\
-i logdev: do integrity testing, logdev is the dm log writes device\n\
-j logid: prefix logs with this id\n\
-l flen: the upper bound on file size (default 262144)\n\
@@ -1873,7 +1879,7 @@ main(int argc, char **argv)
setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
while ((ch = getopt_long(argc, argv,
- "b:c:dfi:j:l:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
+ "b:c:dfg:i:j:l:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
longopts, NULL)) != EOF)
switch (ch) {
case 'b':
@@ -1900,6 +1906,9 @@ main(int argc, char **argv)
case 'f':
flush = 1;
break;
+ case 'g':
+ filldata = *optarg;
+ break;
case 'i':
integrity = 1;
logdev = strdup(optarg);
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 05/13] fsx: add support for keeping existing file
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
` (3 preceding siblings ...)
2017-09-05 19:11 ` [PATCH v3 04/13] fsx: add support for writing constant instead of random data Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 06/13] log-writes: add replay-log program to replay dm-log-writes target Amir Goldstein
` (7 subsequent siblings)
12 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
With fsxk -k, do not truncate existing file and use its size as upper
bound on file size.
This is needed to prevent fsx from truncating the file on start of
test when testing fsx on cloned files.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
ltp/fsx.c | 49 ++++++++++++++++++++++++++++++++++++-------------
1 file changed, 36 insertions(+), 13 deletions(-)
diff --git a/ltp/fsx.c b/ltp/fsx.c
index 3510ace..7a34577 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -629,17 +629,18 @@ void
check_trunc_hack(void)
{
struct stat statbuf;
+ off_t offset = file_size + (off_t)100000;
- if (ftruncate(fd, (off_t)0))
+ if (ftruncate(fd, file_size))
goto ftruncate_err;
- if (ftruncate(fd, (off_t)100000))
+ if (ftruncate(fd, offset))
goto ftruncate_err;
fstat(fd, &statbuf);
- if (statbuf.st_size != (off_t)100000) {
+ if (statbuf.st_size != offset) {
prt("no extend on truncate! not posix!\n");
exit(130);
}
- if (ftruncate(fd, 0)) {
+ if (ftruncate(fd, file_size)) {
ftruncate_err:
prterr("check_trunc_hack: ftruncate");
exit(131);
@@ -1636,7 +1637,7 @@ void
usage(void)
{
fprintf(stdout, "usage: %s",
- "fsx [-dnqxAFLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
+ "fsx [-dknqxAFLOWZ] [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid] [-l flen] [-m start:end] [-o oplen] [-p progressinterval] [-r readbdy] [-s style] [-t truncbdy] [-w writebdy] [-D startingop] [-N numops] [-P dirpath] [-S seed] fname\n\
-b opnum: beginning operation number (default 1)\n\
-c P: 1 in P chance of file close+open at each op (default infinity)\n\
-d: debug output for all operations\n\
@@ -1644,6 +1645,7 @@ usage(void)
-g X: write character X instead of random generated data\n\
-i logdev: do integrity testing, logdev is the dm log writes device\n\
-j logid: prefix logs with this id\n\
+ -k: do not truncate existing file and use its size as upper bound on file size\n\
-l flen: the upper bound on file size (default 262144)\n\
-m startop:endop: monitor (print debug output) specified byte range (default 0:infinity)\n\
-n: no verifications of file size\n\
@@ -1834,7 +1836,7 @@ __test_fallocate(int mode, const char *mode_str)
#ifdef HAVE_LINUX_FALLOC_H
int ret = 0;
if (!lite) {
- if (fallocate(fd, mode, 0, 1) && errno == EOPNOTSUPP) {
+ if (fallocate(fd, mode, file_size, 1) && errno == EOPNOTSUPP) {
if(!quiet)
fprintf(stderr,
"main: filesystem does not support "
@@ -1842,7 +1844,7 @@ __test_fallocate(int mode, const char *mode_str)
mode_str);
} else {
ret = 1;
- if (ftruncate(fd, 0)) {
+ if (ftruncate(fd, file_size)) {
warn("main: ftruncate");
exit(132);
}
@@ -1866,6 +1868,7 @@ main(int argc, char **argv)
char goodfile[1024];
char logfile[1024];
struct stat statbuf;
+ int o_flags = O_RDWR|O_CREAT|O_TRUNC;
goodfile[0] = 0;
logfile[0] = 0;
@@ -1879,7 +1882,7 @@ main(int argc, char **argv)
setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */
while ((ch = getopt_long(argc, argv,
- "b:c:dfg:i:j:l:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
+ "b:c:dfg:i:j:kl:m:no:p:qr:s:t:w:xyAD:FKHzCILN:OP:RS:WZ",
longopts, NULL)) != EOF)
switch (ch) {
case 'b':
@@ -1920,6 +1923,9 @@ main(int argc, char **argv)
case 'j':
logid = getnum(optarg, &endp);
break;
+ case 'k':
+ o_flags &= ~O_TRUNC;
+ break;
case 'l':
maxfilelen = getnum(optarg, &endp);
if (maxfilelen <= 0)
@@ -2007,6 +2013,7 @@ main(int argc, char **argv)
break;
case 'L':
lite = 1;
+ o_flags &= ~(O_CREAT|O_TRUNC);
break;
case 'N':
numops = getnum(optarg, &endp);
@@ -2046,6 +2053,7 @@ main(int argc, char **argv)
break;
case 'Z':
o_direct = O_DIRECT;
+ o_flags |= O_DIRECT;
break;
case 255: /* --record-ops */
if (optarg)
@@ -2089,8 +2097,7 @@ main(int argc, char **argv)
signal(SIGUSR2, cleanup);
srandom(seed);
- fd = open(fname,
- O_RDWR|(lite ? 0 : O_CREAT|O_TRUNC)|o_direct, 0666);
+ fd = open(fname, o_flags, 0666);
if (fd < 0) {
prterr(fname);
exit(91);
@@ -2150,9 +2157,10 @@ main(int argc, char **argv)
aio_setup();
#endif
- if (lite) {
+ if (!(o_flags & O_TRUNC)) {
off_t ret;
- file_size = maxfilelen = lseek(fd, (off_t)0, SEEK_END);
+ file_size = maxfilelen = biggest = lseek(fd, (off_t)0, SEEK_END);
+ printf("file_size=%lu\n", file_size);
if (file_size == (off_t)-1) {
prterr(fname);
warn("main: lseek eof");
@@ -2189,8 +2197,23 @@ main(int argc, char **argv)
maxfilelen);
exit(98);
}
- } else
+ } else {
+ ssize_t ret, len = file_size;
+ off_t off = 0;
+
+ while (len > 0) {
+ ret = read(fd, good_buf + off, len);
+ if (ret == -1) {
+ prterr(fname);
+ warn("main: error on read");
+ exit(98);
+ }
+ len -= ret;
+ off += ret;
+ }
+
check_trunc_hack();
+ }
if (fallocate_calls)
fallocate_calls = test_fallocate(0);
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 06/13] log-writes: add replay-log program to replay dm-log-writes target
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
` (4 preceding siblings ...)
2017-09-05 19:11 ` [PATCH v3 05/13] fsx: add support for keeping existing file Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 07/13] replay-log: add validations for corrupt log entries Amir Goldstein
` (6 subsequent siblings)
12 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
Imported Josef Bacik's code from:
https://github.com/josefbacik/log-writes.git
Specialized program for replaying a write log that was recorded by
device mapper log-writes target. The tools is used to perform
crash consistency tests, allowing to run an arbitrary check tool
(fsck) at specified checkpoints in the write log.
[Amir:]
- Add project Makefile and SOURCE files
- Document the replay-log auxiliary program
- Address review comments by Eryu Guan
Cc: Josef Bacik <jbacik@fb.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
.gitignore | 1 +
doc/auxiliary-programs.txt | 8 +
src/Makefile | 2 +-
src/log-writes/Makefile | 23 +++
src/log-writes/SOURCE | 6 +
src/log-writes/log-writes.c | 381 ++++++++++++++++++++++++++++++++++++++++++++
src/log-writes/log-writes.h | 77 +++++++++
src/log-writes/replay-log.c | 357 +++++++++++++++++++++++++++++++++++++++++
8 files changed, 854 insertions(+), 1 deletion(-)
create mode 100644 src/log-writes/Makefile
create mode 100644 src/log-writes/SOURCE
create mode 100644 src/log-writes/log-writes.c
create mode 100644 src/log-writes/log-writes.h
create mode 100644 src/log-writes/replay-log.c
diff --git a/.gitignore b/.gitignore
index 28fe84d..beb9504 100644
--- a/.gitignore
+++ b/.gitignore
@@ -154,6 +154,7 @@
/src/t_mmap_stale_pmd
/src/t_mmap_cow_race
/src/t_mmap_fallocate
+/src/log-writes/replay-log
# dmapi/ binaries
/dmapi/src/common/cmd/read_invis
diff --git a/doc/auxiliary-programs.txt b/doc/auxiliary-programs.txt
index bcab453..de15832 100644
--- a/doc/auxiliary-programs.txt
+++ b/doc/auxiliary-programs.txt
@@ -18,6 +18,7 @@ Contents:
- af_unix -- Create an AF_UNIX socket
- dmerror -- fault injection block device control
- fsync-err -- tests fsync error reporting after failed writeback
+ - log-writes/replay-log -- Replay log from device mapper log-writes target
- open_by_handle -- open_by_handle_at syscall exercise
- stat_test -- statx syscall exercise
- t_dir_type -- print directory entries and their file type
@@ -46,6 +47,13 @@ fsync-err
writeback and test that errors are reported during fsync and cleared
afterward.
+log-writes/replay-log
+
+ Specialized program for replaying a write log that was recorded by
+ device mapper log-writes target. The tools is used to perform crash
+ consistency tests, allowing to run an arbitrary check tool (fsck) at
+ specified checkpoints in the write log.
+
open_by_handle
The open_by_handle program exercises the open_by_handle_at() system
diff --git a/src/Makefile b/src/Makefile
index b8aff49..7d1306b 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -25,7 +25,7 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \
attr-list-by-handle-cursor-test listxattr dio-interleaved t_dir_type \
dio-invalidate-cache stat_test t_encrypted_d_revalidate
-SUBDIRS =
+SUBDIRS = log-writes
LLDLIBS = $(LIBATTR) $(LIBHANDLE) $(LIBACL) -lpthread
diff --git a/src/log-writes/Makefile b/src/log-writes/Makefile
new file mode 100644
index 0000000..d114177
--- /dev/null
+++ b/src/log-writes/Makefile
@@ -0,0 +1,23 @@
+TOPDIR = ../..
+include $(TOPDIR)/include/builddefs
+
+TARGETS = replay-log
+
+CFILES = replay-log.c log-writes.c
+LDIRT = $(TARGETS)
+
+default: depend $(TARGETS)
+
+depend: .dep
+
+include $(BUILDRULES)
+
+$(TARGETS): $(CFILES)
+ @echo " [CC] $@"
+ $(Q)$(LTLINK) $(CFILES) -o $@ $(CFLAGS) $(LDFLAGS) $(LDLIBS)
+
+install:
+ $(INSTALL) -m 755 -d $(PKG_LIB_DIR)/src/log-writes
+ $(INSTALL) -m 755 $(TARGETS) $(PKG_LIB_DIR)/src/log-writes
+
+-include .dep
diff --git a/src/log-writes/SOURCE b/src/log-writes/SOURCE
new file mode 100644
index 0000000..d6d143c
--- /dev/null
+++ b/src/log-writes/SOURCE
@@ -0,0 +1,6 @@
+From:
+https://github.com/josefbacik/log-writes.git
+
+description Helper code for dm-log-writes target
+owner Josef Bacik <jbacik@fb.com>
+URL https://github.com/josefbacik/log-writes.git
diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c
new file mode 100644
index 0000000..a215fef
--- /dev/null
+++ b/src/log-writes/log-writes.c
@@ -0,0 +1,381 @@
+#include <linux/fs.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/ioctl.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <unistd.h>
+#include <string.h>
+#include "log-writes.h"
+
+int log_writes_verbose = 0;
+
+/*
+ * @log: the log to free.
+ *
+ * This will close any open fd's the log has and free up its memory.
+ */
+void log_free(struct log *log)
+{
+ if (log->replayfd >= 0)
+ close(log->replayfd);
+ if (log->logfd >= 0)
+ close(log->logfd);
+ free(log);
+}
+
+static int discard_range(struct log *log, u64 start, u64 len)
+{
+ u64 range[2] = { start, len };
+
+ if (ioctl(log->replayfd, BLKDISCARD, &range) < 0) {
+ if (log_writes_verbose)
+ printf("replay device doesn't support discard, "
+ "switching to writing zeros\n");
+ log->flags |= LOG_DISCARD_NOT_SUPP;
+ }
+ return 0;
+}
+
+static int zero_range(struct log *log, u64 start, u64 len)
+{
+ u64 bufsize = len;
+ ssize_t ret;
+ char *buf = NULL;
+
+ if (log->max_zero_size < len) {
+ if (log_writes_verbose)
+ printf("discard len %llu larger than max %llu\n",
+ (unsigned long long)len,
+ (unsigned long long)log->max_zero_size);
+ return 0;
+ }
+
+ while (!buf) {
+ buf = malloc(bufsize);
+ if (!buf)
+ bufsize >>= 1;
+ if (!bufsize) {
+ fprintf(stderr, "Couldn't allocate zero buffer");
+ return -1;
+ }
+ }
+
+ memset(buf, 0, bufsize);
+ while (len) {
+ ret = pwrite(log->replayfd, buf, bufsize, start);
+ if (ret != bufsize) {
+ fprintf(stderr, "Error zeroing file: %d\n", errno);
+ free(buf);
+ return -1;
+ }
+ len -= ret;
+ start += ret;
+ }
+ free(buf);
+ return 0;
+}
+
+/*
+ * @log: the log we are replaying.
+ * @entry: the discard entry.
+ *
+ * Discard the given length. If the device supports discard we will call that
+ * ioctl, otherwise we will write 0's to emulate discard. If the discard size
+ * is larger than log->max_zero_size then we will simply skip the zero'ing if
+ * the drive doesn't support discard.
+ */
+int log_discard(struct log *log, struct log_write_entry *entry)
+{
+ u64 start = le64_to_cpu(entry->sector) * log->sectorsize;
+ u64 size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
+ u64 max_chunk = 1 * 1024 * 1024 * 1024;
+
+ if (log->flags & LOG_IGNORE_DISCARD)
+ return 0;
+
+ while (size) {
+ u64 len = size > max_chunk ? max_chunk : size;
+ int ret;
+
+ /*
+ * Do this check first in case it is our first discard, that way
+ * if we return EOPNOTSUPP we will fall back to the 0 method
+ * automatically.
+ */
+ if (!(log->flags & LOG_DISCARD_NOT_SUPP))
+ ret = discard_range(log, start, len);
+ if (log->flags & LOG_DISCARD_NOT_SUPP)
+ ret = zero_range(log, start, len);
+ if (ret)
+ return -1;
+ size -= len;
+ start += len;
+ }
+ return 0;
+}
+
+/*
+ * @log: the log we are replaying.
+ * @entry: where we put the entry.
+ * @read_data: read the entry data as well, entry must be log->sectorsize sized
+ * if this is set.
+ *
+ * @return: 0 if we replayed, 1 if we are at the end, -1 if there was an error.
+ *
+ * Replay the next entry in our log onto the replay device.
+ */
+int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
+ int read_data)
+{
+ u64 size;
+ u64 flags;
+ size_t read_size = read_data ? log->sectorsize :
+ sizeof(struct log_write_entry);
+ char *buf;
+ ssize_t ret;
+ off_t offset;
+
+ if (log->cur_entry >= log->nr_entries)
+ return 1;
+
+ ret = read(log->logfd, entry, read_size);
+ if (ret != read_size) {
+ fprintf(stderr, "Error reading entry: %d\n", errno);
+ return -1;
+ }
+ log->cur_entry++;
+
+ size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
+ if (read_size < log->sectorsize) {
+ if (lseek(log->logfd,
+ log->sectorsize - sizeof(struct log_write_entry),
+ SEEK_CUR) == (off_t)-1) {
+ fprintf(stderr, "Error seeking in log: %d\n", errno);
+ return -1;
+ }
+ }
+
+ if (log_writes_verbose)
+ printf("replaying %d: sector %llu, size %llu, flags %llu\n",
+ (int)log->cur_entry - 1,
+ (unsigned long long)le64_to_cpu(entry->sector),
+ (unsigned long long)size,
+ (unsigned long long)le64_to_cpu(entry->flags));
+ if (!size)
+ return 0;
+
+ flags = le64_to_cpu(entry->flags);
+ if (flags & LOG_DISCARD_FLAG)
+ return log_discard(log, entry);
+
+ buf = malloc(size);
+ if (!buf) {
+ fprintf(stderr, "Error allocating buffer %llu entry %llu\n", (unsigned long long)size, (unsigned long long)log->cur_entry - 1);
+ return -1;
+ }
+
+ ret = read(log->logfd, buf, size);
+ if (ret != size) {
+ fprintf(stderr, "Error reading data: %d\n", errno);
+ free(buf);
+ return -1;
+ }
+
+ offset = le64_to_cpu(entry->sector) * log->sectorsize;
+ ret = pwrite(log->replayfd, buf, size, offset);
+ free(buf);
+ if (ret != size) {
+ fprintf(stderr, "Error writing data: %d\n", errno);
+ return -1;
+ }
+
+ return 0;
+}
+
+/*
+ * @log: the log we are manipulating.
+ * @entry_num: the entry we want.
+ *
+ * Seek to the given entry in the log, starting at 0 and ending at
+ * log->nr_entries - 1.
+ */
+int log_seek_entry(struct log *log, u64 entry_num)
+{
+ u64 i = 0;
+
+ if (entry_num >= log->nr_entries) {
+ fprintf(stderr, "Invalid entry number\n");
+ return -1;
+ }
+
+ /* Skip the first sector containing the log super block */
+ if (lseek(log->logfd, log->sectorsize, SEEK_SET) == (off_t)-1) {
+ fprintf(stderr, "Error seeking in file: %d\n", errno);
+ return -1;
+ }
+
+ log->cur_entry = 0;
+ for (i = 0; i < entry_num; i++) {
+ struct log_write_entry entry;
+ ssize_t ret;
+ off_t seek_size;
+ u64 flags;
+
+ ret = read(log->logfd, &entry, sizeof(entry));
+ if (ret != sizeof(entry)) {
+ fprintf(stderr, "Error reading entry: %d\n", errno);
+ return -1;
+ }
+ if (log_writes_verbose > 1)
+ printf("seek entry %d: %llu, size %llu, flags %llu\n",
+ (int)i,
+ (unsigned long long)le64_to_cpu(entry.sector),
+ (unsigned long long)le64_to_cpu(entry.nr_sectors),
+ (unsigned long long)le64_to_cpu(entry.flags));
+ flags = le64_to_cpu(entry.flags);
+ seek_size = log->sectorsize - sizeof(entry);
+ if (!(flags & LOG_DISCARD_FLAG))
+ seek_size += le64_to_cpu(entry.nr_sectors) *
+ log->sectorsize;
+ if (lseek(log->logfd, seek_size, SEEK_CUR) == (off_t)-1) {
+ fprintf(stderr, "Error seeking in file: %d\n", errno);
+ return -1;
+ }
+ log->cur_entry++;
+ }
+
+ return 0;
+}
+
+/*
+ * @log: the log we are manipulating.
+ * @entry: the entry we read.
+ * @read_data: read the extra data for the entry, your entry must be
+ * log->sectorsize large.
+ *
+ * @return: 1 if we hit the end of the log, 0 we got the next entry, < 0 if
+ * there was an error.
+ *
+ * Seek to the next entry in the log.
+ */
+int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
+ int read_data)
+{
+ size_t read_size = read_data ? log->sectorsize :
+ sizeof(struct log_write_entry);
+ u64 flags;
+ ssize_t ret;
+
+ if (log->cur_entry >= log->nr_entries)
+ return 1;
+
+ ret = read(log->logfd, entry, read_size);
+ if (ret != read_size) {
+ fprintf(stderr, "Error reading entry: %d\n", errno);
+ return -1;
+ }
+ log->cur_entry++;
+
+ if (read_size < log->sectorsize) {
+ if (lseek(log->logfd,
+ log->sectorsize - sizeof(struct log_write_entry),
+ SEEK_CUR) == (off_t)-1) {
+ fprintf(stderr, "Error seeking in log: %d\n", errno);
+ return -1;
+ }
+ }
+ if (log_writes_verbose > 1)
+ printf("seek entry %d: %llu, size %llu, flags %llu\n",
+ (int)log->cur_entry - 1,
+ (unsigned long long)le64_to_cpu(entry->sector),
+ (unsigned long long)le64_to_cpu(entry->nr_sectors),
+ (unsigned long long)le64_to_cpu(entry->flags));
+
+ flags = le32_to_cpu(entry->flags);
+ read_size = le32_to_cpu(entry->nr_sectors) * log->sectorsize;
+ if (!read_size || (flags & LOG_DISCARD_FLAG))
+ return 0;
+
+ if (lseek(log->logfd, read_size, SEEK_CUR) == (off_t)-1) {
+ fprintf(stderr, "Error seeking in log: %d\n", errno);
+ return -1;
+ }
+
+ return 0;
+}
+
+/*
+ * @logfile: the file that contains the write log.
+ * @replayfile: the file/device to replay onto, can be NULL.
+ *
+ * Opens a logfile and makes sure it is valid and returns a struct log.
+ */
+struct log *log_open(char *logfile, char *replayfile)
+{
+ struct log *log;
+ struct log_write_super super;
+ ssize_t ret;
+
+ log = malloc(sizeof(struct log));
+ if (!log) {
+ fprintf(stderr, "Couldn't alloc log\n");
+ return NULL;
+ }
+
+ log->replayfd = -1;
+
+ log->logfd = open(logfile, O_RDONLY);
+ if (log->logfd < 0) {
+ fprintf(stderr, "Couldn't open log %s: %d\n", logfile,
+ errno);
+ log_free(log);
+ return NULL;
+ }
+
+ if (replayfile) {
+ log->replayfd = open(replayfile, O_WRONLY);
+ if (log->replayfd < 0) {
+ fprintf(stderr, "Couldn't open replay file %s: %d\n",
+ replayfile, errno);
+ log_free(log);
+ return NULL;
+ }
+ }
+
+ ret = read(log->logfd, &super, sizeof(struct log_write_super));
+ if (ret < sizeof(struct log_write_super)) {
+ fprintf(stderr, "Error reading super: %d\n", errno);
+ log_free(log);
+ return NULL;
+ }
+
+ if (le64_to_cpu(super.magic) != WRITE_LOG_MAGIC) {
+ fprintf(stderr, "Magic doesn't match\n");
+ log_free(log);
+ return NULL;
+ }
+
+ if (le64_to_cpu(super.version) != WRITE_LOG_VERSION) {
+ fprintf(stderr, "Version mismatch, wanted %d, have %d\n",
+ WRITE_LOG_VERSION, (int)le64_to_cpu(super.version));
+ log_free(log);
+ return NULL;
+ }
+
+ log->sectorsize = le32_to_cpu(super.sectorsize);
+ log->nr_entries = le64_to_cpu(super.nr_entries);
+ log->max_zero_size = 128 * 1024 * 1024;
+
+ if (lseek(log->logfd, log->sectorsize - sizeof(super), SEEK_CUR) ==
+ (off_t) -1) {
+ fprintf(stderr, "Error seeking to first entry: %d\n", errno);
+ log_free(log);
+ return NULL;
+ }
+ log->cur_entry = 0;
+
+ return log;
+}
diff --git a/src/log-writes/log-writes.h b/src/log-writes/log-writes.h
new file mode 100644
index 0000000..6cadb66
--- /dev/null
+++ b/src/log-writes/log-writes.h
@@ -0,0 +1,77 @@
+#ifndef _LOG_WRITES_H_
+#define _LOG_WRITES_H_
+
+#include <linux/types.h>
+#include <linux/byteorder/little_endian.h>
+
+extern int log_writes_verbose;
+
+#define le64_to_cpu __le64_to_cpu
+#define le32_to_cpu __le32_to_cpu
+
+typedef __u64 u64;
+typedef __u32 u32;
+
+/*
+ * Constants copied from kernel file drivers/md/dm-log-writes.c
+ */
+#define LOG_FLUSH_FLAG (1 << 0)
+#define LOG_FUA_FLAG (1 << 1)
+#define LOG_DISCARD_FLAG (1 << 2)
+#define LOG_MARK_FLAG (1 << 3)
+
+#define WRITE_LOG_VERSION 1
+#define WRITE_LOG_MAGIC 0x6a736677736872
+
+
+/*
+ * Basic info about the log for userspace.
+ *
+ * Copied from kernel file drivers/md/dm-log-writes.c
+ */
+struct log_write_super {
+ __le64 magic;
+ __le64 version;
+ __le64 nr_entries;
+ __le32 sectorsize;
+};
+
+/*
+ * sector - the sector we wrote.
+ * nr_sectors - the number of sectors we wrote.
+ * flags - flags for this log entry.
+ * data_len - the size of the data in this log entry, this is for private log
+ * entry stuff, the MARK data provided by userspace for example.
+ *
+ * Copied from kernel file drivers/md/dm-log-writes.c
+ */
+struct log_write_entry {
+ __le64 sector;
+ __le64 nr_sectors;
+ __le64 flags;
+ __le64 data_len;
+};
+
+#define LOG_IGNORE_DISCARD (1 << 0)
+#define LOG_DISCARD_NOT_SUPP (1 << 1)
+
+struct log {
+ int logfd;
+ int replayfd;
+ unsigned long flags;
+ u64 sectorsize;
+ u64 nr_entries;
+ u64 cur_entry;
+ u64 max_zero_size;
+ off_t cur_pos;
+};
+
+struct log *log_open(char *logfile, char *replayfile);
+int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
+ int read_data);
+int log_seek_entry(struct log *log, u64 entry_num);
+int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
+ int read_data);
+void log_free(struct log *log);
+
+#endif
diff --git a/src/log-writes/replay-log.c b/src/log-writes/replay-log.c
new file mode 100644
index 0000000..c3de9c4
--- /dev/null
+++ b/src/log-writes/replay-log.c
@@ -0,0 +1,357 @@
+#include <stdio.h>
+#include <unistd.h>
+#include <getopt.h>
+#include <stdlib.h>
+#include <string.h>
+#include "log-writes.h"
+
+enum option_indexes {
+ NEXT_FLUSH,
+ NEXT_FUA,
+ START_ENTRY,
+ END_MARK,
+ LOG,
+ REPLAY,
+ LIMIT,
+ VERBOSE,
+ FIND,
+ NUM_ENTRIES,
+ NO_DISCARD,
+ FSCK,
+ CHECK,
+ START_MARK,
+};
+
+static struct option long_options[] = {
+ {"next-flush", no_argument, NULL, 0},
+ {"next-fua", no_argument, NULL, 0},
+ {"start-entry", required_argument, NULL, 0},
+ {"end-mark", required_argument, NULL, 0},
+ {"log", required_argument, NULL, 0},
+ {"replay", required_argument, NULL, 0},
+ {"limit", required_argument, NULL, 0},
+ {"verbose", no_argument, NULL, 'v'},
+ {"find", no_argument, NULL, 0},
+ {"num-entries", no_argument, NULL, 0},
+ {"no-discard", no_argument, NULL, 0},
+ {"fsck", required_argument, NULL, 0},
+ {"check", required_argument, NULL, 0},
+ {"start-mark", required_argument, NULL, 0},
+ { NULL, 0, NULL, 0 },
+};
+
+static void usage(void)
+{
+ fprintf(stderr, "Usage: replay-log --log <logfile> [options]\n");
+ fprintf(stderr, "\t--replay <device> - replay onto a specific "
+ "device\n");
+ fprintf(stderr, "\t--limit <number> - number of entries to replay\n");
+ fprintf(stderr, "\t--next-flush - replay to/find the next flush\n");
+ fprintf(stderr, "\t--next-fua - replay to/find the next fua\n");
+ fprintf(stderr, "\t--start-entry <entry> - start at the given "
+ "entry #\n");
+ fprintf(stderr, "\t--start-mark <mark> - mark to start from\n");
+ fprintf(stderr, "\t--end-mark <mark> - replay to/find the given mark\n");
+ fprintf(stderr, "\t--find - put replay-log in find mode, will search "
+ "based on the other options\n");
+ fprintf(stderr, "\t--number-entries - print the number of entries in "
+ "the log\n");
+ fprintf(stderr, "\t--no-discard - don't process discard entries\n");
+ fprintf(stderr, "\t--fsck - the fsck command to run, must specify "
+ "--check\n");
+ fprintf(stderr, "\t--check [<number>|flush|fua] when to check the "
+ "file system, mush specify --fsck\n");
+ exit(1);
+}
+
+/*
+ * Check if the log entry flag matches one of the stop_flags.
+ * If stop_flag has LOG_MARK, then looking also for match of
+ * the mark label.
+ */
+static int should_stop(struct log_write_entry *entry, u64 stop_flags,
+ char *mark)
+{
+ u64 flags = le64_to_cpu(entry->flags);
+ int check_mark = (stop_flags & LOG_MARK_FLAG);
+ /* mark data begins after entry header */
+ char *buf = (char *)(entry + 1);
+ /* entry buffer is padded with at least 1 zero after data_len */
+ u64 buflen = le64_to_cpu(entry->data_len) + 1;
+
+ if (flags & stop_flags) {
+ if (!check_mark)
+ return 1;
+ if ((flags & LOG_MARK_FLAG) &&
+ !strncmp(mark, buf, buflen))
+ return 1;
+ }
+ return 0;
+}
+
+static int run_fsck(struct log *log, char *fsck_command)
+{
+ int ret = fsync(log->replayfd);
+ if (ret)
+ return ret;
+ ret = system(fsck_command);
+ if (ret >= 0)
+ ret = WEXITSTATUS(ret);
+ return ret ? -1 : 0;
+}
+
+enum log_replay_check_mode {
+ CHECK_NUMBER = 1,
+ CHECK_FUA = 2,
+ CHECK_FLUSH = 3,
+};
+
+static int seek_to_mark(struct log *log, struct log_write_entry *entry,
+ char *mark)
+{
+ int ret;
+
+ while ((ret = log_seek_next_entry(log, entry, 1)) == 0) {
+ if (should_stop(entry, LOG_MARK_FLAG, mark))
+ break;
+ }
+ if (ret == 1) {
+ fprintf(stderr, "Couldn't find starting mark\n");
+ ret = -1;
+ }
+
+ return ret;
+}
+
+int main(int argc, char **argv)
+{
+ char *logfile = NULL, *replayfile = NULL, *fsck_command = NULL;
+ struct log_write_entry *entry;
+ u64 stop_flags = 0;
+ u64 start_entry = 0;
+ u64 run_limit = 0;
+ u64 num_entries = 0;
+ u64 check_number = 0;
+ char *end_mark = NULL, *start_mark = NULL;
+ char *tmp = NULL;
+ struct log *log;
+ int find_mode = 0;
+ int c;
+ int opt_index;
+ int ret;
+ int print_num_entries = 0;
+ int discard = 1;
+ enum log_replay_check_mode check_mode = 0;
+
+ while ((c = getopt_long(argc, argv, "v", long_options,
+ &opt_index)) >= 0) {
+ switch(c) {
+ case 'v':
+ log_writes_verbose++;
+ continue;
+ default:
+ break;
+ }
+
+ switch(opt_index) {
+ case NEXT_FLUSH:
+ stop_flags |= LOG_FLUSH_FLAG;
+ break;
+ case NEXT_FUA:
+ stop_flags |= LOG_FUA_FLAG;
+ break;
+ case START_ENTRY:
+ start_entry = strtoull(optarg, &tmp, 0);
+ if (tmp && *tmp != '\0') {
+ fprintf(stderr, "Invalid entry number\n");
+ exit(1);
+ }
+ tmp = NULL;
+ break;
+ case START_MARK:
+ /*
+ * Biggest sectorsize is 4k atm, so limit the mark to 4k
+ * minus the size of the entry. Say 4097 since we want
+ * an extra slot for \0.
+ */
+ start_mark = strndup(optarg, 4097 -
+ sizeof(struct log_write_entry));
+ if (!start_mark) {
+ fprintf(stderr, "Couldn't allocate memory\n");
+ exit(1);
+ }
+ break;
+ case END_MARK:
+ /*
+ * Biggest sectorsize is 4k atm, so limit the mark to 4k
+ * minus the size of the entry. Say 4097 since we want
+ * an extra slot for \0.
+ */
+ end_mark = strndup(optarg, 4097 -
+ sizeof(struct log_write_entry));
+ if (!end_mark) {
+ fprintf(stderr, "Couldn't allocate memory\n");
+ exit(1);
+ }
+ stop_flags |= LOG_MARK_FLAG;
+ break;
+ case LOG:
+ logfile = strdup(optarg);
+ if (!logfile) {
+ fprintf(stderr, "Couldn't allocate memory\n");
+ exit(1);
+ }
+ break;
+ case REPLAY:
+ replayfile = strdup(optarg);
+ if (!replayfile) {
+ fprintf(stderr, "Couldn't allocate memory\n");
+ exit(1);
+ }
+ break;
+ case LIMIT:
+ run_limit = strtoull(optarg, &tmp, 0);
+ if (tmp && *tmp != '\0') {
+ fprintf(stderr, "Invalid entry number\n");
+ exit(1);
+ }
+ tmp = NULL;
+ break;
+ case FIND:
+ find_mode = 1;
+ break;
+ case NUM_ENTRIES:
+ print_num_entries = 1;
+ break;
+ case NO_DISCARD:
+ discard = 0;
+ break;
+ case FSCK:
+ fsck_command = strdup(optarg);
+ if (!fsck_command) {
+ fprintf(stderr, "Couldn't allocate memory\n");
+ exit(1);
+ }
+ break;
+ case CHECK:
+ if (!strcmp(optarg, "flush")) {
+ check_mode = CHECK_FLUSH;
+ } else if (!strcmp(optarg, "fua")) {
+ check_mode = CHECK_FUA;
+ } else {
+ check_mode = CHECK_NUMBER;
+ check_number = strtoull(optarg, &tmp, 0);
+ if (!check_number || (tmp && *tmp != '\0')) {
+ fprintf(stderr,
+ "Invalid entry number\n");
+ exit(1);
+ }
+ tmp = NULL;
+ }
+ break;
+ default:
+ usage();
+ }
+ }
+
+ if (!logfile)
+ usage();
+
+ log = log_open(logfile, replayfile);
+ if (!log)
+ exit(1);
+ free(logfile);
+ free(replayfile);
+
+ if (!discard)
+ log->flags |= LOG_IGNORE_DISCARD;
+
+ entry = malloc(log->sectorsize);
+ if (!entry) {
+ fprintf(stderr, "Couldn't allocate buffer\n");
+ log_free(log);
+ exit(1);
+ }
+
+ if (start_mark) {
+ ret = seek_to_mark(log, entry, start_mark);
+ if (ret)
+ exit(1);
+ free(start_mark);
+ } else {
+ ret = log_seek_entry(log, start_entry);
+ if (ret)
+ exit(1);
+ }
+
+ if ((fsck_command && !check_mode) || (!fsck_command && check_mode))
+ usage();
+
+ /* We just want to find a given entry */
+ if (find_mode) {
+ while ((ret = log_seek_next_entry(log, entry, 1)) == 0) {
+ num_entries++;
+ if ((run_limit && num_entries == run_limit) ||
+ should_stop(entry, stop_flags, end_mark)) {
+ printf("%llu\n",
+ (unsigned long long)log->cur_entry - 1);
+ log_free(log);
+ return 0;
+ }
+ }
+ log_free(log);
+ if (ret < 0)
+ return ret;
+ fprintf(stderr, "Couldn't find entry\n");
+ return 1;
+ }
+
+ /* Used for scripts, just print the number of entries in the log */
+ if (print_num_entries) {
+ printf("%llu\n", (unsigned long long)log->nr_entries);
+ log_free(log);
+ return 0;
+ }
+
+ /* No replay, just spit out the log info. */
+ if (!replayfile) {
+ printf("Log version=%d, sectorsize=%lu, entries=%llu\n",
+ WRITE_LOG_VERSION, (unsigned long)log->sectorsize,
+ (unsigned long long)log->nr_entries);
+ log_free(log);
+ return 0;
+ }
+
+ while ((ret = log_replay_next_entry(log, entry, 1)) == 0) {
+ num_entries++;
+ if (fsck_command) {
+ if ((check_mode == CHECK_NUMBER) &&
+ !(num_entries % check_number))
+ ret = run_fsck(log, fsck_command);
+ else if ((check_mode == CHECK_FUA) &&
+ should_stop(entry, LOG_FUA_FLAG, NULL))
+ ret = run_fsck(log, fsck_command);
+ else if ((check_mode == CHECK_FLUSH) &&
+ should_stop(entry, LOG_FLUSH_FLAG, NULL))
+ ret = run_fsck(log, fsck_command);
+ else
+ ret = 0;
+ if (ret) {
+ fprintf(stderr, "Fsck errored out on entry "
+ "%llu\n",
+ (unsigned long long)log->cur_entry - 1);
+ break;
+ }
+ }
+
+ if ((run_limit && num_entries == run_limit) ||
+ should_stop(entry, stop_flags, end_mark))
+ break;
+ }
+ fsync(log->replayfd);
+ log_free(log);
+ free(end_mark);
+ if (ret < 0)
+ exit(1);
+ return 0;
+}
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 07/13] replay-log: add validations for corrupt log entries
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
` (5 preceding siblings ...)
2017-09-05 19:11 ` [PATCH v3 06/13] log-writes: add replay-log program to replay dm-log-writes target Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 08/13] replay-log: add support for replaying ops in target device sector range Amir Goldstein
` (5 subsequent siblings)
12 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
Check for all zeros entry and for non zero padded entry
and report log offset of corrupted log entry.
Also report log offsets with -v and -vv debug prints.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
src/log-writes/log-writes.c | 84 ++++++++++++++++++++++++++++++++++-----------
src/log-writes/log-writes.h | 2 ++
src/log-writes/replay-log.c | 7 ++--
3 files changed, 70 insertions(+), 23 deletions(-)
diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c
index a215fef..dbfeef7 100644
--- a/src/log-writes/log-writes.c
+++ b/src/log-writes/log-writes.c
@@ -118,6 +118,26 @@ int log_discard(struct log *log, struct log_write_entry *entry)
}
/*
+ * @entry: entry to be replayed.
+ *
+ * @return: 1 if the entry is sane, 0 if it is invalid.
+ *
+ * Check if this is a sane log entry.
+ */
+int log_entry_valid(struct log_write_entry *entry)
+{
+ u64 flags = le64_to_cpu(entry->flags);
+
+ /* Suspect all zeroes entry */
+ if (!flags && !entry->nr_sectors)
+ return 0;
+ /* Suspect non zero padded entry */
+ if (flags != LOG_MARK_FLAG && entry->data[0] != 0)
+ return 0;
+ return 1;
+}
+
+/*
* @log: the log we are replaying.
* @entry: where we put the entry.
* @read_data: read the entry data as well, entry must be log->sectorsize sized
@@ -146,24 +166,32 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
fprintf(stderr, "Error reading entry: %d\n", errno);
return -1;
}
+ if (!log_entry_valid(entry)) {
+ fprintf(stderr, "Malformed entry @%llu\n",
+ log->cur_pos / log->sectorsize);
+ return -1;
+ }
log->cur_entry++;
size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
if (read_size < log->sectorsize) {
- if (lseek(log->logfd,
- log->sectorsize - sizeof(struct log_write_entry),
- SEEK_CUR) == (off_t)-1) {
+ log->cur_pos = lseek(log->logfd,
+ log->sectorsize - sizeof(struct log_write_entry), SEEK_CUR);
+ if (log->cur_pos == (off_t)-1) {
fprintf(stderr, "Error seeking in log: %d\n", errno);
return -1;
}
+ } else {
+ log->cur_pos += read_size;
}
- if (log_writes_verbose)
- printf("replaying %d: sector %llu, size %llu, flags %llu\n",
- (int)log->cur_entry - 1,
+ if (log_writes_verbose) {
+ printf("replaying %d@%llu: sector %llu, size %llu, flags %llu\n",
+ (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
(unsigned long long)le64_to_cpu(entry->sector),
(unsigned long long)size,
(unsigned long long)le64_to_cpu(entry->flags));
+ }
if (!size)
return 0;
@@ -183,6 +211,7 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
free(buf);
return -1;
}
+ log->cur_pos += size;
offset = le64_to_cpu(entry->sector) * log->sectorsize;
ret = pwrite(log->replayfd, buf, size, offset);
@@ -212,7 +241,8 @@ int log_seek_entry(struct log *log, u64 entry_num)
}
/* Skip the first sector containing the log super block */
- if (lseek(log->logfd, log->sectorsize, SEEK_SET) == (off_t)-1) {
+ log->cur_pos = lseek(log->logfd, log->sectorsize, SEEK_SET);
+ if (log->cur_pos == (off_t)-1) {
fprintf(stderr, "Error seeking in file: %d\n", errno);
return -1;
}
@@ -229,9 +259,14 @@ int log_seek_entry(struct log *log, u64 entry_num)
fprintf(stderr, "Error reading entry: %d\n", errno);
return -1;
}
+ if (!log_entry_valid(&entry)) {
+ fprintf(stderr, "Malformed entry @%llu\n",
+ log->cur_pos / log->sectorsize);
+ return -1;
+ }
if (log_writes_verbose > 1)
- printf("seek entry %d: %llu, size %llu, flags %llu\n",
- (int)i,
+ printf("seek entry %d@%llu: %llu, size %llu, flags %llu\n",
+ (int)i, log->cur_pos / log->sectorsize,
(unsigned long long)le64_to_cpu(entry.sector),
(unsigned long long)le64_to_cpu(entry.nr_sectors),
(unsigned long long)le64_to_cpu(entry.flags));
@@ -240,7 +275,8 @@ int log_seek_entry(struct log *log, u64 entry_num)
if (!(flags & LOG_DISCARD_FLAG))
seek_size += le64_to_cpu(entry.nr_sectors) *
log->sectorsize;
- if (lseek(log->logfd, seek_size, SEEK_CUR) == (off_t)-1) {
+ log->cur_pos = lseek(log->logfd, seek_size, SEEK_CUR);
+ if (log->cur_pos == (off_t)-1) {
fprintf(stderr, "Error seeking in file: %d\n", errno);
return -1;
}
@@ -277,29 +313,37 @@ int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
fprintf(stderr, "Error reading entry: %d\n", errno);
return -1;
}
+ if (!log_entry_valid(entry)) {
+ fprintf(stderr, "Malformed entry @%llu\n",
+ log->cur_pos / log->sectorsize);
+ return -1;
+ }
log->cur_entry++;
if (read_size < log->sectorsize) {
- if (lseek(log->logfd,
- log->sectorsize - sizeof(struct log_write_entry),
- SEEK_CUR) == (off_t)-1) {
+ log->cur_pos = lseek(log->logfd,
+ log->sectorsize - sizeof(struct log_write_entry), SEEK_CUR);
+ if (log->cur_pos == (off_t)-1) {
fprintf(stderr, "Error seeking in log: %d\n", errno);
return -1;
}
+ } else {
+ log->cur_pos += read_size;
}
if (log_writes_verbose > 1)
- printf("seek entry %d: %llu, size %llu, flags %llu\n",
- (int)log->cur_entry - 1,
+ printf("seek entry %d@%llu: %llu, size %llu, flags %llu\n",
+ (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
(unsigned long long)le64_to_cpu(entry->sector),
(unsigned long long)le64_to_cpu(entry->nr_sectors),
(unsigned long long)le64_to_cpu(entry->flags));
- flags = le32_to_cpu(entry->flags);
- read_size = le32_to_cpu(entry->nr_sectors) * log->sectorsize;
+ flags = le64_to_cpu(entry->flags);
+ read_size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
if (!read_size || (flags & LOG_DISCARD_FLAG))
return 0;
- if (lseek(log->logfd, read_size, SEEK_CUR) == (off_t)-1) {
+ log->cur_pos = lseek(log->logfd, read_size, SEEK_CUR);
+ if (log->cur_pos == (off_t)-1) {
fprintf(stderr, "Error seeking in log: %d\n", errno);
return -1;
}
@@ -369,8 +413,8 @@ struct log *log_open(char *logfile, char *replayfile)
log->nr_entries = le64_to_cpu(super.nr_entries);
log->max_zero_size = 128 * 1024 * 1024;
- if (lseek(log->logfd, log->sectorsize - sizeof(super), SEEK_CUR) ==
- (off_t) -1) {
+ log->cur_pos = lseek(log->logfd, log->sectorsize - sizeof(super), SEEK_CUR);
+ if (log->cur_pos == (off_t) -1) {
fprintf(stderr, "Error seeking to first entry: %d\n", errno);
log_free(log);
return NULL;
diff --git a/src/log-writes/log-writes.h b/src/log-writes/log-writes.h
index 6cadb66..c89b119 100644
--- a/src/log-writes/log-writes.h
+++ b/src/log-writes/log-writes.h
@@ -50,6 +50,8 @@ struct log_write_entry {
__le64 nr_sectors;
__le64 flags;
__le64 data_len;
+ /* Read extra byte when seeking to verify that header is zero padded */
+ char data[1];
};
#define LOG_IGNORE_DISCARD (1 << 0)
diff --git a/src/log-writes/replay-log.c b/src/log-writes/replay-log.c
index c3de9c4..cf67931 100644
--- a/src/log-writes/replay-log.c
+++ b/src/log-writes/replay-log.c
@@ -75,7 +75,7 @@ static int should_stop(struct log_write_entry *entry, u64 stop_flags,
u64 flags = le64_to_cpu(entry->flags);
int check_mark = (stop_flags & LOG_MARK_FLAG);
/* mark data begins after entry header */
- char *buf = (char *)(entry + 1);
+ char *buf = entry->data;
/* entry buffer is padded with at least 1 zero after data_len */
u64 buflen = le64_to_cpu(entry->data_len) + 1;
@@ -293,8 +293,9 @@ int main(int argc, char **argv)
num_entries++;
if ((run_limit && num_entries == run_limit) ||
should_stop(entry, stop_flags, end_mark)) {
- printf("%llu\n",
- (unsigned long long)log->cur_entry - 1);
+ printf("%llu@%llu\n",
+ (unsigned long long)log->cur_entry - 1,
+ log->cur_pos / log->sectorsize);
log_free(log);
return 0;
}
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 08/13] replay-log: add support for replaying ops in target device sector range
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
` (6 preceding siblings ...)
2017-09-05 19:11 ` [PATCH v3 07/13] replay-log: add validations for corrupt log entries Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 09/13] fstests: add support for working with dm-log-writes target Amir Goldstein
` (4 subsequent siblings)
12 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
Using command line options --start-sector and --end-sector, only
operations acting on the specified target device range will be
replayed.
Single vebbose mode (-v) prints out only replayed operations.
Double verbose mode (-vv) prints out also skipped operations.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
src/log-writes/log-writes.c | 37 +++++++++++++++++++++++++++++++++++--
src/log-writes/log-writes.h | 2 ++
src/log-writes/replay-log.c | 31 +++++++++++++++++++++++++++++++
3 files changed, 68 insertions(+), 2 deletions(-)
diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c
index dbfeef7..fab447a 100644
--- a/src/log-writes/log-writes.c
+++ b/src/log-writes/log-writes.c
@@ -118,6 +118,27 @@ int log_discard(struct log *log, struct log_write_entry *entry)
}
/*
+ * @log: the log we are replaying.
+ * @entry: entry to be replayed.
+ *
+ * @return: 0 if we should replay the entry, > 0 if we should skip it.
+ *
+ * Should we skip the entry in our log or replay onto the replay device.
+ */
+int log_should_skip(struct log *log, struct log_write_entry *entry)
+{
+ u64 sector = le64_to_cpu(entry->sector);
+ u64 nr_sectors = le64_to_cpu(entry->nr_sectors);
+
+ if (!nr_sectors)
+ return 0;
+ if (sector + nr_sectors <= log->start_sector ||
+ sector > log->end_sector)
+ return 1;
+ return 0;
+}
+
+/*
* @entry: entry to be replayed.
*
* @return: 1 if the entry is sane, 0 if it is invalid.
@@ -157,6 +178,7 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
char *buf;
ssize_t ret;
off_t offset;
+ int skip = 0;
if (log->cur_entry >= log->nr_entries)
return 1;
@@ -185,8 +207,10 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
log->cur_pos += read_size;
}
- if (log_writes_verbose) {
- printf("replaying %d@%llu: sector %llu, size %llu, flags %llu\n",
+ skip = log_should_skip(log, entry);
+ if (log_writes_verbose > 1 || (log_writes_verbose && !skip)) {
+ printf("%s %d@%llu: sector %llu, size %llu, flags %llu\n",
+ skip ? "skipping" : "replaying",
(int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
(unsigned long long)le64_to_cpu(entry->sector),
(unsigned long long)size,
@@ -199,6 +223,15 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
if (flags & LOG_DISCARD_FLAG)
return log_discard(log, entry);
+ if (skip) {
+ log->cur_pos = lseek(log->logfd, size, SEEK_CUR);
+ if (log->cur_pos == (off_t)-1) {
+ fprintf(stderr, "Error seeking in log: %d\n", errno);
+ return -1;
+ }
+ return 0;
+ }
+
buf = malloc(size);
if (!buf) {
fprintf(stderr, "Error allocating buffer %llu entry %llu\n", (unsigned long long)size, (unsigned long long)log->cur_entry - 1);
diff --git a/src/log-writes/log-writes.h b/src/log-writes/log-writes.h
index c89b119..14242ee 100644
--- a/src/log-writes/log-writes.h
+++ b/src/log-writes/log-writes.h
@@ -62,6 +62,8 @@ struct log {
int replayfd;
unsigned long flags;
u64 sectorsize;
+ u64 start_sector;
+ u64 end_sector;
u64 nr_entries;
u64 cur_entry;
u64 max_zero_size;
diff --git a/src/log-writes/replay-log.c b/src/log-writes/replay-log.c
index cf67931..8457937 100644
--- a/src/log-writes/replay-log.c
+++ b/src/log-writes/replay-log.c
@@ -20,6 +20,8 @@ enum option_indexes {
FSCK,
CHECK,
START_MARK,
+ START_SECTOR,
+ END_SECTOR,
};
static struct option long_options[] = {
@@ -37,6 +39,8 @@ static struct option long_options[] = {
{"fsck", required_argument, NULL, 0},
{"check", required_argument, NULL, 0},
{"start-mark", required_argument, NULL, 0},
+ {"start-sector", required_argument, NULL, 0},
+ {"end-sector", required_argument, NULL, 0},
{ NULL, 0, NULL, 0 },
};
@@ -61,6 +65,12 @@ static void usage(void)
"--check\n");
fprintf(stderr, "\t--check [<number>|flush|fua] when to check the "
"file system, mush specify --fsck\n");
+ fprintf(stderr, "\t--start-sector <sector> - replay ops on region "
+ "from <sector> onto <device>\n");
+ fprintf(stderr, "\t--end-sector <sector> - replay ops on region "
+ "to <sector> onto <device>\n");
+ fprintf(stderr, "\t-v or --verbose - print replayed ops\n");
+ fprintf(stderr, "\t-vv - print also skipped ops\n");
exit(1);
}
@@ -129,6 +139,8 @@ int main(int argc, char **argv)
struct log_write_entry *entry;
u64 stop_flags = 0;
u64 start_entry = 0;
+ u64 start_sector = 0;
+ u64 end_sector = -1ULL;
u64 run_limit = 0;
u64 num_entries = 0;
u64 check_number = 0;
@@ -249,6 +261,22 @@ int main(int argc, char **argv)
tmp = NULL;
}
break;
+ case START_SECTOR:
+ start_sector = strtoull(optarg, &tmp, 0);
+ if (tmp && *tmp != '\0') {
+ fprintf(stderr, "Invalid sector number\n");
+ exit(1);
+ }
+ tmp = NULL;
+ break;
+ case END_SECTOR:
+ end_sector = strtoull(optarg, &tmp, 0);
+ if (tmp && *tmp != '\0') {
+ fprintf(stderr, "Invalid sector number\n");
+ exit(1);
+ }
+ tmp = NULL;
+ break;
default:
usage();
}
@@ -266,6 +294,9 @@ int main(int argc, char **argv)
if (!discard)
log->flags |= LOG_IGNORE_DISCARD;
+ log->start_sector = start_sector;
+ log->end_sector = end_sector;
+
entry = malloc(log->sectorsize);
if (!entry) {
fprintf(stderr, "Couldn't allocate buffer\n");
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 09/13] fstests: add support for working with dm-log-writes target
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
` (7 preceding siblings ...)
2017-09-05 19:11 ` [PATCH v3 08/13] replay-log: add support for replaying ops in target device sector range Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-07 7:45 ` Eryu Guan
2017-09-07 7:47 ` Eryu Guan
2017-09-05 19:11 ` [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes Amir Goldstein
` (3 subsequent siblings)
12 siblings, 2 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
Cherry-picked the relevant common bits from commit 70d41e17164b
in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
Quoting from Josef's commit message:
This patch adds the supporting code for using the dm-log-writes
target. The dmlogwrites code is similar to the dmflakey code, it just
gives us functions to build and tear down a dm-log-writes target. We
add a new LOGWRITES_DEV variable to take in the device we will use as
the log and add checks for that.
[Amir:]
- Removed unneeded _test_falloc_support
- Moved _require_log_writes to dmlogwrites
- Document _require_log_writes
- Address review comments by Eryu Guan
Cc: Josef Bacik <jbacik@fb.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
README | 2 +
common/dmlogwrites | 94 ++++++++++++++++++++++++++++++++++++++++++++
doc/requirement-checking.txt | 20 ++++++++++
3 files changed, 116 insertions(+)
create mode 100644 common/dmlogwrites
diff --git a/README b/README
index 9456fa7..4963d28 100644
--- a/README
+++ b/README
@@ -91,6 +91,8 @@ Preparing system for tests:
- set TEST_XFS_SCRUB=1 to have _check_xfs_filesystem run
xfs_scrub -vd to scrub the filesystem metadata online before
unmounting to run the offline check.
+ - setenv LOGWRITES_DEV to a block device to use for power fail
+ testing.
- or add a case to the switch in common/config assigning
these variables based on the hostname of your test
diff --git a/common/dmlogwrites b/common/dmlogwrites
new file mode 100644
index 0000000..592b047
--- /dev/null
+++ b/common/dmlogwrites
@@ -0,0 +1,94 @@
+##/bin/bash
+#
+# Copyright (c) 2015 Facebook, Inc. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#
+#
+# common functions for setting up and tearing down a dm log-writes device
+
+echo $MOUNT_OPTIONS | grep -q dax
+if [ $? -eq 0 ]; then
+ _notrun "Cannot run tests with DAX on dmerror devices"
+fi
+
+_require_log_writes()
+{
+ [ -z "$LOGWRITES_DEV" -o ! -b "$LOGWRITES_DEV" ] && \
+ _notrun "This test requires a valid \$LOGWRITES_DEV"
+
+ _require_dm_target log-writes
+ _require_test_program "log-writes/replay-log"
+}
+
+_log_writes_init()
+{
+ local BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+ LOGWRITES_NAME=logwrites-test
+ LOGWRITES_DMDEV=/dev/mapper/$LOGWRITES_NAME
+ LOGWRITES_TABLE="0 $BLK_DEV_SIZE log-writes $SCRATCH_DEV $LOGWRITES_DEV"
+ $DMSETUP_PROG create $LOGWRITES_NAME --table "$LOGWRITES_TABLE" || \
+ _fail "failed to create log-writes device"
+ $DMSETUP_PROG mknodes > /dev/null 2>&1
+}
+
+_log_writes_mark()
+{
+ [ $# -ne 1 ] && _fail "_log_writes_mark takes one argument"
+ $DMSETUP_PROG message $LOGWRITES_NAME 0 mark $1
+}
+
+_log_writes_mkfs()
+{
+ _scratch_options mkfs
+ _mkfs_dev $SCRATCH_OPTIONS $LOGWRITES_DMDEV
+ _log_writes_mark mkfs
+}
+
+_log_writes_mount()
+{
+ _scratch_options mount
+ $MOUNT_PROG -t $FSTYP `_common_dev_mount_options $*` $SCRATCH_OPTIONS \
+ $LOGWRITES_DMDEV $SCRATCH_MNT
+}
+
+_log_writes_unmount()
+{
+ $UMOUNT_PROG $SCRATCH_MNT
+}
+
+# _replay_log <mark>
+#
+# This replays the log contained on $LOGWRITES_DEV onto $SCRATCH_DEV upto the
+# mark passed in.
+_log_writes_replay_log()
+{
+ _mark=$1
+
+ $here/src/log-writes/replay-log --log $LOGWRITES_DEV --replay $SCRATCH_DEV \
+ --end-mark $_mark >> $seqres.full 2>&1
+ [ $? -ne 0 ] && _fail "replay failed"
+}
+
+_log_writes_remove()
+{
+ $DMSETUP_PROG remove $LOGWRITES_NAME > /dev/null 2>&1
+ $DMSETUP_PROG mknodes > /dev/null 2>&1
+}
+
+_log_writes_cleanup()
+{
+ $UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+ _log_writes_remove
+}
diff --git a/doc/requirement-checking.txt b/doc/requirement-checking.txt
index 95d10e6..4e01b1f 100644
--- a/doc/requirement-checking.txt
+++ b/doc/requirement-checking.txt
@@ -21,6 +21,10 @@ they have. This is done with _require_<xxx> macros, which may take parameters.
_require_statx
+ (4) Device mapper requirement.
+
+ _require_dm_target
+ _require_log_writes
====================
GENERAL REQUIREMENTS
@@ -102,3 +106,19 @@ _require_statx
The test requires the use of the statx() system call and will be skipped
if it isn't available in the kernel.
+
+
+==========================
+DEVICE MAPPER REQUIREMENTS
+==========================
+
+_require_dm_target <name>
+
+ The test requires the use of the device mapper target and will be skipped
+ if it isn't available in the kernel.
+
+_require_log_writes
+
+ The test requires the use of the device mapper target log-writes.
+ The test also requires the test program log-writes/replay-log is built
+ and will be skipped if either isn't available.
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
` (8 preceding siblings ...)
2017-09-05 19:11 ` [PATCH v3 09/13] fstests: add support for working with dm-log-writes target Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-07 7:50 ` Eryu Guan
2017-11-27 9:56 ` Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 11/13] fstests: regression test for ext4 crash consistency bug Amir Goldstein
` (2 subsequent siblings)
12 siblings, 2 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
Cherry-picked the test from commit 70d41e17164b
in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
Quoting from Josef's commit message:
The test just runs some ops and exits, then finds all of the good buffers
in the directory we provided and:
- replays up to the mark given
- mounts the file system and compares the md5sum
- unmounts and fsck's to check for metadata integrity
dm-log-writes will pretend to do discard and the replay-log tool will
replay it properly depending on the underlying device, either by writing
0's or actually calling the discard ioctl, so I've enabled discard in the
test for maximum fun.
[Amir:]
- Removed unneeded _test_falloc_support dynamic FSX_OPTS
- Fold repetitions into for loops
- Added place holders for using constant random seeds
- Add pre umount checkpint
- Add test to new 'replay' group
- Address review comments by Eryu Guan
Cc: Josef Bacik <jbacik@fb.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
tests/generic/500 | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++
tests/generic/500.out | 2 +
tests/generic/group | 1 +
3 files changed, 138 insertions(+)
create mode 100755 tests/generic/500
create mode 100644 tests/generic/500.out
diff --git a/tests/generic/500 b/tests/generic/500
new file mode 100755
index 0000000..82f7a93
--- /dev/null
+++ b/tests/generic/500
@@ -0,0 +1,135 @@
+#! /bin/bash
+# FS QA Test No. 500
+#
+# Run fsx with log writes to verify power fail safeness.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2015 Facebook. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _log_writes_cleanup
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmlogwrites
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_scratch_nocheck
+_require_log_writes
+
+rm -f $seqres.full
+
+check_files()
+{
+ local name=$1
+
+ # Now look for our files
+ for i in $(find $SANITY_DIR -type f | grep $name | grep mark)
+ do
+ local filename=$(basename $i)
+ local mark="${filename##*.}"
+ echo "checking $filename" >> $seqres.full
+ _log_writes_replay_log $filename
+ _scratch_mount
+ local expected_md5=$(_md5_checksum $i)
+ local md5=$(_md5_checksum $SCRATCH_MNT/$name)
+ [ "${md5}" != "${expected_md5}" ] && _fail "$filename md5sum mismatched"
+ _scratch_unmount
+ _check_scratch_fs
+ done
+}
+
+SANITY_DIR=$TEST_DIR/fsxtests
+rm -rf $SANITY_DIR
+mkdir $SANITY_DIR
+
+# Create the log
+_log_writes_init
+
+_log_writes_mkfs >> $seqres.full 2>&1
+
+# Log writes emulates discard support, turn it on for maximum crying.
+_log_writes_mount -o discard
+
+NUM_FILES=4
+NUM_OPS=200
+FSX_OPTS="-N $NUM_OPS -d -P $SANITY_DIR -i $LOGWRITES_DMDEV"
+# Set random seeds for fsx runs (0 for timestamp + pid)
+seeds=(0 0 0 0)
+# Run fsx for a while
+for j in `seq 0 $((NUM_FILES-1))`
+do
+ run_check $here/ltp/fsx $FSX_OPTS -S ${seeds[$j]} -j $j $SCRATCH_MNT/testfile$j &
+done
+wait
+
+test_md5=()
+for j in `seq 0 $((NUM_FILES-1))`
+do
+ test_md5[$j]=$(_md5_checksum $SCRATCH_MNT/testfile$j)
+done
+
+# Unmount the scratch dir and tear down the log writes target
+_log_writes_mark last
+_log_writes_unmount
+_log_writes_mark end
+_log_writes_remove
+_check_scratch_fs
+
+# check pre umount
+echo "checking pre umount" >> $seqres.full
+_log_writes_replay_log last
+_scratch_mount
+_scratch_unmount
+_check_scratch_fs
+
+for j in `seq 0 $((NUM_FILES-1))`
+do
+ check_files testfile$j
+done
+
+# Check the end
+echo "checking post umount" >> $seqres.full
+_log_writes_replay_log end
+_scratch_mount
+for j in `seq 0 $((NUM_FILES-1))`
+do
+ md5=$(_md5_checksum $SCRATCH_MNT/testfile$j)
+ [ "${md5}" != "${test_md5[$j]}" ] && _fail "testfile$j end md5sum mismatched"
+done
+_scratch_unmount
+_check_scratch_fs
+
+echo "Silence is golden"
+status=0
+exit
+
diff --git a/tests/generic/500.out b/tests/generic/500.out
new file mode 100644
index 0000000..883b2ca
--- /dev/null
+++ b/tests/generic/500.out
@@ -0,0 +1,2 @@
+QA output created by 500
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 1894de0..c857153 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -457,3 +457,4 @@
452 auto quick
453 auto quick dir
454 auto quick attr
+500 auto log replay
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 11/13] fstests: regression test for ext4 crash consistency bug
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
` (9 preceding siblings ...)
2017-09-05 19:11 ` [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-07 7:52 ` Eryu Guan
2017-09-05 19:11 ` [PATCH v3 12/13] fstests: crash consistency fsx test for cloned files Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 13/13] fstests: regression test for xfs leftover CoW extent error Amir Goldstein
12 siblings, 1 reply; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
This test is motivated by a bug found in ext4 during random crash
consistency tests.
This test uses device mapper flakey target to demonstrate the bug
found using device mapper log-writes target.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
tests/generic/501 | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++
tests/generic/501.out | 2 ++
tests/generic/group | 1 +
3 files changed, 83 insertions(+)
create mode 100755 tests/generic/501
create mode 100644 tests/generic/501.out
diff --git a/tests/generic/501 b/tests/generic/501
new file mode 100755
index 0000000..ccb513d
--- /dev/null
+++ b/tests/generic/501
@@ -0,0 +1,80 @@
+#! /bin/bash
+# FS QA Test No. 501
+#
+# This test is motivated by a bug found in ext4 during random crash
+# consistency tests.
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2017 CTERA Networks. All Rights Reserved.
+# Author: Amir Goldstein <amir73il@gmail.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_flakey
+ cd /
+ rm -f $tmp.*
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+_require_metadata_journaling $SCRATCH_DEV
+
+rm -f $seqres.full
+
+_scratch_mkfs >> $seqres.full 2>&1
+
+_init_flakey
+_mount_flakey
+
+fsxops=$tmp.fsxops
+cat <<EOF > $fsxops
+write 0x137dd 0xdc69 0x0
+fallocate 0xb531 0xb5ad 0x21446
+collapse_range 0x1c000 0x4000 0x21446
+write 0x3e5ec 0x1a14 0x21446
+zero_range 0x20fac 0x6d9c 0x40000 keep_size
+mapwrite 0x216ad 0x274f 0x40000
+EOF
+run_check $here/ltp/fsx -d --replay-ops $fsxops $SCRATCH_MNT/testfile
+
+_flakey_drop_and_remount
+_unmount_flakey
+_cleanup_flakey
+_check_scratch_fs
+
+echo "Silence is golden"
+
+status=0
+exit
diff --git a/tests/generic/501.out b/tests/generic/501.out
new file mode 100644
index 0000000..00133b6
--- /dev/null
+++ b/tests/generic/501.out
@@ -0,0 +1,2 @@
+QA output created by 501
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index c857153..1e1b524 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -458,3 +458,4 @@
453 auto quick dir
454 auto quick attr
500 auto log replay
+501 auto quick metadata
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 12/13] fstests: crash consistency fsx test for cloned files
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
` (10 preceding siblings ...)
2017-09-05 19:11 ` [PATCH v3 11/13] fstests: regression test for ext4 crash consistency bug Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 13/13] fstests: regression test for xfs leftover CoW extent error Amir Goldstein
12 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
tests/generic/502 | 142 ++++++++++++++++++++++++++++++++++++++++++++++++++
tests/generic/502.out | 2 +
tests/generic/group | 1 +
3 files changed, 145 insertions(+)
create mode 100755 tests/generic/502
create mode 100644 tests/generic/502.out
diff --git a/tests/generic/502 b/tests/generic/502
new file mode 100755
index 0000000..a15a311
--- /dev/null
+++ b/tests/generic/502
@@ -0,0 +1,142 @@
+#! /bin/bash
+# FS QA Test No. 502
+#
+# Run fsx with log writes on cloned files to verify power fail safeness.
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2017 CTERA Networks. All Rights Reserved.
+# Author: Amir Goldstein <amir73il@gmail.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _log_writes_cleanup
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+. ./common/dmlogwrites
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_scratch_reflink
+_require_cp_reflink
+_require_log_writes
+
+rm -f $seqres.full
+
+check_files()
+{
+ local name=$1
+
+ # Now look for our files
+ for i in $(find $SANITY_DIR -type f | grep $name | grep mark)
+ do
+ local filename=$(basename $i)
+ local mark="${filename##*.}"
+ echo "checking $filename" >> $seqres.full
+ _log_writes_replay_log $filename
+ _scratch_mount
+ local expected_md5=$(_md5_checksum $i)
+ local md5=$(_md5_checksum $SCRATCH_MNT/$name)
+ [ "${md5}" != "${expected_md5}" ] && _fail "$filename md5sum mismatched"
+ _scratch_unmount
+ _check_scratch_fs
+ done
+}
+
+SANITY_DIR=$TEST_DIR/fsxtests
+rm -rf $SANITY_DIR
+mkdir $SANITY_DIR
+
+# Create the log
+_log_writes_init
+
+_log_writes_mkfs >> $seqres.full 2>&1
+
+# Log writes emulates discard support, turn it on for maximum crying.
+_log_writes_mount -o discard
+
+# write testfile index -1 to be cloned to testfile0
+$XFS_IO_PROG -f -c "pwrite -S 0xff 0 256k" -c "fsync" \
+ $SCRATCH_MNT/testfile-1 > /dev/null 2>&1
+
+NUM_FILES=10
+NUM_OPS=100
+FSX_OPTS="-N $NUM_OPS -d -k -P $SANITY_DIR -i $LOGWRITES_DMDEV"
+# Set random seeds for fsx runs (0 for timestamp + pid)
+seeds=(0 0 0 0 0 0 0 0 0 0)
+# Run fsx for a while
+for j in `seq 0 $((NUM_FILES-1))`
+do
+ # clone the clone from prev iteration which has already mutated
+ cp --reflink $SCRATCH_MNT/testfile$((j-1)) $SCRATCH_MNT/testfile$j
+ run_check $here/ltp/fsx $FSX_OPTS -S ${seeds[$j]} -j $j $SCRATCH_MNT/testfile$j &
+done
+wait
+
+test_md5=()
+for j in `seq 0 $((NUM_FILES-1))`
+do
+ test_md5[$j]=$(_md5_checksum $SCRATCH_MNT/testfile$j)
+done
+
+# Unmount the scratch dir and tear down the log writes target
+_log_writes_mark last
+_log_writes_unmount
+_log_writes_mark end
+_log_writes_remove
+_check_scratch_fs
+
+# check pre umount
+echo "checking pre umount" >> $seqres.full
+_log_writes_replay_log last
+_scratch_mount
+_scratch_unmount
+_check_scratch_fs
+
+for j in `seq 0 $((NUM_FILES-1))`
+do
+ check_files testfile$j
+done
+
+# Check the end
+echo "checking post umount" >> $seqres.full
+_log_writes_replay_log end
+_scratch_mount
+for j in `seq 0 $((NUM_FILES-1))`
+do
+ md5=$(_md5_checksum $SCRATCH_MNT/testfile$j)
+ [ "${md5}" != "${test_md5[$j]}" ] && _fail "testfile$j end md5sum mismatched"
+done
+
+echo "Silence is golden"
+status=0
+exit
+
diff --git a/tests/generic/502.out b/tests/generic/502.out
new file mode 100644
index 0000000..930f6d4
--- /dev/null
+++ b/tests/generic/502.out
@@ -0,0 +1,2 @@
+QA output created by 502
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 1e1b524..4324775 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -459,3 +459,4 @@
454 auto quick attr
500 auto log replay
501 auto quick metadata
+502 auto log replay clone
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v3 13/13] fstests: regression test for xfs leftover CoW extent error
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
` (11 preceding siblings ...)
2017-09-05 19:11 ` [PATCH v3 12/13] fstests: crash consistency fsx test for cloned files Amir Goldstein
@ 2017-09-05 19:11 ` Amir Goldstein
2017-09-07 7:55 ` Eryu Guan
12 siblings, 1 reply; 35+ messages in thread
From: Amir Goldstein @ 2017-09-05 19:11 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
tests/generic/503 | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++
tests/generic/503.out | 2 ++
tests/generic/group | 1 +
3 files changed, 72 insertions(+)
create mode 100755 tests/generic/503
create mode 100644 tests/generic/503.out
diff --git a/tests/generic/503 b/tests/generic/503
new file mode 100755
index 0000000..ebda756
--- /dev/null
+++ b/tests/generic/503
@@ -0,0 +1,69 @@
+#! /bin/bash
+# FS QA Test No. 503
+#
+# Regression test for xfs leftover CoW extents error
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2017 CTERA Networks. All Rights Reserved.
+# Author: Amir Goldstein <amir73il@gmail.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch_reflink
+_require_cp_reflink
+
+rm -f $seqres.full
+
+_scratch_mkfs >/dev/null 2>&1
+_scratch_mount
+
+$XFS_IO_PROG -f -c "pwrite 0 0x40000" \
+ $SCRATCH_MNT/foo > /dev/null 2>&1
+
+cp --reflink $SCRATCH_MNT/foo $SCRATCH_MNT/bar
+
+$XFS_IO_PROG -f -c "fzero -k 0x169f 0x387c" \
+ -c "fcollapse 0x29000 0xd000" \
+ -c "finsert 0 0x8000" \
+ -c "truncate 0x8000" \
+ $SCRATCH_MNT/foo > /dev/null 2>&1
+
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/generic/503.out b/tests/generic/503.out
new file mode 100644
index 0000000..6c1400a
--- /dev/null
+++ b/tests/generic/503.out
@@ -0,0 +1,2 @@
+QA output created by 503
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 4324775..7a9cd78 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -460,3 +460,4 @@
500 auto log replay
501 auto quick metadata
502 auto log replay clone
+503 auto quick clone
--
2.7.4
^ permalink raw reply related [flat|nested] 35+ messages in thread
* Re: [PATCH v3 09/13] fstests: add support for working with dm-log-writes target
2017-09-05 19:11 ` [PATCH v3 09/13] fstests: add support for working with dm-log-writes target Amir Goldstein
@ 2017-09-07 7:45 ` Eryu Guan
2017-09-07 7:47 ` Eryu Guan
1 sibling, 0 replies; 35+ messages in thread
From: Eryu Guan @ 2017-09-07 7:45 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Josef Bacik, fstests, linux-fsdevel
On Tue, Sep 05, 2017 at 10:11:16PM +0300, Amir Goldstein wrote:
> Cherry-picked the relevant common bits from commit 70d41e17164b
> in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
> Quoting from Josef's commit message:
>
> This patch adds the supporting code for using the dm-log-writes
> target. The dmlogwrites code is similar to the dmflakey code, it just
> gives us functions to build and tear down a dm-log-writes target. We
> add a new LOGWRITES_DEV variable to take in the device we will use as
> the log and add checks for that.
>
> [Amir:]
> - Removed unneeded _test_falloc_support
> - Moved _require_log_writes to dmlogwrites
> - Document _require_log_writes
> - Address review comments by Eryu Guan
>
> Cc: Josef Bacik <jbacik@fb.com>
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> ---
> README | 2 +
> common/dmlogwrites | 94 ++++++++++++++++++++++++++++++++++++++++++++
> doc/requirement-checking.txt | 20 ++++++++++
> 3 files changed, 116 insertions(+)
> create mode 100644 common/dmlogwrites
>
> diff --git a/README b/README
> index 9456fa7..4963d28 100644
> --- a/README
> +++ b/README
> @@ -91,6 +91,8 @@ Preparing system for tests:
> - set TEST_XFS_SCRUB=1 to have _check_xfs_filesystem run
> xfs_scrub -vd to scrub the filesystem metadata online before
> unmounting to run the offline check.
> + - setenv LOGWRITES_DEV to a block device to use for power fail
> + testing.
>
> - or add a case to the switch in common/config assigning
> these variables based on the hostname of your test
> diff --git a/common/dmlogwrites b/common/dmlogwrites
> new file mode 100644
> index 0000000..592b047
> --- /dev/null
> +++ b/common/dmlogwrites
> @@ -0,0 +1,94 @@
> +##/bin/bash
> +#
> +# Copyright (c) 2015 Facebook, Inc. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#
> +#
> +# common functions for setting up and tearing down a dm log-writes device
> +
> +echo $MOUNT_OPTIONS | grep -q dax
> +if [ $? -eq 0 ]; then
> + _notrun "Cannot run tests with DAX on dmerror devices"
> +fi
Sorry, I just realized that we can take use of
_exclude_scratch_mount_option helper to do this, and I think we can move
the check inside _require_log_writes() too.
> +
> +_require_log_writes()
> +{
> + [ -z "$LOGWRITES_DEV" -o ! -b "$LOGWRITES_DEV" ] && \
> + _notrun "This test requires a valid \$LOGWRITES_DEV"
> +
> + _require_dm_target log-writes
> + _require_test_program "log-writes/replay-log"
> +}
Otherwise this looks good to me.
Thanks,
Eryu
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 09/13] fstests: add support for working with dm-log-writes target
2017-09-05 19:11 ` [PATCH v3 09/13] fstests: add support for working with dm-log-writes target Amir Goldstein
2017-09-07 7:45 ` Eryu Guan
@ 2017-09-07 7:47 ` Eryu Guan
1 sibling, 0 replies; 35+ messages in thread
From: Eryu Guan @ 2017-09-07 7:47 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Josef Bacik, fstests, linux-fsdevel
On Tue, Sep 05, 2017 at 10:11:16PM +0300, Amir Goldstein wrote:
> Cherry-picked the relevant common bits from commit 70d41e17164b
> in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
> Quoting from Josef's commit message:
>
> This patch adds the supporting code for using the dm-log-writes
> target. The dmlogwrites code is similar to the dmflakey code, it just
> gives us functions to build and tear down a dm-log-writes target. We
> add a new LOGWRITES_DEV variable to take in the device we will use as
> the log and add checks for that.
>
> [Amir:]
> - Removed unneeded _test_falloc_support
> - Moved _require_log_writes to dmlogwrites
> - Document _require_log_writes
> - Address review comments by Eryu Guan
>
> Cc: Josef Bacik <jbacik@fb.com>
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> ---
...
> +# _replay_log <mark>
One additional nit, this comment can be updated to reflect new name :)
> +#
> +# This replays the log contained on $LOGWRITES_DEV onto $SCRATCH_DEV upto the
> +# mark passed in.
> +_log_writes_replay_log()
Thanks,
Eryu
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-09-05 19:11 ` [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes Amir Goldstein
@ 2017-09-07 7:50 ` Eryu Guan
2017-09-07 8:50 ` Amir Goldstein
2017-11-27 9:56 ` Amir Goldstein
1 sibling, 1 reply; 35+ messages in thread
From: Eryu Guan @ 2017-09-07 7:50 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Josef Bacik, fstests, linux-fsdevel
On Tue, Sep 05, 2017 at 10:11:17PM +0300, Amir Goldstein wrote:
> Cherry-picked the test from commit 70d41e17164b
> in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
> Quoting from Josef's commit message:
>
> The test just runs some ops and exits, then finds all of the good buffers
> in the directory we provided and:
> - replays up to the mark given
> - mounts the file system and compares the md5sum
> - unmounts and fsck's to check for metadata integrity
>
> dm-log-writes will pretend to do discard and the replay-log tool will
> replay it properly depending on the underlying device, either by writing
> 0's or actually calling the discard ioctl, so I've enabled discard in the
> test for maximum fun.
>
> [Amir:]
> - Removed unneeded _test_falloc_support dynamic FSX_OPTS
> - Fold repetitions into for loops
> - Added place holders for using constant random seeds
> - Add pre umount checkpint
> - Add test to new 'replay' group
> - Address review comments by Eryu Guan
>
> Cc: Josef Bacik <jbacik@fb.com>
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> ---
> tests/generic/500 | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/generic/500.out | 2 +
> tests/generic/group | 1 +
> 3 files changed, 138 insertions(+)
> create mode 100755 tests/generic/500
> create mode 100644 tests/generic/500.out
>
> diff --git a/tests/generic/500 b/tests/generic/500
> new file mode 100755
> index 0000000..82f7a93
> --- /dev/null
> +++ b/tests/generic/500
> @@ -0,0 +1,135 @@
> +#! /bin/bash
> +# FS QA Test No. 500
> +#
> +# Run fsx with log writes to verify power fail safeness.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2015 Facebook. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +status=1 # failure is the default!
> +
> +_cleanup()
> +{
> + _log_writes_cleanup
> +}
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmlogwrites
> +
> +# real QA test starts here
> +_supported_fs generic
> +_supported_os Linux
> +_require_test
> +_require_scratch_nocheck
> +_require_log_writes
> +
> +rm -f $seqres.full
> +
> +check_files()
> +{
> + local name=$1
> +
> + # Now look for our files
> + for i in $(find $SANITY_DIR -type f | grep $name | grep mark)
> + do
> + local filename=$(basename $i)
> + local mark="${filename##*.}"
> + echo "checking $filename" >> $seqres.full
> + _log_writes_replay_log $filename
> + _scratch_mount
> + local expected_md5=$(_md5_checksum $i)
> + local md5=$(_md5_checksum $SCRATCH_MNT/$name)
> + [ "${md5}" != "${expected_md5}" ] && _fail "$filename md5sum mismatched"
> + _scratch_unmount
> + _check_scratch_fs
> + done
> +}
> +
> +SANITY_DIR=$TEST_DIR/fsxtests
> +rm -rf $SANITY_DIR
> +mkdir $SANITY_DIR
> +
> +# Create the log
> +_log_writes_init
> +
> +_log_writes_mkfs >> $seqres.full 2>&1
> +
> +# Log writes emulates discard support, turn it on for maximum crying.
> +_log_writes_mount -o discard
> +
> +NUM_FILES=4
> +NUM_OPS=200
> +FSX_OPTS="-N $NUM_OPS -d -P $SANITY_DIR -i $LOGWRITES_DMDEV"
> +# Set random seeds for fsx runs (0 for timestamp + pid)
> +seeds=(0 0 0 0)
Hmm, is this array necessary? Why not pass 0 to -S directly in the
commandline? And generic/502 follows the same pattern.
> +# Run fsx for a while
> +for j in `seq 0 $((NUM_FILES-1))`
> +do
> + run_check $here/ltp/fsx $FSX_OPTS -S ${seeds[$j]} -j $j $SCRATCH_MNT/testfile$j &
> +done
> +wait
Thanks,
Eryu
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 11/13] fstests: regression test for ext4 crash consistency bug
2017-09-05 19:11 ` [PATCH v3 11/13] fstests: regression test for ext4 crash consistency bug Amir Goldstein
@ 2017-09-07 7:52 ` Eryu Guan
0 siblings, 0 replies; 35+ messages in thread
From: Eryu Guan @ 2017-09-07 7:52 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Josef Bacik, fstests, linux-fsdevel
On Tue, Sep 05, 2017 at 10:11:18PM +0300, Amir Goldstein wrote:
> This test is motivated by a bug found in ext4 during random crash
> consistency tests.
>
> This test uses device mapper flakey target to demonstrate the bug
> found using device mapper log-writes target.
>
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> ---
> tests/generic/501 | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/generic/501.out | 2 ++
> tests/generic/group | 1 +
> 3 files changed, 83 insertions(+)
> create mode 100755 tests/generic/501
> create mode 100644 tests/generic/501.out
>
> diff --git a/tests/generic/501 b/tests/generic/501
> new file mode 100755
> index 0000000..ccb513d
> --- /dev/null
> +++ b/tests/generic/501
> @@ -0,0 +1,80 @@
> +#! /bin/bash
> +# FS QA Test No. 501
> +#
> +# This test is motivated by a bug found in ext4 during random crash
> +# consistency tests.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (C) 2017 CTERA Networks. All Rights Reserved.
> +# Author: Amir Goldstein <amir73il@gmail.com>
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +
> +_cleanup()
> +{
> + _cleanup_flakey
> + cd /
> + rm -f $tmp.*
> +}
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmflakey
> +
> +# real QA test starts here
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch
> +_require_dm_target flakey
> +_require_metadata_journaling $SCRATCH_DEV
This should be moved after _scratch_mkfs, otherwise it's checking
scratch device from previous test.
> +
> +rm -f $seqres.full
> +
> +_scratch_mkfs >> $seqres.full 2>&1
Thanks,
Eryu
> +
> +_init_flakey
> +_mount_flakey
> +
> +fsxops=$tmp.fsxops
> +cat <<EOF > $fsxops
> +write 0x137dd 0xdc69 0x0
> +fallocate 0xb531 0xb5ad 0x21446
> +collapse_range 0x1c000 0x4000 0x21446
> +write 0x3e5ec 0x1a14 0x21446
> +zero_range 0x20fac 0x6d9c 0x40000 keep_size
> +mapwrite 0x216ad 0x274f 0x40000
> +EOF
> +run_check $here/ltp/fsx -d --replay-ops $fsxops $SCRATCH_MNT/testfile
> +
> +_flakey_drop_and_remount
> +_unmount_flakey
> +_cleanup_flakey
> +_check_scratch_fs
> +
> +echo "Silence is golden"
> +
> +status=0
> +exit
> diff --git a/tests/generic/501.out b/tests/generic/501.out
> new file mode 100644
> index 0000000..00133b6
> --- /dev/null
> +++ b/tests/generic/501.out
> @@ -0,0 +1,2 @@
> +QA output created by 501
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index c857153..1e1b524 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -458,3 +458,4 @@
> 453 auto quick dir
> 454 auto quick attr
> 500 auto log replay
> +501 auto quick metadata
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 13/13] fstests: regression test for xfs leftover CoW extent error
2017-09-05 19:11 ` [PATCH v3 13/13] fstests: regression test for xfs leftover CoW extent error Amir Goldstein
@ 2017-09-07 7:55 ` Eryu Guan
2017-09-07 9:34 ` Amir Goldstein
0 siblings, 1 reply; 35+ messages in thread
From: Eryu Guan @ 2017-09-07 7:55 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Josef Bacik, fstests, linux-fsdevel
On Tue, Sep 05, 2017 at 10:11:20PM +0300, Amir Goldstein wrote:
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Better to have some more descriptions or background information on this
test, either in commit log or in test description.
> ---
> tests/generic/503 | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/generic/503.out | 2 ++
> tests/generic/group | 1 +
> 3 files changed, 72 insertions(+)
> create mode 100755 tests/generic/503
> create mode 100644 tests/generic/503.out
>
> diff --git a/tests/generic/503 b/tests/generic/503
> new file mode 100755
> index 0000000..ebda756
> --- /dev/null
> +++ b/tests/generic/503
> @@ -0,0 +1,69 @@
> +#! /bin/bash
> +# FS QA Test No. 503
> +#
> +# Regression test for xfs leftover CoW extents error
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (C) 2017 CTERA Networks. All Rights Reserved.
> +# Author: Amir Goldstein <amir73il@gmail.com>
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch_reflink
> +_require_cp_reflink
> +
> +rm -f $seqres.full
> +
> +_scratch_mkfs >/dev/null 2>&1
> +_scratch_mount
> +
> +$XFS_IO_PROG -f -c "pwrite 0 0x40000" \
> + $SCRATCH_MNT/foo > /dev/null 2>&1
> +
> +cp --reflink $SCRATCH_MNT/foo $SCRATCH_MNT/bar
_cp_reflink
> +
> +$XFS_IO_PROG -f -c "fzero -k 0x169f 0x387c" \
> + -c "fcollapse 0x29000 0xd000" \
> + -c "finsert 0 0x8000" \
> + -c "truncate 0x8000" \
> + $SCRATCH_MNT/foo > /dev/null 2>&1
_require_xfs_io_command these commands first, fzero, fcollapse, finsert.
Above two nits apply to generic/502 too.
Thanks,
Eryu
> +
> +echo "Silence is golden"
> +status=0
> +exit
> diff --git a/tests/generic/503.out b/tests/generic/503.out
> new file mode 100644
> index 0000000..6c1400a
> --- /dev/null
> +++ b/tests/generic/503.out
> @@ -0,0 +1,2 @@
> +QA output created by 503
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index 4324775..7a9cd78 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -460,3 +460,4 @@
> 500 auto log replay
> 501 auto quick metadata
> 502 auto log replay clone
> +503 auto quick clone
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-09-07 7:50 ` Eryu Guan
@ 2017-09-07 8:50 ` Amir Goldstein
2017-09-07 8:55 ` Eryu Guan
0 siblings, 1 reply; 35+ messages in thread
From: Amir Goldstein @ 2017-09-07 8:50 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
On Thu, Sep 7, 2017 at 10:50 AM, Eryu Guan <eguan@redhat.com> wrote:
> On Tue, Sep 05, 2017 at 10:11:17PM +0300, Amir Goldstein wrote:
>> Cherry-picked the test from commit 70d41e17164b
>> in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
>> Quoting from Josef's commit message:
>>
>> The test just runs some ops and exits, then finds all of the good buffers
>> in the directory we provided and:
>> - replays up to the mark given
>> - mounts the file system and compares the md5sum
>> - unmounts and fsck's to check for metadata integrity
>>
>> dm-log-writes will pretend to do discard and the replay-log tool will
>> replay it properly depending on the underlying device, either by writing
>> 0's or actually calling the discard ioctl, so I've enabled discard in the
>> test for maximum fun.
>>
>> [Amir:]
>> - Removed unneeded _test_falloc_support dynamic FSX_OPTS
>> - Fold repetitions into for loops
>> - Added place holders for using constant random seeds
>> - Add pre umount checkpint
>> - Add test to new 'replay' group
>> - Address review comments by Eryu Guan
>>
>> Cc: Josef Bacik <jbacik@fb.com>
>> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
>> ---
>> tests/generic/500 | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++
>> tests/generic/500.out | 2 +
>> tests/generic/group | 1 +
>> 3 files changed, 138 insertions(+)
>> create mode 100755 tests/generic/500
>> create mode 100644 tests/generic/500.out
>>
>> diff --git a/tests/generic/500 b/tests/generic/500
>> new file mode 100755
>> index 0000000..82f7a93
>> --- /dev/null
>> +++ b/tests/generic/500
>> @@ -0,0 +1,135 @@
>> +#! /bin/bash
>> +# FS QA Test No. 500
>> +#
>> +# Run fsx with log writes to verify power fail safeness.
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (c) 2015 Facebook. All Rights Reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +status=1 # failure is the default!
>> +
>> +_cleanup()
>> +{
>> + _log_writes_cleanup
>> +}
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +. ./common/dmlogwrites
>> +
>> +# real QA test starts here
>> +_supported_fs generic
>> +_supported_os Linux
>> +_require_test
>> +_require_scratch_nocheck
>> +_require_log_writes
>> +
>> +rm -f $seqres.full
>> +
>> +check_files()
>> +{
>> + local name=$1
>> +
>> + # Now look for our files
>> + for i in $(find $SANITY_DIR -type f | grep $name | grep mark)
>> + do
>> + local filename=$(basename $i)
>> + local mark="${filename##*.}"
>> + echo "checking $filename" >> $seqres.full
>> + _log_writes_replay_log $filename
>> + _scratch_mount
>> + local expected_md5=$(_md5_checksum $i)
>> + local md5=$(_md5_checksum $SCRATCH_MNT/$name)
>> + [ "${md5}" != "${expected_md5}" ] && _fail "$filename md5sum mismatched"
>> + _scratch_unmount
>> + _check_scratch_fs
>> + done
>> +}
>> +
>> +SANITY_DIR=$TEST_DIR/fsxtests
>> +rm -rf $SANITY_DIR
>> +mkdir $SANITY_DIR
>> +
>> +# Create the log
>> +_log_writes_init
>> +
>> +_log_writes_mkfs >> $seqres.full 2>&1
>> +
>> +# Log writes emulates discard support, turn it on for maximum crying.
>> +_log_writes_mount -o discard
>> +
>> +NUM_FILES=4
>> +NUM_OPS=200
>> +FSX_OPTS="-N $NUM_OPS -d -P $SANITY_DIR -i $LOGWRITES_DMDEV"
>> +# Set random seeds for fsx runs (0 for timestamp + pid)
>> +seeds=(0 0 0 0)
>
> Hmm, is this array necessary? Why not pass 0 to -S directly in the
> commandline? And generic/502 follows the same pattern.
>
The reason for the array is trying to follow your guidelines for running
the test:
- Normally test runs with random seed (for which the array is not needed)
and every fsx process prints out the seed to the full log
- When test detects a failure, you want to retry the test with same seed
numbers, but those are different seed numbers for every fsx process
- So the array is here to make setting those presets possible
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-09-07 8:50 ` Amir Goldstein
@ 2017-09-07 8:55 ` Eryu Guan
2017-09-07 10:10 ` Amir Goldstein
0 siblings, 1 reply; 35+ messages in thread
From: Eryu Guan @ 2017-09-07 8:55 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Josef Bacik, fstests, linux-fsdevel
On Thu, Sep 07, 2017 at 11:50:25AM +0300, Amir Goldstein wrote:
> On Thu, Sep 7, 2017 at 10:50 AM, Eryu Guan <eguan@redhat.com> wrote:
> > On Tue, Sep 05, 2017 at 10:11:17PM +0300, Amir Goldstein wrote:
> >> Cherry-picked the test from commit 70d41e17164b
> >> in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
> >> Quoting from Josef's commit message:
> >>
> >> The test just runs some ops and exits, then finds all of the good buffers
> >> in the directory we provided and:
> >> - replays up to the mark given
> >> - mounts the file system and compares the md5sum
> >> - unmounts and fsck's to check for metadata integrity
> >>
> >> dm-log-writes will pretend to do discard and the replay-log tool will
> >> replay it properly depending on the underlying device, either by writing
> >> 0's or actually calling the discard ioctl, so I've enabled discard in the
> >> test for maximum fun.
> >>
> >> [Amir:]
> >> - Removed unneeded _test_falloc_support dynamic FSX_OPTS
> >> - Fold repetitions into for loops
> >> - Added place holders for using constant random seeds
> >> - Add pre umount checkpint
> >> - Add test to new 'replay' group
> >> - Address review comments by Eryu Guan
> >>
> >> Cc: Josef Bacik <jbacik@fb.com>
> >> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> >> ---
> >> tests/generic/500 | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++
> >> tests/generic/500.out | 2 +
> >> tests/generic/group | 1 +
> >> 3 files changed, 138 insertions(+)
> >> create mode 100755 tests/generic/500
> >> create mode 100644 tests/generic/500.out
> >>
> >> diff --git a/tests/generic/500 b/tests/generic/500
> >> new file mode 100755
> >> index 0000000..82f7a93
> >> --- /dev/null
> >> +++ b/tests/generic/500
> >> @@ -0,0 +1,135 @@
> >> +#! /bin/bash
> >> +# FS QA Test No. 500
> >> +#
> >> +# Run fsx with log writes to verify power fail safeness.
> >> +#
> >> +#-----------------------------------------------------------------------
> >> +# Copyright (c) 2015 Facebook. All Rights Reserved.
> >> +#
> >> +# This program is free software; you can redistribute it and/or
> >> +# modify it under the terms of the GNU General Public License as
> >> +# published by the Free Software Foundation.
> >> +#
> >> +# This program is distributed in the hope that it would be useful,
> >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> >> +# GNU General Public License for more details.
> >> +#
> >> +# You should have received a copy of the GNU General Public License
> >> +# along with this program; if not, write the Free Software Foundation,
> >> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> >> +#-----------------------------------------------------------------------
> >> +#
> >> +
> >> +seq=`basename $0`
> >> +seqres=$RESULT_DIR/$seq
> >> +echo "QA output created by $seq"
> >> +
> >> +here=`pwd`
> >> +status=1 # failure is the default!
> >> +
> >> +_cleanup()
> >> +{
> >> + _log_writes_cleanup
> >> +}
> >> +trap "_cleanup; exit \$status" 0 1 2 3 15
> >> +
> >> +# get standard environment, filters and checks
> >> +. ./common/rc
> >> +. ./common/filter
> >> +. ./common/dmlogwrites
> >> +
> >> +# real QA test starts here
> >> +_supported_fs generic
> >> +_supported_os Linux
> >> +_require_test
> >> +_require_scratch_nocheck
> >> +_require_log_writes
> >> +
> >> +rm -f $seqres.full
> >> +
> >> +check_files()
> >> +{
> >> + local name=$1
> >> +
> >> + # Now look for our files
> >> + for i in $(find $SANITY_DIR -type f | grep $name | grep mark)
> >> + do
> >> + local filename=$(basename $i)
> >> + local mark="${filename##*.}"
> >> + echo "checking $filename" >> $seqres.full
> >> + _log_writes_replay_log $filename
> >> + _scratch_mount
> >> + local expected_md5=$(_md5_checksum $i)
> >> + local md5=$(_md5_checksum $SCRATCH_MNT/$name)
> >> + [ "${md5}" != "${expected_md5}" ] && _fail "$filename md5sum mismatched"
> >> + _scratch_unmount
> >> + _check_scratch_fs
> >> + done
> >> +}
> >> +
> >> +SANITY_DIR=$TEST_DIR/fsxtests
> >> +rm -rf $SANITY_DIR
> >> +mkdir $SANITY_DIR
> >> +
> >> +# Create the log
> >> +_log_writes_init
> >> +
> >> +_log_writes_mkfs >> $seqres.full 2>&1
> >> +
> >> +# Log writes emulates discard support, turn it on for maximum crying.
> >> +_log_writes_mount -o discard
> >> +
> >> +NUM_FILES=4
> >> +NUM_OPS=200
> >> +FSX_OPTS="-N $NUM_OPS -d -P $SANITY_DIR -i $LOGWRITES_DMDEV"
> >> +# Set random seeds for fsx runs (0 for timestamp + pid)
> >> +seeds=(0 0 0 0)
> >
> > Hmm, is this array necessary? Why not pass 0 to -S directly in the
> > commandline? And generic/502 follows the same pattern.
> >
>
> The reason for the array is trying to follow your guidelines for running
> the test:
> - Normally test runs with random seed (for which the array is not needed)
> and every fsx process prints out the seed to the full log
> - When test detects a failure, you want to retry the test with same seed
> numbers, but those are different seed numbers for every fsx process
> - So the array is here to make setting those presets possible
Yeah, an array makes debug easier. Then I think a comment about the
debug usage of the array would be sufficient. Thanks!
Eryu
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 13/13] fstests: regression test for xfs leftover CoW extent error
2017-09-07 7:55 ` Eryu Guan
@ 2017-09-07 9:34 ` Amir Goldstein
0 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-07 9:34 UTC (permalink / raw)
To: Eryu Guan
Cc: Josef Bacik, fstests, linux-fsdevel, Theodore Tso, Darrick J. Wong
On Thu, Sep 7, 2017 at 10:55 AM, Eryu Guan <eguan@redhat.com> wrote:
> On Tue, Sep 05, 2017 at 10:11:20PM +0300, Amir Goldstein wrote:
>> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
>
> Better to have some more descriptions or background information on this
> test, either in commit log or in test description.
Sure, I had this info in the original patch posting, but forgot to
update my branch.
Regarding test 501,503, they are both "dumb" test in the sense
that they don't explain why the specific sequence of events cause
the problem or why the specific set of random offset/length values
are used.
Darrick has already analyzed the report and posted a fix for 503,
which I tested.
Ted has not responded to the bug report yet.
I suggested in original posting that perhaps, more insight about the
cause should be added to the test before merging and possibly
a reference to the fix commit.
You may want to wait on the maintainers input about this bug report
before merging, or not. After all the "dumb" test does fail and it is
possible to "undumb" it after there is a fix upstream.
>
>> ---
>> tests/generic/503 | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++
>> tests/generic/503.out | 2 ++
>> tests/generic/group | 1 +
>> 3 files changed, 72 insertions(+)
>> create mode 100755 tests/generic/503
>> create mode 100644 tests/generic/503.out
>>
>> diff --git a/tests/generic/503 b/tests/generic/503
>> new file mode 100755
>> index 0000000..ebda756
>> --- /dev/null
>> +++ b/tests/generic/503
>> @@ -0,0 +1,69 @@
>> +#! /bin/bash
>> +# FS QA Test No. 503
>> +#
>> +# Regression test for xfs leftover CoW extents error
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (C) 2017 CTERA Networks. All Rights Reserved.
>> +# Author: Amir Goldstein <amir73il@gmail.com>
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1 # failure is the default!
>> +
>> +_cleanup()
>> +{
>> + cd /
>> + rm -f $tmp.*
>> +}
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +. ./common/reflink
>> +
>> +# real QA test starts here
>> +_supported_fs generic
>> +_supported_os Linux
>> +_require_scratch_reflink
>> +_require_cp_reflink
>> +
>> +rm -f $seqres.full
>> +
>> +_scratch_mkfs >/dev/null 2>&1
>> +_scratch_mount
>> +
>> +$XFS_IO_PROG -f -c "pwrite 0 0x40000" \
>> + $SCRATCH_MNT/foo > /dev/null 2>&1
>> +
>> +cp --reflink $SCRATCH_MNT/foo $SCRATCH_MNT/bar
>
> _cp_reflink
>
>> +
>> +$XFS_IO_PROG -f -c "fzero -k 0x169f 0x387c" \
>> + -c "fcollapse 0x29000 0xd000" \
>> + -c "finsert 0 0x8000" \
>> + -c "truncate 0x8000" \
>> + $SCRATCH_MNT/foo > /dev/null 2>&1
>
> _require_xfs_io_command these commands first, fzero, fcollapse, finsert.
>
> Above two nits apply to generic/502 too.
>
FYI, tests 501 uses fsx to replay events, so it does not require that fs
actually supports fallocate/fcollapse, because fsx automatically disables
fallocate ops that are not supported by fs.
Amir.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-09-07 8:55 ` Eryu Guan
@ 2017-09-07 10:10 ` Amir Goldstein
0 siblings, 0 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-09-07 10:10 UTC (permalink / raw)
To: Eryu Guan; +Cc: Josef Bacik, fstests, linux-fsdevel
On Thu, Sep 7, 2017 at 11:55 AM, Eryu Guan <eguan@redhat.com> wrote:
>> >> +NUM_FILES=4
>> >> +NUM_OPS=200
>> >> +FSX_OPTS="-N $NUM_OPS -d -P $SANITY_DIR -i $LOGWRITES_DMDEV"
>> >> +# Set random seeds for fsx runs (0 for timestamp + pid)
>> >> +seeds=(0 0 0 0)
>> >
>> > Hmm, is this array necessary? Why not pass 0 to -S directly in the
>> > commandline? And generic/502 follows the same pattern.
>> >
>>
>> The reason for the array is trying to follow your guidelines for running
>> the test:
>> - Normally test runs with random seed (for which the array is not needed)
>> and every fsx process prints out the seed to the full log
>> - When test detects a failure, you want to retry the test with same seed
>> numbers, but those are different seed numbers for every fsx process
>> - So the array is here to make setting those presets possible
>
> Yeah, an array makes debug easier. Then I think a comment about the
> debug usage of the array would be sufficient. Thanks!
>
FYI, I removed the array from generic/502, because it was never useful
during debugging. I guess the randomness stems mostly from timing
of forking the fsx processes in that test.
I reduced NUM_OPS in cloned files test to 10 in v4, because after a few
holes and writes the clone is no longer a clone anyway.
I already found a seed value to reproduce a failure of generic/502 on xfs
failures are still random, but on spinning disk, I get them every 2 test runs.
Amir.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-09-05 19:11 ` [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes Amir Goldstein
2017-09-07 7:50 ` Eryu Guan
@ 2017-11-27 9:56 ` Amir Goldstein
2017-11-27 14:23 ` Ashlie Martinez
2017-11-27 15:04 ` Josef Bacik
1 sibling, 2 replies; 35+ messages in thread
From: Amir Goldstein @ 2017-11-27 9:56 UTC (permalink / raw)
To: Josef Bacik; +Cc: fstests, linux-fsdevel, Eryu Guan
On Tue, Sep 5, 2017 at 10:11 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> Cherry-picked the test from commit 70d41e17164b
> in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
> Quoting from Josef's commit message:
>
> The test just runs some ops and exits, then finds all of the good buffers
> in the directory we provided and:
> - replays up to the mark given
> - mounts the file system and compares the md5sum
> - unmounts and fsck's to check for metadata integrity
>
> dm-log-writes will pretend to do discard and the replay-log tool will
> replay it properly depending on the underlying device, either by writing
> 0's or actually calling the discard ioctl, so I've enabled discard in the
> test for maximum fun.
>
> [Amir:]
> - Removed unneeded _test_falloc_support dynamic FSX_OPTS
> - Fold repetitions into for loops
> - Added place holders for using constant random seeds
> - Add pre umount checkpint
> - Add test to new 'replay' group
> - Address review comments by Eryu Guan
>
> Cc: Josef Bacik <jbacik@fb.com>
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
Josef,
As you know, this test is now merged to xfstest as generic/455.
I have been running the test for a while on xfs and it occasionally
reports inconsistencies which I try to investigate.
In some of the reports, it appears that dm-log-writes may be exhibiting
a reliability issue (see below).
> ---
> tests/generic/500 | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/generic/500.out | 2 +
> tests/generic/group | 1 +
> 3 files changed, 138 insertions(+)
> create mode 100755 tests/generic/500
> create mode 100644 tests/generic/500.out
>
> diff --git a/tests/generic/500 b/tests/generic/500
> new file mode 100755
> index 0000000..82f7a93
> --- /dev/null
> +++ b/tests/generic/500
> @@ -0,0 +1,135 @@
> +#! /bin/bash
> +# FS QA Test No. 500
> +#
> +# Run fsx with log writes to verify power fail safeness.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2015 Facebook. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +status=1 # failure is the default!
> +
> +_cleanup()
> +{
> + _log_writes_cleanup
> +}
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmlogwrites
> +
> +# real QA test starts here
> +_supported_fs generic
> +_supported_os Linux
> +_require_test
> +_require_scratch_nocheck
> +_require_log_writes
> +
> +rm -f $seqres.full
> +
> +check_files()
> +{
> + local name=$1
> +
> + # Now look for our files
> + for i in $(find $SANITY_DIR -type f | grep $name | grep mark)
> + do
> + local filename=$(basename $i)
> + local mark="${filename##*.}"
> + echo "checking $filename" >> $seqres.full
> + _log_writes_replay_log $filename
> + _scratch_mount
> + local expected_md5=$(_md5_checksum $i)
> + local md5=$(_md5_checksum $SCRATCH_MNT/$name)
> + [ "${md5}" != "${expected_md5}" ] && _fail "$filename md5sum mismatched"
One time, the test reported md5 mismatch on a file, but when I replayed
the log to the same mark I found that md5 of the file is correct compared to the
'snapshot' file in test partition.
> + _scratch_unmount
> + _check_scratch_fs
> + done
> +}
> +
> +SANITY_DIR=$TEST_DIR/fsxtests
> +rm -rf $SANITY_DIR
> +mkdir $SANITY_DIR
> +
> +# Create the log
> +_log_writes_init
> +
> +_log_writes_mkfs >> $seqres.full 2>&1
> +
> +# Log writes emulates discard support, turn it on for maximum crying.
> +_log_writes_mount -o discard
> +
> +NUM_FILES=4
> +NUM_OPS=200
> +FSX_OPTS="-N $NUM_OPS -d -P $SANITY_DIR -i $LOGWRITES_DMDEV"
> +# Set random seeds for fsx runs (0 for timestamp + pid)
> +seeds=(0 0 0 0)
> +# Run fsx for a while
> +for j in `seq 0 $((NUM_FILES-1))`
> +do
> + run_check $here/ltp/fsx $FSX_OPTS -S ${seeds[$j]} -j $j $SCRATCH_MNT/testfile$j &
> +done
> +wait
> +
> +test_md5=()
> +for j in `seq 0 $((NUM_FILES-1))`
> +do
> + test_md5[$j]=$(_md5_checksum $SCRATCH_MNT/testfile$j)
> +done
> +
> +# Unmount the scratch dir and tear down the log writes target
> +_log_writes_mark last
> +_log_writes_unmount
> +_log_writes_mark end
> +_log_writes_remove
> +_check_scratch_fs
> +
Another time xfs_repair complained about dirty log right after
_log_writes_remove
and I wasn't able to seek to the "end" mark. Log entries were valid a
few entries
after the "last" mark.
This leads me to believe that perhaps dm-log-writes doesn't flush all
its pending
log io before remove? Anyway, I added a "sync $LOGWRITES_DEV" call inside
_log_writes_remove. Not sure if that helps or if that is required
before removing the
target?
I will report if I see that the problem persists.
Thought? suggestions?
Thanks,
Amir.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-11-27 9:56 ` Amir Goldstein
@ 2017-11-27 14:23 ` Ashlie Martinez
2017-11-27 15:07 ` Josef Bacik
2017-11-27 15:04 ` Josef Bacik
1 sibling, 1 reply; 35+ messages in thread
From: Ashlie Martinez @ 2017-11-27 14:23 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Josef Bacik, fstests, linux-fsdevel, Eryu Guan
Amir,
Just to throw in what I believe I've found about dm-log-writes (though
Josef would know more about this as it's just what I've concluded
after looking at the code): dm-log-writes logs at IO completion,
meaning disks that ignore flush operations could exhibit bugs where
there are none, and dm-log-writes does synchronous marks though this
may not line up with the stream of IO operations (due to buffering)
unless you just did a sync. The CrashMonkey team wanted to make
something similar to the "mark" operation dm-log-writes has and we
concluded that the only place we could guarantee a mark would line up
with the stream of IO operations the user had performed was right
after a call to sync as that would force cached updates to disk. If
you call mark without a sync, then you could insert a mark after a
write(2) returns, but before the delayed allocation for that write(2)
actually allocates blocks and changes the extent tree on disk.
On Mon, Nov 27, 2017 at 3:56 AM, Amir Goldstein <amir73il@gmail.com> wrote:
> On Tue, Sep 5, 2017 at 10:11 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>> Cherry-picked the test from commit 70d41e17164b
>> in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
>> Quoting from Josef's commit message:
>>
>> The test just runs some ops and exits, then finds all of the good buffers
>> in the directory we provided and:
>> - replays up to the mark given
>> - mounts the file system and compares the md5sum
>> - unmounts and fsck's to check for metadata integrity
>>
>> dm-log-writes will pretend to do discard and the replay-log tool will
>> replay it properly depending on the underlying device, either by writing
>> 0's or actually calling the discard ioctl, so I've enabled discard in the
>> test for maximum fun.
>>
>> [Amir:]
>> - Removed unneeded _test_falloc_support dynamic FSX_OPTS
>> - Fold repetitions into for loops
>> - Added place holders for using constant random seeds
>> - Add pre umount checkpint
>> - Add test to new 'replay' group
>> - Address review comments by Eryu Guan
>>
>> Cc: Josef Bacik <jbacik@fb.com>
>> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
>
>
> Josef,
>
> As you know, this test is now merged to xfstest as generic/455.
> I have been running the test for a while on xfs and it occasionally
> reports inconsistencies which I try to investigate.
>
> In some of the reports, it appears that dm-log-writes may be exhibiting
> a reliability issue (see below).
>
>> ---
>> tests/generic/500 | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++
>> tests/generic/500.out | 2 +
>> tests/generic/group | 1 +
>> 3 files changed, 138 insertions(+)
>> create mode 100755 tests/generic/500
>> create mode 100644 tests/generic/500.out
>>
>> diff --git a/tests/generic/500 b/tests/generic/500
>> new file mode 100755
>> index 0000000..82f7a93
>> --- /dev/null
>> +++ b/tests/generic/500
>> @@ -0,0 +1,135 @@
>> +#! /bin/bash
>> +# FS QA Test No. 500
>> +#
>> +# Run fsx with log writes to verify power fail safeness.
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (c) 2015 Facebook. All Rights Reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +status=1 # failure is the default!
>> +
>> +_cleanup()
>> +{
>> + _log_writes_cleanup
>> +}
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +. ./common/dmlogwrites
>> +
>> +# real QA test starts here
>> +_supported_fs generic
>> +_supported_os Linux
>> +_require_test
>> +_require_scratch_nocheck
>> +_require_log_writes
>> +
>> +rm -f $seqres.full
>> +
>> +check_files()
>> +{
>> + local name=$1
>> +
>> + # Now look for our files
>> + for i in $(find $SANITY_DIR -type f | grep $name | grep mark)
>> + do
>> + local filename=$(basename $i)
>> + local mark="${filename##*.}"
>> + echo "checking $filename" >> $seqres.full
>> + _log_writes_replay_log $filename
>> + _scratch_mount
>> + local expected_md5=$(_md5_checksum $i)
>> + local md5=$(_md5_checksum $SCRATCH_MNT/$name)
>> + [ "${md5}" != "${expected_md5}" ] && _fail "$filename md5sum mismatched"
>
> One time, the test reported md5 mismatch on a file, but when I replayed
> the log to the same mark I found that md5 of the file is correct compared to the
> 'snapshot' file in test partition.
>
>> + _scratch_unmount
>> + _check_scratch_fs
>> + done
>> +}
>> +
>> +SANITY_DIR=$TEST_DIR/fsxtests
>> +rm -rf $SANITY_DIR
>> +mkdir $SANITY_DIR
>> +
>> +# Create the log
>> +_log_writes_init
>> +
>> +_log_writes_mkfs >> $seqres.full 2>&1
>> +
>> +# Log writes emulates discard support, turn it on for maximum crying.
>> +_log_writes_mount -o discard
>> +
>> +NUM_FILES=4
>> +NUM_OPS=200
>> +FSX_OPTS="-N $NUM_OPS -d -P $SANITY_DIR -i $LOGWRITES_DMDEV"
>> +# Set random seeds for fsx runs (0 for timestamp + pid)
>> +seeds=(0 0 0 0)
>> +# Run fsx for a while
>> +for j in `seq 0 $((NUM_FILES-1))`
>> +do
>> + run_check $here/ltp/fsx $FSX_OPTS -S ${seeds[$j]} -j $j $SCRATCH_MNT/testfile$j &
>> +done
>> +wait
>> +
>> +test_md5=()
>> +for j in `seq 0 $((NUM_FILES-1))`
>> +do
>> + test_md5[$j]=$(_md5_checksum $SCRATCH_MNT/testfile$j)
>> +done
>> +
>> +# Unmount the scratch dir and tear down the log writes target
>> +_log_writes_mark last
>> +_log_writes_unmount
>> +_log_writes_mark end
>> +_log_writes_remove
>> +_check_scratch_fs
>> +
>
> Another time xfs_repair complained about dirty log right after
> _log_writes_remove
> and I wasn't able to seek to the "end" mark. Log entries were valid a
> few entries
> after the "last" mark.
>
> This leads me to believe that perhaps dm-log-writes doesn't flush all
> its pending
> log io before remove? Anyway, I added a "sync $LOGWRITES_DEV" call inside
> _log_writes_remove. Not sure if that helps or if that is required
> before removing the
> target?
>
> I will report if I see that the problem persists.
> Thought? suggestions?
>
> Thanks,
> Amir.
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-11-27 9:56 ` Amir Goldstein
2017-11-27 14:23 ` Ashlie Martinez
@ 2017-11-27 15:04 ` Josef Bacik
2017-11-28 16:48 ` Amir Goldstein
1 sibling, 1 reply; 35+ messages in thread
From: Josef Bacik @ 2017-11-27 15:04 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Josef Bacik, fstests, linux-fsdevel, Eryu Guan
On Mon, Nov 27, 2017 at 11:56:58AM +0200, Amir Goldstein wrote:
> On Tue, Sep 5, 2017 at 10:11 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> > Cherry-picked the test from commit 70d41e17164b
> > in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
> > Quoting from Josef's commit message:
> >
> > The test just runs some ops and exits, then finds all of the good buffers
> > in the directory we provided and:
> > - replays up to the mark given
> > - mounts the file system and compares the md5sum
> > - unmounts and fsck's to check for metadata integrity
> >
> > dm-log-writes will pretend to do discard and the replay-log tool will
> > replay it properly depending on the underlying device, either by writing
> > 0's or actually calling the discard ioctl, so I've enabled discard in the
> > test for maximum fun.
> >
> > [Amir:]
> > - Removed unneeded _test_falloc_support dynamic FSX_OPTS
> > - Fold repetitions into for loops
> > - Added place holders for using constant random seeds
> > - Add pre umount checkpint
> > - Add test to new 'replay' group
> > - Address review comments by Eryu Guan
> >
> > Cc: Josef Bacik <jbacik@fb.com>
> > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
>
>
> Josef,
>
> As you know, this test is now merged to xfstest as generic/455.
> I have been running the test for a while on xfs and it occasionally
> reports inconsistencies which I try to investigate.
>
> In some of the reports, it appears that dm-log-writes may be exhibiting
> a reliability issue (see below).
>
It's not a reliability issue, its a caching issue. dm-log-writes is just
issuing bio's to the log device, and our destructor waits for all pending io
blocks to complete before exiting, so unless I've missed how dm is destroying
devices everything should be on disk.
However since we replay in userspace we are going through the blockdevice's
pagecache, so we could have stale pages left in place which is screwing us up.
Will you try this patch and see if it fixes the problem? Thanks,
Josef
diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
index 8b80a9ce9ea9..1c502930af5e 100644
--- a/drivers/md/dm-log-writes.c
+++ b/drivers/md/dm-log-writes.c
@@ -545,6 +545,8 @@ static void log_writes_dtr(struct dm_target *ti)
!atomic_read(&lc->pending_blocks));
kthread_stop(lc->log_kthread);
+ invalidate_bdev(lc->logdev->bdev);
+ invalidate_bdev(lc->dev->bdev);
WARN_ON(!list_empty(&lc->logging_blocks));
WARN_ON(!list_empty(&lc->unflushed_blocks));
dm_put_device(ti, lc->dev);
^ permalink raw reply related [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-11-27 14:23 ` Ashlie Martinez
@ 2017-11-27 15:07 ` Josef Bacik
0 siblings, 0 replies; 35+ messages in thread
From: Josef Bacik @ 2017-11-27 15:07 UTC (permalink / raw)
To: Ashlie Martinez
Cc: Amir Goldstein, Josef Bacik, fstests, linux-fsdevel, Eryu Guan
On Mon, Nov 27, 2017 at 08:23:49AM -0600, Ashlie Martinez wrote:
> Amir,
>
> Just to throw in what I believe I've found about dm-log-writes (though
> Josef would know more about this as it's just what I've concluded
> after looking at the code): dm-log-writes logs at IO completion,
> meaning disks that ignore flush operations could exhibit bugs where
> there are none, and dm-log-writes does synchronous marks though this
> may not line up with the stream of IO operations (due to buffering)
> unless you just did a sync. The CrashMonkey team wanted to make
> something similar to the "mark" operation dm-log-writes has and we
> concluded that the only place we could guarantee a mark would line up
> with the stream of IO operations the user had performed was right
> after a call to sync as that would force cached updates to disk. If
> you call mark without a sync, then you could insert a mark after a
> write(2) returns, but before the delayed allocation for that write(2)
> actually allocates blocks and changes the extent tree on disk.
>
Yeah dm-log-writes gladly hands you the loaded gun, it's your job to not shoot
yourself with it. This fsx test is specifically set up to only do the mark
_after_ a successful call of fsync(), which means that the mark will be after
where we expect our data to be consistent. Thanks,
Josef
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-11-27 15:04 ` Josef Bacik
@ 2017-11-28 16:48 ` Amir Goldstein
2017-11-28 17:21 ` Josef Bacik
0 siblings, 1 reply; 35+ messages in thread
From: Amir Goldstein @ 2017-11-28 16:48 UTC (permalink / raw)
To: Josef Bacik; +Cc: Josef Bacik, fstests, linux-fsdevel, Eryu Guan
[-- Attachment #1: Type: text/plain, Size: 3120 bytes --]
On Mon, Nov 27, 2017 at 5:04 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> On Mon, Nov 27, 2017 at 11:56:58AM +0200, Amir Goldstein wrote:
>> On Tue, Sep 5, 2017 at 10:11 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>> > Cherry-picked the test from commit 70d41e17164b
>> > in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
>> > Quoting from Josef's commit message:
>> >
>> > The test just runs some ops and exits, then finds all of the good buffers
>> > in the directory we provided and:
>> > - replays up to the mark given
>> > - mounts the file system and compares the md5sum
>> > - unmounts and fsck's to check for metadata integrity
>> >
>> > dm-log-writes will pretend to do discard and the replay-log tool will
>> > replay it properly depending on the underlying device, either by writing
>> > 0's or actually calling the discard ioctl, so I've enabled discard in the
>> > test for maximum fun.
>> >
>> > [Amir:]
>> > - Removed unneeded _test_falloc_support dynamic FSX_OPTS
>> > - Fold repetitions into for loops
>> > - Added place holders for using constant random seeds
>> > - Add pre umount checkpint
>> > - Add test to new 'replay' group
>> > - Address review comments by Eryu Guan
>> >
>> > Cc: Josef Bacik <jbacik@fb.com>
>> > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
>>
>>
>> Josef,
>>
>> As you know, this test is now merged to xfstest as generic/455.
>> I have been running the test for a while on xfs and it occasionally
>> reports inconsistencies which I try to investigate.
>>
>> In some of the reports, it appears that dm-log-writes may be exhibiting
>> a reliability issue (see below).
>>
>
> It's not a reliability issue, its a caching issue. dm-log-writes is just
> issuing bio's to the log device, and our destructor waits for all pending io
> blocks to complete before exiting, so unless I've missed how dm is destroying
> devices everything should be on disk.
>
> However since we replay in userspace we are going through the blockdevice's
> pagecache, so we could have stale pages left in place which is screwing us up.
> Will you try this patch and see if it fixes the problem? Thanks,
>
> Josef
>
>
> diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
> index 8b80a9ce9ea9..1c502930af5e 100644
> --- a/drivers/md/dm-log-writes.c
> +++ b/drivers/md/dm-log-writes.c
> @@ -545,6 +545,8 @@ static void log_writes_dtr(struct dm_target *ti)
> !atomic_read(&lc->pending_blocks));
> kthread_stop(lc->log_kthread);
>
> + invalidate_bdev(lc->logdev->bdev);
> + invalidate_bdev(lc->dev->bdev);
> WARN_ON(!list_empty(&lc->logging_blocks));
> WARN_ON(!list_empty(&lc->unflushed_blocks));
> dm_put_device(ti, lc->dev);
Josef,
With your patch OR with my xfstest patch that adds "sync" I did not yet see
another problem of garbage fs after _log_writes_remove.
I did however, encounter this error (failure to verify read data during fsx)
from scratch/log-writes device (see attached full log).
I will keep running the test to collect more information.
Thanks,
Amir.
[-- Attachment #2: 455.full.replay.failed --]
[-- Type: application/octet-stream, Size: 54172 bytes --]
# /old/home/amir/src/xfstests-dev/ltp/fsx -N 200 -d -P /mnt/test/fsxtests -i /dev/mapper/logwrites-test -S 0 -j 2 /mnt/scratch/testfile2
# /old/home/amir/src/xfstests-dev/ltp/fsx -N 200 -d -P /mnt/test/fsxtests -i /dev/mapper/logwrites-test -S 0 -j 0 /mnt/scratch/testfile0
# /old/home/amir/src/xfstests-dev/ltp/fsx -N 200 -d -P /mnt/test/fsxtests -i /dev/mapper/logwrites-test -S 0 -j 1 /mnt/scratch/testfile1
# /old/home/amir/src/xfstests-dev/ltp/fsx -N 200 -d -P /mnt/test/fsxtests -i /dev/mapper/logwrites-test -S 0 -j 3 /mnt/scratch/testfile3
Seed set to 27204
Seed set to 27205
Seed set to 27206
Seed set to 27207
2: 1 mapwrite 0x36a28 thru 0x3ffff (0x95d8 bytes)
0: skipping zero length punch hole
0: skipping insert range behind EOF
0: skipping collapse range behind EOF
0: skipping insert range behind EOF
0: truncating to largest ever: 0x38843
0: 5 trunc from 0x0 to 0x38843
0: 6 falloc from 0x3965f to 0x40000 (0x69a1 bytes)
0: 7 trunc from 0x38843 to 0x43b5
0: 8 zero from 0x401d to 0x43b5, (0x398 bytes)
0: 9 mapread 0xa0e thru 0x43b4 (0x39a7 bytes)
0: 10 fsync
3: 1 mapwrite 0x61d8 thru 0x100d7 (0x9f00 bytes)
1: skipping zero size read
1: 2 write 0x39450 thru 0x3ffff (0x6bb0 bytes)
1: 3 write 0x3a8e9 thru 0x3ffff (0x5717 bytes)
1: 4 collapse from 0x3000 to 0xf000, (0xc000 bytes)
1: zero_range to largest ever: 0x2946c
1: 5 zero from 0x2706a to 0x2946c, (0x2402 bytes)
1: 6 insert from 0x9000 to 0x14000, (0xb000 bytes)
1: 7 fsync
2: 2 fsync
0: Dumped fsync buffer to testfile0.mark0
0: 11 insert from 0x1000 to 0x7000, (0x6000 bytes)
0: 12 zero from 0x7d3d to 0xa3b5, (0x2678 bytes)
0: 13 punch from 0xf2b to 0x5e94, (0x4f69 bytes)
0: 14 mapread 0x116a thru 0x74ab (0x6342 bytes)
0: 15 insert from 0x0 to 0xc000, (0xc000 bytes)
0: 16 read 0x9b23 thru 0xd3f7 (0x38d5 bytes)
0: 17 trunc from 0x163b5 to 0x1304
0: 18 trunc from 0x1304 to 0x2e6e1
0: 19 mapread 0x9046 thru 0xc8b2 (0x386d bytes)
0: 20 punch from 0x99f6 to 0x14cc6, (0xb2d0 bytes)
0: 21 read 0x19ed9 thru 0x21641 (0x7769 bytes)
0: 22 punch from 0x1d6dd to 0x20933, (0x3256 bytes)
0: truncating to largest ever: 0x3a896
0: 23 trunc from 0x2e6e1 to 0x3a896
0: 24 falloc from 0x14989 to 0x1aba6 (0x621d bytes)
3: 2 mapwrite 0xd137 thru 0x16550 (0x941a bytes)
0: 25 fsync
2: Dumped fsync buffer to testfile2.mark0
2: 3 mapwrite 0xdf72 thru 0x15824 (0x78b3 bytes)
1: Dumped fsync buffer to testfile1.mark0
1: 8 punch from 0x2d44f to 0x35f12, (0x8ac3 bytes)
1: 9 punch from 0x312dc to 0x3835f, (0x7083 bytes)
1: 10 insert from 0x17000 to 0x18000, (0x1000 bytes)
1: 11 write 0xc239 thru 0x18f6b (0xcd33 bytes)
1: 13 punch from 0xe026 to 0x1650a, (0x84e4 bytes)
0: Dumped fsync buffer to testfile0.mark1
0: 26 read 0x1caaa thru 0x22f8b (0x64e2 bytes)
0: 27 punch from 0x2a8c4 to 0x3602c, (0xb768 bytes)
0: 28 insert from 0x1b000 to 0x20000, (0x5000 bytes)
0: 29 write 0x3e5ed thru 0x3ffff (0x1a13 bytes)
0: 30 zero from 0x57b7 to 0x7691, (0x1eda bytes)
0: 31 collapse from 0xd000 to 0x19000, (0xc000 bytes)
0: 32 write 0x29efe thru 0x2a95e (0xa61 bytes)
0: 33 zero from 0x17654 to 0x1de45, (0x67f1 bytes)
0: 34 falloc from 0x14eba to 0x1b9a2 (0x6ae8 bytes)
0: 35 read 0x2aecf thru 0x33fff (0x9131 bytes)
2: 4 fsync
1: 14 read 0x38301 thru 0x3ffff (0x7cff bytes)
0: 36 write 0x1ec35 thru 0x2236e (0x373a bytes)
0: 37 collapse from 0x9000 to 0xe000, (0x5000 bytes)
3: 3 mapwrite 0x27e98 thru 0x31859 (0x99c2 bytes)
1: 15 fsync
0: 38 falloc from 0x2a913 to 0x35f7c (0xb669 bytes)
0: 39 write 0xdb39 thru 0x11e5c (0x4324 bytes)
0: 40 trunc from 0x35f7c to 0x26035
0: 41 falloc from 0x12b38 to 0x135c2 (0xa8a bytes)
0: 42 insert from 0x1a000 to 0x23000, (0x9000 bytes)
0: 43 falloc from 0x2e544 to 0x2fb0e (0x15ca bytes)
0: 44 insert from 0x5000 to 0x14000, (0xf000 bytes)
0: 45 trunc from 0x3e035 to 0x31bde
0: 46 punch from 0x10da to 0x3d1e, (0x2c44 bytes)
0: 47 fsync
2: Dumped fsync buffer to testfile2.mark1
2: fallocating to largest ever: 0x16a44
2: 5 falloc from 0x6e37 to 0x16a44 (0xfc0d bytes)
2: 6 read 0x2c946 thru 0x31e1b (0x54d6 bytes)
2: fallocating to largest ever: 0x26611
2: 8 falloc from 0x17dd8 to 0x26611 (0xe839 bytes)
2: 9 fsync
3: truncating to largest ever: 0x67d3
3: 4 trunc from 0x3185a to 0x67d3
3: 5 collapse from 0x4000 to 0x6000, (0x2000 bytes)
3: truncating to largest ever: 0x11f12
3: 7 trunc from 0x47d3 to 0x11f12
1: Dumped fsync buffer to testfile1.mark1
1: 16 trunc from 0x40000 to 0x16c11
0: Dumped fsync buffer to testfile0.mark2
0: 48 mapread 0x1b360 thru 0x2a281 (0xef22 bytes)
1: 17 read 0xc5ea thru 0x115b8 (0x4fcf bytes)
1: 18 fsync
3: 8 mapwrite 0x250b6 thru 0x31fde (0xcf29 bytes)
0: 49 zero from 0x25da0 to 0x2b3b3, (0x5613 bytes)
0: 50 read 0x24787 thru 0x251f0 (0xa6a bytes)
0: 51 read 0x308c8 thru 0x31bdd (0x1316 bytes)
0: 52 read 0x24111 thru 0x30448 (0xc338 bytes)
0: 53 write 0x149d thru 0x10fee (0xfb52 bytes)
0: 54 punch from 0x145ec to 0x2195a, (0xd36e bytes)
0: 55 mapwrite 0x2d496 thru 0x34075 (0x6be0 bytes)
2: Dumped fsync buffer to testfile2.mark2
2: 10 mapwrite 0x2f8d1 thru 0x38644 (0x8d74 bytes)
1: Dumped fsync buffer to testfile1.mark2
1: 19 mapread 0x15fe4 thru 0x16c10 (0xc2d bytes)
1: 20 mapread 0x20cf thru 0x11e38 (0xfd6a bytes)
1: 21 read 0x15993 thru 0x16c10 (0x127e bytes)
1: fallocating to largest ever: 0x40000
1: 22 falloc from 0x3261f to 0x40000 (0xd9e1 bytes)
3: 9 punch from 0x1b308 to 0x1fc63, (0x495b bytes)
3: 10 mapread 0x25033 thru 0x31310 (0xc2de bytes)
3: 11 mapread 0x3a97 thru 0xf2ad (0xb817 bytes)
3: 12 mapwrite 0x2e485 thru 0x2f8a8 (0x1424 bytes)
1: 23 falloc from 0x1724f to 0x2308e (0xbe3f bytes)
1: 24 mapread 0x3cc1c thru 0x3ffff (0x33e4 bytes)
1: 25 mapread 0x2d017 thru 0x3322a (0x6214 bytes)
1: 26 write 0x2527a thru 0x27ad9 (0x2860 bytes)
1: 27 mapwrite 0x15261 thru 0x23d3b (0xeadb bytes)
2: 11 fsync
0: 56 trunc from 0x34076 to 0x48b9
0: 57 insert from 0x1000 to 0xc000, (0xb000 bytes)
2: Dumped fsync buffer to testfile2.mark3
2: 12 fsync
3: 13 read 0xa335 thru 0x143ad (0xa079 bytes)
3: 14 collapse from 0xf000 to 0x12000, (0x3000 bytes)
1: 28 write 0x13d2a thru 0x16142 (0x2419 bytes)
1: 29 write 0x2a5c6 thru 0x2a6a1 (0xdc bytes)
1: 30 falloc from 0x2ba42 to 0x309e9 (0x4fa7 bytes)
1: 31 falloc from 0x29ed to 0x5e09 (0x341c bytes)
1: 32 falloc from 0x3b571 to 0x40000 (0x4a8f bytes)
1: 33 mapwrite 0x247ef thru 0x25a85 (0x1297 bytes)
3: 15 zero from 0x2131b to 0x2dba5, (0xc88a bytes)
2: Dumped fsync buffer to testfile2.mark4
2: 13 collapse from 0x29000 to 0x30000, (0x7000 bytes)
2: 14 read 0xe2c6 thru 0x1a7ef (0xc52a bytes)
2: 16 mapread 0x31928 thru 0x38fff (0x76d8 bytes)
2: 17 fsync
0: 58 trunc from 0xf8b9 to 0x4c57
0: 59 insert from 0x4000 to 0x8000, (0x4000 bytes)
0: 60 punch from 0x2070 to 0x8c57, (0x6be7 bytes)
0: 61 trunc from 0x8c57 to 0x14634
0: 62 write 0x22482 thru 0x2aa48 (0x85c7 bytes)
0: 63 zero from 0x6e54 to 0x15ba3, (0xed4f bytes)
0: 64 read 0x1fb41 thru 0x2aa48 (0xaf08 bytes)
0: 65 falloc from 0x32b24 to 0x3e763 (0xbc3f bytes)
0: 66 punch from 0x5725 to 0xc767, (0x7042 bytes)
0: 67 collapse from 0x1e000 to 0x21000, (0x3000 bytes)
3: 16 write 0x2a2db thru 0x38018 (0xdd3e bytes)
3: 17 read 0x20d8d thru 0x23216 (0x248a bytes)
3: 18 collapse from 0x2b000 to 0x2f000, (0x4000 bytes)
1: 34 write 0x3a308 thru 0x3ffff (0x5cf8 bytes)
1: 35 fsync
0: 68 write 0x1a765 thru 0x24bb1 (0xa44d bytes)
3: truncating to largest ever: 0x33ffb
3: 19 trunc from 0x34019 to 0x33ffb
0: 69 read 0x1d4df thru 0x26de5 (0x9907 bytes)
3: 20 insert from 0x2f000 to 0x3b000, (0xc000 bytes)
2: Dumped fsync buffer to testfile2.mark5
2: 18 falloc from 0x29b29 to 0x2ca4b (0x2f22 bytes)
2: 19 collapse from 0x36000 to 0x38000, (0x2000 bytes)
2: 20 collapse from 0x30000 to 0x36000, (0x6000 bytes)
2: 21 write 0xf08f thru 0x1c6b7 (0xd629 bytes)
2: 22 falloc from 0x4829 to 0xed78 (0xa54f bytes)
2: 23 read 0x1998c thru 0x2313a (0x97af bytes)
2: 24 write 0xa37b thru 0x135f4 (0x927a bytes)
2: 25 fsync
0: 70 falloc from 0x2f922 to 0x37577 (0x7c55 bytes)
3: 21 falloc from 0x37bf6 to 0x40000 (0x840a bytes)
3: 22 fsync
0: 71 mapwrite 0x3f750 thru 0x3ffff (0x8b0 bytes)
1: Dumped fsync buffer to testfile1.mark3
1: 36 punch from 0xcf4a to 0x14614, (0x76ca bytes)
1: 37 punch from 0x95b4 to 0x15bd2, (0xc61e bytes)
3: Dumped fsync buffer to testfile3.mark0
3: 23 collapse from 0x18000 to 0x26000, (0xe000 bytes)
0: 73 write 0x3d39e thru 0x3ffff (0x2c62 bytes)
0: zero_range to largest ever: 0x40000
0: 74 zero from 0x384b0 to 0x40000, (0x7b50 bytes)
2: Dumped fsync buffer to testfile2.mark6
2: 26 collapse from 0x2000 to 0xe000, (0xc000 bytes)
3: 24 falloc from 0x10f63 to 0x1918d (0x822a bytes)
3: 25 zero from 0x2bdb2 to 0x2d4dd, (0x172b bytes)
2: 27 fsync
1: 38 write 0x25f74 thru 0x32d77 (0xce04 bytes)
1: 39 zero from 0x10522 to 0x17ac4, (0x75a2 bytes)
3: 26 falloc from 0x109d3 to 0x14fbb (0x45e8 bytes)
3: 27 read 0x26e7f thru 0x306ca (0x984c bytes)
0: 75 write 0x1991b thru 0x1c8b8 (0x2f9e bytes)
0: 76 fsync
1: 40 falloc from 0x22522 to 0x31c5b (0xf739 bytes)
1: 41 falloc from 0x1ccbb to 0x2bb93 (0xeed8 bytes)
1: 42 zero from 0x19dfb to 0x23331, (0x9536 bytes)
3: 28 zero from 0x312e3 to 0x31ffb, (0xd18 bytes)
2: Dumped fsync buffer to testfile2.mark7
2: 28 write 0x387a2 thru 0x3ca8a (0x42e9 bytes)
2: 29 punch from 0x1eeb6 to 0x27939, (0x8a83 bytes)
3: 29 read 0x1eea3 thru 0x2cced (0xde4b bytes)
2: 30 read 0x27427 thru 0x2fcde (0x88b8 bytes)
2: 31 falloc from 0x3dd0d to 0x40000 (0x22f3 bytes)
2: 32 collapse from 0x24000 to 0x31000, (0xd000 bytes)
3: 30 zero from 0x7edd to 0xa3f2, (0x2515 bytes)
3: 31 fsync
1: 43 mapread 0x150fd thru 0x1b48b (0x638f bytes)
1: 44 read 0x1f46b thru 0x25f1b (0x6ab1 bytes)
1: 46 write 0x2b634 thru 0x3283c (0x7209 bytes)
1: 47 mapread 0x3301c thru 0x3ffff (0xcfe4 bytes)
1: 48 read 0x3e60 thru 0x10b7b (0xcd1c bytes)
1: 49 fsync
2: 33 mapread 0x1ef6 thru 0xa062 (0x816d bytes)
0: Dumped fsync buffer to testfile0.mark3
0: 77 punch from 0x13870 to 0x14a84, (0x1214 bytes)
0: 78 mapwrite 0x22eb0 thru 0x2636d (0x34be bytes)
2: 34 read 0x2cb72 thru 0x2fa8a (0x2f19 bytes)
2: 35 fsync
3: Dumped fsync buffer to testfile3.mark1
3: 32 write 0x13ec2 thru 0x1463c (0x77b bytes)
3: 33 mapread 0x1834d thru 0x1a47e (0x2132 bytes)
3: 34 insert from 0x11000 to 0x1f000, (0xe000 bytes)
3: 35 read 0x25670 thru 0x2847f (0x2e10 bytes)
3: 36 falloc from 0x11a8e to 0x19c7a (0x81ec bytes)
3: 38 fsync
1: Dumped fsync buffer to testfile1.mark4
1: 50 falloc from 0x35f3e to 0x37e68 (0x1f2a bytes)
1: 51 trunc from 0x40000 to 0x3d0fc
1: 52 trunc from 0x3d0fc to 0x1120e
1: 53 write 0x2ba44 thru 0x32c9e (0x725b bytes)
1: 54 mapread 0x2d90a thru 0x32c9e (0x5395 bytes)
1: 55 fsync
0: 79 mapwrite 0x13b4 thru 0xfe2f (0xea7c bytes)
2: Dumped fsync buffer to testfile2.mark8
2: 36 insert from 0x1b000 to 0x24000, (0x9000 bytes)
2: 37 punch from 0x2509e to 0x28745, (0x36a7 bytes)
2: 38 read 0x21d7d thru 0x2f66f (0xd8f3 bytes)
2: 39 zero from 0x1e708 to 0x21853, (0x314b bytes)
2: 40 punch from 0x193a6 to 0x21924, (0x857e bytes)
2: 41 fsync
3: Dumped fsync buffer to testfile3.mark2
3: 39 falloc from 0x3aa6a to 0x40000 (0x5596 bytes)
3: 40 punch from 0x33ca3 to 0x3815e, (0x44bb bytes)
3: 41 zero from 0x291bb to 0x29ae5, (0x92a bytes)
3: 42 zero from 0x451b to 0xb7d2, (0x72b7 bytes)
3: 43 mapread 0x1a264 thru 0x1fdbb (0x5b58 bytes)
3: 46 write 0x19ea9 thru 0x26b2f (0xcc87 bytes)
3: 47 trunc from 0x3fffb to 0x29d95
3: 48 mapwrite 0x163d3 thru 0x25dbc (0xf9ea bytes)
1: Dumped fsync buffer to testfile1.mark5
1: 56 mapread 0x2e22b thru 0x2f74d (0x1523 bytes)
1: 57 collapse from 0xf000 to 0x18000, (0x9000 bytes)
0: 80 write 0x36efc thru 0x3ffff (0x9104 bytes)
0: 81 write 0xf4a9 thru 0x14069 (0x4bc1 bytes)
0: 82 mapread 0x33045 thru 0x3efd0 (0xbf8c bytes)
0: 83 trunc from 0x40000 to 0xacee
0: 84 mapwrite 0x59ba thru 0x67a2 (0xde9 bytes)
1: 58 write 0x2305 thru 0x2ed3 (0xbcf bytes)
1: 59 punch from 0x25d36 to 0x29c9f, (0x3f69 bytes)
2: Dumped fsync buffer to testfile2.mark9
2: 42 insert from 0x25000 to 0x26000, (0x1000 bytes)
1: 60 mapwrite 0x11ec2 thru 0x18c06 (0x6d45 bytes)
2: 43 read 0x7926 thru 0xf1dc (0x78b7 bytes)
2: 44 falloc from 0x2f331 to 0x39a6a (0xa739 bytes)
2: 45 falloc from 0x39af to 0xc200 (0x8851 bytes)
2: 46 falloc from 0x3e262 to 0x3efb3 (0xd51 bytes)
2: 47 insert from 0x37000 to 0x3d000, (0x6000 bytes)
0: 85 collapse from 0x8000 to 0xa000, (0x2000 bytes)
3: 49 mapread 0xa24b thru 0x1921a (0xefd0 bytes)
3: 50 read 0x1db9a thru 0x20aa7 (0x2f0e bytes)
3: 51 mapread 0x380b thru 0x6404 (0x2bfa bytes)
3: 52 fsync
2: 48 trunc from 0x3fa8b to 0x23b36
2: 49 insert from 0x1a000 to 0x1c000, (0x2000 bytes)
2: 50 punch from 0xb0e6 to 0xd01c, (0x1f36 bytes)
0: 86 trunc from 0x8cee to 0xc1d8
1: 61 read 0x8863 thru 0xaf92 (0x2730 bytes)
1: 62 write 0xbcb5 thru 0xcac3 (0xe0f bytes)
1: 63 read 0xd70a thru 0x1a911 (0xd208 bytes)
1: 64 trunc from 0x29c9f to 0x30cb2
0: 87 read 0x34ac thru 0x8db0 (0x5905 bytes)
0: 88 trunc from 0xc1d8 to 0x35330
0: 90 zero from 0x55c1 to 0x96e3, (0x4122 bytes)
2: 51 mapread 0xbcd4 thru 0x1905c (0xd389 bytes)
2: 52 write 0x37c75 thru 0x3f5aa (0x7936 bytes)
2: 53 punch from 0x1a1b2 to 0x1f0ec, (0x4f3a bytes)
2: 54 punch from 0x36d56 to 0x3f5ab, (0x8855 bytes)
0: 91 mapwrite 0x3da15 thru 0x3ffff (0x25eb bytes)
1: 65 falloc from 0x1955c to 0x23969 (0xa40d bytes)
1: 66 write 0x17e4d thru 0x1d70c (0x58c0 bytes)
1: 67 fsync
2: 55 fsync
3: Dumped fsync buffer to testfile3.mark3
3: 53 falloc from 0x31b2e to 0x3789d (0x5d6f bytes)
3: 54 mapwrite 0x3fb90 thru 0x3ffff (0x470 bytes)
0: 92 read 0x1fe6e thru 0x2787f (0x7a12 bytes)
0: 93 mapwrite 0x306ba thru 0x39b72 (0x94b9 bytes)
3: 55 mapread 0x1668e thru 0x25cd7 (0xf64a bytes)
3: 56 falloc from 0xe573 to 0x13bc8 (0x5655 bytes)
3: 57 zero from 0x9e85 to 0xaba3, (0xd1e bytes)
3: 58 write 0x380dd thru 0x3ed89 (0x6cad bytes)
3: 59 read 0x3ba0 thru 0x10289 (0xc6ea bytes)
3: 60 collapse from 0x30000 to 0x38000, (0x8000 bytes)
3: 61 read 0xd1bb thru 0xdcae (0xaf4 bytes)
3: 62 fsync
2: Dumped fsync buffer to testfile2.mark10
2: 56 collapse from 0x34000 to 0x3b000, (0x7000 bytes)
1: Dumped fsync buffer to testfile1.mark6
1: 68 falloc from 0x1f66f to 0x29b55 (0xa4e6 bytes)
1: 69 fsync
0: 96 collapse from 0x3b000 to 0x3f000, (0x4000 bytes)
0: 97 fsync
2: 57 fsync
3: Dumped fsync buffer to testfile3.mark4
3: 63 mapread 0x48b1 thru 0x120b9 (0xd809 bytes)
3: 64 zero from 0x2e34b to 0x323f6, (0x40ab bytes)
3: 65 zero from 0x19f31 to 0x29bf0, (0xfcbf bytes)
1: Dumped fsync buffer to testfile1.mark7
1: 70 trunc from 0x30cb2 to 0x3559f
1: 71 insert from 0x24000 to 0x2e000, (0xa000 bytes)
1: 72 punch from 0xed6a to 0x1966e, (0xa904 bytes)
1: 73 punch from 0x1e121 to 0x27798, (0x9677 bytes)
1: 74 collapse from 0xd000 to 0x10000, (0x3000 bytes)
3: 66 zero from 0x2dfa9 to 0x3600a, (0x8061 bytes)
2: Dumped fsync buffer to testfile2.mark11
2: 58 collapse from 0x33000 to 0x37000, (0x4000 bytes)
1: 75 falloc from 0x1a45f to 0x214f5 (0x7096 bytes)
1: 76 collapse from 0xc000 to 0x1a000, (0xe000 bytes)
1: 77 read 0x12fa3 thru 0x14d3e (0x1d9c bytes)
1: 78 trunc from 0x2e59f to 0x1cbd4
1: 79 punch from 0x10117 to 0x13d7e, (0x3c67 bytes)
1: 80 insert from 0x3000 to 0xe000, (0xb000 bytes)
1: 81 mapread 0x65c thru 0xed73 (0xe718 bytes)
1: 82 write 0x25150 thru 0x2b052 (0x5f03 bytes)
1: 83 trunc from 0x2b053 to 0x11771
2: 59 falloc from 0x20abb to 0x2235c (0x18a1 bytes)
2: 60 read 0x30837 thru 0x3104a (0x814 bytes)
2: 61 zero from 0x459f to 0x7d8c, (0x37ed bytes)
1: 84 fsync
0: Dumped fsync buffer to testfile0.mark4
0: 98 zero from 0xac77 to 0x16812, (0xbb9b bytes)
2: 62 read 0x113a5 thru 0x1b835 (0xa491 bytes)
3: 67 punch from 0x15031 to 0x218c9, (0xc898 bytes)
0: 99 mapread 0x33515 thru 0x38c2a (0x5716 bytes)
2: 63 falloc from 0x20450 to 0x2c740 (0xc2f0 bytes)
0: 100 collapse from 0x20000 to 0x21000, (0x1000 bytes)
2: 64 read 0x9b83 thru 0x1042e (0x68ac bytes)
2: 65 read 0x342f3 thru 0x345aa (0x2b8 bytes)
2: 66 falloc from 0x1b4fc to 0x1d9f9 (0x24fd bytes)
2: zero_range to largest ever: 0x27206
2: 67 zero from 0x262fa to 0x27206, (0xf0c bytes)
0: 101 mapread 0x130a6 thru 0x22b49 (0xfaa4 bytes)
2: 68 trunc from 0x345ab to 0x10990
0: 102 trunc from 0x3b000 to 0x367be
2: 69 zero from 0xe0b5 to 0x10990, (0x28db bytes)
3: 68 fsync
0: 103 write 0xe32e thru 0x1d796 (0xf469 bytes)
0: 104 collapse from 0x28000 to 0x36000, (0xe000 bytes)
2: 70 zero from 0x17d7 to 0x4d6f, (0x3598 bytes)
1: Dumped fsync buffer to testfile1.mark8
1: 85 collapse from 0x1000 to 0xe000, (0xd000 bytes)
1: 86 write 0x2f878 thru 0x33e68 (0x45f1 bytes)
1: 87 punch from 0x16d75 to 0x23b94, (0xce1f bytes)
1: 88 write 0x261e6 thru 0x2fec1 (0x9cdc bytes)
1: 89 mapread 0xf1f5 thru 0x103dc (0x11e8 bytes)
1: 90 punch from 0x211c5 to 0x2e408, (0xd243 bytes)
0: 105 read 0x2ba5 thru 0x3ab3 (0xf0f bytes)
0: 106 mapread 0x2077f thru 0x287bd (0x803f bytes)
0: 107 zero from 0xb2d5 to 0x1299b, (0x76c6 bytes)
2: truncating to largest ever: 0x3b369
2: 71 trunc from 0x10990 to 0x3b369
1: 91 collapse from 0x4000 to 0xb000, (0x7000 bytes)
2: 72 punch from 0x15943 to 0x221d1, (0xc88e bytes)
2: 73 fsync
1: 92 collapse from 0x2a000 to 0x2c000, (0x2000 bytes)
1: 93 punch from 0x178cb to 0x26f28, (0xf65d bytes)
1: 94 falloc from 0x23b5f to 0x26c51 (0x30f2 bytes)
1: 95 punch from 0x216c to 0x9ef4, (0x7d88 bytes)
1: 96 zero from 0x230cd to 0x2ae69, (0x7d9c bytes)
0: 108 mapwrite 0x255bf thru 0x31f17 (0xc959 bytes)
3: Dumped fsync buffer to testfile3.mark5
3: 69 falloc from 0x29b87 to 0x2a4e1 (0x95a bytes)
3: 70 mapwrite 0x20a10 thru 0x21d3d (0x132e bytes)
1: 97 mapread 0x20653 thru 0x25ad6 (0x5484 bytes)
1: 98 trunc from 0x2ae69 to 0x1b973
1: 99 falloc from 0x38a90 to 0x40000 (0x7570 bytes)
1: 100 collapse from 0x2c000 to 0x37000, (0xb000 bytes)
1: 101 punch from 0x1bfe to 0xe9a2, (0xcda4 bytes)
1: 102 zero from 0x102bc to 0x13efd, (0x3c41 bytes)
1: 103 mapwrite 0xe1b6 thru 0xe296 (0xe1 bytes)
2: Dumped fsync buffer to testfile2.mark12
2: 74 fsync
2: Dumped fsync buffer to testfile2.mark13
2: 75 punch from 0x2d9e5 to 0x3962d, (0xbc48 bytes)
2: 76 insert from 0x4000 to 0x8000, (0x4000 bytes)
2: fallocating to largest ever: 0x40000
2: 77 falloc from 0x3ba92 to 0x40000 (0x456e bytes)
2: 78 mapread 0x1a8d5 thru 0x2809b (0xd7c7 bytes)
0: 109 write 0x32173 thru 0x3621b (0x40a9 bytes)
0: 110 punch from 0x1a171 to 0x27ce8, (0xdb77 bytes)
2: 79 mapread 0x2c2a5 thru 0x39d9a (0xdaf6 bytes)
2: 80 trunc from 0x40000 to 0x21fd0
2: 81 fsync
3: 71 mapread 0xf37e thru 0x18664 (0x92e7 bytes)
3: 72 write 0x3323 thru 0xe65b (0xb339 bytes)
3: 73 punch from 0xa5ad to 0xf825, (0x5278 bytes)
1: 104 zero from 0x6db to 0x856, (0x17b bytes)
1: 105 trunc from 0x35000 to 0x3e4be
1: 106 read 0x30d5c thru 0x3a16a (0x940f bytes)
1: 107 read 0x9407 thru 0xc609 (0x3203 bytes)
1: 108 trunc from 0x3e4be to 0xa09
1: 109 read 0x27e thru 0xa08 (0x78b bytes)
1: 110 trunc from 0xa09 to 0x37d97
1: 111 punch from 0x20563 to 0x25ef3, (0x5990 bytes)
1: 112 read 0x2e2c1 thru 0x37d96 (0x9ad6 bytes)
1: 113 mapread 0x319c0 thru 0x37884 (0x5ec5 bytes)
1: 114 collapse from 0x9000 to 0x17000, (0xe000 bytes)
1: 115 falloc from 0x2c23e to 0x30ae6 (0x48a8 bytes)
1: 116 mapwrite 0x15efd thru 0x22c19 (0xcd1d bytes)
3: 74 mapread 0x22896 thru 0x2de0a (0xb575 bytes)
3: 75 collapse from 0x1f000 to 0x28000, (0x9000 bytes)
0: 111 punch from 0xcd7 to 0xc2b8, (0xb5e1 bytes)
0: 112 trunc from 0x3621c to 0x155d7
0: 113 fsync
3: 76 mapread 0x299ce thru 0x2efff (0x5632 bytes)
3: 77 read 0xc393 thru 0xd12c (0xd9a bytes)
3: 78 insert from 0x9000 to 0xe000, (0x5000 bytes)
2: Dumped fsync buffer to testfile2.mark14
2: 82 collapse from 0x13000 to 0x18000, (0x5000 bytes)
2: 83 insert from 0x1a000 to 0x23000, (0x9000 bytes)
3: 79 mapread 0x253b0 thru 0x25671 (0x2c2 bytes)
2: 84 mapwrite 0xcf0a thru 0x11c6d (0x4d64 bytes)
3: 80 insert from 0x16000 to 0x22000, (0xc000 bytes)
0: Dumped fsync buffer to testfile0.mark5
0: 114 fsync
3: 82 mapread 0xa5de thru 0xcad6 (0x24f9 bytes)
3: 83 mapread 0x2ded2 thru 0x3a956 (0xca85 bytes)
3: 84 punch from 0xf217 to 0x1680c, (0x75f5 bytes)
1: 117 mapread 0x23e2c thru 0x26b27 (0x2cfc bytes)
1: 118 collapse from 0xb000 to 0x12000, (0x7000 bytes)
1: 119 punch from 0x16e3c to 0x1cabb, (0x5c7f bytes)
1: 120 mapwrite 0x5bba thru 0xf612 (0x9a59 bytes)
0: Dumped fsync buffer to testfile0.mark6
0: 115 trunc from 0x155d7 to 0x8102
0: 116 read 0x4346 thru 0x8101 (0x3dbc bytes)
0: 117 write 0x5e90 thru 0xc9eb (0x6b5c bytes)
0: 118 punch from 0x55ad to 0xc9ec, (0x743f bytes)
2: 85 mapwrite 0x30ce3 thru 0x386b9 (0x79d7 bytes)
3: 85 mapwrite 0x36da9 thru 0x3e20f (0x7467 bytes)
1: 121 insert from 0x13000 to 0x21000, (0xe000 bytes)
0: 119 write 0xf2ea thru 0x1376e (0x4485 bytes)
0: 120 mapwrite 0x707a thru 0x1320d (0xc194 bytes)
2: 86 falloc from 0x20eed to 0x30374 (0xf487 bytes)
2: 87 mapread 0x2709e thru 0x33c63 (0xcbc6 bytes)
2: 88 zero from 0x1190c to 0x16f23, (0x5617 bytes)
1: 122 mapread 0x29961 thru 0x2cadd (0x317d bytes)
1: 123 zero from 0x2d864 to 0x30d97, (0x3533 bytes)
1: 124 insert from 0xf000 to 0x18000, (0x9000 bytes)
2: 89 mapwrite 0x17ec5 thru 0x21a54 (0x9b90 bytes)
3: 86 mapread 0xaafc thru 0x1245f (0x7964 bytes)
3: 87 punch from 0x16a6 to 0x3786, (0x20e0 bytes)
1: 125 write 0x2ffda thru 0x3821d (0x8244 bytes)
1: 126 insert from 0x9000 to 0xf000, (0x6000 bytes)
3: 88 falloc from 0xf555 to 0x1b150 (0xbbfb bytes)
3: fallocating to largest ever: 0x40000
3: 89 falloc from 0x3a0b3 to 0x40000 (0x5f4d bytes)
3: 90 mapread 0xd820 thru 0xe474 (0xc55 bytes)
3: 91 collapse from 0x25000 to 0x33000, (0xe000 bytes)
1: 127 fsync
3: 92 punch from 0x15810 to 0x1edf6, (0x95e6 bytes)
3: 93 punch from 0x1723d to 0x1caca, (0x588d bytes)
3: 94 trunc from 0x32000 to 0x12e85
3: 95 fsync
0: 121 mapread 0xd7b6 thru 0x1376e (0x5fb9 bytes)
0: 122 zero from 0xfa4a to 0x1376f, (0x3d25 bytes)
0: 123 zero from 0x110a5 to 0x1376f, (0x26ca bytes)
1: Dumped fsync buffer to testfile1.mark9
1: 128 mapwrite 0x6afc thru 0x10a21 (0x9f26 bytes)
2: 90 trunc from 0x386ba to 0x2fffe
2: 91 punch from 0x912f to 0xd870, (0x4741 bytes)
2: 92 collapse from 0x1f000 to 0x28000, (0x9000 bytes)
2: 93 read 0x1ab22 thru 0x24e46 (0xa325 bytes)
2: 94 trunc from 0x26ffe to 0x2bdfc
2: 95 punch from 0x23c15 to 0x2bdfc, (0x81e7 bytes)
2: 96 mapwrite 0x3b2f7 thru 0x3cc8b (0x1995 bytes)
0: 124 trunc from 0x1376f to 0x2eaed
0: 125 write 0x1f802 thru 0x21b7f (0x237e bytes)
0: 127 fsync
3: Dumped fsync buffer to testfile3.mark6
3: 96 write 0x3e757 thru 0x3ffff (0x18a9 bytes)
3: 97 zero from 0x4b1 to 0x48ad, (0x43fc bytes)
3: 99 collapse from 0xd000 to 0x12000, (0x5000 bytes)
2: 97 fsync
3: 100 read 0x36303 thru 0x3afff (0x4cfd bytes)
1: 129 write 0x38c85 thru 0x3e190 (0x550c bytes)
1: 130 falloc from 0x393bb to 0x40000 (0x6c45 bytes)
1: 132 mapread 0x8e06 thru 0xdd04 (0x4eff bytes)
1: 133 zero from 0x19c7c to 0x26259, (0xc5dd bytes)
1: 134 mapwrite 0x12d11 thru 0x19acf (0x6dbf bytes)
3: 101 mapread 0x8a0b thru 0x18509 (0xfaff bytes)
3: 102 mapwrite 0x1ac3a thru 0x24971 (0x9d38 bytes)
0: Dumped fsync buffer to testfile0.mark7
0: 128 zero from 0x1959c to 0x27a78, (0xe4dc bytes)
0: 129 punch from 0x2d8c4 to 0x2eaed, (0x1229 bytes)
0: 130 falloc from 0x3baac to 0x40000 (0x4554 bytes)
0: 131 write 0x31597 thru 0x3e519 (0xcf83 bytes)
0: 132 mapread 0x2447a thru 0x2fe9b (0xba22 bytes)
0: 133 write 0x1d195 thru 0x252fb (0x8167 bytes)
0: 134 read 0x3918a thru 0x3a490 (0x1307 bytes)
0: 136 write 0x2cdd2 thru 0x394a4 (0xc6d3 bytes)
0: 138 mapwrite 0x3a61d thru 0x3ec39 (0x461d bytes)
2: Dumped fsync buffer to testfile2.mark15
2: 98 write 0x2e16d thru 0x2fa53 (0x18e7 bytes)
2: 99 zero from 0x4358 to 0x60e9, (0x1d91 bytes)
2: 100 punch from 0x3b4b3 to 0x3cc8c, (0x17d9 bytes)
2: 101 insert from 0x1e000 to 0x21000, (0x3000 bytes)
0: 140 falloc from 0xe13a to 0x1b3f0 (0xd2b6 bytes)
0: 141 mapread 0x1a02e thru 0x2460e (0xa5e1 bytes)
0: 142 fsync
3: 103 insert from 0x20000 to 0x25000, (0x5000 bytes)
3: 104 fsync
1: 135 mapread 0x14641 thru 0x16dd9 (0x2799 bytes)
1: 136 trunc from 0x40000 to 0x15ef6
1: 137 zero from 0x10248 to 0x15ef6, (0x5cae bytes)
2: 102 fsync
1: 138 fsync
2: Dumped fsync buffer to testfile2.mark16
2: 104 read 0x3f2cd thru 0x3fc8b (0x9bf bytes)
3: Dumped fsync buffer to testfile3.mark7
3: 106 punch from 0x38187 to 0x40000, (0x7e79 bytes)
3: 107 fsync
2: 105 collapse from 0x1000 to 0xb000, (0xa000 bytes)
0: Dumped fsync buffer to testfile0.mark8
0: 143 zero from 0x1506b to 0x18e22, (0x3db7 bytes)
0: 144 mapwrite 0x31d1b thru 0x3c912 (0xabf8 bytes)
2: 106 fsync
1: Dumped fsync buffer to testfile1.mark10
1: 139 mapread 0x138d0 thru 0x15ef5 (0x2626 bytes)
1: 140 mapread 0x8018 thru 0x15ee8 (0xded1 bytes)
1: 141 punch from 0x11c4f to 0x15ef6, (0x42a7 bytes)
1: 142 trunc from 0x15ef6 to 0x1268d
1: 143 mapwrite 0x257f thru 0x2e87 (0x909 bytes)
2: Dumped fsync buffer to testfile2.mark17
2: 107 punch from 0x1bae6 to 0x2b1c6, (0xf6e0 bytes)
2: 108 insert from 0x2b000 to 0x32000, (0x7000 bytes)
0: 145 collapse from 0x38000 to 0x3f000, (0x7000 bytes)
0: 146 mapwrite 0x3cdd4 thru 0x3ffff (0x322c bytes)
3: Dumped fsync buffer to testfile3.mark8
3: 108 collapse from 0x39000 to 0x3a000, (0x1000 bytes)
3: 109 punch from 0x1260 to 0xb853, (0xa5f3 bytes)
2: 109 mapread 0x23379 thru 0x2e785 (0xb40d bytes)
2: 110 punch from 0x37722 to 0x3918a, (0x1a68 bytes)
2: 111 zero from 0xeb5e to 0x1da4b, (0xeeed bytes)
3: 110 write 0x35197 thru 0x3ffff (0xae69 bytes)
3: 112 zero from 0x252ee to 0x282dc, (0x2fee bytes)
1: 144 zero from 0x544e to 0x8c87, (0x3839 bytes)
2: 112 fsync
3: 113 zero from 0x2b74b to 0x2ee94, (0x3749 bytes)
3: 114 trunc from 0x40000 to 0x10d00
3: 115 zero from 0x14e6 to 0x591f, (0x4439 bytes)
3: 116 write 0x1687e thru 0x1daa4 (0x7227 bytes)
3: 117 mapread 0x1c459 thru 0x1daa4 (0x164c bytes)
3: 118 insert from 0x10000 to 0x15000, (0x5000 bytes)
0: 148 read 0x2699b thru 0x308b6 (0x9f1c bytes)
0: 149 collapse from 0x27000 to 0x34000, (0xd000 bytes)
1: 145 zero from 0x2465 to 0x3c39, (0x17d4 bytes)
3: 119 trunc from 0x22aa5 to 0x1f4de
0: 150 falloc from 0x233ac to 0x26159 (0x2dad bytes)
0: 151 fsync
1: 146 read 0xe25a thru 0x1268c (0x4433 bytes)
1: 147 insert from 0xe000 to 0x14000, (0x6000 bytes)
3: 120 fsync
1: 148 mapread 0x11083 thru 0x1868c (0x760a bytes)
1: 149 fsync
2: Dumped fsync buffer to testfile2.mark18
2: 113 write 0x33ae thru 0xab14 (0x7767 bytes)
2: 114 insert from 0x12000 to 0x15000, (0x3000 bytes)
2: 115 zero from 0x33543 to 0x3ee68, (0xb925 bytes)
0: Dumped fsync buffer to testfile0.mark9
0: 152 trunc from 0x33000 to 0x14cf0
0: 153 zero from 0x1294b to 0x14cf0, (0x23a5 bytes)
0: 154 write 0x3cbb6 thru 0x3dd5a (0x11a5 bytes)
0: 155 insert from 0x30000 to 0x32000, (0x2000 bytes)
1: Dumped fsync buffer to testfile1.mark11
1: 150 falloc from 0x36da0 to 0x3fd7f (0x8fdf bytes)
1: 151 falloc from 0x3a101 to 0x3ac46 (0xb45 bytes)
1: 152 falloc from 0x23839 to 0x2a137 (0x68fe bytes)
1: 153 read 0xb6e4 thru 0x19b8c (0xe4a9 bytes)
1: 154 trunc from 0x3ac46 to 0x18170
1: 155 falloc from 0x2e8f6 to 0x3d123 (0xe82d bytes)
2: 116 falloc from 0x3f009 to 0x40000 (0xff7 bytes)
3: Dumped fsync buffer to testfile3.mark9
3: 121 insert from 0xb000 to 0x16000, (0xb000 bytes)
3: 122 fsync
2: 117 mapread 0x1a1e0 thru 0x27eeb (0xdd0c bytes)
0: 156 read 0x15829 thru 0x1f12f (0x9907 bytes)
1: 156 read 0x25702 thru 0x2d411 (0x7d10 bytes)
1: 157 read 0xabc1 thru 0x16ac9 (0xbf09 bytes)
0: 157 write 0x26d87 thru 0x328a8 (0xbb22 bytes)
1: 158 fsync
2: 119 falloc from 0x186a0 to 0x2212c (0x9a8c bytes)
0: 158 read 0x1794a thru 0x1f7f1 (0x7ea8 bytes)
0: 159 falloc from 0x7d89 to 0x9e60 (0x20d7 bytes)
0: 160 trunc from 0x3fd5b to 0x1b75f
2: 120 read 0x9d45 thru 0x13677 (0x9933 bytes)
0: 161 read 0x12d3 thru 0x7675 (0x63a3 bytes)
0: 162 falloc from 0x16ed to 0x550b (0x3e1e bytes)
0: 163 mapwrite 0x2f852 thru 0x376cc (0x7e7b bytes)
2: 121 trunc from 0x40000 to 0x4eb0
2: 122 zero from 0x41b4 to 0x4eb0, (0xcfc bytes)
3: Dumped fsync buffer to testfile3.mark10
3: 123 trunc from 0x2a4de to 0x2e75c
2: 123 falloc from 0x2c77d to 0x30bfe (0x4481 bytes)
3: 124 punch from 0x1640d to 0x1906a, (0x2c5d bytes)
3: 125 write 0x64ff thru 0x6d09 (0x80b bytes)
3: 126 fsync
0: 164 falloc from 0x982e to 0xab8d (0x135f bytes)
0: 165 insert from 0x1b000 to 0x23000, (0x8000 bytes)
2: 124 fsync
1: Dumped fsync buffer to testfile1.mark12
1: 159 write 0x126a8 thru 0x1b0ae (0x8a07 bytes)
1: 160 read 0x8469 thru 0xab02 (0x269a bytes)
1: 161 insert from 0x38000 to 0x3a000, (0x2000 bytes)
1: 162 read 0x1baa5 thru 0x1df99 (0x24f5 bytes)
1: 163 zero from 0xfa5f to 0x181e0, (0x8781 bytes)
0: 166 read 0x3532d thru 0x3b5f4 (0x62c8 bytes)
0: 167 zero from 0x3947 to 0x4ea8, (0x1561 bytes)
0: 168 trunc from 0x3f6cd to 0x293d2
0: 169 fsync
1: 164 fsync
3: Dumped fsync buffer to testfile3.mark11
3: 127 mapwrite 0x9e40 thru 0x174eb (0xd6ac bytes)
2: Dumped fsync buffer to testfile2.mark19
2: 125 mapwrite 0x22ce thru 0xbd05 (0x9a38 bytes)
0: Dumped fsync buffer to testfile0.mark10
0: 170 falloc from 0x7adb to 0x112ce (0x97f3 bytes)
0: 171 mapread 0x9145 thru 0xfadd (0x6999 bytes)
0: 172 punch from 0xa859 to 0xf490, (0x4c37 bytes)
0: 173 zero from 0x19286 to 0x19bfb, (0x975 bytes)
0: 174 collapse from 0x1f000 to 0x28000, (0x9000 bytes)
0: 175 punch from 0x154de to 0x1e114, (0x8c36 bytes)
0: 176 insert from 0x11000 to 0x1d000, (0xc000 bytes)
0: 177 fsync
1: Dumped fsync buffer to testfile1.mark13
1: 165 mapwrite 0x1970 thru 0x9ffc (0x868d bytes)
3: 128 mapread 0x2847c thru 0x2e75b (0x62e0 bytes)
3: 129 zero from 0xc359 to 0x19f32, (0xdbd9 bytes)
3: 130 read 0x66ce thru 0xcc68 (0x659b bytes)
3: 131 trunc from 0x2e75c to 0xec66
3: 132 punch from 0x2900 to 0xec66, (0xc366 bytes)
2: 126 read 0xf0f0 thru 0x17deb (0x8cfc bytes)
2: 127 punch from 0x2b886 to 0x30bfe, (0x5378 bytes)
2: 128 write 0x1991b thru 0x1cbdb (0x32c1 bytes)
2: 129 mapread 0x20c66 thru 0x2ec2e (0xdfc9 bytes)
2: 130 read 0x14de4 thru 0x24437 (0xf654 bytes)
2: 131 mapread 0xb573 thru 0xfbaf (0x463d bytes)
3: 133 collapse from 0x4000 to 0xe000, (0xa000 bytes)
2: 132 zero from 0x15cd7 to 0x1f921, (0x9c4a bytes)
3: 134 mapread 0x2f8b thru 0x4c65 (0x1cdb bytes)
3: 135 write 0x1963e thru 0x1fcf6 (0x66b9 bytes)
3: 136 insert from 0x16000 to 0x25000, (0xf000 bytes)
3: 137 collapse from 0x20000 to 0x2e000, (0xe000 bytes)
2: 133 read 0x6b07 thru 0x14ff1 (0xe4eb bytes)
2: 134 mapread 0x2be4a thru 0x30bfd (0x4db4 bytes)
2: 135 insert from 0x28000 to 0x2b000, (0x3000 bytes)
2: 136 fsync
3: 138 write 0x4783 thru 0xfc56 (0xb4d4 bytes)
3: 139 fsync
1: 166 falloc from 0x3c1a9 to 0x3d2b4 (0x110b bytes)
1: 167 falloc from 0x1e0fa to 0x2dad1 (0xf9d7 bytes)
1: 168 mapwrite 0x1d9af thru 0x1ddff (0x451 bytes)
0: Dumped fsync buffer to testfile0.mark11
0: 178 mapwrite 0x1589a thru 0x18d3b (0x34a2 bytes)
2: Dumped fsync buffer to testfile2.mark20
2: 137 fsync
2: Dumped fsync buffer to testfile2.mark21
2: 138 write 0xb308 thru 0x157e4 (0xa4dd bytes)
2: 139 zero from 0x255eb to 0x272b7, (0x1ccc bytes)
2: 140 mapread 0x11531 thru 0x1d8d4 (0xc3a4 bytes)
2: 141 trunc from 0x33bfe to 0xfcbb
2: 142 insert from 0x7000 to 0x14000, (0xd000 bytes)
1: 169 punch from 0x5988 to 0x7e0d, (0x2485 bytes)
2: 143 falloc from 0x16be0 to 0x228c7 (0xbce7 bytes)
3: Dumped fsync buffer to testfile3.mark12
3: 140 insert from 0x20000 to 0x2d000, (0xd000 bytes)
0: 179 trunc from 0x2c3d2 to 0x303bf
0: 180 zero from 0x3c01 to 0x9282, (0x5681 bytes)
3: 141 punch from 0x1df9c to 0x2a9f8, (0xca5c bytes)
3: 142 falloc from 0x31fd7 to 0x40000 (0xe029 bytes)
2: 144 mapwrite 0x325a thru 0x11e8f (0xec36 bytes)
0: 181 write 0x34906 thru 0x3d5a8 (0x8ca3 bytes)
0: 182 mapwrite 0xecb9 thru 0x17b12 (0x8e5a bytes)
1: 170 read 0x3efa5 thru 0x3f122 (0x17e bytes)
1: 171 punch from 0x157c to 0x25e8, (0x106c bytes)
3: 143 trunc from 0x40000 to 0x11dae
3: 144 falloc from 0xd162 to 0xf971 (0x280f bytes)
3: 145 trunc from 0x11dae to 0x9a00
3: 146 mapwrite 0x1be24 thru 0x23bbc (0x7d99 bytes)
1: 172 trunc from 0x3f123 to 0x13806
1: 173 fsync
0: 183 mapwrite 0x23d6e thru 0x33b38 (0xfdcb bytes)
3: 147 collapse from 0x1a000 to 0x1d000, (0x3000 bytes)
2: 145 mapwrite 0x32a24 thru 0x36fb9 (0x4596 bytes)
3: 148 zero from 0x20137 to 0x20bbd, (0xa86 bytes)
0: 184 mapwrite 0x2bcf8 thru 0x2c483 (0x78c bytes)
1: Dumped fsync buffer to testfile1.mark14
1: 174 mapread 0xccc4 thru 0xdd9f (0x10dc bytes)
1: 175 collapse from 0xd000 to 0x12000, (0x5000 bytes)
1: 176 trunc from 0xe806 to 0x1301d
1: 177 falloc from 0x2d706 to 0x385e7 (0xaee1 bytes)
1: 178 zero from 0x22c9d to 0x31d09, (0xf06c bytes)
1: 179 zero from 0x2aff8 to 0x3222e, (0x7236 bytes)
1: 180 punch from 0x1cf4b to 0x27a5c, (0xab11 bytes)
1: 181 mapread 0x20d71 thru 0x28435 (0x76c5 bytes)
1: 182 punch from 0x27db9 to 0x32c11, (0xae58 bytes)
1: 183 trunc from 0x385e7 to 0x1d604
1: 184 read 0xe94f thru 0x1130e (0x29c0 bytes)
1: 185 fsync
3: 149 punch from 0x6d86 to 0x11679, (0xa8f3 bytes)
3: 150 punch from 0x5be3 to 0xcd7f, (0x719c bytes)
0: 185 mapread 0x2553a thru 0x2650f (0xfd6 bytes)
2: 146 collapse from 0x26000 to 0x2c000, (0x6000 bytes)
0: 186 punch from 0x30f60 to 0x3d5a9, (0xc649 bytes)
2: 147 fsync
3: 151 collapse from 0x14000 to 0x17000, (0x3000 bytes)
3: 152 trunc from 0x1dbbd to 0x17c87
0: 187 read 0x34d6e thru 0x3d351 (0x85e4 bytes)
0: 188 write 0x1cfff thru 0x25df6 (0x8df8 bytes)
3: 153 collapse from 0x4000 to 0xf000, (0xb000 bytes)
0: 189 mapread 0x24967 thru 0x32556 (0xdbf0 bytes)
0: 190 write 0x11950 thru 0x17107 (0x57b8 bytes)
0: 191 fsync
1: Dumped fsync buffer to testfile1.mark15
1: 186 insert from 0x17000 to 0x19000, (0x2000 bytes)
1: 187 mapwrite 0x28c30 thru 0x2fd27 (0x70f8 bytes)
2: Dumped fsync buffer to testfile2.mark22
2: 148 zero from 0x1e5f9 to 0x2abc7, (0xc5ce bytes)
2: 149 mapread 0x24ece thru 0x2ca65 (0x7b98 bytes)
3: 154 zero from 0x5c9f to 0xac45, (0x4fa6 bytes)
3: 155 zero from 0xade to 0x5795, (0x4cb7 bytes)
2: 150 punch from 0x732c to 0xaa7f, (0x3753 bytes)
3: 156 write 0x1dca5 thru 0x23749 (0x5aa5 bytes)
3: 157 read 0xfd6 thru 0xcfec (0xc017 bytes)
3: 158 insert from 0x1f000 to 0x2b000, (0xc000 bytes)
2: 151 mapwrite 0x231bc thru 0x29ae2 (0x6927 bytes)
3: 159 falloc from 0x25c87 to 0x3469a (0xea13 bytes)
3: 160 write 0x26b15 thru 0x35bb1 (0xf09d bytes)
3: 161 fsync
0: Dumped fsync buffer to testfile0.mark12
0: 192 read 0xcc18 thru 0x17539 (0xa922 bytes)
0: 193 collapse from 0x36000 to 0x3c000, (0x6000 bytes)
1: 188 falloc from 0x34d4e to 0x40000 (0xb2b2 bytes)
0: 194 collapse from 0x1f000 to 0x27000, (0x8000 bytes)
1: 189 mapwrite 0x22d59 thru 0x2a85f (0x7b07 bytes)
0: 195 mapread 0x1ee8b thru 0x243a3 (0x5519 bytes)
2: 152 write 0x1325 thru 0x53be (0x409a bytes)
2: 153 read 0xb71f thru 0x1ab2f (0xf411 bytes)
2: 154 trunc from 0x30fba to 0x24a56
0: 196 mapwrite 0x16005 thru 0x1e1e4 (0x81e0 bytes)
2: 155 mapread 0x1ef56 thru 0x24a55 (0x5b00 bytes)
2: 157 mapwrite 0x1ba3b thru 0x1d61d (0x1be3 bytes)
1: 190 mapwrite 0x1e784 thru 0x2151a (0x2d97 bytes)
3: Dumped fsync buffer to testfile3.mark13
3: 162 fsync
2: 158 zero from 0x1bdaa to 0x20af5, (0x4d4b bytes)
0: 197 mapread 0x246b1 thru 0x2f5a8 (0xaef8 bytes)
0: 198 trunc from 0x2f5a9 to 0x3c99f
2: 160 write 0x2d643 thru 0x30cdb (0x3699 bytes)
2: 161 insert from 0xd000 to 0x1b000, (0xe000 bytes)
0: 199 falloc from 0x26de1 to 0x2dd44 (0x6f63 bytes)
0: 200 zero from 0x1735c to 0x22ec4, (0xbb68 bytes)
3: Dumped fsync buffer to testfile3.mark14
3: 163 falloc from 0x13e41 to 0x1ce27 (0x8fe6 bytes)
3: 164 insert from 0xa000 to 0xb000, (0x1000 bytes)
1: 191 fsync
2: 162 mapread 0x3520e thru 0x3badf (0x68d2 bytes)
3: 165 read 0xdb7a thru 0xeb24 (0xfab bytes)
2: 163 collapse from 0x1d000 to 0x2b000, (0xe000 bytes)
3: 166 mapread 0x25538 thru 0x2abfe (0x56c7 bytes)
0: All 200 operations completed A-OK!
3: 167 insert from 0x2a000 to 0x33000, (0x9000 bytes)
1: Dumped fsync buffer to testfile1.mark16
1: 192 write 0x4526 thru 0xa22a (0x5d05 bytes)
1: 193 falloc from 0x31ab4 to 0x3aacf (0x901b bytes)
1: 194 falloc from 0x76c1 to 0xa3c0 (0x2cff bytes)
1: 195 mapwrite 0x1c3f7 thru 0x23d75 (0x797f bytes)
3: 168 falloc from 0x34e67 to 0x3c4b7 (0x7650 bytes)
3: 169 punch from 0x28e2f to 0x33fe8, (0xb1b9 bytes)
2: 164 falloc from 0x10990 to 0x113ae (0xa1e bytes)
2: 165 insert from 0x5000 to 0xc000, (0x7000 bytes)
3: 170 collapse from 0x3a000 to 0x3f000, (0x5000 bytes)
3: 171 insert from 0xd000 to 0x12000, (0x5000 bytes)
2: 166 zero from 0x67e9 to 0x12d00, (0xc517 bytes)
1: 196 mapread 0x63e thru 0xd9c (0x75f bytes)
2: 167 insert from 0xf000 to 0x17000, (0x8000 bytes)
1: 200 zero from 0x166e to 0xd749, (0xc0db bytes)
2: 169 trunc from 0x3fcdc to 0x1d788
2: 170 mapwrite 0x25a02 thru 0x2fbf0 (0xa1ef bytes)
1: All 200 operations completed A-OK!
3: 172 zero from 0x38162 to 0x3fbb2, (0x7a50 bytes)
3: 173 write 0x19356 thru 0x292ec (0xff97 bytes)
3: 174 fsync
2: 171 collapse from 0x1e000 to 0x1f000, (0x1000 bytes)
2: 172 write 0x1eb43 thru 0x2b480 (0xc93e bytes)
2: 173 zero from 0x2a8eb to 0x2ebf1, (0x4306 bytes)
2: 174 fsync
3: Dumped fsync buffer to testfile3.mark15
3: 175 punch from 0x2d286 to 0x38560, (0xb2da bytes)
3: 177 fsync
2: Dumped fsync buffer to testfile2.mark23
2: 175 trunc from 0x2ebf1 to 0x3e7ba
2: 176 zero from 0x20b7e to 0x2ac22, (0xa0a4 bytes)
2: 177 punch from 0x18726 to 0x1fdb7, (0x7691 bytes)
2: 178 falloc from 0x27604 to 0x33978 (0xc374 bytes)
2: 179 write 0xfb25 thru 0x1a3cb (0xa8a7 bytes)
2: 180 collapse from 0x2d000 to 0x30000, (0x3000 bytes)
2: 181 mapread 0x1cd93 thru 0x2b733 (0xe9a1 bytes)
2: 182 mapwrite 0x305d2 thru 0x33459 (0x2e88 bytes)
3: Dumped fsync buffer to testfile3.mark16
3: 178 mapwrite 0x36012 thru 0x3ffff (0x9fee bytes)
2: 183 insert from 0x36000 to 0x3a000, (0x4000 bytes)
2: 184 write 0x18804 thru 0x21e3a (0x9637 bytes)
2: 185 fsync
2: Dumped fsync buffer to testfile2.mark24
2: 186 read 0xf4d6 thru 0x1bd3d (0xc868 bytes)
2: 187 punch from 0x28f3f to 0x3778e, (0xe84f bytes)
2: 188 collapse from 0x0 to 0x4000, (0x4000 bytes)
2: 189 punch from 0x28e82 to 0x300f0, (0x726e bytes)
2: 190 punch from 0x184e6 to 0x1c404, (0x3f1e bytes)
3: 180 falloc from 0x16ed0 to 0x2623f (0xf36f bytes)
3: 181 mapread 0x1ed9f thru 0x2249b (0x36fd bytes)
3: 182 mapread 0x153ac thru 0x22d68 (0xd9bd bytes)
3: 183 falloc from 0x15c2b to 0x1e9f0 (0x8dc5 bytes)
3: 184 read 0x3519b thru 0x3ffff (0xae65 bytes)
3: 185 mapwrite 0x299c1 thru 0x2d91b (0x3f5b bytes)
2: 191 mapwrite 0x10674 thru 0x10e85 (0x812 bytes)
2: 192 mapwrite 0x7e46 thru 0x11e6b (0xa026 bytes)
3: 186 collapse from 0x2d000 to 0x32000, (0x5000 bytes)
3: 187 mapwrite 0xa0eb thru 0xe332 (0x4248 bytes)
2: 193 mapread 0x179d9 thru 0x23d46 (0xc36e bytes)
2: 194 falloc from 0x1f85b to 0x2d4ca (0xdc6f bytes)
2: 195 write 0x4536 thru 0x13cf7 (0xf7c2 bytes)
2: 196 collapse from 0x18000 to 0x1d000, (0x5000 bytes)
2: 197 mapwrite 0xdae6 thru 0x1bd9e (0xe2b9 bytes)
3: 188 insert from 0x18000 to 0x1d000, (0x5000 bytes)
3: 189 collapse from 0x26000 to 0x33000, (0xd000 bytes)
3: 190 mapread 0x10aa8 thru 0x16c75 (0x61ce bytes)
3: 191 mapread 0x15b40 thru 0x22801 (0xccc2 bytes)
3: READ BAD DATA: offset = 0x15b40, size = 0xccc2, fname = /mnt/scratch/testfile3
3: OFFSET GOOD BAD RANGE
3: 0x1e000 0x0000 0x92623: 0x00000
3: operation# (mod 256) for the bad data may be 146
3: 0x1e001 0x0000 0x62923: 0x00001
3: operation# (mod 256) for the bad data may be 146
3: 0x1e002 0x0000 0x92fb3: 0x00002
3: operation# (mod 256) for the bad data may be 146
3: 0x1e003 0x0000 0xfb923: 0x00003
3: operation# (mod 256) for the bad data may be 146
3: 0x1e004 0x0000 0x921d3: 0x00004
3: operation# (mod 256) for the bad data may be 146
3: 0x1e005 0x0000 0x1d923: 0x00005
3: operation# (mod 256) for the bad data may be 146
3: 0x1e006 0x0000 0x924c3: 0x00006
3: operation# (mod 256) for the bad data may be 146
3: 0x1e007 0x0000 0x4c923: 0x00007
3: operation# (mod 256) for the bad data may be 146
3: 0x1e008 0x0000 0x92123: 0x00008
3: operation# (mod 256) for the bad data may be 146
3: 0x1e009 0x0000 0x12923: 0x00009
3: operation# (mod 256) for the bad data may be 146
3: 0x1e00a 0x0000 0x92a03: 0x0000a
3: operation# (mod 256) for the bad data may be 146
3: 0x1e00b 0x0000 0xa0923: 0x0000b
3: operation# (mod 256) for the bad data may be 146
3: 0x1e00c 0x0000 0x925c3: 0x0000c
3: operation# (mod 256) for the bad data may be 146
3: 0x1e00d 0x0000 0x5c923: 0x0000d
3: operation# (mod 256) for the bad data may be 146
3: 0x1e00e 0x0000 0x92f73: 0x0000e
3: operation# (mod 256) for the bad data may be 146
3: 0x1e00f 0x0000 0xf7923: 0x0000f
3: operation# (mod 256) for the bad data may be 146
3: LOG DUMP (191 total operations):
3: /mnt/test/fsxtests/: Is a directory
3: 1( 1 mod 256): 3: MAPWRITE 0x61d8 thru 0x100d7 (0x9f00 bytes)3:
3: 2( 2 mod 256): 3: MAPWRITE 0xd137 thru 0x16550 (0x941a bytes)3:
3: 3( 3 mod 256): 3: MAPWRITE 0x27e98 thru 0x31859 (0x99c2 bytes)3:
3: 4( 4 mod 256): 3: TRUNCATE DOWN from 0x3185a to 0x67d33: ******WWWW3:
3: 5( 5 mod 256): 3: COLLAPSE 0x4000 thru 0x5fff (0x2000 bytes)3:
3: 6( 6 mod 256): 3: SKIPPED (no operation)3:
3: 7( 7 mod 256): 3: TRUNCATE UP from 0x47d3 to 0x11f123:
3: 8( 8 mod 256): 3: MAPWRITE 0x250b6 thru 0x31fde (0xcf29 bytes)3:
3: 9( 9 mod 256): 3: PUNCH 0x1b308 thru 0x1fc62 (0x495b bytes)3:
3: 10( 10 mod 256): 3: MAPREAD 0x25033 thru 0x31310 (0xc2de bytes)3:
3: 11( 11 mod 256): 3: MAPREAD 0x3a97 thru 0xf2ad (0xb817 bytes)3:
3: 12( 12 mod 256): 3: MAPWRITE 0x2e485 thru 0x2f8a8 (0x1424 bytes)3:
3: 13( 13 mod 256): 3: READ 0xa335 thru 0x143ad (0xa079 bytes)3:
3: 14( 14 mod 256): 3: COLLAPSE 0xf000 thru 0x11fff (0x3000 bytes)3:
3: 15( 15 mod 256): 3: ZERO 0x2131b thru 0x2dba4 (0xc88a bytes)3: ******ZZZZ3:
3: 16( 16 mod 256): 3: WRITE 0x2a2db thru 0x38018 (0xdd3e bytes)3: EXTEND3:
3: 17( 17 mod 256): 3: READ 0x20d8d thru 0x23216 (0x248a bytes)3: ***RRRR***3:
3: 18( 18 mod 256): 3: COLLAPSE 0x2b000 thru 0x2efff (0x4000 bytes)3:
3: 19( 19 mod 256): 3: TRUNCATE DOWN from 0x34019 to 0x33ffb3:
3: 20( 20 mod 256): 3: INSERT 0x2f000 thru 0x3afff (0xc000 bytes)3:
3: 21( 21 mod 256): 3: FALLOC 0x37bf6 thru 0x40000 (0x840a bytes) 3: PAST_EOF3:
3: 22( 22 mod 256): 3: FSYNC3:
3: 23( 23 mod 256): 3: COLLAPSE 0x18000 thru 0x25fff (0xe000 bytes)3: ******CCCC3:
3: 24( 24 mod 256): 3: FALLOC 0x10f63 thru 0x1918d (0x822a bytes) 3: INTERIOR3:
3: 25( 25 mod 256): 3: ZERO 0x2bdb2 thru 0x2d4dc (0x172b bytes)3:
3: 26( 26 mod 256): 3: FALLOC 0x109d3 thru 0x14fbb (0x45e8 bytes) 3: INTERIOR3:
3: 27( 27 mod 256): 3: READ 0x26e7f thru 0x306ca (0x984c bytes)3:
3: 28( 28 mod 256): 3: ZERO 0x312e3 thru 0x31ffa (0xd18 bytes)3:
3: 29( 29 mod 256): 3: READ 0x1eea3 thru 0x2cced (0xde4b bytes)3: ***RRRR***3:
3: 30( 30 mod 256): 3: ZERO 0x7edd thru 0xa3f1 (0x2515 bytes)3:
3: 31( 31 mod 256): 3: FSYNC3:
3: 32( 32 mod 256): 3: WRITE 0x13ec2 thru 0x1463c (0x77b bytes)3:
3: 33( 33 mod 256): 3: MAPREAD 0x1834d thru 0x1a47e (0x2132 bytes)3:
3: 34( 34 mod 256): 3: INSERT 0x11000 thru 0x1efff (0xe000 bytes)3:
3: 35( 35 mod 256): 3: READ 0x25670 thru 0x2847f (0x2e10 bytes)3:
3: 36( 36 mod 256): 3: FALLOC 0x11a8e thru 0x19c7a (0x81ec bytes) 3: INTERIOR3:
3: 37( 37 mod 256): 3: SKIPPED (no operation)3:
3: 38( 38 mod 256): 3: FSYNC3:
3: 39( 39 mod 256): 3: FALLOC 0x3aa6a thru 0x40000 (0x5596 bytes) 3: PAST_EOF3:
3: 40( 40 mod 256): 3: PUNCH 0x33ca3 thru 0x3815d (0x44bb bytes)3:
3: 41( 41 mod 256): 3: ZERO 0x291bb thru 0x29ae4 (0x92a bytes)3:
3: 42( 42 mod 256): 3: ZERO 0x451b thru 0xb7d1 (0x72b7 bytes)3:
3: 43( 43 mod 256): 3: MAPREAD 0x1a264 thru 0x1fdbb (0x5b58 bytes)3:
3: 44( 44 mod 256): 3: SKIPPED (no operation)3:
3: 45( 45 mod 256): 3: SKIPPED (no operation)3:
3: 46( 46 mod 256): 3: WRITE 0x19ea9 thru 0x26b2f (0xcc87 bytes)3: ***WWWW3:
3: 47( 47 mod 256): 3: TRUNCATE DOWN from 0x3fffb to 0x29d953:
3: 48( 48 mod 256): 3: MAPWRITE 0x163d3 thru 0x25dbc (0xf9ea bytes)3: ******WWWW3:
3: 49( 49 mod 256): 3: MAPREAD 0xa24b thru 0x1921a (0xefd0 bytes)3:
3: 50( 50 mod 256): 3: READ 0x1db9a thru 0x20aa7 (0x2f0e bytes)3:
3: 51( 51 mod 256): 3: MAPREAD 0x380b thru 0x6404 (0x2bfa bytes)3:
3: 52( 52 mod 256): 3: FSYNC3:
3: 53( 53 mod 256): 3: FALLOC 0x31b2e thru 0x3789d (0x5d6f bytes) 3: PAST_EOF3:
3: 54( 54 mod 256): 3: MAPWRITE 0x3fb90 thru 0x3ffff (0x470 bytes)3:
3: 55( 55 mod 256): 3: MAPREAD 0x1668e thru 0x25cd7 (0xf64a bytes)3: ***RRRR***3:
3: 56( 56 mod 256): 3: FALLOC 0xe573 thru 0x13bc8 (0x5655 bytes) 3: INTERIOR3:
3: 57( 57 mod 256): 3: ZERO 0x9e85 thru 0xaba2 (0xd1e bytes)3:
3: 58( 58 mod 256): 3: WRITE 0x380dd thru 0x3ed89 (0x6cad bytes)3:
3: 59( 59 mod 256): 3: READ 0x3ba0 thru 0x10289 (0xc6ea bytes)3:
3: 60( 60 mod 256): 3: COLLAPSE 0x30000 thru 0x37fff (0x8000 bytes)3:
3: 61( 61 mod 256): 3: READ 0xd1bb thru 0xdcae (0xaf4 bytes)3:
3: 62( 62 mod 256): 3: FSYNC3:
3: 63( 63 mod 256): 3: MAPREAD 0x48b1 thru 0x120b9 (0xd809 bytes)3:
3: 64( 64 mod 256): 3: ZERO 0x2e34b thru 0x323f5 (0x40ab bytes)3:
3: 65( 65 mod 256): 3: ZERO 0x19f31 thru 0x29bef (0xfcbf bytes)3: ******ZZZZ3:
3: 66( 66 mod 256): 3: ZERO 0x2dfa9 thru 0x36009 (0x8061 bytes)3:
3: 67( 67 mod 256): 3: PUNCH 0x15031 thru 0x218c8 (0xc898 bytes)3:
3: 68( 68 mod 256): 3: FSYNC3:
3: 69( 69 mod 256): 3: FALLOC 0x29b87 thru 0x2a4e1 (0x95a bytes) 3: INTERIOR3:
3: 70( 70 mod 256): 3: MAPWRITE 0x20a10 thru 0x21d3d (0x132e bytes)3:
3: 71( 71 mod 256): 3: MAPREAD 0xf37e thru 0x18664 (0x92e7 bytes)3:
3: 72( 72 mod 256): 3: WRITE 0x3323 thru 0xe65b (0xb339 bytes)3:
3: 73( 73 mod 256): 3: PUNCH 0xa5ad thru 0xf824 (0x5278 bytes)3:
3: 74( 74 mod 256): 3: MAPREAD 0x22896 thru 0x2de0a (0xb575 bytes)3:
3: 75( 75 mod 256): 3: COLLAPSE 0x1f000 thru 0x27fff (0x9000 bytes)3: ******CCCC3:
3: 76( 76 mod 256): 3: MAPREAD 0x299ce thru 0x2efff (0x5632 bytes)3:
3: 77( 77 mod 256): 3: READ 0xc393 thru 0xd12c (0xd9a bytes)3:
3: 78( 78 mod 256): 3: INSERT 0x9000 thru 0xdfff (0x5000 bytes)3:
3: 79( 79 mod 256): 3: MAPREAD 0x253b0 thru 0x25671 (0x2c2 bytes)3:
3: 80( 80 mod 256): 3: INSERT 0x16000 thru 0x21fff (0xc000 bytes)3:
3: 81( 81 mod 256): 3: SKIPPED (no operation)3:
3: 82( 82 mod 256): 3: MAPREAD 0xa5de thru 0xcad6 (0x24f9 bytes)3:
3: 83( 83 mod 256): 3: MAPREAD 0x2ded2 thru 0x3a956 (0xca85 bytes)3:
3: 84( 84 mod 256): 3: PUNCH 0xf217 thru 0x1680b (0x75f5 bytes)3:
3: 85( 85 mod 256): 3: MAPWRITE 0x36da9 thru 0x3e20f (0x7467 bytes)3:
3: 86( 86 mod 256): 3: MAPREAD 0xaafc thru 0x1245f (0x7964 bytes)3:
3: 87( 87 mod 256): 3: PUNCH 0x16a6 thru 0x3785 (0x20e0 bytes)3:
3: 88( 88 mod 256): 3: FALLOC 0xf555 thru 0x1b150 (0xbbfb bytes) 3: INTERIOR3:
3: 89( 89 mod 256): 3: FALLOC 0x3a0b3 thru 0x40000 (0x5f4d bytes) 3: INTERIOR3:
3: 90( 90 mod 256): 3: MAPREAD 0xd820 thru 0xe474 (0xc55 bytes)3:
3: 91( 91 mod 256): 3: COLLAPSE 0x25000 thru 0x32fff (0xe000 bytes)3:
3: 92( 92 mod 256): 3: PUNCH 0x15810 thru 0x1edf5 (0x95e6 bytes)3:
3: 93( 93 mod 256): 3: PUNCH 0x1723d thru 0x1cac9 (0x588d bytes)3:
3: 94( 94 mod 256): 3: TRUNCATE DOWN from 0x32000 to 0x12e853: ******WWWW3:
3: 95( 95 mod 256): 3: FSYNC3:
3: 96( 96 mod 256): 3: WRITE 0x3e757 thru 0x3ffff (0x18a9 bytes)3: HOLE3: ***WWWW3:
3: 97( 97 mod 256): 3: ZERO 0x4b1 thru 0x48ac (0x43fc bytes)3:
3: 98( 98 mod 256): 3: SKIPPED (no operation)3:
3: 99( 99 mod 256): 3: COLLAPSE 0xd000 thru 0x11fff (0x5000 bytes)3:
3: 100(100 mod 256): 3: READ 0x36303 thru 0x3afff (0x4cfd bytes)3:
3: 101(101 mod 256): 3: MAPREAD 0x8a0b thru 0x18509 (0xfaff bytes)3:
3: 102(102 mod 256): 3: MAPWRITE 0x1ac3a thru 0x24971 (0x9d38 bytes)3: ******WWWW3:
3: 103(103 mod 256): 3: INSERT 0x20000 thru 0x24fff (0x5000 bytes)3: ******IIII3:
3: 104(104 mod 256): 3: FSYNC3:
3: 105(105 mod 256): 3: SKIPPED (no operation)3:
3: 106(106 mod 256): 3: PUNCH 0x38187 thru 0x3ffff (0x7e79 bytes)3:
3: 107(107 mod 256): 3: FSYNC3:
3: 108(108 mod 256): 3: COLLAPSE 0x39000 thru 0x39fff (0x1000 bytes)3:
3: 109(109 mod 256): 3: PUNCH 0x1260 thru 0xb852 (0xa5f3 bytes)3:
3: 110(110 mod 256): 3: WRITE 0x35197 thru 0x3ffff (0xae69 bytes)3: EXTEND3:
3: 111(111 mod 256): 3: SKIPPED (no operation)3:
3: 112(112 mod 256): 3: ZERO 0x252ee thru 0x282db (0x2fee bytes)3:
3: 113(113 mod 256): 3: ZERO 0x2b74b thru 0x2ee93 (0x3749 bytes)3:
3: 114(114 mod 256): 3: TRUNCATE DOWN from 0x40000 to 0x10d003: ******WWWW3:
3: 115(115 mod 256): 3: ZERO 0x14e6 thru 0x591e (0x4439 bytes)3:
3: 116(116 mod 256): 3: WRITE 0x1687e thru 0x1daa4 (0x7227 bytes)3: HOLE3:
3: 117(117 mod 256): 3: MAPREAD 0x1c459 thru 0x1daa4 (0x164c bytes)3:
3: 118(118 mod 256): 3: INSERT 0x10000 thru 0x14fff (0x5000 bytes)3:
3: 119(119 mod 256): 3: TRUNCATE DOWN from 0x22aa5 to 0x1f4de3: ******WWWW3:
3: 120(120 mod 256): 3: FSYNC3:
3: 121(121 mod 256): 3: INSERT 0xb000 thru 0x15fff (0xb000 bytes)3:
3: 122(122 mod 256): 3: FSYNC3:
3: 123(123 mod 256): 3: TRUNCATE UP from 0x2a4de to 0x2e75c3:
3: 124(124 mod 256): 3: PUNCH 0x1640d thru 0x19069 (0x2c5d bytes)3:
3: 125(125 mod 256): 3: WRITE 0x64ff thru 0x6d09 (0x80b bytes)3:
3: 126(126 mod 256): 3: FSYNC3:
3: 127(127 mod 256): 3: MAPWRITE 0x9e40 thru 0x174eb (0xd6ac bytes)3:
3: 128(128 mod 256): 3: MAPREAD 0x2847c thru 0x2e75b (0x62e0 bytes)3:
3: 129(129 mod 256): 3: ZERO 0xc359 thru 0x19f31 (0xdbd9 bytes)3:
3: 130(130 mod 256): 3: READ 0x66ce thru 0xcc68 (0x659b bytes)3:
3: 131(131 mod 256): 3: TRUNCATE DOWN from 0x2e75c to 0xec663: ******WWWW3:
3: 132(132 mod 256): 3: PUNCH 0x2900 thru 0xec65 (0xc366 bytes)3:
3: 133(133 mod 256): 3: COLLAPSE 0x4000 thru 0xdfff (0xa000 bytes)3:
3: 134(134 mod 256): 3: MAPREAD 0x2f8b thru 0x4c65 (0x1cdb bytes)3:
3: 135(135 mod 256): 3: WRITE 0x1963e thru 0x1fcf6 (0x66b9 bytes)3: HOLE3:
3: 136(136 mod 256): 3: INSERT 0x16000 thru 0x24fff (0xf000 bytes)3: ******IIII3:
3: 137(137 mod 256): 3: COLLAPSE 0x20000 thru 0x2dfff (0xe000 bytes)3: ******CCCC3:
3: 138(138 mod 256): 3: WRITE 0x4783 thru 0xfc56 (0xb4d4 bytes)3:
3: 139(139 mod 256): 3: FSYNC3:
3: 140(140 mod 256): 3: INSERT 0x20000 thru 0x2cfff (0xd000 bytes)3: ******IIII3:
3: 141(141 mod 256): 3: PUNCH 0x1df9c thru 0x2a9f7 (0xca5c bytes)3: ******PPPP3:
3: 142(142 mod 256): 3: FALLOC 0x31fd7 thru 0x40000 (0xe029 bytes) 3: EXTENDING3:
3: 143(143 mod 256): 3: TRUNCATE DOWN from 0x40000 to 0x11dae3: ******WWWW3:
3: 144(144 mod 256): 3: FALLOC 0xd162 thru 0xf971 (0x280f bytes) 3: INTERIOR3:
3: 145(145 mod 256): 3: TRUNCATE DOWN from 0x11dae to 0x9a003:
3: 146(146 mod 256): 3: MAPWRITE 0x1be24 thru 0x23bbc (0x7d99 bytes)3: ******WWWW3:
3: 147(147 mod 256): 3: COLLAPSE 0x1a000 thru 0x1cfff (0x3000 bytes)3:
3: 148(148 mod 256): 3: ZERO 0x20137 thru 0x20bbc (0xa86 bytes)3:
3: 149(149 mod 256): 3: PUNCH 0x6d86 thru 0x11678 (0xa8f3 bytes)3:
3: 150(150 mod 256): 3: PUNCH 0x5be3 thru 0xcd7e (0x719c bytes)3:
3: 151(151 mod 256): 3: COLLAPSE 0x14000 thru 0x16fff (0x3000 bytes)3:
3: 152(152 mod 256): 3: TRUNCATE DOWN from 0x1dbbd to 0x17c873:
3: 153(153 mod 256): 3: COLLAPSE 0x4000 thru 0xefff (0xb000 bytes)3:
3: 154(154 mod 256): 3: ZERO 0x5c9f thru 0xac44 (0x4fa6 bytes)3:
3: 155(155 mod 256): 3: ZERO 0xade thru 0x5794 (0x4cb7 bytes)3:
3: 156(156 mod 256): 3: WRITE 0x1dca5 thru 0x23749 (0x5aa5 bytes)3: HOLE3: ***WWWW3:
3: 157(157 mod 256): 3: READ 0xfd6 thru 0xcfec (0xc017 bytes)3:
3: 158(158 mod 256): 3: INSERT 0x1f000 thru 0x2afff (0xc000 bytes)3: ******IIII3:
3: 159(159 mod 256): 3: FALLOC 0x25c87 thru 0x3469a (0xea13 bytes) 3: EXTENDING3:
3: 160(160 mod 256): 3: WRITE 0x26b15 thru 0x35bb1 (0xf09d bytes)3: EXTEND3:
3: 161(161 mod 256): 3: FSYNC3:
3: 162(162 mod 256): 3: FSYNC3:
3: 163(163 mod 256): 3: FALLOC 0x13e41 thru 0x1ce27 (0x8fe6 bytes) 3: INTERIOR3:
3: 164(164 mod 256): 3: INSERT 0xa000 thru 0xafff (0x1000 bytes)3:
3: 165(165 mod 256): 3: READ 0xdb7a thru 0xeb24 (0xfab bytes)3:
3: 166(166 mod 256): 3: MAPREAD 0x25538 thru 0x2abfe (0x56c7 bytes)3:
3: 167(167 mod 256): 3: INSERT 0x2a000 thru 0x32fff (0x9000 bytes)3:
3: 168(168 mod 256): 3: FALLOC 0x34e67 thru 0x3c4b7 (0x7650 bytes) 3: INTERIOR3:
3: 169(169 mod 256): 3: PUNCH 0x28e2f thru 0x33fe7 (0xb1b9 bytes)3:
3: 170(170 mod 256): 3: COLLAPSE 0x3a000 thru 0x3efff (0x5000 bytes)3:
3: 171(171 mod 256): 3: INSERT 0xd000 thru 0x11fff (0x5000 bytes)3:
3: 172(172 mod 256): 3: ZERO 0x38162 thru 0x3fbb1 (0x7a50 bytes)3:
3: 173(173 mod 256): 3: WRITE 0x19356 thru 0x292ec (0xff97 bytes)3: ***WWWW3:
3: 174(174 mod 256): 3: FSYNC3:
3: 175(175 mod 256): 3: PUNCH 0x2d286 thru 0x3855f (0xb2da bytes)3:
3: 176(176 mod 256): 3: SKIPPED (no operation)3:
3: 177(177 mod 256): 3: FSYNC3:
3: 178(178 mod 256): 3: MAPWRITE 0x36012 thru 0x3ffff (0x9fee bytes)3:
3: 179(179 mod 256): 3: SKIPPED (no operation)3:
3: 180(180 mod 256): 3: FALLOC 0x16ed0 thru 0x2623f (0xf36f bytes) 3: INTERIOR3: ******FFFF3:
3: 181(181 mod 256): 3: MAPREAD 0x1ed9f thru 0x2249b (0x36fd bytes)3:
3: 182(182 mod 256): 3: MAPREAD 0x153ac thru 0x22d68 (0xd9bd bytes)3: ***RRRR***3:
3: 183(183 mod 256): 3: FALLOC 0x15c2b thru 0x1e9f0 (0x8dc5 bytes) 3: INTERIOR3:
3: 184(184 mod 256): 3: READ 0x3519b thru 0x3ffff (0xae65 bytes)3:
3: 185(185 mod 256): 3: MAPWRITE 0x299c1 thru 0x2d91b (0x3f5b bytes)3:
3: 186(186 mod 256): 3: COLLAPSE 0x2d000 thru 0x31fff (0x5000 bytes)3:
3: 187(187 mod 256): 3: MAPWRITE 0xa0eb thru 0xe332 (0x4248 bytes)3:
3: 188(188 mod 256): 3: INSERT 0x18000 thru 0x1cfff (0x5000 bytes)3:
3: 189(189 mod 256): 3: COLLAPSE 0x26000 thru 0x32fff (0xd000 bytes)3:
3: 190(190 mod 256): 3: MAPREAD 0x10aa8 thru 0x16c75 (0x61ce bytes)3:
3: 191(191 mod 256): 3: MAPREAD 0x15b40 thru 0x22801 (0xccc2 bytes)3: ***RRRR***3:
3: Correct content saved for comparison
3: (maybe hexdump "/mnt/scratch/testfile3" vs "/mnt/scratch/testfile3.fsxgood")
failed: '/old/home/amir/src/xfstests-dev/ltp/fsx -N 200 -d -P /mnt/test/fsxtests -i /dev/mapper/logwrites-test -S 0 -j 3 /mnt/scratch/testfile3'
2: 198 insert from 0x18000 to 0x19000, (0x1000 bytes)
2: 199 fsync
2: Dumped fsync buffer to testfile2.mark25
2: 200 fsync
2: Dumped fsync buffer to testfile2.mark26
2: All 200 operations completed A-OK!
checking pre umount
Malformed entry @116985
replay failed
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-11-28 16:48 ` Amir Goldstein
@ 2017-11-28 17:21 ` Josef Bacik
2017-11-28 19:32 ` Amir Goldstein
0 siblings, 1 reply; 35+ messages in thread
From: Josef Bacik @ 2017-11-28 17:21 UTC (permalink / raw)
To: Amir Goldstein
Cc: Josef Bacik, Josef Bacik, fstests, linux-fsdevel, Eryu Guan
On Tue, Nov 28, 2017 at 06:48:43PM +0200, Amir Goldstein wrote:
> On Mon, Nov 27, 2017 at 5:04 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> > On Mon, Nov 27, 2017 at 11:56:58AM +0200, Amir Goldstein wrote:
> >> On Tue, Sep 5, 2017 at 10:11 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> >> > Cherry-picked the test from commit 70d41e17164b
> >> > in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
> >> > Quoting from Josef's commit message:
> >> >
> >> > The test just runs some ops and exits, then finds all of the good buffers
> >> > in the directory we provided and:
> >> > - replays up to the mark given
> >> > - mounts the file system and compares the md5sum
> >> > - unmounts and fsck's to check for metadata integrity
> >> >
> >> > dm-log-writes will pretend to do discard and the replay-log tool will
> >> > replay it properly depending on the underlying device, either by writing
> >> > 0's or actually calling the discard ioctl, so I've enabled discard in the
> >> > test for maximum fun.
> >> >
> >> > [Amir:]
> >> > - Removed unneeded _test_falloc_support dynamic FSX_OPTS
> >> > - Fold repetitions into for loops
> >> > - Added place holders for using constant random seeds
> >> > - Add pre umount checkpint
> >> > - Add test to new 'replay' group
> >> > - Address review comments by Eryu Guan
> >> >
> >> > Cc: Josef Bacik <jbacik@fb.com>
> >> > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> >>
> >>
> >> Josef,
> >>
> >> As you know, this test is now merged to xfstest as generic/455.
> >> I have been running the test for a while on xfs and it occasionally
> >> reports inconsistencies which I try to investigate.
> >>
> >> In some of the reports, it appears that dm-log-writes may be exhibiting
> >> a reliability issue (see below).
> >>
> >
> > It's not a reliability issue, its a caching issue. dm-log-writes is just
> > issuing bio's to the log device, and our destructor waits for all pending io
> > blocks to complete before exiting, so unless I've missed how dm is destroying
> > devices everything should be on disk.
> >
> > However since we replay in userspace we are going through the blockdevice's
> > pagecache, so we could have stale pages left in place which is screwing us up.
> > Will you try this patch and see if it fixes the problem? Thanks,
> >
> > Josef
> >
> >
> > diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
> > index 8b80a9ce9ea9..1c502930af5e 100644
> > --- a/drivers/md/dm-log-writes.c
> > +++ b/drivers/md/dm-log-writes.c
> > @@ -545,6 +545,8 @@ static void log_writes_dtr(struct dm_target *ti)
> > !atomic_read(&lc->pending_blocks));
> > kthread_stop(lc->log_kthread);
> >
> > + invalidate_bdev(lc->logdev->bdev);
> > + invalidate_bdev(lc->dev->bdev);
> > WARN_ON(!list_empty(&lc->logging_blocks));
> > WARN_ON(!list_empty(&lc->unflushed_blocks));
> > dm_put_device(ti, lc->dev);
>
> Josef,
>
> With your patch OR with my xfstest patch that adds "sync" I did not yet see
> another problem of garbage fs after _log_writes_remove.
>
> I did however, encounter this error (failure to verify read data during fsx)
> from scratch/log-writes device (see attached full log).
>
> I will keep running the test to collect more information.
>
That failure I'll lay at the feet of whatever fs you are testing ;). I'm glad
my patch fixed the replay problem, I'll send that up. Thanks,
Josef
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-11-28 17:21 ` Josef Bacik
@ 2017-11-28 19:32 ` Amir Goldstein
2017-11-28 20:00 ` Josef Bacik
0 siblings, 1 reply; 35+ messages in thread
From: Amir Goldstein @ 2017-11-28 19:32 UTC (permalink / raw)
To: Josef Bacik; +Cc: Josef Bacik, fstests, linux-fsdevel, Eryu Guan
On Tue, Nov 28, 2017 at 7:21 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> On Tue, Nov 28, 2017 at 06:48:43PM +0200, Amir Goldstein wrote:
>> On Mon, Nov 27, 2017 at 5:04 PM, Josef Bacik <josef@toxicpanda.com> wrote:
>> > On Mon, Nov 27, 2017 at 11:56:58AM +0200, Amir Goldstein wrote:
>> >> On Tue, Sep 5, 2017 at 10:11 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>> >> > Cherry-picked the test from commit 70d41e17164b
>> >> > in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
>> >> > Quoting from Josef's commit message:
>> >> >
>> >> > The test just runs some ops and exits, then finds all of the good buffers
>> >> > in the directory we provided and:
>> >> > - replays up to the mark given
>> >> > - mounts the file system and compares the md5sum
>> >> > - unmounts and fsck's to check for metadata integrity
>> >> >
>> >> > dm-log-writes will pretend to do discard and the replay-log tool will
>> >> > replay it properly depending on the underlying device, either by writing
>> >> > 0's or actually calling the discard ioctl, so I've enabled discard in the
>> >> > test for maximum fun.
>> >> >
>> >> > [Amir:]
>> >> > - Removed unneeded _test_falloc_support dynamic FSX_OPTS
>> >> > - Fold repetitions into for loops
>> >> > - Added place holders for using constant random seeds
>> >> > - Add pre umount checkpint
>> >> > - Add test to new 'replay' group
>> >> > - Address review comments by Eryu Guan
>> >> >
>> >> > Cc: Josef Bacik <jbacik@fb.com>
>> >> > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
>> >>
>> >>
>> >> Josef,
>> >>
>> >> As you know, this test is now merged to xfstest as generic/455.
>> >> I have been running the test for a while on xfs and it occasionally
>> >> reports inconsistencies which I try to investigate.
>> >>
>> >> In some of the reports, it appears that dm-log-writes may be exhibiting
>> >> a reliability issue (see below).
>> >>
>> >
>> > It's not a reliability issue, its a caching issue. dm-log-writes is just
>> > issuing bio's to the log device, and our destructor waits for all pending io
>> > blocks to complete before exiting, so unless I've missed how dm is destroying
>> > devices everything should be on disk.
>> >
>> > However since we replay in userspace we are going through the blockdevice's
>> > pagecache, so we could have stale pages left in place which is screwing us up.
>> > Will you try this patch and see if it fixes the problem? Thanks,
>> >
>> > Josef
>> >
>> >
>> > diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
>> > index 8b80a9ce9ea9..1c502930af5e 100644
>> > --- a/drivers/md/dm-log-writes.c
>> > +++ b/drivers/md/dm-log-writes.c
>> > @@ -545,6 +545,8 @@ static void log_writes_dtr(struct dm_target *ti)
>> > !atomic_read(&lc->pending_blocks));
>> > kthread_stop(lc->log_kthread);
>> >
>> > + invalidate_bdev(lc->logdev->bdev);
>> > + invalidate_bdev(lc->dev->bdev);
>> > WARN_ON(!list_empty(&lc->logging_blocks));
>> > WARN_ON(!list_empty(&lc->unflushed_blocks));
>> > dm_put_device(ti, lc->dev);
>>
>> Josef,
>>
>> With your patch OR with my xfstest patch that adds "sync" I did not yet see
>> another problem of garbage fs after _log_writes_remove.
>>
>> I did however, encounter this error (failure to verify read data during fsx)
>> from scratch/log-writes device (see attached full log).
>>
>> I will keep running the test to collect more information.
>>
>
> That failure I'll lay at the feet of whatever fs you are testing ;). I'm glad
> my patch fixed the replay problem, I'll send that up. Thanks,
>
O oh!. You are implying that xfs fails plain fsx and nobody noticed.
That is not where I would place my bet.
Let's wait and see what other testers find and what other incidents
my test machine will find.
Cheers,
Amir.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-11-28 19:32 ` Amir Goldstein
@ 2017-11-28 20:00 ` Josef Bacik
2017-11-28 20:26 ` Amir Goldstein
0 siblings, 1 reply; 35+ messages in thread
From: Josef Bacik @ 2017-11-28 20:00 UTC (permalink / raw)
To: Amir Goldstein
Cc: Josef Bacik, Josef Bacik, fstests, linux-fsdevel, Eryu Guan
On Tue, Nov 28, 2017 at 09:32:59PM +0200, Amir Goldstein wrote:
> On Tue, Nov 28, 2017 at 7:21 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> > On Tue, Nov 28, 2017 at 06:48:43PM +0200, Amir Goldstein wrote:
> >> On Mon, Nov 27, 2017 at 5:04 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> >> > On Mon, Nov 27, 2017 at 11:56:58AM +0200, Amir Goldstein wrote:
> >> >> On Tue, Sep 5, 2017 at 10:11 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> >> >> > Cherry-picked the test from commit 70d41e17164b
> >> >> > in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
> >> >> > Quoting from Josef's commit message:
> >> >> >
> >> >> > The test just runs some ops and exits, then finds all of the good buffers
> >> >> > in the directory we provided and:
> >> >> > - replays up to the mark given
> >> >> > - mounts the file system and compares the md5sum
> >> >> > - unmounts and fsck's to check for metadata integrity
> >> >> >
> >> >> > dm-log-writes will pretend to do discard and the replay-log tool will
> >> >> > replay it properly depending on the underlying device, either by writing
> >> >> > 0's or actually calling the discard ioctl, so I've enabled discard in the
> >> >> > test for maximum fun.
> >> >> >
> >> >> > [Amir:]
> >> >> > - Removed unneeded _test_falloc_support dynamic FSX_OPTS
> >> >> > - Fold repetitions into for loops
> >> >> > - Added place holders for using constant random seeds
> >> >> > - Add pre umount checkpint
> >> >> > - Add test to new 'replay' group
> >> >> > - Address review comments by Eryu Guan
> >> >> >
> >> >> > Cc: Josef Bacik <jbacik@fb.com>
> >> >> > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> >> >>
> >> >>
> >> >> Josef,
> >> >>
> >> >> As you know, this test is now merged to xfstest as generic/455.
> >> >> I have been running the test for a while on xfs and it occasionally
> >> >> reports inconsistencies which I try to investigate.
> >> >>
> >> >> In some of the reports, it appears that dm-log-writes may be exhibiting
> >> >> a reliability issue (see below).
> >> >>
> >> >
> >> > It's not a reliability issue, its a caching issue. dm-log-writes is just
> >> > issuing bio's to the log device, and our destructor waits for all pending io
> >> > blocks to complete before exiting, so unless I've missed how dm is destroying
> >> > devices everything should be on disk.
> >> >
> >> > However since we replay in userspace we are going through the blockdevice's
> >> > pagecache, so we could have stale pages left in place which is screwing us up.
> >> > Will you try this patch and see if it fixes the problem? Thanks,
> >> >
> >> > Josef
> >> >
> >> >
> >> > diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
> >> > index 8b80a9ce9ea9..1c502930af5e 100644
> >> > --- a/drivers/md/dm-log-writes.c
> >> > +++ b/drivers/md/dm-log-writes.c
> >> > @@ -545,6 +545,8 @@ static void log_writes_dtr(struct dm_target *ti)
> >> > !atomic_read(&lc->pending_blocks));
> >> > kthread_stop(lc->log_kthread);
> >> >
> >> > + invalidate_bdev(lc->logdev->bdev);
> >> > + invalidate_bdev(lc->dev->bdev);
> >> > WARN_ON(!list_empty(&lc->logging_blocks));
> >> > WARN_ON(!list_empty(&lc->unflushed_blocks));
> >> > dm_put_device(ti, lc->dev);
> >>
> >> Josef,
> >>
> >> With your patch OR with my xfstest patch that adds "sync" I did not yet see
> >> another problem of garbage fs after _log_writes_remove.
> >>
> >> I did however, encounter this error (failure to verify read data during fsx)
> >> from scratch/log-writes device (see attached full log).
> >>
> >> I will keep running the test to collect more information.
> >>
> >
> > That failure I'll lay at the feet of whatever fs you are testing ;). I'm glad
> > my patch fixed the replay problem, I'll send that up. Thanks,
> >
>
> O oh!. You are implying that xfs fails plain fsx and nobody noticed.
> That is not where I would place my bet.
>
Why not? This is one of the only tests that runs multiple threads of fsx at the
same time. dm-log-writes does nothing special with the actual target disk, it
just clones the bio's as it gets them, sets the bdev for the normal device and
sends them on their way, there's no reason to think we're breaking that.
Furthermore fsx is in normal page cache mode, not O_DIRECT, so it's more likely
something else is going wrong and not dm-log-writes, unless there is so little
memory on the system that we are constantly reading cold-cache. Thanks,
Josef
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-11-28 20:00 ` Josef Bacik
@ 2017-11-28 20:26 ` Amir Goldstein
[not found] ` <CAOQ4uxhQu-1AK71zg4Ce0cJd+xRt3Gf9zMMVb=Rs00zFuWA3hQ@mail.gmail.com>
0 siblings, 1 reply; 35+ messages in thread
From: Amir Goldstein @ 2017-11-28 20:26 UTC (permalink / raw)
To: Josef Bacik; +Cc: Josef Bacik, fstests, linux-fsdevel, Eryu Guan
On Tue, Nov 28, 2017 at 10:00 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> On Tue, Nov 28, 2017 at 09:32:59PM +0200, Amir Goldstein wrote:
>> On Tue, Nov 28, 2017 at 7:21 PM, Josef Bacik <josef@toxicpanda.com> wrote:
>> > On Tue, Nov 28, 2017 at 06:48:43PM +0200, Amir Goldstein wrote:
>> >> On Mon, Nov 27, 2017 at 5:04 PM, Josef Bacik <josef@toxicpanda.com> wrote:
>> >> > On Mon, Nov 27, 2017 at 11:56:58AM +0200, Amir Goldstein wrote:
>> >> >> On Tue, Sep 5, 2017 at 10:11 PM, Amir Goldstein <amir73il@gmail.com> wrote:
>> >> >> > Cherry-picked the test from commit 70d41e17164b
>> >> >> > in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
>> >> >> > Quoting from Josef's commit message:
>> >> >> >
>> >> >> > The test just runs some ops and exits, then finds all of the good buffers
>> >> >> > in the directory we provided and:
>> >> >> > - replays up to the mark given
>> >> >> > - mounts the file system and compares the md5sum
>> >> >> > - unmounts and fsck's to check for metadata integrity
>> >> >> >
>> >> >> > dm-log-writes will pretend to do discard and the replay-log tool will
>> >> >> > replay it properly depending on the underlying device, either by writing
>> >> >> > 0's or actually calling the discard ioctl, so I've enabled discard in the
>> >> >> > test for maximum fun.
>> >> >> >
>> >> >> > [Amir:]
>> >> >> > - Removed unneeded _test_falloc_support dynamic FSX_OPTS
>> >> >> > - Fold repetitions into for loops
>> >> >> > - Added place holders for using constant random seeds
>> >> >> > - Add pre umount checkpint
>> >> >> > - Add test to new 'replay' group
>> >> >> > - Address review comments by Eryu Guan
>> >> >> >
>> >> >> > Cc: Josef Bacik <jbacik@fb.com>
>> >> >> > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
>> >> >>
>> >> >>
>> >> >> Josef,
>> >> >>
>> >> >> As you know, this test is now merged to xfstest as generic/455.
>> >> >> I have been running the test for a while on xfs and it occasionally
>> >> >> reports inconsistencies which I try to investigate.
>> >> >>
>> >> >> In some of the reports, it appears that dm-log-writes may be exhibiting
>> >> >> a reliability issue (see below).
>> >> >>
>> >> >
>> >> > It's not a reliability issue, its a caching issue. dm-log-writes is just
>> >> > issuing bio's to the log device, and our destructor waits for all pending io
>> >> > blocks to complete before exiting, so unless I've missed how dm is destroying
>> >> > devices everything should be on disk.
>> >> >
>> >> > However since we replay in userspace we are going through the blockdevice's
>> >> > pagecache, so we could have stale pages left in place which is screwing us up.
>> >> > Will you try this patch and see if it fixes the problem? Thanks,
>> >> >
>> >> > Josef
>> >> >
>> >> >
>> >> > diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
>> >> > index 8b80a9ce9ea9..1c502930af5e 100644
>> >> > --- a/drivers/md/dm-log-writes.c
>> >> > +++ b/drivers/md/dm-log-writes.c
>> >> > @@ -545,6 +545,8 @@ static void log_writes_dtr(struct dm_target *ti)
>> >> > !atomic_read(&lc->pending_blocks));
>> >> > kthread_stop(lc->log_kthread);
>> >> >
>> >> > + invalidate_bdev(lc->logdev->bdev);
>> >> > + invalidate_bdev(lc->dev->bdev);
>> >> > WARN_ON(!list_empty(&lc->logging_blocks));
>> >> > WARN_ON(!list_empty(&lc->unflushed_blocks));
>> >> > dm_put_device(ti, lc->dev);
>> >>
>> >> Josef,
>> >>
>> >> With your patch OR with my xfstest patch that adds "sync" I did not yet see
>> >> another problem of garbage fs after _log_writes_remove.
>> >>
>> >> I did however, encounter this error (failure to verify read data during fsx)
>> >> from scratch/log-writes device (see attached full log).
>> >>
>> >> I will keep running the test to collect more information.
>> >>
>> >
>> > That failure I'll lay at the feet of whatever fs you are testing ;). I'm glad
>> > my patch fixed the replay problem, I'll send that up. Thanks,
>> >
>>
>> O oh!. You are implying that xfs fails plain fsx and nobody noticed.
>> That is not where I would place my bet.
>>
>
> Why not? This is one of the only tests that runs multiple threads of fsx at the
> same time. dm-log-writes does nothing special with the actual target disk, it
> just clones the bio's as it gets them, sets the bdev for the normal device and
> sends them on their way, there's no reason to think we're breaking that.
> Furthermore fsx is in normal page cache mode, not O_DIRECT, so it's more likely
> something else is going wrong and not dm-log-writes, unless there is so little
> memory on the system that we are constantly reading cold-cache. Thanks,
>
Very well, I'll run the same test on bare xfs for comparison.
Amir.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
[not found] ` <CAOQ4uxhQu-1AK71zg4Ce0cJd+xRt3Gf9zMMVb=Rs00zFuWA3hQ@mail.gmail.com>
@ 2017-11-28 22:33 ` Darrick J. Wong
2017-11-29 3:33 ` Amir Goldstein
0 siblings, 1 reply; 35+ messages in thread
From: Darrick J. Wong @ 2017-11-28 22:33 UTC (permalink / raw)
To: Amir Goldstein
Cc: Josef Bacik, fstests, linux-fsdevel, Eryu Guan, linux-xfs, Josef Bacik
On Tue, Nov 28, 2017 at 11:17:01PM +0200, Amir Goldstein wrote:
> On Tue, Nov 28, 2017 at 10:26 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> > On Tue, Nov 28, 2017 at 10:00 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> >> On Tue, Nov 28, 2017 at 09:32:59PM +0200, Amir Goldstein wrote:
> >>> On Tue, Nov 28, 2017 at 7:21 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> >>> > On Tue, Nov 28, 2017 at 06:48:43PM +0200, Amir Goldstein wrote:
> >>> >> On Mon, Nov 27, 2017 at 5:04 PM, Josef Bacik <josef@toxicpanda.com> wrote:
> >>> >> > On Mon, Nov 27, 2017 at 11:56:58AM +0200, Amir Goldstein wrote:
> >>> >> >> On Tue, Sep 5, 2017 at 10:11 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> >>> >> >> > Cherry-picked the test from commit 70d41e17164b
> >>> >> >> > in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
> >>> >> >> > Quoting from Josef's commit message:
> >>> >> >> >
> >>> >> >> > The test just runs some ops and exits, then finds all of the good buffers
> >>> >> >> > in the directory we provided and:
> >>> >> >> > - replays up to the mark given
> >>> >> >> > - mounts the file system and compares the md5sum
> >>> >> >> > - unmounts and fsck's to check for metadata integrity
> >>> >> >> >
> >>> >> >> > dm-log-writes will pretend to do discard and the replay-log tool will
> >>> >> >> > replay it properly depending on the underlying device, either by writing
> >>> >> >> > 0's or actually calling the discard ioctl, so I've enabled discard in the
> >>> >> >> > test for maximum fun.
> >>> >> >> >
> >>> >> >> > [Amir:]
> >>> >> >> > - Removed unneeded _test_falloc_support dynamic FSX_OPTS
> >>> >> >> > - Fold repetitions into for loops
> >>> >> >> > - Added place holders for using constant random seeds
> >>> >> >> > - Add pre umount checkpint
> >>> >> >> > - Add test to new 'replay' group
> >>> >> >> > - Address review comments by Eryu Guan
> >>> >> >> >
> >>> >> >> > Cc: Josef Bacik <jbacik@fb.com>
> >>> >> >> > Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> >>> >> >>
> >>> >> >>
> >>> >> >> Josef,
> >>> >> >>
> >>> >> >> As you know, this test is now merged to xfstest as generic/455.
> >>> >> >> I have been running the test for a while on xfs and it occasionally
> >>> >> >> reports inconsistencies which I try to investigate.
> >>> >> >>
> >>> >> >> In some of the reports, it appears that dm-log-writes may be exhibiting
> >>> >> >> a reliability issue (see below).
> >>> >> >>
> >>> >> >
> >>> >> > It's not a reliability issue, its a caching issue. dm-log-writes is just
> >>> >> > issuing bio's to the log device, and our destructor waits for all pending io
> >>> >> > blocks to complete before exiting, so unless I've missed how dm is destroying
> >>> >> > devices everything should be on disk.
> >>> >> >
> >>> >> > However since we replay in userspace we are going through the blockdevice's
> >>> >> > pagecache, so we could have stale pages left in place which is screwing us up.
> >>> >> > Will you try this patch and see if it fixes the problem? Thanks,
> >>> >> >
> >>> >> > Josef
> >>> >> >
> >>> >> >
> >>> >> > diff --git a/drivers/md/dm-log-writes.c b/drivers/md/dm-log-writes.c
> >>> >> > index 8b80a9ce9ea9..1c502930af5e 100644
> >>> >> > --- a/drivers/md/dm-log-writes.c
> >>> >> > +++ b/drivers/md/dm-log-writes.c
> >>> >> > @@ -545,6 +545,8 @@ static void log_writes_dtr(struct dm_target *ti)
> >>> >> > !atomic_read(&lc->pending_blocks));
> >>> >> > kthread_stop(lc->log_kthread);
> >>> >> >
> >>> >> > + invalidate_bdev(lc->logdev->bdev);
> >>> >> > + invalidate_bdev(lc->dev->bdev);
> >>> >> > WARN_ON(!list_empty(&lc->logging_blocks));
> >>> >> > WARN_ON(!list_empty(&lc->unflushed_blocks));
> >>> >> > dm_put_device(ti, lc->dev);
> >>> >>
> >>> >> Josef,
> >>> >>
> >>> >> With your patch OR with my xfstest patch that adds "sync" I did not yet see
> >>> >> another problem of garbage fs after _log_writes_remove.
> >>> >>
> >>> >> I did however, encounter this error (failure to verify read data during fsx)
> >>> >> from scratch/log-writes device (see attached full log).
> >>> >>
> >>> >> I will keep running the test to collect more information.
> >>> >>
> >>> >
> >>> > That failure I'll lay at the feet of whatever fs you are testing ;). I'm glad
> >>> > my patch fixed the replay problem, I'll send that up. Thanks,
> >>> >
> >>>
> >>> O oh!. You are implying that xfs fails plain fsx and nobody noticed.
> >>> That is not where I would place my bet.
> >>>
> >>
> >> Why not? This is one of the only tests that runs multiple threads of fsx at the
> >> same time. dm-log-writes does nothing special with the actual target disk, it
> >> just clones the bio's as it gets them, sets the bdev for the normal device and
> >> sends them on their way, there's no reason to think we're breaking that.
> >> Furthermore fsx is in normal page cache mode, not O_DIRECT, so it's more likely
> >> something else is going wrong and not dm-log-writes, unless there is so little
> >> memory on the system that we are constantly reading cold-cache. Thanks,
> >>
> >
> > Very well, I'll run the same test on bare xfs for comparison.
> >
>
> Well, what do you know.
> xfs crashed and burned after 18 rounds of 455 on test partition.
> Same check fs error I blamed dm-log-writes for.
>
> Darrick,
>
> Attached the modified 455 test which runs fsx on test partition
> and full log (+ dmesg with nothing of interest AFAICT).
> xfs code is 4.14.0-rc8 + your patch
> "xfs: log recovery should replay deferred ops in order"
>
> The test was run on a 100G partition on a spinning disk.
>
> Let me know what you think.
I didn't even get /that/ far; with 4.15-rc1 and a vanilla g/455 I see
immediately:
[18395.236285] run fstests generic/457 at 2017-11-28 14:15:49
[18395.561112] XFS (sdf): Unmounting Filesystem
[18395.752713] XFS (sdf): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
[18395.753469] XFS (sdf): EXPERIMENTAL reflink feature enabled. Use at your own risk!
[18395.754259] XFS (sdf): Mounting V5 Filesystem
[18395.757525] XFS (sdf): Ending clean mount
[18395.796992] XFS (sdf): Unmounting Filesystem
[18395.949561] XFS (dm-0): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
[18395.950316] XFS (dm-0): EXPERIMENTAL reflink feature enabled. Use at your own risk!
[18395.951103] XFS (dm-0): Mounting V5 Filesystem
[18395.954946] XFS (dm-0): Ending clean mount
[18396.132701] XFS (dm-0): Unmounting Filesystem
[18396.309939] XFS (sdf): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
[18396.310913] XFS (sdf): EXPERIMENTAL reflink feature enabled. Use at your own risk!
[18396.311798] XFS (sdf): Mounting V5 Filesystem
[18396.330304] XFS (sdf): Starting recovery (logdev: internal)
[18396.334996] XFS (sdf): Metadata corruption detected at xfs_inode_buf_verify+0xc4/0x330 [xfs], xfs_inode block 0x80
[18396.336389] XFS (sdf): Unmount and run xfs_repair
[18396.336945] XFS (sdf): First 64 bytes of corrupted metadata buffer:
[18396.337679] ffffc90004080000: 00 4e 41 ed 03 01 00 00 00 00 00 00 00 00 00 00 .NA.............
[18396.338647] ffffc90004080010: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 ................
[18396.339576] ffffc90004080020: 00 00 00 00 00 00 00 00 5a 1d e0 16 0b 69 8a 10 ........Z....i..
[18396.340341] ffffc90004080030: 5a 1d e0 16 0b 69 8a 10 00 00 00 00 00 00 00 06 Z....i..........
[18396.341091] XFS (sdf): bad inode magic/vsn daddr 128 #0 (magic=4e)
[18396.341754] XFS (sdf): metadata I/O error: block 0x80 ("xlog_recover_do..(read#2)") error 117 numblks 32
[18396.343131] XFS (sdf): log mount/recovery failed: error -117
[18396.343852] XFS (sdf): log mount failed
Off by one in the inode verifier... am I supposed to have Josef's patch?
Also, fwiw I don't see any test failures (or interesting output) with
your modified 455:
[19344.527290] run fstests generic/455 at 2017-11-28 14:31:38
[19344.948061] XFS (pmem4): Unmounting Filesystem
[19345.098299] XFS (dm-0): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
[19345.099328] XFS (dm-0): EXPERIMENTAL reflink feature enabled. Use at your own risk!
[19345.100427] XFS (dm-0): Mounting V5 Filesystem
[19345.105724] XFS (dm-0): Ending clean mount
[19345.476248] XFS (dm-0): Unmounting Filesystem
[19346.055840] XFS (pmem4): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
[19346.057092] XFS (pmem4): EXPERIMENTAL reflink feature enabled. Use at your own risk!
[19346.057883] XFS (pmem4): Mounting V5 Filesystem
[19346.059640] XFS (pmem4): Ending clean mount
[19346.084019] XFS (pmem4): Unmounting Filesystem
[19346.492297] XFS (pmem4): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
[19346.492950] XFS (pmem4): EXPERIMENTAL reflink feature enabled. Use at your own risk!
[19346.493865] XFS (pmem4): Mounting V5 Filesystem
[19346.495732] XFS (pmem4): Ending clean mount
[19346.556181] XFS (pmem4): Unmounting Filesystem
[19346.780951] XFS (pmem3): Unmounting Filesystem
[19346.857051] XFS (pmem3): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
[19346.858352] XFS (pmem3): EXPERIMENTAL reflink feature enabled. Use at your own risk!
[19346.859053] XFS (pmem3): Mounting V5 Filesystem
[19346.861319] XFS (pmem3): Ending clean mount
[19346.940162] XFS (pmem3): Unmounting Filesystem
[19346.980932] XFS (pmem3): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
[19346.981640] XFS (pmem3): EXPERIMENTAL reflink feature enabled. Use at your own risk!
[19346.982359] XFS (pmem3): Mounting V5 Filesystem
[19346.984765] XFS (pmem3): Ending clean mount
[19347.204449] XFS (pmem3): Unmounting Filesystem
I think it's supposed to cycle more than that, right?
--D
>
> Amir.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
2017-11-28 22:33 ` Darrick J. Wong
@ 2017-11-29 3:33 ` Amir Goldstein
[not found] ` <CAOQ4uxhXWxkre7L7RDvpH8E4cwsHGZzVHKmCpBESfTUZhmQpUg@mail.gmail.com>
0 siblings, 1 reply; 35+ messages in thread
From: Amir Goldstein @ 2017-11-29 3:33 UTC (permalink / raw)
To: Darrick J. Wong
Cc: Josef Bacik, fstests, linux-fsdevel, Eryu Guan, linux-xfs, Josef Bacik
On Wed, Nov 29, 2017 at 12:33 AM, Darrick J. Wong
<darrick.wong@oracle.com> wrote:
> On Tue, Nov 28, 2017 at 11:17:01PM +0200, Amir Goldstein wrote:
[...]
>> Well, what do you know.
>> xfs crashed and burned after 18 rounds of 455 on test partition.
>> Same check fs error I blamed dm-log-writes for.
>>
>> Darrick,
>>
>> Attached the modified 455 test which runs fsx on test partition
>> and full log (+ dmesg with nothing of interest AFAICT).
>> xfs code is 4.14.0-rc8 + your patch
>> "xfs: log recovery should replay deferred ops in order"
>>
>> The test was run on a 100G partition on a spinning disk.
>>
>> Let me know what you think.
>
> I didn't even get /that/ far; with 4.15-rc1 and a vanilla g/455 I see
> immediately:
You mean vanilla g/457?
>
> [18395.236285] run fstests generic/457 at 2017-11-28 14:15:49
> [18395.561112] XFS (sdf): Unmounting Filesystem
> [18395.752713] XFS (sdf): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
> [18395.753469] XFS (sdf): EXPERIMENTAL reflink feature enabled. Use at your own risk!
> [18395.754259] XFS (sdf): Mounting V5 Filesystem
> [18395.757525] XFS (sdf): Ending clean mount
> [18395.796992] XFS (sdf): Unmounting Filesystem
> [18395.949561] XFS (dm-0): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
> [18395.950316] XFS (dm-0): EXPERIMENTAL reflink feature enabled. Use at your own risk!
> [18395.951103] XFS (dm-0): Mounting V5 Filesystem
> [18395.954946] XFS (dm-0): Ending clean mount
> [18396.132701] XFS (dm-0): Unmounting Filesystem
> [18396.309939] XFS (sdf): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
> [18396.310913] XFS (sdf): EXPERIMENTAL reflink feature enabled. Use at your own risk!
> [18396.311798] XFS (sdf): Mounting V5 Filesystem
> [18396.330304] XFS (sdf): Starting recovery (logdev: internal)
> [18396.334996] XFS (sdf): Metadata corruption detected at xfs_inode_buf_verify+0xc4/0x330 [xfs], xfs_inode block 0x80
> [18396.336389] XFS (sdf): Unmount and run xfs_repair
> [18396.336945] XFS (sdf): First 64 bytes of corrupted metadata buffer:
> [18396.337679] ffffc90004080000: 00 4e 41 ed 03 01 00 00 00 00 00 00 00 00 00 00 .NA.............
> [18396.338647] ffffc90004080010: 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 ................
> [18396.339576] ffffc90004080020: 00 00 00 00 00 00 00 00 5a 1d e0 16 0b 69 8a 10 ........Z....i..
> [18396.340341] ffffc90004080030: 5a 1d e0 16 0b 69 8a 10 00 00 00 00 00 00 00 06 Z....i..........
> [18396.341091] XFS (sdf): bad inode magic/vsn daddr 128 #0 (magic=4e)
> [18396.341754] XFS (sdf): metadata I/O error: block 0x80 ("xlog_recover_do..(read#2)") error 117 numblks 32
> [18396.343131] XFS (sdf): log mount/recovery failed: error -117
> [18396.343852] XFS (sdf): log mount failed
>
> Off by one in the inode verifier... am I supposed to have Josef's patch?
>
I donno. Josef's patch makes sense, but it only matters for cleaning
logdev page cache
from stale data in between 2 different runs
> Also, fwiw I don't see any test failures (or interesting output) with
> your modified 455:
>
> [19344.527290] run fstests generic/455 at 2017-11-28 14:31:38
> [19344.948061] XFS (pmem4): Unmounting Filesystem
> [19345.098299] XFS (dm-0): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
> [19345.099328] XFS (dm-0): EXPERIMENTAL reflink feature enabled. Use at your own risk!
> [19345.100427] XFS (dm-0): Mounting V5 Filesystem
> [19345.105724] XFS (dm-0): Ending clean mount
> [19345.476248] XFS (dm-0): Unmounting Filesystem
> [19346.055840] XFS (pmem4): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
> [19346.057092] XFS (pmem4): EXPERIMENTAL reflink feature enabled. Use at your own risk!
> [19346.057883] XFS (pmem4): Mounting V5 Filesystem
> [19346.059640] XFS (pmem4): Ending clean mount
> [19346.084019] XFS (pmem4): Unmounting Filesystem
> [19346.492297] XFS (pmem4): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
> [19346.492950] XFS (pmem4): EXPERIMENTAL reflink feature enabled. Use at your own risk!
> [19346.493865] XFS (pmem4): Mounting V5 Filesystem
> [19346.495732] XFS (pmem4): Ending clean mount
> [19346.556181] XFS (pmem4): Unmounting Filesystem
> [19346.780951] XFS (pmem3): Unmounting Filesystem
> [19346.857051] XFS (pmem3): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
> [19346.858352] XFS (pmem3): EXPERIMENTAL reflink feature enabled. Use at your own risk!
> [19346.859053] XFS (pmem3): Mounting V5 Filesystem
> [19346.861319] XFS (pmem3): Ending clean mount
> [19346.940162] XFS (pmem3): Unmounting Filesystem
> [19346.980932] XFS (pmem3): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
> [19346.981640] XFS (pmem3): EXPERIMENTAL reflink feature enabled. Use at your own risk!
> [19346.982359] XFS (pmem3): Mounting V5 Filesystem
> [19346.984765] XFS (pmem3): Ending clean mount
> [19347.204449] XFS (pmem3): Unmounting Filesystem
>
> I think it's supposed to cycle more than that, right?
>
Usually yes. This time around I got it after 18 rounds, but sometimes
it takes more than 100 rounds.
Modified 455 is much faster because it skips all the replays, so I can
expedite the testing.
Also, since most of the failures I have seen were on the slow spinning
disk, not sure you would get
them on pmem device.
I'll continue to bisect.
Amir.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes
[not found] ` <CAOQ4uxjBQ9ZzPe9GKCRYCjNFv3jP8NMAVQDb=LiNqNcEeRp47w@mail.gmail.com>
@ 2017-12-04 20:53 ` Darrick J. Wong
0 siblings, 0 replies; 35+ messages in thread
From: Darrick J. Wong @ 2017-12-04 20:53 UTC (permalink / raw)
To: Amir Goldstein
Cc: Josef Bacik, fstests, linux-fsdevel, Eryu Guan, linux-xfs, Josef Bacik
On Mon, Dec 04, 2017 at 10:17:30PM +0200, Amir Goldstein wrote:
> On Thu, Nov 30, 2017 at 10:28 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> > On Wed, Nov 29, 2017 at 5:33 AM, Amir Goldstein <amir73il@gmail.com> wrote:
> [...]
> > So far I was able to determine that your patch
> > "xfs: log recovery should replay deferred ops in order" is NOT the
> > cause of the problem.
> > This took some time, because at one point it took me 23 hr to get to
> > the dirty log
> > in test partition with modified 455 (no dm-log-writes).
> >
> > Attached metadump of corrupt test partition.
> > The xfs code this test was running with is v4.14-rc8.
> > I did not try to bisect any further because of the time it takes per commit.
> >
> > Let me know if you need any other info or if you want me to run the test
> > on my setup for specific patch and/or bisection points.
> >
>
> I figured out what was going on in my test setup.
> The answer was in the attached dmesg, but I overlooked it:
>
> [33816.533286] ata3.00: failed command: FLUSH CACHE EXT
> [33816.533294] ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 21
> res 40/00:00:20:44:ba/00:00:0c:00:00/40 Emask
> 0x10 (ATA bus error)
> [33816.533300] ata3.00: status: { DRDY }
> [33816.533309] ata3: hard resetting link
>
> It appears that that test machine had a faulty SATA cable.
>
> This is probably more cruel to fs than a dm-flakey/dm-log-writes test...
Not as bad as the time when I discovered that one of my UASP bridges was
arbitrarily injecting 'USBUSBUSB' into bus transfers.
> Cable replaced. Back to sanity. Sorry for the noise.
:)
--D
> Amir.
^ permalink raw reply [flat|nested] 35+ messages in thread
end of thread, other threads:[~2017-12-04 23:08 UTC | newest]
Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-05 19:11 [PATCH v3 00/13] Crash consistency xfstest using dm-log-writes Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 01/13] fsx: add support for integrity check with dm-log-writes target Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 02/13] fsx: add optional logid prefix to log messages Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 03/13] fsx: add support for recording operations to a file Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 04/13] fsx: add support for writing constant instead of random data Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 05/13] fsx: add support for keeping existing file Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 06/13] log-writes: add replay-log program to replay dm-log-writes target Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 07/13] replay-log: add validations for corrupt log entries Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 08/13] replay-log: add support for replaying ops in target device sector range Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 09/13] fstests: add support for working with dm-log-writes target Amir Goldstein
2017-09-07 7:45 ` Eryu Guan
2017-09-07 7:47 ` Eryu Guan
2017-09-05 19:11 ` [PATCH v3 10/13] fstests: crash consistency fsx test using dm-log-writes Amir Goldstein
2017-09-07 7:50 ` Eryu Guan
2017-09-07 8:50 ` Amir Goldstein
2017-09-07 8:55 ` Eryu Guan
2017-09-07 10:10 ` Amir Goldstein
2017-11-27 9:56 ` Amir Goldstein
2017-11-27 14:23 ` Ashlie Martinez
2017-11-27 15:07 ` Josef Bacik
2017-11-27 15:04 ` Josef Bacik
2017-11-28 16:48 ` Amir Goldstein
2017-11-28 17:21 ` Josef Bacik
2017-11-28 19:32 ` Amir Goldstein
2017-11-28 20:00 ` Josef Bacik
2017-11-28 20:26 ` Amir Goldstein
[not found] ` <CAOQ4uxhQu-1AK71zg4Ce0cJd+xRt3Gf9zMMVb=Rs00zFuWA3hQ@mail.gmail.com>
2017-11-28 22:33 ` Darrick J. Wong
2017-11-29 3:33 ` Amir Goldstein
[not found] ` <CAOQ4uxhXWxkre7L7RDvpH8E4cwsHGZzVHKmCpBESfTUZhmQpUg@mail.gmail.com>
[not found] ` <CAOQ4uxjBQ9ZzPe9GKCRYCjNFv3jP8NMAVQDb=LiNqNcEeRp47w@mail.gmail.com>
2017-12-04 20:53 ` Darrick J. Wong
2017-09-05 19:11 ` [PATCH v3 11/13] fstests: regression test for ext4 crash consistency bug Amir Goldstein
2017-09-07 7:52 ` Eryu Guan
2017-09-05 19:11 ` [PATCH v3 12/13] fstests: crash consistency fsx test for cloned files Amir Goldstein
2017-09-05 19:11 ` [PATCH v3 13/13] fstests: regression test for xfs leftover CoW extent error Amir Goldstein
2017-09-07 7:55 ` Eryu Guan
2017-09-07 9:34 ` Amir Goldstein
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.