[RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

This test case is originally designed to expose unexpected corruption
for btrfs, where there are several reports about btrfs serious metadata
corruption after power loss.

The test case itself will trigger heavy fsstress for the fs, and use
dm-flakey to emulate power loss by dropping all later writes.

For btrfs, it should be completely fine, as long as superblock write
(FUA write) finishes atomically, since with metadata CoW, superblock
either points to old trees or new tress, the fs should be as atomic as
superblock.

For journal based filesystems, each metadata update should be journaled,
so metadata operation is as atomic as journal updates.

It does show that XFS is doing the best work among the tested
filesystems (Btrfs, XFS, ext4), no kernel nor xfs_repair problem at all.

For btrfs, although btrfs check doesn't report any problem, kernel
reports some data checksum error, which is a little unexpected as data
is CoWed by default, which should be as atomic as superblock.
(Unfortunately, still not the exact problem I'm chasing for)

For EXT4, kernel is fine, but later e2fsck reports problem, which may
indicates there is still something to be improved.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/generic/479     | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/479.out |   2 +
 tests/generic/group   |   1 +
 3 files changed, 112 insertions(+)
 create mode 100755 tests/generic/479
 create mode 100644 tests/generic/479.out

diff --git a/tests/generic/479 b/tests/generic/479
new file mode 100755
index 00000000..ab530231
--- /dev/null
+++ b/tests/generic/479
@@ -0,0 +1,109 @@
+#! /bin/bash
+# FS QA Test 479
+#
+# Test if a filesystem can survive emulated powerloss.
+#
+# No matter what the solution a filesystem uses (journal or CoW),
+# it should survive unexpected powerloss, without major metadata
+# corruption.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2018 SuSE.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	ps -e | grep fsstress > /dev/null 2>&1
+	while [ $? -eq 0 ]; do
+		$KILLALL_PROG -KILL fsstress > /dev/null 2>&1
+		wait > /dev/null 2>&1
+		ps -e | grep fsstress > /dev/null 2>&1
+	done
+	_unmount_flakey &> /dev/null
+	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+_require_command "$KILLALL_PROG" "killall"
+
+runtime=$(($TIME_FACTOR * 15))
+loops=$(($LOAD_FACTOR * 4))
+
+for i in $(seq -w $loops); do
+	echo "=== Loop $i: $(date) ===" >> $seqres.full
+
+	_scratch_mkfs >/dev/null 2>&1
+	_init_flakey
+	_mount_flakey
+
+	($FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n 1000000 \
+		-p 100 >> $seqres.full &) > /dev/null 2>&1
+
+	sleep $runtime
+
+	# Here we only want to drop all write, don't need to umount the fs
+	_load_flakey_table $FLAKEY_DROP_WRITES
+
+	ps -e | grep fsstress > /dev/null 2>&1
+	while [ $? -eq 0 ]; do
+		$KILLALL_PROG -KILL fsstress > /dev/null 2>&1
+		wait > /dev/null 2>&1
+		ps -e | grep fsstress > /dev/null 2>&1
+	done
+
+	_unmount_flakey
+	_cleanup_flakey
+
+	# Mount the fs to do proper log replay for journal based fs
+	# so later check won't report annoying dirty log and only
+	# report real problem.
+	_scratch_mount
+	_scratch_unmount
+
+	_check_scratch_fs
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/479.out b/tests/generic/479.out
new file mode 100644
index 00000000..290f18b3
--- /dev/null
+++ b/tests/generic/479.out
@@ -0,0 +1,2 @@
+QA output created by 479
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 1e808865..5ce3db1d 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -481,3 +481,4 @@
 476 auto rw
 477 auto quick exportfs
 478 auto quick
+479 auto
-- 
2.15.1

[PATCH 1/2] fstests: log-writes: Add support to output human readable flags

[Qu Wenruo]

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 src/log-writes/log-writes.c | 68 +++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 62 insertions(+), 6 deletions(-)

diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c
index 09391574c4d2..181dabf442de 100644
--- a/src/log-writes/log-writes.c
+++ b/src/log-writes/log-writes.c
@@ -120,6 +120,58 @@ int log_discard(struct log *log, struct log_write_entry *entry)
 	return 0;
 }
 
+#define DEFINE_LOG_FLAGS_STR_ENTRY(x)	\
+	{LOG_##x##_FLAG, #x}
+
+struct flags_to_str_entry {
+	u64 flags;
+	const char *str;
+} log_flags_table[] = {
+	DEFINE_LOG_FLAGS_STR_ENTRY(FLUSH),
+	DEFINE_LOG_FLAGS_STR_ENTRY(FUA),
+	DEFINE_LOG_FLAGS_STR_ENTRY(DISCARD),
+	DEFINE_LOG_FLAGS_STR_ENTRY(MARK)
+};
+
+#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
+#define LOG_FLAGS_BUF_SIZE	128
+/*
+ * Convert numeric flags to human readable flags.
+ * @flags:	numeric flags
+ * @buf:	output buffer for human readable string.
+ * 		must have enough space (LOG_FLAGS_BUF_SIZE) to contain all
+ * 		the string
+ */
+static void entry_flags_to_str(u64 flags, char *buf)
+{
+	int empty = 1;
+	int left_len;
+	int i;
+
+	buf[0] = '\0';
+	for (i = 0; i < ARRAY_SIZE(log_flags_table); i++) {
+		if (flags & log_flags_table[i].flags) {
+			if (!empty)
+				strncat(buf, "|", LOG_FLAGS_BUF_SIZE);
+			empty = 0;
+			strncat(buf, log_flags_table[i].str, LOG_FLAGS_BUF_SIZE);
+			flags &= ~log_flags_table[i].flags;
+		}
+	}
+	if (flags) {
+		if (!empty)
+			strncat(buf, "|", LOG_FLAGS_BUF_SIZE);
+		empty = 0;
+		left_len = LOG_FLAGS_BUF_SIZE - strnlen(buf,
+						        LOG_FLAGS_BUF_SIZE);
+		if (left_len > 0)
+			snprintf(buf + strnlen(buf, LOG_FLAGS_BUF_SIZE),
+				 left_len, "UNKNOWN.%llu", flags);
+	}
+	if (empty)
+		strncpy(buf, "NONE", LOG_FLAGS_BUF_SIZE);
+}
+
 /*
  * @log: the log we are replaying.
  * @entry: entry to be replayed.
@@ -179,6 +231,7 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
 	size_t read_size = read_data ? log->sectorsize :
 		sizeof(struct log_write_entry);
 	char *buf;
+	char flags_buf[LOG_FLAGS_BUF_SIZE];
 	ssize_t ret;
 	off_t offset;
 	int skip = 0;
@@ -210,19 +263,20 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
 		log->cur_pos += read_size;
 	}
 
+	flags = le64_to_cpu(entry->flags);
+	entry_flags_to_str(flags, flags_buf);
 	skip = log_should_skip(log, entry);
 	if (log_writes_verbose > 1 || (log_writes_verbose && !skip)) {
-		printf("%s %d@%llu: sector %llu, size %llu, flags %llu\n",
+		printf("%s %d@%llu: sector %llu, size %llu, flags %llu(%s)\n",
 		       skip ? "skipping" : "replaying",
 		       (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
 		       (unsigned long long)le64_to_cpu(entry->sector),
 		       (unsigned long long)size,
-		       (unsigned long long)le64_to_cpu(entry->flags));
+		       (unsigned long long)flags, flags_buf);
 	}
 	if (!size)
 		return 0;
 
-	flags = le64_to_cpu(entry->flags);
 	if (flags & LOG_DISCARD_FLAG)
 		return log_discard(log, entry);
 
@@ -339,6 +393,7 @@ int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
 	size_t read_size = read_data ? log->sectorsize :
 		sizeof(struct log_write_entry);
 	u64 flags;
+	char flags_buf[LOG_FLAGS_BUF_SIZE];
 	ssize_t ret;
 
 	if (log->cur_entry >= log->nr_entries)
@@ -366,14 +421,15 @@ int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
 	} else {
 		log->cur_pos += read_size;
 	}
+	flags = le64_to_cpu(entry->flags);
+	entry_flags_to_str(flags, flags_buf);
 	if (log_writes_verbose > 1)
-		printf("seek entry %d@%llu: %llu, size %llu, flags %llu\n",
+		printf("seek entry %d@%llu: %llu, size %llu, flags %llu(%s)\n",
 		       (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
 		       (unsigned long long)le64_to_cpu(entry->sector),
 		       (unsigned long long)le64_to_cpu(entry->nr_sectors),
-		       (unsigned long long)le64_to_cpu(entry->flags));
+		       (unsigned long long)flags, flags_buf);
 
-	flags = le64_to_cpu(entry->flags);
 	read_size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
 	if (!read_size || (flags & LOG_DISCARD_FLAG))
 		return 0;
-- 
2.16.2

[PATCH 2/2] fstests: log-writes: Add support for METADATA flag

[Qu Wenruo]

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 src/log-writes/log-writes.c | 3 ++-
 src/log-writes/log-writes.h | 9 +++++----
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c
index 181dabf442de..168fea905ef3 100644
--- a/src/log-writes/log-writes.c
+++ b/src/log-writes/log-writes.c
@@ -130,7 +130,8 @@ struct flags_to_str_entry {
 	DEFINE_LOG_FLAGS_STR_ENTRY(FLUSH),
 	DEFINE_LOG_FLAGS_STR_ENTRY(FUA),
 	DEFINE_LOG_FLAGS_STR_ENTRY(DISCARD),
-	DEFINE_LOG_FLAGS_STR_ENTRY(MARK)
+	DEFINE_LOG_FLAGS_STR_ENTRY(MARK),
+	DEFINE_LOG_FLAGS_STR_ENTRY(METADATA)
 };
 
 #define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
diff --git a/src/log-writes/log-writes.h b/src/log-writes/log-writes.h
index 35ca35838aac..75fb8ac0bf79 100644
--- a/src/log-writes/log-writes.h
+++ b/src/log-writes/log-writes.h
@@ -20,10 +20,11 @@ typedef __u32 u32;
 /*
  * Constants copied from kernel file drivers/md/dm-log-writes.c
  */
-#define LOG_FLUSH_FLAG (1 << 0)
-#define LOG_FUA_FLAG (1 << 1)
-#define LOG_DISCARD_FLAG (1 << 2)
-#define LOG_MARK_FLAG (1 << 3)
+#define LOG_FLUSH_FLAG		(1 << 0)
+#define LOG_FUA_FLAG		(1 << 1)
+#define LOG_DISCARD_FLAG	(1 << 2)
+#define LOG_MARK_FLAG		(1 << 3)
+#define LOG_METADATA_FLAG	(1 << 4)
 
 #define WRITE_LOG_VERSION 1
 #define WRITE_LOG_MAGIC 0x6a736677736872
-- 
2.16.2

Re: [PATCH 1/2] fstests: log-writes: Add support to output human readable flags

[Amir Goldstein]

On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>  src/log-writes/log-writes.c | 68 +++++++++++++++++++++++++++++++++++++++++----
>  1 file changed, 62 insertions(+), 6 deletions(-)
>
> diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c
> index 09391574c4d2..181dabf442de 100644
> --- a/src/log-writes/log-writes.c
> +++ b/src/log-writes/log-writes.c
> @@ -120,6 +120,58 @@ int log_discard(struct log *log, struct log_write_entry *entry)
>         return 0;
>  }
>
> +#define DEFINE_LOG_FLAGS_STR_ENTRY(x)  \
> +       {LOG_##x##_FLAG, #x}
> +
> +struct flags_to_str_entry {
> +       u64 flags;
> +       const char *str;
> +} log_flags_table[] = {
> +       DEFINE_LOG_FLAGS_STR_ENTRY(FLUSH),
> +       DEFINE_LOG_FLAGS_STR_ENTRY(FUA),
> +       DEFINE_LOG_FLAGS_STR_ENTRY(DISCARD),
> +       DEFINE_LOG_FLAGS_STR_ENTRY(MARK)
> +};
> +
> +#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
> +#define LOG_FLAGS_BUF_SIZE     128
> +/*
> + * Convert numeric flags to human readable flags.
> + * @flags:     numeric flags
> + * @buf:       output buffer for human readable string.
> + *             must have enough space (LOG_FLAGS_BUF_SIZE) to contain all
> + *             the string
> + */
> +static void entry_flags_to_str(u64 flags, char *buf)
> +{
> +       int empty = 1;
> +       int left_len;
> +       int i;
> +
> +       buf[0] = '\0';
> +       for (i = 0; i < ARRAY_SIZE(log_flags_table); i++) {
> +               if (flags & log_flags_table[i].flags) {
> +                       if (!empty)
> +                               strncat(buf, "|", LOG_FLAGS_BUF_SIZE);
> +                       empty = 0;
> +                       strncat(buf, log_flags_table[i].str, LOG_FLAGS_BUF_SIZE);
> +                       flags &= ~log_flags_table[i].flags;
> +               }
> +       }
> +       if (flags) {
> +               if (!empty)
> +                       strncat(buf, "|", LOG_FLAGS_BUF_SIZE);
> +               empty = 0;
> +               left_len = LOG_FLAGS_BUF_SIZE - strnlen(buf,
> +                                                       LOG_FLAGS_BUF_SIZE);
> +               if (left_len > 0)
> +                       snprintf(buf + strnlen(buf, LOG_FLAGS_BUF_SIZE),
> +                                left_len, "UNKNOWN.%llu", flags);

I think it is best if all numeric prints of flags used %llx
below as well.

> +       }
> +       if (empty)
> +               strncpy(buf, "NONE", LOG_FLAGS_BUF_SIZE);
> +}
> +
>  /*
>   * @log: the log we are replaying.
>   * @entry: entry to be replayed.
> @@ -179,6 +231,7 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
>         size_t read_size = read_data ? log->sectorsize :
>                 sizeof(struct log_write_entry);
>         char *buf;
> +       char flags_buf[LOG_FLAGS_BUF_SIZE];
>         ssize_t ret;
>         off_t offset;
>         int skip = 0;
> @@ -210,19 +263,20 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
>                 log->cur_pos += read_size;
>         }
>
> +       flags = le64_to_cpu(entry->flags);
> +       entry_flags_to_str(flags, flags_buf);
>         skip = log_should_skip(log, entry);
>         if (log_writes_verbose > 1 || (log_writes_verbose && !skip)) {
> -               printf("%s %d@%llu: sector %llu, size %llu, flags %llu\n",
> +               printf("%s %d@%llu: sector %llu, size %llu, flags %llu(%s)\n",
>                        skip ? "skipping" : "replaying",
>                        (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
>                        (unsigned long long)le64_to_cpu(entry->sector),
>                        (unsigned long long)size,
> -                      (unsigned long long)le64_to_cpu(entry->flags));
> +                      (unsigned long long)flags, flags_buf);
>         }
>         if (!size)
>                 return 0;
>
> -       flags = le64_to_cpu(entry->flags);
>         if (flags & LOG_DISCARD_FLAG)
>                 return log_discard(log, entry);
>
> @@ -339,6 +393,7 @@ int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
>         size_t read_size = read_data ? log->sectorsize :
>                 sizeof(struct log_write_entry);
>         u64 flags;
> +       char flags_buf[LOG_FLAGS_BUF_SIZE];
>         ssize_t ret;
>
>         if (log->cur_entry >= log->nr_entries)
> @@ -366,14 +421,15 @@ int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
>         } else {
>                 log->cur_pos += read_size;
>         }
> +       flags = le64_to_cpu(entry->flags);
> +       entry_flags_to_str(flags, flags_buf);
>         if (log_writes_verbose > 1)
> -               printf("seek entry %d@%llu: %llu, size %llu, flags %llu\n",
> +               printf("seek entry %d@%llu: %llu, size %llu, flags %llu(%s)\n",
>                        (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
>                        (unsigned long long)le64_to_cpu(entry->sector),
>                        (unsigned long long)le64_to_cpu(entry->nr_sectors),
> -                      (unsigned long long)le64_to_cpu(entry->flags));
> +                      (unsigned long long)flags, flags_buf);
>
> -       flags = le64_to_cpu(entry->flags);
>         read_size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
>         if (!read_size || (flags & LOG_DISCARD_FLAG))
>                 return 0;
> --
> 2.16.2
>
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/2] fstests: log-writes: Add support to output human readable flags

[Amir Goldstein]

On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>  src/log-writes/log-writes.c | 68 +++++++++++++++++++++++++++++++++++++++++----
>  1 file changed, 62 insertions(+), 6 deletions(-)
>
> diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c
> index 09391574c4d2..181dabf442de 100644
> --- a/src/log-writes/log-writes.c
> +++ b/src/log-writes/log-writes.c
> @@ -120,6 +120,58 @@ int log_discard(struct log *log, struct log_write_entry *entry)
>         return 0;
>  }
>
> +#define DEFINE_LOG_FLAGS_STR_ENTRY(x)  \
> +       {LOG_##x##_FLAG, #x}
> +
> +struct flags_to_str_entry {
> +       u64 flags;
> +       const char *str;
> +} log_flags_table[] = {
> +       DEFINE_LOG_FLAGS_STR_ENTRY(FLUSH),
> +       DEFINE_LOG_FLAGS_STR_ENTRY(FUA),
> +       DEFINE_LOG_FLAGS_STR_ENTRY(DISCARD),
> +       DEFINE_LOG_FLAGS_STR_ENTRY(MARK)
> +};
> +
> +#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
> +#define LOG_FLAGS_BUF_SIZE     128
> +/*
> + * Convert numeric flags to human readable flags.
> + * @flags:     numeric flags
> + * @buf:       output buffer for human readable string.
> + *             must have enough space (LOG_FLAGS_BUF_SIZE) to contain all
> + *             the string
> + */
> +static void entry_flags_to_str(u64 flags, char *buf)
> +{
> +       int empty = 1;
> +       int left_len;
> +       int i;
> +
> +       buf[0] = '\0';
> +       for (i = 0; i < ARRAY_SIZE(log_flags_table); i++) {
> +               if (flags & log_flags_table[i].flags) {
> +                       if (!empty)
> +                               strncat(buf, "|", LOG_FLAGS_BUF_SIZE);
> +                       empty = 0;
> +                       strncat(buf, log_flags_table[i].str, LOG_FLAGS_BUF_SIZE);
> +                       flags &= ~log_flags_table[i].flags;
> +               }
> +       }
> +       if (flags) {
> +               if (!empty)
> +                       strncat(buf, "|", LOG_FLAGS_BUF_SIZE);
> +               empty = 0;
> +               left_len = LOG_FLAGS_BUF_SIZE - strnlen(buf,
> +                                                       LOG_FLAGS_BUF_SIZE);
> +               if (left_len > 0)
> +                       snprintf(buf + strnlen(buf, LOG_FLAGS_BUF_SIZE),
> +                                left_len, "UNKNOWN.%llu", flags);

I think it is best if all numeric prints of flags used %llx
below as well.

> +       }
> +       if (empty)
> +               strncpy(buf, "NONE", LOG_FLAGS_BUF_SIZE);
> +}
> +
>  /*
>   * @log: the log we are replaying.
>   * @entry: entry to be replayed.
> @@ -179,6 +231,7 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
>         size_t read_size = read_data ? log->sectorsize :
>                 sizeof(struct log_write_entry);
>         char *buf;
> +       char flags_buf[LOG_FLAGS_BUF_SIZE];
>         ssize_t ret;
>         off_t offset;
>         int skip = 0;
> @@ -210,19 +263,20 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
>                 log->cur_pos += read_size;
>         }
>
> +       flags = le64_to_cpu(entry->flags);
> +       entry_flags_to_str(flags, flags_buf);
>         skip = log_should_skip(log, entry);
>         if (log_writes_verbose > 1 || (log_writes_verbose && !skip)) {
> -               printf("%s %d@%llu: sector %llu, size %llu, flags %llu\n",
> +               printf("%s %d@%llu: sector %llu, size %llu, flags %llu(%s)\n",
>                        skip ? "skipping" : "replaying",
>                        (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
>                        (unsigned long long)le64_to_cpu(entry->sector),
>                        (unsigned long long)size,
> -                      (unsigned long long)le64_to_cpu(entry->flags));
> +                      (unsigned long long)flags, flags_buf);
>         }
>         if (!size)
>                 return 0;
>
> -       flags = le64_to_cpu(entry->flags);
>         if (flags & LOG_DISCARD_FLAG)
>                 return log_discard(log, entry);
>
> @@ -339,6 +393,7 @@ int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
>         size_t read_size = read_data ? log->sectorsize :
>                 sizeof(struct log_write_entry);
>         u64 flags;
> +       char flags_buf[LOG_FLAGS_BUF_SIZE];
>         ssize_t ret;
>
>         if (log->cur_entry >= log->nr_entries)
> @@ -366,14 +421,15 @@ int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
>         } else {
>                 log->cur_pos += read_size;
>         }
> +       flags = le64_to_cpu(entry->flags);
> +       entry_flags_to_str(flags, flags_buf);
>         if (log_writes_verbose > 1)
> -               printf("seek entry %d@%llu: %llu, size %llu, flags %llu\n",
> +               printf("seek entry %d@%llu: %llu, size %llu, flags %llu(%s)\n",
>                        (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
>                        (unsigned long long)le64_to_cpu(entry->sector),
>                        (unsigned long long)le64_to_cpu(entry->nr_sectors),
> -                      (unsigned long long)le64_to_cpu(entry->flags));
> +                      (unsigned long long)flags, flags_buf);
>
> -       flags = le64_to_cpu(entry->flags);
>         read_size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
>         if (!read_size || (flags & LOG_DISCARD_FLAG))
>                 return 0;
> --
> 2.16.2
>
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Amir Goldstein]

On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
> This test case is originally designed to expose unexpected corruption
> for btrfs, where there are several reports about btrfs serious metadata
> corruption after power loss.
>
> The test case itself will trigger heavy fsstress for the fs, and use
> dm-flakey to emulate power loss by dropping all later writes.

So you are re-posting the test with dm-flakey or converting it to
dm-log-writes??

>
> For btrfs, it should be completely fine, as long as superblock write
> (FUA write) finishes atomically, since with metadata CoW, superblock
> either points to old trees or new tress, the fs should be as atomic as
> superblock.
>
> For journal based filesystems, each metadata update should be journaled,
> so metadata operation is as atomic as journal updates.
>
> It does show that XFS is doing the best work among the tested
> filesystems (Btrfs, XFS, ext4), no kernel nor xfs_repair problem at all.
>
> For btrfs, although btrfs check doesn't report any problem, kernel
> reports some data checksum error, which is a little unexpected as data
> is CoWed by default, which should be as atomic as superblock.
> (Unfortunately, still not the exact problem I'm chasing for)
>
> For EXT4, kernel is fine, but later e2fsck reports problem, which may
> indicates there is still something to be improved.
>
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>  tests/generic/479     | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/479.out |   2 +
>  tests/generic/group   |   1 +
>  3 files changed, 112 insertions(+)
>  create mode 100755 tests/generic/479
>  create mode 100644 tests/generic/479.out
>
> diff --git a/tests/generic/479 b/tests/generic/479
> new file mode 100755
> index 00000000..ab530231
> --- /dev/null
> +++ b/tests/generic/479
> @@ -0,0 +1,109 @@
> +#! /bin/bash
> +# FS QA Test 479
> +#
> +# Test if a filesystem can survive emulated powerloss.
> +#
> +# No matter what the solution a filesystem uses (journal or CoW),
> +# it should survive unexpected powerloss, without major metadata
> +# corruption.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2018 SuSE.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1       # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +       ps -e | grep fsstress > /dev/null 2>&1
> +       while [ $? -eq 0 ]; do
> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
> +               wait > /dev/null 2>&1
> +               ps -e | grep fsstress > /dev/null 2>&1
> +       done
> +       _unmount_flakey &> /dev/null
> +       _cleanup_flakey
> +       cd /
> +       rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmflakey
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch
> +_require_dm_target flakey
> +_require_command "$KILLALL_PROG" "killall"
> +
> +runtime=$(($TIME_FACTOR * 15))
> +loops=$(($LOAD_FACTOR * 4))
> +
> +for i in $(seq -w $loops); do
> +       echo "=== Loop $i: $(date) ===" >> $seqres.full
> +
> +       _scratch_mkfs >/dev/null 2>&1
> +       _init_flakey
> +       _mount_flakey
> +
> +       ($FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n 1000000 \
> +               -p 100 >> $seqres.full &) > /dev/null 2>&1
> +
> +       sleep $runtime
> +
> +       # Here we only want to drop all write, don't need to umount the fs
> +       _load_flakey_table $FLAKEY_DROP_WRITES
> +
> +       ps -e | grep fsstress > /dev/null 2>&1
> +       while [ $? -eq 0 ]; do
> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
> +               wait > /dev/null 2>&1
> +               ps -e | grep fsstress > /dev/null 2>&1
> +       done
> +
> +       _unmount_flakey
> +       _cleanup_flakey
> +
> +       # Mount the fs to do proper log replay for journal based fs
> +       # so later check won't report annoying dirty log and only
> +       # report real problem.
> +       _scratch_mount
> +       _scratch_unmount
> +
> +       _check_scratch_fs
> +done
> +
> +echo "Silence is golden"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/479.out b/tests/generic/479.out
> new file mode 100644
> index 00000000..290f18b3
> --- /dev/null
> +++ b/tests/generic/479.out
> @@ -0,0 +1,2 @@
> +QA output created by 479
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index 1e808865..5ce3db1d 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -481,3 +481,4 @@
>  476 auto rw
>  477 auto quick exportfs
>  478 auto quick
> +479 auto

+ stress

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Amir Goldstein]

On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
> This test case is originally designed to expose unexpected corruption
> for btrfs, where there are several reports about btrfs serious metadata
> corruption after power loss.
>
> The test case itself will trigger heavy fsstress for the fs, and use
> dm-flakey to emulate power loss by dropping all later writes.

So you are re-posting the test with dm-flakey or converting it to
dm-log-writes??

>
> For btrfs, it should be completely fine, as long as superblock write
> (FUA write) finishes atomically, since with metadata CoW, superblock
> either points to old trees or new tress, the fs should be as atomic as
> superblock.
>
> For journal based filesystems, each metadata update should be journaled,
> so metadata operation is as atomic as journal updates.
>
> It does show that XFS is doing the best work among the tested
> filesystems (Btrfs, XFS, ext4), no kernel nor xfs_repair problem at all.
>
> For btrfs, although btrfs check doesn't report any problem, kernel
> reports some data checksum error, which is a little unexpected as data
> is CoWed by default, which should be as atomic as superblock.
> (Unfortunately, still not the exact problem I'm chasing for)
>
> For EXT4, kernel is fine, but later e2fsck reports problem, which may
> indicates there is still something to be improved.
>
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>  tests/generic/479     | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/479.out |   2 +
>  tests/generic/group   |   1 +
>  3 files changed, 112 insertions(+)
>  create mode 100755 tests/generic/479
>  create mode 100644 tests/generic/479.out
>
> diff --git a/tests/generic/479 b/tests/generic/479
> new file mode 100755
> index 00000000..ab530231
> --- /dev/null
> +++ b/tests/generic/479
> @@ -0,0 +1,109 @@
> +#! /bin/bash
> +# FS QA Test 479
> +#
> +# Test if a filesystem can survive emulated powerloss.
> +#
> +# No matter what the solution a filesystem uses (journal or CoW),
> +# it should survive unexpected powerloss, without major metadata
> +# corruption.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2018 SuSE.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1       # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +       ps -e | grep fsstress > /dev/null 2>&1
> +       while [ $? -eq 0 ]; do
> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
> +               wait > /dev/null 2>&1
> +               ps -e | grep fsstress > /dev/null 2>&1
> +       done
> +       _unmount_flakey &> /dev/null
> +       _cleanup_flakey
> +       cd /
> +       rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmflakey
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch
> +_require_dm_target flakey
> +_require_command "$KILLALL_PROG" "killall"
> +
> +runtime=$(($TIME_FACTOR * 15))
> +loops=$(($LOAD_FACTOR * 4))
> +
> +for i in $(seq -w $loops); do
> +       echo "=== Loop $i: $(date) ===" >> $seqres.full
> +
> +       _scratch_mkfs >/dev/null 2>&1
> +       _init_flakey
> +       _mount_flakey
> +
> +       ($FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n 1000000 \
> +               -p 100 >> $seqres.full &) > /dev/null 2>&1
> +
> +       sleep $runtime
> +
> +       # Here we only want to drop all write, don't need to umount the fs
> +       _load_flakey_table $FLAKEY_DROP_WRITES
> +
> +       ps -e | grep fsstress > /dev/null 2>&1
> +       while [ $? -eq 0 ]; do
> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
> +               wait > /dev/null 2>&1
> +               ps -e | grep fsstress > /dev/null 2>&1
> +       done
> +
> +       _unmount_flakey
> +       _cleanup_flakey
> +
> +       # Mount the fs to do proper log replay for journal based fs
> +       # so later check won't report annoying dirty log and only
> +       # report real problem.
> +       _scratch_mount
> +       _scratch_unmount
> +
> +       _check_scratch_fs
> +done
> +
> +echo "Silence is golden"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/479.out b/tests/generic/479.out
> new file mode 100644
> index 00000000..290f18b3
> --- /dev/null
> +++ b/tests/generic/479.out
> @@ -0,0 +1,2 @@
> +QA output created by 479
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index 1e808865..5ce3db1d 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -481,3 +481,4 @@
>  476 auto rw
>  477 auto quick exportfs
>  478 auto quick
> +479 auto

+ stress

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 6638 bytes --]



On 2018年03月01日 16:39, Amir Goldstein wrote:
> On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
>> This test case is originally designed to expose unexpected corruption
>> for btrfs, where there are several reports about btrfs serious metadata
>> corruption after power loss.
>>
>> The test case itself will trigger heavy fsstress for the fs, and use
>> dm-flakey to emulate power loss by dropping all later writes.
> 
> So you are re-posting the test with dm-flakey or converting it to
> dm-log-writes??

Working on the scripts to allow us to do --find and then replay.

Since for xfs and ext4, their fsck would report false alerts just for
dirty journal.

I'm adding new macro to locate next flush and replay to it, then mount
it RW before we call fsck.

Or do we have options for those fscks to skip dirty journal?

Thanks,
Qu

> 
>>
>> For btrfs, it should be completely fine, as long as superblock write
>> (FUA write) finishes atomically, since with metadata CoW, superblock
>> either points to old trees or new tress, the fs should be as atomic as
>> superblock.
>>
>> For journal based filesystems, each metadata update should be journaled,
>> so metadata operation is as atomic as journal updates.
>>
>> It does show that XFS is doing the best work among the tested
>> filesystems (Btrfs, XFS, ext4), no kernel nor xfs_repair problem at all.
>>
>> For btrfs, although btrfs check doesn't report any problem, kernel
>> reports some data checksum error, which is a little unexpected as data
>> is CoWed by default, which should be as atomic as superblock.
>> (Unfortunately, still not the exact problem I'm chasing for)
>>
>> For EXT4, kernel is fine, but later e2fsck reports problem, which may
>> indicates there is still something to be improved.
>>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>> ---
>>  tests/generic/479     | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++
>>  tests/generic/479.out |   2 +
>>  tests/generic/group   |   1 +
>>  3 files changed, 112 insertions(+)
>>  create mode 100755 tests/generic/479
>>  create mode 100644 tests/generic/479.out
>>
>> diff --git a/tests/generic/479 b/tests/generic/479
>> new file mode 100755
>> index 00000000..ab530231
>> --- /dev/null
>> +++ b/tests/generic/479
>> @@ -0,0 +1,109 @@
>> +#! /bin/bash
>> +# FS QA Test 479
>> +#
>> +# Test if a filesystem can survive emulated powerloss.
>> +#
>> +# No matter what the solution a filesystem uses (journal or CoW),
>> +# it should survive unexpected powerloss, without major metadata
>> +# corruption.
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (c) 2018 SuSE.  All Rights Reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1       # failure is the default!
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +_cleanup()
>> +{
>> +       ps -e | grep fsstress > /dev/null 2>&1
>> +       while [ $? -eq 0 ]; do
>> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
>> +               wait > /dev/null 2>&1
>> +               ps -e | grep fsstress > /dev/null 2>&1
>> +       done
>> +       _unmount_flakey &> /dev/null
>> +       _cleanup_flakey
>> +       cd /
>> +       rm -f $tmp.*
>> +}
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +. ./common/dmflakey
>> +
>> +# remove previous $seqres.full before test
>> +rm -f $seqres.full
>> +
>> +# real QA test starts here
>> +
>> +# Modify as appropriate.
>> +_supported_fs generic
>> +_supported_os Linux
>> +_require_scratch
>> +_require_dm_target flakey
>> +_require_command "$KILLALL_PROG" "killall"
>> +
>> +runtime=$(($TIME_FACTOR * 15))
>> +loops=$(($LOAD_FACTOR * 4))
>> +
>> +for i in $(seq -w $loops); do
>> +       echo "=== Loop $i: $(date) ===" >> $seqres.full
>> +
>> +       _scratch_mkfs >/dev/null 2>&1
>> +       _init_flakey
>> +       _mount_flakey
>> +
>> +       ($FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n 1000000 \
>> +               -p 100 >> $seqres.full &) > /dev/null 2>&1
>> +
>> +       sleep $runtime
>> +
>> +       # Here we only want to drop all write, don't need to umount the fs
>> +       _load_flakey_table $FLAKEY_DROP_WRITES
>> +
>> +       ps -e | grep fsstress > /dev/null 2>&1
>> +       while [ $? -eq 0 ]; do
>> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
>> +               wait > /dev/null 2>&1
>> +               ps -e | grep fsstress > /dev/null 2>&1
>> +       done
>> +
>> +       _unmount_flakey
>> +       _cleanup_flakey
>> +
>> +       # Mount the fs to do proper log replay for journal based fs
>> +       # so later check won't report annoying dirty log and only
>> +       # report real problem.
>> +       _scratch_mount
>> +       _scratch_unmount
>> +
>> +       _check_scratch_fs
>> +done
>> +
>> +echo "Silence is golden"
>> +
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/generic/479.out b/tests/generic/479.out
>> new file mode 100644
>> index 00000000..290f18b3
>> --- /dev/null
>> +++ b/tests/generic/479.out
>> @@ -0,0 +1,2 @@
>> +QA output created by 479
>> +Silence is golden
>> diff --git a/tests/generic/group b/tests/generic/group
>> index 1e808865..5ce3db1d 100644
>> --- a/tests/generic/group
>> +++ b/tests/generic/group
>> @@ -481,3 +481,4 @@
>>  476 auto rw
>>  477 auto quick exportfs
>>  478 auto quick
>> +479 auto
> 
> + stress
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

[-- Attachment #1.1.1: Type: text/plain, Size: 6638 bytes --]



On 2018年03月01日 16:39, Amir Goldstein wrote:
> On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
>> This test case is originally designed to expose unexpected corruption
>> for btrfs, where there are several reports about btrfs serious metadata
>> corruption after power loss.
>>
>> The test case itself will trigger heavy fsstress for the fs, and use
>> dm-flakey to emulate power loss by dropping all later writes.
> 
> So you are re-posting the test with dm-flakey or converting it to
> dm-log-writes??

Working on the scripts to allow us to do --find and then replay.

Since for xfs and ext4, their fsck would report false alerts just for
dirty journal.

I'm adding new macro to locate next flush and replay to it, then mount
it RW before we call fsck.

Or do we have options for those fscks to skip dirty journal?

Thanks,
Qu

> 
>>
>> For btrfs, it should be completely fine, as long as superblock write
>> (FUA write) finishes atomically, since with metadata CoW, superblock
>> either points to old trees or new tress, the fs should be as atomic as
>> superblock.
>>
>> For journal based filesystems, each metadata update should be journaled,
>> so metadata operation is as atomic as journal updates.
>>
>> It does show that XFS is doing the best work among the tested
>> filesystems (Btrfs, XFS, ext4), no kernel nor xfs_repair problem at all.
>>
>> For btrfs, although btrfs check doesn't report any problem, kernel
>> reports some data checksum error, which is a little unexpected as data
>> is CoWed by default, which should be as atomic as superblock.
>> (Unfortunately, still not the exact problem I'm chasing for)
>>
>> For EXT4, kernel is fine, but later e2fsck reports problem, which may
>> indicates there is still something to be improved.
>>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>> ---
>>  tests/generic/479     | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++
>>  tests/generic/479.out |   2 +
>>  tests/generic/group   |   1 +
>>  3 files changed, 112 insertions(+)
>>  create mode 100755 tests/generic/479
>>  create mode 100644 tests/generic/479.out
>>
>> diff --git a/tests/generic/479 b/tests/generic/479
>> new file mode 100755
>> index 00000000..ab530231
>> --- /dev/null
>> +++ b/tests/generic/479
>> @@ -0,0 +1,109 @@
>> +#! /bin/bash
>> +# FS QA Test 479
>> +#
>> +# Test if a filesystem can survive emulated powerloss.
>> +#
>> +# No matter what the solution a filesystem uses (journal or CoW),
>> +# it should survive unexpected powerloss, without major metadata
>> +# corruption.
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (c) 2018 SuSE.  All Rights Reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1       # failure is the default!
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +_cleanup()
>> +{
>> +       ps -e | grep fsstress > /dev/null 2>&1
>> +       while [ $? -eq 0 ]; do
>> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
>> +               wait > /dev/null 2>&1
>> +               ps -e | grep fsstress > /dev/null 2>&1
>> +       done
>> +       _unmount_flakey &> /dev/null
>> +       _cleanup_flakey
>> +       cd /
>> +       rm -f $tmp.*
>> +}
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +. ./common/dmflakey
>> +
>> +# remove previous $seqres.full before test
>> +rm -f $seqres.full
>> +
>> +# real QA test starts here
>> +
>> +# Modify as appropriate.
>> +_supported_fs generic
>> +_supported_os Linux
>> +_require_scratch
>> +_require_dm_target flakey
>> +_require_command "$KILLALL_PROG" "killall"
>> +
>> +runtime=$(($TIME_FACTOR * 15))
>> +loops=$(($LOAD_FACTOR * 4))
>> +
>> +for i in $(seq -w $loops); do
>> +       echo "=== Loop $i: $(date) ===" >> $seqres.full
>> +
>> +       _scratch_mkfs >/dev/null 2>&1
>> +       _init_flakey
>> +       _mount_flakey
>> +
>> +       ($FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n 1000000 \
>> +               -p 100 >> $seqres.full &) > /dev/null 2>&1
>> +
>> +       sleep $runtime
>> +
>> +       # Here we only want to drop all write, don't need to umount the fs
>> +       _load_flakey_table $FLAKEY_DROP_WRITES
>> +
>> +       ps -e | grep fsstress > /dev/null 2>&1
>> +       while [ $? -eq 0 ]; do
>> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
>> +               wait > /dev/null 2>&1
>> +               ps -e | grep fsstress > /dev/null 2>&1
>> +       done
>> +
>> +       _unmount_flakey
>> +       _cleanup_flakey
>> +
>> +       # Mount the fs to do proper log replay for journal based fs
>> +       # so later check won't report annoying dirty log and only
>> +       # report real problem.
>> +       _scratch_mount
>> +       _scratch_unmount
>> +
>> +       _check_scratch_fs
>> +done
>> +
>> +echo "Silence is golden"
>> +
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/generic/479.out b/tests/generic/479.out
>> new file mode 100644
>> index 00000000..290f18b3
>> --- /dev/null
>> +++ b/tests/generic/479.out
>> @@ -0,0 +1,2 @@
>> +QA output created by 479
>> +Silence is golden
>> diff --git a/tests/generic/group b/tests/generic/group
>> index 1e808865..5ce3db1d 100644
>> --- a/tests/generic/group
>> +++ b/tests/generic/group
>> @@ -481,3 +481,4 @@
>>  476 auto rw
>>  477 auto quick exportfs
>>  478 auto quick
>> +479 auto
> 
> + stress
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 6454 bytes --]



On 2018年03月01日 16:39, Amir Goldstein wrote:
> On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
>> This test case is originally designed to expose unexpected corruption
>> for btrfs, where there are several reports about btrfs serious metadata
>> corruption after power loss.
>>
>> The test case itself will trigger heavy fsstress for the fs, and use
>> dm-flakey to emulate power loss by dropping all later writes.
> 
> So you are re-posting the test with dm-flakey or converting it to
> dm-log-writes??

Sorry, I just noticed the date right now.

I was formatting the patch to the wrong place with old patches.
Please ignore this patch.

Thanks,
Qu

> 
>>
>> For btrfs, it should be completely fine, as long as superblock write
>> (FUA write) finishes atomically, since with metadata CoW, superblock
>> either points to old trees or new tress, the fs should be as atomic as
>> superblock.
>>
>> For journal based filesystems, each metadata update should be journaled,
>> so metadata operation is as atomic as journal updates.
>>
>> It does show that XFS is doing the best work among the tested
>> filesystems (Btrfs, XFS, ext4), no kernel nor xfs_repair problem at all.
>>
>> For btrfs, although btrfs check doesn't report any problem, kernel
>> reports some data checksum error, which is a little unexpected as data
>> is CoWed by default, which should be as atomic as superblock.
>> (Unfortunately, still not the exact problem I'm chasing for)
>>
>> For EXT4, kernel is fine, but later e2fsck reports problem, which may
>> indicates there is still something to be improved.
>>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>> ---
>>  tests/generic/479     | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++
>>  tests/generic/479.out |   2 +
>>  tests/generic/group   |   1 +
>>  3 files changed, 112 insertions(+)
>>  create mode 100755 tests/generic/479
>>  create mode 100644 tests/generic/479.out
>>
>> diff --git a/tests/generic/479 b/tests/generic/479
>> new file mode 100755
>> index 00000000..ab530231
>> --- /dev/null
>> +++ b/tests/generic/479
>> @@ -0,0 +1,109 @@
>> +#! /bin/bash
>> +# FS QA Test 479
>> +#
>> +# Test if a filesystem can survive emulated powerloss.
>> +#
>> +# No matter what the solution a filesystem uses (journal or CoW),
>> +# it should survive unexpected powerloss, without major metadata
>> +# corruption.
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (c) 2018 SuSE.  All Rights Reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1       # failure is the default!
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +_cleanup()
>> +{
>> +       ps -e | grep fsstress > /dev/null 2>&1
>> +       while [ $? -eq 0 ]; do
>> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
>> +               wait > /dev/null 2>&1
>> +               ps -e | grep fsstress > /dev/null 2>&1
>> +       done
>> +       _unmount_flakey &> /dev/null
>> +       _cleanup_flakey
>> +       cd /
>> +       rm -f $tmp.*
>> +}
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +. ./common/dmflakey
>> +
>> +# remove previous $seqres.full before test
>> +rm -f $seqres.full
>> +
>> +# real QA test starts here
>> +
>> +# Modify as appropriate.
>> +_supported_fs generic
>> +_supported_os Linux
>> +_require_scratch
>> +_require_dm_target flakey
>> +_require_command "$KILLALL_PROG" "killall"
>> +
>> +runtime=$(($TIME_FACTOR * 15))
>> +loops=$(($LOAD_FACTOR * 4))
>> +
>> +for i in $(seq -w $loops); do
>> +       echo "=== Loop $i: $(date) ===" >> $seqres.full
>> +
>> +       _scratch_mkfs >/dev/null 2>&1
>> +       _init_flakey
>> +       _mount_flakey
>> +
>> +       ($FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n 1000000 \
>> +               -p 100 >> $seqres.full &) > /dev/null 2>&1
>> +
>> +       sleep $runtime
>> +
>> +       # Here we only want to drop all write, don't need to umount the fs
>> +       _load_flakey_table $FLAKEY_DROP_WRITES
>> +
>> +       ps -e | grep fsstress > /dev/null 2>&1
>> +       while [ $? -eq 0 ]; do
>> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
>> +               wait > /dev/null 2>&1
>> +               ps -e | grep fsstress > /dev/null 2>&1
>> +       done
>> +
>> +       _unmount_flakey
>> +       _cleanup_flakey
>> +
>> +       # Mount the fs to do proper log replay for journal based fs
>> +       # so later check won't report annoying dirty log and only
>> +       # report real problem.
>> +       _scratch_mount
>> +       _scratch_unmount
>> +
>> +       _check_scratch_fs
>> +done
>> +
>> +echo "Silence is golden"
>> +
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/generic/479.out b/tests/generic/479.out
>> new file mode 100644
>> index 00000000..290f18b3
>> --- /dev/null
>> +++ b/tests/generic/479.out
>> @@ -0,0 +1,2 @@
>> +QA output created by 479
>> +Silence is golden
>> diff --git a/tests/generic/group b/tests/generic/group
>> index 1e808865..5ce3db1d 100644
>> --- a/tests/generic/group
>> +++ b/tests/generic/group
>> @@ -481,3 +481,4 @@
>>  476 auto rw
>>  477 auto quick exportfs
>>  478 auto quick
>> +479 auto
> 
> + stress
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

[-- Attachment #1.1.1: Type: text/plain, Size: 6454 bytes --]



On 2018年03月01日 16:39, Amir Goldstein wrote:
> On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
>> This test case is originally designed to expose unexpected corruption
>> for btrfs, where there are several reports about btrfs serious metadata
>> corruption after power loss.
>>
>> The test case itself will trigger heavy fsstress for the fs, and use
>> dm-flakey to emulate power loss by dropping all later writes.
> 
> So you are re-posting the test with dm-flakey or converting it to
> dm-log-writes??

Sorry, I just noticed the date right now.

I was formatting the patch to the wrong place with old patches.
Please ignore this patch.

Thanks,
Qu

> 
>>
>> For btrfs, it should be completely fine, as long as superblock write
>> (FUA write) finishes atomically, since with metadata CoW, superblock
>> either points to old trees or new tress, the fs should be as atomic as
>> superblock.
>>
>> For journal based filesystems, each metadata update should be journaled,
>> so metadata operation is as atomic as journal updates.
>>
>> It does show that XFS is doing the best work among the tested
>> filesystems (Btrfs, XFS, ext4), no kernel nor xfs_repair problem at all.
>>
>> For btrfs, although btrfs check doesn't report any problem, kernel
>> reports some data checksum error, which is a little unexpected as data
>> is CoWed by default, which should be as atomic as superblock.
>> (Unfortunately, still not the exact problem I'm chasing for)
>>
>> For EXT4, kernel is fine, but later e2fsck reports problem, which may
>> indicates there is still something to be improved.
>>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>> ---
>>  tests/generic/479     | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++
>>  tests/generic/479.out |   2 +
>>  tests/generic/group   |   1 +
>>  3 files changed, 112 insertions(+)
>>  create mode 100755 tests/generic/479
>>  create mode 100644 tests/generic/479.out
>>
>> diff --git a/tests/generic/479 b/tests/generic/479
>> new file mode 100755
>> index 00000000..ab530231
>> --- /dev/null
>> +++ b/tests/generic/479
>> @@ -0,0 +1,109 @@
>> +#! /bin/bash
>> +# FS QA Test 479
>> +#
>> +# Test if a filesystem can survive emulated powerloss.
>> +#
>> +# No matter what the solution a filesystem uses (journal or CoW),
>> +# it should survive unexpected powerloss, without major metadata
>> +# corruption.
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (c) 2018 SuSE.  All Rights Reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1       # failure is the default!
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +_cleanup()
>> +{
>> +       ps -e | grep fsstress > /dev/null 2>&1
>> +       while [ $? -eq 0 ]; do
>> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
>> +               wait > /dev/null 2>&1
>> +               ps -e | grep fsstress > /dev/null 2>&1
>> +       done
>> +       _unmount_flakey &> /dev/null
>> +       _cleanup_flakey
>> +       cd /
>> +       rm -f $tmp.*
>> +}
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +. ./common/dmflakey
>> +
>> +# remove previous $seqres.full before test
>> +rm -f $seqres.full
>> +
>> +# real QA test starts here
>> +
>> +# Modify as appropriate.
>> +_supported_fs generic
>> +_supported_os Linux
>> +_require_scratch
>> +_require_dm_target flakey
>> +_require_command "$KILLALL_PROG" "killall"
>> +
>> +runtime=$(($TIME_FACTOR * 15))
>> +loops=$(($LOAD_FACTOR * 4))
>> +
>> +for i in $(seq -w $loops); do
>> +       echo "=== Loop $i: $(date) ===" >> $seqres.full
>> +
>> +       _scratch_mkfs >/dev/null 2>&1
>> +       _init_flakey
>> +       _mount_flakey
>> +
>> +       ($FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n 1000000 \
>> +               -p 100 >> $seqres.full &) > /dev/null 2>&1
>> +
>> +       sleep $runtime
>> +
>> +       # Here we only want to drop all write, don't need to umount the fs
>> +       _load_flakey_table $FLAKEY_DROP_WRITES
>> +
>> +       ps -e | grep fsstress > /dev/null 2>&1
>> +       while [ $? -eq 0 ]; do
>> +               $KILLALL_PROG -KILL fsstress > /dev/null 2>&1
>> +               wait > /dev/null 2>&1
>> +               ps -e | grep fsstress > /dev/null 2>&1
>> +       done
>> +
>> +       _unmount_flakey
>> +       _cleanup_flakey
>> +
>> +       # Mount the fs to do proper log replay for journal based fs
>> +       # so later check won't report annoying dirty log and only
>> +       # report real problem.
>> +       _scratch_mount
>> +       _scratch_unmount
>> +
>> +       _check_scratch_fs
>> +done
>> +
>> +echo "Silence is golden"
>> +
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/generic/479.out b/tests/generic/479.out
>> new file mode 100644
>> index 00000000..290f18b3
>> --- /dev/null
>> +++ b/tests/generic/479.out
>> @@ -0,0 +1,2 @@
>> +QA output created by 479
>> +Silence is golden
>> diff --git a/tests/generic/group b/tests/generic/group
>> index 1e808865..5ce3db1d 100644
>> --- a/tests/generic/group
>> +++ b/tests/generic/group
>> @@ -481,3 +481,4 @@
>>  476 auto rw
>>  477 auto quick exportfs
>>  478 auto quick
>> +479 auto
> 
> + stress
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Amir Goldstein]

On Thu, Mar 1, 2018 at 11:25 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
> On 2018年03月01日 16:39, Amir Goldstein wrote:
>> On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
>>> This test case is originally designed to expose unexpected corruption
>>> for btrfs, where there are several reports about btrfs serious metadata
>>> corruption after power loss.
>>>
>>> The test case itself will trigger heavy fsstress for the fs, and use
>>> dm-flakey to emulate power loss by dropping all later writes.
>>
>> So you are re-posting the test with dm-flakey or converting it to
>> dm-log-writes??
>
> Working on the scripts to allow us to do --find and then replay.
>
> Since for xfs and ext4, their fsck would report false alerts just for
> dirty journal.
>
> I'm adding new macro to locate next flush and replay to it, then mount
> it RW before we call fsck.
>
> Or do we have options for those fscks to skip dirty journal?
>

No, you are much better off doing mount/umount before fsck.
Even though e2fsck can replay a journal, it does that much slower
then the kernel does.

But why do you need to teach --find to find next flush?
You could use a helper script to run every fua with --fsck --check fua.
Granted, for fstests context, I agree that --find next fua may look
nicer, so I have no objection to this implementation.

Thanks,
Amir.

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Amir Goldstein]

On Thu, Mar 1, 2018 at 11:25 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
> On 2018年03月01日 16:39, Amir Goldstein wrote:
>> On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
>>> This test case is originally designed to expose unexpected corruption
>>> for btrfs, where there are several reports about btrfs serious metadata
>>> corruption after power loss.
>>>
>>> The test case itself will trigger heavy fsstress for the fs, and use
>>> dm-flakey to emulate power loss by dropping all later writes.
>>
>> So you are re-posting the test with dm-flakey or converting it to
>> dm-log-writes??
>
> Working on the scripts to allow us to do --find and then replay.
>
> Since for xfs and ext4, their fsck would report false alerts just for
> dirty journal.
>
> I'm adding new macro to locate next flush and replay to it, then mount
> it RW before we call fsck.
>
> Or do we have options for those fscks to skip dirty journal?
>

No, you are much better off doing mount/umount before fsck.
Even though e2fsck can replay a journal, it does that much slower
then the kernel does.

But why do you need to teach --find to find next flush?
You could use a helper script to run every fua with --fsck --check fua.
Granted, for fstests context, I agree that --find next fua may look
nicer, so I have no objection to this implementation.

Thanks,
Amir.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2842 bytes --]



On 2018年03月01日 19:15, Amir Goldstein wrote:
> On Thu, Mar 1, 2018 at 11:25 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>> On 2018年03月01日 16:39, Amir Goldstein wrote:
>>> On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
>>>> This test case is originally designed to expose unexpected corruption
>>>> for btrfs, where there are several reports about btrfs serious metadata
>>>> corruption after power loss.
>>>>
>>>> The test case itself will trigger heavy fsstress for the fs, and use
>>>> dm-flakey to emulate power loss by dropping all later writes.
>>>
>>> So you are re-posting the test with dm-flakey or converting it to
>>> dm-log-writes??
>>
>> Working on the scripts to allow us to do --find and then replay.
>>
>> Since for xfs and ext4, their fsck would report false alerts just for
>> dirty journal.
>>
>> I'm adding new macro to locate next flush and replay to it, then mount
>> it RW before we call fsck.
>>
>> Or do we have options for those fscks to skip dirty journal?
>>
> 
> No, you are much better off doing mount/umount before fsck.
> Even though e2fsck can replay a journal, it does that much slower
> then the kernel does.
> 
> But why do you need to teach --find to find next flush?
> You could use a helper script to run every fua with --fsck --check fua.
> Granted, for fstests context, I agree that --find next fua may look
> nicer, so I have no objection to this implementation.

The point is, in my opinion fua is not the worst case we need to test.
Only flush could leads us to the worst case we really need to test.

In btrfs' case, if we finished flush, but without fua, we have a super
block points to all old trees, but all new trees are already written to
disk.

In that flush entry, we could reach to the worst case scenario to verify
all btrfs tricks are working all together to get a completely sane btrfs
(even all data should be correct).

This should also apply to journal based filesystems (if I understand the
journal thing correctly), even when all journals written but superblock
not updated, we should be completely fine.
(Although for journal, we may need to reach fua entry instead of flush?)

And the other reason why we need to find next flush/fua manually is,
mount will write new data, and we need to replay all the sequence until
next flush/fua.


And finally the reason about why need manually mount is, we need to
workaround e2fsck/xfs_repair, so that they won't report dirty journal as
error. If we have extra options to disable such behavior, I'm completely
OK with current --check flush/fua --fsck method.
(BTW, for my btrfs testing, --check flush --fsck is completely good
enough, to exposed possible free space cache related problems)

Thanks,
Qu

> 
> Thanks,
> Amir.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

[-- Attachment #1.1.1: Type: text/plain, Size: 2842 bytes --]



On 2018年03月01日 19:15, Amir Goldstein wrote:
> On Thu, Mar 1, 2018 at 11:25 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>> On 2018年03月01日 16:39, Amir Goldstein wrote:
>>> On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
>>>> This test case is originally designed to expose unexpected corruption
>>>> for btrfs, where there are several reports about btrfs serious metadata
>>>> corruption after power loss.
>>>>
>>>> The test case itself will trigger heavy fsstress for the fs, and use
>>>> dm-flakey to emulate power loss by dropping all later writes.
>>>
>>> So you are re-posting the test with dm-flakey or converting it to
>>> dm-log-writes??
>>
>> Working on the scripts to allow us to do --find and then replay.
>>
>> Since for xfs and ext4, their fsck would report false alerts just for
>> dirty journal.
>>
>> I'm adding new macro to locate next flush and replay to it, then mount
>> it RW before we call fsck.
>>
>> Or do we have options for those fscks to skip dirty journal?
>>
> 
> No, you are much better off doing mount/umount before fsck.
> Even though e2fsck can replay a journal, it does that much slower
> then the kernel does.
> 
> But why do you need to teach --find to find next flush?
> You could use a helper script to run every fua with --fsck --check fua.
> Granted, for fstests context, I agree that --find next fua may look
> nicer, so I have no objection to this implementation.

The point is, in my opinion fua is not the worst case we need to test.
Only flush could leads us to the worst case we really need to test.

In btrfs' case, if we finished flush, but without fua, we have a super
block points to all old trees, but all new trees are already written to
disk.

In that flush entry, we could reach to the worst case scenario to verify
all btrfs tricks are working all together to get a completely sane btrfs
(even all data should be correct).

This should also apply to journal based filesystems (if I understand the
journal thing correctly), even when all journals written but superblock
not updated, we should be completely fine.
(Although for journal, we may need to reach fua entry instead of flush?)

And the other reason why we need to find next flush/fua manually is,
mount will write new data, and we need to replay all the sequence until
next flush/fua.


And finally the reason about why need manually mount is, we need to
workaround e2fsck/xfs_repair, so that they won't report dirty journal as
error. If we have extra options to disable such behavior, I'm completely
OK with current --check flush/fua --fsck method.
(BTW, for my btrfs testing, --check flush --fsck is completely good
enough, to exposed possible free space cache related problems)

Thanks,
Qu

> 
> Thanks,
> Amir.
> 


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Amir Goldstein]

On Thu, Mar 1, 2018 at 1:48 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
> On 2018年03月01日 19:15, Amir Goldstein wrote:
>> On Thu, Mar 1, 2018 at 11:25 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>> On 2018年03月01日 16:39, Amir Goldstein wrote:
>>>> On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
>>>>> This test case is originally designed to expose unexpected corruption
>>>>> for btrfs, where there are several reports about btrfs serious metadata
>>>>> corruption after power loss.
>>>>>
>>>>> The test case itself will trigger heavy fsstress for the fs, and use
>>>>> dm-flakey to emulate power loss by dropping all later writes.
>>>>
>>>> So you are re-posting the test with dm-flakey or converting it to
>>>> dm-log-writes??
>>>
>>> Working on the scripts to allow us to do --find and then replay.
>>>
>>> Since for xfs and ext4, their fsck would report false alerts just for
>>> dirty journal.
>>>
>>> I'm adding new macro to locate next flush and replay to it, then mount
>>> it RW before we call fsck.
>>>
>>> Or do we have options for those fscks to skip dirty journal?
>>>
>>
>> No, you are much better off doing mount/umount before fsck.
>> Even though e2fsck can replay a journal, it does that much slower
>> then the kernel does.
>>
>> But why do you need to teach --find to find next flush?
>> You could use a helper script to run every fua with --fsck --check fua.
>> Granted, for fstests context, I agree that --find next fua may look
>> nicer, so I have no objection to this implementation.
>
> The point is, in my opinion fua is not the worst case we need to test.
> Only flush could leads us to the worst case we really need to test.
>
> In btrfs' case, if we finished flush, but without fua, we have a super
> block points to all old trees, but all new trees are already written to
> disk.
>
> In that flush entry, we could reach to the worst case scenario to verify
> all btrfs tricks are working all together to get a completely sane btrfs
> (even all data should be correct).
>
> This should also apply to journal based filesystems (if I understand the
> journal thing correctly), even when all journals written but superblock
> not updated, we should be completely fine.
> (Although for journal, we may need to reach fua entry instead of flush?)
>
> And the other reason why we need to find next flush/fua manually is,
> mount will write new data, and we need to replay all the sequence until
> next flush/fua.
>

OK. but Josef addressed this in his script using dm snapshots, rather
than replaying each time. I guess this is the reason the script is called
replay-individual-faster.sh. You don't have to do the same, but I expect
the test would run faster if you learn from experience of Josef.

>
> And finally the reason about why need manually mount is, we need to
> workaround e2fsck/xfs_repair, so that they won't report dirty journal as
> error. If we have extra options to disable such behavior, I'm completely
> OK with current --check flush/fua --fsck method.
> (BTW, for my btrfs testing, --check flush --fsck is completely good
> enough, to exposed possible free space cache related problems)
>

What I was suggesting as an alternative is --fsck ./replay-fsck-wrapper.sh
where the wrapper script does the needed mount/umount and if you
also use dm snapshot for the mounted volume you can continue to replay
from the same point and don't need to replay from the start.

Cheers,
Amir.

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Amir Goldstein]

On Thu, Mar 1, 2018 at 1:48 PM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
> On 2018年03月01日 19:15, Amir Goldstein wrote:
>> On Thu, Mar 1, 2018 at 11:25 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>> On 2018年03月01日 16:39, Amir Goldstein wrote:
>>>> On Thu, Mar 1, 2018 at 7:38 AM, Qu Wenruo <wqu@suse.com> wrote:
>>>>> This test case is originally designed to expose unexpected corruption
>>>>> for btrfs, where there are several reports about btrfs serious metadata
>>>>> corruption after power loss.
>>>>>
>>>>> The test case itself will trigger heavy fsstress for the fs, and use
>>>>> dm-flakey to emulate power loss by dropping all later writes.
>>>>
>>>> So you are re-posting the test with dm-flakey or converting it to
>>>> dm-log-writes??
>>>
>>> Working on the scripts to allow us to do --find and then replay.
>>>
>>> Since for xfs and ext4, their fsck would report false alerts just for
>>> dirty journal.
>>>
>>> I'm adding new macro to locate next flush and replay to it, then mount
>>> it RW before we call fsck.
>>>
>>> Or do we have options for those fscks to skip dirty journal?
>>>
>>
>> No, you are much better off doing mount/umount before fsck.
>> Even though e2fsck can replay a journal, it does that much slower
>> then the kernel does.
>>
>> But why do you need to teach --find to find next flush?
>> You could use a helper script to run every fua with --fsck --check fua.
>> Granted, for fstests context, I agree that --find next fua may look
>> nicer, so I have no objection to this implementation.
>
> The point is, in my opinion fua is not the worst case we need to test.
> Only flush could leads us to the worst case we really need to test.
>
> In btrfs' case, if we finished flush, but without fua, we have a super
> block points to all old trees, but all new trees are already written to
> disk.
>
> In that flush entry, we could reach to the worst case scenario to verify
> all btrfs tricks are working all together to get a completely sane btrfs
> (even all data should be correct).
>
> This should also apply to journal based filesystems (if I understand the
> journal thing correctly), even when all journals written but superblock
> not updated, we should be completely fine.
> (Although for journal, we may need to reach fua entry instead of flush?)
>
> And the other reason why we need to find next flush/fua manually is,
> mount will write new data, and we need to replay all the sequence until
> next flush/fua.
>

OK. but Josef addressed this in his script using dm snapshots, rather
than replaying each time. I guess this is the reason the script is called
replay-individual-faster.sh. You don't have to do the same, but I expect
the test would run faster if you learn from experience of Josef.

>
> And finally the reason about why need manually mount is, we need to
> workaround e2fsck/xfs_repair, so that they won't report dirty journal as
> error. If we have extra options to disable such behavior, I'm completely
> OK with current --check flush/fua --fsck method.
> (BTW, for my btrfs testing, --check flush --fsck is completely good
> enough, to exposed possible free space cache related problems)
>

What I was suggesting as an alternative is --fsck ./replay-fsck-wrapper.sh
where the wrapper script does the needed mount/umount and if you
also use dm snapshot for the mounted volume you can continue to replay
from the same point and don't need to replay from the start.

Cheers,
Amir.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2955 bytes --]



On 2018年02月26日 16:45, Amir Goldstein wrote:
> On Mon, Feb 26, 2018 at 10:41 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>> On 2018年02月26日 16:33, Amir Goldstein wrote:
>>> On Mon, Feb 26, 2018 at 10:20 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>> On 2018年02月26日 16:15, Amir Goldstein wrote:
>>>>> On Mon, Feb 26, 2018 at 9:31 AM, Qu Wenruo <wqu@suse.com> wrote:
>>>>>> This test case is originally designed to expose unexpected corruption
>>>>>> for btrfs, where there are several reports about btrfs serious metadata
>>>>>> corruption after power loss.
>>>>>>
>>>>>> The test case itself will trigger heavy fsstress for the fs, and use
>>>>>> dm-flakey to emulate power loss by dropping all later writes.
>>>>>>
>>>>>
>>>>> Come on... dm-flakey is so 2016
>>>>> You should take Josef's fsstress+log-writes test and bring it to fstests:
>>>>> https://github.com/josefbacik/log-writes
>>>>>
>>>>> By doing that you will gain two very important features from the test:
>>>>>
>>>>> 1. Problems will be discovered much faster, because the test can run fsck
>>>>>     after every single block write has been replayed instead of just at random
>>>>>     times like in your test
>>>>
>>>> That's what exactly I want!!!
>>>>
>>>> Great thanks for this one! I would definitely look into this.
>>>> (Although the initial commit is even older than 2016)
>>>>
>>>
>>> Please note that Josef's replay-individual-faster.sh script runs fsck
>>> every 1000 writes (i.e. --check 1000), so you can play with this argument
>>> in your test. Can also run --fsck every --check fua or --check flush, which
>>> may be more indicative of real world problems. not sure.
>>>
>>>>
>>>> But the test itself could already expose something on EXT4, it still
>>>> makes some sense for ext4 developers as a verification test case.
>>>>
>>>
>>> Please take a look at generic/456
>>> When generic/455 found a reproduciable problem in ext4,
>>> I created a specific test without any randomness to pin point the
>>> problem found (using dm-flakey).
>>> If the problem you found is reproduciable, then it will be easy for you
>>> to create a similar "bisected" test.
>>
>> Yep, it's definitely needed for a pin-point test case, but I'm also
>> wondering if a random, stress test could also help.
>>
>> Test case with plain fsstress is already super helpful to expose some
>> bugs, such stress test won't hurt.
>>
> 
> 
> Yes, but the same stress test with dm-log-writes instead of dm-flakey
> will be as useful and much more, so no reason to merge the less useful
> stress test.

OK, I'll try to use dm-log to enhance the test case.

Thanks,
Qu

> 
> Thanks,
> Amir.
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Amir Goldstein]

On Mon, Feb 26, 2018 at 10:41 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
> On 2018年02月26日 16:33, Amir Goldstein wrote:
>> On Mon, Feb 26, 2018 at 10:20 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>> On 2018年02月26日 16:15, Amir Goldstein wrote:
>>>> On Mon, Feb 26, 2018 at 9:31 AM, Qu Wenruo <wqu@suse.com> wrote:
>>>>> This test case is originally designed to expose unexpected corruption
>>>>> for btrfs, where there are several reports about btrfs serious metadata
>>>>> corruption after power loss.
>>>>>
>>>>> The test case itself will trigger heavy fsstress for the fs, and use
>>>>> dm-flakey to emulate power loss by dropping all later writes.
>>>>>
>>>>
>>>> Come on... dm-flakey is so 2016
>>>> You should take Josef's fsstress+log-writes test and bring it to fstests:
>>>> https://github.com/josefbacik/log-writes
>>>>
>>>> By doing that you will gain two very important features from the test:
>>>>
>>>> 1. Problems will be discovered much faster, because the test can run fsck
>>>>     after every single block write has been replayed instead of just at random
>>>>     times like in your test
>>>
>>> That's what exactly I want!!!
>>>
>>> Great thanks for this one! I would definitely look into this.
>>> (Although the initial commit is even older than 2016)
>>>
>>
>> Please note that Josef's replay-individual-faster.sh script runs fsck
>> every 1000 writes (i.e. --check 1000), so you can play with this argument
>> in your test. Can also run --fsck every --check fua or --check flush, which
>> may be more indicative of real world problems. not sure.
>>
>>>
>>> But the test itself could already expose something on EXT4, it still
>>> makes some sense for ext4 developers as a verification test case.
>>>
>>
>> Please take a look at generic/456
>> When generic/455 found a reproduciable problem in ext4,
>> I created a specific test without any randomness to pin point the
>> problem found (using dm-flakey).
>> If the problem you found is reproduciable, then it will be easy for you
>> to create a similar "bisected" test.
>
> Yep, it's definitely needed for a pin-point test case, but I'm also
> wondering if a random, stress test could also help.
>
> Test case with plain fsstress is already super helpful to expose some
> bugs, such stress test won't hurt.
>


Yes, but the same stress test with dm-log-writes instead of dm-flakey
will be as useful and much more, so no reason to merge the less useful
stress test.

Thanks,
Amir.

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2275 bytes --]



On 2018年02月26日 16:33, Amir Goldstein wrote:
> On Mon, Feb 26, 2018 at 10:20 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>> On 2018年02月26日 16:15, Amir Goldstein wrote:
>>> On Mon, Feb 26, 2018 at 9:31 AM, Qu Wenruo <wqu@suse.com> wrote:
>>>> This test case is originally designed to expose unexpected corruption
>>>> for btrfs, where there are several reports about btrfs serious metadata
>>>> corruption after power loss.
>>>>
>>>> The test case itself will trigger heavy fsstress for the fs, and use
>>>> dm-flakey to emulate power loss by dropping all later writes.
>>>>
>>>
>>> Come on... dm-flakey is so 2016
>>> You should take Josef's fsstress+log-writes test and bring it to fstests:
>>> https://github.com/josefbacik/log-writes
>>>
>>> By doing that you will gain two very important features from the test:
>>>
>>> 1. Problems will be discovered much faster, because the test can run fsck
>>>     after every single block write has been replayed instead of just at random
>>>     times like in your test
>>
>> That's what exactly I want!!!
>>
>> Great thanks for this one! I would definitely look into this.
>> (Although the initial commit is even older than 2016)
>>
> 
> Please note that Josef's replay-individual-faster.sh script runs fsck
> every 1000 writes (i.e. --check 1000), so you can play with this argument
> in your test. Can also run --fsck every --check fua or --check flush, which
> may be more indicative of real world problems. not sure.
> 
>>
>> But the test itself could already expose something on EXT4, it still
>> makes some sense for ext4 developers as a verification test case.
>>
> 
> Please take a look at generic/456
> When generic/455 found a reproduciable problem in ext4,
> I created a specific test without any randomness to pin point the
> problem found (using dm-flakey).
> If the problem you found is reproduciable, then it will be easy for you
> to create a similar "bisected" test.

Yep, it's definitely needed for a pin-point test case, but I'm also
wondering if a random, stress test could also help.

Test case with plain fsstress is already super helpful to expose some
bugs, such stress test won't hurt.

Thanks,
Qu
> 
> Thanks,
> Amir.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Amir Goldstein]

On Mon, Feb 26, 2018 at 10:20 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
> On 2018年02月26日 16:15, Amir Goldstein wrote:
>> On Mon, Feb 26, 2018 at 9:31 AM, Qu Wenruo <wqu@suse.com> wrote:
>>> This test case is originally designed to expose unexpected corruption
>>> for btrfs, where there are several reports about btrfs serious metadata
>>> corruption after power loss.
>>>
>>> The test case itself will trigger heavy fsstress for the fs, and use
>>> dm-flakey to emulate power loss by dropping all later writes.
>>>
>>
>> Come on... dm-flakey is so 2016
>> You should take Josef's fsstress+log-writes test and bring it to fstests:
>> https://github.com/josefbacik/log-writes
>>
>> By doing that you will gain two very important features from the test:
>>
>> 1. Problems will be discovered much faster, because the test can run fsck
>>     after every single block write has been replayed instead of just at random
>>     times like in your test
>
> That's what exactly I want!!!
>
> Great thanks for this one! I would definitely look into this.
> (Although the initial commit is even older than 2016)
>

Please note that Josef's replay-individual-faster.sh script runs fsck
every 1000 writes (i.e. --check 1000), so you can play with this argument
in your test. Can also run --fsck every --check fua or --check flush, which
may be more indicative of real world problems. not sure.

>
> But the test itself could already expose something on EXT4, it still
> makes some sense for ext4 developers as a verification test case.
>

Please take a look at generic/456
When generic/455 found a reproduciable problem in ext4,
I created a specific test without any randomness to pin point the
problem found (using dm-flakey).
If the problem you found is reproduciable, then it will be easy for you
to create a similar "bisected" test.

Thanks,
Amir.

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2019 bytes --]



On 2018年02月26日 16:15, Amir Goldstein wrote:
> On Mon, Feb 26, 2018 at 9:31 AM, Qu Wenruo <wqu@suse.com> wrote:
>> This test case is originally designed to expose unexpected corruption
>> for btrfs, where there are several reports about btrfs serious metadata
>> corruption after power loss.
>>
>> The test case itself will trigger heavy fsstress for the fs, and use
>> dm-flakey to emulate power loss by dropping all later writes.
>>
> 
> Come on... dm-flakey is so 2016
> You should take Josef's fsstress+log-writes test and bring it to fstests:
> https://github.com/josefbacik/log-writes
> 
> By doing that you will gain two very important features from the test:
> 
> 1. Problems will be discovered much faster, because the test can run fsck
>     after every single block write has been replayed instead of just at random
>     times like in your test

That's what exactly I want!!!

Great thanks for this one! I would definitely look into this.
(Although the initial commit is even older than 2016)


But the test itself could already expose something on EXT4, it still
makes some sense for ext4 developers as a verification test case.

Thanks,
Qu

> 
> 2. Absolute guaranty to reproducing the problem by replaying the write log.
>     Even though your fsstress could use a pre-defined random seed to results
>     will be far from reproduciable, because of process and IO scheduling
>     differences between subsequent test runs.
>     When you catch an inconsistency with log-writes test, you can send the
>     write-log recording to the maintainer to analyze the problem, even if it is
>     a hard problem to hit. I used that useful technique for ext4,btrfs,xfs when
>     ran tests with generic/455 and found problems.
> 
> Cheers,
> Amir.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: [RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Amir Goldstein]

On Mon, Feb 26, 2018 at 9:31 AM, Qu Wenruo <wqu@suse.com> wrote:
> This test case is originally designed to expose unexpected corruption
> for btrfs, where there are several reports about btrfs serious metadata
> corruption after power loss.
>
> The test case itself will trigger heavy fsstress for the fs, and use
> dm-flakey to emulate power loss by dropping all later writes.
>

Come on... dm-flakey is so 2016
You should take Josef's fsstress+log-writes test and bring it to fstests:
https://github.com/josefbacik/log-writes

By doing that you will gain two very important features from the test:

1. Problems will be discovered much faster, because the test can run fsck
    after every single block write has been replayed instead of just at random
    times like in your test

2. Absolute guaranty to reproducing the problem by replaying the write log.
    Even though your fsstress could use a pre-defined random seed to results
    will be far from reproduciable, because of process and IO scheduling
    differences between subsequent test runs.
    When you catch an inconsistency with log-writes test, you can send the
    write-log recording to the maintainer to analyze the problem, even if it is
    a hard problem to hit. I used that useful technique for ext4,btrfs,xfs when
    ran tests with generic/455 and found problems.

Cheers,
Amir.

[RFC PATCH] fstests: Check if a fs can survive random (emulated) power loss

[Qu Wenruo]

This test case is originally designed to expose unexpected corruption
for btrfs, where there are several reports about btrfs serious metadata
corruption after power loss.

The test case itself will trigger heavy fsstress for the fs, and use
dm-flakey to emulate power loss by dropping all later writes.

For btrfs, it should be completely fine, as long as superblock write
(FUA write) finishes atomically, since with metadata CoW, superblock
either points to old trees or new tress, the fs should be as atomic as
superblock.

For journal based filesystems, each metadata update should be journaled,
so metadata operation is as atomic as journal updates.

It does show that XFS is doing the best work among the tested
filesystems (Btrfs, XFS, ext4), no kernel nor xfs_repair problem at all.

For btrfs, although btrfs check doesn't report any problem, kernel
reports some data checksum error, which is a little unexpected as data
is CoWed by default, which should be as atomic as superblock.
(Unfortunately, still not the exact problem I'm chasing for)

For EXT4, kernel is fine, but later e2fsck reports problem, which may
indicates there is still something to be improved.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/generic/479     | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/479.out |   2 +
 tests/generic/group   |   1 +
 3 files changed, 112 insertions(+)
 create mode 100755 tests/generic/479
 create mode 100644 tests/generic/479.out

diff --git a/tests/generic/479 b/tests/generic/479
new file mode 100755
index 00000000..ab530231
--- /dev/null
+++ b/tests/generic/479
@@ -0,0 +1,109 @@
+#! /bin/bash
+# FS QA Test 479
+#
+# Test if a filesystem can survive emulated powerloss.
+#
+# No matter what the solution a filesystem uses (journal or CoW),
+# it should survive unexpected powerloss, without major metadata
+# corruption.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2018 SuSE.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	ps -e | grep fsstress > /dev/null 2>&1
+	while [ $? -eq 0 ]; do
+		$KILLALL_PROG -KILL fsstress > /dev/null 2>&1
+		wait > /dev/null 2>&1
+		ps -e | grep fsstress > /dev/null 2>&1
+	done
+	_unmount_flakey &> /dev/null
+	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs generic
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+_require_command "$KILLALL_PROG" "killall"
+
+runtime=$(($TIME_FACTOR * 15))
+loops=$(($LOAD_FACTOR * 4))
+
+for i in $(seq -w $loops); do
+	echo "=== Loop $i: $(date) ===" >> $seqres.full
+
+	_scratch_mkfs >/dev/null 2>&1
+	_init_flakey
+	_mount_flakey
+
+	($FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n 1000000 \
+		-p 100 >> $seqres.full &) > /dev/null 2>&1
+
+	sleep $runtime
+
+	# Here we only want to drop all write, don't need to umount the fs
+	_load_flakey_table $FLAKEY_DROP_WRITES
+
+	ps -e | grep fsstress > /dev/null 2>&1
+	while [ $? -eq 0 ]; do
+		$KILLALL_PROG -KILL fsstress > /dev/null 2>&1
+		wait > /dev/null 2>&1
+		ps -e | grep fsstress > /dev/null 2>&1
+	done
+
+	_unmount_flakey
+	_cleanup_flakey
+
+	# Mount the fs to do proper log replay for journal based fs
+	# so later check won't report annoying dirty log and only
+	# report real problem.
+	_scratch_mount
+	_scratch_unmount
+
+	_check_scratch_fs
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/479.out b/tests/generic/479.out
new file mode 100644
index 00000000..290f18b3
--- /dev/null
+++ b/tests/generic/479.out
@@ -0,0 +1,2 @@
+QA output created by 479
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 1e808865..5ce3db1d 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -481,3 +481,4 @@
 476 auto rw
 477 auto quick exportfs
 478 auto quick
+479 auto
-- 
2.15.1