[PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

This is v2 of my earlier series "Btrfs: fix free space tree
bitmaps+tests on big-endian systems" [1]. Patches 1, 4, and 5 are the
same as patches 1, 2, and 3 from the original series. I've added patch 2
to fix another bug I noticed (an xfstest went out earlier). Patch 3 is
the result of the earlier discussion here [2]. Finally, patch 6 was
necessary to get the sanity tests to run on my MIPS emulator.

This series applies to v4.8-rc7. The sanity tests pass on both x86-64
and MIPS, and there are no xfstests regressions. Chandan and Anatoly,
could you test these out as well?

I'm working on the btrfs-progs follow up, but these patches are safe
without that -- the new FREE_SPACE_TREE_VALID bit will stop all versions
of btrfs-progs from mounting read-write.

Thanks!

1: http://marc.info/?l=linux-btrfs&m=146853909905570&w=2
2: http://marc.info/?l=linux-btrfs&m=147448992301110&w=2

Cc: Chandan Rajendra <chandan@linux.vnet.ibm.com>
Cc: Anatoly Pugachev <matorola@gmail.com>

Omar Sandoval (6):
  Btrfs: fix free space tree bitmaps on big-endian systems
  Btrfs: fix mount -o clear_cache,space_cache=v2
  Btrfs: catch invalid free space trees
  Btrfs: fix extent buffer bitmap tests on big-endian systems
  Btrfs: expand free space tree sanity tests to catch endianness bug
  Btrfs: use less memory for delalloc sanity tests

 fs/btrfs/ctree.h                       |   3 +-
 fs/btrfs/disk-io.c                     |  33 ++++---
 fs/btrfs/extent_io.c                   |  64 +++++++++----
 fs/btrfs/extent_io.h                   |  22 +++++
 fs/btrfs/free-space-tree.c             |  19 ++--
 fs/btrfs/tests/extent-io-tests.c       |  95 +++++++++++--------
 fs/btrfs/tests/free-space-tree-tests.c | 164 +++++++++++++++++++--------------
 include/uapi/linux/btrfs.h             |  10 +-
 8 files changed, 261 insertions(+), 149 deletions(-)

-- 
2.10.0

[PATCH v2 1/6] Btrfs: fix free space tree bitmaps on big-endian systems

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

In convert_free_space_to_{bitmaps,extents}(), we buffer the free space
bitmaps in memory and copy them directly to/from the extent buffers with
{read,write}_extent_buffer(). The extent buffer bitmap helpers use byte
granularity, which is equivalent to a little-endian bitmap. This means
that on big-endian systems, the in-memory bitmaps will be written to
disk byte-swapped. To fix this, use byte-granularity for the bitmaps in
memory.

Fixes: a5ed91828518 ("Btrfs: implement the free space B-tree")
Cc: stable@vger.kernel.org # 4.5+
Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 fs/btrfs/extent_io.c       | 64 +++++++++++++++++++++++++++++++++-------------
 fs/btrfs/extent_io.h       | 22 ++++++++++++++++
 fs/btrfs/free-space-tree.c | 17 ++++++------
 3 files changed, 76 insertions(+), 27 deletions(-)

diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 44fe66b..c3ec30d 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -5524,17 +5524,45 @@ void copy_extent_buffer(struct extent_buffer *dst, struct extent_buffer *src,
 	}
 }
 
-/*
- * The extent buffer bitmap operations are done with byte granularity because
- * bitmap items are not guaranteed to be aligned to a word and therefore a
- * single word in a bitmap may straddle two pages in the extent buffer.
- */
-#define BIT_BYTE(nr) ((nr) / BITS_PER_BYTE)
-#define BYTE_MASK ((1 << BITS_PER_BYTE) - 1)
-#define BITMAP_FIRST_BYTE_MASK(start) \
-	((BYTE_MASK << ((start) & (BITS_PER_BYTE - 1))) & BYTE_MASK)
-#define BITMAP_LAST_BYTE_MASK(nbits) \
-	(BYTE_MASK >> (-(nbits) & (BITS_PER_BYTE - 1)))
+void le_bitmap_set(u8 *map, unsigned int start, int len)
+{
+	u8 *p = map + BIT_BYTE(start);
+	const unsigned int size = start + len;
+	int bits_to_set = BITS_PER_BYTE - (start % BITS_PER_BYTE);
+	u8 mask_to_set = BITMAP_FIRST_BYTE_MASK(start);
+
+	while (len - bits_to_set >= 0) {
+		*p |= mask_to_set;
+		len -= bits_to_set;
+		bits_to_set = BITS_PER_BYTE;
+		mask_to_set = ~(u8)0;
+		p++;
+	}
+	if (len) {
+		mask_to_set &= BITMAP_LAST_BYTE_MASK(size);
+		*p |= mask_to_set;
+	}
+}
+
+void le_bitmap_clear(u8 *map, unsigned int start, int len)
+{
+	u8 *p = map + BIT_BYTE(start);
+	const unsigned int size = start + len;
+	int bits_to_clear = BITS_PER_BYTE - (start % BITS_PER_BYTE);
+	u8 mask_to_clear = BITMAP_FIRST_BYTE_MASK(start);
+
+	while (len - bits_to_clear >= 0) {
+		*p &= ~mask_to_clear;
+		len -= bits_to_clear;
+		bits_to_clear = BITS_PER_BYTE;
+		mask_to_clear = ~(u8)0;
+		p++;
+	}
+	if (len) {
+		mask_to_clear &= BITMAP_LAST_BYTE_MASK(size);
+		*p &= ~mask_to_clear;
+	}
+}
 
 /*
  * eb_bitmap_offset() - calculate the page and offset of the byte containing the
@@ -5578,7 +5606,7 @@ static inline void eb_bitmap_offset(struct extent_buffer *eb,
 int extent_buffer_test_bit(struct extent_buffer *eb, unsigned long start,
 			   unsigned long nr)
 {
-	char *kaddr;
+	u8 *kaddr;
 	struct page *page;
 	unsigned long i;
 	size_t offset;
@@ -5600,13 +5628,13 @@ int extent_buffer_test_bit(struct extent_buffer *eb, unsigned long start,
 void extent_buffer_bitmap_set(struct extent_buffer *eb, unsigned long start,
 			      unsigned long pos, unsigned long len)
 {
-	char *kaddr;
+	u8 *kaddr;
 	struct page *page;
 	unsigned long i;
 	size_t offset;
 	const unsigned int size = pos + len;
 	int bits_to_set = BITS_PER_BYTE - (pos % BITS_PER_BYTE);
-	unsigned int mask_to_set = BITMAP_FIRST_BYTE_MASK(pos);
+	u8 mask_to_set = BITMAP_FIRST_BYTE_MASK(pos);
 
 	eb_bitmap_offset(eb, start, pos, &i, &offset);
 	page = eb->pages[i];
@@ -5617,7 +5645,7 @@ void extent_buffer_bitmap_set(struct extent_buffer *eb, unsigned long start,
 		kaddr[offset] |= mask_to_set;
 		len -= bits_to_set;
 		bits_to_set = BITS_PER_BYTE;
-		mask_to_set = ~0U;
+		mask_to_set = ~(u8)0;
 		if (++offset >= PAGE_SIZE && len > 0) {
 			offset = 0;
 			page = eb->pages[++i];
@@ -5642,13 +5670,13 @@ void extent_buffer_bitmap_set(struct extent_buffer *eb, unsigned long start,
 void extent_buffer_bitmap_clear(struct extent_buffer *eb, unsigned long start,
 				unsigned long pos, unsigned long len)
 {
-	char *kaddr;
+	u8 *kaddr;
 	struct page *page;
 	unsigned long i;
 	size_t offset;
 	const unsigned int size = pos + len;
 	int bits_to_clear = BITS_PER_BYTE - (pos % BITS_PER_BYTE);
-	unsigned int mask_to_clear = BITMAP_FIRST_BYTE_MASK(pos);
+	u8 mask_to_clear = BITMAP_FIRST_BYTE_MASK(pos);
 
 	eb_bitmap_offset(eb, start, pos, &i, &offset);
 	page = eb->pages[i];
@@ -5659,7 +5687,7 @@ void extent_buffer_bitmap_clear(struct extent_buffer *eb, unsigned long start,
 		kaddr[offset] &= ~mask_to_clear;
 		len -= bits_to_clear;
 		bits_to_clear = BITS_PER_BYTE;
-		mask_to_clear = ~0U;
+		mask_to_clear = ~(u8)0;
 		if (++offset >= PAGE_SIZE && len > 0) {
 			offset = 0;
 			page = eb->pages[++i];
diff --git a/fs/btrfs/extent_io.h b/fs/btrfs/extent_io.h
index 28cd88f..1cf4e42 100644
--- a/fs/btrfs/extent_io.h
+++ b/fs/btrfs/extent_io.h
@@ -59,6 +59,28 @@
  */
 #define EXTENT_PAGE_PRIVATE 1
 
+/*
+ * The extent buffer bitmap operations are done with byte granularity instead of
+ * word granularity for two reasons:
+ * 1. The bitmaps must be little-endian on disk.
+ * 2. Bitmap items are not guaranteed to be aligned to a word and therefore a
+ *    single word in a bitmap may straddle two pages in the extent buffer.
+ */
+#define BIT_BYTE(nr) ((nr) / BITS_PER_BYTE)
+#define BYTE_MASK ((1 << BITS_PER_BYTE) - 1)
+#define BITMAP_FIRST_BYTE_MASK(start) \
+	((BYTE_MASK << ((start) & (BITS_PER_BYTE - 1))) & BYTE_MASK)
+#define BITMAP_LAST_BYTE_MASK(nbits) \
+	(BYTE_MASK >> (-(nbits) & (BITS_PER_BYTE - 1)))
+
+static inline int le_test_bit(int nr, const u8 *addr)
+{
+	return 1U & (addr[BIT_BYTE(nr)] >> (nr & (BITS_PER_BYTE-1)));
+}
+
+extern void le_bitmap_set(u8 *map, unsigned int start, int len);
+extern void le_bitmap_clear(u8 *map, unsigned int start, int len);
+
 struct extent_state;
 struct btrfs_root;
 struct btrfs_io_bio;
diff --git a/fs/btrfs/free-space-tree.c b/fs/btrfs/free-space-tree.c
index 87e7e3d..8fd85bf 100644
--- a/fs/btrfs/free-space-tree.c
+++ b/fs/btrfs/free-space-tree.c
@@ -151,7 +151,7 @@ static inline u32 free_space_bitmap_size(u64 size, u32 sectorsize)
 	return DIV_ROUND_UP((u32)div_u64(size, sectorsize), BITS_PER_BYTE);
 }
 
-static unsigned long *alloc_bitmap(u32 bitmap_size)
+static u8 *alloc_bitmap(u32 bitmap_size)
 {
 	void *mem;
 
@@ -180,8 +180,7 @@ int convert_free_space_to_bitmaps(struct btrfs_trans_handle *trans,
 	struct btrfs_free_space_info *info;
 	struct btrfs_key key, found_key;
 	struct extent_buffer *leaf;
-	unsigned long *bitmap;
-	char *bitmap_cursor;
+	u8 *bitmap, *bitmap_cursor;
 	u64 start, end;
 	u64 bitmap_range, i;
 	u32 bitmap_size, flags, expected_extent_count;
@@ -231,7 +230,7 @@ int convert_free_space_to_bitmaps(struct btrfs_trans_handle *trans,
 						block_group->sectorsize);
 				last = div_u64(found_key.objectid + found_key.offset - start,
 					       block_group->sectorsize);
-				bitmap_set(bitmap, first, last - first);
+				le_bitmap_set(bitmap, first, last - first);
 
 				extent_count++;
 				nr++;
@@ -269,7 +268,7 @@ int convert_free_space_to_bitmaps(struct btrfs_trans_handle *trans,
 		goto out;
 	}
 
-	bitmap_cursor = (char *)bitmap;
+	bitmap_cursor = bitmap;
 	bitmap_range = block_group->sectorsize * BTRFS_FREE_SPACE_BITMAP_BITS;
 	i = start;
 	while (i < end) {
@@ -318,7 +317,7 @@ int convert_free_space_to_extents(struct btrfs_trans_handle *trans,
 	struct btrfs_free_space_info *info;
 	struct btrfs_key key, found_key;
 	struct extent_buffer *leaf;
-	unsigned long *bitmap;
+	u8 *bitmap;
 	u64 start, end;
 	/* Initialize to silence GCC. */
 	u64 extent_start = 0;
@@ -362,7 +361,7 @@ int convert_free_space_to_extents(struct btrfs_trans_handle *trans,
 				break;
 			} else if (found_key.type == BTRFS_FREE_SPACE_BITMAP_KEY) {
 				unsigned long ptr;
-				char *bitmap_cursor;
+				u8 *bitmap_cursor;
 				u32 bitmap_pos, data_size;
 
 				ASSERT(found_key.objectid >= start);
@@ -372,7 +371,7 @@ int convert_free_space_to_extents(struct btrfs_trans_handle *trans,
 				bitmap_pos = div_u64(found_key.objectid - start,
 						     block_group->sectorsize *
 						     BITS_PER_BYTE);
-				bitmap_cursor = ((char *)bitmap) + bitmap_pos;
+				bitmap_cursor = bitmap + bitmap_pos;
 				data_size = free_space_bitmap_size(found_key.offset,
 								   block_group->sectorsize);
 
@@ -409,7 +408,7 @@ int convert_free_space_to_extents(struct btrfs_trans_handle *trans,
 	offset = start;
 	bitnr = 0;
 	while (offset < end) {
-		bit = !!test_bit(bitnr, bitmap);
+		bit = !!le_test_bit(bitnr, bitmap);
 		if (prev_bit == 0 && bit == 1) {
 			extent_start = offset;
 		} else if (prev_bit == 1 && bit == 0) {
-- 
2.10.0

Re: [PATCH v2 1/6] Btrfs: fix free space tree bitmaps on big-endian systems

[Holger Hoffstätte]

On Thu, 22 Sep 2016 17:24:20 -0700, Omar Sandoval wrote:

> From: Omar Sandoval <osandov@fb.com>
> 
> In convert_free_space_to_{bitmaps,extents}(), we buffer the free space
> bitmaps in memory and copy them directly to/from the extent buffers with
> {read,write}_extent_buffer(). The extent buffer bitmap helpers use byte
> granularity, which is equivalent to a little-endian bitmap. This means
> that on big-endian systems, the in-memory bitmaps will be written to
> disk byte-swapped. To fix this, use byte-granularity for the bitmaps in
> memory.
> 
> Fixes: a5ed91828518 ("Btrfs: implement the free space B-tree")
> Cc: stable@vger.kernel.org # 4.5+
> Signed-off-by: Omar Sandoval <osandov@fb.com>

Tested-by: Holger Hoffstätte <holger@applied-asynchrony.com>
(only for regressions on little-endian x86-64)

Thanks!
Holger

[PATCH v2 2/6] Btrfs: fix mount -o clear_cache,space_cache=v2

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

We moved the code for creating the free space tree the first time that
it's enabled, but didn't move the clearing code along with it. This
breaks my (undocumented) intention that `mount -o
clear_cache,space_cache=v2` would clear the free space tree and then
recreate it.

Fixes: 511711af91f2 ("btrfs: don't run delayed references while we are creating the free space tree")
Cc: stable@vger.kernel.org # 4.5+
Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 fs/btrfs/disk-io.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 54bc8c7..c0bfc6c 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -3129,6 +3129,18 @@ retry_root_backup:
 	if (sb->s_flags & MS_RDONLY)
 		return 0;
 
+	if (btrfs_test_opt(fs_info, CLEAR_CACHE) &&
+	    btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE)) {
+		btrfs_info(fs_info, "clearing free space tree");
+		ret = btrfs_clear_free_space_tree(fs_info);
+		if (ret) {
+			btrfs_warn(fs_info,
+				   "failed to clear free space tree: %d", ret);
+			close_ctree(tree_root);
+			return ret;
+		}
+	}
+
 	if (btrfs_test_opt(tree_root->fs_info, FREE_SPACE_TREE) &&
 	    !btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE)) {
 		btrfs_info(fs_info, "creating free space tree");
@@ -3166,18 +3178,6 @@ retry_root_backup:
 
 	btrfs_qgroup_rescan_resume(fs_info);
 
-	if (btrfs_test_opt(tree_root->fs_info, CLEAR_CACHE) &&
-	    btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE)) {
-		btrfs_info(fs_info, "clearing free space tree");
-		ret = btrfs_clear_free_space_tree(fs_info);
-		if (ret) {
-			btrfs_warn(fs_info,
-				"failed to clear free space tree: %d", ret);
-			close_ctree(tree_root);
-			return ret;
-		}
-	}
-
 	if (!fs_info->uuid_root) {
 		btrfs_info(fs_info, "creating UUID tree");
 		ret = btrfs_create_uuid_tree(fs_info);
-- 
2.10.0

Re: [PATCH v2 2/6] Btrfs: fix mount -o clear_cache,space_cache=v2

[Holger Hoffstätte]

On Thu, 22 Sep 2016 17:24:21 -0700, Omar Sandoval wrote:

> From: Omar Sandoval <osandov@fb.com>
> 
> We moved the code for creating the free space tree the first time that
> it's enabled, but didn't move the clearing code along with it. This
> breaks my (undocumented) intention that `mount -o
> clear_cache,space_cache=v2` would clear the free space tree and then
> recreate it.
> 
> Fixes: 511711af91f2 ("btrfs: don't run delayed references while we are creating the free space tree")
> Cc: stable@vger.kernel.org # 4.5+
> Signed-off-by: Omar Sandoval <osandov@fb.com>

Tested-by: Holger Hoffstätte <holger@applied-asynchrony.com>

Thanks!
Holger

[PATCH v2 3/6] Btrfs: catch invalid free space trees

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

There are two separate issues that can lead to corrupted free space
trees.

1. The free space tree bitmaps had an endianness issue on big-endian
   systems which is fixed by an earlier patch in this series.
2. btrfs-progs before v4.7.3 modified filesystems without updating the
   free space tree.

To catch both of these issues at once, we need to force the free space
tree to be rebuilt. To do so, add a FREE_SPACE_TREE_VALID compat_ro bit.
If the bit isn't set, we know that it was either produced by a broken
big-endian kernel or may have been corrupted by btrfs-progs.

This also provides us with a way to add rudimentary read-write support
for the free space tree to btrfs-progs: it can just clear this bit and
have the kernel rebuild the free space tree.

Cc: stable@vger.kernel.org # 4.5+
Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 fs/btrfs/ctree.h           |  3 ++-
 fs/btrfs/disk-io.c         |  9 +++++++++
 fs/btrfs/free-space-tree.c |  2 ++
 include/uapi/linux/btrfs.h | 10 +++++++++-
 4 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 33fe035..791e47c 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -251,7 +251,8 @@ struct btrfs_super_block {
 #define BTRFS_FEATURE_COMPAT_SAFE_CLEAR		0ULL
 
 #define BTRFS_FEATURE_COMPAT_RO_SUPP			\
-	(BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE)
+	(BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE |	\
+	 BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID)
 
 #define BTRFS_FEATURE_COMPAT_RO_SAFE_SET	0ULL
 #define BTRFS_FEATURE_COMPAT_RO_SAFE_CLEAR	0ULL
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index c0bfc6c..3dede6d 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -2566,6 +2566,7 @@ int open_ctree(struct super_block *sb,
 	int num_backups_tried = 0;
 	int backup_index = 0;
 	int max_active;
+	int clear_free_space_tree = 0;
 
 	tree_root = fs_info->tree_root = btrfs_alloc_root(fs_info, GFP_KERNEL);
 	chunk_root = fs_info->chunk_root = btrfs_alloc_root(fs_info, GFP_KERNEL);
@@ -3131,6 +3132,14 @@ retry_root_backup:
 
 	if (btrfs_test_opt(fs_info, CLEAR_CACHE) &&
 	    btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE)) {
+		clear_free_space_tree = 1;
+	} else if (btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE) &&
+		   !btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE_VALID)) {
+		btrfs_warn(fs_info, "free space tree is invalid");
+		clear_free_space_tree = 1;
+	}
+
+	if (clear_free_space_tree) {
 		btrfs_info(fs_info, "clearing free space tree");
 		ret = btrfs_clear_free_space_tree(fs_info);
 		if (ret) {
diff --git a/fs/btrfs/free-space-tree.c b/fs/btrfs/free-space-tree.c
index 8fd85bf..ea605ff 100644
--- a/fs/btrfs/free-space-tree.c
+++ b/fs/btrfs/free-space-tree.c
@@ -1182,6 +1182,7 @@ int btrfs_create_free_space_tree(struct btrfs_fs_info *fs_info)
 	}
 
 	btrfs_set_fs_compat_ro(fs_info, FREE_SPACE_TREE);
+	btrfs_set_fs_compat_ro(fs_info, FREE_SPACE_TREE_VALID);
 	fs_info->creating_free_space_tree = 0;
 
 	ret = btrfs_commit_transaction(trans, tree_root);
@@ -1250,6 +1251,7 @@ int btrfs_clear_free_space_tree(struct btrfs_fs_info *fs_info)
 		return PTR_ERR(trans);
 
 	btrfs_clear_fs_compat_ro(fs_info, FREE_SPACE_TREE);
+	btrfs_clear_fs_compat_ro(fs_info, FREE_SPACE_TREE_VALID);
 	fs_info->free_space_root = NULL;
 
 	ret = clear_free_space_tree(trans, free_space_root);
diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h
index ac5eacd..549ecf4 100644
--- a/include/uapi/linux/btrfs.h
+++ b/include/uapi/linux/btrfs.h
@@ -239,7 +239,15 @@ struct btrfs_ioctl_fs_info_args {
  * Used by:
  * struct btrfs_ioctl_feature_flags
  */
-#define BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE	(1ULL << 0)
+#define BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE		(1ULL << 0)
+/*
+ * Older kernels on big-endian systems produced broken free space tree bitmaps,
+ * and btrfs-progs also used to corrupt the free space tree. If this bit is
+ * clear, then the free space tree cannot be trusted. btrfs-progs can also
+ * intentionally clear this bit to ask the kernel to rebuild the free space
+ * tree.
+ */
+#define BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID	(1ULL << 1)
 
 #define BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF	(1ULL << 0)
 #define BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL	(1ULL << 1)
-- 
2.10.0

Re: [PATCH v2 3/6] Btrfs: catch invalid free space trees

[Holger Hoffstätte]

On Thu, 22 Sep 2016 17:24:22 -0700, Omar Sandoval wrote:

> From: Omar Sandoval <osandov@fb.com>
> 
> There are two separate issues that can lead to corrupted free space
> trees.
> 
> 1. The free space tree bitmaps had an endianness issue on big-endian
>    systems which is fixed by an earlier patch in this series.
> 2. btrfs-progs before v4.7.3 modified filesystems without updating the
>    free space tree.
> 
> To catch both of these issues at once, we need to force the free space
> tree to be rebuilt. To do so, add a FREE_SPACE_TREE_VALID compat_ro bit.
> If the bit isn't set, we know that it was either produced by a broken
> big-endian kernel or may have been corrupted by btrfs-progs.
> 
> This also provides us with a way to add rudimentary read-write support
> for the free space tree to btrfs-progs: it can just clear this bit and
> have the kernel rebuild the free space tree.
> 
> Cc: stable@vger.kernel.org # 4.5+
> Signed-off-by: Omar Sandoval <osandov@fb.com>

Tested-by: Holger Hoffstätte <holger@applied-asynchrony.com>

Initial mount with this successfully detected the absent _VALID bit;
fst was rebuilt, subsequent mounts are good without rebuild. \o/

Thanks!
Holger

Re: [PATCH v2 3/6] Btrfs: catch invalid free space trees

[Hans van Kranenburg]

On 09/23/2016 02:24 AM, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
> 
> There are two separate issues that can lead to corrupted free space
> trees.
> 
> 1. The free space tree bitmaps had an endianness issue on big-endian
>    systems which is fixed by an earlier patch in this series.
> 2. btrfs-progs before v4.7.3 modified filesystems without updating the
>    free space tree.
> 
> To catch both of these issues at once, we need to force the free space
> tree to be rebuilt. To do so, add a FREE_SPACE_TREE_VALID compat_ro bit.
> If the bit isn't set, we know that it was either produced by a broken
> big-endian kernel or may have been corrupted by btrfs-progs.

This tekst will be read by anyone git blaming the source to find out
what FREE_SPACE_TREE_VALID does, and maybe to find an answer to why
their filesystem just got corrupted after using progs < v4.7.3, even if
they run a new kernel which knows about this bit.

Since the above text suggests this situation can be dealt with, the text
is a bit misleading/incomplete. The construction with the bit requires
active cooperation from whatever external tool that is changing the fs,
to also flip this bit, to keep the filesystem from subsequently
corrupting itself.

So, starting to use this bit can only detect corruption by btrfs-progs
before v4.7.3 once, and only exactly once.

My suggestion is to just add a sentence like the following after "[...]
may have been corrupted by btrfs-progs.": "Caution: Since btrfs-progs
before v4.7.3 will not clear this bit after modifying the filesystem,
keeping to use these older versions will again result in an inconsistent
free space tree, without having an ability to detect this."

> This also provides us with a way to add rudimentary read-write support
> for the free space tree to btrfs-progs: it can just clear this bit and
> have the kernel rebuild the free space tree.
> 
> Cc: stable@vger.kernel.org # 4.5+
> Signed-off-by: Omar Sandoval <osandov@fb.com>
> ---
>  fs/btrfs/ctree.h           |  3 ++-
>  fs/btrfs/disk-io.c         |  9 +++++++++
>  fs/btrfs/free-space-tree.c |  2 ++
>  include/uapi/linux/btrfs.h | 10 +++++++++-
>  4 files changed, 22 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index 33fe035..791e47c 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -251,7 +251,8 @@ struct btrfs_super_block {
>  #define BTRFS_FEATURE_COMPAT_SAFE_CLEAR		0ULL
>  
>  #define BTRFS_FEATURE_COMPAT_RO_SUPP			\
> -	(BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE)
> +	(BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE |	\
> +	 BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID)
>  
>  #define BTRFS_FEATURE_COMPAT_RO_SAFE_SET	0ULL
>  #define BTRFS_FEATURE_COMPAT_RO_SAFE_CLEAR	0ULL
> diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
> index c0bfc6c..3dede6d 100644
> --- a/fs/btrfs/disk-io.c
> +++ b/fs/btrfs/disk-io.c
> @@ -2566,6 +2566,7 @@ int open_ctree(struct super_block *sb,
>  	int num_backups_tried = 0;
>  	int backup_index = 0;
>  	int max_active;
> +	int clear_free_space_tree = 0;
>  
>  	tree_root = fs_info->tree_root = btrfs_alloc_root(fs_info, GFP_KERNEL);
>  	chunk_root = fs_info->chunk_root = btrfs_alloc_root(fs_info, GFP_KERNEL);
> @@ -3131,6 +3132,14 @@ retry_root_backup:
>  
>  	if (btrfs_test_opt(fs_info, CLEAR_CACHE) &&
>  	    btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE)) {
> +		clear_free_space_tree = 1;
> +	} else if (btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE) &&
> +		   !btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE_VALID)) {
> +		btrfs_warn(fs_info, "free space tree is invalid");
> +		clear_free_space_tree = 1;
> +	}
> +
> +	if (clear_free_space_tree) {
>  		btrfs_info(fs_info, "clearing free space tree");
>  		ret = btrfs_clear_free_space_tree(fs_info);
>  		if (ret) {
> diff --git a/fs/btrfs/free-space-tree.c b/fs/btrfs/free-space-tree.c
> index 8fd85bf..ea605ff 100644
> --- a/fs/btrfs/free-space-tree.c
> +++ b/fs/btrfs/free-space-tree.c
> @@ -1182,6 +1182,7 @@ int btrfs_create_free_space_tree(struct btrfs_fs_info *fs_info)
>  	}
>  
>  	btrfs_set_fs_compat_ro(fs_info, FREE_SPACE_TREE);
> +	btrfs_set_fs_compat_ro(fs_info, FREE_SPACE_TREE_VALID);
>  	fs_info->creating_free_space_tree = 0;
>  
>  	ret = btrfs_commit_transaction(trans, tree_root);
> @@ -1250,6 +1251,7 @@ int btrfs_clear_free_space_tree(struct btrfs_fs_info *fs_info)
>  		return PTR_ERR(trans);
>  
>  	btrfs_clear_fs_compat_ro(fs_info, FREE_SPACE_TREE);
> +	btrfs_clear_fs_compat_ro(fs_info, FREE_SPACE_TREE_VALID);
>  	fs_info->free_space_root = NULL;
>  
>  	ret = clear_free_space_tree(trans, free_space_root);
> diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h
> index ac5eacd..549ecf4 100644
> --- a/include/uapi/linux/btrfs.h
> +++ b/include/uapi/linux/btrfs.h
> @@ -239,7 +239,15 @@ struct btrfs_ioctl_fs_info_args {
>   * Used by:
>   * struct btrfs_ioctl_feature_flags
>   */
> -#define BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE	(1ULL << 0)
> +#define BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE		(1ULL << 0)
> +/*
> + * Older kernels on big-endian systems produced broken free space tree bitmaps,
> + * and btrfs-progs also used to corrupt the free space tree. If this bit is
> + * clear, then the free space tree cannot be trusted. btrfs-progs can also
> + * intentionally clear this bit to ask the kernel to rebuild the free space
> + * tree.
> + */
> +#define BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID	(1ULL << 1)
>  
>  #define BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF	(1ULL << 0)
>  #define BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL	(1ULL << 1)
> 


-- 
Hans van Kranenburg

Re: [PATCH v2 3/6] Btrfs: catch invalid free space trees

[Omar Sandoval]

On Sat, Sep 24, 2016 at 09:50:53PM +0200, Hans van Kranenburg wrote:
> On 09/23/2016 02:24 AM, Omar Sandoval wrote:
> > From: Omar Sandoval <osandov@fb.com>
> > 
> > There are two separate issues that can lead to corrupted free space
> > trees.
> > 
> > 1. The free space tree bitmaps had an endianness issue on big-endian
> >    systems which is fixed by an earlier patch in this series.
> > 2. btrfs-progs before v4.7.3 modified filesystems without updating the
> >    free space tree.
> > 
> > To catch both of these issues at once, we need to force the free space
> > tree to be rebuilt. To do so, add a FREE_SPACE_TREE_VALID compat_ro bit.
> > If the bit isn't set, we know that it was either produced by a broken
> > big-endian kernel or may have been corrupted by btrfs-progs.
> 
> This tekst will be read by anyone git blaming the source to find out
> what FREE_SPACE_TREE_VALID does, and maybe to find an answer to why
> their filesystem just got corrupted after using progs < v4.7.3, even if
> they run a new kernel which knows about this bit.
> 
> Since the above text suggests this situation can be dealt with, the text
> is a bit misleading/incomplete. The construction with the bit requires
> active cooperation from whatever external tool that is changing the fs,
> to also flip this bit, to keep the filesystem from subsequently
> corrupting itself.
> 
> So, starting to use this bit can only detect corruption by btrfs-progs
> before v4.7.3 once, and only exactly once.
> 
> My suggestion is to just add a sentence like the following after "[...]
> may have been corrupted by btrfs-progs.": "Caution: Since btrfs-progs
> before v4.7.3 will not clear this bit after modifying the filesystem,
> keeping to use these older versions will again result in an inconsistent
> free space tree, without having an ability to detect this."

Like I mentioned in the cover letter of the series, btrfs-progs won't
touch filesystems with the new bit set:

┌[root@silver ~]
└# btrfs --version
btrfs-progs v4.7.2
┌[root@silver ~]
└# btrfstune -x /dev/vdb1
couldn't open RDWR because of unsupported option features (2).
Open ctree failed

That's the point of compat bits. They're a whitelist rather than a
blacklist, and until we explicitly update btrfs-progs to allow that bit,
it will only allow read-only access.

What happened with the earlier FREE_SPACE_TREE compat_ro bit is that we
mistakenly added it to the mask of supported compat_ro bits before it
was safe to do so.

-- 
Omar

Re: [PATCH v2 3/6] Btrfs: catch invalid free space trees

[Hans van Kranenburg]

On 09/26/2016 07:39 PM, Omar Sandoval wrote:
> On Sat, Sep 24, 2016 at 09:50:53PM +0200, Hans van Kranenburg wrote:
>> On 09/23/2016 02:24 AM, Omar Sandoval wrote:
>>> From: Omar Sandoval <osandov@fb.com>
>>>
>>> There are two separate issues that can lead to corrupted free space
>>> trees.
>>>
>>> 1. The free space tree bitmaps had an endianness issue on big-endian
>>>    systems which is fixed by an earlier patch in this series.
>>> 2. btrfs-progs before v4.7.3 modified filesystems without updating the
>>>    free space tree.
>>>
>>> To catch both of these issues at once, we need to force the free space
>>> tree to be rebuilt. To do so, add a FREE_SPACE_TREE_VALID compat_ro bit.
>>> If the bit isn't set, we know that it was either produced by a broken
>>> big-endian kernel or may have been corrupted by btrfs-progs.
>>
>> This tekst will be read by anyone git blaming the source to find out
>> what FREE_SPACE_TREE_VALID does, and maybe to find an answer to why
>> their filesystem just got corrupted after using progs < v4.7.3, even if
>> they run a new kernel which knows about this bit.
>>
>> Since the above text suggests this situation can be dealt with, the text
>> is a bit misleading/incomplete. The construction with the bit requires
>> active cooperation from whatever external tool that is changing the fs,
>> to also flip this bit, to keep the filesystem from subsequently
>> corrupting itself.
>>
>> So, starting to use this bit can only detect corruption by btrfs-progs
>> before v4.7.3 once, and only exactly once.
>>
>> My suggestion is to just add a sentence like the following after "[...]
>> may have been corrupted by btrfs-progs.": "Caution: Since btrfs-progs
>> before v4.7.3 will not clear this bit after modifying the filesystem,
>> keeping to use these older versions will again result in an inconsistent
>> free space tree, without having an ability to detect this."
> 
> Like I mentioned in the cover letter of the series, btrfs-progs won't
> touch filesystems with the new bit set:
> 
> ┌[root@silver ~]
> └# btrfs --version
> btrfs-progs v4.7.2
> ┌[root@silver ~]
> └# btrfstune -x /dev/vdb1
> couldn't open RDWR because of unsupported option features (2).
> Open ctree failed
> 
> That's the point of compat bits. They're a whitelist rather than a
> blacklist, and until we explicitly update btrfs-progs to allow that bit,
> it will only allow read-only access.
> 
> What happened with the earlier FREE_SPACE_TREE compat_ro bit is that we
> mistakenly added it to the mask of supported compat_ro bits before it
> was safe to do so.

Aha! Now I see. If it sees something from the future, it just thinks:
"whoa, not going to touch this".

Thanks for your patience. :)

-- 
Hans van Kranenburg

Re: [PATCH v2 3/6] Btrfs: catch invalid free space trees

[Omar Sandoval]

On Mon, Sep 26, 2016 at 07:46:02PM +0200, Hans van Kranenburg wrote:
> On 09/26/2016 07:39 PM, Omar Sandoval wrote:
> > On Sat, Sep 24, 2016 at 09:50:53PM +0200, Hans van Kranenburg wrote:
> >> On 09/23/2016 02:24 AM, Omar Sandoval wrote:
> >>> From: Omar Sandoval <osandov@fb.com>
> >>>
> >>> There are two separate issues that can lead to corrupted free space
> >>> trees.
> >>>
> >>> 1. The free space tree bitmaps had an endianness issue on big-endian
> >>>    systems which is fixed by an earlier patch in this series.
> >>> 2. btrfs-progs before v4.7.3 modified filesystems without updating the
> >>>    free space tree.
> >>>
> >>> To catch both of these issues at once, we need to force the free space
> >>> tree to be rebuilt. To do so, add a FREE_SPACE_TREE_VALID compat_ro bit.
> >>> If the bit isn't set, we know that it was either produced by a broken
> >>> big-endian kernel or may have been corrupted by btrfs-progs.
> >>
> >> This tekst will be read by anyone git blaming the source to find out
> >> what FREE_SPACE_TREE_VALID does, and maybe to find an answer to why
> >> their filesystem just got corrupted after using progs < v4.7.3, even if
> >> they run a new kernel which knows about this bit.
> >>
> >> Since the above text suggests this situation can be dealt with, the text
> >> is a bit misleading/incomplete. The construction with the bit requires
> >> active cooperation from whatever external tool that is changing the fs,
> >> to also flip this bit, to keep the filesystem from subsequently
> >> corrupting itself.
> >>
> >> So, starting to use this bit can only detect corruption by btrfs-progs
> >> before v4.7.3 once, and only exactly once.
> >>
> >> My suggestion is to just add a sentence like the following after "[...]
> >> may have been corrupted by btrfs-progs.": "Caution: Since btrfs-progs
> >> before v4.7.3 will not clear this bit after modifying the filesystem,
> >> keeping to use these older versions will again result in an inconsistent
> >> free space tree, without having an ability to detect this."
> > 
> > Like I mentioned in the cover letter of the series, btrfs-progs won't
> > touch filesystems with the new bit set:
> > 
> > ┌[root@silver ~]
> > └# btrfs --version
> > btrfs-progs v4.7.2
> > ┌[root@silver ~]
> > └# btrfstune -x /dev/vdb1
> > couldn't open RDWR because of unsupported option features (2).
> > Open ctree failed
> > 
> > That's the point of compat bits. They're a whitelist rather than a
> > blacklist, and until we explicitly update btrfs-progs to allow that bit,
> > it will only allow read-only access.
> > 
> > What happened with the earlier FREE_SPACE_TREE compat_ro bit is that we
> > mistakenly added it to the mask of supported compat_ro bits before it
> > was safe to do so.
> 
> Aha! Now I see. If it sees something from the future, it just thinks:
> "whoa, not going to touch this".

Exactly.

> Thanks for your patience. :)

Thanks for taking a look :)

-- 
Omar

Re: [PATCH v2 3/6] Btrfs: catch invalid free space trees

[Omar Sandoval]

On Thu, Sep 22, 2016 at 05:24:22PM -0700, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
> 
> There are two separate issues that can lead to corrupted free space
> trees.
> 
> 1. The free space tree bitmaps had an endianness issue on big-endian
>    systems which is fixed by an earlier patch in this series.
> 2. btrfs-progs before v4.7.3 modified filesystems without updating the
>    free space tree.
> 
> To catch both of these issues at once, we need to force the free space
> tree to be rebuilt. To do so, add a FREE_SPACE_TREE_VALID compat_ro bit.
> If the bit isn't set, we know that it was either produced by a broken
> big-endian kernel or may have been corrupted by btrfs-progs.
> 
> This also provides us with a way to add rudimentary read-write support
> for the free space tree to btrfs-progs: it can just clear this bit and
> have the kernel rebuild the free space tree.
> 
> Cc: stable@vger.kernel.org # 4.5+
> Signed-off-by: Omar Sandoval <osandov@fb.com>
> ---
>  fs/btrfs/ctree.h           |  3 ++-
>  fs/btrfs/disk-io.c         |  9 +++++++++
>  fs/btrfs/free-space-tree.c |  2 ++
>  include/uapi/linux/btrfs.h | 10 +++++++++-
>  4 files changed, 22 insertions(+), 2 deletions(-)
> 

[snip]

> diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h
> index ac5eacd..549ecf4 100644
> --- a/include/uapi/linux/btrfs.h
> +++ b/include/uapi/linux/btrfs.h
> @@ -239,7 +239,15 @@ struct btrfs_ioctl_fs_info_args {
>   * Used by:
>   * struct btrfs_ioctl_feature_flags
>   */
> -#define BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE	(1ULL << 0)
> +#define BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE		(1ULL << 0)
> +/*
> + * Older kernels on big-endian systems produced broken free space tree bitmaps,
> + * and btrfs-progs also used to corrupt the free space tree. If this bit is
> + * clear, then the free space tree cannot be trusted. btrfs-progs can also
> + * intentionally clear this bit to ask the kernel to rebuild the free space
> + * tree.
> + */

Hm, I think I changed my mind about allowing btrfs-progs to clear the
bit intentionally. This creates a problem where we have a valid free
space tree, modify it with btrfs-progs and clear the bit, and then mount
it on an older kernel that doesn't check for the valid bit. If we really
wanted to, we could add yet another bit, say, FREE_SPACE_TREE_INVALID,
which prevents old kernels from mounting it, but I don't want to add
more hacks just because write support hasn't been implemented in progs
yet. That doesn't change this patch at all except for the comment here.
Should I resend it or can this be fixed on applying?

> +#define BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID	(1ULL << 1)
>  
>  #define BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF	(1ULL << 0)
>  #define BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL	(1ULL << 1)
> -- 
> 2.10.0
> 

-- 
Omar

Re: [PATCH v2 3/6] Btrfs: catch invalid free space trees

[David Sterba]

On Mon, Sep 26, 2016 at 04:13:29PM -0700, Omar Sandoval wrote:
> > +/*
> > + * Older kernels on big-endian systems produced broken free space tree bitmaps,
> > + * and btrfs-progs also used to corrupt the free space tree. If this bit is
> > + * clear, then the free space tree cannot be trusted. btrfs-progs can also
> > + * intentionally clear this bit to ask the kernel to rebuild the free space
> > + * tree.
> > + */
> 
> Hm, I think I changed my mind about allowing btrfs-progs to clear the
> bit intentionally. This creates a problem where we have a valid free
> space tree, modify it with btrfs-progs and clear the bit, and then mount
> it on an older kernel that doesn't check for the valid bit. If we really
> wanted to, we could add yet another bit, say, FREE_SPACE_TREE_INVALID,
> which prevents old kernels from mounting it, but I don't want to add
> more hacks just because write support hasn't been implemented in progs
> yet. That doesn't change this patch at all except for the comment here.

What you describe is possible to happen but still rare, the lowest
recommended kernel for general FST feature use will be at least 4.9. We
can describe the buggy kernel/tools combinations and recommended stafety
steps, like clearing the cache manually etc.

> Should I resend it or can this be fixed on applying?

I can update the wording.

[PATCH v2 4/6] Btrfs: fix extent buffer bitmap tests on big-endian systems

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

The in-memory bitmap code manipulates words and is therefore sensitive
to endianness, while the extent buffer bitmap code addresses bytes and
is byte-order agnostic. Because the byte addressing of the extent buffer
bitmaps is equivalent to a little-endian in-memory bitmap, the extent
buffer bitmap tests fail on big-endian systems.

34b3e6c92af1 ("Btrfs: self-tests: Fix extent buffer bitmap test fail on
BE system") worked around another endianness bug in the tests but missed
this one because ed9e4afdb055 ("Btrfs: self-tests: Execute page
straddling test only when nodesize < PAGE_SIZE") disables this part of
the test on ppc64. That change lost the original meaning of the test,
however. We really want to test that an equivalent series of operations
using the in-memory bitmap API and the extent buffer bitmap API produces
equivalent results.

To fix this, don't use memcmp_extent_buffer() or write_extent_buffer();
do everything bit-by-bit.

Reported-by: Anatoly Pugachev <matorola@gmail.com>
Tested-by: Anatoly Pugachev <matorola@gmail.com>
Tested-by: Feifei Xu <xufeifei@linux.vnet.ibm.com>
Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 fs/btrfs/tests/extent-io-tests.c | 87 +++++++++++++++++++++++-----------------
 1 file changed, 51 insertions(+), 36 deletions(-)

diff --git a/fs/btrfs/tests/extent-io-tests.c b/fs/btrfs/tests/extent-io-tests.c
index d19ab03..caad80b 100644
--- a/fs/btrfs/tests/extent-io-tests.c
+++ b/fs/btrfs/tests/extent-io-tests.c
@@ -273,20 +273,37 @@ out:
 	return ret;
 }
 
-/**
- * test_bit_in_byte - Determine whether a bit is set in a byte
- * @nr: bit number to test
- * @addr: Address to start counting from
- */
-static inline int test_bit_in_byte(int nr, const u8 *addr)
+static int check_eb_bitmap(unsigned long *bitmap, struct extent_buffer *eb,
+			   unsigned long len)
 {
-	return 1UL & (addr[nr / BITS_PER_BYTE] >> (nr & (BITS_PER_BYTE - 1)));
+	unsigned long i;
+
+	for (i = 0; i < len * BITS_PER_BYTE; i++) {
+		int bit, bit1;
+
+		bit = !!test_bit(i, bitmap);
+		bit1 = !!extent_buffer_test_bit(eb, 0, i);
+		if (bit1 != bit) {
+			test_msg("Bits do not match\n");
+			return -EINVAL;
+		}
+
+		bit1 = !!extent_buffer_test_bit(eb, i / BITS_PER_BYTE,
+						i % BITS_PER_BYTE);
+		if (bit1 != bit) {
+			test_msg("Offset bits do not match\n");
+			return -EINVAL;
+		}
+	}
+	return 0;
 }
 
 static int __test_eb_bitmaps(unsigned long *bitmap, struct extent_buffer *eb,
 			     unsigned long len)
 {
-	unsigned long i, x;
+	unsigned long i, j;
+	u32 x;
+	int ret;
 
 	memset(bitmap, 0, len);
 	memset_extent_buffer(eb, 0, 0, len);
@@ -297,16 +314,18 @@ static int __test_eb_bitmaps(unsigned long *bitmap, struct extent_buffer *eb,
 
 	bitmap_set(bitmap, 0, len * BITS_PER_BYTE);
 	extent_buffer_bitmap_set(eb, 0, 0, len * BITS_PER_BYTE);
-	if (memcmp_extent_buffer(eb, bitmap, 0, len) != 0) {
+	ret = check_eb_bitmap(bitmap, eb, len);
+	if (ret) {
 		test_msg("Setting all bits failed\n");
-		return -EINVAL;
+		return ret;
 	}
 
 	bitmap_clear(bitmap, 0, len * BITS_PER_BYTE);
 	extent_buffer_bitmap_clear(eb, 0, 0, len * BITS_PER_BYTE);
-	if (memcmp_extent_buffer(eb, bitmap, 0, len) != 0) {
+	ret = check_eb_bitmap(bitmap, eb, len);
+	if (ret) {
 		test_msg("Clearing all bits failed\n");
-		return -EINVAL;
+		return ret;
 	}
 
 	/* Straddling pages test */
@@ -316,9 +335,10 @@ static int __test_eb_bitmaps(unsigned long *bitmap, struct extent_buffer *eb,
 			sizeof(long) * BITS_PER_BYTE);
 		extent_buffer_bitmap_set(eb, PAGE_SIZE - sizeof(long) / 2, 0,
 					sizeof(long) * BITS_PER_BYTE);
-		if (memcmp_extent_buffer(eb, bitmap, 0, len) != 0) {
+		ret = check_eb_bitmap(bitmap, eb, len);
+		if (ret) {
 			test_msg("Setting straddling pages failed\n");
-			return -EINVAL;
+			return ret;
 		}
 
 		bitmap_set(bitmap, 0, len * BITS_PER_BYTE);
@@ -328,9 +348,10 @@ static int __test_eb_bitmaps(unsigned long *bitmap, struct extent_buffer *eb,
 		extent_buffer_bitmap_set(eb, 0, 0, len * BITS_PER_BYTE);
 		extent_buffer_bitmap_clear(eb, PAGE_SIZE - sizeof(long) / 2, 0,
 					sizeof(long) * BITS_PER_BYTE);
-		if (memcmp_extent_buffer(eb, bitmap, 0, len) != 0) {
+		ret = check_eb_bitmap(bitmap, eb, len);
+		if (ret) {
 			test_msg("Clearing straddling pages failed\n");
-			return -EINVAL;
+			return ret;
 		}
 	}
 
@@ -339,28 +360,22 @@ static int __test_eb_bitmaps(unsigned long *bitmap, struct extent_buffer *eb,
 	 * something repetitive that could miss some hypothetical off-by-n bug.
 	 */
 	x = 0;
-	for (i = 0; i < len / sizeof(long); i++) {
-		x = (0x19660dULL * (u64)x + 0x3c6ef35fULL) & 0xffffffffUL;
-		bitmap[i] = x;
-	}
-	write_extent_buffer(eb, bitmap, 0, len);
-
-	for (i = 0; i < len * BITS_PER_BYTE; i++) {
-		int bit, bit1;
-
-		bit = !!test_bit_in_byte(i, (u8 *)bitmap);
-		bit1 = !!extent_buffer_test_bit(eb, 0, i);
-		if (bit1 != bit) {
-			test_msg("Testing bit pattern failed\n");
-			return -EINVAL;
+	bitmap_clear(bitmap, 0, len * BITS_PER_BYTE);
+	extent_buffer_bitmap_clear(eb, 0, 0, len * BITS_PER_BYTE);
+	for (i = 0; i < len * BITS_PER_BYTE / 32; i++) {
+		x = (0x19660dULL * (u64)x + 0x3c6ef35fULL) & 0xffffffffU;
+		for (j = 0; j < 32; j++) {
+			if (x & (1U << j)) {
+				bitmap_set(bitmap, i * 32 + j, 1);
+				extent_buffer_bitmap_set(eb, 0, i * 32 + j, 1);
+			}
 		}
+	}
 
-		bit1 = !!extent_buffer_test_bit(eb, i / BITS_PER_BYTE,
-						i % BITS_PER_BYTE);
-		if (bit1 != bit) {
-			test_msg("Testing bit pattern with offset failed\n");
-			return -EINVAL;
-		}
+	ret = check_eb_bitmap(bitmap, eb, len);
+	if (ret) {
+		test_msg("Random bit pattern failed\n");
+		return ret;
 	}
 
 	return 0;
-- 
2.10.0

[PATCH v2 5/6] Btrfs: expand free space tree sanity tests to catch endianness bug

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

The free space tree format conversion functions were broken on
big-endian systems, but the sanity tests didn't catch it because all of
the operations were aligned to multiple words. This was meant to catch
any bugs in the extent buffer code's handling of high memory, but it
ended up hiding the endianness bug. Expand the tests to do both
sector-aligned and page-aligned operations.

Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 fs/btrfs/tests/free-space-tree-tests.c | 164 +++++++++++++++++++--------------
 1 file changed, 96 insertions(+), 68 deletions(-)

diff --git a/fs/btrfs/tests/free-space-tree-tests.c b/fs/btrfs/tests/free-space-tree-tests.c
index 7508d3b..c449da4 100644
--- a/fs/btrfs/tests/free-space-tree-tests.c
+++ b/fs/btrfs/tests/free-space-tree-tests.c
@@ -27,12 +27,6 @@ struct free_space_extent {
 	u64 start, length;
 };
 
-/*
- * The test cases align their operations to this in order to hit some of the
- * edge cases in the bitmap code.
- */
-#define BITMAP_RANGE (BTRFS_FREE_SPACE_BITMAP_BITS * PAGE_SIZE)
-
 static int __check_free_space_extents(struct btrfs_trans_handle *trans,
 				      struct btrfs_fs_info *fs_info,
 				      struct btrfs_block_group_cache *cache,
@@ -168,7 +162,8 @@ static int check_free_space_extents(struct btrfs_trans_handle *trans,
 static int test_empty_block_group(struct btrfs_trans_handle *trans,
 				  struct btrfs_fs_info *fs_info,
 				  struct btrfs_block_group_cache *cache,
-				  struct btrfs_path *path)
+				  struct btrfs_path *path,
+				  u32 alignment)
 {
 	struct free_space_extent extents[] = {
 		{cache->key.objectid, cache->key.offset},
@@ -181,7 +176,8 @@ static int test_empty_block_group(struct btrfs_trans_handle *trans,
 static int test_remove_all(struct btrfs_trans_handle *trans,
 			   struct btrfs_fs_info *fs_info,
 			   struct btrfs_block_group_cache *cache,
-			   struct btrfs_path *path)
+			   struct btrfs_path *path,
+			   u32 alignment)
 {
 	struct free_space_extent extents[] = {};
 	int ret;
@@ -201,16 +197,17 @@ static int test_remove_all(struct btrfs_trans_handle *trans,
 static int test_remove_beginning(struct btrfs_trans_handle *trans,
 				 struct btrfs_fs_info *fs_info,
 				 struct btrfs_block_group_cache *cache,
-				 struct btrfs_path *path)
+				 struct btrfs_path *path,
+				 u32 alignment)
 {
 	struct free_space_extent extents[] = {
-		{cache->key.objectid + BITMAP_RANGE,
-			cache->key.offset - BITMAP_RANGE},
+		{cache->key.objectid + alignment,
+			cache->key.offset - alignment},
 	};
 	int ret;
 
 	ret = __remove_from_free_space_tree(trans, fs_info, cache, path,
-					    cache->key.objectid, BITMAP_RANGE);
+					    cache->key.objectid, alignment);
 	if (ret) {
 		test_msg("Could not remove free space\n");
 		return ret;
@@ -224,17 +221,18 @@ static int test_remove_beginning(struct btrfs_trans_handle *trans,
 static int test_remove_end(struct btrfs_trans_handle *trans,
 			   struct btrfs_fs_info *fs_info,
 			   struct btrfs_block_group_cache *cache,
-			   struct btrfs_path *path)
+			   struct btrfs_path *path,
+			   u32 alignment)
 {
 	struct free_space_extent extents[] = {
-		{cache->key.objectid, cache->key.offset - BITMAP_RANGE},
+		{cache->key.objectid, cache->key.offset - alignment},
 	};
 	int ret;
 
 	ret = __remove_from_free_space_tree(trans, fs_info, cache, path,
 					    cache->key.objectid +
-					    cache->key.offset - BITMAP_RANGE,
-					    BITMAP_RANGE);
+					    cache->key.offset - alignment,
+					    alignment);
 	if (ret) {
 		test_msg("Could not remove free space\n");
 		return ret;
@@ -247,18 +245,19 @@ static int test_remove_end(struct btrfs_trans_handle *trans,
 static int test_remove_middle(struct btrfs_trans_handle *trans,
 			      struct btrfs_fs_info *fs_info,
 			      struct btrfs_block_group_cache *cache,
-			      struct btrfs_path *path)
+			      struct btrfs_path *path,
+			      u32 alignment)
 {
 	struct free_space_extent extents[] = {
-		{cache->key.objectid, BITMAP_RANGE},
-		{cache->key.objectid + 2 * BITMAP_RANGE,
-			cache->key.offset - 2 * BITMAP_RANGE},
+		{cache->key.objectid, alignment},
+		{cache->key.objectid + 2 * alignment,
+			cache->key.offset - 2 * alignment},
 	};
 	int ret;
 
 	ret = __remove_from_free_space_tree(trans, fs_info, cache, path,
-					    cache->key.objectid + BITMAP_RANGE,
-					    BITMAP_RANGE);
+					    cache->key.objectid + alignment,
+					    alignment);
 	if (ret) {
 		test_msg("Could not remove free space\n");
 		return ret;
@@ -271,10 +270,11 @@ static int test_remove_middle(struct btrfs_trans_handle *trans,
 static int test_merge_left(struct btrfs_trans_handle *trans,
 			   struct btrfs_fs_info *fs_info,
 			   struct btrfs_block_group_cache *cache,
-			   struct btrfs_path *path)
+			   struct btrfs_path *path,
+			   u32 alignment)
 {
 	struct free_space_extent extents[] = {
-		{cache->key.objectid, 2 * BITMAP_RANGE},
+		{cache->key.objectid, 2 * alignment},
 	};
 	int ret;
 
@@ -287,15 +287,15 @@ static int test_merge_left(struct btrfs_trans_handle *trans,
 	}
 
 	ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-				       cache->key.objectid, BITMAP_RANGE);
+				       cache->key.objectid, alignment);
 	if (ret) {
 		test_msg("Could not add free space\n");
 		return ret;
 	}
 
 	ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-				       cache->key.objectid + BITMAP_RANGE,
-				       BITMAP_RANGE);
+				       cache->key.objectid + alignment,
+				       alignment);
 	if (ret) {
 		test_msg("Could not add free space\n");
 		return ret;
@@ -308,10 +308,11 @@ static int test_merge_left(struct btrfs_trans_handle *trans,
 static int test_merge_right(struct btrfs_trans_handle *trans,
 			   struct btrfs_fs_info *fs_info,
 			   struct btrfs_block_group_cache *cache,
-			   struct btrfs_path *path)
+			   struct btrfs_path *path,
+			   u32 alignment)
 {
 	struct free_space_extent extents[] = {
-		{cache->key.objectid + BITMAP_RANGE, 2 * BITMAP_RANGE},
+		{cache->key.objectid + alignment, 2 * alignment},
 	};
 	int ret;
 
@@ -324,16 +325,16 @@ static int test_merge_right(struct btrfs_trans_handle *trans,
 	}
 
 	ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-				       cache->key.objectid + 2 * BITMAP_RANGE,
-				       BITMAP_RANGE);
+				       cache->key.objectid + 2 * alignment,
+				       alignment);
 	if (ret) {
 		test_msg("Could not add free space\n");
 		return ret;
 	}
 
 	ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-				       cache->key.objectid + BITMAP_RANGE,
-				       BITMAP_RANGE);
+				       cache->key.objectid + alignment,
+				       alignment);
 	if (ret) {
 		test_msg("Could not add free space\n");
 		return ret;
@@ -346,10 +347,11 @@ static int test_merge_right(struct btrfs_trans_handle *trans,
 static int test_merge_both(struct btrfs_trans_handle *trans,
 			   struct btrfs_fs_info *fs_info,
 			   struct btrfs_block_group_cache *cache,
-			   struct btrfs_path *path)
+			   struct btrfs_path *path,
+			   u32 alignment)
 {
 	struct free_space_extent extents[] = {
-		{cache->key.objectid, 3 * BITMAP_RANGE},
+		{cache->key.objectid, 3 * alignment},
 	};
 	int ret;
 
@@ -362,23 +364,23 @@ static int test_merge_both(struct btrfs_trans_handle *trans,
 	}
 
 	ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-				       cache->key.objectid, BITMAP_RANGE);
+				       cache->key.objectid, alignment);
 	if (ret) {
 		test_msg("Could not add free space\n");
 		return ret;
 	}
 
 	ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-				       cache->key.objectid + 2 * BITMAP_RANGE,
-				       BITMAP_RANGE);
+				       cache->key.objectid + 2 * alignment,
+				       alignment);
 	if (ret) {
 		test_msg("Could not add free space\n");
 		return ret;
 	}
 
 	ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-				       cache->key.objectid + BITMAP_RANGE,
-				       BITMAP_RANGE);
+				       cache->key.objectid + alignment,
+				       alignment);
 	if (ret) {
 		test_msg("Could not add free space\n");
 		return ret;
@@ -391,12 +393,13 @@ static int test_merge_both(struct btrfs_trans_handle *trans,
 static int test_merge_none(struct btrfs_trans_handle *trans,
 			   struct btrfs_fs_info *fs_info,
 			   struct btrfs_block_group_cache *cache,
-			   struct btrfs_path *path)
+			   struct btrfs_path *path,
+			   u32 alignment)
 {
 	struct free_space_extent extents[] = {
-		{cache->key.objectid, BITMAP_RANGE},
-		{cache->key.objectid + 2 * BITMAP_RANGE, BITMAP_RANGE},
-		{cache->key.objectid + 4 * BITMAP_RANGE, BITMAP_RANGE},
+		{cache->key.objectid, alignment},
+		{cache->key.objectid + 2 * alignment, alignment},
+		{cache->key.objectid + 4 * alignment, alignment},
 	};
 	int ret;
 
@@ -409,23 +412,23 @@ static int test_merge_none(struct btrfs_trans_handle *trans,
 	}
 
 	ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-				       cache->key.objectid, BITMAP_RANGE);
+				       cache->key.objectid, alignment);
 	if (ret) {
 		test_msg("Could not add free space\n");
 		return ret;
 	}
 
 	ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-				       cache->key.objectid + 4 * BITMAP_RANGE,
-				       BITMAP_RANGE);
+				       cache->key.objectid + 4 * alignment,
+				       alignment);
 	if (ret) {
 		test_msg("Could not add free space\n");
 		return ret;
 	}
 
 	ret = __add_to_free_space_tree(trans, fs_info, cache, path,
-				       cache->key.objectid + 2 * BITMAP_RANGE,
-				       BITMAP_RANGE);
+				       cache->key.objectid + 2 * alignment,
+				       alignment);
 	if (ret) {
 		test_msg("Could not add free space\n");
 		return ret;
@@ -438,10 +441,11 @@ static int test_merge_none(struct btrfs_trans_handle *trans,
 typedef int (*test_func_t)(struct btrfs_trans_handle *,
 			   struct btrfs_fs_info *,
 			   struct btrfs_block_group_cache *,
-			   struct btrfs_path *);
+			   struct btrfs_path *,
+			   u32 alignment);
 
-static int run_test(test_func_t test_func, int bitmaps,
-		u32 sectorsize, u32 nodesize)
+static int run_test(test_func_t test_func, int bitmaps, u32 sectorsize,
+		    u32 nodesize, u32 alignment)
 {
 	struct btrfs_fs_info *fs_info;
 	struct btrfs_root *root = NULL;
@@ -480,7 +484,7 @@ static int run_test(test_func_t test_func, int bitmaps,
 	btrfs_set_header_nritems(root->node, 0);
 	root->alloc_bytenr += 2 * nodesize;
 
-	cache = btrfs_alloc_dummy_block_group(8 * BITMAP_RANGE, sectorsize);
+	cache = btrfs_alloc_dummy_block_group(8 * alignment, sectorsize);
 	if (!cache) {
 		test_msg("Couldn't allocate dummy block group cache\n");
 		ret = -ENOMEM;
@@ -514,7 +518,7 @@ static int run_test(test_func_t test_func, int bitmaps,
 		}
 	}
 
-	ret = test_func(&trans, root->fs_info, cache, path);
+	ret = test_func(&trans, root->fs_info, cache, path, alignment);
 	if (ret)
 		goto out;
 
@@ -539,15 +543,27 @@ out:
 	return ret;
 }
 
-static int run_test_both_formats(test_func_t test_func,
-	u32 sectorsize, u32 nodesize)
+static int run_test_both_formats(test_func_t test_func, u32 sectorsize,
+				 u32 nodesize, u32 alignment)
 {
+	int test_ret = 0;
 	int ret;
 
-	ret = run_test(test_func, 0, sectorsize, nodesize);
-	if (ret)
-		return ret;
-	return run_test(test_func, 1, sectorsize, nodesize);
+	ret = run_test(test_func, 0, sectorsize, nodesize, alignment);
+	if (ret) {
+		test_msg("%pf failed with extents, sectorsize=%u, nodesize=%u, alignment=%u\n",
+			 test_func, sectorsize, nodesize, alignment);
+		test_ret = ret;
+	}
+
+	ret = run_test(test_func, 1, sectorsize, nodesize, alignment);
+	if (ret) {
+		test_msg("%pf failed with bitmaps, sectorsize=%u, nodesize=%u, alignment=%u\n",
+			 test_func, sectorsize, nodesize, alignment);
+		test_ret = ret;
+	}
+
+	return test_ret;
 }
 
 int btrfs_test_free_space_tree(u32 sectorsize, u32 nodesize)
@@ -563,18 +579,30 @@ int btrfs_test_free_space_tree(u32 sectorsize, u32 nodesize)
 		test_merge_both,
 		test_merge_none,
 	};
+	u32 bitmap_alignment;
+	int test_ret = 0;
 	int i;
 
+	/*
+	 * Align some operations to a page to flush out bugs in the extent
+	 * buffer bitmap handling of highmem.
+	 */
+	bitmap_alignment = BTRFS_FREE_SPACE_BITMAP_BITS * PAGE_SIZE;
+
 	test_msg("Running free space tree tests\n");
 	for (i = 0; i < ARRAY_SIZE(tests); i++) {
-		int ret = run_test_both_formats(tests[i], sectorsize,
-			nodesize);
-		if (ret) {
-			test_msg("%pf : sectorsize %u failed\n",
-				tests[i], sectorsize);
-			return ret;
-		}
+		int ret;
+
+		ret = run_test_both_formats(tests[i], sectorsize, nodesize,
+					    sectorsize);
+		if (ret)
+			test_ret = ret;
+
+		ret = run_test_both_formats(tests[i], sectorsize, nodesize,
+					    bitmap_alignment);
+		if (ret)
+			test_ret = ret;
 	}
 
-	return 0;
+	return test_ret;
 }
-- 
2.10.0

[PATCH v2 6/6] Btrfs: use less memory for delalloc sanity tests

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

test_find_delalloc() allocates 256 MB worth of pages. That's all of the
RAM that my MIPS emulator has, so it ends up panicking after it OOM
kills everything. We don't actually need to use that much for this test.

Signed-off-by: Omar Sandoval <osandov@fb.com>
---
 fs/btrfs/tests/extent-io-tests.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/fs/btrfs/tests/extent-io-tests.c b/fs/btrfs/tests/extent-io-tests.c
index caad80b..ea9c91b 100644
--- a/fs/btrfs/tests/extent-io-tests.c
+++ b/fs/btrfs/tests/extent-io-tests.c
@@ -73,8 +73,8 @@ static int test_find_delalloc(u32 sectorsize)
 	struct page *page;
 	struct page *locked_page = NULL;
 	unsigned long index = 0;
-	u64 total_dirty = SZ_256M;
-	u64 max_bytes = SZ_128M;
+	u64 total_dirty = sectorsize * 256;
+	u64 max_bytes = total_dirty >> 1;
 	u64 start, end, test_start;
 	u64 found;
 	int ret = -EINVAL;
@@ -138,7 +138,7 @@ static int test_find_delalloc(u32 sectorsize)
 	 * |--- delalloc ---|
 	 *           |--- search ---|
 	 */
-	test_start = SZ_64M;
+	test_start = max_bytes >> 1;
 	locked_page = find_lock_page(inode->i_mapping,
 				     test_start >> PAGE_SHIFT);
 	if (!locked_page) {
@@ -226,7 +226,7 @@ static int test_find_delalloc(u32 sectorsize)
 	 * range we want to find.
 	 */
 	page = find_get_page(inode->i_mapping,
-			     (max_bytes + SZ_1M) >> PAGE_SHIFT);
+			     (max_bytes + PAGE_SIZE) >> PAGE_SHIFT);
 	if (!page) {
 		test_msg("Couldn't find our page\n");
 		goto out_bits;
-- 
2.10.0

Re: [PATCH v2 6/6] Btrfs: use less memory for delalloc sanity tests

[David Sterba]

On Thu, Sep 22, 2016 at 05:24:25PM -0700, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
> 
> test_find_delalloc() allocates 256 MB worth of pages. That's all of the
> RAM that my MIPS emulator has, so it ends up panicking after it OOM
> kills everything. We don't actually need to use that much for this test.

I'm not sure we should limit it that way as more bytes can give it more
stress. Can we do it somehow dynamically ? Like start with 256M and fall
back to the numbers you've used.

Or maybe start from the low bound and allocate until it fails with first
ENOMEM.

Re: [PATCH v2 6/6] Btrfs: use less memory for delalloc sanity tests

[Omar Sandoval]

On Fri, Sep 23, 2016 at 11:27:27AM +0200, David Sterba wrote:
> On Thu, Sep 22, 2016 at 05:24:25PM -0700, Omar Sandoval wrote:
> > From: Omar Sandoval <osandov@fb.com>
> > 
> > test_find_delalloc() allocates 256 MB worth of pages. That's all of the
> > RAM that my MIPS emulator has, so it ends up panicking after it OOM
> > kills everything. We don't actually need to use that much for this test.
> 
> I'm not sure we should limit it that way as more bytes can give it more
> stress. Can we do it somehow dynamically ? Like start with 256M and fall
> back to the numbers you've used.
> 
> Or maybe start from the low bound and allocate until it fails with first
> ENOMEM.

I figured out how to get qemu to do highmem on MIPS, so we can drop this
patch. It's reasonable to assume that the sanity tests are running on a
machine with a decent amount of memory.

-- 
Omar

Re: [PATCH v2 6/6] Btrfs: use less memory for delalloc sanity tests

[Omar Sandoval]

On Fri, Sep 23, 2016 at 09:52:01AM -0700, Omar Sandoval wrote:
> On Fri, Sep 23, 2016 at 11:27:27AM +0200, David Sterba wrote:
> > On Thu, Sep 22, 2016 at 05:24:25PM -0700, Omar Sandoval wrote:
> > > From: Omar Sandoval <osandov@fb.com>
> > > 
> > > test_find_delalloc() allocates 256 MB worth of pages. That's all of the
> > > RAM that my MIPS emulator has, so it ends up panicking after it OOM
> > > kills everything. We don't actually need to use that much for this test.
> > 
> > I'm not sure we should limit it that way as more bytes can give it more
> > stress. Can we do it somehow dynamically ? Like start with 256M and fall
> > back to the numbers you've used.
> > 
> > Or maybe start from the low bound and allocate until it fails with first
> > ENOMEM.
> 
> I figured out how to get qemu to do highmem on MIPS, so we can drop this
> patch. It's reasonable to assume that the sanity tests are running on a
> machine with a decent amount of memory.

I lied, that didn't fix the problem for me, since these page allocations
can't use highmem. Anyways, the tests seem to be testing more for logic
errors manipulating the extent io tree, so just scaling it down
shouldn't be a huge issue, right?

The problem with relying on ENOMEM for anything is that the OOM killer
will kick in before we actually get the ENOMEM. I tried adding
__GFP_NORETRY, which claims that it doesn't invoke the OOM killer, but
it still did :(

Anyways, Cc Josef to see what he thinks since he wrote the test.

-- 
Omar

Re: [PATCH v2 6/6] Btrfs: use less memory for delalloc sanity tests

[David Sterba]

On Fri, Sep 23, 2016 at 02:22:53PM -0700, Omar Sandoval wrote:
> On Fri, Sep 23, 2016 at 09:52:01AM -0700, Omar Sandoval wrote:
> > On Fri, Sep 23, 2016 at 11:27:27AM +0200, David Sterba wrote:
> > > On Thu, Sep 22, 2016 at 05:24:25PM -0700, Omar Sandoval wrote:
> > > > From: Omar Sandoval <osandov@fb.com>
> > > > 
> > > > test_find_delalloc() allocates 256 MB worth of pages. That's all of the
> > > > RAM that my MIPS emulator has, so it ends up panicking after it OOM
> > > > kills everything. We don't actually need to use that much for this test.
> > > 
> > > I'm not sure we should limit it that way as more bytes can give it more
> > > stress. Can we do it somehow dynamically ? Like start with 256M and fall
> > > back to the numbers you've used.
> > > 
> > > Or maybe start from the low bound and allocate until it fails with first
> > > ENOMEM.
> > 
> > I figured out how to get qemu to do highmem on MIPS, so we can drop this
> > patch. It's reasonable to assume that the sanity tests are running on a
> > machine with a decent amount of memory.
> 
> I lied, that didn't fix the problem for me, since these page allocations
> can't use highmem. Anyways, the tests seem to be testing more for logic
> errors manipulating the extent io tree, so just scaling it down
> shouldn't be a huge issue, right?
> 
> The problem with relying on ENOMEM for anything is that the OOM killer
> will kick in before we actually get the ENOMEM. I tried adding
> __GFP_NORETRY, which claims that it doesn't invoke the OOM killer, but
> it still did :(

I think we cannot rely on NORETRY behaviour wrt OOM killer anyway. For
now I'd rather leave out the patch and let people test on machines with
enough memory. Any furhter ideas to make it work on low memory are
welcome but I don't want to block this patchset.

Re: [PATCH v2 6/6] Btrfs: use less memory for delalloc sanity tests

[Omar Sandoval]

On Mon, Sep 26, 2016 at 05:58:02PM +0200, David Sterba wrote:
> On Fri, Sep 23, 2016 at 02:22:53PM -0700, Omar Sandoval wrote:
> > On Fri, Sep 23, 2016 at 09:52:01AM -0700, Omar Sandoval wrote:
> > > On Fri, Sep 23, 2016 at 11:27:27AM +0200, David Sterba wrote:
> > > > On Thu, Sep 22, 2016 at 05:24:25PM -0700, Omar Sandoval wrote:
> > > > > From: Omar Sandoval <osandov@fb.com>
> > > > > 
> > > > > test_find_delalloc() allocates 256 MB worth of pages. That's all of the
> > > > > RAM that my MIPS emulator has, so it ends up panicking after it OOM
> > > > > kills everything. We don't actually need to use that much for this test.
> > > > 
> > > > I'm not sure we should limit it that way as more bytes can give it more
> > > > stress. Can we do it somehow dynamically ? Like start with 256M and fall
> > > > back to the numbers you've used.
> > > > 
> > > > Or maybe start from the low bound and allocate until it fails with first
> > > > ENOMEM.
> > > 
> > > I figured out how to get qemu to do highmem on MIPS, so we can drop this
> > > patch. It's reasonable to assume that the sanity tests are running on a
> > > machine with a decent amount of memory.
> > 
> > I lied, that didn't fix the problem for me, since these page allocations
> > can't use highmem. Anyways, the tests seem to be testing more for logic
> > errors manipulating the extent io tree, so just scaling it down
> > shouldn't be a huge issue, right?
> > 
> > The problem with relying on ENOMEM for anything is that the OOM killer
> > will kick in before we actually get the ENOMEM. I tried adding
> > __GFP_NORETRY, which claims that it doesn't invoke the OOM killer, but
> > it still did :(
> 
> I think we cannot rely on NORETRY behaviour wrt OOM killer anyway. For
> now I'd rather leave out the patch and let people test on machines with
> enough memory. Any furhter ideas to make it work on low memory are
> welcome but I don't want to block this patchset.

Yeah that's fine, we can come back to this another time.

-- 
Omar

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[Anatoly Pugachev]

On Thu, Sep 22, 2016 at 05:22:31PM -0700, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
> 
> This is v2 of my earlier series "Btrfs: fix free space tree
> bitmaps+tests on big-endian systems" [1]. Patches 1, 4, and 5 are the
> same as patches 1, 2, and 3 from the original series. I've added patch 2
> to fix another bug I noticed (an xfstest went out earlier). Patch 3 is
> the result of the earlier discussion here [2]. Finally, patch 6 was
> necessary to get the sanity tests to run on my MIPS emulator.
> 
> This series applies to v4.8-rc7. The sanity tests pass on both x86-64
> and MIPS, and there are no xfstests regressions. Chandan and Anatoly,
> could you test these out as well?
> 
> I'm working on the btrfs-progs follow up, but these patches are safe
> without that -- the new FREE_SPACE_TREE_VALID bit will stop all versions
> of btrfs-progs from mounting read-write.

Omar,

applied patch to git kernel (v4.8-rc7-172-gbd5dbcb) cleanly. Did not used
btrfs-progs.git, but debian shipped 4.7.3-1 .

(re)booted to a newly patched kernel and used xfstests.git (v1.1.0-1328-g06d4001):

# mount tmpfs -t tmpfs -o size=26g /ramdisk
# cd /ramdisk
# fallocate -l 10g testvol1
# for i in 1 2 3 4; do fallocate -l 4g scratch${i}; done
# for i in *; do losetup -f $i; done
# losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE         DIO
/dev/loop0         0      0         0  0 /ramdisk/scratch1   0
/dev/loop1         0      0         0  0 /ramdisk/scratch2   0
/dev/loop2         0      0         0  0 /ramdisk/scratch3   0
/dev/loop3         0      0         0  0 /ramdisk/scratch4   0
/dev/loop4         0      0         0  0 /ramdisk/testvol1   0

mator@ttip:~/xfstests-dev$ cat local.config
export TEST_DEV=/dev/loop4
export TEST_DIR=/testvol
export SCRATCH_DEV_POOL="/dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3"
export SCRATCH_MNT=/mnt/scratch

root@ttip:/home/mator/xfstests-dev# mkfs.btrfs /dev/loop4
btrfs-progs v4.7.3
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM (10.00GiB) ...
Label:              (null)
UUID:
Node size:          16384
Sector size:        8192
Filesystem size:    10.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1    10.00GiB  /dev/loop4

root@ttip:/home/mator/xfstests-dev# ./check 'btrfs/*'
FSTYP         -- btrfs
PLATFORM      -- Linux/sparc64 ttip 4.8.0-rc7+
MKFS_OPTIONS  -- /dev/loop0
MOUNT_OPTIONS -- /dev/loop0 /mnt/scratch

_check_btrfs_filesystem: filesystem on /dev/loop4 is inconsistent (see /home/mator/xfstests-dev/results//check.full)
btrfs/001


(on another screen/tmux window shell)

$ pstree -A | grep check 
        |-sshd-+-sshd---sshd---bash---sudo---bash---check---001---mount

$ ps ax | grep mount
 76344 pts/0    D+     0:00 /bin/mount -t btrfs /dev/loop4 /testvol

$ cat /home/mator/xfstests-dev/results//check.full
_check_btrfs_filesystem: filesystem on /dev/loop4 is inconsistent
*** fsck.btrfs output ***
ERROR: cannot open device '/dev/loop4': Device or resource busy
Couldn't open file system
*** end fsck.btrfs output
*** mount output ***
sysfs on /sys type sysfs (rw,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16479064k,nr_inodes=2059883,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3314128k,mode=755)
/dev/vdiska2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=23073)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/vdiska1 on /boot type ext3 (rw,relatime,data=ordered)
/dev/vdiskb1 on /home type xfs (rw,relatime,attr2,inode64,noquota)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3314120k,mode=700,uid=1000,gid=1000)
tmpfs on /ramdisk type tmpfs (rw,relatime,size=27262976k)
/dev/loop0 on /mnt/scratch type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
*** end mount output

in kernel logs and on console:

[3184224.438566] BTRFS: device fsid b54ec0aa-4187-419d-9de1-5ef284cd3b32 devid 1 transid 5 /dev/loop4
[3184239.417845] BTRFS info (device loop4): disk space caching is enabled
[3184239.417865] BTRFS info (device loop4): has skinny extents
[3184239.417872] BTRFS info (device loop4): flagging fs with big metadata feature
[3184239.421147] BTRFS info (device loop4): creating UUID tree
[3184240.026601] BTRFS: device fsid 5383d227-9ea2-4514-8857-641c6e2e2063 devid 1 transid 5 /dev/loop0
[3184240.131927] BTRFS info (device loop0): disk space caching is enabled
[3184240.131956] BTRFS info (device loop0): has skinny extents
[3184240.131971] BTRFS info (device loop0): flagging fs with big metadata feature
[3184240.135182] BTRFS info (device loop0): creating UUID tree
[3184240.252534] BTRFS critical (device loop4): corrupt leaf, non-root leaf's nritems is 0: block=29556736,root=1, slot=0
[3184240.252558] BTRFS info (device loop4): leaf 29556736 total ptrs 0 free space 16283
[3184240.252567] BTRFS: assertion failed: 0, file: fs/btrfs/disk-io.c, line: 4059kernel BUG at fs/btrfs/ctree.h:3369!
[3184240.252581]               \|/ ____ \|/
[3184240.252581]               "@'/ .. \`@"
[3184240.252581]               /_| \__/ |_\
[3184240.252581]                  \__U_/
[3184240.252597] umount(76111): Kernel bad sw trap 5 [#1]
[3184240.252605] CPU: 21 PID: 76111 Comm: umount Not tainted 4.8.0-rc7+ #9
[3184240.252612] task: fff8000805feed20 task.stack: fff8000804824000
[3184240.252620] TSTATE: 0000004411001600 TPC: 00000000100d6ccc TNPC: 00000000100d6cd0 Y: 00000000    Not tainted
[3184240.252679] TPC: <btrfs_mark_buffer_dirty+0x1ac/0x1e0 [btrfs]>
[3184240.252686] g0: 0000000000000000 g1: 0000000000000000 g2: 0000000000000007 g3: 0000000000000000
[3184240.252694] g4: fff8000805feed20 g5: fff800082c66e000 g6: fff8000804824000 g7: fff800082ce00000
[3184240.252702] o0: 000000001019cac8 o1: 0000000000000d29 o2: 000000001019c930 o3: 0000000000000fdb
[3184240.252710] o4: 0000000001c30000 o5: 0000000000000000 sp: fff80008048263c1 ret_pc: 00000000100d6cc4
[3184240.252731] RPC: <btrfs_mark_buffer_dirty+0x1a4/0x1e0 [btrfs]>
[3184240.252738] l0: 000000000000154c l1: 0000000000002000 l2: 0000000000002000 l3: fff80008115fbf10
[3184240.252746] l4: fff8000805d7a698 l5: 00000000101b8400 l6: 0000000001000000 l7: 0000000000000000
[3184240.252754] i0: fff8000812f97968 i1: 0000000000000000 i2: 0700000000000000 i3: 000600001229fa48
[3184240.252762] i4: fff80008134fb000 i5: 0000000000000007 i6: fff8000804826471 i7: 00000000100a0388
[3184240.252780] I7: <__btrfs_cow_block+0x7e8/0xb80 [btrfs]>
[3184240.252786] Call Trace:
[3184240.252801]  [00000000100a0388] __btrfs_cow_block+0x7e8/0xb80 [btrfs]
[3184240.252816]  [00000000100a0948] btrfs_cow_block+0x1a8/0x340 [btrfs]
[3184240.252832]  [00000000100a6000] btrfs_search_slot+0x2a0/0xf60 [btrfs]
[3184240.252849]  [00000000100ccff4] btrfs_del_csums+0x1b4/0x6a0 [btrfs]
[3184240.252865]  [00000000100b4ecc] __btrfs_free_extent.isra.19+0x9ec/0x12c0 [btrfs]
[3184240.252884]  [00000000100ba788] __btrfs_run_delayed_refs+0x548/0x1980 [btrfs]
[3184240.252902]  [00000000100bfec8] btrfs_run_delayed_refs+0x88/0x2c0 [btrfs]
[3184240.252921]  [00000000100dcc20] commit_cowonly_roots+0xe0/0x5c0 [btrfs]
[3184240.252939]  [00000000100df9b4] btrfs_commit_transaction+0x534/0xd60 [btrfs]
[3184240.252959]  [00000000100d54c4] btrfs_commit_super+0x64/0x80 [btrfs]
[3184240.252977]  [00000000100d8a60] close_ctree+0x260/0x320 [btrfs]
[3184240.252992]  [0000000010096cd0] btrfs_put_super+0x10/0x20 [btrfs]
[3184240.253006]  [00000000005e7d74] generic_shutdown_super+0x74/0x100
[3184240.253013]  [00000000005e8050] kill_anon_super+0x10/0x40
[3184240.253028]  [00000000100969b4] btrfs_kill_super+0x14/0x100 [btrfs]
[3184240.253035]  [00000000005e8244] deactivate_locked_super+0x44/0x80
[3184240.253041] Disabling lock debugging due to kernel taint
[3184240.253057] Caller[00000000100a0388]: __btrfs_cow_block+0x7e8/0xb80 [btrfs]
[3184240.253072] Caller[00000000100a0948]: btrfs_cow_block+0x1a8/0x340 [btrfs]
[3184240.253088] Caller[00000000100a6000]: btrfs_search_slot+0x2a0/0xf60 [btrfs]
[3184240.253105] Caller[00000000100ccff4]: btrfs_del_csums+0x1b4/0x6a0 [btrfs]
[3184240.253121] Caller[00000000100b4ecc]: __btrfs_free_extent.isra.19+0x9ec/0x12c0 [btrfs]
[3184240.253139] Caller[00000000100ba788]: __btrfs_run_delayed_refs+0x548/0x1980 [btrfs]
[3184240.253158] Caller[00000000100bfec8]: btrfs_run_delayed_refs+0x88/0x2c0 [btrfs]
[3184240.253178] Caller[00000000100dcc20]: commit_cowonly_roots+0xe0/0x5c0 [btrfs]
[3184240.253198] Caller[00000000100df9b4]: btrfs_commit_transaction+0x534/0xd60 [btrfs]
[3184240.253218] Caller[00000000100d54c4]: btrfs_commit_super+0x64/0x80 [btrfs]
[3184240.253236] Caller[00000000100d8a60]: close_ctree+0x260/0x320 [btrfs]
[3184240.253251] Caller[0000000010096cd0]: btrfs_put_super+0x10/0x20 [btrfs]
[3184240.253258] Caller[00000000005e7d74]: generic_shutdown_super+0x74/0x100
[3184240.253265] Caller[00000000005e8050]: kill_anon_super+0x10/0x40
[3184240.253279] Caller[00000000100969b4]: btrfs_kill_super+0x14/0x100 [btrfs]
[3184240.253287] Caller[00000000005e8244]: deactivate_locked_super+0x44/0x80
[3184240.253293] Caller[00000000005e8f44]: deactivate_super+0x64/0x80
[3184240.253304] Caller[000000000060707c]: cleanup_mnt+0x3c/0xa0
[3184240.253310] Caller[000000000060714c]: __cleanup_mnt+0xc/0x20
[3184240.253321] Caller[00000000004859e0]: task_work_run+0xa0/0xe0
[3184240.253330] Caller[000000000042e144]: do_notify_resume+0x64/0x80
[3184240.253338] Caller[0000000000404b44]: __handle_signal+0xc/0x2c
[3184240.253344] Caller[fff8000100150db4]: 0xfff8000100150db4
[3184240.253349] Instruction DUMP: 92102d29  7c0d43df  901222c8 <91d02005> d25e2008  d4422b40  7c19cac2  90022af0  106fffe1
[3184240.312832] run fstests btrfs/001 at 2016-09-25 10:25:48


Do I need to redo test with btrfs-progs.git version instead of debian one?

Thanks.

PS: boot log of btrfs module:

[3102837.407118] xor: automatically using best checksumming function:
[3102837.446156]    Niagara   :  4898.000 MB/sec
[3102837.870398] Btrfs loaded, crc32c=crc32c-sparc64, debug=on, assert=on, integrity-checker=on
[3102837.871047] BTRFS: selftest: sectorsize: 8192  nodesize: 8192
[3102837.871055] BTRFS: selftest: Running btrfs free space cache tests
[3102837.871242] BTRFS: selftest: Running extent only tests
[3102837.871261] BTRFS: selftest: Running bitmap only tests
[3102837.871338] BTRFS: selftest: Running bitmap and extent tests
[3102837.871443] BTRFS: selftest: Running space stealing from bitmap to extent
[3102837.872016] BTRFS: selftest: Free space cache tests finished
[3102837.872023] BTRFS: selftest: Running extent buffer operation tests
[3102837.872028] BTRFS: selftest: Running btrfs_split_item tests
[3102837.872394] BTRFS: selftest: Running extent I/O tests
[3102837.872401] BTRFS: selftest: Running find delalloc tests
[3102837.872953] BTRFS: selftest: Running extent buffer bitmap tests
[3102837.921398] BTRFS: selftest: Extent I/O tests finished
[3102837.921409] BTRFS: selftest: Running btrfs_get_extent tests
[3102837.921901] BTRFS: selftest: Running hole first btrfs_get_extent test
[3102837.922261] BTRFS: selftest: Running outstanding_extents tests
[3102837.922831] BTRFS: selftest: Running qgroup tests
[3102837.922838] BTRFS: selftest: Qgroup basic add
[3102837.922901] BTRFS: selftest: Qgroup multiple refs test
[3102837.924768] BTRFS: selftest: Running free space tree tests
[3102837.962292] BTRFS: selftest: sectorsize: 8192  nodesize: 16384
[3102837.962303] BTRFS: selftest: Running btrfs free space cache tests
[3102837.962486] BTRFS: selftest: Running extent only tests
[3102837.962502] BTRFS: selftest: Running bitmap only tests
[3102837.962576] BTRFS: selftest: Running bitmap and extent tests
[3102837.962681] BTRFS: selftest: Running space stealing from bitmap to extent
[3102837.963250] BTRFS: selftest: Free space cache tests finished
[3102837.963257] BTRFS: selftest: Running extent buffer operation tests
[3102837.963262] BTRFS: selftest: Running btrfs_split_item tests
[3102837.963601] BTRFS: selftest: Running extent I/O tests
[3102837.963607] BTRFS: selftest: Running find delalloc tests
[3102837.964145] BTRFS: selftest: Running extent buffer bitmap tests
[3102838.012532] BTRFS: selftest: Extent I/O tests finished
[3102838.012539] BTRFS: selftest: Running btrfs_get_extent tests
[3102838.012951] BTRFS: selftest: Running hole first btrfs_get_extent test
[3102838.013292] BTRFS: selftest: Running outstanding_extents tests
[3102838.013830] BTRFS: selftest: Running qgroup tests
[3102838.013836] BTRFS: selftest: Qgroup basic add
[3102838.013879] BTRFS: selftest: Qgroup multiple refs test
[3102838.015744] BTRFS: selftest: Running free space tree tests
[3102838.053230] BTRFS: selftest: sectorsize: 8192  nodesize: 32768
[3102838.053237] BTRFS: selftest: Running btrfs free space cache tests
[3102838.053419] BTRFS: selftest: Running extent only tests
[3102838.053430] BTRFS: selftest: Running bitmap only tests
[3102838.053493] BTRFS: selftest: Running bitmap and extent tests
[3102838.053597] BTRFS: selftest: Running space stealing from bitmap to extent
[3102838.054167] BTRFS: selftest: Free space cache tests finished
[3102838.054173] BTRFS: selftest: Running extent buffer operation tests
[3102838.054179] BTRFS: selftest: Running btrfs_split_item tests
[3102838.054512] BTRFS: selftest: Running extent I/O tests
[3102838.054518] BTRFS: selftest: Running find delalloc tests
[3102838.054956] BTRFS: selftest: Running extent buffer bitmap tests
[3102838.103317] BTRFS: selftest: Extent I/O tests finished
[3102838.103324] BTRFS: selftest: Running btrfs_get_extent tests
[3102838.103728] BTRFS: selftest: Running hole first btrfs_get_extent test
[3102838.104068] BTRFS: selftest: Running outstanding_extents tests
[3102838.104603] BTRFS: selftest: Running qgroup tests
[3102838.104609] BTRFS: selftest: Qgroup basic add
[3102838.104644] BTRFS: selftest: Qgroup multiple refs test
[3102838.106503] BTRFS: selftest: Running free space tree tests
[3102838.143985] BTRFS: selftest: sectorsize: 8192  nodesize: 65536
[3102838.143992] BTRFS: selftest: Running btrfs free space cache tests
[3102838.144173] BTRFS: selftest: Running extent only tests
[3102838.144184] BTRFS: selftest: Running bitmap only tests
[3102838.144247] BTRFS: selftest: Running bitmap and extent tests
[3102838.144350] BTRFS: selftest: Running space stealing from bitmap to extent
[3102838.144915] BTRFS: selftest: Free space cache tests finished
[3102838.144921] BTRFS: selftest: Running extent buffer operation tests
[3102838.144926] BTRFS: selftest: Running btrfs_split_item tests
[3102838.145260] BTRFS: selftest: Running extent I/O tests
[3102838.145266] BTRFS: selftest: Running find delalloc tests
[3102838.145700] BTRFS: selftest: Running extent buffer bitmap tests
[3102838.194039] BTRFS: selftest: Extent I/O tests finished
[3102838.194045] BTRFS: selftest: Running btrfs_get_extent tests
[3102838.194462] BTRFS: selftest: Running hole first btrfs_get_extent test
[3102838.194804] BTRFS: selftest: Running outstanding_extents tests
[3102838.195339] BTRFS: selftest: Running qgroup tests
[3102838.195345] BTRFS: selftest: Qgroup basic add
[3102838.195380] BTRFS: selftest: Qgroup multiple refs test
[3102838.197235] BTRFS: selftest: Running free space tree tests
[3102838.288732] random: fast init done

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[David Sterba]

On Sun, Sep 25, 2016 at 10:55:24AM +0300, Anatoly Pugachev wrote:
> applied patch to git kernel (v4.8-rc7-172-gbd5dbcb) cleanly. Did not used
> btrfs-progs.git, but debian shipped 4.7.3-1 .
> 
> (re)booted to a newly patched kernel and used xfstests.git (v1.1.0-1328-g06d4001):
> 
> # mount tmpfs -t tmpfs -o size=26g /ramdisk
> # cd /ramdisk
> # fallocate -l 10g testvol1
> # for i in 1 2 3 4; do fallocate -l 4g scratch${i}; done
> # for i in *; do losetup -f $i; done
> # losetup
> NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE         DIO
> /dev/loop0         0      0         0  0 /ramdisk/scratch1   0
> /dev/loop1         0      0         0  0 /ramdisk/scratch2   0
> /dev/loop2         0      0         0  0 /ramdisk/scratch3   0
> /dev/loop3         0      0         0  0 /ramdisk/scratch4   0
> /dev/loop4         0      0         0  0 /ramdisk/testvol1   0
> 
> mator@ttip:~/xfstests-dev$ cat local.config
> export TEST_DEV=/dev/loop4
> export TEST_DIR=/testvol
> export SCRATCH_DEV_POOL="/dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3"
> export SCRATCH_MNT=/mnt/scratch
> 
> root@ttip:/home/mator/xfstests-dev# mkfs.btrfs /dev/loop4
> btrfs-progs v4.7.3
> See http://btrfs.wiki.kernel.org for more information.
> 
> Performing full device TRIM (10.00GiB) ...
> Label:              (null)
> UUID:
> Node size:          16384
> Sector size:        8192
> Filesystem size:    10.00GiB
> Block group profiles:
>   Data:             single            8.00MiB
>   Metadata:         DUP               1.00GiB
>   System:           DUP               8.00MiB
> SSD detected:       no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>    ID        SIZE  PATH
>     1    10.00GiB  /dev/loop4
> 
> root@ttip:/home/mator/xfstests-dev# ./check 'btrfs/*'
> FSTYP         -- btrfs
> PLATFORM      -- Linux/sparc64 ttip 4.8.0-rc7+
> MKFS_OPTIONS  -- /dev/loop0
> MOUNT_OPTIONS -- /dev/loop0 /mnt/scratch
> 
> _check_btrfs_filesystem: filesystem on /dev/loop4 is inconsistent (see /home/mator/xfstests-dev/results//check.full)
> btrfs/001
> 
> 
> (on another screen/tmux window shell)
> 
> $ pstree -A | grep check 
>         |-sshd-+-sshd---sshd---bash---sudo---bash---check---001---mount
> 
> $ ps ax | grep mount
>  76344 pts/0    D+     0:00 /bin/mount -t btrfs /dev/loop4 /testvol
> 
> $ cat /home/mator/xfstests-dev/results//check.full
> _check_btrfs_filesystem: filesystem on /dev/loop4 is inconsistent
> *** fsck.btrfs output ***
> ERROR: cannot open device '/dev/loop4': Device or resource busy
> Couldn't open file system
> *** end fsck.btrfs output
> *** mount output ***
> sysfs on /sys type sysfs (rw,relatime)
> proc on /proc type proc (rw,relatime)
> udev on /dev type devtmpfs (rw,nosuid,relatime,size=16479064k,nr_inodes=2059883,mode=755)
> devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
> tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3314128k,mode=755)
> /dev/vdiska2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
> tmpfs on /dev/shm type tmpfs (rw)
> tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
> tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
> cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
> cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
> cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
> cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
> cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
> cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
> cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
> cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
> cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
> cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
> systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=23073)
> mqueue on /dev/mqueue type mqueue (rw,relatime)
> debugfs on /sys/kernel/debug type debugfs (rw,relatime)
> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
> /dev/vdiska1 on /boot type ext3 (rw,relatime,data=ordered)
> /dev/vdiskb1 on /home type xfs (rw,relatime,attr2,inode64,noquota)
> binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
> tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3314120k,mode=700,uid=1000,gid=1000)
> tmpfs on /ramdisk type tmpfs (rw,relatime,size=27262976k)
> /dev/loop0 on /mnt/scratch type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
> *** end mount output
> 
> in kernel logs and on console:
> 
> [3184224.438566] BTRFS: device fsid b54ec0aa-4187-419d-9de1-5ef284cd3b32 devid 1 transid 5 /dev/loop4
> [3184239.417845] BTRFS info (device loop4): disk space caching is enabled
> [3184239.417865] BTRFS info (device loop4): has skinny extents
> [3184239.417872] BTRFS info (device loop4): flagging fs with big metadata feature
> [3184239.421147] BTRFS info (device loop4): creating UUID tree
> [3184240.026601] BTRFS: device fsid 5383d227-9ea2-4514-8857-641c6e2e2063 devid 1 transid 5 /dev/loop0
> [3184240.131927] BTRFS info (device loop0): disk space caching is enabled
> [3184240.131956] BTRFS info (device loop0): has skinny extents
> [3184240.131971] BTRFS info (device loop0): flagging fs with big metadata feature
> [3184240.135182] BTRFS info (device loop0): creating UUID tree
> [3184240.252534] BTRFS critical (device loop4): corrupt leaf, non-root leaf's nritems is 0: block=29556736,root=1, slot=0

The error does not seem to be related to the free space bitmap issues
(at least I don't see a connection). The message is from

  1ba98d086fe3a14d6a31f2f66dbab70c45d00f63
  "Btrfs: detect corruption when non-root leaf has zero item"

called from btrfs_mark_buffer_dirty with integrity checker on.
Confirmed from the log:

> [3102837.870398] Btrfs loaded, crc32c=crc32c-sparc64, debug=on, assert=on, integrity-checker=on
...

This is fixed by patch

  "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in split_leaf"

that's in the 4.9 queue. Other than that, the self-tests seem to pass,
thanks for the test. Would be good if you can test with the mentioned
patch included or without integrity checker. Thanks for testing.

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[Omar Sandoval]

On Mon, Sep 26, 2016 at 07:50:00PM +0200, David Sterba wrote:
> On Sun, Sep 25, 2016 at 10:55:24AM +0300, Anatoly Pugachev wrote:
> > applied patch to git kernel (v4.8-rc7-172-gbd5dbcb) cleanly. Did not used
> > btrfs-progs.git, but debian shipped 4.7.3-1 .
> > 
> > (re)booted to a newly patched kernel and used xfstests.git (v1.1.0-1328-g06d4001):
> > 
> > # mount tmpfs -t tmpfs -o size=26g /ramdisk
> > # cd /ramdisk
> > # fallocate -l 10g testvol1
> > # for i in 1 2 3 4; do fallocate -l 4g scratch${i}; done
> > # for i in *; do losetup -f $i; done
> > # losetup
> > NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE         DIO
> > /dev/loop0         0      0         0  0 /ramdisk/scratch1   0
> > /dev/loop1         0      0         0  0 /ramdisk/scratch2   0
> > /dev/loop2         0      0         0  0 /ramdisk/scratch3   0
> > /dev/loop3         0      0         0  0 /ramdisk/scratch4   0
> > /dev/loop4         0      0         0  0 /ramdisk/testvol1   0
> > 
> > mator@ttip:~/xfstests-dev$ cat local.config
> > export TEST_DEV=/dev/loop4
> > export TEST_DIR=/testvol
> > export SCRATCH_DEV_POOL="/dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3"
> > export SCRATCH_MNT=/mnt/scratch
> > 
> > root@ttip:/home/mator/xfstests-dev# mkfs.btrfs /dev/loop4
> > btrfs-progs v4.7.3
> > See http://btrfs.wiki.kernel.org for more information.
> > 
> > Performing full device TRIM (10.00GiB) ...
> > Label:              (null)
> > UUID:
> > Node size:          16384
> > Sector size:        8192
> > Filesystem size:    10.00GiB
> > Block group profiles:
> >   Data:             single            8.00MiB
> >   Metadata:         DUP               1.00GiB
> >   System:           DUP               8.00MiB
> > SSD detected:       no
> > Incompat features:  extref, skinny-metadata
> > Number of devices:  1
> > Devices:
> >    ID        SIZE  PATH
> >     1    10.00GiB  /dev/loop4
> > 
> > root@ttip:/home/mator/xfstests-dev# ./check 'btrfs/*'
> > FSTYP         -- btrfs
> > PLATFORM      -- Linux/sparc64 ttip 4.8.0-rc7+
> > MKFS_OPTIONS  -- /dev/loop0
> > MOUNT_OPTIONS -- /dev/loop0 /mnt/scratch
> > 
> > _check_btrfs_filesystem: filesystem on /dev/loop4 is inconsistent (see /home/mator/xfstests-dev/results//check.full)
> > btrfs/001
> > 
> > 
> > (on another screen/tmux window shell)
> > 
> > $ pstree -A | grep check 
> >         |-sshd-+-sshd---sshd---bash---sudo---bash---check---001---mount
> > 
> > $ ps ax | grep mount
> >  76344 pts/0    D+     0:00 /bin/mount -t btrfs /dev/loop4 /testvol
> > 
> > $ cat /home/mator/xfstests-dev/results//check.full
> > _check_btrfs_filesystem: filesystem on /dev/loop4 is inconsistent
> > *** fsck.btrfs output ***
> > ERROR: cannot open device '/dev/loop4': Device or resource busy
> > Couldn't open file system
> > *** end fsck.btrfs output
> > *** mount output ***
> > sysfs on /sys type sysfs (rw,relatime)
> > proc on /proc type proc (rw,relatime)
> > udev on /dev type devtmpfs (rw,nosuid,relatime,size=16479064k,nr_inodes=2059883,mode=755)
> > devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
> > tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3314128k,mode=755)
> > /dev/vdiska2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> > securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
> > tmpfs on /dev/shm type tmpfs (rw)
> > tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
> > tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
> > cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
> > cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
> > cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
> > cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
> > cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
> > cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
> > cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
> > cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
> > cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
> > cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
> > systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=23073)
> > mqueue on /dev/mqueue type mqueue (rw,relatime)
> > debugfs on /sys/kernel/debug type debugfs (rw,relatime)
> > hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
> > /dev/vdiska1 on /boot type ext3 (rw,relatime,data=ordered)
> > /dev/vdiskb1 on /home type xfs (rw,relatime,attr2,inode64,noquota)
> > binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
> > tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3314120k,mode=700,uid=1000,gid=1000)
> > tmpfs on /ramdisk type tmpfs (rw,relatime,size=27262976k)
> > /dev/loop0 on /mnt/scratch type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
> > *** end mount output
> > 
> > in kernel logs and on console:
> > 
> > [3184224.438566] BTRFS: device fsid b54ec0aa-4187-419d-9de1-5ef284cd3b32 devid 1 transid 5 /dev/loop4
> > [3184239.417845] BTRFS info (device loop4): disk space caching is enabled
> > [3184239.417865] BTRFS info (device loop4): has skinny extents
> > [3184239.417872] BTRFS info (device loop4): flagging fs with big metadata feature
> > [3184239.421147] BTRFS info (device loop4): creating UUID tree
> > [3184240.026601] BTRFS: device fsid 5383d227-9ea2-4514-8857-641c6e2e2063 devid 1 transid 5 /dev/loop0
> > [3184240.131927] BTRFS info (device loop0): disk space caching is enabled
> > [3184240.131956] BTRFS info (device loop0): has skinny extents
> > [3184240.131971] BTRFS info (device loop0): flagging fs with big metadata feature
> > [3184240.135182] BTRFS info (device loop0): creating UUID tree
> > [3184240.252534] BTRFS critical (device loop4): corrupt leaf, non-root leaf's nritems is 0: block=29556736,root=1, slot=0
> 
> The error does not seem to be related to the free space bitmap issues
> (at least I don't see a connection). The message is from
> 
>   1ba98d086fe3a14d6a31f2f66dbab70c45d00f63
>   "Btrfs: detect corruption when non-root leaf has zero item"
> 
> called from btrfs_mark_buffer_dirty with integrity checker on.
> Confirmed from the log:
> 
> > [3102837.870398] Btrfs loaded, crc32c=crc32c-sparc64, debug=on, assert=on, integrity-checker=on
> ...
> 
> This is fixed by patch
> 
>   "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in split_leaf"
> 
> that's in the 4.9 queue.

Ah, awesome, good to know.

> Other than that, the self-tests seem to pass,
> thanks for the test. Would be good if you can test with the mentioned
> patch included or without integrity checker. Thanks for testing.

-- 
Omar

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[Anatoly Pugachev]

On Mon, Sep 26, 2016 at 07:50:00PM +0200, David Sterba wrote:
> On Sun, Sep 25, 2016 at 10:55:24AM +0300, Anatoly Pugachev wrote:
> > applied patch to git kernel (v4.8-rc7-172-gbd5dbcb) cleanly. Did not used
> > btrfs-progs.git, but debian shipped 4.7.3-1 .
> > 
> > [3184240.135182] BTRFS info (device loop0): creating UUID tree
> > [3184240.252534] BTRFS critical (device loop4): corrupt leaf, non-root leaf's nritems is 0: block=29556736,root=1, slot=0
> 
> The error does not seem to be related to the free space bitmap issues
> (at least I don't see a connection). The message is from
> 
>   1ba98d086fe3a14d6a31f2f66dbab70c45d00f63
>   "Btrfs: detect corruption when non-root leaf has zero item"
> 
> called from btrfs_mark_buffer_dirty with integrity checker on.
> Confirmed from the log:
> 
> > [3102837.870398] Btrfs loaded, crc32c=crc32c-sparc64, debug=on, assert=on, integrity-checker=on
> ...
> 
> This is fixed by patch
> 
>   "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in split_leaf"
> 
> that's in the 4.9 queue. Other than that, the self-tests seem to pass,
> thanks for the test. Would be good if you can test with the mentioned
> patch included or without integrity checker. Thanks for testing.


updated git kernel to v4.8-rc8-8-gae6dd8d , applied this 
"Btrfs: free space tree and sanity test fixes" patchset and added/applied 
"Btrfs: remove unnecessary btrfs_mark_buffer_dirty in split_leaf" :


1) kernel config BTRFS integrity-checker=on :

Sep 29 00:32:15 ttip kernel: raid6: using intx1 recovery algorithm
Sep 29 00:32:15 ttip kernel: xor: automatically using best checksumming function:
Sep 29 00:32:15 ttip kernel:    Niagara   :  4746.000 MB/sec
Sep 29 00:32:15 ttip kernel: Btrfs loaded, crc32c=crc32c-sparc64, debug=on, assert=on, integrity-checker=on
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: sectorsize: 8192  nodesize: 8192
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs free space cache tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent only tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running bitmap only tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running bitmap and extent tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running space stealing from bitmap to extent
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Free space cache tests finished
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent buffer operation tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs_split_item tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent I/O tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running find delalloc tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent buffer bitmap tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Extent I/O tests finished
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs_get_extent tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running hole first btrfs_get_extent test
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running outstanding_extents tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running qgroup tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Qgroup basic add
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Qgroup multiple refs test
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running free space tree tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: sectorsize: 8192  nodesize: 16384
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs free space cache tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent only tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running bitmap only tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running bitmap and extent tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running space stealing from bitmap to extent
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Free space cache tests finished
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent buffer operation tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs_split_item tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent I/O tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running find delalloc tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent buffer bitmap tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Extent I/O tests finished
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs_get_extent tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running hole first btrfs_get_extent test
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running outstanding_extents tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running qgroup tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Qgroup basic add
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Qgroup multiple refs test
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running free space tree tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: sectorsize: 8192  nodesize: 32768
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs free space cache tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent only tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running bitmap only tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running bitmap and extent tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running space stealing from bitmap to extent
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Free space cache tests finished
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent buffer operation tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs_split_item tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent I/O tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running find delalloc tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent buffer bitmap tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Extent I/O tests finished
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs_get_extent tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running hole first btrfs_get_extent test
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running outstanding_extents tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running qgroup tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Qgroup basic add
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Qgroup multiple refs test
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running free space tree tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: sectorsize: 8192  nodesize: 65536
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs free space cache tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent only tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running bitmap only tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running bitmap and extent tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running space stealing from bitmap to extent
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Free space cache tests finished
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent buffer operation tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs_split_item tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent I/O tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running find delalloc tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running extent buffer bitmap tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Extent I/O tests finished
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running btrfs_get_extent tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running hole first btrfs_get_extent test
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running outstanding_extents tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running qgroup tests
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Qgroup basic add
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Qgroup multiple refs test
Sep 29 00:32:15 ttip kernel: BTRFS: selftest: Running free space tree tests
Sep 29 00:32:15 ttip kernel: random: fast init done
Sep 29 00:32:15 ttip kernel: EXT4-fs (vdiska2): mounted filesystem with ordered data mode. Opts: (null)
Sep 29 00:32:15 ttip kernel: random: crng init done
Sep 29 00:32:15 ttip kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 29 00:32:15 ttip systemd[1]: systemd 231 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
Sep 29 00:32:15 ttip systemd[1]: Detected architecture sparc64.
Sep 29 00:32:15 ttip systemd[1]: Set hostname to <ttip>.
Sep 29 00:32:15 ttip systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
Sep 29 00:32:15 ttip systemd[1]: Reached target Swap.
Sep 29 00:32:15 ttip systemd[1]: Listening on fsck to fsckd communication Socket.
Sep 29 00:32:15 ttip kernel: nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
Sep 29 00:32:16 ttip kernel: EXT4-fs (vdiska2): re-mounted. Opts: errors=remount-ro
Sep 29 00:32:22 ttip kernel: n2rng.c:v0.2 (July 27, 2011)
Sep 29 00:32:27 ttip kernel: n2rng f029b124: Registered RNG HVAPI major 2 minor 0
Sep 29 00:32:27 ttip kernel: n2rng f029b124: Found multi-unit-capable RNG, units: 2
Sep 29 00:32:27 ttip kernel: n2rng f029b124: RNG ready
Sep 29 00:32:27 ttip kernel: sha1_sparc64: Using sparc64 sha1 opcode optimized SHA-1 implementation
Sep 29 00:32:27 ttip kernel: sha256_sparc64: Using sparc64 sha256 opcode optimized SHA-256/SHA-224 implementation
Sep 29 00:32:27 ttip kernel: sha512_sparc64: Using sparc64 sha512 opcode optimized SHA-512/SHA-384 implementation
Sep 29 00:32:27 ttip kernel: md5_sparc64: Using sparc64 md5 opcode optimized MD5 implementation
Sep 29 00:32:27 ttip kernel: aes_sparc64: Using sparc64 aes opcodes optimized AES implementation
Sep 29 00:32:27 ttip kernel: des_sparc64: Using sparc64 des opcodes optimized DES implementation
Sep 29 00:32:27 ttip kernel: camellia_sparc64: Using sparc64 camellia opcodes optimized CAMELLIA implementation
Sep 29 00:32:27 ttip kernel: SGI XFS with ACLs, security attributes, realtime, no debug enabled
Sep 29 00:32:27 ttip kernel: XFS (vdiskb1): Mounting V5 Filesystem
Sep 29 00:32:27 ttip kernel: XFS (vdiskb1): Ending clean mount
Sep 29 00:32:27 ttip kernel: EXT4-fs (vdiska1): mounting ext3 file system using the ext4 subsystem
Sep 29 00:32:27 ttip kernel: EXT4-fs (vdiska1): mounted filesystem with ordered data mode. Opts: (null)
Sep 29 00:32:27 ttip systemd[1]: apt-daily.timer: Adding 10h 46min 38.851540s random time.
Sep 29 00:32:27 ttip kernel: tun: Universal TUN/TAP device driver, 1.6
Sep 29 00:32:27 ttip kernel: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Sep 29 00:38:32 ttip kernel: loop: module loaded
Sep 29 00:40:55 ttip kernel: BTRFS: device fsid 7bb81df9-0e2b-47f2-81ff-c08502d38da6 devid 1 transid 5 /dev/loop4
Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): disk space caching is enabled
Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): has skinny extents
Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): flagging fs with big metadata feature
Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): creating UUID tree
Sep 29 00:41:31 ttip kernel: BTRFS: device fsid d0ee7ca3-3be0-465f-857b-19e681181923 devid 1 transid 5 /dev/loop0
Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): enabling free space tree
Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): using free space tree
Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): has skinny extents
Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): flagging fs with big metadata feature
Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): creating free space tree
Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): setting 1 ro feature flag
Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): setting 2 ro feature flag
Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): creating UUID tree
Sep 29 00:41:32 ttip kernel: BTRFS critical (device loop4): corrupt leaf, non-root leaf's nritems is 0: block=29556736,root=1, slot=0
Sep 29 00:41:32 ttip kernel: BTRFS info (device loop4): leaf 29556736 total ptrs 0 free space 16283
Sep 29 00:41:32 ttip kernel: BTRFS: assertion failed: 0, file: fs/btrfs/disk-io.c, line: 4059kernel BUG at fs/btrfs/ctree.h:3369!
Sep 29 00:41:32 ttip kernel:               \|/ ____ \|/
                                           "@'/ .. \`@"
                                           /_| \__/ |_\
                                              \__U_/
Sep 29 00:41:32 ttip kernel: umount(9847): Kernel bad sw trap 5 [#1]
Sep 29 00:41:32 ttip kernel: CPU: 22 PID: 9847 Comm: umount Not tainted 4.8.0-rc8+ #10
Sep 29 00:41:32 ttip kernel: task: fff800007db1a480 task.stack: fff80007f2dc8000
Sep 29 00:41:32 ttip kernel: TSTATE: 0000004411001606 TPC: 00000000100d6ccc TNPC: 00000000100d6cd0 Y: 00000000    Not tainted
Sep 29 00:41:32 ttip kernel: TPC: <btrfs_mark_buffer_dirty+0x1ac/0x1e0 [btrfs]>
Sep 29 00:41:32 ttip kernel: g0: 0000000000000000 g1: 0000000000000000 g2: 0000000000000007 g3: 0000000000000000
Sep 29 00:41:32 ttip kernel: g4: fff800007db1a480 g5: fff800082c692000 g6: fff80007f2dc8000 g7: fff800082ce00000
Sep 29 00:41:32 ttip kernel: o0: 000000001019cac8 o1: 0000000000000d29 o2: 000000001019c930 o3: 0000000000000fdb
Sep 29 00:41:32 ttip kernel: o4: 0000000001c30000 o5: 0000000000000000 sp: fff80007f2dca3c1 ret_pc: 00000000100d6cc4
Sep 29 00:41:32 ttip kernel: RPC: <btrfs_mark_buffer_dirty+0x1a4/0x1e0 [btrfs]>
Sep 29 00:41:32 ttip kernel: l0: 000000000000154c l1: 0000000000002000 l2: 0000000000002000 l3: fff8000812764b10
Sep 29 00:41:32 ttip kernel: l4: fff800080f33aa18 l5: 00000000101b8400 l6: 0000000001000000 l7: 0000000000000000
Sep 29 00:41:32 ttip kernel: i0: fff800081372bbe8 i1: 0000000000000000 i2: 0700000000000000 i3: 00060000122b38f0
Sep 29 00:41:32 ttip kernel: i4: fff8000813156800 i5: 0000000000000007 i6: fff80007f2dca471 i7: 00000000100a0388
Sep 29 00:41:32 ttip kernel: I7: <__btrfs_cow_block+0x7e8/0xb80 [btrfs]>
Sep 29 00:41:32 ttip kernel: Call Trace:
Sep 29 00:41:32 ttip kernel:  [00000000100a0388] __btrfs_cow_block+0x7e8/0xb80 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000100a0948] btrfs_cow_block+0x1a8/0x340 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000100a6000] btrfs_search_slot+0x2a0/0xf60 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000100ccff4] btrfs_del_csums+0x1b4/0x6a0 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000100b4ecc] __btrfs_free_extent.isra.19+0x9ec/0x12c0 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000100ba788] __btrfs_run_delayed_refs+0x548/0x1980 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000100bfec8] btrfs_run_delayed_refs+0x88/0x2c0 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000100dcc20] commit_cowonly_roots+0xe0/0x5c0 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000100df9b4] btrfs_commit_transaction+0x534/0xd60 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000100d54c4] btrfs_commit_super+0x64/0x80 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000100d8a60] close_ctree+0x260/0x320 [btrfs]
Sep 29 00:41:32 ttip kernel:  [0000000010096cd0] btrfs_put_super+0x10/0x20 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000005e62d4] generic_shutdown_super+0x74/0x100
Sep 29 00:41:32 ttip kernel:  [00000000005e65b0] kill_anon_super+0x10/0x40
Sep 29 00:41:32 ttip kernel:  [00000000100969b4] btrfs_kill_super+0x14/0x100 [btrfs]
Sep 29 00:41:32 ttip kernel:  [00000000005e67a4] deactivate_locked_super+0x44/0x80
Sep 29 00:41:32 ttip kernel: Disabling lock debugging due to kernel taint
Sep 29 00:41:32 ttip kernel: Caller[00000000100a0388]: __btrfs_cow_block+0x7e8/0xb80 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000100a0948]: btrfs_cow_block+0x1a8/0x340 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000100a6000]: btrfs_search_slot+0x2a0/0xf60 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000100ccff4]: btrfs_del_csums+0x1b4/0x6a0 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000100b4ecc]: __btrfs_free_extent.isra.19+0x9ec/0x12c0 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000100ba788]: __btrfs_run_delayed_refs+0x548/0x1980 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000100bfec8]: btrfs_run_delayed_refs+0x88/0x2c0 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000100dcc20]: commit_cowonly_roots+0xe0/0x5c0 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000100df9b4]: btrfs_commit_transaction+0x534/0xd60 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000100d54c4]: btrfs_commit_super+0x64/0x80 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000100d8a60]: close_ctree+0x260/0x320 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[0000000010096cd0]: btrfs_put_super+0x10/0x20 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000005e62d4]: generic_shutdown_super+0x74/0x100
Sep 29 00:41:32 ttip kernel: Caller[00000000005e65b0]: kill_anon_super+0x10/0x40
Sep 29 00:41:32 ttip kernel: Caller[00000000100969b4]: btrfs_kill_super+0x14/0x100 [btrfs]
Sep 29 00:41:32 ttip kernel: Caller[00000000005e67a4]: deactivate_locked_super+0x44/0x80
Sep 29 00:41:32 ttip kernel: Caller[00000000005e74a4]: deactivate_super+0x64/0x80
Sep 29 00:41:32 ttip kernel: Caller[00000000006055dc]: cleanup_mnt+0x3c/0xa0
Sep 29 00:41:32 ttip kernel: Caller[00000000006056ac]: __cleanup_mnt+0xc/0x20
Sep 29 00:41:32 ttip kernel: Caller[00000000004859c0]: task_work_run+0xa0/0xe0
Sep 29 00:41:32 ttip kernel: Caller[000000000042e144]: do_notify_resume+0x64/0x80
Sep 29 00:41:32 ttip kernel: Caller[0000000000404b44]: __handle_signal+0xc/0x2c
Sep 29 00:41:32 ttip kernel: Caller[fff8000100150db4]: 0xfff8000100150db4
Sep 29 00:41:32 ttip kernel: Instruction DUMP: 92102d29  7c0d43df  901222c8 <91d02005> d25e2008  d4422b40  7c19c0d2  90022af0  106fffe1 
Sep 29 00:41:32 ttip unknown: run fstests btrfs/001 at 2016-09-29 00:41:32


kernel oops and hangs (as like in my previous email)





2) kernel config BTRFS integrity-checker=off

[3535986.809765] raid6: using intx1 recovery algorithm
[3535986.810822] xor: automatically using best checksumming function:
[3535986.849741]    Niagara   :  4786.000 MB/sec
[3535987.273958] Btrfs loaded, crc32c=crc32c-sparc64, debug=on, assert=on
[3535987.274599] BTRFS: selftest: sectorsize: 8192  nodesize: 8192
[3535987.274607] BTRFS: selftest: Running btrfs free space cache tests
[3535987.274798] BTRFS: selftest: Running extent only tests
[3535987.274817] BTRFS: selftest: Running bitmap only tests
[3535987.274891] BTRFS: selftest: Running bitmap and extent tests
[3535987.274997] BTRFS: selftest: Running space stealing from bitmap to extent
[3535987.275569] BTRFS: selftest: Free space cache tests finished
[3535987.275575] BTRFS: selftest: Running extent buffer operation tests
[3535987.275580] BTRFS: selftest: Running btrfs_split_item tests
[3535987.275950] BTRFS: selftest: Running extent I/O tests
[3535987.275956] BTRFS: selftest: Running find delalloc tests
[3535987.276520] BTRFS: selftest: Running extent buffer bitmap tests
[3535987.324943] BTRFS: selftest: Extent I/O tests finished
[3535987.324950] BTRFS: selftest: Running btrfs_get_extent tests
[3535987.325385] BTRFS: selftest: Running hole first btrfs_get_extent test
[3535987.325729] BTRFS: selftest: Running outstanding_extents tests
[3535987.326282] BTRFS: selftest: Running qgroup tests
[3535987.326288] BTRFS: selftest: Qgroup basic add
[3535987.326346] BTRFS: selftest: Qgroup multiple refs test
[3535987.328207] BTRFS: selftest: Running free space tree tests
[3535987.365715] BTRFS: selftest: sectorsize: 8192  nodesize: 16384
[3535987.365722] BTRFS: selftest: Running btrfs free space cache tests
[3535987.365911] BTRFS: selftest: Running extent only tests
[3535987.365922] BTRFS: selftest: Running bitmap only tests
[3535987.365986] BTRFS: selftest: Running bitmap and extent tests
[3535987.366089] BTRFS: selftest: Running space stealing from bitmap to extent
[3535987.366654] BTRFS: selftest: Free space cache tests finished
[3535987.366660] BTRFS: selftest: Running extent buffer operation tests
[3535987.366665] BTRFS: selftest: Running btrfs_split_item tests
[3535987.366997] BTRFS: selftest: Running extent I/O tests
[3535987.367003] BTRFS: selftest: Running find delalloc tests
[3535987.367450] BTRFS: selftest: Running extent buffer bitmap tests
[3535987.415830] BTRFS: selftest: Extent I/O tests finished
[3535987.415846] BTRFS: selftest: Running btrfs_get_extent tests
[3535987.416310] BTRFS: selftest: Running hole first btrfs_get_extent test
[3535987.416653] BTRFS: selftest: Running outstanding_extents tests
[3535987.417198] BTRFS: selftest: Running qgroup tests
[3535987.417204] BTRFS: selftest: Qgroup basic add
[3535987.417253] BTRFS: selftest: Qgroup multiple refs test
[3535987.419119] BTRFS: selftest: Running free space tree tests
[3535987.456670] BTRFS: selftest: sectorsize: 8192  nodesize: 32768
[3535987.456677] BTRFS: selftest: Running btrfs free space cache tests
[3535987.456860] BTRFS: selftest: Running extent only tests
[3535987.456873] BTRFS: selftest: Running bitmap only tests
[3535987.456938] BTRFS: selftest: Running bitmap and extent tests
[3535987.457041] BTRFS: selftest: Running space stealing from bitmap to extent
[3535987.457607] BTRFS: selftest: Free space cache tests finished
[3535987.457613] BTRFS: selftest: Running extent buffer operation tests
[3535987.457618] BTRFS: selftest: Running btrfs_split_item tests
[3535987.457962] BTRFS: selftest: Running extent I/O tests
[3535987.457968] BTRFS: selftest: Running find delalloc tests
[3535987.458465] BTRFS: selftest: Running extent buffer bitmap tests
[3535987.506819] BTRFS: selftest: Extent I/O tests finished
[3535987.506831] BTRFS: selftest: Running btrfs_get_extent tests
[3535987.507272] BTRFS: selftest: Running hole first btrfs_get_extent test
[3535987.507614] BTRFS: selftest: Running outstanding_extents tests
[3535987.508154] BTRFS: selftest: Running qgroup tests
[3535987.508160] BTRFS: selftest: Qgroup basic add
[3535987.508205] BTRFS: selftest: Qgroup multiple refs test
[3535987.510075] BTRFS: selftest: Running free space tree tests
[3535987.547635] BTRFS: selftest: sectorsize: 8192  nodesize: 65536
[3535987.547646] BTRFS: selftest: Running btrfs free space cache tests
[3535987.547829] BTRFS: selftest: Running extent only tests
[3535987.547843] BTRFS: selftest: Running bitmap only tests
[3535987.547907] BTRFS: selftest: Running bitmap and extent tests
[3535987.548011] BTRFS: selftest: Running space stealing from bitmap to extent
[3535987.548577] BTRFS: selftest: Free space cache tests finished
[3535987.548583] BTRFS: selftest: Running extent buffer operation tests
[3535987.548588] BTRFS: selftest: Running btrfs_split_item tests
[3535987.548927] BTRFS: selftest: Running extent I/O tests
[3535987.548933] BTRFS: selftest: Running find delalloc tests
[3535987.549449] BTRFS: selftest: Running extent buffer bitmap tests
[3535987.597810] BTRFS: selftest: Extent I/O tests finished
[3535987.597817] BTRFS: selftest: Running btrfs_get_extent tests
[3535987.598229] BTRFS: selftest: Running hole first btrfs_get_extent test
[3535987.598572] BTRFS: selftest: Running outstanding_extents tests
[3535987.599111] BTRFS: selftest: Running qgroup tests
[3535987.599117] BTRFS: selftest: Qgroup basic add
[3535987.599157] BTRFS: selftest: Qgroup multiple refs test
[3535987.601014] BTRFS: selftest: Running free space tree tests
Scanning for Btrfs filesystems
[3535987.695327] random: fast init done


# mount tmpfs -t tmpfs -o size=26g /ramdisk/
# cd /ramdisk/
# for i in 1 2 3 4; do fallocate -l 4g scratch${i}; losetup --show -f scratch${i}; done
/dev/loop0
/dev/loop1
/dev/loop2
/dev/loop3

# fallocate -l 10g testvol1
# losetup --show -f testvol1
/dev/loop4
# mkfs.btrfs /dev/loop4
btrfs-progs v4.7.3
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM (10.00GiB) ...
Label:              (null)
UUID:
Node size:          16384
Sector size:        8192
Filesystem size:    10.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1    10.00GiB  /dev/loop4
	

Sep 29 12:09:44 ttip kernel: loop: module loaded
Sep 29 12:10:01 ttip kernel: BTRFS: device fsid 65949799-6eb6-4f95-bff6-40a84f522e16 devid 1 transid 5 /dev/loop4
	
	
root@ttip:/home/mator/xfstests-dev# cat local.config
export TEST_DEV=/dev/loop4
export TEST_DIR=/testvol
export SCRATCH_DEV_POOL="/dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3"
export SCRATCH_MNT=/mnt/scratch
export MOUNT_OPTIONS="-o space_cache=v2"

root@ttip:/home/mator/xfstests-dev# ./check -g auto
FSTYP         -- btrfs
PLATFORM      -- Linux/sparc64 ttip 4.8.0-rc8+
MKFS_OPTIONS  -- /dev/loop0
MOUNT_OPTIONS -- -o space_cache=v2 /dev/loop0 /mnt/scratch

btrfs/001        2s
btrfs/002        11s
btrfs/003        [not run] /dev/loop0 is a device which is not deletable
btrfs/004        18s
btrfs/005        14s
btrfs/006        3s
btrfs/007        2s
btrfs/008        1s
btrfs/009        1s
btrfs/010        [failed, exit status 1] - output mismatch (see /home/mator/xfstests-dev/results//btrfs/010.out.bad)
    --- tests/btrfs/010.out     2016-09-24 11:51:48.190888875 +0300
    +++ /home/mator/xfstests-dev/results//btrfs/010.out.bad     2016-09-29 12:11:51.102710463 +0300
    @@ -1,2 +1,2 @@
     QA output created by 010
    -done
    +number of extents mis-match!
    ...
    (Run 'diff -u tests/btrfs/010.out /home/mator/xfstests-dev/results//btrfs/010.out.bad'  to see the entire diff)
btrfs/011        [failed, exit status 1] - output mismatch (see /home/mator/xfstests-dev/results//btrfs/011.out.bad)
    --- tests/btrfs/011.out     2016-09-24 11:51:48.190888875 +0300
    +++ /home/mator/xfstests-dev/results//btrfs/011.out.bad     2016-09-29 12:12:25.764134794 +0300
    @@ -1,3 +1,2 @@
     QA output created by 011
     *** test btrfs replace
    -*** done   
    ...
    (Run 'diff -u tests/btrfs/011.out /home/mator/xfstests-dev/results//btrfs/011.out.bad'  to see the entire diff)
btrfs/012        [failed, exit status 1] - output mismatch (see /home/mator/xfstests-dev/results//btrfs/012.out.bad)
    --- tests/btrfs/012.out     2016-09-24 11:51:48.190888875 +0300
    +++ /home/mator/xfstests-dev/results//btrfs/012.out.bad     2016-09-29 12:12:33.672459590 +0300
    @@ -1 +1,3 @@
     == QA output created by 012
    +btrfs-convert failed
    +(see /home/mator/xfstests-dev/results//btrfs/012.full for details)
    ...
    (Run 'diff -u tests/btrfs/012.out /home/mator/xfstests-dev/results//btrfs/012.out.bad'  to see the entire diff)
btrfs/013        11s
btrfs/014        201s
btrfs/015        0s
btrfs/016        1s
btrfs/017        1s
btrfs/018        1s
btrfs/019        1s
btrfs/020        2s
btrfs/021        16s
btrfs/022        4s
btrfs/023        7s
btrfs/024        2s
btrfs/025        2s
btrfs/026        5s
btrfs/027        [not run] btrfs and this test needs 5 or more disks in SCRATCH_DEV_POOL
btrfs/028        31s
btrfs/029        1s
btrfs/030        2s
btrfs/031        1s
btrfs/032        1s
btrfs/033        28s
btrfs/034        9s
btrfs/035        1s
btrfs/036        47s
btrfs/037        7s
btrfs/038        2s
btrfs/039        2s
btrfs/040        1s
btrfs/041        2s
btrfs/042        1s
btrfs/043        1s
btrfs/044        2s
btrfs/045        2s
btrfs/046        7s
btrfs/047        [not run] Missing btrfs-progs send --stream-version command line option, skipped this test
btrfs/048        3s
btrfs/049        6s
btrfs/050        17s
btrfs/051        2s
btrfs/052        9s
btrfs/053        2s
btrfs/054        2s
btrfs/055        2s
btrfs/056        4s
btrfs/057        [failed, exit status 1] - output mismatch (see /home/mator/xfstests-dev/results//btrfs/057.out.bad)
    --- tests/btrfs/057.out     2016-09-24 11:51:48.206889520 +0300
    +++ /home/mator/xfstests-dev/results//btrfs/057.out.bad     2016-09-29 12:20:27.184192306 +0300
    @@ -1,3 +1,3 @@
     QA output created by 057
    -4096 4096  
    -4096 4096  
    +failed: '_scratch_mkfs -b 1g --nodesize 4096'
    +(see /home/mator/xfstests-dev/results//btrfs/057.full for details)
    ...
    (Run 'diff -u tests/btrfs/057.out /home/mator/xfstests-dev/results//btrfs/057.out.bad'  to see the entire diff)
btrfs/058        4s
btrfs/059        1s
btrfs/060        101s
btrfs/061        99s
btrfs/062        201s
btrfs/063        97s
btrfs/064        [not run] btrfs and this test needs 5 or more disks in SCRATCH_DEV_POOL
btrfs/065        [not run] btrfs and this test needs 5 or more disks in SCRATCH_DEV_POOL
btrfs/066        16s
btrfs/067        45s
btrfs/068        16s
btrfs/069        [not run] btrfs and this test needs 5 or more disks in SCRATCH_DEV_POOL
btrfs/070        [not run] btrfs and this test needs 5 or more disks in SCRATCH_DEV_POOL
btrfs/071        [not run] btrfs and this test needs 5 or more disks in SCRATCH_DEV_POOL
btrfs/072        44s
btrfs/073        17s
btrfs/074        48s
btrfs/075        [not run] Require selinux to be enabled
btrfs/076        1s
btrfs/077        2s
btrfs/078        14s
btrfs/079        [not run] This test requires at least 10GB free on /mnt/scratch to run
btrfs/080        139s
btrfs/081        1s
btrfs/082        1s
btrfs/083        2s
btrfs/084        2s
btrfs/085        4s
btrfs/086        1s
btrfs/087        2s
btrfs/088        [not run] /sys/kernel/debug/fail_make_request  not found. Seems that CONFIG_FAIL_MAKE_REQUEST kernel config option not enabled
btrfs/089        1s
btrfs/090        2s
btrfs/091        1s
btrfs/092        1s
btrfs/093        2s
btrfs/094        1s
btrfs/095        2s
btrfs/096        1s
btrfs/097        1s
btrfs/098        1s
btrfs/099        1s
btrfs/100        14s
btrfs/101       



mator@ttip:~/xfstests-dev/results/btrfs$ top -bin 1
top - 14:37:17 up  2:28,  2 users,  load average: 1.00, 1.00, 1.00
Tasks: 342 total,   2 running, 340 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.4 us,  3.8 sy,  0.0 ni, 94.8 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 33141248 total, 30396208 free,   248920 used,  2496120 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 32184752 avail Mem 

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
193017 root      20   0    9448   1008    800 R  90.9  0.0 120:59.48 btrfs


btrfs/101 hangs, unable to strace it or attach via gdb to get stack trace. Just
to remind that I have btrgs-progs from debian sid/unstable:
# apt-cache show btrfs-progs  | grep Ver 
Version: 4.7.3-1
	

Current ~/xfstests-dev/results/ (with stuck btrfs/101) available at 
http://u163.east.ru/btrfs/xfstests-dev-sparc64-4.8-btrfs-free-space-tree-and-sanity-test-fixes.tar.gz

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[Holger Hoffstätte]

On 09/29/16 14:21, Anatoly Pugachev wrote:
>> ...
>>
>> This is fixed by patch
>>
>>   "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in split_leaf"
>>
>> that's in the 4.9 queue. Other than that, the self-tests seem to pass,
>> thanks for the test. Would be good if you can test with the mentioned
>> patch included or without integrity checker. Thanks for testing.
> 
> updated git kernel to v4.8-rc8-8-gae6dd8d , applied this 
> "Btrfs: free space tree and sanity test fixes" patchset and added/applied 
> "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in split_leaf" :
> 
(snip)
> Sep 29 00:40:55 ttip kernel: BTRFS: device fsid 7bb81df9-0e2b-47f2-81ff-c08502d38da6 devid 1 transid 5 /dev/loop4
> Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): disk space caching is enabled
> Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): has skinny extents
> Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): flagging fs with big metadata feature
> Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): creating UUID tree
> Sep 29 00:41:31 ttip kernel: BTRFS: device fsid d0ee7ca3-3be0-465f-857b-19e681181923 devid 1 transid 5 /dev/loop0
> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): enabling free space tree
> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): using free space tree
> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): has skinny extents
> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): flagging fs with big metadata feature
> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): creating free space tree
> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): setting 1 ro feature flag
> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): setting 2 ro feature flag
> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): creating UUID tree
> Sep 29 00:41:32 ttip kernel: BTRFS critical (device loop4): corrupt leaf, non-root leaf's nritems is 0: block=29556736,root=1, slot=0
> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop4): leaf 29556736 total ptrs 0 free space 16283
> Sep 29 00:41:32 ttip kernel: BTRFS: assertion failed: 0, file: fs/btrfs/disk-io.c, line: 4059kernel BUG at fs/btrfs/ctree.h:3369!

Try to add https://patchwork.kernel.org/patch/9332707/ aka
"Btrfs: improve check_node to avoid reading corrupted nodes" which should return
-EIO and prevent the BUG().

-h

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[Anatoly Pugachev]

On Thu, Sep 29, 2016 at 3:52 PM, Holger Hoffstätte
<holger@applied-asynchrony.com> wrote:
> On 09/29/16 14:21, Anatoly Pugachev wrote:
>>> ...
>>>
>>> This is fixed by patch
>>>
>>>   "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in split_leaf"
>>>
>>> that's in the 4.9 queue. Other than that, the self-tests seem to pass,
>>> thanks for the test. Would be good if you can test with the mentioned
>>> patch included or without integrity checker. Thanks for testing.
>>
>> updated git kernel to v4.8-rc8-8-gae6dd8d , applied this
>> "Btrfs: free space tree and sanity test fixes" patchset and added/applied
>> "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in split_leaf" :
>>
> (snip)
>> Sep 29 00:40:55 ttip kernel: BTRFS: device fsid 7bb81df9-0e2b-47f2-81ff-c08502d38da6 devid 1 transid 5 /dev/loop4
>> Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): disk space caching is enabled
>> Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): has skinny extents
>> Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): flagging fs with big metadata feature
>> Sep 29 00:41:30 ttip kernel: BTRFS info (device loop4): creating UUID tree
>> Sep 29 00:41:31 ttip kernel: BTRFS: device fsid d0ee7ca3-3be0-465f-857b-19e681181923 devid 1 transid 5 /dev/loop0
>> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): enabling free space tree
>> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): using free space tree
>> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): has skinny extents
>> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): flagging fs with big metadata feature
>> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): creating free space tree
>> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): setting 1 ro feature flag
>> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): setting 2 ro feature flag
>> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop0): creating UUID tree
>> Sep 29 00:41:32 ttip kernel: BTRFS critical (device loop4): corrupt leaf, non-root leaf's nritems is 0: block=29556736,root=1, slot=0
>> Sep 29 00:41:32 ttip kernel: BTRFS info (device loop4): leaf 29556736 total ptrs 0 free space 16283
>> Sep 29 00:41:32 ttip kernel: BTRFS: assertion failed: 0, file: fs/btrfs/disk-io.c, line: 4059kernel BUG at fs/btrfs/ctree.h:3369!
>
> Try to add https://patchwork.kernel.org/patch/9332707/ aka
> "Btrfs: improve check_node to avoid reading corrupted nodes" which should return
> -EIO and prevent the BUG().

Guys,

I was wrong with "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in
split_leaf" , because I've used patch from email
https://marc.info/?l=linux-btrfs&m=147319873029152&w=2 which patches
fs/btrfs/ctree.c , but forgot to apply kdave git kernel commit
1ba98d086fe3a14d6a31f2f66dbab70c45d00f63 "Btrfs: detect corruption
when non-root leaf has zero item"

:-/

Should I remake all the tests?

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[David Sterba]

On Thu, Sep 29, 2016 at 04:02:35PM +0300, Anatoly Pugachev wrote:
> > Try to add https://patchwork.kernel.org/patch/9332707/ aka
> > "Btrfs: improve check_node to avoid reading corrupted nodes" which should return
> > -EIO and prevent the BUG().
> 
> Guys,
> 
> I was wrong with "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in
> split_leaf" , because I've used patch from email
> https://marc.info/?l=linux-btrfs&m=147319873029152&w=2 which patches
> fs/btrfs/ctree.c , but forgot to apply kdave git kernel commit
> 1ba98d086fe3a14d6a31f2f66dbab70c45d00f63 "Btrfs: detect corruption
> when non-root leaf has zero item"
> 
> :-/
> 
> Should I remake all the tests?

It would be much appreciated, as we probably won't get any other sparc64
testing. Thanks.

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[Anatoly Pugachev]

On Thu, Sep 29, 2016 at 5:29 PM, David Sterba <dsterba@suse.cz> wrote:
> On Thu, Sep 29, 2016 at 04:02:35PM +0300, Anatoly Pugachev wrote:
>> > Try to add https://patchwork.kernel.org/patch/9332707/ aka
>> > "Btrfs: improve check_node to avoid reading corrupted nodes" which should return
>> > -EIO and prevent the BUG().
>>
>> Guys,
>>
>> I was wrong with "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in
>> split_leaf" , because I've used patch from email
>> https://marc.info/?l=linux-btrfs&m=147319873029152&w=2 which patches
>> fs/btrfs/ctree.c , but forgot to apply kdave git kernel commit
>> 1ba98d086fe3a14d6a31f2f66dbab70c45d00f63 "Btrfs: detect corruption
>> when non-root leaf has zero item"
>>
>> :-/
>>
>> Should I remake all the tests?
>
> It would be much appreciated, as we probably won't get any other sparc64
> testing. Thanks.

David,

just tried to apply 1ba98d086fe3a14d6a31f2f66dbab70c45d00f63 "Btrfs:
detect corruption when non-root leaf has zero item"
from your btrfs development kernel tree, and patch command said that
it already applied...

mator@ttip:~/linux-2.6$ git remote -v
origin  git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
(fetch)
origin  git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
(push)
mator@ttip:~/linux-2.6$ git show -s --oneline
1ba98d086fe3a14d6a31f2f66dbab70c45d00f63
1ba98d0 Btrfs: detect corruption when non-root leaf has zero item

So my tests was correct and included all the needed patches?!

Sorry for the mess. :-/

I'm going to check patch suggested by Holger Hoffstätte ,
https://patchwork.kernel.org/patch/9332707/ ...

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[Omar Sandoval]

Hey, Anatoly,

On Sun, Sep 25, 2016 at 10:55:24AM +0300, Anatoly Pugachev wrote:
> On Thu, Sep 22, 2016 at 05:22:31PM -0700, Omar Sandoval wrote:
> > From: Omar Sandoval <osandov@fb.com>
> > 
> > This is v2 of my earlier series "Btrfs: fix free space tree
> > bitmaps+tests on big-endian systems" [1]. Patches 1, 4, and 5 are the
> > same as patches 1, 2, and 3 from the original series. I've added patch 2
> > to fix another bug I noticed (an xfstest went out earlier). Patch 3 is
> > the result of the earlier discussion here [2]. Finally, patch 6 was
> > necessary to get the sanity tests to run on my MIPS emulator.
> > 
> > This series applies to v4.8-rc7. The sanity tests pass on both x86-64
> > and MIPS, and there are no xfstests regressions. Chandan and Anatoly,
> > could you test these out as well?
> > 
> > I'm working on the btrfs-progs follow up, but these patches are safe
> > without that -- the new FREE_SPACE_TREE_VALID bit will stop all versions
> > of btrfs-progs from mounting read-write.
> 
> Omar,
> 
> applied patch to git kernel (v4.8-rc7-172-gbd5dbcb) cleanly. Did not used
> btrfs-progs.git, but debian shipped 4.7.3-1 .
> 
> (re)booted to a newly patched kernel and used xfstests.git (v1.1.0-1328-g06d4001):
> 
> # mount tmpfs -t tmpfs -o size=26g /ramdisk
> # cd /ramdisk
> # fallocate -l 10g testvol1
> # for i in 1 2 3 4; do fallocate -l 4g scratch${i}; done
> # for i in *; do losetup -f $i; done
> # losetup
> NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE         DIO
> /dev/loop0         0      0         0  0 /ramdisk/scratch1   0
> /dev/loop1         0      0         0  0 /ramdisk/scratch2   0
> /dev/loop2         0      0         0  0 /ramdisk/scratch3   0
> /dev/loop3         0      0         0  0 /ramdisk/scratch4   0
> /dev/loop4         0      0         0  0 /ramdisk/testvol1   0
> 
> mator@ttip:~/xfstests-dev$ cat local.config
> export TEST_DEV=/dev/loop4
> export TEST_DIR=/testvol
> export SCRATCH_DEV_POOL="/dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3"
> export SCRATCH_MNT=/mnt/scratch

Thanks for trying this out. I should've mentioned how best to test this.
You're going to want to add:

export MOUNT_OPTIONS="-o space_cache=v2"

to local.config. So I don't think this actually tested the free space
tree (although it looks like it might have turned up something else).

> root@ttip:/home/mator/xfstests-dev# mkfs.btrfs /dev/loop4
> btrfs-progs v4.7.3
> See http://btrfs.wiki.kernel.org for more information.
> 
> Performing full device TRIM (10.00GiB) ...
> Label:              (null)
> UUID:
> Node size:          16384
> Sector size:        8192
> Filesystem size:    10.00GiB
> Block group profiles:
>   Data:             single            8.00MiB
>   Metadata:         DUP               1.00GiB
>   System:           DUP               8.00MiB
> SSD detected:       no
> Incompat features:  extref, skinny-metadata
> Number of devices:  1
> Devices:
>    ID        SIZE  PATH
>     1    10.00GiB  /dev/loop4
> 
> root@ttip:/home/mator/xfstests-dev# ./check 'btrfs/*'

I recommend `./check -g auto` instead in order to run the relevant
generic tests as well.

> FSTYP         -- btrfs
> PLATFORM      -- Linux/sparc64 ttip 4.8.0-rc7+
> MKFS_OPTIONS  -- /dev/loop0
> MOUNT_OPTIONS -- /dev/loop0 /mnt/scratch
> 
> _check_btrfs_filesystem: filesystem on /dev/loop4 is inconsistent (see /home/mator/xfstests-dev/results//check.full)
> btrfs/001
> 
> 
> (on another screen/tmux window shell)
> 
> $ pstree -A | grep check 
>         |-sshd-+-sshd---sshd---bash---sudo---bash---check---001---mount
> 
> $ ps ax | grep mount
>  76344 pts/0    D+     0:00 /bin/mount -t btrfs /dev/loop4 /testvol
> 
> $ cat /home/mator/xfstests-dev/results//check.full
> _check_btrfs_filesystem: filesystem on /dev/loop4 is inconsistent
> *** fsck.btrfs output ***
> ERROR: cannot open device '/dev/loop4': Device or resource busy
> Couldn't open file system
> *** end fsck.btrfs output
> *** mount output ***
> sysfs on /sys type sysfs (rw,relatime)
> proc on /proc type proc (rw,relatime)
> udev on /dev type devtmpfs (rw,nosuid,relatime,size=16479064k,nr_inodes=2059883,mode=755)
> devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
> tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3314128k,mode=755)
> /dev/vdiska2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
> securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
> tmpfs on /dev/shm type tmpfs (rw)
> tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
> tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
> cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
> cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
> cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
> cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
> cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
> cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
> cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
> cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
> cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
> cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
> systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=23073)
> mqueue on /dev/mqueue type mqueue (rw,relatime)
> debugfs on /sys/kernel/debug type debugfs (rw,relatime)
> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
> /dev/vdiska1 on /boot type ext3 (rw,relatime,data=ordered)
> /dev/vdiskb1 on /home type xfs (rw,relatime,attr2,inode64,noquota)
> binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
> tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3314120k,mode=700,uid=1000,gid=1000)
> tmpfs on /ramdisk type tmpfs (rw,relatime,size=27262976k)
> /dev/loop0 on /mnt/scratch type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
> *** end mount output
> 
> in kernel logs and on console:
> 
> [3184224.438566] BTRFS: device fsid b54ec0aa-4187-419d-9de1-5ef284cd3b32 devid 1 transid 5 /dev/loop4
> [3184239.417845] BTRFS info (device loop4): disk space caching is enabled

So we're definitely not using the free space tree.

> [3184239.417865] BTRFS info (device loop4): has skinny extents
> [3184239.417872] BTRFS info (device loop4): flagging fs with big metadata feature
> [3184239.421147] BTRFS info (device loop4): creating UUID tree
> [3184240.026601] BTRFS: device fsid 5383d227-9ea2-4514-8857-641c6e2e2063 devid 1 transid 5 /dev/loop0
> [3184240.131927] BTRFS info (device loop0): disk space caching is enabled
> [3184240.131956] BTRFS info (device loop0): has skinny extents
> [3184240.131971] BTRFS info (device loop0): flagging fs with big metadata feature
> [3184240.135182] BTRFS info (device loop0): creating UUID tree
> [3184240.252534] BTRFS critical (device loop4): corrupt leaf, non-root leaf's nritems is 0: block=29556736,root=1, slot=0

That's weird. This was on a freshly created filesystem? If you don't
mind, could you try reproducing this without my patch series and making
a new thread if it still happens?

> [3184240.252558] BTRFS info (device loop4): leaf 29556736 total ptrs 0 free space 16283
> [3184240.252567] BTRFS: assertion failed: 0, file: fs/btrfs/disk-io.c, line: 4059kernel BUG at fs/btrfs/ctree.h:3369!
> [3184240.252581]               \|/ ____ \|/
> [3184240.252581]               "@'/ .. \`@"
> [3184240.252581]               /_| \__/ |_\
> [3184240.252581]                  \__U_/

Wtf, sparc?

> [3184240.252597] umount(76111): Kernel bad sw trap 5 [#1]
> [3184240.252605] CPU: 21 PID: 76111 Comm: umount Not tainted 4.8.0-rc7+ #9
> [3184240.252612] task: fff8000805feed20 task.stack: fff8000804824000
> [3184240.252620] TSTATE: 0000004411001600 TPC: 00000000100d6ccc TNPC: 00000000100d6cd0 Y: 00000000    Not tainted
> [3184240.252679] TPC: <btrfs_mark_buffer_dirty+0x1ac/0x1e0 [btrfs]>
> [3184240.252686] g0: 0000000000000000 g1: 0000000000000000 g2: 0000000000000007 g3: 0000000000000000
> [3184240.252694] g4: fff8000805feed20 g5: fff800082c66e000 g6: fff8000804824000 g7: fff800082ce00000
> [3184240.252702] o0: 000000001019cac8 o1: 0000000000000d29 o2: 000000001019c930 o3: 0000000000000fdb
> [3184240.252710] o4: 0000000001c30000 o5: 0000000000000000 sp: fff80008048263c1 ret_pc: 00000000100d6cc4
> [3184240.252731] RPC: <btrfs_mark_buffer_dirty+0x1a4/0x1e0 [btrfs]>
> [3184240.252738] l0: 000000000000154c l1: 0000000000002000 l2: 0000000000002000 l3: fff80008115fbf10
> [3184240.252746] l4: fff8000805d7a698 l5: 00000000101b8400 l6: 0000000001000000 l7: 0000000000000000
> [3184240.252754] i0: fff8000812f97968 i1: 0000000000000000 i2: 0700000000000000 i3: 000600001229fa48
> [3184240.252762] i4: fff80008134fb000 i5: 0000000000000007 i6: fff8000804826471 i7: 00000000100a0388
> [3184240.252780] I7: <__btrfs_cow_block+0x7e8/0xb80 [btrfs]>
> [3184240.252786] Call Trace:
> [3184240.252801]  [00000000100a0388] __btrfs_cow_block+0x7e8/0xb80 [btrfs]
> [3184240.252816]  [00000000100a0948] btrfs_cow_block+0x1a8/0x340 [btrfs]
> [3184240.252832]  [00000000100a6000] btrfs_search_slot+0x2a0/0xf60 [btrfs]
> [3184240.252849]  [00000000100ccff4] btrfs_del_csums+0x1b4/0x6a0 [btrfs]
> [3184240.252865]  [00000000100b4ecc] __btrfs_free_extent.isra.19+0x9ec/0x12c0 [btrfs]
> [3184240.252884]  [00000000100ba788] __btrfs_run_delayed_refs+0x548/0x1980 [btrfs]
> [3184240.252902]  [00000000100bfec8] btrfs_run_delayed_refs+0x88/0x2c0 [btrfs]
> [3184240.252921]  [00000000100dcc20] commit_cowonly_roots+0xe0/0x5c0 [btrfs]
> [3184240.252939]  [00000000100df9b4] btrfs_commit_transaction+0x534/0xd60 [btrfs]
> [3184240.252959]  [00000000100d54c4] btrfs_commit_super+0x64/0x80 [btrfs]
> [3184240.252977]  [00000000100d8a60] close_ctree+0x260/0x320 [btrfs]
> [3184240.252992]  [0000000010096cd0] btrfs_put_super+0x10/0x20 [btrfs]
> [3184240.253006]  [00000000005e7d74] generic_shutdown_super+0x74/0x100
> [3184240.253013]  [00000000005e8050] kill_anon_super+0x10/0x40
> [3184240.253028]  [00000000100969b4] btrfs_kill_super+0x14/0x100 [btrfs]
> [3184240.253035]  [00000000005e8244] deactivate_locked_super+0x44/0x80
> [3184240.253041] Disabling lock debugging due to kernel taint
> [3184240.253057] Caller[00000000100a0388]: __btrfs_cow_block+0x7e8/0xb80 [btrfs]
> [3184240.253072] Caller[00000000100a0948]: btrfs_cow_block+0x1a8/0x340 [btrfs]
> [3184240.253088] Caller[00000000100a6000]: btrfs_search_slot+0x2a0/0xf60 [btrfs]
> [3184240.253105] Caller[00000000100ccff4]: btrfs_del_csums+0x1b4/0x6a0 [btrfs]
> [3184240.253121] Caller[00000000100b4ecc]: __btrfs_free_extent.isra.19+0x9ec/0x12c0 [btrfs]
> [3184240.253139] Caller[00000000100ba788]: __btrfs_run_delayed_refs+0x548/0x1980 [btrfs]
> [3184240.253158] Caller[00000000100bfec8]: btrfs_run_delayed_refs+0x88/0x2c0 [btrfs]
> [3184240.253178] Caller[00000000100dcc20]: commit_cowonly_roots+0xe0/0x5c0 [btrfs]
> [3184240.253198] Caller[00000000100df9b4]: btrfs_commit_transaction+0x534/0xd60 [btrfs]
> [3184240.253218] Caller[00000000100d54c4]: btrfs_commit_super+0x64/0x80 [btrfs]
> [3184240.253236] Caller[00000000100d8a60]: close_ctree+0x260/0x320 [btrfs]
> [3184240.253251] Caller[0000000010096cd0]: btrfs_put_super+0x10/0x20 [btrfs]
> [3184240.253258] Caller[00000000005e7d74]: generic_shutdown_super+0x74/0x100
> [3184240.253265] Caller[00000000005e8050]: kill_anon_super+0x10/0x40
> [3184240.253279] Caller[00000000100969b4]: btrfs_kill_super+0x14/0x100 [btrfs]
> [3184240.253287] Caller[00000000005e8244]: deactivate_locked_super+0x44/0x80
> [3184240.253293] Caller[00000000005e8f44]: deactivate_super+0x64/0x80
> [3184240.253304] Caller[000000000060707c]: cleanup_mnt+0x3c/0xa0
> [3184240.253310] Caller[000000000060714c]: __cleanup_mnt+0xc/0x20
> [3184240.253321] Caller[00000000004859e0]: task_work_run+0xa0/0xe0
> [3184240.253330] Caller[000000000042e144]: do_notify_resume+0x64/0x80
> [3184240.253338] Caller[0000000000404b44]: __handle_signal+0xc/0x2c
> [3184240.253344] Caller[fff8000100150db4]: 0xfff8000100150db4
> [3184240.253349] Instruction DUMP: 92102d29  7c0d43df  901222c8 <91d02005> d25e2008  d4422b40  7c19cac2  90022af0  106fffe1
> [3184240.312832] run fstests btrfs/001 at 2016-09-25 10:25:48
> 
> 
> Do I need to redo test with btrfs-progs.git version instead of debian one?

Unless progs is making bogus filesystems, this shouldn't make a
difference.

> Thanks.
> 
> PS: boot log of btrfs module:
> 
> [3102837.407118] xor: automatically using best checksumming function:
> [3102837.446156]    Niagara   :  4898.000 MB/sec
> [3102837.870398] Btrfs loaded, crc32c=crc32c-sparc64, debug=on, assert=on, integrity-checker=on
> [3102837.871047] BTRFS: selftest: sectorsize: 8192  nodesize: 8192
> [3102837.871055] BTRFS: selftest: Running btrfs free space cache tests
> [3102837.871242] BTRFS: selftest: Running extent only tests
> [3102837.871261] BTRFS: selftest: Running bitmap only tests
> [3102837.871338] BTRFS: selftest: Running bitmap and extent tests
> [3102837.871443] BTRFS: selftest: Running space stealing from bitmap to extent
> [3102837.872016] BTRFS: selftest: Free space cache tests finished
> [3102837.872023] BTRFS: selftest: Running extent buffer operation tests
> [3102837.872028] BTRFS: selftest: Running btrfs_split_item tests
> [3102837.872394] BTRFS: selftest: Running extent I/O tests
> [3102837.872401] BTRFS: selftest: Running find delalloc tests
> [3102837.872953] BTRFS: selftest: Running extent buffer bitmap tests
> [3102837.921398] BTRFS: selftest: Extent I/O tests finished
> [3102837.921409] BTRFS: selftest: Running btrfs_get_extent tests
> [3102837.921901] BTRFS: selftest: Running hole first btrfs_get_extent test
> [3102837.922261] BTRFS: selftest: Running outstanding_extents tests
> [3102837.922831] BTRFS: selftest: Running qgroup tests
> [3102837.922838] BTRFS: selftest: Qgroup basic add
> [3102837.922901] BTRFS: selftest: Qgroup multiple refs test
> [3102837.924768] BTRFS: selftest: Running free space tree tests
> [3102837.962292] BTRFS: selftest: sectorsize: 8192  nodesize: 16384
> [3102837.962303] BTRFS: selftest: Running btrfs free space cache tests
> [3102837.962486] BTRFS: selftest: Running extent only tests
> [3102837.962502] BTRFS: selftest: Running bitmap only tests
> [3102837.962576] BTRFS: selftest: Running bitmap and extent tests
> [3102837.962681] BTRFS: selftest: Running space stealing from bitmap to extent
> [3102837.963250] BTRFS: selftest: Free space cache tests finished
> [3102837.963257] BTRFS: selftest: Running extent buffer operation tests
> [3102837.963262] BTRFS: selftest: Running btrfs_split_item tests
> [3102837.963601] BTRFS: selftest: Running extent I/O tests
> [3102837.963607] BTRFS: selftest: Running find delalloc tests
> [3102837.964145] BTRFS: selftest: Running extent buffer bitmap tests
> [3102838.012532] BTRFS: selftest: Extent I/O tests finished
> [3102838.012539] BTRFS: selftest: Running btrfs_get_extent tests
> [3102838.012951] BTRFS: selftest: Running hole first btrfs_get_extent test
> [3102838.013292] BTRFS: selftest: Running outstanding_extents tests
> [3102838.013830] BTRFS: selftest: Running qgroup tests
> [3102838.013836] BTRFS: selftest: Qgroup basic add
> [3102838.013879] BTRFS: selftest: Qgroup multiple refs test
> [3102838.015744] BTRFS: selftest: Running free space tree tests
> [3102838.053230] BTRFS: selftest: sectorsize: 8192  nodesize: 32768
> [3102838.053237] BTRFS: selftest: Running btrfs free space cache tests
> [3102838.053419] BTRFS: selftest: Running extent only tests
> [3102838.053430] BTRFS: selftest: Running bitmap only tests
> [3102838.053493] BTRFS: selftest: Running bitmap and extent tests
> [3102838.053597] BTRFS: selftest: Running space stealing from bitmap to extent
> [3102838.054167] BTRFS: selftest: Free space cache tests finished
> [3102838.054173] BTRFS: selftest: Running extent buffer operation tests
> [3102838.054179] BTRFS: selftest: Running btrfs_split_item tests
> [3102838.054512] BTRFS: selftest: Running extent I/O tests
> [3102838.054518] BTRFS: selftest: Running find delalloc tests
> [3102838.054956] BTRFS: selftest: Running extent buffer bitmap tests
> [3102838.103317] BTRFS: selftest: Extent I/O tests finished
> [3102838.103324] BTRFS: selftest: Running btrfs_get_extent tests
> [3102838.103728] BTRFS: selftest: Running hole first btrfs_get_extent test
> [3102838.104068] BTRFS: selftest: Running outstanding_extents tests
> [3102838.104603] BTRFS: selftest: Running qgroup tests
> [3102838.104609] BTRFS: selftest: Qgroup basic add
> [3102838.104644] BTRFS: selftest: Qgroup multiple refs test
> [3102838.106503] BTRFS: selftest: Running free space tree tests
> [3102838.143985] BTRFS: selftest: sectorsize: 8192  nodesize: 65536
> [3102838.143992] BTRFS: selftest: Running btrfs free space cache tests
> [3102838.144173] BTRFS: selftest: Running extent only tests
> [3102838.144184] BTRFS: selftest: Running bitmap only tests
> [3102838.144247] BTRFS: selftest: Running bitmap and extent tests
> [3102838.144350] BTRFS: selftest: Running space stealing from bitmap to extent
> [3102838.144915] BTRFS: selftest: Free space cache tests finished
> [3102838.144921] BTRFS: selftest: Running extent buffer operation tests
> [3102838.144926] BTRFS: selftest: Running btrfs_split_item tests
> [3102838.145260] BTRFS: selftest: Running extent I/O tests
> [3102838.145266] BTRFS: selftest: Running find delalloc tests
> [3102838.145700] BTRFS: selftest: Running extent buffer bitmap tests
> [3102838.194039] BTRFS: selftest: Extent I/O tests finished
> [3102838.194045] BTRFS: selftest: Running btrfs_get_extent tests
> [3102838.194462] BTRFS: selftest: Running hole first btrfs_get_extent test
> [3102838.194804] BTRFS: selftest: Running outstanding_extents tests
> [3102838.195339] BTRFS: selftest: Running qgroup tests
> [3102838.195345] BTRFS: selftest: Qgroup basic add
> [3102838.195380] BTRFS: selftest: Qgroup multiple refs test
> [3102838.197235] BTRFS: selftest: Running free space tree tests
> [3102838.288732] random: fast init done

At least it looks like the selftests passed :)

Thanks so much for trying this out! Let me know what you find.

-- 
Omar

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

[Chandan Rajendra]

On Thursday, September 22, 2016 05:22:31 PM Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
> 
> This is v2 of my earlier series "Btrfs: fix free space tree
> bitmaps+tests on big-endian systems" [1]. Patches 1, 4, and 5 are the
> same as patches 1, 2, and 3 from the original series. I've added patch 2
> to fix another bug I noticed (an xfstest went out earlier). Patch 3 is
> the result of the earlier discussion here [2]. Finally, patch 6 was
> necessary to get the sanity tests to run on my MIPS emulator.
> 
> This series applies to v4.8-rc7. The sanity tests pass on both x86-64
> and MIPS, and there are no xfstests regressions. Chandan and Anatoly,
> could you test these out as well?

Hello Omar,

I have executed xfstests on a big endian ppc64 guest with 'MOUNT_OPTIONS="-o
space_cache=v2"' config option. I have also executed generic/127 on a
filesystem created using "fragment-free-space-tree.py" that you had provided
sometime ago. I did not notice any regressions during the test runs.

Tested-by: Chandan Rajendra <chandan@linux.vnet.ibm.com>

> 
> I'm working on the btrfs-progs follow up, but these patches are safe
> without that -- the new FREE_SPACE_TREE_VALID bit will stop all versions
> of btrfs-progs from mounting read-write.
> 
> Thanks!
> 
> 1: http://marc.info/?l=linux-btrfs&m=146853909905570&w=2
> 2: http://marc.info/?l=linux-btrfs&m=147448992301110&w=2
> 
> Cc: Chandan Rajendra <chandan@linux.vnet.ibm.com>
> Cc: Anatoly Pugachev <matorola@gmail.com>
> 
> Omar Sandoval (6):
>   Btrfs: fix free space tree bitmaps on big-endian systems
>   Btrfs: fix mount -o clear_cache,space_cache=v2
>   Btrfs: catch invalid free space trees
>   Btrfs: fix extent buffer bitmap tests on big-endian systems
>   Btrfs: expand free space tree sanity tests to catch endianness bug
>   Btrfs: use less memory for delalloc sanity tests
> 
>  fs/btrfs/ctree.h                       |   3 +-
>  fs/btrfs/disk-io.c                     |  33 ++++---
>  fs/btrfs/extent_io.c                   |  64 +++++++++----
>  fs/btrfs/extent_io.h                   |  22 +++++
>  fs/btrfs/free-space-tree.c             |  19 ++--
>  fs/btrfs/tests/extent-io-tests.c       |  95 +++++++++++--------
>  fs/btrfs/tests/free-space-tree-tests.c | 164 +++++++++++++++++++--------------
>  include/uapi/linux/btrfs.h             |  10 +-
>  8 files changed, 261 insertions(+), 149 deletions(-)
> 
> 

-- 
chandan