[PATCH v4 00/18] btrfs-progs: lowmem check: avoid extents overwrite

[PATCH v4 00/18] btrfs-progs: lowmem check: avoid extents overwrite

[Su Yue]

This patchset can be fetched from my github:
https://github.com/Damenly/btrfs-progs/tree/lowmem
based on unmerged patchset whose cover:
      [PATCH 0/3] btrfs-progs: Split original mode check to its own
      Author: Qu Wenruo <wqu@suse.com>
      
I'm sorry to send patches based on unmerged patch if it let you feel
uncomfortable.
I think the three patches from Qu are good enough so I send it before
my vacation.

Patch[1-3] fix minor problems of lowmem repair.

Patch[4-8] introduce two ways to avoid extents overwrite:
1) Traverse trees and exclude all metadata blocks.
   It's time-inefficient for large filesystems.
2) Mark all existed chunks full, allocate new chunk for CoW and
   records chunk start.
   If the last allocated chunk is almost full, allocated a new one.
2) is More efficient than 1). However, it can't handle situations
like no space(fsck/004).
Lowmem repair will try method 2 first and then method 1.

Patch[9-17] remove parameters @trans in functions for lowmem repair.
They try to avoid extents overwrite if necessary and start
transactions by themselves.

Patch[18] adds a test image.
Those patches are mainly for lowmem repair. Original mode is not
influenced.

---
Changlog:
v4->v3:
 - Remove global enum extents_operation to simplify
   avoid_extents_overwrite() and its cleanup.
 - Rebase after work of check split.
 
v3->v2:
 - check_btrfs_root() returns FATAL_ERROR if check_fs_first_inode()
   failed. Thanks Nikolay Borisov.
 - Add function try_to_force_cow_in_new_chunk() and global u64
   varaiable to record start of the last allocated chunk.
 - Remove unused EXTENTS_PIN in enum lowmem_extents_operation.
 
v2->v1:
 - Let @err in check_btrfs_root() record err bits but excluded
   negative values.
 - Do not delete a line of code to release path after extent item'
   insertion in repair_extent_data_item().
 - Add patch[3].
 - Force CoW in new allocated chunk to avoid extents overwrite.
 - Remove parameters @trans in check_chunks_and_extents_v2() and
   related callees.
 - Repair functions for lowmem mode call try_avoid_extents_overwrite()
   and start transactions.
   
Su Yue (18):
  btrfs-progs: lowmem check: release path in repair_extent_data_item()
  btrfs-progs: lowmem check: record returned errors after
    walk_down_tree_v2()
  btrfs-progs: lowmem check: assign @parent early in
    repair_extent_data_item()
  btrfs-progs: lowmem check: exclude extents of metadata blocks
  btrfs-progs: lowmem check: introduce mark/clear_block_groups_full()
  btrfs-progs: lowmem check: introduce try_force_cow_in_new_chunk()
  btrfs-progs: lowmem check: introduce avoid_extents_overwrite()
  btrfs-progs: lowmem check: exclude extents if init-extent-tree in
    lowmem
  btrfs-progs: lowmem check: start to remove parameters @trans in lowmem
  btrfs-progs: lowmem check: remove parameter @trans of
    delete_extent_item()
  btrfs-progs: lowmem check: remove parameter @trans of
    repair_chunk_item()
  btrfs-progs: lowmem check: remove parameter @trans of
    repair_extent_item()
  btrfs-progs: lowmem check: remove parameter @trans of
    check_leaf_items()
  btrfs-progs: lowmem check: remove parameter @trans of
    repair_tree_back_ref()
  btrfs-progs: lowmem check: remove parameter @trans of
    check_btrfs_root()
  btrfs-progs: lowmem check: introduce repair_block_accounting()
  btrfs-progs: lowmem check: end of removing parameters @trans in lowmem
  btrfs-progs: fsck-tests: add image no extent with normal device size

 check/mode-common.c                                | 100 +++-
 check/mode-common.h                                |   4 +-
 check/mode-lowmem.c                                | 560 +++++++++++++++++----
 check/mode-lowmem.h                                |   1 +
 cmds-check.c                                       |   3 +-
 .../014-no-extent-info/.lowmem_repairable          |   0
 .../fsck-tests/014-no-extent-info/no_extent.raw.xz | Bin 0 -> 28084 bytes
 .../{default_case.img => no_extent_bad_dev.img}    | Bin
 8 files changed, 561 insertions(+), 107 deletions(-)
 create mode 100644 tests/fsck-tests/014-no-extent-info/.lowmem_repairable
 create mode 100644 tests/fsck-tests/014-no-extent-info/no_extent.raw.xz
 rename tests/fsck-tests/014-no-extent-info/{default_case.img => no_extent_bad_dev.img} (100%)

-- 
2.16.1

[PATCH v4 01/18] btrfs-progs: lowmem check: release path in repair_extent_data_item()

[Su Yue]

In repair_extent_data_item(), path is not be released if some
errors occurs which causes extent buffer leak.

So release path in end of the function.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index 62bcf3d2e126..d168a3ddd5e5 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -2537,6 +2537,7 @@ static int repair_extent_data_item(struct btrfs_trans_handle *trans,
 
 	err &= ~BACKREF_MISSING;
 out:
+	btrfs_release_path(&path);
 	if (ret)
 		error("can't repair root %llu extent data item[%llu %llu]",
 		      root->objectid, disk_bytenr, num_bytes);
-- 
2.16.1

[PATCH v4 02/18] btrfs-progs: lowmem check: record returned errors after walk_down_tree_v2()

[Su Yue]

In lowmem mode with '--repair', check_chunks_and_extents_v2()
will fix accounting in block groups and clear the error
bit BG_ACCOUNTING_ERROR.
However, return value of check_btrfs_root() doesn't contain error bits.

If extent tree is on error, lowmem repair always prints error and
returns nonzero value even the filesystem is fine after repair.

Introduce FATAL_ERROR for lowmem mode to represents negative return
values since negative and positive can't not be mixed in bits operations.

Then let check_btrfs_root() return error bits.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 10 +++++-----
 check/mode-lowmem.h |  1 +
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index d168a3ddd5e5..18ec6db098e7 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -4215,7 +4215,7 @@ out:
  *                otherwise means check fs tree(s) items relationship and
  *		  @root MUST be a fs tree root.
  * Returns 0      represents OK.
- * Returns not 0  represents error.
+ * Returns >0     represents error bits.
  */
 static int check_btrfs_root(struct btrfs_trans_handle *trans,
 			    struct btrfs_root *root, unsigned int ext_ref,
@@ -4238,7 +4238,7 @@ static int check_btrfs_root(struct btrfs_trans_handle *trans,
 		 */
 		ret = check_fs_first_inode(root, ext_ref);
 		if (ret < 0)
-			return ret;
+			return FATAL_ERROR;
 	}
 
 
@@ -4266,11 +4266,11 @@ static int check_btrfs_root(struct btrfs_trans_handle *trans,
 		ret = walk_down_tree(trans, root, &path, &level, &nrefs,
 				     ext_ref, check_all);
 
-		err |= !!ret;
-
+		if (ret > 0)
+			err |= ret;
 		/* if ret is negative, walk shall stop */
 		if (ret < 0) {
-			ret = err;
+			ret = err | FATAL_ERROR;
 			break;
 		}
 
diff --git a/check/mode-lowmem.h b/check/mode-lowmem.h
index 73d5799951b7..e7ba62e2413e 100644
--- a/check/mode-lowmem.h
+++ b/check/mode-lowmem.h
@@ -43,6 +43,7 @@
 #define DIR_INDEX_MISMATCH      (1<<19) /* INODE_INDEX found but not match */
 #define DIR_COUNT_AGAIN         (1<<20) /* DIR isize should be recalculated */
 #define BG_ACCOUNTING_ERROR     (1<<21) /* Block group accounting error */
+#define FATAL_ERROR             (1<<22) /* Fatal bit for errno */
 
 /*
  * Error bit for low memory mode check.
-- 
2.16.1

[PATCH v4 03/18] btrfs-progs: lowmem check: assign @parent early in repair_extent_data_item()

[Su Yue]

The variable @eb is assigned to leaf in fs_tree before insertion of
backref. It will causes wrong parent of new inserted backref.

Set @parent at beginning solves the problem.

Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index 18ec6db098e7..1fc84f1e8c44 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -2475,6 +2475,11 @@ static int repair_extent_data_item(struct btrfs_trans_handle *trans,
 	extent_offset = btrfs_file_extent_offset(eb, fi);
 	offset = file_offset - extent_offset;
 
+	if (nrefs->full_backref[0])
+		parent = btrfs_header_bytenr(eb);
+	else
+		parent = 0;
+
 	/* now repair only adds backref */
 	if ((err & BACKREF_MISSING) == 0)
 		return err;
@@ -2516,11 +2521,6 @@ static int repair_extent_data_item(struct btrfs_trans_handle *trans,
 		btrfs_release_path(&path);
 	}
 
-	if (nrefs->full_backref[0])
-		parent = btrfs_header_bytenr(eb);
-	else
-		parent = 0;
-
 	ret = btrfs_inc_extent_ref(trans, root, disk_bytenr, num_bytes, parent,
 				   root->objectid,
 		   parent ? BTRFS_FIRST_FREE_OBJECTID : fi_key.objectid,
-- 
2.16.1

[PATCH v4 04/18] btrfs-progs: lowmem check: exclude extents of metadata blocks

[Su Yue]

Commit d17d6663c99c ("btrfs-progs: lowmem check: Fix regression which
screws up extent allocator") removes pin_metadata_blocks() from
lowmem repair.
So we have to find another way to exclude extents which should be
occupied by tree blocks.

Introduce exclude_metadata_blocks() to mark extents of all tree
blocks dirty in fs_info->excluded_extents.
Export it since it will be used in lowmem too.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-common.c | 73 +++++++++++++++++++++++++++++++++++++++++++++--------
 check/mode-common.h |  2 ++
 2 files changed, 65 insertions(+), 10 deletions(-)

diff --git a/check/mode-common.c b/check/mode-common.c
index e6d8ebe8b9b7..acceb24b9597 100644
--- a/check/mode-common.c
+++ b/check/mode-common.c
@@ -377,40 +377,54 @@ int zero_log_tree(struct btrfs_root *root)
 	return ret;
 }
 
-static int pin_down_tree_blocks(struct btrfs_fs_info *fs_info,
-				struct extent_buffer *eb, int tree_root)
+static int traverse_tree_blocks(struct btrfs_fs_info *fs_info,
+				struct extent_buffer *eb, int tree_root,
+				int pin)
 {
 	struct extent_buffer *tmp;
 	struct btrfs_root_item *ri;
 	struct btrfs_key key;
+	struct extent_io_tree *tree;
 	u64 bytenr;
 	int level = btrfs_header_level(eb);
 	int nritems;
 	int ret;
 	int i;
+	u64 end = eb->start + eb->len;
 
+	if (pin)
+		tree = &fs_info->pinned_extents;
+	else
+		tree = fs_info->excluded_extents;
 	/*
-	 * If we have pinned this block before, don't pin it again.
+	 * If we have pinned/excluded this block before, don't do it again.
 	 * This can not only avoid forever loop with broken filesystem
 	 * but also give us some speedups.
 	 */
-	if (test_range_bit(&fs_info->pinned_extents, eb->start,
-			   eb->start + eb->len - 1, EXTENT_DIRTY, 0))
+	if (test_range_bit(tree, eb->start, end - 1, EXTENT_DIRTY, 0))
 		return 0;
 
-	btrfs_pin_extent(fs_info, eb->start, eb->len);
+	if (pin)
+		btrfs_pin_extent(fs_info, eb->start, eb->len);
+	else
+		set_extent_dirty(tree, eb->start, end - 1);
 
 	nritems = btrfs_header_nritems(eb);
 	for (i = 0; i < nritems; i++) {
 		if (level == 0) {
+			bool is_extent_root;
 			btrfs_item_key_to_cpu(eb, &key, i);
 			if (key.type != BTRFS_ROOT_ITEM_KEY)
 				continue;
 			/* Skip the extent root and reloc roots */
-			if (key.objectid == BTRFS_EXTENT_TREE_OBJECTID ||
-			    key.objectid == BTRFS_TREE_RELOC_OBJECTID ||
+			if (key.objectid == BTRFS_TREE_RELOC_OBJECTID ||
 			    key.objectid == BTRFS_DATA_RELOC_TREE_OBJECTID)
 				continue;
+			is_extent_root =
+				key.objectid == BTRFS_EXTENT_TREE_OBJECTID;
+			/* If pin, skip the extent root */
+			if (pin && is_extent_root)
+				continue;
 			ri = btrfs_item_ptr(eb, i, struct btrfs_root_item);
 			bytenr = btrfs_disk_root_bytenr(eb, ri);
 
@@ -425,7 +439,7 @@ static int pin_down_tree_blocks(struct btrfs_fs_info *fs_info,
 				fprintf(stderr, "Error reading root block\n");
 				return -EIO;
 			}
-			ret = pin_down_tree_blocks(fs_info, tmp, 0);
+			ret = traverse_tree_blocks(fs_info, tmp, 0, pin);
 			free_extent_buffer(tmp);
 			if (ret)
 				return ret;
@@ -444,7 +458,8 @@ static int pin_down_tree_blocks(struct btrfs_fs_info *fs_info,
 				fprintf(stderr, "Error reading tree block\n");
 				return -EIO;
 			}
-			ret = pin_down_tree_blocks(fs_info, tmp, tree_root);
+			ret = traverse_tree_blocks(fs_info, tmp, tree_root,
+						   pin);
 			free_extent_buffer(tmp);
 			if (ret)
 				return ret;
@@ -454,6 +469,12 @@ static int pin_down_tree_blocks(struct btrfs_fs_info *fs_info,
 	return 0;
 }
 
+static int pin_down_tree_blocks(struct btrfs_fs_info *fs_info,
+				struct extent_buffer *eb, int tree_root)
+{
+	return traverse_tree_blocks(fs_info, eb, tree_root, 1);
+}
+
 static int pin_metadata_blocks(struct btrfs_fs_info *fs_info)
 {
 	int ret;
@@ -465,6 +486,38 @@ static int pin_metadata_blocks(struct btrfs_fs_info *fs_info)
 	return pin_down_tree_blocks(fs_info, fs_info->tree_root->node, 1);
 }
 
+static int exclude_tree_blocks(struct btrfs_fs_info *fs_info,
+				struct extent_buffer *eb, int tree_root)
+{
+	return traverse_tree_blocks(fs_info, eb, tree_root, 0);
+}
+
+int exclude_metadata_blocks(struct btrfs_fs_info *fs_info)
+{
+	int ret;
+	struct extent_io_tree *excluded_extents;
+
+	excluded_extents = malloc(sizeof(*excluded_extents));
+	if (!excluded_extents)
+		return -ENOMEM;
+	extent_io_tree_init(excluded_extents);
+	fs_info->excluded_extents = excluded_extents;
+
+	ret = exclude_tree_blocks(fs_info, fs_info->chunk_root->node, 0);
+	if (ret)
+		return ret;
+	return exclude_tree_blocks(fs_info, fs_info->tree_root->node, 1);
+}
+
+void cleanup_excluded_extents(struct btrfs_fs_info *fs_info)
+{
+	if (fs_info->excluded_extents) {
+		extent_io_tree_cleanup(fs_info->excluded_extents);
+		free(fs_info->excluded_extents);
+	}
+	fs_info->excluded_extents = NULL;
+}
+
 static int reset_block_groups(struct btrfs_fs_info *fs_info)
 {
 	struct btrfs_block_group_cache *cache;
diff --git a/check/mode-common.h b/check/mode-common.h
index e1835d94ab11..e2a824a318c1 100644
--- a/check/mode-common.h
+++ b/check/mode-common.h
@@ -134,4 +134,6 @@ bool is_super_size_valid(struct btrfs_fs_info *fs_info);
 int check_space_cache(struct btrfs_root *root);
 int check_csums(struct btrfs_root *root);
 int recow_extent_buffer(struct btrfs_root *root, struct extent_buffer *eb);
+int exclude_metadata_blocks(struct btrfs_fs_info *fs_info);
+void cleanup_excluded_extents(struct btrfs_fs_info *fs_info);
 #endif
-- 
2.16.1

[PATCH v4 05/18] btrfs-progs: lowmem check: introduce mark/clear_block_groups_full()

[Su Yue]

Excluding or pining all metadata blocks is not time-efficient for large
storage filesystems.
Here is another way to mark all metadata block groups full and allocate
a new chunk for CoW. So new reservered extents never overwrite
extents.

Introduce modify_block_groups_cache() to modify all blocks groups
cache state and set all extents in block groups unfree in free space
cache.
mark/clear_block_groups_full() wraps above function.

Suggested-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index 1fc84f1e8c44..a200c28a9cf7 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -233,6 +233,99 @@ static int update_nodes_refs(struct btrfs_root *root, u64 bytenr,
 	return 0;
 }
 
+/*
+ * Mark all extents unfree in the block group. And set @block_group->cached
+ * according to @cache.
+ */
+static int modify_block_group_cache(struct btrfs_fs_info *fs_info,
+		    struct btrfs_block_group_cache *block_group, int cache)
+{
+	struct extent_io_tree *free_space_cache = &fs_info->free_space_cache;
+	u64 start = block_group->key.objectid;
+	u64 end = start + block_group->key.offset;
+
+	if (cache && !block_group->cached) {
+		block_group->cached = 1;
+		clear_extent_dirty(free_space_cache, start, end - 1);
+	}
+
+	if (!cache && block_group->cached) {
+		block_group->cached = 0;
+		clear_extent_dirty(free_space_cache, start, end - 1);
+	}
+	return 0;
+}
+
+/*
+ * Modify block groups which have @flags unfree in free space cache.
+ *
+ * @cache: if 0, clear block groups cache state;
+ *         not 0, mark blocks groups cached.
+ */
+static int modify_block_groups_cache(struct btrfs_fs_info *fs_info, u64 flags,
+				     int cache)
+{
+	struct btrfs_root *root = fs_info->extent_root;
+	struct btrfs_key key;
+	struct btrfs_path path;
+	struct btrfs_block_group_cache *bg_cache;
+	struct btrfs_block_group_item *bi;
+	struct btrfs_block_group_item bg_item;
+	struct extent_buffer *eb;
+	int slot;
+	int ret;
+
+	key.objectid = 0;
+	key.type = BTRFS_BLOCK_GROUP_ITEM_KEY;
+	key.offset = 0;
+
+	btrfs_init_path(&path);
+	ret = btrfs_search_slot(NULL, root, &key, &path, 0, 0);
+	if (ret < 0) {
+		error("fail to search block groups due to %s", strerror(-ret));
+		goto out;
+	}
+
+	while (1) {
+		eb = path.nodes[0];
+		slot = path.slots[0];
+		btrfs_item_key_to_cpu(eb, &key, slot);
+		bg_cache = btrfs_lookup_block_group(fs_info, key.objectid);
+		if (!bg_cache) {
+			ret = -ENOENT;
+			goto out;
+		}
+
+		bi = btrfs_item_ptr(eb, slot, struct btrfs_block_group_item);
+		read_extent_buffer(eb, &bg_item, (unsigned long)bi,
+				   sizeof(bg_item));
+		if (btrfs_block_group_flags(&bg_item) & flags)
+			modify_block_group_cache(fs_info, bg_cache, cache);
+
+		ret = btrfs_next_item(root, &path);
+		if (ret > 0) {
+			ret = 0;
+			goto out;
+		}
+		if (ret < 0)
+			goto out;
+	}
+
+out:
+	btrfs_release_path(&path);
+	return ret;
+}
+
+static int mark_block_groups_full(struct btrfs_fs_info *fs_info, u64 flags)
+{
+	return modify_block_groups_cache(fs_info, flags, 1);
+}
+
+static int clear_block_groups_full(struct btrfs_fs_info *fs_info, u64 flags)
+{
+	return modify_block_groups_cache(fs_info, flags, 0);
+}
+
 /*
  * This function only handles BACKREF_MISSING,
  * If corresponding extent item exists, increase the ref, else insert an extent
-- 
2.16.1

[PATCH v4 06/18] btrfs-progs: lowmem check: introduce try_force_cow_in_new_chunk()

[Su Yue]

Introduce create_chunk_and_block_block_group() to allocate new chunk
and corresponding block group.

The new function force_cow_in_new_chunk() first allocates new chunk
and records its start.
Then it modifies all metadata block groups cached and full.
Finally it marks the new block group uncached and unfree.
In the next CoW, extents states will be updated automatically by
cache_block_group().

New function try_to_force_cow_in_new_chunk() will try to mark block
groups full, allocate a new chunk and records the start.
If the last allocated chunk is almost full, a new chunk will be
allocated.

Suggested-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 165 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 165 insertions(+)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index a200c28a9cf7..3649d570e11c 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -326,6 +326,171 @@ static int clear_block_groups_full(struct btrfs_fs_info *fs_info, u64 flags)
 	return modify_block_groups_cache(fs_info, flags, 0);
 }
 
+static int create_chunk_and_block_group(struct btrfs_fs_info *fs_info,
+					u64 flags, u64 *start, u64 *nbytes)
+{
+	struct btrfs_trans_handle *trans;
+	struct btrfs_root *root = fs_info->extent_root;
+	int ret;
+
+	if ((flags & BTRFS_BLOCK_GROUP_TYPE_MASK) == 0)
+		return -EINVAL;
+
+	trans = btrfs_start_transaction(root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		error("error starting transaction %s", strerror(-ret));
+		return ret;
+	}
+	ret = btrfs_alloc_chunk(trans, fs_info, start, nbytes, flags);
+	if (ret) {
+		error("fail to allocate new chunk %s", strerror(-ret));
+		goto out;
+	}
+	ret = btrfs_make_block_group(trans, fs_info, 0, flags, *start,
+				     *nbytes);
+	if (ret) {
+		error("fail to make block group for chunk %llu %llu %s",
+		      *start, *nbytes, strerror(-ret));
+		goto out;
+	}
+out:
+	btrfs_commit_transaction(trans, root);
+	return ret;
+}
+
+static int force_cow_in_new_chunk(struct btrfs_fs_info *fs_info,
+				  u64 *start_ret)
+{
+	struct btrfs_block_group_cache *bg;
+	u64 start;
+	u64 nbytes;
+	u64 alloc_profile;
+	u64 flags;
+	int ret;
+
+	alloc_profile = (fs_info->avail_metadata_alloc_bits &
+			 fs_info->metadata_alloc_profile);
+	flags = BTRFS_BLOCK_GROUP_METADATA | alloc_profile;
+	if (btrfs_fs_incompat(fs_info, MIXED_GROUPS))
+		flags |= BTRFS_BLOCK_GROUP_DATA;
+
+	ret = create_chunk_and_block_group(fs_info, flags, &start, &nbytes);
+	if (ret)
+		goto err;
+	printf("Created new chunk [%llu %llu]\n", start, nbytes);
+
+	flags = BTRFS_BLOCK_GROUP_METADATA;
+	/* Mark all metadata block groups cached and full in free space*/
+	ret = mark_block_groups_full(fs_info, flags);
+	if (ret)
+		goto clear_bgs_full;
+
+	bg = btrfs_lookup_block_group(fs_info, start);
+	if (!bg) {
+		ret = -ENOENT;
+		error("fail to look up block group %llu %llu", start, nbytes);
+		goto clear_bgs_full;
+	}
+
+	/* Clear block group cache just allocated */
+	ret = modify_block_group_cache(fs_info, bg, 0);
+	if (ret)
+		goto clear_bgs_full;
+	if (start_ret)
+		*start_ret = start;
+	return 0;
+
+clear_bgs_full:
+	clear_block_groups_full(fs_info, flags);
+err:
+	return ret;
+}
+
+/*
+ * Returns 0 means not almost full.
+ * Returns >0 means almost full.
+ * Returns <0 means fatal error.
+ */
+static int is_chunk_almost_full(struct btrfs_fs_info *fs_info, u64 start)
+{
+	struct btrfs_path path;
+	struct btrfs_key key;
+	struct btrfs_root *root = fs_info->extent_root;
+	struct btrfs_block_group_item *bi;
+	struct btrfs_block_group_item bg_item;
+	struct extent_buffer *eb;
+	u64 used;
+	u64 total;
+	u64 min_free;
+	int ret;
+	int slot;
+
+	key.objectid = start;
+	key.type = BTRFS_BLOCK_GROUP_ITEM_KEY;
+	key.offset = (u64)-1;
+
+	btrfs_init_path(&path);
+	ret = btrfs_search_slot(NULL, root, &key, &path, 0, 0);
+	if (!ret)
+		ret = -EIO;
+	if (ret < 0)
+		goto out;
+	ret = btrfs_previous_item(root, &path, start,
+				  BTRFS_BLOCK_GROUP_ITEM_KEY);
+	if (ret) {
+		error("failed to find block group %llu", start);
+		ret = -ENOENT;
+		goto out;
+	}
+
+	eb = path.nodes[0];
+	slot = path.slots[0];
+	btrfs_item_key_to_cpu(eb, &key, slot);
+	if (key.objectid != start) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	total = key.offset;
+	bi = btrfs_item_ptr(eb, slot, struct btrfs_block_group_item);
+	read_extent_buffer(eb, &bg_item, (unsigned long)bi, sizeof(bg_item));
+	used = btrfs_block_group_used(&bg_item);
+
+	/*
+	 * if the free space in the chunk is less than %10 of total,
+	 * or not not enough for CoW once, we think the chunk is almost full.
+	 */
+	min_free = max_t(u64, (BTRFS_MAX_LEVEL + 1) * fs_info->nodesize,
+			 div_factor(total, 1));
+
+	if ((total - used) > min_free)
+		ret = 0;
+	else
+		ret = 1;
+out:
+	btrfs_release_path(&path);
+	return ret;
+}
+
+/*
+ * Returns <0 for error.
+ * Returns 0 for success.
+ */
+static int try_to_force_cow_in_new_chunk(struct btrfs_fs_info *fs_info,
+					u64 old_start, u64 *new_start)
+{
+	int ret;
+
+	if (old_start) {
+		ret = is_chunk_almost_full(fs_info, old_start);
+		if (ret <= 0)
+			return ret;
+	}
+	ret = force_cow_in_new_chunk(fs_info, new_start);
+	return ret;
+}
+
 /*
  * This function only handles BACKREF_MISSING,
  * If corresponding extent item exists, increase the ref, else insert an extent
-- 
2.16.1

[PATCH v4 07/18] btrfs-progs: lowmem check: introduce avoid_extents_overwrite()

[Su Yue]

Another global u64 last_allocated_chunk records the last chunk start
allocated by lowmem repair.
Although global variable is not so graceful, it simplifies codes much.

avoid_extents_overwrite() prefer to allocates new chunk first.
If it failed because of no space or wrong used bytes(fsck-tests/004),
then it try to exclude metadata blocks but costs lots of time in
large filesystem.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index 3649d570e11c..ea4019c32a3f 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -28,6 +28,8 @@
 #include "check/mode-common.h"
 #include "check/mode-lowmem.h"
 
+static u64 last_allocated_chunk;
+
 static int calc_extent_flag(struct btrfs_root *root, struct extent_buffer *eb,
 			    u64 *flags_ret)
 {
@@ -491,6 +493,50 @@ static int try_to_force_cow_in_new_chunk(struct btrfs_fs_info *fs_info,
 	return ret;
 }
 
+static int avoid_extents_overwrite(struct btrfs_fs_info *fs_info)
+{
+	int ret;
+	int mixed = btrfs_fs_incompat(fs_info, MIXED_GROUPS);
+
+	if (fs_info->excluded_extents)
+		return 0;
+
+	if (last_allocated_chunk != (u64)-1) {
+		ret = try_to_force_cow_in_new_chunk(fs_info,
+			last_allocated_chunk, &last_allocated_chunk);
+		if (!ret)
+			goto out;
+		/*
+		 * If failed, do not try to allocate chunk again in
+		 * next call.
+		 * If there is no space left to allocate, try to exclude all
+		 * metadata blocks. Mixed filesystem is unsupported.
+		 */
+		last_allocated_chunk = (u64)-1;
+		if (ret != -ENOSPC || mixed)
+			goto out;
+	}
+
+	printf(
+	"Try to exclude all metadata blcoks and extents, it may be slow\n");
+	ret = exclude_metadata_blocks(fs_info);
+out:
+	if (ret)
+		error("failed to avoid extents overwrite %s", strerror(-ret));
+	return ret;
+}
+
+static int end_avoid_extents_overwrite(struct btrfs_fs_info *fs_info)
+{
+	int ret = 0;
+
+	cleanup_excluded_extents(fs_info);
+	if (last_allocated_chunk)
+		ret = clear_block_groups_full(fs_info,
+					BTRFS_BLOCK_GROUP_METADATA);
+	return ret;
+}
+
 /*
  * This function only handles BACKREF_MISSING,
  * If corresponding extent item exists, increase the ref, else insert an extent
-- 
2.16.1

[PATCH v4 08/18] btrfs-progs: lowmem check: exclude extents if init-extent-tree in lowmem

[Su Yue]

If options '--init-extent-tree' and '--mode=lowmem' are both
input, all metadata blocks will be traversed twice.
First one is done by pin_metadata_blocks() in reinit_extent_tree().
Second one is in check_chunks_and_extents_v2().

Excluding instead of pining metadata blocks before reinit extent tree
in lowmem can save some time.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-common.c | 27 ++++++++++++++++++++-------
 check/mode-common.h |  2 +-
 check/mode-lowmem.c |  8 +++++++-
 cmds-check.c        |  3 ++-
 4 files changed, 30 insertions(+), 10 deletions(-)

diff --git a/check/mode-common.c b/check/mode-common.c
index acceb24b9597..afe5f04d1deb 100644
--- a/check/mode-common.c
+++ b/check/mode-common.c
@@ -706,7 +706,7 @@ out:
  * Using fs and other trees to rebuild extent tree.
  */
 int reinit_extent_tree(struct btrfs_trans_handle *trans,
-		       struct btrfs_fs_info *fs_info)
+		       struct btrfs_fs_info *fs_info, bool pin)
 {
 	u64 start = 0;
 	int ret;
@@ -728,13 +728,26 @@ int reinit_extent_tree(struct btrfs_trans_handle *trans,
 
 	/*
 	 * first we need to walk all of the trees except the extent tree and pin
-	 * down the bytes that are in use so we don't overwrite any existing
-	 * metadata.
+	 * down/exclude the bytes that are in use so we don't overwrite any
+	 * existing metadata.
+	 * If pin, unpin will be done in end of transaction.
+	 * If exclude, cleanup will be done in check_chunks_and_extents_lowmem.
 	 */
-	ret = pin_metadata_blocks(fs_info);
-	if (ret) {
-		fprintf(stderr, "error pinning down used bytes\n");
-		return ret;
+again:
+	if (pin) {
+		ret = pin_metadata_blocks(fs_info);
+		if (ret) {
+			fprintf(stderr, "error pinning down used bytes\n");
+			return ret;
+		}
+	} else {
+		ret = exclude_metadata_blocks(fs_info);
+		if (ret) {
+			fprintf(stderr, "error excluding used bytes\n");
+			printf("try to pin down used bytes\n");
+			pin = true;
+			goto again;
+		}
 	}
 
 	/*
diff --git a/check/mode-common.h b/check/mode-common.h
index e2a824a318c1..8af7dd3066ff 100644
--- a/check/mode-common.h
+++ b/check/mode-common.h
@@ -122,7 +122,7 @@ int check_child_node(struct extent_buffer *parent, int slot,
 void reset_cached_block_groups(struct btrfs_fs_info *fs_info);
 int zero_log_tree(struct btrfs_root *root);
 int reinit_extent_tree(struct btrfs_trans_handle *trans,
-		       struct btrfs_fs_info *fs_info);
+		       struct btrfs_fs_info *fs_info, bool pin);
 int btrfs_fsck_reinit_root(struct btrfs_trans_handle *trans,
 			   struct btrfs_root *root, int overwrite);
 int fill_csum_tree(struct btrfs_trans_handle *trans,
diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index ea4019c32a3f..1e0545e6249d 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -4860,8 +4860,14 @@ next:
 	}
 out:
 
-	/* if repair, update block accounting */
 	if (repair) {
+		ret = end_avoid_extents_overwrite(fs_info);
+		if (ret < 0)
+			ret = FATAL_ERROR;
+		err |= ret;
+
+		reset_cached_block_groups(fs_info);
+		/* update block accounting */
 		ret = btrfs_fix_block_accounting(trans, root);
 		if (ret)
 			err |= ret;
diff --git a/cmds-check.c b/cmds-check.c
index 28746712fac1..ed81fd3c22b4 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -453,7 +453,8 @@ int cmd_check(int argc, char **argv)
 
 		if (init_extent_tree) {
 			printf("Creating a new extent tree\n");
-			ret = reinit_extent_tree(trans, info);
+			ret = reinit_extent_tree(trans, info,
+					 check_mode == CHECK_MODE_ORIGINAL);
 			err |= !!ret;
 			if (ret)
 				goto close_out;
-- 
2.16.1

[PATCH v4 09/18] btrfs-progs: lowmem check: start to remove parameters @trans in lowmem

[Su Yue]

Since extents can be avoid overwrite by excluding or new chunk
allocation. It's unnessesary to do all repairs in one transaction.

This patch removes parameter @trans of repair_extent_data_item().
repair_extent_data_item() calls try_avoid_extents_overwrite()
and starts a transaction by itself.

Note: This patch and next patches cause error in lowmem repair like:
"Error: Commit_root already set when starting transaction".
This error will disappear after removing @trans finished.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index 1e0545e6249d..446ea4a21bfa 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -2739,12 +2739,12 @@ out:
  *
  * Returns error bits after reapir.
  */
-static int repair_extent_data_item(struct btrfs_trans_handle *trans,
-				   struct btrfs_root *root,
+static int repair_extent_data_item(struct btrfs_root *root,
 				   struct btrfs_path *pathp,
 				   struct node_refs *nrefs,
 				   int err)
 {
+	struct btrfs_trans_handle *trans = NULL;
 	struct btrfs_file_extent_item *fi;
 	struct btrfs_key fi_key;
 	struct btrfs_key key;
@@ -2761,6 +2761,7 @@ static int repair_extent_data_item(struct btrfs_trans_handle *trans,
 	u64 file_offset;
 	int generation;
 	int slot;
+	int need_insert = 0;
 	int ret = 0;
 
 	eb = pathp->nodes[0];
@@ -2799,9 +2800,20 @@ static int repair_extent_data_item(struct btrfs_trans_handle *trans,
 		ret = -EIO;
 		goto out;
 	}
+	need_insert = ret;
 
+	ret = avoid_extents_overwrite(root->fs_info);
+	if (ret)
+		goto out;
+	trans = btrfs_start_transaction(root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		trans = NULL;
+		error("fail to start transaction %s", strerror(-ret));
+		goto out;
+	}
 	/* insert an extent item */
-	if (ret > 0) {
+	if (need_insert) {
 		key.objectid = disk_bytenr;
 		key.type = BTRFS_EXTENT_ITEM_KEY;
 		key.offset = num_bytes;
@@ -2841,6 +2853,8 @@ static int repair_extent_data_item(struct btrfs_trans_handle *trans,
 
 	err &= ~BACKREF_MISSING;
 out:
+	if (trans)
+		btrfs_commit_transaction(trans, root);
 	btrfs_release_path(&path);
 	if (ret)
 		error("can't repair root %llu extent data item[%llu %llu]",
@@ -4117,8 +4131,7 @@ again:
 	case BTRFS_EXTENT_DATA_KEY:
 		ret = check_extent_data_item(root, path, nrefs, account_bytes);
 		if (repair && ret)
-			ret = repair_extent_data_item(trans, root, path, nrefs,
-						      ret);
+			ret = repair_extent_data_item(root, path, nrefs, ret);
 		err |= ret;
 		break;
 	case BTRFS_BLOCK_GROUP_ITEM_KEY:
-- 
2.16.1

[PATCH v4 10/18] btrfs-progs: lowmem check: remove parameter @trans of delete_extent_item()

[Su Yue]

This patch removes the parameter @trans of delete_extent_item().
It calls try_avoid_extents_overwrite() and starts a transaction by itself.

Note: This patch and next patches cause error in lowmem repair like:
"Error: Commit_root already set when starting transaction".
This error will disappear after removing @trans finished.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index 446ea4a21bfa..272e658296e7 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -4063,13 +4063,22 @@ out:
 	return err;
 }
 
-static int delete_extent_tree_item(struct btrfs_trans_handle *trans,
-				   struct btrfs_root *root,
+static int delete_extent_tree_item(struct btrfs_root *root,
 				   struct btrfs_path *path)
 {
 	struct btrfs_key key;
+	struct btrfs_trans_handle *trans;
 	int ret = 0;
 
+	ret = avoid_extents_overwrite(root->fs_info);
+	if (ret)
+		return ret;
+	trans = btrfs_start_transaction(root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		error("fail to start transaction %s", strerror(-ret));
+		goto out;
+	}
 	btrfs_item_key_to_cpu(path->nodes[0], &key, path->slots[0]);
 	btrfs_release_path(path);
 	ret = btrfs_search_slot(trans, root, &key, path, -1, 1);
@@ -4087,6 +4096,7 @@ static int delete_extent_tree_item(struct btrfs_trans_handle *trans,
 	else
 		path->slots[0]--;
 out:
+	btrfs_commit_transaction(trans, root);
 	if (ret)
 		error("failed to delete root %llu item[%llu, %u, %llu]",
 		      root->objectid, key.objectid, key.type, key.offset);
@@ -4138,7 +4148,7 @@ again:
 		ret = check_block_group_item(fs_info, eb, slot);
 		if (repair &&
 		    ret & REFERENCER_MISSING)
-			ret = delete_extent_tree_item(trans, root, path);
+			ret = delete_extent_tree_item(root, path);
 		err |= ret;
 		break;
 	case BTRFS_DEV_ITEM_KEY:
@@ -4169,7 +4179,7 @@ again:
 					       key.objectid, -1);
 		if (repair &&
 		    ret & (REFERENCER_MISMATCH | REFERENCER_MISSING))
-			ret = delete_extent_tree_item(trans, root, path);
+			ret = delete_extent_tree_item(root, path);
 		err |= ret;
 		break;
 	case BTRFS_EXTENT_DATA_REF_KEY:
@@ -4182,7 +4192,7 @@ again:
 				btrfs_extent_data_ref_count(eb, dref));
 		if (repair &&
 		    ret & (REFERENCER_MISMATCH | REFERENCER_MISSING))
-			ret = delete_extent_tree_item(trans, root, path);
+			ret = delete_extent_tree_item(root, path);
 		err |= ret;
 		break;
 	case BTRFS_SHARED_BLOCK_REF_KEY:
@@ -4190,7 +4200,7 @@ again:
 						 key.objectid, -1);
 		if (repair &&
 		    ret & (REFERENCER_MISMATCH | REFERENCER_MISSING))
-			ret = delete_extent_tree_item(trans, root, path);
+			ret = delete_extent_tree_item(root, path);
 		err |= ret;
 		break;
 	case BTRFS_SHARED_DATA_REF_KEY:
@@ -4198,7 +4208,7 @@ again:
 						key.objectid);
 		if (repair &&
 		    ret & (REFERENCER_MISMATCH | REFERENCER_MISSING))
-			ret = delete_extent_tree_item(trans, root, path);
+			ret = delete_extent_tree_item(root, path);
 		err |= ret;
 		break;
 	default:
-- 
2.16.1

[PATCH v4 11/18] btrfs-progs: lowmem check: remove parameter @trans of repair_chunk_item()

[Su Yue]

This patch removes parameter @trans of repair_chunk_item().
It calls try_avoid_extents_overwrite() and starts a transaction by
itself.

Note: This patch and next patches cause error in lowmem repair like:
"Error: Commit_root already set when starting transaction".
This error will disappear after removing @trans finished.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 48 ++++++++++++++++++++++++++++++++----------------
 1 file changed, 32 insertions(+), 16 deletions(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index 272e658296e7..53377848f361 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -4026,13 +4026,14 @@ out:
  *
  * Returns error after repair.
  */
-static int repair_chunk_item(struct btrfs_trans_handle *trans,
-			     struct btrfs_root *chunk_root,
+static int repair_chunk_item(struct btrfs_root *chunk_root,
 			     struct btrfs_path *path, int err)
 {
 	struct btrfs_chunk *chunk;
 	struct btrfs_key chunk_key;
 	struct extent_buffer *eb = path->nodes[0];
+	struct btrfs_root *extent_root = chunk_root->fs_info->extent_root;
+	struct btrfs_trans_handle *trans;
 	u64 length;
 	int slot = path->slots[0];
 	u64 type;
@@ -4045,21 +4046,36 @@ static int repair_chunk_item(struct btrfs_trans_handle *trans,
 	type = btrfs_chunk_type(path->nodes[0], chunk);
 	length = btrfs_chunk_length(eb, chunk);
 
-	if (err & REFERENCER_MISSING) {
-		ret = btrfs_make_block_group(trans, chunk_root->fs_info, 0,
-					     type, chunk_key.offset, length);
-		if (ret) {
-			error("fail to add block group item[%llu %llu]",
-			      chunk_key.offset, length);
-			goto out;
-		} else {
-			err &= ~REFERENCER_MISSING;
-			printf("Added block group item[%llu %llu]\n",
-			       chunk_key.offset, length);
-		}
+	/* now repair only adds block group */
+	if ((err & REFERENCER_MISSING) == 0)
+		return err;
+
+	ret = avoid_extents_overwrite(chunk_root->fs_info);
+	if (ret)
+		return ret;
+
+	trans = btrfs_start_transaction(extent_root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		error("fail to start transaction %s", strerror(-ret));
+		return ret;
 	}
 
-out:
+	ret = btrfs_make_block_group(trans, chunk_root->fs_info, 0, type,
+				     chunk_key.offset, length);
+	if (ret) {
+		error("fail to add block group item[%llu %llu]",
+		      chunk_key.offset, length);
+	} else {
+		err &= ~REFERENCER_MISSING;
+		printf("Added block group item[%llu %llu]\n", chunk_key.offset,
+		       length);
+	}
+
+	btrfs_commit_transaction(trans, extent_root);
+	if (ret)
+		error("fail to repair item(s) related to chunk item[%llu %llu]",
+		      chunk_key.objectid, chunk_key.offset);
 	return err;
 }
 
@@ -4158,7 +4174,7 @@ again:
 	case BTRFS_CHUNK_ITEM_KEY:
 		ret = check_chunk_item(fs_info, eb, slot);
 		if (repair && ret)
-			ret = repair_chunk_item(trans, root, path, ret);
+			ret = repair_chunk_item(root, path, ret);
 		err |= ret;
 		break;
 	case BTRFS_DEV_EXTENT_KEY:
-- 
2.16.1

[PATCH v4 12/18] btrfs-progs: lowmem check: remove parameter @trans of repair_extent_item()

[Su Yue]

This patch removes parameter @trans of repair_extent_item().
It calls try_avoid_extents_overwrite() and starts a transaction by
itself.

Note: This patch and next patches cause error in lowmem repair like:
"Error: Commit_root already set when starting transaction".
This error will disappear after removing @trans finished.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 54 +++++++++++++++++++++++++++++++++--------------------
 1 file changed, 34 insertions(+), 20 deletions(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index 53377848f361..443fa513a13e 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -3588,40 +3588,55 @@ out:
  *               means error after repair
  * Returns  0   nothing happened
  */
-static int repair_extent_item(struct btrfs_trans_handle *trans,
-		      struct btrfs_root *root, struct btrfs_path *path,
+static int repair_extent_item(struct btrfs_root *root, struct btrfs_path *path,
 		      u64 bytenr, u64 num_bytes, u64 parent, u64 root_objectid,
 		      u64 owner, u64 offset, int err)
 {
+	struct btrfs_trans_handle *trans;
+	struct btrfs_root *extent_root = root->fs_info->extent_root;
 	struct btrfs_key old_key;
 	int freed = 0;
 	int ret;
 
 	btrfs_item_key_to_cpu(path->nodes[0], &old_key, path->slots[0]);
 
-	if (err & (REFERENCER_MISSING | REFERENCER_MISMATCH)) {
-		/* delete the backref */
-		ret = btrfs_free_extent(trans, root->fs_info->fs_root, bytenr,
-			  num_bytes, parent, root_objectid, owner, offset);
-		if (!ret) {
-			freed = 1;
-			err &= ~REFERENCER_MISSING;
-			printf("Delete backref in extent [%llu %llu]\n",
-			       bytenr, num_bytes);
-		} else {
-			error("fail to delete backref in extent [%llu %llu]",
-			       bytenr, num_bytes);
-		}
+	if ((err & (REFERENCER_MISSING | REFERENCER_MISMATCH)) == 0)
+		return err;
+
+	ret = avoid_extents_overwrite(root->fs_info);
+	if (ret)
+		return err;
+
+	trans = btrfs_start_transaction(extent_root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		error("fail to start transaction %s", strerror(-ret));
+		/* nothing happened */
+		ret = 0;
+		goto out;
 	}
+	/* delete the backref */
+	ret = btrfs_free_extent(trans, root->fs_info->fs_root, bytenr,
+			num_bytes, parent, root_objectid, owner, offset);
+	if (!ret) {
+		freed = 1;
+		err &= ~REFERENCER_MISSING;
+		printf("Delete backref in extent [%llu %llu]\n",
+		       bytenr, num_bytes);
+	} else {
+		error("fail to delete backref in extent [%llu %llu]",
+		      bytenr, num_bytes);
+	}
+	btrfs_commit_transaction(trans, extent_root);
 
 	/* btrfs_free_extent may delete the extent */
 	btrfs_release_path(path);
 	ret = btrfs_search_slot(NULL, root, &old_key, path, 0, 0);
-
 	if (ret)
 		ret = -ENOENT;
 	else if (freed)
 		ret = err;
+out:
 	return ret;
 }
 
@@ -3631,8 +3646,7 @@ static int repair_extent_item(struct btrfs_trans_handle *trans,
  *
  * Since we don't use extent_record anymore, introduce new error bit
  */
-static int check_extent_item(struct btrfs_trans_handle *trans,
-			     struct btrfs_fs_info *fs_info,
+static int check_extent_item(struct btrfs_fs_info *fs_info,
 			     struct btrfs_path *path)
 {
 	struct btrfs_extent_item *ei;
@@ -3763,7 +3777,7 @@ next:
 	}
 
 	if (err && repair) {
-		ret = repair_extent_item(trans, fs_info->extent_root, path,
+		ret = repair_extent_item(fs_info->extent_root, path,
 			 key.objectid, num_bytes, parent, root_objectid,
 			 owner, owner_offset, ret);
 		if (ret < 0)
@@ -4183,7 +4197,7 @@ again:
 		break;
 	case BTRFS_EXTENT_ITEM_KEY:
 	case BTRFS_METADATA_ITEM_KEY:
-		ret = check_extent_item(trans, fs_info, path);
+		ret = check_extent_item(fs_info, path);
 		err |= ret;
 		break;
 	case BTRFS_EXTENT_CSUM_KEY:
-- 
2.16.1

[PATCH v4 13/18] btrfs-progs: lowmem check: remove parameter @trans of check_leaf_items()

[Su Yue]

This patch removes parameter @trans of check_leaf_items().

Note: This patch and next patches cause error in lowmem repair like:
"Error: Commit_root already set when starting transaction".
This error will disappear after removing @trans finished.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index 443fa513a13e..a7660a25b844 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -4139,8 +4139,7 @@ out:
 /*
  * Main entry function to check known items and update related accounting info
  */
-static int check_leaf_items(struct btrfs_trans_handle *trans,
-			    struct btrfs_root *root, struct btrfs_path *path,
+static int check_leaf_items(struct btrfs_root *root, struct btrfs_path *path,
 			    struct node_refs *nrefs, int account_bytes)
 {
 	struct btrfs_fs_info *fs_info = root->fs_info;
@@ -4336,7 +4335,7 @@ static int walk_down_tree(struct btrfs_trans_handle *trans,
 				ret = process_one_leaf(root, path, nrefs,
 						       level, ext_ref);
 			else
-				ret = check_leaf_items(trans, root, path,
+				ret = check_leaf_items(root, path,
 					       nrefs, account_file_data);
 			err |= ret;
 			break;
-- 
2.16.1

[PATCH v4 14/18] btrfs-progs: lowmem check: remove parameter @trans of repair_tree_back_ref()

[Su Yue]

This patch removes parameter @trans of repair_tree_back_ref().
It calls try_avoid_extents_overwrite() and starts a transaction by
itself.

Note: This patch and next patches cause error in lowmem repair like:
"Error: Commit_root already set when starting transaction".
This error will disappear after removing @trans finished.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index a7660a25b844..d4c8de4e69af 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -544,11 +544,11 @@ static int end_avoid_extents_overwrite(struct btrfs_fs_info *fs_info)
  *
  * Returns error bits after repair.
  */
-static int repair_tree_block_ref(struct btrfs_trans_handle *trans,
-				 struct btrfs_root *root,
+static int repair_tree_block_ref(struct btrfs_root *root,
 				 struct extent_buffer *node,
 				 struct node_refs *nrefs, int level, int err)
 {
+	struct btrfs_trans_handle *trans = NULL;
 	struct btrfs_fs_info *fs_info = root->fs_info;
 	struct btrfs_root *extent_root = fs_info->extent_root;
 	struct btrfs_path path;
@@ -598,6 +598,16 @@ static int repair_tree_block_ref(struct btrfs_trans_handle *trans,
 	if (nrefs->full_backref[level] != 0)
 		flags |= BTRFS_BLOCK_FLAG_FULL_BACKREF;
 
+	ret = avoid_extents_overwrite(root->fs_info);
+	if (ret)
+		goto out;
+	trans = btrfs_start_transaction(extent_root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		trans = NULL;
+		error("fail to start transaction %s", strerror(-ret));
+		goto out;
+	}
 	/* insert an extent item */
 	if (insert_extent) {
 		struct btrfs_disk_key copy_key;
@@ -663,6 +673,8 @@ static int repair_tree_block_ref(struct btrfs_trans_handle *trans,
 
 	nrefs->refs[level]++;
 out:
+	if (trans)
+		btrfs_commit_transaction(trans, extent_root);
 	btrfs_release_path(&path);
 	if (ret) {
 		error(
@@ -4304,7 +4316,7 @@ static int walk_down_tree(struct btrfs_trans_handle *trans,
 			   btrfs_header_owner(cur), nrefs);
 
 			if (repair && ret)
-				ret = repair_tree_block_ref(trans, root,
+				ret = repair_tree_block_ref(root,
 				    path->nodes[*level], nrefs, *level, ret);
 			err |= ret;
 
-- 
2.16.1

[PATCH v4 15/18] btrfs-progs: lowmem check: remove parameter @trans of check_btrfs_root()

[Su Yue]

Remove parameters @trans of delete_extent_item() and walk_down_tree_v2().

Note: This patch and next patches cause error in lowmem repair like:
"Error: Commit_root already set when starting transaction".
This error will disappear after removing @trans finished.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index d4c8de4e69af..d92278d2993c 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -4271,8 +4271,7 @@ out:
  * Returns <0  Fatal error, must exit the whole check
  * Returns 0   No errors found
  */
-static int walk_down_tree(struct btrfs_trans_handle *trans,
-			  struct btrfs_root *root, struct btrfs_path *path,
+static int walk_down_tree(struct btrfs_root *root, struct btrfs_path *path,
 			  int *level, struct node_refs *nrefs, int ext_ref,
 			  int check_all)
 {
@@ -4585,8 +4584,7 @@ out:
  * Returns 0      represents OK.
  * Returns >0     represents error bits.
  */
-static int check_btrfs_root(struct btrfs_trans_handle *trans,
-			    struct btrfs_root *root, unsigned int ext_ref,
+static int check_btrfs_root(struct btrfs_root *root, unsigned int ext_ref,
 			    int check_all)
 {
 	struct btrfs_path path;
@@ -4631,7 +4629,7 @@ static int check_btrfs_root(struct btrfs_trans_handle *trans,
 	}
 
 	while (1) {
-		ret = walk_down_tree(trans, root, &path, &level, &nrefs,
+		ret = walk_down_tree(root, &path, &level, &nrefs,
 				     ext_ref, check_all);
 
 		if (ret > 0)
@@ -4667,7 +4665,7 @@ out:
 static int check_fs_root(struct btrfs_root *root, unsigned int ext_ref)
 {
 	reset_cached_block_groups(root->fs_info);
-	return check_btrfs_root(NULL, root, ext_ref, 0);
+	return check_btrfs_root(root, ext_ref, 0);
 }
 
 /*
@@ -4871,11 +4869,11 @@ int check_chunks_and_extents_lowmem(struct btrfs_fs_info *fs_info)
 	}
 
 	root1 = root->fs_info->chunk_root;
-	ret = check_btrfs_root(trans, root1, 0, 1);
+	ret = check_btrfs_root(root1, 0, 1);
 	err |= ret;
 
 	root1 = root->fs_info->tree_root;
-	ret = check_btrfs_root(trans, root1, 0, 1);
+	ret = check_btrfs_root(root1, 0, 1);
 	err |= ret;
 
 	btrfs_init_path(&path);
@@ -4906,7 +4904,7 @@ int check_chunks_and_extents_lowmem(struct btrfs_fs_info *fs_info)
 			goto next;
 		}
 
-		ret = check_btrfs_root(trans, cur_root, 0, 1);
+		ret = check_btrfs_root(cur_root, 0, 1);
 		err |= ret;
 
 		if (key.objectid == BTRFS_TREE_RELOC_OBJECTID)
-- 
2.16.1

[PATCH v4 16/18] btrfs-progs: lowmem check: introduce repair_block_accounting()

[Su Yue]

Introduce repair_block_accounting() which calls
btrfs_fix_block_accounting() to repair block group accouting.

Replace btrfs_fix_block_accounting() with the new function.

Note: This patch and next patches cause error in lowmem repair like:
"Error: Commit_root already set when starting transaction".
This error will disappear after removing @trans finished.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index d92278d2993c..40a179f75319 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -537,6 +537,30 @@ static int end_avoid_extents_overwrite(struct btrfs_fs_info *fs_info)
 	return ret;
 }
 
+/*
+ * Wrapper function for btrfs_fix_block_accounting().
+ *
+ * Returns 0     on success.
+ * Returns != 0  on error.
+ */
+static int repair_block_accounting(struct btrfs_fs_info *fs_info)
+{
+	struct btrfs_trans_handle *trans = NULL;
+	struct btrfs_root *root = fs_info->extent_root;
+	int ret;
+
+	trans = btrfs_start_transaction(root, 1);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		error("fail to start transaction %s", strerror(-ret));
+		return ret;
+	}
+
+	ret = btrfs_fix_block_accounting(trans, root);
+	btrfs_commit_transaction(trans, root);
+	return ret;
+}
+
 /*
  * This function only handles BACKREF_MISSING,
  * If corresponding extent item exists, increase the ref, else insert an extent
@@ -4930,7 +4954,7 @@ out:
 
 		reset_cached_block_groups(fs_info);
 		/* update block accounting */
-		ret = btrfs_fix_block_accounting(trans, root);
+		ret = repair_block_accounting(fs_info);
 		if (ret)
 			err |= ret;
 		else
-- 
2.16.1

[PATCH v4 17/18] btrfs-progs: lowmem check: end of removing parameters @trans in lowmem

[Su Yue]

Remove @trans in check_chunks_and_extents().

This patch let lowmem repair work again.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 check/mode-lowmem.c | 13 -------------
 1 file changed, 13 deletions(-)

diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c
index 40a179f75319..4aad69fc9eb1 100644
--- a/check/mode-lowmem.c
+++ b/check/mode-lowmem.c
@@ -4872,7 +4872,6 @@ out:
  */
 int check_chunks_and_extents_lowmem(struct btrfs_fs_info *fs_info)
 {
-	struct btrfs_trans_handle *trans = NULL;
 	struct btrfs_path path;
 	struct btrfs_key old_key;
 	struct btrfs_key key;
@@ -4884,14 +4883,6 @@ int check_chunks_and_extents_lowmem(struct btrfs_fs_info *fs_info)
 
 	root = fs_info->fs_root;
 
-	if (repair) {
-		trans = btrfs_start_transaction(fs_info->extent_root, 1);
-		if (IS_ERR(trans)) {
-			error("failed to start transaction before check");
-			return PTR_ERR(trans);
-		}
-	}
-
 	root1 = root->fs_info->chunk_root;
 	ret = check_btrfs_root(root1, 0, 1);
 	err |= ret;
@@ -4961,10 +4952,6 @@ out:
 			err &= ~BG_ACCOUNTING_ERROR;
 	}
 
-	if (trans)
-		btrfs_commit_transaction(trans, root->fs_info->extent_root);
-
 	btrfs_release_path(&path);
-
 	return err;
 }
-- 
2.16.1

[PATCH v4 18/18] btrfs-progs: fsck-tests: add image no extent with normal device size

[Su Yue]

This new image only misses one extent which leads lowmem mode to
allocate new chunk in repair.
Original image renamed to no_extent_bad_dev.img should let lowmem mode
exclude blocks in repair.

Due to problems of btrfs-image, choose xz as compression tool.

Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
---
 tests/fsck-tests/014-no-extent-info/.lowmem_repairable  |   0
 tests/fsck-tests/014-no-extent-info/no_extent.raw.xz    | Bin 0 -> 28084 bytes
 .../{default_case.img => no_extent_bad_dev.img}         | Bin
 3 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 tests/fsck-tests/014-no-extent-info/.lowmem_repairable
 create mode 100644 tests/fsck-tests/014-no-extent-info/no_extent.raw.xz
 rename tests/fsck-tests/014-no-extent-info/{default_case.img => no_extent_bad_dev.img} (100%)

diff --git a/tests/fsck-tests/014-no-extent-info/.lowmem_repairable b/tests/fsck-tests/014-no-extent-info/.lowmem_repairable
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tests/fsck-tests/014-no-extent-info/no_extent.raw.xz b/tests/fsck-tests/014-no-extent-info/no_extent.raw.xz
new file mode 100644
index 0000000000000000000000000000000000000000..6e568a9cf1f0a1d1bcd00222b07cf14d3c09afc5
GIT binary patch
literal 28084
zcmeHwRdglUlAV~Dr4lnUGcz+Yl&Zw65;HTGm`hY*W@ct)W@gDf9*;jfYmHxzU(agW
z_wTOzlbMlmc0}y6&(3#_ADY@gKwt+8b>bjEM8LQ}KtM>7nq!}zeqgG4KtQ(dpP%`S
zpA!6%=nh;)N=@;U2l>H}&lh|6{<94I+`Pat)!xJ5;c^W5@XJ0$yOAWwDqC7u{m$V#
zu1^h#K5*Z<Hk_nPb#uyk@TWA-lsn%){ls!d^sIcP9gZ~zYnsr4q}ybl#F1lDvq#bO
z^9}<s#2LmQ!0el=gtT=&MCBgf;SOx~fVQ@Yp<J~hzxKcB5FvtNi5;etJXt{2&I^0{
z*zT?3JVd<eSars9=XOo3#|dyP1U}c*jujsjEh;JYCg9Fx8nX}akjLp3VFqTGF=&Wg
zhffr`8_?IsI`AUuBKg`o7;>_dr^8P5T*g?2u6pL-W6*Z@NVn9#3fEQ(6T&7fx32XO
zWHO>g#(kt{ScOiSKvXh&evRV3EMjeu4_;Plq;WL7Iu{J`?+x@6^!FpdHHm&<;)*WY
z8Rn^5bFn|!@yXFcCT5$omu(i5<CJ%yf|T+JI5x-#GJ9RL5lapcXVL!V`}zR)3%B=G
z8TruK9=z=UA*4>h&Tm2nXCtaA4Ciq!juG8#5Bf@(Wh;hXy3Jb%gsIt8E1kIoVm*L*
zitD=5-f$&(3aIQmZ34KrcEr%@!=(sJn2K1QUYwkk3q;%dQeDuxfR>hmL!Y>Wc`2e6
zZ>)H>s!r;AO0@Hsz$r<5p@s~;wA!0tS=04;ka*1Xw4Fd>Yxhc~Ht>!p8dc(Yh$Ht8
z80Al_c*_u!60JwnsT_O(B=Q5a0Wd*XFuO|&wHJ)9l;UrS&a{lHJ&LMv#3>M<2E%qd
zXkE-4sfFmkg-7|&)Muri+T`U9a2WB$9b7S_+jd#SBnD@cqe0V2so~z#aW@pskJw!h
zsdB0BOO^Anh<#)#Pxo=`OyGr87q;DYauRUUamb|=hIlO#)f8B|TEWNW%9<U=E`_|f
zaYvlr)EXLuG_eBcsN&vwI&k@dFo?2Rk|=Rg=UYC0Vi&x^5%*J!@qe)!W;9(Nd^SBF
zwQDXY2$sA=c_#4;Lfn%+%9`f&l#JWy2vyZ*j-<MzEzraGacMwS-f?I+hvh5O@5qvd
z?;yQuA|C`47N+=yL(=s7h{+f}vf?5LvdN60<NWT<O+4}WH4k`I4VgT9Y}~+3OR;EV
z0ctA^F1vbsnS+38;cl0faKP&QaPdw!diPeEU#2A=h+`uW{L*uzr6tNjkY=Daxn4Me
z(-sMXnJO)g-jB-|ZUX{!iz?jfLcyV0dA|@bi<Gj(C!d!vc>R%3L9t(2S230fHD+v>
zhH71<bV#?(O7nL!8|@W79`1p1DLm4r5zjY$Lo^31ZffuQR*HL{svpicQ=|a~hfn?=
zX_6zHY<ZsZdZ)p{sC*VVqE8%jhU=#eHGAR@s#l@^?6>*&@SdFgi3CgwbmpkuGEE=E
zi!Hg|vm2gyo{&3l9X}VnV!kbZ!j~q6L6h_>@gjeyyb}+hj-4f%wKMbA+|fUjU*%eR
z`#RuI&x0!!Z?DQUPUHAGgu>73Dtu2=rqDdTqtsTL^*mR7?eqqvi2O3<!m+3bpGXN$
z_HJ5Fu;ZUel;VgbZ=M^kKN;!OyJCWF1N^$o{?XJfNj&Dt{!9e>0mNU#0v;n-v^qH7
z`M_I0<<R5n7)ri^E2*3aevvNjET70MJ^Do>%8j;o=(6zmEH>x^I1sG&^xH!0<W8p6
z)Tn}}XXm1?+iw)Xvj7^#**IFfK&j_qA0rIEE{zairJI^NkL2p{w0&Fau;!|IOZ9Dd
za?vKI?g3SGzJ^+|;;w!Karbtvmgv<O*>Z6)@&^Iqvy+y@?cj1Sh)>gcHaub@=$)I7
zyYtf3dwWgOogeh@h%TO3eZ@ojl(}<*A#ESS=zZ(<HyIP%EOC{WIUy)fJ=t5?ORL?u
zt2m-_W`67n_1yKxn_MI$um++K@gaA7bhC^@zN11v1wGHGTJc1Qb^--=`|XstUPa1Q
zobahq(R{)$Jq%^0(s+@J8}GNj^~H#ZH$wvFdU2b<h&|yhAEwm{w=3f&oc1~Accfu;
zH7w_zG`w4I`eD1N6tkQwPVn@e>IfNvKp-gsag^7kueFatlh(D=etfmHOW;?~)a>Id
zXEcc1;_~Hs%w8LII?HV#rFoLD{cTxx=}Z54asuOPF|&K0?9LDKZdcf2cK91KE6Bpz
z9Foopcj)N!2m+|ao0M1vp7xi?2#5FUQS8xmp*-8-!Y5Jnv}B}mGrj0RDr2#u>h-GO
zvvunwE1lMH9lXzW5>+R3irInLbo7W_l;Xn}OGG?L$bq6fHQ9X+t&=Ow!-5-dyX}VI
zR-PdpGFFd7N1`GzWDvd%bm_Xwx7W`ldnk{kcu@rh7{ef}PNR2!G}dZI_!GX*@K=A%
z-=jIggtI{HW-h4IHck1<g`9bk-Vd$?WILYCUtiEczpI(2%W3AOCpfj$m<I&OHBqdA
z`PA{!3P{)IVN$`1L>_nLll`7M_-eZ8Hp-{BdO!A57bpBox@ylo7svCWJ%n6v^UK#m
z;K7$CC)AcsH__zAScFVjg$K%3MqDz<MD0CHD@F|WE-uIpq&H=-^@SleYo(aX)Y2-Z
zJ-dnBAJnU<Xh;Mx>8T0$WmJOT%T_JF+!uq1U^7g5oHEf5bA`<*Q}>&a26T3Ak_dQ1
z#G(ZT&q>YsOx)Ot%%s7C3e@i!c(d^-`gwNDU355*jO!NeUTdJ~gT$^f{6oc1<n&FG
zYh=CW=sLAvA@wipm-|*5v>?**4vF3?7HS!5fzIne)=`NkGMl1xN|ZtXK&sGy3pf8a
z#8xSO@A&P>gp=JE#boo(s@?xtZ1q6jpxksBhO7CzRR%mELzsVw7|kCw?r#;be?m3-
z&%Uhx@X|_%0yThu{fW*25U@YlY5@WU2-yD|76BOcCr6OKFkno`WkBsMLD5vF=tfja
z8ZkO##<-MiCI(YJcR(>;(3c)f^=o>uTp%ocGDOTP(Q)S1e2Iv<TR69f)P0>&x)>%f
zw$i0z`x&lS#4#f)2(t~iYHxcx7Cdd0&{lxU8MH7!h_H@-v$dq0KGNq{H-)|oyTCwV
z_i(xb*&0Mil;?sw$`(TK+a@D!d(_xLdWUy2h{x!5tb@wf!5T1LCwM~~B`N;^ooRYs
zKsA=kHRp32@jwyLzg+}NA%-Mcs2T=ANK{KOGxsKaaw|b39qcj45q&1*jqYUV#QP!W
z<Yj4Z%syynIyakhiGWa5>~FIU^9;U=f9lU%UB?=D2>n|XZK|m#z0>TH5-oUMisp5#
zx~lld4MP3u7;WL_1#e2ZY5)ViaH9?niYI|1%3#Ly<L+w>h=8W89F$2F{~a&p?p)Dt
zCwn<D!J3Ysfq_Rg3k=u&zW2!bz>L!gE#_3)g`}xB!|SKF+GP3@8XjMg5Wn=Iohnq&
zoqBCKN^|&ZRga`Oa&Q?cszBo+(9{J&q(wb_&pa~iS*%ZbFcyk2j5%l*+Efr%{$c8<
zP1gb@lm0aG>>*INdYYv@)1qm22nH6&R`U3;2)v#sLKCKP3mS3(7^3fsl!}mIh|`?@
zr9I~XP)`i2v7r(dqQ>*U?Jzd^+yX;Ql-cC9LOp29Y|;a(;_^qk0qFgX&__@m`W$R%
z#Qrw|Cb#(<p7)q#BAlXJ(eclxo}oljXqrGc*V)saQ^Ko9#U(_g5H)?zH36pcI&Sbj
z$|_#fBrYIb6jbpLC)-vC&`q=gaI1W`vJ<h3M=4lZ0X%`I+@6>crn3=GXWMDVOm`$;
zPbJBBmi4`aQM?vYKF}Gg)QW24tcOs4<W*xCD1m}4ljpfonvZf|u0nHdi(yvJkCB2J
zu+_2t!}UUcx*E36y4ZYgOX`$by6ACYBGg`^a=X@+?^1KJM6km-#wEn&H5Ido$H`qc
zIEkEO<jjwU+>Y&8uLo%G7WSZQjLa|mp0b=G0ouMb#QIQqc1Z~+Nl`c7>)!gDT;XGL
zeK6l@RoE_!R0&*yd#}aT<OE!3$cmi3F40chTlw#h*AtX6Bn6%yH`E1Yi1ar>s>zbB
za~DBI#y73*x2|otsWz6tUu)3%yY65&*X3}6<EEnZ3;bxuR@Rlvd+o3Jqmbt`r@zli
z^+Lb6YCH0(+Er71zY8+Nd@M66o`>9R#Zovoq#QND^lzy#WXu%OBQ8M_`^mG<z{n_L
zR?h%FBaT!e7O{m^XAW<MmVC28k)HTAib+1gwQ~Wc&!K{fD7}wmdex&|!vJSeeIg1^
zz3`h*Yd{Oypxrd5^Cqj)yUCq80&7Bwgzp0GyL7IEJ+gmJsX}zZLV=FY)dgglNx@Z~
z4NVQ&a4uw346|+BS6gadUVmllTDz$#pt_v&8fPXO!NmM8Kd9AC?o?+if&4fnY7KW-
z{EwPoemg0ApV3O4fuiPf3bze$93XTXI^w*)2=47i(TUSw{i;rsglb|lo&kERD{s7B
z0=}dV^^8$&P8{5A$ug@K{~(->_(~Oe<BmMN3a)aO@`4i~;ZL!+4GlL<7D-*8guclz
zU(9EtfwOr-Mxo~XG}_qttm8N2Z=~+Lts6ODfvL}I9$+$0)9MI?II(Jz|A6E@Xnm0n
zqd=v#CviY?{S744!zI7E{x(&9A^y&leWP$a83S*mGKnVqNt7#LHFMGVUS<1m34>r6
zuN*r}ogoK|w-^Oz7G{}Wce0yuw2AOYd`vrqO`wTpGRf-Jz}-DWj(U$V>Hr*d@gx<c
zQQcg}s-&*8Y|BYI9fU`0jEt)V@&`4=TnrEXIhrfEYvZ_nKX}wU_X&!6^HLbDWWrm5
zC(_U};(Do<4X_}^B!8uCWm&9kyWI8l*>nzG9`v;rQfNF2^v~li9%pBbv5LE7X)+#P
z7Q6^Hsf03QEf~DYr?=q);I_SwI}+jnc4|_*3GK{KEsu|H(9s`5+R;$bQjy@K)e~Gw
zlZolO-_U{*ejy_l`fTCwL@#f>Ri$OP0m*r|3Sr60`!u4Dowsg^WTjNJ^>5N*tck5a
zAnz*<`q0!JD<xIPKep&4p>I>0U03CBCDNKYGu7wkx)rOe&K;I<ba*_m%{b{|h!9Pv
zhEfL^81Lk&$MUkOjf2y^sI@g?U!C0Cj~wOXvcTc(+_7<^lm!Vk;dAUq$GYLQQA2B^
zDa}y2Puq=MPuw|%&yY7an@`7@&u>gq;I&M}ILE-#_cKz>Y}0Uu0V&`eW)os4!L`j0
z2+7GMQ4DTX#7aj7in$f@uB@~gg$1_ppa+8bJD+9hw~HyN;tH<PwR9T@-BsFVcWh|`
zpM@^f6;hft)>&KkkSqF-yoyuK)bUdS;Uogf86t;2h!}HVrF(IbiNeN&+B?48?Ic%L
zaWXm~wCO50w`J^uP=IB6(8VL)38`5o*V$5&Lm#H>AM&6&cs4y0;N__P+7}1&t@$E}
z2x+2kr--+a97LE!A-cLWyTQE0^@ziLVl>Y7$9t|ka<T;XNhx^iLP4;b(Erk_5}rhu
zPYu4XEYMT0B3Wb7fk>l+i@T!11%=U2=|Q54Gz+F9V#z>ly~Ux0RsSk8X8jag&HIyR
zM?(fBZN1j!Sp5AXc#%quZ%nWw#NH$X2_J+YZv)2a9KGv)tN7ziZ1Gdk*jhr#MHaj;
zaS4&W#0SON?q<?9vWNlgS?FjPX(6U$*zIz12&C=g)yOXdNFtth<hNbszKAVsD!6h|
zy4g25X|8pyjw=-h3NNZ$;zZ|UeN{S8VXC{E<vm9Ooh4f_j#hP;fj7xzw#WUsD1>M}
z3CU$m|5*Kc{(DGEbk6=XmejPILm>y=Zl-o@`wL;V%(OB0Y5}MR-s@27_}<jMGnPx5
zsp}Gt!;Sr3KKo9q%JRVy`I4!<zSn2>;DM*Qxsirj{Wax$ANt*MupERKjFAjDLKaDS
z%}<iY_;>D7F|aG`ByGrv4%G-f&_nEv!Jg`4K2d+8bDfoY4DvzxUut>qG&0zj6^h6S
zE*X<z^(Qi}TFz;8pxabz+b$Uj?fAN5hC90MRa(nkO!i7+r~(&!N+g#}2=w<KSM)U0
z4q3MXp>T^2k22Fbu1d`{>K|V97j5creQilEs^fS%%r{K*HGaXA5gQv;8iEhhJSt0X
zcgcpP{oC59-fCui<SKJj-bz1kF8E80(z@aY#t!x0x>o758b5Ji{gC!blX&mlUsl8d
z{bhy&_~u6k$UReL=CPOGlbhOg%=>r+S0kWLLX>|%4#W{5>}(NhXkb?ue^#($>FJ?<
zE1M73PcP!z12Ko7^m%ATF?J0@WX{A;wPQzWA)LB)Q)>>|`Yv#|EIXwNqet6-GaDoH
z5~Xd>Ki_(Q>=wGhhRS2M9<+W6($_sS2mdx*P)}ax`Jh6aWfk%*M~mcQ&e01KGBdz;
z4(8F+m~ZZiv)Zy%M4F!d*)`7`%qD=G{i{!|=f~Z9Y`CTEvagYsU#k`V4u55cJ%_)v
ztRg#ElyP50rcEL3*QPb!_h1fcx`9zI18zA?2{@fkd!Bnb!?tQO`$$d~96|Ca*N!i(
zbswAGDE3y-#rZi;5cpP2;i0nZ3tM7%Jlza|EXNyDoG{C&-zRf$T`2CS4P**=-#v2V
z`Xn!9Uh}+Pf1To|%d`XteqswhW~P6_Gk<zjG$oQs*d3uCry}sl?Q^JHQl(7oG;O{l
zg0PZ>+@IjLYbShn`|57~X?}>Gp~FJU9~IPLK{U3dPY`LAhm^Une_ZHrV(Wh?0b+Q!
zM%PQn+Sno1ZB$h&&K+fbSDLlVyj(S~++Jgv)^a$sFXHwi(7uop9g!mP)$UrB{TMZd
zfy$G{cNpaC`1H3)C2<eF&~T`HPy{$GVNh0yMakk9_iRl`FpZPTCRi;-CtpfTK_qWD
zEcb?}ZT^pgY5bgyWb9>xvEVq8E{DrsUhFJM$suHjQC8Wyrn&1du9euW0Z>it>6<m6
zP#S0GT~j27By7odzQyhP`kT2>RmkXeg~&W*O7q)Ni*!7U1xQ&D?#)P`2?>MXP2nQJ
zA8O3-5ZJxJv$S(ZmDP?F@2lTV6rHM;f4C$Gzl@NYxlP}-Z}n4E`)MI=ek7xWV;T9;
z@N-i9EK}P=f-E#^@tX0*c~%bd{h76n^z&nSJNd(DxKImpx{7Z;vw-9NlZ9gk)Hww@
z?`sfFOv(uS?mgn`mlbJARmk}_j|ss8cVEgCO#N55C39pZss^LY&qbohT!S^MMHuV;
z%P87YHNNyIli`?<-QYvBS?8g;X>|Iv=naPDlo4<z(z-eeDtTQol}nzlk+KPc_wiFV
zNR0Pi!$u0;%aief6a<m$4iFmY_g7(ic>PE_Z5^5>28L;!4_3V_=g=}6#s%o`_4J-B
zf*+_xMrp9Qqw-Z#ku%Oyx%yE<14AdDEd_gl)i27rtg^nCsv{|zsH!;4MD^<KH_m=6
zn&47|oOO0GCpv||S*Y*H(=XUEWUxubpqFeN(<M=HIyOAO!vC5sf$8qK&2C749T;a-
zaj4IVU9o2g0(ITD$qe=3YFIU1_w5ymV)|GU)>N*)F@uVCT}@4M)fft6`szmf3k3h5
zHJ{V-suC;2wVrcjk|(ECwQ1GM@4jyw_RTSSYBaJT8t8W-=N=y)?uLDAe`k(+w9zq$
zB6+A){fl{^bEK__JEGq0&T+q)3<<e(&6{GkR+$w1{Szn>nJZha+_ICPG5CGB#WcE#
z)H|}m*v_){)z|ROVOSG~uQX`h3ev&CHoaYQON4QPCPRAInI<IF!n!r26#i{_-Y`-W
zJq<vJ@?*B<v1-RIJq?UF@a*6gCy<5f^-u&flR~p^Qo6ls6hxb9h$O_>7hWh=bKZ5-
zkTCn&$cu85Sx2KCo$a}aA!-RYYhgT07QtDupRn~<cgiJ1`W|2)(e&ltEO@mtbFOpg
z+2N-JM@=e%mCNr^%&9Cj7a9Zx&momX+3vyo<*gKvdh{?t{UrmA_8i$i@$ED7de+DT
z<g<96H-aD=a}bLTj-q@dckVaX;cAO2Cd?Y6tiH(UA0Sq&n{-hXh{etmx7~-9JLZfF
zrT^B^u_n-x;&82M1b3{4u1fPZV1@pw(Zb9|M_R7Wn1Et+V2t$4X-UeQR{OiveQGOP
zSkvPI6V=<hvJu;|cpRM?)M_<`|MI6t<dFCQb)!PTck~F3{>7%wFsQZ0xy1!gO>Do@
z=;JP8=GdTZ_8MifcT;QCnu&1(LsSEpU4ORZ+g{vmjKWX596Ktzw1)CwjjODl7in2y
zeRC%Fs{HSyZ(9D`jT~k5qlFfvl(-cHGsb>cPB=Bv0}2<}P6gQXG7(>>cA$?ygGC~8
zAc&TioM%`9Jy)5F4GN@xVBms;n(GpauhwLspz`adf=XuGMg`+z$Fi^Y&diWJjZSAp
z<6G``p4TgCJm$WsRy}U*xNux5y8#(rQ2N^!kSv8lV5dlqSOEh`Pu%rRpYmQjpETGl
zM9%eV4!l~F#R-=oB4if88I_%QS2Sp$`m;N%sC@JIJe)5f*|6}C@Vt|VI8Wkdclt3h
zJTt`gem@y-)92L7Uxy){Y`^l&+%pl6zFVwis9wro_>|F|V7+3IG+P`=Pxo|n8ZJOd
zWWX|($4R(1I*d}0i;?obZqxklu|<tve8>!7=>aVLe;3^Rg_Iy*<Ln<+ko@bmhBojk
z(Bj6GY0BY(1xTXa?^MnawCW64FNfK2lRoH%F$rCN_CN_^Y&VoI76hYmF2`71L4DhF
z9UT$x8zVADez`br@kl6?BI-K*_LhxIyBwZ^0hTr^ix!BFOApnC*FpwrsuL@2@BX5t
zHx1;zsJW^Wu5b8<&A|www`3R8Cy8+kOl{jq3dpq4Xx^E-ek+&Ywh+xMgbTymjW<fJ
z_NHFN5!ysBzM_uzEy_yXuUu7V^{+4#s3z5-{7kV*mV$yE97F9f`W#;C#kIdqkJK65
z&%O?*VNtq0<B@(T9=dasxaBzfC#uE%lH~ozglhlXOW)l8?lS~nG60hSnEaoS*8OuA
z<Nr|g{VOIP!B_xomoYgO8bs)g=F+1QKu``RMV(TlP*Q7pOLtD+)jiFYLWMOlbU-eD
zOM9Y^)@2n=w=lcdVi>NTwwq@bzIfrvMd2#*zmm?AL;D?;5hqBY9ObDRwm!gajoDMD
z(KTxftRi6-nnsf}AJSX|*4I`-bzFqO*&1KV5%vha*C@dh%glWpVnsr1vpOA2q0fKW
z?@Jj=1Wg=g%!01rEe;|4gArL8SoHg1mu1o$c1H!e;DbD+bH(TPS}lTGQEJ0UUh%V_
zlec4@Gw7NAdiYDe<|TPf=FqfMt`Aw8d|AUA8PJo#Q{*u>1KC%oW)f&7puntE7YHdS
zx;<u!qXTdbD|CpsMJ;+Bfv2GkyJq*-P0_H)Dv7G@;>#B!-6GemMzOT-fnO__1-fmI
zR(=kgVU>Q-6+f-Qq1JdHJd)cCpW>+Irj611ZY{(<C8GLNN4*(U^dfbXmrTnFSAkUl
zEO2n!aoclb?1xTpgK6+3hx(V>Q|GMAHn<{u_i-4O7M!~cE8JiiaMf)`=t^#c-asxo
z1iGA^)lM=e^bGR8aUZ#wWStLdG&k()FRT2-Y(5z$$(oTB>TjXOX@u1<7OMCl(`lxu
zawBgBB)Dtb40!5JH(s3^HgDp!PQNt|4-;i$<tE#@!OjE4P{n_Wh*o)?3PW1y@iAr8
zCf~ha>YYgGUXa-wG>L3KV8=P#efL&}D!qW<{q7{PbFJ$Vb^wdo{j0+%y--aIPS%iG
zhBmb#-}RO8CMMz}B?RMbu)w!=Q%@1E(uP68O9;VP=1XAT3d8X`${%xhZ2MD&ym9)j
zK{so|mrxC9O@%Vm_gNJF)3PFILL2RCbyak<M`87Y9IX7ijZf?SDaK<67HlToX6c|b
zdS6}|b|M}!qzOGl6zEmM>zvT)KK-ALc!)1j1F3x46#U6_QSS8q$n2`#5^<0o#7d`H
z2p<S%x0TX_y=z7!Od0{0)im2(3)b%ROXt%wk~e4DSZ_i<QSFGE+~Mg!2fw98hakZZ
zemC(6%qbLD@X_&hEWD&j8|$1HT1XD#>AjgPth;gOPDks6XCwCaO0dmC4Lu+=3SAZw
z1=}omLAiSg?8BQ%ni(BQqS&N#y<GMi&K>MzxW^)nCX$My;&kOv3VShI$2DGj--fa2
zH13|lc+B5*`Q<XeT1{5UJm56?E~O90U=sf;7Nn@7=fi2gIJZ+7BcWfrNKM8K)~giw
z$*G%+T`4&$A4R@H>_?bEETMo|sv~b>0q720iWap{zH-_6fVz(0xetzIlNR<BO-9Y>
zoP7o|v5!b5m6>vegj~q@&$c)1cq%YK))kGq@z97w$Re{1m$v0eXN`2e9&`=-28vef
z7MRIFx6o-UpuoeED!pF{F&J>%M+|6ttE~hb5R&f=KUnw4)P>MjIG=o%y0v!n6C-Dr
z&f(=eB6<u@sIL|@5#lm<@l{TwZ$005)5qiT#!J0U_2;i(c6}zN^kc!Zq1}Y)%w9r}
z%dj76&2HCQ-%UqT)Mh|QBh1>dih4P55S&)dyiF&+RU8E$GlAkY?Z1QgX9!MwyFT`V
zJ}RvKiCS3D)ZztHePB%;?j<>FmYdR;&Q0T$lT=)3cMfFbG-3pCvdUOd4B}d-GSaWe
zBJ2q|Sff$w&7OtOE?U+Cz6+7E1RnZLk1Qn3b8s#(waq=%Wz`T^ZOfWfYJNW)w=1!+
zI~G5}J)=#+jq%l!n!aP%Gt0QBK{*#*a*H2I(c=5FA8#`Zhq1a-#ws)nUTVEJ$<H4I
z+vC-Y<uB5cSYrghB?i^K4lhNu{^I!TR41g|l7x;Jt}lzNyhCG~Ert-AY@VpzCt=YP
zqDofV42(49U@RZ3ON6<|=)f(DX`_ULdzToVve)iR=d`@Zv^Jo$P+R>MClF!H0UPr{
z4Rl0NKh${P50{-EWJYa2O$fPYhn1h=AjmnNQUlg#2oI-;I~Egf>Ym-hFvom<KOqhT
zl1OY6@;F50*4Y;CNYfl9H`gD@RK+@0cZ2f<hbiY0@f@$laE^BxL|%pOD1Y@>KI&jy
z@XskO748~N)nefXiyZ-m5=#Q%!eV6KWL2#|-yG6$xwu_!I>IOm#Hw73%nEQM_YF?H
znK$_<K;`NbvRJZjQD-8VbSa6x$$zv#B)Sb#UDk%XuBMe-e&n`KW_B?suUfWdiBOt`
zQ$z>rihI@kqq~T5V;|Eo6ml7XCZAsbMItc%1$a(xEq3J?mb_gi>cR7k%j}^x`u-A~
zD`+^P6@+!ZYA7yC5c68GwneAs&iojZ(z#DNZWmb{guqu{fC?mq{+6S&a~s<c*70Z8
zs;ZZ^xVZv^%5l!dHB!;>?7(@x#9nc|n~wzfXv`$!UZ^bUl*q}KwO?G(cjDON)Rm}H
z9Ld9<o1118A_QpRDtTVmNZ=pwzdQnNV?-$9MzIFV=A3#26oFyc_A`tIE^-zg*I%Qz
zh<i@7U(vvoFZH7mVIPzIYMjzP`?*=6YP&~6Zz%JtyzXkwM);oFF1@Xk{MXugLWXgA
zX)slvTzJ|lTlGG-<0NFQ{l2uS<Kd}Kmt9Y+GMZl^=8>wSbv`d6-Q&%O-@NmYPh04W
z`Lne|BtNZhxrJS!B8GW}B`0~?9)UDZ*y*IIB^ORX$y$uW>qpd)Zs)D)vMw^Vayyab
z{d%HN@H=lomu3kCzd@Dy+AbhmFikV2z41Lnkg<X38OXh(92A<0P7={kM)-?IdAcj5
zQvVvHbL!{BBx7?~>1Nz^Xea)60(hdf^My9>rSk-&nsj{1NFC$r9NsdjNhnX?U+Oa4
zUQ|iD#E>P>rzdPN&3<`O=cxtxf@{<nBQc8olSmiSu~=Z?7Ea#->uPC{>D4}Xm>T6R
zbPds9Pd}}1U^g4R;&{k7_rrvQ{C?CYExxowkB$vwRaP`Kp#D&_&WmP6aBu5eWfsep
zMc$CAg@&fk4M}fy6v)IR^0#&!(CoTznP=fNQPV$RBv<_CDGu)G%lL8Ajs>d!g+Q>g
zOt}B;OZPY{i6DiifMcno!OspfzC-d**F6(450gv_>XpPrEi7hw7m<r$E^P}4UXEaW
zW%?Whv*mFu?M;t7+7of~MyP0}ZO^I+l)5mQ-C&V4)?r520KwEK2^mG;wr}}x{F>S<
z{-4E!n(T%~g*f^{+<i_c;;0wf5IcLz-f)ZL^0#=h^H8Ky-lO!)JoTHCQx39PpPXtI
z%VEwCWv>4PXEXki2J%k~l3*_iQe%Mm4>11$=D)w${0AU0|0IbC2pAw>e;nlfODZZr
z{SQ$81JwWj*#pRcCHeO`E&-^LfGP>7l7HSr;V;B69$;ahQ`bS^%><=S=uUGfX{evc
zb7=~UoA>vcNxLQW4?KbnBd8x@wVMCOE+>D<Q~U|K;V&}4ASH?<0+1Mgn#9NfW&t`K
zw6E7w<5;8u7faw^kr_^*`=mep`M<j3{6CsE{~J%BfB)3leWiqYfK2<-nFi=O09^;5
z>-bxB9e^bXSdxGx30RVUx&7R~@1p(*&UXMp7C^`X2w4Cj`(I?C3Ba(w6NCMS^{bQf
zvV;6Fnq4WHZJRq${V$dn<3KNMCF6J}DcwHnn)t(<p$SN^i|ma;;n7e5U7(mL`ATxi
zM0*0h|F2GgG;u`E1Lz6>T>(({0MtGIx2hTct=0b*!*75YDiTzqVFoqtf9SYQ!3cZ{
zKz#t}15h7;`v141KEML|AN&@?zm-WqzyJXQh+zNhH&Oru&)<pQ0btmlz%XCVF(4?Q
zGlyR#DT$ANe0y#0r8%#cJlO{bIARsp*Vk9@kN=8@QH0Sjc^E<e_zeW6rTyOkz9;_#

literal 0
HcmV?d00001

diff --git a/tests/fsck-tests/014-no-extent-info/default_case.img b/tests/fsck-tests/014-no-extent-info/no_extent_bad_dev.img
similarity index 100%
rename from tests/fsck-tests/014-no-extent-info/default_case.img
rename to tests/fsck-tests/014-no-extent-info/no_extent_bad_dev.img
-- 
2.16.1

Re: [PATCH v4 00/18] btrfs-progs: lowmem check: avoid extents overwrite

[David Sterba]

On Thu, Feb 08, 2018 at 11:27:19AM +0800, Su Yue wrote:
> This patchset can be fetched from my github:
> https://github.com/Damenly/btrfs-progs/tree/lowmem
> based on unmerged patchset whose cover:
>       [PATCH 0/3] btrfs-progs: Split original mode check to its own
>       Author: Qu Wenruo <wqu@suse.com>
>       
> I'm sorry to send patches based on unmerged patch if it let you feel
> uncomfortable.
> I think the three patches from Qu are good enough so I send it before
> my vacation.
> 
> Patch[1-3] fix minor problems of lowmem repair.
> 
> Patch[4-8] introduce two ways to avoid extents overwrite:
> 1) Traverse trees and exclude all metadata blocks.
>    It's time-inefficient for large filesystems.
> 2) Mark all existed chunks full, allocate new chunk for CoW and
>    records chunk start.
>    If the last allocated chunk is almost full, allocated a new one.
> 2) is More efficient than 1). However, it can't handle situations
> like no space(fsck/004).
> Lowmem repair will try method 2 first and then method 1.
> 
> Patch[9-17] remove parameters @trans in functions for lowmem repair.
> They try to avoid extents overwrite if necessary and start
> transactions by themselves.
> 
> Patch[18] adds a test image.
> Those patches are mainly for lowmem repair. Original mode is not
> influenced.
> 
> ---
> Changlog:
> v4->v3:
>  - Remove global enum extents_operation to simplify
>    avoid_extents_overwrite() and its cleanup.
>  - Rebase after work of check split.
>  
> v3->v2:
>  - check_btrfs_root() returns FATAL_ERROR if check_fs_first_inode()
>    failed. Thanks Nikolay Borisov.
>  - Add function try_to_force_cow_in_new_chunk() and global u64
>    varaiable to record start of the last allocated chunk.
>  - Remove unused EXTENTS_PIN in enum lowmem_extents_operation.
>  
> v2->v1:
>  - Let @err in check_btrfs_root() record err bits but excluded
>    negative values.
>  - Do not delete a line of code to release path after extent item'
>    insertion in repair_extent_data_item().
>  - Add patch[3].
>  - Force CoW in new allocated chunk to avoid extents overwrite.
>  - Remove parameters @trans in check_chunks_and_extents_v2() and
>    related callees.
>  - Repair functions for lowmem mode call try_avoid_extents_overwrite()
>    and start transactions.
>    
> Su Yue (18):
>   btrfs-progs: lowmem check: release path in repair_extent_data_item()
>   btrfs-progs: lowmem check: record returned errors after
>     walk_down_tree_v2()
>   btrfs-progs: lowmem check: assign @parent early in
>     repair_extent_data_item()
>   btrfs-progs: lowmem check: exclude extents of metadata blocks

Patch 1-4 added to devel.

>   btrfs-progs: lowmem check: introduce mark/clear_block_groups_full()

There were too many conflicts starting with this patch.

Can you please refresh teh rest of the series and resend?

For subject please use "btrfs-progs: check: lowmem: ..." and try to fix
the typos in the changelogs. Thanks.



>   btrfs-progs: lowmem check: introduce try_force_cow_in_new_chunk()
>   btrfs-progs: lowmem check: introduce avoid_extents_overwrite()
>   btrfs-progs: lowmem check: exclude extents if init-extent-tree in
>     lowmem
>   btrfs-progs: lowmem check: start to remove parameters @trans in lowmem
>   btrfs-progs: lowmem check: remove parameter @trans of
>     delete_extent_item()
>   btrfs-progs: lowmem check: remove parameter @trans of
>     repair_chunk_item()
>   btrfs-progs: lowmem check: remove parameter @trans of
>     repair_extent_item()
>   btrfs-progs: lowmem check: remove parameter @trans of
>     check_leaf_items()
>   btrfs-progs: lowmem check: remove parameter @trans of
>     repair_tree_back_ref()
>   btrfs-progs: lowmem check: remove parameter @trans of
>     check_btrfs_root()
>   btrfs-progs: lowmem check: introduce repair_block_accounting()
>   btrfs-progs: lowmem check: end of removing parameters @trans in lowmem
>   btrfs-progs: fsck-tests: add image no extent with normal device size

Re: [PATCH v4 00/18] btrfs-progs: lowmem check: avoid extents overwrite

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 5471 bytes --]



On 2018年05月08日 10:09, Su Yue wrote:
> 
> 
> On 05/08/2018 02:17 AM, David Sterba wrote:
>> On Thu, Feb 08, 2018 at 11:27:19AM +0800, Su Yue wrote:
>>> This patchset can be fetched from my github:
>>> https://github.com/Damenly/btrfs-progs/tree/lowmem
>>> based on unmerged patchset whose cover:
>>>        [PATCH 0/3] btrfs-progs: Split original mode check to its own
>>>        Author: Qu Wenruo <wqu@suse.com>
>>>        I'm sorry to send patches based on unmerged patch if it let
>>> you feel
>>> uncomfortable.
>>> I think the three patches from Qu are good enough so I send it before
>>> my vacation.
>>>
>>> Patch[1-3] fix minor problems of lowmem repair.
>>>
>>> Patch[4-8] introduce two ways to avoid extents overwrite:
>>> 1) Traverse trees and exclude all metadata blocks.
>>>     It's time-inefficient for large filesystems.
>>> 2) Mark all existed chunks full, allocate new chunk for CoW and
>>>     records chunk start.
>>>     If the last allocated chunk is almost full, allocated a new one.
>>> 2) is More efficient than 1). However, it can't handle situations
>>> like no space(fsck/004).
>>> Lowmem repair will try method 2 first and then method 1.
>>>
>>> Patch[9-17] remove parameters @trans in functions for lowmem repair.
>>> They try to avoid extents overwrite if necessary and start
>>> transactions by themselves.
>>>
>>> Patch[18] adds a test image.
>>> Those patches are mainly for lowmem repair. Original mode is not
>>> influenced.
>>>
>>> ---
>>> Changlog:
>>> v4->v3:
>>>   - Remove global enum extents_operation to simplify
>>>     avoid_extents_overwrite() and its cleanup.
>>>   - Rebase after work of check split.
>>>   v3->v2:
>>>   - check_btrfs_root() returns FATAL_ERROR if check_fs_first_inode()
>>>     failed. Thanks Nikolay Borisov.
>>>   - Add function try_to_force_cow_in_new_chunk() and global u64
>>>     varaiable to record start of the last allocated chunk.
>>>   - Remove unused EXTENTS_PIN in enum lowmem_extents_operation.
>>>   v2->v1:
>>>   - Let @err in check_btrfs_root() record err bits but excluded
>>>     negative values.
>>>   - Do not delete a line of code to release path after extent item'
>>>     insertion in repair_extent_data_item().
>>>   - Add patch[3].
>>>   - Force CoW in new allocated chunk to avoid extents overwrite.
>>>   - Remove parameters @trans in check_chunks_and_extents_v2() and
>>>     related callees.
>>>   - Repair functions for lowmem mode call try_avoid_extents_overwrite()
>>>     and start transactions.
>>>     Su Yue (18):
>>>    btrfs-progs: lowmem check: release path in repair_extent_data_item()
>>>    btrfs-progs: lowmem check: record returned errors after
>>>      walk_down_tree_v2()
>>>    btrfs-progs: lowmem check: assign @parent early in
>>>      repair_extent_data_item()
>>>    btrfs-progs: lowmem check: exclude extents of metadata blocks
>>
>> Patch 1-4 added to devel.
>>
>>>    btrfs-progs: lowmem check: introduce mark/clear_block_groups_full()
>>
>> There were too many conflicts starting with this patch.
>>
>> Can you please refresh teh rest of the series and resend?
>>
> OK of cource. An small question below.
> 
> Though this patchset is for lowmem mode, it also changes original mode
> for code reuse.
> 
> This patch and remainings base on Qu's unmerged patchset:
>     [PATCH 0/3] btrfs-progs: Split original mode check to its own
>         Author: Qu Wenruo <wqu@suse.com>
> New version is named
>     [PATCH 0/3] btrfs-progs: Split mode-original code
>     Author: Qu Wenruo <wqu@suse.com>

Sorry, that patchset will not be updated until we get a relatively
stable check code base.
So I'm afraid you need to base them on current devel branch other than
my patchset.

Thanks,
Qu

> 
> I just tried to rebase his brach named split_check_for_david to devel
> but failed with conflictions.
> 
> Does V2 need to get rid of the precondition and just rebases to devel?
> Or to wait completion of split?
> 
> Thanks,
> Su
> 
>> For subject please use "btrfs-progs: check: lowmem: ..." and try to fix
>> the typos in the changelogs. Thanks.
>>
>>
>>
>>>    btrfs-progs: lowmem check: introduce try_force_cow_in_new_chunk()
>>>    btrfs-progs: lowmem check: introduce avoid_extents_overwrite()
>>>    btrfs-progs: lowmem check: exclude extents if init-extent-tree in
>>>      lowmem
>>>    btrfs-progs: lowmem check: start to remove parameters @trans in
>>> lowmem
>>>    btrfs-progs: lowmem check: remove parameter @trans of
>>>      delete_extent_item()
>>>    btrfs-progs: lowmem check: remove parameter @trans of
>>>      repair_chunk_item()
>>>    btrfs-progs: lowmem check: remove parameter @trans of
>>>      repair_extent_item()
>>>    btrfs-progs: lowmem check: remove parameter @trans of
>>>      check_leaf_items()
>>>    btrfs-progs: lowmem check: remove parameter @trans of
>>>      repair_tree_back_ref()
>>>    btrfs-progs: lowmem check: remove parameter @trans of
>>>      check_btrfs_root()
>>>    btrfs-progs: lowmem check: introduce repair_block_accounting()
>>>    btrfs-progs: lowmem check: end of removing parameters @trans in
>>> lowmem
>>>    btrfs-progs: fsck-tests: add image no extent with normal device size
>>
>>
>>
> 
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v4 00/18] btrfs-progs: lowmem check: avoid extents overwrite

[Su Yue]

On 05/08/2018 02:17 AM, David Sterba wrote:
> On Thu, Feb 08, 2018 at 11:27:19AM +0800, Su Yue wrote:
>> This patchset can be fetched from my github:
>> https://github.com/Damenly/btrfs-progs/tree/lowmem
>> based on unmerged patchset whose cover:
>>        [PATCH 0/3] btrfs-progs: Split original mode check to its own
>>        Author: Qu Wenruo <wqu@suse.com>
>>        
>> I'm sorry to send patches based on unmerged patch if it let you feel
>> uncomfortable.
>> I think the three patches from Qu are good enough so I send it before
>> my vacation.
>>
>> Patch[1-3] fix minor problems of lowmem repair.
>>
>> Patch[4-8] introduce two ways to avoid extents overwrite:
>> 1) Traverse trees and exclude all metadata blocks.
>>     It's time-inefficient for large filesystems.
>> 2) Mark all existed chunks full, allocate new chunk for CoW and
>>     records chunk start.
>>     If the last allocated chunk is almost full, allocated a new one.
>> 2) is More efficient than 1). However, it can't handle situations
>> like no space(fsck/004).
>> Lowmem repair will try method 2 first and then method 1.
>>
>> Patch[9-17] remove parameters @trans in functions for lowmem repair.
>> They try to avoid extents overwrite if necessary and start
>> transactions by themselves.
>>
>> Patch[18] adds a test image.
>> Those patches are mainly for lowmem repair. Original mode is not
>> influenced.
>>
>> ---
>> Changlog:
>> v4->v3:
>>   - Remove global enum extents_operation to simplify
>>     avoid_extents_overwrite() and its cleanup.
>>   - Rebase after work of check split.
>>   
>> v3->v2:
>>   - check_btrfs_root() returns FATAL_ERROR if check_fs_first_inode()
>>     failed. Thanks Nikolay Borisov.
>>   - Add function try_to_force_cow_in_new_chunk() and global u64
>>     varaiable to record start of the last allocated chunk.
>>   - Remove unused EXTENTS_PIN in enum lowmem_extents_operation.
>>   
>> v2->v1:
>>   - Let @err in check_btrfs_root() record err bits but excluded
>>     negative values.
>>   - Do not delete a line of code to release path after extent item'
>>     insertion in repair_extent_data_item().
>>   - Add patch[3].
>>   - Force CoW in new allocated chunk to avoid extents overwrite.
>>   - Remove parameters @trans in check_chunks_and_extents_v2() and
>>     related callees.
>>   - Repair functions for lowmem mode call try_avoid_extents_overwrite()
>>     and start transactions.
>>     
>> Su Yue (18):
>>    btrfs-progs: lowmem check: release path in repair_extent_data_item()
>>    btrfs-progs: lowmem check: record returned errors after
>>      walk_down_tree_v2()
>>    btrfs-progs: lowmem check: assign @parent early in
>>      repair_extent_data_item()
>>    btrfs-progs: lowmem check: exclude extents of metadata blocks
> 
> Patch 1-4 added to devel.
> 
>>    btrfs-progs: lowmem check: introduce mark/clear_block_groups_full()
> 
> There were too many conflicts starting with this patch.
> 
> Can you please refresh teh rest of the series and resend?
> 
OK of cource. An small question below.

Though this patchset is for lowmem mode, it also changes original mode
for code reuse.

This patch and remainings base on Qu's unmerged patchset:
	[PATCH 0/3] btrfs-progs: Split original mode check to its own
         Author: Qu Wenruo <wqu@suse.com>
New version is named
	[PATCH 0/3] btrfs-progs: Split mode-original code
	Author: Qu Wenruo <wqu@suse.com>

I just tried to rebase his brach named split_check_for_david to devel 
but failed with conflictions.

Does V2 need to get rid of the precondition and just rebases to devel?
Or to wait completion of split?

Thanks,
Su

> For subject please use "btrfs-progs: check: lowmem: ..." and try to fix
> the typos in the changelogs. Thanks.
> 
> 
> 
>>    btrfs-progs: lowmem check: introduce try_force_cow_in_new_chunk()
>>    btrfs-progs: lowmem check: introduce avoid_extents_overwrite()
>>    btrfs-progs: lowmem check: exclude extents if init-extent-tree in
>>      lowmem
>>    btrfs-progs: lowmem check: start to remove parameters @trans in lowmem
>>    btrfs-progs: lowmem check: remove parameter @trans of
>>      delete_extent_item()
>>    btrfs-progs: lowmem check: remove parameter @trans of
>>      repair_chunk_item()
>>    btrfs-progs: lowmem check: remove parameter @trans of
>>      repair_extent_item()
>>    btrfs-progs: lowmem check: remove parameter @trans of
>>      check_leaf_items()
>>    btrfs-progs: lowmem check: remove parameter @trans of
>>      repair_tree_back_ref()
>>    btrfs-progs: lowmem check: remove parameter @trans of
>>      check_btrfs_root()
>>    btrfs-progs: lowmem check: introduce repair_block_accounting()
>>    btrfs-progs: lowmem check: end of removing parameters @trans in lowmem
>>    btrfs-progs: fsck-tests: add image no extent with normal device size
> 
> 
> 

Re: [PATCH v4 00/18] btrfs-progs: lowmem check: avoid extents overwrite

[Su Yue]

On 05/08/2018 10:06 AM, Qu Wenruo wrote:
> 
> 
> On 2018年05月08日 10:09, Su Yue wrote:
>>
>>
>> On 05/08/2018 02:17 AM, David Sterba wrote:
>>> On Thu, Feb 08, 2018 at 11:27:19AM +0800, Su Yue wrote:
>>>> This patchset can be fetched from my github:
>>>> https://github.com/Damenly/btrfs-progs/tree/lowmem
>>>> based on unmerged patchset whose cover:
>>>>         [PATCH 0/3] btrfs-progs: Split original mode check to its own
>>>>         Author: Qu Wenruo <wqu@suse.com>
>>>>         I'm sorry to send patches based on unmerged patch if it let
>>>> you feel
>>>> uncomfortable.
>>>> I think the three patches from Qu are good enough so I send it before
>>>> my vacation.
>>>>
>>>> Patch[1-3] fix minor problems of lowmem repair.
>>>>
>>>> Patch[4-8] introduce two ways to avoid extents overwrite:
>>>> 1) Traverse trees and exclude all metadata blocks.
>>>>      It's time-inefficient for large filesystems.
>>>> 2) Mark all existed chunks full, allocate new chunk for CoW and
>>>>      records chunk start.
>>>>      If the last allocated chunk is almost full, allocated a new one.
>>>> 2) is More efficient than 1). However, it can't handle situations
>>>> like no space(fsck/004).
>>>> Lowmem repair will try method 2 first and then method 1.
>>>>
>>>> Patch[9-17] remove parameters @trans in functions for lowmem repair.
>>>> They try to avoid extents overwrite if necessary and start
>>>> transactions by themselves.
>>>>
>>>> Patch[18] adds a test image.
>>>> Those patches are mainly for lowmem repair. Original mode is not
>>>> influenced.
>>>>
>>>> ---
>>>> Changlog:
>>>> v4->v3:
>>>>    - Remove global enum extents_operation to simplify
>>>>      avoid_extents_overwrite() and its cleanup.
>>>>    - Rebase after work of check split.
>>>>    v3->v2:
>>>>    - check_btrfs_root() returns FATAL_ERROR if check_fs_first_inode()
>>>>      failed. Thanks Nikolay Borisov.
>>>>    - Add function try_to_force_cow_in_new_chunk() and global u64
>>>>      varaiable to record start of the last allocated chunk.
>>>>    - Remove unused EXTENTS_PIN in enum lowmem_extents_operation.
>>>>    v2->v1:
>>>>    - Let @err in check_btrfs_root() record err bits but excluded
>>>>      negative values.
>>>>    - Do not delete a line of code to release path after extent item'
>>>>      insertion in repair_extent_data_item().
>>>>    - Add patch[3].
>>>>    - Force CoW in new allocated chunk to avoid extents overwrite.
>>>>    - Remove parameters @trans in check_chunks_and_extents_v2() and
>>>>      related callees.
>>>>    - Repair functions for lowmem mode call try_avoid_extents_overwrite()
>>>>      and start transactions.
>>>>      Su Yue (18):
>>>>     btrfs-progs: lowmem check: release path in repair_extent_data_item()
>>>>     btrfs-progs: lowmem check: record returned errors after
>>>>       walk_down_tree_v2()
>>>>     btrfs-progs: lowmem check: assign @parent early in
>>>>       repair_extent_data_item()
>>>>     btrfs-progs: lowmem check: exclude extents of metadata blocks
>>>
>>> Patch 1-4 added to devel.
>>>
>>>>     btrfs-progs: lowmem check: introduce mark/clear_block_groups_full()
>>>
>>> There were too many conflicts starting with this patch.
>>>
>>> Can you please refresh teh rest of the series and resend?
>>>
>> OK of cource. An small question below.
>>
>> Though this patchset is for lowmem mode, it also changes original mode
>> for code reuse.
>>
>> This patch and remainings base on Qu's unmerged patchset:
>>      [PATCH 0/3] btrfs-progs: Split original mode check to its own
>>          Author: Qu Wenruo <wqu@suse.com>
>> New version is named
>>      [PATCH 0/3] btrfs-progs: Split mode-original code
>>      Author: Qu Wenruo <wqu@suse.com>
> 
> Sorry, that patchset will not be updated until we get a relatively
> stable check code base.
> So I'm afraid you need to base them on current devel branch other than
> my patchset.
> 
Understood. Thanks a lot.

Thanks,
Su

> Thanks,
> Qu
> 
>>
>> I just tried to rebase his brach named split_check_for_david to devel
>> but failed with conflictions.
>>
>> Does V2 need to get rid of the precondition and just rebases to devel?
>> Or to wait completion of split?
>>
>> Thanks,
>> Su
>>
>>> For subject please use "btrfs-progs: check: lowmem: ..." and try to fix
>>> the typos in the changelogs. Thanks.
>>>
>>>
>>>
>>>>     btrfs-progs: lowmem check: introduce try_force_cow_in_new_chunk()
>>>>     btrfs-progs: lowmem check: introduce avoid_extents_overwrite()
>>>>     btrfs-progs: lowmem check: exclude extents if init-extent-tree in
>>>>       lowmem
>>>>     btrfs-progs: lowmem check: start to remove parameters @trans in
>>>> lowmem
>>>>     btrfs-progs: lowmem check: remove parameter @trans of
>>>>       delete_extent_item()
>>>>     btrfs-progs: lowmem check: remove parameter @trans of
>>>>       repair_chunk_item()
>>>>     btrfs-progs: lowmem check: remove parameter @trans of
>>>>       repair_extent_item()
>>>>     btrfs-progs: lowmem check: remove parameter @trans of
>>>>       check_leaf_items()
>>>>     btrfs-progs: lowmem check: remove parameter @trans of
>>>>       repair_tree_back_ref()
>>>>     btrfs-progs: lowmem check: remove parameter @trans of
>>>>       check_btrfs_root()
>>>>     btrfs-progs: lowmem check: introduce repair_block_accounting()
>>>>     btrfs-progs: lowmem check: end of removing parameters @trans in
>>>> lowmem
>>>>     btrfs-progs: fsck-tests: add image no extent with normal device size
>>>
>>>
>>>
>>
>>
>>
> 

Re: [PATCH v4 00/18] btrfs-progs: lowmem check: avoid extents overwrite

[David Sterba]

On Tue, May 08, 2018 at 10:06:05AM +0800, Qu Wenruo wrote:
> >>>   - Repair functions for lowmem mode call try_avoid_extents_overwrite()
> >>>     and start transactions.
> >>>     Su Yue (18):
> >>>    btrfs-progs: lowmem check: release path in repair_extent_data_item()
> >>>    btrfs-progs: lowmem check: record returned errors after
> >>>      walk_down_tree_v2()
> >>>    btrfs-progs: lowmem check: assign @parent early in
> >>>      repair_extent_data_item()
> >>>    btrfs-progs: lowmem check: exclude extents of metadata blocks
> >>
> >> Patch 1-4 added to devel.
> >>
> >>>    btrfs-progs: lowmem check: introduce mark/clear_block_groups_full()
> >>
> >> There were too many conflicts starting with this patch.
> >>
> >> Can you please refresh teh rest of the series and resend?
> >>
> > OK of cource. An small question below.
> > 
> > Though this patchset is for lowmem mode, it also changes original mode
> > for code reuse.
> > 
> > This patch and remainings base on Qu's unmerged patchset:
> >     [PATCH 0/3] btrfs-progs: Split original mode check to its own
> >         Author: Qu Wenruo <wqu@suse.com>
> > New version is named
> >     [PATCH 0/3] btrfs-progs: Split mode-original code
> >     Author: Qu Wenruo <wqu@suse.com>
> 
> Sorry, that patchset will not be updated until we get a relatively
> stable check code base.
> So I'm afraid you need to base them on current devel branch other than
> my patchset.

Yeah, my plan for the upcoming weeks is to get all the pending patchsets
with fixes merged before any bigger code cleanups, in check/.