[PATCH v3 00/22] Btrfs-convert rework to support separate chunk type

[PATCH v3 00/22] Btrfs-convert rework to support separate chunk type

[Qu Wenruo]

Here comes the 2st version of btrfs-convert rework.
This time, it is rebased to co-operate with David new abstraction work
for incoming reiserfs support.
(Sorry David, your reiserfs support will need more modification as the
abstraction facility is also modified in this patchset)

Any test is welcomed, and it can already pass the convert test from
btrfs-progs. (Since the test doesn't test rollback function)

I also did some tests like create ext4 and fill it with fsstress to test
convert, at least no bug spotted yet.

The new btrfs-convert has the following bugfix/feature:
1. True separate meta/data chunk
   The converted filesystem will has correct chunk layout, and all ext*
   data will be covered by data chunk and metadata will be covered by
   metadata chunk.
   Allowing converted filesystem to pass latest btrfs check.

2. Structurized make_btrfs_v2()
   Unlike old make_btrfs() which is a mess of hand written codes, new
   make_btrfs_v2() is all structurized and should be easy for later
   expansion.

But it still has something to do:
1) Ability to handle 10+ TB level filesystem.
   Current implement rely on the initial system chunk to contain all
   chunks until we inserted all data chunks.
   However the initial system chunk are only 32M, in simple calculation,
   it can only hold at most 400K chunks.
   If every chunk is in the smallest size (16M), it can only handle
   less than 6T size in its worst case.
   Better calculate the initial system chunk size according to the
   filesystem size

   But now, I'll focus on current implement to ensure it's safe and
   stable.

2) Ability to create DUP metadata chunk without using balance
   Now the data and metadata chunk are all in SINGLE profile.
   User needs to convert metadata to DUP if they want DUP metadata
   profile.

   This is a little hard, as we must do it at make_btrfs() time, where
   we don't have full btrfs facilities to modify the metadata tree.

   But should be easy to add DUP support since the make_btrfs() is much
   clearer than ever.

The core idea of new btrfs-convert is quite simple:
Do reserve at *CHUNK* level.

Instead of old extent level reserve, we just insert a big enough SYS/META
chunks at make_btrfs() time, then insert all data chunks to cover old fs
data.
Then things will get very clear. No custom extent allocation function,
all chunk/extent allocation facility can be used without modification
and won't screw up chunk layout.

For rollback function, as old and new convert has completely different
chunk type (ANY vs DATA), so make rollback a little less restricted to
handle both old and new behavior.

Changelog:
v1:
  Add rollback support
  Make btrfs-convert more safe on chunk type

v2:
  Rebased to latest devel branch.

v3:
  Rebased to latest devel branch.
  Fix a btrfs patch leak.

Qu Wenruo (22):
  btrfs-progs: convert: Introduce functions to read used space
  btrfs-progs: convert: Introduce new function to remove reserved ranges
  btrfs-progs: convert: Introduce function to calculate the available
    space
  btrfs-progs: utils: Introduce new function for convert
  btrfs-progs: Introduce function to setup temporary superblock
  btrfs-progs: Introduce function to setup temporary tree root
  btrfs-progs: Introduce function to setup temporary chunk root
  btrfs-progs: Introduce function to initialize device tree
  btrfs-progs: Introduce function to initialize fs tree
  btrfs-progs: Introduce function to initialize csum tree
  btrfs-progs: Introduce function to setup temporary extent tree
  btrfs-progs: Introduce function to create convert data chunks
  btrfs-progs: extent-tree: Introduce function to find the first overlap
    extent.
  btrfs-progs: extent-tree: Enhance btrfs_record_file_extent
  btrfs-progs: convert: Introduce new function to create converted image
  btrfs-progs: convert: Introduce function to migrate reserved ranges
  btrfs-progs: convert: Enhance record_file_blocks to handle reserved
    ranges
  btrfs-progs: convert: Introduce init_btrfs_v2 function.
  btrfs-progs: Introduce do_convert_v2 function
  btrfs-progs: Convert: Add support for rollback new convert behavior
  btrfs-progs: convert: Strictly avoid meta or system chunk allocation
  btrfs-progs: Cleanup old btrfs-convert

 btrfs-convert.c | 1873 ++++++++++++++++++++++---------------------------------
 ctree.c         |   24 +
 ctree.h         |   14 +-
 extent-tree.c   |  251 ++++++--
 mkfs.c          |    4 +-
 utils.c         |  828 +++++++++++++++++++++++-
 utils.h         |   26 +-
 volumes.c       |   46 +-
 volumes.h       |    2 +-
 9 files changed, 1858 insertions(+), 1210 deletions(-)

-- 
2.7.0

[PATCH v3 01/22] btrfs-progs: convert: Introduce functions to read used space

[Qu Wenruo]

Before we do real convert, we need to read and build up used space cache
tree for later data/meta separate chunk layout.

This patch will iterate all used blocks in ext2 filesystem and record it
into cctx->used cache tree, for later use.

This provides the very basic of later btrfs-convert rework.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 4baa68e..65841bd 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -81,6 +81,7 @@ struct btrfs_convert_context;
 struct btrfs_convert_operations {
 	const char *name;
 	int (*open_fs)(struct btrfs_convert_context *cctx, const char *devname);
+	int (*read_used_space)(struct btrfs_convert_context *cctx);
 	int (*alloc_block)(struct btrfs_convert_context *cctx, u64 goal,
 			   u64 *block_ret);
 	int (*alloc_block_range)(struct btrfs_convert_context *cctx, u64 goal,
@@ -230,6 +231,73 @@ fail:
 	return -1;
 }
 
+static int __ext2_add_one_block(ext2_filsys fs, char *bitmap,
+				unsigned long group_nr, struct cache_tree *used)
+{
+	unsigned long offset;
+	unsigned i;
+	int ret = 0;
+
+	offset = fs->super->s_first_data_block;
+	offset /= EXT2FS_CLUSTER_RATIO(fs);
+	offset += group_nr * fs->super->s_clusters_per_group;
+	for (i = 0; i < fs->super->s_clusters_per_group; i++) {
+		if (ext2fs_test_bit(i, bitmap)) {
+			u64 start;
+
+			start = (i + offset) * EXT2FS_CLUSTER_RATIO(fs);
+			start *= fs->blocksize;
+			ret = add_merge_cache_extent(used, start,
+						     fs->blocksize);
+			if (ret < 0)
+				break;
+		}
+	}
+	return ret;
+}
+
+/*
+ * Read all used ext2 space into cctx->used cache tree
+ */
+static int ext2_read_used_space(struct btrfs_convert_context *cctx)
+{
+	ext2_filsys fs = (ext2_filsys)cctx->fs_data;
+	blk64_t blk_itr = EXT2FS_B2C(fs, fs->super->s_first_data_block);
+	struct cache_tree *used_tree = &cctx->used;
+	char *block_bitmap = NULL;
+	unsigned long i;
+	int block_nbytes;
+	int ret = 0;
+
+	block_nbytes = EXT2_CLUSTERS_PER_GROUP(fs->super) / 8;
+	/* Shouldn't happen */
+	BUG_ON(!fs->block_map);
+
+	block_bitmap = malloc(block_nbytes);
+	if (!block_bitmap)
+		return -ENOMEM;
+
+	for (i = 0; i < fs->group_desc_count; i++) {
+		ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
+						block_nbytes * 8, block_bitmap);
+		if (ret) {
+			error("fail to get bitmap from ext2, %s",
+			      strerror(-ret));
+			break;
+		}
+		ret = __ext2_add_one_block(fs, block_bitmap, i, used_tree);
+		if (ret < 0) {
+			error("fail to build used space tree, %s",
+			      strerror(-ret));
+			break;
+		}
+		blk_itr += fs->super->s_clusters_per_group;
+	}
+
+	free(block_bitmap);
+	return ret;
+}
+
 static void ext2_close_fs(struct btrfs_convert_context *cctx)
 {
 	if (cctx->volume_name) {
@@ -2418,6 +2486,7 @@ err:
 static const struct btrfs_convert_operations ext2_convert_ops = {
 	.name			= "ext2",
 	.open_fs		= ext2_open_fs,
+	.read_used_space	= ext2_read_used_space,
 	.alloc_block		= ext2_alloc_block,
 	.alloc_block_range	= ext2_alloc_block_range,
 	.copy_inodes		= ext2_copy_inodes,
@@ -2451,6 +2520,14 @@ static int convert_open_fs(const char *devname,
 	return -1;
 }
 
+/*
+ * Read used space
+ */
+static int convert_read_used_space(struct btrfs_convert_context *cctx)
+{
+	return cctx->convert_ops->read_used_space(cctx);
+}
+
 static int do_convert(const char *devname, int datacsum, int packing, int noxattr,
 		u32 nodesize, int copylabel, const char *fslabel, int progress,
 		u64 features)
@@ -2474,6 +2551,9 @@ static int do_convert(const char *devname, int datacsum, int packing, int noxatt
 	ret = convert_open_fs(devname, &cctx);
 	if (ret)
 		goto fail;
+	ret = convert_read_used_space(&cctx);
+	if (ret)
+		goto fail;
 
 	blocksize = cctx.blocksize;
 	total_bytes = (u64)blocksize * (u64)cctx.block_count;
-- 
2.7.0

[PATCH v3 02/22] btrfs-progs: convert: Introduce new function to remove reserved ranges

[Qu Wenruo]

Introduce functions to remove reserved ranges for later btrfs-convert
rework.

The reserved ranges includes:
1. [0,1M)
2. [btrfs_sb_offset(1), +BTRFS_STRIP_LEN)
3. [btrfs_sb_offset(2), +BTRFS_STRIP_LEN)

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 65841bd..2fef1ed 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -2521,6 +2521,123 @@ static int convert_open_fs(const char *devname,
 }
 
 /*
+ * Remove one reserve range from given cache tree
+ * if min_stripe_size is non-zero, it will ensure for split case,
+ * all its split cache extent is no smaller than @min_strip_size / 2.
+ */
+static int wipe_one_reserved_range(struct cache_tree *tree,
+				   u64 start, u64 len, u64 min_stripe_size,
+				   int ensure_size)
+{
+	struct cache_extent *cache;
+	int ret;
+
+	BUG_ON(ensure_size && min_stripe_size == 0);
+	/*
+	 * The logical here is simplified to handle special cases only
+	 * So we don't need to consider merge case for ensure_size
+	 */
+	BUG_ON(min_stripe_size && (min_stripe_size < len * 2 ||
+	       min_stripe_size / 2 < BTRFS_STRIPE_LEN));
+
+	/* Also, wipe range should already be aligned */
+	BUG_ON(start != round_down(start, BTRFS_STRIPE_LEN) ||
+	       start + len != round_up(start + len, BTRFS_STRIPE_LEN));
+
+	min_stripe_size /= 2;
+
+	cache = lookup_cache_extent(tree, start, len);
+	if (!cache)
+		return 0;
+
+	if (start <= cache->start) {
+		/*
+		 *	|--------cache---------|
+		 * |-wipe-|
+		 */
+		BUG_ON(start + len <= cache->start);
+
+		/*
+		 * The wipe size is smaller than min_stripe_size / 2,
+		 * so the result length should still meet min_stripe_size
+		 * And no need to do alignment
+		 */
+		cache->size -= (start + len - cache->start);
+		if (cache->size == 0) {
+			remove_cache_extent(tree, cache);
+			free(cache);
+			return 0;
+		}
+
+		BUG_ON(ensure_size && cache->size < min_stripe_size);
+
+		cache->start = start + len;
+		return 0;
+	} else if (start > cache->start && start + len < cache->start +
+		   cache->size) {
+		/*
+		 * |-------cache-----|
+		 *	|-wipe-|
+		 */
+		u64 old_len = cache->size;
+		u64 insert_start = start + len;
+		u64 insert_len;
+
+		cache->size = start - cache->start;
+		if (ensure_size)
+			cache->size = max(cache->size, min_stripe_size);
+		cache->start = start - cache->size;
+
+		/* And insert the new one */
+		insert_len = old_len - start - len;
+		if (ensure_size)
+			insert_len = max(insert_len, min_stripe_size);
+
+		ret = add_merge_cache_extent(tree, insert_start, insert_len);
+		return ret;
+	}
+	/*
+	 * |----cache-----|
+	 *		|--wipe-|
+	 * Wipe len should be small enough and no need to expand the
+	 * remaining extent
+	 */
+	cache->size = start - cache->start;
+	BUG_ON(ensure_size && cache->size < min_stripe_size);
+	return 0;
+}
+
+/*
+ * Remove reserved ranges from given cache_tree
+ *
+ * It will remove the following ranges
+ * 1) 0~1M
+ * 2) 2nd superblock, +64K(make sure chunks are 64K aligned)
+ * 3) 3rd superblock, +64K
+ *
+ * @min_stripe must be given for safety check
+ * and if @ensure_size is given, it will ensure affected cache_extent will be
+ * larger than min_stripe_size
+ */
+static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
+				int ensure_size)
+{
+	int ret;
+
+	ret = wipe_one_reserved_range(tree, 0, 1024 * 1024, min_stripe_size,
+				      ensure_size);
+	if (ret < 0)
+		return ret;
+	ret = wipe_one_reserved_range(tree, btrfs_sb_offset(1),
+			BTRFS_STRIPE_LEN, min_stripe_size, ensure_size);
+	if (ret < 0)
+		return ret;
+	ret = wipe_one_reserved_range(tree, btrfs_sb_offset(2),
+			BTRFS_STRIPE_LEN, min_stripe_size, ensure_size);
+	return ret;
+}
+
+/*
  * Read used space
  */
 static int convert_read_used_space(struct btrfs_convert_context *cctx)
-- 
2.7.0

[PATCH v3 03/22] btrfs-progs: convert: Introduce function to calculate the available space

[Qu Wenruo]

Introduce a new function, calculate_available_space() to get available
space cache_tree data_chunks cache_tree.

Unlike old implement, this function will do the new work:
1) batch used ext* data space.
   To ensure data chunks will recovery them all.
   And restore the result into mkfs_cfg->convert_data_chunks for later
   user.

2) avoid SB and reserved space at chunk level
   Both batched data space or free space will not cover reserved space,
   like sb or the first 1M.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 90 insertions(+), 2 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 2fef1ed..8026907 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -2637,12 +2637,100 @@ static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
 	return ret;
 }
 
+static int calculate_available_space(struct btrfs_convert_context *cctx)
+{
+	struct cache_tree *used = &cctx->used;
+	struct cache_tree *data_chunks = &cctx->data_chunks;
+	struct cache_tree *free = &cctx->free;
+	struct cache_extent *cache;
+	u64 cur_off = 0;
+	/*
+	 * Twice the minimal chunk size, to allow later wipe_reserved_ranges()
+	 * works without need to consider overlap
+	 */
+	u64 min_stripe_size = 2 * 16 * 1024 * 1024;
+	int ret;
+
+	/* Calculate data_chunks */
+	for (cache = first_cache_extent(used); cache;
+	     cache = next_cache_extent(cache)) {
+		u64 cur_len;
+
+		if (cache->start + cache->size < cur_off)
+			continue;
+		if (cache->start > cur_off + min_stripe_size)
+			cur_off = cache->start;
+		cur_len = max(cache->start + cache->size - cur_off,
+			      min_stripe_size);
+		ret = add_merge_cache_extent(data_chunks, cur_off, cur_len);
+		if (ret < 0)
+			goto out;
+		cur_off += cur_len;
+	}
+	/*
+	 * remove reserved ranges, so we won't ever bother relocating an old
+	 * filesystem extent to other place.
+	 */
+	ret = wipe_reserved_ranges(data_chunks, min_stripe_size, 1);
+	if (ret < 0)
+		goto out;
+
+	cur_off = 0;
+	/*
+	 * Calculate free space
+	 * Always round up the start bytenr, to avoid metadata extent corss
+	 * stripe boundary, as later mkfs_convert() won't have all the extent
+	 * allocation check
+	 */
+	for (cache = first_cache_extent(data_chunks); cache;
+	     cache = next_cache_extent(cache)) {
+		if (cache->start < cur_off)
+			continue;
+		if (cache->start > cur_off) {
+			u64 insert_start;
+			u64 len;
+
+			len = cache->start - round_up(cur_off,
+						      BTRFS_STRIPE_LEN);
+			insert_start = round_up(cur_off, BTRFS_STRIPE_LEN);
+
+			ret = add_merge_cache_extent(free, insert_start, len);
+			if (ret < 0)
+				goto out;
+		}
+		cur_off = cache->start + cache->size;
+	}
+	/* Don't forget the last range */
+	if (cctx->total_bytes > cur_off) {
+		u64 len = cctx->total_bytes - cur_off;
+		u64 insert_start;
+
+		insert_start = round_up(cur_off, BTRFS_STRIPE_LEN);
+
+		ret = add_merge_cache_extent(free, insert_start, len);
+		if (ret < 0)
+			goto out;
+	}
+
+	/* Remove reserved bytes */
+	ret = wipe_reserved_ranges(free, min_stripe_size, 0);
+out:
+	return ret;
+}
 /*
- * Read used space
+ * Read used space, and since we have the used space,
+ * calcuate data_chunks and free for later mkfs
  */
 static int convert_read_used_space(struct btrfs_convert_context *cctx)
 {
-	return cctx->convert_ops->read_used_space(cctx);
+	int ret;
+
+	ret = cctx->convert_ops->read_used_space(cctx);
+	if (ret)
+		return ret;
+
+	ret = calculate_available_space(cctx);
+	return ret;
 }
 
 static int do_convert(const char *devname, int datacsum, int packing, int noxattr,
-- 
2.7.0

[PATCH v3 04/22] btrfs-progs: utils: Introduce new function for convert

[Qu Wenruo]

Introduce new function make_convert_btrfs() for convert.

This new function will have the following features:
1) Alloc temporary sb/metadata/system chunk, avoiding old used data
2) More structurizd functions
   No more over 1000 lines function, better function split and code
   reuse

This will finally replace current make_btrfs(), but now only used for
convert.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 24 +--------------
 mkfs.c          |  2 +-
 utils.c         | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 utils.h         | 26 ++++++++++++++++-
 4 files changed, 117 insertions(+), 26 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 8026907..1573e41 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -96,28 +96,6 @@ struct btrfs_convert_operations {
 	void (*close_fs)(struct btrfs_convert_context *cctx);
 };
 
-struct btrfs_convert_context {
-	u32 blocksize;
-	u32 first_data_block;
-	u32 block_count;
-	u32 inodes_count;
-	u32 free_inodes_count;
-	u64 total_bytes;
-	char *volume_name;
-	const struct btrfs_convert_operations *convert_ops;
-
-	/* The accurate used space of old filesystem */
-	struct cache_tree used;
-
-	/* Batched ranges which must be covered by data chunks */
-	struct cache_tree data_chunks;
-
-	/* Free space which is not covered by data_chunks */
-	struct cache_tree free;
-
-	void *fs_data;
-};
-
 static void init_convert_context(struct btrfs_convert_context *cctx)
 {
 	cache_tree_init(&cctx->used);
@@ -2807,7 +2785,7 @@ static int do_convert(const char *devname, int datacsum, int packing, int noxatt
 	mkfs_cfg.stripesize = blocksize;
 	mkfs_cfg.features = features;
 
-	ret = make_btrfs(fd, &mkfs_cfg);
+	ret = make_btrfs(fd, &mkfs_cfg, NULL);
 	if (ret) {
 		fprintf(stderr, "unable to create initial ctree: %s\n",
 			strerror(-ret));
diff --git a/mkfs.c b/mkfs.c
index ea58404..5b77324 100644
--- a/mkfs.c
+++ b/mkfs.c
@@ -1675,7 +1675,7 @@ int main(int ac, char **av)
 	mkfs_cfg.stripesize = stripesize;
 	mkfs_cfg.features = features;
 
-	ret = make_btrfs(fd, &mkfs_cfg);
+	ret = make_btrfs(fd, &mkfs_cfg, NULL);
 	if (ret) {
 		fprintf(stderr, "error during mkfs: %s\n", strerror(-ret));
 		exit(1);
diff --git a/utils.c b/utils.c
index 3df8b42..bc10293 100644
--- a/utils.c
+++ b/utils.c
@@ -179,9 +179,96 @@ int test_uuid_unique(char *fs_uuid)
 }
 
 /*
+ * Reserve space from free_tree.
+ * The algorithm is very simple, find the first cache_extent with enough space
+ * and allocate from its beginning.
+ */
+static int reserve_free_space(struct cache_tree *free_tree, u64 len,
+			      u64 *ret_start)
+{
+	struct cache_extent *cache;
+	int found = 0;
+
+	BUG_ON(!ret_start);
+	cache = first_cache_extent(free_tree);
+	while (cache) {
+		if (cache->size > len) {
+			found = 1;
+			*ret_start = cache->start;
+
+			cache->size -= len;
+			if (cache->size == 0) {
+				remove_cache_extent(free_tree, cache);
+				free(cache);
+			} else {
+				cache->start += len;
+			}
+			break;
+		}
+		cache = next_cache_extent(cache);
+	}
+	if (!found)
+		return -ENOSPC;
+	return 0;
+}
+
+/*
+ * Improved version of make_btrfs().
+ *
+ * This one will
+ * 1) Do chunk allocation to avoid used data
+ *    And after this function, extent type matches chunk type
+ * 2) Better structurized code
+ *    No super long hand written codes to initialized all tree blocks
+ *    Split into small blocks and reuse codes.
+ *    TODO: Reuse tree operation facilities by introducing new flags
+ */
+static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
+			      struct btrfs_convert_context *cctx)
+{
+	struct cache_tree *free = &cctx->free;
+	struct cache_tree *used = &cctx->used;
+	u64 sys_chunk_start;
+	u64 meta_chunk_start;
+	int ret;
+
+	/* Shouldn't happen */
+	BUG_ON(cache_tree_empty(used));
+
+	/*
+	 * reserve space for temporary superblock first
+	 * Here we allocate a little larger space, to keep later
+	 * free space will be STRIPE_LEN aligned
+	 */
+	ret = reserve_free_space(free, BTRFS_STRIPE_LEN,
+				 &cfg->super_bytenr);
+	if (ret < 0)
+		goto out;
+
+	/*
+	 * Then reserve system chunk space
+	 * TODO: Change system group size depending on cctx->total_bytes.
+	 * If using current 4M, it can only handle less than one TB for
+	 * worst case and then run out of sys space.
+	 */
+	ret = reserve_free_space(free, BTRFS_MKFS_SYSTEM_GROUP_SIZE,
+				 &sys_chunk_start);
+	if (ret < 0)
+		goto out;
+	ret = reserve_free_space(free, BTRFS_CONVERT_META_GROUP_SIZE,
+				 &meta_chunk_start);
+	if (ret < 0)
+		goto out;
+
+out:
+	return ret;
+}
+
+/*
  * @fs_uuid - if NULL, generates a UUID, returns back the new filesystem UUID
  */
-int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
+int make_btrfs(int fd, struct btrfs_mkfs_config *cfg,
+		struct btrfs_convert_context *cctx)
 {
 	struct btrfs_super_block super;
 	struct extent_buffer *buf;
@@ -206,6 +293,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 				 BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA);
 	u64 num_bytes;
 
+	if (cctx)
+		return make_convert_btrfs(fd, cfg, cctx);
 	buf = malloc(sizeof(*buf) + max(cfg->sectorsize, cfg->nodesize));
 	if (!buf)
 		return -ENOMEM;
diff --git a/utils.h b/utils.h
index d53357a..feccebd 100644
--- a/utils.h
+++ b/utils.h
@@ -43,6 +43,7 @@
 	| BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF			\
 	| BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA		\
 	| BTRFS_FEATURE_INCOMPAT_NO_HOLES)
+#define BTRFS_CONVERT_META_GROUP_SIZE (32 * 1024 * 1024)
 
 #define BTRFS_FEATURE_LIST_ALL		(1ULL << 63)
 
@@ -121,7 +122,30 @@ struct btrfs_mkfs_config {
 	u64 super_bytenr;
 };
 
-int make_btrfs(int fd, struct btrfs_mkfs_config *cfg);
+struct btrfs_convert_context {
+	u32 blocksize;
+	u32 first_data_block;
+	u32 block_count;
+	u32 inodes_count;
+	u32 free_inodes_count;
+	u64 total_bytes;
+	char *volume_name;
+	const struct btrfs_convert_operations *convert_ops;
+
+	/* The accurate used space of old filesystem */
+	struct cache_tree used;
+
+	/* Batched ranges which must be covered by data chunks */
+	struct cache_tree data_chunks;
+
+	/* Free space which is not covered by data_chunks */
+	struct cache_tree free;
+
+	void *fs_data;
+};
+
+int make_btrfs(int fd, struct btrfs_mkfs_config *cfg,
+		struct btrfs_convert_context *cctx);
 int btrfs_make_root_dir(struct btrfs_trans_handle *trans,
 			struct btrfs_root *root, u64 objectid);
 int btrfs_prepare_device(int fd, char *file, int zero_end, u64 *block_count_ret,
-- 
2.7.0

[PATCH v3 05/22] btrfs-progs: Introduce function to setup temporary superblock

[Qu Wenruo]

Introduce a new function, setup_temp_super(), to setup temporary super
for make_btrfs_v2().

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 utils.c | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)

diff --git a/utils.c b/utils.c
index bc10293..ed5476d 100644
--- a/utils.c
+++ b/utils.c
@@ -212,6 +212,98 @@ static int reserve_free_space(struct cache_tree *free_tree, u64 len,
 	return 0;
 }
 
+static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
+				   u64 sb_bytenr)
+{
+	u32 crc = ~(u32)0;
+	int ret;
+
+	crc = btrfs_csum_data(NULL, (char *)sb + BTRFS_CSUM_SIZE, crc,
+			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+	btrfs_csum_final(crc, (char *)&sb->csum[0]);
+	ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
+	if (ret < BTRFS_SUPER_INFO_SIZE)
+		ret = (ret < 0 ? -errno : -EIO);
+	else
+		ret = 0;
+	return ret;
+}
+
+/*
+ * Setup temporary superblock at cfg->super_bynter
+ * Needed info are extracted from cfg, and root_bytenr, chunk_bytenr
+ *
+ * For now sys chunk array will be empty and dev_item is empty
+ * too.
+ * They will be re-initialized at temp chunk tree setup.
+ */
+static int setup_temp_super(int fd, struct btrfs_mkfs_config *cfg,
+			    u64 root_bytenr, u64 chunk_bytenr)
+{
+	unsigned char chunk_uuid[BTRFS_UUID_SIZE];
+	char super_buf[BTRFS_SUPER_INFO_SIZE];
+	struct btrfs_super_block *super = (struct btrfs_super_block *)super_buf;
+	int ret;
+
+	/*
+	 * We rely on cfg->chunk_uuid and cfg->fs_uuid to pass uuid
+	 * for other functions.
+	 * Caller must allocation space for them
+	 */
+	BUG_ON(!cfg->chunk_uuid || !cfg->fs_uuid);
+	memset(super_buf, 0, BTRFS_SUPER_INFO_SIZE);
+	cfg->num_bytes = round_down(cfg->num_bytes, cfg->sectorsize);
+
+	if (cfg->fs_uuid && *cfg->fs_uuid) {
+		if (uuid_parse(cfg->fs_uuid, super->fsid) != 0) {
+			error("cound not parse UUID: %s", cfg->fs_uuid);
+			ret = -EINVAL;
+			goto out;
+		}
+		if (!test_uuid_unique(cfg->fs_uuid)) {
+			error("non-unique UUID: %s", cfg->fs_uuid);
+			ret = -EINVAL;
+			goto out;
+		}
+	} else {
+		uuid_generate(super->fsid);
+		uuid_unparse(super->fsid, cfg->fs_uuid);
+	}
+	uuid_generate(chunk_uuid);
+	uuid_unparse(chunk_uuid, cfg->chunk_uuid);
+
+	btrfs_set_super_bytenr(super, cfg->super_bytenr);
+	btrfs_set_super_num_devices(super, 1);
+	btrfs_set_super_magic(super, BTRFS_MAGIC);
+	btrfs_set_super_generation(super, 1);
+	btrfs_set_super_root(super, root_bytenr);
+	btrfs_set_super_chunk_root(super, chunk_bytenr);
+	btrfs_set_super_total_bytes(super, cfg->num_bytes);
+	/*
+	 * Temporary btrfs will only has 6 tree roots:
+	 * chunk tree, root tree, extent_tree, device tree, fs tree
+	 * and csum tree.
+	 */
+	btrfs_set_super_bytes_used(super, 6 * cfg->nodesize);
+	btrfs_set_super_sectorsize(super, cfg->sectorsize);
+	btrfs_set_super_leafsize(super, cfg->nodesize);
+	btrfs_set_super_nodesize(super, cfg->nodesize);
+	btrfs_set_super_stripesize(super, cfg->stripesize);
+	btrfs_set_super_csum_type(super, BTRFS_CSUM_TYPE_CRC32);
+	btrfs_set_super_chunk_root(super, chunk_bytenr);
+	btrfs_set_super_cache_generation(super, -1);
+	btrfs_set_super_incompat_flags(super, cfg->features);
+	if (cfg->label)
+		strncpy(super->label, cfg->label, BTRFS_LABEL_SIZE - 1);
+
+	/* Sys chunk array will be re-initialized at chunk tree init time */
+	super->sys_chunk_array_size = 0;
+
+	ret = write_temp_super(fd, super, cfg->super_bytenr);
+out:
+	return ret;
+}
+
 /*
  * Improved version of make_btrfs().
  *
@@ -230,6 +322,10 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
 	struct cache_tree *used = &cctx->used;
 	u64 sys_chunk_start;
 	u64 meta_chunk_start;
+	/* chunk tree bytenr, in system chunk */
+	u64 chunk_bytenr;
+	/* metadata trees bytenr, in metadata chunk */
+	u64 root_bytenr;
 	int ret;
 
 	/* Shouldn't happen */
@@ -260,6 +356,27 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
 	if (ret < 0)
 		goto out;
 
+	/*
+	 * Inside the allocate metadata chunk, its layout will be:
+	 *  | offset		| contents	|
+	 *  -------------------------------------
+	 *  | +0		| tree root	|
+	 *  | +nodesize		| extent root	|
+	 *  | +nodesize * 2	| device root	|
+	 *  | +nodesize * 3	| fs tree	|
+	 *  | +nodesize * 4	| csum tree	|
+	 *  -------------------------------------
+	 * Inside the allocated system chunk, its layout will be:
+	 *  | offset		| contents	|
+	 *  -------------------------------------
+	 *  | +0		| chunk root	|
+	 *  -------------------------------------
+	 */
+	chunk_bytenr = sys_chunk_start;
+	root_bytenr = meta_chunk_start;
+	ret = setup_temp_super(fd, cfg, root_bytenr, chunk_bytenr);
+	if (ret < 0)
+		goto out;
 out:
 	return ret;
 }
-- 
2.7.0

[PATCH v3 06/22] btrfs-progs: Introduce function to setup temporary tree root

[Qu Wenruo]

Introduce new function, setup_temp_tree_root(), to initialize temporary
tree root for make_btrfs_v2().

The new function will setup tree root at metadata chunk and ensure data
won't be written into metadata chunk.

Also, new make_btrfs_v2() will have a much better code structure than
old make_btrfs().

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 utils.c | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 148 insertions(+)

diff --git a/utils.c b/utils.c
index ed5476d..0b5df7e 100644
--- a/utils.c
+++ b/utils.c
@@ -305,6 +305,136 @@ out:
 }
 
 /*
+ * Setup an extent buffer for tree block.
+ */
+static int setup_temp_extent_buffer(struct extent_buffer *buf,
+				    struct btrfs_mkfs_config *cfg,
+				    u64 bytenr, u64 owner)
+{
+	unsigned char fsid[BTRFS_FSID_SIZE];
+	unsigned char chunk_uuid[BTRFS_UUID_SIZE];
+	int ret;
+
+	/* We rely on cfg->fs_uuid and chunk_uuid to fsid and chunk uuid */
+	BUG_ON(!cfg->fs_uuid || !cfg->chunk_uuid);
+	ret = uuid_parse(cfg->fs_uuid, fsid);
+	if (ret)
+		return -EINVAL;
+	ret = uuid_parse(cfg->chunk_uuid, chunk_uuid);
+	if (ret)
+		return -EINVAL;
+
+	memset(buf->data, 0, cfg->nodesize);
+	buf->len = cfg->nodesize;
+	btrfs_set_header_bytenr(buf, bytenr);
+	btrfs_set_header_generation(buf, 1);
+	btrfs_set_header_backref_rev(buf, BTRFS_MIXED_BACKREF_REV);
+	btrfs_set_header_owner(buf, owner);
+	btrfs_set_header_flags(buf, BTRFS_HEADER_FLAG_WRITTEN);
+	write_extent_buffer(buf, chunk_uuid, btrfs_header_chunk_tree_uuid(buf),
+			    BTRFS_UUID_SIZE);
+	write_extent_buffer(buf, fsid, btrfs_header_fsid(), BTRFS_FSID_SIZE);
+	return 0;
+}
+
+static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
+					   u64 bytenr)
+{
+	int ret;
+
+	csum_tree_block_size(buf, BTRFS_CRC32_SIZE, 0);
+
+	/* Temporary extent buffer is always mapped 1:1 on disk */
+	ret = pwrite(fd, buf->data, buf->len, bytenr);
+	if (ret < buf->len)
+		ret = (ret < 0 ? ret : -EIO);
+	else
+		ret = 0;
+	return ret;
+}
+
+/*
+ * Insert a root item for temporary tree root
+ *
+ * Only used in make_btrfs_v2().
+ */
+static void insert_temp_root_item(struct extent_buffer *buf,
+				  struct btrfs_mkfs_config *cfg,
+				  int *slot, u32 *itemoff, u64 objectid,
+				  u64 bytenr)
+{
+	struct btrfs_root_item root_item;
+	struct btrfs_inode_item *inode_item;
+	struct btrfs_disk_key disk_key;
+
+	btrfs_set_header_nritems(buf, *slot + 1);
+	(*itemoff) -= sizeof(root_item);
+	memset(&root_item, 0, sizeof(root_item));
+	inode_item = &root_item.inode;
+	btrfs_set_stack_inode_generation(inode_item, 1);
+	btrfs_set_stack_inode_size(inode_item, 3);
+	btrfs_set_stack_inode_nlink(inode_item, 1);
+	btrfs_set_stack_inode_nbytes(inode_item, cfg->nodesize);
+	btrfs_set_stack_inode_mode(inode_item, S_IFDIR | 0755);
+	btrfs_set_root_refs(&root_item, 1);
+	btrfs_set_root_used(&root_item, cfg->nodesize);
+	btrfs_set_root_generation(&root_item, 1);
+	btrfs_set_root_bytenr(&root_item, bytenr);
+
+	memset(&disk_key, 0, sizeof(disk_key));
+	btrfs_set_disk_key_type(&disk_key, BTRFS_ROOT_ITEM_KEY);
+	btrfs_set_disk_key_objectid(&disk_key, objectid);
+	btrfs_set_disk_key_offset(&disk_key, 0);
+
+	btrfs_set_item_key(buf, &disk_key, *slot);
+	btrfs_set_item_offset(buf, btrfs_item_nr(*slot), *itemoff);
+	btrfs_set_item_size(buf, btrfs_item_nr(*slot), sizeof(root_item));
+	write_extent_buffer(buf, &root_item,
+			    btrfs_item_ptr_offset(buf, *slot),
+			    sizeof(root_item));
+	(*slot)++;
+}
+
+static int setup_temp_root_tree(int fd, struct btrfs_mkfs_config *cfg,
+				u64 root_bytenr, u64 extent_bytenr,
+				u64 dev_bytenr, u64 fs_bytenr, u64 csum_bytenr)
+{
+	struct extent_buffer *buf = NULL;
+	u32 itemoff = __BTRFS_LEAF_DATA_SIZE(cfg->nodesize);
+	int slot = 0;
+	int ret;
+
+	/*
+	 * Provided bytenr must in ascending order, or tree root will have a
+	 * bad key order.
+	 */
+	BUG_ON(!(root_bytenr < extent_bytenr && extent_bytenr < dev_bytenr &&
+		 dev_bytenr < fs_bytenr && fs_bytenr < csum_bytenr));
+	buf = malloc(sizeof(*buf) + cfg->nodesize);
+	if (!buf)
+		return -ENOMEM;
+
+	ret = setup_temp_extent_buffer(buf, cfg, root_bytenr,
+				       BTRFS_ROOT_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+
+	insert_temp_root_item(buf, cfg, &slot, &itemoff,
+			      BTRFS_EXTENT_TREE_OBJECTID, extent_bytenr);
+	insert_temp_root_item(buf, cfg, &slot, &itemoff,
+			      BTRFS_DEV_TREE_OBJECTID, dev_bytenr);
+	insert_temp_root_item(buf, cfg, &slot, &itemoff,
+			      BTRFS_FS_TREE_OBJECTID, fs_bytenr);
+	insert_temp_root_item(buf, cfg, &slot, &itemoff,
+			      BTRFS_CSUM_TREE_OBJECTID, csum_bytenr);
+
+	ret = write_temp_extent_buffer(fd, buf, root_bytenr);
+out:
+	free(buf);
+	return ret;
+}
+
+/*
  * Improved version of make_btrfs().
  *
  * This one will
@@ -326,6 +456,11 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
 	u64 chunk_bytenr;
 	/* metadata trees bytenr, in metadata chunk */
 	u64 root_bytenr;
+	u64 extent_bytenr;
+	u64 dev_bytenr;
+	u64 fs_bytenr;
+	u64 csum_bytenr;
+	/* Above bytenrs are just for better readablity */
 	int ret;
 
 	/* Shouldn't happen */
@@ -357,6 +492,8 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
 		goto out;
 
 	/*
+	 * Allocated meta/sys chunks will be mapped 1:1 with device offset.
+	 *
 	 * Inside the allocate metadata chunk, its layout will be:
 	 *  | offset		| contents	|
 	 *  -------------------------------------
@@ -374,9 +511,20 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
 	 */
 	chunk_bytenr = sys_chunk_start;
 	root_bytenr = meta_chunk_start;
+	extent_bytenr = meta_chunk_start + cfg->nodesize;
+	dev_bytenr = meta_chunk_start + cfg->nodesize * 2;
+	fs_bytenr = meta_chunk_start + cfg->nodesize * 3;
+	csum_bytenr = meta_chunk_start + cfg->nodesize * 4;
+
 	ret = setup_temp_super(fd, cfg, root_bytenr, chunk_bytenr);
 	if (ret < 0)
 		goto out;
+
+	ret = setup_temp_root_tree(fd, cfg, root_bytenr, extent_bytenr,
+				   dev_bytenr, fs_bytenr, csum_bytenr);
+	if (ret < 0)
+		goto out;
+
 out:
 	return ret;
 }
-- 
2.7.0

[PATCH v3 07/22] btrfs-progs: Introduce function to setup temporary chunk root

[Qu Wenruo]

Introduce new function, setup_temp_chunk_root(), to initialize temporary
chunk root for make_btrfs_v2().

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 utils.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 171 insertions(+)

diff --git a/utils.c b/utils.c
index 0b5df7e..65bb11f 100644
--- a/utils.c
+++ b/utils.c
@@ -434,6 +434,173 @@ out:
 	return ret;
 }
 
+static int insert_temp_dev_item(int fd, struct extent_buffer *buf,
+				struct btrfs_mkfs_config *cfg,
+				int *slot, u32 *itemoff)
+{
+	struct btrfs_disk_key disk_key;
+	struct btrfs_dev_item *dev_item;
+	char super_buf[BTRFS_SUPER_INFO_SIZE];
+	unsigned char dev_uuid[BTRFS_UUID_SIZE];
+	unsigned char fsid[BTRFS_FSID_SIZE];
+	struct btrfs_super_block *super = (struct btrfs_super_block *)super_buf;
+	int ret;
+
+	ret = pread(fd, super_buf, BTRFS_SUPER_INFO_SIZE, cfg->super_bytenr);
+	if (ret < BTRFS_SUPER_INFO_SIZE) {
+		ret = (ret < 0 ? -errno : -EIO);
+		goto out;
+	}
+
+	btrfs_set_header_nritems(buf, *slot + 1);
+	(*itemoff) -= sizeof(*dev_item);
+	/* setup device item 1, 0 is for replace case */
+	btrfs_set_disk_key_type(&disk_key, BTRFS_DEV_ITEM_KEY);
+	btrfs_set_disk_key_objectid(&disk_key, BTRFS_DEV_ITEMS_OBJECTID);
+	btrfs_set_disk_key_offset(&disk_key, 1);
+	btrfs_set_item_key(buf, &disk_key, *slot);
+	btrfs_set_item_offset(buf, btrfs_item_nr(*slot), *itemoff);
+	btrfs_set_item_size(buf, btrfs_item_nr(*slot), sizeof(*dev_item));
+
+	dev_item = btrfs_item_ptr(buf, *slot, struct btrfs_dev_item);
+	/* Generate device uuid */
+	uuid_generate(dev_uuid);
+	write_extent_buffer(buf, dev_uuid,
+			(unsigned long)btrfs_device_uuid(dev_item),
+			BTRFS_UUID_SIZE);
+	uuid_parse(cfg->fs_uuid, fsid);
+	write_extent_buffer(buf, fsid,
+			(unsigned long)btrfs_device_fsid(dev_item),
+			BTRFS_FSID_SIZE);
+	btrfs_set_device_id(buf, dev_item, 1);
+	btrfs_set_device_generation(buf, dev_item, 0);
+	btrfs_set_device_total_bytes(buf, dev_item, cfg->num_bytes);
+	/*
+	 * The number must match the initial SYSTEM and META chunk size
+	 */
+	btrfs_set_device_bytes_used(buf, dev_item,
+			BTRFS_MKFS_SYSTEM_GROUP_SIZE +
+			BTRFS_CONVERT_META_GROUP_SIZE);
+	btrfs_set_device_io_align(buf, dev_item, cfg->sectorsize);
+	btrfs_set_device_io_width(buf, dev_item, cfg->sectorsize);
+	btrfs_set_device_sector_size(buf, dev_item, cfg->sectorsize);
+	btrfs_set_device_type(buf, dev_item, 0);
+
+	/* Super dev_item is not complete, copy the complete one to sb */
+	read_extent_buffer(buf, &super->dev_item, (unsigned long)dev_item,
+			   sizeof(*dev_item));
+	ret = write_temp_super(fd, super, cfg->super_bytenr);
+	(*slot)++;
+out:
+	return ret;
+}
+
+static int insert_temp_chunk_item(int fd, struct extent_buffer *buf,
+				  struct btrfs_mkfs_config *cfg,
+				  int *slot, u32 *itemoff, u64 start, u64 len,
+				  u64 type)
+{
+	struct btrfs_chunk *chunk;
+	struct btrfs_disk_key disk_key;
+	char super_buf[BTRFS_SUPER_INFO_SIZE];
+	struct btrfs_super_block *sb = (struct btrfs_super_block *)super_buf;
+	int ret = 0;
+
+	ret = pread(fd, super_buf, BTRFS_SUPER_INFO_SIZE,
+		    cfg->super_bytenr);
+	if (ret < BTRFS_SUPER_INFO_SIZE) {
+		ret = (ret < 0 ? ret : -EIO);
+		return ret;
+	}
+
+	btrfs_set_header_nritems(buf, *slot + 1);
+	(*itemoff) -= btrfs_chunk_item_size(1);
+	btrfs_set_disk_key_type(&disk_key, BTRFS_CHUNK_ITEM_KEY);
+	btrfs_set_disk_key_objectid(&disk_key, BTRFS_FIRST_CHUNK_TREE_OBJECTID);
+	btrfs_set_disk_key_offset(&disk_key, start);
+	btrfs_set_item_key(buf, &disk_key, *slot);
+	btrfs_set_item_offset(buf, btrfs_item_nr(*slot), *itemoff);
+	btrfs_set_item_size(buf, btrfs_item_nr(*slot),
+			    btrfs_chunk_item_size(1));
+
+	chunk = btrfs_item_ptr(buf, *slot, struct btrfs_chunk);
+	btrfs_set_chunk_length(buf, chunk, len);
+	btrfs_set_chunk_owner(buf, chunk, BTRFS_EXTENT_TREE_OBJECTID);
+	btrfs_set_chunk_stripe_len(buf, chunk, 64 * 1024);
+	btrfs_set_chunk_type(buf, chunk, type);
+	btrfs_set_chunk_io_align(buf, chunk, cfg->sectorsize);
+	btrfs_set_chunk_io_width(buf, chunk, cfg->sectorsize);
+	btrfs_set_chunk_sector_size(buf, chunk, cfg->sectorsize);
+	btrfs_set_chunk_num_stripes(buf, chunk, 1);
+	/* TODO: Support DUP profile for system chunk */
+	btrfs_set_stripe_devid_nr(buf, chunk, 0, 1);
+	/* We are doing 1:1 mapping, so start is its dev offset */
+	btrfs_set_stripe_offset_nr(buf, chunk, 0, start);
+	write_extent_buffer(buf, &sb->dev_item.uuid,
+			    (unsigned long)btrfs_stripe_dev_uuid_nr(chunk, 0),
+			    BTRFS_UUID_SIZE);
+	(*slot)++;
+
+	/*
+	 * If it's system chunk, also copy it to super block.
+	 */
+	if (type & BTRFS_BLOCK_GROUP_SYSTEM) {
+		char *cur;
+
+		cur = (char *)sb->sys_chunk_array + sb->sys_chunk_array_size;
+		memcpy(cur, &disk_key, sizeof(disk_key));
+		cur += sizeof(disk_key);
+		read_extent_buffer(buf, cur, (unsigned long int)chunk,
+				   btrfs_chunk_item_size(1));
+		sb->sys_chunk_array_size += btrfs_chunk_item_size(1) +
+					    sizeof(disk_key);
+
+		ret = write_temp_super(fd, sb, cfg->super_bytenr);
+	}
+	return ret;
+}
+
+static int setup_temp_chunk_tree(int fd, struct btrfs_mkfs_config *cfg,
+				 u64 sys_chunk_start, u64 meta_chunk_start,
+				 u64 chunk_bytenr)
+{
+	struct extent_buffer *buf = NULL;
+	u32 itemoff = __BTRFS_LEAF_DATA_SIZE(cfg->nodesize);
+	int slot = 0;
+	int ret;
+
+	/* Must ensure SYS chunk starts before META chunk */
+	BUG_ON(meta_chunk_start < sys_chunk_start);
+	buf = malloc(sizeof(*buf) + cfg->nodesize);
+	if (!buf)
+		return -ENOMEM;
+	ret = setup_temp_extent_buffer(buf, cfg, chunk_bytenr,
+				       BTRFS_CHUNK_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+
+	ret = insert_temp_dev_item(fd, buf, cfg, &slot, &itemoff);
+	if (ret < 0)
+		goto out;
+	ret = insert_temp_chunk_item(fd, buf, cfg, &slot, &itemoff,
+				     sys_chunk_start,
+				     BTRFS_MKFS_SYSTEM_GROUP_SIZE,
+				     BTRFS_BLOCK_GROUP_SYSTEM);
+	if (ret < 0)
+		goto out;
+	ret = insert_temp_chunk_item(fd, buf, cfg, &slot, &itemoff,
+				     meta_chunk_start,
+				     BTRFS_CONVERT_META_GROUP_SIZE,
+				     BTRFS_BLOCK_GROUP_METADATA);
+	if (ret < 0)
+		goto out;
+	ret = write_temp_extent_buffer(fd, buf, chunk_bytenr);
+
+out:
+	free(buf);
+	return ret;
+}
+
 /*
  * Improved version of make_btrfs().
  *
@@ -524,6 +691,10 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
 				   dev_bytenr, fs_bytenr, csum_bytenr);
 	if (ret < 0)
 		goto out;
+	ret = setup_temp_chunk_tree(fd, cfg, sys_chunk_start, meta_chunk_start,
+				    chunk_bytenr);
+	if (ret < 0)
+		goto out;
 
 out:
 	return ret;
-- 
2.7.0

[PATCH v3 08/22] btrfs-progs: Introduce function to initialize device tree

[Qu Wenruo]

Introduce new function, setup_temp_device_tree(), to setup temporary
device tree for make_btrfs_v2().

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 utils.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/utils.c b/utils.c
index 65bb11f..4d29b1e 100644
--- a/utils.c
+++ b/utils.c
@@ -601,6 +601,59 @@ out:
 	return ret;
 }
 
+static void insert_temp_dev_extent(struct extent_buffer *buf,
+				   int *slot, u32 *itemoff, u64 start, u64 len)
+{
+	struct btrfs_dev_extent *dev_extent;
+	struct btrfs_disk_key disk_key;
+
+	btrfs_set_header_nritems(buf, *slot + 1);
+	(*itemoff) -= sizeof(*dev_extent);
+	btrfs_set_disk_key_type(&disk_key, BTRFS_DEV_EXTENT_KEY);
+	btrfs_set_disk_key_objectid(&disk_key, 1);
+	btrfs_set_disk_key_offset(&disk_key, start);
+	btrfs_set_item_key(buf, &disk_key, *slot);
+	btrfs_set_item_offset(buf, btrfs_item_nr(*slot), *itemoff);
+	btrfs_set_item_size(buf, btrfs_item_nr(*slot), sizeof(*dev_extent));
+
+	dev_extent = btrfs_item_ptr(buf, *slot, struct btrfs_dev_extent);
+	btrfs_set_dev_extent_chunk_objectid(buf, dev_extent,
+					    BTRFS_FIRST_CHUNK_TREE_OBJECTID);
+	btrfs_set_dev_extent_length(buf, dev_extent, len);
+	btrfs_set_dev_extent_chunk_offset(buf, dev_extent, start);
+	btrfs_set_dev_extent_chunk_tree(buf, dev_extent,
+					BTRFS_CHUNK_TREE_OBJECTID);
+	(*slot)++;
+}
+
+static int setup_temp_dev_tree(int fd, struct btrfs_mkfs_config *cfg,
+			       u64 sys_chunk_start, u64 meta_chunk_start,
+			       u64 dev_bytenr)
+{
+	struct extent_buffer *buf = NULL;
+	u32 itemoff = __BTRFS_LEAF_DATA_SIZE(cfg->nodesize);
+	int slot = 0;
+	int ret;
+
+	/* Must ensure SYS chunk starts before META chunk */
+	BUG_ON(meta_chunk_start < sys_chunk_start);
+	buf = malloc(sizeof(*buf) + cfg->nodesize);
+	if (!buf)
+		return -ENOMEM;
+	ret = setup_temp_extent_buffer(buf, cfg, dev_bytenr,
+				       BTRFS_DEV_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+	insert_temp_dev_extent(buf, &slot, &itemoff, sys_chunk_start,
+			       BTRFS_MKFS_SYSTEM_GROUP_SIZE);
+	insert_temp_dev_extent(buf, &slot, &itemoff, meta_chunk_start,
+			       BTRFS_CONVERT_META_GROUP_SIZE);
+	ret = write_temp_extent_buffer(fd, buf, dev_bytenr);
+out:
+	free(buf);
+	return ret;
+}
+
 /*
  * Improved version of make_btrfs().
  *
@@ -695,7 +748,8 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
 				    chunk_bytenr);
 	if (ret < 0)
 		goto out;
-
+	ret = setup_temp_dev_tree(fd, cfg, sys_chunk_start, meta_chunk_start,
+				  dev_bytenr);
 out:
 	return ret;
 }
-- 
2.7.0

[PATCH v3 09/22] btrfs-progs: Introduce function to initialize fs tree

[Qu Wenruo]

Introudce new function, setup_temp_fs_tree(), to setup temporary fs tree
for make_btrfs_v2().

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 utils.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/utils.c b/utils.c
index 4d29b1e..54085b0 100644
--- a/utils.c
+++ b/utils.c
@@ -654,6 +654,28 @@ out:
 	return ret;
 }
 
+static int setup_temp_fs_tree(int fd, struct btrfs_mkfs_config *cfg,
+			      u64 fs_bytenr)
+{
+	struct extent_buffer *buf = NULL;
+	int ret;
+
+	buf = malloc(sizeof(*buf) + cfg->nodesize);
+	if (!buf)
+		return -ENOMEM;
+	ret = setup_temp_extent_buffer(buf, cfg, fs_bytenr,
+				       BTRFS_FS_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+	/*
+	 * Temporary fs tree is completely empty.
+	 */
+	ret = write_temp_extent_buffer(fd, buf, fs_bytenr);
+out:
+	free(buf);
+	return ret;
+}
+
 /*
  * Improved version of make_btrfs().
  *
@@ -750,6 +772,10 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
 		goto out;
 	ret = setup_temp_dev_tree(fd, cfg, sys_chunk_start, meta_chunk_start,
 				  dev_bytenr);
+	if (ret < 0)
+		goto out;
+	ret = setup_temp_fs_tree(fd, cfg, fs_bytenr);
+
 out:
 	return ret;
 }
-- 
2.7.0

[PATCH v3 10/22] btrfs-progs: Introduce function to initialize csum tree

[Qu Wenruo]

Introduce new function, setup_temp_csum_tree(), to setup temporary
csum tree for make_btrfs_v2().

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 utils.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/utils.c b/utils.c
index 54085b0..51daad9 100644
--- a/utils.c
+++ b/utils.c
@@ -676,6 +676,28 @@ out:
 	return ret;
 }
 
+static int setup_temp_csum_tree(int fd, struct btrfs_mkfs_config *cfg,
+				u64 csum_bytenr)
+{
+	struct extent_buffer *buf = NULL;
+	int ret;
+
+	buf = malloc(sizeof(*buf) + cfg->nodesize);
+	if (!buf)
+		return -ENOMEM;
+	ret = setup_temp_extent_buffer(buf, cfg, csum_bytenr,
+				       BTRFS_CSUM_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+	/*
+	 * Temporary csum tree is completely empty.
+	 */
+	ret = write_temp_extent_buffer(fd, buf, csum_bytenr);
+out:
+	free(buf);
+	return ret;
+}
+
 /*
  * Improved version of make_btrfs().
  *
@@ -775,6 +797,9 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
 	if (ret < 0)
 		goto out;
 	ret = setup_temp_fs_tree(fd, cfg, fs_bytenr);
+	if (ret < 0)
+		goto out;
+	ret = setup_temp_csum_tree(fd, cfg, csum_bytenr);
 
 out:
 	return ret;
-- 
2.7.0

[PATCH v3 11/22] btrfs-progs: Introduce function to setup temporary extent tree

[Qu Wenruo]

Introduce new function, setup_temporary_extent_tree() to build a
temporary extent tree for make_btrfs_v2().

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 utils.c | 198 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 197 insertions(+), 1 deletion(-)

diff --git a/utils.c b/utils.c
index 51daad9..506d362 100644
--- a/utils.c
+++ b/utils.c
@@ -699,6 +699,194 @@ out:
 }
 
 /*
+ * Insert one temporary extent item.
+ *
+ * NOTE: if skinny_metadata is not enabled, this function must be called
+ * after all other trees are initialized.
+ * Or fs without skinny-metadata will be screwed up.
+ */
+static int insert_temp_extent_item(int fd, struct extent_buffer *buf,
+				   struct btrfs_mkfs_config *cfg,
+				   int *slot, u32 *itemoff, u64 bytenr,
+				   u64 ref_root)
+{
+	struct extent_buffer *tmp;
+	struct btrfs_extent_item *ei;
+	struct btrfs_extent_inline_ref *iref;
+	struct btrfs_disk_key disk_key;
+	struct btrfs_disk_key tree_info_key;
+	struct btrfs_tree_block_info *info;
+	int itemsize;
+	int skinny_metadata = cfg->features &
+			      BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA;
+	int ret;
+
+	if (skinny_metadata)
+		itemsize = sizeof(*ei) + sizeof(*iref);
+	else
+		itemsize = sizeof(*ei) + sizeof(*iref) +
+			   sizeof(struct btrfs_tree_block_info);
+
+	btrfs_set_header_nritems(buf, *slot + 1);
+	*(itemoff) -= itemsize;
+
+	if (skinny_metadata) {
+		btrfs_set_disk_key_type(&disk_key, BTRFS_METADATA_ITEM_KEY);
+		btrfs_set_disk_key_offset(&disk_key, 0);
+	} else {
+		btrfs_set_disk_key_type(&disk_key, BTRFS_EXTENT_ITEM_KEY);
+		btrfs_set_disk_key_offset(&disk_key, cfg->nodesize);
+	}
+	btrfs_set_disk_key_objectid(&disk_key, bytenr);
+
+	btrfs_set_item_key(buf, &disk_key, *slot);
+	btrfs_set_item_offset(buf, btrfs_item_nr(*slot), *itemoff);
+	btrfs_set_item_size(buf, btrfs_item_nr(*slot), itemsize);
+
+	ei = btrfs_item_ptr(buf, *slot, struct btrfs_extent_item);
+	btrfs_set_extent_refs(buf, ei, 1);
+	btrfs_set_extent_generation(buf, ei, 1);
+	btrfs_set_extent_flags(buf, ei, BTRFS_EXTENT_FLAG_TREE_BLOCK);
+
+	if (skinny_metadata) {
+		iref = (struct btrfs_extent_inline_ref *)(ei + 1);
+	} else {
+		info = (struct btrfs_tree_block_info *)(ei + 1);
+		iref = (struct btrfs_extent_inline_ref *)(info + 1);
+	}
+	btrfs_set_extent_inline_ref_type(buf, iref,
+					 BTRFS_TREE_BLOCK_REF_KEY);
+	btrfs_set_extent_inline_ref_offset(buf, iref, ref_root);
+
+	(*slot)++;
+	if (skinny_metadata)
+		return 0;
+
+	/*
+	 * Lastly, check the tree block key by read the tree block
+	 * Since we do 1:1 mapping for convert case, we can directly
+	 * read the bytenr from disk
+	 */
+	tmp = malloc(sizeof(*tmp) + cfg->nodesize);
+	if (!tmp)
+		return -ENOMEM;
+	ret = setup_temp_extent_buffer(tmp, cfg, bytenr, ref_root);
+	if (ret < 0)
+		goto out;
+	ret = pread(fd, tmp->data, cfg->nodesize, bytenr);
+	if (ret < cfg->nodesize) {
+		ret = (ret < 0 ? -errno : -EIO);
+		goto out;
+	}
+	if (btrfs_header_nritems(tmp) == 0) {
+		btrfs_set_disk_key_type(&tree_info_key, 0);
+		btrfs_set_disk_key_objectid(&tree_info_key, 0);
+		btrfs_set_disk_key_offset(&tree_info_key, 0);
+	} else {
+		btrfs_item_key(tmp, &tree_info_key, 0);
+	}
+	btrfs_set_tree_block_key(buf, info, &tree_info_key);
+
+out:
+	free(tmp);
+	return ret;
+}
+
+static void insert_temp_block_group(struct extent_buffer *buf,
+				   struct btrfs_mkfs_config *cfg,
+				   int *slot, u32 *itemoff,
+				   u64 bytenr, u64 len, u64 used, u64 flag)
+{
+	struct btrfs_block_group_item bgi;
+	struct btrfs_disk_key disk_key;
+
+	btrfs_set_header_nritems(buf, *slot + 1);
+	(*itemoff) -= sizeof(bgi);
+	btrfs_set_disk_key_type(&disk_key, BTRFS_BLOCK_GROUP_ITEM_KEY);
+	btrfs_set_disk_key_objectid(&disk_key, bytenr);
+	btrfs_set_disk_key_offset(&disk_key, len);
+	btrfs_set_item_key(buf, &disk_key, *slot);
+	btrfs_set_item_offset(buf, btrfs_item_nr(*slot), *itemoff);
+	btrfs_set_item_size(buf, btrfs_item_nr(*slot), sizeof(bgi));
+
+	btrfs_set_block_group_flags(&bgi, flag);
+	btrfs_set_block_group_used(&bgi, used);
+	btrfs_set_block_group_chunk_objectid(&bgi,
+			BTRFS_FIRST_CHUNK_TREE_OBJECTID);
+	write_extent_buffer(buf, &bgi, btrfs_item_ptr_offset(buf, *slot),
+			    sizeof(bgi));
+	(*slot)++;
+}
+
+static int setup_temp_extent_tree(int fd, struct btrfs_mkfs_config *cfg,
+				  u64 chunk_bytenr, u64 root_bytenr,
+				  u64 extent_bytenr, u64 dev_bytenr,
+				  u64 fs_bytenr, u64 csum_bytenr)
+{
+	struct extent_buffer *buf = NULL;
+	u32 itemoff = __BTRFS_LEAF_DATA_SIZE(cfg->nodesize);
+	int slot = 0;
+	int ret;
+
+	/*
+	 * We must ensure provided bytenr are in ascending order,
+	 * or extent tree key order will be broken.
+	 */
+	BUG_ON(!(chunk_bytenr < root_bytenr && root_bytenr < extent_bytenr &&
+		 extent_bytenr < dev_bytenr && dev_bytenr < fs_bytenr &&
+		 fs_bytenr < csum_bytenr));
+	buf = malloc(sizeof(*buf) + cfg->nodesize);
+	if (!buf)
+		return -ENOMEM;
+
+	ret = setup_temp_extent_buffer(buf, cfg, extent_bytenr,
+				       BTRFS_EXTENT_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+
+	ret = insert_temp_extent_item(fd, buf, cfg, &slot, &itemoff,
+			chunk_bytenr, BTRFS_CHUNK_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+
+	insert_temp_block_group(buf, cfg, &slot, &itemoff, chunk_bytenr,
+			BTRFS_MKFS_SYSTEM_GROUP_SIZE, cfg->nodesize,
+			BTRFS_BLOCK_GROUP_SYSTEM);
+
+	ret = insert_temp_extent_item(fd, buf, cfg, &slot, &itemoff,
+			root_bytenr, BTRFS_ROOT_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+
+	/* 5 tree block used, root, extent, dev, fs and csum*/
+	insert_temp_block_group(buf, cfg, &slot, &itemoff, root_bytenr,
+			BTRFS_CONVERT_META_GROUP_SIZE, cfg->nodesize * 5,
+			BTRFS_BLOCK_GROUP_METADATA);
+
+	ret = insert_temp_extent_item(fd, buf, cfg, &slot, &itemoff,
+			extent_bytenr, BTRFS_EXTENT_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+	ret = insert_temp_extent_item(fd, buf, cfg, &slot, &itemoff,
+			dev_bytenr, BTRFS_DEV_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+	ret = insert_temp_extent_item(fd, buf, cfg, &slot, &itemoff,
+			fs_bytenr, BTRFS_FS_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+	ret = insert_temp_extent_item(fd, buf, cfg, &slot, &itemoff,
+			csum_bytenr, BTRFS_CSUM_TREE_OBJECTID);
+	if (ret < 0)
+		goto out;
+
+	ret = write_temp_extent_buffer(fd, buf, extent_bytenr);
+out:
+	free(buf);
+	return ret;
+}
+
+/*
  * Improved version of make_btrfs().
  *
  * This one will
@@ -800,7 +988,15 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
 	if (ret < 0)
 		goto out;
 	ret = setup_temp_csum_tree(fd, cfg, csum_bytenr);
-
+	if (ret < 0)
+		goto out;
+	/*
+	 * Setup extent tree lastly, since it may need to read tree block key
+	 * for non-skinny metadata case.
+	 */
+	ret = setup_temp_extent_tree(fd, cfg, chunk_bytenr, root_bytenr,
+				     extent_bytenr, dev_bytenr, fs_bytenr,
+				     csum_bytenr);
 out:
 	return ret;
 }
-- 
2.7.0

[PATCH v3 12/22] btrfs-progs: Introduce function to create convert data chunks

[Qu Wenruo]

Introduce new function, make_convert_data_chunks(), to build up data
chunks for convert.

It will call a modified verion of btrfs_alloc_data_chunk() to force
data chunks to cover all known ext* data.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 mkfs.c          |  2 +-
 volumes.c       | 46 ++++++++++++++++++++++++++++++++++------------
 volumes.h       |  2 +-
 4 files changed, 86 insertions(+), 14 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 1573e41..102953a 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -1863,6 +1863,56 @@ static int create_subvol(struct btrfs_trans_handle *trans,
 	return 0;
 }
 
+/*
+ * New make_btrfs_v2() has handle system and meta chunks quite well.
+ * So only need to add remaining data chunks.
+ */
+static int make_convert_data_block_groups(struct btrfs_trans_handle *trans,
+					  struct btrfs_fs_info *fs_info,
+					  struct btrfs_mkfs_config *cfg,
+					  struct btrfs_convert_context *cctx)
+{
+	struct btrfs_root *extent_root = fs_info->extent_root;
+	struct cache_tree *data_chunks = &cctx->data_chunks;
+	struct cache_extent *cache;
+	u64 max_chunk_size;
+	int ret = 0;
+
+	/*
+	 * Don't create data chunk over 10% of the convert device
+	 * And for single chunk, don't create chunk larger than 1G.
+	 */
+	max_chunk_size = cfg->num_bytes / 10;
+	max_chunk_size = min((u64)(1024 * 1024 * 1024), max_chunk_size);
+	max_chunk_size = round_down(max_chunk_size, extent_root->sectorsize);
+
+	for (cache = first_cache_extent(data_chunks); cache;
+	     cache = next_cache_extent(cache)) {
+		u64 cur = cache->start;
+
+		while (cur < cache->start + cache->size) {
+			u64 len;
+			u64 cur_backup = cur;
+
+			len = min(max_chunk_size,
+				  cache->start + cache->size - cur);
+			ret = btrfs_alloc_data_chunk(trans, extent_root,
+					&cur_backup, len,
+					BTRFS_BLOCK_GROUP_DATA, 1);
+			if (ret < 0)
+				break;
+			ret = btrfs_make_block_group(trans, extent_root, 0,
+					BTRFS_BLOCK_GROUP_DATA,
+					BTRFS_FIRST_CHUNK_TREE_OBJECTID,
+					cur, len);
+			if (ret < 0)
+				break;
+			cur += len;
+		}
+	}
+	return ret;
+}
+
 static int init_btrfs(struct btrfs_root *root)
 {
 	int ret;
diff --git a/mkfs.c b/mkfs.c
index 5b77324..2d690cb 100644
--- a/mkfs.c
+++ b/mkfs.c
@@ -971,7 +971,7 @@ static int create_chunks(struct btrfs_trans_handle *trans,
 		size_of_data = minimum_data_chunk_size;
 
 	ret = btrfs_alloc_data_chunk(trans, root->fs_info->extent_root,
-				     &chunk_start, size_of_data, data_type);
+				     &chunk_start, size_of_data, data_type, 0);
 	BUG_ON(ret);
 	ret = btrfs_make_block_group(trans, root->fs_info->extent_root, 0,
 				     data_type, BTRFS_FIRST_CHUNK_TREE_OBJECTID,
diff --git a/volumes.c b/volumes.c
index a94be0e..7ce92a0 100644
--- a/volumes.c
+++ b/volumes.c
@@ -399,7 +399,7 @@ static int btrfs_alloc_dev_extent(struct btrfs_trans_handle *trans,
 				  struct btrfs_device *device,
 				  u64 chunk_tree, u64 chunk_objectid,
 				  u64 chunk_offset,
-				  u64 num_bytes, u64 *start)
+				  u64 num_bytes, u64 *start, int convert)
 {
 	int ret;
 	struct btrfs_path *path;
@@ -412,9 +412,15 @@ static int btrfs_alloc_dev_extent(struct btrfs_trans_handle *trans,
 	if (!path)
 		return -ENOMEM;
 
-	ret = find_free_dev_extent(trans, device, path, num_bytes, start);
-	if (ret) {
-		goto err;
+	/*
+	 * For convert case, just skip search free dev_extent, as caller
+	 * is responsible to ensure it's free.
+	 */
+	if (!convert) {
+		ret = find_free_dev_extent(trans, device, path, num_bytes,
+					   start);
+		if (ret)
+			goto err;
 	}
 
 	key.objectid = device->devid;
@@ -973,7 +979,7 @@ again:
 		ret = btrfs_alloc_dev_extent(trans, device,
 			     info->chunk_root->root_key.objectid,
 			     BTRFS_FIRST_CHUNK_TREE_OBJECTID, key.offset,
-			     calc_size, &dev_offset);
+			     calc_size, &dev_offset, 0);
 		BUG_ON(ret);
 
 		device->bytes_used += calc_size;
@@ -1029,9 +1035,17 @@ again:
 	return ret;
 }
 
+/*
+ * Alloc a DATA chunk with SINGLE profile.
+ *
+ * If 'convert' is set, it will alloc a chunk with 1:1 mapping
+ * (btrfs logical bytenr == on-disk bytenr)
+ * For that case, caller must ensure the chunk and dev_extent is not
+ * occupied.
+ */
 int btrfs_alloc_data_chunk(struct btrfs_trans_handle *trans,
 			   struct btrfs_root *extent_root, u64 *start,
-			   u64 num_bytes, u64 type)
+			   u64 num_bytes, u64 type, int convert)
 {
 	u64 dev_offset;
 	struct btrfs_fs_info *info = extent_root->fs_info;
@@ -1052,10 +1066,17 @@ int btrfs_alloc_data_chunk(struct btrfs_trans_handle *trans,
 
 	key.objectid = BTRFS_FIRST_CHUNK_TREE_OBJECTID;
 	key.type = BTRFS_CHUNK_ITEM_KEY;
-	ret = find_next_chunk(chunk_root, BTRFS_FIRST_CHUNK_TREE_OBJECTID,
-			      &key.offset);
-	if (ret)
-		return ret;
+	if (convert) {
+		BUG_ON(*start != round_down(*start, extent_root->sectorsize));
+		key.offset = *start;
+		dev_offset = *start;
+	} else {
+		ret = find_next_chunk(chunk_root,
+				      BTRFS_FIRST_CHUNK_TREE_OBJECTID,
+				      &key.offset);
+		if (ret)
+			return ret;
+	}
 
 	chunk = kmalloc(btrfs_chunk_item_size(num_stripes), GFP_NOFS);
 	if (!chunk)
@@ -1080,7 +1101,7 @@ int btrfs_alloc_data_chunk(struct btrfs_trans_handle *trans,
 		ret = btrfs_alloc_dev_extent(trans, device,
 			     info->chunk_root->root_key.objectid,
 			     BTRFS_FIRST_CHUNK_TREE_OBJECTID, key.offset,
-			     calc_size, &dev_offset);
+			     calc_size, &dev_offset, convert);
 		BUG_ON(ret);
 
 		device->bytes_used += calc_size;
@@ -1117,7 +1138,8 @@ int btrfs_alloc_data_chunk(struct btrfs_trans_handle *trans,
 	ret = btrfs_insert_item(trans, chunk_root, &key, chunk,
 				btrfs_chunk_item_size(num_stripes));
 	BUG_ON(ret);
-	*start = key.offset;
+	if (!convert)
+		*start = key.offset;
 
 	map->ce.start = key.offset;
 	map->ce.size = num_bytes;
diff --git a/volumes.h b/volumes.h
index c0007ad..d88e1cf 100644
--- a/volumes.h
+++ b/volumes.h
@@ -194,7 +194,7 @@ int btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
 		      u64 *num_bytes, u64 type);
 int btrfs_alloc_data_chunk(struct btrfs_trans_handle *trans,
 			   struct btrfs_root *extent_root, u64 *start,
-			   u64 num_bytes, u64 type);
+			   u64 num_bytes, u64 type, int convert);
 int btrfs_read_super_device(struct btrfs_root *root, struct extent_buffer *buf);
 int btrfs_add_device(struct btrfs_trans_handle *trans,
 		     struct btrfs_root *root,
-- 
2.7.0

[PATCH v3 13/22] btrfs-progs: extent-tree: Introduce function to find the first overlap extent.

[Qu Wenruo]

Introduce a new function, btrfs_search_overlap_extent() to find the first
overlap extent.

It's useful for later btrfs-convert rework.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 ctree.c       | 24 +++++++++++++++++++++++
 ctree.h       |  2 ++
 extent-tree.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 89 insertions(+)

diff --git a/ctree.c b/ctree.c
index 04cc476..76f3440 100644
--- a/ctree.c
+++ b/ctree.c
@@ -2938,3 +2938,27 @@ int btrfs_previous_extent_item(struct btrfs_root *root,
 	}
 	return 1;
 }
+
+/*
+ * Search in extent tree to found next meta/data extent
+ * Caller need to check no-hole or skinny metadata
+ */
+int btrfs_next_extent_item(struct btrfs_root *root,
+			struct btrfs_path *path, u64 max_objectid)
+{
+	struct btrfs_key found_key;
+	int ret;
+
+	while (1) {
+		ret = btrfs_next_item(root, path);
+		if (ret)
+			return ret;
+		btrfs_item_key_to_cpu(path->nodes[0], &found_key,
+				      path->slots[0]);
+		if (found_key.objectid > max_objectid)
+			return 1;
+		if (found_key.type == BTRFS_EXTENT_ITEM_KEY ||
+		    found_key.type == BTRFS_METADATA_ITEM_KEY)
+		return 0;
+	}
+}
diff --git a/ctree.h b/ctree.h
index 21b0445..59ea01c 100644
--- a/ctree.h
+++ b/ctree.h
@@ -2355,6 +2355,8 @@ int btrfs_previous_item(struct btrfs_root *root,
 			int type);
 int btrfs_previous_extent_item(struct btrfs_root *root,
 			struct btrfs_path *path, u64 min_objectid);
+int btrfs_next_extent_item(struct btrfs_root *root,
+			struct btrfs_path *path, u64 max_objectid);
 int btrfs_cow_block(struct btrfs_trans_handle *trans,
 		    struct btrfs_root *root, struct extent_buffer *buf,
 		    struct extent_buffer *parent, int parent_slot,
diff --git a/extent-tree.c b/extent-tree.c
index 1650bdb..29469ff 100644
--- a/extent-tree.c
+++ b/extent-tree.c
@@ -3908,6 +3908,69 @@ int btrfs_fix_block_accounting(struct btrfs_trans_handle *trans,
 	return 0;
 }
 
+static void __get_extent_size(struct btrfs_root *root, struct btrfs_path *path,
+			      u64 *start, u64 *len)
+{
+	struct btrfs_key key;
+
+	btrfs_item_key_to_cpu(path->nodes[0], &key, path->slots[0]);
+	BUG_ON(!(key.type == BTRFS_EXTENT_ITEM_KEY ||
+		 key.type == BTRFS_METADATA_ITEM_KEY));
+	*start = key.objectid;
+	if (key.type == BTRFS_EXTENT_ITEM_KEY)
+		*len = key.offset;
+	else
+		*len = root->nodesize;
+}
+
+/*
+ * Find first overlap extent for range [bytenr, bytenr + len)
+ * Return 0 for found and point path to it.
+ * Return >0 for not found.
+ * Return <0 for err
+ */
+int btrfs_search_overlap_extent(struct btrfs_root *root,
+				struct btrfs_path *path, u64 bytenr, u64 len)
+{
+	struct btrfs_key key;
+	u64 cur_start;
+	u64 cur_len;
+	int ret;
+
+	key.objectid = bytenr;
+	key.type = BTRFS_EXTENT_DATA_KEY;
+	key.offset = (u64)-1;
+
+	ret = btrfs_search_slot(NULL, root, &key, path, 0, 0);
+	if (ret < 0)
+		return ret;
+	BUG_ON(ret == 0);
+
+	ret = btrfs_previous_extent_item(root, path, 0);
+	if (ret < 0)
+		return ret;
+	/* no previous, check next extent */
+	if (ret > 0)
+		goto next;
+	__get_extent_size(root, path, &cur_start, &cur_len);
+	/* Tail overlap */
+	if (cur_start + cur_len > bytenr)
+		return 1;
+
+next:
+	ret = btrfs_next_extent_item(root, path, bytenr + len);
+	if (ret < 0)
+		return ret;
+	/* No next, prev already checked, no overlap */
+	if (ret > 0)
+		return 0;
+	__get_extent_size(root, path, &cur_start, &cur_len);
+	/* head overlap*/
+	if (cur_start < bytenr + len)
+		return 1;
+	return 0;
+}
+
 /*
  * Record a file extent. Do all the required works, such as inserting
  * file extent item, inserting extent item and backref item into extent
-- 
2.7.0

[PATCH v3 14/22] btrfs-progs: extent-tree: Enhance btrfs_record_file_extent

[Qu Wenruo]

Btrfs_record_file_extent() has some small problems like:
1) Can't handle overlap extent
2) May create extent larger than BTRFS_MAX_EXTENT_SIZE

So enhance it using previous added facilites.
This is used for later btrfs-convert, as for new convert, we create save
image first, then copy inode.
Which will also cause extent overlap.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 ctree.h       |   1 +
 extent-tree.c | 165 ++++++++++++++++++++++++++++++++++++++++------------------
 2 files changed, 115 insertions(+), 51 deletions(-)

diff --git a/ctree.h b/ctree.h
index 59ea01c..1443746 100644
--- a/ctree.h
+++ b/ctree.h
@@ -577,6 +577,7 @@ struct btrfs_extent_item_v0 {
 
 #define BTRFS_MAX_EXTENT_ITEM_SIZE(r) ((BTRFS_LEAF_DATA_SIZE(r) >> 4) - \
 					sizeof(struct btrfs_item))
+#define BTRFS_MAX_EXTENT_SIZE		(128 * 1024 * 1024)
 
 #define BTRFS_EXTENT_FLAG_DATA		(1ULL << 0)
 #define BTRFS_EXTENT_FLAG_TREE_BLOCK	(1ULL << 1)
diff --git a/extent-tree.c b/extent-tree.c
index 29469ff..93b1945 100644
--- a/extent-tree.c
+++ b/extent-tree.c
@@ -3971,16 +3971,11 @@ next:
 	return 0;
 }
 
-/*
- * Record a file extent. Do all the required works, such as inserting
- * file extent item, inserting extent item and backref item into extent
- * tree and updating block accounting.
- */
-int btrfs_record_file_extent(struct btrfs_trans_handle *trans,
-			      struct btrfs_root *root, u64 objectid,
-			      struct btrfs_inode_item *inode,
-			      u64 file_pos, u64 disk_bytenr,
-			      u64 num_bytes)
+static int __btrfs_record_file_extent(struct btrfs_trans_handle *trans,
+				      struct btrfs_root *root, u64 objectid,
+				      struct btrfs_inode_item *inode,
+				      u64 file_pos, u64 disk_bytenr,
+				      u64 *ret_num_bytes)
 {
 	int ret;
 	struct btrfs_fs_info *info = root->fs_info;
@@ -3988,10 +3983,19 @@ int btrfs_record_file_extent(struct btrfs_trans_handle *trans,
 	struct extent_buffer *leaf;
 	struct btrfs_file_extent_item *fi;
 	struct btrfs_key ins_key;
-	struct btrfs_path path;
+	struct btrfs_path *path;
 	struct btrfs_extent_item *ei;
 	u64 nbytes;
+	u64 extent_num_bytes;
+	u64 extent_bytenr;
+	u64 extent_offset;
+	u64 num_bytes = *ret_num_bytes;
 
+	num_bytes = min_t(u64, num_bytes, BTRFS_MAX_EXTENT_SIZE);
+	/*
+	 * All supported file system should not use its 0 extent.
+	 * As it's for hole
+	 */
 	if (disk_bytenr == 0) {
 		ret = btrfs_insert_file_extent(trans, root, objectid,
 						file_pos, disk_bytenr,
@@ -3999,25 +4003,80 @@ int btrfs_record_file_extent(struct btrfs_trans_handle *trans,
 		return ret;
 	}
 
-	btrfs_init_path(&path);
+	path = btrfs_alloc_path();
+	if (!path)
+		return -ENOMEM;
+
+	/* First to check extent overlap */
+	ret = btrfs_search_overlap_extent(extent_root, path, disk_bytenr,
+					  num_bytes);
+	if (ret < 0)
+		goto fail;
+	if (ret > 0) {
+		/* Found overlap */
+		u64 cur_start;
+		u64 cur_len;
+
+		__get_extent_size(extent_root, path, &cur_start, &cur_len);
+		/*
+		 * For convert case, this extent should be a subset of
+		 * existing one.
+		 */
+		BUG_ON(disk_bytenr < cur_start);
 
+		extent_bytenr = cur_start;
+		extent_num_bytes = cur_len;
+		extent_offset = disk_bytenr - extent_bytenr;
+	} else {
+		/* No overlap, create new extent */
+		btrfs_release_path(path);
+		ins_key.objectid = disk_bytenr;
+		ins_key.offset = num_bytes;
+		ins_key.type = BTRFS_EXTENT_ITEM_KEY;
+
+		ret = btrfs_insert_empty_item(trans, extent_root, path,
+					      &ins_key, sizeof(*ei));
+		if (ret == 0) {
+			leaf = path->nodes[0];
+			ei = btrfs_item_ptr(leaf, path->slots[0],
+					    struct btrfs_extent_item);
+
+			btrfs_set_extent_refs(leaf, ei, 0);
+			btrfs_set_extent_generation(leaf, ei, 0);
+			btrfs_set_extent_flags(leaf, ei,
+					       BTRFS_EXTENT_FLAG_DATA);
+			btrfs_mark_buffer_dirty(leaf);
+
+			ret = btrfs_update_block_group(trans, root, disk_bytenr,
+						       num_bytes, 1, 0);
+			if (ret)
+				goto fail;
+		} else if (ret != -EEXIST) {
+			goto fail;
+		}
+		btrfs_extent_post_op(trans, extent_root);
+		extent_bytenr = disk_bytenr;
+		extent_num_bytes = num_bytes;
+		extent_offset = 0;
+	}
+	btrfs_release_path(path);
 	ins_key.objectid = objectid;
 	ins_key.offset = file_pos;
 	btrfs_set_key_type(&ins_key, BTRFS_EXTENT_DATA_KEY);
-	ret = btrfs_insert_empty_item(trans, root, &path, &ins_key,
+	ret = btrfs_insert_empty_item(trans, root, path, &ins_key,
 				      sizeof(*fi));
 	if (ret)
 		goto fail;
-	leaf = path.nodes[0];
-	fi = btrfs_item_ptr(leaf, path.slots[0],
+	leaf = path->nodes[0];
+	fi = btrfs_item_ptr(leaf, path->slots[0],
 			    struct btrfs_file_extent_item);
 	btrfs_set_file_extent_generation(leaf, fi, trans->transid);
 	btrfs_set_file_extent_type(leaf, fi, BTRFS_FILE_EXTENT_REG);
-	btrfs_set_file_extent_disk_bytenr(leaf, fi, disk_bytenr);
-	btrfs_set_file_extent_disk_num_bytes(leaf, fi, num_bytes);
-	btrfs_set_file_extent_offset(leaf, fi, 0);
+	btrfs_set_file_extent_disk_bytenr(leaf, fi, extent_bytenr);
+	btrfs_set_file_extent_disk_num_bytes(leaf, fi, extent_num_bytes);
+	btrfs_set_file_extent_offset(leaf, fi, extent_offset);
 	btrfs_set_file_extent_num_bytes(leaf, fi, num_bytes);
-	btrfs_set_file_extent_ram_bytes(leaf, fi, num_bytes);
+	btrfs_set_file_extent_ram_bytes(leaf, fi, extent_num_bytes);
 	btrfs_set_file_extent_compression(leaf, fi, 0);
 	btrfs_set_file_extent_encryption(leaf, fi, 0);
 	btrfs_set_file_extent_other_encoding(leaf, fi, 0);
@@ -4025,43 +4084,47 @@ int btrfs_record_file_extent(struct btrfs_trans_handle *trans,
 
 	nbytes = btrfs_stack_inode_nbytes(inode) + num_bytes;
 	btrfs_set_stack_inode_nbytes(inode, nbytes);
+	btrfs_release_path(path);
 
-	btrfs_release_path(&path);
-
-	ins_key.objectid = disk_bytenr;
-	ins_key.offset = num_bytes;
-	ins_key.type = BTRFS_EXTENT_ITEM_KEY;
-
-	ret = btrfs_insert_empty_item(trans, extent_root, &path,
-				      &ins_key, sizeof(*ei));
-	if (ret == 0) {
-		leaf = path.nodes[0];
-		ei = btrfs_item_ptr(leaf, path.slots[0],
-				    struct btrfs_extent_item);
-
-		btrfs_set_extent_refs(leaf, ei, 0);
-		btrfs_set_extent_generation(leaf, ei, 0);
-		btrfs_set_extent_flags(leaf, ei, BTRFS_EXTENT_FLAG_DATA);
-
-		btrfs_mark_buffer_dirty(leaf);
-
-		ret = btrfs_update_block_group(trans, root, disk_bytenr,
-					       num_bytes, 1, 0);
-		if (ret)
-			goto fail;
-	} else if (ret != -EEXIST) {
-		goto fail;
-	}
-	btrfs_extent_post_op(trans, extent_root);
-
-	ret = btrfs_inc_extent_ref(trans, root, disk_bytenr, num_bytes, 0,
-				   root->root_key.objectid,
-				   objectid, file_pos);
+	ret = btrfs_inc_extent_ref(trans, root, extent_bytenr, extent_num_bytes,
+				   0, root->root_key.objectid, objectid,
+				   file_pos - extent_offset);
 	if (ret)
 		goto fail;
 	ret = 0;
+	*ret_num_bytes = min(extent_num_bytes - extent_offset, num_bytes);
 fail:
-	btrfs_release_path(&path);
+	btrfs_free_path(path);
+	return ret;
+}
+
+/*
+ * Record a file extent. Do all the required works, such as inserting
+ * file extent item, inserting extent item and backref item into extent
+ * tree and updating block accounting.
+ */
+int btrfs_record_file_extent(struct btrfs_trans_handle *trans,
+			      struct btrfs_root *root, u64 objectid,
+			      struct btrfs_inode_item *inode,
+			      u64 file_pos, u64 disk_bytenr,
+			      u64 num_bytes)
+{
+	u64 cur_disk_bytenr = disk_bytenr;
+	u64 cur_file_pos = file_pos;
+	u64 cur_num_bytes = num_bytes;
+	int ret = 0;
+
+	while (num_bytes > 0) {
+		ret = __btrfs_record_file_extent(trans, root, objectid,
+						 inode, cur_file_pos,
+						 cur_disk_bytenr,
+						 &cur_num_bytes);
+		if (ret < 0)
+			break;
+		cur_disk_bytenr += cur_num_bytes;
+		cur_file_pos += cur_num_bytes;
+		num_bytes -= cur_num_bytes;
+	}
 	return ret;
 }
 
-- 
2.7.0

[PATCH v3 15/22] btrfs-progs: convert: Introduce new function to create converted image

[Qu Wenruo]

Use new function, create_convert_image_v2() to create snapshot of old
filesystem.

Unlike old function which is called after copying all inodes, this
function need to be called before copying inodes.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 184 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 184 insertions(+)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 102953a..16e2309 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -1412,6 +1412,7 @@ static int create_image_file_range(struct btrfs_trans_handle *trans,
 fail:
 	return ret;
 }
+
 /*
  * Create the fs image file.
  */
@@ -1605,6 +1606,189 @@ fail:
 	return ret;
 }
 
+static int create_image_file_range_v2(struct btrfs_trans_handle *trans,
+				      struct btrfs_root *root,
+				      struct cache_tree *used,
+				      struct btrfs_inode_item *inode,
+				      u64 ino, u64 bytenr, u64 *ret_len,
+				      int datacsum)
+{
+	struct cache_extent *cache;
+	struct btrfs_block_group_cache *bg_cache;
+	u64 len = *ret_len;
+	u64 disk_bytenr;
+	int ret;
+
+	BUG_ON(bytenr != round_down(bytenr, root->sectorsize));
+	BUG_ON(len != round_down(len, root->sectorsize));
+	len = min_t(u64, len, BTRFS_MAX_EXTENT_SIZE);
+
+	cache = search_cache_extent(used, bytenr);
+	if (cache) {
+		if (cache->start <= bytenr) {
+			/*
+			 * |///////Used///////|
+			 *	|<--insert--->|
+			 *	bytenr
+			 */
+			len = min_t(u64, len, cache->start + cache->size -
+				    bytenr);
+			disk_bytenr = bytenr;
+		} else {
+			/*
+			 *		|//Used//|
+			 *  |<-insert-->|
+			 *  bytenr
+			 */
+			len = min(len, cache->start - bytenr);
+			disk_bytenr = 0;
+			datacsum = 0;
+		}
+	} else {
+		/*
+		 * |//Used//|		|EOF
+		 *	    |<-insert-->|
+		 *	    bytenr
+		 */
+		disk_bytenr = 0;
+		datacsum = 0;
+	}
+
+	if (disk_bytenr) {
+		/* Check if the range is in a data block group */
+		bg_cache = btrfs_lookup_block_group(root->fs_info, bytenr);
+		if (!bg_cache)
+			return -ENOENT;
+		if (!(bg_cache->flags & BTRFS_BLOCK_GROUP_DATA))
+			return -EINVAL;
+
+		/* The extent should never cross block group boundary */
+		len = min_t(u64, len, bg_cache->key.objectid +
+			    bg_cache->key.offset - bytenr);
+	}
+
+	BUG_ON(len != round_down(len, root->sectorsize));
+	ret = btrfs_record_file_extent(trans, root, ino, inode, bytenr,
+				       disk_bytenr, len);
+	if (ret < 0)
+		return ret;
+
+	if (datacsum)
+		ret = csum_disk_extent(trans, root, bytenr, len);
+	*ret_len = len;
+	return ret;
+}
+
+static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
+				int ensure_size);
+
+/*
+ * Create the fs image file of old filesystem.
+ *
+ * This is completely fs independent as we have cctx->used, only
+ * need to create file extents point to all the posistions.
+ * TODO: Add handler for reserved ranges in next patch
+ */
+static int create_image_v2(struct btrfs_root *root,
+			   struct btrfs_mkfs_config *cfg,
+			   struct btrfs_convert_context *cctx,
+			   u64 size, char *name, int datacsum)
+{
+	struct btrfs_inode_item buf;
+	struct btrfs_trans_handle *trans;
+	struct btrfs_path *path = NULL;
+	struct btrfs_key key;
+	struct cache_extent *cache;
+	struct cache_tree used_tmp;
+	u64 cur;
+	u64 ino;
+	int ret;
+
+	trans = btrfs_start_transaction(root, 1);
+	if (!trans)
+		return -ENOMEM;
+
+	cache_tree_init(&used_tmp);
+
+	ret = btrfs_find_free_objectid(trans, root, BTRFS_FIRST_FREE_OBJECTID,
+				       &ino);
+	if (ret < 0)
+		goto out;
+	ret = btrfs_new_inode(trans, root, ino, 0600 | S_IFREG);
+	if (ret < 0)
+		goto out;
+	ret = btrfs_add_link(trans, root, ino, BTRFS_FIRST_FREE_OBJECTID, name,
+			     strlen(name), BTRFS_FT_REG_FILE, NULL, 1);
+	if (ret < 0)
+		goto out;
+
+	path = btrfs_alloc_path();
+	if (!path) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	key.objectid = ino;
+	key.type = BTRFS_INODE_ITEM_KEY;
+	key.offset = 0;
+
+	ret = btrfs_search_slot(trans, root, &key, path, 0, 1);
+	if (ret) {
+		ret = (ret > 0 ? -ENOENT : ret);
+		goto out;
+	}
+	read_extent_buffer(path->nodes[0], &buf,
+			btrfs_item_ptr_offset(path->nodes[0], path->slots[0]),
+			sizeof(buf));
+	btrfs_release_path(path);
+
+	/*
+	 * Create a new used space cache, which doesn't contain the reserved
+	 * range
+	 */
+	for (cache = first_cache_extent(&cctx->used); cache;
+	     cache = next_cache_extent(cache)) {
+		ret = add_cache_extent(&used_tmp, cache->start, cache->size);
+		if (ret < 0)
+			goto out;
+	}
+	ret = wipe_reserved_ranges(&used_tmp, 0, 0);
+	if (ret < 0)
+		goto out;
+
+	/*
+	 * Start from 1M, as 0~1M is reserved, and create_image_file_range_v2()
+	 * can't handle bytenr 0(will consider it as a hole)
+	 */
+	cur = 1024 * 1024;
+	while (cur < size) {
+		u64 len = size - cur;
+
+		ret = create_image_file_range_v2(trans, root, &used_tmp,
+						&buf, ino, cur, &len, datacsum);
+		if (ret < 0)
+			goto out;
+		cur += len;
+	}
+
+	key.objectid = ino;
+	key.type = BTRFS_INODE_ITEM_KEY;
+	key.offset = 0;
+	ret = btrfs_search_slot(trans, root, &key, path, 0, 1);
+	if (ret) {
+		ret = (ret > 0 ? -ENOENT : ret);
+		goto out;
+	}
+	btrfs_set_stack_inode_size(&buf, cfg->num_bytes);
+	write_extent_buffer(path->nodes[0], &buf,
+			btrfs_item_ptr_offset(path->nodes[0], path->slots[0]),
+			sizeof(buf));
+out:
+	free_extent_cache_tree(&used_tmp);
+	btrfs_free_path(path);
+	btrfs_commit_transaction(trans, root);
+	return ret;
+}
+
 static struct btrfs_root * link_subvol(struct btrfs_root *root,
 		const char *base, u64 root_objectid)
 {
-- 
2.7.0

[PATCH v3 16/22] btrfs-progs: convert: Introduce function to migrate reserved ranges

[Qu Wenruo]

Introduce new function, migrate_reserved_ranges() to migrate used fs
data in btrfs reserved space.

Unlike old implement, which will need to relocate all the complicated
csum and reference relocation, previous patches already ensure such
reserved ranges won't be allocated.
So here we only need copy these data out and create new
extent/csum/reference.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 124 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 122 insertions(+), 2 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 16e2309..f6126db 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -1679,6 +1679,123 @@ static int create_image_file_range_v2(struct btrfs_trans_handle *trans,
 	return ret;
 }
 
+
+/*
+ * Relocate old fs data in one reserved ranges
+ *
+ * Since all old fs data in reserved range is not covered by any chunk nor
+ * data extent, we don't need to handle any reference but add new
+ * extent/reference, which makes codes more clear
+ */
+static int migrate_one_reserved_range(struct btrfs_trans_handle *trans,
+				      struct btrfs_root *root,
+				      struct cache_tree *used,
+				      struct btrfs_inode_item *inode, int fd,
+				      u64 ino, u64 start, u64 len, int datacsum)
+{
+	u64 cur_off = start;
+	u64 cur_len = len;
+	struct cache_extent *cache;
+	struct btrfs_key key;
+	struct extent_buffer *eb;
+	int ret = 0;
+
+	while (cur_off < start + len) {
+		cache = lookup_cache_extent(used, cur_off, cur_len);
+		if (!cache)
+			break;
+		cur_off = max(cache->start, cur_off);
+		cur_len = min(cache->start + cache->size, start + len) -
+			  cur_off;
+		BUG_ON(cur_len < root->sectorsize);
+
+		/* reserve extent for the data */
+		ret = btrfs_reserve_extent(trans, root, cur_len, 0, 0, (u64)-1,
+					   &key, 1);
+		if (ret < 0)
+			break;
+
+		eb = malloc(sizeof(*eb) + cur_len);
+		if (!eb) {
+			ret = -ENOMEM;
+			break;
+		}
+
+		ret = pread(fd, eb->data, cur_len, cur_off);
+		if (ret < cur_len) {
+			ret = (ret < 0 ? ret : -EIO);
+			free(eb);
+			break;
+		}
+		eb->start = key.objectid;
+		eb->len = key.offset;
+
+		/* Write the data */
+		ret = write_and_map_eb(trans, root, eb);
+		free(eb);
+		if (ret < 0)
+			break;
+
+		/* Now handle extent item and file extent things */
+		ret = btrfs_record_file_extent(trans, root, ino, inode, cur_off,
+					       key.objectid, key.offset);
+		if (ret < 0)
+			break;
+		/* Finally, insert csum items */
+		if (datacsum)
+			ret = csum_disk_extent(trans, root, key.objectid,
+					       key.offset);
+
+		cur_off += key.offset;
+		cur_len = start + len - cur_off;
+	}
+	return ret;
+}
+
+/*
+ * Relocate the used ext2 data in reserved ranges
+ * [0,1M)
+ * [btrfs_sb_offset(1), +BTRFS_STRIPE_LEN)
+ * [btrfs_sb_offset(2), +BTRFS_STRIPE_LEN)
+ */
+static int migrate_reserved_ranges(struct btrfs_trans_handle *trans,
+				   struct btrfs_root *root,
+				   struct cache_tree *used,
+				   struct btrfs_inode_item *inode, int fd,
+				   u64 ino, u64 total_bytes, int datacsum)
+{
+	u64 cur_off;
+	u64 cur_len;
+	int ret = 0;
+
+	/* 0 ~ 1M */
+	cur_off = 0;
+	cur_len = 1024 * 1024;
+	ret = migrate_one_reserved_range(trans, root, used, inode, fd, ino,
+					 cur_off, cur_len, datacsum);
+	if (ret < 0)
+		return ret;
+
+	/* second sb(fisrt sb is included in 0~1M) */
+	cur_off = btrfs_sb_offset(1);
+	cur_len = min(total_bytes, cur_off + BTRFS_STRIPE_LEN) - cur_off;
+	if (cur_off < total_bytes)
+		return ret;
+	ret = migrate_one_reserved_range(trans, root, used, inode, fd, ino,
+					 cur_off, cur_len, datacsum);
+	if (ret < 0)
+		return ret;
+
+	/* Last sb */
+	cur_off = btrfs_sb_offset(2);
+	cur_len = min(total_bytes, cur_off + BTRFS_STRIPE_LEN) - cur_off;
+	if (cur_off < total_bytes)
+		return ret;
+	ret = migrate_one_reserved_range(trans, root, used, inode, fd, ino,
+					 cur_off, cur_len, datacsum);
+	return ret;
+}
+
 static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
 				int ensure_size);
 
@@ -1687,11 +1804,10 @@ static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
  *
  * This is completely fs independent as we have cctx->used, only
  * need to create file extents point to all the posistions.
- * TODO: Add handler for reserved ranges in next patch
  */
 static int create_image_v2(struct btrfs_root *root,
 			   struct btrfs_mkfs_config *cfg,
-			   struct btrfs_convert_context *cctx,
+			   struct btrfs_convert_context *cctx, int fd,
 			   u64 size, char *name, int datacsum)
 {
 	struct btrfs_inode_item buf;
@@ -1769,6 +1885,10 @@ static int create_image_v2(struct btrfs_root *root,
 			goto out;
 		cur += len;
 	}
+	/* Handle the reserved ranges */
+	ret = migrate_reserved_ranges(trans, root, &cctx->used, &buf, fd, ino,
+				      cfg->num_bytes, datacsum);
+
 
 	key.objectid = ino;
 	key.type = BTRFS_INODE_ITEM_KEY;
-- 
2.7.0

[PATCH v3 17/22] btrfs-progs: convert: Enhance record_file_blocks to handle reserved ranges

[Qu Wenruo]

Enhance record_file_blocks() to handle reserved ranges.

Old file system can use the space in btrfs reserved ranges.
So we could not use the bytenr of old filesystem directly.

Thanks to previous patches, we have a full fs image in convert_root, and
it has already relocated the blocks in reserved ranges.
So here we just search the convert_root to get correct disk_bytenr and
use it.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 103 +++++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 94 insertions(+), 9 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index f6126db..8956a86 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -629,7 +629,9 @@ static int csum_disk_extent(struct btrfs_trans_handle *trans,
 struct blk_iterate_data {
 	struct btrfs_trans_handle *trans;
 	struct btrfs_root *root;
+	struct btrfs_root *convert_root;
 	struct btrfs_inode_item *inode;
+	u64 convert_ino;
 	u64 objectid;
 	u64 first_block;
 	u64 disk_block;
@@ -645,6 +647,8 @@ static void init_blk_iterate_data(struct blk_iterate_data *data,
 				  struct btrfs_inode_item *inode,
 				  u64 objectid, int checksum)
 {
+	struct btrfs_key key;
+
 	data->trans		= trans;
 	data->root		= root;
 	data->inode		= inode;
@@ -655,25 +659,106 @@ static void init_blk_iterate_data(struct blk_iterate_data *data,
 	data->boundary		= (u64)-1;
 	data->checksum		= checksum;
 	data->errcode		= 0;
+
+	key.objectid = CONV_IMAGE_SUBVOL_OBJECTID;
+	key.type = BTRFS_ROOT_ITEM_KEY;
+	key.offset = (u64)-1;
+	data->convert_root = btrfs_read_fs_root(root->fs_info, &key);
+	/* Impossible as we just opened it before */
+	BUG_ON(!data->convert_root || IS_ERR(data->convert_root));
+	data->convert_ino = BTRFS_FIRST_FREE_OBJECTID + 1;
 }
 
+/*
+ * Record a file extent in original file system into btrfs one.
+ * The special point is, old disk_block can point to a reserved range.
+ * So here, we don't use disk_block directly but search convert_root
+ * to get the real disk_bytenr.
+ */
 static int record_file_blocks(struct blk_iterate_data *data,
 			      u64 file_block, u64 disk_block, u64 num_blocks)
 {
-	int ret;
+	int ret = 0;
 	struct btrfs_root *root = data->root;
+	struct btrfs_root *convert_root = data->convert_root;
+	struct btrfs_path *path;
 	u64 file_pos = file_block * root->sectorsize;
-	u64 disk_bytenr = disk_block * root->sectorsize;
+	u64 old_disk_bytenr = disk_block * root->sectorsize;
 	u64 num_bytes = num_blocks * root->sectorsize;
-	ret = btrfs_record_file_extent(data->trans, data->root,
-				       data->objectid, data->inode, file_pos,
-				       disk_bytenr, num_bytes);
-
-	if (ret || !data->checksum || disk_bytenr == 0)
-		return ret;
+	u64 cur_off = old_disk_bytenr;
 
-	return csum_disk_extent(data->trans, data->root, disk_bytenr,
+	/* Hole, pass it to record_file_extent directly */
+	if (old_disk_bytenr == 0)
+		return btrfs_record_file_extent(data->trans, root,
+				data->objectid, data->inode, file_pos, 0,
 				num_bytes);
+
+	path = btrfs_alloc_path();
+	if (!path)
+		return -ENOMEM;
+
+	/*
+	 * Search real disk bytenr from convert root
+	 */
+	while (cur_off < old_disk_bytenr + num_bytes) {
+		struct btrfs_key key;
+		struct btrfs_file_extent_item *fi;
+		struct extent_buffer *node;
+		int slot;
+		u64 extent_disk_bytenr;
+		u64 extent_num_bytes;
+		u64 real_disk_bytenr;
+		u64 cur_len;
+
+		key.objectid = data->convert_ino;
+		key.type = BTRFS_EXTENT_DATA_KEY;
+		key.offset = cur_off;
+
+		ret = btrfs_search_slot(NULL, convert_root, &key, path, 0, 0);
+		if (ret < 0)
+			break;
+		if (ret > 0) {
+			ret = btrfs_previous_item(convert_root, path,
+						  data->convert_ino,
+						  BTRFS_EXTENT_DATA_KEY);
+			if (ret < 0)
+				break;
+			if (ret > 0) {
+				ret = -ENOENT;
+				break;
+			}
+		}
+		node = path->nodes[0];
+		slot = path->slots[0];
+		btrfs_item_key_to_cpu(node, &key, slot);
+		BUG_ON(key.type != BTRFS_EXTENT_DATA_KEY ||
+		       key.objectid != data->convert_ino ||
+		       key.offset > cur_off);
+		fi = btrfs_item_ptr(node, slot, struct btrfs_file_extent_item);
+		extent_disk_bytenr = btrfs_file_extent_disk_bytenr(node, fi);
+		extent_num_bytes = btrfs_file_extent_disk_num_bytes(node, fi);
+		BUG_ON(cur_off - key.offset >= extent_num_bytes);
+		btrfs_release_path(path);
+
+		real_disk_bytenr = cur_off - key.offset + extent_disk_bytenr;
+		cur_len = min(key.offset + extent_num_bytes,
+			      old_disk_bytenr + num_bytes) - cur_off;
+		ret = btrfs_record_file_extent(data->trans, data->root,
+					data->objectid, data->inode, file_pos,
+					real_disk_bytenr, cur_len);
+		if (ret < 0)
+			break;
+		cur_off += cur_len;
+		file_pos += cur_len;
+
+		/*
+		 * No need to care about csum
+		 * As every byte of old fs image is calculated for csum, no
+		 * need to waste cpu cycle now.
+		 */
+	}
+	btrfs_free_path(path);
+	return ret;
 }
 
 static int block_iterate_proc(u64 disk_block, u64 file_block,
-- 
2.7.0

[PATCH v3 18/22] btrfs-progs: convert: Introduce init_btrfs_v2 function.

[Qu Wenruo]

Introduce new init_btrfs_v2() function for later newer do_convert().

Since we have good enough chunk allocation, a lot of wired chunk hack
won't ever be used.
We only need to insert data chunks and create needed subvolume.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 8956a86..ce6c59b 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -2302,6 +2302,67 @@ static int make_convert_data_block_groups(struct btrfs_trans_handle *trans,
 	return ret;
 }
 
+/*
+ * Init the temp btrfs to a operational status.
+ *
+ * It will fix the extent usage accounting(XXX: Do we really need?) and
+ * insert needed data chunks, to ensure all old fs data extents are covered
+ * by DATA chunks, preventing wrong chunks are allocated.
+ *
+ * And also create convert image subvolume and relocation tree.
+ * (XXX: Not need again?)
+ * But the convert image subvolume is *NOT* linked to fs tree yet.
+ */
+static int init_btrfs_v2(struct btrfs_mkfs_config *cfg, struct btrfs_root *root,
+			 struct btrfs_convert_context *cctx, int datacsum,
+			 int packing, int noxattr)
+{
+	struct btrfs_key location;
+	struct btrfs_trans_handle *trans;
+	struct btrfs_fs_info *fs_info = root->fs_info;
+	int ret;
+
+	trans = btrfs_start_transaction(root, 1);
+	BUG_ON(!trans);
+	ret = btrfs_fix_block_accounting(trans, root);
+	if (ret)
+		goto err;
+	ret = make_convert_data_block_groups(trans, fs_info, cfg, cctx);
+	if (ret)
+		goto err;
+	ret = btrfs_make_root_dir(trans, fs_info->tree_root,
+				  BTRFS_ROOT_TREE_DIR_OBJECTID);
+	if (ret)
+		goto err;
+	memcpy(&location, &root->root_key, sizeof(location));
+	location.offset = (u64)-1;
+	ret = btrfs_insert_dir_item(trans, fs_info->tree_root, "default", 7,
+				btrfs_super_root_dir(fs_info->super_copy),
+				&location, BTRFS_FT_DIR, 0);
+	if (ret)
+		goto err;
+	ret = btrfs_insert_inode_ref(trans, fs_info->tree_root, "default", 7,
+				location.objectid,
+				btrfs_super_root_dir(fs_info->super_copy), 0);
+	if (ret)
+		goto err;
+	btrfs_set_root_dirid(&fs_info->fs_root->root_item,
+			     BTRFS_FIRST_FREE_OBJECTID);
+
+	/* subvol for fs image file */
+	ret = create_subvol(trans, root, CONV_IMAGE_SUBVOL_OBJECTID);
+	if (ret < 0)
+		goto err;
+	/* subvol for data relocation tree */
+	ret = create_subvol(trans, root, BTRFS_DATA_RELOC_TREE_OBJECTID);
+	if (ret < 0)
+		goto err;
+
+	ret = btrfs_commit_transaction(trans, root);
+err:
+	return ret;
+}
+
 static int init_btrfs(struct btrfs_root *root)
 {
 	int ret;
-- 
2.7.0

[PATCH v3 19/22] btrfs-progs: Introduce do_convert_v2 function

[Qu Wenruo]

Introduce new function do_convert_v2() to do new convert.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 177 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 176 insertions(+), 1 deletion(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index ce6c59b..17ce8ac 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -3211,6 +3211,181 @@ static int convert_read_used_space(struct btrfs_convert_context *cctx)
 	return ret;
 }
 
+static int do_convert_v2(const char *devname, int datacsum, int packing,
+		int noxattr, u32 nodesize, int copylabel, const char *fslabel,
+		int progress, u64 features)
+{
+	int ret;
+	int fd = -1;
+	int is_btrfs = 0;
+	u32 blocksize;
+	u64 total_bytes;
+	struct btrfs_root *root;
+	struct btrfs_root *image_root;
+	struct btrfs_convert_context cctx;
+	struct btrfs_key key;
+	char *subvol_name = NULL;
+	struct task_ctx ctx;
+	char features_buf[64];
+	struct btrfs_mkfs_config mkfs_cfg;
+
+	init_convert_context(&cctx);
+	ret = convert_open_fs(devname, &cctx);
+	if (ret)
+		goto fail;
+	ret = convert_read_used_space(&cctx);
+	if (ret)
+		goto fail;
+
+	blocksize = cctx.blocksize;
+	total_bytes = (u64)blocksize * (u64)cctx.block_count;
+	if (blocksize < 4096) {
+		fprintf(stderr, "block size is too small\n");
+		goto fail;
+	}
+	if (btrfs_check_nodesize(nodesize, blocksize, features))
+		goto fail;
+	fd = open(devname, O_RDWR);
+	if (fd < 0) {
+		fprintf(stderr, "unable to open %s\n", devname);
+		goto fail;
+	}
+	btrfs_parse_features_to_string(features_buf, features);
+	if (features == BTRFS_MKFS_DEFAULT_FEATURES)
+		strcat(features_buf, " (default)");
+
+	printf("create btrfs filesystem:\n");
+	printf("\tblocksize: %u\n", blocksize);
+	printf("\tnodesize:  %u\n", nodesize);
+	printf("\tfeatures:  %s\n", features_buf);
+
+	mkfs_cfg.label = cctx.volume_name;
+	mkfs_cfg.num_bytes = total_bytes;
+	mkfs_cfg.nodesize = nodesize;
+	mkfs_cfg.sectorsize = blocksize;
+	mkfs_cfg.stripesize = blocksize;
+	mkfs_cfg.features = features;
+	/* New convert need these space */
+	mkfs_cfg.fs_uuid = malloc(BTRFS_UUID_UNPARSED_SIZE);
+	mkfs_cfg.chunk_uuid = malloc(BTRFS_UUID_UNPARSED_SIZE);
+	*(mkfs_cfg.fs_uuid) = '\0';
+	*(mkfs_cfg.chunk_uuid) = '\0';
+
+	ret = make_btrfs(fd, &mkfs_cfg, &cctx);
+	if (ret) {
+		fprintf(stderr, "unable to create initial ctree: %s\n",
+			strerror(-ret));
+		goto fail;
+	}
+
+	root = open_ctree_fd(fd, devname, mkfs_cfg.super_bytenr,
+			     OPEN_CTREE_WRITES);
+	if (!root) {
+		fprintf(stderr, "unable to open ctree\n");
+		goto fail;
+	}
+	ret = init_btrfs_v2(&mkfs_cfg, root, &cctx, datacsum, packing, noxattr);
+	if (ret) {
+		fprintf(stderr, "unable to setup the root tree\n");
+		goto fail;
+	}
+
+	printf("creating %s image file.\n", cctx.convert_ops->name);
+	ret = asprintf(&subvol_name, "%s_saved", cctx.convert_ops->name);
+	if (ret < 0) {
+		fprintf(stderr, "error allocating subvolume name: %s_saved\n",
+			cctx.convert_ops->name);
+		goto fail;
+	}
+	key.objectid = CONV_IMAGE_SUBVOL_OBJECTID;
+	key.offset = (u64)-1;
+	key.type = BTRFS_ROOT_ITEM_KEY;
+	image_root = btrfs_read_fs_root(root->fs_info, &key);
+	if (!image_root) {
+		fprintf(stderr, "unable to create subvol\n");
+		goto fail;
+	}
+	ret = create_image_v2(image_root, &mkfs_cfg, &cctx, fd,
+			      mkfs_cfg.num_bytes, "image", datacsum);
+	if (ret) {
+		fprintf(stderr, "error during create_image %d\n", ret);
+		goto fail;
+	}
+
+	printf("creating btrfs metadata.\n");
+	ctx.max_copy_inodes = (cctx.inodes_count - cctx.free_inodes_count);
+	ctx.cur_copy_inodes = 0;
+
+	if (progress) {
+		ctx.info = task_init(print_copied_inodes, after_copied_inodes,
+				     &ctx);
+		task_start(ctx.info);
+	}
+	ret = copy_inodes(&cctx, root, datacsum, packing, noxattr, &ctx);
+	if (ret) {
+		fprintf(stderr, "error during copy_inodes %d\n", ret);
+		goto fail;
+	}
+	if (progress) {
+		task_stop(ctx.info);
+		task_deinit(ctx.info);
+	}
+
+	image_root = link_subvol(root, subvol_name, CONV_IMAGE_SUBVOL_OBJECTID);
+
+	free(subvol_name);
+
+	memset(root->fs_info->super_copy->label, 0, BTRFS_LABEL_SIZE);
+	if (copylabel == 1) {
+		strncpy(root->fs_info->super_copy->label,
+				cctx.volume_name, BTRFS_LABEL_SIZE);
+		fprintf(stderr, "copy label '%s'\n",
+				root->fs_info->super_copy->label);
+	} else if (copylabel == -1) {
+		strcpy(root->fs_info->super_copy->label, fslabel);
+		fprintf(stderr, "set label to '%s'\n", fslabel);
+	}
+
+	ret = close_ctree(root);
+	if (ret) {
+		fprintf(stderr, "error during close_ctree %d\n", ret);
+		goto fail;
+	}
+	convert_close_fs(&cctx);
+	clean_convert_context(&cctx);
+
+	/*
+	 * If this step succeed, we get a mountable btrfs. Otherwise
+	 * the source fs is left unchanged.
+	 */
+	ret = migrate_super_block(fd, mkfs_cfg.super_bytenr, blocksize);
+	if (ret) {
+		fprintf(stderr, "unable to migrate super block\n");
+		goto fail;
+	}
+	is_btrfs = 1;
+
+	root = open_ctree_fd(fd, devname, 0, OPEN_CTREE_WRITES);
+	if (!root) {
+		fprintf(stderr, "unable to open ctree\n");
+		goto fail;
+	}
+	close(fd);
+
+	printf("conversion complete.\n");
+	return 0;
+fail:
+	clean_convert_context(&cctx);
+	if (fd != -1)
+		close(fd);
+	if (is_btrfs)
+		fprintf(stderr,
+			"WARNING: an error occurred during chunk mapping fixup, filesystem mountable but not finalized\n");
+	else
+		fprintf(stderr, "conversion aborted\n");
+	return -1;
+}
+
 static int do_convert(const char *devname, int datacsum, int packing, int noxattr,
 		u32 nodesize, int copylabel, const char *fslabel, int progress,
 		u64 features)
@@ -3960,7 +4135,7 @@ int main(int argc, char *argv[])
 	if (rollback) {
 		ret = do_rollback(file);
 	} else {
-		ret = do_convert(file, datacsum, packing, noxattr, nodesize,
+		ret = do_convert_v2(file, datacsum, packing, noxattr, nodesize,
 				copylabel, fslabel, progress, features);
 	}
 	if (ret)
-- 
2.7.0

[PATCH v3 20/22] btrfs-progs: Convert: Add support for rollback new convert behavior

[Qu Wenruo]

Add support to rollback new btrfs-convert.

The support is quite easy unlike the new convert behavior, which in fact
makes the rollback less restricted.

The core is, rollback must support both old and new convert result.

For old convert, all fs image extents can be covered by any chunk type.
But any extents inside reserved range must be covered by chunk.
For new convert, all fs image extents are covered by data chunk.

Allowing any condition will just make another fail to pass.
So make the convert condition a little less restricted to allow both can
be converted.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 17ce8ac..efa3b02 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -3801,11 +3801,19 @@ static int do_rollback(const char *devname)
 		num_bytes = btrfs_file_extent_num_bytes(leaf, fi);
 
 		cache1 = btrfs_lookup_block_group(root->fs_info, offset);
-		cache2 =  btrfs_lookup_block_group(root->fs_info,
-						   offset + num_bytes - 1);
-		if (!cache1 || cache1 != cache2 ||
-		    (!(cache1->flags & BTRFS_BLOCK_GROUP_SYSTEM) &&
-		     !intersect_with_sb(offset, num_bytes)))
+		cache2 = btrfs_lookup_block_group(root->fs_info,
+						  offset + num_bytes - 1);
+		/*
+		 * Here we must take consideration of old and new convert
+		 * behavior.
+		 * For old convert case, sign, there is no consist chunk type
+		 * that will cover the extent. META/DATA/SYS are all possible.
+		 * Just ensure relocate one is in SYS chunk.
+		 * For new convert case, they are all covered by DATA chunk.
+		 *
+		 * So, there is not valid chunk type check for it now.
+		 */
+		if (cache1 != cache2)
 			break;
 
 		set_extent_bits(&io_tree, offset, offset + num_bytes - 1,
@@ -3819,6 +3827,7 @@ next_extent:
 
 	if (offset < total_bytes) {
 		fprintf(stderr, "unable to build extent mapping\n");
+		fprintf(stderr, "converted filesystem after balance is unable to rollback\n");
 		goto fail;
 	}
 
-- 
2.7.0

[PATCH v3 21/22] btrfs-progs: convert: Strictly avoid meta or system chunk allocation

[Qu Wenruo]

Before this patch, btrfs-convert only rely on large enough initial
system/metadata chunk size to ensure no newer system/meta chunk will be
created.

But that's not safe enough. So add two new members in fs_info,
avoid_sys/meta_chunk_alloc flags to prevent any newer system or meta
chunks to be created before init_btrfs_v2().

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c |  9 +++++++++
 ctree.h         |  3 +++
 extent-tree.c   | 10 ++++++++++
 3 files changed, 22 insertions(+)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index efa3b02..333f413 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -2322,6 +2322,13 @@ static int init_btrfs_v2(struct btrfs_mkfs_config *cfg, struct btrfs_root *root,
 	struct btrfs_fs_info *fs_info = root->fs_info;
 	int ret;
 
+	/*
+	 * Don't alloc any metadata/system chunk, as we don't want
+	 * any meta/sys chunk allcated before all data chunks are inserted.
+	 * Or we screw up the chunk layout just like the old implement.
+	 */
+	fs_info->avoid_sys_chunk_alloc = 1;
+	fs_info->avoid_meta_chunk_alloc = 1;
 	trans = btrfs_start_transaction(root, 1);
 	BUG_ON(!trans);
 	ret = btrfs_fix_block_accounting(trans, root);
@@ -2359,6 +2366,8 @@ static int init_btrfs_v2(struct btrfs_mkfs_config *cfg, struct btrfs_root *root,
 		goto err;
 
 	ret = btrfs_commit_transaction(trans, root);
+	fs_info->avoid_sys_chunk_alloc = 0;
+	fs_info->avoid_meta_chunk_alloc = 0;
 err:
 	return ret;
 }
diff --git a/ctree.h b/ctree.h
index 1443746..187bd27 100644
--- a/ctree.h
+++ b/ctree.h
@@ -1030,6 +1030,9 @@ struct btrfs_fs_info {
 	unsigned int quota_enabled:1;
 	unsigned int suppress_check_block_errors:1;
 	unsigned int ignore_fsid_mismatch:1;
+	unsigned int avoid_meta_chunk_alloc:1;
+	unsigned int avoid_sys_chunk_alloc:1;
+
 
 	int (*free_extent_hook)(struct btrfs_trans_handle *trans,
 				struct btrfs_root *root,
diff --git a/extent-tree.c b/extent-tree.c
index 93b1945..e7c61b1 100644
--- a/extent-tree.c
+++ b/extent-tree.c
@@ -1904,6 +1904,16 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
 	    thresh)
 		return 0;
 
+	/*
+	 * Avoid allocating given chunk type
+	 */
+	if (extent_root->fs_info->avoid_meta_chunk_alloc &&
+	    (flags & BTRFS_BLOCK_GROUP_METADATA))
+		return 0;
+	if (extent_root->fs_info->avoid_sys_chunk_alloc &&
+	    (flags & BTRFS_BLOCK_GROUP_SYSTEM))
+		return 0;
+
 	ret = btrfs_alloc_chunk(trans, extent_root, &start, &num_bytes,
 	                        space_info->flags);
 	if (ret == -ENOSPC) {
-- 
2.7.0

[PATCH v3 22/22] btrfs-progs: Cleanup old btrfs-convert

[Qu Wenruo]

Cleanup all the old btrfs-convert facilities, including:
1) btrfs_convert_operations->alloc/free/test_extents*
   No need to do non-standard extent allocation.
   After init_btrfs() everything can be done by normal routine.

   Now only 4 functions are needed in btrfs_convert_operations.
   1) open_fs
   2) read_used_space
   3) copy_inodes
   4) close_fs

2) fs_info->extent_ops
   Same as above.

3) Old init_btrfs(), create_image(), create_file_image_range()
   Replaced with newer and cleaner one.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
---
 btrfs-convert.c | 1425 ++-----------------------------------------------------
 ctree.h         |    8 -
 extent-tree.c   |   17 -
 3 files changed, 52 insertions(+), 1398 deletions(-)

diff --git a/btrfs-convert.c b/btrfs-convert.c
index 333f413..9c844c2 100644
--- a/btrfs-convert.c
+++ b/btrfs-convert.c
@@ -82,14 +82,6 @@ struct btrfs_convert_operations {
 	const char *name;
 	int (*open_fs)(struct btrfs_convert_context *cctx, const char *devname);
 	int (*read_used_space)(struct btrfs_convert_context *cctx);
-	int (*alloc_block)(struct btrfs_convert_context *cctx, u64 goal,
-			   u64 *block_ret);
-	int (*alloc_block_range)(struct btrfs_convert_context *cctx, u64 goal,
-			   int num, u64 *block_ret);
-	int (*test_block)(struct btrfs_convert_context *cctx, u64 block);
-	void (*free_block)(struct btrfs_convert_context *cctx, u64 block);
-	void (*free_block_range)(struct btrfs_convert_context *cctx, u64 block,
-			   int num);
 	int (*copy_inodes)(struct btrfs_convert_context *cctx,
 			 struct btrfs_root *root, int datacsum,
 			 int packing, int noxattr, struct task_ctx *p);
@@ -110,36 +102,6 @@ static void clean_convert_context(struct btrfs_convert_context *cctx)
 	free_extent_cache_tree(&cctx->free);
 }
 
-static inline int convert_alloc_block(struct btrfs_convert_context *cctx,
-				      u64 goal, u64 *ret)
-{
-	return  cctx->convert_ops->alloc_block(cctx, goal, ret);
-}
-
-static inline int convert_alloc_block_range(struct btrfs_convert_context *cctx,
-				      u64 goal, int num, u64 *ret)
-{
-	return  cctx->convert_ops->alloc_block_range(cctx, goal, num, ret);
-}
-
-static inline int convert_test_block(struct btrfs_convert_context *cctx,
-				     u64 block)
-{
-	return cctx->convert_ops->test_block(cctx, block);
-}
-
-static inline void convert_free_block(struct btrfs_convert_context *cctx,
-				      u64 block)
-{
-	cctx->convert_ops->free_block(cctx, block);
-}
-
-static inline void convert_free_block_range(struct btrfs_convert_context *cctx,
-				      u64 block, int num)
-{
-	cctx->convert_ops->free_block_range(cctx, block, num);
-}
-
 static inline int copy_inodes(struct btrfs_convert_context *cctx,
 			      struct btrfs_root *root, int datacsum,
 			      int packing, int noxattr, struct task_ctx *p)
@@ -285,152 +247,6 @@ static void ext2_close_fs(struct btrfs_convert_context *cctx)
 	ext2fs_close(cctx->fs_data);
 }
 
-static int ext2_alloc_block(struct btrfs_convert_context *cctx,
-			    u64 goal, u64 *block_ret)
-{
-	ext2_filsys fs = cctx->fs_data;
-	blk_t block;
-
-	if (!ext2fs_new_block(fs, goal, NULL, &block)) {
-		ext2fs_fast_mark_block_bitmap(fs->block_map, block);
-		*block_ret = block;
-		return 0;
-	}
-	return -ENOSPC;
-}
-
-static int ext2_alloc_block_range(struct btrfs_convert_context *cctx, u64 goal,
-		int num, u64 *block_ret)
-{
-	ext2_filsys fs = cctx->fs_data;
-	blk_t block;
-	ext2fs_block_bitmap bitmap = fs->block_map;
-	blk_t start = ext2fs_get_block_bitmap_start(bitmap);
-	blk_t end = ext2fs_get_block_bitmap_end(bitmap);
-
-	for (block = max_t(u64, goal, start); block + num < end; block++) {
-		if (ext2fs_fast_test_block_bitmap_range(bitmap, block, num)) {
-			ext2fs_fast_mark_block_bitmap_range(bitmap, block,
-					num);
-			*block_ret = block;
-			return 0;
-		}
-	}
-	return -ENOSPC;
-}
-
-static void ext2_free_block(struct btrfs_convert_context *cctx, u64 block)
-{
-	ext2_filsys fs = cctx->fs_data;
-
-	BUG_ON(block != (blk_t)block);
-	ext2fs_fast_unmark_block_bitmap(fs->block_map, block);
-}
-
-static void ext2_free_block_range(struct btrfs_convert_context *cctx, u64 block, int num)
-{
-	ext2_filsys fs = cctx->fs_data;
-
-	BUG_ON(block != (blk_t)block);
-	ext2fs_fast_unmark_block_bitmap_range(fs->block_map, block, num);
-}
-
-static int cache_free_extents(struct btrfs_root *root,
-			      struct btrfs_convert_context *cctx)
-
-{
-	int i, ret = 0;
-	blk_t block;
-	u64 bytenr;
-	u64 blocksize = cctx->blocksize;
-
-	block = cctx->first_data_block;
-	for (; block < cctx->block_count; block++) {
-		if (convert_test_block(cctx, block))
-			continue;
-		bytenr = block * blocksize;
-		ret = set_extent_dirty(&root->fs_info->free_space_cache,
-				       bytenr, bytenr + blocksize - 1, 0);
-		BUG_ON(ret);
-	}
-
-	for (i = 0; i < BTRFS_SUPER_MIRROR_MAX; i++) {
-		bytenr = btrfs_sb_offset(i);
-		bytenr &= ~((u64)BTRFS_STRIPE_LEN - 1);
-		if (bytenr >= blocksize * cctx->block_count)
-			break;
-		clear_extent_dirty(&root->fs_info->free_space_cache, bytenr,
-				   bytenr + BTRFS_STRIPE_LEN - 1, 0);
-	}
-
-	clear_extent_dirty(&root->fs_info->free_space_cache,
-			   0, BTRFS_SUPER_INFO_OFFSET - 1, 0);
-
-	return 0;
-}
-
-static int custom_alloc_extent(struct btrfs_root *root, u64 num_bytes,
-			       u64 hint_byte, struct btrfs_key *ins,
-			       int metadata)
-{
-	u64 start;
-	u64 end;
-	u64 last = hint_byte;
-	int ret;
-	int wrapped = 0;
-	struct btrfs_block_group_cache *cache;
-
-	while(1) {
-		ret = find_first_extent_bit(&root->fs_info->free_space_cache,
-					    last, &start, &end, EXTENT_DIRTY);
-		if (ret) {
-			if (wrapped++ == 0) {
-				last = 0;
-				continue;
-			} else {
-				goto fail;
-			}
-		}
-
-		start = max(last, start);
-		last = end + 1;
-		if (last - start < num_bytes)
-			continue;
-
-		last = start + num_bytes;
-		if (test_range_bit(&root->fs_info->pinned_extents,
-				   start, last - 1, EXTENT_DIRTY, 0))
-			continue;
-
-		cache = btrfs_lookup_block_group(root->fs_info, start);
-		BUG_ON(!cache);
-		if (cache->flags & BTRFS_BLOCK_GROUP_SYSTEM ||
-		    last > cache->key.objectid + cache->key.offset) {
-			last = cache->key.objectid + cache->key.offset;
-			continue;
-		}
-
-		if (metadata) {
-			BUG_ON(num_bytes != root->nodesize);
-			if (check_crossing_stripes(start, num_bytes)) {
-				last = round_down(start + num_bytes,
-						  BTRFS_STRIPE_LEN);
-				continue;
-			}
-		}
-		clear_extent_dirty(&root->fs_info->free_space_cache,
-				   start, start + num_bytes - 1, 0);
-
-		ins->objectid = start;
-		ins->offset = num_bytes;
-		ins->type = BTRFS_EXTENT_ITEM_KEY;
-		return 0;
-	}
-fail:
-	fprintf(stderr, "not enough free space\n");
-	return -ENOSPC;
-}
-
 static int intersect_with_sb(u64 bytenr, u64 num_bytes)
 {
 	int i;
@@ -447,17 +263,6 @@ static int intersect_with_sb(u64 bytenr, u64 num_bytes)
 	return 0;
 }
 
-static int custom_free_extent(struct btrfs_root *root, u64 bytenr,
-			      u64 num_bytes)
-{
-	return intersect_with_sb(bytenr, num_bytes);
-}
-
-static struct btrfs_extent_ops extent_ops = {
-	.alloc_extent = custom_alloc_extent,
-	.free_extent = custom_free_extent,
-};
-
 static int convert_insert_dirent(struct btrfs_trans_handle *trans,
 				 struct btrfs_root *root,
 				 const char *name, size_t name_len,
@@ -1361,29 +1166,6 @@ static int copy_single_inode(struct btrfs_trans_handle *trans,
 	return btrfs_insert_inode(trans, root, objectid, &btrfs_inode);
 }
 
-static int copy_disk_extent(struct btrfs_root *root, u64 dst_bytenr,
-		            u64 src_bytenr, u32 num_bytes)
-{
-	int ret;
-	char *buffer;
-	struct btrfs_fs_devices *fs_devs = root->fs_info->fs_devices;
-
-	buffer = malloc(num_bytes);
-	if (!buffer)
-		return -ENOMEM;
-	ret = pread(fs_devs->latest_bdev, buffer, num_bytes, src_bytenr);
-	if (ret != num_bytes)
-		goto fail;
-	ret = pwrite(fs_devs->latest_bdev, buffer, num_bytes, dst_bytenr);
-	if (ret != num_bytes)
-		goto fail;
-	ret = 0;
-fail:
-	free(buffer);
-	if (ret > 0)
-		ret = -1;
-	return ret;
-}
 /*
  * scan ext2's inode bitmap and copy all used inodes.
  */
@@ -1443,255 +1225,7 @@ static int ext2_copy_inodes(struct btrfs_convert_context *cctx,
 	return ret;
 }
 
-static int ext2_test_block(struct btrfs_convert_context *cctx, u64 block)
-{
-	ext2_filsys ext2_fs = cctx->fs_data;
-
-	BUG_ON(block != (u32)block);
-	return ext2fs_fast_test_block_bitmap(ext2_fs->block_map, block);
-}
-
-/*
- * Construct a range of ext2fs image file.
- * scan block allocation bitmap, find all blocks used by the ext2fs
- * in this range and create file extents that point to these blocks.
- *
- * Note: Before calling the function, no file extent points to blocks
- * 	 in this range
- */
 static int create_image_file_range(struct btrfs_trans_handle *trans,
-				   struct btrfs_root *root, u64 objectid,
-				   struct btrfs_inode_item *inode,
-				   u64 start_byte, u64 end_byte,
-				   struct btrfs_convert_context *cctx, int datacsum)
-{
-	u32 blocksize = cctx->blocksize;
-	u32 block = start_byte / blocksize;
-	u32 last_block = (end_byte + blocksize - 1) / blocksize;
-	int ret = 0;
-	struct blk_iterate_data data;
-
-	init_blk_iterate_data(&data, trans, root, inode, objectid, datacsum);
-	data.first_block = block;
-
-	for (; start_byte < end_byte; block++, start_byte += blocksize) {
-		if (!convert_test_block(cctx, block))
-			continue;
-		ret = block_iterate_proc(block, block, &data);
-		if (ret < 0)
-			goto fail;
-	}
-	if (data.num_blocks > 0) {
-		ret = record_file_blocks(&data, data.first_block,
-					 data.disk_block, data.num_blocks);
-		if (ret)
-			goto fail;
-		data.first_block += data.num_blocks;
-	}
-	if (last_block > data.first_block) {
-		ret = record_file_blocks(&data, data.first_block, 0,
-					 last_block - data.first_block);
-		if (ret)
-			goto fail;
-	}
-fail:
-	return ret;
-}
-
-/*
- * Create the fs image file.
- */
-static int create_image(struct btrfs_convert_context *cctx,
-			struct btrfs_root *root, const char *name, int datacsum)
-{
-	int ret;
-	struct btrfs_key key;
-	struct btrfs_key location;
-	struct btrfs_path path;
-	struct btrfs_inode_item btrfs_inode;
-	struct btrfs_inode_item *inode_item;
-	struct extent_buffer *leaf;
-	struct btrfs_fs_info *fs_info = root->fs_info;
-	struct btrfs_root *extent_root = fs_info->extent_root;
-	struct btrfs_trans_handle *trans;
-	struct btrfs_extent_item *ei;
-	struct btrfs_extent_inline_ref *iref;
-	struct btrfs_extent_data_ref *dref;
-	u64 bytenr;
-	u64 num_bytes;
-	u64 objectid;
-	u64 last_byte;
-	u64 first_free;
-	u64 total_bytes;
-	u64 flags = BTRFS_INODE_READONLY;
-	u32 sectorsize = root->sectorsize;
-
-	total_bytes = btrfs_super_total_bytes(fs_info->super_copy);
-	first_free =  BTRFS_SUPER_INFO_OFFSET + sectorsize * 2 - 1;
-	first_free &= ~((u64)sectorsize - 1);
-	if (!datacsum)
-		flags |= BTRFS_INODE_NODATASUM;
-
-	memset(&btrfs_inode, 0, sizeof(btrfs_inode));
-	btrfs_set_stack_inode_generation(&btrfs_inode, 1);
-	btrfs_set_stack_inode_size(&btrfs_inode, total_bytes);
-	btrfs_set_stack_inode_nlink(&btrfs_inode, 1);
-	btrfs_set_stack_inode_nbytes(&btrfs_inode, 0);
-	btrfs_set_stack_inode_mode(&btrfs_inode, S_IFREG | 0400);
-	btrfs_set_stack_inode_flags(&btrfs_inode,  flags);
-	btrfs_init_path(&path);
-	trans = btrfs_start_transaction(root, 1);
-	BUG_ON(!trans);
-
-	objectid = btrfs_root_dirid(&root->root_item);
-	ret = btrfs_find_free_objectid(trans, root, objectid, &objectid);
-	if (ret)
-		goto fail;
-
-	/*
-	 * copy blocks covered by extent #0 to new positions. extent #0 is
-	 * special, we can't rely on relocate_extents_range to relocate it.
-	 */
-	for (last_byte = 0; last_byte < first_free; last_byte += sectorsize) {
-		ret = custom_alloc_extent(root, sectorsize, 0, &key, 0);
-		if (ret)
-			goto fail;
-		ret = copy_disk_extent(root, key.objectid, last_byte,
-				       sectorsize);
-		if (ret)
-			goto fail;
-		ret = btrfs_record_file_extent(trans, root, objectid,
-					       &btrfs_inode, last_byte,
-					       key.objectid, sectorsize);
-		if (ret)
-			goto fail;
-		if (datacsum) {
-			ret = csum_disk_extent(trans, root, key.objectid,
-					       sectorsize);
-			if (ret)
-				goto fail;
-		}
-	}
-
-	while(1) {
-		key.objectid = last_byte;
-		key.offset = 0;
-		btrfs_set_key_type(&key, BTRFS_EXTENT_ITEM_KEY);
-		ret = btrfs_search_slot(trans, fs_info->extent_root,
-					&key, &path, 0, 0);
-		if (ret < 0)
-			goto fail;
-next:
-		leaf = path.nodes[0];
-		if (path.slots[0] >= btrfs_header_nritems(leaf)) {
-			ret = btrfs_next_leaf(extent_root, &path);
-			if (ret < 0)
-				goto fail;
-			if (ret > 0)
-				break;
-			leaf = path.nodes[0];
-		}
-		btrfs_item_key_to_cpu(leaf, &key, path.slots[0]);
-		if (last_byte > key.objectid ||
-		    key.type != BTRFS_EXTENT_ITEM_KEY) {
-			path.slots[0]++;
-			goto next;
-		}
-
-		bytenr = key.objectid;
-		num_bytes = key.offset;
-		ei = btrfs_item_ptr(leaf, path.slots[0],
-				    struct btrfs_extent_item);
-		if (!(btrfs_extent_flags(leaf, ei) & BTRFS_EXTENT_FLAG_DATA)) {
-			path.slots[0]++;
-			goto next;
-		}
-
-		BUG_ON(btrfs_item_size_nr(leaf, path.slots[0]) != sizeof(*ei) +
-		       btrfs_extent_inline_ref_size(BTRFS_EXTENT_DATA_REF_KEY));
-
-		iref = (struct btrfs_extent_inline_ref *)(ei + 1);
-		key.type = btrfs_extent_inline_ref_type(leaf, iref);
-		BUG_ON(key.type != BTRFS_EXTENT_DATA_REF_KEY);
-		dref = (struct btrfs_extent_data_ref *)(&iref->offset);
-		if (btrfs_extent_data_ref_root(leaf, dref) !=
-		    BTRFS_FS_TREE_OBJECTID) {
-			path.slots[0]++;
-			goto next;
-		}
-
-		if (bytenr > last_byte) {
-			ret = create_image_file_range(trans, root, objectid,
-						      &btrfs_inode, last_byte,
-						      bytenr, cctx,
-						      datacsum);
-			if (ret)
-				goto fail;
-		}
-		ret = btrfs_record_file_extent(trans, root, objectid,
-					       &btrfs_inode, bytenr, bytenr,
-					       num_bytes);
-		if (ret)
-			goto fail;
-		last_byte = bytenr + num_bytes;
-		btrfs_release_path(&path);
-
-		if (trans->blocks_used >= 4096) {
-			ret = btrfs_commit_transaction(trans, root);
-			BUG_ON(ret);
-			trans = btrfs_start_transaction(root, 1);
-			BUG_ON(!trans);
-		}
-	}
-	btrfs_release_path(&path);
-	if (total_bytes > last_byte) {
-		ret = create_image_file_range(trans, root, objectid,
-					      &btrfs_inode, last_byte,
-					      total_bytes, cctx,
-					      datacsum);
-		if (ret)
-			goto fail;
-	}
-
-	ret = btrfs_insert_inode(trans, root, objectid, &btrfs_inode);
-	if (ret)
-		goto fail;
-
-	location.objectid = objectid;
-	location.offset = 0;
-	btrfs_set_key_type(&location, BTRFS_INODE_ITEM_KEY);
-	ret = btrfs_insert_dir_item(trans, root, name, strlen(name),
-				    btrfs_root_dirid(&root->root_item),
-				    &location, BTRFS_FT_REG_FILE, objectid);
-	if (ret)
-		goto fail;
-	ret = btrfs_insert_inode_ref(trans, root, name, strlen(name),
-				     objectid,
-				     btrfs_root_dirid(&root->root_item),
-				     objectid);
-	if (ret)
-		goto fail;
-	location.objectid = btrfs_root_dirid(&root->root_item);
-	location.offset = 0;
-	btrfs_set_key_type(&location, BTRFS_INODE_ITEM_KEY);
-	ret = btrfs_lookup_inode(trans, root, &path, &location, 1);
-	if (ret)
-		goto fail;
-	leaf = path.nodes[0];
-	inode_item = btrfs_item_ptr(leaf, path.slots[0],
-				    struct btrfs_inode_item);
-	btrfs_set_inode_size(leaf, inode_item, strlen(name) * 2 +
-			     btrfs_inode_size(leaf, inode_item));
-	btrfs_mark_buffer_dirty(leaf);
-	btrfs_release_path(&path);
-	ret = btrfs_commit_transaction(trans, root);
-	BUG_ON(ret);
-fail:
-	btrfs_release_path(&path);
-	return ret;
-}
-
-static int create_image_file_range_v2(struct btrfs_trans_handle *trans,
 				      struct btrfs_root *root,
 				      struct cache_tree *used,
 				      struct btrfs_inode_item *inode,
@@ -1890,7 +1424,7 @@ static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
  * This is completely fs independent as we have cctx->used, only
  * need to create file extents point to all the posistions.
  */
-static int create_image_v2(struct btrfs_root *root,
+static int create_image(struct btrfs_root *root,
 			   struct btrfs_mkfs_config *cfg,
 			   struct btrfs_convert_context *cctx, int fd,
 			   u64 size, char *name, int datacsum)
@@ -1957,14 +1491,14 @@ static int create_image_v2(struct btrfs_root *root,
 		goto out;
 
 	/*
-	 * Start from 1M, as 0~1M is reserved, and create_image_file_range_v2()
+	 * Start from 1M, as 0~1M is reserved, and create_image_file_range()
 	 * can't handle bytenr 0(will consider it as a hole)
 	 */
 	cur = 1024 * 1024;
 	while (cur < size) {
 		u64 len = size - cur;
 
-		ret = create_image_file_range_v2(trans, root, &used_tmp,
+		ret = create_image_file_range(trans, root, &used_tmp,
 						&buf, ino, cur, &len, datacsum);
 		if (ret < 0)
 			goto out;
@@ -2094,147 +1628,24 @@ fail:
 	return new_root;
 }
 
-static int create_chunk_mapping(struct btrfs_trans_handle *trans,
-				struct btrfs_root *root)
+static int create_subvol(struct btrfs_trans_handle *trans,
+			 struct btrfs_root *root, u64 root_objectid)
 {
-	struct btrfs_fs_info *info = root->fs_info;
-	struct btrfs_root *chunk_root = info->chunk_root;
-	struct btrfs_root *extent_root = info->extent_root;
-	struct btrfs_device *device;
-	struct btrfs_block_group_cache *cache;
-	struct btrfs_dev_extent *extent;
-	struct extent_buffer *leaf;
-	struct btrfs_chunk chunk;
+	struct extent_buffer *tmp;
+	struct btrfs_root *new_root;
 	struct btrfs_key key;
-	struct btrfs_path path;
-	u64 cur_start;
-	u64 total_bytes;
-	u64 chunk_objectid;
+	struct btrfs_root_item root_item;
 	int ret;
 
-	btrfs_init_path(&path);
-
-	total_bytes = btrfs_super_total_bytes(root->fs_info->super_copy);
-	chunk_objectid = BTRFS_FIRST_CHUNK_TREE_OBJECTID;
+	ret = btrfs_copy_root(trans, root, root->node, &tmp,
+			      root_objectid);
+	BUG_ON(ret);
 
-	BUG_ON(list_empty(&info->fs_devices->devices));
-	device = list_entry(info->fs_devices->devices.next,
-			    struct btrfs_device, dev_list);
-	BUG_ON(device->devid != info->fs_devices->latest_devid);
-
-	/* delete device extent created by make_btrfs */
-	key.objectid = device->devid;
-	key.offset = 0;
-	key.type = BTRFS_DEV_EXTENT_KEY;
-	ret = btrfs_search_slot(trans, device->dev_root, &key, &path, -1, 1);
-	if (ret < 0)
-		goto err;
-
-	BUG_ON(ret > 0);
-	ret = btrfs_del_item(trans, device->dev_root, &path);
-	if (ret)
-		goto err;
-	btrfs_release_path(&path);
-
-	/* delete chunk item created by make_btrfs */
-	key.objectid = chunk_objectid;
-	key.offset = 0;
-	key.type = BTRFS_CHUNK_ITEM_KEY;
-	ret = btrfs_search_slot(trans, chunk_root, &key, &path, -1, 1);
-	if (ret < 0)
-		goto err;
-
-	BUG_ON(ret > 0);
-	ret = btrfs_del_item(trans, chunk_root, &path);
-	if (ret)
-		goto err;
-	btrfs_release_path(&path);
-
-	/* for each block group, create device extent and chunk item */
-	cur_start = 0;
-	while (cur_start < total_bytes) {
-		cache = btrfs_lookup_block_group(root->fs_info, cur_start);
-		BUG_ON(!cache);
-
-		/* insert device extent */
-		key.objectid = device->devid;
-		key.offset = cache->key.objectid;
-		key.type = BTRFS_DEV_EXTENT_KEY;
-		ret = btrfs_insert_empty_item(trans, device->dev_root, &path,
-					      &key, sizeof(*extent));
-		if (ret)
-			goto err;
-
-		leaf = path.nodes[0];
-		extent = btrfs_item_ptr(leaf, path.slots[0],
-					struct btrfs_dev_extent);
-
-		btrfs_set_dev_extent_chunk_tree(leaf, extent,
-						chunk_root->root_key.objectid);
-		btrfs_set_dev_extent_chunk_objectid(leaf, extent,
-						    chunk_objectid);
-		btrfs_set_dev_extent_chunk_offset(leaf, extent,
-						  cache->key.objectid);
-		btrfs_set_dev_extent_length(leaf, extent, cache->key.offset);
-		write_extent_buffer(leaf, root->fs_info->chunk_tree_uuid,
-		    (unsigned long)btrfs_dev_extent_chunk_tree_uuid(extent),
-		    BTRFS_UUID_SIZE);
-		btrfs_mark_buffer_dirty(leaf);
-		btrfs_release_path(&path);
-
-		/* insert chunk item */
-		btrfs_set_stack_chunk_length(&chunk, cache->key.offset);
-		btrfs_set_stack_chunk_owner(&chunk,
-					    extent_root->root_key.objectid);
-		btrfs_set_stack_chunk_stripe_len(&chunk, BTRFS_STRIPE_LEN);
-		btrfs_set_stack_chunk_type(&chunk, cache->flags);
-		btrfs_set_stack_chunk_io_align(&chunk, device->io_align);
-		btrfs_set_stack_chunk_io_width(&chunk, device->io_width);
-		btrfs_set_stack_chunk_sector_size(&chunk, device->sector_size);
-		btrfs_set_stack_chunk_num_stripes(&chunk, 1);
-		btrfs_set_stack_chunk_sub_stripes(&chunk, 0);
-		btrfs_set_stack_stripe_devid(&chunk.stripe, device->devid);
-		btrfs_set_stack_stripe_offset(&chunk.stripe,
-					      cache->key.objectid);
-		memcpy(&chunk.stripe.dev_uuid, device->uuid, BTRFS_UUID_SIZE);
-
-		key.objectid = chunk_objectid;
-		key.offset = cache->key.objectid;
-		key.type = BTRFS_CHUNK_ITEM_KEY;
-
-		ret = btrfs_insert_item(trans, chunk_root, &key, &chunk,
-					btrfs_chunk_item_size(1));
-		if (ret)
-			goto err;
-
-		cur_start = cache->key.objectid + cache->key.offset;
-	}
-
-	device->bytes_used = total_bytes;
-	ret = btrfs_update_device(trans, device);
-err:
-	btrfs_release_path(&path);
-	return ret;
-}
-
-static int create_subvol(struct btrfs_trans_handle *trans,
-			 struct btrfs_root *root, u64 root_objectid)
-{
-	struct extent_buffer *tmp;
-	struct btrfs_root *new_root;
-	struct btrfs_key key;
-	struct btrfs_root_item root_item;
-	int ret;
-
-	ret = btrfs_copy_root(trans, root, root->node, &tmp,
-			      root_objectid);
-	BUG_ON(ret);
-
-	memcpy(&root_item, &root->root_item, sizeof(root_item));
-	btrfs_set_root_bytenr(&root_item, tmp->start);
-	btrfs_set_root_level(&root_item, btrfs_header_level(tmp));
-	btrfs_set_root_generation(&root_item, trans->transid);
-	free_extent_buffer(tmp);
+	memcpy(&root_item, &root->root_item, sizeof(root_item));
+	btrfs_set_root_bytenr(&root_item, tmp->start);
+	btrfs_set_root_level(&root_item, btrfs_header_level(tmp));
+	btrfs_set_root_generation(&root_item, trans->transid);
+	free_extent_buffer(tmp);
 
 	key.objectid = root_objectid;
 	key.type = BTRFS_ROOT_ITEM_KEY;
@@ -2253,7 +1664,7 @@ static int create_subvol(struct btrfs_trans_handle *trans,
 }
 
 /*
- * New make_btrfs_v2() has handle system and meta chunks quite well.
+ * New make_btrfs() has handle system and meta chunks quite well.
  * So only need to add remaining data chunks.
  */
 static int make_convert_data_block_groups(struct btrfs_trans_handle *trans,
@@ -2313,7 +1724,7 @@ static int make_convert_data_block_groups(struct btrfs_trans_handle *trans,
  * (XXX: Not need again?)
  * But the convert image subvolume is *NOT* linked to fs tree yet.
  */
-static int init_btrfs_v2(struct btrfs_mkfs_config *cfg, struct btrfs_root *root,
+static int init_btrfs(struct btrfs_mkfs_config *cfg, struct btrfs_root *root,
 			 struct btrfs_convert_context *cctx, int datacsum,
 			 int packing, int noxattr)
 {
@@ -2372,63 +1783,6 @@ err:
 	return ret;
 }
 
-static int init_btrfs(struct btrfs_root *root)
-{
-	int ret;
-	struct btrfs_key location;
-	struct btrfs_trans_handle *trans;
-	struct btrfs_fs_info *fs_info = root->fs_info;
-	struct extent_buffer *tmp;
-
-	trans = btrfs_start_transaction(root, 1);
-	BUG_ON(!trans);
-	ret = btrfs_make_block_groups(trans, root);
-	if (ret)
-		goto err;
-	ret = btrfs_fix_block_accounting(trans, root);
-	if (ret)
-		goto err;
-	ret = create_chunk_mapping(trans, root);
-	if (ret)
-		goto err;
-	ret = btrfs_make_root_dir(trans, fs_info->tree_root,
-				  BTRFS_ROOT_TREE_DIR_OBJECTID);
-	if (ret)
-		goto err;
-	memcpy(&location, &root->root_key, sizeof(location));
-	location.offset = (u64)-1;
-	ret = btrfs_insert_dir_item(trans, fs_info->tree_root, "default", 7,
-				btrfs_super_root_dir(fs_info->super_copy),
-				&location, BTRFS_FT_DIR, 0);
-	if (ret)
-		goto err;
-	ret = btrfs_insert_inode_ref(trans, fs_info->tree_root, "default", 7,
-				location.objectid,
-				btrfs_super_root_dir(fs_info->super_copy), 0);
-	if (ret)
-		goto err;
-	btrfs_set_root_dirid(&fs_info->fs_root->root_item,
-			     BTRFS_FIRST_FREE_OBJECTID);
-
-	/* subvol for fs image file */
-	ret = create_subvol(trans, root, CONV_IMAGE_SUBVOL_OBJECTID);
-	BUG_ON(ret);
-	/* subvol for data relocation */
-	ret = create_subvol(trans, root, BTRFS_DATA_RELOC_TREE_OBJECTID);
-	BUG_ON(ret);
-
-	extent_buffer_get(fs_info->csum_root->node);
-	ret = __btrfs_cow_block(trans, fs_info->csum_root,
-				fs_info->csum_root->node, NULL, 0, &tmp, 0, 0);
-	BUG_ON(ret);
-	free_extent_buffer(tmp);
-
-	ret = btrfs_commit_transaction(trans, root);
-	BUG_ON(ret);
-err:
-	return ret;
-}
-
 /*
  * Migrate super block to its default position and zero 0 ~ 16k
  */
@@ -2471,515 +1825,54 @@ static int migrate_super_block(int fd, u64 old_bytenr, u32 sectorsize)
 		ret = pwrite(fd, buf->data, len, bytenr);
 		if (ret != len) {
 			fprintf(stderr, "unable to zero fill device\n");
-			break;
-		}
-		bytenr += len;
-	}
-	ret = 0;
-	fsync(fd);
-fail:
-	free(buf);
-	if (ret > 0)
-		ret = -1;
-	return ret;
-}
-
-static int prepare_system_chunk_sb(struct btrfs_super_block *super)
-{
-	struct btrfs_chunk *chunk;
-	struct btrfs_disk_key *key;
-	u32 sectorsize = btrfs_super_sectorsize(super);
-
-	key = (struct btrfs_disk_key *)(super->sys_chunk_array);
-	chunk = (struct btrfs_chunk *)(super->sys_chunk_array +
-				       sizeof(struct btrfs_disk_key));
-
-	btrfs_set_disk_key_objectid(key, BTRFS_FIRST_CHUNK_TREE_OBJECTID);
-	btrfs_set_disk_key_type(key, BTRFS_CHUNK_ITEM_KEY);
-	btrfs_set_disk_key_offset(key, 0);
-
-	btrfs_set_stack_chunk_length(chunk, btrfs_super_total_bytes(super));
-	btrfs_set_stack_chunk_owner(chunk, BTRFS_EXTENT_TREE_OBJECTID);
-	btrfs_set_stack_chunk_stripe_len(chunk, BTRFS_STRIPE_LEN);
-	btrfs_set_stack_chunk_type(chunk, BTRFS_BLOCK_GROUP_SYSTEM);
-	btrfs_set_stack_chunk_io_align(chunk, sectorsize);
-	btrfs_set_stack_chunk_io_width(chunk, sectorsize);
-	btrfs_set_stack_chunk_sector_size(chunk, sectorsize);
-	btrfs_set_stack_chunk_num_stripes(chunk, 1);
-	btrfs_set_stack_chunk_sub_stripes(chunk, 0);
-	chunk->stripe.devid = super->dev_item.devid;
-	btrfs_set_stack_stripe_offset(&chunk->stripe, 0);
-	memcpy(chunk->stripe.dev_uuid, super->dev_item.uuid, BTRFS_UUID_SIZE);
-	btrfs_set_super_sys_array_size(super, sizeof(*key) + sizeof(*chunk));
-	return 0;
-}
-
-static int prepare_system_chunk(int fd, u64 sb_bytenr)
-{
-	int ret;
-	struct extent_buffer *buf;
-	struct btrfs_super_block *super;
-
-	BUG_ON(BTRFS_SUPER_INFO_SIZE < sizeof(*super));
-	buf = malloc(sizeof(*buf) + BTRFS_SUPER_INFO_SIZE);
-	if (!buf)
-		return -ENOMEM;
-
-	buf->len = BTRFS_SUPER_INFO_SIZE;
-	ret = pread(fd, buf->data, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
-	if (ret != BTRFS_SUPER_INFO_SIZE)
-		goto fail;
-
-	super = (struct btrfs_super_block *)buf->data;
-	BUG_ON(btrfs_super_bytenr(super) != sb_bytenr);
-	BUG_ON(btrfs_super_num_devices(super) != 1);
-
-	ret = prepare_system_chunk_sb(super);
-	if (ret)
-		goto fail;
-
-	csum_tree_block_size(buf, BTRFS_CRC32_SIZE, 0);
-	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
-	if (ret != BTRFS_SUPER_INFO_SIZE)
-		goto fail;
-
-	ret = 0;
-fail:
-	free(buf);
-	if (ret > 0)
-		ret = -1;
-	return ret;
-}
-
-static int relocate_one_reference(struct btrfs_trans_handle *trans,
-				  struct btrfs_root *root,
-				  u64 extent_start, u64 extent_size,
-				  struct btrfs_key *extent_key,
-				  struct extent_io_tree *reloc_tree)
-{
-	struct extent_buffer *leaf;
-	struct btrfs_file_extent_item *fi;
-	struct btrfs_key key;
-	struct btrfs_path path;
-	struct btrfs_inode_item inode;
-	struct blk_iterate_data data;
-	u64 bytenr;
-	u64 num_bytes;
-	u64 cur_offset;
-	u64 new_pos;
-	u64 nbytes;
-	u64 sector_end;
-	u32 sectorsize = root->sectorsize;
-	unsigned long ptr;
-	int datacsum;
-	int fd;
-	int ret;
-
-	btrfs_init_path(&path);
-	ret = btrfs_search_slot(trans, root, extent_key, &path, -1, 1);
-	if (ret)
-		goto fail;
-
-	leaf = path.nodes[0];
-	fi = btrfs_item_ptr(leaf, path.slots[0],
-			    struct btrfs_file_extent_item);
-	BUG_ON(btrfs_file_extent_offset(leaf, fi) > 0);
-	if (extent_start != btrfs_file_extent_disk_bytenr(leaf, fi) ||
-	    extent_size != btrfs_file_extent_disk_num_bytes(leaf, fi)) {
-		ret = 1;
-		goto fail;
-	}
-
-	bytenr = extent_start + btrfs_file_extent_offset(leaf, fi);
-	num_bytes = btrfs_file_extent_num_bytes(leaf, fi);
-
-	ret = btrfs_del_item(trans, root, &path);
-	if (ret)
-		goto fail;
-
-	ret = btrfs_free_extent(trans, root, extent_start, extent_size, 0,
-				root->root_key.objectid,
-				extent_key->objectid, extent_key->offset);
-	if (ret)
-		goto fail;
-
-	btrfs_release_path(&path);
-
-	key.objectid = extent_key->objectid;
-	key.offset = 0;
-	key.type =  BTRFS_INODE_ITEM_KEY;
-	ret = btrfs_lookup_inode(trans, root, &path, &key, 0);
-	if (ret)
-		goto fail;
-
-	leaf = path.nodes[0];
-	ptr = btrfs_item_ptr_offset(leaf, path.slots[0]);
-	read_extent_buffer(leaf, &inode, ptr, sizeof(inode));
-	btrfs_release_path(&path);
-
-	BUG_ON(num_bytes & (sectorsize - 1));
-	nbytes = btrfs_stack_inode_nbytes(&inode) - num_bytes;
-	btrfs_set_stack_inode_nbytes(&inode, nbytes);
-	datacsum = !(btrfs_stack_inode_flags(&inode) & BTRFS_INODE_NODATASUM);
-
-	init_blk_iterate_data(&data, trans, root, &inode, extent_key->objectid,
-			      datacsum);
-	data.first_block = extent_key->offset;
-
-	cur_offset = extent_key->offset;
-	while (num_bytes > 0) {
-		sector_end = bytenr + sectorsize - 1;
-		if (test_range_bit(reloc_tree, bytenr, sector_end,
-				   EXTENT_LOCKED, 1)) {
-			ret = get_state_private(reloc_tree, bytenr, &new_pos);
-			BUG_ON(ret);
-		} else {
-			ret = custom_alloc_extent(root, sectorsize, 0, &key, 0);
-			if (ret)
-				goto fail;
-			new_pos = key.objectid;
-
-			if (cur_offset == extent_key->offset) {
-				fd = root->fs_info->fs_devices->latest_bdev;
-				readahead(fd, bytenr, num_bytes);
-			}
-			ret = copy_disk_extent(root, new_pos, bytenr,
-					       sectorsize);
-			if (ret)
-				goto fail;
-			ret = set_extent_bits(reloc_tree, bytenr, sector_end,
-					      EXTENT_LOCKED, GFP_NOFS);
-			BUG_ON(ret);
-			ret = set_state_private(reloc_tree, bytenr, new_pos);
-			BUG_ON(ret);
-		}
-
-		ret = block_iterate_proc(new_pos / sectorsize,
-					 cur_offset / sectorsize, &data);
-		if (ret < 0)
-			goto fail;
-
-		cur_offset += sectorsize;
-		bytenr += sectorsize;
-		num_bytes -= sectorsize;
-	}
-
-	if (data.num_blocks > 0) {
-		ret = record_file_blocks(&data, data.first_block,
-					 data.disk_block, data.num_blocks);
-		if (ret)
-			goto fail;
-	}
-
-	key.objectid = extent_key->objectid;
-	key.offset = 0;
-	key.type =  BTRFS_INODE_ITEM_KEY;
-	ret = btrfs_lookup_inode(trans, root, &path, &key, 1);
-	if (ret)
-		goto fail;
-
-	leaf = path.nodes[0];
-	ptr = btrfs_item_ptr_offset(leaf, path.slots[0]);
-	write_extent_buffer(leaf, &inode, ptr, sizeof(inode));
-	btrfs_mark_buffer_dirty(leaf);
-	btrfs_release_path(&path);
-
-fail:
-	btrfs_release_path(&path);
-	return ret;
-}
-
-static int relocate_extents_range(struct btrfs_root *fs_root,
-				  struct btrfs_root *image_root,
-				  u64 start_byte, u64 end_byte)
-{
-	struct btrfs_fs_info *info = fs_root->fs_info;
-	struct btrfs_root *extent_root = info->extent_root;
-	struct btrfs_root *cur_root = NULL;
-	struct btrfs_trans_handle *trans;
-	struct btrfs_extent_data_ref *dref;
-	struct btrfs_extent_inline_ref *iref;
-	struct btrfs_extent_item *ei;
-	struct extent_buffer *leaf;
-	struct btrfs_key key;
-	struct btrfs_key extent_key;
-	struct btrfs_path path;
-	struct extent_io_tree reloc_tree;
-	unsigned long ptr;
-	unsigned long end;
-	u64 cur_byte;
-	u64 num_bytes;
-	u64 ref_root;
-	u64 num_extents;
-	int pass = 0;
-	int ret;
-
-	btrfs_init_path(&path);
-	extent_io_tree_init(&reloc_tree);
-
-	key.objectid = start_byte;
-	key.offset = 0;
-	key.type = BTRFS_EXTENT_ITEM_KEY;
-	ret = btrfs_search_slot(NULL, extent_root, &key, &path, 0, 0);
-	if (ret < 0)
-		goto fail;
-	if (ret > 0) {
-		ret = btrfs_previous_item(extent_root, &path, 0,
-					  BTRFS_EXTENT_ITEM_KEY);
-		if (ret < 0)
-			goto fail;
-		if (ret == 0) {
-			leaf = path.nodes[0];
-			btrfs_item_key_to_cpu(leaf, &key, path.slots[0]);
-			if (key.objectid + key.offset > start_byte)
-				start_byte = key.objectid;
-		}
-	}
-	btrfs_release_path(&path);
-again:
-	cur_root = (pass % 2 == 0) ? image_root : fs_root;
-	num_extents = 0;
-
-	trans = btrfs_start_transaction(cur_root, 1);
-	BUG_ON(!trans);
-
-	cur_byte = start_byte;
-	while (1) {
-		key.objectid = cur_byte;
-		key.offset = 0;
-		key.type = BTRFS_EXTENT_ITEM_KEY;
-		ret = btrfs_search_slot(trans, extent_root,
-					&key, &path, 0, 0);
-		if (ret < 0)
-			goto fail;
-next:
-		leaf = path.nodes[0];
-		if (path.slots[0] >= btrfs_header_nritems(leaf)) {
-			ret = btrfs_next_leaf(extent_root, &path);
-			if (ret < 0)
-				goto fail;
-			if (ret > 0)
-				break;
-			leaf = path.nodes[0];
-		}
-
-		btrfs_item_key_to_cpu(leaf, &key, path.slots[0]);
-		if (key.objectid < cur_byte ||
-		    key.type != BTRFS_EXTENT_ITEM_KEY) {
-			path.slots[0]++;
-			goto next;
-		}
-		if (key.objectid >= end_byte)
-			break;
-
-		num_extents++;
-
-		cur_byte = key.objectid;
-		num_bytes = key.offset;
-		ei = btrfs_item_ptr(leaf, path.slots[0],
-				    struct btrfs_extent_item);
-		BUG_ON(!(btrfs_extent_flags(leaf, ei) &
-			 BTRFS_EXTENT_FLAG_DATA));
-
-		ptr = btrfs_item_ptr_offset(leaf, path.slots[0]);
-		end = ptr + btrfs_item_size_nr(leaf, path.slots[0]);
-
-		ptr += sizeof(struct btrfs_extent_item);
-
-		while (ptr < end) {
-			iref = (struct btrfs_extent_inline_ref *)ptr;
-			key.type = btrfs_extent_inline_ref_type(leaf, iref);
-			BUG_ON(key.type != BTRFS_EXTENT_DATA_REF_KEY);
-			dref = (struct btrfs_extent_data_ref *)(&iref->offset);
-			ref_root = btrfs_extent_data_ref_root(leaf, dref);
-			extent_key.objectid =
-				btrfs_extent_data_ref_objectid(leaf, dref);
-			extent_key.offset =
-				btrfs_extent_data_ref_offset(leaf, dref);
-			extent_key.type = BTRFS_EXTENT_DATA_KEY;
-			BUG_ON(btrfs_extent_data_ref_count(leaf, dref) != 1);
-
-			if (ref_root == cur_root->root_key.objectid)
-				break;
-
-			ptr += btrfs_extent_inline_ref_size(key.type);
-		}
-
-		if (ptr >= end) {
-			path.slots[0]++;
-			goto next;
-		}
-
-		ret = relocate_one_reference(trans, cur_root, cur_byte,
-					     num_bytes, &extent_key,
-					     &reloc_tree);
-		if (ret < 0)
-			goto fail;
-
-		cur_byte += num_bytes;
-		btrfs_release_path(&path);
-
-		if (trans->blocks_used >= 4096) {
-			ret = btrfs_commit_transaction(trans, cur_root);
-			BUG_ON(ret);
-			trans = btrfs_start_transaction(cur_root, 1);
-			BUG_ON(!trans);
-		}
-	}
-	btrfs_release_path(&path);
-
-	ret = btrfs_commit_transaction(trans, cur_root);
-	BUG_ON(ret);
-
-	if (num_extents > 0 && pass++ < 16)
-		goto again;
-
-	ret = (num_extents > 0) ? -1 : 0;
-fail:
-	btrfs_release_path(&path);
-	extent_io_tree_cleanup(&reloc_tree);
-	return ret;
-}
-
-/*
- * relocate data in system chunk
- */
-static int cleanup_sys_chunk(struct btrfs_root *fs_root,
-			     struct btrfs_root *image_root)
-{
-	struct btrfs_block_group_cache *cache;
-	int i, ret = 0;
-	u64 offset = 0;
-	u64 end_byte;
-
-	while(1) {
-		cache = btrfs_lookup_block_group(fs_root->fs_info, offset);
-		if (!cache)
-			break;
-
-		end_byte = cache->key.objectid + cache->key.offset;
-		if (cache->flags & BTRFS_BLOCK_GROUP_SYSTEM) {
-			ret = relocate_extents_range(fs_root, image_root,
-						     cache->key.objectid,
-						     end_byte);
-			if (ret)
-				goto fail;
+			break;
 		}
-		offset = end_byte;
-	}
-	for (i = 0; i < BTRFS_SUPER_MIRROR_MAX; i++) {
-		offset = btrfs_sb_offset(i);
-		offset &= ~((u64)BTRFS_STRIPE_LEN - 1);
-
-		ret = relocate_extents_range(fs_root, image_root,
-					     offset, offset + BTRFS_STRIPE_LEN);
-		if (ret)
-			goto fail;
+		bytenr += len;
 	}
 	ret = 0;
+	fsync(fd);
 fail:
+	free(buf);
+	if (ret > 0)
+		ret = -1;
 	return ret;
 }
 
-static int fixup_chunk_mapping(struct btrfs_root *root)
+static int prepare_system_chunk_sb(struct btrfs_super_block *super)
 {
-	struct btrfs_trans_handle *trans;
-	struct btrfs_fs_info *info = root->fs_info;
-	struct btrfs_root *chunk_root = info->chunk_root;
-	struct extent_buffer *leaf;
-	struct btrfs_key key;
-	struct btrfs_path path;
-	struct btrfs_chunk chunk;
-	unsigned long ptr;
-	u32 size;
-	u64 type;
-	int ret;
-
-	btrfs_init_path(&path);
-
-	trans = btrfs_start_transaction(root, 1);
-	BUG_ON(!trans);
-
-	/*
-	 * recow the whole chunk tree. this will move all chunk tree blocks
-	 * into system block group.
-	 */
-	memset(&key, 0, sizeof(key));
-	while (1) {
-		ret = btrfs_search_slot(trans, chunk_root, &key, &path, 0, 1);
-		if (ret < 0)
-			goto err;
-
-		ret = btrfs_next_leaf(chunk_root, &path);
-		if (ret < 0)
-			goto err;
-		if (ret > 0)
-			break;
-
-		btrfs_item_key_to_cpu(path.nodes[0], &key, path.slots[0]);
-		btrfs_release_path(&path);
-	}
-	btrfs_release_path(&path);
-
-	/* fixup the system chunk array in super block */
-	btrfs_set_super_sys_array_size(info->super_copy, 0);
-
-	key.objectid = BTRFS_FIRST_CHUNK_TREE_OBJECTID;
-	key.offset = 0;
-	key.type = BTRFS_CHUNK_ITEM_KEY;
-
-	ret = btrfs_search_slot(trans, chunk_root, &key, &path, 0, 0);
-	if (ret < 0)
-		goto err;
-	BUG_ON(ret != 0);
-	while(1) {
-		leaf = path.nodes[0];
-		if (path.slots[0] >= btrfs_header_nritems(leaf)) {
-			ret = btrfs_next_leaf(chunk_root, &path);
-			if (ret < 0)
-				goto err;
-			if (ret > 0)
-				break;
-			leaf = path.nodes[0];
-		}
-		btrfs_item_key_to_cpu(leaf, &key, path.slots[0]);
-		if (key.type != BTRFS_CHUNK_ITEM_KEY)
-			goto next;
-
-		ptr = btrfs_item_ptr_offset(leaf, path.slots[0]);
-		size = btrfs_item_size_nr(leaf, path.slots[0]);
-		BUG_ON(size != sizeof(chunk));
-		read_extent_buffer(leaf, &chunk, ptr, size);
-		type = btrfs_stack_chunk_type(&chunk);
+	struct btrfs_chunk *chunk;
+	struct btrfs_disk_key *key;
+	u32 sectorsize = btrfs_super_sectorsize(super);
 
-		if (!(type & BTRFS_BLOCK_GROUP_SYSTEM))
-			goto next;
+	key = (struct btrfs_disk_key *)(super->sys_chunk_array);
+	chunk = (struct btrfs_chunk *)(super->sys_chunk_array +
+				       sizeof(struct btrfs_disk_key));
 
-		ret = btrfs_add_system_chunk(trans, chunk_root, &key,
-					     &chunk, size);
-		if (ret)
-			goto err;
-next:
-		path.slots[0]++;
-	}
+	btrfs_set_disk_key_objectid(key, BTRFS_FIRST_CHUNK_TREE_OBJECTID);
+	btrfs_set_disk_key_type(key, BTRFS_CHUNK_ITEM_KEY);
+	btrfs_set_disk_key_offset(key, 0);
 
-	ret = btrfs_commit_transaction(trans, root);
-	BUG_ON(ret);
-err:
-	btrfs_release_path(&path);
-	return ret;
+	btrfs_set_stack_chunk_length(chunk, btrfs_super_total_bytes(super));
+	btrfs_set_stack_chunk_owner(chunk, BTRFS_EXTENT_TREE_OBJECTID);
+	btrfs_set_stack_chunk_stripe_len(chunk, BTRFS_STRIPE_LEN);
+	btrfs_set_stack_chunk_type(chunk, BTRFS_BLOCK_GROUP_SYSTEM);
+	btrfs_set_stack_chunk_io_align(chunk, sectorsize);
+	btrfs_set_stack_chunk_io_width(chunk, sectorsize);
+	btrfs_set_stack_chunk_sector_size(chunk, sectorsize);
+	btrfs_set_stack_chunk_num_stripes(chunk, 1);
+	btrfs_set_stack_chunk_sub_stripes(chunk, 0);
+	chunk->stripe.devid = super->dev_item.devid;
+	btrfs_set_stack_stripe_offset(&chunk->stripe, 0);
+	memcpy(chunk->stripe.dev_uuid, super->dev_item.uuid, BTRFS_UUID_SIZE);
+	btrfs_set_super_sys_array_size(super, sizeof(*key) + sizeof(*chunk));
+	return 0;
 }
 
 static const struct btrfs_convert_operations ext2_convert_ops = {
 	.name			= "ext2",
 	.open_fs		= ext2_open_fs,
 	.read_used_space	= ext2_read_used_space,
-	.alloc_block		= ext2_alloc_block,
-	.alloc_block_range	= ext2_alloc_block_range,
 	.copy_inodes		= ext2_copy_inodes,
-	.test_block		= ext2_test_block,
-	.free_block		= ext2_free_block,
-	.free_block_range	= ext2_free_block_range,
 	.close_fs		= ext2_close_fs,
 };
 
@@ -3220,7 +2113,7 @@ static int convert_read_used_space(struct btrfs_convert_context *cctx)
 	return ret;
 }
 
-static int do_convert_v2(const char *devname, int datacsum, int packing,
+static int do_convert(const char *devname, int datacsum, int packing,
 		int noxattr, u32 nodesize, int copylabel, const char *fslabel,
 		int progress, u64 features)
 {
@@ -3293,7 +2186,7 @@ static int do_convert_v2(const char *devname, int datacsum, int packing,
 		fprintf(stderr, "unable to open ctree\n");
 		goto fail;
 	}
-	ret = init_btrfs_v2(&mkfs_cfg, root, &cctx, datacsum, packing, noxattr);
+	ret = init_btrfs(&mkfs_cfg, root, &cctx, datacsum, packing, noxattr);
 	if (ret) {
 		fprintf(stderr, "unable to setup the root tree\n");
 		goto fail;
@@ -3314,7 +2207,7 @@ static int do_convert_v2(const char *devname, int datacsum, int packing,
 		fprintf(stderr, "unable to create subvol\n");
 		goto fail;
 	}
-	ret = create_image_v2(image_root, &mkfs_cfg, &cctx, fd,
+	ret = create_image(image_root, &mkfs_cfg, &cctx, fd,
 			      mkfs_cfg.num_bytes, "image", datacsum);
 	if (ret) {
 		fprintf(stderr, "error during create_image %d\n", ret);
@@ -3395,220 +2288,6 @@ fail:
 	return -1;
 }
 
-static int do_convert(const char *devname, int datacsum, int packing, int noxattr,
-		u32 nodesize, int copylabel, const char *fslabel, int progress,
-		u64 features)
-{
-	int i, ret, blocks_per_node;
-	int fd = -1;
-	int is_btrfs = 0;
-	u32 blocksize;
-	u64 blocks[7];
-	u64 total_bytes;
-	u64 super_bytenr;
-	struct btrfs_root *root;
-	struct btrfs_root *image_root;
-	struct btrfs_convert_context cctx;
-	char *subvol_name = NULL;
-	struct task_ctx ctx;
-	char features_buf[64];
-	struct btrfs_mkfs_config mkfs_cfg;
-
-	init_convert_context(&cctx);
-	ret = convert_open_fs(devname, &cctx);
-	if (ret)
-		goto fail;
-	ret = convert_read_used_space(&cctx);
-	if (ret)
-		goto fail;
-
-	blocksize = cctx.blocksize;
-	total_bytes = (u64)blocksize * (u64)cctx.block_count;
-	if (blocksize < 4096) {
-		fprintf(stderr, "block size is too small\n");
-		goto fail;
-	}
-	if (btrfs_check_nodesize(nodesize, blocksize, features))
-		goto fail;
-	blocks_per_node = nodesize / blocksize;
-	ret = -blocks_per_node;
-	for (i = 0; i < 7; i++) {
-		if (nodesize == blocksize)
-			ret = convert_alloc_block(&cctx, 0, blocks + i);
-		else
-			ret = convert_alloc_block_range(&cctx,
-					ret + blocks_per_node, blocks_per_node,
-					blocks + i);
-		if (ret) {
-			fprintf(stderr, "not enough free space\n");
-			goto fail;
-		}
-		blocks[i] *= blocksize;
-	}
-	super_bytenr = blocks[0];
-	fd = open(devname, O_RDWR);
-	if (fd < 0) {
-		fprintf(stderr, "unable to open %s\n", devname);
-		goto fail;
-	}
-	btrfs_parse_features_to_string(features_buf, features);
-	if (features == BTRFS_MKFS_DEFAULT_FEATURES)
-		strcat(features_buf, " (default)");
-
-	printf("create btrfs filesystem:\n");
-	printf("\tblocksize: %u\n", blocksize);
-	printf("\tnodesize:  %u\n", nodesize);
-	printf("\tfeatures:  %s\n", features_buf);
-
-	mkfs_cfg.label = cctx.volume_name;
-	mkfs_cfg.fs_uuid = NULL;
-	memcpy(mkfs_cfg.blocks, blocks, sizeof(blocks));
-	mkfs_cfg.num_bytes = total_bytes;
-	mkfs_cfg.nodesize = nodesize;
-	mkfs_cfg.sectorsize = blocksize;
-	mkfs_cfg.stripesize = blocksize;
-	mkfs_cfg.features = features;
-
-	ret = make_btrfs(fd, &mkfs_cfg, NULL);
-	if (ret) {
-		fprintf(stderr, "unable to create initial ctree: %s\n",
-			strerror(-ret));
-		goto fail;
-	}
-	/* create a system chunk that maps the whole device */
-	ret = prepare_system_chunk(fd, super_bytenr);
-	if (ret) {
-		fprintf(stderr, "unable to update system chunk\n");
-		goto fail;
-	}
-	root = open_ctree_fd(fd, devname, super_bytenr, OPEN_CTREE_WRITES);
-	if (!root) {
-		fprintf(stderr, "unable to open ctree\n");
-		goto fail;
-	}
-	ret = cache_free_extents(root, &cctx);
-	if (ret) {
-		fprintf(stderr, "error during cache_free_extents %d\n", ret);
-		goto fail;
-	}
-	root->fs_info->extent_ops = &extent_ops;
-	/* recover block allocation bitmap */
-	for (i = 0; i < 7; i++) {
-		blocks[i] /= blocksize;
-		if (nodesize == blocksize)
-			convert_free_block(&cctx, blocks[i]);
-		else
-			convert_free_block_range(&cctx, blocks[i],
-					blocks_per_node);
-	}
-	ret = init_btrfs(root);
-	if (ret) {
-		fprintf(stderr, "unable to setup the root tree\n");
-		goto fail;
-	}
-	printf("creating btrfs metadata.\n");
-	ctx.max_copy_inodes = (cctx.inodes_count - cctx.free_inodes_count);
-	ctx.cur_copy_inodes = 0;
-
-	if (progress) {
-		ctx.info = task_init(print_copied_inodes, after_copied_inodes, &ctx);
-		task_start(ctx.info);
-	}
-	ret = copy_inodes(&cctx, root, datacsum, packing, noxattr, &ctx);
-	if (ret) {
-		fprintf(stderr, "error during copy_inodes %d\n", ret);
-		goto fail;
-	}
-	if (progress) {
-		task_stop(ctx.info);
-		task_deinit(ctx.info);
-	}
-
-	printf("creating %s image file.\n", cctx.convert_ops->name);
-	ret = asprintf(&subvol_name, "%s_saved", cctx.convert_ops->name);
-	if (ret < 0) {
-		fprintf(stderr, "error allocating subvolume name: %s_saved\n",
-			cctx.convert_ops->name);
-		goto fail;
-	}
-
-	image_root = link_subvol(root, subvol_name, CONV_IMAGE_SUBVOL_OBJECTID);
-
-	free(subvol_name);
-
-	if (!image_root) {
-		fprintf(stderr, "unable to create subvol\n");
-		goto fail;
-	}
-	ret = create_image(&cctx, image_root, "image", datacsum);
-	if (ret) {
-		fprintf(stderr, "error during create_image %d\n", ret);
-		goto fail;
-	}
-	memset(root->fs_info->super_copy->label, 0, BTRFS_LABEL_SIZE);
-	if (copylabel == 1) {
-		strncpy(root->fs_info->super_copy->label,
-				cctx.volume_name, BTRFS_LABEL_SIZE);
-		fprintf(stderr, "copy label '%s'\n",
-				root->fs_info->super_copy->label);
-	} else if (copylabel == -1) {
-		strcpy(root->fs_info->super_copy->label, fslabel);
-		fprintf(stderr, "set label to '%s'\n", fslabel);
-	}
-
-	printf("cleaning up system chunk.\n");
-	ret = cleanup_sys_chunk(root, image_root);
-	if (ret) {
-		fprintf(stderr, "error during cleanup_sys_chunk %d\n", ret);
-		goto fail;
-	}
-	ret = close_ctree(root);
-	if (ret) {
-		fprintf(stderr, "error during close_ctree %d\n", ret);
-		goto fail;
-	}
-	convert_close_fs(&cctx);
-	clean_convert_context(&cctx);
-
-	/*
-	 * If this step succeed, we get a mountable btrfs. Otherwise
-	 * the source fs is left unchanged.
-	 */
-	ret = migrate_super_block(fd, super_bytenr, blocksize);
-	if (ret) {
-		fprintf(stderr, "unable to migrate super block\n");
-		goto fail;
-	}
-	is_btrfs = 1;
-
-	root = open_ctree_fd(fd, devname, 0, OPEN_CTREE_WRITES);
-	if (!root) {
-		fprintf(stderr, "unable to open ctree\n");
-		goto fail;
-	}
-	/* move chunk tree into system chunk. */
-	ret = fixup_chunk_mapping(root);
-	if (ret) {
-		fprintf(stderr, "error during fixup_chunk_tree\n");
-		goto fail;
-	}
-	ret = close_ctree(root);
-	close(fd);
-
-	printf("conversion complete.\n");
-	return 0;
-fail:
-	clean_convert_context(&cctx);
-	if (fd != -1)
-		close(fd);
-	if (is_btrfs)
-		fprintf(stderr,
-			"WARNING: an error occured during chunk mapping fixup, filesystem mountable but not finalized\n");
-	else
-		fprintf(stderr, "conversion aborted\n");
-	return -1;
-}
-
 static int may_rollback(struct btrfs_root *root)
 {
 	struct btrfs_fs_info *info = root->fs_info;
@@ -4153,7 +2832,7 @@ int main(int argc, char *argv[])
 	if (rollback) {
 		ret = do_rollback(file);
 	} else {
-		ret = do_convert_v2(file, datacsum, packing, noxattr, nodesize,
+		ret = do_convert(file, datacsum, packing, noxattr, nodesize,
 				copylabel, fslabel, progress, features);
 	}
 	if (ret)
diff --git a/ctree.h b/ctree.h
index 187bd27..257bd01 100644
--- a/ctree.h
+++ b/ctree.h
@@ -959,13 +959,6 @@ struct btrfs_block_group_cache {
 	int ro;
 };
 
-struct btrfs_extent_ops {
-       int (*alloc_extent)(struct btrfs_root *root, u64 num_bytes,
-			   u64 hint_byte, struct btrfs_key *ins, int metadata);
-       int (*free_extent)(struct btrfs_root *root, u64 bytenr,
-		          u64 num_bytes);
-};
-
 struct btrfs_device;
 struct btrfs_fs_devices;
 struct btrfs_fs_info {
@@ -1016,7 +1009,6 @@ struct btrfs_fs_info {
 	u64 super_bytenr;
 	u64 total_pinned;
 
-	struct btrfs_extent_ops *extent_ops;
 	struct list_head dirty_cowonly_roots;
 	struct list_head recow_ebs;
 
diff --git a/extent-tree.c b/extent-tree.c
index e7c61b1..aba34e1 100644
--- a/extent-tree.c
+++ b/extent-tree.c
@@ -2181,7 +2181,6 @@ static int __free_extent(struct btrfs_trans_handle *trans,
 
 	struct btrfs_key key;
 	struct btrfs_path *path;
-	struct btrfs_extent_ops *ops = root->fs_info->extent_ops;
 	struct btrfs_root *extent_root = root->fs_info->extent_root;
 	struct extent_buffer *leaf;
 	struct btrfs_extent_item *ei;
@@ -2382,14 +2381,6 @@ static int __free_extent(struct btrfs_trans_handle *trans,
 			}
 		}
 
-		if (ops && ops->free_extent) {
-			ret = ops->free_extent(root, bytenr, num_bytes);
-			if (ret > 0) {
-				pin = 0;
-				mark_free = 0;
-			}
-		}
-
 		if (pin) {
 			ret = pin_down_bytes(trans, root, bytenr, num_bytes,
 					     is_data);
@@ -2662,13 +2653,6 @@ int btrfs_reserve_extent(struct btrfs_trans_handle *trans,
 	u64 alloc_profile;
 	struct btrfs_fs_info *info = root->fs_info;
 
-	if (info->extent_ops) {
-		struct btrfs_extent_ops *ops = info->extent_ops;
-		ret = ops->alloc_extent(root, num_bytes, hint_byte, ins, !data);
-		BUG_ON(ret);
-		goto found;
-	}
-
 	if (data) {
 		alloc_profile = info->avail_data_alloc_bits &
 			        info->data_alloc_profile;
@@ -2702,7 +2686,6 @@ int btrfs_reserve_extent(struct btrfs_trans_handle *trans,
 			       trans->alloc_exclude_start,
 			       trans->alloc_exclude_nr, data);
 	BUG_ON(ret);
-found:
 	clear_extent_dirty(&root->fs_info->free_space_cache,
 			   ins->objectid, ins->objectid + ins->offset - 1,
 			   GFP_NOFS);
-- 
2.7.0

Re: [PATCH v3 00/22] Btrfs-convert rework to support separate chunk type

[David Sterba]

On Fri, Jan 29, 2016 at 01:03:10PM +0800, Qu Wenruo wrote:
> v3:
>   Rebased to latest devel branch.
>   Fix a btrfs patch leak.

I'll add that to the integration branch, review of this patchset is
stalled.

Re: [PATCH v3 01/22] btrfs-progs: convert: Introduce functions to read used space

[David Sterba]

On Fri, Jan 29, 2016 at 01:03:11PM +0800, Qu Wenruo wrote:
> Before we do real convert, we need to read and build up used space cache
> tree for later data/meta separate chunk layout.
> 
> This patch will iterate all used blocks in ext2 filesystem and record it
> into cctx->used cache tree, for later use.
> 
> This provides the very basic of later btrfs-convert rework.
> 
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Signed-off-by: David Sterba <dsterba@suse.com>
> ---
>  btrfs-convert.c | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 80 insertions(+)
> 
> diff --git a/btrfs-convert.c b/btrfs-convert.c
> index 4baa68e..65841bd 100644
> --- a/btrfs-convert.c
> +++ b/btrfs-convert.c
> @@ -81,6 +81,7 @@ struct btrfs_convert_context;
>  struct btrfs_convert_operations {
>  	const char *name;
>  	int (*open_fs)(struct btrfs_convert_context *cctx, const char *devname);
> +	int (*read_used_space)(struct btrfs_convert_context *cctx);
>  	int (*alloc_block)(struct btrfs_convert_context *cctx, u64 goal,
>  			   u64 *block_ret);
>  	int (*alloc_block_range)(struct btrfs_convert_context *cctx, u64 goal,
> @@ -230,6 +231,73 @@ fail:
>  	return -1;
>  }
>  
> +static int __ext2_add_one_block(ext2_filsys fs, char *bitmap,
> +				unsigned long group_nr, struct cache_tree *used)
> +{
> +	unsigned long offset;
> +	unsigned i;
> +	int ret = 0;
> +
> +	offset = fs->super->s_first_data_block;
> +	offset /= EXT2FS_CLUSTER_RATIO(fs);

This macro does not exist on my reference host for old distros. The
e2fsprogs version is 1.41.14 and I'd like to keep the compatibility at
least at that level.

The clustering has been added in 1.42 but can we add some compatibility
layer that will work on both version?

Re: [PATCH v3 01/22] btrfs-progs: convert: Introduce functions to read used space

[Qu Wenruo]

David Sterba wrote on 2016/04/04 15:35 +0200:
> On Fri, Jan 29, 2016 at 01:03:11PM +0800, Qu Wenruo wrote:
>> Before we do real convert, we need to read and build up used space cache
>> tree for later data/meta separate chunk layout.
>>
>> This patch will iterate all used blocks in ext2 filesystem and record it
>> into cctx->used cache tree, for later use.
>>
>> This provides the very basic of later btrfs-convert rework.
>>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> Signed-off-by: David Sterba <dsterba@suse.com>
>> ---
>>   btrfs-convert.c | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 80 insertions(+)
>>
>> diff --git a/btrfs-convert.c b/btrfs-convert.c
>> index 4baa68e..65841bd 100644
>> --- a/btrfs-convert.c
>> +++ b/btrfs-convert.c
>> @@ -81,6 +81,7 @@ struct btrfs_convert_context;
>>   struct btrfs_convert_operations {
>>   	const char *name;
>>   	int (*open_fs)(struct btrfs_convert_context *cctx, const char *devname);
>> +	int (*read_used_space)(struct btrfs_convert_context *cctx);
>>   	int (*alloc_block)(struct btrfs_convert_context *cctx, u64 goal,
>>   			   u64 *block_ret);
>>   	int (*alloc_block_range)(struct btrfs_convert_context *cctx, u64 goal,
>> @@ -230,6 +231,73 @@ fail:
>>   	return -1;
>>   }
>>
>> +static int __ext2_add_one_block(ext2_filsys fs, char *bitmap,
>> +				unsigned long group_nr, struct cache_tree *used)
>> +{
>> +	unsigned long offset;
>> +	unsigned i;
>> +	int ret = 0;
>> +
>> +	offset = fs->super->s_first_data_block;
>> +	offset /= EXT2FS_CLUSTER_RATIO(fs);
>
> This macro does not exist on my reference host for old distros. The
> e2fsprogs version is 1.41.14 and I'd like to keep the compatibility at
> least at that level.
>
> The clustering has been added in 1.42 but can we add some compatibility
> layer that will work on both version?
>
No problem.

It's a simple macro. For older version which doesn't provide it, we can 
just define it in btrfs-covert.c.

Thanks,
Qu

Re: [PATCH v3 05/22] btrfs-progs: Introduce function to setup temporary superblock

[Liu Bo]

On Fri, Jan 29, 2016 at 01:03:15PM +0800, Qu Wenruo wrote:
> Introduce a new function, setup_temp_super(), to setup temporary super
> for make_btrfs_v2().
> 
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Signed-off-by: David Sterba <dsterba@suse.com>
> ---
>  utils.c | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 117 insertions(+)
> 
> diff --git a/utils.c b/utils.c
> index bc10293..ed5476d 100644
> --- a/utils.c
> +++ b/utils.c
> @@ -212,6 +212,98 @@ static int reserve_free_space(struct cache_tree *free_tree, u64 len,
>  	return 0;
>  }
>  
> +static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
> +				   u64 sb_bytenr)
> +{
> +	u32 crc = ~(u32)0;
> +	int ret;
> +
> +	crc = btrfs_csum_data(NULL, (char *)sb + BTRFS_CSUM_SIZE, crc,
> +			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> +	btrfs_csum_final(crc, (char *)&sb->csum[0]);
> +	ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
> +	if (ret < BTRFS_SUPER_INFO_SIZE)
> +		ret = (ret < 0 ? -errno : -EIO);
> +	else
> +		ret = 0;
> +	return ret;
> +}
> +
> +/*
> + * Setup temporary superblock at cfg->super_bynter
> + * Needed info are extracted from cfg, and root_bytenr, chunk_bytenr
> + *
> + * For now sys chunk array will be empty and dev_item is empty
> + * too.
> + * They will be re-initialized at temp chunk tree setup.
> + */
> +static int setup_temp_super(int fd, struct btrfs_mkfs_config *cfg,
> +			    u64 root_bytenr, u64 chunk_bytenr)
> +{
> +	unsigned char chunk_uuid[BTRFS_UUID_SIZE];
> +	char super_buf[BTRFS_SUPER_INFO_SIZE];
> +	struct btrfs_super_block *super = (struct btrfs_super_block *)super_buf;
> +	int ret;
> +
> +	/*
> +	 * We rely on cfg->chunk_uuid and cfg->fs_uuid to pass uuid
> +	 * for other functions.
> +	 * Caller must allocation space for them
> +	 */
> +	BUG_ON(!cfg->chunk_uuid || !cfg->fs_uuid);
> +	memset(super_buf, 0, BTRFS_SUPER_INFO_SIZE);
> +	cfg->num_bytes = round_down(cfg->num_bytes, cfg->sectorsize);
> +
> +	if (cfg->fs_uuid && *cfg->fs_uuid) {
> +		if (uuid_parse(cfg->fs_uuid, super->fsid) != 0) {
> +			error("cound not parse UUID: %s", cfg->fs_uuid);
> +			ret = -EINVAL;
> +			goto out;
> +		}
> +		if (!test_uuid_unique(cfg->fs_uuid)) {
> +			error("non-unique UUID: %s", cfg->fs_uuid);
> +			ret = -EINVAL;
> +			goto out;
> +		}
> +	} else {
> +		uuid_generate(super->fsid);
> +		uuid_unparse(super->fsid, cfg->fs_uuid);
> +	}
> +	uuid_generate(chunk_uuid);
> +	uuid_unparse(chunk_uuid, cfg->chunk_uuid);
> +
> +	btrfs_set_super_bytenr(super, cfg->super_bytenr);
> +	btrfs_set_super_num_devices(super, 1);
> +	btrfs_set_super_magic(super, BTRFS_MAGIC);
> +	btrfs_set_super_generation(super, 1);
> +	btrfs_set_super_root(super, root_bytenr);
> +	btrfs_set_super_chunk_root(super, chunk_bytenr);
> +	btrfs_set_super_total_bytes(super, cfg->num_bytes);
> +	/*
> +	 * Temporary btrfs will only has 6 tree roots:
> +	 * chunk tree, root tree, extent_tree, device tree, fs tree
> +	 * and csum tree.
> +	 */
> +	btrfs_set_super_bytes_used(super, 6 * cfg->nodesize);
> +	btrfs_set_super_sectorsize(super, cfg->sectorsize);
> +	btrfs_set_super_leafsize(super, cfg->nodesize);
> +	btrfs_set_super_nodesize(super, cfg->nodesize);
> +	btrfs_set_super_stripesize(super, cfg->stripesize);
> +	btrfs_set_super_csum_type(super, BTRFS_CSUM_TYPE_CRC32);
> +	btrfs_set_super_chunk_root(super, chunk_bytenr);
> +	btrfs_set_super_cache_generation(super, -1);
> +	btrfs_set_super_incompat_flags(super, cfg->features);
> +	if (cfg->label)
> +		strncpy(super->label, cfg->label, BTRFS_LABEL_SIZE - 1);

Why not use __strncpy_null?

Thanks,

-liubo

> +
> +	/* Sys chunk array will be re-initialized at chunk tree init time */
> +	super->sys_chunk_array_size = 0;
> +
> +	ret = write_temp_super(fd, super, cfg->super_bytenr);
> +out:
> +	return ret;
> +}
> +
>  /*
>   * Improved version of make_btrfs().
>   *
> @@ -230,6 +322,10 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
>  	struct cache_tree *used = &cctx->used;
>  	u64 sys_chunk_start;
>  	u64 meta_chunk_start;
> +	/* chunk tree bytenr, in system chunk */
> +	u64 chunk_bytenr;
> +	/* metadata trees bytenr, in metadata chunk */
> +	u64 root_bytenr;
>  	int ret;
>  
>  	/* Shouldn't happen */
> @@ -260,6 +356,27 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
>  	if (ret < 0)
>  		goto out;
>  
> +	/*
> +	 * Inside the allocate metadata chunk, its layout will be:
> +	 *  | offset		| contents	|
> +	 *  -------------------------------------
> +	 *  | +0		| tree root	|
> +	 *  | +nodesize		| extent root	|
> +	 *  | +nodesize * 2	| device root	|
> +	 *  | +nodesize * 3	| fs tree	|
> +	 *  | +nodesize * 4	| csum tree	|
> +	 *  -------------------------------------
> +	 * Inside the allocated system chunk, its layout will be:
> +	 *  | offset		| contents	|
> +	 *  -------------------------------------
> +	 *  | +0		| chunk root	|
> +	 *  -------------------------------------
> +	 */
> +	chunk_bytenr = sys_chunk_start;
> +	root_bytenr = meta_chunk_start;
> +	ret = setup_temp_super(fd, cfg, root_bytenr, chunk_bytenr);
> +	if (ret < 0)
> +		goto out;
>  out:
>  	return ret;
>  }
> -- 
> 2.7.0
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 15/22] btrfs-progs: convert: Introduce new function to create converted image

[Liu Bo]

On Fri, Jan 29, 2016 at 01:03:25PM +0800, Qu Wenruo wrote:
> Use new function, create_convert_image_v2() to create snapshot of old
> filesystem.
> 
> Unlike old function which is called after copying all inodes, this
> function need to be called before copying inodes.
> 
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Signed-off-by: David Sterba <dsterba@suse.com>
> ---
>  btrfs-convert.c | 184 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 184 insertions(+)
> 
> diff --git a/btrfs-convert.c b/btrfs-convert.c
> index 102953a..16e2309 100644
> --- a/btrfs-convert.c
> +++ b/btrfs-convert.c
> @@ -1412,6 +1412,7 @@ static int create_image_file_range(struct btrfs_trans_handle *trans,
>  fail:
>  	return ret;
>  }
> +
>  /*
>   * Create the fs image file.
>   */
> @@ -1605,6 +1606,189 @@ fail:
>  	return ret;
>  }
>  
> +static int create_image_file_range_v2(struct btrfs_trans_handle *trans,
> +				      struct btrfs_root *root,
> +				      struct cache_tree *used,
> +				      struct btrfs_inode_item *inode,
> +				      u64 ino, u64 bytenr, u64 *ret_len,
> +				      int datacsum)
> +{
> +	struct cache_extent *cache;
> +	struct btrfs_block_group_cache *bg_cache;
> +	u64 len = *ret_len;
> +	u64 disk_bytenr;
> +	int ret;
> +
> +	BUG_ON(bytenr != round_down(bytenr, root->sectorsize));
> +	BUG_ON(len != round_down(len, root->sectorsize));
> +	len = min_t(u64, len, BTRFS_MAX_EXTENT_SIZE);
> +
> +	cache = search_cache_extent(used, bytenr);
> +	if (cache) {
> +		if (cache->start <= bytenr) {
> +			/*
> +			 * |///////Used///////|
> +			 *	|<--insert--->|
> +			 *	bytenr
> +			 */
> +			len = min_t(u64, len, cache->start + cache->size -
> +				    bytenr);
> +			disk_bytenr = bytenr;
> +		} else {
> +			/*
> +			 *		|//Used//|
> +			 *  |<-insert-->|
> +			 *  bytenr
> +			 */
> +			len = min(len, cache->start - bytenr);
> +			disk_bytenr = 0;
> +			datacsum = 0;
> +		}
> +	} else {
> +		/*
> +		 * |//Used//|		|EOF
> +		 *	    |<-insert-->|
> +		 *	    bytenr
> +		 */
> +		disk_bytenr = 0;
> +		datacsum = 0;
> +	}
> +
> +	if (disk_bytenr) {
> +		/* Check if the range is in a data block group */
> +		bg_cache = btrfs_lookup_block_group(root->fs_info, bytenr);
> +		if (!bg_cache)
> +			return -ENOENT;
> +		if (!(bg_cache->flags & BTRFS_BLOCK_GROUP_DATA))
> +			return -EINVAL;
> +
> +		/* The extent should never cross block group boundary */
> +		len = min_t(u64, len, bg_cache->key.objectid +
> +			    bg_cache->key.offset - bytenr);
> +	}
> +
> +	BUG_ON(len != round_down(len, root->sectorsize));
> +	ret = btrfs_record_file_extent(trans, root, ino, inode, bytenr,
> +				       disk_bytenr, len);
> +	if (ret < 0)
> +		return ret;
> +
> +	if (datacsum)
> +		ret = csum_disk_extent(trans, root, bytenr, len);
> +	*ret_len = len;
> +	return ret;
> +}
> +
> +static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
> +				int ensure_size);
> +
> +/*
> + * Create the fs image file of old filesystem.
> + *
> + * This is completely fs independent as we have cctx->used, only
> + * need to create file extents point to all the posistions.

s/posistions/positions/g


> + * TODO: Add handler for reserved ranges in next patch
> + */
> +static int create_image_v2(struct btrfs_root *root,
> +			   struct btrfs_mkfs_config *cfg,
> +			   struct btrfs_convert_context *cctx,
> +			   u64 size, char *name, int datacsum)
> +{
> +	struct btrfs_inode_item buf;
> +	struct btrfs_trans_handle *trans;
> +	struct btrfs_path *path = NULL;
> +	struct btrfs_key key;
> +	struct cache_extent *cache;
> +	struct cache_tree used_tmp;
> +	u64 cur;
> +	u64 ino;
> +	int ret;
> +
> +	trans = btrfs_start_transaction(root, 1);
> +	if (!trans)
> +		return -ENOMEM;
> +
> +	cache_tree_init(&used_tmp);
> +
> +	ret = btrfs_find_free_objectid(trans, root, BTRFS_FIRST_FREE_OBJECTID,
> +				       &ino);
> +	if (ret < 0)
> +		goto out;
> +	ret = btrfs_new_inode(trans, root, ino, 0600 | S_IFREG);
> +	if (ret < 0)
> +		goto out;
> +	ret = btrfs_add_link(trans, root, ino, BTRFS_FIRST_FREE_OBJECTID, name,
> +			     strlen(name), BTRFS_FT_REG_FILE, NULL, 1);
> +	if (ret < 0)
> +		goto out;
> +
> +	path = btrfs_alloc_path();
> +	if (!path) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +	key.objectid = ino;
> +	key.type = BTRFS_INODE_ITEM_KEY;
> +	key.offset = 0;
> +
> +	ret = btrfs_search_slot(trans, root, &key, path, 0, 1);
> +	if (ret) {
> +		ret = (ret > 0 ? -ENOENT : ret);
> +		goto out;
> +	}
> +	read_extent_buffer(path->nodes[0], &buf,
> +			btrfs_item_ptr_offset(path->nodes[0], path->slots[0]),
> +			sizeof(buf));
> +	btrfs_release_path(path);

It seems unnecessary to search for inode_item, inode_item is
initialized by fill_inode_item(), so fill_inode_item(...&buf...) is fine.

> +
> +	/*
> +	 * Create a new used space cache, which doesn't contain the reserved
> +	 * range
> +	 */
> +	for (cache = first_cache_extent(&cctx->used); cache;
> +	     cache = next_cache_extent(cache)) {
> +		ret = add_cache_extent(&used_tmp, cache->start, cache->size);
> +		if (ret < 0)
> +			goto out;
> +	}

It's not clear why we need used_tmp.

Thanks,

-liubo
> +	ret = wipe_reserved_ranges(&used_tmp, 0, 0);
> +	if (ret < 0)
> +		goto out;
> +
> +	/*
> +	 * Start from 1M, as 0~1M is reserved, and create_image_file_range_v2()
> +	 * can't handle bytenr 0(will consider it as a hole)
> +	 */
> +	cur = 1024 * 1024;
> +	while (cur < size) {
> +		u64 len = size - cur;
> +
> +		ret = create_image_file_range_v2(trans, root, &used_tmp,
> +						&buf, ino, cur, &len, datacsum);
> +		if (ret < 0)
> +			goto out;
> +		cur += len;
> +	}
> +
> +	key.objectid = ino;
> +	key.type = BTRFS_INODE_ITEM_KEY;
> +	key.offset = 0;
> +	ret = btrfs_search_slot(trans, root, &key, path, 0, 1);
> +	if (ret) {
> +		ret = (ret > 0 ? -ENOENT : ret);
> +		goto out;
> +	}
> +	btrfs_set_stack_inode_size(&buf, cfg->num_bytes);
> +	write_extent_buffer(path->nodes[0], &buf,
> +			btrfs_item_ptr_offset(path->nodes[0], path->slots[0]),
> +			sizeof(buf));
> +out:
> +	free_extent_cache_tree(&used_tmp);
> +	btrfs_free_path(path);
> +	btrfs_commit_transaction(trans, root);
> +	return ret;
> +}
> +
>  static struct btrfs_root * link_subvol(struct btrfs_root *root,
>  		const char *base, u64 root_objectid)
>  {
> -- 
> 2.7.0
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 16/22] btrfs-progs: convert: Introduce function to migrate reserved ranges

[Liu Bo]

On Fri, Jan 29, 2016 at 01:03:26PM +0800, Qu Wenruo wrote:
> Introduce new function, migrate_reserved_ranges() to migrate used fs
> data in btrfs reserved space.
> 
> Unlike old implement, which will need to relocate all the complicated
> csum and reference relocation, previous patches already ensure such
> reserved ranges won't be allocated.
> So here we only need copy these data out and create new
> extent/csum/reference.
> 
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Signed-off-by: David Sterba <dsterba@suse.com>
> ---
>  btrfs-convert.c | 124 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 122 insertions(+), 2 deletions(-)
> 
> diff --git a/btrfs-convert.c b/btrfs-convert.c
> index 16e2309..f6126db 100644
> --- a/btrfs-convert.c
> +++ b/btrfs-convert.c
> @@ -1679,6 +1679,123 @@ static int create_image_file_range_v2(struct btrfs_trans_handle *trans,
>  	return ret;
>  }
>  
> +
> +/*
> + * Relocate old fs data in one reserved ranges
> + *
> + * Since all old fs data in reserved range is not covered by any chunk nor
> + * data extent, we don't need to handle any reference but add new
> + * extent/reference, which makes codes more clear
> + */
> +static int migrate_one_reserved_range(struct btrfs_trans_handle *trans,
> +				      struct btrfs_root *root,
> +				      struct cache_tree *used,
> +				      struct btrfs_inode_item *inode, int fd,
> +				      u64 ino, u64 start, u64 len, int datacsum)
> +{
> +	u64 cur_off = start;
> +	u64 cur_len = len;
> +	struct cache_extent *cache;
> +	struct btrfs_key key;
> +	struct extent_buffer *eb;
> +	int ret = 0;
> +
> +	while (cur_off < start + len) {
> +		cache = lookup_cache_extent(used, cur_off, cur_len);
> +		if (!cache)
> +			break;
> +		cur_off = max(cache->start, cur_off);
> +		cur_len = min(cache->start + cache->size, start + len) -
> +			  cur_off;
> +		BUG_ON(cur_len < root->sectorsize);
> +
> +		/* reserve extent for the data */
> +		ret = btrfs_reserve_extent(trans, root, cur_len, 0, 0, (u64)-1,
> +					   &key, 1);
> +		if (ret < 0)
> +			break;
> +
> +		eb = malloc(sizeof(*eb) + cur_len);
> +		if (!eb) {
> +			ret = -ENOMEM;
> +			break;
> +		}
> +
> +		ret = pread(fd, eb->data, cur_len, cur_off);
> +		if (ret < cur_len) {
> +			ret = (ret < 0 ? ret : -EIO);
> +			free(eb);
> +			break;
> +		}
> +		eb->start = key.objectid;
> +		eb->len = key.offset;
> +
> +		/* Write the data */
> +		ret = write_and_map_eb(trans, root, eb);
> +		free(eb);
> +		if (ret < 0)
> +			break;

With write_data_to_disk(), we don't have to create eb for write.

Thanks,

-liubo
> +
> +		/* Now handle extent item and file extent things */
> +		ret = btrfs_record_file_extent(trans, root, ino, inode, cur_off,
> +					       key.objectid, key.offset);
> +		if (ret < 0)
> +			break;
> +		/* Finally, insert csum items */
> +		if (datacsum)
> +			ret = csum_disk_extent(trans, root, key.objectid,
> +					       key.offset);
> +
> +		cur_off += key.offset;
> +		cur_len = start + len - cur_off;
> +	}
> +	return ret;
> +}
> +
> +/*
> + * Relocate the used ext2 data in reserved ranges
> + * [0,1M)
> + * [btrfs_sb_offset(1), +BTRFS_STRIPE_LEN)
> + * [btrfs_sb_offset(2), +BTRFS_STRIPE_LEN)
> + */
> +static int migrate_reserved_ranges(struct btrfs_trans_handle *trans,
> +				   struct btrfs_root *root,
> +				   struct cache_tree *used,
> +				   struct btrfs_inode_item *inode, int fd,
> +				   u64 ino, u64 total_bytes, int datacsum)
> +{
> +	u64 cur_off;
> +	u64 cur_len;
> +	int ret = 0;
> +
> +	/* 0 ~ 1M */
> +	cur_off = 0;
> +	cur_len = 1024 * 1024;
> +	ret = migrate_one_reserved_range(trans, root, used, inode, fd, ino,
> +					 cur_off, cur_len, datacsum);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* second sb(fisrt sb is included in 0~1M) */
> +	cur_off = btrfs_sb_offset(1);
> +	cur_len = min(total_bytes, cur_off + BTRFS_STRIPE_LEN) - cur_off;
> +	if (cur_off < total_bytes)
> +		return ret;
> +	ret = migrate_one_reserved_range(trans, root, used, inode, fd, ino,
> +					 cur_off, cur_len, datacsum);
> +	if (ret < 0)
> +		return ret;
> +
> +	/* Last sb */
> +	cur_off = btrfs_sb_offset(2);
> +	cur_len = min(total_bytes, cur_off + BTRFS_STRIPE_LEN) - cur_off;
> +	if (cur_off < total_bytes)
> +		return ret;
> +	ret = migrate_one_reserved_range(trans, root, used, inode, fd, ino,
> +					 cur_off, cur_len, datacsum);
> +	return ret;
> +}
> +
>  static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
>  				int ensure_size);
>  
> @@ -1687,11 +1804,10 @@ static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
>   *
>   * This is completely fs independent as we have cctx->used, only
>   * need to create file extents point to all the posistions.
> - * TODO: Add handler for reserved ranges in next patch
>   */
>  static int create_image_v2(struct btrfs_root *root,
>  			   struct btrfs_mkfs_config *cfg,
> -			   struct btrfs_convert_context *cctx,
> +			   struct btrfs_convert_context *cctx, int fd,
>  			   u64 size, char *name, int datacsum)
>  {
>  	struct btrfs_inode_item buf;
> @@ -1769,6 +1885,10 @@ static int create_image_v2(struct btrfs_root *root,
>  			goto out;
>  		cur += len;
>  	}
> +	/* Handle the reserved ranges */
> +	ret = migrate_reserved_ranges(trans, root, &cctx->used, &buf, fd, ino,
> +				      cfg->num_bytes, datacsum);
> +
>  
>  	key.objectid = ino;
>  	key.type = BTRFS_INODE_ITEM_KEY;
> -- 
> 2.7.0
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 21/22] btrfs-progs: convert: Strictly avoid meta or system chunk allocation

[Liu Bo]

On Fri, Jan 29, 2016 at 01:03:31PM +0800, Qu Wenruo wrote:
> Before this patch, btrfs-convert only rely on large enough initial
> system/metadata chunk size to ensure no newer system/meta chunk will be
> created.
> 
> But that's not safe enough. So add two new members in fs_info,
> avoid_sys/meta_chunk_alloc flags to prevent any newer system or meta
> chunks to be created before init_btrfs_v2().
> 
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Signed-off-by: David Sterba <dsterba@suse.com>
> ---
>  btrfs-convert.c |  9 +++++++++
>  ctree.h         |  3 +++
>  extent-tree.c   | 10 ++++++++++
>  3 files changed, 22 insertions(+)
> 
> diff --git a/btrfs-convert.c b/btrfs-convert.c
> index efa3b02..333f413 100644
> --- a/btrfs-convert.c
> +++ b/btrfs-convert.c
> @@ -2322,6 +2322,13 @@ static int init_btrfs_v2(struct btrfs_mkfs_config *cfg, struct btrfs_root *root,
>  	struct btrfs_fs_info *fs_info = root->fs_info;
>  	int ret;
>  
> +	/*
> +	 * Don't alloc any metadata/system chunk, as we don't want
> +	 * any meta/sys chunk allcated before all data chunks are inserted.
> +	 * Or we screw up the chunk layout just like the old implement.
> +	 */

I don't get this, with this patch set, we can allocate data from DATA chunk,
allocate metadata from METADATA chunk, but then we're not allowed to allocate new chunks?

Thanks,

-liubo
> +	fs_info->avoid_sys_chunk_alloc = 1;
> +	fs_info->avoid_meta_chunk_alloc = 1;
>  	trans = btrfs_start_transaction(root, 1);
>  	BUG_ON(!trans);
>  	ret = btrfs_fix_block_accounting(trans, root);
> @@ -2359,6 +2366,8 @@ static int init_btrfs_v2(struct btrfs_mkfs_config *cfg, struct btrfs_root *root,
>  		goto err;
>  
>  	ret = btrfs_commit_transaction(trans, root);
> +	fs_info->avoid_sys_chunk_alloc = 0;
> +	fs_info->avoid_meta_chunk_alloc = 0;
>  err:
>  	return ret;
>  }
> diff --git a/ctree.h b/ctree.h
> index 1443746..187bd27 100644
> --- a/ctree.h
> +++ b/ctree.h
> @@ -1030,6 +1030,9 @@ struct btrfs_fs_info {
>  	unsigned int quota_enabled:1;
>  	unsigned int suppress_check_block_errors:1;
>  	unsigned int ignore_fsid_mismatch:1;
> +	unsigned int avoid_meta_chunk_alloc:1;
> +	unsigned int avoid_sys_chunk_alloc:1;
> +
>  
>  	int (*free_extent_hook)(struct btrfs_trans_handle *trans,
>  				struct btrfs_root *root,
> diff --git a/extent-tree.c b/extent-tree.c
> index 93b1945..e7c61b1 100644
> --- a/extent-tree.c
> +++ b/extent-tree.c
> @@ -1904,6 +1904,16 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
>  	    thresh)
>  		return 0;
>  
> +	/*
> +	 * Avoid allocating given chunk type
> +	 */
> +	if (extent_root->fs_info->avoid_meta_chunk_alloc &&
> +	    (flags & BTRFS_BLOCK_GROUP_METADATA))
> +		return 0;
> +	if (extent_root->fs_info->avoid_sys_chunk_alloc &&
> +	    (flags & BTRFS_BLOCK_GROUP_SYSTEM))
> +		return 0;
> +
>  	ret = btrfs_alloc_chunk(trans, extent_root, &start, &num_bytes,
>  	                        space_info->flags);
>  	if (ret == -ENOSPC) {
> -- 
> 2.7.0
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v3 05/22] btrfs-progs: Introduce function to setup temporary superblock

[Qu Wenruo]

On 05/28/2016 11:04 AM, Liu Bo wrote:
> On Fri, Jan 29, 2016 at 01:03:15PM +0800, Qu Wenruo wrote:
>> Introduce a new function, setup_temp_super(), to setup temporary super
>> for make_btrfs_v2().
>>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> Signed-off-by: David Sterba <dsterba@suse.com>
>> ---
>>  utils.c | 117 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 117 insertions(+)
>>
>> diff --git a/utils.c b/utils.c
>> index bc10293..ed5476d 100644
>> --- a/utils.c
>> +++ b/utils.c
>> @@ -212,6 +212,98 @@ static int reserve_free_space(struct cache_tree *free_tree, u64 len,
>>  	return 0;
>>  }
>>
>> +static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
>> +				   u64 sb_bytenr)
>> +{
>> +	u32 crc = ~(u32)0;
>> +	int ret;
>> +
>> +	crc = btrfs_csum_data(NULL, (char *)sb + BTRFS_CSUM_SIZE, crc,
>> +			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
>> +	btrfs_csum_final(crc, (char *)&sb->csum[0]);
>> +	ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
>> +	if (ret < BTRFS_SUPER_INFO_SIZE)
>> +		ret = (ret < 0 ? -errno : -EIO);
>> +	else
>> +		ret = 0;
>> +	return ret;
>> +}
>> +
>> +/*
>> + * Setup temporary superblock at cfg->super_bynter
>> + * Needed info are extracted from cfg, and root_bytenr, chunk_bytenr
>> + *
>> + * For now sys chunk array will be empty and dev_item is empty
>> + * too.
>> + * They will be re-initialized at temp chunk tree setup.
>> + */
>> +static int setup_temp_super(int fd, struct btrfs_mkfs_config *cfg,
>> +			    u64 root_bytenr, u64 chunk_bytenr)
>> +{
>> +	unsigned char chunk_uuid[BTRFS_UUID_SIZE];
>> +	char super_buf[BTRFS_SUPER_INFO_SIZE];
>> +	struct btrfs_super_block *super = (struct btrfs_super_block *)super_buf;
>> +	int ret;
>> +
>> +	/*
>> +	 * We rely on cfg->chunk_uuid and cfg->fs_uuid to pass uuid
>> +	 * for other functions.
>> +	 * Caller must allocation space for them
>> +	 */
>> +	BUG_ON(!cfg->chunk_uuid || !cfg->fs_uuid);
>> +	memset(super_buf, 0, BTRFS_SUPER_INFO_SIZE);
>> +	cfg->num_bytes = round_down(cfg->num_bytes, cfg->sectorsize);
>> +
>> +	if (cfg->fs_uuid && *cfg->fs_uuid) {
>> +		if (uuid_parse(cfg->fs_uuid, super->fsid) != 0) {
>> +			error("cound not parse UUID: %s", cfg->fs_uuid);
>> +			ret = -EINVAL;
>> +			goto out;
>> +		}
>> +		if (!test_uuid_unique(cfg->fs_uuid)) {
>> +			error("non-unique UUID: %s", cfg->fs_uuid);
>> +			ret = -EINVAL;
>> +			goto out;
>> +		}
>> +	} else {
>> +		uuid_generate(super->fsid);
>> +		uuid_unparse(super->fsid, cfg->fs_uuid);
>> +	}
>> +	uuid_generate(chunk_uuid);
>> +	uuid_unparse(chunk_uuid, cfg->chunk_uuid);
>> +
>> +	btrfs_set_super_bytenr(super, cfg->super_bytenr);
>> +	btrfs_set_super_num_devices(super, 1);
>> +	btrfs_set_super_magic(super, BTRFS_MAGIC);
>> +	btrfs_set_super_generation(super, 1);
>> +	btrfs_set_super_root(super, root_bytenr);
>> +	btrfs_set_super_chunk_root(super, chunk_bytenr);
>> +	btrfs_set_super_total_bytes(super, cfg->num_bytes);
>> +	/*
>> +	 * Temporary btrfs will only has 6 tree roots:
>> +	 * chunk tree, root tree, extent_tree, device tree, fs tree
>> +	 * and csum tree.
>> +	 */
>> +	btrfs_set_super_bytes_used(super, 6 * cfg->nodesize);
>> +	btrfs_set_super_sectorsize(super, cfg->sectorsize);
>> +	btrfs_set_super_leafsize(super, cfg->nodesize);
>> +	btrfs_set_super_nodesize(super, cfg->nodesize);
>> +	btrfs_set_super_stripesize(super, cfg->stripesize);
>> +	btrfs_set_super_csum_type(super, BTRFS_CSUM_TYPE_CRC32);
>> +	btrfs_set_super_chunk_root(super, chunk_bytenr);
>> +	btrfs_set_super_cache_generation(super, -1);
>> +	btrfs_set_super_incompat_flags(super, cfg->features);
>> +	if (cfg->label)
>> +		strncpy(super->label, cfg->label, BTRFS_LABEL_SIZE - 1);
>
> Why not use __strncpy_null?
>
> Thanks,
>
> -liubo

Good idea, I'll add new patch to use it.

Thanks,
Qu
>
>> +
>> +	/* Sys chunk array will be re-initialized at chunk tree init time */
>> +	super->sys_chunk_array_size = 0;
>> +
>> +	ret = write_temp_super(fd, super, cfg->super_bytenr);
>> +out:
>> +	return ret;
>> +}
>> +
>>  /*
>>   * Improved version of make_btrfs().
>>   *
>> @@ -230,6 +322,10 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
>>  	struct cache_tree *used = &cctx->used;
>>  	u64 sys_chunk_start;
>>  	u64 meta_chunk_start;
>> +	/* chunk tree bytenr, in system chunk */
>> +	u64 chunk_bytenr;
>> +	/* metadata trees bytenr, in metadata chunk */
>> +	u64 root_bytenr;
>>  	int ret;
>>
>>  	/* Shouldn't happen */
>> @@ -260,6 +356,27 @@ static int make_convert_btrfs(int fd, struct btrfs_mkfs_config *cfg,
>>  	if (ret < 0)
>>  		goto out;
>>
>> +	/*
>> +	 * Inside the allocate metadata chunk, its layout will be:
>> +	 *  | offset		| contents	|
>> +	 *  -------------------------------------
>> +	 *  | +0		| tree root	|
>> +	 *  | +nodesize		| extent root	|
>> +	 *  | +nodesize * 2	| device root	|
>> +	 *  | +nodesize * 3	| fs tree	|
>> +	 *  | +nodesize * 4	| csum tree	|
>> +	 *  -------------------------------------
>> +	 * Inside the allocated system chunk, its layout will be:
>> +	 *  | offset		| contents	|
>> +	 *  -------------------------------------
>> +	 *  | +0		| chunk root	|
>> +	 *  -------------------------------------
>> +	 */
>> +	chunk_bytenr = sys_chunk_start;
>> +	root_bytenr = meta_chunk_start;
>> +	ret = setup_temp_super(fd, cfg, root_bytenr, chunk_bytenr);
>> +	if (ret < 0)
>> +		goto out;
>>  out:
>>  	return ret;
>>  }
>> --
>> 2.7.0
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH v3 21/22] btrfs-progs: convert: Strictly avoid meta or system chunk allocation

[Qu Wenruo]

On 05/28/2016 11:30 AM, Liu Bo wrote:
> On Fri, Jan 29, 2016 at 01:03:31PM +0800, Qu Wenruo wrote:
>> Before this patch, btrfs-convert only rely on large enough initial
>> system/metadata chunk size to ensure no newer system/meta chunk will be
>> created.
>>
>> But that's not safe enough. So add two new members in fs_info,
>> avoid_sys/meta_chunk_alloc flags to prevent any newer system or meta
>> chunks to be created before init_btrfs_v2().
>>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> Signed-off-by: David Sterba <dsterba@suse.com>
>> ---
>>  btrfs-convert.c |  9 +++++++++
>>  ctree.h         |  3 +++
>>  extent-tree.c   | 10 ++++++++++
>>  3 files changed, 22 insertions(+)
>>
>> diff --git a/btrfs-convert.c b/btrfs-convert.c
>> index efa3b02..333f413 100644
>> --- a/btrfs-convert.c
>> +++ b/btrfs-convert.c
>> @@ -2322,6 +2322,13 @@ static int init_btrfs_v2(struct btrfs_mkfs_config *cfg, struct btrfs_root *root,
>>  	struct btrfs_fs_info *fs_info = root->fs_info;
>>  	int ret;
>>
>> +	/*
>> +	 * Don't alloc any metadata/system chunk, as we don't want
>> +	 * any meta/sys chunk allcated before all data chunks are inserted.
>> +	 * Or we screw up the chunk layout just like the old implement.
>> +	 */
>
> I don't get this, with this patch set, we can allocate data from DATA chunk,
> allocate metadata from METADATA chunk, but then we're not allowed to allocate new chunks?
>
> Thanks,

For new convert, we are going through the following steps:
1) Create initial meta/sys chunks into unused space, manually
2) Open fs
3) Insert data chunks to covert all ext* used data
4) Do per inode copying

The whole patchset rely on a key assumption: All data chunks are already 
allocated to cover all ext* used data, so new chunk/extent allocation 
can follow the normal routine.

Before that, only chunks created in step 1) is completely safe, and new 
chunk allocation before step 3) are all unsafe, as the key assumption is 
not met yet.

So, unitl step 3), we must not allocate any new data/metadata chunks.

Thanks,
Qu
>
> -liubo
>> +	fs_info->avoid_sys_chunk_alloc = 1;
>> +	fs_info->avoid_meta_chunk_alloc = 1;
>>  	trans = btrfs_start_transaction(root, 1);
>>  	BUG_ON(!trans);
>>  	ret = btrfs_fix_block_accounting(trans, root);
>> @@ -2359,6 +2366,8 @@ static int init_btrfs_v2(struct btrfs_mkfs_config *cfg, struct btrfs_root *root,
>>  		goto err;
>>
>>  	ret = btrfs_commit_transaction(trans, root);
>> +	fs_info->avoid_sys_chunk_alloc = 0;
>> +	fs_info->avoid_meta_chunk_alloc = 0;
>>  err:
>>  	return ret;
>>  }
>> diff --git a/ctree.h b/ctree.h
>> index 1443746..187bd27 100644
>> --- a/ctree.h
>> +++ b/ctree.h
>> @@ -1030,6 +1030,9 @@ struct btrfs_fs_info {
>>  	unsigned int quota_enabled:1;
>>  	unsigned int suppress_check_block_errors:1;
>>  	unsigned int ignore_fsid_mismatch:1;
>> +	unsigned int avoid_meta_chunk_alloc:1;
>> +	unsigned int avoid_sys_chunk_alloc:1;
>> +
>>
>>  	int (*free_extent_hook)(struct btrfs_trans_handle *trans,
>>  				struct btrfs_root *root,
>> diff --git a/extent-tree.c b/extent-tree.c
>> index 93b1945..e7c61b1 100644
>> --- a/extent-tree.c
>> +++ b/extent-tree.c
>> @@ -1904,6 +1904,16 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
>>  	    thresh)
>>  		return 0;
>>
>> +	/*
>> +	 * Avoid allocating given chunk type
>> +	 */
>> +	if (extent_root->fs_info->avoid_meta_chunk_alloc &&
>> +	    (flags & BTRFS_BLOCK_GROUP_METADATA))
>> +		return 0;
>> +	if (extent_root->fs_info->avoid_sys_chunk_alloc &&
>> +	    (flags & BTRFS_BLOCK_GROUP_SYSTEM))
>> +		return 0;
>> +
>>  	ret = btrfs_alloc_chunk(trans, extent_root, &start, &num_bytes,
>>  	                        space_info->flags);
>>  	if (ret == -ENOSPC) {
>> --
>> 2.7.0
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH v3 16/22] btrfs-progs: convert: Introduce function to migrate reserved ranges

[Qu Wenruo]

On 05/28/2016 11:16 AM, Liu Bo wrote:
> On Fri, Jan 29, 2016 at 01:03:26PM +0800, Qu Wenruo wrote:
>> Introduce new function, migrate_reserved_ranges() to migrate used fs
>> data in btrfs reserved space.
>>
>> Unlike old implement, which will need to relocate all the complicated
>> csum and reference relocation, previous patches already ensure such
>> reserved ranges won't be allocated.
>> So here we only need copy these data out and create new
>> extent/csum/reference.
>>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> Signed-off-by: David Sterba <dsterba@suse.com>
>> ---
>>  btrfs-convert.c | 124 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>>  1 file changed, 122 insertions(+), 2 deletions(-)
>>
>> diff --git a/btrfs-convert.c b/btrfs-convert.c
>> index 16e2309..f6126db 100644
>> --- a/btrfs-convert.c
>> +++ b/btrfs-convert.c
>> @@ -1679,6 +1679,123 @@ static int create_image_file_range_v2(struct btrfs_trans_handle *trans,
>>  	return ret;
>>  }
>>
>> +
>> +/*
>> + * Relocate old fs data in one reserved ranges
>> + *
>> + * Since all old fs data in reserved range is not covered by any chunk nor
>> + * data extent, we don't need to handle any reference but add new
>> + * extent/reference, which makes codes more clear
>> + */
>> +static int migrate_one_reserved_range(struct btrfs_trans_handle *trans,
>> +				      struct btrfs_root *root,
>> +				      struct cache_tree *used,
>> +				      struct btrfs_inode_item *inode, int fd,
>> +				      u64 ino, u64 start, u64 len, int datacsum)
>> +{
>> +	u64 cur_off = start;
>> +	u64 cur_len = len;
>> +	struct cache_extent *cache;
>> +	struct btrfs_key key;
>> +	struct extent_buffer *eb;
>> +	int ret = 0;
>> +
>> +	while (cur_off < start + len) {
>> +		cache = lookup_cache_extent(used, cur_off, cur_len);
>> +		if (!cache)
>> +			break;
>> +		cur_off = max(cache->start, cur_off);
>> +		cur_len = min(cache->start + cache->size, start + len) -
>> +			  cur_off;
>> +		BUG_ON(cur_len < root->sectorsize);
>> +
>> +		/* reserve extent for the data */
>> +		ret = btrfs_reserve_extent(trans, root, cur_len, 0, 0, (u64)-1,
>> +					   &key, 1);
>> +		if (ret < 0)
>> +			break;
>> +
>> +		eb = malloc(sizeof(*eb) + cur_len);
>> +		if (!eb) {
>> +			ret = -ENOMEM;
>> +			break;
>> +		}
>> +
>> +		ret = pread(fd, eb->data, cur_len, cur_off);
>> +		if (ret < cur_len) {
>> +			ret = (ret < 0 ? ret : -EIO);
>> +			free(eb);
>> +			break;
>> +		}
>> +		eb->start = key.objectid;
>> +		eb->len = key.offset;
>> +
>> +		/* Write the data */
>> +		ret = write_and_map_eb(trans, root, eb);
>> +		free(eb);
>> +		if (ret < 0)
>> +			break;
>
> With write_data_to_disk(), we don't have to create eb for write.
>
> Thanks,
>
> -liubo

Nice advice.

I didn't remember whether write_data_to_disk() was there when the 
patchset was written, but always a good idea to get rid of the temporary eb.

Thanks,
Qu

>> +
>> +		/* Now handle extent item and file extent things */
>> +		ret = btrfs_record_file_extent(trans, root, ino, inode, cur_off,
>> +					       key.objectid, key.offset);
>> +		if (ret < 0)
>> +			break;
>> +		/* Finally, insert csum items */
>> +		if (datacsum)
>> +			ret = csum_disk_extent(trans, root, key.objectid,
>> +					       key.offset);
>> +
>> +		cur_off += key.offset;
>> +		cur_len = start + len - cur_off;
>> +	}
>> +	return ret;
>> +}
>> +
>> +/*
>> + * Relocate the used ext2 data in reserved ranges
>> + * [0,1M)
>> + * [btrfs_sb_offset(1), +BTRFS_STRIPE_LEN)
>> + * [btrfs_sb_offset(2), +BTRFS_STRIPE_LEN)
>> + */
>> +static int migrate_reserved_ranges(struct btrfs_trans_handle *trans,
>> +				   struct btrfs_root *root,
>> +				   struct cache_tree *used,
>> +				   struct btrfs_inode_item *inode, int fd,
>> +				   u64 ino, u64 total_bytes, int datacsum)
>> +{
>> +	u64 cur_off;
>> +	u64 cur_len;
>> +	int ret = 0;
>> +
>> +	/* 0 ~ 1M */
>> +	cur_off = 0;
>> +	cur_len = 1024 * 1024;
>> +	ret = migrate_one_reserved_range(trans, root, used, inode, fd, ino,
>> +					 cur_off, cur_len, datacsum);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* second sb(fisrt sb is included in 0~1M) */
>> +	cur_off = btrfs_sb_offset(1);
>> +	cur_len = min(total_bytes, cur_off + BTRFS_STRIPE_LEN) - cur_off;
>> +	if (cur_off < total_bytes)
>> +		return ret;
>> +	ret = migrate_one_reserved_range(trans, root, used, inode, fd, ino,
>> +					 cur_off, cur_len, datacsum);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	/* Last sb */
>> +	cur_off = btrfs_sb_offset(2);
>> +	cur_len = min(total_bytes, cur_off + BTRFS_STRIPE_LEN) - cur_off;
>> +	if (cur_off < total_bytes)
>> +		return ret;
>> +	ret = migrate_one_reserved_range(trans, root, used, inode, fd, ino,
>> +					 cur_off, cur_len, datacsum);
>> +	return ret;
>> +}
>> +
>>  static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
>>  				int ensure_size);
>>
>> @@ -1687,11 +1804,10 @@ static int wipe_reserved_ranges(struct cache_tree *tree, u64 min_stripe_size,
>>   *
>>   * This is completely fs independent as we have cctx->used, only
>>   * need to create file extents point to all the posistions.
>> - * TODO: Add handler for reserved ranges in next patch
>>   */
>>  static int create_image_v2(struct btrfs_root *root,
>>  			   struct btrfs_mkfs_config *cfg,
>> -			   struct btrfs_convert_context *cctx,
>> +			   struct btrfs_convert_context *cctx, int fd,
>>  			   u64 size, char *name, int datacsum)
>>  {
>>  	struct btrfs_inode_item buf;
>> @@ -1769,6 +1885,10 @@ static int create_image_v2(struct btrfs_root *root,
>>  			goto out;
>>  		cur += len;
>>  	}
>> +	/* Handle the reserved ranges */
>> +	ret = migrate_reserved_ranges(trans, root, &cctx->used, &buf, fd, ino,
>> +				      cfg->num_bytes, datacsum);
>> +
>>
>>  	key.objectid = ino;
>>  	key.type = BTRFS_INODE_ITEM_KEY;
>> --
>> 2.7.0
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH v3 05/22] btrfs-progs: Introduce function to setup temporary superblock

[David Sterba]

On Sun, May 29, 2016 at 06:52:41PM +0800, Qu Wenruo wrote:
> >> +	btrfs_set_super_leafsize(super, cfg->nodesize);
> >> +	btrfs_set_super_nodesize(super, cfg->nodesize);
> >> +	btrfs_set_super_stripesize(super, cfg->stripesize);
> >> +	btrfs_set_super_csum_type(super, BTRFS_CSUM_TYPE_CRC32);
> >> +	btrfs_set_super_chunk_root(super, chunk_bytenr);
> >> +	btrfs_set_super_cache_generation(super, -1);
> >> +	btrfs_set_super_incompat_flags(super, cfg->features);
> >> +	if (cfg->label)
> >> +		strncpy(super->label, cfg->label, BTRFS_LABEL_SIZE - 1);
> >
> > Why not use __strncpy_null?
> 
> Good idea, I'll add new patch to use it.

I've updated it in the patch, there was one more that used strncpy and
coverity reported it.