[PATCH 00/16] btrfs-progs: recover from failed metadata_uuid

[PATCH 00/16] btrfs-progs: recover from failed metadata_uuid

[Anand Jain]

The kernel reunites split-brained devices after a failed `btrfstune -m|M`
operation. We can achieve the same in btrfs-progs. So port it here.
Ref the discussion here:

   https://lore.kernel.org/all/1fa6802b-5812-14a8-3fc8-5da54bb5f79d@oracle.com/

Patch 1/16 wasn't integrated as part of the set
	[PATCH 00/10 v2] fixes and preparatory related to metadata_uuid
it's now merged with this patchset.

Patches [2-6,11,12] are cleanup patches.

Patches [7,8,10] are preparatory.

Patch [9] addresses a bug.

Patches [13, 14, 15] provide recovery from previously failed
`btrfstune -m|M` operations.

Patch [16] enhances the misc-test `034-metadata-uuid` to also validate this
new recovery feature.

This set has been successfully tested with the btrfs-progs testsuite.

This patchset is on top the latest devel last commit:
 8aba9b0052b6 btrfs-progs: btrfstune: consolidate error handling in main()


Anand Jain (16):
  btrfs-progs: track num_devices per fs_devices
  btrfs-progs: tune can use local fs_info variable
  btrfs-progs: rename set_metadata_uuid arg to new_fsid_str
  btrfs-progs: rename set_metadata_uuid new_fsid to fsid
  btrfs-progs: rename set_metadata_uuid new_uuid to new_fsid
  btrfs-progs: rename set_metadata_uuid uuid_changed to fsid_changed
  btrfs-progs: pass fsid in check_unfinished_fsid_change arg2
  btrfs-progs: pass metadata_uuid in check_unfinished_fsid_change arg3
  btrfs-progs: fix return without flag reset commit in tune
  btrfs-progs: preparing the latest device's superblock for commit
  btrfs-progs: rename fs_devices::list to match the kernel
  btrfs-progs: rename fs_devices::latest_trans to match the kernel
  btrfs-progs: tune use the latest bdev in fs_devices for super_copy
  btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag
  btrfs-progs: recover from the failed btrfstune -m|M
  btrfs-progs: test btrfstune -m|M ability to fix previous failures

 cmds/filesystem.c                          |  14 +-
 common/device-scan.c                       |   2 +-
 kernel-shared/disk-io.c                    |   3 +
 kernel-shared/disk-io.h                    |   5 +
 kernel-shared/volumes.c                    | 203 +++++++++++++++++++--
 kernel-shared/volumes.h                    |   5 +-
 tests/misc-tests/034-metadata-uuid/test.sh |  70 +++++--
 tune/change-metadata-uuid.c                |  56 +++---
 tune/main.c                                |  35 ++--
 9 files changed, 312 insertions(+), 81 deletions(-)

-- 
2.39.3

[PATCH 01/16] btrfs-progs: track num_devices per fs_devices

[Anand Jain]

Similar to the kernel we need to track the number of devices scanned
per fs_devices. A preparation patch to fix incomplete fsid changing.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 kernel-shared/volumes.c | 1 +
 kernel-shared/volumes.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/kernel-shared/volumes.c b/kernel-shared/volumes.c
index 946c2e7a931a..3ca7a5a62da8 100644
--- a/kernel-shared/volumes.c
+++ b/kernel-shared/volumes.c
@@ -406,6 +406,7 @@ static int device_list_add(const char *path,
 			btrfs_stack_device_bytes_used(&disk_super->dev_item);
 		list_add(&device->dev_list, &fs_devices->devices);
 		device->fs_devices = fs_devices;
+		fs_devices->num_devices++;
 	} else if (!device->name || strcmp(device->name, path)) {
 		char *name;
 
diff --git a/kernel-shared/volumes.h b/kernel-shared/volumes.h
index c89c1659e60c..23559b43e749 100644
--- a/kernel-shared/volumes.h
+++ b/kernel-shared/volumes.h
@@ -88,6 +88,7 @@ struct btrfs_fs_devices {
 	u64 latest_trans;
 	u64 lowest_devid;
 
+	u64 num_devices;
 	u64 missing_devices;
 	u64 total_rw_bytes;
 
-- 
2.39.3

[PATCH 02/16] btrfs-progs: tune can use local fs_info variable

[Anand Jain]

Since the root pointer dereferences for the fs_info several times,
it is rational to save the fs_info.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tune/main.c | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)

diff --git a/tune/main.c b/tune/main.c
index c49c24298187..e3b199c10dad 100644
--- a/tune/main.c
+++ b/tune/main.c
@@ -133,6 +133,7 @@ static const struct cmd_struct tune_cmd = {
 int BOX_MAIN(btrfstune)(int argc, char *argv[])
 {
 	struct btrfs_root *root;
+	struct btrfs_fs_info *fs_info;
 	unsigned ctree_flags = OPEN_CTREE_WRITES;
 	int success = 0;
 	int total = 0;
@@ -296,6 +297,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		ret = 1;
 		goto free_out;
 	}
+	fs_info = root->fs_info;
 
 	/*
 	 * As we increment the generation number here, it is unlikely that the
@@ -309,9 +311,9 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 	 * all the partner devices.
 	 */
 	if ((change_metadata_uuid || random_fsid || new_fsid_str) &&
-	     root->fs_info->fs_devices->missing_devices > 0) {
+	     fs_info->fs_devices->missing_devices > 0) {
 		error("missing %lld device(s), failing the command",
-		       root->fs_info->fs_devices->missing_devices);
+		       fs_info->fs_devices->missing_devices);
 		ret = 1;
 		goto out;
 	}
@@ -322,17 +324,17 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 			ret = 1;
 			goto out;
 		}
-		if (btrfs_fs_compat_ro(root->fs_info, BLOCK_GROUP_TREE)) {
+		if (btrfs_fs_compat_ro(fs_info, BLOCK_GROUP_TREE)) {
 			error("the filesystem already has block group tree feature");
 			ret = 1;
 			goto out;
 		}
-		if (!btrfs_fs_compat_ro(root->fs_info, FREE_SPACE_TREE_VALID)) {
+		if (!btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE_VALID)) {
 			error("the filesystem doesn't have space cache v2, needs to be mounted with \"-o space_cache=v2\" first");
 			ret = 1;
 			goto out;
 		}
-		ret = convert_to_bg_tree(root->fs_info);
+		ret = convert_to_bg_tree(fs_info);
 		if (ret < 0) {
 			error("failed to convert the filesystem to block group tree feature");
 			goto out;
@@ -340,12 +342,12 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		goto out;
 	}
 	if (to_fst) {
-		if (btrfs_fs_compat_ro(root->fs_info, FREE_SPACE_TREE_VALID)) {
+		if (btrfs_fs_compat_ro(fs_info, FREE_SPACE_TREE_VALID)) {
 			error("filesystem already has free-space-tree feature");
 			ret = 1;
 			goto out;
 		}
-		ret = convert_to_fst(root->fs_info);
+		ret = convert_to_fst(fs_info);
 		if (ret < 0)
 			error("failed to convert the filesystem to free-space-tree feature");
 		goto out;
@@ -356,12 +358,12 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 			ret = 1;
 			goto out;
 		}
-		if (!btrfs_fs_compat_ro(root->fs_info, BLOCK_GROUP_TREE)) {
+		if (!btrfs_fs_compat_ro(fs_info, BLOCK_GROUP_TREE)) {
 			error("filesystem doesn't have block-group-tree feature");
 			ret = 1;
 			goto out;
 		}
-		ret = convert_to_extent_tree(root->fs_info);
+		ret = convert_to_extent_tree(fs_info);
 		if (ret < 0) {
 			error("failed to convert the filesystem from block group tree feature");
 			goto out;
@@ -369,7 +371,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		goto out;
 	}
 	if (seeding_flag) {
-		if (btrfs_fs_incompat(root->fs_info, METADATA_UUID)) {
+		if (btrfs_fs_incompat(fs_info, METADATA_UUID)) {
 			error("SEED flag cannot be changed on a metadata-uuid changed fs");
 			ret = 1;
 			goto out;
@@ -402,7 +404,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 	if (csum_type != -1) {
 		/* TODO: check conflicting flags */
 		pr_verbose(LOG_DEFAULT, "Proceed to switch checksums\n");
-		ret = btrfs_change_csum_type(root->fs_info, csum_type);
+		ret = btrfs_change_csum_type(fs_info, csum_type);
 	}
 
 	if (change_metadata_uuid) {
@@ -424,8 +426,8 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 	}
 
 	if (random_fsid || (new_fsid_str && !change_metadata_uuid)) {
-		if (btrfs_fs_incompat(root->fs_info, METADATA_UUID) ||
-		    root->fs_info->fs_devices->active_metadata_uuid) {
+		if (btrfs_fs_incompat(fs_info, METADATA_UUID) ||
+		    fs_info->fs_devices->active_metadata_uuid) {
 			error(
 		"Cannot rewrite fsid while METADATA_UUID flag is active. \n"
 		"Ensure fsid and metadata_uuid match before retrying.");
@@ -445,7 +447,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 				goto out;
 			}
 		}
-		ret = change_uuid(root->fs_info, new_fsid_str);
+		ret = change_uuid(fs_info, new_fsid_str);
 		if (!ret)
 			success++;
 		total++;
@@ -454,7 +456,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 	if (success == total) {
 		ret = 0;
 	} else {
-		root->fs_info->readonly = 1;
+		fs_info->readonly = 1;
 		ret = 1;
 		error("btrfstune failed");
 	}
-- 
2.39.3

[PATCH 03/16] btrfs-progs: rename set_metadata_uuid arg to new_fsid_str

[Anand Jain]

In preparation to use check_unfinished_fsid_change() to support the
ability to reunite devices after a failed 'btrfstune -m|M' command,
%uuid_string arg is actually carries new fsid to be used. So just name
it to %new_fsid_str.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tune/change-metadata-uuid.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tune/change-metadata-uuid.c b/tune/change-metadata-uuid.c
index 295308f299aa..f356de8b57ce 100644
--- a/tune/change-metadata-uuid.c
+++ b/tune/change-metadata-uuid.c
@@ -24,7 +24,7 @@
 #include "common/messages.h"
 #include "tune/tune.h"
 
-int set_metadata_uuid(struct btrfs_root *root, const char *uuid_string)
+int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 {
 	struct btrfs_super_block *disk_super;
 	uuid_t new_fsid, unused1, unused2;
@@ -50,8 +50,8 @@ int set_metadata_uuid(struct btrfs_root *root, const char *uuid_string)
 		return 1;
 	}
 
-	if (uuid_string)
-		uuid_parse(uuid_string, new_fsid);
+	if (new_fsid_string)
+		uuid_parse(new_fsid_string, new_fsid);
 	else
 		uuid_generate(new_fsid);
 
-- 
2.39.3

[PATCH 04/16] btrfs-progs: rename set_metadata_uuid new_fsid to fsid

[Anand Jain]

The %new_fsid is not only new it can be the fsid from the passed disk
so just rename it to %fsid. Also, in the next patch the %new_fsid will
be a bool variable to indicate if the %fsid is new from the fsid in the
disk.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tune/change-metadata-uuid.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/tune/change-metadata-uuid.c b/tune/change-metadata-uuid.c
index f356de8b57ce..0e5760194b54 100644
--- a/tune/change-metadata-uuid.c
+++ b/tune/change-metadata-uuid.c
@@ -27,7 +27,7 @@
 int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 {
 	struct btrfs_super_block *disk_super;
-	uuid_t new_fsid, unused1, unused2;
+	uuid_t fsid, unused1, unused2;
 	struct btrfs_trans_handle *trans;
 	bool new_uuid = true;
 	u64 incompat_flags;
@@ -51,11 +51,11 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	}
 
 	if (new_fsid_string)
-		uuid_parse(new_fsid_string, new_fsid);
+		uuid_parse(new_fsid_string, fsid);
 	else
-		uuid_generate(new_fsid);
+		uuid_generate(fsid);
 
-	new_uuid = (memcmp(new_fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0);
+	new_uuid = (memcmp(fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0);
 
 	/* Step 1 sets the in progress flag */
 	trans = btrfs_start_transaction(root, 1);
@@ -66,23 +66,23 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		return ret;
 
 	if (new_uuid && uuid_changed && memcmp(disk_super->metadata_uuid,
-					       new_fsid, BTRFS_FSID_SIZE) == 0) {
+					       fsid, BTRFS_FSID_SIZE) == 0) {
 		/*
 		 * Changing fsid to be the same as metadata uuid, so just
 		 * disable the flag
 		 */
-		memcpy(disk_super->fsid, &new_fsid, BTRFS_FSID_SIZE);
+		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
 		incompat_flags &= ~BTRFS_FEATURE_INCOMPAT_METADATA_UUID;
 		btrfs_set_super_incompat_flags(disk_super, incompat_flags);
 		memset(disk_super->metadata_uuid, 0, BTRFS_FSID_SIZE);
 	} else if (new_uuid && uuid_changed && memcmp(disk_super->metadata_uuid,
-						new_fsid, BTRFS_FSID_SIZE)) {
+						      fsid, BTRFS_FSID_SIZE)) {
 		/*
 		 * Changing fsid on an already changed FS, in this case we
 		 * only change the fsid and don't touch metadata uuid as it
 		 * has already the correct value
 		 */
-		memcpy(disk_super->fsid, &new_fsid, BTRFS_FSID_SIZE);
+		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
 	} else if (new_uuid && !uuid_changed) {
 		/*
 		 * First time changing the fsid, copy the fsid to metadata_uuid
@@ -91,7 +91,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		btrfs_set_super_incompat_flags(disk_super, incompat_flags);
 		memcpy(disk_super->metadata_uuid, disk_super->fsid,
 		       BTRFS_FSID_SIZE);
-		memcpy(disk_super->fsid, &new_fsid, BTRFS_FSID_SIZE);
+		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
 	} else {
 		/* Setting the same fsid as current, do nothing */
 		return 0;
-- 
2.39.3

[PATCH 05/16] btrfs-progs: rename set_metadata_uuid new_uuid to new_fsid

[Anand Jain]

%new_uuid is being used to say there is a new fsid. So why not just call it
%new_fsid.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tune/change-metadata-uuid.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/tune/change-metadata-uuid.c b/tune/change-metadata-uuid.c
index 0e5760194b54..83299f990b50 100644
--- a/tune/change-metadata-uuid.c
+++ b/tune/change-metadata-uuid.c
@@ -29,7 +29,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	struct btrfs_super_block *disk_super;
 	uuid_t fsid, unused1, unused2;
 	struct btrfs_trans_handle *trans;
-	bool new_uuid = true;
+	bool new_fsid = true;
 	u64 incompat_flags;
 	bool uuid_changed;
 	u64 super_flags;
@@ -55,7 +55,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	else
 		uuid_generate(fsid);
 
-	new_uuid = (memcmp(fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0);
+	new_fsid = (memcmp(fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0);
 
 	/* Step 1 sets the in progress flag */
 	trans = btrfs_start_transaction(root, 1);
@@ -65,7 +65,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	if (ret < 0)
 		return ret;
 
-	if (new_uuid && uuid_changed && memcmp(disk_super->metadata_uuid,
+	if (new_fsid && uuid_changed && memcmp(disk_super->metadata_uuid,
 					       fsid, BTRFS_FSID_SIZE) == 0) {
 		/*
 		 * Changing fsid to be the same as metadata uuid, so just
@@ -75,7 +75,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		incompat_flags &= ~BTRFS_FEATURE_INCOMPAT_METADATA_UUID;
 		btrfs_set_super_incompat_flags(disk_super, incompat_flags);
 		memset(disk_super->metadata_uuid, 0, BTRFS_FSID_SIZE);
-	} else if (new_uuid && uuid_changed && memcmp(disk_super->metadata_uuid,
+	} else if (new_fsid && uuid_changed && memcmp(disk_super->metadata_uuid,
 						      fsid, BTRFS_FSID_SIZE)) {
 		/*
 		 * Changing fsid on an already changed FS, in this case we
@@ -83,7 +83,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		 * has already the correct value
 		 */
 		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
-	} else if (new_uuid && !uuid_changed) {
+	} else if (new_fsid && !uuid_changed) {
 		/*
 		 * First time changing the fsid, copy the fsid to metadata_uuid
 		 */
-- 
2.39.3

[PATCH 06/16] btrfs-progs: rename set_metadata_uuid uuid_changed to fsid_changed

[Anand Jain]

We never change the metadata_uuid; we only change the fsid.
So '%fsid_changed' flows more appropriately than '%uuid_changed'.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tune/change-metadata-uuid.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/tune/change-metadata-uuid.c b/tune/change-metadata-uuid.c
index 83299f990b50..7bf30da4c3b0 100644
--- a/tune/change-metadata-uuid.c
+++ b/tune/change-metadata-uuid.c
@@ -31,14 +31,14 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	struct btrfs_trans_handle *trans;
 	bool new_fsid = true;
 	u64 incompat_flags;
-	bool uuid_changed;
+	bool fsid_changed;
 	u64 super_flags;
 	int ret;
 
 	disk_super = root->fs_info->super_copy;
 	super_flags = btrfs_super_flags(disk_super);
 	incompat_flags = btrfs_super_incompat_flags(disk_super);
-	uuid_changed = incompat_flags & BTRFS_FEATURE_INCOMPAT_METADATA_UUID;
+	fsid_changed = incompat_flags & BTRFS_FEATURE_INCOMPAT_METADATA_UUID;
 
 	if (super_flags & BTRFS_SUPER_FLAG_SEEDING) {
 		error("cannot set metadata UUID on a seed device");
@@ -65,7 +65,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	if (ret < 0)
 		return ret;
 
-	if (new_fsid && uuid_changed && memcmp(disk_super->metadata_uuid,
+	if (new_fsid && fsid_changed && memcmp(disk_super->metadata_uuid,
 					       fsid, BTRFS_FSID_SIZE) == 0) {
 		/*
 		 * Changing fsid to be the same as metadata uuid, so just
@@ -75,7 +75,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		incompat_flags &= ~BTRFS_FEATURE_INCOMPAT_METADATA_UUID;
 		btrfs_set_super_incompat_flags(disk_super, incompat_flags);
 		memset(disk_super->metadata_uuid, 0, BTRFS_FSID_SIZE);
-	} else if (new_fsid && uuid_changed && memcmp(disk_super->metadata_uuid,
+	} else if (new_fsid && fsid_changed && memcmp(disk_super->metadata_uuid,
 						      fsid, BTRFS_FSID_SIZE)) {
 		/*
 		 * Changing fsid on an already changed FS, in this case we
@@ -83,7 +83,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		 * has already the correct value
 		 */
 		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
-	} else if (new_fsid && !uuid_changed) {
+	} else if (new_fsid && !fsid_changed) {
 		/*
 		 * First time changing the fsid, copy the fsid to metadata_uuid
 		 */
-- 
2.39.3

[PATCH 07/16] btrfs-progs: pass fsid in check_unfinished_fsid_change arg2

[Anand Jain]

In preparation to use check_unfinished_fsid_change() to support the
ability to reunite devices after a failed 'btrfstune -m|M' command,
delete unused1 argument instead reuse %fsid as the function
check_unfinished_fsid_change() returns the fsid.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tune/change-metadata-uuid.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tune/change-metadata-uuid.c b/tune/change-metadata-uuid.c
index 7bf30da4c3b0..a49adda8dd29 100644
--- a/tune/change-metadata-uuid.c
+++ b/tune/change-metadata-uuid.c
@@ -27,7 +27,7 @@
 int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 {
 	struct btrfs_super_block *disk_super;
-	uuid_t fsid, unused1, unused2;
+	uuid_t fsid, unused2;
 	struct btrfs_trans_handle *trans;
 	bool new_fsid = true;
 	u64 incompat_flags;
@@ -45,7 +45,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		return 1;
 	}
 
-	if (check_unfinished_fsid_change(root->fs_info, unused1, unused2)) {
+	if (check_unfinished_fsid_change(root->fs_info, fsid, unused2)) {
 		error("UUID rewrite in progress, cannot change metadata_uuid");
 		return 1;
 	}
-- 
2.39.3

[PATCH 08/16] btrfs-progs: pass metadata_uuid in check_unfinished_fsid_change arg3

[Anand Jain]

In preparation to use check_unfinished_fsid_change() to support the
ability to reunite devices after a failed 'btrfstune -m|M' command,
rename %unused2 to %metadata_uuid as the function
check_unfinished_fsid_change() write the metadata_uuid from the ctree to
it.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tune/change-metadata-uuid.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tune/change-metadata-uuid.c b/tune/change-metadata-uuid.c
index a49adda8dd29..b161f6757d13 100644
--- a/tune/change-metadata-uuid.c
+++ b/tune/change-metadata-uuid.c
@@ -27,7 +27,7 @@
 int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 {
 	struct btrfs_super_block *disk_super;
-	uuid_t fsid, unused2;
+	uuid_t fsid, metadata_uuid;
 	struct btrfs_trans_handle *trans;
 	bool new_fsid = true;
 	u64 incompat_flags;
@@ -45,7 +45,7 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		return 1;
 	}
 
-	if (check_unfinished_fsid_change(root->fs_info, fsid, unused2)) {
+	if (check_unfinished_fsid_change(root->fs_info, fsid, metadata_uuid)) {
 		error("UUID rewrite in progress, cannot change metadata_uuid");
 		return 1;
 	}
-- 
2.39.3

[PATCH 09/16] btrfs-progs: fix return without flag reset commit in tune

[Anand Jain]

In the function set_metadata_uuid(), we set the flag
BTRFS_SUPER_FLAG_CHANGING_FSID_V2 in step 1 at line 71 as shown below:

   71         super_flags |= BTRFS_SUPER_FLAG_CHANGING_FSID_V2;
   72         btrfs_set_super_flags(disk_super, super_flags);
   73         ret = btrfs_commit_transaction(trans, root);

However, we fail to reset this flag if there is no change in the fsid on
the incoming disks, as we return too early.

  105       } else {
  106               /* Setting the same fsid as current, do nothing */
  107               return 0;

Fix this by allowing the thread to pass through the step 2, where we
reset the flag.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tune/change-metadata-uuid.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/tune/change-metadata-uuid.c b/tune/change-metadata-uuid.c
index b161f6757d13..ada3149ad549 100644
--- a/tune/change-metadata-uuid.c
+++ b/tune/change-metadata-uuid.c
@@ -92,9 +92,6 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 		memcpy(disk_super->metadata_uuid, disk_super->fsid,
 		       BTRFS_FSID_SIZE);
 		memcpy(disk_super->fsid, &fsid, BTRFS_FSID_SIZE);
-	} else {
-		/* Setting the same fsid as current, do nothing */
-		return 0;
 	}
 
 	trans = btrfs_start_transaction(root, 1);
-- 
2.39.3

[PATCH 10/16] btrfs-progs: preparing the latest device's superblock for commit

[Anand Jain]

This patch provides a flag to copy the superblock of the latest device to the
fs_info::super_copy for the commit process, rather than using the superblock
from the device specified in the argument.

This serves as groundwork to enable recovery from an incomplete
btrfstune -M|m|u|U operation.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 kernel-shared/disk-io.c | 3 +++
 kernel-shared/disk-io.h | 5 +++++
 2 files changed, 8 insertions(+)

diff --git a/kernel-shared/disk-io.c b/kernel-shared/disk-io.c
index 6a3178a84c88..1ef28ba33f28 100644
--- a/kernel-shared/disk-io.c
+++ b/kernel-shared/disk-io.c
@@ -1513,6 +1513,9 @@ static struct btrfs_fs_info *__open_ctree_fd(int fp, struct open_ctree_args *oca
 	if (flags & OPEN_CTREE_RECOVER_SUPER)
 		ret = btrfs_read_dev_super(fs_devices->latest_bdev, disk_super,
 				sb_bytenr, SBREAD_RECOVER);
+	else if (flags & OPEN_CTREE_USE_LATEST_BDEV)
+		ret = btrfs_read_dev_super(fs_devices->latest_bdev, disk_super,
+					   sb_bytenr, sbflags);
 	else
 		ret = btrfs_read_dev_super(fp, disk_super, sb_bytenr,
 				sbflags);
diff --git a/kernel-shared/disk-io.h b/kernel-shared/disk-io.h
index 424b953e0363..4f9ef633227d 100644
--- a/kernel-shared/disk-io.h
+++ b/kernel-shared/disk-io.h
@@ -104,6 +104,11 @@ enum btrfs_open_ctree_flags {
 	 * specific checks and only do the superficial checks.
 	 */
 	OPEN_CTREE_SKIP_LEAF_ITEM_CHECKS	= (1U << 17),
+
+	/*
+	 * Use the superblock of the latest device for the transaction commit.
+	 */
+	OPEN_CTREE_USE_LATEST_BDEV		= (1U << 18),
 };
 
 /*
-- 
2.39.3

[PATCH 11/16] btrfs-progs: rename fs_devices::list to match the kernel

[Anand Jain]

Aligning with the kernel's struct btrfs_fs_devices:fs_list, rename
btrfs_fs_devices::list to btrfs_fs_devices::fs_list.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 cmds/filesystem.c       | 14 +++++++-------
 common/device-scan.c    |  2 +-
 kernel-shared/volumes.c | 12 ++++++------
 kernel-shared/volumes.h |  2 +-
 4 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/cmds/filesystem.c b/cmds/filesystem.c
index 79f3e799250a..7dad5f6a0d25 100644
--- a/cmds/filesystem.c
+++ b/cmds/filesystem.c
@@ -485,7 +485,7 @@ static void free_fs_devices(struct btrfs_fs_devices *fs_devices)
 		cur_seed = next_seed;
 	}
 
-	list_del(&fs_devices->list);
+	list_del(&fs_devices->fs_list);
 	free(fs_devices);
 }
 
@@ -555,7 +555,7 @@ static int find_and_copy_seed(struct btrfs_fs_devices *seed,
 			      struct list_head *fs_uuids) {
 	struct btrfs_fs_devices *cur_fs;
 
-	list_for_each_entry(cur_fs, fs_uuids, list)
+	list_for_each_entry(cur_fs, fs_uuids, fs_list)
 		if (!memcmp(seed->fsid, cur_fs->fsid, BTRFS_FSID_SIZE))
 			return copy_fs_devices(copy, cur_fs);
 
@@ -591,7 +591,7 @@ static int search_umounted_fs_uuids(struct list_head *all_uuids,
 	 * The fs_uuids list is global, and open_ctree_* will
 	 * modify it, make a private copy here
 	 */
-	list_for_each_entry(cur_fs, fs_uuids, list) {
+	list_for_each_entry(cur_fs, fs_uuids, fs_list) {
 		/* don't bother handle all fs, if search target specified */
 		if (search) {
 			if (uuid_search(cur_fs, search) == 0)
@@ -616,7 +616,7 @@ static int search_umounted_fs_uuids(struct list_head *all_uuids,
 			goto out;
 		}
 
-		list_add(&fs_copy->list, all_uuids);
+		list_add(&fs_copy->fs_list, all_uuids);
 	}
 
 out:
@@ -635,7 +635,7 @@ static int map_seed_devices(struct list_head *all_uuids)
 
 	fs_uuids = btrfs_scanned_uuids();
 
-	list_for_each_entry(cur_fs, all_uuids, list) {
+	list_for_each_entry(cur_fs, all_uuids, fs_list) {
 		struct open_ctree_args oca = { 0 };
 
 		device = list_first_entry(&cur_fs->devices,
@@ -837,7 +837,7 @@ devs_only:
 		goto out;
 	}
 
-	list_for_each_entry(fs_devices, &all_uuids, list)
+	list_for_each_entry(fs_devices, &all_uuids, fs_list)
 		print_one_uuid(fs_devices, unit_mode);
 
 	if (search && !found) {
@@ -846,7 +846,7 @@ devs_only:
 	}
 	while (!list_empty(&all_uuids)) {
 		fs_devices = list_entry(all_uuids.next,
-					struct btrfs_fs_devices, list);
+					struct btrfs_fs_devices, fs_list);
 		free_fs_devices(fs_devices);
 	}
 out:
diff --git a/common/device-scan.c b/common/device-scan.c
index a140634f5d88..d61018a86f5c 100644
--- a/common/device-scan.c
+++ b/common/device-scan.c
@@ -257,7 +257,7 @@ int btrfs_register_all_devices(void)
 
 	all_uuids = btrfs_scanned_uuids();
 
-	list_for_each_entry(fs_devices, all_uuids, list) {
+	list_for_each_entry(fs_devices, all_uuids, fs_list) {
 		list_for_each_entry(device, &fs_devices->devices, dev_list) {
 			if (*device->name)
 				err = btrfs_register_one_device(device->name);
diff --git a/kernel-shared/volumes.c b/kernel-shared/volumes.c
index 3ca7a5a62da8..0a3b295930f0 100644
--- a/kernel-shared/volumes.c
+++ b/kernel-shared/volumes.c
@@ -319,7 +319,7 @@ static struct btrfs_fs_devices *find_fsid(u8 *fsid, u8 *metadata_uuid)
 {
 	struct btrfs_fs_devices *fs_devices;
 
-	list_for_each_entry(fs_devices, &fs_uuids, list) {
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
 		if (metadata_uuid && (memcmp(fsid, fs_devices->fsid,
 					     BTRFS_FSID_SIZE) == 0) &&
 		    (memcmp(metadata_uuid, fs_devices->metadata_uuid,
@@ -357,7 +357,7 @@ static int device_list_add(const char *path,
 		if (!fs_devices)
 			return -ENOMEM;
 		INIT_LIST_HEAD(&fs_devices->devices);
-		list_add(&fs_devices->list, &fs_uuids);
+		list_add(&fs_devices->fs_list, &fs_uuids);
 		memcpy(fs_devices->fsid, disk_super->fsid, BTRFS_FSID_SIZE);
 		if (metadata_uuid)
 			memcpy(fs_devices->metadata_uuid,
@@ -489,11 +489,11 @@ again:
 
 		orig = fs_devices;
 		fs_devices = seed_devices;
-		list_del(&orig->list);
+		list_del(&orig->fs_list);
 		free(orig);
 		goto again;
 	} else {
-		list_del(&fs_devices->list);
+		list_del(&fs_devices->fs_list);
 		free(fs_devices);
 	}
 
@@ -506,7 +506,7 @@ void btrfs_close_all_devices(void)
 
 	while (!list_empty(&fs_uuids)) {
 		fs_devices = list_entry(fs_uuids.next, struct btrfs_fs_devices,
-					list);
+					fs_list);
 		btrfs_close_devices(fs_devices);
 	}
 }
@@ -2227,7 +2227,7 @@ static int open_seed_devices(struct btrfs_fs_info *fs_info, u8 *fsid)
 			goto out;
 		}
 		INIT_LIST_HEAD(&fs_devices->devices);
-		list_add(&fs_devices->list, &fs_uuids);
+		list_add(&fs_devices->fs_list, &fs_uuids);
 		memcpy(fs_devices->fsid, fsid, BTRFS_FSID_SIZE);
 	}
 
diff --git a/kernel-shared/volumes.h b/kernel-shared/volumes.h
index 23559b43e749..2bf7b9d78b39 100644
--- a/kernel-shared/volumes.h
+++ b/kernel-shared/volumes.h
@@ -96,7 +96,7 @@ struct btrfs_fs_devices {
 	int latest_bdev;
 	int lowest_bdev;
 	struct list_head devices;
-	struct list_head list;
+	struct list_head fs_list;
 
 	struct btrfs_fs_devices *seed;
 
-- 
2.39.3

[PATCH 12/16] btrfs-progs: rename fs_devices::latest_trans to match the kernel

[Anand Jain]

Aligning progs's struct btrfs_fs_devices with the kernel rename
btrfs_fs_devices::latest_trans to btrfs_fs_devices::latest_generation.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 kernel-shared/volumes.c | 6 +++---
 kernel-shared/volumes.h | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/kernel-shared/volumes.c b/kernel-shared/volumes.c
index 0a3b295930f0..ad006b9de315 100644
--- a/kernel-shared/volumes.c
+++ b/kernel-shared/volumes.c
@@ -368,7 +368,7 @@ static int device_list_add(const char *path,
 
 		fs_devices->latest_devid = devid;
 		/* Below we would set this to found_transid */
-		fs_devices->latest_trans = 0;
+		fs_devices->latest_generation = 0;
 		fs_devices->lowest_devid = (u64)-1;
 		fs_devices->chunk_alloc_policy = BTRFS_CHUNK_ALLOC_REGULAR;
 		device = NULL;
@@ -438,9 +438,9 @@ static int device_list_add(const char *path,
 	if (metadata_uuid)
 		fs_devices->active_metadata_uuid = true;
 
-	if (found_transid > fs_devices->latest_trans) {
+	if (found_transid > fs_devices->latest_generation) {
 		fs_devices->latest_devid = devid;
-		fs_devices->latest_trans = found_transid;
+		fs_devices->latest_generation = found_transid;
 		fs_devices->total_devices = device->total_devs;
 	}
 	if (fs_devices->lowest_devid > devid) {
diff --git a/kernel-shared/volumes.h b/kernel-shared/volumes.h
index 2bf7b9d78b39..7f571bddee87 100644
--- a/kernel-shared/volumes.h
+++ b/kernel-shared/volumes.h
@@ -85,7 +85,7 @@ struct btrfs_fs_devices {
 
 	/* the device with this id has the most recent copy of the super */
 	u64 latest_devid;
-	u64 latest_trans;
+	u64 latest_generation;
 	u64 lowest_devid;
 
 	u64 num_devices;
-- 
2.39.3

[PATCH 13/16] btrfs-progs: tune use the latest bdev in fs_devices for super_copy

[Anand Jain]

btrfstune relies on the superblock of the device specified in the
btrfstune argument for fs_info::super_copy. Instead, should use
fs_devices::latest_bdev.

To support for reuniting devices following past failures of
btrfstune -m|M|u|U as in the following patch, use
fs_devices::latest_bdev.

 btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag
 btrfs-progs: recover from the failed btrfstune -m|M

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tune/main.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tune/main.c b/tune/main.c
index e3b199c10dad..e47047450b24 100644
--- a/tune/main.c
+++ b/tune/main.c
@@ -290,6 +290,9 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])
 		goto free_out;
 	}
 
+	if (change_metadata_uuid || random_fsid || new_fsid_str)
+		ctree_flags |= OPEN_CTREE_USE_LATEST_BDEV;
+
 	root = open_ctree_fd(fd, device, 0, ctree_flags);
 
 	if (!root) {
-- 
2.39.3

[PATCH 14/16] btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag

[Anand Jain]

Most of the code and functions in this patch is copied from the kernel.
Now, with this patch applied, there is no need to mount the device to
complete the incomplete 'btrfstune -m|M' command (CHANING_FSID_V2 flag).
Instead, the same command could be run, which will successfully complete
the operation.

Currently, the 'tests/misc-tests/034-metadata-uuid' tests the kernel using
four sets of disk images with CHANING_FSID_V2. Now, this test case has been
updated (as in the next patch) to test the the progs part.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 kernel-shared/volumes.c | 184 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 179 insertions(+), 5 deletions(-)

diff --git a/kernel-shared/volumes.c b/kernel-shared/volumes.c
index ad006b9de315..62015053afe3 100644
--- a/kernel-shared/volumes.c
+++ b/kernel-shared/volumes.c
@@ -332,6 +332,159 @@ static struct btrfs_fs_devices *find_fsid(u8 *fsid, u8 *metadata_uuid)
 	return NULL;
 }
 
+static u8 *btrfs_sb_fsid_ptr(struct btrfs_super_block *sb)
+{
+	bool has_metadata_uuid = (btrfs_super_incompat_flags(sb) &
+				  BTRFS_FEATURE_INCOMPAT_METADATA_UUID);
+
+	return has_metadata_uuid ? sb->metadata_uuid : sb->fsid;
+}
+
+static bool match_fsid_fs_devices(const struct btrfs_fs_devices *fs_devices,
+				  const u8 *fsid, const u8 *metadata_fsid)
+{
+	if (memcmp(fsid, fs_devices->fsid, BTRFS_FSID_SIZE) != 0)
+		return false;
+
+	if (!metadata_fsid)
+		return true;
+
+	if (memcmp(metadata_fsid, fs_devices->metadata_uuid, BTRFS_FSID_SIZE) != 0)
+		return false;
+
+	return true;
+}
+
+/*
+ * First check if the metadata_uuid is different from the fsid in the given
+ * fs_devices. Then check if the given fsid is the same as the metadata_uuid
+ * in the fs_devices. If it is, return true; otherwise, return false.
+ */
+static inline bool check_fsid_changed(const struct btrfs_fs_devices *fs_devices,
+				      const u8 *fsid)
+{
+	return memcmp(fs_devices->fsid, fs_devices->metadata_uuid,
+		      BTRFS_FSID_SIZE) != 0 &&
+	       memcmp(fs_devices->metadata_uuid, fsid, BTRFS_FSID_SIZE) == 0;
+}
+
+static struct btrfs_fs_devices *find_fsid_with_metadata_uuid(
+				struct btrfs_super_block *disk_super)
+{
+
+	struct btrfs_fs_devices *fs_devices;
+
+	/*
+	 * Handle scanned device having completed its fsid change but
+	 * belonging to a fs_devices that was created by first scanning
+	 * a device which didn't have its fsid/metadata_uuid changed
+	 * at all and the CHANGING_FSID_V2 flag set.
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (!fs_devices->changing_fsid)
+			continue;
+
+		if (match_fsid_fs_devices(fs_devices, disk_super->metadata_uuid,
+					  fs_devices->fsid))
+			return fs_devices;
+	}
+
+	/*
+	 * Handle scanned device having completed its fsid change but
+	 * belonging to a fs_devices that was created by a device that
+	 * has an outdated pair of fsid/metadata_uuid and
+	 * CHANGING_FSID_V2 flag set.
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (!fs_devices->changing_fsid)
+			continue;
+
+		if (check_fsid_changed(fs_devices, disk_super->metadata_uuid))
+			return fs_devices;
+	}
+
+	return find_fsid(disk_super->fsid, disk_super->metadata_uuid);
+}
+
+/*
+ * Handle scanned device having its CHANGING_FSID_V2 flag set and the fs_devices
+ * being created with a disk that has already completed its fsid change. Such
+ * disk can belong to an fs which has its FSID changed or to one which doesn't.
+ * Handle both cases here.
+ */
+static struct btrfs_fs_devices *find_fsid_inprogress(
+					struct btrfs_super_block *disk_super)
+{
+	struct btrfs_fs_devices *fs_devices;
+
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (fs_devices->changing_fsid)
+			continue;
+
+		if (check_fsid_changed(fs_devices,  disk_super->fsid))
+			return fs_devices;
+	}
+
+	return find_fsid(disk_super->fsid, NULL);
+}
+
+static struct btrfs_fs_devices *find_fsid_changed(
+					struct btrfs_super_block *disk_super)
+{
+	struct btrfs_fs_devices *fs_devices;
+
+	/*
+	 * Handles the case where scanned device is part of an fs that had
+	 * multiple successful changes of FSID but currently device didn't
+	 * observe it. Meaning our fsid will be different than theirs. We need
+	 * to handle two subcases :
+	 *  1 - The fs still continues to have different METADATA/FSID uuids.
+	 *  2 - The fs is switched back to its original FSID (METADATA/FSID
+	 *  are equal).
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		/* Changed UUIDs */
+		if (check_fsid_changed(fs_devices, disk_super->metadata_uuid) &&
+		    memcmp(fs_devices->fsid, disk_super->fsid,
+			   BTRFS_FSID_SIZE) != 0)
+			return fs_devices;
+
+		/* Unchanged UUIDs */
+		if (memcmp(fs_devices->metadata_uuid, fs_devices->fsid,
+			   BTRFS_FSID_SIZE) == 0 &&
+		    memcmp(fs_devices->fsid, disk_super->metadata_uuid,
+			   BTRFS_FSID_SIZE) == 0)
+			return fs_devices;
+	}
+
+	return NULL;
+}
+
+static struct btrfs_fs_devices *find_fsid_reverted_metadata(
+				struct btrfs_super_block *disk_super)
+{
+	struct btrfs_fs_devices *fs_devices;
+
+	/*
+	 * Handle the case where the scanned device is part of an fs whose last
+	 * metadata UUID change reverted it to the original FSID. At the same
+	 * time fs_devices was first created by another constituent device
+	 * which didn't fully observe the operation. This results in an
+	 * btrfs_fs_devices created with metadata/fsid different AND
+	 * btrfs_fs_devices::fsid_change set AND the metadata_uuid of the
+	 * fs_devices equal to the FSID of the disk.
+	 */
+	list_for_each_entry(fs_devices, &fs_uuids, fs_list) {
+		if (!fs_devices->changing_fsid)
+			continue;
+
+		if (check_fsid_changed(fs_devices, disk_super->fsid))
+			return fs_devices;
+	}
+
+	return NULL;
+}
+
 static int device_list_add(const char *path,
 			   struct btrfs_super_block *disk_super,
 			   struct btrfs_fs_devices **fs_devices_ret)
@@ -346,11 +499,18 @@ static int device_list_add(const char *path,
 			      (BTRFS_SUPER_FLAG_CHANGING_FSID |
 			       BTRFS_SUPER_FLAG_CHANGING_FSID_V2));
 
-	if (metadata_uuid)
-		fs_devices = find_fsid(disk_super->fsid,
-				       disk_super->metadata_uuid);
-	else
-		fs_devices = find_fsid(disk_super->fsid, NULL);
+	if (changing_fsid) {
+		if (!metadata_uuid)
+			fs_devices = find_fsid_inprogress(disk_super);
+		else
+			fs_devices = find_fsid_changed(disk_super);
+	} else if (metadata_uuid) {
+		fs_devices = find_fsid_with_metadata_uuid(disk_super);
+	} else {
+		fs_devices = find_fsid_reverted_metadata(disk_super);
+		if (!fs_devices)
+			fs_devices = find_fsid(disk_super->fsid, NULL);
+	}
 
 	if (!fs_devices) {
 		fs_devices = kzalloc(sizeof(*fs_devices), GFP_NOFS);
@@ -375,7 +535,21 @@ static int device_list_add(const char *path,
 	} else {
 		device = find_device(fs_devices, devid,
 				       disk_super->dev_item.uuid);
+		/*
+		 * If this disk has been pulled into an fs devices created by
+		 * a device which had the CHANGING_FSID_V2 flag then replace the
+		 * metadata_uuid/fsid values of the fs_devices.
+		 */
+		if (fs_devices->changing_fsid &&
+		    found_transid > fs_devices->latest_generation) {
+			memcpy(fs_devices->fsid, disk_super->fsid,
+			       BTRFS_FSID_SIZE);
+			memcpy(fs_devices->metadata_uuid,
+			       btrfs_sb_fsid_ptr(disk_super), BTRFS_FSID_SIZE);
+			fs_devices->changing_fsid = false;
+		}
 	}
+
 	if (!device) {
 		device = kzalloc(sizeof(*device), GFP_NOFS);
 		if (!device) {
-- 
2.39.3

[PATCH 15/16] btrfs-progs: recover from the failed btrfstune -m|M

[Anand Jain]

Currently, to fix device following the write failure of one or more devices
during btrfstune -m|M, we rely on the kernel's ability to reassemble devices,
even when they possess distinct fsids.

Kernel hinges combinations of metadata_uuid and generation number, with
additional cues taken from the fsid and the BTRFS_SUPER_FLAG_CHANGING_FSID_V2
flag. This patch adds this capability to btrfs-progs.

In complex scenarios (such as multiple fsids with the same metadata_uuid and
matching generation), user intervention becomes necessary to resolve the
situations which btrfs-prog can do better.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tune/change-metadata-uuid.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)

diff --git a/tune/change-metadata-uuid.c b/tune/change-metadata-uuid.c
index ada3149ad549..371f34e679b4 100644
--- a/tune/change-metadata-uuid.c
+++ b/tune/change-metadata-uuid.c
@@ -46,14 +46,23 @@ int set_metadata_uuid(struct btrfs_root *root, const char *new_fsid_string)
 	}
 
 	if (check_unfinished_fsid_change(root->fs_info, fsid, metadata_uuid)) {
-		error("UUID rewrite in progress, cannot change metadata_uuid");
-		return 1;
-	}
+		if (new_fsid_string) {
+			uuid_t tmp;
 
-	if (new_fsid_string)
-		uuid_parse(new_fsid_string, fsid);
-	else
-		uuid_generate(fsid);
+			uuid_parse(new_fsid_string, tmp);
+			if (memcmp(tmp, fsid, BTRFS_FSID_SIZE)) {
+				error(
+		"new fsid %s is not the same with unfinished fsid change",
+				      new_fsid_string);
+				return -EINVAL;
+			}
+		}
+	} else {
+		if (new_fsid_string)
+			uuid_parse(new_fsid_string, fsid);
+		else
+			uuid_generate(fsid);
+	}
 
 	new_fsid = (memcmp(fsid, disk_super->fsid, BTRFS_FSID_SIZE) != 0);
 
-- 
2.39.3

[PATCH 16/16] btrfs-progs: test btrfstune -m|M ability to fix previous failures

[Anand Jain]

The misc-test/034-metadata_uuid test case, has four sets of disk images to
simulate failed writes during btrfstune -m|M operations. As of now, this
tests kernel only. Update the test case to verify btrfstune -m|M's
capacity to recover from the same scenarios.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 tests/misc-tests/034-metadata-uuid/test.sh | 70 ++++++++++++++++------
 1 file changed, 53 insertions(+), 17 deletions(-)

diff --git a/tests/misc-tests/034-metadata-uuid/test.sh b/tests/misc-tests/034-metadata-uuid/test.sh
index f2daa76304de..6aa1cdcb47ae 100755
--- a/tests/misc-tests/034-metadata-uuid/test.sh
+++ b/tests/misc-tests/034-metadata-uuid/test.sh
@@ -195,13 +195,42 @@ check_multi_fsid_unchanged() {
 	check_flag_cleared "$1" "$2"
 }
 
-failure_recovery() {
+failure_recovery_progs() {
+	local image1
+	local image2
+	local loop1
+	local loop2
+	local devcount
+
+	image1=$(extract_image "$1")
+	image2=$(extract_image "$2")
+	loop1=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image1")
+	loop2=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image2")
+
+	run_check $SUDO_HELPER udevadm settle
+
+	# Scan to make sure btrfs detects both devices before trying to mount
+	#run_check "$TOP/btrfstune" -m --noscan --device="$loop1" "$loop2"
+	run_check "$TOP/btrfstune" -m "$loop2"
+
+	# perform any specific check
+	"$3" "$loop1" "$loop2"
+
+	# cleanup
+	run_check $SUDO_HELPER losetup -d "$loop1"
+	run_check $SUDO_HELPER losetup -d "$loop2"
+	rm -f -- "$image1" "$image2"
+}
+
+failure_recovery_kernel() {
 	local image1
 	local image2
 	local loop1
 	local loop2
 	local devcount
 
+	reload_btrfs
+
 	image1=$(extract_image "$1")
 	image2=$(extract_image "$2")
 	loop1=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image1")
@@ -226,47 +255,55 @@ failure_recovery() {
 	rm -f -- "$image1" "$image2"
 }
 
+failure_recovery() {
+	failure_recovery_progs $@
+	failure_recovery_kernel $@
+}
+
 reload_btrfs() {
 	run_check $SUDO_HELPER rmmod btrfs
 	run_check $SUDO_HELPER modprobe btrfs
 }
 
-# for full coverage we need btrfs to actually be a module
-modinfo btrfs > /dev/null 2>&1 || _not_run "btrfs must be a module"
-run_mayfail $SUDO_HELPER modprobe -r btrfs || _not_run "btrfs must be unloadable"
-run_mayfail $SUDO_HELPER modprobe btrfs || _not_run "loading btrfs module failed"
+test_progs() {
+	run_check_mkfs_test_dev
+	check_btrfstune
+
+	run_check_mkfs_test_dev
+	check_dump_super_output
 
-run_check_mkfs_test_dev
-check_btrfstune
+	run_check_mkfs_test_dev
+	check_image_restore
+}
+
+check_kernel_reloadable() {
+	# for full coverage we need btrfs to actually be a module
+	modinfo btrfs > /dev/null 2>&1 || _not_run "btrfs must be a module"
+	run_mayfail $SUDO_HELPER modprobe -r btrfs || _not_run "btrfs must be unloadable"
+	run_mayfail $SUDO_HELPER modprobe btrfs || _not_run "loading btrfs module failed"
+}
 
-run_check_mkfs_test_dev
-check_dump_super_output
+check_kernel_reloadable
 
-run_check_mkfs_test_dev
-check_image_restore
+test_progs
 
 # disk1 is an image which has no metadata uuid flags set and disk2 is part of
 # the same fs but has the in-progress flag set. Test that whicever is scanned
 # first will result in consistent filesystem.
 failure_recovery "./disk1.raw.xz" "./disk2.raw.xz" check_inprogress_flag
-reload_btrfs
 failure_recovery "./disk2.raw.xz" "./disk1.raw.xz" check_inprogress_flag
 
 # disk4 contains an image in with the in-progress flag set and disk 3 is part
 # of the same filesystem but has both METADATA_UUID incompat and a new
 # metadata uuid set. So disk 3 must always take precedence
-reload_btrfs
 failure_recovery "./disk3.raw.xz" "./disk4.raw.xz" check_completed
-reload_btrfs
 failure_recovery "./disk4.raw.xz" "./disk3.raw.xz" check_completed
 
 # disk5 contains an image which has undergone a successful fsid change more
 # than once, disk6 on the other hand is member of the same filesystem but
 # hasn't completed its last change. Thus it has both the FSID_CHANGING flag set
 # and METADATA_UUID flag set.
-reload_btrfs
 failure_recovery "./disk5.raw.xz" "./disk6.raw.xz" check_multi_fsid_change
-reload_btrfs
 failure_recovery "./disk6.raw.xz" "./disk5.raw.xz" check_multi_fsid_change
 
 # disk7 contains an image which has undergone a successful fsid change once to
@@ -275,5 +312,4 @@ failure_recovery "./disk6.raw.xz" "./disk5.raw.xz" check_multi_fsid_change
 # during the process change. So disk 7 looks as if it never underwent fsid change
 # and disk 8 has FSID_CHANGING_FLAG and METADATA_UUID but is stale.
 failure_recovery "./disk7.raw.xz" "./disk8.raw.xz" check_multi_fsid_unchanged
-reload_btrfs
 failure_recovery "./disk8.raw.xz" "./disk7.raw.xz" check_multi_fsid_unchanged
-- 
2.39.3

Re: [PATCH 16/16] btrfs-progs: test btrfstune -m|M ability to fix previous failures

[David Sterba]

On Mon, Aug 14, 2023 at 11:28:12PM +0800, Anand Jain wrote:
> The misc-test/034-metadata_uuid test case, has four sets of disk images to
> simulate failed writes during btrfstune -m|M operations. As of now, this
> tests kernel only. Update the test case to verify btrfstune -m|M's
> capacity to recover from the same scenarios.
> 
> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> ---
>  tests/misc-tests/034-metadata-uuid/test.sh | 70 ++++++++++++++++------
>  1 file changed, 53 insertions(+), 17 deletions(-)
> 
> diff --git a/tests/misc-tests/034-metadata-uuid/test.sh b/tests/misc-tests/034-metadata-uuid/test.sh
> index f2daa76304de..6aa1cdcb47ae 100755
> --- a/tests/misc-tests/034-metadata-uuid/test.sh
> +++ b/tests/misc-tests/034-metadata-uuid/test.sh
> @@ -195,13 +195,42 @@ check_multi_fsid_unchanged() {
>  	check_flag_cleared "$1" "$2"
>  }
>  
> -failure_recovery() {
> +failure_recovery_progs() {
> +	local image1
> +	local image2
> +	local loop1
> +	local loop2
> +	local devcount
> +
> +	image1=$(extract_image "$1")
> +	image2=$(extract_image "$2")
> +	loop1=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image1")
> +	loop2=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image2")
> +
> +	run_check $SUDO_HELPER udevadm settle
> +
> +	# Scan to make sure btrfs detects both devices before trying to mount
> +	#run_check "$TOP/btrfstune" -m --noscan --device="$loop1" "$loop2"

Why is this line here? It looks like a valid command line and then it's
confusing. If there's some condition like that you want to require
scanning then put it to comment and explain it, e.g. "we can't use
--noscan here because ...".

> +	run_check "$TOP/btrfstune" -m "$loop2"
> +
> +	# perform any specific check
> +	"$3" "$loop1" "$loop2"
> +
> +	# cleanup
> +	run_check $SUDO_HELPER losetup -d "$loop1"
> +	run_check $SUDO_HELPER losetup -d "$loop2"
> +	rm -f -- "$image1" "$image2"
> +}

Re: [PATCH 00/16] btrfs-progs: recover from failed metadata_uuid

[David Sterba]

On Mon, Aug 14, 2023 at 11:27:56PM +0800, Anand Jain wrote:
> The kernel reunites split-brained devices after a failed `btrfstune -m|M`
> operation. We can achieve the same in btrfs-progs. So port it here.
> Ref the discussion here:
> 
>    https://lore.kernel.org/all/1fa6802b-5812-14a8-3fc8-5da54bb5f79d@oracle.com/
> 
> Patch 1/16 wasn't integrated as part of the set
> 	[PATCH 00/10 v2] fixes and preparatory related to metadata_uuid
> it's now merged with this patchset.
> 
> Patches [2-6,11,12] are cleanup patches.
> 
> Patches [7,8,10] are preparatory.
> 
> Patch [9] addresses a bug.
> 
> Patches [13, 14, 15] provide recovery from previously failed
> `btrfstune -m|M` operations.
> 
> Patch [16] enhances the misc-test `034-metadata-uuid` to also validate this
> new recovery feature.
> 
> This set has been successfully tested with the btrfs-progs testsuite.
> 
> This patchset is on top the latest devel last commit:
>  8aba9b0052b6 btrfs-progs: btrfstune: consolidate error handling in main()
> 
> 
> Anand Jain (16):
>   btrfs-progs: track num_devices per fs_devices
>   btrfs-progs: tune can use local fs_info variable
>   btrfs-progs: rename set_metadata_uuid arg to new_fsid_str
>   btrfs-progs: rename set_metadata_uuid new_fsid to fsid
>   btrfs-progs: rename set_metadata_uuid new_uuid to new_fsid
>   btrfs-progs: rename set_metadata_uuid uuid_changed to fsid_changed
>   btrfs-progs: pass fsid in check_unfinished_fsid_change arg2
>   btrfs-progs: pass metadata_uuid in check_unfinished_fsid_change arg3
>   btrfs-progs: fix return without flag reset commit in tune
>   btrfs-progs: preparing the latest device's superblock for commit
>   btrfs-progs: rename fs_devices::list to match the kernel
>   btrfs-progs: rename fs_devices::latest_trans to match the kernel
>   btrfs-progs: tune use the latest bdev in fs_devices for super_copy
>   btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag
>   btrfs-progs: recover from the failed btrfstune -m|M
>   btrfs-progs: test btrfstune -m|M ability to fix previous failures

Patches added to devel, thanks.

Re: [PATCH 00/16] btrfs-progs: recover from failed metadata_uuid

[David Sterba]

On Thu, Aug 24, 2023 at 12:13:15AM +0200, David Sterba wrote:
> On Mon, Aug 14, 2023 at 11:27:56PM +0800, Anand Jain wrote:
> > The kernel reunites split-brained devices after a failed `btrfstune -m|M`
> > operation. We can achieve the same in btrfs-progs. So port it here.
> > Ref the discussion here:
> > 
> >    https://lore.kernel.org/all/1fa6802b-5812-14a8-3fc8-5da54bb5f79d@oracle.com/
> > 
> > Patch 1/16 wasn't integrated as part of the set
> > 	[PATCH 00/10 v2] fixes and preparatory related to metadata_uuid
> > it's now merged with this patchset.
> > 
> > Patches [2-6,11,12] are cleanup patches.
> > 
> > Patches [7,8,10] are preparatory.
> > 
> > Patch [9] addresses a bug.
> > 
> > Patches [13, 14, 15] provide recovery from previously failed
> > `btrfstune -m|M` operations.
> > 
> > Patch [16] enhances the misc-test `034-metadata-uuid` to also validate this
> > new recovery feature.
> > 
> > This set has been successfully tested with the btrfs-progs testsuite.
> > 
> > This patchset is on top the latest devel last commit:
> >  8aba9b0052b6 btrfs-progs: btrfstune: consolidate error handling in main()
> > 
> > 
> > Anand Jain (16):
> >   btrfs-progs: track num_devices per fs_devices
> >   btrfs-progs: tune can use local fs_info variable
> >   btrfs-progs: rename set_metadata_uuid arg to new_fsid_str
> >   btrfs-progs: rename set_metadata_uuid new_fsid to fsid
> >   btrfs-progs: rename set_metadata_uuid new_uuid to new_fsid
> >   btrfs-progs: rename set_metadata_uuid uuid_changed to fsid_changed
> >   btrfs-progs: pass fsid in check_unfinished_fsid_change arg2
> >   btrfs-progs: pass metadata_uuid in check_unfinished_fsid_change arg3
> >   btrfs-progs: fix return without flag reset commit in tune
> >   btrfs-progs: preparing the latest device's superblock for commit
> >   btrfs-progs: rename fs_devices::list to match the kernel
> >   btrfs-progs: rename fs_devices::latest_trans to match the kernel
> >   btrfs-progs: tune use the latest bdev in fs_devices for super_copy
> >   btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag
> >   btrfs-progs: recover from the failed btrfstune -m|M
> >   btrfs-progs: test btrfstune -m|M ability to fix previous failures
> 
> Patches added to devel, thanks.

On my machine the metadata uuid test does not run because the module is
not loadable, but the GH actions report a failure:
https://github.com/kdave/btrfs-progs/actions/runs/5956097489/job/16156138260

Re: [PATCH 00/16] btrfs-progs: recover from failed metadata_uuid

[Anand Jain]

On 24/08/2023 06:24, David Sterba wrote:
> On Thu, Aug 24, 2023 at 12:13:15AM +0200, David Sterba wrote:
>> On Mon, Aug 14, 2023 at 11:27:56PM +0800, Anand Jain wrote:
>>> The kernel reunites split-brained devices after a failed `btrfstune -m|M`
>>> operation. We can achieve the same in btrfs-progs. So port it here.
>>> Ref the discussion here:
>>>
>>>     https://lore.kernel.org/all/1fa6802b-5812-14a8-3fc8-5da54bb5f79d@oracle.com/
>>>
>>> Patch 1/16 wasn't integrated as part of the set
>>> 	[PATCH 00/10 v2] fixes and preparatory related to metadata_uuid
>>> it's now merged with this patchset.
>>>
>>> Patches [2-6,11,12] are cleanup patches.
>>>
>>> Patches [7,8,10] are preparatory.
>>>
>>> Patch [9] addresses a bug.
>>>
>>> Patches [13, 14, 15] provide recovery from previously failed
>>> `btrfstune -m|M` operations.
>>>
>>> Patch [16] enhances the misc-test `034-metadata-uuid` to also validate this
>>> new recovery feature.
>>>
>>> This set has been successfully tested with the btrfs-progs testsuite.
>>>
>>> This patchset is on top the latest devel last commit:
>>>   8aba9b0052b6 btrfs-progs: btrfstune: consolidate error handling in main()
>>>
>>>
>>> Anand Jain (16):
>>>    btrfs-progs: track num_devices per fs_devices
>>>    btrfs-progs: tune can use local fs_info variable
>>>    btrfs-progs: rename set_metadata_uuid arg to new_fsid_str
>>>    btrfs-progs: rename set_metadata_uuid new_fsid to fsid
>>>    btrfs-progs: rename set_metadata_uuid new_uuid to new_fsid
>>>    btrfs-progs: rename set_metadata_uuid uuid_changed to fsid_changed
>>>    btrfs-progs: pass fsid in check_unfinished_fsid_change arg2
>>>    btrfs-progs: pass metadata_uuid in check_unfinished_fsid_change arg3
>>>    btrfs-progs: fix return without flag reset commit in tune
>>>    btrfs-progs: preparing the latest device's superblock for commit
>>>    btrfs-progs: rename fs_devices::list to match the kernel
>>>    btrfs-progs: rename fs_devices::latest_trans to match the kernel
>>>    btrfs-progs: tune use the latest bdev in fs_devices for super_copy
>>>    btrfs-progs: add support to fix superblock with CHANGING_FSID_V2 flag
>>>    btrfs-progs: recover from the failed btrfstune -m|M
>>>    btrfs-progs: test btrfstune -m|M ability to fix previous failures
>>
>> Patches added to devel, thanks.
> 
> On my machine the metadata uuid test does not run because the module is
> not loadable, but the GH actions report a failure:
> https://github.com/kdave/btrfs-progs/actions/runs/5956097489/job/1615613826

Local VM successfully runs misc-tests/034*. However, on OCI, the same
error as GH. Error reports missing device. It appears, inconsistent
results due to the varying device scan order from system to system.
I am looking more.

Re: [PATCH 16/16] btrfs-progs: test btrfstune -m|M ability to fix previous failures

[Anand Jain]

On 24/08/2023 04:10, David Sterba wrote:
> On Mon, Aug 14, 2023 at 11:28:12PM +0800, Anand Jain wrote:
>> The misc-test/034-metadata_uuid test case, has four sets of disk images to
>> simulate failed writes during btrfstune -m|M operations. As of now, this
>> tests kernel only. Update the test case to verify btrfstune -m|M's
>> capacity to recover from the same scenarios.
>>
>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
>> ---
>>   tests/misc-tests/034-metadata-uuid/test.sh | 70 ++++++++++++++++------
>>   1 file changed, 53 insertions(+), 17 deletions(-)
>>
>> diff --git a/tests/misc-tests/034-metadata-uuid/test.sh b/tests/misc-tests/034-metadata-uuid/test.sh
>> index f2daa76304de..6aa1cdcb47ae 100755
>> --- a/tests/misc-tests/034-metadata-uuid/test.sh
>> +++ b/tests/misc-tests/034-metadata-uuid/test.sh
>> @@ -195,13 +195,42 @@ check_multi_fsid_unchanged() {
>>   	check_flag_cleared "$1" "$2"
>>   }
>>   
>> -failure_recovery() {
>> +failure_recovery_progs() {
>> +	local image1
>> +	local image2
>> +	local loop1
>> +	local loop2
>> +	local devcount
>> +
>> +	image1=$(extract_image "$1")
>> +	image2=$(extract_image "$2")
>> +	loop1=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image1")
>> +	loop2=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image2")
>> +
>> +	run_check $SUDO_HELPER udevadm settle
>> +

>> +	# Scan to make sure btrfs detects both devices before trying to mount >> +	#run_check "$TOP/btrfstune" -m --noscan --device="$loop1" "$loop2"
> 
> Why is this line here? It looks like a valid command line and then it's
> confusing. If there's some condition like that you want to require
> scanning then put it to comment and explain it, e.g. "we can't use
> --noscan here because ...".

My bad. Those two comment lines shouldn't be here, and it's okay to
remove them.
It was taken from the patchset that added the --device and --noscan
options. I'll clean it up.

Thanks, Anand

>> +	run_check "$TOP/btrfstune" -m "$loop2"
>> +
>> +	# perform any specific check
>> +	"$3" "$loop1" "$loop2"
>> +
>> +	# cleanup
>> +	run_check $SUDO_HELPER losetup -d "$loop1"
>> +	run_check $SUDO_HELPER losetup -d "$loop2"
>> +	rm -f -- "$image1" "$image2"
>> +}

Re: [PATCH 00/16] btrfs-progs: recover from failed metadata_uuid

[David Sterba]

On Thu, Aug 24, 2023 at 09:54:30PM +0800, Anand Jain wrote:
> >> Patches added to devel, thanks.
> > 
> > On my machine the metadata uuid test does not run because the module is
> > not loadable, but the GH actions report a failure:
> > https://github.com/kdave/btrfs-progs/actions/runs/5956097489/job/1615613826
> 
> Local VM successfully runs misc-tests/034*. However, on OCI, the same
> error as GH. Error reports missing device. It appears, inconsistent
> results due to the varying device scan order from system to system.
> I am looking more.

Patches 13-16 have been removed from devel until the issue is resolved.
I've enabled build tests for pull requests you can use the github CI for
testing too (open a PR against devel or master branch).

Re: [PATCH 00/16] btrfs-progs: recover from failed metadata_uuid

[Anand Jain]

On 8/25/23 19:53, David Sterba wrote:
> On Thu, Aug 24, 2023 at 09:54:30PM +0800, Anand Jain wrote:
>>>> Patches added to devel, thanks.
>>>
>>> On my machine the metadata uuid test does not run because the module is
>>> not loadable, but the GH actions report a failure:
>>> https://github.com/kdave/btrfs-progs/actions/runs/5956097489/job/1615613826
>>
>> Local VM successfully runs misc-tests/034*. However, on OCI, the same
>> error as GH. Error reports missing device. It appears, inconsistent
>> results due to the varying device scan order from system to system.
>> I am looking more.
> 
> Patches 13-16 have been removed from devel until the issue is resolved.
> I've enabled build tests for pull requests you can use the github CI for
> testing too (open a PR against devel or master branch).

Yes, I noticed that patches 1 to 12 have been merged. I am able to
reproduce the bug. V3 is tested working fine.