[PATCH 0/9] Introduce btrfs-modify prog to make corruption easier

[PATCH 0/9] Introduce btrfs-modify prog to make corruption easier

[Qu Wenruo]

Introduce a new command, btrfs-modify, which is not part of 'btrfs', but
an independent command, bring minimal impact to existing btrfs commands.

Btrfs-modify is designed to provides better documentation than current
btrfs-corrupt-block with better subcommand division to reduce confusing
or conflicting options.

Btrfs-modify paired with offline-scrub patchset (not merged yet) could
provide a full suite for test case writers to do corruption and recovery
verification.

Qu Wenruo (9):
  btrfs-progs: Introduce new btrfs_map_block function which returns more
    unified result.
  btrfs-progs: Allow __btrfs_map_block_v2 to remove unrelated stripes
  btrfs-progs: Export commands processing code to commands.c from
    btrfs.c
  btrfs-progs: help: Unbind short help description from btrfs
  btrfs-progs: utils: Introduce new function arg_strtou32
  btrfs-progs: Introduce btrfs-modify tool to modify btrfs internal
    structures
  btrfs-progs: modify: Add support to corrupt specified mirror
  btrfs-progs: modify: Introduce option to specify range by root,ino and
    offset
  btrfs-progs: modify: Introduce option to specify the pattern to fill
    mirror

 .gitignore                          |   1 +
 Documentation/Makefile.in           |   1 +
 Documentation/btrfs-modify.asciidoc |  64 +++++
 Makefile                            |  14 +-
 btrfs.c                             | 116 +--------
 commands.c                          | 127 ++++++++++
 commands.h                          |   4 +
 help.c                              |  14 +-
 help.h                              |   3 +-
 modify/main.c                       | 110 +++++++++
 modify/mirror.c                     | 472 ++++++++++++++++++++++++++++++++++++
 modify/modify_commands.h            |  25 ++
 utils-lib.c                         |  15 ++
 utils.h                             |   1 +
 volumes.c                           | 283 +++++++++++++++++++++
 volumes.h                           |  78 ++++++
 16 files changed, 1213 insertions(+), 115 deletions(-)
 create mode 100644 Documentation/btrfs-modify.asciidoc
 create mode 100644 commands.c
 create mode 100644 modify/main.c
 create mode 100644 modify/mirror.c
 create mode 100644 modify/modify_commands.h

-- 
2.12.2

[PATCH 1/9] btrfs-progs: Introduce new btrfs_map_block function which returns more unified result.

[Qu Wenruo]

Introduce a new function, __btrfs_map_block_v2().

Unlike old btrfs_map_block(), which needs different parameter to handle
different RAID profile, this new function uses unified btrfs_map_block
structure to handle all RAID profile in a more meaningful method:

Return physical address along with logical address for each stripe.

For RAID1/Single/DUP (none-stripped):
result would be like:
Map block: Logical 128M, Len 10M, Type RAID1, Stripe len 0, Nr_stripes 2
Stripe 0: Logical 128M, Physical X, Len: 10M Dev dev1
Stripe 1: Logical 128M, Physical Y, Len: 10M Dev dev2

Result will be as long as possible, since it's not stripped at all.

For RAID0/10 (stripped without parity):
Result will be aligned to full stripe size:
Map block: Logical 64K, Len 128K, Type RAID10, Stripe len 64K, Nr_stripes 4
Stripe 0: Logical 64K, Physical X, Len 64K Dev dev1
Stripe 1: Logical 64K, Physical Y, Len 64K Dev dev2
Stripe 2: Logical 128K, Physical Z, Len 64K Dev dev3
Stripe 3: Logical 128K, Physical W, Len 64K Dev dev4

For RAID5/6 (stripped with parity and dev-rotation)
Result will be aligned to full stripe size:
Map block: Logical 64K, Len 128K, Type RAID6, Stripe len 64K, Nr_stripes 4
Stripe 0: Logical 64K, Physical X, Len 64K Dev dev1
Stripe 1: Logical 128K, Physical Y, Len 64K Dev dev2
Stripe 2: Logical RAID5_P, Physical Z, Len 64K Dev dev3
Stripe 3: Logical RAID6_Q, Physical W, Len 64K Dev dev4

The new unified layout should be very flex and can even handle things
like N-way RAID1 (which old mirror_num basic one can't handle well).

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 volumes.c | 181 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 volumes.h |  78 +++++++++++++++++++++++++++
 2 files changed, 259 insertions(+)

diff --git a/volumes.c b/volumes.c
index b350e259..29611bf7 100644
--- a/volumes.c
+++ b/volumes.c
@@ -1596,6 +1596,187 @@ out:
 	return 0;
 }
 
+static inline struct btrfs_map_block *alloc_map_block(int num_stripes)
+{
+	struct btrfs_map_block *ret;
+	int size;
+
+	size = sizeof(struct btrfs_map_stripe) * num_stripes +
+		sizeof(struct btrfs_map_block);
+	ret = malloc(size);
+	if (!ret)
+		return NULL;
+	memset(ret, 0, size);
+	return ret;
+}
+
+static int fill_full_map_block(struct map_lookup *map, u64 start, u64 length,
+			       struct btrfs_map_block *map_block)
+{
+	u64 profile = map->type & BTRFS_BLOCK_GROUP_PROFILE_MASK;
+	u64 bg_start = map->ce.start;
+	u64 bg_end = bg_start + map->ce.size;
+	u64 bg_offset = start - bg_start; /* offset inside the block group */
+	u64 fstripe_logical = 0;	/* Full stripe start logical bytenr */
+	u64 fstripe_size = 0;		/* Full stripe logical size */
+	u64 fstripe_phy_off = 0;	/* Full stripe offset in each dev */
+	u32 stripe_len = map->stripe_len;
+	int sub_stripes = map->sub_stripes;
+	int data_stripes = nr_data_stripes(map);
+	int dev_rotation;
+	int i;
+
+	map_block->num_stripes = map->num_stripes;
+	map_block->type = profile;
+
+	/*
+	 * Common full stripe data for stripe based profiles
+	 */
+	if (profile & (BTRFS_BLOCK_GROUP_RAID0 | BTRFS_BLOCK_GROUP_RAID10 |
+		       BTRFS_BLOCK_GROUP_RAID5 | BTRFS_BLOCK_GROUP_RAID6)) {
+		fstripe_size = stripe_len * data_stripes;
+		if (sub_stripes)
+			fstripe_size /= sub_stripes;
+		fstripe_logical = bg_offset / fstripe_size * fstripe_size +
+				    bg_start;
+		fstripe_phy_off = bg_offset / fstripe_size * stripe_len;
+	}
+
+	switch (profile) {
+	case BTRFS_BLOCK_GROUP_DUP:
+	case BTRFS_BLOCK_GROUP_RAID1:
+	case 0: /* SINGLE */
+		/*
+		 * None-stripe mode,(Single, DUP and RAID1)
+		 * Just use offset to fill map_block
+		 */
+		map_block->stripe_len = 0;
+		map_block->start = start;
+		map_block->length = min(bg_end, start + length) - start;
+		for (i = 0; i < map->num_stripes; i++) {
+			struct btrfs_map_stripe *stripe;
+
+			stripe = &map_block->stripes[i];
+
+			stripe->dev = map->stripes[i].dev;
+			stripe->logical = start;
+			stripe->physical = map->stripes[i].physical + bg_offset;
+			stripe->length = map_block->length;
+		}
+		break;
+	case BTRFS_BLOCK_GROUP_RAID10:
+	case BTRFS_BLOCK_GROUP_RAID0:
+		/*
+		 * Stripe modes without parity(0 and 10)
+		 * Return the whole full stripe
+		 */
+
+		map_block->start = fstripe_logical;
+		map_block->length = fstripe_size;
+		map_block->stripe_len = map->stripe_len;
+		for (i = 0; i < map->num_stripes; i++) {
+			struct btrfs_map_stripe *stripe;
+			u64 cur_offset;
+
+			/* Handle RAID10 sub stripes */
+			if (sub_stripes)
+				cur_offset = i / sub_stripes * stripe_len;
+			else
+				cur_offset = stripe_len * i;
+			stripe = &map_block->stripes[i];
+
+			stripe->dev = map->stripes[i].dev;
+			stripe->logical = fstripe_logical + cur_offset;
+			stripe->length = stripe_len;
+			stripe->physical = map->stripes[i].physical +
+					   fstripe_phy_off;
+		}
+		break;
+	case BTRFS_BLOCK_GROUP_RAID5:
+	case BTRFS_BLOCK_GROUP_RAID6:
+		/*
+		 * Stripe modes with parity and device rotation(5 and 6)
+		 *
+		 * Return the whole full stripe
+		 */
+
+		dev_rotation = (bg_offset / fstripe_size) % map->num_stripes;
+
+		map_block->start = fstripe_logical;
+		map_block->length = fstripe_size;
+		map_block->stripe_len = map->stripe_len;
+		for (i = 0; i < map->num_stripes; i++) {
+			struct btrfs_map_stripe *stripe;
+			int dest_index;
+			u64 cur_offset = stripe_len * i;
+
+			stripe = &map_block->stripes[i];
+
+			dest_index = (i + dev_rotation) % map->num_stripes;
+			stripe->dev = map->stripes[dest_index].dev;
+			stripe->length = stripe_len;
+			stripe->physical = map->stripes[dest_index].physical +
+					   fstripe_phy_off;
+			if (i < data_stripes) {
+				/* data stripe */
+				stripe->logical = fstripe_logical +
+						  cur_offset;
+			} else if (i == data_stripes) {
+				/* P */
+				stripe->logical = BTRFS_RAID5_P_STRIPE;
+			} else {
+				/* Q */
+				stripe->logical = BTRFS_RAID6_Q_STRIPE;
+			}
+		}
+		break;
+	default:
+		return -EINVAL;
+	}
+	return 0;
+}
+
+int __btrfs_map_block_v2(struct btrfs_fs_info *fs_info, int rw, u64 logical,
+			 u64 length, struct btrfs_map_block **map_ret)
+{
+	struct cache_extent *ce;
+	struct map_lookup *map;
+	struct btrfs_map_block *map_block;
+	int ret;
+
+	/* Eearly parameter check */
+	if (!length || !map_ret) {
+		error("wrong parameter for %s", __func__);
+		return -EINVAL;
+	}
+
+	ce = search_cache_extent(&fs_info->mapping_tree.cache_tree, logical);
+	if (!ce)
+		return -ENOENT;
+	if (ce->start > logical)
+		return -ENOENT;
+
+	map = container_of(ce, struct map_lookup, ce);
+	/*
+	 * Allocate a full map_block anyway
+	 *
+	 * For write, we need the full map_block anyway.
+	 * For read, it will be striped to the needed stripe before returning.
+	 */
+	map_block = alloc_map_block(map->num_stripes);
+	if (!map_block)
+		return -ENOMEM;
+	ret = fill_full_map_block(map, logical, length, map_block);
+	if (ret < 0) {
+		free(map_block);
+		return ret;
+	}
+	/* TODO: Remove unrelated map_stripes for READ operation */
+
+	*map_ret = map_block;
+	return 0;
+}
+
 struct btrfs_device *btrfs_find_device(struct btrfs_root *root, u64 devid,
 				       u8 *uuid, u8 *fsid)
 {
diff --git a/volumes.h b/volumes.h
index 699b0bae..7454fbbc 100644
--- a/volumes.h
+++ b/volumes.h
@@ -108,6 +108,51 @@ struct map_lookup {
 	struct btrfs_bio_stripe stripes[];
 };
 
+struct btrfs_map_stripe {
+	struct btrfs_device *dev;
+
+	/*
+	 * Logical address of the stripe start.
+	 * Caller should check if this logical is the desired map start.
+	 * It's possible that the logical is smaller or larger than desired
+	 * map range.
+	 *
+	 * For P/Q stipre, it will be BTRFS_RAID5_P_STRIPE
+	 * and BTRFS_RAID6_Q_STRIPE.
+	 */
+	u64 logical;
+
+	u64 physical;
+
+	/* The length of the stripe */
+	u64 length;
+};
+
+struct btrfs_map_block {
+	/*
+	 * The logical start of the whole map block.
+	 * For RAID5/6 it will be the bytenr of the full stripe start,
+	 * so it's possible that @start is smaller than desired map range
+	 * start.
+	 */
+	u64 start;
+
+	/*
+	 * The logical length of the map block.
+	 * For RAID5/6 it will be total data stripe size
+	 */
+	u64 length;
+
+	/* Block group type */
+	u64 type;
+
+	/* Stripe length, for non-stripped mode, it will be 0 */
+	u32 stripe_len;
+
+	int num_stripes;
+	struct btrfs_map_stripe stripes[];
+};
+
 #define btrfs_multi_bio_size(n) (sizeof(struct btrfs_multi_bio) + \
 			    (sizeof(struct btrfs_bio_stripe) * (n)))
 #define btrfs_map_lookup_size(n) (sizeof(struct map_lookup) + \
@@ -187,6 +232,39 @@ int btrfs_map_block(struct btrfs_mapping_tree *map_tree, int rw,
 		    u64 logical, u64 *length,
 		    struct btrfs_multi_bio **multi_ret, int mirror_num,
 		    u64 **raid_map_ret);
+
+/*
+ * TODO: Use this map_block_v2 to replace __btrfs_map_block()
+ *
+ * New btrfs_map_block(), unlike old one, each stripe will contain the
+ * physical offset *AND* logical address.
+ * So caller won't ever need to care about how the stripe/mirror is organized.
+ * Which makes csum check quite easy.
+ *
+ * Only P/Q based profile needs to care their P/Q stripe.
+ *
+ * @map_ret example:
+ * Raid1:
+ * Map block: logical=128M len=10M type=RAID1 stripe_len=0 nr_stripes=2
+ * Stripe 0: logical=128M physical=X len=10M dev=devid1
+ * Stripe 1: logical=128M physical=Y len=10M dev=devid2
+ *
+ * Raid10:
+ * Map block: logical=64K len=128K type=RAID10 stripe_len=64K nr_stripes=4
+ * Stripe 0: logical=64K physical=X len=64K dev=devid1
+ * Stripe 1: logical=64K physical=Y len=64K dev=devid2
+ * Stripe 2: logical=128K physical=Z len=64K dev=devid3
+ * Stripe 3: logical=128K physical=W len=64K dev=devid4
+ *
+ * Raid6:
+ * Map block: logical=64K len=128K type=RAID6 stripe_len=64K nr_stripes=4
+ * Stripe 0: logical=64K physical=X len=64K dev=devid1
+ * Stripe 1: logical=128K physical=Y len=64K dev=devid2
+ * Stripe 2: logical=RAID5_P physical=Z len=64K dev=devid3
+ * Stripe 3: logical=RAID6_Q physical=W len=64K dev=devid4
+ */
+int __btrfs_map_block_v2(struct btrfs_fs_info *fs_info, int rw, u64 logical,
+			 u64 length, struct btrfs_map_block **map_ret);
 int btrfs_next_bg(struct btrfs_mapping_tree *map_tree, u64 *logical,
 		     u64 *size, u64 type);
 static inline int btrfs_next_bg_metadata(struct btrfs_mapping_tree *map_tree,
-- 
2.12.2

[PATCH 2/9] btrfs-progs: Allow __btrfs_map_block_v2 to remove unrelated stripes

[Qu Wenruo]

For READ, caller normally hopes to get what they request, other than
full stripe map.

In this case, we should remove unrelated stripe map, just like the
following case:
               32K               96K
               |<-request range->|
         0              64k           128K
RAID0:   |    Data 1    |   Data 2    |
              disk1         disk2
Before this patch, we return the full stripe:
Stripe 0: Logical 0, Physical X, Len 64K, Dev disk1
Stripe 1: Logical 64k, Physical Y, Len 64K, Dev disk2

After this patch, we limit the stripe result to the request range:
Stripe 0: Logical 32K, Physical X+32K, Len 32K, Dev disk1
Stripe 1: Logical 64k, Physical Y, Len 32K, Dev disk2

And if it's a RAID5/6 stripe, we just handle it like RAID0, ignoring
parities.

This should make caller easier to use.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 volumes.c | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 103 insertions(+), 1 deletion(-)

diff --git a/volumes.c b/volumes.c
index 29611bf7..5c40d153 100644
--- a/volumes.c
+++ b/volumes.c
@@ -1736,6 +1736,108 @@ static int fill_full_map_block(struct map_lookup *map, u64 start, u64 length,
 	return 0;
 }
 
+static void del_one_stripe(struct btrfs_map_block *map_block, int i)
+{
+	int cur_nr = map_block->num_stripes;
+	int size_left = (cur_nr - 1 - i) * sizeof(struct btrfs_map_stripe);
+
+	memmove(&map_block->stripes[i], &map_block->stripes[i + 1], size_left);
+	map_block->num_stripes--;
+}
+
+static void remove_unrelated_stripes(struct map_lookup *map,
+				     int rw, u64 start, u64 length,
+				     struct btrfs_map_block *map_block)
+{
+	int i = 0;
+	/*
+	 * RAID5/6 write must use full stripe.
+	 * No need to do anything.
+	 */
+	if (map->type & (BTRFS_BLOCK_GROUP_RAID5 | BTRFS_BLOCK_GROUP_RAID6) &&
+	    rw == WRITE)
+		return;
+
+	/*
+	 * For RAID0/1/10/DUP, whatever read/write, we can remove unrelated
+	 * stripes without causing anything wrong.
+	 * RAID5/6 READ is just like RAID0, we don't care parity unless we need
+	 * to recovery.
+	 * For recovery, rw should be set to WRITE.
+	 */
+	while (i < map_block->num_stripes) {
+		struct btrfs_map_stripe *stripe;
+		u64 orig_logical; /* Original stripe logical start */
+		u64 orig_end; /* Original stripe logical end */
+
+		stripe = &map_block->stripes[i];
+
+		/*
+		 * For READ, we don't really care parity
+		 */
+		if (stripe->logical == BTRFS_RAID5_P_STRIPE ||
+		    stripe->logical == BTRFS_RAID6_Q_STRIPE) {
+			del_one_stripe(map_block, i);
+			continue;
+		}
+		/* Completely unrelated stripe */
+		if (stripe->logical >= start + length ||
+		    stripe->logical + stripe->length <= start) {
+			del_one_stripe(map_block, i);
+			continue;
+		}
+		/* Covered stripe, modify its logical and physical */
+		orig_logical = stripe->logical;
+		orig_end = stripe->logical + stripe->length;
+		if (start + length <= orig_end) {
+			/*
+			 * |<--range-->|
+			 *   |  stripe   |
+			 * Or
+			 *     |<range>|
+			 *   |  stripe   |
+			 */
+			stripe->logical = max(orig_logical, start);
+			stripe->length = start + length - stripe->logical;
+			stripe->physical += stripe->logical - orig_logical;
+		} else if (start >= orig_logical) {
+			/*
+			 *     |<-range--->|
+			 * |  stripe     |
+			 * Or
+			 *     |<range>|
+			 * |  stripe     |
+			 */
+			stripe->logical = start;
+			stripe->length = min(orig_end, start + length) -
+					 stripe->logical;
+			stripe->physical += stripe->logical - orig_logical;
+		}
+		/*
+		 * Remaining case:
+		 * |<----range----->|
+		 *   | stripe |
+		 * No need to do any modification
+		 */
+		i++;
+	}
+
+	/* Recaculate map_block size */
+	map_block->start = 0;
+	map_block->length = 0;
+	for (i = 0; i < map_block->num_stripes; i++) {
+		struct btrfs_map_stripe *stripe;
+
+		stripe = &map_block->stripes[i];
+		if (stripe->logical > map_block->start)
+			map_block->start = stripe->logical;
+		if (stripe->logical + stripe->length >
+		    map_block->start + map_block->length)
+			map_block->length = stripe->logical + stripe->length -
+					    map_block->start;
+	}
+}
+
 int __btrfs_map_block_v2(struct btrfs_fs_info *fs_info, int rw, u64 logical,
 			 u64 length, struct btrfs_map_block **map_ret)
 {
@@ -1771,7 +1873,7 @@ int __btrfs_map_block_v2(struct btrfs_fs_info *fs_info, int rw, u64 logical,
 		free(map_block);
 		return ret;
 	}
-	/* TODO: Remove unrelated map_stripes for READ operation */
+	remove_unrelated_stripes(map, rw, logical, length, map_block);
 
 	*map_ret = map_block;
 	return 0;
-- 
2.12.2

[PATCH 3/9] btrfs-progs: Export commands processing code to commands.c from btrfs.c

[Qu Wenruo]

This allow us to the command group facility for other progs other than
btrfs command.
And since it's not bond to 'btrfs' command, move it to commands.c.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 Makefile   |   2 +-
 btrfs.c    | 104 --------------------------------------------------
 commands.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 commands.h |   4 ++
 4 files changed, 132 insertions(+), 105 deletions(-)
 create mode 100644 commands.c

diff --git a/Makefile b/Makefile
index 81598df1..f405c1ec 100644
--- a/Makefile
+++ b/Makefile
@@ -95,7 +95,7 @@ objects = ctree.o disk-io.o kernel-lib/radix-tree.o extent-tree.o print-tree.o \
 	  qgroup.o raid56.o free-space-cache.o kernel-lib/list_sort.o props.o \
 	  kernel-shared/ulist.o qgroup-verify.o backref.o string-table.o task-utils.o \
 	  inode.o file.o find-root.o free-space-tree.o help.o send-dump.o \
-	  fsfeatures.o
+	  fsfeatures.o commands.o
 cmds_objects = cmds-subvolume.o cmds-filesystem.o cmds-device.o cmds-scrub.o \
 	       cmds-inspect.o cmds-balance.o cmds-send.o cmds-receive.o \
 	       cmds-quota.o cmds-qgroup.o cmds-replace.o cmds-check.o \
diff --git a/btrfs.c b/btrfs.c
index 9214ae6e..f096e780 100644
--- a/btrfs.c
+++ b/btrfs.c
@@ -32,110 +32,6 @@ static const char * const btrfs_cmd_group_usage[] = {
 static const char btrfs_cmd_group_info[] =
 	"Use --help as an argument for information on a specific group or command.";
 
-static inline const char *skip_prefix(const char *str, const char *prefix)
-{
-	size_t len = strlen(prefix);
-	return strncmp(str, prefix, len) ? NULL : str + len;
-}
-
-static int parse_one_token(const char *arg, const struct cmd_group *grp,
-			   const struct cmd_struct **cmd_ret)
-{
-	const struct cmd_struct *cmd = grp->commands;
-	const struct cmd_struct *abbrev_cmd = NULL, *ambiguous_cmd = NULL;
-
-	for (; cmd->token; cmd++) {
-		const char *rest;
-
-		rest = skip_prefix(arg, cmd->token);
-		if (!rest) {
-			if (!prefixcmp(cmd->token, arg)) {
-				if (abbrev_cmd) {
-					/*
-					 * If this is abbreviated, it is
-					 * ambiguous. So when there is no
-					 * exact match later, we need to
-					 * error out.
-					 */
-					ambiguous_cmd = abbrev_cmd;
-				}
-				abbrev_cmd = cmd;
-			}
-			continue;
-		}
-		if (*rest)
-			continue;
-
-		*cmd_ret = cmd;
-		return 0;
-	}
-
-	if (ambiguous_cmd)
-		return -2;
-
-	if (abbrev_cmd) {
-		*cmd_ret = abbrev_cmd;
-		return 0;
-	}
-
-	return -1;
-}
-
-static const struct cmd_struct *
-parse_command_token(const char *arg, const struct cmd_group *grp)
-{
-	const struct cmd_struct *cmd = NULL;
-
-	switch(parse_one_token(arg, grp, &cmd)) {
-	case -1:
-		help_unknown_token(arg, grp);
-	case -2:
-		help_ambiguous_token(arg, grp);
-	}
-
-	return cmd;
-}
-
-static void handle_help_options_next_level(const struct cmd_struct *cmd,
-		int argc, char **argv)
-{
-	if (argc < 2)
-		return;
-
-	if (!strcmp(argv[1], "--help")) {
-		if (cmd->next) {
-			argc--;
-			argv++;
-			help_command_group(cmd->next, argc, argv);
-		} else {
-			usage_command(cmd, 1, 0);
-		}
-
-		exit(0);
-	}
-}
-
-int handle_command_group(const struct cmd_group *grp, int argc,
-			 char **argv)
-
-{
-	const struct cmd_struct *cmd;
-
-	argc--;
-	argv++;
-	if (argc < 1) {
-		usage_command_group(grp, 0, 0);
-		exit(1);
-	}
-
-	cmd = parse_command_token(argv[0], grp);
-
-	handle_help_options_next_level(cmd, argc, argv);
-
-	fixup_argv0(argv, cmd->token);
-	return cmd->fn(argc, argv);
-}
-
 static const struct cmd_group btrfs_cmd_group;
 
 static const char * const cmd_help_usage[] = {
diff --git a/commands.c b/commands.c
new file mode 100644
index 00000000..dc016e97
--- /dev/null
+++ b/commands.c
@@ -0,0 +1,127 @@
+/*
+ * Copyright (C) 2017 Fujitsu.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "commands.h"
+#include "utils.h"
+#include "help.h"
+
+static inline const char *skip_prefix(const char *str, const char *prefix)
+{
+	size_t len = strlen(prefix);
+	return strncmp(str, prefix, len) ? NULL : str + len;
+}
+
+static int parse_one_token(const char *arg, const struct cmd_group *grp,
+			   const struct cmd_struct **cmd_ret)
+{
+	const struct cmd_struct *cmd = grp->commands;
+	const struct cmd_struct *abbrev_cmd = NULL, *ambiguous_cmd = NULL;
+
+	for (; cmd->token; cmd++) {
+		const char *rest;
+
+		rest = skip_prefix(arg, cmd->token);
+		if (!rest) {
+			if (!prefixcmp(cmd->token, arg)) {
+				if (abbrev_cmd) {
+					/*
+					 * If this is abbreviated, it is
+					 * ambiguous. So when there is no
+					 * exact match later, we need to
+					 * error out.
+					 */
+					ambiguous_cmd = abbrev_cmd;
+				}
+				abbrev_cmd = cmd;
+			}
+			continue;
+		}
+		if (*rest)
+			continue;
+
+		*cmd_ret = cmd;
+		return 0;
+	}
+
+	if (ambiguous_cmd)
+		return -2;
+
+	if (abbrev_cmd) {
+		*cmd_ret = abbrev_cmd;
+		return 0;
+	}
+
+	return -1;
+}
+
+const struct cmd_struct *
+parse_command_token(const char *arg, const struct cmd_group *grp)
+{
+	const struct cmd_struct *cmd = NULL;
+
+	switch(parse_one_token(arg, grp, &cmd)) {
+	case -1:
+		help_unknown_token(arg, grp);
+	case -2:
+		help_ambiguous_token(arg, grp);
+	}
+
+	return cmd;
+}
+
+void handle_help_options_next_level(const struct cmd_struct *cmd,
+				    int argc, char **argv)
+{
+	if (argc < 2)
+		return;
+
+	if (!strcmp(argv[1], "--help")) {
+		if (cmd->next) {
+			argc--;
+			argv++;
+			help_command_group(cmd->next, argc, argv);
+		} else {
+			usage_command(cmd, 1, 0);
+		}
+
+		exit(0);
+	}
+}
+
+int handle_command_group(const struct cmd_group *grp, int argc,
+			 char **argv)
+{
+	const struct cmd_struct *cmd;
+
+	argc--;
+	argv++;
+	if (argc < 1) {
+		usage_command_group(grp, 0, 0);
+		exit(1);
+	}
+
+	cmd = parse_command_token(argv[0], grp);
+
+	handle_help_options_next_level(cmd, argc, argv);
+
+	fixup_argv0(argv, cmd->token);
+	return cmd->fn(argc, argv);
+}
diff --git a/commands.h b/commands.h
index 01bf387e..5e109537 100644
--- a/commands.h
+++ b/commands.h
@@ -111,4 +111,8 @@ int cmd_dump_super(int argc, char **argv);
 int cmd_debug_tree(int argc, char **argv);
 int cmd_rescue(int argc, char **argv);
 
+const struct cmd_struct *
+parse_command_token(const char *arg, const struct cmd_group *grp);
+void handle_help_options_next_level(const struct cmd_struct *cmd,
+				    int argc, char **argv);
 #endif
-- 
2.12.2

[PATCH 4/9] btrfs-progs: help: Unbind short help description from btrfs

[Qu Wenruo]

usage_command_group_short() always binds its description to 'btrfs',
making us unable to this function in other progs.

This patch makes the short description independent, so callers need to
pass the short description by themselves.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 btrfs.c | 12 +++++++++++-
 help.c  | 14 +++++++-------
 help.h  |  3 ++-
 3 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/btrfs.c b/btrfs.c
index f096e780..b3686c4b 100644
--- a/btrfs.c
+++ b/btrfs.c
@@ -24,6 +24,15 @@
 #include "utils.h"
 #include "help.h"
 
+static const char * const btrfs_short_desc[] = {
+	"For an overview of a given command use 'btrfs command --help'",
+	"or 'btrfs [command...] --help --full' to print all available options.",
+	"Any command name can be shortened as far as it stays unambiguous,",
+	"however it is recommended to use full command names in scripts.",
+	"All command groups have their manual page named 'btrfs-<group>'.",
+	NULL
+};
+
 static const char * const btrfs_cmd_group_usage[] = {
 	"btrfs [--help] [--version] <group> [<group>...] <command> [<args>]",
 	NULL
@@ -126,7 +135,8 @@ int main(int argc, char **argv)
 			if (!prefixcmp(argv[0], "--"))
 				argv[0] += 2;
 		} else {
-			usage_command_group_short(&btrfs_cmd_group);
+			usage_command_group_short(&btrfs_cmd_group,
+						  btrfs_short_desc);
 			exit(1);
 		}
 	}
diff --git a/help.c b/help.c
index 19b0d357..13c45ffd 100644
--- a/help.c
+++ b/help.c
@@ -262,7 +262,8 @@ static void usage_command_group_internal(const struct cmd_group *grp, int full,
 	}
 }
 
-void usage_command_group_short(const struct cmd_group *grp)
+void usage_command_group_short(const struct cmd_group *grp,
+			       const char * const *short_desc)
 {
 	const char * const *usagestr = grp->usagestr;
 	FILE *outf = stdout;
@@ -298,12 +299,11 @@ void usage_command_group_short(const struct cmd_group *grp)
 		fprintf(outf, "  %-16s  %s\n", cmd->token, cmd->usagestr[1]);
 	}
 
-	fputc('\n', outf);
-	fprintf(stderr, "For an overview of a given command use 'btrfs command --help'\n");
-	fprintf(stderr, "or 'btrfs [command...] --help --full' to print all available options.\n");
-	fprintf(stderr, "Any command name can be shortened as far as it stays unambiguous,\n");
-	fprintf(stderr, "however it is recommended to use full command names in scripts.\n");
-	fprintf(stderr, "All command groups have their manual page named 'btrfs-<group>'.\n");
+	if (short_desc) {
+		fputc('\n', outf);
+		while (*short_desc && **short_desc)
+			fprintf(outf, "%s\n", *short_desc++);
+	}
 }
 
 void usage_command_group(const struct cmd_group *grp, int full, int err)
diff --git a/help.h b/help.h
index 7458e745..9b190fb1 100644
--- a/help.h
+++ b/help.h
@@ -58,7 +58,8 @@ struct cmd_group;
 void usage(const char * const *usagestr) __attribute__((noreturn));
 void usage_command(const struct cmd_struct *cmd, int full, int err);
 void usage_command_group(const struct cmd_group *grp, int all, int err);
-void usage_command_group_short(const struct cmd_group *grp);
+void usage_command_group_short(const struct cmd_group *grp,
+			       const char * const *short_desc);
 
 void help_unknown_token(const char *arg, const struct cmd_group *grp) __attribute__((noreturn));
 void help_ambiguous_token(const char *arg, const struct cmd_group *grp) __attribute__((noreturn));
-- 
2.12.2

[PATCH 5/9] btrfs-progs: utils: Introduce new function arg_strtou32

[Qu Wenruo]

This function is just calling arg_strtou64() and then do extra
UINT32_MAX check.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 utils-lib.c | 15 +++++++++++++++
 utils.h     |  1 +
 2 files changed, 16 insertions(+)

diff --git a/utils-lib.c b/utils-lib.c
index 044f93fc..544f6750 100644
--- a/utils-lib.c
+++ b/utils-lib.c
@@ -44,6 +44,21 @@ u64 arg_strtou64(const char *str)
 }
 
 /*
+ * u32 version of arg_strtou*()
+ */
+u32 arg_strtou32(const char *str)
+{
+	u64 tmp;
+
+	tmp = arg_strtou64(str);
+	if (tmp >= UINT32_MAX) {
+		fprintf(stderr, "ERROR: %s is too large.\n", str);
+		exit(1);
+	}
+	return (u32)tmp;
+}
+
+/*
  * For a given:
  * - file or directory return the containing tree root id
  * - subvolume return its own tree id
diff --git a/utils.h b/utils.h
index 24d0a200..f6deb26c 100644
--- a/utils.h
+++ b/utils.h
@@ -94,6 +94,7 @@ const char *pretty_size_mode(u64 size, unsigned mode);
 u64 parse_size(char *s);
 u64 parse_qgroupid(const char *p);
 u64 arg_strtou64(const char *str);
+u32 arg_strtou32(const char *str);
 int arg_copy_path(char *dest, const char *src, int destlen);
 int open_file_or_dir(const char *fname, DIR **dirstream);
 int open_file_or_dir3(const char *fname, DIR **dirstream, int open_flags);
-- 
2.12.2

[PATCH 6/9] btrfs-progs: Introduce btrfs-modify tool to modify btrfs internal structures

[Qu Wenruo]

This patch introduce the basic framework of a new prog, btrfs-modify,
which is designed to modify variant internal btrfs structures and
provide the basis for btrfs recovery test cases.

Btrfs-modify prog uses the command group facility to make it extendable,
the final objective is to replace the less maintained and poorly
documented btrfs-corrupt prog.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 .gitignore                          |   1 +
 Documentation/Makefile.in           |   1 +
 Documentation/btrfs-modify.asciidoc |  35 ++++++++++++
 Makefile                            |  10 +++-
 modify/main.c                       | 108 ++++++++++++++++++++++++++++++++++++
 5 files changed, 154 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/btrfs-modify.asciidoc
 create mode 100644 modify/main.c

diff --git a/.gitignore b/.gitignore
index 43c0ed88..69bb2f7a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -41,6 +41,7 @@ libbtrfs.so.0
 libbtrfs.so.0.1
 library-test
 library-test-static
+btrfs-modify
 /fssum
 
 /tests/*-tests-results.txt
diff --git a/Documentation/Makefile.in b/Documentation/Makefile.in
index 539c6b55..272432f5 100644
--- a/Documentation/Makefile.in
+++ b/Documentation/Makefile.in
@@ -11,6 +11,7 @@ MAN8_TXT += btrfs-select-super.asciidoc
 MAN8_TXT += btrfstune.asciidoc
 MAN8_TXT += fsck.btrfs.asciidoc
 MAN8_TXT += mkfs.btrfs.asciidoc
+MAN8_TXT += btrfs-modify.asciidoc
 
 # Sub commands for btrfs
 MAN8_TXT += btrfs-subvolume.asciidoc
diff --git a/Documentation/btrfs-modify.asciidoc b/Documentation/btrfs-modify.asciidoc
new file mode 100644
index 00000000..7d6e8e73
--- /dev/null
+++ b/Documentation/btrfs-modify.asciidoc
@@ -0,0 +1,35 @@
+btrfs-modify(8)
+===============
+
+NAME
+----
+btrfs-modify - modify variant internal structure of a unmounted btrfs
+
+SYNOPSIS
+--------
+*btrfs-modify* <command> <options> <device>
+
+DESCRIPTION
+-----------
+*btrfs-modify* is used to modify variant btrfs internal structures, either for
+experienced user to fix filesystem, or corrupt fs for test purpose.
+
+COMMANDS
+--------
+Nothing yet
+
+OPTIONS
+-------
+Nothing yet
+
+AVAILABILITY
+------------
+*btrfs-modify* is part of btrfs-progs.
+Please refer to the btrfs wiki http://btrfs.wiki.kernel.org for
+further details.
+
+SEE ALSO
+--------
+`btrfs`(5),
+`btrfs`(8),
+`btrfs-inspect-internal`(8)
diff --git a/Makefile b/Makefile
index f405c1ec..633c4447 100644
--- a/Makefile
+++ b/Makefile
@@ -168,7 +168,7 @@ endif
 MAKEOPTS = --no-print-directory Q=$(Q)
 
 # build all by default
-progs = $(progs_install) btrfsck btrfs-corrupt-block
+progs = $(progs_install) btrfsck btrfs-corrupt-block btrfs-modify
 
 # install only selected
 progs_install = btrfs mkfs.btrfs btrfs-debug-tree \
@@ -394,6 +394,14 @@ btrfs-image.static: image/main.static.o $(static_objects) $(static_libbtrfs_obje
 	@echo "    [LD]     $@"
 	$(Q)$(CC) $(STATIC_CFLAGS) -o $@ $^ $(STATIC_LDFLAGS) $(STATIC_LIBS) $(STATIC_LIBS_COMP)
 
+btrfs-modify: modify/main.o $(objects) $(libs_static)
+	@echo "    [LD]     $@"
+	$(Q)$(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS) $(LIBS) $(LIBS_COMP)
+
+btrfs-modify.static: modify/main.static.o $(static_objects) $(static_libbtrfs_objects)
+	@echo "    [LD]     $@"
+	$(Q)$(CC) $(CFLAGS) -o $@ $^ $(STATIC_LDFLAGS) $(STATIC_LIBS) $(STATIC_LIBS_COMP)
+
 btrfs-convert: $(convert_objects) $(objects) $(libs_static)
 	@echo "    [LD]     $@"
 	$(Q)$(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS) $(btrfs_convert_libs) $(LIBS)
diff --git a/modify/main.c b/modify/main.c
new file mode 100644
index 00000000..107e2e7e
--- /dev/null
+++ b/modify/main.c
@@ -0,0 +1,108 @@
+/*
+ * Copyright (C) 2017 Fujitsu.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <getopt.h>
+#include <limits.h>
+
+#include "kerncompat.h"
+#include "ctree.h"
+#include "volumes.h"
+#include "disk-io.h"
+#include "transaction.h"
+#include "list.h"
+#include "utils.h"
+#include "help.h"
+#include "commands.h"
+#include "crc32c.h"
+
+const char * const modify_group_usage[] = {
+	"btrfs-modify <command> <dest_options> <device>",
+	NULL
+};
+
+static const char * const modify_short_desc[] = {
+	"For an overview of a given command use 'btrfs-modify command --help'",
+	"or 'btrfs-modify [command...] --help --full' to print all available options.",
+	"Any command name can be shortened as far as it stays unambiguous,",
+	"however it is recommended to use full command names in scripts.",
+	"All command groups share the same man page named 'btrfs-modify'.",
+	NULL
+};
+
+static const char modify_group_info[] =
+	"Use --help as an argument for information on a specific group or command.";
+
+static const struct cmd_group modify_cmd_group = {
+	modify_group_usage, modify_group_info, {
+		NULL_CMD_STRUCT
+	},
+};
+
+static void check_options(int argc, char **argv)
+{
+	const char *arg;
+
+	if (argc == 0)
+		return;
+
+	arg = argv[0];
+
+	if (arg[0] != '-' ||
+	    !strncmp(arg, "--help", strlen("--help")))
+		return;
+	fprintf(stderr, "Unknown option: %s\n", arg);
+	fprintf(stderr, "usage: %s\n",
+		modify_cmd_group.usagestr[0]);
+	exit(129);
+}
+
+int main(int argc, char **argv)
+{
+	const struct cmd_struct *command;
+	int ret;
+
+	btrfs_config_init();
+
+	set_argv0(argv);
+	argc--;
+	argv++;
+
+	check_options(argc, argv);
+	if (argc == 0) {
+		usage_command_group_short(&modify_cmd_group, modify_short_desc);
+		exit(1);
+	}
+
+	command = parse_command_token(argv[0], &modify_cmd_group);
+
+	handle_help_options_next_level(command, argc, argv);
+
+	crc32c_optimization_init();
+
+	fixup_argv0(argv, command->token);
+
+	ret = command->fn(argc, argv);
+
+	btrfs_close_all_devices();
+
+	exit(!!ret);
+}
-- 
2.12.2

[PATCH 7/9] btrfs-progs: modify: Add support to corrupt specified mirror

[Qu Wenruo]

Add a subcommand, mirror, for btrfs-modify.

The mirror subcommand is used to modify specified mirror (P/Q included) of
a logical range.

Currently the mirror can be specified by --logical, --length and
--mirror.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 Documentation/btrfs-modify.asciidoc |  28 +++-
 Makefile                            |   6 +-
 modify/main.c                       |   2 +
 modify/mirror.c                     | 282 ++++++++++++++++++++++++++++++++++++
 modify/modify_commands.h            |  25 ++++
 5 files changed, 339 insertions(+), 4 deletions(-)
 create mode 100644 modify/mirror.c
 create mode 100644 modify/modify_commands.h

diff --git a/Documentation/btrfs-modify.asciidoc b/Documentation/btrfs-modify.asciidoc
index 7d6e8e73..ae2ada65 100644
--- a/Documentation/btrfs-modify.asciidoc
+++ b/Documentation/btrfs-modify.asciidoc
@@ -16,11 +16,35 @@ experienced user to fix filesystem, or corrupt fs for test purpose.
 
 COMMANDS
 --------
-Nothing yet
+*mirror*::
+Modify specified mirror, including RAID56 P/Q stripe.
 
 OPTIONS
 -------
-Nothing yet
+*Mirror command options*::
+--logical <address>::::
+Specify the start logical address to modify. Must be aligned to sectorsize.
++
+Must be specified
+
+--length <length>::::
+Specify the length to modify. Must be aligned to sectorsize.
++
+Default value is the sectorsize of the filesystem, if not specified.
+
+--stripe <stripe number>::::
+Specify the stripe number to modify.
++
+0 for 1st data stripe. All profiles support stripe number 0. +
+1 for backup data stripe. Only mirror based profiles (DUP,RAID1,RAID10) support
+stripe number 1.
+P for RAID56 1st parity stripe. Only RAID5 and RAID6 support stripe number P.
+Q for RAID6 2nd parity stripe.
+
+EXIT STATUS
+-----------
+*btrfs-modify* returns a zero exit status if all its operations succeed.
+None zero is returned if any error happens.
 
 AVAILABILITY
 ------------
diff --git a/Makefile b/Makefile
index 633c4447..18ea6051 100644
--- a/Makefile
+++ b/Makefile
@@ -103,6 +103,7 @@ cmds_objects = cmds-subvolume.o cmds-filesystem.o cmds-device.o cmds-scrub.o \
 	       cmds-property.o cmds-fi-usage.o cmds-inspect-dump-tree.o \
 	       cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o \
 	       mkfs/common.o
+modify_objects = modify/mirror.o
 libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
 		   kernel-lib/crc32c.o messages.o \
 		   uuid-tree.o utils-lib.o rbtree-utils.o
@@ -394,11 +395,11 @@ btrfs-image.static: image/main.static.o $(static_objects) $(static_libbtrfs_obje
 	@echo "    [LD]     $@"
 	$(Q)$(CC) $(STATIC_CFLAGS) -o $@ $^ $(STATIC_LDFLAGS) $(STATIC_LIBS) $(STATIC_LIBS_COMP)
 
-btrfs-modify: modify/main.o $(objects) $(libs_static)
+btrfs-modify: modify/main.o $(modify_objects) $(objects) $(libs_static)
 	@echo "    [LD]     $@"
 	$(Q)$(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS) $(LIBS) $(LIBS_COMP)
 
-btrfs-modify.static: modify/main.static.o $(static_objects) $(static_libbtrfs_objects)
+btrfs-modify.static: modify/main.static.o $(static_objects) $(static_libbtrfs_objects) $(modify_objects)
 	@echo "    [LD]     $@"
 	$(Q)$(CC) $(CFLAGS) -o $@ $^ $(STATIC_LDFLAGS) $(STATIC_LIBS) $(STATIC_LIBS_COMP)
 
@@ -514,6 +515,7 @@ clean: $(CLEANDIRS)
 		image/*.o image/*.o.d \
 		convert/*.o convert/*.o.d \
 		mkfs/*.o mkfs/*.o.d \
+		modify/*.o modify/*.o.d \
 	      dir-test ioctl-test quick-test library-test library-test-static \
 	      btrfs.static mkfs.btrfs.static fssum \
 	      $(check_defs) \
diff --git a/modify/main.c b/modify/main.c
index 107e2e7e..eca93fc2 100644
--- a/modify/main.c
+++ b/modify/main.c
@@ -33,6 +33,7 @@
 #include "help.h"
 #include "commands.h"
 #include "crc32c.h"
+#include "modify/modify_commands.h"
 
 const char * const modify_group_usage[] = {
 	"btrfs-modify <command> <dest_options> <device>",
@@ -53,6 +54,7 @@ static const char modify_group_info[] =
 
 static const struct cmd_group modify_cmd_group = {
 	modify_group_usage, modify_group_info, {
+		{ "mirror", modify_mirror, modify_mirror_usage, NULL, 0 },
 		NULL_CMD_STRUCT
 	},
 };
diff --git a/modify/mirror.c b/modify/mirror.c
new file mode 100644
index 00000000..f17dc9a7
--- /dev/null
+++ b/modify/mirror.c
@@ -0,0 +1,282 @@
+/*
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#include <strings.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <limits.h>
+#include <getopt.h>
+#include <unistd.h>
+#include "utils.h"
+#include "ctree.h"
+#include "kerncompat.h"
+#include "help.h"
+#include "disk-io.h"
+#include "volumes.h"
+#include "modify/modify_commands.h"
+
+const char * const modify_mirror_usage[] = {
+	"btrfs-modify mirror <options> <device>",
+	"Modify specified mirror/parity of a filesystem(unmounted).",
+	"<options> are used to specify the destination.",
+	"See 'btrfs-modify'(8) for supported options",
+	NULL
+};
+
+#define STRIPE_UNINITILIZED	(-1)
+#define STRIPE_P		(-2)
+#define STRIPE_Q		(-3)
+
+static char write_buf[BTRFS_STRIPE_LEN] = { 0 };
+
+static int strtostripe(const char *str)
+{
+	u32 tmp;
+
+	if (!strncasecmp(str, "p", strlen("p") + 1))
+		return STRIPE_P;
+	if (!strncasecmp(str, "q", strlen("q") + 1))
+		return STRIPE_Q;
+	tmp = arg_strtou32(str);
+	return (int)tmp;
+}
+
+static int write_range_fd(int fd, u64 offset, u64 len)
+{
+	u64 cur = offset;
+	int ret;
+
+	while (cur < offset + len) {
+		u64 write_len = min(offset + len - cur, (u64)BTRFS_STRIPE_LEN);
+		ret = pwrite(fd, write_buf, write_len, cur);
+		if (ret < 0)
+			return -errno;
+		cur += ret;
+	}
+	return 0;
+}
+
+static int corrupt_mapped_range(struct btrfs_fs_info *fs_info,
+				struct btrfs_map_block *map, u64 logical,
+				u64 len, int stripe_num)
+{
+	u64 mirror_profiles = BTRFS_BLOCK_GROUP_RAID1 |
+			      BTRFS_BLOCK_GROUP_RAID10 |
+			      BTRFS_BLOCK_GROUP_DUP;
+	u64 parity_profiles = BTRFS_BLOCK_GROUP_RAID5 |
+			      BTRFS_BLOCK_GROUP_RAID6;
+	int i;
+	int ret;
+
+	/* Check stripe_num with map->profiles */
+	if (!(map->type & mirror_profiles) && stripe_num > 0) {
+		error("logical range [%llu, %llu) doesn't have extra mirror",
+			map->start, map->length);
+		return -EINVAL;
+	}
+	if (stripe_num == STRIPE_P && !(map->type & parity_profiles)) {
+		error("logical range [%llu, %llu) doesn't have P stripe",
+			map->start, map->length);
+		return -EINVAL;
+	}
+	if (stripe_num == STRIPE_Q && !(map->type & BTRFS_BLOCK_GROUP_RAID6)) {
+		error("logical range [%llu, %llu) doesn't have Q stripe",
+			map->start, map->length);
+		return -EINVAL;
+	}
+
+	for (i = 0; i < map->num_stripes; i++) {
+		struct btrfs_map_stripe *stripe = &map->stripes[i];
+
+		u64 corrupt_logical = 0;
+		u64 corrupt_phy;
+		u64 corrupt_len;
+
+		if (stripe_num == STRIPE_P || stripe_num == STRIPE_Q) {
+			u64 dest_logical;
+
+			if (stripe_num == STRIPE_P)
+				dest_logical = BTRFS_RAID5_P_STRIPE;
+			else
+				dest_logical = BTRFS_RAID6_Q_STRIPE;
+			if (stripe->logical != dest_logical)
+				continue;
+
+			/* For P/Q, corrupt the whole stripe */
+			corrupt_phy = stripe->physical;
+			corrupt_len = stripe->length;
+		} else  {
+			/* Skip unrelated mirror stripe */
+			if (map->type & mirror_profiles && i % 2 != stripe_num)
+				continue;
+
+			corrupt_logical = max(stripe->logical, logical);
+			corrupt_phy = corrupt_logical - stripe->logical +
+					stripe->physical;
+			corrupt_len = min(stripe->logical + stripe->length,
+					logical + len) - corrupt_logical;
+		}
+		ret = write_range_fd(stripe->dev->fd, corrupt_phy,
+				     corrupt_len);
+		if (ret < 0) {
+			if (stripe_num == STRIPE_P || stripe_num == STRIPE_Q)
+				error(
+			"failded to write %s stripe for full stripe [%llu, %llu): %s",
+					(stripe_num == STRIPE_P ? "P" : "Q"),
+					map->start, map->start + map->length,
+					strerror(-ret));
+			else
+				error(
+			"failed to write data for logical range [%llu, %llu): %s",
+					corrupt_logical,
+					corrupt_logical + corrupt_len,
+					strerror(-ret));
+			return ret;
+		}
+	}
+	return 0;
+}
+
+static int modify_logical(struct btrfs_fs_info *fs_info, u64 logical, u64 len,
+			  int stripe)
+{
+	u32 sectorsize = fs_info->tree_root->sectorsize;
+	u64 cur;
+	int ret;
+
+	if (!IS_ALIGNED(logical, sectorsize)) {
+		error("logical address %llu is not aligned to sectorsize %d",
+			logical, sectorsize);
+		return -EINVAL;
+	}
+	if (!IS_ALIGNED(len, sectorsize)) {
+		error("length %llu is not aligned to sectorsize %d",
+			len, sectorsize);
+		return -EINVAL;
+	}
+	/* Current btrfs only support 1 mirror */
+	if (stripe > 1) {
+		error("btrfs only supports 1 mirror, stripe number %d is invalid",
+			stripe);
+		return -EINVAL;
+	}
+
+	cur = logical;
+
+	while (cur < logical + len) {
+		struct btrfs_map_block *map;
+
+		ret = __btrfs_map_block_v2(fs_info, WRITE, cur,
+					   logical + len - cur, &map);
+		if (ret < 0) {
+			error("failed to map logical range [%llu, %llu): %s",
+				cur, logical + len, strerror(-ret));
+			return ret;
+		}
+		ret = corrupt_mapped_range(fs_info, map, cur,
+					   logical + len - cur, stripe);
+		if (ret < 0) {
+			error("failed to modify on-disk data for range [%llu, %llu): %s",
+				cur, logical + len, strerror(-ret));
+			free(map);
+			return ret;
+		}
+		cur = map->start + map->length;
+	}
+	return 0;
+}
+
+int modify_mirror(int argc, char **argv)
+{
+	struct btrfs_fs_info *fs_info;
+	char *device;
+	u64 length = (u64)-1;
+	u64 logical = (u64)-1;
+	int stripe = STRIPE_UNINITILIZED;
+	int ret;
+
+	while (1) {
+		int c;
+		enum { GETOPT_VAL_LOGICAL = 257, GETOPT_VAL_LENGTH,
+			GETOPT_VAL_STRIPE };
+		static const struct option long_options[] = {
+			{ "logical", required_argument, NULL,
+				GETOPT_VAL_LOGICAL },
+			{ "length", required_argument, NULL,
+				GETOPT_VAL_LENGTH },
+			{ "stripe", required_argument, NULL, GETOPT_VAL_STRIPE }
+		};
+
+		c = getopt_long(argc, argv, "", long_options, NULL);
+		if (c < 0)
+			break;
+		switch (c) {
+		case GETOPT_VAL_LOGICAL:
+			logical = arg_strtou64(optarg);
+			break;
+		case GETOPT_VAL_LENGTH:
+			length = arg_strtou64(optarg);
+			break;
+		case GETOPT_VAL_STRIPE:
+			stripe = strtostripe(optarg);
+			break;
+		case '?':
+		case 'h':
+			usage(modify_mirror_usage);
+		}
+	}
+	if (check_argc_exact(argc - optind, 1))
+		usage(modify_mirror_usage);
+	device = argv[optind];
+	
+	ret = check_mounted(device);
+	if (ret < 0) {
+		error("could not check mount status for device %s: %s",
+			device, strerror(-ret));
+		return ret;
+	}
+	if (ret > 0) {
+		error("%s is currently mounted, aborting", device);
+		return -EINVAL;
+	}
+	if (logical == (u64)-1) {
+		error("--logical must be specified");
+		return -EINVAL;
+	}
+	if (stripe == STRIPE_UNINITILIZED) {
+		printf("--stripe not specified, fallback to 0 (1st stripe)\n");
+		stripe = 0;
+	}
+
+	fs_info = open_ctree_fs_info(device, 0, 0, 0, OPEN_CTREE_WRITES);
+	if (!fs_info) {
+		error("failed to open btrfs on device %s\n", device);
+		return -EIO;
+	}
+	if (length == (u64)-1) {
+		printf("--length not specified, fallback to sectorsize (%d)\n",
+			fs_info->tree_root->sectorsize);
+		length = fs_info->tree_root->sectorsize;
+	}
+	ret = modify_logical(fs_info, logical, length, stripe);
+	if (ret < 0)
+		error("failed to modify btrfs: %s", strerror(-ret));
+	else
+		printf("Succeeded in modifying specified mirror\n");
+
+	close_ctree(fs_info->tree_root);
+	return ret;
+}
diff --git a/modify/modify_commands.h b/modify/modify_commands.h
new file mode 100644
index 00000000..0d3e188f
--- /dev/null
+++ b/modify/modify_commands.h
@@ -0,0 +1,25 @@
+/*
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#ifndef __BTRFS_MODIFY_COMMANDS_H__
+#define __BTRFS_MODIFY_COMMANDS_H__
+
+#include "commands.h"
+
+extern const char * const modify_mirror_usage[];
+int modify_mirror(int argc, char **argv);
+
+#endif
-- 
2.12.2

[PATCH 8/9] btrfs-progs: modify: Introduce option to specify range by root,ino and offset

[Qu Wenruo]

The new option, --root-ino-offset, provides a better method to allow
test case scripts to corrupt specified range more easily.

Unlike the old method to call fiemap and then btrfs-map-logical, we can
use more easy-to-get values like rootid, ino, and in-file offset to
specify the destination.

Special note for special extents:
Compressed extent will be all modified if any of the extent get covered
by the destination.
Inline/prealloc/hole extent will be skipped.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 modify/mirror.c | 193 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 187 insertions(+), 6 deletions(-)

diff --git a/modify/mirror.c b/modify/mirror.c
index f17dc9a7..c89927f6 100644
--- a/modify/mirror.c
+++ b/modify/mirror.c
@@ -199,9 +199,177 @@ static int modify_logical(struct btrfs_fs_info *fs_info, u64 logical, u64 len,
 	return 0;
 }
 
+struct root_ino_offset {
+	u64 root;
+	u64 ino;
+	u64 offset;
+	bool set;
+};
+
+static int modify_root_ino_offset(struct btrfs_fs_info *fs_info,
+				  struct root_ino_offset *dest,
+				  u64 length, int stripe)
+{
+	struct btrfs_root *root;
+	struct btrfs_key key;
+	struct btrfs_path path;
+	u32 sectorsize = fs_info->tree_root->sectorsize;
+	u64 cur = dest->offset;
+	int ret;
+
+	if (!is_fstree(dest->root)) {
+		error("rootid %llu is not a valid subvolume id", dest->root);
+		return -EINVAL;
+	}
+	if (!IS_ALIGNED(dest->offset, sectorsize)) {
+		error("offset %llu is not aligned to sectorsize %u",
+			dest->offset, sectorsize);
+		return -EINVAL;
+	}
+
+	key.objectid = dest->root;
+	key.type = BTRFS_ROOT_ITEM_KEY;
+	key.offset = (u64)-1;
+
+	root = btrfs_read_fs_root(fs_info, &key);
+	if (IS_ERR(root)) {
+		ret = PTR_ERR(root);
+		error("failed to read out root %llu: %s",
+			dest->root, strerror(-ret));
+		return ret;
+	}
+
+	btrfs_init_path(&path);
+	key.objectid = dest->ino;
+	key.type = BTRFS_EXTENT_DATA_KEY;
+	key.offset = cur;
+
+	ret = btrfs_search_slot(NULL, root, &key, &path, 0, 0);
+	if (ret < 0)
+		goto out;
+	if (ret > 0) {
+		ret = btrfs_previous_item(root, &path, dest->ino,
+					  BTRFS_EXTENT_DATA_KEY);
+		if (ret < 0)
+			goto out;
+		if (ret > 0) {
+			error("root %llu ino %llu offset %llu not found",
+				dest->root, dest->ino, dest->offset);
+			ret = -ENOENT;
+			goto out;
+		}
+	}
+	while (cur < dest->offset + length) {
+		struct extent_buffer *leaf = path.nodes[0];
+		struct btrfs_file_extent_item *fi;
+		int slot = path.slots[0];
+		u64 corrupt_start;
+		u64 corrupt_len;
+
+		btrfs_item_key_to_cpu(leaf, &key, slot);
+		if (key.objectid != dest->ino ||
+		    key.type != BTRFS_EXTENT_DATA_KEY)
+			goto out;
+
+		fi = btrfs_item_ptr(leaf, slot, struct btrfs_file_extent_item);
+		/* Skip inline extent */
+		if (btrfs_file_extent_type(leaf, fi) ==
+				BTRFS_FILE_EXTENT_INLINE) {
+			cur = key.offset + sectorsize;
+			goto next;
+		}
+
+		/* Skip unrelated extent */
+		if (key.offset + btrfs_file_extent_num_bytes(leaf, fi) <=
+				dest->offset) {
+			cur = key.offset + btrfs_file_extent_num_bytes(leaf,
+					fi);
+			goto next;
+		}
+
+		/* Skip hole or prealloc extent */
+		if (btrfs_file_extent_disk_num_bytes(leaf, fi) == 0 ||
+		    btrfs_file_extent_type(leaf, fi) ==
+				BTRFS_FILE_EXTENT_PREALLOC) {
+			cur = key.offset + btrfs_file_extent_num_bytes(leaf,
+						fi);
+			goto next;
+		}
+
+		/* For compressed extent, corrupt all on-disk data */
+		if (btrfs_file_extent_compression(leaf, fi) !=
+			BTRFS_COMPRESS_NONE) {
+			ret = modify_logical(fs_info,
+				btrfs_file_extent_disk_bytenr(leaf, fi),
+				btrfs_file_extent_disk_num_bytes(leaf, fi),
+				stripe);
+			if (ret < 0)
+				goto out;
+			cur = key.offset +
+				btrfs_file_extent_num_bytes(leaf, fi);
+			goto next;
+		}
+
+		/* Regular plain extents, corrupt given range */
+		corrupt_start = btrfs_file_extent_disk_bytenr(leaf, fi) +
+			cur - key.offset + btrfs_file_extent_offset(leaf, fi);
+		corrupt_len = min(dest->offset + length, key.offset +
+				btrfs_file_extent_num_bytes(leaf, fi)) - cur;
+		ret = modify_logical(fs_info, corrupt_start, corrupt_len, stripe);
+		if (ret < 0)
+			goto out;
+		cur += corrupt_len;
+
+next:
+		ret = btrfs_next_item(root, &path);
+		if (ret < 0)
+			goto out;
+		if (ret > 0) {
+			ret = 0;
+			goto out;
+		}
+	}
+out:
+	btrfs_release_path(&path);
+	return ret;
+}
+
+static void parse_root_ino_offset(struct root_ino_offset *dest, char *optarg)
+{
+	char *this_char;
+	char *save_ptr = NULL;
+	int i = 0;
+
+	for (this_char = strtok_r(optarg, ",", &save_ptr);
+	     this_char != NULL;
+	     this_char = strtok_r(NULL, ",", &save_ptr)) {
+		switch (i) {
+		case 0:
+			dest->root = arg_strtou64(this_char);
+			break;
+		case 1:
+			dest->ino = arg_strtou64(this_char);
+			break;
+		case 2:
+			dest->offset = arg_strtou64(this_char);
+			break;
+		default:
+			goto error;
+		}
+		i++;
+	}
+error:
+	if (i != 3) {
+		error("--root-ino-offset must be specified in number,number,number form");
+		exit(1);
+	}
+	dest->set = true;
+}
+
 int modify_mirror(int argc, char **argv)
 {
 	struct btrfs_fs_info *fs_info;
+	struct root_ino_offset dest = { 0 };
 	char *device;
 	u64 length = (u64)-1;
 	u64 logical = (u64)-1;
@@ -211,13 +379,16 @@ int modify_mirror(int argc, char **argv)
 	while (1) {
 		int c;
 		enum { GETOPT_VAL_LOGICAL = 257, GETOPT_VAL_LENGTH,
-			GETOPT_VAL_STRIPE };
+			GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET };
 		static const struct option long_options[] = {
 			{ "logical", required_argument, NULL,
 				GETOPT_VAL_LOGICAL },
 			{ "length", required_argument, NULL,
 				GETOPT_VAL_LENGTH },
-			{ "stripe", required_argument, NULL, GETOPT_VAL_STRIPE }
+			{ "stripe", required_argument, NULL, GETOPT_VAL_STRIPE },
+			{ "root-ino-offset", required_argument, NULL,
+				GETOPT_VAL_ROOT_INO_OFFSET},
+			{ NULL, 0, NULL, 0 }
 		};
 
 		c = getopt_long(argc, argv, "", long_options, NULL);
@@ -233,6 +404,9 @@ int modify_mirror(int argc, char **argv)
 		case GETOPT_VAL_STRIPE:
 			stripe = strtostripe(optarg);
 			break;
+		case GETOPT_VAL_ROOT_INO_OFFSET:
+			parse_root_ino_offset(&dest, optarg);
+			break;
 		case '?':
 		case 'h':
 			usage(modify_mirror_usage);
@@ -252,9 +426,13 @@ int modify_mirror(int argc, char **argv)
 		error("%s is currently mounted, aborting", device);
 		return -EINVAL;
 	}
-	if (logical == (u64)-1) {
-		error("--logical must be specified");
-		return -EINVAL;
+	if (logical == (u64)-1 && !dest.set) {
+		error("--logical or --root-ino-offset must be specified");
+		return 1;
+	}
+	if (logical != (u64)-1 && dest.set) {
+		error("--logical conflicts with --root-ino-offset");
+		return 1;
 	}
 	if (stripe == STRIPE_UNINITILIZED) {
 		printf("--stripe not specified, fallback to 0 (1st stripe)\n");
@@ -271,7 +449,10 @@ int modify_mirror(int argc, char **argv)
 			fs_info->tree_root->sectorsize);
 		length = fs_info->tree_root->sectorsize;
 	}
-	ret = modify_logical(fs_info, logical, length, stripe);
+	if (logical != (u64)-1)
+		ret = modify_logical(fs_info, logical, length, stripe);
+	else
+		ret = modify_root_ino_offset(fs_info, &dest, length, stripe);
 	if (ret < 0)
 		error("failed to modify btrfs: %s", strerror(-ret));
 	else
-- 
2.12.2

[PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror

[Qu Wenruo]

Introduce a new option, --pattern, to specify the pattern to fill
mirror.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 Documentation/btrfs-modify.asciidoc |  5 +++++
 modify/mirror.c                     | 11 ++++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/Documentation/btrfs-modify.asciidoc b/Documentation/btrfs-modify.asciidoc
index ae2ada65..d82fb0df 100644
--- a/Documentation/btrfs-modify.asciidoc
+++ b/Documentation/btrfs-modify.asciidoc
@@ -41,6 +41,11 @@ stripe number 1.
 P for RAID56 1st parity stripe. Only RAID5 and RAID6 support stripe number P.
 Q for RAID6 2nd parity stripe.
 
+--pattern <pattern>::::
+Specify to pattern to modify the mirror data5f536652.
++
+Default value is "0x66524842" (ascii "BHRf", part of the btrfs magic "_BHRfS_M").
+
 EXIT STATUS
 -----------
 *btrfs-modify* returns a zero exit status if all its operations succeed.
diff --git a/modify/mirror.c b/modify/mirror.c
index c89927f6..dd717fbe 100644
--- a/modify/mirror.c
+++ b/modify/mirror.c
@@ -373,13 +373,15 @@ int modify_mirror(int argc, char **argv)
 	char *device;
 	u64 length = (u64)-1;
 	u64 logical = (u64)-1;
+	u32 pattern = 0x66524842;
 	int stripe = STRIPE_UNINITILIZED;
 	int ret;
 
 	while (1) {
 		int c;
 		enum { GETOPT_VAL_LOGICAL = 257, GETOPT_VAL_LENGTH,
-			GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET };
+			GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET,
+			GETOPT_VAL_PATTERN };
 		static const struct option long_options[] = {
 			{ "logical", required_argument, NULL,
 				GETOPT_VAL_LOGICAL },
@@ -388,6 +390,8 @@ int modify_mirror(int argc, char **argv)
 			{ "stripe", required_argument, NULL, GETOPT_VAL_STRIPE },
 			{ "root-ino-offset", required_argument, NULL,
 				GETOPT_VAL_ROOT_INO_OFFSET},
+			{ "pattern", required_argument, NULL,
+				GETOPT_VAL_PATTERN},
 			{ NULL, 0, NULL, 0 }
 		};
 
@@ -407,6 +411,9 @@ int modify_mirror(int argc, char **argv)
 		case GETOPT_VAL_ROOT_INO_OFFSET:
 			parse_root_ino_offset(&dest, optarg);
 			break;
+		case GETOPT_VAL_PATTERN:
+			pattern = arg_strtou32(optarg);
+			break;
 		case '?':
 		case 'h':
 			usage(modify_mirror_usage);
@@ -439,6 +446,8 @@ int modify_mirror(int argc, char **argv)
 		stripe = 0;
 	}
 
+	memset(write_buf, pattern, sizeof(write_buf) / sizeof(pattern));
+
 	fs_info = open_ctree_fs_info(device, 0, 0, 0, OPEN_CTREE_WRITES);
 	if (!fs_info) {
 		error("failed to open btrfs on device %s\n", device);
-- 
2.12.2

Re: [PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror

[Lakshmipathi.G]

Nice. With this new option, it should be possible to achieve something
like RAID56 corruption test-script say,
https://patchwork.kernel.org/patch/9583455/  right?
----
Cheers,
Lakshmipathi.G
FOSS Programmer.
http://www.giis.co.in http://www.webminal.org


On Mon, Apr 17, 2017 at 8:56 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
> Introduce a new option, --pattern, to specify the pattern to fill
> mirror.
>
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> ---
>  Documentation/btrfs-modify.asciidoc |  5 +++++
>  modify/mirror.c                     | 11 ++++++++++-
>  2 files changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/btrfs-modify.asciidoc b/Documentation/btrfs-modify.asciidoc
> index ae2ada65..d82fb0df 100644
> --- a/Documentation/btrfs-modify.asciidoc
> +++ b/Documentation/btrfs-modify.asciidoc
> @@ -41,6 +41,11 @@ stripe number 1.
>  P for RAID56 1st parity stripe. Only RAID5 and RAID6 support stripe number P.
>  Q for RAID6 2nd parity stripe.
>
> +--pattern <pattern>::::
> +Specify to pattern to modify the mirror data5f536652.
> ++
> +Default value is "0x66524842" (ascii "BHRf", part of the btrfs magic "_BHRfS_M").
> +
>  EXIT STATUS
>  -----------
>  *btrfs-modify* returns a zero exit status if all its operations succeed.
> diff --git a/modify/mirror.c b/modify/mirror.c
> index c89927f6..dd717fbe 100644
> --- a/modify/mirror.c
> +++ b/modify/mirror.c
> @@ -373,13 +373,15 @@ int modify_mirror(int argc, char **argv)
>         char *device;
>         u64 length = (u64)-1;
>         u64 logical = (u64)-1;
> +       u32 pattern = 0x66524842;
>         int stripe = STRIPE_UNINITILIZED;
>         int ret;
>
>         while (1) {
>                 int c;
>                 enum { GETOPT_VAL_LOGICAL = 257, GETOPT_VAL_LENGTH,
> -                       GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET };
> +                       GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET,
> +                       GETOPT_VAL_PATTERN };
>                 static const struct option long_options[] = {
>                         { "logical", required_argument, NULL,
>                                 GETOPT_VAL_LOGICAL },
> @@ -388,6 +390,8 @@ int modify_mirror(int argc, char **argv)
>                         { "stripe", required_argument, NULL, GETOPT_VAL_STRIPE },
>                         { "root-ino-offset", required_argument, NULL,
>                                 GETOPT_VAL_ROOT_INO_OFFSET},
> +                       { "pattern", required_argument, NULL,
> +                               GETOPT_VAL_PATTERN},
>                         { NULL, 0, NULL, 0 }
>                 };
>
> @@ -407,6 +411,9 @@ int modify_mirror(int argc, char **argv)
>                 case GETOPT_VAL_ROOT_INO_OFFSET:
>                         parse_root_ino_offset(&dest, optarg);
>                         break;
> +               case GETOPT_VAL_PATTERN:
> +                       pattern = arg_strtou32(optarg);
> +                       break;
>                 case '?':
>                 case 'h':
>                         usage(modify_mirror_usage);
> @@ -439,6 +446,8 @@ int modify_mirror(int argc, char **argv)
>                 stripe = 0;
>         }
>
> +       memset(write_buf, pattern, sizeof(write_buf) / sizeof(pattern));
> +
>         fs_info = open_ctree_fs_info(device, 0, 0, 0, OPEN_CTREE_WRITES);
>         if (!fs_info) {
>                 error("failed to open btrfs on device %s\n", device);
> --
> 2.12.2
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/9] Introduce btrfs-modify prog to make corruption easier

[David Sterba]

On Mon, Apr 17, 2017 at 11:26:33AM +0800, Qu Wenruo wrote:
> Introduce a new command, btrfs-modify, which is not part of 'btrfs', but
> an independent command, bring minimal impact to existing btrfs commands.
> 
> Btrfs-modify is designed to provides better documentation than current
> btrfs-corrupt-block with better subcommand division to reduce confusing
> or conflicting options.
> 
> Btrfs-modify paired with offline-scrub patchset (not merged yet) could
> provide a full suite for test case writers to do corruption and recovery
> verification.

Starting a new tool makes sense, the btrfs-corrupt-block lacks the
subcommand hierarchy and it would be tedious to sew it in.

The commands that you add now still seem ad-hoc, adressing current
needs. This is how the corrupt-block utility started as well. I'd like
see some proposal of more potential uses and some command gouping. For
example main group: set, get, map, delete, insert, show. Then, define
for which objects the commands are applicable. We can start with the
raid56 testing usecase. You want to locate and modify a block baced on
the logical offset, so this should comprise of 'map' + 'set'. For
specific tasks we can add shortcuts or compound commands, not from the
start.

Re: [PATCH 0/9] Introduce btrfs-modify prog to make corruption easier

[Qu Wenruo]

At 04/20/2017 02:15 AM, David Sterba wrote:
> On Mon, Apr 17, 2017 at 11:26:33AM +0800, Qu Wenruo wrote:
>> Introduce a new command, btrfs-modify, which is not part of 'btrfs', but
>> an independent command, bring minimal impact to existing btrfs commands.
>>
>> Btrfs-modify is designed to provides better documentation than current
>> btrfs-corrupt-block with better subcommand division to reduce confusing
>> or conflicting options.
>>
>> Btrfs-modify paired with offline-scrub patchset (not merged yet) could
>> provide a full suite for test case writers to do corruption and recovery
>> verification.
> 
> Starting a new tool makes sense, the btrfs-corrupt-block lacks the
> subcommand hierarchy and it would be tedious to sew it in.
> 
> The commands that you add now still seem ad-hoc, adressing current
> needs.

Yes, the "mirror" command is indeed for current corruption recovery test 
cases.

But the ability to add new command easily could make the tool more generic.

> This is how the corrupt-block utility started as well. I'd like
> see some proposal of more potential uses and some command gouping. For
> example main group: set, get, map, delete, insert, show.

While I prefer to create command groups by their logical level.
For example, "mirror" as the lowest level.
And then "leaf", allowing us to modify specified leaf headers.

Then "item", allowing us to modify btrfs item and its stored structure.
Including modifying, inserting, removing.

The short objective is to allow us to modify one-item-one-structure 
structure, like INODE_ITEM or EXTENT_DATA, but not EXTENT_ITEM(inlined).

By this method, we can reducing the option combinations by limiting some 
options under certain commands.
Or the logic to check option validation will be a hell.

Thanks,
Qu

> Then, define
> for which objects the commands are applicable. We can start with the
> raid56 testing usecase. You want to locate and modify a block baced on
> the logical offset, so this should comprise of 'map' + 'set'. For
> specific tasks we can add shortcuts or compound commands, not from the
> start.

Re: [PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror

[Lakshmipathi.G]

Thanks for the example and details. I understood some and need to
re-read couple of more times to understand the remaining.

btw, I created a corruption framework(with previous org), the sample
usage and example is below. It looks similar to Btrfs corruption tool.
thanks.

--
corrupt.py --help
Usage: corrupt.py [options] <arg1> <arg2> <arg3> ...]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -t object, --object=object
                        The object type to be corrupted [ choose from (inode,

l_inner,l_master,l_leaf,fileinode_inner,fileinode_leaf,dirinode_in
                        ner,dirinode_leaf,data_block,fec_block,baddr)
default: inode]
  -p /fs/path, --path=/fs/path
                        The file or directory path in /fs
  -q optype, --optype=optype
                        The corruption operation on the object [choose from
                        (set,xor,add) default: set]
  -o offset, --offset=offset
                        The valid offset value of the object [default:0]
  -m mirror, --mirror=mirror
                        The mirror number [default: 1]
  -s size, --size=size  The offset field/byte to be corrupted [default: 1]
  -c fixcrc, --fixcrc=fixcrc
                        This fixes the CRC's of the corrupted objects
  -n lbn, --lbn=lbn     The lbn of the data block to be corrupted
  -b baddr, --baddr=baddr
                        The baddr to be corrupted
  --log=LOG_FILE        This logs the result into the log file

Example usage:
corrupt.py --object=inode --offset=0 --size=4 --path=/fs/path/to/file.txt
Will corrupt first 4 bytes of inode structure.

corrupt.py --object=dirinode_inner --offset=0 --size=4
--path=/fs/path/to/file.txt
Will corrupt first 4 bytes of directory-inode structure.

corrupt.py --object=dirinode_leaf --offset=0 --size=4
--path=/fs/path/to/file.txt
Will corrupt first 4 bytes of directory-inode leaf structure.
(file.txt should be
large enough to contain leafs).

corrupt.py --object=inode --offset=0 --size=4
--path=/fs/path/to/file.txt  --mirror=2
Will corrupt first 4 bytes of inode structure that reside on Node/drive-2.

--optype=optype ==> default is set. It sets '0' to specified offset.
add => Will try given value to existing value. For example.
offset=(nlink's offset) --optype=add will increase the nlink count by
1.
(running fsck, expected to find this and fix it)

where as optype xor => should flip the bits at given offset.
--

----
Cheers,
Lakshmipathi.G
FOSS Programmer.
http://www.giis.co.in http://www.webminal.org

On Thu, Apr 20, 2017 at 6:30 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
>
> At 04/19/2017 03:46 PM, Lakshmipathi.G wrote:
>>
>> Thanks. This should make easier to create corruption test scripts.
>>
>> when using --root-ino-offset 5,257,0  Does final entry 0 has to be default value ? Does it represent any value at
>> this moment?
>
>
> For case 5,257,0, it means the start point of the corruption is root 5, inode 257 and file offset 0.
>
> So let's take the following file layout as an example (a little complex one, to show off all the ability)
>
> Root 5 inode 257:
>
>        \/ This is file offset
> Extent 0:   disk bytenr X, disk len 4K, nr_bytes 4K offset 0  plain
> Extent 4K:  disk bytenr Y, disk len 8K, nr_bytes 4K offset 0  plain
> Extent 8K:  disk bytenr Z, disk len 8K, nr_bytes 4K offset 4K plain
>
> And if you specified --root-ino-offset 5,257,0 and --length 12288
>
> Then the following on-disk range will be corrupted:
> [X, X+4K)
> [Y, Y+4K)
> [Z+4K, Z+8K)
>
> Making all the 0~12K of root5, inode257 get corrupted, while unrelated ranges like [Z, Z+4K) and [Y+4K, Y+8K) not modified.
>
> While for RAID1/RAID10/DUP usage, just create a file on a new fs, then pass --root-ino-offset 5,257,0 with --length <file_size> will corrupt the whole file for you, allowing us to test recovery.
>
>>
>> One more thing, does btrfs-modify command has option to print out existing parity value (all parity of a file or specific a parity)?
>> something like get_parity option?
>
>
> 1) For print parity value
> I didn't see the need to print parity value.
> As if you want to verify if the parity is good or not, just use offline scrub.
> It will detect parity/data corruption.
>
> 2) For printing/corrupting value of a file in RAID56 profile
> Not as easy as one may think.
>
> Think about the following case:
>
> Root 5 ino 257:
> Extent 0: disk bytenr X(rounded to stripe boundary), disk len 32K, nr_bytes 32K, offset 0.
> Extent 1: disk bytenr X+32K, disk len 64K, nr_bytes 64K, offset 0
>
> On disk it will be:
>
> |    Dev  1       |       Dev   2    |       Dev 3      |
> X                 X+64K              X+128K             X+192K
> |<-Ext 0->|<----- Ext 1---->|        |///// Parity /////|
>
> For Extent 0, it's OK to just print out/corrupt its value, as it's completely inside one data stripe.
>
> But for Extent 1, it crosses 2 device.
> If we corrupt on disk data of Extent1, we will corrupt both data stripe, leaving the full stripe unrecoverable.
>
> So for P/Q corrupting, I'm considering to record which full stripe we have corrupted and never corrupt any full stripe already damaged in next version.
>
> Thanks,
> Qu
>>
>>
>>
>>
>> ----
>> Cheers,
>> Lakshmipathi.G
>> FOSS Programmer.
>> http://www.giis.co.in http://www.webminal.org
>>
>> On Tue, Apr 18, 2017 at 12:43 PM, Qu Wenruo <quwenruo@cn.fujitsu.com <mailto:quwenruo@cn.fujitsu.com>> wrote:
>>
>>
>>
>>     At 04/18/2017 03:05 PM, Lakshmipathi.G wrote:
>>
>>         Nice. With this new option, it should be possible to achieve
>>         something
>>         like RAID56 corruption test-script say,
>>         https://patchwork.kernel.org/patch/9583455/
>>         <https://patchwork.kernel.org/patch/9583455/>  right?
>>
>>
>>     Yes, just for your use case!
>>
>>     It would be much more simpler to craft such recovery test case.
>>
>>     Just create a file in default subvolume (5), write large enough
>>     data, get the file ino (257 for first inode).
>>
>>     Then pass --root-ino-offset 5,257,0 --length 64k --stripe 0 to
>>     corrupt data stripe.
>>     Or change to --stripe P to corrupt parity.
>>
>>     Although it's still a little tricky to use current "--stripe" option
>>     to only corrupt data stripe (above 64K may crosses 2 data stripes).
>>     It shouldn't to be hard to add a new parameter --raid-stripe for
>>     such usage.
>>
>>     But it should be good enough for DUP/RAID1/RAID10.
>>
>>     Thanks,
>>     Qu
>>
>>
>>         ----
>>         Cheers,
>>         Lakshmipathi.G
>>         FOSS Programmer.
>>         http://www.giis.co.in http://www.webminal.org
>>
>>
>>         On Mon, Apr 17, 2017 at 8:56 AM, Qu Wenruo
>>         <quwenruo@cn.fujitsu.com <mailto:quwenruo@cn.fujitsu.com>> wrote:
>>
>>             Introduce a new option, --pattern, to specify the pattern to
>>             fill
>>             mirror.
>>
>>             Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com
>>             <mailto:quwenruo@cn.fujitsu.com>>
>>
>>             ---
>>                Documentation/btrfs-modify.asciidoc |  5 +++++
>>                modify/mirror.c                     | 11 ++++++++++-
>>                2 files changed, 15 insertions(+), 1 deletion(-)
>>
>>             diff --git a/Documentation/btrfs-modify.asciidoc
>>             b/Documentation/btrfs-modify.asciidoc
>>             index ae2ada65..d82fb0df 100644
>>             --- a/Documentation/btrfs-modify.asciidoc
>>             +++ b/Documentation/btrfs-modify.asciidoc
>>             @@ -41,6 +41,11 @@ stripe number 1.
>>                P for RAID56 1st parity stripe. Only RAID5 and RAID6
>>             support stripe number P.
>>                Q for RAID6 2nd parity stripe.
>>
>>             +--pattern <pattern>::::
>>             +Specify to pattern to modify the mirror data5f536652.
>>             ++
>>             +Default value is "0x66524842" (ascii "BHRf", part of the
>>             btrfs magic "_BHRfS_M").
>>             +
>>                EXIT STATUS
>>                -----------
>>                *btrfs-modify* returns a zero exit status if all its
>>             operations succeed.
>>             diff --git a/modify/mirror.c b/modify/mirror.c
>>             index c89927f6..dd717fbe 100644
>>             --- a/modify/mirror.c
>>             +++ b/modify/mirror.c
>>             @@ -373,13 +373,15 @@ int modify_mirror(int argc, char **argv)
>>                       char *device;
>>                       u64 length = (u64)-1;
>>                       u64 logical = (u64)-1;
>>             +       u32 pattern = 0x66524842;
>>                       int stripe = STRIPE_UNINITILIZED;
>>                       int ret;
>>
>>                       while (1) {
>>                               int c;
>>                               enum { GETOPT_VAL_LOGICAL = 257,
>>             GETOPT_VAL_LENGTH,
>>             -                       GETOPT_VAL_STRIPE,
>>             GETOPT_VAL_ROOT_INO_OFFSET };
>>             +                       GETOPT_VAL_STRIPE,
>>             GETOPT_VAL_ROOT_INO_OFFSET,
>>             +                       GETOPT_VAL_PATTERN };
>>                               static const struct option long_options[] = {
>>                                       { "logical", required_argument, NULL,
>>                                               GETOPT_VAL_LOGICAL },
>>             @@ -388,6 +390,8 @@ int modify_mirror(int argc, char **argv)
>>                                       { "stripe", required_argument,
>>             NULL, GETOPT_VAL_STRIPE },
>>                                       { "root-ino-offset",
>>             required_argument, NULL,
>>                                               GETOPT_VAL_ROOT_INO_OFFSET},
>>             +                       { "pattern", required_argument, NULL,
>>             +                               GETOPT_VAL_PATTERN},
>>                                       { NULL, 0, NULL, 0 }
>>                               };
>>
>>             @@ -407,6 +411,9 @@ int modify_mirror(int argc, char **argv)
>>                               case GETOPT_VAL_ROOT_INO_OFFSET:
>>                                       parse_root_ino_offset(&dest, optarg);
>>                                       break;
>>             +               case GETOPT_VAL_PATTERN:
>>             +                       pattern = arg_strtou32(optarg);
>>             +                       break;
>>                               case '?':
>>                               case 'h':
>>                                       usage(modify_mirror_usage);
>>             @@ -439,6 +446,8 @@ int modify_mirror(int argc, char **argv)
>>                               stripe = 0;
>>                       }
>>
>>             +       memset(write_buf, pattern, sizeof(write_buf) /
>>             sizeof(pattern));
>>             +
>>                       fs_info = open_ctree_fs_info(device, 0, 0, 0,
>>             OPEN_CTREE_WRITES);
>>                       if (!fs_info) {
>>                               error("failed to open btrfs on device
>>             %s\n", device);
>>             --
>>             2.12.2
>>
>>
>>
>>             --
>>             To unsubscribe from this list: send the line "unsubscribe
>>             linux-btrfs" in
>>             the body of a message to majordomo@vger.kernel.org
>>             <mailto:majordomo@vger.kernel.org>
>>             More majordomo info at
>>             http://vger.kernel.org/majordomo-info.html
>>             <http://vger.kernel.org/majordomo-info.html>
>>
>>         --
>>         To unsubscribe from this list: send the line "unsubscribe
>>         linux-btrfs" in
>>         the body of a message to majordomo@vger.kernel.org
>>         <mailto:majordomo@vger.kernel.org>
>>         More majordomo info at
>>         http://vger.kernel.org/majordomo-info.html
>>         <http://vger.kernel.org/majordomo-info.html>
>>
>>
>>
>>
>>
>
>

Re: [PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror

[David Sterba]

On Sun, Apr 23, 2017 at 01:12:42PM +0530, Lakshmipathi.G wrote:
> Thanks for the example and details. I understood some and need to
> re-read couple of more times to understand the remaining.
> 
> btw, I created a corruption framework(with previous org), the sample
> usage and example is below. It looks similar to Btrfs corruption tool.
> thanks.
> 
> --
> corrupt.py --help
[...]

Interesting, can you please share the script? This is another
alternative that seems more plausible for rapid prototyping of various
corruption scenarios. The C utility (either existing btrfs-corrupt-block
or the proposed btrfs-modify) can become tedious to change, but can be
compiled and distributed without the python dependency.

I wanted to use something python-based for tests when Hans announced the
python-btrfs project, but it has broader goals than just the testsuite
needs.  So we could have our own corrupt.py, just for our internal use.

I'm not sure if a compiled tool like btrfs-modify is really needed, but
why we can't have both.

Re: [PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror

[Lakshmipathi.G]

okay, but I'm not sure whether it can be useful here other than the
sample cli usage. The reason, this tool relies heavily on Python/C
layer[1]. The framework works like this:

C-api(fs-progs) <--> Python/C api <--> Python program.

C-api is typical fs library (like libext2fs/libbtrfs) and C/Python
used  to export every on-disk fs structure as an python-object along
with few fs-library functions. Finally in python script we can perform
stuffs like:

--
import fslib as fs

disk_object=fs.get_inode("/file/path")
#set few values finally make a call
retval = fs.corrupt_object(disk_object, offset, size, optype, fixcrc)
---

at C-python layer above python-object converted into C appropriate
data-type and passed on to C-layer.

hard-part will be troubleshooting issues, tracking them is difficult
because both gdb and pdb can help only upto certain point.

[1]: https://docs.python.org/2/extending/extending.html

Cheers.
Lakshmipathi.G


On 5/5/17, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> At 05/05/2017 12:53 AM, David Sterba wrote:
>> On Sun, Apr 23, 2017 at 01:12:42PM +0530, Lakshmipathi.G wrote:
>>> Thanks for the example and details. I understood some and need to
>>> re-read couple of more times to understand the remaining.
>>>
>>> btw, I created a corruption framework(with previous org), the sample
>>> usage and example is below. It looks similar to Btrfs corruption tool.
>>> thanks.
>>>
>>> --
>>> corrupt.py --help
>> [...]
>>
>> Interesting, can you please share the script? This is another
>> alternative that seems more plausible for rapid prototyping of various
>> corruption scenarios. The C utility (either existing btrfs-corrupt-block
>> or the proposed btrfs-modify) can become tedious to change, but can be
>> compiled and distributed without the python dependency.
>>
>> I wanted to use something python-based for tests when Hans announced the
>> python-btrfs project, but it has broader goals than just the testsuite
>> needs.  So we could have our own corrupt.py, just for our internal use.
>>
>> I'm not sure if a compiled tool like btrfs-modify is really needed, but
>> why we can't have both.
>>
>>
> Python based tool is always a good idea.
>
> If python based lib can provide the same capability of current C
> facilities (btrfs_search_slot() at least) and have better encapsulation,
> I'm totally committed to python based one.
>
> Although I'm a little afraid that the python script is not using the low
> level btrfs_search_slot(), but just getting chunk layout then do
> mathematics to get stripe location.
>
> Thanks,
> Qu
>
>
>