[PATCH 0/7] btrfs-progs: qgroup related enhance.

[PATCH 0/7] btrfs-progs: qgroup related enhance.

[Qu Wenruo]

The patchset include the following bugfix:
1) Update qgroup status flags to keep consistent with kernel

And the following enhancement:
1) debug-tree: Add human readable qgroup status flag.
2) qgroup show: Check status and print warning before if INCONSISTENT or
RESCANNING(*1)
3) parse_qgroup: Accept path and resolve it into qgroupid.
4) qgroup assign/remove: Auto schedule a rescan if the assign/remove
needs.(*2)

*1: The kernel has a bug that never clear INCONSISTENT flag, so need to
cooperate with the kernel patchset or warning is always output.

*2: Need to cooperate with the kernel patchset to detect such change.

Qu Wenruo (7):
  btrfs-progs: Update qgroup status flags and replace qgroup
    level/subvid     calculation with inline function.
  btrfs-progs: Allow btrfs-debug-tree to print human readable qgroup    
    status flag.
  btrfs-progs: Move parse_qgroupid() to utils.c
  btrfs-progs: Allow parse_qgroupid() to resolve subvolume path into    
    qgroupid.
  btrfs-progs: Add stack get/set functions for btrfs_qgroup_status_item.
  btrfs-progs: Print warning message if qgroup data is inconsistent.
  btrfs-progs: Schedule quota rescan if qgroup assign caused    
    inconsistence.

 cmds-qgroup.c | 26 ++++++++++++++++---
 ctree.h       | 26 ++++++++++++++++---
 print-tree.c  | 38 +++++++++++++++++++--------
 qgroup.c      | 82 +++++++++++++++++++++++++++++------------------------------
 qgroup.h      |  1 -
 utils.c       | 66 +++++++++++++++++++++++++++++++++++++++++++++++
 utils.h       |  1 +
 7 files changed, 179 insertions(+), 61 deletions(-)

-- 
2.3.0

[PATCH 1/7] btrfs-progs: Update qgroup status flags and replace qgroup level/subvid calculation with inline function.

[Qu Wenruo]

Ctree.h of btrfs-progs contains wrong flags for btrfs_qgroup_status.
Update it with the one in kernel.

Also, introduce the inline function btrfs_qgroup_(level/subvid) to get
the level/subvolid of qgroup, to replace the old open-coded bit
operations.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 cmds-qgroup.c |  2 +-
 ctree.h       | 17 +++++++++++++----
 print-tree.c  |  8 ++++----
 qgroup.c      | 28 ++++++++++++++++------------
 4 files changed, 34 insertions(+), 21 deletions(-)

diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index c5082f7..2d6d84b 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -52,7 +52,7 @@ static int qgroup_assign(int assign, int argc, char **argv)
 	/*
 	 * FIXME src should accept subvol path
 	 */
-	if ((args.src >> 48) >= (args.dst >> 48)) {
+	if (btrfs_qgroup_level(args.src) >= btrfs_qgroup_level(args.dst)) {
 		fprintf(stderr, "ERROR: bad relation requested '%s'\n", path);
 		return 1;
 	}
diff --git a/ctree.h b/ctree.h
index 901c340..1914a70 100644
--- a/ctree.h
+++ b/ctree.h
@@ -868,11 +868,20 @@ struct btrfs_csum_item {
  */
 #define BTRFS_SPACE_INFO_GLOBAL_RSV    (1ULL << 49)
 
-#define BTRFS_QGROUP_STATUS_OFF			0
-#define BTRFS_QGROUP_STATUS_ON			1
-#define BTRFS_QGROUP_STATUS_SCANNING		2
+#define BTRFS_QGROUP_LEVEL_SHIFT		48
+static inline u64 btrfs_qgroup_level(u64 qgroupid)
+{
+	return qgroupid >> BTRFS_QGROUP_LEVEL_SHIFT;
+}
+
+static inline u64 btrfs_qgroup_subvid(u64 qgroupid)
+{
+	return qgroupid & ((1ULL << BTRFS_QGROUP_LEVEL_SHIFT) - 1);
+}
 
-#define BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT	(1 << 0)
+#define BTRFS_QGROUP_STATUS_FLAG_ON		(1ULL << 0)
+#define BTRFS_QGROUP_STATUS_FLAG_RESCAN		(1ULL << 1)
+#define BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT	(1ULL << 2)
 
 struct btrfs_qgroup_status_item {
 	__le64 version;
diff --git a/print-tree.c b/print-tree.c
index 931a321..30adc1a 100644
--- a/print-tree.c
+++ b/print-tree.c
@@ -645,8 +645,8 @@ static void print_objectid(u64 objectid, u8 type)
 		printf("%llu", (unsigned long long)objectid); /* device id */
 		return;
 	case BTRFS_QGROUP_RELATION_KEY:
-		printf("%llu/%llu", objectid >> 48,
-			objectid & ((1ll << 48) - 1));
+		printf("%llu/%llu", btrfs_qgroup_level(objectid),
+		       btrfs_qgroup_subvid(objectid));
 		return;
 	case BTRFS_UUID_KEY_SUBVOL:
 	case BTRFS_UUID_KEY_RECEIVED_SUBVOL:
@@ -743,8 +743,8 @@ void btrfs_print_key(struct btrfs_disk_key *disk_key)
 	case BTRFS_QGROUP_RELATION_KEY:
 	case BTRFS_QGROUP_INFO_KEY:
 	case BTRFS_QGROUP_LIMIT_KEY:
-		printf(" %llu/%llu)", (unsigned long long)(offset >> 48),
-			(unsigned long long)(offset & ((1ll << 48) - 1)));
+		printf(" %llu/%llu)", btrfs_qgroup_level(offset),
+		       btrfs_qgroup_subvid(offset));
 		break;
 	case BTRFS_UUID_KEY_SUBVOL:
 	case BTRFS_UUID_KEY_RECEIVED_SUBVOL:
diff --git a/qgroup.c b/qgroup.c
index d59f4bb..62f8777 100644
--- a/qgroup.c
+++ b/qgroup.c
@@ -171,8 +171,9 @@ static int print_parent_column(struct btrfs_qgroup *qgroup)
 	int len = 0;
 
 	list_for_each_entry(list, &qgroup->qgroups, next_qgroup) {
-		len += printf("%llu/%llu", (list->qgroup)->qgroupid >> 48,
-			((1ll << 48) - 1) & (list->qgroup)->qgroupid);
+		len += printf("%llu/%llu",
+			      btrfs_qgroup_level(list->qgroup->qgroupid),
+			      btrfs_qgroup_subvid(list->qgroup->qgroupid));
 		if (!list_is_last(&list->next_qgroup, &qgroup->qgroups))
 			len += printf(",");
 	}
@@ -188,8 +189,9 @@ static int print_child_column(struct btrfs_qgroup *qgroup)
 	int len = 0;
 
 	list_for_each_entry(list, &qgroup->members, next_member) {
-		len += printf("%llu/%llu", (list->member)->qgroupid >> 48,
-				((1ll << 48) - 1) & (list->member)->qgroupid);
+		len += printf("%llu/%llu",
+			      btrfs_qgroup_level(list->member->qgroupid),
+			      btrfs_qgroup_subvid(list->member->qgroupid));
 		if (!list_is_last(&list->next_member, &qgroup->members))
 			len += printf(",");
 	}
@@ -218,8 +220,9 @@ static void print_qgroup_column(struct btrfs_qgroup *qgroup,
 	switch (column) {
 
 	case BTRFS_QGROUP_QGROUPID:
-		len = printf("%llu/%llu", qgroup->qgroupid >> 48,
-				((1ll << 48) - 1) & qgroup->qgroupid);
+		len = printf("%llu/%llu",
+			     btrfs_qgroup_level(qgroup->qgroupid),
+			     btrfs_qgroup_subvid(qgroup->qgroupid));
 		print_qgroup_column_add_blank(BTRFS_QGROUP_QGROUPID, len);
 		break;
 	case BTRFS_QGROUP_RFER:
@@ -920,8 +923,9 @@ static void __update_columns_max_len(struct btrfs_qgroup *bq,
 	switch (column) {
 
 	case BTRFS_QGROUP_QGROUPID:
-		sprintf(tmp, "%llu/%llu", (bq->qgroupid >> 48),
-			bq->qgroupid & ((1ll << 48) - 1));
+		sprintf(tmp, "%llu/%llu",
+			btrfs_qgroup_level(bq->qgroupid),
+			btrfs_qgroup_subvid(bq->qgroupid));
 		len = strlen(tmp);
 		if (btrfs_qgroup_columns[column].max_len < len)
 			btrfs_qgroup_columns[column].max_len = len;
@@ -950,8 +954,8 @@ static void __update_columns_max_len(struct btrfs_qgroup *bq,
 		len = 0;
 		list_for_each_entry(list, &bq->qgroups, next_qgroup) {
 			len += sprintf(tmp, "%llu/%llu",
-				(list->qgroup)->qgroupid >> 48,
-				((1ll << 48) - 1) & (list->qgroup)->qgroupid);
+				btrfs_qgroup_level(list->qgroup->qgroupid),
+				btrfs_qgroup_subvid(list->qgroup->qgroupid));
 			if (!list_is_last(&list->next_qgroup, &bq->qgroups))
 				len += 1;
 		}
@@ -962,8 +966,8 @@ static void __update_columns_max_len(struct btrfs_qgroup *bq,
 		len = 0;
 		list_for_each_entry(list, &bq->members, next_member) {
 			len += sprintf(tmp, "%llu/%llu",
-				(list->member)->qgroupid >> 48,
-				((1ll << 48) - 1) & (list->member)->qgroupid);
+				btrfs_qgroup_level(list->member->qgroupid),
+				btrfs_qgroup_subvid(list->member->qgroupid));
 			if (!list_is_last(&list->next_member, &bq->members))
 				len += 1;
 		}
-- 
2.3.0

[PATCH 2/7] btrfs-progs: Allow btrfs-debug-tree to print human readable qgroup status flag.

[Qu Wenruo]

Now btrfs-debug-tree can print qgroup status flag as ON|INCONSISTENT
instead of 0x5.

BTW, this patch helped us to find a bug that INCONSISTENT flag is never
cleared in kernel.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 print-tree.c | 30 +++++++++++++++++++++++-------
 1 file changed, 23 insertions(+), 7 deletions(-)

diff --git a/print-tree.c b/print-tree.c
index 30adc1a..41891fc 100644
--- a/print-tree.c
+++ b/print-tree.c
@@ -203,6 +203,20 @@ static void bg_flags_to_str(u64 flags, char *ret)
 	}
 }
 
+/* Caller should ensure sizeof(*ret)>= 26 "OFF|SCANNING|INCONSISTENT" */
+static void qg_flags_to_str(u64 flags, char *ret)
+{
+	if (flags & BTRFS_QGROUP_STATUS_FLAG_ON)
+		strcpy(ret, "ON");
+	else
+		strcpy(ret, "OFF");
+
+	if (flags & BTRFS_QGROUP_STATUS_FLAG_RESCAN)
+		strcat(ret, "|SCANNING");
+	if (flags & BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT)
+		strcat(ret, "|INCONSISTENT");
+}
+
 void print_chunk(struct extent_buffer *eb, struct btrfs_chunk *chunk)
 {
 	int num_stripes = btrfs_chunk_num_stripes(eb, chunk);
@@ -801,7 +815,7 @@ void btrfs_print_leaf(struct btrfs_root *root, struct extent_buffer *l)
 	u32 nr = btrfs_header_nritems(l);
 	u64 objectid;
 	u32 type;
-	char bg_flags_str[32];
+	char flags_str[32];
 
 	printf("leaf %llu items %d free space %d generation %llu owner %llu\n",
 		(unsigned long long)btrfs_header_bytenr(l), nr,
@@ -920,13 +934,13 @@ void btrfs_print_leaf(struct btrfs_root *root, struct extent_buffer *l)
 					    struct btrfs_block_group_item);
 			read_extent_buffer(l, &bg_item, (unsigned long)bi,
 					   sizeof(bg_item));
-			memset(bg_flags_str, 0, sizeof(bg_flags_str));
+			memset(flags_str, 0, sizeof(flags_str));
 			bg_flags_to_str(btrfs_block_group_flags(&bg_item),
-					bg_flags_str);
+					flags_str);
 			printf("\t\tblock group used %llu chunk_objectid %llu flags %s\n",
 			       (unsigned long long)btrfs_block_group_used(&bg_item),
 			       (unsigned long long)btrfs_block_group_chunk_objectid(&bg_item),
-			       bg_flags_str);
+			       flags_str);
 			break;
 		case BTRFS_CHUNK_ITEM_KEY:
 			print_chunk(l, btrfs_item_ptr(l, i, struct btrfs_chunk));
@@ -953,14 +967,16 @@ void btrfs_print_leaf(struct btrfs_root *root, struct extent_buffer *l)
 		case BTRFS_QGROUP_STATUS_KEY:
 			qg_status = btrfs_item_ptr(l, i,
 					struct btrfs_qgroup_status_item);
-			printf("\t\tversion %llu generation %llu flags %#llx "
+			memset(flags_str, 0, sizeof(flags_str));
+			qg_flags_to_str(btrfs_qgroup_status_flags(l, qg_status),
+					flags_str);
+			printf("\t\tversion %llu generation %llu flags %s "
 				"scan %lld\n",
 				(unsigned long long)
 				btrfs_qgroup_status_version(l, qg_status),
 				(unsigned long long)
 				btrfs_qgroup_status_generation(l, qg_status),
-				(unsigned long long)
-				btrfs_qgroup_status_flags(l, qg_status),
+				flags_str,
 				(unsigned long long)
 				btrfs_qgroup_status_scan(l, qg_status));
 			break;
-- 
2.3.0

[PATCH 3/7] btrfs-progs: Move parse_qgroupid() to utils.c

[Qu Wenruo]

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 qgroup.c | 28 ----------------------------
 qgroup.h |  1 -
 utils.c  | 28 ++++++++++++++++++++++++++++
 utils.h  |  1 +
 4 files changed, 29 insertions(+), 29 deletions(-)

diff --git a/qgroup.c b/qgroup.c
index 62f8777..7288365 100644
--- a/qgroup.c
+++ b/qgroup.c
@@ -1248,34 +1248,6 @@ int btrfs_qgroup_parse_sort_string(char *opt_arg,
 	return 0;
 }
 
-u64 parse_qgroupid(char *p)
-{
-	char *s = strchr(p, '/');
-	char *ptr_src_end = p + strlen(p);
-	char *ptr_parse_end = NULL;
-	u64 level;
-	u64 id;
-
-	if (!s) {
-		id = strtoull(p, &ptr_parse_end, 10);
-		if (ptr_parse_end != ptr_src_end)
-			goto err;
-		return id;
-	}
-	level = strtoull(p, &ptr_parse_end, 10);
-	if (ptr_parse_end != s)
-		goto err;
-
-	id = strtoull(s+1, &ptr_parse_end, 10);
-	if (ptr_parse_end != ptr_src_end)
-		goto  err;
-
-	return (level << 48) | id;
-err:
-	fprintf(stderr, "ERROR:invalid qgroupid\n");
-	exit(-1);
-}
-
 int qgroup_inherit_size(struct btrfs_qgroup_inherit *p)
 {
 	return sizeof(*p) + sizeof(p->qgroups[0]) *
diff --git a/qgroup.h b/qgroup.h
index 6e65ef7..22737fa 100644
--- a/qgroup.h
+++ b/qgroup.h
@@ -93,7 +93,6 @@ void btrfs_qgroup_free_comparer_set(struct btrfs_qgroup_comparer_set *comp_set);
 int btrfs_qgroup_setup_comparer(struct btrfs_qgroup_comparer_set **comp_set,
 				enum btrfs_qgroup_comp_enum comparer,
 				int is_descending);
-u64 parse_qgroupid(char *p);
 int qgroup_inherit_size(struct btrfs_qgroup_inherit *p);
 int qgroup_inherit_add_group(struct btrfs_qgroup_inherit **inherit, char *arg);
 int qgroup_inherit_add_copy(struct btrfs_qgroup_inherit **inherit, char *arg,
diff --git a/utils.c b/utils.c
index 6228c7f..fc2791b 100644
--- a/utils.c
+++ b/utils.c
@@ -1795,6 +1795,34 @@ u64 parse_size(char *s)
 	return ret;
 }
 
+u64 parse_qgroupid(char *p)
+{
+	char *s = strchr(p, '/');
+	char *ptr_src_end = p + strlen(p);
+	char *ptr_parse_end = NULL;
+	u64 level;
+	u64 id;
+
+	if (!s) {
+		id = strtoull(p, &ptr_parse_end, 10);
+		if (ptr_parse_end != ptr_src_end)
+			goto err;
+		return id;
+	}
+	level = strtoull(p, &ptr_parse_end, 10);
+	if (ptr_parse_end != s)
+		goto err;
+
+	id = strtoull(s+1, &ptr_parse_end, 10);
+	if (ptr_parse_end != ptr_src_end)
+		goto  err;
+
+	return (level << BTRFS_QGROUP_LEVEL_SHIFT) | id;
+err:
+	fprintf(stderr, "ERROR:invalid qgroupid\n");
+	exit(-1);
+}
+
 int open_file_or_dir3(const char *fname, DIR **dirstream, int open_flags)
 {
 	int ret;
diff --git a/utils.h b/utils.h
index 82ab5e8..97f3eee 100644
--- a/utils.h
+++ b/utils.h
@@ -113,6 +113,7 @@ int pretty_size_snprintf(u64 size, char *str, size_t str_bytes, unsigned unit_mo
 int get_mountpt(char *dev, char *mntpt, size_t size);
 int btrfs_scan_block_devices(int run_ioctl);
 u64 parse_size(char *s);
+u64 parse_qgroupid(char *p);
 u64 arg_strtou64(const char *str);
 int open_file_or_dir(const char *fname, DIR **dirstream);
 int open_file_or_dir3(const char *fname, DIR **dirstream, int open_flags);
-- 
2.3.0

[PATCH 4/7] btrfs-progs: Allow parse_qgroupid() to resolve subvolume path into qgroupid.

[Qu Wenruo]

Now parse_qgroupid() can resolve subvolume path into qgroupid.
This is quite handy for handling level 0 qgroupid, and user don't need
to resolve rootid by hand now.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 utils.c | 46 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 42 insertions(+), 4 deletions(-)

diff --git a/utils.c b/utils.c
index fc2791b..509c5ec 100644
--- a/utils.c
+++ b/utils.c
@@ -1708,6 +1708,25 @@ scan_again:
 }
 
 /*
+ * Unsafe subvolume check.
+ *
+ * This only checks ino == BTRFS_FIRST_FREE_OBJECTID, even it is not in a
+ * btrfs mount point.
+ * Must use together with other reliable method like btrfs ioctl.
+ */
+static int __is_subvol(char *path)
+{
+	struct stat st;
+	int ret;
+
+	ret = lstat(path, &st);
+	if (ret < 0)
+		return ret;
+
+	return st.st_ino == BTRFS_FIRST_FREE_OBJECTID;
+}
+
+/*
  * A not-so-good version fls64. No fascinating optimization since
  * no one except parse_size use it
  */
@@ -1802,24 +1821,43 @@ u64 parse_qgroupid(char *p)
 	char *ptr_parse_end = NULL;
 	u64 level;
 	u64 id;
+	int fd;
+	int ret = 0;
 
+	if (p[0] == '/')
+		goto path;
+
+	/* Numeric format like '0/257' is the primary case */
 	if (!s) {
 		id = strtoull(p, &ptr_parse_end, 10);
 		if (ptr_parse_end != ptr_src_end)
-			goto err;
+			goto path;
 		return id;
 	}
 	level = strtoull(p, &ptr_parse_end, 10);
 	if (ptr_parse_end != s)
-		goto err;
+		goto path;
 
 	id = strtoull(s+1, &ptr_parse_end, 10);
 	if (ptr_parse_end != ptr_src_end)
-		goto  err;
+		goto path;
 
 	return (level << BTRFS_QGROUP_LEVEL_SHIFT) | id;
+path:
+	/* Path format like subv at 'my_subvol' is the fallback case */
+	ret = __is_subvol(p);
+	if (ret < 0 || !ret)
+		goto err;
+	fd = open(p, O_RDONLY);
+	if (fd < 0)
+		goto err;
+	ret = lookup_ino_rootid(fd, &id);
+	close(fd);
+	if (ret < 0)
+		goto err;
+	return id;
 err:
-	fprintf(stderr, "ERROR:invalid qgroupid\n");
+	fprintf(stderr, "ERROR:invalid qgroupid or subvolume path\n");
 	exit(-1);
 }
 
-- 
2.3.0

[PATCH 5/7] btrfs-progs: Add stack get/set functions for btrfs_qgroup_status_item.

[Qu Wenruo]

This provides the basis for later qgroup related changes.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 ctree.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/ctree.h b/ctree.h
index 1914a70..bf29e85 100644
--- a/ctree.h
+++ b/ctree.h
@@ -2079,6 +2079,15 @@ BTRFS_SETGET_FUNCS(qgroup_status_flags, struct btrfs_qgroup_status_item,
 BTRFS_SETGET_FUNCS(qgroup_status_scan, struct btrfs_qgroup_status_item,
 		   scan, 64);
 
+BTRFS_SETGET_STACK_FUNCS(stack_qgroup_status_version,
+			 struct btrfs_qgroup_status_item, version, 64);
+BTRFS_SETGET_STACK_FUNCS(stack_qgroup_status_generation,
+			 struct btrfs_qgroup_status_item, generation, 64);
+BTRFS_SETGET_STACK_FUNCS(stack_qgroup_status_flags,
+			 struct btrfs_qgroup_status_item, flags, 64);
+BTRFS_SETGET_STACK_FUNCS(stack_qgroup_status_scan,
+			 struct btrfs_qgroup_status_item, scan, 64);
+
 /* btrfs_qgroup_info_item */
 BTRFS_SETGET_FUNCS(qgroup_info_generation, struct btrfs_qgroup_info_item,
 		   generation, 64);
-- 
2.3.0

[PATCH 6/7] btrfs-progs: Print warning message if qgroup data is inconsistent.

[Qu Wenruo]

Before this patch, qgroup show won't check btrfs qgroup status, so even
the INCONSISTENT flags is set, user is not aware of it.

This patch will include BTRFS_QGROUP_STATUS_ITEM in the search range and
check the flag, if there is any flag meaning the inconsistence of qgroup
data, info user.

NOTE: There is several kernel bugs from INCONSISTENT flags is always set
to RUNNING flags is not cleared until umount.
So this warning will always be here if using a newer kernel fixing these
bugs.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 qgroup.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/qgroup.c b/qgroup.c
index 7288365..4173846 100644
--- a/qgroup.c
+++ b/qgroup.c
@@ -1016,6 +1016,20 @@ static void __filter_and_sort_qgroups(struct qgroup_lookup *all_qgroups,
 		n = rb_prev(n);
 	}
 }
+
+static inline void print_status_flag_warning(u64 flags)
+{
+	if (!(flags & BTRFS_QGROUP_STATUS_FLAG_ON))
+		fprintf(stderr,
+		"WARNING: Quota disabled, qgroup data may be out of date\n");
+	else if (flags & BTRFS_QGROUP_STATUS_FLAG_RESCAN)
+		fprintf(stderr,
+		"WARNING: Rescan is running, qgroup data may be incorrect\n");
+	else if (flags & BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT)
+		fprintf(stderr,
+		"WARNING: Qgroup data inconsistent, rescan recommended\n");
+}
+
 static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
 {
 	int ret;
@@ -1039,7 +1053,7 @@ static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
 
 	sk->tree_id = BTRFS_QUOTA_TREE_OBJECTID;
 	sk->max_type = BTRFS_QGROUP_RELATION_KEY;
-	sk->min_type = BTRFS_QGROUP_INFO_KEY;
+	sk->min_type = BTRFS_QGROUP_STATUS_KEY;
 	sk->max_objectid = (u64)-1;
 	sk->max_offset = (u64)-1;
 	sk->max_transid = (u64)-1;
@@ -1070,7 +1084,15 @@ static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
 								  off);
 			off += sizeof(*sh);
 
-			if (sh->type == BTRFS_QGROUP_INFO_KEY) {
+			if (sh->type == BTRFS_QGROUP_STATUS_KEY) {
+				struct btrfs_qgroup_status_item *si;
+				u64 flags;
+
+				si = (struct btrfs_qgroup_status_item *)
+				     (args.buf + off);
+				flags = btrfs_stack_qgroup_status_flags(si);
+				print_status_flag_warning(flags);
+			} else if (sh->type == BTRFS_QGROUP_INFO_KEY) {
 				info = (struct btrfs_qgroup_info_item *)
 				       (args.buf + off);
 				a1 = btrfs_stack_qgroup_info_generation(info);
-- 
2.3.0

[PATCH 7/7] btrfs-progs: Schedule quota rescan if qgroup assign caused inconsistence.

[Qu Wenruo]

NOTE: This patch needs to cooperate with kernel patches, which will fix
a kernel bug that never clear INCONSISTENT bit and return 1 if quota
assign makes qgroup data inconsistent.

Some qgroup assign will cause qgroup data inconsistent, like remove a
qgroup with shared extents from a parent qgroup. But some won't, like
assign a empty(OK, nodesize rfer and exel) to a qgroup.

Newer kernel will return 1 if qgroup data inconsistent and in that case
we should schedule a quota rescan.

This patch will do this in btrfs-progs.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 cmds-qgroup.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index 2d6d84b..7d5f41f 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -64,13 +64,33 @@ static int qgroup_assign(int assign, int argc, char **argv)
 
 	ret = ioctl(fd, BTRFS_IOC_QGROUP_ASSIGN, &args);
 	e = errno;
-	close_file_or_dir(fd, dirstream);
 	if (ret < 0) {
 		fprintf(stderr, "ERROR: unable to assign quota group: %s\n",
 			strerror(e));
+		close_file_or_dir(fd, dirstream);
 		return 1;
 	}
-	return 0;
+
+	/*
+	 * If ret > 0, it means the assign caused qgroup data inconsistent.
+	 * Schedule a quota rescan.
+	 *
+	 * The return value change only happens in newer kernel. But will not
+	 * cause problem since old kernel has a bug that will never clear
+	 * INCONSISTENT bit.
+	 */
+	if (ret > 0) {
+		struct btrfs_ioctl_quota_rescan_args args;
+
+		printf("Quota data changed, quota rescan scheduled\n");
+		memset(&args, 0, sizeof(args));
+		ret = ioctl(fd, BTRFS_IOC_QUOTA_RESCAN, &args);
+		if (ret < 0)
+			fprintf(stderr, "ERROR: quota rescan failed: %s\n",
+				strerror(errno));
+	}
+	close_file_or_dir(fd, dirstream);
+	return ret;
 }
 
 static int qgroup_create(int create, int argc, char **argv)
-- 
2.3.0

Re: [PATCH 4/7] btrfs-progs: Allow parse_qgroupid() to resolve subvolume path into qgroupid.

[Qu Wenruo]

-------- Original Message  --------
Subject: Re: [PATCH 4/7] btrfs-progs: Allow parse_qgroupid() to resolve 
subvolume path into qgroupid.
From: Wang Shilong <wangshilong1991@gmail.com>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: 2015年02月27日 16:41

>
>>
>> Now parse_qgroupid() can resolve subvolume path into qgroupid.
>> This is quite handy for handling level 0 qgroupid, and user don't need
>> to resolve rootid by hand now.
>
> oh, not sure if this is safe, for example, if user do something like:
>
> # btrfs sub create 258(subvolume id = 300)
>
> if 258 is passing, so it will be treated as subvolume id?
Immediate number has higher privilege than path, so 258 is first 
considered as a subvolid and path will not be resolved, just as the old 
days.

If user really want to do things like that, passing "./258" will be the 
best method.

Thanks,
Qu

>
>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> ---
>> utils.c | 46 ++++++++++++++++++++++++++++++++++++++++++----
>> 1 file changed, 42 insertions(+), 4 deletions(-)
>>
>> diff --git a/utils.c b/utils.c
>> index fc2791b..509c5ec 100644
>> --- a/utils.c
>> +++ b/utils.c
>> @@ -1708,6 +1708,25 @@ scan_again:
>> }
>>
>> /*
>> + * Unsafe subvolume check.
>> + *
>> + * This only checks ino == BTRFS_FIRST_FREE_OBJECTID, even it is not in a
>> + * btrfs mount point.
>> + * Must use together with other reliable method like btrfs ioctl.
>> + */
>> +static int __is_subvol(char *path)
>> +{
>> +	struct stat st;
>> +	int ret;
>> +
>> +	ret = lstat(path, &st);
>> +	if (ret < 0)
>> +		return ret;
>> +
>> +	return st.st_ino == BTRFS_FIRST_FREE_OBJECTID;
>> +}
>> +
>> +/*
>>   * A not-so-good version fls64. No fascinating optimization since
>>   * no one except parse_size use it
>>   */
>> @@ -1802,24 +1821,43 @@ u64 parse_qgroupid(char *p)
>> 	char *ptr_parse_end = NULL;
>> 	u64 level;
>> 	u64 id;
>> +	int fd;
>> +	int ret = 0;
>>
>> +	if (p[0] == '/')
>> +		goto path;
>> +
>> +	/* Numeric format like '0/257' is the primary case */
>> 	if (!s) {
>> 		id = strtoull(p, &ptr_parse_end, 10);
>> 		if (ptr_parse_end != ptr_src_end)
>> -			goto err;
>> +			goto path;
>> 		return id;
>> 	}
>> 	level = strtoull(p, &ptr_parse_end, 10);
>> 	if (ptr_parse_end != s)
>> -		goto err;
>> +		goto path;
>>
>> 	id = strtoull(s+1, &ptr_parse_end, 10);
>> 	if (ptr_parse_end != ptr_src_end)
>> -		goto  err;
>> +		goto path;
>>
>> 	return (level << BTRFS_QGROUP_LEVEL_SHIFT) | id;
>> +path:
>> +	/* Path format like subv at 'my_subvol' is the fallback case */
>> +	ret = __is_subvol(p);
>> +	if (ret < 0 || !ret)
>> +		goto err;
>> +	fd = open(p, O_RDONLY);
>> +	if (fd < 0)
>> +		goto err;
>> +	ret = lookup_ino_rootid(fd, &id);
>> +	close(fd);
>> +	if (ret < 0)
>> +		goto err;
>> +	return id;
>> err:
>> -	fprintf(stderr, "ERROR:invalid qgroupid\n");
>> +	fprintf(stderr, "ERROR:invalid qgroupid or subvolume path\n");
>> 	exit(-1);
>> }
>>
>> --
>> 2.3.0
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> Best Regards,
> Wang Shilong
>

Re: [PATCH 0/7] btrfs-progs: qgroup related enhance.

[David Sterba]

On Fri, Feb 27, 2015 at 04:26:32PM +0800, Qu Wenruo wrote:
> Qu Wenruo (7):
>   btrfs-progs: Update qgroup status flags and replace qgroup
>     level/subvid     calculation with inline function.
>   btrfs-progs: Allow btrfs-debug-tree to print human readable qgroup    
>     status flag.
>   btrfs-progs: Move parse_qgroupid() to utils.c
>   btrfs-progs: Allow parse_qgroupid() to resolve subvolume path into    
>     qgroupid.
>   btrfs-progs: Add stack get/set functions for btrfs_qgroup_status_item.
>   btrfs-progs: Print warning message if qgroup data is inconsistent.

All of the above merged.

>   btrfs-progs: Schedule quota rescan if qgroup assign caused    
>     inconsistence.

The rescan can be a long running operation and touches a lot of
metadata, so I'd rather avoid starting the rescan automatically. Right
now the user may start rescan manually, or please add an commandline
option (eg. --rescan) to do the rescan in one go with the 'assign'
command. Thanks.

Re: [PATCH 0/7] btrfs-progs: qgroup related enhance.

[Qu Wenruo]

-------- Original Message  --------
Subject: Re: [PATCH 0/7] btrfs-progs: qgroup related enhance.
From: David Sterba <dsterba@suse.cz>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: 2015年03月24日 07:38

> On Fri, Feb 27, 2015 at 04:26:32PM +0800, Qu Wenruo wrote:
>> Qu Wenruo (7):
>>    btrfs-progs: Update qgroup status flags and replace qgroup
>>      level/subvid     calculation with inline function.
>>    btrfs-progs: Allow btrfs-debug-tree to print human readable qgroup
>>      status flag.
>>    btrfs-progs: Move parse_qgroupid() to utils.c
>>    btrfs-progs: Allow parse_qgroupid() to resolve subvolume path into
>>      qgroupid.
>>    btrfs-progs: Add stack get/set functions for btrfs_qgroup_status_item.
>>    btrfs-progs: Print warning message if qgroup data is inconsistent.
>
> All of the above merged.
>
>>    btrfs-progs: Schedule quota rescan if qgroup assign caused
>>      inconsistence.
>
> The rescan can be a long running operation and touches a lot of
> metadata, so I'd rather avoid starting the rescan automatically. Right
> now the user may start rescan manually, or please add an commandline
> option (eg. --rescan) to do the rescan in one go with the 'assign'
> command. Thanks.
>
Makes sense, I'll update the last patch soon.

Thanks,
Qu

Re: [PATCH 6/7] btrfs-progs: Print warning message if qgroup data is inconsistent.

[Filipe David Manana]

On Fri, Feb 27, 2015 at 8:26 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
> Before this patch, qgroup show won't check btrfs qgroup status, so even
> the INCONSISTENT flags is set, user is not aware of it.
>
> This patch will include BTRFS_QGROUP_STATUS_ITEM in the search range and
> check the flag, if there is any flag meaning the inconsistence of qgroup
> data, info user.
>
> NOTE: There is several kernel bugs from INCONSISTENT flags is always set
> to RUNNING flags is not cleared until umount.
> So this warning will always be here if using a newer kernel fixing these
> bugs.
>
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> ---
>  qgroup.c | 26 ++++++++++++++++++++++++--
>  1 file changed, 24 insertions(+), 2 deletions(-)
>
> diff --git a/qgroup.c b/qgroup.c
> index 7288365..4173846 100644
> --- a/qgroup.c
> +++ b/qgroup.c
> @@ -1016,6 +1016,20 @@ static void __filter_and_sort_qgroups(struct qgroup_lookup *all_qgroups,
>                 n = rb_prev(n);
>         }
>  }
> +
> +static inline void print_status_flag_warning(u64 flags)
> +{
> +       if (!(flags & BTRFS_QGROUP_STATUS_FLAG_ON))
> +               fprintf(stderr,
> +               "WARNING: Quota disabled, qgroup data may be out of date\n");
> +       else if (flags & BTRFS_QGROUP_STATUS_FLAG_RESCAN)
> +               fprintf(stderr,
> +               "WARNING: Rescan is running, qgroup data may be incorrect\n");

Hi Qu, did you ran xfstests? Did btrfs/022 passed for you?

btrfs/022 47s ... - output mismatch (see
/home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad)
    --- tests/btrfs/022.out 2014-11-17 20:59:51.178203000 +0000
    +++ /home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad
2015-05-30 12:35:55.917146846 +0100
    @@ -1,2 +1,3 @@
     QA output created by 022
    +WARNING: Rescan is running, qgroup data may be incorrect
     Silence is golden
    ...
    (Run 'diff -u tests/btrfs/022.out
/home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad'  to see
the entire diff)

thanks


> +       else if (flags & BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT)
> +               fprintf(stderr,
> +               "WARNING: Qgroup data inconsistent, rescan recommended\n");
> +}
> +
>  static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
>  {
>         int ret;
> @@ -1039,7 +1053,7 @@ static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
>
>         sk->tree_id = BTRFS_QUOTA_TREE_OBJECTID;
>         sk->max_type = BTRFS_QGROUP_RELATION_KEY;
> -       sk->min_type = BTRFS_QGROUP_INFO_KEY;
> +       sk->min_type = BTRFS_QGROUP_STATUS_KEY;
>         sk->max_objectid = (u64)-1;
>         sk->max_offset = (u64)-1;
>         sk->max_transid = (u64)-1;
> @@ -1070,7 +1084,15 @@ static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
>                                                                   off);
>                         off += sizeof(*sh);
>
> -                       if (sh->type == BTRFS_QGROUP_INFO_KEY) {
> +                       if (sh->type == BTRFS_QGROUP_STATUS_KEY) {
> +                               struct btrfs_qgroup_status_item *si;
> +                               u64 flags;
> +
> +                               si = (struct btrfs_qgroup_status_item *)
> +                                    (args.buf + off);
> +                               flags = btrfs_stack_qgroup_status_flags(si);
> +                               print_status_flag_warning(flags);
> +                       } else if (sh->type == BTRFS_QGROUP_INFO_KEY) {
>                                 info = (struct btrfs_qgroup_info_item *)
>                                        (args.buf + off);
>                                 a1 = btrfs_stack_qgroup_info_generation(info);
> --
> 2.3.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: [PATCH 6/7] btrfs-progs: Print warning message if qgroup data is inconsistent.

[Qu Wenruo]

-------- Original Message  --------
Subject: Re: [PATCH 6/7] btrfs-progs: Print warning message if qgroup 
data is inconsistent.
From: Filipe David Manana <fdmanana@gmail.com>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: 2015年05月30日 19:39

> On Fri, Feb 27, 2015 at 8:26 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>> Before this patch, qgroup show won't check btrfs qgroup status, so even
>> the INCONSISTENT flags is set, user is not aware of it.
>>
>> This patch will include BTRFS_QGROUP_STATUS_ITEM in the search range and
>> check the flag, if there is any flag meaning the inconsistence of qgroup
>> data, info user.
>>
>> NOTE: There is several kernel bugs from INCONSISTENT flags is always set
>> to RUNNING flags is not cleared until umount.
>> So this warning will always be here if using a newer kernel fixing these
>> bugs.
>>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> ---
>>   qgroup.c | 26 ++++++++++++++++++++++++--
>>   1 file changed, 24 insertions(+), 2 deletions(-)
>>
>> diff --git a/qgroup.c b/qgroup.c
>> index 7288365..4173846 100644
>> --- a/qgroup.c
>> +++ b/qgroup.c
>> @@ -1016,6 +1016,20 @@ static void __filter_and_sort_qgroups(struct qgroup_lookup *all_qgroups,
>>                  n = rb_prev(n);
>>          }
>>   }
>> +
>> +static inline void print_status_flag_warning(u64 flags)
>> +{
>> +       if (!(flags & BTRFS_QGROUP_STATUS_FLAG_ON))
>> +               fprintf(stderr,
>> +               "WARNING: Quota disabled, qgroup data may be out of date\n");
>> +       else if (flags & BTRFS_QGROUP_STATUS_FLAG_RESCAN)
>> +               fprintf(stderr,
>> +               "WARNING: Rescan is running, qgroup data may be incorrect\n");
>
> Hi Qu, did you ran xfstests? Did btrfs/022 passed for you?
>
> btrfs/022 47s ... - output mismatch (see
> /home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad)
>      --- tests/btrfs/022.out 2014-11-17 20:59:51.178203000 +0000
>      +++ /home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad
> 2015-05-30 12:35:55.917146846 +0100
>      @@ -1,2 +1,3 @@
>       QA output created by 022
>      +WARNING: Rescan is running, qgroup data may be incorrect
>       Silence is golden
>      ...
>      (Run 'diff -u tests/btrfs/022.out
> /home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad'  to see
> the entire diff)
I'll investigate this bug soon.

Thanks,
Qu
>
> thanks
>
>
>> +       else if (flags & BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT)
>> +               fprintf(stderr,
>> +               "WARNING: Qgroup data inconsistent, rescan recommended\n");
>> +}
>> +
>>   static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
>>   {
>>          int ret;
>> @@ -1039,7 +1053,7 @@ static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
>>
>>          sk->tree_id = BTRFS_QUOTA_TREE_OBJECTID;
>>          sk->max_type = BTRFS_QGROUP_RELATION_KEY;
>> -       sk->min_type = BTRFS_QGROUP_INFO_KEY;
>> +       sk->min_type = BTRFS_QGROUP_STATUS_KEY;
>>          sk->max_objectid = (u64)-1;
>>          sk->max_offset = (u64)-1;
>>          sk->max_transid = (u64)-1;
>> @@ -1070,7 +1084,15 @@ static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
>>                                                                    off);
>>                          off += sizeof(*sh);
>>
>> -                       if (sh->type == BTRFS_QGROUP_INFO_KEY) {
>> +                       if (sh->type == BTRFS_QGROUP_STATUS_KEY) {
>> +                               struct btrfs_qgroup_status_item *si;
>> +                               u64 flags;
>> +
>> +                               si = (struct btrfs_qgroup_status_item *)
>> +                                    (args.buf + off);
>> +                               flags = btrfs_stack_qgroup_status_flags(si);
>> +                               print_status_flag_warning(flags);
>> +                       } else if (sh->type == BTRFS_QGROUP_INFO_KEY) {
>>                                  info = (struct btrfs_qgroup_info_item *)
>>                                         (args.buf + off);
>>                                  a1 = btrfs_stack_qgroup_info_generation(info);
>> --
>> 2.3.0
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>

Re: [PATCH 6/7] btrfs-progs: Print warning message if qgroup data is inconsistent.

[Qu Wenruo]

-------- Original Message  --------
Subject: Re: [PATCH 6/7] btrfs-progs: Print warning message if qgroup 
data is inconsistent.
From: Filipe David Manana <fdmanana@gmail.com>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: 2015年05月30日 19:39

> On Fri, Feb 27, 2015 at 8:26 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>> Before this patch, qgroup show won't check btrfs qgroup status, so even
>> the INCONSISTENT flags is set, user is not aware of it.
>>
>> This patch will include BTRFS_QGROUP_STATUS_ITEM in the search range and
>> check the flag, if there is any flag meaning the inconsistence of qgroup
>> data, info user.
>>
>> NOTE: There is several kernel bugs from INCONSISTENT flags is always set
>> to RUNNING flags is not cleared until umount.
>> So this warning will always be here if using a newer kernel fixing these
>> bugs.
>>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> ---
>>   qgroup.c | 26 ++++++++++++++++++++++++--
>>   1 file changed, 24 insertions(+), 2 deletions(-)
>>
>> diff --git a/qgroup.c b/qgroup.c
>> index 7288365..4173846 100644
>> --- a/qgroup.c
>> +++ b/qgroup.c
>> @@ -1016,6 +1016,20 @@ static void __filter_and_sort_qgroups(struct qgroup_lookup *all_qgroups,
>>                  n = rb_prev(n);
>>          }
>>   }
>> +
>> +static inline void print_status_flag_warning(u64 flags)
>> +{
>> +       if (!(flags & BTRFS_QGROUP_STATUS_FLAG_ON))
>> +               fprintf(stderr,
>> +               "WARNING: Quota disabled, qgroup data may be out of date\n");
>> +       else if (flags & BTRFS_QGROUP_STATUS_FLAG_RESCAN)
>> +               fprintf(stderr,
>> +               "WARNING: Rescan is running, qgroup data may be incorrect\n");
>
> Hi Qu, did you ran xfstests? Did btrfs/022 passed for you?
>
> btrfs/022 47s ... - output mismatch (see
> /home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad)
>      --- tests/btrfs/022.out 2014-11-17 20:59:51.178203000 +0000
>      +++ /home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad
> 2015-05-30 12:35:55.917146846 +0100
>      @@ -1,2 +1,3 @@
>       QA output created by 022
>      +WARNING: Rescan is running, qgroup data may be incorrect
>       Silence is golden
>      ...
>      (Run 'diff -u tests/btrfs/022.out
> /home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad'  to see
> the entire diff)
>
> thanks
Unfortunately, it's quite hard to trigger in my environment.
I tried about 15 times, and can only trigger it once.

Any hint about mount options or other things to improve the reproducibility?

Thanks,
Qu
>
>
>> +       else if (flags & BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT)
>> +               fprintf(stderr,
>> +               "WARNING: Qgroup data inconsistent, rescan recommended\n");
>> +}
>> +
>>   static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
>>   {
>>          int ret;
>> @@ -1039,7 +1053,7 @@ static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
>>
>>          sk->tree_id = BTRFS_QUOTA_TREE_OBJECTID;
>>          sk->max_type = BTRFS_QGROUP_RELATION_KEY;
>> -       sk->min_type = BTRFS_QGROUP_INFO_KEY;
>> +       sk->min_type = BTRFS_QGROUP_STATUS_KEY;
>>          sk->max_objectid = (u64)-1;
>>          sk->max_offset = (u64)-1;
>>          sk->max_transid = (u64)-1;
>> @@ -1070,7 +1084,15 @@ static int __qgroups_search(int fd, struct qgroup_lookup *qgroup_lookup)
>>                                                                    off);
>>                          off += sizeof(*sh);
>>
>> -                       if (sh->type == BTRFS_QGROUP_INFO_KEY) {
>> +                       if (sh->type == BTRFS_QGROUP_STATUS_KEY) {
>> +                               struct btrfs_qgroup_status_item *si;
>> +                               u64 flags;
>> +
>> +                               si = (struct btrfs_qgroup_status_item *)
>> +                                    (args.buf + off);
>> +                               flags = btrfs_stack_qgroup_status_flags(si);
>> +                               print_status_flag_warning(flags);
>> +                       } else if (sh->type == BTRFS_QGROUP_INFO_KEY) {
>>                                  info = (struct btrfs_qgroup_info_item *)
>>                                         (args.buf + off);
>>                                  a1 = btrfs_stack_qgroup_info_generation(info);
>> --
>> 2.3.0
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>

Re: [PATCH 6/7] btrfs-progs: Print warning message if qgroup data is inconsistent.

[Filipe David Manana]

On Mon, Jun 1, 2015 at 2:25 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> -------- Original Message  --------
> Subject: Re: [PATCH 6/7] btrfs-progs: Print warning message if qgroup data
> is inconsistent.
> From: Filipe David Manana <fdmanana@gmail.com>
> To: Qu Wenruo <quwenruo@cn.fujitsu.com>
> Date: 2015年05月30日 19:39
>
>> On Fri, Feb 27, 2015 at 8:26 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>> wrote:
>>>
>>> Before this patch, qgroup show won't check btrfs qgroup status, so even
>>> the INCONSISTENT flags is set, user is not aware of it.
>>>
>>> This patch will include BTRFS_QGROUP_STATUS_ITEM in the search range and
>>> check the flag, if there is any flag meaning the inconsistence of qgroup
>>> data, info user.
>>>
>>> NOTE: There is several kernel bugs from INCONSISTENT flags is always set
>>> to RUNNING flags is not cleared until umount.
>>> So this warning will always be here if using a newer kernel fixing these
>>> bugs.
>>>
>>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>>> ---
>>>   qgroup.c | 26 ++++++++++++++++++++++++--
>>>   1 file changed, 24 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/qgroup.c b/qgroup.c
>>> index 7288365..4173846 100644
>>> --- a/qgroup.c
>>> +++ b/qgroup.c
>>> @@ -1016,6 +1016,20 @@ static void __filter_and_sort_qgroups(struct
>>> qgroup_lookup *all_qgroups,
>>>                  n = rb_prev(n);
>>>          }
>>>   }
>>> +
>>> +static inline void print_status_flag_warning(u64 flags)
>>> +{
>>> +       if (!(flags & BTRFS_QGROUP_STATUS_FLAG_ON))
>>> +               fprintf(stderr,
>>> +               "WARNING: Quota disabled, qgroup data may be out of
>>> date\n");
>>> +       else if (flags & BTRFS_QGROUP_STATUS_FLAG_RESCAN)
>>> +               fprintf(stderr,
>>> +               "WARNING: Rescan is running, qgroup data may be
>>> incorrect\n");
>>
>>
>> Hi Qu, did you ran xfstests? Did btrfs/022 passed for you?
>>
>> btrfs/022 47s ... - output mismatch (see
>> /home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad)
>>      --- tests/btrfs/022.out 2014-11-17 20:59:51.178203000 +0000
>>      +++ /home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad
>> 2015-05-30 12:35:55.917146846 +0100
>>      @@ -1,2 +1,3 @@
>>       QA output created by 022
>>      +WARNING: Rescan is running, qgroup data may be incorrect
>>       Silence is golden
>>      ...
>>      (Run 'diff -u tests/btrfs/022.out
>> /home/fdmanana/git/hub/xfstests/results//btrfs/022.out.bad'  to see
>> the entire diff)
>>
>> thanks
>
> Unfortunately, it's quite hard to trigger in my environment.
> I tried about 15 times, and can only trigger it once.
>
> Any hint about mount options or other things to improve the reproducibility?

No mount options at all (nor mkfs -O features). Happens all the time here.

Thanks for looking into it.

>
> Thanks,
> Qu
>
>>
>>
>>> +       else if (flags & BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT)
>>> +               fprintf(stderr,
>>> +               "WARNING: Qgroup data inconsistent, rescan
>>> recommended\n");
>>> +}
>>> +
>>>   static int __qgroups_search(int fd, struct qgroup_lookup
>>> *qgroup_lookup)
>>>   {
>>>          int ret;
>>> @@ -1039,7 +1053,7 @@ static int __qgroups_search(int fd, struct
>>> qgroup_lookup *qgroup_lookup)
>>>
>>>          sk->tree_id = BTRFS_QUOTA_TREE_OBJECTID;
>>>          sk->max_type = BTRFS_QGROUP_RELATION_KEY;
>>> -       sk->min_type = BTRFS_QGROUP_INFO_KEY;
>>> +       sk->min_type = BTRFS_QGROUP_STATUS_KEY;
>>>          sk->max_objectid = (u64)-1;
>>>          sk->max_offset = (u64)-1;
>>>          sk->max_transid = (u64)-1;
>>> @@ -1070,7 +1084,15 @@ static int __qgroups_search(int fd, struct
>>> qgroup_lookup *qgroup_lookup)
>>>                                                                    off);
>>>                          off += sizeof(*sh);
>>>
>>> -                       if (sh->type == BTRFS_QGROUP_INFO_KEY) {
>>> +                       if (sh->type == BTRFS_QGROUP_STATUS_KEY) {
>>> +                               struct btrfs_qgroup_status_item *si;
>>> +                               u64 flags;
>>> +
>>> +                               si = (struct btrfs_qgroup_status_item *)
>>> +                                    (args.buf + off);
>>> +                               flags =
>>> btrfs_stack_qgroup_status_flags(si);
>>> +                               print_status_flag_warning(flags);
>>> +                       } else if (sh->type == BTRFS_QGROUP_INFO_KEY) {
>>>                                  info = (struct btrfs_qgroup_info_item *)
>>>                                         (args.buf + off);
>>>                                  a1 =
>>> btrfs_stack_qgroup_info_generation(info);
>>> --
>>> 2.3.0
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>>
>



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

[PATCH] xfstests: btrfs: 022: add a quota rescan -w to wait rescan finished.

[Dongsheng Yang]

When we enable quota, btrfs will rescan quota numbers. We need
to wait the rescan finished before any more operations on btrfs
qgroups. Otherwith, the new btrfs-progs would WARN out:

WARNING: Rescan is running, qgroup data may be incorrect.

It would make btrfs/022 failed.

Signed-off-by: Dongsheng Yang <yangds.fnst@cn.fujitsu.com>
---
 tests/btrfs/022 | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/btrfs/022 b/tests/btrfs/022
index 5c1a82d..56d4f3d 100755
--- a/tests/btrfs/022
+++ b/tests/btrfs/022
@@ -51,6 +51,7 @@ _basic_test()
 {
 	_run_btrfs_util_prog subvolume create $SCRATCH_MNT/a
 	_run_btrfs_util_prog quota enable $SCRATCH_MNT/a
+	_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
 	subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT a)
 	$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep $subvolid >> \
 		$seqres.full 2>&1
-- 
1.8.4.2

[PATCH] xfstests: btrfs: 022: add a quota rescan -w to wait rescan finished.

[Dongsheng Yang]

When we enable quota, btrfs will rescan quota numbers. We need
to wait the rescan finished before any more operations on btrfs
qgroups. Otherwith, the new btrfs-progs would WARN out:

WARNING: Rescan is running, qgroup data may be incorrect.

It would make btrfs/022 failed.

Signed-off-by: Dongsheng Yang <yangds.fnst@cn.fujitsu.com>
---
 tests/btrfs/022 | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/btrfs/022 b/tests/btrfs/022
index 5c1a82d..56d4f3d 100755
--- a/tests/btrfs/022
+++ b/tests/btrfs/022
@@ -51,6 +51,7 @@ _basic_test()
 {
 	_run_btrfs_util_prog subvolume create $SCRATCH_MNT/a
 	_run_btrfs_util_prog quota enable $SCRATCH_MNT/a
+	_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
 	subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT a)
 	$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep $subvolid >> \
 		$seqres.full 2>&1
-- 
1.8.4.2

Re: [PATCH] xfstests: btrfs: 022: add a quota rescan -w to wait rescan finished.

[Dongsheng Yang]

Hi Filip,

Qu is off duty this week, would you please try this patch here?

Thanx
Yang

On 06/03/2015 03:10 PM, Dongsheng Yang wrote:
> When we enable quota, btrfs will rescan quota numbers. We need
> to wait the rescan finished before any more operations on btrfs
> qgroups. Otherwith, the new btrfs-progs would WARN out:
>
> WARNING: Rescan is running, qgroup data may be incorrect.
>
> It would make btrfs/022 failed.
>
> Signed-off-by: Dongsheng Yang <yangds.fnst@cn.fujitsu.com>
> ---
>   tests/btrfs/022 | 1 +
>   1 file changed, 1 insertion(+)
>
> diff --git a/tests/btrfs/022 b/tests/btrfs/022
> index 5c1a82d..56d4f3d 100755
> --- a/tests/btrfs/022
> +++ b/tests/btrfs/022
> @@ -51,6 +51,7 @@ _basic_test()
>   {
>   	_run_btrfs_util_prog subvolume create $SCRATCH_MNT/a
>   	_run_btrfs_util_prog quota enable $SCRATCH_MNT/a
> +	_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
>   	subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT a)
>   	$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep $subvolid >> \
>   		$seqres.full 2>&1
>

Re: [PATCH] xfstests: btrfs: 022: add a quota rescan -w to wait rescan finished.

[Dongsheng Yang]

Hi Filip,

Qu is off duty this week, would you please try this patch here?

Thanx
Yang

On 06/03/2015 03:10 PM, Dongsheng Yang wrote:
> When we enable quota, btrfs will rescan quota numbers. We need
> to wait the rescan finished before any more operations on btrfs
> qgroups. Otherwith, the new btrfs-progs would WARN out:
>
> WARNING: Rescan is running, qgroup data may be incorrect.
>
> It would make btrfs/022 failed.
>
> Signed-off-by: Dongsheng Yang <yangds.fnst@cn.fujitsu.com>
> ---
>   tests/btrfs/022 | 1 +
>   1 file changed, 1 insertion(+)
>
> diff --git a/tests/btrfs/022 b/tests/btrfs/022
> index 5c1a82d..56d4f3d 100755
> --- a/tests/btrfs/022
> +++ b/tests/btrfs/022
> @@ -51,6 +51,7 @@ _basic_test()
>   {
>   	_run_btrfs_util_prog subvolume create $SCRATCH_MNT/a
>   	_run_btrfs_util_prog quota enable $SCRATCH_MNT/a
> +	_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
>   	subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT a)
>   	$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep $subvolid >> \
>   		$seqres.full 2>&1
>

Re: [PATCH] xfstests: btrfs: 022: add a quota rescan -w to wait rescan finished.

[Filipe David Manana]

On Wed, Jun 3, 2015 at 8:10 AM, Dongsheng Yang
<yangds.fnst@cn.fujitsu.com> wrote:
> When we enable quota, btrfs will rescan quota numbers. We need
> to wait the rescan finished before any more operations on btrfs
> qgroups. Otherwith, the new btrfs-progs would WARN out:
>
> WARNING: Rescan is running, qgroup data may be incorrect.
>
> It would make btrfs/022 failed.
>
> Signed-off-by: Dongsheng Yang <yangds.fnst@cn.fujitsu.com>
Reviewed-by: Filipe Manana <fdmanana@suse.com>

Thanks, it works and it makes sense.

> ---
>  tests/btrfs/022 | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/tests/btrfs/022 b/tests/btrfs/022
> index 5c1a82d..56d4f3d 100755
> --- a/tests/btrfs/022
> +++ b/tests/btrfs/022
> @@ -51,6 +51,7 @@ _basic_test()
>  {
>         _run_btrfs_util_prog subvolume create $SCRATCH_MNT/a
>         _run_btrfs_util_prog quota enable $SCRATCH_MNT/a
> +       _run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
>         subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT a)
>         $BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep $subvolid >> \
>                 $seqres.full 2>&1
> --
> 1.8.4.2
>



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: [PATCH 0/7] btrfs-progs: qgroup related enhance.

[David Sterba]

On Tue, Mar 24, 2015 at 08:36:33AM +0800, Qu Wenruo wrote:
> > All of the above merged.
> >
> >>    btrfs-progs: Schedule quota rescan if qgroup assign caused
> >>      inconsistence.
> >
> > The rescan can be a long running operation and touches a lot of
> > metadata, so I'd rather avoid starting the rescan automatically. Right
> > now the user may start rescan manually, or please add an commandline
> > option (eg. --rescan) to do the rescan in one go with the 'assign'
> > command. Thanks.
> >
> Makes sense, I'll update the last patch soon.

I've merged your patch without changes and implemented the --rescan
option myself.

Re: [PATCH 0/7] btrfs-progs: qgroup related enhance.

[Qu Wenruo]

David Sterba wrote on 2015/07/27 16:35 +0200:
> On Tue, Mar 24, 2015 at 08:36:33AM +0800, Qu Wenruo wrote:
>>> All of the above merged.
>>>
>>>>     btrfs-progs: Schedule quota rescan if qgroup assign caused
>>>>       inconsistence.
>>>
>>> The rescan can be a long running operation and touches a lot of
>>> metadata, so I'd rather avoid starting the rescan automatically. Right
>>> now the user may start rescan manually, or please add an commandline
>>> option (eg. --rescan) to do the rescan in one go with the 'assign'
>>> command. Thanks.
>>>
>> Makes sense, I'll update the last patch soon.
>
> I've merged your patch without changes and implemented the --rescan
> option myself.
>
Oh, I forgot to update this one...

Thanks a lot for implement --rescan option.

Thanks,
Qu