[PATCH 0/8] mkfs: Quota support

[PATCH 0/8] mkfs: Quota support

[Qu Wenruo]

This patchset adds quota support, which means the result fs will have
quota enabled by default, and its accounting is already consistent, no
manually rescan or quota enable is needed.

The overall design of such support is:
1) Create needed tree
   Both btrfs_root and real root item and tree root leaf.
   For this, a new infrastructure, btrfs_create_tree(), is added for
   this.

2) Fill quota root with basic skeleton
   Only 3 items are really needed
   a) global quota status item
   b) quota info for 0/5 qgroup
   c) quota limit for 0/5 qgroup

   The content doesn't matter at all.

3) Repair qgroups using infrastructure from qgroup-verify
   In fact, qgroup repair is just offline rescan.
   Although the original qgroup-verify infrastructure is mostly noisy,
   modify it a little to make it silent to function as offline quota
   rescan.

And such support is mainly designed for developers and QA guys.

As to enable quota, we must normally mount the fs, enable quota (and
rescan if needed).
This ioctl based procedure is not common, and fstests doesn't provide
such support.

There are several attempts to make fstests to support it, but due to
different reasons, all these attempts failed.

To make it easier to test all existing test cases with btrfs quota
enabled, the current best method is to support quota at mkfs time, and
here comes the patchset.

Qu Wenruo (8):
  btrfs-progs: qgroup-verify: Also repair qgroup status version
  btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we
    should repair qgroups
  btrfs-progs: qgroup-verify: Move qgroup classification out of
    report_qgroups
  btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent
    repair
  btrfs-progs: ctree: Introduce function to create an empty tree
  btrfs-progs: mkfs: Introduce function to insert qgroup info and limit
    items
  btrfs-progs: mkfs: Introduce option to enable quota at mkfs time
  btrfs-progs: test/mkfs: Add test case for --enable-quota option

 Documentation/mkfs.btrfs.asciidoc           |   3 +
 cmds-check.c                                |   2 +-
 ctree.c                                     | 109 ++++++++++++++++++++++
 ctree.h                                     |   3 +
 mkfs/main.c                                 | 136 +++++++++++++++++++++++++++-
 qgroup-verify.c                             |  51 ++++++++---
 qgroup-verify.h                             |   2 +-
 tests/mkfs-tests/001-basic-profiles/test.sh |  10 ++
 8 files changed, 302 insertions(+), 14 deletions(-)

-- 
2.14.3

[PATCH 1/8] btrfs-progs: qgroup-verify: Also repair qgroup status version

[Qu Wenruo]

Current kernel only supports qgroup version 1.
Make qgroup-verify to follow this standard.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 ctree.h         | 1 +
 qgroup-verify.c | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/ctree.h b/ctree.h
index 22806599a744..506b76766579 100644
--- a/ctree.h
+++ b/ctree.h
@@ -1011,6 +1011,7 @@ struct btrfs_qgroup_status_item {
 	__le64 rescan;		/* progress during scanning */
 } __attribute__ ((__packed__));
 
+#define BTRFS_QGROUP_STATUS_VERSION		1
 struct btrfs_block_group_item {
 	__le64 used;
 	__le64 chunk_objectid;
diff --git a/qgroup-verify.c b/qgroup-verify.c
index 571b4d4f7171..58c735788977 100644
--- a/qgroup-verify.c
+++ b/qgroup-verify.c
@@ -1563,6 +1563,8 @@ static int repair_qgroup_status(struct btrfs_fs_info *info)
 	btrfs_set_qgroup_status_rescan(path.nodes[0], status_item, 0);
 	btrfs_set_qgroup_status_generation(path.nodes[0], status_item,
 					   trans->transid);
+	btrfs_set_qgroup_status_version(path.nodes[0], status_item,
+					BTRFS_QGROUP_STATUS_VERSION);
 
 	btrfs_mark_buffer_dirty(path.nodes[0]);
 
-- 
2.14.3

[PATCH 2/8] btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we should repair qgroups

[Qu Wenruo]

In fact qgroup-verify is just kind of offline qgroup rescan, and later
mkfs qgroup support will reuse it.

So qgroup-verify doesn't really need to rely the global variable @repair
to check if it should repair qgroups.

Instead check fs_info->readonly to do the repair.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 qgroup-verify.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/qgroup-verify.c b/qgroup-verify.c
index 58c735788977..f509cdb9317a 100644
--- a/qgroup-verify.c
+++ b/qgroup-verify.c
@@ -1582,7 +1582,7 @@ int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
 
 	*repaired = 0;
 
-	if (!repair)
+	if (info->readonly)
 		return 0;
 
 	list_for_each_entry_safe(count, tmpcount, &bad_qgroups, bad_list) {
-- 
2.14.3

[PATCH 3/8] btrfs-progs: qgroup-verify: Move qgroup classification out of report_qgroups

[Qu Wenruo]

The original qgroup-verify integrates qgroup classification into
report_qgroups().
This behavior makes silent qgroup repair (or offline rescan) impossible.

To repair qgroup, we must call report_qgroups() to trigger bad qgroup
classification, which will output error message.

This patch moves bad qgroup classification from report_qgroups() to
qgroup_verify_all().
The functionality is not changed, but provides the basis for later
silent qgroup repair.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 qgroup-verify.c | 28 ++++++++++++++++++++++++++--
 1 file changed, 26 insertions(+), 2 deletions(-)

diff --git a/qgroup-verify.c b/qgroup-verify.c
index f509cdb9317a..67bd8b96ddd3 100644
--- a/qgroup-verify.c
+++ b/qgroup-verify.c
@@ -1319,8 +1319,7 @@ void report_qgroups(int all)
 	while (node) {
 		c = rb_entry(node, struct qgroup_count, rb_node);
 
-		if (report_qgroup_difference(c, all))
-			list_add_tail(&c->bad_list, &bad_qgroups);
+		report_qgroup_difference(c, all);
 
 		node = rb_next(node);
 	}
@@ -1358,9 +1357,20 @@ void free_qgroup_counts(void)
 	}
 }
 
+static bool is_bad_qgroup(struct qgroup_count *count)
+{
+	struct qgroup_info *info = &count->info;
+	struct qgroup_info *disk = &count->diskinfo;
+	s64 excl_diff = info->exclusive - disk->exclusive;
+	s64 ref_diff = info->referenced - disk->referenced;
+
+	return (excl_diff || ref_diff);
+}
+
 int qgroup_verify_all(struct btrfs_fs_info *info)
 {
 	int ret;
+	struct rb_node *node;
 
 	if (!info->quota_enabled)
 		return 0;
@@ -1395,6 +1405,20 @@ int qgroup_verify_all(struct btrfs_fs_info *info)
 
 	ret = account_all_refs(1, 0);
 
+	/*
+	 * Do the correctness check here, so for callers who don't want
+	 * verbose report can skip calling report_qgroups()
+	 */
+	node = rb_first(&counts.root);
+	while (node) {
+		struct qgroup_count *c;
+
+		c = rb_entry(node, struct qgroup_count, rb_node);
+		if (is_bad_qgroup(c))
+			list_add_tail(&c->bad_list, &bad_qgroups);
+		node = rb_next(node);
+	}
+
 out:
 	/*
 	 * Don't free the qgroup count records as they will be walked
-- 
2.14.3

[PATCH 4/8] btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent repair

[Qu Wenruo]

Allow repair_qgroups() to do silent repair, so it can acts as offline
qgroup rescan.

This provides the basis for later mkfs quota support.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 cmds-check.c    |  2 +-
 qgroup-verify.c | 19 +++++++++++--------
 qgroup-verify.h |  2 +-
 3 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index 5c822b848608..955049b95f3c 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -13360,7 +13360,7 @@ int cmd_check(int argc, char **argv)
 			goto out;
 		}
 		report_qgroups(0);
-		ret = repair_qgroups(info, &qgroups_repaired);
+		ret = repair_qgroups(info, &qgroups_repaired, false);
 		err |= !!ret;
 		if (err) {
 			error("failed to repair quota groups");
diff --git a/qgroup-verify.c b/qgroup-verify.c
index 67bd8b96ddd3..bb77eac139aa 100644
--- a/qgroup-verify.c
+++ b/qgroup-verify.c
@@ -1498,7 +1498,7 @@ out:
 }
 
 static int repair_qgroup_info(struct btrfs_fs_info *info,
-			      struct qgroup_count *count)
+			      struct qgroup_count *count, bool silent)
 {
 	int ret;
 	struct btrfs_root *root = info->quota_root;
@@ -1507,8 +1507,10 @@ static int repair_qgroup_info(struct btrfs_fs_info *info,
 	struct btrfs_qgroup_info_item *info_item;
 	struct btrfs_key key;
 
-	printf("Repair qgroup %llu/%llu\n", btrfs_qgroup_level(count->qgroupid),
-	       btrfs_qgroup_subvid(count->qgroupid));
+	if (!silent)
+		printf("Repair qgroup %llu/%llu\n",
+			btrfs_qgroup_level(count->qgroupid),
+			btrfs_qgroup_subvid(count->qgroupid));
 
 	trans = btrfs_start_transaction(root, 1);
 	if (IS_ERR(trans))
@@ -1553,7 +1555,7 @@ out:
 	return ret;
 }
 
-static int repair_qgroup_status(struct btrfs_fs_info *info)
+static int repair_qgroup_status(struct btrfs_fs_info *info, bool silent)
 {
 	int ret;
 	struct btrfs_root *root = info->quota_root;
@@ -1562,7 +1564,8 @@ static int repair_qgroup_status(struct btrfs_fs_info *info)
 	struct btrfs_key key;
 	struct btrfs_qgroup_status_item *status_item;
 
-	printf("Repair qgroup status item\n");
+	if (!silent)
+		printf("Repair qgroup status item\n");
 
 	trans = btrfs_start_transaction(root, 1);
 	if (IS_ERR(trans))
@@ -1599,7 +1602,7 @@ out:
 	return ret;
 }
 
-int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
+int repair_qgroups(struct btrfs_fs_info *info, int *repaired, bool silent)
 {
 	int ret = 0;
 	struct qgroup_count *count, *tmpcount;
@@ -1610,7 +1613,7 @@ int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
 		return 0;
 
 	list_for_each_entry_safe(count, tmpcount, &bad_qgroups, bad_list) {
-		ret = repair_qgroup_info(info, count);
+		ret = repair_qgroup_info(info, count, silent);
 		if (ret) {
 			goto out;
 		}
@@ -1626,7 +1629,7 @@ int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
 	 * mount.
 	 */
 	if (*repaired || counts.qgroup_inconsist || counts.rescan_running) {
-		ret = repair_qgroup_status(info);
+		ret = repair_qgroup_status(info, silent);
 		if (ret)
 			goto out;
 
diff --git a/qgroup-verify.h b/qgroup-verify.h
index d7d83a46ed5a..7e5697024f31 100644
--- a/qgroup-verify.h
+++ b/qgroup-verify.h
@@ -24,7 +24,7 @@
 
 int qgroup_verify_all(struct btrfs_fs_info *info);
 void report_qgroups(int all);
-int repair_qgroups(struct btrfs_fs_info *info, int *repaired);
+int repair_qgroups(struct btrfs_fs_info *info, int *repaired, bool silent);
 
 int print_extent_state(struct btrfs_fs_info *info, u64 subvol);
 
-- 
2.14.3

[PATCH 5/8] btrfs-progs: ctree: Introduce function to create an empty tree

[Qu Wenruo]

Introduce a new function, btrfs_create_tree(), to create an empty tree.

Currently, there is only one caller to create new tree, namely
data reloc tree in mkfs.
However it's copying fs tree to create a new root.

This copy fs tree method is not a good idea if we only need an empty
tree.

So here introduce a new function, btrfs_create_tree() to create new
tree.
Which will handle the following things:
1) New tree root leaf
   Using generic tree allocation

2) New root item in tree root

3) Modify special tree root pointers in fs_info
   Only quota_root is supported yet, but can be expended easily

This patch provides the basis to implement quota support in mkfs.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 ctree.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 ctree.h |   2 ++
 2 files changed, 111 insertions(+)

diff --git a/ctree.c b/ctree.c
index 4fc33b14000a..c707be58c413 100644
--- a/ctree.c
+++ b/ctree.c
@@ -22,6 +22,7 @@
 #include "repair.h"
 #include "internal.h"
 #include "sizes.h"
+#include "utils.h"
 
 static int split_node(struct btrfs_trans_handle *trans, struct btrfs_root
 		      *root, struct btrfs_path *path, int level);
@@ -136,6 +137,114 @@ int btrfs_copy_root(struct btrfs_trans_handle *trans,
 	return 0;
 }
 
+/*
+ * Create a new tree root, with root objectid set to @objectid.
+ *
+ * NOTE: Doesn't support tree with non-zero offset, like tree reloc tree.
+ */
+int btrfs_create_root(struct btrfs_trans_handle *trans,
+		      struct btrfs_fs_info *fs_info, u64 objectid)
+{
+	struct extent_buffer *node;
+	struct btrfs_root *new_root;
+	struct btrfs_disk_key disk_key;
+	struct btrfs_key location;
+	struct btrfs_root_item root_item = { 0 };
+	int ret;
+
+	new_root = malloc(sizeof(*new_root));
+	if (!new_root)
+		return -ENOMEM;
+
+	btrfs_setup_root(new_root, fs_info, objectid);
+	if (!is_fstree(objectid))
+		new_root->track_dirty = 1;
+	add_root_to_dirty_list(new_root);
+
+	new_root->objectid = objectid;
+	new_root->root_key.objectid = objectid;
+	new_root->root_key.type = BTRFS_ROOT_ITEM_KEY;
+	new_root->root_key.offset = 0;
+
+	node = btrfs_alloc_free_block(trans, new_root, fs_info->nodesize,
+				      objectid, &disk_key, 0, 0, 0);
+	if (IS_ERR(node)) {
+		ret = PTR_ERR(node);
+		error("failed to create root node for tree %llu: %d (%s)",
+		      objectid, ret, strerror(-ret));
+		return ret;
+	}
+	new_root->node = node;
+
+	btrfs_set_header_generation(node, trans->transid);
+	btrfs_set_header_backref_rev(node, BTRFS_MIXED_BACKREF_REV);
+	btrfs_clear_header_flag(node, BTRFS_HEADER_FLAG_RELOC |
+				      BTRFS_HEADER_FLAG_WRITTEN);
+	btrfs_set_header_owner(node, objectid);
+	btrfs_set_header_nritems(node, 0);
+	btrfs_set_header_level(node, 0);
+	write_extent_buffer(node, fs_info->fsid, btrfs_header_fsid(),
+			    BTRFS_FSID_SIZE);
+	ret = btrfs_inc_ref(trans, new_root, node, 0);
+	if (ret < 0)
+		goto free;
+
+	/*
+	 * Special tree roots may need to modify pointers in @fs_info
+	 * Only quota is supported yet.
+	 */
+	switch (objectid) {
+	case BTRFS_QUOTA_TREE_OBJECTID:
+		if (fs_info->quota_root) {
+			error("quota root already exists");
+			ret = -EEXIST;
+			goto free;
+		}
+		fs_info->quota_root = new_root;
+		fs_info->quota_enabled = 1;
+		break;
+	/*
+	 * Essential trees can't be created by this function, yet.
+	 * As we expect such skeleton exists, or a lot of functions like
+	 * btrfs_alloc_free_block() doesn't work at all
+	 */
+	case BTRFS_ROOT_TREE_OBJECTID:
+	case BTRFS_EXTENT_TREE_OBJECTID:
+	case BTRFS_CHUNK_TREE_OBJECTID:
+	case BTRFS_FS_TREE_OBJECTID:
+		ret = -EEXIST;
+		goto free;
+	default:
+		/* Subvolume trees don't need special handles */
+		if (is_fstree(objectid))
+			break;
+		/* Other special trees are not supported yet */
+		ret = -ENOTTY;
+		goto free;
+	}
+	btrfs_mark_buffer_dirty(node);
+	btrfs_set_root_bytenr(&root_item, btrfs_header_bytenr(node));
+	btrfs_set_root_level(&root_item, 0);
+	btrfs_set_root_generation(&root_item, trans->transid);
+	btrfs_set_root_dirid(&root_item, 0);
+	btrfs_set_root_refs(&root_item, 1);
+	btrfs_set_root_used(&root_item, fs_info->nodesize);
+	location.objectid = objectid;
+	location.type = BTRFS_ROOT_ITEM_KEY;
+	location.offset = 0;
+
+	ret = btrfs_insert_root(trans, fs_info->tree_root, &location,
+				&root_item);
+	if (ret < 0)
+		goto free;
+	return ret;
+
+free:
+	free_extent_buffer(node);
+	free(new_root);
+	return ret;
+}
+
 /*
  * check if the tree block can be shared by multiple trees
  */
diff --git a/ctree.h b/ctree.h
index 506b76766579..ac8ce70a55e7 100644
--- a/ctree.h
+++ b/ctree.h
@@ -2558,6 +2558,8 @@ int btrfs_copy_root(struct btrfs_trans_handle *trans,
 		      struct btrfs_root *root,
 		      struct extent_buffer *buf,
 		      struct extent_buffer **cow_ret, u64 new_root_objectid);
+int btrfs_create_root(struct btrfs_trans_handle *trans,
+		      struct btrfs_fs_info *fs_info, u64 objectid);
 int btrfs_extend_item(struct btrfs_root *root, struct btrfs_path *path,
 		u32 data_size);
 int btrfs_truncate_item(struct btrfs_root *root, struct btrfs_path *path,
-- 
2.14.3

[PATCH 6/8] btrfs-progs: mkfs: Introduce function to insert qgroup info and limit items

[Qu Wenruo]

Introduce a new function, insert_qgroup_items(), to insert qgroup info
item and qgroup limit item for later mkfs qgroup support.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 mkfs/main.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/mkfs/main.c b/mkfs/main.c
index 1d72702414bc..c50e7e74daf1 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -702,6 +702,40 @@ static void update_chunk_allocation(struct btrfs_fs_info *fs_info,
 	}
 }
 
+static int insert_qgroup_items(struct btrfs_trans_handle *trans,
+			       struct btrfs_fs_info *fs_info,
+			       u64 qgroupid)
+{
+	struct btrfs_path path;
+	struct btrfs_root *quota_root = fs_info->quota_root;
+	struct btrfs_key key;
+	int ret;
+
+	if (qgroupid >> BTRFS_QGROUP_LEVEL_SHIFT) {
+		error("qgroup level other than 0 is not supported yet");
+		return -ENOTTY;
+	}
+
+	key.objectid = 0;
+	key.type = BTRFS_QGROUP_INFO_KEY;
+	key.offset = qgroupid;
+
+	btrfs_init_path(&path);
+	ret = btrfs_insert_empty_item(trans, quota_root, &path, &key,
+				      sizeof(struct btrfs_qgroup_info_item));
+	btrfs_release_path(&path);
+	if (ret < 0)
+		return ret;
+
+	key.objectid = 0;
+	key.type = BTRFS_QGROUP_LIMIT_KEY;
+	key.offset = qgroupid;
+	ret = btrfs_insert_empty_item(trans, quota_root, &path, &key,
+				      sizeof(struct btrfs_qgroup_limit_item));
+	btrfs_release_path(&path);
+	return ret;
+}
+
 int main(int argc, char **argv)
 {
 	char *file;
-- 
2.14.3

[PATCH 7/8] btrfs-progs: mkfs: Introduce option to enable quota at mkfs time

[Qu Wenruo]

Introduce new parameter, --enable-quota, to enable quota at mkfs time.

The result fs will has quota enabled, with consistent qgroup accounting.

This is quite handy to test quota with fstests, which doesn't support to
call ioctl for btrfs at mount time.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 Documentation/mkfs.btrfs.asciidoc |   3 ++
 mkfs/main.c                       | 102 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 104 insertions(+), 1 deletion(-)

diff --git a/Documentation/mkfs.btrfs.asciidoc b/Documentation/mkfs.btrfs.asciidoc
index 5ddbedbcea97..78ddd5a7017a 100644
--- a/Documentation/mkfs.btrfs.asciidoc
+++ b/Documentation/mkfs.btrfs.asciidoc
@@ -140,6 +140,9 @@ Print only error or warning messages. Options --features or --help are unaffecte
 Create the filesystem with the given 'UUID'. The UUID must not exist on any
 filesystem currently present.
 
+*--enable-quota*::
+Create the filesystem with quota enabled. See `btrfs quota`(8) for more info.
+
 *-V|--version*::
 Print the *mkfs.btrfs* version and exit.
 
diff --git a/mkfs/main.c b/mkfs/main.c
index c50e7e74daf1..8d93a1613053 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -31,6 +31,7 @@
 #include <uuid/uuid.h>
 #include <ctype.h>
 #include <blkid/blkid.h>
+#include <string.h>
 #include "ctree.h"
 #include "disk-io.h"
 #include "volumes.h"
@@ -41,6 +42,7 @@
 #include "mkfs/common.h"
 #include "mkfs/rootdir.h"
 #include "fsfeatures.h"
+#include "qgroup-verify.h"
 
 static int verbose = 1;
 
@@ -349,6 +351,7 @@ static void print_usage(int ret)
 	printf("\t-O|--features LIST      comma separated list of filesystem features (use '-O list-all' to list features)\n");
 	printf("\t-L|--label LABEL        set the filesystem label\n");
 	printf("\t-U|--uuid UUID          specify the filesystem UUID (must be unique)\n");
+	printf("\t--enable-quota          enable btrfs quota feature\n");
 	printf("  creation:\n");
 	printf("\t-b|--byte-count SIZE    set filesystem size to SIZE (on the first device)\n");
 	printf("\t-r|--rootdir DIR        copy files from DIR to the image root directory\n");
@@ -736,6 +739,91 @@ static int insert_qgroup_items(struct btrfs_trans_handle *trans,
 	return ret;
 }
 
+static int setup_quota_root(struct btrfs_fs_info *fs_info)
+{
+	struct btrfs_trans_handle *trans;
+	struct btrfs_qgroup_status_item *qsi;
+	struct btrfs_root *quota_root;
+	struct btrfs_path path;
+	struct btrfs_key key;
+	int qgroup_repaired = 0;
+	int ret;
+
+	/* One to modify tree root, one for quota root */
+	trans = btrfs_start_transaction(fs_info->tree_root, 2);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		error("failed to start transaction: %d (%s)",
+			ret, strerror(-ret));
+		return ret;
+	}
+	ret = btrfs_create_root(trans, fs_info, BTRFS_QUOTA_TREE_OBJECTID);
+	if (ret < 0) {
+		error("failed to create quota root: %d (%s)",
+			ret, strerror(-ret));
+		goto fail;
+	}
+	quota_root = fs_info->quota_root;
+
+	key.objectid = 0;
+	key.type = BTRFS_QGROUP_STATUS_KEY;
+	key.offset = 0;
+
+	btrfs_init_path(&path);
+	ret = btrfs_insert_empty_item(trans, quota_root, &path, &key,
+				      sizeof(*qsi));
+	if (ret < 0) {
+		error("failed to insert qgroup status item: %d (%s)",
+			ret, strerror(-ret));
+		goto fail;
+	}
+
+	qsi = btrfs_item_ptr(path.nodes[0], path.slots[0],
+			     struct btrfs_qgroup_status_item);
+	btrfs_set_qgroup_status_generation(path.nodes[0], qsi, 0);
+	btrfs_set_qgroup_status_rescan(path.nodes[0], qsi, 0);
+
+	/* Mark current status info inconsistent, and fix it later */
+	btrfs_set_qgroup_status_flags(path.nodes[0], qsi,
+			BTRFS_QGROUP_STATUS_FLAG_ON |
+			BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT);
+	btrfs_release_path(&path);
+
+	/* Currently mkfs will only create one subvolume */
+	ret = insert_qgroup_items(trans, fs_info, BTRFS_FS_TREE_OBJECTID);
+	if (ret < 0) {
+		error("failed to insert qgroup items: %d (%s)",
+			ret, strerror(-ret));
+		goto fail;
+	}
+
+	ret = btrfs_commit_transaction(trans, fs_info->tree_root);
+	if (ret < 0) {
+		error("failed to commit current transaction: %d (%s)",
+			ret, strerror(-ret));
+		return ret;
+	}
+
+	/*
+	 * Qgroup is setup but with wrong info, use qgroup-verify
+	 * infrastructure to repair them.
+	 * (Just acts as offline rescan)
+	 */
+	ret = qgroup_verify_all(fs_info);
+	if (ret < 0) {
+		error("qgroup rescan failed: %d (%s)", ret, strerror(-ret));
+		return ret;
+	}
+	ret = repair_qgroups(fs_info, &qgroup_repaired, true);
+	if (ret < 0)
+		error("failed to fill qgroup info: %d (%s)", ret,
+			strerror(-ret));
+	return ret;
+fail:
+	btrfs_commit_transaction(trans, fs_info->tree_root);
+	return ret;
+}
+
 int main(int argc, char **argv)
 {
 	char *file;
@@ -767,6 +855,7 @@ int main(int argc, char **argv)
 	char *source_dir = NULL;
 	bool source_dir_set = false;
 	bool shrink_rootdir = false;
+	bool enable_quota = false;
 	u64 source_dir_size = 0;
 	u64 min_dev_size;
 	u64 shrink_size;
@@ -779,7 +868,7 @@ int main(int argc, char **argv)
 
 	while(1) {
 		int c;
-		enum { GETOPT_VAL_SHRINK = 257 };
+		enum { GETOPT_VAL_SHRINK = 257, GETOPT_VAL_QUOTA };
 		static const struct option long_options[] = {
 			{ "alloc-start", required_argument, NULL, 'A'},
 			{ "byte-count", required_argument, NULL, 'b' },
@@ -798,6 +887,7 @@ int main(int argc, char **argv)
 			{ "uuid", required_argument, NULL, 'U' },
 			{ "quiet", 0, NULL, 'q' },
 			{ "shrink", no_argument, NULL, GETOPT_VAL_SHRINK },
+			{ "enable-quota", no_argument, NULL, GETOPT_VAL_QUOTA },
 			{ "help", no_argument, NULL, GETOPT_VAL_HELP },
 			{ NULL, 0, NULL, 0}
 		};
@@ -880,6 +970,8 @@ int main(int argc, char **argv)
 			case GETOPT_VAL_SHRINK:
 				shrink_rootdir = true;
 				break;
+			case GETOPT_VAL_QUOTA:
+				enable_quota = true;
 				break;
 			case GETOPT_VAL_HELP:
 			default:
@@ -1242,6 +1334,14 @@ raid_groups:
 		}
 	}
 
+	if (enable_quota) {
+		ret = setup_quota_root(fs_info);
+		if (ret < 0) {
+			error("failed to initialize quota: %d (%s)", ret,
+				strerror(-ret));
+			goto out;
+		}
+	}
 	if (verbose) {
 		char features_buf[64];
 
-- 
2.14.3

[PATCH 8/8] btrfs-progs: test/mkfs: Add test case for --enable-quota option

[Qu Wenruo]

Only test if btrfs check (which will check qgroup by default) and kernel
mount success.

Comprehensive qgroup test cases still belongs to fstests.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/mkfs-tests/001-basic-profiles/test.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tests/mkfs-tests/001-basic-profiles/test.sh b/tests/mkfs-tests/001-basic-profiles/test.sh
index 854ee0072170..a8ef01627b20 100755
--- a/tests/mkfs-tests/001-basic-profiles/test.sh
+++ b/tests/mkfs-tests/001-basic-profiles/test.sh
@@ -48,6 +48,16 @@ test_mkfs_single  -d  dup     -m  single
 test_mkfs_single  -d  dup     -m  dup
 test_mkfs_single  -d  dup     -m  dup     --mixed
 
+test_mkfs_single --enable-quota
+test_mkfs_single --enable-quota -d  single  -m  single
+test_mkfs_single --enable-quota -d  single  -m  single  --mixed
+test_mkfs_single --enable-quota -d  single  -m  dup
+test_mkfs_single --enable-quota -d  dup     -m  single
+test_mkfs_single --enable-quota -d  dup     -m  dup
+test_mkfs_single --enable-quota -d  dup     -m  dup     --mixed
+
+# Profile doesn't really affect quota, skip them to save some time
+
 test_mkfs_multi
 test_mkfs_multi   -d  single  -m  single
 test_mkfs_multi   -d  single  -m  single  --mixed
-- 
2.14.3

Re: [PATCH 0/8] mkfs: Quota support

[David Sterba]

On Fri, Oct 27, 2017 at 03:29:28PM +0800, Qu Wenruo wrote:
> This patchset adds quota support, which means the result fs will have
> quota enabled by default, and its accounting is already consistent, no
> manually rescan or quota enable is needed.

Great, thanks for working on that. I'm going to merge the preparatory
patches next. I'm not sure about the command line option, we might want
to design it in a similar way as -O/--features. As this is a runtime
feature (unlike the on-disk format features), a new option will be
needed, backed by some research what other mkfs.* tools use.

The uuid_tree would at least use the btrfs_create_tree, but I don't
think we'd need to create it only on request, so quotas will be the only
option.

> 2) Fill quota root with basic skeleton
>    Only 3 items are really needed
>    a) global quota status item
>    b) quota info for 0/5 qgroup
>    c) quota limit for 0/5 qgroup
> 
>    The content doesn't matter at all.

How would --rootdir and quotas behave in that case? A related question
is what will be the semantics of rootdir+quotas+subvolume created at
mkfs time.

We can certainly start with only quotas on the empty filesystem, but I'm
sure somebody will ask for the wild combination at some point.

Re: [PATCH 0/8] mkfs: Quota support

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2174 bytes --]



On 2017年10月28日 01:37, David Sterba wrote:
> On Fri, Oct 27, 2017 at 03:29:28PM +0800, Qu Wenruo wrote:
>> This patchset adds quota support, which means the result fs will have
>> quota enabled by default, and its accounting is already consistent, no
>> manually rescan or quota enable is needed.
> 
> Great, thanks for working on that. I'm going to merge the preparatory
> patches next. I'm not sure about the command line option, we might want
> to design it in a similar way as -O/--features. As this is a runtime
> feature (unlike the on-disk format features), a new option will be
> needed, backed by some research what other mkfs.* tools use.

OK, I'll reconsider the command line interface.

Although I'm not pretty sure if other fs will has such design.

> 
> The uuid_tree would at least use the btrfs_create_tree, but I don't
> think we'd need to create it only on request, so quotas will be the only
> option.
> 
>> 2) Fill quota root with basic skeleton
>>    Only 3 items are really needed
>>    a) global quota status item
>>    b) quota info for 0/5 qgroup
>>    c) quota limit for 0/5 qgroup
>>
>>    The content doesn't matter at all.
> 
> How would --rootdir and quotas behave in that case? A related question
> is what will be the semantics of rootdir+quotas+subvolume created at
> mkfs time.

Already considered! (Although not in test case)

For current behavior, rootdir works fine, as we only use one subvolume
(fs tree) and rescan is placed just after we filled fs tree.

So, as you could try your self, it's completely fine.

And the infrastructure I used in the helpers can easily handle the case
where extra subvolume qgroups are used.
Just pass insert_qgroup_items() with new qgroupid, then rescan will
handle them all.

Thanks,
Qu

> 
> We can certainly start with only quotas on the empty filesystem, but I'm
> sure somebody will ask for the wild combination at some point.> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: [PATCH 0/8] mkfs: Quota support

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2050 bytes --]



On 2017年10月28日 01:37, David Sterba wrote:
> On Fri, Oct 27, 2017 at 03:29:28PM +0800, Qu Wenruo wrote:
>> This patchset adds quota support, which means the result fs will have
>> quota enabled by default, and its accounting is already consistent, no
>> manually rescan or quota enable is needed.
> 
> Great, thanks for working on that. I'm going to merge the preparatory
> patches next. I'm not sure about the command line option, we might want
> to design it in a similar way as -O/--features. As this is a runtime
> feature (unlike the on-disk format features), a new option will be
> needed, backed by some research what other mkfs.* tools use.

Well, checked xfs and ext4, they mostly co-operate with mount options
and mkfs options to handle optional feature like their quota.


But considering they have multiple options for journal/extra setting, it
won't be reasonable for mkfs.btrfs to have something like
"--runtime-features|-R"

If you're OK with that option name, I can use it to implement "-R quota".

> 
> The uuid_tree would at least use the btrfs_create_tree, but I don't
> think we'd need to create it only on request, so quotas will be the only
> option.
> 
>> 2) Fill quota root with basic skeleton
>>    Only 3 items are really needed
>>    a) global quota status item
>>    b) quota info for 0/5 qgroup
>>    c) quota limit for 0/5 qgroup
>>
>>    The content doesn't matter at all.
> 
> How would --rootdir and quotas behave in that case? A related question
> is what will be the semantics of rootdir+quotas+subvolume created at
> mkfs time.
> 
> We can certainly start with only quotas on the empty filesystem, but I'm
> sure somebody will ask for the wild combination at some point.

I'll add test case for quota with --rootdir in next update.

Thanks,
Qu

> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: [PATCH 5/8] btrfs-progs: ctree: Introduce function to create an empty tree

[Lu Fengqi]

On 10/27/2017 03:29 PM, Qu Wenruo wrote:
> Introduce a new function, btrfs_create_tree(), to create an empty tree.
> 
> Currently, there is only one caller to create new tree, namely
> data reloc tree in mkfs.
> However it's copying fs tree to create a new root.
> 
> This copy fs tree method is not a good idea if we only need an empty
> tree.
> 
> So here introduce a new function, btrfs_create_tree() to create new
> tree.
> Which will handle the following things:
> 1) New tree root leaf
>     Using generic tree allocation
> 
> 2) New root item in tree root
> 
> 3) Modify special tree root pointers in fs_info
>     Only quota_root is supported yet, but can be expended easily
> 
> This patch provides the basis to implement quota support in mkfs.
> 
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>   ctree.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   ctree.h |   2 ++
>   2 files changed, 111 insertions(+)
> 
> diff --git a/ctree.c b/ctree.c
> index 4fc33b14000a..c707be58c413 100644
> --- a/ctree.c
> +++ b/ctree.c
> @@ -22,6 +22,7 @@
>   #include "repair.h"
>   #include "internal.h"
>   #include "sizes.h"
> +#include "utils.h"
>   
>   static int split_node(struct btrfs_trans_handle *trans, struct btrfs_root
>   		      *root, struct btrfs_path *path, int level);
> @@ -136,6 +137,114 @@ int btrfs_copy_root(struct btrfs_trans_handle *trans,
>   	return 0;
>   }
>   
> +/*
> + * Create a new tree root, with root objectid set to @objectid.
> + *
> + * NOTE: Doesn't support tree with non-zero offset, like tree reloc tree.
> + */
> +int btrfs_create_root(struct btrfs_trans_handle *trans,
> +		      struct btrfs_fs_info *fs_info, u64 objectid)
> +{
> +	struct extent_buffer *node;
> +	struct btrfs_root *new_root;
> +	struct btrfs_disk_key disk_key;
> +	struct btrfs_key location;
> +	struct btrfs_root_item root_item = { 0 };
> +	int ret;
> +
> +	new_root = malloc(sizeof(*new_root));
> +	if (!new_root)
> +		return -ENOMEM;
> +
> +	btrfs_setup_root(new_root, fs_info, objectid);
> +	if (!is_fstree(objectid))
> +		new_root->track_dirty = 1;
> +	add_root_to_dirty_list(new_root);

Since add_root_to_dirty_list only add root which track_dirty != 0 to 
dirty list, why not write like the following?

         if (!is_fstree(objectid)) {
                 new_root->track_dirty = 1;
                 add_root_to_dirty_list(new_root);
         }

> +
> +	new_root->objectid = objectid;
> +	new_root->root_key.objectid = objectid;

These have been initialized in btrfs_setup_root, so we don't need to 
initialize again.

> +	new_root->root_key.type = BTRFS_ROOT_ITEM_KEY;
> +	new_root->root_key.offset = 0;
> +
> +	node = btrfs_alloc_free_block(trans, new_root, fs_info->nodesize,
> +				      objectid, &disk_key, 0, 0, 0);
> +	if (IS_ERR(node)) {
> +		ret = PTR_ERR(node);
> +		error("failed to create root node for tree %llu: %d (%s)",
> +		      objectid, ret, strerror(-ret));
> +		return ret;
> +	}
> +	new_root->node = node;
> +
> +	btrfs_set_header_generation(node, trans->transid);
> +	btrfs_set_header_backref_rev(node, BTRFS_MIXED_BACKREF_REV);
> +	btrfs_clear_header_flag(node, BTRFS_HEADER_FLAG_RELOC |
> +				      BTRFS_HEADER_FLAG_WRITTEN);
> +	btrfs_set_header_owner(node, objectid);
> +	btrfs_set_header_nritems(node, 0);
> +	btrfs_set_header_level(node, 0);
> +	write_extent_buffer(node, fs_info->fsid, btrfs_header_fsid(),
> +			    BTRFS_FSID_SIZE);
> +	ret = btrfs_inc_ref(trans, new_root, node, 0);
> +	if (ret < 0)
> +		goto free;
> +
> +	/*
> +	 * Special tree roots may need to modify pointers in @fs_info
> +	 * Only quota is supported yet.
> +	 */
> +	switch (objectid) {
> +	case BTRFS_QUOTA_TREE_OBJECTID:
> +		if (fs_info->quota_root) {
> +			error("quota root already exists");
> +			ret = -EEXIST;
> +			goto free;
> +		}
> +		fs_info->quota_root = new_root;
> +		fs_info->quota_enabled = 1;
> +		break;
> +	/*
> +	 * Essential trees can't be created by this function, yet.
> +	 * As we expect such skeleton exists, or a lot of functions like
> +	 * btrfs_alloc_free_block() doesn't work at all
> +	 */
> +	case BTRFS_ROOT_TREE_OBJECTID:
> +	case BTRFS_EXTENT_TREE_OBJECTID:
> +	case BTRFS_CHUNK_TREE_OBJECTID:
> +	case BTRFS_FS_TREE_OBJECTID:
> +		ret = -EEXIST;
> +		goto free;
> +	default:
> +		/* Subvolume trees don't need special handles */
> +		if (is_fstree(objectid))
> +			break;
> +		/* Other special trees are not supported yet */
> +		ret = -ENOTTY;
> +		goto free;
> +	}
> +	btrfs_mark_buffer_dirty(node);
> +	btrfs_set_root_bytenr(&root_item, btrfs_header_bytenr(node));
> +	btrfs_set_root_level(&root_item, 0);
> +	btrfs_set_root_generation(&root_item, trans->transid);
> +	btrfs_set_root_dirid(&root_item, 0);
> +	btrfs_set_root_refs(&root_item, 1);
> +	btrfs_set_root_used(&root_item, fs_info->nodesize);
> +	location.objectid = objectid;
> +	location.type = BTRFS_ROOT_ITEM_KEY;
> +	location.offset = 0;
> +
> +	ret = btrfs_insert_root(trans, fs_info->tree_root, &location,
> +				&root_item);
> +	if (ret < 0)
> +		goto free;
> +	return ret;
> +
> +free:
> +	free_extent_buffer(node);
> +	free(new_root);
> +	return ret;
> +}
> +
>   /*
>    * check if the tree block can be shared by multiple trees
>    */
> diff --git a/ctree.h b/ctree.h
> index 506b76766579..ac8ce70a55e7 100644
> --- a/ctree.h
> +++ b/ctree.h
> @@ -2558,6 +2558,8 @@ int btrfs_copy_root(struct btrfs_trans_handle *trans,
>   		      struct btrfs_root *root,
>   		      struct extent_buffer *buf,
>   		      struct extent_buffer **cow_ret, u64 new_root_objectid);
> +int btrfs_create_root(struct btrfs_trans_handle *trans,
> +		      struct btrfs_fs_info *fs_info, u64 objectid);
>   int btrfs_extend_item(struct btrfs_root *root, struct btrfs_path *path,
>   		u32 data_size);
>   int btrfs_truncate_item(struct btrfs_root *root, struct btrfs_path *path,
> 


-- 
Thanks,
Lu

Re: [PATCH 3/8] btrfs-progs: qgroup-verify: Move qgroup classification out of report_qgroups

[Lu Fengqi]

On Fri, Oct 27, 2017 at 03:29:31PM +0800, Qu Wenruo wrote:
>The original qgroup-verify integrates qgroup classification into
>report_qgroups().
>This behavior makes silent qgroup repair (or offline rescan) impossible.
>
>To repair qgroup, we must call report_qgroups() to trigger bad qgroup
>classification, which will output error message.
>
>This patch moves bad qgroup classification from report_qgroups() to
>qgroup_verify_all().
>The functionality is not changed, but provides the basis for later
>silent qgroup repair.
>
>Signed-off-by: Qu Wenruo <wqu@suse.com>
>---
> qgroup-verify.c | 28 ++++++++++++++++++++++++++--
> 1 file changed, 26 insertions(+), 2 deletions(-)
>
>diff --git a/qgroup-verify.c b/qgroup-verify.c
>index f509cdb9317a..67bd8b96ddd3 100644
>--- a/qgroup-verify.c
>+++ b/qgroup-verify.c
>@@ -1319,8 +1319,7 @@ void report_qgroups(int all)
> 	while (node) {
> 		c = rb_entry(node, struct qgroup_count, rb_node);
> 
>-		if (report_qgroup_difference(c, all))
>-			list_add_tail(&c->bad_list, &bad_qgroups);
>+		report_qgroup_difference(c, all);
> 
> 		node = rb_next(node);
> 	}
>@@ -1358,9 +1357,20 @@ void free_qgroup_counts(void)
> 	}
> }
> 
>+static bool is_bad_qgroup(struct qgroup_count *count)
>+{
>+	struct qgroup_info *info = &count->info;
>+	struct qgroup_info *disk = &count->diskinfo;
>+	s64 excl_diff = info->exclusive - disk->exclusive;
>+	s64 ref_diff = info->referenced - disk->referenced;
>+
>+	return (excl_diff || ref_diff);
>+}
>+

The function is_bad_qgroup can be reused in report_qgroup_difference.


> int qgroup_verify_all(struct btrfs_fs_info *info)
> {
> 	int ret;
>+	struct rb_node *node;
> 
> 	if (!info->quota_enabled)
> 		return 0;
>@@ -1395,6 +1405,20 @@ int qgroup_verify_all(struct btrfs_fs_info *info)
> 
> 	ret = account_all_refs(1, 0);
> 
>+	/*
>+	 * Do the correctness check here, so for callers who don't want
>+	 * verbose report can skip calling report_qgroups()
>+	 */
>+	node = rb_first(&counts.root);
>+	while (node) {
>+		struct qgroup_count *c;
>+
>+		c = rb_entry(node, struct qgroup_count, rb_node);
>+		if (is_bad_qgroup(c))
>+			list_add_tail(&c->bad_list, &bad_qgroups);
>+		node = rb_next(node);
>+	}
>+
> out:
> 	/*
> 	 * Don't free the qgroup count records as they will be walked
>-- 
>2.14.3
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

-- 
Thanks,
Lu

Re: [PATCH 0/8] mkfs: Quota support

[Lu Fengqi]

On Fri, Oct 27, 2017 at 03:29:28PM +0800, Qu Wenruo wrote:
>This patchset adds quota support, which means the result fs will have
>quota enabled by default, and its accounting is already consistent, no
>manually rescan or quota enable is needed.
>
>The overall design of such support is:
>1) Create needed tree
>   Both btrfs_root and real root item and tree root leaf.
>   For this, a new infrastructure, btrfs_create_tree(), is added for
>   this.
>
>2) Fill quota root with basic skeleton
>   Only 3 items are really needed
>   a) global quota status item
>   b) quota info for 0/5 qgroup
>   c) quota limit for 0/5 qgroup
>
>   The content doesn't matter at all.
>
>3) Repair qgroups using infrastructure from qgroup-verify
>   In fact, qgroup repair is just offline rescan.
>   Although the original qgroup-verify infrastructure is mostly noisy,
>   modify it a little to make it silent to function as offline quota
>   rescan.
>
>And such support is mainly designed for developers and QA guys.
>
>As to enable quota, we must normally mount the fs, enable quota (and
>rescan if needed).
>This ioctl based procedure is not common, and fstests doesn't provide
>such support.
>
>There are several attempts to make fstests to support it, but due to
>different reasons, all these attempts failed.
>
>To make it easier to test all existing test cases with btrfs quota
>enabled, the current best method is to support quota at mkfs time, and
>here comes the patchset.
>

In addition to some trivial problems mentioned earlier, all series of
patches look good to me.

Reviewed-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>

-- 
Thanks,
Lu

>Qu Wenruo (8):
>  btrfs-progs: qgroup-verify: Also repair qgroup status version
>  btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we
>    should repair qgroups
>  btrfs-progs: qgroup-verify: Move qgroup classification out of
>    report_qgroups
>  btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent
>    repair
>  btrfs-progs: ctree: Introduce function to create an empty tree
>  btrfs-progs: mkfs: Introduce function to insert qgroup info and limit
>    items
>  btrfs-progs: mkfs: Introduce option to enable quota at mkfs time
>  btrfs-progs: test/mkfs: Add test case for --enable-quota option
>
> Documentation/mkfs.btrfs.asciidoc           |   3 +
> cmds-check.c                                |   2 +-
> ctree.c                                     | 109 ++++++++++++++++++++++
> ctree.h                                     |   3 +
> mkfs/main.c                                 | 136 +++++++++++++++++++++++++++-
> qgroup-verify.c                             |  51 ++++++++---
> qgroup-verify.h                             |   2 +-
> tests/mkfs-tests/001-basic-profiles/test.sh |  10 ++
> 8 files changed, 302 insertions(+), 14 deletions(-)
>
>-- 
>2.14.3
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>