[PATCH v4 00/11] btrfs-progs: mkfs: Quota support through -Q|--quota

[PATCH v4 00/11] btrfs-progs: mkfs: Quota support through -Q|--quota

[Marcos Paulo de Souza]

From: Marcos Paulo de Souza <mpdesouza@suse.com>

Hi guys,

This if the forth version of this patchset. The last submission of these patches
was in 2018[1]. This version is based on top on the current devel branch, with
minor cleanups, minor conflicts and only a real fix in patch 0008. I would like
to ask you guys to review these patches, since v3 didn't receive any feedback at
the time.

I only added my SoB in three patches, which were those were I needed a manual
intervention, or a specific fix as I mentioned.

Thanks for your review,
  Marcos

Original cover letter for Wenruo:
This patchset adds quota support, which means the result fs will have
quota enabled by default, and its accounting is already consistent, no
manually rescan or quota enable is needed.

The overall design of such support is:
1) Create needed tree
   Both btrfs_root and real root item and tree root leaf.
   For this, a new infrastructure, btrfs_create_tree(), is added for
   this.

2) Fill quota root with basic skeleton
   Only 3 items are really needed
   a) global quota status item
   b) quota info for specified qgroup
   c) quota limit for specified qgroup

   Currently we insert all qgroup items for all existing file trees.
   If we're going to support extra subvolume at mkfs time, just pass the
   subvolume id into insert_qgroup_items().

   The content doesn't matter at all.

3) Repair qgroups using infrastructure from qgroup-verify
   In fact, qgroup repair is just offline rescan.
   Although the original qgroup-verify infrastructure is mostly noisy,
   modify it a little to make it silent to function as offline quota
   rescan.

And such support is mainly designed for developers and QA guys.

As to enable quota, before we must normally mount the fs, enable quota
(and rescan if needed).
This ioctl based procedure is not common, and fstests doesn't provide
such support.
(Not to mention sometimes rescan itself can be buggy)

There are several attempts to make fstests to support it, but due to
different reasons, all these attempts failed.

To make it easier to test all existing test cases with btrfs quota
enabled, the current best method is to support quota at mkfs time, and
here comes the patchset.

[1]: https://lore.kernel.org/linux-btrfs/20180807081938.21348-1-wqu@suse.com/T/#m107735cecbf4729b599e6e4eee0a54802909b30d

Qu Wenruo (11):
  btrfs-progs: qgroup-verify: Avoid NULL pointer dereference for later
    silent qgroup repair
  btrfs-progs: qgroup-verify: Also repair qgroup status version
  btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we
    should repair qgroups
  btrfs-progs: qgroup-verify: Move qgroup classification out of
    report_qgroups
  btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent
    repair
  btrfs-progs: ctree: Introduce function to create an empty tree
  btrfs-progs: mkfs: Introduce function to insert qgroup info and limit
    items
  btrfs-progs: mkfs: Introduce function to setup quota root and rescan
  btrfs-progs: mkfs: Introduce mkfs time quota support
  btrfs-progs: test/mkfs: Add test case for -Q|--quota option
  btrfs-progs: test/mkfs: Add test case for --rootdir and --quota

 Documentation/mkfs.btrfs.asciidoc             |   5 +
 check/main.c                                  |  20 +--
 check/qgroup-verify.c                         |  93 ++++++++----
 check/qgroup-verify.h                         |   4 +-
 ctree.c                                       | 109 +++++++++++++-
 ctree.h                                       |   3 +
 mkfs/main.c                                   | 136 +++++++++++++++++-
 tests/mkfs-tests/001-basic-profiles/test.sh   |  10 ++
 .../mkfs-tests/018-rootdir-with-quota/test.sh |  51 +++++++
 9 files changed, 389 insertions(+), 42 deletions(-)
 create mode 100755 tests/mkfs-tests/018-rootdir-with-quota/test.sh

-- 
2.25.0

[PATCH v4 01/11] btrfs-progs: qgroup-verify: Avoid NULL pointer dereference for later silent qgroup repair

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

Commit 078e9a1cc973 ("btrfs-progs: check: enhanced progress indicator")
introduced @qgroup_item_count for progress indicator.

However since we will later introduce silent qgroup rescan
functionality, the @qgroup_item_count pointer can be NULL.

So check if @qgroup_item_count is NULL before accessing it.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 check/qgroup-verify.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/check/qgroup-verify.c b/check/qgroup-verify.c
index afe15acf..17c266d4 100644
--- a/check/qgroup-verify.c
+++ b/check/qgroup-verify.c
@@ -743,7 +743,8 @@ static int travel_tree(struct btrfs_fs_info *info, struct btrfs_root *root,
 	 */
 	nr = btrfs_header_nritems(eb);
 	for (i = 0; i < nr; i++) {
-		(*qgroup_item_count)++;
+		if (qgroup_item_count)
+			(*qgroup_item_count)++;
 		new_bytenr = btrfs_node_blockptr(eb, i);
 		new_num_bytes = info->nodesize;
 
-- 
2.25.0

[PATCH v4 02/11] btrfs-progs: qgroup-verify: Also repair qgroup status version

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

Current kernel only supports qgroup version 1.
Make qgroup-verify to follow this standard.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 check/qgroup-verify.c | 2 ++
 ctree.h               | 1 +
 2 files changed, 3 insertions(+)

diff --git a/check/qgroup-verify.c b/check/qgroup-verify.c
index 17c266d4..0509ecec 100644
--- a/check/qgroup-verify.c
+++ b/check/qgroup-verify.c
@@ -1598,6 +1598,8 @@ static int repair_qgroup_status(struct btrfs_fs_info *info)
 	btrfs_set_qgroup_status_rescan(path.nodes[0], status_item, 0);
 	btrfs_set_qgroup_status_generation(path.nodes[0], status_item,
 					   trans->transid);
+	btrfs_set_qgroup_status_version(path.nodes[0], status_item,
+					BTRFS_QGROUP_STATUS_VERSION);
 
 	btrfs_mark_buffer_dirty(path.nodes[0]);
 
diff --git a/ctree.h b/ctree.h
index 36f62732..083bde3c 100644
--- a/ctree.h
+++ b/ctree.h
@@ -1034,6 +1034,7 @@ struct btrfs_qgroup_status_item {
 	__le64 rescan;		/* progress during scanning */
 } __attribute__ ((__packed__));
 
+#define BTRFS_QGROUP_STATUS_VERSION		1
 struct btrfs_block_group_item {
 	__le64 used;
 	__le64 chunk_objectid;
-- 
2.25.0

[PATCH v4 03/11] btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we should repair qgroups

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

In fact qgroup-verify is just kind of offline qgroup rescan, and later
mkfs qgroup support will reuse it.

So qgroup-verify doesn't really need to rely the global variable @repair
to check if it should repair qgroups.

Instead check fs_info->readonly to do the repair.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 check/qgroup-verify.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/check/qgroup-verify.c b/check/qgroup-verify.c
index 0509ecec..b7b63095 100644
--- a/check/qgroup-verify.c
+++ b/check/qgroup-verify.c
@@ -1617,7 +1617,7 @@ int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
 
 	*repaired = 0;
 
-	if (!repair)
+	if (info->readonly)
 		return 0;
 
 	list_for_each_entry_safe(count, tmpcount, &bad_qgroups, bad_list) {
-- 
2.25.0

[PATCH v4 04/11] btrfs-progs: qgroup-verify: Move qgroup classification out of report_qgroups

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

The original qgroup-verify integrates qgroup classification into
report_qgroups().
This behavior makes silent qgroup repair (or offline rescan) impossible.

To repair qgroup, we must call report_qgroups() to trigger bad qgroup
classification, which will output error message.

This patch moves bad qgroup classification from report_qgroups() to
qgroup_verify_all().
Now report_qgroups() is pretty lightweight, only doing basic qgroup
difference report thus change it type to void.

And since the functionality of qgroup_verify_all() changes, change
callers to handle the new return value correctly.

Signed-off-by: Qu Wenruo <wqu@suse.com>
[ removed some comments ]
Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
---
 check/main.c          | 18 ++++++------
 check/qgroup-verify.c | 67 +++++++++++++++++++++++++++++++------------
 check/qgroup-verify.h |  2 +-
 3 files changed, 59 insertions(+), 28 deletions(-)

diff --git a/check/main.c b/check/main.c
index 49bdbfec..7632b60a 100644
--- a/check/main.c
+++ b/check/main.c
@@ -9941,7 +9941,7 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 	int clear_space_cache = 0;
 	int qgroup_report = 0;
 	int qgroups_repaired = 0;
-	int qgroup_report_ret;
+	int qgroup_verify_ret;
 	unsigned ctree_flags = OPEN_CTREE_EXCLUSIVE;
 	int force = 0;
 
@@ -10198,8 +10198,8 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 		       uuidbuf);
 		ret = qgroup_verify_all(info);
 		err |= !!ret;
-		if (ret == 0)
-			err |= !!report_qgroups(1);
+		if (ret >= 0)
+			report_qgroups(1);
 		goto close_out;
 	}
 	if (subvolid) {
@@ -10433,21 +10433,21 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 			ctx.tp = TASK_QGROUPS;
 			task_start(ctx.info, &ctx.start_time, &ctx.item_count);
 		}
-		ret = qgroup_verify_all(info);
+		qgroup_verify_ret = qgroup_verify_all(info);
 		task_stop(ctx.info);
-		err |= !!ret;
-		if (ret) {
+		if (qgroup_verify_ret < 0) {
 			error("failed to check quota groups");
+			err |= !!qgroup_verify_ret;
 			goto out;
 		}
-		qgroup_report_ret = report_qgroups(0);
+		report_qgroups(0);
 		ret = repair_qgroups(info, &qgroups_repaired);
 		if (ret) {
 			error("failed to repair quota groups");
 			goto out;
 		}
-		if (qgroup_report_ret && (!qgroups_repaired || ret))
-			err |= qgroup_report_ret;
+		if (qgroup_verify_ret && (!qgroups_repaired || ret))
+			err |= !!qgroup_verify_ret;
 		ret = 0;
 	} else {
 		fprintf(stderr,
diff --git a/check/qgroup-verify.c b/check/qgroup-verify.c
index b7b63095..b1e6b26c 100644
--- a/check/qgroup-verify.c
+++ b/check/qgroup-verify.c
@@ -1311,18 +1311,13 @@ static int report_qgroup_difference(struct qgroup_count *count, int verbose)
 
 /*
  * Report qgroups errors
- * Return 0 if nothing wrong.
- * Return <0 if any qgroup is inconsistent.
- *
  * @all:	if set, all qgroup will be checked and reported even already
  * 		inconsistent or under rescan.
  */
-int report_qgroups(int all)
+void report_qgroups(int all)
 {
 	struct rb_node *node;
 	struct qgroup_count *c;
-	bool found_err = false;
-	bool skip_err = false;
 
 	if (!repair && counts.rescan_running) {
 		if (all) {
@@ -1331,34 +1326,26 @@ int report_qgroups(int all)
 		} else {
 			printf(
 	"Qgroup rescan is running, qgroups will not be printed.\n");
-			return 0;
+			return;
 		}
 	}
 	/*
 	 * It's possible that rescan hasn't been initialized yet.
 	 */
 	if (counts.qgroup_inconsist && !counts.rescan_running &&
-	    counts.rescan_running == 0) {
+	    counts.rescan_running == 0)
 		printf(
-"Rescan hasn't been initialized, a difference in qgroup accounting is expected\n");
-		skip_err = true;
-	}
+"Rescan hasn't been initialzied, a difference in qgroup accounting is expected\n");
 	if (counts.qgroup_inconsist && !counts.rescan_running)
 		fprintf(stderr, "Qgroup are marked as inconsistent.\n");
 	node = rb_first(&counts.root);
 	while (node) {
 		c = rb_entry(node, struct qgroup_count, rb_node);
 
-		if (report_qgroup_difference(c, all)) {
-			list_add_tail(&c->bad_list, &bad_qgroups);
-			found_err = true;
-		}
+		report_qgroup_difference(c, all);
 
 		node = rb_next(node);
 	}
-	if (found_err && !skip_err)
-		return -EUCLEAN;
-	return 0;
 }
 
 void free_qgroup_counts(void)
@@ -1393,9 +1380,29 @@ void free_qgroup_counts(void)
 	}
 }
 
+static bool is_bad_qgroup(struct qgroup_count *count)
+{
+	struct qgroup_info *info = &count->info;
+	struct qgroup_info *disk = &count->diskinfo;
+	s64 excl_diff = info->exclusive - disk->exclusive;
+	s64 ref_diff = info->referenced - disk->referenced;
+
+	return (excl_diff || ref_diff);
+}
+
+/*
+ * Verify all qgroup numbers.
+ *
+ * Return <0 for fatal errors (e.g. ENOMEM or failed to read quota tree)
+ * Return 0 if all qgroup numbers are correct or no need to check (under rescan)
+ * Return >0 if qgroup numbers are inconsistent.
+ */
 int qgroup_verify_all(struct btrfs_fs_info *info)
 {
 	int ret;
+	bool found_err = false;
+	bool skip_err = false;
+	struct rb_node *node;
 
 	if (!info->quota_enabled)
 		return 0;
@@ -1413,6 +1420,12 @@ int qgroup_verify_all(struct btrfs_fs_info *info)
 		goto out;
 	}
 
+	if (counts.rescan_running)
+		skip_err = true;
+	if (counts.qgroup_inconsist && !counts.rescan_running &&
+	    counts.rescan_running == 0)
+		skip_err = true;
+
 	/*
 	 * Put all extent refs into our rbtree
 	 */
@@ -1430,6 +1443,22 @@ int qgroup_verify_all(struct btrfs_fs_info *info)
 
 	ret = account_all_refs(1, 0);
 
+	/*
+	 * Do the correctness check here, so for callers who don't want
+	 * verbose report can skip calling report_qgroups()
+	 */
+	node = rb_first(&counts.root);
+	while (node) {
+		struct qgroup_count *c;
+
+		c = rb_entry(node, struct qgroup_count, rb_node);
+		if (is_bad_qgroup(c)) {
+			list_add_tail(&c->bad_list, &bad_qgroups);
+			found_err = true;
+		}
+		node = rb_next(node);
+	}
+
 out:
 	/*
 	 * Don't free the qgroup count records as they will be walked
@@ -1437,6 +1466,8 @@ out:
 	 */
 	free_tree_blocks();
 	free_ref_tree(&by_bytenr);
+	if (!ret && !skip_err && found_err)
+		ret = 1;
 	return ret;
 }
 
diff --git a/check/qgroup-verify.h b/check/qgroup-verify.h
index 20e93708..6495dd18 100644
--- a/check/qgroup-verify.h
+++ b/check/qgroup-verify.h
@@ -23,7 +23,7 @@
 #include "ctree.h"
 
 int qgroup_verify_all(struct btrfs_fs_info *info);
-int report_qgroups(int all);
+void report_qgroups(int all);
 int repair_qgroups(struct btrfs_fs_info *info, int *repaired);
 
 int print_extent_state(struct btrfs_fs_info *info, u64 subvol);
-- 
2.25.0

[PATCH v4 05/11] btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent repair

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

Allow repair_qgroups() to do silent repair, so it can acts as offline
qgroup rescan.

This provides the basis for later mkfs quota support.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 check/main.c          |  2 +-
 check/qgroup-verify.c | 19 +++++++++++--------
 check/qgroup-verify.h |  2 +-
 3 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/check/main.c b/check/main.c
index 7632b60a..375aef8f 100644
--- a/check/main.c
+++ b/check/main.c
@@ -10441,7 +10441,7 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)
 			goto out;
 		}
 		report_qgroups(0);
-		ret = repair_qgroups(info, &qgroups_repaired);
+		ret = repair_qgroups(info, &qgroups_repaired, false);
 		if (ret) {
 			error("failed to repair quota groups");
 			goto out;
diff --git a/check/qgroup-verify.c b/check/qgroup-verify.c
index b1e6b26c..b1736aab 100644
--- a/check/qgroup-verify.c
+++ b/check/qgroup-verify.c
@@ -1540,7 +1540,7 @@ out:
 }
 
 static int repair_qgroup_info(struct btrfs_fs_info *info,
-			      struct qgroup_count *count)
+			      struct qgroup_count *count, bool silent)
 {
 	int ret;
 	struct btrfs_root *root = info->quota_root;
@@ -1549,8 +1549,10 @@ static int repair_qgroup_info(struct btrfs_fs_info *info,
 	struct btrfs_qgroup_info_item *info_item;
 	struct btrfs_key key;
 
-	printf("Repair qgroup %llu/%llu\n", btrfs_qgroup_level(count->qgroupid),
-	       btrfs_qgroup_subvid(count->qgroupid));
+	if (!silent)
+		printf("Repair qgroup %llu/%llu\n",
+			btrfs_qgroup_level(count->qgroupid),
+			btrfs_qgroup_subvid(count->qgroupid));
 
 	trans = btrfs_start_transaction(root, 1);
 	if (IS_ERR(trans))
@@ -1595,7 +1597,7 @@ out:
 	return ret;
 }
 
-static int repair_qgroup_status(struct btrfs_fs_info *info)
+static int repair_qgroup_status(struct btrfs_fs_info *info, bool silent)
 {
 	int ret;
 	struct btrfs_root *root = info->quota_root;
@@ -1604,7 +1606,8 @@ static int repair_qgroup_status(struct btrfs_fs_info *info)
 	struct btrfs_key key;
 	struct btrfs_qgroup_status_item *status_item;
 
-	printf("Repair qgroup status item\n");
+	if (!silent)
+		printf("Repair qgroup status item\n");
 
 	trans = btrfs_start_transaction(root, 1);
 	if (IS_ERR(trans))
@@ -1641,7 +1644,7 @@ out:
 	return ret;
 }
 
-int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
+int repair_qgroups(struct btrfs_fs_info *info, int *repaired, bool silent)
 {
 	int ret = 0;
 	struct qgroup_count *count, *tmpcount;
@@ -1652,7 +1655,7 @@ int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
 		return 0;
 
 	list_for_each_entry_safe(count, tmpcount, &bad_qgroups, bad_list) {
-		ret = repair_qgroup_info(info, count);
+		ret = repair_qgroup_info(info, count, silent);
 		if (ret) {
 			goto out;
 		}
@@ -1668,7 +1671,7 @@ int repair_qgroups(struct btrfs_fs_info *info, int *repaired)
 	 * mount.
 	 */
 	if (*repaired || counts.qgroup_inconsist || counts.rescan_running) {
-		ret = repair_qgroup_status(info);
+		ret = repair_qgroup_status(info, silent);
 		if (ret)
 			goto out;
 
diff --git a/check/qgroup-verify.h b/check/qgroup-verify.h
index 6495dd18..8a8694b6 100644
--- a/check/qgroup-verify.h
+++ b/check/qgroup-verify.h
@@ -24,7 +24,7 @@
 
 int qgroup_verify_all(struct btrfs_fs_info *info);
 void report_qgroups(int all);
-int repair_qgroups(struct btrfs_fs_info *info, int *repaired);
+int repair_qgroups(struct btrfs_fs_info *info, int *repaired, bool silent);
 
 int print_extent_state(struct btrfs_fs_info *info, u64 subvol);
 
-- 
2.25.0

[PATCH v4 06/11] btrfs-progs: ctree: Introduce function to create an empty tree

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

Introduce a new function, btrfs_create_tree(), to create an empty tree.

Currently, there is only one caller to create new tree, namely
data reloc tree in mkfs.
However it's copying fs tree to create a new root.

This copy fs tree method is not a good idea if we only need an empty
tree.

So here introduce a new function, btrfs_create_tree() to create new
tree.
Which will handle the following things:
1) New tree root leaf
   Using generic tree allocation

2) New root item in tree root

3) Modify special tree root pointers in fs_info
   Only quota_root is supported yet, but can be expended easily

This patch provides the basis to implement quota support in mkfs.

Signed-off-by: Qu Wenruo <wqu@suse.com>
[ solved minor conflicts ]
Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
---
 ctree.c | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 ctree.h |   2 ++
 2 files changed, 110 insertions(+), 1 deletion(-)

diff --git a/ctree.c b/ctree.c
index 97b44d48..49598a45 100644
--- a/ctree.c
+++ b/ctree.c
@@ -21,8 +21,9 @@
 #include "print-tree.h"
 #include "repair.h"
 #include "common/internal.h"
-#include "kernel-lib/sizes.h"
 #include "common/messages.h"
+#include "common/utils.h"
+#include "kernel-lib/sizes.h"
 #include "volumes.h"
 
 static int split_node(struct btrfs_trans_handle *trans, struct btrfs_root
@@ -182,6 +183,112 @@ int btrfs_copy_root(struct btrfs_trans_handle *trans,
 	return 0;
 }
 
+/*
+ * Create a new tree root, with root objectid set to @objectid.
+ *
+ * NOTE: Doesn't support tree with non-zero offset, like data reloc tree.
+ */
+int btrfs_create_root(struct btrfs_trans_handle *trans,
+		      struct btrfs_fs_info *fs_info, u64 objectid)
+{
+	struct extent_buffer *node;
+	struct btrfs_root *new_root;
+	struct btrfs_disk_key disk_key;
+	struct btrfs_key location;
+	struct btrfs_root_item root_item = { 0 };
+	int ret;
+
+	new_root = malloc(sizeof(*new_root));
+	if (!new_root)
+		return -ENOMEM;
+
+	btrfs_setup_root(new_root, fs_info, objectid);
+	if (!is_fstree(objectid))
+		new_root->track_dirty = 1;
+	add_root_to_dirty_list(new_root);
+
+	new_root->objectid = objectid;
+	new_root->root_key.objectid = objectid;
+	new_root->root_key.type = BTRFS_ROOT_ITEM_KEY;
+	new_root->root_key.offset = 0;
+
+	node = btrfs_alloc_free_block(trans, new_root, fs_info->nodesize,
+				      objectid, &disk_key, 0, 0, 0);
+	if (IS_ERR(node)) {
+		ret = PTR_ERR(node);
+		error("failed to create root node for tree %llu: %d (%s)",
+		      objectid, ret, strerror(-ret));
+		return ret;
+	}
+	new_root->node = node;
+
+	btrfs_set_header_generation(node, trans->transid);
+	btrfs_set_header_backref_rev(node, BTRFS_MIXED_BACKREF_REV);
+	btrfs_clear_header_flag(node, BTRFS_HEADER_FLAG_RELOC |
+				      BTRFS_HEADER_FLAG_WRITTEN);
+	btrfs_set_header_owner(node, objectid);
+	btrfs_set_header_nritems(node, 0);
+	btrfs_set_header_level(node, 0);
+	ret = btrfs_inc_ref(trans, new_root, node, 0);
+	if (ret < 0)
+		goto free;
+
+	/*
+	 * Special tree roots may need to modify pointers in @fs_info
+	 * Only quota is supported yet.
+	 */
+	switch (objectid) {
+	case BTRFS_QUOTA_TREE_OBJECTID:
+		if (fs_info->quota_root) {
+			error("quota root already exists");
+			ret = -EEXIST;
+			goto free;
+		}
+		fs_info->quota_root = new_root;
+		fs_info->quota_enabled = 1;
+		break;
+	/*
+	 * Essential trees can't be created by this function, yet.
+	 * As we expect such skeleton exists, or a lot of functions like
+	 * btrfs_alloc_free_block() doesn't work at all
+	 */
+	case BTRFS_ROOT_TREE_OBJECTID:
+	case BTRFS_EXTENT_TREE_OBJECTID:
+	case BTRFS_CHUNK_TREE_OBJECTID:
+	case BTRFS_FS_TREE_OBJECTID:
+		ret = -EEXIST;
+		goto free;
+	default:
+		/* Subvolume trees don't need special handles */
+		if (is_fstree(objectid))
+			break;
+		/* Other special trees are not supported yet */
+		ret = -ENOTTY;
+		goto free;
+	}
+	btrfs_mark_buffer_dirty(node);
+	btrfs_set_root_bytenr(&root_item, btrfs_header_bytenr(node));
+	btrfs_set_root_level(&root_item, 0);
+	btrfs_set_root_generation(&root_item, trans->transid);
+	btrfs_set_root_dirid(&root_item, 0);
+	btrfs_set_root_refs(&root_item, 1);
+	btrfs_set_root_used(&root_item, fs_info->nodesize);
+	location.objectid = objectid;
+	location.type = BTRFS_ROOT_ITEM_KEY;
+	location.offset = 0;
+
+	ret = btrfs_insert_root(trans, fs_info->tree_root, &location,
+				&root_item);
+	if (ret < 0)
+		goto free;
+	return ret;
+
+free:
+	free_extent_buffer(node);
+	free(new_root);
+	return ret;
+}
+
 /*
  * check if the tree block can be shared by multiple trees
  */
diff --git a/ctree.h b/ctree.h
index 083bde3c..41565b52 100644
--- a/ctree.h
+++ b/ctree.h
@@ -2641,6 +2641,8 @@ int btrfs_copy_root(struct btrfs_trans_handle *trans,
 		      struct btrfs_root *root,
 		      struct extent_buffer *buf,
 		      struct extent_buffer **cow_ret, u64 new_root_objectid);
+int btrfs_create_root(struct btrfs_trans_handle *trans,
+		      struct btrfs_fs_info *fs_info, u64 objectid);
 int btrfs_extend_item(struct btrfs_root *root, struct btrfs_path *path,
 		u32 data_size);
 int btrfs_truncate_item(struct btrfs_root *root, struct btrfs_path *path,
-- 
2.25.0

[PATCH v4 07/11] btrfs-progs: mkfs: Introduce function to insert qgroup info and limit items

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

Introduce a new function, insert_qgroup_items(), to insert qgroup info
item and qgroup limit item for later mkfs qgroup support.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 mkfs/main.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/mkfs/main.c b/mkfs/main.c
index 316ea82e..6d9b3265 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -798,6 +798,40 @@ out:
 	return ret;
 }
 
+static int insert_qgroup_items(struct btrfs_trans_handle *trans,
+			       struct btrfs_fs_info *fs_info,
+			       u64 qgroupid)
+{
+	struct btrfs_path path;
+	struct btrfs_root *quota_root = fs_info->quota_root;
+	struct btrfs_key key;
+	int ret;
+
+	if (qgroupid >> BTRFS_QGROUP_LEVEL_SHIFT) {
+		error("qgroup level other than 0 is not supported yet");
+		return -ENOTTY;
+	}
+
+	key.objectid = 0;
+	key.type = BTRFS_QGROUP_INFO_KEY;
+	key.offset = qgroupid;
+
+	btrfs_init_path(&path);
+	ret = btrfs_insert_empty_item(trans, quota_root, &path, &key,
+				      sizeof(struct btrfs_qgroup_info_item));
+	btrfs_release_path(&path);
+	if (ret < 0)
+		return ret;
+
+	key.objectid = 0;
+	key.type = BTRFS_QGROUP_LIMIT_KEY;
+	key.offset = qgroupid;
+	ret = btrfs_insert_empty_item(trans, quota_root, &path, &key,
+				      sizeof(struct btrfs_qgroup_limit_item));
+	btrfs_release_path(&path);
+	return ret;
+}
+
 int BOX_MAIN(mkfs)(int argc, char **argv)
 {
 	char *file;
-- 
2.25.0

[PATCH v4 08/11] btrfs-progs: mkfs: Introduce function to setup quota root and rescan

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

Introduce a new function, setup_quota_root(), which will create quota
root, and do an offline rescan to ensure all quota accounting numbers
are correct.

Signed-off-by: Qu Wenruo <wqu@suse.com>
[ minor improvement in the fail path ]
Signed-off-by: Marcos Paulo de Souza <mpdesouza@suse.com>
---
 mkfs/main.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)

diff --git a/mkfs/main.c b/mkfs/main.c
index 6d9b3265..1fb25a9d 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -48,6 +48,7 @@
 #include "crypto/crc32c.h"
 #include "common/fsfeatures.h"
 #include "common/box.h"
+#include "check/qgroup-verify.h"
 
 static int verbose = 1;
 
@@ -832,6 +833,91 @@ static int insert_qgroup_items(struct btrfs_trans_handle *trans,
 	return ret;
 }
 
+static int setup_quota_root(struct btrfs_fs_info *fs_info)
+{
+	struct btrfs_trans_handle *trans;
+	struct btrfs_qgroup_status_item *qsi;
+	struct btrfs_root *quota_root;
+	struct btrfs_path path;
+	struct btrfs_key key;
+	int qgroup_repaired = 0;
+	int ret;
+
+	/* One to modify tree root, one for quota root */
+	trans = btrfs_start_transaction(fs_info->tree_root, 2);
+	if (IS_ERR(trans)) {
+		ret = PTR_ERR(trans);
+		error("failed to start transaction: %d (%s)",
+			ret, strerror(-ret));
+		return ret;
+	}
+	ret = btrfs_create_root(trans, fs_info, BTRFS_QUOTA_TREE_OBJECTID);
+	if (ret < 0) {
+		error("failed to create quota root: %d (%s)",
+			ret, strerror(-ret));
+		goto fail;
+	}
+	quota_root = fs_info->quota_root;
+
+	key.objectid = 0;
+	key.type = BTRFS_QGROUP_STATUS_KEY;
+	key.offset = 0;
+
+	btrfs_init_path(&path);
+	ret = btrfs_insert_empty_item(trans, quota_root, &path, &key,
+				      sizeof(*qsi));
+	if (ret < 0) {
+		error("failed to insert qgroup status item: %d (%s)",
+			ret, strerror(-ret));
+		goto fail;
+	}
+
+	qsi = btrfs_item_ptr(path.nodes[0], path.slots[0],
+			     struct btrfs_qgroup_status_item);
+	btrfs_set_qgroup_status_generation(path.nodes[0], qsi, 0);
+	btrfs_set_qgroup_status_rescan(path.nodes[0], qsi, 0);
+
+	/* Mark current status info inconsistent, and fix it later */
+	btrfs_set_qgroup_status_flags(path.nodes[0], qsi,
+			BTRFS_QGROUP_STATUS_FLAG_ON |
+			BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT);
+	btrfs_release_path(&path);
+
+	/* Currently mkfs will only create one subvolume */
+	ret = insert_qgroup_items(trans, fs_info, BTRFS_FS_TREE_OBJECTID);
+	if (ret < 0) {
+		error("failed to insert qgroup items: %d (%s)",
+			ret, strerror(-ret));
+		goto fail;
+	}
+
+	ret = btrfs_commit_transaction(trans, fs_info->tree_root);
+	if (ret < 0) {
+		error("failed to commit current transaction: %d (%s)",
+			ret, strerror(-ret));
+		return ret;
+	}
+
+	/*
+	 * Qgroup is setup but with wrong info, use qgroup-verify
+	 * infrastructure to repair them.
+	 * (Just acts as offline rescan)
+	 */
+	ret = qgroup_verify_all(fs_info);
+	if (ret < 0) {
+		error("qgroup rescan failed: %d (%s)", ret, strerror(-ret));
+		return ret;
+	}
+	ret = repair_qgroups(fs_info, &qgroup_repaired, true);
+	if (ret < 0)
+		error("failed to fill qgroup info: %d (%s)", ret,
+			strerror(-ret));
+	return ret;
+fail:
+	btrfs_abort_transaction(trans, ret);
+	return ret;
+}
+
 int BOX_MAIN(mkfs)(int argc, char **argv)
 {
 	char *file;
-- 
2.25.0

[PATCH v4 09/11] btrfs-progs: mkfs: Introduce mkfs time quota support

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

The result fs will has quota enabled, with consistent qgroup accounting.

This is quite handy to test quota with fstests, which doesn't support to
call ioctl for btrfs at mount time.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 Documentation/mkfs.btrfs.asciidoc |  5 +++++
 mkfs/main.c                       | 16 +++++++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/Documentation/mkfs.btrfs.asciidoc b/Documentation/mkfs.btrfs.asciidoc
index 0502d1d8..611ea9c2 100644
--- a/Documentation/mkfs.btrfs.asciidoc
+++ b/Documentation/mkfs.btrfs.asciidoc
@@ -87,6 +87,11 @@ updating the metadata blocks.
 +
 NOTE: versions up to 3.11 set the nodesize to 4k.
 
+*-Q|--quota*::
+Enable btrfs quota support. Result filesystem will have quota enabled and all
+qgroup accounting correct.
+See also `btrfs-quota`(8).
+
 *-s|--sectorsize <size>*::
 Specify the sectorsize, the minimum data block allocation unit.
 +
diff --git a/mkfs/main.c b/mkfs/main.c
index 1fb25a9d..70a66cef 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -949,6 +949,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	char *source_dir = NULL;
 	bool source_dir_set = false;
 	bool shrink_rootdir = false;
+	bool enable_quota = false;
 	u64 source_dir_size = 0;
 	u64 min_dev_size;
 	u64 shrink_size;
@@ -985,13 +986,14 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 			{ "nodiscard", no_argument, NULL, 'K' },
 			{ "features", required_argument, NULL, 'O' },
 			{ "uuid", required_argument, NULL, 'U' },
+			{ "quota", required_argument, NULL, 'Q' },
 			{ "quiet", 0, NULL, 'q' },
 			{ "shrink", no_argument, NULL, GETOPT_VAL_SHRINK },
 			{ "help", no_argument, NULL, GETOPT_VAL_HELP },
 			{ NULL, 0, NULL, 0}
 		};
 
-		c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKq",
+		c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKqQ",
 				long_options, NULL);
 		if (c < 0)
 			break;
@@ -1066,6 +1068,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 			case 'q':
 				verbose = 0;
 				break;
+			case 'Q':
+				enable_quota = true;
+				break;
 			case GETOPT_VAL_SHRINK:
 				shrink_rootdir = true;
 				break;
@@ -1470,6 +1475,15 @@ raid_groups:
 		}
 	}
 
+	if (enable_quota) {
+		ret = setup_quota_root(fs_info);
+		if (ret < 0) {
+			error("failed to initialize quota: %d (%s)", ret,
+				strerror(-ret));
+			goto out;
+		}
+	}
+
 	if (verbose) {
 		char features_buf[64];
 
-- 
2.25.0

[PATCH v4 10/11] btrfs-progs: test/mkfs: Add test case for -Q|--quota option

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

Only test if btrfs check (which will check qgroup by default) and kernel
mount success.

Comprehensive qgroup test cases still belongs to fstests.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/mkfs-tests/001-basic-profiles/test.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/tests/mkfs-tests/001-basic-profiles/test.sh b/tests/mkfs-tests/001-basic-profiles/test.sh
index 4b6c2f42..c273f1df 100755
--- a/tests/mkfs-tests/001-basic-profiles/test.sh
+++ b/tests/mkfs-tests/001-basic-profiles/test.sh
@@ -47,6 +47,16 @@ test_mkfs_single  -d  dup     -m  single
 test_mkfs_single  -d  dup     -m  dup
 test_mkfs_single  -d  dup     -m  dup     --mixed
 
+test_mkfs_single -Q
+test_mkfs_single -Q -d  single  -m  single
+test_mkfs_single -Q -d  single  -m  single  --mixed
+test_mkfs_single -Q -d  single  -m  dup
+test_mkfs_single -Q -d  dup     -m  single
+test_mkfs_single -Q -d  dup     -m  dup
+test_mkfs_single -Q -d  dup     -m  dup     --mixed
+
+# Profile doesn't really affect quota, skip them to save some time
+
 test_mkfs_multi
 test_mkfs_multi   -d  single  -m  single
 test_mkfs_multi   -d  single  -m  single  --mixed
-- 
2.25.0

[PATCH v4 11/11] btrfs-progs: test/mkfs: Add test case for --rootdir and --quota

[Marcos Paulo de Souza]

From: Qu Wenruo <wqu@suse.com>

Nothing interesting, since such combination can be handled easily by
qgroup-verify.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 .../mkfs-tests/018-rootdir-with-quota/test.sh | 51 +++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100755 tests/mkfs-tests/018-rootdir-with-quota/test.sh

diff --git a/tests/mkfs-tests/018-rootdir-with-quota/test.sh b/tests/mkfs-tests/018-rootdir-with-quota/test.sh
new file mode 100755
index 00000000..97e5f592
--- /dev/null
+++ b/tests/mkfs-tests/018-rootdir-with-quota/test.sh
@@ -0,0 +1,51 @@
+#!/bin/bash
+# Check if runtime feature quota can handle rootdir
+#
+
+source "$TOP/tests/common"
+
+check_prereq mkfs.btrfs
+check_prereq btrfs
+
+setup_root_helper		# For mknod
+prepare_test_dev 1G		# make it large since we will fill the fs
+
+# mknod can create FIFO/CHAR/BLOCK file but not SOCK.
+# No neat tool to create socket file, unless using python or similar.
+# So no SOCK is tested here
+check_global_prereq mknod
+check_global_prereq dd
+
+tmp=$(mktemp -d --tmpdir btrfs-progs-mkfs.rootdirXXXXXXX)
+
+run_check mkdir "$tmp/dir"
+run_check mkdir -p "$tmp/dir/in/dir"
+
+# More dir, there is no good way to pump metadata since we have no trigger
+# to enable/disable inline extent data, so here create enough dirs to bump
+# metadata
+run_check mkdir "$tmp/a_lot_of_dirs"
+for i in $(seq -w 0 8192); do
+	run_check mkdir "$tmp/a_lot_of_dirs/dir_$i"
+done
+
+# Then some data
+run_check dd if=/dev/zero bs=1M count=1 of="$tmp/1M"
+run_check dd if=/dev/zero bs=2M count=1 of="$tmp/2M"
+run_check dd if=/dev/zero bs=4M count=1 of="$tmp/4M"
+run_check dd if=/dev/zero bs=8M count=1 of="$tmp/8M"
+
+run_check dd if=/dev/zero bs=1K count=1 of="$tmp/1K"
+run_check dd if=/dev/zero bs=2K count=1 of="$tmp/2K"
+run_check dd if=/dev/zero bs=4K count=1 of="$tmp/4K"
+run_check dd if=/dev/zero bs=8K count=1 of="$tmp/8K"
+
+run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f -r "$tmp" -Q "$TEST_DEV"
+
+rm -rf -- "$tmp"
+
+# Normal check already includes quota check
+run_check $SUDO_HELPER "$TOP/btrfs" check "$TEST_DEV"
+
+# Just in case
+run_check $SUDO_HELPER "$TOP/btrfs" check --qgroup-report "$TEST_DEV"
-- 
2.25.0

Re: [PATCH v4 00/11] btrfs-progs: mkfs: Quota support through -Q|--quota

[David Sterba]

On Wed, Mar 18, 2020 at 05:21:37PM -0300, Marcos Paulo de Souza wrote:
> From: Marcos Paulo de Souza <mpdesouza@suse.com>
> 
> Hi guys,
> 
> This if the forth version of this patchset. The last submission of these patches
> was in 2018[1]. This version is based on top on the current devel branch, with
> minor cleanups, minor conflicts and only a real fix in patch 0008. I would like
> to ask you guys to review these patches, since v3 didn't receive any feedback at
> the time.
> 
> I only added my SoB in three patches, which were those were I needed a manual
> intervention, or a specific fix as I mentioned.
> 
> Thanks for your review,
>   Marcos
> 
> Original cover letter for Wenruo:
> This patchset adds quota support, which means the result fs will have
> quota enabled by default, and its accounting is already consistent, no
> manually rescan or quota enable is needed.
> 
> The overall design of such support is:
> 1) Create needed tree
>    Both btrfs_root and real root item and tree root leaf.
>    For this, a new infrastructure, btrfs_create_tree(), is added for
>    this.
> 
> 2) Fill quota root with basic skeleton
>    Only 3 items are really needed
>    a) global quota status item
>    b) quota info for specified qgroup
>    c) quota limit for specified qgroup
> 
>    Currently we insert all qgroup items for all existing file trees.
>    If we're going to support extra subvolume at mkfs time, just pass the
>    subvolume id into insert_qgroup_items().
> 
>    The content doesn't matter at all.
> 
> 3) Repair qgroups using infrastructure from qgroup-verify
>    In fact, qgroup repair is just offline rescan.
>    Although the original qgroup-verify infrastructure is mostly noisy,
>    modify it a little to make it silent to function as offline quota
>    rescan.
> 
> And such support is mainly designed for developers and QA guys.
> 
> As to enable quota, before we must normally mount the fs, enable quota
> (and rescan if needed).
> This ioctl based procedure is not common, and fstests doesn't provide
> such support.
> (Not to mention sometimes rescan itself can be buggy)
> 
> There are several attempts to make fstests to support it, but due to
> different reasons, all these attempts failed.
> 
> To make it easier to test all existing test cases with btrfs quota
> enabled, the current best method is to support quota at mkfs time, and
> here comes the patchset.
> 
> [1]: https://lore.kernel.org/linux-btrfs/20180807081938.21348-1-wqu@suse.com/T/#m107735cecbf4729b599e6e4eee0a54802909b30d
> 
> Qu Wenruo (11):
>   btrfs-progs: qgroup-verify: Avoid NULL pointer dereference for later
>     silent qgroup repair
>   btrfs-progs: qgroup-verify: Also repair qgroup status version
>   btrfs-progs: qgroup-verify: Use fs_info->readonly to check if we
>     should repair qgroups
>   btrfs-progs: qgroup-verify: Move qgroup classification out of
>     report_qgroups
>   btrfs-progs: qgroup-verify: Allow repair_qgroups function to do silent
>     repair
>   btrfs-progs: ctree: Introduce function to create an empty tree
>   btrfs-progs: mkfs: Introduce function to insert qgroup info and limit
>     items
>   btrfs-progs: mkfs: Introduce function to setup quota root and rescan

I've applied the above with some fixes to devel.

>   btrfs-progs: mkfs: Introduce mkfs time quota support
>   btrfs-progs: test/mkfs: Add test case for -Q|--quota option
>   btrfs-progs: test/mkfs: Add test case for --rootdir and --quota

The option name needs to be -R as used to be in V2 of Qu's original
patchset. I don't know why this got changed to the single purpose -Q but
-R will be used to specifiy runtime options, similar to what -O does
now.