linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v2 0/5] btrfs: qgroup: address the performance penalty for subvolume dropping
@ 2021-12-07  1:16 Qu Wenruo
  2021-12-07  1:16 ` [PATCH v2 1/5] btrfs: sysfs: introduce qgroup global attribute groups Qu Wenruo
                   ` (4 more replies)
  0 siblings, 5 replies; 6+ messages in thread
From: Qu Wenruo @ 2021-12-07  1:16 UTC (permalink / raw)
  To: linux-btrfs

Btrfs qgroup has a long history of bringing huge performance penalty,
from subvolume dropping to balance.

Although we solved the problem for balance, but the subvolume dropping
problem is still unresolved, as we really need to do all the costly
backref for all the involved subtrees, or qgroup numbers will be
inconsistent.

But the performance penalty is sometimes too big, so big that it's
better just to disable qgroup, do the drop, then do the rescan.

This patchset will address the problem by introducing a user
configurable sysfs interface, to allow certain high subtree dropping to
mark qgroup inconsistent, and skip the whole accounting.

The following things are needed for this objective:

- New qgroups attributes

  Instead of plain qgroup kobjects, we need extra attributes like
  drop_subtree_threshold.

  This patchset will introduce two new attributes to the existing
  qgroups kobject:
  * qgroups_flags
    To indicate the qgroup status flags like ON, RESCAN, INCONSISTENT.

  * drop_subtree_threshold
    To show the subtree dropping level threshold.
    The default value is BTRFS_MAX_LEVEL (8), which means all subtree
    dropping will go through the qgroup accounting, while costly it will
    try to keep qgroup numbers as consistent as possible.

    Users can specify values like 3, meaning any subtree which is at
    level 3 or higher will mark qgroup inconsistent and skip all the
    costly accounting.

    NOTE: if a snapshot is create with tree root level 3, dropping the
    snapshot with drop_subtree_threshold 3 will not mark the qgroup
    inconsistent.
    Since the level threshold is for shared subtree node, not the
    snapshot root node.

    In the case of newly created snapshot, only its (root level - 1)
    tree blocks are shared subtrees.

    This only affects subvolume dropping.

- Skip qgroup accounting when the numbers are already inconsistent

  But still keeps the qgroup relationship correct, thus users can keep
  its qgroup organization while do the rescan later.


This sysfs interface needs user space tools to monitor and set the
values for each btrfs.
And it's also user space daemon's responsibility to save the
drop_subtree_threshold values.

As introducing a new on-disk format just for qgroup is a little
overkilled to an optional feature to me.

Currently the target user space tool is snapper, which by default
utilizes qgroups for its space-aware snapshots reclaim mechanism.

Changelog:
v2:
- Split /sys/fs/btrfs/<uuid>/qgroups/qgroup_flags into two
  entries

- Update the cover letter to explain the drop_subtree_threshold better

Qu Wenruo (5):
  btrfs: sysfs: introduce qgroup global attribute groups
  btrfs: introduce BTRFS_QGROUP_STATUS_FLAGS_MASK for later expansion
  btrfs: introduce BTRFS_QGROUP_RUNTIME_FLAG_CANCEL_RESCAN
  btrfs: introduce BTRFS_QGROUP_RUNTIME_FLAG_NO_ACCOUNTING to skip
    qgroup accounting
  btrfs: skip subtree scan if it's too high to avoid low stall in
    btrfs_commit_transaction()

 fs/btrfs/ctree.h                |   1 +
 fs/btrfs/disk-io.c              |   1 +
 fs/btrfs/qgroup.c               |  87 ++++++++++++++++++-------
 fs/btrfs/qgroup.h               |   3 +
 fs/btrfs/sysfs.c                | 112 ++++++++++++++++++++++++++++++--
 include/uapi/linux/btrfs_tree.h |   4 ++
 6 files changed, 182 insertions(+), 26 deletions(-)

-- 
2.34.1


^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-12-07  1:17 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-07  1:16 [PATCH v2 0/5] btrfs: qgroup: address the performance penalty for subvolume dropping Qu Wenruo
2021-12-07  1:16 ` [PATCH v2 1/5] btrfs: sysfs: introduce qgroup global attribute groups Qu Wenruo
2021-12-07  1:16 ` [PATCH v2 2/5] btrfs: introduce BTRFS_QGROUP_STATUS_FLAGS_MASK for later expansion Qu Wenruo
2021-12-07  1:16 ` [PATCH v2 3/5] btrfs: introduce BTRFS_QGROUP_RUNTIME_FLAG_CANCEL_RESCAN Qu Wenruo
2021-12-07  1:16 ` [PATCH v2 4/5] btrfs: introduce BTRFS_QGROUP_RUNTIME_FLAG_NO_ACCOUNTING to skip qgroup accounting Qu Wenruo
2021-12-07  1:16 ` [PATCH v2 5/5] btrfs: skip subtree scan if it's too high to avoid low stall in btrfs_commit_transaction() Qu Wenruo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).