[PATCH 00/14] Qgroup metadata reservation rework

[Qu Wenruo <wqu@suse.com>]

[Overall]
The previous rework on qgroup reservation system put a lot of effort on
data, which works quite fine.

But it takes less focus on metadata reservation, causing some problem
like metadata reservation underflow and noisy kernel warning.

This patchset will try to address the remaining problem of metadata
reservation.

The idea of new qgroup metadata reservation is to use 2 types of
metadata reservation:
1) Per-transaction reservation
   Life span will be inside a transaction. Will be freed at transaction
   commit time.

2) Preallocated reservation
   For case where we reserve space before starting a transaction.
   Operation like dealloc and delayed-inode/item belongs to this type.

   This works similar to block_rsv, its reservation can be
   reserved/released at any timing caller like.

   The only point to notice is, if preallocated reservation is used and
   finished without problem, it should be converted to per-transaction
   type instead of just freeing.
   This is to co-operate with qgroup update at commit time.

For preallocated type, this patch will integrate them into inode_rsv
mechanism reworked by Josef, and delayed-inode/item reservation.


[Problem: Over-reserve]
Currently the patchset addresses metadata underflow quite well, but
due to the over-reserve nature of btrfs and highly bounded to inode_rsv,
qgroup metadata reservation also tends to be over-reserved.

This is especially obvious for small limit.
For 128M limit, it's will only be able to write about 70M before hitting
quota limit.
Although for larger limit, like 5G limit, it can reach 4.5G or more
before hitting limit.

Such over-reserved behavior can lead to some problem with existing test
cases (where limit is normally less than 20M).

While it's also possible to be addressed by use more accurate space other
than max estimations.

For example, to calculate metadata needed for delalloc, we use
btrfs_calc_trans_metadata_size(), which always try to reserve space for
CoW a full-height tree, and will also include csum size.
Both calculate is way over-killed for qgroup metadata reservation.

[Patch structure]
The patch is consist of 2 main parts:
1) Type based qgroup reservation
   The original patchset is sent several months ago.
   Nothing is modified at all, just rebased. And not conflict at all.

   It's from patch 1 to patch 6.

2) Split meta qgroup reservation into per-trans and prealloc sub types
   The real work to address metadata underflow.
   Due to the over-reserve problem, this part is still in RFC state.
   But the framework should mostly be fine, only needs extra fine-tuning
   to get more accurate qgroup rsv to avoid too early limit.

   It's from patch 7 to 14.

Qu Wenruo (14):
  btrfs: qgroup: Skeleton to support separate qgroup reservation type
  btrfs: qgroup: Introduce helpers to update and access new qgroup rsv
  btrfs: qgroup: Make qgroup_reserve and its callers to use separate
    reservation type
  btrfs: qgroup: Fix wrong qgroup reservation update for relationship
    modification
  btrfs: qgroup: Update trace events to use new separate rsv types
  btrfs: qgroup: Cleanup the remaining old reservation counters
  btrfs: qgroup: Split meta rsv type into meta_prealloc and
    meta_pertrans
  btrfs: qgroup: Don't use root->qgroup_meta_rsv for qgroup
  btrfs: qgroup: Introduce function to convert META_PREALLOC into
    META_PERTRANS
  btrfs: qgroup: Use separate meta reservation type for delalloc
  btrfs: delayed-inode: Use new qgroup meta rsv for delayed inode and
    item
  btrfs: qgroup: Use root->qgroup_meta_rsv_* to record qgroup meta
    reserved space
  btrfs: qgroup: Update trace events for metadata reservation
  Revert "btrfs: qgroups: Retry after commit on getting EDQUOT"

 fs/btrfs/ctree.h             |  15 +-
 fs/btrfs/delayed-inode.c     |  50 +++++--
 fs/btrfs/disk-io.c           |   2 +-
 fs/btrfs/extent-tree.c       |  49 +++---
 fs/btrfs/file.c              |  15 +-
 fs/btrfs/free-space-cache.c  |   2 +-
 fs/btrfs/inode-map.c         |   4 +-
 fs/btrfs/inode.c             |  27 ++--
 fs/btrfs/ioctl.c             |  10 +-
 fs/btrfs/ordered-data.c      |   2 +-
 fs/btrfs/qgroup.c            | 350 ++++++++++++++++++++++++++++++++-----------
 fs/btrfs/qgroup.h            | 102 ++++++++++++-
 fs/btrfs/relocation.c        |   9 +-
 fs/btrfs/transaction.c       |   8 +-
 include/trace/events/btrfs.h |  73 ++++++++-
 15 files changed, 537 insertions(+), 181 deletions(-)

-- 
2.15.1