Linux-BTRFS Archive on
 help / color / Atom feed
From: Qu Wenruo <>
To: Josef Bacik <>
Cc: Qu Wenruo <>,
Subject: Re: [PATCH v4 0/7] btrfs: qgroup: Delay subtree scan to reduce overhead
Date: Wed, 16 Jan 2019 09:29:29 +0800
Message-ID: <> (raw)
In-Reply-To: <>

[-- Attachment #1.1: Type: text/plain, Size: 3749 bytes --]

On 2019/1/16 上午9:15, Josef Bacik wrote:
> On Wed, Jan 16, 2019 at 09:07:26AM +0800, Qu Wenruo wrote:
>> On 2019/1/16 上午8:31, Qu Wenruo wrote:
>>> On 2019/1/16 上午1:26, Josef Bacik wrote:
>>>> On Tue, Jan 15, 2019 at 04:15:57PM +0800, Qu Wenruo wrote:
>>>>> This patchset can be fetched from github:
>>>>> Which is based on v5.0-rc1.
>>>> I've been testing these patches hoping to get rid of the qgroup deadlock that
>>>> these patches are supposed to fix, but instead they make the box completely
>>>> unusable with 100% cpu usage for minutes at a time at every transaction commit.
>>> I'm afraid it's related to the v5.0-rc1 base, not the patchset itself.
>>> Just try to balance metadata with 16 snapshots, you'll see btrfs bumping
>>> its generation like crazy, no matter if quota is enabled or not.
>>> And since btrfs is committing transaction like crazy, no wonder it will
>>> do tons of qgroup accounting.
>>> My bisect leads to commit 64403612b73a94bc7b02cf8ca126e3b8ced6e921
>>> btrfs: rework btrfs_check_space_for_delayed_refs.
>> Furthermore, you could try this RFC test case to see the problem.
>> It would only take around 100s for v4.20 but over 500 for v5.0-rc1 with
>> tons of unnecessary transaction committed for nothing, no quota enabled.
>> So I'm afraid that commit is blocking my qgroup patchset.
> I've fixed the balance problem, it took 2 seconds to figure out, I'm just
> waiting on xfstests to finish running.
> And your patch making things worse has nothing to do with that problem.  Our
> test doesn't run balance, so the issue you reported has nothing to do with the
> fact that your patch makes our boxes unusable with qgroups on.
> The problem is with your deadlock avoidence patch we're now making a lot more
> dirty extents to run in the critical section of the transaction commit.  Also
> because we're no longer pre-fetching the old roots, we're doing the old roots
> and new roots lookup inside the critical section, so now each dirty extent takes
> 2x as long.  With my basic test it was taking 5 minutes to do the qgroup
> accounting, which keeps the box from booting essentially.
> Without your patch it's still awful because btrfs-cleaner just sits there at
> 100% while deleting snapshots, but at least it's not making the whole system
> stop running while it does all that work in the transaction commit.
> And if you had done the proper root cause analysis you would have noticed that
> we're taking tree locks in the find_parent_nodes() case when we're searching the
> commit root, something we shouldn't be doing.  So all that really needs to be
> done is to check if (!path->skip_locking) btrfs_tree_read_lock(eb); in those
> cases and the deadlock goes away.  Because no matter what we shouldn't be taking
> locks when we're not given a trans in the backref lookup code.

That indeed looks much better than my current solution.

Although I'm not 100% sure for cases like tree blocks shared between
commit and current root (tree block not modified yet).

I'll definitely invest more time to try to fix this bug.


>  So the fact that
> we were doing just that and thus deadlocking should have been a red flag.
> I will be sending these patches in the morning, once all of the various testing
> that should take place on patches is complete.  The balance patches you have for
> qgroups don't appear to be a problem, but that deadlock one is bogus and needs
> to be dropped.  Thanks,
> Josef

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

  reply index

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-15  8:15 Qu Wenruo
2019-01-15  8:15 ` [PATCH v4 1/7] btrfs: qgroup: Move reserved data account from btrfs_delayed_ref_head to btrfs_qgroup_extent_record Qu Wenruo
2019-01-15  8:15 ` [PATCH v4 2/7] btrfs: qgroup: Don't trigger backref walk at delayed ref insert time Qu Wenruo
2019-01-15  8:16 ` [PATCH v4 3/7] btrfs: relocation: Delay reloc tree deletion after merge_reloc_roots() Qu Wenruo
2019-01-22 16:32   ` David Sterba
2019-01-23  6:01     ` Qu Wenruo
2019-01-15  8:16 ` [PATCH v4 4/7] btrfs: qgroup: Refactor btrfs_qgroup_trace_subtree_swap() Qu Wenruo
2019-01-22 16:38   ` David Sterba
2019-01-15  8:16 ` [PATCH v4 5/7] btrfs: qgroup: Introduce per-root swapped blocks infrastructure Qu Wenruo
2019-01-22 16:55   ` David Sterba
2019-01-22 23:07     ` Qu Wenruo
2019-01-24 18:44       ` David Sterba
2019-01-15  8:16 ` [PATCH v4 6/7] btrfs: qgroup: Use delayed subtree rescan for balance Qu Wenruo
2019-01-22 17:05   ` David Sterba
2019-01-15  8:16 ` [PATCH v4 7/7] btrfs: qgroup: Cleanup old subtree swap code Qu Wenruo
2019-01-15 17:26 ` [PATCH v4 0/7] btrfs: qgroup: Delay subtree scan to reduce overhead Josef Bacik
2019-01-16  0:31   ` Qu Wenruo
2019-01-16  1:07     ` Qu Wenruo
2019-01-16  1:15       ` Josef Bacik
2019-01-16  1:29         ` Qu Wenruo [this message]
2019-01-16  1:34           ` Josef Bacik
2019-01-16  1:36             ` Qu Wenruo
2019-01-22 16:28 ` David Sterba
2019-01-23  5:43   ` Qu Wenruo

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-BTRFS Archive on

Archives are clonable:
	git clone --mirror linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ \
	public-inbox-index linux-btrfs

Newsgroup available over NNTP:

AGPL code for this site: git clone public-inbox