Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
From: Qu Wenruo <wqu@suse.de>
To: Josef Bacik <josef@toxicpanda.com>, Qu Wenruo <quwenruo.btrfs@gmx.com>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: [PATCH v4 0/7] btrfs: qgroup: Delay subtree scan to reduce overhead
Date: Wed, 16 Jan 2019 09:36:25 +0800
Message-ID: <8ff38dbb-7edb-b83a-9027-53beb52df29e@suse.de> (raw)
In-Reply-To: <20190116013408.adq7rny5n3p2sa2q@macbook-pro-91.dhcp.thefacebook.com>

[-- Attachment #1.1: Type: text/plain, Size: 4056 bytes --]



On 2019/1/16 上午9:34, Josef Bacik wrote:
> On Wed, Jan 16, 2019 at 09:29:29AM +0800, Qu Wenruo wrote:
>>
>>
>> On 2019/1/16 上午9:15, Josef Bacik wrote:
>>> On Wed, Jan 16, 2019 at 09:07:26AM +0800, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2019/1/16 上午8:31, Qu Wenruo wrote:
>>>>>
>>>>>
>>>>> On 2019/1/16 上午1:26, Josef Bacik wrote:
>>>>>> On Tue, Jan 15, 2019 at 04:15:57PM +0800, Qu Wenruo wrote:
>>>>>>> This patchset can be fetched from github:
>>>>>>> https://github.com/adam900710/linux/tree/qgroup_delayed_subtree
>>>>>>>
>>>>>>> Which is based on v5.0-rc1.
>>>>>>>
>>>>>>
>>>>>> I've been testing these patches hoping to get rid of the qgroup deadlock that
>>>>>> these patches are supposed to fix, but instead they make the box completely
>>>>>> unusable with 100% cpu usage for minutes at a time at every transaction commit.
>>>>>
>>>>> I'm afraid it's related to the v5.0-rc1 base, not the patchset itself.
>>>>>
>>>>> Just try to balance metadata with 16 snapshots, you'll see btrfs bumping
>>>>> its generation like crazy, no matter if quota is enabled or not.
>>>>>
>>>>> And since btrfs is committing transaction like crazy, no wonder it will
>>>>> do tons of qgroup accounting.
>>>>>
>>>>> My bisect leads to commit 64403612b73a94bc7b02cf8ca126e3b8ced6e921
>>>>> btrfs: rework btrfs_check_space_for_delayed_refs.
>>>>
>>>> Furthermore, you could try this RFC test case to see the problem.
>>>> https://patchwork.kernel.org/patch/10761715/
>>>>
>>>> It would only take around 100s for v4.20 but over 500 for v5.0-rc1 with
>>>> tons of unnecessary transaction committed for nothing, no quota enabled.
>>>>
>>>> So I'm afraid that commit is blocking my qgroup patchset.
>>>>
>>>
>>> I've fixed the balance problem, it took 2 seconds to figure out, I'm just
>>> waiting on xfstests to finish running.
>>>
>>> And your patch making things worse has nothing to do with that problem.  Our
>>> test doesn't run balance, so the issue you reported has nothing to do with the
>>> fact that your patch makes our boxes unusable with qgroups on.
>>>
>>> The problem is with your deadlock avoidence patch we're now making a lot more
>>> dirty extents to run in the critical section of the transaction commit.  Also
>>> because we're no longer pre-fetching the old roots, we're doing the old roots
>>> and new roots lookup inside the critical section, so now each dirty extent takes
>>> 2x as long.  With my basic test it was taking 5 minutes to do the qgroup
>>> accounting, which keeps the box from booting essentially.
>>>
>>> Without your patch it's still awful because btrfs-cleaner just sits there at
>>> 100% while deleting snapshots, but at least it's not making the whole system
>>> stop running while it does all that work in the transaction commit.
>>>
>>> And if you had done the proper root cause analysis you would have noticed that
>>> we're taking tree locks in the find_parent_nodes() case when we're searching the
>>> commit root, something we shouldn't be doing.  So all that really needs to be
>>> done is to check if (!path->skip_locking) btrfs_tree_read_lock(eb); in those
>>> cases and the deadlock goes away.  Because no matter what we shouldn't be taking
>>> locks when we're not given a trans in the backref lookup code.
>>
>> That indeed looks much better than my current solution.
>>
>> Although I'm not 100% sure for cases like tree blocks shared between
>> commit and current root (tree block not modified yet).
> 
> Doesn't matter, the block can't be modified so we don't need the locking,  If
> the current root needs to change it then it cow's it and messes with the new eb,
> not the one attached to the commit root.
> 
>>
>> I'll definitely invest more time to try to fix this bug.
>>
> 
> Don't worry about it, I'm already running the patch through my tests, I'll send
> them in the morning when the tests are finished.  Thanks,

I can't be more happier dropping that deadlock fix patch.

Thanks,
Qu

> 
> Josef
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

  reply index

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-15  8:15 Qu Wenruo
2019-01-15  8:15 ` [PATCH v4 1/7] btrfs: qgroup: Move reserved data account from btrfs_delayed_ref_head to btrfs_qgroup_extent_record Qu Wenruo
2019-01-15  8:15 ` [PATCH v4 2/7] btrfs: qgroup: Don't trigger backref walk at delayed ref insert time Qu Wenruo
2019-01-15  8:16 ` [PATCH v4 3/7] btrfs: relocation: Delay reloc tree deletion after merge_reloc_roots() Qu Wenruo
2019-01-22 16:32   ` David Sterba
2019-01-23  6:01     ` Qu Wenruo
2019-01-15  8:16 ` [PATCH v4 4/7] btrfs: qgroup: Refactor btrfs_qgroup_trace_subtree_swap() Qu Wenruo
2019-01-22 16:38   ` David Sterba
2019-01-15  8:16 ` [PATCH v4 5/7] btrfs: qgroup: Introduce per-root swapped blocks infrastructure Qu Wenruo
2019-01-22 16:55   ` David Sterba
2019-01-22 23:07     ` Qu Wenruo
2019-01-24 18:44       ` David Sterba
2019-01-15  8:16 ` [PATCH v4 6/7] btrfs: qgroup: Use delayed subtree rescan for balance Qu Wenruo
2019-01-22 17:05   ` David Sterba
2019-01-15  8:16 ` [PATCH v4 7/7] btrfs: qgroup: Cleanup old subtree swap code Qu Wenruo
2019-01-15 17:26 ` [PATCH v4 0/7] btrfs: qgroup: Delay subtree scan to reduce overhead Josef Bacik
2019-01-16  0:31   ` Qu Wenruo
2019-01-16  1:07     ` Qu Wenruo
2019-01-16  1:15       ` Josef Bacik
2019-01-16  1:29         ` Qu Wenruo
2019-01-16  1:34           ` Josef Bacik
2019-01-16  1:36             ` Qu Wenruo [this message]
2019-01-22 16:28 ` David Sterba
2019-01-23  5:43   ` Qu Wenruo

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8ff38dbb-7edb-b83a-9027-53beb52df29e@suse.de \
    --to=wqu@suse.de \
    --cc=josef@toxicpanda.com \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=quwenruo.btrfs@gmx.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org linux-btrfs@archiver.kernel.org
	public-inbox-index linux-btrfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/ public-inbox