Re: [PATCH] Btrfs: send, skip backreference walking for extents with many references

From: Zygo Blaxell <ce3g8jdj@umail.furryterror.org>
To: Filipe Manana <fdmanana@kernel.org>
Cc: linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: [PATCH] Btrfs: send, skip backreference walking for extents with many references
Date: Wed, 27 Nov 2019 13:09:55 -0500	[thread overview]
Message-ID: <20191127180954.GE1046@hungrycats.org> (raw)
In-Reply-To: <CAL3q7H5RCgs53nBG9HKLMHqUXLM=xiUKTQi8dS8nvuvW9x5bDA@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 5006 bytes --]

On Wed, Nov 27, 2019 at 11:42:21AM +0000, Filipe Manana wrote:
> Walking the backreferences was always slow, and having a few hundred
> references for an extent is enough to notice a significant slowdown
> already.
> That affects the logical_ino ioctls, send, etc.
> 
> Worst, for logical_ino, is that we do the backreference walking while
> either holding a transaction (if one is currently running) or
> holding a semaphore (commit_root_sem) to prevent transaction commits
> from happening (to ensure consistency and that things don't disappear
> while we're using them).
> So for a slow backreference walking, that can hang a lot of other
> tasks that are trying to commit a transaction.
> 
> There was someone working on improving backreference walking (Mark),
> but I think he's not working on btrfs anymore, and I don't know what
> happened to that work (if it was ever finished, etc).

Writing a userspace backref walker to replace LOGICAL_INO has been on
my TODO list for a while, but I had to go after the kernel crashing
bugs first.  Now the top of the kernel crashing bug list is LOGICAL_INO
too, so I guess there is no escape.

I'd love to help with the kernel implementation of backref walking,
but it's used by everything in btrfs, and it's full of wonderful things
like log unwinding and negative reference counting that I know I don't
understand well enough to mess with.

> > Simultaneous LOGICAL_INO + balance correlates with these three failure
> > modes:
> >
> >         https://www.spinics.net/lists/linux-btrfs/msg89425.html
> >
> >         https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg88892.html
> 
> For the bug involving btrfs_search_old_slot(), try the following fix:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=efad8a853ad2057f96664328a0d327a05ce39c76
> 
> Both reports where made before that fix was posted/merged.

5.3.13 already contains that commit (backported as
f59d80e2f12b695d38e646f00de1e46e123dfcc9 in 5.3.4), and:

	[58446.332688][ T4514] ------------[ cut here ]------------
	[58446.336367][ T4514] kernel BUG at fs/btrfs/ctree.c:1222!
	[58446.339993][ T4514] invalid opcode: 0000 [#1] SMP PTI
	[58446.343408][ T4514] CPU: 3 PID: 4514 Comm: crawl_35277 Not tainted 5.3.13-zb64-c0eaf2b82f8c+ #1
	[58446.348872][ T4514] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
	[58446.354272][ T4514] RIP: 0010:__tree_mod_log_rewind+0x235/0x240
	[58446.356393][ T4514] Code: 45 31 c0 4c 89 e7 48 89 d6 48 c1 e6 05 48 8d 74 32 65 ba 19 00 00 00 e8 59 d1 04 00 e9 89 fe ff ff 49 63 57 2c e9 18 ff ff ff <0f> 0b 0f 0b 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 83 c2 01 48 63
	[58446.361072][ T4514] RSP: 0000:ffffa34340e1b830 EFLAGS: 00010293
	[58446.364072][ T4514] RAX: 000003ff696b4000 RBX: 000000000000000f RCX: ffff981b2bf7db00
	[58446.366818][ T4514] RDX: 0000000000000000 RSI: 000000000000007e RDI: ffff981b2bf7d300
	[58446.368673][ T4514] RBP: ffffa34340e1b860 R08: ffffa34340e1b7d8 R09: ffffa34340e1b7e0
	[58446.371392][ T4514] R10: 000000000008b74a R11: 0000000000000000 R12: ffff981c62b2e4a0
	[58446.375829][ T4514] R13: ffff981b2bf7d300 R14: 0000000000000006 R15: ffff981b2bf7db00
	[58446.378642][ T4514] FS:  00007fccbd589700(0000) GS:ffff981cf6800000(0000) knlGS:0000000000000000
	[58446.380289][ T4514] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
	[58446.381567][ T4514] CR2: 00007fcc447d5000 CR3: 0000000205248003 CR4: 00000000001606e0
	[58446.383172][ T4514] Call Trace:
	[58446.383835][ T4514]  btrfs_search_old_slot+0x14a/0x7e0
	[58446.384850][ T4514]  resolve_indirect_refs+0x1f0/0x9a0
	[58446.385864][ T4514]  find_parent_nodes+0x427/0x1330
	[58446.386849][ T4514]  btrfs_find_all_roots_safe+0xc1/0x130
	[58446.387924][ T4514]  ? btrfs_inode_flags_to_xflags+0x50/0x50
	[58446.389055][ T4514]  iterate_extent_inodes+0x134/0x390
	[58446.390078][ T4514]  ? _raw_spin_unlock+0x27/0x40
	[58446.391008][ T4514]  iterate_inodes_from_logical+0x9c/0xd0
	[58446.392090][ T4514]  ? iterate_inodes_from_logical+0x9c/0xd0
	[58446.393245][ T4514]  ? btrfs_inode_flags_to_xflags+0x50/0x50
	[58446.394409][ T4514]  btrfs_ioctl_logical_to_ino+0x10e/0x1b0
	[58446.395554][ T4514]  btrfs_ioctl+0x1417/0x2c30
	[58446.396459][ T4514]  ? kvm_sched_clock_read+0x18/0x30
	[58446.397462][ T4514]  ? __lock_acquire+0x4b2/0x1c00
	[58446.398414][ T4514]  ? retint_kernel+0x10/0x10
	[58446.399329][ T4514]  ? kvm_sched_clock_read+0x18/0x30
	[58446.400357][ T4514]  ? sched_clock+0x9/0x10
	[58446.401219][ T4514]  do_vfs_ioctl+0xa9/0x6f0
	[58446.402099][ T4514]  ? do_vfs_ioctl+0xa9/0x6f0
	[58446.403025][ T4514]  ? __fget+0x11e/0x200
	[58446.403863][ T4514]  ksys_ioctl+0x67/0x90
	[58446.404704][ T4514]  __x64_sys_ioctl+0x1a/0x20
	[58446.405619][ T4514]  do_syscall_64+0x65/0x1b0
	[58446.406529][ T4514]  entry_SYSCALL_64_after_hwframe+0x49/0xbe

I was finally able to hit this bug on a 5.3.x kernel last night.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]