Re: Debugging abysmal write performance with 100% cpu kworker/u16:X+flush-btrfs-2

[Chris Murphy <lists@colorremedies.com>]

On Mon, Jul 27, 2020 at 11:17 AM Hans van Kranenburg <hans@knorrie.org> wrote:
>
> https://syrinx.knorrie.org/~knorrie/btrfs/keep/2020-07-25-find_free_extent.txt
>
> From the timestamps you can see how long this takes. It's not much that
> gets done per second.
>
> The spin lock part must be spin_lock(&space_info->lock) because that's
> the only one in find_free_extent.
>
> So, what I think is that, like I mentioned on saturday already,
> find_free_extent is probably doing things in a really inefficient way,
> scanning around for a small single free space gap all the time in a
> really expensive way, and doing that over again for each tiny metadata
> block or file that needs to be placed somewhere (also notice the
> empty_size=0), instead of just throwing all of it on disk after each
> other, when it's otherwise slowing down everyone.

What are the latencies for the iscsi device while this is happening?
Does your trace have the ability to distinguish between inefficiency
in find_free_extent versus waiting on IO latency?

However, this hypothesis could be tested by completely balancing all
metadata block groups, then trying to reproduce this workload. If it's
really this expensive to do metadata insertions, eliminating the need
for insertions should show up as a remarkable performance improvement
that degrades as metadata block groups become fragmented again.


> What I *can* try is switch to the ssd_spread option, to force it to do
> much more yolo allocation, but I'm afraid this will result in a sudden
> blast of metadata block groups getting allocated. Or, maybe try it for a
> minute or so and compare the trace pipe output...

OK or this.


-- 
Chris Murphy