Re: autodefrag causing freezes under heavy writes?

From: Yan Li <elliot.li.tech@gmail.com>
To: dsterba@suse.cz, Yan Li <elliot.li.tech@gmail.com>,
	linux-btrfs@vger.kernel.org
Subject: Re: autodefrag causing freezes under heavy writes?
Date: Fri, 9 Jul 2021 12:10:00 -0700	[thread overview]
Message-ID: <CALc-jWzcGL0Jsb+K6ziokFMmJwLKy-ibDC_h-xcTBu7fh4NeXQ@mail.gmail.com> (raw)
In-Reply-To: <20210707190032.GT2610@twin.jikos.cz>

On Wed, Jul 7, 2021 at 12:03 PM David Sterba <dsterba@suse.cz> wrote:
> On Mon, Jul 05, 2021 at 08:56:23PM -0700, Yan Li wrote:
> > I found that when I added the autodefrag mount option, the system
> > would freeze under heavy write workload for a long time before the
>
> Do you have an estimate for 'long time' ? Like human percievable
> "seconds" or like 5 seconds and more.

It would freeze for minutes, during which the GUI was totally
unresponsive. After a few minutes, presumably after the dd was
finished, the machine would resume normal operation.

The full dmesg is here: https://pastebin.com/TwChmFmC

You could see that the blocked I/O messed up journald's output so that
the message towards the end were not even ordered by the timestamp.

> The autodefrag can cause problems like this, yes, but it depends on
> other factors too. Autodefrag can read additional pages from disk in
> case they aren't contiguous and then writes them (in a small cluster)
> together. You're using compression, so this may add a slightly more
> delay before the data are written. On the default level it should be
> unnoticeable and you mention that's on a Ryzen 5 so I'd rule that out.

The workload was just a simple:
dd if=/dev/urandom of=test_data bs=1M count=2000
so there should be no reason for it to block for such a long time.
And, yes, it's was a new workstation, and works flawlessly on much
heavier workloads when autodefrag was removed.

> IIRC autodefrag can help some workloads but may hurt others so if it's
> making things worse you, then drop it. It helps when seeks are expensive
> ie. on rotational disks but you use SSD so it should not be necessary.

I was advised to add it since I'm running VirtualBox VMs out of these
btrfs. But autodefrag made *everything* worse on this filesystem. It's
weird.

> If you'd still like to debug it, please take a snapshot of all process
> stacks at the time the hang happens.

This was from before I removed the autodefrag option.
https://pastebin.com/TwChmFmC

It's a production system so I can't reboot it every day. I can try to
add autodefrag back a few days later and retest.

Thanks!

-- 
Yan