Re: [PATCH] btrfs: scrub: per-device bandwidth control

[David Sterba <dsterba@suse.cz>]

On Wed, May 19, 2021 at 05:20:50PM +0100, Graham Cobb wrote:
> On 19/05/2021 16:32, Johannes Thumshirn wrote:
> > On 19/05/2021 16:28, David Sterba wrote:
> >> On Wed, May 19, 2021 at 06:53:54AM +0000, Johannes Thumshirn wrote:
> >>> On 18/05/2021 16:52, David Sterba wrote:
> >>> I wonder if this interface would make sense for limiting balance
> >>> bandwidth as well?
> >>
> >> Balance is not contained to one device, so this makes the scrub case
> >> easy. For balance there are data and metadata involved, both read and
> >> write accross several threads so this is really something that the
> >> cgroups io controler is supposed to do.
> >>
> > 
> > For a user initiated balance a cgroups io controller would work well, yes.
> 
> Hmmm. I might give this a try. On my main mail server balance takes a
> long time and a lot of IO, which is why I created my "balance_slowly"
> script which shuts down mail (and some other services) then runs balance
> for 20 mins, then cancels the balance and allows mail to run for 10
> minutes, then resumes the balance for 20 mins, etc. Using this each
> month, a balance takes over 24 hours but at least the only problem is
> short mail delays for 1 day a month, not timeouts, users seeing mail
> error reports, etc.
> 
> Before I did this, the impact was horrible: btrfs spent all its time
> doing backref searches and any process which touched the filesystem (for
> example to deliver a tiny email) could be stuck for over an hour.
> 
> I am wondering whether the cgroups io controller would help, or whether
> it would cause a priority inversion because the backrefs couldn't do the
> IO they needed so the delays to other processes locked out would get
> even **longer**. Any thoughts?

Do you do a full balance? On a mail server where files get created and
deleted the space could become quite fragmented so a balance from time
to time would make the space more compact. You can also start balance in
smaller batches eg. using the limit=N filter.

I haven't fully tested the cgroup io limiting, no success setting it up
with raw cgroups, but the systemd unit files have support for that so
maybe I'm doing it the wrong way.

The priority inversion could be an issue, relocation needs to commit the
changes so in general the metadata operations should not be throttled.