All of lore.kernel.org
 help / color / mirror / Atom feed
From: Josef Bacik <josef@toxicpanda.com>
To: linux-btrfs@vger.kernel.org, kernel-team@fb.com
Subject: [PATCH 00/13] Throttle delayed refs based on time
Date: Fri, 13 Mar 2020 17:23:17 -0400	[thread overview]
Message-ID: <20200313212330.149024-1-josef@toxicpanda.com> (raw)

Zygo reported a problem on IRC where he was seeing multi-hour long latencies
with his test rig with transaction commits.  This turned out to be because his
test rig runs rsync, balance, snapshot create and delete, dedup, a infinite loop
of mkdir/rmdirs, and I'm sure some other horrors I'm forgetting.

When I added the delayed refs reserve, I assumed that the space pressure
generated by generating a lot of delayed refs would result in transactions being
ended if they needed to be, and thus we no longer needed to throttle delayed
refs based on time.

This assumption was wrong, because in Zygo's case he has a multi terabyte file
system, so overcommit allows him to generate as many delayed refs as he wants.
This meant that he would need to run hundreds of thousands of delayed refs at
commit time.  To make matters worse, we didn't have a way to stop people from
generating more delayed refs, so the transaction commit could be held open
indefinitely by balance and snapshot delete.  This is how we were getting
transaction commits happening every few hours.

The solution to this problem is to bring back the time based delayed ref
flushing, and then add the ability for people to throttle themselves on that.

Balance and truncate already had this ability, it only needed to be added to
snapshot delete.

I've also added back the async delayed ref flushing, and I've added code to help
throttle people when we're generating delayed refs too fast for the system to
keep up with them.

This whole patch queue has been running in some form or another on Zygo's awful
test bed, and appears to be performing better than it was before I ripped out
the original code.  Thanks,

Josef


             reply	other threads:[~2020-03-13 21:23 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-13 21:23 Josef Bacik [this message]
2020-03-13 21:23 ` [PATCH 01/13] btrfs: use a stable rolling avg for delayed refs avg Josef Bacik
2020-03-13 21:23 ` [PATCH 02/13] btrfs: change btrfs_should_throttle_delayed_refs to a bool Josef Bacik
2020-03-13 21:23 ` [PATCH 03/13] btrfs: make btrfs_should_throttle_delayed_refs only check run time Josef Bacik
2020-03-13 21:23 ` [PATCH 04/13] btrfs: make should_end_transaction check time to run delayed refs Josef Bacik
2020-03-13 21:23 ` [PATCH 05/13] btrfs: squash should_end_transaction Josef Bacik
2020-03-13 21:23 ` [PATCH 06/13] btrfs: add a mode for delayed ref time based throttling Josef Bacik
2020-03-13 21:23 ` [PATCH 07/13] btrfs: kick off async delayed ref flushing if we are over time budget Josef Bacik
2020-04-09 13:11   ` Nikolay Borisov
2020-04-09 13:26   ` Nikolay Borisov
2020-03-13 21:23 ` [PATCH 08/13] btrfs: adjust the arguments for btrfs_should_throttle_delayed_refs Josef Bacik
2020-03-13 21:23 ` [PATCH 09/13] btrfs: throttle delayed refs based on time Josef Bacik
2020-03-13 21:23 ` [PATCH 10/13] btrfs: handle uncontrolled delayed ref generation Josef Bacik
2020-03-13 21:23 ` [PATCH 11/13] btrfs: check delayed refs while relocating Josef Bacik
2020-03-13 21:23 ` [PATCH 12/13] btrfs: throttle truncate for delayed ref generation Josef Bacik
2020-03-13 21:23 ` [PATCH 13/13] btrfs: throttle snapshot delete on delayed refs Josef Bacik

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200313212330.149024-1-josef@toxicpanda.com \
    --to=josef@toxicpanda.com \
    --cc=kernel-team@fb.com \
    --cc=linux-btrfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.