* [PATCH v6 00/22] btrfs: async discard support
@ 2019-12-14 0:22 Dennis Zhou
2019-12-14 0:22 ` [PATCH 01/22] bitmap: genericize percpu bitmap region iterators Dennis Zhou
` (23 more replies)
0 siblings, 24 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Hello,
Dave reported a lockdep issue [1]. I'm a bit surprised as I can't repro
it, but it obviously is right. I believe I fixed the issue by moving the
fully trimmed check outside of the block_group lock. I mistakingly
thought the btrfs_block_group lock subsumed btrfs_free_space_ctl
tree_lock. This clearly isn't the case.
Changes in v6:
- Move the fully trimmed check outside of the block_group lock.
v5 is available here: [2].
This series is on top of btrfs-devel#misc-next 7ee98bb808e2 + [3] and
[4].
[1] https://lore.kernel.org/linux-btrfs/20191210140438.GU2734@twin.jikos.cz/
[2] https://lore.kernel.org/linux-btrfs/cover.1575919745.git.dennis@kernel.org/
[3] https://lore.kernel.org/linux-btrfs/d934383ea528d920a95b6107daad6023b516f0f4.1576109087.git.dennis@kernel.org/
[4] https://lore.kernel.org/linux-btrfs/20191209193846.18162-1-dennis@kernel.org/
Dennis Zhou (22):
bitmap: genericize percpu bitmap region iterators
btrfs: rename DISCARD opt to DISCARD_SYNC
btrfs: keep track of which extents have been discarded
btrfs: keep track of cleanliness of the bitmap
btrfs: add the beginning of async discard, discard workqueue
btrfs: handle empty block_group removal
btrfs: discard one region at a time in async discard
btrfs: add removal calls for sysfs debug/
btrfs: make UUID/debug have its own kobject
btrfs: add discard sysfs directory
btrfs: track discardable extents for async discard
btrfs: keep track of discardable_bytes
btrfs: calculate discard delay based on number of extents
btrfs: add bps discard rate limit
btrfs: limit max discard size for async discard
btrfs: make max async discard size tunable
btrfs: have multiple discard lists
btrfs: only keep track of data extents for async discard
btrfs: keep track of discard reuse stats
btrfs: add async discard header
btrfs: increase the metadata allowance for the free_space_cache
btrfs: make smaller extents more likely to go into bitmaps
fs/btrfs/Makefile | 2 +-
fs/btrfs/block-group.c | 87 ++++-
fs/btrfs/block-group.h | 30 ++
fs/btrfs/ctree.h | 52 ++-
fs/btrfs/discard.c | 684 ++++++++++++++++++++++++++++++++++++
fs/btrfs/discard.h | 42 +++
fs/btrfs/disk-io.c | 15 +-
fs/btrfs/extent-tree.c | 8 +-
fs/btrfs/free-space-cache.c | 611 +++++++++++++++++++++++++++-----
fs/btrfs/free-space-cache.h | 41 ++-
fs/btrfs/inode-map.c | 13 +-
fs/btrfs/inode.c | 2 +-
fs/btrfs/scrub.c | 7 +-
fs/btrfs/super.c | 39 +-
fs/btrfs/sysfs.c | 205 ++++++++++-
fs/btrfs/volumes.c | 7 +
include/linux/bitmap.h | 35 ++
mm/percpu.c | 61 +---
18 files changed, 1789 insertions(+), 152 deletions(-)
create mode 100644 fs/btrfs/discard.c
create mode 100644 fs/btrfs/discard.h
Thanks,
Dennis
^ permalink raw reply [flat|nested] 61+ messages in thread
* [PATCH 01/22] bitmap: genericize percpu bitmap region iterators
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 02/22] btrfs: rename DISCARD opt to DISCARD_SYNC Dennis Zhou
` (22 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Bitmaps are fairly popular for their space efficiency, but we don't have
generic iterators available. Make percpu's bitmap region iterators
available to everyone.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
include/linux/bitmap.h | 35 ++++++++++++++++++++++++
mm/percpu.c | 61 +++++++++++-------------------------------
2 files changed, 51 insertions(+), 45 deletions(-)
diff --git a/include/linux/bitmap.h b/include/linux/bitmap.h
index ff335b22f23c..cb63feb3cfbe 100644
--- a/include/linux/bitmap.h
+++ b/include/linux/bitmap.h
@@ -456,6 +456,41 @@ static inline int bitmap_parse(const char *buf, unsigned int buflen,
return __bitmap_parse(buf, buflen, 0, maskp, nmaskbits);
}
+static inline void bitmap_next_clear_region(unsigned long *bitmap,
+ unsigned int *rs, unsigned int *re,
+ unsigned int end)
+{
+ *rs = find_next_zero_bit(bitmap, end, *rs);
+ *re = find_next_bit(bitmap, end, *rs + 1);
+}
+
+static inline void bitmap_next_set_region(unsigned long *bitmap,
+ unsigned int *rs, unsigned int *re,
+ unsigned int end)
+{
+ *rs = find_next_bit(bitmap, end, *rs);
+ *re = find_next_zero_bit(bitmap, end, *rs + 1);
+}
+
+/*
+ * Bitmap region iterators. Iterates over the bitmap between [@start, @end).
+ * @rs and @re should be integer variables and will be set to start and end
+ * index of the current clear or set region.
+ */
+#define bitmap_for_each_clear_region(bitmap, rs, re, start, end) \
+ for ((rs) = (start), \
+ bitmap_next_clear_region((bitmap), &(rs), &(re), (end)); \
+ (rs) < (re); \
+ (rs) = (re) + 1, \
+ bitmap_next_clear_region((bitmap), &(rs), &(re), (end)))
+
+#define bitmap_for_each_set_region(bitmap, rs, re, start, end) \
+ for ((rs) = (start), \
+ bitmap_next_set_region((bitmap), &(rs), &(re), (end)); \
+ (rs) < (re); \
+ (rs) = (re) + 1, \
+ bitmap_next_set_region((bitmap), &(rs), &(re), (end)))
+
/**
* BITMAP_FROM_U64() - Represent u64 value in the format suitable for bitmap.
* @n: u64 value
diff --git a/mm/percpu.c b/mm/percpu.c
index 7e06a1e58720..e9844086b236 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -270,33 +270,6 @@ static unsigned long pcpu_chunk_addr(struct pcpu_chunk *chunk,
pcpu_unit_page_offset(cpu, page_idx);
}
-static void pcpu_next_unpop(unsigned long *bitmap, int *rs, int *re, int end)
-{
- *rs = find_next_zero_bit(bitmap, end, *rs);
- *re = find_next_bit(bitmap, end, *rs + 1);
-}
-
-static void pcpu_next_pop(unsigned long *bitmap, int *rs, int *re, int end)
-{
- *rs = find_next_bit(bitmap, end, *rs);
- *re = find_next_zero_bit(bitmap, end, *rs + 1);
-}
-
-/*
- * Bitmap region iterators. Iterates over the bitmap between
- * [@start, @end) in @chunk. @rs and @re should be integer variables
- * and will be set to start and end index of the current free region.
- */
-#define pcpu_for_each_unpop_region(bitmap, rs, re, start, end) \
- for ((rs) = (start), pcpu_next_unpop((bitmap), &(rs), &(re), (end)); \
- (rs) < (re); \
- (rs) = (re) + 1, pcpu_next_unpop((bitmap), &(rs), &(re), (end)))
-
-#define pcpu_for_each_pop_region(bitmap, rs, re, start, end) \
- for ((rs) = (start), pcpu_next_pop((bitmap), &(rs), &(re), (end)); \
- (rs) < (re); \
- (rs) = (re) + 1, pcpu_next_pop((bitmap), &(rs), &(re), (end)))
-
/*
* The following are helper functions to help access bitmaps and convert
* between bitmap offsets to address offsets.
@@ -732,9 +705,8 @@ static void pcpu_chunk_refresh_hint(struct pcpu_chunk *chunk, bool full_scan)
}
bits = 0;
- pcpu_for_each_md_free_region(chunk, bit_off, bits) {
+ pcpu_for_each_md_free_region(chunk, bit_off, bits)
pcpu_block_update(chunk_md, bit_off, bit_off + bits);
- }
}
/**
@@ -749,7 +721,7 @@ static void pcpu_block_refresh_hint(struct pcpu_chunk *chunk, int index)
{
struct pcpu_block_md *block = chunk->md_blocks + index;
unsigned long *alloc_map = pcpu_index_alloc_map(chunk, index);
- int rs, re, start; /* region start, region end */
+ unsigned int rs, re, start; /* region start, region end */
/* promote scan_hint to contig_hint */
if (block->scan_hint) {
@@ -765,10 +737,9 @@ static void pcpu_block_refresh_hint(struct pcpu_chunk *chunk, int index)
block->right_free = 0;
/* iterate over free areas and update the contig hints */
- pcpu_for_each_unpop_region(alloc_map, rs, re, start,
- PCPU_BITMAP_BLOCK_BITS) {
+ bitmap_for_each_clear_region(alloc_map, rs, re, start,
+ PCPU_BITMAP_BLOCK_BITS)
pcpu_block_update(block, rs, re);
- }
}
/**
@@ -1041,13 +1012,13 @@ static void pcpu_block_update_hint_free(struct pcpu_chunk *chunk, int bit_off,
static bool pcpu_is_populated(struct pcpu_chunk *chunk, int bit_off, int bits,
int *next_off)
{
- int page_start, page_end, rs, re;
+ unsigned int page_start, page_end, rs, re;
page_start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE);
page_end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE);
rs = page_start;
- pcpu_next_unpop(chunk->populated, &rs, &re, page_end);
+ bitmap_next_clear_region(chunk->populated, &rs, &re, page_end);
if (rs >= page_end)
return true;
@@ -1702,13 +1673,13 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved,
/* populate if not all pages are already there */
if (!is_atomic) {
- int page_start, page_end, rs, re;
+ unsigned int page_start, page_end, rs, re;
page_start = PFN_DOWN(off);
page_end = PFN_UP(off + size);
- pcpu_for_each_unpop_region(chunk->populated, rs, re,
- page_start, page_end) {
+ bitmap_for_each_clear_region(chunk->populated, rs, re,
+ page_start, page_end) {
WARN_ON(chunk->immutable);
ret = pcpu_populate_chunk(chunk, rs, re, pcpu_gfp);
@@ -1858,10 +1829,10 @@ static void pcpu_balance_workfn(struct work_struct *work)
spin_unlock_irq(&pcpu_lock);
list_for_each_entry_safe(chunk, next, &to_free, list) {
- int rs, re;
+ unsigned int rs, re;
- pcpu_for_each_pop_region(chunk->populated, rs, re, 0,
- chunk->nr_pages) {
+ bitmap_for_each_set_region(chunk->populated, rs, re, 0,
+ chunk->nr_pages) {
pcpu_depopulate_chunk(chunk, rs, re);
spin_lock_irq(&pcpu_lock);
pcpu_chunk_depopulated(chunk, rs, re);
@@ -1893,7 +1864,7 @@ static void pcpu_balance_workfn(struct work_struct *work)
}
for (slot = pcpu_size_to_slot(PAGE_SIZE); slot < pcpu_nr_slots; slot++) {
- int nr_unpop = 0, rs, re;
+ unsigned int nr_unpop = 0, rs, re;
if (!nr_to_pop)
break;
@@ -1910,9 +1881,9 @@ static void pcpu_balance_workfn(struct work_struct *work)
continue;
/* @chunk can't go away while pcpu_alloc_mutex is held */
- pcpu_for_each_unpop_region(chunk->populated, rs, re, 0,
- chunk->nr_pages) {
- int nr = min(re - rs, nr_to_pop);
+ bitmap_for_each_clear_region(chunk->populated, rs, re, 0,
+ chunk->nr_pages) {
+ int nr = min_t(int, re - rs, nr_to_pop);
ret = pcpu_populate_chunk(chunk, rs, rs + nr, gfp);
if (!ret) {
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 02/22] btrfs: rename DISCARD opt to DISCARD_SYNC
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
2019-12-14 0:22 ` [PATCH 01/22] bitmap: genericize percpu bitmap region iterators Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 03/22] btrfs: keep track of which extents have been discarded Dennis Zhou
` (21 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
This series introduces async discard which will use the flag
DISCARD_ASYNC, so rename the original flag to DISCARD_SYNC as it is
synchronously done in transaction commit.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
---
fs/btrfs/block-group.c | 2 +-
fs/btrfs/ctree.h | 2 +-
fs/btrfs/extent-tree.c | 2 +-
fs/btrfs/inode.c | 2 +-
fs/btrfs/super.c | 8 ++++----
5 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c
index 66fa39632cde..be1938dc94fd 100644
--- a/fs/btrfs/block-group.c
+++ b/fs/btrfs/block-group.c
@@ -1349,7 +1349,7 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
spin_unlock(&space_info->lock);
/* DISCARD can flip during remount */
- trimming = btrfs_test_opt(fs_info, DISCARD);
+ trimming = btrfs_test_opt(fs_info, DISCARD_SYNC);
/* Implicit trim during transaction commit. */
if (trimming)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index e416ef6c9415..2f6c21ea84af 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -1170,7 +1170,7 @@ static inline u32 BTRFS_MAX_XATTR_SIZE(const struct btrfs_fs_info *info)
#define BTRFS_MOUNT_FLUSHONCOMMIT (1 << 7)
#define BTRFS_MOUNT_SSD_SPREAD (1 << 8)
#define BTRFS_MOUNT_NOSSD (1 << 9)
-#define BTRFS_MOUNT_DISCARD (1 << 10)
+#define BTRFS_MOUNT_DISCARD_SYNC (1 << 10)
#define BTRFS_MOUNT_FORCE_COMPRESS (1 << 11)
#define BTRFS_MOUNT_SPACE_CACHE (1 << 12)
#define BTRFS_MOUNT_CLEAR_CACHE (1 << 13)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 180b9b81d01a..1a8bf943c3e7 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -2923,7 +2923,7 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
break;
}
- if (btrfs_test_opt(fs_info, DISCARD))
+ if (btrfs_test_opt(fs_info, DISCARD_SYNC))
ret = btrfs_discard_extent(fs_info, start,
end + 1 - start, NULL);
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index ddb2a806c038..101de0189a3b 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -2580,7 +2580,7 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)
clear_reserved_extent &&
!test_bit(BTRFS_ORDERED_NOCOW, &ordered_extent->flags) &&
!test_bit(BTRFS_ORDERED_PREALLOC, &ordered_extent->flags)) {
- if (ret && btrfs_test_opt(fs_info, DISCARD))
+ if (ret && btrfs_test_opt(fs_info, DISCARD_SYNC))
btrfs_discard_extent(fs_info,
ordered_extent->disk_bytenr,
ordered_extent->disk_num_bytes,
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index f452a94abdc3..08ac6a7a67f0 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -695,11 +695,11 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
info->metadata_ratio);
break;
case Opt_discard:
- btrfs_set_and_info(info, DISCARD,
- "turning on discard");
+ btrfs_set_and_info(info, DISCARD_SYNC,
+ "turning on sync discard");
break;
case Opt_nodiscard:
- btrfs_clear_and_info(info, DISCARD,
+ btrfs_clear_and_info(info, DISCARD_SYNC,
"turning off discard");
break;
case Opt_space_cache:
@@ -1322,7 +1322,7 @@ static int btrfs_show_options(struct seq_file *seq, struct dentry *dentry)
seq_puts(seq, ",nologreplay");
if (btrfs_test_opt(info, FLUSHONCOMMIT))
seq_puts(seq, ",flushoncommit");
- if (btrfs_test_opt(info, DISCARD))
+ if (btrfs_test_opt(info, DISCARD_SYNC))
seq_puts(seq, ",discard");
if (!(info->sb->s_flags & SB_POSIXACL))
seq_puts(seq, ",noacl");
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 03/22] btrfs: keep track of which extents have been discarded
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
2019-12-14 0:22 ` [PATCH 01/22] bitmap: genericize percpu bitmap region iterators Dennis Zhou
2019-12-14 0:22 ` [PATCH 02/22] btrfs: rename DISCARD opt to DISCARD_SYNC Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 04/22] btrfs: keep track of cleanliness of the bitmap Dennis Zhou
` (20 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Async discard will use the free space cache as backing knowledge for
which extents to discard. This patch plumbs knowledge about which
extents need to be discarded into the free space cache from
unpin_extent_range().
An untrimmed extent can merge with everything as this is a new region.
Absorbing trimmed extents is a tradeoff to for greater coalescing which
makes life better for find_free_extent(). Additionally, it seems the
size of a trim isn't as problematic as the trim io itself.
When reading in the free space cache from disk, if sync is set, mark all
extents as trimmed. The current code ensures at transaction commit that
all free space is trimmed when sync is set, so this reflects that.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
---
fs/btrfs/free-space-cache.c | 64 +++++++++++++++++++++++++++++++++----
fs/btrfs/free-space-cache.h | 17 +++++++++-
fs/btrfs/inode-map.c | 13 ++++----
3 files changed, 80 insertions(+), 14 deletions(-)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 3283da419200..5f8e2171efbf 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -752,6 +752,14 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
goto free_cache;
}
+ /*
+ * Sync discard ensures that the free space cache is always
+ * trimmed. So when reading this in, the state should reflect
+ * that.
+ */
+ if (btrfs_test_opt(fs_info, DISCARD_SYNC))
+ e->trim_state = BTRFS_TRIM_STATE_TRIMMED;
+
if (!e->bytes) {
kmem_cache_free(btrfs_free_space_cachep, e);
goto free_cache;
@@ -2161,6 +2169,22 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
return ret;
}
+/*
+ * Free space merging rules:
+ * 1) Merge trimmed areas together
+ * 2) Let untrimmed areas coalesce with trimmed areas
+ * 3) Always pull neighboring regions from bitmaps
+ *
+ * The above rules are for when we merge free space based on btrfs_trim_state.
+ * Rules 2 and 3 are subtle because they are suboptimal, but are done for the
+ * same reason: to promote larger extent regions which makes life easier for
+ * find_free_extent(). Rule 2 enables coalescing based on the common path
+ * being returning free space from btrfs_finish_extent_commit(). So when free
+ * space is trimmed, it will prevent aggregating trimmed new region and
+ * untrimmed regions in the rb_tree. Rule 3 is purely to obtain larger extents
+ * and provide find_free_extent() with the largest extents possible hoping for
+ * the reuse path.
+ */
static bool try_merge_free_space(struct btrfs_free_space_ctl *ctl,
struct btrfs_free_space *info, bool update_stat)
{
@@ -2169,6 +2193,7 @@ static bool try_merge_free_space(struct btrfs_free_space_ctl *ctl,
bool merged = false;
u64 offset = info->offset;
u64 bytes = info->bytes;
+ const bool is_trimmed = btrfs_free_space_trimmed(info);
/*
* first we want to see if there is free space adjacent to the range we
@@ -2182,7 +2207,9 @@ static bool try_merge_free_space(struct btrfs_free_space_ctl *ctl,
else
left_info = tree_search_offset(ctl, offset - 1, 0, 0);
- if (right_info && !right_info->bitmap) {
+ /* See try_merge_free_space() comment. */
+ if (right_info && !right_info->bitmap &&
+ (!is_trimmed || btrfs_free_space_trimmed(right_info))) {
if (update_stat)
unlink_free_space(ctl, right_info);
else
@@ -2192,8 +2219,10 @@ static bool try_merge_free_space(struct btrfs_free_space_ctl *ctl,
merged = true;
}
+ /* See try_merge_free_space() comment. */
if (left_info && !left_info->bitmap &&
- left_info->offset + left_info->bytes == offset) {
+ left_info->offset + left_info->bytes == offset &&
+ (!is_trimmed || btrfs_free_space_trimmed(left_info))) {
if (update_stat)
unlink_free_space(ctl, left_info);
else
@@ -2229,6 +2258,10 @@ static bool steal_from_bitmap_to_end(struct btrfs_free_space_ctl *ctl,
bytes = (j - i) * ctl->unit;
info->bytes += bytes;
+ /* See try_merge_free_space() comment. */
+ if (!btrfs_free_space_trimmed(bitmap))
+ info->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
+
if (update_stat)
bitmap_clear_bits(ctl, bitmap, end, bytes);
else
@@ -2282,6 +2315,10 @@ static bool steal_from_bitmap_to_front(struct btrfs_free_space_ctl *ctl,
info->offset -= bytes;
info->bytes += bytes;
+ /* See try_merge_free_space() comment. */
+ if (!btrfs_free_space_trimmed(bitmap))
+ info->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
+
if (update_stat)
bitmap_clear_bits(ctl, bitmap, info->offset, bytes);
else
@@ -2331,7 +2368,8 @@ static void steal_from_bitmap(struct btrfs_free_space_ctl *ctl,
int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
struct btrfs_free_space_ctl *ctl,
- u64 offset, u64 bytes)
+ u64 offset, u64 bytes,
+ enum btrfs_trim_state trim_state)
{
struct btrfs_free_space *info;
int ret = 0;
@@ -2342,6 +2380,7 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
info->offset = offset;
info->bytes = bytes;
+ info->trim_state = trim_state;
RB_CLEAR_NODE(&info->offset_index);
spin_lock(&ctl->tree_lock);
@@ -2387,9 +2426,14 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
int btrfs_add_free_space(struct btrfs_block_group *block_group,
u64 bytenr, u64 size)
{
+ enum btrfs_trim_state trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
+
+ if (btrfs_test_opt(block_group->fs_info, DISCARD_SYNC))
+ trim_state = BTRFS_TRIM_STATE_TRIMMED;
+
return __btrfs_add_free_space(block_group->fs_info,
block_group->free_space_ctl,
- bytenr, size);
+ bytenr, size, trim_state);
}
int btrfs_remove_free_space(struct btrfs_block_group *block_group,
@@ -2464,8 +2508,11 @@ int btrfs_remove_free_space(struct btrfs_block_group *block_group,
}
spin_unlock(&ctl->tree_lock);
- ret = btrfs_add_free_space(block_group, offset + bytes,
- old_end - (offset + bytes));
+ ret = __btrfs_add_free_space(block_group->fs_info,
+ ctl,
+ offset + bytes,
+ old_end - (offset + bytes),
+ info->trim_state);
WARN_ON(ret);
goto out;
}
@@ -2634,6 +2681,7 @@ u64 btrfs_find_space_for_alloc(struct btrfs_block_group *block_group,
u64 ret = 0;
u64 align_gap = 0;
u64 align_gap_len = 0;
+ enum btrfs_trim_state align_gap_trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
spin_lock(&ctl->tree_lock);
entry = find_free_space(ctl, &offset, &bytes_search,
@@ -2650,6 +2698,7 @@ u64 btrfs_find_space_for_alloc(struct btrfs_block_group *block_group,
unlink_free_space(ctl, entry);
align_gap_len = offset - entry->offset;
align_gap = entry->offset;
+ align_gap_trim_state = entry->trim_state;
entry->offset = offset + bytes;
WARN_ON(entry->bytes < bytes + align_gap_len);
@@ -2665,7 +2714,8 @@ u64 btrfs_find_space_for_alloc(struct btrfs_block_group *block_group,
if (align_gap_len)
__btrfs_add_free_space(block_group->fs_info, ctl,
- align_gap, align_gap_len);
+ align_gap, align_gap_len,
+ align_gap_trim_state);
return ret;
}
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index ba9a23241101..66c073f854dc 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -6,6 +6,14 @@
#ifndef BTRFS_FREE_SPACE_CACHE_H
#define BTRFS_FREE_SPACE_CACHE_H
+/*
+ * This is the trim state of an extent or bitmap.
+ */
+enum btrfs_trim_state {
+ BTRFS_TRIM_STATE_UNTRIMMED,
+ BTRFS_TRIM_STATE_TRIMMED,
+};
+
struct btrfs_free_space {
struct rb_node offset_index;
u64 offset;
@@ -13,8 +21,14 @@ struct btrfs_free_space {
u64 max_extent_size;
unsigned long *bitmap;
struct list_head list;
+ enum btrfs_trim_state trim_state;
};
+static inline bool btrfs_free_space_trimmed(struct btrfs_free_space *info)
+{
+ return (info->trim_state == BTRFS_TRIM_STATE_TRIMMED);
+}
+
struct btrfs_free_space_ctl {
spinlock_t tree_lock;
struct rb_root free_space_offset;
@@ -83,7 +97,8 @@ int btrfs_write_out_ino_cache(struct btrfs_root *root,
void btrfs_init_free_space_ctl(struct btrfs_block_group *block_group);
int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
struct btrfs_free_space_ctl *ctl,
- u64 bytenr, u64 size);
+ u64 bytenr, u64 size,
+ enum btrfs_trim_state trim_state);
int btrfs_add_free_space(struct btrfs_block_group *block_group,
u64 bytenr, u64 size);
int btrfs_remove_free_space(struct btrfs_block_group *block_group,
diff --git a/fs/btrfs/inode-map.c b/fs/btrfs/inode-map.c
index 37345fb6191d..d5c9c69d8263 100644
--- a/fs/btrfs/inode-map.c
+++ b/fs/btrfs/inode-map.c
@@ -107,7 +107,7 @@ static int caching_kthread(void *data)
if (last != (u64)-1 && last + 1 != key.objectid) {
__btrfs_add_free_space(fs_info, ctl, last + 1,
- key.objectid - last - 1);
+ key.objectid - last - 1, 0);
wake_up(&root->ino_cache_wait);
}
@@ -118,7 +118,7 @@ static int caching_kthread(void *data)
if (last < root->highest_objectid - 1) {
__btrfs_add_free_space(fs_info, ctl, last + 1,
- root->highest_objectid - last - 1);
+ root->highest_objectid - last - 1, 0);
}
spin_lock(&root->ino_cache_lock);
@@ -175,7 +175,8 @@ static void start_caching(struct btrfs_root *root)
ret = btrfs_find_free_objectid(root, &objectid);
if (!ret && objectid <= BTRFS_LAST_FREE_OBJECTID) {
__btrfs_add_free_space(fs_info, ctl, objectid,
- BTRFS_LAST_FREE_OBJECTID - objectid + 1);
+ BTRFS_LAST_FREE_OBJECTID - objectid + 1,
+ 0);
wake_up(&root->ino_cache_wait);
}
@@ -221,7 +222,7 @@ void btrfs_return_ino(struct btrfs_root *root, u64 objectid)
return;
again:
if (root->ino_cache_state == BTRFS_CACHE_FINISHED) {
- __btrfs_add_free_space(fs_info, pinned, objectid, 1);
+ __btrfs_add_free_space(fs_info, pinned, objectid, 1, 0);
} else {
down_write(&fs_info->commit_root_sem);
spin_lock(&root->ino_cache_lock);
@@ -234,7 +235,7 @@ void btrfs_return_ino(struct btrfs_root *root, u64 objectid)
start_caching(root);
- __btrfs_add_free_space(fs_info, pinned, objectid, 1);
+ __btrfs_add_free_space(fs_info, pinned, objectid, 1, 0);
up_write(&fs_info->commit_root_sem);
}
@@ -281,7 +282,7 @@ void btrfs_unpin_free_ino(struct btrfs_root *root)
spin_unlock(rbroot_lock);
if (count)
__btrfs_add_free_space(root->fs_info, ctl,
- info->offset, count);
+ info->offset, count, 0);
kmem_cache_free(btrfs_free_space_cachep, info);
}
}
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 04/22] btrfs: keep track of cleanliness of the bitmap
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (2 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 03/22] btrfs: keep track of which extents have been discarded Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue Dennis Zhou
` (19 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
There is a cap in btrfs in the amount of free extents that a block group
can have. When it surpasses that threshold, future extents are placed
into bitmaps. Instead of keeping track of if a certain bit is trimmed or
not in a second bitmap, keep track of the relative state of the bitmap.
With async discard, trimming bitmaps becomes a more frequent operation.
As a trade off with simplicity, we keep track of if discarding a bitmap
is in progress. If we fully scan a bitmap and trim as necessary, the
bitmap is marked clean. This has some caveats as the min block size may
skip over regions deemed too small. But this should be a reasonable
trade off rather than keeping a second bitmap and making allocation
paths more complex. The downside is we may overtrim, but ideally the min
block size should prevent us from doing that too often and getting stuck
trimming pathological cases.
BTRFS_TRIM_STATE_TRIMMING is added to indicate a bitmap is in the
process of being trimmed. If additional free space is added to that
bitmap, the bit is cleared. A bitmap will be marked
BTRFS_TRIM_STATE_TRIMMED if the trimming code was able to reach the end
of it and the former is still set.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/free-space-cache.c | 91 +++++++++++++++++++++++++++++++++----
fs/btrfs/free-space-cache.h | 12 +++++
2 files changed, 94 insertions(+), 9 deletions(-)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 5f8e2171efbf..7108962e208d 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -1979,11 +1979,18 @@ static noinline int remove_from_bitmap(struct btrfs_free_space_ctl *ctl,
static u64 add_bytes_to_bitmap(struct btrfs_free_space_ctl *ctl,
struct btrfs_free_space *info, u64 offset,
- u64 bytes)
+ u64 bytes, enum btrfs_trim_state trim_state)
{
u64 bytes_to_set = 0;
u64 end;
+ /*
+ * This is a tradeoff to make bitmap trim state minimal. We mark the
+ * whole bitmap untrimmed if at any point we add untrimmed regions.
+ */
+ if (trim_state == BTRFS_TRIM_STATE_UNTRIMMED)
+ info->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
+
end = info->offset + (u64)(BITS_PER_BITMAP * ctl->unit);
bytes_to_set = min(end - offset, bytes);
@@ -2058,10 +2065,12 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
struct btrfs_block_group *block_group = NULL;
int added = 0;
u64 bytes, offset, bytes_added;
+ enum btrfs_trim_state trim_state;
int ret;
bytes = info->bytes;
offset = info->offset;
+ trim_state = info->trim_state;
if (!ctl->op->use_bitmap(ctl, info))
return 0;
@@ -2096,8 +2105,8 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
}
if (entry->offset == offset_to_bitmap(ctl, offset)) {
- bytes_added = add_bytes_to_bitmap(ctl, entry,
- offset, bytes);
+ bytes_added = add_bytes_to_bitmap(ctl, entry, offset,
+ bytes, trim_state);
bytes -= bytes_added;
offset += bytes_added;
}
@@ -2116,7 +2125,8 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
goto new_bitmap;
}
- bytes_added = add_bytes_to_bitmap(ctl, bitmap_info, offset, bytes);
+ bytes_added = add_bytes_to_bitmap(ctl, bitmap_info, offset, bytes,
+ trim_state);
bytes -= bytes_added;
offset += bytes_added;
added = 0;
@@ -2150,6 +2160,7 @@ static int insert_into_bitmap(struct btrfs_free_space_ctl *ctl,
/* allocate the bitmap */
info->bitmap = kmem_cache_zalloc(btrfs_free_space_bitmap_cachep,
GFP_NOFS);
+ info->trim_state = BTRFS_TRIM_STATE_TRIMMED;
spin_lock(&ctl->tree_lock);
if (!info->bitmap) {
ret = -ENOMEM;
@@ -3325,6 +3336,39 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
return ret;
}
+/*
+ * If we break out of trimming a bitmap prematurely, we should reset the
+ * trimming bit. In a rather contrieved case, it's possible to race here so
+ * reset the state to BTRFS_TRIM_STATE_UNTRIMMED.
+ *
+ * start = start of bitmap
+ * end = near end of bitmap
+ *
+ * Thread 1: Thread 2:
+ * trim_bitmaps(start)
+ * trim_bitmaps(end)
+ * end_trimming_bitmap()
+ * reset_trimming_bitmap()
+ */
+static void reset_trimming_bitmap(struct btrfs_free_space_ctl *ctl, u64 offset)
+{
+ struct btrfs_free_space *entry;
+
+ spin_lock(&ctl->tree_lock);
+
+ entry = tree_search_offset(ctl, offset, 1, 0);
+ if (entry)
+ entry->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
+
+ spin_unlock(&ctl->tree_lock);
+}
+
+static void end_trimming_bitmap(struct btrfs_free_space *entry)
+{
+ if (btrfs_free_space_trimming_bitmap(entry))
+ entry->trim_state = BTRFS_TRIM_STATE_TRIMMED;
+}
+
static int trim_bitmaps(struct btrfs_block_group *block_group,
u64 *total_trimmed, u64 start, u64 end, u64 minlen)
{
@@ -3349,16 +3393,33 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
}
entry = tree_search_offset(ctl, offset, 1, 0);
- if (!entry) {
+ if (!entry || btrfs_free_space_trimmed(entry)) {
spin_unlock(&ctl->tree_lock);
mutex_unlock(&ctl->cache_writeout_mutex);
next_bitmap = true;
goto next;
}
+ /*
+ * Async discard bitmap trimming begins at by setting the start
+ * to be key.objectid and the offset_to_bitmap() aligns to the
+ * start of the bitmap. This lets us know we are fully
+ * scanning the bitmap rather than only some portion of it.
+ */
+ if (start == offset)
+ entry->trim_state = BTRFS_TRIM_STATE_TRIMMING;
+
bytes = minlen;
ret2 = search_bitmap(ctl, entry, &start, &bytes, false);
if (ret2 || start >= end) {
+ /*
+ * This keeps the invariant that all bytes are trimmed
+ * if BTRFS_TRIM_STATE_TRIMMED is set on a bitmap.
+ */
+ if (ret2 && !minlen)
+ end_trimming_bitmap(entry);
+ else
+ entry->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
spin_unlock(&ctl->tree_lock);
mutex_unlock(&ctl->cache_writeout_mutex);
next_bitmap = true;
@@ -3367,6 +3428,7 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
bytes = min(bytes, end - start);
if (bytes < minlen) {
+ entry->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
spin_unlock(&ctl->tree_lock);
mutex_unlock(&ctl->cache_writeout_mutex);
goto next;
@@ -3384,18 +3446,21 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
ret = do_trimming(block_group, total_trimmed, start, bytes,
start, bytes, &trim_entry);
- if (ret)
+ if (ret) {
+ reset_trimming_bitmap(ctl, offset);
break;
+ }
next:
if (next_bitmap) {
offset += BITS_PER_BITMAP * ctl->unit;
+ start = offset;
} else {
start += bytes;
- if (start >= offset + BITS_PER_BITMAP * ctl->unit)
- offset += BITS_PER_BITMAP * ctl->unit;
}
if (fatal_signal_pending(current)) {
+ if (start != offset)
+ reset_trimming_bitmap(ctl, offset);
ret = -ERESTARTSYS;
break;
}
@@ -3449,7 +3514,9 @@ void btrfs_put_block_group_trimming(struct btrfs_block_group *block_group)
int btrfs_trim_block_group(struct btrfs_block_group *block_group,
u64 *trimmed, u64 start, u64 end, u64 minlen)
{
+ struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
int ret;
+ u64 rem = 0;
*trimmed = 0;
@@ -3466,6 +3533,10 @@ int btrfs_trim_block_group(struct btrfs_block_group *block_group,
goto out;
ret = trim_bitmaps(block_group, trimmed, start, end, minlen);
+ div64_u64_rem(end, BITS_PER_BITMAP * ctl->unit, &rem);
+ /* If we ended in the middle of a bitmap, reset the trimming flag. */
+ if (rem)
+ reset_trimming_bitmap(ctl, offset_to_bitmap(ctl, end));
out:
btrfs_put_block_group_trimming(block_group);
return ret;
@@ -3650,6 +3721,7 @@ int test_add_free_space_entry(struct btrfs_block_group *cache,
struct btrfs_free_space_ctl *ctl = cache->free_space_ctl;
struct btrfs_free_space *info = NULL, *bitmap_info;
void *map = NULL;
+ enum btrfs_trim_state trim_state = BTRFS_TRIM_STATE_TRIMMED;
u64 bytes_added;
int ret;
@@ -3691,7 +3763,8 @@ int test_add_free_space_entry(struct btrfs_block_group *cache,
info = NULL;
}
- bytes_added = add_bytes_to_bitmap(ctl, bitmap_info, offset, bytes);
+ bytes_added = add_bytes_to_bitmap(ctl, bitmap_info, offset, bytes,
+ trim_state);
bytes -= bytes_added;
offset += bytes_added;
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index 66c073f854dc..29d16f58b40b 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -8,10 +8,16 @@
/*
* This is the trim state of an extent or bitmap.
+ *
+ * BTRFS_TRIM_STATE_TRIMMING is special and used to maintain the state of a
+ * bitmap as we may need several trims to fully trim a single bitmap entry.
+ * This is reset should any free space other than trimmed space is added to the
+ * bitmap.
*/
enum btrfs_trim_state {
BTRFS_TRIM_STATE_UNTRIMMED,
BTRFS_TRIM_STATE_TRIMMED,
+ BTRFS_TRIM_STATE_TRIMMING,
};
struct btrfs_free_space {
@@ -29,6 +35,12 @@ static inline bool btrfs_free_space_trimmed(struct btrfs_free_space *info)
return (info->trim_state == BTRFS_TRIM_STATE_TRIMMED);
}
+static inline bool btrfs_free_space_trimming_bitmap(
+ struct btrfs_free_space *info)
+{
+ return (info->trim_state == BTRFS_TRIM_STATE_TRIMMING);
+}
+
struct btrfs_free_space_ctl {
spinlock_t tree_lock;
struct rb_root free_space_offset;
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (3 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 04/22] btrfs: keep track of cleanliness of the bitmap Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 06/22] btrfs: handle empty block_group removal Dennis Zhou
` (18 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
When discard is enabled, everytime a pinned extent is released back to
the block_group's free space cache, a discard is issued for the extent.
This is an overeager approach when it comes to discarding and helping
the SSD maintain enough free space to prevent severe garbage collection
situations.
This adds the beginning of async discard. Instead of issuing a discard
prior to returning it to the free space, it is just marked as untrimmed.
The block_group is then added to a LRU which then feeds into a workqueue
to issue discards at a much slower rate. Full discarding of unused block
groups is still done and will be address in a future patch in this
series.
For now, we don't persist the discard state of extents and bitmaps.
Therefore, our failure recovery mode will be to consider extents
untrimmed. This lets us handle failure and unmounting as one in the
same.
On a number of Facebook webservers, I collected data every minute
accounting the time we spent in btrfs_finish_extent_commit() (col. 1)
and in btrfs_commit_transaction() (col. 2). btrfs_finish_extent_commit()
is where we discard extents synchronously before returning them to the
free space cache.
discard=sync:
p99 total per minute p99 total per minute
Drive | extent_commit() (ms) | commit_trans() (ms)
---------------------------------------------------------------
Drive A | 434 | 1170
Drive B | 880 | 2330
Drive C | 2943 | 3920
Drive D | 4763 | 5701
discard=async:
p99 total per minute p99 total per minute
Drive | extent_commit() (ms) | commit_trans() (ms)
--------------------------------------------------------------
Drive A | 134 | 956
Drive B | 64 | 1972
Drive C | 59 | 1032
Drive D | 62 | 1200
While it's not great that the stats are cumulative over 1m, all of these
servers are running the same workload and and the delta between the two
are substantial. We are spending significantly less time in
btrfs_finish_extent_commit() which is responsible for discarding.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/Makefile | 2 +-
fs/btrfs/block-group.c | 37 ++++-
fs/btrfs/block-group.h | 9 ++
fs/btrfs/ctree.h | 21 +++
fs/btrfs/discard.c | 278 ++++++++++++++++++++++++++++++++++++
fs/btrfs/discard.h | 23 +++
fs/btrfs/disk-io.c | 15 +-
fs/btrfs/extent-tree.c | 4 +
fs/btrfs/free-space-cache.c | 54 ++++++-
fs/btrfs/free-space-cache.h | 2 +
fs/btrfs/super.c | 35 ++++-
fs/btrfs/volumes.c | 7 +
12 files changed, 474 insertions(+), 13 deletions(-)
create mode 100644 fs/btrfs/discard.c
create mode 100644 fs/btrfs/discard.h
diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
index 82200dbca5ac..9a0ff3384381 100644
--- a/fs/btrfs/Makefile
+++ b/fs/btrfs/Makefile
@@ -11,7 +11,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
uuid-tree.o props.o free-space-tree.o tree-checker.o space-info.o \
- block-rsv.o delalloc-space.o block-group.o
+ block-rsv.o delalloc-space.o block-group.o discard.o
btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c
index be1938dc94fd..c1b1b59343bd 100644
--- a/fs/btrfs/block-group.c
+++ b/fs/btrfs/block-group.c
@@ -14,6 +14,7 @@
#include "sysfs.h"
#include "tree-log.h"
#include "delalloc-space.h"
+#include "discard.h"
/*
* Return target flags in extended format or 0 if restripe for this chunk_type
@@ -131,6 +132,15 @@ void btrfs_put_block_group(struct btrfs_block_group *cache)
WARN_ON(cache->pinned > 0);
WARN_ON(cache->reserved > 0);
+ /*
+ * A block_group shouldn't be on the discard_list anymore.
+ * Remove the block_group from the discard_list to prevent us
+ * from causing a panic due to NPE.
+ */
+ if (WARN_ON(!list_empty(&cache->discard_list)))
+ btrfs_discard_cancel_work(&cache->fs_info->discard_ctl,
+ cache);
+
/*
* If not empty, someone is still holding mutex of
* full_stripe_lock, which can only be released by caller.
@@ -466,8 +476,8 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 end
} else if (extent_start > start && extent_start < end) {
size = extent_start - start;
total_added += size;
- ret = btrfs_add_free_space(block_group, start,
- size);
+ ret = btrfs_add_free_space_async_trimmed(block_group,
+ start, size);
BUG_ON(ret); /* -ENOMEM or logic error */
start = extent_end + 1;
} else {
@@ -478,7 +488,8 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 end
if (start < end) {
size = end - start;
total_added += size;
- ret = btrfs_add_free_space(block_group, start, size);
+ ret = btrfs_add_free_space_async_trimmed(block_group, start,
+ size);
BUG_ON(ret); /* -ENOMEM or logic error */
}
@@ -1258,6 +1269,8 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
}
spin_unlock(&fs_info->unused_bgs_lock);
+ btrfs_discard_cancel_work(&fs_info->discard_ctl, block_group);
+
mutex_lock(&fs_info->delete_unused_bgs_mutex);
/* Don't want to race with allocators so take the groups_sem */
@@ -1333,6 +1346,23 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
}
mutex_unlock(&fs_info->unused_bg_unpin_mutex);
+ /*
+ * At this point, the block_group is read only and should fail
+ * new allocations. However, btrfs_finish_extent_commit() can
+ * cause this block_group to be placed back on the discard
+ * lists because now the block_group isn't fully discarded.
+ * Bail here and try again later after discarding everything.
+ */
+ spin_lock(&fs_info->discard_ctl.lock);
+ if (!list_empty(&block_group->discard_list)) {
+ spin_unlock(&fs_info->discard_ctl.lock);
+ btrfs_dec_block_group_ro(block_group);
+ btrfs_discard_queue_work(&fs_info->discard_ctl,
+ block_group);
+ goto end_trans;
+ }
+ spin_unlock(&fs_info->discard_ctl.lock);
+
/* Reset pinned so btrfs_put_block_group doesn't complain */
spin_lock(&space_info->lock);
spin_lock(&block_group->lock);
@@ -1603,6 +1633,7 @@ static struct btrfs_block_group *btrfs_create_block_group_cache(
INIT_LIST_HEAD(&cache->cluster_list);
INIT_LIST_HEAD(&cache->bg_list);
INIT_LIST_HEAD(&cache->ro_list);
+ INIT_LIST_HEAD(&cache->discard_list);
INIT_LIST_HEAD(&cache->dirty_list);
INIT_LIST_HEAD(&cache->io_list);
btrfs_init_free_space_ctl(cache);
diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h
index 9b409676c4b2..884defd61dcd 100644
--- a/fs/btrfs/block-group.h
+++ b/fs/btrfs/block-group.h
@@ -116,7 +116,11 @@ struct btrfs_block_group {
/* For read-only block groups */
struct list_head ro_list;
+ /* For discard operations */
atomic_t trimming;
+ struct list_head discard_list;
+ int discard_index;
+ u64 discard_eligible_time;
/* For dirty block groups */
struct list_head dirty_list;
@@ -158,6 +162,11 @@ struct btrfs_block_group {
struct btrfs_full_stripe_locks_tree full_stripe_locks_root;
};
+static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
+{
+ return (block_group->start + block_group->length);
+}
+
#ifdef CONFIG_BTRFS_DEBUG
static inline int btrfs_should_fragment_free_space(
struct btrfs_block_group *block_group)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 2f6c21ea84af..f7b429277089 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -440,6 +440,21 @@ struct btrfs_full_stripe_locks_tree {
struct mutex lock;
};
+/* Discard control. */
+/*
+ * Async discard uses multiple lists to differentiate the discard filter
+ * parameters.
+ */
+#define BTRFS_NR_DISCARD_LISTS 1
+
+struct btrfs_discard_ctl {
+ struct workqueue_struct *discard_workers;
+ struct delayed_work work;
+ spinlock_t lock;
+ struct btrfs_block_group *block_group;
+ struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
+};
+
/* delayed seq elem */
struct seq_list {
struct list_head list;
@@ -526,6 +541,9 @@ enum {
* so we don't need to offload checksums to workqueues.
*/
BTRFS_FS_CSUM_IMPL_FAST,
+
+ /* Indicate that the discard workqueue can service discards. */
+ BTRFS_FS_DISCARD_RUNNING,
};
struct btrfs_fs_info {
@@ -816,6 +834,8 @@ struct btrfs_fs_info {
struct btrfs_workqueue *scrub_wr_completion_workers;
struct btrfs_workqueue *scrub_parity_workers;
+ struct btrfs_discard_ctl discard_ctl;
+
#ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY
u32 check_integrity_print_mask;
#endif
@@ -1189,6 +1209,7 @@ static inline u32 BTRFS_MAX_XATTR_SIZE(const struct btrfs_fs_info *info)
#define BTRFS_MOUNT_FREE_SPACE_TREE (1 << 26)
#define BTRFS_MOUNT_NOLOGREPLAY (1 << 27)
#define BTRFS_MOUNT_REF_VERIFY (1 << 28)
+#define BTRFS_MOUNT_DISCARD_ASYNC (1 << 29)
#define BTRFS_DEFAULT_COMMIT_INTERVAL (30)
#define BTRFS_DEFAULT_MAX_INLINE (2048)
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
new file mode 100644
index 000000000000..692d64025802
--- /dev/null
+++ b/fs/btrfs/discard.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/jiffies.h>
+#include <linux/kernel.h>
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/sizes.h>
+#include <linux/workqueue.h>
+#include "ctree.h"
+#include "block-group.h"
+#include "discard.h"
+#include "free-space-cache.h"
+
+/* This is an initial delay to give some chance for lba reuse. */
+#define BTRFS_DISCARD_DELAY (120ULL * NSEC_PER_SEC)
+
+static struct list_head *get_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ return &discard_ctl->discard_list[block_group->discard_index];
+}
+
+static void add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ spin_lock(&discard_ctl->lock);
+
+ if (!btrfs_run_discard_work(discard_ctl)) {
+ spin_unlock(&discard_ctl->lock);
+ return;
+ }
+
+ if (list_empty(&block_group->discard_list))
+ block_group->discard_eligible_time = (ktime_get_ns() +
+ BTRFS_DISCARD_DELAY);
+
+ list_move_tail(&block_group->discard_list,
+ get_discard_list(discard_ctl, block_group));
+
+ spin_unlock(&discard_ctl->lock);
+}
+
+static bool remove_from_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ bool running = false;
+
+ spin_lock(&discard_ctl->lock);
+
+ if (block_group == discard_ctl->block_group) {
+ running = true;
+ discard_ctl->block_group = NULL;
+ }
+
+ block_group->discard_eligible_time = 0;
+ list_del_init(&block_group->discard_list);
+
+ spin_unlock(&discard_ctl->lock);
+
+ return running;
+}
+
+/**
+ * find_next_block_group - find block_group that's up next for discarding
+ * @discard_ctl: discard control
+ * @now: current time
+ *
+ * Iterate over the discard lists to find the next block_group up for
+ * discarding checking the discard_eligible_time of block_group.
+ */
+static struct btrfs_block_group *find_next_block_group(
+ struct btrfs_discard_ctl *discard_ctl,
+ u64 now)
+{
+ struct btrfs_block_group *ret_block_group = NULL, *block_group;
+ int i;
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++) {
+ struct list_head *discard_list = &discard_ctl->discard_list[i];
+
+ if (!list_empty(discard_list)) {
+ block_group = list_first_entry(discard_list,
+ struct btrfs_block_group,
+ discard_list);
+
+ if (!ret_block_group)
+ ret_block_group = block_group;
+
+ if (ret_block_group->discard_eligible_time < now)
+ break;
+
+ if (ret_block_group->discard_eligible_time >
+ block_group->discard_eligible_time)
+ ret_block_group = block_group;
+ }
+ }
+
+ return ret_block_group;
+}
+
+/**
+ * peek_discard_list - wrap find_next_block_group()
+ * @discard_ctl: discard control
+ *
+ * This wraps find_next_block_group() and sets the block_group to be in use.
+ */
+static struct btrfs_block_group *peek_discard_list(
+ struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_block_group *block_group;
+ u64 now = ktime_get_ns();
+
+ spin_lock(&discard_ctl->lock);
+
+ block_group = find_next_block_group(discard_ctl, now);
+
+ if (block_group && now < block_group->discard_eligible_time)
+ block_group = NULL;
+
+ discard_ctl->block_group = block_group;
+
+ spin_unlock(&discard_ctl->lock);
+
+ return block_group;
+}
+
+/**
+ * btrfs_discard_cancel_work - remove a block_group from the discard lists
+ * @discard_ctl: discard control
+ * @block_group: block_group of interest
+ *
+ * This removes @block_group from the discard lists. If necessary, it waits on
+ * the current work and then reschedules the delayed work.
+ */
+void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ if (remove_from_discard_list(discard_ctl, block_group)) {
+ cancel_delayed_work_sync(&discard_ctl->work);
+ btrfs_discard_schedule_work(discard_ctl, true);
+ }
+}
+
+/**
+ * btrfs_discard_queue_work - handles queuing the block_groups
+ * @discard_ctl: discard control
+ * @block_group: block_group of interest
+ *
+ * This maintains the LRU order of the discard lists.
+ */
+void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ if (!block_group ||
+ !btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
+ return;
+
+ add_to_discard_list(discard_ctl, block_group);
+
+ if (!delayed_work_pending(&discard_ctl->work))
+ btrfs_discard_schedule_work(discard_ctl, false);
+}
+
+/**
+ * btrfs_discard_schedule_work - responsible for scheduling the discard work
+ * @discard_ctl: discard control
+ * @override: override the current timer
+ *
+ * Discards are issued by a delayed workqueue item. @override is used to
+ * update the current delay as the baseline delay interview is reevaluated
+ * on transaction commit. This is also maxed with any other rate limit.
+ */
+void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
+ bool override)
+{
+ struct btrfs_block_group *block_group;
+ u64 now = ktime_get_ns();
+
+ spin_lock(&discard_ctl->lock);
+
+ if (!btrfs_run_discard_work(discard_ctl))
+ goto out;
+
+ if (!override && delayed_work_pending(&discard_ctl->work))
+ goto out;
+
+ block_group = find_next_block_group(discard_ctl, now);
+ if (block_group) {
+ u64 delay = 0;
+
+ if (now < block_group->discard_eligible_time)
+ delay = nsecs_to_jiffies(
+ block_group->discard_eligible_time - now);
+
+ mod_delayed_work(discard_ctl->discard_workers,
+ &discard_ctl->work,
+ delay);
+ }
+
+out:
+ spin_unlock(&discard_ctl->lock);
+}
+
+/**
+ * btrfs_discard_workfn - discard work function
+ * @work: work
+ *
+ * This finds the next block_group to start discarding and then discards it.
+ */
+static void btrfs_discard_workfn(struct work_struct *work)
+{
+ struct btrfs_discard_ctl *discard_ctl;
+ struct btrfs_block_group *block_group;
+ u64 trimmed = 0;
+
+ discard_ctl = container_of(work, struct btrfs_discard_ctl, work.work);
+
+ block_group = peek_discard_list(discard_ctl);
+ if (!block_group || !btrfs_run_discard_work(discard_ctl))
+ return;
+
+ btrfs_trim_block_group(block_group, &trimmed, block_group->start,
+ btrfs_block_group_end(block_group), 0);
+
+ remove_from_discard_list(discard_ctl, block_group);
+
+ btrfs_discard_schedule_work(discard_ctl, false);
+}
+
+/**
+ * btrfs_run_discard_work - determines if async discard should be running
+ * @discard_ctl: discard control
+ *
+ * Checks if the file system is writeable and BTRFS_FS_DISCARD_RUNNING is set.
+ */
+bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_fs_info *fs_info = container_of(discard_ctl,
+ struct btrfs_fs_info,
+ discard_ctl);
+
+ return (!(fs_info->sb->s_flags & SB_RDONLY) &&
+ test_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags));
+}
+
+void btrfs_discard_resume(struct btrfs_fs_info *fs_info)
+{
+ if (!btrfs_test_opt(fs_info, DISCARD_ASYNC)) {
+ btrfs_discard_cleanup(fs_info);
+ return;
+ }
+
+ set_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
+}
+
+void btrfs_discard_stop(struct btrfs_fs_info *fs_info)
+{
+ clear_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
+}
+
+void btrfs_discard_init(struct btrfs_fs_info *fs_info)
+{
+ struct btrfs_discard_ctl *discard_ctl = &fs_info->discard_ctl;
+ int i;
+
+ spin_lock_init(&discard_ctl->lock);
+
+ INIT_DELAYED_WORK(&discard_ctl->work, btrfs_discard_workfn);
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++)
+ INIT_LIST_HEAD(&discard_ctl->discard_list[i]);
+}
+
+void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
+{
+ btrfs_discard_stop(fs_info);
+ cancel_delayed_work_sync(&fs_info->discard_ctl.work);
+}
diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
new file mode 100644
index 000000000000..f3775e84d35a
--- /dev/null
+++ b/fs/btrfs/discard.h
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#ifndef BTRFS_DISCARD_H
+#define BTRFS_DISCARD_H
+
+struct btrfs_fs_info;
+struct btrfs_discard_ctl;
+struct btrfs_block_group;
+
+void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group);
+void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group);
+void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
+ bool override);
+bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl);
+
+void btrfs_discard_resume(struct btrfs_fs_info *fs_info);
+void btrfs_discard_stop(struct btrfs_fs_info *fs_info);
+void btrfs_discard_init(struct btrfs_fs_info *fs_info);
+void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info);
+
+#endif
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 881aba162e4e..d517c4b31338 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -41,6 +41,7 @@
#include "tree-checker.h"
#include "ref-verify.h"
#include "block-group.h"
+#include "discard.h"
#define BTRFS_SUPER_FLAG_SUPP (BTRFS_HEADER_FLAG_WRITTEN |\
BTRFS_HEADER_FLAG_RELOC |\
@@ -1953,6 +1954,8 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info)
btrfs_destroy_workqueue(fs_info->readahead_workers);
btrfs_destroy_workqueue(fs_info->flush_workers);
btrfs_destroy_workqueue(fs_info->qgroup_rescan_workers);
+ if (fs_info->discard_ctl.discard_workers)
+ destroy_workqueue(fs_info->discard_ctl.discard_workers);
/*
* Now that all other work queues are destroyed, we can safely destroy
* the queues used for metadata I/O, since tasks from those other work
@@ -2148,6 +2151,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info,
max_active, 2);
fs_info->qgroup_rescan_workers =
btrfs_alloc_workqueue(fs_info, "qgroup-rescan", flags, 1, 0);
+ fs_info->discard_ctl.discard_workers =
+ alloc_workqueue("btrfs_discard", WQ_UNBOUND | WQ_FREEZABLE, 1);
if (!(fs_info->workers && fs_info->delalloc_workers &&
fs_info->flush_workers &&
@@ -2158,7 +2163,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info,
fs_info->endio_freespace_worker && fs_info->rmw_workers &&
fs_info->caching_workers && fs_info->readahead_workers &&
fs_info->fixup_workers && fs_info->delayed_workers &&
- fs_info->qgroup_rescan_workers)) {
+ fs_info->qgroup_rescan_workers &&
+ fs_info->discard_ctl.discard_workers)) {
return -ENOMEM;
}
@@ -2793,6 +2799,8 @@ int __cold open_ctree(struct super_block *sb,
btrfs_init_dev_replace_locks(fs_info);
btrfs_init_qgroup(fs_info);
+ btrfs_discard_init(fs_info);
+
btrfs_init_free_cluster(&fs_info->meta_alloc_cluster);
btrfs_init_free_cluster(&fs_info->data_alloc_cluster);
@@ -3256,6 +3264,8 @@ int __cold open_ctree(struct super_block *sb,
btrfs_qgroup_rescan_resume(fs_info);
+ btrfs_discard_resume(fs_info);
+
if (!fs_info->uuid_root) {
btrfs_info(fs_info, "creating UUID tree");
ret = btrfs_create_uuid_tree(fs_info);
@@ -3971,6 +3981,9 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
cancel_work_sync(&fs_info->async_reclaim_work);
+ /* cancel or finish ongoing discard work */
+ btrfs_discard_cleanup(fs_info);
+
if (!sb_rdonly(fs_info->sb)) {
/*
* The cleaner kthread is stopped, so do one final pass over
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 1a8bf943c3e7..2c12366cfde5 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -32,6 +32,7 @@
#include "block-rsv.h"
#include "delalloc-space.h"
#include "block-group.h"
+#include "discard.h"
#undef SCRAMBLE_DELAYED_REFS
@@ -2934,6 +2935,9 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
cond_resched();
}
+ if (btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_schedule_work(&fs_info->discard_ctl, true);
+
/*
* Transaction is finished. We don't need the lock anymore. We
* do need to clean up the block groups in case of a transaction
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 7108962e208d..0a868b6df893 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -21,6 +21,7 @@
#include "space-info.h"
#include "delalloc-space.h"
#include "block-group.h"
+#include "discard.h"
#define BITS_PER_BITMAP (PAGE_SIZE * 8UL)
#define MAX_CACHE_BYTES_PER_GIG SZ_32K
@@ -755,9 +756,11 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
/*
* Sync discard ensures that the free space cache is always
* trimmed. So when reading this in, the state should reflect
- * that.
+ * that. We also do this for async as a stop gap for lack of
+ * persistence.
*/
- if (btrfs_test_opt(fs_info, DISCARD_SYNC))
+ if (btrfs_test_opt(fs_info, DISCARD_SYNC) ||
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
e->trim_state = BTRFS_TRIM_STATE_TRIMMED;
if (!e->bytes) {
@@ -2382,6 +2385,7 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
u64 offset, u64 bytes,
enum btrfs_trim_state trim_state)
{
+ struct btrfs_block_group *block_group = ctl->private;
struct btrfs_free_space *info;
int ret = 0;
@@ -2431,6 +2435,9 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
ASSERT(ret != -EEXIST);
}
+ if (trim_state != BTRFS_TRIM_STATE_TRIMMED)
+ btrfs_discard_queue_work(&fs_info->discard_ctl, block_group);
+
return ret;
}
@@ -2447,6 +2454,25 @@ int btrfs_add_free_space(struct btrfs_block_group *block_group,
bytenr, size, trim_state);
}
+/*
+ * This is a subtle distinction because when adding free space back in general,
+ * we want it to be added as untrimmed for async. But in the case where we add
+ * it on loading of a block group, we want to consider it trimmed.
+ */
+int btrfs_add_free_space_async_trimmed(struct btrfs_block_group *block_group,
+ u64 bytenr, u64 size)
+{
+ enum btrfs_trim_state trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
+
+ if (btrfs_test_opt(block_group->fs_info, DISCARD_SYNC) ||
+ btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
+ trim_state = BTRFS_TRIM_STATE_TRIMMED;
+
+ return __btrfs_add_free_space(block_group->fs_info,
+ block_group->free_space_ctl,
+ bytenr, size, trim_state);
+}
+
int btrfs_remove_free_space(struct btrfs_block_group *block_group,
u64 offset, u64 bytes)
{
@@ -3209,6 +3235,7 @@ void btrfs_init_free_cluster(struct btrfs_free_cluster *cluster)
static int do_trimming(struct btrfs_block_group *block_group,
u64 *total_trimmed, u64 start, u64 bytes,
u64 reserved_start, u64 reserved_bytes,
+ enum btrfs_trim_state reserved_trim_state,
struct btrfs_trim_range *trim_entry)
{
struct btrfs_space_info *space_info = block_group->space_info;
@@ -3216,6 +3243,9 @@ static int do_trimming(struct btrfs_block_group *block_group,
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
int ret;
int update = 0;
+ u64 end = start + bytes;
+ u64 reserved_end = reserved_start + reserved_bytes;
+ enum btrfs_trim_state trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
u64 trimmed = 0;
spin_lock(&space_info->lock);
@@ -3229,11 +3259,20 @@ static int do_trimming(struct btrfs_block_group *block_group,
spin_unlock(&space_info->lock);
ret = btrfs_discard_extent(fs_info, start, bytes, &trimmed);
- if (!ret)
+ if (!ret) {
*total_trimmed += trimmed;
+ trim_state = BTRFS_TRIM_STATE_TRIMMED;
+ }
mutex_lock(&ctl->cache_writeout_mutex);
- btrfs_add_free_space(block_group, reserved_start, reserved_bytes);
+ if (reserved_start < start)
+ __btrfs_add_free_space(fs_info, ctl, reserved_start,
+ start - reserved_start,
+ reserved_trim_state);
+ if (start + bytes < reserved_start + reserved_bytes)
+ __btrfs_add_free_space(fs_info, ctl, end, reserved_end - end,
+ reserved_trim_state);
+ __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
list_del(&trim_entry->list);
mutex_unlock(&ctl->cache_writeout_mutex);
@@ -3260,6 +3299,7 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
int ret = 0;
u64 extent_start;
u64 extent_bytes;
+ enum btrfs_trim_state extent_trim_state;
u64 bytes;
while (start < end) {
@@ -3301,6 +3341,7 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
extent_start = entry->offset;
extent_bytes = entry->bytes;
+ extent_trim_state = entry->trim_state;
start = max(start, extent_start);
bytes = min(extent_start + extent_bytes, end) - start;
if (bytes < minlen) {
@@ -3319,7 +3360,8 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
mutex_unlock(&ctl->cache_writeout_mutex);
ret = do_trimming(block_group, total_trimmed, start, bytes,
- extent_start, extent_bytes, &trim_entry);
+ extent_start, extent_bytes, extent_trim_state,
+ &trim_entry);
if (ret)
break;
next:
@@ -3445,7 +3487,7 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
mutex_unlock(&ctl->cache_writeout_mutex);
ret = do_trimming(block_group, total_trimmed, start, bytes,
- start, bytes, &trim_entry);
+ start, bytes, 0, &trim_entry);
if (ret) {
reset_trimming_bitmap(ctl, offset);
break;
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index 29d16f58b40b..a88c30cb3b2b 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -113,6 +113,8 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
enum btrfs_trim_state trim_state);
int btrfs_add_free_space(struct btrfs_block_group *block_group,
u64 bytenr, u64 size);
+int btrfs_add_free_space_async_trimmed(struct btrfs_block_group *block_group,
+ u64 bytenr, u64 size);
int btrfs_remove_free_space(struct btrfs_block_group *block_group,
u64 bytenr, u64 size);
void __btrfs_remove_free_space_cache(struct btrfs_free_space_ctl *ctl);
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index 08ac6a7a67f0..215b25012e49 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -46,6 +46,7 @@
#include "sysfs.h"
#include "tests/btrfs-tests.h"
#include "block-group.h"
+#include "discard.h"
#include "qgroup.h"
#define CREATE_TRACE_POINTS
@@ -146,6 +147,8 @@ void __btrfs_handle_fs_error(struct btrfs_fs_info *fs_info, const char *function
if (sb_rdonly(sb))
return;
+ btrfs_discard_stop(fs_info);
+
/* btrfs handle error by forcing the filesystem readonly */
sb->s_flags |= SB_RDONLY;
btrfs_info(fs_info, "forced readonly");
@@ -313,6 +316,7 @@ enum {
Opt_datasum, Opt_nodatasum,
Opt_defrag, Opt_nodefrag,
Opt_discard, Opt_nodiscard,
+ Opt_discard_mode,
Opt_nologreplay,
Opt_norecovery,
Opt_ratio,
@@ -376,6 +380,7 @@ static const match_table_t tokens = {
{Opt_nodefrag, "noautodefrag"},
{Opt_discard, "discard"},
{Opt_nodiscard, "nodiscard"},
+ {Opt_discard_mode, "discard=%s"},
{Opt_nologreplay, "nologreplay"},
{Opt_norecovery, "norecovery"},
{Opt_ratio, "metadata_ratio=%u"},
@@ -695,12 +700,26 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
info->metadata_ratio);
break;
case Opt_discard:
- btrfs_set_and_info(info, DISCARD_SYNC,
- "turning on sync discard");
+ case Opt_discard_mode:
+ if (token == Opt_discard ||
+ strcmp(args[0].from, "sync") == 0) {
+ btrfs_clear_opt(info->mount_opt, DISCARD_ASYNC);
+ btrfs_set_and_info(info, DISCARD_SYNC,
+ "turning on sync discard");
+ } else if (strcmp(args[0].from, "async") == 0) {
+ btrfs_clear_opt(info->mount_opt, DISCARD_SYNC);
+ btrfs_set_and_info(info, DISCARD_ASYNC,
+ "turning on async discard");
+ } else {
+ ret = -EINVAL;
+ goto out;
+ }
break;
case Opt_nodiscard:
btrfs_clear_and_info(info, DISCARD_SYNC,
"turning off discard");
+ btrfs_clear_and_info(info, DISCARD_ASYNC,
+ "turning off async discard");
break;
case Opt_space_cache:
case Opt_space_cache_version:
@@ -1324,6 +1343,8 @@ static int btrfs_show_options(struct seq_file *seq, struct dentry *dentry)
seq_puts(seq, ",flushoncommit");
if (btrfs_test_opt(info, DISCARD_SYNC))
seq_puts(seq, ",discard");
+ if (btrfs_test_opt(info, DISCARD_ASYNC))
+ seq_puts(seq, ",discard=async");
if (!(info->sb->s_flags & SB_POSIXACL))
seq_puts(seq, ",noacl");
if (btrfs_test_opt(info, SPACE_CACHE))
@@ -1713,6 +1734,14 @@ static inline void btrfs_remount_cleanup(struct btrfs_fs_info *fs_info,
btrfs_cleanup_defrag_inodes(fs_info);
}
+ /* If we toggled discard async. */
+ if (!btrfs_raw_test_opt(old_opts, DISCARD_ASYNC) &&
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_resume(fs_info);
+ else if (btrfs_raw_test_opt(old_opts, DISCARD_ASYNC) &&
+ !btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_cleanup(fs_info);
+
clear_bit(BTRFS_FS_STATE_REMOUNTING, &fs_info->fs_state);
}
@@ -1760,6 +1789,8 @@ static int btrfs_remount(struct super_block *sb, int *flags, char *data)
*/
cancel_work_sync(&fs_info->async_reclaim_work);
+ btrfs_discard_cleanup(fs_info);
+
/* wait for the uuid_scan task to finish */
down(&fs_info->uuid_tree_rescan_sem);
/* avoid complains from lockdep et al. */
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index 66377e678504..65e78e59d5c4 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -30,6 +30,7 @@
#include "tree-checker.h"
#include "space-info.h"
#include "block-group.h"
+#include "discard.h"
const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = {
[BTRFS_RAID_RAID10] = {
@@ -2869,6 +2870,7 @@ static int btrfs_relocate_chunk(struct btrfs_fs_info *fs_info, u64 chunk_offset)
{
struct btrfs_root *root = fs_info->chunk_root;
struct btrfs_trans_handle *trans;
+ struct btrfs_block_group *block_group;
int ret;
/*
@@ -2892,6 +2894,11 @@ static int btrfs_relocate_chunk(struct btrfs_fs_info *fs_info, u64 chunk_offset)
if (ret)
return ret;
+ block_group = btrfs_lookup_block_group(fs_info, chunk_offset);
+ if (!block_group)
+ return -ENOENT;
+ btrfs_discard_cancel_work(&fs_info->discard_ctl, block_group);
+
trans = btrfs_start_trans_remove_block_group(root->fs_info,
chunk_offset);
if (IS_ERR(trans)) {
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 06/22] btrfs: handle empty block_group removal
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (4 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 07/22] btrfs: discard one region at a time in async discard Dennis Zhou
` (17 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
block_group removal is a little tricky. It can race with the extent
allocator, the cleaner thread, and balancing. The current path is for a
block_group to be added to the unused_bgs list. Then, when the cleaner
thread comes around, it starts a transaction and then proceeds with
removing the block_group. Extents that are pinned are subsequently
removed from the pinned trees and then eventually a discard is issued
for the entire block_group.
Async discard introduces another player into the game, the discard
workqueue. While it has none of the racing issues, the new problem is
ensuring we don't leave free space untrimmed prior to forgetting the
block_group. This is handled by placing fully free block_groups on a
separate discard queue. This is necessary to maintain discarding order
as in the future we will slowly trim even fully free block_groups. The
ordering helps us make progress on the same block_group rather than say
the last fully freed block_group or needing to search through the fully
freed block groups at the beginning of a list and insert after.
The new order of events is a fully freed block group gets placed on the
unused discard queue first. Once it's processed, it will be placed on
the unusued_bgs list and then the original sequence of events will
happen, just without the final whole block_group discard.
The mount flags can change when processing unused_bgs, so when flipping
from DISCARD to DISCARD_ASYNC, the unused_bgs must be punted to the
discard_list to be trimmed. If we flip off DISCARD_ASYNC, we punt
free block groups on the discard_list to the unused_bg queue which will
do the final discard for us.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/block-group.c | 48 ++++++++++++++-
fs/btrfs/ctree.h | 9 ++-
fs/btrfs/discard.c | 120 +++++++++++++++++++++++++++++++++++-
fs/btrfs/discard.h | 3 +
fs/btrfs/free-space-cache.c | 34 ++++++++++
fs/btrfs/free-space-cache.h | 1 +
fs/btrfs/scrub.c | 7 ++-
7 files changed, 213 insertions(+), 9 deletions(-)
diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c
index c1b1b59343bd..a0a15fad0ac4 100644
--- a/fs/btrfs/block-group.c
+++ b/fs/btrfs/block-group.c
@@ -1246,6 +1246,7 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
struct btrfs_block_group *block_group;
struct btrfs_space_info *space_info;
struct btrfs_trans_handle *trans;
+ bool async_trim_enabled = btrfs_test_opt(fs_info, DISCARD_ASYNC);
int ret = 0;
if (!test_bit(BTRFS_FS_OPEN, &fs_info->flags))
@@ -1275,6 +1276,22 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
/* Don't want to race with allocators so take the groups_sem */
down_write(&space_info->groups_sem);
+
+ /*
+ * Async discard moves the final block group discard to be prior
+ * to the unused_bgs code path. Therefore, if it's not fully
+ * trimmed, punt it back to the async discard lists.
+ */
+ if (btrfs_test_opt(fs_info, DISCARD_ASYNC) &&
+ !btrfs_is_free_space_trimmed(block_group)) {
+ trace_btrfs_skip_unused_block_group(block_group);
+ up_write(&space_info->groups_sem);
+ /* Requeue if we failed because of async discard. */
+ btrfs_discard_queue_work(&fs_info->discard_ctl,
+ block_group);
+ goto next;
+ }
+
spin_lock(&block_group->lock);
if (block_group->reserved || block_group->pinned ||
block_group->used || block_group->ro ||
@@ -1378,6 +1395,17 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
spin_unlock(&block_group->lock);
spin_unlock(&space_info->lock);
+ /*
+ * The normal path here is an unused block group is passed here,
+ * then trimming is handled in the transaction commit path.
+ * Async discard interposes before this to do the trimming
+ * before coming down the unused block group path as trimming
+ * will no longer be done later in the transaction commit path.
+ */
+ if (!async_trim_enabled &&
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ goto flip_async;
+
/* DISCARD can flip during remount */
trimming = btrfs_test_opt(fs_info, DISCARD_SYNC);
@@ -1422,6 +1450,13 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
spin_lock(&fs_info->unused_bgs_lock);
}
spin_unlock(&fs_info->unused_bgs_lock);
+ return;
+
+flip_async:
+ btrfs_end_transaction(trans);
+ mutex_unlock(&fs_info->delete_unused_bgs_mutex);
+ btrfs_put_block_group(block_group);
+ btrfs_discard_punt_unused_bgs_list(fs_info);
}
void btrfs_mark_bg_unused(struct btrfs_block_group *bg)
@@ -1626,6 +1661,8 @@ static struct btrfs_block_group *btrfs_create_block_group_cache(
cache->full_stripe_len = btrfs_full_stripe_len(fs_info, start);
set_free_space_tree_thresholds(cache);
+ cache->discard_index = BTRFS_DISCARD_INDEX_UNUSED;
+
atomic_set(&cache->count, 1);
spin_lock_init(&cache->lock);
init_rwsem(&cache->data_rwsem);
@@ -1792,7 +1829,10 @@ static int read_one_block_group(struct btrfs_fs_info *info,
inc_block_group_ro(cache, 1);
} else if (cache->used == 0) {
ASSERT(list_empty(&cache->bg_list));
- btrfs_mark_bg_unused(cache);
+ if (btrfs_test_opt(info, DISCARD_ASYNC))
+ btrfs_discard_queue_work(&info->discard_ctl, cache);
+ else
+ btrfs_mark_bg_unused(cache);
}
return 0;
error:
@@ -2755,8 +2795,10 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans,
* dirty list to avoid races between cleaner kthread and space
* cache writeout.
*/
- if (!alloc && old_val == 0)
- btrfs_mark_bg_unused(cache);
+ if (!alloc && old_val == 0) {
+ if (!btrfs_test_opt(info, DISCARD_ASYNC))
+ btrfs_mark_bg_unused(cache);
+ }
btrfs_put_block_group(cache);
total -= num_bytes;
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index f7b429277089..d15a4aa721aa 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -443,9 +443,14 @@ struct btrfs_full_stripe_locks_tree {
/* Discard control. */
/*
* Async discard uses multiple lists to differentiate the discard filter
- * parameters.
+ * parameters. Index 0 is for completely free block groups where we need to
+ * ensure the entire block group is trimmed without being lossy. Indices
+ * afterwards represent monotonically decreasing discard filter sizes to
+ * prioritize what should be discarded next.
*/
-#define BTRFS_NR_DISCARD_LISTS 1
+#define BTRFS_NR_DISCARD_LISTS 2
+#define BTRFS_DISCARD_INDEX_UNUSED 0
+#define BTRFS_DISCARD_INDEX_START 1
struct btrfs_discard_ctl {
struct workqueue_struct *discard_workers;
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index 692d64025802..752b38642b6b 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -13,6 +13,7 @@
/* This is an initial delay to give some chance for lba reuse. */
#define BTRFS_DISCARD_DELAY (120ULL * NSEC_PER_SEC)
+#define BTRFS_DISCARD_UNUSED_DELAY (10ULL * NSEC_PER_SEC)
static struct list_head *get_discard_list(struct btrfs_discard_ctl *discard_ctl,
struct btrfs_block_group *block_group)
@@ -30,9 +31,13 @@ static void add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
return;
}
- if (list_empty(&block_group->discard_list))
+ if (list_empty(&block_group->discard_list) ||
+ block_group->discard_index == BTRFS_DISCARD_INDEX_UNUSED) {
+ if (block_group->discard_index == BTRFS_DISCARD_INDEX_UNUSED)
+ block_group->discard_index = BTRFS_DISCARD_INDEX_START;
block_group->discard_eligible_time = (ktime_get_ns() +
BTRFS_DISCARD_DELAY);
+ }
list_move_tail(&block_group->discard_list,
get_discard_list(discard_ctl, block_group));
@@ -40,6 +45,27 @@ static void add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
spin_unlock(&discard_ctl->lock);
}
+static void add_to_discard_unused_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ spin_lock(&discard_ctl->lock);
+
+ if (!btrfs_run_discard_work(discard_ctl)) {
+ spin_unlock(&discard_ctl->lock);
+ return;
+ }
+
+ list_del_init(&block_group->discard_list);
+
+ block_group->discard_index = BTRFS_DISCARD_INDEX_UNUSED;
+ block_group->discard_eligible_time = (ktime_get_ns() +
+ BTRFS_DISCARD_UNUSED_DELAY);
+ list_add_tail(&block_group->discard_list,
+ &discard_ctl->discard_list[BTRFS_DISCARD_INDEX_UNUSED]);
+
+ spin_unlock(&discard_ctl->lock);
+}
+
static bool remove_from_discard_list(struct btrfs_discard_ctl *discard_ctl,
struct btrfs_block_group *block_group)
{
@@ -155,7 +181,10 @@ void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
!btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
return;
- add_to_discard_list(discard_ctl, block_group);
+ if (block_group->used == 0)
+ add_to_discard_unused_list(discard_ctl, block_group);
+ else
+ add_to_discard_list(discard_ctl, block_group);
if (!delayed_work_pending(&discard_ctl->work))
btrfs_discard_schedule_work(discard_ctl, false);
@@ -201,6 +230,29 @@ void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
spin_unlock(&discard_ctl->lock);
}
+/**
+ * btrfs_finish_discard_pass - determine next step of a block_group
+ * @discard_ctl: discard control
+ * @block_group: block_group of interest
+ *
+ * This determines the next step for a block group after it's finished going
+ * through a pass on a discard list. If it is unused and fully trimmed, we can
+ * mark it unused and send it to the unused_bgs path. Otherwise, pass it onto
+ * the appropriate filter list or let it fall off.
+ */
+static void btrfs_finish_discard_pass(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ remove_from_discard_list(discard_ctl, block_group);
+
+ if (block_group->used == 0) {
+ if (btrfs_is_free_space_trimmed(block_group))
+ btrfs_mark_bg_unused(block_group);
+ else
+ add_to_discard_unused_list(discard_ctl, block_group);
+ }
+}
+
/**
* btrfs_discard_workfn - discard work function
* @work: work
@@ -222,7 +274,7 @@ static void btrfs_discard_workfn(struct work_struct *work)
btrfs_trim_block_group(block_group, &trimmed, block_group->start,
btrfs_block_group_end(block_group), 0);
- remove_from_discard_list(discard_ctl, block_group);
+ btrfs_finish_discard_pass(discard_ctl, block_group);
btrfs_discard_schedule_work(discard_ctl, false);
}
@@ -243,6 +295,64 @@ bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl)
test_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags));
}
+/**
+ * btrfs_discard_punt_unused_bgs_list - punt unused_bgs list to discard lists
+ * @fs_info: fs_info of interest
+ *
+ * The unused_bgs list needs to be punted to the discard lists because the
+ * order of operations is changed. In the normal sychronous discard path, the
+ * block groups are trimmed via a single large trim in transaction commit. This
+ * is ultimately what we are trying to avoid with asynchronous discard. Thus,
+ * it must be done before going down the unused_bgs path.
+ */
+void btrfs_discard_punt_unused_bgs_list(struct btrfs_fs_info *fs_info)
+{
+ struct btrfs_block_group *block_group, *next;
+
+ spin_lock(&fs_info->unused_bgs_lock);
+
+ /* We enabled async discard, so punt all to the queue. */
+ list_for_each_entry_safe(block_group, next, &fs_info->unused_bgs,
+ bg_list) {
+ list_del_init(&block_group->bg_list);
+ btrfs_discard_queue_work(&fs_info->discard_ctl, block_group);
+ }
+
+ spin_unlock(&fs_info->unused_bgs_lock);
+}
+
+/**
+ * btrfs_discard_purge_list - purge discard lists
+ * @discard_ctl: discard control
+ *
+ * If we are disabling async discard, we may have intercepted block groups that
+ * are completely free and ready for the unused_bgs path. As discarding will
+ * now happen in transaction commit or not at all, we can safely mark the
+ * corresponding block groups as unused and they will be sent on their merry
+ * way to the unused_bgs list.
+ */
+static void btrfs_discard_purge_list(struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_block_group *block_group, *next;
+ int i;
+
+ spin_lock(&discard_ctl->lock);
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++) {
+ list_for_each_entry_safe(block_group, next,
+ &discard_ctl->discard_list[i],
+ discard_list) {
+ list_del_init(&block_group->discard_list);
+ spin_unlock(&discard_ctl->lock);
+ if (block_group->used == 0)
+ btrfs_mark_bg_unused(block_group);
+ spin_lock(&discard_ctl->lock);
+ }
+ }
+
+ spin_unlock(&discard_ctl->lock);
+}
+
void btrfs_discard_resume(struct btrfs_fs_info *fs_info)
{
if (!btrfs_test_opt(fs_info, DISCARD_ASYNC)) {
@@ -250,6 +360,8 @@ void btrfs_discard_resume(struct btrfs_fs_info *fs_info)
return;
}
+ btrfs_discard_punt_unused_bgs_list(fs_info);
+
set_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
}
@@ -275,4 +387,6 @@ void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
{
btrfs_discard_stop(fs_info);
cancel_delayed_work_sync(&fs_info->discard_ctl.work);
+
+ btrfs_discard_purge_list(&fs_info->discard_ctl);
}
diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
index f3775e84d35a..a13eb9d86ccf 100644
--- a/fs/btrfs/discard.h
+++ b/fs/btrfs/discard.h
@@ -7,6 +7,7 @@ struct btrfs_fs_info;
struct btrfs_discard_ctl;
struct btrfs_block_group;
+/* Work operations. */
void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
struct btrfs_block_group *block_group);
void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
@@ -15,6 +16,8 @@ void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
bool override);
bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl);
+/* Setup/Cleanup operations. */
+void btrfs_discard_punt_unused_bgs_list(struct btrfs_fs_info *fs_info);
void btrfs_discard_resume(struct btrfs_fs_info *fs_info);
void btrfs_discard_stop(struct btrfs_fs_info *fs_info);
void btrfs_discard_init(struct btrfs_fs_info *fs_info);
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 0a868b6df893..af0092cafc85 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -2708,6 +2708,37 @@ void btrfs_remove_free_space_cache(struct btrfs_block_group *block_group)
}
+/**
+ * btrfs_is_free_space_trimmed - see if everything is trimmed
+ * @block_group: block_group of interest
+ *
+ * Walk @block_group's free space rb_tree to determine if everything is trimmed.
+ */
+bool btrfs_is_free_space_trimmed(struct btrfs_block_group *block_group)
+{
+ struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
+ struct btrfs_free_space *info;
+ struct rb_node *node;
+ bool ret = true;
+
+ spin_lock(&ctl->tree_lock);
+ node = rb_first(&ctl->free_space_offset);
+
+ while (node) {
+ info = rb_entry(node, struct btrfs_free_space, offset_index);
+
+ if (!btrfs_free_space_trimmed(info)) {
+ ret = false;
+ break;
+ }
+
+ node = rb_next(node);
+ }
+
+ spin_unlock(&ctl->tree_lock);
+ return ret;
+}
+
u64 btrfs_find_space_for_alloc(struct btrfs_block_group *block_group,
u64 offset, u64 bytes, u64 empty_size,
u64 *max_extent_size)
@@ -2794,6 +2825,9 @@ int btrfs_return_cluster_to_free_space(
ret = __btrfs_return_cluster_to_free_space(block_group, cluster);
spin_unlock(&ctl->tree_lock);
+ btrfs_discard_queue_work(&block_group->fs_info->discard_ctl,
+ block_group);
+
/* finally drop our ref */
btrfs_put_block_group(block_group);
return ret;
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index a88c30cb3b2b..3100c7d5e646 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -119,6 +119,7 @@ int btrfs_remove_free_space(struct btrfs_block_group *block_group,
u64 bytenr, u64 size);
void __btrfs_remove_free_space_cache(struct btrfs_free_space_ctl *ctl);
void btrfs_remove_free_space_cache(struct btrfs_block_group *block_group);
+bool btrfs_is_free_space_trimmed(struct btrfs_block_group *block_group);
u64 btrfs_find_space_for_alloc(struct btrfs_block_group *block_group,
u64 offset, u64 bytes, u64 empty_size,
u64 *max_extent_size);
diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c
index 21de630b0730..22cf69e6e5bc 100644
--- a/fs/btrfs/scrub.c
+++ b/fs/btrfs/scrub.c
@@ -8,6 +8,7 @@
#include <linux/sched/mm.h>
#include <crypto/hash.h>
#include "ctree.h"
+#include "discard.h"
#include "volumes.h"
#include "disk-io.h"
#include "ordered-data.h"
@@ -3659,7 +3660,11 @@ int scrub_enumerate_chunks(struct scrub_ctx *sctx,
if (!cache->removed && !cache->ro && cache->reserved == 0 &&
cache->used == 0) {
spin_unlock(&cache->lock);
- btrfs_mark_bg_unused(cache);
+ if (btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_queue_work(&fs_info->discard_ctl,
+ cache);
+ else
+ btrfs_mark_bg_unused(cache);
} else {
spin_unlock(&cache->lock);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 07/22] btrfs: discard one region at a time in async discard
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (5 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 06/22] btrfs: handle empty block_group removal Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 08/22] btrfs: add removal calls for sysfs debug/ Dennis Zhou
` (16 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
The prior two patches added discarding via a background workqueue. This
just piggybacked off of the fstrim code to trim the whole block at once.
Well inevitably this is worse performance wise and will aggressively
overtrim. But it was nice to plumb the other infrastructure to keep the
patches easier to review.
This adds the real goal of this series which is discarding slowly (ie a
slow long running fstrim). The discarding is split into two phases,
extents and then bitmaps. The reason for this is two fold. First, the
bitmap regions overlap the extent regions. Second, discarding the
extents first will let the newly trimmed bitmaps have the highest chance
of coalescing when being readded to the free space cache.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/block-group.h | 15 +++++
fs/btrfs/discard.c | 85 ++++++++++++++++++-----
fs/btrfs/free-space-cache.c | 131 ++++++++++++++++++++++++++++--------
fs/btrfs/free-space-cache.h | 6 ++
4 files changed, 192 insertions(+), 45 deletions(-)
diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h
index 884defd61dcd..601e1d217e22 100644
--- a/fs/btrfs/block-group.h
+++ b/fs/btrfs/block-group.h
@@ -12,6 +12,19 @@ enum btrfs_disk_cache_state {
BTRFS_DC_SETUP,
};
+/*
+ * This describes the state of the block_group for async discard. This is due
+ * to the two pass nature of it where extent discarding is prioritized over
+ * bitmap discarding. BTRFS_DISCARD_RESET_CURSOR is set when we are resetting
+ * between lists to prevent contention for discard state variables
+ * (eg discard_cursor).
+ */
+enum btrfs_discard_state {
+ BTRFS_DISCARD_EXTENTS,
+ BTRFS_DISCARD_BITMAPS,
+ BTRFS_DISCARD_RESET_CURSOR,
+};
+
/*
* Control flags for do_chunk_alloc's force field CHUNK_ALLOC_NO_FORCE means to
* only allocate a chunk if we really need one.
@@ -121,6 +134,8 @@ struct btrfs_block_group {
struct list_head discard_list;
int discard_index;
u64 discard_eligible_time;
+ u64 discard_cursor;
+ enum btrfs_discard_state discard_state;
/* For dirty block groups */
struct list_head dirty_list;
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index 752b38642b6b..a7a091f4af4b 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -21,15 +21,11 @@ static struct list_head *get_discard_list(struct btrfs_discard_ctl *discard_ctl,
return &discard_ctl->discard_list[block_group->discard_index];
}
-static void add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
- struct btrfs_block_group *block_group)
+static void __add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
{
- spin_lock(&discard_ctl->lock);
-
- if (!btrfs_run_discard_work(discard_ctl)) {
- spin_unlock(&discard_ctl->lock);
+ if (!btrfs_run_discard_work(discard_ctl))
return;
- }
if (list_empty(&block_group->discard_list) ||
block_group->discard_index == BTRFS_DISCARD_INDEX_UNUSED) {
@@ -37,10 +33,19 @@ static void add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
block_group->discard_index = BTRFS_DISCARD_INDEX_START;
block_group->discard_eligible_time = (ktime_get_ns() +
BTRFS_DISCARD_DELAY);
+ block_group->discard_state = BTRFS_DISCARD_RESET_CURSOR;
}
list_move_tail(&block_group->discard_list,
get_discard_list(discard_ctl, block_group));
+}
+
+static void add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ spin_lock(&discard_ctl->lock);
+
+ __add_to_discard_list(discard_ctl, block_group);
spin_unlock(&discard_ctl->lock);
}
@@ -60,6 +65,7 @@ static void add_to_discard_unused_list(struct btrfs_discard_ctl *discard_ctl,
block_group->discard_index = BTRFS_DISCARD_INDEX_UNUSED;
block_group->discard_eligible_time = (ktime_get_ns() +
BTRFS_DISCARD_UNUSED_DELAY);
+ block_group->discard_state = BTRFS_DISCARD_RESET_CURSOR;
list_add_tail(&block_group->discard_list,
&discard_ctl->discard_list[BTRFS_DISCARD_INDEX_UNUSED]);
@@ -127,23 +133,41 @@ static struct btrfs_block_group *find_next_block_group(
/**
* peek_discard_list - wrap find_next_block_group()
* @discard_ctl: discard control
+ * @discard_state: the discard_state of the block_group after state management
*
* This wraps find_next_block_group() and sets the block_group to be in use.
+ * discard_state's control flow is managed here. Variables related to
+ * discard_state are reset here as needed (eg discard_cursor). @discard_state
+ * is remembered as it may change while we're discarding, but we want the
+ * discard to execute in the context determined here.
*/
static struct btrfs_block_group *peek_discard_list(
- struct btrfs_discard_ctl *discard_ctl)
+ struct btrfs_discard_ctl *discard_ctl,
+ enum btrfs_discard_state *discard_state)
{
struct btrfs_block_group *block_group;
u64 now = ktime_get_ns();
spin_lock(&discard_ctl->lock);
+again:
block_group = find_next_block_group(discard_ctl, now);
- if (block_group && now < block_group->discard_eligible_time)
+ if (block_group && now > block_group->discard_eligible_time) {
+ if (block_group->discard_index == BTRFS_DISCARD_INDEX_UNUSED &&
+ block_group->used != 0) {
+ __add_to_discard_list(discard_ctl, block_group);
+ goto again;
+ }
+ if (block_group->discard_state == BTRFS_DISCARD_RESET_CURSOR) {
+ block_group->discard_cursor = block_group->start;
+ block_group->discard_state = BTRFS_DISCARD_EXTENTS;
+ }
+ discard_ctl->block_group = block_group;
+ *discard_state = block_group->discard_state;
+ } else {
block_group = NULL;
-
- discard_ctl->block_group = block_group;
+ }
spin_unlock(&discard_ctl->lock);
@@ -257,24 +281,53 @@ static void btrfs_finish_discard_pass(struct btrfs_discard_ctl *discard_ctl,
* btrfs_discard_workfn - discard work function
* @work: work
*
- * This finds the next block_group to start discarding and then discards it.
+ * This finds the next block_group to start discarding and then discards a
+ * single region. It does this in a two-pass fashion: first extents and second
+ * bitmaps. Completely discarded block groups are sent to the unused_bgs path.
*/
static void btrfs_discard_workfn(struct work_struct *work)
{
struct btrfs_discard_ctl *discard_ctl;
struct btrfs_block_group *block_group;
+ enum btrfs_discard_state discard_state;
u64 trimmed = 0;
discard_ctl = container_of(work, struct btrfs_discard_ctl, work.work);
- block_group = peek_discard_list(discard_ctl);
+ block_group = peek_discard_list(discard_ctl, &discard_state);
if (!block_group || !btrfs_run_discard_work(discard_ctl))
return;
- btrfs_trim_block_group(block_group, &trimmed, block_group->start,
- btrfs_block_group_end(block_group), 0);
+ /* Perform discarding. */
+ if (discard_state == BTRFS_DISCARD_BITMAPS)
+ btrfs_trim_block_group_bitmaps(block_group, &trimmed,
+ block_group->discard_cursor,
+ btrfs_block_group_end(block_group),
+ 0, true);
+ else
+ btrfs_trim_block_group_extents(block_group, &trimmed,
+ block_group->discard_cursor,
+ btrfs_block_group_end(block_group),
+ 0, true);
+
+ /* Determine next steps for a block_group. */
+ if (block_group->discard_cursor >= btrfs_block_group_end(block_group)) {
+ if (discard_state == BTRFS_DISCARD_BITMAPS) {
+ btrfs_finish_discard_pass(discard_ctl, block_group);
+ } else {
+ block_group->discard_cursor = block_group->start;
+ spin_lock(&discard_ctl->lock);
+ if (block_group->discard_state !=
+ BTRFS_DISCARD_RESET_CURSOR)
+ block_group->discard_state =
+ BTRFS_DISCARD_BITMAPS;
+ spin_unlock(&discard_ctl->lock);
+ }
+ }
- btrfs_finish_discard_pass(discard_ctl, block_group);
+ spin_lock(&discard_ctl->lock);
+ discard_ctl->block_group = NULL;
+ spin_unlock(&discard_ctl->lock);
btrfs_discard_schedule_work(discard_ctl, false);
}
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index af0092cafc85..6e07387622cf 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -3324,8 +3324,12 @@ static int do_trimming(struct btrfs_block_group *block_group,
return ret;
}
+/*
+ * If @async is set, then we will trim 1 region and return.
+ */
static int trim_no_bitmap(struct btrfs_block_group *block_group,
- u64 *total_trimmed, u64 start, u64 end, u64 minlen)
+ u64 *total_trimmed, u64 start, u64 end, u64 minlen,
+ bool async)
{
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
struct btrfs_free_space *entry;
@@ -3342,36 +3346,25 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
mutex_lock(&ctl->cache_writeout_mutex);
spin_lock(&ctl->tree_lock);
- if (ctl->free_space < minlen) {
- spin_unlock(&ctl->tree_lock);
- mutex_unlock(&ctl->cache_writeout_mutex);
- break;
- }
+ if (ctl->free_space < minlen)
+ goto out_unlock;
entry = tree_search_offset(ctl, start, 0, 1);
- if (!entry) {
- spin_unlock(&ctl->tree_lock);
- mutex_unlock(&ctl->cache_writeout_mutex);
- break;
- }
+ if (!entry)
+ goto out_unlock;
- /* skip bitmaps */
- while (entry->bitmap) {
+ /* skip bitmaps and if async, already trimmed entries */
+ while (entry->bitmap ||
+ (async && btrfs_free_space_trimmed(entry))) {
node = rb_next(&entry->offset_index);
- if (!node) {
- spin_unlock(&ctl->tree_lock);
- mutex_unlock(&ctl->cache_writeout_mutex);
- goto out;
- }
+ if (!node)
+ goto out_unlock;
entry = rb_entry(node, struct btrfs_free_space,
offset_index);
}
- if (entry->offset >= end) {
- spin_unlock(&ctl->tree_lock);
- mutex_unlock(&ctl->cache_writeout_mutex);
- break;
- }
+ if (entry->offset >= end)
+ goto out_unlock;
extent_start = entry->offset;
extent_bytes = entry->bytes;
@@ -3396,10 +3389,15 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
ret = do_trimming(block_group, total_trimmed, start, bytes,
extent_start, extent_bytes, extent_trim_state,
&trim_entry);
- if (ret)
+ if (ret) {
+ block_group->discard_cursor = start + bytes;
break;
+ }
next:
start += bytes;
+ block_group->discard_cursor = start;
+ if (async && *total_trimmed)
+ break;
if (fatal_signal_pending(current)) {
ret = -ERESTARTSYS;
@@ -3408,7 +3406,14 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
cond_resched();
}
-out:
+
+ return ret;
+
+out_unlock:
+ block_group->discard_cursor = btrfs_block_group_end(block_group);
+ spin_unlock(&ctl->tree_lock);
+ mutex_unlock(&ctl->cache_writeout_mutex);
+
return ret;
}
@@ -3445,8 +3450,12 @@ static void end_trimming_bitmap(struct btrfs_free_space *entry)
entry->trim_state = BTRFS_TRIM_STATE_TRIMMED;
}
+/*
+ * If @async is set, then we will trim 1 region and return.
+ */
static int trim_bitmaps(struct btrfs_block_group *block_group,
- u64 *total_trimmed, u64 start, u64 end, u64 minlen)
+ u64 *total_trimmed, u64 start, u64 end, u64 minlen,
+ bool async)
{
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
struct btrfs_free_space *entry;
@@ -3463,13 +3472,16 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
spin_lock(&ctl->tree_lock);
if (ctl->free_space < minlen) {
+ block_group->discard_cursor =
+ btrfs_block_group_end(block_group);
spin_unlock(&ctl->tree_lock);
mutex_unlock(&ctl->cache_writeout_mutex);
break;
}
entry = tree_search_offset(ctl, offset, 1, 0);
- if (!entry || btrfs_free_space_trimmed(entry)) {
+ if (!entry || (async && start == offset &&
+ btrfs_free_space_trimmed(entry))) {
spin_unlock(&ctl->tree_lock);
mutex_unlock(&ctl->cache_writeout_mutex);
next_bitmap = true;
@@ -3502,6 +3514,16 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
goto next;
}
+ /*
+ * We already trimmed a region, but are using the locking above
+ * to reset the trim_state.
+ */
+ if (async && *total_trimmed) {
+ spin_unlock(&ctl->tree_lock);
+ mutex_unlock(&ctl->cache_writeout_mutex);
+ return ret;
+ }
+
bytes = min(bytes, end - start);
if (bytes < minlen) {
entry->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
@@ -3524,6 +3546,8 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
start, bytes, 0, &trim_entry);
if (ret) {
reset_trimming_bitmap(ctl, offset);
+ block_group->discard_cursor =
+ btrfs_block_group_end(block_group);
break;
}
next:
@@ -3533,6 +3557,7 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
} else {
start += bytes;
}
+ block_group->discard_cursor = start;
if (fatal_signal_pending(current)) {
if (start != offset)
@@ -3544,6 +3569,9 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
cond_resched();
}
+ if (offset >= end)
+ block_group->discard_cursor = end;
+
return ret;
}
@@ -3604,11 +3632,11 @@ int btrfs_trim_block_group(struct btrfs_block_group *block_group,
btrfs_get_block_group_trimming(block_group);
spin_unlock(&block_group->lock);
- ret = trim_no_bitmap(block_group, trimmed, start, end, minlen);
+ ret = trim_no_bitmap(block_group, trimmed, start, end, minlen, false);
if (ret)
goto out;
- ret = trim_bitmaps(block_group, trimmed, start, end, minlen);
+ ret = trim_bitmaps(block_group, trimmed, start, end, minlen, false);
div64_u64_rem(end, BITS_PER_BITMAP * ctl->unit, &rem);
/* If we ended in the middle of a bitmap, reset the trimming flag. */
if (rem)
@@ -3618,6 +3646,51 @@ int btrfs_trim_block_group(struct btrfs_block_group *block_group,
return ret;
}
+int btrfs_trim_block_group_extents(struct btrfs_block_group *block_group,
+ u64 *trimmed, u64 start, u64 end, u64 minlen,
+ bool async)
+{
+ int ret;
+
+ *trimmed = 0;
+
+ spin_lock(&block_group->lock);
+ if (block_group->removed) {
+ spin_unlock(&block_group->lock);
+ return 0;
+ }
+ btrfs_get_block_group_trimming(block_group);
+ spin_unlock(&block_group->lock);
+
+ ret = trim_no_bitmap(block_group, trimmed, start, end, minlen, async);
+
+ btrfs_put_block_group_trimming(block_group);
+ return ret;
+}
+
+int btrfs_trim_block_group_bitmaps(struct btrfs_block_group *block_group,
+ u64 *trimmed, u64 start, u64 end, u64 minlen,
+ bool async)
+{
+ int ret;
+
+ *trimmed = 0;
+
+ spin_lock(&block_group->lock);
+ if (block_group->removed) {
+ spin_unlock(&block_group->lock);
+ return 0;
+ }
+ btrfs_get_block_group_trimming(block_group);
+ spin_unlock(&block_group->lock);
+
+ ret = trim_bitmaps(block_group, trimmed, start, end, minlen, async);
+
+ btrfs_put_block_group_trimming(block_group);
+ return ret;
+
+}
+
/*
* Find the left-most item in the cache tree, and then return the
* smallest inode number in the item.
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index 3100c7d5e646..55d6f6aaeb9b 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -138,6 +138,12 @@ int btrfs_return_cluster_to_free_space(
struct btrfs_free_cluster *cluster);
int btrfs_trim_block_group(struct btrfs_block_group *block_group,
u64 *trimmed, u64 start, u64 end, u64 minlen);
+int btrfs_trim_block_group_extents(struct btrfs_block_group *block_group,
+ u64 *trimmed, u64 start, u64 end, u64 minlen,
+ bool async);
+int btrfs_trim_block_group_bitmaps(struct btrfs_block_group *block_group,
+ u64 *trimmed, u64 start, u64 end, u64 minlen,
+ bool async);
/* Support functions for running our sanity tests */
#ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 08/22] btrfs: add removal calls for sysfs debug/
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (6 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 07/22] btrfs: discard one region at a time in async discard Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-18 11:45 ` Anand Jain
2019-12-14 0:22 ` [PATCH 09/22] btrfs: make UUID/debug have its own kobject Dennis Zhou
` (15 subsequent siblings)
23 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
We probably should call sysfs_remove_group() on debug/.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/sysfs.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 16379f491ca1..4c022757ffa4 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -771,6 +771,10 @@ void btrfs_sysfs_remove_mounted(struct btrfs_fs_info *fs_info)
kobject_del(fs_info->space_info_kobj);
kobject_put(fs_info->space_info_kobj);
}
+#ifdef CONFIG_BTRFS_DEBUG
+ sysfs_remove_group(&fs_info->fs_devices->fsid_kobj,
+ &btrfs_debug_feature_attr_group);
+#endif
addrm_unknown_feature_attrs(fs_info, false);
sysfs_remove_group(&fs_info->fs_devices->fsid_kobj, &btrfs_feature_attr_group);
sysfs_remove_files(&fs_info->fs_devices->fsid_kobj, btrfs_attrs);
@@ -1205,6 +1209,9 @@ void __cold btrfs_exit_sysfs(void)
sysfs_unmerge_group(&btrfs_kset->kobj,
&btrfs_static_feature_attr_group);
sysfs_remove_group(&btrfs_kset->kobj, &btrfs_feature_attr_group);
+#ifdef CONFIG_BTRFS_DEBUG
+ sysfs_remove_group(&btrfs_kset->kobj, &btrfs_debug_feature_attr_group);
+#endif
kset_unregister(btrfs_kset);
}
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 09/22] btrfs: make UUID/debug have its own kobject
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (7 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 08/22] btrfs: add removal calls for sysfs debug/ Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-18 11:45 ` Anand Jain
2019-12-14 0:22 ` [PATCH 10/22] btrfs: add discard sysfs directory Dennis Zhou
` (14 subsequent siblings)
23 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Btrfs only allowed attributes to be exposed in debug/. Let's let other
groups be created by making debug its own kobject.
This also makes the per-fs debug options separate from the global
features mount attributes. This seems to be needed as
sysfs_create_files() requires const struct attribute * while
sysfs_create_group() can take struct attribute *. This seems nicer as
per file system, you'll probably use to_fs_info().
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/ctree.h | 4 ++++
fs/btrfs/sysfs.c | 20 ++++++++++++++++----
2 files changed, 20 insertions(+), 4 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index d15a4aa721aa..0626e5562993 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -927,6 +927,10 @@ struct btrfs_fs_info {
spinlock_t ref_verify_lock;
struct rb_root block_tree;
#endif
+
+#ifdef CONFIG_BTRFS_DEBUG
+ struct kobject *debug_kobj;
+#endif
};
static inline struct btrfs_fs_info *btrfs_sb(struct super_block *sb)
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 4c022757ffa4..3a56e0cdd673 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -344,6 +344,10 @@ static const struct attribute_group btrfs_static_feature_attr_group = {
* /sys/fs/btrfs/debug - applies to module or all filesystems
* /sys/fs/btrfs/UUID - applies only to the given filesystem
*/
+static const struct attribute *btrfs_debug_mount_attrs[] = {
+ NULL,
+};
+
static struct attribute *btrfs_debug_feature_attrs[] = {
NULL
};
@@ -772,8 +776,12 @@ void btrfs_sysfs_remove_mounted(struct btrfs_fs_info *fs_info)
kobject_put(fs_info->space_info_kobj);
}
#ifdef CONFIG_BTRFS_DEBUG
- sysfs_remove_group(&fs_info->fs_devices->fsid_kobj,
- &btrfs_debug_feature_attr_group);
+ if (fs_info->debug_kobj) {
+ sysfs_remove_files(fs_info->debug_kobj,
+ btrfs_debug_mount_attrs);
+ kobject_del(fs_info->debug_kobj);
+ kobject_put(fs_info->debug_kobj);
+ }
#endif
addrm_unknown_feature_attrs(fs_info, false);
sysfs_remove_group(&fs_info->fs_devices->fsid_kobj, &btrfs_feature_attr_group);
@@ -1111,8 +1119,12 @@ int btrfs_sysfs_add_mounted(struct btrfs_fs_info *fs_info)
goto failure;
#ifdef CONFIG_BTRFS_DEBUG
- error = sysfs_create_group(fsid_kobj,
- &btrfs_debug_feature_attr_group);
+ fs_info->debug_kobj = kobject_create_and_add("debug", fsid_kobj);
+ if (!fs_info->debug_kobj)
+ goto failure;
+
+ error = sysfs_create_files(fs_info->debug_kobj,
+ btrfs_debug_mount_attrs);
if (error)
goto failure;
#endif
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 10/22] btrfs: add discard sysfs directory
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (8 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 09/22] btrfs: make UUID/debug have its own kobject Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-18 11:45 ` Anand Jain
2019-12-14 0:22 ` [PATCH 11/22] btrfs: track discardable extents for async discard Dennis Zhou
` (13 subsequent siblings)
23 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Setup sysfs directory for discard stats + tunables.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/ctree.h | 1 +
fs/btrfs/sysfs.c | 26 ++++++++++++++++++++++++++
2 files changed, 27 insertions(+)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 0626e5562993..71317047c321 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -930,6 +930,7 @@ struct btrfs_fs_info {
#ifdef CONFIG_BTRFS_DEBUG
struct kobject *debug_kobj;
+ struct kobject *discard_debug_kobj;
#endif
};
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 3a56e0cdd673..bc5415f67987 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -338,6 +338,13 @@ static const struct attribute_group btrfs_static_feature_attr_group = {
#ifdef CONFIG_BTRFS_DEBUG
+/*
+ * Discard statistics and tunables.
+ */
+static const struct attribute *discard_debug_attrs[] = {
+ NULL,
+};
+
/*
* Runtime debugging exported via sysfs
*
@@ -776,6 +783,12 @@ void btrfs_sysfs_remove_mounted(struct btrfs_fs_info *fs_info)
kobject_put(fs_info->space_info_kobj);
}
#ifdef CONFIG_BTRFS_DEBUG
+ if (fs_info->discard_debug_kobj) {
+ sysfs_remove_files(fs_info->discard_debug_kobj,
+ discard_debug_attrs);
+ kobject_del(fs_info->discard_debug_kobj);
+ kobject_put(fs_info->discard_debug_kobj);
+ }
if (fs_info->debug_kobj) {
sysfs_remove_files(fs_info->debug_kobj,
btrfs_debug_mount_attrs);
@@ -1127,6 +1140,19 @@ int btrfs_sysfs_add_mounted(struct btrfs_fs_info *fs_info)
btrfs_debug_mount_attrs);
if (error)
goto failure;
+
+ /* Discard directory. */
+ fs_info->discard_debug_kobj = kobject_create_and_add("discard",
+ fs_info->debug_kobj);
+ if (!fs_info->discard_debug_kobj) {
+ error = -ENOMEM;
+ goto failure;
+ }
+
+ error = sysfs_create_files(fs_info->discard_debug_kobj,
+ discard_debug_attrs);
+ if (error)
+ goto failure;
#endif
error = addrm_unknown_feature_attrs(fs_info, true);
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 11/22] btrfs: track discardable extents for async discard
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (9 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 10/22] btrfs: add discard sysfs directory Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 12/22] btrfs: keep track of discardable_bytes Dennis Zhou
` (12 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
The number of discardable extents will serve as the rate limiting metric
for how often we should discard. This keeps track of discardable extents
in the free space caches by maintaining deltas and propagating them to
the global count.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
---
fs/btrfs/ctree.h | 10 ++++
fs/btrfs/discard.c | 32 +++++++++++
fs/btrfs/discard.h | 4 ++
fs/btrfs/free-space-cache.c | 106 +++++++++++++++++++++++++++++++++---
fs/btrfs/free-space-cache.h | 2 +
fs/btrfs/sysfs.c | 15 +++++
6 files changed, 160 insertions(+), 9 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 71317047c321..968bc8f5d0f8 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -101,6 +101,15 @@ struct btrfs_ref;
#define BTRFS_MAX_EXTENT_SIZE SZ_128M
+/*
+ * Deltas are an effective way to populate global statistics. Give macro names
+ * to make it clear what we're doing. An example is discard_extents in
+ * btrfs_free_space_ctl.
+ */
+#define BTRFS_STAT_NR_ENTRIES 2
+#define BTRFS_STAT_CURR 0
+#define BTRFS_STAT_PREV 1
+
/*
* Count how many BTRFS_MAX_EXTENT_SIZE cover the @size
@@ -458,6 +467,7 @@ struct btrfs_discard_ctl {
spinlock_t lock;
struct btrfs_block_group *block_group;
struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
+ atomic_t discardable_extents;
};
/* delayed seq elem */
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index a7a091f4af4b..1f18bbfe8372 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -348,6 +348,36 @@ bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl)
test_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags));
}
+/**
+ * btrfs_discard_update_discardable - propagate discard counters
+ * @block_group: block_group of interest
+ * @ctl: free_space_ctl of @block_group
+ *
+ * This propagates deltas of counters up to the discard_ctl. It maintains a
+ * current counter and a previous counter passing the delta up to the global
+ * stat. Then the current counter value becomes the previous counter value.
+ */
+void btrfs_discard_update_discardable(struct btrfs_block_group *block_group,
+ struct btrfs_free_space_ctl *ctl)
+{
+ struct btrfs_discard_ctl *discard_ctl;
+ s32 extents_delta;
+
+ if (!block_group ||
+ !btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
+ return;
+
+ discard_ctl = &block_group->fs_info->discard_ctl;
+
+ extents_delta = (ctl->discardable_extents[BTRFS_STAT_CURR] -
+ ctl->discardable_extents[BTRFS_STAT_PREV]);
+ if (extents_delta) {
+ atomic_add(extents_delta, &discard_ctl->discardable_extents);
+ ctl->discardable_extents[BTRFS_STAT_PREV] =
+ ctl->discardable_extents[BTRFS_STAT_CURR];
+ }
+}
+
/**
* btrfs_discard_punt_unused_bgs_list - punt unused_bgs list to discard lists
* @fs_info: fs_info of interest
@@ -434,6 +464,8 @@ void btrfs_discard_init(struct btrfs_fs_info *fs_info)
for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++)
INIT_LIST_HEAD(&discard_ctl->discard_list[i]);
+
+ atomic_set(&discard_ctl->discardable_extents, 0);
}
void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
index a13eb9d86ccf..7e3680dd82ce 100644
--- a/fs/btrfs/discard.h
+++ b/fs/btrfs/discard.h
@@ -16,6 +16,10 @@ void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
bool override);
bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl);
+/* Update operations. */
+void btrfs_discard_update_discardable(struct btrfs_block_group *block_group,
+ struct btrfs_free_space_ctl *ctl);
+
/* Setup/Cleanup operations. */
void btrfs_discard_punt_unused_bgs_list(struct btrfs_fs_info *fs_info);
void btrfs_discard_resume(struct btrfs_fs_info *fs_info);
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 6e07387622cf..7a3b07ef4dd5 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -32,6 +32,9 @@ struct btrfs_trim_range {
struct list_head list;
};
+static int count_bitmap_extents(struct btrfs_free_space_ctl *ctl,
+ struct btrfs_free_space *bitmap_info);
+
static int link_free_space(struct btrfs_free_space_ctl *ctl,
struct btrfs_free_space *info);
static void unlink_free_space(struct btrfs_free_space_ctl *ctl,
@@ -816,12 +819,17 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
ret = io_ctl_read_bitmap(&io_ctl, e);
if (ret)
goto free_cache;
+ e->bitmap_extents = count_bitmap_extents(ctl, e);
+ if (!btrfs_free_space_trimmed(e))
+ ctl->discardable_extents[BTRFS_STAT_CURR] +=
+ e->bitmap_extents;
}
io_ctl_drop_pages(&io_ctl);
merge_space_tree(ctl);
ret = 1;
out:
+ btrfs_discard_update_discardable(ctl->private, ctl);
io_ctl_free(&io_ctl);
return ret;
free_cache:
@@ -1635,6 +1643,9 @@ __unlink_free_space(struct btrfs_free_space_ctl *ctl,
{
rb_erase(&info->offset_index, &ctl->free_space_offset);
ctl->free_extents--;
+
+ if (!info->bitmap && !btrfs_free_space_trimmed(info))
+ ctl->discardable_extents[BTRFS_STAT_CURR]--;
}
static void unlink_free_space(struct btrfs_free_space_ctl *ctl,
@@ -1655,6 +1666,9 @@ static int link_free_space(struct btrfs_free_space_ctl *ctl,
if (ret)
return ret;
+ if (!info->bitmap && !btrfs_free_space_trimmed(info))
+ ctl->discardable_extents[BTRFS_STAT_CURR]++;
+
ctl->free_space += info->bytes;
ctl->free_extents++;
return ret;
@@ -1711,17 +1725,29 @@ static inline void __bitmap_clear_bits(struct btrfs_free_space_ctl *ctl,
struct btrfs_free_space *info,
u64 offset, u64 bytes)
{
- unsigned long start, count;
+ unsigned long start, count, end;
+ int extent_delta = -1;
start = offset_to_bit(info->offset, ctl->unit, offset);
count = bytes_to_bits(bytes, ctl->unit);
- ASSERT(start + count <= BITS_PER_BITMAP);
+ end = start + count;
+ ASSERT(end <= BITS_PER_BITMAP);
bitmap_clear(info->bitmap, start, count);
info->bytes -= bytes;
if (info->max_extent_size > ctl->unit)
info->max_extent_size = 0;
+
+ if (start && test_bit(start - 1, info->bitmap))
+ extent_delta++;
+
+ if (end < BITS_PER_BITMAP && test_bit(end, info->bitmap))
+ extent_delta++;
+
+ info->bitmap_extents += extent_delta;
+ if (!btrfs_free_space_trimmed(info))
+ ctl->discardable_extents[BTRFS_STAT_CURR] += extent_delta;
}
static void bitmap_clear_bits(struct btrfs_free_space_ctl *ctl,
@@ -1736,16 +1762,28 @@ static void bitmap_set_bits(struct btrfs_free_space_ctl *ctl,
struct btrfs_free_space *info, u64 offset,
u64 bytes)
{
- unsigned long start, count;
+ unsigned long start, count, end;
+ int extent_delta = 1;
start = offset_to_bit(info->offset, ctl->unit, offset);
count = bytes_to_bits(bytes, ctl->unit);
- ASSERT(start + count <= BITS_PER_BITMAP);
+ end = start + count;
+ ASSERT(end <= BITS_PER_BITMAP);
bitmap_set(info->bitmap, start, count);
info->bytes += bytes;
ctl->free_space += bytes;
+
+ if (start && test_bit(start - 1, info->bitmap))
+ extent_delta--;
+
+ if (end < BITS_PER_BITMAP && test_bit(end, info->bitmap))
+ extent_delta--;
+
+ info->bitmap_extents += extent_delta;
+ if (!btrfs_free_space_trimmed(info))
+ ctl->discardable_extents[BTRFS_STAT_CURR] += extent_delta;
}
/*
@@ -1881,11 +1919,35 @@ find_free_space(struct btrfs_free_space_ctl *ctl, u64 *offset, u64 *bytes,
return NULL;
}
+static int count_bitmap_extents(struct btrfs_free_space_ctl *ctl,
+ struct btrfs_free_space *bitmap_info)
+{
+ struct btrfs_block_group *block_group = ctl->private;
+ u64 bytes = bitmap_info->bytes;
+ unsigned int rs, re;
+ int count = 0;
+
+ if (!block_group || !bytes)
+ return count;
+
+ bitmap_for_each_set_region(bitmap_info->bitmap, rs, re, 0,
+ BITS_PER_BITMAP) {
+ bytes -= (rs - re) * ctl->unit;
+ count++;
+
+ if (!bytes)
+ break;
+ }
+
+ return count;
+}
+
static void add_new_bitmap(struct btrfs_free_space_ctl *ctl,
struct btrfs_free_space *info, u64 offset)
{
info->offset = offset_to_bitmap(ctl, offset);
info->bytes = 0;
+ info->bitmap_extents = 0;
INIT_LIST_HEAD(&info->list);
link_free_space(ctl, info);
ctl->total_bitmaps++;
@@ -1991,8 +2053,12 @@ static u64 add_bytes_to_bitmap(struct btrfs_free_space_ctl *ctl,
* This is a tradeoff to make bitmap trim state minimal. We mark the
* whole bitmap untrimmed if at any point we add untrimmed regions.
*/
- if (trim_state == BTRFS_TRIM_STATE_UNTRIMMED)
+ if (trim_state == BTRFS_TRIM_STATE_UNTRIMMED) {
+ if (btrfs_free_space_trimmed(info))
+ ctl->discardable_extents[BTRFS_STAT_CURR] +=
+ info->bitmap_extents;
info->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
+ }
end = info->offset + (u64)(BITS_PER_BITMAP * ctl->unit);
@@ -2428,6 +2494,7 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
if (ret)
kmem_cache_free(btrfs_free_space_cachep, info);
out:
+ btrfs_discard_update_discardable(block_group, ctl);
spin_unlock(&ctl->tree_lock);
if (ret) {
@@ -2561,6 +2628,7 @@ int btrfs_remove_free_space(struct btrfs_block_group *block_group,
goto again;
}
out_lock:
+ btrfs_discard_update_discardable(block_group, ctl);
spin_unlock(&ctl->tree_lock);
out:
return ret;
@@ -2646,8 +2714,16 @@ __btrfs_return_cluster_to_free_space(
bitmap = (entry->bitmap != NULL);
if (!bitmap) {
+ /* merging treats extents as if they were new */
+ if (!btrfs_free_space_trimmed(entry))
+ ctl->discardable_extents[BTRFS_STAT_CURR]--;
+
try_merge_free_space(ctl, entry, false);
steal_from_bitmap(ctl, entry, false);
+
+ /* as we insert directly, update these statistics */
+ if (!btrfs_free_space_trimmed(entry))
+ ctl->discardable_extents[BTRFS_STAT_CURR]++;
}
tree_insert_offset(&ctl->free_space_offset,
entry->offset, &entry->offset_index, bitmap);
@@ -2704,6 +2780,7 @@ void btrfs_remove_free_space_cache(struct btrfs_block_group *block_group)
cond_resched_lock(&ctl->tree_lock);
}
__btrfs_remove_free_space_cache_locked(ctl);
+ btrfs_discard_update_discardable(block_group, ctl);
spin_unlock(&ctl->tree_lock);
}
@@ -2778,6 +2855,7 @@ u64 btrfs_find_space_for_alloc(struct btrfs_block_group *block_group,
link_free_space(ctl, entry);
}
out:
+ btrfs_discard_update_discardable(block_group, ctl);
spin_unlock(&ctl->tree_lock);
if (align_gap_len)
@@ -2943,6 +3021,8 @@ u64 btrfs_alloc_from_cluster(struct btrfs_block_group *block_group,
entry->bitmap);
ctl->total_bitmaps--;
ctl->op->recalc_thresholds(ctl);
+ } else if (!btrfs_free_space_trimmed(entry)) {
+ ctl->discardable_extents[BTRFS_STAT_CURR]--;
}
kmem_cache_free(btrfs_free_space_cachep, entry);
}
@@ -3438,16 +3518,24 @@ static void reset_trimming_bitmap(struct btrfs_free_space_ctl *ctl, u64 offset)
spin_lock(&ctl->tree_lock);
entry = tree_search_offset(ctl, offset, 1, 0);
- if (entry)
+ if (entry) {
+ if (btrfs_free_space_trimmed(entry))
+ ctl->discardable_extents[BTRFS_STAT_CURR] +=
+ entry->bitmap_extents;
entry->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
+ }
spin_unlock(&ctl->tree_lock);
}
-static void end_trimming_bitmap(struct btrfs_free_space *entry)
+static void end_trimming_bitmap(struct btrfs_free_space_ctl *ctl,
+ struct btrfs_free_space *entry)
{
- if (btrfs_free_space_trimming_bitmap(entry))
+ if (btrfs_free_space_trimming_bitmap(entry)) {
entry->trim_state = BTRFS_TRIM_STATE_TRIMMED;
+ ctl->discardable_extents[BTRFS_STAT_CURR] -=
+ entry->bitmap_extents;
+ }
}
/*
@@ -3505,7 +3593,7 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
* if BTRFS_TRIM_STATE_TRIMMED is set on a bitmap.
*/
if (ret2 && !minlen)
- end_trimming_bitmap(entry);
+ end_trimming_bitmap(ctl, entry);
else
entry->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
spin_unlock(&ctl->tree_lock);
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index 55d6f6aaeb9b..822b19310872 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -28,6 +28,7 @@ struct btrfs_free_space {
unsigned long *bitmap;
struct list_head list;
enum btrfs_trim_state trim_state;
+ s32 bitmap_extents;
};
static inline bool btrfs_free_space_trimmed(struct btrfs_free_space *info)
@@ -50,6 +51,7 @@ struct btrfs_free_space_ctl {
int total_bitmaps;
int unit;
u64 start;
+ s32 discardable_extents[BTRFS_STAT_NR_ENTRIES];
const struct btrfs_free_space_op *op;
void *private;
struct mutex cache_writeout_mutex;
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index bc5415f67987..48bd254133be 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -12,6 +12,7 @@
#include <crypto/hash.h>
#include "ctree.h"
+#include "discard.h"
#include "disk-io.h"
#include "transaction.h"
#include "sysfs.h"
@@ -341,7 +342,21 @@ static const struct attribute_group btrfs_static_feature_attr_group = {
/*
* Discard statistics and tunables.
*/
+#define discard_to_fs_info(_kobj) to_fs_info((_kobj)->parent->parent)
+
+static ssize_t btrfs_discardable_extents_show(struct kobject *kobj,
+ struct kobj_attribute *a,
+ char *buf)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n",
+ atomic_read(&fs_info->discard_ctl.discardable_extents));
+}
+BTRFS_ATTR(discard, discardable_extents, btrfs_discardable_extents_show);
+
static const struct attribute *discard_debug_attrs[] = {
+ BTRFS_ATTR_PTR(discard, discardable_extents),
NULL,
};
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 12/22] btrfs: keep track of discardable_bytes
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (10 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 11/22] btrfs: track discardable extents for async discard Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 13/22] btrfs: calculate discard delay based on number of extents Dennis Zhou
` (11 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Keep track of this metric so that we can understand how ahead or behind
we are in discarding rate.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
---
fs/btrfs/ctree.h | 1 +
fs/btrfs/discard.c | 10 +++++++++
fs/btrfs/free-space-cache.c | 41 +++++++++++++++++++++++++++++--------
fs/btrfs/free-space-cache.h | 1 +
fs/btrfs/sysfs.c | 12 +++++++++++
5 files changed, 56 insertions(+), 9 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 968bc8f5d0f8..18df0e40a282 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -468,6 +468,7 @@ struct btrfs_discard_ctl {
struct btrfs_block_group *block_group;
struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
atomic_t discardable_extents;
+ atomic64_t discardable_bytes;
};
/* delayed seq elem */
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index 1f18bbfe8372..e8c2623617fd 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -362,6 +362,7 @@ void btrfs_discard_update_discardable(struct btrfs_block_group *block_group,
{
struct btrfs_discard_ctl *discard_ctl;
s32 extents_delta;
+ s64 bytes_delta;
if (!block_group ||
!btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
@@ -376,6 +377,14 @@ void btrfs_discard_update_discardable(struct btrfs_block_group *block_group,
ctl->discardable_extents[BTRFS_STAT_PREV] =
ctl->discardable_extents[BTRFS_STAT_CURR];
}
+
+ bytes_delta = (ctl->discardable_bytes[BTRFS_STAT_CURR] -
+ ctl->discardable_bytes[BTRFS_STAT_PREV]);
+ if (bytes_delta) {
+ atomic64_add(bytes_delta, &discard_ctl->discardable_bytes);
+ ctl->discardable_bytes[BTRFS_STAT_PREV] =
+ ctl->discardable_bytes[BTRFS_STAT_CURR];
+ }
}
/**
@@ -466,6 +475,7 @@ void btrfs_discard_init(struct btrfs_fs_info *fs_info)
INIT_LIST_HEAD(&discard_ctl->discard_list[i]);
atomic_set(&discard_ctl->discardable_extents, 0);
+ atomic64_set(&discard_ctl->discardable_bytes, 0);
}
void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 7a3b07ef4dd5..57df34480b93 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -820,9 +820,11 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
if (ret)
goto free_cache;
e->bitmap_extents = count_bitmap_extents(ctl, e);
- if (!btrfs_free_space_trimmed(e))
+ if (!btrfs_free_space_trimmed(e)) {
ctl->discardable_extents[BTRFS_STAT_CURR] +=
e->bitmap_extents;
+ ctl->discardable_bytes[BTRFS_STAT_CURR] += e->bytes;
+ }
}
io_ctl_drop_pages(&io_ctl);
@@ -1644,8 +1646,10 @@ __unlink_free_space(struct btrfs_free_space_ctl *ctl,
rb_erase(&info->offset_index, &ctl->free_space_offset);
ctl->free_extents--;
- if (!info->bitmap && !btrfs_free_space_trimmed(info))
+ if (!info->bitmap && !btrfs_free_space_trimmed(info)) {
ctl->discardable_extents[BTRFS_STAT_CURR]--;
+ ctl->discardable_bytes[BTRFS_STAT_CURR] -= info->bytes;
+ }
}
static void unlink_free_space(struct btrfs_free_space_ctl *ctl,
@@ -1666,8 +1670,10 @@ static int link_free_space(struct btrfs_free_space_ctl *ctl,
if (ret)
return ret;
- if (!info->bitmap && !btrfs_free_space_trimmed(info))
+ if (!info->bitmap && !btrfs_free_space_trimmed(info)) {
ctl->discardable_extents[BTRFS_STAT_CURR]++;
+ ctl->discardable_bytes[BTRFS_STAT_CURR] += info->bytes;
+ }
ctl->free_space += info->bytes;
ctl->free_extents++;
@@ -1746,8 +1752,10 @@ static inline void __bitmap_clear_bits(struct btrfs_free_space_ctl *ctl,
extent_delta++;
info->bitmap_extents += extent_delta;
- if (!btrfs_free_space_trimmed(info))
+ if (!btrfs_free_space_trimmed(info)) {
ctl->discardable_extents[BTRFS_STAT_CURR] += extent_delta;
+ ctl->discardable_bytes[BTRFS_STAT_CURR] -= bytes;
+ }
}
static void bitmap_clear_bits(struct btrfs_free_space_ctl *ctl,
@@ -1782,8 +1790,10 @@ static void bitmap_set_bits(struct btrfs_free_space_ctl *ctl,
extent_delta--;
info->bitmap_extents += extent_delta;
- if (!btrfs_free_space_trimmed(info))
+ if (!btrfs_free_space_trimmed(info)) {
ctl->discardable_extents[BTRFS_STAT_CURR] += extent_delta;
+ ctl->discardable_bytes[BTRFS_STAT_CURR] += bytes;
+ }
}
/*
@@ -2054,9 +2064,11 @@ static u64 add_bytes_to_bitmap(struct btrfs_free_space_ctl *ctl,
* whole bitmap untrimmed if at any point we add untrimmed regions.
*/
if (trim_state == BTRFS_TRIM_STATE_UNTRIMMED) {
- if (btrfs_free_space_trimmed(info))
+ if (btrfs_free_space_trimmed(info)) {
ctl->discardable_extents[BTRFS_STAT_CURR] +=
info->bitmap_extents;
+ ctl->discardable_bytes[BTRFS_STAT_CURR] += info->bytes;
+ }
info->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
}
@@ -2715,15 +2727,21 @@ __btrfs_return_cluster_to_free_space(
bitmap = (entry->bitmap != NULL);
if (!bitmap) {
/* merging treats extents as if they were new */
- if (!btrfs_free_space_trimmed(entry))
+ if (!btrfs_free_space_trimmed(entry)) {
ctl->discardable_extents[BTRFS_STAT_CURR]--;
+ ctl->discardable_bytes[BTRFS_STAT_CURR] -=
+ entry->bytes;
+ }
try_merge_free_space(ctl, entry, false);
steal_from_bitmap(ctl, entry, false);
/* as we insert directly, update these statistics */
- if (!btrfs_free_space_trimmed(entry))
+ if (!btrfs_free_space_trimmed(entry)) {
ctl->discardable_extents[BTRFS_STAT_CURR]++;
+ ctl->discardable_bytes[BTRFS_STAT_CURR] +=
+ entry->bytes;
+ }
}
tree_insert_offset(&ctl->free_space_offset,
entry->offset, &entry->offset_index, bitmap);
@@ -3014,6 +3032,8 @@ u64 btrfs_alloc_from_cluster(struct btrfs_block_group *block_group,
spin_lock(&ctl->tree_lock);
ctl->free_space -= bytes;
+ if (!entry->bitmap && !btrfs_free_space_trimmed(entry))
+ ctl->discardable_bytes[BTRFS_STAT_CURR] -= bytes;
if (entry->bytes == 0) {
ctl->free_extents--;
if (entry->bitmap) {
@@ -3519,9 +3539,11 @@ static void reset_trimming_bitmap(struct btrfs_free_space_ctl *ctl, u64 offset)
entry = tree_search_offset(ctl, offset, 1, 0);
if (entry) {
- if (btrfs_free_space_trimmed(entry))
+ if (btrfs_free_space_trimmed(entry)) {
ctl->discardable_extents[BTRFS_STAT_CURR] +=
entry->bitmap_extents;
+ ctl->discardable_bytes[BTRFS_STAT_CURR] += entry->bytes;
+ }
entry->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
}
@@ -3535,6 +3557,7 @@ static void end_trimming_bitmap(struct btrfs_free_space_ctl *ctl,
entry->trim_state = BTRFS_TRIM_STATE_TRIMMED;
ctl->discardable_extents[BTRFS_STAT_CURR] -=
entry->bitmap_extents;
+ ctl->discardable_bytes[BTRFS_STAT_CURR] -= entry->bytes;
}
}
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index 822b19310872..600633ed403b 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -52,6 +52,7 @@ struct btrfs_free_space_ctl {
int unit;
u64 start;
s32 discardable_extents[BTRFS_STAT_NR_ENTRIES];
+ s64 discardable_bytes[BTRFS_STAT_NR_ENTRIES];
const struct btrfs_free_space_op *op;
void *private;
struct mutex cache_writeout_mutex;
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 48bd254133be..e799d4488d72 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -355,8 +355,20 @@ static ssize_t btrfs_discardable_extents_show(struct kobject *kobj,
}
BTRFS_ATTR(discard, discardable_extents, btrfs_discardable_extents_show);
+static ssize_t btrfs_discardable_bytes_show(struct kobject *kobj,
+ struct kobj_attribute *a,
+ char *buf)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+
+ return snprintf(buf, PAGE_SIZE, "%lld\n",
+ atomic64_read(&fs_info->discard_ctl.discardable_bytes));
+}
+BTRFS_ATTR(discard, discardable_bytes, btrfs_discardable_bytes_show);
+
static const struct attribute *discard_debug_attrs[] = {
BTRFS_ATTR_PTR(discard, discardable_extents),
+ BTRFS_ATTR_PTR(discard, discardable_bytes),
NULL,
};
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 13/22] btrfs: calculate discard delay based on number of extents
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (11 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 12/22] btrfs: keep track of discardable_bytes Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-30 16:50 ` David Sterba
2019-12-14 0:22 ` [PATCH 14/22] btrfs: add bps discard rate limit Dennis Zhou
` (10 subsequent siblings)
23 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Use the number of discardable extents to help guide our discard delay
interval. This value is reevaluated every transaction commit.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/ctree.h | 2 ++
fs/btrfs/discard.c | 53 ++++++++++++++++++++++++++++++++++++++----
fs/btrfs/discard.h | 1 +
fs/btrfs/extent-tree.c | 4 +++-
fs/btrfs/sysfs.c | 31 ++++++++++++++++++++++++
5 files changed, 86 insertions(+), 5 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 18df0e40a282..98979566e281 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -469,6 +469,8 @@ struct btrfs_discard_ctl {
struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
atomic_t discardable_extents;
atomic64_t discardable_bytes;
+ u32 delay;
+ u32 iops_limit;
};
/* delayed seq elem */
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index e8c2623617fd..160713e04d0c 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -15,6 +15,11 @@
#define BTRFS_DISCARD_DELAY (120ULL * NSEC_PER_SEC)
#define BTRFS_DISCARD_UNUSED_DELAY (10ULL * NSEC_PER_SEC)
+/* Target completion latency of discarding all discardable extents. */
+#define BTRFS_DISCARD_TARGET_MSEC (6 * 60 * 60UL * MSEC_PER_SEC)
+#define BTRFS_DISCARD_MAX_DELAY (10000UL)
+#define BTRFS_DISCARD_MAX_IOPS (10UL)
+
static struct list_head *get_discard_list(struct btrfs_discard_ctl *discard_ctl,
struct btrfs_block_group *block_group)
{
@@ -239,11 +244,18 @@ void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
block_group = find_next_block_group(discard_ctl, now);
if (block_group) {
- u64 delay = 0;
+ u32 delay = discard_ctl->delay;
+
+ /*
+ * This timeout is to hopefully prevent immediate discarding
+ * in a recently allocated block group.
+ */
+ if (now < block_group->discard_eligible_time) {
+ u64 bg_timeout = (block_group->discard_eligible_time -
+ now);
- if (now < block_group->discard_eligible_time)
- delay = nsecs_to_jiffies(
- block_group->discard_eligible_time - now);
+ delay = max_t(u64, delay, nsecs_to_jiffies(bg_timeout));
+ }
mod_delayed_work(discard_ctl->discard_workers,
&discard_ctl->work,
@@ -348,6 +360,37 @@ bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl)
test_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags));
}
+/**
+ * btrfs_discard_calc_delay - recalculate the base delay
+ * @discard_ctl: discard control
+ *
+ * Recalculate the base delay which is based off the total number of
+ * discardable_extents. Clamp this with the iops_limit and
+ * BTRFS_DISCARD_MAX_DELAY.
+ */
+void btrfs_discard_calc_delay(struct btrfs_discard_ctl *discard_ctl)
+{
+ s32 discardable_extents =
+ atomic_read(&discard_ctl->discardable_extents);
+ s32 iops_limit;
+ unsigned long delay;
+
+ if (!discardable_extents)
+ return;
+
+ spin_lock(&discard_ctl->lock);
+
+ iops_limit = READ_ONCE(discard_ctl->iops_limit);
+ if (iops_limit)
+ iops_limit = MSEC_PER_SEC / iops_limit;
+
+ delay = BTRFS_DISCARD_TARGET_MSEC / discardable_extents;
+ delay = clamp_t(s32, delay, iops_limit, BTRFS_DISCARD_MAX_DELAY);
+ discard_ctl->delay = msecs_to_jiffies(delay);
+
+ spin_unlock(&discard_ctl->lock);
+}
+
/**
* btrfs_discard_update_discardable - propagate discard counters
* @block_group: block_group of interest
@@ -476,6 +519,8 @@ void btrfs_discard_init(struct btrfs_fs_info *fs_info)
atomic_set(&discard_ctl->discardable_extents, 0);
atomic64_set(&discard_ctl->discardable_bytes, 0);
+ discard_ctl->delay = BTRFS_DISCARD_MAX_DELAY;
+ discard_ctl->iops_limit = BTRFS_DISCARD_MAX_IOPS;
}
void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
index 7e3680dd82ce..3ed6855e24da 100644
--- a/fs/btrfs/discard.h
+++ b/fs/btrfs/discard.h
@@ -17,6 +17,7 @@ void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl);
/* Update operations. */
+void btrfs_discard_calc_delay(struct btrfs_discard_ctl *discard_ctl);
void btrfs_discard_update_discardable(struct btrfs_block_group *block_group,
struct btrfs_free_space_ctl *ctl);
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 2c12366cfde5..0163fdd59f8f 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -2935,8 +2935,10 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
cond_resched();
}
- if (btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ if (btrfs_test_opt(fs_info, DISCARD_ASYNC)) {
+ btrfs_discard_calc_delay(&fs_info->discard_ctl);
btrfs_discard_schedule_work(&fs_info->discard_ctl, true);
+ }
/*
* Transaction is finished. We don't need the lock anymore. We
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index e799d4488d72..79139b4b4f0a 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -344,6 +344,36 @@ static const struct attribute_group btrfs_static_feature_attr_group = {
*/
#define discard_to_fs_info(_kobj) to_fs_info((_kobj)->parent->parent)
+static ssize_t btrfs_discard_iops_limit_show(struct kobject *kobj,
+ struct kobj_attribute *a,
+ char *buf)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+
+ return snprintf(buf, PAGE_SIZE, "%u\n",
+ READ_ONCE(fs_info->discard_ctl.iops_limit));
+}
+
+static ssize_t btrfs_discard_iops_limit_store(struct kobject *kobj,
+ struct kobj_attribute *a,
+ const char *buf, size_t len)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+ struct btrfs_discard_ctl *discard_ctl = &fs_info->discard_ctl;
+ u32 iops_limit;
+ int ret;
+
+ ret = kstrtou32(buf, 10, &iops_limit);
+ if (ret)
+ return -EINVAL;
+
+ WRITE_ONCE(discard_ctl->iops_limit, iops_limit);
+
+ return len;
+}
+BTRFS_ATTR_RW(discard, iops_limit, btrfs_discard_iops_limit_show,
+ btrfs_discard_iops_limit_store);
+
static ssize_t btrfs_discardable_extents_show(struct kobject *kobj,
struct kobj_attribute *a,
char *buf)
@@ -367,6 +397,7 @@ static ssize_t btrfs_discardable_bytes_show(struct kobject *kobj,
BTRFS_ATTR(discard, discardable_bytes, btrfs_discardable_bytes_show);
static const struct attribute *discard_debug_attrs[] = {
+ BTRFS_ATTR_PTR(discard, iops_limit),
BTRFS_ATTR_PTR(discard, discardable_extents),
BTRFS_ATTR_PTR(discard, discardable_bytes),
NULL,
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 14/22] btrfs: add bps discard rate limit
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (12 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 13/22] btrfs: calculate discard delay based on number of extents Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-30 17:58 ` David Sterba
2019-12-14 0:22 ` [PATCH 15/22] btrfs: limit max discard size for async discard Dennis Zhou
` (9 subsequent siblings)
23 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Provide an ability to rate limit based on mbps in addition to the iops
delay calculated from number of discardable extents.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
---
fs/btrfs/ctree.h | 2 ++
fs/btrfs/discard.c | 18 ++++++++++++++++++
fs/btrfs/sysfs.c | 31 +++++++++++++++++++++++++++++++
3 files changed, 51 insertions(+)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 98979566e281..1b2dae5962de 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -467,10 +467,12 @@ struct btrfs_discard_ctl {
spinlock_t lock;
struct btrfs_block_group *block_group;
struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
+ u64 prev_discard;
atomic_t discardable_extents;
atomic64_t discardable_bytes;
u32 delay;
u32 iops_limit;
+ u64 bps_limit;
};
/* delayed seq elem */
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index 160713e04d0c..085f36808e7f 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -4,6 +4,7 @@
#include <linux/kernel.h>
#include <linux/ktime.h>
#include <linux/list.h>
+#include <linux/math64.h>
#include <linux/sizes.h>
#include <linux/workqueue.h>
#include "ctree.h"
@@ -245,6 +246,19 @@ void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
block_group = find_next_block_group(discard_ctl, now);
if (block_group) {
u32 delay = discard_ctl->delay;
+ u64 bps_limit = READ_ONCE(discard_ctl->bps_limit);
+
+ /*
+ * A single delayed workqueue item is responsible for
+ * discarding, so we can manage the bytes rate limit by keeping
+ * track of the previous discard.
+ */
+ if (bps_limit && discard_ctl->prev_discard) {
+ u64 bps_delay = div64_u64(discard_ctl->prev_discard *
+ MSEC_PER_SEC, bps_limit);
+
+ delay = max_t(u64, delay, msecs_to_jiffies(bps_delay));
+ }
/*
* This timeout is to hopefully prevent immediate discarding
@@ -322,6 +336,8 @@ static void btrfs_discard_workfn(struct work_struct *work)
btrfs_block_group_end(block_group),
0, true);
+ discard_ctl->prev_discard = trimmed;
+
/* Determine next steps for a block_group. */
if (block_group->discard_cursor >= btrfs_block_group_end(block_group)) {
if (discard_state == BTRFS_DISCARD_BITMAPS) {
@@ -517,10 +533,12 @@ void btrfs_discard_init(struct btrfs_fs_info *fs_info)
for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++)
INIT_LIST_HEAD(&discard_ctl->discard_list[i]);
+ discard_ctl->prev_discard = 0;
atomic_set(&discard_ctl->discardable_extents, 0);
atomic64_set(&discard_ctl->discardable_bytes, 0);
discard_ctl->delay = BTRFS_DISCARD_MAX_DELAY;
discard_ctl->iops_limit = BTRFS_DISCARD_MAX_IOPS;
+ discard_ctl->bps_limit = 0;
}
void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 79139b4b4f0a..12f7a906a36f 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -374,6 +374,36 @@ static ssize_t btrfs_discard_iops_limit_store(struct kobject *kobj,
BTRFS_ATTR_RW(discard, iops_limit, btrfs_discard_iops_limit_show,
btrfs_discard_iops_limit_store);
+static ssize_t btrfs_discard_bps_limit_show(struct kobject *kobj,
+ struct kobj_attribute *a,
+ char *buf)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+
+ return snprintf(buf, PAGE_SIZE, "%llu\n",
+ READ_ONCE(fs_info->discard_ctl.bps_limit));
+}
+
+static ssize_t btrfs_discard_bps_limit_store(struct kobject *kobj,
+ struct kobj_attribute *a,
+ const char *buf, size_t len)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+ struct btrfs_discard_ctl *discard_ctl = &fs_info->discard_ctl;
+ u64 bps_limit;
+ int ret;
+
+ ret = kstrtou64(buf, 10, &bps_limit);
+ if (ret)
+ return -EINVAL;
+
+ WRITE_ONCE(discard_ctl->bps_limit, bps_limit);
+
+ return len;
+}
+BTRFS_ATTR_RW(discard, bps_limit, btrfs_discard_bps_limit_show,
+ btrfs_discard_bps_limit_store);
+
static ssize_t btrfs_discardable_extents_show(struct kobject *kobj,
struct kobj_attribute *a,
char *buf)
@@ -398,6 +428,7 @@ BTRFS_ATTR(discard, discardable_bytes, btrfs_discardable_bytes_show);
static const struct attribute *discard_debug_attrs[] = {
BTRFS_ATTR_PTR(discard, iops_limit),
+ BTRFS_ATTR_PTR(discard, bps_limit),
BTRFS_ATTR_PTR(discard, discardable_extents),
BTRFS_ATTR_PTR(discard, discardable_bytes),
NULL,
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 15/22] btrfs: limit max discard size for async discard
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (13 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 14/22] btrfs: add bps discard rate limit Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-30 18:00 ` David Sterba
2019-12-30 18:08 ` David Sterba
2019-12-14 0:22 ` [PATCH 16/22] btrfs: make max async discard size tunable Dennis Zhou
` (8 subsequent siblings)
23 siblings, 2 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Throttle the maximum size of a discard so that we can provide an upper
bound for the rate of async discard. While the block layer is able to
split discards into the appropriate sized discards, we want to be able
to account more accurately the rate at which we are consuming ncq slots
as well as limit the upper bound of work for a discard.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/discard.h | 5 ++++
fs/btrfs/free-space-cache.c | 48 +++++++++++++++++++++++++++----------
2 files changed, 41 insertions(+), 12 deletions(-)
diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
index 3ed6855e24da..cb6ef0ab879d 100644
--- a/fs/btrfs/discard.h
+++ b/fs/btrfs/discard.h
@@ -3,10 +3,15 @@
#ifndef BTRFS_DISCARD_H
#define BTRFS_DISCARD_H
+#include <linux/sizes.h>
+
struct btrfs_fs_info;
struct btrfs_discard_ctl;
struct btrfs_block_group;
+/* Discard size limits. */
+#define BTRFS_ASYNC_DISCARD_MAX_SIZE (SZ_64M)
+
/* Work operations. */
void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
struct btrfs_block_group *block_group);
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 57df34480b93..0dbcea6c59f9 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -3466,19 +3466,40 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
if (entry->offset >= end)
goto out_unlock;
- extent_start = entry->offset;
- extent_bytes = entry->bytes;
- extent_trim_state = entry->trim_state;
- start = max(start, extent_start);
- bytes = min(extent_start + extent_bytes, end) - start;
- if (bytes < minlen) {
- spin_unlock(&ctl->tree_lock);
- mutex_unlock(&ctl->cache_writeout_mutex);
- goto next;
- }
+ if (async) {
+ start = extent_start = entry->offset;
+ bytes = extent_bytes = entry->bytes;
+ extent_trim_state = entry->trim_state;
+ if (bytes < minlen) {
+ spin_unlock(&ctl->tree_lock);
+ mutex_unlock(&ctl->cache_writeout_mutex);
+ goto next;
+ }
+ unlink_free_space(ctl, entry);
+ if (bytes > BTRFS_ASYNC_DISCARD_MAX_SIZE) {
+ bytes = extent_bytes =
+ BTRFS_ASYNC_DISCARD_MAX_SIZE;
+ entry->offset += BTRFS_ASYNC_DISCARD_MAX_SIZE;
+ entry->bytes -= BTRFS_ASYNC_DISCARD_MAX_SIZE;
+ link_free_space(ctl, entry);
+ } else {
+ kmem_cache_free(btrfs_free_space_cachep, entry);
+ }
+ } else {
+ extent_start = entry->offset;
+ extent_bytes = entry->bytes;
+ extent_trim_state = entry->trim_state;
+ start = max(start, extent_start);
+ bytes = min(extent_start + extent_bytes, end) - start;
+ if (bytes < minlen) {
+ spin_unlock(&ctl->tree_lock);
+ mutex_unlock(&ctl->cache_writeout_mutex);
+ goto next;
+ }
- unlink_free_space(ctl, entry);
- kmem_cache_free(btrfs_free_space_cachep, entry);
+ unlink_free_space(ctl, entry);
+ kmem_cache_free(btrfs_free_space_cachep, entry);
+ }
spin_unlock(&ctl->tree_lock);
trim_entry.start = extent_start;
@@ -3643,6 +3664,9 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
goto next;
}
+ if (async && bytes > BTRFS_ASYNC_DISCARD_MAX_SIZE)
+ bytes = BTRFS_ASYNC_DISCARD_MAX_SIZE;
+
bitmap_clear_bits(ctl, entry, start, bytes);
if (entry->bytes == 0)
free_bitmap(ctl, entry);
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 16/22] btrfs: make max async discard size tunable
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (14 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 15/22] btrfs: limit max discard size for async discard Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-30 18:05 ` David Sterba
2019-12-14 0:22 ` [PATCH 17/22] btrfs: have multiple discard lists Dennis Zhou
` (7 subsequent siblings)
23 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Expose max_discard_size as a tunable via sysfs.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
---
fs/btrfs/ctree.h | 1 +
fs/btrfs/discard.c | 1 +
fs/btrfs/free-space-cache.c | 19 ++++++++++++-------
fs/btrfs/sysfs.c | 31 +++++++++++++++++++++++++++++++
4 files changed, 45 insertions(+), 7 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 1b2dae5962de..bf93ddbc773f 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -470,6 +470,7 @@ struct btrfs_discard_ctl {
u64 prev_discard;
atomic_t discardable_extents;
atomic64_t discardable_bytes;
+ u64 max_discard_size;
u32 delay;
u32 iops_limit;
u64 bps_limit;
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index 085f36808e7f..dd5143f0283f 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -536,6 +536,7 @@ void btrfs_discard_init(struct btrfs_fs_info *fs_info)
discard_ctl->prev_discard = 0;
atomic_set(&discard_ctl->discardable_extents, 0);
atomic64_set(&discard_ctl->discardable_bytes, 0);
+ discard_ctl->max_discard_size = BTRFS_ASYNC_DISCARD_MAX_SIZE;
discard_ctl->delay = BTRFS_DISCARD_MAX_DELAY;
discard_ctl->iops_limit = BTRFS_DISCARD_MAX_IOPS;
discard_ctl->bps_limit = 0;
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 0dbcea6c59f9..e5bb37627807 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -3431,6 +3431,8 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
u64 *total_trimmed, u64 start, u64 end, u64 minlen,
bool async)
{
+ struct btrfs_discard_ctl *discard_ctl =
+ &block_group->fs_info->discard_ctl;
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
struct btrfs_free_space *entry;
struct rb_node *node;
@@ -3439,6 +3441,7 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
u64 extent_bytes;
enum btrfs_trim_state extent_trim_state;
u64 bytes;
+ u64 max_discard_size = READ_ONCE(discard_ctl->max_discard_size);
while (start < end) {
struct btrfs_trim_range trim_entry;
@@ -3476,11 +3479,10 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
goto next;
}
unlink_free_space(ctl, entry);
- if (bytes > BTRFS_ASYNC_DISCARD_MAX_SIZE) {
- bytes = extent_bytes =
- BTRFS_ASYNC_DISCARD_MAX_SIZE;
- entry->offset += BTRFS_ASYNC_DISCARD_MAX_SIZE;
- entry->bytes -= BTRFS_ASYNC_DISCARD_MAX_SIZE;
+ if (max_discard_size && bytes > max_discard_size) {
+ bytes = extent_bytes = max_discard_size;
+ entry->offset += max_discard_size;
+ entry->bytes -= max_discard_size;
link_free_space(ctl, entry);
} else {
kmem_cache_free(btrfs_free_space_cachep, entry);
@@ -3589,12 +3591,15 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
u64 *total_trimmed, u64 start, u64 end, u64 minlen,
bool async)
{
+ struct btrfs_discard_ctl *discard_ctl =
+ &block_group->fs_info->discard_ctl;
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
struct btrfs_free_space *entry;
int ret = 0;
int ret2;
u64 bytes;
u64 offset = offset_to_bitmap(ctl, start);
+ u64 max_discard_size = READ_ONCE(discard_ctl->max_discard_size);
while (offset < end) {
bool next_bitmap = false;
@@ -3664,8 +3669,8 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
goto next;
}
- if (async && bytes > BTRFS_ASYNC_DISCARD_MAX_SIZE)
- bytes = BTRFS_ASYNC_DISCARD_MAX_SIZE;
+ if (async && max_discard_size && bytes > max_discard_size)
+ bytes = max_discard_size;
bitmap_clear_bits(ctl, entry, start, bytes);
if (entry->bytes == 0)
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index 12f7a906a36f..e71e8bbc57d0 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -404,6 +404,36 @@ static ssize_t btrfs_discard_bps_limit_store(struct kobject *kobj,
BTRFS_ATTR_RW(discard, bps_limit, btrfs_discard_bps_limit_show,
btrfs_discard_bps_limit_store);
+static ssize_t btrfs_discard_max_discard_size_show(struct kobject *kobj,
+ struct kobj_attribute *a,
+ char *buf)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+
+ return snprintf(buf, PAGE_SIZE, "%llu\n",
+ READ_ONCE(fs_info->discard_ctl.max_discard_size));
+}
+
+static ssize_t btrfs_discard_max_discard_size_store(struct kobject *kobj,
+ struct kobj_attribute *a,
+ const char *buf, size_t len)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+ struct btrfs_discard_ctl *discard_ctl = &fs_info->discard_ctl;
+ u64 max_discard_size;
+ int ret;
+
+ ret = kstrtou64(buf, 10, &max_discard_size);
+ if (ret)
+ return -EINVAL;
+
+ WRITE_ONCE(discard_ctl->max_discard_size, max_discard_size);
+
+ return len;
+}
+BTRFS_ATTR_RW(discard, max_discard_size, btrfs_discard_max_discard_size_show,
+ btrfs_discard_max_discard_size_store);
+
static ssize_t btrfs_discardable_extents_show(struct kobject *kobj,
struct kobj_attribute *a,
char *buf)
@@ -429,6 +459,7 @@ BTRFS_ATTR(discard, discardable_bytes, btrfs_discardable_bytes_show);
static const struct attribute *discard_debug_attrs[] = {
BTRFS_ATTR_PTR(discard, iops_limit),
BTRFS_ATTR_PTR(discard, bps_limit),
+ BTRFS_ATTR_PTR(discard, max_discard_size),
BTRFS_ATTR_PTR(discard, discardable_extents),
BTRFS_ATTR_PTR(discard, discardable_bytes),
NULL,
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 17/22] btrfs: have multiple discard lists
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (15 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 16/22] btrfs: make max async discard size tunable Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 18/22] btrfs: only keep track of data extents for async discard Dennis Zhou
` (6 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Non-block group destruction discarding currently only had a single list
with no minimum discard length. This can lead to caravaning more
meaningful discards behind a heavily fragmented block group.
This adds support for multiple lists with minimum discard lengths to
prevent the caravan effect. We promote block groups back up when we
exceed the BTRFS_ASYNC_DISCARD_MAX_FILTER size, currently we support
only 2 lists with filters of 1MB and 32KB respectively.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/ctree.h | 2 +-
fs/btrfs/discard.c | 100 +++++++++++++++++++++++++++++++++---
fs/btrfs/discard.h | 6 +++
fs/btrfs/free-space-cache.c | 53 ++++++++++++++-----
fs/btrfs/free-space-cache.h | 2 +-
5 files changed, 140 insertions(+), 23 deletions(-)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index bf93ddbc773f..dddf51e27bed 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -457,7 +457,7 @@ struct btrfs_full_stripe_locks_tree {
* afterwards represent monotonically decreasing discard filter sizes to
* prioritize what should be discarded next.
*/
-#define BTRFS_NR_DISCARD_LISTS 2
+#define BTRFS_NR_DISCARD_LISTS 3
#define BTRFS_DISCARD_INDEX_UNUSED 0
#define BTRFS_DISCARD_INDEX_START 1
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index dd5143f0283f..884dffd28596 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -21,6 +21,10 @@
#define BTRFS_DISCARD_MAX_DELAY (10000UL)
#define BTRFS_DISCARD_MAX_IOPS (10UL)
+/* Montonically decreasing minimum length filters after index 0. */
+static int discard_minlen[BTRFS_NR_DISCARD_LISTS] = {0,
+ BTRFS_ASYNC_DISCARD_MAX_FILTER, BTRFS_ASYNC_DISCARD_MIN_FILTER};
+
static struct list_head *get_discard_list(struct btrfs_discard_ctl *discard_ctl,
struct btrfs_block_group *block_group)
{
@@ -140,16 +144,18 @@ static struct btrfs_block_group *find_next_block_group(
* peek_discard_list - wrap find_next_block_group()
* @discard_ctl: discard control
* @discard_state: the discard_state of the block_group after state management
+ * @discard_index: the discard_index of the block_group after state management
*
* This wraps find_next_block_group() and sets the block_group to be in use.
* discard_state's control flow is managed here. Variables related to
* discard_state are reset here as needed (eg discard_cursor). @discard_state
- * is remembered as it may change while we're discarding, but we want the
- * discard to execute in the context determined here.
+ * and @discard_index are remembered as it may change while we're discarding,
+ * but we want the discard to execute in the context determined here.
*/
static struct btrfs_block_group *peek_discard_list(
struct btrfs_discard_ctl *discard_ctl,
- enum btrfs_discard_state *discard_state)
+ enum btrfs_discard_state *discard_state,
+ int *discard_index)
{
struct btrfs_block_group *block_group;
u64 now = ktime_get_ns();
@@ -171,6 +177,7 @@ static struct btrfs_block_group *peek_discard_list(
}
discard_ctl->block_group = block_group;
*discard_state = block_group->discard_state;
+ *discard_index = block_group->discard_index;
} else {
block_group = NULL;
}
@@ -180,6 +187,64 @@ static struct btrfs_block_group *peek_discard_list(
return block_group;
}
+/**
+ * btrfs_discard_check_filter - updates a block groups filters
+ * @block_group: block group of interest
+ * @bytes: recently freed region size after coalescing
+ *
+ * Async discard maintains multiple lists with progressively smaller filters
+ * to prioritize discarding based on size. Should a free space that matches
+ * a larger filter be returned to the free_space_cache, prioritize that discard
+ * by moving @block_group to the proper filter.
+ */
+void btrfs_discard_check_filter(struct btrfs_block_group *block_group,
+ u64 bytes)
+{
+ struct btrfs_discard_ctl *discard_ctl;
+
+ if (!block_group ||
+ !btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
+ return;
+
+ discard_ctl = &block_group->fs_info->discard_ctl;
+
+ if (block_group->discard_index > BTRFS_DISCARD_INDEX_START &&
+ bytes >= discard_minlen[block_group->discard_index - 1]) {
+ int i;
+
+ remove_from_discard_list(discard_ctl, block_group);
+
+ for (i = BTRFS_DISCARD_INDEX_START; i < BTRFS_NR_DISCARD_LISTS;
+ i++) {
+ if (bytes >= discard_minlen[i]) {
+ block_group->discard_index = i;
+ add_to_discard_list(discard_ctl, block_group);
+ break;
+ }
+ }
+ }
+}
+
+/**
+ * btrfs_update_discard_index - moves a block_group along the discard lists
+ * @discard_ctl: discard control
+ * @block_group: block_group of interest
+ *
+ * Increment @block_group's discard_index. If it falls of the list, let it be.
+ * Otherwise add it back to the appropriate list.
+ */
+static void btrfs_update_discard_index(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ block_group->discard_index++;
+ if (block_group->discard_index == BTRFS_NR_DISCARD_LISTS) {
+ block_group->discard_index = 1;
+ return;
+ }
+
+ add_to_discard_list(discard_ctl, block_group);
+}
+
/**
* btrfs_discard_cancel_work - remove a block_group from the discard lists
* @discard_ctl: discard control
@@ -300,6 +365,8 @@ static void btrfs_finish_discard_pass(struct btrfs_discard_ctl *discard_ctl,
btrfs_mark_bg_unused(block_group);
else
add_to_discard_unused_list(discard_ctl, block_group);
+ } else {
+ btrfs_update_discard_index(discard_ctl, block_group);
}
}
@@ -316,25 +383,42 @@ static void btrfs_discard_workfn(struct work_struct *work)
struct btrfs_discard_ctl *discard_ctl;
struct btrfs_block_group *block_group;
enum btrfs_discard_state discard_state;
+ int discard_index = 0;
u64 trimmed = 0;
+ u64 minlen = 0;
discard_ctl = container_of(work, struct btrfs_discard_ctl, work.work);
- block_group = peek_discard_list(discard_ctl, &discard_state);
+ block_group = peek_discard_list(discard_ctl, &discard_state,
+ &discard_index);
if (!block_group || !btrfs_run_discard_work(discard_ctl))
return;
/* Perform discarding. */
- if (discard_state == BTRFS_DISCARD_BITMAPS)
+ minlen = discard_minlen[discard_index];
+
+ if (discard_state == BTRFS_DISCARD_BITMAPS) {
+ u64 maxlen = 0;
+
+ /*
+ * Use the previous levels minimum discard length as the max
+ * length filter. In the case something is added to make a
+ * region go beyond the max filter, the entire bitmap is set
+ * back to BTRFS_TRIM_STATE_UNTRIMMED.
+ */
+ if (discard_index != BTRFS_DISCARD_INDEX_UNUSED)
+ maxlen = discard_minlen[discard_index - 1];
+
btrfs_trim_block_group_bitmaps(block_group, &trimmed,
block_group->discard_cursor,
btrfs_block_group_end(block_group),
- 0, true);
- else
+ minlen, maxlen, true);
+ } else {
btrfs_trim_block_group_extents(block_group, &trimmed,
block_group->discard_cursor,
btrfs_block_group_end(block_group),
- 0, true);
+ minlen, true);
+ }
discard_ctl->prev_discard = trimmed;
diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
index cb6ef0ab879d..91d5eb8e8bd9 100644
--- a/fs/btrfs/discard.h
+++ b/fs/btrfs/discard.h
@@ -11,6 +11,12 @@ struct btrfs_block_group;
/* Discard size limits. */
#define BTRFS_ASYNC_DISCARD_MAX_SIZE (SZ_64M)
+#define BTRFS_ASYNC_DISCARD_MAX_FILTER (SZ_1M)
+#define BTRFS_ASYNC_DISCARD_MIN_FILTER (SZ_32K)
+
+/* List operations. */
+void btrfs_discard_check_filter(struct btrfs_block_group *block_group,
+ u64 bytes);
/* Work operations. */
void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index e5bb37627807..c9bf0207960d 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -2466,6 +2466,7 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
struct btrfs_block_group *block_group = ctl->private;
struct btrfs_free_space *info;
int ret = 0;
+ u64 filter_bytes = bytes;
info = kmem_cache_zalloc(btrfs_free_space_cachep, GFP_NOFS);
if (!info)
@@ -2502,6 +2503,8 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
*/
steal_from_bitmap(ctl, info, true);
+ filter_bytes = max(filter_bytes, info->bytes);
+
ret = link_free_space(ctl, info);
if (ret)
kmem_cache_free(btrfs_free_space_cachep, info);
@@ -2514,8 +2517,10 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
ASSERT(ret != -EEXIST);
}
- if (trim_state != BTRFS_TRIM_STATE_TRIMMED)
+ if (trim_state != BTRFS_TRIM_STATE_TRIMMED) {
+ btrfs_discard_check_filter(block_group, filter_bytes);
btrfs_discard_queue_work(&fs_info->discard_ctl, block_group);
+ }
return ret;
}
@@ -3479,7 +3484,14 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
goto next;
}
unlink_free_space(ctl, entry);
- if (max_discard_size && bytes > max_discard_size) {
+ /*
+ * Let bytes = BTRFS_MAX_DISCARD_SIZE + X.
+ * If X < BTRFS_ASYNC_DISCARD_MIN_FILTER, we won't trim
+ * X when we come back around. So trim it now.
+ */
+ if (max_discard_size &&
+ bytes >= (max_discard_size +
+ BTRFS_ASYNC_DISCARD_MIN_FILTER)) {
bytes = extent_bytes = max_discard_size;
entry->offset += max_discard_size;
entry->bytes -= max_discard_size;
@@ -3589,7 +3601,7 @@ static void end_trimming_bitmap(struct btrfs_free_space_ctl *ctl,
*/
static int trim_bitmaps(struct btrfs_block_group *block_group,
u64 *total_trimmed, u64 start, u64 end, u64 minlen,
- bool async)
+ u64 maxlen, bool async)
{
struct btrfs_discard_ctl *discard_ctl =
&block_group->fs_info->discard_ctl;
@@ -3617,7 +3629,15 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
}
entry = tree_search_offset(ctl, offset, 1, 0);
- if (!entry || (async && start == offset &&
+ /*
+ * Bitmaps are marked trimmed lossily now to prevent constant
+ * discarding of the same bitmap (the reason why we are bound
+ * by the filters). So, retrim the block group bitmaps when we
+ * are preparing to punt to the unused_bgs list. This uses
+ * @minlen to determine if we are in BTRFS_DISCARD_INDEX_UNUSED
+ * which is the only discard index which sets minlen to 0.
+ */
+ if (!entry || (async && minlen && start == offset &&
btrfs_free_space_trimmed(entry))) {
spin_unlock(&ctl->tree_lock);
mutex_unlock(&ctl->cache_writeout_mutex);
@@ -3638,10 +3658,10 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
ret2 = search_bitmap(ctl, entry, &start, &bytes, false);
if (ret2 || start >= end) {
/*
- * This keeps the invariant that all bytes are trimmed
- * if BTRFS_TRIM_STATE_TRIMMED is set on a bitmap.
+ * We lossily consider a bitmap trimmed if we only skip
+ * over regions <= BTRFS_ASYNC_DISCARD_MIN_FILTER.
*/
- if (ret2 && !minlen)
+ if (ret2 && minlen <= BTRFS_ASYNC_DISCARD_MIN_FILTER)
end_trimming_bitmap(ctl, entry);
else
entry->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
@@ -3662,14 +3682,20 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
}
bytes = min(bytes, end - start);
- if (bytes < minlen) {
- entry->trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
+ if (bytes < minlen || (async && maxlen && bytes > maxlen)) {
spin_unlock(&ctl->tree_lock);
mutex_unlock(&ctl->cache_writeout_mutex);
goto next;
}
- if (async && max_discard_size && bytes > max_discard_size)
+ /*
+ * Let bytes = BTRFS_MAX_DISCARD_SIZE + X.
+ * If X < @minlen, we won't trim X when we come back around.
+ * So trim it now. We differ here from trimming extents as we
+ * don't keep individual state per bit.
+ */
+ if (async && max_discard_size &&
+ bytes > (max_discard_size + minlen))
bytes = max_discard_size;
bitmap_clear_bits(ctl, entry, start, bytes);
@@ -3776,7 +3802,7 @@ int btrfs_trim_block_group(struct btrfs_block_group *block_group,
if (ret)
goto out;
- ret = trim_bitmaps(block_group, trimmed, start, end, minlen, false);
+ ret = trim_bitmaps(block_group, trimmed, start, end, minlen, 0, false);
div64_u64_rem(end, BITS_PER_BITMAP * ctl->unit, &rem);
/* If we ended in the middle of a bitmap, reset the trimming flag. */
if (rem)
@@ -3810,7 +3836,7 @@ int btrfs_trim_block_group_extents(struct btrfs_block_group *block_group,
int btrfs_trim_block_group_bitmaps(struct btrfs_block_group *block_group,
u64 *trimmed, u64 start, u64 end, u64 minlen,
- bool async)
+ u64 maxlen, bool async)
{
int ret;
@@ -3824,7 +3850,8 @@ int btrfs_trim_block_group_bitmaps(struct btrfs_block_group *block_group,
btrfs_get_block_group_trimming(block_group);
spin_unlock(&block_group->lock);
- ret = trim_bitmaps(block_group, trimmed, start, end, minlen, async);
+ ret = trim_bitmaps(block_group, trimmed, start, end, minlen, maxlen,
+ async);
btrfs_put_block_group_trimming(block_group);
return ret;
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index 600633ed403b..e305fa0117d1 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -146,7 +146,7 @@ int btrfs_trim_block_group_extents(struct btrfs_block_group *block_group,
bool async);
int btrfs_trim_block_group_bitmaps(struct btrfs_block_group *block_group,
u64 *trimmed, u64 start, u64 end, u64 minlen,
- bool async);
+ u64 maxlen, bool async);
/* Support functions for running our sanity tests */
#ifdef CONFIG_BTRFS_FS_RUN_SANITY_TESTS
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 18/22] btrfs: only keep track of data extents for async discard
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (16 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 17/22] btrfs: have multiple discard lists Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-30 17:39 ` David Sterba
2019-12-14 0:22 ` [PATCH 19/22] btrfs: keep track of discard reuse stats Dennis Zhou
` (5 subsequent siblings)
23 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
As mentioned earlier, discarding data can be done either by issuing an
explicit discard or implicitly by reusing the LBA. Metadata chunks see
much more frequent reuse due to well it being metadata. So instead of
explicitly discarding metadata blocks, just leave them be and let the
latter implicit discarding be done for them.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/block-group.h | 6 ++++++
fs/btrfs/discard.c | 11 +++++++++--
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h
index 601e1d217e22..ee8441439a56 100644
--- a/fs/btrfs/block-group.h
+++ b/fs/btrfs/block-group.h
@@ -182,6 +182,12 @@ static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
return (block_group->start + block_group->length);
}
+static inline bool btrfs_is_block_group_data(
+ struct btrfs_block_group *block_group)
+{
+ return (block_group->flags & BTRFS_BLOCK_GROUP_DATA);
+}
+
#ifdef CONFIG_BTRFS_DEBUG
static inline int btrfs_should_fragment_free_space(
struct btrfs_block_group *block_group)
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index 884dffd28596..55ad357e65f3 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -53,6 +53,9 @@ static void __add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
static void add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
struct btrfs_block_group *block_group)
{
+ if (!btrfs_is_block_group_data(block_group))
+ return;
+
spin_lock(&discard_ctl->lock);
__add_to_discard_list(discard_ctl, block_group);
@@ -168,7 +171,10 @@ static struct btrfs_block_group *peek_discard_list(
if (block_group && now > block_group->discard_eligible_time) {
if (block_group->discard_index == BTRFS_DISCARD_INDEX_UNUSED &&
block_group->used != 0) {
- __add_to_discard_list(discard_ctl, block_group);
+ if (btrfs_is_block_group_data(block_group))
+ __add_to_discard_list(discard_ctl, block_group);
+ else
+ list_del_init(&block_group->discard_list);
goto again;
}
if (block_group->discard_state == BTRFS_DISCARD_RESET_CURSOR) {
@@ -508,7 +514,8 @@ void btrfs_discard_update_discardable(struct btrfs_block_group *block_group,
s64 bytes_delta;
if (!block_group ||
- !btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
+ !btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC) ||
+ !btrfs_is_block_group_data(block_group))
return;
discard_ctl = &block_group->fs_info->discard_ctl;
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 19/22] btrfs: keep track of discard reuse stats
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (17 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 18/22] btrfs: only keep track of data extents for async discard Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-30 17:33 ` David Sterba
2019-12-14 0:22 ` [PATCH 20/22] btrfs: add async discard header Dennis Zhou
` (4 subsequent siblings)
23 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Keep track of how much we are discarding and how often we are reusing
with async discard.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/ctree.h | 3 +++
fs/btrfs/discard.c | 7 +++++++
fs/btrfs/free-space-cache.c | 14 ++++++++++++++
fs/btrfs/sysfs.c | 36 ++++++++++++++++++++++++++++++++++++
4 files changed, 60 insertions(+)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index dddf51e27bed..edfbe6060e8d 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -474,6 +474,9 @@ struct btrfs_discard_ctl {
u32 delay;
u32 iops_limit;
u64 bps_limit;
+ u64 discard_extent_bytes;
+ u64 discard_bitmap_bytes;
+ atomic64_t discard_bytes_saved;
};
/* delayed seq elem */
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index 55ad357e65f3..fe73814526ef 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -419,11 +419,15 @@ static void btrfs_discard_workfn(struct work_struct *work)
block_group->discard_cursor,
btrfs_block_group_end(block_group),
minlen, maxlen, true);
+ WRITE_ONCE(discard_ctl->discard_bitmap_bytes,
+ discard_ctl->discard_bitmap_bytes + trimmed);
} else {
btrfs_trim_block_group_extents(block_group, &trimmed,
block_group->discard_cursor,
btrfs_block_group_end(block_group),
minlen, true);
+ WRITE_ONCE(discard_ctl->discard_extent_bytes,
+ discard_ctl->discard_extent_bytes + trimmed);
}
discard_ctl->prev_discard = trimmed;
@@ -631,6 +635,9 @@ void btrfs_discard_init(struct btrfs_fs_info *fs_info)
discard_ctl->delay = BTRFS_DISCARD_MAX_DELAY;
discard_ctl->iops_limit = BTRFS_DISCARD_MAX_IOPS;
discard_ctl->bps_limit = 0;
+ discard_ctl->discard_extent_bytes = 0;
+ discard_ctl->discard_bitmap_bytes = 0;
+ atomic64_set(&discard_ctl->discard_bytes_saved, 0);
}
void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index c9bf0207960d..612726a91e98 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -2844,6 +2844,8 @@ u64 btrfs_find_space_for_alloc(struct btrfs_block_group *block_group,
u64 *max_extent_size)
{
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
+ struct btrfs_discard_ctl *discard_ctl =
+ &block_group->fs_info->discard_ctl;
struct btrfs_free_space *entry = NULL;
u64 bytes_search = bytes + empty_size;
u64 ret = 0;
@@ -2860,6 +2862,10 @@ u64 btrfs_find_space_for_alloc(struct btrfs_block_group *block_group,
ret = offset;
if (entry->bitmap) {
bitmap_clear_bits(ctl, entry, offset, bytes);
+
+ if (!btrfs_free_space_trimmed(entry))
+ atomic64_add(bytes, &discard_ctl->discard_bytes_saved);
+
if (!entry->bytes)
free_bitmap(ctl, entry);
} else {
@@ -2868,6 +2874,9 @@ u64 btrfs_find_space_for_alloc(struct btrfs_block_group *block_group,
align_gap = entry->offset;
align_gap_trim_state = entry->trim_state;
+ if (!btrfs_free_space_trimmed(entry))
+ atomic64_add(bytes, &discard_ctl->discard_bytes_saved);
+
entry->offset = offset + bytes;
WARN_ON(entry->bytes < bytes + align_gap_len);
@@ -2972,6 +2981,8 @@ u64 btrfs_alloc_from_cluster(struct btrfs_block_group *block_group,
u64 min_start, u64 *max_extent_size)
{
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
+ struct btrfs_discard_ctl *discard_ctl =
+ &block_group->fs_info->discard_ctl;
struct btrfs_free_space *entry = NULL;
struct rb_node *node;
u64 ret = 0;
@@ -3036,6 +3047,9 @@ u64 btrfs_alloc_from_cluster(struct btrfs_block_group *block_group,
spin_lock(&ctl->tree_lock);
+ if (!btrfs_free_space_trimmed(entry))
+ atomic64_add(bytes, &discard_ctl->discard_bytes_saved);
+
ctl->free_space -= bytes;
if (!entry->bitmap && !btrfs_free_space_trimmed(entry))
ctl->discardable_bytes[BTRFS_STAT_CURR] -= bytes;
diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
index e71e8bbc57d0..613606389072 100644
--- a/fs/btrfs/sysfs.c
+++ b/fs/btrfs/sysfs.c
@@ -456,12 +456,48 @@ static ssize_t btrfs_discardable_bytes_show(struct kobject *kobj,
}
BTRFS_ATTR(discard, discardable_bytes, btrfs_discardable_bytes_show);
+static ssize_t btrfs_discard_extent_bytes_show(struct kobject *kobj,
+ struct kobj_attribute *a,
+ char *buf)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+
+ return snprintf(buf, PAGE_SIZE, "%lld\n",
+ READ_ONCE(fs_info->discard_ctl.discard_extent_bytes));
+}
+BTRFS_ATTR(discard, discard_extent_bytes, btrfs_discard_extent_bytes_show);
+
+static ssize_t btrfs_discard_bitmap_bytes_show(struct kobject *kobj,
+ struct kobj_attribute *a,
+ char *buf)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+
+ return snprintf(buf, PAGE_SIZE, "%lld\n",
+ READ_ONCE(fs_info->discard_ctl.discard_bitmap_bytes));
+}
+BTRFS_ATTR(discard, discard_bitmap_bytes, btrfs_discard_bitmap_bytes_show);
+
+static ssize_t btrfs_discard_bytes_saved_show(struct kobject *kobj,
+ struct kobj_attribute *a,
+ char *buf)
+{
+ struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
+
+ return snprintf(buf, PAGE_SIZE, "%lld\n",
+ atomic64_read(&fs_info->discard_ctl.discard_bytes_saved));
+}
+BTRFS_ATTR(discard, discard_bytes_saved, btrfs_discard_bytes_saved_show);
+
static const struct attribute *discard_debug_attrs[] = {
BTRFS_ATTR_PTR(discard, iops_limit),
BTRFS_ATTR_PTR(discard, bps_limit),
BTRFS_ATTR_PTR(discard, max_discard_size),
BTRFS_ATTR_PTR(discard, discardable_extents),
BTRFS_ATTR_PTR(discard, discardable_bytes),
+ BTRFS_ATTR_PTR(discard, discard_extent_bytes),
+ BTRFS_ATTR_PTR(discard, discard_bitmap_bytes),
+ BTRFS_ATTR_PTR(discard, discard_bytes_saved),
NULL,
};
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 20/22] btrfs: add async discard header
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (18 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 19/22] btrfs: keep track of discard reuse stats Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 21/22] btrfs: increase the metadata allowance for the free_space_cache Dennis Zhou
` (3 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Give a brief overview for how async discard is implemented.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/discard.c | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
index fe73814526ef..fe5719797eee 100644
--- a/fs/btrfs/discard.c
+++ b/fs/btrfs/discard.c
@@ -1,4 +1,39 @@
// SPDX-License-Identifier: GPL-2.0
+/*
+ * This contains the logic to handle async discard.
+ *
+ * Async discard manages trimming of free space outside of transaction commit.
+ * Discarding is done by managing the block_groups on a LRU list based on free
+ * space recency. Two passes are used to first prioritize discarding extents
+ * and then allow for trimming in the bitmap the best opportunity to coalesce.
+ * The block_groups are maintained on multiple lists to allow for multiple
+ * passes with different discard filter requirements. A delayed work item is
+ * used to manage discarding with timeout determined by a max of the delay
+ * incurred by the iops rate limit, byte rate limit, and the timeout of max
+ * delay of BTRFS_DISCARD_MAX_DELAY.
+ *
+ * The first list is special to manage discarding of fully free block groups.
+ * This is necessary because we issue a final trim for a full free block group
+ * after forgetting it. When a block group becomes unused, instead of directly
+ * being added to the unused_bgs list, we add it to this first list. Then
+ * from there, if it becomes fully discarded, we place it onto the unused_bgs
+ * list.
+ *
+ * The in-memory free space cache serves as the backing state for discard.
+ * Consequently this means there is no persistence. We opt to load all the
+ * block groups in as not discarded, so the mount case degenerates to the
+ * crashing case.
+ *
+ * As the free space cache uses bitmaps, there exists a tradeoff between
+ * ease/efficiency for find_free_extent() and the accuracy of discard state.
+ * Here we opt to let untrimmed regions merge with everything while only letting
+ * trimmed regions merge with other trimmed regions. This can cause
+ * overtrimming, but the coalescing benefit seems to be worth it. Additionally,
+ * bitmap state is tracked as a whole. If we're able to fully trim a bitmap,
+ * the trimmed flag is set on the bitmap. Otherwise, if an allocation comes in,
+ * this resets the state and we will retry trimming the whole bitmap. This is a
+ * tradeoff between discard state accuracy and the cost of accounting.
+ */
#include <linux/jiffies.h>
#include <linux/kernel.h>
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 21/22] btrfs: increase the metadata allowance for the free_space_cache
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (19 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 20/22] btrfs: add async discard header Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 22/22] btrfs: make smaller extents more likely to go into bitmaps Dennis Zhou
` (2 subsequent siblings)
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
Currently, there is no way for the free space cache to recover from
being serviced by purely bitmaps because the extent threshold is set to
0 in recalculate_thresholds() when we surpass the metadata allowance.
This adds a recovery mechanism by keeping large extents out of the
bitmaps and increases the metadata upper bound to 64KB. The recovery
mechanism bypasses this upper bound, thus making it a soft upper bound.
But, with the bypass being 1MB or greater, it shouldn't add unbounded
overhead.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/free-space-cache.c | 26 +++++++++++---------------
1 file changed, 11 insertions(+), 15 deletions(-)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 612726a91e98..6a313946a8da 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -24,7 +24,8 @@
#include "discard.h"
#define BITS_PER_BITMAP (PAGE_SIZE * 8UL)
-#define MAX_CACHE_BYTES_PER_GIG SZ_32K
+#define MAX_CACHE_BYTES_PER_GIG SZ_64K
+#define FORCE_EXTENT_THRESHOLD SZ_1M
struct btrfs_trim_range {
u64 start;
@@ -1695,26 +1696,17 @@ static void recalculate_thresholds(struct btrfs_free_space_ctl *ctl)
ASSERT(ctl->total_bitmaps <= max_bitmaps);
/*
- * The goal is to keep the total amount of memory used per 1gb of space
- * at or below 32k, so we need to adjust how much memory we allow to be
- * used by extent based free space tracking
+ * We are trying to keep the total amount of memory used per 1gb of
+ * space to be MAX_CACHE_BYTES_PER_GIG. However, with a reclamation
+ * mechanism of pulling extents >= FORCE_EXTENT_THRESHOLD out of
+ * bitmaps, we may end up using more memory than this.
*/
if (size < SZ_1G)
max_bytes = MAX_CACHE_BYTES_PER_GIG;
else
max_bytes = MAX_CACHE_BYTES_PER_GIG * div_u64(size, SZ_1G);
- /*
- * we want to account for 1 more bitmap than what we have so we can make
- * sure we don't go over our overall goal of MAX_CACHE_BYTES_PER_GIG as
- * we add more bitmaps.
- */
- bitmap_bytes = (ctl->total_bitmaps + 1) * ctl->unit;
-
- if (bitmap_bytes >= max_bytes) {
- ctl->extents_thresh = 0;
- return;
- }
+ bitmap_bytes = ctl->total_bitmaps * ctl->unit;
/*
* we want the extent entry threshold to always be at most 1/2 the max
@@ -2100,6 +2092,10 @@ static bool use_bitmap(struct btrfs_free_space_ctl *ctl,
forced = true;
#endif
+ /* This is a way to reclaim large regions from the bitmaps. */
+ if (!forced && info->bytes >= FORCE_EXTENT_THRESHOLD)
+ return false;
+
/*
* If we are below the extents threshold then we can add this as an
* extent, and don't have to deal with the bitmap
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* [PATCH 22/22] btrfs: make smaller extents more likely to go into bitmaps
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (20 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 21/22] btrfs: increase the metadata allowance for the free_space_cache Dennis Zhou
@ 2019-12-14 0:22 ` Dennis Zhou
2019-12-17 14:55 ` [PATCH v6 00/22] btrfs: async discard support David Sterba
2019-12-30 18:13 ` David Sterba
23 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-14 0:22 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
It's less than ideal for small extents to eat into our extent budget, so
force extents <= 32KB into the bitmaps save for the first handful.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/free-space-cache.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 6a313946a8da..42db751aa896 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -2108,8 +2108,8 @@ static bool use_bitmap(struct btrfs_free_space_ctl *ctl,
* of cache left then go ahead an dadd them, no sense in adding
* the overhead of a bitmap if we don't have to.
*/
- if (info->bytes <= fs_info->sectorsize * 4) {
- if (ctl->free_extents * 2 <= ctl->extents_thresh)
+ if (info->bytes <= fs_info->sectorsize * 8) {
+ if (ctl->free_extents * 3 <= ctl->extents_thresh)
return false;
} else {
return false;
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* Re: [PATCH v6 00/22] btrfs: async discard support
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (21 preceding siblings ...)
2019-12-14 0:22 ` [PATCH 22/22] btrfs: make smaller extents more likely to go into bitmaps Dennis Zhou
@ 2019-12-17 14:55 ` David Sterba
2019-12-18 0:06 ` Dennis Zhou
2019-12-30 18:13 ` David Sterba
23 siblings, 1 reply; 61+ messages in thread
From: David Sterba @ 2019-12-17 14:55 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Fri, Dec 13, 2019 at 04:22:09PM -0800, Dennis Zhou wrote:
> Hello,
>
> Dave reported a lockdep issue [1]. I'm a bit surprised as I can't repro
> it, but it obviously is right. I believe I fixed the issue by moving the
> fully trimmed check outside of the block_group lock. I mistakingly
> thought the btrfs_block_group lock subsumed btrfs_free_space_ctl
> tree_lock. This clearly isn't the case.
>
> Changes in v6:
> - Move the fully trimmed check outside of the block_group lock.
v6 passed fstests, with some weird test failures that don't seem to be
related to the patchset.
Meanwhile I did manual test how the discard behaves. The workload was
a series of linux git checkouts of various release tags (ie. this should
provide some freed extents and coalesce them eventually to get larger
chunks to discard), then a simple large file copy, sync, remove, sync.
The discards going down to the device were followin the maximum default
size (64M) but I observed that only one range was discarded per 10
seconds, while the other stats there are many more extents to discard.
For the large file it took like 5-10 cycles to send all the trimmed
ranges, the discardable_extents decreased by one each time until it
reached ... -1. At this point the discardable bytes were -16384, so
thre's some accounting problem.
This happened also when I deleted everything from the filesystem and ran
full balance.
Regarding the slow io submission, I tried to increase the iops value,
default was 10, but 100 and 1000 made no change. Increasing the maximum
discard request size to 128M worked (when there was such long extent
ready). I was expecting a burst of like 4 consecutive IOs after a 600MB
file is deleted. I did not try to tweak bps_limit because there was
nothing to limit.
So this is something to fix but otherwise the patchset seems to be ok
for adding to misc-next. Due to the timing of the end of the year and
that we're already at rc2, this will be the main feature in 5.6.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH v6 00/22] btrfs: async discard support
2019-12-17 14:55 ` [PATCH v6 00/22] btrfs: async discard support David Sterba
@ 2019-12-18 0:06 ` Dennis Zhou
2019-12-19 2:03 ` Dennis Zhou
2019-12-19 20:34 ` David Sterba
0 siblings, 2 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-18 0:06 UTC (permalink / raw)
To: David Sterba
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Tue, Dec 17, 2019 at 03:55:41PM +0100, David Sterba wrote:
> On Fri, Dec 13, 2019 at 04:22:09PM -0800, Dennis Zhou wrote:
> > Hello,
> >
> > Dave reported a lockdep issue [1]. I'm a bit surprised as I can't repro
> > it, but it obviously is right. I believe I fixed the issue by moving the
> > fully trimmed check outside of the block_group lock. I mistakingly
> > thought the btrfs_block_group lock subsumed btrfs_free_space_ctl
> > tree_lock. This clearly isn't the case.
> >
> > Changes in v6:
> > - Move the fully trimmed check outside of the block_group lock.
>
> v6 passed fstests, with some weird test failures that don't seem to be
> related to the patchset.
Yay!
>
> Meanwhile I did manual test how the discard behaves. The workload was
> a series of linux git checkouts of various release tags (ie. this should
> provide some freed extents and coalesce them eventually to get larger
> chunks to discard), then a simple large file copy, sync, remove, sync.
>
> The discards going down to the device were followin the maximum default
> size (64M) but I observed that only one range was discarded per 10
> seconds, while the other stats there are many more extents to discard.
>
> For the large file it took like 5-10 cycles to send all the trimmed
> ranges, the discardable_extents decreased by one each time until it
> reached ... -1. At this point the discardable bytes were -16384, so
> thre's some accounting problem.
>
> This happened also when I deleted everything from the filesystem and ran
> full balance.
>
Oh no :(. I've been trying to repro with some limited checking out and
syncing, then subsequently removing everything and calling balance. It
is coming out to be 0 for me. I'll try harder to repro this and fix it.
> Regarding the slow io submission, I tried to increase the iops value,
> default was 10, but 100 and 1000 made no change. Increasing the maximum
> discard request size to 128M worked (when there was such long extent
> ready). I was expecting a burst of like 4 consecutive IOs after a 600MB
> file is deleted. I did not try to tweak bps_limit because there was
> nothing to limit.
>
Ah so there's actually a max time between discards set to 10 seconds as
the maximum timeout is calculated over 6 hours. So if we only have 6
extents, we'd discard 1 per hour(ish given it decays), but this is
clamped to 10 seconds.
At least on our servers, we seem to discard at a reasonable rate to
prevent performance penalties during a large number of reads and writes
while maintaining reasonable write amplification performance. Also,
metadata blocks aren't tracked, so on freeing of a whole metadata block
group (minus relocation), we'll trickle discards slightly slower than
expected.
> So this is something to fix but otherwise the patchset seems to be ok
> for adding to misc-next. Due to the timing of the end of the year and
> that we're already at rc2, this will be the main feature in 5.6.
I'll report back if I continue having trouble reproing it.
Thanks v5.6 sounds good to me!
Dennis
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 09/22] btrfs: make UUID/debug have its own kobject
2019-12-14 0:22 ` [PATCH 09/22] btrfs: make UUID/debug have its own kobject Dennis Zhou
@ 2019-12-18 11:45 ` Anand Jain
0 siblings, 0 replies; 61+ messages in thread
From: Anand Jain @ 2019-12-18 11:45 UTC (permalink / raw)
To: Dennis Zhou, David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs
On 12/14/19 8:22 AM, Dennis Zhou wrote:
> Btrfs only allowed attributes to be exposed in debug/. Let's let other
> groups be created by making debug its own kobject.
>
> This also makes the per-fs debug options separate from the global
> features mount attributes. This seems to be needed as
> sysfs_create_files() requires const struct attribute * while
> sysfs_create_group() can take struct attribute *. This seems nicer as
> per file system, you'll probably use to_fs_info().
>
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 10/22] btrfs: add discard sysfs directory
2019-12-14 0:22 ` [PATCH 10/22] btrfs: add discard sysfs directory Dennis Zhou
@ 2019-12-18 11:45 ` Anand Jain
0 siblings, 0 replies; 61+ messages in thread
From: Anand Jain @ 2019-12-18 11:45 UTC (permalink / raw)
To: Dennis Zhou, David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs
On 12/14/19 8:22 AM, Dennis Zhou wrote:
> Setup sysfs directory for discard stats + tunables.
>
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 08/22] btrfs: add removal calls for sysfs debug/
2019-12-14 0:22 ` [PATCH 08/22] btrfs: add removal calls for sysfs debug/ Dennis Zhou
@ 2019-12-18 11:45 ` Anand Jain
0 siblings, 0 replies; 61+ messages in thread
From: Anand Jain @ 2019-12-18 11:45 UTC (permalink / raw)
To: Dennis Zhou, David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs
On 12/14/19 8:22 AM, Dennis Zhou wrote:
> We probably should call sysfs_remove_group() on debug/.
>
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
Reviewed-by: Anand Jain <anand.jain@oracle.com>
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH v6 00/22] btrfs: async discard support
2019-12-18 0:06 ` Dennis Zhou
@ 2019-12-19 2:03 ` Dennis Zhou
2019-12-19 20:06 ` David Sterba
2019-12-19 20:34 ` David Sterba
1 sibling, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-12-19 2:03 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, David Sterba, Chris Mason, Josef Bacik,
Omar Sandoval, kernel-team, linux-btrfs
On Tue, Dec 17, 2019 at 07:06:00PM -0500, Dennis Zhou wrote:
> On Tue, Dec 17, 2019 at 03:55:41PM +0100, David Sterba wrote:
> > On Fri, Dec 13, 2019 at 04:22:09PM -0800, Dennis Zhou wrote:
> > > Hello,
> > >
> > > Dave reported a lockdep issue [1]. I'm a bit surprised as I can't repro
> > > it, but it obviously is right. I believe I fixed the issue by moving the
> > > fully trimmed check outside of the block_group lock. I mistakingly
> > > thought the btrfs_block_group lock subsumed btrfs_free_space_ctl
> > > tree_lock. This clearly isn't the case.
> > >
> > > Changes in v6:
> > > - Move the fully trimmed check outside of the block_group lock.
> >
> > v6 passed fstests, with some weird test failures that don't seem to be
> > related to the patchset.
>
> Yay!
>
> >
> > Meanwhile I did manual test how the discard behaves. The workload was
> > a series of linux git checkouts of various release tags (ie. this should
> > provide some freed extents and coalesce them eventually to get larger
> > chunks to discard), then a simple large file copy, sync, remove, sync.
> >
> > The discards going down to the device were followin the maximum default
> > size (64M) but I observed that only one range was discarded per 10
> > seconds, while the other stats there are many more extents to discard.
> >
> > For the large file it took like 5-10 cycles to send all the trimmed
> > ranges, the discardable_extents decreased by one each time until it
> > reached ... -1. At this point the discardable bytes were -16384, so
> > thre's some accounting problem.
> >
> > This happened also when I deleted everything from the filesystem and ran
> > full balance.
> >
Also were these both on fresh file systems so it seems reproducible for
you?
>
> Oh no :(. I've been trying to repro with some limited checking out and
> syncing, then subsequently removing everything and calling balance. It
> is coming out to be 0 for me. I'll try harder to repro this and fix it.
>
> > Regarding the slow io submission, I tried to increase the iops value,
> > default was 10, but 100 and 1000 made no change. Increasing the maximum
> > discard request size to 128M worked (when there was such long extent
> > ready). I was expecting a burst of like 4 consecutive IOs after a 600MB
> > file is deleted. I did not try to tweak bps_limit because there was
> > nothing to limit.
> >
>
> Ah so there's actually a max time between discards set to 10 seconds as
> the maximum timeout is calculated over 6 hours. So if we only have 6
> extents, we'd discard 1 per hour(ish given it decays), but this is
> clamped to 10 seconds.
>
> At least on our servers, we seem to discard at a reasonable rate to
> prevent performance penalties during a large number of reads and writes
> while maintaining reasonable write amplification performance. Also,
> metadata blocks aren't tracked, so on freeing of a whole metadata block
> group (minus relocation), we'll trickle discards slightly slower than
> expected.
>
>
> > So this is something to fix but otherwise the patchset seems to be ok
> > for adding to misc-next. Due to the timing of the end of the year and
> > that we're already at rc2, this will be the main feature in 5.6.
>
> I'll report back if I continue having trouble reproing it.
>
I spent the day trying to repro against ext/dzhou-async-discard-v6
without any luck... I've been running the following:
$ mkfs.btrfs -f /dev/nvme0n1
$ mount -t btrfs -o discard=async /dev/nvme0n1 mnt
$ cd mnt
$ bash ../age_btrfs.sh .
where age_btrfs.sh is from [1].
If I delete arbitrary subvolumes, sync, and then run balance:
$ btrfs balance start --full-balance .
It all seems to resolve to 0 after some time. I haven't seen a negative
case on either of my 2 boxes. I've also tried unmounting and then
remounting, deleting and removing more free space items.
I'm still considering how this can happen. Possibly bad load of free
space cache and then freeing of the block group? Because being off by
just 1 and it not accumulating seems to be a real corner case here.
Adding asserts in btrfs_discard_update_discardable() might give us
insight to which callsite is responsible for going below 0.
[1] https://github.com/osandov/osandov-linux/blob/master/scripts/age_btrfs.sh
Thanks,
Dennis
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH v6 00/22] btrfs: async discard support
2019-12-19 2:03 ` Dennis Zhou
@ 2019-12-19 20:06 ` David Sterba
2019-12-19 21:22 ` Dennis Zhou
0 siblings, 1 reply; 61+ messages in thread
From: David Sterba @ 2019-12-19 20:06 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, David Sterba, Chris Mason, Josef Bacik,
Omar Sandoval, kernel-team, linux-btrfs
On Wed, Dec 18, 2019 at 08:03:37PM -0600, Dennis Zhou wrote:
> > > This happened also when I deleted everything from the filesystem and ran
> > > full balance.
>
> Also were these both on fresh file systems so it seems reproducible for
> you?
Yes the filesystem was freshly created before the test.
No luck reproducing it, I tried to repeat the steps as before but the
timing must make a difference and the numbers always ended up as 0
(bytes) 0 (extents).
> > I'll report back if I continue having trouble reproing it.
>
> I spent the day trying to repro against ext/dzhou-async-discard-v6
> without any luck... I've been running the following:
>
> $ mkfs.btrfs -f /dev/nvme0n1
> $ mount -t btrfs -o discard=async /dev/nvme0n1 mnt
> $ cd mnt
> $ bash ../age_btrfs.sh .
>
> where age_btrfs.sh is from [1].
>
> If I delete arbitrary subvolumes, sync, and then run balance:
> $ btrfs balance start --full-balance .
> It all seems to resolve to 0 after some time. I haven't seen a negative
> case on either of my 2 boxes. I've also tried unmounting and then
> remounting, deleting and removing more free space items.
>
> I'm still considering how this can happen. Possibly bad load of free
> space cache and then freeing of the block group? Because being off by
> just 1 and it not accumulating seems to be a real corner case here.
>
> Adding asserts in btrfs_discard_update_discardable() might give us
> insight to which callsite is responsible for going below 0.
Yeah more asserts would be good.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH v6 00/22] btrfs: async discard support
2019-12-18 0:06 ` Dennis Zhou
2019-12-19 2:03 ` Dennis Zhou
@ 2019-12-19 20:34 ` David Sterba
2019-12-19 21:17 ` Dennis Zhou
1 sibling, 1 reply; 61+ messages in thread
From: David Sterba @ 2019-12-19 20:34 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, David Sterba, Chris Mason, Josef Bacik,
Omar Sandoval, kernel-team, linux-btrfs
On Tue, Dec 17, 2019 at 07:06:00PM -0500, Dennis Zhou wrote:
> > Regarding the slow io submission, I tried to increase the iops value,
> > default was 10, but 100 and 1000 made no change. Increasing the maximum
> > discard request size to 128M worked (when there was such long extent
> > ready). I was expecting a burst of like 4 consecutive IOs after a 600MB
> > file is deleted. I did not try to tweak bps_limit because there was
> > nothing to limit.
>
> Ah so there's actually a max time between discards set to 10 seconds as
> the maximum timeout is calculated over 6 hours. So if we only have 6
> extents, we'd discard 1 per hour(ish given it decays), but this is
> clamped to 10 seconds.
>
> At least on our servers, we seem to discard at a reasonable rate to
> prevent performance penalties during a large number of reads and writes
> while maintaining reasonable write amplification performance. Also,
> metadata blocks aren't tracked, so on freeing of a whole metadata block
> group (minus relocation), we'll trickle discards slightly slower than
> expected.
So after watching the sysfs numbers, my observation is that the overall
strategy of the async discard is to wait for larger ranges and discard
one range every 10 seconds. This is a slow process, but this makes sense
when there are reads or writes going on so the discard IO penalty is
marginal.
Running full fstrim will flush all the discardable extents so there's a
way to reset the discardable queue. What I still don't see as optimal is
the single discard request sent per one period. Namely because there's
the iops_limit knob.
My idea is that each timeout, 'iops_limit' times 'max_discard_size' is
called, so the discard batches are large in total. However, this has
impact on reads and writes and also on the device itself, I'm not sure
if the too frequent discards are not making things worse (as this is a
known problem).
I'm interested in more strategies that you could have tested in your
setups, either bps based or rate limited etc. The current one seems ok
for first implementation but we might want to tune it once we get
feedback from more users.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH v6 00/22] btrfs: async discard support
2019-12-19 20:34 ` David Sterba
@ 2019-12-19 21:17 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-19 21:17 UTC (permalink / raw)
To: dsterba, Dennis Zhou, David Sterba, Chris Mason, Josef Bacik,
Omar Sandoval, kernel-team, linux-btrfs
On Thu, Dec 19, 2019 at 09:34:38PM +0100, David Sterba wrote:
> On Tue, Dec 17, 2019 at 07:06:00PM -0500, Dennis Zhou wrote:
> > > Regarding the slow io submission, I tried to increase the iops value,
> > > default was 10, but 100 and 1000 made no change. Increasing the maximum
> > > discard request size to 128M worked (when there was such long extent
> > > ready). I was expecting a burst of like 4 consecutive IOs after a 600MB
> > > file is deleted. I did not try to tweak bps_limit because there was
> > > nothing to limit.
> >
> > Ah so there's actually a max time between discards set to 10 seconds as
> > the maximum timeout is calculated over 6 hours. So if we only have 6
> > extents, we'd discard 1 per hour(ish given it decays), but this is
> > clamped to 10 seconds.
> >
> > At least on our servers, we seem to discard at a reasonable rate to
> > prevent performance penalties during a large number of reads and writes
> > while maintaining reasonable write amplification performance. Also,
> > metadata blocks aren't tracked, so on freeing of a whole metadata block
> > group (minus relocation), we'll trickle discards slightly slower than
> > expected.
>
> So after watching the sysfs numbers, my observation is that the overall
> strategy of the async discard is to wait for larger ranges and discard
> one range every 10 seconds. This is a slow process, but this makes sense
> when there are reads or writes going on so the discard IO penalty is
> marginal.
>
Yeah, (un)fortunately on our systems we're running chef fairly
frequently which results in a lot of IO in addition to package
deployment. This actually drives the system to have a fairly high steady
state number of untrimmed extents and results in a bit faster paced
discarding rate.
> Running full fstrim will flush all the discardable extents so there's a
> way to reset the discardable queue. What I still don't see as optimal is
> the single discard request sent per one period. Namely because there's
> the iops_limit knob.
>
Yeah, it's not really ideal at the moment for much slower paced systems
such as our own laptops. Adding persistence would also make a big
difference here.
> My idea is that each timeout, 'iops_limit' times 'max_discard_size' is
> called, so the discard batches are large in total. However, this has
> impact on reads and writes and also on the device itself, I'm not sure
> if the too frequent discards are not making things worse (as this is a
> known problem).
>
I spent a bit of time looking at the impact of discard on some drives
and my conclusion was that the iops rate is more impactful than the size
of the discards (within reason, which is why there's the
max_discard_size). On a particular drive, I noticed if I went over 10
iops of discards on a sustained simple read write workload, the
latencies would double. That's kind of where the 10 iops limit comes
from. Given the latency impact, that's why this more or less trickles it
down in pieces rather than as a larger batch.
> I'm interested in more strategies that you could have tested in your
> setups, either bps based or rate limited etc. The current one seems ok
> for first implementation but we might want to tune it once we get
> feedback from more users.
Definitely, one of the things I want to do is experiment with different
limits and see how this all correlates with write amplification. I'm
sure there's some happy medium that we can identify that's a lot less
arbitrary than what's current set forth. I imagine it should result in
some graph that we can correlate delay and rate of discarding to a
particular write amp given a fixed workload.
Thanks,
Dennis
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH v6 00/22] btrfs: async discard support
2019-12-19 20:06 ` David Sterba
@ 2019-12-19 21:22 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-19 21:22 UTC (permalink / raw)
To: dsterba, David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Thu, Dec 19, 2019 at 09:06:07PM +0100, David Sterba wrote:
> On Wed, Dec 18, 2019 at 08:03:37PM -0600, Dennis Zhou wrote:
> > > > This happened also when I deleted everything from the filesystem and ran
> > > > full balance.
> >
> > Also were these both on fresh file systems so it seems reproducible for
> > you?
>
> Yes the filesystem was freshly created before the test.
>
> No luck reproducing it, I tried to repeat the steps as before but the
> timing must make a difference and the numbers always ended up as 0
> (bytes) 0 (extents).
>
> > > I'll report back if I continue having trouble reproing it.
> >
> > I spent the day trying to repro against ext/dzhou-async-discard-v6
> > without any luck... I've been running the following:
> >
> > $ mkfs.btrfs -f /dev/nvme0n1
> > $ mount -t btrfs -o discard=async /dev/nvme0n1 mnt
> > $ cd mnt
> > $ bash ../age_btrfs.sh .
> >
> > where age_btrfs.sh is from [1].
> >
> > If I delete arbitrary subvolumes, sync, and then run balance:
> > $ btrfs balance start --full-balance .
> > It all seems to resolve to 0 after some time. I haven't seen a negative
> > case on either of my 2 boxes. I've also tried unmounting and then
> > remounting, deleting and removing more free space items.
> >
> > I'm still considering how this can happen. Possibly bad load of free
> > space cache and then freeing of the block group? Because being off by
> > just 1 and it not accumulating seems to be a real corner case here.
> >
> > Adding asserts in btrfs_discard_update_discardable() might give us
> > insight to which callsite is responsible for going below 0.
>
> Yeah more asserts would be good.
I'll add a few assert patches and some code to ensure that life can
still move on properly if we do hit the -1 case. I think it probably has
something to do with free space cache removal as it can't be a simple
corner case, otherwise we'd see the -1 accumulating much more easily.
What does puzzle me is it's a single nodesize that I'm off by and not
some other random number.
Thanks,
Dennis
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 13/22] btrfs: calculate discard delay based on number of extents
2019-12-14 0:22 ` [PATCH 13/22] btrfs: calculate discard delay based on number of extents Dennis Zhou
@ 2019-12-30 16:50 ` David Sterba
2020-01-02 16:45 ` Dennis Zhou
0 siblings, 1 reply; 61+ messages in thread
From: David Sterba @ 2019-12-30 16:50 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Fri, Dec 13, 2019 at 04:22:22PM -0800, Dennis Zhou wrote:
> Use the number of discardable extents to help guide our discard delay
> interval. This value is reevaluated every transaction commit.
This is too short compared to the added code, please enhance it eg. by
briefly describing what values are tracked and how are they related
together.
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> ---
> fs/btrfs/ctree.h | 2 ++
> fs/btrfs/discard.c | 53 ++++++++++++++++++++++++++++++++++++++----
> fs/btrfs/discard.h | 1 +
> fs/btrfs/extent-tree.c | 4 +++-
> fs/btrfs/sysfs.c | 31 ++++++++++++++++++++++++
> 5 files changed, 86 insertions(+), 5 deletions(-)
>
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index 18df0e40a282..98979566e281 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -469,6 +469,8 @@ struct btrfs_discard_ctl {
> struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
> atomic_t discardable_extents;
> atomic64_t discardable_bytes;
There are some type mismatches for the two:
> + u32 delay;
> + u32 iops_limit;
unsigned for iops
> };
>
> /* delayed seq elem */
> diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
> index e8c2623617fd..160713e04d0c 100644
> --- a/fs/btrfs/discard.c
> +++ b/fs/btrfs/discard.c
> @@ -15,6 +15,11 @@
> #define BTRFS_DISCARD_DELAY (120ULL * NSEC_PER_SEC)
> #define BTRFS_DISCARD_UNUSED_DELAY (10ULL * NSEC_PER_SEC)
>
> +/* Target completion latency of discarding all discardable extents. */
> +#define BTRFS_DISCARD_TARGET_MSEC (6 * 60 * 60UL * MSEC_PER_SEC)
> +#define BTRFS_DISCARD_MAX_DELAY (10000UL)
> +#define BTRFS_DISCARD_MAX_IOPS (10UL)
All unsigned longs
> +
> static struct list_head *get_discard_list(struct btrfs_discard_ctl *discard_ctl,
> struct btrfs_block_group *block_group)
> {
> @@ -239,11 +244,18 @@ void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
>
> block_group = find_next_block_group(discard_ctl, now);
> if (block_group) {
> - u64 delay = 0;
> + u32 delay = discard_ctl->delay;
> +
> + /*
> + * This timeout is to hopefully prevent immediate discarding
> + * in a recently allocated block group.
> + */
> + if (now < block_group->discard_eligible_time) {
> + u64 bg_timeout = (block_group->discard_eligible_time -
> + now);
>
> - if (now < block_group->discard_eligible_time)
> - delay = nsecs_to_jiffies(
> - block_group->discard_eligible_time - now);
> + delay = max_t(u64, delay, nsecs_to_jiffies(bg_timeout));
delay is u32, nsecs_to_jiffies returns unsigned long, so delay should be
also unsigned long everywhere.
> + }
>
> mod_delayed_work(discard_ctl->discard_workers,
> &discard_ctl->work,
> @@ -348,6 +360,37 @@ bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl)
> test_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags));
> }
>
> +/**
> + * btrfs_discard_calc_delay - recalculate the base delay
> + * @discard_ctl: discard control
> + *
> + * Recalculate the base delay which is based off the total number of
> + * discardable_extents. Clamp this with the iops_limit and
> + * BTRFS_DISCARD_MAX_DELAY.
> + */
> +void btrfs_discard_calc_delay(struct btrfs_discard_ctl *discard_ctl)
> +{
> + s32 discardable_extents =
> + atomic_read(&discard_ctl->discardable_extents);
> + s32 iops_limit;
signed for iops
> + unsigned long delay;
> +
> + if (!discardable_extents)
> + return;
> +
> + spin_lock(&discard_ctl->lock);
> +
> + iops_limit = READ_ONCE(discard_ctl->iops_limit);
unsigned -> signed
> + if (iops_limit)
> + iops_limit = MSEC_PER_SEC / iops_limit;
> +
> + delay = BTRFS_DISCARD_TARGET_MSEC / discardable_extents;
> + delay = clamp_t(s32, delay, iops_limit, BTRFS_DISCARD_MAX_DELAY);
> + discard_ctl->delay = msecs_to_jiffies(delay);
unsigned long -> u32
> +
> + spin_unlock(&discard_ctl->lock);
> +}
> +
> /**
> * btrfs_discard_update_discardable - propagate discard counters
> * @block_group: block_group of interest
> @@ -476,6 +519,8 @@ void btrfs_discard_init(struct btrfs_fs_info *fs_info)
>
> atomic_set(&discard_ctl->discardable_extents, 0);
> atomic64_set(&discard_ctl->discardable_bytes, 0);
> + discard_ctl->delay = BTRFS_DISCARD_MAX_DELAY;
> + discard_ctl->iops_limit = BTRFS_DISCARD_MAX_IOPS;
> }
>
> void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
> diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
> index 7e3680dd82ce..3ed6855e24da 100644
> --- a/fs/btrfs/discard.h
> +++ b/fs/btrfs/discard.h
> @@ -17,6 +17,7 @@ void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
> bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl);
>
> /* Update operations. */
> +void btrfs_discard_calc_delay(struct btrfs_discard_ctl *discard_ctl);
> void btrfs_discard_update_discardable(struct btrfs_block_group *block_group,
> struct btrfs_free_space_ctl *ctl);
>
> diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
> index 2c12366cfde5..0163fdd59f8f 100644
> --- a/fs/btrfs/extent-tree.c
> +++ b/fs/btrfs/extent-tree.c
> @@ -2935,8 +2935,10 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
> cond_resched();
> }
>
> - if (btrfs_test_opt(fs_info, DISCARD_ASYNC))
> + if (btrfs_test_opt(fs_info, DISCARD_ASYNC)) {
> + btrfs_discard_calc_delay(&fs_info->discard_ctl);
> btrfs_discard_schedule_work(&fs_info->discard_ctl, true);
> + }
>
> /*
> * Transaction is finished. We don't need the lock anymore. We
> diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
> index e799d4488d72..79139b4b4f0a 100644
> --- a/fs/btrfs/sysfs.c
> +++ b/fs/btrfs/sysfs.c
> @@ -344,6 +344,36 @@ static const struct attribute_group btrfs_static_feature_attr_group = {
> */
> #define discard_to_fs_info(_kobj) to_fs_info((_kobj)->parent->parent)
>
> +static ssize_t btrfs_discard_iops_limit_show(struct kobject *kobj,
> + struct kobj_attribute *a,
> + char *buf)
> +{
> + struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
> +
> + return snprintf(buf, PAGE_SIZE, "%u\n",
> + READ_ONCE(fs_info->discard_ctl.iops_limit));
> +}
> +
> +static ssize_t btrfs_discard_iops_limit_store(struct kobject *kobj,
> + struct kobj_attribute *a,
> + const char *buf, size_t len)
> +{
> + struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
> + struct btrfs_discard_ctl *discard_ctl = &fs_info->discard_ctl;
> + u32 iops_limit;
> + int ret;
> +
> + ret = kstrtou32(buf, 10, &iops_limit);
> + if (ret)
> + return -EINVAL;
> +
> + WRITE_ONCE(discard_ctl->iops_limit, iops_limit);
> +
> + return len;
> +}
> +BTRFS_ATTR_RW(discard, iops_limit, btrfs_discard_iops_limit_show,
> + btrfs_discard_iops_limit_store);
> +
> static ssize_t btrfs_discardable_extents_show(struct kobject *kobj,
> struct kobj_attribute *a,
> char *buf)
> @@ -367,6 +397,7 @@ static ssize_t btrfs_discardable_bytes_show(struct kobject *kobj,
> BTRFS_ATTR(discard, discardable_bytes, btrfs_discardable_bytes_show);
>
> static const struct attribute *discard_debug_attrs[] = {
> + BTRFS_ATTR_PTR(discard, iops_limit),
> BTRFS_ATTR_PTR(discard, discardable_extents),
> BTRFS_ATTR_PTR(discard, discardable_bytes),
> NULL,
> --
> 2.17.1
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 19/22] btrfs: keep track of discard reuse stats
2019-12-14 0:22 ` [PATCH 19/22] btrfs: keep track of discard reuse stats Dennis Zhou
@ 2019-12-30 17:33 ` David Sterba
2020-01-02 16:57 ` Dennis Zhou
0 siblings, 1 reply; 61+ messages in thread
From: David Sterba @ 2019-12-30 17:33 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Fri, Dec 13, 2019 at 04:22:28PM -0800, Dennis Zhou wrote:
> Keep track of how much we are discarding and how often we are reusing
> with async discard.
>
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> ---
> fs/btrfs/ctree.h | 3 +++
> fs/btrfs/discard.c | 7 +++++++
> fs/btrfs/free-space-cache.c | 14 ++++++++++++++
> fs/btrfs/sysfs.c | 36 ++++++++++++++++++++++++++++++++++++
> 4 files changed, 60 insertions(+)
>
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index dddf51e27bed..edfbe6060e8d 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -474,6 +474,9 @@ struct btrfs_discard_ctl {
> u32 delay;
> u32 iops_limit;
> u64 bps_limit;
> + u64 discard_extent_bytes;
> + u64 discard_bitmap_bytes;
> + atomic64_t discard_bytes_saved;
> };
>
> /* delayed seq elem */
> diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
> index 55ad357e65f3..fe73814526ef 100644
> --- a/fs/btrfs/discard.c
> +++ b/fs/btrfs/discard.c
> @@ -419,11 +419,15 @@ static void btrfs_discard_workfn(struct work_struct *work)
> block_group->discard_cursor,
> btrfs_block_group_end(block_group),
> minlen, maxlen, true);
> + WRITE_ONCE(discard_ctl->discard_bitmap_bytes,
> + discard_ctl->discard_bitmap_bytes + trimmed);
The same argument is used for read and write, this does not seem to be a
clear usage pattern for WRITE_ONCE, is there are cleaner way to do that?
> } else {
> btrfs_trim_block_group_extents(block_group, &trimmed,
> block_group->discard_cursor,
> btrfs_block_group_end(block_group),
> minlen, true);
> + WRITE_ONCE(discard_ctl->discard_extent_bytes,
> + discard_ctl->discard_extent_bytes + trimmed);
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 18/22] btrfs: only keep track of data extents for async discard
2019-12-14 0:22 ` [PATCH 18/22] btrfs: only keep track of data extents for async discard Dennis Zhou
@ 2019-12-30 17:39 ` David Sterba
2020-01-02 16:55 ` Dennis Zhou
0 siblings, 1 reply; 61+ messages in thread
From: David Sterba @ 2019-12-30 17:39 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Fri, Dec 13, 2019 at 04:22:27PM -0800, Dennis Zhou wrote:
> As mentioned earlier, discarding data can be done either by issuing an
> explicit discard or implicitly by reusing the LBA. Metadata chunks see
> much more frequent reuse due to well it being metadata. So instead of
> explicitly discarding metadata blocks, just leave them be and let the
> latter implicit discarding be done for them.
>
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> ---
> fs/btrfs/block-group.h | 6 ++++++
> fs/btrfs/discard.c | 11 +++++++++--
> 2 files changed, 15 insertions(+), 2 deletions(-)
>
> diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h
> index 601e1d217e22..ee8441439a56 100644
> --- a/fs/btrfs/block-group.h
> +++ b/fs/btrfs/block-group.h
> @@ -182,6 +182,12 @@ static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
> return (block_group->start + block_group->length);
> }
>
> +static inline bool btrfs_is_block_group_data(
> + struct btrfs_block_group *block_group)
> +{
> + return (block_group->flags & BTRFS_BLOCK_GROUP_DATA);
What happens for mixed data and metadata block groups? As this is a
special mode that will likely lead to fragmented block groups I think
that async discard won't be able to help much.
I'd suggest to make the test explicit only for the separate block group
types and comment that mixed bg's are not supported.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 14/22] btrfs: add bps discard rate limit
2019-12-14 0:22 ` [PATCH 14/22] btrfs: add bps discard rate limit Dennis Zhou
@ 2019-12-30 17:58 ` David Sterba
2020-01-02 16:46 ` Dennis Zhou
0 siblings, 1 reply; 61+ messages in thread
From: David Sterba @ 2019-12-30 17:58 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Fri, Dec 13, 2019 at 04:22:23PM -0800, Dennis Zhou wrote:
> Provide an ability to rate limit based on mbps in addition to the iops
> delay calculated from number of discardable extents.
>
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> ---
> fs/btrfs/ctree.h | 2 ++
> fs/btrfs/discard.c | 18 ++++++++++++++++++
> fs/btrfs/sysfs.c | 31 +++++++++++++++++++++++++++++++
> 3 files changed, 51 insertions(+)
>
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index 98979566e281..1b2dae5962de 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -467,10 +467,12 @@ struct btrfs_discard_ctl {
> spinlock_t lock;
> struct btrfs_block_group *block_group;
> struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
> + u64 prev_discard;
> atomic_t discardable_extents;
> atomic64_t discardable_bytes;
> u32 delay;
> u32 iops_limit;
> + u64 bps_limit;
I think this could be u32, which makes it 4G per second. Byte
granularity is IMO not necessary so it could be KiB/sec instead and it
would be future safe with maximum limit of 4TiB/sec.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 15/22] btrfs: limit max discard size for async discard
2019-12-14 0:22 ` [PATCH 15/22] btrfs: limit max discard size for async discard Dennis Zhou
@ 2019-12-30 18:00 ` David Sterba
2019-12-30 18:08 ` David Sterba
1 sibling, 0 replies; 61+ messages in thread
From: David Sterba @ 2019-12-30 18:00 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Fri, Dec 13, 2019 at 04:22:24PM -0800, Dennis Zhou wrote:
> Throttle the maximum size of a discard so that we can provide an upper
> bound for the rate of async discard. While the block layer is able to
> split discards into the appropriate sized discards, we want to be able
> to account more accurately the rate at which we are consuming ncq slots
> as well as limit the upper bound of work for a discard.
>
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> ---
> fs/btrfs/discard.h | 5 ++++
> fs/btrfs/free-space-cache.c | 48 +++++++++++++++++++++++++++----------
> 2 files changed, 41 insertions(+), 12 deletions(-)
>
> diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
> index 3ed6855e24da..cb6ef0ab879d 100644
> --- a/fs/btrfs/discard.h
> +++ b/fs/btrfs/discard.h
> @@ -3,10 +3,15 @@
> #ifndef BTRFS_DISCARD_H
> #define BTRFS_DISCARD_H
>
> +#include <linux/sizes.h>
> +
> struct btrfs_fs_info;
> struct btrfs_discard_ctl;
> struct btrfs_block_group;
>
> +/* Discard size limits. */
> +#define BTRFS_ASYNC_DISCARD_MAX_SIZE (SZ_64M)
> +
> /* Work operations. */
> void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
> struct btrfs_block_group *block_group);
> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
> index 57df34480b93..0dbcea6c59f9 100644
> --- a/fs/btrfs/free-space-cache.c
> +++ b/fs/btrfs/free-space-cache.c
> @@ -3466,19 +3466,40 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
> if (entry->offset >= end)
> goto out_unlock;
>
> - extent_start = entry->offset;
> - extent_bytes = entry->bytes;
> - extent_trim_state = entry->trim_state;
> - start = max(start, extent_start);
> - bytes = min(extent_start + extent_bytes, end) - start;
> - if (bytes < minlen) {
> - spin_unlock(&ctl->tree_lock);
> - mutex_unlock(&ctl->cache_writeout_mutex);
> - goto next;
> - }
> + if (async) {
> + start = extent_start = entry->offset;
> + bytes = extent_bytes = entry->bytes;
Please avoid chain initializations, I'll fix that unless there are more
updates to this patch.
> + extent_trim_state = entry->trim_state;
> + if (bytes < minlen) {
> + spin_unlock(&ctl->tree_lock);
> + mutex_unlock(&ctl->cache_writeout_mutex);
> + goto next;
> + }
> + unlink_free_space(ctl, entry);
> + if (bytes > BTRFS_ASYNC_DISCARD_MAX_SIZE) {
> + bytes = extent_bytes =
> + BTRFS_ASYNC_DISCARD_MAX_SIZE;
> + entry->offset += BTRFS_ASYNC_DISCARD_MAX_SIZE;
> + entry->bytes -= BTRFS_ASYNC_DISCARD_MAX_SIZE;
> + link_free_space(ctl, entry);
> + } else {
> + kmem_cache_free(btrfs_free_space_cachep, entry);
> + }
> + } else {
> + extent_start = entry->offset;
> + extent_bytes = entry->bytes;
> + extent_trim_state = entry->trim_state;
> + start = max(start, extent_start);
> + bytes = min(extent_start + extent_bytes, end) - start;
> + if (bytes < minlen) {
> + spin_unlock(&ctl->tree_lock);
> + mutex_unlock(&ctl->cache_writeout_mutex);
> + goto next;
> + }
>
> - unlink_free_space(ctl, entry);
> - kmem_cache_free(btrfs_free_space_cachep, entry);
> + unlink_free_space(ctl, entry);
> + kmem_cache_free(btrfs_free_space_cachep, entry);
> + }
>
> spin_unlock(&ctl->tree_lock);
> trim_entry.start = extent_start;
> @@ -3643,6 +3664,9 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
> goto next;
> }
>
> + if (async && bytes > BTRFS_ASYNC_DISCARD_MAX_SIZE)
> + bytes = BTRFS_ASYNC_DISCARD_MAX_SIZE;
> +
> bitmap_clear_bits(ctl, entry, start, bytes);
> if (entry->bytes == 0)
> free_bitmap(ctl, entry);
> --
> 2.17.1
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 16/22] btrfs: make max async discard size tunable
2019-12-14 0:22 ` [PATCH 16/22] btrfs: make max async discard size tunable Dennis Zhou
@ 2019-12-30 18:05 ` David Sterba
2020-01-02 16:50 ` Dennis Zhou
0 siblings, 1 reply; 61+ messages in thread
From: David Sterba @ 2019-12-30 18:05 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Fri, Dec 13, 2019 at 04:22:25PM -0800, Dennis Zhou wrote:
> Expose max_discard_size as a tunable via sysfs.
>
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> ---
> fs/btrfs/ctree.h | 1 +
> fs/btrfs/discard.c | 1 +
> fs/btrfs/free-space-cache.c | 19 ++++++++++++-------
> fs/btrfs/sysfs.c | 31 +++++++++++++++++++++++++++++++
> 4 files changed, 45 insertions(+), 7 deletions(-)
>
> diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> index 1b2dae5962de..bf93ddbc773f 100644
> --- a/fs/btrfs/ctree.h
> +++ b/fs/btrfs/ctree.h
> @@ -470,6 +470,7 @@ struct btrfs_discard_ctl {
> u64 prev_discard;
> atomic_t discardable_extents;
> atomic64_t discardable_bytes;
> + u64 max_discard_size;
> u32 delay;
> u32 iops_limit;
> u64 bps_limit;
> diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
> index 085f36808e7f..dd5143f0283f 100644
> --- a/fs/btrfs/discard.c
> +++ b/fs/btrfs/discard.c
> @@ -536,6 +536,7 @@ void btrfs_discard_init(struct btrfs_fs_info *fs_info)
> discard_ctl->prev_discard = 0;
> atomic_set(&discard_ctl->discardable_extents, 0);
> atomic64_set(&discard_ctl->discardable_bytes, 0);
> + discard_ctl->max_discard_size = BTRFS_ASYNC_DISCARD_MAX_SIZE;
BTRFS_ASYNC_DISCARD_MAX_SIZE won't be the upper limit anymore but a
default so it should be named as such.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 15/22] btrfs: limit max discard size for async discard
2019-12-14 0:22 ` [PATCH 15/22] btrfs: limit max discard size for async discard Dennis Zhou
2019-12-30 18:00 ` David Sterba
@ 2019-12-30 18:08 ` David Sterba
2020-01-02 16:48 ` Dennis Zhou
1 sibling, 1 reply; 61+ messages in thread
From: David Sterba @ 2019-12-30 18:08 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Fri, Dec 13, 2019 at 04:22:24PM -0800, Dennis Zhou wrote:
> Throttle the maximum size of a discard so that we can provide an upper
> bound for the rate of async discard. While the block layer is able to
> split discards into the appropriate sized discards, we want to be able
> to account more accurately the rate at which we are consuming ncq slots
> as well as limit the upper bound of work for a discard.
>
> Signed-off-by: Dennis Zhou <dennis@kernel.org>
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> ---
> fs/btrfs/discard.h | 5 ++++
> fs/btrfs/free-space-cache.c | 48 +++++++++++++++++++++++++++----------
> 2 files changed, 41 insertions(+), 12 deletions(-)
>
> diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
> index 3ed6855e24da..cb6ef0ab879d 100644
> --- a/fs/btrfs/discard.h
> +++ b/fs/btrfs/discard.h
> @@ -3,10 +3,15 @@
> #ifndef BTRFS_DISCARD_H
> #define BTRFS_DISCARD_H
>
> +#include <linux/sizes.h>
> +
> struct btrfs_fs_info;
> struct btrfs_discard_ctl;
> struct btrfs_block_group;
>
> +/* Discard size limits. */
> +#define BTRFS_ASYNC_DISCARD_MAX_SIZE (SZ_64M)
> +
> /* Work operations. */
> void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
> struct btrfs_block_group *block_group);
> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
> index 57df34480b93..0dbcea6c59f9 100644
> --- a/fs/btrfs/free-space-cache.c
> +++ b/fs/btrfs/free-space-cache.c
> @@ -3466,19 +3466,40 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
> if (entry->offset >= end)
> goto out_unlock;
>
> - extent_start = entry->offset;
> - extent_bytes = entry->bytes;
> - extent_trim_state = entry->trim_state;
> - start = max(start, extent_start);
> - bytes = min(extent_start + extent_bytes, end) - start;
> - if (bytes < minlen) {
> - spin_unlock(&ctl->tree_lock);
> - mutex_unlock(&ctl->cache_writeout_mutex);
> - goto next;
> - }
> + if (async) {
> + start = extent_start = entry->offset;
> + bytes = extent_bytes = entry->bytes;
> + extent_trim_state = entry->trim_state;
> + if (bytes < minlen) {
> + spin_unlock(&ctl->tree_lock);
> + mutex_unlock(&ctl->cache_writeout_mutex);
> + goto next;
> + }
> + unlink_free_space(ctl, entry);
> + if (bytes > BTRFS_ASYNC_DISCARD_MAX_SIZE) {
> + bytes = extent_bytes =
> + BTRFS_ASYNC_DISCARD_MAX_SIZE;
> + entry->offset += BTRFS_ASYNC_DISCARD_MAX_SIZE;
> + entry->bytes -= BTRFS_ASYNC_DISCARD_MAX_SIZE;
> + link_free_space(ctl, entry);
> + } else {
> + kmem_cache_free(btrfs_free_space_cachep, entry);
> + }
> + } else {
> + extent_start = entry->offset;
> + extent_bytes = entry->bytes;
> + extent_trim_state = entry->trim_state;
This is common initialization for both async and sync cases so it could
be merged to a common block.
> + start = max(start, extent_start);
> + bytes = min(extent_start + extent_bytes, end) - start;
> + if (bytes < minlen) {
> + spin_unlock(&ctl->tree_lock);
> + mutex_unlock(&ctl->cache_writeout_mutex);
> + goto next;
> + }
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH v6 00/22] btrfs: async discard support
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
` (22 preceding siblings ...)
2019-12-17 14:55 ` [PATCH v6 00/22] btrfs: async discard support David Sterba
@ 2019-12-30 18:13 ` David Sterba
2019-12-30 18:49 ` Dennis Zhou
23 siblings, 1 reply; 61+ messages in thread
From: David Sterba @ 2019-12-30 18:13 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Fri, Dec 13, 2019 at 04:22:09PM -0800, Dennis Zhou wrote:
> Hello,
>
> Dave reported a lockdep issue [1]. I'm a bit surprised as I can't repro
> it, but it obviously is right. I believe I fixed the issue by moving the
> fully trimmed check outside of the block_group lock. I mistakingly
> thought the btrfs_block_group lock subsumed btrfs_free_space_ctl
> tree_lock. This clearly isn't the case.
>
> Changes in v6:
> - Move the fully trimmed check outside of the block_group lock.
>
> v5 is available here: [2].
>
> This series is on top of btrfs-devel#misc-next 7ee98bb808e2 + [3] and
> [4].
>
> [1] https://lore.kernel.org/linux-btrfs/20191210140438.GU2734@twin.jikos.cz/
> [2] https://lore.kernel.org/linux-btrfs/cover.1575919745.git.dennis@kernel.org/
> [3] https://lore.kernel.org/linux-btrfs/d934383ea528d920a95b6107daad6023b516f0f4.1576109087.git.dennis@kernel.org/
> [4] https://lore.kernel.org/linux-btrfs/20191209193846.18162-1-dennis@kernel.org/
>
> Dennis Zhou (22):
> bitmap: genericize percpu bitmap region iterators
> btrfs: rename DISCARD opt to DISCARD_SYNC
> btrfs: keep track of which extents have been discarded
> btrfs: keep track of cleanliness of the bitmap
> btrfs: add the beginning of async discard, discard workqueue
> btrfs: handle empty block_group removal
> btrfs: discard one region at a time in async discard
> btrfs: add removal calls for sysfs debug/
> btrfs: make UUID/debug have its own kobject
> btrfs: add discard sysfs directory
> btrfs: track discardable extents for async discard
> btrfs: keep track of discardable_bytes
> btrfs: calculate discard delay based on number of extents
> btrfs: add bps discard rate limit
> btrfs: limit max discard size for async discard
> btrfs: make max async discard size tunable
> btrfs: have multiple discard lists
> btrfs: only keep track of data extents for async discard
> btrfs: keep track of discard reuse stats
> btrfs: add async discard header
> btrfs: increase the metadata allowance for the free_space_cache
> btrfs: make smaller extents more likely to go into bitmaps
Patches 1-12 merged to a temporary misc-next but I haven't pushed it as
misc-next yet (it's misc-next-with-discard-v6 in my github repo). There
are some comments to patch 13 and up so please send them either as
replies or as a shorter series. Thanks.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH v6 00/22] btrfs: async discard support
2019-12-30 18:13 ` David Sterba
@ 2019-12-30 18:49 ` Dennis Zhou
2020-01-02 13:22 ` David Sterba
0 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-12-30 18:49 UTC (permalink / raw)
To: David Sterba
Cc: Dennis Zhou, David Sterba, Chris Mason, Josef Bacik,
Omar Sandoval, kernel-team, linux-btrfs
On Mon, Dec 30, 2019 at 07:13:18PM +0100, David Sterba wrote:
> On Fri, Dec 13, 2019 at 04:22:09PM -0800, Dennis Zhou wrote:
> > Hello,
> >
> > Dave reported a lockdep issue [1]. I'm a bit surprised as I can't repro
> > it, but it obviously is right. I believe I fixed the issue by moving the
> > fully trimmed check outside of the block_group lock. I mistakingly
> > thought the btrfs_block_group lock subsumed btrfs_free_space_ctl
> > tree_lock. This clearly isn't the case.
> >
> > Changes in v6:
> > - Move the fully trimmed check outside of the block_group lock.
> >
> > v5 is available here: [2].
> >
> > This series is on top of btrfs-devel#misc-next 7ee98bb808e2 + [3] and
> > [4].
> >
> > [1] https://lore.kernel.org/linux-btrfs/20191210140438.GU2734@twin.jikos.cz/
> > [2] https://lore.kernel.org/linux-btrfs/cover.1575919745.git.dennis@kernel.org/
> > [3] https://lore.kernel.org/linux-btrfs/d934383ea528d920a95b6107daad6023b516f0f4.1576109087.git.dennis@kernel.org/
> > [4] https://lore.kernel.org/linux-btrfs/20191209193846.18162-1-dennis@kernel.org/
> >
> > Dennis Zhou (22):
> > bitmap: genericize percpu bitmap region iterators
> > btrfs: rename DISCARD opt to DISCARD_SYNC
> > btrfs: keep track of which extents have been discarded
> > btrfs: keep track of cleanliness of the bitmap
> > btrfs: add the beginning of async discard, discard workqueue
> > btrfs: handle empty block_group removal
> > btrfs: discard one region at a time in async discard
> > btrfs: add removal calls for sysfs debug/
> > btrfs: make UUID/debug have its own kobject
> > btrfs: add discard sysfs directory
> > btrfs: track discardable extents for async discard
> > btrfs: keep track of discardable_bytes
> > btrfs: calculate discard delay based on number of extents
> > btrfs: add bps discard rate limit
> > btrfs: limit max discard size for async discard
> > btrfs: make max async discard size tunable
> > btrfs: have multiple discard lists
> > btrfs: only keep track of data extents for async discard
> > btrfs: keep track of discard reuse stats
> > btrfs: add async discard header
> > btrfs: increase the metadata allowance for the free_space_cache
> > btrfs: make smaller extents more likely to go into bitmaps
>
> Patches 1-12 merged to a temporary misc-next but I haven't pushed it as
> misc-next yet (it's misc-next-with-discard-v6 in my github repo). There
> are some comments to patch 13 and up so please send them either as
> replies or as a shorter series. Thanks.
Great! Thanks for taking another pass at it all. Would you prefer a pull
request or just another series? I'll throw on top a couple patches to
hopefully address the -1 (I'm still not fully sure how it can happen).
Thanks,
Dennis
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH v6 00/22] btrfs: async discard support
2019-12-30 18:49 ` Dennis Zhou
@ 2020-01-02 13:22 ` David Sterba
0 siblings, 0 replies; 61+ messages in thread
From: David Sterba @ 2020-01-02 13:22 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, David Sterba, Chris Mason, Josef Bacik,
Omar Sandoval, kernel-team, linux-btrfs
On Mon, Dec 30, 2019 at 01:49:30PM -0500, Dennis Zhou wrote:
> On Mon, Dec 30, 2019 at 07:13:18PM +0100, David Sterba wrote:
> > On Fri, Dec 13, 2019 at 04:22:09PM -0800, Dennis Zhou wrote:
> > Patches 1-12 merged to a temporary misc-next but I haven't pushed it as
> > misc-next yet (it's misc-next-with-discard-v6 in my github repo). There
> > are some comments to patch 13 and up so please send them either as
> > replies or as a shorter series. Thanks.
>
> Great! Thanks for taking another pass at it all. Would you prefer a pull
> request or just another series? I'll throw on top a couple patches to
> hopefully address the -1 (I'm still not fully sure how it can happen).
Send the patch series please, you can add a link to git repo/branch but
that's not necessary.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 13/22] btrfs: calculate discard delay based on number of extents
2019-12-30 16:50 ` David Sterba
@ 2020-01-02 16:45 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2020-01-02 16:45 UTC (permalink / raw)
To: David Sterba
Cc: Dennis Zhou, David Sterba, Chris Mason, Josef Bacik,
Omar Sandoval, kernel-team, linux-btrfs
On Mon, Dec 30, 2019 at 05:50:43PM +0100, David Sterba wrote:
> On Fri, Dec 13, 2019 at 04:22:22PM -0800, Dennis Zhou wrote:
> > Use the number of discardable extents to help guide our discard delay
> > interval. This value is reevaluated every transaction commit.
>
> This is too short compared to the added code, please enhance it eg. by
> briefly describing what values are tracked and how are they related
> together.
>
Done.
> > Signed-off-by: Dennis Zhou <dennis@kernel.org>
> > Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> > ---
> > fs/btrfs/ctree.h | 2 ++
> > fs/btrfs/discard.c | 53 ++++++++++++++++++++++++++++++++++++++----
> > fs/btrfs/discard.h | 1 +
> > fs/btrfs/extent-tree.c | 4 +++-
> > fs/btrfs/sysfs.c | 31 ++++++++++++++++++++++++
> > 5 files changed, 86 insertions(+), 5 deletions(-)
> >
> > diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> > index 18df0e40a282..98979566e281 100644
> > --- a/fs/btrfs/ctree.h
> > +++ b/fs/btrfs/ctree.h
> > @@ -469,6 +469,8 @@ struct btrfs_discard_ctl {
> > struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
> > atomic_t discardable_extents;
> > atomic64_t discardable_bytes;
>
> There are some type mismatches for the two:
>
> > + u32 delay;
> > + u32 iops_limit;
>
> unsigned for iops
I switched iops_limit to unsigned and delay to unsigned long.
>
> > };
> >
> > /* delayed seq elem */
> > diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
> > index e8c2623617fd..160713e04d0c 100644
> > --- a/fs/btrfs/discard.c
> > +++ b/fs/btrfs/discard.c
> > @@ -15,6 +15,11 @@
> > #define BTRFS_DISCARD_DELAY (120ULL * NSEC_PER_SEC)
> > #define BTRFS_DISCARD_UNUSED_DELAY (10ULL * NSEC_PER_SEC)
> >
> > +/* Target completion latency of discarding all discardable extents. */
> > +#define BTRFS_DISCARD_TARGET_MSEC (6 * 60 * 60UL * MSEC_PER_SEC)
> > +#define BTRFS_DISCARD_MAX_DELAY (10000UL)
> > +#define BTRFS_DISCARD_MAX_IOPS (10UL)
>
> All unsigned longs
>
I've switched BTRFS_DISCARD_MAX_IOPS to 10U and lowered the max delay to
be 1s from our previous conversation.
> > +
> > static struct list_head *get_discard_list(struct btrfs_discard_ctl *discard_ctl,
> > struct btrfs_block_group *block_group)
> > {
> > @@ -239,11 +244,18 @@ void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
> >
> > block_group = find_next_block_group(discard_ctl, now);
> > if (block_group) {
> > - u64 delay = 0;
> > + u32 delay = discard_ctl->delay;
> > +
> > + /*
> > + * This timeout is to hopefully prevent immediate discarding
> > + * in a recently allocated block group.
> > + */
> > + if (now < block_group->discard_eligible_time) {
> > + u64 bg_timeout = (block_group->discard_eligible_time -
> > + now);
> >
> > - if (now < block_group->discard_eligible_time)
> > - delay = nsecs_to_jiffies(
> > - block_group->discard_eligible_time - now);
> > + delay = max_t(u64, delay, nsecs_to_jiffies(bg_timeout));
>
> delay is u32, nsecs_to_jiffies returns unsigned long, so delay should be
> also unsigned long everywhere.
>
> > + }
> >
> > mod_delayed_work(discard_ctl->discard_workers,
> > &discard_ctl->work,
> > @@ -348,6 +360,37 @@ bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl)
> > test_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags));
> > }
> >
> > +/**
> > + * btrfs_discard_calc_delay - recalculate the base delay
> > + * @discard_ctl: discard control
> > + *
> > + * Recalculate the base delay which is based off the total number of
> > + * discardable_extents. Clamp this with the iops_limit and
> > + * BTRFS_DISCARD_MAX_DELAY.
> > + */
> > +void btrfs_discard_calc_delay(struct btrfs_discard_ctl *discard_ctl)
> > +{
> > + s32 discardable_extents =
> > + atomic_read(&discard_ctl->discardable_extents);
> > + s32 iops_limit;
>
> signed for iops
>
> > + unsigned long delay;
> > +
> > + if (!discardable_extents)
> > + return;
> > +
> > + spin_lock(&discard_ctl->lock);
> > +
> > + iops_limit = READ_ONCE(discard_ctl->iops_limit);
>
> unsigned -> signed
>
> > + if (iops_limit)
> > + iops_limit = MSEC_PER_SEC / iops_limit;
> > +
> > + delay = BTRFS_DISCARD_TARGET_MSEC / discardable_extents;
> > + delay = clamp_t(s32, delay, iops_limit, BTRFS_DISCARD_MAX_DELAY);
> > + discard_ctl->delay = msecs_to_jiffies(delay);
>
> unsigned long -> u32
>
> > +
> > + spin_unlock(&discard_ctl->lock);
> > +}
> > +
> > /**
> > * btrfs_discard_update_discardable - propagate discard counters
> > * @block_group: block_group of interest
> > @@ -476,6 +519,8 @@ void btrfs_discard_init(struct btrfs_fs_info *fs_info)
> >
> > atomic_set(&discard_ctl->discardable_extents, 0);
> > atomic64_set(&discard_ctl->discardable_bytes, 0);
> > + discard_ctl->delay = BTRFS_DISCARD_MAX_DELAY;
> > + discard_ctl->iops_limit = BTRFS_DISCARD_MAX_IOPS;
> > }
> >
> > void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
> > diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
> > index 7e3680dd82ce..3ed6855e24da 100644
> > --- a/fs/btrfs/discard.h
> > +++ b/fs/btrfs/discard.h
> > @@ -17,6 +17,7 @@ void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
> > bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl);
> >
> > /* Update operations. */
> > +void btrfs_discard_calc_delay(struct btrfs_discard_ctl *discard_ctl);
> > void btrfs_discard_update_discardable(struct btrfs_block_group *block_group,
> > struct btrfs_free_space_ctl *ctl);
> >
> > diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
> > index 2c12366cfde5..0163fdd59f8f 100644
> > --- a/fs/btrfs/extent-tree.c
> > +++ b/fs/btrfs/extent-tree.c
> > @@ -2935,8 +2935,10 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
> > cond_resched();
> > }
> >
> > - if (btrfs_test_opt(fs_info, DISCARD_ASYNC))
> > + if (btrfs_test_opt(fs_info, DISCARD_ASYNC)) {
> > + btrfs_discard_calc_delay(&fs_info->discard_ctl);
> > btrfs_discard_schedule_work(&fs_info->discard_ctl, true);
> > + }
> >
> > /*
> > * Transaction is finished. We don't need the lock anymore. We
> > diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c
> > index e799d4488d72..79139b4b4f0a 100644
> > --- a/fs/btrfs/sysfs.c
> > +++ b/fs/btrfs/sysfs.c
> > @@ -344,6 +344,36 @@ static const struct attribute_group btrfs_static_feature_attr_group = {
> > */
> > #define discard_to_fs_info(_kobj) to_fs_info((_kobj)->parent->parent)
> >
> > +static ssize_t btrfs_discard_iops_limit_show(struct kobject *kobj,
> > + struct kobj_attribute *a,
> > + char *buf)
> > +{
> > + struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
> > +
> > + return snprintf(buf, PAGE_SIZE, "%u\n",
> > + READ_ONCE(fs_info->discard_ctl.iops_limit));
> > +}
> > +
> > +static ssize_t btrfs_discard_iops_limit_store(struct kobject *kobj,
> > + struct kobj_attribute *a,
> > + const char *buf, size_t len)
> > +{
> > + struct btrfs_fs_info *fs_info = discard_to_fs_info(kobj);
> > + struct btrfs_discard_ctl *discard_ctl = &fs_info->discard_ctl;
> > + u32 iops_limit;
> > + int ret;
> > +
> > + ret = kstrtou32(buf, 10, &iops_limit);
> > + if (ret)
> > + return -EINVAL;
> > +
> > + WRITE_ONCE(discard_ctl->iops_limit, iops_limit);
> > +
> > + return len;
> > +}
> > +BTRFS_ATTR_RW(discard, iops_limit, btrfs_discard_iops_limit_show,
> > + btrfs_discard_iops_limit_store);
> > +
> > static ssize_t btrfs_discardable_extents_show(struct kobject *kobj,
> > struct kobj_attribute *a,
> > char *buf)
> > @@ -367,6 +397,7 @@ static ssize_t btrfs_discardable_bytes_show(struct kobject *kobj,
> > BTRFS_ATTR(discard, discardable_bytes, btrfs_discardable_bytes_show);
> >
> > static const struct attribute *discard_debug_attrs[] = {
> > + BTRFS_ATTR_PTR(discard, iops_limit),
> > BTRFS_ATTR_PTR(discard, discardable_extents),
> > BTRFS_ATTR_PTR(discard, discardable_bytes),
> > NULL,
> > --
> > 2.17.1
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 14/22] btrfs: add bps discard rate limit
2019-12-30 17:58 ` David Sterba
@ 2020-01-02 16:46 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2020-01-02 16:46 UTC (permalink / raw)
To: David Sterba
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Mon, Dec 30, 2019 at 06:58:47PM +0100, David Sterba wrote:
> On Fri, Dec 13, 2019 at 04:22:23PM -0800, Dennis Zhou wrote:
> > Provide an ability to rate limit based on mbps in addition to the iops
> > delay calculated from number of discardable extents.
> >
> > Signed-off-by: Dennis Zhou <dennis@kernel.org>
> > ---
> > fs/btrfs/ctree.h | 2 ++
> > fs/btrfs/discard.c | 18 ++++++++++++++++++
> > fs/btrfs/sysfs.c | 31 +++++++++++++++++++++++++++++++
> > 3 files changed, 51 insertions(+)
> >
> > diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> > index 98979566e281..1b2dae5962de 100644
> > --- a/fs/btrfs/ctree.h
> > +++ b/fs/btrfs/ctree.h
> > @@ -467,10 +467,12 @@ struct btrfs_discard_ctl {
> > spinlock_t lock;
> > struct btrfs_block_group *block_group;
> > struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
> > + u64 prev_discard;
> > atomic_t discardable_extents;
> > atomic64_t discardable_bytes;
> > u32 delay;
> > u32 iops_limit;
> > + u64 bps_limit;
>
> I think this could be u32, which makes it 4G per second. Byte
> granularity is IMO not necessary so it could be KiB/sec instead and it
> would be future safe with maximum limit of 4TiB/sec.
Sounds good, done.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 15/22] btrfs: limit max discard size for async discard
2019-12-30 18:08 ` David Sterba
@ 2020-01-02 16:48 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2020-01-02 16:48 UTC (permalink / raw)
To: dsterba, David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Mon, Dec 30, 2019 at 07:08:03PM +0100, David Sterba wrote:
> On Fri, Dec 13, 2019 at 04:22:24PM -0800, Dennis Zhou wrote:
> > Throttle the maximum size of a discard so that we can provide an upper
> > bound for the rate of async discard. While the block layer is able to
> > split discards into the appropriate sized discards, we want to be able
> > to account more accurately the rate at which we are consuming ncq slots
> > as well as limit the upper bound of work for a discard.
> >
> > Signed-off-by: Dennis Zhou <dennis@kernel.org>
> > Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> > ---
> > fs/btrfs/discard.h | 5 ++++
> > fs/btrfs/free-space-cache.c | 48 +++++++++++++++++++++++++++----------
> > 2 files changed, 41 insertions(+), 12 deletions(-)
> >
> > diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
> > index 3ed6855e24da..cb6ef0ab879d 100644
> > --- a/fs/btrfs/discard.h
> > +++ b/fs/btrfs/discard.h
> > @@ -3,10 +3,15 @@
> > #ifndef BTRFS_DISCARD_H
> > #define BTRFS_DISCARD_H
> >
> > +#include <linux/sizes.h>
> > +
> > struct btrfs_fs_info;
> > struct btrfs_discard_ctl;
> > struct btrfs_block_group;
> >
> > +/* Discard size limits. */
> > +#define BTRFS_ASYNC_DISCARD_MAX_SIZE (SZ_64M)
> > +
> > /* Work operations. */
> > void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
> > struct btrfs_block_group *block_group);
> > diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
> > index 57df34480b93..0dbcea6c59f9 100644
> > --- a/fs/btrfs/free-space-cache.c
> > +++ b/fs/btrfs/free-space-cache.c
> > @@ -3466,19 +3466,40 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
> > if (entry->offset >= end)
> > goto out_unlock;
> >
> > - extent_start = entry->offset;
> > - extent_bytes = entry->bytes;
> > - extent_trim_state = entry->trim_state;
> > - start = max(start, extent_start);
> > - bytes = min(extent_start + extent_bytes, end) - start;
> > - if (bytes < minlen) {
> > - spin_unlock(&ctl->tree_lock);
> > - mutex_unlock(&ctl->cache_writeout_mutex);
> > - goto next;
> > - }
> > + if (async) {
> > + start = extent_start = entry->offset;
> > + bytes = extent_bytes = entry->bytes;
> > + extent_trim_state = entry->trim_state;
> > + if (bytes < minlen) {
> > + spin_unlock(&ctl->tree_lock);
> > + mutex_unlock(&ctl->cache_writeout_mutex);
> > + goto next;
> > + }
> > + unlink_free_space(ctl, entry);
> > + if (bytes > BTRFS_ASYNC_DISCARD_MAX_SIZE) {
> > + bytes = extent_bytes =
> > + BTRFS_ASYNC_DISCARD_MAX_SIZE;
> > + entry->offset += BTRFS_ASYNC_DISCARD_MAX_SIZE;
> > + entry->bytes -= BTRFS_ASYNC_DISCARD_MAX_SIZE;
> > + link_free_space(ctl, entry);
> > + } else {
> > + kmem_cache_free(btrfs_free_space_cachep, entry);
> > + }
> > + } else {
>
> > + extent_start = entry->offset;
> > + extent_bytes = entry->bytes;
> > + extent_trim_state = entry->trim_state;
>
> This is common initialization for both async and sync cases so it could
> be merged to a common block.
>
I removed the chain initialization and moved the common out.
> > + start = max(start, extent_start);
> > + bytes = min(extent_start + extent_bytes, end) - start;
> > + if (bytes < minlen) {
> > + spin_unlock(&ctl->tree_lock);
> > + mutex_unlock(&ctl->cache_writeout_mutex);
> > + goto next;
> > + }
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 16/22] btrfs: make max async discard size tunable
2019-12-30 18:05 ` David Sterba
@ 2020-01-02 16:50 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2020-01-02 16:50 UTC (permalink / raw)
To: dsterba, David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Mon, Dec 30, 2019 at 07:05:03PM +0100, David Sterba wrote:
> On Fri, Dec 13, 2019 at 04:22:25PM -0800, Dennis Zhou wrote:
> > Expose max_discard_size as a tunable via sysfs.
> >
> > Signed-off-by: Dennis Zhou <dennis@kernel.org>
> > ---
> > fs/btrfs/ctree.h | 1 +
> > fs/btrfs/discard.c | 1 +
> > fs/btrfs/free-space-cache.c | 19 ++++++++++++-------
> > fs/btrfs/sysfs.c | 31 +++++++++++++++++++++++++++++++
> > 4 files changed, 45 insertions(+), 7 deletions(-)
> >
> > diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> > index 1b2dae5962de..bf93ddbc773f 100644
> > --- a/fs/btrfs/ctree.h
> > +++ b/fs/btrfs/ctree.h
> > @@ -470,6 +470,7 @@ struct btrfs_discard_ctl {
> > u64 prev_discard;
> > atomic_t discardable_extents;
> > atomic64_t discardable_bytes;
> > + u64 max_discard_size;
> > u32 delay;
> > u32 iops_limit;
> > u64 bps_limit;
> > diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
> > index 085f36808e7f..dd5143f0283f 100644
> > --- a/fs/btrfs/discard.c
> > +++ b/fs/btrfs/discard.c
> > @@ -536,6 +536,7 @@ void btrfs_discard_init(struct btrfs_fs_info *fs_info)
> > discard_ctl->prev_discard = 0;
> > atomic_set(&discard_ctl->discardable_extents, 0);
> > atomic64_set(&discard_ctl->discardable_bytes, 0);
> > + discard_ctl->max_discard_size = BTRFS_ASYNC_DISCARD_MAX_SIZE;
>
> BTRFS_ASYNC_DISCARD_MAX_SIZE won't be the upper limit anymore but a
> default so it should be named as such.
Renamed to BTRFS_ASYNC_DISCARD_MAX_SIZE_DFL.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 18/22] btrfs: only keep track of data extents for async discard
2019-12-30 17:39 ` David Sterba
@ 2020-01-02 16:55 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2020-01-02 16:55 UTC (permalink / raw)
To: dsterba, David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Mon, Dec 30, 2019 at 06:39:54PM +0100, David Sterba wrote:
> On Fri, Dec 13, 2019 at 04:22:27PM -0800, Dennis Zhou wrote:
> > As mentioned earlier, discarding data can be done either by issuing an
> > explicit discard or implicitly by reusing the LBA. Metadata chunks see
> > much more frequent reuse due to well it being metadata. So instead of
> > explicitly discarding metadata blocks, just leave them be and let the
> > latter implicit discarding be done for them.
> >
> > Signed-off-by: Dennis Zhou <dennis@kernel.org>
> > Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> > ---
> > fs/btrfs/block-group.h | 6 ++++++
> > fs/btrfs/discard.c | 11 +++++++++--
> > 2 files changed, 15 insertions(+), 2 deletions(-)
> >
> > diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h
> > index 601e1d217e22..ee8441439a56 100644
> > --- a/fs/btrfs/block-group.h
> > +++ b/fs/btrfs/block-group.h
> > @@ -182,6 +182,12 @@ static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
> > return (block_group->start + block_group->length);
> > }
> >
> > +static inline bool btrfs_is_block_group_data(
> > + struct btrfs_block_group *block_group)
> > +{
> > + return (block_group->flags & BTRFS_BLOCK_GROUP_DATA);
>
> What happens for mixed data and metadata block groups? As this is a
> special mode that will likely lead to fragmented block groups I think
> that async discard won't be able to help much.
>
> I'd suggest to make the test explicit only for the separate block group
> types and comment that mixed bg's are not supported.
Yeah it probably wouldn't do the system much good. I've renamed the test
to btrfs_is_block_group_data_only() and added a comment in the async
discard header patch to state that mixed block_groups aren't supported.
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 19/22] btrfs: keep track of discard reuse stats
2019-12-30 17:33 ` David Sterba
@ 2020-01-02 16:57 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2020-01-02 16:57 UTC (permalink / raw)
To: dsterba, David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Mon, Dec 30, 2019 at 06:33:52PM +0100, David Sterba wrote:
> On Fri, Dec 13, 2019 at 04:22:28PM -0800, Dennis Zhou wrote:
> > Keep track of how much we are discarding and how often we are reusing
> > with async discard.
> >
> > Signed-off-by: Dennis Zhou <dennis@kernel.org>
> > Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> > ---
> > fs/btrfs/ctree.h | 3 +++
> > fs/btrfs/discard.c | 7 +++++++
> > fs/btrfs/free-space-cache.c | 14 ++++++++++++++
> > fs/btrfs/sysfs.c | 36 ++++++++++++++++++++++++++++++++++++
> > 4 files changed, 60 insertions(+)
> >
> > diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
> > index dddf51e27bed..edfbe6060e8d 100644
> > --- a/fs/btrfs/ctree.h
> > +++ b/fs/btrfs/ctree.h
> > @@ -474,6 +474,9 @@ struct btrfs_discard_ctl {
> > u32 delay;
> > u32 iops_limit;
> > u64 bps_limit;
> > + u64 discard_extent_bytes;
> > + u64 discard_bitmap_bytes;
> > + atomic64_t discard_bytes_saved;
> > };
> >
> > /* delayed seq elem */
> > diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
> > index 55ad357e65f3..fe73814526ef 100644
> > --- a/fs/btrfs/discard.c
> > +++ b/fs/btrfs/discard.c
> > @@ -419,11 +419,15 @@ static void btrfs_discard_workfn(struct work_struct *work)
> > block_group->discard_cursor,
> > btrfs_block_group_end(block_group),
> > minlen, maxlen, true);
> > + WRITE_ONCE(discard_ctl->discard_bitmap_bytes,
> > + discard_ctl->discard_bitmap_bytes + trimmed);
>
> The same argument is used for read and write, this does not seem to be a
> clear usage pattern for WRITE_ONCE, is there are cleaner way to do that?
>
Yeah that doesn't really make any sense. I think we are okay here anyway
as btrfs_discard_workfn() provides synchronization by being a single
work item. So, I've removed the WRITE_ONCE.
> > } else {
> > btrfs_trim_block_group_extents(block_group, &trimmed,
> > block_group->discard_cursor,
> > btrfs_block_group_end(block_group),
> > minlen, true);
> > + WRITE_ONCE(discard_ctl->discard_extent_bytes,
> > + discard_ctl->discard_extent_bytes + trimmed);
^ permalink raw reply [flat|nested] 61+ messages in thread
* [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-12-09 19:45 [PATCH v5 " Dennis Zhou
@ 2019-12-09 19:45 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-12-09 19:45 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
When discard is enabled, everytime a pinned extent is released back to
the block_group's free space cache, a discard is issued for the extent.
This is an overeager approach when it comes to discarding and helping
the SSD maintain enough free space to prevent severe garbage collection
situations.
This adds the beginning of async discard. Instead of issuing a discard
prior to returning it to the free space, it is just marked as untrimmed.
The block_group is then added to a LRU which then feeds into a workqueue
to issue discards at a much slower rate. Full discarding of unused block
groups is still done and will be address in a future patch in this
series.
For now, we don't persist the discard state of extents and bitmaps.
Therefore, our failure recovery mode will be to consider extents
untrimmed. This lets us handle failure and unmounting as one in the
same.
On a number of Facebook webservers, I collected data every minute
accounting the time we spent in btrfs_finish_extent_commit() (col. 1)
and in btrfs_commit_transaction() (col. 2). btrfs_finish_extent_commit()
is where we discard extents synchronously before returning them to the
free space cache.
discard=sync:
p99 total per minute p99 total per minute
Drive | extent_commit() (ms) | commit_trans() (ms)
---------------------------------------------------------------
Drive A | 434 | 1170
Drive B | 880 | 2330
Drive C | 2943 | 3920
Drive D | 4763 | 5701
discard=async:
p99 total per minute p99 total per minute
Drive | extent_commit() (ms) | commit_trans() (ms)
--------------------------------------------------------------
Drive A | 134 | 956
Drive B | 64 | 1972
Drive C | 59 | 1032
Drive D | 62 | 1200
While it's not great that the stats are cumulative over 1m, all of these
servers are running the same workload and and the delta between the two
are substantial. We are spending significantly less time in
btrfs_finish_extent_commit() which is responsible for discarding.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/Makefile | 2 +-
fs/btrfs/block-group.c | 37 ++++-
fs/btrfs/block-group.h | 9 ++
fs/btrfs/ctree.h | 21 +++
fs/btrfs/discard.c | 278 ++++++++++++++++++++++++++++++++++++
fs/btrfs/discard.h | 23 +++
fs/btrfs/disk-io.c | 15 +-
fs/btrfs/extent-tree.c | 4 +
fs/btrfs/free-space-cache.c | 54 ++++++-
fs/btrfs/free-space-cache.h | 2 +
fs/btrfs/super.c | 35 ++++-
fs/btrfs/volumes.c | 7 +
12 files changed, 474 insertions(+), 13 deletions(-)
create mode 100644 fs/btrfs/discard.c
create mode 100644 fs/btrfs/discard.h
diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
index 82200dbca5ac..9a0ff3384381 100644
--- a/fs/btrfs/Makefile
+++ b/fs/btrfs/Makefile
@@ -11,7 +11,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
uuid-tree.o props.o free-space-tree.o tree-checker.o space-info.o \
- block-rsv.o delalloc-space.o block-group.o
+ block-rsv.o delalloc-space.o block-group.o discard.o
btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c
index be1938dc94fd..c1b1b59343bd 100644
--- a/fs/btrfs/block-group.c
+++ b/fs/btrfs/block-group.c
@@ -14,6 +14,7 @@
#include "sysfs.h"
#include "tree-log.h"
#include "delalloc-space.h"
+#include "discard.h"
/*
* Return target flags in extended format or 0 if restripe for this chunk_type
@@ -131,6 +132,15 @@ void btrfs_put_block_group(struct btrfs_block_group *cache)
WARN_ON(cache->pinned > 0);
WARN_ON(cache->reserved > 0);
+ /*
+ * A block_group shouldn't be on the discard_list anymore.
+ * Remove the block_group from the discard_list to prevent us
+ * from causing a panic due to NPE.
+ */
+ if (WARN_ON(!list_empty(&cache->discard_list)))
+ btrfs_discard_cancel_work(&cache->fs_info->discard_ctl,
+ cache);
+
/*
* If not empty, someone is still holding mutex of
* full_stripe_lock, which can only be released by caller.
@@ -466,8 +476,8 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 end
} else if (extent_start > start && extent_start < end) {
size = extent_start - start;
total_added += size;
- ret = btrfs_add_free_space(block_group, start,
- size);
+ ret = btrfs_add_free_space_async_trimmed(block_group,
+ start, size);
BUG_ON(ret); /* -ENOMEM or logic error */
start = extent_end + 1;
} else {
@@ -478,7 +488,8 @@ u64 add_new_free_space(struct btrfs_block_group *block_group, u64 start, u64 end
if (start < end) {
size = end - start;
total_added += size;
- ret = btrfs_add_free_space(block_group, start, size);
+ ret = btrfs_add_free_space_async_trimmed(block_group, start,
+ size);
BUG_ON(ret); /* -ENOMEM or logic error */
}
@@ -1258,6 +1269,8 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
}
spin_unlock(&fs_info->unused_bgs_lock);
+ btrfs_discard_cancel_work(&fs_info->discard_ctl, block_group);
+
mutex_lock(&fs_info->delete_unused_bgs_mutex);
/* Don't want to race with allocators so take the groups_sem */
@@ -1333,6 +1346,23 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
}
mutex_unlock(&fs_info->unused_bg_unpin_mutex);
+ /*
+ * At this point, the block_group is read only and should fail
+ * new allocations. However, btrfs_finish_extent_commit() can
+ * cause this block_group to be placed back on the discard
+ * lists because now the block_group isn't fully discarded.
+ * Bail here and try again later after discarding everything.
+ */
+ spin_lock(&fs_info->discard_ctl.lock);
+ if (!list_empty(&block_group->discard_list)) {
+ spin_unlock(&fs_info->discard_ctl.lock);
+ btrfs_dec_block_group_ro(block_group);
+ btrfs_discard_queue_work(&fs_info->discard_ctl,
+ block_group);
+ goto end_trans;
+ }
+ spin_unlock(&fs_info->discard_ctl.lock);
+
/* Reset pinned so btrfs_put_block_group doesn't complain */
spin_lock(&space_info->lock);
spin_lock(&block_group->lock);
@@ -1603,6 +1633,7 @@ static struct btrfs_block_group *btrfs_create_block_group_cache(
INIT_LIST_HEAD(&cache->cluster_list);
INIT_LIST_HEAD(&cache->bg_list);
INIT_LIST_HEAD(&cache->ro_list);
+ INIT_LIST_HEAD(&cache->discard_list);
INIT_LIST_HEAD(&cache->dirty_list);
INIT_LIST_HEAD(&cache->io_list);
btrfs_init_free_space_ctl(cache);
diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h
index 9b409676c4b2..884defd61dcd 100644
--- a/fs/btrfs/block-group.h
+++ b/fs/btrfs/block-group.h
@@ -116,7 +116,11 @@ struct btrfs_block_group {
/* For read-only block groups */
struct list_head ro_list;
+ /* For discard operations */
atomic_t trimming;
+ struct list_head discard_list;
+ int discard_index;
+ u64 discard_eligible_time;
/* For dirty block groups */
struct list_head dirty_list;
@@ -158,6 +162,11 @@ struct btrfs_block_group {
struct btrfs_full_stripe_locks_tree full_stripe_locks_root;
};
+static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
+{
+ return (block_group->start + block_group->length);
+}
+
#ifdef CONFIG_BTRFS_DEBUG
static inline int btrfs_should_fragment_free_space(
struct btrfs_block_group *block_group)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 51a303441802..d67bbbb07925 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -440,6 +440,21 @@ struct btrfs_full_stripe_locks_tree {
struct mutex lock;
};
+/* Discard control. */
+/*
+ * Async discard uses multiple lists to differentiate the discard filter
+ * parameters.
+ */
+#define BTRFS_NR_DISCARD_LISTS 1
+
+struct btrfs_discard_ctl {
+ struct workqueue_struct *discard_workers;
+ struct delayed_work work;
+ spinlock_t lock;
+ struct btrfs_block_group *block_group;
+ struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
+};
+
/* delayed seq elem */
struct seq_list {
struct list_head list;
@@ -526,6 +541,9 @@ enum {
* so we don't need to offload checksums to workqueues.
*/
BTRFS_FS_CSUM_IMPL_FAST,
+
+ /* Indicate that the discard workqueue can service discards. */
+ BTRFS_FS_DISCARD_RUNNING,
};
struct btrfs_fs_info {
@@ -816,6 +834,8 @@ struct btrfs_fs_info {
struct btrfs_workqueue *scrub_wr_completion_workers;
struct btrfs_workqueue *scrub_parity_workers;
+ struct btrfs_discard_ctl discard_ctl;
+
#ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY
u32 check_integrity_print_mask;
#endif
@@ -1189,6 +1209,7 @@ static inline u32 BTRFS_MAX_XATTR_SIZE(const struct btrfs_fs_info *info)
#define BTRFS_MOUNT_FREE_SPACE_TREE (1 << 26)
#define BTRFS_MOUNT_NOLOGREPLAY (1 << 27)
#define BTRFS_MOUNT_REF_VERIFY (1 << 28)
+#define BTRFS_MOUNT_DISCARD_ASYNC (1 << 29)
#define BTRFS_DEFAULT_COMMIT_INTERVAL (30)
#define BTRFS_DEFAULT_MAX_INLINE (2048)
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
new file mode 100644
index 000000000000..692d64025802
--- /dev/null
+++ b/fs/btrfs/discard.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/jiffies.h>
+#include <linux/kernel.h>
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/sizes.h>
+#include <linux/workqueue.h>
+#include "ctree.h"
+#include "block-group.h"
+#include "discard.h"
+#include "free-space-cache.h"
+
+/* This is an initial delay to give some chance for lba reuse. */
+#define BTRFS_DISCARD_DELAY (120ULL * NSEC_PER_SEC)
+
+static struct list_head *get_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ return &discard_ctl->discard_list[block_group->discard_index];
+}
+
+static void add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ spin_lock(&discard_ctl->lock);
+
+ if (!btrfs_run_discard_work(discard_ctl)) {
+ spin_unlock(&discard_ctl->lock);
+ return;
+ }
+
+ if (list_empty(&block_group->discard_list))
+ block_group->discard_eligible_time = (ktime_get_ns() +
+ BTRFS_DISCARD_DELAY);
+
+ list_move_tail(&block_group->discard_list,
+ get_discard_list(discard_ctl, block_group));
+
+ spin_unlock(&discard_ctl->lock);
+}
+
+static bool remove_from_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ bool running = false;
+
+ spin_lock(&discard_ctl->lock);
+
+ if (block_group == discard_ctl->block_group) {
+ running = true;
+ discard_ctl->block_group = NULL;
+ }
+
+ block_group->discard_eligible_time = 0;
+ list_del_init(&block_group->discard_list);
+
+ spin_unlock(&discard_ctl->lock);
+
+ return running;
+}
+
+/**
+ * find_next_block_group - find block_group that's up next for discarding
+ * @discard_ctl: discard control
+ * @now: current time
+ *
+ * Iterate over the discard lists to find the next block_group up for
+ * discarding checking the discard_eligible_time of block_group.
+ */
+static struct btrfs_block_group *find_next_block_group(
+ struct btrfs_discard_ctl *discard_ctl,
+ u64 now)
+{
+ struct btrfs_block_group *ret_block_group = NULL, *block_group;
+ int i;
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++) {
+ struct list_head *discard_list = &discard_ctl->discard_list[i];
+
+ if (!list_empty(discard_list)) {
+ block_group = list_first_entry(discard_list,
+ struct btrfs_block_group,
+ discard_list);
+
+ if (!ret_block_group)
+ ret_block_group = block_group;
+
+ if (ret_block_group->discard_eligible_time < now)
+ break;
+
+ if (ret_block_group->discard_eligible_time >
+ block_group->discard_eligible_time)
+ ret_block_group = block_group;
+ }
+ }
+
+ return ret_block_group;
+}
+
+/**
+ * peek_discard_list - wrap find_next_block_group()
+ * @discard_ctl: discard control
+ *
+ * This wraps find_next_block_group() and sets the block_group to be in use.
+ */
+static struct btrfs_block_group *peek_discard_list(
+ struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_block_group *block_group;
+ u64 now = ktime_get_ns();
+
+ spin_lock(&discard_ctl->lock);
+
+ block_group = find_next_block_group(discard_ctl, now);
+
+ if (block_group && now < block_group->discard_eligible_time)
+ block_group = NULL;
+
+ discard_ctl->block_group = block_group;
+
+ spin_unlock(&discard_ctl->lock);
+
+ return block_group;
+}
+
+/**
+ * btrfs_discard_cancel_work - remove a block_group from the discard lists
+ * @discard_ctl: discard control
+ * @block_group: block_group of interest
+ *
+ * This removes @block_group from the discard lists. If necessary, it waits on
+ * the current work and then reschedules the delayed work.
+ */
+void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ if (remove_from_discard_list(discard_ctl, block_group)) {
+ cancel_delayed_work_sync(&discard_ctl->work);
+ btrfs_discard_schedule_work(discard_ctl, true);
+ }
+}
+
+/**
+ * btrfs_discard_queue_work - handles queuing the block_groups
+ * @discard_ctl: discard control
+ * @block_group: block_group of interest
+ *
+ * This maintains the LRU order of the discard lists.
+ */
+void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ if (!block_group ||
+ !btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
+ return;
+
+ add_to_discard_list(discard_ctl, block_group);
+
+ if (!delayed_work_pending(&discard_ctl->work))
+ btrfs_discard_schedule_work(discard_ctl, false);
+}
+
+/**
+ * btrfs_discard_schedule_work - responsible for scheduling the discard work
+ * @discard_ctl: discard control
+ * @override: override the current timer
+ *
+ * Discards are issued by a delayed workqueue item. @override is used to
+ * update the current delay as the baseline delay interview is reevaluated
+ * on transaction commit. This is also maxed with any other rate limit.
+ */
+void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
+ bool override)
+{
+ struct btrfs_block_group *block_group;
+ u64 now = ktime_get_ns();
+
+ spin_lock(&discard_ctl->lock);
+
+ if (!btrfs_run_discard_work(discard_ctl))
+ goto out;
+
+ if (!override && delayed_work_pending(&discard_ctl->work))
+ goto out;
+
+ block_group = find_next_block_group(discard_ctl, now);
+ if (block_group) {
+ u64 delay = 0;
+
+ if (now < block_group->discard_eligible_time)
+ delay = nsecs_to_jiffies(
+ block_group->discard_eligible_time - now);
+
+ mod_delayed_work(discard_ctl->discard_workers,
+ &discard_ctl->work,
+ delay);
+ }
+
+out:
+ spin_unlock(&discard_ctl->lock);
+}
+
+/**
+ * btrfs_discard_workfn - discard work function
+ * @work: work
+ *
+ * This finds the next block_group to start discarding and then discards it.
+ */
+static void btrfs_discard_workfn(struct work_struct *work)
+{
+ struct btrfs_discard_ctl *discard_ctl;
+ struct btrfs_block_group *block_group;
+ u64 trimmed = 0;
+
+ discard_ctl = container_of(work, struct btrfs_discard_ctl, work.work);
+
+ block_group = peek_discard_list(discard_ctl);
+ if (!block_group || !btrfs_run_discard_work(discard_ctl))
+ return;
+
+ btrfs_trim_block_group(block_group, &trimmed, block_group->start,
+ btrfs_block_group_end(block_group), 0);
+
+ remove_from_discard_list(discard_ctl, block_group);
+
+ btrfs_discard_schedule_work(discard_ctl, false);
+}
+
+/**
+ * btrfs_run_discard_work - determines if async discard should be running
+ * @discard_ctl: discard control
+ *
+ * Checks if the file system is writeable and BTRFS_FS_DISCARD_RUNNING is set.
+ */
+bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_fs_info *fs_info = container_of(discard_ctl,
+ struct btrfs_fs_info,
+ discard_ctl);
+
+ return (!(fs_info->sb->s_flags & SB_RDONLY) &&
+ test_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags));
+}
+
+void btrfs_discard_resume(struct btrfs_fs_info *fs_info)
+{
+ if (!btrfs_test_opt(fs_info, DISCARD_ASYNC)) {
+ btrfs_discard_cleanup(fs_info);
+ return;
+ }
+
+ set_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
+}
+
+void btrfs_discard_stop(struct btrfs_fs_info *fs_info)
+{
+ clear_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
+}
+
+void btrfs_discard_init(struct btrfs_fs_info *fs_info)
+{
+ struct btrfs_discard_ctl *discard_ctl = &fs_info->discard_ctl;
+ int i;
+
+ spin_lock_init(&discard_ctl->lock);
+
+ INIT_DELAYED_WORK(&discard_ctl->work, btrfs_discard_workfn);
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++)
+ INIT_LIST_HEAD(&discard_ctl->discard_list[i]);
+}
+
+void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
+{
+ btrfs_discard_stop(fs_info);
+ cancel_delayed_work_sync(&fs_info->discard_ctl.work);
+}
diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
new file mode 100644
index 000000000000..f3775e84d35a
--- /dev/null
+++ b/fs/btrfs/discard.h
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#ifndef BTRFS_DISCARD_H
+#define BTRFS_DISCARD_H
+
+struct btrfs_fs_info;
+struct btrfs_discard_ctl;
+struct btrfs_block_group;
+
+void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group);
+void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group);
+void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
+ bool override);
+bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl);
+
+void btrfs_discard_resume(struct btrfs_fs_info *fs_info);
+void btrfs_discard_stop(struct btrfs_fs_info *fs_info);
+void btrfs_discard_init(struct btrfs_fs_info *fs_info);
+void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info);
+
+#endif
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index ab888d89d844..fe22ba265ced 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -41,6 +41,7 @@
#include "tree-checker.h"
#include "ref-verify.h"
#include "block-group.h"
+#include "discard.h"
#define BTRFS_SUPER_FLAG_SUPP (BTRFS_HEADER_FLAG_WRITTEN |\
BTRFS_HEADER_FLAG_RELOC |\
@@ -1953,6 +1954,8 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info)
btrfs_destroy_workqueue(fs_info->readahead_workers);
btrfs_destroy_workqueue(fs_info->flush_workers);
btrfs_destroy_workqueue(fs_info->qgroup_rescan_workers);
+ if (fs_info->discard_ctl.discard_workers)
+ destroy_workqueue(fs_info->discard_ctl.discard_workers);
/*
* Now that all other work queues are destroyed, we can safely destroy
* the queues used for metadata I/O, since tasks from those other work
@@ -2148,6 +2151,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info,
max_active, 2);
fs_info->qgroup_rescan_workers =
btrfs_alloc_workqueue(fs_info, "qgroup-rescan", flags, 1, 0);
+ fs_info->discard_ctl.discard_workers =
+ alloc_workqueue("btrfs_discard", WQ_UNBOUND | WQ_FREEZABLE, 1);
if (!(fs_info->workers && fs_info->delalloc_workers &&
fs_info->flush_workers &&
@@ -2158,7 +2163,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info,
fs_info->endio_freespace_worker && fs_info->rmw_workers &&
fs_info->caching_workers && fs_info->readahead_workers &&
fs_info->fixup_workers && fs_info->delayed_workers &&
- fs_info->qgroup_rescan_workers)) {
+ fs_info->qgroup_rescan_workers &&
+ fs_info->discard_ctl.discard_workers)) {
return -ENOMEM;
}
@@ -2793,6 +2799,8 @@ int __cold open_ctree(struct super_block *sb,
btrfs_init_dev_replace_locks(fs_info);
btrfs_init_qgroup(fs_info);
+ btrfs_discard_init(fs_info);
+
btrfs_init_free_cluster(&fs_info->meta_alloc_cluster);
btrfs_init_free_cluster(&fs_info->data_alloc_cluster);
@@ -3256,6 +3264,8 @@ int __cold open_ctree(struct super_block *sb,
btrfs_qgroup_rescan_resume(fs_info);
+ btrfs_discard_resume(fs_info);
+
if (!fs_info->uuid_root) {
btrfs_info(fs_info, "creating UUID tree");
ret = btrfs_create_uuid_tree(fs_info);
@@ -3971,6 +3981,9 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
cancel_work_sync(&fs_info->async_reclaim_work);
+ /* cancel or finish ongoing discard work */
+ btrfs_discard_cleanup(fs_info);
+
if (!sb_rdonly(fs_info->sb)) {
/*
* The cleaner kthread is stopped, so do one final pass over
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 9c8ff4307b7c..574be68e9659 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -32,6 +32,7 @@
#include "block-rsv.h"
#include "delalloc-space.h"
#include "block-group.h"
+#include "discard.h"
#undef SCRAMBLE_DELAYED_REFS
@@ -2934,6 +2935,9 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
cond_resched();
}
+ if (btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_schedule_work(&fs_info->discard_ctl, true);
+
/*
* Transaction is finished. We don't need the lock anymore. We
* do need to clean up the block groups in case of a transaction
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 7108962e208d..0a868b6df893 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -21,6 +21,7 @@
#include "space-info.h"
#include "delalloc-space.h"
#include "block-group.h"
+#include "discard.h"
#define BITS_PER_BITMAP (PAGE_SIZE * 8UL)
#define MAX_CACHE_BYTES_PER_GIG SZ_32K
@@ -755,9 +756,11 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
/*
* Sync discard ensures that the free space cache is always
* trimmed. So when reading this in, the state should reflect
- * that.
+ * that. We also do this for async as a stop gap for lack of
+ * persistence.
*/
- if (btrfs_test_opt(fs_info, DISCARD_SYNC))
+ if (btrfs_test_opt(fs_info, DISCARD_SYNC) ||
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
e->trim_state = BTRFS_TRIM_STATE_TRIMMED;
if (!e->bytes) {
@@ -2382,6 +2385,7 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
u64 offset, u64 bytes,
enum btrfs_trim_state trim_state)
{
+ struct btrfs_block_group *block_group = ctl->private;
struct btrfs_free_space *info;
int ret = 0;
@@ -2431,6 +2435,9 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
ASSERT(ret != -EEXIST);
}
+ if (trim_state != BTRFS_TRIM_STATE_TRIMMED)
+ btrfs_discard_queue_work(&fs_info->discard_ctl, block_group);
+
return ret;
}
@@ -2447,6 +2454,25 @@ int btrfs_add_free_space(struct btrfs_block_group *block_group,
bytenr, size, trim_state);
}
+/*
+ * This is a subtle distinction because when adding free space back in general,
+ * we want it to be added as untrimmed for async. But in the case where we add
+ * it on loading of a block group, we want to consider it trimmed.
+ */
+int btrfs_add_free_space_async_trimmed(struct btrfs_block_group *block_group,
+ u64 bytenr, u64 size)
+{
+ enum btrfs_trim_state trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
+
+ if (btrfs_test_opt(block_group->fs_info, DISCARD_SYNC) ||
+ btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
+ trim_state = BTRFS_TRIM_STATE_TRIMMED;
+
+ return __btrfs_add_free_space(block_group->fs_info,
+ block_group->free_space_ctl,
+ bytenr, size, trim_state);
+}
+
int btrfs_remove_free_space(struct btrfs_block_group *block_group,
u64 offset, u64 bytes)
{
@@ -3209,6 +3235,7 @@ void btrfs_init_free_cluster(struct btrfs_free_cluster *cluster)
static int do_trimming(struct btrfs_block_group *block_group,
u64 *total_trimmed, u64 start, u64 bytes,
u64 reserved_start, u64 reserved_bytes,
+ enum btrfs_trim_state reserved_trim_state,
struct btrfs_trim_range *trim_entry)
{
struct btrfs_space_info *space_info = block_group->space_info;
@@ -3216,6 +3243,9 @@ static int do_trimming(struct btrfs_block_group *block_group,
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
int ret;
int update = 0;
+ u64 end = start + bytes;
+ u64 reserved_end = reserved_start + reserved_bytes;
+ enum btrfs_trim_state trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
u64 trimmed = 0;
spin_lock(&space_info->lock);
@@ -3229,11 +3259,20 @@ static int do_trimming(struct btrfs_block_group *block_group,
spin_unlock(&space_info->lock);
ret = btrfs_discard_extent(fs_info, start, bytes, &trimmed);
- if (!ret)
+ if (!ret) {
*total_trimmed += trimmed;
+ trim_state = BTRFS_TRIM_STATE_TRIMMED;
+ }
mutex_lock(&ctl->cache_writeout_mutex);
- btrfs_add_free_space(block_group, reserved_start, reserved_bytes);
+ if (reserved_start < start)
+ __btrfs_add_free_space(fs_info, ctl, reserved_start,
+ start - reserved_start,
+ reserved_trim_state);
+ if (start + bytes < reserved_start + reserved_bytes)
+ __btrfs_add_free_space(fs_info, ctl, end, reserved_end - end,
+ reserved_trim_state);
+ __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
list_del(&trim_entry->list);
mutex_unlock(&ctl->cache_writeout_mutex);
@@ -3260,6 +3299,7 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
int ret = 0;
u64 extent_start;
u64 extent_bytes;
+ enum btrfs_trim_state extent_trim_state;
u64 bytes;
while (start < end) {
@@ -3301,6 +3341,7 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
extent_start = entry->offset;
extent_bytes = entry->bytes;
+ extent_trim_state = entry->trim_state;
start = max(start, extent_start);
bytes = min(extent_start + extent_bytes, end) - start;
if (bytes < minlen) {
@@ -3319,7 +3360,8 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
mutex_unlock(&ctl->cache_writeout_mutex);
ret = do_trimming(block_group, total_trimmed, start, bytes,
- extent_start, extent_bytes, &trim_entry);
+ extent_start, extent_bytes, extent_trim_state,
+ &trim_entry);
if (ret)
break;
next:
@@ -3445,7 +3487,7 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
mutex_unlock(&ctl->cache_writeout_mutex);
ret = do_trimming(block_group, total_trimmed, start, bytes,
- start, bytes, &trim_entry);
+ start, bytes, 0, &trim_entry);
if (ret) {
reset_trimming_bitmap(ctl, offset);
break;
diff --git a/fs/btrfs/free-space-cache.h b/fs/btrfs/free-space-cache.h
index 29d16f58b40b..a88c30cb3b2b 100644
--- a/fs/btrfs/free-space-cache.h
+++ b/fs/btrfs/free-space-cache.h
@@ -113,6 +113,8 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
enum btrfs_trim_state trim_state);
int btrfs_add_free_space(struct btrfs_block_group *block_group,
u64 bytenr, u64 size);
+int btrfs_add_free_space_async_trimmed(struct btrfs_block_group *block_group,
+ u64 bytenr, u64 size);
int btrfs_remove_free_space(struct btrfs_block_group *block_group,
u64 bytenr, u64 size);
void __btrfs_remove_free_space_cache(struct btrfs_free_space_ctl *ctl);
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index 08ac6a7a67f0..215b25012e49 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -46,6 +46,7 @@
#include "sysfs.h"
#include "tests/btrfs-tests.h"
#include "block-group.h"
+#include "discard.h"
#include "qgroup.h"
#define CREATE_TRACE_POINTS
@@ -146,6 +147,8 @@ void __btrfs_handle_fs_error(struct btrfs_fs_info *fs_info, const char *function
if (sb_rdonly(sb))
return;
+ btrfs_discard_stop(fs_info);
+
/* btrfs handle error by forcing the filesystem readonly */
sb->s_flags |= SB_RDONLY;
btrfs_info(fs_info, "forced readonly");
@@ -313,6 +316,7 @@ enum {
Opt_datasum, Opt_nodatasum,
Opt_defrag, Opt_nodefrag,
Opt_discard, Opt_nodiscard,
+ Opt_discard_mode,
Opt_nologreplay,
Opt_norecovery,
Opt_ratio,
@@ -376,6 +380,7 @@ static const match_table_t tokens = {
{Opt_nodefrag, "noautodefrag"},
{Opt_discard, "discard"},
{Opt_nodiscard, "nodiscard"},
+ {Opt_discard_mode, "discard=%s"},
{Opt_nologreplay, "nologreplay"},
{Opt_norecovery, "norecovery"},
{Opt_ratio, "metadata_ratio=%u"},
@@ -695,12 +700,26 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
info->metadata_ratio);
break;
case Opt_discard:
- btrfs_set_and_info(info, DISCARD_SYNC,
- "turning on sync discard");
+ case Opt_discard_mode:
+ if (token == Opt_discard ||
+ strcmp(args[0].from, "sync") == 0) {
+ btrfs_clear_opt(info->mount_opt, DISCARD_ASYNC);
+ btrfs_set_and_info(info, DISCARD_SYNC,
+ "turning on sync discard");
+ } else if (strcmp(args[0].from, "async") == 0) {
+ btrfs_clear_opt(info->mount_opt, DISCARD_SYNC);
+ btrfs_set_and_info(info, DISCARD_ASYNC,
+ "turning on async discard");
+ } else {
+ ret = -EINVAL;
+ goto out;
+ }
break;
case Opt_nodiscard:
btrfs_clear_and_info(info, DISCARD_SYNC,
"turning off discard");
+ btrfs_clear_and_info(info, DISCARD_ASYNC,
+ "turning off async discard");
break;
case Opt_space_cache:
case Opt_space_cache_version:
@@ -1324,6 +1343,8 @@ static int btrfs_show_options(struct seq_file *seq, struct dentry *dentry)
seq_puts(seq, ",flushoncommit");
if (btrfs_test_opt(info, DISCARD_SYNC))
seq_puts(seq, ",discard");
+ if (btrfs_test_opt(info, DISCARD_ASYNC))
+ seq_puts(seq, ",discard=async");
if (!(info->sb->s_flags & SB_POSIXACL))
seq_puts(seq, ",noacl");
if (btrfs_test_opt(info, SPACE_CACHE))
@@ -1713,6 +1734,14 @@ static inline void btrfs_remount_cleanup(struct btrfs_fs_info *fs_info,
btrfs_cleanup_defrag_inodes(fs_info);
}
+ /* If we toggled discard async. */
+ if (!btrfs_raw_test_opt(old_opts, DISCARD_ASYNC) &&
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_resume(fs_info);
+ else if (btrfs_raw_test_opt(old_opts, DISCARD_ASYNC) &&
+ !btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_cleanup(fs_info);
+
clear_bit(BTRFS_FS_STATE_REMOUNTING, &fs_info->fs_state);
}
@@ -1760,6 +1789,8 @@ static int btrfs_remount(struct super_block *sb, int *flags, char *data)
*/
cancel_work_sync(&fs_info->async_reclaim_work);
+ btrfs_discard_cleanup(fs_info);
+
/* wait for the uuid_scan task to finish */
down(&fs_info->uuid_tree_rescan_sem);
/* avoid complains from lockdep et al. */
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index f5c0c401c330..d25f5e6d5280 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -30,6 +30,7 @@
#include "tree-checker.h"
#include "space-info.h"
#include "block-group.h"
+#include "discard.h"
const struct btrfs_raid_attr btrfs_raid_array[BTRFS_NR_RAID_TYPES] = {
[BTRFS_RAID_RAID10] = {
@@ -2876,6 +2877,7 @@ static int btrfs_relocate_chunk(struct btrfs_fs_info *fs_info, u64 chunk_offset)
{
struct btrfs_root *root = fs_info->chunk_root;
struct btrfs_trans_handle *trans;
+ struct btrfs_block_group *block_group;
int ret;
/*
@@ -2899,6 +2901,11 @@ static int btrfs_relocate_chunk(struct btrfs_fs_info *fs_info, u64 chunk_offset)
if (ret)
return ret;
+ block_group = btrfs_lookup_block_group(fs_info, chunk_offset);
+ if (!block_group)
+ return -ENOENT;
+ btrfs_discard_cancel_work(&fs_info->discard_ctl, block_group);
+
trans = btrfs_start_trans_remove_block_group(root->fs_info,
chunk_offset);
if (IS_ERR(trans)) {
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* Re: [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-11-26 1:47 ` Nick Desaulniers
2019-11-26 4:07 ` Philip Li
@ 2019-11-26 4:09 ` Philip Li
1 sibling, 0 replies; 61+ messages in thread
From: Philip Li @ 2019-11-26 4:09 UTC (permalink / raw)
To: Nick Desaulniers
Cc: Dennis Zhou, Chen Rong, kbuild, clang-built-linux,
kbuild test robot, kbuild-all, David Sterba, Chris Mason,
Josef Bacik, Omar Sandoval, kernel-team, linux-btrfs
On Mon, Nov 25, 2019 at 05:47:00PM -0800, Nick Desaulniers wrote:
> On Mon, Nov 25, 2019 at 5:35 PM Philip Li <philip.li@intel.com> wrote:
> >
> > On Mon, Nov 25, 2019 at 11:39:08AM -0800, Nick Desaulniers wrote:
> > > On Mon, Nov 25, 2019 at 10:59 AM Dennis Zhou <dennis@kernel.org> wrote:
> > > >
> > > > On Thu, Nov 21, 2019 at 08:27:43PM -0800, Nick Desaulniers wrote:
> > > > > Hi Dennis,
> > > > > Below is a 0day bot report from a build w/ Clang. Warning looks legit,
> > > > > can you please take a look?
> > > > >
> > > >
> > > > Ah thanks for this! Yeah that was a miss when I switched from flags ->
> > > > an enum and didn't update the declaration properly. I'll be sending out
> > > > a v4 as another fix for arm has some rebase conflicts.
> > > >
> > > > Is there a way to enable so I get these emails directly?
> > >
> > > + Rong, Philip
> > >
> > > The reports have only been sent to our mailing list where we've been
> > > manually triaging them. The issue with enabling them globally was
> > > that the script to reproduce the warning still doesn't mention how to
> > > build w/ Clang.
Hi Nick, i forgot one question. Is it still expected to use latest clang
to build test? Any possibility the issue is related to clang compiler itself?
Thanks
> > Thanks Nick for continuous caring on this. One thing we initially worry
> > is how to avoid duplicated reports to developer, like the one that can
> > be same as gcc's finding. We haven't found a way to effectively handle
> > this.
>
> Thanks for maintaining an invaluable tool.
>
> How would the reports be duplicated? Does 0day bot build with GCC,
> then rebuild with Clang?
>
> Regardless, does it matter? If I make a mistake, and get two build
> failure emails from 0day bot instead of one, does it matter? Sometimes
> developers may just get one, as some warnings are unique to each
> compiler. Maybe it runs the risk of folks ignoring the email if the
> volume is too much, but do authors generally ignore 0day bot emails?
> (I'd hope not).
>
> >
> > >
> > > In general the reports have been high value (I ignore most reports
> > > with -Wimplicit-function-declaration, which is the most frequent as it
> > > shows the patch was not compile tested at all).
> > Do we mean the report with -Wimplicit-function-declaration can be duplicated
> > to gcc, so we can ignore them to avoid duplication to developer?
>
> Many of the warnings GCC has Clang does as well.
> -Wimplicit-function-declaration is the most common warning I see in
> triage, which developers would see regardless of toolchain had they
> compiled first before pushing. It might be interesting to see maybe
> the intersection or common flags between GCC and Clang, and only email
> Clang reports for warnings unique to clang? I think CFLAGS can even
> be passed into make invocations so you could do:
> $ make CC=clang KBUILD_CFLAGS=<list of flags common to GCC and Clang;
> -Wno-implicit-function-declaration -Wno-...>
> such that any resulting warnings were unique to Clang. I'd expect
> such a list to quickly get stale over time though.
>
> >
> > >
> > > Rong, Philip, it's been a while since we talked about this last. Is
> > > there a general timeline of when these reports will be turned on
> > > globally? Even if the directions to reproduce aren't quite right,
> > For the timeline, it's not decided due to the duplication concern. We tend
> > to look into next year after other priorities are solved for this year.
> >
> > > generally there's enough info in the existing bugs where authors can
> > > rewrite their patch without even needing to rebuild with Clang (though
> > > having correct directions to reproduce would be nice, we could wait
> > > until someone asked for them explicitly).
> > >
> > > --
> > > Thanks,
> > > ~Nick Desaulniers
>
>
>
> --
> Thanks,
> ~Nick Desaulniers
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-11-26 1:47 ` Nick Desaulniers
@ 2019-11-26 4:07 ` Philip Li
2019-11-26 4:09 ` Philip Li
1 sibling, 0 replies; 61+ messages in thread
From: Philip Li @ 2019-11-26 4:07 UTC (permalink / raw)
To: Nick Desaulniers
Cc: Dennis Zhou, Chen Rong, kbuild, clang-built-linux,
kbuild test robot, kbuild-all, David Sterba, Chris Mason,
Josef Bacik, Omar Sandoval, kernel-team, linux-btrfs
On Mon, Nov 25, 2019 at 05:47:00PM -0800, Nick Desaulniers wrote:
> On Mon, Nov 25, 2019 at 5:35 PM Philip Li <philip.li@intel.com> wrote:
> >
> > On Mon, Nov 25, 2019 at 11:39:08AM -0800, Nick Desaulniers wrote:
> > > On Mon, Nov 25, 2019 at 10:59 AM Dennis Zhou <dennis@kernel.org> wrote:
> > > >
> > > > On Thu, Nov 21, 2019 at 08:27:43PM -0800, Nick Desaulniers wrote:
> > > > > Hi Dennis,
> > > > > Below is a 0day bot report from a build w/ Clang. Warning looks legit,
> > > > > can you please take a look?
> > > > >
> > > >
> > > > Ah thanks for this! Yeah that was a miss when I switched from flags ->
> > > > an enum and didn't update the declaration properly. I'll be sending out
> > > > a v4 as another fix for arm has some rebase conflicts.
> > > >
> > > > Is there a way to enable so I get these emails directly?
> > >
> > > + Rong, Philip
> > >
> > > The reports have only been sent to our mailing list where we've been
> > > manually triaging them. The issue with enabling them globally was
> > > that the script to reproduce the warning still doesn't mention how to
> > > build w/ Clang.
> > Thanks Nick for continuous caring on this. One thing we initially worry
> > is how to avoid duplicated reports to developer, like the one that can
> > be same as gcc's finding. We haven't found a way to effectively handle
> > this.
>
> Thanks for maintaining an invaluable tool.
>
> How would the reports be duplicated? Does 0day bot build with GCC,
> then rebuild with Clang?
no, they are built separately. For duplication, i refer to the issue
can be detected by both tool, and gcc reports out already (or clang first).
>
> Regardless, does it matter? If I make a mistake, and get two build
> failure emails from 0day bot instead of one, does it matter? Sometimes
> developers may just get one, as some warnings are unique to each
> compiler. Maybe it runs the risk of folks ignoring the email if the
> volume is too much, but do authors generally ignore 0day bot emails?
> (I'd hope not).
:-) this is a good point, and recently we are working to make the
service more stable to generate reports in time.
>
> >
> > >
> > > In general the reports have been high value (I ignore most reports
> > > with -Wimplicit-function-declaration, which is the most frequent as it
> > > shows the patch was not compile tested at all).
> > Do we mean the report with -Wimplicit-function-declaration can be duplicated
> > to gcc, so we can ignore them to avoid duplication to developer?
>
> Many of the warnings GCC has Clang does as well.
> -Wimplicit-function-declaration is the most common warning I see in
> triage, which developers would see regardless of toolchain had they
> compiled first before pushing. It might be interesting to see maybe
> the intersection or common flags between GCC and Clang, and only email
> Clang reports for warnings unique to clang? I think CFLAGS can even
> be passed into make invocations so you could do:
> $ make CC=clang KBUILD_CFLAGS=<list of flags common to GCC and Clang;
> -Wno-implicit-function-declaration -Wno-...>
> such that any resulting warnings were unique to Clang. I'd expect
> such a list to quickly get stale over time though.
thanks for the idea, we will look into this.
>
> >
> > >
> > > Rong, Philip, it's been a while since we talked about this last. Is
> > > there a general timeline of when these reports will be turned on
> > > globally? Even if the directions to reproduce aren't quite right,
> > For the timeline, it's not decided due to the duplication concern. We tend
> > to look into next year after other priorities are solved for this year.
> >
> > > generally there's enough info in the existing bugs where authors can
> > > rewrite their patch without even needing to rebuild with Clang (though
> > > having correct directions to reproduce would be nice, we could wait
> > > until someone asked for them explicitly).
> > >
> > > --
> > > Thanks,
> > > ~Nick Desaulniers
>
>
>
> --
> Thanks,
> ~Nick Desaulniers
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-11-26 1:42 ` Philip Li
@ 2019-11-26 1:47 ` Nick Desaulniers
2019-11-26 4:07 ` Philip Li
2019-11-26 4:09 ` Philip Li
0 siblings, 2 replies; 61+ messages in thread
From: Nick Desaulniers @ 2019-11-26 1:47 UTC (permalink / raw)
To: Philip Li
Cc: Dennis Zhou, Chen Rong, kbuild, clang-built-linux,
kbuild test robot, kbuild-all, David Sterba, Chris Mason,
Josef Bacik, Omar Sandoval, kernel-team, linux-btrfs
On Mon, Nov 25, 2019 at 5:35 PM Philip Li <philip.li@intel.com> wrote:
>
> On Mon, Nov 25, 2019 at 11:39:08AM -0800, Nick Desaulniers wrote:
> > On Mon, Nov 25, 2019 at 10:59 AM Dennis Zhou <dennis@kernel.org> wrote:
> > >
> > > On Thu, Nov 21, 2019 at 08:27:43PM -0800, Nick Desaulniers wrote:
> > > > Hi Dennis,
> > > > Below is a 0day bot report from a build w/ Clang. Warning looks legit,
> > > > can you please take a look?
> > > >
> > >
> > > Ah thanks for this! Yeah that was a miss when I switched from flags ->
> > > an enum and didn't update the declaration properly. I'll be sending out
> > > a v4 as another fix for arm has some rebase conflicts.
> > >
> > > Is there a way to enable so I get these emails directly?
> >
> > + Rong, Philip
> >
> > The reports have only been sent to our mailing list where we've been
> > manually triaging them. The issue with enabling them globally was
> > that the script to reproduce the warning still doesn't mention how to
> > build w/ Clang.
> Thanks Nick for continuous caring on this. One thing we initially worry
> is how to avoid duplicated reports to developer, like the one that can
> be same as gcc's finding. We haven't found a way to effectively handle
> this.
Thanks for maintaining an invaluable tool.
How would the reports be duplicated? Does 0day bot build with GCC,
then rebuild with Clang?
Regardless, does it matter? If I make a mistake, and get two build
failure emails from 0day bot instead of one, does it matter? Sometimes
developers may just get one, as some warnings are unique to each
compiler. Maybe it runs the risk of folks ignoring the email if the
volume is too much, but do authors generally ignore 0day bot emails?
(I'd hope not).
>
> >
> > In general the reports have been high value (I ignore most reports
> > with -Wimplicit-function-declaration, which is the most frequent as it
> > shows the patch was not compile tested at all).
> Do we mean the report with -Wimplicit-function-declaration can be duplicated
> to gcc, so we can ignore them to avoid duplication to developer?
Many of the warnings GCC has Clang does as well.
-Wimplicit-function-declaration is the most common warning I see in
triage, which developers would see regardless of toolchain had they
compiled first before pushing. It might be interesting to see maybe
the intersection or common flags between GCC and Clang, and only email
Clang reports for warnings unique to clang? I think CFLAGS can even
be passed into make invocations so you could do:
$ make CC=clang KBUILD_CFLAGS=<list of flags common to GCC and Clang;
-Wno-implicit-function-declaration -Wno-...>
such that any resulting warnings were unique to Clang. I'd expect
such a list to quickly get stale over time though.
>
> >
> > Rong, Philip, it's been a while since we talked about this last. Is
> > there a general timeline of when these reports will be turned on
> > globally? Even if the directions to reproduce aren't quite right,
> For the timeline, it's not decided due to the duplication concern. We tend
> to look into next year after other priorities are solved for this year.
>
> > generally there's enough info in the existing bugs where authors can
> > rewrite their patch without even needing to rebuild with Clang (though
> > having correct directions to reproduce would be nice, we could wait
> > until someone asked for them explicitly).
> >
> > --
> > Thanks,
> > ~Nick Desaulniers
--
Thanks,
~Nick Desaulniers
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-11-25 19:39 ` Nick Desaulniers
@ 2019-11-26 1:42 ` Philip Li
2019-11-26 1:47 ` Nick Desaulniers
0 siblings, 1 reply; 61+ messages in thread
From: Philip Li @ 2019-11-26 1:42 UTC (permalink / raw)
To: Nick Desaulniers
Cc: Dennis Zhou, Chen Rong, kbuild, clang-built-linux,
kbuild test robot, kbuild-all, David Sterba, Chris Mason,
Josef Bacik, Omar Sandoval, kernel-team, linux-btrfs
On Mon, Nov 25, 2019 at 11:39:08AM -0800, Nick Desaulniers wrote:
> On Mon, Nov 25, 2019 at 10:59 AM Dennis Zhou <dennis@kernel.org> wrote:
> >
> > On Thu, Nov 21, 2019 at 08:27:43PM -0800, Nick Desaulniers wrote:
> > > Hi Dennis,
> > > Below is a 0day bot report from a build w/ Clang. Warning looks legit,
> > > can you please take a look?
> > >
> >
> > Ah thanks for this! Yeah that was a miss when I switched from flags ->
> > an enum and didn't update the declaration properly. I'll be sending out
> > a v4 as another fix for arm has some rebase conflicts.
> >
> > Is there a way to enable so I get these emails directly?
>
> + Rong, Philip
>
> The reports have only been sent to our mailing list where we've been
> manually triaging them. The issue with enabling them globally was
> that the script to reproduce the warning still doesn't mention how to
> build w/ Clang.
Thanks Nick for continuous caring on this. One thing we initially worry
is how to avoid duplicated reports to developer, like the one that can
be same as gcc's finding. We haven't found a way to effectively handle
this.
>
> In general the reports have been high value (I ignore most reports
> with -Wimplicit-function-declaration, which is the most frequent as it
> shows the patch was not compile tested at all).
Do we mean the report with -Wimplicit-function-declaration can be duplicated
to gcc, so we can ignore them to avoid duplication to developer?
>
> Rong, Philip, it's been a while since we talked about this last. Is
> there a general timeline of when these reports will be turned on
> globally? Even if the directions to reproduce aren't quite right,
For the timeline, it's not decided due to the duplication concern. We tend
to look into next year after other priorities are solved for this year.
> generally there's enough info in the existing bugs where authors can
> rewrite their patch without even needing to rebuild with Clang (though
> having correct directions to reproduce would be nice, we could wait
> until someone asked for them explicitly).
>
> --
> Thanks,
> ~Nick Desaulniers
^ permalink raw reply [flat|nested] 61+ messages in thread
* [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-11-25 19:46 [PATCH v4 00/22] btrfs: async discard support Dennis Zhou
@ 2019-11-25 19:46 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-11-25 19:46 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
When discard is enabled, everytime a pinned extent is released back to
the block_group's free space cache, a discard is issued for the extent.
This is an overeager approach when it comes to discarding and helping
the SSD maintain enough free space to prevent severe garbage collection
situations.
This adds the beginning of async discard. Instead of issuing a discard
prior to returning it to the free space, it is just marked as untrimmed.
The block_group is then added to a LRU which then feeds into a workqueue
to issue discards at a much slower rate. Full discarding of unused block
groups is still done and will be address in a future patch in this
series.
For now, we don't persist the discard state of extents and bitmaps.
Therefore, our failure recovery mode will be to consider extents
untrimmed. This lets us handle failure and unmounting as one in the
same.
On a number of Facebook webservers, I collected data every minute
accounting the time we spent in btrfs_finish_extent_commit() (col. 1)
and in btrfs_commit_transaction() (col. 2). btrfs_finish_extent_commit()
is where we discard extents synchronously before returning them to the
free space cache.
discard=sync:
p99 total per minute p99 total per minute
Drive | extent_commit() (ms) | commit_trans() (ms)
---------------------------------------------------------------
Drive A | 434 | 1170
Drive B | 880 | 2330
Drive C | 2943 | 3920
Drive D | 4763 | 5701
discard=async:
p99 total per minute p99 total per minute
Drive | extent_commit() (ms) | commit_trans() (ms)
--------------------------------------------------------------
Drive A | 134 | 956
Drive B | 64 | 1972
Drive C | 59 | 1032
Drive D | 62 | 1200
While it's not great that the stats are cumulative over 1m, all of these
servers are running the same workload and and the delta between the two
are substantial. We are spending significantly less time in
btrfs_finish_extent_commit() which is responsible for discarding.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/Makefile | 2 +-
fs/btrfs/block-group.c | 4 +
fs/btrfs/block-group.h | 9 ++
fs/btrfs/ctree.h | 21 +++
fs/btrfs/discard.c | 273 ++++++++++++++++++++++++++++++++++++
fs/btrfs/discard.h | 26 ++++
fs/btrfs/disk-io.c | 15 +-
fs/btrfs/extent-tree.c | 4 +
fs/btrfs/free-space-cache.c | 35 ++++-
fs/btrfs/super.c | 35 ++++-
10 files changed, 414 insertions(+), 10 deletions(-)
create mode 100644 fs/btrfs/discard.c
create mode 100644 fs/btrfs/discard.h
diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
index 82200dbca5ac..9a0ff3384381 100644
--- a/fs/btrfs/Makefile
+++ b/fs/btrfs/Makefile
@@ -11,7 +11,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
uuid-tree.o props.o free-space-tree.o tree-checker.o space-info.o \
- block-rsv.o delalloc-space.o block-group.o
+ block-rsv.o delalloc-space.o block-group.o discard.o
btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c
index 6064be2d5556..d2bc46c365f4 100644
--- a/fs/btrfs/block-group.c
+++ b/fs/btrfs/block-group.c
@@ -14,6 +14,7 @@
#include "sysfs.h"
#include "tree-log.h"
#include "delalloc-space.h"
+#include "discard.h"
/*
* Return target flags in extended format or 0 if restripe for this chunk_type
@@ -1272,6 +1273,8 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
}
spin_unlock(&fs_info->unused_bgs_lock);
+ btrfs_discard_cancel_work(&fs_info->discard_ctl, block_group);
+
mutex_lock(&fs_info->delete_unused_bgs_mutex);
/* Don't want to race with allocators so take the groups_sem */
@@ -1617,6 +1620,7 @@ static struct btrfs_block_group *btrfs_create_block_group_cache(
INIT_LIST_HEAD(&cache->cluster_list);
INIT_LIST_HEAD(&cache->bg_list);
INIT_LIST_HEAD(&cache->ro_list);
+ INIT_LIST_HEAD(&cache->discard_list);
INIT_LIST_HEAD(&cache->dirty_list);
INIT_LIST_HEAD(&cache->io_list);
btrfs_init_free_space_ctl(cache);
diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h
index 9b409676c4b2..884defd61dcd 100644
--- a/fs/btrfs/block-group.h
+++ b/fs/btrfs/block-group.h
@@ -116,7 +116,11 @@ struct btrfs_block_group {
/* For read-only block groups */
struct list_head ro_list;
+ /* For discard operations */
atomic_t trimming;
+ struct list_head discard_list;
+ int discard_index;
+ u64 discard_eligible_time;
/* For dirty block groups */
struct list_head dirty_list;
@@ -158,6 +162,11 @@ struct btrfs_block_group {
struct btrfs_full_stripe_locks_tree full_stripe_locks_root;
};
+static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
+{
+ return (block_group->start + block_group->length);
+}
+
#ifdef CONFIG_BTRFS_DEBUG
static inline int btrfs_should_fragment_free_space(
struct btrfs_block_group *block_group)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 8ac3b2deef4a..d5ce8054f074 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -440,6 +440,21 @@ struct btrfs_full_stripe_locks_tree {
struct mutex lock;
};
+/* Discard control. */
+/*
+ * Async discard uses multiple lists to differentiate the discard filter
+ * parameters.
+ */
+#define BTRFS_NR_DISCARD_LISTS 1
+
+struct btrfs_discard_ctl {
+ struct workqueue_struct *discard_workers;
+ struct delayed_work work;
+ spinlock_t lock;
+ struct btrfs_block_group *block_group;
+ struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
+};
+
/* delayed seq elem */
struct seq_list {
struct list_head list;
@@ -526,6 +541,9 @@ enum {
* so we don't need to offload checksums to workqueues.
*/
BTRFS_FS_CSUM_IMPL_FAST,
+
+ /* Indicate that the discard workqueue can service discards. */
+ BTRFS_FS_DISCARD_RUNNING,
};
struct btrfs_fs_info {
@@ -816,6 +834,8 @@ struct btrfs_fs_info {
struct btrfs_workqueue *scrub_wr_completion_workers;
struct btrfs_workqueue *scrub_parity_workers;
+ struct btrfs_discard_ctl discard_ctl;
+
#ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY
u32 check_integrity_print_mask;
#endif
@@ -1189,6 +1209,7 @@ static inline u32 BTRFS_MAX_XATTR_SIZE(const struct btrfs_fs_info *info)
#define BTRFS_MOUNT_FREE_SPACE_TREE (1 << 26)
#define BTRFS_MOUNT_NOLOGREPLAY (1 << 27)
#define BTRFS_MOUNT_REF_VERIFY (1 << 28)
+#define BTRFS_MOUNT_DISCARD_ASYNC (1 << 29)
#define BTRFS_DEFAULT_COMMIT_INTERVAL (30)
#define BTRFS_DEFAULT_MAX_INLINE (2048)
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
new file mode 100644
index 000000000000..15d54de3d682
--- /dev/null
+++ b/fs/btrfs/discard.c
@@ -0,0 +1,273 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/jiffies.h>
+#include <linux/kernel.h>
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/sizes.h>
+#include <linux/workqueue.h>
+#include "ctree.h"
+#include "block-group.h"
+#include "discard.h"
+#include "free-space-cache.h"
+
+/* This is an initial delay to give some chance for lba reuse. */
+#define BTRFS_DISCARD_DELAY (120ULL * NSEC_PER_SEC)
+
+static struct list_head *btrfs_get_discard_list(
+ struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ return &discard_ctl->discard_list[block_group->discard_index];
+}
+
+void btrfs_add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ spin_lock(&discard_ctl->lock);
+
+ if (list_empty(&block_group->discard_list))
+ block_group->discard_eligible_time = (ktime_get_ns() +
+ BTRFS_DISCARD_DELAY);
+
+ list_move_tail(&block_group->discard_list,
+ btrfs_get_discard_list(discard_ctl, block_group));
+
+ spin_unlock(&discard_ctl->lock);
+}
+
+static bool remove_from_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ bool running = false;
+
+ spin_lock(&discard_ctl->lock);
+
+ if (block_group == discard_ctl->block_group) {
+ running = true;
+ discard_ctl->block_group = NULL;
+ }
+
+ block_group->discard_eligible_time = 0;
+ list_del_init(&block_group->discard_list);
+
+ spin_unlock(&discard_ctl->lock);
+
+ return running;
+}
+
+/**
+ * find_next_block_group - find block_group that's up next for discarding
+ * @discard_ctl: discard control
+ * @now: current time
+ *
+ * Iterate over the discard lists to find the next block_group up for
+ * discarding checking the discard_eligible_time of block_group.
+ */
+static struct btrfs_block_group *find_next_block_group(
+ struct btrfs_discard_ctl *discard_ctl,
+ u64 now)
+{
+ struct btrfs_block_group *ret_block_group = NULL, *block_group;
+ int i;
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++) {
+ struct list_head *discard_list = &discard_ctl->discard_list[i];
+
+ if (!list_empty(discard_list)) {
+ block_group = list_first_entry(discard_list,
+ struct btrfs_block_group,
+ discard_list);
+
+ if (!ret_block_group)
+ ret_block_group = block_group;
+
+ if (ret_block_group->discard_eligible_time < now)
+ break;
+
+ if (ret_block_group->discard_eligible_time >
+ block_group->discard_eligible_time)
+ ret_block_group = block_group;
+ }
+ }
+
+ return ret_block_group;
+}
+
+/**
+ * peek_discard_list - wrap find_next_block_group()
+ * @discard_ctl: discard control
+ *
+ * This wraps find_next_block_group() and sets the block_group to be in use.
+ */
+static struct btrfs_block_group *peek_discard_list(
+ struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_block_group *block_group;
+ u64 now = ktime_get_ns();
+
+ spin_lock(&discard_ctl->lock);
+
+ block_group = find_next_block_group(discard_ctl, now);
+
+ if (block_group && now < block_group->discard_eligible_time)
+ block_group = NULL;
+
+ discard_ctl->block_group = block_group;
+
+ spin_unlock(&discard_ctl->lock);
+
+ return block_group;
+}
+
+/**
+ * btrfs_discard_cancel_work - remove a block_group from the discard lists
+ * @discard_ctl: discard control
+ * @block_group: block_group of interest
+ *
+ * This removes @block_group from the discard lists. If necessary, it waits on
+ * the current work and then reschedules the delayed work.
+ */
+void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ if (remove_from_discard_list(discard_ctl, block_group)) {
+ cancel_delayed_work_sync(&discard_ctl->work);
+ btrfs_discard_schedule_work(discard_ctl, true);
+ }
+}
+
+/**
+ * btrfs_discard_queue_work - handles queuing the block_groups
+ * @discard_ctl: discard control
+ * @block_group: block_group of interest
+ *
+ * This maintains the LRU order of the discard lists.
+ */
+void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ if (!block_group ||
+ !btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
+ return;
+
+ btrfs_add_to_discard_list(discard_ctl, block_group);
+ if (!delayed_work_pending(&discard_ctl->work))
+ btrfs_discard_schedule_work(discard_ctl, false);
+}
+
+/**
+ * btrfs_discard_schedule_work - responsible for scheduling the discard work
+ * @discard_ctl: discard control
+ * @override: override the current timer
+ *
+ * Discards are issued by a delayed workqueue item. @override is used to
+ * update the current delay as the baseline delay interview is reevaluated
+ * on transaction commit. This is also maxed with any other rate limit.
+ */
+void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
+ bool override)
+{
+ struct btrfs_block_group *block_group;
+ u64 now = ktime_get_ns();
+
+ spin_lock(&discard_ctl->lock);
+
+ if (!btrfs_run_discard_work(discard_ctl))
+ goto out;
+
+ if (!override && delayed_work_pending(&discard_ctl->work))
+ goto out;
+
+ block_group = find_next_block_group(discard_ctl, now);
+ if (block_group) {
+ u64 delay = 0;
+
+ if (now < block_group->discard_eligible_time)
+ delay = nsecs_to_jiffies(
+ block_group->discard_eligible_time - now);
+
+ mod_delayed_work(discard_ctl->discard_workers,
+ &discard_ctl->work,
+ delay);
+ }
+
+out:
+ spin_unlock(&discard_ctl->lock);
+}
+
+/**
+ * btrfs_discard_workfn - discard work function
+ * @work: work
+ *
+ * This finds the next block_group to start discarding and then discards it.
+ */
+static void btrfs_discard_workfn(struct work_struct *work)
+{
+ struct btrfs_discard_ctl *discard_ctl;
+ struct btrfs_block_group *block_group;
+ u64 trimmed = 0;
+
+ discard_ctl = container_of(work, struct btrfs_discard_ctl, work.work);
+
+ block_group = peek_discard_list(discard_ctl);
+ if (!block_group || !btrfs_run_discard_work(discard_ctl))
+ return;
+
+ btrfs_trim_block_group(block_group, &trimmed, block_group->start,
+ btrfs_block_group_end(block_group), 0);
+
+ remove_from_discard_list(discard_ctl, block_group);
+
+ btrfs_discard_schedule_work(discard_ctl, false);
+}
+
+/**
+ * btrfs_run_discard_work - determines if async discard should be running
+ * @discard_ctl: discard control
+ *
+ * Checks if the file system is writeable and BTRFS_FS_DISCARD_RUNNING is set.
+ */
+bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_fs_info *fs_info = container_of(discard_ctl,
+ struct btrfs_fs_info,
+ discard_ctl);
+
+ return (!(fs_info->sb->s_flags & SB_RDONLY) &&
+ test_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags));
+}
+
+void btrfs_discard_resume(struct btrfs_fs_info *fs_info)
+{
+ if (!btrfs_test_opt(fs_info, DISCARD_ASYNC)) {
+ btrfs_discard_cleanup(fs_info);
+ return;
+ }
+
+ set_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
+}
+
+void btrfs_discard_stop(struct btrfs_fs_info *fs_info)
+{
+ clear_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
+}
+
+void btrfs_discard_init(struct btrfs_fs_info *fs_info)
+{
+ struct btrfs_discard_ctl *discard_ctl = &fs_info->discard_ctl;
+ int i;
+
+ spin_lock_init(&discard_ctl->lock);
+
+ INIT_DELAYED_WORK(&discard_ctl->work, btrfs_discard_workfn);
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++)
+ INIT_LIST_HEAD(&discard_ctl->discard_list[i]);
+}
+
+void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
+{
+ btrfs_discard_stop(fs_info);
+ cancel_delayed_work_sync(&fs_info->discard_ctl.work);
+}
diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
new file mode 100644
index 000000000000..439ca8c51877
--- /dev/null
+++ b/fs/btrfs/discard.h
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#ifndef BTRFS_DISCARD_H
+#define BTRFS_DISCARD_H
+
+struct btrfs_fs_info;
+struct btrfs_discard_ctl;
+struct btrfs_block_group;
+
+void btrfs_add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group);
+
+void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group);
+void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group);
+void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
+ bool override);
+bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl);
+
+void btrfs_discard_resume(struct btrfs_fs_info *fs_info);
+void btrfs_discard_stop(struct btrfs_fs_info *fs_info);
+void btrfs_discard_init(struct btrfs_fs_info *fs_info);
+void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info);
+
+#endif
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index e0edfdc9c82b..8785b5c999b0 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -41,6 +41,7 @@
#include "tree-checker.h"
#include "ref-verify.h"
#include "block-group.h"
+#include "discard.h"
#define BTRFS_SUPER_FLAG_SUPP (BTRFS_HEADER_FLAG_WRITTEN |\
BTRFS_HEADER_FLAG_RELOC |\
@@ -1953,6 +1954,8 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info)
btrfs_destroy_workqueue(fs_info->readahead_workers);
btrfs_destroy_workqueue(fs_info->flush_workers);
btrfs_destroy_workqueue(fs_info->qgroup_rescan_workers);
+ if (fs_info->discard_ctl.discard_workers)
+ destroy_workqueue(fs_info->discard_ctl.discard_workers);
/*
* Now that all other work queues are destroyed, we can safely destroy
* the queues used for metadata I/O, since tasks from those other work
@@ -2148,6 +2151,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info,
max_active, 2);
fs_info->qgroup_rescan_workers =
btrfs_alloc_workqueue(fs_info, "qgroup-rescan", flags, 1, 0);
+ fs_info->discard_ctl.discard_workers =
+ alloc_workqueue("btrfs_discard", WQ_UNBOUND | WQ_FREEZABLE, 1);
if (!(fs_info->workers && fs_info->delalloc_workers &&
fs_info->flush_workers &&
@@ -2158,7 +2163,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info,
fs_info->endio_freespace_worker && fs_info->rmw_workers &&
fs_info->caching_workers && fs_info->readahead_workers &&
fs_info->fixup_workers && fs_info->delayed_workers &&
- fs_info->qgroup_rescan_workers)) {
+ fs_info->qgroup_rescan_workers &&
+ fs_info->discard_ctl.discard_workers)) {
return -ENOMEM;
}
@@ -2793,6 +2799,8 @@ int __cold open_ctree(struct super_block *sb,
btrfs_init_dev_replace_locks(fs_info);
btrfs_init_qgroup(fs_info);
+ btrfs_discard_init(fs_info);
+
btrfs_init_free_cluster(&fs_info->meta_alloc_cluster);
btrfs_init_free_cluster(&fs_info->data_alloc_cluster);
@@ -3263,6 +3271,8 @@ int __cold open_ctree(struct super_block *sb,
btrfs_qgroup_rescan_resume(fs_info);
+ btrfs_discard_resume(fs_info);
+
if (!fs_info->uuid_root) {
btrfs_info(fs_info, "creating UUID tree");
ret = btrfs_create_uuid_tree(fs_info);
@@ -3954,6 +3964,9 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
*/
kthread_park(fs_info->cleaner_kthread);
+ /* cancel or finish ongoing work */
+ btrfs_discard_cleanup(fs_info);
+
/* wait for the qgroup rescan worker to stop */
btrfs_qgroup_wait_for_completion(fs_info, false);
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 5dc696d0c5b2..857642dc8589 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -32,6 +32,7 @@
#include "block-rsv.h"
#include "delalloc-space.h"
#include "block-group.h"
+#include "discard.h"
#undef SCRAMBLE_DELAYED_REFS
@@ -2940,6 +2941,9 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
cond_resched();
}
+ if (btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_schedule_work(&fs_info->discard_ctl, true);
+
/*
* Transaction is finished. We don't need the lock anymore. We
* do need to clean up the block groups in case of a transaction
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 49ae1e6ee104..449bfaf6030f 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -21,6 +21,7 @@
#include "space-info.h"
#include "delalloc-space.h"
#include "block-group.h"
+#include "discard.h"
#define BITS_PER_BITMAP (PAGE_SIZE * 8UL)
#define MAX_CACHE_BYTES_PER_GIG SZ_32K
@@ -755,9 +756,11 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
/*
* Sync discard ensures that the free space cache is always
* trimmed. So when reading this in, the state should reflect
- * that.
+ * that. We also do this for async as a stop gap for lack of
+ * persistence.
*/
- if (btrfs_test_opt(fs_info, DISCARD_SYNC))
+ if (btrfs_test_opt(fs_info, DISCARD_SYNC) ||
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
e->trim_state = BTRFS_TRIM_STATE_TRIMMED;
if (!e->bytes) {
@@ -2382,6 +2385,7 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
u64 offset, u64 bytes,
enum btrfs_trim_state trim_state)
{
+ struct btrfs_block_group *block_group = ctl->private;
struct btrfs_free_space *info;
int ret = 0;
@@ -2431,6 +2435,9 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
ASSERT(ret != -EEXIST);
}
+ if (trim_state != BTRFS_TRIM_STATE_TRIMMED)
+ btrfs_discard_queue_work(&fs_info->discard_ctl, block_group);
+
return ret;
}
@@ -3204,6 +3211,7 @@ void btrfs_init_free_cluster(struct btrfs_free_cluster *cluster)
static int do_trimming(struct btrfs_block_group *block_group,
u64 *total_trimmed, u64 start, u64 bytes,
u64 reserved_start, u64 reserved_bytes,
+ enum btrfs_trim_state reserved_trim_state,
struct btrfs_trim_range *trim_entry)
{
struct btrfs_space_info *space_info = block_group->space_info;
@@ -3211,6 +3219,9 @@ static int do_trimming(struct btrfs_block_group *block_group,
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
int ret;
int update = 0;
+ u64 end = start + bytes;
+ u64 reserved_end = reserved_start + reserved_bytes;
+ enum btrfs_trim_state trim_state = BTRFS_TRIM_STATE_UNTRIMMED;
u64 trimmed = 0;
spin_lock(&space_info->lock);
@@ -3224,11 +3235,20 @@ static int do_trimming(struct btrfs_block_group *block_group,
spin_unlock(&space_info->lock);
ret = btrfs_discard_extent(fs_info, start, bytes, &trimmed);
- if (!ret)
+ if (!ret) {
*total_trimmed += trimmed;
+ trim_state = BTRFS_TRIM_STATE_TRIMMED;
+ }
mutex_lock(&ctl->cache_writeout_mutex);
- btrfs_add_free_space(block_group, reserved_start, reserved_bytes);
+ if (reserved_start < start)
+ __btrfs_add_free_space(fs_info, ctl, reserved_start,
+ start - reserved_start,
+ reserved_trim_state);
+ if (start + bytes < reserved_start + reserved_bytes)
+ __btrfs_add_free_space(fs_info, ctl, end, reserved_end - end,
+ reserved_trim_state);
+ __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
list_del(&trim_entry->list);
mutex_unlock(&ctl->cache_writeout_mutex);
@@ -3255,6 +3275,7 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
int ret = 0;
u64 extent_start;
u64 extent_bytes;
+ enum btrfs_trim_state extent_trim_state;
u64 bytes;
while (start < end) {
@@ -3296,6 +3317,7 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
extent_start = entry->offset;
extent_bytes = entry->bytes;
+ extent_trim_state = entry->trim_state;
start = max(start, extent_start);
bytes = min(extent_start + extent_bytes, end) - start;
if (bytes < minlen) {
@@ -3314,7 +3336,8 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
mutex_unlock(&ctl->cache_writeout_mutex);
ret = do_trimming(block_group, total_trimmed, start, bytes,
- extent_start, extent_bytes, &trim_entry);
+ extent_start, extent_bytes, extent_trim_state,
+ &trim_entry);
if (ret)
break;
next:
@@ -3440,7 +3463,7 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
mutex_unlock(&ctl->cache_writeout_mutex);
ret = do_trimming(block_group, total_trimmed, start, bytes,
- start, bytes, &trim_entry);
+ start, bytes, 0, &trim_entry);
if (ret) {
reset_trimming_bitmap(ctl, offset);
break;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index f131fb9f0f69..0edf00753c00 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -46,6 +46,7 @@
#include "sysfs.h"
#include "tests/btrfs-tests.h"
#include "block-group.h"
+#include "discard.h"
#include "qgroup.h"
#define CREATE_TRACE_POINTS
@@ -146,6 +147,8 @@ void __btrfs_handle_fs_error(struct btrfs_fs_info *fs_info, const char *function
if (sb_rdonly(sb))
return;
+ btrfs_discard_stop(fs_info);
+
/* btrfs handle error by forcing the filesystem readonly */
sb->s_flags |= SB_RDONLY;
btrfs_info(fs_info, "forced readonly");
@@ -313,6 +316,7 @@ enum {
Opt_datasum, Opt_nodatasum,
Opt_defrag, Opt_nodefrag,
Opt_discard, Opt_nodiscard,
+ Opt_discard_mode,
Opt_nologreplay,
Opt_norecovery,
Opt_ratio,
@@ -376,6 +380,7 @@ static const match_table_t tokens = {
{Opt_nodefrag, "noautodefrag"},
{Opt_discard, "discard"},
{Opt_nodiscard, "nodiscard"},
+ {Opt_discard_mode, "discard=%s"},
{Opt_nologreplay, "nologreplay"},
{Opt_norecovery, "norecovery"},
{Opt_ratio, "metadata_ratio=%u"},
@@ -695,12 +700,26 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
info->metadata_ratio);
break;
case Opt_discard:
- btrfs_set_and_info(info, DISCARD_SYNC,
- "turning on sync discard");
+ case Opt_discard_mode:
+ if (token == Opt_discard ||
+ strcmp(args[0].from, "sync") == 0) {
+ btrfs_clear_opt(info->mount_opt, DISCARD_ASYNC);
+ btrfs_set_and_info(info, DISCARD_SYNC,
+ "turning on sync discard");
+ } else if (strcmp(args[0].from, "async") == 0) {
+ btrfs_clear_opt(info->mount_opt, DISCARD_SYNC);
+ btrfs_set_and_info(info, DISCARD_ASYNC,
+ "turning on async discard");
+ } else {
+ ret = -EINVAL;
+ goto out;
+ }
break;
case Opt_nodiscard:
btrfs_clear_and_info(info, DISCARD_SYNC,
"turning off discard");
+ btrfs_clear_and_info(info, DISCARD_ASYNC,
+ "turning off async discard");
break;
case Opt_space_cache:
case Opt_space_cache_version:
@@ -1324,6 +1343,8 @@ static int btrfs_show_options(struct seq_file *seq, struct dentry *dentry)
seq_puts(seq, ",flushoncommit");
if (btrfs_test_opt(info, DISCARD_SYNC))
seq_puts(seq, ",discard");
+ if (btrfs_test_opt(info, DISCARD_ASYNC))
+ seq_puts(seq, ",discard=async");
if (!(info->sb->s_flags & SB_POSIXACL))
seq_puts(seq, ",noacl");
if (btrfs_test_opt(info, SPACE_CACHE))
@@ -1713,6 +1734,14 @@ static inline void btrfs_remount_cleanup(struct btrfs_fs_info *fs_info,
btrfs_cleanup_defrag_inodes(fs_info);
}
+ /* If we toggled discard async. */
+ if (!btrfs_raw_test_opt(old_opts, DISCARD_ASYNC) &&
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_resume(fs_info);
+ else if (btrfs_raw_test_opt(old_opts, DISCARD_ASYNC) &&
+ !btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_cleanup(fs_info);
+
clear_bit(BTRFS_FS_STATE_REMOUNTING, &fs_info->fs_state);
}
@@ -1760,6 +1789,8 @@ static int btrfs_remount(struct super_block *sb, int *flags, char *data)
*/
cancel_work_sync(&fs_info->async_reclaim_work);
+ btrfs_discard_cleanup(fs_info);
+
/* wait for the uuid_scan task to finish */
down(&fs_info->uuid_tree_rescan_sem);
/* avoid complains from lockdep et al. */
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* Re: [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-11-25 18:59 ` Dennis Zhou
@ 2019-11-25 19:39 ` Nick Desaulniers
2019-11-26 1:42 ` Philip Li
0 siblings, 1 reply; 61+ messages in thread
From: Nick Desaulniers @ 2019-11-25 19:39 UTC (permalink / raw)
To: Dennis Zhou, Chen Rong, Philip Li
Cc: kbuild, clang-built-linux, kbuild test robot, kbuild-all,
David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Mon, Nov 25, 2019 at 10:59 AM Dennis Zhou <dennis@kernel.org> wrote:
>
> On Thu, Nov 21, 2019 at 08:27:43PM -0800, Nick Desaulniers wrote:
> > Hi Dennis,
> > Below is a 0day bot report from a build w/ Clang. Warning looks legit,
> > can you please take a look?
> >
>
> Ah thanks for this! Yeah that was a miss when I switched from flags ->
> an enum and didn't update the declaration properly. I'll be sending out
> a v4 as another fix for arm has some rebase conflicts.
>
> Is there a way to enable so I get these emails directly?
+ Rong, Philip
The reports have only been sent to our mailing list where we've been
manually triaging them. The issue with enabling them globally was
that the script to reproduce the warning still doesn't mention how to
build w/ Clang.
In general the reports have been high value (I ignore most reports
with -Wimplicit-function-declaration, which is the most frequent as it
shows the patch was not compile tested at all).
Rong, Philip, it's been a while since we talked about this last. Is
there a general timeline of when these reports will be turned on
globally? Even if the directions to reproduce aren't quite right,
generally there's enough info in the existing bugs where authors can
rewrite their patch without even needing to rebuild with Clang (though
having correct directions to reproduce would be nice, we could wait
until someone asked for them explicitly).
--
Thanks,
~Nick Desaulniers
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-11-22 4:27 ` Nick Desaulniers
@ 2019-11-25 18:59 ` Dennis Zhou
2019-11-25 19:39 ` Nick Desaulniers
0 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-11-25 18:59 UTC (permalink / raw)
To: Nick Desaulniers
Cc: dennis, kbuild, clang-built-linux, kbuild test robot, kbuild-all,
David Sterba, Chris Mason, Josef Bacik, osandov, kernel-team,
linux-btrfs
On Thu, Nov 21, 2019 at 08:27:43PM -0800, Nick Desaulniers wrote:
> Hi Dennis,
> Below is a 0day bot report from a build w/ Clang. Warning looks legit,
> can you please take a look?
>
Ah thanks for this! Yeah that was a miss when I switched from flags ->
an enum and didn't update the declaration properly. I'll be sending out
a v4 as another fix for arm has some rebase conflicts.
Is there a way to enable so I get these emails directly?
Thanks,
Dennis
> On Thu, Nov 21, 2019 at 11:27 AM kbuild test robot <lkp@intel.com> wrote:
> >
> > CC: kbuild-all@lists.01.org
> > In-Reply-To: <63d3257efe1158a6fbbd7abe865cd9250b494438.1574282259.git.dennis@kernel.org>
> > References: <63d3257efe1158a6fbbd7abe865cd9250b494438.1574282259.git.dennis@kernel.org>
> > TO: Dennis Zhou <dennis@kernel.org>
> > CC: David Sterba <dsterba@suse.com>, Chris Mason <clm@fb.com>, Josef Bacik <josef@toxicpanda.com>, Omar Sandoval <osandov@osandov.com>
> > CC: kernel-team@fb.com, linux-btrfs@vger.kernel.org, Dennis Zhou <dennis@kernel.org>
> >
> > Hi Dennis,
> >
> > I love your patch! Perhaps something to improve:
> >
> > [auto build test WARNING on kdave/for-next]
> > [also build test WARNING on next-20191121]
> > [cannot apply to v5.4-rc8]
> > [if your patch is applied to the wrong git tree, please drop us a note to help
> > improve the system. BTW, we also suggest to use '--base' option to specify the
> > base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
> >
> > url: https://github.com/0day-ci/linux/commits/Dennis-Zhou/btrfs-async-discard-support/20191121-230429
> > base: https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-next
> > config: arm64-defconfig (attached as .config)
> > compiler: clang version 10.0.0 (git://gitmirror/llvm_project cf823ce4ad9d04c69b7c29d236f7b14c875111c2)
> > reproduce:
> > wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
> > chmod +x ~/bin/make.cross
> > # save the attached .config to linux build tree
> > make.cross ARCH=arm64
> >
> > If you fix the issue, kindly add following tag
> > Reported-by: kbuild test robot <lkp@intel.com>
> >
> > All warnings (new ones prefixed by >>):
> >
> > >> fs/btrfs/free-space-cache.c:3238:6: warning: variable 'trim_state' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized]
> > if (!ret) {
> > ^~~~
> > fs/btrfs/free-space-cache.c:3251:53: note: uninitialized use occurs here
> > __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
> > ^~~~~~~~~~
> > fs/btrfs/free-space-cache.c:3238:2: note: remove the 'if' if its condition is always true
> > if (!ret) {
> > ^~~~~~~~~~
> > fs/btrfs/free-space-cache.c:3224:2: note: variable 'trim_state' is declared here
> > enum btrfs_trim_state trim_state;
> > ^
> > 1 warning generated.
> >
> > vim +3238 fs/btrfs/free-space-cache.c
> >
> > 3210
> > 3211 static int do_trimming(struct btrfs_block_group *block_group,
> > 3212 u64 *total_trimmed, u64 start, u64 bytes,
> > 3213 u64 reserved_start, u64 reserved_bytes,
> > 3214 enum btrfs_trim_state reserved_trim_state,
> > 3215 struct btrfs_trim_range *trim_entry)
> > 3216 {
> > 3217 struct btrfs_space_info *space_info = block_group->space_info;
> > 3218 struct btrfs_fs_info *fs_info = block_group->fs_info;
> > 3219 struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
> > 3220 int ret;
> > 3221 int update = 0;
> > 3222 u64 end = start + bytes;
> > 3223 u64 reserved_end = reserved_start + reserved_bytes;
> > 3224 enum btrfs_trim_state trim_state;
> > 3225 u64 trimmed = 0;
> > 3226
> > 3227 spin_lock(&space_info->lock);
> > 3228 spin_lock(&block_group->lock);
> > 3229 if (!block_group->ro) {
> > 3230 block_group->reserved += reserved_bytes;
> > 3231 space_info->bytes_reserved += reserved_bytes;
> > 3232 update = 1;
> > 3233 }
> > 3234 spin_unlock(&block_group->lock);
> > 3235 spin_unlock(&space_info->lock);
> > 3236
> > 3237 ret = btrfs_discard_extent(fs_info, start, bytes, &trimmed);
> > > 3238 if (!ret) {
> > 3239 *total_trimmed += trimmed;
> > 3240 trim_state = BTRFS_TRIM_STATE_TRIMMED;
> > 3241 }
> > 3242
> > 3243 mutex_lock(&ctl->cache_writeout_mutex);
> > 3244 if (reserved_start < start)
> > 3245 __btrfs_add_free_space(fs_info, ctl, reserved_start,
> > 3246 start - reserved_start,
> > 3247 reserved_trim_state);
> > 3248 if (start + bytes < reserved_start + reserved_bytes)
> > 3249 __btrfs_add_free_space(fs_info, ctl, end, reserved_end - end,
> > 3250 reserved_trim_state);
> > 3251 __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
>
> ^ oops
>
> > 3252 list_del(&trim_entry->list);
> > 3253 mutex_unlock(&ctl->cache_writeout_mutex);
> > 3254
> > 3255 if (update) {
> > 3256 spin_lock(&space_info->lock);
> > 3257 spin_lock(&block_group->lock);
> > 3258 if (block_group->ro)
> > 3259 space_info->bytes_readonly += reserved_bytes;
> > 3260 block_group->reserved -= reserved_bytes;
> > 3261 space_info->bytes_reserved -= reserved_bytes;
> > 3262 spin_unlock(&block_group->lock);
> > 3263 spin_unlock(&space_info->lock);
> > 3264 }
> > 3265
> > 3266 return ret;
> > 3267 }
> > 3268
> >
> > ---
> > 0-DAY kernel test infrastructure Open Source Technology Center
> > https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org Intel Corporation
>
>
>
> --
> Thanks,
> ~Nick Desaulniers
^ permalink raw reply [flat|nested] 61+ messages in thread
* Re: [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
[not found] <201911220351.HPI9gxNo%lkp@intel.com>
@ 2019-11-22 4:27 ` Nick Desaulniers
2019-11-25 18:59 ` Dennis Zhou
0 siblings, 1 reply; 61+ messages in thread
From: Nick Desaulniers @ 2019-11-22 4:27 UTC (permalink / raw)
To: dennis
Cc: kbuild, clang-built-linux, kbuild test robot, kbuild-all,
David Sterba, Chris Mason, Josef Bacik, osandov, kernel-team,
linux-btrfs
Hi Dennis,
Below is a 0day bot report from a build w/ Clang. Warning looks legit,
can you please take a look?
On Thu, Nov 21, 2019 at 11:27 AM kbuild test robot <lkp@intel.com> wrote:
>
> CC: kbuild-all@lists.01.org
> In-Reply-To: <63d3257efe1158a6fbbd7abe865cd9250b494438.1574282259.git.dennis@kernel.org>
> References: <63d3257efe1158a6fbbd7abe865cd9250b494438.1574282259.git.dennis@kernel.org>
> TO: Dennis Zhou <dennis@kernel.org>
> CC: David Sterba <dsterba@suse.com>, Chris Mason <clm@fb.com>, Josef Bacik <josef@toxicpanda.com>, Omar Sandoval <osandov@osandov.com>
> CC: kernel-team@fb.com, linux-btrfs@vger.kernel.org, Dennis Zhou <dennis@kernel.org>
>
> Hi Dennis,
>
> I love your patch! Perhaps something to improve:
>
> [auto build test WARNING on kdave/for-next]
> [also build test WARNING on next-20191121]
> [cannot apply to v5.4-rc8]
> [if your patch is applied to the wrong git tree, please drop us a note to help
> improve the system. BTW, we also suggest to use '--base' option to specify the
> base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
>
> url: https://github.com/0day-ci/linux/commits/Dennis-Zhou/btrfs-async-discard-support/20191121-230429
> base: https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-next
> config: arm64-defconfig (attached as .config)
> compiler: clang version 10.0.0 (git://gitmirror/llvm_project cf823ce4ad9d04c69b7c29d236f7b14c875111c2)
> reproduce:
> wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
> chmod +x ~/bin/make.cross
> # save the attached .config to linux build tree
> make.cross ARCH=arm64
>
> If you fix the issue, kindly add following tag
> Reported-by: kbuild test robot <lkp@intel.com>
>
> All warnings (new ones prefixed by >>):
>
> >> fs/btrfs/free-space-cache.c:3238:6: warning: variable 'trim_state' is used uninitialized whenever 'if' condition is false [-Wsometimes-uninitialized]
> if (!ret) {
> ^~~~
> fs/btrfs/free-space-cache.c:3251:53: note: uninitialized use occurs here
> __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
> ^~~~~~~~~~
> fs/btrfs/free-space-cache.c:3238:2: note: remove the 'if' if its condition is always true
> if (!ret) {
> ^~~~~~~~~~
> fs/btrfs/free-space-cache.c:3224:2: note: variable 'trim_state' is declared here
> enum btrfs_trim_state trim_state;
> ^
> 1 warning generated.
>
> vim +3238 fs/btrfs/free-space-cache.c
>
> 3210
> 3211 static int do_trimming(struct btrfs_block_group *block_group,
> 3212 u64 *total_trimmed, u64 start, u64 bytes,
> 3213 u64 reserved_start, u64 reserved_bytes,
> 3214 enum btrfs_trim_state reserved_trim_state,
> 3215 struct btrfs_trim_range *trim_entry)
> 3216 {
> 3217 struct btrfs_space_info *space_info = block_group->space_info;
> 3218 struct btrfs_fs_info *fs_info = block_group->fs_info;
> 3219 struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
> 3220 int ret;
> 3221 int update = 0;
> 3222 u64 end = start + bytes;
> 3223 u64 reserved_end = reserved_start + reserved_bytes;
> 3224 enum btrfs_trim_state trim_state;
> 3225 u64 trimmed = 0;
> 3226
> 3227 spin_lock(&space_info->lock);
> 3228 spin_lock(&block_group->lock);
> 3229 if (!block_group->ro) {
> 3230 block_group->reserved += reserved_bytes;
> 3231 space_info->bytes_reserved += reserved_bytes;
> 3232 update = 1;
> 3233 }
> 3234 spin_unlock(&block_group->lock);
> 3235 spin_unlock(&space_info->lock);
> 3236
> 3237 ret = btrfs_discard_extent(fs_info, start, bytes, &trimmed);
> > 3238 if (!ret) {
> 3239 *total_trimmed += trimmed;
> 3240 trim_state = BTRFS_TRIM_STATE_TRIMMED;
> 3241 }
> 3242
> 3243 mutex_lock(&ctl->cache_writeout_mutex);
> 3244 if (reserved_start < start)
> 3245 __btrfs_add_free_space(fs_info, ctl, reserved_start,
> 3246 start - reserved_start,
> 3247 reserved_trim_state);
> 3248 if (start + bytes < reserved_start + reserved_bytes)
> 3249 __btrfs_add_free_space(fs_info, ctl, end, reserved_end - end,
> 3250 reserved_trim_state);
> 3251 __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
^ oops
> 3252 list_del(&trim_entry->list);
> 3253 mutex_unlock(&ctl->cache_writeout_mutex);
> 3254
> 3255 if (update) {
> 3256 spin_lock(&space_info->lock);
> 3257 spin_lock(&block_group->lock);
> 3258 if (block_group->ro)
> 3259 space_info->bytes_readonly += reserved_bytes;
> 3260 block_group->reserved -= reserved_bytes;
> 3261 space_info->bytes_reserved -= reserved_bytes;
> 3262 spin_unlock(&block_group->lock);
> 3263 spin_unlock(&space_info->lock);
> 3264 }
> 3265
> 3266 return ret;
> 3267 }
> 3268
>
> ---
> 0-DAY kernel test infrastructure Open Source Technology Center
> https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org Intel Corporation
--
Thanks,
~Nick Desaulniers
^ permalink raw reply [flat|nested] 61+ messages in thread
* [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-11-20 21:50 [PATCH v3 00/22] btrfs: async discard support Dennis Zhou
@ 2019-11-20 21:51 ` Dennis Zhou
0 siblings, 0 replies; 61+ messages in thread
From: Dennis Zhou @ 2019-11-20 21:51 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
When discard is enabled, everytime a pinned extent is released back to
the block_group's free space cache, a discard is issued for the extent.
This is an overeager approach when it comes to discarding and helping
the SSD maintain enough free space to prevent severe garbage collection
situations.
This adds the beginning of async discard. Instead of issuing a discard
prior to returning it to the free space, it is just marked as untrimmed.
The block_group is then added to a LRU which then feeds into a workqueue
to issue discards at a much slower rate. Full discarding of unused block
groups is still done and will be address in a future patch in this
series.
For now, we don't persist the discard state of extents and bitmaps.
Therefore, our failure recovery mode will be to consider extents
untrimmed. This lets us handle failure and unmounting as one in the
same.
On a number of Facebook webservers, I collected data every minute
accounting the time we spent in btrfs_finish_extent_commit() (col. 1)
and in btrfs_commit_transaction() (col. 2). btrfs_finish_extent_commit()
is where we discard extents synchronously before returning them to the
free space cache.
discard=sync:
p99 total per minute p99 total per minute
Drive | extent_commit() (ms) | commit_trans() (ms)
---------------------------------------------------------------
Drive A | 434 | 1170
Drive B | 880 | 2330
Drive C | 2943 | 3920
Drive D | 4763 | 5701
discard=async:
p99 total per minute p99 total per minute
Drive | extent_commit() (ms) | commit_trans() (ms)
--------------------------------------------------------------
Drive A | 134 | 956
Drive B | 64 | 1972
Drive C | 59 | 1032
Drive D | 62 | 1200
While it's not great that the stats are cumulative over 1m, all of these
servers are running the same workload and and the delta between the two
are substantial. We are spending significantly less time in
btrfs_finish_extent_commit() which is responsible for discarding.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/Makefile | 2 +-
fs/btrfs/block-group.c | 4 +
fs/btrfs/block-group.h | 9 ++
fs/btrfs/ctree.h | 21 +++
fs/btrfs/discard.c | 273 ++++++++++++++++++++++++++++++++++++
fs/btrfs/discard.h | 26 ++++
fs/btrfs/disk-io.c | 15 +-
fs/btrfs/extent-tree.c | 4 +
fs/btrfs/free-space-cache.c | 35 ++++-
fs/btrfs/super.c | 35 ++++-
10 files changed, 414 insertions(+), 10 deletions(-)
create mode 100644 fs/btrfs/discard.c
create mode 100644 fs/btrfs/discard.h
diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
index 82200dbca5ac..9a0ff3384381 100644
--- a/fs/btrfs/Makefile
+++ b/fs/btrfs/Makefile
@@ -11,7 +11,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
uuid-tree.o props.o free-space-tree.o tree-checker.o space-info.o \
- block-rsv.o delalloc-space.o block-group.o
+ block-rsv.o delalloc-space.o block-group.o discard.o
btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c
index 6064be2d5556..d2bc46c365f4 100644
--- a/fs/btrfs/block-group.c
+++ b/fs/btrfs/block-group.c
@@ -14,6 +14,7 @@
#include "sysfs.h"
#include "tree-log.h"
#include "delalloc-space.h"
+#include "discard.h"
/*
* Return target flags in extended format or 0 if restripe for this chunk_type
@@ -1272,6 +1273,8 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
}
spin_unlock(&fs_info->unused_bgs_lock);
+ btrfs_discard_cancel_work(&fs_info->discard_ctl, block_group);
+
mutex_lock(&fs_info->delete_unused_bgs_mutex);
/* Don't want to race with allocators so take the groups_sem */
@@ -1617,6 +1620,7 @@ static struct btrfs_block_group *btrfs_create_block_group_cache(
INIT_LIST_HEAD(&cache->cluster_list);
INIT_LIST_HEAD(&cache->bg_list);
INIT_LIST_HEAD(&cache->ro_list);
+ INIT_LIST_HEAD(&cache->discard_list);
INIT_LIST_HEAD(&cache->dirty_list);
INIT_LIST_HEAD(&cache->io_list);
btrfs_init_free_space_ctl(cache);
diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h
index 9b409676c4b2..884defd61dcd 100644
--- a/fs/btrfs/block-group.h
+++ b/fs/btrfs/block-group.h
@@ -116,7 +116,11 @@ struct btrfs_block_group {
/* For read-only block groups */
struct list_head ro_list;
+ /* For discard operations */
atomic_t trimming;
+ struct list_head discard_list;
+ int discard_index;
+ u64 discard_eligible_time;
/* For dirty block groups */
struct list_head dirty_list;
@@ -158,6 +162,11 @@ struct btrfs_block_group {
struct btrfs_full_stripe_locks_tree full_stripe_locks_root;
};
+static inline u64 btrfs_block_group_end(struct btrfs_block_group *block_group)
+{
+ return (block_group->start + block_group->length);
+}
+
#ifdef CONFIG_BTRFS_DEBUG
static inline int btrfs_should_fragment_free_space(
struct btrfs_block_group *block_group)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 8ac3b2deef4a..d5ce8054f074 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -440,6 +440,21 @@ struct btrfs_full_stripe_locks_tree {
struct mutex lock;
};
+/* Discard control. */
+/*
+ * Async discard uses multiple lists to differentiate the discard filter
+ * parameters.
+ */
+#define BTRFS_NR_DISCARD_LISTS 1
+
+struct btrfs_discard_ctl {
+ struct workqueue_struct *discard_workers;
+ struct delayed_work work;
+ spinlock_t lock;
+ struct btrfs_block_group *block_group;
+ struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
+};
+
/* delayed seq elem */
struct seq_list {
struct list_head list;
@@ -526,6 +541,9 @@ enum {
* so we don't need to offload checksums to workqueues.
*/
BTRFS_FS_CSUM_IMPL_FAST,
+
+ /* Indicate that the discard workqueue can service discards. */
+ BTRFS_FS_DISCARD_RUNNING,
};
struct btrfs_fs_info {
@@ -816,6 +834,8 @@ struct btrfs_fs_info {
struct btrfs_workqueue *scrub_wr_completion_workers;
struct btrfs_workqueue *scrub_parity_workers;
+ struct btrfs_discard_ctl discard_ctl;
+
#ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY
u32 check_integrity_print_mask;
#endif
@@ -1189,6 +1209,7 @@ static inline u32 BTRFS_MAX_XATTR_SIZE(const struct btrfs_fs_info *info)
#define BTRFS_MOUNT_FREE_SPACE_TREE (1 << 26)
#define BTRFS_MOUNT_NOLOGREPLAY (1 << 27)
#define BTRFS_MOUNT_REF_VERIFY (1 << 28)
+#define BTRFS_MOUNT_DISCARD_ASYNC (1 << 29)
#define BTRFS_DEFAULT_COMMIT_INTERVAL (30)
#define BTRFS_DEFAULT_MAX_INLINE (2048)
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
new file mode 100644
index 000000000000..15d54de3d682
--- /dev/null
+++ b/fs/btrfs/discard.c
@@ -0,0 +1,273 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/jiffies.h>
+#include <linux/kernel.h>
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/sizes.h>
+#include <linux/workqueue.h>
+#include "ctree.h"
+#include "block-group.h"
+#include "discard.h"
+#include "free-space-cache.h"
+
+/* This is an initial delay to give some chance for lba reuse. */
+#define BTRFS_DISCARD_DELAY (120ULL * NSEC_PER_SEC)
+
+static struct list_head *btrfs_get_discard_list(
+ struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ return &discard_ctl->discard_list[block_group->discard_index];
+}
+
+void btrfs_add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ spin_lock(&discard_ctl->lock);
+
+ if (list_empty(&block_group->discard_list))
+ block_group->discard_eligible_time = (ktime_get_ns() +
+ BTRFS_DISCARD_DELAY);
+
+ list_move_tail(&block_group->discard_list,
+ btrfs_get_discard_list(discard_ctl, block_group));
+
+ spin_unlock(&discard_ctl->lock);
+}
+
+static bool remove_from_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ bool running = false;
+
+ spin_lock(&discard_ctl->lock);
+
+ if (block_group == discard_ctl->block_group) {
+ running = true;
+ discard_ctl->block_group = NULL;
+ }
+
+ block_group->discard_eligible_time = 0;
+ list_del_init(&block_group->discard_list);
+
+ spin_unlock(&discard_ctl->lock);
+
+ return running;
+}
+
+/**
+ * find_next_block_group - find block_group that's up next for discarding
+ * @discard_ctl: discard control
+ * @now: current time
+ *
+ * Iterate over the discard lists to find the next block_group up for
+ * discarding checking the discard_eligible_time of block_group.
+ */
+static struct btrfs_block_group *find_next_block_group(
+ struct btrfs_discard_ctl *discard_ctl,
+ u64 now)
+{
+ struct btrfs_block_group *ret_block_group = NULL, *block_group;
+ int i;
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++) {
+ struct list_head *discard_list = &discard_ctl->discard_list[i];
+
+ if (!list_empty(discard_list)) {
+ block_group = list_first_entry(discard_list,
+ struct btrfs_block_group,
+ discard_list);
+
+ if (!ret_block_group)
+ ret_block_group = block_group;
+
+ if (ret_block_group->discard_eligible_time < now)
+ break;
+
+ if (ret_block_group->discard_eligible_time >
+ block_group->discard_eligible_time)
+ ret_block_group = block_group;
+ }
+ }
+
+ return ret_block_group;
+}
+
+/**
+ * peek_discard_list - wrap find_next_block_group()
+ * @discard_ctl: discard control
+ *
+ * This wraps find_next_block_group() and sets the block_group to be in use.
+ */
+static struct btrfs_block_group *peek_discard_list(
+ struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_block_group *block_group;
+ u64 now = ktime_get_ns();
+
+ spin_lock(&discard_ctl->lock);
+
+ block_group = find_next_block_group(discard_ctl, now);
+
+ if (block_group && now < block_group->discard_eligible_time)
+ block_group = NULL;
+
+ discard_ctl->block_group = block_group;
+
+ spin_unlock(&discard_ctl->lock);
+
+ return block_group;
+}
+
+/**
+ * btrfs_discard_cancel_work - remove a block_group from the discard lists
+ * @discard_ctl: discard control
+ * @block_group: block_group of interest
+ *
+ * This removes @block_group from the discard lists. If necessary, it waits on
+ * the current work and then reschedules the delayed work.
+ */
+void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ if (remove_from_discard_list(discard_ctl, block_group)) {
+ cancel_delayed_work_sync(&discard_ctl->work);
+ btrfs_discard_schedule_work(discard_ctl, true);
+ }
+}
+
+/**
+ * btrfs_discard_queue_work - handles queuing the block_groups
+ * @discard_ctl: discard control
+ * @block_group: block_group of interest
+ *
+ * This maintains the LRU order of the discard lists.
+ */
+void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group)
+{
+ if (!block_group ||
+ !btrfs_test_opt(block_group->fs_info, DISCARD_ASYNC))
+ return;
+
+ btrfs_add_to_discard_list(discard_ctl, block_group);
+ if (!delayed_work_pending(&discard_ctl->work))
+ btrfs_discard_schedule_work(discard_ctl, false);
+}
+
+/**
+ * btrfs_discard_schedule_work - responsible for scheduling the discard work
+ * @discard_ctl: discard control
+ * @override: override the current timer
+ *
+ * Discards are issued by a delayed workqueue item. @override is used to
+ * update the current delay as the baseline delay interview is reevaluated
+ * on transaction commit. This is also maxed with any other rate limit.
+ */
+void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
+ bool override)
+{
+ struct btrfs_block_group *block_group;
+ u64 now = ktime_get_ns();
+
+ spin_lock(&discard_ctl->lock);
+
+ if (!btrfs_run_discard_work(discard_ctl))
+ goto out;
+
+ if (!override && delayed_work_pending(&discard_ctl->work))
+ goto out;
+
+ block_group = find_next_block_group(discard_ctl, now);
+ if (block_group) {
+ u64 delay = 0;
+
+ if (now < block_group->discard_eligible_time)
+ delay = nsecs_to_jiffies(
+ block_group->discard_eligible_time - now);
+
+ mod_delayed_work(discard_ctl->discard_workers,
+ &discard_ctl->work,
+ delay);
+ }
+
+out:
+ spin_unlock(&discard_ctl->lock);
+}
+
+/**
+ * btrfs_discard_workfn - discard work function
+ * @work: work
+ *
+ * This finds the next block_group to start discarding and then discards it.
+ */
+static void btrfs_discard_workfn(struct work_struct *work)
+{
+ struct btrfs_discard_ctl *discard_ctl;
+ struct btrfs_block_group *block_group;
+ u64 trimmed = 0;
+
+ discard_ctl = container_of(work, struct btrfs_discard_ctl, work.work);
+
+ block_group = peek_discard_list(discard_ctl);
+ if (!block_group || !btrfs_run_discard_work(discard_ctl))
+ return;
+
+ btrfs_trim_block_group(block_group, &trimmed, block_group->start,
+ btrfs_block_group_end(block_group), 0);
+
+ remove_from_discard_list(discard_ctl, block_group);
+
+ btrfs_discard_schedule_work(discard_ctl, false);
+}
+
+/**
+ * btrfs_run_discard_work - determines if async discard should be running
+ * @discard_ctl: discard control
+ *
+ * Checks if the file system is writeable and BTRFS_FS_DISCARD_RUNNING is set.
+ */
+bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_fs_info *fs_info = container_of(discard_ctl,
+ struct btrfs_fs_info,
+ discard_ctl);
+
+ return (!(fs_info->sb->s_flags & SB_RDONLY) &&
+ test_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags));
+}
+
+void btrfs_discard_resume(struct btrfs_fs_info *fs_info)
+{
+ if (!btrfs_test_opt(fs_info, DISCARD_ASYNC)) {
+ btrfs_discard_cleanup(fs_info);
+ return;
+ }
+
+ set_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
+}
+
+void btrfs_discard_stop(struct btrfs_fs_info *fs_info)
+{
+ clear_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
+}
+
+void btrfs_discard_init(struct btrfs_fs_info *fs_info)
+{
+ struct btrfs_discard_ctl *discard_ctl = &fs_info->discard_ctl;
+ int i;
+
+ spin_lock_init(&discard_ctl->lock);
+
+ INIT_DELAYED_WORK(&discard_ctl->work, btrfs_discard_workfn);
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++)
+ INIT_LIST_HEAD(&discard_ctl->discard_list[i]);
+}
+
+void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
+{
+ btrfs_discard_stop(fs_info);
+ cancel_delayed_work_sync(&fs_info->discard_ctl.work);
+}
diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
new file mode 100644
index 000000000000..439ca8c51877
--- /dev/null
+++ b/fs/btrfs/discard.h
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#ifndef BTRFS_DISCARD_H
+#define BTRFS_DISCARD_H
+
+struct btrfs_fs_info;
+struct btrfs_discard_ctl;
+struct btrfs_block_group;
+
+void btrfs_add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group);
+
+void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group);
+void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group *block_group);
+void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
+ bool override);
+bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl);
+
+void btrfs_discard_resume(struct btrfs_fs_info *fs_info);
+void btrfs_discard_stop(struct btrfs_fs_info *fs_info);
+void btrfs_discard_init(struct btrfs_fs_info *fs_info);
+void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info);
+
+#endif
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index e0edfdc9c82b..8785b5c999b0 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -41,6 +41,7 @@
#include "tree-checker.h"
#include "ref-verify.h"
#include "block-group.h"
+#include "discard.h"
#define BTRFS_SUPER_FLAG_SUPP (BTRFS_HEADER_FLAG_WRITTEN |\
BTRFS_HEADER_FLAG_RELOC |\
@@ -1953,6 +1954,8 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info)
btrfs_destroy_workqueue(fs_info->readahead_workers);
btrfs_destroy_workqueue(fs_info->flush_workers);
btrfs_destroy_workqueue(fs_info->qgroup_rescan_workers);
+ if (fs_info->discard_ctl.discard_workers)
+ destroy_workqueue(fs_info->discard_ctl.discard_workers);
/*
* Now that all other work queues are destroyed, we can safely destroy
* the queues used for metadata I/O, since tasks from those other work
@@ -2148,6 +2151,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info,
max_active, 2);
fs_info->qgroup_rescan_workers =
btrfs_alloc_workqueue(fs_info, "qgroup-rescan", flags, 1, 0);
+ fs_info->discard_ctl.discard_workers =
+ alloc_workqueue("btrfs_discard", WQ_UNBOUND | WQ_FREEZABLE, 1);
if (!(fs_info->workers && fs_info->delalloc_workers &&
fs_info->flush_workers &&
@@ -2158,7 +2163,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info,
fs_info->endio_freespace_worker && fs_info->rmw_workers &&
fs_info->caching_workers && fs_info->readahead_workers &&
fs_info->fixup_workers && fs_info->delayed_workers &&
- fs_info->qgroup_rescan_workers)) {
+ fs_info->qgroup_rescan_workers &&
+ fs_info->discard_ctl.discard_workers)) {
return -ENOMEM;
}
@@ -2793,6 +2799,8 @@ int __cold open_ctree(struct super_block *sb,
btrfs_init_dev_replace_locks(fs_info);
btrfs_init_qgroup(fs_info);
+ btrfs_discard_init(fs_info);
+
btrfs_init_free_cluster(&fs_info->meta_alloc_cluster);
btrfs_init_free_cluster(&fs_info->data_alloc_cluster);
@@ -3263,6 +3271,8 @@ int __cold open_ctree(struct super_block *sb,
btrfs_qgroup_rescan_resume(fs_info);
+ btrfs_discard_resume(fs_info);
+
if (!fs_info->uuid_root) {
btrfs_info(fs_info, "creating UUID tree");
ret = btrfs_create_uuid_tree(fs_info);
@@ -3954,6 +3964,9 @@ void __cold close_ctree(struct btrfs_fs_info *fs_info)
*/
kthread_park(fs_info->cleaner_kthread);
+ /* cancel or finish ongoing work */
+ btrfs_discard_cleanup(fs_info);
+
/* wait for the qgroup rescan worker to stop */
btrfs_qgroup_wait_for_completion(fs_info, false);
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index e1ff2f115182..40e965b06f8f 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -32,6 +32,7 @@
#include "block-rsv.h"
#include "delalloc-space.h"
#include "block-group.h"
+#include "discard.h"
#undef SCRAMBLE_DELAYED_REFS
@@ -2940,6 +2941,9 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
cond_resched();
}
+ if (btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_schedule_work(&fs_info->discard_ctl, true);
+
/*
* Transaction is finished. We don't need the lock anymore. We
* do need to clean up the block groups in case of a transaction
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 946a7df33249..72933996e743 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -21,6 +21,7 @@
#include "space-info.h"
#include "delalloc-space.h"
#include "block-group.h"
+#include "discard.h"
#define BITS_PER_BITMAP (PAGE_SIZE * 8UL)
#define MAX_CACHE_BYTES_PER_GIG SZ_32K
@@ -755,9 +756,11 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
/*
* Sync discard ensures that the free space cache is always
* trimmed. So when reading this in, the state should reflect
- * that.
+ * that. We also do this for async as a stop gap for lack of
+ * persistence.
*/
- if (btrfs_test_opt(fs_info, DISCARD_SYNC))
+ if (btrfs_test_opt(fs_info, DISCARD_SYNC) ||
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
e->trim_state = BTRFS_TRIM_STATE_TRIMMED;
if (!e->bytes) {
@@ -2382,6 +2385,7 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
u64 offset, u64 bytes,
enum btrfs_trim_state trim_state)
{
+ struct btrfs_block_group *block_group = ctl->private;
struct btrfs_free_space *info;
int ret = 0;
@@ -2431,6 +2435,9 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
ASSERT(ret != -EEXIST);
}
+ if (trim_state != BTRFS_TRIM_STATE_TRIMMED)
+ btrfs_discard_queue_work(&fs_info->discard_ctl, block_group);
+
return ret;
}
@@ -3204,6 +3211,7 @@ void btrfs_init_free_cluster(struct btrfs_free_cluster *cluster)
static int do_trimming(struct btrfs_block_group *block_group,
u64 *total_trimmed, u64 start, u64 bytes,
u64 reserved_start, u64 reserved_bytes,
+ enum btrfs_trim_state reserved_trim_state,
struct btrfs_trim_range *trim_entry)
{
struct btrfs_space_info *space_info = block_group->space_info;
@@ -3211,6 +3219,9 @@ static int do_trimming(struct btrfs_block_group *block_group,
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
int ret;
int update = 0;
+ u64 end = start + bytes;
+ u64 reserved_end = reserved_start + reserved_bytes;
+ enum btrfs_trim_state trim_state;
u64 trimmed = 0;
spin_lock(&space_info->lock);
@@ -3224,11 +3235,20 @@ static int do_trimming(struct btrfs_block_group *block_group,
spin_unlock(&space_info->lock);
ret = btrfs_discard_extent(fs_info, start, bytes, &trimmed);
- if (!ret)
+ if (!ret) {
*total_trimmed += trimmed;
+ trim_state = BTRFS_TRIM_STATE_TRIMMED;
+ }
mutex_lock(&ctl->cache_writeout_mutex);
- btrfs_add_free_space(block_group, reserved_start, reserved_bytes);
+ if (reserved_start < start)
+ __btrfs_add_free_space(fs_info, ctl, reserved_start,
+ start - reserved_start,
+ reserved_trim_state);
+ if (start + bytes < reserved_start + reserved_bytes)
+ __btrfs_add_free_space(fs_info, ctl, end, reserved_end - end,
+ reserved_trim_state);
+ __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
list_del(&trim_entry->list);
mutex_unlock(&ctl->cache_writeout_mutex);
@@ -3255,6 +3275,7 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
int ret = 0;
u64 extent_start;
u64 extent_bytes;
+ enum btrfs_trim_state extent_trim_state;
u64 bytes;
while (start < end) {
@@ -3296,6 +3317,7 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
extent_start = entry->offset;
extent_bytes = entry->bytes;
+ extent_trim_state = entry->trim_state;
start = max(start, extent_start);
bytes = min(extent_start + extent_bytes, end) - start;
if (bytes < minlen) {
@@ -3314,7 +3336,8 @@ static int trim_no_bitmap(struct btrfs_block_group *block_group,
mutex_unlock(&ctl->cache_writeout_mutex);
ret = do_trimming(block_group, total_trimmed, start, bytes,
- extent_start, extent_bytes, &trim_entry);
+ extent_start, extent_bytes, extent_trim_state,
+ &trim_entry);
if (ret)
break;
next:
@@ -3440,7 +3463,7 @@ static int trim_bitmaps(struct btrfs_block_group *block_group,
mutex_unlock(&ctl->cache_writeout_mutex);
ret = do_trimming(block_group, total_trimmed, start, bytes,
- start, bytes, &trim_entry);
+ start, bytes, 0, &trim_entry);
if (ret) {
reset_trimming_bitmap(ctl, offset);
break;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index f131fb9f0f69..0edf00753c00 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -46,6 +46,7 @@
#include "sysfs.h"
#include "tests/btrfs-tests.h"
#include "block-group.h"
+#include "discard.h"
#include "qgroup.h"
#define CREATE_TRACE_POINTS
@@ -146,6 +147,8 @@ void __btrfs_handle_fs_error(struct btrfs_fs_info *fs_info, const char *function
if (sb_rdonly(sb))
return;
+ btrfs_discard_stop(fs_info);
+
/* btrfs handle error by forcing the filesystem readonly */
sb->s_flags |= SB_RDONLY;
btrfs_info(fs_info, "forced readonly");
@@ -313,6 +316,7 @@ enum {
Opt_datasum, Opt_nodatasum,
Opt_defrag, Opt_nodefrag,
Opt_discard, Opt_nodiscard,
+ Opt_discard_mode,
Opt_nologreplay,
Opt_norecovery,
Opt_ratio,
@@ -376,6 +380,7 @@ static const match_table_t tokens = {
{Opt_nodefrag, "noautodefrag"},
{Opt_discard, "discard"},
{Opt_nodiscard, "nodiscard"},
+ {Opt_discard_mode, "discard=%s"},
{Opt_nologreplay, "nologreplay"},
{Opt_norecovery, "norecovery"},
{Opt_ratio, "metadata_ratio=%u"},
@@ -695,12 +700,26 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
info->metadata_ratio);
break;
case Opt_discard:
- btrfs_set_and_info(info, DISCARD_SYNC,
- "turning on sync discard");
+ case Opt_discard_mode:
+ if (token == Opt_discard ||
+ strcmp(args[0].from, "sync") == 0) {
+ btrfs_clear_opt(info->mount_opt, DISCARD_ASYNC);
+ btrfs_set_and_info(info, DISCARD_SYNC,
+ "turning on sync discard");
+ } else if (strcmp(args[0].from, "async") == 0) {
+ btrfs_clear_opt(info->mount_opt, DISCARD_SYNC);
+ btrfs_set_and_info(info, DISCARD_ASYNC,
+ "turning on async discard");
+ } else {
+ ret = -EINVAL;
+ goto out;
+ }
break;
case Opt_nodiscard:
btrfs_clear_and_info(info, DISCARD_SYNC,
"turning off discard");
+ btrfs_clear_and_info(info, DISCARD_ASYNC,
+ "turning off async discard");
break;
case Opt_space_cache:
case Opt_space_cache_version:
@@ -1324,6 +1343,8 @@ static int btrfs_show_options(struct seq_file *seq, struct dentry *dentry)
seq_puts(seq, ",flushoncommit");
if (btrfs_test_opt(info, DISCARD_SYNC))
seq_puts(seq, ",discard");
+ if (btrfs_test_opt(info, DISCARD_ASYNC))
+ seq_puts(seq, ",discard=async");
if (!(info->sb->s_flags & SB_POSIXACL))
seq_puts(seq, ",noacl");
if (btrfs_test_opt(info, SPACE_CACHE))
@@ -1713,6 +1734,14 @@ static inline void btrfs_remount_cleanup(struct btrfs_fs_info *fs_info,
btrfs_cleanup_defrag_inodes(fs_info);
}
+ /* If we toggled discard async. */
+ if (!btrfs_raw_test_opt(old_opts, DISCARD_ASYNC) &&
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_resume(fs_info);
+ else if (btrfs_raw_test_opt(old_opts, DISCARD_ASYNC) &&
+ !btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_cleanup(fs_info);
+
clear_bit(BTRFS_FS_STATE_REMOUNTING, &fs_info->fs_state);
}
@@ -1760,6 +1789,8 @@ static int btrfs_remount(struct super_block *sb, int *flags, char *data)
*/
cancel_work_sync(&fs_info->async_reclaim_work);
+ btrfs_discard_cleanup(fs_info);
+
/* wait for the uuid_scan task to finish */
down(&fs_info->uuid_tree_rescan_sem);
/* avoid complains from lockdep et al. */
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
* Re: [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-10-23 22:52 ` [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue Dennis Zhou
@ 2019-11-11 18:49 ` David Sterba
0 siblings, 0 replies; 61+ messages in thread
From: David Sterba @ 2019-11-11 18:49 UTC (permalink / raw)
To: Dennis Zhou
Cc: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval,
kernel-team, linux-btrfs
On Wed, Oct 23, 2019 at 06:52:59PM -0400, Dennis Zhou wrote:
> --- /dev/null
> +++ b/fs/btrfs/discard.c
> @@ -0,0 +1,274 @@
> +/*
> + * Copyright (C) 2019 Facebook. All rights reserved.
> + */
It's the other way around, SPDX should be there,
https://btrfs.wiki.kernel.org/index.php/Developer%27s_FAQ#Copyright_notices_in_files.2C_SPDX
^ permalink raw reply [flat|nested] 61+ messages in thread
* [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue
2019-10-23 22:52 [PATCH v2 00/22] btrfs: async discard support Dennis Zhou
@ 2019-10-23 22:52 ` Dennis Zhou
2019-11-11 18:49 ` David Sterba
0 siblings, 1 reply; 61+ messages in thread
From: Dennis Zhou @ 2019-10-23 22:52 UTC (permalink / raw)
To: David Sterba, Chris Mason, Josef Bacik, Omar Sandoval
Cc: kernel-team, linux-btrfs, Dennis Zhou
When discard is enabled, everytime a pinned extent is released back to
the block_group's free space cache, a discard is issued for the extent.
This is an overeager approach when it comes to discarding and helping
the SSD maintain enough free space to prevent severe garbage collection
situations.
This adds the beginning of async discard. Instead of issuing a discard
prior to returning it to the free space, it is just marked as untrimmed.
The block_group is then added to a LRU which then feeds into a workqueue
to issue discards at a much slower rate. Full discarding of unused block
groups is still done and will be address in a future patch in this
series.
For now, we don't persist the discard state of extents and bitmaps.
Therefore, our failure recovery mode will be to consider extents
untrimmed. This lets us handle failure and unmounting as one in the
same.
Signed-off-by: Dennis Zhou <dennis@kernel.org>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
fs/btrfs/Makefile | 2 +-
fs/btrfs/block-group.c | 4 +
fs/btrfs/block-group.h | 9 ++
fs/btrfs/ctree.h | 21 +++
fs/btrfs/discard.c | 274 ++++++++++++++++++++++++++++++++++++
fs/btrfs/discard.h | 28 ++++
fs/btrfs/disk-io.c | 15 +-
fs/btrfs/extent-tree.c | 4 +
fs/btrfs/free-space-cache.c | 35 ++++-
fs/btrfs/super.c | 35 ++++-
10 files changed, 417 insertions(+), 10 deletions(-)
create mode 100644 fs/btrfs/discard.c
create mode 100644 fs/btrfs/discard.h
diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
index 82200dbca5ac..9a0ff3384381 100644
--- a/fs/btrfs/Makefile
+++ b/fs/btrfs/Makefile
@@ -11,7 +11,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
uuid-tree.o props.o free-space-tree.o tree-checker.o space-info.o \
- block-rsv.o delalloc-space.o block-group.o
+ block-rsv.o delalloc-space.o block-group.o discard.o
btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c
index afe86028246a..8bbbe7488328 100644
--- a/fs/btrfs/block-group.c
+++ b/fs/btrfs/block-group.c
@@ -14,6 +14,7 @@
#include "sysfs.h"
#include "tree-log.h"
#include "delalloc-space.h"
+#include "discard.h"
/*
* Return target flags in extended format or 0 if restripe for this chunk_type
@@ -1273,6 +1274,8 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
}
spin_unlock(&fs_info->unused_bgs_lock);
+ btrfs_discard_cancel_work(&fs_info->discard_ctl, block_group);
+
mutex_lock(&fs_info->delete_unused_bgs_mutex);
/* Don't want to race with allocators so take the groups_sem */
@@ -1622,6 +1625,7 @@ static struct btrfs_block_group_cache *btrfs_create_block_group_cache(
INIT_LIST_HEAD(&cache->cluster_list);
INIT_LIST_HEAD(&cache->bg_list);
INIT_LIST_HEAD(&cache->ro_list);
+ INIT_LIST_HEAD(&cache->discard_list);
INIT_LIST_HEAD(&cache->dirty_list);
INIT_LIST_HEAD(&cache->io_list);
btrfs_init_free_space_ctl(cache);
diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h
index c391800388dd..633dce5b9d57 100644
--- a/fs/btrfs/block-group.h
+++ b/fs/btrfs/block-group.h
@@ -115,7 +115,11 @@ struct btrfs_block_group_cache {
/* For read-only block groups */
struct list_head ro_list;
+ /* For discard operations */
atomic_t trimming;
+ struct list_head discard_list;
+ int discard_index;
+ u64 discard_eligible_time;
/* For dirty block groups */
struct list_head dirty_list;
@@ -157,6 +161,11 @@ struct btrfs_block_group_cache {
struct btrfs_full_stripe_locks_tree full_stripe_locks_root;
};
+static inline u64 btrfs_block_group_end(struct btrfs_block_group_cache *cache)
+{
+ return (cache->key.objectid + cache->key.offset);
+}
+
#ifdef CONFIG_BTRFS_DEBUG
static inline int btrfs_should_fragment_free_space(
struct btrfs_block_group_cache *block_group)
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 1877586576aa..efa8390e8419 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -438,6 +438,21 @@ struct btrfs_full_stripe_locks_tree {
struct mutex lock;
};
+/* Discard control. */
+/*
+ * Async discard uses multiple lists to differentiate the discard filter
+ * parameters.
+ */
+#define BTRFS_NR_DISCARD_LISTS 1
+
+struct btrfs_discard_ctl {
+ struct workqueue_struct *discard_workers;
+ struct delayed_work work;
+ spinlock_t lock;
+ struct btrfs_block_group_cache *cache;
+ struct list_head discard_list[BTRFS_NR_DISCARD_LISTS];
+};
+
/* delayed seq elem */
struct seq_list {
struct list_head list;
@@ -524,6 +539,9 @@ enum {
* so we don't need to offload checksums to workqueues.
*/
BTRFS_FS_CSUM_IMPL_FAST,
+
+ /* Indicate that the discard workqueue can service discards. */
+ BTRFS_FS_DISCARD_RUNNING,
};
struct btrfs_fs_info {
@@ -817,6 +835,8 @@ struct btrfs_fs_info {
struct btrfs_workqueue *scrub_wr_completion_workers;
struct btrfs_workqueue *scrub_parity_workers;
+ struct btrfs_discard_ctl discard_ctl;
+
#ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY
u32 check_integrity_print_mask;
#endif
@@ -1190,6 +1210,7 @@ static inline u32 BTRFS_MAX_XATTR_SIZE(const struct btrfs_fs_info *info)
#define BTRFS_MOUNT_FREE_SPACE_TREE (1 << 26)
#define BTRFS_MOUNT_NOLOGREPLAY (1 << 27)
#define BTRFS_MOUNT_REF_VERIFY (1 << 28)
+#define BTRFS_MOUNT_DISCARD_ASYNC (1 << 29)
#define BTRFS_DEFAULT_COMMIT_INTERVAL (30)
#define BTRFS_DEFAULT_MAX_INLINE (2048)
diff --git a/fs/btrfs/discard.c b/fs/btrfs/discard.c
new file mode 100644
index 000000000000..0a72a1902ca6
--- /dev/null
+++ b/fs/btrfs/discard.c
@@ -0,0 +1,274 @@
+/*
+ * Copyright (C) 2019 Facebook. All rights reserved.
+ */
+
+#include <linux/jiffies.h>
+#include <linux/kernel.h>
+#include <linux/ktime.h>
+#include <linux/list.h>
+#include <linux/sizes.h>
+#include <linux/workqueue.h>
+#include "ctree.h"
+#include "block-group.h"
+#include "discard.h"
+#include "free-space-cache.h"
+
+/* This is an initial delay to give some chance for lba reuse. */
+#define BTRFS_DISCARD_DELAY (120ULL * NSEC_PER_SEC)
+
+static struct list_head *btrfs_get_discard_list(
+ struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group_cache *cache)
+{
+ return &discard_ctl->discard_list[cache->discard_index];
+}
+
+void btrfs_add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group_cache *cache)
+{
+ spin_lock(&discard_ctl->lock);
+
+ if (list_empty(&cache->discard_list))
+ cache->discard_eligible_time = (ktime_get_ns() +
+ BTRFS_DISCARD_DELAY);
+
+ list_move_tail(&cache->discard_list,
+ btrfs_get_discard_list(discard_ctl, cache));
+
+ spin_unlock(&discard_ctl->lock);
+}
+
+static bool remove_from_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group_cache *cache)
+{
+ bool running = false;
+
+ spin_lock(&discard_ctl->lock);
+
+ if (cache == discard_ctl->cache) {
+ running = true;
+ discard_ctl->cache = NULL;
+ }
+
+ cache->discard_eligible_time = 0;
+ list_del_init(&cache->discard_list);
+
+ spin_unlock(&discard_ctl->lock);
+
+ return running;
+}
+
+/**
+ * find_next_cache - find cache that's up next for discarding
+ * @discard_ctl: discard control
+ * @now: current time
+ *
+ * Iterate over the discard lists to find the next block_group up for
+ * discarding checking the discard_eligible_time of block_group.
+ */
+static struct btrfs_block_group_cache *find_next_cache(
+ struct btrfs_discard_ctl *discard_ctl,
+ u64 now)
+{
+ struct btrfs_block_group_cache *ret_cache = NULL, *cache;
+ int i;
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++) {
+ struct list_head *discard_list = &discard_ctl->discard_list[i];
+
+ if (!list_empty(discard_list)) {
+ cache = list_first_entry(discard_list,
+ struct btrfs_block_group_cache,
+ discard_list);
+
+ if (!ret_cache)
+ ret_cache = cache;
+
+ if (ret_cache->discard_eligible_time < now)
+ break;
+
+ if (ret_cache->discard_eligible_time >
+ cache->discard_eligible_time)
+ ret_cache = cache;
+ }
+ }
+
+ return ret_cache;
+}
+
+/**
+ * peek_discard_list - wrap find_next_cache()
+ * @discard_ctl: discard control
+ *
+ * This wraps find_next_cache() and sets the cache to be in use.
+ */
+static struct btrfs_block_group_cache *peek_discard_list(
+ struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_block_group_cache *cache;
+ u64 now = ktime_get_ns();
+
+ spin_lock(&discard_ctl->lock);
+
+ cache = find_next_cache(discard_ctl, now);
+
+ if (cache && now < cache->discard_eligible_time)
+ cache = NULL;
+
+ discard_ctl->cache = cache;
+
+ spin_unlock(&discard_ctl->lock);
+
+ return cache;
+}
+
+/**
+ * btrfs_discard_cancel_work - remove a block_group from the discard lists
+ * @discard_ctl: discard control
+ * @cache: block_group of interest
+ *
+ * This removes @cache from the discard lists. If necessary, it waits on the
+ * current work and then reschedules the delayed work.
+ */
+void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group_cache *cache)
+{
+ if (remove_from_discard_list(discard_ctl, cache)) {
+ cancel_delayed_work_sync(&discard_ctl->work);
+ btrfs_discard_schedule_work(discard_ctl, true);
+ }
+}
+
+/**
+ * btrfs_discard_queue_work - handles queuing the block_groups
+ * @discard_ctl: discard control
+ * @cache: block_group of interest
+ *
+ * This maintains the LRU order of the discard lists.
+ */
+void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group_cache *cache)
+{
+ if (!cache || !btrfs_test_opt(cache->fs_info, DISCARD_ASYNC))
+ return;
+
+ btrfs_add_to_discard_list(discard_ctl, cache);
+ if (!delayed_work_pending(&discard_ctl->work))
+ btrfs_discard_schedule_work(discard_ctl, false);
+}
+
+/**
+ * btrfs_discard_schedule_work - responsible for scheduling the discard work
+ * @discard_ctl: discard control
+ * @override: override the current timer
+ *
+ * Discards are issued by a delayed workqueue item. @override is used to
+ * update the current delay as the baseline delay interview is reevaluated
+ * on transaction commit. This is also maxed with any other rate limit.
+ */
+void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
+ bool override)
+{
+ struct btrfs_block_group_cache *cache;
+ u64 now = ktime_get_ns();
+
+ spin_lock(&discard_ctl->lock);
+
+ if (!btrfs_run_discard_work(discard_ctl))
+ goto out;
+
+ if (!override && delayed_work_pending(&discard_ctl->work))
+ goto out;
+
+ cache = find_next_cache(discard_ctl, now);
+ if (cache) {
+ u64 delay = 0;
+
+ if (now < cache->discard_eligible_time)
+ delay = nsecs_to_jiffies(cache->discard_eligible_time -
+ now);
+
+ mod_delayed_work(discard_ctl->discard_workers,
+ &discard_ctl->work,
+ delay);
+ }
+
+out:
+ spin_unlock(&discard_ctl->lock);
+}
+
+/**
+ * btrfs_discard_workfn - discard work function
+ * @work: work
+ *
+ * This finds the next cache to start discarding and then discards it.
+ */
+static void btrfs_discard_workfn(struct work_struct *work)
+{
+ struct btrfs_discard_ctl *discard_ctl;
+ struct btrfs_block_group_cache *cache;
+ u64 trimmed = 0;
+
+ discard_ctl = container_of(work, struct btrfs_discard_ctl, work.work);
+
+ cache = peek_discard_list(discard_ctl);
+ if (!cache || !btrfs_run_discard_work(discard_ctl))
+ return;
+
+ btrfs_trim_block_group(cache, &trimmed, cache->key.objectid,
+ btrfs_block_group_end(cache), 0);
+
+ remove_from_discard_list(discard_ctl, cache);
+
+ btrfs_discard_schedule_work(discard_ctl, false);
+}
+
+/**
+ * btrfs_run_discard_work - determines if async discard should be running
+ * @discard_ctl: discard control
+ *
+ * Checks if the file system is writeable and BTRFS_FS_DISCARD_RUNNING is set.
+ */
+bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl)
+{
+ struct btrfs_fs_info *fs_info = container_of(discard_ctl,
+ struct btrfs_fs_info,
+ discard_ctl);
+
+ return (!(fs_info->sb->s_flags & SB_RDONLY) &&
+ test_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags));
+}
+
+void btrfs_discard_resume(struct btrfs_fs_info *fs_info)
+{
+ if (!btrfs_test_opt(fs_info, DISCARD_ASYNC)) {
+ btrfs_discard_cleanup(fs_info);
+ return;
+ }
+
+ set_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
+}
+
+void btrfs_discard_stop(struct btrfs_fs_info *fs_info)
+{
+ clear_bit(BTRFS_FS_DISCARD_RUNNING, &fs_info->flags);
+}
+
+void btrfs_discard_init(struct btrfs_fs_info *fs_info)
+{
+ struct btrfs_discard_ctl *discard_ctl = &fs_info->discard_ctl;
+ int i;
+
+ spin_lock_init(&discard_ctl->lock);
+
+ INIT_DELAYED_WORK(&discard_ctl->work, btrfs_discard_workfn);
+
+ for (i = 0; i < BTRFS_NR_DISCARD_LISTS; i++)
+ INIT_LIST_HEAD(&discard_ctl->discard_list[i]);
+}
+
+void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info)
+{
+ btrfs_discard_stop(fs_info);
+ cancel_delayed_work_sync(&fs_info->discard_ctl.work);
+}
diff --git a/fs/btrfs/discard.h b/fs/btrfs/discard.h
new file mode 100644
index 000000000000..48b4710a80d0
--- /dev/null
+++ b/fs/btrfs/discard.h
@@ -0,0 +1,28 @@
+/*
+ * Copyright (C) 2019 Facebook. All rights reserved.
+ */
+
+#ifndef BTRFS_DISCARD_H
+#define BTRFS_DISCARD_H
+
+struct btrfs_fs_info;
+struct btrfs_discard_ctl;
+struct btrfs_block_group_cache;
+
+void btrfs_add_to_discard_list(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group_cache *cache);
+
+void btrfs_discard_cancel_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group_cache *cache);
+void btrfs_discard_queue_work(struct btrfs_discard_ctl *discard_ctl,
+ struct btrfs_block_group_cache *cache);
+void btrfs_discard_schedule_work(struct btrfs_discard_ctl *discard_ctl,
+ bool override);
+bool btrfs_run_discard_work(struct btrfs_discard_ctl *discard_ctl);
+
+void btrfs_discard_resume(struct btrfs_fs_info *fs_info);
+void btrfs_discard_stop(struct btrfs_fs_info *fs_info);
+void btrfs_discard_init(struct btrfs_fs_info *fs_info);
+void btrfs_discard_cleanup(struct btrfs_fs_info *fs_info);
+
+#endif
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 044981cf6df9..a304ec972f67 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -41,6 +41,7 @@
#include "tree-checker.h"
#include "ref-verify.h"
#include "block-group.h"
+#include "discard.h"
#define BTRFS_SUPER_FLAG_SUPP (BTRFS_HEADER_FLAG_WRITTEN |\
BTRFS_HEADER_FLAG_RELOC |\
@@ -2009,6 +2010,8 @@ static void btrfs_stop_all_workers(struct btrfs_fs_info *fs_info)
btrfs_destroy_workqueue(fs_info->flush_workers);
btrfs_destroy_workqueue(fs_info->qgroup_rescan_workers);
btrfs_destroy_workqueue(fs_info->extent_workers);
+ if (fs_info->discard_ctl.discard_workers)
+ destroy_workqueue(fs_info->discard_ctl.discard_workers);
/*
* Now that all other work queues are destroyed, we can safely destroy
* the queues used for metadata I/O, since tasks from those other work
@@ -2218,6 +2221,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info,
btrfs_alloc_workqueue(fs_info, "extent-refs", flags,
min_t(u64, fs_devices->num_devices,
max_active), 8);
+ fs_info->discard_ctl.discard_workers =
+ alloc_workqueue("btrfs_discard", WQ_UNBOUND | WQ_FREEZABLE, 1);
if (!(fs_info->workers && fs_info->delalloc_workers &&
fs_info->submit_workers && fs_info->flush_workers &&
@@ -2229,7 +2234,8 @@ static int btrfs_init_workqueues(struct btrfs_fs_info *fs_info,
fs_info->caching_workers && fs_info->readahead_workers &&
fs_info->fixup_workers && fs_info->delayed_workers &&
fs_info->extent_workers &&
- fs_info->qgroup_rescan_workers)) {
+ fs_info->qgroup_rescan_workers &&
+ fs_info->discard_ctl.discard_workers)) {
return -ENOMEM;
}
@@ -2772,6 +2778,8 @@ int open_ctree(struct super_block *sb,
btrfs_init_dev_replace_locks(fs_info);
btrfs_init_qgroup(fs_info);
+ btrfs_discard_init(fs_info);
+
btrfs_init_free_cluster(&fs_info->meta_alloc_cluster);
btrfs_init_free_cluster(&fs_info->data_alloc_cluster);
@@ -3284,6 +3292,8 @@ int open_ctree(struct super_block *sb,
btrfs_qgroup_rescan_resume(fs_info);
+ btrfs_discard_resume(fs_info);
+
if (!fs_info->uuid_root) {
btrfs_info(fs_info, "creating UUID tree");
ret = btrfs_create_uuid_tree(fs_info);
@@ -3993,6 +4003,9 @@ void close_ctree(struct btrfs_fs_info *fs_info)
*/
kthread_park(fs_info->cleaner_kthread);
+ /* cancel or finish ongoing work */
+ btrfs_discard_cleanup(fs_info);
+
/* wait for the qgroup rescan worker to stop */
btrfs_qgroup_wait_for_completion(fs_info, false);
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 6a40bba3cb19..de00fd6e338b 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -32,6 +32,7 @@
#include "block-rsv.h"
#include "delalloc-space.h"
#include "block-group.h"
+#include "discard.h"
#undef SCRAMBLE_DELAYED_REFS
@@ -2920,6 +2921,9 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
cond_resched();
}
+ if (btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_schedule_work(&fs_info->discard_ctl, true);
+
/*
* Transaction is finished. We don't need the lock anymore. We
* do need to clean up the block groups in case of a transaction
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index 900b935e5997..8120630e4439 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -21,6 +21,7 @@
#include "space-info.h"
#include "delalloc-space.h"
#include "block-group.h"
+#include "discard.h"
#define BITS_PER_BITMAP (PAGE_SIZE * 8UL)
#define MAX_CACHE_BYTES_PER_GIG SZ_32K
@@ -750,9 +751,11 @@ static int __load_free_space_cache(struct btrfs_root *root, struct inode *inode,
/*
* Sync discard ensures that the free space cache is always
* trimmed. So when reading this in, the state should reflect
- * that.
+ * that. We also do this for async as a stop gap for lack of
+ * persistence.
*/
- if (btrfs_test_opt(fs_info, DISCARD_SYNC))
+ if (btrfs_test_opt(fs_info, DISCARD_SYNC) ||
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
e->trim_state = BTRFS_TRIM_STATE_TRIMMED;
if (!e->bytes) {
@@ -2379,6 +2382,7 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
u64 offset, u64 bytes,
enum btrfs_trim_state trim_state)
{
+ struct btrfs_block_group_cache *cache = ctl->private;
struct btrfs_free_space *info;
int ret = 0;
@@ -2428,6 +2432,9 @@ int __btrfs_add_free_space(struct btrfs_fs_info *fs_info,
ASSERT(ret != -EEXIST);
}
+ if (trim_state != BTRFS_TRIM_STATE_TRIMMED)
+ btrfs_discard_queue_work(&fs_info->discard_ctl, cache);
+
return ret;
}
@@ -3201,6 +3208,7 @@ void btrfs_init_free_cluster(struct btrfs_free_cluster *cluster)
static int do_trimming(struct btrfs_block_group_cache *block_group,
u64 *total_trimmed, u64 start, u64 bytes,
u64 reserved_start, u64 reserved_bytes,
+ enum btrfs_trim_state reserved_trim_state,
struct btrfs_trim_range *trim_entry)
{
struct btrfs_space_info *space_info = block_group->space_info;
@@ -3208,6 +3216,9 @@ static int do_trimming(struct btrfs_block_group_cache *block_group,
struct btrfs_free_space_ctl *ctl = block_group->free_space_ctl;
int ret;
int update = 0;
+ u64 end = start + bytes;
+ u64 reserved_end = reserved_start + reserved_bytes;
+ enum btrfs_trim_state trim_state;
u64 trimmed = 0;
spin_lock(&space_info->lock);
@@ -3221,11 +3232,20 @@ static int do_trimming(struct btrfs_block_group_cache *block_group,
spin_unlock(&space_info->lock);
ret = btrfs_discard_extent(fs_info, start, bytes, &trimmed);
- if (!ret)
+ if (!ret) {
*total_trimmed += trimmed;
+ trim_state = BTRFS_TRIM_STATE_TRIMMED;
+ }
mutex_lock(&ctl->cache_writeout_mutex);
- btrfs_add_free_space(block_group, reserved_start, reserved_bytes);
+ if (reserved_start < start)
+ __btrfs_add_free_space(fs_info, ctl, reserved_start,
+ start - reserved_start,
+ reserved_trim_state);
+ if (start + bytes < reserved_start + reserved_bytes)
+ __btrfs_add_free_space(fs_info, ctl, end, reserved_end - end,
+ reserved_trim_state);
+ __btrfs_add_free_space(fs_info, ctl, start, bytes, trim_state);
list_del(&trim_entry->list);
mutex_unlock(&ctl->cache_writeout_mutex);
@@ -3252,6 +3272,7 @@ static int trim_no_bitmap(struct btrfs_block_group_cache *block_group,
int ret = 0;
u64 extent_start;
u64 extent_bytes;
+ enum btrfs_trim_state extent_trim_state;
u64 bytes;
while (start < end) {
@@ -3293,6 +3314,7 @@ static int trim_no_bitmap(struct btrfs_block_group_cache *block_group,
extent_start = entry->offset;
extent_bytes = entry->bytes;
+ extent_trim_state = entry->trim_state;
start = max(start, extent_start);
bytes = min(extent_start + extent_bytes, end) - start;
if (bytes < minlen) {
@@ -3311,7 +3333,8 @@ static int trim_no_bitmap(struct btrfs_block_group_cache *block_group,
mutex_unlock(&ctl->cache_writeout_mutex);
ret = do_trimming(block_group, total_trimmed, start, bytes,
- extent_start, extent_bytes, &trim_entry);
+ extent_start, extent_bytes, extent_trim_state,
+ &trim_entry);
if (ret)
break;
next:
@@ -3437,7 +3460,7 @@ static int trim_bitmaps(struct btrfs_block_group_cache *block_group,
mutex_unlock(&ctl->cache_writeout_mutex);
ret = do_trimming(block_group, total_trimmed, start, bytes,
- start, bytes, &trim_entry);
+ start, bytes, 0, &trim_entry);
if (ret) {
reset_trimming_bitmap(ctl, offset);
break;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index a02fece949cb..7a1bd85e1981 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -46,6 +46,7 @@
#include "sysfs.h"
#include "tests/btrfs-tests.h"
#include "block-group.h"
+#include "discard.h"
#include "qgroup.h"
#define CREATE_TRACE_POINTS
@@ -146,6 +147,8 @@ void __btrfs_handle_fs_error(struct btrfs_fs_info *fs_info, const char *function
if (sb_rdonly(sb))
return;
+ btrfs_discard_stop(fs_info);
+
/* btrfs handle error by forcing the filesystem readonly */
sb->s_flags |= SB_RDONLY;
btrfs_info(fs_info, "forced readonly");
@@ -313,6 +316,7 @@ enum {
Opt_datasum, Opt_nodatasum,
Opt_defrag, Opt_nodefrag,
Opt_discard, Opt_nodiscard,
+ Opt_discard_mode,
Opt_nologreplay,
Opt_norecovery,
Opt_ratio,
@@ -376,6 +380,7 @@ static const match_table_t tokens = {
{Opt_nodefrag, "noautodefrag"},
{Opt_discard, "discard"},
{Opt_nodiscard, "nodiscard"},
+ {Opt_discard_mode, "discard=%s"},
{Opt_nologreplay, "nologreplay"},
{Opt_norecovery, "norecovery"},
{Opt_ratio, "metadata_ratio=%u"},
@@ -695,12 +700,26 @@ int btrfs_parse_options(struct btrfs_fs_info *info, char *options,
info->metadata_ratio);
break;
case Opt_discard:
- btrfs_set_and_info(info, DISCARD_SYNC,
- "turning on sync discard");
+ case Opt_discard_mode:
+ if (token == Opt_discard ||
+ strcmp(args[0].from, "sync") == 0) {
+ btrfs_clear_opt(info->mount_opt, DISCARD_ASYNC);
+ btrfs_set_and_info(info, DISCARD_SYNC,
+ "turning on sync discard");
+ } else if (strcmp(args[0].from, "async") == 0) {
+ btrfs_clear_opt(info->mount_opt, DISCARD_SYNC);
+ btrfs_set_and_info(info, DISCARD_ASYNC,
+ "turning on async discard");
+ } else {
+ ret = -EINVAL;
+ goto out;
+ }
break;
case Opt_nodiscard:
btrfs_clear_and_info(info, DISCARD_SYNC,
"turning off discard");
+ btrfs_clear_and_info(info, DISCARD_ASYNC,
+ "turning off async discard");
break;
case Opt_space_cache:
case Opt_space_cache_version:
@@ -1324,6 +1343,8 @@ static int btrfs_show_options(struct seq_file *seq, struct dentry *dentry)
seq_puts(seq, ",flushoncommit");
if (btrfs_test_opt(info, DISCARD_SYNC))
seq_puts(seq, ",discard");
+ if (btrfs_test_opt(info, DISCARD_ASYNC))
+ seq_puts(seq, ",discard=async");
if (!(info->sb->s_flags & SB_POSIXACL))
seq_puts(seq, ",noacl");
if (btrfs_test_opt(info, SPACE_CACHE))
@@ -1714,6 +1735,14 @@ static inline void btrfs_remount_cleanup(struct btrfs_fs_info *fs_info,
btrfs_cleanup_defrag_inodes(fs_info);
}
+ /* If we toggled discard async. */
+ if (!btrfs_raw_test_opt(old_opts, DISCARD_ASYNC) &&
+ btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_resume(fs_info);
+ else if (btrfs_raw_test_opt(old_opts, DISCARD_ASYNC) &&
+ !btrfs_test_opt(fs_info, DISCARD_ASYNC))
+ btrfs_discard_cleanup(fs_info);
+
clear_bit(BTRFS_FS_STATE_REMOUNTING, &fs_info->fs_state);
}
@@ -1761,6 +1790,8 @@ static int btrfs_remount(struct super_block *sb, int *flags, char *data)
*/
cancel_work_sync(&fs_info->async_reclaim_work);
+ btrfs_discard_cleanup(fs_info);
+
/* wait for the uuid_scan task to finish */
down(&fs_info->uuid_tree_rescan_sem);
/* avoid complains from lockdep et al. */
--
2.17.1
^ permalink raw reply related [flat|nested] 61+ messages in thread
end of thread, other threads:[~2020-01-02 16:57 UTC | newest]
Thread overview: 61+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-12-14 0:22 [PATCH v6 00/22] btrfs: async discard support Dennis Zhou
2019-12-14 0:22 ` [PATCH 01/22] bitmap: genericize percpu bitmap region iterators Dennis Zhou
2019-12-14 0:22 ` [PATCH 02/22] btrfs: rename DISCARD opt to DISCARD_SYNC Dennis Zhou
2019-12-14 0:22 ` [PATCH 03/22] btrfs: keep track of which extents have been discarded Dennis Zhou
2019-12-14 0:22 ` [PATCH 04/22] btrfs: keep track of cleanliness of the bitmap Dennis Zhou
2019-12-14 0:22 ` [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue Dennis Zhou
2019-12-14 0:22 ` [PATCH 06/22] btrfs: handle empty block_group removal Dennis Zhou
2019-12-14 0:22 ` [PATCH 07/22] btrfs: discard one region at a time in async discard Dennis Zhou
2019-12-14 0:22 ` [PATCH 08/22] btrfs: add removal calls for sysfs debug/ Dennis Zhou
2019-12-18 11:45 ` Anand Jain
2019-12-14 0:22 ` [PATCH 09/22] btrfs: make UUID/debug have its own kobject Dennis Zhou
2019-12-18 11:45 ` Anand Jain
2019-12-14 0:22 ` [PATCH 10/22] btrfs: add discard sysfs directory Dennis Zhou
2019-12-18 11:45 ` Anand Jain
2019-12-14 0:22 ` [PATCH 11/22] btrfs: track discardable extents for async discard Dennis Zhou
2019-12-14 0:22 ` [PATCH 12/22] btrfs: keep track of discardable_bytes Dennis Zhou
2019-12-14 0:22 ` [PATCH 13/22] btrfs: calculate discard delay based on number of extents Dennis Zhou
2019-12-30 16:50 ` David Sterba
2020-01-02 16:45 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 14/22] btrfs: add bps discard rate limit Dennis Zhou
2019-12-30 17:58 ` David Sterba
2020-01-02 16:46 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 15/22] btrfs: limit max discard size for async discard Dennis Zhou
2019-12-30 18:00 ` David Sterba
2019-12-30 18:08 ` David Sterba
2020-01-02 16:48 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 16/22] btrfs: make max async discard size tunable Dennis Zhou
2019-12-30 18:05 ` David Sterba
2020-01-02 16:50 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 17/22] btrfs: have multiple discard lists Dennis Zhou
2019-12-14 0:22 ` [PATCH 18/22] btrfs: only keep track of data extents for async discard Dennis Zhou
2019-12-30 17:39 ` David Sterba
2020-01-02 16:55 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 19/22] btrfs: keep track of discard reuse stats Dennis Zhou
2019-12-30 17:33 ` David Sterba
2020-01-02 16:57 ` Dennis Zhou
2019-12-14 0:22 ` [PATCH 20/22] btrfs: add async discard header Dennis Zhou
2019-12-14 0:22 ` [PATCH 21/22] btrfs: increase the metadata allowance for the free_space_cache Dennis Zhou
2019-12-14 0:22 ` [PATCH 22/22] btrfs: make smaller extents more likely to go into bitmaps Dennis Zhou
2019-12-17 14:55 ` [PATCH v6 00/22] btrfs: async discard support David Sterba
2019-12-18 0:06 ` Dennis Zhou
2019-12-19 2:03 ` Dennis Zhou
2019-12-19 20:06 ` David Sterba
2019-12-19 21:22 ` Dennis Zhou
2019-12-19 20:34 ` David Sterba
2019-12-19 21:17 ` Dennis Zhou
2019-12-30 18:13 ` David Sterba
2019-12-30 18:49 ` Dennis Zhou
2020-01-02 13:22 ` David Sterba
-- strict thread matches above, loose matches on Subject: below --
2019-12-09 19:45 [PATCH v5 " Dennis Zhou
2019-12-09 19:45 ` [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue Dennis Zhou
2019-11-25 19:46 [PATCH v4 00/22] btrfs: async discard support Dennis Zhou
2019-11-25 19:46 ` [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue Dennis Zhou
[not found] <201911220351.HPI9gxNo%lkp@intel.com>
2019-11-22 4:27 ` Nick Desaulniers
2019-11-25 18:59 ` Dennis Zhou
2019-11-25 19:39 ` Nick Desaulniers
2019-11-26 1:42 ` Philip Li
2019-11-26 1:47 ` Nick Desaulniers
2019-11-26 4:07 ` Philip Li
2019-11-26 4:09 ` Philip Li
2019-11-20 21:50 [PATCH v3 00/22] btrfs: async discard support Dennis Zhou
2019-11-20 21:51 ` [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue Dennis Zhou
2019-10-23 22:52 [PATCH v2 00/22] btrfs: async discard support Dennis Zhou
2019-10-23 22:52 ` [PATCH 05/22] btrfs: add the beginning of async discard, discard workqueue Dennis Zhou
2019-11-11 18:49 ` David Sterba
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).