* [PATCH 1/3] btrfs: zoned: unset dedicated block group on allocation failure
2021-12-07 15:35 [PATCH 0/3] btrfs: zoned: fix zoned extent allocator Naohiro Aota
@ 2021-12-07 15:35 ` Naohiro Aota
2021-12-07 15:35 ` [PATCH 2/3] btrfs: add extent allocator hook to decide to allocate chunk or not Naohiro Aota
` (2 subsequent siblings)
3 siblings, 0 replies; 8+ messages in thread
From: Naohiro Aota @ 2021-12-07 15:35 UTC (permalink / raw)
To: linux-btrfs; +Cc: David Sterba, Naohiro Aota
Allocating an extent from a block group can fail for various
reasons. When an allocation from a dedicated block group (for tree-log
or relocation data) fails, we need to unregister it as a dedicated one
so that we can allocate a new block group for the dedicated one.
However, we are returning early when the block group in case it is
read-only, fully used, or not be able to activate the zone. As a
result, we keep the non-usable block group as a dedicated one, leading
to further allocation failure. With many block groups, the allocator
will iterate hopeless loop to find a free extent, results in
a hung task.
Fix the issue by delaying the return and doing the proper cleanups.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
---
fs/btrfs/extent-tree.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 3fd736a02c1e..34200c1a7da0 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3790,23 +3790,35 @@ static int do_allocation_zoned(struct btrfs_block_group *block_group,
spin_unlock(&fs_info->relocation_bg_lock);
if (skip)
return 1;
+
/* Check RO and no space case before trying to activate it */
spin_lock(&block_group->lock);
if (block_group->ro ||
block_group->alloc_offset == block_group->zone_capacity) {
- spin_unlock(&block_group->lock);
- return 1;
+ ret = 1;
+ /*
+ * May need to clear fs_info->{treelog,data_reloc}_bg.
+ * Return the error after taking the locks.
+ */
}
spin_unlock(&block_group->lock);
- if (!btrfs_zone_activate(block_group))
- return 1;
+ if (!ret && !btrfs_zone_activate(block_group)) {
+ ret = 1;
+ /*
+ * May need to clear fs_info->{treelog,data_reloc}_bg.
+ * Return the error after taking the locks.
+ */
+ }
spin_lock(&space_info->lock);
spin_lock(&block_group->lock);
spin_lock(&fs_info->treelog_bg_lock);
spin_lock(&fs_info->relocation_bg_lock);
+ if (ret)
+ goto out;
+
ASSERT(!ffe_ctl->for_treelog ||
block_group->start == fs_info->treelog_bg ||
fs_info->treelog_bg == 0);
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 2/3] btrfs: add extent allocator hook to decide to allocate chunk or not
2021-12-07 15:35 [PATCH 0/3] btrfs: zoned: fix zoned extent allocator Naohiro Aota
2021-12-07 15:35 ` [PATCH 1/3] btrfs: zoned: unset dedicated block group on allocation failure Naohiro Aota
@ 2021-12-07 15:35 ` Naohiro Aota
2021-12-07 15:35 ` [PATCH 3/3] btrfs: zoned: fix chunk allocation condition for zoned allocator Naohiro Aota
2021-12-08 16:18 ` [PATCH 0/3] btrfs: zoned: fix zoned extent allocator David Sterba
3 siblings, 0 replies; 8+ messages in thread
From: Naohiro Aota @ 2021-12-07 15:35 UTC (permalink / raw)
To: linux-btrfs; +Cc: David Sterba, Naohiro Aota
Introduce a new hook for an extent allocator policy. With the new
hook, a policy can decide to allocate a new block group or not. If
not, it will return -ENOSPC, so btrfs_reserve_extent() will cut the
allocation size in half and retry the allocation if min_alloc_size is
large enough.
The hook has a place holder and will be replaced with the real
implementation in the next patch.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
---
fs/btrfs/extent-tree.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 34200c1a7da0..5ec512673dc5 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3959,6 +3959,19 @@ static void found_extent(struct find_free_extent_ctl *ffe_ctl,
}
}
+static bool can_allocate_chunk(struct btrfs_fs_info *fs_info,
+ struct find_free_extent_ctl *ffe_ctl)
+{
+ switch (ffe_ctl->policy) {
+ case BTRFS_EXTENT_ALLOC_CLUSTERED:
+ return true;
+ case BTRFS_EXTENT_ALLOC_ZONED:
+ return true;
+ default:
+ BUG();
+ }
+}
+
static int chunk_allocation_failed(struct find_free_extent_ctl *ffe_ctl)
{
switch (ffe_ctl->policy) {
@@ -4046,6 +4059,10 @@ static int find_free_extent_update_loop(struct btrfs_fs_info *fs_info,
struct btrfs_trans_handle *trans;
int exist = 0;
+ /*Check if allocation policy allows to create a new chunk */
+ if (!can_allocate_chunk(fs_info, ffe_ctl))
+ return -ENOSPC;
+
trans = current->journal_info;
if (trans)
exist = 1;
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 3/3] btrfs: zoned: fix chunk allocation condition for zoned allocator
2021-12-07 15:35 [PATCH 0/3] btrfs: zoned: fix zoned extent allocator Naohiro Aota
2021-12-07 15:35 ` [PATCH 1/3] btrfs: zoned: unset dedicated block group on allocation failure Naohiro Aota
2021-12-07 15:35 ` [PATCH 2/3] btrfs: add extent allocator hook to decide to allocate chunk or not Naohiro Aota
@ 2021-12-07 15:35 ` Naohiro Aota
2021-12-08 16:18 ` [PATCH 0/3] btrfs: zoned: fix zoned extent allocator David Sterba
3 siblings, 0 replies; 8+ messages in thread
From: Naohiro Aota @ 2021-12-07 15:35 UTC (permalink / raw)
To: linux-btrfs; +Cc: David Sterba, Naohiro Aota
The ZNS specification defines a limit on the number of "active"
zones. That limit impose us to limit the number of block groups which
can be used for an allocation at the same time. Not to exceed the
limit, we reuse the existing active block groups as much as possible
when we can't activate any other zones without sacrificing an already
activated block group in commit a85f05e59bc1 ("btrfs: zoned: avoid
chunk allocation if active block group has enough space").
However, the check is wrong in two ways. First, it checks the
condition for every raid index (ffe_ctl->index). Even if it reaches
the condition and "ffe_ctl->max_extent_size >=
ffe_ctl->min_alloc_size" is met, there can be other block groups
having enough space to hold ffe_ctl->num_bytes. (Actually, this won't
happen in the current zoned code as it only supports SINGLE
profile. But, it can happen once it enables other RAID types.)
Second, it checks the active zone availability depending on the
raid index. The raid index is just an index for
space_info->block_groups, so it has nothing to do with chunk allocation.
These mistakes are causing a faulty allocation in a certain
situation. Consider we are running zoned btrfs on a device whose
max_active_zone == 0 (no limit). And, suppose no block group have a
room to fit ffe_ctl->num_bytes but some room to meet
ffe_ctl->min_alloc_size (i.e. max_extent_size > num_bytes >=
min_alloc_size).
In this situation, the following occur:
- With SINGLE raid_index, it reaches the chunk allocation checking
code
- The check returns true because we can activate a new zone (no limit)
- But, before allocating the chunk, it iterates to the next raid index
(RAID5)
- Since there are no RAID5 block groups on zoned mode, it again
reaches the check code
- The check returns false because of btrfs_can_activate_zone()'s "if
(raid_index != BTRFS_RAID_SINGLE)" part
- That results in returning -ENOSPC without allocating a new chunk
As a result, we end up hitting -ENOSPC too early.
Move the check to the right place in the can_allocate_chunk() hook,
and do the active zone check depending on the allocation flag, not on
the raid index.
---
fs/btrfs/extent-tree.c | 21 +++++++++------------
fs/btrfs/zoned.c | 5 ++---
fs/btrfs/zoned.h | 5 ++---
3 files changed, 13 insertions(+), 18 deletions(-)
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 5ec512673dc5..802add9857ed 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3966,6 +3966,15 @@ static bool can_allocate_chunk(struct btrfs_fs_info *fs_info,
case BTRFS_EXTENT_ALLOC_CLUSTERED:
return true;
case BTRFS_EXTENT_ALLOC_ZONED:
+ /*
+ * If we have enough free space left in an already
+ * active block group and we can't activate any other
+ * zone now, do not allow allocating a new chunk and
+ * let find_free_extent() retry with a smaller size.
+ */
+ if (ffe_ctl->max_extent_size >= ffe_ctl->min_alloc_size &&
+ !btrfs_can_activate_zone(fs_info->fs_devices, ffe_ctl->flags))
+ return false;
return true;
default:
BUG();
@@ -4012,18 +4021,6 @@ static int find_free_extent_update_loop(struct btrfs_fs_info *fs_info,
return 0;
}
- if (ffe_ctl->max_extent_size >= ffe_ctl->min_alloc_size &&
- !btrfs_can_activate_zone(fs_info->fs_devices, ffe_ctl->index)) {
- /*
- * If we have enough free space left in an already active block
- * group and we can't activate any other zone now, retry the
- * active ones with a smaller allocation size. Returning early
- * from here will tell btrfs_reserve_extent() to haven the
- * size.
- */
- return -ENOSPC;
- }
-
if (ffe_ctl->loop >= LOOP_CACHING_WAIT && ffe_ctl->have_caching_bg)
return 1;
diff --git a/fs/btrfs/zoned.c b/fs/btrfs/zoned.c
index 67d932d70798..06681fae450a 100644
--- a/fs/btrfs/zoned.c
+++ b/fs/btrfs/zoned.c
@@ -1883,7 +1883,7 @@ int btrfs_zone_finish(struct btrfs_block_group *block_group)
return ret;
}
-bool btrfs_can_activate_zone(struct btrfs_fs_devices *fs_devices, int raid_index)
+bool btrfs_can_activate_zone(struct btrfs_fs_devices *fs_devices, u64 flags)
{
struct btrfs_device *device;
bool ret = false;
@@ -1892,8 +1892,7 @@ bool btrfs_can_activate_zone(struct btrfs_fs_devices *fs_devices, int raid_index
return true;
/* Non-single profiles are not supported yet */
- if (raid_index != BTRFS_RAID_SINGLE)
- return false;
+ ASSERT((flags & BTRFS_BLOCK_GROUP_PROFILE_MASK) == 0);
/* Check if there is a device with active zones left */
mutex_lock(&fs_devices->device_list_mutex);
diff --git a/fs/btrfs/zoned.h b/fs/btrfs/zoned.h
index e53ab7b96437..002ff86c8608 100644
--- a/fs/btrfs/zoned.h
+++ b/fs/btrfs/zoned.h
@@ -71,8 +71,7 @@ struct btrfs_device *btrfs_zoned_get_device(struct btrfs_fs_info *fs_info,
u64 logical, u64 length);
bool btrfs_zone_activate(struct btrfs_block_group *block_group);
int btrfs_zone_finish(struct btrfs_block_group *block_group);
-bool btrfs_can_activate_zone(struct btrfs_fs_devices *fs_devices,
- int raid_index);
+bool btrfs_can_activate_zone(struct btrfs_fs_devices *fs_devices, u64 flags);
void btrfs_zone_finish_endio(struct btrfs_fs_info *fs_info, u64 logical,
u64 length);
void btrfs_clear_data_reloc_bg(struct btrfs_block_group *bg);
@@ -222,7 +221,7 @@ static inline int btrfs_zone_finish(struct btrfs_block_group *block_group)
}
static inline bool btrfs_can_activate_zone(struct btrfs_fs_devices *fs_devices,
- int raid_index)
+ u64 flags)
{
return true;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH 0/3] btrfs: zoned: fix zoned extent allocator
2021-12-07 15:35 [PATCH 0/3] btrfs: zoned: fix zoned extent allocator Naohiro Aota
` (2 preceding siblings ...)
2021-12-07 15:35 ` [PATCH 3/3] btrfs: zoned: fix chunk allocation condition for zoned allocator Naohiro Aota
@ 2021-12-08 16:18 ` David Sterba
2021-12-29 0:22 ` Shinichiro Kawasaki
3 siblings, 1 reply; 8+ messages in thread
From: David Sterba @ 2021-12-08 16:18 UTC (permalink / raw)
To: Naohiro Aota; +Cc: linux-btrfs, David Sterba
On Wed, Dec 08, 2021 at 12:35:46AM +0900, Naohiro Aota wrote:
> There are several reports of hung_task on btrfs recently.
>
> - https://github.com/naota/linux/issues/59
> - https://lore.kernel.org/linux-btrfs/CAJCQCtR=jztS3P34U_iUNoBodExHcud44OQ8oe4Jn3TK=1yFNw@mail.gmail.com/T/
>
> The stack trace of hung is like this:
>
> [ 739.991925][ T25] INFO: task kworker/u4:0:7 blocked for more than 122 seconds.
> [ 739.994821][ T25] Not tainted 5.16.0-rc3+ #56
> [ 739.996676][ T25] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 739.999600][ T25] task:kworker/u4:0 state:D stack: 0 pid: 7 ppid: 2 flags:0x00004000
> [ 740.002656][ T25] Workqueue: writeback wb_workfn (flush-btrfs-5)
> [ 740.004894][ T25] Call Trace:
> [ 740.006113][ T25] <TASK>
> [ 740.007143][ T25] __schedule+0x9e3/0x23b0
> [ 740.008657][ T25] ? io_schedule_timeout+0x190/0x190
> [ 740.010529][ T25] ? blk_start_plug_nr_ios+0x270/0x270
> [ 740.012385][ T25] ? _raw_spin_unlock_irq+0x28/0x50
> [ 740.014163][ T25] schedule+0xed/0x280
> [ 740.015567][ T25] io_schedule+0xfb/0x180
> [ 740.017026][ T25] folio_wait_bit_common+0x386/0x840
> [ 740.018839][ T25] ? delete_from_page_cache+0x220/0x220
> [ 740.020744][ T25] ? __this_cpu_preempt_check+0x13/0x20
> [ 740.022645][ T25] ? filemap_range_has_page+0x210/0x210
> [ 740.024588][ T25] __folio_lock+0x17/0x20
> [ 740.026197][ T25] extent_write_cache_pages+0x78c/0xba0 [btrfs]
> [ 740.028776][ T25] ? __extent_writepage+0x980/0x980 [btrfs]
> [ 740.031026][ T25] ? __kasan_check_read+0x11/0x20
> [ 740.032916][ T25] ? __lock_acquire+0x1772/0x5a10
> [ 740.034611][ T25] extent_writepages+0x1e8/0x3b0 [btrfs]
> [ 740.036636][ T25] ? extent_write_locked_range+0x580/0x580 [btrfs]
> [ 740.038828][ T25] ? lockdep_hardirqs_on_prepare+0x410/0x410
> [ 740.040929][ T25] btrfs_writepages+0xe/0x10 [btrfs]
> [ 740.042879][ T25] do_writepages+0x187/0x610
> [ 740.044239][ T25] ? page_writeback_cpu_online+0x20/0x20
> [ 740.045810][ T25] ? sched_clock+0x9/0x10
> [ 740.047015][ T25] ? sched_clock_cpu+0x18/0x1b0
> [ 740.048341][ T25] ? find_held_lock+0x3c/0x130
> [ 740.049649][ T25] ? __this_cpu_preempt_check+0x13/0x20
> [ 740.051155][ T25] ? lock_release+0x3fd/0xed0
> [ 740.052358][ T25] ? __this_cpu_preempt_check+0x13/0x20
> [ 740.053775][ T25] ? lock_is_held_type+0xe4/0x140
> [ 740.055059][ T25] __writeback_single_inode+0xd7/0xa90
> [ 740.056399][ T25] writeback_sb_inodes+0x4e8/0xff0
> [ 740.057693][ T25] ? sync_inode_metadata+0xe0/0xe0
> [ 740.058918][ T25] ? down_read_trylock+0x45/0x50
> [ 740.060003][ T25] ? trylock_super+0x1b/0xc0
> [ 740.061050][ T25] __writeback_inodes_wb+0xba/0x210
> [ 740.062123][ T25] wb_writeback+0x5b3/0x8b0
> [ 740.063075][ T25] ? __writeback_inodes_wb+0x210/0x210
> [ 740.064158][ T25] ? __local_bh_enable_ip+0xaa/0x120
> [ 740.065196][ T25] ? __local_bh_enable_ip+0xaa/0x120
> [ 740.066180][ T25] ? trace_hardirqs_on+0x2b/0x130
> [ 740.067117][ T25] ? wb_workfn+0x2cc/0xe80
> [ 740.067943][ T25] ? get_nr_dirty_inodes+0x130/0x1c0
> [ 740.068909][ T25] wb_workfn+0x6f5/0xe80
> [ 740.069674][ T25] ? inode_wait_for_writeback+0x40/0x40
> [ 740.070841][ T25] ? __this_cpu_preempt_check+0x13/0x20
> [ 740.071834][ T25] ? lock_acquire+0x1c1/0x4f0
> [ 740.072630][ T25] ? lock_release+0xed0/0xed0
> [ 740.073483][ T25] ? lock_downgrade+0x7c0/0x7c0
> [ 740.074374][ T25] ? __this_cpu_preempt_check+0x13/0x20
> [ 740.075342][ T25] ? lock_is_held_type+0xe4/0x140
> [ 740.076236][ T25] process_one_work+0x826/0x14e0
> [ 740.077045][ T25] ? lock_is_held_type+0xe4/0x140
> [ 740.077921][ T25] ? pwq_dec_nr_in_flight+0x250/0x250
> [ 740.078842][ T25] ? lockdep_hardirqs_off+0x99/0xe0
> [ 740.079699][ T25] worker_thread+0x59b/0x1050
> [ 740.080478][ T25] ? process_one_work+0x14e0/0x14e0
> [ 740.081306][ T25] kthread+0x38f/0x460
> [ 740.081955][ T25] ? set_kthread_struct+0x110/0x110
> [ 740.082769][ T25] ret_from_fork+0x22/0x30
> [ 740.083476][ T25] </TASK>
> [ 740.083972][ T25] INFO: task aio-dio-write-v:1459 blocked for more than 122 seconds.
> [ 740.085202][ T25] Not tainted 5.16.0-rc3+ #56
> [ 740.085970][ T25] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 740.087238][ T25] task:aio-dio-write-v state:D stack: 0 pid: 1459 ppid: 542 flags:0x00004000
> [ 740.088676][ T25] Call Trace:
> [ 740.089269][ T25] <TASK>
> [ 740.089780][ T25] __schedule+0x9e3/0x23b0
> [ 740.090644][ T25] ? io_schedule_timeout+0x190/0x190
> [ 740.091614][ T25] ? blk_start_plug_nr_ios+0x270/0x270
> [ 740.092649][ T25] ? _raw_spin_unlock_irq+0x28/0x50
> [ 740.093531][ T25] schedule+0xed/0x280
> [ 740.094323][ T25] io_schedule+0xfb/0x180
> [ 740.095095][ T25] folio_wait_bit_common+0x386/0x840
> [ 740.095868][ T25] ? delete_from_page_cache+0x220/0x220
> [ 740.096730][ T25] ? lock_is_held_type+0xe4/0x140
> [ 740.097432][ T25] ? filemap_range_has_page+0x210/0x210
> [ 740.098252][ T25] __filemap_get_folio+0x4d3/0x8f0
> [ 740.099016][ T25] ? filemap_range_needs_writeback+0xb0/0xb0
> [ 740.099953][ T25] ? lock_contended+0xdf0/0xdf0
> [ 740.100663][ T25] pagecache_get_page+0x19/0xc0
> [ 740.101400][ T25] prepare_pages+0x205/0x4c0 [btrfs]
> [ 740.102274][ T25] btrfs_buffered_write+0x5e0/0x1060 [btrfs]
> [ 740.103402][ T25] ? btrfs_dirty_pages+0x2c0/0x2c0 [btrfs]
> [ 740.104480][ T25] ? __up_read+0x1a9/0x7b0
> [ 740.105260][ T25] ? up_write+0x480/0x480
> [ 740.106024][ T25] ? btrfs_file_llseek+0x600/0x600 [btrfs]
> [ 740.107157][ T25] btrfs_file_write_iter+0x84e/0xfa0 [btrfs]
> [ 740.108239][ T25] ? lock_downgrade+0x7c0/0x7c0
> [ 740.109091][ T25] aio_write+0x314/0x820
> [ 740.109888][ T25] ? cpumask_weight.constprop.0+0x40/0x40
> [ 740.110846][ T25] ? kvm_sched_clock_read+0x18/0x40
> [ 740.111677][ T25] ? sched_clock_cpu+0x18/0x1b0
> [ 740.112765][ T25] ? __this_cpu_preempt_check+0x13/0x20
> [ 740.114113][ T25] ? lock_release+0x3fd/0xed0
> [ 740.115264][ T25] ? lock_downgrade+0x7c0/0x7c0
> [ 740.116505][ T25] io_submit_one.constprop.0+0xba3/0x1a50
> [ 740.117894][ T25] ? io_submit_one.constprop.0+0xba3/0x1a50
> [ 740.119180][ T25] ? kvm_sched_clock_read+0x18/0x40
> [ 740.120036][ T25] ? sched_clock+0x9/0x10
> [ 740.120764][ T25] ? sched_clock_cpu+0x18/0x1b0
> [ 740.121619][ T25] ? __x64_sys_io_getevents_time32+0x2a0/0x2a0
> [ 740.122628][ T25] ? lock_release+0x3fd/0xed0
> [ 740.123522][ T25] __x64_sys_io_submit+0x15d/0x2b0
> [ 740.124417][ T25] ? __x64_sys_io_submit+0x15d/0x2b0
> [ 740.125363][ T25] ? io_submit_one.constprop.0+0x1a50/0x1a50
> [ 740.126322][ T25] ? __this_cpu_preempt_check+0x13/0x20
> [ 740.127243][ T25] ? lock_is_held_type+0xe4/0x140
> [ 740.128071][ T25] ? lockdep_hardirqs_on+0x7e/0x100
> [ 740.128915][ T25] ? syscall_enter_from_user_mode+0x25/0x80
> [ 740.129872][ T25] ? trace_hardirqs_on+0x2b/0x130
> [ 740.130694][ T25] do_syscall_64+0x3b/0x90
> [ 740.131380][ T25] entry_SYSCALL_64_after_hwframe+0x44/0xae
> [ 740.132351][ T25] RIP: 0033:0x7f5c03578679
> [ 740.133117][ T25] RSP: 002b:00007fffa4f030e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000d1
> [ 740.134468][ T25] RAX: ffffffffffffffda RBX: 00007f5c034746c0 RCX: 00007f5c03578679
> [ 740.135749][ T25] RDX: 0000561f67b701d0 RSI: 0000000000000037 RDI: 00007f5c03676000
> [ 740.137151][ T25] RBP: 00007f5c03676000 R08: 0000000000000000 R09: 0000000000000000
> [ 740.138505][ T25] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000037
> [ 740.139831][ T25] R13: 0000000000000000 R14: 0000561f67b701d0 R15: 0000561f67b701d0
> [ 740.141263][ T25] </TASK>
> [ 740.141793][ T25]
> [ 740.141793][ T25] Showing all locks held in the system:
> [ 740.143126][ T25] 3 locks held by kworker/u4:0/7:
> [ 740.143982][ T25] #0: ffff888100d4d948 ((wq_completion)writeback){+.+.}-{0:0}, at: process_one_work+0x740/0x14e0
> [ 740.145599][ T25] #1: ffffc9000007fda8 ((work_completion)(&(&wb->dwork)->work)){+.+.}-{0:0}, at: process_one_work+0x770/0x14e0
> [ 740.147521][ T25] #2: ffff8881771760e8 (&type->s_umount_key#30){++++}-{3:3}, at: trylock_super+0x1b/0xc0
> [ 740.149084][ T25] 1 lock held by khungtaskd/25:
> [ 740.149916][ T25] #0: ffffffff833cee00 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x5f/0x27f
> [ 740.151491][ T25] 1 lock held by aio-dio-write-v/1459:
> [ 740.152458][ T25] #0: ffff888172999b10 (&sb->s_type->i_mutex_key#13){++++}-{3:3}, at: btrfs_inode_lock+0x3f/0x70 [btrfs]
> [ 740.154487][ T25]
> [ 740.154920][ T25] =============================================
> [ 740.154920][ T25]
> [ 740.156347][ T25] Kernel panic - not syncing: hung_task: blocked tasks
> [ 740.157548][ T25] CPU: 0 PID: 25 Comm: khungtaskd Not tainted 5.16.0-rc3+ #56
> [ 740.158826][ T25] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS d55cb5a 04/01/2014
> [ 740.160265][ T25] Call Trace:
> [ 740.160788][ T25] <TASK>
> [ 740.161240][ T25] dump_stack_lvl+0x49/0x5e
> [ 740.162045][ T25] dump_stack+0x10/0x12
> [ 740.162642][ T25] panic+0x1e2/0x478
> [ 740.163330][ T25] ? __warn_printk+0xf3/0xf3
> [ 740.164041][ T25] watchdog.cold+0x118/0x137
> [ 740.164741][ T25] ? reset_hung_task_detector+0x30/0x30
> [ 740.165537][ T25] kthread+0x38f/0x460
> [ 740.166202][ T25] ? set_kthread_struct+0x110/0x110
> [ 740.167022][ T25] ret_from_fork+0x22/0x30
> [ 740.167782][ T25] </TASK>
> [ 740.168520][ T25] Kernel Offset: disabled
> [ 740.169285][ T25] ---[ end Kernel panic - not syncing: hung_task: blocked tasks ]---
>
> While we are debugging this issue, we found some faulty behaviors on
> the zoned extent allocator. It is not the root cause of the hung as we
> see a similar report also on a regular btrfs. But, it looks like that
> early -ENOSPC is, at least, making the hung to happen often.
>
> So, this series fixes the faulty behaviors of the zoned extent
> allocator.
>
> Patch 1 fixes a case when allocation fails in a dedicated block group.
>
> Patches 2 and 3 fix the chunk allocation condition for zoned
> allocator, so that it won't block a possible chunk allocation.
>
> Naohiro Aota (3):
> btrfs: zoned: unset dedicated block group on allocation failure
> btrfs: add extent allocator hook to decide to allocate chunk or not
> btrfs: zoned: fix chunk allocation condition for zoned allocator
All seem to be relevant for 5.16-rc so I'll add them to misc-next now to
give it some testing, pull request next week. Thanks.
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH 0/3] btrfs: zoned: fix zoned extent allocator
2021-12-08 16:18 ` [PATCH 0/3] btrfs: zoned: fix zoned extent allocator David Sterba
@ 2021-12-29 0:22 ` Shinichiro Kawasaki
2022-01-03 19:13 ` David Sterba
0 siblings, 1 reply; 8+ messages in thread
From: Shinichiro Kawasaki @ 2021-12-29 0:22 UTC (permalink / raw)
To: dsterba, Naohiro Aota, linux-btrfs, David Sterba
On Dec 08, 2021 / 17:18, David Sterba wrote:
> On Wed, Dec 08, 2021 at 12:35:46AM +0900, Naohiro Aota wrote:
> > There are several reports of hung_task on btrfs recently.
> >
> > - https://github.com/naota/linux/issues/59
> > - https://lore.kernel.org/linux-btrfs/CAJCQCtR=jztS3P34U_iUNoBodExHcud44OQ8oe4Jn3TK=1yFNw@mail.gmail.com/T/
(snip)
> > While we are debugging this issue, we found some faulty behaviors on
> > the zoned extent allocator. It is not the root cause of the hung as we
> > see a similar report also on a regular btrfs. But, it looks like that
> > early -ENOSPC is, at least, making the hung to happen often.
> >
> > So, this series fixes the faulty behaviors of the zoned extent
> > allocator.
> >
> > Patch 1 fixes a case when allocation fails in a dedicated block group.
> >
> > Patches 2 and 3 fix the chunk allocation condition for zoned
> > allocator, so that it won't block a possible chunk allocation.
> >
> > Naohiro Aota (3):
> > btrfs: zoned: unset dedicated block group on allocation failure
> > btrfs: add extent allocator hook to decide to allocate chunk or not
> > btrfs: zoned: fix chunk allocation condition for zoned allocator
>
> All seem to be relevant for 5.16-rc so I'll add them to misc-next now to
> give it some testing, pull request next week. Thanks.
Hello David, thanks for your maintainer-ship always.
When I run my test set for zoned btrfs configuration, I keep on observing the
issue that Naohiro addressed with the three patches. The patches are not yet
merged to 5.16-rc7. Can I expect they get merged to rc8?
--
Best Regards,
Shin'ichiro Kawasaki
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 0/3] btrfs: zoned: fix zoned extent allocator
2021-12-29 0:22 ` Shinichiro Kawasaki
@ 2022-01-03 19:13 ` David Sterba
2022-01-05 2:00 ` Shinichiro Kawasaki
0 siblings, 1 reply; 8+ messages in thread
From: David Sterba @ 2022-01-03 19:13 UTC (permalink / raw)
To: Shinichiro Kawasaki; +Cc: dsterba, Naohiro Aota, linux-btrfs, David Sterba
On Wed, Dec 29, 2021 at 12:22:31AM +0000, Shinichiro Kawasaki wrote:
> On Dec 08, 2021 / 17:18, David Sterba wrote:
> > On Wed, Dec 08, 2021 at 12:35:46AM +0900, Naohiro Aota wrote:
> > > There are several reports of hung_task on btrfs recently.
> > >
> > > - https://github.com/naota/linux/issues/59
> > > - https://lore.kernel.org/linux-btrfs/CAJCQCtR=jztS3P34U_iUNoBodExHcud44OQ8oe4Jn3TK=1yFNw@mail.gmail.com/T/
>
> (snip)
>
> > > While we are debugging this issue, we found some faulty behaviors on
> > > the zoned extent allocator. It is not the root cause of the hung as we
> > > see a similar report also on a regular btrfs. But, it looks like that
> > > early -ENOSPC is, at least, making the hung to happen often.
> > >
> > > So, this series fixes the faulty behaviors of the zoned extent
> > > allocator.
> > >
> > > Patch 1 fixes a case when allocation fails in a dedicated block group.
> > >
> > > Patches 2 and 3 fix the chunk allocation condition for zoned
> > > allocator, so that it won't block a possible chunk allocation.
> > >
> > > Naohiro Aota (3):
> > > btrfs: zoned: unset dedicated block group on allocation failure
> > > btrfs: add extent allocator hook to decide to allocate chunk or not
> > > btrfs: zoned: fix chunk allocation condition for zoned allocator
> >
> > All seem to be relevant for 5.16-rc so I'll add them to misc-next now to
> > give it some testing, pull request next week. Thanks.
>
> Hello David, thanks for your maintainer-ship always.
>
> When I run my test set for zoned btrfs configuration, I keep on observing the
> issue that Naohiro addressed with the three patches. The patches are not yet
> merged to 5.16-rc7. Can I expect they get merged to rc8?
Sorry, I did not get to sending the pull request due to holidays. The
timing of 5.16 release next week is too close, I'll tag the patches for
stable so they'll get to 5.16 later.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 0/3] btrfs: zoned: fix zoned extent allocator
2022-01-03 19:13 ` David Sterba
@ 2022-01-05 2:00 ` Shinichiro Kawasaki
0 siblings, 0 replies; 8+ messages in thread
From: Shinichiro Kawasaki @ 2022-01-05 2:00 UTC (permalink / raw)
To: dsterba, Naohiro Aota, linux-btrfs, David Sterba
On Jan 03, 2022 / 20:13, David Sterba wrote:
> On Wed, Dec 29, 2021 at 12:22:31AM +0000, Shinichiro Kawasaki wrote:
> > On Dec 08, 2021 / 17:18, David Sterba wrote:
> > > On Wed, Dec 08, 2021 at 12:35:46AM +0900, Naohiro Aota wrote:
> > > > There are several reports of hung_task on btrfs recently.
> > > >
> > > > - https://github.com/naota/linux/issues/59
> > > > - https://lore.kernel.org/linux-btrfs/CAJCQCtR=jztS3P34U_iUNoBodExHcud44OQ8oe4Jn3TK=1yFNw@mail.gmail.com/T/
> >
> > (snip)
> >
> > > > While we are debugging this issue, we found some faulty behaviors on
> > > > the zoned extent allocator. It is not the root cause of the hung as we
> > > > see a similar report also on a regular btrfs. But, it looks like that
> > > > early -ENOSPC is, at least, making the hung to happen often.
> > > >
> > > > So, this series fixes the faulty behaviors of the zoned extent
> > > > allocator.
> > > >
> > > > Patch 1 fixes a case when allocation fails in a dedicated block group.
> > > >
> > > > Patches 2 and 3 fix the chunk allocation condition for zoned
> > > > allocator, so that it won't block a possible chunk allocation.
> > > >
> > > > Naohiro Aota (3):
> > > > btrfs: zoned: unset dedicated block group on allocation failure
> > > > btrfs: add extent allocator hook to decide to allocate chunk or not
> > > > btrfs: zoned: fix chunk allocation condition for zoned allocator
> > >
> > > All seem to be relevant for 5.16-rc so I'll add them to misc-next now to
> > > give it some testing, pull request next week. Thanks.
> >
> > Hello David, thanks for your maintainer-ship always.
> >
> > When I run my test set for zoned btrfs configuration, I keep on observing the
> > issue that Naohiro addressed with the three patches. The patches are not yet
> > merged to 5.16-rc7. Can I expect they get merged to rc8?
>
> Sorry, I did not get to sending the pull request due to holidays. The
> timing of 5.16 release next week is too close, I'll tag the patches for
> stable so they'll get to 5.16 later.
Sounds good. Thank you for your care.
--
Best Regards,
Shin'ichiro Kawasaki
^ permalink raw reply [flat|nested] 8+ messages in thread