* [PATCH v4] ext4: Fix dead loop in ext4_mb_new_blocks
@ 2020-09-14 10:47 Ye Bin
2020-09-15 12:11 ` Jan Kara
0 siblings, 1 reply; 3+ messages in thread
From: Ye Bin @ 2020-09-14 10:47 UTC (permalink / raw)
To: riteshh, jack, tytso, adilger.kernel, jack, linux-ext4; +Cc: Ye Bin
As we test disk offline/online with running fsstress, we find fsstress
process is keeping running state.
kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
....
kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
ext4_mb_new_blocks
repeat:
ext4_mb_discard_preallocations_should_retry(sb, ac, &seq)
freed = ext4_mb_discard_preallocations
ext4_mb_discard_group_preallocations
this_cpu_inc(discard_pa_seq);
---> freed == 0
seq_retry = ext4_get_discard_pa_seq_sum
for_each_possible_cpu(__cpu)
__seq += per_cpu(discard_pa_seq, __cpu);
if (seq_retry != *seq) {
*seq = seq_retry;
ret = true;
}
As we see seq_retry is sum of discard_pa_seq every cpu, if
ext4_mb_discard_group_preallocations return zero discard_pa_seq in this
cpu maybe increase one, so condition "seq_retry != *seq" have always
been met.
Ritesh Harjani suggest to in ext4_mb_discard_group_preallocations function we
only increase discard_pa_seq when there is some PA to free.
Fixes: 07b5b8e1ac40 ("ext4: mballoc: introduce pcpu seqcnt for freeing PA to improve ENOSPC handling")
Signed-off-by: Ye Bin <yebin10@huawei.com>
---
fs/ext4/mballoc.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index 132c118d12e1..ff47347012f4 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -4189,7 +4189,6 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
INIT_LIST_HEAD(&list);
repeat:
ext4_lock_group(sb, group);
- this_cpu_inc(discard_pa_seq);
list_for_each_entry_safe(pa, tmp,
&grp->bb_prealloc_list, pa_group_list) {
spin_lock(&pa->pa_lock);
@@ -4206,6 +4205,9 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
/* seems this one can be freed ... */
ext4_mb_mark_pa_deleted(sb, pa);
+ if (!free)
+ this_cpu_inc(discard_pa_seq);
+
/* we can trust pa_free ... */
free += pa->pa_free;
--
2.25.4
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH v4] ext4: Fix dead loop in ext4_mb_new_blocks
2020-09-14 10:47 [PATCH v4] ext4: Fix dead loop in ext4_mb_new_blocks Ye Bin
@ 2020-09-15 12:11 ` Jan Kara
2020-09-16 10:16 ` Ritesh Harjani
0 siblings, 1 reply; 3+ messages in thread
From: Jan Kara @ 2020-09-15 12:11 UTC (permalink / raw)
To: Ye Bin; +Cc: riteshh, jack, tytso, adilger.kernel, jack, linux-ext4
[-- Attachment #1: Type: text/plain, Size: 2713 bytes --]
On Mon 14-09-20 18:47:42, Ye Bin wrote:
> As we test disk offline/online with running fsstress, we find fsstress
> process is keeping running state.
> kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
> ....
> kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
>
> ext4_mb_new_blocks
> repeat:
> ext4_mb_discard_preallocations_should_retry(sb, ac, &seq)
> freed = ext4_mb_discard_preallocations
> ext4_mb_discard_group_preallocations
> this_cpu_inc(discard_pa_seq);
> ---> freed == 0
> seq_retry = ext4_get_discard_pa_seq_sum
> for_each_possible_cpu(__cpu)
> __seq += per_cpu(discard_pa_seq, __cpu);
> if (seq_retry != *seq) {
> *seq = seq_retry;
> ret = true;
> }
>
> As we see seq_retry is sum of discard_pa_seq every cpu, if
> ext4_mb_discard_group_preallocations return zero discard_pa_seq in this
> cpu maybe increase one, so condition "seq_retry != *seq" have always
> been met.
> Ritesh Harjani suggest to in ext4_mb_discard_group_preallocations function we
> only increase discard_pa_seq when there is some PA to free.
>
> Fixes: 07b5b8e1ac40 ("ext4: mballoc: introduce pcpu seqcnt for freeing PA to improve ENOSPC handling")
> Signed-off-by: Ye Bin <yebin10@huawei.com>
The patch looks good to me. You can add:
Reviewed-by: Jan Kara <jack@suse.cz>
But as I mentioned in my previous reply I also think the attached patch
also needs to be merged to avoid premature ENOSPC errors (which your change
makes somewhat more likely). Ritesh do you agree?
Honza
> ---
> fs/ext4/mballoc.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
> index 132c118d12e1..ff47347012f4 100644
> --- a/fs/ext4/mballoc.c
> +++ b/fs/ext4/mballoc.c
> @@ -4189,7 +4189,6 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
> INIT_LIST_HEAD(&list);
> repeat:
> ext4_lock_group(sb, group);
> - this_cpu_inc(discard_pa_seq);
> list_for_each_entry_safe(pa, tmp,
> &grp->bb_prealloc_list, pa_group_list) {
> spin_lock(&pa->pa_lock);
> @@ -4206,6 +4205,9 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
> /* seems this one can be freed ... */
> ext4_mb_mark_pa_deleted(sb, pa);
>
> + if (!free)
> + this_cpu_inc(discard_pa_seq);
> +
> /* we can trust pa_free ... */
> free += pa->pa_free;
>
> --
> 2.25.4
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
[-- Attachment #2: 0001-ext4-Discard-preallocations-before-releasing-group-l.patch --]
[-- Type: text/x-patch, Size: 2138 bytes --]
From ce4bb26350da47a6c07be378bf478e5a81bc96d4 Mon Sep 17 00:00:00 2001
From: Jan Kara <jack@suse.cz>
Date: Tue, 15 Sep 2020 13:54:20 +0200
Subject: [PATCH] ext4: Discard preallocations before releasing group lock
ext4_mb_discard_group_preallocations() can be releasing group lock with
preallocations accumulated on its local list. Thus although
discard_pa_seq was incremented and concurrent allocating processes will
be retrying allocations, it can happen that premature ENOSPC error is
returned because blocks used for preallocations are not available for
reuse yet. Make sure we always free locally accumulated preallocations
before releasing group lock.
Fixes: 07b5b8e1ac40 ("ext4: mballoc: introduce pcpu seqcnt for freeing PA to improve ENOSPC handling")
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/ext4/mballoc.c | 26 ++++++++++----------------
1 file changed, 10 insertions(+), 16 deletions(-)
diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index 132c118d12e1..0ded25d55d9b 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -4215,22 +4215,6 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
list_add(&pa->u.pa_tmp_list, &list);
}
- /* if we still need more blocks and some PAs were used, try again */
- if (free < needed && busy) {
- busy = 0;
- ext4_unlock_group(sb, group);
- cond_resched();
- goto repeat;
- }
-
- /* found anything to free? */
- if (list_empty(&list)) {
- BUG_ON(free != 0);
- mb_debug(sb, "Someone else may have freed PA for this group %u\n",
- group);
- goto out;
- }
-
/* now free all selected PAs */
list_for_each_entry_safe(pa, tmp, &list, u.pa_tmp_list) {
@@ -4248,6 +4232,16 @@ ext4_mb_discard_group_preallocations(struct super_block *sb,
call_rcu(&(pa)->u.pa_rcu, ext4_mb_pa_callback);
}
+ /* if we still need more blocks and some PAs were used, try again */
+ if (free < needed && busy) {
+ ext4_unlock_group(sb, group);
+ cond_resched();
+ busy = 0;
+ /* Make sure we increment discard_pa_seq again */
+ needed -= free;
+ free = 0;
+ goto repeat;
+ }
out:
ext4_unlock_group(sb, group);
ext4_mb_unload_buddy(&e4b);
--
2.16.4
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH v4] ext4: Fix dead loop in ext4_mb_new_blocks
2020-09-15 12:11 ` Jan Kara
@ 2020-09-16 10:16 ` Ritesh Harjani
0 siblings, 0 replies; 3+ messages in thread
From: Ritesh Harjani @ 2020-09-16 10:16 UTC (permalink / raw)
To: Ye Bin; +Cc: Jan Kara, tytso, adilger.kernel, jack, linux-ext4
On 9/15/20 5:41 PM, Jan Kara wrote:
> On Mon 14-09-20 18:47:42, Ye Bin wrote:
>> As we test disk offline/online with running fsstress, we find fsstress
>> process is keeping running state.
>> kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
>> ....
>> kworker/u32:3-262 [004] ...1 140.787471: ext4_mb_discard_preallocations: dev 8,32 needed 114
>>
>> ext4_mb_new_blocks
>> repeat:
>> ext4_mb_discard_preallocations_should_retry(sb, ac, &seq)
>> freed = ext4_mb_discard_preallocations
>> ext4_mb_discard_group_preallocations
>> this_cpu_inc(discard_pa_seq);
>> ---> freed == 0
>> seq_retry = ext4_get_discard_pa_seq_sum
>> for_each_possible_cpu(__cpu)
>> __seq += per_cpu(discard_pa_seq, __cpu);
>> if (seq_retry != *seq) {
>> *seq = seq_retry;
>> ret = true;
>> }
>>
>> As we see seq_retry is sum of discard_pa_seq every cpu, if
>> ext4_mb_discard_group_preallocations return zero discard_pa_seq in this
>> cpu maybe increase one, so condition "seq_retry != *seq" have always
>> been met.
>> Ritesh Harjani suggest to in ext4_mb_discard_group_preallocations function we
>> only increase discard_pa_seq when there is some PA to free.
@yebin,
Did you confirm by running your test case that this patch indeed fixes
your reported issue.
With that confirmed, the patch does looks good to me. Feel free to add.
Reviewed-by: Ritesh Harjani <riteshh@linux.ibm.com>
>>
>> Fixes: 07b5b8e1ac40 ("ext4: mballoc: introduce pcpu seqcnt for freeing PA to improve ENOSPC handling")
>> Signed-off-by: Ye Bin <yebin10@huawei.com>
>
> The patch looks good to me. You can add:
>
> Reviewed-by: Jan Kara <jack@suse.cz>
>
> But as I mentioned in my previous reply I also think the attached patch
> also needs to be merged to avoid premature ENOSPC errors (which your change
> makes somewhat more likely). Ritesh do you agree?
Yes, agree that Jan's attached patch should help to avoid premature
ENOSPC errors. We should have his patch too on top of current patch.
@yebin
Should we have a v5 of then, with both patches included for merging?
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-09-16 20:46 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-14 10:47 [PATCH v4] ext4: Fix dead loop in ext4_mb_new_blocks Ye Bin
2020-09-15 12:11 ` Jan Kara
2020-09-16 10:16 ` Ritesh Harjani
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).