* [PATCH v3 0/2] Fixes for fscache volume operations
@ 2023-01-13 11:52 Hou Tao
2023-01-13 11:52 ` [PATCH v3 1/2] fscache: Use wait_on_bit() to wait for the freeing of relinquished volume Hou Tao
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Hou Tao @ 2023-01-13 11:52 UTC (permalink / raw)
To: linux-cachefs
Cc: Jeff Layton, linux-kernel, David Howells, houtao1, linux-fsdevel,
linux-erofs
From: Hou Tao <houtao1@huawei.com>
Hi,
The patchset includes two fixes for fscache volume operations: patch 1
fixes the hang problem during volume acquisition when the volume
acquisition process waits for the freeing of relinquished volume, patch
2 adds the missing memory barrier in fscache_create_volume_work() and it
is spotted through code review when checking whether or not these is
missing smp_mb() before invoking wake_up_bit().
Comments are always welcome.
Chang Log:
v3:
* Use clear_and_wake_up_bit() helper (Suggested by Jingbo Xu)
* Tidy up commit message and add Reviewed-by tag
v2: https://listman.redhat.com/archives/linux-cachefs/2022-December/007402.html
* rebased on v6.1-rc1
* Patch 1: use wait_on_bit() instead (Suggested by David)
* Patch 2: add the missing smp_mb() in fscache_create_volume_work()
v1: https://listman.redhat.com/archives/linux-cachefs/2022-December/007384.html
Hou Tao (2):
fscache: Use wait_on_bit() to wait for the freeing of relinquished
volume
fscache: Use clear_and_wake_up_bit() in fscache_create_volume_work()
fs/fscache/volume.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
--
2.29.2
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v3 1/2] fscache: Use wait_on_bit() to wait for the freeing of relinquished volume
2023-01-13 11:52 [PATCH v3 0/2] Fixes for fscache volume operations Hou Tao
@ 2023-01-13 11:52 ` Hou Tao
2023-01-13 15:27 ` Jeff Layton
2023-01-13 11:52 ` [PATCH v3 2/2] fscache: Use clear_and_wake_up_bit() in fscache_create_volume_work() Hou Tao
2023-01-30 10:56 ` [PATCH v3 0/2] Fixes for fscache volume operations Hou Tao
2 siblings, 1 reply; 6+ messages in thread
From: Hou Tao @ 2023-01-13 11:52 UTC (permalink / raw)
To: linux-cachefs
Cc: Jeff Layton, linux-kernel, David Howells, houtao1, linux-fsdevel,
linux-erofs
From: Hou Tao <houtao1@huawei.com>
The freeing of relinquished volume will wake up the pending volume
acquisition by using wake_up_bit(), however it is mismatched with
wait_var_event() used in fscache_wait_on_volume_collision() and it will
never wake up the waiter in the wait-queue because these two functions
operate on different wait-queues.
According to the implementation in fscache_wait_on_volume_collision(),
if the wake-up of pending acquisition is delayed longer than 20 seconds
(e.g., due to the delay of on-demand fd closing), the first
wait_var_event_timeout() will timeout and the following wait_var_event()
will hang forever as shown below:
FS-Cache: Potential volume collision new=00000024 old=00000022
......
INFO: task mount:1148 blocked for more than 122 seconds.
Not tainted 6.1.0-rc6+ #1
task:mount state:D stack:0 pid:1148 ppid:1
Call Trace:
<TASK>
__schedule+0x2f6/0xb80
schedule+0x67/0xe0
fscache_wait_on_volume_collision.cold+0x80/0x82
__fscache_acquire_volume+0x40d/0x4e0
erofs_fscache_register_volume+0x51/0xe0 [erofs]
erofs_fscache_register_fs+0x19c/0x240 [erofs]
erofs_fc_fill_super+0x746/0xaf0 [erofs]
vfs_get_super+0x7d/0x100
get_tree_nodev+0x16/0x20
erofs_fc_get_tree+0x20/0x30 [erofs]
vfs_get_tree+0x24/0xb0
path_mount+0x2fa/0xa90
do_mount+0x7c/0xa0
__x64_sys_mount+0x8b/0xe0
do_syscall_64+0x30/0x60
entry_SYSCALL_64_after_hwframe+0x46/0xb0
Considering that wake_up_bit() is more selective, so fix it by using
wait_on_bit() instead of wait_var_event() to wait for the freeing of
relinquished volume. In addition because waitqueue_active() is used in
wake_up_bit() and clear_bit() doesn't imply any memory barrier, use
clear_and_wake_up_bit() to add the missing memory barrier between
cursor->flags and waitqueue_active().
Fixes: 62ab63352350 ("fscache: Implement volume registration")
Reviewed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Hou Tao <houtao1@huawei.com>
---
fs/fscache/volume.c | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/fs/fscache/volume.c b/fs/fscache/volume.c
index ab8ceddf9efa..903af9d85f8b 100644
--- a/fs/fscache/volume.c
+++ b/fs/fscache/volume.c
@@ -141,13 +141,14 @@ static bool fscache_is_acquire_pending(struct fscache_volume *volume)
static void fscache_wait_on_volume_collision(struct fscache_volume *candidate,
unsigned int collidee_debug_id)
{
- wait_var_event_timeout(&candidate->flags,
- !fscache_is_acquire_pending(candidate), 20 * HZ);
+ wait_on_bit_timeout(&candidate->flags, FSCACHE_VOLUME_ACQUIRE_PENDING,
+ TASK_UNINTERRUPTIBLE, 20 * HZ);
if (fscache_is_acquire_pending(candidate)) {
pr_notice("Potential volume collision new=%08x old=%08x",
candidate->debug_id, collidee_debug_id);
fscache_stat(&fscache_n_volumes_collision);
- wait_var_event(&candidate->flags, !fscache_is_acquire_pending(candidate));
+ wait_on_bit(&candidate->flags, FSCACHE_VOLUME_ACQUIRE_PENDING,
+ TASK_UNINTERRUPTIBLE);
}
}
@@ -347,8 +348,8 @@ static void fscache_wake_pending_volume(struct fscache_volume *volume,
hlist_bl_for_each_entry(cursor, p, h, hash_link) {
if (fscache_volume_same(cursor, volume)) {
fscache_see_volume(cursor, fscache_volume_see_hash_wake);
- clear_bit(FSCACHE_VOLUME_ACQUIRE_PENDING, &cursor->flags);
- wake_up_bit(&cursor->flags, FSCACHE_VOLUME_ACQUIRE_PENDING);
+ clear_and_wake_up_bit(FSCACHE_VOLUME_ACQUIRE_PENDING,
+ &cursor->flags);
return;
}
}
--
2.29.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH v3 2/2] fscache: Use clear_and_wake_up_bit() in fscache_create_volume_work()
2023-01-13 11:52 [PATCH v3 0/2] Fixes for fscache volume operations Hou Tao
2023-01-13 11:52 ` [PATCH v3 1/2] fscache: Use wait_on_bit() to wait for the freeing of relinquished volume Hou Tao
@ 2023-01-13 11:52 ` Hou Tao
2023-01-13 15:28 ` Jeff Layton
2023-01-30 10:56 ` [PATCH v3 0/2] Fixes for fscache volume operations Hou Tao
2 siblings, 1 reply; 6+ messages in thread
From: Hou Tao @ 2023-01-13 11:52 UTC (permalink / raw)
To: linux-cachefs
Cc: Jeff Layton, linux-kernel, David Howells, houtao1, linux-fsdevel,
linux-erofs
From: Hou Tao <houtao1@huawei.com>
fscache_create_volume_work() uses wake_up_bit() to wake up the processes
which are waiting for the completion of volume creation. According to
comments in wake_up_bit() and waitqueue_active(), an extra smp_mb() is
needed to guarantee the memory order between FSCACHE_VOLUME_CREATING
flag and waitqueue_active() before invoking wake_up_bit().
Fixing it by using clear_and_wake_up_bit() to add the missing memory
barrier.
Reviewed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
Signed-off-by: Hou Tao <houtao1@huawei.com>
---
fs/fscache/volume.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/fs/fscache/volume.c b/fs/fscache/volume.c
index 903af9d85f8b..cdf991bdd9de 100644
--- a/fs/fscache/volume.c
+++ b/fs/fscache/volume.c
@@ -280,8 +280,7 @@ static void fscache_create_volume_work(struct work_struct *work)
fscache_end_cache_access(volume->cache,
fscache_access_acquire_volume_end);
- clear_bit_unlock(FSCACHE_VOLUME_CREATING, &volume->flags);
- wake_up_bit(&volume->flags, FSCACHE_VOLUME_CREATING);
+ clear_and_wake_up_bit(FSCACHE_VOLUME_CREATING, &volume->flags);
fscache_put_volume(volume, fscache_volume_put_create_work);
}
--
2.29.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v3 1/2] fscache: Use wait_on_bit() to wait for the freeing of relinquished volume
2023-01-13 11:52 ` [PATCH v3 1/2] fscache: Use wait_on_bit() to wait for the freeing of relinquished volume Hou Tao
@ 2023-01-13 15:27 ` Jeff Layton
0 siblings, 0 replies; 6+ messages in thread
From: Jeff Layton @ 2023-01-13 15:27 UTC (permalink / raw)
To: Hou Tao, linux-cachefs
Cc: linux-kernel, David Howells, houtao1, linux-fsdevel, linux-erofs
On Fri, 2023-01-13 at 19:52 +0800, Hou Tao wrote:
> From: Hou Tao <houtao1@huawei.com>
>
> The freeing of relinquished volume will wake up the pending volume
> acquisition by using wake_up_bit(), however it is mismatched with
> wait_var_event() used in fscache_wait_on_volume_collision() and it will
> never wake up the waiter in the wait-queue because these two functions
> operate on different wait-queues.
>
> According to the implementation in fscache_wait_on_volume_collision(),
> if the wake-up of pending acquisition is delayed longer than 20 seconds
> (e.g., due to the delay of on-demand fd closing), the first
> wait_var_event_timeout() will timeout and the following wait_var_event()
> will hang forever as shown below:
>
> FS-Cache: Potential volume collision new=00000024 old=00000022
> ......
> INFO: task mount:1148 blocked for more than 122 seconds.
> Not tainted 6.1.0-rc6+ #1
> task:mount state:D stack:0 pid:1148 ppid:1
> Call Trace:
> <TASK>
> __schedule+0x2f6/0xb80
> schedule+0x67/0xe0
> fscache_wait_on_volume_collision.cold+0x80/0x82
> __fscache_acquire_volume+0x40d/0x4e0
> erofs_fscache_register_volume+0x51/0xe0 [erofs]
> erofs_fscache_register_fs+0x19c/0x240 [erofs]
> erofs_fc_fill_super+0x746/0xaf0 [erofs]
> vfs_get_super+0x7d/0x100
> get_tree_nodev+0x16/0x20
> erofs_fc_get_tree+0x20/0x30 [erofs]
> vfs_get_tree+0x24/0xb0
> path_mount+0x2fa/0xa90
> do_mount+0x7c/0xa0
> __x64_sys_mount+0x8b/0xe0
> do_syscall_64+0x30/0x60
> entry_SYSCALL_64_after_hwframe+0x46/0xb0
>
> Considering that wake_up_bit() is more selective, so fix it by using
> wait_on_bit() instead of wait_var_event() to wait for the freeing of
> relinquished volume. In addition because waitqueue_active() is used in
> wake_up_bit() and clear_bit() doesn't imply any memory barrier, use
> clear_and_wake_up_bit() to add the missing memory barrier between
> cursor->flags and waitqueue_active().
>
> Fixes: 62ab63352350 ("fscache: Implement volume registration")
> Reviewed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
> Signed-off-by: Hou Tao <houtao1@huawei.com>
> ---
> fs/fscache/volume.c | 11 ++++++-----
> 1 file changed, 6 insertions(+), 5 deletions(-)
>
> diff --git a/fs/fscache/volume.c b/fs/fscache/volume.c
> index ab8ceddf9efa..903af9d85f8b 100644
> --- a/fs/fscache/volume.c
> +++ b/fs/fscache/volume.c
> @@ -141,13 +141,14 @@ static bool fscache_is_acquire_pending(struct fscache_volume *volume)
> static void fscache_wait_on_volume_collision(struct fscache_volume *candidate,
> unsigned int collidee_debug_id)
> {
> - wait_var_event_timeout(&candidate->flags,
> - !fscache_is_acquire_pending(candidate), 20 * HZ);
> + wait_on_bit_timeout(&candidate->flags, FSCACHE_VOLUME_ACQUIRE_PENDING,
> + TASK_UNINTERRUPTIBLE, 20 * HZ);
> if (fscache_is_acquire_pending(candidate)) {
> pr_notice("Potential volume collision new=%08x old=%08x",
> candidate->debug_id, collidee_debug_id);
> fscache_stat(&fscache_n_volumes_collision);
> - wait_var_event(&candidate->flags, !fscache_is_acquire_pending(candidate));
> + wait_on_bit(&candidate->flags, FSCACHE_VOLUME_ACQUIRE_PENDING,
> + TASK_UNINTERRUPTIBLE);
> }
> }
>
> @@ -347,8 +348,8 @@ static void fscache_wake_pending_volume(struct fscache_volume *volume,
> hlist_bl_for_each_entry(cursor, p, h, hash_link) {
> if (fscache_volume_same(cursor, volume)) {
> fscache_see_volume(cursor, fscache_volume_see_hash_wake);
> - clear_bit(FSCACHE_VOLUME_ACQUIRE_PENDING, &cursor->flags);
> - wake_up_bit(&cursor->flags, FSCACHE_VOLUME_ACQUIRE_PENDING);
> + clear_and_wake_up_bit(FSCACHE_VOLUME_ACQUIRE_PENDING,
> + &cursor->flags);
> return;
> }
> }
Reviewed-by: Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3 2/2] fscache: Use clear_and_wake_up_bit() in fscache_create_volume_work()
2023-01-13 11:52 ` [PATCH v3 2/2] fscache: Use clear_and_wake_up_bit() in fscache_create_volume_work() Hou Tao
@ 2023-01-13 15:28 ` Jeff Layton
0 siblings, 0 replies; 6+ messages in thread
From: Jeff Layton @ 2023-01-13 15:28 UTC (permalink / raw)
To: Hou Tao, linux-cachefs
Cc: linux-kernel, David Howells, houtao1, linux-fsdevel, linux-erofs
On Fri, 2023-01-13 at 19:52 +0800, Hou Tao wrote:
> From: Hou Tao <houtao1@huawei.com>
>
> fscache_create_volume_work() uses wake_up_bit() to wake up the processes
> which are waiting for the completion of volume creation. According to
> comments in wake_up_bit() and waitqueue_active(), an extra smp_mb() is
> needed to guarantee the memory order between FSCACHE_VOLUME_CREATING
> flag and waitqueue_active() before invoking wake_up_bit().
>
> Fixing it by using clear_and_wake_up_bit() to add the missing memory
> barrier.
>
> Reviewed-by: Jingbo Xu <jefflexu@linux.alibaba.com>
> Signed-off-by: Hou Tao <houtao1@huawei.com>
> ---
> fs/fscache/volume.c | 3 +--
> 1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/fs/fscache/volume.c b/fs/fscache/volume.c
> index 903af9d85f8b..cdf991bdd9de 100644
> --- a/fs/fscache/volume.c
> +++ b/fs/fscache/volume.c
> @@ -280,8 +280,7 @@ static void fscache_create_volume_work(struct work_struct *work)
> fscache_end_cache_access(volume->cache,
> fscache_access_acquire_volume_end);
>
> - clear_bit_unlock(FSCACHE_VOLUME_CREATING, &volume->flags);
> - wake_up_bit(&volume->flags, FSCACHE_VOLUME_CREATING);
> + clear_and_wake_up_bit(FSCACHE_VOLUME_CREATING, &volume->flags);
> fscache_put_volume(volume, fscache_volume_put_create_work);
> }
>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3 0/2] Fixes for fscache volume operations
2023-01-13 11:52 [PATCH v3 0/2] Fixes for fscache volume operations Hou Tao
2023-01-13 11:52 ` [PATCH v3 1/2] fscache: Use wait_on_bit() to wait for the freeing of relinquished volume Hou Tao
2023-01-13 11:52 ` [PATCH v3 2/2] fscache: Use clear_and_wake_up_bit() in fscache_create_volume_work() Hou Tao
@ 2023-01-30 10:56 ` Hou Tao
2 siblings, 0 replies; 6+ messages in thread
From: Hou Tao @ 2023-01-30 10:56 UTC (permalink / raw)
To: linux-cachefs, David Howells
Cc: Jeff Layton, linux-kernel, houtao1, linux-fsdevel, linux-erofs
Hi David,
Could you please pick it up for v6.2 ?
On 1/13/2023 7:52 PM, Hou Tao wrote:
> From: Hou Tao <houtao1@huawei.com>
>
> Hi,
>
> The patchset includes two fixes for fscache volume operations: patch 1
> fixes the hang problem during volume acquisition when the volume
> acquisition process waits for the freeing of relinquished volume, patch
> 2 adds the missing memory barrier in fscache_create_volume_work() and it
> is spotted through code review when checking whether or not these is
> missing smp_mb() before invoking wake_up_bit().
>
> Comments are always welcome.
>
> Chang Log:
> v3:
> * Use clear_and_wake_up_bit() helper (Suggested by Jingbo Xu)
> * Tidy up commit message and add Reviewed-by tag
>
> v2: https://listman.redhat.com/archives/linux-cachefs/2022-December/007402.html
> * rebased on v6.1-rc1
> * Patch 1: use wait_on_bit() instead (Suggested by David)
> * Patch 2: add the missing smp_mb() in fscache_create_volume_work()
>
> v1: https://listman.redhat.com/archives/linux-cachefs/2022-December/007384.html
>
>
> Hou Tao (2):
> fscache: Use wait_on_bit() to wait for the freeing of relinquished
> volume
> fscache: Use clear_and_wake_up_bit() in fscache_create_volume_work()
>
> fs/fscache/volume.c | 14 +++++++-------
> 1 file changed, 7 insertions(+), 7 deletions(-)
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-01-30 10:57 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-13 11:52 [PATCH v3 0/2] Fixes for fscache volume operations Hou Tao
2023-01-13 11:52 ` [PATCH v3 1/2] fscache: Use wait_on_bit() to wait for the freeing of relinquished volume Hou Tao
2023-01-13 15:27 ` Jeff Layton
2023-01-13 11:52 ` [PATCH v3 2/2] fscache: Use clear_and_wake_up_bit() in fscache_create_volume_work() Hou Tao
2023-01-13 15:28 ` Jeff Layton
2023-01-30 10:56 ` [PATCH v3 0/2] Fixes for fscache volume operations Hou Tao
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).