All of lore.kernel.org
 help / color / mirror / Atom feed
* [Ocfs2-devel] [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head
@ 2021-10-05 11:54 Gautham Ananthakrishna
  2021-10-08  6:39 ` Joseph Qi
  0 siblings, 1 reply; 6+ messages in thread
From: Gautham Ananthakrishna @ 2021-10-05 11:54 UTC (permalink / raw)
  To: ocfs2-devel; +Cc: rajesh.sivaramasubramaniom, gautham.ananthakrishna

Encountered a race between ocfs2_test_bg_bit_allocatable() and
jbd2_journal_put_journal_head() resulting in the below vmcore.

PID: 106879  TASK: ffff880244ba9c00  CPU: 2   COMMAND: "loop3"
 0 [ffff8802435ff1c0] panic at ffffffff816ed175
 1 [ffff8802435ff240] oops_end at ffffffff8101a7c9
 2 [ffff8802435ff270] no_context at ffffffff8106eccf
 3 [ffff8802435ff2e0] __bad_area_nosemaphore at ffffffff8106ef9d
 4 [ffff8802435ff330] bad_area_nosemaphore at ffffffff8106f143
 5 [ffff8802435ff340] __do_page_fault at ffffffff8106f80b
 6 [ffff8802435ff3a0] do_page_fault at ffffffff8106fc2f
 7 [ffff8802435ff3e0] page_fault at ffffffff816fd667
    [exception RIP: ocfs2_block_group_find_clear_bits+316]
    RIP: ffffffffc11ef6fc  RSP: ffff8802435ff498  RFLAGS: 00010206
    RAX: 0000000000003918  RBX: 0000000000000001  RCX: 0000000000000018
    RDX: 0000000000003918  RSI: 0000000000000000  RDI: ffff880060194040
    RBP: ffff8802435ff4f8   R8: ffffffffff000000   R9: ffffffffffffffff
    R10: ffff8802435ff730  R11: ffff8802a94e5800  R12: 0000000000000007
    R13: 0000000000007e00  R14: 0000000000003918  R15: ffff88017c973a28
    ORIG_RAX: ffffffffffffffff  CS: e030  SS: e02b
 8 [ffff8802435ff490] ocfs2_block_group_find_clear_bits at ffffffffc11ef680 [ocfs2]
 9 [ffff8802435ff500] ocfs2_cluster_group_search at ffffffffc11ef916 [ocfs2]
10 [ffff8802435ff580] ocfs2_search_chain at ffffffffc11f0fb6 [ocfs2]
11 [ffff8802435ff660] ocfs2_claim_suballoc_bits at ffffffffc11f1b1b [ocfs2]
12 [ffff8802435ff6f0] __ocfs2_claim_clusters at ffffffffc11f32cb [ocfs2]
13 [ffff8802435ff770] ocfs2_claim_clusters at ffffffffc11f5caf [ocfs2]
14 [ffff8802435ff780] ocfs2_local_alloc_slide_window at ffffffffc11cc0db [ocfs2]
15 [ffff8802435ff820] ocfs2_reserve_local_alloc_bits at ffffffffc11ce53f [ocfs2]
16 [ffff8802435ff890] ocfs2_reserve_clusters_with_limit at ffffffffc11f59b5 [ocfs2]
17 [ffff8802435ff8e0] ocfs2_reserve_clusters at ffffffffc11f5c88 [ocfs2]
18 [ffff8802435ff8f0] ocfs2_lock_refcount_allocators at ffffffffc11dc169 [ocfs2]
19 [ffff8802435ff960] ocfs2_make_clusters_writable at ffffffffc11e4274 [ocfs2]
20 [ffff8802435ffa50] ocfs2_replace_cow at ffffffffc11e4df1 [ocfs2]
21 [ffff8802435ffac0] ocfs2_refcount_cow at ffffffffc11e54b1 [ocfs2]
22 [ffff8802435ffb80] ocfs2_file_write_iter at ffffffffc11bf8f4 [ocfs2]
23 [ffff8802435ffcd0] lo_rw_aio at ffffffff814a1b5d
24 [ffff8802435ffd80] loop_queue_work at ffffffff814a2802
25 [ffff8802435ffe60] kthread_worker_fn at ffffffff810a80d2
26 [ffff8802435ffec0] kthread at ffffffff810a7afb
27 [ffff8802435fff50] ret_from_fork at ffffffff816f7da1

When ocfs2_test_bg_bit_allocatable() called bh2jh(bg_bh), the bg_bh->b_private
NULL as jbd2_journal_put_journal_head() raced and released the jounal head
from the buffer head. Needed to take bit lock for the bit 'BH_JournalHead'
to fix this race.

Signed-off-by: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>
---
 fs/ocfs2/suballoc.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/fs/ocfs2/suballoc.c b/fs/ocfs2/suballoc.c
index 8521942..0e4e11b 100644
--- a/fs/ocfs2/suballoc.c
+++ b/fs/ocfs2/suballoc.c
@@ -1256,8 +1256,11 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
 	if (ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap))
 		return 0;
 
-	if (!buffer_jbd(bg_bh))
+	jbd_lock_bh_journal_head(bg_bh);
+	if (!buffer_jbd(bg_bh)){
+		jbd_unlock_bh_journal_head(bg_bh);
 		return 1;
+	}
 
 	jh = bh2jh(bg_bh);
 	spin_lock(&jh->b_state_lock);
@@ -1267,6 +1270,7 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
 	else
 		ret = 1;
 	spin_unlock(&jh->b_state_lock);
+	jbd_unlock_bh_journal_head(bg_bh);
 
 	return ret;
 }
-- 
1.8.3.1


_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

^ permalink raw reply related	[flat|nested] 6+ messages in thread
* Re: [Ocfs2-devel] [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head
  2021-10-05 11:54 [Ocfs2-devel] [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head Gautham Ananthakrishna
@ 2021-10-08  6:39 ` Joseph Qi
  2021-10-13  4:08   ` Gautham Ananthakrishna
  0 siblings, 1 reply; 6+ messages in thread
From: Joseph Qi @ 2021-10-08  6:39 UTC (permalink / raw)
  To: Gautham Ananthakrishna, ocfs2-devel; +Cc: rajesh.sivaramasubramaniom

Hi Gautham,

On 10/5/21 7:54 PM, Gautham Ananthakrishna wrote:
> Encountered a race between ocfs2_test_bg_bit_allocatable() and
> jbd2_journal_put_journal_head() resulting in the below vmcore.
> 
> PID: 106879  TASK: ffff880244ba9c00  CPU: 2   COMMAND: "loop3"
>  0 [ffff8802435ff1c0] panic at ffffffff816ed175
>  1 [ffff8802435ff240] oops_end at ffffffff8101a7c9
>  2 [ffff8802435ff270] no_context at ffffffff8106eccf
>  3 [ffff8802435ff2e0] __bad_area_nosemaphore at ffffffff8106ef9d
>  4 [ffff8802435ff330] bad_area_nosemaphore at ffffffff8106f143
>  5 [ffff8802435ff340] __do_page_fault at ffffffff8106f80b
>  6 [ffff8802435ff3a0] do_page_fault at ffffffff8106fc2f
>  7 [ffff8802435ff3e0] page_fault at ffffffff816fd667
>     [exception RIP: ocfs2_block_group_find_clear_bits+316]
>     RIP: ffffffffc11ef6fc  RSP: ffff8802435ff498  RFLAGS: 00010206
>     RAX: 0000000000003918  RBX: 0000000000000001  RCX: 0000000000000018
>     RDX: 0000000000003918  RSI: 0000000000000000  RDI: ffff880060194040
>     RBP: ffff8802435ff4f8   R8: ffffffffff000000   R9: ffffffffffffffff
>     R10: ffff8802435ff730  R11: ffff8802a94e5800  R12: 0000000000000007
>     R13: 0000000000007e00  R14: 0000000000003918  R15: ffff88017c973a28
>     ORIG_RAX: ffffffffffffffff  CS: e030  SS: e02b
>  8 [ffff8802435ff490] ocfs2_block_group_find_clear_bits at ffffffffc11ef680 [ocfs2]
>  9 [ffff8802435ff500] ocfs2_cluster_group_search at ffffffffc11ef916 [ocfs2]
> 10 [ffff8802435ff580] ocfs2_search_chain at ffffffffc11f0fb6 [ocfs2]
> 11 [ffff8802435ff660] ocfs2_claim_suballoc_bits at ffffffffc11f1b1b [ocfs2]
> 12 [ffff8802435ff6f0] __ocfs2_claim_clusters at ffffffffc11f32cb [ocfs2]
> 13 [ffff8802435ff770] ocfs2_claim_clusters at ffffffffc11f5caf [ocfs2]
> 14 [ffff8802435ff780] ocfs2_local_alloc_slide_window at ffffffffc11cc0db [ocfs2]
> 15 [ffff8802435ff820] ocfs2_reserve_local_alloc_bits at ffffffffc11ce53f [ocfs2]
> 16 [ffff8802435ff890] ocfs2_reserve_clusters_with_limit at ffffffffc11f59b5 [ocfs2]
> 17 [ffff8802435ff8e0] ocfs2_reserve_clusters at ffffffffc11f5c88 [ocfs2]
> 18 [ffff8802435ff8f0] ocfs2_lock_refcount_allocators at ffffffffc11dc169 [ocfs2]
> 19 [ffff8802435ff960] ocfs2_make_clusters_writable at ffffffffc11e4274 [ocfs2]
> 20 [ffff8802435ffa50] ocfs2_replace_cow at ffffffffc11e4df1 [ocfs2]
> 21 [ffff8802435ffac0] ocfs2_refcount_cow at ffffffffc11e54b1 [ocfs2]
> 22 [ffff8802435ffb80] ocfs2_file_write_iter at ffffffffc11bf8f4 [ocfs2]
> 23 [ffff8802435ffcd0] lo_rw_aio at ffffffff814a1b5d
> 24 [ffff8802435ffd80] loop_queue_work at ffffffff814a2802
> 25 [ffff8802435ffe60] kthread_worker_fn at ffffffff810a80d2
> 26 [ffff8802435ffec0] kthread at ffffffff810a7afb
> 27 [ffff8802435fff50] ret_from_fork at ffffffff816f7da1
> 
> When ocfs2_test_bg_bit_allocatable() called bh2jh(bg_bh), the bg_bh->b_private
> NULL as jbd2_journal_put_journal_head() raced and released the jounal head
> from the buffer head. Needed to take bit lock for the bit 'BH_JournalHead'
> to fix this race.
> 
> Signed-off-by: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>
> ---
>  fs/ocfs2/suballoc.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/ocfs2/suballoc.c b/fs/ocfs2/suballoc.c
> index 8521942..0e4e11b 100644
> --- a/fs/ocfs2/suballoc.c
> +++ b/fs/ocfs2/suballoc.c
> @@ -1256,8 +1256,11 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
>  	if (ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap))
>  		return 0;
>  
> -	if (!buffer_jbd(bg_bh))
> +	jbd_lock_bh_journal_head(bg_bh);
> +	if (!buffer_jbd(bg_bh)){
> +		jbd_unlock_bh_journal_head(bg_bh);
>  		return 1;
> +	}

Seems !buffer_jbd() case we don't have to lock bit BH_JournalHead.

Thanks,
Joseph

>  
>  	jh = bh2jh(bg_bh);
>  	spin_lock(&jh->b_state_lock);
> @@ -1267,6 +1270,7 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
>  	else
>  		ret = 1;
>  	spin_unlock(&jh->b_state_lock);
> +	jbd_unlock_bh_journal_head(bg_bh);
>  
>  	return ret;
>  }
> 

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [Ocfs2-devel] [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head
  2021-10-08  6:39 ` Joseph Qi
@ 2021-10-13  4:08   ` Gautham Ananthakrishna
  2021-10-13  8:08     ` Joseph Qi
  0 siblings, 1 reply; 6+ messages in thread
From: Gautham Ananthakrishna @ 2021-10-13  4:08 UTC (permalink / raw)
  To: Joseph Qi, ocfs2-devel; +Cc: Rajesh Sivaramasubramaniom

Hi Joseph.

In jbd2_journal_put_journal_head(), we decrement jh->b_jcount before calling __journal_remove_journal_head().

However in any of the calling functions of ocfs2_test_bg_bit_allocatable(), we dont increment jh->b_jcount.
Because of this, __journal_remove_journal_head() raced and set bh->b_private to NULL in ocfs2_test_bg_bit_allocatable().
This race happened after we checked "if (!buffer_jbd(bg_bh))" but before we referenced b_privatelater. This is how we go the
stack described in this patch. Hence we need to lock bit BH_JournalHead while checking ""if (!buffer_jbd(bg_bh))" as well as referencing b_private.

Thanks,
Gautham.

-----Original Message-----
From: Joseph Qi <joseph.qi@linux.alibaba.com> 
Sent: Friday, October 8, 2021 12:10 PM
To: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>; ocfs2-devel@oss.oracle.com
Cc: Junxiao Bi <junxiao.bi@oracle.com>; Rajesh Sivaramasubramaniom <rajesh.sivaramasubramaniom@oracle.com>
Subject: Re: [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head

Hi Gautham,

On 10/5/21 7:54 PM, Gautham Ananthakrishna wrote:
> Encountered a race between ocfs2_test_bg_bit_allocatable() and
> jbd2_journal_put_journal_head() resulting in the below vmcore.
> 
> PID: 106879  TASK: ffff880244ba9c00  CPU: 2   COMMAND: "loop3"
>  0 [ffff8802435ff1c0] panic at ffffffff816ed175
>  1 [ffff8802435ff240] oops_end at ffffffff8101a7c9
>  2 [ffff8802435ff270] no_context at ffffffff8106eccf
>  3 [ffff8802435ff2e0] __bad_area_nosemaphore at ffffffff8106ef9d
>  4 [ffff8802435ff330] bad_area_nosemaphore at ffffffff8106f143
>  5 [ffff8802435ff340] __do_page_fault at ffffffff8106f80b
>  6 [ffff8802435ff3a0] do_page_fault at ffffffff8106fc2f
>  7 [ffff8802435ff3e0] page_fault at ffffffff816fd667
>     [exception RIP: ocfs2_block_group_find_clear_bits+316]
>     RIP: ffffffffc11ef6fc  RSP: ffff8802435ff498  RFLAGS: 00010206
>     RAX: 0000000000003918  RBX: 0000000000000001  RCX: 0000000000000018
>     RDX: 0000000000003918  RSI: 0000000000000000  RDI: ffff880060194040
>     RBP: ffff8802435ff4f8   R8: ffffffffff000000   R9: ffffffffffffffff
>     R10: ffff8802435ff730  R11: ffff8802a94e5800  R12: 0000000000000007
>     R13: 0000000000007e00  R14: 0000000000003918  R15: ffff88017c973a28
>     ORIG_RAX: ffffffffffffffff  CS: e030  SS: e02b
>  8 [ffff8802435ff490] ocfs2_block_group_find_clear_bits at 
> ffffffffc11ef680 [ocfs2]
>  9 [ffff8802435ff500] ocfs2_cluster_group_search at ffffffffc11ef916 
> [ocfs2]
> 10 [ffff8802435ff580] ocfs2_search_chain at ffffffffc11f0fb6 [ocfs2]
> 11 [ffff8802435ff660] ocfs2_claim_suballoc_bits at ffffffffc11f1b1b 
> [ocfs2]
> 12 [ffff8802435ff6f0] __ocfs2_claim_clusters at ffffffffc11f32cb 
> [ocfs2]
> 13 [ffff8802435ff770] ocfs2_claim_clusters at ffffffffc11f5caf [ocfs2]
> 14 [ffff8802435ff780] ocfs2_local_alloc_slide_window at 
> ffffffffc11cc0db [ocfs2]
> 15 [ffff8802435ff820] ocfs2_reserve_local_alloc_bits at 
> ffffffffc11ce53f [ocfs2]
> 16 [ffff8802435ff890] ocfs2_reserve_clusters_with_limit at 
> ffffffffc11f59b5 [ocfs2]
> 17 [ffff8802435ff8e0] ocfs2_reserve_clusters at ffffffffc11f5c88 
> [ocfs2]
> 18 [ffff8802435ff8f0] ocfs2_lock_refcount_allocators at 
> ffffffffc11dc169 [ocfs2]
> 19 [ffff8802435ff960] ocfs2_make_clusters_writable at ffffffffc11e4274 
> [ocfs2]
> 20 [ffff8802435ffa50] ocfs2_replace_cow at ffffffffc11e4df1 [ocfs2]
> 21 [ffff8802435ffac0] ocfs2_refcount_cow at ffffffffc11e54b1 [ocfs2]
> 22 [ffff8802435ffb80] ocfs2_file_write_iter at ffffffffc11bf8f4 
> [ocfs2]
> 23 [ffff8802435ffcd0] lo_rw_aio at ffffffff814a1b5d
> 24 [ffff8802435ffd80] loop_queue_work at ffffffff814a2802
> 25 [ffff8802435ffe60] kthread_worker_fn at ffffffff810a80d2
> 26 [ffff8802435ffec0] kthread at ffffffff810a7afb
> 27 [ffff8802435fff50] ret_from_fork at ffffffff816f7da1
> 
> When ocfs2_test_bg_bit_allocatable() called bh2jh(bg_bh), the 
> bg_bh->b_private NULL as jbd2_journal_put_journal_head() raced and 
> released the jounal head from the buffer head. Needed to take bit lock for the bit 'BH_JournalHead'
> to fix this race.
> 
> Signed-off-by: Gautham Ananthakrishna 
> <gautham.ananthakrishna@oracle.com>
> ---
>  fs/ocfs2/suballoc.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/ocfs2/suballoc.c b/fs/ocfs2/suballoc.c index 
> 8521942..0e4e11b 100644
> --- a/fs/ocfs2/suballoc.c
> +++ b/fs/ocfs2/suballoc.c
> @@ -1256,8 +1256,11 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
>  	if (ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap))
>  		return 0;
>  
> -	if (!buffer_jbd(bg_bh))
> +	jbd_lock_bh_journal_head(bg_bh);
> +	if (!buffer_jbd(bg_bh)){
> +		jbd_unlock_bh_journal_head(bg_bh);
>  		return 1;
> +	}

Seems !buffer_jbd() case we don't have to lock bit BH_JournalHead.

Thanks,
Joseph

>  
>  	jh = bh2jh(bg_bh);
>  	spin_lock(&jh->b_state_lock);
> @@ -1267,6 +1270,7 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
>  	else
>  		ret = 1;
>  	spin_unlock(&jh->b_state_lock);
> +	jbd_unlock_bh_journal_head(bg_bh);
>  
>  	return ret;
>  }
> 
_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [Ocfs2-devel] [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head
  2021-10-13  4:08   ` Gautham Ananthakrishna
@ 2021-10-13  8:08     ` Joseph Qi
  2021-10-19 14:14       ` Gautham Ananthakrishna
  0 siblings, 1 reply; 6+ messages in thread
From: Joseph Qi @ 2021-10-13  8:08 UTC (permalink / raw)
  To: Gautham Ananthakrishna, ocfs2-devel; +Cc: Rajesh Sivaramasubramaniom



On 10/13/21 12:08 PM, Gautham Ananthakrishna wrote:
> Hi Joseph.
> 
> In jbd2_journal_put_journal_head(), we decrement jh->b_jcount before calling __journal_remove_journal_head().
> 
> However in any of the calling functions of ocfs2_test_bg_bit_allocatable(), we dont increment jh->b_jcount.
> Because of this, __journal_remove_journal_head() raced and set bh->b_private to NULL in ocfs2_test_bg_bit_allocatable().

Agree.

> This race happened after we checked "if (!buffer_jbd(bg_bh))" but before we referenced b_privatelater. This is how we go the
> stack described in this patch. Hence we need to lock bit BH_JournalHead while checking ""if (!buffer_jbd(bg_bh))" as well as referencing b_private.

What I mean is we can still keep !buffer_jbd(bg_bh) as 'fast path'.
So the code may be like:

if (!buffer_jbd(bg_bh))
	return 1;

jbd_lock_bh_journal_head(bg_bh);
if (buffer_jbd(bg_bh)) {
	jh = bh2jh(bg_bh);
	...
}
jbd_unlock_bh_journal_head(bg_bh);

Thanks,
Joseph	


> 
> Thanks,
> Gautham.
> 
> -----Original Message-----
> From: Joseph Qi <joseph.qi@linux.alibaba.com> 
> Sent: Friday, October 8, 2021 12:10 PM
> To: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>; ocfs2-devel@oss.oracle.com
> Cc: Junxiao Bi <junxiao.bi@oracle.com>; Rajesh Sivaramasubramaniom <rajesh.sivaramasubramaniom@oracle.com>
> Subject: Re: [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head
> 
> Hi Gautham,
> 
> On 10/5/21 7:54 PM, Gautham Ananthakrishna wrote:
>> Encountered a race between ocfs2_test_bg_bit_allocatable() and
>> jbd2_journal_put_journal_head() resulting in the below vmcore.
>>
>> PID: 106879  TASK: ffff880244ba9c00  CPU: 2   COMMAND: "loop3"
>>  0 [ffff8802435ff1c0] panic at ffffffff816ed175
>>  1 [ffff8802435ff240] oops_end at ffffffff8101a7c9
>>  2 [ffff8802435ff270] no_context at ffffffff8106eccf
>>  3 [ffff8802435ff2e0] __bad_area_nosemaphore at ffffffff8106ef9d
>>  4 [ffff8802435ff330] bad_area_nosemaphore at ffffffff8106f143
>>  5 [ffff8802435ff340] __do_page_fault at ffffffff8106f80b
>>  6 [ffff8802435ff3a0] do_page_fault at ffffffff8106fc2f
>>  7 [ffff8802435ff3e0] page_fault at ffffffff816fd667
>>     [exception RIP: ocfs2_block_group_find_clear_bits+316]
>>     RIP: ffffffffc11ef6fc  RSP: ffff8802435ff498  RFLAGS: 00010206
>>     RAX: 0000000000003918  RBX: 0000000000000001  RCX: 0000000000000018
>>     RDX: 0000000000003918  RSI: 0000000000000000  RDI: ffff880060194040
>>     RBP: ffff8802435ff4f8   R8: ffffffffff000000   R9: ffffffffffffffff
>>     R10: ffff8802435ff730  R11: ffff8802a94e5800  R12: 0000000000000007
>>     R13: 0000000000007e00  R14: 0000000000003918  R15: ffff88017c973a28
>>     ORIG_RAX: ffffffffffffffff  CS: e030  SS: e02b
>>  8 [ffff8802435ff490] ocfs2_block_group_find_clear_bits at 
>> ffffffffc11ef680 [ocfs2]
>>  9 [ffff8802435ff500] ocfs2_cluster_group_search at ffffffffc11ef916 
>> [ocfs2]
>> 10 [ffff8802435ff580] ocfs2_search_chain at ffffffffc11f0fb6 [ocfs2]
>> 11 [ffff8802435ff660] ocfs2_claim_suballoc_bits at ffffffffc11f1b1b 
>> [ocfs2]
>> 12 [ffff8802435ff6f0] __ocfs2_claim_clusters at ffffffffc11f32cb 
>> [ocfs2]
>> 13 [ffff8802435ff770] ocfs2_claim_clusters at ffffffffc11f5caf [ocfs2]
>> 14 [ffff8802435ff780] ocfs2_local_alloc_slide_window at 
>> ffffffffc11cc0db [ocfs2]
>> 15 [ffff8802435ff820] ocfs2_reserve_local_alloc_bits at 
>> ffffffffc11ce53f [ocfs2]
>> 16 [ffff8802435ff890] ocfs2_reserve_clusters_with_limit at 
>> ffffffffc11f59b5 [ocfs2]
>> 17 [ffff8802435ff8e0] ocfs2_reserve_clusters at ffffffffc11f5c88 
>> [ocfs2]
>> 18 [ffff8802435ff8f0] ocfs2_lock_refcount_allocators at 
>> ffffffffc11dc169 [ocfs2]
>> 19 [ffff8802435ff960] ocfs2_make_clusters_writable at ffffffffc11e4274 
>> [ocfs2]
>> 20 [ffff8802435ffa50] ocfs2_replace_cow at ffffffffc11e4df1 [ocfs2]
>> 21 [ffff8802435ffac0] ocfs2_refcount_cow at ffffffffc11e54b1 [ocfs2]
>> 22 [ffff8802435ffb80] ocfs2_file_write_iter at ffffffffc11bf8f4 
>> [ocfs2]
>> 23 [ffff8802435ffcd0] lo_rw_aio at ffffffff814a1b5d
>> 24 [ffff8802435ffd80] loop_queue_work at ffffffff814a2802
>> 25 [ffff8802435ffe60] kthread_worker_fn at ffffffff810a80d2
>> 26 [ffff8802435ffec0] kthread at ffffffff810a7afb
>> 27 [ffff8802435fff50] ret_from_fork at ffffffff816f7da1
>>
>> When ocfs2_test_bg_bit_allocatable() called bh2jh(bg_bh), the 
>> bg_bh->b_private NULL as jbd2_journal_put_journal_head() raced and 
>> released the jounal head from the buffer head. Needed to take bit lock for the bit 'BH_JournalHead'
>> to fix this race.
>>
>> Signed-off-by: Gautham Ananthakrishna 
>> <gautham.ananthakrishna@oracle.com>
>> ---
>>  fs/ocfs2/suballoc.c | 6 +++++-
>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/fs/ocfs2/suballoc.c b/fs/ocfs2/suballoc.c index 
>> 8521942..0e4e11b 100644
>> --- a/fs/ocfs2/suballoc.c
>> +++ b/fs/ocfs2/suballoc.c
>> @@ -1256,8 +1256,11 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
>>  	if (ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap))
>>  		return 0;
>>  
>> -	if (!buffer_jbd(bg_bh))
>> +	jbd_lock_bh_journal_head(bg_bh);
>> +	if (!buffer_jbd(bg_bh)){
>> +		jbd_unlock_bh_journal_head(bg_bh);
>>  		return 1;
>> +	}
> 
> Seems !buffer_jbd() case we don't have to lock bit BH_JournalHead.
> 
> Thanks,
> Joseph
> 
>>  
>>  	jh = bh2jh(bg_bh);
>>  	spin_lock(&jh->b_state_lock);
>> @@ -1267,6 +1270,7 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
>>  	else
>>  		ret = 1;
>>  	spin_unlock(&jh->b_state_lock);
>> +	jbd_unlock_bh_journal_head(bg_bh);
>>  
>>  	return ret;
>>  }
>>

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [Ocfs2-devel] [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head
  2021-10-13  8:08     ` Joseph Qi
@ 2021-10-19 14:14       ` Gautham Ananthakrishna
  0 siblings, 0 replies; 6+ messages in thread
From: Gautham Ananthakrishna @ 2021-10-19 14:14 UTC (permalink / raw)
  To: Joseph Qi, ocfs2-devel; +Cc: Rajesh Sivaramasubramaniom

Hi Joseph,

Yes.. we can have fast and slow paths. I will make changes and send V1 version of the patch.

Thanks,
Gautham.


-----Original Message-----
From: Joseph Qi <joseph.qi@linux.alibaba.com> 
Sent: Wednesday, October 13, 2021 1:38 PM
To: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>; ocfs2-devel@oss.oracle.com
Cc: Junxiao Bi <junxiao.bi@oracle.com>; Rajesh Sivaramasubramaniom <rajesh.sivaramasubramaniom@oracle.com>
Subject: Re: [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head



On 10/13/21 12:08 PM, Gautham Ananthakrishna wrote:
> Hi Joseph.
> 
> In jbd2_journal_put_journal_head(), we decrement jh->b_jcount before calling __journal_remove_journal_head().
> 
> However in any of the calling functions of ocfs2_test_bg_bit_allocatable(), we dont increment jh->b_jcount.
> Because of this, __journal_remove_journal_head() raced and set bh->b_private to NULL in ocfs2_test_bg_bit_allocatable().

Agree.

> This race happened after we checked "if (!buffer_jbd(bg_bh))" but 
> before we referenced b_privatelater. This is how we go the stack described in this patch. Hence we need to lock bit BH_JournalHead while checking ""if (!buffer_jbd(bg_bh))" as well as referencing b_private.

What I mean is we can still keep !buffer_jbd(bg_bh) as 'fast path'.
So the code may be like:

if (!buffer_jbd(bg_bh))
	return 1;

jbd_lock_bh_journal_head(bg_bh);
if (buffer_jbd(bg_bh)) {
	jh = bh2jh(bg_bh);
	...
}
jbd_unlock_bh_journal_head(bg_bh);

Thanks,
Joseph	


> 
> Thanks,
> Gautham.
> 
> -----Original Message-----
> From: Joseph Qi <joseph.qi@linux.alibaba.com>
> Sent: Friday, October 8, 2021 12:10 PM
> To: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>; 
> ocfs2-devel@oss.oracle.com
> Cc: Junxiao Bi <junxiao.bi@oracle.com>; Rajesh Sivaramasubramaniom 
> <rajesh.sivaramasubramaniom@oracle.com>
> Subject: Re: [PATCH RFC 1/1] ocfs2: race between searching chunks and 
> release journal_head from buffer_head
> 
> Hi Gautham,
> 
> On 10/5/21 7:54 PM, Gautham Ananthakrishna wrote:
>> Encountered a race between ocfs2_test_bg_bit_allocatable() and
>> jbd2_journal_put_journal_head() resulting in the below vmcore.
>>
>> PID: 106879  TASK: ffff880244ba9c00  CPU: 2   COMMAND: "loop3"
>>  0 [ffff8802435ff1c0] panic at ffffffff816ed175
>>  1 [ffff8802435ff240] oops_end at ffffffff8101a7c9
>>  2 [ffff8802435ff270] no_context at ffffffff8106eccf
>>  3 [ffff8802435ff2e0] __bad_area_nosemaphore at ffffffff8106ef9d
>>  4 [ffff8802435ff330] bad_area_nosemaphore at ffffffff8106f143
>>  5 [ffff8802435ff340] __do_page_fault at ffffffff8106f80b
>>  6 [ffff8802435ff3a0] do_page_fault at ffffffff8106fc2f
>>  7 [ffff8802435ff3e0] page_fault at ffffffff816fd667
>>     [exception RIP: ocfs2_block_group_find_clear_bits+316]
>>     RIP: ffffffffc11ef6fc  RSP: ffff8802435ff498  RFLAGS: 00010206
>>     RAX: 0000000000003918  RBX: 0000000000000001  RCX: 0000000000000018
>>     RDX: 0000000000003918  RSI: 0000000000000000  RDI: ffff880060194040
>>     RBP: ffff8802435ff4f8   R8: ffffffffff000000   R9: ffffffffffffffff
>>     R10: ffff8802435ff730  R11: ffff8802a94e5800  R12: 0000000000000007
>>     R13: 0000000000007e00  R14: 0000000000003918  R15: ffff88017c973a28
>>     ORIG_RAX: ffffffffffffffff  CS: e030  SS: e02b
>>  8 [ffff8802435ff490] ocfs2_block_group_find_clear_bits at
>> ffffffffc11ef680 [ocfs2]
>>  9 [ffff8802435ff500] ocfs2_cluster_group_search at ffffffffc11ef916 
>> [ocfs2]
>> 10 [ffff8802435ff580] ocfs2_search_chain at ffffffffc11f0fb6 [ocfs2]
>> 11 [ffff8802435ff660] ocfs2_claim_suballoc_bits at ffffffffc11f1b1b 
>> [ocfs2]
>> 12 [ffff8802435ff6f0] __ocfs2_claim_clusters at ffffffffc11f32cb 
>> [ocfs2]
>> 13 [ffff8802435ff770] ocfs2_claim_clusters at ffffffffc11f5caf 
>> [ocfs2]
>> 14 [ffff8802435ff780] ocfs2_local_alloc_slide_window at 
>> ffffffffc11cc0db [ocfs2]
>> 15 [ffff8802435ff820] ocfs2_reserve_local_alloc_bits at 
>> ffffffffc11ce53f [ocfs2]
>> 16 [ffff8802435ff890] ocfs2_reserve_clusters_with_limit at
>> ffffffffc11f59b5 [ocfs2]
>> 17 [ffff8802435ff8e0] ocfs2_reserve_clusters at ffffffffc11f5c88 
>> [ocfs2]
>> 18 [ffff8802435ff8f0] ocfs2_lock_refcount_allocators at
>> ffffffffc11dc169 [ocfs2]
>> 19 [ffff8802435ff960] ocfs2_make_clusters_writable at 
>> ffffffffc11e4274 [ocfs2]
>> 20 [ffff8802435ffa50] ocfs2_replace_cow at ffffffffc11e4df1 [ocfs2]
>> 21 [ffff8802435ffac0] ocfs2_refcount_cow at ffffffffc11e54b1 [ocfs2]
>> 22 [ffff8802435ffb80] ocfs2_file_write_iter at ffffffffc11bf8f4 
>> [ocfs2]
>> 23 [ffff8802435ffcd0] lo_rw_aio at ffffffff814a1b5d
>> 24 [ffff8802435ffd80] loop_queue_work at ffffffff814a2802
>> 25 [ffff8802435ffe60] kthread_worker_fn at ffffffff810a80d2
>> 26 [ffff8802435ffec0] kthread at ffffffff810a7afb
>> 27 [ffff8802435fff50] ret_from_fork at ffffffff816f7da1
>>
>> When ocfs2_test_bg_bit_allocatable() called bh2jh(bg_bh), the 
>> bg_bh->b_private NULL as jbd2_journal_put_journal_head() raced and 
>> released the jounal head from the buffer head. Needed to take bit lock for the bit 'BH_JournalHead'
>> to fix this race.
>>
>> Signed-off-by: Gautham Ananthakrishna 
>> <gautham.ananthakrishna@oracle.com>
>> ---
>>  fs/ocfs2/suballoc.c | 6 +++++-
>>  1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/fs/ocfs2/suballoc.c b/fs/ocfs2/suballoc.c index 
>> 8521942..0e4e11b 100644
>> --- a/fs/ocfs2/suballoc.c
>> +++ b/fs/ocfs2/suballoc.c
>> @@ -1256,8 +1256,11 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
>>  	if (ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap))
>>  		return 0;
>>  
>> -	if (!buffer_jbd(bg_bh))
>> +	jbd_lock_bh_journal_head(bg_bh);
>> +	if (!buffer_jbd(bg_bh)){
>> +		jbd_unlock_bh_journal_head(bg_bh);
>>  		return 1;
>> +	}
> 
> Seems !buffer_jbd() case we don't have to lock bit BH_JournalHead.
> 
> Thanks,
> Joseph
> 
>>  
>>  	jh = bh2jh(bg_bh);
>>  	spin_lock(&jh->b_state_lock);
>> @@ -1267,6 +1270,7 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
>>  	else
>>  		ret = 1;
>>  	spin_unlock(&jh->b_state_lock);
>> +	jbd_unlock_bh_journal_head(bg_bh);
>>  
>>  	return ret;
>>  }
>>
_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

^ permalink raw reply	[flat|nested] 6+ messages in thread
* [Ocfs2-devel] [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head
@ 2021-09-30  6:57 Gautham Ananthakrishna
  0 siblings, 0 replies; 6+ messages in thread
From: Gautham Ananthakrishna @ 2021-09-30  6:57 UTC (permalink / raw)
  To: ocfs2-devel; +Cc: rajesh.sivaramasubramaniom

Encountered a race between ocfs2_test_bg_bit_allocatable() and
jbd2_journal_put_journal_head() due to which ocfs2_test_bg_bit_allocatable()
dereferenced NULL pointer. When ocfs2_test_bg_bit_allocatable() called bh2jh(bg_bh),
the bg_bh->b_private was NULL as jbd2_journal_put_journal_head() raced and released
the journal_head from the buffer head. Needed to take bit lock for the bit
'BH_JournalHead' to fix this race.

Signed-off-by: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>
---
 fs/ocfs2/suballoc.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/fs/ocfs2/suballoc.c b/fs/ocfs2/suballoc.c
index 8521942..0e4e11b 100644
--- a/fs/ocfs2/suballoc.c
+++ b/fs/ocfs2/suballoc.c
@@ -1256,8 +1256,11 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
 	if (ocfs2_test_bit(nr, (unsigned long *)bg->bg_bitmap))
 		return 0;
 
-	if (!buffer_jbd(bg_bh))
+	jbd_lock_bh_journal_head(bg_bh);
+	if (!buffer_jbd(bg_bh)){
+		jbd_unlock_bh_journal_head(bg_bh);
 		return 1;
+	}
 
 	jh = bh2jh(bg_bh);
 	spin_lock(&jh->b_state_lock);
@@ -1267,6 +1270,7 @@ static int ocfs2_test_bg_bit_allocatable(struct buffer_head *bg_bh,
 	else
 		ret = 1;
 	spin_unlock(&jh->b_state_lock);
+	jbd_unlock_bh_journal_head(bg_bh);
 
 	return ret;
 }
-- 
1.8.3.1


_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel

^ permalink raw reply related	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-10-19 14:15 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-05 11:54 [Ocfs2-devel] [PATCH RFC 1/1] ocfs2: race between searching chunks and release journal_head from buffer_head Gautham Ananthakrishna
2021-10-08  6:39 ` Joseph Qi
2021-10-13  4:08   ` Gautham Ananthakrishna
2021-10-13  8:08     ` Joseph Qi
2021-10-19 14:14       ` Gautham Ananthakrishna
  -- strict thread matches above, loose matches on Subject: below --
2021-09-30  6:57 Gautham Ananthakrishna

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.