[PATCH] jbd2: Fix use after free in jbd2_log_do_checkpoint()

[PATCH] jbd2: Fix use after free in jbd2_log_do_checkpoint()

[Jan Kara]

The code cleaning transaction's lists of checkpoint buffers has a bug
where it increases bh refcount only after releasing
journal->j_list_lock. Thus the following race is possible:

CPU0					CPU1
jbd2_log_do_checkpoint()
					jbd2_journal_try_to_free_buffers()
					  __journal_try_to_free_buffer(bh)
  ...
  while (transaction->t_checkpoint_io_list)
  ...
    if (buffer_locked(bh)) {

<-- IO completes now, buffer gets unlocked -->

      spin_unlock(&journal->j_list_lock);
					    spin_lock(&journal->j_list_lock);
					    __jbd2_journal_remove_checkpoint(jh);
					    spin_unlock(&journal->j_list_lock);
					  try_to_free_buffers(page);
      get_bh(bh) <-- accesses freed bh

Fix the problem by grabbing bh reference before unlocking
journal->j_list_lock.

Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845
Fixes: be1158cc615fd723552f0d9912087423c7cadda5
Reported-by: syzbot+7f4a27091759e2fe7453@syzkaller.appspotmail.com
CC: stable@vger.kernel.org
Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/jbd2/checkpoint.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
index c125d662777c..26f8d7e46462 100644
--- a/fs/jbd2/checkpoint.c
+++ b/fs/jbd2/checkpoint.c
@@ -251,8 +251,8 @@ int jbd2_log_do_checkpoint(journal_t *journal)
 		bh = jh2bh(jh);
 
 		if (buffer_locked(bh)) {
-			spin_unlock(&journal->j_list_lock);
 			get_bh(bh);
+			spin_unlock(&journal->j_list_lock);
 			wait_on_buffer(bh);
 			/* the journal_head may have gone by now */
 			BUFFER_TRACE(bh, "brelse");
@@ -333,8 +333,8 @@ int jbd2_log_do_checkpoint(journal_t *journal)
 		jh = transaction->t_checkpoint_io_list;
 		bh = jh2bh(jh);
 		if (buffer_locked(bh)) {
-			spin_unlock(&journal->j_list_lock);
 			get_bh(bh);
+			spin_unlock(&journal->j_list_lock);
 			wait_on_buffer(bh);
 			/* the journal_head may have gone by now */
 			BUFFER_TRACE(bh, "brelse");
-- 
2.16.4

Re: [PATCH] jbd2: Fix use after free in jbd2_log_do_checkpoint()

[Lukas Czerner]

On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
> The code cleaning transaction's lists of checkpoint buffers has a bug
> where it increases bh refcount only after releasing
> journal->j_list_lock. Thus the following race is possible:
> 
> CPU0					CPU1
> jbd2_log_do_checkpoint()
> 					jbd2_journal_try_to_free_buffers()
> 					  __journal_try_to_free_buffer(bh)
>   ...
>   while (transaction->t_checkpoint_io_list)
>   ...
>     if (buffer_locked(bh)) {
> 
> <-- IO completes now, buffer gets unlocked -->
> 
>       spin_unlock(&journal->j_list_lock);
> 					    spin_lock(&journal->j_list_lock);
> 					    __jbd2_journal_remove_checkpoint(jh);
> 					    spin_unlock(&journal->j_list_lock);
> 					  try_to_free_buffers(page);
>       get_bh(bh) <-- accesses freed bh
> 
> Fix the problem by grabbing bh reference before unlocking
> journal->j_list_lock.

Hi Jan,

nice catch. The patch looks good, you can add

Reviewed-by: Lukas Czerner <lczerner@redhat.com>

Btw, do you by any chance have a reproducer for this ?

-Lukas

> 
> Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845
> Fixes: be1158cc615fd723552f0d9912087423c7cadda5
> Reported-by: syzbot+7f4a27091759e2fe7453@syzkaller.appspotmail.com
> CC: stable@vger.kernel.org
> Signed-off-by: Jan Kara <jack@suse.cz>
> ---
>  fs/jbd2/checkpoint.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
> index c125d662777c..26f8d7e46462 100644
> --- a/fs/jbd2/checkpoint.c
> +++ b/fs/jbd2/checkpoint.c
> @@ -251,8 +251,8 @@ int jbd2_log_do_checkpoint(journal_t *journal)
>  		bh = jh2bh(jh);
>  
>  		if (buffer_locked(bh)) {
> -			spin_unlock(&journal->j_list_lock);
>  			get_bh(bh);
> +			spin_unlock(&journal->j_list_lock);
>  			wait_on_buffer(bh);
>  			/* the journal_head may have gone by now */
>  			BUFFER_TRACE(bh, "brelse");
> @@ -333,8 +333,8 @@ int jbd2_log_do_checkpoint(journal_t *journal)
>  		jh = transaction->t_checkpoint_io_list;
>  		bh = jh2bh(jh);
>  		if (buffer_locked(bh)) {
> -			spin_unlock(&journal->j_list_lock);
>  			get_bh(bh);
> +			spin_unlock(&journal->j_list_lock);
>  			wait_on_buffer(bh);
>  			/* the journal_head may have gone by now */
>  			BUFFER_TRACE(bh, "brelse");
> -- 
> 2.16.4
> 

Re: [PATCH] jbd2: Fix use after free in jbd2_log_do_checkpoint()

[Jan Kara]

On Thu 04-10-18 13:50:12, Lukas Czerner wrote:
> On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
> > The code cleaning transaction's lists of checkpoint buffers has a bug
> > where it increases bh refcount only after releasing
> > journal->j_list_lock. Thus the following race is possible:
> > 
> > CPU0					CPU1
> > jbd2_log_do_checkpoint()
> > 					jbd2_journal_try_to_free_buffers()
> > 					  __journal_try_to_free_buffer(bh)
> >   ...
> >   while (transaction->t_checkpoint_io_list)
> >   ...
> >     if (buffer_locked(bh)) {
> > 
> > <-- IO completes now, buffer gets unlocked -->
> > 
> >       spin_unlock(&journal->j_list_lock);
> > 					    spin_lock(&journal->j_list_lock);
> > 					    __jbd2_journal_remove_checkpoint(jh);
> > 					    spin_unlock(&journal->j_list_lock);
> > 					  try_to_free_buffers(page);
> >       get_bh(bh) <-- accesses freed bh
> > 
> > Fix the problem by grabbing bh reference before unlocking
> > journal->j_list_lock.
> 
> Hi Jan,
> 
> nice catch. The patch looks good, you can add
> 
> Reviewed-by: Lukas Czerner <lczerner@redhat.com>
> 
> Btw, do you by any chance have a reproducer for this ?

No, syzbot hit it but the race window is really small so I don't think you
can create reasonably reliable reproducer...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH] jbd2: Fix use after free in jbd2_log_do_checkpoint()

[Greg KH]

On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
> The code cleaning transaction's lists of checkpoint buffers has a bug
> where it increases bh refcount only after releasing
> journal->j_list_lock. Thus the following race is possible:
> 
> CPU0					CPU1
> jbd2_log_do_checkpoint()
> 					jbd2_journal_try_to_free_buffers()
> 					  __journal_try_to_free_buffer(bh)
>   ...
>   while (transaction->t_checkpoint_io_list)
>   ...
>     if (buffer_locked(bh)) {
> 
> <-- IO completes now, buffer gets unlocked -->
> 
>       spin_unlock(&journal->j_list_lock);
> 					    spin_lock(&journal->j_list_lock);
> 					    __jbd2_journal_remove_checkpoint(jh);
> 					    spin_unlock(&journal->j_list_lock);
> 					  try_to_free_buffers(page);
>       get_bh(bh) <-- accesses freed bh
> 
> Fix the problem by grabbing bh reference before unlocking
> journal->j_list_lock.
> 
> Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845
> Fixes: be1158cc615fd723552f0d9912087423c7cadda5

Nit, this normally looks like:

Fixes: dc6e8d669cf5 ("jbd2: don't call get_bh() before calling __jbd2_journal_remove_checkpoint()")

And this is created by:
	git show -s --abbrev-commit --abbrev=12 --pretty=format:"%h (\"%s\")%n"

thanks

greg k-h

Re: [PATCH] jbd2: Fix use after free in jbd2_log_do_checkpoint()

[Jan Kara]

On Thu 04-10-18 08:44:02, Greg KH wrote:
> On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
> > The code cleaning transaction's lists of checkpoint buffers has a bug
> > where it increases bh refcount only after releasing
> > journal->j_list_lock. Thus the following race is possible:
> > 
> > CPU0					CPU1
> > jbd2_log_do_checkpoint()
> > 					jbd2_journal_try_to_free_buffers()
> > 					  __journal_try_to_free_buffer(bh)
> >   ...
> >   while (transaction->t_checkpoint_io_list)
> >   ...
> >     if (buffer_locked(bh)) {
> > 
> > <-- IO completes now, buffer gets unlocked -->
> > 
> >       spin_unlock(&journal->j_list_lock);
> > 					    spin_lock(&journal->j_list_lock);
> > 					    __jbd2_journal_remove_checkpoint(jh);
> > 					    spin_unlock(&journal->j_list_lock);
> > 					  try_to_free_buffers(page);
> >       get_bh(bh) <-- accesses freed bh
> > 
> > Fix the problem by grabbing bh reference before unlocking
> > journal->j_list_lock.
> > 
> > Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845
> > Fixes: be1158cc615fd723552f0d9912087423c7cadda5
> 
> Nit, this normally looks like:
> 
> Fixes: dc6e8d669cf5 ("jbd2: don't call get_bh() before calling __jbd2_journal_remove_checkpoint()")
> 
> And this is created by:
> 	git show -s --abbrev-commit --abbrev=12 --pretty=format:"%h (\"%s\")%n"

OK, ok, will do next time. The full git commit ID is faster to cut-n-paste
which shows how lazy I'm ;) 

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH] jbd2: Fix use after free in jbd2_log_do_checkpoint()

[Lukas Czerner]

On Thu, Oct 04, 2018 at 06:05:46PM +0200, Jan Kara wrote:
> On Thu 04-10-18 08:44:02, Greg KH wrote:
> > On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
> > > The code cleaning transaction's lists of checkpoint buffers has a bug
> > > where it increases bh refcount only after releasing
> > > journal->j_list_lock. Thus the following race is possible:
> > > 
> > > CPU0					CPU1
> > > jbd2_log_do_checkpoint()
> > > 					jbd2_journal_try_to_free_buffers()
> > > 					  __journal_try_to_free_buffer(bh)
> > >   ...
> > >   while (transaction->t_checkpoint_io_list)
> > >   ...
> > >     if (buffer_locked(bh)) {
> > > 
> > > <-- IO completes now, buffer gets unlocked -->
> > > 
> > >       spin_unlock(&journal->j_list_lock);
> > > 					    spin_lock(&journal->j_list_lock);
> > > 					    __jbd2_journal_remove_checkpoint(jh);
> > > 					    spin_unlock(&journal->j_list_lock);
> > > 					  try_to_free_buffers(page);
> > >       get_bh(bh) <-- accesses freed bh
> > > 
> > > Fix the problem by grabbing bh reference before unlocking
> > > journal->j_list_lock.
> > > 
> > > Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845
> > > Fixes: be1158cc615fd723552f0d9912087423c7cadda5
> > 
> > Nit, this normally looks like:
> > 
> > Fixes: dc6e8d669cf5 ("jbd2: don't call get_bh() before calling __jbd2_journal_remove_checkpoint()")
> > 
> > And this is created by:
> > 	git show -s --abbrev-commit --abbrev=12 --pretty=format:"%h (\"%s\")%n"
> 
> OK, ok, will do next time. The full git commit ID is faster to cut-n-paste
> which shows how lazy I'm ;) 

I have this in the .gitconfig

[pretty]
	fixes = Fixes: %h (\"%s\")

which helps. I think I found it in the docs somewhere.

-Lukas

> 
> 								Honza
> -- 
> Jan Kara <jack@suse.com>
> SUSE Labs, CR

Re: [PATCH] jbd2: Fix use after free in jbd2_log_do_checkpoint()

[Theodore Y. Ts'o]

On Thu, Oct 04, 2018 at 06:05:46PM +0200, Jan Kara wrote:
> 
> OK, ok, will do next time. The full git commit ID is faster to cut-n-paste
> which shows how lazy I'm ;)

No worires, I'll fix that up for you when I apply it.

   	    	     	     	     - Ted

Re: [PATCH] jbd2: Fix use after free in jbd2_log_do_checkpoint()

[Jan Kara]

On Thu 04-10-18 17:22:18, Theodore Y. Ts'o wrote:
> On Thu, Oct 04, 2018 at 06:05:46PM +0200, Jan Kara wrote:
> > 
> > OK, ok, will do next time. The full git commit ID is faster to cut-n-paste
> > which shows how lazy I'm ;)
> 
> No worires, I'll fix that up for you when I apply it.

Thanks!

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH] jbd2: Fix use after free in jbd2_log_do_checkpoint()

[Theodore Y. Ts'o]

On Thu, Oct 04, 2018 at 12:46:40PM +0200, Jan Kara wrote:
> The code cleaning transaction's lists of checkpoint buffers has a bug
> where it increases bh refcount only after releasing
> journal->j_list_lock. Thus the following race is possible:
> 
> CPU0					CPU1
> jbd2_log_do_checkpoint()
> 					jbd2_journal_try_to_free_buffers()
> 					  __journal_try_to_free_buffer(bh)
>   ...
>   while (transaction->t_checkpoint_io_list)
>   ...
>     if (buffer_locked(bh)) {
> 
> <-- IO completes now, buffer gets unlocked -->
> 
>       spin_unlock(&journal->j_list_lock);
> 					    spin_lock(&journal->j_list_lock);
> 					    __jbd2_journal_remove_checkpoint(jh);
> 					    spin_unlock(&journal->j_list_lock);
> 					  try_to_free_buffers(page);
>       get_bh(bh) <-- accesses freed bh
> 
> Fix the problem by grabbing bh reference before unlocking
> journal->j_list_lock.
> 
> Fixes: dc6e8d669cf5cb3ff84707c372c0a2a8a5e80845
> Fixes: be1158cc615fd723552f0d9912087423c7cadda5
> Reported-by: syzbot+7f4a27091759e2fe7453@syzkaller.appspotmail.com
> CC: stable@vger.kernel.org
> Signed-off-by: Jan Kara <jack@suse.cz>

Applied, thanks (with the Fixes field adjusted).

					- Ted