* WARNING in percpu_ref_exit (2)
@ 2019-12-21 8:05 syzbot
2019-12-21 8:43 ` syzbot
2020-11-11 3:09 ` syzbot
0 siblings, 2 replies; 6+ messages in thread
From: syzbot @ 2019-12-21 8:05 UTC (permalink / raw)
To: axboe, io-uring, linux-fsdevel, linux-kernel, syzkaller-bugs, viro
Hello,
syzbot found the following crash on:
HEAD commit: 7ddd09fc Add linux-next specific files for 20191220
git tree: linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=12a18cc6e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=f183b01c3088afc6
dashboard link: https://syzkaller.appspot.com/bug?extid=8c4a14856e657b43487c
compiler: gcc (GCC) 9.0.0 20181231 (experimental)
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14b8f351e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=14b51925e00000
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+8c4a14856e657b43487c@syzkaller.appspotmail.com
------------[ cut here ]------------
WARNING: CPU: 1 PID: 11482 at lib/percpu-refcount.c:111
percpu_ref_exit+0xab/0xd0 lib/percpu-refcount.c:111
Kernel panic - not syncing: panic_on_warn set ...
CPU: 1 PID: 11482 Comm: syz-executor051 Not tainted
5.5.0-rc2-next-20191220-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x197/0x210 lib/dump_stack.c:118
panic+0x2e3/0x75c kernel/panic.c:221
__warn.cold+0x2f/0x3e kernel/panic.c:582
report_bug+0x289/0x300 lib/bug.c:195
fixup_bug arch/x86/kernel/traps.c:174 [inline]
fixup_bug arch/x86/kernel/traps.c:169 [inline]
do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:267
do_invalid_op+0x37/0x50 arch/x86/kernel/traps.c:286
invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027
RIP: 0010:percpu_ref_exit+0xab/0xd0 lib/percpu-refcount.c:111
Code: 00 00 00 00 fc ff df 48 c1 ea 03 80 3c 02 00 75 1d 48 c7 43 08 03 00
00 00 e8 01 41 e5 fd 5b 41 5c 41 5d 5d c3 e8 f5 40 e5 fd <0f> 0b eb bf 4c
89 ef e8 29 2c 23 fe eb d9 e8 82 2b 23 fe eb a7 4c
RSP: 0018:ffffc9000cb17968 EFLAGS: 00010293
RAX: ffff8880a3390640 RBX: ffff8880a83a8010 RCX: ffffffff83901432
RDX: 0000000000000000 RSI: ffffffff8390149b RDI: ffff8880a83a8028
RBP: ffffc9000cb17980 R08: ffff8880a3390640 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000607f51435750
R13: ffff8880a83a8018 R14: ffff888097b95000 R15: ffff888097b95228
io_sqe_files_unregister+0x7d/0x2f0 fs/io_uring.c:4623
io_ring_ctx_free fs/io_uring.c:5575 [inline]
io_ring_ctx_wait_and_kill+0x430/0x9a0 fs/io_uring.c:5644
io_uring_release+0x42/0x50 fs/io_uring.c:5652
__fput+0x2ff/0x890 fs/file_table.c:280
____fput+0x16/0x20 fs/file_table.c:313
task_work_run+0x145/0x1c0 kernel/task_work.c:113
exit_task_work include/linux/task_work.h:22 [inline]
do_exit+0x909/0x2f20 kernel/exit.c:797
do_group_exit+0x135/0x360 kernel/exit.c:895
get_signal+0x47c/0x24f0 kernel/signal.c:2734
do_signal+0x87/0x1700 arch/x86/kernel/signal.c:815
exit_to_usermode_loop+0x286/0x380 arch/x86/entry/common.c:160
prepare_exit_to_usermode arch/x86/entry/common.c:195 [inline]
syscall_return_slowpath arch/x86/entry/common.c:278 [inline]
do_syscall_64+0x676/0x790 arch/x86/entry/common.c:304
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4468f9
Code: e8 0c e8 ff ff 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff
ff 0f 83 5b 07 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f7ef700ddb8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: fffffffffffffe00 RBX: 00000000006dbc38 RCX: 00000000004468f9
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 00000000006dbc38
RBP: 00000000006dbc30 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dbc3c
R13: 00007fff8371e42f R14: 00007f7ef700e9c0 R15: 0000000000000001
Kernel Offset: disabled
Rebooting in 86400 seconds..
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: WARNING in percpu_ref_exit (2)
2019-12-21 8:05 WARNING in percpu_ref_exit (2) syzbot
@ 2019-12-21 8:43 ` syzbot
2020-11-11 3:09 ` syzbot
1 sibling, 0 replies; 6+ messages in thread
From: syzbot @ 2019-12-21 8:43 UTC (permalink / raw)
To: axboe, io-uring, linux-block, linux-fsdevel, linux-kernel,
syzkaller-bugs, viro
syzbot has bisected this bug to:
commit cbb537634780172137459dead490d668d437ef4d
Author: Jens Axboe <axboe@kernel.dk>
Date: Mon Dec 9 18:22:50 2019 +0000
io_uring: avoid ring quiesce for fixed file set unregister and update
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=1118aac1e00000
start commit: 7ddd09fc Add linux-next specific files for 20191220
git tree: linux-next
final crash: https://syzkaller.appspot.com/x/report.txt?x=1318aac1e00000
console output: https://syzkaller.appspot.com/x/log.txt?x=1518aac1e00000
kernel config: https://syzkaller.appspot.com/x/.config?x=f183b01c3088afc6
dashboard link: https://syzkaller.appspot.com/bug?extid=8c4a14856e657b43487c
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14b8f351e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=14b51925e00000
Reported-by: syzbot+8c4a14856e657b43487c@syzkaller.appspotmail.com
Fixes: cbb537634780 ("io_uring: avoid ring quiesce for fixed file set
unregister and update")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: WARNING in percpu_ref_exit (2)
2019-12-21 8:05 WARNING in percpu_ref_exit (2) syzbot
2019-12-21 8:43 ` syzbot
@ 2020-11-11 3:09 ` syzbot
2020-11-11 13:30 ` Dmitry Vyukov
1 sibling, 1 reply; 6+ messages in thread
From: syzbot @ 2020-11-11 3:09 UTC (permalink / raw)
To: axboe, ebiggers, hdanton, io-uring, linux-block, linux-fsdevel,
linux-kernel, paulmck, syzkaller-bugs, viro
syzbot suspects this issue was fixed by commit:
commit c1e2148f8ecb26863b899d402a823dab8e26efd1
Author: Jens Axboe <axboe@kernel.dk>
Date: Wed Mar 4 14:25:50 2020 +0000
io_uring: free fixed_file_data after RCU grace period
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=161ea46e500000
start commit: 63849c8f Merge tag 'linux-kselftest-5.6-rc5' of git://git...
git tree: upstream
kernel config: https://syzkaller.appspot.com/x/.config?x=4527d1e2fb19fd5c
dashboard link: https://syzkaller.appspot.com/bug?extid=8c4a14856e657b43487c
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=13c30061e00000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1251b731e00000
If the result looks correct, please mark the issue as fixed by replying with:
#syz fix: io_uring: free fixed_file_data after RCU grace period
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <20191221134330.7376-1-hdanton@sina.com>]
* Re: WARNING in percpu_ref_exit (2)
[not found] <20191221134330.7376-1-hdanton@sina.com>
@ 2019-12-21 14:02 ` Jens Axboe
2020-06-28 15:48 ` Eric Biggers
0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2019-12-21 14:02 UTC (permalink / raw)
To: Hillf Danton, syzbot
Cc: io-uring, viro, linux-fsdevel, linux-kernel, syzkaller-bugs
On 12/21/19 6:43 AM, Hillf Danton wrote:
>
> On Sat, 21 Dec 2019 00:05:07 -0800
>> Hello,
>>
>> syzbot found the following crash on:
>>
>> HEAD commit: 7ddd09fc Add linux-next specific files for 20191220
>> git tree: linux-next
>> console output: https://syzkaller.appspot.com/x/log.txt?x=12a18cc6e00000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=f183b01c3088afc6
>> dashboard link: https://syzkaller.appspot.com/bug?extid=8c4a14856e657b43487c
>> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14b8f351e00000
>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=14b51925e00000
>>
>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> Reported-by: syzbot+8c4a14856e657b43487c@syzkaller.appspotmail.com
>>
>> ------------[ cut here ]------------
>> WARNING: CPU: 1 PID: 11482 at lib/percpu-refcount.c:111
>> percpu_ref_exit+0xab/0xd0 lib/percpu-refcount.c:111
>> Kernel panic - not syncing: panic_on_warn set ...
>> CPU: 1 PID: 11482 Comm: syz-executor051 Not tainted
>> 5.5.0-rc2-next-20191220-syzkaller #0
>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>> Google 01/01/2011
>> Call Trace:
>> __dump_stack lib/dump_stack.c:77 [inline]
>> dump_stack+0x197/0x210 lib/dump_stack.c:118
>> panic+0x2e3/0x75c kernel/panic.c:221
>> __warn.cold+0x2f/0x3e kernel/panic.c:582
>> report_bug+0x289/0x300 lib/bug.c:195
>> fixup_bug arch/x86/kernel/traps.c:174 [inline]
>> fixup_bug arch/x86/kernel/traps.c:169 [inline]
>> do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:267
>> do_invalid_op+0x37/0x50 arch/x86/kernel/traps.c:286
>> invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027
>> RIP: 0010:percpu_ref_exit+0xab/0xd0 lib/percpu-refcount.c:111
>> Code: 00 00 00 00 fc ff df 48 c1 ea 03 80 3c 02 00 75 1d 48 c7 43 08 03 00
>> 00 00 e8 01 41 e5 fd 5b 41 5c 41 5d 5d c3 e8 f5 40 e5 fd <0f> 0b eb bf 4c
>> 89 ef e8 29 2c 23 fe eb d9 e8 82 2b 23 fe eb a7 4c
>> RSP: 0018:ffffc9000cb17968 EFLAGS: 00010293
>> RAX: ffff8880a3390640 RBX: ffff8880a83a8010 RCX: ffffffff83901432
>> RDX: 0000000000000000 RSI: ffffffff8390149b RDI: ffff8880a83a8028
>> RBP: ffffc9000cb17980 R08: ffff8880a3390640 R09: 0000000000000000
>> R10: 0000000000000000 R11: 0000000000000000 R12: 0000607f51435750
>> R13: ffff8880a83a8018 R14: ffff888097b95000 R15: ffff888097b95228
>> io_sqe_files_unregister+0x7d/0x2f0 fs/io_uring.c:4623
>> io_ring_ctx_free fs/io_uring.c:5575 [inline]
>> io_ring_ctx_wait_and_kill+0x430/0x9a0 fs/io_uring.c:5644
>> io_uring_release+0x42/0x50 fs/io_uring.c:5652
>> __fput+0x2ff/0x890 fs/file_table.c:280
>> ____fput+0x16/0x20 fs/file_table.c:313
>> task_work_run+0x145/0x1c0 kernel/task_work.c:113
>> exit_task_work include/linux/task_work.h:22 [inline]
>> do_exit+0x909/0x2f20 kernel/exit.c:797
>> do_group_exit+0x135/0x360 kernel/exit.c:895
>> get_signal+0x47c/0x24f0 kernel/signal.c:2734
>> do_signal+0x87/0x1700 arch/x86/kernel/signal.c:815
>> exit_to_usermode_loop+0x286/0x380 arch/x86/entry/common.c:160
>> prepare_exit_to_usermode arch/x86/entry/common.c:195 [inline]
>> syscall_return_slowpath arch/x86/entry/common.c:278 [inline]
>> do_syscall_64+0x676/0x790 arch/x86/entry/common.c:304
>> entry_SYSCALL_64_after_hwframe+0x49/0xbe
>
> Flush work before killing.
>
> --- a/fs/io_uring.c
> +++ b/fs/io_uring.c
> @@ -4618,10 +4618,10 @@ static int io_sqe_files_unregister(struc
> if (!data)
> return -ENXIO;
>
> + flush_work(&data->ref_work);
> percpu_ref_kill_and_confirm(&data->refs, io_file_ref_kill);
> wait_for_completion(&data->done);
> percpu_ref_exit(&data->refs);
> - flush_work(&data->ref_work);
>
> __io_sqe_files_unregister(ctx);
> nr_tables = DIV_ROUND_UP(ctx->nr_user_files, IORING_MAX_FILES_TABLE);
Oh indeed, good catch! Thanks, I'll fold this in.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: WARNING in percpu_ref_exit (2)
2019-12-21 14:02 ` Jens Axboe
@ 2020-06-28 15:48 ` Eric Biggers
0 siblings, 0 replies; 6+ messages in thread
From: Eric Biggers @ 2020-06-28 15:48 UTC (permalink / raw)
To: Jens Axboe
Cc: Hillf Danton, syzbot, io-uring, viro, linux-fsdevel,
linux-kernel, syzkaller-bugs
On Sat, Dec 21, 2019 at 07:02:25AM -0700, Jens Axboe wrote:
> On 12/21/19 6:43 AM, Hillf Danton wrote:
> >
> > On Sat, 21 Dec 2019 00:05:07 -0800
> >> Hello,
> >>
> >> syzbot found the following crash on:
> >>
> >> HEAD commit: 7ddd09fc Add linux-next specific files for 20191220
> >> git tree: linux-next
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=12a18cc6e00000
> >> kernel config: https://syzkaller.appspot.com/x/.config?x=f183b01c3088afc6
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=8c4a14856e657b43487c
> >> compiler: gcc (GCC) 9.0.0 20181231 (experimental)
> >> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=14b8f351e00000
> >> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=14b51925e00000
> >>
> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> >> Reported-by: syzbot+8c4a14856e657b43487c@syzkaller.appspotmail.com
> >>
> >> ------------[ cut here ]------------
> >> WARNING: CPU: 1 PID: 11482 at lib/percpu-refcount.c:111
> >> percpu_ref_exit+0xab/0xd0 lib/percpu-refcount.c:111
> >> Kernel panic - not syncing: panic_on_warn set ...
> >> CPU: 1 PID: 11482 Comm: syz-executor051 Not tainted
> >> 5.5.0-rc2-next-20191220-syzkaller #0
> >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> >> Google 01/01/2011
> >> Call Trace:
> >> __dump_stack lib/dump_stack.c:77 [inline]
> >> dump_stack+0x197/0x210 lib/dump_stack.c:118
> >> panic+0x2e3/0x75c kernel/panic.c:221
> >> __warn.cold+0x2f/0x3e kernel/panic.c:582
> >> report_bug+0x289/0x300 lib/bug.c:195
> >> fixup_bug arch/x86/kernel/traps.c:174 [inline]
> >> fixup_bug arch/x86/kernel/traps.c:169 [inline]
> >> do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:267
> >> do_invalid_op+0x37/0x50 arch/x86/kernel/traps.c:286
> >> invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027
> >> RIP: 0010:percpu_ref_exit+0xab/0xd0 lib/percpu-refcount.c:111
> >> Code: 00 00 00 00 fc ff df 48 c1 ea 03 80 3c 02 00 75 1d 48 c7 43 08 03 00
> >> 00 00 e8 01 41 e5 fd 5b 41 5c 41 5d 5d c3 e8 f5 40 e5 fd <0f> 0b eb bf 4c
> >> 89 ef e8 29 2c 23 fe eb d9 e8 82 2b 23 fe eb a7 4c
> >> RSP: 0018:ffffc9000cb17968 EFLAGS: 00010293
> >> RAX: ffff8880a3390640 RBX: ffff8880a83a8010 RCX: ffffffff83901432
> >> RDX: 0000000000000000 RSI: ffffffff8390149b RDI: ffff8880a83a8028
> >> RBP: ffffc9000cb17980 R08: ffff8880a3390640 R09: 0000000000000000
> >> R10: 0000000000000000 R11: 0000000000000000 R12: 0000607f51435750
> >> R13: ffff8880a83a8018 R14: ffff888097b95000 R15: ffff888097b95228
> >> io_sqe_files_unregister+0x7d/0x2f0 fs/io_uring.c:4623
> >> io_ring_ctx_free fs/io_uring.c:5575 [inline]
> >> io_ring_ctx_wait_and_kill+0x430/0x9a0 fs/io_uring.c:5644
> >> io_uring_release+0x42/0x50 fs/io_uring.c:5652
> >> __fput+0x2ff/0x890 fs/file_table.c:280
> >> ____fput+0x16/0x20 fs/file_table.c:313
> >> task_work_run+0x145/0x1c0 kernel/task_work.c:113
> >> exit_task_work include/linux/task_work.h:22 [inline]
> >> do_exit+0x909/0x2f20 kernel/exit.c:797
> >> do_group_exit+0x135/0x360 kernel/exit.c:895
> >> get_signal+0x47c/0x24f0 kernel/signal.c:2734
> >> do_signal+0x87/0x1700 arch/x86/kernel/signal.c:815
> >> exit_to_usermode_loop+0x286/0x380 arch/x86/entry/common.c:160
> >> prepare_exit_to_usermode arch/x86/entry/common.c:195 [inline]
> >> syscall_return_slowpath arch/x86/entry/common.c:278 [inline]
> >> do_syscall_64+0x676/0x790 arch/x86/entry/common.c:304
> >> entry_SYSCALL_64_after_hwframe+0x49/0xbe
> >
> > Flush work before killing.
> >
> > --- a/fs/io_uring.c
> > +++ b/fs/io_uring.c
> > @@ -4618,10 +4618,10 @@ static int io_sqe_files_unregister(struc
> > if (!data)
> > return -ENXIO;
> >
> > + flush_work(&data->ref_work);
> > percpu_ref_kill_and_confirm(&data->refs, io_file_ref_kill);
> > wait_for_completion(&data->done);
> > percpu_ref_exit(&data->refs);
> > - flush_work(&data->ref_work);
> >
> > __io_sqe_files_unregister(ctx);
> > nr_tables = DIV_ROUND_UP(ctx->nr_user_files, IORING_MAX_FILES_TABLE);
>
> Oh indeed, good catch! Thanks, I'll fold this in.
Hi Jens, this bug is still occurring. See the syzbot dashboard:
https://syzkaller.appspot.com/bug?extid=8c4a14856e657b43487c
Here's a crash on upstream commit 7ae77150d94d (Fri Jun 5 12:39:30 2020 -0700):
https://syzkaller.appspot.com/text?tag=CrashReport&x=1754d169100000
- Eric
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-11-11 13:31 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-12-21 8:05 WARNING in percpu_ref_exit (2) syzbot
2019-12-21 8:43 ` syzbot
2020-11-11 3:09 ` syzbot
2020-11-11 13:30 ` Dmitry Vyukov
[not found] <20191221134330.7376-1-hdanton@sina.com>
2019-12-21 14:02 ` Jens Axboe
2020-06-28 15:48 ` Eric Biggers
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).