All of lore.kernel.org
 help / color / mirror / Atom feed
* use-after-free" with v5.10.y caused by backport of a298232ee6b9 ("io_uring: fix link timeout refs")
@ 2021-07-26 10:29 Sudip Mukherjee
  2021-07-26 10:57 ` Pavel Begunkov
  0 siblings, 1 reply; 5+ messages in thread
From: Sudip Mukherjee @ 2021-07-26 10:29 UTC (permalink / raw)
  To: Pavel Begunkov; +Cc: stable, Greg Kroah-Hartman, Jens Axboe, Sasha Levin

Hi Pavel,

We had been running syzkaller on v5.10.y and a "use after free" is being
reported for v5.10.43+ kernels.

The syzkaller report is at: https://elisa-builder-00.iol.unh.edu/syzkaller/file?name=crashes%2fb23bc4ad436bbe4afc620d9503730ddd78c382c0%2freport19

The trace is:

refcount_t: underflow; use-after-free.
WARNING: CPU: 1 PID: 8769 at lib/refcount.c:28 refcount_warn_saturate+0x103/0x1f0 lib/refcount.c:28
Modules linked in:
CPU: 1 PID: 8769 Comm: syz-executor.6 Not tainted 5.10.52 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
RIP: 0010:refcount_warn_saturate+0x103/0x1f0 lib/refcount.c:28
Code: 1d d2 63 54 03 31 ff 89 de e8 c9 22 51 ff 84 db 75 a3 e8 90 29 51 ff 48 c7 c7 20 38 3b 84 c6 05 b2 63 54 03 01 e8 cc 0c c9 01 <0f> 0b eb 87 e8 74 29 51 ff 0f b6 1d 9b 63 54 03 31 ff 89 de e8 94
RSP: 0018:ffff88804ec5f9f8 EFLAGS: 00010286
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000040000 RSI: ffffffff81293053 RDI: ffffed1009d8bf31
RBP: ffff888048ceb41c R08: 0000000000000001 R09: ffff88806cf1ff9b
R10: 0000000000000000 R11: 0000000000000001 R12: ffff888048ceb41c
R13: 0000000000000000 R14: ffff888048ceb558 R15: ffff88800c857180
FS:  00007f0e798e9700(0000) GS:ffff88806cf00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000558bbbc7d898 CR3: 0000000048fb6000 CR4: 0000000000350ee0
Call Trace:
 __refcount_sub_and_test include/linux/refcount.h:283 [inline]
 __refcount_dec_and_test include/linux/refcount.h:315 [inline]
 refcount_dec_and_test include/linux/refcount.h:333 [inline]
 io_put_req+0xc6/0x100 fs/io_uring.c:2220
 __io_queue_sqe+0x2b1/0xd00 fs/io_uring.c:6358
 io_queue_sqe+0x5bc/0x1020 fs/io_uring.c:6403
 io_queue_link_head fs/io_uring.c:6414 [inline]
 io_submit_sqe fs/io_uring.c:6455 [inline]
 io_submit_sqes+0x17b5/0x2310 fs/io_uring.c:6700
 __do_sys_io_uring_enter+0x1092/0x1910 fs/io_uring.c:9092
 do_syscall_64+0x33/0x40 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x466609
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f0e798e9188 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
RAX: ffffffffffffffda RBX: 000000000056bf80 RCX: 0000000000466609
RDX: 0000000000000000 RSI: 00000000000058ab RDI: 0000000000000004
RBP: 00000000004bfcb9 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf80
R13: 00007ffdbe4f1c5f R14: 00007f0e798e9300 R15: 0000000000022000
irq event stamp: 1473
hardirqs last  enabled at (1483): [<ffffffff8128f5d9>] console_unlock+0x929/0xb30 kernel/printk/printk.c:2552
hardirqs last disabled at (1494): [<ffffffff83c43a7b>] sysvec_apic_timer_interrupt+0xb/0xa0 arch/x86/kernel/apic/apic.c:1095
softirqs last  enabled at (1072): [<ffffffff83e00f92>] asm_call_irq_on_stack+0x12/0x20
softirqs last disabled at (1067): [<ffffffff83e00f92>] asm_call_irq_on_stack+0x12/0x20

I have done a bisect and the commit causing this is: 0b2a990e5d2f ("io_uring: fix link timeout refs")

The git bisect log is:
# bad: [71046eac2db9aeccf10763d034a1a123911c9a81] Linux 5.10.53
# good: [2c85ebc57b3e1817b6ce1a6b703928e113a90442] Linux 5.10
git bisect start 'v5.10.53' 'v5.10'
# good: [d29c38dd926d5aba65d177c0b99381ea632ff0a0] staging: rtl8192e: Change state information from u16 to u8
git bisect good d29c38dd926d5aba65d177c0b99381ea632ff0a0
# good: [b67c3d74adc3f7f832f57c170234bbe1fc69c87c] Revert "net: fujitsu: fix a potential NULL pointer dereference"
git bisect good b67c3d74adc3f7f832f57c170234bbe1fc69c87c
# bad: [f1f30b3373df2e5ab96dd3781df5c02e5366f845] mmc: usdhi6rol0: fix error return code in usdhi6_probe()
git bisect bad f1f30b3373df2e5ab96dd3781df5c02e5366f845
# bad: [3d60457d74d9cc7b36f78f9cb74f29bc6182c1e8] cxgb4: fix endianness when flashing boot image
git bisect bad 3d60457d74d9cc7b36f78f9cb74f29bc6182c1e8
# bad: [3a6b69221f96f87c680bbc9fba01db3415b18f27] drm/amdgpu: make sure we unpin the UVD BO
git bisect bad 3a6b69221f96f87c680bbc9fba01db3415b18f27
# good: [65859eca4dff1af0db5e36d1cfbac15b834c6a65] Linux 5.10.42
git bisect good 65859eca4dff1af0db5e36d1cfbac15b834c6a65
# good: [a1bf16616d8351a2e79400d6d19608befb2ce1dd] ixgbe: add correct exception tracing for XDP
git bisect good a1bf16616d8351a2e79400d6d19608befb2ce1dd
# bad: [c5155c741a484e036e7997420559431a951f2106] wireguard: allowedips: allocate nodes in kmem_cache
git bisect bad c5155c741a484e036e7997420559431a951f2106
# good: [3c23e23c7ad9844a645f4e2bd8ec34a0a2ee5514] riscv: vdso: fix and clean-up Makefile
git bisect good 3c23e23c7ad9844a645f4e2bd8ec34a0a2ee5514
# bad: [74caf718cc7422a957aac381c73d798c0a999a65] Bluetooth: use correct lock to prevent UAF of hdev object
git bisect bad 74caf718cc7422a957aac381c73d798c0a999a65
# bad: [58f4d45d8d4d391f60b6f0db6308df1994a265b3] drm/amdgpu/vcn3: add cancel_delayed_work_sync before power gate
git bisect bad 58f4d45d8d4d391f60b6f0db6308df1994a265b3
# bad: [ec72cb50c1db39816eae7296686449bba8ca0b2e] io_uring: use better types for cflags
git bisect bad ec72cb50c1db39816eae7296686449bba8ca0b2e
# bad: [0b2a990e5d2f76d020cb840c456e6ec5f0c27530] io_uring: fix link timeout refs
git bisect bad 0b2a990e5d2f76d020cb840c456e6ec5f0c27530
# first bad commit: [0b2a990e5d2f76d020cb840c456e6ec5f0c27530] io_uring: fix link timeout refs

The mainline commit for the bad LTS commit is:
a298232ee6b9 ("io_uring: fix link timeout refs") and I have tested the
reproducer on mainline with 'a298232ee6b9' as HEAD and the issue is not
reproduced. I think we are missing some change in v5.10.y kernel which
was missed while the mainline fix was backported to LTS.

I can reproduce the crash using syzkaller and will be happy to test any
patch for this.


--
Regards
Sudip

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: use-after-free" with v5.10.y caused by backport of a298232ee6b9 ("io_uring: fix link timeout refs")
  2021-07-26 10:29 use-after-free" with v5.10.y caused by backport of a298232ee6b9 ("io_uring: fix link timeout refs") Sudip Mukherjee
@ 2021-07-26 10:57 ` Pavel Begunkov
  2021-07-26 11:08   ` Greg Kroah-Hartman
  0 siblings, 1 reply; 5+ messages in thread
From: Pavel Begunkov @ 2021-07-26 10:57 UTC (permalink / raw)
  To: Sudip Mukherjee; +Cc: stable, Greg Kroah-Hartman, Jens Axboe, Sasha Levin

On 7/26/21 11:29 AM, Sudip Mukherjee wrote:
> Hi Pavel,
> 
> We had been running syzkaller on v5.10.y and a "use after free" is being
> reported for v5.10.43+ kernels.

"... # 5.12+", weird, but perhaps due to dependencies.
Thanks for letting know.


Greg, Sasha, should be same as reported for 5.12

https://www.spinics.net/lists/stable/msg485116.html

Can you try to apply it to 5.10 or should I resend?


> The syzkaller report is at: https://elisa-builder-00.iol.unh.edu/syzkaller/file?name=crashes%2fb23bc4ad436bbe4afc620d9503730ddd78c382c0%2freport19
> 
> The trace is:
> 
> refcount_t: underflow; use-after-free.
> WARNING: CPU: 1 PID: 8769 at lib/refcount.c:28 refcount_warn_saturate+0x103/0x1f0 lib/refcount.c:28
> Modules linked in:
> CPU: 1 PID: 8769 Comm: syz-executor.6 Not tainted 5.10.52 #1
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
> RIP: 0010:refcount_warn_saturate+0x103/0x1f0 lib/refcount.c:28
> Code: 1d d2 63 54 03 31 ff 89 de e8 c9 22 51 ff 84 db 75 a3 e8 90 29 51 ff 48 c7 c7 20 38 3b 84 c6 05 b2 63 54 03 01 e8 cc 0c c9 01 <0f> 0b eb 87 e8 74 29 51 ff 0f b6 1d 9b 63 54 03 31 ff 89 de e8 94
> RSP: 0018:ffff88804ec5f9f8 EFLAGS: 00010286
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> RDX: 0000000000040000 RSI: ffffffff81293053 RDI: ffffed1009d8bf31
> RBP: ffff888048ceb41c R08: 0000000000000001 R09: ffff88806cf1ff9b
> R10: 0000000000000000 R11: 0000000000000001 R12: ffff888048ceb41c
> R13: 0000000000000000 R14: ffff888048ceb558 R15: ffff88800c857180
> FS:  00007f0e798e9700(0000) GS:ffff88806cf00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000558bbbc7d898 CR3: 0000000048fb6000 CR4: 0000000000350ee0
> Call Trace:
>  __refcount_sub_and_test include/linux/refcount.h:283 [inline]
>  __refcount_dec_and_test include/linux/refcount.h:315 [inline]
>  refcount_dec_and_test include/linux/refcount.h:333 [inline]
>  io_put_req+0xc6/0x100 fs/io_uring.c:2220
>  __io_queue_sqe+0x2b1/0xd00 fs/io_uring.c:6358
>  io_queue_sqe+0x5bc/0x1020 fs/io_uring.c:6403
>  io_queue_link_head fs/io_uring.c:6414 [inline]
>  io_submit_sqe fs/io_uring.c:6455 [inline]
>  io_submit_sqes+0x17b5/0x2310 fs/io_uring.c:6700
>  __do_sys_io_uring_enter+0x1092/0x1910 fs/io_uring.c:9092
>  do_syscall_64+0x33/0x40 arch/x86/entry/common.c:46
>  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> RIP: 0033:0x466609
> Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007f0e798e9188 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
> RAX: ffffffffffffffda RBX: 000000000056bf80 RCX: 0000000000466609
> RDX: 0000000000000000 RSI: 00000000000058ab RDI: 0000000000000004
> RBP: 00000000004bfcb9 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf80
> R13: 00007ffdbe4f1c5f R14: 00007f0e798e9300 R15: 0000000000022000
> irq event stamp: 1473
> hardirqs last  enabled at (1483): [<ffffffff8128f5d9>] console_unlock+0x929/0xb30 kernel/printk/printk.c:2552
> hardirqs last disabled at (1494): [<ffffffff83c43a7b>] sysvec_apic_timer_interrupt+0xb/0xa0 arch/x86/kernel/apic/apic.c:1095
> softirqs last  enabled at (1072): [<ffffffff83e00f92>] asm_call_irq_on_stack+0x12/0x20
> softirqs last disabled at (1067): [<ffffffff83e00f92>] asm_call_irq_on_stack+0x12/0x20
> 
> I have done a bisect and the commit causing this is: 0b2a990e5d2f ("io_uring: fix link timeout refs")
> 
> The git bisect log is:
> # bad: [71046eac2db9aeccf10763d034a1a123911c9a81] Linux 5.10.53
> # good: [2c85ebc57b3e1817b6ce1a6b703928e113a90442] Linux 5.10
> git bisect start 'v5.10.53' 'v5.10'
> # good: [d29c38dd926d5aba65d177c0b99381ea632ff0a0] staging: rtl8192e: Change state information from u16 to u8
> git bisect good d29c38dd926d5aba65d177c0b99381ea632ff0a0
> # good: [b67c3d74adc3f7f832f57c170234bbe1fc69c87c] Revert "net: fujitsu: fix a potential NULL pointer dereference"
> git bisect good b67c3d74adc3f7f832f57c170234bbe1fc69c87c
> # bad: [f1f30b3373df2e5ab96dd3781df5c02e5366f845] mmc: usdhi6rol0: fix error return code in usdhi6_probe()
> git bisect bad f1f30b3373df2e5ab96dd3781df5c02e5366f845
> # bad: [3d60457d74d9cc7b36f78f9cb74f29bc6182c1e8] cxgb4: fix endianness when flashing boot image
> git bisect bad 3d60457d74d9cc7b36f78f9cb74f29bc6182c1e8
> # bad: [3a6b69221f96f87c680bbc9fba01db3415b18f27] drm/amdgpu: make sure we unpin the UVD BO
> git bisect bad 3a6b69221f96f87c680bbc9fba01db3415b18f27
> # good: [65859eca4dff1af0db5e36d1cfbac15b834c6a65] Linux 5.10.42
> git bisect good 65859eca4dff1af0db5e36d1cfbac15b834c6a65
> # good: [a1bf16616d8351a2e79400d6d19608befb2ce1dd] ixgbe: add correct exception tracing for XDP
> git bisect good a1bf16616d8351a2e79400d6d19608befb2ce1dd
> # bad: [c5155c741a484e036e7997420559431a951f2106] wireguard: allowedips: allocate nodes in kmem_cache
> git bisect bad c5155c741a484e036e7997420559431a951f2106
> # good: [3c23e23c7ad9844a645f4e2bd8ec34a0a2ee5514] riscv: vdso: fix and clean-up Makefile
> git bisect good 3c23e23c7ad9844a645f4e2bd8ec34a0a2ee5514
> # bad: [74caf718cc7422a957aac381c73d798c0a999a65] Bluetooth: use correct lock to prevent UAF of hdev object
> git bisect bad 74caf718cc7422a957aac381c73d798c0a999a65
> # bad: [58f4d45d8d4d391f60b6f0db6308df1994a265b3] drm/amdgpu/vcn3: add cancel_delayed_work_sync before power gate
> git bisect bad 58f4d45d8d4d391f60b6f0db6308df1994a265b3
> # bad: [ec72cb50c1db39816eae7296686449bba8ca0b2e] io_uring: use better types for cflags
> git bisect bad ec72cb50c1db39816eae7296686449bba8ca0b2e
> # bad: [0b2a990e5d2f76d020cb840c456e6ec5f0c27530] io_uring: fix link timeout refs
> git bisect bad 0b2a990e5d2f76d020cb840c456e6ec5f0c27530
> # first bad commit: [0b2a990e5d2f76d020cb840c456e6ec5f0c27530] io_uring: fix link timeout refs
> 
> The mainline commit for the bad LTS commit is:
> a298232ee6b9 ("io_uring: fix link timeout refs") and I have tested the
> reproducer on mainline with 'a298232ee6b9' as HEAD and the issue is not
> reproduced. I think we are missing some change in v5.10.y kernel which
> was missed while the mainline fix was backported to LTS.
> 
> I can reproduce the crash using syzkaller and will be happy to test any
> patch for this.
> 
> 
> --
> Regards
> Sudip
> 

-- 
Pavel Begunkov

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: use-after-free" with v5.10.y caused by backport of a298232ee6b9 ("io_uring: fix link timeout refs")
  2021-07-26 10:57 ` Pavel Begunkov
@ 2021-07-26 11:08   ` Greg Kroah-Hartman
  2021-07-26 12:44     ` Sudip Mukherjee
  0 siblings, 1 reply; 5+ messages in thread
From: Greg Kroah-Hartman @ 2021-07-26 11:08 UTC (permalink / raw)
  To: Pavel Begunkov; +Cc: Sudip Mukherjee, stable, Jens Axboe, Sasha Levin

On Mon, Jul 26, 2021 at 11:57:22AM +0100, Pavel Begunkov wrote:
> On 7/26/21 11:29 AM, Sudip Mukherjee wrote:
> > Hi Pavel,
> > 
> > We had been running syzkaller on v5.10.y and a "use after free" is being
> > reported for v5.10.43+ kernels.
> 
> "... # 5.12+", weird, but perhaps due to dependencies.
> Thanks for letting know.
> 
> 
> Greg, Sasha, should be same as reported for 5.12
> 
> https://www.spinics.net/lists/stable/msg485116.html
> 
> Can you try to apply it to 5.10 or should I resend?

I just tried applying those patches and they did not work.  So can you
provide some new backports?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: use-after-free" with v5.10.y caused by backport of a298232ee6b9 ("io_uring: fix link timeout refs")
  2021-07-26 11:08   ` Greg Kroah-Hartman
@ 2021-07-26 12:44     ` Sudip Mukherjee
  2021-07-26 12:53       ` Pavel Begunkov
  0 siblings, 1 reply; 5+ messages in thread
From: Sudip Mukherjee @ 2021-07-26 12:44 UTC (permalink / raw)
  To: Greg Kroah-Hartman; +Cc: Pavel Begunkov, stable, Jens Axboe, Sasha Levin

[-- Attachment #1: Type: text/plain, Size: 843 bytes --]

Hi Greg,

On Mon, Jul 26, 2021 at 01:08:38PM +0200, Greg Kroah-Hartman wrote:
> On Mon, Jul 26, 2021 at 11:57:22AM +0100, Pavel Begunkov wrote:
> > On 7/26/21 11:29 AM, Sudip Mukherjee wrote:
> > > Hi Pavel,
> > > 
> > > We had been running syzkaller on v5.10.y and a "use after free" is being
> > > reported for v5.10.43+ kernels.
> > 
> > "... # 5.12+", weird, but perhaps due to dependencies.
> > Thanks for letting know.
> > 
> > 
> > Greg, Sasha, should be same as reported for 5.12
> > 
> > https://www.spinics.net/lists/stable/msg485116.html
> > 
> > Can you try to apply it to 5.10 or should I resend?
> 
> I just tried applying those patches and they did not work.  So can you
> provide some new backports?

Here is the backport for v5.10.y. I have also tested these with the
syzkaller repro and the issue is fixed.

--
Regards
Sudip

[-- Attachment #2: 0001-io_uring-put-link-timeout-req-consistently.patch --]
[-- Type: text/x-diff, Size: 1961 bytes --]

From bd136e16d2f1b5480c60f4e78f18727d568c7d86 Mon Sep 17 00:00:00 2001
From: Pavel Begunkov <asml.silence@gmail.com>
Date: Thu, 1 Apr 2021 15:43:59 +0100
Subject: [PATCH 1/2] io_uring: put link timeout req consistently

commit df9727affa058f4f18e388b30247650f8ae13cd8 upstream

Don't put linked timeout req in io_async_find_and_cancel() but do it in
io_link_timeout_fn(), so we have only one point for that and won't have
to do it differently as it's now (put vs put_deferred). Btw, improve a
bit io_async_find_and_cancel()'s locking.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/d75b70957f245275ab7cba83e0ac9c1b86aae78a.1617287883.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
[sudip: adjust context]
Signed-off-by: Sudip Mukherjee <sudipm.mukherjee@gmail.com>
---
 fs/io_uring.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 42153106b7bc..a6c9c55ca3a3 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -5697,12 +5697,9 @@ static void io_async_find_and_cancel(struct io_ring_ctx *ctx,
 	int ret;
 
 	ret = io_async_cancel_one(ctx, (void *) (unsigned long) sqe_addr);
-	if (ret != -ENOENT) {
-		spin_lock_irqsave(&ctx->completion_lock, flags);
-		goto done;
-	}
-
 	spin_lock_irqsave(&ctx->completion_lock, flags);
+	if (ret != -ENOENT)
+		goto done;
 	ret = io_timeout_cancel(ctx, sqe_addr);
 	if (ret != -ENOENT)
 		goto done;
@@ -5717,7 +5714,6 @@ static void io_async_find_and_cancel(struct io_ring_ctx *ctx,
 
 	if (ret < 0)
 		req_set_fail_links(req);
-	io_put_req(req);
 }
 
 static int io_async_cancel_prep(struct io_kiocb *req,
@@ -6263,8 +6259,8 @@ static enum hrtimer_restart io_link_timeout_fn(struct hrtimer *timer)
 		io_put_req_deferred(req, 1);
 	} else {
 		io_cqring_add_event(req, -ETIME, 0);
-		io_put_req_deferred(req, 1);
 	}
+	io_put_req_deferred(req, 1);
 	return HRTIMER_NORESTART;
 }
 
-- 
2.30.2


[-- Attachment #3: 0002-io_uring-fix-link-timeout-refs.patch --]
[-- Type: text/x-diff, Size: 2031 bytes --]

From b901fb09894731c4ad84a359509508eff50a4920 Mon Sep 17 00:00:00 2001
From: Pavel Begunkov <asml.silence@gmail.com>
Date: Fri, 7 May 2021 21:06:38 +0100
Subject: [PATCH 2/2] io_uring: fix link timeout refs

commit a298232ee6b9a1d5d732aa497ff8be0d45b5bd82 upstream

WARNING: CPU: 0 PID: 10242 at lib/refcount.c:28 refcount_warn_saturate+0x15b/0x1a0 lib/refcount.c:28
RIP: 0010:refcount_warn_saturate+0x15b/0x1a0 lib/refcount.c:28
Call Trace:
 __refcount_sub_and_test include/linux/refcount.h:283 [inline]
 __refcount_dec_and_test include/linux/refcount.h:315 [inline]
 refcount_dec_and_test include/linux/refcount.h:333 [inline]
 io_put_req fs/io_uring.c:2140 [inline]
 io_queue_linked_timeout fs/io_uring.c:6300 [inline]
 __io_queue_sqe+0xbef/0xec0 fs/io_uring.c:6354
 io_submit_sqe fs/io_uring.c:6534 [inline]
 io_submit_sqes+0x2bbd/0x7c50 fs/io_uring.c:6660
 __do_sys_io_uring_enter fs/io_uring.c:9240 [inline]
 __se_sys_io_uring_enter+0x256/0x1d60 fs/io_uring.c:9182

io_link_timeout_fn() should put only one reference of the linked timeout
request, however in case of racing with the master request's completion
first io_req_complete() puts one and then io_put_req_deferred() is
called.

Cc: stable@vger.kernel.org # 5.12+
Fixes: 9ae1f8dd372e0 ("io_uring: fix inconsistent lock state")
Reported-by: syzbot+a2910119328ce8e7996f@syzkaller.appspotmail.com
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/ff51018ff29de5ffa76f09273ef48cb24c720368.1620417627.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sudip Mukherjee <sudipm.mukherjee@gmail.com>
---
 fs/io_uring.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/fs/io_uring.c b/fs/io_uring.c
index a6c9c55ca3a3..1a5951fbc763 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -6260,7 +6260,6 @@ static enum hrtimer_restart io_link_timeout_fn(struct hrtimer *timer)
 	} else {
 		io_cqring_add_event(req, -ETIME, 0);
 	}
-	io_put_req_deferred(req, 1);
 	return HRTIMER_NORESTART;
 }
 
-- 
2.30.2


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: use-after-free" with v5.10.y caused by backport of a298232ee6b9 ("io_uring: fix link timeout refs")
  2021-07-26 12:44     ` Sudip Mukherjee
@ 2021-07-26 12:53       ` Pavel Begunkov
  0 siblings, 0 replies; 5+ messages in thread
From: Pavel Begunkov @ 2021-07-26 12:53 UTC (permalink / raw)
  To: Sudip Mukherjee, Greg Kroah-Hartman; +Cc: stable, Jens Axboe, Sasha Levin

On 7/26/21 1:44 PM, Sudip Mukherjee wrote:
> Hi Greg,
> 
> On Mon, Jul 26, 2021 at 01:08:38PM +0200, Greg Kroah-Hartman wrote:
>> On Mon, Jul 26, 2021 at 11:57:22AM +0100, Pavel Begunkov wrote:
>>> On 7/26/21 11:29 AM, Sudip Mukherjee wrote:
>>>> Hi Pavel,
>>>>
>>>> We had been running syzkaller on v5.10.y and a "use after free" is being
>>>> reported for v5.10.43+ kernels.
>>>
>>> "... # 5.12+", weird, but perhaps due to dependencies.
>>> Thanks for letting know.
>>>
>>>
>>> Greg, Sasha, should be same as reported for 5.12
>>>
>>> https://www.spinics.net/lists/stable/msg485116.html
>>>
>>> Can you try to apply it to 5.10 or should I resend?
>>
>> I just tried applying those patches and they did not work.  So can you
>> provide some new backports?
> 
> Here is the backport for v5.10.y. I have also tested these with the
> syzkaller repro and the issue is fixed.

Thanks trying out, but it should be leaking requests (rarely),
because io_cqring_add_event() doesn't put a ref unlike
io_req_complete_post() from the original patch.

I'll get to it today double checking refcounting.

-- 
Pavel Begunkov

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-07-26 12:53 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-26 10:29 use-after-free" with v5.10.y caused by backport of a298232ee6b9 ("io_uring: fix link timeout refs") Sudip Mukherjee
2021-07-26 10:57 ` Pavel Begunkov
2021-07-26 11:08   ` Greg Kroah-Hartman
2021-07-26 12:44     ` Sudip Mukherjee
2021-07-26 12:53       ` Pavel Begunkov

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.