* possible deadlock in skb_queue_tail
@ 2018-04-02 9:20 syzbot
2018-04-03 9:50 ` Kirill Tkhai
0 siblings, 1 reply; 6+ messages in thread
From: syzbot @ 2018-04-02 9:20 UTC (permalink / raw)
To: davem, dh.herrmann, dvlasenk, dwindsor, elena.reshetova,
ishkamiel, keescook, ktkhai, linux-kernel, matthew, mjurczyk,
netdev, syzkaller-bugs, viro
Hello,
syzbot hit the following crash on net-next commit
06b19fe9a6df7aaa423cd8404ebe5ac9ec4b2960 (Sun Apr 1 03:37:33 2018 +0000)
Merge branch 'chelsio-inline-tls'
syzbot dashboard link:
https://syzkaller.appspot.com/bug?extid=6b495100f17ca8554ab9
Unfortunately, I don't have any reproducer for this crash yet.
Raw console output:
https://syzkaller.appspot.com/x/log.txt?id=6218830443446272
Kernel config:
https://syzkaller.appspot.com/x/.config?id=3327544840960562528
compiler: gcc (GCC) 7.1.1 20170620
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+6b495100f17ca8554ab9@syzkaller.appspotmail.com
It will help syzbot understand when the bug is fixed. See footer for
details.
If you forward the report, please keep this part and the footer.
======================================================
WARNING: possible circular locking dependency detected
4.16.0-rc6+ #290 Not tainted
------------------------------------------------------
syz-executor7/20971 is trying to acquire lock:
(&af_unix_sk_receive_queue_lock_key){+.+.}, at: [<00000000271ef0d8>]
skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
but task is already holding lock:
(&(&u->lock)->rlock/1){+.+.}, at: [<000000004e725e14>]
unix_state_double_lock+0x7b/0xb0 net/unix/af_unix.c:1088
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&(&u->lock)->rlock/1){+.+.}:
_raw_spin_lock_nested+0x28/0x40 kernel/locking/spinlock.c:354
sk_diag_dump_icons net/unix/diag.c:82 [inline]
sk_diag_fill.isra.4+0xa52/0xfe0 net/unix/diag.c:144
sk_diag_dump net/unix/diag.c:178 [inline]
unix_diag_dump+0x400/0x4f0 net/unix/diag.c:206
netlink_dump+0x492/0xcf0 net/netlink/af_netlink.c:2221
__netlink_dump_start+0x4ec/0x710 net/netlink/af_netlink.c:2318
netlink_dump_start include/linux/netlink.h:214 [inline]
unix_diag_handler_dump+0x3e7/0x750 net/unix/diag.c:307
__sock_diag_cmd net/core/sock_diag.c:230 [inline]
sock_diag_rcv_msg+0x204/0x360 net/core/sock_diag.c:261
netlink_rcv_skb+0x14b/0x380 net/netlink/af_netlink.c:2443
sock_diag_rcv+0x2a/0x40 net/core/sock_diag.c:272
netlink_unicast_kernel net/netlink/af_netlink.c:1307 [inline]
netlink_unicast+0x4c4/0x6b0 net/netlink/af_netlink.c:1333
netlink_sendmsg+0xa4a/0xe80 net/netlink/af_netlink.c:1896
sock_sendmsg_nosec net/socket.c:629 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:639
sock_write_iter+0x31a/0x5d0 net/socket.c:908
call_write_iter include/linux/fs.h:1782 [inline]
new_sync_write fs/read_write.c:469 [inline]
__vfs_write+0x684/0x970 fs/read_write.c:482
vfs_write+0x189/0x510 fs/read_write.c:544
SYSC_write fs/read_write.c:589 [inline]
SyS_write+0xef/0x220 fs/read_write.c:581
do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
entry_SYSCALL_64_after_hwframe+0x42/0xb7
-> #0 (&af_unix_sk_receive_queue_lock_key){+.+.}:
lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
unix_dgram_sendmsg+0xa30/0x1610 net/unix/af_unix.c:1807
sock_sendmsg_nosec net/socket.c:629 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:639
___sys_sendmsg+0x320/0x8b0 net/socket.c:2047
__sys_sendmmsg+0x1ee/0x620 net/socket.c:2137
SYSC_sendmmsg net/socket.c:2168 [inline]
SyS_sendmmsg+0x35/0x60 net/socket.c:2163
do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
entry_SYSCALL_64_after_hwframe+0x42/0xb7
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&(&u->lock)->rlock/1);
lock(&af_unix_sk_receive_queue_lock_key);
lock(&(&u->lock)->rlock/1);
lock(&af_unix_sk_receive_queue_lock_key);
*** DEADLOCK ***
1 lock held by syz-executor7/20971:
#0: (&(&u->lock)->rlock/1){+.+.}, at: [<000000004e725e14>]
unix_state_double_lock+0x7b/0xb0 net/unix/af_unix.c:1088
stack backtrace:
CPU: 0 PID: 20971 Comm: syz-executor7 Not tainted 4.16.0-rc6+ #290
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:17 [inline]
dump_stack+0x194/0x24d lib/dump_stack.c:53
print_circular_bug.isra.38+0x2cd/0x2dc kernel/locking/lockdep.c:1223
check_prev_add kernel/locking/lockdep.c:1863 [inline]
check_prevs_add kernel/locking/lockdep.c:1976 [inline]
validate_chain kernel/locking/lockdep.c:2417 [inline]
__lock_acquire+0x30a8/0x3e00 kernel/locking/lockdep.c:3431
lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
__raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
_raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
unix_dgram_sendmsg+0xa30/0x1610 net/unix/af_unix.c:1807
sock_sendmsg_nosec net/socket.c:629 [inline]
sock_sendmsg+0xca/0x110 net/socket.c:639
___sys_sendmsg+0x320/0x8b0 net/socket.c:2047
__sys_sendmmsg+0x1ee/0x620 net/socket.c:2137
SYSC_sendmmsg net/socket.c:2168 [inline]
SyS_sendmmsg+0x35/0x60 net/socket.c:2163
do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x455269
RSP: 002b:00007f71ffad6c68 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
RAX: ffffffffffffffda RBX: 00007f71ffad76d4 RCX: 0000000000455269
RDX: 04924924924924f4 RSI: 0000000020000200 RDI: 0000000000000016
RBP: 000000000072bf58 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000200000d4 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000000004ca R14: 00000000006f9390 R15: 0000000000000001
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: sync thread started: state = BACKUP, mcast_ifn = bcsh0, syncid = 0,
id = 0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
IPVS: Unknown mcast interface: bcsh0
---
This bug is generated by a dumb bot. It may contain errors.
See https://goo.gl/tpsmEJ for details.
Direct all questions to syzkaller@googlegroups.com.
syzbot will keep track of this bug report.
If you forgot to add the Reported-by tag, once the fix for this bug is
merged
into any tree, please reply to this email with:
#syz fix: exact-commit-title
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug
report.
Note: all commands must start from beginning of the line in the email body.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: possible deadlock in skb_queue_tail
2018-04-02 9:20 possible deadlock in skb_queue_tail syzbot
@ 2018-04-03 9:50 ` Kirill Tkhai
2018-04-03 11:25 ` Dmitry Vyukov
0 siblings, 1 reply; 6+ messages in thread
From: Kirill Tkhai @ 2018-04-03 9:50 UTC (permalink / raw)
To: syzbot, davem, dh.herrmann, dvlasenk, dwindsor, elena.reshetova,
ishkamiel, keescook, linux-kernel, matthew, mjurczyk, netdev,
syzkaller-bugs, viro, xemul
On 02.04.2018 12:20, syzbot wrote:
> Hello,
>
> syzbot hit the following crash on net-next commit
> 06b19fe9a6df7aaa423cd8404ebe5ac9ec4b2960 (Sun Apr 1 03:37:33 2018 +0000)
> Merge branch 'chelsio-inline-tls'
> syzbot dashboard link: https://syzkaller.appspot.com/bug?extid=6b495100f17ca8554ab9
>
> Unfortunately, I don't have any reproducer for this crash yet.
> Raw console output: https://syzkaller.appspot.com/x/log.txt?id=6218830443446272
> Kernel config: https://syzkaller.appspot.com/x/.config?id=3327544840960562528
> compiler: gcc (GCC) 7.1.1 20170620
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+6b495100f17ca8554ab9@syzkaller.appspotmail.com
> It will help syzbot understand when the bug is fixed. See footer for details.
> If you forward the report, please keep this part and the footer.
>
>
> ======================================================
> WARNING: possible circular locking dependency detected
> 4.16.0-rc6+ #290 Not tainted
> ------------------------------------------------------
> syz-executor7/20971 is trying to acquire lock:
> (&af_unix_sk_receive_queue_lock_key){+.+.}, at: [<00000000271ef0d8>] skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
>
> but task is already holding lock:
> (&(&u->lock)->rlock/1){+.+.}, at: [<000000004e725e14>] unix_state_double_lock+0x7b/0xb0 net/unix/af_unix.c:1088
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #1 (&(&u->lock)->rlock/1){+.+.}:
> _raw_spin_lock_nested+0x28/0x40 kernel/locking/spinlock.c:354
> sk_diag_dump_icons net/unix/diag.c:82 [inline]
> sk_diag_fill.isra.4+0xa52/0xfe0 net/unix/diag.c:144
> sk_diag_dump net/unix/diag.c:178 [inline]
> unix_diag_dump+0x400/0x4f0 net/unix/diag.c:206
> netlink_dump+0x492/0xcf0 net/netlink/af_netlink.c:2221
> __netlink_dump_start+0x4ec/0x710 net/netlink/af_netlink.c:2318
> netlink_dump_start include/linux/netlink.h:214 [inline]
> unix_diag_handler_dump+0x3e7/0x750 net/unix/diag.c:307
> __sock_diag_cmd net/core/sock_diag.c:230 [inline]
> sock_diag_rcv_msg+0x204/0x360 net/core/sock_diag.c:261
> netlink_rcv_skb+0x14b/0x380 net/netlink/af_netlink.c:2443
> sock_diag_rcv+0x2a/0x40 net/core/sock_diag.c:272
> netlink_unicast_kernel net/netlink/af_netlink.c:1307 [inline]
> netlink_unicast+0x4c4/0x6b0 net/netlink/af_netlink.c:1333
> netlink_sendmsg+0xa4a/0xe80 net/netlink/af_netlink.c:1896
> sock_sendmsg_nosec net/socket.c:629 [inline]
> sock_sendmsg+0xca/0x110 net/socket.c:639
> sock_write_iter+0x31a/0x5d0 net/socket.c:908
> call_write_iter include/linux/fs.h:1782 [inline]
> new_sync_write fs/read_write.c:469 [inline]
> __vfs_write+0x684/0x970 fs/read_write.c:482
> vfs_write+0x189/0x510 fs/read_write.c:544
> SYSC_write fs/read_write.c:589 [inline]
> SyS_write+0xef/0x220 fs/read_write.c:581
> do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>
> -> #0 (&af_unix_sk_receive_queue_lock_key){+.+.}:
> lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
> skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
> unix_dgram_sendmsg+0xa30/0x1610 net/unix/af_unix.c:1807
> sock_sendmsg_nosec net/socket.c:629 [inline]
> sock_sendmsg+0xca/0x110 net/socket.c:639
> ___sys_sendmsg+0x320/0x8b0 net/socket.c:2047
> __sys_sendmmsg+0x1ee/0x620 net/socket.c:2137
> SYSC_sendmmsg net/socket.c:2168 [inline]
> SyS_sendmmsg+0x35/0x60 net/socket.c:2163
> do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
> entry_SYSCALL_64_after_hwframe+0x42/0xb7
sk_diag_dump_icons() dumps only sockets in TCP_LISTEN state.
TCP_LISTEN state may be assigned in only place in net/unix/af_unix.c:
it's unix_listen(). The function is applied to stream and seqpacket
socket types.
It can't be stream because of the second stack, and seqpacket also can't,
as I don't think it's possible for gcc to inline unix_seqpacket_sendmsg()
in the way, we don't see it in the stack.
So, this is looks like false positive result for me.
Kirill
>
> other info that might help us debug this:
>
> Possible unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(&(&u->lock)->rlock/1);
> lock(&af_unix_sk_receive_queue_lock_key);
> lock(&(&u->lock)->rlock/1);
> lock(&af_unix_sk_receive_queue_lock_key);
>
> *** DEADLOCK ***
>
> 1 lock held by syz-executor7/20971:
> #0: (&(&u->lock)->rlock/1){+.+.}, at: [<000000004e725e14>] unix_state_double_lock+0x7b/0xb0 net/unix/af_unix.c:1088
>
> stack backtrace:
> CPU: 0 PID: 20971 Comm: syz-executor7 Not tainted 4.16.0-rc6+ #290
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:17 [inline]
> dump_stack+0x194/0x24d lib/dump_stack.c:53
> print_circular_bug.isra.38+0x2cd/0x2dc kernel/locking/lockdep.c:1223
> check_prev_add kernel/locking/lockdep.c:1863 [inline]
> check_prevs_add kernel/locking/lockdep.c:1976 [inline]
> validate_chain kernel/locking/lockdep.c:2417 [inline]
> __lock_acquire+0x30a8/0x3e00 kernel/locking/lockdep.c:3431
> lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
> skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
> unix_dgram_sendmsg+0xa30/0x1610 net/unix/af_unix.c:1807
> sock_sendmsg_nosec net/socket.c:629 [inline]
> sock_sendmsg+0xca/0x110 net/socket.c:639
> ___sys_sendmsg+0x320/0x8b0 net/socket.c:2047
> __sys_sendmmsg+0x1ee/0x620 net/socket.c:2137
> SYSC_sendmmsg net/socket.c:2168 [inline]
> SyS_sendmmsg+0x35/0x60 net/socket.c:2163
> do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
> entry_SYSCALL_64_after_hwframe+0x42/0xb7
> RIP: 0033:0x455269
> RSP: 002b:00007f71ffad6c68 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
> RAX: ffffffffffffffda RBX: 00007f71ffad76d4 RCX: 0000000000455269
> RDX: 04924924924924f4 RSI: 0000000020000200 RDI: 0000000000000016
> RBP: 000000000072bf58 R08: 0000000000000000 R09: 0000000000000000
> R10: 00000000200000d4 R11: 0000000000000246 R12: 00000000ffffffff
> R13: 00000000000004ca R14: 00000000006f9390 R15: 0000000000000001
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: sync thread started: state = BACKUP, mcast_ifn = bcsh0, syncid = 0, id = 0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
> IPVS: Unknown mcast interface: bcsh0
>
>
> ---
> This bug is generated by a dumb bot. It may contain errors.
> See https://goo.gl/tpsmEJ for details.
> Direct all questions to syzkaller@googlegroups.com.
>
> syzbot will keep track of this bug report.
> If you forgot to add the Reported-by tag, once the fix for this bug is merged
> into any tree, please reply to this email with:
> #syz fix: exact-commit-title
> To mark this as a duplicate of another syzbot report, please reply with:
> #syz dup: exact-subject-of-another-report
> If it's a one-off invalid bug report, please reply with:
> #syz invalid
> Note: if the crash happens again, it will cause creation of a new bug report.
> Note: all commands must start from beginning of the line in the email body.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: possible deadlock in skb_queue_tail
2018-04-03 9:50 ` Kirill Tkhai
@ 2018-04-03 11:25 ` Dmitry Vyukov
2018-04-03 11:42 ` Kirill Tkhai
0 siblings, 1 reply; 6+ messages in thread
From: Dmitry Vyukov @ 2018-04-03 11:25 UTC (permalink / raw)
To: Kirill Tkhai, Ingo Molnar
Cc: syzbot, David Miller, David Herrmann, Denys Vlasenko,
David Windsor, elena.reshetova, ishkamiel, Kees Cook, LKML,
matthew, Mateusz Jurczyk, netdev, syzkaller-bugs, Al Viro, xemul
On Tue, Apr 3, 2018 at 11:50 AM, Kirill Tkhai <ktkhai@virtuozzo.com> wrote:
> On 02.04.2018 12:20, syzbot wrote:
>> Hello,
>>
>> syzbot hit the following crash on net-next commit
>> 06b19fe9a6df7aaa423cd8404ebe5ac9ec4b2960 (Sun Apr 1 03:37:33 2018 +0000)
>> Merge branch 'chelsio-inline-tls'
>> syzbot dashboard link: https://syzkaller.appspot.com/bug?extid=6b495100f17ca8554ab9
>>
>> Unfortunately, I don't have any reproducer for this crash yet.
>> Raw console output: https://syzkaller.appspot.com/x/log.txt?id=6218830443446272
>> Kernel config: https://syzkaller.appspot.com/x/.config?id=3327544840960562528
>> compiler: gcc (GCC) 7.1.1 20170620
>>
>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> Reported-by: syzbot+6b495100f17ca8554ab9@syzkaller.appspotmail.com
>> It will help syzbot understand when the bug is fixed. See footer for details.
>> If you forward the report, please keep this part and the footer.
>>
>>
>> ======================================================
>> WARNING: possible circular locking dependency detected
>> 4.16.0-rc6+ #290 Not tainted
>> ------------------------------------------------------
>> syz-executor7/20971 is trying to acquire lock:
>> (&af_unix_sk_receive_queue_lock_key){+.+.}, at: [<00000000271ef0d8>] skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
>>
>> but task is already holding lock:
>> (&(&u->lock)->rlock/1){+.+.}, at: [<000000004e725e14>] unix_state_double_lock+0x7b/0xb0 net/unix/af_unix.c:1088
>>
>> which lock already depends on the new lock.
>>
>>
>> the existing dependency chain (in reverse order) is:
>>
>> -> #1 (&(&u->lock)->rlock/1){+.+.}:
>> _raw_spin_lock_nested+0x28/0x40 kernel/locking/spinlock.c:354
>> sk_diag_dump_icons net/unix/diag.c:82 [inline]
>> sk_diag_fill.isra.4+0xa52/0xfe0 net/unix/diag.c:144
>> sk_diag_dump net/unix/diag.c:178 [inline]
>> unix_diag_dump+0x400/0x4f0 net/unix/diag.c:206
>> netlink_dump+0x492/0xcf0 net/netlink/af_netlink.c:2221
>> __netlink_dump_start+0x4ec/0x710 net/netlink/af_netlink.c:2318
>> netlink_dump_start include/linux/netlink.h:214 [inline]
>> unix_diag_handler_dump+0x3e7/0x750 net/unix/diag.c:307
>> __sock_diag_cmd net/core/sock_diag.c:230 [inline]
>> sock_diag_rcv_msg+0x204/0x360 net/core/sock_diag.c:261
>> netlink_rcv_skb+0x14b/0x380 net/netlink/af_netlink.c:2443
>> sock_diag_rcv+0x2a/0x40 net/core/sock_diag.c:272
>> netlink_unicast_kernel net/netlink/af_netlink.c:1307 [inline]
>> netlink_unicast+0x4c4/0x6b0 net/netlink/af_netlink.c:1333
>> netlink_sendmsg+0xa4a/0xe80 net/netlink/af_netlink.c:1896
>> sock_sendmsg_nosec net/socket.c:629 [inline]
>> sock_sendmsg+0xca/0x110 net/socket.c:639
>> sock_write_iter+0x31a/0x5d0 net/socket.c:908
>> call_write_iter include/linux/fs.h:1782 [inline]
>> new_sync_write fs/read_write.c:469 [inline]
>> __vfs_write+0x684/0x970 fs/read_write.c:482
>> vfs_write+0x189/0x510 fs/read_write.c:544
>> SYSC_write fs/read_write.c:589 [inline]
>> SyS_write+0xef/0x220 fs/read_write.c:581
>> do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>>
>> -> #0 (&af_unix_sk_receive_queue_lock_key){+.+.}:
>> lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
>> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
>> _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
>> skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
>> unix_dgram_sendmsg+0xa30/0x1610 net/unix/af_unix.c:1807
>> sock_sendmsg_nosec net/socket.c:629 [inline]
>> sock_sendmsg+0xca/0x110 net/socket.c:639
>> ___sys_sendmsg+0x320/0x8b0 net/socket.c:2047
>> __sys_sendmmsg+0x1ee/0x620 net/socket.c:2137
>> SYSC_sendmmsg net/socket.c:2168 [inline]
>> SyS_sendmmsg+0x35/0x60 net/socket.c:2163
>> do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>
> sk_diag_dump_icons() dumps only sockets in TCP_LISTEN state.
> TCP_LISTEN state may be assigned in only place in net/unix/af_unix.c:
> it's unix_listen(). The function is applied to stream and seqpacket
> socket types.
>
> It can't be stream because of the second stack, and seqpacket also can't,
> as I don't think it's possible for gcc to inline unix_seqpacket_sendmsg()
> in the way, we don't see it in the stack.
>
> So, this is looks like false positive result for me.
>
> Kirill
Do you mean that these &(&u->lock)->rlock/1 referenced in 2 stacks are
always different?
+Ingo for lockdep false positive
Do we need some kind of annotation here?
>> other info that might help us debug this:
>>
>> Possible unsafe locking scenario:
>>
>> CPU0 CPU1
>> ---- ----
>> lock(&(&u->lock)->rlock/1);
>> lock(&af_unix_sk_receive_queue_lock_key);
>> lock(&(&u->lock)->rlock/1);
>> lock(&af_unix_sk_receive_queue_lock_key);
>>
>> *** DEADLOCK ***
>>
>> 1 lock held by syz-executor7/20971:
>> #0: (&(&u->lock)->rlock/1){+.+.}, at: [<000000004e725e14>] unix_state_double_lock+0x7b/0xb0 net/unix/af_unix.c:1088
>>
>> stack backtrace:
>> CPU: 0 PID: 20971 Comm: syz-executor7 Not tainted 4.16.0-rc6+ #290
>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
>> Call Trace:
>> __dump_stack lib/dump_stack.c:17 [inline]
>> dump_stack+0x194/0x24d lib/dump_stack.c:53
>> print_circular_bug.isra.38+0x2cd/0x2dc kernel/locking/lockdep.c:1223
>> check_prev_add kernel/locking/lockdep.c:1863 [inline]
>> check_prevs_add kernel/locking/lockdep.c:1976 [inline]
>> validate_chain kernel/locking/lockdep.c:2417 [inline]
>> __lock_acquire+0x30a8/0x3e00 kernel/locking/lockdep.c:3431
>> lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
>> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
>> _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
>> skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
>> unix_dgram_sendmsg+0xa30/0x1610 net/unix/af_unix.c:1807
>> sock_sendmsg_nosec net/socket.c:629 [inline]
>> sock_sendmsg+0xca/0x110 net/socket.c:639
>> ___sys_sendmsg+0x320/0x8b0 net/socket.c:2047
>> __sys_sendmmsg+0x1ee/0x620 net/socket.c:2137
>> SYSC_sendmmsg net/socket.c:2168 [inline]
>> SyS_sendmmsg+0x35/0x60 net/socket.c:2163
>> do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>> RIP: 0033:0x455269
>> RSP: 002b:00007f71ffad6c68 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
>> RAX: ffffffffffffffda RBX: 00007f71ffad76d4 RCX: 0000000000455269
>> RDX: 04924924924924f4 RSI: 0000000020000200 RDI: 0000000000000016
>> RBP: 000000000072bf58 R08: 0000000000000000 R09: 0000000000000000
>> R10: 00000000200000d4 R11: 0000000000000246 R12: 00000000ffffffff
>> R13: 00000000000004ca R14: 00000000006f9390 R15: 0000000000000001
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: sync thread started: state = BACKUP, mcast_ifn = bcsh0, syncid = 0, id = 0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>> IPVS: Unknown mcast interface: bcsh0
>>
>>
>> ---
>> This bug is generated by a dumb bot. It may contain errors.
>> See https://goo.gl/tpsmEJ for details.
>> Direct all questions to syzkaller@googlegroups.com.
>>
>> syzbot will keep track of this bug report.
>> If you forgot to add the Reported-by tag, once the fix for this bug is merged
>> into any tree, please reply to this email with:
>> #syz fix: exact-commit-title
>> To mark this as a duplicate of another syzbot report, please reply with:
>> #syz dup: exact-subject-of-another-report
>> If it's a one-off invalid bug report, please reply with:
>> #syz invalid
>> Note: if the crash happens again, it will cause creation of a new bug report.
>> Note: all commands must start from beginning of the line in the email body.
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/06c79d3f-3f28-7f1e-9431-66c18149c9e6%40virtuozzo.com.
> For more options, visit https://groups.google.com/d/optout.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: possible deadlock in skb_queue_tail
2018-04-03 11:25 ` Dmitry Vyukov
@ 2018-04-03 11:42 ` Kirill Tkhai
2018-04-04 5:08 ` Cong Wang
0 siblings, 1 reply; 6+ messages in thread
From: Kirill Tkhai @ 2018-04-03 11:42 UTC (permalink / raw)
To: Dmitry Vyukov, Ingo Molnar
Cc: syzbot, David Miller, David Herrmann, Denys Vlasenko,
David Windsor, elena.reshetova, ishkamiel, Kees Cook, LKML,
matthew, Mateusz Jurczyk, netdev, syzkaller-bugs, Al Viro, xemul
On 03.04.2018 14:25, Dmitry Vyukov wrote:
> On Tue, Apr 3, 2018 at 11:50 AM, Kirill Tkhai <ktkhai@virtuozzo.com> wrote:
>> On 02.04.2018 12:20, syzbot wrote:
>>> Hello,
>>>
>>> syzbot hit the following crash on net-next commit
>>> 06b19fe9a6df7aaa423cd8404ebe5ac9ec4b2960 (Sun Apr 1 03:37:33 2018 +0000)
>>> Merge branch 'chelsio-inline-tls'
>>> syzbot dashboard link: https://syzkaller.appspot.com/bug?extid=6b495100f17ca8554ab9
>>>
>>> Unfortunately, I don't have any reproducer for this crash yet.
>>> Raw console output: https://syzkaller.appspot.com/x/log.txt?id=6218830443446272
>>> Kernel config: https://syzkaller.appspot.com/x/.config?id=3327544840960562528
>>> compiler: gcc (GCC) 7.1.1 20170620
>>>
>>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>>> Reported-by: syzbot+6b495100f17ca8554ab9@syzkaller.appspotmail.com
>>> It will help syzbot understand when the bug is fixed. See footer for details.
>>> If you forward the report, please keep this part and the footer.
>>>
>>>
>>> ======================================================
>>> WARNING: possible circular locking dependency detected
>>> 4.16.0-rc6+ #290 Not tainted
>>> ------------------------------------------------------
>>> syz-executor7/20971 is trying to acquire lock:
>>> (&af_unix_sk_receive_queue_lock_key){+.+.}, at: [<00000000271ef0d8>] skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
>>>
>>> but task is already holding lock:
>>> (&(&u->lock)->rlock/1){+.+.}, at: [<000000004e725e14>] unix_state_double_lock+0x7b/0xb0 net/unix/af_unix.c:1088
>>>
>>> which lock already depends on the new lock.
>>>
>>>
>>> the existing dependency chain (in reverse order) is:
>>>
>>> -> #1 (&(&u->lock)->rlock/1){+.+.}:
>>> _raw_spin_lock_nested+0x28/0x40 kernel/locking/spinlock.c:354
>>> sk_diag_dump_icons net/unix/diag.c:82 [inline]
>>> sk_diag_fill.isra.4+0xa52/0xfe0 net/unix/diag.c:144
>>> sk_diag_dump net/unix/diag.c:178 [inline]
>>> unix_diag_dump+0x400/0x4f0 net/unix/diag.c:206
>>> netlink_dump+0x492/0xcf0 net/netlink/af_netlink.c:2221
>>> __netlink_dump_start+0x4ec/0x710 net/netlink/af_netlink.c:2318
>>> netlink_dump_start include/linux/netlink.h:214 [inline]
>>> unix_diag_handler_dump+0x3e7/0x750 net/unix/diag.c:307
>>> __sock_diag_cmd net/core/sock_diag.c:230 [inline]
>>> sock_diag_rcv_msg+0x204/0x360 net/core/sock_diag.c:261
>>> netlink_rcv_skb+0x14b/0x380 net/netlink/af_netlink.c:2443
>>> sock_diag_rcv+0x2a/0x40 net/core/sock_diag.c:272
>>> netlink_unicast_kernel net/netlink/af_netlink.c:1307 [inline]
>>> netlink_unicast+0x4c4/0x6b0 net/netlink/af_netlink.c:1333
>>> netlink_sendmsg+0xa4a/0xe80 net/netlink/af_netlink.c:1896
>>> sock_sendmsg_nosec net/socket.c:629 [inline]
>>> sock_sendmsg+0xca/0x110 net/socket.c:639
>>> sock_write_iter+0x31a/0x5d0 net/socket.c:908
>>> call_write_iter include/linux/fs.h:1782 [inline]
>>> new_sync_write fs/read_write.c:469 [inline]
>>> __vfs_write+0x684/0x970 fs/read_write.c:482
>>> vfs_write+0x189/0x510 fs/read_write.c:544
>>> SYSC_write fs/read_write.c:589 [inline]
>>> SyS_write+0xef/0x220 fs/read_write.c:581
>>> do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
>>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>>>
>>> -> #0 (&af_unix_sk_receive_queue_lock_key){+.+.}:
>>> lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
>>> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
>>> _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
>>> skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
>>> unix_dgram_sendmsg+0xa30/0x1610 net/unix/af_unix.c:1807
>>> sock_sendmsg_nosec net/socket.c:629 [inline]
>>> sock_sendmsg+0xca/0x110 net/socket.c:639
>>> ___sys_sendmsg+0x320/0x8b0 net/socket.c:2047
>>> __sys_sendmmsg+0x1ee/0x620 net/socket.c:2137
>>> SYSC_sendmmsg net/socket.c:2168 [inline]
>>> SyS_sendmmsg+0x35/0x60 net/socket.c:2163
>>> do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
>>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>>
>> sk_diag_dump_icons() dumps only sockets in TCP_LISTEN state.
>> TCP_LISTEN state may be assigned in only place in net/unix/af_unix.c:
>> it's unix_listen(). The function is applied to stream and seqpacket
>> socket types.
>>
>> It can't be stream because of the second stack, and seqpacket also can't,
>> as I don't think it's possible for gcc to inline unix_seqpacket_sendmsg()
>> in the way, we don't see it in the stack.
>>
>> So, this is looks like false positive result for me.
>>
>> Kirill
>
> Do you mean that these &(&u->lock)->rlock/1 referenced in 2 stacks are
> always different?
In these 2 particular stacks they have to be different.
But we may meet another stacks, where stream or seqpacket
functions are used instead of unix_dgram_sendmsg(), and
they may be true positive.
Kirill
> +Ingo for lockdep false positive
> Do we need some kind of annotation here?
>
>
>>> other info that might help us debug this:
>>>
>>> Possible unsafe locking scenario:
>>>
>>> CPU0 CPU1
>>> ---- ----
>>> lock(&(&u->lock)->rlock/1);
>>> lock(&af_unix_sk_receive_queue_lock_key);
>>> lock(&(&u->lock)->rlock/1);
>>> lock(&af_unix_sk_receive_queue_lock_key);
>>>
>>> *** DEADLOCK ***
>>>
>>> 1 lock held by syz-executor7/20971:
>>> #0: (&(&u->lock)->rlock/1){+.+.}, at: [<000000004e725e14>] unix_state_double_lock+0x7b/0xb0 net/unix/af_unix.c:1088
>>>
>>> stack backtrace:
>>> CPU: 0 PID: 20971 Comm: syz-executor7 Not tainted 4.16.0-rc6+ #290
>>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
>>> Call Trace:
>>> __dump_stack lib/dump_stack.c:17 [inline]
>>> dump_stack+0x194/0x24d lib/dump_stack.c:53
>>> print_circular_bug.isra.38+0x2cd/0x2dc kernel/locking/lockdep.c:1223
>>> check_prev_add kernel/locking/lockdep.c:1863 [inline]
>>> check_prevs_add kernel/locking/lockdep.c:1976 [inline]
>>> validate_chain kernel/locking/lockdep.c:2417 [inline]
>>> __lock_acquire+0x30a8/0x3e00 kernel/locking/lockdep.c:3431
>>> lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:3920
>>> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
>>> _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
>>> skb_queue_tail+0x26/0x150 net/core/skbuff.c:2899
>>> unix_dgram_sendmsg+0xa30/0x1610 net/unix/af_unix.c:1807
>>> sock_sendmsg_nosec net/socket.c:629 [inline]
>>> sock_sendmsg+0xca/0x110 net/socket.c:639
>>> ___sys_sendmsg+0x320/0x8b0 net/socket.c:2047
>>> __sys_sendmmsg+0x1ee/0x620 net/socket.c:2137
>>> SYSC_sendmmsg net/socket.c:2168 [inline]
>>> SyS_sendmmsg+0x35/0x60 net/socket.c:2163
>>> do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
>>> entry_SYSCALL_64_after_hwframe+0x42/0xb7
>>> RIP: 0033:0x455269
>>> RSP: 002b:00007f71ffad6c68 EFLAGS: 00000246 ORIG_RAX: 0000000000000133
>>> RAX: ffffffffffffffda RBX: 00007f71ffad76d4 RCX: 0000000000455269
>>> RDX: 04924924924924f4 RSI: 0000000020000200 RDI: 0000000000000016
>>> RBP: 000000000072bf58 R08: 0000000000000000 R09: 0000000000000000
>>> R10: 00000000200000d4 R11: 0000000000000246 R12: 00000000ffffffff
>>> R13: 00000000000004ca R14: 00000000006f9390 R15: 0000000000000001
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: sync thread started: state = BACKUP, mcast_ifn = bcsh0, syncid = 0, id = 0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>> IPVS: Unknown mcast interface: bcsh0
>>>
>>>
>>> ---
>>> This bug is generated by a dumb bot. It may contain errors.
>>> See https://goo.gl/tpsmEJ for details.
>>> Direct all questions to syzkaller@googlegroups.com.
>>>
>>> syzbot will keep track of this bug report.
>>> If you forgot to add the Reported-by tag, once the fix for this bug is merged
>>> into any tree, please reply to this email with:
>>> #syz fix: exact-commit-title
>>> To mark this as a duplicate of another syzbot report, please reply with:
>>> #syz dup: exact-subject-of-another-report
>>> If it's a one-off invalid bug report, please reply with:
>>> #syz invalid
>>> Note: if the crash happens again, it will cause creation of a new bug report.
>>> Note: all commands must start from beginning of the line in the email body.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/06c79d3f-3f28-7f1e-9431-66c18149c9e6%40virtuozzo.com.
>> For more options, visit https://groups.google.com/d/optout.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: possible deadlock in skb_queue_tail
2018-04-03 11:42 ` Kirill Tkhai
@ 2018-04-04 5:08 ` Cong Wang
2018-04-04 19:00 ` Dmitry Vyukov
0 siblings, 1 reply; 6+ messages in thread
From: Cong Wang @ 2018-04-04 5:08 UTC (permalink / raw)
To: Kirill Tkhai
Cc: Dmitry Vyukov, Ingo Molnar, syzbot, David Miller, David Herrmann,
Denys Vlasenko, David Windsor, Reshetova, Elena, ishkamiel,
Kees Cook, LKML, matthew, Mateusz Jurczyk, netdev,
syzkaller-bugs, Al Viro, xemul
On Tue, Apr 3, 2018 at 4:42 AM, Kirill Tkhai <ktkhai@virtuozzo.com> wrote:
> On 03.04.2018 14:25, Dmitry Vyukov wrote:
>> On Tue, Apr 3, 2018 at 11:50 AM, Kirill Tkhai <ktkhai@virtuozzo.com> wrote:
>>> sk_diag_dump_icons() dumps only sockets in TCP_LISTEN state.
>>> TCP_LISTEN state may be assigned in only place in net/unix/af_unix.c:
>>> it's unix_listen(). The function is applied to stream and seqpacket
>>> socket types.
>>>
>>> It can't be stream because of the second stack, and seqpacket also can't,
>>> as I don't think it's possible for gcc to inline unix_seqpacket_sendmsg()
>>> in the way, we don't see it in the stack.
>>>
>>> So, this is looks like false positive result for me.
>>>
>>> Kirill
>>
>> Do you mean that these &(&u->lock)->rlock/1 referenced in 2 stacks are
>> always different?
>
> In these 2 particular stacks they have to be different.
So actually my patch could fix this false positive? I thought it couldn't.
https://patchwork.ozlabs.org/patch/894342/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: possible deadlock in skb_queue_tail
2018-04-04 5:08 ` Cong Wang
@ 2018-04-04 19:00 ` Dmitry Vyukov
0 siblings, 0 replies; 6+ messages in thread
From: Dmitry Vyukov @ 2018-04-04 19:00 UTC (permalink / raw)
To: Cong Wang
Cc: Kirill Tkhai, Ingo Molnar, syzbot, David Miller, David Herrmann,
Denys Vlasenko, David Windsor, Reshetova, Elena,
Hans Liljestrand, Kees Cook, LKML, Matthew Dawson,
Mateusz Jurczyk, netdev, syzkaller-bugs, Al Viro, xemul
On Wed, Apr 4, 2018 at 7:08 AM, Cong Wang <xiyou.wangcong@gmail.com> wrote:
> On Tue, Apr 3, 2018 at 4:42 AM, Kirill Tkhai <ktkhai@virtuozzo.com> wrote:
>> On 03.04.2018 14:25, Dmitry Vyukov wrote:
>>> On Tue, Apr 3, 2018 at 11:50 AM, Kirill Tkhai <ktkhai@virtuozzo.com> wrote:
>>>> sk_diag_dump_icons() dumps only sockets in TCP_LISTEN state.
>>>> TCP_LISTEN state may be assigned in only place in net/unix/af_unix.c:
>>>> it's unix_listen(). The function is applied to stream and seqpacket
>>>> socket types.
>>>>
>>>> It can't be stream because of the second stack, and seqpacket also can't,
>>>> as I don't think it's possible for gcc to inline unix_seqpacket_sendmsg()
>>>> in the way, we don't see it in the stack.
>>>>
>>>> So, this is looks like false positive result for me.
>>>>
>>>> Kirill
>>>
>>> Do you mean that these &(&u->lock)->rlock/1 referenced in 2 stacks are
>>> always different?
>>
>> In these 2 particular stacks they have to be different.
>
> So actually my patch could fix this false positive? I thought it couldn't.
> https://patchwork.ozlabs.org/patch/894342/
You know better!
If you suspect it can fix this report, and nobody has better
proposals, then we can just mark this as being fixed with your commit
and then see if it triggers again with your commit or not.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-04-04 19:00 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-02 9:20 possible deadlock in skb_queue_tail syzbot
2018-04-03 9:50 ` Kirill Tkhai
2018-04-03 11:25 ` Dmitry Vyukov
2018-04-03 11:42 ` Kirill Tkhai
2018-04-04 5:08 ` Cong Wang
2018-04-04 19:00 ` Dmitry Vyukov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).