All of lore.kernel.org
 help / color / mirror / Atom feed
* INFO: rcu detected stall in corrupted
@ 2018-05-21 18:05 syzbot
  2018-05-21 18:09 ` David Miller
  0 siblings, 1 reply; 11+ messages in thread
From: syzbot @ 2018-05-21 18:05 UTC (permalink / raw)
  To: davem, kuznet, linux-kernel, netdev, syzkaller-bugs, yoshfuji

Hello,

syzbot found the following crash on:

HEAD commit:    771c577c23ba Linux 4.17-rc6
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=17134357800000
kernel config:  https://syzkaller.appspot.com/x/.config?x=982e2df1b9e60b02
dashboard link: https://syzkaller.appspot.com/bug?extid=f116bc1994efe725d51b
compiler:       gcc (GCC) 8.0.1 20180413 (experimental)
userspace arch: i386
syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=14e5a7cf800000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+f116bc1994efe725d51b@syzkaller.appspotmail.com

IPv6: ADDRCONF(NETDEV_UP): veth1: link is not ready
IPv6: ADDRCONF(NETDEV_CHANGE): veth1: link becomes ready
IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
8021q: adding VLAN 0 to HW filter on device team0
8021q: adding VLAN 0 to HW filter on device team0
INFO: rcu_sched self-detected stall on CPU
INFO: rcu_sched detected stalls on CPUs/tasks:
	0-...!: (124975 ticks this GP) idle=a36/1/4611686018427387906  
softirq=14002/14002 fqs=10
	
	0-...!: (124975 ticks this GP) idle=a36/1/4611686018427387906  
softirq=14002/14002 fqs=10
	
  (t=125002 jiffies g=7347 c=7346 q=349000)
(detected by 1, t=125002 jiffies, g=7347, c=7346, q=349000)
rcu_sched kthread starved for 124927 jiffies! g7347 c7346 f0x2  
RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=1
Sending NMI from CPU 1 to CPUs 0:
RCU grace-period kthread stack dump:
NMI backtrace for cpu 0
CPU: 0 PID: 8 Comm: ksoftirqd/0 Not tainted 4.17.0-rc6+ #86
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
RIP: 0010:get_current arch/x86/include/asm/current.h:15 [inline]
RIP: 0010:write_comp_data+0xa/0x70 kernel/kcov.c:121
RSP: 0018:ffff8801dae06d30 EFLAGS: 00000006
RAX: 0000000000010105 RBX: 0000000000000006 RCX: ffffffff876bdc58
RDX: 0000000000000000 RSI: 0000000000000005 RDI: 0000000000000001
RBP: ffff8801dae06d68 R08: ffff8801d9a9c200 R09: fffffbfff14da4bc
R10: fffffbfff14da4bc R11: ffffffff8a6d25e0 R12: ffffffff88644220
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000008
FS:  0000000000000000(0000) GS:ffff8801dae00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f595e194270 CR3: 00000001b09c2000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
  <IRQ>
  vsnprintf+0x1b8/0x1b40 lib/vsprintf.c:2252
  sprintf+0xa7/0xd0 lib/vsprintf.c:2498
  print_time kernel/printk/printk.c:1223 [inline]
  print_prefix+0x26a/0x3f0 kernel/printk/printk.c:1246
  msg_print_text+0xca/0x1c0 kernel/printk/printk.c:1273
  console_unlock+0x4f5/0x1100 kernel/printk/printk.c:2369
  vprintk_emit+0x6ad/0xdd0 kernel/printk/printk.c:1907
  vprintk_default+0x28/0x30 kernel/printk/printk.c:1947
  vprintk_func+0x7a/0xe7 kernel/printk/printk_safe.c:379
  printk+0x9e/0xba kernel/printk/printk.c:1980
  rcu_check_gp_kthread_starvation+0x325/0x3a4 kernel/rcu/tree.c:1353
  print_cpu_stall kernel/rcu/tree.c:1523 [inline]
  check_cpu_stall.isra.61.cold.80+0x364/0x59a kernel/rcu/tree.c:1593
  __rcu_pending kernel/rcu/tree.c:3356 [inline]
  rcu_pending kernel/rcu/tree.c:3401 [inline]
  rcu_check_callbacks+0x21b/0xad0 kernel/rcu/tree.c:2763
  update_process_times+0x2d/0x70 kernel/time/timer.c:1636
  tick_sched_handle+0x9f/0x180 kernel/time/tick-sched.c:164
  tick_sched_timer+0x45/0x130 kernel/time/tick-sched.c:1274
  __run_hrtimer kernel/time/hrtimer.c:1398 [inline]
  __hrtimer_run_queues+0x3e3/0x10a0 kernel/time/hrtimer.c:1460
  hrtimer_interrupt+0x2f3/0x750 kernel/time/hrtimer.c:1518
  local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1025 [inline]
  smp_apic_timer_interrupt+0x15d/0x710 arch/x86/kernel/apic/apic.c:1050
  apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:863
  </IRQ>
RIP: 0010:__sanitizer_cov_trace_pc+0x1/0x50 kernel/kcov.c:94
RSP: 0018:ffff8801d9aad680 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000103 RBX: 0000000000000002 RCX: ffffffff867e02e0
RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000005
RBP: ffff8801d9aad7e0 R08: ffff8801d9a9c200 R09: ffff8801d9aadaf0
R10: ffffed003b5c46c2 R11: ffff8801dae23613 R12: ffff8801ce597c40
R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000000
  find_match+0x244/0x13a0 net/ipv6/route.c:691
  find_rr_leaf net/ipv6/route.c:729 [inline]
  rt6_select net/ipv6/route.c:779 [inline]
  ip6_pol_route+0x946/0x3d40 net/ipv6/route.c:1705
  ip6_pol_route_output+0x54/0x70 net/ipv6/route.c:1969
  fib6_rule_lookup+0x211/0x6d0 net/ipv6/fib6_rules.c:89
  ip6_route_output_flags+0x2c5/0x350 net/ipv6/route.c:1997
  ip6_dst_lookup_tail+0x47b/0x1b30 net/ipv6/ip6_output.c:995
  ip6_dst_lookup_flow+0xc1/0x260 net/ipv6/ip6_output.c:1096
  sctp_v6_get_dst+0x16b4/0x20b0 net/sctp/ipv6.c:327
  sctp_transport_route+0xad/0x450 net/sctp/transport.c:293
  sctp_packet_config+0xb89/0xfd0 net/sctp/output.c:123
  sctp_outq_flush+0x79c/0x4370 net/sctp/outqueue.c:894
  ? trace_hardirqs_off
Lost 148 message(s)!
rcu_sched kthread starved for 124927 jiffies! g7347 c7346 f0x2  
RCU_GP_WAIT_FQS(3) ->state=0x0 ->cpu=1
rcu_sched       R
RCU grace-period kthread stack dump:
rcu_sched       R
   running task
   running task    23896     9      2 0x80000000
23896     9      2 0x80000000
Call Trace:
Call Trace:
  context_switch kernel/sched/core.c:2859 [inline]
  __schedule+0x801/0x1e30 kernel/sched/core.c:3501
  context_switch kernel/sched/core.c:2859 [inline]
  __schedule+0x801/0x1e30 kernel/sched/core.c:3501
  schedule+0xef/0x430 kernel/sched/core.c:3545
  schedule+0xef/0x430 kernel/sched/core.c:3545
  schedule_timeout+0x138/0x240 kernel/time/timer.c:1801
  schedule_timeout+0x138/0x240 kernel/time/timer.c:1801
  rcu_gp_kthread+0x6b5/0x1940 kernel/rcu/tree.c:2231
  rcu_gp_kthread+0x6b5/0x1940 kernel/rcu/tree.c:2231
  kthread+0x345/0x410 kernel/kthread.c:240
  kthread+0x345/0x410 kernel/kthread.c:240
  ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412
  ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412
NMI backtrace for cpu 0
CPU: 0 PID: 8 Comm: ksoftirqd/0 Not tainted 4.17.0-rc6+ #86
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Call Trace:
  <IRQ>
  __dump_stack lib/dump_stack.c:77 [inline]
  dump_stack+0x1b9/0x294 lib/dump_stack.c:113
  nmi_cpu_backtrace.cold.4+0x19/0xce lib/nmi_backtrace.c:103
  nmi_trigger_cpumask_backtrace+0x151/0x192 lib/nmi_backtrace.c:62
  arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
  trigger_single_cpu_backtrace include/linux/nmi.h:156 [inline]
  rcu_dump_cpu_stacks+0x175/0x1c2 kernel/rcu/tree.c:1376
  print_cpu_stall kernel/rcu/tree.c:1525 [inline]
  check_cpu_stall.isra.61.cold.80+0x36c/0x59a kernel/rcu/tree.c:1593
  __rcu_pending kernel/rcu/tree.c:3356 [inline]
  rcu_pending kernel/rcu/tree.c:3401 [inline]
  rcu_check_callbacks+0x21b/0xad0 kernel/rcu/tree.c:2763
  update_process_times+0x2d/0x70 kernel/time/timer.c:1636
  tick_sched_handle+0x9f/0x180 kernel/time/tick-sched.c:164
  tick_sched_timer+0x45/0x130 kernel/time/tick-sched.c:1274
  __run_hrtimer kernel/time/hrtimer.c:1398 [inline]
  __hrtimer_run_queues+0x3e3/0x10a0 kernel/time/hrtimer.c:1460
  hrtimer_interrupt+0x2f3/0x750 kernel/time/hrtimer.c:1518
  local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1025 [inline]
  smp_apic_timer_interrupt+0x15d/0x710 arch/x86/kernel/apic/apic.c:1050
  apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:863
  </IRQ>
RIP: 0010:__sanitizer_cov_trace_pc+0x1/0x50 kernel/kcov.c:94
RSP: 0018:ffff8801d9aad680 EFLAGS: 00000293 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000103 RBX: 0000000000000002 RCX: ffffffff867e02e0
RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000005
RBP: ffff8801d9aad7e0 R08: ffff8801d9a9c200 R09: ffff8801d9aadaf0
R10: ffffed003b5c46c2 R11: ffff8801dae23613 R12: ffff8801ce597c40
R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000000
  find_match+0x244/0x13a0 net/ipv6/route.c:691
  find_rr_leaf net/ipv6/route.c:729 [inline]
  rt6_select net/ipv6/route.c:779 [inline]
  ip6_pol_route+0x946/0x3d40 net/ipv6/route.c:1705
  ip6_pol_route_output+0x54/0x70 net/ipv6/route.c:1969
  fib6_rule_lookup+0x211/0x6d0 net/ipv6/fib6_rules.c:89
  ip6_route_output_flags+0x2c5/0x350 net/ipv6/route.c:1997
  ip6_dst_lookup_tail+0x47b/0x1b30 net/ipv6/ip6_output.c:995
  ip6_dst_lookup_flow+0xc1/0x260 net/ipv6/ip6_output.c:1096
  sctp_v6_get_dst+0x16b4/0x20b0 net/sctp/ipv6.c:327
  sctp_transport_route+0xad/0x450 net/sctp/transport.c:293
  sctp_packet_config+0xb89/0xfd0 net/sctp/output.c:123
  sctp_outq_flush+0x79c/0x4370 net/sctp/outqueue.c:894
  sctp_outq_uncork+0x6a/0x80 net/sctp/outqueue.c:776
  sctp_cmd_interpreter net/sctp/sm_sideeffect.c:1820 [inline]
  sctp_side_effects net/sctp/sm_sideeffect.c:1220 [inline]
  sctp_do_sm+0x596/0x7160 net/sctp/sm_sideeffect.c:1191
  sctp_generate_heartbeat_event+0x218/0x450 net/sctp/sm_sideeffect.c:406
  call_timer_fn+0x230/0x940 kernel/time/timer.c:1326
  expire_timers kernel/time/timer.c:1363 [inline]
  __run_timers+0x79e/0xc50 kernel/time/timer.c:1666
  run_timer_softirq+0x4c/0x70 kernel/time/timer.c:1692
  __do_softirq+0x2e0/0xaf5 kernel/softirq.c:285
  run_ksoftirqd+0x86/0x100 kernel/softirq.c:646
  smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164
  kthread+0x345/0x410 kernel/kthread.c:240
  ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.929  
msecs
BUG: workqueue lockup - pool cpus=0-1 flags=0x4 nice=0 stuck for 127s!
Showing busy workqueues and worker pools:
workqueue events: flags=0x0
   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=10/256
     pending: pcpu_balance_workfn, defense_work_handler,  
defense_work_handler, defense_work_handler, defense_work_handler,  
defense_work_handler, defense_work_handler, check_corruption,  
vmstat_shepherd, cache_reap
workqueue events_power_efficient: flags=0x80
   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
     pending: check_lifetime
workqueue mm_percpu_wq: flags=0x8
   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
     pending: vmstat_update
workqueue writeback: flags=0x4e
   pwq 4: cpus=0-1 flags=0x4 nice=0 active=4/256
     pending: wb_workfn, wb_workfn, wb_workfn, wb_workfn
workqueue kblockd: flags=0x18
   pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=1/256
     pending: blk_mq_timeout_work
workqueue ib_addr: flags=0xa0002
   pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1
     pending: process_req
workqueue gid-cache-wq: flags=0xa0002
   pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1
     pending: update_gid_event_work_handler
     delayed: netdevice_event_work_handler
workqueue ipv6_addrconf: flags=0x40008
   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1
     pending: addrconf_dad_work
     delayed: addrconf_dad_work, addrconf_dad_work, addrconf_dad_work,  
addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work,  
addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work,  
addrconf_dad_work, addrconf_dad_work, addrconf_dad_work, addrconf_dad_work,  
addrconf_dad_work, addrconf_dad_work, addrconf_dad_work


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with  
syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: INFO: rcu detected stall in corrupted
  2018-05-21 18:05 INFO: rcu detected stall in corrupted syzbot
@ 2018-05-21 18:09 ` David Miller
  2018-05-21 18:13   ` Eric Dumazet
  0 siblings, 1 reply; 11+ messages in thread
From: David Miller @ 2018-05-21 18:09 UTC (permalink / raw)
  To: syzbot+f116bc1994efe725d51b
  Cc: kuznet, linux-kernel, netdev, syzkaller-bugs, yoshfuji, dsahern, roopa

From: syzbot <syzbot+f116bc1994efe725d51b@syzkaller.appspotmail.com>
Date: Mon, 21 May 2018 11:05:02 -0700

>  find_match+0x244/0x13a0 net/ipv6/route.c:691
>  find_rr_leaf net/ipv6/route.c:729 [inline]
>  rt6_select net/ipv6/route.c:779 [inline]

Hmmm, endless loop in find_rr_leaf or similar?

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: INFO: rcu detected stall in corrupted
  2018-05-21 18:09 ` David Miller
@ 2018-05-21 18:13   ` Eric Dumazet
  2018-05-21 18:15     ` David Ahern
  2018-05-23 23:13       ` Marcelo Ricardo Leitner
  0 siblings, 2 replies; 11+ messages in thread
From: Eric Dumazet @ 2018-05-21 18:13 UTC (permalink / raw)
  To: David Miller, syzbot+f116bc1994efe725d51b
  Cc: kuznet, linux-kernel, netdev, syzkaller-bugs, yoshfuji, dsahern, roopa



On 05/21/2018 11:09 AM, David Miller wrote:
> From: syzbot <syzbot+f116bc1994efe725d51b@syzkaller.appspotmail.com>
> Date: Mon, 21 May 2018 11:05:02 -0700
> 
>>  find_match+0x244/0x13a0 net/ipv6/route.c:691
>>  find_rr_leaf net/ipv6/route.c:729 [inline]
>>  rt6_select net/ipv6/route.c:779 [inline]
> 
> Hmmm, endless loop in find_rr_leaf or similar?
> 


I do not think so, this really looks like SCTP specific 
, we now have dozens of traces all sharing :

 sctp_transport_route+0xad/0x450 net/sctp/transport.c:293
 sctp_packet_config+0xb89/0xfd0 net/sctp/output.c:123
 sctp_outq_flush+0x79c/0x4370 net/sctp/outqueue.c:894
 sctp_outq_uncork+0x6a/0x80 net/sctp/outqueue.c:776
 sctp_cmd_interpreter net/sctp/sm_sideeffect.c:1820 [inline]
 sctp_side_effects net/sctp/sm_sideeffect.c:1220 [inline]
 sctp_do_sm+0x596/0x7160 net/sctp/sm_sideeffect.c:1191
 sctp_generate_heartbeat_event+0x218/0x450 net/sctp/sm_sideeffect.c:406
 call_timer_fn+0x230/0x940 kernel/time/timer.c:1326 


Some kind of infinite loop.

When the hrtimer fires, it can point to any code that sits below but does not necessarily have a bug.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: INFO: rcu detected stall in corrupted
  2018-05-21 18:13   ` Eric Dumazet
@ 2018-05-21 18:15     ` David Ahern
  2018-05-23 23:13       ` Marcelo Ricardo Leitner
  1 sibling, 0 replies; 11+ messages in thread
From: David Ahern @ 2018-05-21 18:15 UTC (permalink / raw)
  To: Eric Dumazet, David Miller, syzbot+f116bc1994efe725d51b
  Cc: kuznet, linux-kernel, netdev, syzkaller-bugs, yoshfuji, dsahern, roopa

On 5/21/18 12:13 PM, Eric Dumazet wrote:
> 
> 
> On 05/21/2018 11:09 AM, David Miller wrote:
>> From: syzbot <syzbot+f116bc1994efe725d51b@syzkaller.appspotmail.com>
>> Date: Mon, 21 May 2018 11:05:02 -0700
>>
>>>  find_match+0x244/0x13a0 net/ipv6/route.c:691
>>>  find_rr_leaf net/ipv6/route.c:729 [inline]
>>>  rt6_select net/ipv6/route.c:779 [inline]
>>
>> Hmmm, endless loop in find_rr_leaf or similar?
>>
> 
> 
> I do not think so, this really looks like SCTP specific 
> , we now have dozens of traces all sharing :
> 
>  sctp_transport_route+0xad/0x450 net/sctp/transport.c:293
>  sctp_packet_config+0xb89/0xfd0 net/sctp/output.c:123
>  sctp_outq_flush+0x79c/0x4370 net/sctp/outqueue.c:894
>  sctp_outq_uncork+0x6a/0x80 net/sctp/outqueue.c:776
>  sctp_cmd_interpreter net/sctp/sm_sideeffect.c:1820 [inline]
>  sctp_side_effects net/sctp/sm_sideeffect.c:1220 [inline]
>  sctp_do_sm+0x596/0x7160 net/sctp/sm_sideeffect.c:1191
>  sctp_generate_heartbeat_event+0x218/0x450 net/sctp/sm_sideeffect.c:406
>  call_timer_fn+0x230/0x940 kernel/time/timer.c:1326 
> 
> 
> Some kind of infinite loop.
> 
> When the hrtimer fires, it can point to any code that sits below but does not necessarily have a bug.
> 

Just so we are looking at the right tree, the original message shows:

> HEAD commit:    771c577c23ba Linux 4.17-rc6
> git tree:       upstream

I take that to mean this is Linus' tree.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: INFO: rcu detected stall in corrupted
  2018-05-21 18:13   ` Eric Dumazet
@ 2018-05-23 23:13       ` Marcelo Ricardo Leitner
  2018-05-23 23:13       ` Marcelo Ricardo Leitner
  1 sibling, 0 replies; 11+ messages in thread
From: Marcelo Ricardo Leitner @ 2018-05-23 23:13 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: David Miller, syzbot+f116bc1994efe725d51b, kuznet, linux-kernel,
	netdev, syzkaller-bugs, yoshfuji, dsahern, roopa, linux-sctp,
	Xin Long

On Mon, May 21, 2018 at 11:13:46AM -0700, Eric Dumazet wrote:
> 
> 
> On 05/21/2018 11:09 AM, David Miller wrote:
> > From: syzbot <syzbot+f116bc1994efe725d51b@syzkaller.appspotmail.com>
> > Date: Mon, 21 May 2018 11:05:02 -0700
> > 
> >>  find_match+0x244/0x13a0 net/ipv6/route.c:691
> >>  find_rr_leaf net/ipv6/route.c:729 [inline]
> >>  rt6_select net/ipv6/route.c:779 [inline]
> > 
> > Hmmm, endless loop in find_rr_leaf or similar?
> > 
> 
> 
> I do not think so, this really looks like SCTP specific 
> , we now have dozens of traces all sharing :
> 
>  sctp_transport_route+0xad/0x450 net/sctp/transport.c:293
>  sctp_packet_config+0xb89/0xfd0 net/sctp/output.c:123
>  sctp_outq_flush+0x79c/0x4370 net/sctp/outqueue.c:894
>  sctp_outq_uncork+0x6a/0x80 net/sctp/outqueue.c:776
>  sctp_cmd_interpreter net/sctp/sm_sideeffect.c:1820 [inline]
>  sctp_side_effects net/sctp/sm_sideeffect.c:1220 [inline]
>  sctp_do_sm+0x596/0x7160 net/sctp/sm_sideeffect.c:1191
>  sctp_generate_heartbeat_event+0x218/0x450 net/sctp/sm_sideeffect.c:406
>  call_timer_fn+0x230/0x940 kernel/time/timer.c:1326 
> 
> 
> Some kind of infinite loop.
> 
> When the hrtimer fires, it can point to any code that sits below but does not necessarily have a bug.

Agreed. Xin Long identified the root cause. syzkaller is setting too
aggressive parameters to SCTP RTO, leading to issues with the
heartbeat timer.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: INFO: rcu detected stall in corrupted
@ 2018-05-23 23:13       ` Marcelo Ricardo Leitner
  0 siblings, 0 replies; 11+ messages in thread
From: Marcelo Ricardo Leitner @ 2018-05-23 23:13 UTC (permalink / raw)
  To: Eric Dumazet
  Cc: David Miller, syzbot+f116bc1994efe725d51b, kuznet, linux-kernel,
	netdev, syzkaller-bugs, yoshfuji, dsahern, roopa, linux-sctp,
	Xin Long

On Mon, May 21, 2018 at 11:13:46AM -0700, Eric Dumazet wrote:
> 
> 
> On 05/21/2018 11:09 AM, David Miller wrote:
> > From: syzbot <syzbot+f116bc1994efe725d51b@syzkaller.appspotmail.com>
> > Date: Mon, 21 May 2018 11:05:02 -0700
> > 
> >>  find_match+0x244/0x13a0 net/ipv6/route.c:691
> >>  find_rr_leaf net/ipv6/route.c:729 [inline]
> >>  rt6_select net/ipv6/route.c:779 [inline]
> > 
> > Hmmm, endless loop in find_rr_leaf or similar?
> > 
> 
> 
> I do not think so, this really looks like SCTP specific 
> , we now have dozens of traces all sharing :
> 
>  sctp_transport_route+0xad/0x450 net/sctp/transport.c:293
>  sctp_packet_config+0xb89/0xfd0 net/sctp/output.c:123
>  sctp_outq_flush+0x79c/0x4370 net/sctp/outqueue.c:894
>  sctp_outq_uncork+0x6a/0x80 net/sctp/outqueue.c:776
>  sctp_cmd_interpreter net/sctp/sm_sideeffect.c:1820 [inline]
>  sctp_side_effects net/sctp/sm_sideeffect.c:1220 [inline]
>  sctp_do_sm+0x596/0x7160 net/sctp/sm_sideeffect.c:1191
>  sctp_generate_heartbeat_event+0x218/0x450 net/sctp/sm_sideeffect.c:406
>  call_timer_fn+0x230/0x940 kernel/time/timer.c:1326 
> 
> 
> Some kind of infinite loop.
> 
> When the hrtimer fires, it can point to any code that sits below but does not necessarily have a bug.

Agreed. Xin Long identified the root cause. syzkaller is setting too
aggressive parameters to SCTP RTO, leading to issues with the
heartbeat timer.


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: INFO: rcu detected stall in corrupted
  2018-05-23 23:13       ` Marcelo Ricardo Leitner
@ 2018-05-24  9:02         ` Xin Long
  -1 siblings, 0 replies; 11+ messages in thread
From: Xin Long @ 2018-05-24  9:02 UTC (permalink / raw)
  To: Marcelo Ricardo Leitner
  Cc: Eric Dumazet, David Miller, syzbot+f116bc1994efe725d51b, kuznet,
	LKML, network dev, syzkaller-bugs, yoshfuji, dsahern,
	Roopa Prabhu, linux-sctp

On Thu, May 24, 2018 at 7:13 AM, Marcelo Ricardo Leitner
<marcelo.leitner@gmail.com> wrote:
> On Mon, May 21, 2018 at 11:13:46AM -0700, Eric Dumazet wrote:
>>
>>
>> On 05/21/2018 11:09 AM, David Miller wrote:
>> > From: syzbot <syzbot+f116bc1994efe725d51b@syzkaller.appspotmail.com>
>> > Date: Mon, 21 May 2018 11:05:02 -0700
>> >
>> >>  find_match+0x244/0x13a0 net/ipv6/route.c:691
>> >>  find_rr_leaf net/ipv6/route.c:729 [inline]
>> >>  rt6_select net/ipv6/route.c:779 [inline]
>> >
>> > Hmmm, endless loop in find_rr_leaf or similar?
>> >
>>
>>
>> I do not think so, this really looks like SCTP specific
>> , we now have dozens of traces all sharing :
>>
>>  sctp_transport_route+0xad/0x450 net/sctp/transport.c:293
>>  sctp_packet_config+0xb89/0xfd0 net/sctp/output.c:123
>>  sctp_outq_flush+0x79c/0x4370 net/sctp/outqueue.c:894
>>  sctp_outq_uncork+0x6a/0x80 net/sctp/outqueue.c:776
>>  sctp_cmd_interpreter net/sctp/sm_sideeffect.c:1820 [inline]
>>  sctp_side_effects net/sctp/sm_sideeffect.c:1220 [inline]
>>  sctp_do_sm+0x596/0x7160 net/sctp/sm_sideeffect.c:1191
>>  sctp_generate_heartbeat_event+0x218/0x450 net/sctp/sm_sideeffect.c:406
>>  call_timer_fn+0x230/0x940 kernel/time/timer.c:1326
>>
>>
>> Some kind of infinite loop.
>>
>> When the hrtimer fires, it can point to any code that sits below but does not necessarily have a bug.
>
> Agreed. Xin Long identified the root cause. syzkaller is setting too
> aggressive parameters to SCTP RTO, leading to issues with the
> heartbeat timer.
Right, I will prepare a fix soon with your suggestion rto_min value "HZ/5"
Thanks.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: INFO: rcu detected stall in corrupted
@ 2018-05-24  9:02         ` Xin Long
  0 siblings, 0 replies; 11+ messages in thread
From: Xin Long @ 2018-05-24  9:02 UTC (permalink / raw)
  To: Marcelo Ricardo Leitner
  Cc: Eric Dumazet, David Miller, syzbot+f116bc1994efe725d51b, kuznet,
	LKML, network dev, syzkaller-bugs, yoshfuji, dsahern,
	Roopa Prabhu, linux-sctp

On Thu, May 24, 2018 at 7:13 AM, Marcelo Ricardo Leitner
<marcelo.leitner@gmail.com> wrote:
> On Mon, May 21, 2018 at 11:13:46AM -0700, Eric Dumazet wrote:
>>
>>
>> On 05/21/2018 11:09 AM, David Miller wrote:
>> > From: syzbot <syzbot+f116bc1994efe725d51b@syzkaller.appspotmail.com>
>> > Date: Mon, 21 May 2018 11:05:02 -0700
>> >
>> >>  find_match+0x244/0x13a0 net/ipv6/route.c:691
>> >>  find_rr_leaf net/ipv6/route.c:729 [inline]
>> >>  rt6_select net/ipv6/route.c:779 [inline]
>> >
>> > Hmmm, endless loop in find_rr_leaf or similar?
>> >
>>
>>
>> I do not think so, this really looks like SCTP specific
>> , we now have dozens of traces all sharing :
>>
>>  sctp_transport_route+0xad/0x450 net/sctp/transport.c:293
>>  sctp_packet_config+0xb89/0xfd0 net/sctp/output.c:123
>>  sctp_outq_flush+0x79c/0x4370 net/sctp/outqueue.c:894
>>  sctp_outq_uncork+0x6a/0x80 net/sctp/outqueue.c:776
>>  sctp_cmd_interpreter net/sctp/sm_sideeffect.c:1820 [inline]
>>  sctp_side_effects net/sctp/sm_sideeffect.c:1220 [inline]
>>  sctp_do_sm+0x596/0x7160 net/sctp/sm_sideeffect.c:1191
>>  sctp_generate_heartbeat_event+0x218/0x450 net/sctp/sm_sideeffect.c:406
>>  call_timer_fn+0x230/0x940 kernel/time/timer.c:1326
>>
>>
>> Some kind of infinite loop.
>>
>> When the hrtimer fires, it can point to any code that sits below but does not necessarily have a bug.
>
> Agreed. Xin Long identified the root cause. syzkaller is setting too
> aggressive parameters to SCTP RTO, leading to issues with the
> heartbeat timer.
Right, I will prepare a fix soon with your suggestion rto_min value "HZ/5"
Thanks.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: INFO: rcu detected stall in corrupted
  2018-05-24  9:02         ` Xin Long
@ 2018-05-26 15:28           ` Dmitry Vyukov
  -1 siblings, 0 replies; 11+ messages in thread
From: Dmitry Vyukov @ 2018-05-26 15:28 UTC (permalink / raw)
  To: Xin Long
  Cc: Marcelo Ricardo Leitner, Eric Dumazet, David Miller,
	syzbot+f116bc1994efe725d51b, kuznet, LKML, network dev,
	syzkaller-bugs, yoshfuji, dsahern, Roopa Prabhu, linux-sctp

On Thu, May 24, 2018 at 11:02 AM, Xin Long <lucien.xin@gmail.com> wrote:
> On Thu, May 24, 2018 at 7:13 AM, Marcelo Ricardo Leitner
> <marcelo.leitner@gmail.com> wrote:
>> On Mon, May 21, 2018 at 11:13:46AM -0700, Eric Dumazet wrote:
>>>
>>>
>>> On 05/21/2018 11:09 AM, David Miller wrote:
>>> > From: syzbot <syzbot+f116bc1994efe725d51b@syzkaller.appspotmail.com>
>>> > Date: Mon, 21 May 2018 11:05:02 -0700
>>> >
>>> >>  find_match+0x244/0x13a0 net/ipv6/route.c:691
>>> >>  find_rr_leaf net/ipv6/route.c:729 [inline]
>>> >>  rt6_select net/ipv6/route.c:779 [inline]
>>> >
>>> > Hmmm, endless loop in find_rr_leaf or similar?
>>> >
>>>
>>>
>>> I do not think so, this really looks like SCTP specific
>>> , we now have dozens of traces all sharing :
>>>
>>>  sctp_transport_route+0xad/0x450 net/sctp/transport.c:293
>>>  sctp_packet_config+0xb89/0xfd0 net/sctp/output.c:123
>>>  sctp_outq_flush+0x79c/0x4370 net/sctp/outqueue.c:894
>>>  sctp_outq_uncork+0x6a/0x80 net/sctp/outqueue.c:776
>>>  sctp_cmd_interpreter net/sctp/sm_sideeffect.c:1820 [inline]
>>>  sctp_side_effects net/sctp/sm_sideeffect.c:1220 [inline]
>>>  sctp_do_sm+0x596/0x7160 net/sctp/sm_sideeffect.c:1191
>>>  sctp_generate_heartbeat_event+0x218/0x450 net/sctp/sm_sideeffect.c:406
>>>  call_timer_fn+0x230/0x940 kernel/time/timer.c:1326
>>>
>>>
>>> Some kind of infinite loop.
>>>
>>> When the hrtimer fires, it can point to any code that sits below but does not necessarily have a bug.
>>
>> Agreed. Xin Long identified the root cause. syzkaller is setting too
>> aggressive parameters to SCTP RTO, leading to issues with the
>> heartbeat timer.
> Right, I will prepare a fix soon with your suggestion rto_min value "HZ/5"
> Thanks.

#syz fix: sctp: not allow to set rto_min with a value below 200 msecs

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: INFO: rcu detected stall in corrupted
@ 2018-05-26 15:28           ` Dmitry Vyukov
  0 siblings, 0 replies; 11+ messages in thread
From: Dmitry Vyukov @ 2018-05-26 15:28 UTC (permalink / raw)
  To: Xin Long
  Cc: Marcelo Ricardo Leitner, Eric Dumazet, David Miller,
	syzbot+f116bc1994efe725d51b, kuznet, LKML, network dev,
	syzkaller-bugs, yoshfuji, dsahern, Roopa Prabhu, linux-sctp

On Thu, May 24, 2018 at 11:02 AM, Xin Long <lucien.xin@gmail.com> wrote:
> On Thu, May 24, 2018 at 7:13 AM, Marcelo Ricardo Leitner
> <marcelo.leitner@gmail.com> wrote:
>> On Mon, May 21, 2018 at 11:13:46AM -0700, Eric Dumazet wrote:
>>>
>>>
>>> On 05/21/2018 11:09 AM, David Miller wrote:
>>> > From: syzbot <syzbot+f116bc1994efe725d51b@syzkaller.appspotmail.com>
>>> > Date: Mon, 21 May 2018 11:05:02 -0700
>>> >
>>> >>  find_match+0x244/0x13a0 net/ipv6/route.c:691
>>> >>  find_rr_leaf net/ipv6/route.c:729 [inline]
>>> >>  rt6_select net/ipv6/route.c:779 [inline]
>>> >
>>> > Hmmm, endless loop in find_rr_leaf or similar?
>>> >
>>>
>>>
>>> I do not think so, this really looks like SCTP specific
>>> , we now have dozens of traces all sharing :
>>>
>>>  sctp_transport_route+0xad/0x450 net/sctp/transport.c:293
>>>  sctp_packet_config+0xb89/0xfd0 net/sctp/output.c:123
>>>  sctp_outq_flush+0x79c/0x4370 net/sctp/outqueue.c:894
>>>  sctp_outq_uncork+0x6a/0x80 net/sctp/outqueue.c:776
>>>  sctp_cmd_interpreter net/sctp/sm_sideeffect.c:1820 [inline]
>>>  sctp_side_effects net/sctp/sm_sideeffect.c:1220 [inline]
>>>  sctp_do_sm+0x596/0x7160 net/sctp/sm_sideeffect.c:1191
>>>  sctp_generate_heartbeat_event+0x218/0x450 net/sctp/sm_sideeffect.c:406
>>>  call_timer_fn+0x230/0x940 kernel/time/timer.c:1326
>>>
>>>
>>> Some kind of infinite loop.
>>>
>>> When the hrtimer fires, it can point to any code that sits below but does not necessarily have a bug.
>>
>> Agreed. Xin Long identified the root cause. syzkaller is setting too
>> aggressive parameters to SCTP RTO, leading to issues with the
>> heartbeat timer.
> Right, I will prepare a fix soon with your suggestion rto_min value "HZ/5"
> Thanks.

#syz fix: sctp: not allow to set rto_min with a value below 200 msecs

^ permalink raw reply	[flat|nested] 11+ messages in thread

* INFO: rcu detected stall in corrupted
@ 2024-03-16  3:25 cheung wall
  0 siblings, 0 replies; 11+ messages in thread
From: cheung wall @ 2024-03-16  3:25 UTC (permalink / raw)
  To: Bjorn Helgaas; +Cc: linux-pci, linux-kernel

Hello,


when using Healer to fuzz the latest Linux Kernel, the following crash

was triggered on:


HEAD commit: 0dd3ee31125508cd67f7e7172247f05b7fd1753a  (tag: v6.7)

git tree: upstream

console output: https://pastebin.com/raw/1JtKSypv

kernel config: https://pastebin.com/raw/VecrLrRN

C reproducer: https://pastebin.com/raw/CUsatTHW

Syzlang reproducer: https://pastebin.com/raw/gJFKLvkN


If you fix this issue, please add the following tag to the commit:

Reported-by: Qiang Zhang <zzqq0103.hey@gmail.com>

----------------------------------------------------------

rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: 1-...0: (0 ticks this GP) idle=84e4/1/0x4000000000000000
softirq=698253/698254 fqs=4779
rcu: (detected by 3, t=21002 jiffies, g=1119041, q=13818 ncpus=8)
Sending NMI from CPU 3 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 510326 Comm: syz-executor.1 Not tainted 6.7.0 #2
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
RIP: 0010:vga_put+0x42/0x100 drivers/pci/vgaarb.c:547
Code: 85 e4 0f 84 aa 00 00 00 e8 8b 59 38 ff 48 c7 c7 00 31 71 b9 48
bb 00 00 00 00 00 fc ff df e8 75 c0 d0 01 48 8b 2d fe a1 f9 02 <49> 89
c6 48 81 fd 00 ba 1f b8 75 25 eb 59 e8 5b 59 38 ff 48 89 e8
RSP: 0018:ffff888117917e28 EFLAGS: 00000082
RAX: 0000000000000093 RBX: dffffc0000000000 RCX: ffffffffb6f6d916
RDX: ffff8881138ea200 RSI: 0000000000000004 RDI: ffff888117917dc8
RBP: ffff888100fd5000 R08: 0000000000000001 R09: ffffed1022f22fb9
R10: 0000000000000003 R11: 0000000000000000 R12: ffff888100ee8000
R13: 0000000000000001 R14: ffff888116747818 R15: dffffc0000000000
FS: 0000555556ccd480(0000) GS:ffff8881f7080000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7b9128aa4f CR3: 0000000114c00003 CR4: 0000000000770ef0
PKRU: 55555554
Call Trace:
<NMI>
</NMI>
<TASK>
vga_arb_release+0x19a/0x2e0 drivers/pci/vgaarb.c:1455
__fput+0x235/0xb20 fs/file_table.c:394
__fput_sync+0x35/0x40 fs/file_table.c:475
__do_sys_close fs/open.c:1587 [inline]
__se_sys_close fs/open.c:1572 [inline]
__x64_sys_close+0x87/0xf0 fs/open.c:1572
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0x46/0xf0 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x6f/0x77
RIP: 0033:0x7f7b91217f0b
Code: 03 00 00 00 0f 05 48 3d 00 f0 ff ff 77 41 c3 48 83 ec 18 89 7c
24 0c e8 f3 fb 02 00 8b 7c 24 0c 41 89 c0 b8 03 00 00 00 0f 05 <48> 3d
00 f0 ff ff 77 35 44 89 c7 89 44 24 0c e8 41 fc 02 00 8b 44
RSP: 002b:00007fffbef11d10 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f7b91217f0b
RDX: 0000000000000000 RSI: ffffffffffffffff RDI: 0000000000000003
RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f7b90d89bd8
R10: 00007fffbef11e10 R11: 0000000000000293 R12: 00007f7b90d89000
R13: 00007fffbef11e10 R14: 00007f7b90d89c80 R15: 00007f7b90d89c78
</TASK>

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2024-03-16  3:26 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-05-21 18:05 INFO: rcu detected stall in corrupted syzbot
2018-05-21 18:09 ` David Miller
2018-05-21 18:13   ` Eric Dumazet
2018-05-21 18:15     ` David Ahern
2018-05-23 23:13     ` Marcelo Ricardo Leitner
2018-05-23 23:13       ` Marcelo Ricardo Leitner
2018-05-24  9:02       ` Xin Long
2018-05-24  9:02         ` Xin Long
2018-05-26 15:28         ` Dmitry Vyukov
2018-05-26 15:28           ` Dmitry Vyukov
2024-03-16  3:25 cheung wall

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.