From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Sasha Levin <sasha.levin@oracle.com>
Cc: Dave Jones <davej@redhat.com>,
Linux Kernel <linux-kernel@vger.kernel.org>,
htejun@gmail.com
Subject: Re: rcu_preempt detected stalls.
Date: Fri, 24 Oct 2014 09:13:37 -0700 [thread overview]
Message-ID: <20141024161337.GQ4977@linux.vnet.ibm.com> (raw)
In-Reply-To: <544A45F8.2030207@oracle.com>
On Fri, Oct 24, 2014 at 08:28:40AM -0400, Sasha Levin wrote:
> On 10/23/2014 03:58 PM, Paul E. McKenney wrote:
> > On Thu, Oct 23, 2014 at 02:55:43PM -0400, Sasha Levin wrote:
> >> > On 10/23/2014 02:39 PM, Paul E. McKenney wrote:
> >>> > > On Tue, Oct 14, 2014 at 10:35:10PM -0400, Sasha Levin wrote:
> >>>> > >> On 10/13/2014 01:35 PM, Dave Jones wrote:
> >>>>> > >>> oday in "rcu stall while fuzzing" news:
> >>>>> > >>>
> >>>>> > >>> INFO: rcu_preempt detected stalls on CPUs/tasks:
> >>>>> > >>> Tasks blocked on level-0 rcu_node (CPUs 0-3): P766 P646
> >>>>> > >>> Tasks blocked on level-0 rcu_node (CPUs 0-3): P766 P646
> >>>>> > >>> (detected by 0, t=6502 jiffies, g=75434, c=75433, q=0)
> >>>> > >>
> >>>> > >> I've complained about RCU stalls couple days ago (in a different context)
> >>>> > >> on -next. I guess whatever causing them made it into Linus's tree?
> >>>> > >>
> >>>> > >> https://lkml.org/lkml/2014/10/11/64
> >>> > >
> >>> > > And on that one, I must confess that I don't see where the RCU read-side
> >>> > > critical section might be.
> >>> > >
> >>> > > Hmmm... Maybe someone forgot to put an rcu_read_unlock() somewhere.
> >>> > > Can you reproduce this with CONFIG_PROVE_RCU=y?
> >> >
> >> > Paul, if that was directed to me - Yes, I see stalls with CONFIG_PROVE_RCU
> >> > set and nothing else is showing up before/after that.
> > Indeed it was directed to you. ;-)
> >
> > Does the following crude diagnostic patch turn up anything?
>
> Nope, seeing stalls but not seeing that pr_err() you added.
OK, color me confused. Could you please send me the full dmesg or a
pointer to it?
Thanx, Paul
> [ 5107.395916] INFO: rcu_preempt detected stalls on CPUs/tasks:
> [ 5107.395916] 0: (776 ticks this GP) idle=a8d/140000000000002/0 softirq=16356/16356 last_accelerate: f5b7/55e5, nonlazy_posted: 24252, ..
> [ 5107.395916] (detected by 1, t=20502 jiffies, g=13949, c=13948, q=0)
> [ 5107.395916] Task dump for CPU 0:
> [ 5107.395916] trinity-c0 R running task 12848 20357 9041 0x0008000e
> [ 5107.395916] 0000000000000000 ffff88006bfd76c0 ffff88065722b988 ffffffffa10af964
> [ 5107.395916] ffff88065722b998 ffffffffa106ad23 ffff88065722b9c8 ffffffffa119dce5
> [ 5107.395916] 00000000001d76c0 ffff88006bfd76c0 00000000001d76c0 ffff8806473cbd10
> [ 5107.395916] Call Trace:
> [ 5107.395916] [<ffffffffa10af964>] ? kvm_clock_read+0x24/0x40
> [ 5107.395916] [<ffffffffa106ad23>] ? sched_clock+0x13/0x30
> [ 5107.395916] [<ffffffffa119dce5>] ? sched_clock_local+0x25/0x90
> [ 5107.395916] [<ffffffffa1303dfb>] ? __slab_free+0xbb/0x3a0
> [ 5107.395916] [<ffffffffa1b71167>] ? debug_smp_processor_id+0x17/0x20
> [ 5107.395916] [<ffffffffa451cb64>] ? _raw_spin_unlock_irqrestore+0x64/0xa0
> [ 5107.395916] [<ffffffffa1303dfb>] ? __slab_free+0xbb/0x3a0
> [ 5107.395916] [<ffffffffa1b71bce>] ? __debug_check_no_obj_freed+0x10e/0x210
> [ 5107.395916] [<ffffffffa1305871>] ? kmem_cache_free+0xb1/0x4f0
> [ 5107.395916] [<ffffffffa1305883>] ? kmem_cache_free+0xc3/0x4f0
> [ 5107.395916] [<ffffffffa1305bb2>] ? kmem_cache_free+0x3f2/0x4f0
> [ 5107.395916] [<ffffffffa12e0cbe>] ? unlink_anon_vmas+0x10e/0x180
> [ 5107.395916] [<ffffffffa12e0cbe>] ? unlink_anon_vmas+0x10e/0x180
> [ 5107.395916] [<ffffffffa12cfbdf>] ? free_pgtables+0x3f/0x130
> [ 5107.395916] [<ffffffffa12dc1a4>] ? exit_mmap+0xc4/0x180
> [ 5107.395916] [<ffffffffa13143fe>] ? __khugepaged_exit+0xbe/0x120
> [ 5107.395916] [<ffffffffa115bbb3>] ? mmput+0x73/0x110
> [ 5107.395916] [<ffffffffa1162eb7>] ? do_exit+0x2c7/0xd30
> [ 5107.395916] [<ffffffffa1173fb9>] ? get_signal+0x3c9/0xaf0
> [ 5107.395916] [<ffffffffa1b71167>] ? debug_smp_processor_id+0x17/0x20
> [ 5107.395916] [<ffffffffa11bccbe>] ? put_lock_stats.isra.13+0xe/0x30
> [ 5107.395916] [<ffffffffa451c810>] ? _raw_spin_unlock_irq+0x30/0x70
> [ 5107.395916] [<ffffffffa11639c2>] ? do_group_exit+0x52/0xe0
> [ 5107.395916] [<ffffffffa1173ef6>] ? get_signal+0x306/0xaf0
> [ 5107.395916] [<ffffffffa119dce5>] ? sched_clock_local+0x25/0x90
> [ 5107.395916] [<ffffffffa105f2f0>] ? do_signal+0x20/0x130
> [ 5107.395916] [<ffffffffa1298558>] ? context_tracking_user_exit+0x78/0x2d0
> [ 5107.395916] [<ffffffffa1b71183>] ? __this_cpu_preempt_check+0x13/0x20
> [ 5107.395916] [<ffffffffa11c04cb>] ? trace_hardirqs_on_caller+0xfb/0x280
> [ 5107.395916] [<ffffffffa11c065d>] ? trace_hardirqs_on+0xd/0x10
> [ 5107.395916] [<ffffffffa105f469>] ? do_notify_resume+0x69/0xb0
> [ 5107.395916] [<ffffffffa451d74f>] ? int_signal+0x12/0x17
>
>
> Thanks,
> Sasha
>
next prev parent reply other threads:[~2014-10-24 16:17 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-10-13 17:35 rcu_preempt detected stalls Dave Jones
2014-10-15 2:35 ` Sasha Levin
2014-10-23 18:39 ` Paul E. McKenney
2014-10-23 18:55 ` Sasha Levin
2014-10-23 19:58 ` Paul E. McKenney
2014-10-24 12:28 ` Sasha Levin
2014-10-24 16:13 ` Paul E. McKenney [this message]
2014-10-24 16:39 ` Sasha Levin
2014-10-27 21:13 ` Paul E. McKenney
2014-10-27 23:44 ` Paul E. McKenney
2014-10-27 23:44 ` Paul E. McKenney
2014-11-13 23:07 ` Paul E. McKenney
2014-11-13 23:07 ` Paul E. McKenney
2014-11-13 23:10 ` Sasha Levin
2014-11-13 23:10 ` Sasha Levin
2014-10-30 23:41 ` Sasha Levin
2014-10-23 18:32 ` Paul E. McKenney
2014-10-23 18:40 ` Dave Jones
2014-10-23 19:28 ` Paul E. McKenney
2014-10-23 19:37 ` Dave Jones
2014-10-23 19:52 ` Paul E. McKenney
2014-10-23 20:28 ` Dave Jones
2014-10-23 20:44 ` Paul E. McKenney
2014-10-23 19:13 ` Oleg Nesterov
2014-10-23 19:38 ` Paul E. McKenney
2014-10-23 19:53 ` Oleg Nesterov
2014-10-23 20:24 ` Paul E. McKenney
2014-10-23 21:13 ` Oleg Nesterov
2014-10-23 21:38 ` Paul E. McKenney
2014-10-25 3:16 ` Dâniel Fraga
2021-08-31 15:21 Jorge Ramirez-Ortiz, Foundries
2021-08-31 15:21 ` Jorge Ramirez-Ortiz, Foundries
2021-08-31 15:53 ` Paul E. McKenney
2021-08-31 15:53 ` Paul E. McKenney
2021-08-31 17:01 ` Zhouyi Zhou
2021-08-31 17:01 ` Zhouyi Zhou
2021-08-31 17:01 ` Zhouyi Zhou
2021-08-31 17:11 ` Zhouyi Zhou
2021-08-31 17:11 ` Zhouyi Zhou
2021-08-31 17:11 ` Zhouyi Zhou
2021-09-01 1:03 ` Zhouyi Zhou
2021-09-01 1:03 ` Zhouyi Zhou
2021-09-01 1:03 ` Zhouyi Zhou
2021-09-01 4:08 ` Neeraj Upadhyay
2021-09-01 6:47 ` Zhouyi Zhou
2021-09-01 6:47 ` Zhouyi Zhou
2021-09-01 6:47 ` Zhouyi Zhou
2021-09-01 8:23 ` Jorge Ramirez-Ortiz, Foundries
2021-09-01 8:23 ` Jorge Ramirez-Ortiz, Foundries
2021-09-01 9:17 ` Zhouyi Zhou
2021-09-01 9:17 ` Zhouyi Zhou
2021-09-01 9:17 ` Zhouyi Zhou
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20141024161337.GQ4977@linux.vnet.ibm.com \
--to=paulmck@linux.vnet.ibm.com \
--cc=davej@redhat.com \
--cc=htejun@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=sasha.levin@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.