From: Frederic Weisbecker <fweisbec@gmail.com>
To: Sasha Levin <sasha.levin@oracle.com>
Cc: linux-kernel@vger.kernel.org, mingo@kernel.org,
tglx@linutronix.de, hpa@zytor.com, torvalds@linux-foundation.org,
peterz@infradead.org,
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Subject: Re: [tip:sched/core] sched/preempt: Optimize preemption operations on __schedule() callers
Date: Tue, 21 Jul 2015 00:48:59 +0200 [thread overview]
Message-ID: <20150720224858.GD13032@lerouge> (raw)
In-Reply-To: <55AD72C3.4000300@oracle.com>
On Mon, Jul 20, 2015 at 06:14:27PM -0400, Sasha Levin wrote:
> On 05/19/2015 03:17 AM, tip-bot for Frederic Weisbecker wrote:
> > Commit-ID: b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
> > Gitweb: http://git.kernel.org/tip/b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
> > Author: Frederic Weisbecker <fweisbec@gmail.com>
> > AuthorDate: Tue, 12 May 2015 16:41:49 +0200
> > Committer: Ingo Molnar <mingo@kernel.org>
> > CommitDate: Tue, 19 May 2015 08:39:12 +0200
> >
> > sched/preempt: Optimize preemption operations on __schedule() callers
> >
> > __schedule() disables preemption and some of its callers
> > (the preempt_schedule*() family) also set PREEMPT_ACTIVE.
> >
> > So we have two preempt_count() modifications that could be performed
> > at once.
> >
> > Lets remove the preemption disablement from __schedule() and pull
> > this responsibility to its callers in order to optimize preempt_count()
> > operations in a single place.
> >
> > Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
> > Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Link: http://lkml.kernel.org/r/1431441711-29753-5-git-send-email-fweisbec@gmail.com
> > Signed-off-by: Ingo Molnar <mingo@kernel.org>
>
> Hi all,
>
> I was seeing RCU stall warnings that appeared to have different backtrace
> each time, but would reliably trigger when fuzzing and looked something like this:
>
> [4394886.414687] INFO: rcu_preempt detected stalls on CPUs/tasks:
> [4394886.416717] Tasks blocked on level-0 rcu_node (CPUs 0-23): P13356
> [4394886.418699] (detected by 2, t=20502 jiffies, g=3548, c=3547, q=120)
> [4394886.420712] trinity-c42 R running task 26936 13356 9574 0x10000000
> [4394886.422466] ffff8807d1c7fbe8 ffff8807d1c7fb88 0000000000000022 ffff8801081e1bb0
> [4394886.423711] ffff8801081e1b88 ffff8807d1eb8780 ffff8801081e11d8 ffff8807e2564000
> [4394886.424956] ffff8807d1eb8000 ffff8807d1c7fbd8 ffff8807d1c78000 0000000000000000
> [4394886.426196] Call Trace:
> [4394886.426627] preempt_schedule_irq (./arch/x86/include/asm/paravirt.h:807 kernel/sched/core.c:3218)
> [4394886.427582] ? lockdep_reset_lock (kernel/locking/lockdep.c:3105)
> [4394886.428548] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
> [4394886.429438] retint_kernel (arch/x86/entry/entry_64.S:578)
> [4394886.430286] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
> [4394886.431176] ? native_restore_fl (./arch/x86/include/asm/irqflags.h:35)
> [4394886.432376] lock_is_held (kernel/locking/lockdep.c:3661)
> [4394886.433241] ? kill_pid_info (include/linux/rcupdate.h:914 kernel/signal.c:1344)
> [4394886.434135] rcu_read_lock_held (kernel/rcu/update.c:275)
> [4394886.435012] pid_task (kernel/pid.c:440 (discriminator 5))
> [4394886.435656] kill_pid_info (kernel/signal.c:1341)
> [4394886.436351] ? kill_pid_info (include/linux/rcupdate.h:857 kernel/signal.c:1340)
> [4394886.437065] SYSC_kill (kernel/signal.c:1426 kernel/signal.c:2903)
> [4394886.437726] ? SYSC_kill (include/linux/rcupdate.h:857 kernel/signal.c:1425 kernel/signal.c:2903)
> [4394886.438419] ? find_get_pid (include/linux/rcupdate.h:914 kernel/pid.c:494)
> [4394886.439134] ? kill_pid (kernel/signal.c:2894)
> [4394886.439782] ? find_get_pid (kernel/pid.c:497)
> [4394886.440492] ? find_get_pid (kernel/pid.c:489)
> [4394886.441183] ? lock_is_held (kernel/locking/lockdep.c:3661)
> [4394886.441929] ? rcu_read_lock_sched_held (kernel/rcu/update.c:109)
> [4394886.442772] ? syscall_trace_enter_phase2 (arch/x86/entry/common.c:196)
> [4394886.443632] SyS_kill (kernel/signal.c:2893)
> [4394886.444255] tracesys_phase2 (arch/x86/entry/entry_64.S:270)
>
> I worked with Paul to rule out RCU as the cause.
>
> I've noticed that all traces had one thing in common: being stuck in preempt_schedule_irq(),
> so I've looked at recent changes there and noticed this commit.
>
> I've tried testing the commit before that, and the problem went away. Checking out this
> commit the problem reappeared.
Do you have any idea how to reproduce that?
Thanks.
next prev parent reply other threads:[~2015-07-20 22:49 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-05-12 14:41 [PATCH 0/6] preempt: A few headers cleanups and preempt_schedule*() optimizations v2 Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 1/6] preempt: Merge preempt_mask.h into preempt.h Frederic Weisbecker
2015-05-19 7:16 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 2/6] preempt: Rearrange a few symbols after headers merge Frederic Weisbecker
2015-05-19 7:16 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 3/6] preempt: Rename PREEMPT_CHECK_OFFSET to PREEMPT_DISABLE_OFFSET Frederic Weisbecker
2015-05-19 7:16 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 4/6] sched: Optimize preemption operations on __schedule() callers Frederic Weisbecker
2015-05-19 7:17 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
2015-07-20 22:14 ` Sasha Levin
2015-07-20 22:48 ` Frederic Weisbecker [this message]
2015-07-20 22:52 ` Sasha Levin
2015-07-21 11:07 ` Peter Zijlstra
2015-05-12 14:41 ` [PATCH 5/6] preempt: Fix out of date comment Frederic Weisbecker
2015-05-19 7:17 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
2015-05-12 14:41 ` [PATCH 6/6] preempt: Remove PREEMPT_ACTIVE unmasking off in_atomic() Frederic Weisbecker
2015-05-19 7:17 ` [tip:sched/core] sched/preempt: " tip-bot for Frederic Weisbecker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150720224858.GD13032@lerouge \
--to=fweisbec@gmail.com \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=sasha.levin@oracle.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).