From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754636AbbCPMM0 (ORCPT ); Mon, 16 Mar 2015 08:12:26 -0400 Received: from terminus.zytor.com ([198.137.202.10]:36167 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750948AbbCPMMV (ORCPT ); Mon, 16 Mar 2015 08:12:21 -0400 Date: Mon, 16 Mar 2015 05:11:25 -0700 From: tip-bot for Mike Galbraith Message-ID: Cc: linux-kernel@vger.kernel.org, efault@gmx.de, mingo@kernel.org, lenb@kernel.org, tglx@linutronix.de, jwboyer@redhat.com, peterz@infradead.org, bitbucket@online.de, torvalds@linux-foundation.org, hpa@zytor.com, ibmalone@gmail.com, bp@alien8.de, len.brown@intel.com Reply-To: jwboyer@redhat.com, tglx@linutronix.de, mingo@kernel.org, lenb@kernel.org, bp@alien8.de, len.brown@intel.com, hpa@zytor.com, ibmalone@gmail.com, torvalds@linux-foundation.org, peterz@infradead.org, bitbucket@online.de, efault@gmx.de, linux-kernel@vger.kernel.org In-Reply-To: <1390061684.5566.4.camel@marge.simpson.net> References: <1390061684.5566.4.camel@marge.simpson.net> To: linux-tip-commits@vger.kernel.org Subject: [tip:sched/core] sched/idle/x86: Optimize unnecessary mwait_idle( ) resched IPIs Git-Commit-ID: f8e617f4582995f7c25ef25b4167213120ad122b X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: f8e617f4582995f7c25ef25b4167213120ad122b Gitweb: http://git.kernel.org/tip/f8e617f4582995f7c25ef25b4167213120ad122b Author: Mike Galbraith AuthorDate: Sat, 18 Jan 2014 17:14:44 +0100 Committer: Ingo Molnar CommitDate: Mon, 16 Mar 2015 11:14:22 +0100 sched/idle/x86: Optimize unnecessary mwait_idle() resched IPIs To fully take advantage of MWAIT, apparently the CLFLUSH instruction needs another quirk on certain CPUs: proper barriers around it on certain machines. On a Q6600 SMP system, pipe-test scheduling performance, cross core, improves significantly: 3.8.13 487.2 KHz 1.000 3.13.0-master 415.5 KHz .852 3.13.0-master+ 415.2 KHz .852 + restore mwait_idle 3.13.0-master++ 488.5 KHz 1.002 + restore mwait_idle + IPI fix Since X86_BUG_CLFLUSH_MONITOR is already a quirk, don't create a separate quirk for the extra smp_mb()s. Signed-off-by: Mike Galbraith Cc: # 3.10+ Cc: Borislav Petkov Cc: H. Peter Anvin Cc: Ian Malone Cc: Josh Boyer Cc: Len Brown Cc: Len Brown Cc: Linus Torvalds Cc: Mike Galbraith Cc: Peter Zijlstra Cc: Thomas Gleixner Link: http://lkml.kernel.org/r/1390061684.5566.4.camel@marge.simpson.net [ Ported to recent kernel, added comments about the quirk. ] Signed-off-by: Ingo Molnar --- arch/x86/kernel/process.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index da06f74..6ad8a63 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -428,18 +428,22 @@ static int prefer_mwait_c1_over_halt(const struct cpuinfo_x86 *c) static void mwait_idle(void) { - if (!need_resched()) { - if (this_cpu_has(X86_BUG_CLFLUSH_MONITOR)) + if (!current_set_polling_and_test()) { + if (this_cpu_has(X86_BUG_CLFLUSH_MONITOR)) { + smp_mb(); /* quirk */ clflush((void *)¤t_thread_info()->flags); + smp_mb(); /* quirk */ + } __monitor((void *)¤t_thread_info()->flags, 0, 0); - smp_mb(); if (!need_resched()) __sti_mwait(0, 0); else local_irq_enable(); - } else + } else { local_irq_enable(); + } + __current_clr_polling(); } void select_idle_routine(const struct cpuinfo_x86 *c)