From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBA6FC4360C for ; Fri, 4 Oct 2019 18:17:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D1D5A21D81 for ; Fri, 4 Oct 2019 18:17:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730611AbfJDSRo (ORCPT ); Fri, 4 Oct 2019 14:17:44 -0400 Received: from mga07.intel.com ([134.134.136.100]:15531 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730588AbfJDSRE (ORCPT ); Fri, 4 Oct 2019 14:17:04 -0400 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 04 Oct 2019 11:17:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.67,257,1566889200"; d="scan'208";a="204394796" Received: from chang-linux-3.sc.intel.com ([172.25.66.185]) by orsmga002.jf.intel.com with ESMTP; 04 Oct 2019 11:17:03 -0700 From: "Chang S. Bae" To: linux-kernel@vger.kernel.org, tglx@linutronix.de, bp@alien8.de, luto@kernel.org Cc: hpa@zytor.com, dave.hansen@intel.com, tony.luck@intel.com, ak@linux.intel.com, ravi.v.shankar@intel.com, chang.seok.bae@intel.com, Andrew Cooper Subject: [PATCH v9 10/17] x86/fsgsbase/64: Enable FSGSBASE instructions in helper functions Date: Fri, 4 Oct 2019 11:16:02 -0700 Message-Id: <1570212969-21888-11-git-send-email-chang.seok.bae@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1570212969-21888-1-git-send-email-chang.seok.bae@intel.com> References: <1570212969-21888-1-git-send-email-chang.seok.bae@intel.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add CPU feature conditional FS/GS base access to the relevant helper functions. That allows accelerating certain FS/GS base operations in subsequent changes. Note, that while possible, the user space entry/exit GS base operations are not going to use the new FSGSBASE instructions. The reason is that it would require additional storage for the user space value which adds more complexity to the low level code and experiments have shown marginal benefit. This may be revisited later but for now the SWAPGS based handling in the entry code is preserved except for the paranoid entry/exit code. Suggested-by: Tony Luck Signed-off-by: Chang S. Bae Reviewed-by: Tony Luck Cc: Thomas Gleixner Cc: Borislav Petkov Cc: Andy Lutomirski Cc: H. Peter Anvin Cc: Dave Hansen Cc: Tony Luck Cc: Andi Kleen Cc: Andrew Cooper --- Changes from v8: * Internalized the interrupt condition check in the helper functions (Andy L.) * Simplified the GS base read/write helper functions (Tony) * Massaged the changelog to reflect the helper changes Changes from v7: * Added interrupt-related warning messages by Thomas * Massaged changelog by Thomas * Used '[FS|GS] base' consistently, instead of '[FS|GS]BASE' --- arch/x86/include/asm/fsgsbase.h | 27 +++++++++---------- arch/x86/kernel/process_64.c | 58 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/fsgsbase.h b/arch/x86/include/asm/fsgsbase.h index fdd1177..aefd537 100644 --- a/arch/x86/include/asm/fsgsbase.h +++ b/arch/x86/include/asm/fsgsbase.h @@ -49,35 +49,32 @@ static __always_inline void wrgsbase(unsigned long gsbase) asm volatile("wrgsbase %0" :: "r" (gsbase) : "memory"); } +#include + /* Helper functions for reading/writing FS/GS base */ static inline unsigned long x86_fsbase_read_cpu(void) { unsigned long fsbase; - rdmsrl(MSR_FS_BASE, fsbase); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) + fsbase = rdfsbase(); + else + rdmsrl(MSR_FS_BASE, fsbase); return fsbase; } -static inline unsigned long x86_gsbase_read_cpu_inactive(void) -{ - unsigned long gsbase; - - rdmsrl(MSR_KERNEL_GS_BASE, gsbase); - - return gsbase; -} - static inline void x86_fsbase_write_cpu(unsigned long fsbase) { - wrmsrl(MSR_FS_BASE, fsbase); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) + wrfsbase(fsbase); + else + wrmsrl(MSR_FS_BASE, fsbase); } -static inline void x86_gsbase_write_cpu_inactive(unsigned long gsbase) -{ - wrmsrl(MSR_KERNEL_GS_BASE, gsbase); -} +extern unsigned long x86_gsbase_read_cpu_inactive(void); +extern void x86_gsbase_write_cpu_inactive(unsigned long gsbase); #endif /* CONFIG_X86_64 */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index af64519..295aa0c 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -329,6 +329,64 @@ static unsigned long x86_fsgsbase_read_task(struct task_struct *task, return base; } +unsigned long x86_gsbase_read_cpu_inactive(void) +{ + unsigned long gsbase; + + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + bool need_restore = false; + unsigned long flags; + + /* + * We read the inactive GS base value by swapping + * to make it the active one. But we cannot allow + * an interrupt while we switch to and from. + */ + if (!irqs_disabled()) { + local_irq_save(flags); + need_restore = true; + } + + native_swapgs(); + gsbase = rdgsbase(); + native_swapgs(); + + if (need_restore) + local_irq_restore(flags); + } else { + rdmsrl(MSR_KERNEL_GS_BASE, gsbase); + } + + return gsbase; +} + +void x86_gsbase_write_cpu_inactive(unsigned long gsbase) +{ + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + bool need_restore = false; + unsigned long flags; + + /* + * We write the inactive GS base value by swapping + * to make it the active one. But we cannot allow + * an interrupt while we switch to and from. + */ + if (!irqs_disabled()) { + local_irq_save(flags); + need_restore = true; + } + + native_swapgs(); + wrgsbase(gsbase); + native_swapgs(); + + if (need_restore) + local_irq_restore(flags); + } else { + wrmsrl(MSR_KERNEL_GS_BASE, gsbase); + } +} + unsigned long x86_fsbase_read_task(struct task_struct *task) { unsigned long fsbase; -- 2.7.4