From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9AE8C433DF for ; Thu, 28 May 2020 20:14:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A72A32088E for ; Thu, 28 May 2020 20:14:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1590696886; bh=TPKI62BqBvZhsuUBfCwO33Stb+N1jFBOmDUnrHw33C0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=N9RCzf5sejwphock8+DknCcRkzolEK/FFwsJbDR5Q+m9OsmPfdggKOYOjpaGXAGmc 4V7hHKMCEzNSdgiMrRVzLoaV+s+zfIzKiUTFwvmBGI4K/9YBqC0q02idZ5rM4JjUov IYw7QO/Xg0uyNONHONKBkGK0HrC6Y96DjIdyORI8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2407062AbgE1UOp (ORCPT ); Thu, 28 May 2020 16:14:45 -0400 Received: from mail.kernel.org ([198.145.29.99]:41758 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2407011AbgE1UOW (ORCPT ); Thu, 28 May 2020 16:14:22 -0400 Received: from sasha-vm.mshome.net (c-73-47-72-35.hsd1.nh.comcast.net [73.47.72.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 186C2214F1; Thu, 28 May 2020 20:14:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1590696861; bh=TPKI62BqBvZhsuUBfCwO33Stb+N1jFBOmDUnrHw33C0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f8/4xF0MgTlsClMU1iKlEI5LTrwokkIgX40gbSxCSGtBYjocm+sM/rRNlkZk5Vdco PWfgYjS0IqC3QDYErVCE3SqFl1A3uwMZ8fxjKEYseLjjerhqVPnQH/mDdLRm91EK7S XUJ7T7yyBHJJX8ZdH3jXLWnatzXMXgKZBGlvON1A= From: Sasha Levin To: tglx@linutronix.de, luto@kernel.org, ak@linux.intel.com Cc: corbet@lwn.net, mingo@redhat.com, bp@alien8.de, x86@kernel.org, shuah@kernel.org, gregkh@linuxfoundation.org, tony.luck@intel.com, chang.seok.bae@intel.com, dave.hansen@linux.intel.com, peterz@infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, jarkko.sakkinen@linux.intel.com, Ravi Shankar , "H . Peter Anvin" , Sasha Levin Subject: [PATCH v13 05/16] x86/process/64: Use FSBSBASE in switch_to() if available Date: Thu, 28 May 2020 16:13:51 -0400 Message-Id: <20200528201402.1708239-6-sashal@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200528201402.1708239-1-sashal@kernel.org> References: <20200528201402.1708239-1-sashal@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Andy Lutomirski With the new FSGSBASE instructions, FS and GSABSE can be efficiently read and writen in __switch_to(). Use that capability to preserve the full state. This will enable user code to do whatever it wants with the new instructions without any kernel-induced gotchas. (There can still be architectural gotchas: movl %gs,%eax; movl %eax,%gs may change GSBASE if WRGSBASE was used, but users are expected to read the CPU manual before doing things like that.) This is a considerable speedup. It seems to save about 100 cycles per context switch compared to the baseline 4.6-rc1 behavior on a Skylake laptop. [ chang: 5~10% performance improvements were seen with a context switch benchmark that ran threads with different FS/GSBASE values (to the baseline 4.16). Minor edit on the changelog. ] [ tglx: Masaage changelog ] Signed-off-by: Andy Lutomirski Signed-off-by: Chang S. Bae Signed-off-by: Thomas Gleixner Reviewed-by: Andi Kleen Cc: Ravi Shankar Cc: H. Peter Anvin Link: https://lkml.kernel.org/r/1557309753-24073-8-git-send-email-chang.seok.bae@intel.com Signed-off-by: Sasha Levin --- arch/x86/kernel/process_64.c | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 0624689825a3..85c7f9cabde2 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -233,8 +233,18 @@ static __always_inline void save_fsgs(struct task_struct *task) { savesegment(fs, task->thread.fsindex); savesegment(gs, task->thread.gsindex); - save_base_legacy(task, task->thread.fsindex, FS); - save_base_legacy(task, task->thread.gsindex, GS); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + /* + * If FSGSBASE is enabled, we can't make any useful guesses + * about the base, and user code expects us to save the current + * value. Fortunately, reading the base directly is efficient. + */ + task->thread.fsbase = rdfsbase(); + task->thread.gsbase = __rdgsbase_inactive(); + } else { + save_base_legacy(task, task->thread.fsindex, FS); + save_base_legacy(task, task->thread.gsindex, GS); + } } #if IS_ENABLED(CONFIG_KVM) @@ -313,10 +323,22 @@ static __always_inline void load_seg_legacy(unsigned short prev_index, static __always_inline void x86_fsgsbase_load(struct thread_struct *prev, struct thread_struct *next) { - load_seg_legacy(prev->fsindex, prev->fsbase, - next->fsindex, next->fsbase, FS); - load_seg_legacy(prev->gsindex, prev->gsbase, - next->gsindex, next->gsbase, GS); + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + /* Update the FS and GS selectors if they could have changed. */ + if (unlikely(prev->fsindex || next->fsindex)) + loadseg(FS, next->fsindex); + if (unlikely(prev->gsindex || next->gsindex)) + loadseg(GS, next->gsindex); + + /* Update the bases. */ + wrfsbase(next->fsbase); + __wrgsbase_inactive(next->gsbase); + } else { + load_seg_legacy(prev->fsindex, prev->fsbase, + next->fsindex, next->fsbase, FS); + load_seg_legacy(prev->gsindex, prev->gsbase, + next->gsindex, next->gsbase, GS); + } } static unsigned long x86_fsgsbase_read_task(struct task_struct *task, -- 2.25.1