From mboxrd@z Thu Jan 1 00:00:00 1970 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754130AbeAJT3Y (ORCPT + 1 other); Wed, 10 Jan 2018 14:29:24 -0500 Received: from wtarreau.pck.nerim.net ([62.212.114.60]:39471 "EHLO 1wt.eu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753909AbeAJT3V (ORCPT ); Wed, 10 Jan 2018 14:29:21 -0500 From: Willy Tarreau To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Willy Tarreau , Andy Lutomirski , Borislav Petkov , Brian Gerst , Dave Hansen , Ingo Molnar , Linus Torvalds , Peter Zijlstra , Thomas Gleixner , Josh Poimboeuf , "H. Peter Anvin" , Kees Cook Subject: [RFC PATCH v3 7/8] x86/entry/pti: avoid setting CR3 when it's already correct Date: Wed, 10 Jan 2018 20:28:19 +0100 Message-Id: <1515612500-14505-8-git-send-email-w@1wt.eu> X-Mailer: git-send-email 2.8.0.rc2.1.gbe9624a In-Reply-To: <1515612500-14505-1-git-send-email-w@1wt.eu> References: <1515612500-14505-1-git-send-email-w@1wt.eu> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Return-Path: When entering the kernel with CR3 pointing to the kernel's PGD, there's no need to set it again. This will avoid a TLB flush on syscalls for tasks running with the kernel's PGD (see next patch). Signed-off-by: Willy Tarreau Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Brian Gerst Cc: Dave Hansen Cc: Ingo Molnar Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Josh Poimboeuf Cc: "H. Peter Anvin" Cc: Kees Cook v2: - updated comments according to Ingo's suggestions - split the code to keep only the CR3 changes here --- arch/x86/entry/calling.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h index 45a63e0..19c6790 100644 --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -214,6 +214,11 @@ .macro SWITCH_TO_KERNEL_CR3 scratch_reg:req ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI mov %cr3, \scratch_reg + + /* If we're already on the kernel PGD, we don't switch */ + testq $(PTI_SWITCH_PGTABLES_MASK), \scratch_reg + jz .Lend_\@ + ADJUST_KERNEL_CR3 \scratch_reg mov \scratch_reg, %cr3 .Lend_\@: @@ -262,6 +267,14 @@ ALTERNATIVE "jmp .Ldone_\@", "", X86_FEATURE_PTI movq %cr3, \scratch_reg movq \scratch_reg, \save_reg + + /* + * If we're already on the kernel PGD, we don't switch, + * we just save the current CR3. + */ + testq $(PTI_SWITCH_PGTABLES_MASK), \scratch_reg + jz .Ldone_\@ + /* * Is the "switch mask" all zero? That means that both of * these are zero: @@ -284,6 +297,13 @@ .macro RESTORE_CR3 scratch_reg:req save_reg:req ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI + /* + * If we saved a kernel context on entry, we didn't switch the CR3, + * so we don't need to restore it on the way out either: + */ + testq $(PTI_SWITCH_PGTABLES_MASK), \save_reg + jz .Lend_\@ + ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID /* -- 1.7.12.1