From: Thadeu Lima de Souza Cascardo <cascardo@canonical.com>
To: stable@vger.kernel.org
Cc: x86@kernel.org, kvm@vger.kernel.org, bp@alien8.de,
pbonzini@redhat.com, peterz@infradead.org, jpoimboe@kernel.org
Subject: [PATCH 5.4 13/37] x86/bugs: Optimize SPEC_CTRL MSR writes
Date: Mon, 3 Oct 2022 10:10:14 -0300 [thread overview]
Message-ID: <20221003131038.12645-14-cascardo@canonical.com> (raw)
In-Reply-To: <20221003131038.12645-1-cascardo@canonical.com>
From: Peter Zijlstra <peterz@infradead.org>
commit c779bc1a9002fa474175b80e72b85c9bf628abb0 upstream.
When changing SPEC_CTRL for user control, the WRMSR can be delayed
until return-to-user when KERNEL_IBRS has been enabled.
This avoids an MSR write during context switch.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@canonical.com>
---
arch/x86/include/asm/nospec-branch.h | 2 +-
arch/x86/kernel/cpu/bugs.c | 18 ++++++++++++------
arch/x86/kernel/process.c | 2 +-
3 files changed, 14 insertions(+), 8 deletions(-)
diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
index 9ac1fd82f479..2e7da90a5f73 100644
--- a/arch/x86/include/asm/nospec-branch.h
+++ b/arch/x86/include/asm/nospec-branch.h
@@ -297,7 +297,7 @@ static inline void indirect_branch_prediction_barrier(void)
/* The Intel SPEC CTRL MSR base value cache */
extern u64 x86_spec_ctrl_base;
-extern void write_spec_ctrl_current(u64 val);
+extern void write_spec_ctrl_current(u64 val, bool force);
/*
* With retpoline, we must use IBRS to restrict branch prediction
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index e68f8bc45d3d..549d603b5f6e 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -61,13 +61,19 @@ static DEFINE_MUTEX(spec_ctrl_mutex);
* Keep track of the SPEC_CTRL MSR value for the current task, which may differ
* from x86_spec_ctrl_base due to STIBP/SSB in __speculation_ctrl_update().
*/
-void write_spec_ctrl_current(u64 val)
+void write_spec_ctrl_current(u64 val, bool force)
{
if (this_cpu_read(x86_spec_ctrl_current) == val)
return;
this_cpu_write(x86_spec_ctrl_current, val);
- wrmsrl(MSR_IA32_SPEC_CTRL, val);
+
+ /*
+ * When KERNEL_IBRS this MSR is written on return-to-user, unless
+ * forced the update can be delayed until that time.
+ */
+ if (force || !cpu_feature_enabled(X86_FEATURE_KERNEL_IBRS))
+ wrmsrl(MSR_IA32_SPEC_CTRL, val);
}
/*
@@ -1195,7 +1201,7 @@ static void __init spectre_v2_select_mitigation(void)
if (spectre_v2_in_eibrs_mode(mode)) {
/* Force it so VMEXIT will restore correctly */
x86_spec_ctrl_base |= SPEC_CTRL_IBRS;
- write_spec_ctrl_current(x86_spec_ctrl_base);
+ write_spec_ctrl_current(x86_spec_ctrl_base, true);
}
switch (mode) {
@@ -1250,7 +1256,7 @@ static void __init spectre_v2_select_mitigation(void)
static void update_stibp_msr(void * __unused)
{
- write_spec_ctrl_current(x86_spec_ctrl_base);
+ write_spec_ctrl_current(x86_spec_ctrl_base, true);
}
/* Update x86_spec_ctrl_base in case SMT state changed. */
@@ -1493,7 +1499,7 @@ static enum ssb_mitigation __init __ssb_select_mitigation(void)
x86_amd_ssb_disable();
} else {
x86_spec_ctrl_base |= SPEC_CTRL_SSBD;
- write_spec_ctrl_current(x86_spec_ctrl_base);
+ write_spec_ctrl_current(x86_spec_ctrl_base, true);
}
}
@@ -1710,7 +1716,7 @@ int arch_prctl_spec_ctrl_get(struct task_struct *task, unsigned long which)
void x86_spec_ctrl_setup_ap(void)
{
if (boot_cpu_has(X86_FEATURE_MSR_SPEC_CTRL))
- write_spec_ctrl_current(x86_spec_ctrl_base);
+ write_spec_ctrl_current(x86_spec_ctrl_base, true);
if (ssb_mode == SPEC_STORE_BYPASS_DISABLE)
x86_amd_ssb_disable();
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index 0ddc1504ad57..87cfd2ee9ca0 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -449,7 +449,7 @@ static __always_inline void __speculation_ctrl_update(unsigned long tifp,
}
if (updmsr)
- write_spec_ctrl_current(msr);
+ write_spec_ctrl_current(msr, false);
}
static unsigned long speculation_ctrl_update_tif(struct task_struct *tsk)
--
2.34.1
next prev parent reply other threads:[~2022-10-03 13:12 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-03 13:10 [PATCH 5.4 00/37] IBRS support // Retbleed mitigations Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 01/37] Revert "x86/speculation: Add RSB VM Exit protections" Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 02/37] Revert "x86/cpu: Add a steppings field to struct x86_cpu_id" Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 03/37] x86/devicetable: Move x86 specific macro out of generic code Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 04/37] x86/cpu: Add consistent CPU match macros Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 05/37] x86/cpu: Add a steppings field to struct x86_cpu_id Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 06/37] x86/kvm/vmx: Make noinstr clean Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 07/37] x86/cpufeatures: Move RETPOLINE flags to word 11 Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 08/37] x86/bugs: Report AMD retbleed vulnerability Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 09/37] x86/bugs: Add AMD retbleed= boot parameter Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 10/37] x86/bugs: Keep a per-CPU IA32_SPEC_CTRL value Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 11/37] x86/entry: Remove skip_r11rcx Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 12/37] x86/entry: Add kernel IBRS implementation Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` Thadeu Lima de Souza Cascardo [this message]
2022-10-03 13:10 ` [PATCH 5.4 14/37] x86/speculation: Add spectre_v2=ibrs option to support Kernel IBRS Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 15/37] x86/bugs: Split spectre_v2_select_mitigation() and spectre_v2_user_select_mitigation() Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 16/37] x86/bugs: Report Intel retbleed vulnerability Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 17/37] intel_idle: Disable IBRS during long idle Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 18/37] x86/speculation: Change FILL_RETURN_BUFFER to work with objtool Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 19/37] x86/speculation: Fix RSB filling with CONFIG_RETPOLINE=n Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 20/37] x86/speculation: Fix firmware entry SPEC_CTRL handling Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 21/37] x86/speculation: Fix SPEC_CTRL write on SMT state change Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 22/37] x86/speculation: Use cached host SPEC_CTRL value for guest entry/exit Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 23/37] x86/speculation: Remove x86_spec_ctrl_mask Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 24/37] KVM/VMX: Use TEST %REG,%REG instead of CMP $0,%REG in vmenter.S Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 25/37] KVM/nVMX: Use __vmx_vcpu_run in nested_vmx_check_vmentry_hw Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 26/37] KVM: VMX: Flatten __vmx_vcpu_run() Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 27/37] KVM: VMX: Convert launched argument to flags Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 28/37] KVM: VMX: Prevent guest RSB poisoning attacks with eIBRS Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 29/37] KVM: VMX: Fix IBRS handling after vmexit Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 30/37] x86/speculation: Fill RSB on vmexit for IBRS Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 31/37] x86/common: Stamp out the stepping madness Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 32/37] x86/cpu/amd: Enumerate BTC_NO Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 33/37] x86/bugs: Add Cannon lake to RETBleed affected CPU list Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 34/37] x86/speculation: Disable RRSBA behavior Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 35/37] x86/speculation: Use DECLARE_PER_CPU for x86_spec_ctrl_current Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 36/37] x86/bugs: Warn when "ibrs" mitigation is selected on Enhanced IBRS parts Thadeu Lima de Souza Cascardo
2022-10-03 13:10 ` [PATCH 5.4 37/37] x86/speculation: Add RSB VM Exit protections Thadeu Lima de Souza Cascardo
2022-10-05 10:34 ` [PATCH 5.4 00/37] IBRS support // Retbleed mitigations Greg KH
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221003131038.12645-14-cascardo@canonical.com \
--to=cascardo@canonical.com \
--cc=bp@alien8.de \
--cc=jpoimboe@kernel.org \
--cc=kvm@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=peterz@infradead.org \
--cc=stable@vger.kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).