From: Thomas Gleixner <tglx@linutronix.de> To: LKML <linux-kernel@vger.kernel.org> Cc: x86@kernel.org, Paul McKenney <paulmck@kernel.org>, Josh Poimboeuf <jpoimboe@redhat.com>, "Joel Fernandes (Google)" <joel@joelfernandes.org>, "Steven Rostedt (VMware)" <rostedt@goodmis.org>, Masami Hiramatsu <mhiramat@kernel.org>, Alexei Starovoitov <ast@kernel.org>, Frederic Weisbecker <frederic@kernel.org>, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>, Brian Gerst <brgerst@gmail.com>, Juergen Gross <jgross@suse.com>, Alexandre Chartre <alexandre.chartre@oracle.com>, Tom Lendacky <thomas.lendacky@amd.com>, Paolo Bonzini <pbonzini@redhat.com>, kvm@vger.kernel.org, Peter Zijlstra <peterz@infradead.org> Subject: [patch V3 23/23] x86/kvm/svm: Move guest enter/exit into .noinstr.text Date: Fri, 20 Mar 2020 19:00:19 +0100 [thread overview] Message-ID: <20200320180034.672927065@linutronix.de> (raw) In-Reply-To: 20200320175956.033706968@linutronix.de Split out the really last steps of guest enter and the early guest exit code and mark it .noinstr.text. Add the required instr_begin()/end() pairs around "safe" code and replace the wrmsr() with native_wrmsr() to prevent a tracepoint injection. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Cc: Tom Lendacky <thomas.lendacky@amd.com> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: kvm@vger.kernel.org --- arch/x86/kvm/svm.c | 114 ++++++++++++++++++++++++++++------------------------- 1 file changed, 62 insertions(+), 52 deletions(-) --- a/arch/x86/kvm/svm.c +++ b/arch/x86/kvm/svm.c @@ -5714,58 +5714,9 @@ static void svm_cancel_injection(struct svm_complete_interrupts(svm); } -static void svm_vcpu_run(struct kvm_vcpu *vcpu) +static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, + struct vcpu_svm *svm) { - struct vcpu_svm *svm = to_svm(vcpu); - - svm->vmcb->save.rax = vcpu->arch.regs[VCPU_REGS_RAX]; - svm->vmcb->save.rsp = vcpu->arch.regs[VCPU_REGS_RSP]; - svm->vmcb->save.rip = vcpu->arch.regs[VCPU_REGS_RIP]; - - /* - * A vmexit emulation is required before the vcpu can be executed - * again. - */ - if (unlikely(svm->nested.exit_required)) - return; - - /* - * Disable singlestep if we're injecting an interrupt/exception. - * We don't want our modified rflags to be pushed on the stack where - * we might not be able to easily reset them if we disabled NMI - * singlestep later. - */ - if (svm->nmi_singlestep && svm->vmcb->control.event_inj) { - /* - * Event injection happens before external interrupts cause a - * vmexit and interrupts are disabled here, so smp_send_reschedule - * is enough to force an immediate vmexit. - */ - disable_nmi_singlestep(svm); - smp_send_reschedule(vcpu->cpu); - } - - pre_svm_run(svm); - - sync_lapic_to_cr8(vcpu); - - svm->vmcb->save.cr2 = vcpu->arch.cr2; - - clgi(); - kvm_load_guest_xsave_state(vcpu); - - if (lapic_in_kernel(vcpu) && - vcpu->arch.apic->lapic_timer.timer_advance_ns) - kvm_wait_lapic_expire(vcpu); - - /* - * If this vCPU has touched SPEC_CTRL, restore the guest's value if - * it's non-zero. Since vmentry is serialising on affected CPUs, there - * is no need to worry about the conditional branch over the wrmsr - * being speculatively taken. - */ - x86_spec_ctrl_set_guest(svm->spec_ctrl, svm->virt_spec_ctrl); - /* * VMENTER enables interrupts (host state), but the kernel state is * interrupts disabled when this is invoked. Also tell RCU about @@ -5780,8 +5731,10 @@ static void svm_vcpu_run(struct kvm_vcpu * take locks (lockdep needs RCU) and calls into world and some * more. */ + instr_begin(); __trace_hardirqs_on(); lockdep_hardirqs_on_prepare(CALLER_ADDR0); + instr_end(); guest_enter_irqoff(); lockdep_hardirqs_on(CALLER_ADDR0); @@ -5881,7 +5834,7 @@ static void svm_vcpu_run(struct kvm_vcpu vmexit_fill_RSB(); #ifdef CONFIG_X86_64 - wrmsrl(MSR_GS_BASE, svm->host.gs_base); + native_wrmsrl(MSR_GS_BASE, svm->host.gs_base); #else loadsegment(fs, svm->host.fs); #ifndef CONFIG_X86_32_LAZY_GS @@ -5904,7 +5857,64 @@ static void svm_vcpu_run(struct kvm_vcpu */ lockdep_hardirqs_off(CALLER_ADDR0); guest_exit_irqoff(); + instr_begin(); __trace_hardirqs_off(); + instr_end(); +} + +static void svm_vcpu_run(struct kvm_vcpu *vcpu) +{ + struct vcpu_svm *svm = to_svm(vcpu); + + svm->vmcb->save.rax = vcpu->arch.regs[VCPU_REGS_RAX]; + svm->vmcb->save.rsp = vcpu->arch.regs[VCPU_REGS_RSP]; + svm->vmcb->save.rip = vcpu->arch.regs[VCPU_REGS_RIP]; + + /* + * A vmexit emulation is required before the vcpu can be executed + * again. + */ + if (unlikely(svm->nested.exit_required)) + return; + + /* + * Disable singlestep if we're injecting an interrupt/exception. + * We don't want our modified rflags to be pushed on the stack where + * we might not be able to easily reset them if we disabled NMI + * singlestep later. + */ + if (svm->nmi_singlestep && svm->vmcb->control.event_inj) { + /* + * Event injection happens before external interrupts cause a + * vmexit and interrupts are disabled here, so smp_send_reschedule + * is enough to force an immediate vmexit. + */ + disable_nmi_singlestep(svm); + smp_send_reschedule(vcpu->cpu); + } + + pre_svm_run(svm); + + sync_lapic_to_cr8(vcpu); + + svm->vmcb->save.cr2 = vcpu->arch.cr2; + + clgi(); + kvm_load_guest_xsave_state(vcpu); + + if (lapic_in_kernel(vcpu) && + vcpu->arch.apic->lapic_timer.timer_advance_ns) + kvm_wait_lapic_expire(vcpu); + + /* + * If this vCPU has touched SPEC_CTRL, restore the guest's value if + * it's non-zero. Since vmentry is serialising on affected CPUs, there + * is no need to worry about the conditional branch over the wrmsr + * being speculatively taken. + */ + x86_spec_ctrl_set_guest(svm->spec_ctrl, svm->virt_spec_ctrl); + + svm_vcpu_enter_exit(vcpu, svm); /* * We do not use IBRS in the kernel. If this vCPU has used the
WARNING: multiple messages have this Message-ID (diff)
From: Thomas Gleixner <tglx@linutronix.de> To: LKML <linux-kernel@vger.kernel.org> Cc: x86@kernel.org, Paul McKenney <paulmck@kernel.org>, Josh Poimboeuf <jpoimboe@redhat.com>, "Joel Fernandes (Google)" <joel@joelfernandes.org>, "Steven Rostedt (VMware)" <rostedt@goodmis.org>, Masami Hiramatsu <mhiramat@kernel.org>, Alexei Starovoitov <ast@kernel.org>, Frederic Weisbecker <frederic@kernel.org>, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>, Brian Gerst <brgerst@gmail.com>, Juergen Gross <jgross@suse.com>, Alexandre Chartre <alexandre.chartre@oracle.com>, Tom Lendacky <thomas.lendacky@amd.com>, Paolo Bonzini <pbonzini@redhat.com>, kvm@vger.kernel.org, Peter Zijlstra <peterz@infradead.org> Subject: [RESEND][patch V3 23/23] x86/kvm/svm: Move guest enter/exit into .noinstr.text Date: Fri, 20 Mar 2020 19:00:19 +0100 [thread overview] Message-ID: <20200320180034.672927065@linutronix.de> (raw) Message-ID: <20200320180019.tGt75kXKPu3rh71MrB-Aa1X6fh_fHfuzoS_FyEFTOT4@z> (raw) In-Reply-To: 20200320175956.033706968@linutronix.de Split out the really last steps of guest enter and the early guest exit code and mark it .noinstr.text. Add the required instr_begin()/end() pairs around "safe" code and replace the wrmsr() with native_wrmsr() to prevent a tracepoint injection. Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Cc: Tom Lendacky <thomas.lendacky@amd.com> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: kvm@vger.kernel.org --- arch/x86/kvm/svm.c | 114 ++++++++++++++++++++++++++++------------------------- 1 file changed, 62 insertions(+), 52 deletions(-) --- a/arch/x86/kvm/svm.c +++ b/arch/x86/kvm/svm.c @@ -5714,58 +5714,9 @@ static void svm_cancel_injection(struct svm_complete_interrupts(svm); } -static void svm_vcpu_run(struct kvm_vcpu *vcpu) +static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, + struct vcpu_svm *svm) { - struct vcpu_svm *svm = to_svm(vcpu); - - svm->vmcb->save.rax = vcpu->arch.regs[VCPU_REGS_RAX]; - svm->vmcb->save.rsp = vcpu->arch.regs[VCPU_REGS_RSP]; - svm->vmcb->save.rip = vcpu->arch.regs[VCPU_REGS_RIP]; - - /* - * A vmexit emulation is required before the vcpu can be executed - * again. - */ - if (unlikely(svm->nested.exit_required)) - return; - - /* - * Disable singlestep if we're injecting an interrupt/exception. - * We don't want our modified rflags to be pushed on the stack where - * we might not be able to easily reset them if we disabled NMI - * singlestep later. - */ - if (svm->nmi_singlestep && svm->vmcb->control.event_inj) { - /* - * Event injection happens before external interrupts cause a - * vmexit and interrupts are disabled here, so smp_send_reschedule - * is enough to force an immediate vmexit. - */ - disable_nmi_singlestep(svm); - smp_send_reschedule(vcpu->cpu); - } - - pre_svm_run(svm); - - sync_lapic_to_cr8(vcpu); - - svm->vmcb->save.cr2 = vcpu->arch.cr2; - - clgi(); - kvm_load_guest_xsave_state(vcpu); - - if (lapic_in_kernel(vcpu) && - vcpu->arch.apic->lapic_timer.timer_advance_ns) - kvm_wait_lapic_expire(vcpu); - - /* - * If this vCPU has touched SPEC_CTRL, restore the guest's value if - * it's non-zero. Since vmentry is serialising on affected CPUs, there - * is no need to worry about the conditional branch over the wrmsr - * being speculatively taken. - */ - x86_spec_ctrl_set_guest(svm->spec_ctrl, svm->virt_spec_ctrl); - /* * VMENTER enables interrupts (host state), but the kernel state is * interrupts disabled when this is invoked. Also tell RCU about @@ -5780,8 +5731,10 @@ static void svm_vcpu_run(struct kvm_vcpu * take locks (lockdep needs RCU) and calls into world and some * more. */ + instr_begin(); __trace_hardirqs_on(); lockdep_hardirqs_on_prepare(CALLER_ADDR0); + instr_end(); guest_enter_irqoff(); lockdep_hardirqs_on(CALLER_ADDR0); @@ -5881,7 +5834,7 @@ static void svm_vcpu_run(struct kvm_vcpu vmexit_fill_RSB(); #ifdef CONFIG_X86_64 - wrmsrl(MSR_GS_BASE, svm->host.gs_base); + native_wrmsrl(MSR_GS_BASE, svm->host.gs_base); #else loadsegment(fs, svm->host.fs); #ifndef CONFIG_X86_32_LAZY_GS @@ -5904,7 +5857,64 @@ static void svm_vcpu_run(struct kvm_vcpu */ lockdep_hardirqs_off(CALLER_ADDR0); guest_exit_irqoff(); + instr_begin(); __trace_hardirqs_off(); + instr_end(); +} + +static void svm_vcpu_run(struct kvm_vcpu *vcpu) +{ + struct vcpu_svm *svm = to_svm(vcpu); + + svm->vmcb->save.rax = vcpu->arch.regs[VCPU_REGS_RAX]; + svm->vmcb->save.rsp = vcpu->arch.regs[VCPU_REGS_RSP]; + svm->vmcb->save.rip = vcpu->arch.regs[VCPU_REGS_RIP]; + + /* + * A vmexit emulation is required before the vcpu can be executed + * again. + */ + if (unlikely(svm->nested.exit_required)) + return; + + /* + * Disable singlestep if we're injecting an interrupt/exception. + * We don't want our modified rflags to be pushed on the stack where + * we might not be able to easily reset them if we disabled NMI + * singlestep later. + */ + if (svm->nmi_singlestep && svm->vmcb->control.event_inj) { + /* + * Event injection happens before external interrupts cause a + * vmexit and interrupts are disabled here, so smp_send_reschedule + * is enough to force an immediate vmexit. + */ + disable_nmi_singlestep(svm); + smp_send_reschedule(vcpu->cpu); + } + + pre_svm_run(svm); + + sync_lapic_to_cr8(vcpu); + + svm->vmcb->save.cr2 = vcpu->arch.cr2; + + clgi(); + kvm_load_guest_xsave_state(vcpu); + + if (lapic_in_kernel(vcpu) && + vcpu->arch.apic->lapic_timer.timer_advance_ns) + kvm_wait_lapic_expire(vcpu); + + /* + * If this vCPU has touched SPEC_CTRL, restore the guest's value if + * it's non-zero. Since vmentry is serialising on affected CPUs, there + * is no need to worry about the conditional branch over the wrmsr + * being speculatively taken. + */ + x86_spec_ctrl_set_guest(svm->spec_ctrl, svm->virt_spec_ctrl); + + svm_vcpu_enter_exit(vcpu, svm); /* * We do not use IBRS in the kernel. If this vCPU has used the
next prev parent reply other threads:[~2020-03-20 18:38 UTC|newest] Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-03-20 17:59 [patch V3 00/23] x86/entry: Consolidation part II (syscalls) Thomas Gleixner 2020-03-20 17:59 ` [RESEND][patch " Thomas Gleixner 2020-03-20 17:59 ` [patch V3 01/23] rcu: Dont acquire lock in NMI handler in rcu_nmi_enter_common() Thomas Gleixner 2020-03-20 17:59 ` [RESEND][patch " Thomas Gleixner 2020-03-24 15:37 ` [patch " Frederic Weisbecker 2020-03-20 17:59 ` [patch V3 02/23] rcu: Add comments marking transitions between RCU watching and not Thomas Gleixner 2020-03-20 17:59 ` [RESEND][patch " Thomas Gleixner 2020-03-24 15:38 ` [patch " Frederic Weisbecker 2020-03-20 17:59 ` [patch V3 03/23] vmlinux.lds.h: Create section for protection against instrumentation Thomas Gleixner 2020-03-20 17:59 ` [RESEND][patch " Thomas Gleixner 2020-03-24 12:26 ` Borislav Petkov 2020-04-03 8:08 ` Alexandre Chartre 2020-03-20 18:00 ` [patch V3 04/23] kprobes: Prevent probes in .noinstr.text section Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-23 14:00 ` [patch " Masami Hiramatsu 2020-03-23 16:03 ` Thomas Gleixner 2020-03-24 5:49 ` Masami Hiramatsu 2020-03-24 9:47 ` Thomas Gleixner 2020-03-25 13:39 ` Masami Hiramatsu 2020-03-20 18:00 ` [patch V3 05/23] tracing: Provide lockdep less trace_hardirqs_on/off() variants Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-04-03 8:34 ` Alexandre Chartre 2020-03-20 18:00 ` [patch V3 06/23] bug: Annotate WARN/BUG/stackfail as noinstr safe Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-04-02 21:01 ` Josh Poimboeuf 2020-04-02 21:34 ` Peter Zijlstra 2020-04-02 21:43 ` Josh Poimboeuf 2020-04-02 21:49 ` Thomas Gleixner 2020-03-20 18:00 ` [patch V3 07/23] lockdep: Prepare for noinstr sections Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 08/23] x86/entry: Mark enter_from_user_mode() noinstr Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 09/23] x86/entry/common: Protect against instrumentation Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 10/23] x86/entry: Move irq tracing on syscall entry to C-code Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 11/23] x86/entry: Move irq flags tracing to prepare_exit_to_usermode() Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 12/23] context_tracking: Ensure that the critical path cannot be instrumented Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 13/23] lib/smp_processor_id: Move it into noinstr section Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 14/23] x86/speculation/mds: Mark mds_user_clear_cpu_buffers() __always_inline Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 15/23] x86/entry/64: Check IF in __preempt_enable_notrace() thunk Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 16/23] x86/entry/64: Mark ___preempt_schedule_notrace() thunk noinstr Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 17/23] rcu/tree: Mark the idle relevant functions noinstr Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-24 16:09 ` Paul E. McKenney 2020-03-24 19:28 ` Thomas Gleixner 2020-03-24 19:58 ` Paul E. McKenney 2020-03-20 18:00 ` [patch V3 18/23] x86/kvm: Move context tracking where it belongs Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 19/23] x86/kvm/vmx: Add hardirq tracing to guest enter/exit Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-24 23:03 ` Peter Zijlstra 2020-03-24 23:21 ` Thomas Gleixner 2020-03-20 18:00 ` [patch V3 20/23] x86/kvm/svm: Handle hardirqs proper on " Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 21/23] context_tracking: Make guest_enter/exit_irqoff() .noinstr ready Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` [patch V3 22/23] x86/kvm/vmx: Move guest enter/exit into .noinstr.text Thomas Gleixner 2020-03-20 18:00 ` [RESEND][patch " Thomas Gleixner 2020-03-20 18:00 ` Thomas Gleixner [this message] 2020-03-20 18:00 ` [RESEND][patch V3 23/23] x86/kvm/svm: " Thomas Gleixner
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200320180034.672927065@linutronix.de \ --to=tglx@linutronix.de \ --cc=alexandre.chartre@oracle.com \ --cc=ast@kernel.org \ --cc=brgerst@gmail.com \ --cc=frederic@kernel.org \ --cc=jgross@suse.com \ --cc=joel@joelfernandes.org \ --cc=jpoimboe@redhat.com \ --cc=kvm@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mathieu.desnoyers@efficios.com \ --cc=mhiramat@kernel.org \ --cc=paulmck@kernel.org \ --cc=pbonzini@redhat.com \ --cc=peterz@infradead.org \ --cc=rostedt@goodmis.org \ --cc=thomas.lendacky@amd.com \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).