From: Xiaoyao Li <xiaoyao.li@intel.com>
To: Sean Christopherson <sean.j.christopherson@intel.com>,
Thomas Gleixner <tglx@linutronix.de>
Cc: Fenghua Yu <fenghua.yu@intel.com>, Ingo Molnar <mingo@redhat.com>,
Borislav Petkov <bp@alien8.de>, H Peter Anvin <hpa@zytor.com>,
Peter Zijlstra <peterz@infradead.org>,
Andrew Morton <akpm@linux-foundation.org>,
Dave Hansen <dave.hansen@intel.com>,
Paolo Bonzini <pbonzini@redhat.com>,
Radim Krcmar <rkrcmar@redhat.com>,
Ashok Raj <ashok.raj@intel.com>, Tony Luck <tony.luck@intel.com>,
Dan Williams <dan.j.williams@intel.com>,
Sai Praneeth Prakhya <sai.praneeth.prakhya@intel.com>,
Ravi V Shankar <ravi.v.shankar@intel.com>,
linux-kernel <linux-kernel@vger.kernel.org>, x86 <x86@kernel.org>,
kvm@vger.kernel.org
Subject: Re: [PATCH v9 09/17] x86/split_lock: Handle #AC exception for split lock
Date: Wed, 16 Oct 2019 14:58:05 +0800 [thread overview]
Message-ID: <bc8f5850-ba7d-45b7-a30b-5560764edcc8@intel.com> (raw)
In-Reply-To: <20190925180931.GG31852@linux.intel.com>
On 9/26/2019 2:09 AM, Sean Christopherson wrote:
> On Wed, Jun 26, 2019 at 11:47:40PM +0200, Thomas Gleixner wrote:
>> So only one of the CPUs will win the cmpxchg race, set te variable to 1 and
>> warn, the other and any subsequent AC on any other CPU will not warn
>> either. So you don't need WARN_ONCE() at all. It's redundant and confusing
>> along with the atomic_set().
>>
>> Whithout reading that link [1], what Ingo proposed was surely not the
>> trainwreck which you decided to put into that debugfs thing.
>
> We're trying to sort out the trainwreck, but there's an additional wrinkle
> that I'd like your input on.
>
> We overlooked the fact that MSR_TEST_CTRL is per-core, i.e. shared by
> sibling hyperthreads. This is especially problematic for KVM, as loading
> MSR_TEST_CTRL during VM-Enter could cause spurious #AC faults in the kernel
> and bounce MSR_TEST_CTRL.split_lock.
>
> E.g. if CPU0 and CPU1 are siblings and CPU1 is running a KVM guest with
> MSR_TEST_CTRL.split_lock=1, hitting an #AC on CPU0 in the host kernel will
> lead to suprious #AC faults and constant toggling of of the MSR.
>
> CPU0 CPU1
>
> split_lock=enabled
>
> #AC -> disabled
>
> VM-Enter -> enabled
>
> #AC -> disabled
>
> VM-Enter -> enabled
>
> #AC -> disabled
>
>
>
> My thought to handle this:
>
> - Remove the per-cpu cache.
>
> - Rework the atomic variable to differentiate between "disabled globally"
> and "disabled by kernel (on some CPUs)".
>
> - Modify the #AC handler to test/set the same atomic variable as the
> sysfs knob. This is the "disabled by kernel" flow.
>
> - Modify the debugfs/sysfs knob to only allow disabling split-lock
> detection. This is the "disabled globally" path, i.e. sends IPIs to
> clear MSR_TEST_CTRL.split_lock on all online CPUs.
>
> - Modify the resume/init flow to clear MSR_TEST_CTRL.split_lock if it's
> been disabled on *any* CPU via #AC or via the knob.
>
> - Modify the debugs/sysfs read function to either print the raw atomic
> variable, or differentiate between "enabled", "disabled globally" and
> "disabled by kernel".
>
> - Remove KVM loading of MSR_TEST_CTRL, i.e. KVM *never* writes the CPU's
> actual MSR_TEST_CTRL. KVM still emulates MSR_TEST_CTRL so that the
> guest can do WRMSR and handle its own #AC faults, but KVM doesn't
> change the value in hardware.
>
> * Allowing guest to enable split-lock detection can induce #AC on
> the host after it has been explicitly turned off, e.g. the sibling
> hyperthread hits an #AC in the host kernel, or worse, causes a
> different process in the host to SIGBUS.
>
> * Allowing guest to disable split-lock detection opens up the host
> to DoS attacks.
>
> - KVM advertises split-lock detection to guest/userspace if and only if
> split_lock_detect_disabled is zero.
>
> - Add a pr_warn_once() in KVM that triggers if split locks are disabled
> after support has been advertised to a guest.
>
> Does this sound sane?
>
> The question at the forefront of my mind is: why not have the #AC handler
> send a fire-and-forget IPI to online CPUs to disable split-lock detection
> on all CPUs? Would the IPI be problematic? Globally disabling split-lock
> on any #AC would (marginally) simplify the code and would eliminate the
> oddity of userspace process (and KVM guest) #AC behavior varying based on
> the physical CPU it's running on.
>
>
> Something like:
>
> #define SPLIT_LOCK_DISABLED_IN_KERNEL BIT(0)
> #define SPLIT_LOCK_DISABLED_GLOBALLY BIT(1)
>
> static atomic_t split_lock_detect_disabled = ATOMIT_INIT(0);
>
> void split_lock_detect_ac(void)
> {
> lockdep_assert_irqs_disabled();
>
> /* Disable split lock detection on this CPU to avoid reentrant #AC. */
> wrmsrl(MSR_TEST_CTRL,
> rdmsrl(MSR_TEST_CTRL) & ~MSR_TEST_CTRL_SPLIT_LOCK_DETECT);
>
> /*
> * If split-lock detection has not been disabled, either by the kernel
> * or globally, record that it has been disabled by the kernel and
> * WARN. Guarding WARN with the atomic ensures only the first #AC due
> * to split-lock is logged, e.g. if multiple CPUs encounter #AC or if
> * #AC is retriggered by a perf context NMI that interrupts the
> * original WARN.
> */
> if (atomic_cmpxchg(&split_lock_detect_disabled, 0,
> SPLIT_LOCK_DISABLED_IN_KERNEL) == 0)
> WARN(1, "split lock operation detected\n");
> }
>
> static ssize_t split_lock_detect_wr(struct file *f, const char __user *user_buf,
> size_t count, loff_t *ppos)
> {
> int old;
>
> <parse or ignore input value?>
>
> old = atomic_fetch_or(SPLIT_LOCK_DISABLED_GLOBALLY,
> &split_lock_detect_disabled);
>
> /* Update MSR_TEST_CTRL unless split-lock was already disabled. */
> if (!(old & SPLIT_LOCK_DISABLED_GLOBALLY))
> on_each_cpu(split_lock_update, NULL, 1);
>
> return count;
> }
>
Hi Thomas,
Could you please have a look at Sean's proposal and give your opinion.
next prev parent reply other threads:[~2019-10-16 6:58 UTC|newest]
Thread overview: 85+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-18 22:41 [PATCH v9 00/17] x86/split_lock: Enable split lock detection Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 01/17] x86/common: Align cpu_caps_cleared and cpu_caps_set to unsigned long Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 02/17] drivers/net/b44: Align pwol_mask to unsigned long for better performance Fenghua Yu
2019-06-24 15:12 ` David Laight
2019-06-24 18:43 ` Paolo Bonzini
2019-06-18 22:41 ` [PATCH v9 03/17] x86/split_lock: Align x86_capability to unsigned long to avoid split locked access Fenghua Yu
2019-06-24 15:12 ` David Laight
2019-06-25 23:54 ` Fenghua Yu
2019-06-26 19:15 ` Thomas Gleixner
2019-06-18 22:41 ` [PATCH v9 04/17] x86/msr-index: Define MSR_IA32_CORE_CAP and split lock detection bit Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 05/17] x86/cpufeatures: Enumerate MSR_IA32_CORE_CAP Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 06/17] x86/split_lock: Enumerate split lock detection by MSR_IA32_CORE_CAP Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 07/17] x86/split_lock: Enumerate split lock detection on Icelake mobile processor Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 08/17] x86/split_lock: Define MSR TEST_CTL register Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 09/17] x86/split_lock: Handle #AC exception for split lock Fenghua Yu
2019-06-26 20:20 ` Thomas Gleixner
2019-06-26 20:36 ` Fenghua Yu
2019-06-26 21:47 ` Thomas Gleixner
2019-09-25 18:09 ` Sean Christopherson
2019-10-16 6:58 ` Xiaoyao Li [this message]
2019-10-16 9:29 ` Thomas Gleixner
2019-10-16 15:59 ` Sean Christopherson
2019-10-16 9:40 ` Paolo Bonzini
2019-10-16 9:47 ` Thomas Gleixner
2019-10-16 10:16 ` Paolo Bonzini
2019-10-16 11:23 ` Xiaoyao Li
2019-10-16 11:26 ` Paolo Bonzini
2019-10-16 13:13 ` Xiaoyao Li
2019-10-16 14:43 ` Thomas Gleixner
2019-10-16 15:37 ` Paolo Bonzini
2019-10-16 16:25 ` Xiaoyao Li
2019-10-16 16:38 ` Paolo Bonzini
2019-10-17 12:29 ` [RFD] x86/split_lock: Request to Intel Thomas Gleixner
2019-10-17 17:23 ` Sean Christopherson
2019-10-17 21:31 ` Thomas Gleixner
2019-10-17 23:38 ` Sean Christopherson
2019-10-17 23:28 ` Luck, Tony
2019-10-18 10:45 ` David Laight
2019-10-18 21:03 ` hpa
2019-10-18 2:36 ` Xiaoyao Li
2019-10-18 9:02 ` Thomas Gleixner
2019-10-18 10:20 ` Xiaoyao Li
2019-10-18 10:43 ` Peter Zijlstra
2019-10-16 11:49 ` [PATCH v9 09/17] x86/split_lock: Handle #AC exception for split lock Thomas Gleixner
2019-10-16 11:58 ` Paolo Bonzini
2019-10-16 13:51 ` Xiaoyao Li
2019-10-16 14:08 ` Paolo Bonzini
2019-10-16 14:14 ` David Laight
2019-10-16 15:03 ` Thomas Gleixner
2019-10-16 15:41 ` Sean Christopherson
2019-10-16 15:43 ` Paolo Bonzini
2019-10-16 16:23 ` Sean Christopherson
2019-10-16 17:42 ` Sean Christopherson
2019-10-17 1:23 ` Xiaoyao Li
2019-10-21 13:06 ` Paolo Bonzini
2019-10-21 13:03 ` Paolo Bonzini
2019-10-21 13:02 ` Paolo Bonzini
2019-10-16 14:50 ` Thomas Gleixner
2019-06-18 22:41 ` [PATCH v9 10/17] kvm/x86: Emulate MSR IA32_CORE_CAPABILITY Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 11/17] kvm/vmx: Emulate MSR TEST_CTL Fenghua Yu
2019-06-27 2:24 ` Xiaoyao Li
2019-06-27 7:12 ` Thomas Gleixner
2019-06-27 7:58 ` Xiaoyao Li
2019-06-27 12:11 ` Thomas Gleixner
2019-06-27 12:22 ` Xiaoyao Li
2019-06-18 22:41 ` [PATCH v9 12/17] x86/split_lock: Enable split lock detection by default Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 13/17] x86/split_lock: Disable split lock detection by kernel parameter "nosplit_lock_detect" Fenghua Yu
2019-06-26 20:34 ` Thomas Gleixner
2019-06-26 20:37 ` Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 14/17] x86/split_lock: Add a debugfs interface to enable/disable split lock detection during run time Fenghua Yu
2019-06-26 21:37 ` Thomas Gleixner
2019-06-18 22:41 ` [PATCH v9 15/17] x86/split_lock: Add documentation for split lock detection interface Fenghua Yu
2019-06-26 21:51 ` Thomas Gleixner
2019-06-18 22:41 ` [PATCH v9 16/17] x86/split_lock: Reorganize few header files in order to call WARN_ON_ONCE() in atomic bit ops Fenghua Yu
2019-06-18 22:41 ` [PATCH v9 17/17] x86/split_lock: Warn on unaligned address in atomic bit operations Fenghua Yu
2019-06-26 22:00 ` Thomas Gleixner
2019-09-16 22:39 ` [PATCH 0/3] Fix some 4-byte vs. 8-byte alignment issues Tony Luck
2019-09-16 22:39 ` [PATCH 1/3] x86/common: Align cpu_caps_cleared and cpu_caps_set to unsigned long Tony Luck
2019-11-15 19:26 ` [tip: x86/cpu] x86/cpu: " tip-bot2 for Fenghua Yu
2019-09-16 22:39 ` [PATCH 2/3] drivers/net/b44: Align pwol_mask to unsigned long for better performance Tony Luck
2019-09-16 22:39 ` [PATCH 3/3] x86/split_lock: Align the x86_capability array to size of unsigned long Tony Luck
2019-09-17 8:29 ` David Laight
2019-09-17 19:14 ` Luck, Tony
2019-09-18 8:54 ` David Laight
2019-11-15 19:26 ` [tip: x86/cpu] x86/cpu: " tip-bot2 for Fenghua Yu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bc8f5850-ba7d-45b7-a30b-5560764edcc8@intel.com \
--to=xiaoyao.li@intel.com \
--cc=akpm@linux-foundation.org \
--cc=ashok.raj@intel.com \
--cc=bp@alien8.de \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@intel.com \
--cc=fenghua.yu@intel.com \
--cc=hpa@zytor.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=pbonzini@redhat.com \
--cc=peterz@infradead.org \
--cc=ravi.v.shankar@intel.com \
--cc=rkrcmar@redhat.com \
--cc=sai.praneeth.prakhya@intel.com \
--cc=sean.j.christopherson@intel.com \
--cc=tglx@linutronix.de \
--cc=tony.luck@intel.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).