From: Sean Christopherson <seanjc@google.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: Marc Zyngier <maz@kernel.org>,
Huacai Chen <chenhuacai@kernel.org>,
Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>,
Paul Mackerras <paulus@ozlabs.org>,
James Morse <james.morse@arm.com>,
Julien Thierry <julien.thierry.kdev@gmail.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
Vitaly Kuznetsov <vkuznets@redhat.com>,
Wanpeng Li <wanpengli@tencent.com>,
Jim Mattson <jmattson@google.com>, Joerg Roedel <joro@8bytes.org>,
linux-arm-kernel@lists.infradead.org,
kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org,
kvm@vger.kernel.org, kvm-ppc@vger.kernel.org,
linux-kernel@vger.kernel.org, Ben Gardon <bgardon@google.com>
Subject: Re: [PATCH 16/18] KVM: Don't take mmu_lock for range invalidation unless necessary
Date: Wed, 31 Mar 2021 16:41:21 +0000 [thread overview]
Message-ID: <YGSmMeSOPcjxRwf6@google.com> (raw)
In-Reply-To: <6e7dc7d0-f5dc-85d9-1c50-d23b761b5ff3@redhat.com>
On Wed, Mar 31, 2021, Paolo Bonzini wrote:
> On 26/03/21 03:19, Sean Christopherson wrote:
> > + /*
> > + * Reset the lock used to prevent memslot updates between MMU notifier
> > + * range_start and range_end. At this point no more MMU notifiers will
> > + * run, but the lock could still be held if KVM's notifier was removed
> > + * between range_start and range_end. No threads can be waiting on the
> > + * lock as the last reference on KVM has been dropped. If the lock is
> > + * still held, freeing memslots will deadlock.
> > + */
> > + init_rwsem(&kvm->mmu_notifier_slots_lock);
>
> I was going to say that this is nasty,
Heh, I still think it's nasty.
> then I noticed that
> mmu_notifier_unregister uses SRCU to ensure completion of concurrent calls
> to the MMU notifier. So I guess it's fine, but it's better to point it out:
>
> /*
> * At this point no more MMU notifiers will run and pending
> * calls to range_start have completed, but the lock would
> * still be held and never released if the MMU notifier was
> * removed between range_start and range_end. Since the last
> * reference to the struct kvm has been dropped, no threads can
> * be waiting on the lock, but we might still end up taking it
> * when freeing memslots in kvm_arch_destroy_vm. Reset the lock
> * to avoid deadlocks.
> */
>
> That said, the easiest way to avoid this would be to always update
> mmu_notifier_count.
Updating mmu_notifier_count requires taking mmu_lock, which would defeat the
purpose of these shenanigans. I think it could be made atomic, since mmu_lock
would be taken before the elevated count _must_ be visible, but that would
break the mmu_notifier_range_{start,end} optimization that was recently added.
Or did I completely misunderstand what you're suggesting?
> I don't mind the rwsem, but at least I suggest that you
> split the patch in two---the first one keeping the mmu_notifier_count update
> unconditional, and the second one introducing the rwsem and the on_lock
> function kvm_inc_notifier_count. Please document the new lock in
> Documentation/virt/kvm/locking.rst too.
Note, will update docs.
> Also, related to the first part of the series, perhaps you could structure
> the series in a slightly different way:
>
> 1) introduce the HVA walking API in common code, complete with on_lock and
> patch 15, so that you can use on_lock to increase mmu_notifier_seq
>
> 2) then migrate all architectures including x86 to the new API
>
> IOW, first half of patch 10 and all of patch 15; then the second half of
> patch 10; then patches 11-14.
>
> > +#if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
> > + down_write(&kvm->mmu_notifier_slots_lock);
> > +#endif
> > rcu_assign_pointer(kvm->memslots[as_id], slots);
> > +#if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
> > + up_write(&kvm->mmu_notifier_slots_lock);
> > +#endif
>
> Please do this unconditionally, the cost is minimal if the rwsem is not
> contended (as is the case if the architecture doesn't use MMU notifiers at
> all).
It's not the cost, it's that mmu_notifier_slots_lock doesn't exist. That's an
easily solved problem, but then the lock wouldn't be initialized since
kvm_init_mmu_notifier() is a nop. That's again easy to solve, but IMO would
look rather weird. I guess the counter argument is that __kvm_memslots()
wouldn't need #ifdeffery.
These are the to ideas I've come up with:
Option 1:
static int kvm_init_mmu_notifier(struct kvm *kvm)
{
init_rwsem(&kvm->mmu_notifier_slots_lock);
#if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
kvm->mmu_notifier.ops = &kvm_mmu_notifier_ops;
return mmu_notifier_register(&kvm->mmu_notifier, current->mm);
#else
return 0;
#endif
}
Option 2:
kvm_mmu_notifier_lock(kvm);
rcu_assign_pointer(kvm->memslots[as_id], slots);
kvm_mmu_notifier_unlock(kvm);
next prev parent reply other threads:[~2021-03-31 16:42 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-26 2:19 [PATCH 00/18] KVM: Consolidate and optimize MMU notifiers Sean Christopherson
2021-03-26 2:19 ` [PATCH 01/18] KVM: x86/mmu: Coalesce TDP MMU TLB flushes when zapping collapsible SPTEs Sean Christopherson
2021-03-26 2:19 ` [PATCH 02/18] KVM: x86/mmu: Move flushing for "slot" handlers to caller for legacy MMU Sean Christopherson
2021-03-26 2:19 ` [PATCH 03/18] KVM: x86/mmu: Coalesce TLB flushes when zapping collapsible SPTEs Sean Christopherson
2021-03-26 2:19 ` [PATCH 04/18] KVM: x86/mmu: Coalesce TLB flushes across address spaces for gfn range zap Sean Christopherson
2021-03-26 2:19 ` [PATCH 05/18] KVM: x86/mmu: Pass address space ID to __kvm_tdp_mmu_zap_gfn_range() Sean Christopherson
2021-03-26 2:19 ` [PATCH 06/18] KVM: x86/mmu: Pass address space ID to TDP MMU root walkers Sean Christopherson
2021-03-26 2:19 ` [PATCH 07/18] KVM: x86/mmu: Use leaf-only loop for walking TDP SPTEs when changing SPTE Sean Christopherson
2021-03-26 2:19 ` [PATCH 08/18] KVM: Move prototypes for MMU notifier callbacks to generic code Sean Christopherson
2021-03-26 2:19 ` [PATCH 09/18] KVM: Move arm64's MMU notifier trace events " Sean Christopherson
2021-03-26 2:19 ` [PATCH 10/18] KVM: Move x86's MMU notifier memslot walkers " Sean Christopherson
2021-03-31 7:52 ` Paolo Bonzini
2021-03-31 16:20 ` Sean Christopherson
2021-03-31 16:36 ` Paolo Bonzini
2021-03-26 2:19 ` [PATCH 11/18] KVM: arm64: Convert to the gfn-based MMU notifier callbacks Sean Christopherson
2021-03-26 2:19 ` [PATCH 12/18] KVM: MIPS/MMU: " Sean Christopherson
2021-03-31 7:41 ` Paolo Bonzini
2021-03-26 2:19 ` [PATCH 13/18] KVM: PPC: " Sean Christopherson
2021-03-26 2:19 ` [PATCH 14/18] KVM: Kill off the old hva-based " Sean Christopherson
2021-03-26 2:19 ` [PATCH 15/18] KVM: Take mmu_lock when handling MMU notifier iff the hva hits a memslot Sean Christopherson
2021-03-26 2:19 ` [PATCH 16/18] KVM: Don't take mmu_lock for range invalidation unless necessary Sean Christopherson
2021-03-31 7:52 ` Paolo Bonzini
2021-03-31 8:35 ` Paolo Bonzini
2021-03-31 16:41 ` Sean Christopherson [this message]
2021-03-31 16:47 ` Paolo Bonzini
2021-03-31 19:47 ` Sean Christopherson
2021-03-31 20:42 ` Paolo Bonzini
2021-03-31 21:05 ` Sean Christopherson
2021-03-31 21:22 ` Sean Christopherson
2021-03-31 21:36 ` Paolo Bonzini
2021-03-31 21:35 ` Paolo Bonzini
2021-03-31 21:47 ` Sean Christopherson
2021-03-31 20:15 ` Sean Christopherson
2021-03-31 20:30 ` Paolo Bonzini
2021-03-31 20:52 ` Sean Christopherson
2021-03-31 21:00 ` Paolo Bonzini
2021-03-26 2:19 ` [PATCH 17/18] KVM: x86/mmu: Allow yielding during MMU notifier unmap/zap, if possible Sean Christopherson
2021-03-26 2:19 ` [PATCH 18/18] KVM: x86/mmu: Drop trace_kvm_age_page() tracepoint Sean Christopherson
2021-03-30 18:32 ` [PATCH 00/18] KVM: Consolidate and optimize MMU notifiers Ben Gardon
2021-03-30 19:48 ` Paolo Bonzini
2021-03-30 19:58 ` Sean Christopherson
2021-03-31 7:57 ` Paolo Bonzini
2021-03-31 9:34 ` Marc Zyngier
2021-03-31 9:41 ` Paolo Bonzini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YGSmMeSOPcjxRwf6@google.com \
--to=seanjc@google.com \
--cc=aleksandar.qemu.devel@gmail.com \
--cc=bgardon@google.com \
--cc=chenhuacai@kernel.org \
--cc=james.morse@arm.com \
--cc=jmattson@google.com \
--cc=joro@8bytes.org \
--cc=julien.thierry.kdev@gmail.com \
--cc=kvm-ppc@vger.kernel.org \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mips@vger.kernel.org \
--cc=maz@kernel.org \
--cc=paulus@ozlabs.org \
--cc=pbonzini@redhat.com \
--cc=suzuki.poulose@arm.com \
--cc=vkuznets@redhat.com \
--cc=wanpengli@tencent.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).