From: Hao Peng <flyingpenghao@gmail.com>
To: Sean Christopherson <seanjc@google.com>
Cc: pbonzini@redhat.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH] kvm: x86: Improve virtual machine startup performance
Date: Thu, 3 Mar 2022 10:56:59 +0800 [thread overview]
Message-ID: <CAPm50aLJ51mm9JVpTMQCkNENX_9-Do5UeH5zxu-5byOcOFsJBg@mail.gmail.com> (raw)
In-Reply-To: <YiAZ3wTICeLTVnJz@google.com>
On Thu, Mar 3, 2022 at 9:29 AM Sean Christopherson <seanjc@google.com> wrote:
>
> On Wed, Mar 02, 2022, Hao Peng wrote:
> > On Wed, Mar 2, 2022 at 1:54 AM Sean Christopherson <seanjc@google.com> wrote:
> > >
> > > On Tue, Mar 01, 2022, Peng Hao wrote:
> > > > From: Peng Hao <flyingpeng@tencent.com>
> > > >
> > > > vcpu 0 will repeatedly enter/exit the smm state during the startup
> > > > phase, and kvm_init_mmu will be called repeatedly during this process.
> > > > There are parts of the mmu initialization code that do not need to be
> > > > modified after the first initialization.
> > > >
> > > > Statistics on my server, vcpu0 when starting the virtual machine
> > > > Calling kvm_init_mmu more than 600 times (due to smm state switching).
> > > > The patch can save about 36 microseconds in total.
> > > >
> > > > Signed-off-by: Peng Hao <flyingpeng@tencent.com>
> > > > ---
> > > > @@ -5054,7 +5059,7 @@ void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu)
> > > > void kvm_mmu_reset_context(struct kvm_vcpu *vcpu)
> > > > {
> > > > kvm_mmu_unload(vcpu);
> > > > - kvm_init_mmu(vcpu);
> > > > + kvm_init_mmu(vcpu, false);
> > >
> > > This is wrong, kvm_mmu_reset_context() is the "big hammer" and is expected to
> > > unconditionally get the MMU to a known good state. E.g. failure to initialize
> > > means this code:
> > >
> > > context->shadow_root_level = kvm_mmu_get_tdp_level(vcpu);
> > >
> > > will not update the shadow_root_level as expected in response to userspace changing
> > > guest.MAXPHYADDR in such a way that KVM enables/disables 5-level paging.
> > >
> > Thanks for pointing this out. However, other than shadow_root_level,
> > other fields of context will not
> > change during the entire operation, such as
> > page_fault/sync_page/direct_map and so on under
> > the condition of tdp_mmu.
> > Is this patch still viable after careful confirmation of the fields
> > that won't be modified?
>
> No, passing around the "init" flag is a hack.
>
> But, we can achieve what you want simply by initializing the constant data once
> per vCPU. There's a _lot_ of state that is constant for a given MMU now that KVM
> uses separate MMUs for L1 vs. L2 when TDP is enabled. I should get patches posted
> tomorrow, just need to test (famous last words).
>
> Also, based on the number of SMM transitions, I'm guessing you're using SeaBIOS.
> Have you tried disabling CONFIG_CALL32_SMM, or CONFIG_USE_SMM altogether? That
> might be an even better way to improve performance in your environment.
>
Both options are disabled in guest.
> Last question, do you happen to know why eliminating this code shaves 36us? The
> raw writes don't seem like they'd take that long. Maybe the writes to function
> pointers trigger stalls or mispredicts or something? If you don't have an easy
> answer, don't bother investigating, I'm just curious.
I'm guessing it's because of the cache. At first, I wanted to replace
it with memcpy, if the modified fields are continuous enough, I can
use instructions such as erms/fsrm.
next prev parent reply other threads:[~2022-03-03 2:58 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-01 6:37 [PATCH] kvm: x86: Improve virtual machine startup performance Peng Hao
2022-03-01 17:54 ` Sean Christopherson
2022-03-02 1:30 ` Hao Peng
2022-03-03 1:29 ` Sean Christopherson
2022-03-03 2:56 ` Hao Peng [this message]
2022-03-08 2:03 ` Sean Christopherson
2022-03-08 8:13 ` Paolo Bonzini
2022-03-08 16:04 ` Sean Christopherson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAPm50aLJ51mm9JVpTMQCkNENX_9-Do5UeH5zxu-5byOcOFsJBg@mail.gmail.com \
--to=flyingpenghao@gmail.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=seanjc@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).