From: Zeng Guang <guang.zeng@intel.com>
To: Wanpeng Li <kernellwp@gmail.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>,
Sean Christopherson <seanjc@google.com>,
Vitaly Kuznetsov <vkuznets@redhat.com>,
Wanpeng Li <wanpengli@tencent.com>,
Jim Mattson <jmattson@google.com>, Joerg Roedel <joro@8bytes.org>,
kvm <kvm@vger.kernel.org>,
Dave Hansen <dave.hansen@linux.intel.com>,
"Luck, Tony" <tony.luck@intel.com>,
Kan Liang <kan.liang@linux.intel.com>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
"H. Peter Anvin" <hpa@zytor.com>,
Kim Phillips <kim.phillips@amd.com>,
Jarkko Sakkinen <jarkko@kernel.org>,
Jethro Beekman <jethro@fortanix.com>,
"Huang, Kai" <kai.huang@intel.com>,
the arch/x86 maintainers <x86@kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
"Hu, Robert" <robert.hu@intel.com>,
"Gao, Chao" <chao.gao@intel.com>
Subject: Re: [PATCH 0/5] IPI virtualization support for VM
Date: Fri, 23 Jul 2021 14:15:53 +0800 [thread overview]
Message-ID: <d0fa2542-8c64-f6ab-a864-b71f64d50304@intel.com> (raw)
In-Reply-To: <CANRm+CyinezdL4udNv1fkCymCUdOjG7wjBPKsbcMTVw0pAbcjA@mail.gmail.com>
On 7/19/2021 3:37 PM, Wanpeng Li wrote:
> On Mon, 19 Jul 2021 at 15:26, Zeng Guang <guang.zeng@intel.com> wrote:
>> On 7/16/2021 5:25 PM, Wanpeng Li wrote:
>>> On Fri, 16 Jul 2021 at 15:14, Zeng Guang <guang.zeng@intel.com> wrote:
>>>> Current IPI process in guest VM will virtualize the writing to interrupt
>>>> command register(ICR) of the local APIC which will cause VM-exit anyway
>>>> on source vCPU. Frequent VM-exit could induce much overhead accumulated
>>>> if running IPI intensive task.
>>>>
>>>> IPI virtualization as a new VT-x feature targets to eliminate VM-exits
>>>> when issuing IPI on source vCPU. It introduces a new VM-execution
>>>> control - "IPI virtualization"(bit4) in the tertiary processor-based
>>>> VM-exection controls and a new data structure - "PID-pointer table
>>>> address" and "Last PID-pointer index" referenced by the VMCS. When "IPI
>>>> virtualization" is enabled, processor emulateds following kind of writes
>>>> to APIC registers that would send IPIs, moreover without causing VM-exits.
>>>> - Memory-mapped ICR writes
>>>> - MSR-mapped ICR writes
>>>> - SENDUIPI execution
>>>>
>>>> This patch series implement IPI virtualization support in KVM.
>>>>
>>>> Patches 1-3 add tertiary processor-based VM-execution support
>>>> framework.
>>>>
>>>> Patch 4 implement interrupt dispatch support in x2APIC mode with
>>>> APIC-write VM exit. In previous platform, no CPU would produce
>>>> APIC-write VM exit with exit qulification 300H when the "virtual x2APIC
>>>> mode" VM-execution control was 1.
>>>>
>>>> Patch 5 implement IPI virtualization related function including
>>>> feature enabling through tertiary processor-based VM-execution in
>>>> various scenario of VMCS configuration, PID table setup in vCPU creation
>>>> and vCPU block consideration.
>>>>
>>>> Document for IPI virtualization is now available at the latest "Intel
>>>> Architecture Instruction Set Extensions Programming Reference".
>>>>
>>>> Document Link:
>>>> https://software.intel.com/content/www/us/en/develop/download/intel-architecture-instruction-set-extensions-programming-reference.html
>>>>
>>>> We did experiment to measure average time sending IPI from source vCPU
>>>> to the target vCPU completing the IPI handling by kvm unittest w/ and
>>>> w/o IPI virtualization. When IPI virtualizatin enabled, it will reduce
>>>> 22.21% and 15.98% cycles comsuming in xAPIC mode and x2APIC mode
>>>> respectly.
>>>>
>>>> KMV unittest:vmexit/ipi, 2 vCPU, AP runs without halt to ensure no VM
>>>> exit impact on target vCPU.
>>>>
>>>> Cycles of IPI
>>>> xAPIC mode x2APIC mode
>>>> test w/o IPIv w/ IPIv w/o IPIv w/ IPIv
>>>> 1 6106 4816 4265 3768
>>>> 2 6244 4656 4404 3546
>>>> 3 6165 4658 4233 3474
>>>> 4 5992 4710 4363 3430
>>>> 5 6083 4741 4215 3551
>>>> 6 6238 4904 4304 3547
>>>> 7 6164 4617 4263 3709
>>>> 8 5984 4763 4518 3779
>>>> 9 5931 4712 4645 3667
>>>> 10 5955 4530 4332 3724
>>>> 11 5897 4673 4283 3569
>>>> 12 6140 4794 4178 3598
>>>> 13 6183 4728 4363 3628
>>>> 14 5991 4994 4509 3842
>>>> 15 5866 4665 4520 3739
>>>> 16 6032 4654 4229 3701
>>>> 17 6050 4653 4185 3726
>>>> 18 6004 4792 4319 3746
>>>> 19 5961 4626 4196 3392
>>>> 20 6194 4576 4433 3760
>>>>
>>>> Average cycles 6059 4713.1 4337.85 3644.8
>>>> %Reduction -22.21% -15.98%
>>> Commit a9ab13ff6e (KVM: X86: Improve latency for single target IPI
>>> fastpath) mentioned that the whole ipi fastpath feature reduces the
>>> latency from 4238 to 3293 around 22.3% on SKX server, why your IPIv
>>> hardware acceleration is worse than software emulation? In addition,
>> Actually this performance data was measured on the basis of fastpath
>> optimization while cpu runs at base frequency.
>>
>> As a result, IPI virtualization could have extra 15.98% cost reduction
>> over IPI fastpath process in x2apic mode.
> I observed that adaptive advance lapic timer and adaptive halt-polling
> will influence kvm-unit-tests/vmexit.flat IPI testing score, could you
> post the score after disabling these features as commit a9ab13ff6e
> (KVM: X86: Improve latency for single target IPI fastpath) mentioned?
> In addition, please post the hackbench(./hackbench -l 1000000) and ipi
> microbenchmark scores.
We modified unittest to make AP runing with idle loop instead of hlt .
This eliminates the impact
from adaptive halt-polling. So far we don't observe the influence from
adaptive advance lapic timer
by test either. vmexit/ipi test should not involve lapic timer.
We post the hackbench and ipi microbenchmark score in patch V2 for your
reference.
Thanks.
>
> Wanpeng
prev parent reply other threads:[~2021-07-23 6:16 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-16 6:48 [PATCH 0/5] IPI virtualization support for VM Zeng Guang
2021-07-16 6:48 ` [PATCH 1/6] x86/feat_ctl: Add new VMX feature, Tertiary VM-Execution control Zeng Guang
2021-07-28 23:44 ` Sean Christopherson
2021-07-29 15:56 ` Paolo Bonzini
2021-08-02 8:22 ` Zeng Guang
2021-08-02 16:20 ` Sean Christopherson
2021-07-16 6:48 ` [PATCH 2/6] KVM: VMX: Extend BUILD_CONTROLS_SHADOW macro to support 64-bit variation Zeng Guang
2021-07-16 6:48 ` [PATCH 3/6] KVM: VMX: Detect Tertiary VM-Execution control when setup VMCS config Zeng Guang
2021-07-29 0:03 ` Sean Christopherson
2021-08-02 6:59 ` Zeng Guang
2021-07-16 6:48 ` [PATCH 4/6] KVM: VMX: dump_vmcs() reports tertiary_exec_control field as well Zeng Guang
2021-07-16 6:48 ` [PATCH 5/6] KVM: x86: Support interrupt dispatch in x2APIC mode with APIC-write VM exit Zeng Guang
2021-07-16 6:48 ` [PATCH 6/6] KVM: VMX: enable IPI virtualization Zeng Guang
2021-07-16 9:52 ` Paolo Bonzini
2021-07-17 3:55 ` Zeng Guang
2021-07-18 20:32 ` Paolo Bonzini
2021-07-19 12:38 ` Zeng Guang
2021-07-19 13:58 ` Paolo Bonzini
2021-07-20 1:07 ` Zeng Guang
2021-07-19 13:16 ` Zeng Guang
2021-07-16 9:25 ` [PATCH 0/5] IPI virtualization support for VM Wanpeng Li
2021-07-17 1:46 ` Zeng Guang
2021-07-19 7:26 ` Zeng Guang
2021-07-19 7:37 ` Wanpeng Li
2021-07-23 6:15 ` Zeng Guang [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d0fa2542-8c64-f6ab-a864-b71f64d50304@intel.com \
--to=guang.zeng@intel.com \
--cc=bp@alien8.de \
--cc=chao.gao@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=hpa@zytor.com \
--cc=jarkko@kernel.org \
--cc=jethro@fortanix.com \
--cc=jmattson@google.com \
--cc=joro@8bytes.org \
--cc=kai.huang@intel.com \
--cc=kan.liang@linux.intel.com \
--cc=kernellwp@gmail.com \
--cc=kim.phillips@amd.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=pbonzini@redhat.com \
--cc=robert.hu@intel.com \
--cc=seanjc@google.com \
--cc=tglx@linutronix.de \
--cc=tony.luck@intel.com \
--cc=vkuznets@redhat.com \
--cc=wanpengli@tencent.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).