* About the performance of hyper-v
@ 2021-05-19 14:00 ` Liang Li
0 siblings, 0 replies; 14+ messages in thread
From: Liang Li @ 2021-05-19 14:00 UTC (permalink / raw)
To: vkuznets; +Cc: qemu-devel, kvm, Tianyu.Lan
[resend for missing cc]
Hi Vitaly,
I found a case that the virtualization overhead was almost doubled
when turning on Hper-v related features compared to that without any
no hyper-v feature. It happens when running a 3D game in windows
guest in qemu kvm environment.
By investigation, I found there are a lot of IPIs triggered by guest,
when turning on the hyer-v related features including stimer, for the
apicv is turned off, at least two vm exits are needed for processing a
single IPI.
perf stat will show something like below [recorded for 5 seconds]
---------
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max
Time Avg time
EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
65.42us 2.34us ( +- 0.11% )
MSR_WRITE 238932 30.33% 23.07% 0.48us
41.05us 1.56us ( +- 0.14% )
Total Samples:787803, Total events handled time:1611193.84us.
I tried turning off hyper-v for the same workload and repeat the test,
the overall virtualization overhead reduced by about of 50%:
-------
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max
Time Avg time
APIC_WRITE 255152 74.43% 50.72% 0.49us
50.01us 1.42us ( +- 0.14% )
EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
686.05us 7.27us ( +- 0.43% )
DR_ACCESS 35003 10.21% 4.64% 0.32us
40.03us 0.95us ( +- 0.32% )
EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
57.38us 2.25us ( +- 1.42% )
Total Samples:342788, Total events handled time:715695.62us.
For this scenario, hyper-v works really bad. stimer works better
than hpet, but on the other hand, it relies on SynIC which has
negative effects for IPI intensive workloads.
Do you have any plans for improvement?
Thanks!
Liang
^ permalink raw reply [flat|nested] 14+ messages in thread
* About the performance of hyper-v
@ 2021-05-19 14:00 ` Liang Li
0 siblings, 0 replies; 14+ messages in thread
From: Liang Li @ 2021-05-19 14:00 UTC (permalink / raw)
To: vkuznets; +Cc: Tianyu.Lan, qemu-devel, kvm
[resend for missing cc]
Hi Vitaly,
I found a case that the virtualization overhead was almost doubled
when turning on Hper-v related features compared to that without any
no hyper-v feature. It happens when running a 3D game in windows
guest in qemu kvm environment.
By investigation, I found there are a lot of IPIs triggered by guest,
when turning on the hyer-v related features including stimer, for the
apicv is turned off, at least two vm exits are needed for processing a
single IPI.
perf stat will show something like below [recorded for 5 seconds]
---------
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max
Time Avg time
EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
65.42us 2.34us ( +- 0.11% )
MSR_WRITE 238932 30.33% 23.07% 0.48us
41.05us 1.56us ( +- 0.14% )
Total Samples:787803, Total events handled time:1611193.84us.
I tried turning off hyper-v for the same workload and repeat the test,
the overall virtualization overhead reduced by about of 50%:
-------
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max
Time Avg time
APIC_WRITE 255152 74.43% 50.72% 0.49us
50.01us 1.42us ( +- 0.14% )
EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
686.05us 7.27us ( +- 0.43% )
DR_ACCESS 35003 10.21% 4.64% 0.32us
40.03us 0.95us ( +- 0.32% )
EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
57.38us 2.25us ( +- 1.42% )
Total Samples:342788, Total events handled time:715695.62us.
For this scenario, hyper-v works really bad. stimer works better
than hpet, but on the other hand, it relies on SynIC which has
negative effects for IPI intensive workloads.
Do you have any plans for improvement?
Thanks!
Liang
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
2021-05-19 14:00 ` Liang Li
@ 2021-05-19 14:22 ` Vitaly Kuznetsov
-1 siblings, 0 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2021-05-19 14:22 UTC (permalink / raw)
To: Liang Li; +Cc: qemu-devel, kvm, Tianyu.Lan
Liang Li <liliang324@gmail.com> writes:
> [resend for missing cc]
>
> Hi Vitaly,
>
> I found a case that the virtualization overhead was almost doubled
> when turning on Hper-v related features compared to that without any
> no hyper-v feature. It happens when running a 3D game in windows
> guest in qemu kvm environment.
>
> By investigation, I found there are a lot of IPIs triggered by guest,
> when turning on the hyer-v related features including stimer, for the
> apicv is turned off, at least two vm exits are needed for processing a
> single IPI.
>
>
> perf stat will show something like below [recorded for 5 seconds]
>
> ---------
>
> Analyze events for all VMs, all VCPUs:
> VM-EXIT Samples Samples% Time% Min Time Max
> Time Avg time
> EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
> 65.42us 2.34us ( +- 0.11% )
> MSR_WRITE 238932 30.33% 23.07% 0.48us
> 41.05us 1.56us ( +- 0.14% )
>
> Total Samples:787803, Total events handled time:1611193.84us.
>
> I tried turning off hyper-v for the same workload and repeat the test,
> the overall virtualization overhead reduced by about of 50%:
>
> -------
>
> Analyze events for all VMs, all VCPUs:
>
> VM-EXIT Samples Samples% Time% Min Time Max
> Time Avg time
> APIC_WRITE 255152 74.43% 50.72% 0.49us
> 50.01us 1.42us ( +- 0.14% )
> EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
> 686.05us 7.27us ( +- 0.43% )
> DR_ACCESS 35003 10.21% 4.64% 0.32us
> 40.03us 0.95us ( +- 0.32% )
> EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
> 57.38us 2.25us ( +- 1.42% )
>
> Total Samples:342788, Total events handled time:715695.62us.
>
> For this scenario, hyper-v works really bad. stimer works better
> than hpet, but on the other hand, it relies on SynIC which has
> negative effects for IPI intensive workloads.
> Do you have any plans for improvement?
>
Hey,
the above can be caused by the fact that when 'hv-synic' is enabled, KVM
automatically disables APICv and this can explain the overhead and the
fact that you're seeing more vmexits. KVM disables APICv because SynIC's
'AutoEOI' feature is incompatible with it. We can, however, tell Windows
to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
inhibit APICv if the recommendation was ignored. This is implemented in
the following KVM patch series:
https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
It will, however, require a new 'hv-something' flag to QEMU. For now, it
can be tested with 'hv-passthrough'.
It would be great if you could give it a spin!
--
Vitaly
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
@ 2021-05-19 14:22 ` Vitaly Kuznetsov
0 siblings, 0 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2021-05-19 14:22 UTC (permalink / raw)
To: Liang Li; +Cc: Tianyu.Lan, qemu-devel, kvm
Liang Li <liliang324@gmail.com> writes:
> [resend for missing cc]
>
> Hi Vitaly,
>
> I found a case that the virtualization overhead was almost doubled
> when turning on Hper-v related features compared to that without any
> no hyper-v feature. It happens when running a 3D game in windows
> guest in qemu kvm environment.
>
> By investigation, I found there are a lot of IPIs triggered by guest,
> when turning on the hyer-v related features including stimer, for the
> apicv is turned off, at least two vm exits are needed for processing a
> single IPI.
>
>
> perf stat will show something like below [recorded for 5 seconds]
>
> ---------
>
> Analyze events for all VMs, all VCPUs:
> VM-EXIT Samples Samples% Time% Min Time Max
> Time Avg time
> EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
> 65.42us 2.34us ( +- 0.11% )
> MSR_WRITE 238932 30.33% 23.07% 0.48us
> 41.05us 1.56us ( +- 0.14% )
>
> Total Samples:787803, Total events handled time:1611193.84us.
>
> I tried turning off hyper-v for the same workload and repeat the test,
> the overall virtualization overhead reduced by about of 50%:
>
> -------
>
> Analyze events for all VMs, all VCPUs:
>
> VM-EXIT Samples Samples% Time% Min Time Max
> Time Avg time
> APIC_WRITE 255152 74.43% 50.72% 0.49us
> 50.01us 1.42us ( +- 0.14% )
> EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
> 686.05us 7.27us ( +- 0.43% )
> DR_ACCESS 35003 10.21% 4.64% 0.32us
> 40.03us 0.95us ( +- 0.32% )
> EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
> 57.38us 2.25us ( +- 1.42% )
>
> Total Samples:342788, Total events handled time:715695.62us.
>
> For this scenario, hyper-v works really bad. stimer works better
> than hpet, but on the other hand, it relies on SynIC which has
> negative effects for IPI intensive workloads.
> Do you have any plans for improvement?
>
Hey,
the above can be caused by the fact that when 'hv-synic' is enabled, KVM
automatically disables APICv and this can explain the overhead and the
fact that you're seeing more vmexits. KVM disables APICv because SynIC's
'AutoEOI' feature is incompatible with it. We can, however, tell Windows
to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
inhibit APICv if the recommendation was ignored. This is implemented in
the following KVM patch series:
https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
It will, however, require a new 'hv-something' flag to QEMU. For now, it
can be tested with 'hv-passthrough'.
It would be great if you could give it a spin!
--
Vitaly
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
2021-05-19 14:22 ` Vitaly Kuznetsov
@ 2021-05-21 8:22 ` Liang Li
-1 siblings, 0 replies; 14+ messages in thread
From: Liang Li @ 2021-05-21 8:22 UTC (permalink / raw)
To: Vitaly Kuznetsov; +Cc: qemu-devel, kvm, Tianyu.Lan
> > Hi Vitaly,
> >
> > I found a case that the virtualization overhead was almost doubled
> > when turning on Hper-v related features compared to that without any
> > no hyper-v feature. It happens when running a 3D game in windows
> > guest in qemu kvm environment.
> >
> > By investigation, I found there are a lot of IPIs triggered by guest,
> > when turning on the hyer-v related features including stimer, for the
> > apicv is turned off, at least two vm exits are needed for processing a
> > single IPI.
> >
> >
> > perf stat will show something like below [recorded for 5 seconds]
> >
> > ---------
> >
> > Analyze events for all VMs, all VCPUs:
> > VM-EXIT Samples Samples% Time% Min Time Max
> > Time Avg time
> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
> > 65.42us 2.34us ( +- 0.11% )
> > MSR_WRITE 238932 30.33% 23.07% 0.48us
> > 41.05us 1.56us ( +- 0.14% )
> >
> > Total Samples:787803, Total events handled time:1611193.84us.
> >
> > I tried turning off hyper-v for the same workload and repeat the test,
> > the overall virtualization overhead reduced by about of 50%:
> >
> > -------
> >
> > Analyze events for all VMs, all VCPUs:
> >
> > VM-EXIT Samples Samples% Time% Min Time Max
> > Time Avg time
> > APIC_WRITE 255152 74.43% 50.72% 0.49us
> > 50.01us 1.42us ( +- 0.14% )
> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
> > 686.05us 7.27us ( +- 0.43% )
> > DR_ACCESS 35003 10.21% 4.64% 0.32us
> > 40.03us 0.95us ( +- 0.32% )
> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
> > 57.38us 2.25us ( +- 1.42% )
> >
> > Total Samples:342788, Total events handled time:715695.62us.
> >
> > For this scenario, hyper-v works really bad. stimer works better
> > than hpet, but on the other hand, it relies on SynIC which has
> > negative effects for IPI intensive workloads.
> > Do you have any plans for improvement?
> >
>
> Hey,
>
> the above can be caused by the fact that when 'hv-synic' is enabled, KVM
> automatically disables APICv and this can explain the overhead and the
> fact that you're seeing more vmexits. KVM disables APICv because SynIC's
> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows
> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
> inhibit APICv if the recommendation was ignored. This is implemented in
> the following KVM patch series:
> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
>
> It will, however, require a new 'hv-something' flag to QEMU. For now, it
> can be tested with 'hv-passthrough'.
>
> It would be great if you could give it a spin!
>
> --
> Vitaly
It's great to know that you already have a solution for this. :)
By the way, is there any requirement for the version of windows or
windows updates for the new feature to work?
Thanks!
Liang
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
@ 2021-05-21 8:22 ` Liang Li
0 siblings, 0 replies; 14+ messages in thread
From: Liang Li @ 2021-05-21 8:22 UTC (permalink / raw)
To: Vitaly Kuznetsov; +Cc: Tianyu.Lan, qemu-devel, kvm
> > Hi Vitaly,
> >
> > I found a case that the virtualization overhead was almost doubled
> > when turning on Hper-v related features compared to that without any
> > no hyper-v feature. It happens when running a 3D game in windows
> > guest in qemu kvm environment.
> >
> > By investigation, I found there are a lot of IPIs triggered by guest,
> > when turning on the hyer-v related features including stimer, for the
> > apicv is turned off, at least two vm exits are needed for processing a
> > single IPI.
> >
> >
> > perf stat will show something like below [recorded for 5 seconds]
> >
> > ---------
> >
> > Analyze events for all VMs, all VCPUs:
> > VM-EXIT Samples Samples% Time% Min Time Max
> > Time Avg time
> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
> > 65.42us 2.34us ( +- 0.11% )
> > MSR_WRITE 238932 30.33% 23.07% 0.48us
> > 41.05us 1.56us ( +- 0.14% )
> >
> > Total Samples:787803, Total events handled time:1611193.84us.
> >
> > I tried turning off hyper-v for the same workload and repeat the test,
> > the overall virtualization overhead reduced by about of 50%:
> >
> > -------
> >
> > Analyze events for all VMs, all VCPUs:
> >
> > VM-EXIT Samples Samples% Time% Min Time Max
> > Time Avg time
> > APIC_WRITE 255152 74.43% 50.72% 0.49us
> > 50.01us 1.42us ( +- 0.14% )
> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
> > 686.05us 7.27us ( +- 0.43% )
> > DR_ACCESS 35003 10.21% 4.64% 0.32us
> > 40.03us 0.95us ( +- 0.32% )
> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
> > 57.38us 2.25us ( +- 1.42% )
> >
> > Total Samples:342788, Total events handled time:715695.62us.
> >
> > For this scenario, hyper-v works really bad. stimer works better
> > than hpet, but on the other hand, it relies on SynIC which has
> > negative effects for IPI intensive workloads.
> > Do you have any plans for improvement?
> >
>
> Hey,
>
> the above can be caused by the fact that when 'hv-synic' is enabled, KVM
> automatically disables APICv and this can explain the overhead and the
> fact that you're seeing more vmexits. KVM disables APICv because SynIC's
> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows
> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
> inhibit APICv if the recommendation was ignored. This is implemented in
> the following KVM patch series:
> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
>
> It will, however, require a new 'hv-something' flag to QEMU. For now, it
> can be tested with 'hv-passthrough'.
>
> It would be great if you could give it a spin!
>
> --
> Vitaly
It's great to know that you already have a solution for this. :)
By the way, is there any requirement for the version of windows or
windows updates for the new feature to work?
Thanks!
Liang
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
2021-05-21 8:22 ` Liang Li
@ 2021-05-21 8:48 ` Vitaly Kuznetsov
-1 siblings, 0 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2021-05-21 8:48 UTC (permalink / raw)
To: Liang Li; +Cc: qemu-devel, kvm, Tianyu.Lan
Liang Li <liliang324@gmail.com> writes:
>> > Hi Vitaly,
>> >
>> > I found a case that the virtualization overhead was almost doubled
>> > when turning on Hper-v related features compared to that without any
>> > no hyper-v feature. It happens when running a 3D game in windows
>> > guest in qemu kvm environment.
>> >
>> > By investigation, I found there are a lot of IPIs triggered by guest,
>> > when turning on the hyer-v related features including stimer, for the
>> > apicv is turned off, at least two vm exits are needed for processing a
>> > single IPI.
>> >
>> >
>> > perf stat will show something like below [recorded for 5 seconds]
>> >
>> > ---------
>> >
>> > Analyze events for all VMs, all VCPUs:
>> > VM-EXIT Samples Samples% Time% Min Time Max
>> > Time Avg time
>> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
>> > 65.42us 2.34us ( +- 0.11% )
>> > MSR_WRITE 238932 30.33% 23.07% 0.48us
>> > 41.05us 1.56us ( +- 0.14% )
>> >
>> > Total Samples:787803, Total events handled time:1611193.84us.
>> >
>> > I tried turning off hyper-v for the same workload and repeat the test,
>> > the overall virtualization overhead reduced by about of 50%:
>> >
>> > -------
>> >
>> > Analyze events for all VMs, all VCPUs:
>> >
>> > VM-EXIT Samples Samples% Time% Min Time Max
>> > Time Avg time
>> > APIC_WRITE 255152 74.43% 50.72% 0.49us
>> > 50.01us 1.42us ( +- 0.14% )
>> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
>> > 686.05us 7.27us ( +- 0.43% )
>> > DR_ACCESS 35003 10.21% 4.64% 0.32us
>> > 40.03us 0.95us ( +- 0.32% )
>> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
>> > 57.38us 2.25us ( +- 1.42% )
>> >
>> > Total Samples:342788, Total events handled time:715695.62us.
>> >
>> > For this scenario, hyper-v works really bad. stimer works better
>> > than hpet, but on the other hand, it relies on SynIC which has
>> > negative effects for IPI intensive workloads.
>> > Do you have any plans for improvement?
>> >
>>
>> Hey,
>>
>> the above can be caused by the fact that when 'hv-synic' is enabled, KVM
>> automatically disables APICv and this can explain the overhead and the
>> fact that you're seeing more vmexits. KVM disables APICv because SynIC's
>> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows
>> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
>> inhibit APICv if the recommendation was ignored. This is implemented in
>> the following KVM patch series:
>> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
>>
>> It will, however, require a new 'hv-something' flag to QEMU. For now, it
>> can be tested with 'hv-passthrough'.
>>
>> It would be great if you could give it a spin!
>>
>> --
>> Vitaly
>
> It's great to know that you already have a solution for this. :)
>
> By the way, is there any requirement for the version of windows or
> windows updates for the new feature to work?
AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd
expect WS2008 to ignore it completely (and thus SynIC will always be
disabling APICv for it).
--
Vitaly
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
@ 2021-05-21 8:48 ` Vitaly Kuznetsov
0 siblings, 0 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2021-05-21 8:48 UTC (permalink / raw)
To: Liang Li; +Cc: Tianyu.Lan, qemu-devel, kvm
Liang Li <liliang324@gmail.com> writes:
>> > Hi Vitaly,
>> >
>> > I found a case that the virtualization overhead was almost doubled
>> > when turning on Hper-v related features compared to that without any
>> > no hyper-v feature. It happens when running a 3D game in windows
>> > guest in qemu kvm environment.
>> >
>> > By investigation, I found there are a lot of IPIs triggered by guest,
>> > when turning on the hyer-v related features including stimer, for the
>> > apicv is turned off, at least two vm exits are needed for processing a
>> > single IPI.
>> >
>> >
>> > perf stat will show something like below [recorded for 5 seconds]
>> >
>> > ---------
>> >
>> > Analyze events for all VMs, all VCPUs:
>> > VM-EXIT Samples Samples% Time% Min Time Max
>> > Time Avg time
>> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
>> > 65.42us 2.34us ( +- 0.11% )
>> > MSR_WRITE 238932 30.33% 23.07% 0.48us
>> > 41.05us 1.56us ( +- 0.14% )
>> >
>> > Total Samples:787803, Total events handled time:1611193.84us.
>> >
>> > I tried turning off hyper-v for the same workload and repeat the test,
>> > the overall virtualization overhead reduced by about of 50%:
>> >
>> > -------
>> >
>> > Analyze events for all VMs, all VCPUs:
>> >
>> > VM-EXIT Samples Samples% Time% Min Time Max
>> > Time Avg time
>> > APIC_WRITE 255152 74.43% 50.72% 0.49us
>> > 50.01us 1.42us ( +- 0.14% )
>> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
>> > 686.05us 7.27us ( +- 0.43% )
>> > DR_ACCESS 35003 10.21% 4.64% 0.32us
>> > 40.03us 0.95us ( +- 0.32% )
>> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
>> > 57.38us 2.25us ( +- 1.42% )
>> >
>> > Total Samples:342788, Total events handled time:715695.62us.
>> >
>> > For this scenario, hyper-v works really bad. stimer works better
>> > than hpet, but on the other hand, it relies on SynIC which has
>> > negative effects for IPI intensive workloads.
>> > Do you have any plans for improvement?
>> >
>>
>> Hey,
>>
>> the above can be caused by the fact that when 'hv-synic' is enabled, KVM
>> automatically disables APICv and this can explain the overhead and the
>> fact that you're seeing more vmexits. KVM disables APICv because SynIC's
>> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows
>> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
>> inhibit APICv if the recommendation was ignored. This is implemented in
>> the following KVM patch series:
>> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
>>
>> It will, however, require a new 'hv-something' flag to QEMU. For now, it
>> can be tested with 'hv-passthrough'.
>>
>> It would be great if you could give it a spin!
>>
>> --
>> Vitaly
>
> It's great to know that you already have a solution for this. :)
>
> By the way, is there any requirement for the version of windows or
> windows updates for the new feature to work?
AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd
expect WS2008 to ignore it completely (and thus SynIC will always be
disabling APICv for it).
--
Vitaly
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
2021-05-21 8:48 ` Vitaly Kuznetsov
@ 2021-05-24 2:41 ` Liang Li
-1 siblings, 0 replies; 14+ messages in thread
From: Liang Li @ 2021-05-24 2:41 UTC (permalink / raw)
To: Vitaly Kuznetsov; +Cc: qemu-devel, kvm, Tianyu.Lan
> >> > Analyze events for all VMs, all VCPUs:
> >> > VM-EXIT Samples Samples% Time% Min Time Max
> >> > Time Avg time
> >> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
> >> > 65.42us 2.34us ( +- 0.11% )
> >> > MSR_WRITE 238932 30.33% 23.07% 0.48us
> >> > 41.05us 1.56us ( +- 0.14% )
> >> >
> >> > Total Samples:787803, Total events handled time:1611193.84us.
> >> >
> >> > I tried turning off hyper-v for the same workload and repeat the test,
> >> > the overall virtualization overhead reduced by about of 50%:
> >> >
> >> > -------
> >> >
> >> > Analyze events for all VMs, all VCPUs:
> >> >
> >> > VM-EXIT Samples Samples% Time% Min Time Max
> >> > Time Avg time
> >> > APIC_WRITE 255152 74.43% 50.72% 0.49us
> >> > 50.01us 1.42us ( +- 0.14% )
> >> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
> >> > 686.05us 7.27us ( +- 0.43% )
> >> > DR_ACCESS 35003 10.21% 4.64% 0.32us
> >> > 40.03us 0.95us ( +- 0.32% )
> >> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
> >> > 57.38us 2.25us ( +- 1.42% )
> >> >
> >> > Total Samples:342788, Total events handled time:715695.62us.
> >> >
> >> > For this scenario, hyper-v works really bad. stimer works better
> >> > than hpet, but on the other hand, it relies on SynIC which has
> >> > negative effects for IPI intensive workloads.
> >> > Do you have any plans for improvement?
> >> >
> >>
> >> Hey,
> >>
> >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM
> >> automatically disables APICv and this can explain the overhead and the
> >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's
> >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows
> >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
> >> inhibit APICv if the recommendation was ignored. This is implemented in
> >> the following KVM patch series:
> >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
> >>
> >> It will, however, require a new 'hv-something' flag to QEMU. For now, it
> >> can be tested with 'hv-passthrough'.
> >>
> >> It would be great if you could give it a spin!
> >>
> >> --
> >> Vitaly
> >
> > It's great to know that you already have a solution for this. :)
> >
> > By the way, is there any requirement for the version of windows or
> > windows updates for the new feature to work?
>
> AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd
> expect WS2008 to ignore it completely (and thus SynIC will always be
> disabling APICv for it).
>
Hi Vitaly,
I tried your patchset and found it's not helpful to reduce the
virtualization overhead.
here are some perfdata with the same workload
===============================
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max
Time Avg time
MSR_WRITE 924045 89.96% 81.10% 0.42us
68.42us 1.26us ( +- 0.07% )
DR_ACCESS 44669 4.35% 2.36% 0.32us
50.74us 0.76us ( +- 0.32% )
EXTERNAL_INTERRUPT 29809 2.90% 6.42% 0.66us
70.75us 3.10us ( +- 0.54% )
VMCALL 17819 1.73% 5.21% 0.75us
15.64us 4.20us ( +- 0.33%
Total Samples:1027227, Total events handled time:1436343.94us.
===============================
The result shows the overhead increased. enable the apicv can help to
reduce the vm-exit
caused by interrupt injection, but on the other side, there are a lot
of vm-exit caused by APIC_EOI.
When turning off the hyper-v and using the kvm apicv, there is no such
overhead. It seems turning
on hyper V related features is not always the best choice for a windows guest.
Thanks!
Liang
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
@ 2021-05-24 2:41 ` Liang Li
0 siblings, 0 replies; 14+ messages in thread
From: Liang Li @ 2021-05-24 2:41 UTC (permalink / raw)
To: Vitaly Kuznetsov; +Cc: Tianyu.Lan, qemu-devel, kvm
> >> > Analyze events for all VMs, all VCPUs:
> >> > VM-EXIT Samples Samples% Time% Min Time Max
> >> > Time Avg time
> >> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
> >> > 65.42us 2.34us ( +- 0.11% )
> >> > MSR_WRITE 238932 30.33% 23.07% 0.48us
> >> > 41.05us 1.56us ( +- 0.14% )
> >> >
> >> > Total Samples:787803, Total events handled time:1611193.84us.
> >> >
> >> > I tried turning off hyper-v for the same workload and repeat the test,
> >> > the overall virtualization overhead reduced by about of 50%:
> >> >
> >> > -------
> >> >
> >> > Analyze events for all VMs, all VCPUs:
> >> >
> >> > VM-EXIT Samples Samples% Time% Min Time Max
> >> > Time Avg time
> >> > APIC_WRITE 255152 74.43% 50.72% 0.49us
> >> > 50.01us 1.42us ( +- 0.14% )
> >> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
> >> > 686.05us 7.27us ( +- 0.43% )
> >> > DR_ACCESS 35003 10.21% 4.64% 0.32us
> >> > 40.03us 0.95us ( +- 0.32% )
> >> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
> >> > 57.38us 2.25us ( +- 1.42% )
> >> >
> >> > Total Samples:342788, Total events handled time:715695.62us.
> >> >
> >> > For this scenario, hyper-v works really bad. stimer works better
> >> > than hpet, but on the other hand, it relies on SynIC which has
> >> > negative effects for IPI intensive workloads.
> >> > Do you have any plans for improvement?
> >> >
> >>
> >> Hey,
> >>
> >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM
> >> automatically disables APICv and this can explain the overhead and the
> >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's
> >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows
> >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
> >> inhibit APICv if the recommendation was ignored. This is implemented in
> >> the following KVM patch series:
> >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
> >>
> >> It will, however, require a new 'hv-something' flag to QEMU. For now, it
> >> can be tested with 'hv-passthrough'.
> >>
> >> It would be great if you could give it a spin!
> >>
> >> --
> >> Vitaly
> >
> > It's great to know that you already have a solution for this. :)
> >
> > By the way, is there any requirement for the version of windows or
> > windows updates for the new feature to work?
>
> AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd
> expect WS2008 to ignore it completely (and thus SynIC will always be
> disabling APICv for it).
>
Hi Vitaly,
I tried your patchset and found it's not helpful to reduce the
virtualization overhead.
here are some perfdata with the same workload
===============================
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max
Time Avg time
MSR_WRITE 924045 89.96% 81.10% 0.42us
68.42us 1.26us ( +- 0.07% )
DR_ACCESS 44669 4.35% 2.36% 0.32us
50.74us 0.76us ( +- 0.32% )
EXTERNAL_INTERRUPT 29809 2.90% 6.42% 0.66us
70.75us 3.10us ( +- 0.54% )
VMCALL 17819 1.73% 5.21% 0.75us
15.64us 4.20us ( +- 0.33%
Total Samples:1027227, Total events handled time:1436343.94us.
===============================
The result shows the overhead increased. enable the apicv can help to
reduce the vm-exit
caused by interrupt injection, but on the other side, there are a lot
of vm-exit caused by APIC_EOI.
When turning off the hyper-v and using the kvm apicv, there is no such
overhead. It seems turning
on hyper V related features is not always the best choice for a windows guest.
Thanks!
Liang
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
2021-05-24 2:41 ` Liang Li
@ 2021-05-24 8:21 ` Vitaly Kuznetsov
-1 siblings, 0 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2021-05-24 8:21 UTC (permalink / raw)
To: Liang Li; +Cc: qemu-devel, kvm, Tianyu.Lan
Liang Li <liliang324@gmail.com> writes:
>> >> > Analyze events for all VMs, all VCPUs:
>> >> > VM-EXIT Samples Samples% Time% Min Time Max
>> >> > Time Avg time
>> >> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
>> >> > 65.42us 2.34us ( +- 0.11% )
>> >> > MSR_WRITE 238932 30.33% 23.07% 0.48us
>> >> > 41.05us 1.56us ( +- 0.14% )
>> >> >
>> >> > Total Samples:787803, Total events handled time:1611193.84us.
>> >> >
>> >> > I tried turning off hyper-v for the same workload and repeat the test,
>> >> > the overall virtualization overhead reduced by about of 50%:
>> >> >
>> >> > -------
>> >> >
>> >> > Analyze events for all VMs, all VCPUs:
>> >> >
>> >> > VM-EXIT Samples Samples% Time% Min Time Max
>> >> > Time Avg time
>> >> > APIC_WRITE 255152 74.43% 50.72% 0.49us
>> >> > 50.01us 1.42us ( +- 0.14% )
>> >> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
>> >> > 686.05us 7.27us ( +- 0.43% )
>> >> > DR_ACCESS 35003 10.21% 4.64% 0.32us
>> >> > 40.03us 0.95us ( +- 0.32% )
>> >> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
>> >> > 57.38us 2.25us ( +- 1.42% )
>> >> >
>> >> > Total Samples:342788, Total events handled time:715695.62us.
>> >> >
>> >> > For this scenario, hyper-v works really bad. stimer works better
>> >> > than hpet, but on the other hand, it relies on SynIC which has
>> >> > negative effects for IPI intensive workloads.
>> >> > Do you have any plans for improvement?
>> >> >
>> >>
>> >> Hey,
>> >>
>> >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM
>> >> automatically disables APICv and this can explain the overhead and the
>> >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's
>> >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows
>> >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
>> >> inhibit APICv if the recommendation was ignored. This is implemented in
>> >> the following KVM patch series:
>> >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
>> >>
>> >> It will, however, require a new 'hv-something' flag to QEMU. For now, it
>> >> can be tested with 'hv-passthrough'.
>> >>
>> >> It would be great if you could give it a spin!
>> >>
>> >> --
>> >> Vitaly
>> >
>> > It's great to know that you already have a solution for this. :)
>> >
>> > By the way, is there any requirement for the version of windows or
>> > windows updates for the new feature to work?
>>
>> AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd
>> expect WS2008 to ignore it completely (and thus SynIC will always be
>> disabling APICv for it).
>>
>
> Hi Vitaly,
> I tried your patchset and found it's not helpful to reduce the
> virtualization overhead.
> here are some perfdata with the same workload
>
> ===============================
> Analyze events for all VMs, all VCPUs:
> VM-EXIT Samples Samples% Time% Min Time Max
> Time Avg time
> MSR_WRITE 924045 89.96% 81.10% 0.42us
> 68.42us 1.26us ( +- 0.07% )
> DR_ACCESS 44669 4.35% 2.36% 0.32us
> 50.74us 0.76us ( +- 0.32% )
> EXTERNAL_INTERRUPT 29809 2.90% 6.42% 0.66us
> 70.75us 3.10us ( +- 0.54% )
> VMCALL 17819 1.73% 5.21% 0.75us
> 15.64us 4.20us ( +- 0.33%
>
> Total Samples:1027227, Total events handled time:1436343.94us.
> ===============================
>
> The result shows the overhead increased. enable the apicv can help to
> reduce the vm-exit
> caused by interrupt injection, but on the other side, there are a lot
> of vm-exit caused by APIC_EOI.
>
> When turning off the hyper-v and using the kvm apicv, there is no such
> overhead.
I think I know what's happening. We've asked Windows to use synthetic
MSRs to access APIC (HV_APIC_ACCESS_RECOMMENDED) and this can't be
accelerated in hardware.
Could you please try the following hack (KVM):
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index c8f2592ccc99..66ee85a83e9a 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -145,6 +145,13 @@ void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu)
vcpu->arch.ia32_misc_enable_msr &
MSR_IA32_MISC_ENABLE_MWAIT);
}
+
+ /* Dirty hack: force HV_DEPRECATING_AEOI_RECOMMENDED. Not to be merged! */
+ best = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO, 0);
+ if (best) {
+ best->eax &= ~HV_X64_APIC_ACCESS_RECOMMENDED;
+ best->eax |= HV_DEPRECATING_AEOI_RECOMMENDED;
+ }
}
EXPORT_SYMBOL_GPL(kvm_update_cpuid_runtime);
> It seems turning on hyper V related features is not always the best
> choice for a windows guest.
Generally it is, we'll just need to make QEMU smarter when setting
'recommendation' bits.
--
Vitaly
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
@ 2021-05-24 8:21 ` Vitaly Kuznetsov
0 siblings, 0 replies; 14+ messages in thread
From: Vitaly Kuznetsov @ 2021-05-24 8:21 UTC (permalink / raw)
To: Liang Li; +Cc: Tianyu.Lan, qemu-devel, kvm
Liang Li <liliang324@gmail.com> writes:
>> >> > Analyze events for all VMs, all VCPUs:
>> >> > VM-EXIT Samples Samples% Time% Min Time Max
>> >> > Time Avg time
>> >> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us
>> >> > 65.42us 2.34us ( +- 0.11% )
>> >> > MSR_WRITE 238932 30.33% 23.07% 0.48us
>> >> > 41.05us 1.56us ( +- 0.14% )
>> >> >
>> >> > Total Samples:787803, Total events handled time:1611193.84us.
>> >> >
>> >> > I tried turning off hyper-v for the same workload and repeat the test,
>> >> > the overall virtualization overhead reduced by about of 50%:
>> >> >
>> >> > -------
>> >> >
>> >> > Analyze events for all VMs, all VCPUs:
>> >> >
>> >> > VM-EXIT Samples Samples% Time% Min Time Max
>> >> > Time Avg time
>> >> > APIC_WRITE 255152 74.43% 50.72% 0.49us
>> >> > 50.01us 1.42us ( +- 0.14% )
>> >> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us
>> >> > 686.05us 7.27us ( +- 0.43% )
>> >> > DR_ACCESS 35003 10.21% 4.64% 0.32us
>> >> > 40.03us 0.95us ( +- 0.32% )
>> >> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us
>> >> > 57.38us 2.25us ( +- 1.42% )
>> >> >
>> >> > Total Samples:342788, Total events handled time:715695.62us.
>> >> >
>> >> > For this scenario, hyper-v works really bad. stimer works better
>> >> > than hpet, but on the other hand, it relies on SynIC which has
>> >> > negative effects for IPI intensive workloads.
>> >> > Do you have any plans for improvement?
>> >> >
>> >>
>> >> Hey,
>> >>
>> >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM
>> >> automatically disables APICv and this can explain the overhead and the
>> >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's
>> >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows
>> >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
>> >> inhibit APICv if the recommendation was ignored. This is implemented in
>> >> the following KVM patch series:
>> >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
>> >>
>> >> It will, however, require a new 'hv-something' flag to QEMU. For now, it
>> >> can be tested with 'hv-passthrough'.
>> >>
>> >> It would be great if you could give it a spin!
>> >>
>> >> --
>> >> Vitaly
>> >
>> > It's great to know that you already have a solution for this. :)
>> >
>> > By the way, is there any requirement for the version of windows or
>> > windows updates for the new feature to work?
>>
>> AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd
>> expect WS2008 to ignore it completely (and thus SynIC will always be
>> disabling APICv for it).
>>
>
> Hi Vitaly,
> I tried your patchset and found it's not helpful to reduce the
> virtualization overhead.
> here are some perfdata with the same workload
>
> ===============================
> Analyze events for all VMs, all VCPUs:
> VM-EXIT Samples Samples% Time% Min Time Max
> Time Avg time
> MSR_WRITE 924045 89.96% 81.10% 0.42us
> 68.42us 1.26us ( +- 0.07% )
> DR_ACCESS 44669 4.35% 2.36% 0.32us
> 50.74us 0.76us ( +- 0.32% )
> EXTERNAL_INTERRUPT 29809 2.90% 6.42% 0.66us
> 70.75us 3.10us ( +- 0.54% )
> VMCALL 17819 1.73% 5.21% 0.75us
> 15.64us 4.20us ( +- 0.33%
>
> Total Samples:1027227, Total events handled time:1436343.94us.
> ===============================
>
> The result shows the overhead increased. enable the apicv can help to
> reduce the vm-exit
> caused by interrupt injection, but on the other side, there are a lot
> of vm-exit caused by APIC_EOI.
>
> When turning off the hyper-v and using the kvm apicv, there is no such
> overhead.
I think I know what's happening. We've asked Windows to use synthetic
MSRs to access APIC (HV_APIC_ACCESS_RECOMMENDED) and this can't be
accelerated in hardware.
Could you please try the following hack (KVM):
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index c8f2592ccc99..66ee85a83e9a 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -145,6 +145,13 @@ void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu)
vcpu->arch.ia32_misc_enable_msr &
MSR_IA32_MISC_ENABLE_MWAIT);
}
+
+ /* Dirty hack: force HV_DEPRECATING_AEOI_RECOMMENDED. Not to be merged! */
+ best = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO, 0);
+ if (best) {
+ best->eax &= ~HV_X64_APIC_ACCESS_RECOMMENDED;
+ best->eax |= HV_DEPRECATING_AEOI_RECOMMENDED;
+ }
}
EXPORT_SYMBOL_GPL(kvm_update_cpuid_runtime);
> It seems turning on hyper V related features is not always the best
> choice for a windows guest.
Generally it is, we'll just need to make QEMU smarter when setting
'recommendation' bits.
--
Vitaly
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
2021-05-24 8:21 ` Vitaly Kuznetsov
@ 2021-06-01 13:29 ` Liang Li
-1 siblings, 0 replies; 14+ messages in thread
From: Liang Li @ 2021-06-01 13:29 UTC (permalink / raw)
To: Vitaly Kuznetsov; +Cc: qemu-devel, kvm, Tianyu.Lan
==========================
> > Analyze events for all VMs, all VCPUs:
> > VM-EXIT Samples Samples% Time% Min Time Max
> > Time Avg time
> > MSR_WRITE 924045 89.96% 81.10% 0.42us
> > 68.42us 1.26us ( +- 0.07% )
> > DR_ACCESS 44669 4.35% 2.36% 0.32us
> > 50.74us 0.76us ( +- 0.32% )
> > EXTERNAL_INTERRUPT 29809 2.90% 6.42% 0.66us
> > 70.75us 3.10us ( +- 0.54% )
> > VMCALL 17819 1.73% 5.21% 0.75us
> > 15.64us 4.20us ( +- 0.33%
> >
> > Total Samples:1027227, Total events handled time:1436343.94us.
> > ===============================
> >
> > The result shows the overhead increased. enable the apicv can help to
> > reduce the vm-exit
> > caused by interrupt injection, but on the other side, there are a lot
> > of vm-exit caused by APIC_EOI.
> >
> > When turning off the hyper-v and using the kvm apicv, there is no such
> > overhead.
>
> I think I know what's happening. We've asked Windows to use synthetic
> MSRs to access APIC (HV_APIC_ACCESS_RECOMMENDED) and this can't be
> accelerated in hardware.
>
> Could you please try the following hack (KVM):
>
> diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
> index c8f2592ccc99..66ee85a83e9a 100644
> --- a/arch/x86/kvm/cpuid.c
> +++ b/arch/x86/kvm/cpuid.c
> @@ -145,6 +145,13 @@ void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu)
> vcpu->arch.ia32_misc_enable_msr &
> MSR_IA32_MISC_ENABLE_MWAIT);
> }
> +
> + /* Dirty hack: force HV_DEPRECATING_AEOI_RECOMMENDED. Not to be merged! */
> + best = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO, 0);
> + if (best) {
> + best->eax &= ~HV_X64_APIC_ACCESS_RECOMMENDED;
> + best->eax |= HV_DEPRECATING_AEOI_RECOMMENDED;
> + }
> }
> EXPORT_SYMBOL_GPL(kvm_update_cpuid_runtime);
>
> > It seems turning on hyper V related features is not always the best
> > choice for a windows guest.
>
> Generally it is, we'll just need to make QEMU smarter when setting
> 'recommendation' bits.
>
Hi Vitaly,
I have tried your patch and found it can help to reduce the overhead.
it works as well as
the option "<feature policy='disable' name='hypervisor'/>" is set in
libvirt xml.
=======with your patch and stimer enabled=====
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max
Time Avg time
APIC_WRITE 172232 78.36% 68.99% 0.70us
47.71us 1.48us ( +- 0.18% )
DR_ACCESS 19136 8.71% 4.42% 0.55us
4.42us 0.85us ( +- 0.32% )
EXTERNAL_INTERRUPT 15921 7.24% 13.84% 0.87us
55.28us 3.21us ( +- 0.55% )
VMCALL 6971 3.17% 10.34% 1.16us
12.02us 5.48us ( +- 0.49%
Total Samples:219802, Total events handled time:369310.30us.
===========with hypervisor disabled=========
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max
Time Avg time
APIC_WRITE 200482 78.51% 68.62% 0.64us
49.51us 1.37us ( +- 0.16% )
DR_ACCESS 24235 9.49% 4.92% 0.55us
3.65us 0.81us ( +- 0.26% )
EXTERNAL_INTERRUPT 17084 6.69% 13.20% 0.89us
56.38us 3.09us ( +- 0.53% )
VMCALL 7124 2.79% 9.87% 1.26us
12.39us 5.54us ( +- 0.49% )
EOI_INDUCED 5066 1.98% 1.36% 0.66us
2.64us 1.07us ( +- 0.25% )
IO_INSTRUCTION 591 0.23% 1.27% 3.37us
673.23us 8.59us ( +- 13.69% )
Total Samples:255363, Total events handled time:399954.27us.
Thanks!
Liang
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: About the performance of hyper-v
@ 2021-06-01 13:29 ` Liang Li
0 siblings, 0 replies; 14+ messages in thread
From: Liang Li @ 2021-06-01 13:29 UTC (permalink / raw)
To: Vitaly Kuznetsov; +Cc: Tianyu.Lan, qemu-devel, kvm
==========================
> > Analyze events for all VMs, all VCPUs:
> > VM-EXIT Samples Samples% Time% Min Time Max
> > Time Avg time
> > MSR_WRITE 924045 89.96% 81.10% 0.42us
> > 68.42us 1.26us ( +- 0.07% )
> > DR_ACCESS 44669 4.35% 2.36% 0.32us
> > 50.74us 0.76us ( +- 0.32% )
> > EXTERNAL_INTERRUPT 29809 2.90% 6.42% 0.66us
> > 70.75us 3.10us ( +- 0.54% )
> > VMCALL 17819 1.73% 5.21% 0.75us
> > 15.64us 4.20us ( +- 0.33%
> >
> > Total Samples:1027227, Total events handled time:1436343.94us.
> > ===============================
> >
> > The result shows the overhead increased. enable the apicv can help to
> > reduce the vm-exit
> > caused by interrupt injection, but on the other side, there are a lot
> > of vm-exit caused by APIC_EOI.
> >
> > When turning off the hyper-v and using the kvm apicv, there is no such
> > overhead.
>
> I think I know what's happening. We've asked Windows to use synthetic
> MSRs to access APIC (HV_APIC_ACCESS_RECOMMENDED) and this can't be
> accelerated in hardware.
>
> Could you please try the following hack (KVM):
>
> diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
> index c8f2592ccc99..66ee85a83e9a 100644
> --- a/arch/x86/kvm/cpuid.c
> +++ b/arch/x86/kvm/cpuid.c
> @@ -145,6 +145,13 @@ void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu)
> vcpu->arch.ia32_misc_enable_msr &
> MSR_IA32_MISC_ENABLE_MWAIT);
> }
> +
> + /* Dirty hack: force HV_DEPRECATING_AEOI_RECOMMENDED. Not to be merged! */
> + best = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO, 0);
> + if (best) {
> + best->eax &= ~HV_X64_APIC_ACCESS_RECOMMENDED;
> + best->eax |= HV_DEPRECATING_AEOI_RECOMMENDED;
> + }
> }
> EXPORT_SYMBOL_GPL(kvm_update_cpuid_runtime);
>
> > It seems turning on hyper V related features is not always the best
> > choice for a windows guest.
>
> Generally it is, we'll just need to make QEMU smarter when setting
> 'recommendation' bits.
>
Hi Vitaly,
I have tried your patch and found it can help to reduce the overhead.
it works as well as
the option "<feature policy='disable' name='hypervisor'/>" is set in
libvirt xml.
=======with your patch and stimer enabled=====
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max
Time Avg time
APIC_WRITE 172232 78.36% 68.99% 0.70us
47.71us 1.48us ( +- 0.18% )
DR_ACCESS 19136 8.71% 4.42% 0.55us
4.42us 0.85us ( +- 0.32% )
EXTERNAL_INTERRUPT 15921 7.24% 13.84% 0.87us
55.28us 3.21us ( +- 0.55% )
VMCALL 6971 3.17% 10.34% 1.16us
12.02us 5.48us ( +- 0.49%
Total Samples:219802, Total events handled time:369310.30us.
===========with hypervisor disabled=========
Analyze events for all VMs, all VCPUs:
VM-EXIT Samples Samples% Time% Min Time Max
Time Avg time
APIC_WRITE 200482 78.51% 68.62% 0.64us
49.51us 1.37us ( +- 0.16% )
DR_ACCESS 24235 9.49% 4.92% 0.55us
3.65us 0.81us ( +- 0.26% )
EXTERNAL_INTERRUPT 17084 6.69% 13.20% 0.89us
56.38us 3.09us ( +- 0.53% )
VMCALL 7124 2.79% 9.87% 1.26us
12.39us 5.54us ( +- 0.49% )
EOI_INDUCED 5066 1.98% 1.36% 0.66us
2.64us 1.07us ( +- 0.25% )
IO_INSTRUCTION 591 0.23% 1.27% 3.37us
673.23us 8.59us ( +- 13.69% )
Total Samples:255363, Total events handled time:399954.27us.
Thanks!
Liang
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-06-01 13:37 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-19 14:00 About the performance of hyper-v Liang Li
2021-05-19 14:00 ` Liang Li
2021-05-19 14:22 ` Vitaly Kuznetsov
2021-05-19 14:22 ` Vitaly Kuznetsov
2021-05-21 8:22 ` Liang Li
2021-05-21 8:22 ` Liang Li
2021-05-21 8:48 ` Vitaly Kuznetsov
2021-05-21 8:48 ` Vitaly Kuznetsov
2021-05-24 2:41 ` Liang Li
2021-05-24 2:41 ` Liang Li
2021-05-24 8:21 ` Vitaly Kuznetsov
2021-05-24 8:21 ` Vitaly Kuznetsov
2021-06-01 13:29 ` Liang Li
2021-06-01 13:29 ` Liang Li
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.