* apic-v reduce network performance in my test case
@ 2015-01-31 10:29 Liuqiming (John)
2015-02-02 10:12 ` Jan Beulich
0 siblings, 1 reply; 7+ messages in thread
From: Liuqiming (John) @ 2015-01-31 10:29 UTC (permalink / raw)
To: xen-devel
Hi all,
Recently I met an odd performance problem: when I turn on APIC
Virtualization feature (apicv=1), the network performance of a windows
guest become worse.
My test case like this: host only have one windows 2008 R2 HVM
guest running,and this guest has a SR-IOV VF network passthrough to it.
Guest using this network access a NAS device. No fontend or backend of
network and storage, all data transfered through network.
The xentrace data shows: the mainly difference between apicv and
non-apicv, is the way guest write apic registers, and
EXIT_REASON_MSR_WRITE vmexit cost much more time than
EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit is much
less than using APIC-v.
This is the odd part,any ideas?
APIC-v OFF:
4099582 VMEXIT 3467051359128 TSC HLT
10135140 VMEXIT 42484175528 TSC WRMSR
1651714 VMEXIT 9785961276 TSC I/O instruction
532702 VMEXIT 3887971388 TSC External interrupt
290546 VMEXIT 2262312440 TSC PAUSE
588077 VMEXIT 914905312 TSC Control-register accesses
383617 VMEXIT 453329940 TSC Exception or non-maskable
interrupt (NMI)
132717 VMEXIT 232289792 TSC Interrupt window
25534 VMEXIT 198718764 TSC EPT violation
53969 VMEXIT 62886752 TSC TPR below threshold
7996 VMEXIT 34735376 TSC RDMSR
1615 VMEXIT 16042768 TSC VMCALL
147 VMEXIT 272320 TSC CPUID
7 VMEXIT 6484 TSC WBINVD
2 VMEXIT 6308 TSC MOV DR
APIC-v ON:
3717629 VMEXIT 3459905385332 TSC HLT
2282403 VMEXIT 23099880196 TSC APIC write
3900448 VMEXIT 13073253548 TSC PAUSE
1643729 VMEXIT 11719626776 TSC I/O instruction
2194667 VMEXIT 5321640708 TSC WRMSR
214425 VMEXIT 2198994944 TSC External interrupt
566795 VMEXIT 1940710108 TSC Control-register accesses
342688 VMEXIT 659665532 TSC Exception or non-maskable
interrupt (NMI)
190623 VMEXIT 644411612 TSC VMCALL
188657 VMEXIT 295956932 TSC Virtualized EOI
24350 VMEXIT 194817152 TSC EPT violation
4393 VMEXIT 23282044 TSC RDMSR
179 VMEXIT 1688676 TSC CPUID
7 VMEXIT 6884 TSC WBINVD
1 VMEXIT 4200 TSC MOV DR
In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
"VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V",
msr based apic is disabled when apic-v is on, I wonder can they co-exist
in some way? seems for windows guest msr-based apic has better performance.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apic-v reduce network performance in my test case
2015-01-31 10:29 apic-v reduce network performance in my test case Liuqiming (John)
@ 2015-02-02 10:12 ` Jan Beulich
2015-02-02 13:58 ` Liuqiming (John)
0 siblings, 1 reply; 7+ messages in thread
From: Jan Beulich @ 2015-02-02 10:12 UTC (permalink / raw)
To: Liuqiming (John); +Cc: xen-devel
>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> Recently I met an odd performance problem: when I turn on APIC
> Virtualization feature (apicv=1), the network performance of a windows
> guest become worse.
>
> My test case like this: host only have one windows 2008 R2 HVM
> guest running,and this guest has a SR-IOV VF network passthrough to it.
> Guest using this network access a NAS device. No fontend or backend of
> network and storage, all data transfered through network.
>
> The xentrace data shows: the mainly difference between apicv and
> non-apicv, is the way guest write apic registers, and
> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit is much
> less than using APIC-v.
There being heavier use of the pause VMEXIT doesn't by itself say
anything, I'm afraid. It may suggest that you have a C-state exit
latency problem - try lowering the maximum C-state allowed, or
disabling use of C-states altogether.
> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> "VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V",
> msr based apic is disabled when apic-v is on, I wonder can they co-exist
> in some way? seems for windows guest msr-based apic has better performance.
The whole purpose is to avoid the costly MSR access exits. Why
would you want to reintroduce that overhead?
Jan
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apic-v reduce network performance in my test case
2015-02-02 10:12 ` Jan Beulich
@ 2015-02-02 13:58 ` Liuqiming (John)
2015-02-02 14:59 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 7+ messages in thread
From: Liuqiming (John) @ 2015-02-02 13:58 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel
Hi Jan,
Thanks for the reply.
On 2015/2/2 18:12, Jan Beulich wrote:
> >>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> > Recently I met an odd performance problem: when I turn on APIC
> > Virtualization feature (apicv=1), the network performance of a windows
> > guest become worse.
> >
> > My test case like this: host only have one windows 2008 R2 HVM
> > guest running,and this guest has a SR-IOV VF network passthrough to it.
> > Guest using this network access a NAS device. No fontend or backend of
> > network and storage, all data transfered through network.
> >
> > The xentrace data shows: the mainly difference between apicv and
> > non-apicv, is the way guest write apic registers, and
> > EXIT_REASON_MSR_WRITE vmexit cost much more time than
> > EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit is much
> > less than using APIC-v.
>
> There being heavier use of the pause VMEXIT doesn't by itself say
> anything, I'm afraid. It may suggest that you have a C-state exit
> latency problem - try lowering the maximum C-state allowed, or
> disabling use of C-states altogether.
Sorry, I forgot to mention my test scenario:
Its a video test suite,I am not sure what the logic inside the tools exactly (not opensource tool).
The basic flow is:
1) test suite start several thread to read video file from disk (from NAS through network in my case)
2) decode these video data as a frame one by one
3) if any frame delay more than 40ms, then mark as lost
test result:
apicv=1, there can be 15 thread running at the same time without lost frame
apicv=0, there can be 22 thread running at the same time without lost frame
so when I'm saying apicv reduce the performance, I got the conclusion from the test result not from what xentrace shows.
>
> > In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> > "VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V",
> > msr based apic is disabled when apic-v is on, I wonder can they co-exist
> > in some way? seems for windows guest msr-based apic has better performance.
>
> The whole purpose is to avoid the costly MSR access exits. Why
> would you want to reintroduce that overhead?
>
> Jan
>
I agree to avoid the MSR access vmexit by using apicv, I just do not know what's the side effect.
Because from the test result, apicv replacing msr-based access brings performance reduction.
>
> .
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apic-v reduce network performance in my test case
2015-02-02 13:58 ` Liuqiming (John)
@ 2015-02-02 14:59 ` Konrad Rzeszutek Wilk
[not found] ` <54D03148.50604@huawei.com>
0 siblings, 1 reply; 7+ messages in thread
From: Konrad Rzeszutek Wilk @ 2015-02-02 14:59 UTC (permalink / raw)
To: Liuqiming (John); +Cc: Jan Beulich, xen-devel
On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
> Hi Jan,
> Thanks for the reply.
>
> On 2015/2/2 18:12, Jan Beulich wrote:
> >>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> >> Recently I met an odd performance problem: when I turn on APIC
> >> Virtualization feature (apicv=1), the network performance of a windows
> >> guest become worse.
> >>
> >> My test case like this: host only have one windows 2008 R2 HVM
> >> guest running,and this guest has a SR-IOV VF network passthrough to it.
> >> Guest using this network access a NAS device. No fontend or backend of
> >> network and storage, all data transfered through network.
> >>
> >> The xentrace data shows: the mainly difference between apicv and
> >> non-apicv, is the way guest write apic registers, and
> >> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> >> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit is much
> >> less than using APIC-v.
> >
> >There being heavier use of the pause VMEXIT doesn't by itself say
> >anything, I'm afraid. It may suggest that you have a C-state exit
> >latency problem - try lowering the maximum C-state allowed, or
> >disabling use of C-states altogether.
> Sorry, I forgot to mention my test scenario:
> Its a video test suite,I am not sure what the logic inside the tools exactly (not opensource tool).
> The basic flow is:
> 1) test suite start several thread to read video file from disk (from NAS through network in my case)
> 2) decode these video data as a frame one by one
> 3) if any frame delay more than 40ms, then mark as lost
>
> test result:
> apicv=1, there can be 15 thread running at the same time without lost frame
> apicv=0, there can be 22 thread running at the same time without lost frame
>
> so when I'm saying apicv reduce the performance, I got the conclusion from the test result not from what xentrace shows.
> >
> >> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> >> "VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V",
> >> msr based apic is disabled when apic-v is on, I wonder can they co-exist
> >> in some way? seems for windows guest msr-based apic has better performance.
> >
> >The whole purpose is to avoid the costly MSR access exits. Why
> >would you want to reintroduce that overhead?
> >
> >Jan
> >
> I agree to avoid the MSR access vmexit by using apicv, I just do not know what's the side effect.
> Because from the test result, apicv replacing msr-based access brings performance reduction.
It sounds like it brings latency disruption, not neccessarily performance reduction.
What happens if you run with the cpufreq turned to performance, or as Jan suggested -
with disabling C-states?
> >
> >.
> >
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apic-v reduce network performance in my test case
[not found] ` <54D2DC35.6070205@huawei.com>
@ 2015-02-05 3:04 ` Liuqiming (John)
2015-02-05 19:26 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 7+ messages in thread
From: Liuqiming (John) @ 2015-02-05 3:04 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, xen-devel
On 2015/2/5 10:57, Liuqiming (John) wrote:
> sorry for late replay
>
> On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote:
> > On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote:
> > > On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote:
> > >> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
> > >>> Hi Jan,
> > >>> Thanks for the reply.
> > >>>
> > >>> On 2015/2/2 18:12, Jan Beulich wrote:
> > >>>>>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> > >>>>> Recently I met an odd performance problem: when I turn
> > on APIC
> > >>>>> Virtualization feature (apicv=1), the network performance of a
> > windows
> > >>>>> guest become worse.
> > >>>>>
> > >>>>> My test case like this: host only have one windows 2008
> > R2 HVM
> > >>>>> guest running,and this guest has a SR-IOV VF network passthrough
> > to it.
> > >>>>> Guest using this network access a NAS device. No fontend or
> > backend of
> > >>>>> network and storage, all data transfered through network.
> > >>>>>
> > >>>>> The xentrace data shows: the mainly difference between
> > apicv and
> > >>>>> non-apicv, is the way guest write apic registers, and
> > >>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> > >>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit
> > is much
> > >>>>> less than using APIC-v.
> > >>>>
> > >>>> There being heavier use of the pause VMEXIT doesn't by itself say
> > >>>> anything, I'm afraid. It may suggest that you have a C-state exit
> > >>>> latency problem - try lowering the maximum C-state allowed, or
> > >>>> disabling use of C-states altogether.
> > >>> Sorry, I forgot to mention my test scenario:
> > >>> Its a video test suite,I am not sure what the logic inside the
> > tools exactly (not opensource tool).
> > >>> The basic flow is:
> > >>> 1) test suite start several thread to read video file from
> > disk (from NAS through network in my case)
> > >>> 2) decode these video data as a frame one by one
> > >>> 3) if any frame delay more than 40ms, then mark as lost
> > >>>
> > >>> test result:
> > >>> apicv=1, there can be 15 thread running at the same time
> > without lost frame
> > >>> apicv=0, there can be 22 thread running at the same time
> > without lost frame
> > >>>
> > >>> so when I'm saying apicv reduce the performance, I got the
> > conclusion from the test result not from what xentrace shows.
> > >>>>
> > >>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> > >>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having
> > APIC-V",
> > >>>>> msr based apic is disabled when apic-v is on, I wonder can they
> > co-exist
> > >>>>> in some way? seems for windows guest msr-based apic has better
> > performance.
> > >>>>
> > >>>> The whole purpose is to avoid the costly MSR access exits. Why
> > >>>> would you want to reintroduce that overhead?
> > >>>>
> > >>>> Jan
> > >>>>
> > >>> I agree to avoid the MSR access vmexit by using apicv, I just do
> > not know what's the side effect.
> > >>> Because from the test result, apicv replacing msr-based access
> > brings performance reduction.
> > >>
> > >> It sounds like it brings latency disruption, not neccessarily
> > performance reduction.
> > >>
> > >> What happens if you run with the cpufreq turned to performance, or
> > as Jan suggested -
> > >> with disabling C-states?
> > >>
> > > I have tried disable C-states in BIOS, the test result is still the
> > same.
> > > There is no frontend-backend device in my test, dom0 is not
> > involved, so you mean cpufreq in DomU, right?
> >
> > I meant on Xen (cpufreq=performance). But if you disable C-states then
> > that should
> > not matter.
> >
> > > I have no idea how to change cpufreq of a windows os but I will
> > figure out and try.
> >
> > That should not be needed.
> >
> > But back to your problem. When you say you have 'frame delay' - is it
> > because
> > of the NAS storage not getting the I/Os fast enough?
> >
> The NAS storage should not be the bottleneck, it's a ramdisk connected
> with 10G network card.
> Besides, I'am using the same test environment, the only difference is
> whether apicv=1 or not.
>
> > What are you using as network device? Is it an SR-IOV device or an
> > emulated
> > device (e1000, ne2k, rtl81..?)?
> The vm using Intel Corporation 82599 Ethernet Controller Virtual
> Function passthrough network and has the corresponding driver installed.
> > >>
> > >>> _______________________________________________
> > >>> Xen-devel mailing list
> > >>> Xen-devel@lists.xen.org
> > >>> http://lists.xen.org/xen-devel
> > >>
> > >> .
> > >>
> > >
> When tracing the performance problem, I found whether apicv is on
> change the result.
> And after investigating the code, turns out CPUID4A_MSR_BASED_APIC
> presents or not is the key factor.
>
>
>
>
sorry, forget cc the list
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apic-v reduce network performance in my test case
2015-02-05 3:04 ` Liuqiming (John)
@ 2015-02-05 19:26 ` Konrad Rzeszutek Wilk
2015-02-06 2:48 ` Liuqiming (John)
0 siblings, 1 reply; 7+ messages in thread
From: Konrad Rzeszutek Wilk @ 2015-02-05 19:26 UTC (permalink / raw)
To: Liuqiming (John); +Cc: xen-devel
On Thu, Feb 05, 2015 at 11:04:37AM +0800, Liuqiming (John) wrote:
> On 2015/2/5 10:57, Liuqiming (John) wrote:
> >sorry for late replay
> >
> >On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote:
> >> On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote:
> >> > On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote:
> >> >> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
> >> >>> Hi Jan,
> >> >>> Thanks for the reply.
> >> >>>
> >> >>> On 2015/2/2 18:12, Jan Beulich wrote:
> >> >>>>>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> >> >>>>> Recently I met an odd performance problem: when I turn
> >> on APIC
> >> >>>>> Virtualization feature (apicv=1), the network performance of a
> >> windows
> >> >>>>> guest become worse.
> >> >>>>>
> >> >>>>> My test case like this: host only have one windows 2008
> >> R2 HVM
> >> >>>>> guest running,and this guest has a SR-IOV VF network passthrough
> >> to it.
> >> >>>>> Guest using this network access a NAS device. No fontend or
> >> backend of
> >> >>>>> network and storage, all data transfered through network.
> >> >>>>>
> >> >>>>> The xentrace data shows: the mainly difference between
> >> apicv and
> >> >>>>> non-apicv, is the way guest write apic registers, and
> >> >>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> >> >>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit
> >> is much
> >> >>>>> less than using APIC-v.
> >> >>>>
> >> >>>> There being heavier use of the pause VMEXIT doesn't by itself say
> >> >>>> anything, I'm afraid. It may suggest that you have a C-state exit
> >> >>>> latency problem - try lowering the maximum C-state allowed, or
> >> >>>> disabling use of C-states altogether.
> >> >>> Sorry, I forgot to mention my test scenario:
> >> >>> Its a video test suite,I am not sure what the logic inside the
> >> tools exactly (not opensource tool).
> >> >>> The basic flow is:
> >> >>> 1) test suite start several thread to read video file from
> >> disk (from NAS through network in my case)
> >> >>> 2) decode these video data as a frame one by one
> >> >>> 3) if any frame delay more than 40ms, then mark as lost
> >> >>>
> >> >>> test result:
> >> >>> apicv=1, there can be 15 thread running at the same time
> >> without lost frame
> >> >>> apicv=0, there can be 22 thread running at the same time
> >> without lost frame
> >> >>>
> >> >>> so when I'm saying apicv reduce the performance, I got the
> >> conclusion from the test result not from what xentrace shows.
> >> >>>>
> >> >>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> >> >>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having
> >> APIC-V",
> >> >>>>> msr based apic is disabled when apic-v is on, I wonder can they
> >> co-exist
> >> >>>>> in some way? seems for windows guest msr-based apic has better
> >> performance.
> >> >>>>
> >> >>>> The whole purpose is to avoid the costly MSR access exits. Why
> >> >>>> would you want to reintroduce that overhead?
> >> >>>>
> >> >>>> Jan
> >> >>>>
> >> >>> I agree to avoid the MSR access vmexit by using apicv, I just do
> >> not know what's the side effect.
> >> >>> Because from the test result, apicv replacing msr-based access
> >> brings performance reduction.
> >> >>
> >> >> It sounds like it brings latency disruption, not neccessarily
> >> performance reduction.
> >> >>
> >> >> What happens if you run with the cpufreq turned to performance, or
> >> as Jan suggested -
> >> >> with disabling C-states?
> >> >>
> >> > I have tried disable C-states in BIOS, the test result is still the
> >> same.
> >> > There is no frontend-backend device in my test, dom0 is not
> >> involved, so you mean cpufreq in DomU, right?
> >>
> >> I meant on Xen (cpufreq=performance). But if you disable C-states then
> >> that should
> >> not matter.
> >>
> >> > I have no idea how to change cpufreq of a windows os but I will
> >> figure out and try.
> >>
> >> That should not be needed.
> >>
> >> But back to your problem. When you say you have 'frame delay' - is it
> >> because
> >> of the NAS storage not getting the I/Os fast enough?
> >>
> >The NAS storage should not be the bottleneck, it's a ramdisk connected
> >with 10G network card.
> >Besides, I'am using the same test environment, the only difference is
> >whether apicv=1 or not.
> >
> >> What are you using as network device? Is it an SR-IOV device or an
> >> emulated
> >> device (e1000, ne2k, rtl81..?)?
> >The vm using Intel Corporation 82599 Ethernet Controller Virtual
> >Function passthrough network and has the corresponding driver installed.
> >> >>
> >> >>> _______________________________________________
> >> >>> Xen-devel mailing list
> >> >>> Xen-devel@lists.xen.org
> >> >>> http://lists.xen.org/xen-devel
> >> >>
> >> >> .
> >> >>
> >> >
> >When tracing the performance problem, I found whether apicv is on
> >change the result.
> >And after investigating the code, turns out CPUID4A_MSR_BASED_APIC
> >presents or not is the key factor.
Ah, willing to send out a patch for that?
> >
> >
> >
> >
> sorry, forget cc the list
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: apic-v reduce network performance in my test case
2015-02-05 19:26 ` Konrad Rzeszutek Wilk
@ 2015-02-06 2:48 ` Liuqiming (John)
0 siblings, 0 replies; 7+ messages in thread
From: Liuqiming (John) @ 2015-02-06 2:48 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk; +Cc: xen-devel
On 2015/2/6 3:26, Konrad Rzeszutek Wilk wrote:
> On Thu, Feb 05, 2015 at 11:04:37AM +0800, Liuqiming (John) wrote:
> > On 2015/2/5 10:57, Liuqiming (John) wrote:
> >> sorry for late replay
> >>
> >> On 2015/2/3 23:32, Konrad Rzeszutek Wilk wrote:
> >>> On Tue, Feb 03, 2015 at 10:24:08AM +0800, Liuqiming (John) wrote:
> >>>> On 2015/2/2 22:59, Konrad Rzeszutek Wilk wrote:
> >>>>> On Mon, Feb 02, 2015 at 09:58:57PM +0800, Liuqiming (John) wrote:
> >>>>>> Hi Jan,
> >>>>>> Thanks for the reply.
> >>>>>>
> >>>>>> On 2015/2/2 18:12, Jan Beulich wrote:
> >>>>>>>>>> On 31.01.15 at 11:29, <john.liuqiming@huawei.com> wrote:
> >>>>>>>> Recently I met an odd performance problem: when I turn
> >>> on APIC
> >>>>>>>> Virtualization feature (apicv=1), the network performance of a
> >>> windows
> >>>>>>>> guest become worse.
> >>>>>>>>
> >>>>>>>> My test case like this: host only have one windows 2008
> >>> R2 HVM
> >>>>>>>> guest running,and this guest has a SR-IOV VF network passthrough
> >>> to it.
> >>>>>>>> Guest using this network access a NAS device. No fontend or
> >>> backend of
> >>>>>>>> network and storage, all data transfered through network.
> >>>>>>>>
> >>>>>>>> The xentrace data shows: the mainly difference between
> >>> apicv and
> >>>>>>>> non-apicv, is the way guest write apic registers, and
> >>>>>>>> EXIT_REASON_MSR_WRITE vmexit cost much more time than
> >>>>>>>> EXIT_REASON_APIC_WRITE, but when using WRMSR, the PAUSE vmexit
> >>> is much
> >>>>>>>> less than using APIC-v.
> >>>>>>>
> >>>>>>> There being heavier use of the pause VMEXIT doesn't by itself say
> >>>>>>> anything, I'm afraid. It may suggest that you have a C-state exit
> >>>>>>> latency problem - try lowering the maximum C-state allowed, or
> >>>>>>> disabling use of C-states altogether.
> >>>>>> Sorry, I forgot to mention my test scenario:
> >>>>>> Its a video test suite,I am not sure what the logic inside the
> >>> tools exactly (not opensource tool).
> >>>>>> The basic flow is:
> >>>>>> 1) test suite start several thread to read video file from
> >>> disk (from NAS through network in my case)
> >>>>>> 2) decode these video data as a frame one by one
> >>>>>> 3) if any frame delay more than 40ms, then mark as lost
> >>>>>>
> >>>>>> test result:
> >>>>>> apicv=1, there can be 15 thread running at the same time
> >>> without lost frame
> >>>>>> apicv=0, there can be 22 thread running at the same time
> >>> without lost frame
> >>>>>>
> >>>>>> so when I'm saying apicv reduce the performance, I got the
> >>> conclusion from the test result not from what xentrace shows.
> >>>>>>>
> >>>>>>>> In commit 7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
> >>>>>>>> "VMX/Viridian: suppress MSR-based APIC suggestion when having
> >>> APIC-V",
> >>>>>>>> msr based apic is disabled when apic-v is on, I wonder can they
> >>> co-exist
> >>>>>>>> in some way? seems for windows guest msr-based apic has better
> >>> performance.
> >>>>>>>
> >>>>>>> The whole purpose is to avoid the costly MSR access exits. Why
> >>>>>>> would you want to reintroduce that overhead?
> >>>>>>>
> >>>>>>> Jan
> >>>>>>>
> >>>>>> I agree to avoid the MSR access vmexit by using apicv, I just do
> >>> not know what's the side effect.
> >>>>>> Because from the test result, apicv replacing msr-based access
> >>> brings performance reduction.
> >>>>>
> >>>>> It sounds like it brings latency disruption, not neccessarily
> >>> performance reduction.
> >>>>>
> >>>>> What happens if you run with the cpufreq turned to performance, or
> >>> as Jan suggested -
> >>>>> with disabling C-states?
> >>>>>
> >>>> I have tried disable C-states in BIOS, the test result is still the
> >>> same.
> >>>> There is no frontend-backend device in my test, dom0 is not
> >>> involved, so you mean cpufreq in DomU, right?
> >>>
> >>> I meant on Xen (cpufreq=performance). But if you disable C-states then
> >>> that should
> >>> not matter.
> >>>
> >>>> I have no idea how to change cpufreq of a windows os but I will
> >>> figure out and try.
> >>>
> >>> That should not be needed.
> >>>
> >>> But back to your problem. When you say you have 'frame delay' - is it
> >>> because
> >>> of the NAS storage not getting the I/Os fast enough?
> >>>
> >> The NAS storage should not be the bottleneck, it's a ramdisk connected
> >> with 10G network card.
> >> Besides, I'am using the same test environment, the only difference is
> >> whether apicv=1 or not.
> >>
> >>> What are you using as network device? Is it an SR-IOV device or an
> >>> emulated
> >>> device (e1000, ne2k, rtl81..?)?
> >> The vm using Intel Corporation 82599 Ethernet Controller Virtual
> >> Function passthrough network and has the corresponding driver installed.
> >>>>>
> >>>>>> _______________________________________________
> >>>>>> Xen-devel mailing list
> >>>>>> Xen-devel@lists.xen.org
> >>>>>> http://lists.xen.org/xen-devel
> >>>>>
> >>>>> .
> >>>>>
> >>>>
> >> When tracing the performance problem, I found whether apicv is on
> >> change the result.
> >> And after investigating the code, turns out CPUID4A_MSR_BASED_APIC
> >> presents or not is the key factor.
>
> Ah, willing to send out a patch for that?
I have not get the conclusion: why CPUID4A_MSR_BASED_APIC affect performance.
But I will continue to dig in, and happy to send out what I found.
> >>
> >>
> >>
> >>
> > sorry, forget cc the list
> >
> >
>
for testing purpose, reverting this commit will do:
http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=7f2e992b824ec62a2818e64390ac2ccfbd74e6b7
"VMX/Viridian: suppress MSR-based APIC suggestion when having APIC-V"
> .
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-02-06 2:48 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-01-31 10:29 apic-v reduce network performance in my test case Liuqiming (John)
2015-02-02 10:12 ` Jan Beulich
2015-02-02 13:58 ` Liuqiming (John)
2015-02-02 14:59 ` Konrad Rzeszutek Wilk
[not found] ` <54D03148.50604@huawei.com>
[not found] ` <20150203153228.GD9371@l.oracle.com>
[not found] ` <54D2DC35.6070205@huawei.com>
2015-02-05 3:04 ` Liuqiming (John)
2015-02-05 19:26 ` Konrad Rzeszutek Wilk
2015-02-06 2:48 ` Liuqiming (John)
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.