* performace issue when turn on apicv
@ 2015-06-11 7:23 Liuqiming (John)
2015-06-11 7:35 ` Zhang, Yang Z
0 siblings, 1 reply; 17+ messages in thread
From: Liuqiming (John) @ 2015-06-11 7:23 UTC (permalink / raw)
To: xen-devel; +Cc: yang.z.zhang, peter.huangpeng
Hi,
Recently I encounter a strange performance problem with APIC virtualization.
My host has Intel(R) Xeon(R) CPU E7-4890 v2 CPU installed which support
APIC virtualization and x2apic, and there are 4 socket * 15
cores_per_socket = 60 core available for VM. There is also a SSD disk on
host and the host support vt-d, so I can passsthrough this SSD to VM.
A VM created with 60 vcpus, 400G memory and SSD device assigned.
I pin these vcpus 1:1 to phisical cpu, and in this VM only keep 15 vcpus
online.
The problem is: when apicv turn on, a significant performace decrease
can be observed and it seems related to cpu topology.
I had test follow cases
apicv=1:
1) ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
PIN TO
PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
2) ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
PIN TO
PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
apicv=0:
3) ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
PIN TO
PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
4) ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
PIN TO
PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
the result is (the lower the better):
1) 989
2) 2209
3) 978
4) 1130
It is a database testcase running on suse11sp3 system in the VM, and I
had traced that "read" and "write" syscall get much slower in 2) case.
I have disabled NUMA in BIOS, so it seems apicv cause this bad
performance when using cpus in different nodes.
Can any one shed some light on this?
Btw, I am using xen 4.1.5 version with apicv backported, so I am not
sure whether something broken when backporting or just apicv behaves
this way.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-11 7:23 performace issue when turn on apicv Liuqiming (John)
@ 2015-06-11 7:35 ` Zhang, Yang Z
2015-06-11 8:09 ` Liuqiming (John)
0 siblings, 1 reply; 17+ messages in thread
From: Zhang, Yang Z @ 2015-06-11 7:35 UTC (permalink / raw)
To: Liuqiming (John), xen-devel; +Cc: peter.huangpeng
Liuqiming (John) wrote on 2015-06-11:
> Hi,
>
> Recently I encounter a strange performance problem with APIC virtualization.
>
> My host has Intel(R) Xeon(R) CPU E7-4890 v2 CPU installed which support
> APIC virtualization and x2apic, and there are 4 socket * 15
> cores_per_socket = 60 core available for VM. There is also a SSD disk on
> host and the host support vt-d, so I can passsthrough this SSD to VM.
>
> A VM created with 60 vcpus, 400G memory and SSD device assigned.
> I pin these vcpus 1:1 to phisical cpu, and in this VM only keep 15 vcpus
> online.
> The problem is: when apicv turn on, a significant performace decrease
> can be observed and it seems related to cpu topology.
>
> I had test follow cases
> apicv=1:
> 1) ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> PIN TO
> PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> 2) ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> PIN TO
> PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> apicv=0:
> 3) ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> PIN TO
> PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> 4) ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> PIN TO
> PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> the result is (the lower the better):
> 1) 989
> 2) 2209
> 3) 978
> 4) 1130
>
> It is a database testcase running on suse11sp3 system in the VM, and I
> had traced that "read" and "write" syscall get much slower in 2) case.
>
> I have disabled NUMA in BIOS, so it seems apicv cause this bad
> performance when using cpus in different nodes.
>
> Can any one shed some light on this?
>
> Btw, I am using xen 4.1.5 version with apicv backported, so I am not
> sure whether something broken when backporting or just apicv behaves
> this way.
Can you retest it based on upstream Xen? Just as you suspected, your backporting may be the culprit.
>
>
>
Best regards,
Yang
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-11 7:35 ` Zhang, Yang Z
@ 2015-06-11 8:09 ` Liuqiming (John)
2015-06-11 8:34 ` Zhang, Yang Z
0 siblings, 1 reply; 17+ messages in thread
From: Liuqiming (John) @ 2015-06-11 8:09 UTC (permalink / raw)
To: Zhang, Yang Z; +Cc: xen-devel, peter.huangpeng
I will setup a test environment for xen upstream version but need some time.
Meanwhile, can you give me some idea about what MAY cause this problem?
I have been using xentrace to trace the problem, from what I see, the apicv feature
itself works
apicv=1
2583096 VMEXIT 19027420184 TSC WRMSR
459708 VMEXIT 6924749392 TSC External interrupt
293818 VMEXIT 451974088 TSC Virtualized EOI
843 VMEXIT 54729244 TSC I/O instruction
3260 VMEXIT 15979024 TSC Control-register accesses
1345 VMEXIT 2199736 TSC Exception or non-maskable interrupt (NMI)
39 VMEXIT 1516768 TSC EPT violation
54 VMEXIT 891712 TSC VMCALL
205 VMEXIT 370864 TSC CPUID
apicv=0
3416159 VMEXIT 20929093044 TSC WRMSR
1098428 VMEXIT 11029334704 TSC External interrupt
41128 VMEXIT 64360924 TSC Interrupt window
664 VMEXIT 49245372 TSC I/O instruction
3221 VMEXIT 20116036 TSC Control-register accesses
1401 VMEXIT 2280412 TSC Exception or non-maskable interrupt (NMI)
39 VMEXIT 1581428 TSC EPT violation
53 VMEXIT 749588 TSC VMCALL
205 VMEXIT 355500 TSC CPUID
113 VMEXIT 298568 TSC RDMSR
RDMSR gone,so "APIC Register Virtualization" works
apicv=1
IRQ IRQ_MOVE_CLEANUP_VECTOR( 32): 21
IRQ LOCAL_TIMER_VECTOR( 249): 423804
IRQ CALL_FUNCTION_VECTOR( 251): 1171
IRQ EVENT_CHECK_VECTOR( 252): 1130
IRQ INVALIDATE_TLB_VECTOR( 253): 1
apicv=0
IRQ IRQ_MOVE_CLEANUP_VECTOR( 32): 22
IRQ LAST_DYNAMIC_VECTOR( 223): 27
IRQ LOCAL_TIMER_VECTOR( 249): 448057
IRQ CALL_FUNCTION_VECTOR( 251): 1173
IRQ EVENT_CHECK_VECTOR( 252): 608024
vmexit caused by External interrupt: EVENT_CHECK_VECTOR reduced a lot,
so "Virtual Interrupt Delivery" and "Posted Interrupt Processing" works, I guess.
I think the problem is not caused by apicv itself, maybe some other logic
has conflict with apicv.
On 2015/6/11 15:35, Zhang, Yang Z wrote:
> Liuqiming (John) wrote on 2015-06-11:
> > Hi,
> >
> > Recently I encounter a strange performance problem with APIC virtualization.
> >
> > My host has Intel(R) Xeon(R) CPU E7-4890 v2 CPU installed which support
> > APIC virtualization and x2apic, and there are 4 socket * 15
> > cores_per_socket = 60 core available for VM. There is also a SSD disk on
> > host and the host support vt-d, so I can passsthrough this SSD to VM.
> >
> > A VM created with 60 vcpus, 400G memory and SSD device assigned.
> > I pin these vcpus 1:1 to phisical cpu, and in this VM only keep 15 vcpus
> > online.
> > The problem is: when apicv turn on, a significant performace decrease
> > can be observed and it seems related to cpu topology.
> >
> > I had test follow cases
> > apicv=1:
> > 1) ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> > PIN TO
> > PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> > 2) ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> > PIN TO
> > PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> > apicv=0:
> > 3) ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> > PIN TO
> > PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> > 4) ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> > PIN TO
> > PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> > the result is (the lower the better):
> > 1) 989
> > 2) 2209
> > 3) 978
> > 4) 1130
> >
> > It is a database testcase running on suse11sp3 system in the VM, and I
> > had traced that "read" and "write" syscall get much slower in 2) case.
> >
> > I have disabled NUMA in BIOS, so it seems apicv cause this bad
> > performance when using cpus in different nodes.
> >
> > Can any one shed some light on this?
> >
> > Btw, I am using xen 4.1.5 version with apicv backported, so I am not
> > sure whether something broken when backporting or just apicv behaves
> > this way.
>
> Can you retest it based on upstream Xen? Just as you suspected, your backporting may be the culprit.
>
> >
> >
> >
>
>
> Best regards,
> Yang
>
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-11 8:09 ` Liuqiming (John)
@ 2015-06-11 8:34 ` Zhang, Yang Z
2015-06-18 8:02 ` Liuqiming (John)
0 siblings, 1 reply; 17+ messages in thread
From: Zhang, Yang Z @ 2015-06-11 8:34 UTC (permalink / raw)
To: Liuqiming (John); +Cc: xen-devel, peter.huangpeng
Liuqiming (John) wrote on 2015-06-11:
> I will setup a test environment for xen upstream version but need some time.
> Meanwhile, can you give me some idea about what MAY cause this problem?
How you set the irq affinity of assigned SSD device? IIRC, the irq is migrated with vcpu by default in current Xen and you may not benefit from posted interrupt with pass-through device. So it's better to set the interrupt affinity of the pass-through device in a different core manually.
>
> I have been using xentrace to trace the problem, from what I see, the
> apicv feature itself works
>
> apicv=1
> 2583096 VMEXIT 19027420184 TSC WRMSR
> 459708 VMEXIT 6924749392 TSC External interrupt
> 293818 VMEXIT 451974088 TSC Virtualized
> EOI
> 843 VMEXIT 54729244 TSC I/O
> instruction
> 3260 VMEXIT 15979024 TSC Control-register
> accesses 1345 VMEXIT 2199736 TSC Exception
> or
> non-maskable interrupt (NMI)
> 39 VMEXIT 1516768 TSC EPT violation
> 54 VMEXIT 891712 TSC VMCALL
> 205 VMEXIT 370864 TSC CPUID
> apicv=0
> 3416159 VMEXIT 20929093044 TSC WRMSR
> 1098428 VMEXIT 11029334704 TSC External
> interrupt
> 41128 VMEXIT 64360924 TSC Interrupt
> window
> 664 VMEXIT 49245372 TSC I/O
> instruction
> 3221 VMEXIT 20116036 TSC Control-register
> accesses 1401 VMEXIT 2280412 TSC Exception
> or
> non-maskable interrupt (NMI)
> 39 VMEXIT 1581428 TSC EPT violation
> 53 VMEXIT 749588 TSC VMCALL
> 205 VMEXIT 355500 TSC CPUID
> 113 VMEXIT 298568 TSC RDMSR
> RDMSR gone,so "APIC Register Virtualization" works
>
> apicv=1
> IRQ IRQ_MOVE_CLEANUP_VECTOR( 32):
> 21
> IRQ LOCAL_TIMER_VECTOR( 249):
> 423804
> IRQ CALL_FUNCTION_VECTOR( 251):
> 1171
> IRQ EVENT_CHECK_VECTOR( 252):
> 1130
> IRQ INVALIDATE_TLB_VECTOR( 253):
> 1
> apicv=0
> IRQ IRQ_MOVE_CLEANUP_VECTOR( 32):
> 22
> IRQ LAST_DYNAMIC_VECTOR( 223):
> 27
> IRQ LOCAL_TIMER_VECTOR( 249):
> 448057
> IRQ CALL_FUNCTION_VECTOR( 251):
> 1173
> IRQ EVENT_CHECK_VECTOR( 252):
> 608024
>
> vmexit caused by External interrupt: EVENT_CHECK_VECTOR reduced a lot,
> so "Virtual Interrupt Delivery" and "Posted Interrupt Processing" works, I guess.
Maybe you can turn on the three features one by one(need some changes in code). Then it will help us to know which feature really causes the problem
>
> I think the problem is not caused by apicv itself, maybe some other
> logic has conflict with apicv.
>
> On 2015/6/11 15:35, Zhang, Yang Z wrote:
>> Liuqiming (John) wrote on 2015-06-11:
>>> Hi,
>>>
>>> Recently I encounter a strange performance problem with APIC
>>> virtualization.
>>>
>>> My host has Intel(R) Xeon(R) CPU E7-4890 v2 CPU installed which
>>> support APIC virtualization and x2apic, and there are 4 socket * 15
>>> cores_per_socket = 60 core available for VM. There is also a SSD disk
>>> on host and the host support vt-d, so I can passsthrough this SSD to
>>> VM.
>>>
>>> A VM created with 60 vcpus, 400G memory and SSD device assigned.
>>> I pin these vcpus 1:1 to phisical cpu, and in this VM only keep 15 vcpus
>>> online.
>>> The problem is: when apicv turn on, a significant performace
>>> decrease can be observed and it seems related to cpu topology.
>>>
>>> I had test follow cases
>>> apicv=1:
>>> 1) ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
>>> PIN TO
>>> PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
>>> 2) ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
>>> PIN TO
>>> PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
>>> apicv=0:
>>> 3) ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
>>> PIN TO
>>> PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
>>> 4) ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
>>> PIN TO
>>> PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
>>> the result is (the lower the better):
>>> 1) 989
>>> 2) 2209
>>> 3) 978
>>> 4) 1130
>>>
>>> It is a database testcase running on suse11sp3 system in the VM,
>>> and I had traced that "read" and "write" syscall get much slower in 2) case.
>>>
>>> I have disabled NUMA in BIOS, so it seems apicv cause this bad
>>> performance when using cpus in different nodes.
>>>
>>> Can any one shed some light on this?
>>>
>>> Btw, I am using xen 4.1.5 version with apicv backported, so I am
>>> not sure whether something broken when backporting or just apicv
>>> behaves this way.
>>
>> Can you retest it based on upstream Xen? Just as you suspected, your
>> backporting may be the culprit.
>>
>>>
>>>
>>>
>>
>>
>> Best regards,
>> Yang
>>
>>
>
Best regards,
Yang
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-11 8:34 ` Zhang, Yang Z
@ 2015-06-18 8:02 ` Liuqiming (John)
2015-06-18 8:20 ` Jan Beulich
0 siblings, 1 reply; 17+ messages in thread
From: Liuqiming (John) @ 2015-06-18 8:02 UTC (permalink / raw)
To: Zhang, Yang Z; +Cc: xen-devel, peter.huangpeng
Hi,
We have test on upstream Xen, it seem the upstream version also has the same problem.
When using FIO to test the performance of SSD passthroughed in vm the result show that: When the apicv is on, each EXIT_REASON_MSR_WRITE event spent more time than apicv is off.
Following is the xentrace result:
apicv on:
VMExitCode VMExitReason VMExitCnt VMExitTicks VMExitTicks/VMExitCnt
0x0000000001 EXIT_REASON_EXTERNAL_INTERRUPT 270334 2730912532 10101.99432
0x0000000012 EXIT_REASON_VMCALL 20 438736 21936.8
0x000000001c EXIT_REASON_CR_ACCESS 381340 1096174160 2874.532333
0x000000001e EXIT_REASON_IO_INSTRUCTION 413 32958356 79802.31477
0x0000000020 EXIT_REASON_MSR_WRITE 111830 818317724 7317.515193
0x000000002d EXIT_REASON_EOI_INDUCED 58944 234914864 3985.390608
0x0000000030 EXIT_REASON_EPT_VIOLATION 10 298368 29836.8
Total 822891 4914014740
apicv off:
VMExitCode VMExitReason VMExitCnt VMExitTicks VMExitTicks/VMExitCnt
0x0000000001 EXIT_REASON_EXTERNAL_INTERRUPT 237100 2419717824 10205.47374
0x0000000007 EXIT_REASON_PENDING_VIRT_INTR 792 2324824 2935.383838
0x0000000012 EXIT_REASON_VMCALL 19 415168 21850.94737
0x000000001c EXIT_REASON_CR_ACCESS 406848 1075393292 2643.231113
0x000000001e EXIT_REASON_IO_INSTRUCTION 413 39433068 95479.58354
0x000000001f EXIT_REASON_MSR_READ 48 150528 3136
0x0000000020 EXIT_REASON_MSR_WRITE 229609 1004000084 4372.651264
0x0000000030 EXIT_REASON_EPT_VIOLATION 10 249172 24917.2
Total 874839 4541683960
On 2015/6/11 16:34, Zhang, Yang Z wrote:
> Liuqiming (John) wrote on 2015-06-11:
> > I will setup a test environment for xen upstream version but need some time.
> > Meanwhile, can you give me some idea about what MAY cause this problem?
>
> How you set the irq affinity of assigned SSD device? IIRC, the irq is migrated with vcpu by default in current Xen and you may not benefit from posted interrupt with pass-through device. So it's better to set the interrupt affinity of the pass-through device in a different core manually.
>
> >
> > I have been using xentrace to trace the problem, from what I see, the
> > apicv feature itself works
> >
> > apicv=1
> > 2583096 VMEXIT 19027420184 TSC WRMSR
> > 459708 VMEXIT 6924749392 TSC External interrupt
> > 293818 VMEXIT 451974088 TSC Virtualized
> > EOI
> > 843 VMEXIT 54729244 TSC I/O
> > instruction
> > 3260 VMEXIT 15979024 TSC Control-register
> > accesses 1345 VMEXIT 2199736 TSC Exception
> > or
> > non-maskable interrupt (NMI)
> > 39 VMEXIT 1516768 TSC EPT violation
> > 54 VMEXIT 891712 TSC VMCALL
> > 205 VMEXIT 370864 TSC CPUID
> > apicv=0
> > 3416159 VMEXIT 20929093044 TSC WRMSR
> > 1098428 VMEXIT 11029334704 TSC External
> > interrupt
> > 41128 VMEXIT 64360924 TSC Interrupt
> > window
> > 664 VMEXIT 49245372 TSC I/O
> > instruction
> > 3221 VMEXIT 20116036 TSC Control-register
> > accesses 1401 VMEXIT 2280412 TSC Exception
> > or
> > non-maskable interrupt (NMI)
> > 39 VMEXIT 1581428 TSC EPT violation
> > 53 VMEXIT 749588 TSC VMCALL
> > 205 VMEXIT 355500 TSC CPUID
> > 113 VMEXIT 298568 TSC RDMSR
> > RDMSR gone,so "APIC Register Virtualization" works
> >
> > apicv=1
> > IRQ IRQ_MOVE_CLEANUP_VECTOR( 32):
> > 21
> > IRQ LOCAL_TIMER_VECTOR( 249):
> > 423804
> > IRQ CALL_FUNCTION_VECTOR( 251):
> > 1171
> > IRQ EVENT_CHECK_VECTOR( 252):
> > 1130
> > IRQ INVALIDATE_TLB_VECTOR( 253):
> > 1
> > apicv=0
> > IRQ IRQ_MOVE_CLEANUP_VECTOR( 32):
> > 22
> > IRQ LAST_DYNAMIC_VECTOR( 223):
> > 27
> > IRQ LOCAL_TIMER_VECTOR( 249):
> > 448057
> > IRQ CALL_FUNCTION_VECTOR( 251):
> > 1173
> > IRQ EVENT_CHECK_VECTOR( 252):
> > 608024
> >
> > vmexit caused by External interrupt: EVENT_CHECK_VECTOR reduced a lot,
> > so "Virtual Interrupt Delivery" and "Posted Interrupt Processing" works, I guess.
>
> Maybe you can turn on the three features one by one(need some changes in code). Then it will help us to know which feature really causes the problem
>
> >
> > I think the problem is not caused by apicv itself, maybe some other
> > logic has conflict with apicv.
> >
> > On 2015/6/11 15:35, Zhang, Yang Z wrote:
> >> Liuqiming (John) wrote on 2015-06-11:
> >>> Hi,
> >>>
> >>> Recently I encounter a strange performance problem with APIC
> >>> virtualization.
> >>>
> >>> My host has Intel(R) Xeon(R) CPU E7-4890 v2 CPU installed which
> >>> support APIC virtualization and x2apic, and there are 4 socket * 15
> >>> cores_per_socket = 60 core available for VM. There is also a SSD disk
> >>> on host and the host support vt-d, so I can passsthrough this SSD to
> >>> VM.
> >>>
> >>> A VM created with 60 vcpus, 400G memory and SSD device assigned.
> >>> I pin these vcpus 1:1 to phisical cpu, and in this VM only keep 15 vcpus
> >>> online.
> >>> The problem is: when apicv turn on, a significant performace
> >>> decrease can be observed and it seems related to cpu topology.
> >>>
> >>> I had test follow cases
> >>> apicv=1:
> >>> 1) ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> >>> PIN TO
> >>> PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> >>> 2) ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> >>> PIN TO
> >>> PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> >>> apicv=0:
> >>> 3) ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> >>> PIN TO
> >>> PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> >>> 4) ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> >>> PIN TO
> >>> PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> >>> the result is (the lower the better):
> >>> 1) 989
> >>> 2) 2209
> >>> 3) 978
> >>> 4) 1130
> >>>
> >>> It is a database testcase running on suse11sp3 system in the VM,
> >>> and I had traced that "read" and "write" syscall get much slower in 2) case.
> >>>
> >>> I have disabled NUMA in BIOS, so it seems apicv cause this bad
> >>> performance when using cpus in different nodes.
> >>>
> >>> Can any one shed some light on this?
> >>>
> >>> Btw, I am using xen 4.1.5 version with apicv backported, so I am
> >>> not sure whether something broken when backporting or just apicv
> >>> behaves this way.
> >>
> >> Can you retest it based on upstream Xen? Just as you suspected, your
> >> backporting may be the culprit.
> >>
> >>>
> >>>
> >>>
> >>
> >>
> >> Best regards,
> >> Yang
> >>
> >>
> >
>
>
> Best regards,
> Yang
>
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-18 8:02 ` Liuqiming (John)
@ 2015-06-18 8:20 ` Jan Beulich
2015-06-18 8:52 ` Jan Beulich
2015-06-18 9:31 ` Zhang, Yang Z
0 siblings, 2 replies; 17+ messages in thread
From: Jan Beulich @ 2015-06-18 8:20 UTC (permalink / raw)
To: Liuqiming (John); +Cc: Yang Z Zhang, xen-devel, peter.huangpeng
>>> On 18.06.15 at 10:02, <john.liuqiming@huawei.com> wrote:
> When using FIO to test the performance of SSD passthroughed in vm the
> result show that: When the apicv is on, each EXIT_REASON_MSR_WRITE event
> spent more time than apicv is off.
>
> Following is the xentrace result:
>
> apicv on:
>
> VMExitCode VMExitReason VMExitCnt VMExitTicks VMExitTicks/VMExitCnt
>
> 0x0000000001 EXIT_REASON_EXTERNAL_INTERRUPT 270334 2730912532 10101.99432
> 0x0000000012 EXIT_REASON_VMCALL 20 438736 21936.8
> 0x000000001c EXIT_REASON_CR_ACCESS 381340 1096174160 2874.532333
> 0x000000001e EXIT_REASON_IO_INSTRUCTION 413 32958356 79802.31477
> 0x0000000020 EXIT_REASON_MSR_WRITE 111830 818317724 7317.515193
> 0x000000002d EXIT_REASON_EOI_INDUCED 58944 234914864 3985.390608
> 0x0000000030 EXIT_REASON_EPT_VIOLATION 10 298368 29836.8
>
> Total 822891 4914014740
>
> apicv off:
>
> VMExitCode VMExitReason VMExitCnt VMExitTicks VMExitTicks/VMExitCnt
>
> 0x0000000001 EXIT_REASON_EXTERNAL_INTERRUPT 237100 2419717824 10205.47374
> 0x0000000007 EXIT_REASON_PENDING_VIRT_INTR 792 2324824 2935.383838
> 0x0000000012 EXIT_REASON_VMCALL 19 415168 21850.94737
> 0x000000001c EXIT_REASON_CR_ACCESS 406848 1075393292 2643.231113
> 0x000000001e EXIT_REASON_IO_INSTRUCTION 413 39433068 95479.58354
> 0x000000001f EXIT_REASON_MSR_READ 48 150528 3136
> 0x0000000020 EXIT_REASON_MSR_WRITE 229609 1004000084 4372.651264
> 0x0000000030 EXIT_REASON_EPT_VIOLATION 10 249172 24917.2
>
> Total 874839 4541683960
And did you drill into _which_ MSR(s) are requiring this much longer to
have their writes handled? After all, that's the relevant thing, provided
the increase of this indeed has anything to do with the performance
issue you're seeing (the absolute increase of 200M ticks there doesn't
mean much for the performance effect without knowing what the total
execution time was).
Apart from that I notice that the EXIT_REASON_EOI_INDUCED handling
also adds about the same number of ticks...
Jan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-18 8:20 ` Jan Beulich
@ 2015-06-18 8:52 ` Jan Beulich
2015-06-18 8:53 ` Zhang, Yang Z
2015-06-18 9:31 ` Zhang, Yang Z
1 sibling, 1 reply; 17+ messages in thread
From: Jan Beulich @ 2015-06-18 8:52 UTC (permalink / raw)
To: Liuqiming (John); +Cc: Yang Z Zhang, xen-devel, peter.huangpeng
>>> On 18.06.15 at 10:20, <JBeulich@suse.com> wrote:
> Apart from that I notice that the EXIT_REASON_EOI_INDUCED handling
> also adds about the same number of ticks...
Are there any other devices in the guest causing meaningful amounts
of interrupts (I suppose the SSD itself is using just one)? I ask since I
wonder whether e.g. there may be lock contention in
vioapic_update_EOI().
Jan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-18 8:52 ` Jan Beulich
@ 2015-06-18 8:53 ` Zhang, Yang Z
2015-06-18 9:15 ` Jan Beulich
0 siblings, 1 reply; 17+ messages in thread
From: Zhang, Yang Z @ 2015-06-18 8:53 UTC (permalink / raw)
To: Jan Beulich, Liuqiming (John); +Cc: xen-devel, peter.huangpeng
Jan Beulich wrote on 2015-06-18:
>>>> On 18.06.15 at 10:20, <JBeulich@suse.com> wrote:
>> Apart from that I notice that the EXIT_REASON_EOI_INDUCED handling
>> also adds about the same number of ticks...
>
> Are there any other devices in the guest causing meaningful amounts of
> interrupts (I suppose the SSD itself is using just one)? I ask since I
> wonder whether e.g. there may be lock contention in vioapic_update_EOI().
The EXIT_REASON_EOI_INDUCED is for local apic time.
>
> Jan
Best regards,
Yang
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-18 8:53 ` Zhang, Yang Z
@ 2015-06-18 9:15 ` Jan Beulich
2015-06-18 9:18 ` Zhang, Yang Z
0 siblings, 1 reply; 17+ messages in thread
From: Jan Beulich @ 2015-06-18 9:15 UTC (permalink / raw)
To: Yang Z Zhang; +Cc: Liuqiming (John), xen-devel, peter.huangpeng
>>> On 18.06.15 at 10:53, <yang.z.zhang@intel.com> wrote:
> Jan Beulich wrote on 2015-06-18:
>>>>> On 18.06.15 at 10:20, <JBeulich@suse.com> wrote:
>>> Apart from that I notice that the EXIT_REASON_EOI_INDUCED handling
>>> also adds about the same number of ticks...
>>
>> Are there any other devices in the guest causing meaningful amounts of
>> interrupts (I suppose the SSD itself is using just one)? I ask since I
>> wonder whether e.g. there may be lock contention in vioapic_update_EOI().
>
> The EXIT_REASON_EOI_INDUCED is for local apic time.
??? (I.e. I don't see the connection between the context and your
reply. Perhaps I'm missing something obvious...)
Jan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-18 9:15 ` Jan Beulich
@ 2015-06-18 9:18 ` Zhang, Yang Z
2015-06-18 9:35 ` Jan Beulich
0 siblings, 1 reply; 17+ messages in thread
From: Zhang, Yang Z @ 2015-06-18 9:18 UTC (permalink / raw)
To: Jan Beulich; +Cc: Liuqiming (John), xen-devel, peter.huangpeng
Jan Beulich wrote on 2015-06-18:
>>>> On 18.06.15 at 10:53, <yang.z.zhang@intel.com> wrote:
>> Jan Beulich wrote on 2015-06-18:
>>>>>> On 18.06.15 at 10:20, <JBeulich@suse.com> wrote:
>>>> Apart from that I notice that the EXIT_REASON_EOI_INDUCED handling
>>>> also adds about the same number of ticks...
>>>
>>> Are there any other devices in the guest causing meaningful amounts
>>> of interrupts (I suppose the SSD itself is using just one)? I ask
>>> since I wonder whether e.g. there may be lock contention in
> vioapic_update_EOI().
>>
>> The EXIT_REASON_EOI_INDUCED is for local apic time.
>
> ??? (I.e. I don't see the connection between the context and your reply.
> Perhaps I'm missing something obvious...)
I mean the lots of EXIT_REASON_EOI_INDUCEDis for local apic time which is edge-trigger. So it will not call vioapic_update_EOI.
>
> Jan
Best regards,
Yang
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-18 8:20 ` Jan Beulich
2015-06-18 8:52 ` Jan Beulich
@ 2015-06-18 9:31 ` Zhang, Yang Z
1 sibling, 0 replies; 17+ messages in thread
From: Zhang, Yang Z @ 2015-06-18 9:31 UTC (permalink / raw)
To: Jan Beulich, Liuqiming (John); +Cc: xen-devel, peter.huangpeng
Jan Beulich wrote on 2015-06-18:
>>>> On 18.06.15 at 10:02, <john.liuqiming@huawei.com> wrote:
>> When using FIO to test the performance of SSD passthroughed in vm
>> the result show that: When the apicv is on, each
>> EXIT_REASON_MSR_WRITE event spent more time than apicv is off.
>>
>> Following is the xentrace result:
>>
>> apicv on:
>>
>> VMExitCode VMExitReason
> VMExitCnt VMExitTicks VMExitTicks/VMExitCnt
>>
>> 0x0000000001 EXIT_REASON_EXTERNAL_INTERRUPT 270334
>> 2730912532 10101.99432 0x0000000012
>> EXIT_REASON_VMCALL 20 438736 21936.8 0x000000001c
>> EXIT_REASON_CR_ACCESS 381340 1096174160
>> 2874.532333 0x000000001e EXIT_REASON_IO_INSTRUCTION 413
>> 32958356 79802.31477 0x0000000020
>> EXIT_REASON_MSR_WRITE 111830 818317724 7317.515193
>> 0x000000002d EXIT_REASON_EOI_INDUCED 58944
>> 234914864 3985.390608 0x0000000030
>> EXIT_REASON_EPT_VIOLATION 10 298368 29836.8
>>
>> Total 822891 4914014740
>>
>> apicv off:
>>
>> VMExitCode VMExitReason
> VMExitCnt VMExitTicks VMExitTicks/VMExitCnt
>>
>> 0x0000000001 EXIT_REASON_EXTERNAL_INTERRUPT 237100
>> 2419717824 10205.47374 0x0000000007
>> EXIT_REASON_PENDING_VIRT_INTR 792 2324824 2935.383838
>> 0x0000000012 EXIT_REASON_VMCALL 19
>> 415168 21850.94737 0x000000001c
>> EXIT_REASON_CR_ACCESS 406848 1075393292 2643.231113
>> 0x000000001e EXIT_REASON_IO_INSTRUCTION 413
>> 39433068 95479.58354 0x000000001f
>> EXIT_REASON_MSR_READ 48 150528 3136 0x0000000020
>> EXIT_REASON_MSR_WRITE 229609 1004000084
>> 4372.651264 0x0000000030 EXIT_REASON_EPT_VIOLATION 10
>> 249172 24917.2
>>
>> Total
> 874839 4541683960
>
> And did you drill into _which_ MSR(s) are requiring this much longer
> to have their writes handled? After all, that's the relevant thing,
> provided the increase of this indeed has anything to do with the
> performance issue you're seeing (the absolute increase of 200M ticks
> there doesn't mean much for the performance effect without knowing what the total execution time was).
IIRC, APICv doesn't impact the WRMSR's behavior (except EOI, self-IPI and TPR). Cannot understand why APICv will impact it.
>
> Apart from that I notice that the EXIT_REASON_EOI_INDUCED handling
> also adds about the same number of ticks...
>
> Jan
Best regards,
Yang
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-18 9:18 ` Zhang, Yang Z
@ 2015-06-18 9:35 ` Jan Beulich
2015-06-19 14:23 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 17+ messages in thread
From: Jan Beulich @ 2015-06-18 9:35 UTC (permalink / raw)
To: Yang Z Zhang; +Cc: Liuqiming (John), xen-devel, peter.huangpeng
>>> On 18.06.15 at 11:18, <yang.z.zhang@intel.com> wrote:
> Jan Beulich wrote on 2015-06-18:
>>>>> On 18.06.15 at 10:53, <yang.z.zhang@intel.com> wrote:
>>> Jan Beulich wrote on 2015-06-18:
>>>>>>> On 18.06.15 at 10:20, <JBeulich@suse.com> wrote:
>>>>> Apart from that I notice that the EXIT_REASON_EOI_INDUCED handling
>>>>> also adds about the same number of ticks...
>>>>
>>>> Are there any other devices in the guest causing meaningful amounts
>>>> of interrupts (I suppose the SSD itself is using just one)? I ask
>>>> since I wonder whether e.g. there may be lock contention in
>> vioapic_update_EOI().
>>>
>>> The EXIT_REASON_EOI_INDUCED is for local apic time.
>>
>> ??? (I.e. I don't see the connection between the context and your reply.
>> Perhaps I'm missing something obvious...)
>
> I mean the lots of EXIT_REASON_EOI_INDUCEDis for local apic time which is
> edge-trigger. So it will not call vioapic_update_EOI.
Okay, this then suggests the problem being hvm_dpci_msi_eoi(),
albeit looking at it I can't immediately see why it would take long.
Jan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-18 9:35 ` Jan Beulich
@ 2015-06-19 14:23 ` Konrad Rzeszutek Wilk
0 siblings, 0 replies; 17+ messages in thread
From: Konrad Rzeszutek Wilk @ 2015-06-19 14:23 UTC (permalink / raw)
To: Jan Beulich; +Cc: Yang Z Zhang, Liuqiming (John), xen-devel, peter.huangpeng
On Thu, Jun 18, 2015 at 10:35:37AM +0100, Jan Beulich wrote:
> >>> On 18.06.15 at 11:18, <yang.z.zhang@intel.com> wrote:
> > Jan Beulich wrote on 2015-06-18:
> >>>>> On 18.06.15 at 10:53, <yang.z.zhang@intel.com> wrote:
> >>> Jan Beulich wrote on 2015-06-18:
> >>>>>>> On 18.06.15 at 10:20, <JBeulich@suse.com> wrote:
> >>>>> Apart from that I notice that the EXIT_REASON_EOI_INDUCED handling
> >>>>> also adds about the same number of ticks...
> >>>>
> >>>> Are there any other devices in the guest causing meaningful amounts
> >>>> of interrupts (I suppose the SSD itself is using just one)? I ask
> >>>> since I wonder whether e.g. there may be lock contention in
> >> vioapic_update_EOI().
> >>>
> >>> The EXIT_REASON_EOI_INDUCED is for local apic time.
> >>
> >> ??? (I.e. I don't see the connection between the context and your reply.
> >> Perhaps I'm missing something obvious...)
> >
> > I mean the lots of EXIT_REASON_EOI_INDUCEDis for local apic time which is
> > edge-trigger. So it will not call vioapic_update_EOI.
>
> Okay, this then suggests the problem being hvm_dpci_msi_eoi(),
> albeit looking at it I can't immediately see why it would take long.
We do take two spin-locks (d->event_lock) and (desc->lock) and also
can IPI set_eoi_ready across different CPUs.
Perhaps it might be worth adding some TRACE with the time taken
to get the locks and also doing the IPIs?
>
> Jan
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-22 2:48 ` Jiangyifei
@ 2015-06-22 9:56 ` Jan Beulich
0 siblings, 0 replies; 17+ messages in thread
From: Jan Beulich @ 2015-06-22 9:56 UTC (permalink / raw)
To: Jiangyifei
Cc: yang.z.zhang, Liuqiming (John), Huangpeng (Peter),
Xuzhichuang, xen-devel
>>> On 22.06.15 at 04:48, <jiangyifei@huawei.com> wrote:
> We found that all the events are HVM_EMUL_LAPIC_START_TIMER. The number of
> TRC_HVM_EMUL_LAPIC_STOP_TIMER event is zero.
>
> Below is the details:
>
> apicv on:
> TRC_HVM_EMUL_LAPIC_START_TIMER: count=111480
> TRC_HVM_EMUL_LAPIC_STOP_TIMER: count=0
> apicv off:
> TRC_HVM_EMUL_LAPIC_START_TIMER: count=13595
> TRC_HVM_EMUL_LAPIC_STOP_TIMER: count=0
>
> I want to know what the bad effects of create_periodic_time() being
> repeatedly called are.
Calling the function the way it is being done here, there is a drift in
the time the next interrupt gets signaled, due to the delay incurred
by the code doing the write emulation. See hvm/rtc.c (and its
history) for further details as well as how the issue got addressed
there. It would certainly be nice if you could see whether a similar
approach can be taken (and would help) here. (A fundamental
question, still unanswered, is over what time period these 111k
timer adjustments happened - said problem matters only when
there are many such updates in a relatively short period of time.)
Jan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-19 10:42 ` Jan Beulich
@ 2015-06-22 2:48 ` Jiangyifei
2015-06-22 9:56 ` Jan Beulich
0 siblings, 1 reply; 17+ messages in thread
From: Jiangyifei @ 2015-06-22 2:48 UTC (permalink / raw)
To: Jan Beulich, xen-devel
Cc: yang.z.zhang, Liuqiming (John), Huangpeng (Peter), Xuzhichuang
Hi,
We found that all the events are HVM_EMUL_LAPIC_START_TIMER. The number of TRC_HVM_EMUL_LAPIC_STOP_TIMER event is zero.
Below is the details:
apicv on:
TRC_HVM_EMUL_LAPIC_START_TIMER: count=111480
TRC_HVM_EMUL_LAPIC_STOP_TIMER: count=0
apicv off:
TRC_HVM_EMUL_LAPIC_START_TIMER: count=13595
TRC_HVM_EMUL_LAPIC_STOP_TIMER: count=0
I want to know what the bad effects of create_periodic_time() being repeatedly called are.
Regards
Yifei jiang
-----Original Message-----
From: Jan Beulich [mailto:JBeulich@suse.com]
Sent: Friday, June 19, 2015 6:43 PM
To: Jiangyifei
Cc: Liuqiming (John); Huangpeng (Peter); Xuzhichuang; yang.z.zhang@Intel.com; xen-devel@lists.xen.org
Subject: RE: [Xen-devel] performace issue when turn on apicv
>>> On 18.06.15 at 16:55, <jiangyifei@huawei.com> wrote:
> I'm John's colleague. We looked into the details of the tracing data,
> and found that the number of MSR_IA32_APICTMICT_MSR event is quite
> high when apic-v is enabled(about 9x more compared with apic-v
> disabled).
>
> Below is the details:
>
> EXIT_REASON_MSR_WRITE
> apicv on:
> MSR= 0x00000838(MSR_IA32_APICTMICT_MSR) count= 111480
> MSR= 0x00000830(x2APIC Interrupt Command Register) count= 350
> Total count = 111830
>
> apicv off:
> MSR= 0x00000838(MSR_IA32_APICTMICT_MSR) count= 13595
> MSR= 0x00000830(x2APIC Interrupt Command Register) count= 254
> MSR= 0x0000080b(MSR_IA32_APICEOI_MSR) count= 215760
> Total count = 229609
Yes, accesses to this MSR get specifically intercepted when using APICv (see vmx_vlapic_msr_changed()). I'm therefore surprised that you see more than a handful of writes come through in the opposite case.
As the processing of the writes themselves isn't very involved, I wonder whether the higher count really contributes to your problem in a measurable way. I wonder whether bad effects of
create_periodic_time() being repeatedly called (we observed such
before) might instead play a role here. Iirc you had taken a trace already - could you extract a meaningful subset of HVM_EMUL_LAPIC_{START,STOP}_TIMER events from it?
Jan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
2015-06-18 14:55 ` Jiangyifei
@ 2015-06-19 10:42 ` Jan Beulich
2015-06-22 2:48 ` Jiangyifei
0 siblings, 1 reply; 17+ messages in thread
From: Jan Beulich @ 2015-06-19 10:42 UTC (permalink / raw)
To: Jiangyifei
Cc: yang.z.zhang, Liuqiming (John), Huangpeng (Peter),
Xuzhichuang, xen-devel
>>> On 18.06.15 at 16:55, <jiangyifei@huawei.com> wrote:
> I'm John's colleague. We looked into the details of the tracing data, and
> found that the number of MSR_IA32_APICTMICT_MSR
> event is quite high when apic-v is enabled(about 9x more compared with apic-v
> disabled).
>
> Below is the details:
>
> EXIT_REASON_MSR_WRITE
> apicv on:
> MSR= 0x00000838(MSR_IA32_APICTMICT_MSR) count= 111480
> MSR= 0x00000830(x2APIC Interrupt Command Register) count= 350
> Total count = 111830
>
> apicv off:
> MSR= 0x00000838(MSR_IA32_APICTMICT_MSR) count= 13595
> MSR= 0x00000830(x2APIC Interrupt Command Register) count= 254
> MSR= 0x0000080b(MSR_IA32_APICEOI_MSR) count= 215760
> Total count = 229609
Yes, accesses to this MSR get specifically intercepted when using
APICv (see vmx_vlapic_msr_changed()). I'm therefore surprised
that you see more than a handful of writes come through in the
opposite case.
As the processing of the writes themselves isn't very involved, I
wonder whether the higher count really contributes to your
problem in a measurable way. I wonder whether bad effects of
create_periodic_time() being repeatedly called (we observed such
before) might instead play a role here. Iirc you had taken a
trace already - could you extract a meaningful subset of
HVM_EMUL_LAPIC_{START,STOP}_TIMER events from it?
Jan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: performace issue when turn on apicv
[not found] <mailman.2928.1434616580.28840.xen-devel@lists.xen.org>
@ 2015-06-18 14:55 ` Jiangyifei
2015-06-19 10:42 ` Jan Beulich
0 siblings, 1 reply; 17+ messages in thread
From: Jiangyifei @ 2015-06-18 14:55 UTC (permalink / raw)
To: xen-devel, Jan Beulich
Cc: yang.z.zhang, Liuqiming (John), Jiangyifei, Huangpeng (Peter),
Xuzhichuang
Hi,
I'm John's colleague. We looked into the details of the tracing data, and found that the number of MSR_IA32_APICTMICT_MSR
event is quite high when apic-v is enabled(about 9x more compared with apic-v disabled).
Below is the details:
EXIT_REASON_MSR_WRITE
apicv on:
MSR= 0x00000838(MSR_IA32_APICTMICT_MSR) count= 111480
MSR= 0x00000830(x2APIC Interrupt Command Register) count= 350
Total count = 111830
apicv off:
MSR= 0x00000838(MSR_IA32_APICTMICT_MSR) count= 13595
MSR= 0x00000830(x2APIC Interrupt Command Register) count= 254
MSR= 0x0000080b(MSR_IA32_APICEOI_MSR) count= 215760
Total count = 229609
If there is anything need to get, pls let me know.
Thanks.
Regards
Yifei jiang
-----Original Message-----
Date: Thu, 18 Jun 2015 09:20:39 +0100
From: "Jan Beulich" <JBeulich@suse.com>
To: "Liuqiming (John)" <john.liuqiming@huawei.com>
Cc: Yang Z Zhang <yang.z.zhang@intel.com>,
"xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>,
"peter.huangpeng@huawei.com" <peter.huangpeng@huawei.com>
Subject: Re: [Xen-devel] performace issue when turn on apicv
Message-ID: <55829B770200007800086752@mail.emea.novell.com>
Content-Type: text/plain; charset=US-ASCII
>>> On 18.06.15 at 10:02, <john.liuqiming@huawei.com> wrote:
> When using FIO to test the performance of SSD passthroughed in vm the
> result show that: When the apicv is on, each EXIT_REASON_MSR_WRITE event
> spent more time than apicv is off.
>
> Following is the xentrace result:
>
> apicv on:
>
> VMExitCode VMExitReason VMExitCnt VMExitTicks VMExitTicks/VMExitCnt
>
> 0x0000000001 EXIT_REASON_EXTERNAL_INTERRUPT 270334 2730912532 10101.99432
> 0x0000000012 EXIT_REASON_VMCALL 20 438736 21936.8
> 0x000000001c EXIT_REASON_CR_ACCESS 381340 1096174160 2874.532333
> 0x000000001e EXIT_REASON_IO_INSTRUCTION 413 32958356 79802.31477
> 0x0000000020 EXIT_REASON_MSR_WRITE 111830 818317724 7317.515193
> 0x000000002d EXIT_REASON_EOI_INDUCED 58944 234914864 3985.390608
> 0x0000000030 EXIT_REASON_EPT_VIOLATION 10 298368 29836.8
>
> Total 822891 4914014740
>
> apicv off:
>
> VMExitCode VMExitReason VMExitCnt VMExitTicks VMExitTicks/VMExitCnt
>
> 0x0000000001 EXIT_REASON_EXTERNAL_INTERRUPT 237100 2419717824 10205.47374
> 0x0000000007 EXIT_REASON_PENDING_VIRT_INTR 792 2324824 2935.383838
> 0x0000000012 EXIT_REASON_VMCALL 19 415168 21850.94737
> 0x000000001c EXIT_REASON_CR_ACCESS 406848 1075393292 2643.231113
> 0x000000001e EXIT_REASON_IO_INSTRUCTION 413 39433068 95479.58354
> 0x000000001f EXIT_REASON_MSR_READ 48 150528 3136
> 0x0000000020 EXIT_REASON_MSR_WRITE 229609 1004000084 4372.651264
> 0x0000000030 EXIT_REASON_EPT_VIOLATION 10 249172 24917.2
>
> Total 874839 4541683960
And did you drill into _which_ MSR(s) are requiring this much longer to
have their writes handled? After all, that's the relevant thing, provided
the increase of this indeed has anything to do with the performance
issue you're seeing (the absolute increase of 200M ticks there doesn't
mean much for the performance effect without knowing what the total
execution time was).
Apart from that I notice that the EXIT_REASON_EOI_INDUCED handling
also adds about the same number of ticks...
Jan
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2015-06-22 9:56 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-11 7:23 performace issue when turn on apicv Liuqiming (John)
2015-06-11 7:35 ` Zhang, Yang Z
2015-06-11 8:09 ` Liuqiming (John)
2015-06-11 8:34 ` Zhang, Yang Z
2015-06-18 8:02 ` Liuqiming (John)
2015-06-18 8:20 ` Jan Beulich
2015-06-18 8:52 ` Jan Beulich
2015-06-18 8:53 ` Zhang, Yang Z
2015-06-18 9:15 ` Jan Beulich
2015-06-18 9:18 ` Zhang, Yang Z
2015-06-18 9:35 ` Jan Beulich
2015-06-19 14:23 ` Konrad Rzeszutek Wilk
2015-06-18 9:31 ` Zhang, Yang Z
[not found] <mailman.2928.1434616580.28840.xen-devel@lists.xen.org>
2015-06-18 14:55 ` Jiangyifei
2015-06-19 10:42 ` Jan Beulich
2015-06-22 2:48 ` Jiangyifei
2015-06-22 9:56 ` Jan Beulich
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.