How to handle TLB shootdown IPI on Xen

How to handle TLB shootdown IPI on Xen

[J Ahn]

[-- Attachment #1.1: Type: text/plain, Size: 957 bytes --]

All,

Currently, I have used Xen 4.2.1 with Ubuntu HVM guests.

I am just wondering how to handle the TLB shootdown IPIs in Xen.
Specifically, the guest will send the TLB shootdown IPI to remote CPUs if
TLB synchronization between CPUs is required.

In this case, I guess the hypervisor intervention is required because it
needs to route the interrupt to virtual CPUs of corresponding guest.
However, Xen seems not to intercept the event because the number of IPIs
from Xenperf is very low. Is this right?

I checked the number of IPIs is relatively high in guest OS when using 'cat
/proc/interrupts'.
I also checked the system does not invoke the '*invalidate_interrupt*'
function located in *'arch/x86/smp.c*'.

If there is no need to intervene to deal with that, could anyone explain
that how to maintain the consistent view of TLBs in the guest hosted by Xen?
Otherwise, please let me know the handling TLB shootdown IPI code in Xen.

Thanks

Jeongseob

[-- Attachment #1.2: Type: text/html, Size: 1268 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: How to handle TLB shootdown IPI on Xen

[Jan Beulich]

>>> On 11.08.13 at 18:22, J Ahn <jeong.seob.ahn@gmail.com> wrote:
> I am just wondering how to handle the TLB shootdown IPIs in Xen.
> Specifically, the guest will send the TLB shootdown IPI to remote CPUs if
> TLB synchronization between CPUs is required.
> 
> In this case, I guess the hypervisor intervention is required because it
> needs to route the interrupt to virtual CPUs of corresponding guest.
> However, Xen seems not to intercept the event because the number of IPIs
> from Xenperf is very low. Is this right?
> 
> I checked the number of IPIs is relatively high in guest OS when using 'cat
> /proc/interrupts'.
> I also checked the system does not invoke the '*invalidate_interrupt*'
> function located in *'arch/x86/smp.c*'.
> 
> If there is no need to intervene to deal with that, could anyone explain
> that how to maintain the consistent view of TLBs in the guest hosted by Xen?
> Otherwise, please let me know the handling TLB shootdown IPI code in Xen.

I'm getting the impression that you simply mix up physical IPIs (as
used by the hypervisor) and their virtualized counterparts (as used
by HVM guests).

Jan

Re: How to handle TLB shootdown IPI on Xen

[J Ahn]

[-- Attachment #1.1: Type: text/plain, Size: 1296 bytes --]

Could you tell me the how to handle the virtualized IPI on Xen ??




2013/8/12 Jan Beulich <JBeulich@suse.com>

> >>> On 11.08.13 at 18:22, J Ahn <jeong.seob.ahn@gmail.com> wrote:
> > I am just wondering how to handle the TLB shootdown IPIs in Xen.
> > Specifically, the guest will send the TLB shootdown IPI to remote CPUs if
> > TLB synchronization between CPUs is required.
> >
> > In this case, I guess the hypervisor intervention is required because it
> > needs to route the interrupt to virtual CPUs of corresponding guest.
> > However, Xen seems not to intercept the event because the number of IPIs
> > from Xenperf is very low. Is this right?
> >
> > I checked the number of IPIs is relatively high in guest OS when using
> 'cat
> > /proc/interrupts'.
> > I also checked the system does not invoke the '*invalidate_interrupt*'
> > function located in *'arch/x86/smp.c*'.
> >
> > If there is no need to intervene to deal with that, could anyone explain
> > that how to maintain the consistent view of TLBs in the guest hosted by
> Xen?
> > Otherwise, please let me know the handling TLB shootdown IPI code in Xen.
>
> I'm getting the impression that you simply mix up physical IPIs (as
> used by the hypervisor) and their virtualized counterparts (as used
> by HVM guests).
>
> Jan
>
>

[-- Attachment #1.2: Type: text/html, Size: 1918 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: How to handle TLB shootdown IPI on Xen

[Jan Beulich]

>>> On 12.08.13 at 14:47, J Ahn <jeong.seob.ahn@gmail.com> wrote:

Please don't top post.

> Could you tell me the how to handle the virtualized IPI on Xen ??

The virtualized IPI gets handled by the guest OS, not by Xen.

Jan

Re: How to handle TLB shootdown IPI on Xen

[J Ahn]

[-- Attachment #1.1: Type: text/plain, Size: 489 bytes --]

2013/8/12 Jan Beulich <JBeulich@suse.com>

> >>> On 12.08.13 at 14:47, J Ahn <jeong.seob.ahn@gmail.com> wrote:
>
> Please don't top post.
>
> > Could you tell me the how to handle the virtualized IPI on Xen ??
>
> The virtualized IPI gets handled by the guest OS, not by Xen.
>
> Jan
>
>
Sorry, I asked you the wrong question.

I was wondering how Xen works when the guest sends the TLB shootdown IPI.
You mean that there is no need to intercept the IPI on Xen.
Is my understanding right?

[-- Attachment #1.2: Type: text/html, Size: 1127 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: How to handle TLB shootdown IPI on Xen

[Jan Beulich]

>>> On 12.08.13 at 18:04, J Ahn <jeong.seob.ahn@gmail.com> wrote:
> I was wondering how Xen works when the guest sends the TLB shootdown IPI.
> You mean that there is no need to intercept the IPI on Xen.
> Is my understanding right?

Xen is just involved in the delivery of the virtual interrupt, but
doesn't participate in its handling.

Jan

Re: How to handle TLB shootdown IPI on Xen

[J Ahn]

[-- Attachment #1.1: Type: text/plain, Size: 777 bytes --]

2013/8/13 Jan Beulich <JBeulich@suse.com>

> >>> On 12.08.13 at 18:04, J Ahn <jeong.seob.ahn@gmail.com> wrote:
> > I was wondering how Xen works when the guest sends the TLB shootdown IPI.
> > You mean that there is no need to intercept the IPI on Xen.
> > Is my understanding right?
>
> Xen is just involved in the delivery of the virtual interrupt, but
> doesn't participate in its handling.
>
> Jan
>
>
As I mentioned the first post, I also guess that the hypervisor
intervention is required to route the virtual IPI.
So, I would like to know where the delivery code on Xen has been located
because I could not find the delivery point for the virtual IPI.

Does the *vmx_do_extint()* function delivery the IPI through *
__do_IRQ_guest()* ??

Thank you for your kind advice.

[-- Attachment #1.2: Type: text/html, Size: 1556 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: How to handle TLB shootdown IPI on Xen

[Jan Beulich]

>>> On 12.08.13 at 18:50, J Ahn <jeong.seob.ahn@gmail.com> wrote:
> As I mentioned the first post, I also guess that the hypervisor
> intervention is required to route the virtual IPI.
> So, I would like to know where the delivery code on Xen has been located
> because I could not find the delivery point for the virtual IPI.
> 
> Does the *vmx_do_extint()* function delivery the IPI through *
> __do_IRQ_guest()* ??

How would it? I suppose you know that IPIs get generated through
writing to LAPIC register, so associating this with the functions in
xen/arch/x86/hvm/vlapic.c shouldn't be too difficult; namely you
want to look at vlapic_reg_write()'s handling of APIC_ICR{,2} and
APIC_SELF_IPI.

Jan

Re: How to handle TLB shootdown IPI on Xen

[J Ahn]

[-- Attachment #1.1: Type: text/plain, Size: 945 bytes --]

2013/8/13 Jan Beulich <JBeulich@suse.com>

> >>> On 12.08.13 at 18:50, J Ahn <jeong.seob.ahn@gmail.com> wrote:
> > As I mentioned the first post, I also guess that the hypervisor
> > intervention is required to route the virtual IPI.
> > So, I would like to know where the delivery code on Xen has been located
> > because I could not find the delivery point for the virtual IPI.
> >
> > Does the *vmx_do_extint()* function delivery the IPI through *
> > __do_IRQ_guest()* ??
>
> How would it? I suppose you know that IPIs get generated through
> writing to LAPIC register, so associating this with the functions in
> xen/arch/x86/hvm/vlapic.c shouldn't be too difficult; namely you
> want to look at vlapic_reg_write()'s handling of APIC_ICR{,2} and
> APIC_SELF_IPI.
>
> Jan
>
>

I am missing the important point which you mentioned.
I need to study the code to understand the behavior well.

Your advice is very helpful.
Thank you.

Jeongseob

[-- Attachment #1.2: Type: text/html, Size: 1684 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel