[query] gic_set_lr always uses maintenance Interrupt

[query] gic_set_lr always uses maintenance Interrupt

[Mj Embd]

Hi,

As per section 5.2.2 of IHI0048B_b_gic_architecture_specification, If
hypervisor is injecting a VIRQ into guest, that is actually a HW IRQ,
and guest does EOI (provided conditions) the maintenance interrupt is
not needed.

In xen arch/arm/gic.c always while setting an LR using gic_set_lr ,
the maintenance_int is enabled.

Can some one clear the doubt on why it is done
a) is this because EOI by guest would result in control back to
hypervisor to replenish the LR for   inflight, as there is otherwise
no way to know in hypervisor that the LR is available.
or
b) some other reason ?

Re: [query] gic_set_lr always uses maintenance Interrupt

[Ian Campbell]

On Wed, 2013-10-30 at 15:08 +0530, Mj Embd wrote:
> Hi,
> 
> As per section 5.2.2 of IHI0048B_b_gic_architecture_specification, If
> hypervisor is injecting a VIRQ into guest, that is actually a HW IRQ,

Is it? You mean LR.HW is set? AIUI this means the IRQ is an actual
hardware IRQ which is passed through, whereas the VIRQs are entirely
fictional virtual interrupts. (or maybe you mean some other sort of
VIRQ?)

> and guest does EOI (provided conditions) the maintenance interrupt is
> not needed.
> 
> In xen arch/arm/gic.c always while setting an LR using gic_set_lr ,
> the maintenance_int is enabled.
> 
> Can some one clear the doubt on why it is done
> a) is this because EOI by guest would result in control back to
> hypervisor to replenish the LR for   inflight, as there is otherwise
> no way to know in hypervisor that the LR is available.
> or
> b) some other reason ?

CCing Stefano, he understands this stuff and IIRC there was some reason
your a) didn't work in practice.

Ian.

Re: [query] gic_set_lr always uses maintenance Interrupt

[Mj Embd]

On Thu, Oct 31, 2013 at 9:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Wed, 2013-10-30 at 15:08 +0530, Mj Embd wrote:
>> Hi,
>>
>> As per section 5.2.2 of IHI0048B_b_gic_architecture_specification, If
>> hypervisor is injecting a VIRQ into guest, that is actually a HW IRQ,
>
> Is it? You mean LR.HW is set? AIUI this means the IRQ is an actual
> hardware IRQ which is passed through, whereas the VIRQs are entirely
> fictional virtual interrupts. (or maybe you mean some other sort of
> VIRQ?)
>

If say a timer HW interrupt is injected into guest. And if guest does
EOI, the HW interrupt is deactivated.
This is an optimisation as per ARM which saves a trap to hypervisor
which is in form of a maintenance interrupt.

I see in the code gic_set_lr always chooses an option for maintenance
interrupt. So it is not using ARM's optimisation technique.

The reason I thought why this is done is bcase problem I see with this
is that hypervisor still has its LR used and has no way of determining
when it can replenish LR for further interrupts to be injected in
guest. But I could be wrong, why would ARM provide a way an keep a
loophole

>> and guest does EOI (provided conditions) the maintenance interrupt is
>> not needed.
>>
>> In xen arch/arm/gic.c always while setting an LR using gic_set_lr ,
>> the maintenance_int is enabled.
>>
>> Can some one clear the doubt on why it is done
>> a) is this because EOI by guest would result in control back to
>> hypervisor to replenish the LR for   inflight, as there is otherwise
>> no way to know in hypervisor that the LR is available.
>> or
>> b) some other reason ?
>
> CCing Stefano, he understands this stuff and IIRC there was some reason
> your a) didn't work in practice.
>
> Ian.
>



-- 
-mj

Re: [query] gic_set_lr always uses maintenance Interrupt

[Stefano Stabellini]

On Thu, 31 Oct 2013, Mj Embd wrote:
> On Thu, Oct 31, 2013 at 9:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Wed, 2013-10-30 at 15:08 +0530, Mj Embd wrote:
> >> Hi,
> >>
> >> As per section 5.2.2 of IHI0048B_b_gic_architecture_specification, If
> >> hypervisor is injecting a VIRQ into guest, that is actually a HW IRQ,
> >
> > Is it? You mean LR.HW is set? AIUI this means the IRQ is an actual
> > hardware IRQ which is passed through, whereas the VIRQs are entirely
> > fictional virtual interrupts. (or maybe you mean some other sort of
> > VIRQ?)
> >
> 
> If say a timer HW interrupt is injected into guest. And if guest does
> EOI, the HW interrupt is deactivated.
> This is an optimisation as per ARM which saves a trap to hypervisor
> which is in form of a maintenance interrupt.
> 
> I see in the code gic_set_lr always chooses an option for maintenance
> interrupt. So it is not using ARM's optimisation technique.
> 
> The reason I thought why this is done is bcase problem I see with this
> is that hypervisor still has its LR used and has no way of determining
> when it can replenish LR for further interrupts to be injected in
> guest. But I could be wrong, why would ARM provide a way an keep a
> loophole

Your analysis is correct. Even if the hardware interrupt is deactivated
how can Xen know exactly which LRs need to be cleared and when?

If you can figure out the answer to these questions we might be able to
start avoiding maintenance interrupts on hardware interrupts :)

Re: [query] gic_set_lr always uses maintenance Interrupt

[Mj Embd]

A few thoughts are circling around my mind, don't know how much
interrupt latency would it have.

Rather than the hypervisor entry when guest does EOI, a late / lazy
checkin on LR's can be done
 on next hypervisor entry by
a) guest doing something and trapping to hypervisor
b) schedular timer in hypervisor

What do you think on this...

On Wed, Nov 6, 2013 at 11:54 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 31 Oct 2013, Mj Embd wrote:
>> On Thu, Oct 31, 2013 at 9:13 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > On Wed, 2013-10-30 at 15:08 +0530, Mj Embd wrote:
>> >> Hi,
>> >>
>> >> As per section 5.2.2 of IHI0048B_b_gic_architecture_specification, If
>> >> hypervisor is injecting a VIRQ into guest, that is actually a HW IRQ,
>> >
>> > Is it? You mean LR.HW is set? AIUI this means the IRQ is an actual
>> > hardware IRQ which is passed through, whereas the VIRQs are entirely
>> > fictional virtual interrupts. (or maybe you mean some other sort of
>> > VIRQ?)
>> >
>>
>> If say a timer HW interrupt is injected into guest. And if guest does
>> EOI, the HW interrupt is deactivated.
>> This is an optimisation as per ARM which saves a trap to hypervisor
>> which is in form of a maintenance interrupt.
>>
>> I see in the code gic_set_lr always chooses an option for maintenance
>> interrupt. So it is not using ARM's optimisation technique.
>>
>> The reason I thought why this is done is bcase problem I see with this
>> is that hypervisor still has its LR used and has no way of determining
>> when it can replenish LR for further interrupts to be injected in
>> guest. But I could be wrong, why would ARM provide a way an keep a
>> loophole
>
> Your analysis is correct. Even if the hardware interrupt is deactivated
> how can Xen know exactly which LRs need to be cleared and when?
>
> If you can figure out the answer to these questions we might be able to
> start avoiding maintenance interrupts on hardware interrupts :)



-- 
-mj

Re: [query] gic_set_lr always uses maintenance Interrupt

[Ian Campbell]

(please don't top post)
On Thu, 2013-11-07 at 05:29 +0530, Mj Embd wrote:
> A few thoughts are circling around my mind, don't know how much
> interrupt latency would it have.
> 
> Rather than the hypervisor entry when guest does EOI, a late / lazy
> checkin on LR's can be done
>  on next hypervisor entry by
> a) guest doing something and trapping to hypervisor
> b) schedular timer in hypervisor
> 
> What do you think on this...

If you have e.g. 5 pending interrupts and 4 LR registers then ideally
you would want to inject the 5 highest priorities into the LRs
immediately and then inject the fifth as soon as a slot became
available. Or maybe it could be deferred until all of the original 4 are
acked, since it is lower priority all four would need to be handled
before it came up.

Ian.

Re: [query] gic_set_lr always uses maintenance Interrupt

[Stefano Stabellini]

Please don't top post as it makes it harder to follow the conversation.

On Thu, 7 Nov 2013, Mj Embd wrote:
> A few thoughts are circling around my mind, don't know how much
> interrupt latency would it have.
> 
> Rather than the hypervisor entry when guest does EOI, a late / lazy
> checkin on LR's can be done
>  on next hypervisor entry by
> a) guest doing something and trapping to hypervisor
> b) schedular timer in hypervisor
> 
> What do you think on this...

It might work.
One key issue is how to identify that the guest EOIed a particular irq
and henceforth the corresponding LR can be reused.
I hope that the status bits in the LR register reflect this condition.
Maybe the status becomes 00 invalid after the guest does EOI? Otherwise
I can't really see how we could make it work.

Re: [query] gic_set_lr always uses maintenance Interrupt

[Mj Embd]

On Thu, Nov 7, 2013 at 5:07 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> Please don't top post as it makes it harder to follow the conversation.
>
> On Thu, 7 Nov 2013, Mj Embd wrote:
>> A few thoughts are circling around my mind, don't know how much
>> interrupt latency would it have.
>>
>> Rather than the hypervisor entry when guest does EOI, a late / lazy
>> checkin on LR's can be done
>>  on next hypervisor entry by
>> a) guest doing something and trapping to hypervisor
>> b) schedular timer in hypervisor
>>
>> What do you think on this...
>
> It might work.
> One key issue is how to identify that the guest EOIed a particular irq
> and henceforth the corresponding LR can be reused.
[mj] I believe that GICH_ELSR0/1 can be read anytime to get the status.

> I hope that the status bits in the LR register reflect this condition.
> Maybe the status becomes 00 invalid after the guest does EOI? Otherwise
[mj] The state in LR is marked invalid by Virtual CPU interface.

> I can't really see how we could make it work.

I am planning to stop using gmail, it makes you top post as it hides
everything.


-- 
-mj

Re: [query] gic_set_lr always uses maintenance Interrupt

[Ian Campbell]

On Thu, 2013-11-07 at 22:14 +0530, Mj Embd wrote:
> On Thu, Nov 7, 2013 at 5:07 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > Please don't top post as it makes it harder to follow the conversation.
> >
> > On Thu, 7 Nov 2013, Mj Embd wrote:
> >> A few thoughts are circling around my mind, don't know how much
> >> interrupt latency would it have.
> >>
> >> Rather than the hypervisor entry when guest does EOI, a late / lazy
> >> checkin on LR's can be done
> >>  on next hypervisor entry by
> >> a) guest doing something and trapping to hypervisor
> >> b) schedular timer in hypervisor
> >>
> >> What do you think on this...
> >
> > It might work.
> > One key issue is how to identify that the guest EOIed a particular irq
> > and henceforth the corresponding LR can be reused.
> [mj] I believe that GICH_ELSR0/1 can be read anytime to get the status.
> 
> > I hope that the status bits in the LR register reflect this condition.
> > Maybe the status becomes 00 invalid after the guest does EOI? Otherwise
> [mj] The state in LR is marked invalid by Virtual CPU interface.

If you can make it work then we'd be interested in seeing the patches.

Ian.

Re: [query] gic_set_lr always uses maintenance Interrupt

[Stefano Stabellini]

On Thu, 7 Nov 2013, Mj Embd wrote:
> On Thu, Nov 7, 2013 at 5:07 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > Please don't top post as it makes it harder to follow the conversation.
> >
> > On Thu, 7 Nov 2013, Mj Embd wrote:
> >> A few thoughts are circling around my mind, don't know how much
> >> interrupt latency would it have.
> >>
> >> Rather than the hypervisor entry when guest does EOI, a late / lazy
> >> checkin on LR's can be done
> >>  on next hypervisor entry by
> >> a) guest doing something and trapping to hypervisor
> >> b) schedular timer in hypervisor
> >>
> >> What do you think on this...
> >
> > It might work.
> > One key issue is how to identify that the guest EOIed a particular irq
> > and henceforth the corresponding LR can be reused.
> [mj] I believe that GICH_ELSR0/1 can be read anytime to get the status.
> 
> > I hope that the status bits in the LR register reflect this condition.
> > Maybe the status becomes 00 invalid after the guest does EOI? Otherwise
> [mj] The state in LR is marked invalid by Virtual CPU interface.

Right. In that case the lazy LR clearance should work.

> > I can't really see how we could make it work.
> 
> I am planning to stop using gmail, it makes you top post as it hides
> everything.

Yes, I use gmail for my personal email and I find it very inconvenient
for anything but simple messages. Thank you.

Re: [query] gic_set_lr always uses maintenance Interrupt

[Mj Embd]

On Thu, Nov 7, 2013 at 10:18 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 7 Nov 2013, Mj Embd wrote:
>> On Thu, Nov 7, 2013 at 5:07 PM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > Please don't top post as it makes it harder to follow the conversation.
>> >
>> > On Thu, 7 Nov 2013, Mj Embd wrote:
>> >> A few thoughts are circling around my mind, don't know how much
>> >> interrupt latency would it have.
>> >>
>> >> Rather than the hypervisor entry when guest does EOI, a late / lazy
>> >> checkin on LR's can be done
>> >>  on next hypervisor entry by
>> >> a) guest doing something and trapping to hypervisor
>> >> b) schedular timer in hypervisor
>> >>
>> >> What do you think on this...
>> >
>> > It might work.
>> > One key issue is how to identify that the guest EOIed a particular irq
>> > and henceforth the corresponding LR can be reused.
>> [mj] I believe that GICH_ELSR0/1 can be read anytime to get the status.
>>
>> > I hope that the status bits in the LR register reflect this condition.
>> > Maybe the status becomes 00 invalid after the guest does EOI? Otherwise
>> [mj] The state in LR is marked invalid by Virtual CPU interface.
>
> Right. In that case the lazy LR clearance should work.
>
[mj] I have started analysing it, can i send a patch in near future
>> > I can't really see how we could make it work.
>>
>> I am planning to stop using gmail, it makes you top post as it hides
>> everything.
>
> Yes, I use gmail for my personal email and I find it very inconvenient
> for anything but simple messages. Thank you.



-- 
-mj

Re: [query] gic_set_lr always uses maintenance Interrupt

[Stefano Stabellini]

On Thu, 7 Nov 2013, Mj Embd wrote:
> On Thu, Nov 7, 2013 at 10:18 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Thu, 7 Nov 2013, Mj Embd wrote:
> >> On Thu, Nov 7, 2013 at 5:07 PM, Stefano Stabellini
> >> <stefano.stabellini@eu.citrix.com> wrote:
> >> > Please don't top post as it makes it harder to follow the conversation.
> >> >
> >> > On Thu, 7 Nov 2013, Mj Embd wrote:
> >> >> A few thoughts are circling around my mind, don't know how much
> >> >> interrupt latency would it have.
> >> >>
> >> >> Rather than the hypervisor entry when guest does EOI, a late / lazy
> >> >> checkin on LR's can be done
> >> >>  on next hypervisor entry by
> >> >> a) guest doing something and trapping to hypervisor
> >> >> b) schedular timer in hypervisor
> >> >>
> >> >> What do you think on this...
> >> >
> >> > It might work.
> >> > One key issue is how to identify that the guest EOIed a particular irq
> >> > and henceforth the corresponding LR can be reused.
> >> [mj] I believe that GICH_ELSR0/1 can be read anytime to get the status.
> >>
> >> > I hope that the status bits in the LR register reflect this condition.
> >> > Maybe the status becomes 00 invalid after the guest does EOI? Otherwise
> >> [mj] The state in LR is marked invalid by Virtual CPU interface.
> >
> > Right. In that case the lazy LR clearance should work.
> >
> [mj] I have started analysing it, can i send a patch in near future

Do you have an update on this?
Did you manage to write such a patch?