[Qemu-devel] vIOMMU Posted-interrupt implementation - atomic operation?

[Qemu-devel] vIOMMU Posted-interrupt implementation - atomic operation?

[Jintack Lim]

Hi,

I'm implementing Posted-interrupt functionality in vIOMMU. According
to Vt-d spec 5.2.3, IOMMU performs a coherent atomic read-modify-write
operation of the posted-interrupt descriptor. I wonder how can we
achieve this considering the guest can modify the same
posted-interrupt descriptor anytime. Is there any existing mechanism
that I can use in QEMU?

Thanks,
Jintack

Re: [Qemu-devel] vIOMMU Posted-interrupt implementation - atomic operation?

[Tian, Kevin]

> From: Jintack Lim
> Sent: Friday, June 1, 2018 11:47 AM
> 
> Hi,
> 
> I'm implementing Posted-interrupt functionality in vIOMMU. According
> to Vt-d spec 5.2.3, IOMMU performs a coherent atomic read-modify-write
> operation of the posted-interrupt descriptor. I wonder how can we
> achieve this considering the guest can modify the same
> posted-interrupt descriptor anytime. Is there any existing mechanism
> that I can use in QEMU?
> 

I don't think it's possible to emulate such operation in software, unless
you want to change guest to be cooperative. Actually it is not necessary.
VT-d does so due to some hardware implementation consideration. 
Since you are emulating on CPU, could just follow how CPU posted 
interrupt is conducted. If you look at SDM (29.6 Posted-Interrupt
Processing):

	"There is a requirement, however, that such modifications be 
done using locked read-modify-write instructions."

[instructions] means you can do update multiple times when posting an
interrupt, as long as each update is atomic.

Thanks
Kevin

Re: [Qemu-devel] vIOMMU Posted-interrupt implementation - atomic operation?

[Jintack Lim]

Thanks, Kevin.

On Tue, Jun 5, 2018 at 2:54 AM, Tian, Kevin <kevin.tian@intel.com> wrote:
>> From: Jintack Lim
>> Sent: Friday, June 1, 2018 11:47 AM
>>
>> Hi,
>>
>> I'm implementing Posted-interrupt functionality in vIOMMU. According
>> to Vt-d spec 5.2.3, IOMMU performs a coherent atomic read-modify-write
>> operation of the posted-interrupt descriptor. I wonder how can we
>> achieve this considering the guest can modify the same
>> posted-interrupt descriptor anytime. Is there any existing mechanism
>> that I can use in QEMU?
>>
>
> I don't think it's possible to emulate such operation in software, unless
> you want to change guest to be cooperative. Actually it is not necessary.
> VT-d does so due to some hardware implementation consideration.

Would you mind expanding this? I'm curious what it would be. Is it
because IOMMU can't do something like cmpxchg instructions?

> Since you are emulating on CPU, could just follow how CPU posted
> interrupt is conducted. If you look at SDM (29.6 Posted-Interrupt
> Processing):
>
>         "There is a requirement, however, that such modifications be
> done using locked read-modify-write instructions."
>
> [instructions] means you can do update multiple times when posting an
> interrupt, as long as each update is atomic.

Ah, that's a good point. So the unit of atomic operation doesn't need
to be the whole PI descriptor, but it can be any subset (e.g. just one
bit) of the descriptor? By looking at Linux kernel code, that seems to
be the case.

Best,
Jintack

>
> Thanks
> Kevin
>

Re: [Qemu-devel] vIOMMU Posted-interrupt implementation - atomic operation?

[Tian, Kevin]

> From: Jintack Lim [mailto:jintack@cs.columbia.edu]
> Sent: Tuesday, June 5, 2018 8:57 PM
> 
> Thanks, Kevin.
> 
> On Tue, Jun 5, 2018 at 2:54 AM, Tian, Kevin <kevin.tian@intel.com> wrote:
> >> From: Jintack Lim
> >> Sent: Friday, June 1, 2018 11:47 AM
> >>
> >> Hi,
> >>
> >> I'm implementing Posted-interrupt functionality in vIOMMU. According
> >> to Vt-d spec 5.2.3, IOMMU performs a coherent atomic read-modify-
> write
> >> operation of the posted-interrupt descriptor. I wonder how can we
> >> achieve this considering the guest can modify the same
> >> posted-interrupt descriptor anytime. Is there any existing mechanism
> >> that I can use in QEMU?
> >>
> >
> > I don't think it's possible to emulate such operation in software, unless
> > you want to change guest to be cooperative. Actually it is not necessary.
> > VT-d does so due to some hardware implementation consideration.
> 
> Would you mind expanding this? I'm curious what it would be. Is it
> because IOMMU can't do something like cmpxchg instructions?

I don't have further information. Above is what I was told by hardware
team.

> 
> > Since you are emulating on CPU, could just follow how CPU posted
> > interrupt is conducted. If you look at SDM (29.6 Posted-Interrupt
> > Processing):
> >
> >         "There is a requirement, however, that such modifications be
> > done using locked read-modify-write instructions."
> >
> > [instructions] means you can do update multiple times when posting an
> > interrupt, as long as each update is atomic.
> 
> Ah, that's a good point. So the unit of atomic operation doesn't need
> to be the whole PI descriptor, but it can be any subset (e.g. just one
> bit) of the descriptor? By looking at Linux kernel code, that seems to
> be the case.
> 

Exactly. :-)

Thanks
Kevin

Re: [Qemu-devel] vIOMMU Posted-interrupt implementation - atomic operation?

[Jintack Lim]

On Wed, Jun 6, 2018 at 2:56 AM, Tian, Kevin <kevin.tian@intel.com> wrote:
>> From: Jintack Lim [mailto:jintack@cs.columbia.edu]
>> Sent: Tuesday, June 5, 2018 8:57 PM
>>
>> Thanks, Kevin.
>>
>> On Tue, Jun 5, 2018 at 2:54 AM, Tian, Kevin <kevin.tian@intel.com> wrote:
>> >> From: Jintack Lim
>> >> Sent: Friday, June 1, 2018 11:47 AM
>> >>
>> >> Hi,
>> >>
>> >> I'm implementing Posted-interrupt functionality in vIOMMU. According
>> >> to Vt-d spec 5.2.3, IOMMU performs a coherent atomic read-modify-
>> write
>> >> operation of the posted-interrupt descriptor. I wonder how can we
>> >> achieve this considering the guest can modify the same
>> >> posted-interrupt descriptor anytime. Is there any existing mechanism
>> >> that I can use in QEMU?
>> >>
>> >
>> > I don't think it's possible to emulate such operation in software, unless
>> > you want to change guest to be cooperative. Actually it is not necessary.
>> > VT-d does so due to some hardware implementation consideration.
>>
>> Would you mind expanding this? I'm curious what it would be. Is it
>> because IOMMU can't do something like cmpxchg instructions?
>
> I don't have further information. Above is what I was told by hardware
> team.

Ah, I see. Thanks!

>
>>
>> > Since you are emulating on CPU, could just follow how CPU posted
>> > interrupt is conducted. If you look at SDM (29.6 Posted-Interrupt
>> > Processing):
>> >
>> >         "There is a requirement, however, that such modifications be
>> > done using locked read-modify-write instructions."
>> >
>> > [instructions] means you can do update multiple times when posting an
>> > interrupt, as long as each update is atomic.
>>
>> Ah, that's a good point. So the unit of atomic operation doesn't need
>> to be the whole PI descriptor, but it can be any subset (e.g. just one
>> bit) of the descriptor? By looking at Linux kernel code, that seems to
>> be the case.
>>
>
> Exactly. :-)

Cool. Thanks for the confirmation.

Thanks,
Jintack

>
> Thanks
> Kevin