[Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Jintack Lim]

Hi,

I'm using vhost with the virtual intel-iommu, and this page[1] shows
the QEMU command line example.

qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \
                   -device intel-iommu,intremap=on,device-iotlb=on \
                   -device ioh3420,id=pcie.1,chassis=1 \
                   -device
virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on
\
                   -netdev tap,id=net0,vhostforce \
                   $IMAGE_PATH

I wonder what's the impact of using device-iotlb and ats options as
they are described necessary.

In my understanding, vhost in the kernel only looks at
VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a
device-iotlb. In addition, vhost and QEMU communicate using vhost_msg
basically to cache mappings correctly in the vhost, so I wonder what's
the role of ats in this case.

A related question is that if we use SMMU emulation[2] on ARM without
those options, does vhost cache mappings as if it has a device-iotlb?
(I guess this is the case.)

I'm pretty new to QEMU code, so I might be missing something. Can
somebody shed some light on it?

[1] https://wiki.qemu.org/Features/VT-d
[2] http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html

Thanks,
Jintack

Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Peter Xu]

On Tue, Feb 20, 2018 at 11:03:46PM -0500, Jintack Lim wrote:
> Hi,
> 
> I'm using vhost with the virtual intel-iommu, and this page[1] shows
> the QEMU command line example.
> 
> qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \
>                    -device intel-iommu,intremap=on,device-iotlb=on \
>                    -device ioh3420,id=pcie.1,chassis=1 \
>                    -device
> virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on
> \
>                    -netdev tap,id=net0,vhostforce \
>                    $IMAGE_PATH
> 
> I wonder what's the impact of using device-iotlb and ats options as
> they are described necessary.
> 
> In my understanding, vhost in the kernel only looks at
> VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a
> device-iotlb. In addition, vhost and QEMU communicate using vhost_msg
> basically to cache mappings correctly in the vhost, so I wonder what's
> the role of ats in this case.

The "ats" as virtio device parameter will add ATS capability to the
PCI device.

The "device-iotlb" as intel-iommu parameter will enable ATS in the
IOMMU device (and also report that in ACPI field).

If both parameters are provided IIUC it means guest will know virtio
device has device-iotlb and it'll treat the device specially (e.g.,
guest will need to send device-iotlb invalidations).

We'd better keep these parameters when running virtio devices with
vIOMMU.  For the rest of vhost/arm specific questions, I'll leave to
others.

PS: Though IIUC the whole ATS thing may not really be necessary for
current VT-d emulation, since even with ATS vhost is registering UNMAP
IOMMU notifiers (see vhost_iommu_region_add()), and IIUC that means
vhost will receive IOTLB invalidations even without ATS support, and
it _might_ still work.  But there can be other differences, like
performance, etc.

> 
> A related question is that if we use SMMU emulation[2] on ARM without
> those options, does vhost cache mappings as if it has a device-iotlb?
> (I guess this is the case.)
> 
> I'm pretty new to QEMU code, so I might be missing something. Can
> somebody shed some light on it?
> 
> [1] https://wiki.qemu.org/Features/VT-d
> [2] http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html
> 
> Thanks,
> Jintack
> 

-- 
Peter Xu

Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Jintack Lim]

Hi Peter,

Hope you had great holidays!

On Thu, Feb 22, 2018 at 10:55 PM, Peter Xu <peterx@redhat.com> wrote:
> On Tue, Feb 20, 2018 at 11:03:46PM -0500, Jintack Lim wrote:
>> Hi,
>>
>> I'm using vhost with the virtual intel-iommu, and this page[1] shows
>> the QEMU command line example.
>>
>> qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \
>>                    -device intel-iommu,intremap=on,device-iotlb=on \
>>                    -device ioh3420,id=pcie.1,chassis=1 \
>>                    -device
>> virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on
>> \
>>                    -netdev tap,id=net0,vhostforce \
>>                    $IMAGE_PATH
>>
>> I wonder what's the impact of using device-iotlb and ats options as
>> they are described necessary.
>>
>> In my understanding, vhost in the kernel only looks at
>> VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a
>> device-iotlb. In addition, vhost and QEMU communicate using vhost_msg
>> basically to cache mappings correctly in the vhost, so I wonder what's
>> the role of ats in this case.
>
> The "ats" as virtio device parameter will add ATS capability to the
> PCI device.
>
> The "device-iotlb" as intel-iommu parameter will enable ATS in the
> IOMMU device (and also report that in ACPI field).
>
> If both parameters are provided IIUC it means guest will know virtio
> device has device-iotlb and it'll treat the device specially (e.g.,
> guest will need to send device-iotlb invalidations).

Oh, I see. I was focusing on how QEMU and vhost work in the host, but
I think I missed the guest part! Thanks. I see that the Intel IOMMU
driver has has_iotlb_device flag for that purpose.

>
> We'd better keep these parameters when running virtio devices with
> vIOMMU.  For the rest of vhost/arm specific questions, I'll leave to
> others.

It seems like SMMU is not checking ATS capability - at least
ats_enabled flag - but I may miss something here as well :)

>
> PS: Though IIUC the whole ATS thing may not really be necessary for
> current VT-d emulation, since even with ATS vhost is registering UNMAP
> IOMMU notifiers (see vhost_iommu_region_add()), and IIUC that means
> vhost will receive IOTLB invalidations even without ATS support, and
> it _might_ still work.

Right. That's what I thought.

Come to think of it, I'm not sure why we need to flush mappings in
IOMMU and devices separately in the first place... Any thoughts?

Your reply was really helpful to me. I appreciate it.

Thanks,
Jintack

> But there can be other differences, like
> performance, etc.
>
>>
>> A related question is that if we use SMMU emulation[2] on ARM without
>> those options, does vhost cache mappings as if it has a device-iotlb?
>> (I guess this is the case.)
>>
>> I'm pretty new to QEMU code, so I might be missing something. Can
>> somebody shed some light on it?
>>
>> [1] https://wiki.qemu.org/Features/VT-d
>> [2] http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html
>>
>> Thanks,
>> Jintack
>>
>
> --
> Peter Xu
>

Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Peter Xu]

On Fri, Feb 23, 2018 at 12:32:13AM -0500, Jintack Lim wrote:
> Hi Peter,
> 
> Hope you had great holidays!
> 
> On Thu, Feb 22, 2018 at 10:55 PM, Peter Xu <peterx@redhat.com> wrote:
> > On Tue, Feb 20, 2018 at 11:03:46PM -0500, Jintack Lim wrote:
> >> Hi,
> >>
> >> I'm using vhost with the virtual intel-iommu, and this page[1] shows
> >> the QEMU command line example.
> >>
> >> qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \
> >>                    -device intel-iommu,intremap=on,device-iotlb=on \
> >>                    -device ioh3420,id=pcie.1,chassis=1 \
> >>                    -device
> >> virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on
> >> \
> >>                    -netdev tap,id=net0,vhostforce \
> >>                    $IMAGE_PATH
> >>
> >> I wonder what's the impact of using device-iotlb and ats options as
> >> they are described necessary.
> >>
> >> In my understanding, vhost in the kernel only looks at
> >> VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a
> >> device-iotlb. In addition, vhost and QEMU communicate using vhost_msg
> >> basically to cache mappings correctly in the vhost, so I wonder what's
> >> the role of ats in this case.
> >
> > The "ats" as virtio device parameter will add ATS capability to the
> > PCI device.
> >
> > The "device-iotlb" as intel-iommu parameter will enable ATS in the
> > IOMMU device (and also report that in ACPI field).
> >
> > If both parameters are provided IIUC it means guest will know virtio
> > device has device-iotlb and it'll treat the device specially (e.g.,
> > guest will need to send device-iotlb invalidations).
> 
> Oh, I see. I was focusing on how QEMU and vhost work in the host, but
> I think I missed the guest part! Thanks. I see that the Intel IOMMU
> driver has has_iotlb_device flag for that purpose.
> 
> >
> > We'd better keep these parameters when running virtio devices with
> > vIOMMU.  For the rest of vhost/arm specific questions, I'll leave to
> > others.
> 
> It seems like SMMU is not checking ATS capability - at least
> ats_enabled flag - but I may miss something here as well :)
> 
> >
> > PS: Though IIUC the whole ATS thing may not really be necessary for
> > current VT-d emulation, since even with ATS vhost is registering UNMAP
> > IOMMU notifiers (see vhost_iommu_region_add()), and IIUC that means
> > vhost will receive IOTLB invalidations even without ATS support, and
> > it _might_ still work.
> 
> Right. That's what I thought.
> 
> Come to think of it, I'm not sure why we need to flush mappings in
> IOMMU and devices separately in the first place... Any thoughts?

I don't know ATS much, neither.

You can have a look at chap 4 of vt-d spec:

        One approach to scaling IOTLBs is to enable I/O devices to
        participate in the DMA remapping with IOTLBs implemented at
        the devices. The Device-IOTLBs alleviate pressure for IOTLB
        resources in the core logic, and provide opportunities for
        devices to improve performance by pre-fetching address
        translations before issuing DMA requests. This may be useful
        for devices with strict DMA latency requirements (such as
        isochronous devices), and for devices that have large DMA
        working set or multiple active DMA streams.

So I think it's for performance's sake. For example, the DMA operation
won't need to be translated at all if it's pre-translated, so it can
have less latency.  And also, that'll offload some of the translation
process so that workload can be more distributed.

When with that (caches located both on IOMMU's and device's side), we
need to invalidate all the cache when needed.

> 
> Your reply was really helpful to me. I appreciate it.

My pleasure.  Thanks,

> 
> Thanks,
> Jintack
> 
> > But there can be other differences, like
> > performance, etc.
> >
> >>
> >> A related question is that if we use SMMU emulation[2] on ARM without
> >> those options, does vhost cache mappings as if it has a device-iotlb?
> >> (I guess this is the case.)
> >>
> >> I'm pretty new to QEMU code, so I might be missing something. Can
> >> somebody shed some light on it?
> >>
> >> [1] https://wiki.qemu.org/Features/VT-d
> >> [2] http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html
> >>
> >> Thanks,
> >> Jintack
> >>
> >
> > --
> > Peter Xu
> >
> 

-- 
Peter Xu

Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Jintack Lim]

On Fri, Feb 23, 2018 at 1:10 AM Peter Xu <peterx@redhat.com> wrote:

> On Fri, Feb 23, 2018 at 12:32:13AM -0500, Jintack Lim wrote:
> > Hi Peter,
> >
> > Hope you had great holidays!
> >
> > On Thu, Feb 22, 2018 at 10:55 PM, Peter Xu <peterx@redhat.com> wrote:
> > > On Tue, Feb 20, 2018 at 11:03:46PM -0500, Jintack Lim wrote:
> > >> Hi,
> > >>
> > >> I'm using vhost with the virtual intel-iommu, and this page[1] shows
> > >> the QEMU command line example.
> > >>
> > >> qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \
> > >>                    -device intel-iommu,intremap=on,device-iotlb=on \
> > >>                    -device ioh3420,id=pcie.1,chassis=1 \
> > >>                    -device
> > >>
> virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on
> > >> \
> > >>                    -netdev tap,id=net0,vhostforce \
> > >>                    $IMAGE_PATH
> > >>
> > >> I wonder what's the impact of using device-iotlb and ats options as
> > >> they are described necessary.
> > >>
> > >> In my understanding, vhost in the kernel only looks at
> > >> VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a
> > >> device-iotlb. In addition, vhost and QEMU communicate using vhost_msg
> > >> basically to cache mappings correctly in the vhost, so I wonder what's
> > >> the role of ats in this case.
> > >
> > > The "ats" as virtio device parameter will add ATS capability to the
> > > PCI device.
> > >
> > > The "device-iotlb" as intel-iommu parameter will enable ATS in the
> > > IOMMU device (and also report that in ACPI field).
> > >
> > > If both parameters are provided IIUC it means guest will know virtio
> > > device has device-iotlb and it'll treat the device specially (e.g.,
> > > guest will need to send device-iotlb invalidations).
> >
> > Oh, I see. I was focusing on how QEMU and vhost work in the host, but
> > I think I missed the guest part! Thanks. I see that the Intel IOMMU
> > driver has has_iotlb_device flag for that purpose.
> >
> > >
> > > We'd better keep these parameters when running virtio devices with
> > > vIOMMU.  For the rest of vhost/arm specific questions, I'll leave to
> > > others.
> >
> > It seems like SMMU is not checking ATS capability - at least
> > ats_enabled flag - but I may miss something here as well :)
> >
> > >
> > > PS: Though IIUC the whole ATS thing may not really be necessary for
> > > current VT-d emulation, since even with ATS vhost is registering UNMAP
> > > IOMMU notifiers (see vhost_iommu_region_add()), and IIUC that means
> > > vhost will receive IOTLB invalidations even without ATS support, and
> > > it _might_ still work.
> >
> > Right. That's what I thought.
> >
> > Come to think of it, I'm not sure why we need to flush mappings in
> > IOMMU and devices separately in the first place... Any thoughts?
>
> I don't know ATS much, neither.
>
> You can have a look at chap 4 of vt-d spec:
>
>         One approach to scaling IOTLBs is to enable I/O devices to
>         participate in the DMA remapping with IOTLBs implemented at
>         the devices. The Device-IOTLBs alleviate pressure for IOTLB
>         resources in the core logic, and provide opportunities for
>         devices to improve performance by pre-fetching address
>         translations before issuing DMA requests. This may be useful
>         for devices with strict DMA latency requirements (such as
>         isochronous devices), and for devices that have large DMA
>         working set or multiple active DMA streams.
>
> So I think it's for performance's sake. For example, the DMA operation
> won't need to be translated at all if it's pre-translated, so it can
> have less latency.  And also, that'll offload some of the translation
> process so that workload can be more distributed.
>
> When with that (caches located both on IOMMU's and device's side), we
> need to invalidate all the cache when needed.
>

Right. I think my question was not clear. My question was that why don’t
IOMMU invalidate device-iotlb along with its mappings in one go. Then IOMMU
device driver doesn’t need to flush device-iotlb explicitly. Maybe the
reason is that ATS and IOMMU are not always coupled.. but I guess it’s time
for me to get some more background :)


> >
> > Your reply was really helpful to me. I appreciate it.
>
> My pleasure.  Thanks,
>
> >
> > Thanks,
> > Jintack
> >
> > > But there can be other differences, like
> > > performance, etc.
> > >
> > >>
> > >> A related question is that if we use SMMU emulation[2] on ARM without
> > >> those options, does vhost cache mappings as if it has a device-iotlb?
> > >> (I guess this is the case.)
> > >>
> > >> I'm pretty new to QEMU code, so I might be missing something. Can
> > >> somebody shed some light on it?
> > >>
> > >> [1] https://wiki.qemu.org/Features/VT-d
> > >> [2]
> http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html
> > >>
> > >> Thanks,
> > >> Jintack
> > >>
> > >
> > > --
> > > Peter Xu
> > >
> >
>
> --
> Peter Xu
>
>

Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Peter Xu]

On Fri, Feb 23, 2018 at 06:34:04AM +0000, Jintack Lim wrote:
> On Fri, Feb 23, 2018 at 1:10 AM Peter Xu <peterx@redhat.com> wrote:
> 
> > On Fri, Feb 23, 2018 at 12:32:13AM -0500, Jintack Lim wrote:
> > > Hi Peter,
> > >
> > > Hope you had great holidays!
> > >
> > > On Thu, Feb 22, 2018 at 10:55 PM, Peter Xu <peterx@redhat.com> wrote:
> > > > On Tue, Feb 20, 2018 at 11:03:46PM -0500, Jintack Lim wrote:
> > > >> Hi,
> > > >>
> > > >> I'm using vhost with the virtual intel-iommu, and this page[1] shows
> > > >> the QEMU command line example.
> > > >>
> > > >> qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \
> > > >>                    -device intel-iommu,intremap=on,device-iotlb=on \
> > > >>                    -device ioh3420,id=pcie.1,chassis=1 \
> > > >>                    -device
> > > >>
> > virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on
> > > >> \
> > > >>                    -netdev tap,id=net0,vhostforce \
> > > >>                    $IMAGE_PATH
> > > >>
> > > >> I wonder what's the impact of using device-iotlb and ats options as
> > > >> they are described necessary.
> > > >>
> > > >> In my understanding, vhost in the kernel only looks at
> > > >> VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a
> > > >> device-iotlb. In addition, vhost and QEMU communicate using vhost_msg
> > > >> basically to cache mappings correctly in the vhost, so I wonder what's
> > > >> the role of ats in this case.
> > > >
> > > > The "ats" as virtio device parameter will add ATS capability to the
> > > > PCI device.
> > > >
> > > > The "device-iotlb" as intel-iommu parameter will enable ATS in the
> > > > IOMMU device (and also report that in ACPI field).
> > > >
> > > > If both parameters are provided IIUC it means guest will know virtio
> > > > device has device-iotlb and it'll treat the device specially (e.g.,
> > > > guest will need to send device-iotlb invalidations).
> > >
> > > Oh, I see. I was focusing on how QEMU and vhost work in the host, but
> > > I think I missed the guest part! Thanks. I see that the Intel IOMMU
> > > driver has has_iotlb_device flag for that purpose.
> > >
> > > >
> > > > We'd better keep these parameters when running virtio devices with
> > > > vIOMMU.  For the rest of vhost/arm specific questions, I'll leave to
> > > > others.
> > >
> > > It seems like SMMU is not checking ATS capability - at least
> > > ats_enabled flag - but I may miss something here as well :)
> > >
> > > >
> > > > PS: Though IIUC the whole ATS thing may not really be necessary for
> > > > current VT-d emulation, since even with ATS vhost is registering UNMAP
> > > > IOMMU notifiers (see vhost_iommu_region_add()), and IIUC that means
> > > > vhost will receive IOTLB invalidations even without ATS support, and
> > > > it _might_ still work.
> > >
> > > Right. That's what I thought.
> > >
> > > Come to think of it, I'm not sure why we need to flush mappings in
> > > IOMMU and devices separately in the first place... Any thoughts?
> >
> > I don't know ATS much, neither.
> >
> > You can have a look at chap 4 of vt-d spec:
> >
> >         One approach to scaling IOTLBs is to enable I/O devices to
> >         participate in the DMA remapping with IOTLBs implemented at
> >         the devices. The Device-IOTLBs alleviate pressure for IOTLB
> >         resources in the core logic, and provide opportunities for
> >         devices to improve performance by pre-fetching address
> >         translations before issuing DMA requests. This may be useful
> >         for devices with strict DMA latency requirements (such as
> >         isochronous devices), and for devices that have large DMA
> >         working set or multiple active DMA streams.
> >
> > So I think it's for performance's sake. For example, the DMA operation
> > won't need to be translated at all if it's pre-translated, so it can
> > have less latency.  And also, that'll offload some of the translation
> > process so that workload can be more distributed.
> >
> > When with that (caches located both on IOMMU's and device's side), we
> > need to invalidate all the cache when needed.
> >
> 
> Right. I think my question was not clear. My question was that why don’t
> IOMMU invalidate device-iotlb along with its mappings in one go. Then IOMMU
> device driver doesn’t need to flush device-iotlb explicitly. Maybe the
> reason is that ATS and IOMMU are not always coupled.. but I guess it’s time
> for me to get some more background :)

Ah, I see your point.

I don't know the answer.  My wild guess is that IOMMU is just trying
to be simple and only provide most basic functionalities, leaving
complex stuff to CPU.  For example, if IOMMU takes over the ownership
to deliever device-iotlb invalidations when receiving iotlb
invalidations, it possibly needs to traverse the device tree sometimes
(e.g., for domain invalidations) to know what device is under what
domain, which is really compliated.  While it'll be simpler for CPU to
do this since it's very possible that the OS keeps a list of devices
for a domain already.

IMHO that follows the *nix philosophy too - Do One Thing And Do It
Well.  Though again, it's wild guess and I may be wrong. :)

CCing Alex, in case he has quick answers.

> 
> 
> > >
> > > Your reply was really helpful to me. I appreciate it.
> >
> > My pleasure.  Thanks,
> >
> > >
> > > Thanks,
> > > Jintack
> > >
> > > > But there can be other differences, like
> > > > performance, etc.
> > > >
> > > >>
> > > >> A related question is that if we use SMMU emulation[2] on ARM without
> > > >> those options, does vhost cache mappings as if it has a device-iotlb?
> > > >> (I guess this is the case.)
> > > >>
> > > >> I'm pretty new to QEMU code, so I might be missing something. Can
> > > >> somebody shed some light on it?
> > > >>
> > > >> [1] https://wiki.qemu.org/Features/VT-d
> > > >> [2]
> > http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html
> > > >>
> > > >> Thanks,
> > > >> Jintack
> > > >>
> > > >
> > > > --
> > > > Peter Xu
> > > >
> > >
> >
> > --
> > Peter Xu
> >
> >

-- 
Peter Xu

Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Tian, Kevin]

> From: Peter Xu
> Sent: Friday, February 23, 2018 3:09 PM
> 
> >
> > Right. I think my question was not clear. My question was that why don’t
> > IOMMU invalidate device-iotlb along with its mappings in one go. Then
> IOMMU
> > device driver doesn’t need to flush device-iotlb explicitly. Maybe the
> > reason is that ATS and IOMMU are not always coupled.. but I guess it’s
> time
> > for me to get some more background :)
> 
> Ah, I see your point.
> 
> I don't know the answer.  My wild guess is that IOMMU is just trying
> to be simple and only provide most basic functionalities, leaving
> complex stuff to CPU.  For example, if IOMMU takes over the ownership
> to deliever device-iotlb invalidations when receiving iotlb
> invalidations, it possibly needs to traverse the device tree sometimes
> (e.g., for domain invalidations) to know what device is under what
> domain, which is really compliated.  While it'll be simpler for CPU to
> do this since it's very possible that the OS keeps a list of devices
> for a domain already.
> 
> IMHO that follows the *nix philosophy too - Do One Thing And Do It
> Well.  Though again, it's wild guess and I may be wrong. :)
> 
> CCing Alex, in case he has quick answers.
> 

IOMMU and devices are de-coupled. You need a protocol so IOMMU
knows which device enables translation caches and thus requires
explicit invalidation, which is how ATS comes to play. ATS is not
mandatory for vhost, but doing so provides more flexibility e.g.
to enable I/O page fault if further emulating PCI PRS cap.

Thanks
Kevin

Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Jintack Lim]

On Fri, Feb 23, 2018 at 2:09 AM, Peter Xu <peterx@redhat.com> wrote:
> On Fri, Feb 23, 2018 at 06:34:04AM +0000, Jintack Lim wrote:
>> On Fri, Feb 23, 2018 at 1:10 AM Peter Xu <peterx@redhat.com> wrote:
>>
>> > On Fri, Feb 23, 2018 at 12:32:13AM -0500, Jintack Lim wrote:
>> > > Hi Peter,
>> > >
>> > > Hope you had great holidays!
>> > >
>> > > On Thu, Feb 22, 2018 at 10:55 PM, Peter Xu <peterx@redhat.com> wrote:
>> > > > On Tue, Feb 20, 2018 at 11:03:46PM -0500, Jintack Lim wrote:
>> > > >> Hi,
>> > > >>
>> > > >> I'm using vhost with the virtual intel-iommu, and this page[1] shows
>> > > >> the QEMU command line example.
>> > > >>
>> > > >> qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \
>> > > >>                    -device intel-iommu,intremap=on,device-iotlb=on \
>> > > >>                    -device ioh3420,id=pcie.1,chassis=1 \
>> > > >>                    -device
>> > > >>
>> > virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on
>> > > >> \
>> > > >>                    -netdev tap,id=net0,vhostforce \
>> > > >>                    $IMAGE_PATH
>> > > >>
>> > > >> I wonder what's the impact of using device-iotlb and ats options as
>> > > >> they are described necessary.
>> > > >>
>> > > >> In my understanding, vhost in the kernel only looks at
>> > > >> VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a
>> > > >> device-iotlb. In addition, vhost and QEMU communicate using vhost_msg
>> > > >> basically to cache mappings correctly in the vhost, so I wonder what's
>> > > >> the role of ats in this case.
>> > > >
>> > > > The "ats" as virtio device parameter will add ATS capability to the
>> > > > PCI device.
>> > > >
>> > > > The "device-iotlb" as intel-iommu parameter will enable ATS in the
>> > > > IOMMU device (and also report that in ACPI field).
>> > > >
>> > > > If both parameters are provided IIUC it means guest will know virtio
>> > > > device has device-iotlb and it'll treat the device specially (e.g.,
>> > > > guest will need to send device-iotlb invalidations).
>> > >
>> > > Oh, I see. I was focusing on how QEMU and vhost work in the host, but
>> > > I think I missed the guest part! Thanks. I see that the Intel IOMMU
>> > > driver has has_iotlb_device flag for that purpose.
>> > >
>> > > >
>> > > > We'd better keep these parameters when running virtio devices with
>> > > > vIOMMU.  For the rest of vhost/arm specific questions, I'll leave to
>> > > > others.
>> > >
>> > > It seems like SMMU is not checking ATS capability - at least
>> > > ats_enabled flag - but I may miss something here as well :)
>> > >
>> > > >
>> > > > PS: Though IIUC the whole ATS thing may not really be necessary for
>> > > > current VT-d emulation, since even with ATS vhost is registering UNMAP
>> > > > IOMMU notifiers (see vhost_iommu_region_add()), and IIUC that means
>> > > > vhost will receive IOTLB invalidations even without ATS support, and
>> > > > it _might_ still work.
>> > >
>> > > Right. That's what I thought.
>> > >
>> > > Come to think of it, I'm not sure why we need to flush mappings in
>> > > IOMMU and devices separately in the first place... Any thoughts?
>> >
>> > I don't know ATS much, neither.
>> >
>> > You can have a look at chap 4 of vt-d spec:
>> >
>> >         One approach to scaling IOTLBs is to enable I/O devices to
>> >         participate in the DMA remapping with IOTLBs implemented at
>> >         the devices. The Device-IOTLBs alleviate pressure for IOTLB
>> >         resources in the core logic, and provide opportunities for
>> >         devices to improve performance by pre-fetching address
>> >         translations before issuing DMA requests. This may be useful
>> >         for devices with strict DMA latency requirements (such as
>> >         isochronous devices), and for devices that have large DMA
>> >         working set or multiple active DMA streams.
>> >
>> > So I think it's for performance's sake. For example, the DMA operation
>> > won't need to be translated at all if it's pre-translated, so it can
>> > have less latency.  And also, that'll offload some of the translation
>> > process so that workload can be more distributed.
>> >
>> > When with that (caches located both on IOMMU's and device's side), we
>> > need to invalidate all the cache when needed.
>> >
>>
>> Right. I think my question was not clear. My question was that why don’t
>> IOMMU invalidate device-iotlb along with its mappings in one go. Then IOMMU
>> device driver doesn’t need to flush device-iotlb explicitly. Maybe the
>> reason is that ATS and IOMMU are not always coupled.. but I guess it’s time
>> for me to get some more background :)
>
> Ah, I see your point.
>
> I don't know the answer.  My wild guess is that IOMMU is just trying
> to be simple and only provide most basic functionalities, leaving
> complex stuff to CPU.  For example, if IOMMU takes over the ownership
> to deliever device-iotlb invalidations when receiving iotlb
> invalidations, it possibly needs to traverse the device tree sometimes
> (e.g., for domain invalidations) to know what device is under what
> domain, which is really compliated.  While it'll be simpler for CPU to
> do this since it's very possible that the OS keeps a list of devices
> for a domain already.
>
> IMHO that follows the *nix philosophy too - Do One Thing And Do It
> Well.  Though again, it's wild guess and I may be wrong. :)

Cool. Makes sense to me!

>
> CCing Alex, in case he has quick answers.
>
>>
>>
>> > >
>> > > Your reply was really helpful to me. I appreciate it.
>> >
>> > My pleasure.  Thanks,
>> >
>> > >
>> > > Thanks,
>> > > Jintack
>> > >
>> > > > But there can be other differences, like
>> > > > performance, etc.
>> > > >
>> > > >>
>> > > >> A related question is that if we use SMMU emulation[2] on ARM without
>> > > >> those options, does vhost cache mappings as if it has a device-iotlb?
>> > > >> (I guess this is the case.)
>> > > >>
>> > > >> I'm pretty new to QEMU code, so I might be missing something. Can
>> > > >> somebody shed some light on it?
>> > > >>
>> > > >> [1] https://wiki.qemu.org/Features/VT-d
>> > > >> [2]
>> > http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html
>> > > >>
>> > > >> Thanks,
>> > > >> Jintack
>> > > >>
>> > > >
>> > > > --
>> > > > Peter Xu
>> > > >
>> > >
>> >
>> > --
>> > Peter Xu
>> >
>> >
>
> --
> Peter Xu
>

Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Jintack Lim]

Hi Kevin,

On Fri, Feb 23, 2018 at 2:34 AM, Tian, Kevin <kevin.tian@intel.com> wrote:
>> From: Peter Xu
>> Sent: Friday, February 23, 2018 3:09 PM
>>
>> >
>> > Right. I think my question was not clear. My question was that why don’t
>> > IOMMU invalidate device-iotlb along with its mappings in one go. Then
>> IOMMU
>> > device driver doesn’t need to flush device-iotlb explicitly. Maybe the
>> > reason is that ATS and IOMMU are not always coupled.. but I guess it’s
>> time
>> > for me to get some more background :)
>>
>> Ah, I see your point.
>>
>> I don't know the answer.  My wild guess is that IOMMU is just trying
>> to be simple and only provide most basic functionalities, leaving
>> complex stuff to CPU.  For example, if IOMMU takes over the ownership
>> to deliever device-iotlb invalidations when receiving iotlb
>> invalidations, it possibly needs to traverse the device tree sometimes
>> (e.g., for domain invalidations) to know what device is under what
>> domain, which is really compliated.  While it'll be simpler for CPU to
>> do this since it's very possible that the OS keeps a list of devices
>> for a domain already.
>>
>> IMHO that follows the *nix philosophy too - Do One Thing And Do It
>> Well.  Though again, it's wild guess and I may be wrong. :)
>>
>> CCing Alex, in case he has quick answers.
>>
>
> IOMMU and devices are de-coupled. You need a protocol so IOMMU
> knows which device enables translation caches and thus requires
> explicit invalidation, which is how ATS comes to play. ATS is not
> mandatory for vhost, but doing so provides more flexibility e.g.
> to enable I/O page fault if further emulating PCI PRS cap.

Thanks for the explanation!

Thanks,
Jintack

>
> Thanks
> Kevin

Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Auger Eric]

Hi Jintack,

On 21/02/18 05:03, Jintack Lim wrote:
> Hi,
> 
> I'm using vhost with the virtual intel-iommu, and this page[1] shows
> the QEMU command line example.
> 
> qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \
>                    -device intel-iommu,intremap=on,device-iotlb=on \
>                    -device ioh3420,id=pcie.1,chassis=1 \
>                    -device
> virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on
> \
>                    -netdev tap,id=net0,vhostforce \
>                    $IMAGE_PATH
> 
> I wonder what's the impact of using device-iotlb and ats options as
> they are described necessary.
> 
> In my understanding, vhost in the kernel only looks at
> VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a
> device-iotlb. In addition, vhost and QEMU communicate using vhost_msg
> basically to cache mappings correctly in the vhost, so I wonder what's
> the role of ats in this case.
> 
> A related question is that if we use SMMU emulation[2] on ARM without
> those options, does vhost cache mappings as if it has a device-iotlb?
> (I guess this is the case.)
vsmmuv3 emulation code does not support ATS at the moment. vhost support
is something different. As Peter explained it comes with the capability
of the virtio device to register unmap notifiers. Those notifiers get
called each time there are TLB invalidation commands. That way the
in-kernel vhost cache can be invalidated. vhost support was there until
vsmmuv3 v7. With latest versions, I removed it to help reviewers
concentrate on the root functionality. However I will send it to you
based on v9.

Thanks

Eric
> 
> I'm pretty new to QEMU code, so I might be missing something. Can
> somebody shed some light on it?
> 
> [1] https://wiki.qemu.org/Features/VT-d
> [2] http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html
> 
> Thanks,
> Jintack
> 
> 

Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Jintack Lim]

Hi Eric,

On Mon, Feb 26, 2018 at 5:14 AM, Auger Eric <eric.auger@redhat.com> wrote:
> Hi Jintack,
>
> On 21/02/18 05:03, Jintack Lim wrote:
>> Hi,
>>
>> I'm using vhost with the virtual intel-iommu, and this page[1] shows
>> the QEMU command line example.
>>
>> qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \
>>                    -device intel-iommu,intremap=on,device-iotlb=on \
>>                    -device ioh3420,id=pcie.1,chassis=1 \
>>                    -device
>> virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on
>> \
>>                    -netdev tap,id=net0,vhostforce \
>>                    $IMAGE_PATH
>>
>> I wonder what's the impact of using device-iotlb and ats options as
>> they are described necessary.
>>
>> In my understanding, vhost in the kernel only looks at
>> VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a
>> device-iotlb. In addition, vhost and QEMU communicate using vhost_msg
>> basically to cache mappings correctly in the vhost, so I wonder what's
>> the role of ats in this case.
>>
>> A related question is that if we use SMMU emulation[2] on ARM without
>> those options, does vhost cache mappings as if it has a device-iotlb?
>> (I guess this is the case.)
> vsmmuv3 emulation code does not support ATS at the moment. vhost support
> is something different. As Peter explained it comes with the capability
> of the virtio device to register unmap notifiers. Those notifiers get
> called each time there are TLB invalidation commands. That way the
> in-kernel vhost cache can be invalidated. vhost support was there until
> vsmmuv3 v7. With latest versions, I removed it to help reviewers
> concentrate on the root functionality. However I will send it to you
> based on v9.

Thanks, Eric. I'm happy to take a look at those patches!

Thanks,
Jintack

>
> Thanks
>
> Eric
>>
>> I'm pretty new to QEMU code, so I might be missing something. Can
>> somebody shed some light on it?
>>
>> [1] https://wiki.qemu.org/Features/VT-d
>> [2] http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html
>>
>> Thanks,
>> Jintack
>>
>>
>