[PATCH] docs: Add '-device intel-iommu' entry

[PATCH] docs: Add '-device intel-iommu' entry

[Peter Xu]

The parameters of intel-iommu device are non-trivial to understand.  Add an
entry for it so that people can reference to it when using.

There're actually a few more options there, but I hide them explicitly because
they shouldn't be used by normal QEMU users.

Cc: Chao Yang <chayang@redhat.com>
Cc: Lei Yang <leiyang@redhat.com>
Cc: Jing Zhao <jinzhao@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/qemu-options.hx b/qemu-options.hx
index 14258784b3a..4bb04243907 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -926,6 +926,38 @@ SRST
 
 ``-device pci-ipmi-bt,bmc=id``
     Like the KCS interface, but defines a BT interface on the PCI bus.
+
+``-device intel-iommu[,option=...]``
+    This is only supported by ``-machine q35``, which will enable Intel VT-d
+    emulation within the guest.  It supports below options:
+
+    ``intremap=on|off`` (default: auto)
+        This enables interrupt remapping feature in the guest.  It's required
+        to enable complete x2apic.  Currently it only supports kvm
+        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
+        yet supported.
+
+    ``caching-mode=on|off`` (default: off)
+        This enables caching mode for the VT-d emulated device.  When
+        caching-mode is enabled, each guest DMA buffer mapping will generate an
+        IOTLB invalidation from the guest IOMMU driver to the vIOMMU device in
+        a synchronous way.  It is required for ``-device vfio-pci`` to work
+        with the VT-d device, because host assigned devices requires to setup
+        the DMA mapping on the host before guest DMA starts.
+
+    ``device-iotlb=on|off`` (default: off)
+        This enables device-iotlb capability for the emulated VT-d device.  So
+        far virtio/vhost should be the only real user for this parameter,
+        paired with ats=on configured for the device.
+
+    ``aw-bits=39|48`` (default: 39)
+        This decides the address width of IOVA address space.  The address
+        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
+        4-level IOMMU page tables.
+
+    Please also refer to the wiki page for general scenarios of VT-d
+    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
+
 ERST
 
 DEF("name", HAS_ARG, QEMU_OPTION_name,
-- 
2.31.1

Re: [PATCH] docs: Add '-device intel-iommu' entry

[Peter Xu]

On Fri, Jun 11, 2021 at 02:55:00PM -0400, Peter Xu wrote:
> The parameters of intel-iommu device are non-trivial to understand.  Add an
> entry for it so that people can reference to it when using.
> 
> There're actually a few more options there, but I hide them explicitly because
> they shouldn't be used by normal QEMU users.
> 
> Cc: Chao Yang <chayang@redhat.com>
> Cc: Lei Yang <leiyang@redhat.com>
> Cc: Jing Zhao <jinzhao@redhat.com>
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
>  1 file changed, 32 insertions(+)

Ping..

-- 
Peter Xu

Re: [PATCH] docs: Add '-device intel-iommu' entry

[Jason Wang]

在 2021/6/12 上午2:55, Peter Xu 写道:
> The parameters of intel-iommu device are non-trivial to understand.  Add an
> entry for it so that people can reference to it when using.
>
> There're actually a few more options there, but I hide them explicitly because
> they shouldn't be used by normal QEMU users.
>
> Cc: Chao Yang <chayang@redhat.com>
> Cc: Lei Yang <leiyang@redhat.com>
> Cc: Jing Zhao <jinzhao@redhat.com>
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
>   1 file changed, 32 insertions(+)
>
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 14258784b3a..4bb04243907 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -926,6 +926,38 @@ SRST
>   
>   ``-device pci-ipmi-bt,bmc=id``
>       Like the KCS interface, but defines a BT interface on the PCI bus.
> +
> +``-device intel-iommu[,option=...]``
> +    This is only supported by ``-machine q35``, which will enable Intel VT-d
> +    emulation within the guest.  It supports below options:
> +
> +    ``intremap=on|off`` (default: auto)
> +        This enables interrupt remapping feature in the guest.


I think we don't need "in the guest" here.

Other than this:

Reviewed-by: Jason Wang <jasowang@redhat.com>

Thanks


>   It's required
> +        to enable complete x2apic.  Currently it only supports kvm
> +        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
> +        yet supported.
> +
> +    ``caching-mode=on|off`` (default: off)
> +        This enables caching mode for the VT-d emulated device.  When
> +        caching-mode is enabled, each guest DMA buffer mapping will generate an
> +        IOTLB invalidation from the guest IOMMU driver to the vIOMMU device in
> +        a synchronous way.  It is required for ``-device vfio-pci`` to work
> +        with the VT-d device, because host assigned devices requires to setup
> +        the DMA mapping on the host before guest DMA starts.
> +
> +    ``device-iotlb=on|off`` (default: off)
> +        This enables device-iotlb capability for the emulated VT-d device.  So
> +        far virtio/vhost should be the only real user for this parameter,
> +        paired with ats=on configured for the device.
> +
> +    ``aw-bits=39|48`` (default: 39)
> +        This decides the address width of IOVA address space.  The address
> +        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
> +        4-level IOMMU page tables.
> +
> +    Please also refer to the wiki page for general scenarios of VT-d
> +    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
> +
>   ERST
>   
>   DEF("name", HAS_ARG, QEMU_OPTION_name,

Re: [PATCH] docs: Add '-device intel-iommu' entry

[Eric Auger]

Hi Peter,

On 6/11/21 8:55 PM, Peter Xu wrote:
> The parameters of intel-iommu device are non-trivial to understand.  Add an
> entry for it so that people can reference to it when using.
> 
> There're actually a few more options there, but I hide them explicitly because
> they shouldn't be used by normal QEMU users.
> 
> Cc: Chao Yang <chayang@redhat.com>
> Cc: Lei Yang <leiyang@redhat.com>
> Cc: Jing Zhao <jinzhao@redhat.com>
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
>  1 file changed, 32 insertions(+)
> 
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 14258784b3a..4bb04243907 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -926,6 +926,38 @@ SRST
>  
>  ``-device pci-ipmi-bt,bmc=id``
>      Like the KCS interface, but defines a BT interface on the PCI bus.
> +
> +``-device intel-iommu[,option=...]``
> +    This is only supported by ``-machine q35``, which will enable Intel VT-d
> +    emulation within the guest.  It supports below options:
> +
> +    ``intremap=on|off`` (default: auto)
> +        This enables interrupt remapping feature in the guest.  It's required
> +        to enable complete x2apic.  Currently it only supports kvm
> +        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
> +        yet supported.
maybe explain how the default is chosen (based on kernel-irqhcip mode?)
> +
> +    ``caching-mode=on|off`` (default: off)
> +        This enables caching mode for the VT-d emulated device.  When
> +        caching-mode is enabled, each guest DMA buffer mapping will generate an
> +        IOTLB invalidation from the guest IOMMU driver to the vIOMMU device in
> +        a synchronous way.  It is required for ``-device vfio-pci`` to work
> +        with the VT-d device, because host assigned devices requires to setup
> +        the DMA mapping on the host before guest DMA starts.
> +
> +    ``device-iotlb=on|off`` (default: off)
> +        This enables device-iotlb capability for the emulated VT-d device.  So
> +        far virtio/vhost should be the only real user for this parameter,
I thought if was needed for vhost. What is the usage for virtio?
> +        paired with ats=on configured for the device.
What is not obvious to me is whether there is a kind of diagnosis
whether device-iotlb and caching mode settings are consistent with
vfio/vhost usage. Does qemu exit if there is inconsistency?
> +
> +    ``aw-bits=39|48`` (default: 39)
> +        This decides the address width of IOVA address space.  The address
> +        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
> +        4-level IOMMU page tables.
> +
> +    Please also refer to the wiki page for general scenarios of VT-d
> +    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
> +
>  ERST
>  
>  DEF("name", HAS_ARG, QEMU_OPTION_name,
>
Besides, it is quite useful.

Thanks

Eric

RE: [PATCH] docs: Add '-device intel-iommu' entry

[Liu, Yi L]

> From: Peter Xu < peterx@redhat.com >
> Sent: Saturday, June 12, 2021 2:55 AM
> 
> The parameters of intel-iommu device are non-trivial to understand.  Add
> an
> entry for it so that people can reference to it when using.
> 
> There're actually a few more options there, but I hide them explicitly
> because
> they shouldn't be used by normal QEMU users.

yes, it's a good start.

Looks good to me.
Reviewed-by: Liu Yi L <yi.l.liu@intel.com>

Regards,
Yi Liu

> 
> Cc: Chao Yang <chayang@redhat.com>
> Cc: Lei Yang <leiyang@redhat.com>
> Cc: Jing Zhao <jinzhao@redhat.com>
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Alex Williamson <alex.williamson@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
>  1 file changed, 32 insertions(+)
> 
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 14258784b3a..4bb04243907 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -926,6 +926,38 @@ SRST
> 
>  ``-device pci-ipmi-bt,bmc=id``
>      Like the KCS interface, but defines a BT interface on the PCI bus.
> +
> +``-device intel-iommu[,option=...]``
> +    This is only supported by ``-machine q35``, which will enable Intel VT-d
> +    emulation within the guest.  It supports below options:
> +
> +    ``intremap=on|off`` (default: auto)
> +        This enables interrupt remapping feature in the guest.  It's required
> +        to enable complete x2apic.  Currently it only supports kvm
> +        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
> +        yet supported.
> +
> +    ``caching-mode=on|off`` (default: off)
> +        This enables caching mode for the VT-d emulated device.  When
> +        caching-mode is enabled, each guest DMA buffer mapping will
> generate an
> +        IOTLB invalidation from the guest IOMMU driver to the vIOMMU
> device in
> +        a synchronous way.  It is required for ``-device vfio-pci`` to work
> +        with the VT-d device, because host assigned devices requires to setup
> +        the DMA mapping on the host before guest DMA starts.
> +
> +    ``device-iotlb=on|off`` (default: off)
> +        This enables device-iotlb capability for the emulated VT-d device.  So
> +        far virtio/vhost should be the only real user for this parameter,
> +        paired with ats=on configured for the device.
> +
> +    ``aw-bits=39|48`` (default: 39)
> +        This decides the address width of IOVA address space.  The address
> +        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
> +        4-level IOMMU page tables.
> +
> +    Please also refer to the wiki page for general scenarios of VT-d
> +    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
> +
>  ERST
> 
>  DEF("name", HAS_ARG, QEMU_OPTION_name,
> --
> 2.31.1
> 

Re: [PATCH] docs: Add '-device intel-iommu' entry

[Peter Xu]

On Wed, Jul 07, 2021 at 11:35:43AM +0800, Jason Wang wrote:
> 
> 在 2021/6/12 上午2:55, Peter Xu 写道:
> > The parameters of intel-iommu device are non-trivial to understand.  Add an
> > entry for it so that people can reference to it when using.
> > 
> > There're actually a few more options there, but I hide them explicitly because
> > they shouldn't be used by normal QEMU users.
> > 
> > Cc: Chao Yang <chayang@redhat.com>
> > Cc: Lei Yang <leiyang@redhat.com>
> > Cc: Jing Zhao <jinzhao@redhat.com>
> > Cc: Jason Wang <jasowang@redhat.com>
> > Cc: Michael S. Tsirkin <mst@redhat.com>
> > Cc: Alex Williamson <alex.williamson@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >   qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
> >   1 file changed, 32 insertions(+)
> > 
> > diff --git a/qemu-options.hx b/qemu-options.hx
> > index 14258784b3a..4bb04243907 100644
> > --- a/qemu-options.hx
> > +++ b/qemu-options.hx
> > @@ -926,6 +926,38 @@ SRST
> >   ``-device pci-ipmi-bt,bmc=id``
> >       Like the KCS interface, but defines a BT interface on the PCI bus.
> > +
> > +``-device intel-iommu[,option=...]``
> > +    This is only supported by ``-machine q35``, which will enable Intel VT-d
> > +    emulation within the guest.  It supports below options:
> > +
> > +    ``intremap=on|off`` (default: auto)
> > +        This enables interrupt remapping feature in the guest.
> 
> 
> I think we don't need "in the guest" here.

Sure.

> 
> Other than this:
> 
> Reviewed-by: Jason Wang <jasowang@redhat.com>

Thanks,

-- 
Peter Xu

Re: [PATCH] docs: Add '-device intel-iommu' entry

[Peter Xu]

On Wed, Jul 07, 2021 at 11:29:47AM +0200, Eric Auger wrote:
> Hi Peter,

Hi, Eric,

> 
> On 6/11/21 8:55 PM, Peter Xu wrote:
> > The parameters of intel-iommu device are non-trivial to understand.  Add an
> > entry for it so that people can reference to it when using.
> > 
> > There're actually a few more options there, but I hide them explicitly because
> > they shouldn't be used by normal QEMU users.
> > 
> > Cc: Chao Yang <chayang@redhat.com>
> > Cc: Lei Yang <leiyang@redhat.com>
> > Cc: Jing Zhao <jinzhao@redhat.com>
> > Cc: Jason Wang <jasowang@redhat.com>
> > Cc: Michael S. Tsirkin <mst@redhat.com>
> > Cc: Alex Williamson <alex.williamson@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
> >  1 file changed, 32 insertions(+)
> > 
> > diff --git a/qemu-options.hx b/qemu-options.hx
> > index 14258784b3a..4bb04243907 100644
> > --- a/qemu-options.hx
> > +++ b/qemu-options.hx
> > @@ -926,6 +926,38 @@ SRST
> >  
> >  ``-device pci-ipmi-bt,bmc=id``
> >      Like the KCS interface, but defines a BT interface on the PCI bus.
> > +
> > +``-device intel-iommu[,option=...]``
> > +    This is only supported by ``-machine q35``, which will enable Intel VT-d
> > +    emulation within the guest.  It supports below options:
> > +
> > +    ``intremap=on|off`` (default: auto)
> > +        This enables interrupt remapping feature in the guest.  It's required
> > +        to enable complete x2apic.  Currently it only supports kvm
> > +        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
> > +        yet supported.
> maybe explain how the default is chosen (based on kernel-irqhcip mode?)

Will do.

> > +
> > +    ``caching-mode=on|off`` (default: off)
> > +        This enables caching mode for the VT-d emulated device.  When
> > +        caching-mode is enabled, each guest DMA buffer mapping will generate an
> > +        IOTLB invalidation from the guest IOMMU driver to the vIOMMU device in
> > +        a synchronous way.  It is required for ``-device vfio-pci`` to work
> > +        with the VT-d device, because host assigned devices requires to setup
> > +        the DMA mapping on the host before guest DMA starts.
> > +
> > +    ``device-iotlb=on|off`` (default: off)
> > +        This enables device-iotlb capability for the emulated VT-d device.  So
> > +        far virtio/vhost should be the only real user for this parameter,
> I thought if was needed for vhost. What is the usage for virtio?

I used "virtio/vhost" because vhost is only one type of virtio backend, so
normally they should be used in pair.  Also, the "ats=on" parameter should be
an option to virtio frontend rather than vhost, hence I mentioned both.

> > +        paired with ats=on configured for the device.
> What is not obvious to me is whether there is a kind of diagnosis
> whether device-iotlb and caching mode settings are consistent with
> vfio/vhost usage. Does qemu exit if there is inconsistency?

Caching mode check is done in vtd_machine_done_notify_one().

I think there's no such check for vhost, however iirc vhost should work
with/without ats=on; I did't check deeper.

> > +
> > +    ``aw-bits=39|48`` (default: 39)
> > +        This decides the address width of IOVA address space.  The address
> > +        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
> > +        4-level IOMMU page tables.
> > +
> > +    Please also refer to the wiki page for general scenarios of VT-d
> > +    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
> > +
> >  ERST
> >  
> >  DEF("name", HAS_ARG, QEMU_OPTION_name,
> >
> Besides, it is quite useful.

Thanks for taking a look!

-- 
Peter Xu

Re: [PATCH] docs: Add '-device intel-iommu' entry

[Eric Auger]

Hi Peter,
On 7/7/21 5:35 PM, Peter Xu wrote:
> On Wed, Jul 07, 2021 at 11:29:47AM +0200, Eric Auger wrote:
>> Hi Peter,
> 
> Hi, Eric,
> 
>>
>> On 6/11/21 8:55 PM, Peter Xu wrote:
>>> The parameters of intel-iommu device are non-trivial to understand.  Add an
>>> entry for it so that people can reference to it when using.
>>>
>>> There're actually a few more options there, but I hide them explicitly because
>>> they shouldn't be used by normal QEMU users.
>>>
>>> Cc: Chao Yang <chayang@redhat.com>
>>> Cc: Lei Yang <leiyang@redhat.com>
>>> Cc: Jing Zhao <jinzhao@redhat.com>
>>> Cc: Jason Wang <jasowang@redhat.com>
>>> Cc: Michael S. Tsirkin <mst@redhat.com>
>>> Cc: Alex Williamson <alex.williamson@redhat.com>
>>> Signed-off-by: Peter Xu <peterx@redhat.com>
>>> ---
>>>  qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
>>>  1 file changed, 32 insertions(+)
>>>
>>> diff --git a/qemu-options.hx b/qemu-options.hx
>>> index 14258784b3a..4bb04243907 100644
>>> --- a/qemu-options.hx
>>> +++ b/qemu-options.hx
>>> @@ -926,6 +926,38 @@ SRST
>>>  
>>>  ``-device pci-ipmi-bt,bmc=id``
>>>      Like the KCS interface, but defines a BT interface on the PCI bus.
>>> +
>>> +``-device intel-iommu[,option=...]``
>>> +    This is only supported by ``-machine q35``, which will enable Intel VT-d
>>> +    emulation within the guest.  It supports below options:
>>> +
>>> +    ``intremap=on|off`` (default: auto)
>>> +        This enables interrupt remapping feature in the guest.  It's required
>>> +        to enable complete x2apic.  Currently it only supports kvm
>>> +        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
>>> +        yet supported.
>> maybe explain how the default is chosen (based on kernel-irqhcip mode?)
> 
> Will do.
> 
>>> +
>>> +    ``caching-mode=on|off`` (default: off)
>>> +        This enables caching mode for the VT-d emulated device.  When
>>> +        caching-mode is enabled, each guest DMA buffer mapping will generate an
>>> +        IOTLB invalidation from the guest IOMMU driver to the vIOMMU device in
>>> +        a synchronous way.  It is required for ``-device vfio-pci`` to work
>>> +        with the VT-d device, because host assigned devices requires to setup
>>> +        the DMA mapping on the host before guest DMA starts.
>>> +
>>> +    ``device-iotlb=on|off`` (default: off)
>>> +        This enables device-iotlb capability for the emulated VT-d device.  So
>>> +        far virtio/vhost should be the only real user for this parameter,
>> I thought if was needed for vhost. What is the usage for virtio?
> 
> I used "virtio/vhost" because vhost is only one type of virtio backend, so
> normally they should be used in pair.  Also, the "ats=on" parameter should be
> an option to virtio frontend rather than vhost, hence I mentioned both.

OK makes sense.
> 
>>> +        paired with ats=on configured for the device.
>> What is not obvious to me is whether there is a kind of diagnosis
>> whether device-iotlb and caching mode settings are consistent with
>> vfio/vhost usage. Does qemu exit if there is inconsistency?
> 
> Caching mode check is done in vtd_machine_done_notify_one().
> 
> I think there's no such check for vhost, however iirc vhost should work
> with/without ats=on; I did't check deeper.
OK
> 
>>> +
>>> +    ``aw-bits=39|48`` (default: 39)
>>> +        This decides the address width of IOVA address space.  The address
>>> +        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
>>> +        4-level IOMMU page tables.
>>> +
>>> +    Please also refer to the wiki page for general scenarios of VT-d
>>> +    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
>>> +
>>>  ERST
>>>  
>>>  DEF("name", HAS_ARG, QEMU_OPTION_name,
>>>
>> Besides, it is quite useful.
> 
> Thanks for taking a look!
> 
Besides,

Reviewed-by: Eric Auger <eric.auger@redhat.com>


Eric