Re: [PATCH 5/25] Xen/doc: Add Xen virtual IOMMU doc

[Julien Grall <julien.grall@arm.com>]

Hi,

On 06/29/2017 06:50 AM, Lan Tianyu wrote:
> This patch is to add Xen virtual IOMMU doc to introduce motivation,
> framework, vIOMMU hypercall and xl configuration.
> 
> Signed-off-by: Lan Tianyu <tianyu.lan@intel.com>
> ---
>   docs/misc/viommu.txt | 129 +++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 129 insertions(+)
>   create mode 100644 docs/misc/viommu.txt
> 
> diff --git a/docs/misc/viommu.txt b/docs/misc/viommu.txt
> new file mode 100644
> index 0000000..76d4cee
> --- /dev/null
> +++ b/docs/misc/viommu.txt
> @@ -0,0 +1,129 @@
> +Xen virtual IOMMU
> +
> +Motivation
> +==========
> +*) Enable more than 255 vcpu support
> +HPC cloud service requires VM provides high performance parallel
> +computing and we hope to create a huge VM with >255 vcpu on one machine
> +to meet such requirement. Pin each vcpu to separate pcpus.
> +
> +To support >255 vcpus, X2APIC mode in guest is necessary because legacy
> +APIC(XAPIC) just supports 8-bit APIC ID and it only can support 255
> +vcpus at most. X2APIC mode supports 32-bit APIC ID and it requires
> +interrupt mapping function of vIOMMU.
> +
> +The reason for this is that there is no modification to existing PCI MSI
> +and IOAPIC with the introduction of X2APIC. PCI MSI/IOAPIC can only send
> +interrupt message containing 8-bit APIC ID, which cannot address >255
> +cpus. Interrupt remapping supports 32-bit APIC ID and so it's necessary
> +to enable >255 cpus with x2apic mode.
> +
> +
> +vIOMMU Architecture
> +===================
> +vIOMMU device model is inside Xen hypervisor for following factors
> +    1) Avoid round trips between Qemu and Xen hypervisor
> +    2) Ease of integration with the rest of hypervisor
> +    3) HVMlite/PVH doesn't use Qemu
> +
> +* Interrupt remapping overview.
> +Interrupts from virtual devices and physical devices are delivered
> +to vLAPIC from vIOAPIC and vMSI. vIOMMU needs to remap interrupt during
> +this procedure.
> +
> ++---------------------------------------------------+
> +|Qemu                       |VM                     |
> +|                           | +----------------+    |
> +|                           | |  Device driver |    |
> +|                           | +--------+-------+    |
> +|                           |          ^            |
> +|       +----------------+  | +--------+-------+    |
> +|       | Virtual device |  | |  IRQ subsystem |    |
> +|       +-------+--------+  | +--------+-------+    |
> +|               |           |          ^            |
> +|               |           |          |            |
> ++---------------------------+-----------------------+
> +|hyperviosr     |                      | VIRQ       |
> +|               |            +---------+--------+   |
> +|               |            |      vLAPIC      |   |
> +|               |VIRQ        +---------+--------+   |
> +|               |                      ^            |
> +|               |                      |            |
> +|               |            +---------+--------+   |
> +|               |            |      vIOMMU      |   |
> +|               |            +---------+--------+   |
> +|               |                      ^            |
> +|               |                      |            |
> +|               |            +---------+--------+   |
> +|               |            |   vIOAPIC/vMSI   |   |
> +|               |            +----+----+--------+   |
> +|               |                 ^    ^            |
> +|               +-----------------+    |            |
> +|                                      |            |
> ++---------------------------------------------------+
> +HW                                     |IRQ
> +                                +-------------------+
> +                                |   PCI Device      |
> +                                +-------------------+
> +
> +
> +vIOMMU hypercall
> +================
> +Introduce new domctl hypercall "xen_domctl_viommu_op" to create/destroy
> +vIOMMU and query vIOMMU capabilities that device model can support.
> +
> +* vIOMMU hypercall parameter structure
> +struct xen_domctl_viommu_op {
> +    uint32_t cmd;
> +#define XEN_DOMCTL_create_viommu          0
> +#define XEN_DOMCTL_destroy_viommu         1
> +#define XEN_DOMCTL_query_viommu_caps      2

I am a bit confused. This is only creating the vIOMMU. However, there 
might be multiple host IOMMUs, how do you link them together?

> +    union {
> +        struct {
> +            /* IN - vIOMMU type */
> +            uint64_t viommu_type;

This is a bit confusing, you don't define what should be the value of 
viommu_type, ...

> +            /* IN - MMIO base address of vIOMMU. */
> +            uint64_t base_address;
> +            /* IN - Length of MMIO region */
> +            uint64_t length; > +            /* IN - Capabilities with which we want to create */
> +            uint64_t capabilities;

... capabilities ...

> +            /* OUT - vIOMMU identity */
> +            uint32_t viommu_id;
> +        } create_viommu; > +
> +        struct {
> +            /* IN - vIOMMU identity */
> +            uint32_t viommu_id;
> +        } destroy_viommu;
> +
> +        struct {
> +            /* IN - vIOMMU type */
> +            uint64_t viommu_type; > +            /* OUT - vIOMMU Capabilities */
> +            uint64_t caps;

... and caps. I see you have defined them in a separate header 
(viommu.h). But there are no way for the developer to know that they 
should be used.

> +        } query_caps;
> +    } u;
> +};
> +
> +- XEN_DOMCTL_query_viommu_caps
> +    Query capabilities of vIOMMU device model. vIOMMU_type specifies
> +which vendor vIOMMU device model(E,G Intel VTD) is targeted and hypervisor

"E,G" did you mean "e.g"?

> +returns capability bits(E,G interrupt remapping bit).

Ditto.

A given platform may have multiple IOMMUs with different features. Are 
we expecting

> +
> +- XEN_DOMCTL_create_viommu
> +    Create vIOMMU device with vIOMMU_type, capabilities, MMIO
> +base address and length. Hypervisor returns viommu_id. Capabilities should
> +be in range of value returned by query_viommu_caps hypercall.

Can you explain what mmio and length are here for? Do you expect to trap 
and emulate the MMIO region in Xen?

 From just looking at the document. I am struggling to understand how 
this is going to be useful.

> +
> +- XEN_DOMCTL_destroy_viommu
> +    Destroy vIOMMU in Xen hypervisor with viommu_id as parameters.
> +
> +xl vIOMMU configuration
> +=======================
> +viommu="type=vtd,intremap=1,x2apic=1"
> +
> +"type" - Specify vIOMMU device model type. Currently only supports Intel vtd
> +device model.
> +"intremap" - Enable vIOMMU interrupt remapping function.
> +"x2apic" - Support x2apic mode with interrupt remapping function.
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel