Re: [PATCH v7 4/4] docs: Add Documentation for Mediated devices

[Kirti Wankhede <kwankhede@nvidia.com>]

Adding Eric.

Eric,
This is the v7 version of patch. I'll incorporate changes that you
suggested here.

Kirti.

On 8/25/2016 9:23 AM, Kirti Wankhede wrote:
> Add file Documentation/vfio-mediated-device.txt that include details of
> mediated device framework.
> 
> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> Signed-off-by: Neo Jia <cjia@nvidia.com>
> Change-Id: I137dd646442936090d92008b115908b7b2c7bc5d
> Reviewed-on: http://git-master/r/1182512
> Reviewed-by: Automatic_Commit_Validation_User
> ---
>  Documentation/vfio-mediated-device.txt | 203 +++++++++++++++++++++++++++++++++
>  1 file changed, 203 insertions(+)
>  create mode 100644 Documentation/vfio-mediated-device.txt
> 
> diff --git a/Documentation/vfio-mediated-device.txt b/Documentation/vfio-mediated-device.txt
> new file mode 100644
> index 000000000000..237d8eb630b7
> --- /dev/null
> +++ b/Documentation/vfio-mediated-device.txt
> @@ -0,0 +1,203 @@
> +VFIO Mediated devices [1]
> +-------------------------------------------------------------------------------
> +
> +There are more and more use cases/demands to virtualize the DMA devices which
> +doesn't have SR_IOV capability built-in. To do this, drivers of different
> +devices had to develop their own management interface and set of APIs and then
> +integrate it to user space software. We've identified common requirements and
> +unified management interface for such devices to make user space software
> +integration easier.
> +
> +The VFIO driver framework provides unified APIs for direct device access. It is
> +an IOMMU/device agnostic framework for exposing direct device access to
> +user space, in a secure, IOMMU protected environment. This framework is
> +used for multiple devices like GPUs, network adapters and compute accelerators.
> +With direct device access, virtual machines or user space applications have
> +direct access of physical device. This framework is reused for mediated devices.
> +
> +Mediated core driver provides a common interface for mediated device management
> +that can be used by drivers of different devices. This module provides a generic
> +interface to create/destroy mediated device, add/remove it to mediated bus
> +driver, add/remove device to IOMMU group. It also provides an interface to
> +register bus driver, for example, Mediated VFIO mdev driver is designed for
> +mediated devices and supports VFIO APIs. Mediated bus driver add/delete mediated
> +device to VFIO Group.
> +
> +Below is the high Level block diagram, with NVIDIA, Intel and IBM devices
> +as example, since these are the devices which are going to actively use
> +this module as of now.
> +
> +     +---------------+
> +     |               |
> +     | +-----------+ |  mdev_register_driver() +--------------+
> +     | |           | +<------------------------+              |
> +     | |  mdev     | |                         |              |
> +     | |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user
> +     | |  driver   | |     probe()/remove()    |              |    APIs
> +     | |           | |                         +--------------+
> +     | +-----------+ |
> +     |               |
> +     |  MDEV CORE    |
> +     |   MODULE      |
> +     |   mdev.ko     |
> +     | +-----------+ |  mdev_register_device() +--------------+
> +     | |           | +<------------------------+              |
> +     | |           | |                         |  nvidia.ko   |<-> physical
> +     | |           | +------------------------>+              |    device
> +     | |           | |        callbacks        +--------------+
> +     | | Physical  | |
> +     | |  device   | |  mdev_register_device() +--------------+
> +     | | interface | |<------------------------+              |
> +     | |           | |                         |  i915.ko     |<-> physical
> +     | |           | +------------------------>+              |    device
> +     | |           | |        callbacks        +--------------+
> +     | |           | |
> +     | |           | |  mdev_register_device() +--------------+
> +     | |           | +<------------------------+              |
> +     | |           | |                         | ccw_device.ko|<-> physical
> +     | |           | +------------------------>+              |    device
> +     | |           | |        callbacks        +--------------+
> +     | +-----------+ |
> +     +---------------+
> +
> +
> +Registration Interfaces
> +-------------------------------------------------------------------------------
> +
> +Mediated core driver provides two types of registration interfaces:
> +
> +1. Registration interface for mediated bus driver:
> +-------------------------------------------------
> +     /*
> +      * struct mdev_driver [2] - Mediated device's driver
> +      * @name: driver name
> +      * @probe: called when new device created
> +      * @remove: called when device removed
> +      * @driver: device driver structure
> +      */
> +     struct mdev_driver {
> +	     const char *name;
> +	     int  (*probe)  (struct device *dev);
> +	     void (*remove) (struct device *dev);
> +	     struct device_driver    driver;
> +     };
> +
> +Mediated bus driver for mdev should use this interface to register and
> +unregister with core driver respectively:
> +
> +extern int  mdev_register_driver(struct mdev_driver *drv, struct module *owner);
> +extern void mdev_unregister_driver(struct mdev_driver *drv);
> +
> +Mediated bus driver is responsible to add/delete mediated devices to/from VFIO
> +group when devices are bound and unbound to the driver.
> +
> +2. Physical device driver interface:
> +-----------------------------------
> +This interface [3] provides a set of APIs to manage physical device related work
> +in its driver. APIs are:
> +
> +* dev_attr_groups: attributes of the parent device.
> +* mdev_attr_groups: attributes of the mediated device.
> +* supported_config: to provide supported configuration list by the driver.
> +* create: to allocate basic resources in driver for a mediated device.
> +* destroy: to free resources in driver when mediated device is destroyed.
> +* reset: to free and reallocate resources in driver on mediated device reset.
> +* set_online_status: to change online status of mediated device.
> +* get_online_status: to get current (online/offline) status of mediated device.
> +* read : read emulation callback.
> +* write: write emulation callback.
> +* mmap: mmap emulation callback.
> +* get_irq_info: to retrieve information about mediated device's IRQ.
> +* set_irqs: gives interrupt configuration information that VMM sets.
> +* get_region_info: to provide region size and its flags for the mediated device.
> +    Vendor driver can provide the capability id and corresponding capability
> +    structure if want to support a capability.
> +* get_device_info: to retrieve VFIO device related flags, number of regions and
> +  number of IRQs supported.
> +
> +Drivers should use this interface to register and unregister device to mdev core
> +driver respectively:
> +
> +extern int  mdev_register_device(struct device *dev,
> +                                 const struct parent_ops *ops);
> +extern void mdev_unregister_device(struct device *dev);
> +
> +Mediated device management interface via sysfs
> +-------------------------------------------------------------------------------
> +This is the interface that allows user space software, like libvirt, to query
> +and configure mediated device in a HW agnostic fashion. This management
> +interface provide flexibility to underlying physical device's driver to support
> +mediated device hotplug, multiple mediated devices per virtual machine, multiple
> +mediated devices from different physical devices, etc.
> +
> +Under per-physical device sysfs:
> +--------------------------------
> +
> +* mdev_supported_types: (read only)
> +    List the current supported mediated device types and its details.
> +
> +* mdev_create: (write only)
> +	Create a mediated device on target physical device.
> +	Input syntax: <UUID:params>
> +	where,
> +		UUID: mediated device's UUID
> +		params: extra parameters required by driver
> +	Example:
> +	# echo "12345678-1234-1234-1234-123456789abc:0" >
> +				 /sys/bus/pci/devices/0000\:05\:00.0/mdev_create
> +
> +* mdev_destroy: (write only)
> +	Destroy a mediated device on a target physical device.
> +	Input syntax: <UUID>
> +	where,
> +		UUID: mediated device's UUID
> +	Example:
> +	# echo "12345678-1234-1234-1234-123456789abc" >
> +			       /sys/bus/pci/devices/0000\:05\:00.0/mdev_destroy
> +
> +Under per mdev device:
> +----------------------------------------
> +
> +* online: (read write)
> +	Read on this file provides current status of mediated device (0 or 1).
> +	Write on this file (0 or 1) will change the state of mediated device.
> +	This trigger the registration callback to notify the driver to commit
> +	or free mediated device resources. This callback is blocking call,
> +	successful return of this call will indicate requested mdev resources
> +	has been fully committed, the VMM should continue.
> +	Example:
> +	# echo "1|0" > /sys/bus/mdev/devices/$mdev_UUID/online
> +
> +
> +Mediated device Hotplug:
> +-----------------------
> +
> +To support mediated device hotplug, <mdev_create> and <mdev_destroy> can be
> +accessed during VM runtime, and the corresponding registration callback is
> +invoked to allow driver to support hotplug.
> +
> +Translation APIs for Mediated device
> +------------------------------------------------------------------------------
> +
> +Below APIs are provided for user pfn to host pfn translation in VFIO driver:
> +
> +extern long vfio_pin_pages(struct device *dev, unsigned long *user_pfn,
> +                           long npage, int prot, unsigned long *phys_pfn);
> +
> +extern long vfio_unpin_pages(struct device *dev, unsigned long *pfn,
> +			     long npage);
> +
> +These functions call back into the backend IOMMU module using two callbacks of
> +struct vfio_iommu_driver_ops, pin_pages and unpin_pages [4]. Currently these are
> +supported in TYPE1 IOMMU module. To enable the same for other IOMMU backend
> +modules, such as PPC64 sPAPR module, they need to provide these two callback
> +functions.
> +
> +References
> +-------------------------------------------------------------------------------
> +
> +[1] See Documentation/vfio.txt for more information on VFIO.
> +[2] struct mdev_driver in include/linux/mdev.h
> +[3] struct parent_ops in include/linux/mdev.h
> +[4] struct vfio_iommu_driver_ops in include/linux/vfio.h
> +
>