Re: [RFC v4 01/21] vfio-user: introduce vfio-user protocol specification

From: Alex Williamson <alex.williamson@redhat.com>
To: John Johnson <john.g.johnson@oracle.com>
Cc: qemu-devel@nongnu.org
Subject: Re: [RFC v4 01/21] vfio-user: introduce vfio-user protocol specification
Date: Wed, 9 Mar 2022 15:34:53 -0700	[thread overview]
Message-ID: <20220309153453.25eee9dd.alex.williamson@redhat.com> (raw)
In-Reply-To: <a9b696ca38ee2329e371c28bcaa2921cac2a48a2.1641584316.git.john.g.johnson@oracle.com>

On Tue, 11 Jan 2022 16:43:37 -0800
John Johnson <john.g.johnson@oracle.com> wrote:
> +VFIO region info cap sparse mmap
> +""""""""""""""""""""""""""""""""
> +
> ++----------+--------+------+
> +| Name     | Offset | Size |
> ++==========+========+======+
> +| nr_areas | 0      | 4    |
> ++----------+--------+------+
> +| reserved | 4      | 4    |
> ++----------+--------+------+
> +| offset   | 8      | 8    |
> ++----------+--------+------+
> +| size     | 16     | 9    |
> ++----------+--------+------+

Typo, I'm pretty sure size isn't 9 bytes.

> +| ...      |        |      |
> ++----------+--------+------+
> +
> +* *nr_areas* is the number of sparse mmap areas in the region.
> +* *offset* and size describe a single area that can be mapped by the client.
> +  There will be *nr_areas* pairs of offset and size. The offset will be added to
> +  the base offset given in the ``VFIO_USER_DEVICE_GET_REGION_INFO`` to form the
> +  offset argument of the subsequent mmap() call.
> +
> +The VFIO sparse mmap area is defined in ``<linux/vfio.h>`` (``struct
> +vfio_region_info_cap_sparse_mmap``).
> +
> +VFIO region type cap header
> +"""""""""""""""""""""""""""
> +
> ++------------------+---------------------------+
> +| Name             | Value                     |
> ++==================+===========================+
> +| id               | VFIO_REGION_INFO_CAP_TYPE |
> ++------------------+---------------------------+
> +| version          | 0x1                       |
> ++------------------+---------------------------+
> +| next             | <next>                    |
> ++------------------+---------------------------+
> +| region info type | VFIO region info type     |
> ++------------------+---------------------------+
> +
> +This capability is defined when a region is specific to the device.
> +
> +VFIO region info type cap
> +"""""""""""""""""""""""""
> +
> +The VFIO region info type is defined in ``<linux/vfio.h>``
> +(``struct vfio_region_info_cap_type``).
> +
> ++---------+--------+------+
> +| Name    | Offset | Size |
> ++=========+========+======+
> +| type    | 0      | 4    |
> ++---------+--------+------+
> +| subtype | 4      | 4    |
> ++---------+--------+------+
> +
> +The only device-specific region type and subtype supported by vfio-user is
> +``VFIO_REGION_TYPE_MIGRATION`` (3) and ``VFIO_REGION_SUBTYPE_MIGRATION`` (1).

These should be considered deprecated from the kernel interface.  I
hope there are plans for vfio-user to adopt the new interface that's
currently available in linux-next and intended for v5.18.

...
> +Unused VFIO ``ioctl()`` commands
> +--------------------------------
> +
> +The following VFIO commands do not have an equivalent vfio-user command:
> +
> +* ``VFIO_GET_API_VERSION``
> +* ``VFIO_CHECK_EXTENSION``
> +* ``VFIO_SET_IOMMU``
> +* ``VFIO_GROUP_GET_STATUS``
> +* ``VFIO_GROUP_SET_CONTAINER``
> +* ``VFIO_GROUP_UNSET_CONTAINER``
> +* ``VFIO_GROUP_GET_DEVICE_FD``
> +* ``VFIO_IOMMU_GET_INFO``
> +
> +However, once support for live migration for VFIO devices is finalized some
> +of the above commands may have to be handled by the client in their
> +corresponding vfio-user form. This will be addressed in a future protocol
> +version.

As above, I'd go ahead and drop the migration region interface support,
it's being removed from the kernel.  Dirty page handling might also be
something you want to pull back on as we're expecting in-kernel vfio to
essentially deprecate its iommu backends in favor of a new shared
userspace iommufd interface.  We expect to have backwards compatibility
via that interface, but as QEMU migration support for vfio-pci devices
is experimental and there are desires not to consolidate dirty page
tracking behind the iommu interface in the new model, it's not clear if
the kernel will continue to expose the current dirty page tracking.

AIUI, we're expecting to see patches officially proposing the iommufd
interface in the kernel "soon".  Thanks,

Alex