Re: [PATCH RFC] vfio: Documentation for the migration region

[Akira Yokosawa <akiyks@gmail.com>]

On Mon, 22 Nov 2021 20:20:42 -0400, Jason Gunthorpe wrote:
> On Mon, Nov 22, 2021 at 01:31:14PM -0700, Jonathan Corbet wrote:
>> Jason Gunthorpe <jgg@nvidia.com> writes:
>> 
>> > Provide some more complete documentation for the migration region's
>> > behavior, specifically focusing on the device_state bits and the whole
>> > system view from a VMM.
>> >
>> > Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
>> >  Documentation/driver-api/vfio.rst | 208 +++++++++++++++++++++++++++++-
>> >  1 file changed, 207 insertions(+), 1 deletion(-)
>> >
>> > Alex/Cornelia, here is the first draft of the requested documentation I promised
>> >
>> > We think it includes all the feedback from hns, Intel and NVIDIA on this mechanism.
>> >
>> > Our thinking is that NDMA would be implemented like this:
>> >
>> >    +#define VFIO_DEVICE_STATE_NDMA      (1 << 3)
>> >
>> > And a .add_capability ops will be used to signal to userspace driver support:
>> >
>> >    +#define VFIO_REGION_INFO_CAP_MIGRATION_NDMA    6
>> >
>> > I've described DIRTY TRACKING as a seperate concept here. With the current
>> > uAPI this would be controlled by VFIO_IOMMU_DIRTY_PAGES_FLAG_START, with our
>> > change in direction this would be per-tracker control, but no semantic change.
>> >
>> > Upon some agreement we'll include this patch in the next iteration of the mlx5 driver
>> > along with the NDMA bits.
>> >
>> > Thanks,
>> > Jason
>> >
>> > diff --git a/Documentation/driver-api/vfio.rst b/Documentation/driver-api/vfio.rst
>> > index c663b6f978255b..b28c6fb89ee92f 100644
>> > +++ b/Documentation/driver-api/vfio.rst
>> > @@ -242,7 +242,213 @@ group and can access them as follows::
>> >  VFIO User API
>> >  
>> > -Please see include/linux/vfio.h for complete API documentation.
>> > +Please see include/uapi/linux/vfio.h for complete API documentation.
>> > +
>> > +-------------------------------------------------------------------------------
>> > +
>> > +VFIO migration driver API
>> > +-------------------------------------------------------------------------------
>> > +
>> > +VFIO drivers that support migration implement a migration control register
>> > +called device_state in the struct vfio_device_migration_info which is in its
>> > +VFIO_REGION_TYPE_MIGRATION region.
>> > +
>> > +The device_state triggers device action both when bits are set/cleared and
>> > +continuous behavior for each bit. For VMMs they can also control if the VCPUs in
>> > +a VM are executing (VCPU RUNNING) and if the IOMMU is logging DMAs (DIRTY
>> > +TRACKING). These two controls are not part of the device_state register, KVM
>> > +will be used to control the VCPU and VFIO_IOMMU_DIRTY_PAGES_FLAG_START on the
>> > +container controls dirty tracking.
>> > +
>> > +Along with the device_state the migration driver provides a data window which
>> > +allows streaming migration data into or out of the device.
>> > +
>> > +A lot of flexibility is provided to userspace in how it operates these bits. The
>> > +reference flow for saving device state in a live migration, with all features:
>> > +
>> > +  RUNNING, VCPU_RUNNING
>> > +     Normal operating state

Hi.

So if you'd like definition lists of ReST, adding empty lines here

>> > +  RUNNING, DIRTY TRACKING, VCPU RUNNING>> > +     Log DMAs

and here should do the trick.

>> > +     Stream all memory

Ditto for the rest of the lists.

>> 
>> So I'd recommend actually building the docs and looking at the result;
>> this will not render the way you expect it to.  
> 
> Hum... It is really close to what I'd like, with the state names
> bolded and in something like an enumerated list
> 
> But on close inspection I see the text fragments have been assembled
> together. I'd probably try to make them into sentances than go to a
> literal block?

Please see ReST documentation for further info:

    https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#definition-lists

        Thanks, Akira
> 
> Thanks,
> Jason