Re: device compatibility interface for live migration with assigned devices

[Jason Wang <jasowang@redhat.com>]

On 2020/7/16 下午4:32, Yan Zhao wrote:
> On Thu, Jul 16, 2020 at 12:16:26PM +0800, Jason Wang wrote:
>> On 2020/7/14 上午7:29, Yan Zhao wrote:
>>> hi folks,
>>> we are defining a device migration compatibility interface that helps upper
>>> layer stack like openstack/ovirt/libvirt to check if two devices are
>>> live migration compatible.
>>> The "devices" here could be MDEVs, physical devices, or hybrid of the two.
>>> e.g. we could use it to check whether
>>> - a src MDEV can migrate to a target MDEV,
>>> - a src VF in SRIOV can migrate to a target VF in SRIOV,
>>> - a src MDEV can migration to a target VF in SRIOV.
>>>     (e.g. SIOV/SRIOV backward compatibility case)
>>>
>>> The upper layer stack could use this interface as the last step to check
>>> if one device is able to migrate to another device before triggering a real
>>> live migration procedure.
>>> we are not sure if this interface is of value or help to you. please don't
>>> hesitate to drop your valuable comments.
>>>
>>>
>>> (1) interface definition
>>> The interface is defined in below way:
>>>
>>>                __    userspace
>>>                 /\              \
>>>                /                 \write
>>>               / read              \
>>>      ________/__________       ___\|/_____________
>>>     | migration_version |     | migration_version |-->check migration
>>>     ---------------------     ---------------------   compatibility
>>>        device A                    device B
>>>
>>>
>>> a device attribute named migration_version is defined under each device's
>>> sysfs node. e.g. (/sys/bus/pci/devices/0000\:00\:02.0/$mdev_UUID/migration_version).
>>
>> Are you aware of the devlink based device management interface that is
>> proposed upstream? I think it has many advantages over sysfs, do you
>> consider to switch to that?
> not familiar with the devlink. will do some research of it.
>>
>>> userspace tools read the migration_version as a string from the source device,
>>> and write it to the migration_version sysfs attribute in the target device.
>>>
>>> The userspace should treat ANY of below conditions as two devices not compatible:
>>> - any one of the two devices does not have a migration_version attribute
>>> - error when reading from migration_version attribute of one device
>>> - error when writing migration_version string of one device to
>>>     migration_version attribute of the other device
>>>
>>> The string read from migration_version attribute is defined by device vendor
>>> driver and is completely opaque to the userspace.
>>
>> My understanding is that something opaque to userspace is not the philosophy
> but the VFIO live migration in itself is essentially a big opaque stream to userspace.


I think it's better not limit to the kernel interface for a specific use 
case. This is basically the device introspection.


>
>> of Linux. Instead of having a generic API but opaque value, why not do in a
>> vendor specific way like:
>>
>> 1) exposing the device capability in a vendor specific way via sysfs/devlink
>> or other API
>> 2) management read capability in both src and dst and determine whether we
>> can do the migration
>>
>> This is the way we plan to do with vDPA.
>>
> yes, in another reply, Alex proposed to use an interface in json format.
> I guess we can define something like
>
> { "self" :
>    [
>      { "pciid" : "8086591d",
>        "driver" : "i915",
>        "gvt-version" : "v1",
>        "mdev_type"   : "i915-GVTg_V5_2",
>        "aggregator"  : "1",
>        "pv-mode"     : "none",
>      }
>    ],
>    "compatible" :
>    [
>      { "pciid" : "8086591d",
>        "driver" : "i915",
>        "gvt-version" : "v1",
>        "mdev_type"   : "i915-GVTg_V5_2",
>        "aggregator"  : "1"
>        "pv-mode"     : "none",
>      },
>      { "pciid" : "8086591d",
>        "driver" : "i915",
>        "gvt-version" : "v1",
>        "mdev_type"   : "i915-GVTg_V5_4",
>        "aggregator"  : "2"
>        "pv-mode"     : "none",
>      },
>      { "pciid" : "8086591d",
>        "driver" : "i915",
>        "gvt-version" : "v2",
>        "mdev_type"   : "i915-GVTg_V5_4",
>        "aggregator"  : "2"
>        "pv-mode"     : "none, ppgtt, context",
>      }
>      ...
>    ]
> }


This is probably another call for devlink base interface.


>
> But as those fields are mostly vendor specific, the userspace can
> only do simple string comparing, I guess the list would be very long as
> it needs to enumerate all possible targets.
> also, in some fileds like "gvt-version", is there a simple way to express
> things like v2+?


That's total vendor specific I think. If "v2+" means it only support a 
version 2+, we can introduce fields like min_version and max_version. 
But again, the point is to let such interfaces vendor specific instead 
of trying to have a generic format.


>
> If the userspace can read this interface both in src and target and
> check whether both src and target are in corresponding compatible list, I
> think it will work for us.
>
> But still, kernel should not rely on userspace's choice, the opaque
> compatibility string is still required in kernel. No matter whether
> it would be exposed to userspace as an compatibility checking interface,
> vendor driver would keep this part of code and embed the string into the
> migration stream.


Why? Can we simply do:

1) Src support feature A, B, C  (version 1.0)
2) Dst support feature A, B, C, D (version 2.0)
3) only enable feature A, B, C in destination in a version specific way 
(set version to 1.0)
4) migrate metadata A, B, C


>   so exposing it as an interface to be used by libvirt to
> do a safety check before a real live migration is only about enabling
> the kernel part of check to happen ahead.


If we've already exposed the capability, there's no need for an extra 
check like compatibility string.

Thanks


>
>
> Thanks
> Yan
>
>
>>
>>> for a Intel vGPU, string format can be defined like
>>> "parent device PCI ID" + "version of gvt driver" + "mdev type" + "aggregator count".
>>>
>>> for an NVMe VF connecting to a remote storage. it could be
>>> "PCI ID" + "driver version" + "configured remote storage URL"
>>>
>>> for a QAT VF, it may be
>>> "PCI ID" + "driver version" + "supported encryption set".
>>>
>>> (to avoid namespace confliction from each vendor, we may prefix a driver name to
>>> each migration_version string. e.g. i915-v1-8086-591d-i915-GVTg_V5_8-1)
>>>
>>>
>>> (2) backgrounds
>>>
>>> The reason we hope the migration_version string is opaque to the userspace
>>> is that it is hard to generalize standard comparing fields and comparing
>>> methods for different devices from different vendors.
>>> Though userspace now could still do a simple string compare to check if
>>> two devices are compatible, and result should also be right, it's still
>>> too limited as it excludes the possible candidate whose migration_version
>>> string fails to be equal.
>>> e.g. an MDEV with mdev_type_1, aggregator count 3 is probably compatible
>>> with another MDEV with mdev_type_3, aggregator count 1, even their
>>> migration_version strings are not equal.
>>> (assumed mdev_type_3 is of 3 times equal resources of mdev_type_1).
>>>
>>> besides that, driver version + configured resources are all elements demanding
>>> to take into account.
>>>
>>> So, we hope leaving the freedom to vendor driver and let it make the final decision
>>> in a simple reading from source side and writing for test in the target side way.
>>>
>>>
>>> we then think the device compatibility issues for live migration with assigned
>>> devices can be divided into two steps:
>>> a. management tools filter out possible migration target devices.
>>>      Tags could be created according to info from product specification.
>>>      we think openstack/ovirt may have vendor proprietary components to create
>>>      those customized tags for each product from each vendor.
>>>      e.g.
>>>      for Intel vGPU, with a vGPU(a MDEV device) in source side, the tags to
>>>      search target vGPU are like:
>>>      a tag for compatible parent PCI IDs,
>>>      a tag for a range of gvt driver versions,
>>>      a tag for a range of mdev type + aggregator count
>>>
>>>      for NVMe VF, the tags to search target VF may be like:
>>>      a tag for compatible PCI IDs,
>>>      a tag for a range of driver versions,
>>>      a tag for URL of configured remote storage.
>>>
>>> b. with the output from step a, openstack/ovirt/libvirt could use our proposed
>>>      device migration compatibility interface to make sure the two devices are
>>>      indeed live migration compatible before launching the real live migration
>>>      process to start stream copying, src device stopping and target device
>>>      resuming.
>>>      It is supposed that this step would not bring any performance penalty as
>>>      -in kernel it's just a simple string decoding and comparing
>>>      -in openstack/ovirt, it could be done by extending current function
>>>       check_can_live_migrate_destination, along side claiming target resources.[1]
>>>
>>>
>>> [1] https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/libvirt-neutron-sriov-livemigration.html
>>>
>>> Thanks
>>> Yan
>>>