Re: device compatibility interface for live migration with assigned devices

[Jason Wang <jasowang@redhat.com>]

On 2020/7/20 下午6:39, Sean Mooney wrote:
> On Mon, 2020-07-20 at 11:41 +0800, Jason Wang wrote:
>> On 2020/7/18 上午12:12, Alex Williamson wrote:
>>> On Thu, 16 Jul 2020 16:32:30 +0800
>>> Yan Zhao <yan.y.zhao@intel.com> wrote:
>>>
>>>> On Thu, Jul 16, 2020 at 12:16:26PM +0800, Jason Wang wrote:
>>>>> On 2020/7/14 上午7:29, Yan Zhao wrote:
>>>>>> hi folks,
>>>>>> we are defining a device migration compatibility interface that helps upper
>>>>>> layer stack like openstack/ovirt/libvirt to check if two devices are
>>>>>> live migration compatible.
>>>>>> The "devices" here could be MDEVs, physical devices, or hybrid of the two.
>>>>>> e.g. we could use it to check whether
>>>>>> - a src MDEV can migrate to a target MDEV,
>>>>>> - a src VF in SRIOV can migrate to a target VF in SRIOV,
>>>>>> - a src MDEV can migration to a target VF in SRIOV.
>>>>>>      (e.g. SIOV/SRIOV backward compatibility case)
>>>>>>
>>>>>> The upper layer stack could use this interface as the last step to check
>>>>>> if one device is able to migrate to another device before triggering a real
>>>>>> live migration procedure.
>>>>>> we are not sure if this interface is of value or help to you. please don't
>>>>>> hesitate to drop your valuable comments.
>>>>>>
>>>>>>
>>>>>> (1) interface definition
>>>>>> The interface is defined in below way:
>>>>>>
>>>>>>                 __    userspace
>>>>>>                  /\              \
>>>>>>                 /                 \write
>>>>>>                / read              \
>>>>>>       ________/__________       ___\|/_____________
>>>>>>      | migration_version |     | migration_version |-->check migration
>>>>>>      ---------------------     ---------------------   compatibility
>>>>>>         device A                    device B
>>>>>>
>>>>>>
>>>>>> a device attribute named migration_version is defined under each device's
>>>>>> sysfs node. e.g. (/sys/bus/pci/devices/0000\:00\:02.0/$mdev_UUID/migration_version).
>>>>> Are you aware of the devlink based device management interface that is
>>>>> proposed upstream? I think it has many advantages over sysfs, do you
>>>>> consider to switch to that?
>>> Advantages, such as?
>>
>> My understanding for devlink(netlink) over sysfs (some are mentioned at
>> the time of vDPA sysfs mgmt API discussion) are:
> i tought netlink was used more a as a configuration protocoal to qurry and confire nic and i guess
> other devices in its devlink form requireint a tool to be witten that can speak the protocal to interact with.
> the primary advantate of sysfs is that everything is just a file. there are no addtional depleenceis
> needed


Well, if you try to build logic like introspection on top for a 
sophisticated hardware, you probably need to have library on top. And 
it's attribute per file is pretty inefficient.


>   and unlike netlink there are not interoperatblity issues in a coanitnerised env. if you are using diffrenet
> version of libc and gcc in the contaienr vs the host my understanding is tools like ethtool from ubuntu deployed
> in a container on a centos host can have issue communicating with the host kernel.


Kernel provides stable ABI for userspace, so it's not something that we 
can't fix.


> if its jsut a file unless
> the format the data is returnin in chagnes or the layout of sysfs changes its compatiable regardless of what you
> use to read it.


I believe you can't change sysfs layout which is part of uABI. But as I 
mentioned below, sysfs has several drawbacks. It's not harm to compare 
between different approach when you start a new device management API.

Thanks


>> - existing users (NIC, crypto, SCSI, ib), mature and stable
>> - much better error reporting (ext_ack other than string or errno)
>> - namespace aware
>> - do not couple with kobject
>>
>> Thanks
>>