Live Migration of Virtio Virtual Function

From: Max Gurtovoy <mgurtovoy@nvidia.com>
To: "virtio-comment@lists.oasis-open.org"
	<virtio-comment@lists.oasis-open.org>,
	Jason Wang <jasowang@redhat.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	"cohuck@redhat.com" <cohuck@redhat.com>
Cc: Parav Pandit <parav@nvidia.com>,
	Shahaf Shuler <shahafs@nvidia.com>, Ariel Adam <aadam@redhat.com>,
	Amnon Ilan <ailan@redhat.com>, Bodong Wang <bodong@nvidia.com>,
	Jason Gunthorpe <jgg@nvidia.com>,
	Stefan Hajnoczi <stefanha@redhat.com>,
	Eugenio Perez Martin <eperezma@redhat.com>,
	Liran Liss <liranl@nvidia.com>, Oren Duer <oren@nvidia.com>
Subject: Live Migration of Virtio Virtual Function
Date: Thu, 12 Aug 2021 12:08:41 +0000	[thread overview]
Message-ID: <DM4PR12MB504062CEF6BEDABE6F76DEC2DEF99@DM4PR12MB5040.namprd12.prod.outlook.com> (raw)

[-- Attachment #1: Type: text/plain, Size: 1880 bytes --]

Hi all,
Live migration is one of the most important features of virtualization and virtio devices are oftenly found in virtual environments.

The migration process is managed by a migration SW that is running on the hypervisor and the VM is not aware of the process at all.

Unlike the vDPA case, a real pci Virtual Function state resides in the HW.

In our vision, in order to fulfil the Live migration requirements for virtual functions, each physical function device must implement migration operations. Using these operations, it will be able to master the migration process for the virtual function devices. Each capable physical function device has a supervisor permissions to change the virtual function operational states, save/restore its internal state and start/stop dirty pages tracking.

An example of this approach can be seen in the way NVIDIA performs live migration of a ConnectX NIC function:
https://github.com/jgunthorpe/linux/commits/mlx5_vfio_pci

NVIDIAs SNAP technology enables hardware-accelerated software defined PCIe devices. virtio-blk/virtio-net/virtio-fs SNAP used for storage and networking solutions. The host OS/hypervisor uses its standard drivers that are implemented according to a well-known VIRTIO specifications.

In order to implement Live Migration for these virtual function devices, that use a standard drivers as mentioned, the specification should define how HW vendor should build their devices and for SW developers to adjust the drivers.
This will enable specification compliant vendor agnostic solution.

This is exactly how we built the migration driver for ConnectX (internal HW design doc) and I guess that this is the way other vendors work.

For that, I would like to know if the approach of "PF that controls the VF live migration process" is acceptable by the VIRTIO technical group ?

Cheers,
-Max.

[-- Attachment #2: Type: text/html, Size: 4357 bytes --]