Re: [RFC PATCH v2 00/19] Control VQ support in vDPA

From: Jason Wang <jasowang@redhat.com>
To: Gautam Dawar <gautam.dawar@xilinx.com>
Cc: gdawar@xilinx.com, martinh@xilinx.com, hanand@xilinx.com,
	tanujk@xilinx.com, eperezma@redhat.com,
	"Michael S. Tsirkin" <mst@redhat.com>,
	Zhu Lingshan <lingshan.zhu@intel.com>,
	Stefano Garzarella <sgarzare@redhat.com>,
	Xie Yongji <xieyongji@bytedance.com>, Eli Cohen <elic@nvidia.com>,
	Si-Wei Liu <si-wei.liu@oracle.com>,
	Parav Pandit <parav@nvidia.com>, Longpeng <longpeng2@huawei.com>,
	virtualization@lists.linux-foundation.org,
	linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
	netdev@vger.kernel.org
Subject: Re: [RFC PATCH v2 00/19] Control VQ support in vDPA
Date: Mon, 28 Feb 2022 16:17:41 +0800	[thread overview]
Message-ID: <19843d2b-dfe9-5e2d-4d3d-ca55456947dc@redhat.com> (raw)
In-Reply-To: <20220224212314.1326-1-gdawar@xilinx.com>

在 2022/2/25 上午5:22, Gautam Dawar 写道:
> Hi All:
>
> This series tries to add the support for control virtqueue in vDPA.
>
> Control virtqueue is used by networking device for accepting various
> commands from the driver. It's a must to support multiqueue and other
> configurations.
>
> When used by vhost-vDPA bus driver for VM, the control virtqueue
> should be shadowed via userspace VMM (Qemu) instead of being assigned
> directly to Guest. This is because Qemu needs to know the device state
> in order to start and stop device correctly (e.g for Live Migration).
>
> This requies to isolate the memory mapping for control virtqueue
> presented by vhost-vDPA to prevent guest from accessing it directly.
>
> To achieve this, vDPA introduce two new abstractions:
>
> - address space: identified through address space id (ASID) and a set
>                   of memory mapping in maintained
> - virtqueue group: the minimal set of virtqueues that must share an
>                   address space
>
> Device needs to advertise the following attributes to vDPA:
>
> - the number of address spaces supported in the device
> - the number of virtqueue groups supported in the device
> - the mappings from a specific virtqueue to its virtqueue groups
>
> The mappings from virtqueue to virtqueue groups is fixed and defined
> by vDPA device driver. E.g:
>
> - For the device that has hardware ASID support, it can simply
>    advertise a per virtqueue virtqueue group.
> - For the device that does not have hardware ASID support, it can
>    simply advertise a single virtqueue group that contains all
>    virtqueues. Or if it wants a software emulated control virtqueue, it
>    can advertise two virtqueue groups, one is for cvq, another is for
>    the rest virtqueues.
>
> vDPA also allow to change the association between virtqueue group and
> address space. So in the case of control virtqueue, userspace
> VMM(Qemu) may use a dedicated address space for the control virtqueue
> group to isolate the memory mapping.
>
> The vhost/vhost-vDPA is also extend for the userspace to:
>
> - query the number of virtqueue groups and address spaces supported by
>    the device
> - query the virtqueue group for a specific virtqueue
> - assocaite a virtqueue group with an address space
> - send ASID based IOTLB commands
>
> This will help userspace VMM(Qemu) to detect whether the control vq
> could be supported and isolate memory mappings of control virtqueue
> from the others.
>
> To demonstrate the usage, vDPA simulator is extended to support
> setting MAC address via a emulated control virtqueue.
>
> Please review.
>
> Changes since v1:
>
> - Rebased the v1 patch series on vhost branch of MST vhost git repo
>    git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/log/?h=vhost
> - Updates to accommodate vdpa_sim changes from monolithic module in
>    kernel used v1 patch series to current modularized class (net, block)
>    based approach.
> - Added new attributes (ngroups and nas) to "vdpasim_dev_attr" and
>    propagated them from vdpa_sim_net to vdpa_sim
> - Widened the data-type for "asid" member of vhost_msg_v2 to __u32
>    to accommodate PASID

This is great. Then the semantic matches exactly the PASID proposal here[1].

> - Fixed the buildbot warnings
> - Resolved all checkpatch.pl errors and warnings
> - Tested both control and datapath with Xilinx Smartnic SN1000 series
>    device using QEMU implementing the Shadow virtqueue and support for
>    VQ groups and ASID available at:
>    github.com/eugpermar/qemu/releases/tag/vdpa_sw_live_migration.d%2F
>    asid_groups-v1.d%2F00

On top, we may extend the netlink protocol to report the mapping between 
virtqueue to its groups.

Thanks

[1] 
https://www.mail-archive.com/virtio-dev@lists.oasis-open.org/msg08077.html

>
> Changes since RFC:
>
> - tweak vhost uAPI documentation
> - switch to use device specific IOTLB really in patch 4
> - tweak the commit log
> - fix that ASID in vhost is claimed to be 32 actually but 16bit
>    actually
> - fix use after free when using ASID with IOTLB batching requests
> - switch to use Stefano's patch for having separated iov
> - remove unused "used_as" variable
> - fix the iotlb/asid checking in vhost_vdpa_unmap()
>
> Thanks
>
> Gautam Dawar (19):
>    vhost: move the backend feature bits to vhost_types.h
>    virtio-vdpa: don't set callback if virtio doesn't need it
>    vhost-vdpa: passing iotlb to IOMMU mapping helpers
>    vhost-vdpa: switch to use vhost-vdpa specific IOTLB
>    vdpa: introduce virtqueue groups
>    vdpa: multiple address spaces support
>    vdpa: introduce config operations for associating ASID to a virtqueue
>      group
>    vhost_iotlb: split out IOTLB initialization
>    vhost: support ASID in IOTLB API
>    vhost-vdpa: introduce asid based IOTLB
>    vhost-vdpa: introduce uAPI to get the number of virtqueue groups
>    vhost-vdpa: introduce uAPI to get the number of address spaces
>    vhost-vdpa: uAPI to get virtqueue group id
>    vhost-vdpa: introduce uAPI to set group ASID
>    vhost-vdpa: support ASID based IOTLB API
>    vdpa_sim: advertise VIRTIO_NET_F_MTU
>    vdpa_sim: factor out buffer completion logic
>    vdpa_sim: filter destination mac address
>    vdpasim: control virtqueue support
>
>   drivers/vdpa/ifcvf/ifcvf_main.c      |   8 +-
>   drivers/vdpa/mlx5/net/mlx5_vnet.c    |  11 +-
>   drivers/vdpa/vdpa.c                  |   5 +
>   drivers/vdpa/vdpa_sim/vdpa_sim.c     | 100 ++++++++--
>   drivers/vdpa/vdpa_sim/vdpa_sim.h     |   3 +
>   drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 169 +++++++++++++----
>   drivers/vhost/iotlb.c                |  23 ++-
>   drivers/vhost/vdpa.c                 | 272 +++++++++++++++++++++------
>   drivers/vhost/vhost.c                |  23 ++-
>   drivers/vhost/vhost.h                |   4 +-
>   drivers/virtio/virtio_vdpa.c         |   2 +-
>   include/linux/vdpa.h                 |  46 ++++-
>   include/linux/vhost_iotlb.h          |   2 +
>   include/uapi/linux/vhost.h           |  25 ++-
>   include/uapi/linux/vhost_types.h     |  11 +-
>   15 files changed, 566 insertions(+), 138 deletions(-)
>