Re: [PATCH 21/21] vdpasim: control virtqueue support

From: Eugenio Perez Martin <eperezma@redhat.com>
To: Jason Wang <jasowang@redhat.com>
Cc: Eli Cohen <elic@nvidia.com>, Michael Tsirkin <mst@redhat.com>,
	kvm list <kvm@vger.kernel.org>,
	virtualization@lists.linux-foundation.org,
	netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
	Cindy Lu <lulu@redhat.com>, Eli Cohen <eli@mellanox.com>,
	lingshan.zhu@intel.com, Rob Miller <rob.miller@broadcom.com>,
	Stefan Hajnoczi <stefanha@redhat.com>,
	Stefano Garzarella <sgarzare@redhat.com>
Subject: Re: [PATCH 21/21] vdpasim: control virtqueue support
Date: Fri, 22 Jan 2021 20:43:34 +0100	[thread overview]
Message-ID: <CAJaqyWcRirQgz+n63rU2nYVH2RKqjQfwHGFLzOG=H46qRWuTog@mail.gmail.com> (raw)
In-Reply-To: <da50981b-6bbc-ee61-b5b1-a57a09da8e93@redhat.com>

On Tue, Jan 12, 2021 at 4:12 AM Jason Wang <jasowang@redhat.com> wrote:
>
>
> On 2021/1/11 下午8:26, Eli Cohen wrote:
> > On Wed, Dec 16, 2020 at 02:48:18PM +0800, Jason Wang wrote:
> >> This patch introduces the control virtqueue support for vDPA
> >> simulator. This is a requirement for supporting advanced features like
> >> multiqueue.
> >>
> >> A requirement for control virtqueue is to isolate its memory access
> >> from the rx/tx virtqueues. This is because when using vDPA device
> >> for VM, the control virqueue is not directly assigned to VM. Userspace
> >> (Qemu) will present a shadow control virtqueue to control for
> >> recording the device states.
> >>
> >> The isolation is done via the virtqueue groups and ASID support in
> >> vDPA through vhost-vdpa. The simulator is extended to have:
> >>
> >> 1) three virtqueues: RXVQ, TXVQ and CVQ (control virtqueue)
> >> 2) two virtqueue groups: group 0 contains RXVQ and TXVQ; group 1
> >>     contains CVQ
> >> 3) two address spaces and the simulator simply implements the address
> >>     spaces by mapping it 1:1 to IOTLB.
> >>
> >> For the VM use cases, userspace(Qemu) may set AS 0 to group 0 and AS 1
> >> to group 1. So we have:
> >>
> >> 1) The IOTLB for virtqueue group 0 contains the mappings of guest, so
> >>     RX and TX can be assigned to guest directly.
> >> 2) The IOTLB for virtqueue group 1 contains the mappings of CVQ which
> >>     is the buffers that allocated and managed by VMM only. So CVQ of
> >>     vhost-vdpa is visible to VMM only. And Guest can not access the CVQ
> >>     of vhost-vdpa.
> >>
> >> For the other use cases, since AS 0 is associated to all virtqueue
> >> groups by default. All virtqueues share the same mapping by default.
> >>
> >> To demonstrate the function, VIRITO_NET_F_CTRL_MACADDR is
> >> implemented in the simulator for the driver to set mac address.
> >>
> > Hi Jason,
> >
> > is there any version of qemu/libvirt available that I can see the
> > control virtqueue working in action?
>
>
> Not yet, the qemu part depends on the shadow virtqueue work of Eugenio.
> But it will work as:
>
> 1) qemu will use a separated address space for the control virtqueue
> (shadow) exposed through vhost-vDPA
> 2) the commands sent through control virtqueue by guest driver will
> intercept by qemu
> 3) Qemu will send those commands to the shadow control virtqueue
>
> Eugenio, any ETA for the new version of shadow virtqueue support in Qemu?
>

Hi Jason. Sorry for the late response.

For the notification part I have addressed all the issues of the RFC
[1], except the potential race conditions Stefan pointed, and tested
with vdpa devices. You can find at
https://github.com/eugpermar/qemu/tree/vdpa_sw_live_migration.d/notifications.rfc
. Since the shadow path is activated only through QMP and does not
interfere with regular operation, I could post to the qemu list if you
prefer. The series will be smaller if merged in steps.

Adding the buffer forwarding on top should not take long.

[1] https://lkml.org/lkml/2020/9/23/1243

Thanks!