From: "Eugenio Pérez" <eperezma@redhat.com>
To: qemu-devel@nongnu.org
Cc: Parav Pandit <parav@mellanox.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
Guru Prasad <guru.prasad@broadcom.com>,
Jason Wang <jasowang@redhat.com>,
Juan Quintela <quintela@redhat.com>,
Markus Armbruster <armbru@redhat.com>,
virtualization@lists.linux-foundation.org,
Harpreet Singh Anand <hanand@xilinx.com>,
Xiao W Wang <xiao.w.wang@intel.com>, Eli Cohen <eli@mellanox.com>,
Stefano Garzarella <sgarzare@redhat.com>,
Michael Lilja <ml@napatech.com>,
Jim Harford <jim.harford@broadcom.com>,
Rob Miller <rob.miller@broadcom.com>
Subject: [RFC v2 00/13] vDPA software assisted live migration
Date: Mon, 15 Mar 2021 20:48:29 +0100 [thread overview]
Message-ID: <20210315194842.277740-1-eperezma@redhat.com> (raw)
This series enable shadow virtqueue for vhost-net devices. This is a
new method of vhost devices migration: Instead of relay on vhost
device's dirty logging capability, SW assisted LM intercepts dataplane,
forwarding the descriptors between VM and device. Is intended for vDPA
devices with no logging, but this address the basic platform to build
that support on.
In this migration mode, qemu offers a new vring to the device to
read and write into, and disable vhost notifiers, processing guest and
vhost notifications in qemu. On used buffer relay, qemu will mark the
dirty memory as with plain virtio-net devices. This way, devices does
not need to have dirty page logging capability.
This series is a POC doing SW LM for vhost-net devices, which already
have dirty page logging capabilities. For qemu to use shadow virtqueues
these vhost-net devices need to be instantiated:
* With IOMMU (iommu_platform=on,ats=on)
* Without event_idx (event_idx=off)
And shadow virtqueue needs to be enabled for them with QMP command
like:
{
"execute": "x-vhost-enable-shadow-vq",
"arguments": {
"name": "virtio-net",
"enable": false
}
}
Just the notification forwarding (with no descriptor relay) can be
achieved with patches 5 and 6, and starting SVQ. Previous commits
are cleanup ones and declaration of QMP command.
Commit 11 introduces the buffer forwarding. Previous one are for
preparations again, and laters are for enabling some obvious
optimizations.
It is based on the ideas of DPDK SW assisted LM, in the series of
DPDK's https://patchwork.dpdk.org/cover/48370/ . However, these does
not map the shadow vq in guest's VA, but in qemu's.
Comments are welcome! Especially on:
* Different/improved way of synchronization, particularly on the race
of masking.
TODO:
* Event, indirect, packed, and others features of virtio - Waiting for
confirmation of the big picture.
* vDPA devices: Developing solutions for tracking the available IOVA
space for all devices. Small POC available, skipping the get/set
status (since vDPA does not support it) and just allocating more and
more IOVA addresses in a hardcoded range available for the device.
* To sepparate buffers forwarding in its own AIO context, so we can
throw more threads to that task and we don't need to stop the main
event loop.
* IOMMU optimizations, so bacthing and bigger chunks of IOVA can be
sent to device.
* Automatic kick-in on live-migration.
* Proper documentation.
Thanks!
Changes from v1 RFC:
* Use QMP instead of migration to start SVQ mode.
* Only accepting IOMMU devices, closer behavior with target devices
(vDPA)
* Fix invalid masking/unmasking of vhost call fd.
* Use of proper methods for synchronization.
* No need to modify VirtIO device code, all of the changes are
contained in vhost code.
* Delete superfluous code.
* An intermediate RFC was sent with only the notifications forwarding
changes. It can be seen in
https://patchew.org/QEMU/20210129205415.876290-1-eperezma@redhat.com/
* v1 at
https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg05372.html
Eugenio Pérez (13):
virtio: Add virtio_queue_is_host_notifier_enabled
vhost: Save masked_notifier state
vhost: Add VhostShadowVirtqueue
vhost: Add x-vhost-enable-shadow-vq qmp
vhost: Route guest->host notification through shadow virtqueue
vhost: Route host->guest notification through shadow virtqueue
vhost: Avoid re-set masked notifier in shadow vq
virtio: Add vhost_shadow_vq_get_vring_addr
virtio: Add virtio_queue_full
vhost: add vhost_kernel_set_vring_enable
vhost: Shadow virtqueue buffers forwarding
vhost: Check for device VRING_USED_F_NO_NOTIFY at shadow virtqueue
kick
vhost: Use VRING_AVAIL_F_NO_INTERRUPT at device call on shadow
virtqueue
qapi/net.json | 22 ++
hw/virtio/vhost-shadow-virtqueue.h | 36 ++
include/hw/virtio/vhost.h | 6 +
include/hw/virtio/virtio.h | 3 +
hw/virtio/vhost-backend.c | 29 ++
hw/virtio/vhost-shadow-virtqueue.c | 551 +++++++++++++++++++++++++++++
hw/virtio/vhost.c | 283 +++++++++++++++
hw/virtio/virtio.c | 23 +-
hw/virtio/meson.build | 2 +-
9 files changed, 952 insertions(+), 3 deletions(-)
create mode 100644 hw/virtio/vhost-shadow-virtqueue.h
create mode 100644 hw/virtio/vhost-shadow-virtqueue.c
--
2.27.0
next reply other threads:[~2021-03-15 19:51 UTC|newest]
Thread overview: 61+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-15 19:48 Eugenio Pérez [this message]
2021-03-15 19:48 ` [RFC v2 01/13] virtio: Add virtio_queue_is_host_notifier_enabled Eugenio Pérez
2021-03-15 19:48 ` [RFC v2 02/13] vhost: Save masked_notifier state Eugenio Pérez
2021-03-15 19:48 ` [RFC v2 03/13] vhost: Add VhostShadowVirtqueue Eugenio Pérez
2021-03-15 19:48 ` [RFC v2 04/13] vhost: Add x-vhost-enable-shadow-vq qmp Eugenio Pérez
2021-03-16 13:37 ` Eric Blake
2021-03-16 13:37 ` Eric Blake
2021-03-15 19:48 ` [RFC v2 05/13] vhost: Route guest->host notification through shadow virtqueue Eugenio Pérez
2021-03-16 7:18 ` Jason Wang
2021-03-16 7:18 ` Jason Wang
2021-03-16 10:31 ` Eugenio Perez Martin
2021-03-17 2:05 ` Jason Wang
2021-03-17 2:05 ` Jason Wang
2021-03-17 16:47 ` Eugenio Perez Martin
2021-03-18 3:10 ` Jason Wang
2021-03-18 3:10 ` Jason Wang
2021-03-18 9:18 ` Eugenio Perez Martin
2021-03-18 9:29 ` Jason Wang
2021-03-18 9:29 ` Jason Wang
2021-03-18 10:48 ` Eugenio Perez Martin
2021-03-18 12:04 ` Eugenio Perez Martin
2021-03-19 6:55 ` Jason Wang
2021-03-19 6:55 ` Jason Wang
2021-03-15 19:48 ` [RFC v2 06/13] vhost: Route host->guest " Eugenio Pérez
2021-03-16 7:21 ` Jason Wang
2021-03-15 19:48 ` [RFC v2 07/13] vhost: Avoid re-set masked notifier in shadow vq Eugenio Pérez
2021-03-15 19:48 ` [RFC v2 08/13] virtio: Add vhost_shadow_vq_get_vring_addr Eugenio Pérez
2021-03-16 7:50 ` Jason Wang
2021-03-16 7:50 ` Jason Wang
2021-03-16 15:20 ` Eugenio Perez Martin
2021-05-17 17:39 ` Eugenio Perez Martin
2021-03-15 19:48 ` [RFC v2 09/13] virtio: Add virtio_queue_full Eugenio Pérez
2021-03-15 19:48 ` [RFC v2 10/13] vhost: add vhost_kernel_set_vring_enable Eugenio Pérez
2021-03-16 7:29 ` Jason Wang
2021-03-16 7:29 ` Jason Wang
2021-03-16 10:43 ` Eugenio Perez Martin
2021-03-17 2:25 ` Jason Wang
2021-03-17 2:25 ` Jason Wang
2021-03-15 19:48 ` [RFC v2 11/13] vhost: Shadow virtqueue buffers forwarding Eugenio Pérez
2021-03-16 8:15 ` Jason Wang
2021-03-16 8:15 ` Jason Wang
2021-03-16 16:05 ` Eugenio Perez Martin
2021-03-17 2:50 ` Jason Wang
2021-03-17 2:50 ` Jason Wang
2021-03-17 14:38 ` Eugenio Perez Martin
2021-03-18 3:14 ` Jason Wang
2021-03-18 3:14 ` Jason Wang
2021-03-18 8:06 ` Eugenio Perez Martin
2021-03-18 9:16 ` Jason Wang
2021-03-18 9:54 ` Eugenio Perez Martin
2021-03-15 19:48 ` [RFC v2 12/13] vhost: Check for device VRING_USED_F_NO_NOTIFY at shadow virtqueue kick Eugenio Pérez
2021-03-16 8:07 ` Jason Wang
2021-03-16 8:07 ` Jason Wang
2021-05-17 17:11 ` Eugenio Perez Martin
2021-03-15 19:48 ` [RFC v2 13/13] vhost: Use VRING_AVAIL_F_NO_INTERRUPT at device call on shadow virtqueue Eugenio Pérez
2021-03-16 8:08 ` Jason Wang
2021-03-16 8:08 ` Jason Wang
2021-05-17 17:32 ` Eugenio Perez Martin
2021-03-16 8:28 ` [RFC v2 00/13] vDPA software assisted live migration Jason Wang
2021-03-16 8:28 ` Jason Wang
2021-03-16 17:25 ` Eugenio Perez Martin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210315194842.277740-1-eperezma@redhat.com \
--to=eperezma@redhat.com \
--cc=armbru@redhat.com \
--cc=eli@mellanox.com \
--cc=guru.prasad@broadcom.com \
--cc=hanand@xilinx.com \
--cc=jasowang@redhat.com \
--cc=jim.harford@broadcom.com \
--cc=ml@napatech.com \
--cc=mst@redhat.com \
--cc=parav@mellanox.com \
--cc=qemu-devel@nongnu.org \
--cc=quintela@redhat.com \
--cc=rob.miller@broadcom.com \
--cc=sgarzare@redhat.com \
--cc=virtualization@lists.linux-foundation.org \
--cc=xiao.w.wang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.