[PATCH 0/7] Guest announce feature emulation using Shadow VirtQueue

[PATCH 0/7] Guest announce feature emulation using Shadow VirtQueue

[Eugenio Pérez]

A gratuitous ARP is recommended after a live migration to reduce the amount of
time needed by the network links to be aware of the new location. A hypervisor
may not have the knowledge of the guest network configuration, and this is
especially true on passthrough devices, so its simpler to ask the guest to
do it.

However, the device control part of this feature can be totally emulated by
qemu and shadow virtqueue, not needing any special feature from the actual
vdpa device.

VIRTIO_NET_F_STATUS is also needed for the guest to access the status of
virtio net config where announcement status bit is set. Emulating it as
always active in case backend does not support it.

This series need two fixes applied:
* https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03242.html
* https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03241.html

v1:
* Move code from vhost_net_get_config to virtio_net_get_config.

RFC v2:
* Add VIRTIO_NET_F_STATUS emulation.

Eugenio Pérez (7):
  vdpa: Save emulated features list in vhost_vdpa
  virtio_net: Modify virtio_net_get_config to early return
  virtio_net: Handle _F_STATUS emulation in virtio_net_get_config
  vdpa: Expose VIRTIO_NET_F_STATUS unconditionally
  vdpa: handle VIRTIO_NET_CTRL_ANNOUNCE in
    vhost_vdpa_net_handle_ctrl_avail
  virtio_net: copy VIRTIO_NET_S_ANNOUNCE if device model has it
  vdpa: Offer VIRTIO_NET_F_GUEST_ANNOUNCE feature if SVQ is enabled

 include/hw/virtio/vhost-vdpa.h |  2 ++
 hw/net/virtio-net.c            | 40 +++++++++++++++++++++-------------
 hw/virtio/vhost-vdpa.c         |  8 +++----
 net/vhost-vdpa.c               | 25 ++++++++++++++++++---
 4 files changed, 53 insertions(+), 22 deletions(-)

-- 
2.31.1

[PATCH 1/7] vdpa: Save emulated features list in vhost_vdpa

[Eugenio Pérez]

At this moment only _F_LOG is added there.

However future patches add features that depend on the kind of device.
In particular, only net devices can add VIRTIO_F_GUEST_ANNOUNCE. So
let's allow vhost_vdpa creator to set custom emulated device features.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 include/hw/virtio/vhost-vdpa.h | 2 ++
 hw/virtio/vhost-vdpa.c         | 8 ++++----
 net/vhost-vdpa.c               | 4 ++++
 3 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 1111d85643..50083e1e3b 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -31,6 +31,8 @@ typedef struct vhost_vdpa {
     bool iotlb_batch_begin_sent;
     MemoryListener listener;
     struct vhost_vdpa_iova_range iova_range;
+    /* VirtIO device features that can be emulated by qemu */
+    uint64_t added_features;
     uint64_t acked_features;
     bool shadow_vqs_enabled;
     /* IOVA mapping used by the Shadow Virtqueue */
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 7468e44b87..ddb5e29288 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -660,8 +660,8 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev,
 
         v->acked_features = features;
 
-        /* We must not ack _F_LOG if SVQ is enabled */
-        features &= ~BIT_ULL(VHOST_F_LOG_ALL);
+        /* Do not ack features emulated by qemu */
+        features &= ~v->added_features;
     }
 
     trace_vhost_vdpa_set_features(dev, features);
@@ -1244,8 +1244,8 @@ static int vhost_vdpa_get_features(struct vhost_dev *dev,
     int ret = vhost_vdpa_get_dev_features(dev, features);
 
     if (ret == 0 && v->shadow_vqs_enabled) {
-        /* Add SVQ logging capabilities */
-        *features |= BIT_ULL(VHOST_F_LOG_ALL);
+        /* Add emulated capabilities */
+        *features |= v->added_features;
     }
 
     return ret;
diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 6d64000202..e7db40b7cd 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -551,6 +551,10 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
     s->vhost_vdpa.index = queue_pair_index;
     s->vhost_vdpa.shadow_vqs_enabled = svq;
     s->vhost_vdpa.iova_tree = iova_tree;
+    if (svq) {
+        /* Add SVQ logging capabilities */
+        s->vhost_vdpa.added_features |= BIT_ULL(VHOST_F_LOG_ALL);
+    }
     if (!is_datapath) {
         s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
                                             vhost_vdpa_net_cvq_cmd_page_len());
-- 
2.31.1

[PATCH 2/7] virtio_net: Modify virtio_net_get_config to early return

[Eugenio Pérez]

Next patches introduce more code on vhost-vdpa branch, with already have
too much indentation.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 hw/net/virtio-net.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index e9f696b4cf..56ff219196 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -158,20 +158,22 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
     if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
         ret = vhost_net_get_config(get_vhost_net(nc->peer), (uint8_t *)&netcfg,
                                    n->config_size);
-        if (ret != -1) {
-            /*
-             * Some NIC/kernel combinations present 0 as the mac address.  As
-             * that is not a legal address, try to proceed with the
-             * address from the QEMU command line in the hope that the
-             * address has been configured correctly elsewhere - just not
-             * reported by the device.
-             */
-            if (memcmp(&netcfg.mac, &zero, sizeof(zero)) == 0) {
-                info_report("Zero hardware mac address detected. Ignoring.");
-                memcpy(netcfg.mac, n->mac, ETH_ALEN);
-            }
-            memcpy(config, &netcfg, n->config_size);
+        if (ret == -1) {
+            return;
         }
+
+        /*
+         * Some NIC/kernel combinations present 0 as the mac address.  As that
+         * is not a legal address, try to proceed with the address from the
+         * QEMU command line in the hope that the address has been configured
+         * correctly elsewhere - just not reported by the device.
+         */
+        if (memcmp(&netcfg.mac, &zero, sizeof(zero)) == 0) {
+            info_report("Zero hardware mac address detected. Ignoring.");
+            memcpy(netcfg.mac, n->mac, ETH_ALEN);
+        }
+
+        memcpy(config, &netcfg, n->config_size);
     }
 }
 
-- 
2.31.1

[PATCH 3/7] virtio_net: Handle _F_STATUS emulation in virtio_net_get_config

[Eugenio Pérez]

At this moment this code path is not reached, but vdpa devices can offer
VIRTIO_NET_F_STATUS unconditionally. While the guest must assume that
link is always up by the standard, qemu will set the status bit to 1
always in this case.

This makes little use by itself, but VIRTIO_NET_F_STATUS is needed for
the guest to read status bit VIRTIO_NET_F_GUEST_ANNOUNCE, used by feature
VIRTIO_NET_F_GUEST_ANNOUNCE. So qemu must emulate status feature in case
it needs to emulate the guest announce feature.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 hw/net/virtio-net.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 56ff219196..51fe6d5c1a 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -156,8 +156,9 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
      * disconnect/reconnect a VDPA peer.
      */
     if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
-        ret = vhost_net_get_config(get_vhost_net(nc->peer), (uint8_t *)&netcfg,
-                                   n->config_size);
+        struct vhost_net *net = get_vhost_net(nc->peer);
+
+        ret = vhost_net_get_config(net, (uint8_t *)&netcfg, n->config_size);
         if (ret == -1) {
             return;
         }
@@ -173,6 +174,12 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
             memcpy(netcfg.mac, n->mac, ETH_ALEN);
         }
 
+        if (vdev->guest_features & BIT_ULL(VIRTIO_NET_F_STATUS) &&
+            !(net->dev.features & BIT_ULL(VIRTIO_NET_F_STATUS))) {
+            /* Emulating link up in qemu */
+            netcfg.status |= VIRTIO_NET_S_LINK_UP;
+        }
+
         memcpy(config, &netcfg, n->config_size);
     }
 }
-- 
2.31.1

[PATCH 4/7] vdpa: Expose VIRTIO_NET_F_STATUS unconditionally

[Eugenio Pérez]

Now that qemu can handle and emulate it if the vdpa backend does not
support it we can offer it always.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 net/vhost-vdpa.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index e7db40b7cd..3374c21b4d 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -554,6 +554,9 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
     if (svq) {
         /* Add SVQ logging capabilities */
         s->vhost_vdpa.added_features |= BIT_ULL(VHOST_F_LOG_ALL);
+
+        /* VIRTIO_NET_F_STATUS is mandatory for _F_GUEST_ANNOUNCE. */
+        s->vhost_vdpa.added_features |= BIT_ULL(VIRTIO_NET_F_STATUS);
     }
     if (!is_datapath) {
         s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
-- 
2.31.1

[PATCH 5/7] vdpa: handle VIRTIO_NET_CTRL_ANNOUNCE in vhost_vdpa_net_handle_ctrl_avail

[Eugenio Pérez]

Since this capability is emulated by qemu shadowed CVQ cannot forward it
to the device. Process all that command within qemu.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 net/vhost-vdpa.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 3374c21b4d..5fda405a66 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -488,9 +488,18 @@ static int vhost_vdpa_net_handle_ctrl_avail(VhostShadowVirtqueue *svq,
     out.iov_len = iov_to_buf(elem->out_sg, elem->out_num, 0,
                              s->cvq_cmd_out_buffer,
                              vhost_vdpa_net_cvq_cmd_len());
-    dev_written = vhost_vdpa_net_cvq_add(s, out.iov_len, sizeof(status));
-    if (unlikely(dev_written < 0)) {
-        goto out;
+    if (*(uint8_t *)s->cvq_cmd_out_buffer == VIRTIO_NET_CTRL_ANNOUNCE) {
+        /*
+         * Guest announce capability is emulated by qemu, so dont forward to
+         * the device.
+         */
+        dev_written = sizeof(status);
+        *s->status = VIRTIO_NET_OK;
+    } else {
+        dev_written = vhost_vdpa_net_cvq_add(s, out.iov_len, sizeof(status));
+        if (unlikely(dev_written < 0)) {
+            goto out;
+        }
     }
 
     if (unlikely(dev_written < sizeof(status))) {
-- 
2.31.1

[PATCH 6/7] virtio_net: copy VIRTIO_NET_S_ANNOUNCE if device model has it

[Eugenio Pérez]

Status part of the emulated feature. It will follow device model, so we
must copy it as long as NIC device model has it set.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 hw/net/virtio-net.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 51fe6d5c1a..49674dcf32 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -180,6 +180,7 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
             netcfg.status |= VIRTIO_NET_S_LINK_UP;
         }
 
+        netcfg.status |= (n->status & VIRTIO_NET_S_ANNOUNCE);
         memcpy(config, &netcfg, n->config_size);
     }
 }
-- 
2.31.1

[PATCH 7/7] vdpa: Offer VIRTIO_NET_F_GUEST_ANNOUNCE feature if SVQ is enabled

[Eugenio Pérez]

So qemu emulates it in case the device does not support it.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 net/vhost-vdpa.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 5fda405a66..64442e8455 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -566,6 +566,9 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
 
         /* VIRTIO_NET_F_STATUS is mandatory for _F_GUEST_ANNOUNCE. */
         s->vhost_vdpa.added_features |= BIT_ULL(VIRTIO_NET_F_STATUS);
+
+        /* We can emulate guest announce shadowing CVQ */
+        s->vhost_vdpa.added_features |= BIT_ULL(VIRTIO_NET_F_GUEST_ANNOUNCE);
     }
     if (!is_datapath) {
         s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
-- 
2.31.1

Re: [PATCH 0/7] Guest announce feature emulation using Shadow VirtQueue

[Michael S. Tsirkin]

On Thu, Oct 20, 2022 at 12:34:22PM +0200, Eugenio Pérez wrote:
> A gratuitous ARP is recommended after a live migration to reduce the amount of
> time needed by the network links to be aware of the new location. A hypervisor
> may not have the knowledge of the guest network configuration, and this is
> especially true on passthrough devices, so its simpler to ask the guest to
> do it.
> 
> However, the device control part of this feature can be totally emulated by
> qemu and shadow virtqueue, not needing any special feature from the actual
> vdpa device.
> 
> VIRTIO_NET_F_STATUS is also needed for the guest to access the status of
> virtio net config where announcement status bit is set. Emulating it as
> always active in case backend does not support it.
> 
> This series need two fixes applied:
> * https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03242.html
> * https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg03241.html


Jason you seem to have queued these so you will have to take
this patchset too.

Reviewed-by: Michael S. Tsirkin <mst@redhat.com>



> v1:
> * Move code from vhost_net_get_config to virtio_net_get_config.
> 
> RFC v2:
> * Add VIRTIO_NET_F_STATUS emulation.
> 
> Eugenio Pérez (7):
>   vdpa: Save emulated features list in vhost_vdpa
>   virtio_net: Modify virtio_net_get_config to early return
>   virtio_net: Handle _F_STATUS emulation in virtio_net_get_config
>   vdpa: Expose VIRTIO_NET_F_STATUS unconditionally
>   vdpa: handle VIRTIO_NET_CTRL_ANNOUNCE in
>     vhost_vdpa_net_handle_ctrl_avail
>   virtio_net: copy VIRTIO_NET_S_ANNOUNCE if device model has it
>   vdpa: Offer VIRTIO_NET_F_GUEST_ANNOUNCE feature if SVQ is enabled
> 
>  include/hw/virtio/vhost-vdpa.h |  2 ++
>  hw/net/virtio-net.c            | 40 +++++++++++++++++++++-------------
>  hw/virtio/vhost-vdpa.c         |  8 +++----
>  net/vhost-vdpa.c               | 25 ++++++++++++++++++---
>  4 files changed, 53 insertions(+), 22 deletions(-)
> 
> -- 
> 2.31.1
>