* [PATCH v2 00/12] vDPA support in qemu
@ 2020-06-30 17:49 Cindy Lu
2020-06-30 17:49 ` [PATCH v2 01/12] net: introduce qemu_get_peer Cindy Lu
` (12 more replies)
0 siblings, 13 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
aadam, rdunlap, maxime.coquelin, lingshan.zhu
vDPA device is a device that uses a datapath which complies with the
virtio specifications with vendor specific control path. vDPA devices
can be both physically located on the hardware or emulated by software.
This PATCH introduce the vDPA support in qemu
TODO
1) vIOMMU support
2) live migration support
3) docs for vhost-vdpa
4) config interrupt support
change from v1
separate the patch of vhost_vq_get_addr
separate the patch of vhost_dev_start
introduce the docmation for vhost-vdpa.rst
other comments form last version
github address
https://github.com/lulu-github-name/qemutmp.git PATCHV2
Cindy Lu (9):
net: introduce qemu_get_peer
vhost_net: use the function qemu_get_peer
vhost: introduce new VhostOps vhost_dev_start
vhost: implement vhost_dev_start method
vhost: introduce new VhostOps vhost_vq_get_addr
vhost: implement vhost_vq_get_addr method
vhost: introduce new VhostOps vhost_get_device_id
vhost-vdpa: introduce vhost-vdpa backend
vhost-vdpa: introduce vhost-vdpa net client
Jason Wang (3):
virtio-bus: introduce queue_enabled method
virtio-pci: implement queue_enabled method
vhost: check the existence of vhost_set_iotlb_callback
configure | 21 ++
docs/interop/index.rst | 1 +
docs/interop/vhost-vdpa.rst | 17 ++
hw/net/vhost_net.c | 35 ++-
hw/net/virtio-net.c | 22 +-
hw/virtio/Makefile.objs | 1 +
hw/virtio/vhost-backend.c | 7 +-
hw/virtio/vhost-vdpa.c | 469 ++++++++++++++++++++++++++++++
hw/virtio/vhost.c | 48 ++-
hw/virtio/virtio-pci.c | 13 +
hw/virtio/virtio.c | 6 +
include/hw/virtio/vhost-backend.h | 16 +-
include/hw/virtio/vhost-vdpa.h | 26 ++
include/hw/virtio/vhost.h | 7 +
include/hw/virtio/virtio-bus.h | 4 +
include/net/net.h | 1 +
include/net/vhost-vdpa.h | 22 ++
net/Makefile.objs | 2 +-
net/clients.h | 2 +
net/net.c | 10 +
net/vhost-vdpa.c | 228 +++++++++++++++
qapi/net.json | 28 +-
qemu-options.hx | 12 +
23 files changed, 966 insertions(+), 32 deletions(-)
create mode 100644 docs/interop/vhost-vdpa.rst
create mode 100644 hw/virtio/vhost-vdpa.c
create mode 100644 include/hw/virtio/vhost-vdpa.h
create mode 100644 include/net/vhost-vdpa.h
create mode 100644 net/vhost-vdpa.c
--
2.21.1
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v2 01/12] net: introduce qemu_get_peer
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 02/12] vhost_net: use the function qemu_get_peer Cindy Lu
` (11 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
aadam, rdunlap, maxime.coquelin, lingshan.zhu
This is a small function that can get the peer
from given NetClientState and queue_index
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
include/net/net.h | 1 +
net/net.c | 7 +++++++
2 files changed, 8 insertions(+)
diff --git a/include/net/net.h b/include/net/net.h
index 39085d9444..e7ef42d62b 100644
--- a/include/net/net.h
+++ b/include/net/net.h
@@ -176,6 +176,7 @@ void hmp_info_network(Monitor *mon, const QDict *qdict);
void net_socket_rs_init(SocketReadState *rs,
SocketReadStateFinalize *finalize,
bool vnet_hdr);
+NetClientState *qemu_get_peer(NetClientState *nc, int queue_index);
/* NIC info */
diff --git a/net/net.c b/net/net.c
index d1130296e1..9099a327dd 100644
--- a/net/net.c
+++ b/net/net.c
@@ -325,6 +325,13 @@ void *qemu_get_nic_opaque(NetClientState *nc)
return nic->opaque;
}
+NetClientState *qemu_get_peer(NetClientState *nc, int queue_index)
+{
+ assert(nc != NULL);
+ NetClientState *ncs = nc + queue_index;
+ return ncs->peer;
+}
+
static void qemu_cleanup_net_client(NetClientState *nc)
{
QTAILQ_REMOVE(&net_clients, nc, next);
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 02/12] vhost_net: use the function qemu_get_peer
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
2020-06-30 17:49 ` [PATCH v2 01/12] net: introduce qemu_get_peer Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 03/12] virtio-bus: introduce queue_enabled method Cindy Lu
` (10 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, Laurent Vivier, shahafs, kevin.tian, parav,
vmireyno, cunming.liang, gdawar, jiri, xiao.w.wang, stefanha,
zhihong.wang, aadam, rdunlap, maxime.coquelin, lingshan.zhu
user the qemu_get_peer to replace the old process
Signed-off-by: Cindy Lu <lulu@redhat.com>
Reviewed-by: Laurent Vivier <lvivier@redhat.com>
---
hw/net/vhost_net.c | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
index 6b82803fa7..4096d64aaf 100644
--- a/hw/net/vhost_net.c
+++ b/hw/net/vhost_net.c
@@ -306,7 +306,9 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs,
BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(dev)));
VirtioBusState *vbus = VIRTIO_BUS(qbus);
VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(vbus);
+ struct vhost_net *net;
int r, e, i;
+ NetClientState *peer;
if (!k->set_guest_notifiers) {
error_report("binding does not support guest notifiers");
@@ -314,9 +316,9 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs,
}
for (i = 0; i < total_queues; i++) {
- struct vhost_net *net;
- net = get_vhost_net(ncs[i].peer);
+ peer = qemu_get_peer(ncs, i);
+ net = get_vhost_net(peer);
vhost_net_set_vq_index(net, i * 2);
/* Suppress the masking guest notifiers on vhost user
@@ -335,15 +337,16 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs,
}
for (i = 0; i < total_queues; i++) {
- r = vhost_net_start_one(get_vhost_net(ncs[i].peer), dev);
+ peer = qemu_get_peer(ncs, i);
+ r = vhost_net_start_one(get_vhost_net(peer), dev);
if (r < 0) {
goto err_start;
}
- if (ncs[i].peer->vring_enable) {
+ if (peer->vring_enable) {
/* restore vring enable state */
- r = vhost_set_vring_enable(ncs[i].peer, ncs[i].peer->vring_enable);
+ r = vhost_set_vring_enable(peer, peer->vring_enable);
if (r < 0) {
goto err_start;
@@ -355,7 +358,8 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs,
err_start:
while (--i >= 0) {
- vhost_net_stop_one(get_vhost_net(ncs[i].peer), dev);
+ peer = qemu_get_peer(ncs , i);
+ vhost_net_stop_one(get_vhost_net(peer), dev);
}
e = k->set_guest_notifiers(qbus->parent, total_queues * 2, false);
if (e < 0) {
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 03/12] virtio-bus: introduce queue_enabled method
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
2020-06-30 17:49 ` [PATCH v2 01/12] net: introduce qemu_get_peer Cindy Lu
2020-06-30 17:49 ` [PATCH v2 02/12] vhost_net: use the function qemu_get_peer Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 04/12] virtio-pci: implement " Cindy Lu
` (9 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
aadam, rdunlap, maxime.coquelin, lingshan.zhu
From: Jason Wang <jasowang@redhat.com>
This patch introduces queue_enabled() method which allows the
transport to implement its own way to report whether or not a queue is
enabled.
Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
hw/virtio/virtio.c | 6 ++++++
include/hw/virtio/virtio-bus.h | 4 ++++
2 files changed, 10 insertions(+)
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index cc9c9dc162..5bd2a2f621 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -3286,6 +3286,12 @@ hwaddr virtio_queue_get_desc_addr(VirtIODevice *vdev, int n)
bool virtio_queue_enabled(VirtIODevice *vdev, int n)
{
+ BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
+ VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
+
+ if (k->queue_enabled) {
+ return k->queue_enabled(qbus->parent, n);
+ }
return virtio_queue_get_desc_addr(vdev, n) != 0;
}
diff --git a/include/hw/virtio/virtio-bus.h b/include/hw/virtio/virtio-bus.h
index 38c9399cd4..0f6f215925 100644
--- a/include/hw/virtio/virtio-bus.h
+++ b/include/hw/virtio/virtio-bus.h
@@ -83,6 +83,10 @@ typedef struct VirtioBusClass {
*/
int (*ioeventfd_assign)(DeviceState *d, EventNotifier *notifier,
int n, bool assign);
+ /*
+ * Whether queue number n is enabled.
+ */
+ bool (*queue_enabled)(DeviceState *d, int n);
/*
* Does the transport have variable vring alignment?
* (ie can it ever call virtio_queue_set_align()?)
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 04/12] virtio-pci: implement queue_enabled method
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
` (2 preceding siblings ...)
2020-06-30 17:49 ` [PATCH v2 03/12] virtio-bus: introduce queue_enabled method Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 05/12] vhost: check the existence of vhost_set_iotlb_callback Cindy Lu
` (8 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
aadam, rdunlap, maxime.coquelin, lingshan.zhu
From: Jason Wang <jasowang@redhat.com>
With version 1, we can detect whether a queue is enabled via
queue_enabled.
Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
hw/virtio/virtio-pci.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 7bc8c1c056..8554cf2a03 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -1107,6 +1107,18 @@ static AddressSpace *virtio_pci_get_dma_as(DeviceState *d)
return pci_get_address_space(dev);
}
+static bool virtio_pci_queue_enabled(DeviceState *d, int n)
+{
+ VirtIOPCIProxy *proxy = VIRTIO_PCI(d);
+ VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
+
+ if (virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1)) {
+ return proxy->vqs[vdev->queue_sel].enabled;
+ }
+
+ return virtio_queue_enabled(vdev, n);
+}
+
static int virtio_pci_add_mem_cap(VirtIOPCIProxy *proxy,
struct virtio_pci_cap *cap)
{
@@ -2064,6 +2076,7 @@ static void virtio_pci_bus_class_init(ObjectClass *klass, void *data)
k->ioeventfd_enabled = virtio_pci_ioeventfd_enabled;
k->ioeventfd_assign = virtio_pci_ioeventfd_assign;
k->get_dma_as = virtio_pci_get_dma_as;
+ k->queue_enabled = virtio_pci_queue_enabled;
}
static const TypeInfo virtio_pci_bus_info = {
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 05/12] vhost: check the existence of vhost_set_iotlb_callback
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
` (3 preceding siblings ...)
2020-06-30 17:49 ` [PATCH v2 04/12] virtio-pci: implement " Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 06/12] vhost: introduce new VhostOps vhost_dev_start Cindy Lu
` (7 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
aadam, rdunlap, maxime.coquelin, lingshan.zhu
From: Jason Wang <jasowang@redhat.com>
Add the check of vhost_set_iotlb_callback
before calling
Signed-off-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
hw/virtio/vhost.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 5fd25fe520..10304b583e 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1686,8 +1686,9 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
}
}
- if (vhost_dev_has_iommu(hdev)) {
- hdev->vhost_ops->vhost_set_iotlb_callback(hdev, true);
+ if (vhost_dev_has_iommu(hdev) &&
+ hdev->vhost_ops->vhost_set_iotlb_callback) {
+ hdev->vhost_ops->vhost_set_iotlb_callback(hdev, true);
/* Update used ring information for IOTLB to work correctly,
* vhost-kernel code requires for this.*/
@@ -1730,7 +1731,9 @@ void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev)
}
if (vhost_dev_has_iommu(hdev)) {
- hdev->vhost_ops->vhost_set_iotlb_callback(hdev, false);
+ if (hdev->vhost_ops->vhost_set_iotlb_callback) {
+ hdev->vhost_ops->vhost_set_iotlb_callback(hdev, false);
+ }
memory_listener_unregister(&hdev->iommu_listener);
}
vhost_log_put(hdev, true);
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 06/12] vhost: introduce new VhostOps vhost_dev_start
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
` (4 preceding siblings ...)
2020-06-30 17:49 ` [PATCH v2 05/12] vhost: check the existence of vhost_set_iotlb_callback Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 07/12] vhost: implement vhost_dev_start method Cindy Lu
` (6 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
aadam, rdunlap, maxime.coquelin, lingshan.zhu
This patch introduces new VhostOps vhost_dev_start callback which allows the
vhost_net set the start/stop status to backend
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
include/hw/virtio/vhost-backend.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
index 6f6670783f..b80f344cd6 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -112,6 +112,7 @@ typedef int (*vhost_get_inflight_fd_op)(struct vhost_dev *dev,
typedef int (*vhost_set_inflight_fd_op)(struct vhost_dev *dev,
struct vhost_inflight *inflight);
+typedef int (*vhost_dev_start_op)(struct vhost_dev *dev, bool started);
typedef struct VhostOps {
VhostBackendType backend_type;
vhost_backend_init vhost_backend_init;
@@ -152,6 +153,7 @@ typedef struct VhostOps {
vhost_backend_mem_section_filter_op vhost_backend_mem_section_filter;
vhost_get_inflight_fd_op vhost_get_inflight_fd;
vhost_set_inflight_fd_op vhost_set_inflight_fd;
+ vhost_dev_start_op vhost_dev_start;
} VhostOps;
extern const VhostOps user_ops;
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 07/12] vhost: implement vhost_dev_start method
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
` (5 preceding siblings ...)
2020-06-30 17:49 ` [PATCH v2 06/12] vhost: introduce new VhostOps vhost_dev_start Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 08/12] vhost: introduce new VhostOps vhost_vq_get_addr Cindy Lu
` (5 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
aadam, rdunlap, maxime.coquelin, lingshan.zhu
use the vhost_dev_start callback to send the status to backend
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
hw/virtio/vhost.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 10304b583e..32809e54b5 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1685,7 +1685,12 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
goto fail_log;
}
}
-
+ if (hdev->vhost_ops->vhost_dev_start) {
+ r = hdev->vhost_ops->vhost_dev_start(hdev, true);
+ if (r) {
+ goto fail_log;
+ }
+ }
if (vhost_dev_has_iommu(hdev) &&
hdev->vhost_ops->vhost_set_iotlb_callback) {
hdev->vhost_ops->vhost_set_iotlb_callback(hdev, true);
@@ -1723,6 +1728,9 @@ void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice *vdev)
/* should only be called after backend is connected */
assert(hdev->vhost_ops);
+ if (hdev->vhost_ops->vhost_dev_start) {
+ hdev->vhost_ops->vhost_dev_start(hdev, false);
+ }
for (i = 0; i < hdev->nvqs; ++i) {
vhost_virtqueue_stop(hdev,
vdev,
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 08/12] vhost: introduce new VhostOps vhost_vq_get_addr
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
` (6 preceding siblings ...)
2020-06-30 17:49 ` [PATCH v2 07/12] vhost: implement vhost_dev_start method Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 09/12] vhost: implement vhost_vq_get_addr method Cindy Lu
` (4 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
aadam, rdunlap, maxime.coquelin, lingshan.zhu
This patch introduces new VhostOps vhost_vq_get_addr_op callback to get
the vring addr from the backend
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
include/hw/virtio/vhost-backend.h | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
index b80f344cd6..fa84abac97 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -34,6 +34,7 @@ struct vhost_vring_state;
struct vhost_vring_addr;
struct vhost_scsi_target;
struct vhost_iotlb_msg;
+struct vhost_virtqueue;
typedef int (*vhost_backend_init)(struct vhost_dev *dev, void *opaque);
typedef int (*vhost_backend_cleanup)(struct vhost_dev *dev);
@@ -113,6 +114,10 @@ typedef int (*vhost_set_inflight_fd_op)(struct vhost_dev *dev,
struct vhost_inflight *inflight);
typedef int (*vhost_dev_start_op)(struct vhost_dev *dev, bool started);
+
+typedef int (*vhost_vq_get_addr_op)(struct vhost_dev *dev,
+ struct vhost_vring_addr *addr,
+ struct vhost_virtqueue *vq);
typedef struct VhostOps {
VhostBackendType backend_type;
vhost_backend_init vhost_backend_init;
@@ -154,6 +159,7 @@ typedef struct VhostOps {
vhost_get_inflight_fd_op vhost_get_inflight_fd;
vhost_set_inflight_fd_op vhost_set_inflight_fd;
vhost_dev_start_op vhost_dev_start;
+ vhost_vq_get_addr_op vhost_vq_get_addr;
} VhostOps;
extern const VhostOps user_ops;
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 09/12] vhost: implement vhost_vq_get_addr method
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
` (7 preceding siblings ...)
2020-06-30 17:49 ` [PATCH v2 08/12] vhost: introduce new VhostOps vhost_vq_get_addr Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 10/12] vhost: introduce new VhostOps vhost_get_device_id Cindy Lu
` (3 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
aadam, rdunlap, maxime.coquelin, lingshan.zhu
use vhost_vq_get_addr callback to get the vq address from backend
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
hw/virtio/vhost.c | 28 +++++++++++++++++++---------
1 file changed, 19 insertions(+), 9 deletions(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 32809e54b5..1e083a8976 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -773,15 +773,25 @@ static int vhost_virtqueue_set_addr(struct vhost_dev *dev,
struct vhost_virtqueue *vq,
unsigned idx, bool enable_log)
{
- struct vhost_vring_addr addr = {
- .index = idx,
- .desc_user_addr = (uint64_t)(unsigned long)vq->desc,
- .avail_user_addr = (uint64_t)(unsigned long)vq->avail,
- .used_user_addr = (uint64_t)(unsigned long)vq->used,
- .log_guest_addr = vq->used_phys,
- .flags = enable_log ? (1 << VHOST_VRING_F_LOG) : 0,
- };
- int r = dev->vhost_ops->vhost_set_vring_addr(dev, &addr);
+ struct vhost_vring_addr addr;
+ int r;
+ memset(&addr, 0, sizeof(struct vhost_vring_addr));
+
+ if (dev->vhost_ops->vhost_vq_get_addr) {
+ r = dev->vhost_ops->vhost_vq_get_addr(dev, &addr, vq);
+ if (r < 0) {
+ VHOST_OPS_DEBUG("vhost_vq_get_addr failed");
+ return -errno;
+ }
+ } else {
+ addr.desc_user_addr = (uint64_t)(unsigned long)vq->desc;
+ addr.avail_user_addr = (uint64_t)(unsigned long)vq->avail;
+ addr.used_user_addr = (uint64_t)(unsigned long)vq->used;
+ }
+ addr.index = idx;
+ addr.log_guest_addr = vq->used_phys;
+ addr.flags = enable_log ? (1 << VHOST_VRING_F_LOG) : 0;
+ r = dev->vhost_ops->vhost_set_vring_addr(dev, &addr);
if (r < 0) {
VHOST_OPS_DEBUG("vhost_set_vring_addr failed");
return -errno;
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 10/12] vhost: introduce new VhostOps vhost_get_device_id
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
` (8 preceding siblings ...)
2020-06-30 17:49 ` [PATCH v2 09/12] vhost: implement vhost_vq_get_addr method Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 11/12] vhost-vdpa: introduce vhost-vdpa backend Cindy Lu
` (2 subsequent siblings)
12 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
aadam, rdunlap, maxime.coquelin, lingshan.zhu
This patch introduces new VhostOps vhost_get_device_id callback which
can get the device id from backend
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
include/hw/virtio/vhost-backend.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
index fa84abac97..bfc24207e2 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -118,6 +118,9 @@ typedef int (*vhost_dev_start_op)(struct vhost_dev *dev, bool started);
typedef int (*vhost_vq_get_addr_op)(struct vhost_dev *dev,
struct vhost_vring_addr *addr,
struct vhost_virtqueue *vq);
+
+typedef int (*vhost_get_device_id_op)(struct vhost_dev *dev, uint32_t *dev_id);
+
typedef struct VhostOps {
VhostBackendType backend_type;
vhost_backend_init vhost_backend_init;
@@ -160,6 +163,7 @@ typedef struct VhostOps {
vhost_set_inflight_fd_op vhost_set_inflight_fd;
vhost_dev_start_op vhost_dev_start;
vhost_vq_get_addr_op vhost_vq_get_addr;
+ vhost_get_device_id_op vhost_get_device_id;
} VhostOps;
extern const VhostOps user_ops;
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 11/12] vhost-vdpa: introduce vhost-vdpa backend
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
` (9 preceding siblings ...)
2020-06-30 17:49 ` [PATCH v2 10/12] vhost: introduce new VhostOps vhost_get_device_id Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-07-01 4:21 ` Jason Wang
2020-06-30 17:49 ` [PATCH v2 12/12] vhost-vdpa: introduce vhost-vdpa net client Cindy Lu
2020-06-30 18:06 ` [PATCH v2 00/12] vDPA support in qemu no-reply
12 siblings, 1 reply; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
Tiwei Bie, aadam, rdunlap, maxime.coquelin, lingshan.zhu
Currently we have 2 types of vhost backends in QEMU: vhost kernel and
vhost-user. The above patch provides a generic device for vDPA purpose,
this vDPA device exposes to user space a non-vendor-specific configuration
interface for setting up a vhost HW accelerator, this patch set introduces
a third vhost backend called vhost-vdpa based on the vDPA interface.
Vhost-vdpa usage:
qemu-system-x86_64 -cpu host -enable-kvm \
......
-netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-id,id=vhost-vdpa0 \
-device virtio-net-pci,netdev=vhost-vdpa0,page-per-vq=on \
Signed-off-by: Lingshan zhu <lingshan.zhu@intel.com>
Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
configure | 21 ++
docs/interop/index.rst | 1 +
docs/interop/vhost-vdpa.rst | 17 ++
hw/net/vhost_net.c | 19 +-
hw/net/virtio-net.c | 22 +-
hw/virtio/Makefile.objs | 1 +
hw/virtio/vhost-backend.c | 7 +-
hw/virtio/vhost-vdpa.c | 469 ++++++++++++++++++++++++++++++
hw/virtio/vhost.c | 7 +-
include/hw/virtio/vhost-backend.h | 4 +-
include/hw/virtio/vhost-vdpa.h | 26 ++
include/hw/virtio/vhost.h | 7 +
qemu-options.hx | 12 +
13 files changed, 601 insertions(+), 12 deletions(-)
create mode 100644 docs/interop/vhost-vdpa.rst
create mode 100644 hw/virtio/vhost-vdpa.c
create mode 100644 include/hw/virtio/vhost-vdpa.h
diff --git a/configure b/configure
index 4a22dcd563..3db7f20185 100755
--- a/configure
+++ b/configure
@@ -1575,6 +1575,10 @@ for opt do
;;
--enable-vhost-user) vhost_user="yes"
;;
+ --disable-vhost-vdpa) vhost_vdpa="no"
+ ;;
+ --enable-vhost-vdpa) vhost_vdpa="yes"
+ ;;
--disable-vhost-kernel) vhost_kernel="no"
;;
--enable-vhost-kernel) vhost_kernel="yes"
@@ -1883,6 +1887,7 @@ disabled with --disable-FEATURE, default is enabled if available:
vhost-crypto vhost-user-crypto backend support
vhost-kernel vhost kernel backend support
vhost-user vhost-user backend support
+ vhost-vdpa vhost-vdpa kernel backend support
spice spice
rbd rados block device (rbd)
libiscsi iscsi support
@@ -2394,6 +2399,10 @@ test "$vhost_user" = "" && vhost_user=yes
if test "$vhost_user" = "yes" && test "$mingw32" = "yes"; then
error_exit "vhost-user isn't available on win32"
fi
+test "$vhost_vdpa" = "" && vhost_vdpa=$linux
+if test "$vhost_vdpa" = "yes" && test "$linux" != "yes"; then
+ error_exit "vhost-vdpa is only available on Linux"
+fi
test "$vhost_kernel" = "" && vhost_kernel=$linux
if test "$vhost_kernel" = "yes" && test "$linux" != "yes"; then
error_exit "vhost-kernel is only available on Linux"
@@ -2422,6 +2431,11 @@ test "$vhost_user_fs" = "" && vhost_user_fs=$vhost_user
if test "$vhost_user_fs" = "yes" && test "$vhost_user" = "no"; then
error_exit "--enable-vhost-user-fs requires --enable-vhost-user"
fi
+#vhost-vdpa backends
+test "$vhost_net_vdpa" = "" && vhost_net_vdpa=$vhost_vdpa
+if test "$vhost_net_vdpa" = "yes" && test "$vhost_vdpa" = "no"; then
+ error_exit "--enable-vhost-net-vdpa requires --enable-vhost-vdpa"
+fi
# OR the vhost-kernel and vhost-user values for simplicity
if test "$vhost_net" = ""; then
@@ -6936,6 +6950,7 @@ echo "vhost-scsi support $vhost_scsi"
echo "vhost-vsock support $vhost_vsock"
echo "vhost-user support $vhost_user"
echo "vhost-user-fs support $vhost_user_fs"
+echo "vhost-vdpa support $vhost_vdpa"
echo "Trace backends $trace_backends"
if have_backend "simple"; then
echo "Trace output file $trace_file-<pid>"
@@ -7437,6 +7452,9 @@ fi
if test "$vhost_net_user" = "yes" ; then
echo "CONFIG_VHOST_NET_USER=y" >> $config_host_mak
fi
+if test "$vhost_net_vdpa" = "yes" ; then
+ echo "CONFIG_VHOST_NET_VDPA=y" >> $config_host_mak
+fi
if test "$vhost_crypto" = "yes" ; then
echo "CONFIG_VHOST_CRYPTO=y" >> $config_host_mak
fi
@@ -7452,6 +7470,9 @@ fi
if test "$vhost_user" = "yes" ; then
echo "CONFIG_VHOST_USER=y" >> $config_host_mak
fi
+if test "$vhost_vdpa" = "yes" ; then
+ echo "CONFIG_VHOST_VDPA=y" >> $config_host_mak
+fi
if test "$vhost_user_fs" = "yes" ; then
echo "CONFIG_VHOST_USER_FS=y" >> $config_host_mak
fi
diff --git a/docs/interop/index.rst b/docs/interop/index.rst
index 049387ac6d..006f986420 100644
--- a/docs/interop/index.rst
+++ b/docs/interop/index.rst
@@ -20,3 +20,4 @@ Contents:
qemu-ga
vhost-user
vhost-user-gpu
+ vhost-vdpa
diff --git a/docs/interop/vhost-vdpa.rst b/docs/interop/vhost-vdpa.rst
new file mode 100644
index 0000000000..5a075b8a63
--- /dev/null
+++ b/docs/interop/vhost-vdpa.rst
@@ -0,0 +1,17 @@
+=====================
+Vhost-vdpa Protocol
+=====================
+
+Introduction
+=============
+vDPA(Virtual data path acceleration) device is a device that uses
+a datapath which complies with the virtio specifications with vendor
+specific control path. vDPA devices can be both physically located on
+the hardware or emulated by software.
+
+This document describes the vDPA support in qemu
+
+Here is the kernel commit here
+https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4c8cf31885f69e86be0b5b9e6677a26797365e1d
+
+TODO : Moreinfo will add later
diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
index 4096d64aaf..cbdf42ace5 100644
--- a/hw/net/vhost_net.c
+++ b/hw/net/vhost_net.c
@@ -17,8 +17,10 @@
#include "net/net.h"
#include "net/tap.h"
#include "net/vhost-user.h"
+#include "net/vhost-vdpa.h"
#include "standard-headers/linux/vhost_types.h"
+#include "linux-headers/linux/vhost.h"
#include "hw/virtio/virtio-net.h"
#include "net/vhost_net.h"
#include "qemu/error-report.h"
@@ -33,12 +35,6 @@
#include "hw/virtio/vhost.h"
#include "hw/virtio/virtio-bus.h"
-struct vhost_net {
- struct vhost_dev dev;
- struct vhost_virtqueue vqs[2];
- int backend;
- NetClientState *nc;
-};
/* Features supported by host kernel. */
static const int kernel_feature_bits[] = {
@@ -96,6 +92,9 @@ static const int *vhost_net_get_feature_bits(struct vhost_net *net)
case NET_CLIENT_DRIVER_VHOST_USER:
feature_bits = user_feature_bits;
break;
+ case NET_CLIENT_DRIVER_VHOST_VDPA:
+ feature_bits = vdpa_feature_bits;
+ break;
default:
error_report("Feature bits not defined for this type: %d",
net->nc->info->type);
@@ -316,7 +315,6 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs,
}
for (i = 0; i < total_queues; i++) {
-
peer = qemu_get_peer(ncs, i);
net = get_vhost_net(peer);
vhost_net_set_vq_index(net, i * 2);
@@ -337,6 +335,7 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs,
}
for (i = 0; i < total_queues; i++) {
+
peer = qemu_get_peer(ncs, i);
r = vhost_net_start_one(get_vhost_net(peer), dev);
@@ -433,6 +432,12 @@ VHostNetState *get_vhost_net(NetClientState *nc)
vhost_net = vhost_user_get_vhost_net(nc);
assert(vhost_net);
break;
+#endif
+#ifdef CONFIG_VHOST_NET_VDPA
+ case NET_CLIENT_DRIVER_VHOST_VDPA:
+ vhost_net = vhost_vdpa_get_vhost_net(nc);
+ assert(vhost_net);
+ break;
#endif
default:
break;
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index aff67a92df..6aadc2403a 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -43,6 +43,7 @@
#include "monitor/qdev.h"
#include "hw/pci/pci.h"
#include "net_rx_pkt.h"
+#include "hw/virtio/vhost.h"
#define VIRTIO_NET_VM_VERSION 11
@@ -124,7 +125,10 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
{
VirtIONet *n = VIRTIO_NET(vdev);
struct virtio_net_config netcfg;
+ int ret = 0;
+ struct vhost_dev *hdev = NULL;
+ memset(&netcfg, 0 , sizeof(struct virtio_net_config));
virtio_stw_p(vdev, &netcfg.status, n->status);
virtio_stw_p(vdev, &netcfg.max_virtqueue_pairs, n->max_queues);
virtio_stw_p(vdev, &netcfg.mtu, n->net_conf.mtu);
@@ -138,21 +142,37 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
virtio_stl_p(vdev, &netcfg.supported_hash_types,
VIRTIO_NET_RSS_SUPPORTED_HASHES);
memcpy(config, &netcfg, n->config_size);
+
+ NetClientState *nc = qemu_get_queue(n->nic);
+ if (nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
+ hdev = &(get_vhost_net(nc->peer)->dev);
+ ret = vhost_dev_get_config(hdev, (uint8_t *)&netcfg, n->config_size);
+ if (ret != -1) {
+ memcpy(config, &netcfg, n->config_size);
+ }
+ }
}
static void virtio_net_set_config(VirtIODevice *vdev, const uint8_t *config)
{
VirtIONet *n = VIRTIO_NET(vdev);
struct virtio_net_config netcfg = {};
+ struct vhost_dev *hdev = NULL;
memcpy(&netcfg, config, n->config_size);
-
if (!virtio_vdev_has_feature(vdev, VIRTIO_NET_F_CTRL_MAC_ADDR) &&
!virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1) &&
memcmp(netcfg.mac, n->mac, ETH_ALEN)) {
memcpy(n->mac, netcfg.mac, ETH_ALEN);
qemu_format_nic_info_str(qemu_get_queue(n->nic), n->mac);
}
+
+ NetClientState *nc = qemu_get_queue(n->nic);
+ if (nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
+ hdev = &(get_vhost_net(nc->peer)->dev);
+ vhost_dev_set_config(hdev, (uint8_t *)&netcfg, 0, n->config_size,
+ VHOST_SET_CONFIG_TYPE_MASTER);
+ }
}
static bool virtio_net_started(VirtIONet *n, uint8_t status)
diff --git a/hw/virtio/Makefile.objs b/hw/virtio/Makefile.objs
index 13e75f171f..2b31e9b3dc 100644
--- a/hw/virtio/Makefile.objs
+++ b/hw/virtio/Makefile.objs
@@ -5,6 +5,7 @@ obj-y += virtio.o
obj-$(CONFIG_VHOST) += vhost.o vhost-backend.o
common-obj-$(call lnot,$(CONFIG_VHOST)) += vhost-stub.o
obj-$(CONFIG_VHOST_USER) += vhost-user.o
+obj-$(CONFIG_VHOST_VDPA) += vhost-vdpa.o
common-obj-$(CONFIG_VIRTIO_RNG) += virtio-rng.o
common-obj-$(CONFIG_VIRTIO_PCI) += virtio-pci.o
diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c
index 48905383f8..b925a2b1ac 100644
--- a/hw/virtio/vhost-backend.c
+++ b/hw/virtio/vhost-backend.c
@@ -14,7 +14,7 @@
#include "qemu/error-report.h"
#include "qemu/main-loop.h"
#include "standard-headers/linux/vhost_types.h"
-
+#include "hw/virtio/vhost-vdpa.h"
#ifdef CONFIG_VHOST_KERNEL
#include <linux/vhost.h>
#include <sys/ioctl.h>
@@ -285,6 +285,11 @@ int vhost_set_backend_type(struct vhost_dev *dev, VhostBackendType backend_type)
case VHOST_BACKEND_TYPE_USER:
dev->vhost_ops = &user_ops;
break;
+#endif
+#ifdef CONFIG_VHOST_VDPA
+ case VHOST_BACKEND_TYPE_VDPA:
+ dev->vhost_ops = &vdpa_ops;
+ break;
#endif
default:
error_report("Unknown vhost backend type");
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
new file mode 100644
index 0000000000..25c9b537b0
--- /dev/null
+++ b/hw/virtio/vhost-vdpa.c
@@ -0,0 +1,469 @@
+/*
+ * vhost-vdpa
+ *
+ * Copyright(c) 2017-2018 Intel Corporation.
+ * Copyright(c) 2020 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include <linux/vhost.h>
+#include <linux/vfio.h>
+#include <sys/eventfd.h>
+#include <sys/ioctl.h>
+#include "hw/virtio/vhost.h"
+#include "hw/virtio/vhost-backend.h"
+#include "hw/virtio/virtio-net.h"
+#include "hw/virtio/vhost-vdpa.h"
+#include "qemu/main-loop.h"
+#include <linux/kvm.h>
+#include "sysemu/kvm.h"
+
+static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section)
+{
+ return (!memory_region_is_ram(section->mr) &&
+ !memory_region_is_iommu(section->mr)) ||
+ /*
+ * Sizing an enabled 64-bit BAR can cause spurious mappings to
+ * addresses in the upper part of the 64-bit address space. These
+ * are never accessed by the CPU and beyond the address width of
+ * some IOMMU hardware. TODO: VDPA should tell us the IOMMU width.
+ */
+ section->offset_within_address_space & (1ULL << 63);
+}
+
+static int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
+ void *vaddr, bool readonly)
+{
+ struct vhost_msg_v2 msg;
+ int fd = v->device_fd;
+ int ret = 0;
+
+ msg.type = v->msg_type;
+ msg.iotlb.iova = iova;
+ msg.iotlb.size = size;
+ msg.iotlb.uaddr = (uint64_t)vaddr;
+ msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
+ msg.iotlb.type = VHOST_IOTLB_UPDATE;
+
+ if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
+ error_report("failed to write, fd=%d, errno=%d (%s)",
+ fd, errno, strerror(errno));
+ return -EIO ;
+ }
+
+ return ret;
+}
+
+static int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova,
+ hwaddr size)
+{
+ struct vhost_msg_v2 msg;
+ int fd = v->device_fd;
+ int ret = 0;
+
+ msg.type = v->msg_type;
+ msg.iotlb.iova = iova;
+ msg.iotlb.size = size;
+ msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
+
+ if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
+ error_report("failed to write, fd=%d, errno=%d (%s)",
+ fd, errno, strerror(errno));
+ return -EIO ;
+ }
+
+ return ret;
+}
+
+static void vhost_vdpa_listener_region_add(MemoryListener *listener,
+ MemoryRegionSection *section)
+{
+ struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+ hwaddr iova;
+ Int128 llend, llsize;
+ void *vaddr;
+ int ret;
+
+ if (vhost_vdpa_listener_skipped_section(section)) {
+ return;
+ }
+
+ if (unlikely((section->offset_within_address_space & ~TARGET_PAGE_MASK) !=
+ (section->offset_within_region & ~TARGET_PAGE_MASK))) {
+ error_report("%s received unaligned region", __func__);
+ return;
+ }
+
+ iova = TARGET_PAGE_ALIGN(section->offset_within_address_space);
+ llend = int128_make64(section->offset_within_address_space);
+ llend = int128_add(llend, section->size);
+ llend = int128_and(llend, int128_exts64(TARGET_PAGE_MASK));
+
+ if (int128_ge(int128_make64(iova), llend)) {
+ return;
+ }
+
+ memory_region_ref(section->mr);
+
+ /* Here we assume that memory_region_is_ram(section->mr)==true */
+
+ vaddr = memory_region_get_ram_ptr(section->mr) +
+ section->offset_within_region +
+ (iova - section->offset_within_address_space);
+
+ llsize = int128_sub(llend, int128_make64(iova));
+
+ ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize),
+ vaddr, section->readonly);
+ if (ret) {
+ error_report("vhost vdpa map fail!");
+ if (memory_region_is_ram_device(section->mr)) {
+ /* Allow unexpected mappings not to be fatal for RAM devices */
+ error_report("map ram fail!");
+ return ;
+ }
+ goto fail;
+ }
+
+ return;
+
+fail:
+ if (memory_region_is_ram_device(section->mr)) {
+ error_report("failed to vdpa_dma_map. pci p2p may not work");
+ return;
+
+ }
+ /*
+ * On the initfn path, store the first error in the container so we
+ * can gracefully fail. Runtime, there's not much we can do other
+ * than throw a hardware error.
+ */
+ error_report("vhost-vdpa: DMA mapping failed, unable to continue");
+ return;
+
+}
+
+static void vhost_vdpa_listener_region_del(MemoryListener *listener,
+ MemoryRegionSection *section)
+{
+ struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+ hwaddr iova;
+ Int128 llend, llsize;
+ int ret;
+ bool try_unmap = true;
+
+ if (vhost_vdpa_listener_skipped_section(section)) {
+ return;
+ }
+
+ if (unlikely((section->offset_within_address_space & ~TARGET_PAGE_MASK) !=
+ (section->offset_within_region & ~TARGET_PAGE_MASK))) {
+ error_report("%s received unaligned region", __func__);
+ return;
+ }
+
+ iova = TARGET_PAGE_ALIGN(section->offset_within_address_space);
+ llend = int128_make64(section->offset_within_address_space);
+ llend = int128_add(llend, section->size);
+ llend = int128_and(llend, int128_exts64(TARGET_PAGE_MASK));
+
+ if (int128_ge(int128_make64(iova), llend)) {
+ return;
+ }
+
+ llsize = int128_sub(llend, int128_make64(iova));
+
+ if (try_unmap) {
+ ret = vhost_vdpa_dma_unmap(v, iova, int128_get64(llsize));
+ if (ret) {
+ error_report("vhost_vdpa dma unmap error!");
+ }
+ }
+
+ memory_region_unref(section->mr);
+}
+/*
+ * IOTLB API is used by vhost-vpda which requires incremental updating
+ * of the mapping. So we can not use generic vhost memory listener which
+ * depends on the addnop().
+ */
+static const MemoryListener vhost_vdpa_memory_listener = {
+ .region_add = vhost_vdpa_listener_region_add,
+ .region_del = vhost_vdpa_listener_region_del,
+};
+
+static int vhost_vdpa_call(struct vhost_dev *dev, unsigned long int request,
+ void *arg)
+{
+ struct vhost_vdpa *v = dev->opaque;
+ int fd = v->device_fd;
+
+ assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
+
+ return ioctl(fd, request, arg);
+}
+
+static void vhost_vdpa_add_status(struct vhost_dev *dev, uint8_t status)
+{
+ uint8_t s;
+
+ if (vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &s)) {
+ return;
+ }
+
+ s |= status;
+
+ vhost_vdpa_call(dev, VHOST_VDPA_SET_STATUS, &s);
+}
+
+static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque)
+{
+ struct vhost_vdpa *v;
+ uint64_t features;
+ assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
+
+ v = opaque;
+ dev->opaque = opaque ;
+ vhost_vdpa_call(dev, VHOST_GET_FEATURES, &features);
+ dev->backend_features = features;
+ v->listener = vhost_vdpa_memory_listener;
+ v->msg_type = VHOST_IOTLB_MSG_V2;
+
+ vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |
+ VIRTIO_CONFIG_S_DRIVER);
+
+ return 0;
+}
+
+static int vhost_vdpa_cleanup(struct vhost_dev *dev)
+{
+ struct vhost_vdpa *v;
+ assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
+ v = dev->opaque;
+ memory_listener_unregister(&v->listener);
+
+ dev->opaque = NULL;
+ return 0;
+}
+
+static int vhost_vdpa_memslots_limit(struct vhost_dev *dev)
+{
+ return INT_MAX;
+}
+
+static int vhost_vdpa_set_mem_table(struct vhost_dev *dev,
+ struct vhost_memory *mem)
+{
+
+ if (mem->padding) {
+ return -1;
+ }
+
+ return 0;
+}
+
+static int vhost_vdpa_set_features(struct vhost_dev *dev,
+ uint64_t features)
+{
+ int ret;
+ ret = vhost_vdpa_call(dev, VHOST_SET_FEATURES, &features);
+ uint8_t status = 0;
+ if (ret) {
+ return ret;
+ }
+ vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_FEATURES_OK);
+ vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status);
+
+ return !(status & VIRTIO_CONFIG_S_FEATURES_OK);
+}
+
+int vhost_vdpa_get_device_id(struct vhost_dev *dev,
+ uint32_t *device_id)
+{
+ return vhost_vdpa_call(dev, VHOST_VDPA_GET_DEVICE_ID, device_id);
+}
+
+static int vhost_vdpa_reset_device(struct vhost_dev *dev)
+{
+ uint8_t status = 0;
+
+ return vhost_vdpa_call(dev, VHOST_VDPA_SET_STATUS, &status);
+}
+
+static int vhost_vdpa_get_vq_index(struct vhost_dev *dev, int idx)
+{
+ assert(idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs);
+
+ return idx - dev->vq_index;
+}
+
+static int vhost_vdpa_set_vring_ready(struct vhost_dev *dev)
+{
+ int i;
+ for (i = 0; i < dev->nvqs; ++i) {
+ struct vhost_vring_state state = {
+ .index = dev->vq_index + i,
+ .num = 1,
+ };
+ vhost_vdpa_call(dev, VHOST_VDPA_SET_VRING_ENABLE, &state);
+ }
+ return 0;
+}
+
+static int vhost_vdpa_set_config(struct vhost_dev *dev, const uint8_t *data,
+ uint32_t offset, uint32_t size,
+ uint32_t flags)
+{
+ struct vhost_vdpa_config *config;
+ int ret;
+ unsigned long config_size = offsetof(struct vhost_vdpa_config, buf);
+ config = g_malloc(size + config_size);
+ if (config == NULL) {
+ return -1;
+ }
+ config->off = offset;
+ config->len = size;
+ memcpy(config->buf, data, size);
+ ret = vhost_vdpa_call(dev, VHOST_VDPA_SET_CONFIG, config);
+ g_free(config);
+ return ret;
+}
+
+static int vhost_vdpa_get_config(struct vhost_dev *dev, uint8_t *config,
+ uint32_t config_len)
+{
+ struct vhost_vdpa_config *v_config;
+ unsigned long config_size = offsetof(struct vhost_vdpa_config, buf);
+ int ret;
+
+ v_config = g_malloc(config_len + config_size);
+ if (v_config == NULL) {
+ return -1;
+ }
+ v_config->len = config_len;
+ v_config->off = 0;
+ ret = vhost_vdpa_call(dev, VHOST_VDPA_GET_CONFIG, v_config);
+ memcpy(config, v_config->buf, config_len);
+ g_free(v_config);
+ return ret;
+ }
+
+static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started)
+{
+ struct vhost_vdpa *v = dev->opaque;
+ if (started) {
+ uint8_t status = 0;
+ memory_listener_register(&v->listener, &address_space_memory);
+ vhost_vdpa_set_vring_ready(dev);
+ vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK);
+ vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status);
+
+ return !(status & VIRTIO_CONFIG_S_DRIVER_OK);
+ } else {
+ vhost_vdpa_reset_device(dev);
+ vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |
+ VIRTIO_CONFIG_S_DRIVER);
+ memory_listener_unregister(&v->listener);
+
+ return 0;
+ }
+}
+
+static int vhost_vdpa_set_log_base(struct vhost_dev *dev, uint64_t base,
+ struct vhost_log *log)
+{
+ return vhost_vdpa_call(dev, VHOST_SET_LOG_BASE, &base);
+}
+
+static int vhost_vdpa_set_vring_addr(struct vhost_dev *dev,
+ struct vhost_vring_addr *addr)
+{
+ return vhost_vdpa_call(dev, VHOST_SET_VRING_ADDR, addr);
+}
+
+static int vhost_vdpa_set_vring_num(struct vhost_dev *dev,
+ struct vhost_vring_state *ring)
+{
+ return vhost_vdpa_call(dev, VHOST_SET_VRING_NUM, ring);
+}
+
+static int vhost_vdpa_set_vring_base(struct vhost_dev *dev,
+ struct vhost_vring_state *ring)
+{
+ return vhost_vdpa_call(dev, VHOST_SET_VRING_BASE, ring);
+}
+
+static int vhost_vdpa_get_vring_base(struct vhost_dev *dev,
+ struct vhost_vring_state *ring)
+{
+ return vhost_vdpa_call(dev, VHOST_GET_VRING_BASE, ring);
+}
+
+static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev,
+ struct vhost_vring_file *file)
+{
+ return vhost_vdpa_call(dev, VHOST_SET_VRING_KICK, file);
+}
+
+static int vhost_vdpa_set_vring_call(struct vhost_dev *dev,
+ struct vhost_vring_file *file)
+{
+ return vhost_vdpa_call(dev, VHOST_SET_VRING_CALL, file);
+}
+
+static int vhost_vdpa_get_features(struct vhost_dev *dev,
+ uint64_t *features)
+{
+ return vhost_vdpa_call(dev, VHOST_GET_FEATURES, features);
+}
+
+static int vhost_vdpa_set_owner(struct vhost_dev *dev)
+{
+ return vhost_vdpa_call(dev, VHOST_SET_OWNER, NULL);
+}
+
+static int vhost_vdpa_vq_get_addr(struct vhost_dev *dev,
+ struct vhost_vring_addr *addr, struct vhost_virtqueue *vq)
+{
+ assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
+ addr->desc_user_addr = (uint64_t)(unsigned long)vq->desc_phys;
+ addr->avail_user_addr = (uint64_t)(unsigned long)vq->avail_phys;
+ addr->used_user_addr = (uint64_t)(unsigned long)vq->used_phys;
+ return 0;
+}
+
+const VhostOps vdpa_ops = {
+ .backend_type = VHOST_BACKEND_TYPE_VDPA,
+ .vhost_backend_init = vhost_vdpa_init,
+ .vhost_backend_cleanup = vhost_vdpa_cleanup,
+ .vhost_set_log_base = vhost_vdpa_set_log_base,
+ .vhost_set_vring_addr = vhost_vdpa_set_vring_addr,
+ .vhost_set_vring_num = vhost_vdpa_set_vring_num,
+ .vhost_set_vring_base = vhost_vdpa_set_vring_base,
+ .vhost_get_vring_base = vhost_vdpa_get_vring_base,
+ .vhost_set_vring_kick = vhost_vdpa_set_vring_kick,
+ .vhost_set_vring_call = vhost_vdpa_set_vring_call,
+ .vhost_get_features = vhost_vdpa_get_features,
+ .vhost_set_owner = vhost_vdpa_set_owner,
+ .vhost_set_vring_endian = NULL,
+ .vhost_backend_memslots_limit = vhost_vdpa_memslots_limit,
+ .vhost_set_mem_table = vhost_vdpa_set_mem_table,
+ .vhost_set_features = vhost_vdpa_set_features,
+ .vhost_reset_device = vhost_vdpa_reset_device,
+ .vhost_get_vq_index = vhost_vdpa_get_vq_index,
+ .vhost_get_config = vhost_vdpa_get_config,
+ .vhost_set_config = vhost_vdpa_set_config,
+ .vhost_requires_shm_log = NULL,
+ .vhost_migration_done = NULL,
+ .vhost_backend_can_merge = NULL,
+ .vhost_net_set_mtu = NULL,
+ .vhost_set_iotlb_callback = NULL,
+ .vhost_send_device_iotlb_msg = NULL,
+ .vhost_dev_start = vhost_vdpa_dev_start,
+ .vhost_get_device_id = vhost_vdpa_get_device_id,
+ .vhost_vq_get_addr = vhost_vdpa_vq_get_addr,
+};
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 1e083a8976..bcaa66e0d2 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -808,7 +808,9 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
features |= 0x1ULL << VHOST_F_LOG_ALL;
}
if (!vhost_dev_has_iommu(dev)) {
- features &= ~(0x1ULL << VIRTIO_F_IOMMU_PLATFORM);
+ if (dev->vhost_ops->backend_type != VHOST_BACKEND_TYPE_VDPA) {
+ features &= ~(0x1ULL << VIRTIO_F_IOMMU_PLATFORM);
+ }
}
r = dev->vhost_ops->vhost_set_features(dev, features);
if (r < 0) {
@@ -1701,9 +1703,10 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
goto fail_log;
}
}
+
if (vhost_dev_has_iommu(hdev) &&
hdev->vhost_ops->vhost_set_iotlb_callback) {
- hdev->vhost_ops->vhost_set_iotlb_callback(hdev, true);
+ hdev->vhost_ops->vhost_set_iotlb_callback(hdev, true);
/* Update used ring information for IOTLB to work correctly,
* vhost-kernel code requires for this.*/
diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
index bfc24207e2..01f7c0d46f 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -17,7 +17,8 @@ typedef enum VhostBackendType {
VHOST_BACKEND_TYPE_NONE = 0,
VHOST_BACKEND_TYPE_KERNEL = 1,
VHOST_BACKEND_TYPE_USER = 2,
- VHOST_BACKEND_TYPE_MAX = 3,
+ VHOST_BACKEND_TYPE_VDPA = 3,
+ VHOST_BACKEND_TYPE_MAX = 4,
} VhostBackendType;
typedef enum VhostSetConfigType {
@@ -167,6 +168,7 @@ typedef struct VhostOps {
} VhostOps;
extern const VhostOps user_ops;
+extern const VhostOps vdpa_ops;
int vhost_set_backend_type(struct vhost_dev *dev,
VhostBackendType backend_type);
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
new file mode 100644
index 0000000000..6455663388
--- /dev/null
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -0,0 +1,26 @@
+/*
+ * vhost-vdpa.h
+ *
+ * Copyright(c) 2017-2018 Intel Corporation.
+ * Copyright(c) 2020 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef HW_VIRTIO_VHOST_VDPA_H
+#define HW_VIRTIO_VHOST_VDPA_H
+
+#include "hw/virtio/virtio.h"
+
+typedef struct vhost_vdpa {
+ int device_fd;
+ uint32_t msg_type;
+ MemoryListener listener;
+} VhostVDPA;
+
+extern AddressSpace address_space_memory;
+extern int vhost_vdpa_get_device_id(struct vhost_dev *dev,
+ uint32_t *device_id);
+#endif
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index 085450c6f8..767a95ec0b 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -92,6 +92,13 @@ struct vhost_dev {
const VhostDevConfigOps *config_ops;
};
+struct vhost_net {
+ struct vhost_dev dev;
+ struct vhost_virtqueue vqs[2];
+ int backend;
+ NetClientState *nc;
+};
+
int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
VhostBackendType backend_type,
uint32_t busyloop_timeout);
diff --git a/qemu-options.hx b/qemu-options.hx
index 196f468786..fa1b19de4c 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -2418,6 +2418,10 @@ DEF("netdev", HAS_ARG, QEMU_OPTION_netdev,
#ifdef CONFIG_POSIX
"-netdev vhost-user,id=str,chardev=dev[,vhostforce=on|off]\n"
" configure a vhost-user network, backed by a chardev 'dev'\n"
+#endif
+#ifdef __linux__
+ "-netdev vhost-vdpa,id=str,vhostdev=/path/to/dev\n"
+ " configure a vhost-vdpa network,Establish a vhost-vdpa netdev\n"
#endif
"-netdev hubport,id=str,hubid=n[,netdev=nd]\n"
" configure a hub port on the hub with ID 'n'\n", QEMU_ARCH_ALL)
@@ -2897,6 +2901,14 @@ SRST
-netdev type=vhost-user,id=net0,chardev=chr0 \
-device virtio-net-pci,netdev=net0
+``-netdev vhost-vdpa,vhostdev=/path/to/dev``
+ Establish a vhost-vdpa netdev.
+
+ vDPA device is a device that uses a datapath which complies with
+ the virtio specifications with a vendor specific control path.
+ vDPA devices can be both physically located on the hardware or
+ emulated by software.
+
``-netdev hubport,id=id,hubid=hubid[,netdev=nd]``
Create a hub port on the emulated hub with ID hubid.
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH v2 12/12] vhost-vdpa: introduce vhost-vdpa net client
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
` (10 preceding siblings ...)
2020-06-30 17:49 ` [PATCH v2 11/12] vhost-vdpa: introduce vhost-vdpa backend Cindy Lu
@ 2020-06-30 17:49 ` Cindy Lu
2020-07-01 14:24 ` Eric Blake
2020-06-30 18:06 ` [PATCH v2 00/12] vDPA support in qemu no-reply
12 siblings, 1 reply; 20+ messages in thread
From: Cindy Lu @ 2020-06-30 17:49 UTC (permalink / raw)
To: mst, armbru, eblake, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, lulu, hanand, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
Tiwei Bie, aadam, rdunlap, maxime.coquelin, lingshan.zhu
This patch set introduces a new net client type: vhost-vdpa.
vhost-vdpa net client will set up a vDPA device which is specified
by a "vhostdev" parameter.
Signed-off-by: Lingshan Zhu <lingshan.zhu@intel.com>
Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
Signed-off-by: Cindy Lu <lulu@redhat.com>
---
include/net/vhost-vdpa.h | 22 ++++
net/Makefile.objs | 2 +-
net/clients.h | 2 +
net/net.c | 3 +
net/vhost-vdpa.c | 228 +++++++++++++++++++++++++++++++++++++++
qapi/net.json | 28 ++++-
6 files changed, 281 insertions(+), 4 deletions(-)
create mode 100644 include/net/vhost-vdpa.h
create mode 100644 net/vhost-vdpa.c
diff --git a/include/net/vhost-vdpa.h b/include/net/vhost-vdpa.h
new file mode 100644
index 0000000000..45e34b7cfc
--- /dev/null
+++ b/include/net/vhost-vdpa.h
@@ -0,0 +1,22 @@
+/*
+ * vhost-vdpa.h
+ *
+ * Copyright(c) 2017-2018 Intel Corporation.
+ * Copyright(c) 2020 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef VHOST_VDPA_H
+#define VHOST_VDPA_H
+
+#define TYPE_VHOST_VDPA "vhost-vdpa"
+
+struct vhost_net *vhost_vdpa_get_vhost_net(NetClientState *nc);
+uint64_t vhost_vdpa_get_acked_features(NetClientState *nc);
+
+extern const int vdpa_feature_bits[];
+
+#endif /* VHOST_VDPA_H */
diff --git a/net/Makefile.objs b/net/Makefile.objs
index c5d076d19c..5ab45545db 100644
--- a/net/Makefile.objs
+++ b/net/Makefile.objs
@@ -26,7 +26,7 @@ tap-obj-$(CONFIG_SOLARIS) = tap-solaris.o
tap-obj-y ?= tap-stub.o
common-obj-$(CONFIG_POSIX) += tap.o $(tap-obj-y)
common-obj-$(CONFIG_WIN32) += tap-win32.o
-
+common-obj-$(CONFIG_VHOST_NET_VDPA) += vhost-vdpa.o
vde.o-libs = $(VDE_LIBS)
common-obj-$(CONFIG_CAN_BUS) += can/
diff --git a/net/clients.h b/net/clients.h
index a6ef267e19..92f9b59aed 100644
--- a/net/clients.h
+++ b/net/clients.h
@@ -61,4 +61,6 @@ int net_init_netmap(const Netdev *netdev, const char *name,
int net_init_vhost_user(const Netdev *netdev, const char *name,
NetClientState *peer, Error **errp);
+int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
+ NetClientState *peer, Error **errp);
#endif /* QEMU_NET_CLIENTS_H */
diff --git a/net/net.c b/net/net.c
index 9099a327dd..94dc546fb2 100644
--- a/net/net.c
+++ b/net/net.c
@@ -966,6 +966,9 @@ static int (* const net_client_init_fun[NET_CLIENT_DRIVER__MAX])(
#ifdef CONFIG_VHOST_NET_USER
[NET_CLIENT_DRIVER_VHOST_USER] = net_init_vhost_user,
#endif
+#ifdef CONFIG_VHOST_NET_VDPA
+ [NET_CLIENT_DRIVER_VHOST_VDPA] = net_init_vhost_vdpa,
+#endif
#ifdef CONFIG_L2TPV3
[NET_CLIENT_DRIVER_L2TPV3] = net_init_l2tpv3,
#endif
diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
new file mode 100644
index 0000000000..bc0e0d2d35
--- /dev/null
+++ b/net/vhost-vdpa.c
@@ -0,0 +1,228 @@
+/*
+ * vhost-vdpa.c
+ *
+ * Copyright(c) 2017-2018 Intel Corporation.
+ * Copyright(c) 2020 Red Hat, Inc.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu/osdep.h"
+#include "clients.h"
+#include "net/vhost_net.h"
+#include "net/vhost-vdpa.h"
+#include "hw/virtio/vhost-vdpa.h"
+#include "qemu/config-file.h"
+#include "qemu/error-report.h"
+#include "qemu/option.h"
+#include "qapi/error.h"
+#include <sys/ioctl.h>
+#include <err.h>
+#include "standard-headers/linux/virtio_net.h"
+#include "monitor/monitor.h"
+#include "hw/virtio/vhost.h"
+
+/* Todo:need to add the multiqueue support here */
+typedef struct VhostVDPAState {
+ NetClientState nc;
+ struct vhost_vdpa vhost_vdpa;
+ VHostNetState *vhost_net;
+ uint64_t acked_features;
+ bool started;
+} VhostVDPAState;
+
+const int vdpa_feature_bits[] = {
+ VIRTIO_F_NOTIFY_ON_EMPTY,
+ VIRTIO_RING_F_INDIRECT_DESC,
+ VIRTIO_RING_F_EVENT_IDX,
+ VIRTIO_F_ANY_LAYOUT,
+ VIRTIO_F_VERSION_1,
+ VIRTIO_NET_F_CSUM,
+ VIRTIO_NET_F_GUEST_CSUM,
+ VIRTIO_NET_F_GSO,
+ VIRTIO_NET_F_GUEST_TSO4,
+ VIRTIO_NET_F_GUEST_TSO6,
+ VIRTIO_NET_F_GUEST_ECN,
+ VIRTIO_NET_F_GUEST_UFO,
+ VIRTIO_NET_F_HOST_TSO4,
+ VIRTIO_NET_F_HOST_TSO6,
+ VIRTIO_NET_F_HOST_ECN,
+ VIRTIO_NET_F_HOST_UFO,
+ VIRTIO_NET_F_MRG_RXBUF,
+ VIRTIO_NET_F_MTU,
+ VIRTIO_F_IOMMU_PLATFORM,
+ VIRTIO_F_RING_PACKED,
+ VIRTIO_NET_F_GUEST_ANNOUNCE,
+ VHOST_INVALID_FEATURE_BIT
+};
+
+VHostNetState *vhost_vdpa_get_vhost_net(NetClientState *nc)
+{
+ VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
+ assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
+ return s->vhost_net;
+}
+
+uint64_t vhost_vdpa_get_acked_features(NetClientState *nc)
+{
+ VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
+ assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
+ s->acked_features = vhost_net_get_acked_features(s->vhost_net);
+
+ return s->acked_features;
+}
+
+static int vhost_vdpa_net_check_device_id(struct vhost_net *net)
+{
+ uint32_t device_id;
+ int ret;
+ struct vhost_dev *hdev;
+
+ hdev = (struct vhost_dev *)&net->dev;
+ ret = hdev->vhost_ops->vhost_get_device_id(hdev, &device_id);
+ if (device_id != VIRTIO_ID_NET) {
+ return -ENOTSUP;
+ }
+ return ret;
+}
+
+static void vhost_vdpa_del(NetClientState *ncs)
+{
+ VhostVDPAState *s;
+ assert(ncs->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
+ s = DO_UPCAST(VhostVDPAState, nc, ncs);
+ if (s->vhost_net) {
+ vhost_net_cleanup(s->vhost_net);
+ }
+}
+
+static int vhost_vdpa_add(NetClientState *ncs, void *be)
+{
+ VhostNetOptions options;
+ struct vhost_net *net = NULL;
+ VhostVDPAState *s;
+ int ret;
+
+ options.backend_type = VHOST_BACKEND_TYPE_VDPA;
+ assert(ncs->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
+ s = DO_UPCAST(VhostVDPAState, nc, ncs);
+ options.net_backend = ncs;
+ options.opaque = be;
+ options.busyloop_timeout = 0;
+
+ net = vhost_net_init(&options);
+ if (!net) {
+ error_report("failed to init vhost_net for queue");
+ goto err;
+ }
+ if (s->vhost_net) {
+ vhost_net_cleanup(s->vhost_net);
+ g_free(s->vhost_net);
+ }
+ s->vhost_net = net;
+ ret = vhost_vdpa_net_check_device_id(net);
+ if (ret) {
+ goto err;
+ }
+ return 0;
+err:
+ if (net) {
+ vhost_net_cleanup(net);
+ }
+ vhost_vdpa_del(ncs);
+ return -1;
+}
+
+static void vhost_vdpa_cleanup(NetClientState *nc)
+{
+ VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
+
+ if (s->vhost_net) {
+ vhost_net_cleanup(s->vhost_net);
+ g_free(s->vhost_net);
+ s->vhost_net = NULL;
+ }
+}
+
+static bool vhost_vdpa_has_vnet_hdr(NetClientState *nc)
+{
+ assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
+
+ return true;
+}
+
+static bool vhost_vdpa_has_ufo(NetClientState *nc)
+{
+ assert(nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA);
+ VhostVDPAState *s = DO_UPCAST(VhostVDPAState, nc, nc);
+ uint64_t features = 0;
+ features |= (1ULL << VIRTIO_NET_F_HOST_UFO);
+ features = vhost_net_get_features(s->vhost_net, features);
+ return !!(features & (1ULL << VIRTIO_NET_F_HOST_UFO));
+
+}
+
+static NetClientInfo net_vhost_vdpa_info = {
+ .type = NET_CLIENT_DRIVER_VHOST_VDPA,
+ .size = sizeof(VhostVDPAState),
+ .cleanup = vhost_vdpa_cleanup,
+ .has_vnet_hdr = vhost_vdpa_has_vnet_hdr,
+ .has_ufo = vhost_vdpa_has_ufo,
+};
+
+static int net_vhost_vdpa_init(NetClientState *peer, const char *device,
+ const char *name, const char *vhostdev)
+{
+ NetClientState *nc = NULL;
+ VhostVDPAState *s;
+ int vdpa_device_fd = -1;
+ int ret = 0;
+ assert(name);
+ nc = qemu_new_net_client(&net_vhost_vdpa_info, peer, device, name);
+ snprintf(nc->info_str, sizeof(nc->info_str), TYPE_VHOST_VDPA);
+ nc->queue_index = 0;
+ s = DO_UPCAST(VhostVDPAState, nc, nc);
+ vdpa_device_fd = qemu_open(vhostdev, O_RDWR);
+ if (vdpa_device_fd == -1) {
+ return -errno;
+ }
+ s->vhost_vdpa.device_fd = vdpa_device_fd;
+ ret = vhost_vdpa_add(nc, (void *)&s->vhost_vdpa);
+ assert(s->vhost_net);
+ return ret;
+}
+
+static int net_vhost_check_net(void *opaque, QemuOpts *opts, Error **errp)
+{
+ const char *name = opaque;
+ const char *driver, *netdev;
+
+ driver = qemu_opt_get(opts, "driver");
+ netdev = qemu_opt_get(opts, "netdev");
+ if (!driver || !netdev) {
+ return 0;
+ }
+ if (strcmp(netdev, name) == 0 &&
+ !g_str_has_prefix(driver, "virtio-net-")) {
+ error_setg(errp, "vhost-vdpa requires frontend driver virtio-net-*");
+ return -1;
+ }
+ return 0;
+}
+
+int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
+ NetClientState *peer, Error **errp)
+{
+ const NetdevVhostVDPAOptions *opts;
+
+ assert(netdev->type == NET_CLIENT_DRIVER_VHOST_VDPA);
+ opts = &netdev->u.vhost_vdpa;
+ /* verify net frontend */
+ if (qemu_opts_foreach(qemu_find_opts("device"), net_vhost_check_net,
+ (char *)name, errp)) {
+ return -1;
+ }
+ return net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, opts->vhostdev);
+}
diff --git a/qapi/net.json b/qapi/net.json
index 9244c9af56..00fc915640 100644
--- a/qapi/net.json
+++ b/qapi/net.json
@@ -428,16 +428,37 @@
'*vhostforce': 'bool',
'*queues': 'int' } }
+##
+# @NetdevVhostVDPAOptions:
+#
+# Vhost-vdpa network backend
+#
+# vDPA device is a device that uses a datapath which complies with the virtio
+# specifications with a vendor specific control path.
+#
+# @vhostdev: path of vhost-vdpa device
+# (default:'/dev/vhost-vdpa-0')
+#
+# @queues: number of queues to be created for multiqueue vhost-vdpa
+# (default: 1)
+#
+# Since: 5.1
+##
+{ 'struct': 'NetdevVhostVDPAOptions',
+ 'data': {
+ '*vhostdev': 'str',
+ '*queues': 'int' } }
+
##
# @NetClientDriver:
#
# Available netdev drivers.
#
-# Since: 2.7
+# Since: 5.1
##
{ 'enum': 'NetClientDriver',
'data': [ 'none', 'nic', 'user', 'tap', 'l2tpv3', 'socket', 'vde',
- 'bridge', 'hubport', 'netmap', 'vhost-user' ] }
+ 'bridge', 'hubport', 'netmap', 'vhost-user', 'vhost-vdpa' ] }
##
# @Netdev:
@@ -465,7 +486,8 @@
'bridge': 'NetdevBridgeOptions',
'hubport': 'NetdevHubPortOptions',
'netmap': 'NetdevNetmapOptions',
- 'vhost-user': 'NetdevVhostUserOptions' } }
+ 'vhost-user': 'NetdevVhostUserOptions',
+ 'vhost-vdpa': 'NetdevVhostVDPAOptions' } }
##
# @NetFilterDirection:
--
2.21.1
^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH v2 00/12] vDPA support in qemu
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
` (11 preceding siblings ...)
2020-06-30 17:49 ` [PATCH v2 12/12] vhost-vdpa: introduce vhost-vdpa net client Cindy Lu
@ 2020-06-30 18:06 ` no-reply
2020-07-01 5:29 ` Jason Wang
12 siblings, 1 reply; 20+ messages in thread
From: no-reply @ 2020-06-30 18:06 UTC (permalink / raw)
To: lulu
Cc: rdunlap, mst, mhabets, qemu-devel, rob.miller, saugatm, lulu,
armbru, hch, eperezma, jgg, jasowang, shahafs, kevin.tian, parav,
vmireyno, cunming.liang, gdawar, jiri, xiao.w.wang, stefanha,
zhihong.wang, maxime.coquelin, aadam, cohuck, hanand,
lingshan.zhu
Patchew URL: https://patchew.org/QEMU/20200630174937.25560-1-lulu@redhat.com/
Hi,
This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.
=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===
/tmp/qemu-test/src/hw/net/virtio-net.c:173: undefined reference to `vhost_dev_set_config'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.1/../../../../x86_64-w64-mingw32/bin/ld: hw/net/virtio-net.o: in function `virtio_net_get_config':
/tmp/qemu-test/src/hw/net/virtio-net.c:149: undefined reference to `vhost_dev_get_config'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:208: qemu-system-x86_64w.exe] Error 1
make: *** [Makefile:527: x86_64-softmmu/all] Error 2
make: *** Waiting for unfinished jobs....
LINK aarch64-softmmu/qemu-system-aarch64w.exe
/usr/lib/gcc/x86_64-w64-mingw32/9.2.1/../../../../x86_64-w64-mingw32/bin/ld: hw/net/virtio-net.o: in function `virtio_net_set_config':
/tmp/qemu-test/src/hw/net/virtio-net.c:173: undefined reference to `vhost_dev_set_config'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.1/../../../../x86_64-w64-mingw32/bin/ld: hw/net/virtio-net.o: in function `virtio_net_get_config':
/tmp/qemu-test/src/hw/net/virtio-net.c:149: undefined reference to `vhost_dev_get_config'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:208: qemu-system-aarch64w.exe] Error 1
make: *** [Makefile:527: aarch64-softmmu/all] Error 2
Traceback (most recent call last):
File "./tests/docker/docker.py", line 669, in <module>
sys.exit(main())
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=9f5def1ac1a840deb219e8827020f24d', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-lwyumgcs/src/docker-src.2020-06-30-14.01.04.32102:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=9f5def1ac1a840deb219e8827020f24d
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-lwyumgcs/src'
make: *** [docker-run-test-mingw@fedora] Error 2
real 5m52.723s
user 0m8.824s
The full log is available at
http://patchew.org/logs/20200630174937.25560-1-lulu@redhat.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2 11/12] vhost-vdpa: introduce vhost-vdpa backend
2020-06-30 17:49 ` [PATCH v2 11/12] vhost-vdpa: introduce vhost-vdpa backend Cindy Lu
@ 2020-07-01 4:21 ` Jason Wang
2020-07-01 9:17 ` Cindy Lu
0 siblings, 1 reply; 20+ messages in thread
From: Jason Wang @ 2020-07-01 4:21 UTC (permalink / raw)
To: Cindy Lu, mst, armbru, eblake, cohuck
Cc: mhabets, qemu-devel, rob.miller, saugatm, hanand, hch, eperezma,
jgg, shahafs, kevin.tian, parav, vmireyno, cunming.liang, gdawar,
jiri, xiao.w.wang, stefanha, zhihong.wang, Tiwei Bie, aadam,
rdunlap, maxime.coquelin, lingshan.zhu
On 2020/7/1 上午1:49, Cindy Lu wrote:
> Currently we have 2 types of vhost backends in QEMU: vhost kernel and
> vhost-user. The above patch provides a generic device for vDPA purpose,
> this vDPA device exposes to user space a non-vendor-specific configuration
> interface for setting up a vhost HW accelerator, this patch set introduces
> a third vhost backend called vhost-vdpa based on the vDPA interface.
>
> Vhost-vdpa usage:
>
> qemu-system-x86_64 -cpu host -enable-kvm \
> ......
> -netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-id,id=vhost-vdpa0 \
> -device virtio-net-pci,netdev=vhost-vdpa0,page-per-vq=on \
>
> Signed-off-by: Lingshan zhu <lingshan.zhu@intel.com>
> Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
I would like to add my sob here.
> Signed-off-by: Cindy Lu <lulu@redhat.com>
> ---
> configure | 21 ++
> docs/interop/index.rst | 1 +
> docs/interop/vhost-vdpa.rst | 17 ++
> hw/net/vhost_net.c | 19 +-
> hw/net/virtio-net.c | 22 +-
> hw/virtio/Makefile.objs | 1 +
> hw/virtio/vhost-backend.c | 7 +-
> hw/virtio/vhost-vdpa.c | 469 ++++++++++++++++++++++++++++++
> hw/virtio/vhost.c | 7 +-
> include/hw/virtio/vhost-backend.h | 4 +-
> include/hw/virtio/vhost-vdpa.h | 26 ++
> include/hw/virtio/vhost.h | 7 +
> qemu-options.hx | 12 +
> 13 files changed, 601 insertions(+), 12 deletions(-)
> create mode 100644 docs/interop/vhost-vdpa.rst
> create mode 100644 hw/virtio/vhost-vdpa.c
> create mode 100644 include/hw/virtio/vhost-vdpa.h
>
> diff --git a/configure b/configure
> index 4a22dcd563..3db7f20185 100755
> --- a/configure
> +++ b/configure
> @@ -1575,6 +1575,10 @@ for opt do
> ;;
> --enable-vhost-user) vhost_user="yes"
> ;;
> + --disable-vhost-vdpa) vhost_vdpa="no"
> + ;;
> + --enable-vhost-vdpa) vhost_vdpa="yes"
> + ;;
> --disable-vhost-kernel) vhost_kernel="no"
> ;;
> --enable-vhost-kernel) vhost_kernel="yes"
> @@ -1883,6 +1887,7 @@ disabled with --disable-FEATURE, default is enabled if available:
> vhost-crypto vhost-user-crypto backend support
> vhost-kernel vhost kernel backend support
> vhost-user vhost-user backend support
> + vhost-vdpa vhost-vdpa kernel backend support
> spice spice
> rbd rados block device (rbd)
> libiscsi iscsi support
> @@ -2394,6 +2399,10 @@ test "$vhost_user" = "" && vhost_user=yes
> if test "$vhost_user" = "yes" && test "$mingw32" = "yes"; then
> error_exit "vhost-user isn't available on win32"
> fi
> +test "$vhost_vdpa" = "" && vhost_vdpa=$linux
> +if test "$vhost_vdpa" = "yes" && test "$linux" != "yes"; then
> + error_exit "vhost-vdpa is only available on Linux"
> +fi
> test "$vhost_kernel" = "" && vhost_kernel=$linux
> if test "$vhost_kernel" = "yes" && test "$linux" != "yes"; then
> error_exit "vhost-kernel is only available on Linux"
> @@ -2422,6 +2431,11 @@ test "$vhost_user_fs" = "" && vhost_user_fs=$vhost_user
> if test "$vhost_user_fs" = "yes" && test "$vhost_user" = "no"; then
> error_exit "--enable-vhost-user-fs requires --enable-vhost-user"
> fi
> +#vhost-vdpa backends
> +test "$vhost_net_vdpa" = "" && vhost_net_vdpa=$vhost_vdpa
> +if test "$vhost_net_vdpa" = "yes" && test "$vhost_vdpa" = "no"; then
> + error_exit "--enable-vhost-net-vdpa requires --enable-vhost-vdpa"
> +fi
>
> # OR the vhost-kernel and vhost-user values for simplicity
> if test "$vhost_net" = ""; then
> @@ -6936,6 +6950,7 @@ echo "vhost-scsi support $vhost_scsi"
> echo "vhost-vsock support $vhost_vsock"
> echo "vhost-user support $vhost_user"
> echo "vhost-user-fs support $vhost_user_fs"
> +echo "vhost-vdpa support $vhost_vdpa"
> echo "Trace backends $trace_backends"
> if have_backend "simple"; then
> echo "Trace output file $trace_file-<pid>"
> @@ -7437,6 +7452,9 @@ fi
> if test "$vhost_net_user" = "yes" ; then
> echo "CONFIG_VHOST_NET_USER=y" >> $config_host_mak
> fi
> +if test "$vhost_net_vdpa" = "yes" ; then
> + echo "CONFIG_VHOST_NET_VDPA=y" >> $config_host_mak
> +fi
> if test "$vhost_crypto" = "yes" ; then
> echo "CONFIG_VHOST_CRYPTO=y" >> $config_host_mak
> fi
> @@ -7452,6 +7470,9 @@ fi
> if test "$vhost_user" = "yes" ; then
> echo "CONFIG_VHOST_USER=y" >> $config_host_mak
> fi
> +if test "$vhost_vdpa" = "yes" ; then
> + echo "CONFIG_VHOST_VDPA=y" >> $config_host_mak
> +fi
> if test "$vhost_user_fs" = "yes" ; then
> echo "CONFIG_VHOST_USER_FS=y" >> $config_host_mak
> fi
> diff --git a/docs/interop/index.rst b/docs/interop/index.rst
> index 049387ac6d..006f986420 100644
> --- a/docs/interop/index.rst
> +++ b/docs/interop/index.rst
> @@ -20,3 +20,4 @@ Contents:
> qemu-ga
> vhost-user
> vhost-user-gpu
> + vhost-vdpa
> diff --git a/docs/interop/vhost-vdpa.rst b/docs/interop/vhost-vdpa.rst
> new file mode 100644
> index 0000000000..5a075b8a63
> --- /dev/null
> +++ b/docs/interop/vhost-vdpa.rst
> @@ -0,0 +1,17 @@
> +=====================
> +Vhost-vdpa Protocol
> +=====================
> +
> +Introduction
> +=============
> +vDPA(Virtual data path acceleration) device is a device that uses
> +a datapath which complies with the virtio specifications with vendor
> +specific control path. vDPA devices can be both physically located on
> +the hardware or emulated by software.
> +
> +This document describes the vDPA support in qemu
> +
> +Here is the kernel commit here
> +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4c8cf31885f69e86be0b5b9e6677a26797365e1d
> +
> +TODO : Moreinfo will add later
> diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
> index 4096d64aaf..cbdf42ace5 100644
> --- a/hw/net/vhost_net.c
> +++ b/hw/net/vhost_net.c
> @@ -17,8 +17,10 @@
> #include "net/net.h"
> #include "net/tap.h"
> #include "net/vhost-user.h"
> +#include "net/vhost-vdpa.h"
>
> #include "standard-headers/linux/vhost_types.h"
> +#include "linux-headers/linux/vhost.h"
> #include "hw/virtio/virtio-net.h"
> #include "net/vhost_net.h"
> #include "qemu/error-report.h"
> @@ -33,12 +35,6 @@
> #include "hw/virtio/vhost.h"
> #include "hw/virtio/virtio-bus.h"
>
> -struct vhost_net {
> - struct vhost_dev dev;
> - struct vhost_virtqueue vqs[2];
> - int backend;
> - NetClientState *nc;
> -};
>
> /* Features supported by host kernel. */
> static const int kernel_feature_bits[] = {
> @@ -96,6 +92,9 @@ static const int *vhost_net_get_feature_bits(struct vhost_net *net)
> case NET_CLIENT_DRIVER_VHOST_USER:
> feature_bits = user_feature_bits;
> break;
> + case NET_CLIENT_DRIVER_VHOST_VDPA:
> + feature_bits = vdpa_feature_bits;
> + break;
> default:
> error_report("Feature bits not defined for this type: %d",
> net->nc->info->type);
> @@ -316,7 +315,6 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs,
> }
>
> for (i = 0; i < total_queues; i++) {
> -
Unnecessary change.
> peer = qemu_get_peer(ncs, i);
> net = get_vhost_net(peer);
> vhost_net_set_vq_index(net, i * 2);
> @@ -337,6 +335,7 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs,
> }
>
> for (i = 0; i < total_queues; i++) {
> +
Unnecessary change.
> peer = qemu_get_peer(ncs, i);
> r = vhost_net_start_one(get_vhost_net(peer), dev);
>
> @@ -433,6 +432,12 @@ VHostNetState *get_vhost_net(NetClientState *nc)
> vhost_net = vhost_user_get_vhost_net(nc);
> assert(vhost_net);
> break;
> +#endif
> +#ifdef CONFIG_VHOST_NET_VDPA
> + case NET_CLIENT_DRIVER_VHOST_VDPA:
> + vhost_net = vhost_vdpa_get_vhost_net(nc);
> + assert(vhost_net);
> + break;
> #endif
> default:
> break;
> diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> index aff67a92df..6aadc2403a 100644
> --- a/hw/net/virtio-net.c
> +++ b/hw/net/virtio-net.c
> @@ -43,6 +43,7 @@
> #include "monitor/qdev.h"
> #include "hw/pci/pci.h"
> #include "net_rx_pkt.h"
> +#include "hw/virtio/vhost.h"
>
> #define VIRTIO_NET_VM_VERSION 11
>
> @@ -124,7 +125,10 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
> {
> VirtIONet *n = VIRTIO_NET(vdev);
> struct virtio_net_config netcfg;
> + int ret = 0;
> + struct vhost_dev *hdev = NULL;
>
> + memset(&netcfg, 0 , sizeof(struct virtio_net_config));
> virtio_stw_p(vdev, &netcfg.status, n->status);
> virtio_stw_p(vdev, &netcfg.max_virtqueue_pairs, n->max_queues);
> virtio_stw_p(vdev, &netcfg.mtu, n->net_conf.mtu);
> @@ -138,21 +142,37 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
> virtio_stl_p(vdev, &netcfg.supported_hash_types,
> VIRTIO_NET_RSS_SUPPORTED_HASHES);
> memcpy(config, &netcfg, n->config_size);
> +
> + NetClientState *nc = qemu_get_queue(n->nic);
> + if (nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
> + hdev = &(get_vhost_net(nc->peer)->dev);
> + ret = vhost_dev_get_config(hdev, (uint8_t *)&netcfg, n->config_size);
> + if (ret != -1) {
> + memcpy(config, &netcfg, n->config_size);
> + }
> + }
> }
>
> static void virtio_net_set_config(VirtIODevice *vdev, const uint8_t *config)
> {
> VirtIONet *n = VIRTIO_NET(vdev);
> struct virtio_net_config netcfg = {};
> + struct vhost_dev *hdev = NULL;
>
> memcpy(&netcfg, config, n->config_size);
> -
Unnecessary change.
> if (!virtio_vdev_has_feature(vdev, VIRTIO_NET_F_CTRL_MAC_ADDR) &&
> !virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1) &&
> memcmp(netcfg.mac, n->mac, ETH_ALEN)) {
> memcpy(n->mac, netcfg.mac, ETH_ALEN);
> qemu_format_nic_info_str(qemu_get_queue(n->nic), n->mac);
> }
> +
> + NetClientState *nc = qemu_get_queue(n->nic);
> + if (nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
> + hdev = &(get_vhost_net(nc->peer)->dev);
> + vhost_dev_set_config(hdev, (uint8_t *)&netcfg, 0, n->config_size,
> + VHOST_SET_CONFIG_TYPE_MASTER);
> + }
> }
>
> static bool virtio_net_started(VirtIONet *n, uint8_t status)
> diff --git a/hw/virtio/Makefile.objs b/hw/virtio/Makefile.objs
> index 13e75f171f..2b31e9b3dc 100644
> --- a/hw/virtio/Makefile.objs
> +++ b/hw/virtio/Makefile.objs
> @@ -5,6 +5,7 @@ obj-y += virtio.o
> obj-$(CONFIG_VHOST) += vhost.o vhost-backend.o
> common-obj-$(call lnot,$(CONFIG_VHOST)) += vhost-stub.o
> obj-$(CONFIG_VHOST_USER) += vhost-user.o
> +obj-$(CONFIG_VHOST_VDPA) += vhost-vdpa.o
>
> common-obj-$(CONFIG_VIRTIO_RNG) += virtio-rng.o
> common-obj-$(CONFIG_VIRTIO_PCI) += virtio-pci.o
> diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c
> index 48905383f8..b925a2b1ac 100644
> --- a/hw/virtio/vhost-backend.c
> +++ b/hw/virtio/vhost-backend.c
> @@ -14,7 +14,7 @@
> #include "qemu/error-report.h"
> #include "qemu/main-loop.h"
> #include "standard-headers/linux/vhost_types.h"
> -
> +#include "hw/virtio/vhost-vdpa.h"
Let's keep the newline here.
> #ifdef CONFIG_VHOST_KERNEL
> #include <linux/vhost.h>
> #include <sys/ioctl.h>
> @@ -285,6 +285,11 @@ int vhost_set_backend_type(struct vhost_dev *dev, VhostBackendType backend_type)
> case VHOST_BACKEND_TYPE_USER:
> dev->vhost_ops = &user_ops;
> break;
> +#endif
> +#ifdef CONFIG_VHOST_VDPA
> + case VHOST_BACKEND_TYPE_VDPA:
> + dev->vhost_ops = &vdpa_ops;
> + break;
> #endif
> default:
> error_report("Unknown vhost backend type");
> diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
> new file mode 100644
> index 0000000000..25c9b537b0
> --- /dev/null
> +++ b/hw/virtio/vhost-vdpa.c
> @@ -0,0 +1,469 @@
> +/*
> + * vhost-vdpa
> + *
> + * Copyright(c) 2017-2018 Intel Corporation.
> + * Copyright(c) 2020 Red Hat, Inc.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include "qemu/osdep.h"
> +#include <linux/vhost.h>
> +#include <linux/vfio.h>
> +#include <sys/eventfd.h>
> +#include <sys/ioctl.h>
> +#include "hw/virtio/vhost.h"
> +#include "hw/virtio/vhost-backend.h"
> +#include "hw/virtio/virtio-net.h"
> +#include "hw/virtio/vhost-vdpa.h"
> +#include "qemu/main-loop.h"
> +#include <linux/kvm.h>
> +#include "sysemu/kvm.h"
> +
> +static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section)
> +{
> + return (!memory_region_is_ram(section->mr) &&
> + !memory_region_is_iommu(section->mr)) ||
> + /*
> + * Sizing an enabled 64-bit BAR can cause spurious mappings to
> + * addresses in the upper part of the 64-bit address space. These
> + * are never accessed by the CPU and beyond the address width of
> + * some IOMMU hardware. TODO: VDPA should tell us the IOMMU width.
> + */
> + section->offset_within_address_space & (1ULL << 63);
> +}
> +
> +static int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
> + void *vaddr, bool readonly)
> +{
> + struct vhost_msg_v2 msg;
> + int fd = v->device_fd;
> + int ret = 0;
> +
> + msg.type = v->msg_type;
> + msg.iotlb.iova = iova;
> + msg.iotlb.size = size;
> + msg.iotlb.uaddr = (uint64_t)vaddr;
> + msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
> + msg.iotlb.type = VHOST_IOTLB_UPDATE;
> +
> + if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
> + error_report("failed to write, fd=%d, errno=%d (%s)",
> + fd, errno, strerror(errno));
> + return -EIO ;
> + }
> +
> + return ret;
> +}
> +
> +static int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova,
> + hwaddr size)
> +{
> + struct vhost_msg_v2 msg;
> + int fd = v->device_fd;
> + int ret = 0;
> +
> + msg.type = v->msg_type;
> + msg.iotlb.iova = iova;
> + msg.iotlb.size = size;
> + msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
> +
> + if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
> + error_report("failed to write, fd=%d, errno=%d (%s)",
> + fd, errno, strerror(errno));
> + return -EIO ;
> + }
> +
> + return ret;
> +}
> +
> +static void vhost_vdpa_listener_region_add(MemoryListener *listener,
> + MemoryRegionSection *section)
> +{
> + struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
> + hwaddr iova;
> + Int128 llend, llsize;
> + void *vaddr;
> + int ret;
> +
> + if (vhost_vdpa_listener_skipped_section(section)) {
> + return;
> + }
> +
> + if (unlikely((section->offset_within_address_space & ~TARGET_PAGE_MASK) !=
> + (section->offset_within_region & ~TARGET_PAGE_MASK))) {
> + error_report("%s received unaligned region", __func__);
> + return;
> + }
> +
> + iova = TARGET_PAGE_ALIGN(section->offset_within_address_space);
> + llend = int128_make64(section->offset_within_address_space);
> + llend = int128_add(llend, section->size);
> + llend = int128_and(llend, int128_exts64(TARGET_PAGE_MASK));
> +
> + if (int128_ge(int128_make64(iova), llend)) {
> + return;
> + }
> +
> + memory_region_ref(section->mr);
> +
> + /* Here we assume that memory_region_is_ram(section->mr)==true */
> +
> + vaddr = memory_region_get_ram_ptr(section->mr) +
> + section->offset_within_region +
> + (iova - section->offset_within_address_space);
> +
> + llsize = int128_sub(llend, int128_make64(iova));
> +
> + ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize),
> + vaddr, section->readonly);
> + if (ret) {
> + error_report("vhost vdpa map fail!");
> + if (memory_region_is_ram_device(section->mr)) {
> + /* Allow unexpected mappings not to be fatal for RAM devices */
> + error_report("map ram fail!");
> + return ;
> + }
> + goto fail;
> + }
> +
> + return;
> +
> +fail:
> + if (memory_region_is_ram_device(section->mr)) {
> + error_report("failed to vdpa_dma_map. pci p2p may not work");
> + return;
> +
> + }
> + /*
> + * On the initfn path, store the first error in the container so we
> + * can gracefully fail. Runtime, there's not much we can do other
> + * than throw a hardware error.
> + */
> + error_report("vhost-vdpa: DMA mapping failed, unable to continue");
> + return;
> +
> +}
> +
> +static void vhost_vdpa_listener_region_del(MemoryListener *listener,
> + MemoryRegionSection *section)
> +{
> + struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
> + hwaddr iova;
> + Int128 llend, llsize;
> + int ret;
> + bool try_unmap = true;
> +
> + if (vhost_vdpa_listener_skipped_section(section)) {
> + return;
> + }
> +
> + if (unlikely((section->offset_within_address_space & ~TARGET_PAGE_MASK) !=
> + (section->offset_within_region & ~TARGET_PAGE_MASK))) {
> + error_report("%s received unaligned region", __func__);
> + return;
> + }
> +
> + iova = TARGET_PAGE_ALIGN(section->offset_within_address_space);
> + llend = int128_make64(section->offset_within_address_space);
> + llend = int128_add(llend, section->size);
> + llend = int128_and(llend, int128_exts64(TARGET_PAGE_MASK));
> +
> + if (int128_ge(int128_make64(iova), llend)) {
> + return;
> + }
> +
> + llsize = int128_sub(llend, int128_make64(iova));
> +
> + if (try_unmap) {
> + ret = vhost_vdpa_dma_unmap(v, iova, int128_get64(llsize));
> + if (ret) {
> + error_report("vhost_vdpa dma unmap error!");
> + }
> + }
> +
> + memory_region_unref(section->mr);
> +}
> +/*
> + * IOTLB API is used by vhost-vpda which requires incremental updating
> + * of the mapping. So we can not use generic vhost memory listener which
> + * depends on the addnop().
> + */
> +static const MemoryListener vhost_vdpa_memory_listener = {
> + .region_add = vhost_vdpa_listener_region_add,
> + .region_del = vhost_vdpa_listener_region_del,
> +};
> +
> +static int vhost_vdpa_call(struct vhost_dev *dev, unsigned long int request,
> + void *arg)
> +{
> + struct vhost_vdpa *v = dev->opaque;
> + int fd = v->device_fd;
> +
> + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
> +
> + return ioctl(fd, request, arg);
> +}
> +
> +static void vhost_vdpa_add_status(struct vhost_dev *dev, uint8_t status)
> +{
> + uint8_t s;
> +
> + if (vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &s)) {
> + return;
> + }
> +
> + s |= status;
> +
> + vhost_vdpa_call(dev, VHOST_VDPA_SET_STATUS, &s);
> +}
> +
> +static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque)
> +{
> + struct vhost_vdpa *v;
> + uint64_t features;
> + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
> +
> + v = opaque;
> + dev->opaque = opaque ;
> + vhost_vdpa_call(dev, VHOST_GET_FEATURES, &features);
> + dev->backend_features = features;
> + v->listener = vhost_vdpa_memory_listener;
> + v->msg_type = VHOST_IOTLB_MSG_V2;
> +
> + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |
> + VIRTIO_CONFIG_S_DRIVER);
> +
> + return 0;
> +}
> +
> +static int vhost_vdpa_cleanup(struct vhost_dev *dev)
> +{
> + struct vhost_vdpa *v;
> + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
> + v = dev->opaque;
> + memory_listener_unregister(&v->listener);
> +
> + dev->opaque = NULL;
> + return 0;
> +}
> +
> +static int vhost_vdpa_memslots_limit(struct vhost_dev *dev)
> +{
> + return INT_MAX;
> +}
> +
> +static int vhost_vdpa_set_mem_table(struct vhost_dev *dev,
> + struct vhost_memory *mem)
> +{
> +
> + if (mem->padding) {
> + return -1;
> + }
> +
> + return 0;
> +}
> +
> +static int vhost_vdpa_set_features(struct vhost_dev *dev,
> + uint64_t features)
> +{
> + int ret;
> + ret = vhost_vdpa_call(dev, VHOST_SET_FEATURES, &features);
> + uint8_t status = 0;
> + if (ret) {
> + return ret;
> + }
> + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_FEATURES_OK);
> + vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status);
> +
> + return !(status & VIRTIO_CONFIG_S_FEATURES_OK);
> +}
> +
> +int vhost_vdpa_get_device_id(struct vhost_dev *dev,
> + uint32_t *device_id)
> +{
> + return vhost_vdpa_call(dev, VHOST_VDPA_GET_DEVICE_ID, device_id);
> +}
> +
> +static int vhost_vdpa_reset_device(struct vhost_dev *dev)
> +{
> + uint8_t status = 0;
> +
> + return vhost_vdpa_call(dev, VHOST_VDPA_SET_STATUS, &status);
> +}
> +
> +static int vhost_vdpa_get_vq_index(struct vhost_dev *dev, int idx)
> +{
> + assert(idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs);
> +
> + return idx - dev->vq_index;
> +}
> +
> +static int vhost_vdpa_set_vring_ready(struct vhost_dev *dev)
> +{
> + int i;
> + for (i = 0; i < dev->nvqs; ++i) {
> + struct vhost_vring_state state = {
> + .index = dev->vq_index + i,
> + .num = 1,
> + };
> + vhost_vdpa_call(dev, VHOST_VDPA_SET_VRING_ENABLE, &state);
> + }
> + return 0;
> +}
> +
> +static int vhost_vdpa_set_config(struct vhost_dev *dev, const uint8_t *data,
> + uint32_t offset, uint32_t size,
> + uint32_t flags)
> +{
> + struct vhost_vdpa_config *config;
> + int ret;
> + unsigned long config_size = offsetof(struct vhost_vdpa_config, buf);
> + config = g_malloc(size + config_size);
> + if (config == NULL) {
> + return -1;
> + }
> + config->off = offset;
> + config->len = size;
> + memcpy(config->buf, data, size);
> + ret = vhost_vdpa_call(dev, VHOST_VDPA_SET_CONFIG, config);
> + g_free(config);
> + return ret;
> +}
> +
> +static int vhost_vdpa_get_config(struct vhost_dev *dev, uint8_t *config,
> + uint32_t config_len)
> +{
> + struct vhost_vdpa_config *v_config;
> + unsigned long config_size = offsetof(struct vhost_vdpa_config, buf);
> + int ret;
> +
> + v_config = g_malloc(config_len + config_size);
> + if (v_config == NULL) {
> + return -1;
> + }
> + v_config->len = config_len;
> + v_config->off = 0;
> + ret = vhost_vdpa_call(dev, VHOST_VDPA_GET_CONFIG, v_config);
> + memcpy(config, v_config->buf, config_len);
> + g_free(v_config);
> + return ret;
> + }
> +
> +static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started)
> +{
> + struct vhost_vdpa *v = dev->opaque;
> + if (started) {
> + uint8_t status = 0;
> + memory_listener_register(&v->listener, &address_space_memory);
> + vhost_vdpa_set_vring_ready(dev);
> + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK);
> + vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status);
> +
> + return !(status & VIRTIO_CONFIG_S_DRIVER_OK);
> + } else {
> + vhost_vdpa_reset_device(dev);
> + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |
> + VIRTIO_CONFIG_S_DRIVER);
> + memory_listener_unregister(&v->listener);
> +
> + return 0;
> + }
> +}
> +
> +static int vhost_vdpa_set_log_base(struct vhost_dev *dev, uint64_t base,
> + struct vhost_log *log)
> +{
> + return vhost_vdpa_call(dev, VHOST_SET_LOG_BASE, &base);
> +}
> +
> +static int vhost_vdpa_set_vring_addr(struct vhost_dev *dev,
> + struct vhost_vring_addr *addr)
> +{
> + return vhost_vdpa_call(dev, VHOST_SET_VRING_ADDR, addr);
> +}
> +
> +static int vhost_vdpa_set_vring_num(struct vhost_dev *dev,
> + struct vhost_vring_state *ring)
> +{
> + return vhost_vdpa_call(dev, VHOST_SET_VRING_NUM, ring);
> +}
> +
> +static int vhost_vdpa_set_vring_base(struct vhost_dev *dev,
> + struct vhost_vring_state *ring)
> +{
> + return vhost_vdpa_call(dev, VHOST_SET_VRING_BASE, ring);
> +}
> +
> +static int vhost_vdpa_get_vring_base(struct vhost_dev *dev,
> + struct vhost_vring_state *ring)
> +{
> + return vhost_vdpa_call(dev, VHOST_GET_VRING_BASE, ring);
> +}
> +
> +static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev,
> + struct vhost_vring_file *file)
> +{
> + return vhost_vdpa_call(dev, VHOST_SET_VRING_KICK, file);
> +}
> +
> +static int vhost_vdpa_set_vring_call(struct vhost_dev *dev,
> + struct vhost_vring_file *file)
> +{
> + return vhost_vdpa_call(dev, VHOST_SET_VRING_CALL, file);
> +}
> +
> +static int vhost_vdpa_get_features(struct vhost_dev *dev,
> + uint64_t *features)
> +{
> + return vhost_vdpa_call(dev, VHOST_GET_FEATURES, features);
> +}
> +
> +static int vhost_vdpa_set_owner(struct vhost_dev *dev)
> +{
> + return vhost_vdpa_call(dev, VHOST_SET_OWNER, NULL);
> +}
> +
> +static int vhost_vdpa_vq_get_addr(struct vhost_dev *dev,
> + struct vhost_vring_addr *addr, struct vhost_virtqueue *vq)
> +{
> + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
> + addr->desc_user_addr = (uint64_t)(unsigned long)vq->desc_phys;
> + addr->avail_user_addr = (uint64_t)(unsigned long)vq->avail_phys;
> + addr->used_user_addr = (uint64_t)(unsigned long)vq->used_phys;
> + return 0;
> +}
> +
> +const VhostOps vdpa_ops = {
> + .backend_type = VHOST_BACKEND_TYPE_VDPA,
> + .vhost_backend_init = vhost_vdpa_init,
> + .vhost_backend_cleanup = vhost_vdpa_cleanup,
> + .vhost_set_log_base = vhost_vdpa_set_log_base,
> + .vhost_set_vring_addr = vhost_vdpa_set_vring_addr,
> + .vhost_set_vring_num = vhost_vdpa_set_vring_num,
> + .vhost_set_vring_base = vhost_vdpa_set_vring_base,
> + .vhost_get_vring_base = vhost_vdpa_get_vring_base,
> + .vhost_set_vring_kick = vhost_vdpa_set_vring_kick,
> + .vhost_set_vring_call = vhost_vdpa_set_vring_call,
> + .vhost_get_features = vhost_vdpa_get_features,
> + .vhost_set_owner = vhost_vdpa_set_owner,
> + .vhost_set_vring_endian = NULL,
> + .vhost_backend_memslots_limit = vhost_vdpa_memslots_limit,
> + .vhost_set_mem_table = vhost_vdpa_set_mem_table,
> + .vhost_set_features = vhost_vdpa_set_features,
> + .vhost_reset_device = vhost_vdpa_reset_device,
> + .vhost_get_vq_index = vhost_vdpa_get_vq_index,
> + .vhost_get_config = vhost_vdpa_get_config,
> + .vhost_set_config = vhost_vdpa_set_config,
> + .vhost_requires_shm_log = NULL,
> + .vhost_migration_done = NULL,
> + .vhost_backend_can_merge = NULL,
> + .vhost_net_set_mtu = NULL,
> + .vhost_set_iotlb_callback = NULL,
> + .vhost_send_device_iotlb_msg = NULL,
> + .vhost_dev_start = vhost_vdpa_dev_start,
> + .vhost_get_device_id = vhost_vdpa_get_device_id,
> + .vhost_vq_get_addr = vhost_vdpa_vq_get_addr,
> +};
> diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> index 1e083a8976..bcaa66e0d2 100644
> --- a/hw/virtio/vhost.c
> +++ b/hw/virtio/vhost.c
> @@ -808,7 +808,9 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
> features |= 0x1ULL << VHOST_F_LOG_ALL;
> }
> if (!vhost_dev_has_iommu(dev)) {
> - features &= ~(0x1ULL << VIRTIO_F_IOMMU_PLATFORM);
> + if (dev->vhost_ops->backend_type != VHOST_BACKEND_TYPE_VDPA) {
> + features &= ~(0x1ULL << VIRTIO_F_IOMMU_PLATFORM);
> + }
Let's add a vhost_ops here if we want to force IOMMU_PLATFORM.
> }
> r = dev->vhost_ops->vhost_set_features(dev, features);
> if (r < 0) {
> @@ -1701,9 +1703,10 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
> goto fail_log;
> }
> }
> +
Unnecessary change.
> if (vhost_dev_has_iommu(hdev) &&
> hdev->vhost_ops->vhost_set_iotlb_callback) {
> - hdev->vhost_ops->vhost_set_iotlb_callback(hdev, true);
> + hdev->vhost_ops->vhost_set_iotlb_callback(hdev, true);
This change should belong to patch 5
Thanks
>
> /* Update used ring information for IOTLB to work correctly,
> * vhost-kernel code requires for this.*/
> diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
> index bfc24207e2..01f7c0d46f 100644
> --- a/include/hw/virtio/vhost-backend.h
> +++ b/include/hw/virtio/vhost-backend.h
> @@ -17,7 +17,8 @@ typedef enum VhostBackendType {
> VHOST_BACKEND_TYPE_NONE = 0,
> VHOST_BACKEND_TYPE_KERNEL = 1,
> VHOST_BACKEND_TYPE_USER = 2,
> - VHOST_BACKEND_TYPE_MAX = 3,
> + VHOST_BACKEND_TYPE_VDPA = 3,
> + VHOST_BACKEND_TYPE_MAX = 4,
> } VhostBackendType;
>
> typedef enum VhostSetConfigType {
> @@ -167,6 +168,7 @@ typedef struct VhostOps {
> } VhostOps;
>
> extern const VhostOps user_ops;
> +extern const VhostOps vdpa_ops;
>
> int vhost_set_backend_type(struct vhost_dev *dev,
> VhostBackendType backend_type);
> diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
> new file mode 100644
> index 0000000000..6455663388
> --- /dev/null
> +++ b/include/hw/virtio/vhost-vdpa.h
> @@ -0,0 +1,26 @@
> +/*
> + * vhost-vdpa.h
> + *
> + * Copyright(c) 2017-2018 Intel Corporation.
> + * Copyright(c) 2020 Red Hat, Inc.
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#ifndef HW_VIRTIO_VHOST_VDPA_H
> +#define HW_VIRTIO_VHOST_VDPA_H
> +
> +#include "hw/virtio/virtio.h"
> +
> +typedef struct vhost_vdpa {
> + int device_fd;
> + uint32_t msg_type;
> + MemoryListener listener;
> +} VhostVDPA;
> +
> +extern AddressSpace address_space_memory;
> +extern int vhost_vdpa_get_device_id(struct vhost_dev *dev,
> + uint32_t *device_id);
> +#endif
> diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
> index 085450c6f8..767a95ec0b 100644
> --- a/include/hw/virtio/vhost.h
> +++ b/include/hw/virtio/vhost.h
> @@ -92,6 +92,13 @@ struct vhost_dev {
> const VhostDevConfigOps *config_ops;
> };
>
> +struct vhost_net {
> + struct vhost_dev dev;
> + struct vhost_virtqueue vqs[2];
> + int backend;
> + NetClientState *nc;
> +};
> +
> int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
> VhostBackendType backend_type,
> uint32_t busyloop_timeout);
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 196f468786..fa1b19de4c 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -2418,6 +2418,10 @@ DEF("netdev", HAS_ARG, QEMU_OPTION_netdev,
> #ifdef CONFIG_POSIX
> "-netdev vhost-user,id=str,chardev=dev[,vhostforce=on|off]\n"
> " configure a vhost-user network, backed by a chardev 'dev'\n"
> +#endif
> +#ifdef __linux__
> + "-netdev vhost-vdpa,id=str,vhostdev=/path/to/dev\n"
> + " configure a vhost-vdpa network,Establish a vhost-vdpa netdev\n"
> #endif
> "-netdev hubport,id=str,hubid=n[,netdev=nd]\n"
> " configure a hub port on the hub with ID 'n'\n", QEMU_ARCH_ALL)
> @@ -2897,6 +2901,14 @@ SRST
> -netdev type=vhost-user,id=net0,chardev=chr0 \
> -device virtio-net-pci,netdev=net0
>
> +``-netdev vhost-vdpa,vhostdev=/path/to/dev``
> + Establish a vhost-vdpa netdev.
> +
> + vDPA device is a device that uses a datapath which complies with
> + the virtio specifications with a vendor specific control path.
> + vDPA devices can be both physically located on the hardware or
> + emulated by software.
> +
> ``-netdev hubport,id=id,hubid=hubid[,netdev=nd]``
> Create a hub port on the emulated hub with ID hubid.
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2 00/12] vDPA support in qemu
2020-06-30 18:06 ` [PATCH v2 00/12] vDPA support in qemu no-reply
@ 2020-07-01 5:29 ` Jason Wang
2020-07-01 5:30 ` Cindy Lu
0 siblings, 1 reply; 20+ messages in thread
From: Jason Wang @ 2020-07-01 5:29 UTC (permalink / raw)
To: qemu-devel, lulu
Cc: rdunlap, mst, mhabets, rob.miller, saugatm, armbru, hch,
eperezma, jgg, shahafs, kevin.tian, parav, vmireyno,
cunming.liang, gdawar, jiri, xiao.w.wang, stefanha, zhihong.wang,
maxime.coquelin, aadam, cohuck, hanand, lingshan.zhu
On 2020/7/1 上午2:06, no-reply@patchew.org wrote:
> Patchew URL: https://patchew.org/QEMU/20200630174937.25560-1-lulu@redhat.com/
>
>
>
> Hi,
>
> This series failed the docker-mingw@fedora build test. Please find the testing commands and
> their output below. If you have Docker installed, you can probably reproduce it
> locally.
>
> === TEST SCRIPT BEGIN ===
> #! /bin/bash
> export ARCH=x86_64
> make docker-image-fedora V=1 NETWORK=1
> time make docker-test-mingw@fedora J=14 NETWORK=1
Please fix this. You can reproduce this with the above commands.
Thanks
> === TEST SCRIPT END ===
>
> /tmp/qemu-test/src/hw/net/virtio-net.c:173: undefined reference to `vhost_dev_set_config'
> /usr/lib/gcc/x86_64-w64-mingw32/9.2.1/../../../../x86_64-w64-mingw32/bin/ld: hw/net/virtio-net.o: in function `virtio_net_get_config':
> /tmp/qemu-test/src/hw/net/virtio-net.c:149: undefined reference to `vhost_dev_get_config'
> collect2: error: ld returned 1 exit status
> make[1]: *** [Makefile:208: qemu-system-x86_64w.exe] Error 1
> make: *** [Makefile:527: x86_64-softmmu/all] Error 2
> make: *** Waiting for unfinished jobs....
> LINK aarch64-softmmu/qemu-system-aarch64w.exe
> /usr/lib/gcc/x86_64-w64-mingw32/9.2.1/../../../../x86_64-w64-mingw32/bin/ld: hw/net/virtio-net.o: in function `virtio_net_set_config':
> /tmp/qemu-test/src/hw/net/virtio-net.c:173: undefined reference to `vhost_dev_set_config'
> /usr/lib/gcc/x86_64-w64-mingw32/9.2.1/../../../../x86_64-w64-mingw32/bin/ld: hw/net/virtio-net.o: in function `virtio_net_get_config':
> /tmp/qemu-test/src/hw/net/virtio-net.c:149: undefined reference to `vhost_dev_get_config'
> collect2: error: ld returned 1 exit status
> make[1]: *** [Makefile:208: qemu-system-aarch64w.exe] Error 1
> make: *** [Makefile:527: aarch64-softmmu/all] Error 2
> Traceback (most recent call last):
> File "./tests/docker/docker.py", line 669, in <module>
> sys.exit(main())
> ---
> raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=9f5def1ac1a840deb219e8827020f24d', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-lwyumgcs/src/docker-src.2020-06-30-14.01.04.32102:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
> filter=--filter=label=com.qemu.instance.uuid=9f5def1ac1a840deb219e8827020f24d
> make[1]: *** [docker-run] Error 1
> make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-lwyumgcs/src'
> make: *** [docker-run-test-mingw@fedora] Error 2
>
> real 5m52.723s
> user 0m8.824s
>
>
> The full log is available at
> http://patchew.org/logs/20200630174937.25560-1-lulu@redhat.com/testing.docker-mingw@fedora/?type=message.
> ---
> Email generated automatically by Patchew [https://patchew.org/].
> Please send your feedback to patchew-devel@redhat.com
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2 00/12] vDPA support in qemu
2020-07-01 5:29 ` Jason Wang
@ 2020-07-01 5:30 ` Cindy Lu
0 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-07-01 5:30 UTC (permalink / raw)
To: Jason Wang
Cc: rdunlap, Michael Tsirkin, mhabets, qemu-devel, Rob Miller,
saugatm, Markus Armbruster, hch, Eugenio Perez Martin, jgg,
Shahaf Shuler, kevin.tian, parav, Vitaly Mireyno, Liang, Cunming,
gdawar, jiri, xiao.w.wang, Stefan Hajnoczi, Wang, Zhihong,
Maxime Coquelin, Ariel Adam, Cornelia Huck, hanand, Zhu,
Lingshan
Thanks Jason, I'm working in this, Will update a new version soon
On Wed, Jul 1, 2020 at 1:29 PM Jason Wang <jasowang@redhat.com> wrote:
>
>
> On 2020/7/1 上午2:06, no-reply@patchew.org wrote:
> > Patchew URL: https://patchew.org/QEMU/20200630174937.25560-1-lulu@redhat.com/
> >
> >
> >
> > Hi,
> >
> > This series failed the docker-mingw@fedora build test. Please find the testing commands and
> > their output below. If you have Docker installed, you can probably reproduce it
> > locally.
> >
> > === TEST SCRIPT BEGIN ===
> > #! /bin/bash
> > export ARCH=x86_64
> > make docker-image-fedora V=1 NETWORK=1
> > time make docker-test-mingw@fedora J=14 NETWORK=1
>
>
> Please fix this. You can reproduce this with the above commands.
>
> Thanks
>
>
> > === TEST SCRIPT END ===
> >
> > /tmp/qemu-test/src/hw/net/virtio-net.c:173: undefined reference to `vhost_dev_set_config'
> > /usr/lib/gcc/x86_64-w64-mingw32/9.2.1/../../../../x86_64-w64-mingw32/bin/ld: hw/net/virtio-net.o: in function `virtio_net_get_config':
> > /tmp/qemu-test/src/hw/net/virtio-net.c:149: undefined reference to `vhost_dev_get_config'
> > collect2: error: ld returned 1 exit status
> > make[1]: *** [Makefile:208: qemu-system-x86_64w.exe] Error 1
> > make: *** [Makefile:527: x86_64-softmmu/all] Error 2
> > make: *** Waiting for unfinished jobs....
> > LINK aarch64-softmmu/qemu-system-aarch64w.exe
> > /usr/lib/gcc/x86_64-w64-mingw32/9.2.1/../../../../x86_64-w64-mingw32/bin/ld: hw/net/virtio-net.o: in function `virtio_net_set_config':
> > /tmp/qemu-test/src/hw/net/virtio-net.c:173: undefined reference to `vhost_dev_set_config'
> > /usr/lib/gcc/x86_64-w64-mingw32/9.2.1/../../../../x86_64-w64-mingw32/bin/ld: hw/net/virtio-net.o: in function `virtio_net_get_config':
> > /tmp/qemu-test/src/hw/net/virtio-net.c:149: undefined reference to `vhost_dev_get_config'
> > collect2: error: ld returned 1 exit status
> > make[1]: *** [Makefile:208: qemu-system-aarch64w.exe] Error 1
> > make: *** [Makefile:527: aarch64-softmmu/all] Error 2
> > Traceback (most recent call last):
> > File "./tests/docker/docker.py", line 669, in <module>
> > sys.exit(main())
> > ---
> > raise CalledProcessError(retcode, cmd)
> > subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=9f5def1ac1a840deb219e8827020f24d', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-lwyumgcs/src/docker-src.2020-06-30-14.01.04.32102:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
> > filter=--filter=label=com.qemu.instance.uuid=9f5def1ac1a840deb219e8827020f24d
> > make[1]: *** [docker-run] Error 1
> > make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-lwyumgcs/src'
> > make: *** [docker-run-test-mingw@fedora] Error 2
> >
> > real 5m52.723s
> > user 0m8.824s
> >
> >
> > The full log is available at
> > http://patchew.org/logs/20200630174937.25560-1-lulu@redhat.com/testing.docker-mingw@fedora/?type=message.
> > ---
> > Email generated automatically by Patchew [https://patchew.org/].
> > Please send your feedback to patchew-devel@redhat.com
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2 11/12] vhost-vdpa: introduce vhost-vdpa backend
2020-07-01 4:21 ` Jason Wang
@ 2020-07-01 9:17 ` Cindy Lu
0 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-07-01 9:17 UTC (permalink / raw)
To: Jason Wang
Cc: Cornelia Huck, Michael Tsirkin, mhabets, qemu-devel, hanand,
Rob Miller, saugatm, Markus Armbruster, hch,
Eugenio Perez Martin, jgg, Shahaf Shuler, kevin.tian, parav,
Vitaly Mireyno, Liang, Cunming, gdawar, jiri, xiao.w.wang,
Stefan Hajnoczi, Wang, Zhihong, Tiwei Bie, Ariel Adam, rdunlap,
Maxime Coquelin, Zhu, Lingshan
Thanks Jason, will fix all these and send the new version soon
On Wed, Jul 1, 2020 at 12:22 PM Jason Wang <jasowang@redhat.com> wrote:
>
>
> On 2020/7/1 上午1:49, Cindy Lu wrote:
> > Currently we have 2 types of vhost backends in QEMU: vhost kernel and
> > vhost-user. The above patch provides a generic device for vDPA purpose,
> > this vDPA device exposes to user space a non-vendor-specific configuration
> > interface for setting up a vhost HW accelerator, this patch set introduces
> > a third vhost backend called vhost-vdpa based on the vDPA interface.
> >
> > Vhost-vdpa usage:
> >
> > qemu-system-x86_64 -cpu host -enable-kvm \
> > ......
> > -netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-id,id=vhost-vdpa0 \
> > -device virtio-net-pci,netdev=vhost-vdpa0,page-per-vq=on \
> >
> > Signed-off-by: Lingshan zhu <lingshan.zhu@intel.com>
> > Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
>
>
> I would like to add my sob here.
>
>
> > Signed-off-by: Cindy Lu <lulu@redhat.com>
> > ---
> > configure | 21 ++
> > docs/interop/index.rst | 1 +
> > docs/interop/vhost-vdpa.rst | 17 ++
> > hw/net/vhost_net.c | 19 +-
> > hw/net/virtio-net.c | 22 +-
> > hw/virtio/Makefile.objs | 1 +
> > hw/virtio/vhost-backend.c | 7 +-
> > hw/virtio/vhost-vdpa.c | 469 ++++++++++++++++++++++++++++++
> > hw/virtio/vhost.c | 7 +-
> > include/hw/virtio/vhost-backend.h | 4 +-
> > include/hw/virtio/vhost-vdpa.h | 26 ++
> > include/hw/virtio/vhost.h | 7 +
> > qemu-options.hx | 12 +
> > 13 files changed, 601 insertions(+), 12 deletions(-)
> > create mode 100644 docs/interop/vhost-vdpa.rst
> > create mode 100644 hw/virtio/vhost-vdpa.c
> > create mode 100644 include/hw/virtio/vhost-vdpa.h
> >
> > diff --git a/configure b/configure
> > index 4a22dcd563..3db7f20185 100755
> > --- a/configure
> > +++ b/configure
> > @@ -1575,6 +1575,10 @@ for opt do
> > ;;
> > --enable-vhost-user) vhost_user="yes"
> > ;;
> > + --disable-vhost-vdpa) vhost_vdpa="no"
> > + ;;
> > + --enable-vhost-vdpa) vhost_vdpa="yes"
> > + ;;
> > --disable-vhost-kernel) vhost_kernel="no"
> > ;;
> > --enable-vhost-kernel) vhost_kernel="yes"
> > @@ -1883,6 +1887,7 @@ disabled with --disable-FEATURE, default is enabled if available:
> > vhost-crypto vhost-user-crypto backend support
> > vhost-kernel vhost kernel backend support
> > vhost-user vhost-user backend support
> > + vhost-vdpa vhost-vdpa kernel backend support
> > spice spice
> > rbd rados block device (rbd)
> > libiscsi iscsi support
> > @@ -2394,6 +2399,10 @@ test "$vhost_user" = "" && vhost_user=yes
> > if test "$vhost_user" = "yes" && test "$mingw32" = "yes"; then
> > error_exit "vhost-user isn't available on win32"
> > fi
> > +test "$vhost_vdpa" = "" && vhost_vdpa=$linux
> > +if test "$vhost_vdpa" = "yes" && test "$linux" != "yes"; then
> > + error_exit "vhost-vdpa is only available on Linux"
> > +fi
> > test "$vhost_kernel" = "" && vhost_kernel=$linux
> > if test "$vhost_kernel" = "yes" && test "$linux" != "yes"; then
> > error_exit "vhost-kernel is only available on Linux"
> > @@ -2422,6 +2431,11 @@ test "$vhost_user_fs" = "" && vhost_user_fs=$vhost_user
> > if test "$vhost_user_fs" = "yes" && test "$vhost_user" = "no"; then
> > error_exit "--enable-vhost-user-fs requires --enable-vhost-user"
> > fi
> > +#vhost-vdpa backends
> > +test "$vhost_net_vdpa" = "" && vhost_net_vdpa=$vhost_vdpa
> > +if test "$vhost_net_vdpa" = "yes" && test "$vhost_vdpa" = "no"; then
> > + error_exit "--enable-vhost-net-vdpa requires --enable-vhost-vdpa"
> > +fi
> >
> > # OR the vhost-kernel and vhost-user values for simplicity
> > if test "$vhost_net" = ""; then
> > @@ -6936,6 +6950,7 @@ echo "vhost-scsi support $vhost_scsi"
> > echo "vhost-vsock support $vhost_vsock"
> > echo "vhost-user support $vhost_user"
> > echo "vhost-user-fs support $vhost_user_fs"
> > +echo "vhost-vdpa support $vhost_vdpa"
> > echo "Trace backends $trace_backends"
> > if have_backend "simple"; then
> > echo "Trace output file $trace_file-<pid>"
> > @@ -7437,6 +7452,9 @@ fi
> > if test "$vhost_net_user" = "yes" ; then
> > echo "CONFIG_VHOST_NET_USER=y" >> $config_host_mak
> > fi
> > +if test "$vhost_net_vdpa" = "yes" ; then
> > + echo "CONFIG_VHOST_NET_VDPA=y" >> $config_host_mak
> > +fi
> > if test "$vhost_crypto" = "yes" ; then
> > echo "CONFIG_VHOST_CRYPTO=y" >> $config_host_mak
> > fi
> > @@ -7452,6 +7470,9 @@ fi
> > if test "$vhost_user" = "yes" ; then
> > echo "CONFIG_VHOST_USER=y" >> $config_host_mak
> > fi
> > +if test "$vhost_vdpa" = "yes" ; then
> > + echo "CONFIG_VHOST_VDPA=y" >> $config_host_mak
> > +fi
> > if test "$vhost_user_fs" = "yes" ; then
> > echo "CONFIG_VHOST_USER_FS=y" >> $config_host_mak
> > fi
> > diff --git a/docs/interop/index.rst b/docs/interop/index.rst
> > index 049387ac6d..006f986420 100644
> > --- a/docs/interop/index.rst
> > +++ b/docs/interop/index.rst
> > @@ -20,3 +20,4 @@ Contents:
> > qemu-ga
> > vhost-user
> > vhost-user-gpu
> > + vhost-vdpa
> > diff --git a/docs/interop/vhost-vdpa.rst b/docs/interop/vhost-vdpa.rst
> > new file mode 100644
> > index 0000000000..5a075b8a63
> > --- /dev/null
> > +++ b/docs/interop/vhost-vdpa.rst
> > @@ -0,0 +1,17 @@
> > +=====================
> > +Vhost-vdpa Protocol
> > +=====================
> > +
> > +Introduction
> > +=============
> > +vDPA(Virtual data path acceleration) device is a device that uses
> > +a datapath which complies with the virtio specifications with vendor
> > +specific control path. vDPA devices can be both physically located on
> > +the hardware or emulated by software.
> > +
> > +This document describes the vDPA support in qemu
> > +
> > +Here is the kernel commit here
> > +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4c8cf31885f69e86be0b5b9e6677a26797365e1d
> > +
> > +TODO : Moreinfo will add later
> > diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
> > index 4096d64aaf..cbdf42ace5 100644
> > --- a/hw/net/vhost_net.c
> > +++ b/hw/net/vhost_net.c
> > @@ -17,8 +17,10 @@
> > #include "net/net.h"
> > #include "net/tap.h"
> > #include "net/vhost-user.h"
> > +#include "net/vhost-vdpa.h"
> >
> > #include "standard-headers/linux/vhost_types.h"
> > +#include "linux-headers/linux/vhost.h"
> > #include "hw/virtio/virtio-net.h"
> > #include "net/vhost_net.h"
> > #include "qemu/error-report.h"
> > @@ -33,12 +35,6 @@
> > #include "hw/virtio/vhost.h"
> > #include "hw/virtio/virtio-bus.h"
> >
> > -struct vhost_net {
> > - struct vhost_dev dev;
> > - struct vhost_virtqueue vqs[2];
> > - int backend;
> > - NetClientState *nc;
> > -};
> >
> > /* Features supported by host kernel. */
> > static const int kernel_feature_bits[] = {
> > @@ -96,6 +92,9 @@ static const int *vhost_net_get_feature_bits(struct vhost_net *net)
> > case NET_CLIENT_DRIVER_VHOST_USER:
> > feature_bits = user_feature_bits;
> > break;
> > + case NET_CLIENT_DRIVER_VHOST_VDPA:
> > + feature_bits = vdpa_feature_bits;
> > + break;
> > default:
> > error_report("Feature bits not defined for this type: %d",
> > net->nc->info->type);
> > @@ -316,7 +315,6 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs,
> > }
> >
> > for (i = 0; i < total_queues; i++) {
> > -
>
>
> Unnecessary change.
>
>
> > peer = qemu_get_peer(ncs, i);
> > net = get_vhost_net(peer);
> > vhost_net_set_vq_index(net, i * 2);
> > @@ -337,6 +335,7 @@ int vhost_net_start(VirtIODevice *dev, NetClientState *ncs,
> > }
> >
> > for (i = 0; i < total_queues; i++) {
> > +
>
>
> Unnecessary change.
>
>
> > peer = qemu_get_peer(ncs, i);
> > r = vhost_net_start_one(get_vhost_net(peer), dev);
> >
> > @@ -433,6 +432,12 @@ VHostNetState *get_vhost_net(NetClientState *nc)
> > vhost_net = vhost_user_get_vhost_net(nc);
> > assert(vhost_net);
> > break;
> > +#endif
> > +#ifdef CONFIG_VHOST_NET_VDPA
> > + case NET_CLIENT_DRIVER_VHOST_VDPA:
> > + vhost_net = vhost_vdpa_get_vhost_net(nc);
> > + assert(vhost_net);
> > + break;
> > #endif
> > default:
> > break;
> > diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> > index aff67a92df..6aadc2403a 100644
> > --- a/hw/net/virtio-net.c
> > +++ b/hw/net/virtio-net.c
> > @@ -43,6 +43,7 @@
> > #include "monitor/qdev.h"
> > #include "hw/pci/pci.h"
> > #include "net_rx_pkt.h"
> > +#include "hw/virtio/vhost.h"
> >
> > #define VIRTIO_NET_VM_VERSION 11
> >
> > @@ -124,7 +125,10 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
> > {
> > VirtIONet *n = VIRTIO_NET(vdev);
> > struct virtio_net_config netcfg;
> > + int ret = 0;
> > + struct vhost_dev *hdev = NULL;
> >
> > + memset(&netcfg, 0 , sizeof(struct virtio_net_config));
> > virtio_stw_p(vdev, &netcfg.status, n->status);
> > virtio_stw_p(vdev, &netcfg.max_virtqueue_pairs, n->max_queues);
> > virtio_stw_p(vdev, &netcfg.mtu, n->net_conf.mtu);
> > @@ -138,21 +142,37 @@ static void virtio_net_get_config(VirtIODevice *vdev, uint8_t *config)
> > virtio_stl_p(vdev, &netcfg.supported_hash_types,
> > VIRTIO_NET_RSS_SUPPORTED_HASHES);
> > memcpy(config, &netcfg, n->config_size);
> > +
> > + NetClientState *nc = qemu_get_queue(n->nic);
> > + if (nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
> > + hdev = &(get_vhost_net(nc->peer)->dev);
> > + ret = vhost_dev_get_config(hdev, (uint8_t *)&netcfg, n->config_size);
> > + if (ret != -1) {
> > + memcpy(config, &netcfg, n->config_size);
> > + }
> > + }
> > }
> >
> > static void virtio_net_set_config(VirtIODevice *vdev, const uint8_t *config)
> > {
> > VirtIONet *n = VIRTIO_NET(vdev);
> > struct virtio_net_config netcfg = {};
> > + struct vhost_dev *hdev = NULL;
> >
> > memcpy(&netcfg, config, n->config_size);
> > -
>
>
> Unnecessary change.
>
>
> > if (!virtio_vdev_has_feature(vdev, VIRTIO_NET_F_CTRL_MAC_ADDR) &&
> > !virtio_vdev_has_feature(vdev, VIRTIO_F_VERSION_1) &&
> > memcmp(netcfg.mac, n->mac, ETH_ALEN)) {
> > memcpy(n->mac, netcfg.mac, ETH_ALEN);
> > qemu_format_nic_info_str(qemu_get_queue(n->nic), n->mac);
> > }
> > +
> > + NetClientState *nc = qemu_get_queue(n->nic);
> > + if (nc->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
> > + hdev = &(get_vhost_net(nc->peer)->dev);
> > + vhost_dev_set_config(hdev, (uint8_t *)&netcfg, 0, n->config_size,
> > + VHOST_SET_CONFIG_TYPE_MASTER);
> > + }
> > }
> >
> > static bool virtio_net_started(VirtIONet *n, uint8_t status)
> > diff --git a/hw/virtio/Makefile.objs b/hw/virtio/Makefile.objs
> > index 13e75f171f..2b31e9b3dc 100644
> > --- a/hw/virtio/Makefile.objs
> > +++ b/hw/virtio/Makefile.objs
> > @@ -5,6 +5,7 @@ obj-y += virtio.o
> > obj-$(CONFIG_VHOST) += vhost.o vhost-backend.o
> > common-obj-$(call lnot,$(CONFIG_VHOST)) += vhost-stub.o
> > obj-$(CONFIG_VHOST_USER) += vhost-user.o
> > +obj-$(CONFIG_VHOST_VDPA) += vhost-vdpa.o
> >
> > common-obj-$(CONFIG_VIRTIO_RNG) += virtio-rng.o
> > common-obj-$(CONFIG_VIRTIO_PCI) += virtio-pci.o
> > diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c
> > index 48905383f8..b925a2b1ac 100644
> > --- a/hw/virtio/vhost-backend.c
> > +++ b/hw/virtio/vhost-backend.c
> > @@ -14,7 +14,7 @@
> > #include "qemu/error-report.h"
> > #include "qemu/main-loop.h"
> > #include "standard-headers/linux/vhost_types.h"
> > -
> > +#include "hw/virtio/vhost-vdpa.h"
>
>
> Let's keep the newline here.
>
>
> > #ifdef CONFIG_VHOST_KERNEL
> > #include <linux/vhost.h>
> > #include <sys/ioctl.h>
> > @@ -285,6 +285,11 @@ int vhost_set_backend_type(struct vhost_dev *dev, VhostBackendType backend_type)
> > case VHOST_BACKEND_TYPE_USER:
> > dev->vhost_ops = &user_ops;
> > break;
> > +#endif
> > +#ifdef CONFIG_VHOST_VDPA
> > + case VHOST_BACKEND_TYPE_VDPA:
> > + dev->vhost_ops = &vdpa_ops;
> > + break;
> > #endif
> > default:
> > error_report("Unknown vhost backend type");
> > diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
> > new file mode 100644
> > index 0000000000..25c9b537b0
> > --- /dev/null
> > +++ b/hw/virtio/vhost-vdpa.c
> > @@ -0,0 +1,469 @@
> > +/*
> > + * vhost-vdpa
> > + *
> > + * Copyright(c) 2017-2018 Intel Corporation.
> > + * Copyright(c) 2020 Red Hat, Inc.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > + * See the COPYING file in the top-level directory.
> > + *
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +#include <linux/vhost.h>
> > +#include <linux/vfio.h>
> > +#include <sys/eventfd.h>
> > +#include <sys/ioctl.h>
> > +#include "hw/virtio/vhost.h"
> > +#include "hw/virtio/vhost-backend.h"
> > +#include "hw/virtio/virtio-net.h"
> > +#include "hw/virtio/vhost-vdpa.h"
> > +#include "qemu/main-loop.h"
> > +#include <linux/kvm.h>
> > +#include "sysemu/kvm.h"
> > +
> > +static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section)
> > +{
> > + return (!memory_region_is_ram(section->mr) &&
> > + !memory_region_is_iommu(section->mr)) ||
> > + /*
> > + * Sizing an enabled 64-bit BAR can cause spurious mappings to
> > + * addresses in the upper part of the 64-bit address space. These
> > + * are never accessed by the CPU and beyond the address width of
> > + * some IOMMU hardware. TODO: VDPA should tell us the IOMMU width.
> > + */
> > + section->offset_within_address_space & (1ULL << 63);
> > +}
> > +
> > +static int vhost_vdpa_dma_map(struct vhost_vdpa *v, hwaddr iova, hwaddr size,
> > + void *vaddr, bool readonly)
> > +{
> > + struct vhost_msg_v2 msg;
> > + int fd = v->device_fd;
> > + int ret = 0;
> > +
> > + msg.type = v->msg_type;
> > + msg.iotlb.iova = iova;
> > + msg.iotlb.size = size;
> > + msg.iotlb.uaddr = (uint64_t)vaddr;
> > + msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
> > + msg.iotlb.type = VHOST_IOTLB_UPDATE;
> > +
> > + if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
> > + error_report("failed to write, fd=%d, errno=%d (%s)",
> > + fd, errno, strerror(errno));
> > + return -EIO ;
> > + }
> > +
> > + return ret;
> > +}
> > +
> > +static int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, hwaddr iova,
> > + hwaddr size)
> > +{
> > + struct vhost_msg_v2 msg;
> > + int fd = v->device_fd;
> > + int ret = 0;
> > +
> > + msg.type = v->msg_type;
> > + msg.iotlb.iova = iova;
> > + msg.iotlb.size = size;
> > + msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
> > +
> > + if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
> > + error_report("failed to write, fd=%d, errno=%d (%s)",
> > + fd, errno, strerror(errno));
> > + return -EIO ;
> > + }
> > +
> > + return ret;
> > +}
> > +
> > +static void vhost_vdpa_listener_region_add(MemoryListener *listener,
> > + MemoryRegionSection *section)
> > +{
> > + struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
> > + hwaddr iova;
> > + Int128 llend, llsize;
> > + void *vaddr;
> > + int ret;
> > +
> > + if (vhost_vdpa_listener_skipped_section(section)) {
> > + return;
> > + }
> > +
> > + if (unlikely((section->offset_within_address_space & ~TARGET_PAGE_MASK) !=
> > + (section->offset_within_region & ~TARGET_PAGE_MASK))) {
> > + error_report("%s received unaligned region", __func__);
> > + return;
> > + }
> > +
> > + iova = TARGET_PAGE_ALIGN(section->offset_within_address_space);
> > + llend = int128_make64(section->offset_within_address_space);
> > + llend = int128_add(llend, section->size);
> > + llend = int128_and(llend, int128_exts64(TARGET_PAGE_MASK));
> > +
> > + if (int128_ge(int128_make64(iova), llend)) {
> > + return;
> > + }
> > +
> > + memory_region_ref(section->mr);
> > +
> > + /* Here we assume that memory_region_is_ram(section->mr)==true */
> > +
> > + vaddr = memory_region_get_ram_ptr(section->mr) +
> > + section->offset_within_region +
> > + (iova - section->offset_within_address_space);
> > +
> > + llsize = int128_sub(llend, int128_make64(iova));
> > +
> > + ret = vhost_vdpa_dma_map(v, iova, int128_get64(llsize),
> > + vaddr, section->readonly);
> > + if (ret) {
> > + error_report("vhost vdpa map fail!");
> > + if (memory_region_is_ram_device(section->mr)) {
> > + /* Allow unexpected mappings not to be fatal for RAM devices */
> > + error_report("map ram fail!");
> > + return ;
> > + }
> > + goto fail;
> > + }
> > +
> > + return;
> > +
> > +fail:
> > + if (memory_region_is_ram_device(section->mr)) {
> > + error_report("failed to vdpa_dma_map. pci p2p may not work");
> > + return;
> > +
> > + }
> > + /*
> > + * On the initfn path, store the first error in the container so we
> > + * can gracefully fail. Runtime, there's not much we can do other
> > + * than throw a hardware error.
> > + */
> > + error_report("vhost-vdpa: DMA mapping failed, unable to continue");
> > + return;
> > +
> > +}
> > +
> > +static void vhost_vdpa_listener_region_del(MemoryListener *listener,
> > + MemoryRegionSection *section)
> > +{
> > + struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
> > + hwaddr iova;
> > + Int128 llend, llsize;
> > + int ret;
> > + bool try_unmap = true;
> > +
> > + if (vhost_vdpa_listener_skipped_section(section)) {
> > + return;
> > + }
> > +
> > + if (unlikely((section->offset_within_address_space & ~TARGET_PAGE_MASK) !=
> > + (section->offset_within_region & ~TARGET_PAGE_MASK))) {
> > + error_report("%s received unaligned region", __func__);
> > + return;
> > + }
> > +
> > + iova = TARGET_PAGE_ALIGN(section->offset_within_address_space);
> > + llend = int128_make64(section->offset_within_address_space);
> > + llend = int128_add(llend, section->size);
> > + llend = int128_and(llend, int128_exts64(TARGET_PAGE_MASK));
> > +
> > + if (int128_ge(int128_make64(iova), llend)) {
> > + return;
> > + }
> > +
> > + llsize = int128_sub(llend, int128_make64(iova));
> > +
> > + if (try_unmap) {
> > + ret = vhost_vdpa_dma_unmap(v, iova, int128_get64(llsize));
> > + if (ret) {
> > + error_report("vhost_vdpa dma unmap error!");
> > + }
> > + }
> > +
> > + memory_region_unref(section->mr);
> > +}
> > +/*
> > + * IOTLB API is used by vhost-vpda which requires incremental updating
> > + * of the mapping. So we can not use generic vhost memory listener which
> > + * depends on the addnop().
> > + */
> > +static const MemoryListener vhost_vdpa_memory_listener = {
> > + .region_add = vhost_vdpa_listener_region_add,
> > + .region_del = vhost_vdpa_listener_region_del,
> > +};
> > +
> > +static int vhost_vdpa_call(struct vhost_dev *dev, unsigned long int request,
> > + void *arg)
> > +{
> > + struct vhost_vdpa *v = dev->opaque;
> > + int fd = v->device_fd;
> > +
> > + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
> > +
> > + return ioctl(fd, request, arg);
> > +}
> > +
> > +static void vhost_vdpa_add_status(struct vhost_dev *dev, uint8_t status)
> > +{
> > + uint8_t s;
> > +
> > + if (vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &s)) {
> > + return;
> > + }
> > +
> > + s |= status;
> > +
> > + vhost_vdpa_call(dev, VHOST_VDPA_SET_STATUS, &s);
> > +}
> > +
> > +static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque)
> > +{
> > + struct vhost_vdpa *v;
> > + uint64_t features;
> > + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
> > +
> > + v = opaque;
> > + dev->opaque = opaque ;
> > + vhost_vdpa_call(dev, VHOST_GET_FEATURES, &features);
> > + dev->backend_features = features;
> > + v->listener = vhost_vdpa_memory_listener;
> > + v->msg_type = VHOST_IOTLB_MSG_V2;
> > +
> > + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |
> > + VIRTIO_CONFIG_S_DRIVER);
> > +
> > + return 0;
> > +}
> > +
> > +static int vhost_vdpa_cleanup(struct vhost_dev *dev)
> > +{
> > + struct vhost_vdpa *v;
> > + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
> > + v = dev->opaque;
> > + memory_listener_unregister(&v->listener);
> > +
> > + dev->opaque = NULL;
> > + return 0;
> > +}
> > +
> > +static int vhost_vdpa_memslots_limit(struct vhost_dev *dev)
> > +{
> > + return INT_MAX;
> > +}
> > +
> > +static int vhost_vdpa_set_mem_table(struct vhost_dev *dev,
> > + struct vhost_memory *mem)
> > +{
> > +
> > + if (mem->padding) {
> > + return -1;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int vhost_vdpa_set_features(struct vhost_dev *dev,
> > + uint64_t features)
> > +{
> > + int ret;
> > + ret = vhost_vdpa_call(dev, VHOST_SET_FEATURES, &features);
> > + uint8_t status = 0;
> > + if (ret) {
> > + return ret;
> > + }
> > + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_FEATURES_OK);
> > + vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status);
> > +
> > + return !(status & VIRTIO_CONFIG_S_FEATURES_OK);
> > +}
> > +
> > +int vhost_vdpa_get_device_id(struct vhost_dev *dev,
> > + uint32_t *device_id)
> > +{
> > + return vhost_vdpa_call(dev, VHOST_VDPA_GET_DEVICE_ID, device_id);
> > +}
> > +
> > +static int vhost_vdpa_reset_device(struct vhost_dev *dev)
> > +{
> > + uint8_t status = 0;
> > +
> > + return vhost_vdpa_call(dev, VHOST_VDPA_SET_STATUS, &status);
> > +}
> > +
> > +static int vhost_vdpa_get_vq_index(struct vhost_dev *dev, int idx)
> > +{
> > + assert(idx >= dev->vq_index && idx < dev->vq_index + dev->nvqs);
> > +
> > + return idx - dev->vq_index;
> > +}
> > +
> > +static int vhost_vdpa_set_vring_ready(struct vhost_dev *dev)
> > +{
> > + int i;
> > + for (i = 0; i < dev->nvqs; ++i) {
> > + struct vhost_vring_state state = {
> > + .index = dev->vq_index + i,
> > + .num = 1,
> > + };
> > + vhost_vdpa_call(dev, VHOST_VDPA_SET_VRING_ENABLE, &state);
> > + }
> > + return 0;
> > +}
> > +
> > +static int vhost_vdpa_set_config(struct vhost_dev *dev, const uint8_t *data,
> > + uint32_t offset, uint32_t size,
> > + uint32_t flags)
> > +{
> > + struct vhost_vdpa_config *config;
> > + int ret;
> > + unsigned long config_size = offsetof(struct vhost_vdpa_config, buf);
> > + config = g_malloc(size + config_size);
> > + if (config == NULL) {
> > + return -1;
> > + }
> > + config->off = offset;
> > + config->len = size;
> > + memcpy(config->buf, data, size);
> > + ret = vhost_vdpa_call(dev, VHOST_VDPA_SET_CONFIG, config);
> > + g_free(config);
> > + return ret;
> > +}
> > +
> > +static int vhost_vdpa_get_config(struct vhost_dev *dev, uint8_t *config,
> > + uint32_t config_len)
> > +{
> > + struct vhost_vdpa_config *v_config;
> > + unsigned long config_size = offsetof(struct vhost_vdpa_config, buf);
> > + int ret;
> > +
> > + v_config = g_malloc(config_len + config_size);
> > + if (v_config == NULL) {
> > + return -1;
> > + }
> > + v_config->len = config_len;
> > + v_config->off = 0;
> > + ret = vhost_vdpa_call(dev, VHOST_VDPA_GET_CONFIG, v_config);
> > + memcpy(config, v_config->buf, config_len);
> > + g_free(v_config);
> > + return ret;
> > + }
> > +
> > +static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started)
> > +{
> > + struct vhost_vdpa *v = dev->opaque;
> > + if (started) {
> > + uint8_t status = 0;
> > + memory_listener_register(&v->listener, &address_space_memory);
> > + vhost_vdpa_set_vring_ready(dev);
> > + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK);
> > + vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status);
> > +
> > + return !(status & VIRTIO_CONFIG_S_DRIVER_OK);
> > + } else {
> > + vhost_vdpa_reset_device(dev);
> > + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |
> > + VIRTIO_CONFIG_S_DRIVER);
> > + memory_listener_unregister(&v->listener);
> > +
> > + return 0;
> > + }
> > +}
> > +
> > +static int vhost_vdpa_set_log_base(struct vhost_dev *dev, uint64_t base,
> > + struct vhost_log *log)
> > +{
> > + return vhost_vdpa_call(dev, VHOST_SET_LOG_BASE, &base);
> > +}
> > +
> > +static int vhost_vdpa_set_vring_addr(struct vhost_dev *dev,
> > + struct vhost_vring_addr *addr)
> > +{
> > + return vhost_vdpa_call(dev, VHOST_SET_VRING_ADDR, addr);
> > +}
> > +
> > +static int vhost_vdpa_set_vring_num(struct vhost_dev *dev,
> > + struct vhost_vring_state *ring)
> > +{
> > + return vhost_vdpa_call(dev, VHOST_SET_VRING_NUM, ring);
> > +}
> > +
> > +static int vhost_vdpa_set_vring_base(struct vhost_dev *dev,
> > + struct vhost_vring_state *ring)
> > +{
> > + return vhost_vdpa_call(dev, VHOST_SET_VRING_BASE, ring);
> > +}
> > +
> > +static int vhost_vdpa_get_vring_base(struct vhost_dev *dev,
> > + struct vhost_vring_state *ring)
> > +{
> > + return vhost_vdpa_call(dev, VHOST_GET_VRING_BASE, ring);
> > +}
> > +
> > +static int vhost_vdpa_set_vring_kick(struct vhost_dev *dev,
> > + struct vhost_vring_file *file)
> > +{
> > + return vhost_vdpa_call(dev, VHOST_SET_VRING_KICK, file);
> > +}
> > +
> > +static int vhost_vdpa_set_vring_call(struct vhost_dev *dev,
> > + struct vhost_vring_file *file)
> > +{
> > + return vhost_vdpa_call(dev, VHOST_SET_VRING_CALL, file);
> > +}
> > +
> > +static int vhost_vdpa_get_features(struct vhost_dev *dev,
> > + uint64_t *features)
> > +{
> > + return vhost_vdpa_call(dev, VHOST_GET_FEATURES, features);
> > +}
> > +
> > +static int vhost_vdpa_set_owner(struct vhost_dev *dev)
> > +{
> > + return vhost_vdpa_call(dev, VHOST_SET_OWNER, NULL);
> > +}
> > +
> > +static int vhost_vdpa_vq_get_addr(struct vhost_dev *dev,
> > + struct vhost_vring_addr *addr, struct vhost_virtqueue *vq)
> > +{
> > + assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
> > + addr->desc_user_addr = (uint64_t)(unsigned long)vq->desc_phys;
> > + addr->avail_user_addr = (uint64_t)(unsigned long)vq->avail_phys;
> > + addr->used_user_addr = (uint64_t)(unsigned long)vq->used_phys;
> > + return 0;
> > +}
> > +
> > +const VhostOps vdpa_ops = {
> > + .backend_type = VHOST_BACKEND_TYPE_VDPA,
> > + .vhost_backend_init = vhost_vdpa_init,
> > + .vhost_backend_cleanup = vhost_vdpa_cleanup,
> > + .vhost_set_log_base = vhost_vdpa_set_log_base,
> > + .vhost_set_vring_addr = vhost_vdpa_set_vring_addr,
> > + .vhost_set_vring_num = vhost_vdpa_set_vring_num,
> > + .vhost_set_vring_base = vhost_vdpa_set_vring_base,
> > + .vhost_get_vring_base = vhost_vdpa_get_vring_base,
> > + .vhost_set_vring_kick = vhost_vdpa_set_vring_kick,
> > + .vhost_set_vring_call = vhost_vdpa_set_vring_call,
> > + .vhost_get_features = vhost_vdpa_get_features,
> > + .vhost_set_owner = vhost_vdpa_set_owner,
> > + .vhost_set_vring_endian = NULL,
> > + .vhost_backend_memslots_limit = vhost_vdpa_memslots_limit,
> > + .vhost_set_mem_table = vhost_vdpa_set_mem_table,
> > + .vhost_set_features = vhost_vdpa_set_features,
> > + .vhost_reset_device = vhost_vdpa_reset_device,
> > + .vhost_get_vq_index = vhost_vdpa_get_vq_index,
> > + .vhost_get_config = vhost_vdpa_get_config,
> > + .vhost_set_config = vhost_vdpa_set_config,
> > + .vhost_requires_shm_log = NULL,
> > + .vhost_migration_done = NULL,
> > + .vhost_backend_can_merge = NULL,
> > + .vhost_net_set_mtu = NULL,
> > + .vhost_set_iotlb_callback = NULL,
> > + .vhost_send_device_iotlb_msg = NULL,
> > + .vhost_dev_start = vhost_vdpa_dev_start,
> > + .vhost_get_device_id = vhost_vdpa_get_device_id,
> > + .vhost_vq_get_addr = vhost_vdpa_vq_get_addr,
> > +};
> > diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
> > index 1e083a8976..bcaa66e0d2 100644
> > --- a/hw/virtio/vhost.c
> > +++ b/hw/virtio/vhost.c
> > @@ -808,7 +808,9 @@ static int vhost_dev_set_features(struct vhost_dev *dev,
> > features |= 0x1ULL << VHOST_F_LOG_ALL;
> > }
> > if (!vhost_dev_has_iommu(dev)) {
> > - features &= ~(0x1ULL << VIRTIO_F_IOMMU_PLATFORM);
> > + if (dev->vhost_ops->backend_type != VHOST_BACKEND_TYPE_VDPA) {
> > + features &= ~(0x1ULL << VIRTIO_F_IOMMU_PLATFORM);
> > + }
>
>
> Let's add a vhost_ops here if we want to force IOMMU_PLATFORM.
>
>
> > }
> > r = dev->vhost_ops->vhost_set_features(dev, features);
> > if (r < 0) {
> > @@ -1701,9 +1703,10 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev)
> > goto fail_log;
> > }
> > }
> > +
>
>
> Unnecessary change.
>
>
> > if (vhost_dev_has_iommu(hdev) &&
> > hdev->vhost_ops->vhost_set_iotlb_callback) {
> > - hdev->vhost_ops->vhost_set_iotlb_callback(hdev, true);
> > + hdev->vhost_ops->vhost_set_iotlb_callback(hdev, true);
>
>
> This change should belong to patch 5
>
> Thanks
>
>
> >
> > /* Update used ring information for IOTLB to work correctly,
> > * vhost-kernel code requires for this.*/
> > diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
> > index bfc24207e2..01f7c0d46f 100644
> > --- a/include/hw/virtio/vhost-backend.h
> > +++ b/include/hw/virtio/vhost-backend.h
> > @@ -17,7 +17,8 @@ typedef enum VhostBackendType {
> > VHOST_BACKEND_TYPE_NONE = 0,
> > VHOST_BACKEND_TYPE_KERNEL = 1,
> > VHOST_BACKEND_TYPE_USER = 2,
> > - VHOST_BACKEND_TYPE_MAX = 3,
> > + VHOST_BACKEND_TYPE_VDPA = 3,
> > + VHOST_BACKEND_TYPE_MAX = 4,
> > } VhostBackendType;
> >
> > typedef enum VhostSetConfigType {
> > @@ -167,6 +168,7 @@ typedef struct VhostOps {
> > } VhostOps;
> >
> > extern const VhostOps user_ops;
> > +extern const VhostOps vdpa_ops;
> >
> > int vhost_set_backend_type(struct vhost_dev *dev,
> > VhostBackendType backend_type);
> > diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
> > new file mode 100644
> > index 0000000000..6455663388
> > --- /dev/null
> > +++ b/include/hw/virtio/vhost-vdpa.h
> > @@ -0,0 +1,26 @@
> > +/*
> > + * vhost-vdpa.h
> > + *
> > + * Copyright(c) 2017-2018 Intel Corporation.
> > + * Copyright(c) 2020 Red Hat, Inc.
> > + *
> > + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> > + * See the COPYING file in the top-level directory.
> > + *
> > + */
> > +
> > +#ifndef HW_VIRTIO_VHOST_VDPA_H
> > +#define HW_VIRTIO_VHOST_VDPA_H
> > +
> > +#include "hw/virtio/virtio.h"
> > +
> > +typedef struct vhost_vdpa {
> > + int device_fd;
> > + uint32_t msg_type;
> > + MemoryListener listener;
> > +} VhostVDPA;
> > +
> > +extern AddressSpace address_space_memory;
> > +extern int vhost_vdpa_get_device_id(struct vhost_dev *dev,
> > + uint32_t *device_id);
> > +#endif
> > diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
> > index 085450c6f8..767a95ec0b 100644
> > --- a/include/hw/virtio/vhost.h
> > +++ b/include/hw/virtio/vhost.h
> > @@ -92,6 +92,13 @@ struct vhost_dev {
> > const VhostDevConfigOps *config_ops;
> > };
> >
> > +struct vhost_net {
> > + struct vhost_dev dev;
> > + struct vhost_virtqueue vqs[2];
> > + int backend;
> > + NetClientState *nc;
> > +};
> > +
> > int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
> > VhostBackendType backend_type,
> > uint32_t busyloop_timeout);
> > diff --git a/qemu-options.hx b/qemu-options.hx
> > index 196f468786..fa1b19de4c 100644
> > --- a/qemu-options.hx
> > +++ b/qemu-options.hx
> > @@ -2418,6 +2418,10 @@ DEF("netdev", HAS_ARG, QEMU_OPTION_netdev,
> > #ifdef CONFIG_POSIX
> > "-netdev vhost-user,id=str,chardev=dev[,vhostforce=on|off]\n"
> > " configure a vhost-user network, backed by a chardev 'dev'\n"
> > +#endif
> > +#ifdef __linux__
> > + "-netdev vhost-vdpa,id=str,vhostdev=/path/to/dev\n"
> > + " configure a vhost-vdpa network,Establish a vhost-vdpa netdev\n"
> > #endif
> > "-netdev hubport,id=str,hubid=n[,netdev=nd]\n"
> > " configure a hub port on the hub with ID 'n'\n", QEMU_ARCH_ALL)
> > @@ -2897,6 +2901,14 @@ SRST
> > -netdev type=vhost-user,id=net0,chardev=chr0 \
> > -device virtio-net-pci,netdev=net0
> >
> > +``-netdev vhost-vdpa,vhostdev=/path/to/dev``
> > + Establish a vhost-vdpa netdev.
> > +
> > + vDPA device is a device that uses a datapath which complies with
> > + the virtio specifications with a vendor specific control path.
> > + vDPA devices can be both physically located on the hardware or
> > + emulated by software.
> > +
> > ``-netdev hubport,id=id,hubid=hubid[,netdev=nd]``
> > Create a hub port on the emulated hub with ID hubid.
> >
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2 12/12] vhost-vdpa: introduce vhost-vdpa net client
2020-06-30 17:49 ` [PATCH v2 12/12] vhost-vdpa: introduce vhost-vdpa net client Cindy Lu
@ 2020-07-01 14:24 ` Eric Blake
2020-07-01 14:26 ` Cindy Lu
0 siblings, 1 reply; 20+ messages in thread
From: Eric Blake @ 2020-07-01 14:24 UTC (permalink / raw)
To: Cindy Lu, mst, armbru, cohuck, jasowang
Cc: mhabets, qemu-devel, rob.miller, saugatm, hanand, hch, eperezma,
jgg, shahafs, kevin.tian, parav, vmireyno, cunming.liang, gdawar,
jiri, xiao.w.wang, stefanha, zhihong.wang, Tiwei Bie, aadam,
rdunlap, maxime.coquelin, lingshan.zhu
On 6/30/20 12:49 PM, Cindy Lu wrote:
> This patch set introduces a new net client type: vhost-vdpa.
> vhost-vdpa net client will set up a vDPA device which is specified
> by a "vhostdev" parameter.
>
> Signed-off-by: Lingshan Zhu <lingshan.zhu@intel.com>
> Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
> Signed-off-by: Cindy Lu <lulu@redhat.com>
> ---
> +++ b/qapi/net.json
> @@ -428,16 +428,37 @@
> '*vhostforce': 'bool',
> '*queues': 'int' } }
>
> +##
> +# @NetdevVhostVDPAOptions:
> +#
> +# Vhost-vdpa network backend
> +#
> +# vDPA device is a device that uses a datapath which complies with the virtio
> +# specifications with a vendor specific control path.
> +#
> +# @vhostdev: path of vhost-vdpa device
> +# (default:'/dev/vhost-vdpa-0')
> +#
> +# @queues: number of queues to be created for multiqueue vhost-vdpa
> +# (default: 1)
> +#
> +# Since: 5.1
> +##
> +{ 'struct': 'NetdevVhostVDPAOptions',
> + 'data': {
> + '*vhostdev': 'str',
> + '*queues': 'int' } }
> +
> ##
> # @NetClientDriver:
> #
> # Available netdev drivers.
> #
> -# Since: 2.7
> +# Since: 5.1
This should be:
# Since: 2.7
# @vhost-vdpa since 5.1
since the enum has been around a lot longer than the latest addition to it.
> ##
> { 'enum': 'NetClientDriver',
> 'data': [ 'none', 'nic', 'user', 'tap', 'l2tpv3', 'socket', 'vde',
> - 'bridge', 'hubport', 'netmap', 'vhost-user' ] }
> + 'bridge', 'hubport', 'netmap', 'vhost-user', 'vhost-vdpa' ] }
>
> ##
> # @Netdev:
> @@ -465,7 +486,8 @@
> 'bridge': 'NetdevBridgeOptions',
> 'hubport': 'NetdevHubPortOptions',
> 'netmap': 'NetdevNetmapOptions',
> - 'vhost-user': 'NetdevVhostUserOptions' } }
> + 'vhost-user': 'NetdevVhostUserOptions',
> + 'vhost-vdpa': 'NetdevVhostVDPAOptions' } }
>
> ##
> # @NetFilterDirection:
>
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v2 12/12] vhost-vdpa: introduce vhost-vdpa net client
2020-07-01 14:24 ` Eric Blake
@ 2020-07-01 14:26 ` Cindy Lu
0 siblings, 0 replies; 20+ messages in thread
From: Cindy Lu @ 2020-07-01 14:26 UTC (permalink / raw)
To: Eric Blake
Cc: Cornelia Huck, Michael Tsirkin, Jason Wang, qemu-devel, hanand,
Rob Miller, saugatm, Markus Armbruster, hch,
Eugenio Perez Martin, jgg, mhabets, Shahaf Shuler, kevin.tian,
parav, Vitaly Mireyno, Liang, Cunming, gdawar, jiri, xiao.w.wang,
Stefan Hajnoczi, Wang, Zhihong, Tiwei Bie, Ariel Adam, rdunlap,
Maxime Coquelin, Zhu, Lingshan
On Wed, Jul 1, 2020 at 10:24 PM Eric Blake <eblake@redhat.com> wrote:
>
> On 6/30/20 12:49 PM, Cindy Lu wrote:
> > This patch set introduces a new net client type: vhost-vdpa.
> > vhost-vdpa net client will set up a vDPA device which is specified
> > by a "vhostdev" parameter.
> >
> > Signed-off-by: Lingshan Zhu <lingshan.zhu@intel.com>
> > Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
> > Signed-off-by: Cindy Lu <lulu@redhat.com>
> > ---
>
> > +++ b/qapi/net.json
> > @@ -428,16 +428,37 @@
> > '*vhostforce': 'bool',
> > '*queues': 'int' } }
> >
> > +##
> > +# @NetdevVhostVDPAOptions:
> > +#
> > +# Vhost-vdpa network backend
> > +#
> > +# vDPA device is a device that uses a datapath which complies with the virtio
> > +# specifications with a vendor specific control path.
> > +#
> > +# @vhostdev: path of vhost-vdpa device
> > +# (default:'/dev/vhost-vdpa-0')
> > +#
> > +# @queues: number of queues to be created for multiqueue vhost-vdpa
> > +# (default: 1)
> > +#
> > +# Since: 5.1
> > +##
> > +{ 'struct': 'NetdevVhostVDPAOptions',
> > + 'data': {
> > + '*vhostdev': 'str',
> > + '*queues': 'int' } }
> > +
> > ##
> > # @NetClientDriver:
> > #
> > # Available netdev drivers.
> > #
> > -# Since: 2.7
> > +# Since: 5.1
>
> This should be:
>
> # Since: 2.7
> # @vhost-vdpa since 5.1
>
> since the enum has been around a lot longer than the latest addition to it.
>
Thanks Eric, will fix
> > ##
> > { 'enum': 'NetClientDriver',
> > 'data': [ 'none', 'nic', 'user', 'tap', 'l2tpv3', 'socket', 'vde',
> > - 'bridge', 'hubport', 'netmap', 'vhost-user' ] }
> > + 'bridge', 'hubport', 'netmap', 'vhost-user', 'vhost-vdpa' ] }
> >
> > ##
> > # @Netdev:
> > @@ -465,7 +486,8 @@
> > 'bridge': 'NetdevBridgeOptions',
> > 'hubport': 'NetdevHubPortOptions',
> > 'netmap': 'NetdevNetmapOptions',
> > - 'vhost-user': 'NetdevVhostUserOptions' } }
> > + 'vhost-user': 'NetdevVhostUserOptions',
> > + 'vhost-vdpa': 'NetdevVhostVDPAOptions' } }
> >
> > ##
> > # @NetFilterDirection:
> >
>
> --
> Eric Blake, Principal Software Engineer
> Red Hat, Inc. +1-919-301-3226
> Virtualization: qemu.org | libvirt.org
>
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2020-07-01 14:36 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-30 17:49 [PATCH v2 00/12] vDPA support in qemu Cindy Lu
2020-06-30 17:49 ` [PATCH v2 01/12] net: introduce qemu_get_peer Cindy Lu
2020-06-30 17:49 ` [PATCH v2 02/12] vhost_net: use the function qemu_get_peer Cindy Lu
2020-06-30 17:49 ` [PATCH v2 03/12] virtio-bus: introduce queue_enabled method Cindy Lu
2020-06-30 17:49 ` [PATCH v2 04/12] virtio-pci: implement " Cindy Lu
2020-06-30 17:49 ` [PATCH v2 05/12] vhost: check the existence of vhost_set_iotlb_callback Cindy Lu
2020-06-30 17:49 ` [PATCH v2 06/12] vhost: introduce new VhostOps vhost_dev_start Cindy Lu
2020-06-30 17:49 ` [PATCH v2 07/12] vhost: implement vhost_dev_start method Cindy Lu
2020-06-30 17:49 ` [PATCH v2 08/12] vhost: introduce new VhostOps vhost_vq_get_addr Cindy Lu
2020-06-30 17:49 ` [PATCH v2 09/12] vhost: implement vhost_vq_get_addr method Cindy Lu
2020-06-30 17:49 ` [PATCH v2 10/12] vhost: introduce new VhostOps vhost_get_device_id Cindy Lu
2020-06-30 17:49 ` [PATCH v2 11/12] vhost-vdpa: introduce vhost-vdpa backend Cindy Lu
2020-07-01 4:21 ` Jason Wang
2020-07-01 9:17 ` Cindy Lu
2020-06-30 17:49 ` [PATCH v2 12/12] vhost-vdpa: introduce vhost-vdpa net client Cindy Lu
2020-07-01 14:24 ` Eric Blake
2020-07-01 14:26 ` Cindy Lu
2020-06-30 18:06 ` [PATCH v2 00/12] vDPA support in qemu no-reply
2020-07-01 5:29 ` Jason Wang
2020-07-01 5:30 ` Cindy Lu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).