[PATCH v3 00/21] implement packed virtqueues

[PATCH v3 00/21] implement packed virtqueues

[Jens Freimann]

This is a basic implementation of packed virtqueues as specified in the
Virtio 1.1 draft. A compiled version of the current draft is available
at https://github.com/oasis-tcs/virtio-docs.git (or as .pdf at
https://github.com/oasis-tcs/virtio-docs/blob/master/virtio-v1.1-packed-wd10.pdf

It does not implement yet indirect descriptors and checksum offloading.

A packed virtqueue is different from a split virtqueue in that it
consists of only a single descriptor ring that replaces available and
used ring, index and descriptor buffer.

Each descriptor is readable and writable and has a flags field. These flags
will mark if a descriptor is available or used.  To detect new available descriptors
even after the ring has wrapped, device and driver each have a
single-bit wrap counter that is flipped from 0 to 1 and vice versa every time
the last descriptor in the ring is used/made available.

The idea behind this is to 1. improve performance by avoiding cache misses
and 2. be easier for devices to implement.

Regarding performance: with these patches I get 21.13 Mpps on my system
as compared to 18.8 Mpps with the virtio 1.0 code. Packet size was 64
bytes, 0.05% acceptable loss.  Test setup is described as in
http://dpdk.org/doc/guides/howto/pvp_reference_benchmark.html

Packet generator:
MoonGen
Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz
Intel X710 NIC
RHEL 7.4

Device under test:
Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
Intel X710 NIC
RHEL 7.4

VM on DuT: RHEL7.4

I plan to do more performance test with bigger frame sizes.

This patch series is based on a prototype implemented by Yuanhan Liu and
Tiwei Bie.

changes from v2->v3:
* implement event suppression
* add code do dump packed virtqueues
* don't use assert in vhost code
* rename virtio-user parameter to packed-vq
* support rxvf flush 

changes from v1->v2:
* don't use VIRTQ_DESC_F_NEXT in used descriptors (Jason)
* no rte_panice() in guest triggerable code (Maxime)
* use unlikely when checking for vq (Maxime)
* rename everything from _1_1 to _packed  (Yuanhan)
* add two more patches to implement mergeable receive buffers 

regards,
Jens 


Jens Freimann (17):
  net/virtio: by default disable packed virtqueues
  net/virtio: vring init for packed queues
  net/virtio: add virtio 1.1 defines
  net/virtio: add packed virtqueue helpers
  net/virtio: dump packed virtqueue data
  net/virtio: implement transmit path for packed queues
  vhost: disable packed virtqueues by default
  vhost: turn of indirect descriptors for packed virtqueues
  vhost: add virtio 1.1 defines
  vhost: add helpers for packed virtqueues
  vhost: dequeue for packed queues
  vhost: packed queue enqueue path
  net/virtio: disable ctrl virtqueue for packed rings
  net/virtio: add support for mergeable buffers with packed virtqueues
  vhost: support mergeable rx buffers with packed queues
  net/virtio: add support for event suppression
  vhost: add event suppression for packed queues

Yuanhan Liu (4):
  net/virtio-user: add option to use packed queues
  net/virtio: implement receive path for packed queues
  vhost: vring address setup for packed queues
  vhost: enable packed virtqueues

 config/common_base                               |   2 +
 drivers/net/virtio/virtio_ethdev.c               |  51 ++-
 drivers/net/virtio/virtio_ethdev.h               |   7 +-
 drivers/net/virtio/virtio_pci.h                  |   8 +
 drivers/net/virtio/virtio_ring.h                 |  91 +++-
 drivers/net/virtio/virtio_rxtx.c                 | 359 +++++++++++++++-
 drivers/net/virtio/virtio_user/virtio_user_dev.c |  12 +-
 drivers/net/virtio/virtio_user/virtio_user_dev.h |   3 +-
 drivers/net/virtio/virtio_user_ethdev.c          |  15 +-
 drivers/net/virtio/virtqueue.c                   |  10 +
 drivers/net/virtio/virtqueue.h                   |  91 +++-
 lib/librte_vhost/socket.c                        |   5 +
 lib/librte_vhost/vhost.c                         |  16 +-
 lib/librte_vhost/vhost.h                         |  69 ++-
 lib/librte_vhost/vhost_user.c                    |  40 +-
 lib/librte_vhost/virtio-1.1.h                    |  62 +++
 lib/librte_vhost/virtio_net.c                    | 507 +++++++++++++++++++++--
 17 files changed, 1271 insertions(+), 77 deletions(-)
 create mode 100644 lib/librte_vhost/virtio-1.1.h

-- 
2.14.3

[PATCH v3 01/21] net/virtio: by default disable packed virtqueues

[Jens Freimann]

Disable packed virtqueues for now and make it dependend on a build-time
config option. This can be reverted once we have missing features like
indirect descriptors implemented.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 config/common_base                 | 1 +
 drivers/net/virtio/virtio_ethdev.c | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/config/common_base b/config/common_base
index c09c7cf88..cd4b419b4 100644
--- a/config/common_base
+++ b/config/common_base
@@ -346,6 +346,7 @@ CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
 CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=n
 CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=n
 CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DUMP=n
+CONFIG_RTE_LIBRTE_VIRTIO_PQ=n
 
 #
 # Compile virtio device emulation inside virtio PMD driver
diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index 11f758929..06fbf7311 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1149,6 +1149,10 @@ virtio_negotiate_features(struct virtio_hw *hw, uint64_t req_features)
 			req_features &= ~(1ULL << VIRTIO_NET_F_MTU);
 	}
 
+#ifndef RTE_LIBRTE_VIRTIO_PQ
+	req_features &= ~(1ull << VIRTIO_F_RING_PACKED);
+#endif
+
 	/*
 	 * Negotiate features: Subset of device feature bits are written back
 	 * guest feature bits.
-- 
2.14.3

Re: [PATCH v3 01/21] net/virtio: by default disable packed virtqueues

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Disable packed virtqueues for now and make it dependend on a build-time
> config option. This can be reverted once we have missing features like
> indirect descriptors implemented.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   config/common_base                 | 1 +
>   drivers/net/virtio/virtio_ethdev.c | 4 ++++
>   2 files changed, 5 insertions(+)
> 
> diff --git a/config/common_base b/config/common_base
> index c09c7cf88..cd4b419b4 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -346,6 +346,7 @@ CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
>   CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=n
>   CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=n
>   CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DUMP=n
> +CONFIG_RTE_LIBRTE_VIRTIO_PQ=n
>   
>   #
>   # Compile virtio device emulation inside virtio PMD driver
> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
> index 11f758929..06fbf7311 100644
> --- a/drivers/net/virtio/virtio_ethdev.c
> +++ b/drivers/net/virtio/virtio_ethdev.c
> @@ -1149,6 +1149,10 @@ virtio_negotiate_features(struct virtio_hw *hw, uint64_t req_features)
>   			req_features &= ~(1ULL << VIRTIO_NET_F_MTU);
>   	}
>   
> +#ifndef RTE_LIBRTE_VIRTIO_PQ
> +	req_features &= ~(1ull << VIRTIO_F_RING_PACKED);
Is VIRTIO_F_RING_PACKED already declared here?
I don't think it is, so would break build when bisecting.

Maybe the thing to do is to not have VIRTIO_F_RING_PACKED in the feature 
set by default. And when RTE_LIBRTE_VIRTIO_PQ is set, enable it and
explicitly disable indirect descs.

> +#endif
> +
>   	/*
>   	 * Negotiate features: Subset of device feature bits are written back
>   	 * guest feature bits.
> 

Re: [PATCH v3 01/21] net/virtio: by default disable packed virtqueues

[Jens Freimann]

On Thu, Apr 05, 2018 at 03:42:47PM +0200, Maxime Coquelin wrote:
>
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>Disable packed virtqueues for now and make it dependend on a build-time
>>config option. This can be reverted once we have missing features like
>>indirect descriptors implemented.
>>
>>Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>>---
>>  config/common_base                 | 1 +
>>  drivers/net/virtio/virtio_ethdev.c | 4 ++++
>>  2 files changed, 5 insertions(+)
>>
>>diff --git a/config/common_base b/config/common_base
>>index c09c7cf88..cd4b419b4 100644
>>--- a/config/common_base
>>+++ b/config/common_base
>>@@ -346,6 +346,7 @@ CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
>>  CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=n
>>  CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=n
>>  CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DUMP=n
>>+CONFIG_RTE_LIBRTE_VIRTIO_PQ=n
>>  #
>>  # Compile virtio device emulation inside virtio PMD driver
>>diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
>>index 11f758929..06fbf7311 100644
>>--- a/drivers/net/virtio/virtio_ethdev.c
>>+++ b/drivers/net/virtio/virtio_ethdev.c
>>@@ -1149,6 +1149,10 @@ virtio_negotiate_features(struct virtio_hw *hw, uint64_t req_features)
>>  			req_features &= ~(1ULL << VIRTIO_NET_F_MTU);
>>  	}
>>+#ifndef RTE_LIBRTE_VIRTIO_PQ
>>+	req_features &= ~(1ull << VIRTIO_F_RING_PACKED);
>Is VIRTIO_F_RING_PACKED already declared here?
>I don't think it is, so would break build when bisecting.

yes, you are right it is not. I pushed this patch to far up during git
rebase. 
>
>Maybe the thing to do is to not have VIRTIO_F_RING_PACKED in the 
>feature set by default. And when RTE_LIBRTE_VIRTIO_PQ is set, enable 
>it and
>explicitly disable indirect descs.

Yes, both would work. I'll go with your suggestion.
Thanks!

regards,
Jens 
>
>>+#endif
>>+
>>  	/*
>>  	 * Negotiate features: Subset of device feature bits are written back
>>  	 * guest feature bits.
>>

[PATCH v3 02/21] net/virtio: vring init for packed queues

[Jens Freimann]

Add and initialize descriptor data structures.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 drivers/net/virtio/virtio_ethdev.c | 22 +++++++++-------
 drivers/net/virtio/virtio_pci.h    |  8 ++++++
 drivers/net/virtio/virtio_ring.h   | 53 ++++++++++++++++++++++++++++++++++----
 drivers/net/virtio/virtqueue.h     | 10 +++++++
 4 files changed, 78 insertions(+), 15 deletions(-)

diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index 06fbf7311..cccefafe9 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -298,19 +298,21 @@ virtio_init_vring(struct virtqueue *vq)
 
 	PMD_INIT_FUNC_TRACE();
 
-	/*
-	 * Reinitialise since virtio port might have been stopped and restarted
-	 */
 	memset(ring_mem, 0, vq->vq_ring_size);
-	vring_init(vr, size, ring_mem, VIRTIO_PCI_VRING_ALIGN);
-	vq->vq_used_cons_idx = 0;
-	vq->vq_desc_head_idx = 0;
-	vq->vq_avail_idx = 0;
-	vq->vq_desc_tail_idx = (uint16_t)(vq->vq_nentries - 1);
+	vring_init(vq->hw, vr, size, ring_mem, VIRTIO_PCI_VRING_ALIGN);
+
 	vq->vq_free_cnt = vq->vq_nentries;
 	memset(vq->vq_descx, 0, sizeof(struct vq_desc_extra) * vq->vq_nentries);
+	vq->vq_used_cons_idx = 0;
+	vq->vq_avail_idx     = 0;
+	if (vtpci_packed_queue(vq->hw)) {
+		vring_desc_init_packed(vr, size);
+	} else {
+		vq->vq_desc_head_idx = 0;
+		vq->vq_desc_tail_idx = (uint16_t)(vq->vq_nentries - 1);
 
-	vring_desc_init(vr->desc, size);
+		vring_desc_init(vr->desc, size);
+	}
 
 	/*
 	 * Disable device(host) interrupting guest
@@ -385,7 +387,7 @@ virtio_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_queue_idx)
 	/*
 	 * Reserve a memzone for vring elements
 	 */
-	size = vring_size(vq_size, VIRTIO_PCI_VRING_ALIGN);
+	size = vring_size(hw, vq_size, VIRTIO_PCI_VRING_ALIGN);
 	vq->vq_ring_size = RTE_ALIGN_CEIL(size, VIRTIO_PCI_VRING_ALIGN);
 	PMD_INIT_LOG(DEBUG, "vring_size: %d, rounded_vring_size: %d",
 		     size, vq->vq_ring_size);
diff --git a/drivers/net/virtio/virtio_pci.h b/drivers/net/virtio/virtio_pci.h
index a28ba8339..528fb46b9 100644
--- a/drivers/net/virtio/virtio_pci.h
+++ b/drivers/net/virtio/virtio_pci.h
@@ -112,6 +112,8 @@ struct virtnet_ctl;
 
 #define VIRTIO_F_VERSION_1		32
 #define VIRTIO_F_IOMMU_PLATFORM	33
+#define VIRTIO_F_RING_PACKED		34
+#define VIRTIO_F_IN_ORDER		35
 
 /*
  * Some VirtIO feature bits (currently bits 28 through 31) are
@@ -304,6 +306,12 @@ vtpci_with_feature(struct virtio_hw *hw, uint64_t bit)
 	return (hw->guest_features & (1ULL << bit)) != 0;
 }
 
+static inline int
+vtpci_packed_queue(struct virtio_hw *hw)
+{
+	return vtpci_with_feature(hw, VIRTIO_F_RING_PACKED);
+}
+
 /*
  * Function declaration from virtio_pci.c
  */
diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
index 9e3c2a015..1bd7ba98e 100644
--- a/drivers/net/virtio/virtio_ring.h
+++ b/drivers/net/virtio/virtio_ring.h
@@ -9,6 +9,7 @@
 
 #include <rte_common.h>
 
+
 /* This marks a buffer as continuing via the next field. */
 #define VRING_DESC_F_NEXT       1
 /* This marks a buffer as write-only (otherwise read-only). */
@@ -54,11 +55,38 @@ struct vring_used {
 	struct vring_used_elem ring[0];
 };
 
+/* For support of packed virtqueues in Virtio 1.1 the format of descriptors
+ * looks like this.
+ */
+struct vring_desc_packed {
+	uint64_t addr;
+	uint32_t len;
+	uint16_t index;
+	uint16_t flags;
+};
+
+#define RING_EVENT_FLAGS_ENABLE 0x0
+#define RING_EVENT_FLAGS_DISABLE 0x1
+#define RING_EVENT_FLAGS_DESC 0x2
+struct vring_packed_desc_event {
+	uint16_t desc_event_off_wrap;
+	uint16_t desc_event_flags;
+};
+
 struct vring {
 	unsigned int num;
-	struct vring_desc  *desc;
-	struct vring_avail *avail;
-	struct vring_used  *used;
+	union {
+		struct vring_desc_packed *desc_packed;
+		struct vring_desc *desc;
+	};
+	union {
+		struct vring_avail *avail;
+		struct vring_packed_desc_event *driver_event;
+	};
+	union {
+		struct vring_used  *used;
+		struct vring_packed_desc_event *device_event;
+	};
 };
 
 /* The standard layout for the ring is a continuous chunk of memory which
@@ -95,10 +123,16 @@ struct vring {
 #define vring_avail_event(vr) (*(uint16_t *)&(vr)->used->ring[(vr)->num])
 
 static inline size_t
-vring_size(unsigned int num, unsigned long align)
+vring_size(struct virtio_hw *hw, unsigned int num, unsigned long align)
 {
 	size_t size;
 
+	if (vtpci_packed_queue(hw)) {
+		size = num * sizeof(struct vring_desc_packed);
+		size += 2 * sizeof(struct vring_packed_desc_event);
+		return size;
+	}
+
 	size = num * sizeof(struct vring_desc);
 	size += sizeof(struct vring_avail) + (num * sizeof(uint16_t));
 	size = RTE_ALIGN_CEIL(size, align);
@@ -108,10 +142,19 @@ vring_size(unsigned int num, unsigned long align)
 }
 
 static inline void
-vring_init(struct vring *vr, unsigned int num, uint8_t *p,
+vring_init(struct virtio_hw *hw, struct vring *vr, unsigned int num, uint8_t *p,
 	unsigned long align)
 {
 	vr->num = num;
+	if (vtpci_packed_queue(hw)) {
+		vr->desc_packed = (struct vring_desc_packed *)p;
+		vr->driver_event = (struct vring_packed_desc_event *) (p +
+			num * sizeof(struct vring_desc_packed));
+		vr->device_event = (struct vring_packed_desc_event *) (vr->driver_event +
+			sizeof(struct vring_packed_desc_event));
+		return;
+	}
+
 	vr->desc = (struct vring_desc *) p;
 	vr->avail = (struct vring_avail *) (p +
 		num * sizeof(struct vring_desc));
diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
index 14364f356..cc2e7c0f6 100644
--- a/drivers/net/virtio/virtqueue.h
+++ b/drivers/net/virtio/virtqueue.h
@@ -245,6 +245,16 @@ struct virtio_tx_region {
 			   __attribute__((__aligned__(16)));
 };
 
+static inline void
+vring_desc_init_packed(struct vring *vr, int n)
+{
+	int i;
+	for (i = 0; i < n; i++) {
+		struct vring_desc_packed *desc = &vr->desc_packed[i];
+		desc->index = i;
+	}
+}
+
 /* Chain all the descriptors in the ring with an END */
 static inline void
 vring_desc_init(struct vring_desc *dp, uint16_t n)
-- 
2.14.3

Re: [PATCH v3 02/21] net/virtio: vring init for packed queues

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Add and initialize descriptor data structures.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   drivers/net/virtio/virtio_ethdev.c | 22 +++++++++-------
>   drivers/net/virtio/virtio_pci.h    |  8 ++++++
>   drivers/net/virtio/virtio_ring.h   | 53 ++++++++++++++++++++++++++++++++++----
>   drivers/net/virtio/virtqueue.h     | 10 +++++++
>   4 files changed, 78 insertions(+), 15 deletions(-)
> 
> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
> index 06fbf7311..cccefafe9 100644
> --- a/drivers/net/virtio/virtio_ethdev.c
> +++ b/drivers/net/virtio/virtio_ethdev.c
> @@ -298,19 +298,21 @@ virtio_init_vring(struct virtqueue *vq)
>   
>   	PMD_INIT_FUNC_TRACE();
>   
> -	/*
> -	 * Reinitialise since virtio port might have been stopped and restarted
> -	 */
>   	memset(ring_mem, 0, vq->vq_ring_size);
> -	vring_init(vr, size, ring_mem, VIRTIO_PCI_VRING_ALIGN);
> -	vq->vq_used_cons_idx = 0;
> -	vq->vq_desc_head_idx = 0;
> -	vq->vq_avail_idx = 0;
> -	vq->vq_desc_tail_idx = (uint16_t)(vq->vq_nentries - 1);
> +	vring_init(vq->hw, vr, size, ring_mem, VIRTIO_PCI_VRING_ALIGN);
> +
>   	vq->vq_free_cnt = vq->vq_nentries;
>   	memset(vq->vq_descx, 0, sizeof(struct vq_desc_extra) * vq->vq_nentries);
> +	vq->vq_used_cons_idx = 0;
> +	vq->vq_avail_idx     = 0;
> +	if (vtpci_packed_queue(vq->hw)) {
> +		vring_desc_init_packed(vr, size);
> +	} else {
> +		vq->vq_desc_head_idx = 0;
> +		vq->vq_desc_tail_idx = (uint16_t)(vq->vq_nentries - 1);
>   
> -	vring_desc_init(vr->desc, size);
> +		vring_desc_init(vr->desc, size);
> +	}
>   
>   	/*
>   	 * Disable device(host) interrupting guest
> @@ -385,7 +387,7 @@ virtio_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_queue_idx)
>   	/*
>   	 * Reserve a memzone for vring elements
>   	 */
> -	size = vring_size(vq_size, VIRTIO_PCI_VRING_ALIGN);
> +	size = vring_size(hw, vq_size, VIRTIO_PCI_VRING_ALIGN);
>   	vq->vq_ring_size = RTE_ALIGN_CEIL(size, VIRTIO_PCI_VRING_ALIGN);
>   	PMD_INIT_LOG(DEBUG, "vring_size: %d, rounded_vring_size: %d",
>   		     size, vq->vq_ring_size);
> diff --git a/drivers/net/virtio/virtio_pci.h b/drivers/net/virtio/virtio_pci.h
> index a28ba8339..528fb46b9 100644
> --- a/drivers/net/virtio/virtio_pci.h
> +++ b/drivers/net/virtio/virtio_pci.h
> @@ -112,6 +112,8 @@ struct virtnet_ctl;
>   
>   #define VIRTIO_F_VERSION_1		32
>   #define VIRTIO_F_IOMMU_PLATFORM	33
> +#define VIRTIO_F_RING_PACKED		34
> +#define VIRTIO_F_IN_ORDER		35
>   
>   /*
>    * Some VirtIO feature bits (currently bits 28 through 31) are
> @@ -304,6 +306,12 @@ vtpci_with_feature(struct virtio_hw *hw, uint64_t bit)
>   	return (hw->guest_features & (1ULL << bit)) != 0;
>   }
>   
> +static inline int
> +vtpci_packed_queue(struct virtio_hw *hw)
> +{
> +	return vtpci_with_feature(hw, VIRTIO_F_RING_PACKED);
> +}
> +
>   /*
>    * Function declaration from virtio_pci.c
>    */
> diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
> index 9e3c2a015..1bd7ba98e 100644
> --- a/drivers/net/virtio/virtio_ring.h
> +++ b/drivers/net/virtio/virtio_ring.h
> @@ -9,6 +9,7 @@
>   
>   #include <rte_common.h>
>   
> +
Remove new line

>   /* This marks a buffer as continuing via the next field. */
>   #define VRING_DESC_F_NEXT       1
>   /* This marks a buffer as write-only (otherwise read-only). */
> @@ -54,11 +55,38 @@ struct vring_used {
>   	struct vring_used_elem ring[0];
>   };
>   
> +/* For support of packed virtqueues in Virtio 1.1 the format of descriptors
> + * looks like this.
> + */
> +struct vring_desc_packed {
> +	uint64_t addr;
> +	uint32_t len;
> +	uint16_t index;
> +	uint16_t flags;
> +};
> +
> +#define RING_EVENT_FLAGS_ENABLE 0x0
> +#define RING_EVENT_FLAGS_DISABLE 0x1
> +#define RING_EVENT_FLAGS_DESC 0x2
> +struct vring_packed_desc_event {
> +	uint16_t desc_event_off_wrap;
> +	uint16_t desc_event_flags;
> +};
> +
>   struct vring {
>   	unsigned int num;
> -	struct vring_desc  *desc;
> -	struct vring_avail *avail;
> -	struct vring_used  *used;
> +	union {
> +		struct vring_desc_packed *desc_packed;
> +		struct vring_desc *desc;
> +	};
> +	union {
> +		struct vring_avail *avail;
> +		struct vring_packed_desc_event *driver_event;
> +	};
> +	union {
> +		struct vring_used  *used;
> +		struct vring_packed_desc_event *device_event;
> +	};

Not a must, but I wonder if it wouldn't be cleaner to have a unique
union of two structs, one for packed rings, and one for the legacy.
I may be wrong, though.

>   };
>   
>   /* The standard layout for the ring is a continuous chunk of memory which
> @@ -95,10 +123,16 @@ struct vring {
>   #define vring_avail_event(vr) (*(uint16_t *)&(vr)->used->ring[(vr)->num])
>   
>   static inline size_t
> -vring_size(unsigned int num, unsigned long align)
> +vring_size(struct virtio_hw *hw, unsigned int num, unsigned long align)
>   {
>   	size_t size;
>   
> +	if (vtpci_packed_queue(hw)) {
> +		size = num * sizeof(struct vring_desc_packed);
> +		size += 2 * sizeof(struct vring_packed_desc_event);
> +		return size;
> +	}
> +
>   	size = num * sizeof(struct vring_desc);
>   	size += sizeof(struct vring_avail) + (num * sizeof(uint16_t));
>   	size = RTE_ALIGN_CEIL(size, align);
> @@ -108,10 +142,19 @@ vring_size(unsigned int num, unsigned long align)
>   }
>   
>   static inline void
> -vring_init(struct vring *vr, unsigned int num, uint8_t *p,
> +vring_init(struct virtio_hw *hw, struct vring *vr, unsigned int num, uint8_t *p,
>   	unsigned long align)
>   {
>   	vr->num = num;
> +	if (vtpci_packed_queue(hw)) {
> +		vr->desc_packed = (struct vring_desc_packed *)p;
> +		vr->driver_event = (struct vring_packed_desc_event *) (p +
> +			num * sizeof(struct vring_desc_packed));
> +		vr->device_event = (struct vring_packed_desc_event *) (vr->driver_event +
> +			sizeof(struct vring_packed_desc_event));
> +		return;
> +	}
> +
>   	vr->desc = (struct vring_desc *) p;
>   	vr->avail = (struct vring_avail *) (p +
>   		num * sizeof(struct vring_desc));
> diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
> index 14364f356..cc2e7c0f6 100644
> --- a/drivers/net/virtio/virtqueue.h
> +++ b/drivers/net/virtio/virtqueue.h
> @@ -245,6 +245,16 @@ struct virtio_tx_region {
>   			   __attribute__((__aligned__(16)));
>   };
>   
> +static inline void
> +vring_desc_init_packed(struct vring *vr, int n)
> +{
> +	int i;
> +	for (i = 0; i < n; i++) {
> +		struct vring_desc_packed *desc = &vr->desc_packed[i];
> +		desc->index = i;
> +	}
> +}
> +
>   /* Chain all the descriptors in the ring with an END */
>   static inline void
>   vring_desc_init(struct vring_desc *dp, uint16_t n)
> 

Re: [PATCH v3 02/21] net/virtio: vring init for packed queues

[Jens Freimann]

On Thu, Apr 05, 2018 at 04:15:56PM +0200, Maxime Coquelin wrote:
>
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>Add and initialize descriptor data structures.
>>
>>Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>>---
>>  drivers/net/virtio/virtio_ethdev.c | 22 +++++++++-------
>>  drivers/net/virtio/virtio_pci.h    |  8 ++++++
>>  drivers/net/virtio/virtio_ring.h   | 53 ++++++++++++++++++++++++++++++++++----
>>  drivers/net/virtio/virtqueue.h     | 10 +++++++
>>  4 files changed, 78 insertions(+), 15 deletions(-)
>>
>>diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
>>index 06fbf7311..cccefafe9 100644
>>--- a/drivers/net/virtio/virtio_ethdev.c
>>+++ b/drivers/net/virtio/virtio_ethdev.c
>>@@ -298,19 +298,21 @@ virtio_init_vring(struct virtqueue *vq)
>>  	PMD_INIT_FUNC_TRACE();
>>-	/*
>>-	 * Reinitialise since virtio port might have been stopped and restarted
>>-	 */
>>  	memset(ring_mem, 0, vq->vq_ring_size);
>>-	vring_init(vr, size, ring_mem, VIRTIO_PCI_VRING_ALIGN);
>>-	vq->vq_used_cons_idx = 0;
>>-	vq->vq_desc_head_idx = 0;
>>-	vq->vq_avail_idx = 0;
>>-	vq->vq_desc_tail_idx = (uint16_t)(vq->vq_nentries - 1);
>>+	vring_init(vq->hw, vr, size, ring_mem, VIRTIO_PCI_VRING_ALIGN);
>>+
>>  	vq->vq_free_cnt = vq->vq_nentries;
>>  	memset(vq->vq_descx, 0, sizeof(struct vq_desc_extra) * vq->vq_nentries);
>>+	vq->vq_used_cons_idx = 0;
>>+	vq->vq_avail_idx     = 0;
>>+	if (vtpci_packed_queue(vq->hw)) {
>>+		vring_desc_init_packed(vr, size);
>>+	} else {
>>+		vq->vq_desc_head_idx = 0;
>>+		vq->vq_desc_tail_idx = (uint16_t)(vq->vq_nentries - 1);
>>-	vring_desc_init(vr->desc, size);
>>+		vring_desc_init(vr->desc, size);
>>+	}
>>  	/*
>>  	 * Disable device(host) interrupting guest
>>@@ -385,7 +387,7 @@ virtio_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_queue_idx)
>>  	/*
>>  	 * Reserve a memzone for vring elements
>>  	 */
>>-	size = vring_size(vq_size, VIRTIO_PCI_VRING_ALIGN);
>>+	size = vring_size(hw, vq_size, VIRTIO_PCI_VRING_ALIGN);
>>  	vq->vq_ring_size = RTE_ALIGN_CEIL(size, VIRTIO_PCI_VRING_ALIGN);
>>  	PMD_INIT_LOG(DEBUG, "vring_size: %d, rounded_vring_size: %d",
>>  		     size, vq->vq_ring_size);
>>diff --git a/drivers/net/virtio/virtio_pci.h b/drivers/net/virtio/virtio_pci.h
>>index a28ba8339..528fb46b9 100644
>>--- a/drivers/net/virtio/virtio_pci.h
>>+++ b/drivers/net/virtio/virtio_pci.h
>>@@ -112,6 +112,8 @@ struct virtnet_ctl;
>>  #define VIRTIO_F_VERSION_1		32
>>  #define VIRTIO_F_IOMMU_PLATFORM	33
>>+#define VIRTIO_F_RING_PACKED		34
>>+#define VIRTIO_F_IN_ORDER		35
>>  /*
>>   * Some VirtIO feature bits (currently bits 28 through 31) are
>>@@ -304,6 +306,12 @@ vtpci_with_feature(struct virtio_hw *hw, uint64_t bit)
>>  	return (hw->guest_features & (1ULL << bit)) != 0;
>>  }
>>+static inline int
>>+vtpci_packed_queue(struct virtio_hw *hw)
>>+{
>>+	return vtpci_with_feature(hw, VIRTIO_F_RING_PACKED);
>>+}
>>+
>>  /*
>>   * Function declaration from virtio_pci.c
>>   */
>>diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
>>index 9e3c2a015..1bd7ba98e 100644
>>--- a/drivers/net/virtio/virtio_ring.h
>>+++ b/drivers/net/virtio/virtio_ring.h
>>@@ -9,6 +9,7 @@
>>  #include <rte_common.h>
>>+
>Remove new line
>
>>  /* This marks a buffer as continuing via the next field. */
>>  #define VRING_DESC_F_NEXT       1
>>  /* This marks a buffer as write-only (otherwise read-only). */
>>@@ -54,11 +55,38 @@ struct vring_used {
>>  	struct vring_used_elem ring[0];
>>  };
>>+/* For support of packed virtqueues in Virtio 1.1 the format of descriptors
>>+ * looks like this.
>>+ */
>>+struct vring_desc_packed {
>>+	uint64_t addr;
>>+	uint32_t len;
>>+	uint16_t index;
>>+	uint16_t flags;
>>+};
>>+
>>+#define RING_EVENT_FLAGS_ENABLE 0x0
>>+#define RING_EVENT_FLAGS_DISABLE 0x1
>>+#define RING_EVENT_FLAGS_DESC 0x2
>>+struct vring_packed_desc_event {
>>+	uint16_t desc_event_off_wrap;
>>+	uint16_t desc_event_flags;
>>+};
>>+
>>  struct vring {
>>  	unsigned int num;
>>-	struct vring_desc  *desc;
>>-	struct vring_avail *avail;
>>-	struct vring_used  *used;
>>+	union {
>>+		struct vring_desc_packed *desc_packed;
>>+		struct vring_desc *desc;
>>+	};
>>+	union {
>>+		struct vring_avail *avail;
>>+		struct vring_packed_desc_event *driver_event;
>>+	};
>>+	union {
>>+		struct vring_used  *used;
>>+		struct vring_packed_desc_event *device_event;
>>+	};
>
>Not a must, but I wonder if it wouldn't be cleaner to have a unique
>union of two structs, one for packed rings, and one for the legacy.
>I may be wrong, though.

I'm not sure either but this looked best to me. 

Thanks!

regards,
Jens 
>
>>  };
>>  /* The standard layout for the ring is a continuous chunk of memory which
>>@@ -95,10 +123,16 @@ struct vring {
>>  #define vring_avail_event(vr) (*(uint16_t *)&(vr)->used->ring[(vr)->num])
>>  static inline size_t
>>-vring_size(unsigned int num, unsigned long align)
>>+vring_size(struct virtio_hw *hw, unsigned int num, unsigned long align)
>>  {
>>  	size_t size;
>>+	if (vtpci_packed_queue(hw)) {
>>+		size = num * sizeof(struct vring_desc_packed);
>>+		size += 2 * sizeof(struct vring_packed_desc_event);
>>+		return size;
>>+	}
>>+
>>  	size = num * sizeof(struct vring_desc);
>>  	size += sizeof(struct vring_avail) + (num * sizeof(uint16_t));
>>  	size = RTE_ALIGN_CEIL(size, align);
>>@@ -108,10 +142,19 @@ vring_size(unsigned int num, unsigned long align)
>>  }
>>  static inline void
>>-vring_init(struct vring *vr, unsigned int num, uint8_t *p,
>>+vring_init(struct virtio_hw *hw, struct vring *vr, unsigned int num, uint8_t *p,
>>  	unsigned long align)
>>  {
>>  	vr->num = num;
>>+	if (vtpci_packed_queue(hw)) {
>>+		vr->desc_packed = (struct vring_desc_packed *)p;
>>+		vr->driver_event = (struct vring_packed_desc_event *) (p +
>>+			num * sizeof(struct vring_desc_packed));
>>+		vr->device_event = (struct vring_packed_desc_event *) (vr->driver_event +
>>+			sizeof(struct vring_packed_desc_event));
>>+		return;
>>+	}
>>+
>>  	vr->desc = (struct vring_desc *) p;
>>  	vr->avail = (struct vring_avail *) (p +
>>  		num * sizeof(struct vring_desc));
>>diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
>>index 14364f356..cc2e7c0f6 100644
>>--- a/drivers/net/virtio/virtqueue.h
>>+++ b/drivers/net/virtio/virtqueue.h
>>@@ -245,6 +245,16 @@ struct virtio_tx_region {
>>  			   __attribute__((__aligned__(16)));
>>  };
>>+static inline void
>>+vring_desc_init_packed(struct vring *vr, int n)
>>+{
>>+	int i;
>>+	for (i = 0; i < n; i++) {
>>+		struct vring_desc_packed *desc = &vr->desc_packed[i];
>>+		desc->index = i;
>>+	}
>>+}
>>+
>>  /* Chain all the descriptors in the ring with an END */
>>  static inline void
>>  vring_desc_init(struct vring_desc *dp, uint16_t n)
>>

Re: [PATCH v3 02/21] net/virtio: vring init for packed queues

[Tiwei Bie]

On Thu, Apr 05, 2018 at 12:10:12PM +0200, Jens Freimann wrote:
> Add and initialize descriptor data structures.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
[...]
> diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
> index 9e3c2a015..1bd7ba98e 100644
> --- a/drivers/net/virtio/virtio_ring.h
> +++ b/drivers/net/virtio/virtio_ring.h
> @@ -9,6 +9,7 @@
>  
>  #include <rte_common.h>
>  
> +

There is no need to add this empty line.

>  /* This marks a buffer as continuing via the next field. */
>  #define VRING_DESC_F_NEXT       1
>  /* This marks a buffer as write-only (otherwise read-only). */
> @@ -54,11 +55,38 @@ struct vring_used {
>  	struct vring_used_elem ring[0];
>  };
>  
> +/* For support of packed virtqueues in Virtio 1.1 the format of descriptors
> + * looks like this.
> + */
> +struct vring_desc_packed {
> +	uint64_t addr;
> +	uint32_t len;
> +	uint16_t index;
> +	uint16_t flags;
> +};
> +
> +#define RING_EVENT_FLAGS_ENABLE 0x0
> +#define RING_EVENT_FLAGS_DISABLE 0x1
> +#define RING_EVENT_FLAGS_DESC 0x2
> +struct vring_packed_desc_event {
> +	uint16_t desc_event_off_wrap;
> +	uint16_t desc_event_flags;
> +};
> +
>  struct vring {
>  	unsigned int num;
> -	struct vring_desc  *desc;
> -	struct vring_avail *avail;
> -	struct vring_used  *used;
> +	union {
> +		struct vring_desc_packed *desc_packed;
> +		struct vring_desc *desc;
> +	};
> +	union {
> +		struct vring_avail *avail;
> +		struct vring_packed_desc_event *driver_event;
> +	};
> +	union {
> +		struct vring_used  *used;
> +		struct vring_packed_desc_event *device_event;
> +	};
>  };
>  
>  /* The standard layout for the ring is a continuous chunk of memory which
> @@ -95,10 +123,16 @@ struct vring {
>  #define vring_avail_event(vr) (*(uint16_t *)&(vr)->used->ring[(vr)->num])
>  
>  static inline size_t
> -vring_size(unsigned int num, unsigned long align)
> +vring_size(struct virtio_hw *hw, unsigned int num, unsigned long align)
>  {
>  	size_t size;
>  
> +	if (vtpci_packed_queue(hw)) {
> +		size = num * sizeof(struct vring_desc_packed);

I think you need to add paddings based on the alignment.

> +		size += 2 * sizeof(struct vring_packed_desc_event);
> +		return size;
> +	}
> +
>  	size = num * sizeof(struct vring_desc);
>  	size += sizeof(struct vring_avail) + (num * sizeof(uint16_t));
>  	size = RTE_ALIGN_CEIL(size, align);
> @@ -108,10 +142,19 @@ vring_size(unsigned int num, unsigned long align)
>  }
>  
>  static inline void
> -vring_init(struct vring *vr, unsigned int num, uint8_t *p,
> +vring_init(struct virtio_hw *hw, struct vring *vr, unsigned int num, uint8_t *p,
>  	unsigned long align)
>  {
>  	vr->num = num;
> +	if (vtpci_packed_queue(hw)) {
> +		vr->desc_packed = (struct vring_desc_packed *)p;
> +		vr->driver_event = (struct vring_packed_desc_event *) (p +
> +			num * sizeof(struct vring_desc_packed));

I think paddings are needed here.

> +		vr->device_event = (struct vring_packed_desc_event *) (vr->driver_event +
> +			sizeof(struct vring_packed_desc_event));
> +		return;
> +	}
> +
>  	vr->desc = (struct vring_desc *) p;
>  	vr->avail = (struct vring_avail *) (p +
>  		num * sizeof(struct vring_desc));
> diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
> index 14364f356..cc2e7c0f6 100644
> --- a/drivers/net/virtio/virtqueue.h
> +++ b/drivers/net/virtio/virtqueue.h
> @@ -245,6 +245,16 @@ struct virtio_tx_region {
>  			   __attribute__((__aligned__(16)));
>  };
>  
> +static inline void
> +vring_desc_init_packed(struct vring *vr, int n)
> +{
> +	int i;
> +	for (i = 0; i < n; i++) {
> +		struct vring_desc_packed *desc = &vr->desc_packed[i];
> +		desc->index = i;
> +	}
> +}
> +
>  /* Chain all the descriptors in the ring with an END */
>  static inline void
>  vring_desc_init(struct vring_desc *dp, uint16_t n)
> -- 
> 2.14.3
> 

Re: [PATCH v3 02/21] net/virtio: vring init for packed queues

[Jens Freimann]

On Thu, Apr 05, 2018 at 10:22:28PM +0800, Tiwei Bie wrote:
>On Thu, Apr 05, 2018 at 12:10:12PM +0200, Jens Freimann wrote:
>> Add and initialize descriptor data structures.
>>
>> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>> ---
>[...]
>> diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
>> index 9e3c2a015..1bd7ba98e 100644
>> --- a/drivers/net/virtio/virtio_ring.h
>> +++ b/drivers/net/virtio/virtio_ring.h
>> @@ -9,6 +9,7 @@
>>
>>  #include <rte_common.h>
>>
>> +
>
>There is no need to add this empty line.
>
>>  /* This marks a buffer as continuing via the next field. */
>>  #define VRING_DESC_F_NEXT       1
>>  /* This marks a buffer as write-only (otherwise read-only). */
>> @@ -54,11 +55,38 @@ struct vring_used {
>>  	struct vring_used_elem ring[0];
>>  };
>>
>> +/* For support of packed virtqueues in Virtio 1.1 the format of descriptors
>> + * looks like this.
>> + */
>> +struct vring_desc_packed {
>> +	uint64_t addr;
>> +	uint32_t len;
>> +	uint16_t index;
>> +	uint16_t flags;
>> +};
>> +
>> +#define RING_EVENT_FLAGS_ENABLE 0x0
>> +#define RING_EVENT_FLAGS_DISABLE 0x1
>> +#define RING_EVENT_FLAGS_DESC 0x2
>> +struct vring_packed_desc_event {
>> +	uint16_t desc_event_off_wrap;
>> +	uint16_t desc_event_flags;
>> +};
>> +
>>  struct vring {
>>  	unsigned int num;
>> -	struct vring_desc  *desc;
>> -	struct vring_avail *avail;
>> -	struct vring_used  *used;
>> +	union {
>> +		struct vring_desc_packed *desc_packed;
>> +		struct vring_desc *desc;
>> +	};
>> +	union {
>> +		struct vring_avail *avail;
>> +		struct vring_packed_desc_event *driver_event;
>> +	};
>> +	union {
>> +		struct vring_used  *used;
>> +		struct vring_packed_desc_event *device_event;
>> +	};
>>  };
>>
>>  /* The standard layout for the ring is a continuous chunk of memory which
>> @@ -95,10 +123,16 @@ struct vring {
>>  #define vring_avail_event(vr) (*(uint16_t *)&(vr)->used->ring[(vr)->num])
>>
>>  static inline size_t
>> -vring_size(unsigned int num, unsigned long align)
>> +vring_size(struct virtio_hw *hw, unsigned int num, unsigned long align)
>>  {
>>  	size_t size;
>>
>> +	if (vtpci_packed_queue(hw)) {
>> +		size = num * sizeof(struct vring_desc_packed);
>
>I think you need to add paddings based on the alignment.
>
>> +		size += 2 * sizeof(struct vring_packed_desc_event);
>> +		return size;
>> +	}
>> +
>>  	size = num * sizeof(struct vring_desc);
>>  	size += sizeof(struct vring_avail) + (num * sizeof(uint16_t));
>>  	size = RTE_ALIGN_CEIL(size, align);
>> @@ -108,10 +142,19 @@ vring_size(unsigned int num, unsigned long align)
>>  }
>>
>>  static inline void
>> -vring_init(struct vring *vr, unsigned int num, uint8_t *p,
>> +vring_init(struct virtio_hw *hw, struct vring *vr, unsigned int num, uint8_t *p,
>>  	unsigned long align)
>>  {
>>  	vr->num = num;
>> +	if (vtpci_packed_queue(hw)) {
>> +		vr->desc_packed = (struct vring_desc_packed *)p;
>> +		vr->driver_event = (struct vring_packed_desc_event *) (p +
>> +			num * sizeof(struct vring_desc_packed));
>
>I think paddings are needed here.

I'll align it with RTE_ALIGN_CEIL, like it's done for the used ring
address.

Thanks!

regards,
Jens 
>
>> +		vr->device_event = (struct vring_packed_desc_event *) (vr->driver_event +
>> +			sizeof(struct vring_packed_desc_event));
>> +		return;
>> +	}
>> +
>>  	vr->desc = (struct vring_desc *) p;
>>  	vr->avail = (struct vring_avail *) (p +
>>  		num * sizeof(struct vring_desc));
>> diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
>> index 14364f356..cc2e7c0f6 100644
>> --- a/drivers/net/virtio/virtqueue.h
>> +++ b/drivers/net/virtio/virtqueue.h
>> @@ -245,6 +245,16 @@ struct virtio_tx_region {
>>  			   __attribute__((__aligned__(16)));
>>  };
>>
>> +static inline void
>> +vring_desc_init_packed(struct vring *vr, int n)
>> +{
>> +	int i;
>> +	for (i = 0; i < n; i++) {
>> +		struct vring_desc_packed *desc = &vr->desc_packed[i];
>> +		desc->index = i;
>> +	}
>> +}
>> +
>>  /* Chain all the descriptors in the ring with an END */
>>  static inline void
>>  vring_desc_init(struct vring_desc *dp, uint16_t n)
>> --
>> 2.14.3
>>

[PATCH v3 03/21] net/virtio: add virtio 1.1 defines

[Jens Freimann]

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 drivers/net/virtio/virtio_ring.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
index 1bd7ba98e..54a11d2a9 100644
--- a/drivers/net/virtio/virtio_ring.h
+++ b/drivers/net/virtio/virtio_ring.h
@@ -16,7 +16,10 @@
 #define VRING_DESC_F_WRITE      2
 /* This means the buffer contains a list of buffer descriptors. */
 #define VRING_DESC_F_INDIRECT   4
-
+/* This flag means the descriptor was made available by the driver */
+#define VRING_DESC_F_AVAIL	(1ULL << 7)
+/* This flag means the descriptor was used by the device */
+#define VRING_DESC_F_USED	(1ULL << 15)
 /* The Host uses this in used->flags to advise the Guest: don't kick me
  * when you add a buffer.  It's unreliable, so it's simply an
  * optimization.  Guest will still kick if it's out of buffers. */
-- 
2.14.3

Re: [PATCH v3 03/21] net/virtio: add virtio 1.1 defines

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   drivers/net/virtio/virtio_ring.h | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
> index 1bd7ba98e..54a11d2a9 100644
> --- a/drivers/net/virtio/virtio_ring.h
> +++ b/drivers/net/virtio/virtio_ring.h
> @@ -16,7 +16,10 @@
>   #define VRING_DESC_F_WRITE      2
>   /* This means the buffer contains a list of buffer descriptors. */
>   #define VRING_DESC_F_INDIRECT   4
> -
> +/* This flag means the descriptor was made available by the driver */
> +#define VRING_DESC_F_AVAIL	(1ULL << 7)
> +/* This flag means the descriptor was used by the device */
> +#define VRING_DESC_F_USED	(1ULL << 15)
>   /* The Host uses this in used->flags to advise the Guest: don't kick me
>    * when you add a buffer.  It's unreliable, so it's simply an
>    * optimization.  Guest will still kick if it's out of buffers. */
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [PATCH v3 03/21] net/virtio: add virtio 1.1 defines

[Tiwei Bie]

On Thu, Apr 05, 2018 at 12:10:13PM +0200, Jens Freimann wrote:
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>  drivers/net/virtio/virtio_ring.h | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
> index 1bd7ba98e..54a11d2a9 100644
> --- a/drivers/net/virtio/virtio_ring.h
> +++ b/drivers/net/virtio/virtio_ring.h
> @@ -16,7 +16,10 @@
>  #define VRING_DESC_F_WRITE      2
>  /* This means the buffer contains a list of buffer descriptors. */
>  #define VRING_DESC_F_INDIRECT   4
> -

Why remove this empty line?

VRING_DESC_F_* and VRING_USED_F_NO_NOTIFY/VRING_AVAIL_F_NO_INTERRUPT
are unrelated. I think we still need this to make the code more
readable.

> +/* This flag means the descriptor was made available by the driver */
> +#define VRING_DESC_F_AVAIL	(1ULL << 7)
> +/* This flag means the descriptor was used by the device */
> +#define VRING_DESC_F_USED	(1ULL << 15)
>  /* The Host uses this in used->flags to advise the Guest: don't kick me
>   * when you add a buffer.  It's unreliable, so it's simply an
>   * optimization.  Guest will still kick if it's out of buffers. */
> -- 
> 2.14.3
> 

Re: [PATCH v3 03/21] net/virtio: add virtio 1.1 defines

[Jens Freimann]

On Thu, Apr 05, 2018 at 10:27:21PM +0800, Tiwei Bie wrote:
>On Thu, Apr 05, 2018 at 12:10:13PM +0200, Jens Freimann wrote:
>> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>> ---
>>  drivers/net/virtio/virtio_ring.h | 5 ++++-
>>  1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
>> index 1bd7ba98e..54a11d2a9 100644
>> --- a/drivers/net/virtio/virtio_ring.h
>> +++ b/drivers/net/virtio/virtio_ring.h
>> @@ -16,7 +16,10 @@
>>  #define VRING_DESC_F_WRITE      2
>>  /* This means the buffer contains a list of buffer descriptors. */
>>  #define VRING_DESC_F_INDIRECT   4
>> -
>
>Why remove this empty line?
>
>VRING_DESC_F_* and VRING_USED_F_NO_NOTIFY/VRING_AVAIL_F_NO_INTERRUPT
>are unrelated. I think we still need this to make the code more
>readable.

Figured it was unnecessary, but I'll add it back.

Thanks!

regards,
Jens 
>
>> +/* This flag means the descriptor was made available by the driver */
>> +#define VRING_DESC_F_AVAIL	(1ULL << 7)
>> +/* This flag means the descriptor was used by the device */
>> +#define VRING_DESC_F_USED	(1ULL << 15)
>>  /* The Host uses this in used->flags to advise the Guest: don't kick me
>>   * when you add a buffer.  It's unreliable, so it's simply an
>>   * optimization.  Guest will still kick if it's out of buffers. */
>> --
>> 2.14.3
>>

[PATCH v3 04/21] net/virtio: add packed virtqueue helpers

[Jens Freimann]

Add helper functions to set/clear and check descriptor flags.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 drivers/net/virtio/virtio_ring.h | 33 +++++++++++++++++++++++++++++++++
 drivers/net/virtio/virtqueue.c   | 10 ++++++++++
 2 files changed, 43 insertions(+)

diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
index 54a11d2a9..663b4b01d 100644
--- a/drivers/net/virtio/virtio_ring.h
+++ b/drivers/net/virtio/virtio_ring.h
@@ -78,6 +78,7 @@ struct vring_packed_desc_event {
 
 struct vring {
 	unsigned int num;
+	unsigned int avail_wrap_counter;
 	union {
 		struct vring_desc_packed *desc_packed;
 		struct vring_desc *desc;
@@ -92,6 +93,38 @@ struct vring {
 	};
 };
 
+static inline void toggle_wrap_counter(struct vring *vr)
+{
+	vr->avail_wrap_counter ^= 1;
+}
+
+static inline void _set_desc_avail(struct vring_desc_packed *desc,
+				   int wrap_counter)
+{
+	uint16_t flags = desc->flags;
+
+	if (wrap_counter) {
+		flags |= VRING_DESC_F_AVAIL;
+		flags &= ~VRING_DESC_F_USED;
+	} else {
+		flags &= ~VRING_DESC_F_AVAIL;
+		flags |= VRING_DESC_F_USED;
+	}
+
+	desc->flags = flags;
+}
+
+static inline void set_desc_avail(struct vring *vr,
+				  struct vring_desc_packed *desc)
+{
+	_set_desc_avail(desc, vr->avail_wrap_counter);
+}
+
+static inline int desc_is_used(struct vring_desc_packed *desc)
+{
+	return !(desc->flags & VRING_DESC_F_AVAIL) == !(desc->flags & VRING_DESC_F_USED);
+}
+
 /* The standard layout for the ring is a continuous chunk of memory which
  * looks like this.  We assume num is a power of 2.
  *
diff --git a/drivers/net/virtio/virtqueue.c b/drivers/net/virtio/virtqueue.c
index a7d0a9cbe..4f95ed5c8 100644
--- a/drivers/net/virtio/virtqueue.c
+++ b/drivers/net/virtio/virtqueue.c
@@ -58,6 +58,7 @@ virtqueue_detach_unused(struct virtqueue *vq)
 void
 virtqueue_rxvq_flush(struct virtqueue *vq)
 {
+	struct vring_desc_packed *descs = vq->vq_ring.desc_packed;
 	struct virtnet_rx *rxq = &vq->rxq;
 	struct virtio_hw *hw = vq->hw;
 	struct vring_used_elem *uep;
@@ -65,6 +66,15 @@ virtqueue_rxvq_flush(struct virtqueue *vq)
 	uint16_t used_idx, desc_idx;
 	uint16_t nb_used, i;
 
+	if (vtpci_packed_queue(vq->hw)) {
+		i = vq->vq_used_cons_idx & (vq->vq_nentries - 1);
+		while (desc_is_used(&descs[i])) {
+			rte_pktmbuf_free(vq->sw_ring[i]);
+			vq->vq_free_cnt++;
+		}
+		return;
+	}
+
 	nb_used = VIRTQUEUE_NUSED(vq);
 
 	for (i = 0; i < nb_used; i++) {
-- 
2.14.3

Re: [PATCH v3 04/21] net/virtio: add packed virtqueue helpers

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Add helper functions to set/clear and check descriptor flags.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   drivers/net/virtio/virtio_ring.h | 33 +++++++++++++++++++++++++++++++++
>   drivers/net/virtio/virtqueue.c   | 10 ++++++++++
>   2 files changed, 43 insertions(+)
> 
> diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
> index 54a11d2a9..663b4b01d 100644
> --- a/drivers/net/virtio/virtio_ring.h
> +++ b/drivers/net/virtio/virtio_ring.h
> @@ -78,6 +78,7 @@ struct vring_packed_desc_event {
>   
>   struct vring {
>   	unsigned int num;
> +	unsigned int avail_wrap_counter;
>   	union {
>   		struct vring_desc_packed *desc_packed;
>   		struct vring_desc *desc;
> @@ -92,6 +93,38 @@ struct vring {
>   	};
>   };
>   
> +static inline void toggle_wrap_counter(struct vring *vr)
Please follow the coding style of the driver:

static inline void
toggle_wrap_counter(struct vring *vr)

I would also prefix the functions with vring_ to keep consistency.
> +{
> +	vr->avail_wrap_counter ^= 1;
> +}
> +
> +static inline void _set_desc_avail(struct vring_desc_packed *desc,
> +				   int wrap_counter)
> +{
> +	uint16_t flags = desc->flags;
> +
> +	if (wrap_counter) {
> +		flags |= VRING_DESC_F_AVAIL;
> +		flags &= ~VRING_DESC_F_USED;
> +	} else {
> +		flags &= ~VRING_DESC_F_AVAIL;
> +		flags |= VRING_DESC_F_USED;
> +	}
> +
> +	desc->flags = flags;
> +}
> +
> +static inline void set_desc_avail(struct vring *vr,
> +				  struct vring_desc_packed *desc)
> +{
> +	_set_desc_avail(desc, vr->avail_wrap_counter);
> +}
> +
> +static inline int desc_is_used(struct vring_desc_packed *desc)
> +{
> +	return !(desc->flags & VRING_DESC_F_AVAIL) == !(desc->flags & VRING_DESC_F_USED);
> +}
> +
>   /* The standard layout for the ring is a continuous chunk of memory which
>    * looks like this.  We assume num is a power of 2.
>    *
> diff --git a/drivers/net/virtio/virtqueue.c b/drivers/net/virtio/virtqueue.c
> index a7d0a9cbe..4f95ed5c8 100644
> --- a/drivers/net/virtio/virtqueue.c
> +++ b/drivers/net/virtio/virtqueue.c
> @@ -58,6 +58,7 @@ virtqueue_detach_unused(struct virtqueue *vq)
>   void
>   virtqueue_rxvq_flush(struct virtqueue *vq)
>   {
> +	struct vring_desc_packed *descs = vq->vq_ring.desc_packed;
>   	struct virtnet_rx *rxq = &vq->rxq;
>   	struct virtio_hw *hw = vq->hw;
>   	struct vring_used_elem *uep;
> @@ -65,6 +66,15 @@ virtqueue_rxvq_flush(struct virtqueue *vq)
>   	uint16_t used_idx, desc_idx;
>   	uint16_t nb_used, i;
>   
> +	if (vtpci_packed_queue(vq->hw)) {
> +		i = vq->vq_used_cons_idx & (vq->vq_nentries - 1);
> +		while (desc_is_used(&descs[i])) {
> +			rte_pktmbuf_free(vq->sw_ring[i]);
> +			vq->vq_free_cnt++;
> +		}
> +		return;
> +	}
> +
>   	nb_used = VIRTQUEUE_NUSED(vq);
>   
>   	for (i = 0; i < nb_used; i++) {
> 

With style comments taken into account:

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Re: [PATCH v3 04/21] net/virtio: add packed virtqueue helpers

[Jens Freimann]

On Thu, Apr 05, 2018 at 04:27:51PM +0200, Maxime Coquelin wrote:
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>Add helper functions to set/clear and check descriptor flags.
>>
>>Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>>---
>>  drivers/net/virtio/virtio_ring.h | 33 +++++++++++++++++++++++++++++++++
>>  drivers/net/virtio/virtqueue.c   | 10 ++++++++++
>>  2 files changed, 43 insertions(+)
>>
>>diff --git a/drivers/net/virtio/virtio_ring.h b/drivers/net/virtio/virtio_ring.h
>>index 54a11d2a9..663b4b01d 100644
>>--- a/drivers/net/virtio/virtio_ring.h
>>+++ b/drivers/net/virtio/virtio_ring.h
>>@@ -78,6 +78,7 @@ struct vring_packed_desc_event {
>>  struct vring {
>>  	unsigned int num;
>>+	unsigned int avail_wrap_counter;
>>  	union {
>>  		struct vring_desc_packed *desc_packed;
>>  		struct vring_desc *desc;
>>@@ -92,6 +93,38 @@ struct vring {
>>  	};
>>  };
>>+static inline void toggle_wrap_counter(struct vring *vr)
>Please follow the coding style of the driver:
>
>static inline void
>toggle_wrap_counter(struct vring *vr)
>
>I would also prefix the functions with vring_ to keep consistency.

ok

>>+{
>>+	vr->avail_wrap_counter ^= 1;
>>+}
>>+
>>+static inline void _set_desc_avail(struct vring_desc_packed *desc,
>>+				   int wrap_counter)
>>+{
>>+	uint16_t flags = desc->flags;
>>+
>>+	if (wrap_counter) {
>>+		flags |= VRING_DESC_F_AVAIL;
>>+		flags &= ~VRING_DESC_F_USED;
>>+	} else {
>>+		flags &= ~VRING_DESC_F_AVAIL;
>>+		flags |= VRING_DESC_F_USED;
>>+	}
>>+
>>+	desc->flags = flags;
>>+}
>>+
>>+static inline void set_desc_avail(struct vring *vr,
>>+				  struct vring_desc_packed *desc)
>>+{
>>+	_set_desc_avail(desc, vr->avail_wrap_counter);
>>+}
>>+
>>+static inline int desc_is_used(struct vring_desc_packed *desc)
>>+{
>>+	return !(desc->flags & VRING_DESC_F_AVAIL) == !(desc->flags & VRING_DESC_F_USED);
>>+}
>>+
>>  /* The standard layout for the ring is a continuous chunk of memory which
>>   * looks like this.  We assume num is a power of 2.
>>   *
>>diff --git a/drivers/net/virtio/virtqueue.c b/drivers/net/virtio/virtqueue.c
>>index a7d0a9cbe..4f95ed5c8 100644
>>--- a/drivers/net/virtio/virtqueue.c
>>+++ b/drivers/net/virtio/virtqueue.c
>>@@ -58,6 +58,7 @@ virtqueue_detach_unused(struct virtqueue *vq)
>>  void
>>  virtqueue_rxvq_flush(struct virtqueue *vq)
>>  {
>>+	struct vring_desc_packed *descs = vq->vq_ring.desc_packed;
>>  	struct virtnet_rx *rxq = &vq->rxq;
>>  	struct virtio_hw *hw = vq->hw;
>>  	struct vring_used_elem *uep;
>>@@ -65,6 +66,15 @@ virtqueue_rxvq_flush(struct virtqueue *vq)
>>  	uint16_t used_idx, desc_idx;
>>  	uint16_t nb_used, i;
>>+	if (vtpci_packed_queue(vq->hw)) {
>>+		i = vq->vq_used_cons_idx & (vq->vq_nentries - 1);
>>+		while (desc_is_used(&descs[i])) {
>>+			rte_pktmbuf_free(vq->sw_ring[i]);
>>+			vq->vq_free_cnt++;
>>+		}
>>+		return;
>>+	}
>>+
>>  	nb_used = VIRTQUEUE_NUSED(vq);
>>  	for (i = 0; i < nb_used; i++) {
>>
>
>With style comments taken into account:

I'll fix it.

Thanks!
regards,
Jens 
>
>Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>

Re: [PATCH v3 04/21] net/virtio: add packed virtqueue helpers

[Tiwei Bie]

On Thu, Apr 05, 2018 at 04:27:51PM +0200, Maxime Coquelin wrote:
> On 04/05/2018 12:10 PM, Jens Freimann wrote:
> > Add helper functions to set/clear and check descriptor flags.
> > 
> > Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> > ---
> >   drivers/net/virtio/virtio_ring.h | 33 +++++++++++++++++++++++++++++++++
> >   drivers/net/virtio/virtqueue.c   | 10 ++++++++++
> >   2 files changed, 43 insertions(+)
[...]
> > diff --git a/drivers/net/virtio/virtqueue.c b/drivers/net/virtio/virtqueue.c
> > index a7d0a9cbe..4f95ed5c8 100644
> > --- a/drivers/net/virtio/virtqueue.c
> > +++ b/drivers/net/virtio/virtqueue.c
> > @@ -58,6 +58,7 @@ virtqueue_detach_unused(struct virtqueue *vq)
> >   void
> >   virtqueue_rxvq_flush(struct virtqueue *vq)
> >   {
> > +	struct vring_desc_packed *descs = vq->vq_ring.desc_packed;
> >   	struct virtnet_rx *rxq = &vq->rxq;
> >   	struct virtio_hw *hw = vq->hw;
> >   	struct vring_used_elem *uep;
> > @@ -65,6 +66,15 @@ virtqueue_rxvq_flush(struct virtqueue *vq)
> >   	uint16_t used_idx, desc_idx;
> >   	uint16_t nb_used, i;
> > +	if (vtpci_packed_queue(vq->hw)) {
> > +		i = vq->vq_used_cons_idx & (vq->vq_nentries - 1);
> > +		while (desc_is_used(&descs[i])) {
> > +			rte_pktmbuf_free(vq->sw_ring[i]);
> > +			vq->vq_free_cnt++;
> > +		}

You need to increase i here. Otherwise you are looping
on the same entry. To increase i, you also need to keep
track of the number of descriptors need to be skipped
for each used descriptor.

> > +		return;
> > +	}
> > +
> >   	nb_used = VIRTQUEUE_NUSED(vq);
> >   	for (i = 0; i < nb_used; i++) {
> > 
> 
> With style comments taken into account:
> 
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> 

Re: [PATCH v3 04/21] net/virtio: add packed virtqueue helpers

[Jens Freimann]

On Thu, Apr 05, 2018 at 10:40:29PM +0800, Tiwei Bie wrote:
>On Thu, Apr 05, 2018 at 04:27:51PM +0200, Maxime Coquelin wrote:
>> On 04/05/2018 12:10 PM, Jens Freimann wrote:
>> > Add helper functions to set/clear and check descriptor flags.
>> >
>> > Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>> > ---
>> >   drivers/net/virtio/virtio_ring.h | 33 +++++++++++++++++++++++++++++++++
>> >   drivers/net/virtio/virtqueue.c   | 10 ++++++++++
>> >   2 files changed, 43 insertions(+)
>[...]
>> > diff --git a/drivers/net/virtio/virtqueue.c b/drivers/net/virtio/virtqueue.c
>> > index a7d0a9cbe..4f95ed5c8 100644
>> > --- a/drivers/net/virtio/virtqueue.c
>> > +++ b/drivers/net/virtio/virtqueue.c
>> > @@ -58,6 +58,7 @@ virtqueue_detach_unused(struct virtqueue *vq)
>> >   void
>> >   virtqueue_rxvq_flush(struct virtqueue *vq)
>> >   {
>> > +	struct vring_desc_packed *descs = vq->vq_ring.desc_packed;
>> >   	struct virtnet_rx *rxq = &vq->rxq;
>> >   	struct virtio_hw *hw = vq->hw;
>> >   	struct vring_used_elem *uep;
>> > @@ -65,6 +66,15 @@ virtqueue_rxvq_flush(struct virtqueue *vq)
>> >   	uint16_t used_idx, desc_idx;
>> >   	uint16_t nb_used, i;
>> > +	if (vtpci_packed_queue(vq->hw)) {
>> > +		i = vq->vq_used_cons_idx & (vq->vq_nentries - 1);
>> > +		while (desc_is_used(&descs[i])) {
>> > +			rte_pktmbuf_free(vq->sw_ring[i]);
>> > +			vq->vq_free_cnt++;
>> > +		}
>
>You need to increase i here. Otherwise you are looping
>on the same entry. To increase i, you also need to keep
>track of the number of descriptors need to be skipped
>for each used descriptor.

Hmm, you are right. Seems like this code never ran. I'll be more
careful testing even last minute changes now. 

regards,
Jens 
>
>> > +		return;
>> > +	}
>> > +
>> >   	nb_used = VIRTQUEUE_NUSED(vq);
>> >   	for (i = 0; i < nb_used; i++) {
>> >
>>
>> With style comments taken into account:
>>
>> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>>

[PATCH v3 05/21] net/virtio: dump packed virtqueue data

[Jens Freimann]

Add support to dump packed virtqueue data to the
VIRTQUEUE_DUMP() macro.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 drivers/net/virtio/virtqueue.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
index cc2e7c0f6..7e265bf93 100644
--- a/drivers/net/virtio/virtqueue.h
+++ b/drivers/net/virtio/virtqueue.h
@@ -360,6 +360,13 @@ virtqueue_notify(struct virtqueue *vq)
 
 #ifdef RTE_LIBRTE_VIRTIO_DEBUG_DUMP
 #define VIRTQUEUE_DUMP(vq) do { \
+       if (vtpci_packed_queue((vq)->hw)) { \
+         PMD_INIT_LOG(DEBUG, \
+         "VQ: - size=%d; free=%d; last_used_idx=%d;" \
+         (vq)->vq_nentries, (vq)->vq_free_cnt, nused); \
+         break; \
+         } \
+	if (vtpci_packed_queue((vq)->hw)) break; \
 	uint16_t used_idx, nused; \
 	used_idx = (vq)->vq_ring.used->idx; \
 	nused = (uint16_t)(used_idx - (vq)->vq_used_cons_idx); \
-- 
2.14.3

Re: [PATCH v3 05/21] net/virtio: dump packed virtqueue data

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Add support to dump packed virtqueue data to the
> VIRTQUEUE_DUMP() macro.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   drivers/net/virtio/virtqueue.h | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
> index cc2e7c0f6..7e265bf93 100644
> --- a/drivers/net/virtio/virtqueue.h
> +++ b/drivers/net/virtio/virtqueue.h
> @@ -360,6 +360,13 @@ virtqueue_notify(struct virtqueue *vq)
>   
>   #ifdef RTE_LIBRTE_VIRTIO_DEBUG_DUMP
>   #define VIRTQUEUE_DUMP(vq) do { \
> +       if (vtpci_packed_queue((vq)->hw)) { \
> +         PMD_INIT_LOG(DEBUG, \
> +         "VQ: - size=%d; free=%d; last_used_idx=%d;" \
> +         (vq)->vq_nentries, (vq)->vq_free_cnt, nused); \
> +         break; \
> +         } \
> +	if (vtpci_packed_queue((vq)->hw)) break; \
>   	uint16_t used_idx, nused; \
>   	used_idx = (vq)->vq_ring.used->idx; \
>   	nused = (uint16_t)(used_idx - (vq)->vq_used_cons_idx); \
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [PATCH v3 05/21] net/virtio: dump packed virtqueue data

[Tiwei Bie]

On Thu, Apr 05, 2018 at 12:10:15PM +0200, Jens Freimann wrote:
> Add support to dump packed virtqueue data to the
> VIRTQUEUE_DUMP() macro.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>  drivers/net/virtio/virtqueue.h | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
> index cc2e7c0f6..7e265bf93 100644
> --- a/drivers/net/virtio/virtqueue.h
> +++ b/drivers/net/virtio/virtqueue.h
> @@ -360,6 +360,13 @@ virtqueue_notify(struct virtqueue *vq)
>  
>  #ifdef RTE_LIBRTE_VIRTIO_DEBUG_DUMP
>  #define VIRTQUEUE_DUMP(vq) do { \
> +       if (vtpci_packed_queue((vq)->hw)) { \
> +         PMD_INIT_LOG(DEBUG, \
> +         "VQ: - size=%d; free=%d; last_used_idx=%d;" \

You missed the `,` after the format string.

> +         (vq)->vq_nentries, (vq)->vq_free_cnt, nused); \

There is no way to make sure that nused (the last param
in PMD_INIT_LOG()) will be defined and have the meaning
of last_used_idx. It's not the right way to use a variable
in MACRO.

> +         break; \
> +         } \
> +	if (vtpci_packed_queue((vq)->hw)) break; \

Why do the check twice?


>  	uint16_t used_idx, nused; \
>  	used_idx = (vq)->vq_ring.used->idx; \
>  	nused = (uint16_t)(used_idx - (vq)->vq_used_cons_idx); \
> -- 
> 2.14.3
> 

[PATCH v3 06/21] net/virtio-user: add option to use packed queues

[Jens Freimann]

From: Yuanhan Liu <yuanhan.liu@linux.intel.com>

Add option to enable packed queue support for virtio-user
devices.

Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
 drivers/net/virtio/virtio_user/virtio_user_dev.c | 12 ++++++++++--
 drivers/net/virtio/virtio_user/virtio_user_dev.h |  3 ++-
 drivers/net/virtio/virtio_user_ethdev.c          | 15 ++++++++++++++-
 3 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/drivers/net/virtio/virtio_user/virtio_user_dev.c b/drivers/net/virtio/virtio_user/virtio_user_dev.c
index f90fee9e5..e5e7af185 100644
--- a/drivers/net/virtio/virtio_user/virtio_user_dev.c
+++ b/drivers/net/virtio/virtio_user/virtio_user_dev.c
@@ -314,11 +314,13 @@ virtio_user_dev_setup(struct virtio_user_dev *dev)
 	 1ULL << VIRTIO_NET_F_GUEST_CSUM	|	\
 	 1ULL << VIRTIO_NET_F_GUEST_TSO4	|	\
 	 1ULL << VIRTIO_NET_F_GUEST_TSO6	|	\
-	 1ULL << VIRTIO_F_VERSION_1)
+	 1ULL << VIRTIO_F_VERSION_1		|	\
+	 1ULL << VIRTIO_F_RING_PACKED)
 
 int
 virtio_user_dev_init(struct virtio_user_dev *dev, char *path, int queues,
-		     int cq, int queue_size, const char *mac, char **ifname)
+		     int cq, int queue_size, const char *mac, char **ifname,
+		     int packed_vq)
 {
 	snprintf(dev->path, PATH_MAX, "%s", path);
 	dev->max_queue_pairs = queues;
@@ -347,6 +349,12 @@ virtio_user_dev_init(struct virtio_user_dev *dev, char *path, int queues,
 		PMD_INIT_LOG(ERR, "get_features failed: %s", strerror(errno));
 		return -1;
 	}
+
+	if (packed_vq)
+		dev->device_features |= (1ull << VIRTIO_F_RING_PACKED);
+	else
+		dev->device_features &= ~(1ull << VIRTIO_F_RING_PACKED);
+
 	if (dev->mac_specified)
 		dev->device_features |= (1ull << VIRTIO_NET_F_MAC);
 
diff --git a/drivers/net/virtio/virtio_user/virtio_user_dev.h b/drivers/net/virtio/virtio_user/virtio_user_dev.h
index 5f8755771..50c60c8a7 100644
--- a/drivers/net/virtio/virtio_user/virtio_user_dev.h
+++ b/drivers/net/virtio/virtio_user/virtio_user_dev.h
@@ -42,7 +42,8 @@ int is_vhost_user_by_type(const char *path);
 int virtio_user_start_device(struct virtio_user_dev *dev);
 int virtio_user_stop_device(struct virtio_user_dev *dev);
 int virtio_user_dev_init(struct virtio_user_dev *dev, char *path, int queues,
-			 int cq, int queue_size, const char *mac, char **ifname);
+			 int cq, int queue_size, const char *mac, char **ifname,
+			 int packed_vq);
 void virtio_user_dev_uninit(struct virtio_user_dev *dev);
 void virtio_user_handle_cq(struct virtio_user_dev *dev, uint16_t queue_idx);
 #endif
diff --git a/drivers/net/virtio/virtio_user_ethdev.c b/drivers/net/virtio/virtio_user_ethdev.c
index 263649006..1c6086f14 100644
--- a/drivers/net/virtio/virtio_user_ethdev.c
+++ b/drivers/net/virtio/virtio_user_ethdev.c
@@ -278,6 +278,8 @@ static const char *valid_args[] = {
 	VIRTIO_USER_ARG_QUEUE_SIZE,
 #define VIRTIO_USER_ARG_INTERFACE_NAME "iface"
 	VIRTIO_USER_ARG_INTERFACE_NAME,
+#define VIRTIO_USER_ARG_PACKED_VQ "packed_vq"
+	VIRTIO_USER_ARG_PACKED_VQ,
 	NULL
 };
 
@@ -382,6 +384,7 @@ virtio_user_pmd_probe(struct rte_vdev_device *dev)
 	char *ifname = NULL;
 	char *mac_addr = NULL;
 	int ret = -1;
+	uint64_t packed_vq = 0;
 
 	kvlist = rte_kvargs_parse(rte_vdev_device_args(dev), valid_args);
 	if (!kvlist) {
@@ -456,6 +459,15 @@ virtio_user_pmd_probe(struct rte_vdev_device *dev)
 		cq = 1;
 	}
 
+	if (rte_kvargs_count(kvlist, VIRTIO_USER_ARG_PACKED_VQ) == 1) {
+		if (rte_kvargs_process(kvlist, VIRTIO_USER_ARG_PACKED_VQ,
+				       &get_integer_arg, &packed_vq) < 0) {
+			PMD_INIT_LOG(ERR, "error to parse %s",
+				     VIRTIO_USER_ARG_PACKED_VQ);
+			goto end;
+		}
+	}
+
 	if (queues > 1 && cq == 0) {
 		PMD_INIT_LOG(ERR, "multi-q requires ctrl-q");
 		goto end;
@@ -477,7 +489,8 @@ virtio_user_pmd_probe(struct rte_vdev_device *dev)
 
 		hw = eth_dev->data->dev_private;
 		if (virtio_user_dev_init(hw->virtio_user_dev, path, queues, cq,
-				 queue_size, mac_addr, &ifname) < 0) {
+				 queue_size, mac_addr, &ifname,
+				 packed_vq) < 0) {
 			PMD_INIT_LOG(ERR, "virtio_user_dev_init fails");
 			virtio_user_eth_dev_free(eth_dev);
 			goto end;
-- 
2.14.3

Re: [PATCH v3 06/21] net/virtio-user: add option to use packed queues

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> From: Yuanhan Liu <yuanhan.liu@linux.intel.com>
> 
> Add option to enable packed queue support for virtio-user
> devices.
> 
> Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
> ---
>   drivers/net/virtio/virtio_user/virtio_user_dev.c | 12 ++++++++++--
>   drivers/net/virtio/virtio_user/virtio_user_dev.h |  3 ++-
>   drivers/net/virtio/virtio_user_ethdev.c          | 15 ++++++++++++++-
>   3 files changed, 26 insertions(+), 4 deletions(-)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 07/21] net/virtio: implement transmit path for packed queues

[Jens Freimann]

This implements the transmit path for devices with
support for Virtio 1.1.

Add the feature bit for Virtio 1.1 and enable code to
add buffers to vring and mark descriptors as available.

This is based on a patch by Yuanhan Liu.

Signed-off-by: Jens Freiman <jfreimann@redhat.com>
---
 drivers/net/virtio/virtio_ethdev.c |   8 ++-
 drivers/net/virtio/virtio_ethdev.h |   3 ++
 drivers/net/virtio/virtio_rxtx.c   | 102 ++++++++++++++++++++++++++++++++++++-
 3 files changed, 111 insertions(+), 2 deletions(-)

diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index cccefafe9..089a161ac 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -383,6 +383,8 @@ virtio_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_queue_idx)
 	vq->hw = hw;
 	vq->vq_queue_index = vtpci_queue_idx;
 	vq->vq_nentries = vq_size;
+	if (vtpci_packed_queue(hw))
+		vq->vq_ring.avail_wrap_counter = 1;
 
 	/*
 	 * Reserve a memzone for vring elements
@@ -1328,7 +1330,11 @@ set_rxtx_funcs(struct rte_eth_dev *eth_dev)
 		eth_dev->rx_pkt_burst = &virtio_recv_pkts;
 	}
 
-	if (hw->use_simple_tx) {
+	if (vtpci_packed_queue(hw)) {
+		PMD_INIT_LOG(INFO, "virtio: using virtio 1.1 Tx path on port %u",
+			eth_dev->data->port_id);
+		eth_dev->tx_pkt_burst = virtio_xmit_pkts_packed;
+	} else if (hw->use_simple_tx) {
 		PMD_INIT_LOG(INFO, "virtio: using simple Tx path on port %u",
 			eth_dev->data->port_id);
 		eth_dev->tx_pkt_burst = virtio_xmit_pkts_simple;
diff --git a/drivers/net/virtio/virtio_ethdev.h b/drivers/net/virtio/virtio_ethdev.h
index bb40064ea..d457013cb 100644
--- a/drivers/net/virtio/virtio_ethdev.h
+++ b/drivers/net/virtio/virtio_ethdev.h
@@ -36,6 +36,7 @@
 	 1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE |	\
 	 1u << VIRTIO_RING_F_INDIRECT_DESC |    \
 	 1ULL << VIRTIO_F_VERSION_1       |	\
+	 1ULL << VIRTIO_F_RING_PACKED     |	\
 	 1ULL << VIRTIO_F_IOMMU_PLATFORM)
 
 #define VIRTIO_PMD_SUPPORTED_GUEST_FEATURES	\
@@ -85,6 +86,8 @@ uint16_t virtio_recv_mergeable_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 
 uint16_t virtio_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
 		uint16_t nb_pkts);
+uint16_t virtio_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts,
+		uint16_t nb_pkts);
 
 uint16_t virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,
 		uint16_t nb_pkts);
diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
index a8aa87b32..9f9b5a8f8 100644
--- a/drivers/net/virtio/virtio_rxtx.c
+++ b/drivers/net/virtio/virtio_rxtx.c
@@ -38,6 +38,101 @@
 #define  VIRTIO_DUMP_PACKET(m, len) do { } while (0)
 #endif
 
+#define VIRTIO_SIMPLE_FLAGS ((uint32_t)ETH_TXQ_FLAGS_NOMULTSEGS | \
+	ETH_TXQ_FLAGS_NOOFFLOADS)
+
+/* Cleanup from completed transmits. */
+static void
+virtio_xmit_cleanup_packed(struct virtqueue *vq)
+{
+	uint16_t idx;
+	uint16_t size = vq->vq_nentries;
+	struct vring_desc_packed *desc = vq->vq_ring.desc_packed;
+
+	idx = vq->vq_used_cons_idx & (size - 1);
+	while (desc_is_used(&desc[idx]) &&
+	       vq->vq_free_cnt < size) {
+		vq->vq_free_cnt++;
+		idx = ++vq->vq_used_cons_idx & (size - 1);
+	}
+}
+
+uint16_t
+virtio_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts,
+		     uint16_t nb_pkts)
+{
+	struct virtnet_tx *txvq = tx_queue;
+	struct virtqueue *vq = txvq->vq;
+	uint16_t i;
+	struct vring_desc_packed *desc = vq->vq_ring.desc_packed;
+	uint16_t idx;
+	struct vq_desc_extra *dxp;
+
+	if (unlikely(nb_pkts < 1))
+		return nb_pkts;
+
+	PMD_TX_LOG(DEBUG, "%d packets to xmit", nb_pkts);
+
+	if (likely(vq->vq_free_cnt < vq->vq_free_thresh))
+		virtio_xmit_cleanup_packed(vq);
+
+	for (i = 0; i < nb_pkts; i++) {
+		struct rte_mbuf *txm = tx_pkts[i];
+		struct virtio_tx_region *txr = txvq->virtio_net_hdr_mz->addr;
+		uint16_t head_idx;
+		int wrap_counter;
+
+		if (unlikely(txm->nb_segs + 1 > vq->vq_free_cnt)) {
+			virtio_xmit_cleanup_packed(vq);
+
+			if (unlikely(txm->nb_segs + 1 > vq->vq_free_cnt)) {
+				PMD_TX_LOG(ERR,
+					   "No free tx descriptors to transmit");
+				break;
+			}
+		}
+
+		txvq->stats.bytes += txm->pkt_len;
+
+		vq->vq_free_cnt -= txm->nb_segs + 1;
+
+		idx = (vq->vq_avail_idx++) & (vq->vq_nentries - 1);
+		head_idx = idx;
+		wrap_counter = vq->vq_ring.avail_wrap_counter;
+
+		if ((vq->vq_avail_idx & (vq->vq_nentries - 1)) == 0)
+			toggle_wrap_counter(&vq->vq_ring);
+
+		dxp = &vq->vq_descx[idx];
+		if (dxp->cookie != NULL)
+			rte_pktmbuf_free(dxp->cookie);
+		dxp->cookie = txm;
+
+		desc[idx].addr  = txvq->virtio_net_hdr_mem +
+				  RTE_PTR_DIFF(&txr[idx].tx_hdr, txr);
+		desc[idx].len   = vq->hw->vtnet_hdr_size;
+		desc[idx].flags |= VRING_DESC_F_NEXT;
+
+		do {
+			idx = (vq->vq_avail_idx++) & (vq->vq_nentries - 1);
+			desc[idx].addr  = VIRTIO_MBUF_DATA_DMA_ADDR(txm, vq);
+			desc[idx].len   = txm->data_len;
+			desc[idx].flags |= VRING_DESC_F_NEXT;
+			if (idx == (vq->vq_nentries - 1))
+				toggle_wrap_counter(&vq->vq_ring);
+		} while ((txm = txm->next) != NULL);
+
+		desc[idx].flags &= ~VRING_DESC_F_NEXT;
+
+		rte_smp_wmb();
+		_set_desc_avail(&desc[head_idx], wrap_counter);
+	}
+
+	txvq->stats.packets += i;
+	txvq->stats.errors  += nb_pkts - i;
+
+	return i;
+}
 int
 virtio_dev_rx_queue_done(void *rxq, uint16_t offset)
 {
@@ -547,6 +642,10 @@ virtio_dev_tx_queue_setup_finish(struct rte_eth_dev *dev,
 
 	PMD_INIT_FUNC_TRACE();
 
+	if (vtpci_packed_queue(hw)) {
+		vq->vq_ring.avail_wrap_counter = 1;
+	}
+
 	if (hw->use_simple_tx) {
 		for (desc_idx = 0; desc_idx < mid_idx; desc_idx++) {
 			vq->vq_ring.avail->ring[desc_idx] =
@@ -567,7 +666,8 @@ virtio_dev_tx_queue_setup_finish(struct rte_eth_dev *dev,
 			vq->vq_ring.avail->ring[desc_idx] = desc_idx;
 	}
 
-	VIRTQUEUE_DUMP(vq);
+	if (!vtpci_packed_queue(hw))
+		VIRTQUEUE_DUMP(vq);
 
 	return 0;
 }
-- 
2.14.3

Re: [PATCH v3 07/21] net/virtio: implement transmit path for packed queues

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> This implements the transmit path for devices with
> support for Virtio 1.1.
> 
> Add the feature bit for Virtio 1.1 and enable code to
> add buffers to vring and mark descriptors as available.
> 
> This is based on a patch by Yuanhan Liu.
> 
> Signed-off-by: Jens Freiman <jfreimann@redhat.com>
> ---
>   drivers/net/virtio/virtio_ethdev.c |   8 ++-
>   drivers/net/virtio/virtio_ethdev.h |   3 ++
>   drivers/net/virtio/virtio_rxtx.c   | 102 ++++++++++++++++++++++++++++++++++++-
>   3 files changed, 111 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
> index cccefafe9..089a161ac 100644
> --- a/drivers/net/virtio/virtio_ethdev.c
> +++ b/drivers/net/virtio/virtio_ethdev.c
> @@ -383,6 +383,8 @@ virtio_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_queue_idx)
>   	vq->hw = hw;
>   	vq->vq_queue_index = vtpci_queue_idx;
>   	vq->vq_nentries = vq_size;
> +	if (vtpci_packed_queue(hw))
> +		vq->vq_ring.avail_wrap_counter = 1;
>   
>   	/*
>   	 * Reserve a memzone for vring elements
> @@ -1328,7 +1330,11 @@ set_rxtx_funcs(struct rte_eth_dev *eth_dev)
>   		eth_dev->rx_pkt_burst = &virtio_recv_pkts;
>   	}
>   
> -	if (hw->use_simple_tx) {
> +	if (vtpci_packed_queue(hw)) {
> +		PMD_INIT_LOG(INFO, "virtio: using virtio 1.1 Tx path on port %u",
> +			eth_dev->data->port_id);
> +		eth_dev->tx_pkt_burst = virtio_xmit_pkts_packed;
> +	} else if (hw->use_simple_tx) {
>   		PMD_INIT_LOG(INFO, "virtio: using simple Tx path on port %u",
>   			eth_dev->data->port_id);
>   		eth_dev->tx_pkt_burst = virtio_xmit_pkts_simple;
> diff --git a/drivers/net/virtio/virtio_ethdev.h b/drivers/net/virtio/virtio_ethdev.h
> index bb40064ea..d457013cb 100644
> --- a/drivers/net/virtio/virtio_ethdev.h
> +++ b/drivers/net/virtio/virtio_ethdev.h
> @@ -36,6 +36,7 @@
>   	 1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE |	\
>   	 1u << VIRTIO_RING_F_INDIRECT_DESC |    \
>   	 1ULL << VIRTIO_F_VERSION_1       |	\
> +	 1ULL << VIRTIO_F_RING_PACKED     |	\

Should it really advertise VIRTIO_F_RING_PACKED unconditionally, as it
is not yet fully supported? (non-pow2, indirect descs, etc...)

>   	 1ULL << VIRTIO_F_IOMMU_PLATFORM)
>   
>   #define VIRTIO_PMD_SUPPORTED_GUEST_FEATURES	\

Re: [PATCH v3 07/21] net/virtio: implement transmit path for packed queues

[Jens Freimann]

On Fri, Apr 06, 2018 at 09:56:06AM +0200, Maxime Coquelin wrote:
>
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>This implements the transmit path for devices with
>>support for Virtio 1.1.
>>
>>Add the feature bit for Virtio 1.1 and enable code to
>>add buffers to vring and mark descriptors as available.
>>
>>This is based on a patch by Yuanhan Liu.
>>
>>Signed-off-by: Jens Freiman <jfreimann@redhat.com>
>>---
>>  drivers/net/virtio/virtio_ethdev.c |   8 ++-
>>  drivers/net/virtio/virtio_ethdev.h |   3 ++
>>  drivers/net/virtio/virtio_rxtx.c   | 102 ++++++++++++++++++++++++++++++++++++-
>>  3 files changed, 111 insertions(+), 2 deletions(-)
>>
>>diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
>>index cccefafe9..089a161ac 100644
>>--- a/drivers/net/virtio/virtio_ethdev.c
>>+++ b/drivers/net/virtio/virtio_ethdev.c
>>@@ -383,6 +383,8 @@ virtio_init_queue(struct rte_eth_dev *dev, uint16_t vtpci_queue_idx)
>>  	vq->hw = hw;
>>  	vq->vq_queue_index = vtpci_queue_idx;
>>  	vq->vq_nentries = vq_size;
>>+	if (vtpci_packed_queue(hw))
>>+		vq->vq_ring.avail_wrap_counter = 1;
>>  	/*
>>  	 * Reserve a memzone for vring elements
>>@@ -1328,7 +1330,11 @@ set_rxtx_funcs(struct rte_eth_dev *eth_dev)
>>  		eth_dev->rx_pkt_burst = &virtio_recv_pkts;
>>  	}
>>-	if (hw->use_simple_tx) {
>>+	if (vtpci_packed_queue(hw)) {
>>+		PMD_INIT_LOG(INFO, "virtio: using virtio 1.1 Tx path on port %u",
>>+			eth_dev->data->port_id);
>>+		eth_dev->tx_pkt_burst = virtio_xmit_pkts_packed;
>>+	} else if (hw->use_simple_tx) {
>>  		PMD_INIT_LOG(INFO, "virtio: using simple Tx path on port %u",
>>  			eth_dev->data->port_id);
>>  		eth_dev->tx_pkt_burst = virtio_xmit_pkts_simple;
>>diff --git a/drivers/net/virtio/virtio_ethdev.h b/drivers/net/virtio/virtio_ethdev.h
>>index bb40064ea..d457013cb 100644
>>--- a/drivers/net/virtio/virtio_ethdev.h
>>+++ b/drivers/net/virtio/virtio_ethdev.h
>>@@ -36,6 +36,7 @@
>>  	 1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE |	\
>>  	 1u << VIRTIO_RING_F_INDIRECT_DESC |    \
>>  	 1ULL << VIRTIO_F_VERSION_1       |	\
>>+	 1ULL << VIRTIO_F_RING_PACKED     |	\
>
>Should it really advertise VIRTIO_F_RING_PACKED unconditionally, as it
>is not yet fully supported? (non-pow2, indirect descs, etc...)

We can advertise packed ring but have VIRTIO_F_INDIRECT_DESC disabled.
non-pow2 needs to be integrated thoug and will be in v4.

regards,
Jens 
>
>>  	 1ULL << VIRTIO_F_IOMMU_PLATFORM)
>>  #define VIRTIO_PMD_SUPPORTED_GUEST_FEATURES	\

Re: [PATCH v3 07/21] net/virtio: implement transmit path for packed queues

[Tiwei Bie]

On Thu, Apr 05, 2018 at 12:10:17PM +0200, Jens Freimann wrote:
[...]
> diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
> index a8aa87b32..9f9b5a8f8 100644
> --- a/drivers/net/virtio/virtio_rxtx.c
> +++ b/drivers/net/virtio/virtio_rxtx.c
> @@ -38,6 +38,101 @@
>  #define  VIRTIO_DUMP_PACKET(m, len) do { } while (0)
>  #endif
>  
> +#define VIRTIO_SIMPLE_FLAGS ((uint32_t)ETH_TXQ_FLAGS_NOMULTSEGS | \
> +	ETH_TXQ_FLAGS_NOOFFLOADS)

Why add this?


> +
> +/* Cleanup from completed transmits. */
> +static void
> +virtio_xmit_cleanup_packed(struct virtqueue *vq)
> +{
> +	uint16_t idx;
> +	uint16_t size = vq->vq_nentries;
> +	struct vring_desc_packed *desc = vq->vq_ring.desc_packed;
> +
> +	idx = vq->vq_used_cons_idx & (size - 1);
> +	while (desc_is_used(&desc[idx]) &&
> +	       vq->vq_free_cnt < size) {
> +		vq->vq_free_cnt++;
> +		idx = ++vq->vq_used_cons_idx & (size - 1);

Driver needs to keep track of the number of descriptors
to be skipped for each used descriptor.


> +	}
> +}
> +
> +uint16_t
> +virtio_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts,
> +		     uint16_t nb_pkts)
> +{
[...]
> +}
>  int
>  virtio_dev_rx_queue_done(void *rxq, uint16_t offset)
>  {

Please add an empty line between the functions.


> @@ -547,6 +642,10 @@ virtio_dev_tx_queue_setup_finish(struct rte_eth_dev *dev,
>  
>  	PMD_INIT_FUNC_TRACE();
>  
> +	if (vtpci_packed_queue(hw)) {
> +		vq->vq_ring.avail_wrap_counter = 1;

virtio_dev_tx_queue_setup_finish() will be called during a
dev_stop()/dev_start(). The problem is that, the dev_stop()
doesn't really stop the device. So we can't reset the wrap
counter to 1 in dev_start().


> +	}
> +
>  	if (hw->use_simple_tx) {
>  		for (desc_idx = 0; desc_idx < mid_idx; desc_idx++) {
>  			vq->vq_ring.avail->ring[desc_idx] =
> @@ -567,7 +666,8 @@ virtio_dev_tx_queue_setup_finish(struct rte_eth_dev *dev,
>  			vq->vq_ring.avail->ring[desc_idx] = desc_idx;
>  	}
>  
> -	VIRTQUEUE_DUMP(vq);
> +	if (!vtpci_packed_queue(hw))

The check isn't needed.


> +		VIRTQUEUE_DUMP(vq);
>  
>  	return 0;
>  }
> -- 
> 2.14.3
> 

Re: [PATCH v3 07/21] net/virtio: implement transmit path for packed queues

[Jens Freimann]

On Sun, Apr 08, 2018 at 12:51:32PM +0800, Tiwei Bie wrote:
>On Thu, Apr 05, 2018 at 12:10:17PM +0200, Jens Freimann wrote:
>[...]
>> diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
>> index a8aa87b32..9f9b5a8f8 100644
>> --- a/drivers/net/virtio/virtio_rxtx.c
>> +++ b/drivers/net/virtio/virtio_rxtx.c
>> @@ -38,6 +38,101 @@
>>  #define  VIRTIO_DUMP_PACKET(m, len) do { } while (0)
>>  #endif
>>
>> +#define VIRTIO_SIMPLE_FLAGS ((uint32_t)ETH_TXQ_FLAGS_NOMULTSEGS | \
>> +	ETH_TXQ_FLAGS_NOOFFLOADS)
>
>Why add this?

mistake during rebase, will fix.
>
>> +
>> +/* Cleanup from completed transmits. */
>> +static void
>> +virtio_xmit_cleanup_packed(struct virtqueue *vq)
>> +{
>> +	uint16_t idx;
>> +	uint16_t size = vq->vq_nentries;
>> +	struct vring_desc_packed *desc = vq->vq_ring.desc_packed;
>> +
>> +	idx = vq->vq_used_cons_idx & (size - 1);
>> +	while (desc_is_used(&desc[idx]) &&
>> +	       vq->vq_free_cnt < size) {
>> +		vq->vq_free_cnt++;
>> +		idx = ++vq->vq_used_cons_idx & (size - 1);
>
>Driver needs to keep track of the number of descriptors
>to be skipped for each used descriptor.

I've added support for this due already (because of your comment on
the rxvq flush). Will be in v4.

>> +	}
>> +}
>> +
>> +uint16_t
>> +virtio_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts,
>> +		     uint16_t nb_pkts)
>> +{
>[...]
>> +}
>>  int
>>  virtio_dev_rx_queue_done(void *rxq, uint16_t offset)
>>  {
>
>Please add an empty line between the functions.

ok.

>
>
>> @@ -547,6 +642,10 @@ virtio_dev_tx_queue_setup_finish(struct rte_eth_dev *dev,
>>
>>  	PMD_INIT_FUNC_TRACE();
>>
>> +	if (vtpci_packed_queue(hw)) {
>> +		vq->vq_ring.avail_wrap_counter = 1;
>
>virtio_dev_tx_queue_setup_finish() will be called during a
>dev_stop()/dev_start(). The problem is that, the dev_stop()
>doesn't really stop the device. So we can't reset the wrap
>counter to 1 in dev_start().

I see, would virtio_init_device() work? 

thanks!

regards,
Jens 
>
>> +	}
>> +
>>  	if (hw->use_simple_tx) {
>>  		for (desc_idx = 0; desc_idx < mid_idx; desc_idx++) {
>>  			vq->vq_ring.avail->ring[desc_idx] =
>> @@ -567,7 +666,8 @@ virtio_dev_tx_queue_setup_finish(struct rte_eth_dev *dev,
>>  			vq->vq_ring.avail->ring[desc_idx] = desc_idx;
>>  	}
>>
>> -	VIRTQUEUE_DUMP(vq);
>> +	if (!vtpci_packed_queue(hw))
>
>The check isn't needed.
>
>
>> +		VIRTQUEUE_DUMP(vq);
>>
>>  	return 0;
>>  }
>> --
>> 2.14.3
>>

[PATCH v3 08/21] net/virtio: implement receive path for packed queues

[Jens Freimann]

From: Yuanhan Liu <yuanhan.liu@linux.intel.com>

Implement the receive part here. No support for mergeable buffers yet.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
 drivers/net/virtio/virtio_ethdev.c |  10 ++-
 drivers/net/virtio/virtio_ethdev.h |   2 +
 drivers/net/virtio/virtio_rxtx.c   | 137 ++++++++++++++++++++++++++++++++++++-
 3 files changed, 146 insertions(+), 3 deletions(-)

diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index 089a161ac..dc220c743 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1315,10 +1315,15 @@ set_rxtx_funcs(struct rte_eth_dev *eth_dev)
 {
 	struct virtio_hw *hw = eth_dev->data->dev_private;
 
-	if (hw->use_simple_rx) {
+	/* workarount for packed vqs which don't support mrg_rxbuf at this point */
+	if (vtpci_packed_queue(hw) && vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF)) {
+		eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
+	} else if (hw->use_simple_rx) {
 		PMD_INIT_LOG(INFO, "virtio: using simple Rx path on port %u",
 			eth_dev->data->port_id);
 		eth_dev->rx_pkt_burst = virtio_recv_pkts_vec;
+	} else if (vtpci_packed_queue(hw)) {
+		eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
 	} else if (vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF)) {
 		PMD_INIT_LOG(INFO,
 			"virtio: using mergeable buffer Rx path on port %u",
@@ -1474,7 +1479,8 @@ virtio_init_device(struct rte_eth_dev *eth_dev, uint64_t req_features)
 
 	/* Setting up rx_header size for the device */
 	if (vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF) ||
-	    vtpci_with_feature(hw, VIRTIO_F_VERSION_1))
+	    vtpci_with_feature(hw, VIRTIO_F_VERSION_1) ||
+	    vtpci_with_feature(hw, VIRTIO_F_RING_PACKED))
 		hw->vtnet_hdr_size = sizeof(struct virtio_net_hdr_mrg_rxbuf);
 	else
 		hw->vtnet_hdr_size = sizeof(struct virtio_net_hdr);
diff --git a/drivers/net/virtio/virtio_ethdev.h b/drivers/net/virtio/virtio_ethdev.h
index d457013cb..3aeced4bb 100644
--- a/drivers/net/virtio/virtio_ethdev.h
+++ b/drivers/net/virtio/virtio_ethdev.h
@@ -80,6 +80,8 @@ int virtio_dev_tx_queue_setup_finish(struct rte_eth_dev *dev,
 
 uint16_t virtio_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		uint16_t nb_pkts);
+uint16_t virtio_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
+		uint16_t nb_pkts);
 
 uint16_t virtio_recv_mergeable_pkts(void *rx_queue, struct rte_mbuf **rx_pkts,
 		uint16_t nb_pkts);
diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
index 9f9b5a8f8..9220ae661 100644
--- a/drivers/net/virtio/virtio_rxtx.c
+++ b/drivers/net/virtio/virtio_rxtx.c
@@ -31,6 +31,7 @@
 #include "virtqueue.h"
 #include "virtio_rxtx.h"
 #include "virtio_rxtx_simple.h"
+#include "virtio_ring.h"
 
 #ifdef RTE_LIBRTE_VIRTIO_DEBUG_DUMP
 #define VIRTIO_DUMP_PACKET(m, len) rte_pktmbuf_dump(stdout, m, len)
@@ -521,10 +522,38 @@ virtio_dev_rx_queue_setup_finish(struct rte_eth_dev *dev, uint16_t queue_idx)
 	struct virtnet_rx *rxvq = &vq->rxq;
 	struct rte_mbuf *m;
 	uint16_t desc_idx;
-	int error, nbufs;
+	int error, nbufs = 0;
 
 	PMD_INIT_FUNC_TRACE();
 
+	if (vtpci_packed_queue(hw)) {
+		struct vring_desc_packed *desc;
+		struct vq_desc_extra *dxp;
+
+		for (desc_idx = 0; desc_idx < vq->vq_nentries;
+				desc_idx++) {
+			m = rte_mbuf_raw_alloc(rxvq->mpool);
+			if (unlikely(m == NULL))
+				return -ENOMEM;
+
+			dxp = &vq->vq_descx[desc_idx];
+			dxp->cookie = m;
+			dxp->ndescs = 1;
+
+			desc = &vq->vq_ring.desc_packed[desc_idx];
+			desc->addr = VIRTIO_MBUF_ADDR(m, vq) +
+				RTE_PKTMBUF_HEADROOM - hw->vtnet_hdr_size;
+			desc->len = m->buf_len - RTE_PKTMBUF_HEADROOM +
+				hw->vtnet_hdr_size;
+			desc->flags |= VRING_DESC_F_WRITE;
+			rte_smp_wmb();
+			set_desc_avail(&vq->vq_ring, desc);
+		}
+		toggle_wrap_counter(&vq->vq_ring);
+		nbufs = desc_idx;
+		goto out;
+	}
+
 	/* Allocate blank mbufs for the each rx descriptor */
 	nbufs = 0;
 
@@ -569,6 +598,7 @@ virtio_dev_rx_queue_setup_finish(struct rte_eth_dev *dev, uint16_t queue_idx)
 		vq_update_avail_idx(vq);
 	}
 
+out:
 	PMD_INIT_LOG(DEBUG, "Allocated %d bufs", nbufs);
 
 	VIRTQUEUE_DUMP(vq);
@@ -799,6 +829,111 @@ rx_offload_enabled(struct virtio_hw *hw)
 		vtpci_with_feature(hw, VIRTIO_NET_F_GUEST_TSO6);
 }
 
+uint16_t
+virtio_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
+		     uint16_t nb_pkts)
+{
+	struct virtnet_rx *rxvq = rx_queue;
+	struct virtqueue *vq = rxvq->vq;
+	struct virtio_hw *hw = vq->hw;
+	struct rte_mbuf *rxm, *nmb;
+	uint16_t nb_rx;
+	uint32_t len;
+	uint32_t i;
+	uint32_t hdr_size;
+	int offload;
+	struct virtio_net_hdr *hdr;
+	struct vring_desc_packed *descs = vq->vq_ring.desc_packed;
+	struct vring_desc_packed *desc;
+	uint16_t used_idx = vq->vq_used_cons_idx;
+	struct vq_desc_extra *dxp;
+
+	nb_rx = 0;
+	if (unlikely(hw->started == 0))
+		return nb_rx;
+
+	hdr_size = hw->vtnet_hdr_size;
+	offload = rx_offload_enabled(hw);
+
+	for (i = 0; i < nb_pkts; i++) {
+		desc = &descs[used_idx & (vq->vq_nentries - 1)];
+		if (!desc_is_used(desc))
+			break;
+
+		rte_smp_rmb();
+
+		nmb = rte_mbuf_raw_alloc(rxvq->mpool);
+		if (unlikely(nmb == NULL)) {
+			struct rte_eth_dev *dev
+				= &rte_eth_devices[rxvq->port_id];
+			dev->data->rx_mbuf_alloc_failed++;
+			break;
+		}
+
+		dxp = &vq->vq_descx[used_idx & (vq->vq_nentries - 1)];
+
+		len = desc->len;
+		rxm = dxp->cookie;
+		dxp->cookie = nmb;
+		dxp->ndescs = 1;
+
+		desc->addr = VIRTIO_MBUF_ADDR(nmb, vq) +
+			RTE_PKTMBUF_HEADROOM - hw->vtnet_hdr_size;
+		desc->len = nmb->buf_len - RTE_PKTMBUF_HEADROOM +
+			hw->vtnet_hdr_size;
+		desc->flags |= VRING_DESC_F_WRITE;
+
+		PMD_RX_LOG(DEBUG, "packet len:%d", len);
+
+		if (unlikely(len < hdr_size + ETHER_HDR_LEN)) {
+			PMD_RX_LOG(ERR, "Packet drop");
+			rte_pktmbuf_free(rxm);
+			rxvq->stats.errors++;
+			continue;
+		}
+
+		rxm->port = rxvq->port_id;
+		rxm->data_off = RTE_PKTMBUF_HEADROOM;
+		rxm->ol_flags = 0;
+		rxm->vlan_tci = 0;
+
+		rxm->pkt_len = (uint32_t)(len - hdr_size);
+		rxm->data_len = (uint16_t)(len - hdr_size);
+
+		hdr = (struct virtio_net_hdr *)((char *)rxm->buf_addr +
+			RTE_PKTMBUF_HEADROOM - hdr_size);
+
+		if (hw->vlan_strip)
+			rte_vlan_strip(rxm);
+
+		if (offload && virtio_rx_offload(rxm, hdr) < 0) {
+			rte_pktmbuf_free(rxm);
+			rxvq->stats.errors++;
+			continue;
+		}
+
+		VIRTIO_DUMP_PACKET(rxm, rxm->data_len);
+
+		rxvq->stats.bytes += rxm->pkt_len;
+		virtio_update_packet_stats(&rxvq->stats, rxm);
+
+		rte_smp_wmb();
+		set_desc_avail(&vq->vq_ring, desc);
+
+		rx_pkts[nb_rx++] = rxm;
+
+		used_idx++;
+		if ((used_idx & (vq->vq_nentries - 1)) == 0)
+			toggle_wrap_counter(&vq->vq_ring);
+	}
+
+	rxvq->stats.packets += nb_rx;
+
+	vq->vq_used_cons_idx = used_idx;
+
+	return nb_rx;
+}
+
 #define VIRTIO_MBUF_BURST_SZ 64
 #define DESC_PER_CACHELINE (RTE_CACHE_LINE_SIZE / sizeof(struct vring_desc))
 uint16_t
-- 
2.14.3

Re: [PATCH v3 08/21] net/virtio: implement receive path for packed queues

[Maxime Coquelin]

Hi Jens,

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> From: Yuanhan Liu <yuanhan.liu@linux.intel.com>
> 
> Implement the receive part here. No support for mergeable buffers yet.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
> ---
>   drivers/net/virtio/virtio_ethdev.c |  10 ++-
>   drivers/net/virtio/virtio_ethdev.h |   2 +
>   drivers/net/virtio/virtio_rxtx.c   | 137 ++++++++++++++++++++++++++++++++++++-
>   3 files changed, 146 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
> index 089a161ac..dc220c743 100644
> --- a/drivers/net/virtio/virtio_ethdev.c
> +++ b/drivers/net/virtio/virtio_ethdev.c
> @@ -1315,10 +1315,15 @@ set_rxtx_funcs(struct rte_eth_dev *eth_dev)
>   {
>   	struct virtio_hw *hw = eth_dev->data->dev_private;
>   
> -	if (hw->use_simple_rx) {
> +	/* workarount for packed vqs which don't support mrg_rxbuf at this point */

I don't think you need such workarounds, just advertise the packed ring
layout feature once full support is introduced.

Also I'm not cleat what the workaround is needed here, as you set to
virtio_recv_pkts_packed whatever mrg is on or not.

Re: [PATCH v3 08/21] net/virtio: implement receive path for packed queues

[Jens Freimann]

On Fri, Apr 06, 2018 at 09:51:32AM +0200, Maxime Coquelin wrote:
>Hi Jens,
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>From: Yuanhan Liu <yuanhan.liu@linux.intel.com>
>>
>>Implement the receive part here. No support for mergeable buffers yet.
>>
>>Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>>Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
>>---
>>  drivers/net/virtio/virtio_ethdev.c |  10 ++-
>>  drivers/net/virtio/virtio_ethdev.h |   2 +
>>  drivers/net/virtio/virtio_rxtx.c   | 137 ++++++++++++++++++++++++++++++++++++-
>>  3 files changed, 146 insertions(+), 3 deletions(-)
>>
>>diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
>>index 089a161ac..dc220c743 100644
>>--- a/drivers/net/virtio/virtio_ethdev.c
>>+++ b/drivers/net/virtio/virtio_ethdev.c
>>@@ -1315,10 +1315,15 @@ set_rxtx_funcs(struct rte_eth_dev *eth_dev)
>>  {
>>  	struct virtio_hw *hw = eth_dev->data->dev_private;
>>-	if (hw->use_simple_rx) {
>>+	/* workarount for packed vqs which don't support mrg_rxbuf at this point */
>
>I don't think you need such workarounds, just advertise the packed ring
>layout feature once full support is introduced.
>
>Also I'm not cleat what the workaround is needed here, as you set to
>virtio_recv_pkts_packed whatever mrg is on or not.

yes, I'll change it in v4 to advertise packed virtqueues as the last
patch.

Thanks!

regards,
Jens 
>
>

[PATCH v3 09/21] vhost: disable packed virtqueues by default

[Jens Freimann]

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 config/common_base        | 1 +
 lib/librte_vhost/socket.c | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/config/common_base b/config/common_base
index cd4b419b4..bf969d82d 100644
--- a/config/common_base
+++ b/config/common_base
@@ -783,6 +783,7 @@ CONFIG_RTE_LIBRTE_PDUMP=y
 CONFIG_RTE_LIBRTE_VHOST=n
 CONFIG_RTE_LIBRTE_VHOST_NUMA=n
 CONFIG_RTE_LIBRTE_VHOST_DEBUG=n
+CONFIG_RTE_LIBRTE_VHOST_PQ=n
 
 #
 # Compile vhost PMD
diff --git a/lib/librte_vhost/socket.c b/lib/librte_vhost/socket.c
index 636fc25c6..72d769e6a 100644
--- a/lib/librte_vhost/socket.c
+++ b/lib/librte_vhost/socket.c
@@ -850,6 +850,10 @@ rte_vhost_driver_register(const char *path, uint64_t flags)
 		vsocket->features &= ~(1ULL << VIRTIO_F_IOMMU_PLATFORM);
 	}
 
+#ifndef RTE_LIBRTE_VHOST_PQ
+		vsocket->features &= ~(1ULL << VIRTIO_F_RING_PACKED);
+#endif
+
 	if ((flags & RTE_VHOST_USER_CLIENT) != 0) {
 		vsocket->reconnect = !(flags & RTE_VHOST_USER_NO_RECONNECT);
 		if (vsocket->reconnect && reconn_tid == 0) {
-- 
2.14.3

[PATCH v3 10/21] vhost: turn of indirect descriptors for packed virtqueues

[Jens Freimann]

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 lib/librte_vhost/socket.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/librte_vhost/socket.c b/lib/librte_vhost/socket.c
index 72d769e6a..05193e368 100644
--- a/lib/librte_vhost/socket.c
+++ b/lib/librte_vhost/socket.c
@@ -852,6 +852,7 @@ rte_vhost_driver_register(const char *path, uint64_t flags)
 
 #ifndef RTE_LIBRTE_VHOST_PQ
 		vsocket->features &= ~(1ULL << VIRTIO_F_RING_PACKED);
+		vsocket->features &= ~(1ULL << VIRTIO_RING_F_INDIRECT_DESC);
 #endif
 
 	if ((flags & RTE_VHOST_USER_CLIENT) != 0) {
-- 
2.14.3

Re: [PATCH v3 10/21] vhost: turn of indirect descriptors for packed virtqueues

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   lib/librte_vhost/socket.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/lib/librte_vhost/socket.c b/lib/librte_vhost/socket.c
> index 72d769e6a..05193e368 100644
> --- a/lib/librte_vhost/socket.c
> +++ b/lib/librte_vhost/socket.c
> @@ -852,6 +852,7 @@ rte_vhost_driver_register(const char *path, uint64_t flags)
>   
>   #ifndef RTE_LIBRTE_VHOST_PQ
>   		vsocket->features &= ~(1ULL << VIRTIO_F_RING_PACKED);
> +		vsocket->features &= ~(1ULL << VIRTIO_RING_F_INDIRECT_DESC);
hmm.. Shouldn't be the opposite?
>   #endif
>   
>   	if ((flags & RTE_VHOST_USER_CLIENT) != 0) {
> 

[PATCH v3 11/21] vhost: add virtio 1.1 defines

[Jens Freimann]

This should actually be in the kernel header file, but it isn't
yet. For now let's use our own headers.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 lib/librte_vhost/vhost.h      |  4 ++++
 lib/librte_vhost/virtio-1.1.h | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+)
 create mode 100644 lib/librte_vhost/virtio-1.1.h

diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
index c14a90529..3004c26c1 100644
--- a/lib/librte_vhost/vhost.h
+++ b/lib/librte_vhost/vhost.h
@@ -177,6 +177,10 @@ struct vhost_msg {
 #ifndef VIRTIO_F_VERSION_1
  #define VIRTIO_F_VERSION_1 32
 #endif
+#ifndef VIRTIO_F_RING_PACKED
+ #define VIRTIO_F_RING_PACKED 34
+#endif
+#define VHOST_USER_F_PROTOCOL_FEATURES	30
 
 /* Features supported by this builtin vhost-user net driver. */
 #define VIRTIO_NET_SUPPORTED_FEATURES ((1ULL << VIRTIO_NET_F_MRG_RXBUF) | \
diff --git a/lib/librte_vhost/virtio-1.1.h b/lib/librte_vhost/virtio-1.1.h
new file mode 100644
index 000000000..7b48caed7
--- /dev/null
+++ b/lib/librte_vhost/virtio-1.1.h
@@ -0,0 +1,18 @@
+#ifndef __VIRTIO_PACKED_H
+#define __VIRTIO_PACKED_H
+
+#define VRING_DESC_F_NEXT       1
+#define VRING_DESC_F_WRITE      2
+#define VRING_DESC_F_INDIRECT   4
+
+#define VRING_DESC_F_AVAIL      (1ULL << 7)
+#define VRING_DESC_F_USED	(1ULL << 15)
+
+struct vring_desc_packed {
+	uint64_t addr;
+	uint32_t len;
+	uint16_t index;
+	uint16_t flags;
+};
+
+#endif /* __VIRTIO_PACKED_H */
-- 
2.14.3

Re: [PATCH v3 11/21] vhost: add virtio 1.1 defines

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> This should actually be in the kernel header file, but it isn't
> yet. For now let's use our own headers.

I think it is not just temporary, as we will always want to be able to
build with older kernels.

> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   lib/librte_vhost/vhost.h      |  4 ++++
>   lib/librte_vhost/virtio-1.1.h | 18 ++++++++++++++++++
>   2 files changed, 22 insertions(+)
>   create mode 100644 lib/librte_vhost/virtio-1.1.h
> 
> diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
> index c14a90529..3004c26c1 100644
> --- a/lib/librte_vhost/vhost.h
> +++ b/lib/librte_vhost/vhost.h
> @@ -177,6 +177,10 @@ struct vhost_msg {
>   #ifndef VIRTIO_F_VERSION_1
>    #define VIRTIO_F_VERSION_1 32
>   #endif
> +#ifndef VIRTIO_F_RING_PACKED
> + #define VIRTIO_F_RING_PACKED 34
> +#endif
> +#define VHOST_USER_F_PROTOCOL_FEATURES	30
>   
>   /* Features supported by this builtin vhost-user net driver. */
>   #define VIRTIO_NET_SUPPORTED_FEATURES ((1ULL << VIRTIO_NET_F_MRG_RXBUF) | \
> diff --git a/lib/librte_vhost/virtio-1.1.h b/lib/librte_vhost/virtio-1.1.h
> new file mode 100644
> index 000000000..7b48caed7
> --- /dev/null
> +++ b/lib/librte_vhost/virtio-1.1.h
> @@ -0,0 +1,18 @@

You need to add a license for the file.

> +#ifndef __VIRTIO_PACKED_H
> +#define __VIRTIO_PACKED_H
> +
> +#define VRING_DESC_F_NEXT       1
> +#define VRING_DESC_F_WRITE      2
> +#define VRING_DESC_F_INDIRECT   4
> +
> +#define VRING_DESC_F_AVAIL      (1ULL << 7)
> +#define VRING_DESC_F_USED	(1ULL << 15)
> +
> +struct vring_desc_packed {
> +	uint64_t addr;
> +	uint32_t len;
> +	uint16_t index;
> +	uint16_t flags;
> +};
> +
> +#endif /* __VIRTIO_PACKED_H */
> 

[PATCH v3 12/21] vhost: vring address setup for packed queues

[Jens Freimann]

From: Yuanhan Liu <yuanhan.liu@linux.intel.com>

Add code to set up packed queues when enabled.

Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Signed-off-by: Jens Freimann <jfreiman@redhat.com>
---
 lib/librte_vhost/vhost.h      |  1 +
 lib/librte_vhost/vhost_user.c | 21 ++++++++++++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
index 3004c26c1..20d78f883 100644
--- a/lib/librte_vhost/vhost.h
+++ b/lib/librte_vhost/vhost.h
@@ -74,6 +74,7 @@ struct batch_copy_elem {
  */
 struct vhost_virtqueue {
 	struct vring_desc	*desc;
+	struct vring_desc_packed   *desc_packed;
 	struct vring_avail	*avail;
 	struct vring_used	*used;
 	uint32_t		size;
diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c
index 157cf2f60..183893e46 100644
--- a/lib/librte_vhost/vhost_user.c
+++ b/lib/librte_vhost/vhost_user.c
@@ -469,6 +469,23 @@ translate_ring_addresses(struct virtio_net *dev, int vq_index)
 	struct vhost_virtqueue *vq = dev->virtqueue[vq_index];
 	struct vhost_vring_addr *addr = &vq->ring_addrs;
 
+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
+		vq->desc_packed = (struct vring_desc_packed *) ring_addr_to_vva
+			(dev, vq, addr->desc_user_addr, sizeof(vq->desc_packed));
+		vq->desc = NULL;
+		vq->avail = NULL;
+		vq->used = NULL;
+		vq->log_guest_addr = 0;
+
+		if (vq->last_used_idx != 0) {
+			RTE_LOG(WARNING, VHOST_CONFIG,
+				"last_used_idx (%u) not 0\n",
+				vq->last_used_idx);
+			vq->last_used_idx = 0;
+		}
+		return dev;
+	}
+
 	/* The addresses are converted from QEMU virtual to Vhost virtual. */
 	if (vq->desc && vq->avail && vq->used)
 		return dev;
@@ -481,6 +498,7 @@ translate_ring_addresses(struct virtio_net *dev, int vq_index)
 			dev->vid);
 		return dev;
 	}
+	vq->desc_packed = NULL;
 
 	dev = numa_realloc(dev, vq_index);
 	vq = dev->virtqueue[vq_index];
@@ -853,7 +871,8 @@ vhost_user_set_mem_table(struct virtio_net *dev, struct VhostUserMsg *pmsg)
 static int
 vq_is_ready(struct vhost_virtqueue *vq)
 {
-	return vq && vq->desc && vq->avail && vq->used &&
+	return vq &&
+	       (vq->desc_packed || (vq->desc && vq->avail && vq->used)) &&
 	       vq->kickfd != VIRTIO_UNINITIALIZED_EVENTFD &&
 	       vq->callfd != VIRTIO_UNINITIALIZED_EVENTFD;
 }
-- 
2.14.3

Re: [PATCH v3 12/21] vhost: vring address setup for packed queues

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> From: Yuanhan Liu <yuanhan.liu@linux.intel.com>
> 
> Add code to set up packed queues when enabled.
> 
> Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
> Signed-off-by: Jens Freimann <jfreiman@redhat.com>
> ---
>   lib/librte_vhost/vhost.h      |  1 +
>   lib/librte_vhost/vhost_user.c | 21 ++++++++++++++++++++-
>   2 files changed, 21 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
> index 3004c26c1..20d78f883 100644
> --- a/lib/librte_vhost/vhost.h
> +++ b/lib/librte_vhost/vhost.h
> @@ -74,6 +74,7 @@ struct batch_copy_elem {
>    */
>   struct vhost_virtqueue {
>   	struct vring_desc	*desc;
> +	struct vring_desc_packed   *desc_packed;
>   	struct vring_avail	*avail;
>   	struct vring_used	*used;
>   	uint32_t		size;
> diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c
> index 157cf2f60..183893e46 100644
> --- a/lib/librte_vhost/vhost_user.c
> +++ b/lib/librte_vhost/vhost_user.c
> @@ -469,6 +469,23 @@ translate_ring_addresses(struct virtio_net *dev, int vq_index)
>   	struct vhost_virtqueue *vq = dev->virtqueue[vq_index];
>   	struct vhost_vring_addr *addr = &vq->ring_addrs;
>   
> +	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
> +		vq->desc_packed = (struct vring_desc_packed *) ring_addr_to_vva
> +			(dev, vq, addr->desc_user_addr, sizeof(vq->desc_packed));
sizeof(*vq->desc_packed)
or
sizeof(struct vring_desc_packed) for consitency.

> +		vq->desc = NULL;
> +		vq->avail = NULL;
> +		vq->used = NULL;
> +		vq->log_guest_addr = 0;
> +
> +		if (vq->last_used_idx != 0) {
> +			RTE_LOG(WARNING, VHOST_CONFIG,
> +				"last_used_idx (%u) not 0\n",
> +				vq->last_used_idx);
> +			vq->last_used_idx = 0;
> +		}
> +		return dev;
> +	}
> +
>   	/* The addresses are converted from QEMU virtual to Vhost virtual. */
>   	if (vq->desc && vq->avail && vq->used)
>   		return dev;
> @@ -481,6 +498,7 @@ translate_ring_addresses(struct virtio_net *dev, int vq_index)
>   			dev->vid);
>   		return dev;
>   	}
> +	vq->desc_packed = NULL;
>   
>   	dev = numa_realloc(dev, vq_index);
>   	vq = dev->virtqueue[vq_index];
> @@ -853,7 +871,8 @@ vhost_user_set_mem_table(struct virtio_net *dev, struct VhostUserMsg *pmsg)
>   static int
>   vq_is_ready(struct vhost_virtqueue *vq)
>   {
> -	return vq && vq->desc && vq->avail && vq->used &&
> +	return vq &&
> +	       (vq->desc_packed || (vq->desc && vq->avail && vq->used)) &&
>   	       vq->kickfd != VIRTIO_UNINITIALIZED_EVENTFD &&
>   	       vq->callfd != VIRTIO_UNINITIALIZED_EVENTFD;
>   }
> 

Re: [PATCH v3 12/21] vhost: vring address setup for packed queues

[Jens Freimann]

On Fri, Apr 06, 2018 at 10:19:28AM +0200, Maxime Coquelin wrote:
>
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>From: Yuanhan Liu <yuanhan.liu@linux.intel.com>
>>
>>Add code to set up packed queues when enabled.
>>
>>Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
>>Signed-off-by: Jens Freimann <jfreiman@redhat.com>
>>---
>>  lib/librte_vhost/vhost.h      |  1 +
>>  lib/librte_vhost/vhost_user.c | 21 ++++++++++++++++++++-
>>  2 files changed, 21 insertions(+), 1 deletion(-)
>>
>>diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
>>index 3004c26c1..20d78f883 100644
>>--- a/lib/librte_vhost/vhost.h
>>+++ b/lib/librte_vhost/vhost.h
>>@@ -74,6 +74,7 @@ struct batch_copy_elem {
>>   */
>>  struct vhost_virtqueue {
>>  	struct vring_desc	*desc;
>>+	struct vring_desc_packed   *desc_packed;
>>  	struct vring_avail	*avail;
>>  	struct vring_used	*used;
>>  	uint32_t		size;
>>diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c
>>index 157cf2f60..183893e46 100644
>>--- a/lib/librte_vhost/vhost_user.c
>>+++ b/lib/librte_vhost/vhost_user.c
>>@@ -469,6 +469,23 @@ translate_ring_addresses(struct virtio_net *dev, int vq_index)
>>  	struct vhost_virtqueue *vq = dev->virtqueue[vq_index];
>>  	struct vhost_vring_addr *addr = &vq->ring_addrs;
>>+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
>>+		vq->desc_packed = (struct vring_desc_packed *) ring_addr_to_vva
>>+			(dev, vq, addr->desc_user_addr, sizeof(vq->desc_packed));
>sizeof(*vq->desc_packed)
>or
>sizeof(struct vring_desc_packed) for consitency.

yes, will fix. thanks!

regards,
Jens 
>
>>+		vq->desc = NULL;
>>+		vq->avail = NULL;
>>+		vq->used = NULL;
>>+		vq->log_guest_addr = 0;
>>+
>>+		if (vq->last_used_idx != 0) {
>>+			RTE_LOG(WARNING, VHOST_CONFIG,
>>+				"last_used_idx (%u) not 0\n",
>>+				vq->last_used_idx);
>>+			vq->last_used_idx = 0;
>>+		}
>>+		return dev;
>>+	}
>>+
>>  	/* The addresses are converted from QEMU virtual to Vhost virtual. */
>>  	if (vq->desc && vq->avail && vq->used)
>>  		return dev;
>>@@ -481,6 +498,7 @@ translate_ring_addresses(struct virtio_net *dev, int vq_index)
>>  			dev->vid);
>>  		return dev;
>>  	}
>>+	vq->desc_packed = NULL;
>>  	dev = numa_realloc(dev, vq_index);
>>  	vq = dev->virtqueue[vq_index];
>>@@ -853,7 +871,8 @@ vhost_user_set_mem_table(struct virtio_net *dev, struct VhostUserMsg *pmsg)
>>  static int
>>  vq_is_ready(struct vhost_virtqueue *vq)
>>  {
>>-	return vq && vq->desc && vq->avail && vq->used &&
>>+	return vq &&
>>+	       (vq->desc_packed || (vq->desc && vq->avail && vq->used)) &&
>>  	       vq->kickfd != VIRTIO_UNINITIALIZED_EVENTFD &&
>>  	       vq->callfd != VIRTIO_UNINITIALIZED_EVENTFD;
>>  }
>>

[PATCH v3 13/21] vhost: add helpers for packed virtqueues

[Jens Freimann]

Add some helper functions to set/check descriptor flags
and toggle the used wrap counter.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 lib/librte_vhost/virtio-1.1.h | 44 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/lib/librte_vhost/virtio-1.1.h b/lib/librte_vhost/virtio-1.1.h
index 7b48caed7..e77d7aa6c 100644
--- a/lib/librte_vhost/virtio-1.1.h
+++ b/lib/librte_vhost/virtio-1.1.h
@@ -15,4 +15,48 @@ struct vring_desc_packed {
 	uint16_t flags;
 };
 
+static inline void
+toggle_wrap_counter(struct vhost_virtqueue *vq)
+{
+	vq->used_wrap_counter ^= 1;
+}
+
+static inline int
+desc_is_avail(struct vhost_virtqueue *vq, struct vring_desc_packed *desc)
+{
+	if (vq->used_wrap_counter == 1) {
+		if ((desc->flags & VRING_DESC_F_AVAIL) &&
+				!(desc->flags & VRING_DESC_F_USED))
+			return 1;
+	}
+	if (vq->used_wrap_counter == 0) {
+		if (!(desc->flags & VRING_DESC_F_AVAIL) &&
+				(desc->flags & VRING_DESC_F_USED))
+			return 1;
+	}
+	return 0;
+}
+
+static inline void
+_set_desc_used(struct vring_desc_packed *desc, int wrap_counter)
+{
+	uint16_t flags = desc->flags;
+
+	if (wrap_counter == 1) {
+		flags |= VRING_DESC_F_USED;
+		flags |= VRING_DESC_F_AVAIL;
+	} else {
+		flags &= ~VRING_DESC_F_USED;
+		flags &= ~VRING_DESC_F_AVAIL;
+	}
+
+	desc->flags = flags;
+}
+
+static inline void
+set_desc_used(struct vhost_virtqueue *vq, struct vring_desc_packed *desc)
+{
+	_set_desc_used(desc, vq->used_wrap_counter);
+}
+
 #endif /* __VIRTIO_PACKED_H */
-- 
2.14.3

Re: [PATCH v3 13/21] vhost: add helpers for packed virtqueues

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Add some helper functions to set/check descriptor flags
> and toggle the used wrap counter.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   lib/librte_vhost/virtio-1.1.h | 44 +++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 44 insertions(+)
> 
> diff --git a/lib/librte_vhost/virtio-1.1.h b/lib/librte_vhost/virtio-1.1.h
> index 7b48caed7..e77d7aa6c 100644
> --- a/lib/librte_vhost/virtio-1.1.h
> +++ b/lib/librte_vhost/virtio-1.1.h
Shouldn't the file be named virtio-packed.h?

> @@ -15,4 +15,48 @@ struct vring_desc_packed {
>   	uint16_t flags;
>   };
>   
> +static inline void
> +toggle_wrap_counter(struct vhost_virtqueue *vq)
> +{
> +	vq->used_wrap_counter ^= 1;
> +}
> +
> +static inline int
> +desc_is_avail(struct vhost_virtqueue *vq, struct vring_desc_packed *desc)
> +{
> +	if (vq->used_wrap_counter == 1) {
> +		if ((desc->flags & VRING_DESC_F_AVAIL) &&
> +				!(desc->flags & VRING_DESC_F_USED))
> +			return 1;
> +	}
> +	if (vq->used_wrap_counter == 0) {
> +		if (!(desc->flags & VRING_DESC_F_AVAIL) &&
> +				(desc->flags & VRING_DESC_F_USED))
> +			return 1;
> +	}
> +	return 0;
> +}
> +
> +static inline void
> +_set_desc_used(struct vring_desc_packed *desc, int wrap_counter)
> +{
> +	uint16_t flags = desc->flags;
> +
> +	if (wrap_counter == 1) {
> +		flags |= VRING_DESC_F_USED;
> +		flags |= VRING_DESC_F_AVAIL;
> +	} else {
> +		flags &= ~VRING_DESC_F_USED;
> +		flags &= ~VRING_DESC_F_AVAIL;
> +	}
> +
> +	desc->flags = flags;
> +}
> +
> +static inline void
> +set_desc_used(struct vhost_virtqueue *vq, struct vring_desc_packed *desc)
> +{
> +	_set_desc_used(desc, vq->used_wrap_counter);
> +}
> +

Maybe prefix all with vring_
>   #endif /* __VIRTIO_PACKED_H */
> 

Re: [PATCH v3 13/21] vhost: add helpers for packed virtqueues

[Jens Freimann]

On Fri, Apr 06, 2018 at 11:20:10AM +0200, Maxime Coquelin wrote:
>
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>Add some helper functions to set/check descriptor flags
>>and toggle the used wrap counter.
>>
>>Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>>---
>>  lib/librte_vhost/virtio-1.1.h | 44 +++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 44 insertions(+)
>>
>>diff --git a/lib/librte_vhost/virtio-1.1.h b/lib/librte_vhost/virtio-1.1.h
>>index 7b48caed7..e77d7aa6c 100644
>>--- a/lib/librte_vhost/virtio-1.1.h
>>+++ b/lib/librte_vhost/virtio-1.1.h
>Shouldn't the file be named virtio-packed.h?

yes, will rename it. 

>>@@ -15,4 +15,48 @@ struct vring_desc_packed {
>>  	uint16_t flags;
>>  };
>>+static inline void
>>+toggle_wrap_counter(struct vhost_virtqueue *vq)
>>+{
>>+	vq->used_wrap_counter ^= 1;
>>+}
>>+
>>+static inline int
>>+desc_is_avail(struct vhost_virtqueue *vq, struct vring_desc_packed *desc)
>>+{
>>+	if (vq->used_wrap_counter == 1) {
>>+		if ((desc->flags & VRING_DESC_F_AVAIL) &&
>>+				!(desc->flags & VRING_DESC_F_USED))
>>+			return 1;
>>+	}
>>+	if (vq->used_wrap_counter == 0) {
>>+		if (!(desc->flags & VRING_DESC_F_AVAIL) &&
>>+				(desc->flags & VRING_DESC_F_USED))
>>+			return 1;
>>+	}
>>+	return 0;
>>+}
>>+
>>+static inline void
>>+_set_desc_used(struct vring_desc_packed *desc, int wrap_counter)
>>+{
>>+	uint16_t flags = desc->flags;
>>+
>>+	if (wrap_counter == 1) {
>>+		flags |= VRING_DESC_F_USED;
>>+		flags |= VRING_DESC_F_AVAIL;
>>+	} else {
>>+		flags &= ~VRING_DESC_F_USED;
>>+		flags &= ~VRING_DESC_F_AVAIL;
>>+	}
>>+
>>+	desc->flags = flags;
>>+}
>>+
>>+static inline void
>>+set_desc_used(struct vhost_virtqueue *vq, struct vring_desc_packed *desc)
>>+{
>>+	_set_desc_used(desc, vq->used_wrap_counter);
>>+}
>>+
>
>Maybe prefix all with vring_
>>  #endif /* __VIRTIO_PACKED_H */

ok

Thanks!

regards,
Jens 
>>

[PATCH v3 14/21] vhost: dequeue for packed queues

[Jens Freimann]

Implement code to dequeue and process descriptors from
the vring if VIRTIO_F_RING_PACKED is enabled.

Check if descriptor was made available by driver by looking at
VIRTIO_F_DESC_AVAIL flag in descriptor. If so dequeue and set
the used flag VIRTIO_F_DESC_USED to the current value of the
used wrap counter.

Used ring wrap counter needs to be toggled when last descriptor is
written out. This allows the host/guest to detect new descriptors even
after the ring has wrapped.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 lib/librte_vhost/vhost.c      |   1 +
 lib/librte_vhost/vhost.h      |   1 +
 lib/librte_vhost/virtio_net.c | 228 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 230 insertions(+)

diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c
index 1f17cdd75..eb5a98875 100644
--- a/lib/librte_vhost/vhost.c
+++ b/lib/librte_vhost/vhost.c
@@ -185,6 +185,7 @@ init_vring_queue(struct virtio_net *dev, uint32_t vring_idx)
 
 	vq->kickfd = VIRTIO_UNINITIALIZED_EVENTFD;
 	vq->callfd = VIRTIO_UNINITIALIZED_EVENTFD;
+	vq->used_wrap_counter = 1;
 
 	vhost_user_iotlb_init(dev, vring_idx);
 	/* Backends are set to -1 indicating an inactive device. */
diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
index 20d78f883..c8aa946fd 100644
--- a/lib/librte_vhost/vhost.h
+++ b/lib/librte_vhost/vhost.h
@@ -112,6 +112,7 @@ struct vhost_virtqueue {
 
 	struct batch_copy_elem	*batch_copy_elems;
 	uint16_t		batch_copy_nb_elems;
+	uint16_t 		used_wrap_counter;
 
 	rte_rwlock_t	iotlb_lock;
 	rte_rwlock_t	iotlb_pending_lock;
diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
index ed7198dbb..7eea1da04 100644
--- a/lib/librte_vhost/virtio_net.c
+++ b/lib/librte_vhost/virtio_net.c
@@ -19,6 +19,7 @@
 
 #include "iotlb.h"
 #include "vhost.h"
+#include "virtio-1.1.h"
 
 #define MAX_PKT_BURST 32
 
@@ -1118,6 +1119,233 @@ restore_mbuf(struct rte_mbuf *m)
 	}
 }
 
+static inline uint16_t
+dequeue_desc_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
+	     struct rte_mempool *mbuf_pool, struct rte_mbuf *m,
+	     struct vring_desc_packed *descs)
+{
+	struct vring_desc_packed *desc;
+	uint64_t desc_addr;
+	uint32_t desc_avail, desc_offset;
+	uint32_t mbuf_avail, mbuf_offset;
+	uint32_t cpy_len;
+	struct rte_mbuf *cur = m, *prev = m;
+	struct virtio_net_hdr *hdr = NULL;
+	uint16_t head_idx = vq->last_used_idx & (vq->size - 1);
+	int wrap_counter = vq->used_wrap_counter;
+	int rc = 0;
+
+	rte_spinlock_lock(&vq->access_lock);
+
+	if (unlikely(vq->enabled == 0))
+		goto out;
+
+	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
+		vhost_user_iotlb_rd_lock(vq);
+
+	desc = &descs[vq->last_used_idx & (vq->size - 1)];
+	if (unlikely((desc->len < dev->vhost_hlen)) ||
+			(desc->flags & VRING_DESC_F_INDIRECT)) {
+			RTE_LOG(ERR, VHOST_DATA,
+				"INDIRECT not supported yet\n");
+			rc = -1;
+			goto out;
+	}
+
+	desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
+				      sizeof(*desc), VHOST_ACCESS_RO);
+
+	if (unlikely(!desc_addr)) {
+		rc = -1;
+		goto out;
+	}
+
+	if (virtio_net_with_host_offload(dev)) {
+		hdr = (struct virtio_net_hdr *)((uintptr_t)desc_addr);
+		rte_prefetch0(hdr);
+	}
+
+	/*
+	 * A virtio driver normally uses at least 2 desc buffers
+	 * for Tx: the first for storing the header, and others
+	 * for storing the data.
+	 */
+	if (likely((desc->len == dev->vhost_hlen) &&
+		   (desc->flags & VRING_DESC_F_NEXT) != 0)) {
+		if ((++vq->last_used_idx & (vq->size - 1)) == 0)
+			toggle_wrap_counter(vq);
+
+		desc = &descs[vq->last_used_idx & (vq->size - 1)];
+
+		if (unlikely(desc->flags & VRING_DESC_F_INDIRECT)) {
+			RTE_LOG(ERR, VHOST_DATA,
+				"INDIRECT not supported yet\n");
+			rc = -1;
+			goto out;
+		}
+
+		desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
+					      sizeof(*desc), VHOST_ACCESS_RO);
+		if (unlikely(!desc_addr)) {
+			rc = -1;
+			goto out;
+		}
+
+		desc_offset = 0;
+		desc_avail  = desc->len;
+	} else {
+		desc_avail  = desc->len - dev->vhost_hlen;
+		desc_offset = dev->vhost_hlen;
+	}
+
+	rte_prefetch0((void *)(uintptr_t)(desc_addr + desc_offset));
+
+	PRINT_PACKET(dev, (uintptr_t)(desc_addr + desc_offset), desc_avail, 0);
+
+	mbuf_offset = 0;
+	mbuf_avail  = m->buf_len - RTE_PKTMBUF_HEADROOM;
+	while (1) {
+		uint64_t hpa;
+
+		cpy_len = RTE_MIN(desc_avail, mbuf_avail);
+
+		/*
+		 * A desc buf might across two host physical pages that are
+		 * not continuous. In such case (gpa_to_hpa returns 0), data
+		 * will be copied even though zero copy is enabled.
+		 */
+		if (unlikely(dev->dequeue_zero_copy && (hpa = gpa_to_hpa(dev,
+					desc->addr + desc_offset, cpy_len)))) {
+			cur->data_len = cpy_len;
+			cur->data_off = 0;
+			cur->buf_addr = (void *)(uintptr_t)desc_addr;
+			cur->buf_physaddr = hpa;
+
+			/*
+			 * In zero copy mode, one mbuf can only reference data
+			 * for one or partial of one desc buff.
+			 */
+			mbuf_avail = cpy_len;
+		} else {
+			rte_memcpy(rte_pktmbuf_mtod_offset(cur, void *,
+							   mbuf_offset),
+				(void *)((uintptr_t)(desc_addr + desc_offset)),
+				cpy_len);
+		}
+
+		mbuf_avail  -= cpy_len;
+		mbuf_offset += cpy_len;
+		desc_avail  -= cpy_len;
+		desc_offset += cpy_len;
+
+		/* This desc reaches to its end, get the next one */
+		if (desc_avail == 0) {
+			if ((desc->flags & VRING_DESC_F_NEXT) == 0)
+				break;
+
+			if ((++vq->last_used_idx & (vq->size - 1)) == 0)
+				toggle_wrap_counter(vq);
+
+			desc = &descs[vq->last_used_idx & (vq->size - 1)];
+			if (unlikely(desc->flags & VRING_DESC_F_INDIRECT)) {
+				RTE_LOG(ERR, VHOST_DATA, "INDIRECT not supported yet");
+			}
+
+			desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
+						      sizeof(*desc), VHOST_ACCESS_RO);
+			if (unlikely(!desc_addr)) {
+				rc = -1;
+				goto out;
+			}
+
+			rte_prefetch0((void *)(uintptr_t)desc_addr);
+
+			desc_offset = 0;
+			desc_avail  = desc->len;
+
+			PRINT_PACKET(dev, (uintptr_t)desc_addr, desc->len, 0);
+		}
+
+		/*
+		 * This mbuf reaches to its end, get a new one
+		 * to hold more data.
+		 */
+		if (mbuf_avail == 0) {
+			cur = rte_pktmbuf_alloc(mbuf_pool);
+			if (unlikely(cur == NULL)) {
+				RTE_LOG(ERR, VHOST_DATA, "Failed to "
+					"allocate memory for mbuf.\n");
+				rc = -1;
+				goto out;
+			}
+
+			prev->next = cur;
+			prev->data_len = mbuf_offset;
+			m->nb_segs += 1;
+			m->pkt_len += mbuf_offset;
+			prev = cur;
+
+			mbuf_offset = 0;
+			mbuf_avail  = cur->buf_len - RTE_PKTMBUF_HEADROOM;
+		}
+	}
+
+	if (hdr)
+		vhost_dequeue_offload(hdr, m);
+
+	if ((++vq->last_used_idx & (vq->size - 1)) == 0)
+		toggle_wrap_counter(vq);
+
+	rte_smp_wmb();
+	_set_desc_used(&descs[head_idx], wrap_counter);
+
+	prev->data_len = mbuf_offset;
+	m->pkt_len    += mbuf_offset;
+
+out:
+	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
+		vhost_user_iotlb_rd_unlock(vq);
+	rte_spinlock_unlock(&vq->access_lock);
+
+	return rc;
+}
+
+static inline uint16_t
+vhost_dequeue_burst_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
+			struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
+			uint16_t count)
+{
+	uint16_t i;
+	uint16_t idx;
+	struct vring_desc_packed *desc = vq->desc_packed;
+	int err;
+
+	count = RTE_MIN(MAX_PKT_BURST, count);
+	for (i = 0; i < count; i++) {
+		idx = vq->last_used_idx & (vq->size - 1);
+		if (!desc_is_avail(vq, &desc[idx]))
+			break;
+		rte_smp_rmb();
+
+		pkts[i] = rte_pktmbuf_alloc(mbuf_pool);
+		if (unlikely(pkts[i] == NULL)) {
+			RTE_LOG(ERR, VHOST_DATA,
+				"Failed to allocate memory for mbuf.\n");
+			break;
+		}
+
+		err = dequeue_desc_packed(dev, vq, mbuf_pool, pkts[i], desc);
+		if (unlikely(err)) {
+			rte_pktmbuf_free(pkts[i]);
+			break;
+		}
+	}
+
+	rte_spinlock_unlock(&vq->access_lock);
+
+	return i;
+}
+
 uint16_t
 rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count)
-- 
2.14.3

Re: [PATCH v3 14/21] vhost: dequeue for packed queues

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Implement code to dequeue and process descriptors from
> the vring if VIRTIO_F_RING_PACKED is enabled.
> 
> Check if descriptor was made available by driver by looking at
> VIRTIO_F_DESC_AVAIL flag in descriptor. If so dequeue and set
> the used flag VIRTIO_F_DESC_USED to the current value of the
> used wrap counter.
> 
> Used ring wrap counter needs to be toggled when last descriptor is
> written out. This allows the host/guest to detect new descriptors even
> after the ring has wrapped.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   lib/librte_vhost/vhost.c      |   1 +
>   lib/librte_vhost/vhost.h      |   1 +
>   lib/librte_vhost/virtio_net.c | 228 ++++++++++++++++++++++++++++++++++++++++++
>   3 files changed, 230 insertions(+)
> 
> diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c
> index 1f17cdd75..eb5a98875 100644
> --- a/lib/librte_vhost/vhost.c
> +++ b/lib/librte_vhost/vhost.c
> @@ -185,6 +185,7 @@ init_vring_queue(struct virtio_net *dev, uint32_t vring_idx)
>   
>   	vq->kickfd = VIRTIO_UNINITIALIZED_EVENTFD;
>   	vq->callfd = VIRTIO_UNINITIALIZED_EVENTFD;
> +	vq->used_wrap_counter = 1;
>   
>   	vhost_user_iotlb_init(dev, vring_idx);
>   	/* Backends are set to -1 indicating an inactive device. */
> diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
> index 20d78f883..c8aa946fd 100644
> --- a/lib/librte_vhost/vhost.h
> +++ b/lib/librte_vhost/vhost.h
> @@ -112,6 +112,7 @@ struct vhost_virtqueue {
>   
>   	struct batch_copy_elem	*batch_copy_elems;
>   	uint16_t		batch_copy_nb_elems;
> +	uint16_t 		used_wrap_counter;
>   
>   	rte_rwlock_t	iotlb_lock;
>   	rte_rwlock_t	iotlb_pending_lock;
> diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
> index ed7198dbb..7eea1da04 100644
> --- a/lib/librte_vhost/virtio_net.c
> +++ b/lib/librte_vhost/virtio_net.c
> @@ -19,6 +19,7 @@
>   
>   #include "iotlb.h"
>   #include "vhost.h"
> +#include "virtio-1.1.h"
>   
>   #define MAX_PKT_BURST 32
>   
> @@ -1118,6 +1119,233 @@ restore_mbuf(struct rte_mbuf *m)
>   	}
>   }
>   
> +static inline uint16_t
> +dequeue_desc_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
> +	     struct rte_mempool *mbuf_pool, struct rte_mbuf *m,
> +	     struct vring_desc_packed *descs)
> +{
> +	struct vring_desc_packed *desc;
> +	uint64_t desc_addr;
> +	uint32_t desc_avail, desc_offset;
> +	uint32_t mbuf_avail, mbuf_offset;
> +	uint32_t cpy_len;
> +	struct rte_mbuf *cur = m, *prev = m;
> +	struct virtio_net_hdr *hdr = NULL;
> +	uint16_t head_idx = vq->last_used_idx & (vq->size - 1);
> +	int wrap_counter = vq->used_wrap_counter;
> +	int rc = 0;
> +
> +	rte_spinlock_lock(&vq->access_lock);
> +
> +	if (unlikely(vq->enabled == 0))
> +		goto out;

It is unbalanced as it would unlock iotlb_rd_lock that isn't locked yet.

> +	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> +		vhost_user_iotlb_rd_lock(vq);
> +
> +	desc = &descs[vq->last_used_idx & (vq->size - 1)];
> +	if (unlikely((desc->len < dev->vhost_hlen)) ||
> +			(desc->flags & VRING_DESC_F_INDIRECT)) {
> +			RTE_LOG(ERR, VHOST_DATA,
> +				"INDIRECT not supported yet\n");
> +			rc = -1;
> +			goto out;
> +	}
> +
> +	desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
> +				      sizeof(*desc), VHOST_ACCESS_RO);
> +
> +	if (unlikely(!desc_addr)) {
> +		rc = -1;
> +		goto out;
> +	}
> +
> +	if (virtio_net_with_host_offload(dev)) {
> +		hdr = (struct virtio_net_hdr *)((uintptr_t)desc_addr);
> +		rte_prefetch0(hdr);
> +	}
> +
> +	/*
> +	 * A virtio driver normally uses at least 2 desc buffers
> +	 * for Tx: the first for storing the header, and others
> +	 * for storing the data.
> +	 */
> +	if (likely((desc->len == dev->vhost_hlen) &&
> +		   (desc->flags & VRING_DESC_F_NEXT) != 0)) {
> +		if ((++vq->last_used_idx & (vq->size - 1)) == 0)
> +			toggle_wrap_counter(vq);
> +
> +		desc = &descs[vq->last_used_idx & (vq->size - 1)];
> +
> +		if (unlikely(desc->flags & VRING_DESC_F_INDIRECT)) {
> +			RTE_LOG(ERR, VHOST_DATA,
> +				"INDIRECT not supported yet\n");
> +			rc = -1;
> +			goto out;
> +		}
> +
> +		desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
> +					      sizeof(*desc), VHOST_ACCESS_RO);
> +		if (unlikely(!desc_addr)) {
> +			rc = -1;
> +			goto out;
> +		}
> +
> +		desc_offset = 0;
> +		desc_avail  = desc->len;
> +	} else {
> +		desc_avail  = desc->len - dev->vhost_hlen;
> +		desc_offset = dev->vhost_hlen;
> +	}
> +
> +	rte_prefetch0((void *)(uintptr_t)(desc_addr + desc_offset));
> +
> +	PRINT_PACKET(dev, (uintptr_t)(desc_addr + desc_offset), desc_avail, 0);
> +
> +	mbuf_offset = 0;
> +	mbuf_avail  = m->buf_len - RTE_PKTMBUF_HEADROOM;
> +	while (1) {
> +		uint64_t hpa;
> +
> +		cpy_len = RTE_MIN(desc_avail, mbuf_avail);
> +
> +		/*
> +		 * A desc buf might across two host physical pages that are
> +		 * not continuous. In such case (gpa_to_hpa returns 0), data
> +		 * will be copied even though zero copy is enabled.
> +		 */
> +		if (unlikely(dev->dequeue_zero_copy && (hpa = gpa_to_hpa(dev,
> +					desc->addr + desc_offset, cpy_len)))) {
> +			cur->data_len = cpy_len;
> +			cur->data_off = 0;
> +			cur->buf_addr = (void *)(uintptr_t)desc_addr;
> +			cur->buf_physaddr = hpa;
> +
> +			/*
> +			 * In zero copy mode, one mbuf can only reference data
> +			 * for one or partial of one desc buff.
> +			 */
> +			mbuf_avail = cpy_len;
> +		} else {
> +			rte_memcpy(rte_pktmbuf_mtod_offset(cur, void *,
> +							   mbuf_offset),
> +				(void *)((uintptr_t)(desc_addr + desc_offset)),
> +				cpy_len);
> +		}
> +
> +		mbuf_avail  -= cpy_len;
> +		mbuf_offset += cpy_len;
> +		desc_avail  -= cpy_len;
> +		desc_offset += cpy_len;
> +
> +		/* This desc reaches to its end, get the next one */
> +		if (desc_avail == 0) {
> +			if ((desc->flags & VRING_DESC_F_NEXT) == 0)
> +				break;
> +
> +			if ((++vq->last_used_idx & (vq->size - 1)) == 0)
> +				toggle_wrap_counter(vq);
> +
> +			desc = &descs[vq->last_used_idx & (vq->size - 1)];
> +			if (unlikely(desc->flags & VRING_DESC_F_INDIRECT)) {
> +				RTE_LOG(ERR, VHOST_DATA, "INDIRECT not supported yet");
> +			}
> +
> +			desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
> +						      sizeof(*desc), VHOST_ACCESS_RO);
> +			if (unlikely(!desc_addr)) {
> +				rc = -1;
> +				goto out;
Not sure, but, shouldn't you break here instead so rings gets updated if
some of the descs have been processed?
> +			}
> +
> +			rte_prefetch0((void *)(uintptr_t)desc_addr);
> +
> +			desc_offset = 0;
> +			desc_avail  = desc->len;
> +
> +			PRINT_PACKET(dev, (uintptr_t)desc_addr, desc->len, 0);
> +		}
> +
> +		/*
> +		 * This mbuf reaches to its end, get a new one
> +		 * to hold more data.
> +		 */
> +		if (mbuf_avail == 0) {
> +			cur = rte_pktmbuf_alloc(mbuf_pool);
> +			if (unlikely(cur == NULL)) {
> +				RTE_LOG(ERR, VHOST_DATA, "Failed to "
> +					"allocate memory for mbuf.\n");
> +				rc = -1;
> +				goto out;
> +			}
> +
> +			prev->next = cur;
> +			prev->data_len = mbuf_offset;
> +			m->nb_segs += 1;
> +			m->pkt_len += mbuf_offset;
> +			prev = cur;
> +
> +			mbuf_offset = 0;
> +			mbuf_avail  = cur->buf_len - RTE_PKTMBUF_HEADROOM;
> +		}
> +	}
> +
> +	if (hdr)
> +		vhost_dequeue_offload(hdr, m);
> +
> +	if ((++vq->last_used_idx & (vq->size - 1)) == 0)
> +		toggle_wrap_counter(vq);
> +
> +	rte_smp_wmb();
> +	_set_desc_used(&descs[head_idx], wrap_counter);
> +
> +	prev->data_len = mbuf_offset;
> +	m->pkt_len    += mbuf_offset;
> +
> +out:
> +	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> +		vhost_user_iotlb_rd_unlock(vq);
> +	rte_spinlock_unlock(&vq->access_lock);
> +
> +	return rc;
> +}
> +
> +static inline uint16_t
> +vhost_dequeue_burst_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
> +			struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
> +			uint16_t count)
> +{
> +	uint16_t i;
> +	uint16_t idx;
> +	struct vring_desc_packed *desc = vq->desc_packed;
> +	int err;
> +
> +	count = RTE_MIN(MAX_PKT_BURST, count);
> +	for (i = 0; i < count; i++) {
> +		idx = vq->last_used_idx & (vq->size - 1);
> +		if (!desc_is_avail(vq, &desc[idx]))
> +			break;
> +		rte_smp_rmb();
> +
> +		pkts[i] = rte_pktmbuf_alloc(mbuf_pool);
> +		if (unlikely(pkts[i] == NULL)) {
> +			RTE_LOG(ERR, VHOST_DATA,
> +				"Failed to allocate memory for mbuf.\n");
> +			break;
> +		}
> +
> +		err = dequeue_desc_packed(dev, vq, mbuf_pool, pkts[i], desc);
> +		if (unlikely(err)) {
> +			rte_pktmbuf_free(pkts[i]);
> +			break;
> +		}
> +	}
> +
> +	rte_spinlock_unlock(&vq->access_lock);
I think you forgot to remove this line.

> +
> +	return i;
> +}
> +
>   uint16_t
>   rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
>   	struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count)
> 

Re: [PATCH v3 14/21] vhost: dequeue for packed queues

[Jens Freimann]

On Fri, Apr 06, 2018 at 11:30:10AM +0200, Maxime Coquelin wrote:
>
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>Implement code to dequeue and process descriptors from
>>the vring if VIRTIO_F_RING_PACKED is enabled.
>>
>>Check if descriptor was made available by driver by looking at
>>VIRTIO_F_DESC_AVAIL flag in descriptor. If so dequeue and set
>>the used flag VIRTIO_F_DESC_USED to the current value of the
>>used wrap counter.
>>
>>Used ring wrap counter needs to be toggled when last descriptor is
>>written out. This allows the host/guest to detect new descriptors even
>>after the ring has wrapped.
>>
>>Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>>---
>>  lib/librte_vhost/vhost.c      |   1 +
>>  lib/librte_vhost/vhost.h      |   1 +
>>  lib/librte_vhost/virtio_net.c | 228 ++++++++++++++++++++++++++++++++++++++++++
>>  3 files changed, 230 insertions(+)
>>
>>diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c
>>index 1f17cdd75..eb5a98875 100644
>>--- a/lib/librte_vhost/vhost.c
>>+++ b/lib/librte_vhost/vhost.c
>>@@ -185,6 +185,7 @@ init_vring_queue(struct virtio_net *dev, uint32_t vring_idx)
>>  	vq->kickfd = VIRTIO_UNINITIALIZED_EVENTFD;
>>  	vq->callfd = VIRTIO_UNINITIALIZED_EVENTFD;
>>+	vq->used_wrap_counter = 1;
>>  	vhost_user_iotlb_init(dev, vring_idx);
>>  	/* Backends are set to -1 indicating an inactive device. */
>>diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
>>index 20d78f883..c8aa946fd 100644
>>--- a/lib/librte_vhost/vhost.h
>>+++ b/lib/librte_vhost/vhost.h
>>@@ -112,6 +112,7 @@ struct vhost_virtqueue {
>>  	struct batch_copy_elem	*batch_copy_elems;
>>  	uint16_t		batch_copy_nb_elems;
>>+	uint16_t 		used_wrap_counter;
>>  	rte_rwlock_t	iotlb_lock;
>>  	rte_rwlock_t	iotlb_pending_lock;
>>diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
>>index ed7198dbb..7eea1da04 100644
>>--- a/lib/librte_vhost/virtio_net.c
>>+++ b/lib/librte_vhost/virtio_net.c
>>@@ -19,6 +19,7 @@
>>  #include "iotlb.h"
>>  #include "vhost.h"
>>+#include "virtio-1.1.h"
>>  #define MAX_PKT_BURST 32
>>@@ -1118,6 +1119,233 @@ restore_mbuf(struct rte_mbuf *m)
>>  	}
>>  }
>>+static inline uint16_t
>>+dequeue_desc_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
>>+	     struct rte_mempool *mbuf_pool, struct rte_mbuf *m,
>>+	     struct vring_desc_packed *descs)
>>+{
>>+	struct vring_desc_packed *desc;
>>+	uint64_t desc_addr;
>>+	uint32_t desc_avail, desc_offset;
>>+	uint32_t mbuf_avail, mbuf_offset;
>>+	uint32_t cpy_len;
>>+	struct rte_mbuf *cur = m, *prev = m;
>>+	struct virtio_net_hdr *hdr = NULL;
>>+	uint16_t head_idx = vq->last_used_idx & (vq->size - 1);
>>+	int wrap_counter = vq->used_wrap_counter;
>>+	int rc = 0;
>>+
>>+	rte_spinlock_lock(&vq->access_lock);
>>+
>>+	if (unlikely(vq->enabled == 0))
>>+		goto out;
>
>It is unbalanced as it would unlock iotlb_rd_lock that isn't locked yet.

yes, will fix
>
>>+	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
>>+		vhost_user_iotlb_rd_lock(vq);
>>+
>>+	desc = &descs[vq->last_used_idx & (vq->size - 1)];
>>+	if (unlikely((desc->len < dev->vhost_hlen)) ||
>>+			(desc->flags & VRING_DESC_F_INDIRECT)) {
>>+			RTE_LOG(ERR, VHOST_DATA,
>>+				"INDIRECT not supported yet\n");
>>+			rc = -1;
>>+			goto out;
>>+	}
>>+
>>+	desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
>>+				      sizeof(*desc), VHOST_ACCESS_RO);
>>+
>>+	if (unlikely(!desc_addr)) {
>>+		rc = -1;
>>+		goto out;
>>+	}
>>+
>>+	if (virtio_net_with_host_offload(dev)) {
>>+		hdr = (struct virtio_net_hdr *)((uintptr_t)desc_addr);
>>+		rte_prefetch0(hdr);
>>+	}
>>+
>>+	/*
>>+	 * A virtio driver normally uses at least 2 desc buffers
>>+	 * for Tx: the first for storing the header, and others
>>+	 * for storing the data.
>>+	 */
>>+	if (likely((desc->len == dev->vhost_hlen) &&
>>+		   (desc->flags & VRING_DESC_F_NEXT) != 0)) {
>>+		if ((++vq->last_used_idx & (vq->size - 1)) == 0)
>>+			toggle_wrap_counter(vq);
>>+
>>+		desc = &descs[vq->last_used_idx & (vq->size - 1)];
>>+
>>+		if (unlikely(desc->flags & VRING_DESC_F_INDIRECT)) {
>>+			RTE_LOG(ERR, VHOST_DATA,
>>+				"INDIRECT not supported yet\n");
>>+			rc = -1;
>>+			goto out;
>>+		}
>>+
>>+		desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
>>+					      sizeof(*desc), VHOST_ACCESS_RO);
>>+		if (unlikely(!desc_addr)) {
>>+			rc = -1;
>>+			goto out;
>>+		}
>>+
>>+		desc_offset = 0;
>>+		desc_avail  = desc->len;
>>+	} else {
>>+		desc_avail  = desc->len - dev->vhost_hlen;
>>+		desc_offset = dev->vhost_hlen;
>>+	}
>>+
>>+	rte_prefetch0((void *)(uintptr_t)(desc_addr + desc_offset));
>>+
>>+	PRINT_PACKET(dev, (uintptr_t)(desc_addr + desc_offset), desc_avail, 0);
>>+
>>+	mbuf_offset = 0;
>>+	mbuf_avail  = m->buf_len - RTE_PKTMBUF_HEADROOM;
>>+	while (1) {
>>+		uint64_t hpa;
>>+
>>+		cpy_len = RTE_MIN(desc_avail, mbuf_avail);
>>+
>>+		/*
>>+		 * A desc buf might across two host physical pages that are
>>+		 * not continuous. In such case (gpa_to_hpa returns 0), data
>>+		 * will be copied even though zero copy is enabled.
>>+		 */
>>+		if (unlikely(dev->dequeue_zero_copy && (hpa = gpa_to_hpa(dev,
>>+					desc->addr + desc_offset, cpy_len)))) {
>>+			cur->data_len = cpy_len;
>>+			cur->data_off = 0;
>>+			cur->buf_addr = (void *)(uintptr_t)desc_addr;
>>+			cur->buf_physaddr = hpa;
>>+
>>+			/*
>>+			 * In zero copy mode, one mbuf can only reference data
>>+			 * for one or partial of one desc buff.
>>+			 */
>>+			mbuf_avail = cpy_len;
>>+		} else {
>>+			rte_memcpy(rte_pktmbuf_mtod_offset(cur, void *,
>>+							   mbuf_offset),
>>+				(void *)((uintptr_t)(desc_addr + desc_offset)),
>>+				cpy_len);
>>+		}
>>+
>>+		mbuf_avail  -= cpy_len;
>>+		mbuf_offset += cpy_len;
>>+		desc_avail  -= cpy_len;
>>+		desc_offset += cpy_len;
>>+
>>+		/* This desc reaches to its end, get the next one */
>>+		if (desc_avail == 0) {
>>+			if ((desc->flags & VRING_DESC_F_NEXT) == 0)
>>+				break;
>>+
>>+			if ((++vq->last_used_idx & (vq->size - 1)) == 0)
>>+				toggle_wrap_counter(vq);
>>+
>>+			desc = &descs[vq->last_used_idx & (vq->size - 1)];
>>+			if (unlikely(desc->flags & VRING_DESC_F_INDIRECT)) {
>>+				RTE_LOG(ERR, VHOST_DATA, "INDIRECT not supported yet");
>>+			}
>>+
>>+			desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
>>+						      sizeof(*desc), VHOST_ACCESS_RO);
>>+			if (unlikely(!desc_addr)) {
>>+				rc = -1;
>>+				goto out;
>Not sure, but, shouldn't you break here instead so rings gets updated if
>some of the descs have been processed?

It should break here and in other places in this loop as well. will
fix it.
>>+			}
>>+
>>+			rte_prefetch0((void *)(uintptr_t)desc_addr);
>>+
>>+			desc_offset = 0;
>>+			desc_avail  = desc->len;
>>+
>>+			PRINT_PACKET(dev, (uintptr_t)desc_addr, desc->len, 0);
>>+		}
>>+
>>+		/*
>>+		 * This mbuf reaches to its end, get a new one
>>+		 * to hold more data.
>>+		 */
>>+		if (mbuf_avail == 0) {
>>+			cur = rte_pktmbuf_alloc(mbuf_pool);
>>+			if (unlikely(cur == NULL)) {
>>+				RTE_LOG(ERR, VHOST_DATA, "Failed to "
>>+					"allocate memory for mbuf.\n");
>>+				rc = -1;
>>+				goto out;
>>+			}
>>+
>>+			prev->next = cur;
>>+			prev->data_len = mbuf_offset;
>>+			m->nb_segs += 1;
>>+			m->pkt_len += mbuf_offset;
>>+			prev = cur;
>>+
>>+			mbuf_offset = 0;
>>+			mbuf_avail  = cur->buf_len - RTE_PKTMBUF_HEADROOM;
>>+		}
>>+	}
>>+
>>+	if (hdr)
>>+		vhost_dequeue_offload(hdr, m);
>>+
>>+	if ((++vq->last_used_idx & (vq->size - 1)) == 0)
>>+		toggle_wrap_counter(vq);
>>+
>>+	rte_smp_wmb();
>>+	_set_desc_used(&descs[head_idx], wrap_counter);
>>+
>>+	prev->data_len = mbuf_offset;
>>+	m->pkt_len    += mbuf_offset;
>>+
>>+out:
>>+	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
>>+		vhost_user_iotlb_rd_unlock(vq);
>>+	rte_spinlock_unlock(&vq->access_lock);
>>+
>>+	return rc;
>>+}
>>+
>>+static inline uint16_t
>>+vhost_dequeue_burst_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
>>+			struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
>>+			uint16_t count)
>>+{
>>+	uint16_t i;
>>+	uint16_t idx;
>>+	struct vring_desc_packed *desc = vq->desc_packed;
>>+	int err;
>>+
>>+	count = RTE_MIN(MAX_PKT_BURST, count);
>>+	for (i = 0; i < count; i++) {
>>+		idx = vq->last_used_idx & (vq->size - 1);
>>+		if (!desc_is_avail(vq, &desc[idx]))
>>+			break;
>>+		rte_smp_rmb();
>>+
>>+		pkts[i] = rte_pktmbuf_alloc(mbuf_pool);
>>+		if (unlikely(pkts[i] == NULL)) {
>>+			RTE_LOG(ERR, VHOST_DATA,
>>+				"Failed to allocate memory for mbuf.\n");
>>+			break;
>>+		}
>>+
>>+		err = dequeue_desc_packed(dev, vq, mbuf_pool, pkts[i], desc);
>>+		if (unlikely(err)) {
>>+			rte_pktmbuf_free(pkts[i]);
>>+			break;
>>+		}
>>+	}
>>+
>>+	rte_spinlock_unlock(&vq->access_lock);
>I think you forgot to remove this line.

yes

Thanks!

regards,
Jens 

[PATCH v3 15/21] vhost: packed queue enqueue path

[Jens Freimann]

Implement enqueue of packets to the receive virtqueue.

Set descriptor flag VIRTQ_DESC_F_USED and toggle used wrap counter if
last descriptor in ring is used. Perform a write memory barrier before
flags are written to descriptor.

Chained descriptors are not supported with this patch.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 lib/librte_vhost/virtio_net.c | 129 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 129 insertions(+)

diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
index 7eea1da04..578e5612e 100644
--- a/lib/librte_vhost/virtio_net.c
+++ b/lib/librte_vhost/virtio_net.c
@@ -695,6 +695,135 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
 	return pkt_idx;
 }
 
+static inline uint32_t __attribute__((always_inline))
+vhost_enqueue_burst_packed(struct virtio_net *dev, uint16_t queue_id,
+	      struct rte_mbuf **pkts, uint32_t count)
+{
+	struct vhost_virtqueue *vq;
+	struct vring_desc_packed *descs;
+	uint16_t idx;
+	uint16_t mask;
+	uint16_t i;
+
+	vq = dev->virtqueue[queue_id];
+
+	rte_spinlock_lock(&vq->access_lock);
+
+	if (unlikely(vq->enabled == 0)) {
+		i = 0;
+		goto out_access_unlock;
+	}
+
+	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
+		vhost_user_iotlb_rd_lock(vq);
+
+	descs = vq->desc_packed;
+	mask = vq->size - 1;
+
+	for (i = 0; i < count; i++) {
+		uint32_t desc_avail, desc_offset;
+		uint32_t mbuf_avail, mbuf_offset;
+		uint32_t cpy_len;
+		struct vring_desc_packed *desc;
+		uint64_t desc_addr;
+		struct virtio_net_hdr_mrg_rxbuf *hdr;
+		struct rte_mbuf *m = pkts[i];
+
+		/* XXX: there is an assumption that no desc will be chained */
+		idx = vq->last_used_idx & mask;
+		desc = &descs[idx];
+
+		if (!desc_is_avail(vq, desc))
+			break;
+		rte_smp_rmb();
+
+		desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
+					      sizeof(*desc), VHOST_ACCESS_RW);
+		/*
+		 * Checking of 'desc_addr' placed outside of 'unlikely' macro
+		 * to avoid performance issue with some versions of gcc (4.8.4
+		 * and 5.3.0) which otherwise stores offset on the stack instead
+		 * of in a register.
+		 */
+		if (unlikely(desc->len < dev->vhost_hlen) || !desc_addr)
+			break;
+
+		hdr = (struct virtio_net_hdr_mrg_rxbuf *)(uintptr_t)desc_addr;
+		virtio_enqueue_offload(m, &hdr->hdr);
+		vhost_log_write(dev, desc->addr, dev->vhost_hlen);
+		PRINT_PACKET(dev, (uintptr_t)desc_addr, dev->vhost_hlen, 0);
+
+		desc_offset = dev->vhost_hlen;
+		desc_avail  = desc->len - dev->vhost_hlen;
+
+		mbuf_avail  = rte_pktmbuf_data_len(m);
+		mbuf_offset = 0;
+		while (mbuf_avail != 0 || m->next != NULL) {
+			/* done with current mbuf, fetch next */
+			if (mbuf_avail == 0) {
+				m = m->next;
+
+				mbuf_offset = 0;
+				mbuf_avail  = rte_pktmbuf_data_len(m);
+			}
+
+			/* done with current desc buf, fetch next */
+			if (desc_avail == 0) {
+				if ((desc->flags & VRING_DESC_F_NEXT) == 0) {
+					/* Room in vring buffer is not enough */
+					goto out;
+				}
+
+				idx = (idx+1) & (vq->size - 1);
+				desc = &descs[idx];
+				if (unlikely(!desc_is_avail(vq, desc)))
+					goto out ;
+
+				desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
+							      sizeof(*desc),
+							      VHOST_ACCESS_RW);
+				if (unlikely(!desc_addr))
+					goto out;
+
+				desc_offset = 0;
+				desc_avail  = desc->len;
+			}
+
+			cpy_len = RTE_MIN(desc_avail, mbuf_avail);
+			rte_memcpy((void *)((uintptr_t)(desc_addr + desc_offset)),
+				rte_pktmbuf_mtod_offset(m, void *, mbuf_offset),
+				cpy_len);
+			vhost_log_write(dev, desc->addr + desc_offset, cpy_len);
+			PRINT_PACKET(dev, (uintptr_t)(desc_addr + desc_offset),
+				     cpy_len, 0);
+
+			mbuf_avail  -= cpy_len;
+			mbuf_offset += cpy_len;
+			desc_avail  -= cpy_len;
+			desc_offset += cpy_len;
+		}
+
+		descs[idx].len = pkts[i]->pkt_len + dev->vhost_hlen;
+		rte_smp_wmb();
+		set_desc_used(vq, desc);
+
+		vq->last_used_idx++;
+		if ((vq->last_used_idx & (vq->size - 1)) == 0)
+			toggle_wrap_counter(vq);
+	}
+
+out:
+	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
+		vhost_user_iotlb_rd_unlock(vq);
+
+out_access_unlock:
+	rte_spinlock_unlock(&vq->access_lock);
+
+	count = i;
+
+	return count;
+}
+
 uint16_t
 rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
 	struct rte_mbuf **pkts, uint16_t count)
-- 
2.14.3

Re: [PATCH v3 15/21] vhost: packed queue enqueue path

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Implement enqueue of packets to the receive virtqueue.
> 
> Set descriptor flag VIRTQ_DESC_F_USED and toggle used wrap counter if
> last descriptor in ring is used. Perform a write memory barrier before
> flags are written to descriptor.
> 
> Chained descriptors are not supported with this patch.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   lib/librte_vhost/virtio_net.c | 129 ++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 129 insertions(+)
> 
> diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
> index 7eea1da04..578e5612e 100644
> --- a/lib/librte_vhost/virtio_net.c
> +++ b/lib/librte_vhost/virtio_net.c
> @@ -695,6 +695,135 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>   	return pkt_idx;
>   }
>   
> +static inline uint32_t __attribute__((always_inline))
> +vhost_enqueue_burst_packed(struct virtio_net *dev, uint16_t queue_id,
> +	      struct rte_mbuf **pkts, uint32_t count)
> +{
> +	struct vhost_virtqueue *vq;
> +	struct vring_desc_packed *descs;
> +	uint16_t idx;
> +	uint16_t mask;
> +	uint16_t i;
> +
> +	vq = dev->virtqueue[queue_id];
> +
> +	rte_spinlock_lock(&vq->access_lock);
> +
> +	if (unlikely(vq->enabled == 0)) {
> +		i = 0;
> +		goto out_access_unlock;
> +	}
> +
> +	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> +		vhost_user_iotlb_rd_lock(vq);
> +
> +	descs = vq->desc_packed;
> +	mask = vq->size - 1;
> +
> +	for (i = 0; i < count; i++) {
> +		uint32_t desc_avail, desc_offset;
> +		uint32_t mbuf_avail, mbuf_offset;
> +		uint32_t cpy_len;
> +		struct vring_desc_packed *desc;
> +		uint64_t desc_addr;
> +		struct virtio_net_hdr_mrg_rxbuf *hdr;
> +		struct rte_mbuf *m = pkts[i];
> +
> +		/* XXX: there is an assumption that no desc will be chained */
Is this assumption still true?
If not what are the plan to fix this?

> +		idx = vq->last_used_idx & mask;
> +		desc = &descs[idx];
> +
> +		if (!desc_is_avail(vq, desc))
IIUC, it means the ring is full.
I think this is an unlikely case, so maybe better to use the unlikely
macro here.

> +			break;
> +		rte_smp_rmb();
> +
> +		desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
> +					      sizeof(*desc), VHOST_ACCESS_RW);
> +		/*
> +		 * Checking of 'desc_addr' placed outside of 'unlikely' macro
> +		 * to avoid performance issue with some versions of gcc (4.8.4
> +		 * and 5.3.0) which otherwise stores offset on the stack instead
> +		 * of in a register.
> +		 */
> +		if (unlikely(desc->len < dev->vhost_hlen) || !desc_addr)
> +			break;
> +
> +		hdr = (struct virtio_net_hdr_mrg_rxbuf *)(uintptr_t)desc_addr;
> +		virtio_enqueue_offload(m, &hdr->hdr);
> +		vhost_log_write(dev, desc->addr, dev->vhost_hlen);
> +		PRINT_PACKET(dev, (uintptr_t)desc_addr, dev->vhost_hlen, 0);
> +
> +		desc_offset = dev->vhost_hlen;
> +		desc_avail  = desc->len - dev->vhost_hlen;
> +
> +		mbuf_avail  = rte_pktmbuf_data_len(m);
> +		mbuf_offset = 0;
> +		while (mbuf_avail != 0 || m->next != NULL) {
> +			/* done with current mbuf, fetch next */
> +			if (mbuf_avail == 0) {
> +				m = m->next;
> +
> +				mbuf_offset = 0;
> +				mbuf_avail  = rte_pktmbuf_data_len(m);
> +			}
> +
> +			/* done with current desc buf, fetch next */
> +			if (desc_avail == 0) {
> +				if ((desc->flags & VRING_DESC_F_NEXT) == 0) {
> +					/* Room in vring buffer is not enough */
> +					goto out;
> +				}
> +
> +				idx = (idx+1) & (vq->size - 1);
> +				desc = &descs[idx];
> +				if (unlikely(!desc_is_avail(vq, desc)))
> +					goto out ;
> +
> +				desc_addr = vhost_iova_to_vva(dev, vq, desc->addr,
> +							      sizeof(*desc),
> +							      VHOST_ACCESS_RW);
> +				if (unlikely(!desc_addr))
> +					goto out;
> +
> +				desc_offset = 0;
> +				desc_avail  = desc->len;
> +			}
> +
> +			cpy_len = RTE_MIN(desc_avail, mbuf_avail);
> +			rte_memcpy((void *)((uintptr_t)(desc_addr + desc_offset)),
> +				rte_pktmbuf_mtod_offset(m, void *, mbuf_offset),
> +				cpy_len);
> +			vhost_log_write(dev, desc->addr + desc_offset, cpy_len);
> +			PRINT_PACKET(dev, (uintptr_t)(desc_addr + desc_offset),
> +				     cpy_len, 0);
> +
> +			mbuf_avail  -= cpy_len;
> +			mbuf_offset += cpy_len;
> +			desc_avail  -= cpy_len;
> +			desc_offset += cpy_len;
> +		}
> +
> +		descs[idx].len = pkts[i]->pkt_len + dev->vhost_hlen;
> +		rte_smp_wmb();
> +		set_desc_used(vq, desc);
> +
> +		vq->last_used_idx++;
> +		if ((vq->last_used_idx & (vq->size - 1)) == 0)
> +			toggle_wrap_counter(vq);
> +	}
> +
> +out:
> +	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> +		vhost_user_iotlb_rd_unlock(vq);
> +
> +out_access_unlock:
> +	rte_spinlock_unlock(&vq->access_lock);
> +
> +	count = i;
> +
> +	return count;
> +}
> +
>   uint16_t
>   rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
>   	struct rte_mbuf **pkts, uint16_t count)
> 

Re: [PATCH v3 15/21] vhost: packed queue enqueue path

[Jens Freimann]

On Fri, Apr 06, 2018 at 11:36:03AM +0200, Maxime Coquelin wrote:
>
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>Implement enqueue of packets to the receive virtqueue.
>>
>>Set descriptor flag VIRTQ_DESC_F_USED and toggle used wrap counter if
>>last descriptor in ring is used. Perform a write memory barrier before
>>flags are written to descriptor.
>>
>>Chained descriptors are not supported with this patch.
>>
>>Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>>---
>>  lib/librte_vhost/virtio_net.c | 129 ++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 129 insertions(+)
>>
>>diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
>>index 7eea1da04..578e5612e 100644
>>--- a/lib/librte_vhost/virtio_net.c
>>+++ b/lib/librte_vhost/virtio_net.c
>>@@ -695,6 +695,135 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>>  	return pkt_idx;
>>  }
>>+static inline uint32_t __attribute__((always_inline))
>>+vhost_enqueue_burst_packed(struct virtio_net *dev, uint16_t queue_id,
>>+	      struct rte_mbuf **pkts, uint32_t count)
>>+{
>>+	struct vhost_virtqueue *vq;
>>+	struct vring_desc_packed *descs;
>>+	uint16_t idx;
>>+	uint16_t mask;
>>+	uint16_t i;
>>+
>>+	vq = dev->virtqueue[queue_id];
>>+
>>+	rte_spinlock_lock(&vq->access_lock);
>>+
>>+	if (unlikely(vq->enabled == 0)) {
>>+		i = 0;
>>+		goto out_access_unlock;
>>+	}
>>+
>>+	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
>>+		vhost_user_iotlb_rd_lock(vq);
>>+
>>+	descs = vq->desc_packed;
>>+	mask = vq->size - 1;
>>+
>>+	for (i = 0; i < count; i++) {
>>+		uint32_t desc_avail, desc_offset;
>>+		uint32_t mbuf_avail, mbuf_offset;
>>+		uint32_t cpy_len;
>>+		struct vring_desc_packed *desc;
>>+		uint64_t desc_addr;
>>+		struct virtio_net_hdr_mrg_rxbuf *hdr;
>>+		struct rte_mbuf *m = pkts[i];
>>+
>>+		/* XXX: there is an assumption that no desc will be chained */
>Is this assumption still true?
>If not what are the plan to fix this?

This is a leftover from the prototype code. I checked the code and
don't see what it could still relate to except if it is supposed to
mean indirect instead of chained. I think the comment can be removed. 
>
>>+		idx = vq->last_used_idx & mask;
>>+		desc = &descs[idx];
>>+
>>+		if (!desc_is_avail(vq, desc))
>IIUC, it means the ring is full.
>I think this is an unlikely case, so maybe better to use the unlikely
>macro here.

yes, we can use unlikely here, will fix.

thanks!

regards,
Jens 

[PATCH v3 16/21] vhost: enable packed virtqueues

[Jens Freimann]

From: Yuanhan Liu <yuanhan.liu@linux.intel.com>

This patch enables the code do enqueue and dequeue packed to/from a
packed virtqueue.  Add feature bit for packed virtqueues as defined in
Virtio 1.1 draft.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
 lib/librte_vhost/vhost.h      | 1 +
 lib/librte_vhost/virtio_net.c | 7 ++++++-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
index c8aa946fd..a0b61b7b0 100644
--- a/lib/librte_vhost/vhost.h
+++ b/lib/librte_vhost/vhost.h
@@ -192,6 +192,7 @@ struct vhost_msg {
 				(1ULL << VIRTIO_NET_F_GUEST_ANNOUNCE) | \
 				(1ULL << VIRTIO_NET_F_MQ)      | \
 				(1ULL << VIRTIO_F_VERSION_1)   | \
+				(1ULL << VIRTIO_F_RING_PACKED) | \
 				(1ULL << VHOST_F_LOG_ALL)      | \
 				(1ULL << VHOST_USER_F_PROTOCOL_FEATURES) | \
 				(1ULL << VIRTIO_NET_F_GSO) | \
diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
index 578e5612e..18e67fdc1 100644
--- a/lib/librte_vhost/virtio_net.c
+++ b/lib/librte_vhost/virtio_net.c
@@ -840,7 +840,9 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
 		return 0;
 	}
 
-	if (dev->features & (1 << VIRTIO_NET_F_MRG_RXBUF))
+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
+		return vhost_enqueue_burst_packed(dev, queue_id, pkts, count);
+	else if (dev->features & (1 << VIRTIO_NET_F_MRG_RXBUF))
 		return virtio_dev_merge_rx(dev, queue_id, pkts, count);
 	else
 		return virtio_dev_rx(dev, queue_id, pkts, count);
@@ -1513,6 +1515,9 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 	if (unlikely(vq->enabled == 0))
 		goto out_access_unlock;
 
+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
+		return vhost_dequeue_burst_packed(dev, vq, mbuf_pool, pkts, count);
+
 	vq->batch_copy_nb_elems = 0;
 
 	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
-- 
2.14.3

[PATCH v3 17/21] net/virtio: disable ctrl virtqueue for packed rings

[Jens Freimann]

Signed-off-by: Jens Freiman <jfreimann@redhat.com>
---
 drivers/net/virtio/virtio_ethdev.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index dc220c743..7367d9c5d 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1157,6 +1157,13 @@ virtio_negotiate_features(struct virtio_hw *hw, uint64_t req_features)
 	req_features &= ~(1ull << VIRTIO_F_RING_PACKED);
 #endif
 
+	if (req_features & (1ULL << VIRTIO_F_RING_PACKED)) {
+		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_MAC_ADDR);
+		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_VQ);
+		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_RX);
+		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_VLAN);
+	}
+
 	/*
 	 * Negotiate features: Subset of device feature bits are written back
 	 * guest feature bits.
-- 
2.14.3

Re: [PATCH v3 17/21] net/virtio: disable ctrl virtqueue for packed rings

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Signed-off-by: Jens Freiman <jfreimann@redhat.com>
> ---
>   drivers/net/virtio/virtio_ethdev.c | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
> index dc220c743..7367d9c5d 100644
> --- a/drivers/net/virtio/virtio_ethdev.c
> +++ b/drivers/net/virtio/virtio_ethdev.c
> @@ -1157,6 +1157,13 @@ virtio_negotiate_features(struct virtio_hw *hw, uint64_t req_features)
>   	req_features &= ~(1ull << VIRTIO_F_RING_PACKED);
>   #endif
>   
> +	if (req_features & (1ULL << VIRTIO_F_RING_PACKED)) {
> +		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_MAC_ADDR);
> +		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_VQ);
> +		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_RX);
> +		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_VLAN);
> +	}
> +

Does packed ring not support ctrl vqs, or is it just a workaround while
it is implemented?

>   	/*
>   	 * Negotiate features: Subset of device feature bits are written back
>   	 * guest feature bits.
> 

Re: [PATCH v3 17/21] net/virtio: disable ctrl virtqueue for packed rings

[Jens Freimann]

On Fri, Apr 06, 2018 at 11:38:50AM +0200, Maxime Coquelin wrote:
>
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>Signed-off-by: Jens Freiman <jfreimann@redhat.com>
>>---
>>  drivers/net/virtio/virtio_ethdev.c | 7 +++++++
>>  1 file changed, 7 insertions(+)
>>
>>diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
>>index dc220c743..7367d9c5d 100644
>>--- a/drivers/net/virtio/virtio_ethdev.c
>>+++ b/drivers/net/virtio/virtio_ethdev.c
>>@@ -1157,6 +1157,13 @@ virtio_negotiate_features(struct virtio_hw *hw, uint64_t req_features)
>>  	req_features &= ~(1ull << VIRTIO_F_RING_PACKED);
>>  #endif
>>+	if (req_features & (1ULL << VIRTIO_F_RING_PACKED)) {
>>+		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_MAC_ADDR);
>>+		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_VQ);
>>+		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_RX);
>>+		req_features &= ~(1ull << VIRTIO_NET_F_CTRL_VLAN);
>>+	}
>>+
>
>Does packed ring not support ctrl vqs, or is it just a workaround while
>it is implemented?

packed queues support virtqueues, but I had not implemented it yet. I
have a patch for it though and will include it in v4.

regards,
Jens 
>
>>  	/*
>>  	 * Negotiate features: Subset of device feature bits are written back
>>  	 * guest feature bits.
>>

Re: [PATCH v3 17/21] net/virtio: disable ctrl virtqueue for packed rings

[Maxime Coquelin]

On 04/06/2018 11:43 AM, Jens Freimann wrote:
> On Fri, Apr 06, 2018 at 11:38:50AM +0200, Maxime Coquelin wrote:
>>
>>
>> On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>> Signed-off-by: Jens Freiman <jfreimann@redhat.com>
>>> ---
>>>  drivers/net/virtio/virtio_ethdev.c | 7 +++++++
>>>  1 file changed, 7 insertions(+)
>>>
>>> diff --git a/drivers/net/virtio/virtio_ethdev.c 
>>> b/drivers/net/virtio/virtio_ethdev.c
>>> index dc220c743..7367d9c5d 100644
>>> --- a/drivers/net/virtio/virtio_ethdev.c
>>> +++ b/drivers/net/virtio/virtio_ethdev.c
>>> @@ -1157,6 +1157,13 @@ virtio_negotiate_features(struct virtio_hw 
>>> *hw, uint64_t req_features)
>>>      req_features &= ~(1ull << VIRTIO_F_RING_PACKED);
>>>  #endif
>>> +    if (req_features & (1ULL << VIRTIO_F_RING_PACKED)) {
>>> +        req_features &= ~(1ull << VIRTIO_NET_F_CTRL_MAC_ADDR);
>>> +        req_features &= ~(1ull << VIRTIO_NET_F_CTRL_VQ);
>>> +        req_features &= ~(1ull << VIRTIO_NET_F_CTRL_RX);
>>> +        req_features &= ~(1ull << VIRTIO_NET_F_CTRL_VLAN);
>>> +    }
>>> +
>>
>> Does packed ring not support ctrl vqs, or is it just a workaround while
>> it is implemented?
> 
> packed queues support virtqueues, but I had not implemented it yet. I
> have a patch for it though and will include it in v4.

Great, so if you hadn't the patch, I think that a comment would be
welcome. But it does not apply as you have the patch in your queue!

Thanks,
Maxime

> 
> regards,
> Jens
>>
>>>      /*
>>>       * Negotiate features: Subset of device feature bits are written 
>>> back
>>>       * guest feature bits.
>>>

[PATCH v3 18/21] net/virtio: add support for mergeable buffers with packed virtqueues

[Jens Freimann]

Implement support for receiving merged buffers in virtio when packed virtqueues
are enabled.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 drivers/net/virtio/virtio_ethdev.c |  10 ++--
 drivers/net/virtio/virtio_rxtx.c   | 107 +++++++++++++++++++++++++++++++++----
 drivers/net/virtio/virtqueue.h     |   1 +
 3 files changed, 104 insertions(+), 14 deletions(-)

diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index 7367d9c5d..a3c3376d7 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1322,15 +1322,15 @@ set_rxtx_funcs(struct rte_eth_dev *eth_dev)
 {
 	struct virtio_hw *hw = eth_dev->data->dev_private;
 
-	/* workarount for packed vqs which don't support mrg_rxbuf at this point */
-	if (vtpci_packed_queue(hw) && vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF)) {
-		eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
-	} else if (hw->use_simple_rx) {
+	if (hw->use_simple_rx) {
 		PMD_INIT_LOG(INFO, "virtio: using simple Rx path on port %u",
 			eth_dev->data->port_id);
 		eth_dev->rx_pkt_burst = virtio_recv_pkts_vec;
 	} else if (vtpci_packed_queue(hw)) {
-		eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
+		if (vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF))
+			eth_dev->rx_pkt_burst = &virtio_recv_mergeable_pkts;
+		else
+			eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
 	} else if (vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF)) {
 		PMD_INIT_LOG(INFO,
 			"virtio: using mergeable buffer Rx path on port %u",
diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
index 9220ae661..a48ca6aaa 100644
--- a/drivers/net/virtio/virtio_rxtx.c
+++ b/drivers/net/virtio/virtio_rxtx.c
@@ -155,8 +155,8 @@ vq_ring_free_chain(struct virtqueue *vq, uint16_t desc_idx)
 	vq->vq_free_cnt = (uint16_t)(vq->vq_free_cnt + dxp->ndescs);
 	if ((dp->flags & VRING_DESC_F_INDIRECT) == 0) {
 		while (dp->flags & VRING_DESC_F_NEXT) {
-			desc_idx_last = dp->next;
-			dp = &vq->vq_ring.desc[dp->next];
+			desc_idx_last = desc_idx++;
+			dp = &vq->vq_ring.desc[desc_idx];
 		}
 	}
 	dxp->ndescs = 0;
@@ -177,6 +177,76 @@ vq_ring_free_chain(struct virtqueue *vq, uint16_t desc_idx)
 	dp->next = VQ_RING_DESC_CHAIN_END;
 }
 
+static void
+virtio_refill_packed(struct virtqueue *vq, uint16_t used_idx, struct virtnet_rx *rxvq)
+{
+	struct vq_desc_extra *dxp;
+	struct vring_desc_packed *descs = vq->vq_ring.desc_packed;
+	struct vring_desc_packed *desc;
+	struct rte_mbuf *nmb;
+
+	nmb = rte_mbuf_raw_alloc(rxvq->mpool);
+	if (unlikely(nmb == NULL)) {
+		struct rte_eth_dev *dev
+			= &rte_eth_devices[rxvq->port_id];
+		dev->data->rx_mbuf_alloc_failed++;
+		return;
+	}
+
+	desc = &descs[used_idx & (vq->vq_nentries - 1)];
+
+	dxp = &vq->vq_descx[used_idx & (vq->vq_nentries - 1)];
+
+	dxp->cookie = nmb;
+	dxp->ndescs = 1;
+
+	desc->addr = VIRTIO_MBUF_ADDR(nmb, vq) +
+		RTE_PKTMBUF_HEADROOM - vq->hw->vtnet_hdr_size;
+	desc->len = nmb->buf_len - RTE_PKTMBUF_HEADROOM +
+		vq->hw->vtnet_hdr_size;
+	desc->flags |= VRING_DESC_F_WRITE;
+
+
+}
+
+static uint16_t
+virtqueue_dequeue_burst_rx_packed(struct virtqueue *vq, struct rte_mbuf **rx_pkts,
+			   uint32_t *len, uint16_t num, struct virtnet_rx *rx_queue)
+{
+	struct rte_mbuf *cookie;
+	uint16_t used_idx;
+	struct vring_desc_packed *desc;
+	uint16_t i;
+
+	for (i = 0; i < num; i++) {
+		used_idx = (uint16_t)(vq->vq_used_cons_idx & (vq->vq_nentries - 1));
+		desc = &vq->vq_ring.desc_packed[used_idx];
+		if (!desc_is_used(desc))
+			return i;
+		len[i] = desc->len;
+		cookie = (struct rte_mbuf *)vq->vq_descx[used_idx].cookie;
+
+		if (unlikely(cookie == NULL)) {
+			PMD_DRV_LOG(ERR, "vring descriptor with no mbuf cookie at %u",
+				vq->vq_used_cons_idx);
+			break;
+		}
+		rte_prefetch0(cookie);
+		rte_packet_prefetch(rte_pktmbuf_mtod(cookie, void *));
+		rx_pkts[i] = cookie;
+
+		virtio_refill_packed(vq, used_idx, rx_queue);
+
+		rte_smp_wmb();
+		if ((vq->vq_used_cons_idx & (vq->vq_nentries - 1)) == 0)
+			toggle_wrap_counter(&vq->vq_ring);
+		set_desc_avail(&vq->vq_ring, desc);
+		vq->vq_used_cons_idx++;
+	}
+
+	return i;
+}
+
 static uint16_t
 virtqueue_dequeue_burst_rx(struct virtqueue *vq, struct rte_mbuf **rx_pkts,
 			   uint32_t *len, uint16_t num)
@@ -1066,12 +1136,16 @@ virtio_recv_mergeable_pkts(void *rx_queue,
 	uint32_t seg_res;
 	uint32_t hdr_size;
 	int offload;
+	uint32_t rx_num = 0;
 
 	nb_rx = 0;
 	if (unlikely(hw->started == 0))
 		return nb_rx;
 
-	nb_used = VIRTQUEUE_NUSED(vq);
+	if (vtpci_packed_queue(vq->hw))
+		nb_used = VIRTIO_MBUF_BURST_SZ;
+	else
+		nb_used = VIRTQUEUE_NUSED(vq);
 
 	virtio_rmb();
 
@@ -1085,14 +1159,23 @@ virtio_recv_mergeable_pkts(void *rx_queue,
 	hdr_size = hw->vtnet_hdr_size;
 	offload = rx_offload_enabled(hw);
 
+	vq->vq_used_idx = vq->vq_used_cons_idx;
+
 	while (i < nb_used) {
 		struct virtio_net_hdr_mrg_rxbuf *header;
 
 		if (nb_rx == nb_pkts)
 			break;
 
-		num = virtqueue_dequeue_burst_rx(vq, rcv_pkts, len, 1);
-		if (num != 1)
+		if (vtpci_packed_queue(vq->hw))
+			num = virtqueue_dequeue_burst_rx_packed(vq, rcv_pkts, len, 1, 
+				(struct virtnet_rx *) rx_queue);
+		else
+			num = virtqueue_dequeue_burst_rx(vq, rcv_pkts, len, 1);
+		if (num == 0) {
+			return nb_rx;
+		}
+		if (num != 1 )
 			continue;
 
 		i++;
@@ -1143,9 +1226,13 @@ virtio_recv_mergeable_pkts(void *rx_queue,
 			uint16_t  rcv_cnt =
 				RTE_MIN(seg_res, RTE_DIM(rcv_pkts));
 			if (likely(VIRTQUEUE_NUSED(vq) >= rcv_cnt)) {
-				uint32_t rx_num =
-					virtqueue_dequeue_burst_rx(vq,
-					rcv_pkts, len, rcv_cnt);
+				if (vtpci_packed_queue(vq->hw))
+					rx_num = virtqueue_dequeue_burst_rx_packed(vq,
+							     rcv_pkts, len, rcv_cnt,
+							     (struct virtnet_rx *) rx_queue);
+				else
+					rx_num = virtqueue_dequeue_burst_rx(vq,
+							      rcv_pkts, len, rcv_cnt);
 				i += rx_num;
 				rcv_cnt = rx_num;
 			} else {
@@ -1189,6 +1276,9 @@ virtio_recv_mergeable_pkts(void *rx_queue,
 
 	rxvq->stats.packets += nb_rx;
 
+	if (vtpci_packed_queue(vq->hw))
+		return nb_rx;
+
 	/* Allocate new mbuf for the used descriptor */
 	error = ENOSPC;
 	while (likely(!virtqueue_full(vq))) {
@@ -1209,7 +1299,6 @@ virtio_recv_mergeable_pkts(void *rx_queue,
 
 	if (likely(nb_enqueued)) {
 		vq_update_avail_idx(vq);
-
 		if (unlikely(virtqueue_kick_prepare(vq))) {
 			virtqueue_notify(vq);
 			PMD_RX_LOG(DEBUG, "Notified");
diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
index 7e265bf93..7196bd717 100644
--- a/drivers/net/virtio/virtqueue.h
+++ b/drivers/net/virtio/virtqueue.h
@@ -171,6 +171,7 @@ struct virtqueue {
 	 * trails vq_ring.used->idx.
 	 */
 	uint16_t vq_used_cons_idx;
+	uint16_t vq_used_idx;
 	uint16_t vq_nentries;  /**< vring desc numbers */
 	uint16_t vq_free_cnt;  /**< num of desc available */
 	uint16_t vq_avail_idx; /**< sync until needed */
-- 
2.14.3

Re: [PATCH v3 18/21] net/virtio: add support for mergeable buffers with packed virtqueues

[Maxime Coquelin]

Maybe better to place this patch right after the non-mrg one.

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Implement support for receiving merged buffers in virtio when packed virtqueues
> are enabled.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   drivers/net/virtio/virtio_ethdev.c |  10 ++--
>   drivers/net/virtio/virtio_rxtx.c   | 107 +++++++++++++++++++++++++++++++++----
>   drivers/net/virtio/virtqueue.h     |   1 +
>   3 files changed, 104 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
> index 7367d9c5d..a3c3376d7 100644
> --- a/drivers/net/virtio/virtio_ethdev.c
> +++ b/drivers/net/virtio/virtio_ethdev.c
> @@ -1322,15 +1322,15 @@ set_rxtx_funcs(struct rte_eth_dev *eth_dev)
>   {
>   	struct virtio_hw *hw = eth_dev->data->dev_private;
>   
> -	/* workarount for packed vqs which don't support mrg_rxbuf at this point */
> -	if (vtpci_packed_queue(hw) && vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF)) {
> -		eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
> -	} else if (hw->use_simple_rx) {
> +	if (hw->use_simple_rx) {
>   		PMD_INIT_LOG(INFO, "virtio: using simple Rx path on port %u",
>   			eth_dev->data->port_id);
>   		eth_dev->rx_pkt_burst = virtio_recv_pkts_vec;
>   	} else if (vtpci_packed_queue(hw)) {
> -		eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
> +		if (vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF))
> +			eth_dev->rx_pkt_burst = &virtio_recv_mergeable_pkts;
> +		else
> +			eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
>   	} else if (vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF)) {
>   		PMD_INIT_LOG(INFO,
>   			"virtio: using mergeable buffer Rx path on port %u",
> diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
> index 9220ae661..a48ca6aaa 100644
> --- a/drivers/net/virtio/virtio_rxtx.c
> +++ b/drivers/net/virtio/virtio_rxtx.c
> @@ -155,8 +155,8 @@ vq_ring_free_chain(struct virtqueue *vq, uint16_t desc_idx)
>   	vq->vq_free_cnt = (uint16_t)(vq->vq_free_cnt + dxp->ndescs);
>   	if ((dp->flags & VRING_DESC_F_INDIRECT) == 0) {
>   		while (dp->flags & VRING_DESC_F_NEXT) {
> -			desc_idx_last = dp->next;
> -			dp = &vq->vq_ring.desc[dp->next];
> +			desc_idx_last = desc_idx++;
> +			dp = &vq->vq_ring.desc[desc_idx];

Are you sure this change is in the right patch?

>   		}
>   	}
>   	dxp->ndescs = 0;
> @@ -177,6 +177,76 @@ vq_ring_free_chain(struct virtqueue *vq, uint16_t desc_idx)
>   	dp->next = VQ_RING_DESC_CHAIN_END;
>   }
>   

Re: [PATCH v3 18/21] net/virtio: add support for mergeable buffers with packed virtqueues

[Tiwei Bie]

On Thu, Apr 05, 2018 at 12:10:28PM +0200, Jens Freimann wrote:
> Implement support for receiving merged buffers in virtio when packed virtqueues
> are enabled.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>  drivers/net/virtio/virtio_ethdev.c |  10 ++--
>  drivers/net/virtio/virtio_rxtx.c   | 107 +++++++++++++++++++++++++++++++++----
>  drivers/net/virtio/virtqueue.h     |   1 +
>  3 files changed, 104 insertions(+), 14 deletions(-)
> 
> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
> index 7367d9c5d..a3c3376d7 100644
> --- a/drivers/net/virtio/virtio_ethdev.c
> +++ b/drivers/net/virtio/virtio_ethdev.c
> @@ -1322,15 +1322,15 @@ set_rxtx_funcs(struct rte_eth_dev *eth_dev)
>  {
>  	struct virtio_hw *hw = eth_dev->data->dev_private;
>  
> -	/* workarount for packed vqs which don't support mrg_rxbuf at this point */
> -	if (vtpci_packed_queue(hw) && vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF)) {
> -		eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
> -	} else if (hw->use_simple_rx) {
> +	if (hw->use_simple_rx) {
>  		PMD_INIT_LOG(INFO, "virtio: using simple Rx path on port %u",
>  			eth_dev->data->port_id);
>  		eth_dev->rx_pkt_burst = virtio_recv_pkts_vec;

The logic isn't right here. It means when hw->use_simple_rx
is true, virtio_recv_pkts_vec() will be selected as the rx
function for packed ring. But virtio_recv_pkts_vec() doesn't
support packed ring now.


>  	} else if (vtpci_packed_queue(hw)) {
> -		eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
> +		if (vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF))
> +			eth_dev->rx_pkt_burst = &virtio_recv_mergeable_pkts;
> +		else
> +			eth_dev->rx_pkt_burst = &virtio_recv_pkts_packed;
>  	} else if (vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF)) {
>  		PMD_INIT_LOG(INFO,
>  			"virtio: using mergeable buffer Rx path on port %u",
[...]

[PATCH v3 19/21] vhost: support mergeable rx buffers with packed queues

[Jens Freimann]

This implements support for mergeable receive buffers in vhost when using
packed virtqueues. The difference to split virtqueues is not big, it differs
mostly where descriptor flags are touched and virtio features are checked.

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 lib/librte_vhost/vhost.c      |   2 +
 lib/librte_vhost/virtio_net.c | 160 +++++++++++++++++++++++++++++++++---------
 2 files changed, 127 insertions(+), 35 deletions(-)

diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c
index eb5a98875..3c633e71e 100644
--- a/lib/librte_vhost/vhost.c
+++ b/lib/librte_vhost/vhost.c
@@ -580,6 +580,8 @@ rte_vhost_enable_guest_notification(int vid, uint16_t queue_id, int enable)
 
 	if (dev == NULL)
 		return -1;
+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
+		return -1;
 
 	if (enable) {
 		RTE_LOG(ERR, VHOST_CONFIG,
diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
index 18e67fdc1..b82c24081 100644
--- a/lib/librte_vhost/virtio_net.c
+++ b/lib/librte_vhost/virtio_net.c
@@ -401,17 +401,53 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
 }
 
 static __rte_always_inline int
-fill_vec_buf(struct virtio_net *dev, struct vhost_virtqueue *vq,
-			 uint32_t avail_idx, uint32_t *vec_idx,
-			 struct buf_vector *buf_vec, uint16_t *desc_chain_head,
-			 uint16_t *desc_chain_len)
+__fill_vec_buf_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
+			 struct buf_vector *buf_vec,
+			 uint32_t *len, uint32_t *vec_id)
+{
+	uint16_t idx = vq->last_avail_idx & (vq->size - 1);
+	struct vring_desc_packed *descs= vq->desc_packed;
+	uint32_t _vec_id = *vec_id;
+
+	if (vq->desc_packed[idx].flags & VRING_DESC_F_INDIRECT) {
+		descs = (struct vring_desc_packed *)(uintptr_t)
+			vhost_iova_to_vva(dev, vq, vq->desc_packed[idx].addr,
+						vq->desc_packed[idx].len,
+						VHOST_ACCESS_RO);
+		if (unlikely(!descs))
+			return -1;
+
+		idx = 0;
+	}
+
+	while (1) {
+		if (unlikely(_vec_id >= BUF_VECTOR_MAX || idx >= vq->size))
+			return -1;
+
+		*len += descs[idx & (vq->size - 1)].len;
+		buf_vec[_vec_id].buf_addr = descs[idx].addr;
+		buf_vec[_vec_id].buf_len  = descs[idx].len;
+		buf_vec[_vec_id].desc_idx = idx;
+		_vec_id++;
+
+		if ((descs[idx & (vq->size - 1)].flags & VRING_DESC_F_NEXT) == 0)
+			break;
+
+		idx++;
+	}
+	*vec_id = _vec_id;
+
+	return 0;
+}
+
+static __rte_always_inline int
+__fill_vec_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
+			 struct buf_vector *buf_vec,
+			 uint32_t *len, uint32_t *vec_id, uint32_t avail_idx)
 {
 	uint16_t idx = vq->avail->ring[avail_idx & (vq->size - 1)];
-	uint32_t vec_id = *vec_idx;
-	uint32_t len    = 0;
 	struct vring_desc *descs = vq->desc;
-
-	*desc_chain_head = idx;
+	uint32_t _vec_id = *vec_id;
 
 	if (vq->desc[idx].flags & VRING_DESC_F_INDIRECT) {
 		descs = (struct vring_desc *)(uintptr_t)
@@ -425,20 +461,53 @@ fill_vec_buf(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	}
 
 	while (1) {
-		if (unlikely(vec_id >= BUF_VECTOR_MAX || idx >= vq->size))
+		if (unlikely(_vec_id >= BUF_VECTOR_MAX || idx >= vq->size))
 			return -1;
 
-		len += descs[idx].len;
-		buf_vec[vec_id].buf_addr = descs[idx].addr;
-		buf_vec[vec_id].buf_len  = descs[idx].len;
-		buf_vec[vec_id].desc_idx = idx;
-		vec_id++;
+		*len += descs[idx].len;
+		buf_vec[_vec_id].buf_addr = descs[idx].addr;
+		buf_vec[_vec_id].buf_len  = descs[idx].len;
+		buf_vec[_vec_id].desc_idx = idx;
+		_vec_id++;
 
 		if ((descs[idx].flags & VRING_DESC_F_NEXT) == 0)
 			break;
 
 		idx = descs[idx].next;
 	}
+	*vec_id = _vec_id;
+
+	return 0;
+}
+
+static __rte_always_inline int
+fill_vec_buf(struct virtio_net *dev, struct vhost_virtqueue *vq,
+			 uint32_t avail_idx, uint32_t *vec_idx,
+			 struct buf_vector *buf_vec, uint16_t *desc_chain_head,
+			 uint16_t *desc_chain_len)
+{
+	uint16_t idx;
+	uint32_t vec_id = *vec_idx;
+	uint32_t len    = 0;
+
+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
+		idx = vq->last_avail_idx & (vq->size -1);
+	} else {
+		idx = vq->avail->ring[avail_idx & (vq->size - 1)];
+	}
+
+
+	*desc_chain_head = idx;
+
+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
+		if (__fill_vec_buf_packed(dev, vq,
+				buf_vec, &len, &vec_id))
+			return -1;
+	} else {
+		if (__fill_vec_buf_split(dev, vq,
+				buf_vec, &len, &vec_id, avail_idx))
+			return -1;
+	}
 
 	*desc_chain_len = len;
 	*vec_idx = vec_id;
@@ -465,14 +534,16 @@ reserve_avail_buf_mergeable(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	cur_idx  = vq->last_avail_idx;
 
 	while (size > 0) {
-		if (unlikely(cur_idx == avail_head))
+		if (unlikely(cur_idx == avail_head) &&
+			!(dev->features & (1ull < VIRTIO_F_RING_PACKED)))
 			return -1;
 
 		if (unlikely(fill_vec_buf(dev, vq, cur_idx, &vec_idx, buf_vec,
 						&head_idx, &len) < 0))
 			return -1;
 		len = RTE_MIN(len, size);
-		update_shadow_used_ring(vq, head_idx, len);
+		if (!(dev->features & (1ULL << VIRTIO_F_RING_PACKED)))
+			update_shadow_used_ring(vq, head_idx, len);
 		size -= len;
 
 		cur_idx++;
@@ -620,6 +691,8 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
 	uint16_t num_buffers;
 	struct buf_vector buf_vec[BUF_VECTOR_MAX];
 	uint16_t avail_head;
+	uint16_t i;
+	struct vring_desc_packed *descs = NULL;
 
 	VHOST_LOG_DEBUG(VHOST_DATA, "(%d) %s\n", dev->vid, __func__);
 	if (unlikely(!is_valid_virt_queue_idx(queue_id, 0, dev->nr_vring))) {
@@ -634,7 +707,6 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
 
 	if (unlikely(vq->enabled == 0))
 		goto out_access_unlock;
-
 	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
 		vhost_user_iotlb_rd_lock(vq);
 
@@ -648,10 +720,14 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
 
 	vq->batch_copy_nb_elems = 0;
 
-	rte_prefetch0(&vq->avail->ring[vq->last_avail_idx & (vq->size - 1)]);
-
-	vq->shadow_used_idx = 0;
-	avail_head = *((volatile uint16_t *)&vq->avail->idx);
+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
+		avail_head = vq->last_avail_idx;
+		descs = vq->desc_packed;
+	} else {
+		rte_prefetch0(&vq->avail->ring[vq->last_avail_idx & (vq->size - 1)]);
+		avail_head = *((volatile uint16_t *)&vq->avail->idx);
+		vq->shadow_used_idx = 0;
+	}
 	for (pkt_idx = 0; pkt_idx < count; pkt_idx++) {
 		uint32_t pkt_len = pkts[pkt_idx]->pkt_len + dev->vhost_hlen;
 
@@ -661,7 +737,9 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
 			VHOST_LOG_DEBUG(VHOST_DATA,
 				"(%d) failed to get enough desc from vring\n",
 				dev->vid);
-			vq->shadow_used_idx -= num_buffers;
+
+			if (!dev->features & (1ULL & VIRTIO_F_RING_PACKED))
+				vq->shadow_used_idx -= num_buffers;
 			break;
 		}
 
@@ -671,7 +749,8 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
 
 		if (copy_mbuf_to_desc_mergeable(dev, vq, pkts[pkt_idx],
 						buf_vec, num_buffers) < 0) {
-			vq->shadow_used_idx -= num_buffers;
+			if (!dev->features & (1ULL & VIRTIO_F_RING_PACKED))
+				vq->shadow_used_idx -= num_buffers;
 			break;
 		}
 
@@ -680,9 +759,18 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
 
 	do_data_copy_enqueue(dev, vq);
 
-	if (likely(vq->shadow_used_idx)) {
-		flush_shadow_used_ring(dev, vq);
-		vhost_vring_call(dev, vq);
+	if (!(dev->features & (1ULL << VIRTIO_F_RING_PACKED))) {
+		if (likely(vq->shadow_used_idx)) {
+			flush_shadow_used_ring(dev, vq);
+			vhost_vring_call(dev, vq);
+		}
+	} else {
+		rte_smp_wmb();
+		for (i = avail_head; i < vq->last_avail_idx; i++) {
+			if ((i & (vq->size - 1)) == 0)
+				toggle_wrap_counter(vq);
+			set_desc_used(vq, &descs[i & (vq->size - 1)]);
+		}
 	}
 
 out:
@@ -774,7 +862,7 @@ vhost_enqueue_burst_packed(struct virtio_net *dev, uint16_t queue_id,
 					goto out;
 				}
 
-				idx = (idx+1) & (vq->size - 1);
+				idx = (idx + 1) & mask;
 				desc = &descs[idx];
 				if (unlikely(!desc_is_avail(vq, desc)))
 					goto out ;
@@ -840,10 +928,11 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
 		return 0;
 	}
 
-	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
-		return vhost_enqueue_burst_packed(dev, queue_id, pkts, count);
-	else if (dev->features & (1 << VIRTIO_NET_F_MRG_RXBUF))
+	if ((dev->features & (1 << VIRTIO_NET_F_MRG_RXBUF)) &&
+	    (dev->features & (1ULL << VIRTIO_F_RING_PACKED)))
 		return virtio_dev_merge_rx(dev, queue_id, pkts, count);
+	else if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
+		return vhost_enqueue_burst_packed(dev, queue_id, pkts, count);
 	else
 		return virtio_dev_rx(dev, queue_id, pkts, count);
 }
@@ -1266,8 +1355,6 @@ dequeue_desc_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	int wrap_counter = vq->used_wrap_counter;
 	int rc = 0;
 
-	rte_spinlock_lock(&vq->access_lock);
-
 	if (unlikely(vq->enabled == 0))
 		goto out;
 
@@ -1451,6 +1538,9 @@ vhost_dequeue_burst_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
 	struct vring_desc_packed *desc = vq->desc_packed;
 	int err;
 
+	if (unlikely(rte_spinlock_trylock(&vq->access_lock) == 0))
+		return 0;
+
 	count = RTE_MIN(MAX_PKT_BURST, count);
 	for (i = 0; i < count; i++) {
 		idx = vq->last_used_idx & (vq->size - 1);
@@ -1509,15 +1599,15 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
 
 	vq = dev->virtqueue[queue_id];
 
+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
+		return vhost_dequeue_burst_packed(dev, vq, mbuf_pool, pkts, count);
+
 	if (unlikely(rte_spinlock_trylock(&vq->access_lock) == 0))
 		return 0;
 
 	if (unlikely(vq->enabled == 0))
 		goto out_access_unlock;
 
-	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
-		return vhost_dequeue_burst_packed(dev, vq, mbuf_pool, pkts, count);
-
 	vq->batch_copy_nb_elems = 0;
 
 	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
-- 
2.14.3

Re: [PATCH v3 19/21] vhost: support mergeable rx buffers with packed queues

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> This implements support for mergeable receive buffers in vhost when using
> packed virtqueues. The difference to split virtqueues is not big, it differs
> mostly where descriptor flags are touched and virtio features are checked.
> 
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   lib/librte_vhost/vhost.c      |   2 +
>   lib/librte_vhost/virtio_net.c | 160 +++++++++++++++++++++++++++++++++---------
>   2 files changed, 127 insertions(+), 35 deletions(-)
> 
> diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c
> index eb5a98875..3c633e71e 100644
> --- a/lib/librte_vhost/vhost.c
> +++ b/lib/librte_vhost/vhost.c
> @@ -580,6 +580,8 @@ rte_vhost_enable_guest_notification(int vid, uint16_t queue_id, int enable)
>   
>   	if (dev == NULL)
>   		return -1;
> +	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
> +		return -1;

Is this change really related to mergeable buffers?

>   
>   	if (enable) {
>   		RTE_LOG(ERR, VHOST_CONFIG,
> diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
> index 18e67fdc1..b82c24081 100644
> --- a/lib/librte_vhost/virtio_net.c
> +++ b/lib/librte_vhost/virtio_net.c
> @@ -401,17 +401,53 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
>   }
>   
>   static __rte_always_inline int
> -fill_vec_buf(struct virtio_net *dev, struct vhost_virtqueue *vq,
> -			 uint32_t avail_idx, uint32_t *vec_idx,
> -			 struct buf_vector *buf_vec, uint16_t *desc_chain_head,
> -			 uint16_t *desc_chain_len)
> +__fill_vec_buf_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
> +			 struct buf_vector *buf_vec,
> +			 uint32_t *len, uint32_t *vec_id)
> +{
> +	uint16_t idx = vq->last_avail_idx & (vq->size - 1);
> +	struct vring_desc_packed *descs= vq->desc_packed;
> +	uint32_t _vec_id = *vec_id;
> +
> +	if (vq->desc_packed[idx].flags & VRING_DESC_F_INDIRECT) {
> +		descs = (struct vring_desc_packed *)(uintptr_t)
> +			vhost_iova_to_vva(dev, vq, vq->desc_packed[idx].addr,
> +						vq->desc_packed[idx].len,
> +						VHOST_ACCESS_RO);
> +		if (unlikely(!descs))
> +			return -1;
> +
> +		idx = 0;
> +	}
> +
> +	while (1) {
> +		if (unlikely(_vec_id >= BUF_VECTOR_MAX || idx >= vq->size))
> +			return -1;
> +
> +		*len += descs[idx & (vq->size - 1)].len;
> +		buf_vec[_vec_id].buf_addr = descs[idx].addr;
> +		buf_vec[_vec_id].buf_len  = descs[idx].len;
> +		buf_vec[_vec_id].desc_idx = idx;
> +		_vec_id++;
> +
> +		if ((descs[idx & (vq->size - 1)].flags & VRING_DESC_F_NEXT) == 0)
> +			break;
> +
> +		idx++;
> +	}
> +	*vec_id = _vec_id;
> +
> +	return 0;
> +}
> +
> +static __rte_always_inline int
> +__fill_vec_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
> +			 struct buf_vector *buf_vec,
> +			 uint32_t *len, uint32_t *vec_id, uint32_t avail_idx)
>   {
>   	uint16_t idx = vq->avail->ring[avail_idx & (vq->size - 1)];
> -	uint32_t vec_id = *vec_idx;
> -	uint32_t len    = 0;
>   	struct vring_desc *descs = vq->desc;
> -
> -	*desc_chain_head = idx;
> +	uint32_t _vec_id = *vec_id;
>   
>   	if (vq->desc[idx].flags & VRING_DESC_F_INDIRECT) {
>   		descs = (struct vring_desc *)(uintptr_t)
> @@ -425,20 +461,53 @@ fill_vec_buf(struct virtio_net *dev, struct vhost_virtqueue *vq,
>   	}
>   
>   	while (1) {
> -		if (unlikely(vec_id >= BUF_VECTOR_MAX || idx >= vq->size))
> +		if (unlikely(_vec_id >= BUF_VECTOR_MAX || idx >= vq->size))
>   			return -1;
>   
> -		len += descs[idx].len;
> -		buf_vec[vec_id].buf_addr = descs[idx].addr;
> -		buf_vec[vec_id].buf_len  = descs[idx].len;
> -		buf_vec[vec_id].desc_idx = idx;
> -		vec_id++;
> +		*len += descs[idx].len;
> +		buf_vec[_vec_id].buf_addr = descs[idx].addr;
> +		buf_vec[_vec_id].buf_len  = descs[idx].len;
> +		buf_vec[_vec_id].desc_idx = idx;
> +		_vec_id++;
>   
>   		if ((descs[idx].flags & VRING_DESC_F_NEXT) == 0)
>   			break;
>   
>   		idx = descs[idx].next;
>   	}
> +	*vec_id = _vec_id;
> +
> +	return 0;
> +}
> +
> +static __rte_always_inline int
> +fill_vec_buf(struct virtio_net *dev, struct vhost_virtqueue *vq,
> +			 uint32_t avail_idx, uint32_t *vec_idx,
> +			 struct buf_vector *buf_vec, uint16_t *desc_chain_head,
> +			 uint16_t *desc_chain_len)
> +{
> +	uint16_t idx;
> +	uint32_t vec_id = *vec_idx;
> +	uint32_t len    = 0;
> +
> +	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
> +		idx = vq->last_avail_idx & (vq->size -1);
> +	} else {
> +		idx = vq->avail->ring[avail_idx & (vq->size - 1)];
> +	}
> +
> +
> +	*desc_chain_head = idx;
> +
> +	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
> +		if (__fill_vec_buf_packed(dev, vq,
> +				buf_vec, &len, &vec_id))
> +			return -1;
> +	} else {
> +		if (__fill_vec_buf_split(dev, vq,
> +				buf_vec, &len, &vec_id, avail_idx))
> +			return -1;
> +	}
>   
>   	*desc_chain_len = len;
>   	*vec_idx = vec_id;
> @@ -465,14 +534,16 @@ reserve_avail_buf_mergeable(struct virtio_net *dev, struct vhost_virtqueue *vq,
>   	cur_idx  = vq->last_avail_idx;
>   
>   	while (size > 0) {
> -		if (unlikely(cur_idx == avail_head))
> +		if (unlikely(cur_idx == avail_head) &&
> +			!(dev->features & (1ull < VIRTIO_F_RING_PACKED)))
>   			return -1;
>   
>   		if (unlikely(fill_vec_buf(dev, vq, cur_idx, &vec_idx, buf_vec,
>   						&head_idx, &len) < 0))
>   			return -1;
>   		len = RTE_MIN(len, size);
> -		update_shadow_used_ring(vq, head_idx, len);
> +		if (!(dev->features & (1ULL << VIRTIO_F_RING_PACKED)))

This is cosmetics, but I wonder if we could have a helper like:

static inline bool
vring_is_packed(struct virtio_net *dev) {
	return dev->features & (1ULL << VIRTIO_F_RING_PACKED);
}

> +			update_shadow_used_ring(vq, head_idx, len);
>   		size -= len;
>   
>   		cur_idx++;
> @@ -620,6 +691,8 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>   	uint16_t num_buffers;
>   	struct buf_vector buf_vec[BUF_VECTOR_MAX];
>   	uint16_t avail_head;
> +	uint16_t i;
> +	struct vring_desc_packed *descs = NULL;
>   
>   	VHOST_LOG_DEBUG(VHOST_DATA, "(%d) %s\n", dev->vid, __func__);
>   	if (unlikely(!is_valid_virt_queue_idx(queue_id, 0, dev->nr_vring))) {
> @@ -634,7 +707,6 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>   
>   	if (unlikely(vq->enabled == 0))
>   		goto out_access_unlock;
> -
>   	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
>   		vhost_user_iotlb_rd_lock(vq);
>   
> @@ -648,10 +720,14 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>   
>   	vq->batch_copy_nb_elems = 0;
>   
> -	rte_prefetch0(&vq->avail->ring[vq->last_avail_idx & (vq->size - 1)]);
> -
> -	vq->shadow_used_idx = 0;
> -	avail_head = *((volatile uint16_t *)&vq->avail->idx);
> +	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
> +		avail_head = vq->last_avail_idx;
> +		descs = vq->desc_packed;
> +	} else {
> +		rte_prefetch0(&vq->avail->ring[vq->last_avail_idx & (vq->size - 1)]);
> +		avail_head = *((volatile uint16_t *)&vq->avail->idx);
> +		vq->shadow_used_idx = 0;
> +	}
>   	for (pkt_idx = 0; pkt_idx < count; pkt_idx++) {
>   		uint32_t pkt_len = pkts[pkt_idx]->pkt_len + dev->vhost_hlen;
>   
> @@ -661,7 +737,9 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>   			VHOST_LOG_DEBUG(VHOST_DATA,
>   				"(%d) failed to get enough desc from vring\n",
>   				dev->vid);
> -			vq->shadow_used_idx -= num_buffers;
> +
> +			if (!dev->features & (1ULL & VIRTIO_F_RING_PACKED))
> +				vq->shadow_used_idx -= num_buffers;
>   			break;
>   		}
>   
> @@ -671,7 +749,8 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>   
>   		if (copy_mbuf_to_desc_mergeable(dev, vq, pkts[pkt_idx],
>   						buf_vec, num_buffers) < 0) {
> -			vq->shadow_used_idx -= num_buffers;
> +			if (!dev->features & (1ULL & VIRTIO_F_RING_PACKED))
> +				vq->shadow_used_idx -= num_buffers;
>   			break;
>   		}
>   
> @@ -680,9 +759,18 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>   
>   	do_data_copy_enqueue(dev, vq);
>   
> -	if (likely(vq->shadow_used_idx)) {
> -		flush_shadow_used_ring(dev, vq);
> -		vhost_vring_call(dev, vq);
> +	if (!(dev->features & (1ULL << VIRTIO_F_RING_PACKED))) {
> +		if (likely(vq->shadow_used_idx)) {
> +			flush_shadow_used_ring(dev, vq);
> +			vhost_vring_call(dev, vq);
> +		}
> +	} else {
> +		rte_smp_wmb();
> +		for (i = avail_head; i < vq->last_avail_idx; i++) {
> +			if ((i & (vq->size - 1)) == 0)
> +				toggle_wrap_counter(vq);
> +			set_desc_used(vq, &descs[i & (vq->size - 1)]);
> +		}
>   	}
>   
>   out:
> @@ -774,7 +862,7 @@ vhost_enqueue_burst_packed(struct virtio_net *dev, uint16_t queue_id,
>   					goto out;
>   				}
>   
> -				idx = (idx+1) & (vq->size - 1);
> +				idx = (idx + 1) & mask;
Are you sure this change is in the right patch?

>   				desc = &descs[idx];
>   				if (unlikely(!desc_is_avail(vq, desc)))
>   					goto out ;
> @@ -840,10 +928,11 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
>   		return 0;
>   	}
>   
> -	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
> -		return vhost_enqueue_burst_packed(dev, queue_id, pkts, count);
> -	else if (dev->features & (1 << VIRTIO_NET_F_MRG_RXBUF))
> +	if ((dev->features & (1 << VIRTIO_NET_F_MRG_RXBUF)) &&
> +	    (dev->features & (1ULL << VIRTIO_F_RING_PACKED)))
>   		return virtio_dev_merge_rx(dev, queue_id, pkts, count);
> +	else if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
> +		return vhost_enqueue_burst_packed(dev, queue_id, pkts, count);
>   	else
>   		return virtio_dev_rx(dev, queue_id, pkts, count);
>   }
> @@ -1266,8 +1355,6 @@ dequeue_desc_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
>   	int wrap_counter = vq->used_wrap_counter;
>   	int rc = 0;
>   
> -	rte_spinlock_lock(&vq->access_lock);
> -
hmm

>   	if (unlikely(vq->enabled == 0))
>   		goto out;
>   
> @@ -1451,6 +1538,9 @@ vhost_dequeue_burst_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
>   	struct vring_desc_packed *desc = vq->desc_packed;
>   	int err;
>   
> +	if (unlikely(rte_spinlock_trylock(&vq->access_lock) == 0))
> +		return 0;
> +
>   	count = RTE_MIN(MAX_PKT_BURST, count);
>   	for (i = 0; i < count; i++) {
>   		idx = vq->last_used_idx & (vq->size - 1);
> @@ -1509,15 +1599,15 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
>   
>   	vq = dev->virtqueue[queue_id];
>   
> +	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
> +		return vhost_dequeue_burst_packed(dev, vq, mbuf_pool, pkts, count);
> +
>   	if (unlikely(rte_spinlock_trylock(&vq->access_lock) == 0))
>   		return 0;
>   
>   	if (unlikely(vq->enabled == 0))
>   		goto out_access_unlock;
>   
> -	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
> -		return vhost_dequeue_burst_packed(dev, vq, mbuf_pool, pkts, count);
> -
>   	vq->batch_copy_nb_elems = 0;
>   
>   	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> 

all above changes, if valid, are for dequeue, so 'tx', so certainly in 
the wrong patch.

Re: [PATCH v3 19/21] vhost: support mergeable rx buffers with packed queues

[Jens Freimann]

On Fri, Apr 06, 2018 at 03:04:47PM +0200, Maxime Coquelin wrote:
>
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>This implements support for mergeable receive buffers in vhost when using
>>packed virtqueues. The difference to split virtqueues is not big, it differs
>>mostly where descriptor flags are touched and virtio features are checked.
>>
>>Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>>---
>>  lib/librte_vhost/vhost.c      |   2 +
>>  lib/librte_vhost/virtio_net.c | 160 +++++++++++++++++++++++++++++++++---------
>>  2 files changed, 127 insertions(+), 35 deletions(-)
>>
>>diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c
>>index eb5a98875..3c633e71e 100644
>>--- a/lib/librte_vhost/vhost.c
>>+++ b/lib/librte_vhost/vhost.c
>>@@ -580,6 +580,8 @@ rte_vhost_enable_guest_notification(int vid, uint16_t queue_id, int enable)
>>  	if (dev == NULL)
>>  		return -1;
>>+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
>>+		return -1;
>
>Is this change really related to mergeable buffers?

no, will fix

>
>>  	if (enable) {
>>  		RTE_LOG(ERR, VHOST_CONFIG,
>>diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
>>index 18e67fdc1..b82c24081 100644
>>--- a/lib/librte_vhost/virtio_net.c
>>+++ b/lib/librte_vhost/virtio_net.c
>>@@ -401,17 +401,53 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
>>  }
>>  static __rte_always_inline int
>>-fill_vec_buf(struct virtio_net *dev, struct vhost_virtqueue *vq,
>>-			 uint32_t avail_idx, uint32_t *vec_idx,
>>-			 struct buf_vector *buf_vec, uint16_t *desc_chain_head,
>>-			 uint16_t *desc_chain_len)
>>+__fill_vec_buf_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
>>+			 struct buf_vector *buf_vec,
>>+			 uint32_t *len, uint32_t *vec_id)
>>+{
>>+	uint16_t idx = vq->last_avail_idx & (vq->size - 1);
>>+	struct vring_desc_packed *descs= vq->desc_packed;
>>+	uint32_t _vec_id = *vec_id;
>>+
>>+	if (vq->desc_packed[idx].flags & VRING_DESC_F_INDIRECT) {
>>+		descs = (struct vring_desc_packed *)(uintptr_t)
>>+			vhost_iova_to_vva(dev, vq, vq->desc_packed[idx].addr,
>>+						vq->desc_packed[idx].len,
>>+						VHOST_ACCESS_RO);
>>+		if (unlikely(!descs))
>>+			return -1;
>>+
>>+		idx = 0;
>>+	}
>>+
>>+	while (1) {
>>+		if (unlikely(_vec_id >= BUF_VECTOR_MAX || idx >= vq->size))
>>+			return -1;
>>+
>>+		*len += descs[idx & (vq->size - 1)].len;
>>+		buf_vec[_vec_id].buf_addr = descs[idx].addr;
>>+		buf_vec[_vec_id].buf_len  = descs[idx].len;
>>+		buf_vec[_vec_id].desc_idx = idx;
>>+		_vec_id++;
>>+
>>+		if ((descs[idx & (vq->size - 1)].flags & VRING_DESC_F_NEXT) == 0)
>>+			break;
>>+
>>+		idx++;
>>+	}
>>+	*vec_id = _vec_id;
>>+
>>+	return 0;
>>+}
>>+
>>+static __rte_always_inline int
>>+__fill_vec_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
>>+			 struct buf_vector *buf_vec,
>>+			 uint32_t *len, uint32_t *vec_id, uint32_t avail_idx)
>>  {
>>  	uint16_t idx = vq->avail->ring[avail_idx & (vq->size - 1)];
>>-	uint32_t vec_id = *vec_idx;
>>-	uint32_t len    = 0;
>>  	struct vring_desc *descs = vq->desc;
>>-
>>-	*desc_chain_head = idx;
>>+	uint32_t _vec_id = *vec_id;
>>  	if (vq->desc[idx].flags & VRING_DESC_F_INDIRECT) {
>>  		descs = (struct vring_desc *)(uintptr_t)
>>@@ -425,20 +461,53 @@ fill_vec_buf(struct virtio_net *dev, struct vhost_virtqueue *vq,
>>  	}
>>  	while (1) {
>>-		if (unlikely(vec_id >= BUF_VECTOR_MAX || idx >= vq->size))
>>+		if (unlikely(_vec_id >= BUF_VECTOR_MAX || idx >= vq->size))
>>  			return -1;
>>-		len += descs[idx].len;
>>-		buf_vec[vec_id].buf_addr = descs[idx].addr;
>>-		buf_vec[vec_id].buf_len  = descs[idx].len;
>>-		buf_vec[vec_id].desc_idx = idx;
>>-		vec_id++;
>>+		*len += descs[idx].len;
>>+		buf_vec[_vec_id].buf_addr = descs[idx].addr;
>>+		buf_vec[_vec_id].buf_len  = descs[idx].len;
>>+		buf_vec[_vec_id].desc_idx = idx;
>>+		_vec_id++;
>>  		if ((descs[idx].flags & VRING_DESC_F_NEXT) == 0)
>>  			break;
>>  		idx = descs[idx].next;
>>  	}
>>+	*vec_id = _vec_id;
>>+
>>+	return 0;
>>+}
>>+
>>+static __rte_always_inline int
>>+fill_vec_buf(struct virtio_net *dev, struct vhost_virtqueue *vq,
>>+			 uint32_t avail_idx, uint32_t *vec_idx,
>>+			 struct buf_vector *buf_vec, uint16_t *desc_chain_head,
>>+			 uint16_t *desc_chain_len)
>>+{
>>+	uint16_t idx;
>>+	uint32_t vec_id = *vec_idx;
>>+	uint32_t len    = 0;
>>+
>>+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
>>+		idx = vq->last_avail_idx & (vq->size -1);
>>+	} else {
>>+		idx = vq->avail->ring[avail_idx & (vq->size - 1)];
>>+	}
>>+
>>+
>>+	*desc_chain_head = idx;
>>+
>>+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
>>+		if (__fill_vec_buf_packed(dev, vq,
>>+				buf_vec, &len, &vec_id))
>>+			return -1;
>>+	} else {
>>+		if (__fill_vec_buf_split(dev, vq,
>>+				buf_vec, &len, &vec_id, avail_idx))
>>+			return -1;
>>+	}
>>  	*desc_chain_len = len;
>>  	*vec_idx = vec_id;
>>@@ -465,14 +534,16 @@ reserve_avail_buf_mergeable(struct virtio_net *dev, struct vhost_virtqueue *vq,
>>  	cur_idx  = vq->last_avail_idx;
>>  	while (size > 0) {
>>-		if (unlikely(cur_idx == avail_head))
>>+		if (unlikely(cur_idx == avail_head) &&
>>+			!(dev->features & (1ull < VIRTIO_F_RING_PACKED)))
>>  			return -1;
>>  		if (unlikely(fill_vec_buf(dev, vq, cur_idx, &vec_idx, buf_vec,
>>  						&head_idx, &len) < 0))
>>  			return -1;
>>  		len = RTE_MIN(len, size);
>>-		update_shadow_used_ring(vq, head_idx, len);
>>+		if (!(dev->features & (1ULL << VIRTIO_F_RING_PACKED)))
>
>This is cosmetics, but I wonder if we could have a helper like:
>
>static inline bool
>vring_is_packed(struct virtio_net *dev) {
>	return dev->features & (1ULL << VIRTIO_F_RING_PACKED);
>}

Yes, I will add one

>
>>+			update_shadow_used_ring(vq, head_idx, len);
>>  		size -= len;
>>  		cur_idx++;
>>@@ -620,6 +691,8 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>>  	uint16_t num_buffers;
>>  	struct buf_vector buf_vec[BUF_VECTOR_MAX];
>>  	uint16_t avail_head;
>>+	uint16_t i;
>>+	struct vring_desc_packed *descs = NULL;
>>  	VHOST_LOG_DEBUG(VHOST_DATA, "(%d) %s\n", dev->vid, __func__);
>>  	if (unlikely(!is_valid_virt_queue_idx(queue_id, 0, dev->nr_vring))) {
>>@@ -634,7 +707,6 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>>  	if (unlikely(vq->enabled == 0))
>>  		goto out_access_unlock;
>>-
>>  	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
>>  		vhost_user_iotlb_rd_lock(vq);
>>@@ -648,10 +720,14 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>>  	vq->batch_copy_nb_elems = 0;
>>-	rte_prefetch0(&vq->avail->ring[vq->last_avail_idx & (vq->size - 1)]);
>>-
>>-	vq->shadow_used_idx = 0;
>>-	avail_head = *((volatile uint16_t *)&vq->avail->idx);
>>+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
>>+		avail_head = vq->last_avail_idx;
>>+		descs = vq->desc_packed;
>>+	} else {
>>+		rte_prefetch0(&vq->avail->ring[vq->last_avail_idx & (vq->size - 1)]);
>>+		avail_head = *((volatile uint16_t *)&vq->avail->idx);
>>+		vq->shadow_used_idx = 0;
>>+	}
>>  	for (pkt_idx = 0; pkt_idx < count; pkt_idx++) {
>>  		uint32_t pkt_len = pkts[pkt_idx]->pkt_len + dev->vhost_hlen;
>>@@ -661,7 +737,9 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>>  			VHOST_LOG_DEBUG(VHOST_DATA,
>>  				"(%d) failed to get enough desc from vring\n",
>>  				dev->vid);
>>-			vq->shadow_used_idx -= num_buffers;
>>+
>>+			if (!dev->features & (1ULL & VIRTIO_F_RING_PACKED))
>>+				vq->shadow_used_idx -= num_buffers;
>>  			break;
>>  		}
>>@@ -671,7 +749,8 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>>  		if (copy_mbuf_to_desc_mergeable(dev, vq, pkts[pkt_idx],
>>  						buf_vec, num_buffers) < 0) {
>>-			vq->shadow_used_idx -= num_buffers;
>>+			if (!dev->features & (1ULL & VIRTIO_F_RING_PACKED))
>>+				vq->shadow_used_idx -= num_buffers;
>>  			break;
>>  		}
>>@@ -680,9 +759,18 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
>>  	do_data_copy_enqueue(dev, vq);
>>-	if (likely(vq->shadow_used_idx)) {
>>-		flush_shadow_used_ring(dev, vq);
>>-		vhost_vring_call(dev, vq);
>>+	if (!(dev->features & (1ULL << VIRTIO_F_RING_PACKED))) {
>>+		if (likely(vq->shadow_used_idx)) {
>>+			flush_shadow_used_ring(dev, vq);
>>+			vhost_vring_call(dev, vq);
>>+		}
>>+	} else {
>>+		rte_smp_wmb();
>>+		for (i = avail_head; i < vq->last_avail_idx; i++) {
>>+			if ((i & (vq->size - 1)) == 0)
>>+				toggle_wrap_counter(vq);
>>+			set_desc_used(vq, &descs[i & (vq->size - 1)]);
>>+		}
>>  	}
>>  out:
>>@@ -774,7 +862,7 @@ vhost_enqueue_burst_packed(struct virtio_net *dev, uint16_t queue_id,
>>  					goto out;
>>  				}
>>-				idx = (idx+1) & (vq->size - 1);
>>+				idx = (idx + 1) & mask;
>Are you sure this change is in the right patch?

Pretty sure it's not :)

>
>>  				desc = &descs[idx];
>>  				if (unlikely(!desc_is_avail(vq, desc)))
>>  					goto out ;
>>@@ -840,10 +928,11 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
>>  		return 0;
>>  	}
>>-	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
>>-		return vhost_enqueue_burst_packed(dev, queue_id, pkts, count);
>>-	else if (dev->features & (1 << VIRTIO_NET_F_MRG_RXBUF))
>>+	if ((dev->features & (1 << VIRTIO_NET_F_MRG_RXBUF)) &&
>>+	    (dev->features & (1ULL << VIRTIO_F_RING_PACKED)))
>>  		return virtio_dev_merge_rx(dev, queue_id, pkts, count);
>>+	else if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
>>+		return vhost_enqueue_burst_packed(dev, queue_id, pkts, count);
>>  	else
>>  		return virtio_dev_rx(dev, queue_id, pkts, count);
>>  }
>>@@ -1266,8 +1355,6 @@ dequeue_desc_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
>>  	int wrap_counter = vq->used_wrap_counter;
>>  	int rc = 0;
>>-	rte_spinlock_lock(&vq->access_lock);
>>-
>hmm
>
>>  	if (unlikely(vq->enabled == 0))
>>  		goto out;
>>@@ -1451,6 +1538,9 @@ vhost_dequeue_burst_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
>>  	struct vring_desc_packed *desc = vq->desc_packed;
>>  	int err;
>>+	if (unlikely(rte_spinlock_trylock(&vq->access_lock) == 0))
>>+		return 0;
>>+
>>  	count = RTE_MIN(MAX_PKT_BURST, count);
>>  	for (i = 0; i < count; i++) {
>>  		idx = vq->last_used_idx & (vq->size - 1);
>>@@ -1509,15 +1599,15 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
>>  	vq = dev->virtqueue[queue_id];
>>+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
>>+		return vhost_dequeue_burst_packed(dev, vq, mbuf_pool, pkts, count);
>>+
>>  	if (unlikely(rte_spinlock_trylock(&vq->access_lock) == 0))
>>  		return 0;
>>  	if (unlikely(vq->enabled == 0))
>>  		goto out_access_unlock;
>>-	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
>>-		return vhost_dequeue_burst_packed(dev, vq, mbuf_pool, pkts, count);
>>-
>>  	vq->batch_copy_nb_elems = 0;
>>  	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
>>
>
>all above changes, if valid, are for dequeue, so 'tx', so certainly in 
>the wrong patch.

yes, I'll move them. Thanks!

regards,
Jens 

[PATCH v3 20/21] net/virtio: add support for event suppression

[Jens Freimann]

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 drivers/net/virtio/virtio_ethdev.c |  2 +-
 drivers/net/virtio/virtio_ethdev.h |  2 +-
 drivers/net/virtio/virtio_rxtx.c   | 15 +++++++-
 drivers/net/virtio/virtqueue.h     | 73 ++++++++++++++++++++++++++++++++++++--
 4 files changed, 86 insertions(+), 6 deletions(-)

diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index a3c3376d7..65a6a9d89 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -727,7 +727,7 @@ virtio_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
 	struct virtnet_rx *rxvq = dev->data->rx_queues[queue_id];
 	struct virtqueue *vq = rxvq->vq;
 
-	virtqueue_enable_intr(vq);
+	virtqueue_enable_intr(vq, 0, 0);
 	return 0;
 }
 
diff --git a/drivers/net/virtio/virtio_ethdev.h b/drivers/net/virtio/virtio_ethdev.h
index 3aeced4bb..19d3f2617 100644
--- a/drivers/net/virtio/virtio_ethdev.h
+++ b/drivers/net/virtio/virtio_ethdev.h
@@ -37,7 +37,7 @@
 	 1u << VIRTIO_RING_F_INDIRECT_DESC |    \
 	 1ULL << VIRTIO_F_VERSION_1       |	\
 	 1ULL << VIRTIO_F_RING_PACKED     |	\
-	 1ULL << VIRTIO_F_IOMMU_PLATFORM)
+	 1ULL << VIRTIO_RING_F_EVENT_IDX)
 
 #define VIRTIO_PMD_SUPPORTED_GUEST_FEATURES	\
 	(VIRTIO_PMD_DEFAULT_GUEST_FEATURES |	\
diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
index a48ca6aaa..ed65434ce 100644
--- a/drivers/net/virtio/virtio_rxtx.c
+++ b/drivers/net/virtio/virtio_rxtx.c
@@ -127,6 +127,10 @@ virtio_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts,
 
 		rte_smp_wmb();
 		_set_desc_avail(&desc[head_idx], wrap_counter);
+		if (unlikely(virtqueue_kick_prepare_packed(vq))) {
+				virtqueue_notify(vq);
+				PMD_RX_LOG(DEBUG, "Notified");
+		}
 	}
 
 	txvq->stats.packets += i;
@@ -998,6 +1002,10 @@ virtio_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
 	}
 
 	rxvq->stats.packets += nb_rx;
+	if (nb_rx > 0 && unlikely(virtqueue_kick_prepare_packed(vq))) {
+		virtqueue_notify(vq);
+		PMD_RX_LOG(DEBUG, "Notified");
+	}
 
 	vq->vq_used_cons_idx = used_idx;
 
@@ -1276,8 +1284,13 @@ virtio_recv_mergeable_pkts(void *rx_queue,
 
 	rxvq->stats.packets += nb_rx;
 
-	if (vtpci_packed_queue(vq->hw))
+	if (vtpci_packed_queue(vq->hw)) {
+		if (unlikely(virtqueue_kick_prepare(vq))) {
+			virtqueue_notify(vq);
+			PMD_RX_LOG(DEBUG, "Notified");
+		}
 		return nb_rx;
+	}
 
 	/* Allocate new mbuf for the used descriptor */
 	error = ENOSPC;
diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
index 7196bd717..6fd3317d2 100644
--- a/drivers/net/virtio/virtqueue.h
+++ b/drivers/net/virtio/virtqueue.h
@@ -176,6 +176,8 @@ struct virtqueue {
 	uint16_t vq_free_cnt;  /**< num of desc available */
 	uint16_t vq_avail_idx; /**< sync until needed */
 	uint16_t vq_free_thresh; /**< free threshold */
+	uint16_t vq_signalled_avail;
+	int vq_signalled_avail_valid;
 
 	void *vq_ring_virt_mem;  /**< linear address of vring*/
 	unsigned int vq_ring_size;
@@ -273,16 +275,34 @@ vring_desc_init(struct vring_desc *dp, uint16_t n)
 static inline void
 virtqueue_disable_intr(struct virtqueue *vq)
 {
-	vq->vq_ring.avail->flags |= VRING_AVAIL_F_NO_INTERRUPT;
+	if (vtpci_packed_queue(vq->hw) && vtpci_with_feature(vq->hw,
+				VIRTIO_RING_F_EVENT_IDX))
+		vq->vq_ring.device_event->desc_event_flags = RING_EVENT_FLAGS_DISABLE;
+	else
+		vq->vq_ring.avail->flags |= VRING_AVAIL_F_NO_INTERRUPT;
 }
 
 /**
  * Tell the backend to interrupt us.
  */
 static inline void
-virtqueue_enable_intr(struct virtqueue *vq)
+virtqueue_enable_intr(struct virtqueue *vq, uint16_t off, uint16_t wrap_counter)
 {
-	vq->vq_ring.avail->flags &= (~VRING_AVAIL_F_NO_INTERRUPT);
+	uint16_t *flags = &vq->vq_ring.device_event->desc_event_flags;
+	uint16_t *event_off_wrap = &vq->vq_ring.device_event->desc_event_off_wrap;
+	if (vtpci_packed_queue(vq->hw)) {
+		*flags = 0;
+		*event_off_wrap = 0;
+		if (*event_off_wrap & RING_EVENT_FLAGS_DESC) {
+			*event_off_wrap = off | 0x7FFF;
+			*event_off_wrap |= wrap_counter << 15;
+			*flags |= RING_EVENT_FLAGS_DESC;
+		} else
+			*event_off_wrap = 0;
+		*flags |= RING_EVENT_FLAGS_ENABLE;
+	} else {
+		vq->vq_ring.avail->flags &= (~VRING_AVAIL_F_NO_INTERRUPT);
+	}
 }
 
 /**
@@ -342,12 +362,59 @@ vq_update_avail_ring(struct virtqueue *vq, uint16_t desc_idx)
 	vq->vq_avail_idx++;
 }
 
+static int vhost_idx_diff(struct virtqueue *vq, uint16_t old, uint16_t new)
+{
+	if (new > old)
+		return new - old;
+	return  (new + vq->vq_nentries - old);
+}
+
+static int vring_packed_need_event(struct virtqueue *vq,
+		uint16_t event_off, uint16_t new,
+		uint16_t old)
+{
+	return (uint16_t)(vhost_idx_diff(vq, new, event_off) - 1) <
+		(uint16_t)vhost_idx_diff(vq, new, old);
+}
+
+
 static inline int
 virtqueue_kick_prepare(struct virtqueue *vq)
 {
 	return !(vq->vq_ring.used->flags & VRING_USED_F_NO_NOTIFY);
 }
 
+static inline int
+virtqueue_kick_prepare_packed(struct virtqueue *vq)
+{
+	uint16_t notify_offset, flags, wrap;
+	uint16_t old, new;
+	int v;
+
+	if (vtpci_packed_queue(vq->hw)) {
+		flags = vq->vq_ring.device_event->desc_event_flags;
+		if (!(flags & RING_EVENT_FLAGS_DESC))
+			return flags & RING_EVENT_FLAGS_ENABLE;
+		virtio_rmb();
+		notify_offset = vq->vq_ring.device_event->desc_event_off_wrap;
+		wrap = notify_offset & 0x1;
+		notify_offset >>= 1;
+
+		old = vq->vq_signalled_avail;
+		v = vq->vq_signalled_avail_valid;
+		new = vq->vq_signalled_avail = vq->vq_avail_idx;
+		vq->vq_signalled_avail_valid = 1;
+
+		if (unlikely(!v))
+			return 0;
+
+		return (vring_packed_need_event(vq, new, old, notify_offset) &&
+			wrap == vq->vq_ring.avail_wrap_counter);
+	} else {
+		return !(vq->vq_ring.used->flags & VRING_USED_F_NO_NOTIFY);
+	}
+}
+
 static inline void
 virtqueue_notify(struct virtqueue *vq)
 {
-- 
2.14.3

Re: [PATCH v3 20/21] net/virtio: add support for event suppression

[Maxime Coquelin]

On 04/05/2018 12:10 PM, Jens Freimann wrote:
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>   drivers/net/virtio/virtio_ethdev.c |  2 +-
>   drivers/net/virtio/virtio_ethdev.h |  2 +-
>   drivers/net/virtio/virtio_rxtx.c   | 15 +++++++-
>   drivers/net/virtio/virtqueue.h     | 73 ++++++++++++++++++++++++++++++++++++--
>   4 files changed, 86 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
> index a3c3376d7..65a6a9d89 100644
> --- a/drivers/net/virtio/virtio_ethdev.c
> +++ b/drivers/net/virtio/virtio_ethdev.c
> @@ -727,7 +727,7 @@ virtio_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
>   	struct virtnet_rx *rxvq = dev->data->rx_queues[queue_id];
>   	struct virtqueue *vq = rxvq->vq;
>   
> -	virtqueue_enable_intr(vq);
> +	virtqueue_enable_intr(vq, 0, 0);
>   	return 0;
>   }
>   
> diff --git a/drivers/net/virtio/virtio_ethdev.h b/drivers/net/virtio/virtio_ethdev.h
> index 3aeced4bb..19d3f2617 100644
> --- a/drivers/net/virtio/virtio_ethdev.h
> +++ b/drivers/net/virtio/virtio_ethdev.h
> @@ -37,7 +37,7 @@
>   	 1u << VIRTIO_RING_F_INDIRECT_DESC |    \
>   	 1ULL << VIRTIO_F_VERSION_1       |	\
>   	 1ULL << VIRTIO_F_RING_PACKED     |	\
> -	 1ULL << VIRTIO_F_IOMMU_PLATFORM)
> +	 1ULL << VIRTIO_RING_F_EVENT_IDX)
>   
>   #define VIRTIO_PMD_SUPPORTED_GUEST_FEATURES	\
>   	(VIRTIO_PMD_DEFAULT_GUEST_FEATURES |	\
> diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
> index a48ca6aaa..ed65434ce 100644
> --- a/drivers/net/virtio/virtio_rxtx.c
> +++ b/drivers/net/virtio/virtio_rxtx.c
> @@ -127,6 +127,10 @@ virtio_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts,
>   
>   		rte_smp_wmb();
>   		_set_desc_avail(&desc[head_idx], wrap_counter);
> +		if (unlikely(virtqueue_kick_prepare_packed(vq))) {
> +				virtqueue_notify(vq);
> +				PMD_RX_LOG(DEBUG, "Notified");
> +		}
>   	}
>   
>   	txvq->stats.packets += i;
> @@ -998,6 +1002,10 @@ virtio_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
>   	}
>   
>   	rxvq->stats.packets += nb_rx;
> +	if (nb_rx > 0 && unlikely(virtqueue_kick_prepare_packed(vq))) {
> +		virtqueue_notify(vq);
> +		PMD_RX_LOG(DEBUG, "Notified");
> +	}
>   
>   	vq->vq_used_cons_idx = used_idx;
>   
> @@ -1276,8 +1284,13 @@ virtio_recv_mergeable_pkts(void *rx_queue,
>   
>   	rxvq->stats.packets += nb_rx;
>   
> -	if (vtpci_packed_queue(vq->hw))
> +	if (vtpci_packed_queue(vq->hw)) {
> +		if (unlikely(virtqueue_kick_prepare(vq))) {
> +			virtqueue_notify(vq);
> +			PMD_RX_LOG(DEBUG, "Notified");
> +		}
>   		return nb_rx;
> +	}
>   
>   	/* Allocate new mbuf for the used descriptor */
>   	error = ENOSPC;
> diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
> index 7196bd717..6fd3317d2 100644
> --- a/drivers/net/virtio/virtqueue.h
> +++ b/drivers/net/virtio/virtqueue.h
> @@ -176,6 +176,8 @@ struct virtqueue {
>   	uint16_t vq_free_cnt;  /**< num of desc available */
>   	uint16_t vq_avail_idx; /**< sync until needed */
>   	uint16_t vq_free_thresh; /**< free threshold */
> +	uint16_t vq_signalled_avail;
> +	int vq_signalled_avail_valid;
>   
>   	void *vq_ring_virt_mem;  /**< linear address of vring*/
>   	unsigned int vq_ring_size;
> @@ -273,16 +275,34 @@ vring_desc_init(struct vring_desc *dp, uint16_t n)
>   static inline void
>   virtqueue_disable_intr(struct virtqueue *vq)
>   {
> -	vq->vq_ring.avail->flags |= VRING_AVAIL_F_NO_INTERRUPT;
> +	if (vtpci_packed_queue(vq->hw) && vtpci_with_feature(vq->hw,
> +				VIRTIO_RING_F_EVENT_IDX))
> +		vq->vq_ring.device_event->desc_event_flags = RING_EVENT_FLAGS_DISABLE;
> +	else
> +		vq->vq_ring.avail->flags |= VRING_AVAIL_F_NO_INTERRUPT;
>   }
>   
>   /**
>    * Tell the backend to interrupt us.
>    */
>   static inline void
> -virtqueue_enable_intr(struct virtqueue *vq)
> +virtqueue_enable_intr(struct virtqueue *vq, uint16_t off, uint16_t wrap_counter)
>   {
> -	vq->vq_ring.avail->flags &= (~VRING_AVAIL_F_NO_INTERRUPT);
> +	uint16_t *flags = &vq->vq_ring.device_event->desc_event_flags;
> +	uint16_t *event_off_wrap = &vq->vq_ring.device_event->desc_event_off_wrap;
> +	if (vtpci_packed_queue(vq->hw)) {
> +		*flags = 0;
> +		*event_off_wrap = 0;
> +		if (*event_off_wrap & RING_EVENT_FLAGS_DESC) {
> +			*event_off_wrap = off | 0x7FFF;
> +			*event_off_wrap |= wrap_counter << 15;
> +			*flags |= RING_EVENT_FLAGS_DESC;
> +		} else
> +			*event_off_wrap = 0;
> +		*flags |= RING_EVENT_FLAGS_ENABLE;
> +	} else {
> +		vq->vq_ring.avail->flags &= (~VRING_AVAIL_F_NO_INTERRUPT);
> +	}
>   }
>   
>   /**
> @@ -342,12 +362,59 @@ vq_update_avail_ring(struct virtqueue *vq, uint16_t desc_idx)
>   	vq->vq_avail_idx++;
>   }
>   
> +static int vhost_idx_diff(struct virtqueue *vq, uint16_t old, uint16_t new)
> +{
> +	if (new > old)
> +		return new - old;
> +	return  (new + vq->vq_nentries - old);
> +}
> +
> +static int vring_packed_need_event(struct virtqueue *vq,
> +		uint16_t event_off, uint16_t new,
> +		uint16_t old)
> +{
> +	return (uint16_t)(vhost_idx_diff(vq, new, event_off) - 1) <
> +		(uint16_t)vhost_idx_diff(vq, new, old);
> +}
> +
> +
>   static inline int
>   virtqueue_kick_prepare(struct virtqueue *vq)
>   {
>   	return !(vq->vq_ring.used->flags & VRING_USED_F_NO_NOTIFY);
>   }
>   
> +static inline int
> +virtqueue_kick_prepare_packed(struct virtqueue *vq)
> +{
> +	uint16_t notify_offset, flags, wrap;
> +	uint16_t old, new;
> +	int v;
> +
> +	if (vtpci_packed_queue(vq->hw)) {
> +		flags = vq->vq_ring.device_event->desc_event_flags;
> +		if (!(flags & RING_EVENT_FLAGS_DESC))
> +			return flags & RING_EVENT_FLAGS_ENABLE;
> +		virtio_rmb();
> +		notify_offset = vq->vq_ring.device_event->desc_event_off_wrap;
> +		wrap = notify_offset & 0x1;
> +		notify_offset >>= 1;
> +
> +		old = vq->vq_signalled_avail;
> +		v = vq->vq_signalled_avail_valid;
> +		new = vq->vq_signalled_avail = vq->vq_avail_idx;
> +		vq->vq_signalled_avail_valid = 1;
> +
> +		if (unlikely(!v))
> +			return 0;
> +
> +		return (vring_packed_need_event(vq, new, old, notify_offset) &&
> +			wrap == vq->vq_ring.avail_wrap_counter);
> +	} else {
> +		return !(vq->vq_ring.used->flags & VRING_USED_F_NO_NOTIFY);

Maybe I didn't gezt it, but it looks confusing that the function name
contains packed, and that it seems it handles non-packed queue case.

> +	}
> +}
> +
>   static inline void
>   virtqueue_notify(struct virtqueue *vq)
>   {
> 

Re: [PATCH v3 20/21] net/virtio: add support for event suppression

[Jens Freimann]

On Fri, Apr 06, 2018 at 03:50:28PM +0200, Maxime Coquelin wrote:
>
>
>On 04/05/2018 12:10 PM, Jens Freimann wrote:
>>Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>>---
>>  drivers/net/virtio/virtio_ethdev.c |  2 +-
>>  drivers/net/virtio/virtio_ethdev.h |  2 +-
>>  drivers/net/virtio/virtio_rxtx.c   | 15 +++++++-
>>  drivers/net/virtio/virtqueue.h     | 73 ++++++++++++++++++++++++++++++++++++--
>>  4 files changed, 86 insertions(+), 6 deletions(-)
>>
>>diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
>>index a3c3376d7..65a6a9d89 100644
>>--- a/drivers/net/virtio/virtio_ethdev.c
>>+++ b/drivers/net/virtio/virtio_ethdev.c
>>@@ -727,7 +727,7 @@ virtio_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
>>  	struct virtnet_rx *rxvq = dev->data->rx_queues[queue_id];
>>  	struct virtqueue *vq = rxvq->vq;
>>-	virtqueue_enable_intr(vq);
>>+	virtqueue_enable_intr(vq, 0, 0);
>>  	return 0;
>>  }
>>diff --git a/drivers/net/virtio/virtio_ethdev.h b/drivers/net/virtio/virtio_ethdev.h
>>index 3aeced4bb..19d3f2617 100644
>>--- a/drivers/net/virtio/virtio_ethdev.h
>>+++ b/drivers/net/virtio/virtio_ethdev.h
>>@@ -37,7 +37,7 @@
>>  	 1u << VIRTIO_RING_F_INDIRECT_DESC |    \
>>  	 1ULL << VIRTIO_F_VERSION_1       |	\
>>  	 1ULL << VIRTIO_F_RING_PACKED     |	\
>>-	 1ULL << VIRTIO_F_IOMMU_PLATFORM)
>>+	 1ULL << VIRTIO_RING_F_EVENT_IDX)
>>  #define VIRTIO_PMD_SUPPORTED_GUEST_FEATURES	\
>>  	(VIRTIO_PMD_DEFAULT_GUEST_FEATURES |	\
>>diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
>>index a48ca6aaa..ed65434ce 100644
>>--- a/drivers/net/virtio/virtio_rxtx.c
>>+++ b/drivers/net/virtio/virtio_rxtx.c
>>@@ -127,6 +127,10 @@ virtio_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts,
>>  		rte_smp_wmb();
>>  		_set_desc_avail(&desc[head_idx], wrap_counter);
>>+		if (unlikely(virtqueue_kick_prepare_packed(vq))) {
>>+				virtqueue_notify(vq);
>>+				PMD_RX_LOG(DEBUG, "Notified");
>>+		}
>>  	}
>>  	txvq->stats.packets += i;
>>@@ -998,6 +1002,10 @@ virtio_recv_pkts_packed(void *rx_queue, struct rte_mbuf **rx_pkts,
>>  	}
>>  	rxvq->stats.packets += nb_rx;
>>+	if (nb_rx > 0 && unlikely(virtqueue_kick_prepare_packed(vq))) {
>>+		virtqueue_notify(vq);
>>+		PMD_RX_LOG(DEBUG, "Notified");
>>+	}
>>  	vq->vq_used_cons_idx = used_idx;
>>@@ -1276,8 +1284,13 @@ virtio_recv_mergeable_pkts(void *rx_queue,
>>  	rxvq->stats.packets += nb_rx;
>>-	if (vtpci_packed_queue(vq->hw))
>>+	if (vtpci_packed_queue(vq->hw)) {
>>+		if (unlikely(virtqueue_kick_prepare(vq))) {
>>+			virtqueue_notify(vq);
>>+			PMD_RX_LOG(DEBUG, "Notified");
>>+		}
>>  		return nb_rx;
>>+	}
>>  	/* Allocate new mbuf for the used descriptor */
>>  	error = ENOSPC;
>>diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h
>>index 7196bd717..6fd3317d2 100644
>>--- a/drivers/net/virtio/virtqueue.h
>>+++ b/drivers/net/virtio/virtqueue.h
>>@@ -176,6 +176,8 @@ struct virtqueue {
>>  	uint16_t vq_free_cnt;  /**< num of desc available */
>>  	uint16_t vq_avail_idx; /**< sync until needed */
>>  	uint16_t vq_free_thresh; /**< free threshold */
>>+	uint16_t vq_signalled_avail;
>>+	int vq_signalled_avail_valid;
>>  	void *vq_ring_virt_mem;  /**< linear address of vring*/
>>  	unsigned int vq_ring_size;
>>@@ -273,16 +275,34 @@ vring_desc_init(struct vring_desc *dp, uint16_t n)
>>  static inline void
>>  virtqueue_disable_intr(struct virtqueue *vq)
>>  {
>>-	vq->vq_ring.avail->flags |= VRING_AVAIL_F_NO_INTERRUPT;
>>+	if (vtpci_packed_queue(vq->hw) && vtpci_with_feature(vq->hw,
>>+				VIRTIO_RING_F_EVENT_IDX))
>>+		vq->vq_ring.device_event->desc_event_flags = RING_EVENT_FLAGS_DISABLE;
>>+	else
>>+		vq->vq_ring.avail->flags |= VRING_AVAIL_F_NO_INTERRUPT;
>>  }
>>  /**
>>   * Tell the backend to interrupt us.
>>   */
>>  static inline void
>>-virtqueue_enable_intr(struct virtqueue *vq)
>>+virtqueue_enable_intr(struct virtqueue *vq, uint16_t off, uint16_t wrap_counter)
>>  {
>>-	vq->vq_ring.avail->flags &= (~VRING_AVAIL_F_NO_INTERRUPT);
>>+	uint16_t *flags = &vq->vq_ring.device_event->desc_event_flags;
>>+	uint16_t *event_off_wrap = &vq->vq_ring.device_event->desc_event_off_wrap;
>>+	if (vtpci_packed_queue(vq->hw)) {
>>+		*flags = 0;
>>+		*event_off_wrap = 0;
>>+		if (*event_off_wrap & RING_EVENT_FLAGS_DESC) {
>>+			*event_off_wrap = off | 0x7FFF;
>>+			*event_off_wrap |= wrap_counter << 15;
>>+			*flags |= RING_EVENT_FLAGS_DESC;
>>+		} else
>>+			*event_off_wrap = 0;
>>+		*flags |= RING_EVENT_FLAGS_ENABLE;
>>+	} else {
>>+		vq->vq_ring.avail->flags &= (~VRING_AVAIL_F_NO_INTERRUPT);
>>+	}
>>  }
>>  /**
>>@@ -342,12 +362,59 @@ vq_update_avail_ring(struct virtqueue *vq, uint16_t desc_idx)
>>  	vq->vq_avail_idx++;
>>  }
>>+static int vhost_idx_diff(struct virtqueue *vq, uint16_t old, uint16_t new)
>>+{
>>+	if (new > old)
>>+		return new - old;
>>+	return  (new + vq->vq_nentries - old);
>>+}
>>+
>>+static int vring_packed_need_event(struct virtqueue *vq,
>>+		uint16_t event_off, uint16_t new,
>>+		uint16_t old)
>>+{
>>+	return (uint16_t)(vhost_idx_diff(vq, new, event_off) - 1) <
>>+		(uint16_t)vhost_idx_diff(vq, new, old);
>>+}
>>+
>>+
>>  static inline int
>>  virtqueue_kick_prepare(struct virtqueue *vq)
>>  {
>>  	return !(vq->vq_ring.used->flags & VRING_USED_F_NO_NOTIFY);
>>  }
>>+static inline int
>>+virtqueue_kick_prepare_packed(struct virtqueue *vq)
>>+{
>>+	uint16_t notify_offset, flags, wrap;
>>+	uint16_t old, new;
>>+	int v;
>>+
>>+	if (vtpci_packed_queue(vq->hw)) {
>>+		flags = vq->vq_ring.device_event->desc_event_flags;
>>+		if (!(flags & RING_EVENT_FLAGS_DESC))
>>+			return flags & RING_EVENT_FLAGS_ENABLE;
>>+		virtio_rmb();
>>+		notify_offset = vq->vq_ring.device_event->desc_event_off_wrap;
>>+		wrap = notify_offset & 0x1;
>>+		notify_offset >>= 1;
>>+
>>+		old = vq->vq_signalled_avail;
>>+		v = vq->vq_signalled_avail_valid;
>>+		new = vq->vq_signalled_avail = vq->vq_avail_idx;
>>+		vq->vq_signalled_avail_valid = 1;
>>+
>>+		if (unlikely(!v))
>>+			return 0;
>>+
>>+		return (vring_packed_need_event(vq, new, old, notify_offset) &&
>>+			wrap == vq->vq_ring.avail_wrap_counter);
>>+	} else {
>>+		return !(vq->vq_ring.used->flags & VRING_USED_F_NO_NOTIFY);
>
>Maybe I didn't gezt it, but it looks confusing that the function name
>contains packed, and that it seems it handles non-packed queue case.

No, the else part is not needed here. I'll remove it and make sure to
call the right function.

Thanks!

regards,
Jens 
>
>>+	}
>>+}
>>+
>>  static inline void
>>  virtqueue_notify(struct virtqueue *vq)
>>  {
>>

Re: [PATCH v3 20/21] net/virtio: add support for event suppression

[Tiwei Bie]

On Thu, Apr 05, 2018 at 12:10:30PM +0200, Jens Freimann wrote:
> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
> ---
>  drivers/net/virtio/virtio_ethdev.c |  2 +-
>  drivers/net/virtio/virtio_ethdev.h |  2 +-
>  drivers/net/virtio/virtio_rxtx.c   | 15 +++++++-
>  drivers/net/virtio/virtqueue.h     | 73 ++++++++++++++++++++++++++++++++++++--
>  4 files changed, 86 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
> index a3c3376d7..65a6a9d89 100644
> --- a/drivers/net/virtio/virtio_ethdev.c
> +++ b/drivers/net/virtio/virtio_ethdev.c
> @@ -727,7 +727,7 @@ virtio_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
>  	struct virtnet_rx *rxvq = dev->data->rx_queues[queue_id];
>  	struct virtqueue *vq = rxvq->vq;
>  
> -	virtqueue_enable_intr(vq);
> +	virtqueue_enable_intr(vq, 0, 0);
>  	return 0;
>  }
>  
> diff --git a/drivers/net/virtio/virtio_ethdev.h b/drivers/net/virtio/virtio_ethdev.h
> index 3aeced4bb..19d3f2617 100644
> --- a/drivers/net/virtio/virtio_ethdev.h
> +++ b/drivers/net/virtio/virtio_ethdev.h
> @@ -37,7 +37,7 @@
>  	 1u << VIRTIO_RING_F_INDIRECT_DESC |    \
>  	 1ULL << VIRTIO_F_VERSION_1       |	\
>  	 1ULL << VIRTIO_F_RING_PACKED     |	\
> -	 1ULL << VIRTIO_F_IOMMU_PLATFORM)

Why remove this feature bit?


> +	 1ULL << VIRTIO_RING_F_EVENT_IDX)

Supporting event suppression doesn't means supporting
F_EVENT_IDX. Event suppression is mandatory and not
a negotiable feature. F_EVENT_IDX is just an optional
enhancement for event suppression.


>  
>  #define VIRTIO_PMD_SUPPORTED_GUEST_FEATURES	\
>  	(VIRTIO_PMD_DEFAULT_GUEST_FEATURES |	\
> diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
> index a48ca6aaa..ed65434ce 100644
> --- a/drivers/net/virtio/virtio_rxtx.c
> +++ b/drivers/net/virtio/virtio_rxtx.c
> @@ -127,6 +127,10 @@ virtio_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts,
>  
>  		rte_smp_wmb();
>  		_set_desc_avail(&desc[head_idx], wrap_counter);
> +		if (unlikely(virtqueue_kick_prepare_packed(vq))) {
> +				virtqueue_notify(vq);
> +				PMD_RX_LOG(DEBUG, "Notified");

The indent isn't right.


> +		}
>  	}
[...]

Re: [PATCH v3 20/21] net/virtio: add support for event suppression

[Jens Freimann]

On Sun, Apr 08, 2018 at 02:07:50PM +0800, Tiwei Bie wrote:
>On Thu, Apr 05, 2018 at 12:10:30PM +0200, Jens Freimann wrote:
>> Signed-off-by: Jens Freimann <jfreimann@redhat.com>
>> ---
>>  drivers/net/virtio/virtio_ethdev.c |  2 +-
>>  drivers/net/virtio/virtio_ethdev.h |  2 +-
>>  drivers/net/virtio/virtio_rxtx.c   | 15 +++++++-
>>  drivers/net/virtio/virtqueue.h     | 73 ++++++++++++++++++++++++++++++++++++--
>>  4 files changed, 86 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
>> index a3c3376d7..65a6a9d89 100644
>> --- a/drivers/net/virtio/virtio_ethdev.c
>> +++ b/drivers/net/virtio/virtio_ethdev.c
>> @@ -727,7 +727,7 @@ virtio_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t queue_id)
>>  	struct virtnet_rx *rxvq = dev->data->rx_queues[queue_id];
>>  	struct virtqueue *vq = rxvq->vq;
>>
>> -	virtqueue_enable_intr(vq);
>> +	virtqueue_enable_intr(vq, 0, 0);
>>  	return 0;
>>  }
>>
>> diff --git a/drivers/net/virtio/virtio_ethdev.h b/drivers/net/virtio/virtio_ethdev.h
>> index 3aeced4bb..19d3f2617 100644
>> --- a/drivers/net/virtio/virtio_ethdev.h
>> +++ b/drivers/net/virtio/virtio_ethdev.h
>> @@ -37,7 +37,7 @@
>>  	 1u << VIRTIO_RING_F_INDIRECT_DESC |    \
>>  	 1ULL << VIRTIO_F_VERSION_1       |	\
>>  	 1ULL << VIRTIO_F_RING_PACKED     |	\
>> -	 1ULL << VIRTIO_F_IOMMU_PLATFORM)
>
>Why remove this feature bit?

my mistake. will fix it. 
>
>
>> +	 1ULL << VIRTIO_RING_F_EVENT_IDX)
>
>Supporting event suppression doesn't means supporting
>F_EVENT_IDX. Event suppression is mandatory and not
>a negotiable feature. F_EVENT_IDX is just an optional
>enhancement for event suppression.

I understand, will fix that too. 
>
>
>>
>>  #define VIRTIO_PMD_SUPPORTED_GUEST_FEATURES	\
>>  	(VIRTIO_PMD_DEFAULT_GUEST_FEATURES |	\
>> diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
>> index a48ca6aaa..ed65434ce 100644
>> --- a/drivers/net/virtio/virtio_rxtx.c
>> +++ b/drivers/net/virtio/virtio_rxtx.c
>> @@ -127,6 +127,10 @@ virtio_xmit_pkts_packed(void *tx_queue, struct rte_mbuf **tx_pkts,
>>
>>  		rte_smp_wmb();
>>  		_set_desc_avail(&desc[head_idx], wrap_counter);
>> +		if (unlikely(virtqueue_kick_prepare_packed(vq))) {
>> +				virtqueue_notify(vq);
>> +				PMD_RX_LOG(DEBUG, "Notified");
>
>The indent isn't right.

will fix. Thanks!

regards,
Jens 

[PATCH v3 21/21] vhost: add event suppression for packed queues

[Jens Freimann]

Signed-off-by: Jens Freimann <jfreimann@redhat.com>
---
 lib/librte_vhost/vhost.c      | 17 +++++++++---
 lib/librte_vhost/vhost.h      | 62 ++++++++++++++++++++++++++++++++++++-------
 lib/librte_vhost/vhost_user.c | 19 +++++++++++++
 lib/librte_vhost/virtio_net.c |  1 +
 4 files changed, 86 insertions(+), 13 deletions(-)

diff --git a/lib/librte_vhost/vhost.c b/lib/librte_vhost/vhost.c
index 3c633e71e..b0fc1c1ac 100644
--- a/lib/librte_vhost/vhost.c
+++ b/lib/librte_vhost/vhost.c
@@ -577,11 +577,21 @@ int
 rte_vhost_enable_guest_notification(int vid, uint16_t queue_id, int enable)
 {
 	struct virtio_net *dev = get_device(vid);
+	struct vhost_virtqueue *vq;
 
 	if (dev == NULL)
 		return -1;
-	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED))
-		return -1;
+
+	vq = dev->virtqueue[queue_id];
+	if (!vq->enabled)
+		return 0;
+
+	if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
+		if (!enable) {
+			vq->driver_event->desc_event_flags |= RING_EVENT_FLAGS_DISABLE;
+		} else
+			vq->driver_event->desc_event_flags |= RING_EVENT_FLAGS_ENABLE;
+	}
 
 	if (enable) {
 		RTE_LOG(ERR, VHOST_CONFIG,
@@ -589,7 +599,8 @@ rte_vhost_enable_guest_notification(int vid, uint16_t queue_id, int enable)
 		return -1;
 	}
 
-	dev->virtqueue[queue_id]->used->flags = VRING_USED_F_NO_NOTIFY;
+	if (!(dev->features & (1ULL << VIRTIO_F_RING_PACKED)))
+		dev->virtqueue[queue_id]->used->flags = VRING_USED_F_NO_NOTIFY;
 	return 0;
 }
 
diff --git a/lib/librte_vhost/vhost.h b/lib/librte_vhost/vhost.h
index a0b61b7b0..77eeed8c9 100644
--- a/lib/librte_vhost/vhost.h
+++ b/lib/librte_vhost/vhost.h
@@ -69,14 +69,31 @@ struct batch_copy_elem {
 	uint64_t log_addr;
 };
 
+#define RING_EVENT_FLAGS_ENABLE 0x0
+#define RING_EVENT_FLAGS_DISABLE 0x1
+#define RING_EVENT_FLAGS_DESC 0x2
+#define RING_EVENT_FLAGS_MASK 0xFFFC
+#define RING_EVENT_WRAP_MASK 0x8000
+#define RING_EVENT_OFF_MASK 0x7FFF
+struct vring_packed_desc_event {
+        uint16_t desc_event_off_wrap;
+        uint16_t desc_event_flags;
+};
+
 /**
  * Structure contains variables relevant to RX/TX virtqueues.
  */
 struct vhost_virtqueue {
 	struct vring_desc	*desc;
 	struct vring_desc_packed   *desc_packed;
-	struct vring_avail	*avail;
-	struct vring_used	*used;
+	union {
+		struct vring_avail	*avail;
+		struct vring_packed_desc_event *driver_event;
+	};
+	union {
+		struct vring_used	*used;
+		struct vring_packed_desc_event *device_event;
+	};
 	uint32_t		size;
 
 	uint16_t		last_avail_idx;
@@ -211,7 +228,6 @@ struct vhost_msg {
 				(1ULL << VIRTIO_NET_F_MTU) | \
 				(1ULL << VIRTIO_F_IOMMU_PLATFORM))
 
-
 struct guest_page {
 	uint64_t guest_phys_addr;
 	uint64_t host_phys_addr;
@@ -427,6 +443,11 @@ vhost_need_event(uint16_t event_idx, uint16_t new_idx, uint16_t old)
 static __rte_always_inline void
 vhost_vring_call(struct virtio_net *dev, struct vhost_virtqueue *vq)
 {
+	uint16_t off_wrap, wrap = 0;
+	uint16_t event_flags;
+	uint16_t event_idx = 0;
+	int do_kick = 0;
+
 	/* Flush used->idx update before we read avail->flags. */
 	rte_mb();
 
@@ -434,22 +455,43 @@ vhost_vring_call(struct virtio_net *dev, struct vhost_virtqueue *vq)
 	if (dev->features & (1ULL << VIRTIO_RING_F_EVENT_IDX)) {
 		uint16_t old = vq->signalled_used;
 		uint16_t new = vq->last_used_idx;
+		if (dev->features & (1ULL << VIRTIO_F_RING_PACKED)) {
+			event_flags = vq->driver_event->desc_event_flags &
+					RING_EVENT_FLAGS_MASK;
+			if (!(event_flags & RING_EVENT_FLAGS_DESC))
+				do_kick = event_flags & RING_EVENT_FLAGS_ENABLE ? 1 : 0;
+			else {
+				off_wrap = vq->driver_event->desc_event_off_wrap;
+				wrap = off_wrap & RING_EVENT_WRAP_MASK;
+				event_idx = off_wrap & RING_EVENT_OFF_MASK;
+			}
+			if (vhost_need_event(event_idx, new, old)
+				&& (vq->callfd >= 0) && (wrap == vq->used_wrap_counter)) {
+				vq->signalled_used = vq->last_used_idx;
+				do_kick = 1;
+			}
+		} else {
+			event_idx = vhost_used_event(vq);
+			if (vhost_need_event(event_idx, new, old)
+				&& (vq->callfd >= 0)) {
+				vq->signalled_used = vq->last_used_idx;
+				do_kick = 1;
+			}
+		}
 
 		VHOST_LOG_DEBUG(VHOST_DATA, "%s: used_event_idx=%d, old=%d, new=%d\n",
 			__func__,
-			vhost_used_event(vq),
+			event_idx,
 			old, new);
-		if (vhost_need_event(vhost_used_event(vq), new, old)
-			&& (vq->callfd >= 0)) {
-			vq->signalled_used = vq->last_used_idx;
-			eventfd_write(vq->callfd, (eventfd_t) 1);
-		}
 	} else {
 		/* Kick the guest if necessary. */
 		if (!(vq->avail->flags & VRING_AVAIL_F_NO_INTERRUPT)
 				&& (vq->callfd >= 0))
-			eventfd_write(vq->callfd, (eventfd_t)1);
+			do_kick = 1;
 	}
+
+	if (do_kick)
+			eventfd_write(vq->callfd, (eventfd_t)1);
 }
 
 #endif /* _VHOST_NET_CDEV_H_ */
diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c
index 183893e46..6b8fd1474 100644
--- a/lib/librte_vhost/vhost_user.c
+++ b/lib/librte_vhost/vhost_user.c
@@ -39,6 +39,7 @@
 #include "iotlb.h"
 #include "vhost.h"
 #include "vhost_user.h"
+#include "virtio-1.1.h"
 
 #define VIRTIO_MIN_MTU 68
 #define VIRTIO_MAX_MTU 65535
@@ -476,6 +477,24 @@ translate_ring_addresses(struct virtio_net *dev, int vq_index)
 		vq->avail = NULL;
 		vq->used = NULL;
 		vq->log_guest_addr = 0;
+		vq->driver_event = (struct vring_packed_desc_event *)(uintptr_t)ring_addr_to_vva(dev,
+				vq, addr->avail_user_addr, sizeof(struct vring_packed_desc_event));
+		if (vq->driver_event == 0) {
+			RTE_LOG(DEBUG, VHOST_CONFIG,
+				"(%d) failed to find driver area address.\n",
+				dev->vid);
+			return dev;
+		}
+
+		vq->device_event = (struct vring_packed_desc_event *)(uintptr_t)ring_addr_to_vva(dev,
+				vq, addr->used_user_addr, sizeof(struct vring_packed_desc_event));
+		if (vq->device_event == 0) {
+			RTE_LOG(DEBUG, VHOST_CONFIG,
+				"(%d) failed to find device area address.\n",
+				dev->vid);
+			return dev;
+		}
+
 
 		if (vq->last_used_idx != 0) {
 			RTE_LOG(WARNING, VHOST_CONFIG,
diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c
index b82c24081..d3c09c8ba 100644
--- a/lib/librte_vhost/virtio_net.c
+++ b/lib/librte_vhost/virtio_net.c
@@ -770,6 +770,7 @@ virtio_dev_merge_rx(struct virtio_net *dev, uint16_t queue_id,
 			if ((i & (vq->size - 1)) == 0)
 				toggle_wrap_counter(vq);
 			set_desc_used(vq, &descs[i & (vq->size - 1)]);
+			vhost_vring_call(dev, vq);
 		}
 	}
 
-- 
2.14.3