All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH net 0/2] add checking sq is full inside xdp xmit
@ 2023-03-06  4:15 ` Xuan Zhuo
  0 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-06  4:15 UTC (permalink / raw)
  To: netdev
  Cc: Jesper Dangaard Brouer, Daniel Borkmann, Michael S. Tsirkin,
	John Fastabend, Alexei Starovoitov, virtualization, Eric Dumazet,
	Jakub Kicinski, bpf, Paolo Abeni, David S. Miller

If the queue of xdp xmit is not an independent queue, then when the xdp
xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
the following error.

net ens4: Unexpected TXQ (0) queue failure: -28

This patch adds a check whether sq is full in XDP Xmit.

Thanks.

Xuan Zhuo (2):
  virtio_net: separate the logic of checking whether sq is full
  virtio_net: add checking sq is full inside xdp xmit

 drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++----------------
 1 file changed, 47 insertions(+), 31 deletions(-)

--
2.32.0.3.g01195cf9f

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCH net 0/2] add checking sq is full inside xdp xmit
@ 2023-03-06  4:15 ` Xuan Zhuo
  0 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-06  4:15 UTC (permalink / raw)
  To: netdev
  Cc: Michael S. Tsirkin, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf

If the queue of xdp xmit is not an independent queue, then when the xdp
xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
the following error.

net ens4: Unexpected TXQ (0) queue failure: -28

This patch adds a check whether sq is full in XDP Xmit.

Thanks.

Xuan Zhuo (2):
  virtio_net: separate the logic of checking whether sq is full
  virtio_net: add checking sq is full inside xdp xmit

 drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++----------------
 1 file changed, 47 insertions(+), 31 deletions(-)

--
2.32.0.3.g01195cf9f


^ permalink raw reply	[flat|nested] 24+ messages in thread

* [PATCH net 1/2] virtio_net: separate the logic of checking whether sq is full
  2023-03-06  4:15 ` Xuan Zhuo
@ 2023-03-06  4:15   ` Xuan Zhuo
  -1 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-06  4:15 UTC (permalink / raw)
  To: netdev
  Cc: Jesper Dangaard Brouer, Daniel Borkmann, Michael S. Tsirkin,
	John Fastabend, Alexei Starovoitov, virtualization, Eric Dumazet,
	Jakub Kicinski, bpf, Paolo Abeni, David S. Miller

Separate the logic of checking whether sq is full. The subsequent patch
will reuse this func.

Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
---
 drivers/net/virtio_net.c | 59 ++++++++++++++++++++++++----------------
 1 file changed, 35 insertions(+), 24 deletions(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index fb5e68ed3ec2..777de0ec0b1b 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1750,6 +1750,40 @@ static void free_old_xmit_skbs(struct send_queue *sq, bool in_napi)
 	u64_stats_update_end(&sq->stats.syncp);
 }
 
+static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
+			  struct send_queue *sq)
+{
+	bool use_napi = sq->napi.weight;
+	int qnum;
+
+	qnum = sq - vi->sq;
+
+	/* If running out of space, stop queue to avoid getting packets that we
+	 * are then unable to transmit.
+	 * An alternative would be to force queuing layer to requeue the skb by
+	 * returning NETDEV_TX_BUSY. However, NETDEV_TX_BUSY should not be
+	 * returned in a normal path of operation: it means that driver is not
+	 * maintaining the TX queue stop/start state properly, and causes
+	 * the stack to do a non-trivial amount of useless work.
+	 * Since most packets only take 1 or 2 ring slots, stopping the queue
+	 * early means 16 slots are typically wasted.
+	 */
+	if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
+		netif_stop_subqueue(dev, qnum);
+		if (use_napi) {
+			if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
+				virtqueue_napi_schedule(&sq->napi, sq->vq);
+		} else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
+			/* More just got used, free them then recheck. */
+			free_old_xmit_skbs(sq, false);
+			if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
+				netif_start_subqueue(dev, qnum);
+				virtqueue_disable_cb(sq->vq);
+			}
+		}
+	}
+}
+
 static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
 {
 	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
@@ -1989,30 +2023,7 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
 		nf_reset_ct(skb);
 	}
 
-	/* If running out of space, stop queue to avoid getting packets that we
-	 * are then unable to transmit.
-	 * An alternative would be to force queuing layer to requeue the skb by
-	 * returning NETDEV_TX_BUSY. However, NETDEV_TX_BUSY should not be
-	 * returned in a normal path of operation: it means that driver is not
-	 * maintaining the TX queue stop/start state properly, and causes
-	 * the stack to do a non-trivial amount of useless work.
-	 * Since most packets only take 1 or 2 ring slots, stopping the queue
-	 * early means 16 slots are typically wasted.
-	 */
-	if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
-		netif_stop_subqueue(dev, qnum);
-		if (use_napi) {
-			if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
-				virtqueue_napi_schedule(&sq->napi, sq->vq);
-		} else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
-			/* More just got used, free them then recheck. */
-			free_old_xmit_skbs(sq, false);
-			if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
-				netif_start_subqueue(dev, qnum);
-				virtqueue_disable_cb(sq->vq);
-			}
-		}
-	}
+	check_sq_full(vi, dev, sq);
 
 	if (kick || netif_xmit_stopped(txq)) {
 		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) {
-- 
2.32.0.3.g01195cf9f

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH net 1/2] virtio_net: separate the logic of checking whether sq is full
@ 2023-03-06  4:15   ` Xuan Zhuo
  0 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-06  4:15 UTC (permalink / raw)
  To: netdev
  Cc: Michael S. Tsirkin, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf

Separate the logic of checking whether sq is full. The subsequent patch
will reuse this func.

Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
---
 drivers/net/virtio_net.c | 59 ++++++++++++++++++++++++----------------
 1 file changed, 35 insertions(+), 24 deletions(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index fb5e68ed3ec2..777de0ec0b1b 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1750,6 +1750,40 @@ static void free_old_xmit_skbs(struct send_queue *sq, bool in_napi)
 	u64_stats_update_end(&sq->stats.syncp);
 }
 
+static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
+			  struct send_queue *sq)
+{
+	bool use_napi = sq->napi.weight;
+	int qnum;
+
+	qnum = sq - vi->sq;
+
+	/* If running out of space, stop queue to avoid getting packets that we
+	 * are then unable to transmit.
+	 * An alternative would be to force queuing layer to requeue the skb by
+	 * returning NETDEV_TX_BUSY. However, NETDEV_TX_BUSY should not be
+	 * returned in a normal path of operation: it means that driver is not
+	 * maintaining the TX queue stop/start state properly, and causes
+	 * the stack to do a non-trivial amount of useless work.
+	 * Since most packets only take 1 or 2 ring slots, stopping the queue
+	 * early means 16 slots are typically wasted.
+	 */
+	if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
+		netif_stop_subqueue(dev, qnum);
+		if (use_napi) {
+			if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
+				virtqueue_napi_schedule(&sq->napi, sq->vq);
+		} else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
+			/* More just got used, free them then recheck. */
+			free_old_xmit_skbs(sq, false);
+			if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
+				netif_start_subqueue(dev, qnum);
+				virtqueue_disable_cb(sq->vq);
+			}
+		}
+	}
+}
+
 static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
 {
 	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
@@ -1989,30 +2023,7 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
 		nf_reset_ct(skb);
 	}
 
-	/* If running out of space, stop queue to avoid getting packets that we
-	 * are then unable to transmit.
-	 * An alternative would be to force queuing layer to requeue the skb by
-	 * returning NETDEV_TX_BUSY. However, NETDEV_TX_BUSY should not be
-	 * returned in a normal path of operation: it means that driver is not
-	 * maintaining the TX queue stop/start state properly, and causes
-	 * the stack to do a non-trivial amount of useless work.
-	 * Since most packets only take 1 or 2 ring slots, stopping the queue
-	 * early means 16 slots are typically wasted.
-	 */
-	if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
-		netif_stop_subqueue(dev, qnum);
-		if (use_napi) {
-			if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
-				virtqueue_napi_schedule(&sq->napi, sq->vq);
-		} else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
-			/* More just got used, free them then recheck. */
-			free_old_xmit_skbs(sq, false);
-			if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
-				netif_start_subqueue(dev, qnum);
-				virtqueue_disable_cb(sq->vq);
-			}
-		}
-	}
+	check_sq_full(vi, dev, sq);
 
 	if (kick || netif_xmit_stopped(txq)) {
 		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) {
-- 
2.32.0.3.g01195cf9f


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH net 2/2] virtio_net: add checking sq is full inside xdp xmit
  2023-03-06  4:15 ` Xuan Zhuo
@ 2023-03-06  4:15   ` Xuan Zhuo
  -1 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-06  4:15 UTC (permalink / raw)
  To: netdev
  Cc: Yichun Zhang, Jesper Dangaard Brouer, Daniel Borkmann,
	Michael S. Tsirkin, John Fastabend, Alexei Starovoitov,
	virtualization, Eric Dumazet, Jakub Kicinski, bpf, Paolo Abeni,
	David S. Miller

If the queue of xdp xmit is not an independent queue, then when the xdp
xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
the following error.

net ens4: Unexpected TXQ (0) queue failure: -28

This patch adds a check whether sq is full in XDP Xmit.

Reported-by: Yichun Zhang <yichun@openresty.com>
Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
---
 drivers/net/virtio_net.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 777de0ec0b1b..3001b9a548e5 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -302,6 +302,8 @@ struct padded_vnet_hdr {
 
 static void virtnet_rq_free_unused_buf(struct virtqueue *vq, void *buf);
 static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf);
+static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
+			  struct send_queue *sq);
 
 static bool is_xdp_frame(void *ptr)
 {
@@ -341,6 +343,16 @@ static int rxq2vq(int rxq)
 	return rxq * 2;
 }
 
+static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
+{
+	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
+		return false;
+	else if (q < vi->curr_queue_pairs)
+		return true;
+	else
+		return false;
+}
+
 static inline struct virtio_net_hdr_mrg_rxbuf *skb_vnet_hdr(struct sk_buff *skb)
 {
 	return (struct virtio_net_hdr_mrg_rxbuf *)skb->cb;
@@ -686,6 +698,9 @@ static int virtnet_xdp_xmit(struct net_device *dev,
 	}
 	ret = nxmit;
 
+	if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq))
+		check_sq_full(vi, dev, sq);
+
 	if (flags & XDP_XMIT_FLUSH) {
 		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq))
 			kicks = 1;
@@ -1784,16 +1799,6 @@ static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
 	}
 }
 
-static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
-{
-	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
-		return false;
-	else if (q < vi->curr_queue_pairs)
-		return true;
-	else
-		return false;
-}
-
 static void virtnet_poll_cleantx(struct receive_queue *rq)
 {
 	struct virtnet_info *vi = rq->vq->vdev->priv;
-- 
2.32.0.3.g01195cf9f

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply related	[flat|nested] 24+ messages in thread

* [PATCH net 2/2] virtio_net: add checking sq is full inside xdp xmit
@ 2023-03-06  4:15   ` Xuan Zhuo
  0 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-06  4:15 UTC (permalink / raw)
  To: netdev
  Cc: Michael S. Tsirkin, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf,
	Yichun Zhang

If the queue of xdp xmit is not an independent queue, then when the xdp
xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
the following error.

net ens4: Unexpected TXQ (0) queue failure: -28

This patch adds a check whether sq is full in XDP Xmit.

Reported-by: Yichun Zhang <yichun@openresty.com>
Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
---
 drivers/net/virtio_net.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 777de0ec0b1b..3001b9a548e5 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -302,6 +302,8 @@ struct padded_vnet_hdr {
 
 static void virtnet_rq_free_unused_buf(struct virtqueue *vq, void *buf);
 static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf);
+static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
+			  struct send_queue *sq);
 
 static bool is_xdp_frame(void *ptr)
 {
@@ -341,6 +343,16 @@ static int rxq2vq(int rxq)
 	return rxq * 2;
 }
 
+static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
+{
+	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
+		return false;
+	else if (q < vi->curr_queue_pairs)
+		return true;
+	else
+		return false;
+}
+
 static inline struct virtio_net_hdr_mrg_rxbuf *skb_vnet_hdr(struct sk_buff *skb)
 {
 	return (struct virtio_net_hdr_mrg_rxbuf *)skb->cb;
@@ -686,6 +698,9 @@ static int virtnet_xdp_xmit(struct net_device *dev,
 	}
 	ret = nxmit;
 
+	if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq))
+		check_sq_full(vi, dev, sq);
+
 	if (flags & XDP_XMIT_FLUSH) {
 		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq))
 			kicks = 1;
@@ -1784,16 +1799,6 @@ static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
 	}
 }
 
-static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
-{
-	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
-		return false;
-	else if (q < vi->curr_queue_pairs)
-		return true;
-	else
-		return false;
-}
-
 static void virtnet_poll_cleantx(struct receive_queue *rq)
 {
 	struct virtnet_info *vi = rq->vq->vdev->priv;
-- 
2.32.0.3.g01195cf9f


^ permalink raw reply related	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
  2023-03-06  4:15 ` Xuan Zhuo
@ 2023-03-06 16:49   ` Alexander H Duyck
  -1 siblings, 0 replies; 24+ messages in thread
From: Alexander H Duyck @ 2023-03-06 16:49 UTC (permalink / raw)
  To: Xuan Zhuo, netdev
  Cc: Jesper Dangaard Brouer, Daniel Borkmann, Michael S. Tsirkin,
	John Fastabend, Alexei Starovoitov, virtualization, Eric Dumazet,
	Jakub Kicinski, bpf, Paolo Abeni, David S. Miller

On Mon, 2023-03-06 at 12:15 +0800, Xuan Zhuo wrote:
> If the queue of xdp xmit is not an independent queue, then when the xdp
> xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> the following error.
> 
> net ens4: Unexpected TXQ (0) queue failure: -28
> 
> This patch adds a check whether sq is full in XDP Xmit.
> 
> Thanks.
> 
> Xuan Zhuo (2):
>   virtio_net: separate the logic of checking whether sq is full
>   virtio_net: add checking sq is full inside xdp xmit
> 
>  drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++----------------
>  1 file changed, 47 insertions(+), 31 deletions(-)
> 
> --
> 2.32.0.3.g01195cf9f
> 

Series looks good to me.

Reviewed-by: Alexander Duyck <alexanderduyck@fb.com>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
@ 2023-03-06 16:49   ` Alexander H Duyck
  0 siblings, 0 replies; 24+ messages in thread
From: Alexander H Duyck @ 2023-03-06 16:49 UTC (permalink / raw)
  To: Xuan Zhuo, netdev
  Cc: Michael S. Tsirkin, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf

On Mon, 2023-03-06 at 12:15 +0800, Xuan Zhuo wrote:
> If the queue of xdp xmit is not an independent queue, then when the xdp
> xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> the following error.
> 
> net ens4: Unexpected TXQ (0) queue failure: -28
> 
> This patch adds a check whether sq is full in XDP Xmit.
> 
> Thanks.
> 
> Xuan Zhuo (2):
>   virtio_net: separate the logic of checking whether sq is full
>   virtio_net: add checking sq is full inside xdp xmit
> 
>  drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++----------------
>  1 file changed, 47 insertions(+), 31 deletions(-)
> 
> --
> 2.32.0.3.g01195cf9f
> 

Series looks good to me.

Reviewed-by: Alexander Duyck <alexanderduyck@fb.com>


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 2/2] virtio_net: add checking sq is full inside xdp xmit
  2023-03-06  4:15   ` Xuan Zhuo
@ 2023-03-06 17:57     ` Michael S. Tsirkin
  -1 siblings, 0 replies; 24+ messages in thread
From: Michael S. Tsirkin @ 2023-03-06 17:57 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: Yichun Zhang, Jesper Dangaard Brouer, Daniel Borkmann, netdev,
	John Fastabend, Alexei Starovoitov, virtualization, Eric Dumazet,
	Jakub Kicinski, bpf, Paolo Abeni, David S. Miller

On Mon, Mar 06, 2023 at 12:15:35PM +0800, Xuan Zhuo wrote:
> If the queue of xdp xmit is not an independent queue, then when the xdp
> xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> the following error.
> 
> net ens4: Unexpected TXQ (0) queue failure: -28
> 
> This patch adds a check whether sq is full in XDP Xmit.
> 
> Reported-by: Yichun Zhang <yichun@openresty.com>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  drivers/net/virtio_net.c | 25 +++++++++++++++----------
>  1 file changed, 15 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 777de0ec0b1b..3001b9a548e5 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -302,6 +302,8 @@ struct padded_vnet_hdr {
>  
>  static void virtnet_rq_free_unused_buf(struct virtqueue *vq, void *buf);
>  static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf);
> +static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
> +			  struct send_queue *sq);
>  
>  static bool is_xdp_frame(void *ptr)
>  {
> @@ -341,6 +343,16 @@ static int rxq2vq(int rxq)
>  	return rxq * 2;
>  }
>  

I'd really rather we ordered functions reasonably so declarations
are not needed.

> +static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
> +{
> +	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
> +		return false;
> +	else if (q < vi->curr_queue_pairs)
> +		return true;
> +	else
> +		return false;
> +}
> +
>  static inline struct virtio_net_hdr_mrg_rxbuf *skb_vnet_hdr(struct sk_buff *skb)
>  {
>  	return (struct virtio_net_hdr_mrg_rxbuf *)skb->cb;
> @@ -686,6 +698,9 @@ static int virtnet_xdp_xmit(struct net_device *dev,
>  	}
>  	ret = nxmit;
>  
> +	if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq))
> +		check_sq_full(vi, dev, sq);
> +
>  	if (flags & XDP_XMIT_FLUSH) {
>  		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq))
>  			kicks = 1;
> @@ -1784,16 +1799,6 @@ static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
>  	}
>  }
>  
> -static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
> -{
> -	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
> -		return false;
> -	else if (q < vi->curr_queue_pairs)
> -		return true;
> -	else
> -		return false;
> -}
> -
>  static void virtnet_poll_cleantx(struct receive_queue *rq)
>  {
>  	struct virtnet_info *vi = rq->vq->vdev->priv;
> -- 
> 2.32.0.3.g01195cf9f

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 2/2] virtio_net: add checking sq is full inside xdp xmit
@ 2023-03-06 17:57     ` Michael S. Tsirkin
  0 siblings, 0 replies; 24+ messages in thread
From: Michael S. Tsirkin @ 2023-03-06 17:57 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: netdev, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf,
	Yichun Zhang

On Mon, Mar 06, 2023 at 12:15:35PM +0800, Xuan Zhuo wrote:
> If the queue of xdp xmit is not an independent queue, then when the xdp
> xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> the following error.
> 
> net ens4: Unexpected TXQ (0) queue failure: -28
> 
> This patch adds a check whether sq is full in XDP Xmit.
> 
> Reported-by: Yichun Zhang <yichun@openresty.com>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  drivers/net/virtio_net.c | 25 +++++++++++++++----------
>  1 file changed, 15 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 777de0ec0b1b..3001b9a548e5 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -302,6 +302,8 @@ struct padded_vnet_hdr {
>  
>  static void virtnet_rq_free_unused_buf(struct virtqueue *vq, void *buf);
>  static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf);
> +static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
> +			  struct send_queue *sq);
>  
>  static bool is_xdp_frame(void *ptr)
>  {
> @@ -341,6 +343,16 @@ static int rxq2vq(int rxq)
>  	return rxq * 2;
>  }
>  

I'd really rather we ordered functions reasonably so declarations
are not needed.

> +static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
> +{
> +	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
> +		return false;
> +	else if (q < vi->curr_queue_pairs)
> +		return true;
> +	else
> +		return false;
> +}
> +
>  static inline struct virtio_net_hdr_mrg_rxbuf *skb_vnet_hdr(struct sk_buff *skb)
>  {
>  	return (struct virtio_net_hdr_mrg_rxbuf *)skb->cb;
> @@ -686,6 +698,9 @@ static int virtnet_xdp_xmit(struct net_device *dev,
>  	}
>  	ret = nxmit;
>  
> +	if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq))
> +		check_sq_full(vi, dev, sq);
> +
>  	if (flags & XDP_XMIT_FLUSH) {
>  		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq))
>  			kicks = 1;
> @@ -1784,16 +1799,6 @@ static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
>  	}
>  }
>  
> -static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
> -{
> -	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
> -		return false;
> -	else if (q < vi->curr_queue_pairs)
> -		return true;
> -	else
> -		return false;
> -}
> -
>  static void virtnet_poll_cleantx(struct receive_queue *rq)
>  {
>  	struct virtnet_info *vi = rq->vq->vdev->priv;
> -- 
> 2.32.0.3.g01195cf9f


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
  2023-03-06  4:15 ` Xuan Zhuo
@ 2023-03-06 17:58   ` Michael S. Tsirkin
  -1 siblings, 0 replies; 24+ messages in thread
From: Michael S. Tsirkin @ 2023-03-06 17:58 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: Jesper Dangaard Brouer, Daniel Borkmann, netdev, John Fastabend,
	Alexei Starovoitov, virtualization, Eric Dumazet, Jakub Kicinski,
	bpf, Paolo Abeni, David S. Miller

On Mon, Mar 06, 2023 at 12:15:33PM +0800, Xuan Zhuo wrote:
> If the queue of xdp xmit is not an independent queue, then when the xdp
> xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> the following error.
> 
> net ens4: Unexpected TXQ (0) queue failure: -28
> 
> This patch adds a check whether sq is full in XDP Xmit.
> 
> Thanks.

Acked-by: Michael S. Tsirkin <mst@redhat.com>

needed for stable?

> Xuan Zhuo (2):
>   virtio_net: separate the logic of checking whether sq is full
>   virtio_net: add checking sq is full inside xdp xmit
> 
>  drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++----------------
>  1 file changed, 47 insertions(+), 31 deletions(-)
> 
> --
> 2.32.0.3.g01195cf9f

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
@ 2023-03-06 17:58   ` Michael S. Tsirkin
  0 siblings, 0 replies; 24+ messages in thread
From: Michael S. Tsirkin @ 2023-03-06 17:58 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: netdev, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf

On Mon, Mar 06, 2023 at 12:15:33PM +0800, Xuan Zhuo wrote:
> If the queue of xdp xmit is not an independent queue, then when the xdp
> xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> the following error.
> 
> net ens4: Unexpected TXQ (0) queue failure: -28
> 
> This patch adds a check whether sq is full in XDP Xmit.
> 
> Thanks.

Acked-by: Michael S. Tsirkin <mst@redhat.com>

needed for stable?

> Xuan Zhuo (2):
>   virtio_net: separate the logic of checking whether sq is full
>   virtio_net: add checking sq is full inside xdp xmit
> 
>  drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++----------------
>  1 file changed, 47 insertions(+), 31 deletions(-)
> 
> --
> 2.32.0.3.g01195cf9f


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 2/2] virtio_net: add checking sq is full inside xdp xmit
  2023-03-06 17:57     ` Michael S. Tsirkin
@ 2023-03-07  1:47       ` Xuan Zhuo
  -1 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-07  1:47 UTC (permalink / raw)
  To: Michael S. Tsirkin
  Cc: netdev, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf,
	Yichun Zhang

On Mon, 6 Mar 2023 12:57:34 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Mon, Mar 06, 2023 at 12:15:35PM +0800, Xuan Zhuo wrote:
> > If the queue of xdp xmit is not an independent queue, then when the xdp
> > xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> > the following error.
> >
> > net ens4: Unexpected TXQ (0) queue failure: -28
> >
> > This patch adds a check whether sq is full in XDP Xmit.
> >
> > Reported-by: Yichun Zhang <yichun@openresty.com>
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  drivers/net/virtio_net.c | 25 +++++++++++++++----------
> >  1 file changed, 15 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > index 777de0ec0b1b..3001b9a548e5 100644
> > --- a/drivers/net/virtio_net.c
> > +++ b/drivers/net/virtio_net.c
> > @@ -302,6 +302,8 @@ struct padded_vnet_hdr {
> >
> >  static void virtnet_rq_free_unused_buf(struct virtqueue *vq, void *buf);
> >  static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf);
> > +static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
> > +			  struct send_queue *sq);
> >
> >  static bool is_xdp_frame(void *ptr)
> >  {
> > @@ -341,6 +343,16 @@ static int rxq2vq(int rxq)
> >  	return rxq * 2;
> >  }
> >
>
> I'd really rather we ordered functions reasonably so declarations
> are not needed.

Yes, but this function depends on some other functions. If we put this function
above, then these dependent functions must be processed. The current method
should be a relatively simple way.

Thanks.


>
> > +static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
> > +{
> > +	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
> > +		return false;
> > +	else if (q < vi->curr_queue_pairs)
> > +		return true;
> > +	else
> > +		return false;
> > +}
> > +
> >  static inline struct virtio_net_hdr_mrg_rxbuf *skb_vnet_hdr(struct sk_buff *skb)
> >  {
> >  	return (struct virtio_net_hdr_mrg_rxbuf *)skb->cb;
> > @@ -686,6 +698,9 @@ static int virtnet_xdp_xmit(struct net_device *dev,
> >  	}
> >  	ret = nxmit;
> >
> > +	if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq))
> > +		check_sq_full(vi, dev, sq);
> > +
> >  	if (flags & XDP_XMIT_FLUSH) {
> >  		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq))
> >  			kicks = 1;
> > @@ -1784,16 +1799,6 @@ static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
> >  	}
> >  }
> >
> > -static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
> > -{
> > -	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
> > -		return false;
> > -	else if (q < vi->curr_queue_pairs)
> > -		return true;
> > -	else
> > -		return false;
> > -}
> > -
> >  static void virtnet_poll_cleantx(struct receive_queue *rq)
> >  {
> >  	struct virtnet_info *vi = rq->vq->vdev->priv;
> > --
> > 2.32.0.3.g01195cf9f
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 2/2] virtio_net: add checking sq is full inside xdp xmit
@ 2023-03-07  1:47       ` Xuan Zhuo
  0 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-07  1:47 UTC (permalink / raw)
  To: Michael S. Tsirkin
  Cc: Yichun Zhang, Jesper Dangaard Brouer, Daniel Borkmann, netdev,
	John Fastabend, Alexei Starovoitov, virtualization, Eric Dumazet,
	Jakub Kicinski, bpf, Paolo Abeni, David S. Miller

On Mon, 6 Mar 2023 12:57:34 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Mon, Mar 06, 2023 at 12:15:35PM +0800, Xuan Zhuo wrote:
> > If the queue of xdp xmit is not an independent queue, then when the xdp
> > xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> > the following error.
> >
> > net ens4: Unexpected TXQ (0) queue failure: -28
> >
> > This patch adds a check whether sq is full in XDP Xmit.
> >
> > Reported-by: Yichun Zhang <yichun@openresty.com>
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  drivers/net/virtio_net.c | 25 +++++++++++++++----------
> >  1 file changed, 15 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > index 777de0ec0b1b..3001b9a548e5 100644
> > --- a/drivers/net/virtio_net.c
> > +++ b/drivers/net/virtio_net.c
> > @@ -302,6 +302,8 @@ struct padded_vnet_hdr {
> >
> >  static void virtnet_rq_free_unused_buf(struct virtqueue *vq, void *buf);
> >  static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf);
> > +static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
> > +			  struct send_queue *sq);
> >
> >  static bool is_xdp_frame(void *ptr)
> >  {
> > @@ -341,6 +343,16 @@ static int rxq2vq(int rxq)
> >  	return rxq * 2;
> >  }
> >
>
> I'd really rather we ordered functions reasonably so declarations
> are not needed.

Yes, but this function depends on some other functions. If we put this function
above, then these dependent functions must be processed. The current method
should be a relatively simple way.

Thanks.


>
> > +static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
> > +{
> > +	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
> > +		return false;
> > +	else if (q < vi->curr_queue_pairs)
> > +		return true;
> > +	else
> > +		return false;
> > +}
> > +
> >  static inline struct virtio_net_hdr_mrg_rxbuf *skb_vnet_hdr(struct sk_buff *skb)
> >  {
> >  	return (struct virtio_net_hdr_mrg_rxbuf *)skb->cb;
> > @@ -686,6 +698,9 @@ static int virtnet_xdp_xmit(struct net_device *dev,
> >  	}
> >  	ret = nxmit;
> >
> > +	if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq))
> > +		check_sq_full(vi, dev, sq);
> > +
> >  	if (flags & XDP_XMIT_FLUSH) {
> >  		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq))
> >  			kicks = 1;
> > @@ -1784,16 +1799,6 @@ static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
> >  	}
> >  }
> >
> > -static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
> > -{
> > -	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
> > -		return false;
> > -	else if (q < vi->curr_queue_pairs)
> > -		return true;
> > -	else
> > -		return false;
> > -}
> > -
> >  static void virtnet_poll_cleantx(struct receive_queue *rq)
> >  {
> >  	struct virtnet_info *vi = rq->vq->vdev->priv;
> > --
> > 2.32.0.3.g01195cf9f
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
  2023-03-06 17:58   ` Michael S. Tsirkin
@ 2023-03-07  1:49     ` Xuan Zhuo
  -1 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-07  1:49 UTC (permalink / raw)
  To: Michael S. Tsirkin
  Cc: netdev, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf

On Mon, 6 Mar 2023 12:58:22 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Mon, Mar 06, 2023 at 12:15:33PM +0800, Xuan Zhuo wrote:
> > If the queue of xdp xmit is not an independent queue, then when the xdp
> > xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> > the following error.
> >
> > net ens4: Unexpected TXQ (0) queue failure: -28
> >
> > This patch adds a check whether sq is full in XDP Xmit.
> >
> > Thanks.
>
> Acked-by: Michael S. Tsirkin <mst@redhat.com>
>
> needed for stable?


Yes i think.

Thanks.


>
> > Xuan Zhuo (2):
> >   virtio_net: separate the logic of checking whether sq is full
> >   virtio_net: add checking sq is full inside xdp xmit
> >
> >  drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++----------------
> >  1 file changed, 47 insertions(+), 31 deletions(-)
> >
> > --
> > 2.32.0.3.g01195cf9f
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
@ 2023-03-07  1:49     ` Xuan Zhuo
  0 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-07  1:49 UTC (permalink / raw)
  To: Michael S. Tsirkin
  Cc: Jesper Dangaard Brouer, Daniel Borkmann, netdev, John Fastabend,
	Alexei Starovoitov, virtualization, Eric Dumazet, Jakub Kicinski,
	bpf, Paolo Abeni, David S. Miller

On Mon, 6 Mar 2023 12:58:22 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Mon, Mar 06, 2023 at 12:15:33PM +0800, Xuan Zhuo wrote:
> > If the queue of xdp xmit is not an independent queue, then when the xdp
> > xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> > the following error.
> >
> > net ens4: Unexpected TXQ (0) queue failure: -28
> >
> > This patch adds a check whether sq is full in XDP Xmit.
> >
> > Thanks.
>
> Acked-by: Michael S. Tsirkin <mst@redhat.com>
>
> needed for stable?


Yes i think.

Thanks.


>
> > Xuan Zhuo (2):
> >   virtio_net: separate the logic of checking whether sq is full
> >   virtio_net: add checking sq is full inside xdp xmit
> >
> >  drivers/net/virtio_net.c | 78 ++++++++++++++++++++++++----------------
> >  1 file changed, 47 insertions(+), 31 deletions(-)
> >
> > --
> > 2.32.0.3.g01195cf9f
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 1/2] virtio_net: separate the logic of checking whether sq is full
  2023-03-06  4:15   ` Xuan Zhuo
@ 2023-03-07  8:34     ` Jason Wang
  -1 siblings, 0 replies; 24+ messages in thread
From: Jason Wang @ 2023-03-07  8:34 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: Jesper Dangaard Brouer, Daniel Borkmann, Michael S. Tsirkin,
	netdev, John Fastabend, Alexei Starovoitov, virtualization,
	Eric Dumazet, Jakub Kicinski, bpf, Paolo Abeni, David S. Miller

On Mon, Mar 6, 2023 at 12:15 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> Separate the logic of checking whether sq is full. The subsequent patch
> will reuse this func.
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  drivers/net/virtio_net.c | 59 ++++++++++++++++++++++++----------------
>  1 file changed, 35 insertions(+), 24 deletions(-)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index fb5e68ed3ec2..777de0ec0b1b 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1750,6 +1750,40 @@ static void free_old_xmit_skbs(struct send_queue *sq, bool in_napi)
>         u64_stats_update_end(&sq->stats.syncp);
>  }
>
> +static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
> +                         struct send_queue *sq)
> +{

Nit: we need a better name, it's not only the check but also can disable TX.

Thanks

> +       bool use_napi = sq->napi.weight;
> +       int qnum;
> +
> +       qnum = sq - vi->sq;
> +
> +       /* If running out of space, stop queue to avoid getting packets that we
> +        * are then unable to transmit.
> +        * An alternative would be to force queuing layer to requeue the skb by
> +        * returning NETDEV_TX_BUSY. However, NETDEV_TX_BUSY should not be
> +        * returned in a normal path of operation: it means that driver is not
> +        * maintaining the TX queue stop/start state properly, and causes
> +        * the stack to do a non-trivial amount of useless work.
> +        * Since most packets only take 1 or 2 ring slots, stopping the queue
> +        * early means 16 slots are typically wasted.
> +        */
> +       if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
> +               netif_stop_subqueue(dev, qnum);
> +               if (use_napi) {
> +                       if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
> +                               virtqueue_napi_schedule(&sq->napi, sq->vq);
> +               } else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
> +                       /* More just got used, free them then recheck. */
> +                       free_old_xmit_skbs(sq, false);
> +                       if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
> +                               netif_start_subqueue(dev, qnum);
> +                               virtqueue_disable_cb(sq->vq);
> +                       }
> +               }
> +       }
> +}
> +
>  static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
>  {
>         if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
> @@ -1989,30 +2023,7 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
>                 nf_reset_ct(skb);
>         }
>
> -       /* If running out of space, stop queue to avoid getting packets that we
> -        * are then unable to transmit.
> -        * An alternative would be to force queuing layer to requeue the skb by
> -        * returning NETDEV_TX_BUSY. However, NETDEV_TX_BUSY should not be
> -        * returned in a normal path of operation: it means that driver is not
> -        * maintaining the TX queue stop/start state properly, and causes
> -        * the stack to do a non-trivial amount of useless work.
> -        * Since most packets only take 1 or 2 ring slots, stopping the queue
> -        * early means 16 slots are typically wasted.
> -        */
> -       if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
> -               netif_stop_subqueue(dev, qnum);
> -               if (use_napi) {
> -                       if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
> -                               virtqueue_napi_schedule(&sq->napi, sq->vq);
> -               } else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
> -                       /* More just got used, free them then recheck. */
> -                       free_old_xmit_skbs(sq, false);
> -                       if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
> -                               netif_start_subqueue(dev, qnum);
> -                               virtqueue_disable_cb(sq->vq);
> -                       }
> -               }
> -       }
> +       check_sq_full(vi, dev, sq);
>
>         if (kick || netif_xmit_stopped(txq)) {
>                 if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) {
> --
> 2.32.0.3.g01195cf9f
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 1/2] virtio_net: separate the logic of checking whether sq is full
@ 2023-03-07  8:34     ` Jason Wang
  0 siblings, 0 replies; 24+ messages in thread
From: Jason Wang @ 2023-03-07  8:34 UTC (permalink / raw)
  To: Xuan Zhuo
  Cc: netdev, Michael S. Tsirkin, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf

On Mon, Mar 6, 2023 at 12:15 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> Separate the logic of checking whether sq is full. The subsequent patch
> will reuse this func.
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  drivers/net/virtio_net.c | 59 ++++++++++++++++++++++++----------------
>  1 file changed, 35 insertions(+), 24 deletions(-)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index fb5e68ed3ec2..777de0ec0b1b 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1750,6 +1750,40 @@ static void free_old_xmit_skbs(struct send_queue *sq, bool in_napi)
>         u64_stats_update_end(&sq->stats.syncp);
>  }
>
> +static void check_sq_full(struct virtnet_info *vi, struct net_device *dev,
> +                         struct send_queue *sq)
> +{

Nit: we need a better name, it's not only the check but also can disable TX.

Thanks

> +       bool use_napi = sq->napi.weight;
> +       int qnum;
> +
> +       qnum = sq - vi->sq;
> +
> +       /* If running out of space, stop queue to avoid getting packets that we
> +        * are then unable to transmit.
> +        * An alternative would be to force queuing layer to requeue the skb by
> +        * returning NETDEV_TX_BUSY. However, NETDEV_TX_BUSY should not be
> +        * returned in a normal path of operation: it means that driver is not
> +        * maintaining the TX queue stop/start state properly, and causes
> +        * the stack to do a non-trivial amount of useless work.
> +        * Since most packets only take 1 or 2 ring slots, stopping the queue
> +        * early means 16 slots are typically wasted.
> +        */
> +       if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
> +               netif_stop_subqueue(dev, qnum);
> +               if (use_napi) {
> +                       if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
> +                               virtqueue_napi_schedule(&sq->napi, sq->vq);
> +               } else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
> +                       /* More just got used, free them then recheck. */
> +                       free_old_xmit_skbs(sq, false);
> +                       if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
> +                               netif_start_subqueue(dev, qnum);
> +                               virtqueue_disable_cb(sq->vq);
> +                       }
> +               }
> +       }
> +}
> +
>  static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
>  {
>         if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
> @@ -1989,30 +2023,7 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
>                 nf_reset_ct(skb);
>         }
>
> -       /* If running out of space, stop queue to avoid getting packets that we
> -        * are then unable to transmit.
> -        * An alternative would be to force queuing layer to requeue the skb by
> -        * returning NETDEV_TX_BUSY. However, NETDEV_TX_BUSY should not be
> -        * returned in a normal path of operation: it means that driver is not
> -        * maintaining the TX queue stop/start state properly, and causes
> -        * the stack to do a non-trivial amount of useless work.
> -        * Since most packets only take 1 or 2 ring slots, stopping the queue
> -        * early means 16 slots are typically wasted.
> -        */
> -       if (sq->vq->num_free < 2+MAX_SKB_FRAGS) {
> -               netif_stop_subqueue(dev, qnum);
> -               if (use_napi) {
> -                       if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
> -                               virtqueue_napi_schedule(&sq->napi, sq->vq);
> -               } else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
> -                       /* More just got used, free them then recheck. */
> -                       free_old_xmit_skbs(sq, false);
> -                       if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) {
> -                               netif_start_subqueue(dev, qnum);
> -                               virtqueue_disable_cb(sq->vq);
> -                       }
> -               }
> -       }
> +       check_sq_full(vi, dev, sq);
>
>         if (kick || netif_xmit_stopped(txq)) {
>                 if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) {
> --
> 2.32.0.3.g01195cf9f
>


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
  2023-03-07  1:49     ` Xuan Zhuo
@ 2023-03-07  9:53       ` Paolo Abeni
  -1 siblings, 0 replies; 24+ messages in thread
From: Paolo Abeni @ 2023-03-07  9:53 UTC (permalink / raw)
  To: Xuan Zhuo, Michael S. Tsirkin
  Cc: Jesper Dangaard Brouer, Daniel Borkmann, netdev, John Fastabend,
	Alexei Starovoitov, virtualization, Eric Dumazet, Jakub Kicinski,
	bpf, David S. Miller

Hi,
On Tue, 2023-03-07 at 09:49 +0800, Xuan Zhuo wrote:
> On Mon, 6 Mar 2023 12:58:22 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Mon, Mar 06, 2023 at 12:15:33PM +0800, Xuan Zhuo wrote:
> > > If the queue of xdp xmit is not an independent queue, then when the xdp
> > > xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> > > the following error.
> > > 
> > > net ens4: Unexpected TXQ (0) queue failure: -28
> > > 
> > > This patch adds a check whether sq is full in XDP Xmit.
> > > 
> > > Thanks.
> > 
> > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > 
> > needed for stable?
> 
> Yes i think.

Could you please re-post including a suitable 'Fixes' tag? That would
address stable, too. Additionally you could rename check_sq_full() in
patch 1, perhaps 'check_disable_sq_full()' would do.

You can retain the already collected tags.

Thanks!

Paolo

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
@ 2023-03-07  9:53       ` Paolo Abeni
  0 siblings, 0 replies; 24+ messages in thread
From: Paolo Abeni @ 2023-03-07  9:53 UTC (permalink / raw)
  To: Xuan Zhuo, Michael S. Tsirkin
  Cc: netdev, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf

Hi,
On Tue, 2023-03-07 at 09:49 +0800, Xuan Zhuo wrote:
> On Mon, 6 Mar 2023 12:58:22 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Mon, Mar 06, 2023 at 12:15:33PM +0800, Xuan Zhuo wrote:
> > > If the queue of xdp xmit is not an independent queue, then when the xdp
> > > xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> > > the following error.
> > > 
> > > net ens4: Unexpected TXQ (0) queue failure: -28
> > > 
> > > This patch adds a check whether sq is full in XDP Xmit.
> > > 
> > > Thanks.
> > 
> > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > 
> > needed for stable?
> 
> Yes i think.

Could you please re-post including a suitable 'Fixes' tag? That would
address stable, too. Additionally you could rename check_sq_full() in
patch 1, perhaps 'check_disable_sq_full()' would do.

You can retain the already collected tags.

Thanks!

Paolo


^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
  2023-03-07  9:53       ` Paolo Abeni
@ 2023-03-07 11:08         ` Xuan Zhuo
  -1 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-07 11:08 UTC (permalink / raw)
  To: Paolo Abeni
  Cc: Jesper Dangaard Brouer, Daniel Borkmann, Michael S. Tsirkin,
	netdev, John Fastabend, Alexei Starovoitov, virtualization,
	Eric Dumazet, Jakub Kicinski, bpf, David S.  Miller

On Tue, 07 Mar 2023 10:53:41 +0100, Paolo Abeni <pabeni@redhat.com> wrote:
> Hi,
> On Tue, 2023-03-07 at 09:49 +0800, Xuan Zhuo wrote:
> > On Mon, 6 Mar 2023 12:58:22 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Mon, Mar 06, 2023 at 12:15:33PM +0800, Xuan Zhuo wrote:
> > > > If the queue of xdp xmit is not an independent queue, then when the xdp
> > > > xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> > > > the following error.
> > > >
> > > > net ens4: Unexpected TXQ (0) queue failure: -28
> > > >
> > > > This patch adds a check whether sq is full in XDP Xmit.
> > > >
> > > > Thanks.
> > >
> > > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > >
> > > needed for stable?
> >
> > Yes i think.
>
> Could you please re-post including a suitable 'Fixes' tag? That would
> address stable, too. Additionally you could rename check_sq_full() in
> patch 1, perhaps 'check_disable_sq_full()' would do.
>
> You can retain the already collected tags.

OK

Thanks.


>
> Thanks!
>
> Paolo
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
@ 2023-03-07 11:08         ` Xuan Zhuo
  0 siblings, 0 replies; 24+ messages in thread
From: Xuan Zhuo @ 2023-03-07 11:08 UTC (permalink / raw)
  To: Paolo Abeni
  Cc: netdev, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf,
	Michael S. Tsirkin

On Tue, 07 Mar 2023 10:53:41 +0100, Paolo Abeni <pabeni@redhat.com> wrote:
> Hi,
> On Tue, 2023-03-07 at 09:49 +0800, Xuan Zhuo wrote:
> > On Mon, 6 Mar 2023 12:58:22 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Mon, Mar 06, 2023 at 12:15:33PM +0800, Xuan Zhuo wrote:
> > > > If the queue of xdp xmit is not an independent queue, then when the xdp
> > > > xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> > > > the following error.
> > > >
> > > > net ens4: Unexpected TXQ (0) queue failure: -28
> > > >
> > > > This patch adds a check whether sq is full in XDP Xmit.
> > > >
> > > > Thanks.
> > >
> > > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > >
> > > needed for stable?
> >
> > Yes i think.
>
> Could you please re-post including a suitable 'Fixes' tag? That would
> address stable, too. Additionally you could rename check_sq_full() in
> patch 1, perhaps 'check_disable_sq_full()' would do.
>
> You can retain the already collected tags.

OK

Thanks.


>
> Thanks!
>
> Paolo
>

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
  2023-03-07  9:53       ` Paolo Abeni
@ 2023-03-07 15:59         ` Michael S. Tsirkin
  -1 siblings, 0 replies; 24+ messages in thread
From: Michael S. Tsirkin @ 2023-03-07 15:59 UTC (permalink / raw)
  To: Paolo Abeni
  Cc: Jesper Dangaard Brouer, Daniel Borkmann, netdev, John Fastabend,
	Alexei Starovoitov, virtualization, Eric Dumazet, Jakub Kicinski,
	bpf, David S. Miller

On Tue, Mar 07, 2023 at 10:53:41AM +0100, Paolo Abeni wrote:
> Hi,
> On Tue, 2023-03-07 at 09:49 +0800, Xuan Zhuo wrote:
> > On Mon, 6 Mar 2023 12:58:22 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Mon, Mar 06, 2023 at 12:15:33PM +0800, Xuan Zhuo wrote:
> > > > If the queue of xdp xmit is not an independent queue, then when the xdp
> > > > xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> > > > the following error.
> > > > 
> > > > net ens4: Unexpected TXQ (0) queue failure: -28
> > > > 
> > > > This patch adds a check whether sq is full in XDP Xmit.
> > > > 
> > > > Thanks.
> > > 
> > > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > > 
> > > needed for stable?
> > 
> > Yes i think.
> 
> Could you please re-post including a suitable 'Fixes' tag? That would
> address stable, too. Additionally you could rename check_sq_full() in
> patch 1, perhaps 'check_disable_sq_full()' would do.

Not that's even more confusing it sounds like it's checking that
it's checking that sq is disabled unless one looks closely.

I think check_sq_full_and_disable() is better.

> You can retain the already collected tags.
> 
> Thanks!
> 
> Paolo

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: [PATCH net 0/2] add checking sq is full inside xdp xmit
@ 2023-03-07 15:59         ` Michael S. Tsirkin
  0 siblings, 0 replies; 24+ messages in thread
From: Michael S. Tsirkin @ 2023-03-07 15:59 UTC (permalink / raw)
  To: Paolo Abeni
  Cc: Xuan Zhuo, netdev, Jason Wang, David S. Miller, Eric Dumazet,
	Jakub Kicinski, Alexei Starovoitov, Daniel Borkmann,
	Jesper Dangaard Brouer, John Fastabend, virtualization, bpf

On Tue, Mar 07, 2023 at 10:53:41AM +0100, Paolo Abeni wrote:
> Hi,
> On Tue, 2023-03-07 at 09:49 +0800, Xuan Zhuo wrote:
> > On Mon, 6 Mar 2023 12:58:22 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Mon, Mar 06, 2023 at 12:15:33PM +0800, Xuan Zhuo wrote:
> > > > If the queue of xdp xmit is not an independent queue, then when the xdp
> > > > xmit used all the desc, the xmit from the __dev_queue_xmit() may encounter
> > > > the following error.
> > > > 
> > > > net ens4: Unexpected TXQ (0) queue failure: -28
> > > > 
> > > > This patch adds a check whether sq is full in XDP Xmit.
> > > > 
> > > > Thanks.
> > > 
> > > Acked-by: Michael S. Tsirkin <mst@redhat.com>
> > > 
> > > needed for stable?
> > 
> > Yes i think.
> 
> Could you please re-post including a suitable 'Fixes' tag? That would
> address stable, too. Additionally you could rename check_sq_full() in
> patch 1, perhaps 'check_disable_sq_full()' would do.

Not that's even more confusing it sounds like it's checking that
it's checking that sq is disabled unless one looks closely.

I think check_sq_full_and_disable() is better.

> You can retain the already collected tags.
> 
> Thanks!
> 
> Paolo


^ permalink raw reply	[flat|nested] 24+ messages in thread

end of thread, other threads:[~2023-03-07 16:04 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-06  4:15 [PATCH net 0/2] add checking sq is full inside xdp xmit Xuan Zhuo
2023-03-06  4:15 ` Xuan Zhuo
2023-03-06  4:15 ` [PATCH net 1/2] virtio_net: separate the logic of checking whether sq is full Xuan Zhuo
2023-03-06  4:15   ` Xuan Zhuo
2023-03-07  8:34   ` Jason Wang
2023-03-07  8:34     ` Jason Wang
2023-03-06  4:15 ` [PATCH net 2/2] virtio_net: add checking sq is full inside xdp xmit Xuan Zhuo
2023-03-06  4:15   ` Xuan Zhuo
2023-03-06 17:57   ` Michael S. Tsirkin
2023-03-06 17:57     ` Michael S. Tsirkin
2023-03-07  1:47     ` Xuan Zhuo
2023-03-07  1:47       ` Xuan Zhuo
2023-03-06 16:49 ` [PATCH net 0/2] " Alexander H Duyck
2023-03-06 16:49   ` Alexander H Duyck
2023-03-06 17:58 ` Michael S. Tsirkin
2023-03-06 17:58   ` Michael S. Tsirkin
2023-03-07  1:49   ` Xuan Zhuo
2023-03-07  1:49     ` Xuan Zhuo
2023-03-07  9:53     ` Paolo Abeni
2023-03-07  9:53       ` Paolo Abeni
2023-03-07 11:08       ` Xuan Zhuo
2023-03-07 11:08         ` Xuan Zhuo
2023-03-07 15:59       ` Michael S. Tsirkin
2023-03-07 15:59         ` Michael S. Tsirkin

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.