[PATCH bpf-next v2 00/16] AF_XDP infrastructure improvements and mlx5e support

[PATCH bpf-next v2 00/16] AF_XDP infrastructure improvements and mlx5e support

[Maxim Mikityanskiy]

This series contains improvements to the AF_XDP kernel infrastructure
and AF_XDP support in mlx5e. The infrastructure improvements are
required for mlx5e, but also some of them benefit to all drivers, and
some can be useful for other drivers that want to implement AF_XDP.

The performance testing was performed on a machine with the following
configuration:

- 24 cores of Intel Xeon E5-2620 v3 @ 2.40 GHz
- Mellanox ConnectX-5 Ex with 100 Gbit/s link

The results with retpoline disabled, single stream:

txonly: 33.3 Mpps (21.5 Mpps with queue and app pinned to the same CPU)
rxdrop: 12.2 Mpps
l2fwd: 9.4 Mpps

The results with retpoline enabled, single stream:

txonly: 21.3 Mpps (14.1 Mpps with queue and app pinned to the same CPU)
rxdrop: 9.9 Mpps
l2fwd: 6.8 Mpps

v2 changes:

Added patches for mlx5e and addressed the comments for v1. Rebased for
bpf-next (net-next has to be merged first, because this series depends
on some patches from there).

Maxim Mikityanskiy (16):
  xsk: Add API to check for available entries in FQ
  xsk: Add getsockopt XDP_OPTIONS
  libbpf: Support getsockopt XDP_OPTIONS
  xsk: Extend channels to support combined XSK/non-XSK traffic
  xsk: Change the default frame size to 4096 and allow controlling it
  xsk: Return the whole xdp_desc from xsk_umem_consume_tx
  net/mlx5e: Replace deprecated PCI_DMA_TODEVICE
  net/mlx5e: Calculate linear RX frag size considering XSK
  net/mlx5e: Allow ICO SQ to be used by multiple RQs
  net/mlx5e: Refactor struct mlx5e_xdp_info
  net/mlx5e: Share the XDP SQ for XDP_TX between RQs
  net/mlx5e: XDP_TX from UMEM support
  net/mlx5e: Consider XSK in XDP MTU limit calculation
  net/mlx5e: Encapsulate open/close queues into a function
  net/mlx5e: Move queue param structs to en/params.h
  net/mlx5e: Add XSK support

 drivers/net/ethernet/intel/i40e/i40e_xsk.c    |  12 +-
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c  |  15 +-
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 drivers/net/ethernet/mellanox/mlx5/core/en.h  | 147 +++-
 .../ethernet/mellanox/mlx5/core/en/params.c   | 108 ++-
 .../ethernet/mellanox/mlx5/core/en/params.h   |  87 ++-
 .../net/ethernet/mellanox/mlx5/core/en/xdp.c  | 231 ++++--
 .../net/ethernet/mellanox/mlx5/core/en/xdp.h  |  36 +-
 .../mellanox/mlx5/core/en/xsk/Makefile        |   1 +
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   | 192 +++++
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.h   |  27 +
 .../mellanox/mlx5/core/en/xsk/setup.c         | 220 ++++++
 .../mellanox/mlx5/core/en/xsk/setup.h         |  25 +
 .../ethernet/mellanox/mlx5/core/en/xsk/tx.c   | 108 +++
 .../ethernet/mellanox/mlx5/core/en/xsk/tx.h   |  15 +
 .../ethernet/mellanox/mlx5/core/en/xsk/umem.c | 252 +++++++
 .../ethernet/mellanox/mlx5/core/en/xsk/umem.h |  34 +
 .../ethernet/mellanox/mlx5/core/en_ethtool.c  |  21 +-
 .../mellanox/mlx5/core/en_fs_ethtool.c        |  44 +-
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 680 +++++++++++-------
 .../net/ethernet/mellanox/mlx5/core/en_rep.c  |  12 +-
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 104 ++-
 .../ethernet/mellanox/mlx5/core/en_stats.c    | 115 ++-
 .../ethernet/mellanox/mlx5/core/en_stats.h    |  30 +
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |  42 +-
 .../ethernet/mellanox/mlx5/core/ipoib/ipoib.c |  14 +-
 drivers/net/ethernet/mellanox/mlx5/core/wq.h  |   5 -
 include/net/xdp_sock.h                        |  27 +-
 include/uapi/linux/if_xdp.h                   |  18 +
 net/xdp/xsk.c                                 |  43 +-
 net/xdp/xsk_queue.h                           |  14 +
 samples/bpf/xdpsock_user.c                    |  52 +-
 tools/include/uapi/linux/if_xdp.h             |  18 +
 tools/lib/bpf/xsk.c                           | 127 +++-
 tools/lib/bpf/xsk.h                           |   6 +-
 35 files changed, 2384 insertions(+), 500 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/Makefile
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h

-- 
2.19.1

[PATCH bpf-next v2 01/16] xsk: Add API to check for available entries in FQ

[Maxim Mikityanskiy]

Add a function that checks whether the Fill Ring has the specified
amount of descriptors available. It will be useful for mlx5e that wants
to check in advance, whether it can allocate a bulk of RX descriptors,
to get the best performance.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 include/net/xdp_sock.h | 21 +++++++++++++++++++++
 net/xdp/xsk.c          |  6 ++++++
 net/xdp/xsk_queue.h    | 14 ++++++++++++++
 3 files changed, 41 insertions(+)

diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h
index d074b6d60f8a..1acddcbda236 100644
--- a/include/net/xdp_sock.h
+++ b/include/net/xdp_sock.h
@@ -77,6 +77,7 @@ int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp);
 void xsk_flush(struct xdp_sock *xs);
 bool xsk_is_setup_for_bpf_map(struct xdp_sock *xs);
 /* Used from netdev driver */
+bool xsk_umem_has_addrs(struct xdp_umem *umem, u32 cnt);
 u64 *xsk_umem_peek_addr(struct xdp_umem *umem, u64 *addr);
 void xsk_umem_discard_addr(struct xdp_umem *umem);
 void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries);
@@ -99,6 +100,16 @@ static inline dma_addr_t xdp_umem_get_dma(struct xdp_umem *umem, u64 addr)
 }
 
 /* Reuse-queue aware version of FILL queue helpers */
+static inline bool xsk_umem_has_addrs_rq(struct xdp_umem *umem, u32 cnt)
+{
+	struct xdp_umem_fq_reuse *rq = umem->fq_reuse;
+
+	if (rq->length >= cnt)
+		return true;
+
+	return xsk_umem_has_addrs(umem, cnt - rq->length);
+}
+
 static inline u64 *xsk_umem_peek_addr_rq(struct xdp_umem *umem, u64 *addr)
 {
 	struct xdp_umem_fq_reuse *rq = umem->fq_reuse;
@@ -146,6 +157,11 @@ static inline bool xsk_is_setup_for_bpf_map(struct xdp_sock *xs)
 	return false;
 }
 
+static inline bool xsk_umem_has_addrs(struct xdp_umem *umem, u32 cnt)
+{
+	return false;
+}
+
 static inline u64 *xsk_umem_peek_addr(struct xdp_umem *umem, u64 *addr)
 {
 	return NULL;
@@ -200,6 +216,11 @@ static inline dma_addr_t xdp_umem_get_dma(struct xdp_umem *umem, u64 addr)
 	return 0;
 }
 
+static inline bool xsk_umem_has_addrs_rq(struct xdp_umem *umem, u32 cnt)
+{
+	return false;
+}
+
 static inline u64 *xsk_umem_peek_addr_rq(struct xdp_umem *umem, u64 *addr)
 {
 	return NULL;
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index a14e8864e4fa..b68a380f50b3 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c
@@ -37,6 +37,12 @@ bool xsk_is_setup_for_bpf_map(struct xdp_sock *xs)
 		READ_ONCE(xs->umem->fq);
 }
 
+bool xsk_umem_has_addrs(struct xdp_umem *umem, u32 cnt)
+{
+	return xskq_has_addrs(umem->fq, cnt);
+}
+EXPORT_SYMBOL(xsk_umem_has_addrs);
+
 u64 *xsk_umem_peek_addr(struct xdp_umem *umem, u64 *addr)
 {
 	return xskq_peek_addr(umem->fq, addr);
diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h
index 88b9ae24658d..12b49784a6d5 100644
--- a/net/xdp/xsk_queue.h
+++ b/net/xdp/xsk_queue.h
@@ -117,6 +117,20 @@ static inline u32 xskq_nb_free(struct xsk_queue *q, u32 producer, u32 dcnt)
 	return q->nentries - (producer - q->cons_tail);
 }
 
+static inline bool xskq_has_addrs(struct xsk_queue *q, u32 cnt)
+{
+	u32 entries = q->prod_tail - q->cons_tail;
+
+	if (entries >= cnt)
+		return true;
+
+	/* Refresh the local pointer. */
+	q->prod_tail = READ_ONCE(q->ring->producer);
+	entries = q->prod_tail - q->cons_tail;
+
+	return entries >= cnt;
+}
+
 /* UMEM queue */
 
 static inline bool xskq_is_valid_addr(struct xsk_queue *q, u64 addr)
-- 
2.19.1

[PATCH bpf-next v2 02/16] xsk: Add getsockopt XDP_OPTIONS

[Maxim Mikityanskiy]

Make it possible for the application to determine whether the AF_XDP
socket is running in zero-copy mode. To achieve this, add a new
getsockopt option XDP_OPTIONS that returns flags. The only flag
supported for now is the zero-copy mode indicator.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 include/uapi/linux/if_xdp.h       |  7 +++++++
 net/xdp/xsk.c                     | 22 ++++++++++++++++++++++
 tools/include/uapi/linux/if_xdp.h |  7 +++++++
 3 files changed, 36 insertions(+)

diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
index caed8b1614ff..9ae4b4e08b68 100644
--- a/include/uapi/linux/if_xdp.h
+++ b/include/uapi/linux/if_xdp.h
@@ -46,6 +46,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_FILL_RING		5
 #define XDP_UMEM_COMPLETION_RING	6
 #define XDP_STATISTICS			7
+#define XDP_OPTIONS			8
 
 struct xdp_umem_reg {
 	__u64 addr; /* Start of packet data area */
@@ -60,6 +61,12 @@ struct xdp_statistics {
 	__u64 tx_invalid_descs; /* Dropped due to invalid descriptor */
 };
 
+struct xdp_options {
+	__u32 flags;
+};
+
+#define XDP_OPTIONS_FLAG_ZEROCOPY (1 << 0)
+
 /* Pgoff for mmaping the rings */
 #define XDP_PGOFF_RX_RING			  0
 #define XDP_PGOFF_TX_RING		 0x80000000
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index b68a380f50b3..998199109d5c 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c
@@ -650,6 +650,28 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname,
 
 		return 0;
 	}
+	case XDP_OPTIONS:
+	{
+		struct xdp_options opts;
+
+		if (len < sizeof(opts))
+			return -EINVAL;
+
+		opts.flags = 0;
+
+		mutex_lock(&xs->mutex);
+		if (xs->zc)
+			opts.flags |= XDP_OPTIONS_FLAG_ZEROCOPY;
+		mutex_unlock(&xs->mutex);
+
+		len = sizeof(opts);
+		if (copy_to_user(optval, &opts, len))
+			return -EFAULT;
+		if (put_user(len, optlen))
+			return -EFAULT;
+
+		return 0;
+	}
 	default:
 		break;
 	}
diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
index caed8b1614ff..9ae4b4e08b68 100644
--- a/tools/include/uapi/linux/if_xdp.h
+++ b/tools/include/uapi/linux/if_xdp.h
@@ -46,6 +46,7 @@ struct xdp_mmap_offsets {
 #define XDP_UMEM_FILL_RING		5
 #define XDP_UMEM_COMPLETION_RING	6
 #define XDP_STATISTICS			7
+#define XDP_OPTIONS			8
 
 struct xdp_umem_reg {
 	__u64 addr; /* Start of packet data area */
@@ -60,6 +61,12 @@ struct xdp_statistics {
 	__u64 tx_invalid_descs; /* Dropped due to invalid descriptor */
 };
 
+struct xdp_options {
+	__u32 flags;
+};
+
+#define XDP_OPTIONS_FLAG_ZEROCOPY (1 << 0)
+
 /* Pgoff for mmaping the rings */
 #define XDP_PGOFF_RX_RING			  0
 #define XDP_PGOFF_TX_RING		 0x80000000
-- 
2.19.1

Re: [PATCH bpf-next v2 02/16] xsk: Add getsockopt XDP_OPTIONS

[Björn Töpel]

On Tue, 30 Apr 2019 at 20:12, Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>
> Make it possible for the application to determine whether the AF_XDP
> socket is running in zero-copy mode. To achieve this, add a new
> getsockopt option XDP_OPTIONS that returns flags. The only flag
> supported for now is the zero-copy mode indicator.
>
> Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
> Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
> Acked-by: Saeed Mahameed <saeedm@mellanox.com>
> ---
>  include/uapi/linux/if_xdp.h       |  7 +++++++
>  net/xdp/xsk.c                     | 22 ++++++++++++++++++++++
>  tools/include/uapi/linux/if_xdp.h |  7 +++++++
>  3 files changed, 36 insertions(+)
>
> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
> index caed8b1614ff..9ae4b4e08b68 100644
> --- a/include/uapi/linux/if_xdp.h
> +++ b/include/uapi/linux/if_xdp.h
> @@ -46,6 +46,7 @@ struct xdp_mmap_offsets {
>  #define XDP_UMEM_FILL_RING             5
>  #define XDP_UMEM_COMPLETION_RING       6
>  #define XDP_STATISTICS                 7
> +#define XDP_OPTIONS                    8
>
>  struct xdp_umem_reg {
>         __u64 addr; /* Start of packet data area */
> @@ -60,6 +61,12 @@ struct xdp_statistics {
>         __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */
>  };
>
> +struct xdp_options {
> +       __u32 flags;
> +};
> +
> +#define XDP_OPTIONS_FLAG_ZEROCOPY (1 << 0)

Nit: The other flags doesn't use "FLAG" in its name, but that doesn't
really matter.

> +
>  /* Pgoff for mmaping the rings */
>  #define XDP_PGOFF_RX_RING                        0
>  #define XDP_PGOFF_TX_RING               0x80000000
> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
> index b68a380f50b3..998199109d5c 100644
> --- a/net/xdp/xsk.c
> +++ b/net/xdp/xsk.c
> @@ -650,6 +650,28 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname,
>
>                 return 0;
>         }
> +       case XDP_OPTIONS:
> +       {
> +               struct xdp_options opts;
> +
> +               if (len < sizeof(opts))
> +                       return -EINVAL;
> +
> +               opts.flags = 0;

Maybe get rid of this, in favor of "opts = {}" if the structure grows?


> +
> +               mutex_lock(&xs->mutex);
> +               if (xs->zc)
> +                       opts.flags |= XDP_OPTIONS_FLAG_ZEROCOPY;
> +               mutex_unlock(&xs->mutex);
> +
> +               len = sizeof(opts);
> +               if (copy_to_user(optval, &opts, len))
> +                       return -EFAULT;
> +               if (put_user(len, optlen))
> +                       return -EFAULT;
> +
> +               return 0;
> +       }
>         default:
>                 break;
>         }
> diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
> index caed8b1614ff..9ae4b4e08b68 100644
> --- a/tools/include/uapi/linux/if_xdp.h
> +++ b/tools/include/uapi/linux/if_xdp.h
> @@ -46,6 +46,7 @@ struct xdp_mmap_offsets {
>  #define XDP_UMEM_FILL_RING             5
>  #define XDP_UMEM_COMPLETION_RING       6
>  #define XDP_STATISTICS                 7
> +#define XDP_OPTIONS                    8
>
>  struct xdp_umem_reg {
>         __u64 addr; /* Start of packet data area */
> @@ -60,6 +61,12 @@ struct xdp_statistics {
>         __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */
>  };
>
> +struct xdp_options {
> +       __u32 flags;
> +};
> +
> +#define XDP_OPTIONS_FLAG_ZEROCOPY (1 << 0)
> +
>  /* Pgoff for mmaping the rings */
>  #define XDP_PGOFF_RX_RING                        0
>  #define XDP_PGOFF_TX_RING               0x80000000
> --
> 2.19.1
>

Re: [PATCH bpf-next v2 02/16] xsk: Add getsockopt XDP_OPTIONS

[Maxim Mikityanskiy]

On 2019-05-04 20:25, Björn Töpel wrote:
> On Tue, 30 Apr 2019 at 20:12, Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>>
>> Make it possible for the application to determine whether the AF_XDP
>> socket is running in zero-copy mode. To achieve this, add a new
>> getsockopt option XDP_OPTIONS that returns flags. The only flag
>> supported for now is the zero-copy mode indicator.
>>
>> Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
>> Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
>> Acked-by: Saeed Mahameed <saeedm@mellanox.com>
>> ---
>>   include/uapi/linux/if_xdp.h       |  7 +++++++
>>   net/xdp/xsk.c                     | 22 ++++++++++++++++++++++
>>   tools/include/uapi/linux/if_xdp.h |  7 +++++++
>>   3 files changed, 36 insertions(+)
>>
>> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
>> index caed8b1614ff..9ae4b4e08b68 100644
>> --- a/include/uapi/linux/if_xdp.h
>> +++ b/include/uapi/linux/if_xdp.h
>> @@ -46,6 +46,7 @@ struct xdp_mmap_offsets {
>>   #define XDP_UMEM_FILL_RING             5
>>   #define XDP_UMEM_COMPLETION_RING       6
>>   #define XDP_STATISTICS                 7
>> +#define XDP_OPTIONS                    8
>>
>>   struct xdp_umem_reg {
>>          __u64 addr; /* Start of packet data area */
>> @@ -60,6 +61,12 @@ struct xdp_statistics {
>>          __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */
>>   };
>>
>> +struct xdp_options {
>> +       __u32 flags;
>> +};
>> +
>> +#define XDP_OPTIONS_FLAG_ZEROCOPY (1 << 0)
> 
> Nit: The other flags doesn't use "FLAG" in its name, but that doesn't
> really matter.
> 
>> +
>>   /* Pgoff for mmaping the rings */
>>   #define XDP_PGOFF_RX_RING                        0
>>   #define XDP_PGOFF_TX_RING               0x80000000
>> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
>> index b68a380f50b3..998199109d5c 100644
>> --- a/net/xdp/xsk.c
>> +++ b/net/xdp/xsk.c
>> @@ -650,6 +650,28 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname,
>>
>>                  return 0;
>>          }
>> +       case XDP_OPTIONS:
>> +       {
>> +               struct xdp_options opts;
>> +
>> +               if (len < sizeof(opts))
>> +                       return -EINVAL;
>> +
>> +               opts.flags = 0;
> 
> Maybe get rid of this, in favor of "opts = {}" if the structure grows?

I'm OK with any of these options. Should I respin the series, or can I 
follow up with the change in RCs if the series gets to 5.2?

Alexei, is it even possible to still make changes to this series? The 
window appears closed.

> 
>> +
>> +               mutex_lock(&xs->mutex);
>> +               if (xs->zc)
>> +                       opts.flags |= XDP_OPTIONS_FLAG_ZEROCOPY;
>> +               mutex_unlock(&xs->mutex);
>> +
>> +               len = sizeof(opts);
>> +               if (copy_to_user(optval, &opts, len))
>> +                       return -EFAULT;
>> +               if (put_user(len, optlen))
>> +                       return -EFAULT;
>> +
>> +               return 0;
>> +       }
>>          default:
>>                  break;
>>          }
>> diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
>> index caed8b1614ff..9ae4b4e08b68 100644
>> --- a/tools/include/uapi/linux/if_xdp.h
>> +++ b/tools/include/uapi/linux/if_xdp.h
>> @@ -46,6 +46,7 @@ struct xdp_mmap_offsets {
>>   #define XDP_UMEM_FILL_RING             5
>>   #define XDP_UMEM_COMPLETION_RING       6
>>   #define XDP_STATISTICS                 7
>> +#define XDP_OPTIONS                    8
>>
>>   struct xdp_umem_reg {
>>          __u64 addr; /* Start of packet data area */
>> @@ -60,6 +61,12 @@ struct xdp_statistics {
>>          __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */
>>   };
>>
>> +struct xdp_options {
>> +       __u32 flags;
>> +};
>> +
>> +#define XDP_OPTIONS_FLAG_ZEROCOPY (1 << 0)
>> +
>>   /* Pgoff for mmaping the rings */
>>   #define XDP_PGOFF_RX_RING                        0
>>   #define XDP_PGOFF_TX_RING               0x80000000
>> --
>> 2.19.1
>>

Re: [PATCH bpf-next v2 02/16] xsk: Add getsockopt XDP_OPTIONS

[Alexei Starovoitov]

On Mon, May 06, 2019 at 01:45:40PM +0000, Maxim Mikityanskiy wrote:
> On 2019-05-04 20:25, Björn Töpel wrote:
> > On Tue, 30 Apr 2019 at 20:12, Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
> >>
> >> Make it possible for the application to determine whether the AF_XDP
> >> socket is running in zero-copy mode. To achieve this, add a new
> >> getsockopt option XDP_OPTIONS that returns flags. The only flag
> >> supported for now is the zero-copy mode indicator.
> >>
> >> Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
> >> Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
> >> Acked-by: Saeed Mahameed <saeedm@mellanox.com>
> >> ---
> >>   include/uapi/linux/if_xdp.h       |  7 +++++++
> >>   net/xdp/xsk.c                     | 22 ++++++++++++++++++++++
> >>   tools/include/uapi/linux/if_xdp.h |  7 +++++++
> >>   3 files changed, 36 insertions(+)
> >>
> >> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
> >> index caed8b1614ff..9ae4b4e08b68 100644
> >> --- a/include/uapi/linux/if_xdp.h
> >> +++ b/include/uapi/linux/if_xdp.h
> >> @@ -46,6 +46,7 @@ struct xdp_mmap_offsets {
> >>   #define XDP_UMEM_FILL_RING             5
> >>   #define XDP_UMEM_COMPLETION_RING       6
> >>   #define XDP_STATISTICS                 7
> >> +#define XDP_OPTIONS                    8
> >>
> >>   struct xdp_umem_reg {
> >>          __u64 addr; /* Start of packet data area */
> >> @@ -60,6 +61,12 @@ struct xdp_statistics {
> >>          __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */
> >>   };
> >>
> >> +struct xdp_options {
> >> +       __u32 flags;
> >> +};
> >> +
> >> +#define XDP_OPTIONS_FLAG_ZEROCOPY (1 << 0)
> > 
> > Nit: The other flags doesn't use "FLAG" in its name, but that doesn't
> > really matter.
> > 
> >> +
> >>   /* Pgoff for mmaping the rings */
> >>   #define XDP_PGOFF_RX_RING                        0
> >>   #define XDP_PGOFF_TX_RING               0x80000000
> >> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
> >> index b68a380f50b3..998199109d5c 100644
> >> --- a/net/xdp/xsk.c
> >> +++ b/net/xdp/xsk.c
> >> @@ -650,6 +650,28 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname,
> >>
> >>                  return 0;
> >>          }
> >> +       case XDP_OPTIONS:
> >> +       {
> >> +               struct xdp_options opts;
> >> +
> >> +               if (len < sizeof(opts))
> >> +                       return -EINVAL;
> >> +
> >> +               opts.flags = 0;
> > 
> > Maybe get rid of this, in favor of "opts = {}" if the structure grows?
> 
> I'm OK with any of these options. Should I respin the series, or can I 
> follow up with the change in RCs if the series gets to 5.2?
> 
> Alexei, is it even possible to still make changes to this series? The 
> window appears closed.

The series were not applied.
Please resubmit addressing all feedback when bpf-next reopens.
Likely in ~2 weeks.

[PATCH bpf-next v2 03/16] libbpf: Support getsockopt XDP_OPTIONS

[Maxim Mikityanskiy]

Query XDP_OPTIONS in libbpf to determine if the zero-copy mode is active
or not.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 tools/lib/bpf/xsk.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c
index 557ef8d1250d..a95b06d1f81d 100644
--- a/tools/lib/bpf/xsk.c
+++ b/tools/lib/bpf/xsk.c
@@ -67,6 +67,7 @@ struct xsk_socket {
 	int xsks_map_fd;
 	__u32 queue_id;
 	char ifname[IFNAMSIZ];
+	bool zc;
 };
 
 struct xsk_nl_info {
@@ -526,6 +527,7 @@ int xsk_socket__create(struct xsk_socket **xsk_ptr, const char *ifname,
 {
 	struct sockaddr_xdp sxdp = {};
 	struct xdp_mmap_offsets off;
+	struct xdp_options opts;
 	struct xsk_socket *xsk;
 	socklen_t optlen;
 	void *map;
@@ -643,6 +645,15 @@ int xsk_socket__create(struct xsk_socket **xsk_ptr, const char *ifname,
 		goto out_mmap_tx;
 	}
 
+	optlen = sizeof(opts);
+	err = getsockopt(xsk->fd, SOL_XDP, XDP_OPTIONS, &opts, &optlen);
+	if (err) {
+		err = -errno;
+		goto out_mmap_tx;
+	}
+
+	xsk->zc = opts.flags & XDP_OPTIONS_FLAG_ZEROCOPY;
+
 	if (!(xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)) {
 		err = xsk_setup_xdp_prog(xsk);
 		if (err)
-- 
2.19.1

[PATCH bpf-next v2 04/16] xsk: Extend channels to support combined XSK/non-XSK traffic

[Maxim Mikityanskiy]

Currently, the drivers that implement AF_XDP zero-copy support (e.g.,
i40e) switch the channel into a different mode when an XSK is opened. It
causes some issues that have to be taken into account. For example, RSS
needs to be reconfigured to skip the XSK-enabled channels, or the XDP
program should filter out traffic not intended for that socket and
XDP_PASS it with an additional copy. As nothing validates or forces the
proper configuration, it's easy to have packets drops, when they get
into an XSK by mistake, and, in fact, it's the default configuration.
There has to be some tool to have RSS reconfigured on each socket open
and close event, but such a tool is problematic to implement, because no
one reports these events, and it's race-prone.

This commit extends XSK to support both kinds of traffic (XSK and
non-XSK) in the same channel. It implies having two RX queues in
XSK-enabled channels: one for the regular traffic, and the other for
XSK. It solves the problem with RSS: the default configuration just
works without the need to manually reconfigure RSS or to perform some
possibly complicated filtering in the XDP layer. It makes it easy to run
both AF_XDP and regular sockets on the same machine. In the XDP program,
the QID's most significant bit will serve as a flag to indicate whether
it's the XSK queue or not. The extension is compatible with the legacy
configuration, so if one wants to run the legacy mode, they can
reconfigure RSS and ignore the flag in the XDP program (implemented in
the reference XDP program in libbpf). mlx5e will support this extension.

A single XDP program can run both with drivers supporting or not
supporting this extension. The xdpsock sample and libbpf are updated
accordingly.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 include/uapi/linux/if_xdp.h       |  11 +++
 net/xdp/xsk.c                     |   5 +-
 samples/bpf/xdpsock_user.c        |  10 ++-
 tools/include/uapi/linux/if_xdp.h |  11 +++
 tools/lib/bpf/xsk.c               | 116 ++++++++++++++++++++++--------
 tools/lib/bpf/xsk.h               |   4 ++
 6 files changed, 126 insertions(+), 31 deletions(-)

diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
index 9ae4b4e08b68..cf6ff1ecc6bd 100644
--- a/include/uapi/linux/if_xdp.h
+++ b/include/uapi/linux/if_xdp.h
@@ -82,4 +82,15 @@ struct xdp_desc {
 
 /* UMEM descriptor is __u64 */
 
+/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
+ * this flag bit in the queue index to distinguish between two RQs of the same
+ * channel.
+ */
+#define XDP_QID_FLAG_XSKRQ (1 << 31)
+
+static inline __u32 xdp_qid_get_channel(__u32 qid)
+{
+	return qid & ~XDP_QID_FLAG_XSKRQ;
+}
+
 #endif /* _LINUX_IF_XDP_H */
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index 998199109d5c..114ba17acb09 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c
@@ -104,9 +104,12 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
 
 int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
 {
+	struct xdp_rxq_info *rxq = xdp->rxq;
+	u32 channel = xdp_qid_get_channel(rxq->queue_index);
 	u32 len;
 
-	if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
+	if (xs->dev != rxq->dev || xs->queue_id != channel ||
+	    xs->zc != (rxq->mem.type == MEM_TYPE_ZERO_COPY))
 		return -EINVAL;
 
 	len = xdp->data_end - xdp->data;
diff --git a/samples/bpf/xdpsock_user.c b/samples/bpf/xdpsock_user.c
index d08ee1ab7bb4..a6b13025ee79 100644
--- a/samples/bpf/xdpsock_user.c
+++ b/samples/bpf/xdpsock_user.c
@@ -62,6 +62,7 @@ enum benchmark_type {
 
 static enum benchmark_type opt_bench = BENCH_RXDROP;
 static u32 opt_xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
+static u32 opt_libbpf_flags;
 static const char *opt_if = "";
 static int opt_ifindex;
 static int opt_queue;
@@ -306,7 +307,7 @@ static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem)
 	xsk->umem = umem;
 	cfg.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
 	cfg.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS;
-	cfg.libbpf_flags = 0;
+	cfg.libbpf_flags = opt_libbpf_flags;
 	cfg.xdp_flags = opt_xdp_flags;
 	cfg.bind_flags = opt_xdp_bind_flags;
 	ret = xsk_socket__create(&xsk->xsk, opt_if, opt_queue, umem->umem,
@@ -346,6 +347,7 @@ static struct option long_options[] = {
 	{"interval", required_argument, 0, 'n'},
 	{"zero-copy", no_argument, 0, 'z'},
 	{"copy", no_argument, 0, 'c'},
+	{"combined", no_argument, 0, 'C'},
 	{0, 0, 0, 0}
 };
 
@@ -365,6 +367,7 @@ static void usage(const char *prog)
 		"  -n, --interval=n	Specify statistics update interval (default 1 sec).\n"
 		"  -z, --zero-copy      Force zero-copy mode.\n"
 		"  -c, --copy           Force copy mode.\n"
+		"  -C, --combined       Driver supports combined XSK and non-XSK traffic in a channel.\n"
 		"\n";
 	fprintf(stderr, str, prog);
 	exit(EXIT_FAILURE);
@@ -377,7 +380,7 @@ static void parse_command_line(int argc, char **argv)
 	opterr = 0;
 
 	for (;;) {
-		c = getopt_long(argc, argv, "Frtli:q:psSNn:cz", long_options,
+		c = getopt_long(argc, argv, "Frtli:q:psSNn:czC", long_options,
 				&option_index);
 		if (c == -1)
 			break;
@@ -420,6 +423,9 @@ static void parse_command_line(int argc, char **argv)
 		case 'F':
 			opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
 			break;
+		case 'C':
+			opt_libbpf_flags |= XSK_LIBBPF_FLAGS__COMBINED_CHANNELS;
+			break;
 		default:
 			usage(basename(argv[0]));
 		}
diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
index 9ae4b4e08b68..cf6ff1ecc6bd 100644
--- a/tools/include/uapi/linux/if_xdp.h
+++ b/tools/include/uapi/linux/if_xdp.h
@@ -82,4 +82,15 @@ struct xdp_desc {
 
 /* UMEM descriptor is __u64 */
 
+/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
+ * this flag bit in the queue index to distinguish between two RQs of the same
+ * channel.
+ */
+#define XDP_QID_FLAG_XSKRQ (1 << 31)
+
+static inline __u32 xdp_qid_get_channel(__u32 qid)
+{
+	return qid & ~XDP_QID_FLAG_XSKRQ;
+}
+
 #endif /* _LINUX_IF_XDP_H */
diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c
index a95b06d1f81d..969dfd856039 100644
--- a/tools/lib/bpf/xsk.c
+++ b/tools/lib/bpf/xsk.c
@@ -76,6 +76,12 @@ struct xsk_nl_info {
 	int fd;
 };
 
+enum qidconf {
+	QIDCONF_REGULAR,
+	QIDCONF_XSK,
+	QIDCONF_XSK_COMBINED,
+};
+
 /* For 32-bit systems, we need to use mmap2 as the offsets are 64-bit.
  * Unfortunately, it is not part of glibc.
  */
@@ -139,7 +145,7 @@ static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg,
 		return 0;
 	}
 
-	if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)
+	if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS_MASK)
 		return -EINVAL;
 
 	cfg->rx_size = usr_cfg->rx_size;
@@ -267,44 +273,93 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk)
 	/* This is the C-program:
 	 * SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)
 	 * {
-	 *     int *qidconf, index = ctx->rx_queue_index;
+	 *     int *qidconf, qc;
+	 *     int index = ctx->rx_queue_index & ~(1 << 31);
+	 *     bool is_xskrq = ctx->rx_queue_index & (1 << 31);
 	 *
-	 *     // A set entry here means that the correspnding queue_id
-	 *     // has an active AF_XDP socket bound to it.
+	 *     // A set entry here means that the corresponding queue_id
+	 *     // has an active AF_XDP socket bound to it. Value 2 means
+	 *     // it's zero-copy multi-RQ mode.
 	 *     qidconf = bpf_map_lookup_elem(&qidconf_map, &index);
 	 *     if (!qidconf)
 	 *         return XDP_ABORTED;
 	 *
-	 *     if (*qidconf)
+	 *     qc = *qidconf;
+	 *
+	 *     if (qc == 2)
+	 *         qc = is_xskrq ? 1 : 0;
+	 *
+	 *     switch (qc) {
+	 *     case 0:
+	 *         return XDP_PASS;
+	 *     case 1:
 	 *         return bpf_redirect_map(&xsks_map, index, 0);
+	 *     }
 	 *
-	 *     return XDP_PASS;
+	 *     return XDP_ABORTED;
 	 * }
 	 */
 	struct bpf_insn prog[] = {
-		/* r1 = *(u32 *)(r1 + 16) */
-		BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 16),
-		/* *(u32 *)(r10 - 4) = r1 */
-		BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_1, -4),
-		BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
-		BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
-		BPF_LD_MAP_FD(BPF_REG_1, xsk->qidconf_map_fd),
+		/* Load index. */
+		/* r6 = *(u32 *)(r1 + 16) */
+		BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_ARG1, 16),
+		/* w7 = w6 */
+		BPF_MOV32_REG(BPF_REG_7, BPF_REG_6),
+		/* w7 &= 2147483647 */
+		BPF_ALU32_IMM(BPF_AND, BPF_REG_7, ~XDP_QID_FLAG_XSKRQ),
+		/* *(u32 *)(r10 - 4) = r7 */
+		BPF_STX_MEM(BPF_W, BPF_REG_FP, BPF_REG_7, -4),
+
+		/* Call bpf_map_lookup_elem. */
+		/* r2 = r10 */
+		BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP),
+		/* r2 += -4 */
+		BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -4),
+		/* r1 = qidconf_map ll */
+		BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->qidconf_map_fd),
+		/* call 1 */
 		BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
-		BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
-		BPF_MOV32_IMM(BPF_REG_0, 0),
-		/* if r1 == 0 goto +8 */
-		BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 8),
-		BPF_MOV32_IMM(BPF_REG_0, 2),
-		/* r1 = *(u32 *)(r1 + 0) */
-		BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 0),
-		/* if r1 == 0 goto +5 */
-		BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 5),
-		/* r2 = *(u32 *)(r10 - 4) */
-		BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),
-		BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_10, -4),
+
+		/* Check the return value. */
+		/* if r0 == 0 goto +14 */
+		BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 14),
+
+		/* Check qc == QIDCONF_XSK_COMBINED. */
+		/* r6 >>= 31 */
+		BPF_ALU64_IMM(BPF_RSH, BPF_REG_6, 31),
+		/* r1 = *(u32 *)(r0 + 0) */
+		BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
+		/* if r1 == 2 goto +1 */
+		BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, QIDCONF_XSK_COMBINED, 1),
+
+		/* qc != QIDCONF_XSK_COMBINED */
+		/* r6 = r1 */
+		BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+
+		/* switch (qc) */
+		/* w0 = 2 */
+		BPF_MOV32_IMM(BPF_REG_0, XDP_PASS),
+		/* if w6 == 0 goto +8 */
+		BPF_JMP32_IMM(BPF_JEQ, BPF_REG_6, QIDCONF_REGULAR, 8),
+		/* if w6 != 1 goto +6 */
+		BPF_JMP32_IMM(BPF_JNE, BPF_REG_6, QIDCONF_XSK, 6),
+
+		/* Call bpf_redirect_map. */
+		/* r1 = xsks_map ll */
+		BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->xsks_map_fd),
+		/* w2 = w7 */
+		BPF_MOV32_REG(BPF_REG_ARG2, BPF_REG_7),
+		/* w3 = 0 */
 		BPF_MOV32_IMM(BPF_REG_3, 0),
+		/* call 51 */
 		BPF_EMIT_CALL(BPF_FUNC_redirect_map),
-		/* The jumps are to this instruction */
+		/* exit */
+		BPF_EXIT_INSN(),
+
+		/* XDP_ABORTED */
+		/* w0 = 0 */
+		BPF_MOV32_IMM(BPF_REG_0, XDP_ABORTED),
+		/* exit */
 		BPF_EXIT_INSN(),
 	};
 	size_t insns_cnt = sizeof(prog) / sizeof(struct bpf_insn);
@@ -483,6 +538,7 @@ static int xsk_update_bpf_maps(struct xsk_socket *xsk, int qidconf_value,
 
 static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
 {
+	int qidconf_value = QIDCONF_XSK;
 	bool prog_attached = false;
 	__u32 prog_id = 0;
 	int err;
@@ -505,7 +561,11 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
 		xsk->prog_fd = bpf_prog_get_fd_by_id(prog_id);
 	}
 
-	err = xsk_update_bpf_maps(xsk, true, xsk->fd);
+	if (xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
+		if (xsk->zc)
+			qidconf_value = QIDCONF_XSK_COMBINED;
+
+	err = xsk_update_bpf_maps(xsk, qidconf_value, xsk->fd);
 	if (err)
 		goto out_load;
 
@@ -717,7 +777,7 @@ void xsk_socket__delete(struct xsk_socket *xsk)
 	if (!xsk)
 		return;
 
-	(void)xsk_update_bpf_maps(xsk, 0, 0);
+	(void)xsk_update_bpf_maps(xsk, QIDCONF_REGULAR, 0);
 
 	optlen = sizeof(off);
 	err = getsockopt(xsk->fd, SOL_XDP, XDP_MMAP_OFFSETS, &off, &optlen);
diff --git a/tools/lib/bpf/xsk.h b/tools/lib/bpf/xsk.h
index 82ea71a0f3ec..be26a2423c04 100644
--- a/tools/lib/bpf/xsk.h
+++ b/tools/lib/bpf/xsk.h
@@ -180,6 +180,10 @@ struct xsk_umem_config {
 
 /* Flags for the libbpf_flags field. */
 #define XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD (1 << 0)
+#define XSK_LIBBPF_FLAGS__COMBINED_CHANNELS (1 << 1)
+#define XSK_LIBBPF_FLAGS_MASK ( \
+	XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD | \
+	XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
 
 struct xsk_socket_config {
 	__u32 rx_size;
-- 
2.19.1

Re: [PATCH bpf-next v2 04/16] xsk: Extend channels to support combined XSK/non-XSK traffic

[Björn Töpel]

On Tue, 30 Apr 2019 at 20:12, Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>
> Currently, the drivers that implement AF_XDP zero-copy support (e.g.,
> i40e) switch the channel into a different mode when an XSK is opened. It
> causes some issues that have to be taken into account. For example, RSS
> needs to be reconfigured to skip the XSK-enabled channels, or the XDP
> program should filter out traffic not intended for that socket and
> XDP_PASS it with an additional copy. As nothing validates or forces the
> proper configuration, it's easy to have packets drops, when they get
> into an XSK by mistake, and, in fact, it's the default configuration.
> There has to be some tool to have RSS reconfigured on each socket open
> and close event, but such a tool is problematic to implement, because no
> one reports these events, and it's race-prone.
>
> This commit extends XSK to support both kinds of traffic (XSK and
> non-XSK) in the same channel. It implies having two RX queues in
> XSK-enabled channels: one for the regular traffic, and the other for
> XSK. It solves the problem with RSS: the default configuration just
> works without the need to manually reconfigure RSS or to perform some
> possibly complicated filtering in the XDP layer. It makes it easy to run
> both AF_XDP and regular sockets on the same machine. In the XDP program,
> the QID's most significant bit will serve as a flag to indicate whether
> it's the XSK queue or not. The extension is compatible with the legacy
> configuration, so if one wants to run the legacy mode, they can
> reconfigure RSS and ignore the flag in the XDP program (implemented in
> the reference XDP program in libbpf). mlx5e will support this extension.
>
> A single XDP program can run both with drivers supporting or not
> supporting this extension. The xdpsock sample and libbpf are updated
> accordingly.
>

I'm still not a fan of this, or maybe I'm not following you. It makes
it more complex and even harder to use. Let's take a look at the
kernel nomenclature. "ethtool" uses netdevs and channels. A channel is
a Rx queue or a Tx queue. In AF_XDP we call the channel a queue, which
is what kernel uses internally (netdev_rx_queue, netdev_queue).

Today, AF_XDP can attach to an existing queue for ingress. (On the
egress side, we're using "a queue", but the "XDP queue". XDP has these
"shadow queues" which are separated from the netdev. This is a bit
messy, and we can't really configure them. I believe Jakub has some
ideas here. :-) For now, let's leave egress aside.)

If an application would like to get all the traffic from a netdev,
it'll create an equal amout of sockets as the queues and bind to the
queues. Yes, even the queues in the RSS  set.

What you would like (I think):
a) is a way of spawning a new queue for a netdev, that is not part of
the stack and/or RSS set
b) steering traffic to that queue using a configuration mechanism (tc?
some yet to be hacked BPF configuration hook?)

With your mechanism you're doing this in contrived way. This makes the
existing AF_XDP model *more* complex/hard(er) to use.

How do you steer traffic to this dual-channel RQ? So you have a netdev
receiving on all queues. Then, e.g., the last queue is a "dual
channel" queue that can receive traffic from some other filter. How do
you use it?



Björn

> Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
> Acked-by: Saeed Mahameed <saeedm@mellanox.com>
> ---
>  include/uapi/linux/if_xdp.h       |  11 +++
>  net/xdp/xsk.c                     |   5 +-
>  samples/bpf/xdpsock_user.c        |  10 ++-
>  tools/include/uapi/linux/if_xdp.h |  11 +++
>  tools/lib/bpf/xsk.c               | 116 ++++++++++++++++++++++--------
>  tools/lib/bpf/xsk.h               |   4 ++
>  6 files changed, 126 insertions(+), 31 deletions(-)
>
> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
> --- a/include/uapi/linux/if_xdp.h
> +++ b/include/uapi/linux/if_xdp.h
> @@ -82,4 +82,15 @@ struct xdp_desc {
>
>  /* UMEM descriptor is __u64 */
>
> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
> + * this flag bit in the queue index to distinguish between two RQs of the same
> + * channel.
> + */
> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
> +
> +static inline __u32 xdp_qid_get_channel(__u32 qid)
> +{
> +       return qid & ~XDP_QID_FLAG_XSKRQ;
> +}
> +
>  #endif /* _LINUX_IF_XDP_H */
> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
> index 998199109d5c..114ba17acb09 100644
> --- a/net/xdp/xsk.c
> +++ b/net/xdp/xsk.c
> @@ -104,9 +104,12 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
>
>  int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
>  {
> +       struct xdp_rxq_info *rxq = xdp->rxq;
> +       u32 channel = xdp_qid_get_channel(rxq->queue_index);
>         u32 len;
>
> -       if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
> +       if (xs->dev != rxq->dev || xs->queue_id != channel ||
> +           xs->zc != (rxq->mem.type == MEM_TYPE_ZERO_COPY))
>                 return -EINVAL;
>
>         len = xdp->data_end - xdp->data;
> diff --git a/samples/bpf/xdpsock_user.c b/samples/bpf/xdpsock_user.c
> index d08ee1ab7bb4..a6b13025ee79 100644
> --- a/samples/bpf/xdpsock_user.c
> +++ b/samples/bpf/xdpsock_user.c
> @@ -62,6 +62,7 @@ enum benchmark_type {
>
>  static enum benchmark_type opt_bench = BENCH_RXDROP;
>  static u32 opt_xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
> +static u32 opt_libbpf_flags;
>  static const char *opt_if = "";
>  static int opt_ifindex;
>  static int opt_queue;
> @@ -306,7 +307,7 @@ static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem)
>         xsk->umem = umem;
>         cfg.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
>         cfg.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS;
> -       cfg.libbpf_flags = 0;
> +       cfg.libbpf_flags = opt_libbpf_flags;
>         cfg.xdp_flags = opt_xdp_flags;
>         cfg.bind_flags = opt_xdp_bind_flags;
>         ret = xsk_socket__create(&xsk->xsk, opt_if, opt_queue, umem->umem,
> @@ -346,6 +347,7 @@ static struct option long_options[] = {
>         {"interval", required_argument, 0, 'n'},
>         {"zero-copy", no_argument, 0, 'z'},
>         {"copy", no_argument, 0, 'c'},
> +       {"combined", no_argument, 0, 'C'},
>         {0, 0, 0, 0}
>  };
>
> @@ -365,6 +367,7 @@ static void usage(const char *prog)
>                 "  -n, --interval=n     Specify statistics update interval (default 1 sec).\n"
>                 "  -z, --zero-copy      Force zero-copy mode.\n"
>                 "  -c, --copy           Force copy mode.\n"
> +               "  -C, --combined       Driver supports combined XSK and non-XSK traffic in a channel.\n"
>                 "\n";
>         fprintf(stderr, str, prog);
>         exit(EXIT_FAILURE);
> @@ -377,7 +380,7 @@ static void parse_command_line(int argc, char **argv)
>         opterr = 0;
>
>         for (;;) {
> -               c = getopt_long(argc, argv, "Frtli:q:psSNn:cz", long_options,
> +               c = getopt_long(argc, argv, "Frtli:q:psSNn:czC", long_options,
>                                 &option_index);
>                 if (c == -1)
>                         break;
> @@ -420,6 +423,9 @@ static void parse_command_line(int argc, char **argv)
>                 case 'F':
>                         opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
>                         break;
> +               case 'C':
> +                       opt_libbpf_flags |= XSK_LIBBPF_FLAGS__COMBINED_CHANNELS;
> +                       break;
>                 default:
>                         usage(basename(argv[0]));
>                 }
> diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
> --- a/tools/include/uapi/linux/if_xdp.h
> +++ b/tools/include/uapi/linux/if_xdp.h
> @@ -82,4 +82,15 @@ struct xdp_desc {
>
>  /* UMEM descriptor is __u64 */
>
> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
> + * this flag bit in the queue index to distinguish between two RQs of the same
> + * channel.
> + */
> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
> +
> +static inline __u32 xdp_qid_get_channel(__u32 qid)
> +{
> +       return qid & ~XDP_QID_FLAG_XSKRQ;
> +}
> +
>  #endif /* _LINUX_IF_XDP_H */
> diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c
> index a95b06d1f81d..969dfd856039 100644
> --- a/tools/lib/bpf/xsk.c
> +++ b/tools/lib/bpf/xsk.c
> @@ -76,6 +76,12 @@ struct xsk_nl_info {
>         int fd;
>  };
>
> +enum qidconf {
> +       QIDCONF_REGULAR,
> +       QIDCONF_XSK,
> +       QIDCONF_XSK_COMBINED,
> +};
> +
>  /* For 32-bit systems, we need to use mmap2 as the offsets are 64-bit.
>   * Unfortunately, it is not part of glibc.
>   */
> @@ -139,7 +145,7 @@ static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg,
>                 return 0;
>         }
>
> -       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)
> +       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS_MASK)
>                 return -EINVAL;
>
>         cfg->rx_size = usr_cfg->rx_size;
> @@ -267,44 +273,93 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk)
>         /* This is the C-program:
>          * SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)
>          * {
> -        *     int *qidconf, index = ctx->rx_queue_index;
> +        *     int *qidconf, qc;
> +        *     int index = ctx->rx_queue_index & ~(1 << 31);
> +        *     bool is_xskrq = ctx->rx_queue_index & (1 << 31);
>          *
> -        *     // A set entry here means that the correspnding queue_id
> -        *     // has an active AF_XDP socket bound to it.
> +        *     // A set entry here means that the corresponding queue_id
> +        *     // has an active AF_XDP socket bound to it. Value 2 means
> +        *     // it's zero-copy multi-RQ mode.
>          *     qidconf = bpf_map_lookup_elem(&qidconf_map, &index);
>          *     if (!qidconf)
>          *         return XDP_ABORTED;
>          *
> -        *     if (*qidconf)
> +        *     qc = *qidconf;
> +        *
> +        *     if (qc == 2)
> +        *         qc = is_xskrq ? 1 : 0;
> +        *
> +        *     switch (qc) {
> +        *     case 0:
> +        *         return XDP_PASS;
> +        *     case 1:
>          *         return bpf_redirect_map(&xsks_map, index, 0);
> +        *     }
>          *
> -        *     return XDP_PASS;
> +        *     return XDP_ABORTED;
>          * }
>          */
>         struct bpf_insn prog[] = {
> -               /* r1 = *(u32 *)(r1 + 16) */
> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 16),
> -               /* *(u32 *)(r10 - 4) = r1 */
> -               BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_1, -4),
> -               BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
> -               BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->qidconf_map_fd),
> +               /* Load index. */
> +               /* r6 = *(u32 *)(r1 + 16) */
> +               BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_ARG1, 16),
> +               /* w7 = w6 */
> +               BPF_MOV32_REG(BPF_REG_7, BPF_REG_6),
> +               /* w7 &= 2147483647 */
> +               BPF_ALU32_IMM(BPF_AND, BPF_REG_7, ~XDP_QID_FLAG_XSKRQ),
> +               /* *(u32 *)(r10 - 4) = r7 */
> +               BPF_STX_MEM(BPF_W, BPF_REG_FP, BPF_REG_7, -4),
> +
> +               /* Call bpf_map_lookup_elem. */
> +               /* r2 = r10 */
> +               BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP),
> +               /* r2 += -4 */
> +               BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -4),
> +               /* r1 = qidconf_map ll */
> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->qidconf_map_fd),
> +               /* call 1 */
>                 BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
> -               BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
> -               BPF_MOV32_IMM(BPF_REG_0, 0),
> -               /* if r1 == 0 goto +8 */
> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 8),
> -               BPF_MOV32_IMM(BPF_REG_0, 2),
> -               /* r1 = *(u32 *)(r1 + 0) */
> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 0),
> -               /* if r1 == 0 goto +5 */
> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 5),
> -               /* r2 = *(u32 *)(r10 - 4) */
> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),
> -               BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_10, -4),
> +
> +               /* Check the return value. */
> +               /* if r0 == 0 goto +14 */
> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 14),
> +
> +               /* Check qc == QIDCONF_XSK_COMBINED. */
> +               /* r6 >>= 31 */
> +               BPF_ALU64_IMM(BPF_RSH, BPF_REG_6, 31),
> +               /* r1 = *(u32 *)(r0 + 0) */
> +               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
> +               /* if r1 == 2 goto +1 */
> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, QIDCONF_XSK_COMBINED, 1),
> +
> +               /* qc != QIDCONF_XSK_COMBINED */
> +               /* r6 = r1 */
> +               BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
> +
> +               /* switch (qc) */
> +               /* w0 = 2 */
> +               BPF_MOV32_IMM(BPF_REG_0, XDP_PASS),
> +               /* if w6 == 0 goto +8 */
> +               BPF_JMP32_IMM(BPF_JEQ, BPF_REG_6, QIDCONF_REGULAR, 8),
> +               /* if w6 != 1 goto +6 */
> +               BPF_JMP32_IMM(BPF_JNE, BPF_REG_6, QIDCONF_XSK, 6),
> +
> +               /* Call bpf_redirect_map. */
> +               /* r1 = xsks_map ll */
> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->xsks_map_fd),
> +               /* w2 = w7 */
> +               BPF_MOV32_REG(BPF_REG_ARG2, BPF_REG_7),
> +               /* w3 = 0 */
>                 BPF_MOV32_IMM(BPF_REG_3, 0),
> +               /* call 51 */
>                 BPF_EMIT_CALL(BPF_FUNC_redirect_map),
> -               /* The jumps are to this instruction */
> +               /* exit */
> +               BPF_EXIT_INSN(),
> +
> +               /* XDP_ABORTED */
> +               /* w0 = 0 */
> +               BPF_MOV32_IMM(BPF_REG_0, XDP_ABORTED),
> +               /* exit */
>                 BPF_EXIT_INSN(),
>         };
>         size_t insns_cnt = sizeof(prog) / sizeof(struct bpf_insn);
> @@ -483,6 +538,7 @@ static int xsk_update_bpf_maps(struct xsk_socket *xsk, int qidconf_value,
>
>  static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
>  {
> +       int qidconf_value = QIDCONF_XSK;
>         bool prog_attached = false;
>         __u32 prog_id = 0;
>         int err;
> @@ -505,7 +561,11 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
>                 xsk->prog_fd = bpf_prog_get_fd_by_id(prog_id);
>         }
>
> -       err = xsk_update_bpf_maps(xsk, true, xsk->fd);
> +       if (xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
> +               if (xsk->zc)
> +                       qidconf_value = QIDCONF_XSK_COMBINED;
> +
> +       err = xsk_update_bpf_maps(xsk, qidconf_value, xsk->fd);
>         if (err)
>                 goto out_load;
>
> @@ -717,7 +777,7 @@ void xsk_socket__delete(struct xsk_socket *xsk)
>         if (!xsk)
>                 return;
>
> -       (void)xsk_update_bpf_maps(xsk, 0, 0);
> +       (void)xsk_update_bpf_maps(xsk, QIDCONF_REGULAR, 0);
>
>         optlen = sizeof(off);
>         err = getsockopt(xsk->fd, SOL_XDP, XDP_MMAP_OFFSETS, &off, &optlen);
> diff --git a/tools/lib/bpf/xsk.h b/tools/lib/bpf/xsk.h
> index 82ea71a0f3ec..be26a2423c04 100644
> --- a/tools/lib/bpf/xsk.h
> +++ b/tools/lib/bpf/xsk.h
> @@ -180,6 +180,10 @@ struct xsk_umem_config {
>
>  /* Flags for the libbpf_flags field. */
>  #define XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD (1 << 0)
> +#define XSK_LIBBPF_FLAGS__COMBINED_CHANNELS (1 << 1)
> +#define XSK_LIBBPF_FLAGS_MASK ( \
> +       XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD | \
> +       XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
>
>  struct xsk_socket_config {
>         __u32 rx_size;
> --
> 2.19.1
>

Re: [PATCH bpf-next v2 04/16] xsk: Extend channels to support combined XSK/non-XSK traffic

[Maxim Mikityanskiy]

On 2019-05-04 20:26, Björn Töpel wrote:
> On Tue, 30 Apr 2019 at 20:12, Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>>
>> Currently, the drivers that implement AF_XDP zero-copy support (e.g.,
>> i40e) switch the channel into a different mode when an XSK is opened. It
>> causes some issues that have to be taken into account. For example, RSS
>> needs to be reconfigured to skip the XSK-enabled channels, or the XDP
>> program should filter out traffic not intended for that socket and
>> XDP_PASS it with an additional copy. As nothing validates or forces the
>> proper configuration, it's easy to have packets drops, when they get
>> into an XSK by mistake, and, in fact, it's the default configuration.
>> There has to be some tool to have RSS reconfigured on each socket open
>> and close event, but such a tool is problematic to implement, because no
>> one reports these events, and it's race-prone.
>>
>> This commit extends XSK to support both kinds of traffic (XSK and
>> non-XSK) in the same channel. It implies having two RX queues in
>> XSK-enabled channels: one for the regular traffic, and the other for
>> XSK. It solves the problem with RSS: the default configuration just
>> works without the need to manually reconfigure RSS or to perform some
>> possibly complicated filtering in the XDP layer. It makes it easy to run
>> both AF_XDP and regular sockets on the same machine. In the XDP program,
>> the QID's most significant bit will serve as a flag to indicate whether
>> it's the XSK queue or not. The extension is compatible with the legacy
>> configuration, so if one wants to run the legacy mode, they can
>> reconfigure RSS and ignore the flag in the XDP program (implemented in
>> the reference XDP program in libbpf). mlx5e will support this extension.
>>
>> A single XDP program can run both with drivers supporting or not
>> supporting this extension. The xdpsock sample and libbpf are updated
>> accordingly.
>>
> 
> I'm still not a fan of this, or maybe I'm not following you. It makes
> it more complex and even harder to use. Let's take a look at the
> kernel nomenclature. "ethtool" uses netdevs and channels. A channel is
> a Rx queue or a Tx queue.

There are also combined channels that consist of an RX and a TX queue. 
mlx5e has only this kind of channels. For us, a channel is a set of 
queues "pinned to a CPU core" (they use the same NAPI).

> In AF_XDP we call the channel a queue, which
> is what kernel uses internally (netdev_rx_queue, netdev_queue).

You seem to agree it's a channel, right?

AF_XDP doesn't allow to configure RX queue number and TX queue number 
separately. Basically you choose a channel in AF_XDP. For some reason, 
it's referred as a queue in some places, but logically it means "channel".

> Today, AF_XDP can attach to an existing queue for ingress. (On the
> egress side, we're using "a queue", but the "XDP queue". XDP has these
> "shadow queues" which are separated from the netdev. This is a bit
> messy, and we can't really configure them. I believe Jakub has some
> ideas here. :-) For now, let's leave egress aside.)

So, XDP already has "shadow queues" for TX, so I see no problem in 
having the similar concept for AF_XDP RX.

> If an application would like to get all the traffic from a netdev,
> it'll create an equal amout of sockets as the queues and bind to the
> queues. Yes, even the queues in the RSS  set.
> 
> What you would like (I think):
> a) is a way of spawning a new queue for a netdev, that is not part of
> the stack and/or RSS set

Yes - for the simplicity sake and to make configuration easier. The only 
thing needed is to steer the traffic and to open an AF_XDP socket on 
channel X. We don't need to care about removing the queue out of RSS, 
about finding a way to administer this (which is hard because it's racy 
if the configuration in not known in advance). So I don't agree I'm 
complicating things, my goal is to make them easier.

> b) steering traffic to that queue using a configuration mechanism (tc?
> some yet to be hacked BPF configuration hook?)

Currently, ethtool --config-ntuple is used to steer the traffic. The 
user-def argument has a bit that selects XSK RQ/regular RQ, and action 
selects a channel:

ethtool -N eth0 flow-type udp4 dst-port 4242 action 3 user-def 1

> With your mechanism you're doing this in contrived way. This makes the
> existing AF_XDP model *more* complex/hard(er) to use.

No, as I said above, some issues are eliminated with my approach, and no 
new limitations are introduced, so it makes things more universal and 
simpler to configure.

> How do you steer traffic to this dual-channel RQ?

First, there is no dual-channel RQ, a more accurate term is dual-RQ 
channel, cause now the channel contains a regular RQ and can contain an 
XSK RQ.

For the steering itself, see the ethtool command above - the user-def 
argument has a bit that selects one of two RQs.

> So you have a netdev
> receiving on all queues. Then, e.g., the last queue is a "dual
> channel" queue that can receive traffic from some other filter. How do
> you use it?

If I want to take the last (or some) channel and start using AF_XDP with 
it, I simply configure steering to the XSK RQ of that channel and open a 
socket specifying the channel number. I don't need to reconfigure RSS, 
because RSS packets go to the regular RQ of that channel and don't 
interfere with XSK.

No functionality is lost - if you don't distinguish the regular and XSK 
RQs on the XDP level, you'll get the same effect as with i40e's 
implementation. If you want to dedicate the CPU core and channel solely 
for AF_XDP, in i40e you exclude the channel from RSS, and here you can 
do exactly the same thing. So, no use case is complicated comparing to 
i40e, but there are use cases where this feature is to advantage.

I hope I explained the points you were interested in. Please ask more 
questions if there is still something that I should clarify regarding 
this topic.

Thanks,
Max

> 
> 
> Björn
> 
>> Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
>> Acked-by: Saeed Mahameed <saeedm@mellanox.com>
>> ---
>>   include/uapi/linux/if_xdp.h       |  11 +++
>>   net/xdp/xsk.c                     |   5 +-
>>   samples/bpf/xdpsock_user.c        |  10 ++-
>>   tools/include/uapi/linux/if_xdp.h |  11 +++
>>   tools/lib/bpf/xsk.c               | 116 ++++++++++++++++++++++--------
>>   tools/lib/bpf/xsk.h               |   4 ++
>>   6 files changed, 126 insertions(+), 31 deletions(-)
>>
>> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
>> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
>> --- a/include/uapi/linux/if_xdp.h
>> +++ b/include/uapi/linux/if_xdp.h
>> @@ -82,4 +82,15 @@ struct xdp_desc {
>>
>>   /* UMEM descriptor is __u64 */
>>
>> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
>> + * this flag bit in the queue index to distinguish between two RQs of the same
>> + * channel.
>> + */
>> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
>> +
>> +static inline __u32 xdp_qid_get_channel(__u32 qid)
>> +{
>> +       return qid & ~XDP_QID_FLAG_XSKRQ;
>> +}
>> +
>>   #endif /* _LINUX_IF_XDP_H */
>> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
>> index 998199109d5c..114ba17acb09 100644
>> --- a/net/xdp/xsk.c
>> +++ b/net/xdp/xsk.c
>> @@ -104,9 +104,12 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
>>
>>   int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
>>   {
>> +       struct xdp_rxq_info *rxq = xdp->rxq;
>> +       u32 channel = xdp_qid_get_channel(rxq->queue_index);
>>          u32 len;
>>
>> -       if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
>> +       if (xs->dev != rxq->dev || xs->queue_id != channel ||
>> +           xs->zc != (rxq->mem.type == MEM_TYPE_ZERO_COPY))
>>                  return -EINVAL;
>>
>>          len = xdp->data_end - xdp->data;
>> diff --git a/samples/bpf/xdpsock_user.c b/samples/bpf/xdpsock_user.c
>> index d08ee1ab7bb4..a6b13025ee79 100644
>> --- a/samples/bpf/xdpsock_user.c
>> +++ b/samples/bpf/xdpsock_user.c
>> @@ -62,6 +62,7 @@ enum benchmark_type {
>>
>>   static enum benchmark_type opt_bench = BENCH_RXDROP;
>>   static u32 opt_xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
>> +static u32 opt_libbpf_flags;
>>   static const char *opt_if = "";
>>   static int opt_ifindex;
>>   static int opt_queue;
>> @@ -306,7 +307,7 @@ static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem)
>>          xsk->umem = umem;
>>          cfg.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
>>          cfg.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS;
>> -       cfg.libbpf_flags = 0;
>> +       cfg.libbpf_flags = opt_libbpf_flags;
>>          cfg.xdp_flags = opt_xdp_flags;
>>          cfg.bind_flags = opt_xdp_bind_flags;
>>          ret = xsk_socket__create(&xsk->xsk, opt_if, opt_queue, umem->umem,
>> @@ -346,6 +347,7 @@ static struct option long_options[] = {
>>          {"interval", required_argument, 0, 'n'},
>>          {"zero-copy", no_argument, 0, 'z'},
>>          {"copy", no_argument, 0, 'c'},
>> +       {"combined", no_argument, 0, 'C'},
>>          {0, 0, 0, 0}
>>   };
>>
>> @@ -365,6 +367,7 @@ static void usage(const char *prog)
>>                  "  -n, --interval=n     Specify statistics update interval (default 1 sec).\n"
>>                  "  -z, --zero-copy      Force zero-copy mode.\n"
>>                  "  -c, --copy           Force copy mode.\n"
>> +               "  -C, --combined       Driver supports combined XSK and non-XSK traffic in a channel.\n"
>>                  "\n";
>>          fprintf(stderr, str, prog);
>>          exit(EXIT_FAILURE);
>> @@ -377,7 +380,7 @@ static void parse_command_line(int argc, char **argv)
>>          opterr = 0;
>>
>>          for (;;) {
>> -               c = getopt_long(argc, argv, "Frtli:q:psSNn:cz", long_options,
>> +               c = getopt_long(argc, argv, "Frtli:q:psSNn:czC", long_options,
>>                                  &option_index);
>>                  if (c == -1)
>>                          break;
>> @@ -420,6 +423,9 @@ static void parse_command_line(int argc, char **argv)
>>                  case 'F':
>>                          opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
>>                          break;
>> +               case 'C':
>> +                       opt_libbpf_flags |= XSK_LIBBPF_FLAGS__COMBINED_CHANNELS;
>> +                       break;
>>                  default:
>>                          usage(basename(argv[0]));
>>                  }
>> diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
>> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
>> --- a/tools/include/uapi/linux/if_xdp.h
>> +++ b/tools/include/uapi/linux/if_xdp.h
>> @@ -82,4 +82,15 @@ struct xdp_desc {
>>
>>   /* UMEM descriptor is __u64 */
>>
>> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
>> + * this flag bit in the queue index to distinguish between two RQs of the same
>> + * channel.
>> + */
>> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
>> +
>> +static inline __u32 xdp_qid_get_channel(__u32 qid)
>> +{
>> +       return qid & ~XDP_QID_FLAG_XSKRQ;
>> +}
>> +
>>   #endif /* _LINUX_IF_XDP_H */
>> diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c
>> index a95b06d1f81d..969dfd856039 100644
>> --- a/tools/lib/bpf/xsk.c
>> +++ b/tools/lib/bpf/xsk.c
>> @@ -76,6 +76,12 @@ struct xsk_nl_info {
>>          int fd;
>>   };
>>
>> +enum qidconf {
>> +       QIDCONF_REGULAR,
>> +       QIDCONF_XSK,
>> +       QIDCONF_XSK_COMBINED,
>> +};
>> +
>>   /* For 32-bit systems, we need to use mmap2 as the offsets are 64-bit.
>>    * Unfortunately, it is not part of glibc.
>>    */
>> @@ -139,7 +145,7 @@ static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg,
>>                  return 0;
>>          }
>>
>> -       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)
>> +       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS_MASK)
>>                  return -EINVAL;
>>
>>          cfg->rx_size = usr_cfg->rx_size;
>> @@ -267,44 +273,93 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk)
>>          /* This is the C-program:
>>           * SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)
>>           * {
>> -        *     int *qidconf, index = ctx->rx_queue_index;
>> +        *     int *qidconf, qc;
>> +        *     int index = ctx->rx_queue_index & ~(1 << 31);
>> +        *     bool is_xskrq = ctx->rx_queue_index & (1 << 31);
>>           *
>> -        *     // A set entry here means that the correspnding queue_id
>> -        *     // has an active AF_XDP socket bound to it.
>> +        *     // A set entry here means that the corresponding queue_id
>> +        *     // has an active AF_XDP socket bound to it. Value 2 means
>> +        *     // it's zero-copy multi-RQ mode.
>>           *     qidconf = bpf_map_lookup_elem(&qidconf_map, &index);
>>           *     if (!qidconf)
>>           *         return XDP_ABORTED;
>>           *
>> -        *     if (*qidconf)
>> +        *     qc = *qidconf;
>> +        *
>> +        *     if (qc == 2)
>> +        *         qc = is_xskrq ? 1 : 0;
>> +        *
>> +        *     switch (qc) {
>> +        *     case 0:
>> +        *         return XDP_PASS;
>> +        *     case 1:
>>           *         return bpf_redirect_map(&xsks_map, index, 0);
>> +        *     }
>>           *
>> -        *     return XDP_PASS;
>> +        *     return XDP_ABORTED;
>>           * }
>>           */
>>          struct bpf_insn prog[] = {
>> -               /* r1 = *(u32 *)(r1 + 16) */
>> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 16),
>> -               /* *(u32 *)(r10 - 4) = r1 */
>> -               BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_1, -4),
>> -               BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
>> -               BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
>> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->qidconf_map_fd),
>> +               /* Load index. */
>> +               /* r6 = *(u32 *)(r1 + 16) */
>> +               BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_ARG1, 16),
>> +               /* w7 = w6 */
>> +               BPF_MOV32_REG(BPF_REG_7, BPF_REG_6),
>> +               /* w7 &= 2147483647 */
>> +               BPF_ALU32_IMM(BPF_AND, BPF_REG_7, ~XDP_QID_FLAG_XSKRQ),
>> +               /* *(u32 *)(r10 - 4) = r7 */
>> +               BPF_STX_MEM(BPF_W, BPF_REG_FP, BPF_REG_7, -4),
>> +
>> +               /* Call bpf_map_lookup_elem. */
>> +               /* r2 = r10 */
>> +               BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP),
>> +               /* r2 += -4 */
>> +               BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -4),
>> +               /* r1 = qidconf_map ll */
>> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->qidconf_map_fd),
>> +               /* call 1 */
>>                  BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
>> -               BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
>> -               BPF_MOV32_IMM(BPF_REG_0, 0),
>> -               /* if r1 == 0 goto +8 */
>> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 8),
>> -               BPF_MOV32_IMM(BPF_REG_0, 2),
>> -               /* r1 = *(u32 *)(r1 + 0) */
>> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 0),
>> -               /* if r1 == 0 goto +5 */
>> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 5),
>> -               /* r2 = *(u32 *)(r10 - 4) */
>> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),
>> -               BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_10, -4),
>> +
>> +               /* Check the return value. */
>> +               /* if r0 == 0 goto +14 */
>> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 14),
>> +
>> +               /* Check qc == QIDCONF_XSK_COMBINED. */
>> +               /* r6 >>= 31 */
>> +               BPF_ALU64_IMM(BPF_RSH, BPF_REG_6, 31),
>> +               /* r1 = *(u32 *)(r0 + 0) */
>> +               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
>> +               /* if r1 == 2 goto +1 */
>> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, QIDCONF_XSK_COMBINED, 1),
>> +
>> +               /* qc != QIDCONF_XSK_COMBINED */
>> +               /* r6 = r1 */
>> +               BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
>> +
>> +               /* switch (qc) */
>> +               /* w0 = 2 */
>> +               BPF_MOV32_IMM(BPF_REG_0, XDP_PASS),
>> +               /* if w6 == 0 goto +8 */
>> +               BPF_JMP32_IMM(BPF_JEQ, BPF_REG_6, QIDCONF_REGULAR, 8),
>> +               /* if w6 != 1 goto +6 */
>> +               BPF_JMP32_IMM(BPF_JNE, BPF_REG_6, QIDCONF_XSK, 6),
>> +
>> +               /* Call bpf_redirect_map. */
>> +               /* r1 = xsks_map ll */
>> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->xsks_map_fd),
>> +               /* w2 = w7 */
>> +               BPF_MOV32_REG(BPF_REG_ARG2, BPF_REG_7),
>> +               /* w3 = 0 */
>>                  BPF_MOV32_IMM(BPF_REG_3, 0),
>> +               /* call 51 */
>>                  BPF_EMIT_CALL(BPF_FUNC_redirect_map),
>> -               /* The jumps are to this instruction */
>> +               /* exit */
>> +               BPF_EXIT_INSN(),
>> +
>> +               /* XDP_ABORTED */
>> +               /* w0 = 0 */
>> +               BPF_MOV32_IMM(BPF_REG_0, XDP_ABORTED),
>> +               /* exit */
>>                  BPF_EXIT_INSN(),
>>          };
>>          size_t insns_cnt = sizeof(prog) / sizeof(struct bpf_insn);
>> @@ -483,6 +538,7 @@ static int xsk_update_bpf_maps(struct xsk_socket *xsk, int qidconf_value,
>>
>>   static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
>>   {
>> +       int qidconf_value = QIDCONF_XSK;
>>          bool prog_attached = false;
>>          __u32 prog_id = 0;
>>          int err;
>> @@ -505,7 +561,11 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
>>                  xsk->prog_fd = bpf_prog_get_fd_by_id(prog_id);
>>          }
>>
>> -       err = xsk_update_bpf_maps(xsk, true, xsk->fd);
>> +       if (xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
>> +               if (xsk->zc)
>> +                       qidconf_value = QIDCONF_XSK_COMBINED;
>> +
>> +       err = xsk_update_bpf_maps(xsk, qidconf_value, xsk->fd);
>>          if (err)
>>                  goto out_load;
>>
>> @@ -717,7 +777,7 @@ void xsk_socket__delete(struct xsk_socket *xsk)
>>          if (!xsk)
>>                  return;
>>
>> -       (void)xsk_update_bpf_maps(xsk, 0, 0);
>> +       (void)xsk_update_bpf_maps(xsk, QIDCONF_REGULAR, 0);
>>
>>          optlen = sizeof(off);
>>          err = getsockopt(xsk->fd, SOL_XDP, XDP_MMAP_OFFSETS, &off, &optlen);
>> diff --git a/tools/lib/bpf/xsk.h b/tools/lib/bpf/xsk.h
>> index 82ea71a0f3ec..be26a2423c04 100644
>> --- a/tools/lib/bpf/xsk.h
>> +++ b/tools/lib/bpf/xsk.h
>> @@ -180,6 +180,10 @@ struct xsk_umem_config {
>>
>>   /* Flags for the libbpf_flags field. */
>>   #define XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD (1 << 0)
>> +#define XSK_LIBBPF_FLAGS__COMBINED_CHANNELS (1 << 1)
>> +#define XSK_LIBBPF_FLAGS_MASK ( \
>> +       XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD | \
>> +       XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
>>
>>   struct xsk_socket_config {
>>          __u32 rx_size;
>> --
>> 2.19.1
>>

Re: [PATCH bpf-next v2 04/16] xsk: Extend channels to support combined XSK/non-XSK traffic

[Magnus Karlsson]

On Mon, May 6, 2019 at 3:46 PM Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>
> On 2019-05-04 20:26, Björn Töpel wrote:
> > On Tue, 30 Apr 2019 at 20:12, Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
> >>
> >> Currently, the drivers that implement AF_XDP zero-copy support (e.g.,
> >> i40e) switch the channel into a different mode when an XSK is opened. It
> >> causes some issues that have to be taken into account. For example, RSS
> >> needs to be reconfigured to skip the XSK-enabled channels, or the XDP
> >> program should filter out traffic not intended for that socket and
> >> XDP_PASS it with an additional copy. As nothing validates or forces the
> >> proper configuration, it's easy to have packets drops, when they get
> >> into an XSK by mistake, and, in fact, it's the default configuration.
> >> There has to be some tool to have RSS reconfigured on each socket open
> >> and close event, but such a tool is problematic to implement, because no
> >> one reports these events, and it's race-prone.
> >>
> >> This commit extends XSK to support both kinds of traffic (XSK and
> >> non-XSK) in the same channel. It implies having two RX queues in
> >> XSK-enabled channels: one for the regular traffic, and the other for
> >> XSK. It solves the problem with RSS: the default configuration just
> >> works without the need to manually reconfigure RSS or to perform some
> >> possibly complicated filtering in the XDP layer. It makes it easy to run
> >> both AF_XDP and regular sockets on the same machine. In the XDP program,
> >> the QID's most significant bit will serve as a flag to indicate whether
> >> it's the XSK queue or not. The extension is compatible with the legacy
> >> configuration, so if one wants to run the legacy mode, they can
> >> reconfigure RSS and ignore the flag in the XDP program (implemented in
> >> the reference XDP program in libbpf). mlx5e will support this extension.
> >>
> >> A single XDP program can run both with drivers supporting or not
> >> supporting this extension. The xdpsock sample and libbpf are updated
> >> accordingly.
> >>
> >
> > I'm still not a fan of this, or maybe I'm not following you. It makes
> > it more complex and even harder to use. Let's take a look at the
> > kernel nomenclature. "ethtool" uses netdevs and channels. A channel is
> > a Rx queue or a Tx queue.
>
> There are also combined channels that consist of an RX and a TX queue.
> mlx5e has only this kind of channels. For us, a channel is a set of
> queues "pinned to a CPU core" (they use the same NAPI).
>
> > In AF_XDP we call the channel a queue, which
> > is what kernel uses internally (netdev_rx_queue, netdev_queue).
>
> You seem to agree it's a channel, right?
>
> AF_XDP doesn't allow to configure RX queue number and TX queue number
> separately. Basically you choose a channel in AF_XDP. For some reason,
> it's referred as a queue in some places, but logically it means "channel".

You can configure the Rx queue and the Tx queue separately by creating
two sockets tied to the same umem area. But if you just create one,
you are correct.

> > Today, AF_XDP can attach to an existing queue for ingress. (On the
> > egress side, we're using "a queue", but the "XDP queue". XDP has these
> > "shadow queues" which are separated from the netdev. This is a bit
> > messy, and we can't really configure them. I believe Jakub has some
> > ideas here. :-) For now, let's leave egress aside.)
>
> So, XDP already has "shadow queues" for TX, so I see no problem in
> having the similar concept for AF_XDP RX.

The question is if we would like to continue down the path of "shadow
queues" by adding even more. In the busy-poll RFC I sent out last
week, I talk about the possibility to create a new queue (set) not
tied to the napi of the regular Rx queues in order to get better
performance when using busy-poll. How would such a queue set fit into
a shadow queue set approach? When does hiding the real queues created
to support various features break and we have to expose the real queue
number? Trying to wrap my head around these questions.

Maxim, would it be possible for you to respin this set without this
feature? I like the other stuff you have implemented and think that
the rest of the common functionality should be useful for all of us.
This way you can get the AF_XDP support accepted quicker while we
debate the best way to solve the issue in this thread.

Thanks for all your work: Magnus

> > If an application would like to get all the traffic from a netdev,
> > it'll create an equal amout of sockets as the queues and bind to the
> > queues. Yes, even the queues in the RSS  set.
> >
> > What you would like (I think):
> > a) is a way of spawning a new queue for a netdev, that is not part of
> > the stack and/or RSS set
>
> Yes - for the simplicity sake and to make configuration easier. The only
> thing needed is to steer the traffic and to open an AF_XDP socket on
> channel X. We don't need to care about removing the queue out of RSS,
> about finding a way to administer this (which is hard because it's racy
> if the configuration in not known in advance). So I don't agree I'm
> complicating things, my goal is to make them easier.
>
> > b) steering traffic to that queue using a configuration mechanism (tc?
> > some yet to be hacked BPF configuration hook?)
>
> Currently, ethtool --config-ntuple is used to steer the traffic. The
> user-def argument has a bit that selects XSK RQ/regular RQ, and action
> selects a channel:
>
> ethtool -N eth0 flow-type udp4 dst-port 4242 action 3 user-def 1
>
> > With your mechanism you're doing this in contrived way. This makes the
> > existing AF_XDP model *more* complex/hard(er) to use.
>
> No, as I said above, some issues are eliminated with my approach, and no
> new limitations are introduced, so it makes things more universal and
> simpler to configure.
>
> > How do you steer traffic to this dual-channel RQ?
>
> First, there is no dual-channel RQ, a more accurate term is dual-RQ
> channel, cause now the channel contains a regular RQ and can contain an
> XSK RQ.
>
> For the steering itself, see the ethtool command above - the user-def
> argument has a bit that selects one of two RQs.
>
> > So you have a netdev
> > receiving on all queues. Then, e.g., the last queue is a "dual
> > channel" queue that can receive traffic from some other filter. How do
> > you use it?
>
> If I want to take the last (or some) channel and start using AF_XDP with
> it, I simply configure steering to the XSK RQ of that channel and open a
> socket specifying the channel number. I don't need to reconfigure RSS,
> because RSS packets go to the regular RQ of that channel and don't
> interfere with XSK.
>
> No functionality is lost - if you don't distinguish the regular and XSK
> RQs on the XDP level, you'll get the same effect as with i40e's
> implementation. If you want to dedicate the CPU core and channel solely
> for AF_XDP, in i40e you exclude the channel from RSS, and here you can
> do exactly the same thing. So, no use case is complicated comparing to
> i40e, but there are use cases where this feature is to advantage.
>
> I hope I explained the points you were interested in. Please ask more
> questions if there is still something that I should clarify regarding
> this topic.
>
> Thanks,
> Max
>
> >
> >
> > Björn
> >
> >> Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
> >> Acked-by: Saeed Mahameed <saeedm@mellanox.com>
> >> ---
> >>   include/uapi/linux/if_xdp.h       |  11 +++
> >>   net/xdp/xsk.c                     |   5 +-
> >>   samples/bpf/xdpsock_user.c        |  10 ++-
> >>   tools/include/uapi/linux/if_xdp.h |  11 +++
> >>   tools/lib/bpf/xsk.c               | 116 ++++++++++++++++++++++--------
> >>   tools/lib/bpf/xsk.h               |   4 ++
> >>   6 files changed, 126 insertions(+), 31 deletions(-)
> >>
> >> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
> >> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
> >> --- a/include/uapi/linux/if_xdp.h
> >> +++ b/include/uapi/linux/if_xdp.h
> >> @@ -82,4 +82,15 @@ struct xdp_desc {
> >>
> >>   /* UMEM descriptor is __u64 */
> >>
> >> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
> >> + * this flag bit in the queue index to distinguish between two RQs of the same
> >> + * channel.
> >> + */
> >> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
> >> +
> >> +static inline __u32 xdp_qid_get_channel(__u32 qid)
> >> +{
> >> +       return qid & ~XDP_QID_FLAG_XSKRQ;
> >> +}
> >> +
> >>   #endif /* _LINUX_IF_XDP_H */
> >> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
> >> index 998199109d5c..114ba17acb09 100644
> >> --- a/net/xdp/xsk.c
> >> +++ b/net/xdp/xsk.c
> >> @@ -104,9 +104,12 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
> >>
> >>   int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
> >>   {
> >> +       struct xdp_rxq_info *rxq = xdp->rxq;
> >> +       u32 channel = xdp_qid_get_channel(rxq->queue_index);
> >>          u32 len;
> >>
> >> -       if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
> >> +       if (xs->dev != rxq->dev || xs->queue_id != channel ||
> >> +           xs->zc != (rxq->mem.type == MEM_TYPE_ZERO_COPY))
> >>                  return -EINVAL;
> >>
> >>          len = xdp->data_end - xdp->data;
> >> diff --git a/samples/bpf/xdpsock_user.c b/samples/bpf/xdpsock_user.c
> >> index d08ee1ab7bb4..a6b13025ee79 100644
> >> --- a/samples/bpf/xdpsock_user.c
> >> +++ b/samples/bpf/xdpsock_user.c
> >> @@ -62,6 +62,7 @@ enum benchmark_type {
> >>
> >>   static enum benchmark_type opt_bench = BENCH_RXDROP;
> >>   static u32 opt_xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
> >> +static u32 opt_libbpf_flags;
> >>   static const char *opt_if = "";
> >>   static int opt_ifindex;
> >>   static int opt_queue;
> >> @@ -306,7 +307,7 @@ static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem)
> >>          xsk->umem = umem;
> >>          cfg.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
> >>          cfg.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS;
> >> -       cfg.libbpf_flags = 0;
> >> +       cfg.libbpf_flags = opt_libbpf_flags;
> >>          cfg.xdp_flags = opt_xdp_flags;
> >>          cfg.bind_flags = opt_xdp_bind_flags;
> >>          ret = xsk_socket__create(&xsk->xsk, opt_if, opt_queue, umem->umem,
> >> @@ -346,6 +347,7 @@ static struct option long_options[] = {
> >>          {"interval", required_argument, 0, 'n'},
> >>          {"zero-copy", no_argument, 0, 'z'},
> >>          {"copy", no_argument, 0, 'c'},
> >> +       {"combined", no_argument, 0, 'C'},
> >>          {0, 0, 0, 0}
> >>   };
> >>
> >> @@ -365,6 +367,7 @@ static void usage(const char *prog)
> >>                  "  -n, --interval=n     Specify statistics update interval (default 1 sec).\n"
> >>                  "  -z, --zero-copy      Force zero-copy mode.\n"
> >>                  "  -c, --copy           Force copy mode.\n"
> >> +               "  -C, --combined       Driver supports combined XSK and non-XSK traffic in a channel.\n"
> >>                  "\n";
> >>          fprintf(stderr, str, prog);
> >>          exit(EXIT_FAILURE);
> >> @@ -377,7 +380,7 @@ static void parse_command_line(int argc, char **argv)
> >>          opterr = 0;
> >>
> >>          for (;;) {
> >> -               c = getopt_long(argc, argv, "Frtli:q:psSNn:cz", long_options,
> >> +               c = getopt_long(argc, argv, "Frtli:q:psSNn:czC", long_options,
> >>                                  &option_index);
> >>                  if (c == -1)
> >>                          break;
> >> @@ -420,6 +423,9 @@ static void parse_command_line(int argc, char **argv)
> >>                  case 'F':
> >>                          opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
> >>                          break;
> >> +               case 'C':
> >> +                       opt_libbpf_flags |= XSK_LIBBPF_FLAGS__COMBINED_CHANNELS;
> >> +                       break;
> >>                  default:
> >>                          usage(basename(argv[0]));
> >>                  }
> >> diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
> >> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
> >> --- a/tools/include/uapi/linux/if_xdp.h
> >> +++ b/tools/include/uapi/linux/if_xdp.h
> >> @@ -82,4 +82,15 @@ struct xdp_desc {
> >>
> >>   /* UMEM descriptor is __u64 */
> >>
> >> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
> >> + * this flag bit in the queue index to distinguish between two RQs of the same
> >> + * channel.
> >> + */
> >> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
> >> +
> >> +static inline __u32 xdp_qid_get_channel(__u32 qid)
> >> +{
> >> +       return qid & ~XDP_QID_FLAG_XSKRQ;
> >> +}
> >> +
> >>   #endif /* _LINUX_IF_XDP_H */
> >> diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c
> >> index a95b06d1f81d..969dfd856039 100644
> >> --- a/tools/lib/bpf/xsk.c
> >> +++ b/tools/lib/bpf/xsk.c
> >> @@ -76,6 +76,12 @@ struct xsk_nl_info {
> >>          int fd;
> >>   };
> >>
> >> +enum qidconf {
> >> +       QIDCONF_REGULAR,
> >> +       QIDCONF_XSK,
> >> +       QIDCONF_XSK_COMBINED,
> >> +};
> >> +
> >>   /* For 32-bit systems, we need to use mmap2 as the offsets are 64-bit.
> >>    * Unfortunately, it is not part of glibc.
> >>    */
> >> @@ -139,7 +145,7 @@ static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg,
> >>                  return 0;
> >>          }
> >>
> >> -       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)
> >> +       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS_MASK)
> >>                  return -EINVAL;
> >>
> >>          cfg->rx_size = usr_cfg->rx_size;
> >> @@ -267,44 +273,93 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk)
> >>          /* This is the C-program:
> >>           * SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)
> >>           * {
> >> -        *     int *qidconf, index = ctx->rx_queue_index;
> >> +        *     int *qidconf, qc;
> >> +        *     int index = ctx->rx_queue_index & ~(1 << 31);
> >> +        *     bool is_xskrq = ctx->rx_queue_index & (1 << 31);
> >>           *
> >> -        *     // A set entry here means that the correspnding queue_id
> >> -        *     // has an active AF_XDP socket bound to it.
> >> +        *     // A set entry here means that the corresponding queue_id
> >> +        *     // has an active AF_XDP socket bound to it. Value 2 means
> >> +        *     // it's zero-copy multi-RQ mode.
> >>           *     qidconf = bpf_map_lookup_elem(&qidconf_map, &index);
> >>           *     if (!qidconf)
> >>           *         return XDP_ABORTED;
> >>           *
> >> -        *     if (*qidconf)
> >> +        *     qc = *qidconf;
> >> +        *
> >> +        *     if (qc == 2)
> >> +        *         qc = is_xskrq ? 1 : 0;
> >> +        *
> >> +        *     switch (qc) {
> >> +        *     case 0:
> >> +        *         return XDP_PASS;
> >> +        *     case 1:
> >>           *         return bpf_redirect_map(&xsks_map, index, 0);
> >> +        *     }
> >>           *
> >> -        *     return XDP_PASS;
> >> +        *     return XDP_ABORTED;
> >>           * }
> >>           */
> >>          struct bpf_insn prog[] = {
> >> -               /* r1 = *(u32 *)(r1 + 16) */
> >> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 16),
> >> -               /* *(u32 *)(r10 - 4) = r1 */
> >> -               BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_1, -4),
> >> -               BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
> >> -               BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
> >> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->qidconf_map_fd),
> >> +               /* Load index. */
> >> +               /* r6 = *(u32 *)(r1 + 16) */
> >> +               BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_ARG1, 16),
> >> +               /* w7 = w6 */
> >> +               BPF_MOV32_REG(BPF_REG_7, BPF_REG_6),
> >> +               /* w7 &= 2147483647 */
> >> +               BPF_ALU32_IMM(BPF_AND, BPF_REG_7, ~XDP_QID_FLAG_XSKRQ),
> >> +               /* *(u32 *)(r10 - 4) = r7 */
> >> +               BPF_STX_MEM(BPF_W, BPF_REG_FP, BPF_REG_7, -4),
> >> +
> >> +               /* Call bpf_map_lookup_elem. */
> >> +               /* r2 = r10 */
> >> +               BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP),
> >> +               /* r2 += -4 */
> >> +               BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -4),
> >> +               /* r1 = qidconf_map ll */
> >> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->qidconf_map_fd),
> >> +               /* call 1 */
> >>                  BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
> >> -               BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
> >> -               BPF_MOV32_IMM(BPF_REG_0, 0),
> >> -               /* if r1 == 0 goto +8 */
> >> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 8),
> >> -               BPF_MOV32_IMM(BPF_REG_0, 2),
> >> -               /* r1 = *(u32 *)(r1 + 0) */
> >> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 0),
> >> -               /* if r1 == 0 goto +5 */
> >> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 5),
> >> -               /* r2 = *(u32 *)(r10 - 4) */
> >> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),
> >> -               BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_10, -4),
> >> +
> >> +               /* Check the return value. */
> >> +               /* if r0 == 0 goto +14 */
> >> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 14),
> >> +
> >> +               /* Check qc == QIDCONF_XSK_COMBINED. */
> >> +               /* r6 >>= 31 */
> >> +               BPF_ALU64_IMM(BPF_RSH, BPF_REG_6, 31),
> >> +               /* r1 = *(u32 *)(r0 + 0) */
> >> +               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
> >> +               /* if r1 == 2 goto +1 */
> >> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, QIDCONF_XSK_COMBINED, 1),
> >> +
> >> +               /* qc != QIDCONF_XSK_COMBINED */
> >> +               /* r6 = r1 */
> >> +               BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
> >> +
> >> +               /* switch (qc) */
> >> +               /* w0 = 2 */
> >> +               BPF_MOV32_IMM(BPF_REG_0, XDP_PASS),
> >> +               /* if w6 == 0 goto +8 */
> >> +               BPF_JMP32_IMM(BPF_JEQ, BPF_REG_6, QIDCONF_REGULAR, 8),
> >> +               /* if w6 != 1 goto +6 */
> >> +               BPF_JMP32_IMM(BPF_JNE, BPF_REG_6, QIDCONF_XSK, 6),
> >> +
> >> +               /* Call bpf_redirect_map. */
> >> +               /* r1 = xsks_map ll */
> >> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->xsks_map_fd),
> >> +               /* w2 = w7 */
> >> +               BPF_MOV32_REG(BPF_REG_ARG2, BPF_REG_7),
> >> +               /* w3 = 0 */
> >>                  BPF_MOV32_IMM(BPF_REG_3, 0),
> >> +               /* call 51 */
> >>                  BPF_EMIT_CALL(BPF_FUNC_redirect_map),
> >> -               /* The jumps are to this instruction */
> >> +               /* exit */
> >> +               BPF_EXIT_INSN(),
> >> +
> >> +               /* XDP_ABORTED */
> >> +               /* w0 = 0 */
> >> +               BPF_MOV32_IMM(BPF_REG_0, XDP_ABORTED),
> >> +               /* exit */
> >>                  BPF_EXIT_INSN(),
> >>          };
> >>          size_t insns_cnt = sizeof(prog) / sizeof(struct bpf_insn);
> >> @@ -483,6 +538,7 @@ static int xsk_update_bpf_maps(struct xsk_socket *xsk, int qidconf_value,
> >>
> >>   static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
> >>   {
> >> +       int qidconf_value = QIDCONF_XSK;
> >>          bool prog_attached = false;
> >>          __u32 prog_id = 0;
> >>          int err;
> >> @@ -505,7 +561,11 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
> >>                  xsk->prog_fd = bpf_prog_get_fd_by_id(prog_id);
> >>          }
> >>
> >> -       err = xsk_update_bpf_maps(xsk, true, xsk->fd);
> >> +       if (xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
> >> +               if (xsk->zc)
> >> +                       qidconf_value = QIDCONF_XSK_COMBINED;
> >> +
> >> +       err = xsk_update_bpf_maps(xsk, qidconf_value, xsk->fd);
> >>          if (err)
> >>                  goto out_load;
> >>
> >> @@ -717,7 +777,7 @@ void xsk_socket__delete(struct xsk_socket *xsk)
> >>          if (!xsk)
> >>                  return;
> >>
> >> -       (void)xsk_update_bpf_maps(xsk, 0, 0);
> >> +       (void)xsk_update_bpf_maps(xsk, QIDCONF_REGULAR, 0);
> >>
> >>          optlen = sizeof(off);
> >>          err = getsockopt(xsk->fd, SOL_XDP, XDP_MMAP_OFFSETS, &off, &optlen);
> >> diff --git a/tools/lib/bpf/xsk.h b/tools/lib/bpf/xsk.h
> >> index 82ea71a0f3ec..be26a2423c04 100644
> >> --- a/tools/lib/bpf/xsk.h
> >> +++ b/tools/lib/bpf/xsk.h
> >> @@ -180,6 +180,10 @@ struct xsk_umem_config {
> >>
> >>   /* Flags for the libbpf_flags field. */
> >>   #define XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD (1 << 0)
> >> +#define XSK_LIBBPF_FLAGS__COMBINED_CHANNELS (1 << 1)
> >> +#define XSK_LIBBPF_FLAGS_MASK ( \
> >> +       XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD | \
> >> +       XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
> >>
> >>   struct xsk_socket_config {
> >>          __u32 rx_size;
> >> --
> >> 2.19.1
> >>
>

Re: [PATCH bpf-next v2 04/16] xsk: Extend channels to support combined XSK/non-XSK traffic

[Maxim Mikityanskiy]

On 2019-05-06 17:23, Magnus Karlsson wrote:
> On Mon, May 6, 2019 at 3:46 PM Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>>
>> On 2019-05-04 20:26, Björn Töpel wrote:
>>> On Tue, 30 Apr 2019 at 20:12, Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>>>>
>>>> Currently, the drivers that implement AF_XDP zero-copy support (e.g.,
>>>> i40e) switch the channel into a different mode when an XSK is opened. It
>>>> causes some issues that have to be taken into account. For example, RSS
>>>> needs to be reconfigured to skip the XSK-enabled channels, or the XDP
>>>> program should filter out traffic not intended for that socket and
>>>> XDP_PASS it with an additional copy. As nothing validates or forces the
>>>> proper configuration, it's easy to have packets drops, when they get
>>>> into an XSK by mistake, and, in fact, it's the default configuration.
>>>> There has to be some tool to have RSS reconfigured on each socket open
>>>> and close event, but such a tool is problematic to implement, because no
>>>> one reports these events, and it's race-prone.
>>>>
>>>> This commit extends XSK to support both kinds of traffic (XSK and
>>>> non-XSK) in the same channel. It implies having two RX queues in
>>>> XSK-enabled channels: one for the regular traffic, and the other for
>>>> XSK. It solves the problem with RSS: the default configuration just
>>>> works without the need to manually reconfigure RSS or to perform some
>>>> possibly complicated filtering in the XDP layer. It makes it easy to run
>>>> both AF_XDP and regular sockets on the same machine. In the XDP program,
>>>> the QID's most significant bit will serve as a flag to indicate whether
>>>> it's the XSK queue or not. The extension is compatible with the legacy
>>>> configuration, so if one wants to run the legacy mode, they can
>>>> reconfigure RSS and ignore the flag in the XDP program (implemented in
>>>> the reference XDP program in libbpf). mlx5e will support this extension.
>>>>
>>>> A single XDP program can run both with drivers supporting or not
>>>> supporting this extension. The xdpsock sample and libbpf are updated
>>>> accordingly.
>>>>
>>>
>>> I'm still not a fan of this, or maybe I'm not following you. It makes
>>> it more complex and even harder to use. Let's take a look at the
>>> kernel nomenclature. "ethtool" uses netdevs and channels. A channel is
>>> a Rx queue or a Tx queue.
>>
>> There are also combined channels that consist of an RX and a TX queue.
>> mlx5e has only this kind of channels. For us, a channel is a set of
>> queues "pinned to a CPU core" (they use the same NAPI).
>>
>>> In AF_XDP we call the channel a queue, which
>>> is what kernel uses internally (netdev_rx_queue, netdev_queue).
>>
>> You seem to agree it's a channel, right?
>>
>> AF_XDP doesn't allow to configure RX queue number and TX queue number
>> separately. Basically you choose a channel in AF_XDP. For some reason,
>> it's referred as a queue in some places, but logically it means "channel".
> 
> You can configure the Rx queue and the Tx queue separately by creating
> two sockets tied to the same umem area. But if you just create one,
> you are correct.

Yes, I know I can open two sockets, but it's only a workaround. If I 
want to RX on RQ 3 and TX on SQ 5, I'll have one socket bound to RQ 3 
and SQ 3, and the other bound to RQ 5 and SQ 5. It's not a clean way to 
achieve the initial goal. It means that the current implementation 
actually binds a socket to a channel X (that has RQ X and SQ X). It 
could be different if there were two kinds of XSK sockets: RX-only and 
TX-only, but it's not the case.

>>> Today, AF_XDP can attach to an existing queue for ingress. (On the
>>> egress side, we're using "a queue", but the "XDP queue". XDP has these
>>> "shadow queues" which are separated from the netdev. This is a bit
>>> messy, and we can't really configure them. I believe Jakub has some
>>> ideas here. :-) For now, let's leave egress aside.)
>>
>> So, XDP already has "shadow queues" for TX, so I see no problem in
>> having the similar concept for AF_XDP RX.
> 
> The question is if we would like to continue down the path of "shadow
> queues" by adding even more.

OK, I think "shadow queues" is not a valid name, and it's certainly not 
something bad. Initially, kernel had a concept of RX and TX queues. They 
can match the queues the driver has for regular traffic. Then new 
features appeared (XDP, then AF_XDP), and they required new types of 
queues: XDP TX queue, AF_XDP RX queue. These are absolutely new 
concepts, they aren't interchangeable with the legacy RX and TX queues. 
It means we can't just say we have 32 TX queues, 16 of which are regular 
SQs and 16 are XDP SQs. They function differently: the stack must not 
try sending anything from an XDP SQ, and XDP_REDIRECT must not try 
sending from the regular SQ.

However, the kernel didn't learn about new types of queues. That's why 
the new queues remain "in shadow". And we certainly shouldn't use the 
same numeration for different types of queues, i.e. it's incorrect to 
say that TX queues 0..15 are regular SQs, TX queues 16..31 are XDP SQs, 
etc. The correct way is: there are 16 regular SQs 0..15 and 16 XDP SQs 
0..15.

The same considerations apply to AF_XDP RX queues (XSK RQs). This is a 
whole new type of queue, so it can't be mixed with the regular RQs. 
That's why they have their own numeration and are not accounted as RX 
queues (moreover, their numeration can be non-contiguous). If the kernel 
had need to know about them, they could be exposed to the netdev as a 
third type of queue, but I see no need currently.

> In the busy-poll RFC I sent out last
> week, I talk about the possibility to create a new queue (set) not
> tied to the napi of the regular Rx queues in order to get better
> performance when using busy-poll. How would such a queue set fit into
> a shadow queue set approach?

Looking at the question in the RFC, I don't see in what way it 
interferes with XSK RQs. Moreover, it's even more convenient to have a 
separate type of queue - you'll be able to create and destroy "unbound" 
(not driven by an interrupt, having a separate NAPI) XSK RQs without 
interfering with the regular queues. At first sight, I would say it's a 
perfect fit :). For me, it looks cleaner when we have regular RQs in 
channels, XSK RQs in channels and "unbound" XSK RQs than having channels 
that switch the RQ type and having some extra out-of-channel RQs that 
behave differently, but look the same to the kernel.

> When does hiding the real queues created
> to support various features break and we have to expose the real queue
> number? Trying to wrap my head around these questions.

As I said above, different types of queues should not share the 
numeration. So, if need be, they can be exposed as a different type, but 
I don't see the necessity yet.

> Maxim, would it be possible for you to respin this set without this
> feature?

That's not as easy as you may think... It's not just a feature - for the 
kernel it's just a feature, but for the driver it's part of core design 
of AF_XDP support in mlx5. Removing it will require a lot of changes, 
reviewing and testing to adapt the driver to a different approach. I'd 
prefer to come to the conclusion before thinking of such changes :)

Thanks for giving the feedback and joining this discussion!

> I like the other stuff you have implemented and think that
> the rest of the common functionality should be useful for all of us.
> This way you can get the AF_XDP support accepted quicker while we
> debate the best way to solve the issue in this thread.
> 
> Thanks for all your work: Magnus
> 
>>> If an application would like to get all the traffic from a netdev,
>>> it'll create an equal amout of sockets as the queues and bind to the
>>> queues. Yes, even the queues in the RSS  set.
>>>
>>> What you would like (I think):
>>> a) is a way of spawning a new queue for a netdev, that is not part of
>>> the stack and/or RSS set
>>
>> Yes - for the simplicity sake and to make configuration easier. The only
>> thing needed is to steer the traffic and to open an AF_XDP socket on
>> channel X. We don't need to care about removing the queue out of RSS,
>> about finding a way to administer this (which is hard because it's racy
>> if the configuration in not known in advance). So I don't agree I'm
>> complicating things, my goal is to make them easier.
>>
>>> b) steering traffic to that queue using a configuration mechanism (tc?
>>> some yet to be hacked BPF configuration hook?)
>>
>> Currently, ethtool --config-ntuple is used to steer the traffic. The
>> user-def argument has a bit that selects XSK RQ/regular RQ, and action
>> selects a channel:
>>
>> ethtool -N eth0 flow-type udp4 dst-port 4242 action 3 user-def 1
>>
>>> With your mechanism you're doing this in contrived way. This makes the
>>> existing AF_XDP model *more* complex/hard(er) to use.
>>
>> No, as I said above, some issues are eliminated with my approach, and no
>> new limitations are introduced, so it makes things more universal and
>> simpler to configure.
>>
>>> How do you steer traffic to this dual-channel RQ?
>>
>> First, there is no dual-channel RQ, a more accurate term is dual-RQ
>> channel, cause now the channel contains a regular RQ and can contain an
>> XSK RQ.
>>
>> For the steering itself, see the ethtool command above - the user-def
>> argument has a bit that selects one of two RQs.
>>
>>> So you have a netdev
>>> receiving on all queues. Then, e.g., the last queue is a "dual
>>> channel" queue that can receive traffic from some other filter. How do
>>> you use it?
>>
>> If I want to take the last (or some) channel and start using AF_XDP with
>> it, I simply configure steering to the XSK RQ of that channel and open a
>> socket specifying the channel number. I don't need to reconfigure RSS,
>> because RSS packets go to the regular RQ of that channel and don't
>> interfere with XSK.
>>
>> No functionality is lost - if you don't distinguish the regular and XSK
>> RQs on the XDP level, you'll get the same effect as with i40e's
>> implementation. If you want to dedicate the CPU core and channel solely
>> for AF_XDP, in i40e you exclude the channel from RSS, and here you can
>> do exactly the same thing. So, no use case is complicated comparing to
>> i40e, but there are use cases where this feature is to advantage.
>>
>> I hope I explained the points you were interested in. Please ask more
>> questions if there is still something that I should clarify regarding
>> this topic.
>>
>> Thanks,
>> Max
>>
>>>
>>>
>>> Björn
>>>
>>>> Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
>>>> Acked-by: Saeed Mahameed <saeedm@mellanox.com>
>>>> ---
>>>>    include/uapi/linux/if_xdp.h       |  11 +++
>>>>    net/xdp/xsk.c                     |   5 +-
>>>>    samples/bpf/xdpsock_user.c        |  10 ++-
>>>>    tools/include/uapi/linux/if_xdp.h |  11 +++
>>>>    tools/lib/bpf/xsk.c               | 116 ++++++++++++++++++++++--------
>>>>    tools/lib/bpf/xsk.h               |   4 ++
>>>>    6 files changed, 126 insertions(+), 31 deletions(-)
>>>>
>>>> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
>>>> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
>>>> --- a/include/uapi/linux/if_xdp.h
>>>> +++ b/include/uapi/linux/if_xdp.h
>>>> @@ -82,4 +82,15 @@ struct xdp_desc {
>>>>
>>>>    /* UMEM descriptor is __u64 */
>>>>
>>>> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
>>>> + * this flag bit in the queue index to distinguish between two RQs of the same
>>>> + * channel.
>>>> + */
>>>> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
>>>> +
>>>> +static inline __u32 xdp_qid_get_channel(__u32 qid)
>>>> +{
>>>> +       return qid & ~XDP_QID_FLAG_XSKRQ;
>>>> +}
>>>> +
>>>>    #endif /* _LINUX_IF_XDP_H */
>>>> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
>>>> index 998199109d5c..114ba17acb09 100644
>>>> --- a/net/xdp/xsk.c
>>>> +++ b/net/xdp/xsk.c
>>>> @@ -104,9 +104,12 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
>>>>
>>>>    int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
>>>>    {
>>>> +       struct xdp_rxq_info *rxq = xdp->rxq;
>>>> +       u32 channel = xdp_qid_get_channel(rxq->queue_index);
>>>>           u32 len;
>>>>
>>>> -       if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
>>>> +       if (xs->dev != rxq->dev || xs->queue_id != channel ||
>>>> +           xs->zc != (rxq->mem.type == MEM_TYPE_ZERO_COPY))
>>>>                   return -EINVAL;
>>>>
>>>>           len = xdp->data_end - xdp->data;
>>>> diff --git a/samples/bpf/xdpsock_user.c b/samples/bpf/xdpsock_user.c
>>>> index d08ee1ab7bb4..a6b13025ee79 100644
>>>> --- a/samples/bpf/xdpsock_user.c
>>>> +++ b/samples/bpf/xdpsock_user.c
>>>> @@ -62,6 +62,7 @@ enum benchmark_type {
>>>>
>>>>    static enum benchmark_type opt_bench = BENCH_RXDROP;
>>>>    static u32 opt_xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
>>>> +static u32 opt_libbpf_flags;
>>>>    static const char *opt_if = "";
>>>>    static int opt_ifindex;
>>>>    static int opt_queue;
>>>> @@ -306,7 +307,7 @@ static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem)
>>>>           xsk->umem = umem;
>>>>           cfg.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
>>>>           cfg.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS;
>>>> -       cfg.libbpf_flags = 0;
>>>> +       cfg.libbpf_flags = opt_libbpf_flags;
>>>>           cfg.xdp_flags = opt_xdp_flags;
>>>>           cfg.bind_flags = opt_xdp_bind_flags;
>>>>           ret = xsk_socket__create(&xsk->xsk, opt_if, opt_queue, umem->umem,
>>>> @@ -346,6 +347,7 @@ static struct option long_options[] = {
>>>>           {"interval", required_argument, 0, 'n'},
>>>>           {"zero-copy", no_argument, 0, 'z'},
>>>>           {"copy", no_argument, 0, 'c'},
>>>> +       {"combined", no_argument, 0, 'C'},
>>>>           {0, 0, 0, 0}
>>>>    };
>>>>
>>>> @@ -365,6 +367,7 @@ static void usage(const char *prog)
>>>>                   "  -n, --interval=n     Specify statistics update interval (default 1 sec).\n"
>>>>                   "  -z, --zero-copy      Force zero-copy mode.\n"
>>>>                   "  -c, --copy           Force copy mode.\n"
>>>> +               "  -C, --combined       Driver supports combined XSK and non-XSK traffic in a channel.\n"
>>>>                   "\n";
>>>>           fprintf(stderr, str, prog);
>>>>           exit(EXIT_FAILURE);
>>>> @@ -377,7 +380,7 @@ static void parse_command_line(int argc, char **argv)
>>>>           opterr = 0;
>>>>
>>>>           for (;;) {
>>>> -               c = getopt_long(argc, argv, "Frtli:q:psSNn:cz", long_options,
>>>> +               c = getopt_long(argc, argv, "Frtli:q:psSNn:czC", long_options,
>>>>                                   &option_index);
>>>>                   if (c == -1)
>>>>                           break;
>>>> @@ -420,6 +423,9 @@ static void parse_command_line(int argc, char **argv)
>>>>                   case 'F':
>>>>                           opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
>>>>                           break;
>>>> +               case 'C':
>>>> +                       opt_libbpf_flags |= XSK_LIBBPF_FLAGS__COMBINED_CHANNELS;
>>>> +                       break;
>>>>                   default:
>>>>                           usage(basename(argv[0]));
>>>>                   }
>>>> diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
>>>> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
>>>> --- a/tools/include/uapi/linux/if_xdp.h
>>>> +++ b/tools/include/uapi/linux/if_xdp.h
>>>> @@ -82,4 +82,15 @@ struct xdp_desc {
>>>>
>>>>    /* UMEM descriptor is __u64 */
>>>>
>>>> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
>>>> + * this flag bit in the queue index to distinguish between two RQs of the same
>>>> + * channel.
>>>> + */
>>>> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
>>>> +
>>>> +static inline __u32 xdp_qid_get_channel(__u32 qid)
>>>> +{
>>>> +       return qid & ~XDP_QID_FLAG_XSKRQ;
>>>> +}
>>>> +
>>>>    #endif /* _LINUX_IF_XDP_H */
>>>> diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c
>>>> index a95b06d1f81d..969dfd856039 100644
>>>> --- a/tools/lib/bpf/xsk.c
>>>> +++ b/tools/lib/bpf/xsk.c
>>>> @@ -76,6 +76,12 @@ struct xsk_nl_info {
>>>>           int fd;
>>>>    };
>>>>
>>>> +enum qidconf {
>>>> +       QIDCONF_REGULAR,
>>>> +       QIDCONF_XSK,
>>>> +       QIDCONF_XSK_COMBINED,
>>>> +};
>>>> +
>>>>    /* For 32-bit systems, we need to use mmap2 as the offsets are 64-bit.
>>>>     * Unfortunately, it is not part of glibc.
>>>>     */
>>>> @@ -139,7 +145,7 @@ static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg,
>>>>                   return 0;
>>>>           }
>>>>
>>>> -       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)
>>>> +       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS_MASK)
>>>>                   return -EINVAL;
>>>>
>>>>           cfg->rx_size = usr_cfg->rx_size;
>>>> @@ -267,44 +273,93 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk)
>>>>           /* This is the C-program:
>>>>            * SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)
>>>>            * {
>>>> -        *     int *qidconf, index = ctx->rx_queue_index;
>>>> +        *     int *qidconf, qc;
>>>> +        *     int index = ctx->rx_queue_index & ~(1 << 31);
>>>> +        *     bool is_xskrq = ctx->rx_queue_index & (1 << 31);
>>>>            *
>>>> -        *     // A set entry here means that the correspnding queue_id
>>>> -        *     // has an active AF_XDP socket bound to it.
>>>> +        *     // A set entry here means that the corresponding queue_id
>>>> +        *     // has an active AF_XDP socket bound to it. Value 2 means
>>>> +        *     // it's zero-copy multi-RQ mode.
>>>>            *     qidconf = bpf_map_lookup_elem(&qidconf_map, &index);
>>>>            *     if (!qidconf)
>>>>            *         return XDP_ABORTED;
>>>>            *
>>>> -        *     if (*qidconf)
>>>> +        *     qc = *qidconf;
>>>> +        *
>>>> +        *     if (qc == 2)
>>>> +        *         qc = is_xskrq ? 1 : 0;
>>>> +        *
>>>> +        *     switch (qc) {
>>>> +        *     case 0:
>>>> +        *         return XDP_PASS;
>>>> +        *     case 1:
>>>>            *         return bpf_redirect_map(&xsks_map, index, 0);
>>>> +        *     }
>>>>            *
>>>> -        *     return XDP_PASS;
>>>> +        *     return XDP_ABORTED;
>>>>            * }
>>>>            */
>>>>           struct bpf_insn prog[] = {
>>>> -               /* r1 = *(u32 *)(r1 + 16) */
>>>> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 16),
>>>> -               /* *(u32 *)(r10 - 4) = r1 */
>>>> -               BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_1, -4),
>>>> -               BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
>>>> -               BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
>>>> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->qidconf_map_fd),
>>>> +               /* Load index. */
>>>> +               /* r6 = *(u32 *)(r1 + 16) */
>>>> +               BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_ARG1, 16),
>>>> +               /* w7 = w6 */
>>>> +               BPF_MOV32_REG(BPF_REG_7, BPF_REG_6),
>>>> +               /* w7 &= 2147483647 */
>>>> +               BPF_ALU32_IMM(BPF_AND, BPF_REG_7, ~XDP_QID_FLAG_XSKRQ),
>>>> +               /* *(u32 *)(r10 - 4) = r7 */
>>>> +               BPF_STX_MEM(BPF_W, BPF_REG_FP, BPF_REG_7, -4),
>>>> +
>>>> +               /* Call bpf_map_lookup_elem. */
>>>> +               /* r2 = r10 */
>>>> +               BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP),
>>>> +               /* r2 += -4 */
>>>> +               BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -4),
>>>> +               /* r1 = qidconf_map ll */
>>>> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->qidconf_map_fd),
>>>> +               /* call 1 */
>>>>                   BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
>>>> -               BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
>>>> -               BPF_MOV32_IMM(BPF_REG_0, 0),
>>>> -               /* if r1 == 0 goto +8 */
>>>> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 8),
>>>> -               BPF_MOV32_IMM(BPF_REG_0, 2),
>>>> -               /* r1 = *(u32 *)(r1 + 0) */
>>>> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 0),
>>>> -               /* if r1 == 0 goto +5 */
>>>> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 5),
>>>> -               /* r2 = *(u32 *)(r10 - 4) */
>>>> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),
>>>> -               BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_10, -4),
>>>> +
>>>> +               /* Check the return value. */
>>>> +               /* if r0 == 0 goto +14 */
>>>> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 14),
>>>> +
>>>> +               /* Check qc == QIDCONF_XSK_COMBINED. */
>>>> +               /* r6 >>= 31 */
>>>> +               BPF_ALU64_IMM(BPF_RSH, BPF_REG_6, 31),
>>>> +               /* r1 = *(u32 *)(r0 + 0) */
>>>> +               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
>>>> +               /* if r1 == 2 goto +1 */
>>>> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, QIDCONF_XSK_COMBINED, 1),
>>>> +
>>>> +               /* qc != QIDCONF_XSK_COMBINED */
>>>> +               /* r6 = r1 */
>>>> +               BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
>>>> +
>>>> +               /* switch (qc) */
>>>> +               /* w0 = 2 */
>>>> +               BPF_MOV32_IMM(BPF_REG_0, XDP_PASS),
>>>> +               /* if w6 == 0 goto +8 */
>>>> +               BPF_JMP32_IMM(BPF_JEQ, BPF_REG_6, QIDCONF_REGULAR, 8),
>>>> +               /* if w6 != 1 goto +6 */
>>>> +               BPF_JMP32_IMM(BPF_JNE, BPF_REG_6, QIDCONF_XSK, 6),
>>>> +
>>>> +               /* Call bpf_redirect_map. */
>>>> +               /* r1 = xsks_map ll */
>>>> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->xsks_map_fd),
>>>> +               /* w2 = w7 */
>>>> +               BPF_MOV32_REG(BPF_REG_ARG2, BPF_REG_7),
>>>> +               /* w3 = 0 */
>>>>                   BPF_MOV32_IMM(BPF_REG_3, 0),
>>>> +               /* call 51 */
>>>>                   BPF_EMIT_CALL(BPF_FUNC_redirect_map),
>>>> -               /* The jumps are to this instruction */
>>>> +               /* exit */
>>>> +               BPF_EXIT_INSN(),
>>>> +
>>>> +               /* XDP_ABORTED */
>>>> +               /* w0 = 0 */
>>>> +               BPF_MOV32_IMM(BPF_REG_0, XDP_ABORTED),
>>>> +               /* exit */
>>>>                   BPF_EXIT_INSN(),
>>>>           };
>>>>           size_t insns_cnt = sizeof(prog) / sizeof(struct bpf_insn);
>>>> @@ -483,6 +538,7 @@ static int xsk_update_bpf_maps(struct xsk_socket *xsk, int qidconf_value,
>>>>
>>>>    static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
>>>>    {
>>>> +       int qidconf_value = QIDCONF_XSK;
>>>>           bool prog_attached = false;
>>>>           __u32 prog_id = 0;
>>>>           int err;
>>>> @@ -505,7 +561,11 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
>>>>                   xsk->prog_fd = bpf_prog_get_fd_by_id(prog_id);
>>>>           }
>>>>
>>>> -       err = xsk_update_bpf_maps(xsk, true, xsk->fd);
>>>> +       if (xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
>>>> +               if (xsk->zc)
>>>> +                       qidconf_value = QIDCONF_XSK_COMBINED;
>>>> +
>>>> +       err = xsk_update_bpf_maps(xsk, qidconf_value, xsk->fd);
>>>>           if (err)
>>>>                   goto out_load;
>>>>
>>>> @@ -717,7 +777,7 @@ void xsk_socket__delete(struct xsk_socket *xsk)
>>>>           if (!xsk)
>>>>                   return;
>>>>
>>>> -       (void)xsk_update_bpf_maps(xsk, 0, 0);
>>>> +       (void)xsk_update_bpf_maps(xsk, QIDCONF_REGULAR, 0);
>>>>
>>>>           optlen = sizeof(off);
>>>>           err = getsockopt(xsk->fd, SOL_XDP, XDP_MMAP_OFFSETS, &off, &optlen);
>>>> diff --git a/tools/lib/bpf/xsk.h b/tools/lib/bpf/xsk.h
>>>> index 82ea71a0f3ec..be26a2423c04 100644
>>>> --- a/tools/lib/bpf/xsk.h
>>>> +++ b/tools/lib/bpf/xsk.h
>>>> @@ -180,6 +180,10 @@ struct xsk_umem_config {
>>>>
>>>>    /* Flags for the libbpf_flags field. */
>>>>    #define XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD (1 << 0)
>>>> +#define XSK_LIBBPF_FLAGS__COMBINED_CHANNELS (1 << 1)
>>>> +#define XSK_LIBBPF_FLAGS_MASK ( \
>>>> +       XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD | \
>>>> +       XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
>>>>
>>>>    struct xsk_socket_config {
>>>>           __u32 rx_size;
>>>> --
>>>> 2.19.1
>>>>
>>

Re: [PATCH bpf-next v2 04/16] xsk: Extend channels to support combined XSK/non-XSK traffic

[Magnus Karlsson]

On Tue, May 7, 2019 at 4:19 PM Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>
> On 2019-05-06 17:23, Magnus Karlsson wrote:
> > On Mon, May 6, 2019 at 3:46 PM Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
> >>
> >> On 2019-05-04 20:26, Björn Töpel wrote:
> >>> On Tue, 30 Apr 2019 at 20:12, Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
> >>>>
> >>>> Currently, the drivers that implement AF_XDP zero-copy support (e.g.,
> >>>> i40e) switch the channel into a different mode when an XSK is opened. It
> >>>> causes some issues that have to be taken into account. For example, RSS
> >>>> needs to be reconfigured to skip the XSK-enabled channels, or the XDP
> >>>> program should filter out traffic not intended for that socket and
> >>>> XDP_PASS it with an additional copy. As nothing validates or forces the
> >>>> proper configuration, it's easy to have packets drops, when they get
> >>>> into an XSK by mistake, and, in fact, it's the default configuration.
> >>>> There has to be some tool to have RSS reconfigured on each socket open
> >>>> and close event, but such a tool is problematic to implement, because no
> >>>> one reports these events, and it's race-prone.
> >>>>
> >>>> This commit extends XSK to support both kinds of traffic (XSK and
> >>>> non-XSK) in the same channel. It implies having two RX queues in
> >>>> XSK-enabled channels: one for the regular traffic, and the other for
> >>>> XSK. It solves the problem with RSS: the default configuration just
> >>>> works without the need to manually reconfigure RSS or to perform some
> >>>> possibly complicated filtering in the XDP layer. It makes it easy to run
> >>>> both AF_XDP and regular sockets on the same machine. In the XDP program,
> >>>> the QID's most significant bit will serve as a flag to indicate whether
> >>>> it's the XSK queue or not. The extension is compatible with the legacy
> >>>> configuration, so if one wants to run the legacy mode, they can
> >>>> reconfigure RSS and ignore the flag in the XDP program (implemented in
> >>>> the reference XDP program in libbpf). mlx5e will support this extension.
> >>>>
> >>>> A single XDP program can run both with drivers supporting or not
> >>>> supporting this extension. The xdpsock sample and libbpf are updated
> >>>> accordingly.
> >>>>
> >>>
> >>> I'm still not a fan of this, or maybe I'm not following you. It makes
> >>> it more complex and even harder to use. Let's take a look at the
> >>> kernel nomenclature. "ethtool" uses netdevs and channels. A channel is
> >>> a Rx queue or a Tx queue.
> >>
> >> There are also combined channels that consist of an RX and a TX queue.
> >> mlx5e has only this kind of channels. For us, a channel is a set of
> >> queues "pinned to a CPU core" (they use the same NAPI).
> >>
> >>> In AF_XDP we call the channel a queue, which
> >>> is what kernel uses internally (netdev_rx_queue, netdev_queue).
> >>
> >> You seem to agree it's a channel, right?
> >>
> >> AF_XDP doesn't allow to configure RX queue number and TX queue number
> >> separately. Basically you choose a channel in AF_XDP. For some reason,
> >> it's referred as a queue in some places, but logically it means "channel".
> >
> > You can configure the Rx queue and the Tx queue separately by creating
> > two sockets tied to the same umem area. But if you just create one,
> > you are correct.
>
> Yes, I know I can open two sockets, but it's only a workaround. If I
> want to RX on RQ 3 and TX on SQ 5, I'll have one socket bound to RQ 3
> and SQ 3, and the other bound to RQ 5 and SQ 5. It's not a clean way to
> achieve the initial goal. It means that the current implementation
> actually binds a socket to a channel X (that has RQ X and SQ X). It
> could be different if there were two kinds of XSK sockets: RX-only and
> TX-only, but it's not the case.

It is possible to create Rx only or Tx only AF_XDP sockets from the
uapi by only creating an Rx or a Tx ring. Though, the NDO that
registers the umem in the driver would need to be extended with this
information so that the driver could act upon it. Today it is assumed
that it is always Rx and Tx which wastes resources and maybe also
performance. This has been on my todo list for quite some time, but it
has never floated up to the top.

> >>> Today, AF_XDP can attach to an existing queue for ingress. (On the
> >>> egress side, we're using "a queue", but the "XDP queue". XDP has these
> >>> "shadow queues" which are separated from the netdev. This is a bit
> >>> messy, and we can't really configure them. I believe Jakub has some
> >>> ideas here. :-) For now, let's leave egress aside.)
> >>
> >> So, XDP already has "shadow queues" for TX, so I see no problem in
> >> having the similar concept for AF_XDP RX.
> >
> > The question is if we would like to continue down the path of "shadow
> > queues" by adding even more.
>
> OK, I think "shadow queues" is not a valid name, and it's certainly not
> something bad. Initially, kernel had a concept of RX and TX queues. They
> can match the queues the driver has for regular traffic. Then new
> features appeared (XDP, then AF_XDP), and they required new types of
> queues: XDP TX queue, AF_XDP RX queue. These are absolutely new
> concepts, they aren't interchangeable with the legacy RX and TX queues.
> It means we can't just say we have 32 TX queues, 16 of which are regular
> SQs and 16 are XDP SQs. They function differently: the stack must not
> try sending anything from an XDP SQ, and XDP_REDIRECT must not try
> sending from the regular SQ.
>
> However, the kernel didn't learn about new types of queues. That's why
> the new queues remain "in shadow". And we certainly shouldn't use the
> same numeration for different types of queues, i.e. it's incorrect to
> say that TX queues 0..15 are regular SQs, TX queues 16..31 are XDP SQs,
> etc. The correct way is: there are 16 regular SQs 0..15 and 16 XDP SQs
> 0..15.
>
> The same considerations apply to AF_XDP RX queues (XSK RQs). This is a
> whole new type of queue, so it can't be mixed with the regular RQs.
> That's why they have their own numeration and are not accounted as RX
> queues (moreover, their numeration can be non-contiguous). If the kernel
> had need to know about them, they could be exposed to the netdev as a
> third type of queue, but I see no need currently.
>
> > In the busy-poll RFC I sent out last
> > week, I talk about the possibility to create a new queue (set) not
> > tied to the napi of the regular Rx queues in order to get better
> > performance when using busy-poll. How would such a queue set fit into
> > a shadow queue set approach?
>
> Looking at the question in the RFC, I don't see in what way it
> interferes with XSK RQs. Moreover, it's even more convenient to have a
> separate type of queue - you'll be able to create and destroy "unbound"
> (not driven by an interrupt, having a separate NAPI) XSK RQs without
> interfering with the regular queues. At first sight, I would say it's a
> perfect fit :). For me, it looks cleaner when we have regular RQs in
> channels, XSK RQs in channels and "unbound" XSK RQs than having channels
> that switch the RQ type and having some extra out-of-channel RQs that
> behave differently, but look the same to the kernel.

Agree with you that they are not interchangeable and that they
should be explicit. So we are on the same page here. Good. But
let us try to solve the Tx problem first because it has been
there from day one for both XDP and AF_XDP. Jakub Kicinski has
some ideas in this direction that he presented at Linux Kernel
Developers' bpfconf 2019
(http://vger.kernel.org/bpfconf2019.html) where he suggested to
start with Tx first. Let us start a discussion around that. Many
of the concepts developed there should hopefully extend to the Rx
side too.

Note that AF_XDP is today using the Rx queues that XDP operates
on (because it is an extension of XDP) both in the uapi and in
the driver. And these are the regular netdev Rx queues. But as we
all agree on, we need to change this so the choice becomes more
flexible and plays nicely with all side band configuration tools.

> > When does hiding the real queues created
> > to support various features break and we have to expose the real queue
> > number? Trying to wrap my head around these questions.
>
> As I said above, different types of queues should not share the
> numeration. So, if need be, they can be exposed as a different type, but
> I don't see the necessity yet.
>
> > Maxim, would it be possible for you to respin this set without this
> > feature?
>
> That's not as easy as you may think... It's not just a feature - for the
> kernel it's just a feature, but for the driver it's part of core design
> of AF_XDP support in mlx5. Removing it will require a lot of changes,
> reviewing and testing to adapt the driver to a different approach. I'd
> prefer to come to the conclusion before thinking of such changes :)

I leave it up to you if you would like to wait to submit the next
version of your patch set until the "queue id" problem of both Rx
and Tx has been resolved. I hope that it will be speedy so that
you can get your patch set in, but it might not be. It would have
been better if you had partitioned your patch set from the start
as: basic support -> new uapi/kernel feature -> implementation of
the new feature in your driver. Then we could have acked the
basic support quickly. It is good stuff.

Thanks: Magnus

> Thanks for giving the feedback and joining this discussion!
>
> > I like the other stuff you have implemented and think that
> > the rest of the common functionality should be useful for all of us.
> > This way you can get the AF_XDP support accepted quicker while we
> > debate the best way to solve the issue in this thread.
> >
> > Thanks for all your work: Magnus
> >
> >>> If an application would like to get all the traffic from a netdev,
> >>> it'll create an equal amout of sockets as the queues and bind to the
> >>> queues. Yes, even the queues in the RSS  set.
> >>>
> >>> What you would like (I think):
> >>> a) is a way of spawning a new queue for a netdev, that is not part of
> >>> the stack and/or RSS set
> >>
> >> Yes - for the simplicity sake and to make configuration easier. The only
> >> thing needed is to steer the traffic and to open an AF_XDP socket on
> >> channel X. We don't need to care about removing the queue out of RSS,
> >> about finding a way to administer this (which is hard because it's racy
> >> if the configuration in not known in advance). So I don't agree I'm
> >> complicating things, my goal is to make them easier.
> >>
> >>> b) steering traffic to that queue using a configuration mechanism (tc?
> >>> some yet to be hacked BPF configuration hook?)
> >>
> >> Currently, ethtool --config-ntuple is used to steer the traffic. The
> >> user-def argument has a bit that selects XSK RQ/regular RQ, and action
> >> selects a channel:
> >>
> >> ethtool -N eth0 flow-type udp4 dst-port 4242 action 3 user-def 1
> >>
> >>> With your mechanism you're doing this in contrived way. This makes the
> >>> existing AF_XDP model *more* complex/hard(er) to use.
> >>
> >> No, as I said above, some issues are eliminated with my approach, and no
> >> new limitations are introduced, so it makes things more universal and
> >> simpler to configure.
> >>
> >>> How do you steer traffic to this dual-channel RQ?
> >>
> >> First, there is no dual-channel RQ, a more accurate term is dual-RQ
> >> channel, cause now the channel contains a regular RQ and can contain an
> >> XSK RQ.
> >>
> >> For the steering itself, see the ethtool command above - the user-def
> >> argument has a bit that selects one of two RQs.
> >>
> >>> So you have a netdev
> >>> receiving on all queues. Then, e.g., the last queue is a "dual
> >>> channel" queue that can receive traffic from some other filter. How do
> >>> you use it?
> >>
> >> If I want to take the last (or some) channel and start using AF_XDP with
> >> it, I simply configure steering to the XSK RQ of that channel and open a
> >> socket specifying the channel number. I don't need to reconfigure RSS,
> >> because RSS packets go to the regular RQ of that channel and don't
> >> interfere with XSK.
> >>
> >> No functionality is lost - if you don't distinguish the regular and XSK
> >> RQs on the XDP level, you'll get the same effect as with i40e's
> >> implementation. If you want to dedicate the CPU core and channel solely
> >> for AF_XDP, in i40e you exclude the channel from RSS, and here you can
> >> do exactly the same thing. So, no use case is complicated comparing to
> >> i40e, but there are use cases where this feature is to advantage.
> >>
> >> I hope I explained the points you were interested in. Please ask more
> >> questions if there is still something that I should clarify regarding
> >> this topic.
> >>
> >> Thanks,
> >> Max
> >>
> >>>
> >>>
> >>> Björn
> >>>
> >>>> Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
> >>>> Acked-by: Saeed Mahameed <saeedm@mellanox.com>
> >>>> ---
> >>>>    include/uapi/linux/if_xdp.h       |  11 +++
> >>>>    net/xdp/xsk.c                     |   5 +-
> >>>>    samples/bpf/xdpsock_user.c        |  10 ++-
> >>>>    tools/include/uapi/linux/if_xdp.h |  11 +++
> >>>>    tools/lib/bpf/xsk.c               | 116 ++++++++++++++++++++++--------
> >>>>    tools/lib/bpf/xsk.h               |   4 ++
> >>>>    6 files changed, 126 insertions(+), 31 deletions(-)
> >>>>
> >>>> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
> >>>> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
> >>>> --- a/include/uapi/linux/if_xdp.h
> >>>> +++ b/include/uapi/linux/if_xdp.h
> >>>> @@ -82,4 +82,15 @@ struct xdp_desc {
> >>>>
> >>>>    /* UMEM descriptor is __u64 */
> >>>>
> >>>> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
> >>>> + * this flag bit in the queue index to distinguish between two RQs of the same
> >>>> + * channel.
> >>>> + */
> >>>> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
> >>>> +
> >>>> +static inline __u32 xdp_qid_get_channel(__u32 qid)
> >>>> +{
> >>>> +       return qid & ~XDP_QID_FLAG_XSKRQ;
> >>>> +}
> >>>> +
> >>>>    #endif /* _LINUX_IF_XDP_H */
> >>>> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
> >>>> index 998199109d5c..114ba17acb09 100644
> >>>> --- a/net/xdp/xsk.c
> >>>> +++ b/net/xdp/xsk.c
> >>>> @@ -104,9 +104,12 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
> >>>>
> >>>>    int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
> >>>>    {
> >>>> +       struct xdp_rxq_info *rxq = xdp->rxq;
> >>>> +       u32 channel = xdp_qid_get_channel(rxq->queue_index);
> >>>>           u32 len;
> >>>>
> >>>> -       if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
> >>>> +       if (xs->dev != rxq->dev || xs->queue_id != channel ||
> >>>> +           xs->zc != (rxq->mem.type == MEM_TYPE_ZERO_COPY))
> >>>>                   return -EINVAL;
> >>>>
> >>>>           len = xdp->data_end - xdp->data;
> >>>> diff --git a/samples/bpf/xdpsock_user.c b/samples/bpf/xdpsock_user.c
> >>>> index d08ee1ab7bb4..a6b13025ee79 100644
> >>>> --- a/samples/bpf/xdpsock_user.c
> >>>> +++ b/samples/bpf/xdpsock_user.c
> >>>> @@ -62,6 +62,7 @@ enum benchmark_type {
> >>>>
> >>>>    static enum benchmark_type opt_bench = BENCH_RXDROP;
> >>>>    static u32 opt_xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
> >>>> +static u32 opt_libbpf_flags;
> >>>>    static const char *opt_if = "";
> >>>>    static int opt_ifindex;
> >>>>    static int opt_queue;
> >>>> @@ -306,7 +307,7 @@ static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem)
> >>>>           xsk->umem = umem;
> >>>>           cfg.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
> >>>>           cfg.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS;
> >>>> -       cfg.libbpf_flags = 0;
> >>>> +       cfg.libbpf_flags = opt_libbpf_flags;
> >>>>           cfg.xdp_flags = opt_xdp_flags;
> >>>>           cfg.bind_flags = opt_xdp_bind_flags;
> >>>>           ret = xsk_socket__create(&xsk->xsk, opt_if, opt_queue, umem->umem,
> >>>> @@ -346,6 +347,7 @@ static struct option long_options[] = {
> >>>>           {"interval", required_argument, 0, 'n'},
> >>>>           {"zero-copy", no_argument, 0, 'z'},
> >>>>           {"copy", no_argument, 0, 'c'},
> >>>> +       {"combined", no_argument, 0, 'C'},
> >>>>           {0, 0, 0, 0}
> >>>>    };
> >>>>
> >>>> @@ -365,6 +367,7 @@ static void usage(const char *prog)
> >>>>                   "  -n, --interval=n     Specify statistics update interval (default 1 sec).\n"
> >>>>                   "  -z, --zero-copy      Force zero-copy mode.\n"
> >>>>                   "  -c, --copy           Force copy mode.\n"
> >>>> +               "  -C, --combined       Driver supports combined XSK and non-XSK traffic in a channel.\n"
> >>>>                   "\n";
> >>>>           fprintf(stderr, str, prog);
> >>>>           exit(EXIT_FAILURE);
> >>>> @@ -377,7 +380,7 @@ static void parse_command_line(int argc, char **argv)
> >>>>           opterr = 0;
> >>>>
> >>>>           for (;;) {
> >>>> -               c = getopt_long(argc, argv, "Frtli:q:psSNn:cz", long_options,
> >>>> +               c = getopt_long(argc, argv, "Frtli:q:psSNn:czC", long_options,
> >>>>                                   &option_index);
> >>>>                   if (c == -1)
> >>>>                           break;
> >>>> @@ -420,6 +423,9 @@ static void parse_command_line(int argc, char **argv)
> >>>>                   case 'F':
> >>>>                           opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
> >>>>                           break;
> >>>> +               case 'C':
> >>>> +                       opt_libbpf_flags |= XSK_LIBBPF_FLAGS__COMBINED_CHANNELS;
> >>>> +                       break;
> >>>>                   default:
> >>>>                           usage(basename(argv[0]));
> >>>>                   }
> >>>> diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
> >>>> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
> >>>> --- a/tools/include/uapi/linux/if_xdp.h
> >>>> +++ b/tools/include/uapi/linux/if_xdp.h
> >>>> @@ -82,4 +82,15 @@ struct xdp_desc {
> >>>>
> >>>>    /* UMEM descriptor is __u64 */
> >>>>
> >>>> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
> >>>> + * this flag bit in the queue index to distinguish between two RQs of the same
> >>>> + * channel.
> >>>> + */
> >>>> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
> >>>> +
> >>>> +static inline __u32 xdp_qid_get_channel(__u32 qid)
> >>>> +{
> >>>> +       return qid & ~XDP_QID_FLAG_XSKRQ;
> >>>> +}
> >>>> +
> >>>>    #endif /* _LINUX_IF_XDP_H */
> >>>> diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c
> >>>> index a95b06d1f81d..969dfd856039 100644
> >>>> --- a/tools/lib/bpf/xsk.c
> >>>> +++ b/tools/lib/bpf/xsk.c
> >>>> @@ -76,6 +76,12 @@ struct xsk_nl_info {
> >>>>           int fd;
> >>>>    };
> >>>>
> >>>> +enum qidconf {
> >>>> +       QIDCONF_REGULAR,
> >>>> +       QIDCONF_XSK,
> >>>> +       QIDCONF_XSK_COMBINED,
> >>>> +};
> >>>> +
> >>>>    /* For 32-bit systems, we need to use mmap2 as the offsets are 64-bit.
> >>>>     * Unfortunately, it is not part of glibc.
> >>>>     */
> >>>> @@ -139,7 +145,7 @@ static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg,
> >>>>                   return 0;
> >>>>           }
> >>>>
> >>>> -       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)
> >>>> +       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS_MASK)
> >>>>                   return -EINVAL;
> >>>>
> >>>>           cfg->rx_size = usr_cfg->rx_size;
> >>>> @@ -267,44 +273,93 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk)
> >>>>           /* This is the C-program:
> >>>>            * SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)
> >>>>            * {
> >>>> -        *     int *qidconf, index = ctx->rx_queue_index;
> >>>> +        *     int *qidconf, qc;
> >>>> +        *     int index = ctx->rx_queue_index & ~(1 << 31);
> >>>> +        *     bool is_xskrq = ctx->rx_queue_index & (1 << 31);
> >>>>            *
> >>>> -        *     // A set entry here means that the correspnding queue_id
> >>>> -        *     // has an active AF_XDP socket bound to it.
> >>>> +        *     // A set entry here means that the corresponding queue_id
> >>>> +        *     // has an active AF_XDP socket bound to it. Value 2 means
> >>>> +        *     // it's zero-copy multi-RQ mode.
> >>>>            *     qidconf = bpf_map_lookup_elem(&qidconf_map, &index);
> >>>>            *     if (!qidconf)
> >>>>            *         return XDP_ABORTED;
> >>>>            *
> >>>> -        *     if (*qidconf)
> >>>> +        *     qc = *qidconf;
> >>>> +        *
> >>>> +        *     if (qc == 2)
> >>>> +        *         qc = is_xskrq ? 1 : 0;
> >>>> +        *
> >>>> +        *     switch (qc) {
> >>>> +        *     case 0:
> >>>> +        *         return XDP_PASS;
> >>>> +        *     case 1:
> >>>>            *         return bpf_redirect_map(&xsks_map, index, 0);
> >>>> +        *     }
> >>>>            *
> >>>> -        *     return XDP_PASS;
> >>>> +        *     return XDP_ABORTED;
> >>>>            * }
> >>>>            */
> >>>>           struct bpf_insn prog[] = {
> >>>> -               /* r1 = *(u32 *)(r1 + 16) */
> >>>> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 16),
> >>>> -               /* *(u32 *)(r10 - 4) = r1 */
> >>>> -               BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_1, -4),
> >>>> -               BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
> >>>> -               BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
> >>>> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->qidconf_map_fd),
> >>>> +               /* Load index. */
> >>>> +               /* r6 = *(u32 *)(r1 + 16) */
> >>>> +               BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_ARG1, 16),
> >>>> +               /* w7 = w6 */
> >>>> +               BPF_MOV32_REG(BPF_REG_7, BPF_REG_6),
> >>>> +               /* w7 &= 2147483647 */
> >>>> +               BPF_ALU32_IMM(BPF_AND, BPF_REG_7, ~XDP_QID_FLAG_XSKRQ),
> >>>> +               /* *(u32 *)(r10 - 4) = r7 */
> >>>> +               BPF_STX_MEM(BPF_W, BPF_REG_FP, BPF_REG_7, -4),
> >>>> +
> >>>> +               /* Call bpf_map_lookup_elem. */
> >>>> +               /* r2 = r10 */
> >>>> +               BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP),
> >>>> +               /* r2 += -4 */
> >>>> +               BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -4),
> >>>> +               /* r1 = qidconf_map ll */
> >>>> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->qidconf_map_fd),
> >>>> +               /* call 1 */
> >>>>                   BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
> >>>> -               BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
> >>>> -               BPF_MOV32_IMM(BPF_REG_0, 0),
> >>>> -               /* if r1 == 0 goto +8 */
> >>>> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 8),
> >>>> -               BPF_MOV32_IMM(BPF_REG_0, 2),
> >>>> -               /* r1 = *(u32 *)(r1 + 0) */
> >>>> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 0),
> >>>> -               /* if r1 == 0 goto +5 */
> >>>> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 5),
> >>>> -               /* r2 = *(u32 *)(r10 - 4) */
> >>>> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),
> >>>> -               BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_10, -4),
> >>>> +
> >>>> +               /* Check the return value. */
> >>>> +               /* if r0 == 0 goto +14 */
> >>>> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 14),
> >>>> +
> >>>> +               /* Check qc == QIDCONF_XSK_COMBINED. */
> >>>> +               /* r6 >>= 31 */
> >>>> +               BPF_ALU64_IMM(BPF_RSH, BPF_REG_6, 31),
> >>>> +               /* r1 = *(u32 *)(r0 + 0) */
> >>>> +               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
> >>>> +               /* if r1 == 2 goto +1 */
> >>>> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, QIDCONF_XSK_COMBINED, 1),
> >>>> +
> >>>> +               /* qc != QIDCONF_XSK_COMBINED */
> >>>> +               /* r6 = r1 */
> >>>> +               BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
> >>>> +
> >>>> +               /* switch (qc) */
> >>>> +               /* w0 = 2 */
> >>>> +               BPF_MOV32_IMM(BPF_REG_0, XDP_PASS),
> >>>> +               /* if w6 == 0 goto +8 */
> >>>> +               BPF_JMP32_IMM(BPF_JEQ, BPF_REG_6, QIDCONF_REGULAR, 8),
> >>>> +               /* if w6 != 1 goto +6 */
> >>>> +               BPF_JMP32_IMM(BPF_JNE, BPF_REG_6, QIDCONF_XSK, 6),
> >>>> +
> >>>> +               /* Call bpf_redirect_map. */
> >>>> +               /* r1 = xsks_map ll */
> >>>> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->xsks_map_fd),
> >>>> +               /* w2 = w7 */
> >>>> +               BPF_MOV32_REG(BPF_REG_ARG2, BPF_REG_7),
> >>>> +               /* w3 = 0 */
> >>>>                   BPF_MOV32_IMM(BPF_REG_3, 0),
> >>>> +               /* call 51 */
> >>>>                   BPF_EMIT_CALL(BPF_FUNC_redirect_map),
> >>>> -               /* The jumps are to this instruction */
> >>>> +               /* exit */
> >>>> +               BPF_EXIT_INSN(),
> >>>> +
> >>>> +               /* XDP_ABORTED */
> >>>> +               /* w0 = 0 */
> >>>> +               BPF_MOV32_IMM(BPF_REG_0, XDP_ABORTED),
> >>>> +               /* exit */
> >>>>                   BPF_EXIT_INSN(),
> >>>>           };
> >>>>           size_t insns_cnt = sizeof(prog) / sizeof(struct bpf_insn);
> >>>> @@ -483,6 +538,7 @@ static int xsk_update_bpf_maps(struct xsk_socket *xsk, int qidconf_value,
> >>>>
> >>>>    static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
> >>>>    {
> >>>> +       int qidconf_value = QIDCONF_XSK;
> >>>>           bool prog_attached = false;
> >>>>           __u32 prog_id = 0;
> >>>>           int err;
> >>>> @@ -505,7 +561,11 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
> >>>>                   xsk->prog_fd = bpf_prog_get_fd_by_id(prog_id);
> >>>>           }
> >>>>
> >>>> -       err = xsk_update_bpf_maps(xsk, true, xsk->fd);
> >>>> +       if (xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
> >>>> +               if (xsk->zc)
> >>>> +                       qidconf_value = QIDCONF_XSK_COMBINED;
> >>>> +
> >>>> +       err = xsk_update_bpf_maps(xsk, qidconf_value, xsk->fd);
> >>>>           if (err)
> >>>>                   goto out_load;
> >>>>
> >>>> @@ -717,7 +777,7 @@ void xsk_socket__delete(struct xsk_socket *xsk)
> >>>>           if (!xsk)
> >>>>                   return;
> >>>>
> >>>> -       (void)xsk_update_bpf_maps(xsk, 0, 0);
> >>>> +       (void)xsk_update_bpf_maps(xsk, QIDCONF_REGULAR, 0);
> >>>>
> >>>>           optlen = sizeof(off);
> >>>>           err = getsockopt(xsk->fd, SOL_XDP, XDP_MMAP_OFFSETS, &off, &optlen);
> >>>> diff --git a/tools/lib/bpf/xsk.h b/tools/lib/bpf/xsk.h
> >>>> index 82ea71a0f3ec..be26a2423c04 100644
> >>>> --- a/tools/lib/bpf/xsk.h
> >>>> +++ b/tools/lib/bpf/xsk.h
> >>>> @@ -180,6 +180,10 @@ struct xsk_umem_config {
> >>>>
> >>>>    /* Flags for the libbpf_flags field. */
> >>>>    #define XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD (1 << 0)
> >>>> +#define XSK_LIBBPF_FLAGS__COMBINED_CHANNELS (1 << 1)
> >>>> +#define XSK_LIBBPF_FLAGS_MASK ( \
> >>>> +       XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD | \
> >>>> +       XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
> >>>>
> >>>>    struct xsk_socket_config {
> >>>>           __u32 rx_size;
> >>>> --
> >>>> 2.19.1
> >>>>
> >>
>

Re: [PATCH bpf-next v2 04/16] xsk: Extend channels to support combined XSK/non-XSK traffic

[Maxim Mikityanskiy]

On 2019-05-08 16:06, Magnus Karlsson wrote:
> On Tue, May 7, 2019 at 4:19 PM Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>>
>> On 2019-05-06 17:23, Magnus Karlsson wrote:
>>> On Mon, May 6, 2019 at 3:46 PM Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>>>>
>>>> On 2019-05-04 20:26, Björn Töpel wrote:
>>>>> On Tue, 30 Apr 2019 at 20:12, Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>>>>>>
>>>>>> Currently, the drivers that implement AF_XDP zero-copy support (e.g.,
>>>>>> i40e) switch the channel into a different mode when an XSK is opened. It
>>>>>> causes some issues that have to be taken into account. For example, RSS
>>>>>> needs to be reconfigured to skip the XSK-enabled channels, or the XDP
>>>>>> program should filter out traffic not intended for that socket and
>>>>>> XDP_PASS it with an additional copy. As nothing validates or forces the
>>>>>> proper configuration, it's easy to have packets drops, when they get
>>>>>> into an XSK by mistake, and, in fact, it's the default configuration.
>>>>>> There has to be some tool to have RSS reconfigured on each socket open
>>>>>> and close event, but such a tool is problematic to implement, because no
>>>>>> one reports these events, and it's race-prone.
>>>>>>
>>>>>> This commit extends XSK to support both kinds of traffic (XSK and
>>>>>> non-XSK) in the same channel. It implies having two RX queues in
>>>>>> XSK-enabled channels: one for the regular traffic, and the other for
>>>>>> XSK. It solves the problem with RSS: the default configuration just
>>>>>> works without the need to manually reconfigure RSS or to perform some
>>>>>> possibly complicated filtering in the XDP layer. It makes it easy to run
>>>>>> both AF_XDP and regular sockets on the same machine. In the XDP program,
>>>>>> the QID's most significant bit will serve as a flag to indicate whether
>>>>>> it's the XSK queue or not. The extension is compatible with the legacy
>>>>>> configuration, so if one wants to run the legacy mode, they can
>>>>>> reconfigure RSS and ignore the flag in the XDP program (implemented in
>>>>>> the reference XDP program in libbpf). mlx5e will support this extension.
>>>>>>
>>>>>> A single XDP program can run both with drivers supporting or not
>>>>>> supporting this extension. The xdpsock sample and libbpf are updated
>>>>>> accordingly.
>>>>>>
>>>>>
>>>>> I'm still not a fan of this, or maybe I'm not following you. It makes
>>>>> it more complex and even harder to use. Let's take a look at the
>>>>> kernel nomenclature. "ethtool" uses netdevs and channels. A channel is
>>>>> a Rx queue or a Tx queue.
>>>>
>>>> There are also combined channels that consist of an RX and a TX queue.
>>>> mlx5e has only this kind of channels. For us, a channel is a set of
>>>> queues "pinned to a CPU core" (they use the same NAPI).
>>>>
>>>>> In AF_XDP we call the channel a queue, which
>>>>> is what kernel uses internally (netdev_rx_queue, netdev_queue).
>>>>
>>>> You seem to agree it's a channel, right?
>>>>
>>>> AF_XDP doesn't allow to configure RX queue number and TX queue number
>>>> separately. Basically you choose a channel in AF_XDP. For some reason,
>>>> it's referred as a queue in some places, but logically it means "channel".
>>>
>>> You can configure the Rx queue and the Tx queue separately by creating
>>> two sockets tied to the same umem area. But if you just create one,
>>> you are correct.
>>
>> Yes, I know I can open two sockets, but it's only a workaround. If I
>> want to RX on RQ 3 and TX on SQ 5, I'll have one socket bound to RQ 3
>> and SQ 3, and the other bound to RQ 5 and SQ 5. It's not a clean way to
>> achieve the initial goal. It means that the current implementation
>> actually binds a socket to a channel X (that has RQ X and SQ X). It
>> could be different if there were two kinds of XSK sockets: RX-only and
>> TX-only, but it's not the case.
> 
> It is possible to create Rx only or Tx only AF_XDP sockets from the
> uapi by only creating an Rx or a Tx ring. Though, the NDO that
> registers the umem in the driver would need to be extended with this
> information so that the driver could act upon it. Today it is assumed
> that it is always Rx and Tx which wastes resources and maybe also
> performance. This has been on my todo list for quite some time, but it
> has never floated up to the top.
> 
>>>>> Today, AF_XDP can attach to an existing queue for ingress. (On the
>>>>> egress side, we're using "a queue", but the "XDP queue". XDP has these
>>>>> "shadow queues" which are separated from the netdev. This is a bit
>>>>> messy, and we can't really configure them. I believe Jakub has some
>>>>> ideas here. :-) For now, let's leave egress aside.)
>>>>
>>>> So, XDP already has "shadow queues" for TX, so I see no problem in
>>>> having the similar concept for AF_XDP RX.
>>>
>>> The question is if we would like to continue down the path of "shadow
>>> queues" by adding even more.
>>
>> OK, I think "shadow queues" is not a valid name, and it's certainly not
>> something bad. Initially, kernel had a concept of RX and TX queues. They
>> can match the queues the driver has for regular traffic. Then new
>> features appeared (XDP, then AF_XDP), and they required new types of
>> queues: XDP TX queue, AF_XDP RX queue. These are absolutely new
>> concepts, they aren't interchangeable with the legacy RX and TX queues.
>> It means we can't just say we have 32 TX queues, 16 of which are regular
>> SQs and 16 are XDP SQs. They function differently: the stack must not
>> try sending anything from an XDP SQ, and XDP_REDIRECT must not try
>> sending from the regular SQ.
>>
>> However, the kernel didn't learn about new types of queues. That's why
>> the new queues remain "in shadow". And we certainly shouldn't use the
>> same numeration for different types of queues, i.e. it's incorrect to
>> say that TX queues 0..15 are regular SQs, TX queues 16..31 are XDP SQs,
>> etc. The correct way is: there are 16 regular SQs 0..15 and 16 XDP SQs
>> 0..15.
>>
>> The same considerations apply to AF_XDP RX queues (XSK RQs). This is a
>> whole new type of queue, so it can't be mixed with the regular RQs.
>> That's why they have their own numeration and are not accounted as RX
>> queues (moreover, their numeration can be non-contiguous). If the kernel
>> had need to know about them, they could be exposed to the netdev as a
>> third type of queue, but I see no need currently.
>>
>>> In the busy-poll RFC I sent out last
>>> week, I talk about the possibility to create a new queue (set) not
>>> tied to the napi of the regular Rx queues in order to get better
>>> performance when using busy-poll. How would such a queue set fit into
>>> a shadow queue set approach?
>>
>> Looking at the question in the RFC, I don't see in what way it
>> interferes with XSK RQs. Moreover, it's even more convenient to have a
>> separate type of queue - you'll be able to create and destroy "unbound"
>> (not driven by an interrupt, having a separate NAPI) XSK RQs without
>> interfering with the regular queues. At first sight, I would say it's a
>> perfect fit :). For me, it looks cleaner when we have regular RQs in
>> channels, XSK RQs in channels and "unbound" XSK RQs than having channels
>> that switch the RQ type and having some extra out-of-channel RQs that
>> behave differently, but look the same to the kernel.
> 
> Agree with you that they are not interchangeable and that they
> should be explicit. So we are on the same page here. Good. But
> let us try to solve the Tx problem first because it has been
> there from day one for both XDP and AF_XDP. Jakub Kicinski has
> some ideas in this direction that he presented at Linux Kernel
> Developers' bpfconf 2019
> (http://vger.kernel.org/bpfconf2019.html) where he suggested to
> start with Tx first. Let us start a discussion around that. Many
> of the concepts developed there should hopefully extend to the Rx
> side too.

Solving the problem with TX first isn't something that helps get my code 
in. I'm not denying the problem - but it's not related to this patchset.

Regarding XSK RQs, currently things work without registering XSK RQs in 
the kernel. If a need for it arises, it can be addressed later, together 
with XDP TX queues. The new feature doesn't conflict with your RFC, and 
if you push the dedicated NAPI context feature, I'll be able to adapt 
easily with my feature, and it won't be the case without my feature.

> Note that AF_XDP is today using the Rx queues that XDP operates
> on (because it is an extension of XDP) both in the uapi and in
> the driver. And these are the regular netdev Rx queues. But as we
> all agree on, we need to change this so the choice becomes more
> flexible and plays nicely with all side band configuration tools.
> 
>>> When does hiding the real queues created
>>> to support various features break and we have to expose the real queue
>>> number? Trying to wrap my head around these questions.
>>
>> As I said above, different types of queues should not share the
>> numeration. So, if need be, they can be exposed as a different type, but
>> I don't see the necessity yet.
>>
>>> Maxim, would it be possible for you to respin this set without this
>>> feature?
>>
>> That's not as easy as you may think... It's not just a feature - for the
>> kernel it's just a feature, but for the driver it's part of core design
>> of AF_XDP support in mlx5. Removing it will require a lot of changes,
>> reviewing and testing to adapt the driver to a different approach. I'd
>> prefer to come to the conclusion before thinking of such changes :)
> 
> I leave it up to you if you would like to wait to submit the next
> version of your patch set until the "queue id" problem of both Rx
> and Tx has been resolved. I hope that it will be speedy so that
> you can get your patch set in, but it might not be. It would have
> been better if you had partitioned your patch set from the start
> as: basic support -> new uapi/kernel feature -> implementation of
> the new feature in your driver.
I'm not sure you got it right. You probably call "basic support" an 
implementation similar to the one in i40e, right? The problem is that 
the new feature doesn't complement "basic support", it replaces it. It's 
impossible to split this series into "basic support" and "new feature" 
parts. Our design is based on the new feature, and to split the series 
means to throw away the whole design, to start from scratch and 
implement "basic support", then remove it immediately with the "new 
feature" patch. It doesn't make sense for me to do it this way.

Thanks,
Max

> Then we could have acked the
> basic support quickly. It is good stuff.
> 
> Thanks: Magnus
> 
>> Thanks for giving the feedback and joining this discussion!
>>
>>> I like the other stuff you have implemented and think that
>>> the rest of the common functionality should be useful for all of us.
>>> This way you can get the AF_XDP support accepted quicker while we
>>> debate the best way to solve the issue in this thread.
>>>
>>> Thanks for all your work: Magnus
>>>
>>>>> If an application would like to get all the traffic from a netdev,
>>>>> it'll create an equal amout of sockets as the queues and bind to the
>>>>> queues. Yes, even the queues in the RSS  set.
>>>>>
>>>>> What you would like (I think):
>>>>> a) is a way of spawning a new queue for a netdev, that is not part of
>>>>> the stack and/or RSS set
>>>>
>>>> Yes - for the simplicity sake and to make configuration easier. The only
>>>> thing needed is to steer the traffic and to open an AF_XDP socket on
>>>> channel X. We don't need to care about removing the queue out of RSS,
>>>> about finding a way to administer this (which is hard because it's racy
>>>> if the configuration in not known in advance). So I don't agree I'm
>>>> complicating things, my goal is to make them easier.
>>>>
>>>>> b) steering traffic to that queue using a configuration mechanism (tc?
>>>>> some yet to be hacked BPF configuration hook?)
>>>>
>>>> Currently, ethtool --config-ntuple is used to steer the traffic. The
>>>> user-def argument has a bit that selects XSK RQ/regular RQ, and action
>>>> selects a channel:
>>>>
>>>> ethtool -N eth0 flow-type udp4 dst-port 4242 action 3 user-def 1
>>>>
>>>>> With your mechanism you're doing this in contrived way. This makes the
>>>>> existing AF_XDP model *more* complex/hard(er) to use.
>>>>
>>>> No, as I said above, some issues are eliminated with my approach, and no
>>>> new limitations are introduced, so it makes things more universal and
>>>> simpler to configure.
>>>>
>>>>> How do you steer traffic to this dual-channel RQ?
>>>>
>>>> First, there is no dual-channel RQ, a more accurate term is dual-RQ
>>>> channel, cause now the channel contains a regular RQ and can contain an
>>>> XSK RQ.
>>>>
>>>> For the steering itself, see the ethtool command above - the user-def
>>>> argument has a bit that selects one of two RQs.
>>>>
>>>>> So you have a netdev
>>>>> receiving on all queues. Then, e.g., the last queue is a "dual
>>>>> channel" queue that can receive traffic from some other filter. How do
>>>>> you use it?
>>>>
>>>> If I want to take the last (or some) channel and start using AF_XDP with
>>>> it, I simply configure steering to the XSK RQ of that channel and open a
>>>> socket specifying the channel number. I don't need to reconfigure RSS,
>>>> because RSS packets go to the regular RQ of that channel and don't
>>>> interfere with XSK.
>>>>
>>>> No functionality is lost - if you don't distinguish the regular and XSK
>>>> RQs on the XDP level, you'll get the same effect as with i40e's
>>>> implementation. If you want to dedicate the CPU core and channel solely
>>>> for AF_XDP, in i40e you exclude the channel from RSS, and here you can
>>>> do exactly the same thing. So, no use case is complicated comparing to
>>>> i40e, but there are use cases where this feature is to advantage.
>>>>
>>>> I hope I explained the points you were interested in. Please ask more
>>>> questions if there is still something that I should clarify regarding
>>>> this topic.
>>>>
>>>> Thanks,
>>>> Max
>>>>
>>>>>
>>>>>
>>>>> Björn
>>>>>
>>>>>> Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
>>>>>> Acked-by: Saeed Mahameed <saeedm@mellanox.com>
>>>>>> ---
>>>>>>     include/uapi/linux/if_xdp.h       |  11 +++
>>>>>>     net/xdp/xsk.c                     |   5 +-
>>>>>>     samples/bpf/xdpsock_user.c        |  10 ++-
>>>>>>     tools/include/uapi/linux/if_xdp.h |  11 +++
>>>>>>     tools/lib/bpf/xsk.c               | 116 ++++++++++++++++++++++--------
>>>>>>     tools/lib/bpf/xsk.h               |   4 ++
>>>>>>     6 files changed, 126 insertions(+), 31 deletions(-)
>>>>>>
>>>>>> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
>>>>>> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
>>>>>> --- a/include/uapi/linux/if_xdp.h
>>>>>> +++ b/include/uapi/linux/if_xdp.h
>>>>>> @@ -82,4 +82,15 @@ struct xdp_desc {
>>>>>>
>>>>>>     /* UMEM descriptor is __u64 */
>>>>>>
>>>>>> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
>>>>>> + * this flag bit in the queue index to distinguish between two RQs of the same
>>>>>> + * channel.
>>>>>> + */
>>>>>> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
>>>>>> +
>>>>>> +static inline __u32 xdp_qid_get_channel(__u32 qid)
>>>>>> +{
>>>>>> +       return qid & ~XDP_QID_FLAG_XSKRQ;
>>>>>> +}
>>>>>> +
>>>>>>     #endif /* _LINUX_IF_XDP_H */
>>>>>> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
>>>>>> index 998199109d5c..114ba17acb09 100644
>>>>>> --- a/net/xdp/xsk.c
>>>>>> +++ b/net/xdp/xsk.c
>>>>>> @@ -104,9 +104,12 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
>>>>>>
>>>>>>     int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
>>>>>>     {
>>>>>> +       struct xdp_rxq_info *rxq = xdp->rxq;
>>>>>> +       u32 channel = xdp_qid_get_channel(rxq->queue_index);
>>>>>>            u32 len;
>>>>>>
>>>>>> -       if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
>>>>>> +       if (xs->dev != rxq->dev || xs->queue_id != channel ||
>>>>>> +           xs->zc != (rxq->mem.type == MEM_TYPE_ZERO_COPY))
>>>>>>                    return -EINVAL;
>>>>>>
>>>>>>            len = xdp->data_end - xdp->data;
>>>>>> diff --git a/samples/bpf/xdpsock_user.c b/samples/bpf/xdpsock_user.c
>>>>>> index d08ee1ab7bb4..a6b13025ee79 100644
>>>>>> --- a/samples/bpf/xdpsock_user.c
>>>>>> +++ b/samples/bpf/xdpsock_user.c
>>>>>> @@ -62,6 +62,7 @@ enum benchmark_type {
>>>>>>
>>>>>>     static enum benchmark_type opt_bench = BENCH_RXDROP;
>>>>>>     static u32 opt_xdp_flags = XDP_FLAGS_UPDATE_IF_NOEXIST;
>>>>>> +static u32 opt_libbpf_flags;
>>>>>>     static const char *opt_if = "";
>>>>>>     static int opt_ifindex;
>>>>>>     static int opt_queue;
>>>>>> @@ -306,7 +307,7 @@ static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem)
>>>>>>            xsk->umem = umem;
>>>>>>            cfg.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS;
>>>>>>            cfg.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS;
>>>>>> -       cfg.libbpf_flags = 0;
>>>>>> +       cfg.libbpf_flags = opt_libbpf_flags;
>>>>>>            cfg.xdp_flags = opt_xdp_flags;
>>>>>>            cfg.bind_flags = opt_xdp_bind_flags;
>>>>>>            ret = xsk_socket__create(&xsk->xsk, opt_if, opt_queue, umem->umem,
>>>>>> @@ -346,6 +347,7 @@ static struct option long_options[] = {
>>>>>>            {"interval", required_argument, 0, 'n'},
>>>>>>            {"zero-copy", no_argument, 0, 'z'},
>>>>>>            {"copy", no_argument, 0, 'c'},
>>>>>> +       {"combined", no_argument, 0, 'C'},
>>>>>>            {0, 0, 0, 0}
>>>>>>     };
>>>>>>
>>>>>> @@ -365,6 +367,7 @@ static void usage(const char *prog)
>>>>>>                    "  -n, --interval=n     Specify statistics update interval (default 1 sec).\n"
>>>>>>                    "  -z, --zero-copy      Force zero-copy mode.\n"
>>>>>>                    "  -c, --copy           Force copy mode.\n"
>>>>>> +               "  -C, --combined       Driver supports combined XSK and non-XSK traffic in a channel.\n"
>>>>>>                    "\n";
>>>>>>            fprintf(stderr, str, prog);
>>>>>>            exit(EXIT_FAILURE);
>>>>>> @@ -377,7 +380,7 @@ static void parse_command_line(int argc, char **argv)
>>>>>>            opterr = 0;
>>>>>>
>>>>>>            for (;;) {
>>>>>> -               c = getopt_long(argc, argv, "Frtli:q:psSNn:cz", long_options,
>>>>>> +               c = getopt_long(argc, argv, "Frtli:q:psSNn:czC", long_options,
>>>>>>                                    &option_index);
>>>>>>                    if (c == -1)
>>>>>>                            break;
>>>>>> @@ -420,6 +423,9 @@ static void parse_command_line(int argc, char **argv)
>>>>>>                    case 'F':
>>>>>>                            opt_xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
>>>>>>                            break;
>>>>>> +               case 'C':
>>>>>> +                       opt_libbpf_flags |= XSK_LIBBPF_FLAGS__COMBINED_CHANNELS;
>>>>>> +                       break;
>>>>>>                    default:
>>>>>>                            usage(basename(argv[0]));
>>>>>>                    }
>>>>>> diff --git a/tools/include/uapi/linux/if_xdp.h b/tools/include/uapi/linux/if_xdp.h
>>>>>> index 9ae4b4e08b68..cf6ff1ecc6bd 100644
>>>>>> --- a/tools/include/uapi/linux/if_xdp.h
>>>>>> +++ b/tools/include/uapi/linux/if_xdp.h
>>>>>> @@ -82,4 +82,15 @@ struct xdp_desc {
>>>>>>
>>>>>>     /* UMEM descriptor is __u64 */
>>>>>>
>>>>>> +/* The driver may run a dedicated XSK RQ in the channel. The XDP program uses
>>>>>> + * this flag bit in the queue index to distinguish between two RQs of the same
>>>>>> + * channel.
>>>>>> + */
>>>>>> +#define XDP_QID_FLAG_XSKRQ (1 << 31)
>>>>>> +
>>>>>> +static inline __u32 xdp_qid_get_channel(__u32 qid)
>>>>>> +{
>>>>>> +       return qid & ~XDP_QID_FLAG_XSKRQ;
>>>>>> +}
>>>>>> +
>>>>>>     #endif /* _LINUX_IF_XDP_H */
>>>>>> diff --git a/tools/lib/bpf/xsk.c b/tools/lib/bpf/xsk.c
>>>>>> index a95b06d1f81d..969dfd856039 100644
>>>>>> --- a/tools/lib/bpf/xsk.c
>>>>>> +++ b/tools/lib/bpf/xsk.c
>>>>>> @@ -76,6 +76,12 @@ struct xsk_nl_info {
>>>>>>            int fd;
>>>>>>     };
>>>>>>
>>>>>> +enum qidconf {
>>>>>> +       QIDCONF_REGULAR,
>>>>>> +       QIDCONF_XSK,
>>>>>> +       QIDCONF_XSK_COMBINED,
>>>>>> +};
>>>>>> +
>>>>>>     /* For 32-bit systems, we need to use mmap2 as the offsets are 64-bit.
>>>>>>      * Unfortunately, it is not part of glibc.
>>>>>>      */
>>>>>> @@ -139,7 +145,7 @@ static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg,
>>>>>>                    return 0;
>>>>>>            }
>>>>>>
>>>>>> -       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD)
>>>>>> +       if (usr_cfg->libbpf_flags & ~XSK_LIBBPF_FLAGS_MASK)
>>>>>>                    return -EINVAL;
>>>>>>
>>>>>>            cfg->rx_size = usr_cfg->rx_size;
>>>>>> @@ -267,44 +273,93 @@ static int xsk_load_xdp_prog(struct xsk_socket *xsk)
>>>>>>            /* This is the C-program:
>>>>>>             * SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx)
>>>>>>             * {
>>>>>> -        *     int *qidconf, index = ctx->rx_queue_index;
>>>>>> +        *     int *qidconf, qc;
>>>>>> +        *     int index = ctx->rx_queue_index & ~(1 << 31);
>>>>>> +        *     bool is_xskrq = ctx->rx_queue_index & (1 << 31);
>>>>>>             *
>>>>>> -        *     // A set entry here means that the correspnding queue_id
>>>>>> -        *     // has an active AF_XDP socket bound to it.
>>>>>> +        *     // A set entry here means that the corresponding queue_id
>>>>>> +        *     // has an active AF_XDP socket bound to it. Value 2 means
>>>>>> +        *     // it's zero-copy multi-RQ mode.
>>>>>>             *     qidconf = bpf_map_lookup_elem(&qidconf_map, &index);
>>>>>>             *     if (!qidconf)
>>>>>>             *         return XDP_ABORTED;
>>>>>>             *
>>>>>> -        *     if (*qidconf)
>>>>>> +        *     qc = *qidconf;
>>>>>> +        *
>>>>>> +        *     if (qc == 2)
>>>>>> +        *         qc = is_xskrq ? 1 : 0;
>>>>>> +        *
>>>>>> +        *     switch (qc) {
>>>>>> +        *     case 0:
>>>>>> +        *         return XDP_PASS;
>>>>>> +        *     case 1:
>>>>>>             *         return bpf_redirect_map(&xsks_map, index, 0);
>>>>>> +        *     }
>>>>>>             *
>>>>>> -        *     return XDP_PASS;
>>>>>> +        *     return XDP_ABORTED;
>>>>>>             * }
>>>>>>             */
>>>>>>            struct bpf_insn prog[] = {
>>>>>> -               /* r1 = *(u32 *)(r1 + 16) */
>>>>>> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 16),
>>>>>> -               /* *(u32 *)(r10 - 4) = r1 */
>>>>>> -               BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_1, -4),
>>>>>> -               BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
>>>>>> -               BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),
>>>>>> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->qidconf_map_fd),
>>>>>> +               /* Load index. */
>>>>>> +               /* r6 = *(u32 *)(r1 + 16) */
>>>>>> +               BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_ARG1, 16),
>>>>>> +               /* w7 = w6 */
>>>>>> +               BPF_MOV32_REG(BPF_REG_7, BPF_REG_6),
>>>>>> +               /* w7 &= 2147483647 */
>>>>>> +               BPF_ALU32_IMM(BPF_AND, BPF_REG_7, ~XDP_QID_FLAG_XSKRQ),
>>>>>> +               /* *(u32 *)(r10 - 4) = r7 */
>>>>>> +               BPF_STX_MEM(BPF_W, BPF_REG_FP, BPF_REG_7, -4),
>>>>>> +
>>>>>> +               /* Call bpf_map_lookup_elem. */
>>>>>> +               /* r2 = r10 */
>>>>>> +               BPF_MOV64_REG(BPF_REG_ARG2, BPF_REG_FP),
>>>>>> +               /* r2 += -4 */
>>>>>> +               BPF_ALU64_IMM(BPF_ADD, BPF_REG_ARG2, -4),
>>>>>> +               /* r1 = qidconf_map ll */
>>>>>> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->qidconf_map_fd),
>>>>>> +               /* call 1 */
>>>>>>                    BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem),
>>>>>> -               BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
>>>>>> -               BPF_MOV32_IMM(BPF_REG_0, 0),
>>>>>> -               /* if r1 == 0 goto +8 */
>>>>>> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 8),
>>>>>> -               BPF_MOV32_IMM(BPF_REG_0, 2),
>>>>>> -               /* r1 = *(u32 *)(r1 + 0) */
>>>>>> -               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_1, 0),
>>>>>> -               /* if r1 == 0 goto +5 */
>>>>>> -               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 5),
>>>>>> -               /* r2 = *(u32 *)(r10 - 4) */
>>>>>> -               BPF_LD_MAP_FD(BPF_REG_1, xsk->xsks_map_fd),
>>>>>> -               BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_10, -4),
>>>>>> +
>>>>>> +               /* Check the return value. */
>>>>>> +               /* if r0 == 0 goto +14 */
>>>>>> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 14),
>>>>>> +
>>>>>> +               /* Check qc == QIDCONF_XSK_COMBINED. */
>>>>>> +               /* r6 >>= 31 */
>>>>>> +               BPF_ALU64_IMM(BPF_RSH, BPF_REG_6, 31),
>>>>>> +               /* r1 = *(u32 *)(r0 + 0) */
>>>>>> +               BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0),
>>>>>> +               /* if r1 == 2 goto +1 */
>>>>>> +               BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, QIDCONF_XSK_COMBINED, 1),
>>>>>> +
>>>>>> +               /* qc != QIDCONF_XSK_COMBINED */
>>>>>> +               /* r6 = r1 */
>>>>>> +               BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
>>>>>> +
>>>>>> +               /* switch (qc) */
>>>>>> +               /* w0 = 2 */
>>>>>> +               BPF_MOV32_IMM(BPF_REG_0, XDP_PASS),
>>>>>> +               /* if w6 == 0 goto +8 */
>>>>>> +               BPF_JMP32_IMM(BPF_JEQ, BPF_REG_6, QIDCONF_REGULAR, 8),
>>>>>> +               /* if w6 != 1 goto +6 */
>>>>>> +               BPF_JMP32_IMM(BPF_JNE, BPF_REG_6, QIDCONF_XSK, 6),
>>>>>> +
>>>>>> +               /* Call bpf_redirect_map. */
>>>>>> +               /* r1 = xsks_map ll */
>>>>>> +               BPF_LD_MAP_FD(BPF_REG_ARG1, xsk->xsks_map_fd),
>>>>>> +               /* w2 = w7 */
>>>>>> +               BPF_MOV32_REG(BPF_REG_ARG2, BPF_REG_7),
>>>>>> +               /* w3 = 0 */
>>>>>>                    BPF_MOV32_IMM(BPF_REG_3, 0),
>>>>>> +               /* call 51 */
>>>>>>                    BPF_EMIT_CALL(BPF_FUNC_redirect_map),
>>>>>> -               /* The jumps are to this instruction */
>>>>>> +               /* exit */
>>>>>> +               BPF_EXIT_INSN(),
>>>>>> +
>>>>>> +               /* XDP_ABORTED */
>>>>>> +               /* w0 = 0 */
>>>>>> +               BPF_MOV32_IMM(BPF_REG_0, XDP_ABORTED),
>>>>>> +               /* exit */
>>>>>>                    BPF_EXIT_INSN(),
>>>>>>            };
>>>>>>            size_t insns_cnt = sizeof(prog) / sizeof(struct bpf_insn);
>>>>>> @@ -483,6 +538,7 @@ static int xsk_update_bpf_maps(struct xsk_socket *xsk, int qidconf_value,
>>>>>>
>>>>>>     static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
>>>>>>     {
>>>>>> +       int qidconf_value = QIDCONF_XSK;
>>>>>>            bool prog_attached = false;
>>>>>>            __u32 prog_id = 0;
>>>>>>            int err;
>>>>>> @@ -505,7 +561,11 @@ static int xsk_setup_xdp_prog(struct xsk_socket *xsk)
>>>>>>                    xsk->prog_fd = bpf_prog_get_fd_by_id(prog_id);
>>>>>>            }
>>>>>>
>>>>>> -       err = xsk_update_bpf_maps(xsk, true, xsk->fd);
>>>>>> +       if (xsk->config.libbpf_flags & XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
>>>>>> +               if (xsk->zc)
>>>>>> +                       qidconf_value = QIDCONF_XSK_COMBINED;
>>>>>> +
>>>>>> +       err = xsk_update_bpf_maps(xsk, qidconf_value, xsk->fd);
>>>>>>            if (err)
>>>>>>                    goto out_load;
>>>>>>
>>>>>> @@ -717,7 +777,7 @@ void xsk_socket__delete(struct xsk_socket *xsk)
>>>>>>            if (!xsk)
>>>>>>                    return;
>>>>>>
>>>>>> -       (void)xsk_update_bpf_maps(xsk, 0, 0);
>>>>>> +       (void)xsk_update_bpf_maps(xsk, QIDCONF_REGULAR, 0);
>>>>>>
>>>>>>            optlen = sizeof(off);
>>>>>>            err = getsockopt(xsk->fd, SOL_XDP, XDP_MMAP_OFFSETS, &off, &optlen);
>>>>>> diff --git a/tools/lib/bpf/xsk.h b/tools/lib/bpf/xsk.h
>>>>>> index 82ea71a0f3ec..be26a2423c04 100644
>>>>>> --- a/tools/lib/bpf/xsk.h
>>>>>> +++ b/tools/lib/bpf/xsk.h
>>>>>> @@ -180,6 +180,10 @@ struct xsk_umem_config {
>>>>>>
>>>>>>     /* Flags for the libbpf_flags field. */
>>>>>>     #define XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD (1 << 0)
>>>>>> +#define XSK_LIBBPF_FLAGS__COMBINED_CHANNELS (1 << 1)
>>>>>> +#define XSK_LIBBPF_FLAGS_MASK ( \
>>>>>> +       XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD | \
>>>>>> +       XSK_LIBBPF_FLAGS__COMBINED_CHANNELS)
>>>>>>
>>>>>>     struct xsk_socket_config {
>>>>>>            __u32 rx_size;
>>>>>> --
>>>>>> 2.19.1
>>>>>>
>>>>
>>

[PATCH bpf-next v2 05/16] xsk: Change the default frame size to 4096 and allow controlling it

[Maxim Mikityanskiy]

The typical XDP memory scheme is one packet per page. Change the AF_XDP
frame size in libbpf to 4096, which is the page size on x86, to allow
libbpf to be used with the drivers with the packet-per-page scheme.

Add a command line option -f to xdpsock to allow to specify a custom
frame size.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 samples/bpf/xdpsock_user.c | 44 ++++++++++++++++++++++++--------------
 tools/lib/bpf/xsk.h        |  2 +-
 2 files changed, 29 insertions(+), 17 deletions(-)

diff --git a/samples/bpf/xdpsock_user.c b/samples/bpf/xdpsock_user.c
index a6b13025ee79..9a2fd899c3e0 100644
--- a/samples/bpf/xdpsock_user.c
+++ b/samples/bpf/xdpsock_user.c
@@ -69,6 +69,7 @@ static int opt_queue;
 static int opt_poll;
 static int opt_interval = 1;
 static u32 opt_xdp_bind_flags;
+static int opt_xsk_frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE;
 static __u32 prog_id;
 
 struct xsk_umem_info {
@@ -277,6 +278,12 @@ static size_t gen_eth_frame(struct xsk_umem_info *umem, u64 addr)
 static struct xsk_umem_info *xsk_configure_umem(void *buffer, u64 size)
 {
 	struct xsk_umem_info *umem;
+	struct xsk_umem_config cfg = {
+		.fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS,
+		.comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS,
+		.frame_size = opt_xsk_frame_size,
+		.frame_headroom = XSK_UMEM__DEFAULT_FRAME_HEADROOM,
+	};
 	int ret;
 
 	umem = calloc(1, sizeof(*umem));
@@ -284,7 +291,7 @@ static struct xsk_umem_info *xsk_configure_umem(void *buffer, u64 size)
 		exit_with_error(errno);
 
 	ret = xsk_umem__create(&umem->umem, buffer, size, &umem->fq, &umem->cq,
-			       NULL);
+			       &cfg);
 	if (ret)
 		exit_with_error(-ret);
 
@@ -324,11 +331,9 @@ static struct xsk_socket_info *xsk_configure_socket(struct xsk_umem_info *umem)
 				     &idx);
 	if (ret != XSK_RING_PROD__DEFAULT_NUM_DESCS)
 		exit_with_error(-ret);
-	for (i = 0;
-	     i < XSK_RING_PROD__DEFAULT_NUM_DESCS *
-		     XSK_UMEM__DEFAULT_FRAME_SIZE;
-	     i += XSK_UMEM__DEFAULT_FRAME_SIZE)
-		*xsk_ring_prod__fill_addr(&xsk->umem->fq, idx++) = i;
+	for (i = 0; i < XSK_RING_PROD__DEFAULT_NUM_DESCS; i++)
+		*xsk_ring_prod__fill_addr(&xsk->umem->fq, idx++) =
+			i * opt_xsk_frame_size;
 	xsk_ring_prod__submit(&xsk->umem->fq,
 			      XSK_RING_PROD__DEFAULT_NUM_DESCS);
 
@@ -348,6 +353,7 @@ static struct option long_options[] = {
 	{"zero-copy", no_argument, 0, 'z'},
 	{"copy", no_argument, 0, 'c'},
 	{"combined", no_argument, 0, 'C'},
+	{"frame-size", required_argument, 0, 'f'},
 	{0, 0, 0, 0}
 };
 
@@ -368,8 +374,9 @@ static void usage(const char *prog)
 		"  -z, --zero-copy      Force zero-copy mode.\n"
 		"  -c, --copy           Force copy mode.\n"
 		"  -C, --combined       Driver supports combined XSK and non-XSK traffic in a channel.\n"
+		"  -f, --frame-size=n   Set the frame size (must be a power of two, default is %d).\n"
 		"\n";
-	fprintf(stderr, str, prog);
+	fprintf(stderr, str, prog, XSK_UMEM__DEFAULT_FRAME_SIZE);
 	exit(EXIT_FAILURE);
 }
 
@@ -380,7 +387,7 @@ static void parse_command_line(int argc, char **argv)
 	opterr = 0;
 
 	for (;;) {
-		c = getopt_long(argc, argv, "Frtli:q:psSNn:czC", long_options,
+		c = getopt_long(argc, argv, "Frtli:q:psSNn:czCf:", long_options,
 				&option_index);
 		if (c == -1)
 			break;
@@ -426,6 +433,9 @@ static void parse_command_line(int argc, char **argv)
 		case 'C':
 			opt_libbpf_flags |= XSK_LIBBPF_FLAGS__COMBINED_CHANNELS;
 			break;
+		case 'f':
+			opt_xsk_frame_size = atoi(optarg);
+			break;
 		default:
 			usage(basename(argv[0]));
 		}
@@ -438,6 +448,11 @@ static void parse_command_line(int argc, char **argv)
 		usage(basename(argv[0]));
 	}
 
+	if (opt_xsk_frame_size & (opt_xsk_frame_size - 1)) {
+		fprintf(stderr, "--frame-size=%d is not a power of two\n",
+			opt_xsk_frame_size);
+		usage(basename(argv[0]));
+	}
 }
 
 static void kick_tx(struct xsk_socket_info *xsk)
@@ -589,8 +604,7 @@ static void tx_only(struct xsk_socket_info *xsk)
 
 			for (i = 0; i < BATCH_SIZE; i++) {
 				xsk_ring_prod__tx_desc(&xsk->tx, idx + i)->addr
-					= (frame_nb + i) <<
-					XSK_UMEM__DEFAULT_FRAME_SHIFT;
+					= (frame_nb + i) * opt_xsk_frame_size;
 				xsk_ring_prod__tx_desc(&xsk->tx, idx + i)->len =
 					sizeof(pkt_data) - 1;
 			}
@@ -667,21 +681,19 @@ int main(int argc, char **argv)
 	}
 
 	ret = posix_memalign(&bufs, getpagesize(), /* PAGE_SIZE aligned */
-			     NUM_FRAMES * XSK_UMEM__DEFAULT_FRAME_SIZE);
+			     NUM_FRAMES * opt_xsk_frame_size);
 	if (ret)
 		exit_with_error(ret);
 
        /* Create sockets... */
-	umem = xsk_configure_umem(bufs,
-				  NUM_FRAMES * XSK_UMEM__DEFAULT_FRAME_SIZE);
+	umem = xsk_configure_umem(bufs, NUM_FRAMES * opt_xsk_frame_size);
 	xsks[num_socks++] = xsk_configure_socket(umem);
 
 	if (opt_bench == BENCH_TXONLY) {
 		int i;
 
-		for (i = 0; i < NUM_FRAMES * XSK_UMEM__DEFAULT_FRAME_SIZE;
-		     i += XSK_UMEM__DEFAULT_FRAME_SIZE)
-			(void)gen_eth_frame(umem, i);
+		for (i = 0; i < NUM_FRAMES; i++)
+			(void)gen_eth_frame(umem, i * opt_xsk_frame_size);
 	}
 
 	signal(SIGINT, int_exit);
diff --git a/tools/lib/bpf/xsk.h b/tools/lib/bpf/xsk.h
index be26a2423c04..31d3db1bd1d5 100644
--- a/tools/lib/bpf/xsk.h
+++ b/tools/lib/bpf/xsk.h
@@ -167,7 +167,7 @@ LIBBPF_API int xsk_socket__fd(const struct xsk_socket *xsk);
 
 #define XSK_RING_CONS__DEFAULT_NUM_DESCS      2048
 #define XSK_RING_PROD__DEFAULT_NUM_DESCS      2048
-#define XSK_UMEM__DEFAULT_FRAME_SHIFT    11 /* 2048 bytes */
+#define XSK_UMEM__DEFAULT_FRAME_SHIFT    12 /* 4096 bytes */
 #define XSK_UMEM__DEFAULT_FRAME_SIZE     (1 << XSK_UMEM__DEFAULT_FRAME_SHIFT)
 #define XSK_UMEM__DEFAULT_FRAME_HEADROOM 0
 
-- 
2.19.1

[PATCH bpf-next v2 06/16] xsk: Return the whole xdp_desc from xsk_umem_consume_tx

[Maxim Mikityanskiy]

Some drivers want to access the data transmitted in order to implement
acceleration features of the NICs. It is also useful in AF_XDP TX flow.

Change the xsk_umem_consume_tx API to return the whole xdp_desc, that
contains the data pointer, length and DMA address, instead of only the
latter two. Adapt the implementation of i40e and ixgbe to this change.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
Cc: Björn Töpel <bjorn.topel@intel.com>
Cc: Magnus Karlsson <magnus.karlsson@intel.com>
---
 drivers/net/ethernet/intel/i40e/i40e_xsk.c   | 12 +++++++-----
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 15 +++++++++------
 include/net/xdp_sock.h                       |  6 +++---
 net/xdp/xsk.c                                | 10 +++-------
 4 files changed, 22 insertions(+), 21 deletions(-)

diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c
index 1b17486543ac..eae6fafad1b8 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c
@@ -640,8 +640,8 @@ static bool i40e_xmit_zc(struct i40e_ring *xdp_ring, unsigned int budget)
 	struct i40e_tx_desc *tx_desc = NULL;
 	struct i40e_tx_buffer *tx_bi;
 	bool work_done = true;
+	struct xdp_desc desc;
 	dma_addr_t dma;
-	u32 len;
 
 	while (budget-- > 0) {
 		if (!unlikely(I40E_DESC_UNUSED(xdp_ring))) {
@@ -650,21 +650,23 @@ static bool i40e_xmit_zc(struct i40e_ring *xdp_ring, unsigned int budget)
 			break;
 		}
 
-		if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &dma, &len))
+		if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &desc))
 			break;
 
-		dma_sync_single_for_device(xdp_ring->dev, dma, len,
+		dma = xdp_umem_get_dma(xdp_ring->xsk_umem, desc.addr);
+
+		dma_sync_single_for_device(xdp_ring->dev, dma, desc.len,
 					   DMA_BIDIRECTIONAL);
 
 		tx_bi = &xdp_ring->tx_bi[xdp_ring->next_to_use];
-		tx_bi->bytecount = len;
+		tx_bi->bytecount = desc.len;
 
 		tx_desc = I40E_TX_DESC(xdp_ring, xdp_ring->next_to_use);
 		tx_desc->buffer_addr = cpu_to_le64(dma);
 		tx_desc->cmd_type_offset_bsz =
 			build_ctob(I40E_TX_DESC_CMD_ICRC
 				   | I40E_TX_DESC_CMD_EOP,
-				   0, len, 0);
+				   0, desc.len, 0);
 
 		xdp_ring->next_to_use++;
 		if (xdp_ring->next_to_use == xdp_ring->count)
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
index bfe95ce0bd7f..0297a70a4e2d 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
@@ -621,8 +621,9 @@ static bool ixgbe_xmit_zc(struct ixgbe_ring *xdp_ring, unsigned int budget)
 	union ixgbe_adv_tx_desc *tx_desc = NULL;
 	struct ixgbe_tx_buffer *tx_bi;
 	bool work_done = true;
-	u32 len, cmd_type;
+	struct xdp_desc desc;
 	dma_addr_t dma;
+	u32 cmd_type;
 
 	while (budget-- > 0) {
 		if (unlikely(!ixgbe_desc_unused(xdp_ring)) ||
@@ -631,14 +632,16 @@ static bool ixgbe_xmit_zc(struct ixgbe_ring *xdp_ring, unsigned int budget)
 			break;
 		}
 
-		if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &dma, &len))
+		if (!xsk_umem_consume_tx(xdp_ring->xsk_umem, &desc))
 			break;
 
-		dma_sync_single_for_device(xdp_ring->dev, dma, len,
+		dma = xdp_umem_get_dma(xdp_ring->xsk_umem, desc.addr);
+
+		dma_sync_single_for_device(xdp_ring->dev, dma, desc.len,
 					   DMA_BIDIRECTIONAL);
 
 		tx_bi = &xdp_ring->tx_buffer_info[xdp_ring->next_to_use];
-		tx_bi->bytecount = len;
+		tx_bi->bytecount = desc.len;
 		tx_bi->xdpf = NULL;
 
 		tx_desc = IXGBE_TX_DESC(xdp_ring, xdp_ring->next_to_use);
@@ -648,10 +651,10 @@ static bool ixgbe_xmit_zc(struct ixgbe_ring *xdp_ring, unsigned int budget)
 		cmd_type = IXGBE_ADVTXD_DTYP_DATA |
 			   IXGBE_ADVTXD_DCMD_DEXT |
 			   IXGBE_ADVTXD_DCMD_IFCS;
-		cmd_type |= len | IXGBE_TXD_CMD;
+		cmd_type |= desc.len | IXGBE_TXD_CMD;
 		tx_desc->read.cmd_type_len = cpu_to_le32(cmd_type);
 		tx_desc->read.olinfo_status =
-			cpu_to_le32(len << IXGBE_ADVTXD_PAYLEN_SHIFT);
+			cpu_to_le32(desc.len << IXGBE_ADVTXD_PAYLEN_SHIFT);
 
 		xdp_ring->next_to_use++;
 		if (xdp_ring->next_to_use == xdp_ring->count)
diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h
index 1acddcbda236..eec5b66aee93 100644
--- a/include/net/xdp_sock.h
+++ b/include/net/xdp_sock.h
@@ -81,7 +81,7 @@ bool xsk_umem_has_addrs(struct xdp_umem *umem, u32 cnt);
 u64 *xsk_umem_peek_addr(struct xdp_umem *umem, u64 *addr);
 void xsk_umem_discard_addr(struct xdp_umem *umem);
 void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries);
-bool xsk_umem_consume_tx(struct xdp_umem *umem, dma_addr_t *dma, u32 *len);
+bool xsk_umem_consume_tx(struct xdp_umem *umem, struct xdp_desc *desc);
 void xsk_umem_consume_tx_done(struct xdp_umem *umem);
 struct xdp_umem_fq_reuse *xsk_reuseq_prepare(u32 nentries);
 struct xdp_umem_fq_reuse *xsk_reuseq_swap(struct xdp_umem *umem,
@@ -175,8 +175,8 @@ static inline void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries)
 {
 }
 
-static inline bool xsk_umem_consume_tx(struct xdp_umem *umem, dma_addr_t *dma,
-				       u32 *len)
+static inline bool xsk_umem_consume_tx(struct xdp_umem *umem,
+				       struct xdp_desc *desc)
 {
 	return false;
 }
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index 114ba17acb09..99e987e25a4d 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c
@@ -175,22 +175,18 @@ void xsk_umem_consume_tx_done(struct xdp_umem *umem)
 }
 EXPORT_SYMBOL(xsk_umem_consume_tx_done);
 
-bool xsk_umem_consume_tx(struct xdp_umem *umem, dma_addr_t *dma, u32 *len)
+bool xsk_umem_consume_tx(struct xdp_umem *umem, struct xdp_desc *desc)
 {
-	struct xdp_desc desc;
 	struct xdp_sock *xs;
 
 	rcu_read_lock();
 	list_for_each_entry_rcu(xs, &umem->xsk_list, list) {
-		if (!xskq_peek_desc(xs->tx, &desc))
+		if (!xskq_peek_desc(xs->tx, desc))
 			continue;
 
-		if (xskq_produce_addr_lazy(umem->cq, desc.addr))
+		if (xskq_produce_addr_lazy(umem->cq, desc->addr))
 			goto out;
 
-		*dma = xdp_umem_get_dma(umem, desc.addr);
-		*len = desc.len;
-
 		xskq_discard_desc(xs->tx);
 		rcu_read_unlock();
 		return true;
-- 
2.19.1

[PATCH bpf-next v2 07/16] net/mlx5e: Replace deprecated PCI_DMA_TODEVICE

[Maxim Mikityanskiy]

The PCI API for DMA is deprecated, and PCI_DMA_TODEVICE is just defined
to DMA_TO_DEVICE for backward compatibility. Just use DMA_TO_DEVICE.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
index eb8ef78e5626..5a900b70b203 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
@@ -64,7 +64,7 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_dma_info *di,
 		return false;
 	xdpi.dma_addr = di->addr + (xdpi.xdpf->data - (void *)xdpi.xdpf);
 	dma_sync_single_for_device(sq->pdev, xdpi.dma_addr,
-				   xdpi.xdpf->len, PCI_DMA_TODEVICE);
+				   xdpi.xdpf->len, DMA_TO_DEVICE);
 	xdpi.di = *di;
 
 	return sq->xmit_xdp_frame(sq, &xdpi);
-- 
2.19.1

[PATCH bpf-next v2 08/16] net/mlx5e: Calculate linear RX frag size considering XSK

[Maxim Mikityanskiy]

Additional conditions introduced:

- XSK implies XDP.
- Headroom includes the XSK headroom if it exists.
- No space is reserved for struct shared_skb_info in XSK mode.
- Fragment size smaller than the XSK chunk size is not allowed.

A new auxiliary function mlx5e_get_linear_rq_headroom with the support
for XSK is introduced. Use this function in the implementation of
mlx5e_get_rq_headroom. Change headroom to u32 to match the headroom
field in struct xdp_umem.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 .../ethernet/mellanox/mlx5/core/en/params.c   | 65 +++++++++++++------
 .../ethernet/mellanox/mlx5/core/en/params.h   |  8 ++-
 .../net/ethernet/mellanox/mlx5/core/en_main.c |  2 +-
 3 files changed, 52 insertions(+), 23 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
index d3744bffbae3..50a458dc3836 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
@@ -3,33 +3,62 @@
 
 #include "en/params.h"
 
-u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params)
+static inline bool mlx5e_rx_is_xdp(struct mlx5e_params *params,
+				   struct mlx5e_xsk_param *xsk)
 {
-	u16 hw_mtu = MLX5E_SW2HW_MTU(params, params->sw_mtu);
-	u16 linear_rq_headroom = params->xdp_prog ?
-		XDP_PACKET_HEADROOM : MLX5_RX_HEADROOM;
-	u32 frag_sz;
+	return params->xdp_prog || xsk;
+}
+
+static inline u16 mlx5e_get_linear_rq_headroom(struct mlx5e_params *params,
+					       struct mlx5e_xsk_param *xsk)
+{
+	u16 headroom = NET_IP_ALIGN;
+
+	if (mlx5e_rx_is_xdp(params, xsk)) {
+		headroom += XDP_PACKET_HEADROOM;
+		if (xsk)
+			headroom += xsk->headroom;
+	} else {
+		headroom += MLX5_RX_HEADROOM;
+	}
+
+	return headroom;
+}
+
+u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params,
+				struct mlx5e_xsk_param *xsk)
+{
+	u32 hw_mtu = MLX5E_SW2HW_MTU(params, params->sw_mtu);
+	u16 linear_rq_headroom = mlx5e_get_linear_rq_headroom(params, xsk);
+	u32 frag_sz = linear_rq_headroom + hw_mtu;
 
-	linear_rq_headroom += NET_IP_ALIGN;
+	/* AF_XDP doesn't build SKBs in place. */
+	if (!xsk)
+		frag_sz = MLX5_SKB_FRAG_SZ(frag_sz);
 
-	frag_sz = MLX5_SKB_FRAG_SZ(linear_rq_headroom + hw_mtu);
+	/* XDP in mlx5e doesn't support multiple packets per page. */
+	if (mlx5e_rx_is_xdp(params, xsk))
+		frag_sz = max_t(u32, frag_sz, PAGE_SIZE);
 
-	if (params->xdp_prog && frag_sz < PAGE_SIZE)
-		frag_sz = PAGE_SIZE;
+	/* Even if we can go with a smaller fragment size, we must not put
+	 * multiple packets into a single frame.
+	 */
+	if (xsk)
+		frag_sz = max_t(u32, frag_sz, xsk->chunk_size);
 
 	return frag_sz;
 }
 
 u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5e_params *params)
 {
-	u32 linear_frag_sz = mlx5e_rx_get_linear_frag_sz(params);
+	u32 linear_frag_sz = mlx5e_rx_get_linear_frag_sz(params, NULL);
 
 	return MLX5_MPWRQ_LOG_WQE_SZ - order_base_2(linear_frag_sz);
 }
 
 bool mlx5e_rx_is_linear_skb(struct mlx5e_params *params)
 {
-	u32 frag_sz = mlx5e_rx_get_linear_frag_sz(params);
+	u32 frag_sz = mlx5e_rx_get_linear_frag_sz(params, NULL);
 
 	return !params->lro_en && frag_sz <= PAGE_SIZE;
 }
@@ -39,7 +68,7 @@ bool mlx5e_rx_is_linear_skb(struct mlx5e_params *params)
 bool mlx5e_rx_mpwqe_is_linear_skb(struct mlx5_core_dev *mdev,
 				  struct mlx5e_params *params)
 {
-	u32 frag_sz = mlx5e_rx_get_linear_frag_sz(params);
+	u32 frag_sz = mlx5e_rx_get_linear_frag_sz(params, NULL);
 	s8 signed_log_num_strides_param;
 	u8 log_num_strides;
 
@@ -75,7 +104,7 @@ u8 mlx5e_mpwqe_get_log_stride_size(struct mlx5_core_dev *mdev,
 				   struct mlx5e_params *params)
 {
 	if (mlx5e_rx_mpwqe_is_linear_skb(mdev, params))
-		return order_base_2(mlx5e_rx_get_linear_frag_sz(params));
+		return order_base_2(mlx5e_rx_get_linear_frag_sz(params, NULL));
 
 	return MLX5_MPWRQ_DEF_LOG_STRIDE_SZ(mdev);
 }
@@ -90,15 +119,9 @@ u8 mlx5e_mpwqe_get_log_num_strides(struct mlx5_core_dev *mdev,
 u16 mlx5e_get_rq_headroom(struct mlx5_core_dev *mdev,
 			  struct mlx5e_params *params)
 {
-	u16 linear_rq_headroom = params->xdp_prog ?
-		XDP_PACKET_HEADROOM : MLX5_RX_HEADROOM;
-	bool is_linear_skb;
-
-	linear_rq_headroom += NET_IP_ALIGN;
-
-	is_linear_skb = (params->rq_wq_type == MLX5_WQ_TYPE_CYCLIC) ?
+	bool is_linear_skb = (params->rq_wq_type == MLX5_WQ_TYPE_CYCLIC) ?
 		mlx5e_rx_is_linear_skb(params) :
 		mlx5e_rx_mpwqe_is_linear_skb(mdev, params);
 
-	return is_linear_skb ? linear_rq_headroom : 0;
+	return is_linear_skb ? mlx5e_get_linear_rq_headroom(params, NULL) : 0;
 }
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
index b106a0236f36..ed420f3efe52 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
@@ -6,7 +6,13 @@
 
 #include "en.h"
 
-u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params);
+struct mlx5e_xsk_param {
+	u16 headroom;
+	u16 chunk_size;
+};
+
+u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params,
+				struct mlx5e_xsk_param *xsk);
 u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5e_params *params);
 bool mlx5e_rx_is_linear_skb(struct mlx5e_params *params);
 bool mlx5e_rx_mpwqe_is_linear_skb(struct mlx5_core_dev *mdev,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 457cc39423f2..204e199b141e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1955,7 +1955,7 @@ static void mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev,
 	if (mlx5e_rx_is_linear_skb(params)) {
 		int frag_stride;
 
-		frag_stride = mlx5e_rx_get_linear_frag_sz(params);
+		frag_stride = mlx5e_rx_get_linear_frag_sz(params, NULL);
 		frag_stride = roundup_pow_of_two(frag_stride);
 
 		info->arr[0].frag_size = byte_count;
-- 
2.19.1

[PATCH bpf-next v2 09/16] net/mlx5e: Allow ICO SQ to be used by multiple RQs

[Maxim Mikityanskiy]

Prepare to creation of the XSK RQ, which will require posting UMRs, too.
The same ICO SQ will be used for both RQs and also to trigger interrupts
by posting NOPs. UMR WQEs can't be reused any more. Optimization
introduced in commit ab966d7e4ff98 ("net/mlx5e: RX, Recycle buffer of
UMR WQEs") is reverted.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  9 +++++++
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 27 +++++++------------
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |  4 ++-
 drivers/net/ethernet/mellanox/mlx5/core/wq.h  |  5 ----
 4 files changed, 22 insertions(+), 23 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 3a183d690e23..41e22763007c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -348,6 +348,13 @@ enum {
 
 struct mlx5e_sq_wqe_info {
 	u8  opcode;
+
+	/* Auxiliary data for different opcodes. */
+	union {
+		struct {
+			struct mlx5e_rq *rq;
+		} umr;
+	};
 };
 
 struct mlx5e_txqsq {
@@ -570,6 +577,7 @@ struct mlx5e_rq {
 			u8                     log_stride_sz;
 			u8                     umr_in_progress;
 			u8                     umr_last_bulk;
+			u8                     umr_completed;
 		} mpwqe;
 	};
 	struct {
@@ -797,6 +805,7 @@ void mlx5e_page_release(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info,
 void mlx5e_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe);
 void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe);
 bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq);
+void mlx5e_poll_ico_cq(struct mlx5e_cq *cq);
 bool mlx5e_post_rx_mpwqes(struct mlx5e_rq *rq);
 void mlx5e_dealloc_rx_wqe(struct mlx5e_rq *rq, u16 ix);
 void mlx5e_dealloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 13133e7f088e..5d762da6bf9b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -425,11 +425,6 @@ static void mlx5e_post_rx_mpwqe(struct mlx5e_rq *rq, u8 n)
 	mlx5_wq_ll_update_db_record(wq);
 }
 
-static inline u16 mlx5e_icosq_wrap_cnt(struct mlx5e_icosq *sq)
-{
-	return mlx5_wq_cyc_get_ctr_wrap_cnt(&sq->wq, sq->pc);
-}
-
 static inline void mlx5e_fill_icosq_frag_edge(struct mlx5e_icosq *sq,
 					      struct mlx5_wq_cyc *wq,
 					      u16 pi, u16 nnops)
@@ -465,9 +460,7 @@ static int mlx5e_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix)
 	}
 
 	umr_wqe = mlx5_wq_cyc_get_wqe(wq, pi);
-	if (unlikely(mlx5e_icosq_wrap_cnt(sq) < 2))
-		memcpy(umr_wqe, &rq->mpwqe.umr_wqe,
-		       offsetof(struct mlx5e_umr_wqe, inline_mtts));
+	memcpy(umr_wqe, &rq->mpwqe.umr_wqe, offsetof(struct mlx5e_umr_wqe, inline_mtts));
 
 	for (i = 0; i < MLX5_MPWRQ_PAGES_PER_WQE; i++, dma_info++) {
 		err = mlx5e_page_alloc_mapped(rq, dma_info);
@@ -485,6 +478,7 @@ static int mlx5e_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix)
 	umr_wqe->uctrl.xlt_offset = cpu_to_be16(xlt_offset);
 
 	sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_UMR;
+	sq->db.ico_wqe[pi].umr.rq = rq;
 	sq->pc += MLX5E_UMR_WQEBBS;
 
 	sq->doorbell_cseg = &umr_wqe->ctrl;
@@ -542,11 +536,10 @@ bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq)
 	return !!err;
 }
 
-static void mlx5e_poll_ico_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq)
+void mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
 {
 	struct mlx5e_icosq *sq = container_of(cq, struct mlx5e_icosq, cq);
 	struct mlx5_cqe64 *cqe;
-	u8  completed_umr = 0;
 	u16 sqcc;
 	int i;
 
@@ -587,7 +580,7 @@ static void mlx5e_poll_ico_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq)
 
 			if (likely(wi->opcode == MLX5_OPCODE_UMR)) {
 				sqcc += MLX5E_UMR_WQEBBS;
-				completed_umr++;
+				wi->umr.rq->mpwqe.umr_completed++;
 			} else if (likely(wi->opcode == MLX5_OPCODE_NOP)) {
 				sqcc++;
 			} else {
@@ -603,24 +596,24 @@ static void mlx5e_poll_ico_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq)
 	sq->cc = sqcc;
 
 	mlx5_cqwq_update_db_record(&cq->wq);
-
-	if (likely(completed_umr)) {
-		mlx5e_post_rx_mpwqe(rq, completed_umr);
-		rq->mpwqe.umr_in_progress -= completed_umr;
-	}
 }
 
 bool mlx5e_post_rx_mpwqes(struct mlx5e_rq *rq)
 {
 	struct mlx5e_icosq *sq = &rq->channel->icosq;
 	struct mlx5_wq_ll *wq = &rq->mpwqe.wq;
+	u8  umr_completed = rq->mpwqe.umr_completed;
 	u8  missing, i;
 	u16 head;
 
 	if (unlikely(!test_bit(MLX5E_RQ_STATE_ENABLED, &rq->state)))
 		return false;
 
-	mlx5e_poll_ico_cq(&sq->cq, rq);
+	if (umr_completed) {
+		mlx5e_post_rx_mpwqe(rq, umr_completed);
+		rq->mpwqe.umr_in_progress -= umr_completed;
+		rq->mpwqe.umr_completed = 0;
+	}
 
 	missing = mlx5_wq_ll_missing(wq) - rq->mpwqe.umr_in_progress;
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index f9862bf75491..de4d5ae431af 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -107,7 +107,9 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 		busy |= work_done == budget;
 	}
 
-	busy |= c->rq.post_wqes(rq);
+	mlx5e_poll_ico_cq(&c->icosq.cq);
+
+	busy |= rq->post_wqes(rq);
 
 	if (busy) {
 		if (likely(mlx5e_channel_no_affinity_change(c)))
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/wq.h b/drivers/net/ethernet/mellanox/mlx5/core/wq.h
index 1f87cce421e0..f1ec58c9e9e3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/wq.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/wq.h
@@ -134,11 +134,6 @@ static inline void mlx5_wq_cyc_update_db_record(struct mlx5_wq_cyc *wq)
 	*wq->db = cpu_to_be32(wq->wqe_ctr);
 }
 
-static inline u16 mlx5_wq_cyc_get_ctr_wrap_cnt(struct mlx5_wq_cyc *wq, u16 ctr)
-{
-	return ctr >> wq->fbc.log_sz;
-}
-
 static inline u16 mlx5_wq_cyc_ctr2ix(struct mlx5_wq_cyc *wq, u16 ctr)
 {
 	return ctr & wq->fbc.sz_m1;
-- 
2.19.1

[PATCH bpf-next v2 10/16] net/mlx5e: Refactor struct mlx5e_xdp_info

[Maxim Mikityanskiy]

Currently, struct mlx5e_xdp_info has some issues that have to be cleaned
up before the upcoming AF_XDP support makes things too complicated and
messy. This structure is used both when sending the packet and on
completion. Moreover, the cleanup procedure on completion depends on the
origin of the packet (XDP_REDIRECT, XDP_TX). Adding AF_XDP support will
add new flows that use this structure even differently. To avoid
overcomplicating the code, this commit refactors the usage of this
structure in the following ways:

1. struct mlx5e_xdp_info is split into two different structures. One is
struct mlx5e_xdp_xmit_data, a transient structure that doesn't need to
be stored and is only used while sending the packet. The other is still
struct mlx5e_xdp_info that is stored in a FIFO and contains the fields
needed on completion.

2. The fields of struct mlx5e_xdp_info that are used in different flows
are put into a union. A special enum indicates the cleanup mode and
helps choose the right union member. This approach is clear and
explicit. Although it could be possible to "guess" the mode by looking
at the values of the fields and at the XDP SQ type, it wouldn't be that
clear and extendable and would require looking through the whole chain
to understand what's going on.

For the reference, there are the fields of struct mlx5e_xdp_info that
are used in different flows (including AF_XDP ones):

Packet origin          | Fields used on completion | Cleanup steps
-----------------------+---------------------------+------------------
XDP_REDIRECT,          | xdpf, dma_addr            | DMA unmap and
XDP_TX from XSK RQ     |                           | xdp_return_frame.
-----------------------+---------------------------+------------------
XDP_TX from regular RQ | di                        | Recycle page.
-----------------------+---------------------------+------------------
AF_XDP TX              | (none)                    | Increment the
                       |                           | producer index in
                       |                           | Completion Ring.

On send, the same set of mlx5e_xdp_xmit_data fields is used in all
flows: DMA and virtual addresses and length.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en.h  | 46 +++++++++--
 .../net/ethernet/mellanox/mlx5/core/en/xdp.c  | 81 ++++++++++++-------
 .../net/ethernet/mellanox/mlx5/core/en/xdp.h  | 11 ++-
 3 files changed, 97 insertions(+), 41 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 41e22763007c..cdb73568a344 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -402,10 +402,44 @@ struct mlx5e_dma_info {
 	dma_addr_t      addr;
 };
 
+/* XDP packets can be transmitted in different ways. On completion, we need to
+ * distinguish between them to clean up things in a proper way.
+ */
+enum mlx5e_xdp_xmit_mode {
+	/* An xdp_frame was transmitted due to either XDP_REDIRECT from another
+	 * device or XDP_TX from an XSK RQ. The frame has to be unmapped and
+	 * returned.
+	 */
+	MLX5E_XDP_XMIT_MODE_FRAME,
+
+	/* The xdp_frame was created in place as a result of XDP_TX from a
+	 * regular RQ. No DMA remapping happened, and the page belongs to us.
+	 */
+	MLX5E_XDP_XMIT_MODE_PAGE,
+
+	/* No xdp_frame was created at all, the transmit happened from a UMEM
+	 * page. The UMEM Completion Ring producer pointer has to be increased.
+	 */
+	MLX5E_XDP_XMIT_MODE_XSK,
+};
+
 struct mlx5e_xdp_info {
-	struct xdp_frame      *xdpf;
-	dma_addr_t            dma_addr;
-	struct mlx5e_dma_info di;
+	enum mlx5e_xdp_xmit_mode mode;
+	union {
+		struct {
+			struct xdp_frame *xdpf;
+			dma_addr_t dma_addr;
+		} frame;
+		struct {
+			struct mlx5e_dma_info di;
+		} page;
+	};
+};
+
+struct mlx5e_xdp_xmit_data {
+	dma_addr_t  dma_addr;
+	void       *data;
+	u32         len;
 };
 
 struct mlx5e_xdp_info_fifo {
@@ -431,8 +465,10 @@ struct mlx5e_xdp_mpwqe {
 };
 
 struct mlx5e_xdpsq;
-typedef bool (*mlx5e_fp_xmit_xdp_frame)(struct mlx5e_xdpsq*,
-					struct mlx5e_xdp_info*);
+typedef bool (*mlx5e_fp_xmit_xdp_frame)(struct mlx5e_xdpsq *,
+					struct mlx5e_xdp_xmit_data *,
+					struct mlx5e_xdp_info *);
+
 struct mlx5e_xdpsq {
 	/* data path */
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
index 5a900b70b203..89f6eb1109cf 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
@@ -57,17 +57,27 @@ static inline bool
 mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_dma_info *di,
 		    struct xdp_buff *xdp)
 {
+	struct mlx5e_xdp_xmit_data xdptxd;
 	struct mlx5e_xdp_info xdpi;
+	struct xdp_frame *xdpf;
+	dma_addr_t dma_addr;
 
-	xdpi.xdpf = convert_to_xdp_frame(xdp);
-	if (unlikely(!xdpi.xdpf))
+	xdpf = convert_to_xdp_frame(xdp);
+	if (unlikely(!xdpf))
 		return false;
-	xdpi.dma_addr = di->addr + (xdpi.xdpf->data - (void *)xdpi.xdpf);
-	dma_sync_single_for_device(sq->pdev, xdpi.dma_addr,
-				   xdpi.xdpf->len, DMA_TO_DEVICE);
-	xdpi.di = *di;
 
-	return sq->xmit_xdp_frame(sq, &xdpi);
+	xdptxd.data = xdpf->data;
+	xdptxd.len  = xdpf->len;
+
+	xdpi.mode = MLX5E_XDP_XMIT_MODE_PAGE;
+
+	dma_addr = di->addr + (xdpf->data - (void *)xdpf);
+	dma_sync_single_for_device(sq->pdev, dma_addr, xdptxd.len, DMA_TO_DEVICE);
+
+	xdptxd.dma_addr = dma_addr;
+	xdpi.page.di = *di;
+
+	return sq->xmit_xdp_frame(sq, &xdptxd, &xdpi);
 }
 
 /* returns true if packet was consumed by xdp */
@@ -184,14 +194,13 @@ static void mlx5e_xdp_mpwqe_complete(struct mlx5e_xdpsq *sq)
 }
 
 static bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq,
+				       struct mlx5e_xdp_xmit_data *xdptxd,
 				       struct mlx5e_xdp_info *xdpi)
 {
 	struct mlx5e_xdp_mpwqe *session = &sq->mpwqe;
 	struct mlx5e_xdpsq_stats *stats = sq->stats;
 
-	struct xdp_frame *xdpf = xdpi->xdpf;
-
-	if (unlikely(sq->hw_mtu < xdpf->len)) {
+	if (unlikely(xdptxd->len > sq->hw_mtu)) {
 		stats->err++;
 		return false;
 	}
@@ -208,7 +217,7 @@ static bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq,
 		mlx5e_xdp_mpwqe_session_start(sq);
 	}
 
-	mlx5e_xdp_mpwqe_add_dseg(sq, xdpi, stats);
+	mlx5e_xdp_mpwqe_add_dseg(sq, xdptxd, stats);
 
 	if (unlikely(session->complete ||
 		     session->ds_count == session->max_ds_count))
@@ -219,7 +228,9 @@ static bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq,
 	return true;
 }
 
-static bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xdp_info *xdpi)
+static bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq,
+				 struct mlx5e_xdp_xmit_data *xdptxd,
+				 struct mlx5e_xdp_info *xdpi)
 {
 	struct mlx5_wq_cyc       *wq   = &sq->wq;
 	u16                       pi   = mlx5_wq_cyc_ctr2ix(wq, sq->pc);
@@ -229,9 +240,8 @@ static bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xdp_info *
 	struct mlx5_wqe_eth_seg  *eseg = &wqe->eth;
 	struct mlx5_wqe_data_seg *dseg = wqe->data;
 
-	struct xdp_frame *xdpf = xdpi->xdpf;
-	dma_addr_t dma_addr  = xdpi->dma_addr;
-	unsigned int dma_len = xdpf->len;
+	dma_addr_t dma_addr = xdptxd->dma_addr;
+	u32 dma_len = xdptxd->len;
 
 	struct mlx5e_xdpsq_stats *stats = sq->stats;
 
@@ -253,7 +263,7 @@ static bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xdp_info *
 
 	/* copy the inline part if required */
 	if (sq->min_inline_mode != MLX5_INLINE_MODE_NONE) {
-		memcpy(eseg->inline_hdr.start, xdpf->data, MLX5E_XDP_MIN_INLINE);
+		memcpy(eseg->inline_hdr.start, xdptxd->data, MLX5E_XDP_MIN_INLINE);
 		eseg->inline_hdr.sz = cpu_to_be16(MLX5E_XDP_MIN_INLINE);
 		dma_len  -= MLX5E_XDP_MIN_INLINE;
 		dma_addr += MLX5E_XDP_MIN_INLINE;
@@ -286,14 +296,19 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq,
 	for (i = 0; i < wi->num_pkts; i++) {
 		struct mlx5e_xdp_info xdpi = mlx5e_xdpi_fifo_pop(xdpi_fifo);
 
-		if (rq) {
-			/* XDP_TX */
-			mlx5e_page_release(rq, &xdpi.di, recycle);
-		} else {
+		switch (xdpi.mode) {
+		case MLX5E_XDP_XMIT_MODE_FRAME:
 			/* XDP_REDIRECT */
-			dma_unmap_single(sq->pdev, xdpi.dma_addr,
-					 xdpi.xdpf->len, DMA_TO_DEVICE);
-			xdp_return_frame(xdpi.xdpf);
+			dma_unmap_single(sq->pdev, xdpi.frame.dma_addr,
+					 xdpi.frame.xdpf->len, DMA_TO_DEVICE);
+			xdp_return_frame(xdpi.frame.xdpf);
+			break;
+		case MLX5E_XDP_XMIT_MODE_PAGE:
+			/* XDP_TX */
+			mlx5e_page_release(rq, &xdpi.page.di, recycle);
+			break;
+		default:
+			WARN_ON_ONCE(true);
 		}
 	}
 }
@@ -398,21 +413,27 @@ int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames,
 
 	for (i = 0; i < n; i++) {
 		struct xdp_frame *xdpf = frames[i];
+		struct mlx5e_xdp_xmit_data xdptxd;
 		struct mlx5e_xdp_info xdpi;
 
-		xdpi.dma_addr = dma_map_single(sq->pdev, xdpf->data, xdpf->len,
-					       DMA_TO_DEVICE);
-		if (unlikely(dma_mapping_error(sq->pdev, xdpi.dma_addr))) {
+		xdptxd.data = xdpf->data;
+		xdptxd.len = xdpf->len;
+		xdptxd.dma_addr = dma_map_single(sq->pdev, xdptxd.data,
+						 xdptxd.len, DMA_TO_DEVICE);
+
+		if (unlikely(dma_mapping_error(sq->pdev, xdptxd.dma_addr))) {
 			xdp_return_frame_rx_napi(xdpf);
 			drops++;
 			continue;
 		}
 
-		xdpi.xdpf = xdpf;
+		xdpi.mode           = MLX5E_XDP_XMIT_MODE_FRAME;
+		xdpi.frame.xdpf     = xdpf;
+		xdpi.frame.dma_addr = xdptxd.dma_addr;
 
-		if (unlikely(!sq->xmit_xdp_frame(sq, &xdpi))) {
-			dma_unmap_single(sq->pdev, xdpi.dma_addr,
-					 xdpf->len, DMA_TO_DEVICE);
+		if (unlikely(!sq->xmit_xdp_frame(sq, &xdptxd, &xdpi))) {
+			dma_unmap_single(sq->pdev, xdptxd.dma_addr,
+					 xdptxd.len, DMA_TO_DEVICE);
 			xdp_return_frame_rx_napi(xdpf);
 			drops++;
 		}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
index 8b537a4b0840..2a5158993349 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
@@ -97,15 +97,14 @@ static inline void mlx5e_xdp_update_inline_state(struct mlx5e_xdpsq *sq)
 }
 
 static inline void
-mlx5e_xdp_mpwqe_add_dseg(struct mlx5e_xdpsq *sq, struct mlx5e_xdp_info *xdpi,
+mlx5e_xdp_mpwqe_add_dseg(struct mlx5e_xdpsq *sq,
+			 struct mlx5e_xdp_xmit_data *xdptxd,
 			 struct mlx5e_xdpsq_stats *stats)
 {
 	struct mlx5e_xdp_mpwqe *session = &sq->mpwqe;
-	dma_addr_t dma_addr    = xdpi->dma_addr;
-	struct xdp_frame *xdpf = xdpi->xdpf;
 	struct mlx5_wqe_data_seg *dseg =
 		(struct mlx5_wqe_data_seg *)session->wqe + session->ds_count;
-	u16 dma_len = xdpf->len;
+	u32 dma_len = xdptxd->len;
 
 	session->pkt_count++;
 
@@ -124,7 +123,7 @@ mlx5e_xdp_mpwqe_add_dseg(struct mlx5e_xdpsq *sq, struct mlx5e_xdp_info *xdpi,
 		}
 
 		inline_dseg->byte_count = cpu_to_be32(dma_len | MLX5_INLINE_SEG);
-		memcpy(inline_dseg->data, xdpf->data, dma_len);
+		memcpy(inline_dseg->data, xdptxd->data, dma_len);
 
 		session->ds_count += ds_cnt;
 		stats->inlnw++;
@@ -132,7 +131,7 @@ mlx5e_xdp_mpwqe_add_dseg(struct mlx5e_xdpsq *sq, struct mlx5e_xdp_info *xdpi,
 	}
 
 no_inline:
-	dseg->addr       = cpu_to_be64(dma_addr);
+	dseg->addr       = cpu_to_be64(xdptxd->dma_addr);
 	dseg->byte_count = cpu_to_be32(dma_len);
 	dseg->lkey       = sq->mkey_be;
 	session->ds_count++;
-- 
2.19.1

[PATCH bpf-next v2 11/16] net/mlx5e: Share the XDP SQ for XDP_TX between RQs

[Maxim Mikityanskiy]

Put the XDP SQ that is used for XDP_TX into the channel. It used to be a
part of the RQ, but with introduction of AF_XDP there will be one more
RQ that could share the same XDP SQ. This patch is a preparation for
that change.

Separate XDP_TX statistics per RQ were implemented in one of the previous
patches.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  4 ++-
 .../net/ethernet/mellanox/mlx5/core/en/xdp.c  | 20 +++++++-------
 .../net/ethernet/mellanox/mlx5/core/en/xdp.h  |  4 +--
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 26 +++++++++++--------
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |  4 +--
 5 files changed, 32 insertions(+), 26 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index cdb73568a344..8cb28e5604f0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -431,6 +431,7 @@ struct mlx5e_xdp_info {
 			dma_addr_t dma_addr;
 		} frame;
 		struct {
+			struct mlx5e_rq *rq;
 			struct mlx5e_dma_info di;
 		} page;
 	};
@@ -643,7 +644,7 @@ struct mlx5e_rq {
 
 	/* XDP */
 	struct bpf_prog       *xdp_prog;
-	struct mlx5e_xdpsq     xdpsq;
+	struct mlx5e_xdpsq    *xdpsq;
 	DECLARE_BITMAP(flags, 8);
 	struct page_pool      *page_pool;
 
@@ -662,6 +663,7 @@ struct mlx5e_rq {
 struct mlx5e_channel {
 	/* data path */
 	struct mlx5e_rq            rq;
+	struct mlx5e_xdpsq         rq_xdpsq;
 	struct mlx5e_txqsq         sq[MLX5E_MAX_NUM_TC];
 	struct mlx5e_icosq         icosq;   /* internal control operations */
 	bool                       xdp;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
index 89f6eb1109cf..b3e118fc4521 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
@@ -54,8 +54,8 @@ int mlx5e_xdp_max_mtu(struct mlx5e_params *params)
 }
 
 static inline bool
-mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_dma_info *di,
-		    struct xdp_buff *xdp)
+mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq,
+		    struct mlx5e_dma_info *di, struct xdp_buff *xdp)
 {
 	struct mlx5e_xdp_xmit_data xdptxd;
 	struct mlx5e_xdp_info xdpi;
@@ -75,6 +75,7 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_dma_info *di,
 	dma_sync_single_for_device(sq->pdev, dma_addr, xdptxd.len, DMA_TO_DEVICE);
 
 	xdptxd.dma_addr = dma_addr;
+	xdpi.page.rq = rq;
 	xdpi.page.di = *di;
 
 	return sq->xmit_xdp_frame(sq, &xdptxd, &xdpi);
@@ -105,7 +106,7 @@ bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,
 		*len = xdp.data_end - xdp.data;
 		return false;
 	case XDP_TX:
-		if (unlikely(!mlx5e_xmit_xdp_buff(&rq->xdpsq, di, &xdp)))
+		if (unlikely(!mlx5e_xmit_xdp_buff(rq->xdpsq, rq, di, &xdp)))
 			goto xdp_abort;
 		__set_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags); /* non-atomic */
 		return true;
@@ -287,7 +288,6 @@ static bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq,
 
 static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq,
 				  struct mlx5e_xdp_wqe_info *wi,
-				  struct mlx5e_rq *rq,
 				  bool recycle)
 {
 	struct mlx5e_xdp_info_fifo *xdpi_fifo = &sq->db.xdpi_fifo;
@@ -305,7 +305,7 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq,
 			break;
 		case MLX5E_XDP_XMIT_MODE_PAGE:
 			/* XDP_TX */
-			mlx5e_page_release(rq, &xdpi.page.di, recycle);
+			mlx5e_page_release(xdpi.page.rq, &xdpi.page.di, recycle);
 			break;
 		default:
 			WARN_ON_ONCE(true);
@@ -313,7 +313,7 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq,
 	}
 }
 
-bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq)
+bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
 {
 	struct mlx5e_xdpsq *sq;
 	struct mlx5_cqe64 *cqe;
@@ -358,7 +358,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq)
 
 			sqcc += wi->num_wqebbs;
 
-			mlx5e_free_xdpsq_desc(sq, wi, rq, true);
+			mlx5e_free_xdpsq_desc(sq, wi, true);
 		} while (!last_wqe);
 	} while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq)));
 
@@ -373,7 +373,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq)
 	return (i == MLX5E_TX_CQ_POLL_BUDGET);
 }
 
-void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq)
+void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq)
 {
 	while (sq->cc != sq->pc) {
 		struct mlx5e_xdp_wqe_info *wi;
@@ -384,7 +384,7 @@ void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq)
 
 		sq->cc += wi->num_wqebbs;
 
-		mlx5e_free_xdpsq_desc(sq, wi, rq, false);
+		mlx5e_free_xdpsq_desc(sq, wi, false);
 	}
 }
 
@@ -450,7 +450,7 @@ int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames,
 
 void mlx5e_xdp_rx_poll_complete(struct mlx5e_rq *rq)
 {
-	struct mlx5e_xdpsq *xdpsq = &rq->xdpsq;
+	struct mlx5e_xdpsq *xdpsq = rq->xdpsq;
 
 	if (xdpsq->mpwqe.wqe)
 		mlx5e_xdp_mpwqe_complete(xdpsq);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
index 2a5158993349..86db5ad49a42 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
@@ -42,8 +42,8 @@
 int mlx5e_xdp_max_mtu(struct mlx5e_params *params);
 bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,
 		      void *va, u16 *rx_headroom, u32 *len);
-bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq, struct mlx5e_rq *rq);
-void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq);
+bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq);
+void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq);
 void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw);
 void mlx5e_xdp_rx_poll_complete(struct mlx5e_rq *rq);
 int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 204e199b141e..35b42c875cf3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -418,6 +418,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 	rq->mdev    = mdev;
 	rq->hw_mtu  = MLX5E_SW2HW_MTU(params, params->sw_mtu);
 	rq->stats   = &c->priv->channel_stats[c->ix].rq;
+	rq->xdpsq   = &c->rq_xdpsq;
 
 	rq->xdp_prog = params->xdp_prog ? bpf_prog_inc(params->xdp_prog) : NULL;
 	if (IS_ERR(rq->xdp_prog)) {
@@ -1439,7 +1440,7 @@ static int mlx5e_open_xdpsq(struct mlx5e_channel *c,
 	return err;
 }
 
-static void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq)
+static void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq)
 {
 	struct mlx5e_channel *c = sq->channel;
 
@@ -1447,7 +1448,7 @@ static void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq)
 	napi_synchronize(&c->napi);
 
 	mlx5e_destroy_sq(c->mdev, sq->sqn);
-	mlx5e_free_xdpsq_descs(sq, rq);
+	mlx5e_free_xdpsq_descs(sq);
 	mlx5e_free_xdpsq(sq);
 }
 
@@ -1826,7 +1827,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 
 	/* XDP SQ CQ params are same as normal TXQ sq CQ params */
 	err = c->xdp ? mlx5e_open_cq(c, params->tx_cq_moderation,
-				     &cparam->tx_cq, &c->rq.xdpsq.cq) : 0;
+				     &cparam->tx_cq, &c->rq_xdpsq.cq) : 0;
 	if (err)
 		goto err_close_rx_cq;
 
@@ -1840,9 +1841,12 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 	if (err)
 		goto err_close_icosq;
 
-	err = c->xdp ? mlx5e_open_xdpsq(c, params, &cparam->xdp_sq, &c->rq.xdpsq, false) : 0;
-	if (err)
-		goto err_close_sqs;
+	if (c->xdp) {
+		err = mlx5e_open_xdpsq(c, params, &cparam->xdp_sq,
+				       &c->rq_xdpsq, false);
+		if (err)
+			goto err_close_sqs;
+	}
 
 	err = mlx5e_open_rq(c, params, &cparam->rq, &c->rq);
 	if (err)
@@ -1861,7 +1865,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 
 err_close_xdp_sq:
 	if (c->xdp)
-		mlx5e_close_xdpsq(&c->rq.xdpsq, &c->rq);
+		mlx5e_close_xdpsq(&c->rq_xdpsq);
 
 err_close_sqs:
 	mlx5e_close_sqs(c);
@@ -1872,7 +1876,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 err_disable_napi:
 	napi_disable(&c->napi);
 	if (c->xdp)
-		mlx5e_close_cq(&c->rq.xdpsq.cq);
+		mlx5e_close_cq(&c->rq_xdpsq.cq);
 
 err_close_rx_cq:
 	mlx5e_close_cq(&c->rq.cq);
@@ -1917,15 +1921,15 @@ static void mlx5e_deactivate_channel(struct mlx5e_channel *c)
 
 static void mlx5e_close_channel(struct mlx5e_channel *c)
 {
-	mlx5e_close_xdpsq(&c->xdpsq, NULL);
+	mlx5e_close_xdpsq(&c->xdpsq);
 	mlx5e_close_rq(&c->rq);
 	if (c->xdp)
-		mlx5e_close_xdpsq(&c->rq.xdpsq, &c->rq);
+		mlx5e_close_xdpsq(&c->rq_xdpsq);
 	mlx5e_close_sqs(c);
 	mlx5e_close_icosq(&c->icosq);
 	napi_disable(&c->napi);
 	if (c->xdp)
-		mlx5e_close_cq(&c->rq.xdpsq.cq);
+		mlx5e_close_cq(&c->rq_xdpsq.cq);
 	mlx5e_close_cq(&c->rq.cq);
 	mlx5e_close_cq(&c->xdpsq.cq);
 	mlx5e_close_tx_cqs(c);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index de4d5ae431af..d2b8ce5df59c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -97,10 +97,10 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 	for (i = 0; i < c->num_tc; i++)
 		busy |= mlx5e_poll_tx_cq(&c->sq[i].cq, budget);
 
-	busy |= mlx5e_poll_xdpsq_cq(&c->xdpsq.cq, NULL);
+	busy |= mlx5e_poll_xdpsq_cq(&c->xdpsq.cq);
 
 	if (c->xdp)
-		busy |= mlx5e_poll_xdpsq_cq(&rq->xdpsq.cq, rq);
+		busy |= mlx5e_poll_xdpsq_cq(&c->rq_xdpsq.cq);
 
 	if (likely(budget)) { /* budget=0 means: don't poll rx rings */
 		work_done = mlx5e_poll_rx_cq(&rq->cq, budget);
-- 
2.19.1

[PATCH bpf-next v2 12/16] net/mlx5e: XDP_TX from UMEM support

[Maxim Mikityanskiy]

When an XDP program returns XDP_TX, and the RQ is XSK-enabled, it
requires careful handling, because convert_to_xdp_frame creates a new
page and copies the data there, while our driver expects the xdp_frame
to point to the same memory as the xdp_buff. Handle this case
separately: map the page, and in the end unmap it and call
xdp_return_frame.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 .../net/ethernet/mellanox/mlx5/core/en/xdp.c  | 50 ++++++++++++++++---
 1 file changed, 42 insertions(+), 8 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
index b3e118fc4521..1364bdff702c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
@@ -69,14 +69,48 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq,
 	xdptxd.data = xdpf->data;
 	xdptxd.len  = xdpf->len;
 
-	xdpi.mode = MLX5E_XDP_XMIT_MODE_PAGE;
+	if (xdp->rxq->mem.type == MEM_TYPE_ZERO_COPY) {
+		/* The xdp_buff was in the UMEM and was copied into a newly
+		 * allocated page. The UMEM page was returned via the ZCA, and
+		 * this new page has to be mapped at this point and has to be
+		 * unmapped and returned via xdp_return_frame on completion.
+		 */
+
+		/* Prevent double recycling of the UMEM page. Even in case this
+		 * function returns false, the xdp_buff shouldn't be recycled,
+		 * as it was already done in xdp_convert_zc_to_xdp_frame.
+		 */
+		__set_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags); /* non-atomic */
+
+		xdpi.mode = MLX5E_XDP_XMIT_MODE_FRAME;
 
-	dma_addr = di->addr + (xdpf->data - (void *)xdpf);
-	dma_sync_single_for_device(sq->pdev, dma_addr, xdptxd.len, DMA_TO_DEVICE);
+		dma_addr = dma_map_single(sq->pdev, xdptxd.data, xdptxd.len,
+					  DMA_TO_DEVICE);
+		if (dma_mapping_error(sq->pdev, dma_addr)) {
+			xdp_return_frame(xdpf);
+			return false;
+		}
 
-	xdptxd.dma_addr = dma_addr;
-	xdpi.page.rq = rq;
-	xdpi.page.di = *di;
+		xdptxd.dma_addr     = dma_addr;
+		xdpi.frame.xdpf     = xdpf;
+		xdpi.frame.dma_addr = dma_addr;
+	} else {
+		/* Driver assumes that convert_to_xdp_frame returns an xdp_frame
+		 * that points to the same memory region as the original
+		 * xdp_buff. It allows to map the memory only once and to use
+		 * the DMA_BIDIRECTIONAL mode.
+		 */
+
+		xdpi.mode = MLX5E_XDP_XMIT_MODE_PAGE;
+
+		dma_addr = di->addr + (xdpf->data - (void *)xdpf);
+		dma_sync_single_for_device(sq->pdev, dma_addr, xdptxd.len,
+					   DMA_TO_DEVICE);
+
+		xdptxd.dma_addr = dma_addr;
+		xdpi.page.rq    = rq;
+		xdpi.page.di    = *di;
+	}
 
 	return sq->xmit_xdp_frame(sq, &xdptxd, &xdpi);
 }
@@ -298,13 +332,13 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq,
 
 		switch (xdpi.mode) {
 		case MLX5E_XDP_XMIT_MODE_FRAME:
-			/* XDP_REDIRECT */
+			/* XDP_TX from the XSK RQ and XDP_REDIRECT */
 			dma_unmap_single(sq->pdev, xdpi.frame.dma_addr,
 					 xdpi.frame.xdpf->len, DMA_TO_DEVICE);
 			xdp_return_frame(xdpi.frame.xdpf);
 			break;
 		case MLX5E_XDP_XMIT_MODE_PAGE:
-			/* XDP_TX */
+			/* XDP_TX from the regular RQ */
 			mlx5e_page_release(xdpi.page.rq, &xdpi.page.di, recycle);
 			break;
 		default:
-- 
2.19.1

[PATCH bpf-next v2 13/16] net/mlx5e: Consider XSK in XDP MTU limit calculation

[Maxim Mikityanskiy]

Use the existing mlx5e_get_linear_rq_headroom function to calculate the
headroom for mlx5e_xdp_max_mtu. This function takes the XSK headroom
into consideration, which will be used in the following patches.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en/params.c | 4 ++--
 drivers/net/ethernet/mellanox/mlx5/core/en/params.h | 2 ++
 drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c    | 5 +++--
 drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h    | 3 ++-
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c   | 4 ++--
 5 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
index 50a458dc3836..0de908b12fcc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
@@ -9,8 +9,8 @@ static inline bool mlx5e_rx_is_xdp(struct mlx5e_params *params,
 	return params->xdp_prog || xsk;
 }
 
-static inline u16 mlx5e_get_linear_rq_headroom(struct mlx5e_params *params,
-					       struct mlx5e_xsk_param *xsk)
+u16 mlx5e_get_linear_rq_headroom(struct mlx5e_params *params,
+				 struct mlx5e_xsk_param *xsk)
 {
 	u16 headroom = NET_IP_ALIGN;
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
index ed420f3efe52..7f29b82dd8c2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
@@ -11,6 +11,8 @@ struct mlx5e_xsk_param {
 	u16 chunk_size;
 };
 
+u16 mlx5e_get_linear_rq_headroom(struct mlx5e_params *params,
+				 struct mlx5e_xsk_param *xsk);
 u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params,
 				struct mlx5e_xsk_param *xsk);
 u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5e_params *params);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
index 1364bdff702c..ee99efde9143 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
@@ -32,10 +32,11 @@
 
 #include <linux/bpf_trace.h>
 #include "en/xdp.h"
+#include "en/params.h"
 
-int mlx5e_xdp_max_mtu(struct mlx5e_params *params)
+int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_xsk_param *xsk)
 {
-	int hr = NET_IP_ALIGN + XDP_PACKET_HEADROOM;
+	int hr = mlx5e_get_linear_rq_headroom(params, xsk);
 
 	/* Let S := SKB_DATA_ALIGN(sizeof(struct skb_shared_info)).
 	 * The condition checked in mlx5e_rx_is_linear_skb is:
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
index 86db5ad49a42..9200cb9f499b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
@@ -39,7 +39,8 @@
 	(sizeof(struct mlx5e_tx_wqe) / MLX5_SEND_WQE_DS)
 #define MLX5E_XDP_TX_DS_COUNT (MLX5E_XDP_TX_EMPTY_DS_COUNT + 1 /* SG DS */)
 
-int mlx5e_xdp_max_mtu(struct mlx5e_params *params);
+struct mlx5e_xsk_param;
+int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_xsk_param *xsk);
 bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,
 		      void *va, u16 *rx_headroom, u32 *len);
 bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 35b42c875cf3..08c2fd0be7ac 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -3718,7 +3718,7 @@ int mlx5e_change_mtu(struct net_device *netdev, int new_mtu,
 	if (params->xdp_prog &&
 	    !mlx5e_rx_is_linear_skb(&new_channels.params)) {
 		netdev_err(netdev, "MTU(%d) > %d is not allowed while XDP enabled\n",
-			   new_mtu, mlx5e_xdp_max_mtu(params));
+			   new_mtu, mlx5e_xdp_max_mtu(params, NULL));
 		err = -EINVAL;
 		goto out;
 	}
@@ -4160,7 +4160,7 @@ static int mlx5e_xdp_allowed(struct mlx5e_priv *priv, struct bpf_prog *prog)
 	if (!mlx5e_rx_is_linear_skb(&new_channels.params)) {
 		netdev_warn(netdev, "XDP is not allowed with MTU(%d) > %d\n",
 			    new_channels.params.sw_mtu,
-			    mlx5e_xdp_max_mtu(&new_channels.params));
+			    mlx5e_xdp_max_mtu(&new_channels.params, NULL));
 		return -EINVAL;
 	}
 
-- 
2.19.1

[PATCH bpf-next v2 14/16] net/mlx5e: Encapsulate open/close queues into a function

[Maxim Mikityanskiy]

Create new functions mlx5e_{open,close}_queues to encapsulate opening
and closing RQs and SQs, and call the new functions from
mlx5e_{open,close}_channel. It simplifies the existing functions a bit
and prepares them for the upcoming AF_XDP changes.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 125 ++++++++++--------
 1 file changed, 73 insertions(+), 52 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 08c2fd0be7ac..9f74b9f72135 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1769,49 +1769,16 @@ static void mlx5e_free_xps_cpumask(struct mlx5e_channel *c)
 	free_cpumask_var(c->xps_cpumask);
 }
 
-static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
-			      struct mlx5e_params *params,
-			      struct mlx5e_channel_param *cparam,
-			      struct mlx5e_channel **cp)
+static int mlx5e_open_queues(struct mlx5e_channel *c,
+			     struct mlx5e_params *params,
+			     struct mlx5e_channel_param *cparam)
 {
-	int cpu = cpumask_first(mlx5_comp_irq_get_affinity_mask(priv->mdev, ix));
 	struct net_dim_cq_moder icocq_moder = {0, 0};
-	struct net_device *netdev = priv->netdev;
-	struct mlx5e_channel *c;
-	unsigned int irq;
 	int err;
-	int eqn;
-
-	err = mlx5_vector2eqn(priv->mdev, ix, &eqn, &irq);
-	if (err)
-		return err;
-
-	c = kvzalloc_node(sizeof(*c), GFP_KERNEL, cpu_to_node(cpu));
-	if (!c)
-		return -ENOMEM;
-
-	c->priv     = priv;
-	c->mdev     = priv->mdev;
-	c->tstamp   = &priv->tstamp;
-	c->ix       = ix;
-	c->cpu      = cpu;
-	c->pdev     = priv->mdev->device;
-	c->netdev   = priv->netdev;
-	c->mkey_be  = cpu_to_be32(priv->mdev->mlx5e_res.mkey.key);
-	c->num_tc   = params->num_tc;
-	c->xdp      = !!params->xdp_prog;
-	c->stats    = &priv->channel_stats[ix].ch;
-	c->irq_desc = irq_to_desc(irq);
-
-	err = mlx5e_alloc_xps_cpumask(c, params);
-	if (err)
-		goto err_free_channel;
-
-	netif_napi_add(netdev, &c->napi, mlx5e_napi_poll, 64);
 
 	err = mlx5e_open_cq(c, icocq_moder, &cparam->icosq_cq, &c->icosq.cq);
 	if (err)
-		goto err_napi_del;
+		return err;
 
 	err = mlx5e_open_tx_cqs(c, params, cparam);
 	if (err)
@@ -1856,8 +1823,6 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 	if (err)
 		goto err_close_rq;
 
-	*cp = c;
-
 	return 0;
 
 err_close_rq:
@@ -1875,6 +1840,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 
 err_disable_napi:
 	napi_disable(&c->napi);
+
 	if (c->xdp)
 		mlx5e_close_cq(&c->rq_xdpsq.cq);
 
@@ -1890,6 +1856,73 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 err_close_icosq_cq:
 	mlx5e_close_cq(&c->icosq.cq);
 
+	return err;
+}
+
+static void mlx5e_close_queues(struct mlx5e_channel *c)
+{
+	mlx5e_close_xdpsq(&c->xdpsq);
+	mlx5e_close_rq(&c->rq);
+	if (c->xdp)
+		mlx5e_close_xdpsq(&c->rq_xdpsq);
+	mlx5e_close_sqs(c);
+	mlx5e_close_icosq(&c->icosq);
+	napi_disable(&c->napi);
+	if (c->xdp)
+		mlx5e_close_cq(&c->rq_xdpsq.cq);
+	mlx5e_close_cq(&c->rq.cq);
+	mlx5e_close_cq(&c->xdpsq.cq);
+	mlx5e_close_tx_cqs(c);
+	mlx5e_close_cq(&c->icosq.cq);
+}
+
+static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
+			      struct mlx5e_params *params,
+			      struct mlx5e_channel_param *cparam,
+			      struct mlx5e_channel **cp)
+{
+	int cpu = cpumask_first(mlx5_comp_irq_get_affinity_mask(priv->mdev, ix));
+	struct net_device *netdev = priv->netdev;
+	struct mlx5e_channel *c;
+	unsigned int irq;
+	int err;
+	int eqn;
+
+	err = mlx5_vector2eqn(priv->mdev, ix, &eqn, &irq);
+	if (err)
+		return err;
+
+	c = kvzalloc_node(sizeof(*c), GFP_KERNEL, cpu_to_node(cpu));
+	if (!c)
+		return -ENOMEM;
+
+	c->priv     = priv;
+	c->mdev     = priv->mdev;
+	c->tstamp   = &priv->tstamp;
+	c->ix       = ix;
+	c->cpu      = cpu;
+	c->pdev     = priv->mdev->device;
+	c->netdev   = priv->netdev;
+	c->mkey_be  = cpu_to_be32(priv->mdev->mlx5e_res.mkey.key);
+	c->num_tc   = params->num_tc;
+	c->xdp      = !!params->xdp_prog;
+	c->stats    = &priv->channel_stats[ix].ch;
+	c->irq_desc = irq_to_desc(irq);
+
+	err = mlx5e_alloc_xps_cpumask(c, params);
+	if (err)
+		goto err_free_channel;
+
+	netif_napi_add(netdev, &c->napi, mlx5e_napi_poll, 64);
+
+	err = mlx5e_open_queues(c, params, cparam);
+	if (unlikely(err))
+		goto err_napi_del;
+
+	*cp = c;
+
+	return 0;
+
 err_napi_del:
 	netif_napi_del(&c->napi);
 	mlx5e_free_xps_cpumask(c);
@@ -1921,19 +1954,7 @@ static void mlx5e_deactivate_channel(struct mlx5e_channel *c)
 
 static void mlx5e_close_channel(struct mlx5e_channel *c)
 {
-	mlx5e_close_xdpsq(&c->xdpsq);
-	mlx5e_close_rq(&c->rq);
-	if (c->xdp)
-		mlx5e_close_xdpsq(&c->rq_xdpsq);
-	mlx5e_close_sqs(c);
-	mlx5e_close_icosq(&c->icosq);
-	napi_disable(&c->napi);
-	if (c->xdp)
-		mlx5e_close_cq(&c->rq_xdpsq.cq);
-	mlx5e_close_cq(&c->rq.cq);
-	mlx5e_close_cq(&c->xdpsq.cq);
-	mlx5e_close_tx_cqs(c);
-	mlx5e_close_cq(&c->icosq.cq);
+	mlx5e_close_queues(c);
 	netif_napi_del(&c->napi);
 	mlx5e_free_xps_cpumask(c);
 
-- 
2.19.1

[PATCH bpf-next v2 15/16] net/mlx5e: Move queue param structs to en/params.h

[Maxim Mikityanskiy]

structs mlx5e_{rq,sq,cq,channel}_param are going to be used in the
upcoming XSK RX and TX patches. Move them to a header file to make
them accessible from other C files.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 .../ethernet/mellanox/mlx5/core/en/params.h   | 31 +++++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 29 -----------------
 2 files changed, 31 insertions(+), 29 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
index 7f29b82dd8c2..f83417b822bf 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
@@ -11,6 +11,37 @@ struct mlx5e_xsk_param {
 	u16 chunk_size;
 };
 
+struct mlx5e_rq_param {
+	u32                        rqc[MLX5_ST_SZ_DW(rqc)];
+	struct mlx5_wq_param       wq;
+	struct mlx5e_rq_frags_info frags_info;
+};
+
+struct mlx5e_sq_param {
+	u32                        sqc[MLX5_ST_SZ_DW(sqc)];
+	struct mlx5_wq_param       wq;
+	bool                       is_mpw;
+};
+
+struct mlx5e_cq_param {
+	u32                        cqc[MLX5_ST_SZ_DW(cqc)];
+	struct mlx5_wq_param       wq;
+	u16                        eq_ix;
+	u8                         cq_period_mode;
+};
+
+struct mlx5e_channel_param {
+	struct mlx5e_rq_param      rq;
+	struct mlx5e_sq_param      sq;
+	struct mlx5e_sq_param      xdp_sq;
+	struct mlx5e_sq_param      icosq;
+	struct mlx5e_cq_param      rx_cq;
+	struct mlx5e_cq_param      tx_cq;
+	struct mlx5e_cq_param      icosq_cq;
+};
+
+/* Parameter calculations */
+
 u16 mlx5e_get_linear_rq_headroom(struct mlx5e_params *params,
 				 struct mlx5e_xsk_param *xsk);
 u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 9f74b9f72135..3b2d7a1fa7df 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -57,35 +57,6 @@
 #include "en/reporter.h"
 #include "en/params.h"
 
-struct mlx5e_rq_param {
-	u32			rqc[MLX5_ST_SZ_DW(rqc)];
-	struct mlx5_wq_param	wq;
-	struct mlx5e_rq_frags_info frags_info;
-};
-
-struct mlx5e_sq_param {
-	u32                        sqc[MLX5_ST_SZ_DW(sqc)];
-	struct mlx5_wq_param       wq;
-	bool                       is_mpw;
-};
-
-struct mlx5e_cq_param {
-	u32                        cqc[MLX5_ST_SZ_DW(cqc)];
-	struct mlx5_wq_param       wq;
-	u16                        eq_ix;
-	u8                         cq_period_mode;
-};
-
-struct mlx5e_channel_param {
-	struct mlx5e_rq_param      rq;
-	struct mlx5e_sq_param      sq;
-	struct mlx5e_sq_param      xdp_sq;
-	struct mlx5e_sq_param      icosq;
-	struct mlx5e_cq_param      rx_cq;
-	struct mlx5e_cq_param      tx_cq;
-	struct mlx5e_cq_param      icosq_cq;
-};
-
 bool mlx5e_check_fragmented_striding_rq_cap(struct mlx5_core_dev *mdev)
 {
 	bool striding_rq_umr = MLX5_CAP_GEN(mdev, striding_rq) &&
-- 
2.19.1

[PATCH bpf-next v2 16/16] net/mlx5e: Add XSK support

[Maxim Mikityanskiy]

This commit adds support for AF_XDP zero-copy RX and TX.

We create a dedicated XSK RQ inside the channel, it means that two
RQs are running simultaneously: one for non-XSK traffic and the other
for XSK traffic. They need to be distinguished though. Use the MSB of
the 32-bit queue index to indicate it's the XSK RQ in an XDP program.
When configuring flow steering with ethtool, use `user-def 1` to select
the XSK RQ, e.g.:

ethtool -N eth0 flow-type udp4 dst-port 4242 action 3 user-def 1

XSK requires different page allocation and release routines. Such
functions as mlx5e_{alloc,free}_rx_mpwqe and mlx5e_{get,put}_rx_frag are
generic enough to be used for both regular and XSK RQs, and they use the
mlx5e_page_{alloc,release} wrappers around the real allocation
functions. Function pointers are not used to avoid losing the
performance with retpolines. Wherever it's certain that the regular
(non-XSK) page release function should be used, it's called directly.

Only the stats that could be meaningful for XSK are exposed to the
userspace. Those that don't take part in the XSK flow are not
considered.

Note that we don't wait for WQEs on the XSK RQ (unlike the regular RQ),
because the newer xdpsock sample doesn't provide any Fill Ring entries
at the setup stage.

We create a dedicated XSK SQ in the channel. This separation has its
advantages:

1. When the UMEM is closed, the XSK SQ can also be closed and stop
receiving completions. If an existing SQ was used for XSK, it would
continue receiving completions for the packets of the closed socket. If
a new UMEM was opened at that point, it would start getting completions
that don't belong to it.

2. Calculating statistics separately.

When the userspace kicks the TX, the driver triggers a hardware
interrupt by posting a NOP to a dedicated XSK ICO (internal control
operations) SQ, in order to trigger NAPI on the right CPU core. This XSK
ICO SQ is protected by a spinlock, as the userspace application may kick
the TX from any core.

Store the pointers to the UMEMs in the net device private context,
independently from the kernel. This way the driver can distinguish
between the zero-copy and non-zero-copy UMEMs. The kernel function
xdp_get_umem_from_qid does not care about this difference, but the
driver is only interested in zero-copy UMEMs, particularly, on the
cleanup it determines whether to close the XSK RQ and SQ or not by
looking at the presence of the UMEM. Use state_lock to protect the
access to this area of UMEM pointers.

LRO isn't compatible with XDP, but there may be active UMEMs while
XDP is off. If this is the case, don't allow LRO to ensure XDP can
be reenabled at any time.

The validation of XSK parameters typically happens when XSK queues
open. However, when the interface is down or the XDP program isn't
set, it's still possible to have active AF_XDP sockets and even to
open new, but the XSK queues will be closed. To cover these cases,
perform the validation also in these flows:

1. A new UMEM is registered, but the XSK queues aren't going to be
created due to missing XDP program or interface being down.

2. MTU changes while there are UMEMs registered.

Having this early check prevents mlx5e_open_channels from failing
at a later stage, where recovery is impossible and the application
has no chance to handle the error, because it got the successful
return value for an MTU change or XSK open operation.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
---
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 drivers/net/ethernet/mellanox/mlx5/core/en.h  |  90 +++-
 .../ethernet/mellanox/mlx5/core/en/params.c   |  53 +-
 .../ethernet/mellanox/mlx5/core/en/params.h   |  46 +-
 .../net/ethernet/mellanox/mlx5/core/en/xdp.c  | 103 +++-
 .../net/ethernet/mellanox/mlx5/core/en/xdp.h  |  18 +-
 .../mellanox/mlx5/core/en/xsk/Makefile        |   1 +
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   | 192 +++++++
 .../ethernet/mellanox/mlx5/core/en/xsk/rx.h   |  27 +
 .../mellanox/mlx5/core/en/xsk/setup.c         | 220 ++++++++
 .../mellanox/mlx5/core/en/xsk/setup.h         |  25 +
 .../ethernet/mellanox/mlx5/core/en/xsk/tx.c   | 108 ++++
 .../ethernet/mellanox/mlx5/core/en/xsk/tx.h   |  15 +
 .../ethernet/mellanox/mlx5/core/en/xsk/umem.c | 252 +++++++++
 .../ethernet/mellanox/mlx5/core/en/xsk/umem.h |  34 ++
 .../ethernet/mellanox/mlx5/core/en_ethtool.c  |  21 +-
 .../mellanox/mlx5/core/en_fs_ethtool.c        |  44 +-
 .../net/ethernet/mellanox/mlx5/core/en_main.c | 506 ++++++++++++------
 .../net/ethernet/mellanox/mlx5/core/en_rep.c  |  12 +-
 .../net/ethernet/mellanox/mlx5/core/en_rx.c   |  77 ++-
 .../ethernet/mellanox/mlx5/core/en_stats.c    | 115 +++-
 .../ethernet/mellanox/mlx5/core/en_stats.h    |  30 ++
 .../net/ethernet/mellanox/mlx5/core/en_txrx.c |  34 +-
 .../ethernet/mellanox/mlx5/core/ipoib/ipoib.c |  14 +-
 24 files changed, 1788 insertions(+), 251 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/Makefile
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 3dbbe3b643b3..853479c367bf 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -23,7 +23,7 @@ mlx5_core-y :=	main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \
 mlx5_core-$(CONFIG_MLX5_CORE_EN) += en_main.o en_common.o en_fs.o en_ethtool.o \
 		en_tx.o en_rx.o en_dim.o en_txrx.o en/xdp.o en_stats.o \
 		en_selftest.o en/port.o en/monitor_stats.o en/reporter_tx.o \
-		en/params.o
+		en/params.o en/xsk/umem.o en/xsk/setup.o en/xsk/rx.o en/xsk/tx.o
 
 #
 # Netdev extra
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 8cb28e5604f0..26f160c09be9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -137,6 +137,7 @@ struct page_pool;
 #define MLX5E_MAX_NUM_CHANNELS         (MLX5E_INDIR_RQT_SIZE >> 1)
 #define MLX5E_MAX_NUM_SQS              (MLX5E_MAX_NUM_CHANNELS * MLX5E_MAX_NUM_TC)
 #define MLX5E_TX_CQ_POLL_BUDGET        128
+#define MLX5E_TX_XSK_POLL_BUDGET       64
 #define MLX5E_SQ_RECOVER_MIN_INTERVAL  500 /* msecs */
 
 #define MLX5E_UMR_WQE_INLINE_SZ \
@@ -250,6 +251,7 @@ struct mlx5e_params {
 	u32 lro_timeout;
 	u32 pflags;
 	struct bpf_prog *xdp_prog;
+	struct mlx5e_xsk *xsk;
 	unsigned int sw_mtu;
 	int hard_mtu;
 };
@@ -398,8 +400,14 @@ struct mlx5e_txqsq {
 } ____cacheline_aligned_in_smp;
 
 struct mlx5e_dma_info {
-	struct page     *page;
-	dma_addr_t      addr;
+	dma_addr_t addr;
+	union {
+		struct page *page;
+		struct {
+			u64 handle;
+			void *data;
+		} xsk;
+	};
 };
 
 /* XDP packets can be transmitted in different ways. On completion, we need to
@@ -466,9 +474,11 @@ struct mlx5e_xdp_mpwqe {
 };
 
 struct mlx5e_xdpsq;
+typedef int (*mlx5e_fp_xmit_xdp_frame_check)(struct mlx5e_xdpsq *);
 typedef bool (*mlx5e_fp_xmit_xdp_frame)(struct mlx5e_xdpsq *,
 					struct mlx5e_xdp_xmit_data *,
-					struct mlx5e_xdp_info *);
+					struct mlx5e_xdp_info *,
+					int);
 
 struct mlx5e_xdpsq {
 	/* data path */
@@ -486,8 +496,10 @@ struct mlx5e_xdpsq {
 	struct mlx5e_cq            cq;
 
 	/* read only */
+	struct xdp_umem           *umem;
 	struct mlx5_wq_cyc         wq;
 	struct mlx5e_xdpsq_stats  *stats;
+	mlx5e_fp_xmit_xdp_frame_check xmit_xdp_frame_check;
 	mlx5e_fp_xmit_xdp_frame    xmit_xdp_frame;
 	struct {
 		struct mlx5e_xdp_wqe_info *wqe_info;
@@ -618,6 +630,7 @@ struct mlx5e_rq {
 		} mpwqe;
 	};
 	struct {
+		u16            umem_headroom;
 		u16            headroom;
 		u8             map_dir;   /* dma map direction */
 	} buff;
@@ -648,6 +661,10 @@ struct mlx5e_rq {
 	DECLARE_BITMAP(flags, 8);
 	struct page_pool      *page_pool;
 
+	/* AF_XDP zero-copy */
+	struct zero_copy_allocator zca;
+	struct xdp_umem       *umem;
+
 	/* control */
 	struct mlx5_wq_ctrl    wq_ctrl;
 	__be32                 mkey_be;
@@ -660,6 +677,11 @@ struct mlx5e_rq {
 	struct xdp_rxq_info    xdp_rxq;
 } ____cacheline_aligned_in_smp;
 
+enum mlx5e_channel_state {
+	MLX5E_CHANNEL_STATE_XSK,
+	MLX5E_CHANNEL_NUM_STATES
+};
+
 struct mlx5e_channel {
 	/* data path */
 	struct mlx5e_rq            rq;
@@ -676,6 +698,13 @@ struct mlx5e_channel {
 	/* XDP_REDIRECT */
 	struct mlx5e_xdpsq         xdpsq;
 
+	/* AF_XDP zero-copy */
+	struct mlx5e_rq            xskrq;
+	struct mlx5e_xdpsq         xsksq;
+	struct mlx5e_icosq         xskicosq;
+	/* xskicosq can be accessed from any CPU - the spinlock protects it. */
+	spinlock_t                 xskicosq_lock;
+
 	/* data path - accessed per napi poll */
 	struct irq_desc *irq_desc;
 	struct mlx5e_ch_stats     *stats;
@@ -684,6 +713,7 @@ struct mlx5e_channel {
 	struct mlx5e_priv         *priv;
 	struct mlx5_core_dev      *mdev;
 	struct hwtstamp_config    *tstamp;
+	DECLARE_BITMAP(state, MLX5E_CHANNEL_NUM_STATES);
 	int                        ix;
 	int                        cpu;
 	cpumask_var_t              xps_cpumask;
@@ -699,14 +729,17 @@ struct mlx5e_channel_stats {
 	struct mlx5e_ch_stats ch;
 	struct mlx5e_sq_stats sq[MLX5E_MAX_NUM_TC];
 	struct mlx5e_rq_stats rq;
+	struct mlx5e_rq_stats xskrq;
 	struct mlx5e_xdpsq_stats rq_xdpsq;
 	struct mlx5e_xdpsq_stats xdpsq;
+	struct mlx5e_xdpsq_stats xsksq;
 } ____cacheline_aligned_in_smp;
 
 enum {
 	MLX5E_STATE_OPENED,
 	MLX5E_STATE_DESTROYING,
 	MLX5E_STATE_XDP_TX_ENABLED,
+	MLX5E_STATE_XDP_OPEN,
 };
 
 struct mlx5e_rqt {
@@ -739,6 +772,17 @@ struct mlx5e_modify_sq_param {
 	int rl_index;
 };
 
+struct mlx5e_xsk {
+	/* UMEMs are stored separately from channels, because we don't want to
+	 * lose them when channels are recreated. The kernel also stores UMEMs,
+	 * but it doesn't distinguish between zero-copy and non-zero-copy UMEMs,
+	 * so rely on our mechanism.
+	 */
+	struct xdp_umem **umems;
+	u16 refcnt;
+	bool ever_used;
+};
+
 struct mlx5e_priv {
 	/* priv data path fields - start */
 	struct mlx5e_txqsq *txq2sq[MLX5E_MAX_NUM_CHANNELS * MLX5E_MAX_NUM_TC];
@@ -759,6 +803,7 @@ struct mlx5e_priv {
 	struct mlx5e_tir           indir_tir[MLX5E_NUM_INDIR_TIRS];
 	struct mlx5e_tir           inner_indir_tir[MLX5E_NUM_INDIR_TIRS];
 	struct mlx5e_tir           direct_tir[MLX5E_MAX_NUM_CHANNELS];
+	struct mlx5e_tir           xsk_tir[MLX5E_MAX_NUM_CHANNELS];
 	struct mlx5e_rss_params    rss_params;
 	u32                        tx_rates[MLX5E_MAX_NUM_SQS];
 
@@ -795,6 +840,7 @@ struct mlx5e_priv {
 	struct mlx5e_tls          *tls;
 #endif
 	struct devlink_health_reporter *tx_reporter;
+	struct mlx5e_xsk           xsk;
 };
 
 struct mlx5e_profile {
@@ -838,8 +884,9 @@ bool mlx5e_striding_rq_possible(struct mlx5_core_dev *mdev,
 				struct mlx5e_params *params);
 
 void mlx5e_page_dma_unmap(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info);
-void mlx5e_page_release(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info,
-			bool recycle);
+void mlx5e_page_release_dynamic(struct mlx5e_rq *rq,
+				struct mlx5e_dma_info *dma_info,
+				bool recycle);
 void mlx5e_handle_rx_cqe(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe);
 void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe);
 bool mlx5e_post_rx_wqes(struct mlx5e_rq *rq);
@@ -899,6 +946,30 @@ void mlx5e_build_indir_tir_ctx_hash(struct mlx5e_rss_params *rss_params,
 void mlx5e_modify_tirs_hash(struct mlx5e_priv *priv, void *in, int inlen);
 struct mlx5e_tirc_config mlx5e_tirc_get_default_config(enum mlx5e_traffic_types tt);
 
+struct mlx5e_xsk_param;
+
+struct mlx5e_rq_param;
+int mlx5e_open_rq(struct mlx5e_channel *c, struct mlx5e_params *params,
+		  struct mlx5e_rq_param *param, struct mlx5e_xsk_param *xsk,
+		  struct xdp_umem *umem, struct mlx5e_rq *rq);
+int mlx5e_wait_for_min_rx_wqes(struct mlx5e_rq *rq, int wait_time);
+void mlx5e_deactivate_rq(struct mlx5e_rq *rq);
+void mlx5e_close_rq(struct mlx5e_rq *rq);
+
+struct mlx5e_sq_param;
+int mlx5e_open_icosq(struct mlx5e_channel *c, struct mlx5e_params *params,
+		     struct mlx5e_sq_param *param, struct mlx5e_icosq *sq);
+void mlx5e_close_icosq(struct mlx5e_icosq *sq);
+int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params,
+		     struct mlx5e_sq_param *param, struct xdp_umem *umem,
+		     struct mlx5e_xdpsq *sq, bool is_redirect);
+void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq);
+
+struct mlx5e_cq_param;
+int mlx5e_open_cq(struct mlx5e_channel *c, struct net_dim_cq_moder moder,
+		  struct mlx5e_cq_param *param, struct mlx5e_cq *cq);
+void mlx5e_close_cq(struct mlx5e_cq *cq);
+
 int mlx5e_open_locked(struct net_device *netdev);
 int mlx5e_close_locked(struct net_device *netdev);
 
@@ -1069,10 +1140,10 @@ int mlx5e_create_indirect_rqt(struct mlx5e_priv *priv);
 int mlx5e_create_indirect_tirs(struct mlx5e_priv *priv, bool inner_ttc);
 void mlx5e_destroy_indirect_tirs(struct mlx5e_priv *priv, bool inner_ttc);
 
-int mlx5e_create_direct_rqts(struct mlx5e_priv *priv);
-void mlx5e_destroy_direct_rqts(struct mlx5e_priv *priv);
-int mlx5e_create_direct_tirs(struct mlx5e_priv *priv);
-void mlx5e_destroy_direct_tirs(struct mlx5e_priv *priv);
+int mlx5e_create_direct_rqts(struct mlx5e_priv *priv, struct mlx5e_tir *tirs);
+void mlx5e_destroy_direct_rqts(struct mlx5e_priv *priv, struct mlx5e_tir *tirs);
+int mlx5e_create_direct_tirs(struct mlx5e_priv *priv, struct mlx5e_tir *tirs);
+void mlx5e_destroy_direct_tirs(struct mlx5e_priv *priv, struct mlx5e_tir *tirs);
 void mlx5e_destroy_rqt(struct mlx5e_priv *priv, struct mlx5e_rqt *rqt);
 
 int mlx5e_create_tis(struct mlx5_core_dev *mdev, int tc,
@@ -1143,6 +1214,7 @@ void mlx5e_detach_netdev(struct mlx5e_priv *priv);
 void mlx5e_destroy_netdev(struct mlx5e_priv *priv);
 void mlx5e_set_netdev_mtu_boundaries(struct mlx5e_priv *priv);
 void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
+			    struct mlx5e_xsk *xsk,
 			    struct mlx5e_rss_params *rss_params,
 			    struct mlx5e_params *params,
 			    u16 max_channels, u16 mtu);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
index 0de908b12fcc..79301d116667 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.c
@@ -49,48 +49,56 @@ u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params,
 	return frag_sz;
 }
 
-u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5e_params *params)
+u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5e_params *params,
+				struct mlx5e_xsk_param *xsk)
 {
-	u32 linear_frag_sz = mlx5e_rx_get_linear_frag_sz(params, NULL);
+	u32 linear_frag_sz = mlx5e_rx_get_linear_frag_sz(params, xsk);
 
 	return MLX5_MPWRQ_LOG_WQE_SZ - order_base_2(linear_frag_sz);
 }
 
-bool mlx5e_rx_is_linear_skb(struct mlx5e_params *params)
+bool mlx5e_rx_is_linear_skb(struct mlx5e_params *params,
+			    struct mlx5e_xsk_param *xsk)
 {
-	u32 frag_sz = mlx5e_rx_get_linear_frag_sz(params, NULL);
+	/* AF_XDP allocates SKBs on XDP_PASS - ensure they don't occupy more
+	 * than one page. For this, check both with and without xsk.
+	 */
+	u32 linear_frag_sz = max(mlx5e_rx_get_linear_frag_sz(params, xsk),
+				 mlx5e_rx_get_linear_frag_sz(params, NULL));
 
-	return !params->lro_en && frag_sz <= PAGE_SIZE;
+	return !params->lro_en && linear_frag_sz <= PAGE_SIZE;
 }
 
 #define MLX5_MAX_MPWQE_LOG_WQE_STRIDE_SZ ((BIT(__mlx5_bit_sz(wq, log_wqe_stride_size)) - 1) + \
 					  MLX5_MPWQE_LOG_STRIDE_SZ_BASE)
 bool mlx5e_rx_mpwqe_is_linear_skb(struct mlx5_core_dev *mdev,
-				  struct mlx5e_params *params)
+				  struct mlx5e_params *params,
+				  struct mlx5e_xsk_param *xsk)
 {
-	u32 frag_sz = mlx5e_rx_get_linear_frag_sz(params, NULL);
+	u32 linear_frag_sz = mlx5e_rx_get_linear_frag_sz(params, xsk);
 	s8 signed_log_num_strides_param;
 	u8 log_num_strides;
 
-	if (!mlx5e_rx_is_linear_skb(params))
+	if (!mlx5e_rx_is_linear_skb(params, xsk))
 		return false;
 
-	if (order_base_2(frag_sz) > MLX5_MAX_MPWQE_LOG_WQE_STRIDE_SZ)
+	if (order_base_2(linear_frag_sz) > MLX5_MAX_MPWQE_LOG_WQE_STRIDE_SZ)
 		return false;
 
 	if (MLX5_CAP_GEN(mdev, ext_stride_num_range))
 		return true;
 
-	log_num_strides = MLX5_MPWRQ_LOG_WQE_SZ - order_base_2(frag_sz);
+	log_num_strides = MLX5_MPWRQ_LOG_WQE_SZ - order_base_2(linear_frag_sz);
 	signed_log_num_strides_param =
 		(s8)log_num_strides - MLX5_MPWQE_LOG_NUM_STRIDES_BASE;
 
 	return signed_log_num_strides_param >= 0;
 }
 
-u8 mlx5e_mpwqe_get_log_rq_size(struct mlx5e_params *params)
+u8 mlx5e_mpwqe_get_log_rq_size(struct mlx5e_params *params,
+			       struct mlx5e_xsk_param *xsk)
 {
-	u8 log_pkts_per_wqe = mlx5e_mpwqe_log_pkts_per_wqe(params);
+	u8 log_pkts_per_wqe = mlx5e_mpwqe_log_pkts_per_wqe(params, xsk);
 
 	/* Numbers are unsigned, don't subtract to avoid underflow. */
 	if (params->log_rq_mtu_frames <
@@ -101,27 +109,30 @@ u8 mlx5e_mpwqe_get_log_rq_size(struct mlx5e_params *params)
 }
 
 u8 mlx5e_mpwqe_get_log_stride_size(struct mlx5_core_dev *mdev,
-				   struct mlx5e_params *params)
+				   struct mlx5e_params *params,
+				   struct mlx5e_xsk_param *xsk)
 {
-	if (mlx5e_rx_mpwqe_is_linear_skb(mdev, params))
-		return order_base_2(mlx5e_rx_get_linear_frag_sz(params, NULL));
+	if (mlx5e_rx_mpwqe_is_linear_skb(mdev, params, xsk))
+		return order_base_2(mlx5e_rx_get_linear_frag_sz(params, xsk));
 
 	return MLX5_MPWRQ_DEF_LOG_STRIDE_SZ(mdev);
 }
 
 u8 mlx5e_mpwqe_get_log_num_strides(struct mlx5_core_dev *mdev,
-				   struct mlx5e_params *params)
+				   struct mlx5e_params *params,
+				   struct mlx5e_xsk_param *xsk)
 {
 	return MLX5_MPWRQ_LOG_WQE_SZ -
-		mlx5e_mpwqe_get_log_stride_size(mdev, params);
+		mlx5e_mpwqe_get_log_stride_size(mdev, params, xsk);
 }
 
 u16 mlx5e_get_rq_headroom(struct mlx5_core_dev *mdev,
-			  struct mlx5e_params *params)
+			  struct mlx5e_params *params,
+			  struct mlx5e_xsk_param *xsk)
 {
 	bool is_linear_skb = (params->rq_wq_type == MLX5_WQ_TYPE_CYCLIC) ?
-		mlx5e_rx_is_linear_skb(params) :
-		mlx5e_rx_mpwqe_is_linear_skb(mdev, params);
+		mlx5e_rx_is_linear_skb(params, xsk) :
+		mlx5e_rx_mpwqe_is_linear_skb(mdev, params, xsk);
 
-	return is_linear_skb ? mlx5e_get_linear_rq_headroom(params, NULL) : 0;
+	return is_linear_skb ? mlx5e_get_linear_rq_headroom(params, xsk) : 0;
 }
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
index f83417b822bf..352e480fa469 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/params.h
@@ -46,16 +46,48 @@ u16 mlx5e_get_linear_rq_headroom(struct mlx5e_params *params,
 				 struct mlx5e_xsk_param *xsk);
 u32 mlx5e_rx_get_linear_frag_sz(struct mlx5e_params *params,
 				struct mlx5e_xsk_param *xsk);
-u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5e_params *params);
-bool mlx5e_rx_is_linear_skb(struct mlx5e_params *params);
+u8 mlx5e_mpwqe_log_pkts_per_wqe(struct mlx5e_params *params,
+				struct mlx5e_xsk_param *xsk);
+bool mlx5e_rx_is_linear_skb(struct mlx5e_params *params,
+			    struct mlx5e_xsk_param *xsk);
 bool mlx5e_rx_mpwqe_is_linear_skb(struct mlx5_core_dev *mdev,
-				  struct mlx5e_params *params);
-u8 mlx5e_mpwqe_get_log_rq_size(struct mlx5e_params *params);
+				  struct mlx5e_params *params,
+				  struct mlx5e_xsk_param *xsk);
+u8 mlx5e_mpwqe_get_log_rq_size(struct mlx5e_params *params,
+			       struct mlx5e_xsk_param *xsk);
 u8 mlx5e_mpwqe_get_log_stride_size(struct mlx5_core_dev *mdev,
-				   struct mlx5e_params *params);
+				   struct mlx5e_params *params,
+				   struct mlx5e_xsk_param *xsk);
 u8 mlx5e_mpwqe_get_log_num_strides(struct mlx5_core_dev *mdev,
-				   struct mlx5e_params *params);
+				   struct mlx5e_params *params,
+				   struct mlx5e_xsk_param *xsk);
 u16 mlx5e_get_rq_headroom(struct mlx5_core_dev *mdev,
-			  struct mlx5e_params *params);
+			  struct mlx5e_params *params,
+			  struct mlx5e_xsk_param *xsk);
+
+/* Build queue parameters */
+
+void mlx5e_build_rq_param(struct mlx5e_priv *priv,
+			  struct mlx5e_params *params,
+			  struct mlx5e_xsk_param *xsk,
+			  struct mlx5e_rq_param *param);
+void mlx5e_build_sq_param_common(struct mlx5e_priv *priv,
+				 struct mlx5e_sq_param *param);
+void mlx5e_build_rx_cq_param(struct mlx5e_priv *priv,
+			     struct mlx5e_params *params,
+			     struct mlx5e_xsk_param *xsk,
+			     struct mlx5e_cq_param *param);
+void mlx5e_build_tx_cq_param(struct mlx5e_priv *priv,
+			     struct mlx5e_params *params,
+			     struct mlx5e_cq_param *param);
+void mlx5e_build_ico_cq_param(struct mlx5e_priv *priv,
+			      u8 log_wq_size,
+			      struct mlx5e_cq_param *param);
+void mlx5e_build_icosq_param(struct mlx5e_priv *priv,
+			     u8 log_wq_size,
+			     struct mlx5e_sq_param *param);
+void mlx5e_build_xdpsq_param(struct mlx5e_priv *priv,
+			     struct mlx5e_params *params,
+			     struct mlx5e_sq_param *param);
 
 #endif /* __MLX5_EN_PARAMS_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
index ee99efde9143..b0b982cf69bb 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
@@ -31,6 +31,7 @@
  */
 
 #include <linux/bpf_trace.h>
+#include <net/xdp_sock.h>
 #include "en/xdp.h"
 #include "en/params.h"
 
@@ -113,12 +114,12 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq,
 		xdpi.page.di    = *di;
 	}
 
-	return sq->xmit_xdp_frame(sq, &xdptxd, &xdpi);
+	return sq->xmit_xdp_frame(sq, &xdptxd, &xdpi, 0);
 }
 
 /* returns true if packet was consumed by xdp */
 bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,
-		      void *va, u16 *rx_headroom, u32 *len)
+		      void *va, u16 *rx_headroom, u32 *len, bool xsk)
 {
 	struct bpf_prog *prog = READ_ONCE(rq->xdp_prog);
 	struct xdp_buff xdp;
@@ -132,9 +133,13 @@ bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,
 	xdp_set_data_meta_invalid(&xdp);
 	xdp.data_end = xdp.data + *len;
 	xdp.data_hard_start = va;
+	if (xsk)
+		xdp.handle = di->xsk.handle;
 	xdp.rxq = &rq->xdp_rxq;
 
 	act = bpf_prog_run_xdp(prog, &xdp);
+	if (xsk)
+		xdp.handle += xdp.data - xdp.data_hard_start;
 	switch (act) {
 	case XDP_PASS:
 		*rx_headroom = xdp.data - xdp.data_hard_start;
@@ -152,7 +157,8 @@ bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,
 			goto xdp_abort;
 		__set_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags);
 		__set_bit(MLX5E_RQ_FLAG_XDP_REDIRECT, rq->flags);
-		mlx5e_page_dma_unmap(rq, di);
+		if (!xsk)
+			mlx5e_page_dma_unmap(rq, di);
 		rq->stats->xdp_redirect++;
 		return true;
 	default:
@@ -206,7 +212,7 @@ static void mlx5e_xdp_mpwqe_session_start(struct mlx5e_xdpsq *sq)
 	stats->mpwqe++;
 }
 
-static void mlx5e_xdp_mpwqe_complete(struct mlx5e_xdpsq *sq)
+void mlx5e_xdp_mpwqe_complete(struct mlx5e_xdpsq *sq)
 {
 	struct mlx5_wq_cyc       *wq    = &sq->wq;
 	struct mlx5e_xdp_mpwqe *session = &sq->mpwqe;
@@ -229,9 +235,32 @@ static void mlx5e_xdp_mpwqe_complete(struct mlx5e_xdpsq *sq)
 	session->wqe = NULL; /* Close session */
 }
 
+enum {
+	MLX5E_XDP_CHECK_OK = 1,
+	MLX5E_XDP_CHECK_START_MPWQE = 2,
+};
+
+static int mlx5e_xmit_xdp_frame_check_mpwqe(struct mlx5e_xdpsq *sq)
+{
+	if (unlikely(!sq->mpwqe.wqe)) {
+		if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc,
+						     MLX5_SEND_WQE_MAX_WQEBBS))) {
+			/* SQ is full, ring doorbell */
+			mlx5e_xmit_xdp_doorbell(sq);
+			sq->stats->full++;
+			return -EBUSY;
+		}
+
+		return MLX5E_XDP_CHECK_START_MPWQE;
+	}
+
+	return MLX5E_XDP_CHECK_OK;
+}
+
 static bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq,
 				       struct mlx5e_xdp_xmit_data *xdptxd,
-				       struct mlx5e_xdp_info *xdpi)
+				       struct mlx5e_xdp_info *xdpi,
+				       int check_result)
 {
 	struct mlx5e_xdp_mpwqe *session = &sq->mpwqe;
 	struct mlx5e_xdpsq_stats *stats = sq->stats;
@@ -241,15 +270,16 @@ static bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq,
 		return false;
 	}
 
-	if (unlikely(!session->wqe)) {
-		if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc,
-						     MLX5_SEND_WQE_MAX_WQEBBS))) {
-			/* SQ is full, ring doorbell */
-			mlx5e_xmit_xdp_doorbell(sq);
-			stats->full++;
-			return false;
-		}
+	if (!check_result)
+		check_result = mlx5e_xmit_xdp_frame_check_mpwqe(sq);
+	if (unlikely(check_result < 0))
+		return false;
 
+	if (check_result == MLX5E_XDP_CHECK_START_MPWQE) {
+		/* Start the session when nothing can fail, so it's guaranteed
+		 * that if there is an active session, it has at least one dseg,
+		 * and it's safe to complete it at any time.
+		 */
 		mlx5e_xdp_mpwqe_session_start(sq);
 	}
 
@@ -264,9 +294,22 @@ static bool mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq,
 	return true;
 }
 
+static int mlx5e_xmit_xdp_frame_check(struct mlx5e_xdpsq *sq)
+{
+	if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, 1))) {
+		/* SQ is full, ring doorbell */
+		mlx5e_xmit_xdp_doorbell(sq);
+		sq->stats->full++;
+		return -EBUSY;
+	}
+
+	return MLX5E_XDP_CHECK_OK;
+}
+
 static bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq,
 				 struct mlx5e_xdp_xmit_data *xdptxd,
-				 struct mlx5e_xdp_info *xdpi)
+				 struct mlx5e_xdp_info *xdpi,
+				 int check_result)
 {
 	struct mlx5_wq_cyc       *wq   = &sq->wq;
 	u16                       pi   = mlx5_wq_cyc_ctr2ix(wq, sq->pc);
@@ -288,12 +331,10 @@ static bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq,
 		return false;
 	}
 
-	if (unlikely(!mlx5e_wqc_has_room_for(wq, sq->cc, sq->pc, 1))) {
-		/* SQ is full, ring doorbell */
-		mlx5e_xmit_xdp_doorbell(sq);
-		stats->full++;
+	if (!check_result)
+		check_result = mlx5e_xmit_xdp_frame_check(sq);
+	if (unlikely(check_result < 0))
 		return false;
-	}
 
 	cseg->fm_ce_se = 0;
 
@@ -323,6 +364,7 @@ static bool mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq,
 
 static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq,
 				  struct mlx5e_xdp_wqe_info *wi,
+				  u32 *xsk_frames,
 				  bool recycle)
 {
 	struct mlx5e_xdp_info_fifo *xdpi_fifo = &sq->db.xdpi_fifo;
@@ -340,7 +382,11 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq,
 			break;
 		case MLX5E_XDP_XMIT_MODE_PAGE:
 			/* XDP_TX from the regular RQ */
-			mlx5e_page_release(xdpi.page.rq, &xdpi.page.di, recycle);
+			mlx5e_page_release_dynamic(xdpi.page.rq, &xdpi.page.di, recycle);
+			break;
+		case MLX5E_XDP_XMIT_MODE_XSK:
+			/* AF_XDP send */
+			(*xsk_frames)++;
 			break;
 		default:
 			WARN_ON_ONCE(true);
@@ -352,6 +398,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
 {
 	struct mlx5e_xdpsq *sq;
 	struct mlx5_cqe64 *cqe;
+	u32 xsk_frames = 0;
 	u16 sqcc;
 	int i;
 
@@ -393,10 +440,13 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
 
 			sqcc += wi->num_wqebbs;
 
-			mlx5e_free_xdpsq_desc(sq, wi, true);
+			mlx5e_free_xdpsq_desc(sq, wi, &xsk_frames, true);
 		} while (!last_wqe);
 	} while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq)));
 
+	if (xsk_frames)
+		xsk_umem_complete_tx(sq->umem, xsk_frames);
+
 	sq->stats->cqes += i;
 
 	mlx5_cqwq_update_db_record(&cq->wq);
@@ -410,6 +460,8 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
 
 void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq)
 {
+	u32 xsk_frames = 0;
+
 	while (sq->cc != sq->pc) {
 		struct mlx5e_xdp_wqe_info *wi;
 		u16 ci;
@@ -419,8 +471,11 @@ void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq)
 
 		sq->cc += wi->num_wqebbs;
 
-		mlx5e_free_xdpsq_desc(sq, wi, false);
+		mlx5e_free_xdpsq_desc(sq, wi, &xsk_frames, false);
 	}
+
+	if (xsk_frames)
+		xsk_umem_complete_tx(sq->umem, xsk_frames);
 }
 
 int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames,
@@ -466,7 +521,7 @@ int mlx5e_xdp_xmit(struct net_device *dev, int n, struct xdp_frame **frames,
 		xdpi.frame.xdpf     = xdpf;
 		xdpi.frame.dma_addr = xdptxd.dma_addr;
 
-		if (unlikely(!sq->xmit_xdp_frame(sq, &xdptxd, &xdpi))) {
+		if (unlikely(!sq->xmit_xdp_frame(sq, &xdptxd, &xdpi, 0))) {
 			dma_unmap_single(sq->pdev, xdptxd.dma_addr,
 					 xdptxd.len, DMA_TO_DEVICE);
 			xdp_return_frame_rx_napi(xdpf);
@@ -500,6 +555,8 @@ void mlx5e_xdp_rx_poll_complete(struct mlx5e_rq *rq)
 
 void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw)
 {
+	sq->xmit_xdp_frame_check = is_mpw ?
+		mlx5e_xmit_xdp_frame_check_mpwqe : mlx5e_xmit_xdp_frame_check;
 	sq->xmit_xdp_frame = is_mpw ?
 		mlx5e_xmit_xdp_frame_mpwqe : mlx5e_xmit_xdp_frame;
 }
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
index 9200cb9f499b..2d934c8d3807 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
@@ -42,7 +42,8 @@
 struct mlx5e_xsk_param;
 int mlx5e_xdp_max_mtu(struct mlx5e_params *params, struct mlx5e_xsk_param *xsk);
 bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,
-		      void *va, u16 *rx_headroom, u32 *len);
+		      void *va, u16 *rx_headroom, u32 *len, bool xsk);
+void mlx5e_xdp_mpwqe_complete(struct mlx5e_xdpsq *sq);
 bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq);
 void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq);
 void mlx5e_set_xmit_fp(struct mlx5e_xdpsq *sq, bool is_mpw);
@@ -67,6 +68,21 @@ static inline bool mlx5e_xdp_tx_is_enabled(struct mlx5e_priv *priv)
 	return test_bit(MLX5E_STATE_XDP_TX_ENABLED, &priv->state);
 }
 
+static inline void mlx5e_xdp_set_open(struct mlx5e_priv *priv)
+{
+	set_bit(MLX5E_STATE_XDP_OPEN, &priv->state);
+}
+
+static inline void mlx5e_xdp_set_closed(struct mlx5e_priv *priv)
+{
+	clear_bit(MLX5E_STATE_XDP_OPEN, &priv->state);
+}
+
+static inline bool mlx5e_xdp_is_open(struct mlx5e_priv *priv)
+{
+	return test_bit(MLX5E_STATE_XDP_OPEN, &priv->state);
+}
+
 static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_xdpsq *sq)
 {
 	if (sq->doorbell_cseg) {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/Makefile
new file mode 100644
index 000000000000..5ee42991900a
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/Makefile
@@ -0,0 +1 @@
+subdir-ccflags-y += -I$(src)/../..
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
new file mode 100644
index 000000000000..6a55573ec8f2
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
@@ -0,0 +1,192 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2019 Mellanox Technologies. */
+
+#include "rx.h"
+#include "en/xdp.h"
+#include <net/xdp_sock.h>
+
+/* RX data path */
+
+bool mlx5e_xsk_pages_enough_umem(struct mlx5e_rq *rq, int count)
+{
+	/* Check in advance that we have enough frames, instead of allocating
+	 * one-by-one, failing and moving frames to the Reuse Ring.
+	 */
+	return xsk_umem_has_addrs_rq(rq->umem, count);
+}
+
+int mlx5e_xsk_page_alloc_umem(struct mlx5e_rq *rq,
+			      struct mlx5e_dma_info *dma_info)
+{
+	struct xdp_umem *umem = rq->umem;
+	u64 handle;
+
+	if (!xsk_umem_peek_addr_rq(umem, &handle))
+		return -ENOMEM;
+
+	dma_info->xsk.handle = handle + rq->buff.umem_headroom;
+	dma_info->xsk.data = xdp_umem_get_data(umem, dma_info->xsk.handle);
+
+	/* No need to add headroom to the DMA address. In striding RQ case, we
+	 * just provide pages for UMR, and headroom is counted at the setup
+	 * stage when creating a WQE. In non-striding RQ case, headroom is
+	 * accounted in mlx5e_alloc_rx_wqe.
+	 */
+	dma_info->addr = xdp_umem_get_dma(umem, handle);
+
+	xsk_umem_discard_addr_rq(umem);
+
+	dma_sync_single_for_device(rq->pdev, dma_info->addr, PAGE_SIZE,
+				   DMA_BIDIRECTIONAL);
+
+	return 0;
+}
+
+static inline void mlx5e_xsk_recycle_frame(struct mlx5e_rq *rq, u64 handle)
+{
+	xsk_umem_fq_reuse(rq->umem, handle & rq->umem->chunk_mask);
+}
+
+/* XSKRQ uses pages from UMEM, they must not be released. They are returned to
+ * the userspace if possible, and if not, this function is called to reuse them
+ * in the driver.
+ */
+void mlx5e_xsk_page_release(struct mlx5e_rq *rq,
+			    struct mlx5e_dma_info *dma_info)
+{
+	mlx5e_xsk_recycle_frame(rq, dma_info->xsk.handle);
+}
+
+/* Return a frame back to the hardware to fill in again. It is used by XDP when
+ * the XDP program returns XDP_TX or XDP_REDIRECT not to an XSKMAP.
+ */
+void mlx5e_xsk_zca_free(struct zero_copy_allocator *zca, unsigned long handle)
+{
+	struct mlx5e_rq *rq = container_of(zca, struct mlx5e_rq, zca);
+
+	mlx5e_xsk_recycle_frame(rq, handle);
+}
+
+static struct sk_buff *mlx5e_xsk_construct_skb(struct mlx5e_rq *rq, void *data,
+					       u32 cqe_bcnt)
+{
+	struct sk_buff *skb;
+
+	skb = napi_alloc_skb(rq->cq.napi, cqe_bcnt);
+	if (unlikely(!skb)) {
+		rq->stats->buff_alloc_err++;
+		return NULL;
+	}
+
+	skb_put_data(skb, data, cqe_bcnt);
+
+	return skb;
+}
+
+struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,
+						    struct mlx5e_mpw_info *wi,
+						    u16 cqe_bcnt,
+						    u32 head_offset,
+						    u32 page_idx)
+{
+	struct mlx5e_dma_info *di = &wi->umr.dma_info[page_idx];
+	u16 rx_headroom = rq->buff.headroom - rq->buff.umem_headroom;
+	u32 cqe_bcnt32 = cqe_bcnt;
+	void *va, *data;
+	u32 frag_size;
+	bool consumed;
+
+	/* Check packet size. Note LRO doesn't use linear SKB */
+	if (unlikely(cqe_bcnt > rq->hw_mtu)) {
+		rq->stats->oversize_pkts_sw_drop++;
+		return NULL;
+	}
+
+	/* head_offset is not used in this function, because di->xsk.data and
+	 * di->addr point directly to the necessary place. Furthermore, in the
+	 * current implementation, one page = one packet = one frame, so
+	 * head_offset should always be 0.
+	 */
+	WARN_ON_ONCE(head_offset);
+
+	va             = di->xsk.data;
+	data           = va + rx_headroom;
+	frag_size      = rq->buff.headroom + cqe_bcnt32;
+
+	dma_sync_single_for_cpu(rq->pdev, di->addr, frag_size, DMA_BIDIRECTIONAL);
+	prefetch(data);
+
+	rcu_read_lock();
+	consumed = mlx5e_xdp_handle(rq, di, va, &rx_headroom, &cqe_bcnt32, true);
+	rcu_read_unlock();
+
+	/* Possible flows:
+	 * - XDP_REDIRECT to XSKMAP:
+	 *   The page is owned by the userspace from now.
+	 * - XDP_TX and other XDP_REDIRECTs:
+	 *   The page was returned by ZCA and recycled.
+	 * - XDP_DROP:
+	 *   Recycle the page.
+	 * - XDP_PASS:
+	 *   Allocate an SKB, copy the data and recycle the page.
+	 *
+	 * Pages to be recycled go to the Reuse Ring on MPWQE deallocation. Its
+	 * size is the same as the Driver RX Ring's size, and pages for WQEs are
+	 * allocated first from the Reuse Ring, so it has enough space.
+	 */
+
+	if (likely(consumed)) {
+		if (likely(__test_and_clear_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags)))
+			__set_bit(page_idx, wi->xdp_xmit_bitmap); /* non-atomic */
+		return NULL; /* page/packet was consumed by XDP */
+	}
+
+	/* XDP_PASS: copy the data from the UMEM to a new SKB and reuse the
+	 * frame. On SKB allocation failure, NULL is returned.
+	 */
+	return mlx5e_xsk_construct_skb(rq, data, cqe_bcnt32);
+}
+
+struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq *rq,
+					      struct mlx5_cqe64 *cqe,
+					      struct mlx5e_wqe_frag_info *wi,
+					      u32 cqe_bcnt)
+{
+	struct mlx5e_dma_info *di = wi->di;
+	u16 rx_headroom = rq->buff.headroom - rq->buff.umem_headroom;
+	void *va, *data;
+	bool consumed;
+	u32 frag_size;
+
+	/* wi->offset is not used in this function, because di->xsk.data and
+	 * di->addr point directly to the necessary place. Furthermore, in the
+	 * current implementation, one page = one packet = one frame, so
+	 * wi->offset should always be 0.
+	 */
+	WARN_ON_ONCE(wi->offset);
+
+	va             = di->xsk.data;
+	data           = va + rx_headroom;
+	frag_size      = rq->buff.headroom + cqe_bcnt;
+
+	dma_sync_single_for_cpu(rq->pdev, di->addr, frag_size, DMA_BIDIRECTIONAL);
+	prefetch(data);
+
+	if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_RESP_SEND)) {
+		rq->stats->wqe_err++;
+		return NULL;
+	}
+
+	rcu_read_lock();
+	consumed = mlx5e_xdp_handle(rq, di, va, &rx_headroom, &cqe_bcnt, true);
+	rcu_read_unlock();
+
+	if (likely(consumed))
+		return NULL; /* page/packet was consumed by XDP */
+
+	/* XDP_PASS: copy the data from the UMEM to a new SKB. The frame reuse
+	 * will be handled by mlx5e_put_rx_frag.
+	 * On SKB allocation failure, NULL is returned.
+	 */
+	return mlx5e_xsk_construct_skb(rq, data, cqe_bcnt);
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h
new file mode 100644
index 000000000000..307b923a1361
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2019 Mellanox Technologies. */
+
+#ifndef __MLX5_EN_XSK_RX_H__
+#define __MLX5_EN_XSK_RX_H__
+
+#include "en.h"
+
+/* RX data path */
+
+bool mlx5e_xsk_pages_enough_umem(struct mlx5e_rq *rq, int count);
+int mlx5e_xsk_page_alloc_umem(struct mlx5e_rq *rq,
+			      struct mlx5e_dma_info *dma_info);
+void mlx5e_xsk_page_release(struct mlx5e_rq *rq,
+			    struct mlx5e_dma_info *dma_info);
+void mlx5e_xsk_zca_free(struct zero_copy_allocator *zca, unsigned long handle);
+struct sk_buff *mlx5e_xsk_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq,
+						    struct mlx5e_mpw_info *wi,
+						    u16 cqe_bcnt,
+						    u32 head_offset,
+						    u32 page_idx);
+struct sk_buff *mlx5e_xsk_skb_from_cqe_linear(struct mlx5e_rq *rq,
+					      struct mlx5_cqe64 *cqe,
+					      struct mlx5e_wqe_frag_info *wi,
+					      u32 cqe_bcnt);
+
+#endif /* __MLX5_EN_XSK_RX_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c
new file mode 100644
index 000000000000..f2d8c465b61b
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c
@@ -0,0 +1,220 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2019 Mellanox Technologies. */
+
+#include "setup.h"
+#include "en/params.h"
+
+bool mlx5e_validate_xsk_param(struct mlx5e_params *params,
+			      struct mlx5e_xsk_param *xsk,
+			      struct mlx5_core_dev *mdev)
+{
+	/* AF_XDP doesn't support frames larger than PAGE_SIZE, and the current
+	 * mlx5e XDP implementation doesn't support multiple packets per page.
+	 */
+	if (xsk->chunk_size != PAGE_SIZE)
+		return false;
+
+	/* Current MTU and XSK headroom don't allow packets to fit the frames. */
+	if (mlx5e_rx_get_linear_frag_sz(params, xsk) > xsk->chunk_size)
+		return false;
+
+	/* frag_sz is different for regular and XSK RQs, so ensure that linear
+	 * SKB mode is possible.
+	 */
+	switch (params->rq_wq_type) {
+	case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
+		return mlx5e_rx_mpwqe_is_linear_skb(mdev, params, xsk);
+	default: /* MLX5_WQ_TYPE_CYCLIC */
+		return mlx5e_rx_is_linear_skb(params, xsk);
+	}
+}
+
+static void mlx5e_build_xskicosq_param(struct mlx5e_priv *priv,
+				       u8 log_wq_size,
+				       struct mlx5e_sq_param *param)
+{
+	void *sqc = param->sqc;
+	void *wq = MLX5_ADDR_OF(sqc, sqc, wq);
+
+	mlx5e_build_sq_param_common(priv, param);
+
+	MLX5_SET(wq, wq, log_wq_sz, log_wq_size);
+}
+
+static void mlx5e_build_xsk_cparam(struct mlx5e_priv *priv,
+				   struct mlx5e_params *params,
+				   struct mlx5e_xsk_param *xsk,
+				   struct mlx5e_channel_param *cparam)
+{
+	const u8 xskicosq_size = MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE;
+
+	mlx5e_build_rq_param(priv, params, xsk, &cparam->rq);
+	mlx5e_build_xdpsq_param(priv, params, &cparam->xdp_sq);
+	mlx5e_build_xskicosq_param(priv, xskicosq_size, &cparam->icosq);
+	mlx5e_build_rx_cq_param(priv, params, xsk, &cparam->rx_cq);
+	mlx5e_build_tx_cq_param(priv, params, &cparam->tx_cq);
+	mlx5e_build_ico_cq_param(priv, xskicosq_size, &cparam->icosq_cq);
+}
+
+int mlx5e_open_xsk(struct mlx5e_priv *priv, struct mlx5e_params *params,
+		   struct mlx5e_xsk_param *xsk, struct xdp_umem *umem,
+		   struct mlx5e_channel *c)
+{
+	struct mlx5e_channel_param cparam = {};
+	struct net_dim_cq_moder icocq_moder = {};
+	int err;
+
+	if (!mlx5e_validate_xsk_param(params, xsk, priv->mdev))
+		return -EINVAL;
+
+	mlx5e_build_xsk_cparam(priv, params, xsk, &cparam);
+
+	err = mlx5e_open_cq(c, params->rx_cq_moderation, &cparam.rx_cq, &c->xskrq.cq);
+	if (unlikely(err))
+		return err;
+
+	err = mlx5e_open_rq(c, params, &cparam.rq, xsk, umem, &c->xskrq);
+	if (unlikely(err))
+		goto err_close_rx_cq;
+
+	err = mlx5e_open_cq(c, params->tx_cq_moderation, &cparam.tx_cq, &c->xsksq.cq);
+	if (unlikely(err))
+		return err;
+
+	/* Create a separate SQ, so that when the UMEM is disabled, we could
+	 * close this SQ safely and stop receiving CQEs. In other case, e.g., if
+	 * the XDPSQ was used instead, we might run into trouble when the UMEM
+	 * is disabled and then reenabled, but the SQ continues receiving CQEs
+	 * from the old UMEM.
+	 */
+	err = mlx5e_open_xdpsq(c, params, &cparam.xdp_sq, umem, &c->xsksq, true);
+	if (unlikely(err))
+		goto err_close_tx_cq;
+
+	err = mlx5e_open_cq(c, icocq_moder, &cparam.icosq_cq, &c->xskicosq.cq);
+	if (unlikely(err))
+		goto err_close_sq;
+
+	/* Create a dedicated SQ for posting NOPs whenever we need an IRQ to be
+	 * triggered and NAPI to be called on the correct CPU.
+	 */
+	err = mlx5e_open_icosq(c, params, &cparam.icosq, &c->xskicosq);
+	if (unlikely(err))
+		goto err_close_icocq;
+
+	spin_lock_init(&c->xskicosq_lock);
+
+	set_bit(MLX5E_CHANNEL_STATE_XSK, c->state);
+
+	return 0;
+
+err_close_icocq:
+	mlx5e_close_cq(&c->xskicosq.cq);
+
+err_close_sq:
+	mlx5e_close_xdpsq(&c->xsksq);
+
+err_close_tx_cq:
+	mlx5e_close_cq(&c->xsksq.cq);
+
+err_close_rx_cq:
+	mlx5e_close_cq(&c->xskrq.cq);
+
+	return err;
+}
+
+void mlx5e_close_xsk(struct mlx5e_channel *c)
+{
+	clear_bit(MLX5E_CHANNEL_STATE_XSK, c->state);
+	napi_synchronize(&c->napi);
+
+	mlx5e_close_rq(&c->xskrq);
+	mlx5e_close_cq(&c->xskrq.cq);
+	mlx5e_close_icosq(&c->xskicosq);
+	mlx5e_close_cq(&c->xskicosq.cq);
+	mlx5e_close_xdpsq(&c->xsksq);
+	mlx5e_close_cq(&c->xsksq.cq);
+}
+
+void mlx5e_activate_xsk(struct mlx5e_channel *c)
+{
+	set_bit(MLX5E_RQ_STATE_ENABLED, &c->xskrq.state);
+	/* TX queue is created active. */
+	mlx5e_trigger_irq(&c->xskicosq);
+}
+
+void mlx5e_deactivate_xsk(struct mlx5e_channel *c)
+{
+	mlx5e_deactivate_rq(&c->xskrq);
+	/* TX queue is disabled on close. */
+}
+
+static int mlx5e_redirect_xsk_rqt(struct mlx5e_priv *priv, u16 ix, u32 rqn)
+{
+	struct mlx5e_redirect_rqt_param direct_rrp = {
+		.is_rss = false,
+		{
+			.rqn = rqn,
+		},
+	};
+
+	u32 rqtn = priv->xsk_tir[ix].rqt.rqtn;
+
+	return mlx5e_redirect_rqt(priv, rqtn, 1, direct_rrp);
+}
+
+int mlx5e_xsk_redirect_rqt_to_channel(struct mlx5e_priv *priv, struct mlx5e_channel *c)
+{
+	return mlx5e_redirect_xsk_rqt(priv, c->ix, c->xskrq.rqn);
+}
+
+int mlx5e_xsk_redirect_rqt_to_drop(struct mlx5e_priv *priv, u16 ix)
+{
+	return mlx5e_redirect_xsk_rqt(priv, ix, priv->drop_rq.rqn);
+}
+
+int mlx5e_xsk_redirect_rqts_to_channels(struct mlx5e_priv *priv, struct mlx5e_channels *chs)
+{
+	int err, i;
+
+	if (!priv->xsk.refcnt)
+		return 0;
+
+	for (i = 0; i < chs->num; i++) {
+		struct mlx5e_channel *c = chs->c[i];
+
+		if (!test_bit(MLX5E_CHANNEL_STATE_XSK, c->state))
+			continue;
+
+		err = mlx5e_xsk_redirect_rqt_to_channel(priv, c);
+		if (unlikely(err))
+			goto err_stop;
+	}
+
+	return 0;
+
+err_stop:
+	for (i--; i >= 0; i--) {
+		if (!test_bit(MLX5E_CHANNEL_STATE_XSK, chs->c[i]->state))
+			continue;
+
+		mlx5e_xsk_redirect_rqt_to_drop(priv, i);
+	}
+
+	return err;
+}
+
+void mlx5e_xsk_redirect_rqts_to_drop(struct mlx5e_priv *priv, struct mlx5e_channels *chs)
+{
+	int i;
+
+	if (!priv->xsk.refcnt)
+		return;
+
+	for (i = 0; i < chs->num; i++) {
+		if (!test_bit(MLX5E_CHANNEL_STATE_XSK, chs->c[i]->state))
+			continue;
+
+		mlx5e_xsk_redirect_rqt_to_drop(priv, i);
+	}
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.h
new file mode 100644
index 000000000000..0dd11b81c046
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2019 Mellanox Technologies. */
+
+#ifndef __MLX5_EN_XSK_SETUP_H__
+#define __MLX5_EN_XSK_SETUP_H__
+
+#include "en.h"
+
+struct mlx5e_xsk_param;
+
+bool mlx5e_validate_xsk_param(struct mlx5e_params *params,
+			      struct mlx5e_xsk_param *xsk,
+			      struct mlx5_core_dev *mdev);
+int mlx5e_open_xsk(struct mlx5e_priv *priv, struct mlx5e_params *params,
+		   struct mlx5e_xsk_param *xsk, struct xdp_umem *umem,
+		   struct mlx5e_channel *c);
+void mlx5e_close_xsk(struct mlx5e_channel *c);
+void mlx5e_activate_xsk(struct mlx5e_channel *c);
+void mlx5e_deactivate_xsk(struct mlx5e_channel *c);
+int mlx5e_xsk_redirect_rqt_to_channel(struct mlx5e_priv *priv, struct mlx5e_channel *c);
+int mlx5e_xsk_redirect_rqt_to_drop(struct mlx5e_priv *priv, u16 ix);
+int mlx5e_xsk_redirect_rqts_to_channels(struct mlx5e_priv *priv, struct mlx5e_channels *chs);
+void mlx5e_xsk_redirect_rqts_to_drop(struct mlx5e_priv *priv, struct mlx5e_channels *chs);
+
+#endif /* __MLX5_EN_XSK_SETUP_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
new file mode 100644
index 000000000000..9a72b59839ba
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
@@ -0,0 +1,108 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2019 Mellanox Technologies. */
+
+#include "tx.h"
+#include "umem.h"
+#include "en/xdp.h"
+#include <net/xdp_sock.h>
+
+int mlx5e_xsk_async_xmit(struct net_device *dev, u32 qid)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+	struct mlx5e_channel *c;
+
+	if (unlikely(!mlx5e_xdp_is_open(priv)))
+		return -ENETDOWN;
+
+	if (unlikely(!mlx5e_xsk_is_valid_qid(priv, qid)))
+		return -EINVAL;
+
+	c = priv->channels.c[qid];
+
+	if (unlikely(!test_bit(MLX5E_CHANNEL_STATE_XSK, c->state)))
+		return -ENXIO;
+
+	if (!napi_if_scheduled_mark_missed(&c->napi)) {
+		spin_lock(&c->xskicosq_lock);
+		mlx5e_trigger_irq(&c->xskicosq);
+		spin_unlock(&c->xskicosq_lock);
+	}
+
+	return 0;
+}
+
+/* When TX fails (because of the size of the packet), we need to get completions
+ * in order, so post a NOP to get a CQE. Since AF_XDP doesn't distinguish
+ * between successful TX and errors, handling in mlx5e_poll_xdpsq_cq is the
+ * same.
+ */
+static void mlx5e_xsk_tx_post_err(struct mlx5e_xdpsq *sq,
+				  struct mlx5e_xdp_info *xdpi)
+{
+	u16 pi = mlx5_wq_cyc_ctr2ix(&sq->wq, sq->pc);
+	struct mlx5e_xdp_wqe_info *wi = &sq->db.wqe_info[pi];
+	struct mlx5e_tx_wqe *nopwqe;
+
+	wi->num_wqebbs = 1;
+	wi->num_pkts = 1;
+
+	nopwqe = mlx5e_post_nop(&sq->wq, sq->sqn, &sq->pc);
+	mlx5e_xdpi_fifo_push(&sq->db.xdpi_fifo, xdpi);
+	sq->doorbell_cseg = &nopwqe->ctrl;
+}
+
+bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget)
+{
+	struct xdp_umem *umem = sq->umem;
+	struct mlx5e_xdp_info xdpi;
+	struct mlx5e_xdp_xmit_data xdptxd;
+	bool work_done = true;
+	bool flush = false;
+
+	xdpi.mode = MLX5E_XDP_XMIT_MODE_XSK;
+
+	for (; budget; budget--) {
+		int check_result = sq->xmit_xdp_frame_check(sq);
+		struct xdp_desc desc;
+
+		if (unlikely(check_result < 0)) {
+			work_done = false;
+			break;
+		}
+
+		if (!xsk_umem_consume_tx(umem, &desc)) {
+			/* TX will get stuck until something wakes it up by
+			 * triggering NAPI. Currently it's expected that the
+			 * application calls sendto() if there are consumed, but
+			 * not completed frames.
+			 */
+			break;
+		}
+
+		xdptxd.dma_addr = xdp_umem_get_dma(umem, desc.addr);
+		xdptxd.data = xdp_umem_get_data(umem, desc.addr);
+		xdptxd.len = desc.len;
+
+		dma_sync_single_for_device(sq->pdev, xdptxd.dma_addr,
+					   xdptxd.len, DMA_BIDIRECTIONAL);
+
+		if (unlikely(!sq->xmit_xdp_frame(sq, &xdptxd, &xdpi, check_result))) {
+			if (sq->mpwqe.wqe)
+				mlx5e_xdp_mpwqe_complete(sq);
+
+			mlx5e_xsk_tx_post_err(sq, &xdpi);
+		}
+
+		flush = true;
+	}
+
+	if (flush) {
+		if (sq->mpwqe.wqe)
+			mlx5e_xdp_mpwqe_complete(sq);
+		mlx5e_xmit_xdp_doorbell(sq);
+
+		xsk_umem_consume_tx_done(umem);
+	}
+
+	return !(budget && work_done);
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.h
new file mode 100644
index 000000000000..7add18bf78d8
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2019 Mellanox Technologies. */
+
+#ifndef __MLX5_EN_XSK_TX_H__
+#define __MLX5_EN_XSK_TX_H__
+
+#include "en.h"
+
+/* TX data path */
+
+int mlx5e_xsk_async_xmit(struct net_device *dev, u32 qid);
+
+bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget);
+
+#endif /* __MLX5_EN_XSK_TX_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c
new file mode 100644
index 000000000000..c9917c57f054
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c
@@ -0,0 +1,252 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
+/* Copyright (c) 2019 Mellanox Technologies. */
+
+#include <net/xdp_sock.h>
+#include "umem.h"
+#include "setup.h"
+#include "en/params.h"
+
+static int mlx5e_xsk_map_umem(struct mlx5e_priv *priv,
+			      struct xdp_umem *umem)
+{
+	struct device *dev = priv->mdev->device;
+	u32 i;
+
+	for (i = 0; i < umem->npgs; i++) {
+		dma_addr_t dma = dma_map_page(dev, umem->pgs[i], 0, PAGE_SIZE,
+					      DMA_BIDIRECTIONAL);
+
+		if (unlikely(dma_mapping_error(dev, dma)))
+			goto err_unmap;
+		umem->pages[i].dma = dma;
+	}
+
+	return 0;
+
+err_unmap:
+	while (i--) {
+		dma_unmap_page(dev, umem->pages[i].dma, PAGE_SIZE,
+			       DMA_BIDIRECTIONAL);
+		umem->pages[i].dma = 0;
+	}
+
+	return -ENOMEM;
+}
+
+static void mlx5e_xsk_unmap_umem(struct mlx5e_priv *priv,
+				 struct xdp_umem *umem)
+{
+	struct device *dev = priv->mdev->device;
+	u32 i;
+
+	for (i = 0; i < umem->npgs; i++) {
+		dma_unmap_page(dev, umem->pages[i].dma, PAGE_SIZE,
+			       DMA_BIDIRECTIONAL);
+		umem->pages[i].dma = 0;
+	}
+}
+
+static int mlx5e_xsk_get_umems(struct mlx5e_xsk *xsk)
+{
+	if (!xsk->umems) {
+		xsk->umems = kcalloc(MLX5E_MAX_NUM_CHANNELS,
+				     sizeof(*xsk->umems), GFP_KERNEL);
+		if (unlikely(!xsk->umems))
+			return -ENOMEM;
+	}
+
+	xsk->refcnt++;
+	xsk->ever_used = true;
+
+	return 0;
+}
+
+static void mlx5e_xsk_put_umems(struct mlx5e_xsk *xsk)
+{
+	if (!--xsk->refcnt) {
+		kfree(xsk->umems);
+		xsk->umems = NULL;
+	}
+}
+
+static int mlx5e_xsk_add_umem(struct mlx5e_xsk *xsk, struct xdp_umem *umem, u16 ix)
+{
+	int err;
+
+	err = mlx5e_xsk_get_umems(xsk);
+	if (unlikely(err))
+		return err;
+
+	xsk->umems[ix] = umem;
+	return 0;
+}
+
+static void mlx5e_xsk_remove_umem(struct mlx5e_xsk *xsk, u16 ix)
+{
+	xsk->umems[ix] = NULL;
+
+	mlx5e_xsk_put_umems(xsk);
+}
+
+static bool mlx5e_xsk_is_umem_sane(struct xdp_umem *umem)
+{
+	return umem->headroom <= 0xffff && umem->chunk_size_nohr <= 0xffff;
+}
+
+void mlx5e_build_xsk_param(struct xdp_umem *umem, struct mlx5e_xsk_param *xsk)
+{
+	xsk->headroom = umem->headroom;
+	xsk->chunk_size = umem->chunk_size_nohr + umem->headroom;
+}
+
+static int mlx5e_xsk_enable_locked(struct mlx5e_priv *priv,
+				   struct xdp_umem *umem, u16 ix)
+{
+	struct mlx5e_params *params = &priv->channels.params;
+	struct mlx5e_xsk_param xsk;
+	struct mlx5e_channel *c;
+	int err;
+
+	if (unlikely(mlx5e_xsk_get_umem(&priv->channels.params, &priv->xsk, ix)))
+		return -EBUSY;
+
+	if (unlikely(!mlx5e_xsk_is_umem_sane(umem)))
+		return -EINVAL;
+
+	err = mlx5e_xsk_map_umem(priv, umem);
+	if (unlikely(err))
+		return err;
+
+	err = mlx5e_xsk_add_umem(&priv->xsk, umem, ix);
+	if (unlikely(err))
+		goto err_unmap_umem;
+
+	mlx5e_build_xsk_param(umem, &xsk);
+
+	if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+		/* XSK objects will be created on open. */
+		goto validate_closed;
+	}
+
+	if (!params->xdp_prog) {
+		/* XSK objects will be created when an XDP program is set,
+		 * and the channels are reopened.
+		 */
+		goto validate_closed;
+	}
+
+	c = priv->channels.c[ix];
+
+	err = mlx5e_open_xsk(priv, params, &xsk, umem, c);
+	if (unlikely(err))
+		goto err_remove_umem;
+
+	mlx5e_activate_xsk(c);
+
+	/* Don't wait for WQEs, because the newer xdpsock sample doesn't provide
+	 * any Fill Ring entries at the setup stage.
+	 */
+
+	err = mlx5e_xsk_redirect_rqt_to_channel(priv, priv->channels.c[ix]);
+	if (unlikely(err))
+		goto err_deactivate;
+
+	return 0;
+
+err_deactivate:
+	mlx5e_deactivate_xsk(c);
+	mlx5e_close_xsk(c);
+
+err_remove_umem:
+	mlx5e_xsk_remove_umem(&priv->xsk, ix);
+
+err_unmap_umem:
+	mlx5e_xsk_unmap_umem(priv, umem);
+
+	return err;
+
+validate_closed:
+	/* Check the configuration in advance, rather than fail at a later stage
+	 * (in mlx5e_xdp_set or on open) and end up with no channels.
+	 */
+	if (!mlx5e_validate_xsk_param(params, &xsk, priv->mdev)) {
+		err = -EINVAL;
+		goto err_remove_umem;
+	}
+
+	return 0;
+}
+
+static int mlx5e_xsk_disable_locked(struct mlx5e_priv *priv, u16 ix)
+{
+	struct xdp_umem *umem = mlx5e_xsk_get_umem(&priv->channels.params,
+						   &priv->xsk, ix);
+	struct mlx5e_channel *c;
+
+	if (unlikely(!umem))
+		return -EINVAL;
+
+	if (!test_bit(MLX5E_STATE_OPENED, &priv->state))
+		goto remove_umem;
+
+	/* XSK RQ and SQ are only created if XDP program is set. */
+	if (!priv->channels.params.xdp_prog)
+		goto remove_umem;
+
+	c = priv->channels.c[ix];
+	mlx5e_xsk_redirect_rqt_to_drop(priv, ix);
+	mlx5e_deactivate_xsk(c);
+	mlx5e_close_xsk(c);
+
+remove_umem:
+	mlx5e_xsk_remove_umem(&priv->xsk, ix);
+	mlx5e_xsk_unmap_umem(priv, umem);
+
+	return 0;
+}
+
+static int mlx5e_xsk_enable_umem(struct mlx5e_priv *priv, struct xdp_umem *umem,
+				 u16 ix)
+{
+	int err;
+
+	mutex_lock(&priv->state_lock);
+	err = mlx5e_xsk_enable_locked(priv, umem, ix);
+	mutex_unlock(&priv->state_lock);
+
+	return err;
+}
+
+static int mlx5e_xsk_disable_umem(struct mlx5e_priv *priv, u16 ix)
+{
+	int err;
+
+	mutex_lock(&priv->state_lock);
+	err = mlx5e_xsk_disable_locked(priv, ix);
+	mutex_unlock(&priv->state_lock);
+
+	return err;
+}
+
+int mlx5e_xsk_setup_umem(struct net_device *dev, struct xdp_umem *umem, u16 qid)
+{
+	struct mlx5e_priv *priv = netdev_priv(dev);
+
+	if (unlikely(!mlx5e_xsk_is_valid_qid(priv, qid)))
+		return -EINVAL;
+
+	return umem ? mlx5e_xsk_enable_umem(priv, umem, qid) :
+		      mlx5e_xsk_disable_umem(priv, qid);
+}
+
+int mlx5e_xsk_resize_reuseq(struct xdp_umem *umem, u32 nentries)
+{
+	struct xdp_umem_fq_reuse *reuseq;
+
+	reuseq = xsk_reuseq_prepare(nentries);
+	if (unlikely(!reuseq))
+		return -ENOMEM;
+	xsk_reuseq_free(xsk_reuseq_swap(umem, reuseq));
+
+	return 0;
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h
new file mode 100644
index 000000000000..1053ecdf1d88
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2019 Mellanox Technologies. */
+
+#ifndef __MLX5_EN_XSK_UMEM_H__
+#define __MLX5_EN_XSK_UMEM_H__
+
+#include "en.h"
+
+static inline bool mlx5e_xsk_is_valid_qid(struct mlx5e_priv *priv, u16 qid)
+{
+	return qid < priv->channels.params.num_channels;
+}
+
+static inline struct xdp_umem *mlx5e_xsk_get_umem(struct mlx5e_params *params,
+						  struct mlx5e_xsk *xsk, u16 ix)
+{
+	if (!xsk || !xsk->umems)
+		return NULL;
+
+	if (unlikely(ix >= params->num_channels))
+		return NULL;
+
+	return xsk->umems[ix];
+}
+
+struct mlx5e_xsk_param;
+void mlx5e_build_xsk_param(struct xdp_umem *umem, struct mlx5e_xsk_param *xsk);
+
+/* .ndo_bpf callback. */
+int mlx5e_xsk_setup_umem(struct net_device *dev, struct xdp_umem *umem, u16 qid);
+
+int mlx5e_xsk_resize_reuseq(struct xdp_umem *umem, u32 nentries);
+
+#endif /* __MLX5_EN_XSK_UMEM_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
index 76a3d01a489e..010279a3e861 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
@@ -32,6 +32,7 @@
 
 #include "en.h"
 #include "en/port.h"
+#include "en/xsk/umem.h"
 #include "lib/clock.h"
 
 void mlx5e_ethtool_get_drvinfo(struct mlx5e_priv *priv,
@@ -403,6 +404,7 @@ static void mlx5e_get_channels(struct net_device *dev,
 int mlx5e_ethtool_set_channels(struct mlx5e_priv *priv,
 			       struct ethtool_channels *ch)
 {
+	struct mlx5e_params *cur_params = &priv->channels.params;
 	unsigned int count = ch->combined_count;
 	struct mlx5e_channels new_channels = {};
 	bool arfs_enabled;
@@ -414,16 +416,31 @@ int mlx5e_ethtool_set_channels(struct mlx5e_priv *priv,
 		return -EINVAL;
 	}
 
-	if (priv->channels.params.num_channels == count)
+	if (cur_params->num_channels == count)
 		return 0;
 
 	mutex_lock(&priv->state_lock);
 
+	/* Don't allow closing the channels if they have an active XSK. */
+	if (priv->xsk.refcnt) {
+		u16 ix;
+
+		/* Check the channels that are going to be closed. */
+		for (ix = cur_params->num_channels - 1; ix >= count; ix--) {
+			if (mlx5e_xsk_get_umem(cur_params, &priv->xsk, ix)) {
+				err = -EINVAL;
+				netdev_err(priv->netdev, "%s: AF_XDP is active on channel %hu, cannot reduce the number of channels\n",
+					   __func__, ix);
+				goto out;
+			}
+		}
+	}
+
 	new_channels.params = priv->channels.params;
 	new_channels.params.num_channels = count;
 
 	if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
-		priv->channels.params = new_channels.params;
+		*cur_params = new_channels.params;
 		if (!netif_is_rxfh_configured(priv->netdev))
 			mlx5e_build_default_indir_rqt(priv->rss_params.indirection_rqt,
 						      MLX5E_INDIR_RQT_SIZE, count);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c
index 4421c10f58ae..9241e640c8e4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_fs_ethtool.c
@@ -392,6 +392,20 @@ static bool outer_header_zero(u32 *match_criteria)
 						  size - 1);
 }
 
+#define MLX5E_USERDEF_FLAG_XSK BIT_ULL(0)
+
+static bool is_rule_xsk(struct ethtool_rx_flow_spec *fs)
+{
+	u64 userdef;
+
+	if (!(fs->flow_type & FLOW_EXT))
+		return false;
+
+	userdef = be64_to_cpu(*((__be64 *)fs->h_ext.data));
+
+	return userdef & MLX5E_USERDEF_FLAG_XSK;
+}
+
 static struct mlx5_flow_handle *
 add_ethtool_flow_rule(struct mlx5e_priv *priv,
 		      struct mlx5_flow_table *ft,
@@ -414,6 +428,9 @@ add_ethtool_flow_rule(struct mlx5e_priv *priv,
 	if (fs->ring_cookie == RX_CLS_FLOW_DISC) {
 		flow_act.action = MLX5_FLOW_CONTEXT_ACTION_DROP;
 	} else {
+		struct mlx5e_tir *tir = is_rule_xsk(fs) ? priv->xsk_tir
+							: priv->direct_tir;
+
 		dst = kzalloc(sizeof(*dst), GFP_KERNEL);
 		if (!dst) {
 			err = -ENOMEM;
@@ -421,7 +438,7 @@ add_ethtool_flow_rule(struct mlx5e_priv *priv,
 		}
 
 		dst->type = MLX5_FLOW_DESTINATION_TYPE_TIR;
-		dst->tir_num = priv->direct_tir[fs->ring_cookie].tirn;
+		dst->tir_num = tir[fs->ring_cookie].tirn;
 		flow_act.action = MLX5_FLOW_CONTEXT_ACTION_FWD_DEST;
 	}
 
@@ -578,14 +595,27 @@ static int validate_tcpudp6(struct ethtool_rx_flow_spec *fs)
 	return ++ntuples;
 }
 
-static int validate_vlan(struct ethtool_rx_flow_spec *fs)
+static int validate_ext(struct ethtool_rx_flow_spec *fs)
 {
-	if (fs->m_ext.vlan_etype ||
-	    fs->m_ext.vlan_tci != cpu_to_be16(VLAN_VID_MASK))
+	u64 userdef_mask;
+	u64 userdef;
+
+	if (fs->m_ext.vlan_etype)
 		return -EINVAL;
 
-	if (fs->m_ext.vlan_tci &&
-	    (be16_to_cpu(fs->h_ext.vlan_tci) >= VLAN_N_VID))
+	if (fs->m_ext.vlan_tci) {
+		if (fs->m_ext.vlan_tci != cpu_to_be16(VLAN_VID_MASK))
+			return -EINVAL;
+
+		if (be16_to_cpu(fs->h_ext.vlan_tci) >= VLAN_N_VID)
+			return -EINVAL;
+	}
+
+	userdef = be64_to_cpu(*((__be64 *)fs->h_ext.data));
+	userdef_mask = be64_to_cpu(*((__be64 *)fs->m_ext.data));
+	if (userdef & ~MLX5E_USERDEF_FLAG_XSK)
+		return -EINVAL;
+	if (userdef_mask != ~(0ULL))
 		return -EINVAL;
 
 	return 1;
@@ -638,7 +668,7 @@ static int validate_flow(struct mlx5e_priv *priv,
 		return -ENOTSUPP;
 	}
 	if ((fs->flow_type & FLOW_EXT)) {
-		ret = validate_vlan(fs);
+		ret = validate_ext(fs);
 		if (ret < 0)
 			return ret;
 		num_tuples += ret;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 3b2d7a1fa7df..1f7f50ca61d2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -38,6 +38,7 @@
 #include <linux/bpf.h>
 #include <linux/if_bridge.h>
 #include <net/page_pool.h>
+#include <net/xdp_sock.h>
 #include "eswitch.h"
 #include "en.h"
 #include "en_tc.h"
@@ -56,6 +57,10 @@
 #include "en/monitor_stats.h"
 #include "en/reporter.h"
 #include "en/params.h"
+#include "en/xsk/umem.h"
+#include "en/xsk/setup.h"
+#include "en/xsk/rx.h"
+#include "en/xsk/tx.h"
 
 bool mlx5e_check_fragmented_striding_rq_cap(struct mlx5_core_dev *mdev)
 {
@@ -85,18 +90,31 @@ void mlx5e_init_rq_type_params(struct mlx5_core_dev *mdev,
 	mlx5_core_info(mdev, "MLX5E: StrdRq(%d) RqSz(%ld) StrdSz(%ld) RxCqeCmprss(%d)\n",
 		       params->rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ,
 		       params->rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ ?
-		       BIT(mlx5e_mpwqe_get_log_rq_size(params)) :
+		       BIT(mlx5e_mpwqe_get_log_rq_size(params, NULL)) :
 		       BIT(params->log_rq_mtu_frames),
-		       BIT(mlx5e_mpwqe_get_log_stride_size(mdev, params)),
+		       BIT(mlx5e_mpwqe_get_log_stride_size(mdev, params, NULL)),
 		       MLX5E_GET_PFLAG(params, MLX5E_PFLAG_RX_CQE_COMPRESS));
 }
 
 bool mlx5e_striding_rq_possible(struct mlx5_core_dev *mdev,
 				struct mlx5e_params *params)
 {
-	return mlx5e_check_fragmented_striding_rq_cap(mdev) &&
-		!MLX5_IPSEC_DEV(mdev) &&
-		!(params->xdp_prog && !mlx5e_rx_mpwqe_is_linear_skb(mdev, params));
+	if (!mlx5e_check_fragmented_striding_rq_cap(mdev))
+		return false;
+
+	if (MLX5_IPSEC_DEV(mdev))
+		return false;
+
+	if (params->xdp_prog) {
+		/* XSK params are not considered here. If striding RQ is in use,
+		 * and an XSK is being opened, mlx5e_rx_mpwqe_is_linear_skb will
+		 * be called with the known XSK params.
+		 */
+		if (!mlx5e_rx_mpwqe_is_linear_skb(mdev, params, NULL))
+			return false;
+	}
+
+	return true;
 }
 
 void mlx5e_set_rq_type(struct mlx5_core_dev *mdev, struct mlx5e_params *params)
@@ -365,6 +383,8 @@ static void mlx5e_free_di_list(struct mlx5e_rq *rq)
 
 static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 			  struct mlx5e_params *params,
+			  struct mlx5e_xsk_param *xsk,
+			  struct xdp_umem *umem,
 			  struct mlx5e_rq_param *rqp,
 			  struct mlx5e_rq *rq)
 {
@@ -372,6 +392,8 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 	struct mlx5_core_dev *mdev = c->mdev;
 	void *rqc = rqp->rqc;
 	void *rqc_wq = MLX5_ADDR_OF(rqc, rqc, wq);
+	u32 num_xsk_frames = 0;
+	u32 rq_xdp_ix;
 	u32 pool_size;
 	int wq_sz;
 	int err;
@@ -388,8 +410,13 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 	rq->ix      = c->ix;
 	rq->mdev    = mdev;
 	rq->hw_mtu  = MLX5E_SW2HW_MTU(params, params->sw_mtu);
-	rq->stats   = &c->priv->channel_stats[c->ix].rq;
 	rq->xdpsq   = &c->rq_xdpsq;
+	rq->umem    = umem;
+
+	if (rq->umem)
+		rq->stats = &c->priv->channel_stats[c->ix].xskrq;
+	else
+		rq->stats = &c->priv->channel_stats[c->ix].rq;
 
 	rq->xdp_prog = params->xdp_prog ? bpf_prog_inc(params->xdp_prog) : NULL;
 	if (IS_ERR(rq->xdp_prog)) {
@@ -398,12 +425,16 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 		goto err_rq_wq_destroy;
 	}
 
-	err = xdp_rxq_info_reg(&rq->xdp_rxq, rq->netdev, rq->ix);
+	rq_xdp_ix = rq->ix;
+	if (xsk)
+		rq_xdp_ix |= XDP_QID_FLAG_XSKRQ;
+	err = xdp_rxq_info_reg(&rq->xdp_rxq, rq->netdev, rq_xdp_ix);
 	if (err < 0)
 		goto err_rq_wq_destroy;
 
 	rq->buff.map_dir = rq->xdp_prog ? DMA_BIDIRECTIONAL : DMA_FROM_DEVICE;
-	rq->buff.headroom = mlx5e_get_rq_headroom(mdev, params);
+	rq->buff.headroom = mlx5e_get_rq_headroom(mdev, params, xsk);
+	rq->buff.umem_headroom = xsk ? xsk->headroom : 0;
 	pool_size = 1 << params->log_rq_mtu_frames;
 
 	switch (rq->wq_type) {
@@ -417,7 +448,12 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 
 		wq_sz = mlx5_wq_ll_get_size(&rq->mpwqe.wq);
 
-		pool_size = MLX5_MPWRQ_PAGES_PER_WQE << mlx5e_mpwqe_get_log_rq_size(params);
+		if (xsk)
+			num_xsk_frames = wq_sz <<
+				mlx5e_mpwqe_get_log_num_strides(mdev, params, xsk);
+
+		pool_size = MLX5_MPWRQ_PAGES_PER_WQE <<
+			mlx5e_mpwqe_get_log_rq_size(params, xsk);
 
 		rq->post_wqes = mlx5e_post_rx_mpwqes;
 		rq->dealloc_wqe = mlx5e_dealloc_rx_mpwqe;
@@ -436,12 +472,15 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 			goto err_rq_wq_destroy;
 		}
 
-		rq->mpwqe.skb_from_cqe_mpwrq =
-			mlx5e_rx_mpwqe_is_linear_skb(mdev, params) ?
-			mlx5e_skb_from_cqe_mpwrq_linear :
-			mlx5e_skb_from_cqe_mpwrq_nonlinear;
-		rq->mpwqe.log_stride_sz = mlx5e_mpwqe_get_log_stride_size(mdev, params);
-		rq->mpwqe.num_strides = BIT(mlx5e_mpwqe_get_log_num_strides(mdev, params));
+		rq->mpwqe.skb_from_cqe_mpwrq = xsk ?
+			mlx5e_xsk_skb_from_cqe_mpwrq_linear :
+			mlx5e_rx_mpwqe_is_linear_skb(mdev, params, NULL) ?
+				mlx5e_skb_from_cqe_mpwrq_linear :
+				mlx5e_skb_from_cqe_mpwrq_nonlinear;
+
+		rq->mpwqe.log_stride_sz = mlx5e_mpwqe_get_log_stride_size(mdev, params, xsk);
+		rq->mpwqe.num_strides =
+			BIT(mlx5e_mpwqe_get_log_num_strides(mdev, params, xsk));
 
 		err = mlx5e_create_rq_umr_mkey(mdev, rq);
 		if (err)
@@ -462,6 +501,9 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 
 		wq_sz = mlx5_wq_cyc_get_size(&rq->wqe.wq);
 
+		if (xsk)
+			num_xsk_frames = wq_sz << rq->wqe.info.log_num_frags;
+
 		rq->wqe.info = rqp->frags_info;
 		rq->wqe.frags =
 			kvzalloc_node(array_size(sizeof(*rq->wqe.frags),
@@ -475,6 +517,7 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 		err = mlx5e_init_di_list(rq, wq_sz, c->cpu);
 		if (err)
 			goto err_free;
+
 		rq->post_wqes = mlx5e_post_rx_wqes;
 		rq->dealloc_wqe = mlx5e_dealloc_rx_wqe;
 
@@ -490,33 +533,51 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
 			goto err_free;
 		}
 
-		rq->wqe.skb_from_cqe = mlx5e_rx_is_linear_skb(params) ?
-			mlx5e_skb_from_cqe_linear :
-			mlx5e_skb_from_cqe_nonlinear;
+		rq->wqe.skb_from_cqe = xsk ?
+			mlx5e_xsk_skb_from_cqe_linear :
+			mlx5e_rx_is_linear_skb(params, NULL) ?
+				mlx5e_skb_from_cqe_linear :
+				mlx5e_skb_from_cqe_nonlinear;
+
 		rq->mkey_be = c->mkey_be;
 	}
 
-	/* Create a page_pool and register it with rxq */
-	pp_params.order     = 0;
-	pp_params.flags     = 0; /* No-internal DMA mapping in page_pool */
-	pp_params.pool_size = pool_size;
-	pp_params.nid       = cpu_to_node(c->cpu);
-	pp_params.dev       = c->pdev;
-	pp_params.dma_dir   = rq->buff.map_dir;
-
-	/* page_pool can be used even when there is no rq->xdp_prog,
-	 * given page_pool does not handle DMA mapping there is no
-	 * required state to clear. And page_pool gracefully handle
-	 * elevated refcnt.
-	 */
-	rq->page_pool = page_pool_create(&pp_params);
-	if (IS_ERR(rq->page_pool)) {
-		err = PTR_ERR(rq->page_pool);
-		rq->page_pool = NULL;
-		goto err_free;
+	if (xsk) {
+		err = mlx5e_xsk_resize_reuseq(umem, num_xsk_frames);
+		if (unlikely(err)) {
+			mlx5_core_err(mdev, "Unable to allocate the Reuse Ring for %u frames\n",
+				      num_xsk_frames);
+			goto err_free;
+		}
+
+		rq->zca.free = mlx5e_xsk_zca_free;
+		err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
+						 MEM_TYPE_ZERO_COPY,
+						 &rq->zca);
+	} else {
+		/* Create a page_pool and register it with rxq */
+		pp_params.order     = 0;
+		pp_params.flags     = 0; /* No-internal DMA mapping in page_pool */
+		pp_params.pool_size = pool_size;
+		pp_params.nid       = cpu_to_node(c->cpu);
+		pp_params.dev       = c->pdev;
+		pp_params.dma_dir   = rq->buff.map_dir;
+
+		/* page_pool can be used even when there is no rq->xdp_prog,
+		 * given page_pool does not handle DMA mapping there is no
+		 * required state to clear. And page_pool gracefully handle
+		 * elevated refcnt.
+		 */
+		rq->page_pool = page_pool_create(&pp_params);
+		if (IS_ERR(rq->page_pool)) {
+			err = PTR_ERR(rq->page_pool);
+			rq->page_pool = NULL;
+			goto err_free;
+		}
+		err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
+						 MEM_TYPE_PAGE_POOL,
+						 rq->page_pool);
 	}
-	err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
-					 MEM_TYPE_PAGE_POOL, rq->page_pool);
 	if (err)
 		goto err_free;
 
@@ -615,7 +676,11 @@ static void mlx5e_free_rq(struct mlx5e_rq *rq)
 	     i = (i + 1) & (MLX5E_CACHE_SIZE - 1)) {
 		struct mlx5e_dma_info *dma_info = &rq->page_cache.page_cache[i];
 
-		mlx5e_page_release(rq, dma_info, false);
+		/* With AF_XDP, page_cache is not used, so this loop is not
+		 * entered, and it's safe to call mlx5e_page_release_dynamic
+		 * directly.
+		 */
+		mlx5e_page_release_dynamic(rq, dma_info, false);
 	}
 	mlx5_wq_destroy(&rq->wq_ctrl);
 }
@@ -750,7 +815,7 @@ static void mlx5e_destroy_rq(struct mlx5e_rq *rq)
 	mlx5_core_destroy_rq(rq->mdev, rq->rqn);
 }
 
-static int mlx5e_wait_for_min_rx_wqes(struct mlx5e_rq *rq, int wait_time)
+int mlx5e_wait_for_min_rx_wqes(struct mlx5e_rq *rq, int wait_time)
 {
 	unsigned long exp_time = jiffies + msecs_to_jiffies(wait_time);
 	struct mlx5e_channel *c = rq->channel;
@@ -808,14 +873,13 @@ static void mlx5e_free_rx_descs(struct mlx5e_rq *rq)
 
 }
 
-static int mlx5e_open_rq(struct mlx5e_channel *c,
-			 struct mlx5e_params *params,
-			 struct mlx5e_rq_param *param,
-			 struct mlx5e_rq *rq)
+int mlx5e_open_rq(struct mlx5e_channel *c, struct mlx5e_params *params,
+		  struct mlx5e_rq_param *param, struct mlx5e_xsk_param *xsk,
+		  struct xdp_umem *umem, struct mlx5e_rq *rq)
 {
 	int err;
 
-	err = mlx5e_alloc_rq(c, params, param, rq);
+	err = mlx5e_alloc_rq(c, params, xsk, umem, param, rq);
 	if (err)
 		return err;
 
@@ -853,13 +917,13 @@ static void mlx5e_activate_rq(struct mlx5e_rq *rq)
 	mlx5e_trigger_irq(&rq->channel->icosq);
 }
 
-static void mlx5e_deactivate_rq(struct mlx5e_rq *rq)
+void mlx5e_deactivate_rq(struct mlx5e_rq *rq)
 {
 	clear_bit(MLX5E_RQ_STATE_ENABLED, &rq->state);
 	napi_synchronize(&rq->channel->napi); /* prevent mlx5e_post_rx_wqes */
 }
 
-static void mlx5e_close_rq(struct mlx5e_rq *rq)
+void mlx5e_close_rq(struct mlx5e_rq *rq)
 {
 	cancel_work_sync(&rq->dim.work);
 	mlx5e_destroy_rq(rq);
@@ -912,6 +976,7 @@ static int mlx5e_alloc_xdpsq_db(struct mlx5e_xdpsq *sq, int numa)
 
 static int mlx5e_alloc_xdpsq(struct mlx5e_channel *c,
 			     struct mlx5e_params *params,
+			     struct xdp_umem *umem,
 			     struct mlx5e_sq_param *param,
 			     struct mlx5e_xdpsq *sq,
 			     bool is_redirect)
@@ -927,9 +992,13 @@ static int mlx5e_alloc_xdpsq(struct mlx5e_channel *c,
 	sq->uar_map   = mdev->mlx5e_res.bfreg.map;
 	sq->min_inline_mode = params->tx_min_inline_mode;
 	sq->hw_mtu    = MLX5E_SW2HW_MTU(params, params->sw_mtu);
-	sq->stats     = is_redirect ?
-		&c->priv->channel_stats[c->ix].xdpsq :
-		&c->priv->channel_stats[c->ix].rq_xdpsq;
+	sq->umem      = umem;
+
+	sq->stats = sq->umem ?
+		&c->priv->channel_stats[c->ix].xsksq :
+		is_redirect ?
+			&c->priv->channel_stats[c->ix].xdpsq :
+			&c->priv->channel_stats[c->ix].rq_xdpsq;
 
 	param->wq.db_numa_node = cpu_to_node(c->cpu);
 	err = mlx5_wq_cyc_create(mdev, &param->wq, sqc_wq, wq, &sq->wq_ctrl);
@@ -1308,10 +1377,8 @@ static void mlx5e_tx_err_cqe_work(struct work_struct *recover_work)
 	mlx5e_tx_reporter_err_cqe(sq);
 }
 
-static int mlx5e_open_icosq(struct mlx5e_channel *c,
-			    struct mlx5e_params *params,
-			    struct mlx5e_sq_param *param,
-			    struct mlx5e_icosq *sq)
+int mlx5e_open_icosq(struct mlx5e_channel *c, struct mlx5e_params *params,
+		     struct mlx5e_sq_param *param, struct mlx5e_icosq *sq)
 {
 	struct mlx5e_create_sq_param csp = {};
 	int err;
@@ -1337,7 +1404,7 @@ static int mlx5e_open_icosq(struct mlx5e_channel *c,
 	return err;
 }
 
-static void mlx5e_close_icosq(struct mlx5e_icosq *sq)
+void mlx5e_close_icosq(struct mlx5e_icosq *sq)
 {
 	struct mlx5e_channel *c = sq->channel;
 
@@ -1348,16 +1415,14 @@ static void mlx5e_close_icosq(struct mlx5e_icosq *sq)
 	mlx5e_free_icosq(sq);
 }
 
-static int mlx5e_open_xdpsq(struct mlx5e_channel *c,
-			    struct mlx5e_params *params,
-			    struct mlx5e_sq_param *param,
-			    struct mlx5e_xdpsq *sq,
-			    bool is_redirect)
+int mlx5e_open_xdpsq(struct mlx5e_channel *c, struct mlx5e_params *params,
+		     struct mlx5e_sq_param *param, struct xdp_umem *umem,
+		     struct mlx5e_xdpsq *sq, bool is_redirect)
 {
 	struct mlx5e_create_sq_param csp = {};
 	int err;
 
-	err = mlx5e_alloc_xdpsq(c, params, param, sq, is_redirect);
+	err = mlx5e_alloc_xdpsq(c, params, umem, param, sq, is_redirect);
 	if (err)
 		return err;
 
@@ -1411,7 +1476,7 @@ static int mlx5e_open_xdpsq(struct mlx5e_channel *c,
 	return err;
 }
 
-static void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq)
+void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq)
 {
 	struct mlx5e_channel *c = sq->channel;
 
@@ -1540,10 +1605,8 @@ static void mlx5e_destroy_cq(struct mlx5e_cq *cq)
 	mlx5_core_destroy_cq(cq->mdev, &cq->mcq);
 }
 
-static int mlx5e_open_cq(struct mlx5e_channel *c,
-			 struct net_dim_cq_moder moder,
-			 struct mlx5e_cq_param *param,
-			 struct mlx5e_cq *cq)
+int mlx5e_open_cq(struct mlx5e_channel *c, struct net_dim_cq_moder moder,
+		  struct mlx5e_cq_param *param, struct mlx5e_cq *cq)
 {
 	struct mlx5_core_dev *mdev = c->mdev;
 	int err;
@@ -1566,7 +1629,7 @@ static int mlx5e_open_cq(struct mlx5e_channel *c,
 	return err;
 }
 
-static void mlx5e_close_cq(struct mlx5e_cq *cq)
+void mlx5e_close_cq(struct mlx5e_cq *cq)
 {
 	mlx5e_destroy_cq(cq);
 	mlx5e_free_cq(cq);
@@ -1780,17 +1843,17 @@ static int mlx5e_open_queues(struct mlx5e_channel *c,
 		goto err_close_icosq;
 
 	if (c->xdp) {
-		err = mlx5e_open_xdpsq(c, params, &cparam->xdp_sq,
+		err = mlx5e_open_xdpsq(c, params, &cparam->xdp_sq, NULL,
 				       &c->rq_xdpsq, false);
 		if (err)
 			goto err_close_sqs;
 	}
 
-	err = mlx5e_open_rq(c, params, &cparam->rq, &c->rq);
+	err = mlx5e_open_rq(c, params, &cparam->rq, NULL, NULL, &c->rq);
 	if (err)
 		goto err_close_xdp_sq;
 
-	err = mlx5e_open_xdpsq(c, params, &cparam->xdp_sq, &c->xdpsq, true);
+	err = mlx5e_open_xdpsq(c, params, &cparam->xdp_sq, NULL, &c->xdpsq, true);
 	if (err)
 		goto err_close_rq;
 
@@ -1850,10 +1913,12 @@ static void mlx5e_close_queues(struct mlx5e_channel *c)
 static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 			      struct mlx5e_params *params,
 			      struct mlx5e_channel_param *cparam,
+			      struct xdp_umem *umem,
 			      struct mlx5e_channel **cp)
 {
 	int cpu = cpumask_first(mlx5_comp_irq_get_affinity_mask(priv->mdev, ix));
 	struct net_device *netdev = priv->netdev;
+	struct mlx5e_xsk_param xsk;
 	struct mlx5e_channel *c;
 	unsigned int irq;
 	int err;
@@ -1890,10 +1955,20 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
 	if (unlikely(err))
 		goto err_napi_del;
 
+	if (umem) {
+		mlx5e_build_xsk_param(umem, &xsk);
+		err = mlx5e_open_xsk(priv, params, &xsk, umem, c);
+		if (unlikely(err))
+			goto err_close_queues;
+	}
+
 	*cp = c;
 
 	return 0;
 
+err_close_queues:
+	mlx5e_close_queues(c);
+
 err_napi_del:
 	netif_napi_del(&c->napi);
 	mlx5e_free_xps_cpumask(c);
@@ -1912,12 +1987,18 @@ static void mlx5e_activate_channel(struct mlx5e_channel *c)
 		mlx5e_activate_txqsq(&c->sq[tc]);
 	mlx5e_activate_rq(&c->rq);
 	netif_set_xps_queue(c->netdev, c->xps_cpumask, c->ix);
+
+	if (test_bit(MLX5E_CHANNEL_STATE_XSK, c->state))
+		mlx5e_activate_xsk(c);
 }
 
 static void mlx5e_deactivate_channel(struct mlx5e_channel *c)
 {
 	int tc;
 
+	if (test_bit(MLX5E_CHANNEL_STATE_XSK, c->state))
+		mlx5e_deactivate_xsk(c);
+
 	mlx5e_deactivate_rq(&c->rq);
 	for (tc = 0; tc < c->num_tc; tc++)
 		mlx5e_deactivate_txqsq(&c->sq[tc]);
@@ -1925,6 +2006,8 @@ static void mlx5e_deactivate_channel(struct mlx5e_channel *c)
 
 static void mlx5e_close_channel(struct mlx5e_channel *c)
 {
+	if (test_bit(MLX5E_CHANNEL_STATE_XSK, c->state))
+		mlx5e_close_xsk(c);
 	mlx5e_close_queues(c);
 	netif_napi_del(&c->napi);
 	mlx5e_free_xps_cpumask(c);
@@ -1936,6 +2019,7 @@ static void mlx5e_close_channel(struct mlx5e_channel *c)
 
 static void mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev,
 				      struct mlx5e_params *params,
+				      struct mlx5e_xsk_param *xsk,
 				      struct mlx5e_rq_frags_info *info)
 {
 	u32 byte_count = MLX5E_SW2HW_MTU(params, params->sw_mtu);
@@ -1948,10 +2032,10 @@ static void mlx5e_build_rq_frags_info(struct mlx5_core_dev *mdev,
 		byte_count += MLX5E_METADATA_ETHER_LEN;
 #endif
 
-	if (mlx5e_rx_is_linear_skb(params)) {
+	if (mlx5e_rx_is_linear_skb(params, xsk)) {
 		int frag_stride;
 
-		frag_stride = mlx5e_rx_get_linear_frag_sz(params, NULL);
+		frag_stride = mlx5e_rx_get_linear_frag_sz(params, xsk);
 		frag_stride = roundup_pow_of_two(frag_stride);
 
 		info->arr[0].frag_size = byte_count;
@@ -2009,9 +2093,10 @@ static u8 mlx5e_get_rq_log_wq_sz(void *rqc)
 	return MLX5_GET(wq, wq, log_wq_sz);
 }
 
-static void mlx5e_build_rq_param(struct mlx5e_priv *priv,
-				 struct mlx5e_params *params,
-				 struct mlx5e_rq_param *param)
+void mlx5e_build_rq_param(struct mlx5e_priv *priv,
+			  struct mlx5e_params *params,
+			  struct mlx5e_xsk_param *xsk,
+			  struct mlx5e_rq_param *param)
 {
 	struct mlx5_core_dev *mdev = priv->mdev;
 	void *rqc = param->rqc;
@@ -2021,16 +2106,16 @@ static void mlx5e_build_rq_param(struct mlx5e_priv *priv,
 	switch (params->rq_wq_type) {
 	case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
 		MLX5_SET(wq, wq, log_wqe_num_of_strides,
-			 mlx5e_mpwqe_get_log_num_strides(mdev, params) -
+			 mlx5e_mpwqe_get_log_num_strides(mdev, params, xsk) -
 			 MLX5_MPWQE_LOG_NUM_STRIDES_BASE);
 		MLX5_SET(wq, wq, log_wqe_stride_size,
-			 mlx5e_mpwqe_get_log_stride_size(mdev, params) -
+			 mlx5e_mpwqe_get_log_stride_size(mdev, params, xsk) -
 			 MLX5_MPWQE_LOG_STRIDE_SZ_BASE);
-		MLX5_SET(wq, wq, log_wq_sz, mlx5e_mpwqe_get_log_rq_size(params));
+		MLX5_SET(wq, wq, log_wq_sz, mlx5e_mpwqe_get_log_rq_size(params, xsk));
 		break;
 	default: /* MLX5_WQ_TYPE_CYCLIC */
 		MLX5_SET(wq, wq, log_wq_sz, params->log_rq_mtu_frames);
-		mlx5e_build_rq_frags_info(mdev, params, &param->frags_info);
+		mlx5e_build_rq_frags_info(mdev, params, xsk, &param->frags_info);
 		ndsegs = param->frags_info.num_frags;
 	}
 
@@ -2061,8 +2146,8 @@ static void mlx5e_build_drop_rq_param(struct mlx5e_priv *priv,
 	param->wq.buf_numa_node = dev_to_node(mdev->device);
 }
 
-static void mlx5e_build_sq_param_common(struct mlx5e_priv *priv,
-					struct mlx5e_sq_param *param)
+void mlx5e_build_sq_param_common(struct mlx5e_priv *priv,
+				 struct mlx5e_sq_param *param)
 {
 	void *sqc = param->sqc;
 	void *wq = MLX5_ADDR_OF(sqc, sqc, wq);
@@ -2098,9 +2183,10 @@ static void mlx5e_build_common_cq_param(struct mlx5e_priv *priv,
 		MLX5_SET(cqc, cqc, cqe_sz, CQE_STRIDE_128_PAD);
 }
 
-static void mlx5e_build_rx_cq_param(struct mlx5e_priv *priv,
-				    struct mlx5e_params *params,
-				    struct mlx5e_cq_param *param)
+void mlx5e_build_rx_cq_param(struct mlx5e_priv *priv,
+			     struct mlx5e_params *params,
+			     struct mlx5e_xsk_param *xsk,
+			     struct mlx5e_cq_param *param)
 {
 	struct mlx5_core_dev *mdev = priv->mdev;
 	void *cqc = param->cqc;
@@ -2108,8 +2194,8 @@ static void mlx5e_build_rx_cq_param(struct mlx5e_priv *priv,
 
 	switch (params->rq_wq_type) {
 	case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
-		log_cq_size = mlx5e_mpwqe_get_log_rq_size(params) +
-			mlx5e_mpwqe_get_log_num_strides(mdev, params);
+		log_cq_size = mlx5e_mpwqe_get_log_rq_size(params, xsk) +
+			mlx5e_mpwqe_get_log_num_strides(mdev, params, xsk);
 		break;
 	default: /* MLX5_WQ_TYPE_CYCLIC */
 		log_cq_size = params->log_rq_mtu_frames;
@@ -2125,9 +2211,9 @@ static void mlx5e_build_rx_cq_param(struct mlx5e_priv *priv,
 	param->cq_period_mode = params->rx_cq_moderation.cq_period_mode;
 }
 
-static void mlx5e_build_tx_cq_param(struct mlx5e_priv *priv,
-				    struct mlx5e_params *params,
-				    struct mlx5e_cq_param *param)
+void mlx5e_build_tx_cq_param(struct mlx5e_priv *priv,
+			     struct mlx5e_params *params,
+			     struct mlx5e_cq_param *param)
 {
 	void *cqc = param->cqc;
 
@@ -2137,9 +2223,9 @@ static void mlx5e_build_tx_cq_param(struct mlx5e_priv *priv,
 	param->cq_period_mode = params->tx_cq_moderation.cq_period_mode;
 }
 
-static void mlx5e_build_ico_cq_param(struct mlx5e_priv *priv,
-				     u8 log_wq_size,
-				     struct mlx5e_cq_param *param)
+void mlx5e_build_ico_cq_param(struct mlx5e_priv *priv,
+			      u8 log_wq_size,
+			      struct mlx5e_cq_param *param)
 {
 	void *cqc = param->cqc;
 
@@ -2150,9 +2236,9 @@ static void mlx5e_build_ico_cq_param(struct mlx5e_priv *priv,
 	param->cq_period_mode = NET_DIM_CQ_PERIOD_MODE_START_FROM_EQE;
 }
 
-static void mlx5e_build_icosq_param(struct mlx5e_priv *priv,
-				    u8 log_wq_size,
-				    struct mlx5e_sq_param *param)
+void mlx5e_build_icosq_param(struct mlx5e_priv *priv,
+			     u8 log_wq_size,
+			     struct mlx5e_sq_param *param)
 {
 	void *sqc = param->sqc;
 	void *wq = MLX5_ADDR_OF(sqc, sqc, wq);
@@ -2163,9 +2249,9 @@ static void mlx5e_build_icosq_param(struct mlx5e_priv *priv,
 	MLX5_SET(sqc, sqc, reg_umr, MLX5_CAP_ETH(priv->mdev, reg_umr_sq));
 }
 
-static void mlx5e_build_xdpsq_param(struct mlx5e_priv *priv,
-				    struct mlx5e_params *params,
-				    struct mlx5e_sq_param *param)
+void mlx5e_build_xdpsq_param(struct mlx5e_priv *priv,
+			     struct mlx5e_params *params,
+			     struct mlx5e_sq_param *param)
 {
 	void *sqc = param->sqc;
 	void *wq = MLX5_ADDR_OF(sqc, sqc, wq);
@@ -2193,14 +2279,14 @@ static void mlx5e_build_channel_param(struct mlx5e_priv *priv,
 {
 	u8 icosq_log_wq_sz;
 
-	mlx5e_build_rq_param(priv, params, &cparam->rq);
+	mlx5e_build_rq_param(priv, params, NULL, &cparam->rq);
 
 	icosq_log_wq_sz = mlx5e_build_icosq_log_wq_sz(params, &cparam->rq);
 
 	mlx5e_build_sq_param(priv, params, &cparam->sq);
 	mlx5e_build_xdpsq_param(priv, params, &cparam->xdp_sq);
 	mlx5e_build_icosq_param(priv, icosq_log_wq_sz, &cparam->icosq);
-	mlx5e_build_rx_cq_param(priv, params, &cparam->rx_cq);
+	mlx5e_build_rx_cq_param(priv, params, NULL, &cparam->rx_cq);
 	mlx5e_build_tx_cq_param(priv, params, &cparam->tx_cq);
 	mlx5e_build_ico_cq_param(priv, icosq_log_wq_sz, &cparam->icosq_cq);
 }
@@ -2221,7 +2307,12 @@ int mlx5e_open_channels(struct mlx5e_priv *priv,
 
 	mlx5e_build_channel_param(priv, &chs->params, cparam);
 	for (i = 0; i < chs->num; i++) {
-		err = mlx5e_open_channel(priv, i, &chs->params, cparam, &chs->c[i]);
+		struct xdp_umem *umem = NULL;
+
+		if (chs->params.xdp_prog)
+			umem = mlx5e_xsk_get_umem(&chs->params, chs->params.xsk, i);
+
+		err = mlx5e_open_channel(priv, i, &chs->params, cparam, umem, &chs->c[i]);
 		if (err)
 			goto err_close_channels;
 	}
@@ -2263,6 +2354,10 @@ static int mlx5e_wait_channels_min_rx_wqes(struct mlx5e_channels *chs)
 		int timeout = err ? 0 : MLX5E_RQ_WQES_TIMEOUT;
 
 		err |= mlx5e_wait_for_min_rx_wqes(&chs->c[i]->rq, timeout);
+
+		/* Don't wait on the XSK RQ, because the newer xdpsock sample
+		 * doesn't provide any Fill Ring entries at the setup stage.
+		 */
 	}
 
 	return err ? -ETIMEDOUT : 0;
@@ -2335,35 +2430,35 @@ int mlx5e_create_indirect_rqt(struct mlx5e_priv *priv)
 	return err;
 }
 
-int mlx5e_create_direct_rqts(struct mlx5e_priv *priv)
+int mlx5e_create_direct_rqts(struct mlx5e_priv *priv, struct mlx5e_tir *tirs)
 {
-	struct mlx5e_rqt *rqt;
+	const int max_nch = mlx5e_get_netdev_max_channels(priv->netdev);
 	int err;
 	int ix;
 
-	for (ix = 0; ix < mlx5e_get_netdev_max_channels(priv->netdev); ix++) {
-		rqt = &priv->direct_tir[ix].rqt;
-		err = mlx5e_create_rqt(priv, 1 /*size */, rqt);
-		if (err)
+	for (ix = 0; ix < max_nch; ix++) {
+		err = mlx5e_create_rqt(priv, 1 /*size */, &tirs[ix].rqt);
+		if (unlikely(err))
 			goto err_destroy_rqts;
 	}
 
 	return 0;
 
 err_destroy_rqts:
-	mlx5_core_warn(priv->mdev, "create direct rqts failed, %d\n", err);
+	mlx5_core_warn(priv->mdev, "create rqts failed, %d\n", err);
 	for (ix--; ix >= 0; ix--)
-		mlx5e_destroy_rqt(priv, &priv->direct_tir[ix].rqt);
+		mlx5e_destroy_rqt(priv, &tirs[ix].rqt);
 
 	return err;
 }
 
-void mlx5e_destroy_direct_rqts(struct mlx5e_priv *priv)
+void mlx5e_destroy_direct_rqts(struct mlx5e_priv *priv, struct mlx5e_tir *tirs)
 {
+	const int max_nch = mlx5e_get_netdev_max_channels(priv->netdev);
 	int i;
 
-	for (i = 0; i < mlx5e_get_netdev_max_channels(priv->netdev); i++)
-		mlx5e_destroy_rqt(priv, &priv->direct_tir[i].rqt);
+	for (i = 0; i < max_nch; i++)
+		mlx5e_destroy_rqt(priv, &tirs[i].rqt);
 }
 
 static int mlx5e_rx_hash_fn(int hfunc)
@@ -2799,10 +2894,14 @@ void mlx5e_activate_priv_channels(struct mlx5e_priv *priv)
 
 	mlx5e_wait_channels_min_rx_wqes(&priv->channels);
 	mlx5e_redirect_rqts_to_channels(priv, &priv->channels);
+
+	mlx5e_xsk_redirect_rqts_to_channels(priv, &priv->channels);
 }
 
 void mlx5e_deactivate_priv_channels(struct mlx5e_priv *priv)
 {
+	mlx5e_xsk_redirect_rqts_to_drop(priv, &priv->channels);
+
 	mlx5e_redirect_rqts_to_drop(priv);
 
 	if (mlx5e_is_vport_rep(priv))
@@ -2881,9 +2980,12 @@ void mlx5e_timestamp_init(struct mlx5e_priv *priv)
 int mlx5e_open_locked(struct net_device *netdev)
 {
 	struct mlx5e_priv *priv = netdev_priv(netdev);
+	bool is_xdp = priv->channels.params.xdp_prog;
 	int err;
 
 	set_bit(MLX5E_STATE_OPENED, &priv->state);
+	if (is_xdp)
+		mlx5e_xdp_set_open(priv);
 
 	err = mlx5e_open_channels(priv, &priv->channels);
 	if (err)
@@ -2898,6 +3000,8 @@ int mlx5e_open_locked(struct net_device *netdev)
 	return 0;
 
 err_clear_state_opened_flag:
+	if (is_xdp)
+		mlx5e_xdp_set_closed(priv);
 	clear_bit(MLX5E_STATE_OPENED, &priv->state);
 	return err;
 }
@@ -2929,6 +3033,8 @@ int mlx5e_close_locked(struct net_device *netdev)
 	if (!test_bit(MLX5E_STATE_OPENED, &priv->state))
 		return 0;
 
+	if (priv->channels.params.xdp_prog)
+		mlx5e_xdp_set_closed(priv);
 	clear_bit(MLX5E_STATE_OPENED, &priv->state);
 
 	netif_carrier_off(priv->netdev);
@@ -3185,13 +3291,13 @@ int mlx5e_create_indirect_tirs(struct mlx5e_priv *priv, bool inner_ttc)
 	return err;
 }
 
-int mlx5e_create_direct_tirs(struct mlx5e_priv *priv)
+int mlx5e_create_direct_tirs(struct mlx5e_priv *priv, struct mlx5e_tir *tirs)
 {
-	int nch = mlx5e_get_netdev_max_channels(priv->netdev);
+	const int max_nch = mlx5e_get_netdev_max_channels(priv->netdev);
 	struct mlx5e_tir *tir;
 	void *tirc;
 	int inlen;
-	int err;
+	int err = 0;
 	u32 *in;
 	int ix;
 
@@ -3200,25 +3306,24 @@ int mlx5e_create_direct_tirs(struct mlx5e_priv *priv)
 	if (!in)
 		return -ENOMEM;
 
-	for (ix = 0; ix < nch; ix++) {
+	for (ix = 0; ix < max_nch; ix++) {
 		memset(in, 0, inlen);
-		tir = &priv->direct_tir[ix];
+		tir = &tirs[ix];
 		tirc = MLX5_ADDR_OF(create_tir_in, in, ctx);
-		mlx5e_build_direct_tir_ctx(priv, priv->direct_tir[ix].rqt.rqtn, tirc);
+		mlx5e_build_direct_tir_ctx(priv, tir->rqt.rqtn, tirc);
 		err = mlx5e_create_tir(priv->mdev, tir, in, inlen);
-		if (err)
+		if (unlikely(err))
 			goto err_destroy_ch_tirs;
 	}
 
-	kvfree(in);
-
-	return 0;
+	goto out;
 
 err_destroy_ch_tirs:
-	mlx5_core_warn(priv->mdev, "create direct tirs failed, %d\n", err);
+	mlx5_core_warn(priv->mdev, "create tirs failed, %d\n", err);
 	for (ix--; ix >= 0; ix--)
-		mlx5e_destroy_tir(priv->mdev, &priv->direct_tir[ix]);
+		mlx5e_destroy_tir(priv->mdev, &tirs[ix]);
 
+out:
 	kvfree(in);
 
 	return err;
@@ -3238,13 +3343,13 @@ void mlx5e_destroy_indirect_tirs(struct mlx5e_priv *priv, bool inner_ttc)
 		mlx5e_destroy_tir(priv->mdev, &priv->inner_indir_tir[i]);
 }
 
-void mlx5e_destroy_direct_tirs(struct mlx5e_priv *priv)
+void mlx5e_destroy_direct_tirs(struct mlx5e_priv *priv, struct mlx5e_tir *tirs)
 {
-	int nch = mlx5e_get_netdev_max_channels(priv->netdev);
+	const int max_nch = mlx5e_get_netdev_max_channels(priv->netdev);
 	int i;
 
-	for (i = 0; i < nch; i++)
-		mlx5e_destroy_tir(priv->mdev, &priv->direct_tir[i]);
+	for (i = 0; i < max_nch; i++)
+		mlx5e_destroy_tir(priv->mdev, &tirs[i]);
 }
 
 static int mlx5e_modify_channels_scatter_fcs(struct mlx5e_channels *chs, bool enable)
@@ -3386,11 +3491,12 @@ void mlx5e_fold_sw_stats64(struct mlx5e_priv *priv, struct rtnl_link_stats64 *s)
 
 	for (i = 0; i < mlx5e_get_netdev_max_channels(priv->netdev); i++) {
 		struct mlx5e_channel_stats *channel_stats = &priv->channel_stats[i];
+		struct mlx5e_rq_stats *xskrq_stats = &channel_stats->xskrq;
 		struct mlx5e_rq_stats *rq_stats = &channel_stats->rq;
 		int j;
 
-		s->rx_packets   += rq_stats->packets;
-		s->rx_bytes     += rq_stats->bytes;
+		s->rx_packets   += rq_stats->packets + xskrq_stats->packets;
+		s->rx_bytes     += rq_stats->bytes + xskrq_stats->bytes;
 
 		for (j = 0; j < priv->max_opened_tc; j++) {
 			struct mlx5e_sq_stats *sq_stats = &channel_stats->sq[j];
@@ -3489,6 +3595,13 @@ static int set_feature_lro(struct net_device *netdev, bool enable)
 
 	mutex_lock(&priv->state_lock);
 
+	if (enable && priv->xsk.refcnt) {
+		netdev_warn(netdev, "LRO is incompatible with AF_XDP (%hu XSKs are active)\n",
+			    priv->xsk.refcnt);
+		err = -EINVAL;
+		goto out;
+	}
+
 	old_params = &priv->channels.params;
 	if (enable && !MLX5E_GET_PFLAG(old_params, MLX5E_PFLAG_RX_STRIDING_RQ)) {
 		netdev_warn(netdev, "can't set LRO with legacy RQ\n");
@@ -3502,8 +3615,8 @@ static int set_feature_lro(struct net_device *netdev, bool enable)
 	new_channels.params.lro_en = enable;
 
 	if (old_params->rq_wq_type != MLX5_WQ_TYPE_CYCLIC) {
-		if (mlx5e_rx_mpwqe_is_linear_skb(mdev, old_params) ==
-		    mlx5e_rx_mpwqe_is_linear_skb(mdev, &new_channels.params))
+		if (mlx5e_rx_mpwqe_is_linear_skb(mdev, old_params, NULL) ==
+		    mlx5e_rx_mpwqe_is_linear_skb(mdev, &new_channels.params, NULL))
 			reset = false;
 	}
 
@@ -3688,6 +3801,43 @@ static netdev_features_t mlx5e_fix_features(struct net_device *netdev,
 	return features;
 }
 
+static bool mlx5e_xsk_validate_mtu(struct net_device *netdev,
+				   struct mlx5e_channels *chs,
+				   struct mlx5e_params *new_params,
+				   struct mlx5_core_dev *mdev)
+{
+	u16 ix;
+
+	for (ix = 0; ix < chs->params.num_channels; ix++) {
+		struct xdp_umem *umem = mlx5e_xsk_get_umem(&chs->params, chs->params.xsk, ix);
+		struct mlx5e_xsk_param xsk;
+
+		if (!umem)
+			continue;
+
+		mlx5e_build_xsk_param(umem, &xsk);
+
+		if (!mlx5e_validate_xsk_param(new_params, &xsk, mdev)) {
+			u32 hr = mlx5e_get_linear_rq_headroom(new_params, &xsk);
+			int max_mtu_frame, max_mtu_page, max_mtu;
+
+			/* Two criteria must be met:
+			 * 1. HW MTU + all headrooms <= XSK frame size.
+			 * 2. Size of SKBs allocated on XDP_PASS <= PAGE_SIZE.
+			 */
+			max_mtu_frame = MLX5E_HW2SW_MTU(new_params, xsk.chunk_size - hr);
+			max_mtu_page = mlx5e_xdp_max_mtu(new_params, &xsk);
+			max_mtu = min(max_mtu_frame, max_mtu_page);
+
+			netdev_err(netdev, "MTU %d is too big for an XSK running on channel %hu. Try MTU <= %d\n",
+				   new_params->sw_mtu, ix, max_mtu);
+			return false;
+		}
+	}
+
+	return true;
+}
+
 int mlx5e_change_mtu(struct net_device *netdev, int new_mtu,
 		     change_hw_mtu_cb set_mtu_cb)
 {
@@ -3708,18 +3858,31 @@ int mlx5e_change_mtu(struct net_device *netdev, int new_mtu,
 	new_channels.params.sw_mtu = new_mtu;
 
 	if (params->xdp_prog &&
-	    !mlx5e_rx_is_linear_skb(&new_channels.params)) {
+	    !mlx5e_rx_is_linear_skb(&new_channels.params, NULL)) {
 		netdev_err(netdev, "MTU(%d) > %d is not allowed while XDP enabled\n",
 			   new_mtu, mlx5e_xdp_max_mtu(params, NULL));
 		err = -EINVAL;
 		goto out;
 	}
 
+	if (priv->xsk.refcnt &&
+	    !mlx5e_xsk_validate_mtu(netdev, &priv->channels,
+				    &new_channels.params, priv->mdev)) {
+		err = -EINVAL;
+		goto out;
+	}
+
 	if (params->rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ) {
-		bool is_linear = mlx5e_rx_mpwqe_is_linear_skb(priv->mdev, &new_channels.params);
-		u8 ppw_old = mlx5e_mpwqe_log_pkts_per_wqe(params);
-		u8 ppw_new = mlx5e_mpwqe_log_pkts_per_wqe(&new_channels.params);
+		bool is_linear = mlx5e_rx_mpwqe_is_linear_skb(priv->mdev,
+							      &new_channels.params,
+							      NULL);
+		u8 ppw_old = mlx5e_mpwqe_log_pkts_per_wqe(params, NULL);
+		u8 ppw_new = mlx5e_mpwqe_log_pkts_per_wqe(&new_channels.params, NULL);
 
+		/* If XSK is active, XSK RQs are linear. */
+		is_linear |= priv->xsk.refcnt;
+
+		/* Always reset in linear mode - hw_mtu is used in data path. */
 		reset = reset && (is_linear || (ppw_old != ppw_new));
 	}
 
@@ -4149,7 +4312,10 @@ static int mlx5e_xdp_allowed(struct mlx5e_priv *priv, struct bpf_prog *prog)
 	new_channels.params = priv->channels.params;
 	new_channels.params.xdp_prog = prog;
 
-	if (!mlx5e_rx_is_linear_skb(&new_channels.params)) {
+	/* No XSK params: AF_XDP can't be enabled yet at the point of setting
+	 * the XDP program.
+	 */
+	if (!mlx5e_rx_is_linear_skb(&new_channels.params, NULL)) {
 		netdev_warn(netdev, "XDP is not allowed with MTU(%d) > %d\n",
 			    new_channels.params.sw_mtu,
 			    mlx5e_xdp_max_mtu(&new_channels.params, NULL));
@@ -4213,19 +4379,29 @@ static int mlx5e_xdp_set(struct net_device *netdev, struct bpf_prog *prog)
 	 */
 	for (i = 0; i < priv->channels.num; i++) {
 		struct mlx5e_channel *c = priv->channels.c[i];
+		bool xsk_open = test_bit(MLX5E_CHANNEL_STATE_XSK, c->state);
 
 		clear_bit(MLX5E_RQ_STATE_ENABLED, &c->rq.state);
+		if (xsk_open)
+			clear_bit(MLX5E_RQ_STATE_ENABLED, &c->xskrq.state);
 		napi_synchronize(&c->napi);
 		/* prevent mlx5e_poll_rx_cq from accessing rq->xdp_prog */
 
 		old_prog = xchg(&c->rq.xdp_prog, prog);
+		if (old_prog)
+			bpf_prog_put(old_prog);
+
+		if (xsk_open) {
+			old_prog = xchg(&c->xskrq.xdp_prog, prog);
+			if (old_prog)
+				bpf_prog_put(old_prog);
+		}
 
 		set_bit(MLX5E_RQ_STATE_ENABLED, &c->rq.state);
+		if (xsk_open)
+			set_bit(MLX5E_RQ_STATE_ENABLED, &c->xskrq.state);
 		/* napi_schedule in case we have missed anything */
 		napi_schedule(&c->napi);
-
-		if (old_prog)
-			bpf_prog_put(old_prog);
 	}
 
 unlock:
@@ -4256,6 +4432,9 @@ static int mlx5e_xdp(struct net_device *dev, struct netdev_bpf *xdp)
 	case XDP_QUERY_PROG:
 		xdp->prog_id = mlx5e_xdp_query(dev);
 		return 0;
+	case XDP_SETUP_XSK_UMEM:
+		return mlx5e_xsk_setup_umem(dev, xdp->xsk.umem,
+					    xdp->xsk.queue_id);
 	default:
 		return -EINVAL;
 	}
@@ -4338,6 +4517,7 @@ const struct net_device_ops mlx5e_netdev_ops = {
 	.ndo_tx_timeout          = mlx5e_tx_timeout,
 	.ndo_bpf		 = mlx5e_xdp,
 	.ndo_xdp_xmit            = mlx5e_xdp_xmit,
+	.ndo_xsk_async_xmit      = mlx5e_xsk_async_xmit,
 #ifdef CONFIG_MLX5_EN_ARFS
 	.ndo_rx_flow_steer	 = mlx5e_rx_flow_steer,
 #endif
@@ -4489,11 +4669,13 @@ void mlx5e_build_rq_params(struct mlx5_core_dev *mdev,
 	 * - Striding RQ configuration is not possible/supported.
 	 * - Slow PCI heuristic.
 	 * - Legacy RQ would use linear SKB while Striding RQ would use non-linear.
+	 *
+	 * No XSK params: checking the availability of striding RQ in general.
 	 */
 	if (!slow_pci_heuristic(mdev) &&
 	    mlx5e_striding_rq_possible(mdev, params) &&
-	    (mlx5e_rx_mpwqe_is_linear_skb(mdev, params) ||
-	     !mlx5e_rx_is_linear_skb(params)))
+	    (mlx5e_rx_mpwqe_is_linear_skb(mdev, params, NULL) ||
+	     !mlx5e_rx_is_linear_skb(params, NULL)))
 		MLX5E_SET_PFLAG(params, MLX5E_PFLAG_RX_STRIDING_RQ, true);
 	mlx5e_set_rq_type(mdev, params);
 	mlx5e_init_rq_type_params(mdev, params);
@@ -4515,6 +4697,7 @@ void mlx5e_build_rss_params(struct mlx5e_rss_params *rss_params,
 }
 
 void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
+			    struct mlx5e_xsk *xsk,
 			    struct mlx5e_rss_params *rss_params,
 			    struct mlx5e_params *params,
 			    u16 max_channels, u16 mtu)
@@ -4550,9 +4733,11 @@ void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
 	/* HW LRO */
 
 	/* TODO: && MLX5_CAP_ETH(mdev, lro_cap) */
-	if (params->rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ)
-		if (!mlx5e_rx_mpwqe_is_linear_skb(mdev, params))
+	if (params->rq_wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ) {
+		/* No XSK params: checking the availability of striding RQ in general. */
+		if (!mlx5e_rx_mpwqe_is_linear_skb(mdev, params, NULL))
 			params->lro_en = !slow_pci_heuristic(mdev);
+	}
 	params->lro_timeout = mlx5e_choose_lro_timeout(mdev, MLX5E_DEFAULT_LRO_TIMEOUT);
 
 	/* CQ moderation params */
@@ -4571,6 +4756,9 @@ void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
 	mlx5e_build_rss_params(rss_params, params->num_channels);
 	params->tunneled_offload_en =
 		mlx5e_tunnel_inner_ft_supported(mdev);
+
+	/* AF_XDP */
+	params->xsk = xsk;
 }
 
 static void mlx5e_set_netdev_dev_addr(struct net_device *netdev)
@@ -4739,7 +4927,7 @@ static int mlx5e_nic_init(struct mlx5_core_dev *mdev,
 	if (err)
 		return err;
 
-	mlx5e_build_nic_params(mdev, rss, &priv->channels.params,
+	mlx5e_build_nic_params(mdev, &priv->xsk, rss, &priv->channels.params,
 			       mlx5e_get_netdev_max_channels(netdev),
 			       netdev->mtu);
 
@@ -4781,7 +4969,7 @@ static int mlx5e_init_nic_rx(struct mlx5e_priv *priv)
 	if (err)
 		goto err_close_drop_rq;
 
-	err = mlx5e_create_direct_rqts(priv);
+	err = mlx5e_create_direct_rqts(priv, priv->direct_tir);
 	if (err)
 		goto err_destroy_indirect_rqts;
 
@@ -4789,14 +4977,22 @@ static int mlx5e_init_nic_rx(struct mlx5e_priv *priv)
 	if (err)
 		goto err_destroy_direct_rqts;
 
-	err = mlx5e_create_direct_tirs(priv);
+	err = mlx5e_create_direct_tirs(priv, priv->direct_tir);
 	if (err)
 		goto err_destroy_indirect_tirs;
 
+	err = mlx5e_create_direct_rqts(priv, priv->xsk_tir);
+	if (unlikely(err))
+		goto err_destroy_direct_tirs;
+
+	err = mlx5e_create_direct_tirs(priv, priv->xsk_tir);
+	if (unlikely(err))
+		goto err_destroy_xsk_rqts;
+
 	err = mlx5e_create_flow_steering(priv);
 	if (err) {
 		mlx5_core_warn(mdev, "create flow steering failed, %d\n", err);
-		goto err_destroy_direct_tirs;
+		goto err_destroy_xsk_tirs;
 	}
 
 	err = mlx5e_tc_nic_init(priv);
@@ -4807,12 +5003,16 @@ static int mlx5e_init_nic_rx(struct mlx5e_priv *priv)
 
 err_destroy_flow_steering:
 	mlx5e_destroy_flow_steering(priv);
+err_destroy_xsk_tirs:
+	mlx5e_destroy_direct_tirs(priv, priv->xsk_tir);
+err_destroy_xsk_rqts:
+	mlx5e_destroy_direct_rqts(priv, priv->xsk_tir);
 err_destroy_direct_tirs:
-	mlx5e_destroy_direct_tirs(priv);
+	mlx5e_destroy_direct_tirs(priv, priv->direct_tir);
 err_destroy_indirect_tirs:
 	mlx5e_destroy_indirect_tirs(priv, true);
 err_destroy_direct_rqts:
-	mlx5e_destroy_direct_rqts(priv);
+	mlx5e_destroy_direct_rqts(priv, priv->direct_tir);
 err_destroy_indirect_rqts:
 	mlx5e_destroy_rqt(priv, &priv->indir_rqt);
 err_close_drop_rq:
@@ -4826,9 +5026,11 @@ static void mlx5e_cleanup_nic_rx(struct mlx5e_priv *priv)
 {
 	mlx5e_tc_nic_cleanup(priv);
 	mlx5e_destroy_flow_steering(priv);
-	mlx5e_destroy_direct_tirs(priv);
+	mlx5e_destroy_direct_tirs(priv, priv->xsk_tir);
+	mlx5e_destroy_direct_rqts(priv, priv->xsk_tir);
+	mlx5e_destroy_direct_tirs(priv, priv->direct_tir);
 	mlx5e_destroy_indirect_tirs(priv, true);
-	mlx5e_destroy_direct_rqts(priv);
+	mlx5e_destroy_direct_rqts(priv, priv->direct_tir);
 	mlx5e_destroy_rqt(priv, &priv->indir_rqt);
 	mlx5e_close_drop_rq(&priv->drop_rq);
 	mlx5e_destroy_q_counters(priv);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
index 91e24f1cead8..ec229face359 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
@@ -1511,7 +1511,7 @@ static int mlx5e_init_rep_rx(struct mlx5e_priv *priv)
 	if (err)
 		goto err_close_drop_rq;
 
-	err = mlx5e_create_direct_rqts(priv);
+	err = mlx5e_create_direct_rqts(priv, priv->direct_tir);
 	if (err)
 		goto err_destroy_indirect_rqts;
 
@@ -1519,7 +1519,7 @@ static int mlx5e_init_rep_rx(struct mlx5e_priv *priv)
 	if (err)
 		goto err_destroy_direct_rqts;
 
-	err = mlx5e_create_direct_tirs(priv);
+	err = mlx5e_create_direct_tirs(priv, priv->direct_tir);
 	if (err)
 		goto err_destroy_indirect_tirs;
 
@@ -1536,11 +1536,11 @@ static int mlx5e_init_rep_rx(struct mlx5e_priv *priv)
 err_destroy_ttc_table:
 	mlx5e_destroy_ttc_table(priv, &priv->fs.ttc);
 err_destroy_direct_tirs:
-	mlx5e_destroy_direct_tirs(priv);
+	mlx5e_destroy_direct_tirs(priv, priv->direct_tir);
 err_destroy_indirect_tirs:
 	mlx5e_destroy_indirect_tirs(priv, false);
 err_destroy_direct_rqts:
-	mlx5e_destroy_direct_rqts(priv);
+	mlx5e_destroy_direct_rqts(priv, priv->direct_tir);
 err_destroy_indirect_rqts:
 	mlx5e_destroy_rqt(priv, &priv->indir_rqt);
 err_close_drop_rq:
@@ -1554,9 +1554,9 @@ static void mlx5e_cleanup_rep_rx(struct mlx5e_priv *priv)
 
 	mlx5_del_flow_rules(rpriv->vport_rx_rule);
 	mlx5e_destroy_ttc_table(priv, &priv->fs.ttc);
-	mlx5e_destroy_direct_tirs(priv);
+	mlx5e_destroy_direct_tirs(priv, priv->direct_tir);
 	mlx5e_destroy_indirect_tirs(priv, false);
-	mlx5e_destroy_direct_rqts(priv);
+	mlx5e_destroy_direct_rqts(priv, priv->direct_tir);
 	mlx5e_destroy_rqt(priv, &priv->indir_rqt);
 	mlx5e_close_drop_rq(&priv->drop_rq);
 }
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 5d762da6bf9b..9eedc6793984 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -46,6 +46,7 @@
 #include "en_accel/tls_rxtx.h"
 #include "lib/clock.h"
 #include "en/xdp.h"
+#include "en/xsk/rx.h"
 
 static inline bool mlx5e_rx_hw_stamp(struct hwtstamp_config *config)
 {
@@ -234,8 +235,8 @@ static inline bool mlx5e_rx_cache_get(struct mlx5e_rq *rq,
 	return true;
 }
 
-static inline int mlx5e_page_alloc_mapped(struct mlx5e_rq *rq,
-					  struct mlx5e_dma_info *dma_info)
+static inline int mlx5e_page_alloc_pool(struct mlx5e_rq *rq,
+					struct mlx5e_dma_info *dma_info)
 {
 	if (mlx5e_rx_cache_get(rq, dma_info))
 		return 0;
@@ -255,13 +256,23 @@ static inline int mlx5e_page_alloc_mapped(struct mlx5e_rq *rq,
 	return 0;
 }
 
+static inline int mlx5e_page_alloc(struct mlx5e_rq *rq,
+				   struct mlx5e_dma_info *dma_info)
+{
+	if (rq->umem)
+		return mlx5e_xsk_page_alloc_umem(rq, dma_info);
+	else
+		return mlx5e_page_alloc_pool(rq, dma_info);
+}
+
 void mlx5e_page_dma_unmap(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info)
 {
 	dma_unmap_page(rq->pdev, dma_info->addr, PAGE_SIZE, rq->buff.map_dir);
 }
 
-void mlx5e_page_release(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info,
-			bool recycle)
+void mlx5e_page_release_dynamic(struct mlx5e_rq *rq,
+				struct mlx5e_dma_info *dma_info,
+				bool recycle)
 {
 	if (likely(recycle)) {
 		if (mlx5e_rx_cache_put(rq, dma_info))
@@ -275,6 +286,20 @@ void mlx5e_page_release(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info,
 	}
 }
 
+static inline void mlx5e_page_release(struct mlx5e_rq *rq,
+				      struct mlx5e_dma_info *dma_info,
+				      bool recycle)
+{
+	if (rq->umem)
+		/* The `recycle` parameter is ignored, and the page is always
+		 * put into the Reuse Ring, because there is no way to return
+		 * the page to the userspace when the interface goes down.
+		 */
+		mlx5e_xsk_page_release(rq, dma_info);
+	else
+		mlx5e_page_release_dynamic(rq, dma_info, recycle);
+}
+
 static inline int mlx5e_get_rx_frag(struct mlx5e_rq *rq,
 				    struct mlx5e_wqe_frag_info *frag)
 {
@@ -286,7 +311,7 @@ static inline int mlx5e_get_rx_frag(struct mlx5e_rq *rq,
 		 * offset) should just use the new one without replenishing again
 		 * by themselves.
 		 */
-		err = mlx5e_page_alloc_mapped(rq, frag->di);
+		err = mlx5e_page_alloc(rq, frag->di);
 
 	return err;
 }
@@ -352,6 +377,13 @@ static int mlx5e_alloc_rx_wqes(struct mlx5e_rq *rq, u16 ix, u8 wqe_bulk)
 	int err;
 	int i;
 
+	if (rq->umem) {
+		int pages_desired = wqe_bulk << rq->wqe.info.log_num_frags;
+
+		if (unlikely(!mlx5e_xsk_pages_enough_umem(rq, pages_desired)))
+			return -ENOMEM;
+	}
+
 	for (i = 0; i < wqe_bulk; i++) {
 		struct mlx5e_rx_wqe_cyc *wqe = mlx5_wq_cyc_get_wqe(wq, ix + i);
 
@@ -399,11 +431,17 @@ mlx5e_copy_skb_header(struct device *pdev, struct sk_buff *skb,
 static void
 mlx5e_free_rx_mpwqe(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi, bool recycle)
 {
-	const bool no_xdp_xmit =
-		bitmap_empty(wi->xdp_xmit_bitmap, MLX5_MPWRQ_PAGES_PER_WQE);
+	bool no_xdp_xmit;
 	struct mlx5e_dma_info *dma_info = wi->umr.dma_info;
 	int i;
 
+	/* A common case for AF_XDP. */
+	if (bitmap_full(wi->xdp_xmit_bitmap, MLX5_MPWRQ_PAGES_PER_WQE))
+		return;
+
+	no_xdp_xmit = bitmap_empty(wi->xdp_xmit_bitmap,
+				   MLX5_MPWRQ_PAGES_PER_WQE);
+
 	for (i = 0; i < MLX5_MPWRQ_PAGES_PER_WQE; i++)
 		if (no_xdp_xmit || !test_bit(i, wi->xdp_xmit_bitmap))
 			mlx5e_page_release(rq, &dma_info[i], recycle);
@@ -452,6 +490,12 @@ static int mlx5e_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix)
 	int err;
 	int i;
 
+	if (rq->umem &&
+	    unlikely(!mlx5e_xsk_pages_enough_umem(rq, MLX5_MPWRQ_PAGES_PER_WQE))) {
+		err = -ENOMEM;
+		goto err;
+	}
+
 	pi = mlx5_wq_cyc_ctr2ix(wq, sq->pc);
 	contig_wqebbs_room = mlx5_wq_cyc_get_contig_wqebbs(wq, pi);
 	if (unlikely(contig_wqebbs_room < MLX5E_UMR_WQEBBS)) {
@@ -463,7 +507,7 @@ static int mlx5e_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix)
 	memcpy(umr_wqe, &rq->mpwqe.umr_wqe, offsetof(struct mlx5e_umr_wqe, inline_mtts));
 
 	for (i = 0; i < MLX5_MPWRQ_PAGES_PER_WQE; i++, dma_info++) {
-		err = mlx5e_page_alloc_mapped(rq, dma_info);
+		err = mlx5e_page_alloc(rq, dma_info);
 		if (unlikely(err))
 			goto err_unmap;
 		umr_wqe->inline_mtts[i].ptag = cpu_to_be64(dma_info->addr | MLX5_EN_WR);
@@ -490,6 +534,8 @@ static int mlx5e_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix)
 		dma_info--;
 		mlx5e_page_release(rq, dma_info, true);
 	}
+
+err:
 	rq->stats->buff_alloc_err++;
 
 	return err;
@@ -603,6 +649,7 @@ bool mlx5e_post_rx_mpwqes(struct mlx5e_rq *rq)
 	struct mlx5e_icosq *sq = &rq->channel->icosq;
 	struct mlx5_wq_ll *wq = &rq->mpwqe.wq;
 	u8  umr_completed = rq->mpwqe.umr_completed;
+	int alloc_err = 0;
 	u8  missing, i;
 	u16 head;
 
@@ -627,7 +674,9 @@ bool mlx5e_post_rx_mpwqes(struct mlx5e_rq *rq)
 	head = rq->mpwqe.actual_wq_head;
 	i = missing;
 	do {
-		if (unlikely(mlx5e_alloc_rx_mpwqe(rq, head)))
+		alloc_err = mlx5e_alloc_rx_mpwqe(rq, head);
+
+		if (unlikely(alloc_err))
 			break;
 		head = mlx5_wq_ll_get_wqe_next_ix(wq, head);
 	} while (--i);
@@ -641,6 +690,12 @@ bool mlx5e_post_rx_mpwqes(struct mlx5e_rq *rq)
 	rq->mpwqe.umr_in_progress += rq->mpwqe.umr_last_bulk;
 	rq->mpwqe.actual_wq_head   = head;
 
+	/* If XSK Fill Ring doesn't have enough frames, busy poll by
+	 * rescheduling the NAPI poll.
+	 */
+	if (unlikely(alloc_err == -ENOMEM && rq->umem))
+		return true;
+
 	return false;
 }
 
@@ -1009,7 +1064,7 @@ mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,
 	}
 
 	rcu_read_lock();
-	consumed = mlx5e_xdp_handle(rq, di, va, &rx_headroom, &cqe_bcnt);
+	consumed = mlx5e_xdp_handle(rq, di, va, &rx_headroom, &cqe_bcnt, false);
 	rcu_read_unlock();
 	if (consumed)
 		return NULL; /* page/packet was consumed by XDP */
@@ -1223,7 +1278,7 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
 	prefetch(data);
 
 	rcu_read_lock();
-	consumed = mlx5e_xdp_handle(rq, di, va, &rx_headroom, &cqe_bcnt32);
+	consumed = mlx5e_xdp_handle(rq, di, va, &rx_headroom, &cqe_bcnt32, false);
 	rcu_read_unlock();
 	if (consumed) {
 		if (__test_and_clear_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags))
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
index 483d321d2151..5f540db47cc9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
@@ -104,7 +104,33 @@ static const struct counter_desc sw_stats_desc[] = {
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_poll) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_arm) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_aff_change) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_force_irq) },
 	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, ch_eq_rearm) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_packets) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_bytes) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_csum_complete) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_csum_unnecessary) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_csum_unnecessary_inner) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_csum_none) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_ecn_mark) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_removed_vlan_packets) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_xdp_drop) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_xdp_redirect) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_wqe_err) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_mpwqe_filler_cqes) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_mpwqe_filler_strides) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_oversize_pkts_sw_drop) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_buff_alloc_err) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_cqe_compress_blks) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_cqe_compress_pkts) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_congst_umr) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, rx_xsk_arfs_err) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xsk_xmit) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xsk_mpwqe) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xsk_inlnw) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xsk_full) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xsk_err) },
+	{ MLX5E_DECLARE_STAT(struct mlx5e_sw_stats, tx_xsk_cqes) },
 };
 
 #define NUM_SW_COUNTERS			ARRAY_SIZE(sw_stats_desc)
@@ -144,6 +170,8 @@ static void mlx5e_grp_sw_update_stats(struct mlx5e_priv *priv)
 			&priv->channel_stats[i];
 		struct mlx5e_xdpsq_stats *xdpsq_red_stats = &channel_stats->xdpsq;
 		struct mlx5e_xdpsq_stats *xdpsq_stats = &channel_stats->rq_xdpsq;
+		struct mlx5e_xdpsq_stats *xsksq_stats = &channel_stats->xsksq;
+		struct mlx5e_rq_stats *xskrq_stats = &channel_stats->xskrq;
 		struct mlx5e_rq_stats *rq_stats = &channel_stats->rq;
 		struct mlx5e_ch_stats *ch_stats = &channel_stats->ch;
 		int j;
@@ -186,6 +214,7 @@ static void mlx5e_grp_sw_update_stats(struct mlx5e_priv *priv)
 		s->ch_poll        += ch_stats->poll;
 		s->ch_arm         += ch_stats->arm;
 		s->ch_aff_change  += ch_stats->aff_change;
+		s->ch_force_irq   += ch_stats->force_irq;
 		s->ch_eq_rearm    += ch_stats->eq_rearm;
 		/* xdp redirect */
 		s->tx_xdp_xmit    += xdpsq_red_stats->xmit;
@@ -194,6 +223,32 @@ static void mlx5e_grp_sw_update_stats(struct mlx5e_priv *priv)
 		s->tx_xdp_full    += xdpsq_red_stats->full;
 		s->tx_xdp_err     += xdpsq_red_stats->err;
 		s->tx_xdp_cqes    += xdpsq_red_stats->cqes;
+		/* AF_XDP zero-copy */
+		s->rx_xsk_packets                += xskrq_stats->packets;
+		s->rx_xsk_bytes                  += xskrq_stats->bytes;
+		s->rx_xsk_csum_complete          += xskrq_stats->csum_complete;
+		s->rx_xsk_csum_unnecessary       += xskrq_stats->csum_unnecessary;
+		s->rx_xsk_csum_unnecessary_inner += xskrq_stats->csum_unnecessary_inner;
+		s->rx_xsk_csum_none              += xskrq_stats->csum_none;
+		s->rx_xsk_ecn_mark               += xskrq_stats->ecn_mark;
+		s->rx_xsk_removed_vlan_packets   += xskrq_stats->removed_vlan_packets;
+		s->rx_xsk_xdp_drop               += xskrq_stats->xdp_drop;
+		s->rx_xsk_xdp_redirect           += xskrq_stats->xdp_redirect;
+		s->rx_xsk_wqe_err                += xskrq_stats->wqe_err;
+		s->rx_xsk_mpwqe_filler_cqes      += xskrq_stats->mpwqe_filler_cqes;
+		s->rx_xsk_mpwqe_filler_strides   += xskrq_stats->mpwqe_filler_strides;
+		s->rx_xsk_oversize_pkts_sw_drop  += xskrq_stats->oversize_pkts_sw_drop;
+		s->rx_xsk_buff_alloc_err         += xskrq_stats->buff_alloc_err;
+		s->rx_xsk_cqe_compress_blks      += xskrq_stats->cqe_compress_blks;
+		s->rx_xsk_cqe_compress_pkts      += xskrq_stats->cqe_compress_pkts;
+		s->rx_xsk_congst_umr             += xskrq_stats->congst_umr;
+		s->rx_xsk_arfs_err               += xskrq_stats->arfs_err;
+		s->tx_xsk_xmit                   += xsksq_stats->xmit;
+		s->tx_xsk_mpwqe                  += xsksq_stats->mpwqe;
+		s->tx_xsk_inlnw                  += xsksq_stats->inlnw;
+		s->tx_xsk_full                   += xsksq_stats->full;
+		s->tx_xsk_err                    += xsksq_stats->err;
+		s->tx_xsk_cqes                   += xsksq_stats->cqes;
 
 		for (j = 0; j < priv->max_opened_tc; j++) {
 			struct mlx5e_sq_stats *sq_stats = &channel_stats->sq[j];
@@ -1266,11 +1321,43 @@ static const struct counter_desc xdpsq_stats_desc[] = {
 	{ MLX5E_DECLARE_XDPSQ_STAT(struct mlx5e_xdpsq_stats, cqes) },
 };
 
+static const struct counter_desc xskrq_stats_desc[] = {
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, packets) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, bytes) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, csum_complete) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, csum_unnecessary) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, csum_unnecessary_inner) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, csum_none) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, ecn_mark) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, removed_vlan_packets) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, xdp_drop) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, xdp_redirect) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, wqe_err) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, mpwqe_filler_cqes) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, mpwqe_filler_strides) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, oversize_pkts_sw_drop) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, buff_alloc_err) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, cqe_compress_blks) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, cqe_compress_pkts) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, congst_umr) },
+	{ MLX5E_DECLARE_XSKRQ_STAT(struct mlx5e_rq_stats, arfs_err) },
+};
+
+static const struct counter_desc xsksq_stats_desc[] = {
+	{ MLX5E_DECLARE_XSKSQ_STAT(struct mlx5e_xdpsq_stats, xmit) },
+	{ MLX5E_DECLARE_XSKSQ_STAT(struct mlx5e_xdpsq_stats, mpwqe) },
+	{ MLX5E_DECLARE_XSKSQ_STAT(struct mlx5e_xdpsq_stats, inlnw) },
+	{ MLX5E_DECLARE_XSKSQ_STAT(struct mlx5e_xdpsq_stats, full) },
+	{ MLX5E_DECLARE_XSKSQ_STAT(struct mlx5e_xdpsq_stats, err) },
+	{ MLX5E_DECLARE_XSKSQ_STAT(struct mlx5e_xdpsq_stats, cqes) },
+};
+
 static const struct counter_desc ch_stats_desc[] = {
 	{ MLX5E_DECLARE_CH_STAT(struct mlx5e_ch_stats, events) },
 	{ MLX5E_DECLARE_CH_STAT(struct mlx5e_ch_stats, poll) },
 	{ MLX5E_DECLARE_CH_STAT(struct mlx5e_ch_stats, arm) },
 	{ MLX5E_DECLARE_CH_STAT(struct mlx5e_ch_stats, aff_change) },
+	{ MLX5E_DECLARE_CH_STAT(struct mlx5e_ch_stats, force_irq) },
 	{ MLX5E_DECLARE_CH_STAT(struct mlx5e_ch_stats, eq_rearm) },
 };
 
@@ -1278,6 +1365,8 @@ static const struct counter_desc ch_stats_desc[] = {
 #define NUM_SQ_STATS			ARRAY_SIZE(sq_stats_desc)
 #define NUM_XDPSQ_STATS			ARRAY_SIZE(xdpsq_stats_desc)
 #define NUM_RQ_XDPSQ_STATS		ARRAY_SIZE(rq_xdpsq_stats_desc)
+#define NUM_XSKRQ_STATS			ARRAY_SIZE(xskrq_stats_desc)
+#define NUM_XSKSQ_STATS			ARRAY_SIZE(xsksq_stats_desc)
 #define NUM_CH_STATS			ARRAY_SIZE(ch_stats_desc)
 
 static int mlx5e_grp_channels_get_num_stats(struct mlx5e_priv *priv)
@@ -1288,13 +1377,16 @@ static int mlx5e_grp_channels_get_num_stats(struct mlx5e_priv *priv)
 	       (NUM_CH_STATS * max_nch) +
 	       (NUM_SQ_STATS * max_nch * priv->max_opened_tc) +
 	       (NUM_RQ_XDPSQ_STATS * max_nch) +
-	       (NUM_XDPSQ_STATS * max_nch);
+	       (NUM_XDPSQ_STATS * max_nch) +
+	       (NUM_XSKRQ_STATS * max_nch * priv->xsk.ever_used) +
+	       (NUM_XSKSQ_STATS * max_nch * priv->xsk.ever_used);
 }
 
 static int mlx5e_grp_channels_fill_strings(struct mlx5e_priv *priv, u8 *data,
 					   int idx)
 {
 	int max_nch = mlx5e_get_netdev_max_channels(priv->netdev);
+	bool is_xsk = priv->xsk.ever_used;
 	int i, j, tc;
 
 	for (i = 0; i < max_nch; i++)
@@ -1306,6 +1398,9 @@ static int mlx5e_grp_channels_fill_strings(struct mlx5e_priv *priv, u8 *data,
 		for (j = 0; j < NUM_RQ_STATS; j++)
 			sprintf(data + (idx++) * ETH_GSTRING_LEN,
 				rq_stats_desc[j].format, i);
+		for (j = 0; j < NUM_XSKRQ_STATS * is_xsk; j++)
+			sprintf(data + (idx++) * ETH_GSTRING_LEN,
+				xskrq_stats_desc[j].format, i);
 		for (j = 0; j < NUM_RQ_XDPSQ_STATS; j++)
 			sprintf(data + (idx++) * ETH_GSTRING_LEN,
 				rq_xdpsq_stats_desc[j].format, i);
@@ -1318,10 +1413,14 @@ static int mlx5e_grp_channels_fill_strings(struct mlx5e_priv *priv, u8 *data,
 					sq_stats_desc[j].format,
 					priv->channel_tc2txq[i][tc]);
 
-	for (i = 0; i < max_nch; i++)
+	for (i = 0; i < max_nch; i++) {
+		for (j = 0; j < NUM_XSKSQ_STATS * is_xsk; j++)
+			sprintf(data + (idx++) * ETH_GSTRING_LEN,
+				xsksq_stats_desc[j].format, i);
 		for (j = 0; j < NUM_XDPSQ_STATS; j++)
 			sprintf(data + (idx++) * ETH_GSTRING_LEN,
 				xdpsq_stats_desc[j].format, i);
+	}
 
 	return idx;
 }
@@ -1330,6 +1429,7 @@ static int mlx5e_grp_channels_fill_stats(struct mlx5e_priv *priv, u64 *data,
 					 int idx)
 {
 	int max_nch = mlx5e_get_netdev_max_channels(priv->netdev);
+	bool is_xsk = priv->xsk.ever_used;
 	int i, j, tc;
 
 	for (i = 0; i < max_nch; i++)
@@ -1343,6 +1443,10 @@ static int mlx5e_grp_channels_fill_stats(struct mlx5e_priv *priv, u64 *data,
 			data[idx++] =
 				MLX5E_READ_CTR64_CPU(&priv->channel_stats[i].rq,
 						     rq_stats_desc, j);
+		for (j = 0; j < NUM_XSKRQ_STATS * is_xsk; j++)
+			data[idx++] =
+				MLX5E_READ_CTR64_CPU(&priv->channel_stats[i].xskrq,
+						     xskrq_stats_desc, j);
 		for (j = 0; j < NUM_RQ_XDPSQ_STATS; j++)
 			data[idx++] =
 				MLX5E_READ_CTR64_CPU(&priv->channel_stats[i].rq_xdpsq,
@@ -1356,11 +1460,16 @@ static int mlx5e_grp_channels_fill_stats(struct mlx5e_priv *priv, u64 *data,
 					MLX5E_READ_CTR64_CPU(&priv->channel_stats[i].sq[tc],
 							     sq_stats_desc, j);
 
-	for (i = 0; i < max_nch; i++)
+	for (i = 0; i < max_nch; i++) {
+		for (j = 0; j < NUM_XSKSQ_STATS * is_xsk; j++)
+			data[idx++] =
+				MLX5E_READ_CTR64_CPU(&priv->channel_stats[i].xsksq,
+						     xsksq_stats_desc, j);
 		for (j = 0; j < NUM_XDPSQ_STATS; j++)
 			data[idx++] =
 				MLX5E_READ_CTR64_CPU(&priv->channel_stats[i].xdpsq,
 						     xdpsq_stats_desc, j);
+	}
 
 	return idx;
 }
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
index cdddcc46971b..fb3ad7231e11 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
@@ -46,6 +46,8 @@
 #define MLX5E_DECLARE_TX_STAT(type, fld) "tx%d_"#fld, offsetof(type, fld)
 #define MLX5E_DECLARE_XDPSQ_STAT(type, fld) "tx%d_xdp_"#fld, offsetof(type, fld)
 #define MLX5E_DECLARE_RQ_XDPSQ_STAT(type, fld) "rx%d_xdp_tx_"#fld, offsetof(type, fld)
+#define MLX5E_DECLARE_XSKRQ_STAT(type, fld) "rx%d_xsk_"#fld, offsetof(type, fld)
+#define MLX5E_DECLARE_XSKSQ_STAT(type, fld) "tx%d_xsk_"#fld, offsetof(type, fld)
 #define MLX5E_DECLARE_CH_STAT(type, fld) "ch%d_"#fld, offsetof(type, fld)
 
 struct counter_desc {
@@ -116,12 +118,39 @@ struct mlx5e_sw_stats {
 	u64 ch_poll;
 	u64 ch_arm;
 	u64 ch_aff_change;
+	u64 ch_force_irq;
 	u64 ch_eq_rearm;
 
 #ifdef CONFIG_MLX5_EN_TLS
 	u64 tx_tls_ooo;
 	u64 tx_tls_resync_bytes;
 #endif
+
+	u64 rx_xsk_packets;
+	u64 rx_xsk_bytes;
+	u64 rx_xsk_csum_complete;
+	u64 rx_xsk_csum_unnecessary;
+	u64 rx_xsk_csum_unnecessary_inner;
+	u64 rx_xsk_csum_none;
+	u64 rx_xsk_ecn_mark;
+	u64 rx_xsk_removed_vlan_packets;
+	u64 rx_xsk_xdp_drop;
+	u64 rx_xsk_xdp_redirect;
+	u64 rx_xsk_wqe_err;
+	u64 rx_xsk_mpwqe_filler_cqes;
+	u64 rx_xsk_mpwqe_filler_strides;
+	u64 rx_xsk_oversize_pkts_sw_drop;
+	u64 rx_xsk_buff_alloc_err;
+	u64 rx_xsk_cqe_compress_blks;
+	u64 rx_xsk_cqe_compress_pkts;
+	u64 rx_xsk_congst_umr;
+	u64 rx_xsk_arfs_err;
+	u64 tx_xsk_xmit;
+	u64 tx_xsk_mpwqe;
+	u64 tx_xsk_inlnw;
+	u64 tx_xsk_full;
+	u64 tx_xsk_err;
+	u64 tx_xsk_cqes;
 };
 
 struct mlx5e_qcounter_stats {
@@ -256,6 +285,7 @@ struct mlx5e_ch_stats {
 	u64 poll;
 	u64 arm;
 	u64 aff_change;
+	u64 force_irq;
 	u64 eq_rearm;
 };
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index d2b8ce5df59c..9ae327e80d6d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -33,6 +33,7 @@
 #include <linux/irq.h>
 #include "en.h"
 #include "en/xdp.h"
+#include "en/xsk/tx.h"
 
 static inline bool mlx5e_channel_no_affinity_change(struct mlx5e_channel *c)
 {
@@ -87,7 +88,12 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 	struct mlx5e_channel *c = container_of(napi, struct mlx5e_channel,
 					       napi);
 	struct mlx5e_ch_stats *ch_stats = c->stats;
+	struct mlx5e_xdpsq *xsksq = &c->xsksq;
+	struct mlx5e_rq *xskrq = &c->xskrq;
 	struct mlx5e_rq *rq = &c->rq;
+	bool xsk_open = test_bit(MLX5E_CHANNEL_STATE_XSK, c->state);
+	bool aff_change = false;
+	bool busy_xsk = false;
 	bool busy = false;
 	int work_done = 0;
 	int i;
@@ -103,18 +109,32 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 		busy |= mlx5e_poll_xdpsq_cq(&c->rq_xdpsq.cq);
 
 	if (likely(budget)) { /* budget=0 means: don't poll rx rings */
-		work_done = mlx5e_poll_rx_cq(&rq->cq, budget);
+		if (xsk_open)
+			work_done = mlx5e_poll_rx_cq(&xskrq->cq, budget);
+
+		if (likely(budget - work_done))
+			work_done += mlx5e_poll_rx_cq(&rq->cq, budget - work_done);
+
 		busy |= work_done == budget;
 	}
 
 	mlx5e_poll_ico_cq(&c->icosq.cq);
 
 	busy |= rq->post_wqes(rq);
+	if (xsk_open) {
+		mlx5e_poll_ico_cq(&c->xskicosq.cq);
+		busy |= mlx5e_poll_xdpsq_cq(&xsksq->cq);
+		busy_xsk |= mlx5e_xsk_tx(xsksq, MLX5E_TX_XSK_POLL_BUDGET);
+		busy_xsk |= xskrq->post_wqes(xskrq);
+	}
+
+	busy |= busy_xsk;
 
 	if (busy) {
 		if (likely(mlx5e_channel_no_affinity_change(c)))
 			return budget;
 		ch_stats->aff_change++;
+		aff_change = true;
 		if (budget && work_done == budget)
 			work_done--;
 	}
@@ -135,6 +155,18 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
 	mlx5e_cq_arm(&c->icosq.cq);
 	mlx5e_cq_arm(&c->xdpsq.cq);
 
+	if (xsk_open) {
+		mlx5e_handle_rx_dim(xskrq);
+		mlx5e_cq_arm(&c->xskicosq.cq);
+		mlx5e_cq_arm(&xsksq->cq);
+		mlx5e_cq_arm(&xskrq->cq);
+	}
+
+	if (unlikely(aff_change && busy_xsk)) {
+		mlx5e_trigger_irq(&c->icosq);
+		ch_stats->force_irq++;
+	}
+
 	return work_done;
 }
 
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
index ada1b7c0e0b8..1019dfb46c8c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
@@ -87,7 +87,7 @@ int mlx5i_init(struct mlx5_core_dev *mdev,
 	mlx5e_set_netdev_mtu_boundaries(priv);
 	netdev->mtu = netdev->max_mtu;
 
-	mlx5e_build_nic_params(mdev, &priv->rss_params, &priv->channels.params,
+	mlx5e_build_nic_params(mdev, NULL, &priv->rss_params, &priv->channels.params,
 			       mlx5e_get_netdev_max_channels(netdev),
 			       netdev->mtu);
 	mlx5i_build_nic_params(mdev, &priv->channels.params);
@@ -365,7 +365,7 @@ static int mlx5i_init_rx(struct mlx5e_priv *priv)
 	if (err)
 		goto err_close_drop_rq;
 
-	err = mlx5e_create_direct_rqts(priv);
+	err = mlx5e_create_direct_rqts(priv, priv->direct_tir);
 	if (err)
 		goto err_destroy_indirect_rqts;
 
@@ -373,7 +373,7 @@ static int mlx5i_init_rx(struct mlx5e_priv *priv)
 	if (err)
 		goto err_destroy_direct_rqts;
 
-	err = mlx5e_create_direct_tirs(priv);
+	err = mlx5e_create_direct_tirs(priv, priv->direct_tir);
 	if (err)
 		goto err_destroy_indirect_tirs;
 
@@ -384,11 +384,11 @@ static int mlx5i_init_rx(struct mlx5e_priv *priv)
 	return 0;
 
 err_destroy_direct_tirs:
-	mlx5e_destroy_direct_tirs(priv);
+	mlx5e_destroy_direct_tirs(priv, priv->direct_tir);
 err_destroy_indirect_tirs:
 	mlx5e_destroy_indirect_tirs(priv, true);
 err_destroy_direct_rqts:
-	mlx5e_destroy_direct_rqts(priv);
+	mlx5e_destroy_direct_rqts(priv, priv->direct_tir);
 err_destroy_indirect_rqts:
 	mlx5e_destroy_rqt(priv, &priv->indir_rqt);
 err_close_drop_rq:
@@ -401,9 +401,9 @@ static int mlx5i_init_rx(struct mlx5e_priv *priv)
 static void mlx5i_cleanup_rx(struct mlx5e_priv *priv)
 {
 	mlx5i_destroy_flow_steering(priv);
-	mlx5e_destroy_direct_tirs(priv);
+	mlx5e_destroy_direct_tirs(priv, priv->direct_tir);
 	mlx5e_destroy_indirect_tirs(priv, true);
-	mlx5e_destroy_direct_rqts(priv);
+	mlx5e_destroy_direct_rqts(priv, priv->direct_tir);
 	mlx5e_destroy_rqt(priv, &priv->indir_rqt);
 	mlx5e_close_drop_rq(&priv->drop_rq);
 	mlx5e_destroy_q_counters(priv);
-- 
2.19.1

Re: [PATCH bpf-next v2 00/16] AF_XDP infrastructure improvements and mlx5e support

[Björn Töpel]

On Tue, 30 Apr 2019 at 20:12, Maxim Mikityanskiy <maximmi@mellanox.com> wrote:
>
> This series contains improvements to the AF_XDP kernel infrastructure
> and AF_XDP support in mlx5e. The infrastructure improvements are
> required for mlx5e, but also some of them benefit to all drivers, and
> some can be useful for other drivers that want to implement AF_XDP.
>
> The performance testing was performed on a machine with the following
> configuration:
>
> - 24 cores of Intel Xeon E5-2620 v3 @ 2.40 GHz
> - Mellanox ConnectX-5 Ex with 100 Gbit/s link
>
> The results with retpoline disabled, single stream:
>
> txonly: 33.3 Mpps (21.5 Mpps with queue and app pinned to the same CPU)
> rxdrop: 12.2 Mpps
> l2fwd: 9.4 Mpps
>
> The results with retpoline enabled, single stream:
>
> txonly: 21.3 Mpps (14.1 Mpps with queue and app pinned to the same CPU)
> rxdrop: 9.9 Mpps
> l2fwd: 6.8 Mpps
>
> v2 changes:
>
> Added patches for mlx5e and addressed the comments for v1. Rebased for
> bpf-next (net-next has to be merged first, because this series depends
> on some patches from there).
>

Nit: There're some checkpatch warnings (>80 char lines) for the driver parts.


> Maxim Mikityanskiy (16):
>   xsk: Add API to check for available entries in FQ
>   xsk: Add getsockopt XDP_OPTIONS
>   libbpf: Support getsockopt XDP_OPTIONS
>   xsk: Extend channels to support combined XSK/non-XSK traffic
>   xsk: Change the default frame size to 4096 and allow controlling it
>   xsk: Return the whole xdp_desc from xsk_umem_consume_tx
>   net/mlx5e: Replace deprecated PCI_DMA_TODEVICE
>   net/mlx5e: Calculate linear RX frag size considering XSK
>   net/mlx5e: Allow ICO SQ to be used by multiple RQs
>   net/mlx5e: Refactor struct mlx5e_xdp_info
>   net/mlx5e: Share the XDP SQ for XDP_TX between RQs
>   net/mlx5e: XDP_TX from UMEM support
>   net/mlx5e: Consider XSK in XDP MTU limit calculation
>   net/mlx5e: Encapsulate open/close queues into a function
>   net/mlx5e: Move queue param structs to en/params.h
>   net/mlx5e: Add XSK support
>
>  drivers/net/ethernet/intel/i40e/i40e_xsk.c    |  12 +-
>  drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c  |  15 +-
>  .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
>  drivers/net/ethernet/mellanox/mlx5/core/en.h  | 147 +++-
>  .../ethernet/mellanox/mlx5/core/en/params.c   | 108 ++-
>  .../ethernet/mellanox/mlx5/core/en/params.h   |  87 ++-
>  .../net/ethernet/mellanox/mlx5/core/en/xdp.c  | 231 ++++--
>  .../net/ethernet/mellanox/mlx5/core/en/xdp.h  |  36 +-
>  .../mellanox/mlx5/core/en/xsk/Makefile        |   1 +
>  .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   | 192 +++++
>  .../ethernet/mellanox/mlx5/core/en/xsk/rx.h   |  27 +
>  .../mellanox/mlx5/core/en/xsk/setup.c         | 220 ++++++
>  .../mellanox/mlx5/core/en/xsk/setup.h         |  25 +
>  .../ethernet/mellanox/mlx5/core/en/xsk/tx.c   | 108 +++
>  .../ethernet/mellanox/mlx5/core/en/xsk/tx.h   |  15 +
>  .../ethernet/mellanox/mlx5/core/en/xsk/umem.c | 252 +++++++
>  .../ethernet/mellanox/mlx5/core/en/xsk/umem.h |  34 +
>  .../ethernet/mellanox/mlx5/core/en_ethtool.c  |  21 +-
>  .../mellanox/mlx5/core/en_fs_ethtool.c        |  44 +-
>  .../net/ethernet/mellanox/mlx5/core/en_main.c | 680 +++++++++++-------
>  .../net/ethernet/mellanox/mlx5/core/en_rep.c  |  12 +-
>  .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 104 ++-
>  .../ethernet/mellanox/mlx5/core/en_stats.c    | 115 ++-
>  .../ethernet/mellanox/mlx5/core/en_stats.h    |  30 +
>  .../net/ethernet/mellanox/mlx5/core/en_txrx.c |  42 +-
>  .../ethernet/mellanox/mlx5/core/ipoib/ipoib.c |  14 +-
>  drivers/net/ethernet/mellanox/mlx5/core/wq.h  |   5 -
>  include/net/xdp_sock.h                        |  27 +-
>  include/uapi/linux/if_xdp.h                   |  18 +
>  net/xdp/xsk.c                                 |  43 +-
>  net/xdp/xsk_queue.h                           |  14 +
>  samples/bpf/xdpsock_user.c                    |  52 +-
>  tools/include/uapi/linux/if_xdp.h             |  18 +
>  tools/lib/bpf/xsk.c                           | 127 +++-
>  tools/lib/bpf/xsk.h                           |   6 +-
>  35 files changed, 2384 insertions(+), 500 deletions(-)
>  create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/Makefile
>  create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
>  create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.h
>  create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c
>  create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.h
>  create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c
>  create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.h
>  create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.c
>  create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/xsk/umem.h
>
> --
> 2.19.1
>