* [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization
@ 2017-03-24 21:52 Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 01/12] net/mlx5e: Use dma_rmb rather than rmb in CQE fetch routine Saeed Mahameed
` (14 more replies)
0 siblings, 15 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
Hi Dave,
This series provides some preformancee optimizations for mlx5e
driver, especially for XDP TX flows.
1st patch is a simple change of rmb to dma_rmb in CQE fetch routine
which shows a huge gain for both RX and TX packet rates.
2nd patch removes write combining logic from the driver TX handler
and simplifies the TX logic while improving TX CPU utilization.
All other patches combined provide some refactoring to the driver TX
flows to allow some significant XDP TX improvements.
More details and performance numbers per patch can be found in each patch
commit message compared to the preceding patch.
Overall performance improvemnets
System: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Test case Baseline Now improvement
---------------------------------------------------------------
TX packets (24 threads) 45Mpps 54Mpps 20%
TC stack Drop (1 core) 3.45Mpps 3.6Mpps 5%
XDP Drop (1 core) 14Mpps 16.9Mpps 20%
XDP TX (1 core) 10.4Mpps 13.7Mpps 31%
Thanks,
Saeed.
Saeed Mahameed (12):
net/mlx5e: Use dma_rmb rather than rmb in CQE fetch routine
net/mlx5e: Xmit, no write combining
net/mlx5e: Single bfreg (UAR) for all mlx5e SQs and netdevs
net/mlx5e: Move XDP completion functions to rx file
net/mlx5e: Move mlx5e_rq struct declaration
net/mlx5e: Move XDP SQ instance into RQ
net/mlx5e: Poll XDP TX CQ before RX CQ
net/mlx5e: Optimize XDP frame xmit
net/mlx5e: Generalize tx helper functions for different SQ types
net/mlx5e: Proper names for SQ/RQ/CQ functions
net/mlx5e: Generalize SQ create/modify/destroy functions
net/mlx5e: Different SQ types
drivers/net/ethernet/mellanox/mlx5/core/en.h | 319 +++++-----
.../net/ethernet/mellanox/mlx5/core/en_common.c | 9 +
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 644 +++++++++++++--------
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 124 +++-
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 147 +----
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 70 +--
include/linux/mlx5/driver.h | 1 +
7 files changed, 716 insertions(+), 598 deletions(-)
--
2.11.0
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH net-next 01/12] net/mlx5e: Use dma_rmb rather than rmb in CQE fetch routine
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 02/12] net/mlx5e: Xmit, no write combining Saeed Mahameed
` (13 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
Use dma_rmb in mlx5e_get_cqe rather than aggressive rmb (at least on
some architectures), this should help improve the performance on such
CPU archs where dma_rmb is optimized.
Performance improvement:
System: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Test case Baseline Now improvement
---------------------------------------------------------------
TX packets (24 threads) 45Mpps 50Mpps 11%
TC stack Drop (1 core) 3.45Mpps 3.6Mpps 5%
XDP Drop (1 core) 14Mpps 16.9Mpps 20%
XDP TX (1 core) 10.4Mpps 12Mpps 15%
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index e5c12a732aa1..d8cda2f6239b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -44,7 +44,7 @@ struct mlx5_cqe64 *mlx5e_get_cqe(struct mlx5e_cq *cq)
return NULL;
/* ensure cqe content is read after cqe ownership bit */
- rmb();
+ dma_rmb();
return cqe;
}
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 02/12] net/mlx5e: Xmit, no write combining
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 01/12] net/mlx5e: Use dma_rmb rather than rmb in CQE fetch routine Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 03/12] net/mlx5e: Single bfreg (UAR) for all mlx5e SQs and netdevs Saeed Mahameed
` (12 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
mlx5e netdev Blue Flame (write combining) support demands a lot of
overhead for a little latency gain for some special cases, this overhead
is hurting the common case.
Here we remove xmit Blue Flame support by creating all bfregs with no
write combining for all SQs, and we remove a lot of BF logic and
conditions from xmit data path.
Simplify mlx5e_tx_notify_hw (doorbell function) by removing BF related
code and by removing one memory barrier needed for WC mapped SQ doorbell
buffers, which no longer exist.
Performance improvement:
System: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Test case Before Now improvement
---------------------------------------------------------------
TX packets (24 threads) 50Mpps 54Mpps 8%
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 20 ++---------
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 6 +---
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 4 +--
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 42 ++---------------------
4 files changed, 9 insertions(+), 63 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index dc52053128bc..85261e9ccf4a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -111,7 +111,6 @@
#define MLX5E_MAX_NUM_SQS (MLX5E_MAX_NUM_CHANNELS * MLX5E_MAX_NUM_TC)
#define MLX5E_TX_CQ_POLL_BUDGET 128
#define MLX5E_UPDATE_STATS_INTERVAL 200 /* msecs */
-#define MLX5E_SQ_BF_BUDGET 16
#define MLX5E_ICOSQ_MAX_WQEBBS \
(DIV_ROUND_UP(sizeof(struct mlx5e_umr_wqe), MLX5_SEND_WQE_BB))
@@ -426,7 +425,6 @@ struct mlx5e_sq_dma {
enum {
MLX5E_SQ_STATE_ENABLED,
- MLX5E_SQ_STATE_BF_ENABLE,
};
struct mlx5e_sq_wqe_info {
@@ -450,9 +448,6 @@ struct mlx5e_sq {
/* dirtied @xmit */
u16 pc ____cacheline_aligned_in_smp;
u32 dma_fifo_pc;
- u16 bf_offset;
- u16 prev_cc;
- u8 bf_budget;
struct mlx5e_sq_stats stats;
struct mlx5e_cq cq;
@@ -478,7 +473,6 @@ struct mlx5e_sq {
void __iomem *uar_map;
struct netdev_queue *txq;
u32 sqn;
- u16 bf_buf_size;
u16 max_inline;
u8 min_inline_mode;
u16 edge;
@@ -818,11 +812,9 @@ void mlx5e_set_rx_cq_mode_params(struct mlx5e_params *params,
u8 cq_period_mode);
void mlx5e_set_rq_type_params(struct mlx5e_priv *priv, u8 rq_type);
-static inline void mlx5e_tx_notify_hw(struct mlx5e_sq *sq,
- struct mlx5_wqe_ctrl_seg *ctrl, int bf_sz)
+static inline void
+mlx5e_tx_notify_hw(struct mlx5e_sq *sq, struct mlx5_wqe_ctrl_seg *ctrl)
{
- u16 ofst = sq->bf_offset;
-
/* ensure wqe is visible to device before updating doorbell record */
dma_wmb();
@@ -832,14 +824,8 @@ static inline void mlx5e_tx_notify_hw(struct mlx5e_sq *sq,
* doorbell
*/
wmb();
- if (bf_sz)
- __iowrite64_copy(sq->uar_map + ofst, ctrl, bf_sz);
- else
- mlx5_write64((__be32 *)ctrl, sq->uar_map + ofst, NULL);
- /* flush the write-combining mapped buffer */
- wmb();
- sq->bf_offset ^= sq->bf_buf_size;
+ mlx5_write64((__be32 *)ctrl, sq->uar_map, NULL);
}
static inline void mlx5e_cq_arm(struct mlx5e_cq *cq)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index ddd7464c6b45..f9bcbd277adb 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1017,7 +1017,7 @@ static int mlx5e_create_sq(struct mlx5e_channel *c,
sq->channel = c;
sq->tc = tc;
- err = mlx5_alloc_bfreg(mdev, &sq->bfreg, MLX5_CAP_GEN(mdev, bf), false);
+ err = mlx5_alloc_bfreg(mdev, &sq->bfreg, false, false);
if (err)
return err;
@@ -1030,10 +1030,7 @@ static int mlx5e_create_sq(struct mlx5e_channel *c,
goto err_unmap_free_uar;
sq->wq.db = &sq->wq.db[MLX5_SND_DBR];
- if (sq->bfreg.wc)
- set_bit(MLX5E_SQ_STATE_BF_ENABLE, &sq->state);
- sq->bf_buf_size = (1 << MLX5_CAP_GEN(mdev, log_bf_reg_size)) / 2;
sq->max_inline = param->max_inline;
sq->min_inline_mode = param->min_inline_mode;
@@ -1050,7 +1047,6 @@ static int mlx5e_create_sq(struct mlx5e_channel *c,
}
sq->edge = (sq->wq.sz_m1 + 1) - mlx5e_sq_get_max_wqebbs(sq->type);
- sq->bf_budget = MLX5E_SQ_BF_BUDGET;
return 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index bafcb349a50c..873b3085756c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -353,7 +353,7 @@ static inline void mlx5e_post_umr_wqe(struct mlx5e_rq *rq, u16 ix)
sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_UMR;
sq->db.ico_wqe[pi].num_wqebbs = num_wqebbs;
sq->pc += num_wqebbs;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl, 0);
+ mlx5e_tx_notify_hw(sq, &wqe->ctrl);
}
static int mlx5e_alloc_rx_umr_mpwqe(struct mlx5e_rq *rq,
@@ -646,7 +646,7 @@ static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_sq *sq)
wqe = mlx5_wq_cyc_get_wqe(wq, pi);
wqe->ctrl.fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl, 0);
+ mlx5e_tx_notify_hw(sq, &wqe->ctrl);
}
static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index 57f5e2d7ebd1..eec4354208ee 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -57,7 +57,7 @@ void mlx5e_send_nop(struct mlx5e_sq *sq, bool notify_hw)
if (notify_hw) {
cseg->fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl, 0);
+ mlx5e_tx_notify_hw(sq, &wqe->ctrl);
}
}
@@ -175,25 +175,6 @@ static inline unsigned int mlx5e_calc_min_inline(enum mlx5_inline_modes mode,
}
}
-static inline u16 mlx5e_get_inline_hdr_size(struct mlx5e_sq *sq,
- struct sk_buff *skb, bool bf)
-{
- /* Some NIC TX decisions, e.g loopback, are based on the packet
- * headers and occur before the data gather.
- * Therefore these headers must be copied into the WQE
- */
- if (bf) {
- u16 ihs = skb_headlen(skb);
-
- if (skb_vlan_tag_present(skb))
- ihs += VLAN_HLEN;
-
- if (ihs <= sq->max_inline)
- return skb_headlen(skb);
- }
- return mlx5e_calc_min_inline(sq->min_inline_mode, skb);
-}
-
static inline void mlx5e_tx_skb_pull_inline(unsigned char **skb_data,
unsigned int *skb_len,
unsigned int len)
@@ -235,7 +216,6 @@ static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
u8 opcode = MLX5_OPCODE_SEND;
dma_addr_t dma_addr = 0;
unsigned int num_bytes;
- bool bf = false;
u16 headlen;
u16 ds_cnt;
u16 ihs;
@@ -255,11 +235,6 @@ static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
} else
sq->stats.csum_none++;
- if (sq->cc != sq->prev_cc) {
- sq->prev_cc = sq->cc;
- sq->bf_budget = (sq->cc == sq->pc) ? MLX5E_SQ_BF_BUDGET : 0;
- }
-
if (skb_is_gso(skb)) {
eseg->mss = cpu_to_be16(skb_shinfo(skb)->gso_size);
opcode = MLX5_OPCODE_LSO;
@@ -277,10 +252,7 @@ static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
sq->stats.packets += skb_shinfo(skb)->gso_segs;
num_bytes = skb->len + (skb_shinfo(skb)->gso_segs - 1) * ihs;
} else {
- bf = sq->bf_budget &&
- !skb->xmit_more &&
- !skb_shinfo(skb)->nr_frags;
- ihs = mlx5e_get_inline_hdr_size(sq, skb, bf);
+ ihs = mlx5e_calc_min_inline(sq->min_inline_mode, skb);
sq->stats.packets++;
num_bytes = max_t(unsigned int, skb->len, ETH_ZLEN);
}
@@ -366,13 +338,8 @@ static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
sq->stats.xmit_more += skb->xmit_more;
if (!skb->xmit_more || netif_xmit_stopped(sq->txq)) {
- int bf_sz = 0;
-
- if (bf && test_bit(MLX5E_SQ_STATE_BF_ENABLE, &sq->state))
- bf_sz = wi->num_wqebbs << 3;
-
cseg->fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl, bf_sz);
+ mlx5e_tx_notify_hw(sq, &wqe->ctrl);
}
/* fill sq edge with nops to avoid wqe wrap around */
@@ -381,9 +348,6 @@ static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
mlx5e_send_nop(sq, false);
}
- if (bf)
- sq->bf_budget--;
-
return NETDEV_TX_OK;
dma_unmap_wqe_err:
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 03/12] net/mlx5e: Single bfreg (UAR) for all mlx5e SQs and netdevs
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 01/12] net/mlx5e: Use dma_rmb rather than rmb in CQE fetch routine Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 02/12] net/mlx5e: Xmit, no write combining Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 04/12] net/mlx5e: Move XDP completion functions to rx file Saeed Mahameed
` (11 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
One is sufficient since Blue Flame is not supported anymore.
This will also come in handy for switchdev mode to save resources, since
VF representors will use same single UAR as well for their own SQs.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 1 -
drivers/net/ethernet/mellanox/mlx5/core/en_common.c | 9 +++++++++
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 17 +++--------------
include/linux/mlx5/driver.h | 1 +
4 files changed, 13 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 85261e9ccf4a..22e4bad03f05 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -483,7 +483,6 @@ struct mlx5e_sq {
/* control path */
struct mlx5_wq_ctrl wq_ctrl;
- struct mlx5_sq_bfreg bfreg;
struct mlx5e_channel *channel;
int tc;
u32 rate_limit;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
index bd898d8deda0..20bdbe685795 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_common.c
@@ -107,10 +107,18 @@ int mlx5e_create_mdev_resources(struct mlx5_core_dev *mdev)
goto err_dealloc_transport_domain;
}
+ err = mlx5_alloc_bfreg(mdev, &res->bfreg, false, false);
+ if (err) {
+ mlx5_core_err(mdev, "alloc bfreg failed, %d\n", err);
+ goto err_destroy_mkey;
+ }
+
INIT_LIST_HEAD(&mdev->mlx5e_res.td.tirs_list);
return 0;
+err_destroy_mkey:
+ mlx5_core_destroy_mkey(mdev, &res->mkey);
err_dealloc_transport_domain:
mlx5_core_dealloc_transport_domain(mdev, res->td.tdn);
err_dealloc_pd:
@@ -122,6 +130,7 @@ void mlx5e_destroy_mdev_resources(struct mlx5_core_dev *mdev)
{
struct mlx5e_resources *res = &mdev->mlx5e_res;
+ mlx5_free_bfreg(mdev, &res->bfreg);
mlx5_core_destroy_mkey(mdev, &res->mkey);
mlx5_core_dealloc_transport_domain(mdev, res->td.tdn);
mlx5_core_dealloc_pd(mdev, res->pdn);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index f9bcbd277adb..49c1769d13b9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1016,18 +1016,14 @@ static int mlx5e_create_sq(struct mlx5e_channel *c,
sq->mkey_be = c->mkey_be;
sq->channel = c;
sq->tc = tc;
+ sq->uar_map = mdev->mlx5e_res.bfreg.map;
- err = mlx5_alloc_bfreg(mdev, &sq->bfreg, false, false);
- if (err)
- return err;
-
- sq->uar_map = sq->bfreg.map;
param->wq.db_numa_node = cpu_to_node(c->cpu);
err = mlx5_wq_cyc_create(mdev, ¶m->wq, sqc_wq, &sq->wq,
&sq->wq_ctrl);
if (err)
- goto err_unmap_free_uar;
+ return err;
sq->wq.db = &sq->wq.db[MLX5_SND_DBR];
@@ -1053,20 +1049,13 @@ static int mlx5e_create_sq(struct mlx5e_channel *c,
err_sq_wq_destroy:
mlx5_wq_destroy(&sq->wq_ctrl);
-err_unmap_free_uar:
- mlx5_free_bfreg(mdev, &sq->bfreg);
-
return err;
}
static void mlx5e_destroy_sq(struct mlx5e_sq *sq)
{
- struct mlx5e_channel *c = sq->channel;
- struct mlx5e_priv *priv = c->priv;
-
mlx5e_free_sq_db(sq);
mlx5_wq_destroy(&sq->wq_ctrl);
- mlx5_free_bfreg(priv->mdev, &sq->bfreg);
}
static int mlx5e_enable_sq(struct mlx5e_sq *sq, struct mlx5e_sq_param *param)
@@ -1103,7 +1092,7 @@ static int mlx5e_enable_sq(struct mlx5e_sq *sq, struct mlx5e_sq_param *param)
MLX5_SET(sqc, sqc, tis_lst_sz, param->type == MLX5E_SQ_ICO ? 0 : 1);
MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_CYCLIC);
- MLX5_SET(wq, wq, uar_page, sq->bfreg.index);
+ MLX5_SET(wq, wq, uar_page, mdev->mlx5e_res.bfreg.index);
MLX5_SET(wq, wq, log_wq_pg_sz, sq->wq_ctrl.buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
MLX5_SET64(wq, wq, dbr_addr, sq->wq_ctrl.db.dma);
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 2fcff6b4503f..f50864626230 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -728,6 +728,7 @@ struct mlx5e_resources {
u32 pdn;
struct mlx5_td td;
struct mlx5_core_mkey mkey;
+ struct mlx5_sq_bfreg bfreg;
};
struct mlx5_core_dev {
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 04/12] net/mlx5e: Move XDP completion functions to rx file
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (2 preceding siblings ...)
2017-03-24 21:52 ` [PATCH net-next 03/12] net/mlx5e: Single bfreg (UAR) for all mlx5e SQs and netdevs Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 05/12] net/mlx5e: Move mlx5e_rq struct declaration Saeed Mahameed
` (10 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
XDP code belongs to RX path, move mlx5e_poll_xdp_tx_cq and
mlx5e_free_xdp_tx_descs to en_rx.c.
Rename them to mlx5e_poll_xdpsq_cq and mlx5e_free_xdpsq_descs.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 +
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 82 +++++++++++++++++++++++
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 24 +------
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 62 +----------------
4 files changed, 86 insertions(+), 84 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 22e4bad03f05..fce0eca0701c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -737,6 +737,8 @@ void mlx5e_cq_error_event(struct mlx5_core_cq *mcq, enum mlx5_event event);
int mlx5e_napi_poll(struct napi_struct *napi, int budget);
bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget);
int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget);
+bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq);
+void mlx5e_free_xdpsq_descs(struct mlx5e_sq *sq);
void mlx5e_free_sq_descs(struct mlx5e_sq *sq);
void mlx5e_page_release(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 873b3085756c..bc74d6032a5c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -989,3 +989,85 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)
return work_done;
}
+
+bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
+{
+ struct mlx5e_sq *sq;
+ u16 sqcc;
+ int i;
+
+ sq = container_of(cq, struct mlx5e_sq, cq);
+
+ if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &sq->state)))
+ return false;
+
+ /* sq->cc must be updated only after mlx5_cqwq_update_db_record(),
+ * otherwise a cq overrun may occur
+ */
+ sqcc = sq->cc;
+
+ for (i = 0; i < MLX5E_TX_CQ_POLL_BUDGET; i++) {
+ struct mlx5_cqe64 *cqe;
+ u16 wqe_counter;
+ bool last_wqe;
+
+ cqe = mlx5e_get_cqe(cq);
+ if (!cqe)
+ break;
+
+ mlx5_cqwq_pop(&cq->wq);
+
+ wqe_counter = be16_to_cpu(cqe->wqe_counter);
+
+ do {
+ struct mlx5e_sq_wqe_info *wi;
+ struct mlx5e_dma_info *di;
+ u16 ci;
+
+ last_wqe = (sqcc == wqe_counter);
+
+ ci = sqcc & sq->wq.sz_m1;
+ di = &sq->db.xdp.di[ci];
+ wi = &sq->db.xdp.wqe_info[ci];
+
+ if (unlikely(wi->opcode == MLX5_OPCODE_NOP)) {
+ sqcc++;
+ continue;
+ }
+
+ sqcc += wi->num_wqebbs;
+ /* Recycle RX page */
+ mlx5e_page_release(&sq->channel->rq, di, true);
+ } while (!last_wqe);
+ }
+
+ mlx5_cqwq_update_db_record(&cq->wq);
+
+ /* ensure cq space is freed before enabling more cqes */
+ wmb();
+
+ sq->cc = sqcc;
+ return (i == MLX5E_TX_CQ_POLL_BUDGET);
+}
+
+void mlx5e_free_xdpsq_descs(struct mlx5e_sq *sq)
+{
+ struct mlx5e_sq_wqe_info *wi;
+ struct mlx5e_dma_info *di;
+ u16 ci;
+
+ while (sq->cc != sq->pc) {
+ ci = sq->cc & sq->wq.sz_m1;
+ di = &sq->db.xdp.di[ci];
+ wi = &sq->db.xdp.wqe_info[ci];
+
+ if (wi->opcode == MLX5_OPCODE_NOP) {
+ sq->cc++;
+ continue;
+ }
+
+ sq->cc += wi->num_wqebbs;
+
+ mlx5e_page_release(&sq->channel->rq, di, false);
+ }
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index eec4354208ee..7497b6ac4382 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -493,28 +493,6 @@ static void mlx5e_free_txq_sq_descs(struct mlx5e_sq *sq)
}
}
-static void mlx5e_free_xdp_sq_descs(struct mlx5e_sq *sq)
-{
- struct mlx5e_sq_wqe_info *wi;
- struct mlx5e_dma_info *di;
- u16 ci;
-
- while (sq->cc != sq->pc) {
- ci = sq->cc & sq->wq.sz_m1;
- di = &sq->db.xdp.di[ci];
- wi = &sq->db.xdp.wqe_info[ci];
-
- if (wi->opcode == MLX5_OPCODE_NOP) {
- sq->cc++;
- continue;
- }
-
- sq->cc += wi->num_wqebbs;
-
- mlx5e_page_release(&sq->channel->rq, di, false);
- }
-}
-
void mlx5e_free_sq_descs(struct mlx5e_sq *sq)
{
switch (sq->type) {
@@ -522,7 +500,7 @@ void mlx5e_free_sq_descs(struct mlx5e_sq *sq)
mlx5e_free_txq_sq_descs(sq);
break;
case MLX5E_SQ_XDP:
- mlx5e_free_xdp_sq_descs(sq);
+ mlx5e_free_xdpsq_descs(sq);
break;
}
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index d8cda2f6239b..f23dedc58175 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -105,66 +105,6 @@ static void mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
sq->cc = sqcc;
}
-static inline bool mlx5e_poll_xdp_tx_cq(struct mlx5e_cq *cq)
-{
- struct mlx5e_sq *sq;
- u16 sqcc;
- int i;
-
- sq = container_of(cq, struct mlx5e_sq, cq);
-
- if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &sq->state)))
- return false;
-
- /* sq->cc must be updated only after mlx5_cqwq_update_db_record(),
- * otherwise a cq overrun may occur
- */
- sqcc = sq->cc;
-
- for (i = 0; i < MLX5E_TX_CQ_POLL_BUDGET; i++) {
- struct mlx5_cqe64 *cqe;
- u16 wqe_counter;
- bool last_wqe;
-
- cqe = mlx5e_get_cqe(cq);
- if (!cqe)
- break;
-
- mlx5_cqwq_pop(&cq->wq);
-
- wqe_counter = be16_to_cpu(cqe->wqe_counter);
-
- do {
- struct mlx5e_sq_wqe_info *wi;
- struct mlx5e_dma_info *di;
- u16 ci;
-
- last_wqe = (sqcc == wqe_counter);
-
- ci = sqcc & sq->wq.sz_m1;
- di = &sq->db.xdp.di[ci];
- wi = &sq->db.xdp.wqe_info[ci];
-
- if (unlikely(wi->opcode == MLX5_OPCODE_NOP)) {
- sqcc++;
- continue;
- }
-
- sqcc += wi->num_wqebbs;
- /* Recycle RX page */
- mlx5e_page_release(&sq->channel->rq, di, true);
- } while (!last_wqe);
- }
-
- mlx5_cqwq_update_db_record(&cq->wq);
-
- /* ensure cq space is freed before enabling more cqes */
- wmb();
-
- sq->cc = sqcc;
- return (i == MLX5E_TX_CQ_POLL_BUDGET);
-}
-
int mlx5e_napi_poll(struct napi_struct *napi, int budget)
{
struct mlx5e_channel *c = container_of(napi, struct mlx5e_channel,
@@ -182,7 +122,7 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
busy |= work_done == budget;
if (c->xdp)
- busy |= mlx5e_poll_xdp_tx_cq(&c->xdp_sq.cq);
+ busy |= mlx5e_poll_xdpsq_cq(&c->xdp_sq.cq);
mlx5e_poll_ico_cq(&c->icosq.cq);
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 05/12] net/mlx5e: Move mlx5e_rq struct declaration
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (3 preceding siblings ...)
2017-03-24 21:52 ` [PATCH net-next 04/12] net/mlx5e: Move XDP completion functions to rx file Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 06/12] net/mlx5e: Move XDP SQ instance into RQ Saeed Mahameed
` (9 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
Move struct mlx5e_rq and friends to appear after mlx5e_sq declaration in
en.h.
We will need this for next patch to move the mlx5e_sq instance into
mlx5e_rq struct for XDP SQs.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 213 +++++++++++++--------------
1 file changed, 105 insertions(+), 108 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index fce0eca0701c..8d789a25a1c0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -297,19 +297,113 @@ struct mlx5e_cq {
struct mlx5_frag_wq_ctrl wq_ctrl;
} ____cacheline_aligned_in_smp;
-struct mlx5e_rq;
-typedef void (*mlx5e_fp_handle_rx_cqe)(struct mlx5e_rq *rq,
- struct mlx5_cqe64 *cqe);
-typedef int (*mlx5e_fp_alloc_wqe)(struct mlx5e_rq *rq, struct mlx5e_rx_wqe *wqe,
- u16 ix);
+struct mlx5e_tx_wqe_info {
+ u32 num_bytes;
+ u8 num_wqebbs;
+ u8 num_dma;
+};
+
+enum mlx5e_dma_map_type {
+ MLX5E_DMA_MAP_SINGLE,
+ MLX5E_DMA_MAP_PAGE
+};
+
+struct mlx5e_sq_dma {
+ dma_addr_t addr;
+ u32 size;
+ enum mlx5e_dma_map_type type;
+};
+
+enum {
+ MLX5E_SQ_STATE_ENABLED,
+};
+
+struct mlx5e_sq_wqe_info {
+ u8 opcode;
+ u8 num_wqebbs;
+};
-typedef void (*mlx5e_fp_dealloc_wqe)(struct mlx5e_rq *rq, u16 ix);
+enum mlx5e_sq_type {
+ MLX5E_SQ_TXQ,
+ MLX5E_SQ_ICO,
+ MLX5E_SQ_XDP
+};
+
+struct mlx5e_sq {
+ /* data path */
+
+ /* dirtied @completion */
+ u16 cc;
+ u32 dma_fifo_cc;
+
+ /* dirtied @xmit */
+ u16 pc ____cacheline_aligned_in_smp;
+ u32 dma_fifo_pc;
+ struct mlx5e_sq_stats stats;
+
+ struct mlx5e_cq cq;
+
+ /* pointers to per tx element info: write@xmit, read@completion */
+ union {
+ struct {
+ struct sk_buff **skb;
+ struct mlx5e_sq_dma *dma_fifo;
+ struct mlx5e_tx_wqe_info *wqe_info;
+ } txq;
+ struct mlx5e_sq_wqe_info *ico_wqe;
+ struct {
+ struct mlx5e_sq_wqe_info *wqe_info;
+ struct mlx5e_dma_info *di;
+ bool doorbell;
+ } xdp;
+ } db;
+
+ /* read only */
+ struct mlx5_wq_cyc wq;
+ u32 dma_fifo_mask;
+ void __iomem *uar_map;
+ struct netdev_queue *txq;
+ u32 sqn;
+ u16 max_inline;
+ u8 min_inline_mode;
+ u16 edge;
+ struct device *pdev;
+ struct mlx5e_tstamp *tstamp;
+ __be32 mkey_be;
+ unsigned long state;
+
+ /* control path */
+ struct mlx5_wq_ctrl wq_ctrl;
+ struct mlx5e_channel *channel;
+ int tc;
+ u32 rate_limit;
+ u8 type;
+} ____cacheline_aligned_in_smp;
+
+static inline bool mlx5e_sq_has_room_for(struct mlx5e_sq *sq, u16 n)
+{
+ return (((sq->wq.sz_m1 & (sq->cc - sq->pc)) >= n) ||
+ (sq->cc == sq->pc));
+}
struct mlx5e_dma_info {
struct page *page;
dma_addr_t addr;
};
+struct mlx5e_umr_dma_info {
+ __be64 *mtt;
+ dma_addr_t mtt_addr;
+ struct mlx5e_dma_info dma_info[MLX5_MPWRQ_PAGES_PER_WQE];
+ struct mlx5e_umr_wqe wqe;
+};
+
+struct mlx5e_mpw_info {
+ struct mlx5e_umr_dma_info umr;
+ u16 consumed_strides;
+ u16 skbs_frags[MLX5_MPWRQ_PAGES_PER_WQE];
+};
+
struct mlx5e_rx_am_stats {
int ppms; /* packets per msec */
int epms; /* events per msec */
@@ -346,6 +440,11 @@ struct mlx5e_page_cache {
struct mlx5e_dma_info page_cache[MLX5E_CACHE_SIZE];
};
+struct mlx5e_rq;
+typedef void (*mlx5e_fp_handle_rx_cqe)(struct mlx5e_rq*, struct mlx5_cqe64*);
+typedef int (*mlx5e_fp_alloc_wqe)(struct mlx5e_rq*, struct mlx5e_rx_wqe*, u16);
+typedef void (*mlx5e_fp_dealloc_wqe)(struct mlx5e_rq*, u16);
+
struct mlx5e_rq {
/* data path */
struct mlx5_wq_ll wq;
@@ -393,108 +492,6 @@ struct mlx5e_rq {
struct mlx5_core_mkey umr_mkey;
} ____cacheline_aligned_in_smp;
-struct mlx5e_umr_dma_info {
- __be64 *mtt;
- dma_addr_t mtt_addr;
- struct mlx5e_dma_info dma_info[MLX5_MPWRQ_PAGES_PER_WQE];
- struct mlx5e_umr_wqe wqe;
-};
-
-struct mlx5e_mpw_info {
- struct mlx5e_umr_dma_info umr;
- u16 consumed_strides;
- u16 skbs_frags[MLX5_MPWRQ_PAGES_PER_WQE];
-};
-
-struct mlx5e_tx_wqe_info {
- u32 num_bytes;
- u8 num_wqebbs;
- u8 num_dma;
-};
-
-enum mlx5e_dma_map_type {
- MLX5E_DMA_MAP_SINGLE,
- MLX5E_DMA_MAP_PAGE
-};
-
-struct mlx5e_sq_dma {
- dma_addr_t addr;
- u32 size;
- enum mlx5e_dma_map_type type;
-};
-
-enum {
- MLX5E_SQ_STATE_ENABLED,
-};
-
-struct mlx5e_sq_wqe_info {
- u8 opcode;
- u8 num_wqebbs;
-};
-
-enum mlx5e_sq_type {
- MLX5E_SQ_TXQ,
- MLX5E_SQ_ICO,
- MLX5E_SQ_XDP
-};
-
-struct mlx5e_sq {
- /* data path */
-
- /* dirtied @completion */
- u16 cc;
- u32 dma_fifo_cc;
-
- /* dirtied @xmit */
- u16 pc ____cacheline_aligned_in_smp;
- u32 dma_fifo_pc;
- struct mlx5e_sq_stats stats;
-
- struct mlx5e_cq cq;
-
- /* pointers to per tx element info: write@xmit, read@completion */
- union {
- struct {
- struct sk_buff **skb;
- struct mlx5e_sq_dma *dma_fifo;
- struct mlx5e_tx_wqe_info *wqe_info;
- } txq;
- struct mlx5e_sq_wqe_info *ico_wqe;
- struct {
- struct mlx5e_sq_wqe_info *wqe_info;
- struct mlx5e_dma_info *di;
- bool doorbell;
- } xdp;
- } db;
-
- /* read only */
- struct mlx5_wq_cyc wq;
- u32 dma_fifo_mask;
- void __iomem *uar_map;
- struct netdev_queue *txq;
- u32 sqn;
- u16 max_inline;
- u8 min_inline_mode;
- u16 edge;
- struct device *pdev;
- struct mlx5e_tstamp *tstamp;
- __be32 mkey_be;
- unsigned long state;
-
- /* control path */
- struct mlx5_wq_ctrl wq_ctrl;
- struct mlx5e_channel *channel;
- int tc;
- u32 rate_limit;
- u8 type;
-} ____cacheline_aligned_in_smp;
-
-static inline bool mlx5e_sq_has_room_for(struct mlx5e_sq *sq, u16 n)
-{
- return (((sq->wq.sz_m1 & (sq->cc - sq->pc)) >= n) ||
- (sq->cc == sq->pc));
-}
-
enum channel_flags {
MLX5E_CHANNEL_NAPI_SCHED = 1,
};
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 06/12] net/mlx5e: Move XDP SQ instance into RQ
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (4 preceding siblings ...)
2017-03-24 21:52 ` [PATCH net-next 05/12] net/mlx5e: Move mlx5e_rq struct declaration Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 07/12] net/mlx5e: Poll XDP TX CQ before RX CQ Saeed Mahameed
` (8 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
To save many rq->channel->sq dereferences in fast-path.
And rename it to xdpsq.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 +++-
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 12 ++++++------
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 18 +++++++++++-------
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 2 +-
4 files changed, 21 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 8d789a25a1c0..5e4ae94c9f6a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -479,7 +479,10 @@ struct mlx5e_rq {
u16 rx_headroom;
struct mlx5e_rx_am am; /* Adaptive Moderation */
+
+ /* XDP */
struct bpf_prog *xdp_prog;
+ struct mlx5e_sq xdpsq;
/* control */
struct mlx5_wq_ctrl wq_ctrl;
@@ -499,7 +502,6 @@ enum channel_flags {
struct mlx5e_channel {
/* data path */
struct mlx5e_rq rq;
- struct mlx5e_sq xdp_sq;
struct mlx5e_sq sq[MLX5E_MAX_NUM_TC];
struct mlx5e_sq icosq; /* internal control operations */
bool xdp;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 49c1769d13b9..210033187bfe 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1562,7 +1562,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
goto err_close_tx_cqs;
/* XDP SQ CQ params are same as normal TXQ sq CQ params */
- err = c->xdp ? mlx5e_open_cq(c, &cparam->tx_cq, &c->xdp_sq.cq,
+ err = c->xdp ? mlx5e_open_cq(c, &cparam->tx_cq, &c->rq.xdpsq.cq,
priv->params.tx_cq_moderation) : 0;
if (err)
goto err_close_rx_cq;
@@ -1587,7 +1587,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
}
}
- err = c->xdp ? mlx5e_open_sq(c, 0, &cparam->xdp_sq, &c->xdp_sq) : 0;
+ err = c->xdp ? mlx5e_open_sq(c, 0, &cparam->xdp_sq, &c->rq.xdpsq) : 0;
if (err)
goto err_close_sqs;
@@ -1601,7 +1601,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
return 0;
err_close_xdp_sq:
if (c->xdp)
- mlx5e_close_sq(&c->xdp_sq);
+ mlx5e_close_sq(&c->rq.xdpsq);
err_close_sqs:
mlx5e_close_sqs(c);
@@ -1612,7 +1612,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
err_disable_napi:
napi_disable(&c->napi);
if (c->xdp)
- mlx5e_close_cq(&c->xdp_sq.cq);
+ mlx5e_close_cq(&c->rq.xdpsq.cq);
err_close_rx_cq:
mlx5e_close_cq(&c->rq.cq);
@@ -1634,12 +1634,12 @@ static void mlx5e_close_channel(struct mlx5e_channel *c)
{
mlx5e_close_rq(&c->rq);
if (c->xdp)
- mlx5e_close_sq(&c->xdp_sq);
+ mlx5e_close_sq(&c->rq.xdpsq);
mlx5e_close_sqs(c);
mlx5e_close_sq(&c->icosq);
napi_disable(&c->napi);
if (c->xdp)
- mlx5e_close_cq(&c->xdp_sq.cq);
+ mlx5e_close_cq(&c->rq.xdpsq.cq);
mlx5e_close_cq(&c->rq.cq);
mlx5e_close_tx_cqs(c);
mlx5e_close_cq(&c->icosq.cq);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index bc74d6032a5c..040074f36313 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -653,7 +653,7 @@ static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
struct mlx5e_dma_info *di,
const struct xdp_buff *xdp)
{
- struct mlx5e_sq *sq = &rq->channel->xdp_sq;
+ struct mlx5e_sq *sq = &rq->xdpsq;
struct mlx5_wq_cyc *wq = &sq->wq;
u16 pi = sq->pc & wq->sz_m1;
struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
@@ -950,7 +950,7 @@ void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)
{
struct mlx5e_rq *rq = container_of(cq, struct mlx5e_rq, cq);
- struct mlx5e_sq *xdp_sq = &rq->channel->xdp_sq;
+ struct mlx5e_sq *xdpsq = &rq->xdpsq;
int work_done = 0;
if (unlikely(!test_bit(MLX5E_RQ_STATE_ENABLED, &rq->state)))
@@ -977,9 +977,9 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)
rq->handle_rx_cqe(rq, cqe);
}
- if (xdp_sq->db.xdp.doorbell) {
- mlx5e_xmit_xdp_doorbell(xdp_sq);
- xdp_sq->db.xdp.doorbell = false;
+ if (xdpsq->db.xdp.doorbell) {
+ mlx5e_xmit_xdp_doorbell(xdpsq);
+ xdpsq->db.xdp.doorbell = false;
}
mlx5_cqwq_update_db_record(&cq->wq);
@@ -993,6 +993,7 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)
bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
{
struct mlx5e_sq *sq;
+ struct mlx5e_rq *rq;
u16 sqcc;
int i;
@@ -1001,6 +1002,8 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &sq->state)))
return false;
+ rq = container_of(sq, struct mlx5e_rq, xdpsq);
+
/* sq->cc must be updated only after mlx5_cqwq_update_db_record(),
* otherwise a cq overrun may occur
*/
@@ -1037,7 +1040,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
sqcc += wi->num_wqebbs;
/* Recycle RX page */
- mlx5e_page_release(&sq->channel->rq, di, true);
+ mlx5e_page_release(rq, di, true);
} while (!last_wqe);
}
@@ -1052,6 +1055,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
void mlx5e_free_xdpsq_descs(struct mlx5e_sq *sq)
{
+ struct mlx5e_rq *rq = container_of(sq, struct mlx5e_rq, xdpsq);
struct mlx5e_sq_wqe_info *wi;
struct mlx5e_dma_info *di;
u16 ci;
@@ -1068,6 +1072,6 @@ void mlx5e_free_xdpsq_descs(struct mlx5e_sq *sq)
sq->cc += wi->num_wqebbs;
- mlx5e_page_release(&sq->channel->rq, di, false);
+ mlx5e_page_release(rq, di, false);
}
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index f23dedc58175..9beeb4a1212f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -122,7 +122,7 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
busy |= work_done == budget;
if (c->xdp)
- busy |= mlx5e_poll_xdpsq_cq(&c->xdp_sq.cq);
+ busy |= mlx5e_poll_xdpsq_cq(&c->rq.xdpsq.cq);
mlx5e_poll_ico_cq(&c->icosq.cq);
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 07/12] net/mlx5e: Poll XDP TX CQ before RX CQ
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (5 preceding siblings ...)
2017-03-24 21:52 ` [PATCH net-next 06/12] net/mlx5e: Move XDP SQ instance into RQ Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 08/12] net/mlx5e: Optimize XDP frame xmit Saeed Mahameed
` (7 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
Handle XDP TX completions before handling RX packets, to make sure more
free space is available for XDP TX packets a moment before handling
RX packets.
Performance improvement:
System: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Test case Before Now improvement
---------------------------------------------------------------
XDP Drop (1 core) 16.9Mpps 16.9Mpps No change
XDP TX (1 core) 12Mpps 13Mpps 8%
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index 9beeb4a1212f..c880022bb21a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -118,12 +118,12 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
for (i = 0; i < c->num_tc; i++)
busy |= mlx5e_poll_tx_cq(&c->sq[i].cq, budget);
- work_done = mlx5e_poll_rx_cq(&c->rq.cq, budget);
- busy |= work_done == budget;
-
if (c->xdp)
busy |= mlx5e_poll_xdpsq_cq(&c->rq.xdpsq.cq);
+ work_done = mlx5e_poll_rx_cq(&c->rq.cq, budget);
+ busy |= work_done == budget;
+
mlx5e_poll_ico_cq(&c->icosq.cq);
busy |= mlx5e_post_rx_wqes(&c->rq);
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 08/12] net/mlx5e: Optimize XDP frame xmit
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (6 preceding siblings ...)
2017-03-24 21:52 ` [PATCH net-next 07/12] net/mlx5e: Poll XDP TX CQ before RX CQ Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 09/12] net/mlx5e: Generalize tx helper functions for different SQ types Saeed Mahameed
` (6 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
XDP SQ has a fixed size WQE (MLX5E_XDP_TX_WQEBBS = 1) and only posts
one kind of WQE (MLX5_OPCODE_SEND),
Also we initialize SQ descriptors static fields once on open_xdpsq,
rather than every time on critical path.
Optimize the code in light of those facts and add a prefetch of the TX
descriptor first thing in the xdp xmit function.
Performance improvement:
System: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Test case Before Now improvement
---------------------------------------------------------------
XDP TX (1 core) 13Mpps 13.7Mpps 5%
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 5 ---
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 43 +++++++++++++++++++----
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 41 ++++++---------------
3 files changed, 47 insertions(+), 42 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 5e4ae94c9f6a..f02d2cb8d148 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -116,12 +116,8 @@
(DIV_ROUND_UP(sizeof(struct mlx5e_umr_wqe), MLX5_SEND_WQE_BB))
#define MLX5E_XDP_MIN_INLINE (ETH_HLEN + VLAN_HLEN)
-#define MLX5E_XDP_IHS_DS_COUNT \
- DIV_ROUND_UP(MLX5E_XDP_MIN_INLINE - 2, MLX5_SEND_WQE_DS)
#define MLX5E_XDP_TX_DS_COUNT \
((sizeof(struct mlx5e_tx_wqe) / MLX5_SEND_WQE_DS) + 1 /* SG DS */)
-#define MLX5E_XDP_TX_WQEBBS \
- DIV_ROUND_UP(MLX5E_XDP_TX_DS_COUNT, MLX5_SEND_WQEBB_NUM_DS)
#define MLX5E_NUM_MAIN_GROUPS 9
@@ -352,7 +348,6 @@ struct mlx5e_sq {
} txq;
struct mlx5e_sq_wqe_info *ico_wqe;
struct {
- struct mlx5e_sq_wqe_info *wqe_info;
struct mlx5e_dma_info *di;
bool doorbell;
} xdp;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 210033187bfe..d39ee6669b8e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -894,7 +894,6 @@ static void mlx5e_close_rq(struct mlx5e_rq *rq)
static void mlx5e_free_sq_xdp_db(struct mlx5e_sq *sq)
{
kfree(sq->db.xdp.di);
- kfree(sq->db.xdp.wqe_info);
}
static int mlx5e_alloc_sq_xdp_db(struct mlx5e_sq *sq, int numa)
@@ -903,9 +902,7 @@ static int mlx5e_alloc_sq_xdp_db(struct mlx5e_sq *sq, int numa)
sq->db.xdp.di = kzalloc_node(sizeof(*sq->db.xdp.di) * wq_sz,
GFP_KERNEL, numa);
- sq->db.xdp.wqe_info = kzalloc_node(sizeof(*sq->db.xdp.wqe_info) * wq_sz,
- GFP_KERNEL, numa);
- if (!sq->db.xdp.di || !sq->db.xdp.wqe_info) {
+ if (!sq->db.xdp.di) {
mlx5e_free_sq_xdp_db(sq);
return -ENOMEM;
}
@@ -993,7 +990,7 @@ static int mlx5e_sq_get_max_wqebbs(u8 sq_type)
case MLX5E_SQ_ICO:
return MLX5E_ICOSQ_MAX_WQEBBS;
case MLX5E_SQ_XDP:
- return MLX5E_XDP_TX_WQEBBS;
+ return 1;
}
return MLX5_SEND_WQE_MAX_WQEBBS;
}
@@ -1513,6 +1510,40 @@ static inline int mlx5e_get_max_num_channels(struct mlx5_core_dev *mdev)
MLX5E_MAX_NUM_CHANNELS);
}
+static int mlx5e_open_xdpsq(struct mlx5e_channel *c,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_sq *sq)
+{
+ unsigned int ds_cnt = MLX5E_XDP_TX_DS_COUNT;
+ unsigned int inline_hdr_sz = 0;
+ int err;
+ int i;
+
+ err = mlx5e_open_sq(c, 0, param, sq);
+ if (err)
+ return err;
+
+ if (sq->min_inline_mode != MLX5_INLINE_MODE_NONE) {
+ inline_hdr_sz = MLX5E_XDP_MIN_INLINE;
+ ds_cnt++;
+ }
+
+ /* Pre initialize fixed WQE fields */
+ for (i = 0; i < mlx5_wq_cyc_get_size(&sq->wq); i++) {
+ struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(&sq->wq, i);
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ struct mlx5_wqe_eth_seg *eseg = &wqe->eth;
+ struct mlx5_wqe_data_seg *dseg;
+
+ cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt);
+ eseg->inline_hdr.sz = cpu_to_be16(inline_hdr_sz);
+
+ dseg = (struct mlx5_wqe_data_seg *)cseg + (ds_cnt - 1);
+ dseg->lkey = sq->mkey_be;
+ }
+ return 0;
+}
+
static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
struct mlx5e_channel_param *cparam,
struct mlx5e_channel **cp)
@@ -1587,7 +1618,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
}
}
- err = c->xdp ? mlx5e_open_sq(c, 0, &cparam->xdp_sq, &c->rq.xdpsq) : 0;
+ err = c->xdp ? mlx5e_open_xdpsq(c, &cparam->xdp_sq, &c->rq.xdpsq) : 0;
if (err)
goto err_close_sqs;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 040074f36313..1b50c54614ac 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -641,7 +641,7 @@ static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_sq *sq)
{
struct mlx5_wq_cyc *wq = &sq->wq;
struct mlx5e_tx_wqe *wqe;
- u16 pi = (sq->pc - MLX5E_XDP_TX_WQEBBS) & wq->sz_m1; /* last pi */
+ u16 pi = (sq->pc - 1) & wq->sz_m1; /* last pi */
wqe = mlx5_wq_cyc_get_wqe(wq, pi);
@@ -657,17 +657,17 @@ static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
struct mlx5_wq_cyc *wq = &sq->wq;
u16 pi = sq->pc & wq->sz_m1;
struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
- struct mlx5e_sq_wqe_info *wi = &sq->db.xdp.wqe_info[pi];
struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
struct mlx5_wqe_eth_seg *eseg = &wqe->eth;
struct mlx5_wqe_data_seg *dseg;
- u8 ds_cnt = MLX5E_XDP_TX_DS_COUNT;
ptrdiff_t data_offset = xdp->data - xdp->data_hard_start;
dma_addr_t dma_addr = di->addr + data_offset;
unsigned int dma_len = xdp->data_end - xdp->data;
+ prefetchw(wqe);
+
if (unlikely(dma_len < MLX5E_XDP_MIN_INLINE ||
MLX5E_SW2HW_MTU(rq->netdev->mtu) < dma_len)) {
rq->stats.xdp_drop++;
@@ -675,7 +675,7 @@ static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
return false;
}
- if (unlikely(!mlx5e_sq_has_room_for(sq, MLX5E_XDP_TX_WQEBBS))) {
+ if (unlikely(!mlx5e_sq_has_room_for(sq, 1))) {
if (sq->db.xdp.doorbell) {
/* SQ is full, ring doorbell */
mlx5e_xmit_xdp_doorbell(sq);
@@ -686,35 +686,29 @@ static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
return false;
}
- dma_sync_single_for_device(sq->pdev, dma_addr, dma_len,
- PCI_DMA_TODEVICE);
+ dma_sync_single_for_device(sq->pdev, dma_addr, dma_len, PCI_DMA_TODEVICE);
- memset(wqe, 0, sizeof(*wqe));
+ cseg->fm_ce_se = 0;
dseg = (struct mlx5_wqe_data_seg *)eseg + 1;
+
/* copy the inline part if required */
if (sq->min_inline_mode != MLX5_INLINE_MODE_NONE) {
memcpy(eseg->inline_hdr.start, xdp->data, MLX5E_XDP_MIN_INLINE);
eseg->inline_hdr.sz = cpu_to_be16(MLX5E_XDP_MIN_INLINE);
dma_len -= MLX5E_XDP_MIN_INLINE;
dma_addr += MLX5E_XDP_MIN_INLINE;
-
- ds_cnt += MLX5E_XDP_IHS_DS_COUNT;
dseg++;
}
/* write the dma part */
dseg->addr = cpu_to_be64(dma_addr);
dseg->byte_count = cpu_to_be32(dma_len);
- dseg->lkey = sq->mkey_be;
cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_SEND);
- cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt);
sq->db.xdp.di[pi] = *di;
- wi->opcode = MLX5_OPCODE_SEND;
- wi->num_wqebbs = MLX5E_XDP_TX_WQEBBS;
- sq->pc += MLX5E_XDP_TX_WQEBBS;
+ sq->pc++;
sq->db.xdp.doorbell = true;
rq->stats.xdp_tx++;
@@ -1023,7 +1017,6 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
wqe_counter = be16_to_cpu(cqe->wqe_counter);
do {
- struct mlx5e_sq_wqe_info *wi;
struct mlx5e_dma_info *di;
u16 ci;
@@ -1031,14 +1024,8 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
ci = sqcc & sq->wq.sz_m1;
di = &sq->db.xdp.di[ci];
- wi = &sq->db.xdp.wqe_info[ci];
-
- if (unlikely(wi->opcode == MLX5_OPCODE_NOP)) {
- sqcc++;
- continue;
- }
- sqcc += wi->num_wqebbs;
+ sqcc++;
/* Recycle RX page */
mlx5e_page_release(rq, di, true);
} while (!last_wqe);
@@ -1056,21 +1043,13 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
void mlx5e_free_xdpsq_descs(struct mlx5e_sq *sq)
{
struct mlx5e_rq *rq = container_of(sq, struct mlx5e_rq, xdpsq);
- struct mlx5e_sq_wqe_info *wi;
struct mlx5e_dma_info *di;
u16 ci;
while (sq->cc != sq->pc) {
ci = sq->cc & sq->wq.sz_m1;
di = &sq->db.xdp.di[ci];
- wi = &sq->db.xdp.wqe_info[ci];
-
- if (wi->opcode == MLX5_OPCODE_NOP) {
- sq->cc++;
- continue;
- }
-
- sq->cc += wi->num_wqebbs;
+ sq->cc++;
mlx5e_page_release(rq, di, false);
}
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 09/12] net/mlx5e: Generalize tx helper functions for different SQ types
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (7 preceding siblings ...)
2017-03-24 21:52 ` [PATCH net-next 08/12] net/mlx5e: Optimize XDP frame xmit Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 10/12] net/mlx5e: Proper names for SQ/RQ/CQ functions Saeed Mahameed
` (5 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
In the next patches we will introduce different SQ types, for that we here
generalize some TX helper functions to work with more basic SQ parameters,
in order to re-use them for the different SQ types.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 35 ++++++++++++++++-----
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 13 +++++---
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 10 +++---
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 37 +++++------------------
4 files changed, 48 insertions(+), 47 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index f02d2cb8d148..50f895fa5f31 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -375,10 +375,10 @@ struct mlx5e_sq {
u8 type;
} ____cacheline_aligned_in_smp;
-static inline bool mlx5e_sq_has_room_for(struct mlx5e_sq *sq, u16 n)
+static inline bool
+mlx5e_wqc_has_room_for(struct mlx5_wq_cyc *wq, u16 cc, u16 pc, u16 n)
{
- return (((sq->wq.sz_m1 & (sq->cc - sq->pc)) >= n) ||
- (sq->cc == sq->pc));
+ return (((wq->sz_m1 & (cc - pc)) >= n) || (cc == pc));
}
struct mlx5e_dma_info {
@@ -721,7 +721,6 @@ struct mlx5e_priv {
void mlx5e_build_ptys2ethtool_map(void);
-void mlx5e_send_nop(struct mlx5e_sq *sq, bool notify_hw);
u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb,
void *accel_priv, select_queue_fallback_t fallback);
netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev);
@@ -807,20 +806,40 @@ void mlx5e_set_rx_cq_mode_params(struct mlx5e_params *params,
u8 cq_period_mode);
void mlx5e_set_rq_type_params(struct mlx5e_priv *priv, u8 rq_type);
-static inline void
-mlx5e_tx_notify_hw(struct mlx5e_sq *sq, struct mlx5_wqe_ctrl_seg *ctrl)
+static inline
+struct mlx5e_tx_wqe *mlx5e_post_nop(struct mlx5_wq_cyc *wq, u32 sqn, u16 *pc)
{
+ u16 pi = *pc & wq->sz_m1;
+ struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+
+ memset(cseg, 0, sizeof(*cseg));
+
+ cseg->opmod_idx_opcode = cpu_to_be32((*pc << 8) | MLX5_OPCODE_NOP);
+ cseg->qpn_ds = cpu_to_be32((sqn << 8) | 0x01);
+
+ (*pc)++;
+
+ return wqe;
+}
+
+static inline
+void mlx5e_notify_hw(struct mlx5_wq_cyc *wq, u16 pc,
+ void __iomem *uar_map,
+ struct mlx5_wqe_ctrl_seg *ctrl)
+{
+ ctrl->fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
/* ensure wqe is visible to device before updating doorbell record */
dma_wmb();
- *sq->wq.db = cpu_to_be32(sq->pc);
+ *wq->db = cpu_to_be32(pc);
/* ensure doorbell record is visible to device before ringing the
* doorbell
*/
wmb();
- mlx5_write64((__be32 *)ctrl, sq->uar_map, NULL);
+ mlx5_write64((__be32 *)ctrl, uar_map, NULL);
}
static inline void mlx5e_cq_arm(struct mlx5e_cq *cq)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index d39ee6669b8e..7faf2bcccfa6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -847,6 +847,7 @@ static int mlx5e_open_rq(struct mlx5e_channel *c,
{
struct mlx5e_sq *sq = &c->icosq;
u16 pi = sq->pc & sq->wq.sz_m1;
+ struct mlx5e_tx_wqe *nopwqe;
int err;
err = mlx5e_create_rq(c, param, rq);
@@ -867,8 +868,9 @@ static int mlx5e_open_rq(struct mlx5e_channel *c,
sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_NOP;
sq->db.ico_wqe[pi].num_wqebbs = 1;
- mlx5e_send_nop(sq, true); /* trigger mlx5e_post_rx_wqes() */
-
+ nopwqe = mlx5e_post_nop(&sq->wq, sq->sqn, &sq->pc);
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &nopwqe->ctrl);
+ sq->stats.nop++; /* TODO no need for SQ stats in ico */
return 0;
err_disable_rq:
@@ -1202,9 +1204,12 @@ static void mlx5e_close_sq(struct mlx5e_sq *sq)
netif_tx_disable_queue(sq->txq);
/* last doorbell out, godspeed .. */
- if (mlx5e_sq_has_room_for(sq, 1)) {
+ if (mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, 1)) {
+ struct mlx5e_tx_wqe *nop;
+
sq->db.txq.skb[(sq->pc & sq->wq.sz_m1)] = NULL;
- mlx5e_send_nop(sq, true);
+ nop = mlx5e_post_nop(&sq->wq, sq->sqn, &sq->pc);
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &nop->ctrl);
}
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 1b50c54614ac..141dcc486063 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -341,7 +341,8 @@ static inline void mlx5e_post_umr_wqe(struct mlx5e_rq *rq, u16 ix)
while ((pi = (sq->pc & wq->sz_m1)) > sq->edge) {
sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_NOP;
sq->db.ico_wqe[pi].num_wqebbs = 1;
- mlx5e_send_nop(sq, false);
+ mlx5e_post_nop(wq, sq->sqn, &sq->pc);
+ sq->stats.nop++;
}
wqe = mlx5_wq_cyc_get_wqe(wq, pi);
@@ -353,7 +354,7 @@ static inline void mlx5e_post_umr_wqe(struct mlx5e_rq *rq, u16 ix)
sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_UMR;
sq->db.ico_wqe[pi].num_wqebbs = num_wqebbs;
sq->pc += num_wqebbs;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl);
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &wqe->ctrl);
}
static int mlx5e_alloc_rx_umr_mpwqe(struct mlx5e_rq *rq,
@@ -645,8 +646,7 @@ static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_sq *sq)
wqe = mlx5_wq_cyc_get_wqe(wq, pi);
- wqe->ctrl.fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl);
+ mlx5e_notify_hw(wq, sq->pc, sq->uar_map, &wqe->ctrl);
}
static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
@@ -675,7 +675,7 @@ static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
return false;
}
- if (unlikely(!mlx5e_sq_has_room_for(sq, 1))) {
+ if (unlikely(!mlx5e_wqc_has_room_for(wq, sq->cc, sq->pc, 1))) {
if (sq->db.xdp.doorbell) {
/* SQ is full, ring doorbell */
mlx5e_xmit_xdp_doorbell(sq);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index 7497b6ac4382..897eaea6f51f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -38,29 +38,6 @@
#define MLX5E_SQ_STOP_ROOM (MLX5_SEND_WQE_MAX_WQEBBS +\
MLX5E_SQ_NOPS_ROOM)
-void mlx5e_send_nop(struct mlx5e_sq *sq, bool notify_hw)
-{
- struct mlx5_wq_cyc *wq = &sq->wq;
-
- u16 pi = sq->pc & wq->sz_m1;
- struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
-
- struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
-
- memset(cseg, 0, sizeof(*cseg));
-
- cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_NOP);
- cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | 0x01);
-
- sq->pc++;
- sq->stats.nop++;
-
- if (notify_hw) {
- cseg->fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl);
- }
-}
-
static inline void mlx5e_tx_dma_unmap(struct device *pdev,
struct mlx5e_sq_dma *dma)
{
@@ -331,21 +308,21 @@ static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP))
skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
- if (unlikely(!mlx5e_sq_has_room_for(sq, MLX5E_SQ_STOP_ROOM))) {
+ if (unlikely(!mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc,
+ MLX5E_SQ_STOP_ROOM))) {
netif_tx_stop_queue(sq->txq);
sq->stats.stopped++;
}
sq->stats.xmit_more += skb->xmit_more;
- if (!skb->xmit_more || netif_xmit_stopped(sq->txq)) {
- cseg->fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
- mlx5e_tx_notify_hw(sq, &wqe->ctrl);
- }
+ if (!skb->xmit_more || netif_xmit_stopped(sq->txq))
+ mlx5e_notify_hw(wq, sq->pc, sq->uar_map, cseg);
/* fill sq edge with nops to avoid wqe wrap around */
while ((pi = (sq->pc & wq->sz_m1)) > sq->edge) {
sq->db.txq.skb[pi] = NULL;
- mlx5e_send_nop(sq, false);
+ mlx5e_post_nop(&sq->wq, sq->sqn, &sq->pc);
+ sq->stats.nop++;
}
return NETDEV_TX_OK;
@@ -456,7 +433,7 @@ bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget)
netdev_tx_completed_queue(sq->txq, npkts, nbytes);
if (netif_tx_queue_stopped(sq->txq) &&
- mlx5e_sq_has_room_for(sq, MLX5E_SQ_STOP_ROOM)) {
+ mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, MLX5E_SQ_STOP_ROOM)) {
netif_tx_wake_queue(sq->txq);
sq->stats.wake++;
}
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 10/12] net/mlx5e: Proper names for SQ/RQ/CQ functions
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (8 preceding siblings ...)
2017-03-24 21:52 ` [PATCH net-next 09/12] net/mlx5e: Generalize tx helper functions for different SQ types Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 11/12] net/mlx5e: Generalize SQ create/modify/destroy functions Saeed Mahameed
` (4 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
Rename mlx5e_{create,destroy}_{sq,rq,cq} to
mlx5e_{alloc,free}_{sq,rq,cq}.
Rename mlx5e_{enable,disable}_{sq,rq,cq} to
mlx5e_{create,destroy}_{sq,rq,cq}.
mlx5e_{enable,disable}_{sq,rq,cq} used to actually create/destroy the SQ
in FW, so we rename them to align the functions names with FW semantics.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 126 +++++++++++-----------
1 file changed, 63 insertions(+), 63 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 7faf2bcccfa6..d03afa535064 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -539,9 +539,9 @@ static int mlx5e_create_rq_umr_mkey(struct mlx5e_rq *rq)
return mlx5e_create_umr_mkey(priv, num_mtts, PAGE_SHIFT, &rq->umr_mkey);
}
-static int mlx5e_create_rq(struct mlx5e_channel *c,
- struct mlx5e_rq_param *param,
- struct mlx5e_rq *rq)
+static int mlx5e_alloc_rq(struct mlx5e_channel *c,
+ struct mlx5e_rq_param *param,
+ struct mlx5e_rq *rq)
{
struct mlx5e_priv *priv = c->priv;
struct mlx5_core_dev *mdev = priv->mdev;
@@ -674,7 +674,7 @@ static int mlx5e_create_rq(struct mlx5e_channel *c,
return err;
}
-static void mlx5e_destroy_rq(struct mlx5e_rq *rq)
+static void mlx5e_free_rq(struct mlx5e_rq *rq)
{
int i;
@@ -699,7 +699,7 @@ static void mlx5e_destroy_rq(struct mlx5e_rq *rq)
mlx5_wq_destroy(&rq->wq_ctrl);
}
-static int mlx5e_enable_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param)
+static int mlx5e_create_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param)
{
struct mlx5e_priv *priv = rq->priv;
struct mlx5_core_dev *mdev = priv->mdev;
@@ -798,7 +798,7 @@ static int mlx5e_modify_rq_vsd(struct mlx5e_rq *rq, bool vsd)
return err;
}
-static void mlx5e_disable_rq(struct mlx5e_rq *rq)
+static void mlx5e_destroy_rq(struct mlx5e_rq *rq)
{
mlx5_core_destroy_rq(rq->priv->mdev, rq->rqn);
}
@@ -850,18 +850,18 @@ static int mlx5e_open_rq(struct mlx5e_channel *c,
struct mlx5e_tx_wqe *nopwqe;
int err;
- err = mlx5e_create_rq(c, param, rq);
+ err = mlx5e_alloc_rq(c, param, rq);
if (err)
return err;
- err = mlx5e_enable_rq(rq, param);
+ err = mlx5e_create_rq(rq, param);
if (err)
- goto err_destroy_rq;
+ goto err_free_rq;
set_bit(MLX5E_RQ_STATE_ENABLED, &rq->state);
err = mlx5e_modify_rq_state(rq, MLX5_RQC_STATE_RST, MLX5_RQC_STATE_RDY);
if (err)
- goto err_disable_rq;
+ goto err_destroy_rq;
if (param->am_enabled)
set_bit(MLX5E_RQ_STATE_AM, &c->rq.state);
@@ -873,11 +873,11 @@ static int mlx5e_open_rq(struct mlx5e_channel *c,
sq->stats.nop++; /* TODO no need for SQ stats in ico */
return 0;
-err_disable_rq:
- clear_bit(MLX5E_RQ_STATE_ENABLED, &rq->state);
- mlx5e_disable_rq(rq);
err_destroy_rq:
+ clear_bit(MLX5E_RQ_STATE_ENABLED, &rq->state);
mlx5e_destroy_rq(rq);
+err_free_rq:
+ mlx5e_free_rq(rq);
return err;
}
@@ -888,9 +888,9 @@ static void mlx5e_close_rq(struct mlx5e_rq *rq)
napi_synchronize(&rq->channel->napi); /* prevent mlx5e_post_rx_wqes */
cancel_work_sync(&rq->am.work);
- mlx5e_disable_rq(rq);
- mlx5e_free_rx_descs(rq);
mlx5e_destroy_rq(rq);
+ mlx5e_free_rx_descs(rq);
+ mlx5e_free_rq(rq);
}
static void mlx5e_free_sq_xdp_db(struct mlx5e_sq *sq)
@@ -997,10 +997,10 @@ static int mlx5e_sq_get_max_wqebbs(u8 sq_type)
return MLX5_SEND_WQE_MAX_WQEBBS;
}
-static int mlx5e_create_sq(struct mlx5e_channel *c,
- int tc,
- struct mlx5e_sq_param *param,
- struct mlx5e_sq *sq)
+static int mlx5e_alloc_sq(struct mlx5e_channel *c,
+ int tc,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_sq *sq)
{
struct mlx5e_priv *priv = c->priv;
struct mlx5_core_dev *mdev = priv->mdev;
@@ -1051,13 +1051,13 @@ static int mlx5e_create_sq(struct mlx5e_channel *c,
return err;
}
-static void mlx5e_destroy_sq(struct mlx5e_sq *sq)
+static void mlx5e_free_sq(struct mlx5e_sq *sq)
{
mlx5e_free_sq_db(sq);
mlx5_wq_destroy(&sq->wq_ctrl);
}
-static int mlx5e_enable_sq(struct mlx5e_sq *sq, struct mlx5e_sq_param *param)
+static int mlx5e_create_sq(struct mlx5e_sq *sq, struct mlx5e_sq_param *param)
{
struct mlx5e_channel *c = sq->channel;
struct mlx5e_priv *priv = c->priv;
@@ -1139,7 +1139,7 @@ static int mlx5e_modify_sq(struct mlx5e_sq *sq, int curr_state,
return err;
}
-static void mlx5e_disable_sq(struct mlx5e_sq *sq)
+static void mlx5e_destroy_sq(struct mlx5e_sq *sq)
{
struct mlx5e_channel *c = sq->channel;
struct mlx5e_priv *priv = c->priv;
@@ -1157,19 +1157,19 @@ static int mlx5e_open_sq(struct mlx5e_channel *c,
{
int err;
- err = mlx5e_create_sq(c, tc, param, sq);
+ err = mlx5e_alloc_sq(c, tc, param, sq);
if (err)
return err;
- err = mlx5e_enable_sq(sq, param);
+ err = mlx5e_create_sq(sq, param);
if (err)
- goto err_destroy_sq;
+ goto err_free_sq;
set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
err = mlx5e_modify_sq(sq, MLX5_SQC_STATE_RST, MLX5_SQC_STATE_RDY,
false, 0);
if (err)
- goto err_disable_sq;
+ goto err_destroy_sq;
if (sq->txq) {
netdev_tx_reset_queue(sq->txq);
@@ -1178,11 +1178,11 @@ static int mlx5e_open_sq(struct mlx5e_channel *c,
return 0;
-err_disable_sq:
- clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
- mlx5e_disable_sq(sq);
err_destroy_sq:
+ clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
mlx5e_destroy_sq(sq);
+err_free_sq:
+ mlx5e_free_sq(sq);
return err;
}
@@ -1213,14 +1213,14 @@ static void mlx5e_close_sq(struct mlx5e_sq *sq)
}
}
- mlx5e_disable_sq(sq);
- mlx5e_free_sq_descs(sq);
mlx5e_destroy_sq(sq);
+ mlx5e_free_sq_descs(sq);
+ mlx5e_free_sq(sq);
}
-static int mlx5e_create_cq(struct mlx5e_channel *c,
- struct mlx5e_cq_param *param,
- struct mlx5e_cq *cq)
+static int mlx5e_alloc_cq(struct mlx5e_channel *c,
+ struct mlx5e_cq_param *param,
+ struct mlx5e_cq *cq)
{
struct mlx5e_priv *priv = c->priv;
struct mlx5_core_dev *mdev = priv->mdev;
@@ -1265,12 +1265,12 @@ static int mlx5e_create_cq(struct mlx5e_channel *c,
return 0;
}
-static void mlx5e_destroy_cq(struct mlx5e_cq *cq)
+static void mlx5e_free_cq(struct mlx5e_cq *cq)
{
mlx5_cqwq_destroy(&cq->wq_ctrl);
}
-static int mlx5e_enable_cq(struct mlx5e_cq *cq, struct mlx5e_cq_param *param)
+static int mlx5e_create_cq(struct mlx5e_cq *cq, struct mlx5e_cq_param *param)
{
struct mlx5e_priv *priv = cq->priv;
struct mlx5_core_dev *mdev = priv->mdev;
@@ -1317,7 +1317,7 @@ static int mlx5e_enable_cq(struct mlx5e_cq *cq, struct mlx5e_cq_param *param)
return 0;
}
-static void mlx5e_disable_cq(struct mlx5e_cq *cq)
+static void mlx5e_destroy_cq(struct mlx5e_cq *cq)
{
struct mlx5e_priv *priv = cq->priv;
struct mlx5_core_dev *mdev = priv->mdev;
@@ -1334,13 +1334,13 @@ static int mlx5e_open_cq(struct mlx5e_channel *c,
struct mlx5e_priv *priv = c->priv;
struct mlx5_core_dev *mdev = priv->mdev;
- err = mlx5e_create_cq(c, param, cq);
+ err = mlx5e_alloc_cq(c, param, cq);
if (err)
return err;
- err = mlx5e_enable_cq(cq, param);
+ err = mlx5e_create_cq(cq, param);
if (err)
- goto err_destroy_cq;
+ goto err_free_cq;
if (MLX5_CAP_GEN(mdev, cq_moderation))
mlx5_core_modify_cq_moderation(mdev, &cq->mcq,
@@ -1348,16 +1348,16 @@ static int mlx5e_open_cq(struct mlx5e_channel *c,
moderation.pkts);
return 0;
-err_destroy_cq:
- mlx5e_destroy_cq(cq);
+err_free_cq:
+ mlx5e_free_cq(cq);
return err;
}
static void mlx5e_close_cq(struct mlx5e_cq *cq)
{
- mlx5e_disable_cq(cq);
mlx5e_destroy_cq(cq);
+ mlx5e_free_cq(cq);
}
static int mlx5e_get_cpu(struct mlx5e_priv *priv, int ix)
@@ -2422,9 +2422,9 @@ int mlx5e_close(struct net_device *netdev)
return err;
}
-static int mlx5e_create_drop_rq(struct mlx5e_priv *priv,
- struct mlx5e_rq *rq,
- struct mlx5e_rq_param *param)
+static int mlx5e_alloc_drop_rq(struct mlx5e_priv *priv,
+ struct mlx5e_rq *rq,
+ struct mlx5e_rq_param *param)
{
struct mlx5_core_dev *mdev = priv->mdev;
void *rqc = param->rqc;
@@ -2443,9 +2443,9 @@ static int mlx5e_create_drop_rq(struct mlx5e_priv *priv,
return 0;
}
-static int mlx5e_create_drop_cq(struct mlx5e_priv *priv,
- struct mlx5e_cq *cq,
- struct mlx5e_cq_param *param)
+static int mlx5e_alloc_drop_cq(struct mlx5e_priv *priv,
+ struct mlx5e_cq *cq,
+ struct mlx5e_cq_param *param)
{
struct mlx5_core_dev *mdev = priv->mdev;
struct mlx5_core_cq *mcq = &cq->mcq;
@@ -2487,42 +2487,42 @@ static int mlx5e_open_drop_rq(struct mlx5e_priv *priv)
memset(&rq_param, 0, sizeof(rq_param));
mlx5e_build_drop_rq_param(&rq_param);
- err = mlx5e_create_drop_cq(priv, cq, &cq_param);
+ err = mlx5e_alloc_drop_cq(priv, cq, &cq_param);
if (err)
return err;
- err = mlx5e_enable_cq(cq, &cq_param);
+ err = mlx5e_create_cq(cq, &cq_param);
if (err)
- goto err_destroy_cq;
+ goto err_free_cq;
- err = mlx5e_create_drop_rq(priv, rq, &rq_param);
+ err = mlx5e_alloc_drop_rq(priv, rq, &rq_param);
if (err)
- goto err_disable_cq;
+ goto err_destroy_cq;
- err = mlx5e_enable_rq(rq, &rq_param);
+ err = mlx5e_create_rq(rq, &rq_param);
if (err)
- goto err_destroy_rq;
+ goto err_free_rq;
return 0;
-err_destroy_rq:
- mlx5e_destroy_rq(&priv->drop_rq);
-
-err_disable_cq:
- mlx5e_disable_cq(&priv->drop_rq.cq);
+err_free_rq:
+ mlx5e_free_rq(&priv->drop_rq);
err_destroy_cq:
mlx5e_destroy_cq(&priv->drop_rq.cq);
+err_free_cq:
+ mlx5e_free_cq(&priv->drop_rq.cq);
+
return err;
}
static void mlx5e_close_drop_rq(struct mlx5e_priv *priv)
{
- mlx5e_disable_rq(&priv->drop_rq);
mlx5e_destroy_rq(&priv->drop_rq);
- mlx5e_disable_cq(&priv->drop_rq.cq);
+ mlx5e_free_rq(&priv->drop_rq);
mlx5e_destroy_cq(&priv->drop_rq.cq);
+ mlx5e_free_cq(&priv->drop_rq.cq);
}
static int mlx5e_create_tis(struct mlx5e_priv *priv, int tc)
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 11/12] net/mlx5e: Generalize SQ create/modify/destroy functions
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (9 preceding siblings ...)
2017-03-24 21:52 ` [PATCH net-next 10/12] net/mlx5e: Proper names for SQ/RQ/CQ functions Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 12/12] net/mlx5e: Different SQ types Saeed Mahameed
` (3 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
In the next patches we will introduce different SQ types,
and we would want to reuse those functions, in this patch we make them
agnostic to SQ type (txq, xdp, ico).
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 111 ++++++++++++++--------
1 file changed, 69 insertions(+), 42 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index d03afa535064..dcc67df54a5c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1057,10 +1057,19 @@ static void mlx5e_free_sq(struct mlx5e_sq *sq)
mlx5_wq_destroy(&sq->wq_ctrl);
}
-static int mlx5e_create_sq(struct mlx5e_sq *sq, struct mlx5e_sq_param *param)
+struct mlx5e_create_sq_param {
+ struct mlx5_wq_ctrl *wq_ctrl;
+ u32 cqn;
+ u32 tisn;
+ u8 tis_lst_sz;
+ u8 min_inline_mode;
+};
+
+static int mlx5e_create_sq(struct mlx5e_priv *priv,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_create_sq_param *csp,
+ u32 *sqn)
{
- struct mlx5e_channel *c = sq->channel;
- struct mlx5e_priv *priv = c->priv;
struct mlx5_core_dev *mdev = priv->mdev;
void *in;
@@ -1070,7 +1079,7 @@ static int mlx5e_create_sq(struct mlx5e_sq *sq, struct mlx5e_sq_param *param)
int err;
inlen = MLX5_ST_SZ_BYTES(create_sq_in) +
- sizeof(u64) * sq->wq_ctrl.buf.npages;
+ sizeof(u64) * csp->wq_ctrl->buf.npages;
in = mlx5_vzalloc(inlen);
if (!in)
return -ENOMEM;
@@ -1079,38 +1088,41 @@ static int mlx5e_create_sq(struct mlx5e_sq *sq, struct mlx5e_sq_param *param)
wq = MLX5_ADDR_OF(sqc, sqc, wq);
memcpy(sqc, param->sqc, sizeof(param->sqc));
-
- MLX5_SET(sqc, sqc, tis_num_0, param->type == MLX5E_SQ_ICO ?
- 0 : priv->tisn[sq->tc]);
- MLX5_SET(sqc, sqc, cqn, sq->cq.mcq.cqn);
+ MLX5_SET(sqc, sqc, tis_lst_sz, csp->tis_lst_sz);
+ MLX5_SET(sqc, sqc, tis_num_0, csp->tisn);
+ MLX5_SET(sqc, sqc, cqn, csp->cqn);
if (MLX5_CAP_ETH(mdev, wqe_inline_mode) == MLX5_CAP_INLINE_MODE_VPORT_CONTEXT)
- MLX5_SET(sqc, sqc, min_wqe_inline_mode, sq->min_inline_mode);
+ MLX5_SET(sqc, sqc, min_wqe_inline_mode, csp->min_inline_mode);
- MLX5_SET(sqc, sqc, state, MLX5_SQC_STATE_RST);
- MLX5_SET(sqc, sqc, tis_lst_sz, param->type == MLX5E_SQ_ICO ? 0 : 1);
+ MLX5_SET(sqc, sqc, state, MLX5_SQC_STATE_RST);
MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_CYCLIC);
- MLX5_SET(wq, wq, uar_page, mdev->mlx5e_res.bfreg.index);
- MLX5_SET(wq, wq, log_wq_pg_sz, sq->wq_ctrl.buf.page_shift -
+ MLX5_SET(wq, wq, uar_page, priv->mdev->mlx5e_res.bfreg.index);
+ MLX5_SET(wq, wq, log_wq_pg_sz, csp->wq_ctrl->buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
- MLX5_SET64(wq, wq, dbr_addr, sq->wq_ctrl.db.dma);
+ MLX5_SET64(wq, wq, dbr_addr, csp->wq_ctrl->db.dma);
- mlx5_fill_page_array(&sq->wq_ctrl.buf,
- (__be64 *)MLX5_ADDR_OF(wq, wq, pas));
+ mlx5_fill_page_array(&csp->wq_ctrl->buf, (__be64 *)MLX5_ADDR_OF(wq, wq, pas));
- err = mlx5_core_create_sq(mdev, in, inlen, &sq->sqn);
+ err = mlx5_core_create_sq(mdev, in, inlen, sqn);
kvfree(in);
return err;
}
-static int mlx5e_modify_sq(struct mlx5e_sq *sq, int curr_state,
- int next_state, bool update_rl, int rl_index)
+struct mlx5e_modify_sq_param {
+ int curr_state;
+ int next_state;
+ bool rl_update;
+ int rl_index;
+};
+
+static int mlx5e_modify_sq(struct mlx5e_priv *priv,
+ u32 sqn,
+ struct mlx5e_modify_sq_param *p)
{
- struct mlx5e_channel *c = sq->channel;
- struct mlx5e_priv *priv = c->priv;
struct mlx5_core_dev *mdev = priv->mdev;
void *in;
@@ -1125,29 +1137,24 @@ static int mlx5e_modify_sq(struct mlx5e_sq *sq, int curr_state,
sqc = MLX5_ADDR_OF(modify_sq_in, in, ctx);
- MLX5_SET(modify_sq_in, in, sq_state, curr_state);
- MLX5_SET(sqc, sqc, state, next_state);
- if (update_rl && next_state == MLX5_SQC_STATE_RDY) {
+ MLX5_SET(modify_sq_in, in, sq_state, p->curr_state);
+ MLX5_SET(sqc, sqc, state, p->next_state);
+ if (p->rl_update && p->next_state == MLX5_SQC_STATE_RDY) {
MLX5_SET64(modify_sq_in, in, modify_bitmask, 1);
- MLX5_SET(sqc, sqc, packet_pacing_rate_limit_index, rl_index);
+ MLX5_SET(sqc, sqc, packet_pacing_rate_limit_index, p->rl_index);
}
- err = mlx5_core_modify_sq(mdev, sq->sqn, in, inlen);
+ err = mlx5_core_modify_sq(mdev, sqn, in, inlen);
kvfree(in);
return err;
}
-static void mlx5e_destroy_sq(struct mlx5e_sq *sq)
-{
- struct mlx5e_channel *c = sq->channel;
- struct mlx5e_priv *priv = c->priv;
- struct mlx5_core_dev *mdev = priv->mdev;
- mlx5_core_destroy_sq(mdev, sq->sqn);
- if (sq->rate_limit)
- mlx5_rl_remove_rate(mdev, sq->rate_limit);
+static void mlx5e_destroy_sq(struct mlx5e_priv *priv, u32 sqn)
+{
+ mlx5_core_destroy_sq(priv->mdev, sqn);
}
static int mlx5e_open_sq(struct mlx5e_channel *c,
@@ -1155,19 +1162,29 @@ static int mlx5e_open_sq(struct mlx5e_channel *c,
struct mlx5e_sq_param *param,
struct mlx5e_sq *sq)
{
+ struct mlx5e_create_sq_param csp = {0};
+ struct mlx5e_modify_sq_param msp = {0};
+ struct mlx5e_priv *priv = c->priv;
int err;
err = mlx5e_alloc_sq(c, tc, param, sq);
if (err)
return err;
- err = mlx5e_create_sq(sq, param);
+ csp.tisn = param->type == MLX5E_SQ_ICO ? 0 : priv->tisn[sq->tc];
+ csp.tis_lst_sz = param->type == MLX5E_SQ_ICO ? 0 : 1;
+ csp.cqn = sq->cq.mcq.cqn;
+ csp.wq_ctrl = &sq->wq_ctrl;
+ csp.min_inline_mode = sq->min_inline_mode;
+
+ err = mlx5e_create_sq(c->priv, param, &csp, &sq->sqn);
if (err)
goto err_free_sq;
set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
- err = mlx5e_modify_sq(sq, MLX5_SQC_STATE_RST, MLX5_SQC_STATE_RDY,
- false, 0);
+ msp.curr_state = MLX5_SQC_STATE_RST;
+ msp.next_state = MLX5_SQC_STATE_RDY;
+ err = mlx5e_modify_sq(priv, sq->sqn, &msp);
if (err)
goto err_destroy_sq;
@@ -1180,7 +1197,7 @@ static int mlx5e_open_sq(struct mlx5e_channel *c,
err_destroy_sq:
clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
- mlx5e_destroy_sq(sq);
+ mlx5e_destroy_sq(priv, sq->sqn);
err_free_sq:
mlx5e_free_sq(sq);
@@ -1196,9 +1213,13 @@ static inline void netif_tx_disable_queue(struct netdev_queue *txq)
static void mlx5e_close_sq(struct mlx5e_sq *sq)
{
+ struct mlx5e_channel *c = sq->channel;
+ struct mlx5e_priv *priv = c->priv;
+ struct mlx5_core_dev *mdev = priv->mdev;
+
clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
/* prevent netif_tx_wake_queue */
- napi_synchronize(&sq->channel->napi);
+ napi_synchronize(&c->napi);
if (sq->txq) {
netif_tx_disable_queue(sq->txq);
@@ -1213,7 +1234,9 @@ static void mlx5e_close_sq(struct mlx5e_sq *sq)
}
}
- mlx5e_destroy_sq(sq);
+ mlx5e_destroy_sq(priv, sq->sqn);
+ if (sq->rate_limit)
+ mlx5_rl_remove_rate(mdev, sq->rate_limit);
mlx5e_free_sq_descs(sq);
mlx5e_free_sq(sq);
}
@@ -1439,6 +1462,7 @@ static int mlx5e_set_sq_maxrate(struct net_device *dev,
{
struct mlx5e_priv *priv = netdev_priv(dev);
struct mlx5_core_dev *mdev = priv->mdev;
+ struct mlx5e_modify_sq_param msp = {0};
u16 rl_index = 0;
int err;
@@ -1461,8 +1485,11 @@ static int mlx5e_set_sq_maxrate(struct net_device *dev,
}
}
- err = mlx5e_modify_sq(sq, MLX5_SQC_STATE_RDY,
- MLX5_SQC_STATE_RDY, true, rl_index);
+ msp.curr_state = MLX5_SQC_STATE_RDY;
+ msp.next_state = MLX5_SQC_STATE_RDY;
+ msp.rl_index = rl_index;
+ msp.rl_update = true;
+ err = mlx5e_modify_sq(priv, sq->sqn, &msp);
if (err) {
netdev_err(dev, "Failed configuring rate %u: %d\n",
rate, err);
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH net-next 12/12] net/mlx5e: Different SQ types
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (10 preceding siblings ...)
2017-03-24 21:52 ` [PATCH net-next 11/12] net/mlx5e: Generalize SQ create/modify/destroy functions Saeed Mahameed
@ 2017-03-24 21:52 ` Saeed Mahameed
2017-03-24 23:26 ` [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Alexei Starovoitov
` (2 subsequent siblings)
14 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-24 21:52 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, kernel-team, Saeed Mahameed
Different SQ types (tx, xdp, ico) are growing apart, we separate them
and remove unwanted parts in each one of them, to simplify data path and
utilize data cache.
Remove DB union from SQ structures since it is not needed anymore as we
now have different SQ data type for each SQ.
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 99 +++--
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 464 +++++++++++++---------
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 33 +-
drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 50 +--
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 2 +-
5 files changed, 392 insertions(+), 256 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 50f895fa5f31..bace9233dc1f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -319,13 +319,7 @@ struct mlx5e_sq_wqe_info {
u8 num_wqebbs;
};
-enum mlx5e_sq_type {
- MLX5E_SQ_TXQ,
- MLX5E_SQ_ICO,
- MLX5E_SQ_XDP
-};
-
-struct mlx5e_sq {
+struct mlx5e_txqsq {
/* data path */
/* dirtied @completion */
@@ -339,18 +333,11 @@ struct mlx5e_sq {
struct mlx5e_cq cq;
- /* pointers to per tx element info: write@xmit, read@completion */
- union {
- struct {
- struct sk_buff **skb;
- struct mlx5e_sq_dma *dma_fifo;
- struct mlx5e_tx_wqe_info *wqe_info;
- } txq;
- struct mlx5e_sq_wqe_info *ico_wqe;
- struct {
- struct mlx5e_dma_info *di;
- bool doorbell;
- } xdp;
+ /* write@xmit, read@completion */
+ struct {
+ struct sk_buff **skb;
+ struct mlx5e_sq_dma *dma_fifo;
+ struct mlx5e_tx_wqe_info *wqe_info;
} db;
/* read only */
@@ -372,7 +359,67 @@ struct mlx5e_sq {
struct mlx5e_channel *channel;
int tc;
u32 rate_limit;
- u8 type;
+} ____cacheline_aligned_in_smp;
+
+struct mlx5e_xdpsq {
+ /* data path */
+
+ /* dirtied @rx completion */
+ u16 cc;
+ u16 pc;
+
+ struct mlx5e_cq cq;
+
+ /* write@xmit, read@completion */
+ struct {
+ struct mlx5e_dma_info *di;
+ bool doorbell;
+ } db;
+
+ /* read only */
+ struct mlx5_wq_cyc wq;
+ void __iomem *uar_map;
+ u32 sqn;
+ struct device *pdev;
+ __be32 mkey_be;
+ u8 min_inline_mode;
+ unsigned long state;
+
+ /* control path */
+ struct mlx5_wq_ctrl wq_ctrl;
+ struct mlx5e_channel *channel;
+} ____cacheline_aligned_in_smp;
+
+struct mlx5e_icosq {
+ /* data path */
+
+ /* dirtied @completion */
+ u16 cc;
+
+ /* dirtied @xmit */
+ u16 pc ____cacheline_aligned_in_smp;
+ u32 dma_fifo_pc;
+ u16 prev_cc;
+
+ struct mlx5e_cq cq;
+
+ /* write@xmit, read@completion */
+ struct {
+ struct mlx5e_sq_wqe_info *ico_wqe;
+ } db;
+
+ /* read only */
+ struct mlx5_wq_cyc wq;
+ void __iomem *uar_map;
+ u32 sqn;
+ u16 edge;
+ struct device *pdev;
+ __be32 mkey_be;
+ unsigned long state;
+
+ /* control path */
+ struct mlx5_wq_ctrl wq_ctrl;
+ struct mlx5e_channel *channel;
} ____cacheline_aligned_in_smp;
static inline bool
@@ -477,7 +524,7 @@ struct mlx5e_rq {
/* XDP */
struct bpf_prog *xdp_prog;
- struct mlx5e_sq xdpsq;
+ struct mlx5e_xdpsq xdpsq;
/* control */
struct mlx5_wq_ctrl wq_ctrl;
@@ -497,8 +544,8 @@ enum channel_flags {
struct mlx5e_channel {
/* data path */
struct mlx5e_rq rq;
- struct mlx5e_sq sq[MLX5E_MAX_NUM_TC];
- struct mlx5e_sq icosq; /* internal control operations */
+ struct mlx5e_txqsq sq[MLX5E_MAX_NUM_TC];
+ struct mlx5e_icosq icosq; /* internal control operations */
bool xdp;
struct napi_struct napi;
struct device *pdev;
@@ -680,7 +727,7 @@ struct mlx5e_profile {
struct mlx5e_priv {
/* priv data path fields - start */
- struct mlx5e_sq **txq_to_sq_map;
+ struct mlx5e_txqsq **txq_to_sq_map;
int channeltc_to_txq_map[MLX5E_MAX_NUM_CHANNELS][MLX5E_MAX_NUM_TC];
struct bpf_prog *xdp_prog;
/* priv data path fields - end */
@@ -731,8 +778,8 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget);
bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget);
int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget);
bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq);
-void mlx5e_free_xdpsq_descs(struct mlx5e_sq *sq);
-void mlx5e_free_sq_descs(struct mlx5e_sq *sq);
+void mlx5e_free_txqsq_descs(struct mlx5e_txqsq *sq);
+void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq);
void mlx5e_page_release(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info,
bool recycle);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index dcc67df54a5c..e849a0fc2653 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -52,7 +52,6 @@ struct mlx5e_sq_param {
struct mlx5_wq_param wq;
u16 max_inline;
u8 min_inline_mode;
- enum mlx5e_sq_type type;
};
struct mlx5e_cq_param {
@@ -402,8 +401,10 @@ static inline int mlx5e_get_wqe_mtt_sz(void)
MLX5_UMR_MTT_ALIGNMENT);
}
-static inline void mlx5e_build_umr_wqe(struct mlx5e_rq *rq, struct mlx5e_sq *sq,
- struct mlx5e_umr_wqe *wqe, u16 ix)
+static inline void mlx5e_build_umr_wqe(struct mlx5e_rq *rq,
+ struct mlx5e_icosq *sq,
+ struct mlx5e_umr_wqe *wqe,
+ u16 ix)
{
struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
struct mlx5_wqe_umr_ctrl_seg *ucseg = &wqe->uctrl;
@@ -845,7 +846,7 @@ static int mlx5e_open_rq(struct mlx5e_channel *c,
struct mlx5e_rq_param *param,
struct mlx5e_rq *rq)
{
- struct mlx5e_sq *sq = &c->icosq;
+ struct mlx5e_icosq *sq = &c->icosq;
u16 pi = sq->pc & sq->wq.sz_m1;
struct mlx5e_tx_wqe *nopwqe;
int err;
@@ -870,7 +871,6 @@ static int mlx5e_open_rq(struct mlx5e_channel *c,
sq->db.ico_wqe[pi].num_wqebbs = 1;
nopwqe = mlx5e_post_nop(&sq->wq, sq->sqn, &sq->pc);
mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &nopwqe->ctrl);
- sq->stats.nop++; /* TODO no need for SQ stats in ico */
return 0;
err_destroy_rq:
@@ -893,31 +893,70 @@ static void mlx5e_close_rq(struct mlx5e_rq *rq)
mlx5e_free_rq(rq);
}
-static void mlx5e_free_sq_xdp_db(struct mlx5e_sq *sq)
+static void mlx5e_free_xdpsq_db(struct mlx5e_xdpsq *sq)
{
- kfree(sq->db.xdp.di);
+ kfree(sq->db.di);
}
-static int mlx5e_alloc_sq_xdp_db(struct mlx5e_sq *sq, int numa)
+static int mlx5e_alloc_xdpsq_db(struct mlx5e_xdpsq *sq, int numa)
{
int wq_sz = mlx5_wq_cyc_get_size(&sq->wq);
- sq->db.xdp.di = kzalloc_node(sizeof(*sq->db.xdp.di) * wq_sz,
+ sq->db.di = kzalloc_node(sizeof(*sq->db.di) * wq_sz,
GFP_KERNEL, numa);
- if (!sq->db.xdp.di) {
- mlx5e_free_sq_xdp_db(sq);
+ if (!sq->db.di) {
+ mlx5e_free_xdpsq_db(sq);
return -ENOMEM;
}
return 0;
}
-static void mlx5e_free_sq_ico_db(struct mlx5e_sq *sq)
+static int mlx5e_alloc_xdpsq(struct mlx5e_channel *c,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_xdpsq *sq)
+{
+ void *sqc_wq = MLX5_ADDR_OF(sqc, param->sqc, wq);
+ struct mlx5e_priv *priv = c->priv;
+ struct mlx5_core_dev *mdev = priv->mdev;
+ int err;
+
+ sq->pdev = c->pdev;
+ sq->mkey_be = c->mkey_be;
+ sq->channel = c;
+ sq->uar_map = mdev->mlx5e_res.bfreg.map;
+ sq->min_inline_mode = param->min_inline_mode;
+
+ param->wq.db_numa_node = cpu_to_node(c->cpu);
+ err = mlx5_wq_cyc_create(mdev, ¶m->wq, sqc_wq, &sq->wq, &sq->wq_ctrl);
+ if (err)
+ return err;
+ sq->wq.db = &sq->wq.db[MLX5_SND_DBR];
+
+ err = mlx5e_alloc_xdpsq_db(sq, cpu_to_node(c->cpu));
+ if (err)
+ goto err_sq_wq_destroy;
+
+ return 0;
+
+err_sq_wq_destroy:
+ mlx5_wq_destroy(&sq->wq_ctrl);
+
+ return err;
+}
+
+static void mlx5e_free_xdpsq(struct mlx5e_xdpsq *sq)
+{
+ mlx5e_free_xdpsq_db(sq);
+ mlx5_wq_destroy(&sq->wq_ctrl);
+}
+
+static void mlx5e_free_icosq_db(struct mlx5e_icosq *sq)
{
kfree(sq->db.ico_wqe);
}
-static int mlx5e_alloc_sq_ico_db(struct mlx5e_sq *sq, int numa)
+static int mlx5e_alloc_icosq_db(struct mlx5e_icosq *sq, int numa)
{
u8 wq_sz = mlx5_wq_cyc_get_size(&sq->wq);
@@ -929,119 +968,110 @@ static int mlx5e_alloc_sq_ico_db(struct mlx5e_sq *sq, int numa)
return 0;
}
-static void mlx5e_free_sq_txq_db(struct mlx5e_sq *sq)
+static int mlx5e_alloc_icosq(struct mlx5e_channel *c,
+ int tc,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_icosq *sq)
{
- kfree(sq->db.txq.wqe_info);
- kfree(sq->db.txq.dma_fifo);
- kfree(sq->db.txq.skb);
-}
+ void *sqc_wq = MLX5_ADDR_OF(sqc, param->sqc, wq);
+ struct mlx5e_priv *priv = c->priv;
+ struct mlx5_core_dev *mdev = priv->mdev;
+ int err;
-static int mlx5e_alloc_sq_txq_db(struct mlx5e_sq *sq, int numa)
-{
- int wq_sz = mlx5_wq_cyc_get_size(&sq->wq);
- int df_sz = wq_sz * MLX5_SEND_WQEBB_NUM_DS;
+ sq->pdev = c->pdev;
+ sq->mkey_be = c->mkey_be;
+ sq->channel = c;
+ sq->uar_map = mdev->mlx5e_res.bfreg.map;
- sq->db.txq.skb = kzalloc_node(wq_sz * sizeof(*sq->db.txq.skb),
- GFP_KERNEL, numa);
- sq->db.txq.dma_fifo = kzalloc_node(df_sz * sizeof(*sq->db.txq.dma_fifo),
- GFP_KERNEL, numa);
- sq->db.txq.wqe_info = kzalloc_node(wq_sz * sizeof(*sq->db.txq.wqe_info),
- GFP_KERNEL, numa);
- if (!sq->db.txq.skb || !sq->db.txq.dma_fifo || !sq->db.txq.wqe_info) {
- mlx5e_free_sq_txq_db(sq);
- return -ENOMEM;
- }
+ param->wq.db_numa_node = cpu_to_node(c->cpu);
+ err = mlx5_wq_cyc_create(mdev, ¶m->wq, sqc_wq, &sq->wq, &sq->wq_ctrl);
+ if (err)
+ return err;
+ sq->wq.db = &sq->wq.db[MLX5_SND_DBR];
- sq->dma_fifo_mask = df_sz - 1;
+ err = mlx5e_alloc_icosq_db(sq, cpu_to_node(c->cpu));
+ if (err)
+ goto err_sq_wq_destroy;
+
+ sq->edge = (sq->wq.sz_m1 + 1) - MLX5E_ICOSQ_MAX_WQEBBS;
return 0;
+
+err_sq_wq_destroy:
+ mlx5_wq_destroy(&sq->wq_ctrl);
+
+ return err;
}
-static void mlx5e_free_sq_db(struct mlx5e_sq *sq)
+static void mlx5e_free_icosq(struct mlx5e_icosq *sq)
{
- switch (sq->type) {
- case MLX5E_SQ_TXQ:
- mlx5e_free_sq_txq_db(sq);
- break;
- case MLX5E_SQ_ICO:
- mlx5e_free_sq_ico_db(sq);
- break;
- case MLX5E_SQ_XDP:
- mlx5e_free_sq_xdp_db(sq);
- break;
- }
+ mlx5e_free_icosq_db(sq);
+ mlx5_wq_destroy(&sq->wq_ctrl);
}
-static int mlx5e_alloc_sq_db(struct mlx5e_sq *sq, int numa)
+static void mlx5e_free_txqsq_db(struct mlx5e_txqsq *sq)
{
- switch (sq->type) {
- case MLX5E_SQ_TXQ:
- return mlx5e_alloc_sq_txq_db(sq, numa);
- case MLX5E_SQ_ICO:
- return mlx5e_alloc_sq_ico_db(sq, numa);
- case MLX5E_SQ_XDP:
- return mlx5e_alloc_sq_xdp_db(sq, numa);
- }
-
- return 0;
+ kfree(sq->db.wqe_info);
+ kfree(sq->db.dma_fifo);
+ kfree(sq->db.skb);
}
-static int mlx5e_sq_get_max_wqebbs(u8 sq_type)
+static int mlx5e_alloc_txqsq_db(struct mlx5e_txqsq *sq, int numa)
{
- switch (sq_type) {
- case MLX5E_SQ_ICO:
- return MLX5E_ICOSQ_MAX_WQEBBS;
- case MLX5E_SQ_XDP:
- return 1;
+ int wq_sz = mlx5_wq_cyc_get_size(&sq->wq);
+ int df_sz = wq_sz * MLX5_SEND_WQEBB_NUM_DS;
+
+ sq->db.skb = kzalloc_node(wq_sz * sizeof(*sq->db.skb),
+ GFP_KERNEL, numa);
+ sq->db.dma_fifo = kzalloc_node(df_sz * sizeof(*sq->db.dma_fifo),
+ GFP_KERNEL, numa);
+ sq->db.wqe_info = kzalloc_node(wq_sz * sizeof(*sq->db.wqe_info),
+ GFP_KERNEL, numa);
+ if (!sq->db.skb || !sq->db.dma_fifo || !sq->db.wqe_info) {
+ mlx5e_free_txqsq_db(sq);
+ return -ENOMEM;
}
- return MLX5_SEND_WQE_MAX_WQEBBS;
+
+ sq->dma_fifo_mask = df_sz - 1;
+
+ return 0;
}
-static int mlx5e_alloc_sq(struct mlx5e_channel *c,
- int tc,
- struct mlx5e_sq_param *param,
- struct mlx5e_sq *sq)
+static int mlx5e_alloc_txqsq(struct mlx5e_channel *c,
+ int tc,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_txqsq *sq)
{
- struct mlx5e_priv *priv = c->priv;
+ void *sqc_wq = MLX5_ADDR_OF(sqc, param->sqc, wq);
+ struct mlx5e_priv *priv = c->priv;
struct mlx5_core_dev *mdev = priv->mdev;
-
- void *sqc = param->sqc;
- void *sqc_wq = MLX5_ADDR_OF(sqc, sqc, wq);
+ int txq_ix;
int err;
- sq->type = param->type;
sq->pdev = c->pdev;
sq->tstamp = &priv->tstamp;
sq->mkey_be = c->mkey_be;
sq->channel = c;
sq->tc = tc;
sq->uar_map = mdev->mlx5e_res.bfreg.map;
+ sq->max_inline = param->max_inline;
+ sq->min_inline_mode = param->min_inline_mode;
param->wq.db_numa_node = cpu_to_node(c->cpu);
-
- err = mlx5_wq_cyc_create(mdev, ¶m->wq, sqc_wq, &sq->wq,
- &sq->wq_ctrl);
+ err = mlx5_wq_cyc_create(mdev, ¶m->wq, sqc_wq, &sq->wq, &sq->wq_ctrl);
if (err)
return err;
+ sq->wq.db = &sq->wq.db[MLX5_SND_DBR];
- sq->wq.db = &sq->wq.db[MLX5_SND_DBR];
-
- sq->max_inline = param->max_inline;
- sq->min_inline_mode = param->min_inline_mode;
-
- err = mlx5e_alloc_sq_db(sq, cpu_to_node(c->cpu));
+ err = mlx5e_alloc_txqsq_db(sq, cpu_to_node(c->cpu));
if (err)
goto err_sq_wq_destroy;
- if (sq->type == MLX5E_SQ_TXQ) {
- int txq_ix;
-
- txq_ix = c->ix + tc * priv->params.num_channels;
- sq->txq = netdev_get_tx_queue(priv->netdev, txq_ix);
- priv->txq_to_sq_map[txq_ix] = sq;
- }
+ txq_ix = c->ix + tc * priv->params.num_channels;
+ sq->txq = netdev_get_tx_queue(priv->netdev, txq_ix);
+ priv->txq_to_sq_map[txq_ix] = sq;
- sq->edge = (sq->wq.sz_m1 + 1) - mlx5e_sq_get_max_wqebbs(sq->type);
+ sq->edge = (sq->wq.sz_m1 + 1) - MLX5_SEND_WQE_MAX_WQEBBS;
return 0;
@@ -1051,9 +1081,9 @@ static int mlx5e_alloc_sq(struct mlx5e_channel *c,
return err;
}
-static void mlx5e_free_sq(struct mlx5e_sq *sq)
+static void mlx5e_free_txqsq(struct mlx5e_txqsq *sq)
{
- mlx5e_free_sq_db(sq);
+ mlx5e_free_txqsq_db(sq);
mlx5_wq_destroy(&sq->wq_ctrl);
}
@@ -1151,55 +1181,62 @@ static int mlx5e_modify_sq(struct mlx5e_priv *priv,
return err;
}
-
static void mlx5e_destroy_sq(struct mlx5e_priv *priv, u32 sqn)
{
mlx5_core_destroy_sq(priv->mdev, sqn);
}
-static int mlx5e_open_sq(struct mlx5e_channel *c,
- int tc,
- struct mlx5e_sq_param *param,
- struct mlx5e_sq *sq)
+static int mlx5e_create_sq_rdy(struct mlx5e_priv *priv,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_create_sq_param *csp,
+ u32 *sqn)
{
- struct mlx5e_create_sq_param csp = {0};
struct mlx5e_modify_sq_param msp = {0};
+ int err;
+
+ err = mlx5e_create_sq(priv, param, csp, sqn);
+ if (err)
+ return err;
+
+ msp.curr_state = MLX5_SQC_STATE_RST;
+ msp.next_state = MLX5_SQC_STATE_RDY;
+ err = mlx5e_modify_sq(priv, *sqn, &msp);
+ if (err)
+ mlx5e_destroy_sq(priv, *sqn);
+
+ return err;
+}
+
+static int mlx5e_open_txqsq(struct mlx5e_channel *c,
+ int tc,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_txqsq *sq)
+{
+ struct mlx5e_create_sq_param csp = {};
struct mlx5e_priv *priv = c->priv;
int err;
- err = mlx5e_alloc_sq(c, tc, param, sq);
+ err = mlx5e_alloc_txqsq(c, tc, param, sq);
if (err)
return err;
- csp.tisn = param->type == MLX5E_SQ_ICO ? 0 : priv->tisn[sq->tc];
- csp.tis_lst_sz = param->type == MLX5E_SQ_ICO ? 0 : 1;
+ csp.tisn = priv->tisn[sq->tc];
+ csp.tis_lst_sz = 1;
csp.cqn = sq->cq.mcq.cqn;
csp.wq_ctrl = &sq->wq_ctrl;
csp.min_inline_mode = sq->min_inline_mode;
-
- err = mlx5e_create_sq(c->priv, param, &csp, &sq->sqn);
- if (err)
- goto err_free_sq;
-
set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
- msp.curr_state = MLX5_SQC_STATE_RST;
- msp.next_state = MLX5_SQC_STATE_RDY;
- err = mlx5e_modify_sq(priv, sq->sqn, &msp);
+ err = mlx5e_create_sq_rdy(c->priv, param, &csp, &sq->sqn);
if (err)
- goto err_destroy_sq;
-
- if (sq->txq) {
- netdev_tx_reset_queue(sq->txq);
- netif_tx_start_queue(sq->txq);
- }
+ goto err_free_txqsq;
+ netdev_tx_reset_queue(sq->txq);
+ netif_tx_start_queue(sq->txq);
return 0;
-err_destroy_sq:
+err_free_txqsq:
clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
- mlx5e_destroy_sq(priv, sq->sqn);
-err_free_sq:
- mlx5e_free_sq(sq);
+ mlx5e_free_txqsq(sq);
return err;
}
@@ -1211,7 +1248,7 @@ static inline void netif_tx_disable_queue(struct netdev_queue *txq)
__netif_tx_unlock_bh(txq);
}
-static void mlx5e_close_sq(struct mlx5e_sq *sq)
+static void mlx5e_close_txqsq(struct mlx5e_txqsq *sq)
{
struct mlx5e_channel *c = sq->channel;
struct mlx5e_priv *priv = c->priv;
@@ -1221,24 +1258,127 @@ static void mlx5e_close_sq(struct mlx5e_sq *sq)
/* prevent netif_tx_wake_queue */
napi_synchronize(&c->napi);
- if (sq->txq) {
- netif_tx_disable_queue(sq->txq);
+ netif_tx_disable_queue(sq->txq);
- /* last doorbell out, godspeed .. */
- if (mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, 1)) {
- struct mlx5e_tx_wqe *nop;
+ /* last doorbell out, godspeed .. */
+ if (mlx5e_wqc_has_room_for(&sq->wq, sq->cc, sq->pc, 1)) {
+ struct mlx5e_tx_wqe *nop;
- sq->db.txq.skb[(sq->pc & sq->wq.sz_m1)] = NULL;
- nop = mlx5e_post_nop(&sq->wq, sq->sqn, &sq->pc);
- mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &nop->ctrl);
- }
+ sq->db.skb[(sq->pc & sq->wq.sz_m1)] = NULL;
+ nop = mlx5e_post_nop(&sq->wq, sq->sqn, &sq->pc);
+ mlx5e_notify_hw(&sq->wq, sq->pc, sq->uar_map, &nop->ctrl);
}
mlx5e_destroy_sq(priv, sq->sqn);
if (sq->rate_limit)
mlx5_rl_remove_rate(mdev, sq->rate_limit);
- mlx5e_free_sq_descs(sq);
- mlx5e_free_sq(sq);
+ mlx5e_free_txqsq_descs(sq);
+ mlx5e_free_txqsq(sq);
+}
+
+static int mlx5e_open_icosq(struct mlx5e_channel *c,
+ int tc,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_icosq *sq)
+{
+ struct mlx5e_create_sq_param csp = {};
+ int err;
+
+ err = mlx5e_alloc_icosq(c, tc, param, sq);
+ if (err)
+ return err;
+
+ csp.cqn = sq->cq.mcq.cqn;
+ csp.wq_ctrl = &sq->wq_ctrl;
+ csp.min_inline_mode = param->min_inline_mode;
+ set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ err = mlx5e_create_sq_rdy(c->priv, param, &csp, &sq->sqn);
+ if (err)
+ goto err_free_icosq;
+
+ return 0;
+
+err_free_icosq:
+ clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ mlx5e_free_icosq(sq);
+
+ return err;
+}
+
+static void mlx5e_close_icosq(struct mlx5e_icosq *sq)
+{
+ struct mlx5e_channel *c = sq->channel;
+
+ clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ napi_synchronize(&c->napi);
+
+ mlx5e_destroy_sq(c->priv, sq->sqn);
+ mlx5e_free_icosq(sq);
+}
+
+static int mlx5e_open_xdpsq(struct mlx5e_channel *c,
+ struct mlx5e_sq_param *param,
+ struct mlx5e_xdpsq *sq)
+{
+ unsigned int ds_cnt = MLX5E_XDP_TX_DS_COUNT;
+ struct mlx5e_create_sq_param csp = {};
+ struct mlx5e_priv *priv = c->priv;
+ unsigned int inline_hdr_sz = 0;
+ int err;
+ int i;
+
+ err = mlx5e_alloc_xdpsq(c, param, sq);
+ if (err)
+ return err;
+
+ csp.tis_lst_sz = 1;
+ csp.tisn = priv->tisn[0]; /* tc = 0 */
+ csp.cqn = sq->cq.mcq.cqn;
+ csp.wq_ctrl = &sq->wq_ctrl;
+ csp.min_inline_mode = sq->min_inline_mode;
+ set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ err = mlx5e_create_sq_rdy(c->priv, param, &csp, &sq->sqn);
+ if (err)
+ goto err_free_xdpsq;
+
+ if (sq->min_inline_mode != MLX5_INLINE_MODE_NONE) {
+ inline_hdr_sz = MLX5E_XDP_MIN_INLINE;
+ ds_cnt++;
+ }
+
+ /* Pre initialize fixed WQE fields */
+ for (i = 0; i < mlx5_wq_cyc_get_size(&sq->wq); i++) {
+ struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(&sq->wq, i);
+ struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
+ struct mlx5_wqe_eth_seg *eseg = &wqe->eth;
+ struct mlx5_wqe_data_seg *dseg;
+
+ cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt);
+ eseg->inline_hdr.sz = cpu_to_be16(inline_hdr_sz);
+
+ dseg = (struct mlx5_wqe_data_seg *)cseg + (ds_cnt - 1);
+ dseg->lkey = sq->mkey_be;
+ }
+
+ return 0;
+
+err_free_xdpsq:
+ clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ mlx5e_free_xdpsq(sq);
+
+ return err;
+}
+
+static void mlx5e_close_xdpsq(struct mlx5e_xdpsq *sq)
+{
+ struct mlx5e_channel *c = sq->channel;
+
+ clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
+ napi_synchronize(&c->napi);
+
+ mlx5e_destroy_sq(c->priv, sq->sqn);
+ mlx5e_free_xdpsq_descs(sq);
+ mlx5e_free_xdpsq(sq);
}
static int mlx5e_alloc_cq(struct mlx5e_channel *c,
@@ -1426,7 +1566,7 @@ static int mlx5e_open_sqs(struct mlx5e_channel *c,
int tc;
for (tc = 0; tc < c->num_tc; tc++) {
- err = mlx5e_open_sq(c, tc, &cparam->sq, &c->sq[tc]);
+ err = mlx5e_open_txqsq(c, tc, &cparam->sq, &c->sq[tc]);
if (err)
goto err_close_sqs;
}
@@ -1435,7 +1575,7 @@ static int mlx5e_open_sqs(struct mlx5e_channel *c,
err_close_sqs:
for (tc--; tc >= 0; tc--)
- mlx5e_close_sq(&c->sq[tc]);
+ mlx5e_close_txqsq(&c->sq[tc]);
return err;
}
@@ -1445,7 +1585,7 @@ static void mlx5e_close_sqs(struct mlx5e_channel *c)
int tc;
for (tc = 0; tc < c->num_tc; tc++)
- mlx5e_close_sq(&c->sq[tc]);
+ mlx5e_close_txqsq(&c->sq[tc]);
}
static void mlx5e_build_channeltc_to_txq_map(struct mlx5e_priv *priv, int ix)
@@ -1458,7 +1598,7 @@ static void mlx5e_build_channeltc_to_txq_map(struct mlx5e_priv *priv, int ix)
}
static int mlx5e_set_sq_maxrate(struct net_device *dev,
- struct mlx5e_sq *sq, u32 rate)
+ struct mlx5e_txqsq *sq, u32 rate)
{
struct mlx5e_priv *priv = netdev_priv(dev);
struct mlx5_core_dev *mdev = priv->mdev;
@@ -1507,7 +1647,7 @@ static int mlx5e_set_tx_maxrate(struct net_device *dev, int index, u32 rate)
{
struct mlx5e_priv *priv = netdev_priv(dev);
struct mlx5_core_dev *mdev = priv->mdev;
- struct mlx5e_sq *sq = priv->txq_to_sq_map[index];
+ struct mlx5e_txqsq *sq = priv->txq_to_sq_map[index];
int err = 0;
if (!mlx5_rl_is_supported(mdev)) {
@@ -1542,40 +1682,6 @@ static inline int mlx5e_get_max_num_channels(struct mlx5_core_dev *mdev)
MLX5E_MAX_NUM_CHANNELS);
}
-static int mlx5e_open_xdpsq(struct mlx5e_channel *c,
- struct mlx5e_sq_param *param,
- struct mlx5e_sq *sq)
-{
- unsigned int ds_cnt = MLX5E_XDP_TX_DS_COUNT;
- unsigned int inline_hdr_sz = 0;
- int err;
- int i;
-
- err = mlx5e_open_sq(c, 0, param, sq);
- if (err)
- return err;
-
- if (sq->min_inline_mode != MLX5_INLINE_MODE_NONE) {
- inline_hdr_sz = MLX5E_XDP_MIN_INLINE;
- ds_cnt++;
- }
-
- /* Pre initialize fixed WQE fields */
- for (i = 0; i < mlx5_wq_cyc_get_size(&sq->wq); i++) {
- struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(&sq->wq, i);
- struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
- struct mlx5_wqe_eth_seg *eseg = &wqe->eth;
- struct mlx5_wqe_data_seg *dseg;
-
- cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt);
- eseg->inline_hdr.sz = cpu_to_be16(inline_hdr_sz);
-
- dseg = (struct mlx5_wqe_data_seg *)cseg + (ds_cnt - 1);
- dseg->lkey = sq->mkey_be;
- }
- return 0;
-}
-
static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
struct mlx5e_channel_param *cparam,
struct mlx5e_channel **cp)
@@ -1585,7 +1691,6 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
struct mlx5e_cq_moder rx_cq_profile;
int cpu = mlx5e_get_cpu(priv, ix);
struct mlx5e_channel *c;
- struct mlx5e_sq *sq;
int err;
int i;
@@ -1632,7 +1737,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
napi_enable(&c->napi);
- err = mlx5e_open_sq(c, 0, &cparam->icosq, &c->icosq);
+ err = mlx5e_open_icosq(c, 0, &cparam->icosq, &c->icosq);
if (err)
goto err_disable_napi;
@@ -1644,7 +1749,8 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
u32 txq_ix = priv->channeltc_to_txq_map[ix][i];
if (priv->tx_rates[txq_ix]) {
- sq = priv->txq_to_sq_map[txq_ix];
+ struct mlx5e_txqsq *sq = priv->txq_to_sq_map[txq_ix];
+
mlx5e_set_sq_maxrate(priv->netdev, sq,
priv->tx_rates[txq_ix]);
}
@@ -1664,13 +1770,13 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
return 0;
err_close_xdp_sq:
if (c->xdp)
- mlx5e_close_sq(&c->rq.xdpsq);
+ mlx5e_close_xdpsq(&c->rq.xdpsq);
err_close_sqs:
mlx5e_close_sqs(c);
err_close_icosq:
- mlx5e_close_sq(&c->icosq);
+ mlx5e_close_icosq(&c->icosq);
err_disable_napi:
napi_disable(&c->napi);
@@ -1697,9 +1803,9 @@ static void mlx5e_close_channel(struct mlx5e_channel *c)
{
mlx5e_close_rq(&c->rq);
if (c->xdp)
- mlx5e_close_sq(&c->rq.xdpsq);
+ mlx5e_close_xdpsq(&c->rq.xdpsq);
mlx5e_close_sqs(c);
- mlx5e_close_sq(&c->icosq);
+ mlx5e_close_icosq(&c->icosq);
napi_disable(&c->napi);
if (c->xdp)
mlx5e_close_cq(&c->rq.xdpsq.cq);
@@ -1773,7 +1879,6 @@ static void mlx5e_build_sq_param(struct mlx5e_priv *priv,
param->max_inline = priv->params.tx_max_inline;
param->min_inline_mode = priv->params.tx_min_inline_mode;
- param->type = MLX5E_SQ_TXQ;
}
static void mlx5e_build_common_cq_param(struct mlx5e_priv *priv,
@@ -1846,8 +1951,6 @@ static void mlx5e_build_icosq_param(struct mlx5e_priv *priv,
MLX5_SET(wq, wq, log_wq_sz, log_wq_size);
MLX5_SET(sqc, sqc, reg_umr, MLX5_CAP_ETH(priv->mdev, reg_umr_sq));
-
- param->type = MLX5E_SQ_ICO;
}
static void mlx5e_build_xdpsq_param(struct mlx5e_priv *priv,
@@ -1861,7 +1964,6 @@ static void mlx5e_build_xdpsq_param(struct mlx5e_priv *priv,
param->max_inline = priv->params.tx_max_inline;
param->min_inline_mode = priv->params.tx_min_inline_mode;
- param->type = MLX5E_SQ_XDP;
}
static void mlx5e_build_channel_param(struct mlx5e_priv *priv, struct mlx5e_channel_param *cparam)
@@ -3237,7 +3339,7 @@ static void mlx5e_tx_timeout(struct net_device *dev)
netdev_err(dev, "TX timeout detected\n");
for (i = 0; i < priv->params.num_channels * priv->params.num_tc; i++) {
- struct mlx5e_sq *sq = priv->txq_to_sq_map[i];
+ struct mlx5e_txqsq *sq = priv->txq_to_sq_map[i];
if (!netif_xmit_stopped(netdev_get_tx_queue(dev, i)))
continue;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 141dcc486063..3ecbe8c2d5e3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -331,7 +331,7 @@ mlx5e_copy_skb_header_mpwqe(struct device *pdev,
static inline void mlx5e_post_umr_wqe(struct mlx5e_rq *rq, u16 ix)
{
struct mlx5e_mpw_info *wi = &rq->mpwqe.info[ix];
- struct mlx5e_sq *sq = &rq->channel->icosq;
+ struct mlx5e_icosq *sq = &rq->channel->icosq;
struct mlx5_wq_cyc *wq = &sq->wq;
struct mlx5e_umr_wqe *wqe;
u8 num_wqebbs = DIV_ROUND_UP(sizeof(*wqe), MLX5_SEND_WQE_BB);
@@ -342,7 +342,6 @@ static inline void mlx5e_post_umr_wqe(struct mlx5e_rq *rq, u16 ix)
sq->db.ico_wqe[pi].opcode = MLX5_OPCODE_NOP;
sq->db.ico_wqe[pi].num_wqebbs = 1;
mlx5e_post_nop(wq, sq->sqn, &sq->pc);
- sq->stats.nop++;
}
wqe = mlx5_wq_cyc_get_wqe(wq, pi);
@@ -638,7 +637,7 @@ static inline void mlx5e_complete_rx_cqe(struct mlx5e_rq *rq,
mlx5e_build_rx_skb(cqe, cqe_bcnt, rq, skb);
}
-static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_sq *sq)
+static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_xdpsq *sq)
{
struct mlx5_wq_cyc *wq = &sq->wq;
struct mlx5e_tx_wqe *wqe;
@@ -653,9 +652,9 @@ static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
struct mlx5e_dma_info *di,
const struct xdp_buff *xdp)
{
- struct mlx5e_sq *sq = &rq->xdpsq;
+ struct mlx5e_xdpsq *sq = &rq->xdpsq;
struct mlx5_wq_cyc *wq = &sq->wq;
- u16 pi = sq->pc & wq->sz_m1;
+ u16 pi = sq->pc & wq->sz_m1;
struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
@@ -676,10 +675,10 @@ static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
}
if (unlikely(!mlx5e_wqc_has_room_for(wq, sq->cc, sq->pc, 1))) {
- if (sq->db.xdp.doorbell) {
+ if (sq->db.doorbell) {
/* SQ is full, ring doorbell */
mlx5e_xmit_xdp_doorbell(sq);
- sq->db.xdp.doorbell = false;
+ sq->db.doorbell = false;
}
rq->stats.xdp_tx_full++;
mlx5e_page_release(rq, di, true);
@@ -707,10 +706,10 @@ static inline bool mlx5e_xmit_xdp_frame(struct mlx5e_rq *rq,
cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_SEND);
- sq->db.xdp.di[pi] = *di;
+ sq->db.di[pi] = *di;
sq->pc++;
- sq->db.xdp.doorbell = true;
+ sq->db.doorbell = true;
rq->stats.xdp_tx++;
return true;
}
@@ -944,7 +943,7 @@ void mlx5e_handle_rx_cqe_mpwrq(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe)
int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)
{
struct mlx5e_rq *rq = container_of(cq, struct mlx5e_rq, cq);
- struct mlx5e_sq *xdpsq = &rq->xdpsq;
+ struct mlx5e_xdpsq *xdpsq = &rq->xdpsq;
int work_done = 0;
if (unlikely(!test_bit(MLX5E_RQ_STATE_ENABLED, &rq->state)))
@@ -971,9 +970,9 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)
rq->handle_rx_cqe(rq, cqe);
}
- if (xdpsq->db.xdp.doorbell) {
+ if (xdpsq->db.doorbell) {
mlx5e_xmit_xdp_doorbell(xdpsq);
- xdpsq->db.xdp.doorbell = false;
+ xdpsq->db.doorbell = false;
}
mlx5_cqwq_update_db_record(&cq->wq);
@@ -986,12 +985,12 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget)
bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
{
- struct mlx5e_sq *sq;
+ struct mlx5e_xdpsq *sq;
struct mlx5e_rq *rq;
u16 sqcc;
int i;
- sq = container_of(cq, struct mlx5e_sq, cq);
+ sq = container_of(cq, struct mlx5e_xdpsq, cq);
if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &sq->state)))
return false;
@@ -1023,7 +1022,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
last_wqe = (sqcc == wqe_counter);
ci = sqcc & sq->wq.sz_m1;
- di = &sq->db.xdp.di[ci];
+ di = &sq->db.di[ci];
sqcc++;
/* Recycle RX page */
@@ -1040,7 +1039,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq)
return (i == MLX5E_TX_CQ_POLL_BUDGET);
}
-void mlx5e_free_xdpsq_descs(struct mlx5e_sq *sq)
+void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq)
{
struct mlx5e_rq *rq = container_of(sq, struct mlx5e_rq, xdpsq);
struct mlx5e_dma_info *di;
@@ -1048,7 +1047,7 @@ void mlx5e_free_xdpsq_descs(struct mlx5e_sq *sq)
while (sq->cc != sq->pc) {
ci = sq->cc & sq->wq.sz_m1;
- di = &sq->db.xdp.di[ci];
+ di = &sq->db.di[ci];
sq->cc++;
mlx5e_page_release(rq, di, false);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
index 897eaea6f51f..20f71b55651e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c
@@ -53,25 +53,25 @@ static inline void mlx5e_tx_dma_unmap(struct device *pdev,
}
}
-static inline void mlx5e_dma_push(struct mlx5e_sq *sq,
+static inline void mlx5e_dma_push(struct mlx5e_txqsq *sq,
dma_addr_t addr,
u32 size,
enum mlx5e_dma_map_type map_type)
{
u32 i = sq->dma_fifo_pc & sq->dma_fifo_mask;
- sq->db.txq.dma_fifo[i].addr = addr;
- sq->db.txq.dma_fifo[i].size = size;
- sq->db.txq.dma_fifo[i].type = map_type;
+ sq->db.dma_fifo[i].addr = addr;
+ sq->db.dma_fifo[i].size = size;
+ sq->db.dma_fifo[i].type = map_type;
sq->dma_fifo_pc++;
}
-static inline struct mlx5e_sq_dma *mlx5e_dma_get(struct mlx5e_sq *sq, u32 i)
+static inline struct mlx5e_sq_dma *mlx5e_dma_get(struct mlx5e_txqsq *sq, u32 i)
{
- return &sq->db.txq.dma_fifo[i & sq->dma_fifo_mask];
+ return &sq->db.dma_fifo[i & sq->dma_fifo_mask];
}
-static void mlx5e_dma_unmap_wqe_err(struct mlx5e_sq *sq, u8 num_dma)
+static void mlx5e_dma_unmap_wqe_err(struct mlx5e_txqsq *sq, u8 num_dma)
{
int i;
@@ -176,13 +176,13 @@ static inline void mlx5e_insert_vlan(void *start, struct sk_buff *skb, u16 ihs,
mlx5e_tx_skb_pull_inline(skb_data, skb_len, cpy2_sz);
}
-static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
+static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_txqsq *sq, struct sk_buff *skb)
{
struct mlx5_wq_cyc *wq = &sq->wq;
u16 pi = sq->pc & wq->sz_m1;
struct mlx5e_tx_wqe *wqe = mlx5_wq_cyc_get_wqe(wq, pi);
- struct mlx5e_tx_wqe_info *wi = &sq->db.txq.wqe_info[pi];
+ struct mlx5e_tx_wqe_info *wi = &sq->db.wqe_info[pi];
struct mlx5_wqe_ctrl_seg *cseg = &wqe->ctrl;
struct mlx5_wqe_eth_seg *eseg = &wqe->eth;
@@ -298,7 +298,7 @@ static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | opcode);
cseg->qpn_ds = cpu_to_be32((sq->sqn << 8) | ds_cnt);
- sq->db.txq.skb[pi] = skb;
+ sq->db.skb[pi] = skb;
wi->num_wqebbs = DIV_ROUND_UP(ds_cnt, MLX5_SEND_WQEBB_NUM_DS);
sq->pc += wi->num_wqebbs;
@@ -320,7 +320,7 @@ static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
/* fill sq edge with nops to avoid wqe wrap around */
while ((pi = (sq->pc & wq->sz_m1)) > sq->edge) {
- sq->db.txq.skb[pi] = NULL;
+ sq->db.skb[pi] = NULL;
mlx5e_post_nop(&sq->wq, sq->sqn, &sq->pc);
sq->stats.nop++;
}
@@ -339,21 +339,21 @@ static netdev_tx_t mlx5e_sq_xmit(struct mlx5e_sq *sq, struct sk_buff *skb)
netdev_tx_t mlx5e_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct mlx5e_priv *priv = netdev_priv(dev);
- struct mlx5e_sq *sq = priv->txq_to_sq_map[skb_get_queue_mapping(skb)];
+ struct mlx5e_txqsq *sq = priv->txq_to_sq_map[skb_get_queue_mapping(skb)];
return mlx5e_sq_xmit(sq, skb);
}
bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget)
{
- struct mlx5e_sq *sq;
+ struct mlx5e_txqsq *sq;
u32 dma_fifo_cc;
u32 nbytes;
u16 npkts;
u16 sqcc;
int i;
- sq = container_of(cq, struct mlx5e_sq, cq);
+ sq = container_of(cq, struct mlx5e_txqsq, cq);
if (unlikely(!test_bit(MLX5E_SQ_STATE_ENABLED, &sq->state)))
return false;
@@ -391,8 +391,8 @@ bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget)
last_wqe = (sqcc == wqe_counter);
ci = sqcc & sq->wq.sz_m1;
- skb = sq->db.txq.skb[ci];
- wi = &sq->db.txq.wqe_info[ci];
+ skb = sq->db.skb[ci];
+ wi = &sq->db.wqe_info[ci];
if (unlikely(!skb)) { /* nop */
sqcc++;
@@ -441,7 +441,7 @@ bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget)
return (i == MLX5E_TX_CQ_POLL_BUDGET);
}
-static void mlx5e_free_txq_sq_descs(struct mlx5e_sq *sq)
+void mlx5e_free_txqsq_descs(struct mlx5e_txqsq *sq)
{
struct mlx5e_tx_wqe_info *wi;
struct sk_buff *skb;
@@ -450,8 +450,8 @@ static void mlx5e_free_txq_sq_descs(struct mlx5e_sq *sq)
while (sq->cc != sq->pc) {
ci = sq->cc & sq->wq.sz_m1;
- skb = sq->db.txq.skb[ci];
- wi = &sq->db.txq.wqe_info[ci];
+ skb = sq->db.skb[ci];
+ wi = &sq->db.wqe_info[ci];
if (!skb) { /* nop */
sq->cc++;
@@ -469,15 +469,3 @@ static void mlx5e_free_txq_sq_descs(struct mlx5e_sq *sq)
sq->cc += wi->num_wqebbs;
}
}
-
-void mlx5e_free_sq_descs(struct mlx5e_sq *sq)
-{
- switch (sq->type) {
- case MLX5E_SQ_TXQ:
- mlx5e_free_txq_sq_descs(sq);
- break;
- case MLX5E_SQ_XDP:
- mlx5e_free_xdpsq_descs(sq);
- break;
- }
-}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index c880022bb21a..3317ef561a75 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -51,7 +51,7 @@ struct mlx5_cqe64 *mlx5e_get_cqe(struct mlx5e_cq *cq)
static void mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
{
- struct mlx5e_sq *sq = container_of(cq, struct mlx5e_sq, cq);
+ struct mlx5e_icosq *sq = container_of(cq, struct mlx5e_icosq, cq);
struct mlx5_wq_cyc *wq;
struct mlx5_cqe64 *cqe;
u16 sqcc;
--
2.11.0
^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (11 preceding siblings ...)
2017-03-24 21:52 ` [PATCH net-next 12/12] net/mlx5e: Different SQ types Saeed Mahameed
@ 2017-03-24 23:26 ` Alexei Starovoitov
2017-03-25 12:30 ` Saeed Mahameed
2017-03-25 2:12 ` David Miller
2017-03-25 16:54 ` Tom Herbert
14 siblings, 1 reply; 18+ messages in thread
From: Alexei Starovoitov @ 2017-03-24 23:26 UTC (permalink / raw)
To: Saeed Mahameed, David S. Miller; +Cc: netdev, kernel-team
On 3/24/17 2:52 PM, Saeed Mahameed wrote:
> Hi Dave,
>
> This series provides some preformancee optimizations for mlx5e
> driver, especially for XDP TX flows.
>
> 1st patch is a simple change of rmb to dma_rmb in CQE fetch routine
> which shows a huge gain for both RX and TX packet rates.
>
> 2nd patch removes write combining logic from the driver TX handler
> and simplifies the TX logic while improving TX CPU utilization.
>
> All other patches combined provide some refactoring to the driver TX
> flows to allow some significant XDP TX improvements.
>
> More details and performance numbers per patch can be found in each patch
> commit message compared to the preceding patch.
>
> Overall performance improvemnets
> System: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>
> Test case Baseline Now improvement
> ---------------------------------------------------------------
> TX packets (24 threads) 45Mpps 54Mpps 20%
> TC stack Drop (1 core) 3.45Mpps 3.6Mpps 5%
> XDP Drop (1 core) 14Mpps 16.9Mpps 20%
> XDP TX (1 core) 10.4Mpps 13.7Mpps 31%
Excellent work!
All patches look great, so for the series:
Acked-by: Alexei Starovoitov <ast@kernel.org>
in patch 12 I noticed that inline_mode is being evaluated.
I think for xdp queues it's guaranteed to be fixed.
Can we optimize that path little bit more as well?
Thanks!
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (12 preceding siblings ...)
2017-03-24 23:26 ` [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Alexei Starovoitov
@ 2017-03-25 2:12 ` David Miller
2017-03-25 16:54 ` Tom Herbert
14 siblings, 0 replies; 18+ messages in thread
From: David Miller @ 2017-03-25 2:12 UTC (permalink / raw)
To: saeedm; +Cc: netdev, kernel-team
From: Saeed Mahameed <saeedm@mellanox.com>
Date: Sat, 25 Mar 2017 00:52:02 +0300
> This series provides some preformancee optimizations for mlx5e
> driver, especially for XDP TX flows.
This looks really great, series applied, thanks!
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization
2017-03-24 23:26 ` [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Alexei Starovoitov
@ 2017-03-25 12:30 ` Saeed Mahameed
0 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-25 12:30 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Saeed Mahameed, David S. Miller, Linux Netdev List, Kernel Team
On Sat, Mar 25, 2017 at 2:26 AM, Alexei Starovoitov <ast@fb.com> wrote:
> On 3/24/17 2:52 PM, Saeed Mahameed wrote:
>>
>> Hi Dave,
>>
>> This series provides some preformancee optimizations for mlx5e
>> driver, especially for XDP TX flows.
>>
>> 1st patch is a simple change of rmb to dma_rmb in CQE fetch routine
>> which shows a huge gain for both RX and TX packet rates.
>>
>> 2nd patch removes write combining logic from the driver TX handler
>> and simplifies the TX logic while improving TX CPU utilization.
>>
>> All other patches combined provide some refactoring to the driver TX
>> flows to allow some significant XDP TX improvements.
>>
>> More details and performance numbers per patch can be found in each patch
>> commit message compared to the preceding patch.
>>
>> Overall performance improvemnets
>> System: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>>
>> Test case Baseline Now improvement
>> ---------------------------------------------------------------
>> TX packets (24 threads) 45Mpps 54Mpps 20%
>> TC stack Drop (1 core) 3.45Mpps 3.6Mpps 5%
>> XDP Drop (1 core) 14Mpps 16.9Mpps 20%
>> XDP TX (1 core) 10.4Mpps 13.7Mpps 31%
>
>
> Excellent work!
> All patches look great, so for the series:
> Acked-by: Alexei Starovoitov <ast@kernel.org>
>
Thanks Alexei !
> in patch 12 I noticed that inline_mode is being evaluated.
> I think for xdp queues it's guaranteed to be fixed.
> Can we optimize that path little bit more as well?
Yes, you are right, we do evaluate it in mlx5e_alloc_xdpsq
+ if (sq->min_inline_mode != MLX5_INLINE_MODE_NONE) {
+ inline_hdr_sz = MLX5E_XDP_MIN_INLINE;
+ ds_cnt++;
+ }
and check it again in mlx5e_xmit_xdp_frame
+ /* copy the inline part if required */
+ if (sq->min_inline_mode != MLX5_INLINE_MODE_NONE) {
sq->min_inline_mode is fixed in run-time, but it is different across
HW versions.
This condition is needed so we would not copy inline headers and waste
CPU cycles while it is not required from ConnectX-5 and later.
Actually this is a 5% XDP_TX optimization you get when you run over
ConnectX-5 [1].
in ConnectX-4 and 4-LX driver is still required to copy L2 headers
into TX descriptor so the HW can make the loopback decision correctly
(needed in case you want XDP program to switch packets between
different PFs/VFs running on the same box/NIC).
So i don't see anyway to do this without breaking XDP loopback
functionality or removing the connectX-5 optimization.
for my taste this condition is good as is.
[1] https://www.spinics.net/lists/netdev/msg419215.html
> Thanks!
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
` (13 preceding siblings ...)
2017-03-25 2:12 ` David Miller
@ 2017-03-25 16:54 ` Tom Herbert
2017-03-26 9:16 ` Saeed Mahameed
14 siblings, 1 reply; 18+ messages in thread
From: Tom Herbert @ 2017-03-25 16:54 UTC (permalink / raw)
To: Saeed Mahameed
Cc: David S. Miller, Linux Kernel Network Developers, Kernel Team
On Fri, Mar 24, 2017 at 2:52 PM, Saeed Mahameed <saeedm@mellanox.com> wrote:
> Hi Dave,
>
> This series provides some preformancee optimizations for mlx5e
> driver, especially for XDP TX flows.
>
> 1st patch is a simple change of rmb to dma_rmb in CQE fetch routine
> which shows a huge gain for both RX and TX packet rates.
>
> 2nd patch removes write combining logic from the driver TX handler
> and simplifies the TX logic while improving TX CPU utilization.
>
> All other patches combined provide some refactoring to the driver TX
> flows to allow some significant XDP TX improvements.
>
> More details and performance numbers per patch can be found in each patch
> commit message compared to the preceding patch.
>
> Overall performance improvemnets
> System: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>
> Test case Baseline Now improvement
> ---------------------------------------------------------------
> TX packets (24 threads) 45Mpps 54Mpps 20%
> TC stack Drop (1 core) 3.45Mpps 3.6Mpps 5%
> XDP Drop (1 core) 14Mpps 16.9Mpps 20%
> XDP TX (1 core) 10.4Mpps 13.7Mpps 31%
>
Awesome, and good timing. I'll be presenting XDP at IETF next and
would like to include these numbers in the presentation if you don't
mind...
Tom
> Thanks,
> Saeed.
>
> Saeed Mahameed (12):
> net/mlx5e: Use dma_rmb rather than rmb in CQE fetch routine
> net/mlx5e: Xmit, no write combining
> net/mlx5e: Single bfreg (UAR) for all mlx5e SQs and netdevs
> net/mlx5e: Move XDP completion functions to rx file
> net/mlx5e: Move mlx5e_rq struct declaration
> net/mlx5e: Move XDP SQ instance into RQ
> net/mlx5e: Poll XDP TX CQ before RX CQ
> net/mlx5e: Optimize XDP frame xmit
> net/mlx5e: Generalize tx helper functions for different SQ types
> net/mlx5e: Proper names for SQ/RQ/CQ functions
> net/mlx5e: Generalize SQ create/modify/destroy functions
> net/mlx5e: Different SQ types
>
> drivers/net/ethernet/mellanox/mlx5/core/en.h | 319 +++++-----
> .../net/ethernet/mellanox/mlx5/core/en_common.c | 9 +
> drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 644 +++++++++++++--------
> drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 124 +++-
> drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 147 +----
> drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 70 +--
> include/linux/mlx5/driver.h | 1 +
> 7 files changed, 716 insertions(+), 598 deletions(-)
>
> --
> 2.11.0
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization
2017-03-25 16:54 ` Tom Herbert
@ 2017-03-26 9:16 ` Saeed Mahameed
0 siblings, 0 replies; 18+ messages in thread
From: Saeed Mahameed @ 2017-03-26 9:16 UTC (permalink / raw)
To: Tom Herbert
Cc: Saeed Mahameed, David S. Miller, Linux Kernel Network Developers,
Kernel Team
On Sat, Mar 25, 2017 at 6:54 PM, Tom Herbert <tom@herbertland.com> wrote:
> On Fri, Mar 24, 2017 at 2:52 PM, Saeed Mahameed <saeedm@mellanox.com> wrote:
>> Hi Dave,
>>
>> This series provides some preformancee optimizations for mlx5e
>> driver, especially for XDP TX flows.
>>
>> 1st patch is a simple change of rmb to dma_rmb in CQE fetch routine
>> which shows a huge gain for both RX and TX packet rates.
>>
>> 2nd patch removes write combining logic from the driver TX handler
>> and simplifies the TX logic while improving TX CPU utilization.
>>
>> All other patches combined provide some refactoring to the driver TX
>> flows to allow some significant XDP TX improvements.
>>
>> More details and performance numbers per patch can be found in each patch
>> commit message compared to the preceding patch.
>>
>> Overall performance improvemnets
>> System: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>>
>> Test case Baseline Now improvement
>> ---------------------------------------------------------------
>> TX packets (24 threads) 45Mpps 54Mpps 20%
>> TC stack Drop (1 core) 3.45Mpps 3.6Mpps 5%
>> XDP Drop (1 core) 14Mpps 16.9Mpps 20%
>> XDP TX (1 core) 10.4Mpps 13.7Mpps 31%
>>
> Awesome, and good timing. I'll be presenting XDP at IETF next and
> would like to include these numbers in the presentation if you don't
> mind...
>
Not at all, please go ahead.
But as you see, the system i tested on is not that powerful. We can
get even better results with a modern system.
If you want i can provide you those numbers by mid-week.
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2017-03-26 9:17 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-24 21:52 [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 01/12] net/mlx5e: Use dma_rmb rather than rmb in CQE fetch routine Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 02/12] net/mlx5e: Xmit, no write combining Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 03/12] net/mlx5e: Single bfreg (UAR) for all mlx5e SQs and netdevs Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 04/12] net/mlx5e: Move XDP completion functions to rx file Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 05/12] net/mlx5e: Move mlx5e_rq struct declaration Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 06/12] net/mlx5e: Move XDP SQ instance into RQ Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 07/12] net/mlx5e: Poll XDP TX CQ before RX CQ Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 08/12] net/mlx5e: Optimize XDP frame xmit Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 09/12] net/mlx5e: Generalize tx helper functions for different SQ types Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 10/12] net/mlx5e: Proper names for SQ/RQ/CQ functions Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 11/12] net/mlx5e: Generalize SQ create/modify/destroy functions Saeed Mahameed
2017-03-24 21:52 ` [PATCH net-next 12/12] net/mlx5e: Different SQ types Saeed Mahameed
2017-03-24 23:26 ` [PATCH net-next 00/12] Mellanox mlx5e XDP performance optimization Alexei Starovoitov
2017-03-25 12:30 ` Saeed Mahameed
2017-03-25 2:12 ` David Miller
2017-03-25 16:54 ` Tom Herbert
2017-03-26 9:16 ` Saeed Mahameed
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.