* [RFC 0/9] net: create adaptive software irq moderation library
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 1/9] mlx5_en: move interrupt moderation structs to new file Andy Gospodarek
` (12 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
This RFC converts the adaptive interrupt moderation library from the
mlx5_en driver into a library so it can be used by any driver. The last
patch in this set adds support for interrupt moderation in the bnxt_en
driver.
The main purpose of this code in the mlx5 driver is to allow an administrator to make sure that default coalesce settings are optimized for low latency, but quickly adapt to handle high throughput traffic and
optimize how many packets are received during each napi poll.
For any new driver the following changes would ne needed to use this
library:
- add elements in ring struct to track items needed by this library
- create function that can be called to actually set coalesce settings
for the driver
My main reason for making this an RFC is that I would like verification
from Mellanox that the performance of their driver does not change in a
unintended way. I did some basic testing (netperf) and did not note a
statistically significant change in throughput or CPU utilization before
and after this set.
Andy Gospodarek (9):
mlx5_en: move interrupt moderation structs to new file
mlx5_en: move interrupt moderation forward delcarations
mlx5_en: remove rq references in mlx5e_rx_am
mlx5_en: move AM logic enums
mlx5_en: move generic functions to new file
mlx5_en: rename en_rx_am.h to net_rx_am.h
mlx5_en: remove Mellanox references in AM code
net: move adaptive interrpt coalescing code to lib/
bnxt_en: add support for software adaptive interrupt moderation
drivers/net/ethernet/broadcom/bnxt/Makefile | 2 +-
drivers/net/ethernet/broadcom/bnxt/bnxt.c | 51 ++++
drivers/net/ethernet/broadcom/bnxt/bnxt.h | 34 ++-
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 7 +
drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c | 32 +++
drivers/net/ethernet/mellanox/mlx5/core/en.h | 43 +--
.../net/ethernet/mellanox/mlx5/core/en_ethtool.c | 6 +-
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 18 +-
drivers/net/ethernet/mellanox/mlx5/core/en_rep.c | 4 +-
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c | 298 +-------------------
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 5 +-
include/linux/mlx5/mlx5_ifc.h | 6 -
include/linux/net_rx_am.h | 109 ++++++++
lib/Makefile | 2 +-
lib/net_rx_am.c | 306 +++++++++++++++++++++
15 files changed, 558 insertions(+), 365 deletions(-)
create mode 100644 drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c
create mode 100644 include/linux/net_rx_am.h
create mode 100644 lib/net_rx_am.c
--
2.7.4
^ permalink raw reply [flat|nested] 17+ messages in thread* [RFC 1/9] mlx5_en: move interrupt moderation structs to new file
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
2017-11-06 5:44 ` Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 2/9] mlx5_en: move interrupt moderation forward declarations Andy Gospodarek
` (11 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
Create new header file to prepare to move code that handles irq
moderation to a library.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 32 +---------
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h | 74 ++++++++++++++++++++++
include/linux/mlx5/mlx5_ifc.h | 6 --
3 files changed, 75 insertions(+), 37 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index e613ce0..1bde086 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -49,6 +49,7 @@
#include "wq.h"
#include "mlx5_core.h"
#include "en_stats.h"
+#include "en_rx_am.h"
#define MLX5_SET_CFG(p, f, v) MLX5_SET(create_flow_group_in, p, f, v)
@@ -209,11 +210,6 @@ enum mlx5e_priv_flag {
#define MLX5E_MAX_BW_ALLOC 100 /* Max percentage of BW allocation */
#endif
-struct mlx5e_cq_moder {
- u16 usec;
- u16 pkts;
-};
-
struct mlx5e_params {
u8 log_sq_size;
u8 rq_wq_type;
@@ -449,32 +445,6 @@ struct mlx5e_mpw_info {
u16 skbs_frags[MLX5_MPWRQ_PAGES_PER_WQE];
};
-struct mlx5e_rx_am_stats {
- int ppms; /* packets per msec */
- int bpms; /* bytes per msec */
- int epms; /* events per msec */
-};
-
-struct mlx5e_rx_am_sample {
- ktime_t time;
- u32 pkt_ctr;
- u32 byte_ctr;
- u16 event_ctr;
-};
-
-struct mlx5e_rx_am { /* Adaptive Moderation */
- u8 state;
- struct mlx5e_rx_am_stats prev_stats;
- struct mlx5e_rx_am_sample start_sample;
- struct work_struct work;
- u8 profile_ix;
- u8 mode;
- u8 tune_state;
- u8 steps_right;
- u8 steps_left;
- u8 tired;
-};
-
/* a single cache unit is capable to serve one napi call (for non-striding rq)
* or a MPWQE (for striding rq).
*/
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
new file mode 100644
index 0000000..176a732
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
@@ -0,0 +1,74 @@
+/*
+ * Copyright (c) 2013-2015, Mellanox Technologies, Ltd. All rights reserved.
+ * Copyright (c) 2017, Broadcom Limited
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+*/
+
+#ifndef MLX5_AM_H
+#define MLX5_AM_H
+
+struct mlx5e_cq_moder {
+ u16 usec;
+ u16 pkts;
+};
+
+struct mlx5e_rx_am_sample {
+ ktime_t time;
+ u32 pkt_ctr;
+ u32 byte_ctr;
+ u16 event_ctr;
+};
+
+struct mlx5e_rx_am_stats {
+ int ppms; /* packets per msec */
+ int bpms; /* bytes per msec */
+ int epms; /* events per msec */
+};
+
+struct mlx5e_rx_am { /* Adaptive Moderation */
+ u8 state;
+ struct mlx5e_rx_am_stats prev_stats;
+ struct mlx5e_rx_am_sample start_sample;
+ struct work_struct work;
+ u8 profile_ix;
+ u8 mode;
+ u8 tune_state;
+ u8 steps_right;
+ u8 steps_left;
+ u8 tired;
+};
+
+enum {
+ MLX5_CQ_PERIOD_MODE_START_FROM_EQE = 0x0,
+ MLX5_CQ_PERIOD_MODE_START_FROM_CQE = 0x1,
+ MLX5_CQ_PERIOD_NUM_MODES
+};
+
+#endif /* MLX5_AM_H */
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 6977234..716528d 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -2912,12 +2912,6 @@ enum {
MLX5_CQC_ST_FIRED = 0xa,
};
-enum {
- MLX5_CQ_PERIOD_MODE_START_FROM_EQE = 0x0,
- MLX5_CQ_PERIOD_MODE_START_FROM_CQE = 0x1,
- MLX5_CQ_PERIOD_NUM_MODES
-};
-
struct mlx5_ifc_cqc_bits {
u8 status[0x4];
u8 reserved_at_4[0x4];
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [RFC 2/9] mlx5_en: move interrupt moderation forward declarations
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 1/9] mlx5_en: move interrupt moderation structs to new file Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 2/9] mlx5_en: move interrupt moderation forward delcarations Andy Gospodarek
` (10 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
Move these to newly created file to prepare to move these functions to a
library.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 ----
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h | 4 ++++
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 1bde086..1c56d16 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -798,10 +798,6 @@ void mlx5e_dealloc_rx_wqe(struct mlx5e_rq *rq, u16 ix);
void mlx5e_dealloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix);
void mlx5e_free_rx_mpwqe(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi);
-void mlx5e_rx_am(struct mlx5e_rq *rq);
-void mlx5e_rx_am_work(struct work_struct *work);
-struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode);
-
void mlx5e_update_stats(struct mlx5e_priv *priv, bool full);
int mlx5e_create_flow_steering(struct mlx5e_priv *priv);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
index 176a732..869e4e7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
@@ -71,4 +71,8 @@ enum {
MLX5_CQ_PERIOD_NUM_MODES
};
+void mlx5e_rx_am(struct mlx5e_rq *rq);
+void mlx5e_rx_am_work(struct work_struct *work);
+struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode);
+
#endif /* MLX5_AM_H */
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [RFC 2/9] mlx5_en: move interrupt moderation forward delcarations
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (2 preceding siblings ...)
2017-11-06 5:44 ` [RFC 2/9] mlx5_en: move interrupt moderation forward declarations Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 3/9] mlx5_en: remove rq references in mlx5e_rx_am Andy Gospodarek
` (9 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
Move these to newly created file to prepare to move these functions to a
library.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 4 ----
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h | 4 ++++
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 1bde086..1c56d16 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -798,10 +798,6 @@ void mlx5e_dealloc_rx_wqe(struct mlx5e_rq *rq, u16 ix);
void mlx5e_dealloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix);
void mlx5e_free_rx_mpwqe(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi);
-void mlx5e_rx_am(struct mlx5e_rq *rq);
-void mlx5e_rx_am_work(struct work_struct *work);
-struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode);
-
void mlx5e_update_stats(struct mlx5e_priv *priv, bool full);
int mlx5e_create_flow_steering(struct mlx5e_priv *priv);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
index 176a732..869e4e7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
@@ -71,4 +71,8 @@ enum {
MLX5_CQ_PERIOD_NUM_MODES
};
+void mlx5e_rx_am(struct mlx5e_rq *rq);
+void mlx5e_rx_am_work(struct work_struct *work);
+struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode);
+
#endif /* MLX5_AM_H */
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [RFC 3/9] mlx5_en: remove rq references in mlx5e_rx_am
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (3 preceding siblings ...)
2017-11-06 5:44 ` [RFC 2/9] mlx5_en: move interrupt moderation forward delcarations Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 4/9] mlx5_en: move AM logic enums Andy Gospodarek
` (8 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
This makes mlx5e_am_sample more generic so that it can be called easily
from a driver that does not use the same data structure to store these
values in a single structure.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c | 22 +++++++++++++---------
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h | 5 ++++-
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 5 ++++-
3 files changed, 21 insertions(+), 11 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
index acf32fe..845dbb8 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
@@ -260,13 +260,15 @@ static bool mlx5e_am_decision(struct mlx5e_rx_am_stats *curr_stats,
return am->profile_ix != prev_ix;
}
-static void mlx5e_am_sample(struct mlx5e_rq *rq,
+static void mlx5e_am_sample(u16 event_ctr,
+ u64 packets,
+ u64 bytes,
struct mlx5e_rx_am_sample *s)
{
s->time = ktime_get();
- s->pkt_ctr = rq->stats.packets;
- s->byte_ctr = rq->stats.bytes;
- s->event_ctr = rq->cq.event_ctr;
+ s->pkt_ctr = packets;
+ s->byte_ctr = bytes;
+ s->event_ctr = event_ctr;
}
#define MLX5E_AM_NEVENTS 64
@@ -305,20 +307,22 @@ void mlx5e_rx_am_work(struct work_struct *work)
am->state = MLX5E_AM_START_MEASURE;
}
-void mlx5e_rx_am(struct mlx5e_rq *rq)
+void mlx5e_rx_am(struct mlx5e_rx_am *am,
+ u16 event_ctr,
+ u64 packets,
+ u64 bytes)
{
- struct mlx5e_rx_am *am = &rq->am;
struct mlx5e_rx_am_sample end_sample;
struct mlx5e_rx_am_stats curr_stats;
u16 nevents;
switch (am->state) {
case MLX5E_AM_MEASURE_IN_PROGRESS:
- nevents = BIT_GAP(BITS_PER_TYPE(u16), rq->cq.event_ctr,
+ nevents = BIT_GAP(BITS_PER_TYPE(u16), event_ctr,
am->start_sample.event_ctr);
if (nevents < MLX5E_AM_NEVENTS)
break;
- mlx5e_am_sample(rq, &end_sample);
+ mlx5e_am_sample(event_ctr, packets, bytes, &end_sample);
mlx5e_am_calc_stats(&am->start_sample, &end_sample,
&curr_stats);
if (mlx5e_am_decision(&curr_stats, am)) {
@@ -328,7 +332,7 @@ void mlx5e_rx_am(struct mlx5e_rq *rq)
}
/* fall through */
case MLX5E_AM_START_MEASURE:
- mlx5e_am_sample(rq, &am->start_sample);
+ mlx5e_am_sample(event_ctr, packets, bytes, &am->start_sample);
am->state = MLX5E_AM_MEASURE_IN_PROGRESS;
break;
case MLX5E_AM_APPLY_NEW_PROFILE:
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
index 869e4e7..90e4913 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
@@ -71,7 +71,10 @@ enum {
MLX5_CQ_PERIOD_NUM_MODES
};
-void mlx5e_rx_am(struct mlx5e_rq *rq);
+void mlx5e_rx_am(struct mlx5e_rx_am *am,
+ u16 event_ctr,
+ u64 packets,
+ u64 bytes);
void mlx5e_rx_am_work(struct work_struct *work);
struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index e906b75..8fed6c6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -77,7 +77,10 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
mlx5e_cq_arm(&c->sq[i].cq);
if (MLX5E_TEST_BIT(c->rq.state, MLX5E_RQ_STATE_AM))
- mlx5e_rx_am(&c->rq);
+ mlx5e_rx_am(&c->rq.am,
+ c->rq.cq.event_ctr,
+ c->rq.stats.packets,
+ c->rq.stats.bytes);
mlx5e_cq_arm(&c->rq.cq);
mlx5e_cq_arm(&c->icosq.cq);
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [RFC 4/9] mlx5_en: move AM logic enums
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (4 preceding siblings ...)
2017-11-06 5:44 ` [RFC 3/9] mlx5_en: remove rq references in mlx5e_rx_am Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 5/9] mlx5_en: move generic functions to new file Andy Gospodarek
` (7 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
More movement to help make this code more generic.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c | 25 ---------------------
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h | 26 ++++++++++++++++++++++
2 files changed, 26 insertions(+), 25 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
index 845dbb8..02d4f80 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
@@ -78,31 +78,6 @@ struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode)
return profile[rx_cq_period_mode][default_profile_ix];
}
-/* Adaptive moderation logic */
-enum {
- MLX5E_AM_START_MEASURE,
- MLX5E_AM_MEASURE_IN_PROGRESS,
- MLX5E_AM_APPLY_NEW_PROFILE,
-};
-
-enum {
- MLX5E_AM_PARKING_ON_TOP,
- MLX5E_AM_PARKING_TIRED,
- MLX5E_AM_GOING_RIGHT,
- MLX5E_AM_GOING_LEFT,
-};
-
-enum {
- MLX5E_AM_STATS_WORSE,
- MLX5E_AM_STATS_SAME,
- MLX5E_AM_STATS_BETTER,
-};
-
-enum {
- MLX5E_AM_STEPPED,
- MLX5E_AM_TOO_TIRED,
- MLX5E_AM_ON_EDGE,
-};
static bool mlx5e_am_on_top(struct mlx5e_rx_am *am)
{
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
index 90e4913..efbee99 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
@@ -71,6 +71,32 @@ enum {
MLX5_CQ_PERIOD_NUM_MODES
};
+/* Adaptive moderation logic */
+enum {
+ MLX5E_AM_START_MEASURE,
+ MLX5E_AM_MEASURE_IN_PROGRESS,
+ MLX5E_AM_APPLY_NEW_PROFILE,
+};
+
+enum {
+ MLX5E_AM_PARKING_ON_TOP,
+ MLX5E_AM_PARKING_TIRED,
+ MLX5E_AM_GOING_RIGHT,
+ MLX5E_AM_GOING_LEFT,
+};
+
+enum {
+ MLX5E_AM_STATS_WORSE,
+ MLX5E_AM_STATS_SAME,
+ MLX5E_AM_STATS_BETTER,
+};
+
+enum {
+ MLX5E_AM_STEPPED,
+ MLX5E_AM_TOO_TIRED,
+ MLX5E_AM_ON_EDGE,
+};
+
void mlx5e_rx_am(struct mlx5e_rx_am *am,
u16 event_ctr,
u64 packets,
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [RFC 5/9] mlx5_en: move generic functions to new file
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (5 preceding siblings ...)
2017-11-06 5:44 ` [RFC 4/9] mlx5_en: move AM logic enums Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 6/9] mlx5_en: rename en_rx_am.h to net_rx_am.h Andy Gospodarek
` (6 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
These functions were identified as ones that could be made generic and
used by multiple drivers. Most of the contents of en_rx_am.c are moved
to net_rx_am.c.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/mellanox/mlx5/core/Makefile | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c | 272 +-----------------
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h | 1 +
.../net/ethernet/mellanox/mlx5/core/net_rx_am.c | 303 +++++++++++++++++++++
4 files changed, 307 insertions(+), 271 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 19b21b4..d5d6d3d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -15,7 +15,7 @@ mlx5_core-$(CONFIG_MLX5_FPGA) += fpga/cmd.o fpga/core.o fpga/conn.o fpga/sdk.o \
mlx5_core-$(CONFIG_MLX5_CORE_EN) += en_main.o en_common.o en_fs.o en_ethtool.o \
en_tx.o en_rx.o en_rx_am.o en_txrx.o en_stats.o vxlan.o \
- en_arfs.o en_fs_ethtool.o en_selftest.o
+ en_arfs.o en_fs_ethtool.o en_selftest.o net_rx_am.o
mlx5_core-$(CONFIG_MLX5_MPFS) += lib/mpfs.o
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
index 02d4f80..b9b434b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
@@ -32,249 +32,13 @@
#include "en.h"
-/* Adaptive moderation profiles */
-#define MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE 256
-#define MLX5E_RX_AM_DEF_PROFILE_CQE 1
-#define MLX5E_RX_AM_DEF_PROFILE_EQE 1
-#define MLX5E_PARAMS_AM_NUM_PROFILES 5
-
-/* All profiles sizes must be MLX5E_PARAMS_AM_NUM_PROFILES */
-#define MLX5_AM_EQE_PROFILES { \
- {1, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {8, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {64, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {128, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {256, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
-}
-
-#define MLX5_AM_CQE_PROFILES { \
- {2, 256}, \
- {8, 128}, \
- {16, 64}, \
- {32, 64}, \
- {64, 64} \
-}
-
-static const struct mlx5e_cq_moder
-profile[MLX5_CQ_PERIOD_NUM_MODES][MLX5E_PARAMS_AM_NUM_PROFILES] = {
- MLX5_AM_EQE_PROFILES,
- MLX5_AM_CQE_PROFILES,
-};
-
-static inline struct mlx5e_cq_moder mlx5e_am_get_profile(u8 cq_period_mode, int ix)
-{
- return profile[cq_period_mode][ix];
-}
-
-struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode)
-{
- int default_profile_ix;
-
- if (rx_cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE)
- default_profile_ix = MLX5E_RX_AM_DEF_PROFILE_CQE;
- else /* MLX5_CQ_PERIOD_MODE_START_FROM_EQE */
- default_profile_ix = MLX5E_RX_AM_DEF_PROFILE_EQE;
-
- return profile[rx_cq_period_mode][default_profile_ix];
-}
-
-
-static bool mlx5e_am_on_top(struct mlx5e_rx_am *am)
-{
- switch (am->tune_state) {
- case MLX5E_AM_PARKING_ON_TOP:
- case MLX5E_AM_PARKING_TIRED:
- return true;
- case MLX5E_AM_GOING_RIGHT:
- return (am->steps_left > 1) && (am->steps_right == 1);
- default: /* MLX5E_AM_GOING_LEFT */
- return (am->steps_right > 1) && (am->steps_left == 1);
- }
-}
-
-static void mlx5e_am_turn(struct mlx5e_rx_am *am)
-{
- switch (am->tune_state) {
- case MLX5E_AM_PARKING_ON_TOP:
- case MLX5E_AM_PARKING_TIRED:
- break;
- case MLX5E_AM_GOING_RIGHT:
- am->tune_state = MLX5E_AM_GOING_LEFT;
- am->steps_left = 0;
- break;
- case MLX5E_AM_GOING_LEFT:
- am->tune_state = MLX5E_AM_GOING_RIGHT;
- am->steps_right = 0;
- break;
- }
-}
-
-static int mlx5e_am_step(struct mlx5e_rx_am *am)
-{
- if (am->tired == (MLX5E_PARAMS_AM_NUM_PROFILES * 2))
- return MLX5E_AM_TOO_TIRED;
-
- switch (am->tune_state) {
- case MLX5E_AM_PARKING_ON_TOP:
- case MLX5E_AM_PARKING_TIRED:
- break;
- case MLX5E_AM_GOING_RIGHT:
- if (am->profile_ix == (MLX5E_PARAMS_AM_NUM_PROFILES - 1))
- return MLX5E_AM_ON_EDGE;
- am->profile_ix++;
- am->steps_right++;
- break;
- case MLX5E_AM_GOING_LEFT:
- if (am->profile_ix == 0)
- return MLX5E_AM_ON_EDGE;
- am->profile_ix--;
- am->steps_left++;
- break;
- }
-
- am->tired++;
- return MLX5E_AM_STEPPED;
-}
-
-static void mlx5e_am_park_on_top(struct mlx5e_rx_am *am)
-{
- am->steps_right = 0;
- am->steps_left = 0;
- am->tired = 0;
- am->tune_state = MLX5E_AM_PARKING_ON_TOP;
-}
-
-static void mlx5e_am_park_tired(struct mlx5e_rx_am *am)
-{
- am->steps_right = 0;
- am->steps_left = 0;
- am->tune_state = MLX5E_AM_PARKING_TIRED;
-}
-
-static void mlx5e_am_exit_parking(struct mlx5e_rx_am *am)
-{
- am->tune_state = am->profile_ix ? MLX5E_AM_GOING_LEFT :
- MLX5E_AM_GOING_RIGHT;
- mlx5e_am_step(am);
-}
-
-#define IS_SIGNIFICANT_DIFF(val, ref) \
- (((100 * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */
-
-static int mlx5e_am_stats_compare(struct mlx5e_rx_am_stats *curr,
- struct mlx5e_rx_am_stats *prev)
-{
- if (!prev->bpms)
- return curr->bpms ? MLX5E_AM_STATS_BETTER :
- MLX5E_AM_STATS_SAME;
-
- if (IS_SIGNIFICANT_DIFF(curr->bpms, prev->bpms))
- return (curr->bpms > prev->bpms) ? MLX5E_AM_STATS_BETTER :
- MLX5E_AM_STATS_WORSE;
-
- if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms))
- return (curr->ppms > prev->ppms) ? MLX5E_AM_STATS_BETTER :
- MLX5E_AM_STATS_WORSE;
-
- if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms))
- return (curr->epms < prev->epms) ? MLX5E_AM_STATS_BETTER :
- MLX5E_AM_STATS_WORSE;
-
- return MLX5E_AM_STATS_SAME;
-}
-
-static bool mlx5e_am_decision(struct mlx5e_rx_am_stats *curr_stats,
- struct mlx5e_rx_am *am)
-{
- int prev_state = am->tune_state;
- int prev_ix = am->profile_ix;
- int stats_res;
- int step_res;
-
- switch (am->tune_state) {
- case MLX5E_AM_PARKING_ON_TOP:
- stats_res = mlx5e_am_stats_compare(curr_stats, &am->prev_stats);
- if (stats_res != MLX5E_AM_STATS_SAME)
- mlx5e_am_exit_parking(am);
- break;
-
- case MLX5E_AM_PARKING_TIRED:
- am->tired--;
- if (!am->tired)
- mlx5e_am_exit_parking(am);
- break;
-
- case MLX5E_AM_GOING_RIGHT:
- case MLX5E_AM_GOING_LEFT:
- stats_res = mlx5e_am_stats_compare(curr_stats, &am->prev_stats);
- if (stats_res != MLX5E_AM_STATS_BETTER)
- mlx5e_am_turn(am);
-
- if (mlx5e_am_on_top(am)) {
- mlx5e_am_park_on_top(am);
- break;
- }
-
- step_res = mlx5e_am_step(am);
- switch (step_res) {
- case MLX5E_AM_ON_EDGE:
- mlx5e_am_park_on_top(am);
- break;
- case MLX5E_AM_TOO_TIRED:
- mlx5e_am_park_tired(am);
- break;
- }
-
- break;
- }
-
- if ((prev_state != MLX5E_AM_PARKING_ON_TOP) ||
- (am->tune_state != MLX5E_AM_PARKING_ON_TOP))
- am->prev_stats = *curr_stats;
-
- return am->profile_ix != prev_ix;
-}
-
-static void mlx5e_am_sample(u16 event_ctr,
- u64 packets,
- u64 bytes,
- struct mlx5e_rx_am_sample *s)
-{
- s->time = ktime_get();
- s->pkt_ctr = packets;
- s->byte_ctr = bytes;
- s->event_ctr = event_ctr;
-}
-
-#define MLX5E_AM_NEVENTS 64
-#define BITS_PER_TYPE(type) (sizeof(type) * BITS_PER_BYTE)
-#define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1))
-
-static void mlx5e_am_calc_stats(struct mlx5e_rx_am_sample *start,
- struct mlx5e_rx_am_sample *end,
- struct mlx5e_rx_am_stats *curr_stats)
-{
- /* u32 holds up to 71 minutes, should be enough */
- u32 delta_us = ktime_us_delta(end->time, start->time);
- u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr);
- u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr,
- start->byte_ctr);
-
- if (!delta_us)
- return;
-
- curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us);
- curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us);
- curr_stats->epms = DIV_ROUND_UP(MLX5E_AM_NEVENTS * USEC_PER_MSEC,
- delta_us);
-}
-
void mlx5e_rx_am_work(struct work_struct *work)
{
struct mlx5e_rx_am *am = container_of(work, struct mlx5e_rx_am,
work);
struct mlx5e_rq *rq = container_of(am, struct mlx5e_rq, am);
- struct mlx5e_cq_moder cur_profile = profile[am->mode][am->profile_ix];
+ struct mlx5e_cq_moder cur_profile = mlx5e_am_get_profile(am->mode,
+ am->profile_ix);
mlx5_core_modify_cq_moderation(rq->mdev, &rq->cq.mcq,
cur_profile.usec, cur_profile.pkts);
@@ -282,35 +46,3 @@ void mlx5e_rx_am_work(struct work_struct *work)
am->state = MLX5E_AM_START_MEASURE;
}
-void mlx5e_rx_am(struct mlx5e_rx_am *am,
- u16 event_ctr,
- u64 packets,
- u64 bytes)
-{
- struct mlx5e_rx_am_sample end_sample;
- struct mlx5e_rx_am_stats curr_stats;
- u16 nevents;
-
- switch (am->state) {
- case MLX5E_AM_MEASURE_IN_PROGRESS:
- nevents = BIT_GAP(BITS_PER_TYPE(u16), event_ctr,
- am->start_sample.event_ctr);
- if (nevents < MLX5E_AM_NEVENTS)
- break;
- mlx5e_am_sample(event_ctr, packets, bytes, &end_sample);
- mlx5e_am_calc_stats(&am->start_sample, &end_sample,
- &curr_stats);
- if (mlx5e_am_decision(&curr_stats, am)) {
- am->state = MLX5E_AM_APPLY_NEW_PROFILE;
- schedule_work(&am->work);
- break;
- }
- /* fall through */
- case MLX5E_AM_START_MEASURE:
- mlx5e_am_sample(event_ctr, packets, bytes, &am->start_sample);
- am->state = MLX5E_AM_MEASURE_IN_PROGRESS;
- break;
- case MLX5E_AM_APPLY_NEW_PROFILE:
- break;
- }
-}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
index efbee99..ef86bf8 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
@@ -103,5 +103,6 @@ void mlx5e_rx_am(struct mlx5e_rx_am *am,
u64 bytes);
void mlx5e_rx_am_work(struct work_struct *work);
struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode);
+struct mlx5e_cq_moder mlx5e_am_get_profile(u8 cq_period_mode, int ix);
#endif /* MLX5_AM_H */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
new file mode 100644
index 0000000..14cea75
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
@@ -0,0 +1,303 @@
+/*
+ * Copyright (c) 2016, Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2017, Broadcom Limiited. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include "en.h"
+
+#define MLX5E_PARAMS_AM_NUM_PROFILES 5
+/* Adaptive moderation profiles */
+#define MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE 256
+#define MLX5E_RX_AM_DEF_PROFILE_CQE 1
+#define MLX5E_RX_AM_DEF_PROFILE_EQE 1
+
+/* All profiles sizes must be MLX5E_PARAMS_AM_NUM_PROFILES */
+#define MLX5_AM_EQE_PROFILES { \
+ {1, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {8, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {64, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {128, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {256, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+}
+
+#define MLX5_AM_CQE_PROFILES { \
+ {2, 256}, \
+ {8, 128}, \
+ {16, 64}, \
+ {32, 64}, \
+ {64, 64} \
+}
+
+static const struct mlx5e_cq_moder
+profile[MLX5_CQ_PERIOD_NUM_MODES][MLX5E_PARAMS_AM_NUM_PROFILES] = {
+ MLX5_AM_EQE_PROFILES,
+ MLX5_AM_CQE_PROFILES,
+};
+
+struct mlx5e_cq_moder mlx5e_am_get_profile(u8 cq_period_mode, int ix)
+{
+ return profile[cq_period_mode][ix];
+}
+
+struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode)
+{
+ int default_profile_ix;
+
+ if (rx_cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE)
+ default_profile_ix = MLX5E_RX_AM_DEF_PROFILE_CQE;
+ else /* MLX5_CQ_PERIOD_MODE_START_FROM_EQE */
+ default_profile_ix = MLX5E_RX_AM_DEF_PROFILE_EQE;
+
+ return profile[rx_cq_period_mode][default_profile_ix];
+}
+
+static bool mlx5e_am_on_top(struct mlx5e_rx_am *am)
+{
+ switch (am->tune_state) {
+ case MLX5E_AM_PARKING_ON_TOP:
+ case MLX5E_AM_PARKING_TIRED:
+ return true;
+ case MLX5E_AM_GOING_RIGHT:
+ return (am->steps_left > 1) && (am->steps_right == 1);
+ default: /* MLX5E_AM_GOING_LEFT */
+ return (am->steps_right > 1) && (am->steps_left == 1);
+ }
+}
+
+static void mlx5e_am_turn(struct mlx5e_rx_am *am)
+{
+ switch (am->tune_state) {
+ case MLX5E_AM_PARKING_ON_TOP:
+ case MLX5E_AM_PARKING_TIRED:
+ break;
+ case MLX5E_AM_GOING_RIGHT:
+ am->tune_state = MLX5E_AM_GOING_LEFT;
+ am->steps_left = 0;
+ break;
+ case MLX5E_AM_GOING_LEFT:
+ am->tune_state = MLX5E_AM_GOING_RIGHT;
+ am->steps_right = 0;
+ break;
+ }
+}
+
+static int mlx5e_am_step(struct mlx5e_rx_am *am)
+{
+ if (am->tired == (MLX5E_PARAMS_AM_NUM_PROFILES * 2))
+ return MLX5E_AM_TOO_TIRED;
+
+ switch (am->tune_state) {
+ case MLX5E_AM_PARKING_ON_TOP:
+ case MLX5E_AM_PARKING_TIRED:
+ break;
+ case MLX5E_AM_GOING_RIGHT:
+ if (am->profile_ix == (MLX5E_PARAMS_AM_NUM_PROFILES - 1))
+ return MLX5E_AM_ON_EDGE;
+ am->profile_ix++;
+ am->steps_right++;
+ break;
+ case MLX5E_AM_GOING_LEFT:
+ if (am->profile_ix == 0)
+ return MLX5E_AM_ON_EDGE;
+ am->profile_ix--;
+ am->steps_left++;
+ break;
+ }
+
+ am->tired++;
+ return MLX5E_AM_STEPPED;
+}
+
+static void mlx5e_am_park_on_top(struct mlx5e_rx_am *am)
+{
+ am->steps_right = 0;
+ am->steps_left = 0;
+ am->tired = 0;
+ am->tune_state = MLX5E_AM_PARKING_ON_TOP;
+}
+
+static void mlx5e_am_park_tired(struct mlx5e_rx_am *am)
+{
+ am->steps_right = 0;
+ am->steps_left = 0;
+ am->tune_state = MLX5E_AM_PARKING_TIRED;
+}
+
+static void mlx5e_am_exit_parking(struct mlx5e_rx_am *am)
+{
+ am->tune_state = am->profile_ix ? MLX5E_AM_GOING_LEFT :
+ MLX5E_AM_GOING_RIGHT;
+ mlx5e_am_step(am);
+}
+
+#define IS_SIGNIFICANT_DIFF(val, ref) \
+ (((100 * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */
+
+static int mlx5e_am_stats_compare(struct mlx5e_rx_am_stats *curr,
+ struct mlx5e_rx_am_stats *prev)
+{
+ if (!prev->bpms)
+ return curr->bpms ? MLX5E_AM_STATS_BETTER :
+ MLX5E_AM_STATS_SAME;
+
+ if (IS_SIGNIFICANT_DIFF(curr->bpms, prev->bpms))
+ return (curr->bpms > prev->bpms) ? MLX5E_AM_STATS_BETTER :
+ MLX5E_AM_STATS_WORSE;
+
+ if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms))
+ return (curr->ppms > prev->ppms) ? MLX5E_AM_STATS_BETTER :
+ MLX5E_AM_STATS_WORSE;
+
+ if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms))
+ return (curr->epms < prev->epms) ? MLX5E_AM_STATS_BETTER :
+ MLX5E_AM_STATS_WORSE;
+
+ return MLX5E_AM_STATS_SAME;
+}
+
+static bool mlx5e_am_decision(struct mlx5e_rx_am_stats *curr_stats,
+ struct mlx5e_rx_am *am)
+{
+ int prev_state = am->tune_state;
+ int prev_ix = am->profile_ix;
+ int stats_res;
+ int step_res;
+
+ switch (am->tune_state) {
+ case MLX5E_AM_PARKING_ON_TOP:
+ stats_res = mlx5e_am_stats_compare(curr_stats, &am->prev_stats);
+ if (stats_res != MLX5E_AM_STATS_SAME)
+ mlx5e_am_exit_parking(am);
+ break;
+
+ case MLX5E_AM_PARKING_TIRED:
+ am->tired--;
+ if (!am->tired)
+ mlx5e_am_exit_parking(am);
+ break;
+
+ case MLX5E_AM_GOING_RIGHT:
+ case MLX5E_AM_GOING_LEFT:
+ stats_res = mlx5e_am_stats_compare(curr_stats, &am->prev_stats);
+ if (stats_res != MLX5E_AM_STATS_BETTER)
+ mlx5e_am_turn(am);
+
+ if (mlx5e_am_on_top(am)) {
+ mlx5e_am_park_on_top(am);
+ break;
+ }
+
+ step_res = mlx5e_am_step(am);
+ switch (step_res) {
+ case MLX5E_AM_ON_EDGE:
+ mlx5e_am_park_on_top(am);
+ break;
+ case MLX5E_AM_TOO_TIRED:
+ mlx5e_am_park_tired(am);
+ break;
+ }
+
+ break;
+ }
+
+ if ((prev_state != MLX5E_AM_PARKING_ON_TOP) ||
+ (am->tune_state != MLX5E_AM_PARKING_ON_TOP))
+ am->prev_stats = *curr_stats;
+
+ return am->profile_ix != prev_ix;
+}
+
+static void mlx5e_am_sample(u16 event_ctr,
+ u64 packets,
+ u64 bytes,
+ struct mlx5e_rx_am_sample *s)
+{
+ s->time = ktime_get();
+ s->pkt_ctr = packets;
+ s->byte_ctr = bytes;
+ s->event_ctr = event_ctr;
+}
+
+#define MLX5E_AM_NEVENTS 64
+#define BITS_PER_TYPE(type) (sizeof(type) * BITS_PER_BYTE)
+#define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1))
+
+static void mlx5e_am_calc_stats(struct mlx5e_rx_am_sample *start,
+ struct mlx5e_rx_am_sample *end,
+ struct mlx5e_rx_am_stats *curr_stats)
+{
+ /* u32 holds up to 71 minutes, should be enough */
+ u32 delta_us = ktime_us_delta(end->time, start->time);
+ u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr);
+ u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr,
+ start->byte_ctr);
+
+ if (!delta_us)
+ return;
+
+ curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us);
+ curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us);
+ curr_stats->epms = DIV_ROUND_UP(MLX5E_AM_NEVENTS * USEC_PER_MSEC,
+ delta_us);
+}
+
+void mlx5e_rx_am(struct mlx5e_rx_am *am,
+ u16 event_ctr,
+ u64 packets,
+ u64 bytes)
+{
+ struct mlx5e_rx_am_sample end_sample;
+ struct mlx5e_rx_am_stats curr_stats;
+ u16 nevents;
+
+ switch (am->state) {
+ case MLX5E_AM_MEASURE_IN_PROGRESS:
+ nevents = BIT_GAP(BITS_PER_TYPE(u16), event_ctr,
+ am->start_sample.event_ctr);
+ if (nevents < MLX5E_AM_NEVENTS)
+ break;
+ mlx5e_am_sample(event_ctr, packets, bytes, &end_sample);
+ mlx5e_am_calc_stats(&am->start_sample, &end_sample,
+ &curr_stats);
+ if (mlx5e_am_decision(&curr_stats, am)) {
+ am->state = MLX5E_AM_APPLY_NEW_PROFILE;
+ schedule_work(&am->work);
+ break;
+ }
+ /* fall through */
+ case MLX5E_AM_START_MEASURE:
+ mlx5e_am_sample(event_ctr, packets, bytes, &am->start_sample);
+ am->state = MLX5E_AM_MEASURE_IN_PROGRESS;
+ break;
+ case MLX5E_AM_APPLY_NEW_PROFILE:
+ break;
+ }
+}
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [RFC 6/9] mlx5_en: rename en_rx_am.h to net_rx_am.h
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (6 preceding siblings ...)
2017-11-06 5:44 ` [RFC 5/9] mlx5_en: move generic functions to new file Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 7/9] mlx5_en: remove Mellanox references in AM code Andy Gospodarek
` (5 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
This is so net_rx_am.h can be easily moved out of mlx5/core.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h | 108 ---------------------
.../net/ethernet/mellanox/mlx5/core/net_rx_am.h | 108 +++++++++++++++++++++
3 files changed, 109 insertions(+), 109 deletions(-)
delete mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 1c56d16..a9dc118 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -49,7 +49,7 @@
#include "wq.h"
#include "mlx5_core.h"
#include "en_stats.h"
-#include "en_rx_am.h"
+#include "net_rx_am.h"
#define MLX5_SET_CFG(p, f, v) MLX5_SET(create_flow_group_in, p, f, v)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
deleted file mode 100644
index ef86bf8..0000000
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.h
+++ /dev/null
@@ -1,108 +0,0 @@
-/*
- * Copyright (c) 2013-2015, Mellanox Technologies, Ltd. All rights reserved.
- * Copyright (c) 2017, Broadcom Limited
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
-*/
-
-#ifndef MLX5_AM_H
-#define MLX5_AM_H
-
-struct mlx5e_cq_moder {
- u16 usec;
- u16 pkts;
-};
-
-struct mlx5e_rx_am_sample {
- ktime_t time;
- u32 pkt_ctr;
- u32 byte_ctr;
- u16 event_ctr;
-};
-
-struct mlx5e_rx_am_stats {
- int ppms; /* packets per msec */
- int bpms; /* bytes per msec */
- int epms; /* events per msec */
-};
-
-struct mlx5e_rx_am { /* Adaptive Moderation */
- u8 state;
- struct mlx5e_rx_am_stats prev_stats;
- struct mlx5e_rx_am_sample start_sample;
- struct work_struct work;
- u8 profile_ix;
- u8 mode;
- u8 tune_state;
- u8 steps_right;
- u8 steps_left;
- u8 tired;
-};
-
-enum {
- MLX5_CQ_PERIOD_MODE_START_FROM_EQE = 0x0,
- MLX5_CQ_PERIOD_MODE_START_FROM_CQE = 0x1,
- MLX5_CQ_PERIOD_NUM_MODES
-};
-
-/* Adaptive moderation logic */
-enum {
- MLX5E_AM_START_MEASURE,
- MLX5E_AM_MEASURE_IN_PROGRESS,
- MLX5E_AM_APPLY_NEW_PROFILE,
-};
-
-enum {
- MLX5E_AM_PARKING_ON_TOP,
- MLX5E_AM_PARKING_TIRED,
- MLX5E_AM_GOING_RIGHT,
- MLX5E_AM_GOING_LEFT,
-};
-
-enum {
- MLX5E_AM_STATS_WORSE,
- MLX5E_AM_STATS_SAME,
- MLX5E_AM_STATS_BETTER,
-};
-
-enum {
- MLX5E_AM_STEPPED,
- MLX5E_AM_TOO_TIRED,
- MLX5E_AM_ON_EDGE,
-};
-
-void mlx5e_rx_am(struct mlx5e_rx_am *am,
- u16 event_ctr,
- u64 packets,
- u64 bytes);
-void mlx5e_rx_am_work(struct work_struct *work);
-struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode);
-struct mlx5e_cq_moder mlx5e_am_get_profile(u8 cq_period_mode, int ix);
-
-#endif /* MLX5_AM_H */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
new file mode 100644
index 0000000..ef86bf8
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
@@ -0,0 +1,108 @@
+/*
+ * Copyright (c) 2013-2015, Mellanox Technologies, Ltd. All rights reserved.
+ * Copyright (c) 2017, Broadcom Limited
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+*/
+
+#ifndef MLX5_AM_H
+#define MLX5_AM_H
+
+struct mlx5e_cq_moder {
+ u16 usec;
+ u16 pkts;
+};
+
+struct mlx5e_rx_am_sample {
+ ktime_t time;
+ u32 pkt_ctr;
+ u32 byte_ctr;
+ u16 event_ctr;
+};
+
+struct mlx5e_rx_am_stats {
+ int ppms; /* packets per msec */
+ int bpms; /* bytes per msec */
+ int epms; /* events per msec */
+};
+
+struct mlx5e_rx_am { /* Adaptive Moderation */
+ u8 state;
+ struct mlx5e_rx_am_stats prev_stats;
+ struct mlx5e_rx_am_sample start_sample;
+ struct work_struct work;
+ u8 profile_ix;
+ u8 mode;
+ u8 tune_state;
+ u8 steps_right;
+ u8 steps_left;
+ u8 tired;
+};
+
+enum {
+ MLX5_CQ_PERIOD_MODE_START_FROM_EQE = 0x0,
+ MLX5_CQ_PERIOD_MODE_START_FROM_CQE = 0x1,
+ MLX5_CQ_PERIOD_NUM_MODES
+};
+
+/* Adaptive moderation logic */
+enum {
+ MLX5E_AM_START_MEASURE,
+ MLX5E_AM_MEASURE_IN_PROGRESS,
+ MLX5E_AM_APPLY_NEW_PROFILE,
+};
+
+enum {
+ MLX5E_AM_PARKING_ON_TOP,
+ MLX5E_AM_PARKING_TIRED,
+ MLX5E_AM_GOING_RIGHT,
+ MLX5E_AM_GOING_LEFT,
+};
+
+enum {
+ MLX5E_AM_STATS_WORSE,
+ MLX5E_AM_STATS_SAME,
+ MLX5E_AM_STATS_BETTER,
+};
+
+enum {
+ MLX5E_AM_STEPPED,
+ MLX5E_AM_TOO_TIRED,
+ MLX5E_AM_ON_EDGE,
+};
+
+void mlx5e_rx_am(struct mlx5e_rx_am *am,
+ u16 event_ctr,
+ u64 packets,
+ u64 bytes);
+void mlx5e_rx_am_work(struct work_struct *work);
+struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode);
+struct mlx5e_cq_moder mlx5e_am_get_profile(u8 cq_period_mode, int ix);
+
+#endif /* MLX5_AM_H */
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [RFC 7/9] mlx5_en: remove Mellanox references in AM code
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (7 preceding siblings ...)
2017-11-06 5:44 ` [RFC 6/9] mlx5_en: rename en_rx_am.h to net_rx_am.h Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 8/9] net: move adaptive interrupt coalescing code to lib/ Andy Gospodarek
` (4 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
Remove all mlx5* and MLX* references to net_ and NET_, respectively in
code that handles software interrupt moderation.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 7 +-
.../net/ethernet/mellanox/mlx5/core/en_ethtool.c | 6 +-
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 18 +-
drivers/net/ethernet/mellanox/mlx5/core/en_rep.c | 4 +-
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c | 6 +-
drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 2 +-
.../net/ethernet/mellanox/mlx5/core/net_rx_am.c | 214 ++++++++++-----------
.../net/ethernet/mellanox/mlx5/core/net_rx_am.h | 57 +++---
8 files changed, 157 insertions(+), 157 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index a9dc118..203dc7b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -221,8 +221,8 @@ struct mlx5e_params {
u8 num_tc;
u8 rx_cq_period_mode;
bool rx_cqe_compress_def;
- struct mlx5e_cq_moder rx_cq_moderation;
- struct mlx5e_cq_moder tx_cq_moderation;
+ struct net_cq_moder rx_cq_moderation;
+ struct net_cq_moder tx_cq_moderation;
bool lro_en;
u32 lro_wqe_sz;
u16 tx_max_inline;
@@ -505,7 +505,7 @@ struct mlx5e_rq {
unsigned long state;
int ix;
- struct mlx5e_rx_am am; /* Adaptive Moderation */
+ struct net_rx_am am; /* Adaptive Moderation */
/* XDP */
struct bpf_prog *xdp_prog;
@@ -1036,4 +1036,5 @@ void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
struct mlx5e_params *params,
u16 max_channels);
+void mlx5e_rx_am_work(struct work_struct *work);
#endif /* __MLX5_EN_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
index b34aa8e..3955521 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
@@ -1454,11 +1454,11 @@ static int set_pflag_rx_cqe_based_moder(struct net_device *netdev, bool enable)
int err = 0;
rx_cq_period_mode = enable ?
- MLX5_CQ_PERIOD_MODE_START_FROM_CQE :
- MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
+ NET_CQ_PERIOD_MODE_START_FROM_CQE :
+ NET_CQ_PERIOD_MODE_START_FROM_EQE;
rx_mode_changed = rx_cq_period_mode != priv->channels.params.rx_cq_period_mode;
- if (rx_cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE &&
+ if (rx_cq_period_mode == NET_CQ_PERIOD_MODE_START_FROM_CQE &&
!MLX5_CAP_GEN(mdev, cq_period_start_from_cqe))
return -EOPNOTSUPP;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 28ae00b..dcd96fe 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1571,7 +1571,7 @@ static void mlx5e_destroy_cq(struct mlx5e_cq *cq)
}
static int mlx5e_open_cq(struct mlx5e_channel *c,
- struct mlx5e_cq_moder moder,
+ struct net_cq_moder moder,
struct mlx5e_cq_param *param,
struct mlx5e_cq *cq)
{
@@ -1748,7 +1748,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
struct mlx5e_channel_param *cparam,
struct mlx5e_channel **cp)
{
- struct mlx5e_cq_moder icocq_moder = {0, 0};
+ struct net_cq_moder icocq_moder = {0, 0};
struct net_device *netdev = priv->netdev;
struct mlx5e_channel *c;
unsigned int irq;
@@ -1987,7 +1987,7 @@ static void mlx5e_build_tx_cq_param(struct mlx5e_priv *priv,
mlx5e_build_common_cq_param(priv, param);
- param->cq_period_mode = MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
+ param->cq_period_mode = NET_CQ_PERIOD_MODE_START_FROM_EQE;
}
static void mlx5e_build_ico_cq_param(struct mlx5e_priv *priv,
@@ -2000,7 +2000,7 @@ static void mlx5e_build_ico_cq_param(struct mlx5e_priv *priv,
mlx5e_build_common_cq_param(priv, param);
- param->cq_period_mode = MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
+ param->cq_period_mode = NET_CQ_PERIOD_MODE_START_FROM_EQE;
}
static void mlx5e_build_icosq_param(struct mlx5e_priv *priv,
@@ -3996,16 +3996,16 @@ void mlx5e_set_rx_cq_mode_params(struct mlx5e_params *params, u8 cq_period_mode)
params->rx_cq_moderation.usec =
MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_USEC;
- if (cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE)
+ if (cq_period_mode == NET_CQ_PERIOD_MODE_START_FROM_CQE)
params->rx_cq_moderation.usec =
MLX5E_PARAMS_DEFAULT_RX_CQ_MODERATION_USEC_FROM_CQE;
if (params->rx_am_enabled)
params->rx_cq_moderation =
- mlx5e_am_get_def_profile(params->rx_cq_period_mode);
+ net_am_get_def_profile(params->rx_cq_period_mode);
MLX5E_SET_PFLAG(params, MLX5E_PFLAG_RX_CQE_BASED_MODER,
- params->rx_cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE);
+ params->rx_cq_period_mode == NET_CQ_PERIOD_MODE_START_FROM_CQE);
}
u32 mlx5e_choose_lro_timeout(struct mlx5_core_dev *mdev, u32 wanted_timeout)
@@ -4061,8 +4061,8 @@ void mlx5e_build_nic_params(struct mlx5_core_dev *mdev,
/* CQ moderation params */
cq_period_mode = MLX5_CAP_GEN(mdev, cq_period_start_from_cqe) ?
- MLX5_CQ_PERIOD_MODE_START_FROM_CQE :
- MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
+ NET_CQ_PERIOD_MODE_START_FROM_CQE :
+ NET_CQ_PERIOD_MODE_START_FROM_EQE;
params->rx_am_enabled = MLX5_CAP_GEN(mdev, cq_moderation);
mlx5e_set_rx_cq_mode_params(params, cq_period_mode);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
index 2c43606..550d05e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c
@@ -820,8 +820,8 @@ static void mlx5e_build_rep_params(struct mlx5_core_dev *mdev,
struct mlx5e_params *params)
{
u8 cq_period_mode = MLX5_CAP_GEN(mdev, cq_period_start_from_cqe) ?
- MLX5_CQ_PERIOD_MODE_START_FROM_CQE :
- MLX5_CQ_PERIOD_MODE_START_FROM_EQE;
+ NET_CQ_PERIOD_MODE_START_FROM_CQE :
+ NET_CQ_PERIOD_MODE_START_FROM_EQE;
params->log_sq_size = MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE;
params->rq_wq_type = MLX5_WQ_TYPE_LINKED_LIST;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
index b9b434b..1f8fda1 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
@@ -34,15 +34,15 @@
void mlx5e_rx_am_work(struct work_struct *work)
{
- struct mlx5e_rx_am *am = container_of(work, struct mlx5e_rx_am,
+ struct net_rx_am *am = container_of(work, struct net_rx_am,
work);
struct mlx5e_rq *rq = container_of(am, struct mlx5e_rq, am);
- struct mlx5e_cq_moder cur_profile = mlx5e_am_get_profile(am->mode,
+ struct net_cq_moder cur_profile = net_am_get_profile(am->mode,
am->profile_ix);
mlx5_core_modify_cq_moderation(rq->mdev, &rq->cq.mcq,
cur_profile.usec, cur_profile.pkts);
- am->state = MLX5E_AM_START_MEASURE;
+ am->state = NET_AM_START_MEASURE;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
index 8fed6c6..4f8d8d5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c
@@ -77,7 +77,7 @@ int mlx5e_napi_poll(struct napi_struct *napi, int budget)
mlx5e_cq_arm(&c->sq[i].cq);
if (MLX5E_TEST_BIT(c->rq.state, MLX5E_RQ_STATE_AM))
- mlx5e_rx_am(&c->rq.am,
+ net_rx_am(&c->rq.am,
c->rq.cq.event_ctr,
c->rq.stats.packets,
c->rq.stats.bytes);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
index 14cea75..37ea6d1 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
@@ -33,22 +33,22 @@
#include "en.h"
-#define MLX5E_PARAMS_AM_NUM_PROFILES 5
+#define NET_PARAMS_AM_NUM_PROFILES 5
/* Adaptive moderation profiles */
-#define MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE 256
-#define MLX5E_RX_AM_DEF_PROFILE_CQE 1
-#define MLX5E_RX_AM_DEF_PROFILE_EQE 1
-
-/* All profiles sizes must be MLX5E_PARAMS_AM_NUM_PROFILES */
-#define MLX5_AM_EQE_PROFILES { \
- {1, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {8, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {64, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {128, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {256, MLX5E_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+#define NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE 256
+#define NET_RX_AM_DEF_PROFILE_CQE 1
+#define NET_RX_AM_DEF_PROFILE_EQE 1
+
+/* All profiles sizes must be NET_PARAMS_AM_NUM_PROFILES */
+#define NET_AM_EQE_PROFILES { \
+ {1, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {8, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {64, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {128, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {256, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
}
-#define MLX5_AM_CQE_PROFILES { \
+#define NET_AM_CQE_PROFILES { \
{2, 256}, \
{8, 128}, \
{16, 64}, \
@@ -56,135 +56,135 @@
{64, 64} \
}
-static const struct mlx5e_cq_moder
-profile[MLX5_CQ_PERIOD_NUM_MODES][MLX5E_PARAMS_AM_NUM_PROFILES] = {
- MLX5_AM_EQE_PROFILES,
- MLX5_AM_CQE_PROFILES,
+static const struct net_cq_moder
+profile[NET_CQ_PERIOD_NUM_MODES][NET_PARAMS_AM_NUM_PROFILES] = {
+ NET_AM_EQE_PROFILES,
+ NET_AM_CQE_PROFILES,
};
-struct mlx5e_cq_moder mlx5e_am_get_profile(u8 cq_period_mode, int ix)
+struct net_cq_moder net_am_get_profile(u8 cq_period_mode, int ix)
{
return profile[cq_period_mode][ix];
}
-struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode)
+struct net_cq_moder net_am_get_def_profile(u8 rx_cq_period_mode)
{
int default_profile_ix;
- if (rx_cq_period_mode == MLX5_CQ_PERIOD_MODE_START_FROM_CQE)
- default_profile_ix = MLX5E_RX_AM_DEF_PROFILE_CQE;
- else /* MLX5_CQ_PERIOD_MODE_START_FROM_EQE */
- default_profile_ix = MLX5E_RX_AM_DEF_PROFILE_EQE;
+ if (rx_cq_period_mode == NET_CQ_PERIOD_MODE_START_FROM_CQE)
+ default_profile_ix = NET_RX_AM_DEF_PROFILE_CQE;
+ else /* NET_CQ_PERIOD_MODE_START_FROM_EQE */
+ default_profile_ix = NET_RX_AM_DEF_PROFILE_EQE;
return profile[rx_cq_period_mode][default_profile_ix];
}
-static bool mlx5e_am_on_top(struct mlx5e_rx_am *am)
+static bool net_am_on_top(struct net_rx_am *am)
{
switch (am->tune_state) {
- case MLX5E_AM_PARKING_ON_TOP:
- case MLX5E_AM_PARKING_TIRED:
+ case NET_AM_PARKING_ON_TOP:
+ case NET_AM_PARKING_TIRED:
return true;
- case MLX5E_AM_GOING_RIGHT:
+ case NET_AM_GOING_RIGHT:
return (am->steps_left > 1) && (am->steps_right == 1);
- default: /* MLX5E_AM_GOING_LEFT */
+ default: /* NET_AM_GOING_LEFT */
return (am->steps_right > 1) && (am->steps_left == 1);
}
}
-static void mlx5e_am_turn(struct mlx5e_rx_am *am)
+static void net_am_turn(struct net_rx_am *am)
{
switch (am->tune_state) {
- case MLX5E_AM_PARKING_ON_TOP:
- case MLX5E_AM_PARKING_TIRED:
+ case NET_AM_PARKING_ON_TOP:
+ case NET_AM_PARKING_TIRED:
break;
- case MLX5E_AM_GOING_RIGHT:
- am->tune_state = MLX5E_AM_GOING_LEFT;
+ case NET_AM_GOING_RIGHT:
+ am->tune_state = NET_AM_GOING_LEFT;
am->steps_left = 0;
break;
- case MLX5E_AM_GOING_LEFT:
- am->tune_state = MLX5E_AM_GOING_RIGHT;
+ case NET_AM_GOING_LEFT:
+ am->tune_state = NET_AM_GOING_RIGHT;
am->steps_right = 0;
break;
}
}
-static int mlx5e_am_step(struct mlx5e_rx_am *am)
+static int net_am_step(struct net_rx_am *am)
{
- if (am->tired == (MLX5E_PARAMS_AM_NUM_PROFILES * 2))
- return MLX5E_AM_TOO_TIRED;
+ if (am->tired == (NET_PARAMS_AM_NUM_PROFILES * 2))
+ return NET_AM_TOO_TIRED;
switch (am->tune_state) {
- case MLX5E_AM_PARKING_ON_TOP:
- case MLX5E_AM_PARKING_TIRED:
+ case NET_AM_PARKING_ON_TOP:
+ case NET_AM_PARKING_TIRED:
break;
- case MLX5E_AM_GOING_RIGHT:
- if (am->profile_ix == (MLX5E_PARAMS_AM_NUM_PROFILES - 1))
- return MLX5E_AM_ON_EDGE;
+ case NET_AM_GOING_RIGHT:
+ if (am->profile_ix == (NET_PARAMS_AM_NUM_PROFILES - 1))
+ return NET_AM_ON_EDGE;
am->profile_ix++;
am->steps_right++;
break;
- case MLX5E_AM_GOING_LEFT:
+ case NET_AM_GOING_LEFT:
if (am->profile_ix == 0)
- return MLX5E_AM_ON_EDGE;
+ return NET_AM_ON_EDGE;
am->profile_ix--;
am->steps_left++;
break;
}
am->tired++;
- return MLX5E_AM_STEPPED;
+ return NET_AM_STEPPED;
}
-static void mlx5e_am_park_on_top(struct mlx5e_rx_am *am)
+static void net_am_park_on_top(struct net_rx_am *am)
{
am->steps_right = 0;
am->steps_left = 0;
am->tired = 0;
- am->tune_state = MLX5E_AM_PARKING_ON_TOP;
+ am->tune_state = NET_AM_PARKING_ON_TOP;
}
-static void mlx5e_am_park_tired(struct mlx5e_rx_am *am)
+static void net_am_park_tired(struct net_rx_am *am)
{
am->steps_right = 0;
am->steps_left = 0;
- am->tune_state = MLX5E_AM_PARKING_TIRED;
+ am->tune_state = NET_AM_PARKING_TIRED;
}
-static void mlx5e_am_exit_parking(struct mlx5e_rx_am *am)
+static void net_am_exit_parking(struct net_rx_am *am)
{
- am->tune_state = am->profile_ix ? MLX5E_AM_GOING_LEFT :
- MLX5E_AM_GOING_RIGHT;
- mlx5e_am_step(am);
+ am->tune_state = am->profile_ix ? NET_AM_GOING_LEFT :
+ NET_AM_GOING_RIGHT;
+ net_am_step(am);
}
#define IS_SIGNIFICANT_DIFF(val, ref) \
(((100 * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */
-static int mlx5e_am_stats_compare(struct mlx5e_rx_am_stats *curr,
- struct mlx5e_rx_am_stats *prev)
+static int net_am_stats_compare(struct net_rx_am_stats *curr,
+ struct net_rx_am_stats *prev)
{
if (!prev->bpms)
- return curr->bpms ? MLX5E_AM_STATS_BETTER :
- MLX5E_AM_STATS_SAME;
+ return curr->bpms ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_SAME;
if (IS_SIGNIFICANT_DIFF(curr->bpms, prev->bpms))
- return (curr->bpms > prev->bpms) ? MLX5E_AM_STATS_BETTER :
- MLX5E_AM_STATS_WORSE;
+ return (curr->bpms > prev->bpms) ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_WORSE;
if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms))
- return (curr->ppms > prev->ppms) ? MLX5E_AM_STATS_BETTER :
- MLX5E_AM_STATS_WORSE;
+ return (curr->ppms > prev->ppms) ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_WORSE;
if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms))
- return (curr->epms < prev->epms) ? MLX5E_AM_STATS_BETTER :
- MLX5E_AM_STATS_WORSE;
+ return (curr->epms < prev->epms) ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_WORSE;
- return MLX5E_AM_STATS_SAME;
+ return NET_AM_STATS_SAME;
}
-static bool mlx5e_am_decision(struct mlx5e_rx_am_stats *curr_stats,
- struct mlx5e_rx_am *am)
+static bool net_am_decision(struct net_rx_am_stats *curr_stats,
+ struct net_rx_am *am)
{
int prev_state = am->tune_state;
int prev_ix = am->profile_ix;
@@ -192,53 +192,53 @@ static bool mlx5e_am_decision(struct mlx5e_rx_am_stats *curr_stats,
int step_res;
switch (am->tune_state) {
- case MLX5E_AM_PARKING_ON_TOP:
- stats_res = mlx5e_am_stats_compare(curr_stats, &am->prev_stats);
- if (stats_res != MLX5E_AM_STATS_SAME)
- mlx5e_am_exit_parking(am);
+ case NET_AM_PARKING_ON_TOP:
+ stats_res = net_am_stats_compare(curr_stats, &am->prev_stats);
+ if (stats_res != NET_AM_STATS_SAME)
+ net_am_exit_parking(am);
break;
- case MLX5E_AM_PARKING_TIRED:
+ case NET_AM_PARKING_TIRED:
am->tired--;
if (!am->tired)
- mlx5e_am_exit_parking(am);
+ net_am_exit_parking(am);
break;
- case MLX5E_AM_GOING_RIGHT:
- case MLX5E_AM_GOING_LEFT:
- stats_res = mlx5e_am_stats_compare(curr_stats, &am->prev_stats);
- if (stats_res != MLX5E_AM_STATS_BETTER)
- mlx5e_am_turn(am);
+ case NET_AM_GOING_RIGHT:
+ case NET_AM_GOING_LEFT:
+ stats_res = net_am_stats_compare(curr_stats, &am->prev_stats);
+ if (stats_res != NET_AM_STATS_BETTER)
+ net_am_turn(am);
- if (mlx5e_am_on_top(am)) {
- mlx5e_am_park_on_top(am);
+ if (net_am_on_top(am)) {
+ net_am_park_on_top(am);
break;
}
- step_res = mlx5e_am_step(am);
+ step_res = net_am_step(am);
switch (step_res) {
- case MLX5E_AM_ON_EDGE:
- mlx5e_am_park_on_top(am);
+ case NET_AM_ON_EDGE:
+ net_am_park_on_top(am);
break;
- case MLX5E_AM_TOO_TIRED:
- mlx5e_am_park_tired(am);
+ case NET_AM_TOO_TIRED:
+ net_am_park_tired(am);
break;
}
break;
}
- if ((prev_state != MLX5E_AM_PARKING_ON_TOP) ||
- (am->tune_state != MLX5E_AM_PARKING_ON_TOP))
+ if ((prev_state != NET_AM_PARKING_ON_TOP) ||
+ (am->tune_state != NET_AM_PARKING_ON_TOP))
am->prev_stats = *curr_stats;
return am->profile_ix != prev_ix;
}
-static void mlx5e_am_sample(u16 event_ctr,
+static void net_am_sample(u16 event_ctr,
u64 packets,
u64 bytes,
- struct mlx5e_rx_am_sample *s)
+ struct net_rx_am_sample *s)
{
s->time = ktime_get();
s->pkt_ctr = packets;
@@ -246,13 +246,13 @@ static void mlx5e_am_sample(u16 event_ctr,
s->event_ctr = event_ctr;
}
-#define MLX5E_AM_NEVENTS 64
+#define NET_AM_NEVENTS 64
#define BITS_PER_TYPE(type) (sizeof(type) * BITS_PER_BYTE)
#define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1))
-static void mlx5e_am_calc_stats(struct mlx5e_rx_am_sample *start,
- struct mlx5e_rx_am_sample *end,
- struct mlx5e_rx_am_stats *curr_stats)
+static void net_am_calc_stats(struct net_rx_am_sample *start,
+ struct net_rx_am_sample *end,
+ struct net_rx_am_stats *curr_stats)
{
/* u32 holds up to 71 minutes, should be enough */
u32 delta_us = ktime_us_delta(end->time, start->time);
@@ -265,39 +265,39 @@ static void mlx5e_am_calc_stats(struct mlx5e_rx_am_sample *start,
curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us);
curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us);
- curr_stats->epms = DIV_ROUND_UP(MLX5E_AM_NEVENTS * USEC_PER_MSEC,
+ curr_stats->epms = DIV_ROUND_UP(NET_AM_NEVENTS * USEC_PER_MSEC,
delta_us);
}
-void mlx5e_rx_am(struct mlx5e_rx_am *am,
+void net_rx_am(struct net_rx_am *am,
u16 event_ctr,
u64 packets,
u64 bytes)
{
- struct mlx5e_rx_am_sample end_sample;
- struct mlx5e_rx_am_stats curr_stats;
+ struct net_rx_am_sample end_sample;
+ struct net_rx_am_stats curr_stats;
u16 nevents;
switch (am->state) {
- case MLX5E_AM_MEASURE_IN_PROGRESS:
+ case NET_AM_MEASURE_IN_PROGRESS:
nevents = BIT_GAP(BITS_PER_TYPE(u16), event_ctr,
am->start_sample.event_ctr);
- if (nevents < MLX5E_AM_NEVENTS)
+ if (nevents < NET_AM_NEVENTS)
break;
- mlx5e_am_sample(event_ctr, packets, bytes, &end_sample);
- mlx5e_am_calc_stats(&am->start_sample, &end_sample,
+ net_am_sample(event_ctr, packets, bytes, &end_sample);
+ net_am_calc_stats(&am->start_sample, &end_sample,
&curr_stats);
- if (mlx5e_am_decision(&curr_stats, am)) {
- am->state = MLX5E_AM_APPLY_NEW_PROFILE;
+ if (net_am_decision(&curr_stats, am)) {
+ am->state = NET_AM_APPLY_NEW_PROFILE;
schedule_work(&am->work);
break;
}
/* fall through */
- case MLX5E_AM_START_MEASURE:
- mlx5e_am_sample(event_ctr, packets, bytes, &am->start_sample);
- am->state = MLX5E_AM_MEASURE_IN_PROGRESS;
+ case NET_AM_START_MEASURE:
+ net_am_sample(event_ctr, packets, bytes, &am->start_sample);
+ am->state = NET_AM_MEASURE_IN_PROGRESS;
break;
- case MLX5E_AM_APPLY_NEW_PROFILE:
+ case NET_AM_APPLY_NEW_PROFILE:
break;
}
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
index ef86bf8..aa65e79 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
@@ -31,31 +31,31 @@
* SOFTWARE.
*/
-#ifndef MLX5_AM_H
-#define MLX5_AM_H
+#ifndef NET_AM_H
+#define NET_AM_H
-struct mlx5e_cq_moder {
+struct net_cq_moder {
u16 usec;
u16 pkts;
};
-struct mlx5e_rx_am_sample {
+struct net_rx_am_sample {
ktime_t time;
u32 pkt_ctr;
u32 byte_ctr;
u16 event_ctr;
};
-struct mlx5e_rx_am_stats {
+struct net_rx_am_stats {
int ppms; /* packets per msec */
int bpms; /* bytes per msec */
int epms; /* events per msec */
};
-struct mlx5e_rx_am { /* Adaptive Moderation */
+struct net_rx_am { /* Adaptive Moderation */
u8 state;
- struct mlx5e_rx_am_stats prev_stats;
- struct mlx5e_rx_am_sample start_sample;
+ struct net_rx_am_stats prev_stats;
+ struct net_rx_am_sample start_sample;
struct work_struct work;
u8 profile_ix;
u8 mode;
@@ -66,43 +66,42 @@ struct mlx5e_rx_am { /* Adaptive Moderation */
};
enum {
- MLX5_CQ_PERIOD_MODE_START_FROM_EQE = 0x0,
- MLX5_CQ_PERIOD_MODE_START_FROM_CQE = 0x1,
- MLX5_CQ_PERIOD_NUM_MODES
+ NET_CQ_PERIOD_MODE_START_FROM_EQE = 0x0,
+ NET_CQ_PERIOD_MODE_START_FROM_CQE = 0x1,
+ NET_CQ_PERIOD_NUM_MODES
};
/* Adaptive moderation logic */
enum {
- MLX5E_AM_START_MEASURE,
- MLX5E_AM_MEASURE_IN_PROGRESS,
- MLX5E_AM_APPLY_NEW_PROFILE,
+ NET_AM_START_MEASURE,
+ NET_AM_MEASURE_IN_PROGRESS,
+ NET_AM_APPLY_NEW_PROFILE,
};
enum {
- MLX5E_AM_PARKING_ON_TOP,
- MLX5E_AM_PARKING_TIRED,
- MLX5E_AM_GOING_RIGHT,
- MLX5E_AM_GOING_LEFT,
+ NET_AM_PARKING_ON_TOP,
+ NET_AM_PARKING_TIRED,
+ NET_AM_GOING_RIGHT,
+ NET_AM_GOING_LEFT,
};
enum {
- MLX5E_AM_STATS_WORSE,
- MLX5E_AM_STATS_SAME,
- MLX5E_AM_STATS_BETTER,
+ NET_AM_STATS_WORSE,
+ NET_AM_STATS_SAME,
+ NET_AM_STATS_BETTER,
};
enum {
- MLX5E_AM_STEPPED,
- MLX5E_AM_TOO_TIRED,
- MLX5E_AM_ON_EDGE,
+ NET_AM_STEPPED,
+ NET_AM_TOO_TIRED,
+ NET_AM_ON_EDGE,
};
-void mlx5e_rx_am(struct mlx5e_rx_am *am,
+void net_rx_am(struct net_rx_am *am,
u16 event_ctr,
u64 packets,
u64 bytes);
-void mlx5e_rx_am_work(struct work_struct *work);
-struct mlx5e_cq_moder mlx5e_am_get_def_profile(u8 rx_cq_period_mode);
-struct mlx5e_cq_moder mlx5e_am_get_profile(u8 cq_period_mode, int ix);
+struct net_cq_moder net_am_get_def_profile(u8 rx_cq_period_mode);
+struct net_cq_moder net_am_get_profile(u8 cq_period_mode, int ix);
-#endif /* MLX5_AM_H */
+#endif /* NET_AM_H */
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [RFC 8/9] net: move adaptive interrupt coalescing code to lib/
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (8 preceding siblings ...)
2017-11-06 5:44 ` [RFC 7/9] mlx5_en: remove Mellanox references in AM code Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 8/9] net: move adaptive interrpt " Andy Gospodarek
` (3 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
This takes the code that is not generically named to lib/.
This move allows drivers to add private structure elements to track the
number of packets, bytes, and interrupts events per ring. A driver
also defines a workqueue handler to act on this collected data once per
poll and modify the coalescing parameters per ring.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/mellanox/mlx5/core/Makefile | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c | 1 +
.../net/ethernet/mellanox/mlx5/core/net_rx_am.c | 303 --------------------
.../net/ethernet/mellanox/mlx5/core/net_rx_am.h | 107 -------
include/linux/net_rx_am.h | 109 ++++++++
lib/Makefile | 2 +-
lib/net_rx_am.c | 306 +++++++++++++++++++++
8 files changed, 419 insertions(+), 413 deletions(-)
delete mode 100644 drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
delete mode 100644 drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
create mode 100644 include/linux/net_rx_am.h
create mode 100644 lib/net_rx_am.c
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index d5d6d3d..19b21b4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -15,7 +15,7 @@ mlx5_core-$(CONFIG_MLX5_FPGA) += fpga/cmd.o fpga/core.o fpga/conn.o fpga/sdk.o \
mlx5_core-$(CONFIG_MLX5_CORE_EN) += en_main.o en_common.o en_fs.o en_ethtool.o \
en_tx.o en_rx.o en_rx_am.o en_txrx.o en_stats.o vxlan.o \
- en_arfs.o en_fs_ethtool.o en_selftest.o net_rx_am.o
+ en_arfs.o en_fs_ethtool.o en_selftest.o
mlx5_core-$(CONFIG_MLX5_MPFS) += lib/mpfs.o
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 203dc7b..04b36fe 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -49,7 +49,7 @@
#include "wq.h"
#include "mlx5_core.h"
#include "en_stats.h"
-#include "net_rx_am.h"
+#include <linux/net_rx_am.h>
#define MLX5_SET_CFG(p, f, v) MLX5_SET(create_flow_group_in, p, f, v)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
index 1f8fda1..391f1ba 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
@@ -31,6 +31,7 @@
*/
#include "en.h"
+#include <linux/net_rx_am.h>
void mlx5e_rx_am_work(struct work_struct *work)
{
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
deleted file mode 100644
index 37ea6d1..0000000
--- a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
+++ /dev/null
@@ -1,303 +0,0 @@
-/*
- * Copyright (c) 2016, Mellanox Technologies. All rights reserved.
- * Copyright (c) 2017, Broadcom Limiited. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include "en.h"
-
-#define NET_PARAMS_AM_NUM_PROFILES 5
-/* Adaptive moderation profiles */
-#define NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE 256
-#define NET_RX_AM_DEF_PROFILE_CQE 1
-#define NET_RX_AM_DEF_PROFILE_EQE 1
-
-/* All profiles sizes must be NET_PARAMS_AM_NUM_PROFILES */
-#define NET_AM_EQE_PROFILES { \
- {1, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {8, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {64, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {128, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {256, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
-}
-
-#define NET_AM_CQE_PROFILES { \
- {2, 256}, \
- {8, 128}, \
- {16, 64}, \
- {32, 64}, \
- {64, 64} \
-}
-
-static const struct net_cq_moder
-profile[NET_CQ_PERIOD_NUM_MODES][NET_PARAMS_AM_NUM_PROFILES] = {
- NET_AM_EQE_PROFILES,
- NET_AM_CQE_PROFILES,
-};
-
-struct net_cq_moder net_am_get_profile(u8 cq_period_mode, int ix)
-{
- return profile[cq_period_mode][ix];
-}
-
-struct net_cq_moder net_am_get_def_profile(u8 rx_cq_period_mode)
-{
- int default_profile_ix;
-
- if (rx_cq_period_mode == NET_CQ_PERIOD_MODE_START_FROM_CQE)
- default_profile_ix = NET_RX_AM_DEF_PROFILE_CQE;
- else /* NET_CQ_PERIOD_MODE_START_FROM_EQE */
- default_profile_ix = NET_RX_AM_DEF_PROFILE_EQE;
-
- return profile[rx_cq_period_mode][default_profile_ix];
-}
-
-static bool net_am_on_top(struct net_rx_am *am)
-{
- switch (am->tune_state) {
- case NET_AM_PARKING_ON_TOP:
- case NET_AM_PARKING_TIRED:
- return true;
- case NET_AM_GOING_RIGHT:
- return (am->steps_left > 1) && (am->steps_right == 1);
- default: /* NET_AM_GOING_LEFT */
- return (am->steps_right > 1) && (am->steps_left == 1);
- }
-}
-
-static void net_am_turn(struct net_rx_am *am)
-{
- switch (am->tune_state) {
- case NET_AM_PARKING_ON_TOP:
- case NET_AM_PARKING_TIRED:
- break;
- case NET_AM_GOING_RIGHT:
- am->tune_state = NET_AM_GOING_LEFT;
- am->steps_left = 0;
- break;
- case NET_AM_GOING_LEFT:
- am->tune_state = NET_AM_GOING_RIGHT;
- am->steps_right = 0;
- break;
- }
-}
-
-static int net_am_step(struct net_rx_am *am)
-{
- if (am->tired == (NET_PARAMS_AM_NUM_PROFILES * 2))
- return NET_AM_TOO_TIRED;
-
- switch (am->tune_state) {
- case NET_AM_PARKING_ON_TOP:
- case NET_AM_PARKING_TIRED:
- break;
- case NET_AM_GOING_RIGHT:
- if (am->profile_ix == (NET_PARAMS_AM_NUM_PROFILES - 1))
- return NET_AM_ON_EDGE;
- am->profile_ix++;
- am->steps_right++;
- break;
- case NET_AM_GOING_LEFT:
- if (am->profile_ix == 0)
- return NET_AM_ON_EDGE;
- am->profile_ix--;
- am->steps_left++;
- break;
- }
-
- am->tired++;
- return NET_AM_STEPPED;
-}
-
-static void net_am_park_on_top(struct net_rx_am *am)
-{
- am->steps_right = 0;
- am->steps_left = 0;
- am->tired = 0;
- am->tune_state = NET_AM_PARKING_ON_TOP;
-}
-
-static void net_am_park_tired(struct net_rx_am *am)
-{
- am->steps_right = 0;
- am->steps_left = 0;
- am->tune_state = NET_AM_PARKING_TIRED;
-}
-
-static void net_am_exit_parking(struct net_rx_am *am)
-{
- am->tune_state = am->profile_ix ? NET_AM_GOING_LEFT :
- NET_AM_GOING_RIGHT;
- net_am_step(am);
-}
-
-#define IS_SIGNIFICANT_DIFF(val, ref) \
- (((100 * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */
-
-static int net_am_stats_compare(struct net_rx_am_stats *curr,
- struct net_rx_am_stats *prev)
-{
- if (!prev->bpms)
- return curr->bpms ? NET_AM_STATS_BETTER :
- NET_AM_STATS_SAME;
-
- if (IS_SIGNIFICANT_DIFF(curr->bpms, prev->bpms))
- return (curr->bpms > prev->bpms) ? NET_AM_STATS_BETTER :
- NET_AM_STATS_WORSE;
-
- if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms))
- return (curr->ppms > prev->ppms) ? NET_AM_STATS_BETTER :
- NET_AM_STATS_WORSE;
-
- if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms))
- return (curr->epms < prev->epms) ? NET_AM_STATS_BETTER :
- NET_AM_STATS_WORSE;
-
- return NET_AM_STATS_SAME;
-}
-
-static bool net_am_decision(struct net_rx_am_stats *curr_stats,
- struct net_rx_am *am)
-{
- int prev_state = am->tune_state;
- int prev_ix = am->profile_ix;
- int stats_res;
- int step_res;
-
- switch (am->tune_state) {
- case NET_AM_PARKING_ON_TOP:
- stats_res = net_am_stats_compare(curr_stats, &am->prev_stats);
- if (stats_res != NET_AM_STATS_SAME)
- net_am_exit_parking(am);
- break;
-
- case NET_AM_PARKING_TIRED:
- am->tired--;
- if (!am->tired)
- net_am_exit_parking(am);
- break;
-
- case NET_AM_GOING_RIGHT:
- case NET_AM_GOING_LEFT:
- stats_res = net_am_stats_compare(curr_stats, &am->prev_stats);
- if (stats_res != NET_AM_STATS_BETTER)
- net_am_turn(am);
-
- if (net_am_on_top(am)) {
- net_am_park_on_top(am);
- break;
- }
-
- step_res = net_am_step(am);
- switch (step_res) {
- case NET_AM_ON_EDGE:
- net_am_park_on_top(am);
- break;
- case NET_AM_TOO_TIRED:
- net_am_park_tired(am);
- break;
- }
-
- break;
- }
-
- if ((prev_state != NET_AM_PARKING_ON_TOP) ||
- (am->tune_state != NET_AM_PARKING_ON_TOP))
- am->prev_stats = *curr_stats;
-
- return am->profile_ix != prev_ix;
-}
-
-static void net_am_sample(u16 event_ctr,
- u64 packets,
- u64 bytes,
- struct net_rx_am_sample *s)
-{
- s->time = ktime_get();
- s->pkt_ctr = packets;
- s->byte_ctr = bytes;
- s->event_ctr = event_ctr;
-}
-
-#define NET_AM_NEVENTS 64
-#define BITS_PER_TYPE(type) (sizeof(type) * BITS_PER_BYTE)
-#define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1))
-
-static void net_am_calc_stats(struct net_rx_am_sample *start,
- struct net_rx_am_sample *end,
- struct net_rx_am_stats *curr_stats)
-{
- /* u32 holds up to 71 minutes, should be enough */
- u32 delta_us = ktime_us_delta(end->time, start->time);
- u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr);
- u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr,
- start->byte_ctr);
-
- if (!delta_us)
- return;
-
- curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us);
- curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us);
- curr_stats->epms = DIV_ROUND_UP(NET_AM_NEVENTS * USEC_PER_MSEC,
- delta_us);
-}
-
-void net_rx_am(struct net_rx_am *am,
- u16 event_ctr,
- u64 packets,
- u64 bytes)
-{
- struct net_rx_am_sample end_sample;
- struct net_rx_am_stats curr_stats;
- u16 nevents;
-
- switch (am->state) {
- case NET_AM_MEASURE_IN_PROGRESS:
- nevents = BIT_GAP(BITS_PER_TYPE(u16), event_ctr,
- am->start_sample.event_ctr);
- if (nevents < NET_AM_NEVENTS)
- break;
- net_am_sample(event_ctr, packets, bytes, &end_sample);
- net_am_calc_stats(&am->start_sample, &end_sample,
- &curr_stats);
- if (net_am_decision(&curr_stats, am)) {
- am->state = NET_AM_APPLY_NEW_PROFILE;
- schedule_work(&am->work);
- break;
- }
- /* fall through */
- case NET_AM_START_MEASURE:
- net_am_sample(event_ctr, packets, bytes, &am->start_sample);
- am->state = NET_AM_MEASURE_IN_PROGRESS;
- break;
- case NET_AM_APPLY_NEW_PROFILE:
- break;
- }
-}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
deleted file mode 100644
index aa65e79..0000000
--- a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
+++ /dev/null
@@ -1,107 +0,0 @@
-/*
- * Copyright (c) 2013-2015, Mellanox Technologies, Ltd. All rights reserved.
- * Copyright (c) 2017, Broadcom Limited
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
-*/
-
-#ifndef NET_AM_H
-#define NET_AM_H
-
-struct net_cq_moder {
- u16 usec;
- u16 pkts;
-};
-
-struct net_rx_am_sample {
- ktime_t time;
- u32 pkt_ctr;
- u32 byte_ctr;
- u16 event_ctr;
-};
-
-struct net_rx_am_stats {
- int ppms; /* packets per msec */
- int bpms; /* bytes per msec */
- int epms; /* events per msec */
-};
-
-struct net_rx_am { /* Adaptive Moderation */
- u8 state;
- struct net_rx_am_stats prev_stats;
- struct net_rx_am_sample start_sample;
- struct work_struct work;
- u8 profile_ix;
- u8 mode;
- u8 tune_state;
- u8 steps_right;
- u8 steps_left;
- u8 tired;
-};
-
-enum {
- NET_CQ_PERIOD_MODE_START_FROM_EQE = 0x0,
- NET_CQ_PERIOD_MODE_START_FROM_CQE = 0x1,
- NET_CQ_PERIOD_NUM_MODES
-};
-
-/* Adaptive moderation logic */
-enum {
- NET_AM_START_MEASURE,
- NET_AM_MEASURE_IN_PROGRESS,
- NET_AM_APPLY_NEW_PROFILE,
-};
-
-enum {
- NET_AM_PARKING_ON_TOP,
- NET_AM_PARKING_TIRED,
- NET_AM_GOING_RIGHT,
- NET_AM_GOING_LEFT,
-};
-
-enum {
- NET_AM_STATS_WORSE,
- NET_AM_STATS_SAME,
- NET_AM_STATS_BETTER,
-};
-
-enum {
- NET_AM_STEPPED,
- NET_AM_TOO_TIRED,
- NET_AM_ON_EDGE,
-};
-
-void net_rx_am(struct net_rx_am *am,
- u16 event_ctr,
- u64 packets,
- u64 bytes);
-struct net_cq_moder net_am_get_def_profile(u8 rx_cq_period_mode);
-struct net_cq_moder net_am_get_profile(u8 cq_period_mode, int ix);
-
-#endif /* NET_AM_H */
diff --git a/include/linux/net_rx_am.h b/include/linux/net_rx_am.h
new file mode 100644
index 0000000..115cbde
--- /dev/null
+++ b/include/linux/net_rx_am.h
@@ -0,0 +1,109 @@
+/*
+ * Copyright (c) 2013-2015, Mellanox Technologies, Ltd. All rights reserved.
+ * Copyright (c) 2017, Broadcom Limited
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+*/
+
+#ifndef NET_AM_H
+#define NET_AM_H
+
+#include <linux/module.h>
+
+struct net_cq_moder {
+ u16 usec;
+ u16 pkts;
+};
+
+struct net_rx_am_sample {
+ ktime_t time;
+ u32 pkt_ctr;
+ u32 byte_ctr;
+ u16 event_ctr;
+};
+
+struct net_rx_am_stats {
+ int ppms; /* packets per msec */
+ int bpms; /* bytes per msec */
+ int epms; /* events per msec */
+};
+
+struct net_rx_am { /* Adaptive Moderation */
+ u8 state;
+ struct net_rx_am_stats prev_stats;
+ struct net_rx_am_sample start_sample;
+ struct work_struct work;
+ u8 profile_ix;
+ u8 mode;
+ u8 tune_state;
+ u8 steps_right;
+ u8 steps_left;
+ u8 tired;
+};
+
+enum {
+ NET_CQ_PERIOD_MODE_START_FROM_EQE = 0x0,
+ NET_CQ_PERIOD_MODE_START_FROM_CQE = 0x1,
+ NET_CQ_PERIOD_NUM_MODES
+};
+
+/* Adaptive moderation logic */
+enum {
+ NET_AM_START_MEASURE,
+ NET_AM_MEASURE_IN_PROGRESS,
+ NET_AM_APPLY_NEW_PROFILE,
+};
+
+enum {
+ NET_AM_PARKING_ON_TOP,
+ NET_AM_PARKING_TIRED,
+ NET_AM_GOING_RIGHT,
+ NET_AM_GOING_LEFT,
+};
+
+enum {
+ NET_AM_STATS_WORSE,
+ NET_AM_STATS_SAME,
+ NET_AM_STATS_BETTER,
+};
+
+enum {
+ NET_AM_STEPPED,
+ NET_AM_TOO_TIRED,
+ NET_AM_ON_EDGE,
+};
+
+void net_rx_am(struct net_rx_am *am,
+ u16 event_ctr,
+ u64 packets,
+ u64 bytes);
+struct net_cq_moder net_am_get_def_profile(u8 rx_cq_period_mode);
+struct net_cq_moder net_am_get_profile(u8 cq_period_mode, int ix);
+
+#endif /* NET_AM_H */
diff --git a/lib/Makefile b/lib/Makefile
index b8f2c16..8f6bdc6 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -39,7 +39,7 @@ obj-y += bcd.o div64.o sort.o parser.o debug_locks.o random32.o \
gcd.o lcm.o list_sort.o uuid.o flex_array.o iov_iter.o clz_ctz.o \
bsearch.o find_bit.o llist.o memweight.o kfifo.o \
percpu-refcount.o percpu_ida.o rhashtable.o reciprocal_div.o \
- once.o refcount.o usercopy.o errseq.o
+ once.o refcount.o usercopy.o errseq.o net_rx_am.o
obj-y += string_helpers.o
obj-$(CONFIG_TEST_STRING_HELPERS) += test-string_helpers.o
obj-y += hexdump.o
diff --git a/lib/net_rx_am.c b/lib/net_rx_am.c
new file mode 100644
index 0000000..3167ffa
--- /dev/null
+++ b/lib/net_rx_am.c
@@ -0,0 +1,306 @@
+/*
+ * Copyright (c) 2016, Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2017, Broadcom Limiited. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <linux/net_rx_am.h>
+
+#define NET_PARAMS_AM_NUM_PROFILES 5
+/* Adaptive moderation profiles */
+#define NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE 256
+#define NET_RX_AM_DEF_PROFILE_CQE 1
+#define NET_RX_AM_DEF_PROFILE_EQE 1
+
+/* All profiles sizes must be NET_PARAMS_AM_NUM_PROFILES */
+#define NET_AM_EQE_PROFILES { \
+ {1, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {8, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {64, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {128, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {256, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+}
+
+#define NET_AM_CQE_PROFILES { \
+ {2, 256}, \
+ {8, 128}, \
+ {16, 64}, \
+ {32, 64}, \
+ {64, 64} \
+}
+
+static const struct net_cq_moder
+profile[NET_CQ_PERIOD_NUM_MODES][NET_PARAMS_AM_NUM_PROFILES] = {
+ NET_AM_EQE_PROFILES,
+ NET_AM_CQE_PROFILES,
+};
+
+struct net_cq_moder net_am_get_profile(u8 cq_period_mode, int ix)
+{
+ return profile[cq_period_mode][ix];
+}
+EXPORT_SYMBOL_GPL(net_am_get_profile);
+
+struct net_cq_moder net_am_get_def_profile(u8 rx_cq_period_mode)
+{
+ int default_profile_ix;
+
+ if (rx_cq_period_mode == NET_CQ_PERIOD_MODE_START_FROM_CQE)
+ default_profile_ix = NET_RX_AM_DEF_PROFILE_CQE;
+ else /* NET_CQ_PERIOD_MODE_START_FROM_EQE */
+ default_profile_ix = NET_RX_AM_DEF_PROFILE_EQE;
+
+ return profile[rx_cq_period_mode][default_profile_ix];
+}
+EXPORT_SYMBOL_GPL(net_am_get_def_profile);
+
+static bool net_am_on_top(struct net_rx_am *am)
+{
+ switch (am->tune_state) {
+ case NET_AM_PARKING_ON_TOP:
+ case NET_AM_PARKING_TIRED:
+ return true;
+ case NET_AM_GOING_RIGHT:
+ return (am->steps_left > 1) && (am->steps_right == 1);
+ default: /* NET_AM_GOING_LEFT */
+ return (am->steps_right > 1) && (am->steps_left == 1);
+ }
+}
+
+static void net_am_turn(struct net_rx_am *am)
+{
+ switch (am->tune_state) {
+ case NET_AM_PARKING_ON_TOP:
+ case NET_AM_PARKING_TIRED:
+ break;
+ case NET_AM_GOING_RIGHT:
+ am->tune_state = NET_AM_GOING_LEFT;
+ am->steps_left = 0;
+ break;
+ case NET_AM_GOING_LEFT:
+ am->tune_state = NET_AM_GOING_RIGHT;
+ am->steps_right = 0;
+ break;
+ }
+}
+
+static int net_am_step(struct net_rx_am *am)
+{
+ if (am->tired == (NET_PARAMS_AM_NUM_PROFILES * 2))
+ return NET_AM_TOO_TIRED;
+
+ switch (am->tune_state) {
+ case NET_AM_PARKING_ON_TOP:
+ case NET_AM_PARKING_TIRED:
+ break;
+ case NET_AM_GOING_RIGHT:
+ if (am->profile_ix == (NET_PARAMS_AM_NUM_PROFILES - 1))
+ return NET_AM_ON_EDGE;
+ am->profile_ix++;
+ am->steps_right++;
+ break;
+ case NET_AM_GOING_LEFT:
+ if (am->profile_ix == 0)
+ return NET_AM_ON_EDGE;
+ am->profile_ix--;
+ am->steps_left++;
+ break;
+ }
+
+ am->tired++;
+ return NET_AM_STEPPED;
+}
+
+static void net_am_park_on_top(struct net_rx_am *am)
+{
+ am->steps_right = 0;
+ am->steps_left = 0;
+ am->tired = 0;
+ am->tune_state = NET_AM_PARKING_ON_TOP;
+}
+
+static void net_am_park_tired(struct net_rx_am *am)
+{
+ am->steps_right = 0;
+ am->steps_left = 0;
+ am->tune_state = NET_AM_PARKING_TIRED;
+}
+
+static void net_am_exit_parking(struct net_rx_am *am)
+{
+ am->tune_state = am->profile_ix ? NET_AM_GOING_LEFT :
+ NET_AM_GOING_RIGHT;
+ net_am_step(am);
+}
+
+#define IS_SIGNIFICANT_DIFF(val, ref) \
+ (((100 * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */
+
+static int net_am_stats_compare(struct net_rx_am_stats *curr,
+ struct net_rx_am_stats *prev)
+{
+ if (!prev->bpms)
+ return curr->bpms ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_SAME;
+
+ if (IS_SIGNIFICANT_DIFF(curr->bpms, prev->bpms))
+ return (curr->bpms > prev->bpms) ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_WORSE;
+
+ if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms))
+ return (curr->ppms > prev->ppms) ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_WORSE;
+
+ if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms))
+ return (curr->epms < prev->epms) ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_WORSE;
+
+ return NET_AM_STATS_SAME;
+}
+
+static bool net_am_decision(struct net_rx_am_stats *curr_stats,
+ struct net_rx_am *am)
+{
+ int prev_state = am->tune_state;
+ int prev_ix = am->profile_ix;
+ int stats_res;
+ int step_res;
+
+ switch (am->tune_state) {
+ case NET_AM_PARKING_ON_TOP:
+ stats_res = net_am_stats_compare(curr_stats, &am->prev_stats);
+ if (stats_res != NET_AM_STATS_SAME)
+ net_am_exit_parking(am);
+ break;
+
+ case NET_AM_PARKING_TIRED:
+ am->tired--;
+ if (!am->tired)
+ net_am_exit_parking(am);
+ break;
+
+ case NET_AM_GOING_RIGHT:
+ case NET_AM_GOING_LEFT:
+ stats_res = net_am_stats_compare(curr_stats, &am->prev_stats);
+ if (stats_res != NET_AM_STATS_BETTER)
+ net_am_turn(am);
+
+ if (net_am_on_top(am)) {
+ net_am_park_on_top(am);
+ break;
+ }
+
+ step_res = net_am_step(am);
+ switch (step_res) {
+ case NET_AM_ON_EDGE:
+ net_am_park_on_top(am);
+ break;
+ case NET_AM_TOO_TIRED:
+ net_am_park_tired(am);
+ break;
+ }
+
+ break;
+ }
+
+ if ((prev_state != NET_AM_PARKING_ON_TOP) ||
+ (am->tune_state != NET_AM_PARKING_ON_TOP))
+ am->prev_stats = *curr_stats;
+
+ return am->profile_ix != prev_ix;
+}
+
+static void net_am_sample(u16 event_ctr,
+ u64 packets,
+ u64 bytes,
+ struct net_rx_am_sample *s)
+{
+ s->time = ktime_get();
+ s->pkt_ctr = packets;
+ s->byte_ctr = bytes;
+ s->event_ctr = event_ctr;
+}
+
+#define NET_AM_NEVENTS 64
+#define BITS_PER_TYPE(type) (sizeof(type) * BITS_PER_BYTE)
+#define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1))
+
+static void net_am_calc_stats(struct net_rx_am_sample *start,
+ struct net_rx_am_sample *end,
+ struct net_rx_am_stats *curr_stats)
+{
+ /* u32 holds up to 71 minutes, should be enough */
+ u32 delta_us = ktime_us_delta(end->time, start->time);
+ u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr);
+ u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr,
+ start->byte_ctr);
+
+ if (!delta_us)
+ return;
+
+ curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us);
+ curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us);
+ curr_stats->epms = DIV_ROUND_UP(NET_AM_NEVENTS * USEC_PER_MSEC,
+ delta_us);
+}
+
+void net_rx_am(struct net_rx_am *am,
+ u16 event_ctr,
+ u64 packets,
+ u64 bytes)
+{
+ struct net_rx_am_sample end_sample;
+ struct net_rx_am_stats curr_stats;
+ u16 nevents;
+
+ switch (am->state) {
+ case NET_AM_MEASURE_IN_PROGRESS:
+ nevents = BIT_GAP(BITS_PER_TYPE(u16), event_ctr,
+ am->start_sample.event_ctr);
+ if (nevents < NET_AM_NEVENTS)
+ break;
+ net_am_sample(event_ctr, packets, bytes, &end_sample);
+ net_am_calc_stats(&am->start_sample, &end_sample,
+ &curr_stats);
+ if (net_am_decision(&curr_stats, am)) {
+ am->state = NET_AM_APPLY_NEW_PROFILE;
+ schedule_work(&am->work);
+ break;
+ }
+ /* fall through */
+ case NET_AM_START_MEASURE:
+ net_am_sample(event_ctr, packets, bytes, &am->start_sample);
+ am->state = NET_AM_MEASURE_IN_PROGRESS;
+ break;
+ case NET_AM_APPLY_NEW_PROFILE:
+ break;
+ }
+}
+EXPORT_SYMBOL_GPL(net_rx_am);
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [RFC 8/9] net: move adaptive interrpt coalescing code to lib/
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (9 preceding siblings ...)
2017-11-06 5:44 ` [RFC 8/9] net: move adaptive interrupt coalescing code to lib/ Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 5:44 ` [RFC 9/9] bnxt_en: add support for software adaptive interrupt moderation Andy Gospodarek
` (2 subsequent siblings)
13 siblings, 0 replies; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
This takes the code that is not generically named to lib/.
This move allows drivers to add private structure elements to track the
number of packets, bytes, and interrupts events per ring. A driver
also defines a workqueue handler to act on this collected data once per
poll and modify the coalecing paramets per ring.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/mellanox/mlx5/core/Makefile | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/en.h | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c | 1 +
.../net/ethernet/mellanox/mlx5/core/net_rx_am.c | 303 --------------------
.../net/ethernet/mellanox/mlx5/core/net_rx_am.h | 107 -------
include/linux/net_rx_am.h | 109 ++++++++
lib/Makefile | 2 +-
lib/net_rx_am.c | 306 +++++++++++++++++++++
8 files changed, 419 insertions(+), 413 deletions(-)
delete mode 100644 drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
delete mode 100644 drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
create mode 100644 include/linux/net_rx_am.h
create mode 100644 lib/net_rx_am.c
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index d5d6d3d..19b21b4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -15,7 +15,7 @@ mlx5_core-$(CONFIG_MLX5_FPGA) += fpga/cmd.o fpga/core.o fpga/conn.o fpga/sdk.o \
mlx5_core-$(CONFIG_MLX5_CORE_EN) += en_main.o en_common.o en_fs.o en_ethtool.o \
en_tx.o en_rx.o en_rx_am.o en_txrx.o en_stats.o vxlan.o \
- en_arfs.o en_fs_ethtool.o en_selftest.o net_rx_am.o
+ en_arfs.o en_fs_ethtool.o en_selftest.o
mlx5_core-$(CONFIG_MLX5_MPFS) += lib/mpfs.o
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 203dc7b..04b36fe 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -49,7 +49,7 @@
#include "wq.h"
#include "mlx5_core.h"
#include "en_stats.h"
-#include "net_rx_am.h"
+#include <linux/net_rx_am.h>
#define MLX5_SET_CFG(p, f, v) MLX5_SET(create_flow_group_in, p, f, v)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
index 1f8fda1..391f1ba 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
@@ -31,6 +31,7 @@
*/
#include "en.h"
+#include <linux/net_rx_am.h>
void mlx5e_rx_am_work(struct work_struct *work)
{
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
deleted file mode 100644
index 37ea6d1..0000000
--- a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.c
+++ /dev/null
@@ -1,303 +0,0 @@
-/*
- * Copyright (c) 2016, Mellanox Technologies. All rights reserved.
- * Copyright (c) 2017, Broadcom Limiited. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include "en.h"
-
-#define NET_PARAMS_AM_NUM_PROFILES 5
-/* Adaptive moderation profiles */
-#define NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE 256
-#define NET_RX_AM_DEF_PROFILE_CQE 1
-#define NET_RX_AM_DEF_PROFILE_EQE 1
-
-/* All profiles sizes must be NET_PARAMS_AM_NUM_PROFILES */
-#define NET_AM_EQE_PROFILES { \
- {1, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {8, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {64, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {128, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
- {256, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
-}
-
-#define NET_AM_CQE_PROFILES { \
- {2, 256}, \
- {8, 128}, \
- {16, 64}, \
- {32, 64}, \
- {64, 64} \
-}
-
-static const struct net_cq_moder
-profile[NET_CQ_PERIOD_NUM_MODES][NET_PARAMS_AM_NUM_PROFILES] = {
- NET_AM_EQE_PROFILES,
- NET_AM_CQE_PROFILES,
-};
-
-struct net_cq_moder net_am_get_profile(u8 cq_period_mode, int ix)
-{
- return profile[cq_period_mode][ix];
-}
-
-struct net_cq_moder net_am_get_def_profile(u8 rx_cq_period_mode)
-{
- int default_profile_ix;
-
- if (rx_cq_period_mode == NET_CQ_PERIOD_MODE_START_FROM_CQE)
- default_profile_ix = NET_RX_AM_DEF_PROFILE_CQE;
- else /* NET_CQ_PERIOD_MODE_START_FROM_EQE */
- default_profile_ix = NET_RX_AM_DEF_PROFILE_EQE;
-
- return profile[rx_cq_period_mode][default_profile_ix];
-}
-
-static bool net_am_on_top(struct net_rx_am *am)
-{
- switch (am->tune_state) {
- case NET_AM_PARKING_ON_TOP:
- case NET_AM_PARKING_TIRED:
- return true;
- case NET_AM_GOING_RIGHT:
- return (am->steps_left > 1) && (am->steps_right == 1);
- default: /* NET_AM_GOING_LEFT */
- return (am->steps_right > 1) && (am->steps_left == 1);
- }
-}
-
-static void net_am_turn(struct net_rx_am *am)
-{
- switch (am->tune_state) {
- case NET_AM_PARKING_ON_TOP:
- case NET_AM_PARKING_TIRED:
- break;
- case NET_AM_GOING_RIGHT:
- am->tune_state = NET_AM_GOING_LEFT;
- am->steps_left = 0;
- break;
- case NET_AM_GOING_LEFT:
- am->tune_state = NET_AM_GOING_RIGHT;
- am->steps_right = 0;
- break;
- }
-}
-
-static int net_am_step(struct net_rx_am *am)
-{
- if (am->tired == (NET_PARAMS_AM_NUM_PROFILES * 2))
- return NET_AM_TOO_TIRED;
-
- switch (am->tune_state) {
- case NET_AM_PARKING_ON_TOP:
- case NET_AM_PARKING_TIRED:
- break;
- case NET_AM_GOING_RIGHT:
- if (am->profile_ix == (NET_PARAMS_AM_NUM_PROFILES - 1))
- return NET_AM_ON_EDGE;
- am->profile_ix++;
- am->steps_right++;
- break;
- case NET_AM_GOING_LEFT:
- if (am->profile_ix == 0)
- return NET_AM_ON_EDGE;
- am->profile_ix--;
- am->steps_left++;
- break;
- }
-
- am->tired++;
- return NET_AM_STEPPED;
-}
-
-static void net_am_park_on_top(struct net_rx_am *am)
-{
- am->steps_right = 0;
- am->steps_left = 0;
- am->tired = 0;
- am->tune_state = NET_AM_PARKING_ON_TOP;
-}
-
-static void net_am_park_tired(struct net_rx_am *am)
-{
- am->steps_right = 0;
- am->steps_left = 0;
- am->tune_state = NET_AM_PARKING_TIRED;
-}
-
-static void net_am_exit_parking(struct net_rx_am *am)
-{
- am->tune_state = am->profile_ix ? NET_AM_GOING_LEFT :
- NET_AM_GOING_RIGHT;
- net_am_step(am);
-}
-
-#define IS_SIGNIFICANT_DIFF(val, ref) \
- (((100 * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */
-
-static int net_am_stats_compare(struct net_rx_am_stats *curr,
- struct net_rx_am_stats *prev)
-{
- if (!prev->bpms)
- return curr->bpms ? NET_AM_STATS_BETTER :
- NET_AM_STATS_SAME;
-
- if (IS_SIGNIFICANT_DIFF(curr->bpms, prev->bpms))
- return (curr->bpms > prev->bpms) ? NET_AM_STATS_BETTER :
- NET_AM_STATS_WORSE;
-
- if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms))
- return (curr->ppms > prev->ppms) ? NET_AM_STATS_BETTER :
- NET_AM_STATS_WORSE;
-
- if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms))
- return (curr->epms < prev->epms) ? NET_AM_STATS_BETTER :
- NET_AM_STATS_WORSE;
-
- return NET_AM_STATS_SAME;
-}
-
-static bool net_am_decision(struct net_rx_am_stats *curr_stats,
- struct net_rx_am *am)
-{
- int prev_state = am->tune_state;
- int prev_ix = am->profile_ix;
- int stats_res;
- int step_res;
-
- switch (am->tune_state) {
- case NET_AM_PARKING_ON_TOP:
- stats_res = net_am_stats_compare(curr_stats, &am->prev_stats);
- if (stats_res != NET_AM_STATS_SAME)
- net_am_exit_parking(am);
- break;
-
- case NET_AM_PARKING_TIRED:
- am->tired--;
- if (!am->tired)
- net_am_exit_parking(am);
- break;
-
- case NET_AM_GOING_RIGHT:
- case NET_AM_GOING_LEFT:
- stats_res = net_am_stats_compare(curr_stats, &am->prev_stats);
- if (stats_res != NET_AM_STATS_BETTER)
- net_am_turn(am);
-
- if (net_am_on_top(am)) {
- net_am_park_on_top(am);
- break;
- }
-
- step_res = net_am_step(am);
- switch (step_res) {
- case NET_AM_ON_EDGE:
- net_am_park_on_top(am);
- break;
- case NET_AM_TOO_TIRED:
- net_am_park_tired(am);
- break;
- }
-
- break;
- }
-
- if ((prev_state != NET_AM_PARKING_ON_TOP) ||
- (am->tune_state != NET_AM_PARKING_ON_TOP))
- am->prev_stats = *curr_stats;
-
- return am->profile_ix != prev_ix;
-}
-
-static void net_am_sample(u16 event_ctr,
- u64 packets,
- u64 bytes,
- struct net_rx_am_sample *s)
-{
- s->time = ktime_get();
- s->pkt_ctr = packets;
- s->byte_ctr = bytes;
- s->event_ctr = event_ctr;
-}
-
-#define NET_AM_NEVENTS 64
-#define BITS_PER_TYPE(type) (sizeof(type) * BITS_PER_BYTE)
-#define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1))
-
-static void net_am_calc_stats(struct net_rx_am_sample *start,
- struct net_rx_am_sample *end,
- struct net_rx_am_stats *curr_stats)
-{
- /* u32 holds up to 71 minutes, should be enough */
- u32 delta_us = ktime_us_delta(end->time, start->time);
- u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr);
- u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr,
- start->byte_ctr);
-
- if (!delta_us)
- return;
-
- curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us);
- curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us);
- curr_stats->epms = DIV_ROUND_UP(NET_AM_NEVENTS * USEC_PER_MSEC,
- delta_us);
-}
-
-void net_rx_am(struct net_rx_am *am,
- u16 event_ctr,
- u64 packets,
- u64 bytes)
-{
- struct net_rx_am_sample end_sample;
- struct net_rx_am_stats curr_stats;
- u16 nevents;
-
- switch (am->state) {
- case NET_AM_MEASURE_IN_PROGRESS:
- nevents = BIT_GAP(BITS_PER_TYPE(u16), event_ctr,
- am->start_sample.event_ctr);
- if (nevents < NET_AM_NEVENTS)
- break;
- net_am_sample(event_ctr, packets, bytes, &end_sample);
- net_am_calc_stats(&am->start_sample, &end_sample,
- &curr_stats);
- if (net_am_decision(&curr_stats, am)) {
- am->state = NET_AM_APPLY_NEW_PROFILE;
- schedule_work(&am->work);
- break;
- }
- /* fall through */
- case NET_AM_START_MEASURE:
- net_am_sample(event_ctr, packets, bytes, &am->start_sample);
- am->state = NET_AM_MEASURE_IN_PROGRESS;
- break;
- case NET_AM_APPLY_NEW_PROFILE:
- break;
- }
-}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h b/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
deleted file mode 100644
index aa65e79..0000000
--- a/drivers/net/ethernet/mellanox/mlx5/core/net_rx_am.h
+++ /dev/null
@@ -1,107 +0,0 @@
-/*
- * Copyright (c) 2013-2015, Mellanox Technologies, Ltd. All rights reserved.
- * Copyright (c) 2017, Broadcom Limited
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
-*/
-
-#ifndef NET_AM_H
-#define NET_AM_H
-
-struct net_cq_moder {
- u16 usec;
- u16 pkts;
-};
-
-struct net_rx_am_sample {
- ktime_t time;
- u32 pkt_ctr;
- u32 byte_ctr;
- u16 event_ctr;
-};
-
-struct net_rx_am_stats {
- int ppms; /* packets per msec */
- int bpms; /* bytes per msec */
- int epms; /* events per msec */
-};
-
-struct net_rx_am { /* Adaptive Moderation */
- u8 state;
- struct net_rx_am_stats prev_stats;
- struct net_rx_am_sample start_sample;
- struct work_struct work;
- u8 profile_ix;
- u8 mode;
- u8 tune_state;
- u8 steps_right;
- u8 steps_left;
- u8 tired;
-};
-
-enum {
- NET_CQ_PERIOD_MODE_START_FROM_EQE = 0x0,
- NET_CQ_PERIOD_MODE_START_FROM_CQE = 0x1,
- NET_CQ_PERIOD_NUM_MODES
-};
-
-/* Adaptive moderation logic */
-enum {
- NET_AM_START_MEASURE,
- NET_AM_MEASURE_IN_PROGRESS,
- NET_AM_APPLY_NEW_PROFILE,
-};
-
-enum {
- NET_AM_PARKING_ON_TOP,
- NET_AM_PARKING_TIRED,
- NET_AM_GOING_RIGHT,
- NET_AM_GOING_LEFT,
-};
-
-enum {
- NET_AM_STATS_WORSE,
- NET_AM_STATS_SAME,
- NET_AM_STATS_BETTER,
-};
-
-enum {
- NET_AM_STEPPED,
- NET_AM_TOO_TIRED,
- NET_AM_ON_EDGE,
-};
-
-void net_rx_am(struct net_rx_am *am,
- u16 event_ctr,
- u64 packets,
- u64 bytes);
-struct net_cq_moder net_am_get_def_profile(u8 rx_cq_period_mode);
-struct net_cq_moder net_am_get_profile(u8 cq_period_mode, int ix);
-
-#endif /* NET_AM_H */
diff --git a/include/linux/net_rx_am.h b/include/linux/net_rx_am.h
new file mode 100644
index 0000000..115cbde
--- /dev/null
+++ b/include/linux/net_rx_am.h
@@ -0,0 +1,109 @@
+/*
+ * Copyright (c) 2013-2015, Mellanox Technologies, Ltd. All rights reserved.
+ * Copyright (c) 2017, Broadcom Limited
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+*/
+
+#ifndef NET_AM_H
+#define NET_AM_H
+
+#include <linux/module.h>
+
+struct net_cq_moder {
+ u16 usec;
+ u16 pkts;
+};
+
+struct net_rx_am_sample {
+ ktime_t time;
+ u32 pkt_ctr;
+ u32 byte_ctr;
+ u16 event_ctr;
+};
+
+struct net_rx_am_stats {
+ int ppms; /* packets per msec */
+ int bpms; /* bytes per msec */
+ int epms; /* events per msec */
+};
+
+struct net_rx_am { /* Adaptive Moderation */
+ u8 state;
+ struct net_rx_am_stats prev_stats;
+ struct net_rx_am_sample start_sample;
+ struct work_struct work;
+ u8 profile_ix;
+ u8 mode;
+ u8 tune_state;
+ u8 steps_right;
+ u8 steps_left;
+ u8 tired;
+};
+
+enum {
+ NET_CQ_PERIOD_MODE_START_FROM_EQE = 0x0,
+ NET_CQ_PERIOD_MODE_START_FROM_CQE = 0x1,
+ NET_CQ_PERIOD_NUM_MODES
+};
+
+/* Adaptive moderation logic */
+enum {
+ NET_AM_START_MEASURE,
+ NET_AM_MEASURE_IN_PROGRESS,
+ NET_AM_APPLY_NEW_PROFILE,
+};
+
+enum {
+ NET_AM_PARKING_ON_TOP,
+ NET_AM_PARKING_TIRED,
+ NET_AM_GOING_RIGHT,
+ NET_AM_GOING_LEFT,
+};
+
+enum {
+ NET_AM_STATS_WORSE,
+ NET_AM_STATS_SAME,
+ NET_AM_STATS_BETTER,
+};
+
+enum {
+ NET_AM_STEPPED,
+ NET_AM_TOO_TIRED,
+ NET_AM_ON_EDGE,
+};
+
+void net_rx_am(struct net_rx_am *am,
+ u16 event_ctr,
+ u64 packets,
+ u64 bytes);
+struct net_cq_moder net_am_get_def_profile(u8 rx_cq_period_mode);
+struct net_cq_moder net_am_get_profile(u8 cq_period_mode, int ix);
+
+#endif /* NET_AM_H */
diff --git a/lib/Makefile b/lib/Makefile
index b8f2c16..8f6bdc6 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -39,7 +39,7 @@ obj-y += bcd.o div64.o sort.o parser.o debug_locks.o random32.o \
gcd.o lcm.o list_sort.o uuid.o flex_array.o iov_iter.o clz_ctz.o \
bsearch.o find_bit.o llist.o memweight.o kfifo.o \
percpu-refcount.o percpu_ida.o rhashtable.o reciprocal_div.o \
- once.o refcount.o usercopy.o errseq.o
+ once.o refcount.o usercopy.o errseq.o net_rx_am.o
obj-y += string_helpers.o
obj-$(CONFIG_TEST_STRING_HELPERS) += test-string_helpers.o
obj-y += hexdump.o
diff --git a/lib/net_rx_am.c b/lib/net_rx_am.c
new file mode 100644
index 0000000..3167ffa
--- /dev/null
+++ b/lib/net_rx_am.c
@@ -0,0 +1,306 @@
+/*
+ * Copyright (c) 2016, Mellanox Technologies. All rights reserved.
+ * Copyright (c) 2017, Broadcom Limiited. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <linux/net_rx_am.h>
+
+#define NET_PARAMS_AM_NUM_PROFILES 5
+/* Adaptive moderation profiles */
+#define NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE 256
+#define NET_RX_AM_DEF_PROFILE_CQE 1
+#define NET_RX_AM_DEF_PROFILE_EQE 1
+
+/* All profiles sizes must be NET_PARAMS_AM_NUM_PROFILES */
+#define NET_AM_EQE_PROFILES { \
+ {1, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {8, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {64, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {128, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+ {256, NET_AM_DEFAULT_RX_CQ_MODERATION_PKTS_FROM_EQE}, \
+}
+
+#define NET_AM_CQE_PROFILES { \
+ {2, 256}, \
+ {8, 128}, \
+ {16, 64}, \
+ {32, 64}, \
+ {64, 64} \
+}
+
+static const struct net_cq_moder
+profile[NET_CQ_PERIOD_NUM_MODES][NET_PARAMS_AM_NUM_PROFILES] = {
+ NET_AM_EQE_PROFILES,
+ NET_AM_CQE_PROFILES,
+};
+
+struct net_cq_moder net_am_get_profile(u8 cq_period_mode, int ix)
+{
+ return profile[cq_period_mode][ix];
+}
+EXPORT_SYMBOL_GPL(net_am_get_profile);
+
+struct net_cq_moder net_am_get_def_profile(u8 rx_cq_period_mode)
+{
+ int default_profile_ix;
+
+ if (rx_cq_period_mode == NET_CQ_PERIOD_MODE_START_FROM_CQE)
+ default_profile_ix = NET_RX_AM_DEF_PROFILE_CQE;
+ else /* NET_CQ_PERIOD_MODE_START_FROM_EQE */
+ default_profile_ix = NET_RX_AM_DEF_PROFILE_EQE;
+
+ return profile[rx_cq_period_mode][default_profile_ix];
+}
+EXPORT_SYMBOL_GPL(net_am_get_def_profile);
+
+static bool net_am_on_top(struct net_rx_am *am)
+{
+ switch (am->tune_state) {
+ case NET_AM_PARKING_ON_TOP:
+ case NET_AM_PARKING_TIRED:
+ return true;
+ case NET_AM_GOING_RIGHT:
+ return (am->steps_left > 1) && (am->steps_right == 1);
+ default: /* NET_AM_GOING_LEFT */
+ return (am->steps_right > 1) && (am->steps_left == 1);
+ }
+}
+
+static void net_am_turn(struct net_rx_am *am)
+{
+ switch (am->tune_state) {
+ case NET_AM_PARKING_ON_TOP:
+ case NET_AM_PARKING_TIRED:
+ break;
+ case NET_AM_GOING_RIGHT:
+ am->tune_state = NET_AM_GOING_LEFT;
+ am->steps_left = 0;
+ break;
+ case NET_AM_GOING_LEFT:
+ am->tune_state = NET_AM_GOING_RIGHT;
+ am->steps_right = 0;
+ break;
+ }
+}
+
+static int net_am_step(struct net_rx_am *am)
+{
+ if (am->tired == (NET_PARAMS_AM_NUM_PROFILES * 2))
+ return NET_AM_TOO_TIRED;
+
+ switch (am->tune_state) {
+ case NET_AM_PARKING_ON_TOP:
+ case NET_AM_PARKING_TIRED:
+ break;
+ case NET_AM_GOING_RIGHT:
+ if (am->profile_ix == (NET_PARAMS_AM_NUM_PROFILES - 1))
+ return NET_AM_ON_EDGE;
+ am->profile_ix++;
+ am->steps_right++;
+ break;
+ case NET_AM_GOING_LEFT:
+ if (am->profile_ix == 0)
+ return NET_AM_ON_EDGE;
+ am->profile_ix--;
+ am->steps_left++;
+ break;
+ }
+
+ am->tired++;
+ return NET_AM_STEPPED;
+}
+
+static void net_am_park_on_top(struct net_rx_am *am)
+{
+ am->steps_right = 0;
+ am->steps_left = 0;
+ am->tired = 0;
+ am->tune_state = NET_AM_PARKING_ON_TOP;
+}
+
+static void net_am_park_tired(struct net_rx_am *am)
+{
+ am->steps_right = 0;
+ am->steps_left = 0;
+ am->tune_state = NET_AM_PARKING_TIRED;
+}
+
+static void net_am_exit_parking(struct net_rx_am *am)
+{
+ am->tune_state = am->profile_ix ? NET_AM_GOING_LEFT :
+ NET_AM_GOING_RIGHT;
+ net_am_step(am);
+}
+
+#define IS_SIGNIFICANT_DIFF(val, ref) \
+ (((100 * abs((val) - (ref))) / (ref)) > 10) /* more than 10% difference */
+
+static int net_am_stats_compare(struct net_rx_am_stats *curr,
+ struct net_rx_am_stats *prev)
+{
+ if (!prev->bpms)
+ return curr->bpms ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_SAME;
+
+ if (IS_SIGNIFICANT_DIFF(curr->bpms, prev->bpms))
+ return (curr->bpms > prev->bpms) ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_WORSE;
+
+ if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms))
+ return (curr->ppms > prev->ppms) ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_WORSE;
+
+ if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms))
+ return (curr->epms < prev->epms) ? NET_AM_STATS_BETTER :
+ NET_AM_STATS_WORSE;
+
+ return NET_AM_STATS_SAME;
+}
+
+static bool net_am_decision(struct net_rx_am_stats *curr_stats,
+ struct net_rx_am *am)
+{
+ int prev_state = am->tune_state;
+ int prev_ix = am->profile_ix;
+ int stats_res;
+ int step_res;
+
+ switch (am->tune_state) {
+ case NET_AM_PARKING_ON_TOP:
+ stats_res = net_am_stats_compare(curr_stats, &am->prev_stats);
+ if (stats_res != NET_AM_STATS_SAME)
+ net_am_exit_parking(am);
+ break;
+
+ case NET_AM_PARKING_TIRED:
+ am->tired--;
+ if (!am->tired)
+ net_am_exit_parking(am);
+ break;
+
+ case NET_AM_GOING_RIGHT:
+ case NET_AM_GOING_LEFT:
+ stats_res = net_am_stats_compare(curr_stats, &am->prev_stats);
+ if (stats_res != NET_AM_STATS_BETTER)
+ net_am_turn(am);
+
+ if (net_am_on_top(am)) {
+ net_am_park_on_top(am);
+ break;
+ }
+
+ step_res = net_am_step(am);
+ switch (step_res) {
+ case NET_AM_ON_EDGE:
+ net_am_park_on_top(am);
+ break;
+ case NET_AM_TOO_TIRED:
+ net_am_park_tired(am);
+ break;
+ }
+
+ break;
+ }
+
+ if ((prev_state != NET_AM_PARKING_ON_TOP) ||
+ (am->tune_state != NET_AM_PARKING_ON_TOP))
+ am->prev_stats = *curr_stats;
+
+ return am->profile_ix != prev_ix;
+}
+
+static void net_am_sample(u16 event_ctr,
+ u64 packets,
+ u64 bytes,
+ struct net_rx_am_sample *s)
+{
+ s->time = ktime_get();
+ s->pkt_ctr = packets;
+ s->byte_ctr = bytes;
+ s->event_ctr = event_ctr;
+}
+
+#define NET_AM_NEVENTS 64
+#define BITS_PER_TYPE(type) (sizeof(type) * BITS_PER_BYTE)
+#define BIT_GAP(bits, end, start) ((((end) - (start)) + BIT_ULL(bits)) & (BIT_ULL(bits) - 1))
+
+static void net_am_calc_stats(struct net_rx_am_sample *start,
+ struct net_rx_am_sample *end,
+ struct net_rx_am_stats *curr_stats)
+{
+ /* u32 holds up to 71 minutes, should be enough */
+ u32 delta_us = ktime_us_delta(end->time, start->time);
+ u32 npkts = BIT_GAP(BITS_PER_TYPE(u32), end->pkt_ctr, start->pkt_ctr);
+ u32 nbytes = BIT_GAP(BITS_PER_TYPE(u32), end->byte_ctr,
+ start->byte_ctr);
+
+ if (!delta_us)
+ return;
+
+ curr_stats->ppms = DIV_ROUND_UP(npkts * USEC_PER_MSEC, delta_us);
+ curr_stats->bpms = DIV_ROUND_UP(nbytes * USEC_PER_MSEC, delta_us);
+ curr_stats->epms = DIV_ROUND_UP(NET_AM_NEVENTS * USEC_PER_MSEC,
+ delta_us);
+}
+
+void net_rx_am(struct net_rx_am *am,
+ u16 event_ctr,
+ u64 packets,
+ u64 bytes)
+{
+ struct net_rx_am_sample end_sample;
+ struct net_rx_am_stats curr_stats;
+ u16 nevents;
+
+ switch (am->state) {
+ case NET_AM_MEASURE_IN_PROGRESS:
+ nevents = BIT_GAP(BITS_PER_TYPE(u16), event_ctr,
+ am->start_sample.event_ctr);
+ if (nevents < NET_AM_NEVENTS)
+ break;
+ net_am_sample(event_ctr, packets, bytes, &end_sample);
+ net_am_calc_stats(&am->start_sample, &end_sample,
+ &curr_stats);
+ if (net_am_decision(&curr_stats, am)) {
+ am->state = NET_AM_APPLY_NEW_PROFILE;
+ schedule_work(&am->work);
+ break;
+ }
+ /* fall through */
+ case NET_AM_START_MEASURE:
+ net_am_sample(event_ctr, packets, bytes, &am->start_sample);
+ am->state = NET_AM_MEASURE_IN_PROGRESS;
+ break;
+ case NET_AM_APPLY_NEW_PROFILE:
+ break;
+ }
+}
+EXPORT_SYMBOL_GPL(net_rx_am);
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* [RFC 9/9] bnxt_en: add support for software adaptive interrupt moderation
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (10 preceding siblings ...)
2017-11-06 5:44 ` [RFC 8/9] net: move adaptive interrpt " Andy Gospodarek
@ 2017-11-06 5:44 ` Andy Gospodarek
2017-11-06 21:26 ` Michael Chan
2017-11-07 1:05 ` [RFC 0/9] net: create adaptive software irq moderation library Saeed Mahameed
2017-11-24 12:05 ` Saeed Mahameed
13 siblings, 1 reply; 17+ messages in thread
From: Andy Gospodarek @ 2017-11-06 5:44 UTC (permalink / raw)
To: netdev; +Cc: mchan, saeedm, Andy Gospodarek
From: Andy Gospodarek <gospo@broadcom.com>
This implements the changes needed for the bnxt_en driver to add support
for adaptive interrupt moderation per ring.
This does add additional counters in the receive path, but testing shows
that any additional instructions are offset by throughput gain when the
default configuration is for low latency.
Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
---
drivers/net/ethernet/broadcom/bnxt/Makefile | 2 +-
drivers/net/ethernet/broadcom/bnxt/bnxt.c | 51 +++++++++++++++++++++++
drivers/net/ethernet/broadcom/bnxt/bnxt.h | 34 ++++++++++-----
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 7 ++++
drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c | 32 ++++++++++++++
5 files changed, 114 insertions(+), 12 deletions(-)
create mode 100644 drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c
diff --git a/drivers/net/ethernet/broadcom/bnxt/Makefile b/drivers/net/ethernet/broadcom/bnxt/Makefile
index 59c8ec9..1b0c78c 100644
--- a/drivers/net/ethernet/broadcom/bnxt/Makefile
+++ b/drivers/net/ethernet/broadcom/bnxt/Makefile
@@ -1,4 +1,4 @@
obj-$(CONFIG_BNXT) += bnxt_en.o
-bnxt_en-y := bnxt.o bnxt_sriov.o bnxt_ethtool.o bnxt_dcb.o bnxt_ulp.o bnxt_xdp.o bnxt_vfr.o bnxt_devlink.o
+bnxt_en-y := bnxt.o bnxt_sriov.o bnxt_ethtool.o bnxt_dcb.o bnxt_ulp.o bnxt_xdp.o bnxt_vfr.o bnxt_devlink.o bnxt_rx_am.o
bnxt_en-$(CONFIG_BNXT_FLOWER_OFFLOAD) += bnxt_tc.o
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 4e3d569..e1110d9 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -1482,6 +1482,7 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_napi *bnapi, u32 *raw_cons,
u32 tmp_raw_cons = *raw_cons;
u16 cfa_code, cons, prod, cp_cons = RING_CMP(tmp_raw_cons);
struct bnxt_sw_rx_bd *rx_buf;
+ unsigned int pkts = 0;
unsigned int len;
u8 *data_ptr, agg_bufs, cmp_type;
dma_addr_t dma_addr;
@@ -1522,6 +1523,8 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_napi *bnapi, u32 *raw_cons,
rc = -ENOMEM;
if (likely(skb)) {
+ struct skb_shared_info *shinfo = skb_shinfo(skb);
+ pkts = shinfo->nr_frags;
bnxt_deliver_skb(bp, bnapi, skb);
rc = 1;
}
@@ -1645,6 +1648,8 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_napi *bnapi, u32 *raw_cons,
rxr->rx_next_cons = NEXT_RX(cons);
next_rx_no_prod:
+ cpr->rx_packets += pkts ? : 1;
+ cpr->rx_bytes += len;
*raw_cons = tmp_raw_cons;
return rc;
@@ -1798,6 +1803,7 @@ static irqreturn_t bnxt_msix(int irq, void *dev_instance)
struct bnxt_cp_ring_info *cpr = &bnapi->cp_ring;
u32 cons = RING_CMP(cpr->cp_raw_cons);
+ cpr->event_ctr++;
prefetch(&cpr->cp_desc_ring[CP_RING(cons)][CP_IDX(cons)]);
napi_schedule(&bnapi->napi);
return IRQ_HANDLED;
@@ -2021,6 +2027,11 @@ static int bnxt_poll(struct napi_struct *napi, int budget)
break;
}
}
+ if (bp->flags & BNXT_FLAG_RX_AM)
+ net_rx_am(&cpr->am,
+ cpr->event_ctr,
+ cpr->rx_packets,
+ cpr->rx_bytes);
mmiowb();
return work_done;
}
@@ -2606,6 +2617,8 @@ static void bnxt_init_cp_rings(struct bnxt *bp)
struct bnxt_ring_struct *ring = &cpr->cp_ring_struct;
ring->fw_ring_id = INVALID_HW_RING_ID;
+ cpr->rx_ring_coal.coal_ticks = bp->rx_coal.coal_ticks;
+ cpr->rx_ring_coal.coal_bufs = bp->rx_coal.coal_bufs;
}
}
@@ -4579,6 +4592,38 @@ static void bnxt_hwrm_set_coal_params(struct bnxt_coal *hw_coal,
req->flags = cpu_to_le16(flags);
}
+int bnxt_hwrm_set_ring_coal(struct bnxt *bp, struct bnxt_napi *bnapi)
+{
+ struct hwrm_ring_cmpl_ring_cfg_aggint_params_input req_rx = {0};
+ struct bnxt_cp_ring_info *cpr = &bnapi->cp_ring;
+ struct bnxt_coal coal;
+ int rc = 0;
+
+ /* Tick values in micro seconds.
+ * 1 coal_buf x bufs_per_record = 1 completion record.
+ */
+ memcpy(&coal, &bp->rx_coal, sizeof(struct bnxt_coal));
+
+ coal.coal_ticks = cpr->rx_ring_coal.coal_ticks;
+ coal.coal_bufs = cpr->rx_ring_coal.coal_bufs;
+
+ if (!bnapi->rx_ring)
+ return -ENODEV;
+
+ bnxt_hwrm_cmd_hdr_init(bp, &req_rx,
+ HWRM_RING_CMPL_RING_CFG_AGGINT_PARAMS, -1, -1);
+
+ bnxt_hwrm_set_coal_params(&coal, &req_rx);
+
+ mutex_lock(&bp->hwrm_cmd_lock);
+ req_rx.ring_id = cpr->cp_ring_struct.fw_ring_id;
+
+ rc = _hwrm_send_message(bp, &req_rx, sizeof(req_rx),
+ HWRM_CMD_TIMEOUT);
+ mutex_unlock(&bp->hwrm_cmd_lock);
+ return rc;
+}
+
int bnxt_hwrm_set_coal(struct bnxt *bp)
{
int i, rc = 0;
@@ -5706,7 +5751,11 @@ static void bnxt_enable_napi(struct bnxt *bp)
int i;
for (i = 0; i < bp->cp_nr_rings; i++) {
+ struct bnxt_cp_ring_info *cpr = &bp->bnapi[i]->cp_ring;
bp->bnapi[i]->in_reset = false;
+
+ INIT_WORK(&cpr->am.work, bnxt_rx_am_work);
+ cpr->am.mode = NET_CQ_PERIOD_MODE_START_FROM_EQE;
napi_enable(&bp->bnapi[i]->napi);
}
}
@@ -8124,6 +8173,8 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
else
bp->flags |= BNXT_FLAG_DOUBLE_DB;
+ bp->flags = BNXT_FLAG_RX_AM;
+
rc = bnxt_hwrm_func_drv_rgtr(bp);
if (rc)
goto init_err_pci_clean;
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
index 5359a1f..8ab5801d 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
@@ -23,6 +23,7 @@
#include <net/devlink.h>
#include <net/dst_metadata.h>
#include <net/switchdev.h>
+#include <linux/net_rx_am.h>
struct tx_bd {
__le32 tx_bd_len_flags_type;
@@ -607,6 +608,17 @@ struct bnxt_tx_ring_info {
struct bnxt_ring_struct tx_ring_struct;
};
+struct bnxt_coal {
+ u16 coal_ticks;
+ u16 coal_ticks_irq;
+ u16 coal_bufs;
+ u16 coal_bufs_irq;
+ /* RING_IDLE enabled when coal ticks < idle_thresh */
+ u16 idle_thresh;
+ u8 bufs_per_record;
+ u8 budget;
+};
+
struct bnxt_tpa_info {
void *data;
u8 *data_ptr;
@@ -670,6 +682,13 @@ struct bnxt_cp_ring_info {
u32 cp_raw_cons;
void __iomem *cp_doorbell;
+ struct bnxt_coal rx_ring_coal;
+ u64 rx_packets;
+ u64 rx_bytes;
+ u64 event_ctr;
+
+ struct net_rx_am am;
+
struct tx_cmp *cp_desc_ring[MAX_CP_PAGES];
dma_addr_t cp_desc_mapping[MAX_CP_PAGES];
@@ -944,17 +963,6 @@ struct bnxt_test_info {
#define BNXT_CAG_REG_LEGACY_INT_STATUS 0x4014
#define BNXT_CAG_REG_BASE 0x300000
-struct bnxt_coal {
- u16 coal_ticks;
- u16 coal_ticks_irq;
- u16 coal_bufs;
- u16 coal_bufs_irq;
- /* RING_IDLE enabled when coal ticks < idle_thresh */
- u16 idle_thresh;
- u8 bufs_per_record;
- u8 budget;
-};
-
struct bnxt_tc_flow_stats {
u64 packets;
u64 bytes;
@@ -1126,6 +1134,7 @@ struct bnxt {
#define BNXT_FLAG_DOUBLE_DB 0x400000
#define BNXT_FLAG_FW_DCBX_AGENT 0x800000
#define BNXT_FLAG_CHIP_NITRO_A0 0x1000000
+ #define BNXT_FLAG_RX_AM 0x2000000
#define BNXT_FLAG_ALL_CONFIG_FEATS (BNXT_FLAG_TPA | \
BNXT_FLAG_RFS | \
@@ -1423,4 +1432,7 @@ int bnxt_setup_mq_tc(struct net_device *dev, u8 tc);
int bnxt_get_max_rings(struct bnxt *, int *, int *, bool);
void bnxt_restore_pf_fw_resources(struct bnxt *bp);
int bnxt_port_attr_get(struct bnxt *bp, struct switchdev_attr *attr);
+void bnxt_rx_am_work(struct work_struct *work);
+int bnxt_hwrm_set_ring_coal(struct bnxt *bp, struct bnxt_napi *bnapi);
+
#endif
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
index 7ce1d4b..cffc649 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
@@ -49,6 +49,8 @@ static int bnxt_get_coalesce(struct net_device *dev,
memset(coal, 0, sizeof(*coal));
+ coal->use_adaptive_rx_coalesce = bp->flags & BNXT_FLAG_RX_AM;
+
hw_coal = &bp->rx_coal;
mult = hw_coal->bufs_per_record;
coal->rx_coalesce_usecs = hw_coal->coal_ticks;
@@ -77,6 +79,11 @@ static int bnxt_set_coalesce(struct net_device *dev,
int rc = 0;
u16 mult;
+ if (coal->use_adaptive_rx_coalesce)
+ bp->flags |= BNXT_FLAG_RX_AM;
+ else
+ bp->flags &= ~(BNXT_FLAG_RX_AM);
+
hw_coal = &bp->rx_coal;
mult = hw_coal->bufs_per_record;
hw_coal->coal_ticks = coal->rx_coalesce_usecs;
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c
new file mode 100644
index 0000000..8579ca5
--- /dev/null
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c
@@ -0,0 +1,32 @@
+/*
+ * Copyright (c) 2017 Broadcom Limited
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ */
+
+#include <linux/net_rx_am.h>
+#include "bnxt_hsi.h"
+#include "bnxt.h"
+
+void bnxt_rx_am_work(struct work_struct *work)
+{
+ struct net_rx_am *am = container_of(work, struct net_rx_am,
+ work);
+ struct bnxt_cp_ring_info *cpr = container_of(am,
+ struct bnxt_cp_ring_info,
+ am);
+ struct bnxt_napi *bnapi = container_of(cpr,
+ struct bnxt_napi,
+ cp_ring);
+ struct net_cq_moder cur_profile = net_am_get_profile(am->mode,
+ am->profile_ix);
+
+ cpr->rx_ring_coal.coal_ticks = cur_profile.usec;
+ cpr->rx_ring_coal.coal_bufs = cur_profile.pkts;
+
+ bnxt_hwrm_set_ring_coal(bnapi->bp, bnapi);
+ am->state = NET_AM_START_MEASURE;
+}
+
--
2.7.4
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [RFC 9/9] bnxt_en: add support for software adaptive interrupt moderation
2017-11-06 5:44 ` [RFC 9/9] bnxt_en: add support for software adaptive interrupt moderation Andy Gospodarek
@ 2017-11-06 21:26 ` Michael Chan
0 siblings, 0 replies; 17+ messages in thread
From: Michael Chan @ 2017-11-06 21:26 UTC (permalink / raw)
To: Andy Gospodarek
Cc: Netdev, michael.chan@broadcom.com, Saeed Mahameed, Andy Gospodarek
On Sun, Nov 5, 2017 at 9:44 PM, Andy Gospodarek <andy@greyhouse.net> wrote:
> From: Andy Gospodarek <gospo@broadcom.com>
>
> This implements the changes needed for the bnxt_en driver to add support
> for adaptive interrupt moderation per ring.
>
> This does add additional counters in the receive path, but testing shows
> that any additional instructions are offset by throughput gain when the
> default configuration is for low latency.
>
> Signed-off-by: Andy Gospodarek <gospo@broadcom.com>
> ---
> drivers/net/ethernet/broadcom/bnxt/Makefile | 2 +-
> drivers/net/ethernet/broadcom/bnxt/bnxt.c | 51 +++++++++++++++++++++++
> drivers/net/ethernet/broadcom/bnxt/bnxt.h | 34 ++++++++++-----
> drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 7 ++++
> drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c | 32 ++++++++++++++
> 5 files changed, 114 insertions(+), 12 deletions(-)
> create mode 100644 drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c
>
> diff --git a/drivers/net/ethernet/broadcom/bnxt/Makefile b/drivers/net/ethernet/broadcom/bnxt/Makefile
> index 59c8ec9..1b0c78c 100644
> --- a/drivers/net/ethernet/broadcom/bnxt/Makefile
> +++ b/drivers/net/ethernet/broadcom/bnxt/Makefile
> @@ -1,4 +1,4 @@
> obj-$(CONFIG_BNXT) += bnxt_en.o
>
> -bnxt_en-y := bnxt.o bnxt_sriov.o bnxt_ethtool.o bnxt_dcb.o bnxt_ulp.o bnxt_xdp.o bnxt_vfr.o bnxt_devlink.o
> +bnxt_en-y := bnxt.o bnxt_sriov.o bnxt_ethtool.o bnxt_dcb.o bnxt_ulp.o bnxt_xdp.o bnxt_vfr.o bnxt_devlink.o bnxt_rx_am.o
> bnxt_en-$(CONFIG_BNXT_FLOWER_OFFLOAD) += bnxt_tc.o
> diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
> index 4e3d569..e1110d9 100644
> --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
> +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
> @@ -1482,6 +1482,7 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_napi *bnapi, u32 *raw_cons,
> u32 tmp_raw_cons = *raw_cons;
> u16 cfa_code, cons, prod, cp_cons = RING_CMP(tmp_raw_cons);
> struct bnxt_sw_rx_bd *rx_buf;
> + unsigned int pkts = 0;
> unsigned int len;
> u8 *data_ptr, agg_bufs, cmp_type;
> dma_addr_t dma_addr;
> @@ -1522,6 +1523,8 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_napi *bnapi, u32 *raw_cons,
>
> rc = -ENOMEM;
> if (likely(skb)) {
> + struct skb_shared_info *shinfo = skb_shinfo(skb);
> + pkts = shinfo->nr_frags;
> bnxt_deliver_skb(bp, bnapi, skb);
> rc = 1;
> }
> @@ -1645,6 +1648,8 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_napi *bnapi, u32 *raw_cons,
> rxr->rx_next_cons = NEXT_RX(cons);
>
> next_rx_no_prod:
> + cpr->rx_packets += pkts ? : 1;
> + cpr->rx_bytes += len;
These counters are in the RX fast path, including the XDP fast path.
If we are handling tens of millions of packets per second, every
packet has to incur the cost of these counters.
It will be much nicer if we can use the hardware counters instead.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [RFC 0/9] net: create adaptive software irq moderation library
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (11 preceding siblings ...)
2017-11-06 5:44 ` [RFC 9/9] bnxt_en: add support for software adaptive interrupt moderation Andy Gospodarek
@ 2017-11-07 1:05 ` Saeed Mahameed
2017-11-24 12:05 ` Saeed Mahameed
13 siblings, 0 replies; 17+ messages in thread
From: Saeed Mahameed @ 2017-11-07 1:05 UTC (permalink / raw)
To: Andy Gospodarek, Achiad Shochat, talg
Cc: Linux Netdev List, Michael Chan, Saeed Mahameed, Andy Gospodarek
On Sun, Nov 5, 2017 at 9:44 PM, Andy Gospodarek <andy@greyhouse.net> wrote:
> From: Andy Gospodarek <gospo@broadcom.com>
>
> This RFC converts the adaptive interrupt moderation library from the
> mlx5_en driver into a library so it can be used by any driver. The last
> patch in this set adds support for interrupt moderation in the bnxt_en
> driver.
>
> The main purpose of this code in the mlx5_en driver is to allow an
> administrator to make sure that default coalesce settings are optimized
> for low latency, but quickly adapt to handle high throughput traffic and
> optimize how many packets are received during each napi poll.
>
> For any new driver the following changes would be needed to use this
> library:
>
> - add elements in ring struct to track items needed by this library
> - create function that can be called to actually set coalesce settings
> for the driver
>
> My main reason for making this an RFC is that I would like verification
> from Mellanox that the performance of their driver does not change in a
> unintended way. I did some basic testing (netperf) and did not note a
> statistically significant change in throughput or CPU utilization before
> and after this set.
>
Hi Andy, Are you at netdev conf this week ?
Achiad and Tal (the inventors of mlx5 adaptive moderation) and I, will
be in Seoul this week.
So if you are joining the conference then it will be a good
opportunity to sit and talk F2F.
> Credit to Rob Rice and Lee Reed for doing some of the initial proof of
> concept and testing for this patch.
>
> Andy Gospodarek (9):
> mlx5_en: move interrupt moderation structs to new file
> mlx5_en: move interrupt moderation forward delcarations
> mlx5_en: remove rq references in mlx5e_rx_am
> mlx5_en: move AM logic enums
> mlx5_en: move generic functions to new file
> mlx5_en: rename en_rx_am.h to net_rx_am.h
> mlx5_en: remove Mellanox references in AM code
> net: move adaptive interrpt coalescing code to lib/
> bnxt_en: add support for software adaptive interrupt moderation
>
> drivers/net/ethernet/broadcom/bnxt/Makefile | 2 +-
> drivers/net/ethernet/broadcom/bnxt/bnxt.c | 51 ++++
> drivers/net/ethernet/broadcom/bnxt/bnxt.h | 34 ++-
> drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 7 +
> drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c | 32 +++
> drivers/net/ethernet/mellanox/mlx5/core/en.h | 43 +--
> .../net/ethernet/mellanox/mlx5/core/en_ethtool.c | 6 +-
> drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 18 +-
> drivers/net/ethernet/mellanox/mlx5/core/en_rep.c | 4 +-
> drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c | 298 +-------------------
> drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 5 +-
> include/linux/mlx5/mlx5_ifc.h | 6 -
> include/linux/net_rx_am.h | 109 ++++++++
> lib/Makefile | 2 +-
> lib/net_rx_am.c | 306 +++++++++++++++++++++
> 15 files changed, 558 insertions(+), 365 deletions(-)
> create mode 100644 drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c
> create mode 100644 include/linux/net_rx_am.h
> create mode 100644 lib/net_rx_am.c
>
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [RFC 0/9] net: create adaptive software irq moderation library
2017-11-06 5:44 [RFC 0/9] net: create adaptive software irq moderation library Andy Gospodarek
` (12 preceding siblings ...)
2017-11-07 1:05 ` [RFC 0/9] net: create adaptive software irq moderation library Saeed Mahameed
@ 2017-11-24 12:05 ` Saeed Mahameed
2017-11-24 12:20 ` Saeed Mahameed
13 siblings, 1 reply; 17+ messages in thread
From: Saeed Mahameed @ 2017-11-24 12:05 UTC (permalink / raw)
To: Andy Gospodarek, Tariq Toukan, talg
Cc: Linux Netdev List, Michael Chan, Saeed Mahameed, Andy Gospodarek
On Sun, Nov 5, 2017 at 9:44 PM, Andy Gospodarek <andy@greyhouse.net> wrote:
> From: Andy Gospodarek <gospo@broadcom.com>
>
> This RFC converts the adaptive interrupt moderation library from the
> mlx5_en driver into a library so it can be used by any driver. The last
> patch in this set adds support for interrupt moderation in the bnxt_en
> driver.
>
> The main purpose of this code in the mlx5_en driver is to allow an
> administrator to make sure that default coalesce settings are optimized
> for low latency, but quickly adapt to handle high throughput traffic and
> optimize how many packets are received during each napi poll.
>
> For any new driver the following changes would be needed to use this
> library:
>
> - add elements in ring struct to track items needed by this library
> - create function that can be called to actually set coalesce settings
> for the driver
>
> My main reason for making this an RFC is that I would like verification
> from Mellanox that the performance of their driver does not change in a
> unintended way. I did some basic testing (netperf) and did not note a
> statistically significant change in throughput or CPU utilization before
> and after this set.
>
> Credit to Rob Rice and Lee Reed for doing some of the initial proof of
> concept and testing for this patch.
Hi Andy,
Following our conversation in netdev 2.2, i would like to suggest the
following:
Instead of introducing a new API which demands from the driver to
provide callbacks and function pointers to the adaptive moderation
logic, which might be called on every irq interrupt, and to avoid
performance hit, we can move the generic code and the core adaptive
moderation logic to a header file.
the mlx5e am logic and data structures are already written in a very
modular way and can be stripped out of mlx5e fairly easily.
And i would like to suggest to do it in the following manner:
1. naming convention:
I would like to change the generic code naming convention to have the
words DIM (Dynamically-Tuned Interrupt Moderation) instead of mlx5e_am
or am, Following our public blog [1] of the matter and the official
name we prefer for this feature.
[1] https://community.mellanox.com/docs/DOC-2511
Suggested naming convention instead of rx_am: net_dim (DIM for net
applications).
As the rx_am or (dim) logic can be applied to other applications.
2. Data types:
All below mlx5e am data types can be used as is as they hold nothing
mlx5 related.
struct mlx5e_rx_am_sample
- Holds the current stats sample with ktime stamp
- rename to: net_dim_sample
struct mlx5e_rx_am_stats
- Holds the needed stats (delta) calculation of last 2 samples
- rename to: net_dim_stats
struct mlx5e_rx_am
- Adaptive moderation handle
- rename to: net_dim
3. static inline generic functions API (based on the usage from
mlx5e_rx_am function)
//Make a DIM measurement:
net_dim_sample(struct *net_dim_sample sample, packets, bytes, event_ctr)
- previously mlx5e_am_sample()
- Fills a sample struct with the provided stats and the current timestamp
//start a new DIM measurement and handles the DIM state machine initial state:
net_dim_start_sample(struct *net_dim rx_dim)
- Makes a new measurement
- stores it into rx_dim->start
- rx_dim->state = DIM_MEASURE_IN_PROGRESS
// Takes a new sample (curr_sample) and makes the decision (handles
DIM_MEASURE_IN_PROGRESS state)
net_dim_decision(struct *net_dim rx_dim, curr_sample)
- previously mlx5e_am_decision
- Note, instead of providing the current_stats (delta between start
and current_sample) I suggest to provide the current_sample and move
the stats calculation logic into net_dime_decision.
- All the logic in this function will move to the generic code.
4. Driver implementation: (according to the above suggested API)
- Driver should initialize struct net_dim rx_dim, and provide a
work function to handle "dim apply new profile" decision.
- in napi_poll driver should implement the rx_dim state machine
using the above API before arming the completion event queues as
follows:
mlx5e_rx_am:
void mlx5e_rx_am(struct mlx5e_rq *rq)
{
struct net_dim *rx_dim = &rq->dim;
struct net_dim_sample end_sample;
u16 nevents;
switch (rx_dim->state) {
case DIM_MEASURE_IN_PROGRESS:
// driver specific pre condition to decide whether to
continue or skip
// Note that here we only sample and don't calc the delta
stats, this logic moved into net_dim_decision
net_dim_sample(rq, &end_sample, rq->packets, rq->bytes, cq->events);
if (net_dim_decision(rx_dim, &end_sample)) {
rx_dim->state = DIM_APPLY_NEW_PROFILE;
schedule_work(&rx_dim->work);
}
/* fall through */
case DIM_START_MEASURE:
net_dim_start_sample(rx_dim);
break;
case DIM_APPLY_NEW_PROFILE:
break;
}
Thanks,
Saeed.
>
> Andy Gospodarek (9):
> mlx5_en: move interrupt moderation structs to new file
> mlx5_en: move interrupt moderation forward delcarations
> mlx5_en: remove rq references in mlx5e_rx_am
> mlx5_en: move AM logic enums
> mlx5_en: move generic functions to new file
> mlx5_en: rename en_rx_am.h to net_rx_am.h
> mlx5_en: remove Mellanox references in AM code
> net: move adaptive interrpt coalescing code to lib/
> bnxt_en: add support for software adaptive interrupt moderation
>
> drivers/net/ethernet/broadcom/bnxt/Makefile | 2 +-
> drivers/net/ethernet/broadcom/bnxt/bnxt.c | 51 ++++
> drivers/net/ethernet/broadcom/bnxt/bnxt.h | 34 ++-
> drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 7 +
> drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c | 32 +++
> drivers/net/ethernet/mellanox/mlx5/core/en.h | 43 +--
> .../net/ethernet/mellanox/mlx5/core/en_ethtool.c | 6 +-
> drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 18 +-
> drivers/net/ethernet/mellanox/mlx5/core/en_rep.c | 4 +-
> drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c | 298 +-------------------
> drivers/net/ethernet/mellanox/mlx5/core/en_txrx.c | 5 +-
> include/linux/mlx5/mlx5_ifc.h | 6 -
> include/linux/net_rx_am.h | 109 ++++++++
> lib/Makefile | 2 +-
> lib/net_rx_am.c | 306 +++++++++++++++++++++
> 15 files changed, 558 insertions(+), 365 deletions(-)
> create mode 100644 drivers/net/ethernet/broadcom/bnxt/bnxt_rx_am.c
> create mode 100644 include/linux/net_rx_am.h
> create mode 100644 lib/net_rx_am.c
>
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [RFC 0/9] net: create adaptive software irq moderation library
2017-11-24 12:05 ` Saeed Mahameed
@ 2017-11-24 12:20 ` Saeed Mahameed
0 siblings, 0 replies; 17+ messages in thread
From: Saeed Mahameed @ 2017-11-24 12:20 UTC (permalink / raw)
To: Andy Gospodarek, Tariq Toukan, talgi
Cc: Linux Netdev List, Michael Chan, Saeed Mahameed, Andy Gospodarek
On Fri, Nov 24, 2017 at 4:05 AM, Saeed Mahameed
<saeedm@dev.mellanox.co.il> wrote:
> On Sun, Nov 5, 2017 at 9:44 PM, Andy Gospodarek <andy@greyhouse.net> wrote:
>> From: Andy Gospodarek <gospo@broadcom.com>
>>
>> This RFC converts the adaptive interrupt moderation library from the
>> mlx5_en driver into a library so it can be used by any driver. The last
>> patch in this set adds support for interrupt moderation in the bnxt_en
>> driver.
>>
>> The main purpose of this code in the mlx5_en driver is to allow an
>> administrator to make sure that default coalesce settings are optimized
>> for low latency, but quickly adapt to handle high throughput traffic and
>> optimize how many packets are received during each napi poll.
>>
>> For any new driver the following changes would be needed to use this
>> library:
>>
>> - add elements in ring struct to track items needed by this library
>> - create function that can be called to actually set coalesce settings
>> for the driver
>>
>> My main reason for making this an RFC is that I would like verification
>> from Mellanox that the performance of their driver does not change in a
>> unintended way. I did some basic testing (netperf) and did not note a
>> statistically significant change in throughput or CPU utilization before
>> and after this set.
>>
>> Credit to Rob Rice and Lee Reed for doing some of the initial proof of
>> concept and testing for this patch.
>
> Hi Andy,
>
> Following our conversation in netdev 2.2, i would like to suggest the
> following:
>
> Instead of introducing a new API which demands from the driver to
> provide callbacks and function pointers to the adaptive moderation
> logic, which might be called on every irq interrupt, and to avoid
> performance hit, we can move the generic code and the core adaptive
> moderation logic to a header file.
>
I would like also to suggesting adding Tal Gilboa, as the official
maintainer for this new file.
as he is the current maintainer and the co-author of this feature in mlx5.
> the mlx5e am logic and data structures are already written in a very
> modular way and can be stripped out of mlx5e fairly easily.
> And i would like to suggest to do it in the following manner:
>
> 1. naming convention:
> I would like to change the generic code naming convention to have the
> words DIM (Dynamically-Tuned Interrupt Moderation) instead of mlx5e_am
> or am, Following our public blog [1] of the matter and the official
> name we prefer for this feature.
>
> [1] https://community.mellanox.com/docs/DOC-2511
>
> Suggested naming convention instead of rx_am: net_dim (DIM for net
> applications).
> As the rx_am or (dim) logic can be applied to other applications.
>
> 2. Data types:
>
> All below mlx5e am data types can be used as is as they hold nothing
> mlx5 related.
>
> struct mlx5e_rx_am_sample
> - Holds the current stats sample with ktime stamp
> - rename to: net_dim_sample
>
> struct mlx5e_rx_am_stats
> - Holds the needed stats (delta) calculation of last 2 samples
> - rename to: net_dim_stats
>
> struct mlx5e_rx_am
> - Adaptive moderation handle
> - rename to: net_dim
>
> 3. static inline generic functions API (based on the usage from
> mlx5e_rx_am function)
>
> //Make a DIM measurement:
> net_dim_sample(struct *net_dim_sample sample, packets, bytes, event_ctr)
> - previously mlx5e_am_sample()
> - Fills a sample struct with the provided stats and the current timestamp
>
>
> //start a new DIM measurement and handles the DIM state machine initial state:
> net_dim_start_sample(struct *net_dim rx_dim)
> - Makes a new measurement
> - stores it into rx_dim->start
> - rx_dim->state = DIM_MEASURE_IN_PROGRESS
>
>
> // Takes a new sample (curr_sample) and makes the decision (handles
> DIM_MEASURE_IN_PROGRESS state)
> net_dim_decision(struct *net_dim rx_dim, curr_sample)
> - previously mlx5e_am_decision
> - Note, instead of providing the current_stats (delta between start
> and current_sample) I suggest to provide the current_sample and move
> the stats calculation logic into net_dime_decision.
> - All the logic in this function will move to the generic code.
>
> 4. Driver implementation: (according to the above suggested API)
> - Driver should initialize struct net_dim rx_dim, and provide a
> work function to handle "dim apply new profile" decision.
> - in napi_poll driver should implement the rx_dim state machine
> using the above API before arming the completion event queues as
> follows:
>
> mlx5e_rx_am:
>
> void mlx5e_rx_am(struct mlx5e_rq *rq)
> {
> struct net_dim *rx_dim = &rq->dim;
> struct net_dim_sample end_sample;
> u16 nevents;
>
> switch (rx_dim->state) {
> case DIM_MEASURE_IN_PROGRESS:
> // driver specific pre condition to decide whether to
> continue or skip
> // Note that here we only sample and don't calc the delta
> stats, this logic moved into net_dim_decision
> net_dim_sample(rq, &end_sample, rq->packets, rq->bytes, cq->events);
> if (net_dim_decision(rx_dim, &end_sample)) {
> rx_dim->state = DIM_APPLY_NEW_PROFILE;
> schedule_work(&rx_dim->work);
> }
> /* fall through */
> case DIM_START_MEASURE:
> net_dim_start_sample(rx_dim);
> break;
> case DIM_APPLY_NEW_PROFILE:
> break;
> }
>
> Thanks,
> Saeed.
>
^ permalink raw reply [flat|nested] 17+ messages in thread