[PATCH net-next v2 0/6] ethtool: add standard FEC statistics

[PATCH net-next v2 0/6] ethtool: add standard FEC statistics

[Jakub Kicinski]

This set adds uAPI for reporting standard FEC statistics, and
implements it in a handful of drivers.

The statistics are taken from the IEEE standard, with one
extra seemingly popular but not standard statistics added.

The implementation is similar to that of the pause frame
statistics, user requests the stats by setting a bit
(ETHTOOL_FLAG_STATS) in the common ethtool header of
ETHTOOL_MSG_FEC_GET.

Since standard defines the statistics per lane what's
reported is both total and per-lane counters:

 # ethtool -I --show-fec eth0
 FEC parameters for eth0:
 Configured FEC encodings: None
 Active FEC encoding: None
 Statistics:
  corrected_blocks: 256
    Lane 0: 255
    Lane 1: 1
  uncorrectable_blocks: 145
    Lane 0: 128
    Lane 1: 17

v2: check for errors in mlx5 register access

Jakub Kicinski (6):
  ethtool: move ethtool_stats_init
  ethtool: fec_prepare_data() - jump to error handling
  ethtool: add FEC statistics
  bnxt: implement ethtool::get_fec_stats
  sfc: ef10: implement ethtool::get_fec_stats
  mlx5: implement ethtool::get_fec_stats

 Documentation/networking/ethtool-netlink.rst  | 21 +++++
 Documentation/networking/statistics.rst       |  2 +
 .../net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 15 ++++
 .../ethernet/mellanox/mlx5/core/en_ethtool.c  |  9 +++
 .../ethernet/mellanox/mlx5/core/en_stats.c    | 29 ++++++-
 .../ethernet/mellanox/mlx5/core/en_stats.h    |  2 +
 drivers/net/ethernet/sfc/ef10.c               | 17 ++++
 drivers/net/ethernet/sfc/ethtool.c            | 10 +++
 drivers/net/ethernet/sfc/net_driver.h         |  3 +
 include/linux/ethtool.h                       | 46 +++++++++++
 include/uapi/linux/ethtool_netlink.h          | 14 ++++
 net/ethtool/fec.c                             | 80 ++++++++++++++++++-
 net/ethtool/pause.c                           |  6 --
 13 files changed, 242 insertions(+), 12 deletions(-)

-- 
2.30.2

[PATCH net-next v2 1/6] ethtool: move ethtool_stats_init

[Jakub Kicinski]

We'll need it for FEC stats as well.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
 include/linux/ethtool.h | 6 ++++++
 net/ethtool/pause.c     | 6 ------
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/include/linux/ethtool.h b/include/linux/ethtool.h
index 9f6f323af59a..069100b252bd 100644
--- a/include/linux/ethtool.h
+++ b/include/linux/ethtool.h
@@ -244,6 +244,12 @@ bool ethtool_convert_link_mode_to_legacy_u32(u32 *legacy_u32,
 
 #define ETHTOOL_STAT_NOT_SET	(~0ULL)
 
+static inline void ethtool_stats_init(u64 *stats, unsigned int n)
+{
+	while (n--)
+		stats[n] = ETHTOOL_STAT_NOT_SET;
+}
+
 /**
  * struct ethtool_pause_stats - statistics for IEEE 802.3x pause frames
  * @tx_pause_frames: transmitted pause frame count. Reported to user space
diff --git a/net/ethtool/pause.c b/net/ethtool/pause.c
index d4ac02718b72..9009f412151e 100644
--- a/net/ethtool/pause.c
+++ b/net/ethtool/pause.c
@@ -21,12 +21,6 @@ const struct nla_policy ethnl_pause_get_policy[] = {
 		NLA_POLICY_NESTED(ethnl_header_policy_stats),
 };
 
-static void ethtool_stats_init(u64 *stats, unsigned int n)
-{
-	while (n--)
-		stats[n] = ETHTOOL_STAT_NOT_SET;
-}
-
 static int pause_prepare_data(const struct ethnl_req_info *req_base,
 			      struct ethnl_reply_data *reply_base,
 			      struct genl_info *info)
-- 
2.30.2

[PATCH net-next v2 2/6] ethtool: fec_prepare_data() - jump to error handling

[Jakub Kicinski]

Refactor fec_prepare_data() a little bit to skip the body
of the function and exit on error. Currently the code
depends on the fact that we only have one call which
may fail between ethnl_ops_begin() and ethnl_ops_complete()
and simply saves the error code. This will get hairy with
the stats also being queried.

No functional changes.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
 net/ethtool/fec.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/net/ethtool/fec.c b/net/ethtool/fec.c
index 31454b9188bd..3e7d091ee7aa 100644
--- a/net/ethtool/fec.c
+++ b/net/ethtool/fec.c
@@ -80,9 +80,8 @@ static int fec_prepare_data(const struct ethnl_req_info *req_base,
 	if (ret < 0)
 		return ret;
 	ret = dev->ethtool_ops->get_fecparam(dev, &fec);
-	ethnl_ops_complete(dev);
 	if (ret)
-		return ret;
+		goto out_complete;
 
 	WARN_ON_ONCE(fec.reserved);
 
@@ -98,7 +97,9 @@ static int fec_prepare_data(const struct ethnl_req_info *req_base,
 	if (data->active_fec == __ETHTOOL_LINK_MODE_MASK_NBITS)
 		data->active_fec = 0;
 
-	return 0;
+out_complete:
+	ethnl_ops_complete(dev);
+	return ret;
 }
 
 static int fec_reply_size(const struct ethnl_req_info *req_base,
-- 
2.30.2

[PATCH net-next v2 3/6] ethtool: add FEC statistics

[Jakub Kicinski]

Similarly to pause statistics add stats for FEC.

The IEEE standard mandates two sets of counters:
 - 30.5.1.1.17 aFECCorrectedBlocks
 - 30.5.1.1.18 aFECUncorrectableBlocks
where block is a block of bits FEC operates on.
Each of these counters is defined per lane (PCS instance).

Multiple vendors provide number of corrected _bits_ rather
than/as well as blocks.

This set adds the 2 standard-based block counters and a extra
one for corrected bits.

Counters are exposed to user space via netlink in new attributes.
Each attribute carries an array of u64s, first element is
the total count, and the following ones are a per-lane break down.

Much like with pause stats the operation will not fail when driver
does not implement the get_fec_stats callback (nor can the driver
fail the operation by returning an error). If stats can't be
reported the relevant attributes will be empty.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
 Documentation/networking/ethtool-netlink.rst | 21 ++++++
 Documentation/networking/statistics.rst      |  2 +
 include/linux/ethtool.h                      | 40 +++++++++++
 include/uapi/linux/ethtool_netlink.h         | 14 ++++
 net/ethtool/fec.c                            | 73 +++++++++++++++++++-
 5 files changed, 149 insertions(+), 1 deletion(-)

diff --git a/Documentation/networking/ethtool-netlink.rst b/Documentation/networking/ethtool-netlink.rst
index bbecffc7b11a..f8219e2f489e 100644
--- a/Documentation/networking/ethtool-netlink.rst
+++ b/Documentation/networking/ethtool-netlink.rst
@@ -1302,6 +1302,7 @@ Gets FEC configuration and state like ``ETHTOOL_GFECPARAM`` ioctl request.
   ``ETHTOOL_A_FEC_MODES``                bitset  configured modes
   ``ETHTOOL_A_FEC_AUTO``                 bool    FEC mode auto selection
   ``ETHTOOL_A_FEC_ACTIVE``               u32     index of active FEC mode
+  ``ETHTOOL_A_FEC_STATS``                nested  FEC statistics
   =====================================  ======  ==========================
 
 ``ETHTOOL_A_FEC_ACTIVE`` is the bit index of the FEC link mode currently
@@ -1315,6 +1316,26 @@ This is equivalent to the ``ETHTOOL_FEC_AUTO`` bit of the ioctl interface.
 ``ETHTOOL_A_FEC_MODES`` carry the current FEC configuration using link mode
 bits (rather than old ``ETHTOOL_FEC_*`` bits).
 
+``ETHTOOL_A_FEC_STATS`` are reported if ``ETHTOOL_FLAG_STATS`` was set in
+``ETHTOOL_A_HEADER_FLAGS``.
+Each attribute carries an array of 64bit statistics. First entry in the array
+contains the total number of events on the port, while the following entries
+are counters corresponding to lanes/PCS instances. The number of entries in
+the array will be:
+
++--------------+---------------------------------------------+
+| `0`          | device does not support FEC statistics      |
++--------------+---------------------------------------------+
+| `1`          | device does not support per-lane break down |
++--------------+---------------------------------------------+
+| `1 + #lanes` | device has full support for FEC stats       |
++--------------+---------------------------------------------+
+
+Drivers fill in the statistics in the following structure:
+
+.. kernel-doc:: include/linux/ethtool.h
+    :identifiers: ethtool_fec_stats
+
 FEC_SET
 =======
 
diff --git a/Documentation/networking/statistics.rst b/Documentation/networking/statistics.rst
index 234abedc29b2..b748fe44ee02 100644
--- a/Documentation/networking/statistics.rst
+++ b/Documentation/networking/statistics.rst
@@ -130,6 +130,7 @@ the `ETHTOOL_FLAG_STATS` flag in `ETHTOOL_A_HEADER_FLAGS`. Currently
 statistics are supported in the following commands:
 
   - `ETHTOOL_MSG_PAUSE_GET`
+  - `ETHTOOL_MSG_FEC_GET`
 
 debugfs
 -------
@@ -176,3 +177,4 @@ translated to netlink attributes when dumped. Drivers must not overwrite
 the statistics they don't report with 0.
 
 - ethtool_pause_stats()
+- ethtool_fec_stats()
diff --git a/include/linux/ethtool.h b/include/linux/ethtool.h
index 069100b252bd..112a85b57f1f 100644
--- a/include/linux/ethtool.h
+++ b/include/linux/ethtool.h
@@ -269,6 +269,39 @@ struct ethtool_pause_stats {
 	u64 rx_pause_frames;
 };
 
+#define ETHTOOL_MAX_LANES	8
+
+/**
+ * struct ethtool_fec_stats - statistics for IEEE 802.3 FEC
+ * @corrected_blocks: number of received blocks corrected by FEC
+ *	Reported to user space as %ETHTOOL_A_FEC_STAT_CORRECTED.
+ *
+ *	Equivalent to `30.5.1.1.17 aFECCorrectedBlocks` from the standard.
+ *
+ * @uncorrectable_blocks: number of received blocks FEC was not able to correct
+ *	Reported to user space as %ETHTOOL_A_FEC_STAT_UNCORR.
+ *
+ *	Equivalent to `30.5.1.1.18 aFECUncorrectableBlocks` from the standard.
+ *
+ * @corrected_bits: number of bits corrected by FEC
+ *	Similar to @corrected_blocks but counts individual bit changes,
+ *	not entire FEC data blocks. This is a non-standard statistic.
+ *	Reported to user space as %ETHTOOL_A_FEC_STAT_CORR_BITS.
+ *
+ * @lane: per-lane/PCS-instance counts as defined by the standard
+ * @total: error counts for the entire port, for drivers incapable of reporting
+ *	per-lane stats
+ *
+ * Drivers should fill in either only total or per-lane statistics, core
+ * will take care of adding lane values up to produce the total.
+ */
+struct ethtool_fec_stats {
+	struct ethtool_fec_stat {
+		u64 total;
+		u64 lanes[ETHTOOL_MAX_LANES];
+	} corrected_blocks, uncorrectable_blocks, corrected_bits;
+};
+
 #define ETH_MODULE_EEPROM_PAGE_LEN	128
 #define ETH_MODULE_MAX_I2C_ADDRESS	0x7f
 
@@ -439,6 +472,11 @@ struct ethtool_module_eeprom {
  *	ignored (use %__ETHTOOL_LINK_MODE_MASK_NBITS instead of the latter),
  *	any change to them will be overwritten by kernel. Returns a negative
  *	error code or zero.
+ * @get_fec_stats: Report FEC statistics.
+ *	Core will sum up per-lane stats to get the total.
+ *	Drivers must not zero statistics which they don't report. The stats
+ *	structure is initialized to ETHTOOL_STAT_NOT_SET indicating driver does
+ *	not report statistics.
  * @get_fecparam: Get the network device Forward Error Correction parameters.
  * @set_fecparam: Set the network device Forward Error Correction parameters.
  * @get_ethtool_phy_stats: Return extended statistics about the PHY device.
@@ -544,6 +582,8 @@ struct ethtool_ops {
 				      struct ethtool_link_ksettings *);
 	int	(*set_link_ksettings)(struct net_device *,
 				      const struct ethtool_link_ksettings *);
+	void	(*get_fec_stats)(struct net_device *dev,
+				 struct ethtool_fec_stats *fec_stats);
 	int	(*get_fecparam)(struct net_device *,
 				      struct ethtool_fecparam *);
 	int	(*set_fecparam)(struct net_device *,
diff --git a/include/uapi/linux/ethtool_netlink.h b/include/uapi/linux/ethtool_netlink.h
index 9612dcd48a6a..3a2b31ccbc5b 100644
--- a/include/uapi/linux/ethtool_netlink.h
+++ b/include/uapi/linux/ethtool_netlink.h
@@ -643,11 +643,25 @@ enum {
 	ETHTOOL_A_FEC_MODES,				/* bitset */
 	ETHTOOL_A_FEC_AUTO,				/* u8 */
 	ETHTOOL_A_FEC_ACTIVE,				/* u32 */
+	ETHTOOL_A_FEC_STATS,				/* nest - _A_FEC_STAT */
 
 	__ETHTOOL_A_FEC_CNT,
 	ETHTOOL_A_FEC_MAX = (__ETHTOOL_A_FEC_CNT - 1)
 };
 
+enum {
+	ETHTOOL_A_FEC_STAT_UNSPEC,
+	ETHTOOL_A_FEC_STAT_PAD,
+
+	ETHTOOL_A_FEC_STAT_CORRECTED,			/* array, u64 */
+	ETHTOOL_A_FEC_STAT_UNCORR,			/* array, u64 */
+	ETHTOOL_A_FEC_STAT_CORR_BITS,			/* array, u64 */
+
+	/* add new constants above here */
+	__ETHTOOL_A_FEC_STAT_CNT,
+	ETHTOOL_A_FEC_STAT_MAX = (__ETHTOOL_A_FEC_STAT_CNT - 1)
+};
+
 /* MODULE EEPROM */
 
 enum {
diff --git a/net/ethtool/fec.c b/net/ethtool/fec.c
index 3e7d091ee7aa..8738dafd5417 100644
--- a/net/ethtool/fec.c
+++ b/net/ethtool/fec.c
@@ -13,6 +13,10 @@ struct fec_reply_data {
 	__ETHTOOL_DECLARE_LINK_MODE_MASK(fec_link_modes);
 	u32 active_fec;
 	u8 fec_auto;
+	struct fec_stat_grp {
+		u64 stats[1 + ETHTOOL_MAX_LANES];
+		u8 cnt;
+	} corr, uncorr, corr_bits;
 };
 
 #define FEC_REPDATA(__reply_base) \
@@ -21,7 +25,7 @@ struct fec_reply_data {
 #define ETHTOOL_FEC_MASK	((ETHTOOL_FEC_LLRS << 1) - 1)
 
 const struct nla_policy ethnl_fec_get_policy[ETHTOOL_A_FEC_HEADER + 1] = {
-	[ETHTOOL_A_FEC_HEADER]	= NLA_POLICY_NESTED(ethnl_header_policy),
+	[ETHTOOL_A_FEC_HEADER]	= NLA_POLICY_NESTED(ethnl_header_policy_stats),
 };
 
 static void
@@ -64,6 +68,28 @@ ethtool_link_modes_to_fecparam(struct ethtool_fecparam *fec,
 	return 0;
 }
 
+static void
+fec_stats_recalc(struct fec_stat_grp *grp, struct ethtool_fec_stat *stats)
+{
+	int i;
+
+	if (stats->lanes[0] == ETHTOOL_STAT_NOT_SET) {
+		grp->stats[0] = stats->total;
+		grp->cnt = stats->total != ETHTOOL_STAT_NOT_SET;
+		return;
+	}
+
+	grp->cnt = 1;
+	grp->stats[0] = 0;
+	for (i = 0; i < ETHTOOL_MAX_LANES; i++) {
+		if (stats->lanes[i] == ETHTOOL_STAT_NOT_SET)
+			break;
+
+		grp->stats[0] += stats->lanes[i];
+		grp->stats[grp->cnt++] = stats->lanes[i];
+	}
+}
+
 static int fec_prepare_data(const struct ethnl_req_info *req_base,
 			    struct ethnl_reply_data *reply_base,
 			    struct genl_info *info)
@@ -82,6 +108,17 @@ static int fec_prepare_data(const struct ethnl_req_info *req_base,
 	ret = dev->ethtool_ops->get_fecparam(dev, &fec);
 	if (ret)
 		goto out_complete;
+	if (req_base->flags & ETHTOOL_FLAG_STATS &&
+	    dev->ethtool_ops->get_fec_stats) {
+		struct ethtool_fec_stats stats;
+
+		ethtool_stats_init((u64 *)&stats, sizeof(stats) / 8);
+		dev->ethtool_ops->get_fec_stats(dev, &stats);
+
+		fec_stats_recalc(&data->corr, &stats.corrected_blocks);
+		fec_stats_recalc(&data->uncorr, &stats.uncorrectable_blocks);
+		fec_stats_recalc(&data->corr_bits, &stats.corrected_bits);
+	}
 
 	WARN_ON_ONCE(fec.reserved);
 
@@ -120,9 +157,40 @@ static int fec_reply_size(const struct ethnl_req_info *req_base,
 	len += nla_total_size(sizeof(u8)) +	/* _FEC_AUTO */
 	       nla_total_size(sizeof(u32));	/* _FEC_ACTIVE */
 
+	if (req_base->flags & ETHTOOL_FLAG_STATS)
+		len += 3 * nla_total_size_64bit(sizeof(u64) *
+						(1 + ETHTOOL_MAX_LANES));
+
 	return len;
 }
 
+static int fec_put_stats(struct sk_buff *skb, const struct fec_reply_data *data)
+{
+	struct nlattr *nest;
+
+	nest = nla_nest_start(skb, ETHTOOL_A_FEC_STATS);
+	if (!nest)
+		return -EMSGSIZE;
+
+	if (nla_put_64bit(skb, ETHTOOL_A_FEC_STAT_CORRECTED,
+			  sizeof(u64) * data->corr.cnt,
+			  data->corr.stats, ETHTOOL_A_FEC_STAT_PAD) ||
+	    nla_put_64bit(skb, ETHTOOL_A_FEC_STAT_UNCORR,
+			  sizeof(u64) * data->uncorr.cnt,
+			  data->uncorr.stats, ETHTOOL_A_FEC_STAT_PAD) ||
+	    nla_put_64bit(skb, ETHTOOL_A_FEC_STAT_CORR_BITS,
+			  sizeof(u64) * data->corr_bits.cnt,
+			  data->corr_bits.stats, ETHTOOL_A_FEC_STAT_PAD))
+		goto err_cancel;
+
+	nla_nest_end(skb, nest);
+	return 0;
+
+err_cancel:
+	nla_nest_cancel(skb, nest);
+	return -EMSGSIZE;
+}
+
 static int fec_fill_reply(struct sk_buff *skb,
 			  const struct ethnl_req_info *req_base,
 			  const struct ethnl_reply_data *reply_base)
@@ -143,6 +211,9 @@ static int fec_fill_reply(struct sk_buff *skb,
 	     nla_put_u32(skb, ETHTOOL_A_FEC_ACTIVE, data->active_fec)))
 		return -EMSGSIZE;
 
+	if (req_base->flags & ETHTOOL_FLAG_STATS && fec_put_stats(skb, data))
+		return -EMSGSIZE;
+
 	return 0;
 }
 
-- 
2.30.2

[PATCH net-next v2 4/6] bnxt: implement ethtool::get_fec_stats

[Jakub Kicinski]

Report corrected bits.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Reviewed-by: Michael Chan <michael.chan@broadcom.com>
---
 drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
index 2f8b193a772d..7b90357daba1 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
@@ -1930,6 +1930,20 @@ static int bnxt_get_fecparam(struct net_device *dev,
 	return 0;
 }
 
+static void bnxt_get_fec_stats(struct net_device *dev,
+			       struct ethtool_fec_stats *fec_stats)
+{
+	struct bnxt *bp = netdev_priv(dev);
+	u64 *rx;
+
+	if (BNXT_VF(bp) || !(bp->flags & BNXT_FLAG_PORT_STATS_EXT))
+		return;
+
+	rx = bp->rx_port_stats_ext.sw_stats;
+	fec_stats->corrected_bits.total =
+		*(rx + BNXT_RX_STATS_EXT_OFFSET(rx_corrected_bits));
+}
+
 static u32 bnxt_ethtool_forced_fec_to_fw(struct bnxt_link_info *link_info,
 					 u32 fec)
 {
@@ -3991,6 +4005,7 @@ const struct ethtool_ops bnxt_ethtool_ops = {
 				     ETHTOOL_COALESCE_USE_ADAPTIVE_RX,
 	.get_link_ksettings	= bnxt_get_link_ksettings,
 	.set_link_ksettings	= bnxt_set_link_ksettings,
+	.get_fec_stats		= bnxt_get_fec_stats,
 	.get_fecparam		= bnxt_get_fecparam,
 	.set_fecparam		= bnxt_set_fecparam,
 	.get_pause_stats	= bnxt_get_pause_stats,
-- 
2.30.2

[PATCH net-next v2 5/6] sfc: ef10: implement ethtool::get_fec_stats

[Jakub Kicinski]

Report what appears to be the standard block counts:
 - 30.5.1.1.17 aFECCorrectedBlocks
 - 30.5.1.1.18 aFECUncorrectableBlocks

Don't report the per-lane symbol counts, if those really
count symbols they are not what the standard calls for
(even if symbols seem like the most useful thing to count.)

Fingers crossed that fec_corrected_errors is not in symbols.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
 drivers/net/ethernet/sfc/ef10.c       | 17 +++++++++++++++++
 drivers/net/ethernet/sfc/ethtool.c    | 10 ++++++++++
 drivers/net/ethernet/sfc/net_driver.h |  3 +++
 3 files changed, 30 insertions(+)

diff --git a/drivers/net/ethernet/sfc/ef10.c b/drivers/net/ethernet/sfc/ef10.c
index da6886dcac37..c873f961d5a5 100644
--- a/drivers/net/ethernet/sfc/ef10.c
+++ b/drivers/net/ethernet/sfc/ef10.c
@@ -1747,6 +1747,22 @@ static size_t efx_ef10_describe_stats(struct efx_nic *efx, u8 *names)
 				      mask, names);
 }
 
+static void efx_ef10_get_fec_stats(struct efx_nic *efx,
+				   struct ethtool_fec_stats *fec_stats)
+{
+	DECLARE_BITMAP(mask, EF10_STAT_COUNT);
+	struct efx_ef10_nic_data *nic_data = efx->nic_data;
+	u64 *stats = nic_data->stats;
+
+	efx_ef10_get_stat_mask(efx, mask);
+	if (test_bit(EF10_STAT_fec_corrected_errors, mask))
+		fec_stats->corrected_blocks.total =
+			stats[EF10_STAT_fec_corrected_errors];
+	if (test_bit(EF10_STAT_fec_uncorrected_errors, mask))
+		fec_stats->uncorrectable_blocks.total =
+			stats[EF10_STAT_fec_uncorrected_errors];
+}
+
 static size_t efx_ef10_update_stats_common(struct efx_nic *efx, u64 *full_stats,
 					   struct rtnl_link_stats64 *core_stats)
 {
@@ -4122,6 +4138,7 @@ const struct efx_nic_type efx_hunt_a0_nic_type = {
 	.get_wol = efx_ef10_get_wol,
 	.set_wol = efx_ef10_set_wol,
 	.resume_wol = efx_port_dummy_op_void,
+	.get_fec_stats = efx_ef10_get_fec_stats,
 	.test_chip = efx_ef10_test_chip,
 	.test_nvram = efx_mcdi_nvram_test_all,
 	.mcdi_request = efx_ef10_mcdi_request,
diff --git a/drivers/net/ethernet/sfc/ethtool.c b/drivers/net/ethernet/sfc/ethtool.c
index 12a91c559aa2..058d9fe41d99 100644
--- a/drivers/net/ethernet/sfc/ethtool.c
+++ b/drivers/net/ethernet/sfc/ethtool.c
@@ -206,6 +206,15 @@ static int efx_ethtool_set_wol(struct net_device *net_dev,
 	return efx->type->set_wol(efx, wol->wolopts);
 }
 
+static void efx_ethtool_get_fec_stats(struct net_device *net_dev,
+				      struct ethtool_fec_stats *fec_stats)
+{
+	struct efx_nic *efx = netdev_priv(net_dev);
+
+	if (efx->type->get_fec_stats)
+		efx->type->get_fec_stats(efx, fec_stats);
+}
+
 static int efx_ethtool_get_ts_info(struct net_device *net_dev,
 				   struct ethtool_ts_info *ts_info)
 {
@@ -257,6 +266,7 @@ const struct ethtool_ops efx_ethtool_ops = {
 	.get_module_eeprom	= efx_ethtool_get_module_eeprom,
 	.get_link_ksettings	= efx_ethtool_get_link_ksettings,
 	.set_link_ksettings	= efx_ethtool_set_link_ksettings,
+	.get_fec_stats		= efx_ethtool_get_fec_stats,
 	.get_fecparam		= efx_ethtool_get_fecparam,
 	.set_fecparam		= efx_ethtool_set_fecparam,
 };
diff --git a/drivers/net/ethernet/sfc/net_driver.h b/drivers/net/ethernet/sfc/net_driver.h
index 9f7dfdf708cf..9b4b25704271 100644
--- a/drivers/net/ethernet/sfc/net_driver.h
+++ b/drivers/net/ethernet/sfc/net_driver.h
@@ -1187,6 +1187,7 @@ struct efx_udp_tunnel {
  * @get_wol: Get WoL configuration from driver state
  * @set_wol: Push WoL configuration to the NIC
  * @resume_wol: Synchronise WoL state between driver and MC (e.g. after resume)
+ * @get_fec_stats: Get standard FEC statistics.
  * @test_chip: Test registers.  May use efx_farch_test_registers(), and is
  *	expected to reset the NIC.
  * @test_nvram: Test validity of NVRAM contents
@@ -1332,6 +1333,8 @@ struct efx_nic_type {
 	void (*get_wol)(struct efx_nic *efx, struct ethtool_wolinfo *wol);
 	int (*set_wol)(struct efx_nic *efx, u32 type);
 	void (*resume_wol)(struct efx_nic *efx);
+	void (*get_fec_stats)(struct efx_nic *efx,
+			      struct ethtool_fec_stats *fec_stats);
 	unsigned int (*check_caps)(const struct efx_nic *efx,
 				   u8 flag,
 				   u32 offset);
-- 
2.30.2

Re: [PATCH net-next v2 5/6] sfc: ef10: implement ethtool::get_fec_stats

[Edward Cree]

On 15/04/2021 23:53, Jakub Kicinski wrote:
> Report what appears to be the standard block counts:
>  - 30.5.1.1.17 aFECCorrectedBlocks
>  - 30.5.1.1.18 aFECUncorrectableBlocks
> 
> Don't report the per-lane symbol counts, if those really
> count symbols they are not what the standard calls for
> (even if symbols seem like the most useful thing to count.)
> 
> Fingers crossed that fec_corrected_errors is not in symbols.
> 
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>

For the record: yes, fec_[un]corrected_errors are counts of
 FEC codewords, not symbols, so this patch is correct.

Whereas, the per-lane counts definitely are counting symbols.

-ed

Re: [PATCH net-next v2 5/6] sfc: ef10: implement ethtool::get_fec_stats

[Jakub Kicinski]

On Tue, 20 Apr 2021 16:13:21 +0100 Edward Cree wrote:
> On 15/04/2021 23:53, Jakub Kicinski wrote:
> > Report what appears to be the standard block counts:
> >  - 30.5.1.1.17 aFECCorrectedBlocks
> >  - 30.5.1.1.18 aFECUncorrectableBlocks
> > 
> > Don't report the per-lane symbol counts, if those really
> > count symbols they are not what the standard calls for
> > (even if symbols seem like the most useful thing to count.)
> > 
> > Fingers crossed that fec_corrected_errors is not in symbols.
> > 
> > Signed-off-by: Jakub Kicinski <kuba@kernel.org>  
> 
> For the record: yes, fec_[un]corrected_errors are counts of
>  FEC codewords, not symbols, so this patch is correct.
> 
> Whereas, the per-lane counts definitely are counting symbols.

Thanks for reviewing!

[PATCH net-next v2 6/6] mlx5: implement ethtool::get_fec_stats

[Jakub Kicinski]

Report corrected bits.

v2: catch reg access errors (Saeed)

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
Acked-by: Saeed Mahameed <saeedm@nvidia.com>
---
 .../ethernet/mellanox/mlx5/core/en_ethtool.c  |  9 ++++++
 .../ethernet/mellanox/mlx5/core/en_stats.c    | 29 +++++++++++++++++--
 .../ethernet/mellanox/mlx5/core/en_stats.h    |  2 ++
 3 files changed, 38 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
index c8057a44d5ab..f17690cbeeea 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
@@ -1602,6 +1602,14 @@ static int mlx5e_set_wol(struct net_device *netdev, struct ethtool_wolinfo *wol)
 	return mlx5_set_port_wol(mdev, mlx5_wol_mode);
 }
 
+static void mlx5e_get_fec_stats(struct net_device *netdev,
+				struct ethtool_fec_stats *fec_stats)
+{
+	struct mlx5e_priv *priv = netdev_priv(netdev);
+
+	mlx5e_stats_fec_get(priv, fec_stats);
+}
+
 static int mlx5e_get_fecparam(struct net_device *netdev,
 			      struct ethtool_fecparam *fecparam)
 {
@@ -2209,6 +2217,7 @@ const struct ethtool_ops mlx5e_ethtool_ops = {
 	.self_test         = mlx5e_self_test,
 	.get_msglevel      = mlx5e_get_msglevel,
 	.set_msglevel      = mlx5e_set_msglevel,
+	.get_fec_stats     = mlx5e_get_fec_stats,
 	.get_fecparam      = mlx5e_get_fecparam,
 	.set_fecparam      = mlx5e_set_fecparam,
 };
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
index ae0570ea08bf..353513bd0d5e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
@@ -768,10 +768,10 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(802_3)
 	mlx5_core_access_reg(mdev, in, sz, out, sz, MLX5_REG_PPCNT, 0, 0);
 }
 
-#define MLX5E_READ_CTR64_BE_F(ptr, c)			\
+#define MLX5E_READ_CTR64_BE_F(ptr, set, c)		\
 	be64_to_cpu(*(__be64 *)((char *)ptr +		\
 		MLX5_BYTE_OFF(ppcnt_reg,		\
-			counter_set.eth_802_3_cntrs_grp_data_layout.c##_high)))
+			      counter_set.set.c##_high)))
 
 void mlx5e_stats_pause_get(struct mlx5e_priv *priv,
 			   struct ethtool_pause_stats *pause_stats)
@@ -791,9 +791,11 @@ void mlx5e_stats_pause_get(struct mlx5e_priv *priv,
 
 	pause_stats->tx_pause_frames =
 		MLX5E_READ_CTR64_BE_F(ppcnt_ieee_802_3,
+				      eth_802_3_cntrs_grp_data_layout,
 				      a_pause_mac_ctrl_frames_transmitted);
 	pause_stats->rx_pause_frames =
 		MLX5E_READ_CTR64_BE_F(ppcnt_ieee_802_3,
+				      eth_802_3_cntrs_grp_data_layout,
 				      a_pause_mac_ctrl_frames_received);
 }
 
@@ -1015,6 +1017,29 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(phy)
 	mlx5_core_access_reg(mdev, in, sz, out, sz, MLX5_REG_PPCNT, 0, 0);
 }
 
+void mlx5e_stats_fec_get(struct mlx5e_priv *priv,
+			 struct ethtool_fec_stats *fec_stats)
+{
+	u32 ppcnt_phy_statistical[MLX5_ST_SZ_DW(ppcnt_reg)];
+	struct mlx5_core_dev *mdev = priv->mdev;
+	u32 in[MLX5_ST_SZ_DW(ppcnt_reg)] = {0};
+	int sz = MLX5_ST_SZ_BYTES(ppcnt_reg);
+
+	if (!MLX5_CAP_PCAM_FEATURE(mdev, ppcnt_statistical_group))
+		return;
+
+	MLX5_SET(ppcnt_reg, in, local_port, 1);
+	MLX5_SET(ppcnt_reg, in, grp, MLX5_PHYSICAL_LAYER_STATISTICAL_GROUP);
+	if (mlx5_core_access_reg(mdev, in, sz, ppcnt_phy_statistical,
+				 sz, MLX5_REG_PPCNT, 0, 0))
+		return;
+
+	fec_stats->corrected_bits.total =
+		MLX5E_READ_CTR64_BE_F(ppcnt_phy_statistical,
+				      phys_layer_statistical_cntrs,
+				      phy_corrected_bits);
+}
+
 #define PPORT_ETH_EXT_OFF(c) \
 	MLX5_BYTE_OFF(ppcnt_reg, \
 		      counter_set.eth_extended_cntrs_grp_data_layout.c##_high)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
index 21d3b8747f93..3f0789e51eed 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.h
@@ -114,6 +114,8 @@ void mlx5e_stats_update_ndo_stats(struct mlx5e_priv *priv);
 
 void mlx5e_stats_pause_get(struct mlx5e_priv *priv,
 			   struct ethtool_pause_stats *pause_stats);
+void mlx5e_stats_fec_get(struct mlx5e_priv *priv,
+			 struct ethtool_fec_stats *fec_stats);
 
 /* Concrete NIC Stats */
 
-- 
2.30.2

Re: [PATCH net-next v2 0/6] ethtool: add standard FEC statistics

[Saeed Mahameed]

On Thu, 2021-04-15 at 15:53 -0700, Jakub Kicinski wrote:
> This set adds uAPI for reporting standard FEC statistics, and
> implements it in a handful of drivers.
> 
> The statistics are taken from the IEEE standard, with one
> extra seemingly popular but not standard statistics added.
> 
> The implementation is similar to that of the pause frame
> statistics, user requests the stats by setting a bit
> (ETHTOOL_FLAG_STATS) in the common ethtool header of
> ETHTOOL_MSG_FEC_GET.
> 
> Since standard defines the statistics per lane what's
> reported is both total and per-lane counters:
> 
>  # ethtool -I --show-fec eth0
>  FEC parameters for eth0:
>  Configured FEC encodings: None
>  Active FEC encoding: None
>  Statistics:
>   corrected_blocks: 256
>     Lane 0: 255
>     Lane 1: 1
>   uncorrectable_blocks: 145
>     Lane 0: 128
>     Lane 1: 17
> 
> v2: check for errors in mlx5 register access
> 
> Jakub Kicinski (6):
>   ethtool: move ethtool_stats_init
>   ethtool: fec_prepare_data() - jump to error handling
>   ethtool: add FEC statistics
>   bnxt: implement ethtool::get_fec_stats
>   sfc: ef10: implement ethtool::get_fec_stats
>   mlx5: implement ethtool::get_fec_stats

For the series:

Reviewed-by: Saeed Mahameed <saeedm@nvidia.com>

Re: [PATCH net-next v2 0/6] ethtool: add standard FEC statistics

[patchwork-bot+netdevbpf]

Hello:

This series was applied to netdev/net-next.git (refs/heads/master):

On Thu, 15 Apr 2021 15:53:12 -0700 you wrote:
> This set adds uAPI for reporting standard FEC statistics, and
> implements it in a handful of drivers.
> 
> The statistics are taken from the IEEE standard, with one
> extra seemingly popular but not standard statistics added.
> 
> The implementation is similar to that of the pause frame
> statistics, user requests the stats by setting a bit
> (ETHTOOL_FLAG_STATS) in the common ethtool header of
> ETHTOOL_MSG_FEC_GET.
> 
> [...]

Here is the summary with links:
  - [net-next,v2,1/6] ethtool: move ethtool_stats_init
    https://git.kernel.org/netdev/net-next/c/c5797f8a6415
  - [net-next,v2,2/6] ethtool: fec_prepare_data() - jump to error handling
    https://git.kernel.org/netdev/net-next/c/3d7cc109ecf7
  - [net-next,v2,3/6] ethtool: add FEC statistics
    https://git.kernel.org/netdev/net-next/c/be85dbfeb37c
  - [net-next,v2,4/6] bnxt: implement ethtool::get_fec_stats
    https://git.kernel.org/netdev/net-next/c/c9ca5c3aabaf
  - [net-next,v2,5/6] sfc: ef10: implement ethtool::get_fec_stats
    https://git.kernel.org/netdev/net-next/c/cab351be53c2
  - [net-next,v2,6/6] mlx5: implement ethtool::get_fec_stats
    https://git.kernel.org/netdev/net-next/c/1703bb50df0e

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: [PATCH net-next v2 0/6] ethtool: add standard FEC statistics

[huangguangbin (A)]

On 2021/4/16 6:53, Jakub Kicinski wrote:
> This set adds uAPI for reporting standard FEC statistics, and
> implements it in a handful of drivers.
> 
> The statistics are taken from the IEEE standard, with one
> extra seemingly popular but not standard statistics added.
> 
> The implementation is similar to that of the pause frame
> statistics, user requests the stats by setting a bit
> (ETHTOOL_FLAG_STATS) in the common ethtool header of
> ETHTOOL_MSG_FEC_GET.
> 
> Since standard defines the statistics per lane what's
> reported is both total and per-lane counters:
> 
>   # ethtool -I --show-fec eth0
>   FEC parameters for eth0:
>   Configured FEC encodings: None
>   Active FEC encoding: None
>   Statistics:
>    corrected_blocks: 256
>      Lane 0: 255
>      Lane 1: 1
>    uncorrectable_blocks: 145
>      Lane 0: 128
>      Lane 1: 17
> 
Hi, I have a doubt that why active FEC encoding is None here?  Should it actually be BaseR or RS if FEC statistics are reported?

> v2: check for errors in mlx5 register access
> 
> Jakub Kicinski (6):
>    ethtool: move ethtool_stats_init
>    ethtool: fec_prepare_data() - jump to error handling
>    ethtool: add FEC statistics
>    bnxt: implement ethtool::get_fec_stats
>    sfc: ef10: implement ethtool::get_fec_stats
>    mlx5: implement ethtool::get_fec_stats
> 
>   Documentation/networking/ethtool-netlink.rst  | 21 +++++
>   Documentation/networking/statistics.rst       |  2 +
>   .../net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 15 ++++
>   .../ethernet/mellanox/mlx5/core/en_ethtool.c  |  9 +++
>   .../ethernet/mellanox/mlx5/core/en_stats.c    | 29 ++++++-
>   .../ethernet/mellanox/mlx5/core/en_stats.h    |  2 +
>   drivers/net/ethernet/sfc/ef10.c               | 17 ++++
>   drivers/net/ethernet/sfc/ethtool.c            | 10 +++
>   drivers/net/ethernet/sfc/net_driver.h         |  3 +
>   include/linux/ethtool.h                       | 46 +++++++++++
>   include/uapi/linux/ethtool_netlink.h          | 14 ++++
>   net/ethtool/fec.c                             | 80 ++++++++++++++++++-
>   net/ethtool/pause.c                           |  6 --
>   13 files changed, 242 insertions(+), 12 deletions(-)
> 

Re: [PATCH net-next v2 0/6] ethtool: add standard FEC statistics

[Jakub Kicinski]

On Tue, 18 May 2021 14:48:13 +0800 huangguangbin (A) wrote:
> On 2021/4/16 6:53, Jakub Kicinski wrote:
> > This set adds uAPI for reporting standard FEC statistics, and
> > implements it in a handful of drivers.
> > 
> > The statistics are taken from the IEEE standard, with one
> > extra seemingly popular but not standard statistics added.
> > 
> > The implementation is similar to that of the pause frame
> > statistics, user requests the stats by setting a bit
> > (ETHTOOL_FLAG_STATS) in the common ethtool header of
> > ETHTOOL_MSG_FEC_GET.
> > 
> > Since standard defines the statistics per lane what's
> > reported is both total and per-lane counters:
> > 
> >   # ethtool -I --show-fec eth0
> >   FEC parameters for eth0:
> >   Configured FEC encodings: None
> >   Active FEC encoding: None
> >   Statistics:
> >    corrected_blocks: 256
> >      Lane 0: 255
> >      Lane 1: 1
> >    uncorrectable_blocks: 145
> >      Lane 0: 128
> >      Lane 1: 17
> 
> Hi, I have a doubt that why active FEC encoding is None here? 
> Should it actually be BaseR or RS if FEC statistics are reported?

Hi! Good point. The values in the example are collected from a netdevsim
based mock up which I used for testing the interface, not real hardware.
In reality seeing None and corrected/uncorrectable blocks is not valid.
That said please keep in mind that the statistics should not be reset
when settings are changed, so OFF + stats may happen.

Re: [PATCH net-next v2 0/6] ethtool: add standard FEC statistics

[huangguangbin (A)]

On 2021/5/19 1:30, Jakub Kicinski wrote:
> On Tue, 18 May 2021 14:48:13 +0800 huangguangbin (A) wrote:
>> On 2021/4/16 6:53, Jakub Kicinski wrote:
>>> This set adds uAPI for reporting standard FEC statistics, and
>>> implements it in a handful of drivers.
>>>
>>> The statistics are taken from the IEEE standard, with one
>>> extra seemingly popular but not standard statistics added.
>>>
>>> The implementation is similar to that of the pause frame
>>> statistics, user requests the stats by setting a bit
>>> (ETHTOOL_FLAG_STATS) in the common ethtool header of
>>> ETHTOOL_MSG_FEC_GET.
>>>
>>> Since standard defines the statistics per lane what's
>>> reported is both total and per-lane counters:
>>>
>>>    # ethtool -I --show-fec eth0
>>>    FEC parameters for eth0:
>>>    Configured FEC encodings: None
>>>    Active FEC encoding: None
>>>    Statistics:
>>>     corrected_blocks: 256
>>>       Lane 0: 255
>>>       Lane 1: 1
>>>     uncorrectable_blocks: 145
>>>       Lane 0: 128
>>>       Lane 1: 17
>>
>> Hi, I have a doubt that why active FEC encoding is None here?
>> Should it actually be BaseR or RS if FEC statistics are reported?
> 
> Hi! Good point. The values in the example are collected from a netdevsim
> based mock up which I used for testing the interface, not real hardware.
> In reality seeing None and corrected/uncorrectable blocks is not valid.
> That said please keep in mind that the statistics should not be reset
> when settings are changed, so OFF + stats may happen.
> .
> 
Hi, Jakub, I have another question of per-lane counters.
If speed is changed, do the lane number of FEC statistics need to follow the change?
For examples, change speed from 200Gbps to 50Gbps, the actual used lane number is
changed from 4 to 2, in this case, how many lanes are needed to display FEC statistic?
Still 4 lanes or 2 lanes?

Thanks,
Guangbin
.

Re: [PATCH net-next v2 0/6] ethtool: add standard FEC statistics

[Jakub Kicinski]

On Thu, 1 Jul 2021 20:18:15 +0800 huangguangbin (A) wrote:
> On 2021/5/19 1:30, Jakub Kicinski wrote:
> > On Tue, 18 May 2021 14:48:13 +0800 huangguangbin (A) wrote:  
> >> Hi, I have a doubt that why active FEC encoding is None here?
> >> Should it actually be BaseR or RS if FEC statistics are reported?  
> > 
> > Hi! Good point. The values in the example are collected from a netdevsim
> > based mock up which I used for testing the interface, not real hardware.
> > In reality seeing None and corrected/uncorrectable blocks is not valid.
> > That said please keep in mind that the statistics should not be reset
> > when settings are changed, so OFF + stats may happen.
> > .
> >   
> Hi, Jakub, I have another question of per-lane counters.
> If speed is changed, do the lane number of FEC statistics need to follow the change?
> For examples, change speed from 200Gbps to 50Gbps, the actual used lane number is
> changed from 4 to 2, in this case, how many lanes are needed to display FEC statistic?
> Still 4 lanes or 2 lanes?

I'd always report counters for max number of lanes supported by the HW.
Much like you'd do if you were directly exposing the register values,
the registers are still there in HW.