[PATCH for-next 0/4] RDMA/erdma: cleanups and updates 9-9-2022

[PATCH for-next 0/4] RDMA/erdma: cleanups and updates 9-9-2022

[Cheng Xu]

Hi,

This series has new updates for erdma driver:
- #1~#3 aim at making code more clean, including eliminating unnecessary
  castings, removing redundant include headers, and hiding hardware
  internal opcodes.
- #4 introcuces dynamic mtu support.

Thanks,
Cheng Xu

Cheng Xu (4):
  RDMA/erdma: Eliminate unnecessary casting for erdma_post_cmd_wait
  RDMA/erdma: Remove redundant includes
  RDMA/erdma: Make hardware internal opcodes invisible to driver
  RDMA/erdma: Support dynamic mtu

 drivers/infiniband/hw/erdma/erdma.h       |  4 ++-
 drivers/infiniband/hw/erdma/erdma_cm.c    |  8 ------
 drivers/infiniband/hw/erdma/erdma_cmdq.c  |  8 +-----
 drivers/infiniband/hw/erdma/erdma_cq.c    |  4 ---
 drivers/infiniband/hw/erdma/erdma_eq.c    | 13 ++-------
 drivers/infiniband/hw/erdma/erdma_hw.h    | 14 ++++++---
 drivers/infiniband/hw/erdma/erdma_main.c  | 17 +++++------
 drivers/infiniband/hw/erdma/erdma_qp.c    | 15 ++--------
 drivers/infiniband/hw/erdma/erdma_verbs.c | 35 +++++++++++------------
 drivers/infiniband/hw/erdma/erdma_verbs.h |  9 +-----
 10 files changed, 43 insertions(+), 84 deletions(-)

-- 
2.27.0

[PATCH for-next 1/4] RDMA/erdma: Eliminate unnecessary casting for erdma_post_cmd_wait

[Cheng Xu]

erdma_post_cmd_wait does not use the 'u64 *req' input parameter directly.
So it is better to define it to 'void *req', and by this we can eliminate
the casting when calling erdma_post_cmd_wait.

Signed-off-by: Cheng Xu <chengyou@linux.alibaba.com>
---
 drivers/infiniband/hw/erdma/erdma.h       |  2 +-
 drivers/infiniband/hw/erdma/erdma_cmdq.c  |  2 +-
 drivers/infiniband/hw/erdma/erdma_eq.c    |  7 ++-----
 drivers/infiniband/hw/erdma/erdma_qp.c    |  6 ++----
 drivers/infiniband/hw/erdma/erdma_verbs.c | 17 ++++++-----------
 5 files changed, 12 insertions(+), 22 deletions(-)

diff --git a/drivers/infiniband/hw/erdma/erdma.h b/drivers/infiniband/hw/erdma/erdma.h
index 2aae635c1c8d..07bcd688fdb7 100644
--- a/drivers/infiniband/hw/erdma/erdma.h
+++ b/drivers/infiniband/hw/erdma/erdma.h
@@ -269,7 +269,7 @@ void erdma_finish_cmdq_init(struct erdma_dev *dev);
 void erdma_cmdq_destroy(struct erdma_dev *dev);
 
 void erdma_cmdq_build_reqhdr(u64 *hdr, u32 mod, u32 op);
-int erdma_post_cmd_wait(struct erdma_cmdq *cmdq, u64 *req, u32 req_size,
+int erdma_post_cmd_wait(struct erdma_cmdq *cmdq, void *req, u32 req_size,
 			u64 *resp0, u64 *resp1);
 void erdma_cmdq_completion_handler(struct erdma_cmdq *cmdq);
 
diff --git a/drivers/infiniband/hw/erdma/erdma_cmdq.c b/drivers/infiniband/hw/erdma/erdma_cmdq.c
index 57da0c670472..c8f93dc11449 100644
--- a/drivers/infiniband/hw/erdma/erdma_cmdq.c
+++ b/drivers/infiniband/hw/erdma/erdma_cmdq.c
@@ -441,7 +441,7 @@ void erdma_cmdq_build_reqhdr(u64 *hdr, u32 mod, u32 op)
 	       FIELD_PREP(ERDMA_CMD_HDR_OPCODE_MASK, op);
 }
 
-int erdma_post_cmd_wait(struct erdma_cmdq *cmdq, u64 *req, u32 req_size,
+int erdma_post_cmd_wait(struct erdma_cmdq *cmdq, void *req, u32 req_size,
 			u64 *resp0, u64 *resp1)
 {
 	struct erdma_comp_wait *comp_wait;
diff --git a/drivers/infiniband/hw/erdma/erdma_eq.c b/drivers/infiniband/hw/erdma/erdma_eq.c
index 8f2d094e0227..09ddedb5c1b5 100644
--- a/drivers/infiniband/hw/erdma/erdma_eq.c
+++ b/drivers/infiniband/hw/erdma/erdma_eq.c
@@ -229,9 +229,7 @@ static int create_eq_cmd(struct erdma_dev *dev, u32 eqn, struct erdma_eq *eq)
 	req.db_dma_addr_l = lower_32_bits(db_info_dma_addr);
 	req.db_dma_addr_h = upper_32_bits(db_info_dma_addr);
 
-	return erdma_post_cmd_wait(&dev->cmdq, (u64 *)&req,
-				   sizeof(struct erdma_cmdq_create_eq_req),
-				   NULL, NULL);
+	return erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
 }
 
 static int erdma_ceq_init_one(struct erdma_dev *dev, u16 ceqn)
@@ -281,8 +279,7 @@ static void erdma_ceq_uninit_one(struct erdma_dev *dev, u16 ceqn)
 	req.qtype = ERDMA_EQ_TYPE_CEQ;
 	req.vector_idx = ceqn + 1;
 
-	err = erdma_post_cmd_wait(&dev->cmdq, (u64 *)&req, sizeof(req), NULL,
-				  NULL);
+	err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
 	if (err)
 		return;
 
diff --git a/drivers/infiniband/hw/erdma/erdma_qp.c b/drivers/infiniband/hw/erdma/erdma_qp.c
index bc3ec22a62c5..60f898ccc4e4 100644
--- a/drivers/infiniband/hw/erdma/erdma_qp.c
+++ b/drivers/infiniband/hw/erdma/erdma_qp.c
@@ -105,8 +105,7 @@ static int erdma_modify_qp_state_to_rts(struct erdma_qp *qp,
 		req.send_nxt += MPA_DEFAULT_HDR_LEN + qp->attrs.pd_len;
 	req.recv_nxt = tp->rcv_nxt;
 
-	return erdma_post_cmd_wait(&dev->cmdq, (u64 *)&req, sizeof(req), NULL,
-				   NULL);
+	return erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
 }
 
 static int erdma_modify_qp_state_to_stop(struct erdma_qp *qp,
@@ -124,8 +123,7 @@ static int erdma_modify_qp_state_to_stop(struct erdma_qp *qp,
 	req.cfg = FIELD_PREP(ERDMA_CMD_MODIFY_QP_STATE_MASK, attrs->state) |
 		  FIELD_PREP(ERDMA_CMD_MODIFY_QP_QPN_MASK, QP_ID(qp));
 
-	return erdma_post_cmd_wait(&dev->cmdq, (u64 *)&req, sizeof(req), NULL,
-				   NULL);
+	return erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
 }
 
 int erdma_modify_qp_internal(struct erdma_qp *qp, struct erdma_qp_attrs *attrs,
diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c
index 699bd3f59cd3..d04b90549cd6 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.c
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.c
@@ -102,7 +102,7 @@ static int create_qp_cmd(struct erdma_dev *dev, struct erdma_qp *qp)
 		req.rq_db_info_dma_addr = user_qp->rq_db_info_dma_addr;
 	}
 
-	err = erdma_post_cmd_wait(&dev->cmdq, (u64 *)&req, sizeof(req), &resp0,
+	err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), &resp0,
 				  &resp1);
 	if (!err)
 		qp->attrs.cookie =
@@ -151,8 +151,7 @@ static int regmr_cmd(struct erdma_dev *dev, struct erdma_mr *mr)
 	}
 
 post_cmd:
-	return erdma_post_cmd_wait(&dev->cmdq, (u64 *)&req, sizeof(req), NULL,
-				   NULL);
+	return erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
 }
 
 static int create_cq_cmd(struct erdma_dev *dev, struct erdma_cq *cq)
@@ -202,8 +201,7 @@ static int create_cq_cmd(struct erdma_dev *dev, struct erdma_cq *cq)
 		req.cq_db_info_addr = cq->user_cq.db_info_dma_addr;
 	}
 
-	return erdma_post_cmd_wait(&dev->cmdq, (u64 *)&req, sizeof(req), NULL,
-				   NULL);
+	return erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
 }
 
 static int erdma_alloc_idx(struct erdma_resource_cb *res_cb)
@@ -976,8 +974,7 @@ int erdma_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
 	req.cfg = FIELD_PREP(ERDMA_CMD_MR_MPT_IDX_MASK, ibmr->lkey >> 8) |
 		  FIELD_PREP(ERDMA_CMD_MR_KEY_MASK, ibmr->lkey & 0xFF);
 
-	ret = erdma_post_cmd_wait(&dev->cmdq, (u64 *)&req, sizeof(req), NULL,
-				  NULL);
+	ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
 	if (ret)
 		return ret;
 
@@ -1002,8 +999,7 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
 				CMDQ_OPCODE_DESTROY_CQ);
 	req.cqn = cq->cqn;
 
-	err = erdma_post_cmd_wait(&dev->cmdq, (u64 *)&req, sizeof(req), NULL,
-				  NULL);
+	err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
 	if (err)
 		return err;
 
@@ -1040,8 +1036,7 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
 				CMDQ_OPCODE_DESTROY_QP);
 	req.qpn = QP_ID(qp);
 
-	err = erdma_post_cmd_wait(&dev->cmdq, (u64 *)&req, sizeof(req), NULL,
-				  NULL);
+	err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
 	if (err)
 		return err;
 
-- 
2.27.0

[PATCH for-next 2/4] RDMA/erdma: Remove redundant includes

[Cheng Xu]

Many of erdma's includes are redundant, because they are already included
indirectly by kernel headers or custom headers. So we remove all the
unnecessary direct-includes. Besides, add linux/pci.h to erdma.h because
it's also used in the file.

Signed-off-by: Cheng Xu <chengyou@linux.alibaba.com>
---
 drivers/infiniband/hw/erdma/erdma.h       | 1 +
 drivers/infiniband/hw/erdma/erdma_cm.c    | 8 --------
 drivers/infiniband/hw/erdma/erdma_cmdq.c  | 6 ------
 drivers/infiniband/hw/erdma/erdma_cq.c    | 3 ---
 drivers/infiniband/hw/erdma/erdma_eq.c    | 6 ------
 drivers/infiniband/hw/erdma/erdma_main.c  | 9 ---------
 drivers/infiniband/hw/erdma/erdma_qp.c    | 9 ---------
 drivers/infiniband/hw/erdma/erdma_verbs.c | 7 -------
 drivers/infiniband/hw/erdma/erdma_verbs.h | 8 --------
 9 files changed, 1 insertion(+), 56 deletions(-)

diff --git a/drivers/infiniband/hw/erdma/erdma.h b/drivers/infiniband/hw/erdma/erdma.h
index 07bcd688fdb7..cc5e4eb3a21e 100644
--- a/drivers/infiniband/hw/erdma/erdma.h
+++ b/drivers/infiniband/hw/erdma/erdma.h
@@ -9,6 +9,7 @@
 
 #include <linux/bitfield.h>
 #include <linux/netdevice.h>
+#include <linux/pci.h>
 #include <linux/xarray.h>
 #include <rdma/ib_verbs.h>
 
diff --git a/drivers/infiniband/hw/erdma/erdma_cm.c b/drivers/infiniband/hw/erdma/erdma_cm.c
index f13f16479eca..74f6348f240a 100644
--- a/drivers/infiniband/hw/erdma/erdma_cm.c
+++ b/drivers/infiniband/hw/erdma/erdma_cm.c
@@ -10,15 +10,7 @@
 /* Copyright (c) 2008-2019, IBM Corporation */
 /* Copyright (c) 2017, Open Grid Computing, Inc. */
 
-#include <linux/errno.h>
-#include <linux/inetdevice.h>
-#include <linux/net.h>
-#include <linux/types.h>
 #include <linux/workqueue.h>
-#include <net/addrconf.h>
-
-#include <rdma/ib_user_verbs.h>
-#include <rdma/ib_verbs.h>
 
 #include "erdma.h"
 #include "erdma_cm.h"
diff --git a/drivers/infiniband/hw/erdma/erdma_cmdq.c b/drivers/infiniband/hw/erdma/erdma_cmdq.c
index c8f93dc11449..6ebfa6989b11 100644
--- a/drivers/infiniband/hw/erdma/erdma_cmdq.c
+++ b/drivers/infiniband/hw/erdma/erdma_cmdq.c
@@ -4,13 +4,7 @@
 /*          Kai Shen <kaishen@linux.alibaba.com> */
 /* Copyright (c) 2020-2022, Alibaba Group. */
 
-#include <linux/kernel.h>
-#include <linux/pci.h>
-#include <linux/types.h>
-
 #include "erdma.h"
-#include "erdma_hw.h"
-#include "erdma_verbs.h"
 
 static void arm_cmdq_cq(struct erdma_cmdq *cmdq)
 {
diff --git a/drivers/infiniband/hw/erdma/erdma_cq.c b/drivers/infiniband/hw/erdma/erdma_cq.c
index 751c7f9f0de7..2f7390de35d7 100644
--- a/drivers/infiniband/hw/erdma/erdma_cq.c
+++ b/drivers/infiniband/hw/erdma/erdma_cq.c
@@ -4,9 +4,6 @@
 /*          Kai Shen <kaishen@linux.alibaba.com> */
 /* Copyright (c) 2020-2022, Alibaba Group. */
 
-#include <rdma/ib_verbs.h>
-
-#include "erdma_hw.h"
 #include "erdma_verbs.h"
 
 static void *get_next_valid_cqe(struct erdma_cq *cq)
diff --git a/drivers/infiniband/hw/erdma/erdma_eq.c b/drivers/infiniband/hw/erdma/erdma_eq.c
index 09ddedb5c1b5..ed54130d924b 100644
--- a/drivers/infiniband/hw/erdma/erdma_eq.c
+++ b/drivers/infiniband/hw/erdma/erdma_eq.c
@@ -4,12 +4,6 @@
 /*          Kai Shen <kaishen@linux.alibaba.com> */
 /* Copyright (c) 2020-2022, Alibaba Group. */
 
-#include <linux/errno.h>
-#include <linux/pci.h>
-#include <linux/types.h>
-
-#include "erdma.h"
-#include "erdma_hw.h"
 #include "erdma_verbs.h"
 
 #define MAX_POLL_CHUNK_SIZE 16
diff --git a/drivers/infiniband/hw/erdma/erdma_main.c b/drivers/infiniband/hw/erdma/erdma_main.c
index 07e743d24847..6d3e02ba9e77 100644
--- a/drivers/infiniband/hw/erdma/erdma_main.c
+++ b/drivers/infiniband/hw/erdma/erdma_main.c
@@ -4,21 +4,12 @@
 /*          Kai Shen <kaishen@linux.alibaba.com> */
 /* Copyright (c) 2020-2022, Alibaba Group. */
 
-#include <linux/errno.h>
-#include <linux/init.h>
-#include <linux/kernel.h>
-#include <linux/list.h>
 #include <linux/module.h>
-#include <linux/netdevice.h>
-#include <linux/pci.h>
 #include <net/addrconf.h>
 #include <rdma/erdma-abi.h>
-#include <rdma/ib_verbs.h>
-#include <rdma/ib_user_verbs.h>
 
 #include "erdma.h"
 #include "erdma_cm.h"
-#include "erdma_hw.h"
 #include "erdma_verbs.h"
 
 MODULE_AUTHOR("Cheng Xu <chengyou@linux.alibaba.com>");
diff --git a/drivers/infiniband/hw/erdma/erdma_qp.c b/drivers/infiniband/hw/erdma/erdma_qp.c
index 60f898ccc4e4..5fe1a339a435 100644
--- a/drivers/infiniband/hw/erdma/erdma_qp.c
+++ b/drivers/infiniband/hw/erdma/erdma_qp.c
@@ -6,15 +6,6 @@
 /* Authors: Bernard Metzler <bmt@zurich.ibm.com> */
 /* Copyright (c) 2008-2019, IBM Corporation */
 
-#include <linux/errno.h>
-#include <linux/pci.h>
-#include <linux/scatterlist.h>
-#include <linux/types.h>
-
-#include <rdma/ib_user_verbs.h>
-#include <rdma/ib_verbs.h>
-
-#include "erdma.h"
 #include "erdma_cm.h"
 #include "erdma_verbs.h"
 
diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c
index d04b90549cd6..aea93d08af95 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.c
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.c
@@ -9,21 +9,14 @@
 
 /* Copyright (c) 2013-2015, Mellanox Technologies. All rights reserved. */
 
-#include <linux/errno.h>
-#include <linux/pci.h>
-#include <linux/types.h>
-#include <linux/uaccess.h>
 #include <linux/vmalloc.h>
 #include <net/addrconf.h>
 #include <rdma/erdma-abi.h>
 #include <rdma/ib_umem.h>
-#include <rdma/ib_user_verbs.h>
-#include <rdma/ib_verbs.h>
 #include <rdma/uverbs_ioctl.h>
 
 #include "erdma.h"
 #include "erdma_cm.h"
-#include "erdma_hw.h"
 #include "erdma_verbs.h"
 
 static int create_qp_cmd(struct erdma_dev *dev, struct erdma_qp *qp)
diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.h b/drivers/infiniband/hw/erdma/erdma_verbs.h
index c7baddb1f292..fe93e1ac9674 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.h
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.h
@@ -7,15 +7,7 @@
 #ifndef __ERDMA_VERBS_H__
 #define __ERDMA_VERBS_H__
 
-#include <linux/errno.h>
-
-#include <rdma/ib_verbs.h>
-#include <rdma/ib_user_verbs.h>
-#include <rdma/iw_cm.h>
-
 #include "erdma.h"
-#include "erdma_cm.h"
-#include "erdma_hw.h"
 
 /* RDMA Capability. */
 #define ERDMA_MAX_PD (128 * 1024)
-- 
2.27.0

[PATCH for-next 3/4] RDMA/erdma: Make hardware internal opcodes invisible to driver

[Cheng Xu]

Some opcodes are used in hardware internally, and driver does not care
about them. So, we change them to reserved opcodes in driver.

Signed-off-by: Cheng Xu <chengyou@linux.alibaba.com>
---
 drivers/infiniband/hw/erdma/erdma_cq.c | 1 -
 drivers/infiniband/hw/erdma/erdma_hw.h | 8 ++++----
 2 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/hw/erdma/erdma_cq.c b/drivers/infiniband/hw/erdma/erdma_cq.c
index 2f7390de35d7..58e0dc5c75d1 100644
--- a/drivers/infiniband/hw/erdma/erdma_cq.c
+++ b/drivers/infiniband/hw/erdma/erdma_cq.c
@@ -59,7 +59,6 @@ static const enum ib_wc_opcode wc_mapping_table[ERDMA_NUM_OPCODES] = {
 	[ERDMA_OP_RECV_IMM] = IB_WC_RECV_RDMA_WITH_IMM,
 	[ERDMA_OP_RECV_INV] = IB_WC_RECV,
 	[ERDMA_OP_WRITE_WITH_IMM] = IB_WC_RDMA_WRITE,
-	[ERDMA_OP_INVALIDATE] = IB_WC_LOCAL_INV,
 	[ERDMA_OP_RSP_SEND_IMM] = IB_WC_RECV,
 	[ERDMA_OP_SEND_WITH_INV] = IB_WC_SEND,
 	[ERDMA_OP_REG_MR] = IB_WC_REG_MR,
diff --git a/drivers/infiniband/hw/erdma/erdma_hw.h b/drivers/infiniband/hw/erdma/erdma_hw.h
index b210c49c669f..3004cf3ac481 100644
--- a/drivers/infiniband/hw/erdma/erdma_hw.h
+++ b/drivers/infiniband/hw/erdma/erdma_hw.h
@@ -450,13 +450,13 @@ enum erdma_opcode {
 	ERDMA_OP_RECV_IMM = 5,
 	ERDMA_OP_RECV_INV = 6,
 
-	ERDMA_OP_REQ_ERR = 7,
-	ERDMA_OP_READ_RESPONSE = 8,
+	ERDMA_OP_RSVD0 = 7,
+	ERDMA_OP_RSVD1 = 8,
 	ERDMA_OP_WRITE_WITH_IMM = 9,
 
-	ERDMA_OP_RECV_ERR = 10,
+	ERDMA_OP_RSVD2 = 10,
+	ERDMA_OP_RSVD3 = 11,
 
-	ERDMA_OP_INVALIDATE = 11,
 	ERDMA_OP_RSP_SEND_IMM = 12,
 	ERDMA_OP_SEND_WITH_INV = 13,
 
-- 
2.27.0

[PATCH for-next 4/4] RDMA/erdma: Support dynamic mtu

[Cheng Xu]

Hardware now support jumbo frame for RDMA. So we introduce a new CMDQ
message to support mtu change notification.

Signed-off-by: Cheng Xu <chengyou@linux.alibaba.com>
---
 drivers/infiniband/hw/erdma/erdma.h       |  1 +
 drivers/infiniband/hw/erdma/erdma_hw.h    |  6 ++++++
 drivers/infiniband/hw/erdma/erdma_main.c  |  8 +++++++-
 drivers/infiniband/hw/erdma/erdma_verbs.c | 11 +++++++++++
 drivers/infiniband/hw/erdma/erdma_verbs.h |  1 +
 5 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/erdma/erdma.h b/drivers/infiniband/hw/erdma/erdma.h
index cc5e4eb3a21e..730783fbc894 100644
--- a/drivers/infiniband/hw/erdma/erdma.h
+++ b/drivers/infiniband/hw/erdma/erdma.h
@@ -197,6 +197,7 @@ struct erdma_dev {
 	struct erdma_devattr attrs;
 	/* physical port state (only one port per device) */
 	enum ib_port_state state;
+	u32 mtu;
 
 	/* cmdq and aeq use the same msix vector */
 	struct erdma_irq comm_irq;
diff --git a/drivers/infiniband/hw/erdma/erdma_hw.h b/drivers/infiniband/hw/erdma/erdma_hw.h
index 3004cf3ac481..e788887732e1 100644
--- a/drivers/infiniband/hw/erdma/erdma_hw.h
+++ b/drivers/infiniband/hw/erdma/erdma_hw.h
@@ -153,6 +153,7 @@ enum CMDQ_COMMON_OPCODE {
 	CMDQ_OPCODE_CREATE_EQ = 0,
 	CMDQ_OPCODE_DESTROY_EQ = 1,
 	CMDQ_OPCODE_QUERY_FW_INFO = 2,
+	CMDQ_OPCODE_CONF_MTU = 3,
 };
 
 /* cmdq-SQE HDR */
@@ -190,6 +191,11 @@ struct erdma_cmdq_destroy_eq_req {
 	u8 qtype;
 };
 
+struct erdma_cmdq_config_mtu_req {
+	u64 hdr;
+	u32 mtu;
+};
+
 /* create_cq cfg0 */
 #define ERDMA_CMD_CREATE_CQ_DEPTH_MASK GENMASK(31, 24)
 #define ERDMA_CMD_CREATE_CQ_PAGESIZE_MASK GENMASK(23, 20)
diff --git a/drivers/infiniband/hw/erdma/erdma_main.c b/drivers/infiniband/hw/erdma/erdma_main.c
index 6d3e02ba9e77..49778bb294ae 100644
--- a/drivers/infiniband/hw/erdma/erdma_main.c
+++ b/drivers/infiniband/hw/erdma/erdma_main.c
@@ -34,10 +34,15 @@ static int erdma_netdev_event(struct notifier_block *nb, unsigned long event,
 		dev->state = IB_PORT_DOWN;
 		erdma_port_event(dev, IB_EVENT_PORT_ERR);
 		break;
+	case NETDEV_CHANGEMTU:
+		if (dev->mtu != netdev->mtu) {
+			erdma_set_mtu(dev, netdev->mtu);
+			dev->mtu = netdev->mtu;
+		}
+		break;
 	case NETDEV_REGISTER:
 	case NETDEV_UNREGISTER:
 	case NETDEV_CHANGEADDR:
-	case NETDEV_CHANGEMTU:
 	case NETDEV_GOING_DOWN:
 	case NETDEV_CHANGE:
 	default:
@@ -95,6 +100,7 @@ static int erdma_device_register(struct erdma_dev *dev)
 	if (ret)
 		return ret;
 
+	dev->mtu = dev->netdev->mtu;
 	addrconf_addr_eui48((u8 *)&ibdev->node_guid, dev->netdev->dev_addr);
 
 	ret = ib_register_device(ibdev, "erdma_%d", &dev->pdev->dev);
diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c
index aea93d08af95..62be98e2b941 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.c
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.c
@@ -1436,6 +1436,17 @@ int erdma_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
 	return ret;
 }
 
+void erdma_set_mtu(struct erdma_dev *dev, u32 mtu)
+{
+	struct erdma_cmdq_config_mtu_req req;
+
+	erdma_cmdq_build_reqhdr(&req.hdr, CMDQ_SUBMOD_COMMON,
+				CMDQ_OPCODE_CONF_MTU);
+	req.mtu = mtu;
+
+	erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
+}
+
 void erdma_port_event(struct erdma_dev *dev, enum ib_event_type reason)
 {
 	struct ib_event event;
diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.h b/drivers/infiniband/hw/erdma/erdma_verbs.h
index fe93e1ac9674..ab6380635e9e 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.h
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.h
@@ -330,5 +330,6 @@ struct ib_mr *erdma_ib_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type,
 int erdma_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
 		    unsigned int *sg_offset);
 void erdma_port_event(struct erdma_dev *dev, enum ib_event_type reason);
+void erdma_set_mtu(struct erdma_dev *dev, u32 mtu);
 
 #endif
-- 
2.27.0

Re: [PATCH for-next 4/4] RDMA/erdma: Support dynamic mtu

[Leon Romanovsky]

On Fri, Sep 09, 2022 at 05:38:22PM +0800, Cheng Xu wrote:
> Hardware now support jumbo frame for RDMA. So we introduce a new CMDQ
> message to support mtu change notification.
> 
> Signed-off-by: Cheng Xu <chengyou@linux.alibaba.com>
> ---
>  drivers/infiniband/hw/erdma/erdma.h       |  1 +
>  drivers/infiniband/hw/erdma/erdma_hw.h    |  6 ++++++
>  drivers/infiniband/hw/erdma/erdma_main.c  |  8 +++++++-
>  drivers/infiniband/hw/erdma/erdma_verbs.c | 11 +++++++++++
>  drivers/infiniband/hw/erdma/erdma_verbs.h |  1 +
>  5 files changed, 26 insertions(+), 1 deletion(-)

<...>

> --- a/drivers/infiniband/hw/erdma/erdma_main.c
> +++ b/drivers/infiniband/hw/erdma/erdma_main.c
> @@ -34,10 +34,15 @@ static int erdma_netdev_event(struct notifier_block *nb, unsigned long event,
>  		dev->state = IB_PORT_DOWN;
>  		erdma_port_event(dev, IB_EVENT_PORT_ERR);
>  		break;
> +	case NETDEV_CHANGEMTU:
> +		if (dev->mtu != netdev->mtu) {
> +			erdma_set_mtu(dev, netdev->mtu);
> +			dev->mtu = netdev->mtu;
> +		}
> +		break;
>  	case NETDEV_REGISTER:
>  	case NETDEV_UNREGISTER:
>  	case NETDEV_CHANGEADDR:
> -	case NETDEV_CHANGEMTU:
>  	case NETDEV_GOING_DOWN:
>  	case NETDEV_CHANGE:
>  	default:
> @@ -95,6 +100,7 @@ static int erdma_device_register(struct erdma_dev *dev)
>  	if (ret)
>  		return ret;
>  
> +	dev->mtu = dev->netdev->mtu;
>  	addrconf_addr_eui48((u8 *)&ibdev->node_guid, dev->netdev->dev_addr);
>  
>  	ret = ib_register_device(ibdev, "erdma_%d", &dev->pdev->dev);
> diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c
> index aea93d08af95..62be98e2b941 100644
> --- a/drivers/infiniband/hw/erdma/erdma_verbs.c
> +++ b/drivers/infiniband/hw/erdma/erdma_verbs.c
> @@ -1436,6 +1436,17 @@ int erdma_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
>  	return ret;
>  }
>  
> +void erdma_set_mtu(struct erdma_dev *dev, u32 mtu)
> +{
> +	struct erdma_cmdq_config_mtu_req req;
> +
> +	erdma_cmdq_build_reqhdr(&req.hdr, CMDQ_SUBMOD_COMMON,
> +				CMDQ_OPCODE_CONF_MTU);
> +	req.mtu = mtu;
> +
> +	erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
> +}

I don't see any backward compatibility here. How can you make sure that
new code that supports MTU change works correctly on old FW/device?

Thanks

Re: [PATCH for-next 0/4] RDMA/erdma: cleanups and updates 9-9-2022

[Leon Romanovsky]

On Fri, Sep 09, 2022 at 05:38:18PM +0800, Cheng Xu wrote:
> Hi,
> 
> This series has new updates for erdma driver:
> - #1~#3 aim at making code more clean, including eliminating unnecessary
>   castings, removing redundant include headers, and hiding hardware
>   internal opcodes.
> - #4 introcuces dynamic mtu support.
> 
> Thanks,
> Cheng Xu
> 
> Cheng Xu (4):
>   RDMA/erdma: Eliminate unnecessary casting for erdma_post_cmd_wait
>   RDMA/erdma: Remove redundant includes
>   RDMA/erdma: Make hardware internal opcodes invisible to driver

I applied these three to -next.

>   RDMA/erdma: Support dynamic mtu

This patch needs clarification.

Thanks

Re: [PATCH for-next 4/4] RDMA/erdma: Support dynamic mtu

[Cheng Xu]

On 9/20/22 7:32 PM, Leon Romanovsky wrote:
> On Fri, Sep 09, 2022 at 05:38:22PM +0800, Cheng Xu wrote:
>> Hardware now support jumbo frame for RDMA. So we introduce a new CMDQ
>> message to support mtu change notification.
>>
<...>
> 
> I don't see any backward compatibility here. How can you make sure that
> new code that supports MTU change works correctly on old FW/device?
> 

In this case, driver needn't to consider backward compatibility.

ERDMA hardware is programmable part of our iaas infrastructure, and can be
hot-update without BMs/VMs awareness.Before I submitted this patch, all the
FWs has been updated, and support this feature, no old FWs exist.

Thanks,
Cheng Xu

Re: (subset) [PATCH for-next 4/4] RDMA/erdma: Support dynamic mtu

[Leon Romanovsky]

On Fri, 9 Sep 2022 17:38:22 +0800, Cheng Xu wrote:
> Hardware now support jumbo frame for RDMA. So we introduce a new CMDQ
> message to support mtu change notification.
> 
> 

Applied, thanks!

[4/4] RDMA/erdma: Support dynamic mtu
      https://git.kernel.org/rdma/rdma/c/9bdb9350f3808b

Best regards,
-- 
Leon Romanovsky <leon@kernel.org>

Re: [PATCH for-next 4/4] RDMA/erdma: Support dynamic mtu

[Leon Romanovsky]

On Wed, Sep 21, 2022 at 10:28:57AM +0800, Cheng Xu wrote:
> 
> 
> On 9/20/22 7:32 PM, Leon Romanovsky wrote:
> > On Fri, Sep 09, 2022 at 05:38:22PM +0800, Cheng Xu wrote:
> >> Hardware now support jumbo frame for RDMA. So we introduce a new CMDQ
> >> message to support mtu change notification.
> >>
> <...>
> > 
> > I don't see any backward compatibility here. How can you make sure that
> > new code that supports MTU change works correctly on old FW/device?
> > 
> 
> In this case, driver needn't to consider backward compatibility.
> 
> ERDMA hardware is programmable part of our iaas infrastructure, and can be
> hot-update without BMs/VMs awareness.Before I submitted this patch, all the
> FWs has been updated, and support this feature, no old FWs exist.

I'm not big fan of such answers, but ok, applied.

> 
> Thanks,
> Cheng Xu
> 
>