[PATCH for-next 00/16] IB/hfi1, qib, rdmavt: patches for next 08/21/2017

[PATCH for-next 00/16] IB/hfi1, qib, rdmavt: patches for next 08/21/2017

[Dennis Dalessandro]

Hi Doug,

Another week another set of patches. I meant to send on Friday but was waiting
for 0-day builds to complete. I still haven't gotten any notification, the
service may be down but I wanted to get these on the list anyway for review.
If I get any bad result I will reply but there is nothing particularly scary
in here.

Content wise this is more clean ups including a patch for rdmavt to use
the put_swqe/clear_mr_ref helpers that support was added for in qib/hfi1
previously. 

We have some minor bug fixes as well. A large portion of the patch set is
a clean up/refactoring exercise by Harish that I took because it makes the code
cleaner and easier to read.

Note, one of the fixes from Mike is marked stable. I don't know if it's something
that you want to take into this late of an RC so it may be best to just go to
next and flow through the stable process.

Patches can can also be found in my GitHub repo at:
https://github.com/ddalessa/kernel/tree/for-4.14

---

Grzegorz Morys (1):
      IB/hfi1: Ratelimit prints from sdma_interrupt

Harish Chegondi (7):
      IB/hfi1: Clean up hfi1_user_exp_rcv_setup function
      IB/hfi1: Clean up user_sdma_send_pkts() function
      IB/hfi1: Clean up pin_vector_pages() function
      IB/hfi1: Fix the bail out code in pin_vector_pages() function
      IB/hfi1: Remove duplicate definitions of num_user_pages() function
      IB/hfi1: Move structure definitions from user_exp_rcv.c to user_exp_rcv.h
      IB/hfi1: Move structure and MACRO definitions in user_sdma.c to user_sdma.h

Kaike Wan (1):
      IB/hfi1: Add received request info to qp_stats

Kamenee Arumugam (3):
      IB/qib: Remove unnecessary memory allocation for boardname
      IB/qib: Stricter bounds checking for copy and array access
      IB/hfi1: Fix whitespace alignment issue for MAD

Michael J. Ruhl (1):
      IB/hfi1: Improve local kmem_cache_alloc performance

Mike Marciniszyn (3):
      IB/rdmavt: Use rvt_put_swqe() in rvt_clear_mr_ref()
      IB/{qib,hfi1}: Avoid flow control testing for RDMA write operation
      IB/hfi1: Add opcode states to qp_stats


 drivers/infiniband/hw/hfi1/chip.c         |    4 
 drivers/infiniband/hw/hfi1/file_ops.c     |    2 
 drivers/infiniband/hw/hfi1/hfi.h          |    6 
 drivers/infiniband/hw/hfi1/mad.c          |    2 
 drivers/infiniband/hw/hfi1/qp.c           |   13 +
 drivers/infiniband/hw/hfi1/rc.c           |    3 
 drivers/infiniband/hw/hfi1/user_exp_rcv.c |  255 ++++++++++-------
 drivers/infiniband/hw/hfi1/user_exp_rcv.h |   34 ++
 drivers/infiniband/hw/hfi1/user_sdma.c    |  434 +++++++++--------------------
 drivers/infiniband/hw/hfi1/user_sdma.h    |  166 +++++++++++
 drivers/infiniband/hw/hfi1/verbs_txreq.c  |   11 -
 drivers/infiniband/hw/hfi1/vnic_sdma.c    |   13 -
 drivers/infiniband/hw/qib/qib_iba6120.c   |   20 -
 drivers/infiniband/hw/qib/qib_iba7220.c   |   22 +
 drivers/infiniband/hw/qib/qib_iba7322.c   |   40 +--
 drivers/infiniband/hw/qib/qib_init.c      |    1 
 drivers/infiniband/hw/qib/qib_mad.c       |    4 
 drivers/infiniband/hw/qib/qib_rc.c        |    4 
 drivers/infiniband/hw/qib/qib_ruc.c       |    6 
 drivers/infiniband/sw/rdmavt/qp.c         |    6 
 20 files changed, 556 insertions(+), 490 deletions(-)

--
-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 00/16] IB/hfi1, qib, rdmavt: patches for next 08/21/2017

[Dennis Dalessandro]

Hi Doug,

Another week another set of patches. I meant to send on Friday but was waiting
for 0-day builds to complete. I still haven't gotten any notification, the
service may be down but I wanted to get these on the list anyway for review.
If I get any bad result I will reply but there is nothing particularly scary
in here.

Content wise this is more clean ups including a patch for rdmavt to use
the put_swqe/clear_mr_ref helpers that support was added for in qib/hfi1
previously. 

We have some minor bug fixes as well. A large portion of the patch set is
a clean up/refactoring exercise by Harish that I took because it makes the code
cleaner and easier to read.

Note, one of the fixes from Mike is marked stable. I don't know if it's something
that you want to take into this late of an RC so it may be best to just go to
next and flow through the stable process.

Patches can can also be found in my GitHub repo at:
https://github.com/ddalessa/kernel/tree/for-4.14

---

Grzegorz Morys (1):
      IB/hfi1: Ratelimit prints from sdma_interrupt

Harish Chegondi (7):
      IB/hfi1: Clean up hfi1_user_exp_rcv_setup function
      IB/hfi1: Clean up user_sdma_send_pkts() function
      IB/hfi1: Clean up pin_vector_pages() function
      IB/hfi1: Fix the bail out code in pin_vector_pages() function
      IB/hfi1: Remove duplicate definitions of num_user_pages() function
      IB/hfi1: Move structure definitions from user_exp_rcv.c to user_exp_rcv.h
      IB/hfi1: Move structure and MACRO definitions in user_sdma.c to user_sdma.h

Kaike Wan (1):
      IB/hfi1: Add received request info to qp_stats

Kamenee Arumugam (3):
      IB/qib: Remove unnecessary memory allocation for boardname
      IB/qib: Stricter bounds checking for copy and array access
      IB/hfi1: Fix whitespace alignment issue for MAD

Michael J. Ruhl (1):
      IB/hfi1: Improve local kmem_cache_alloc performance

Mike Marciniszyn (3):
      IB/rdmavt: Use rvt_put_swqe() in rvt_clear_mr_ref()
      IB/{qib,hfi1}: Avoid flow control testing for RDMA write operation
      IB/hfi1: Add opcode states to qp_stats


 drivers/infiniband/hw/hfi1/chip.c         |    4 
 drivers/infiniband/hw/hfi1/file_ops.c     |    2 
 drivers/infiniband/hw/hfi1/hfi.h          |    6 
 drivers/infiniband/hw/hfi1/mad.c          |    2 
 drivers/infiniband/hw/hfi1/qp.c           |   13 +
 drivers/infiniband/hw/hfi1/rc.c           |    3 
 drivers/infiniband/hw/hfi1/user_exp_rcv.c |  255 ++++++++++-------
 drivers/infiniband/hw/hfi1/user_exp_rcv.h |   34 ++
 drivers/infiniband/hw/hfi1/user_sdma.c    |  434 +++++++++--------------------
 drivers/infiniband/hw/hfi1/user_sdma.h    |  166 +++++++++++
 drivers/infiniband/hw/hfi1/verbs_txreq.c  |   11 -
 drivers/infiniband/hw/hfi1/vnic_sdma.c    |   13 -
 drivers/infiniband/hw/qib/qib_iba6120.c   |   20 -
 drivers/infiniband/hw/qib/qib_iba7220.c   |   22 +
 drivers/infiniband/hw/qib/qib_iba7322.c   |   40 +--
 drivers/infiniband/hw/qib/qib_init.c      |    1 
 drivers/infiniband/hw/qib/qib_mad.c       |    4 
 drivers/infiniband/hw/qib/qib_rc.c        |    4 
 drivers/infiniband/hw/qib/qib_ruc.c       |    6 
 drivers/infiniband/sw/rdmavt/qp.c         |    6 
 20 files changed, 556 insertions(+), 490 deletions(-)

--
-Denny

[PATCH for-next 01/16] IB/rdmavt: Use rvt_put_swqe() in rvt_clear_mr_ref()

[Dennis Dalessandro]

From: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

hfi1 and qib were converted in previous patches, do the same for rdmavt.

Signed-off-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/sw/rdmavt/qp.c |    6 +-----
 1 files changed, 1 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/qp.c b/drivers/infiniband/sw/rdmavt/qp.c
index 6f6525d..3a238b0 100644
--- a/drivers/infiniband/sw/rdmavt/qp.c
+++ b/drivers/infiniband/sw/rdmavt/qp.c
@@ -440,13 +440,9 @@ static void rvt_clear_mr_refs(struct rvt_qp *qp, int clr_sends)
 	if (clr_sends) {
 		while (qp->s_last != qp->s_head) {
 			struct rvt_swqe *wqe = rvt_get_swqe_ptr(qp, qp->s_last);
-			unsigned i;
 
-			for (i = 0; i < wqe->wr.num_sge; i++) {
-				struct rvt_sge *sge = &wqe->sg_list[i];
+			rvt_put_swqe(wqe);
 
-				rvt_put_mr(sge->mr);
-			}
 			if (qp->ibqp.qp_type == IB_QPT_UD ||
 			    qp->ibqp.qp_type == IB_QPT_SMI ||
 			    qp->ibqp.qp_type == IB_QPT_GSI)

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 02/16] IB/{qib, hfi1}: Avoid flow control testing for RDMA write operation

[Dennis Dalessandro]

From: Mike Marciniszyn <mike.marciniszyn@intel.com>

Section 9.7.7.2.5 of the 1.3 IBTA spec clearly says that receive
credits should never apply to RDMA write.

qib and hfi1 were doing that.  The following situation will result
in a QP hang:
- A prior SEND or RDMA_WRITE with immmediate consumed the last
  credit for a QP using RC receive buffer credits
- The prior op is acked so there are no more acks
- The peer ULP fails to post receive for some reason
- An RDMA write sees that the credits are exhausted and waits
- The peer ULP posts receive buffers
- The ULP posts a send or RDMA write that will be hung

The fix is to avoid the credit test for the RDMA write operation.

Cc: <stable@vger.kernel.org>
Reviewed-by: Kaike Wan <kaike.wan@intel.com>
Signed-off-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro@intel.com>
---
 drivers/infiniband/hw/hfi1/rc.c    |    3 ++-
 drivers/infiniband/hw/qib/qib_rc.c |    4 ++--
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/rc.c b/drivers/infiniband/hw/hfi1/rc.c
index 99defcc..e1cf0c0 100644
--- a/drivers/infiniband/hw/hfi1/rc.c
+++ b/drivers/infiniband/hw/hfi1/rc.c
@@ -442,7 +442,7 @@ int hfi1_make_rc_req(struct rvt_qp *qp, struct hfi1_pkt_state *ps)
 		case IB_WR_RDMA_WRITE:
 			if (newreq && !(qp->s_flags & RVT_S_UNLIMITED_CREDIT))
 				qp->s_lsn++;
-			/* FALLTHROUGH */
+			goto no_flow_control;
 		case IB_WR_RDMA_WRITE_WITH_IMM:
 			/* If no credit, return. */
 			if (!(qp->s_flags & RVT_S_UNLIMITED_CREDIT) &&
@@ -450,6 +450,7 @@ int hfi1_make_rc_req(struct rvt_qp *qp, struct hfi1_pkt_state *ps)
 				qp->s_flags |= RVT_S_WAIT_SSN_CREDIT;
 				goto bail;
 			}
+no_flow_control:
 			put_ib_reth_vaddr(
 				wqe->rdma_wr.remote_addr,
 				&ohdr->u.rc.reth);
diff --git a/drivers/infiniband/hw/qib/qib_rc.c b/drivers/infiniband/hw/qib/qib_rc.c
index 4ddbcac..e9a9173 100644
--- a/drivers/infiniband/hw/qib/qib_rc.c
+++ b/drivers/infiniband/hw/qib/qib_rc.c
@@ -348,7 +348,7 @@ int qib_make_rc_req(struct rvt_qp *qp, unsigned long *flags)
 		case IB_WR_RDMA_WRITE:
 			if (newreq && !(qp->s_flags & RVT_S_UNLIMITED_CREDIT))
 				qp->s_lsn++;
-			/* FALLTHROUGH */
+			goto no_flow_control;
 		case IB_WR_RDMA_WRITE_WITH_IMM:
 			/* If no credit, return. */
 			if (!(qp->s_flags & RVT_S_UNLIMITED_CREDIT) &&
@@ -356,7 +356,7 @@ int qib_make_rc_req(struct rvt_qp *qp, unsigned long *flags)
 				qp->s_flags |= RVT_S_WAIT_SSN_CREDIT;
 				goto bail;
 			}
-
+no_flow_control:
 			ohdr->u.rc.reth.vaddr =
 				cpu_to_be64(wqe->rdma_wr.remote_addr);
 			ohdr->u.rc.reth.rkey =

[PATCH for-next 03/16] IB/qib: Remove unnecessary memory allocation for boardname

[Dennis Dalessandro]

From: Kamenee Arumugam <kamenee.arumugam-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Remove all the memory allocation implemented for boardname and
directly assign the defined string literal.

Reviewed-by: Michael J. Ruhl <michael.j.ruhl-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Kamenee Arumugam <kamenee.arumugam-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/qib/qib_iba6120.c |   20 +++++-----------
 drivers/infiniband/hw/qib/qib_iba7220.c |   22 ++++++-----------
 drivers/infiniband/hw/qib/qib_iba7322.c |   40 +++++++++++++------------------
 drivers/infiniband/hw/qib/qib_init.c    |    1 -
 4 files changed, 32 insertions(+), 51 deletions(-)

diff --git a/drivers/infiniband/hw/qib/qib_iba6120.c b/drivers/infiniband/hw/qib/qib_iba6120.c
index 46045fc..3259a60 100644
--- a/drivers/infiniband/hw/qib/qib_iba6120.c
+++ b/drivers/infiniband/hw/qib/qib_iba6120.c
@@ -1742,38 +1742,32 @@ static void qib_setup_6120_interrupt(struct qib_devdata *dd)
  */
 static void pe_boardname(struct qib_devdata *dd)
 {
-	char *n;
-	u32 boardid, namelen;
+	u32 boardid;
 
 	boardid = SYM_FIELD(dd->revision, Revision,
 			    BoardID);
 
 	switch (boardid) {
 	case 2:
-		n = "InfiniPath_QLE7140";
+		dd->boardname = "InfiniPath_QLE7140";
 		break;
 	default:
 		qib_dev_err(dd, "Unknown 6120 board with ID %u\n", boardid);
-		n = "Unknown_InfiniPath_6120";
+		dd->boardname = "Unknown_InfiniPath_6120";
 		break;
 	}
-	namelen = strlen(n) + 1;
-	dd->boardname = kmalloc(namelen, GFP_KERNEL);
-	if (dd->boardname)
-		snprintf(dd->boardname, namelen, "%s", n);
 
 	if (dd->majrev != 4 || !dd->minrev || dd->minrev > 2)
 		qib_dev_err(dd,
-			"Unsupported InfiniPath hardware revision %u.%u!\n",
-			dd->majrev, dd->minrev);
+			    "Unsupported InfiniPath hardware revision %u.%u!\n",
+			    dd->majrev, dd->minrev);
 
 	snprintf(dd->boardversion, sizeof(dd->boardversion),
 		 "ChipABI %u.%u, %s, InfiniPath%u %u.%u, SW Compat %u\n",
 		 QIB_CHIP_VERS_MAJ, QIB_CHIP_VERS_MIN, dd->boardname,
-		 (unsigned)SYM_FIELD(dd->revision, Revision_R, Arch),
+		 (unsigned int)SYM_FIELD(dd->revision, Revision_R, Arch),
 		 dd->majrev, dd->minrev,
-		 (unsigned)SYM_FIELD(dd->revision, Revision_R, SW));
-
+		 (unsigned int)SYM_FIELD(dd->revision, Revision_R, SW));
 }
 
 /*
diff --git a/drivers/infiniband/hw/qib/qib_iba7220.c b/drivers/infiniband/hw/qib/qib_iba7220.c
index 49cd6e3..04bdd3d 100644
--- a/drivers/infiniband/hw/qib/qib_iba7220.c
+++ b/drivers/infiniband/hw/qib/qib_iba7220.c
@@ -2050,41 +2050,35 @@ static void qib_setup_7220_interrupt(struct qib_devdata *dd)
  */
 static void qib_7220_boardname(struct qib_devdata *dd)
 {
-	char *n;
-	u32 boardid, namelen;
+	u32 boardid;
 
 	boardid = SYM_FIELD(dd->revision, Revision,
 			    BoardID);
 
 	switch (boardid) {
 	case 1:
-		n = "InfiniPath_QLE7240";
+		dd->boardname = "InfiniPath_QLE7240";
 		break;
 	case 2:
-		n = "InfiniPath_QLE7280";
+		dd->boardname = "InfiniPath_QLE7280";
 		break;
 	default:
 		qib_dev_err(dd, "Unknown 7220 board with ID %u\n", boardid);
-		n = "Unknown_InfiniPath_7220";
+		dd->boardname = "Unknown_InfiniPath_7220";
 		break;
 	}
 
-	namelen = strlen(n) + 1;
-	dd->boardname = kmalloc(namelen, GFP_KERNEL);
-	if (dd->boardname)
-		snprintf(dd->boardname, namelen, "%s", n);
-
 	if (dd->majrev != 5 || !dd->minrev || dd->minrev > 2)
 		qib_dev_err(dd,
-			"Unsupported InfiniPath hardware revision %u.%u!\n",
-			dd->majrev, dd->minrev);
+			    "Unsupported InfiniPath hardware revision %u.%u!\n",
+			    dd->majrev, dd->minrev);
 
 	snprintf(dd->boardversion, sizeof(dd->boardversion),
 		 "ChipABI %u.%u, %s, InfiniPath%u %u.%u, SW Compat %u\n",
 		 QIB_CHIP_VERS_MAJ, QIB_CHIP_VERS_MIN, dd->boardname,
-		 (unsigned)SYM_FIELD(dd->revision, Revision_R, Arch),
+		 (unsigned int)SYM_FIELD(dd->revision, Revision_R, Arch),
 		 dd->majrev, dd->minrev,
-		 (unsigned)SYM_FIELD(dd->revision, Revision_R, SW));
+		 (unsigned int)SYM_FIELD(dd->revision, Revision_R, SW));
 }
 
 /*
diff --git a/drivers/infiniband/hw/qib/qib_iba7322.c b/drivers/infiniband/hw/qib/qib_iba7322.c
index 2653064..92ae68c 100644
--- a/drivers/infiniband/hw/qib/qib_iba7322.c
+++ b/drivers/infiniband/hw/qib/qib_iba7322.c
@@ -3582,75 +3582,69 @@ static void qib_setup_7322_interrupt(struct qib_devdata *dd, int clearpend)
 static unsigned qib_7322_boardname(struct qib_devdata *dd)
 {
 	/* Will need enumeration of board-types here */
-	char *n;
-	u32 boardid, namelen;
-	unsigned features = DUAL_PORT_CAP;
+	u32 boardid;
+	unsigned int features = DUAL_PORT_CAP;
 
 	boardid = SYM_FIELD(dd->revision, Revision, BoardID);
 
 	switch (boardid) {
 	case 0:
-		n = "InfiniPath_QLE7342_Emulation";
+		dd->boardname = "InfiniPath_QLE7342_Emulation";
 		break;
 	case 1:
-		n = "InfiniPath_QLE7340";
+		dd->boardname = "InfiniPath_QLE7340";
 		dd->flags |= QIB_HAS_QSFP;
 		features = PORT_SPD_CAP;
 		break;
 	case 2:
-		n = "InfiniPath_QLE7342";
+		dd->boardname = "InfiniPath_QLE7342";
 		dd->flags |= QIB_HAS_QSFP;
 		break;
 	case 3:
-		n = "InfiniPath_QMI7342";
+		dd->boardname = "InfiniPath_QMI7342";
 		break;
 	case 4:
-		n = "InfiniPath_Unsupported7342";
+		dd->boardname = "InfiniPath_Unsupported7342";
 		qib_dev_err(dd, "Unsupported version of QMH7342\n");
 		features = 0;
 		break;
 	case BOARD_QMH7342:
-		n = "InfiniPath_QMH7342";
+		dd->boardname = "InfiniPath_QMH7342";
 		features = 0x24;
 		break;
 	case BOARD_QME7342:
-		n = "InfiniPath_QME7342";
+		dd->boardname = "InfiniPath_QME7342";
 		break;
 	case 8:
-		n = "InfiniPath_QME7362";
+		dd->boardname = "InfiniPath_QME7362";
 		dd->flags |= QIB_HAS_QSFP;
 		break;
 	case BOARD_QMH7360:
-		n = "Intel IB QDR 1P FLR-QSFP Adptr";
+		dd->boardname = "Intel IB QDR 1P FLR-QSFP Adptr";
 		dd->flags |= QIB_HAS_QSFP;
 		break;
 	case 15:
-		n = "InfiniPath_QLE7342_TEST";
+		dd->boardname = "InfiniPath_QLE7342_TEST";
 		dd->flags |= QIB_HAS_QSFP;
 		break;
 	default:
-		n = "InfiniPath_QLE73xy_UNKNOWN";
+		dd->boardname = "InfiniPath_QLE73xy_UNKNOWN";
 		qib_dev_err(dd, "Unknown 7322 board type %u\n", boardid);
 		break;
 	}
 	dd->board_atten = 1; /* index into txdds_Xdr */
 
-	namelen = strlen(n) + 1;
-	dd->boardname = kmalloc(namelen, GFP_KERNEL);
-	if (dd->boardname)
-		snprintf(dd->boardname, namelen, "%s", n);
-
 	snprintf(dd->boardversion, sizeof(dd->boardversion),
 		 "ChipABI %u.%u, %s, InfiniPath%u %u.%u, SW Compat %u\n",
 		 QIB_CHIP_VERS_MAJ, QIB_CHIP_VERS_MIN, dd->boardname,
-		 (unsigned)SYM_FIELD(dd->revision, Revision_R, Arch),
+		 (unsigned int)SYM_FIELD(dd->revision, Revision_R, Arch),
 		 dd->majrev, dd->minrev,
-		 (unsigned)SYM_FIELD(dd->revision, Revision_R, SW));
+		 (unsigned int)SYM_FIELD(dd->revision, Revision_R, SW));
 
 	if (qib_singleport && (features >> PORT_SPD_CAP_SHIFT) & PORT_SPD_CAP) {
 		qib_devinfo(dd->pcidev,
-			"IB%u: Forced to single port mode by module parameter\n",
-			dd->unit);
+			    "IB%u: Forced to single port mode by module parameter\n",
+			    dd->unit);
 		features &= PORT_SPD_CAP;
 	}
 
diff --git a/drivers/infiniband/hw/qib/qib_init.c b/drivers/infiniband/hw/qib/qib_init.c
index 6c16ba1..4a2252a 100644
--- a/drivers/infiniband/hw/qib/qib_init.c
+++ b/drivers/infiniband/hw/qib/qib_init.c
@@ -1398,7 +1398,6 @@ static void cleanup_device_data(struct qib_devdata *dd)
 		qib_free_ctxtdata(dd, rcd);
 	}
 	kfree(tmp);
-	kfree(dd->boardname);
 }
 
 /*

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 04/16] IB/qib: Stricter bounds checking for copy and array access

[Dennis Dalessandro]

From: Kamenee Arumugam <kamenee.arumugam-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Added checking on index value of array 'guids' in qib_ruc.c.
Pass in corrrect size of array for memset operation in qib_mad.c.

Reviewed-by: Michael J. Ruhl <michael.j.ruhl-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Kamenee Arumugam <kamenee.arumugam-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/qib/qib_mad.c |    4 ++--
 drivers/infiniband/hw/qib/qib_ruc.c |    6 ++++--
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/hw/qib/qib_mad.c b/drivers/infiniband/hw/qib/qib_mad.c
index 549c719..82d9da9 100644
--- a/drivers/infiniband/hw/qib/qib_mad.c
+++ b/drivers/infiniband/hw/qib/qib_mad.c
@@ -1573,8 +1573,8 @@ static int pma_get_portcounters_cong(struct ib_pma_mad *pmp,
 	cntrs.port_xmit_packets -= ibp->z_port_xmit_packets;
 	cntrs.port_rcv_packets -= ibp->z_port_rcv_packets;
 
-	memset(pmp->reserved, 0, sizeof(pmp->reserved) +
-	       sizeof(pmp->data));
+	memset(pmp->reserved, 0, sizeof(pmp->reserved));
+	memset(pmp->data, 0, sizeof(pmp->data));
 
 	/*
 	 * Set top 3 bits to indicate interval in picoseconds in
diff --git a/drivers/infiniband/hw/qib/qib_ruc.c b/drivers/infiniband/hw/qib/qib_ruc.c
index e6a42a8..53efbb0 100644
--- a/drivers/infiniband/hw/qib/qib_ruc.c
+++ b/drivers/infiniband/hw/qib/qib_ruc.c
@@ -645,8 +645,10 @@ u32 qib_make_grh(struct qib_ibport *ibp, struct ib_grh *hdr,
 	hdr->hop_limit = grh->hop_limit;
 	/* The SGID is 32-bit aligned. */
 	hdr->sgid.global.subnet_prefix = ibp->rvp.gid_prefix;
-	hdr->sgid.global.interface_id = grh->sgid_index ?
-		ibp->guids[grh->sgid_index - 1] : ppd_from_ibp(ibp)->guid;
+	if (!grh->sgid_index)
+		hdr->sgid.global.interface_id = ppd_from_ibp(ibp)->guid;
+	else if (grh->sgid_index < QIB_GUIDS_PER_PORT)
+		hdr->sgid.global.interface_id = ibp->guids[grh->sgid_index - 1];
 	hdr->dgid = grh->dgid;
 
 	/* GRH header size in 32-bit words. */

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 05/16] IB/hfi1: Ratelimit prints from sdma_interrupt

[Dennis Dalessandro]

From: Grzegorz Morys <grzegorz.morys-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Ratelimit error prints from sdma_interrupt function
that could swarm dmesg otherwise.

Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Grzegorz Morys <grzegorz.morys-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/chip.c |    4 ++--
 drivers/infiniband/hw/hfi1/hfi.h  |    6 ++++++
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/chip.c b/drivers/infiniband/hw/hfi1/chip.c
index 45f6874..ac967a9 100644
--- a/drivers/infiniband/hw/hfi1/chip.c
+++ b/drivers/infiniband/hw/hfi1/chip.c
@@ -8290,8 +8290,8 @@ static irqreturn_t sdma_interrupt(int irq, void *data)
 		/* handle the interrupt(s) */
 		sdma_engine_interrupt(sde, status);
 	} else {
-		dd_dev_err(dd, "SDMA engine %u interrupt, but no status bits set\n",
-			   sde->this_idx);
+		dd_dev_err_ratelimited(dd, "SDMA engine %u interrupt, but no status bits set\n",
+				       sde->this_idx);
 	}
 	return IRQ_HANDLED;
 }
diff --git a/drivers/infiniband/hw/hfi1/hfi.h b/drivers/infiniband/hw/hfi1/hfi.h
index eadb735..3ac9c30 100644
--- a/drivers/infiniband/hw/hfi1/hfi.h
+++ b/drivers/infiniband/hw/hfi1/hfi.h
@@ -2119,9 +2119,15 @@ static inline u64 hfi1_pkt_base_sdma_integrity(struct hfi1_devdata *dd)
 #define dd_dev_emerg(dd, fmt, ...) \
 	dev_emerg(&(dd)->pcidev->dev, "%s: " fmt, \
 		  get_unit_name((dd)->unit), ##__VA_ARGS__)
+
 #define dd_dev_err(dd, fmt, ...) \
 	dev_err(&(dd)->pcidev->dev, "%s: " fmt, \
 			get_unit_name((dd)->unit), ##__VA_ARGS__)
+
+#define dd_dev_err_ratelimited(dd, fmt, ...) \
+	dev_err_ratelimited(&(dd)->pcidev->dev, "%s: " fmt, \
+			get_unit_name((dd)->unit), ##__VA_ARGS__)
+
 #define dd_dev_warn(dd, fmt, ...) \
 	dev_warn(&(dd)->pcidev->dev, "%s: " fmt, \
 			get_unit_name((dd)->unit), ##__VA_ARGS__)

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 06/16] IB/hfi1: Improve local kmem_cache_alloc performance

[Dennis Dalessandro]

From: Michael J. Ruhl <michael.j.ruhl-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Performance analysis shows that the cache callback function
sdma_kmem_cache_ctor contributes to 1/2 of the kmem_cache_allocs
time.

Since all of the fields in the allocated data structure are initialized
in the code path, remove the _ctor function.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Michael J. Ruhl <michael.j.ruhl-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/user_sdma.c   |    9 +--------
 drivers/infiniband/hw/hfi1/verbs_txreq.c |   11 ++---------
 drivers/infiniband/hw/hfi1/vnic_sdma.c   |   13 +++----------
 3 files changed, 6 insertions(+), 27 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index aae1f40..9b89df5 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -331,13 +331,6 @@ static void activate_packet_queue(struct iowait *wait, int reason)
 	wake_up(&wait->wait_dma);
 };
 
-static void sdma_kmem_cache_ctor(void *obj)
-{
-	struct user_sdma_txreq *tx = obj;
-
-	memset(tx, 0, sizeof(*tx));
-}
-
 int hfi1_user_sdma_alloc_queues(struct hfi1_ctxtdata *uctxt,
 				struct hfi1_filedata *fd)
 {
@@ -391,7 +384,7 @@ int hfi1_user_sdma_alloc_queues(struct hfi1_ctxtdata *uctxt,
 					    sizeof(struct user_sdma_txreq),
 					    L1_CACHE_BYTES,
 					    SLAB_HWCACHE_ALIGN,
-					    sdma_kmem_cache_ctor);
+					    NULL);
 	if (!pq->txreq_cache) {
 		dd_dev_err(dd, "[%u] Failed to allocate TxReq cache\n",
 			   uctxt->ctxt);
diff --git a/drivers/infiniband/hw/hfi1/verbs_txreq.c b/drivers/infiniband/hw/hfi1/verbs_txreq.c
index 5d23172..873e48e 100644
--- a/drivers/infiniband/hw/hfi1/verbs_txreq.c
+++ b/drivers/infiniband/hw/hfi1/verbs_txreq.c
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2016 Intel Corporation.
+ * Copyright(c) 2016 - 2017 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -119,13 +119,6 @@ struct verbs_txreq *__get_txreq(struct hfi1_ibdev *dev,
 	return tx;
 }
 
-static void verbs_txreq_kmem_cache_ctor(void *obj)
-{
-	struct verbs_txreq *tx = (struct verbs_txreq *)obj;
-
-	memset(tx, 0, sizeof(*tx));
-}
-
 int verbs_txreq_init(struct hfi1_ibdev *dev)
 {
 	char buf[TXREQ_LEN];
@@ -135,7 +128,7 @@ int verbs_txreq_init(struct hfi1_ibdev *dev)
 	dev->verbs_txreq_cache = kmem_cache_create(buf,
 						   sizeof(struct verbs_txreq),
 						   0, SLAB_HWCACHE_ALIGN,
-						   verbs_txreq_kmem_cache_ctor);
+						   NULL);
 	if (!dev->verbs_txreq_cache)
 		return -ENOMEM;
 	return 0;
diff --git a/drivers/infiniband/hw/hfi1/vnic_sdma.c b/drivers/infiniband/hw/hfi1/vnic_sdma.c
index 7815d74..c3c96c5 100644
--- a/drivers/infiniband/hw/hfi1/vnic_sdma.c
+++ b/drivers/infiniband/hw/hfi1/vnic_sdma.c
@@ -303,22 +303,15 @@ void hfi1_vnic_sdma_init(struct hfi1_vnic_vport_info *vinfo)
 	}
 }
 
-static void hfi1_vnic_txreq_kmem_cache_ctor(void *obj)
-{
-	struct vnic_txreq *tx = (struct vnic_txreq *)obj;
-
-	memset(tx, 0, sizeof(*tx));
-}
-
 int hfi1_vnic_txreq_init(struct hfi1_devdata *dd)
 {
 	char buf[HFI1_VNIC_TXREQ_NAME_LEN];
 
 	snprintf(buf, sizeof(buf), "hfi1_%u_vnic_txreq_cache", dd->unit);
 	dd->vnic.txreq_cache = kmem_cache_create(buf,
-					  sizeof(struct vnic_txreq),
-					  0, SLAB_HWCACHE_ALIGN,
-					  hfi1_vnic_txreq_kmem_cache_ctor);
+						 sizeof(struct vnic_txreq),
+						 0, SLAB_HWCACHE_ALIGN,
+						 NULL);
 	if (!dd->vnic.txreq_cache)
 		return -ENOMEM;
 	return 0;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 07/16] IB/hfi1: Clean up hfi1_user_exp_rcv_setup function

[Dennis Dalessandro]

From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Clean up hfi1_user_exp_rcv_setup function by moving page pinning and
unpinning related code to separate functions. In order to reduce the
number of parameters passed between functions, a new data structure
struct tid_user_buf is defined and used.

Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/user_exp_rcv.c |  232 ++++++++++++++++++-----------
 drivers/infiniband/hw/hfi1/user_exp_rcv.h |    9 +
 2 files changed, 153 insertions(+), 88 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/user_exp_rcv.c b/drivers/infiniband/hw/hfi1/user_exp_rcv.c
index d9036ba..04be178 100644
--- a/drivers/infiniband/hw/hfi1/user_exp_rcv.c
+++ b/drivers/infiniband/hw/hfi1/user_exp_rcv.c
@@ -75,20 +75,21 @@ struct tid_pageset {
 static void unlock_exp_tids(struct hfi1_ctxtdata *uctxt,
 			    struct exp_tid_set *set,
 			    struct hfi1_filedata *fd);
-static u32 find_phys_blocks(struct page **pages, unsigned npages,
-			    struct tid_pageset *list);
-static int set_rcvarray_entry(struct hfi1_filedata *fd, unsigned long vaddr,
+static u32 find_phys_blocks(struct tid_user_buf *tidbuf, unsigned int npages);
+static int set_rcvarray_entry(struct hfi1_filedata *fd,
+			      struct tid_user_buf *tbuf,
 			      u32 rcventry, struct tid_group *grp,
-			      struct page **pages, unsigned npages);
+			      u16 pageidx, unsigned int npages);
 static int tid_rb_insert(void *arg, struct mmu_rb_node *node);
 static void cacheless_tid_rb_remove(struct hfi1_filedata *fdata,
 				    struct tid_rb_node *tnode);
 static void tid_rb_remove(void *arg, struct mmu_rb_node *node);
 static int tid_rb_invalidate(void *arg, struct mmu_rb_node *mnode);
-static int program_rcvarray(struct hfi1_filedata *fd, unsigned long vaddr,
-			    struct tid_group *grp, struct tid_pageset *sets,
-			    unsigned start, u16 count, struct page **pages,
-			    u32 *tidlist, unsigned *tididx, unsigned *pmapped);
+static int program_rcvarray(struct hfi1_filedata *fd, struct tid_user_buf *,
+			    struct tid_group *grp,
+			    unsigned int start, u16 count,
+			    u32 *tidlist, unsigned int *tididx,
+			    unsigned int *pmapped);
 static int unprogram_rcvarray(struct hfi1_filedata *fd, u32 tidinfo,
 			      struct tid_group **grp);
 static void clear_tid_node(struct hfi1_filedata *fd, struct tid_rb_node *node);
@@ -198,6 +199,92 @@ void hfi1_user_exp_rcv_free(struct hfi1_filedata *fd)
 	fd->entry_to_rb = NULL;
 }
 
+/**
+ * Release pinned receive buffer pages.
+ *
+ * @mapped - true if the pages have been DMA mapped. false otherwise.
+ * @idx - Index of the first page to unpin.
+ * @npages - No of pages to unpin.
+ *
+ * If the pages have been DMA mapped (indicated by mapped parameter), their
+ * info will be passed via a struct tid_rb_node. If they haven't been mapped,
+ * their info will be passed via a struct tid_user_buf.
+ */
+static void unpin_rcv_pages(struct hfi1_filedata *fd,
+			    struct tid_user_buf *tidbuf,
+			    struct tid_rb_node *node,
+			    unsigned int idx,
+			    unsigned int npages,
+			    bool mapped)
+{
+	struct page **pages;
+	struct hfi1_devdata *dd = fd->uctxt->dd;
+
+	if (mapped) {
+		pci_unmap_single(dd->pcidev, node->dma_addr,
+				 node->mmu.len, PCI_DMA_FROMDEVICE);
+		pages = &node->pages[idx];
+	} else {
+		pages = &tidbuf->pages[idx];
+	}
+	hfi1_release_user_pages(fd->mm, pages, npages, mapped);
+	fd->tid_n_pinned -= npages;
+}
+
+/**
+ * Pin receive buffer pages.
+ */
+static int pin_rcv_pages(struct hfi1_filedata *fd, struct tid_user_buf *tidbuf)
+{
+	int pinned;
+	unsigned int npages;
+	unsigned long vaddr = tidbuf->vaddr;
+	struct page **pages = NULL;
+	struct hfi1_devdata *dd = fd->uctxt->dd;
+
+	/* Get the number of pages the user buffer spans */
+	npages = num_user_pages(vaddr, tidbuf->length);
+	if (!npages)
+		return -EINVAL;
+
+	if (npages > fd->uctxt->expected_count) {
+		dd_dev_err(dd, "Expected buffer too big\n");
+		return -EINVAL;
+	}
+
+	/* Verify that access is OK for the user buffer */
+	if (!access_ok(VERIFY_WRITE, (void __user *)vaddr,
+		       npages * PAGE_SIZE)) {
+		dd_dev_err(dd, "Fail vaddr %p, %u pages, !access_ok\n",
+			   (void *)vaddr, npages);
+		return -EFAULT;
+	}
+	/* Allocate the array of struct page pointers needed for pinning */
+	pages = kcalloc(npages, sizeof(*pages), GFP_KERNEL);
+	if (!pages)
+		return -ENOMEM;
+
+	/*
+	 * Pin all the pages of the user buffer. If we can't pin all the
+	 * pages, accept the amount pinned so far and program only that.
+	 * User space knows how to deal with partially programmed buffers.
+	 */
+	if (!hfi1_can_pin_pages(dd, fd->mm, fd->tid_n_pinned, npages)) {
+		kfree(pages);
+		return -ENOMEM;
+	}
+
+	pinned = hfi1_acquire_user_pages(fd->mm, vaddr, npages, true, pages);
+	if (pinned <= 0) {
+		kfree(pages);
+		return pinned;
+	}
+	tidbuf->pages = pages;
+	tidbuf->npages = npages;
+	fd->tid_n_pinned += pinned;
+	return pinned;
+}
+
 /*
  * RcvArray entry allocation for Expected Receives is done by the
  * following algorithm:
@@ -253,62 +340,33 @@ int hfi1_user_exp_rcv_setup(struct hfi1_filedata *fd,
 	int ret = 0, need_group = 0, pinned;
 	struct hfi1_ctxtdata *uctxt = fd->uctxt;
 	struct hfi1_devdata *dd = uctxt->dd;
-	unsigned npages, ngroups, pageidx = 0, pageset_count, npagesets,
+	unsigned int ngroups, pageidx = 0, pageset_count,
 		tididx = 0, mapped, mapped_pages = 0;
-	unsigned long vaddr = tinfo->vaddr;
-	struct page **pages = NULL;
 	u32 *tidlist = NULL;
-	struct tid_pageset *pagesets = NULL;
-
-	/* Get the number of pages the user buffer spans */
-	npages = num_user_pages(vaddr, tinfo->length);
-	if (!npages)
-		return -EINVAL;
-
-	if (npages > uctxt->expected_count) {
-		dd_dev_err(dd, "Expected buffer too big\n");
-		return -EINVAL;
-	}
-
-	/* Verify that access is OK for the user buffer */
-	if (!access_ok(VERIFY_WRITE, (void __user *)vaddr,
-		       npages * PAGE_SIZE)) {
-		dd_dev_err(dd, "Fail vaddr %p, %u pages, !access_ok\n",
-			   (void *)vaddr, npages);
-		return -EFAULT;
-	}
+	struct tid_user_buf *tidbuf;
 
-	pagesets = kcalloc(uctxt->expected_count, sizeof(*pagesets),
-			   GFP_KERNEL);
-	if (!pagesets)
+	tidbuf = kzalloc(sizeof(*tidbuf), GFP_KERNEL);
+	if (!tidbuf)
 		return -ENOMEM;
 
-	/* Allocate the array of struct page pointers needed for pinning */
-	pages = kcalloc(npages, sizeof(*pages), GFP_KERNEL);
-	if (!pages) {
-		ret = -ENOMEM;
-		goto bail;
-	}
-
-	/*
-	 * Pin all the pages of the user buffer. If we can't pin all the
-	 * pages, accept the amount pinned so far and program only that.
-	 * User space knows how to deal with partially programmed buffers.
-	 */
-	if (!hfi1_can_pin_pages(dd, fd->mm, fd->tid_n_pinned, npages)) {
-		ret = -ENOMEM;
-		goto bail;
+	tidbuf->vaddr = tinfo->vaddr;
+	tidbuf->length = tinfo->length;
+	tidbuf->psets = kcalloc(uctxt->expected_count, sizeof(*tidbuf->psets),
+				GFP_KERNEL);
+	if (!tidbuf->psets) {
+		kfree(tidbuf);
+		return -ENOMEM;
 	}
 
-	pinned = hfi1_acquire_user_pages(fd->mm, vaddr, npages, true, pages);
+	pinned = pin_rcv_pages(fd, tidbuf);
 	if (pinned <= 0) {
-		ret = pinned;
-		goto bail;
+		kfree(tidbuf->psets);
+		kfree(tidbuf);
+		return pinned;
 	}
-	fd->tid_n_pinned += npages;
 
 	/* Find sets of physically contiguous pages */
-	npagesets = find_phys_blocks(pages, pinned, pagesets);
+	tidbuf->n_psets = find_phys_blocks(tidbuf, pinned);
 
 	/*
 	 * We don't need to access this under a lock since tid_used is per
@@ -316,10 +374,10 @@ int hfi1_user_exp_rcv_setup(struct hfi1_filedata *fd,
 	 * and hfi1_user_exp_rcv_setup() at the same time.
 	 */
 	spin_lock(&fd->tid_lock);
-	if (fd->tid_used + npagesets > fd->tid_limit)
+	if (fd->tid_used + tidbuf->n_psets > fd->tid_limit)
 		pageset_count = fd->tid_limit - fd->tid_used;
 	else
-		pageset_count = npagesets;
+		pageset_count = tidbuf->n_psets;
 	spin_unlock(&fd->tid_lock);
 
 	if (!pageset_count)
@@ -347,9 +405,9 @@ int hfi1_user_exp_rcv_setup(struct hfi1_filedata *fd,
 		struct tid_group *grp =
 			tid_group_pop(&uctxt->tid_group_list);
 
-		ret = program_rcvarray(fd, vaddr, grp, pagesets,
+		ret = program_rcvarray(fd, tidbuf, grp,
 				       pageidx, dd->rcv_entries.group_size,
-				       pages, tidlist, &tididx, &mapped);
+				       tidlist, &tididx, &mapped);
 		/*
 		 * If there was a failure to program the RcvArray
 		 * entries for the entire group, reset the grp fields
@@ -393,8 +451,8 @@ int hfi1_user_exp_rcv_setup(struct hfi1_filedata *fd,
 			unsigned use = min_t(unsigned, pageset_count - pageidx,
 					     grp->size - grp->used);
 
-			ret = program_rcvarray(fd, vaddr, grp, pagesets,
-					       pageidx, use, pages, tidlist,
+			ret = program_rcvarray(fd, tidbuf, grp,
+					       pageidx, use, tidlist,
 					       &tididx, &mapped);
 			if (ret < 0) {
 				hfi1_cdbg(TID,
@@ -454,16 +512,14 @@ int hfi1_user_exp_rcv_setup(struct hfi1_filedata *fd,
 	 * If not everything was mapped (due to insufficient RcvArray entries,
 	 * for example), unpin all unmapped pages so we can pin them nex time.
 	 */
-	if (mapped_pages != pinned) {
-		hfi1_release_user_pages(fd->mm, &pages[mapped_pages],
-					pinned - mapped_pages,
-					false);
-		fd->tid_n_pinned -= pinned - mapped_pages;
-	}
+	if (mapped_pages != pinned)
+		unpin_rcv_pages(fd, tidbuf, NULL, mapped_pages,
+				(pinned - mapped_pages), false);
 bail:
-	kfree(pagesets);
-	kfree(pages);
+	kfree(tidbuf->psets);
 	kfree(tidlist);
+	kfree(tidbuf->pages);
+	kfree(tidbuf);
 	return ret > 0 ? 0 : ret;
 }
 
@@ -553,11 +609,12 @@ int hfi1_user_exp_rcv_invalid(struct hfi1_filedata *fd,
 	return ret;
 }
 
-static u32 find_phys_blocks(struct page **pages, unsigned npages,
-			    struct tid_pageset *list)
+static u32 find_phys_blocks(struct tid_user_buf *tidbuf, unsigned int npages)
 {
 	unsigned pagecount, pageidx, setcount = 0, i;
 	unsigned long pfn, this_pfn;
+	struct page **pages = tidbuf->pages;
+	struct tid_pageset *list = tidbuf->psets;
 
 	if (!npages)
 		return 0;
@@ -620,13 +677,13 @@ static u32 find_phys_blocks(struct page **pages, unsigned npages,
 /**
  * program_rcvarray() - program an RcvArray group with receive buffers
  * @fd: filedata pointer
- * @vaddr: starting user virtual address
+ * @tbuf: pointer to struct tid_user_buf that has the user buffer starting
+ *	  virtual address, buffer length, page pointers, pagesets (array of
+ *	  struct tid_pageset holding information on physically contiguous
+ *	  chunks from the user buffer), and other fields.
  * @grp: RcvArray group
- * @sets: array of struct tid_pageset holding information on physically
- *        contiguous chunks from the user buffer
  * @start: starting index into sets array
  * @count: number of struct tid_pageset's to program
- * @pages: an array of struct page * for the user buffer
  * @tidlist: the array of u32 elements when the information about the
  *           programmed RcvArray entries is to be encoded.
  * @tididx: starting offset into tidlist
@@ -644,11 +701,11 @@ static u32 find_phys_blocks(struct page **pages, unsigned npages,
  * -ENOMEM or -EFAULT on error from set_rcvarray_entry(), or
  * number of RcvArray entries programmed.
  */
-static int program_rcvarray(struct hfi1_filedata *fd, unsigned long vaddr,
+static int program_rcvarray(struct hfi1_filedata *fd, struct tid_user_buf *tbuf,
 			    struct tid_group *grp,
-			    struct tid_pageset *sets,
-			    unsigned start, u16 count, struct page **pages,
-			    u32 *tidlist, unsigned *tididx, unsigned *pmapped)
+			    unsigned int start, u16 count,
+			    u32 *tidlist, unsigned int *tididx,
+			    unsigned int *pmapped)
 {
 	struct hfi1_ctxtdata *uctxt = fd->uctxt;
 	struct hfi1_devdata *dd = uctxt->dd;
@@ -687,11 +744,11 @@ static int program_rcvarray(struct hfi1_filedata *fd, unsigned long vaddr,
 		}
 
 		rcventry = grp->base + useidx;
-		npages = sets[setidx].count;
-		pageidx = sets[setidx].idx;
+		npages = tbuf->psets[setidx].count;
+		pageidx = tbuf->psets[setidx].idx;
 
-		ret = set_rcvarray_entry(fd, vaddr + (pageidx * PAGE_SIZE),
-					 rcventry, grp, pages + pageidx,
+		ret = set_rcvarray_entry(fd, tbuf,
+					 rcventry, grp, pageidx,
 					 npages);
 		if (ret)
 			return ret;
@@ -712,15 +769,17 @@ static int program_rcvarray(struct hfi1_filedata *fd, unsigned long vaddr,
 	return idx;
 }
 
-static int set_rcvarray_entry(struct hfi1_filedata *fd, unsigned long vaddr,
+static int set_rcvarray_entry(struct hfi1_filedata *fd,
+			      struct tid_user_buf *tbuf,
 			      u32 rcventry, struct tid_group *grp,
-			      struct page **pages, unsigned npages)
+			      u16 pageidx, unsigned int npages)
 {
 	int ret;
 	struct hfi1_ctxtdata *uctxt = fd->uctxt;
 	struct tid_rb_node *node;
 	struct hfi1_devdata *dd = uctxt->dd;
 	dma_addr_t phys;
+	struct page **pages = tbuf->pages + pageidx;
 
 	/*
 	 * Allocate the node first so we can handle a potential
@@ -741,7 +800,7 @@ static int set_rcvarray_entry(struct hfi1_filedata *fd, unsigned long vaddr,
 		return -EFAULT;
 	}
 
-	node->mmu.addr = vaddr;
+	node->mmu.addr = tbuf->vaddr + (pageidx * PAGE_SIZE);
 	node->mmu.len = npages * PAGE_SIZE;
 	node->phys = page_to_phys(pages[0]);
 	node->npages = npages;
@@ -820,10 +879,7 @@ static void clear_tid_node(struct hfi1_filedata *fd, struct tid_rb_node *node)
 	 */
 	hfi1_put_tid(dd, node->rcventry, PT_INVALID_FLUSH, 0, 0);
 
-	pci_unmap_single(dd->pcidev, node->dma_addr, node->mmu.len,
-			 PCI_DMA_FROMDEVICE);
-	hfi1_release_user_pages(fd->mm, node->pages, node->npages, true);
-	fd->tid_n_pinned -= node->npages;
+	unpin_rcv_pages(fd, NULL, node, 0, node->npages, true);
 
 	node->grp->used--;
 	node->grp->map &= ~(1 << (node->rcventry - node->grp->base));
diff --git a/drivers/infiniband/hw/hfi1/user_exp_rcv.h b/drivers/infiniband/hw/hfi1/user_exp_rcv.h
index 6cbaa4c..8c4eb5d 100644
--- a/drivers/infiniband/hw/hfi1/user_exp_rcv.h
+++ b/drivers/infiniband/hw/hfi1/user_exp_rcv.h
@@ -51,6 +51,15 @@
 
 #include "exp_rcv.h"
 
+struct tid_user_buf {
+	unsigned long vaddr;
+	unsigned long length;
+	unsigned int npages;
+	struct page **pages;
+	struct tid_pageset *psets;
+	unsigned int n_psets;
+};
+
 int hfi1_user_exp_rcv_init(struct hfi1_filedata *fd,
 			   struct hfi1_ctxtdata *uctxt);
 void hfi1_user_exp_rcv_free(struct hfi1_filedata *fd);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 08/16] IB/hfi1: Clean up user_sdma_send_pkts() function

[Dennis Dalessandro]

From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

user_sdma_send_pkts() function is unnecessarily long. Clean it up by
moving some of its code into separate functions.

Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |  141 +++++++++++++++++++-------------
 1 files changed, 82 insertions(+), 59 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index 9b89df5..d5a2572 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -847,6 +847,84 @@ static inline u32 get_lrh_len(struct hfi1_pkt_header hdr, u32 len)
 	return ((sizeof(hdr) - sizeof(hdr.pbc)) + 4 + len);
 }
 
+static int user_sdma_txadd_ahg(struct user_sdma_request *req,
+			       struct user_sdma_txreq *tx,
+			       u32 datalen)
+{
+	int ret;
+	u16 pbclen = le16_to_cpu(req->hdr.pbc[0]);
+	u32 lrhlen = get_lrh_len(req->hdr, pad_len(datalen));
+	struct hfi1_user_sdma_pkt_q *pq = req->pq;
+
+	/*
+	 * Copy the request header into the tx header
+	 * because the HW needs a cacheline-aligned
+	 * address.
+	 * This copy can be optimized out if the hdr
+	 * member of user_sdma_request were also
+	 * cacheline aligned.
+	 */
+	memcpy(&tx->hdr, &req->hdr, sizeof(tx->hdr));
+	if (PBC2LRH(pbclen) != lrhlen) {
+		pbclen = (pbclen & 0xf000) | LRH2PBC(lrhlen);
+		tx->hdr.pbc[0] = cpu_to_le16(pbclen);
+	}
+	ret = check_header_template(req, &tx->hdr, lrhlen, datalen);
+	if (ret)
+		return ret;
+	ret = sdma_txinit_ahg(&tx->txreq, SDMA_TXREQ_F_AHG_COPY,
+			      sizeof(tx->hdr) + datalen, req->ahg_idx,
+			      0, NULL, 0, user_sdma_txreq_cb);
+	if (ret)
+		return ret;
+	ret = sdma_txadd_kvaddr(pq->dd, &tx->txreq, &tx->hdr, sizeof(tx->hdr));
+	if (ret)
+		sdma_txclean(pq->dd, &tx->txreq);
+	return ret;
+}
+
+static int user_sdma_txadd(struct user_sdma_request *req,
+			   struct user_sdma_txreq *tx,
+			   struct user_sdma_iovec *iovec, u32 datalen,
+			   u32 *queued_ptr, u32 *data_sent_ptr,
+			   u64 *iov_offset_ptr)
+{
+	int ret;
+	unsigned int pageidx, len;
+	unsigned long base, offset;
+	u64 iov_offset = *iov_offset_ptr;
+	u32 queued = *queued_ptr, data_sent = *data_sent_ptr;
+	struct hfi1_user_sdma_pkt_q *pq = req->pq;
+
+	base = (unsigned long)iovec->iov.iov_base;
+	offset = offset_in_page(base + iovec->offset + iov_offset);
+	pageidx = (((iovec->offset + iov_offset + base) - (base & PAGE_MASK)) >>
+		   PAGE_SHIFT);
+	len = offset + req->info.fragsize > PAGE_SIZE ?
+		PAGE_SIZE - offset : req->info.fragsize;
+	len = min((datalen - queued), len);
+	ret = sdma_txadd_page(pq->dd, &tx->txreq, iovec->pages[pageidx],
+			      offset, len);
+	if (ret) {
+		SDMA_DBG(req, "SDMA txreq add page failed %d\n", ret);
+		return ret;
+	}
+	iov_offset += len;
+	queued += len;
+	data_sent += len;
+	if (unlikely(queued < datalen && pageidx == iovec->npages &&
+		     req->iov_idx < req->data_iovs - 1)) {
+		iovec->offset += iov_offset;
+		iovec = &req->iovs[++req->iov_idx];
+		iov_offset = 0;
+	}
+
+	*queued_ptr = queued;
+	*data_sent_ptr = data_sent;
+	*iov_offset_ptr = iov_offset;
+	return ret;
+}
+
 static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 {
 	int ret = 0, count;
@@ -943,39 +1021,9 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 
 		if (req->ahg_idx >= 0) {
 			if (!req->seqnum) {
-				u16 pbclen = le16_to_cpu(req->hdr.pbc[0]);
-				u32 lrhlen = get_lrh_len(req->hdr,
-							 pad_len(datalen));
-				/*
-				 * Copy the request header into the tx header
-				 * because the HW needs a cacheline-aligned
-				 * address.
-				 * This copy can be optimized out if the hdr
-				 * member of user_sdma_request were also
-				 * cacheline aligned.
-				 */
-				memcpy(&tx->hdr, &req->hdr, sizeof(tx->hdr));
-				if (PBC2LRH(pbclen) != lrhlen) {
-					pbclen = (pbclen & 0xf000) |
-						LRH2PBC(lrhlen);
-					tx->hdr.pbc[0] = cpu_to_le16(pbclen);
-				}
-				ret = check_header_template(req, &tx->hdr,
-							    lrhlen, datalen);
-				if (ret)
-					goto free_tx;
-				ret = sdma_txinit_ahg(&tx->txreq,
-						      SDMA_TXREQ_F_AHG_COPY,
-						      sizeof(tx->hdr) + datalen,
-						      req->ahg_idx, 0, NULL, 0,
-						      user_sdma_txreq_cb);
+				ret = user_sdma_txadd_ahg(req, tx, datalen);
 				if (ret)
 					goto free_tx;
-				ret = sdma_txadd_kvaddr(pq->dd, &tx->txreq,
-							&tx->hdr,
-							sizeof(tx->hdr));
-				if (ret)
-					goto free_txreq;
 			} else {
 				int changes;
 
@@ -1006,35 +1054,10 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 		 */
 		while (queued < datalen &&
 		       (req->sent + data_sent) < req->data_len) {
-			unsigned long base, offset;
-			unsigned pageidx, len;
-
-			base = (unsigned long)iovec->iov.iov_base;
-			offset = offset_in_page(base + iovec->offset +
-						iov_offset);
-			pageidx = (((iovec->offset + iov_offset +
-				     base) - (base & PAGE_MASK)) >> PAGE_SHIFT);
-			len = offset + req->info.fragsize > PAGE_SIZE ?
-				PAGE_SIZE - offset : req->info.fragsize;
-			len = min((datalen - queued), len);
-			ret = sdma_txadd_page(pq->dd, &tx->txreq,
-					      iovec->pages[pageidx],
-					      offset, len);
-			if (ret) {
-				SDMA_DBG(req, "SDMA txreq add page failed %d\n",
-					 ret);
+			ret = user_sdma_txadd(req, tx, iovec, datalen,
+					      &queued, &data_sent, &iov_offset);
+			if (ret)
 				goto free_txreq;
-			}
-			iov_offset += len;
-			queued += len;
-			data_sent += len;
-			if (unlikely(queued < datalen &&
-				     pageidx == iovec->npages &&
-				     req->iov_idx < req->data_iovs - 1)) {
-				iovec->offset += iov_offset;
-				iovec = &req->iovs[++req->iov_idx];
-				iov_offset = 0;
-			}
 		}
 		/*
 		 * The txreq was submitted successfully so we can update

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 09/16] IB/hfi1: Clean up pin_vector_pages() function

[Dennis Dalessandro]

From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Clean up pin_vector_pages() function by moving page pinning related code
to a separate function since it really stands on its own.

Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |   79 ++++++++++++++++++--------------
 1 files changed, 45 insertions(+), 34 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index d5a2572..6f26253 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -1124,11 +1124,53 @@ static u32 sdma_cache_evict(struct hfi1_user_sdma_pkt_q *pq, u32 npages)
 	return evict_data.cleared;
 }
 
+static int pin_sdma_pages(struct user_sdma_request *req,
+			  struct user_sdma_iovec *iovec,
+			  struct sdma_mmu_node *node,
+			  int npages)
+{
+	int pinned, cleared;
+	struct page **pages;
+	struct hfi1_user_sdma_pkt_q *pq = req->pq;
+
+	pages = kcalloc(npages, sizeof(*pages), GFP_KERNEL);
+	if (!pages) {
+		SDMA_DBG(req, "Failed page array alloc");
+		return -ENOMEM;
+	}
+	memcpy(pages, node->pages, node->npages * sizeof(*pages));
+
+	npages -= node->npages;
+retry:
+	if (!hfi1_can_pin_pages(pq->dd, pq->mm,
+				atomic_read(&pq->n_locked), npages)) {
+		cleared = sdma_cache_evict(pq, npages);
+		if (cleared >= npages)
+			goto retry;
+	}
+	pinned = hfi1_acquire_user_pages(pq->mm,
+					 ((unsigned long)iovec->iov.iov_base +
+					 (node->npages * PAGE_SIZE)), npages, 0,
+					 pages + node->npages);
+	if (pinned < 0) {
+		kfree(pages);
+		return pinned;
+	}
+	if (pinned != npages) {
+		unpin_vector_pages(pq->mm, pages, node->npages, pinned);
+		return -EFAULT;
+	}
+	kfree(node->pages);
+	node->rb.len = iovec->iov.iov_len;
+	node->pages = pages;
+	atomic_add(pinned, &pq->n_locked);
+	return pinned;
+}
+
 static int pin_vector_pages(struct user_sdma_request *req,
 			    struct user_sdma_iovec *iovec)
 {
-	int ret = 0, pinned, npages, cleared;
-	struct page **pages;
+	int ret = 0, pinned, npages;
 	struct hfi1_user_sdma_pkt_q *pq = req->pq;
 	struct sdma_mmu_node *node = NULL;
 	struct mmu_rb_node *rb_node;
@@ -1162,44 +1204,13 @@ static int pin_vector_pages(struct user_sdma_request *req,
 
 	npages = num_user_pages(&iovec->iov);
 	if (node->npages < npages) {
-		pages = kcalloc(npages, sizeof(*pages), GFP_KERNEL);
-		if (!pages) {
-			SDMA_DBG(req, "Failed page array alloc");
-			ret = -ENOMEM;
-			goto bail;
-		}
-		memcpy(pages, node->pages, node->npages * sizeof(*pages));
-
-		npages -= node->npages;
-
-retry:
-		if (!hfi1_can_pin_pages(pq->dd, pq->mm,
-					atomic_read(&pq->n_locked), npages)) {
-			cleared = sdma_cache_evict(pq, npages);
-			if (cleared >= npages)
-				goto retry;
-		}
-		pinned = hfi1_acquire_user_pages(pq->mm,
-			((unsigned long)iovec->iov.iov_base +
-			 (node->npages * PAGE_SIZE)), npages, 0,
-			pages + node->npages);
+		pinned = pin_sdma_pages(req, iovec, node, npages);
 		if (pinned < 0) {
-			kfree(pages);
 			ret = pinned;
 			goto bail;
 		}
-		if (pinned != npages) {
-			unpin_vector_pages(pq->mm, pages, node->npages,
-					   pinned);
-			ret = -EFAULT;
-			goto bail;
-		}
-		kfree(node->pages);
-		node->rb.len = iovec->iov.iov_len;
-		node->pages = pages;
 		node->npages += pinned;
 		npages = node->npages;
-		atomic_add(pinned, &pq->n_locked);
 	}
 	iovec->pages = node->pages;
 	iovec->npages = npages;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 10/16] IB/hfi1: Fix the bail out code in pin_vector_pages() function

[Dennis Dalessandro]

From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

In pin_vector_pages() function, if there is any error while pinning the
pages or while adding a pinned buffer to the cache, the bail out code
needs to unpin any pinned pages that are not in the cache and adjust the
n_locked counter that counts the total pages pinned. The current bail
out code doesn't seem to be doing it right in two cases:

1. Before pinning required pages for a buffer, the SDMA pinned buffer
cache is searched to see if the virtual address range that needs to be
pinned is already pinned. If there isn't a hit in the cache, a new node
is created for the buffer and is added to the cache after the buffer is
pinned. If adding the new node to the cache fails, the n_locked count is
decremented properly but the pinned pages are not freed. This commit
fixes this issue.

2. If there is a hit in the SDMA cache, but the cached buffer doesn't
have enough pages to cover the entire address range that needs to be
pinned, the node for the cached buffer is extracted from the cache,
remaining pages needed are pinned and added to the node. The node is
finally added back into the cache. If there is an error pinning the
extra pages, the bail out code frees all the pages in the node but the
n_locked count is not being decremented by the no of pages in the node
that are freed. This commit fixes this issue.

This commit fixes the above two issues by creating a new function that
frees the pages in a node and decrements the n_locked count by the
number of pages freed.

Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |   17 ++++++++++-------
 1 files changed, 10 insertions(+), 7 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index 6f26253..a3a9925 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -1167,6 +1167,14 @@ static int pin_sdma_pages(struct user_sdma_request *req,
 	return pinned;
 }
 
+static void unpin_sdma_pages(struct sdma_mmu_node *node)
+{
+	if (node->npages) {
+		unpin_vector_pages(node->pq->mm, node->pages, 0, node->npages);
+		atomic_sub(node->npages, &node->pq->n_locked);
+	}
+}
+
 static int pin_vector_pages(struct user_sdma_request *req,
 			    struct user_sdma_iovec *iovec)
 {
@@ -1218,14 +1226,12 @@ static int pin_vector_pages(struct user_sdma_request *req,
 
 	ret = hfi1_mmu_rb_insert(req->pq->handler, &node->rb);
 	if (ret) {
-		atomic_sub(node->npages, &pq->n_locked);
 		iovec->node = NULL;
 		goto bail;
 	}
 	return 0;
 bail:
-	if (rb_node)
-		unpin_vector_pages(pq->mm, node->pages, 0, node->npages);
+	unpin_sdma_pages(node);
 	kfree(node);
 	return ret;
 }
@@ -1671,10 +1677,7 @@ static void sdma_rb_remove(void *arg, struct mmu_rb_node *mnode)
 	struct sdma_mmu_node *node =
 		container_of(mnode, struct sdma_mmu_node, rb);
 
-	atomic_sub(node->npages, &node->pq->n_locked);
-
-	unpin_vector_pages(node->pq->mm, node->pages, 0, node->npages);
-
+	unpin_sdma_pages(node);
 	kfree(node);
 }
 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 11/16] IB/hfi1: Remove duplicate definitions of num_user_pages() function

[Dennis Dalessandro]

From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

num_user_pages() function has been defined in both user_exp_rcv.c file
and user_sdma.c file. Move the function definition to a header file so
there is only one definition in the source repo.

Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/user_exp_rcv.c |    5 -----
 drivers/infiniband/hw/hfi1/user_exp_rcv.h |    9 +++++++++
 drivers/infiniband/hw/hfi1/user_sdma.c    |   18 +++---------------
 3 files changed, 12 insertions(+), 20 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/user_exp_rcv.c b/drivers/infiniband/hw/hfi1/user_exp_rcv.c
index 04be178..026790e 100644
--- a/drivers/infiniband/hw/hfi1/user_exp_rcv.c
+++ b/drivers/infiniband/hw/hfi1/user_exp_rcv.c
@@ -67,11 +67,6 @@ struct tid_pageset {
 	u16 count;
 };
 
-#define num_user_pages(vaddr, len)				       \
-	(1 + (((((unsigned long)(vaddr) +			       \
-		 (unsigned long)(len) - 1) & PAGE_MASK) -	       \
-	       ((unsigned long)vaddr & PAGE_MASK)) >> PAGE_SHIFT))
-
 static void unlock_exp_tids(struct hfi1_ctxtdata *uctxt,
 			    struct exp_tid_set *set,
 			    struct hfi1_filedata *fd);
diff --git a/drivers/infiniband/hw/hfi1/user_exp_rcv.h b/drivers/infiniband/hw/hfi1/user_exp_rcv.h
index 8c4eb5d..7461d11 100644
--- a/drivers/infiniband/hw/hfi1/user_exp_rcv.h
+++ b/drivers/infiniband/hw/hfi1/user_exp_rcv.h
@@ -60,6 +60,15 @@ struct tid_user_buf {
 	unsigned int n_psets;
 };
 
+static inline int num_user_pages(unsigned long addr,
+				 unsigned long len)
+{
+	const unsigned long spage = addr & PAGE_MASK;
+	const unsigned long epage = (addr + len - 1) & PAGE_MASK;
+
+	return 1 + ((epage - spage) >> PAGE_SHIFT);
+}
+
 int hfi1_user_exp_rcv_init(struct hfi1_filedata *fd,
 			   struct hfi1_ctxtdata *uctxt);
 void hfi1_user_exp_rcv_free(struct hfi1_filedata *fd);
diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index a3a9925..2837407 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -246,7 +246,6 @@ struct user_sdma_txreq {
 
 static int user_sdma_send_pkts(struct user_sdma_request *req,
 			       unsigned maxpkts);
-static int num_user_pages(const struct iovec *iov);
 static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status);
 static inline void pq_update(struct hfi1_user_sdma_pkt_q *pq);
 static void user_sdma_free_request(struct user_sdma_request *req, bool unpin);
@@ -1101,19 +1100,6 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 	return ret;
 }
 
-/*
- * How many pages in this iovec element?
- */
-static inline int num_user_pages(const struct iovec *iov)
-{
-	const unsigned long addr  = (unsigned long)iov->iov_base;
-	const unsigned long len   = iov->iov_len;
-	const unsigned long spage = addr & PAGE_MASK;
-	const unsigned long epage = (addr + len - 1) & PAGE_MASK;
-
-	return 1 + ((epage - spage) >> PAGE_SHIFT);
-}
-
 static u32 sdma_cache_evict(struct hfi1_user_sdma_pkt_q *pq, u32 npages)
 {
 	struct evict_data evict_data;
@@ -1182,6 +1168,7 @@ static int pin_vector_pages(struct user_sdma_request *req,
 	struct hfi1_user_sdma_pkt_q *pq = req->pq;
 	struct sdma_mmu_node *node = NULL;
 	struct mmu_rb_node *rb_node;
+	struct iovec *iov;
 	bool extracted;
 
 	extracted =
@@ -1210,7 +1197,8 @@ static int pin_vector_pages(struct user_sdma_request *req,
 		atomic_set(&node->refcount, 0);
 	}
 
-	npages = num_user_pages(&iovec->iov);
+	iov = &iovec->iov;
+	npages = num_user_pages((unsigned long)iov->iov_base, iov->iov_len);
 	if (node->npages < npages) {
 		pinned = pin_sdma_pages(req, iovec, node, npages);
 		if (pinned < 0) {

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 12/16] IB/hfi1: Move structure definitions from user_exp_rcv.c to user_exp_rcv.h

[Dennis Dalessandro]

From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Clean up user_exp_rcv.c file by moving structure definitions into header
file user_exp_rcv.h. Since these structure definitions depend on the
structure definitions in mmu_rb.h, move #include "mmu_rb.h" above
the include "user_exp_rcv.h" or include of header files that include
user_exp_rcv.h

Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/file_ops.c     |    2 +-
 drivers/infiniband/hw/hfi1/user_exp_rcv.c |   18 +-----------------
 drivers/infiniband/hw/hfi1/user_exp_rcv.h |   16 ++++++++++++++++
 drivers/infiniband/hw/hfi1/user_sdma.c    |    2 +-
 4 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c
index ab8eb2b..e0fd8fc 100644
--- a/drivers/infiniband/hw/hfi1/file_ops.c
+++ b/drivers/infiniband/hw/hfi1/file_ops.c
@@ -58,10 +58,10 @@
 #include "device.h"
 #include "common.h"
 #include "trace.h"
+#include "mmu_rb.h"
 #include "user_sdma.h"
 #include "user_exp_rcv.h"
 #include "aspm.h"
-#include "mmu_rb.h"
 
 #undef pr_fmt
 #define pr_fmt(fmt) DRIVER_NAME ": " fmt
diff --git a/drivers/infiniband/hw/hfi1/user_exp_rcv.c b/drivers/infiniband/hw/hfi1/user_exp_rcv.c
index 026790e..6f6c14d 100644
--- a/drivers/infiniband/hw/hfi1/user_exp_rcv.c
+++ b/drivers/infiniband/hw/hfi1/user_exp_rcv.c
@@ -47,25 +47,9 @@
 #include <asm/page.h>
 #include <linux/string.h>
 
+#include "mmu_rb.h"
 #include "user_exp_rcv.h"
 #include "trace.h"
-#include "mmu_rb.h"
-
-struct tid_rb_node {
-	struct mmu_rb_node mmu;
-	unsigned long phys;
-	struct tid_group *grp;
-	u32 rcventry;
-	dma_addr_t dma_addr;
-	bool freed;
-	unsigned npages;
-	struct page *pages[0];
-};
-
-struct tid_pageset {
-	u16 idx;
-	u16 count;
-};
 
 static void unlock_exp_tids(struct hfi1_ctxtdata *uctxt,
 			    struct exp_tid_set *set,
diff --git a/drivers/infiniband/hw/hfi1/user_exp_rcv.h b/drivers/infiniband/hw/hfi1/user_exp_rcv.h
index 7461d11..e383cc0 100644
--- a/drivers/infiniband/hw/hfi1/user_exp_rcv.h
+++ b/drivers/infiniband/hw/hfi1/user_exp_rcv.h
@@ -51,6 +51,11 @@
 
 #include "exp_rcv.h"
 
+struct tid_pageset {
+	u16 idx;
+	u16 count;
+};
+
 struct tid_user_buf {
 	unsigned long vaddr;
 	unsigned long length;
@@ -60,6 +65,17 @@ struct tid_user_buf {
 	unsigned int n_psets;
 };
 
+struct tid_rb_node {
+	struct mmu_rb_node mmu;
+	unsigned long phys;
+	struct tid_group *grp;
+	u32 rcventry;
+	dma_addr_t dma_addr;
+	bool freed;
+	unsigned int npages;
+	struct page *pages[0];
+};
+
 static inline int num_user_pages(unsigned long addr,
 				 unsigned long len)
 {
diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index 2837407..8a1653a 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -64,11 +64,11 @@
 
 #include "hfi.h"
 #include "sdma.h"
+#include "mmu_rb.h"
 #include "user_sdma.h"
 #include "verbs.h"  /* for the headers */
 #include "common.h" /* for struct hfi1_tid_info */
 #include "trace.h"
-#include "mmu_rb.h"
 
 static uint hfi1_sdma_comp_ring_size = 128;
 module_param_named(sdma_comp_size, hfi1_sdma_comp_ring_size, uint, S_IRUGO);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 13/16] IB/hfi1: Move structure and MACRO definitions in user_sdma.c to user_sdma.h

[Dennis Dalessandro]

From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Clean up user_sdma.c by moving the structure and MACRO definitions into
the header file user_sdma.h

Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |  168 --------------------------------
 drivers/infiniband/hw/hfi1/user_sdma.h |  166 ++++++++++++++++++++++++++++++++
 2 files changed, 166 insertions(+), 168 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index 8a1653a..dacb0fc 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -74,176 +74,8 @@
 module_param_named(sdma_comp_size, hfi1_sdma_comp_ring_size, uint, S_IRUGO);
 MODULE_PARM_DESC(sdma_comp_size, "Size of User SDMA completion ring. Default: 128");
 
-/* The maximum number of Data io vectors per message/request */
-#define MAX_VECTORS_PER_REQ 8
-/*
- * Maximum number of packet to send from each message/request
- * before moving to the next one.
- */
-#define MAX_PKTS_PER_QUEUE 16
-
-#define num_pages(x) (1 + ((((x) - 1) & PAGE_MASK) >> PAGE_SHIFT))
-
-#define req_opcode(x) \
-	(((x) >> HFI1_SDMA_REQ_OPCODE_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
-#define req_version(x) \
-	(((x) >> HFI1_SDMA_REQ_VERSION_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
-#define req_iovcnt(x) \
-	(((x) >> HFI1_SDMA_REQ_IOVCNT_SHIFT) & HFI1_SDMA_REQ_IOVCNT_MASK)
-
-/* Number of BTH.PSN bits used for sequence number in expected rcvs */
-#define BTH_SEQ_MASK 0x7ffull
-
-#define AHG_KDETH_INTR_SHIFT 12
-#define AHG_KDETH_SH_SHIFT   13
-#define AHG_KDETH_ARRAY_SIZE  9
-
-#define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
-#define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)
-
-#define AHG_HEADER_SET(arr, idx, dw, bit, width, value)			\
-	do {								\
-		if ((idx) < ARRAY_SIZE((arr)))				\
-			(arr)[(idx++)] = sdma_build_ahg_descriptor(	\
-				(__force u16)(value), (dw), (bit),	\
-							(width));	\
-		else							\
-			return -ERANGE;					\
-	} while (0)
-
-/* Tx request flag bits */
-#define TXREQ_FLAGS_REQ_ACK   BIT(0)      /* Set the ACK bit in the header */
-#define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
-
-#define SDMA_PKT_Q_INACTIVE BIT(0)
-#define SDMA_PKT_Q_ACTIVE   BIT(1)
-#define SDMA_PKT_Q_DEFERRED BIT(2)
-
-/*
- * Maximum retry attempts to submit a TX request
- * before putting the process to sleep.
- */
-#define MAX_DEFER_RETRY_COUNT 1
-
 static unsigned initial_pkt_count = 8;
 
-#define SDMA_IOWAIT_TIMEOUT 1000 /* in milliseconds */
-
-struct sdma_mmu_node;
-
-struct user_sdma_iovec {
-	struct list_head list;
-	struct iovec iov;
-	/* number of pages in this vector */
-	unsigned npages;
-	/* array of pinned pages for this vector */
-	struct page **pages;
-	/*
-	 * offset into the virtual address space of the vector at
-	 * which we last left off.
-	 */
-	u64 offset;
-	struct sdma_mmu_node *node;
-};
-
-struct sdma_mmu_node {
-	struct mmu_rb_node rb;
-	struct hfi1_user_sdma_pkt_q *pq;
-	atomic_t refcount;
-	struct page **pages;
-	unsigned npages;
-};
-
-/* evict operation argument */
-struct evict_data {
-	u32 cleared;	/* count evicted so far */
-	u32 target;	/* target count to evict */
-};
-
-struct user_sdma_request {
-	/* This is the original header from user space */
-	struct hfi1_pkt_header hdr;
-
-	/* Read mostly fields */
-	struct hfi1_user_sdma_pkt_q *pq ____cacheline_aligned_in_smp;
-	struct hfi1_user_sdma_comp_q *cq;
-	/*
-	 * Pointer to the SDMA engine for this request.
-	 * Since different request could be on different VLs,
-	 * each request will need it's own engine pointer.
-	 */
-	struct sdma_engine *sde;
-	struct sdma_req_info info;
-	/* TID array values copied from the tid_iov vector */
-	u32 *tids;
-	/* total length of the data in the request */
-	u32 data_len;
-	/* number of elements copied to the tids array */
-	u16 n_tids;
-	/*
-	 * We copy the iovs for this request (based on
-	 * info.iovcnt). These are only the data vectors
-	 */
-	u8 data_iovs;
-	s8 ahg_idx;
-
-	/* Writeable fields shared with interrupt */
-	u64 seqcomp ____cacheline_aligned_in_smp;
-	u64 seqsubmitted;
-	/* status of the last txreq completed */
-	int status;
-
-	/* Send side fields */
-	struct list_head txps ____cacheline_aligned_in_smp;
-	u64 seqnum;
-	/*
-	 * KDETH.OFFSET (TID) field
-	 * The offset can cover multiple packets, depending on the
-	 * size of the TID entry.
-	 */
-	u32 tidoffset;
-	/*
-	 * KDETH.Offset (Eager) field
-	 * We need to remember the initial value so the headers
-	 * can be updated properly.
-	 */
-	u32 koffset;
-	u32 sent;
-	/* TID index copied from the tid_iov vector */
-	u16 tididx;
-	/* progress index moving along the iovs array */
-	u8 iov_idx;
-	u8 done;
-	u8 has_error;
-
-	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
-} ____cacheline_aligned_in_smp;
-
-/*
- * A single txreq could span up to 3 physical pages when the MTU
- * is sufficiently large (> 4K). Each of the IOV pointers also
- * needs it's own set of flags so the vector has been handled
- * independently of each other.
- */
-struct user_sdma_txreq {
-	/* Packet header for the txreq */
-	struct hfi1_pkt_header hdr;
-	struct sdma_txreq txreq;
-	struct list_head list;
-	struct user_sdma_request *req;
-	u16 flags;
-	unsigned busycount;
-	u64 seqnum;
-};
-
-#define SDMA_DBG(req, fmt, ...)				     \
-	hfi1_cdbg(SDMA, "[%u:%u:%u:%u] " fmt, (req)->pq->dd->unit, \
-		 (req)->pq->ctxt, (req)->pq->subctxt, (req)->info.comp_idx, \
-		 ##__VA_ARGS__)
-#define SDMA_Q_DBG(pq, fmt, ...)			 \
-	hfi1_cdbg(SDMA, "[%u:%u:%u] " fmt, (pq)->dd->unit, (pq)->ctxt, \
-		 (pq)->subctxt, ##__VA_ARGS__)
-
 static int user_sdma_send_pkts(struct user_sdma_request *req,
 			       unsigned maxpkts);
 static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status);
diff --git a/drivers/infiniband/hw/hfi1/user_sdma.h b/drivers/infiniband/hw/hfi1/user_sdma.h
index 84c199d..6c10484 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.h
+++ b/drivers/infiniband/hw/hfi1/user_sdma.h
@@ -53,6 +53,67 @@
 #include "iowait.h"
 #include "user_exp_rcv.h"
 
+/* The maximum number of Data io vectors per message/request */
+#define MAX_VECTORS_PER_REQ 8
+/*
+ * Maximum number of packet to send from each message/request
+ * before moving to the next one.
+ */
+#define MAX_PKTS_PER_QUEUE 16
+
+#define num_pages(x) (1 + ((((x) - 1) & PAGE_MASK) >> PAGE_SHIFT))
+
+#define req_opcode(x) \
+	(((x) >> HFI1_SDMA_REQ_OPCODE_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
+#define req_version(x) \
+	(((x) >> HFI1_SDMA_REQ_VERSION_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
+#define req_iovcnt(x) \
+	(((x) >> HFI1_SDMA_REQ_IOVCNT_SHIFT) & HFI1_SDMA_REQ_IOVCNT_MASK)
+
+/* Number of BTH.PSN bits used for sequence number in expected rcvs */
+#define BTH_SEQ_MASK 0x7ffull
+
+#define AHG_KDETH_INTR_SHIFT 12
+#define AHG_KDETH_SH_SHIFT   13
+#define AHG_KDETH_ARRAY_SIZE  9
+
+#define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
+#define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)
+
+#define AHG_HEADER_SET(arr, idx, dw, bit, width, value)			\
+	do {								\
+		if ((idx) < ARRAY_SIZE((arr)))				\
+			(arr)[(idx++)] = sdma_build_ahg_descriptor(	\
+				(__force u16)(value), (dw), (bit),	\
+							(width));	\
+		else							\
+			return -ERANGE;					\
+	} while (0)
+
+/* Tx request flag bits */
+#define TXREQ_FLAGS_REQ_ACK   BIT(0)      /* Set the ACK bit in the header */
+#define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
+
+#define SDMA_PKT_Q_INACTIVE BIT(0)
+#define SDMA_PKT_Q_ACTIVE   BIT(1)
+#define SDMA_PKT_Q_DEFERRED BIT(2)
+
+/*
+ * Maximum retry attempts to submit a TX request
+ * before putting the process to sleep.
+ */
+#define MAX_DEFER_RETRY_COUNT 1
+
+#define SDMA_IOWAIT_TIMEOUT 1000 /* in milliseconds */
+
+#define SDMA_DBG(req, fmt, ...)				     \
+	hfi1_cdbg(SDMA, "[%u:%u:%u:%u] " fmt, (req)->pq->dd->unit, \
+		 (req)->pq->ctxt, (req)->pq->subctxt, (req)->info.comp_idx, \
+		 ##__VA_ARGS__)
+#define SDMA_Q_DBG(pq, fmt, ...)			 \
+	hfi1_cdbg(SDMA, "[%u:%u:%u] " fmt, (pq)->dd->unit, (pq)->ctxt, \
+		 (pq)->subctxt, ##__VA_ARGS__)
+
 extern uint extended_psn;
 
 struct hfi1_user_sdma_pkt_q {
@@ -79,6 +140,111 @@ struct hfi1_user_sdma_comp_q {
 	struct hfi1_sdma_comp_entry *comps;
 };
 
+struct sdma_mmu_node {
+	struct mmu_rb_node rb;
+	struct hfi1_user_sdma_pkt_q *pq;
+	atomic_t refcount;
+	struct page **pages;
+	unsigned int npages;
+};
+
+struct user_sdma_iovec {
+	struct list_head list;
+	struct iovec iov;
+	/* number of pages in this vector */
+	unsigned int npages;
+	/* array of pinned pages for this vector */
+	struct page **pages;
+	/*
+	 * offset into the virtual address space of the vector at
+	 * which we last left off.
+	 */
+	u64 offset;
+	struct sdma_mmu_node *node;
+};
+
+/* evict operation argument */
+struct evict_data {
+	u32 cleared;	/* count evicted so far */
+	u32 target;	/* target count to evict */
+};
+
+struct user_sdma_request {
+	/* This is the original header from user space */
+	struct hfi1_pkt_header hdr;
+
+	/* Read mostly fields */
+	struct hfi1_user_sdma_pkt_q *pq ____cacheline_aligned_in_smp;
+	struct hfi1_user_sdma_comp_q *cq;
+	/*
+	 * Pointer to the SDMA engine for this request.
+	 * Since different request could be on different VLs,
+	 * each request will need it's own engine pointer.
+	 */
+	struct sdma_engine *sde;
+	struct sdma_req_info info;
+	/* TID array values copied from the tid_iov vector */
+	u32 *tids;
+	/* total length of the data in the request */
+	u32 data_len;
+	/* number of elements copied to the tids array */
+	u16 n_tids;
+	/*
+	 * We copy the iovs for this request (based on
+	 * info.iovcnt). These are only the data vectors
+	 */
+	u8 data_iovs;
+	s8 ahg_idx;
+
+	/* Writeable fields shared with interrupt */
+	u64 seqcomp ____cacheline_aligned_in_smp;
+	u64 seqsubmitted;
+	/* status of the last txreq completed */
+	int status;
+
+	/* Send side fields */
+	struct list_head txps ____cacheline_aligned_in_smp;
+	u64 seqnum;
+	/*
+	 * KDETH.OFFSET (TID) field
+	 * The offset can cover multiple packets, depending on the
+	 * size of the TID entry.
+	 */
+	u32 tidoffset;
+	/*
+	 * KDETH.Offset (Eager) field
+	 * We need to remember the initial value so the headers
+	 * can be updated properly.
+	 */
+	u32 koffset;
+	u32 sent;
+	/* TID index copied from the tid_iov vector */
+	u16 tididx;
+	/* progress index moving along the iovs array */
+	u8 iov_idx;
+	u8 done;
+	u8 has_error;
+
+	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
+} ____cacheline_aligned_in_smp;
+
+/*
+ * A single txreq could span up to 3 physical pages when the MTU
+ * is sufficiently large (> 4K). Each of the IOV pointers also
+ * needs it's own set of flags so the vector has been handled
+ * independently of each other.
+ */
+struct user_sdma_txreq {
+	/* Packet header for the txreq */
+	struct hfi1_pkt_header hdr;
+	struct sdma_txreq txreq;
+	struct list_head list;
+	struct user_sdma_request *req;
+	u16 flags;
+	unsigned int busycount;
+	u64 seqnum;
+};
+
 int hfi1_user_sdma_alloc_queues(struct hfi1_ctxtdata *uctxt,
 				struct hfi1_filedata *fd);
 int hfi1_user_sdma_free_queues(struct hfi1_filedata *fd,

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 14/16] IB/hfi1: Fix whitespace alignment issue for MAD

[Dennis Dalessandro]

From: Kamenee Arumugam <kamenee.arumugam-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Fix a tab alignment issue present in pr_err_ratelimited
error message.

Reviewed-by: Michael J. Ruhl <michael.j.ruhl-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Kamenee Arumugam <kamenee.arumugam-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/mad.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/mad.c b/drivers/infiniband/hw/hfi1/mad.c
index 9336e78..97aee09 100644
--- a/drivers/infiniband/hw/hfi1/mad.c
+++ b/drivers/infiniband/hw/hfi1/mad.c
@@ -158,7 +158,7 @@ static void cleanup_traps(struct hfi1_ibport *ibp, struct trap_node *trap)
 	if (queue_id >= RVT_MAX_TRAP_LISTS) {
 		trap_count++;
 		pr_err_ratelimited("hfi1: Invalid trap 0x%0x dropped. Total dropped: %d\n",
-				  trap->data.generic_type, trap_count);
+				   trap->data.generic_type, trap_count);
 		kfree(trap);
 		return NULL;
 	}

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 15/16] IB/hfi1: Add received request info to qp_stats

[Dennis Dalessandro]

From: Kaike Wan <kaike.wan-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

The rvt_ack_entry pointed to by s_tail_ack_queue provides important
info about the request that has just been processed or is being processed
on the responder side of a RC connection. This patch adds this info to
the qp_stats to assist debugging.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Kaike Wan <kaike.wan-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/qp.c |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/qp.c b/drivers/infiniband/hw/hfi1/qp.c
index 0fca6df..ee836c0 100644
--- a/drivers/infiniband/hw/hfi1/qp.c
+++ b/drivers/infiniband/hw/hfi1/qp.c
@@ -626,12 +626,15 @@ void qp_iter_print(struct seq_file *s, struct qp_iter *iter)
 	struct hfi1_qp_priv *priv = qp->priv;
 	struct sdma_engine *sde;
 	struct send_context *send_context;
+	struct rvt_ack_entry *e = NULL;
 
 	sde = qp_to_sdma_engine(qp, priv->s_sc);
 	wqe = rvt_get_swqe_ptr(qp, qp->s_last);
 	send_context = qp_to_send_context(qp, priv->s_sc);
+	if (qp->s_ack_queue)
+		e = &qp->s_ack_queue[qp->s_tail_ack_queue];
 	seq_printf(s,
-		   "N %d %s QP %x R %u %s %u %u %u f=%x %u %u %u %u %u %u SPSN %x %x %x %x %x RPSN %x S(%u %u %u %u %u %u %u) R(%u %u %u) RQP %x LID %x SL %u MTU %u %u %u %u %u SDE %p,%u SC %p,%u SCQ %u %u PID %d\n",
+		   "N %d %s QP %x R %u %s %u %u %u f=%x %u %u %u %u %u %u SPSN %x %x %x %x %x RPSN %x S(%u %u %u %u %u %u %u) R(%u %u %u) RQP %x LID %x SL %u MTU %u %u %u %u %u SDE %p,%u SC %p,%u SCQ %u %u PID %d E %x %x %x\n",
 		   iter->n,
 		   qp_idle(qp) ? "I" : "B",
 		   qp->ibqp.qp_num,
@@ -672,7 +675,11 @@ void qp_iter_print(struct seq_file *s, struct qp_iter *iter)
 		   send_context ? send_context->sw_index : 0,
 		   ibcq_to_rvtcq(qp->ibqp.send_cq)->queue->head,
 		   ibcq_to_rvtcq(qp->ibqp.send_cq)->queue->tail,
-		   qp->pid);
+		   qp->pid,
+		   /* ack queue information */
+		   e ? e->opcode : 0,
+		   e ? e->psn : 0,
+		   e ? e->lpsn : 0);
 }
 
 void *qp_priv_alloc(struct rvt_dev_info *rdi, struct rvt_qp *qp)

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 16/16] IB/hfi1: Add opcode states to qp_stats

[Dennis Dalessandro]

From: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

These fields allow for debugging send engine processing.

Reviewed-by: Kaike Wan <kaike.wan-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/qp.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/qp.c b/drivers/infiniband/hw/hfi1/qp.c
index ee836c0..b87889d 100644
--- a/drivers/infiniband/hw/hfi1/qp.c
+++ b/drivers/infiniband/hw/hfi1/qp.c
@@ -634,7 +634,7 @@ void qp_iter_print(struct seq_file *s, struct qp_iter *iter)
 	if (qp->s_ack_queue)
 		e = &qp->s_ack_queue[qp->s_tail_ack_queue];
 	seq_printf(s,
-		   "N %d %s QP %x R %u %s %u %u %u f=%x %u %u %u %u %u %u SPSN %x %x %x %x %x RPSN %x S(%u %u %u %u %u %u %u) R(%u %u %u) RQP %x LID %x SL %u MTU %u %u %u %u %u SDE %p,%u SC %p,%u SCQ %u %u PID %d E %x %x %x\n",
+		   "N %d %s QP %x R %u %s %u %u %u f=%x %u %u %u %u %u %u SPSN %x %x %x %x %x RPSN %x S(%u %u %u %u %u %u %u) R(%u %u %u) RQP %x LID %x SL %u MTU %u %u %u %u %u SDE %p,%u SC %p,%u SCQ %u %u PID %d OS %x %x E %x %x %x\n",
 		   iter->n,
 		   qp_idle(qp) ? "I" : "B",
 		   qp->ibqp.qp_num,
@@ -676,6 +676,8 @@ void qp_iter_print(struct seq_file *s, struct qp_iter *iter)
 		   ibcq_to_rvtcq(qp->ibqp.send_cq)->queue->head,
 		   ibcq_to_rvtcq(qp->ibqp.send_cq)->queue->tail,
 		   qp->pid,
+		   qp->s_state,
+		   qp->s_ack_state,
 		   /* ack queue information */
 		   e ? e->opcode : 0,
 		   e ? e->psn : 0,

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-next 13/16] IB/hfi1: Move structure and MACRO definitions in user_sdma.c to user_sdma.h

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 13113 bytes --]

On Mon, Aug 21, 2017 at 06:27:29PM -0700, Dennis Dalessandro wrote:
> From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
> Clean up user_sdma.c by moving the structure and MACRO definitions into
> the header file user_sdma.h
>
> Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>  drivers/infiniband/hw/hfi1/user_sdma.c |  168 --------------------------------
>  drivers/infiniband/hw/hfi1/user_sdma.h |  166 ++++++++++++++++++++++++++++++++
>  2 files changed, 166 insertions(+), 168 deletions(-)
>
> diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
> index 8a1653a..dacb0fc 100644
> --- a/drivers/infiniband/hw/hfi1/user_sdma.c
> +++ b/drivers/infiniband/hw/hfi1/user_sdma.c
> @@ -74,176 +74,8 @@
>  module_param_named(sdma_comp_size, hfi1_sdma_comp_ring_size, uint, S_IRUGO);
>  MODULE_PARM_DESC(sdma_comp_size, "Size of User SDMA completion ring. Default: 128");
>
> -/* The maximum number of Data io vectors per message/request */
> -#define MAX_VECTORS_PER_REQ 8
> -/*
> - * Maximum number of packet to send from each message/request
> - * before moving to the next one.
> - */
> -#define MAX_PKTS_PER_QUEUE 16
> -
> -#define num_pages(x) (1 + ((((x) - 1) & PAGE_MASK) >> PAGE_SHIFT))
> -
> -#define req_opcode(x) \
> -	(((x) >> HFI1_SDMA_REQ_OPCODE_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
> -#define req_version(x) \
> -	(((x) >> HFI1_SDMA_REQ_VERSION_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
> -#define req_iovcnt(x) \
> -	(((x) >> HFI1_SDMA_REQ_IOVCNT_SHIFT) & HFI1_SDMA_REQ_IOVCNT_MASK)
> -
> -/* Number of BTH.PSN bits used for sequence number in expected rcvs */
> -#define BTH_SEQ_MASK 0x7ffull
> -
> -#define AHG_KDETH_INTR_SHIFT 12
> -#define AHG_KDETH_SH_SHIFT   13
> -#define AHG_KDETH_ARRAY_SIZE  9
> -
> -#define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
> -#define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)
> -
> -#define AHG_HEADER_SET(arr, idx, dw, bit, width, value)			\
> -	do {								\
> -		if ((idx) < ARRAY_SIZE((arr)))				\
> -			(arr)[(idx++)] = sdma_build_ahg_descriptor(	\
> -				(__force u16)(value), (dw), (bit),	\
> -							(width));	\
> -		else							\
> -			return -ERANGE;					\
> -	} while (0)
> -
> -/* Tx request flag bits */
> -#define TXREQ_FLAGS_REQ_ACK   BIT(0)      /* Set the ACK bit in the header */
> -#define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
> -
> -#define SDMA_PKT_Q_INACTIVE BIT(0)
> -#define SDMA_PKT_Q_ACTIVE   BIT(1)
> -#define SDMA_PKT_Q_DEFERRED BIT(2)
> -
> -/*
> - * Maximum retry attempts to submit a TX request
> - * before putting the process to sleep.
> - */
> -#define MAX_DEFER_RETRY_COUNT 1
> -
>  static unsigned initial_pkt_count = 8;
>
> -#define SDMA_IOWAIT_TIMEOUT 1000 /* in milliseconds */
> -
> -struct sdma_mmu_node;
> -
> -struct user_sdma_iovec {
> -	struct list_head list;
> -	struct iovec iov;
> -	/* number of pages in this vector */
> -	unsigned npages;
> -	/* array of pinned pages for this vector */
> -	struct page **pages;
> -	/*
> -	 * offset into the virtual address space of the vector at
> -	 * which we last left off.
> -	 */
> -	u64 offset;
> -	struct sdma_mmu_node *node;
> -};
> -
> -struct sdma_mmu_node {
> -	struct mmu_rb_node rb;
> -	struct hfi1_user_sdma_pkt_q *pq;
> -	atomic_t refcount;
> -	struct page **pages;
> -	unsigned npages;
> -};
> -
> -/* evict operation argument */
> -struct evict_data {
> -	u32 cleared;	/* count evicted so far */
> -	u32 target;	/* target count to evict */
> -};
> -
> -struct user_sdma_request {
> -	/* This is the original header from user space */
> -	struct hfi1_pkt_header hdr;
> -
> -	/* Read mostly fields */
> -	struct hfi1_user_sdma_pkt_q *pq ____cacheline_aligned_in_smp;
> -	struct hfi1_user_sdma_comp_q *cq;
> -	/*
> -	 * Pointer to the SDMA engine for this request.
> -	 * Since different request could be on different VLs,
> -	 * each request will need it's own engine pointer.
> -	 */
> -	struct sdma_engine *sde;
> -	struct sdma_req_info info;
> -	/* TID array values copied from the tid_iov vector */
> -	u32 *tids;
> -	/* total length of the data in the request */
> -	u32 data_len;
> -	/* number of elements copied to the tids array */
> -	u16 n_tids;
> -	/*
> -	 * We copy the iovs for this request (based on
> -	 * info.iovcnt). These are only the data vectors
> -	 */
> -	u8 data_iovs;
> -	s8 ahg_idx;
> -
> -	/* Writeable fields shared with interrupt */
> -	u64 seqcomp ____cacheline_aligned_in_smp;
> -	u64 seqsubmitted;
> -	/* status of the last txreq completed */
> -	int status;
> -
> -	/* Send side fields */
> -	struct list_head txps ____cacheline_aligned_in_smp;
> -	u64 seqnum;
> -	/*
> -	 * KDETH.OFFSET (TID) field
> -	 * The offset can cover multiple packets, depending on the
> -	 * size of the TID entry.
> -	 */
> -	u32 tidoffset;
> -	/*
> -	 * KDETH.Offset (Eager) field
> -	 * We need to remember the initial value so the headers
> -	 * can be updated properly.
> -	 */
> -	u32 koffset;
> -	u32 sent;
> -	/* TID index copied from the tid_iov vector */
> -	u16 tididx;
> -	/* progress index moving along the iovs array */
> -	u8 iov_idx;
> -	u8 done;
> -	u8 has_error;
> -
> -	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
> -} ____cacheline_aligned_in_smp;
> -
> -/*
> - * A single txreq could span up to 3 physical pages when the MTU
> - * is sufficiently large (> 4K). Each of the IOV pointers also
> - * needs it's own set of flags so the vector has been handled
> - * independently of each other.
> - */
> -struct user_sdma_txreq {
> -	/* Packet header for the txreq */
> -	struct hfi1_pkt_header hdr;
> -	struct sdma_txreq txreq;
> -	struct list_head list;
> -	struct user_sdma_request *req;
> -	u16 flags;
> -	unsigned busycount;
> -	u64 seqnum;
> -};
> -
> -#define SDMA_DBG(req, fmt, ...)				     \
> -	hfi1_cdbg(SDMA, "[%u:%u:%u:%u] " fmt, (req)->pq->dd->unit, \
> -		 (req)->pq->ctxt, (req)->pq->subctxt, (req)->info.comp_idx, \
> -		 ##__VA_ARGS__)
> -#define SDMA_Q_DBG(pq, fmt, ...)			 \
> -	hfi1_cdbg(SDMA, "[%u:%u:%u] " fmt, (pq)->dd->unit, (pq)->ctxt, \
> -		 (pq)->subctxt, ##__VA_ARGS__)
> -
>  static int user_sdma_send_pkts(struct user_sdma_request *req,
>  			       unsigned maxpkts);
>  static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status);
> diff --git a/drivers/infiniband/hw/hfi1/user_sdma.h b/drivers/infiniband/hw/hfi1/user_sdma.h
> index 84c199d..6c10484 100644
> --- a/drivers/infiniband/hw/hfi1/user_sdma.h
> +++ b/drivers/infiniband/hw/hfi1/user_sdma.h
> @@ -53,6 +53,67 @@
>  #include "iowait.h"
>  #include "user_exp_rcv.h"
>
> +/* The maximum number of Data io vectors per message/request */
> +#define MAX_VECTORS_PER_REQ 8
> +/*
> + * Maximum number of packet to send from each message/request
> + * before moving to the next one.
> + */
> +#define MAX_PKTS_PER_QUEUE 16
> +
> +#define num_pages(x) (1 + ((((x) - 1) & PAGE_MASK) >> PAGE_SHIFT))
> +
> +#define req_opcode(x) \
> +	(((x) >> HFI1_SDMA_REQ_OPCODE_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
> +#define req_version(x) \
> +	(((x) >> HFI1_SDMA_REQ_VERSION_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
> +#define req_iovcnt(x) \
> +	(((x) >> HFI1_SDMA_REQ_IOVCNT_SHIFT) & HFI1_SDMA_REQ_IOVCNT_MASK)
> +
> +/* Number of BTH.PSN bits used for sequence number in expected rcvs */
> +#define BTH_SEQ_MASK 0x7ffull
> +
> +#define AHG_KDETH_INTR_SHIFT 12
> +#define AHG_KDETH_SH_SHIFT   13
> +#define AHG_KDETH_ARRAY_SIZE  9
> +
> +#define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
> +#define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)
> +
> +#define AHG_HEADER_SET(arr, idx, dw, bit, width, value)			\
> +	do {								\
> +		if ((idx) < ARRAY_SIZE((arr)))				\
> +			(arr)[(idx++)] = sdma_build_ahg_descriptor(	\
> +				(__force u16)(value), (dw), (bit),	\
> +							(width));	\
> +		else							\
> +			return -ERANGE;					\
> +	} while (0)

I know that this was in original code, but it violates Documentation/process/coding-style.rst
 687 12) Macros, Enums and RTL
....
 713 Things to avoid when using macros:
 714
 715 1) macros that affect control flow:
 716
 717 .. code-block:: c
 718
 719         #define FOO(x)                                  \
 720                 do {                                    \
 721                         if (blah(x) < 0)                \
 722                                 return -EBUGGERED;      \
 723                 } while (0)
 724
 725 is a **very** bad idea.  It looks like a function call but exits the ``calling``
 726 function; don't break the internal parsers of those who will read the code.



> +
> +/* Tx request flag bits */
> +#define TXREQ_FLAGS_REQ_ACK   BIT(0)      /* Set the ACK bit in the header */
> +#define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
> +
> +#define SDMA_PKT_Q_INACTIVE BIT(0)
> +#define SDMA_PKT_Q_ACTIVE   BIT(1)
> +#define SDMA_PKT_Q_DEFERRED BIT(2)
> +
> +/*
> + * Maximum retry attempts to submit a TX request
> + * before putting the process to sleep.
> + */
> +#define MAX_DEFER_RETRY_COUNT 1
> +
> +#define SDMA_IOWAIT_TIMEOUT 1000 /* in milliseconds */
> +
> +#define SDMA_DBG(req, fmt, ...)				     \
> +	hfi1_cdbg(SDMA, "[%u:%u:%u:%u] " fmt, (req)->pq->dd->unit, \
> +		 (req)->pq->ctxt, (req)->pq->subctxt, (req)->info.comp_idx, \
> +		 ##__VA_ARGS__)
> +#define SDMA_Q_DBG(pq, fmt, ...)			 \
> +	hfi1_cdbg(SDMA, "[%u:%u:%u] " fmt, (pq)->dd->unit, (pq)->ctxt, \
> +		 (pq)->subctxt, ##__VA_ARGS__)
> +
>  extern uint extended_psn;
>
>  struct hfi1_user_sdma_pkt_q {
> @@ -79,6 +140,111 @@ struct hfi1_user_sdma_comp_q {
>  	struct hfi1_sdma_comp_entry *comps;
>  };
>
> +struct sdma_mmu_node {
> +	struct mmu_rb_node rb;
> +	struct hfi1_user_sdma_pkt_q *pq;
> +	atomic_t refcount;
> +	struct page **pages;
> +	unsigned int npages;
> +};
> +
> +struct user_sdma_iovec {
> +	struct list_head list;
> +	struct iovec iov;
> +	/* number of pages in this vector */
> +	unsigned int npages;
> +	/* array of pinned pages for this vector */
> +	struct page **pages;
> +	/*
> +	 * offset into the virtual address space of the vector at
> +	 * which we last left off.
> +	 */
> +	u64 offset;
> +	struct sdma_mmu_node *node;
> +};
> +
> +/* evict operation argument */
> +struct evict_data {
> +	u32 cleared;	/* count evicted so far */
> +	u32 target;	/* target count to evict */
> +};
> +
> +struct user_sdma_request {
> +	/* This is the original header from user space */
> +	struct hfi1_pkt_header hdr;
> +
> +	/* Read mostly fields */
> +	struct hfi1_user_sdma_pkt_q *pq ____cacheline_aligned_in_smp;
> +	struct hfi1_user_sdma_comp_q *cq;
> +	/*
> +	 * Pointer to the SDMA engine for this request.
> +	 * Since different request could be on different VLs,
> +	 * each request will need it's own engine pointer.
> +	 */
> +	struct sdma_engine *sde;
> +	struct sdma_req_info info;
> +	/* TID array values copied from the tid_iov vector */
> +	u32 *tids;
> +	/* total length of the data in the request */
> +	u32 data_len;
> +	/* number of elements copied to the tids array */
> +	u16 n_tids;
> +	/*
> +	 * We copy the iovs for this request (based on
> +	 * info.iovcnt). These are only the data vectors
> +	 */
> +	u8 data_iovs;
> +	s8 ahg_idx;
> +
> +	/* Writeable fields shared with interrupt */
> +	u64 seqcomp ____cacheline_aligned_in_smp;
> +	u64 seqsubmitted;
> +	/* status of the last txreq completed */
> +	int status;
> +
> +	/* Send side fields */
> +	struct list_head txps ____cacheline_aligned_in_smp;
> +	u64 seqnum;
> +	/*
> +	 * KDETH.OFFSET (TID) field
> +	 * The offset can cover multiple packets, depending on the
> +	 * size of the TID entry.
> +	 */
> +	u32 tidoffset;
> +	/*
> +	 * KDETH.Offset (Eager) field
> +	 * We need to remember the initial value so the headers
> +	 * can be updated properly.
> +	 */
> +	u32 koffset;
> +	u32 sent;
> +	/* TID index copied from the tid_iov vector */
> +	u16 tididx;
> +	/* progress index moving along the iovs array */
> +	u8 iov_idx;
> +	u8 done;
> +	u8 has_error;
> +
> +	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
> +} ____cacheline_aligned_in_smp;
> +
> +/*
> + * A single txreq could span up to 3 physical pages when the MTU
> + * is sufficiently large (> 4K). Each of the IOV pointers also
> + * needs it's own set of flags so the vector has been handled
> + * independently of each other.
> + */
> +struct user_sdma_txreq {
> +	/* Packet header for the txreq */
> +	struct hfi1_pkt_header hdr;
> +	struct sdma_txreq txreq;
> +	struct list_head list;
> +	struct user_sdma_request *req;
> +	u16 flags;
> +	unsigned int busycount;
> +	u64 seqnum;
> +};
> +
>  int hfi1_user_sdma_alloc_queues(struct hfi1_ctxtdata *uctxt,
>  				struct hfi1_filedata *fd);
>  int hfi1_user_sdma_free_queues(struct hfi1_filedata *fd,
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH for-next 09/16] IB/hfi1: Clean up pin_vector_pages() function

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 4291 bytes --]

On Mon, Aug 21, 2017 at 06:27:03PM -0700, Dennis Dalessandro wrote:
> From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
> Clean up pin_vector_pages() function by moving page pinning related code
> to a separate function since it really stands on its own.
>
> Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>  drivers/infiniband/hw/hfi1/user_sdma.c |   79 ++++++++++++++++++--------------
>  1 files changed, 45 insertions(+), 34 deletions(-)
>
> diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
> index d5a2572..6f26253 100644
> --- a/drivers/infiniband/hw/hfi1/user_sdma.c
> +++ b/drivers/infiniband/hw/hfi1/user_sdma.c
> @@ -1124,11 +1124,53 @@ static u32 sdma_cache_evict(struct hfi1_user_sdma_pkt_q *pq, u32 npages)
>  	return evict_data.cleared;
>  }
>
> +static int pin_sdma_pages(struct user_sdma_request *req,
> +			  struct user_sdma_iovec *iovec,
> +			  struct sdma_mmu_node *node,
> +			  int npages)
> +{
> +	int pinned, cleared;
> +	struct page **pages;
> +	struct hfi1_user_sdma_pkt_q *pq = req->pq;
> +
> +	pages = kcalloc(npages, sizeof(*pages), GFP_KERNEL);
> +	if (!pages) {
> +		SDMA_DBG(req, "Failed page array alloc");

Please don't add prints after k[c|m|z]alloc failures, despite the fact
that it was before.

Thanks

> +		return -ENOMEM;
> +	}
> +	memcpy(pages, node->pages, node->npages * sizeof(*pages));
> +
> +	npages -= node->npages;
> +retry:
> +	if (!hfi1_can_pin_pages(pq->dd, pq->mm,
> +				atomic_read(&pq->n_locked), npages)) {
> +		cleared = sdma_cache_evict(pq, npages);
> +		if (cleared >= npages)
> +			goto retry;
> +	}
> +	pinned = hfi1_acquire_user_pages(pq->mm,
> +					 ((unsigned long)iovec->iov.iov_base +
> +					 (node->npages * PAGE_SIZE)), npages, 0,
> +					 pages + node->npages);
> +	if (pinned < 0) {
> +		kfree(pages);
> +		return pinned;
> +	}
> +	if (pinned != npages) {
> +		unpin_vector_pages(pq->mm, pages, node->npages, pinned);
> +		return -EFAULT;
> +	}
> +	kfree(node->pages);
> +	node->rb.len = iovec->iov.iov_len;
> +	node->pages = pages;
> +	atomic_add(pinned, &pq->n_locked);
> +	return pinned;
> +}
> +
>  static int pin_vector_pages(struct user_sdma_request *req,
>  			    struct user_sdma_iovec *iovec)
>  {
> -	int ret = 0, pinned, npages, cleared;
> -	struct page **pages;
> +	int ret = 0, pinned, npages;
>  	struct hfi1_user_sdma_pkt_q *pq = req->pq;
>  	struct sdma_mmu_node *node = NULL;
>  	struct mmu_rb_node *rb_node;
> @@ -1162,44 +1204,13 @@ static int pin_vector_pages(struct user_sdma_request *req,
>
>  	npages = num_user_pages(&iovec->iov);
>  	if (node->npages < npages) {
> -		pages = kcalloc(npages, sizeof(*pages), GFP_KERNEL);
> -		if (!pages) {
> -			SDMA_DBG(req, "Failed page array alloc");
> -			ret = -ENOMEM;
> -			goto bail;
> -		}
> -		memcpy(pages, node->pages, node->npages * sizeof(*pages));
> -
> -		npages -= node->npages;
> -
> -retry:
> -		if (!hfi1_can_pin_pages(pq->dd, pq->mm,
> -					atomic_read(&pq->n_locked), npages)) {
> -			cleared = sdma_cache_evict(pq, npages);
> -			if (cleared >= npages)
> -				goto retry;
> -		}
> -		pinned = hfi1_acquire_user_pages(pq->mm,
> -			((unsigned long)iovec->iov.iov_base +
> -			 (node->npages * PAGE_SIZE)), npages, 0,
> -			pages + node->npages);
> +		pinned = pin_sdma_pages(req, iovec, node, npages);
>  		if (pinned < 0) {
> -			kfree(pages);
>  			ret = pinned;
>  			goto bail;
>  		}
> -		if (pinned != npages) {
> -			unpin_vector_pages(pq->mm, pages, node->npages,
> -					   pinned);
> -			ret = -EFAULT;
> -			goto bail;
> -		}
> -		kfree(node->pages);
> -		node->rb.len = iovec->iov.iov_len;
> -		node->pages = pages;
>  		node->npages += pinned;
>  		npages = node->npages;
> -		atomic_add(pinned, &pq->n_locked);
>  	}
>  	iovec->pages = node->pages;
>  	iovec->npages = npages;
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH for-next 09/16] IB/hfi1: Clean up pin_vector_pages() function

[Harish Chegondi]

On 08/22/2017 08:46 AM, Leon Romanovsky wrote:
> On Mon, Aug 21, 2017 at 06:27:03PM -0700, Dennis Dalessandro wrote:
>> From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>
>> Clean up pin_vector_pages() function by moving page pinning related code
>> to a separate function since it really stands on its own.
>>
>> Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> ---
>>  drivers/infiniband/hw/hfi1/user_sdma.c |   79 ++++++++++++++++++--------------
>>  1 files changed, 45 insertions(+), 34 deletions(-)
>>
>> diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
>> index d5a2572..6f26253 100644
>> --- a/drivers/infiniband/hw/hfi1/user_sdma.c
>> +++ b/drivers/infiniband/hw/hfi1/user_sdma.c
>> @@ -1124,11 +1124,53 @@ static u32 sdma_cache_evict(struct hfi1_user_sdma_pkt_q *pq, u32 npages)
>>  	return evict_data.cleared;
>>  }
>>
>> +static int pin_sdma_pages(struct user_sdma_request *req,
>> +			  struct user_sdma_iovec *iovec,
>> +			  struct sdma_mmu_node *node,
>> +			  int npages)
>> +{
>> +	int pinned, cleared;
>> +	struct page **pages;
>> +	struct hfi1_user_sdma_pkt_q *pq = req->pq;
>> +
>> +	pages = kcalloc(npages, sizeof(*pages), GFP_KERNEL);
>> +	if (!pages) {
>> +		SDMA_DBG(req, "Failed page array alloc");
> 
> Please don't add prints after k[c|m|z]alloc failures, despite the fact
> that it was before.
> 
> Thanks

SDMA_DBG MACRO is a trace for development debugging. It's not a print.

> 
>> +		return -ENOMEM;
>> +	}
>> +	memcpy(pages, node->pages, node->npages * sizeof(*pages));
>> +
>> +	npages -= node->npages;
>> +retry:
>> +	if (!hfi1_can_pin_pages(pq->dd, pq->mm,
>> +				atomic_read(&pq->n_locked), npages)) {
>> +		cleared = sdma_cache_evict(pq, npages);
>> +		if (cleared >= npages)
>> +			goto retry;
>> +	}
>> +	pinned = hfi1_acquire_user_pages(pq->mm,
>> +					 ((unsigned long)iovec->iov.iov_base +
>> +					 (node->npages * PAGE_SIZE)), npages, 0,
>> +					 pages + node->npages);
>> +	if (pinned < 0) {
>> +		kfree(pages);
>> +		return pinned;
>> +	}
>> +	if (pinned != npages) {
>> +		unpin_vector_pages(pq->mm, pages, node->npages, pinned);
>> +		return -EFAULT;
>> +	}
>> +	kfree(node->pages);
>> +	node->rb.len = iovec->iov.iov_len;
>> +	node->pages = pages;
>> +	atomic_add(pinned, &pq->n_locked);
>> +	return pinned;
>> +}
>> +
>>  static int pin_vector_pages(struct user_sdma_request *req,
>>  			    struct user_sdma_iovec *iovec)
>>  {
>> -	int ret = 0, pinned, npages, cleared;
>> -	struct page **pages;
>> +	int ret = 0, pinned, npages;
>>  	struct hfi1_user_sdma_pkt_q *pq = req->pq;
>>  	struct sdma_mmu_node *node = NULL;
>>  	struct mmu_rb_node *rb_node;
>> @@ -1162,44 +1204,13 @@ static int pin_vector_pages(struct user_sdma_request *req,
>>
>>  	npages = num_user_pages(&iovec->iov);
>>  	if (node->npages < npages) {
>> -		pages = kcalloc(npages, sizeof(*pages), GFP_KERNEL);
>> -		if (!pages) {
>> -			SDMA_DBG(req, "Failed page array alloc");
>> -			ret = -ENOMEM;
>> -			goto bail;
>> -		}
>> -		memcpy(pages, node->pages, node->npages * sizeof(*pages));
>> -
>> -		npages -= node->npages;
>> -
>> -retry:
>> -		if (!hfi1_can_pin_pages(pq->dd, pq->mm,
>> -					atomic_read(&pq->n_locked), npages)) {
>> -			cleared = sdma_cache_evict(pq, npages);
>> -			if (cleared >= npages)
>> -				goto retry;
>> -		}
>> -		pinned = hfi1_acquire_user_pages(pq->mm,
>> -			((unsigned long)iovec->iov.iov_base +
>> -			 (node->npages * PAGE_SIZE)), npages, 0,
>> -			pages + node->npages);
>> +		pinned = pin_sdma_pages(req, iovec, node, npages);
>>  		if (pinned < 0) {
>> -			kfree(pages);
>>  			ret = pinned;
>>  			goto bail;
>>  		}
>> -		if (pinned != npages) {
>> -			unpin_vector_pages(pq->mm, pages, node->npages,
>> -					   pinned);
>> -			ret = -EFAULT;
>> -			goto bail;
>> -		}
>> -		kfree(node->pages);
>> -		node->rb.len = iovec->iov.iov_len;
>> -		node->pages = pages;
>>  		node->npages += pinned;
>>  		npages = node->npages;
>> -		atomic_add(pinned, &pq->n_locked);
>>  	}
>>  	iovec->pages = node->pages;
>>  	iovec->npages = npages;
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-next 09/16] IB/hfi1: Clean up pin_vector_pages() function

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2155 bytes --]

On Tue, Aug 22, 2017 at 11:39:48AM -0700, Harish Chegondi wrote:
>
>
> On 08/22/2017 08:46 AM, Leon Romanovsky wrote:
> > On Mon, Aug 21, 2017 at 06:27:03PM -0700, Dennis Dalessandro wrote:
> >> From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >>
> >> Clean up pin_vector_pages() function by moving page pinning related code
> >> to a separate function since it really stands on its own.
> >>
> >> Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> >> ---
> >>  drivers/infiniband/hw/hfi1/user_sdma.c |   79 ++++++++++++++++++--------------
> >>  1 files changed, 45 insertions(+), 34 deletions(-)
> >>
> >> diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
> >> index d5a2572..6f26253 100644
> >> --- a/drivers/infiniband/hw/hfi1/user_sdma.c
> >> +++ b/drivers/infiniband/hw/hfi1/user_sdma.c
> >> @@ -1124,11 +1124,53 @@ static u32 sdma_cache_evict(struct hfi1_user_sdma_pkt_q *pq, u32 npages)
> >>  	return evict_data.cleared;
> >>  }
> >>
> >> +static int pin_sdma_pages(struct user_sdma_request *req,
> >> +			  struct user_sdma_iovec *iovec,
> >> +			  struct sdma_mmu_node *node,
> >> +			  int npages)
> >> +{
> >> +	int pinned, cleared;
> >> +	struct page **pages;
> >> +	struct hfi1_user_sdma_pkt_q *pq = req->pq;
> >> +
> >> +	pages = kcalloc(npages, sizeof(*pages), GFP_KERNEL);
> >> +	if (!pages) {
> >> +		SDMA_DBG(req, "Failed page array alloc");
> >
> > Please don't add prints after k[c|m|z]alloc failures, despite the fact
> > that it was before.
> >
> > Thanks
>
> SDMA_DBG MACRO is a trace for development debugging. It's not a print.
>

No problem, I'll rephrase it for you.

Please don't add prints, tracepoints, debug output and SDMA_DBG after k[c|m|z]alloc failures,
despite the fact that it was before.

Tracepoints output will be less worry for you if your kcalloc starts to fail.

Is it clear enough now?
>
Thanks,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH for-next 09/16] IB/hfi1: Clean up pin_vector_pages() function

[Harish Chegondi]

On 08/22/2017 09:49 PM, Leon Romanovsky wrote:
> On Tue, Aug 22, 2017 at 11:39:48AM -0700, Harish Chegondi wrote:
>>
>>
>> On 08/22/2017 08:46 AM, Leon Romanovsky wrote:
>>> On Mon, Aug 21, 2017 at 06:27:03PM -0700, Dennis Dalessandro wrote:
>>>> From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>>>
>>>> Clean up pin_vector_pages() function by moving page pinning related code
>>>> to a separate function since it really stands on its own.
>>>>
>>>> Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>>> Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>>> Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>>> ---
>>>>  drivers/infiniband/hw/hfi1/user_sdma.c |   79 ++++++++++++++++++--------------
>>>>  1 files changed, 45 insertions(+), 34 deletions(-)
>>>>
>>>> diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
>>>> index d5a2572..6f26253 100644
>>>> --- a/drivers/infiniband/hw/hfi1/user_sdma.c
>>>> +++ b/drivers/infiniband/hw/hfi1/user_sdma.c
>>>> @@ -1124,11 +1124,53 @@ static u32 sdma_cache_evict(struct hfi1_user_sdma_pkt_q *pq, u32 npages)
>>>>  	return evict_data.cleared;
>>>>  }
>>>>
>>>> +static int pin_sdma_pages(struct user_sdma_request *req,
>>>> +			  struct user_sdma_iovec *iovec,
>>>> +			  struct sdma_mmu_node *node,
>>>> +			  int npages)
>>>> +{
>>>> +	int pinned, cleared;
>>>> +	struct page **pages;
>>>> +	struct hfi1_user_sdma_pkt_q *pq = req->pq;
>>>> +
>>>> +	pages = kcalloc(npages, sizeof(*pages), GFP_KERNEL);
>>>> +	if (!pages) {
>>>> +		SDMA_DBG(req, "Failed page array alloc");
>>>
>>> Please don't add prints after k[c|m|z]alloc failures, despite the fact
>>> that it was before.
>>>
>>> Thanks
>>
>> SDMA_DBG MACRO is a trace for development debugging. It's not a print.
>>
> 
> No problem, I'll rephrase it for you.
> 
> Please don't add prints, tracepoints, debug output and SDMA_DBG after k[c|m|z]alloc failures,
> despite the fact that it was before.
> 
> Tracepoints output will be less worry for you if your kcalloc starts to fail.
> 
> Is it clear enough now?
>>
> Thanks,
> 

Ok. I will remove the SDMA_DBG trace print. But I think removing the trace print has to be done in a separate patch. I will soon submit another patch that removes the trace print.

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-next 13/16] IB/hfi1: Move structure and MACRO definitions in user_sdma.c to user_sdma.h

[Harish Chegondi]

On 08/22/2017 08:40 AM, Leon Romanovsky wrote:
> On Mon, Aug 21, 2017 at 06:27:29PM -0700, Dennis Dalessandro wrote:
>> From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>
>> Clean up user_sdma.c by moving the structure and MACRO definitions into
>> the header file user_sdma.h
>>
>> Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> ---
>>  drivers/infiniband/hw/hfi1/user_sdma.c |  168 --------------------------------
>>  drivers/infiniband/hw/hfi1/user_sdma.h |  166 ++++++++++++++++++++++++++++++++
>>  2 files changed, 166 insertions(+), 168 deletions(-)
>>
>> diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
>> index 8a1653a..dacb0fc 100644
>> --- a/drivers/infiniband/hw/hfi1/user_sdma.c
>> +++ b/drivers/infiniband/hw/hfi1/user_sdma.c
>> @@ -74,176 +74,8 @@
>>  module_param_named(sdma_comp_size, hfi1_sdma_comp_ring_size, uint, S_IRUGO);
>>  MODULE_PARM_DESC(sdma_comp_size, "Size of User SDMA completion ring. Default: 128");
>>
>> -/* The maximum number of Data io vectors per message/request */
>> -#define MAX_VECTORS_PER_REQ 8
>> -/*
>> - * Maximum number of packet to send from each message/request
>> - * before moving to the next one.
>> - */
>> -#define MAX_PKTS_PER_QUEUE 16
>> -
>> -#define num_pages(x) (1 + ((((x) - 1) & PAGE_MASK) >> PAGE_SHIFT))
>> -
>> -#define req_opcode(x) \
>> -	(((x) >> HFI1_SDMA_REQ_OPCODE_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
>> -#define req_version(x) \
>> -	(((x) >> HFI1_SDMA_REQ_VERSION_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
>> -#define req_iovcnt(x) \
>> -	(((x) >> HFI1_SDMA_REQ_IOVCNT_SHIFT) & HFI1_SDMA_REQ_IOVCNT_MASK)
>> -
>> -/* Number of BTH.PSN bits used for sequence number in expected rcvs */
>> -#define BTH_SEQ_MASK 0x7ffull
>> -
>> -#define AHG_KDETH_INTR_SHIFT 12
>> -#define AHG_KDETH_SH_SHIFT   13
>> -#define AHG_KDETH_ARRAY_SIZE  9
>> -
>> -#define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
>> -#define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)
>> -
>> -#define AHG_HEADER_SET(arr, idx, dw, bit, width, value)			\
>> -	do {								\
>> -		if ((idx) < ARRAY_SIZE((arr)))				\
>> -			(arr)[(idx++)] = sdma_build_ahg_descriptor(	\
>> -				(__force u16)(value), (dw), (bit),	\
>> -							(width));	\
>> -		else							\
>> -			return -ERANGE;					\
>> -	} while (0)
>> -
>> -/* Tx request flag bits */
>> -#define TXREQ_FLAGS_REQ_ACK   BIT(0)      /* Set the ACK bit in the header */
>> -#define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
>> -
>> -#define SDMA_PKT_Q_INACTIVE BIT(0)
>> -#define SDMA_PKT_Q_ACTIVE   BIT(1)
>> -#define SDMA_PKT_Q_DEFERRED BIT(2)
>> -
>> -/*
>> - * Maximum retry attempts to submit a TX request
>> - * before putting the process to sleep.
>> - */
>> -#define MAX_DEFER_RETRY_COUNT 1
>> -
>>  static unsigned initial_pkt_count = 8;
>>
>> -#define SDMA_IOWAIT_TIMEOUT 1000 /* in milliseconds */
>> -
>> -struct sdma_mmu_node;
>> -
>> -struct user_sdma_iovec {
>> -	struct list_head list;
>> -	struct iovec iov;
>> -	/* number of pages in this vector */
>> -	unsigned npages;
>> -	/* array of pinned pages for this vector */
>> -	struct page **pages;
>> -	/*
>> -	 * offset into the virtual address space of the vector at
>> -	 * which we last left off.
>> -	 */
>> -	u64 offset;
>> -	struct sdma_mmu_node *node;
>> -};
>> -
>> -struct sdma_mmu_node {
>> -	struct mmu_rb_node rb;
>> -	struct hfi1_user_sdma_pkt_q *pq;
>> -	atomic_t refcount;
>> -	struct page **pages;
>> -	unsigned npages;
>> -};
>> -
>> -/* evict operation argument */
>> -struct evict_data {
>> -	u32 cleared;	/* count evicted so far */
>> -	u32 target;	/* target count to evict */
>> -};
>> -
>> -struct user_sdma_request {
>> -	/* This is the original header from user space */
>> -	struct hfi1_pkt_header hdr;
>> -
>> -	/* Read mostly fields */
>> -	struct hfi1_user_sdma_pkt_q *pq ____cacheline_aligned_in_smp;
>> -	struct hfi1_user_sdma_comp_q *cq;
>> -	/*
>> -	 * Pointer to the SDMA engine for this request.
>> -	 * Since different request could be on different VLs,
>> -	 * each request will need it's own engine pointer.
>> -	 */
>> -	struct sdma_engine *sde;
>> -	struct sdma_req_info info;
>> -	/* TID array values copied from the tid_iov vector */
>> -	u32 *tids;
>> -	/* total length of the data in the request */
>> -	u32 data_len;
>> -	/* number of elements copied to the tids array */
>> -	u16 n_tids;
>> -	/*
>> -	 * We copy the iovs for this request (based on
>> -	 * info.iovcnt). These are only the data vectors
>> -	 */
>> -	u8 data_iovs;
>> -	s8 ahg_idx;
>> -
>> -	/* Writeable fields shared with interrupt */
>> -	u64 seqcomp ____cacheline_aligned_in_smp;
>> -	u64 seqsubmitted;
>> -	/* status of the last txreq completed */
>> -	int status;
>> -
>> -	/* Send side fields */
>> -	struct list_head txps ____cacheline_aligned_in_smp;
>> -	u64 seqnum;
>> -	/*
>> -	 * KDETH.OFFSET (TID) field
>> -	 * The offset can cover multiple packets, depending on the
>> -	 * size of the TID entry.
>> -	 */
>> -	u32 tidoffset;
>> -	/*
>> -	 * KDETH.Offset (Eager) field
>> -	 * We need to remember the initial value so the headers
>> -	 * can be updated properly.
>> -	 */
>> -	u32 koffset;
>> -	u32 sent;
>> -	/* TID index copied from the tid_iov vector */
>> -	u16 tididx;
>> -	/* progress index moving along the iovs array */
>> -	u8 iov_idx;
>> -	u8 done;
>> -	u8 has_error;
>> -
>> -	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
>> -} ____cacheline_aligned_in_smp;
>> -
>> -/*
>> - * A single txreq could span up to 3 physical pages when the MTU
>> - * is sufficiently large (> 4K). Each of the IOV pointers also
>> - * needs it's own set of flags so the vector has been handled
>> - * independently of each other.
>> - */
>> -struct user_sdma_txreq {
>> -	/* Packet header for the txreq */
>> -	struct hfi1_pkt_header hdr;
>> -	struct sdma_txreq txreq;
>> -	struct list_head list;
>> -	struct user_sdma_request *req;
>> -	u16 flags;
>> -	unsigned busycount;
>> -	u64 seqnum;
>> -};
>> -
>> -#define SDMA_DBG(req, fmt, ...)				     \
>> -	hfi1_cdbg(SDMA, "[%u:%u:%u:%u] " fmt, (req)->pq->dd->unit, \
>> -		 (req)->pq->ctxt, (req)->pq->subctxt, (req)->info.comp_idx, \
>> -		 ##__VA_ARGS__)
>> -#define SDMA_Q_DBG(pq, fmt, ...)			 \
>> -	hfi1_cdbg(SDMA, "[%u:%u:%u] " fmt, (pq)->dd->unit, (pq)->ctxt, \
>> -		 (pq)->subctxt, ##__VA_ARGS__)
>> -
>>  static int user_sdma_send_pkts(struct user_sdma_request *req,
>>  			       unsigned maxpkts);
>>  static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status);
>> diff --git a/drivers/infiniband/hw/hfi1/user_sdma.h b/drivers/infiniband/hw/hfi1/user_sdma.h
>> index 84c199d..6c10484 100644
>> --- a/drivers/infiniband/hw/hfi1/user_sdma.h
>> +++ b/drivers/infiniband/hw/hfi1/user_sdma.h
>> @@ -53,6 +53,67 @@
>>  #include "iowait.h"
>>  #include "user_exp_rcv.h"
>>
>> +/* The maximum number of Data io vectors per message/request */
>> +#define MAX_VECTORS_PER_REQ 8
>> +/*
>> + * Maximum number of packet to send from each message/request
>> + * before moving to the next one.
>> + */
>> +#define MAX_PKTS_PER_QUEUE 16
>> +
>> +#define num_pages(x) (1 + ((((x) - 1) & PAGE_MASK) >> PAGE_SHIFT))
>> +
>> +#define req_opcode(x) \
>> +	(((x) >> HFI1_SDMA_REQ_OPCODE_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
>> +#define req_version(x) \
>> +	(((x) >> HFI1_SDMA_REQ_VERSION_SHIFT) & HFI1_SDMA_REQ_OPCODE_MASK)
>> +#define req_iovcnt(x) \
>> +	(((x) >> HFI1_SDMA_REQ_IOVCNT_SHIFT) & HFI1_SDMA_REQ_IOVCNT_MASK)
>> +
>> +/* Number of BTH.PSN bits used for sequence number in expected rcvs */
>> +#define BTH_SEQ_MASK 0x7ffull
>> +
>> +#define AHG_KDETH_INTR_SHIFT 12
>> +#define AHG_KDETH_SH_SHIFT   13
>> +#define AHG_KDETH_ARRAY_SIZE  9
>> +
>> +#define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
>> +#define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)
>> +
>> +#define AHG_HEADER_SET(arr, idx, dw, bit, width, value)			\
>> +	do {								\
>> +		if ((idx) < ARRAY_SIZE((arr)))				\
>> +			(arr)[(idx++)] = sdma_build_ahg_descriptor(	\
>> +				(__force u16)(value), (dw), (bit),	\
>> +							(width));	\
>> +		else							\
>> +			return -ERANGE;					\
>> +	} while (0)
> 
> I know that this was in original code, but it violates Documentation/process/coding-style.rst
>  687 12) Macros, Enums and RTL
> ....
>  713 Things to avoid when using macros:
>  714
>  715 1) macros that affect control flow:
>  716
>  717 .. code-block:: c
>  718
>  719         #define FOO(x)                                  \
>  720                 do {                                    \
>  721                         if (blah(x) < 0)                \
>  722                                 return -EBUGGERED;      \
>  723                 } while (0)
>  724
>  725 is a **very** bad idea.  It looks like a function call but exits the ``calling``
>  726 function; don't break the internal parsers of those who will read the code.

I will fix this Macro in a separate patch I will soon submit.

Thanks.

> 
> 
> 
>> +
>> +/* Tx request flag bits */
>> +#define TXREQ_FLAGS_REQ_ACK   BIT(0)      /* Set the ACK bit in the header */
>> +#define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
>> +
>> +#define SDMA_PKT_Q_INACTIVE BIT(0)
>> +#define SDMA_PKT_Q_ACTIVE   BIT(1)
>> +#define SDMA_PKT_Q_DEFERRED BIT(2)
>> +
>> +/*
>> + * Maximum retry attempts to submit a TX request
>> + * before putting the process to sleep.
>> + */
>> +#define MAX_DEFER_RETRY_COUNT 1
>> +
>> +#define SDMA_IOWAIT_TIMEOUT 1000 /* in milliseconds */
>> +
>> +#define SDMA_DBG(req, fmt, ...)				     \
>> +	hfi1_cdbg(SDMA, "[%u:%u:%u:%u] " fmt, (req)->pq->dd->unit, \
>> +		 (req)->pq->ctxt, (req)->pq->subctxt, (req)->info.comp_idx, \
>> +		 ##__VA_ARGS__)
>> +#define SDMA_Q_DBG(pq, fmt, ...)			 \
>> +	hfi1_cdbg(SDMA, "[%u:%u:%u] " fmt, (pq)->dd->unit, (pq)->ctxt, \
>> +		 (pq)->subctxt, ##__VA_ARGS__)
>> +
>>  extern uint extended_psn;
>>
>>  struct hfi1_user_sdma_pkt_q {
>> @@ -79,6 +140,111 @@ struct hfi1_user_sdma_comp_q {
>>  	struct hfi1_sdma_comp_entry *comps;
>>  };
>>
>> +struct sdma_mmu_node {
>> +	struct mmu_rb_node rb;
>> +	struct hfi1_user_sdma_pkt_q *pq;
>> +	atomic_t refcount;
>> +	struct page **pages;
>> +	unsigned int npages;
>> +};
>> +
>> +struct user_sdma_iovec {
>> +	struct list_head list;
>> +	struct iovec iov;
>> +	/* number of pages in this vector */
>> +	unsigned int npages;
>> +	/* array of pinned pages for this vector */
>> +	struct page **pages;
>> +	/*
>> +	 * offset into the virtual address space of the vector at
>> +	 * which we last left off.
>> +	 */
>> +	u64 offset;
>> +	struct sdma_mmu_node *node;
>> +};
>> +
>> +/* evict operation argument */
>> +struct evict_data {
>> +	u32 cleared;	/* count evicted so far */
>> +	u32 target;	/* target count to evict */
>> +};
>> +
>> +struct user_sdma_request {
>> +	/* This is the original header from user space */
>> +	struct hfi1_pkt_header hdr;
>> +
>> +	/* Read mostly fields */
>> +	struct hfi1_user_sdma_pkt_q *pq ____cacheline_aligned_in_smp;
>> +	struct hfi1_user_sdma_comp_q *cq;
>> +	/*
>> +	 * Pointer to the SDMA engine for this request.
>> +	 * Since different request could be on different VLs,
>> +	 * each request will need it's own engine pointer.
>> +	 */
>> +	struct sdma_engine *sde;
>> +	struct sdma_req_info info;
>> +	/* TID array values copied from the tid_iov vector */
>> +	u32 *tids;
>> +	/* total length of the data in the request */
>> +	u32 data_len;
>> +	/* number of elements copied to the tids array */
>> +	u16 n_tids;
>> +	/*
>> +	 * We copy the iovs for this request (based on
>> +	 * info.iovcnt). These are only the data vectors
>> +	 */
>> +	u8 data_iovs;
>> +	s8 ahg_idx;
>> +
>> +	/* Writeable fields shared with interrupt */
>> +	u64 seqcomp ____cacheline_aligned_in_smp;
>> +	u64 seqsubmitted;
>> +	/* status of the last txreq completed */
>> +	int status;
>> +
>> +	/* Send side fields */
>> +	struct list_head txps ____cacheline_aligned_in_smp;
>> +	u64 seqnum;
>> +	/*
>> +	 * KDETH.OFFSET (TID) field
>> +	 * The offset can cover multiple packets, depending on the
>> +	 * size of the TID entry.
>> +	 */
>> +	u32 tidoffset;
>> +	/*
>> +	 * KDETH.Offset (Eager) field
>> +	 * We need to remember the initial value so the headers
>> +	 * can be updated properly.
>> +	 */
>> +	u32 koffset;
>> +	u32 sent;
>> +	/* TID index copied from the tid_iov vector */
>> +	u16 tididx;
>> +	/* progress index moving along the iovs array */
>> +	u8 iov_idx;
>> +	u8 done;
>> +	u8 has_error;
>> +
>> +	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
>> +} ____cacheline_aligned_in_smp;
>> +
>> +/*
>> + * A single txreq could span up to 3 physical pages when the MTU
>> + * is sufficiently large (> 4K). Each of the IOV pointers also
>> + * needs it's own set of flags so the vector has been handled
>> + * independently of each other.
>> + */
>> +struct user_sdma_txreq {
>> +	/* Packet header for the txreq */
>> +	struct hfi1_pkt_header hdr;
>> +	struct sdma_txreq txreq;
>> +	struct list_head list;
>> +	struct user_sdma_request *req;
>> +	u16 flags;
>> +	unsigned int busycount;
>> +	u64 seqnum;
>> +};
>> +
>>  int hfi1_user_sdma_alloc_queues(struct hfi1_ctxtdata *uctxt,
>>  				struct hfi1_filedata *fd);
>>  int hfi1_user_sdma_free_queues(struct hfi1_filedata *fd,
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-next 00/16] IB/hfi1, qib, rdmavt: patches for next 08/21/2017

[Doug Ledford]

On Mon, 2017-08-21 at 18:26 -0700, Dennis Dalessandro wrote:
> Hi Doug,
> 
> Another week another set of patches. I meant to send on Friday but
> was waiting
> for 0-day builds to complete. I still haven't gotten any
> notification, the
> service may be down but I wanted to get these on the list anyway for
> review.
> If I get any bad result I will reply but there is nothing
> particularly scary
> in here.
> 
> Content wise this is more clean ups including a patch for rdmavt to
> use
> the put_swqe/clear_mr_ref helpers that support was added for in
> qib/hfi1
> previously. 
> 
> We have some minor bug fixes as well. A large portion of the patch
> set is
> a clean up/refactoring exercise by Harish that I took because it
> makes the code
> cleaner and easier to read.
> 
> Note, one of the fixes from Mike is marked stable. I don't know if
> it's something
> that you want to take into this late of an RC so it may be best to
> just go to
> next and flow through the stable process.
> 
> Patches can can also be found in my GitHub repo at:
> https://github.com/ddalessa/kernel/tree/for-4.14
> 

Hi Denny, 

I went ahead and took these, but please make sure you follow up with
the review requests from Leon.  Preferably by the time I'm back from
PTO the incremental patches will be waiting for me ;-)

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-next 00/16] IB/hfi1, qib, rdmavt: patches for next 08/21/2017

[Doug Ledford]

On Mon, 2017-08-21 at 18:26 -0700, Dennis Dalessandro wrote:
> Hi Doug,
> 
> Another week another set of patches. I meant to send on Friday but
> was waiting
> for 0-day builds to complete. I still haven't gotten any
> notification, the
> service may be down but I wanted to get these on the list anyway for
> review.
> If I get any bad result I will reply but there is nothing
> particularly scary
> in here.
> 
> Content wise this is more clean ups including a patch for rdmavt to
> use
> the put_swqe/clear_mr_ref helpers that support was added for in
> qib/hfi1
> previously. 
> 
> We have some minor bug fixes as well. A large portion of the patch
> set is
> a clean up/refactoring exercise by Harish that I took because it
> makes the code
> cleaner and easier to read.
> 
> Note, one of the fixes from Mike is marked stable. I don't know if
> it's something
> that you want to take into this late of an RC so it may be best to
> just go to
> next and flow through the stable process.
> 
> Patches can can also be found in my GitHub repo at:
> https://github.com/ddalessa/kernel/tree/for-4.14
> 

Hi Denny, 

I went ahead and took these, but please make sure you follow up with
the review requests from Leon.  Preferably by the time I'm back from
PTO the incremental patches will be waiting for me ;-)

-- 
Doug Ledford <dledford@redhat.com>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD