[PATCH for-next 0/9] IB/core, hfi1, rdmavt: patches for next 05/19/2017

[PATCH for-next 0/9] IB/core, hfi1, rdmavt: patches for next 05/19/2017

[Dennis Dalessandro]

Hi Doug,

Another set of patches for 4.13. These apply on top of the last submitted
series: http://marc.info/?l=linux-rdma&m=149460607918479&w=2. I wanted to
submit this last week but 0-day build service and my GitHub weren't seeing
eye to eye. Things are working now and these have passed 0-day builds.

These are mostly fixes for the hfi1 driver, but there is a core change and one
for rdmavt as well which adds the ability to send OPA cap_mask3 in a trap.

Patches can can also be found in my GitHub repo at:
https://github.com/ddalessa/kernel/tree/for-4.13

---

Ira Weiny (1):
      IB/hfi1: Fix up sdma_init function comment

Michael J. Ruhl (2):
      IB/hfi1: Name function prototype parameters for affinity module
      IB/hfi1: Replace deprecated pci functions with new API

Michael N. Henry (1):
      IB/core,rdmavt,hfi1: Send OPA cap_mask3 in trap

Sebastian Sanchez (5):
      IB/hfi1: Don't remove RB entry when not needed.
      IB/hfi1: Optimize cachelines for user SDMA request structure
      IB/hfi1: Remove atomic SDMA_REQ_SEND_DONE bit operation
      IB/hfi1: Remove atomic SDMA_REQ_HAS_ERROR bit operation
      IB/hfi1: Reclassify type of messages printed for platform config logic


 drivers/infiniband/hw/hfi1/affinity.c  |   18 ++--
 drivers/infiniband/hw/hfi1/affinity.h  |   14 ++-
 drivers/infiniband/hw/hfi1/chip.c      |  105 ++++++++++-----------
 drivers/infiniband/hw/hfi1/hfi.h       |    6 -
 drivers/infiniband/hw/hfi1/mad.c       |    7 -
 drivers/infiniband/hw/hfi1/mad.h       |    2 
 drivers/infiniband/hw/hfi1/mmu_rb.c    |   14 ++-
 drivers/infiniband/hw/hfi1/mmu_rb.h    |    5 +
 drivers/infiniband/hw/hfi1/pcie.c      |   80 +++-------------
 drivers/infiniband/hw/hfi1/platform.c  |   20 ++--
 drivers/infiniband/hw/hfi1/sdma.c      |    6 -
 drivers/infiniband/hw/hfi1/user_sdma.c |  159 +++++++++++++++++---------------
 drivers/infiniband/hw/hfi1/verbs.c     |    7 +
 drivers/infiniband/sw/rdmavt/vt.c      |    9 +-
 include/rdma/ib_verbs.h                |    3 -
 include/rdma/rdma_vt.h                 |    1 
 16 files changed, 215 insertions(+), 241 deletions(-)

--
-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 1/9] IB/hfi1: Don't remove RB entry when not needed.

[Dennis Dalessandro]

From: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

An RB tree is used for the SDMA pinning cache. Cache
entries are extracted and reinserted from the tree
in case the address range for it changes. However,
if the address range for the entry doesn't change,
deleting the entry from the RB tree is not necessary.

This affects performance since the tree needs to be
rebalanced for each insertion, and this happens in
the hot path. Optimize RB search by not removing
entries when it's not needed.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Reviewed-by: Mitko Haralanov <mitko.haralanov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/mmu_rb.c    |   14 ++++++++++----
 drivers/infiniband/hw/hfi1/mmu_rb.h    |    5 +++--
 drivers/infiniband/hw/hfi1/user_sdma.c |   23 ++++++++++++++++-------
 3 files changed, 29 insertions(+), 13 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/mmu_rb.c b/drivers/infiniband/hw/hfi1/mmu_rb.c
index ccbf52c..d41fd87 100644
--- a/drivers/infiniband/hw/hfi1/mmu_rb.c
+++ b/drivers/infiniband/hw/hfi1/mmu_rb.c
@@ -217,21 +217,27 @@ int hfi1_mmu_rb_insert(struct mmu_rb_handler *handler,
 	return node;
 }
 
-struct mmu_rb_node *hfi1_mmu_rb_extract(struct mmu_rb_handler *handler,
-					unsigned long addr, unsigned long len)
+bool hfi1_mmu_rb_remove_unless_exact(struct mmu_rb_handler *handler,
+				     unsigned long addr, unsigned long len,
+				     struct mmu_rb_node **rb_node)
 {
 	struct mmu_rb_node *node;
 	unsigned long flags;
+	bool ret = false;
 
 	spin_lock_irqsave(&handler->lock, flags);
 	node = __mmu_rb_search(handler, addr, len);
 	if (node) {
+		if (node->addr == addr && node->len == len)
+			goto unlock;
 		__mmu_int_rb_remove(node, &handler->root);
 		list_del(&node->list); /* remove from LRU list */
+		ret = true;
 	}
+unlock:
 	spin_unlock_irqrestore(&handler->lock, flags);
-
-	return node;
+	*rb_node = node;
+	return ret;
 }
 
 void hfi1_mmu_rb_evict(struct mmu_rb_handler *handler, void *evict_arg)
diff --git a/drivers/infiniband/hw/hfi1/mmu_rb.h b/drivers/infiniband/hw/hfi1/mmu_rb.h
index 754f6eb..f04cec1 100644
--- a/drivers/infiniband/hw/hfi1/mmu_rb.h
+++ b/drivers/infiniband/hw/hfi1/mmu_rb.h
@@ -81,7 +81,8 @@ int hfi1_mmu_rb_insert(struct mmu_rb_handler *handler,
 void hfi1_mmu_rb_evict(struct mmu_rb_handler *handler, void *evict_arg);
 void hfi1_mmu_rb_remove(struct mmu_rb_handler *handler,
 			struct mmu_rb_node *mnode);
-struct mmu_rb_node *hfi1_mmu_rb_extract(struct mmu_rb_handler *handler,
-					unsigned long addr, unsigned long len);
+bool hfi1_mmu_rb_remove_unless_exact(struct mmu_rb_handler *handler,
+				     unsigned long addr, unsigned long len,
+				     struct mmu_rb_node **rb_node);
 
 #endif /* _HFI1_MMU_RB_H */
diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index 16fd519..79450cf 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -1165,14 +1165,23 @@ static int pin_vector_pages(struct user_sdma_request *req,
 	struct hfi1_user_sdma_pkt_q *pq = req->pq;
 	struct sdma_mmu_node *node = NULL;
 	struct mmu_rb_node *rb_node;
-
-	rb_node = hfi1_mmu_rb_extract(pq->handler,
-				      (unsigned long)iovec->iov.iov_base,
-				      iovec->iov.iov_len);
-	if (rb_node)
+	bool extracted;
+
+	extracted =
+		hfi1_mmu_rb_remove_unless_exact(pq->handler,
+						(unsigned long)
+						iovec->iov.iov_base,
+						iovec->iov.iov_len, &rb_node);
+	if (rb_node) {
 		node = container_of(rb_node, struct sdma_mmu_node, rb);
-	else
-		rb_node = NULL;
+		if (!extracted) {
+			atomic_inc(&node->refcount);
+			iovec->pages = node->pages;
+			iovec->npages = node->npages;
+			iovec->node = node;
+			return 0;
+		}
+	}
 
 	if (!node) {
 		node = kzalloc(sizeof(*node), GFP_KERNEL);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 2/9] IB/hfi1: Optimize cachelines for user SDMA request structure

[Dennis Dalessandro]

From: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

The current user SDMA request structure layout has holes.
The cachelines can be reduced to improve cacheline trading.
Separate fields in the following categories: mostly read,
writable and shared with interrupt.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |  106 ++++++++++++++++++--------------
 1 files changed, 58 insertions(+), 48 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index 79450cf..92517ce 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -117,6 +117,7 @@
 
 #define AHG_KDETH_INTR_SHIFT 12
 #define AHG_KDETH_SH_SHIFT   13
+#define AHG_KDETH_ARRAY_SIZE  9
 
 #define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
 #define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)
@@ -204,25 +205,42 @@ struct evict_data {
 };
 
 struct user_sdma_request {
-	struct sdma_req_info info;
-	struct hfi1_user_sdma_pkt_q *pq;
-	struct hfi1_user_sdma_comp_q *cq;
 	/* This is the original header from user space */
 	struct hfi1_pkt_header hdr;
+
+	/* Read mostly fields */
+	struct hfi1_user_sdma_pkt_q *pq ____cacheline_aligned_in_smp;
+	struct hfi1_user_sdma_comp_q *cq;
 	/*
 	 * Pointer to the SDMA engine for this request.
 	 * Since different request could be on different VLs,
 	 * each request will need it's own engine pointer.
 	 */
 	struct sdma_engine *sde;
-	s8 ahg_idx;
-	u32 ahg[9];
+	struct sdma_req_info info;
+	/* TID array values copied from the tid_iov vector */
+	u32 *tids;
+	/* total length of the data in the request */
+	u32 data_len;
+	/* number of elements copied to the tids array */
+	u16 n_tids;
 	/*
-	 * KDETH.Offset (Eager) field
-	 * We need to remember the initial value so the headers
-	 * can be updated properly.
+	 * We copy the iovs for this request (based on
+	 * info.iovcnt). These are only the data vectors
 	 */
-	u32 koffset;
+	u8 data_iovs;
+	s8 ahg_idx;
+
+	/* Writeable fields shared with interrupt */
+	u64 seqcomp ____cacheline_aligned_in_smp;
+	u64 seqsubmitted;
+	unsigned long flags;
+	/* status of the last txreq completed */
+	int status;
+
+	/* Send side fields */
+	struct list_head txps ____cacheline_aligned_in_smp;
+	u64 seqnum;
 	/*
 	 * KDETH.OFFSET (TID) field
 	 * The offset can cover multiple packets, depending on the
@@ -230,29 +248,19 @@ struct user_sdma_request {
 	 */
 	u32 tidoffset;
 	/*
-	 * We copy the iovs for this request (based on
-	 * info.iovcnt). These are only the data vectors
+	 * KDETH.Offset (Eager) field
+	 * We need to remember the initial value so the headers
+	 * can be updated properly.
 	 */
-	unsigned data_iovs;
-	/* total length of the data in the request */
-	u32 data_len;
+	u32 koffset;
+	u32 sent;
+	/* TID index copied from the tid_iov vector */
+	u16 tididx;
 	/* progress index moving along the iovs array */
-	unsigned iov_idx;
+	u8 iov_idx;
+
 	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
-	/* number of elements copied to the tids array */
-	u16 n_tids;
-	/* TID array values copied from the tid_iov vector */
-	u32 *tids;
-	u16 tididx;
-	u32 sent;
-	u64 seqnum;
-	u64 seqcomp;
-	u64 seqsubmitted;
-	struct list_head txps;
-	unsigned long flags;
-	/* status of the last txreq completed */
-	int status;
-};
+} ____cacheline_aligned_in_smp;
 
 /*
  * A single txreq could span up to 3 physical pages when the MTU
@@ -1034,11 +1042,6 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 							       datalen);
 				if (changes < 0)
 					goto free_tx;
-				sdma_txinit_ahg(&tx->txreq,
-						SDMA_TXREQ_F_USE_AHG,
-						datalen, req->ahg_idx, changes,
-						req->ahg, sizeof(req->hdr),
-						user_sdma_txreq_cb);
 			}
 		} else {
 			ret = sdma_txinit(&tx->txreq, 0, sizeof(req->hdr) +
@@ -1442,21 +1445,22 @@ static int set_txreq_header(struct user_sdma_request *req,
 }
 
 static int set_txreq_header_ahg(struct user_sdma_request *req,
-				struct user_sdma_txreq *tx, u32 len)
+				struct user_sdma_txreq *tx, u32 datalen)
 {
+	u32 ahg[AHG_KDETH_ARRAY_SIZE];
 	int diff = 0;
 	u8 omfactor; /* KDETH.OM */
 	struct hfi1_user_sdma_pkt_q *pq = req->pq;
 	struct hfi1_pkt_header *hdr = &req->hdr;
 	u16 pbclen = le16_to_cpu(hdr->pbc[0]);
-	u32 val32, tidval = 0, lrhlen = get_lrh_len(*hdr, pad_len(len));
+	u32 val32, tidval = 0, lrhlen = get_lrh_len(*hdr, pad_len(datalen));
 
 	if (PBC2LRH(pbclen) != lrhlen) {
 		/* PBC.PbcLengthDWs */
-		AHG_HEADER_SET(req->ahg, diff, 0, 0, 12,
+		AHG_HEADER_SET(ahg, diff, 0, 0, 12,
 			       cpu_to_le16(LRH2PBC(lrhlen)));
 		/* LRH.PktLen (we need the full 16 bits due to byte swap) */
-		AHG_HEADER_SET(req->ahg, diff, 3, 0, 16,
+		AHG_HEADER_SET(ahg, diff, 3, 0, 16,
 			       cpu_to_be16(lrhlen >> 2));
 	}
 
@@ -1468,13 +1472,12 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 		(HFI1_CAP_IS_KSET(EXTENDED_PSN) ? 0x7fffffff : 0xffffff);
 	if (unlikely(tx->flags & TXREQ_FLAGS_REQ_ACK))
 		val32 |= 1UL << 31;
-	AHG_HEADER_SET(req->ahg, diff, 6, 0, 16, cpu_to_be16(val32 >> 16));
-	AHG_HEADER_SET(req->ahg, diff, 6, 16, 16, cpu_to_be16(val32 & 0xffff));
+	AHG_HEADER_SET(ahg, diff, 6, 0, 16, cpu_to_be16(val32 >> 16));
+	AHG_HEADER_SET(ahg, diff, 6, 16, 16, cpu_to_be16(val32 & 0xffff));
 	/* KDETH.Offset */
-	AHG_HEADER_SET(req->ahg, diff, 15, 0, 16,
+	AHG_HEADER_SET(ahg, diff, 15, 0, 16,
 		       cpu_to_le16(req->koffset & 0xffff));
-	AHG_HEADER_SET(req->ahg, diff, 15, 16, 16,
-		       cpu_to_le16(req->koffset >> 16));
+	AHG_HEADER_SET(ahg, diff, 15, 16, 16, cpu_to_le16(req->koffset >> 16));
 	if (req_opcode(req->info.ctrl) == EXPECTED) {
 		__le16 val;
 
@@ -1492,9 +1495,8 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 			 * we have to check again.
 			 */
 			if (++req->tididx > req->n_tids - 1 ||
-			    !req->tids[req->tididx]) {
+			    !req->tids[req->tididx])
 				return -EINVAL;
-			}
 			tidval = req->tids[req->tididx];
 		}
 		omfactor = ((EXP_TID_GET(tidval, LEN) *
@@ -1502,7 +1504,7 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 				 KDETH_OM_MAX_SIZE) ? KDETH_OM_LARGE_SHIFT :
 				 KDETH_OM_SMALL_SHIFT;
 		/* KDETH.OM and KDETH.OFFSET (TID) */
-		AHG_HEADER_SET(req->ahg, diff, 7, 0, 16,
+		AHG_HEADER_SET(ahg, diff, 7, 0, 16,
 			       ((!!(omfactor - KDETH_OM_SMALL_SHIFT)) << 15 |
 				((req->tidoffset >> omfactor)
 				 & 0x7fff)));
@@ -1522,12 +1524,20 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 					     AHG_KDETH_INTR_SHIFT));
 		}
 
-		AHG_HEADER_SET(req->ahg, diff, 7, 16, 14, val);
+		AHG_HEADER_SET(ahg, diff, 7, 16, 14, val);
 	}
+	if (diff < 0)
+		return diff;
 
 	trace_hfi1_sdma_user_header_ahg(pq->dd, pq->ctxt, pq->subctxt,
 					req->info.comp_idx, req->sde->this_idx,
-					req->ahg_idx, req->ahg, diff, tidval);
+					req->ahg_idx, ahg, diff, tidval);
+	sdma_txinit_ahg(&tx->txreq,
+			SDMA_TXREQ_F_USE_AHG,
+			datalen, req->ahg_idx, diff,
+			ahg, sizeof(req->hdr),
+			user_sdma_txreq_cb);
+
 	return diff;
 }
 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 3/9] IB/hfi1: Name function prototype parameters for affinity module

[Dennis Dalessandro]

From: Michael J. Ruhl <michael.j.ruhl-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

To improve the readability of function prototypes, give the parameters
names in the affinity module.

Reviewed-by: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Michael J. Ruhl <michael.j.ruhl-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/affinity.h |   14 ++++++++------
 1 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/affinity.h b/drivers/infiniband/hw/hfi1/affinity.h
index e78c7aa..2a1e374 100644
--- a/drivers/infiniband/hw/hfi1/affinity.h
+++ b/drivers/infiniband/hw/hfi1/affinity.h
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015, 2016 Intel Corporation.
+ * Copyright(c) 2015 - 2017 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -75,24 +75,26 @@ struct cpu_mask_set {
 /* Initialize non-HT cpu cores mask */
 void init_real_cpu_mask(void);
 /* Initialize driver affinity data */
-int hfi1_dev_affinity_init(struct hfi1_devdata *);
+int hfi1_dev_affinity_init(struct hfi1_devdata *dd);
 /*
  * Set IRQ affinity to a CPU. The function will determine the
  * CPU and set the affinity to it.
  */
-int hfi1_get_irq_affinity(struct hfi1_devdata *, struct hfi1_msix_entry *);
+int hfi1_get_irq_affinity(struct hfi1_devdata *dd,
+			  struct hfi1_msix_entry *msix);
 /*
  * Remove the IRQ's CPU affinity. This function also updates
  * any internal CPU tracking data
  */
-void hfi1_put_irq_affinity(struct hfi1_devdata *, struct hfi1_msix_entry *);
+void hfi1_put_irq_affinity(struct hfi1_devdata *dd,
+			   struct hfi1_msix_entry *msix);
 /*
  * Determine a CPU affinity for a user process, if the process does not
  * have an affinity set yet.
  */
-int hfi1_get_proc_affinity(int);
+int hfi1_get_proc_affinity(int node);
 /* Release a CPU used by a user process. */
-void hfi1_put_proc_affinity(int);
+void hfi1_put_proc_affinity(int cpu);
 
 struct hfi1_affinity_node {
 	int node;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 4/9] IB/hfi1: Replace deprecated pci functions with new API

[Dennis Dalessandro]

From: Michael J. Ruhl <michael.j.ruhl-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

pci_enable_msix_range() and pci_disable_msix() have been deprecated.
Updating to the new pci_alloc_irq_vectors() interface.

Reviewed-by: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Michael J. Ruhl <michael.j.ruhl-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/affinity.c |   18 +++---
 drivers/infiniband/hw/hfi1/chip.c     |  105 ++++++++++++++++-----------------
 drivers/infiniband/hw/hfi1/hfi.h      |    6 +-
 drivers/infiniband/hw/hfi1/pcie.c     |   80 +++++--------------------
 4 files changed, 78 insertions(+), 131 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/affinity.c b/drivers/infiniband/hw/hfi1/affinity.c
index e2cd2cd..a97055d 100644
--- a/drivers/infiniband/hw/hfi1/affinity.c
+++ b/drivers/infiniband/hw/hfi1/affinity.c
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015, 2016 Intel Corporation.
+ * Copyright(c) 2015 - 2017 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -335,10 +335,10 @@ static void hfi1_update_sdma_affinity(struct hfi1_msix_entry *msix, int cpu)
 	sde->cpu = cpu;
 	cpumask_clear(&msix->mask);
 	cpumask_set_cpu(cpu, &msix->mask);
-	dd_dev_dbg(dd, "IRQ vector: %u, type %s engine %u -> cpu: %d\n",
-		   msix->msix.vector, irq_type_names[msix->type],
+	dd_dev_dbg(dd, "IRQ: %u, type %s engine %u -> cpu: %d\n",
+		   msix->irq, irq_type_names[msix->type],
 		   sde->this_idx, cpu);
-	irq_set_affinity_hint(msix->msix.vector, &msix->mask);
+	irq_set_affinity_hint(msix->irq, &msix->mask);
 
 	/*
 	 * Set the new cpu in the hfi1_affinity_node and clean
@@ -387,7 +387,7 @@ static void hfi1_setup_sdma_notifier(struct hfi1_msix_entry *msix)
 {
 	struct irq_affinity_notify *notify = &msix->notify;
 
-	notify->irq = msix->msix.vector;
+	notify->irq = msix->irq;
 	notify->notify = hfi1_irq_notifier_notify;
 	notify->release = hfi1_irq_notifier_release;
 
@@ -472,10 +472,10 @@ static int get_irq_affinity(struct hfi1_devdata *dd,
 	}
 
 	cpumask_set_cpu(cpu, &msix->mask);
-	dd_dev_info(dd, "IRQ vector: %u, type %s %s -> cpu: %d\n",
-		    msix->msix.vector, irq_type_names[msix->type],
+	dd_dev_info(dd, "IRQ: %u, type %s %s -> cpu: %d\n",
+		    msix->irq, irq_type_names[msix->type],
 		    extra, cpu);
-	irq_set_affinity_hint(msix->msix.vector, &msix->mask);
+	irq_set_affinity_hint(msix->irq, &msix->mask);
 
 	if (msix->type == IRQ_SDMA) {
 		sde->cpu = cpu;
@@ -533,7 +533,7 @@ void hfi1_put_irq_affinity(struct hfi1_devdata *dd,
 		}
 	}
 
-	irq_set_affinity_hint(msix->msix.vector, NULL);
+	irq_set_affinity_hint(msix->irq, NULL);
 	cpumask_clear(&msix->mask);
 	mutex_unlock(&node_affinity.lock);
 }
diff --git a/drivers/infiniband/hw/hfi1/chip.c b/drivers/infiniband/hw/hfi1/chip.c
index a4fb116..aab107f 100644
--- a/drivers/infiniband/hw/hfi1/chip.c
+++ b/drivers/infiniband/hw/hfi1/chip.c
@@ -12800,30 +12800,24 @@ static void clean_up_interrupts(struct hfi1_devdata *dd)
 		for (i = 0; i < dd->num_msix_entries; i++, me++) {
 			if (!me->arg) /* => no irq, no affinity */
 				continue;
-			hfi1_put_irq_affinity(dd, &dd->msix_entries[i]);
-			free_irq(me->msix.vector, me->arg);
+			hfi1_put_irq_affinity(dd, me);
+			free_irq(me->irq, me->arg);
 		}
+
+		/* clean structures */
+		kfree(dd->msix_entries);
+		dd->msix_entries = NULL;
+		dd->num_msix_entries = 0;
 	} else {
 		/* INTx */
 		if (dd->requested_intx_irq) {
 			free_irq(dd->pcidev->irq, dd);
 			dd->requested_intx_irq = 0;
 		}
-	}
-
-	/* turn off interrupts */
-	if (dd->num_msix_entries) {
-		/* MSI-X */
-		pci_disable_msix(dd->pcidev);
-	} else {
-		/* INTx */
 		disable_intx(dd->pcidev);
 	}
 
-	/* clean structures */
-	kfree(dd->msix_entries);
-	dd->msix_entries = NULL;
-	dd->num_msix_entries = 0;
+	pci_free_irq_vectors(dd->pcidev);
 }
 
 /*
@@ -12972,13 +12966,21 @@ static int request_msix_irqs(struct hfi1_devdata *dd)
 			continue;
 		/* make sure the name is terminated */
 		me->name[sizeof(me->name) - 1] = 0;
+		me->irq = pci_irq_vector(dd->pcidev, i);
+		/*
+		 * On err return me->irq.  Don't need to clear this
+		 * because 'arg' has not been set, and cleanup will
+		 * do the right thing.
+		 */
+		if (me->irq < 0)
+			return me->irq;
 
-		ret = request_threaded_irq(me->msix.vector, handler, thread, 0,
+		ret = request_threaded_irq(me->irq, handler, thread, 0,
 					   me->name, arg);
 		if (ret) {
 			dd_dev_err(dd,
-				   "unable to allocate %s interrupt, vector %d, index %d, err %d\n",
-				   err_info, me->msix.vector, idx, ret);
+				   "unable to allocate %s interrupt, irq %d, index %d, err %d\n",
+				   err_info, me->irq, idx, ret);
 			return ret;
 		}
 		/*
@@ -12989,8 +12991,7 @@ static int request_msix_irqs(struct hfi1_devdata *dd)
 
 		ret = hfi1_get_irq_affinity(dd, me);
 		if (ret)
-			dd_dev_err(dd,
-				   "unable to pin IRQ %d\n", ret);
+			dd_dev_err(dd, "unable to pin IRQ %d\n", ret);
 	}
 
 	return ret;
@@ -13009,7 +13010,7 @@ void hfi1_vnic_synchronize_irq(struct hfi1_devdata *dd)
 		struct hfi1_ctxtdata *rcd = dd->vnic.ctxt[i];
 		struct hfi1_msix_entry *me = &dd->msix_entries[rcd->msix_intr];
 
-		synchronize_irq(me->msix.vector);
+		synchronize_irq(me->irq);
 	}
 }
 
@@ -13022,7 +13023,7 @@ void hfi1_reset_vnic_msix_info(struct hfi1_ctxtdata *rcd)
 		return;
 
 	hfi1_put_irq_affinity(dd, me);
-	free_irq(me->msix.vector, me->arg);
+	free_irq(me->irq, me->arg);
 
 	me->arg = NULL;
 }
@@ -13050,14 +13051,19 @@ void hfi1_set_vnic_msix_info(struct hfi1_ctxtdata *rcd)
 		 DRIVER_NAME "_%d kctxt%d", dd->unit, idx);
 	me->name[sizeof(me->name) - 1] = 0;
 	me->type = IRQ_RCVCTXT;
-
+	me->irq = pci_irq_vector(dd->pcidev, rcd->msix_intr);
+	if (me->irq < 0) {
+		dd_dev_err(dd, "vnic irq vector request (idx %d) fail %d\n",
+			   idx, me->irq);
+		return;
+	}
 	remap_intr(dd, IS_RCVAVAIL_START + idx, rcd->msix_intr);
 
-	ret = request_threaded_irq(me->msix.vector, receive_context_interrupt,
+	ret = request_threaded_irq(me->irq, receive_context_interrupt,
 				   receive_context_thread, 0, me->name, arg);
 	if (ret) {
-		dd_dev_err(dd, "vnic irq request (vector %d, idx %d) fail %d\n",
-			   me->msix.vector, idx, ret);
+		dd_dev_err(dd, "vnic irq request (irq %d, idx %d) fail %d\n",
+			   me->irq, idx, ret);
 		return;
 	}
 	/*
@@ -13070,7 +13076,7 @@ void hfi1_set_vnic_msix_info(struct hfi1_ctxtdata *rcd)
 	if (ret) {
 		dd_dev_err(dd,
 			   "unable to pin IRQ %d\n", ret);
-		free_irq(me->msix.vector, me->arg);
+		free_irq(me->irq, me->arg);
 	}
 }
 
@@ -13093,9 +13099,8 @@ static void reset_interrupts(struct hfi1_devdata *dd)
 
 static int set_up_interrupts(struct hfi1_devdata *dd)
 {
-	struct hfi1_msix_entry *entries;
-	u32 total, request;
-	int i, ret;
+	u32 total;
+	int ret, request;
 	int single_interrupt = 0; /* we expect to have all the interrupts */
 
 	/*
@@ -13107,39 +13112,31 @@ static int set_up_interrupts(struct hfi1_devdata *dd)
 	 */
 	total = 1 + dd->num_sdma + dd->n_krcv_queues + HFI1_NUM_VNIC_CTXT;
 
-	entries = kcalloc(total, sizeof(*entries), GFP_KERNEL);
-	if (!entries) {
-		ret = -ENOMEM;
-		goto fail;
-	}
-	/* 1-1 MSI-X entry assignment */
-	for (i = 0; i < total; i++)
-		entries[i].msix.entry = i;
-
 	/* ask for MSI-X interrupts */
-	request = total;
-	request_msix(dd, &request, entries);
-
-	if (request == 0) {
+	request = request_msix(dd, total);
+	if (request < 0) {
+		ret = request;
+		goto fail;
+	} else if (request == 0) {
 		/* using INTx */
 		/* dd->num_msix_entries already zero */
-		kfree(entries);
 		single_interrupt = 1;
 		dd_dev_err(dd, "MSI-X failed, using INTx interrupts\n");
+	} else if (request < total) {
+		/* using MSI-X, with reduced interrupts */
+		dd_dev_err(dd, "reduced interrupt found, wanted %u, got %u\n",
+			   total, request);
+		ret = -EINVAL;
+		goto fail;
 	} else {
-		/* using MSI-X */
-		dd->num_msix_entries = request;
-		dd->msix_entries = entries;
-
-		if (request != total) {
-			/* using MSI-X, with reduced interrupts */
-			dd_dev_err(
-				dd,
-				"cannot handle reduced interrupt case, want %u, got %u\n",
-				total, request);
-			ret = -EINVAL;
+		dd->msix_entries = kcalloc(total, sizeof(*dd->msix_entries),
+					   GFP_KERNEL);
+		if (!dd->msix_entries) {
+			ret = -ENOMEM;
 			goto fail;
 		}
+		/* using MSI-X */
+		dd->num_msix_entries = total;
 		dd_dev_info(dd, "%u MSI-X interrupts allocated\n", total);
 	}
 
diff --git a/drivers/infiniband/hw/hfi1/hfi.h b/drivers/infiniband/hw/hfi1/hfi.h
index cc69b51..e1cd468 100644
--- a/drivers/infiniband/hw/hfi1/hfi.h
+++ b/drivers/infiniband/hw/hfi1/hfi.h
@@ -521,7 +521,7 @@ static inline void incr_cntr32(u32 *cntr)
 #define MAX_NAME_SIZE 64
 struct hfi1_msix_entry {
 	enum irq_type type;
-	struct msix_entry msix;
+	int irq;
 	void *arg;
 	char name[MAX_NAME_SIZE];
 	cpumask_t mask;
@@ -1839,9 +1839,7 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,
 void hfi1_pcie_ddcleanup(struct hfi1_devdata *);
 void hfi1_pcie_flr(struct hfi1_devdata *dd);
 int pcie_speeds(struct hfi1_devdata *dd);
-void request_msix(struct hfi1_devdata *dd, u32 *nent,
-		  struct hfi1_msix_entry *entry);
-void hfi1_enable_intx(struct pci_dev *pdev);
+int request_msix(struct hfi1_devdata *dd, u32 msireq);
 void restore_pci_variables(struct hfi1_devdata *dd);
 int do_pcie_gen3_transition(struct hfi1_devdata *dd);
 int parse_platform_config(struct hfi1_devdata *dd);
diff --git a/drivers/infiniband/hw/hfi1/pcie.c b/drivers/infiniband/hw/hfi1/pcie.c
index e39e01b..e666264 100644
--- a/drivers/infiniband/hw/hfi1/pcie.c
+++ b/drivers/infiniband/hw/hfi1/pcie.c
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015, 2016 Intel Corporation.
+ * Copyright(c) 2015 - 2017 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -270,50 +270,6 @@ void hfi1_pcie_flr(struct hfi1_devdata *dd)
 	msleep(100);
 }
 
-static void msix_setup(struct hfi1_devdata *dd, int pos, u32 *msixcnt,
-		       struct hfi1_msix_entry *hfi1_msix_entry)
-{
-	int ret;
-	int nvec = *msixcnt;
-	struct msix_entry *msix_entry;
-	int i;
-
-	/*
-	 * We can't pass hfi1_msix_entry array to msix_setup
-	 * so use a dummy msix_entry array and copy the allocated
-	 * irq back to the hfi1_msix_entry array.
-	 */
-	msix_entry = kmalloc_array(nvec, sizeof(*msix_entry), GFP_KERNEL);
-	if (!msix_entry) {
-		ret = -ENOMEM;
-		goto do_intx;
-	}
-
-	for (i = 0; i < nvec; i++)
-		msix_entry[i] = hfi1_msix_entry[i].msix;
-
-	ret = pci_enable_msix_range(dd->pcidev, msix_entry, 1, nvec);
-	if (ret < 0)
-		goto free_msix_entry;
-	nvec = ret;
-
-	for (i = 0; i < nvec; i++)
-		hfi1_msix_entry[i].msix = msix_entry[i];
-
-	kfree(msix_entry);
-	*msixcnt = nvec;
-	return;
-
-free_msix_entry:
-	kfree(msix_entry);
-
-do_intx:
-	dd_dev_err(dd, "pci_enable_msix_range %d vectors failed: %d, falling back to INTx\n",
-		   nvec, ret);
-	*msixcnt = 0;
-	hfi1_enable_intx(dd->pcidev);
-}
-
 /* return the PCIe link speed from the given link status */
 static u32 extract_speed(u16 linkstat)
 {
@@ -394,33 +350,29 @@ int pcie_speeds(struct hfi1_devdata *dd)
 }
 
 /*
- * Returns in *nent:
- *	- actual number of interrupts allocated
+ * Returns:
+ *	- actual number of interrupts allocated or
  *	- 0 if fell back to INTx.
+ *      - error
  */
-void request_msix(struct hfi1_devdata *dd, u32 *nent,
-		  struct hfi1_msix_entry *entry)
+int request_msix(struct hfi1_devdata *dd, u32 msireq)
 {
-	int pos;
+	int nvec;
 
-	pos = dd->pcidev->msix_cap;
-	if (*nent && pos) {
-		msix_setup(dd, pos, nent, entry);
-		/* did it, either MSI-X or INTx */
-	} else {
-		*nent = 0;
-		hfi1_enable_intx(dd->pcidev);
+	nvec = pci_alloc_irq_vectors(dd->pcidev, 1, msireq,
+				     PCI_IRQ_MSIX | PCI_IRQ_LEGACY);
+	if (nvec < 0) {
+		dd_dev_err(dd, "pci_alloc_irq_vectors() failed: %d\n", nvec);
+		return nvec;
 	}
 
 	tune_pcie_caps(dd);
-}
 
-void hfi1_enable_intx(struct pci_dev *pdev)
-{
-	/* first, turn on INTx */
-	pci_intx(pdev, 1);
-	/* then turn off MSI-X */
-	pci_disable_msix(pdev);
+	/* check for legacy IRQ */
+	if (nvec == 1 && !dd->pcidev->msix_enabled)
+		return 0;
+
+	return nvec;
 }
 
 /* restore command and BARs after a reset has wiped them out */

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

[Dennis Dalessandro]

From: Michael N. Henry <michael.n.henry-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Provide the ability for IB clients to modify the OPA specific
capability mask and include this mask in the subsequent trap data.

Reviewed-by: Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Michael N. Henry <michael.n.henry-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/mad.c   |    7 ++-----
 drivers/infiniband/hw/hfi1/mad.h   |    2 +-
 drivers/infiniband/hw/hfi1/verbs.c |    7 ++++++-
 drivers/infiniband/sw/rdmavt/vt.c  |    9 +++++++--
 include/rdma/ib_verbs.h            |    3 ++-
 include/rdma/rdma_vt.h             |    1 +
 6 files changed, 19 insertions(+), 10 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/mad.c b/drivers/infiniband/hw/hfi1/mad.c
index 5977673..70831ad 100644
--- a/drivers/infiniband/hw/hfi1/mad.c
+++ b/drivers/infiniband/hw/hfi1/mad.c
@@ -260,6 +260,7 @@ void hfi1_cap_mask_chg(struct rvt_dev_info *rdi, u8 port_num)
 	data.issuer_lid = cpu_to_be32(lid);
 	data.ntc_144.lid = data.issuer_lid;
 	data.ntc_144.new_cap_mask = cpu_to_be32(ibp->rvp.port_cap_flags);
+	data.ntc_144.cap_mask3 = cpu_to_be16(ibp->rvp.port_cap3_flags);
 
 	send_trap(ibp, &data, sizeof(data));
 }
@@ -704,11 +705,7 @@ static int __subn_get_opa_portinfo(struct opa_smp *smp, u32 am, u8 *data,
 	buffer_units |= (dd->vl15_init << 11) & OPA_PI_MASK_BUF_UNIT_VL15_INIT;
 	pi->buffer_units = cpu_to_be32(buffer_units);
 
-	pi->opa_cap_mask = cpu_to_be16(OPA_CAP_MASK3_IsSharedSpaceSupported |
-				       OPA_CAP_MASK3_IsEthOnFabricSupported);
-	/* Driver does not support mcast/collective configuration */
-	pi->opa_cap_mask &=
-		cpu_to_be16(~OPA_CAP_MASK3_IsAddrRangeConfigSupported);
+	pi->opa_cap_mask = cpu_to_be16(ibp->rvp.port_cap3_flags);
 	pi->collectivemask_multicastmask = ((HFI1_COLLECTIVE_NR & 0x7)
 					    << 3 | (HFI1_MCAST_NR & 0x7));
 
diff --git a/drivers/infiniband/hw/hfi1/mad.h b/drivers/infiniband/hw/hfi1/mad.h
index 5aa3fd1..a4e2506 100644
--- a/drivers/infiniband/hw/hfi1/mad.h
+++ b/drivers/infiniband/hw/hfi1/mad.h
@@ -115,7 +115,7 @@ struct opa_mad_notice_attr {
 			__be32	lid;		/* LID where change occurred */
 			__be32	new_cap_mask;	/* new capability mask */
 			__be16	reserved2;
-			__be16	cap_mask;
+			__be16	cap_mask3;
 			__be16	change_flags;	/* low 4 bits only */
 		} __packed ntc_144;
 
diff --git a/drivers/infiniband/hw/hfi1/verbs.c b/drivers/infiniband/hw/hfi1/verbs.c
index af54d3f..2881798 100644
--- a/drivers/infiniband/hw/hfi1/verbs.c
+++ b/drivers/infiniband/hw/hfi1/verbs.c
@@ -1537,9 +1537,14 @@ static void init_ibport(struct hfi1_pportdata *ppd)
 	/* Set the prefix to the default value (see ch. 4.1.1) */
 	ibp->rvp.gid_prefix = IB_DEFAULT_GID_PREFIX;
 	ibp->rvp.sm_lid = 0;
-	/* Below should only set bits defined in OPA PortInfo.CapabilityMask */
+	/*
+	 * Below should only set bits defined in OPA PortInfo.CapabilityMask
+	 * and PortInfo.CapabilityMask3
+	 */
 	ibp->rvp.port_cap_flags = IB_PORT_AUTO_MIGR_SUP |
 		IB_PORT_CAP_MASK_NOTICE_SUP;
+	ibp->rvp.port_cap3_flags = OPA_CAP_MASK3_IsSharedSpaceSupported |
+				   OPA_CAP_MASK3_IsEthOnFabricSupported;
 	ibp->rvp.pma_counter_select[0] = IB_PMA_PORT_XMIT_DATA;
 	ibp->rvp.pma_counter_select[1] = IB_PMA_PORT_RCV_DATA;
 	ibp->rvp.pma_counter_select[2] = IB_PMA_PORT_XMIT_PKTS;
diff --git a/drivers/infiniband/sw/rdmavt/vt.c b/drivers/infiniband/sw/rdmavt/vt.c
index 0d7c6bb..64bdd44 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -202,8 +202,13 @@ static int rvt_modify_port(struct ib_device *ibdev, u8 port_num,
 		return -EINVAL;
 
 	rvp = rdi->ports[port_index];
-	rvp->port_cap_flags |= props->set_port_cap_mask;
-	rvp->port_cap_flags &= ~props->clr_port_cap_mask;
+	if (port_modify_mask & IB_PORT_OPA_MASK_CHG) {
+		rvp->port_cap3_flags |= props->set_port_cap_mask;
+		rvp->port_cap3_flags &= ~props->clr_port_cap_mask;
+	} else {
+		rvp->port_cap_flags |= props->set_port_cap_mask;
+		rvp->port_cap_flags &= ~props->clr_port_cap_mask;
+	}
 
 	if (props->set_port_cap_mask || props->clr_port_cap_mask)
 		rdi->driver_f.cap_mask_chg(rdi, port_num);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 785e927..6c4f6da 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -576,7 +576,8 @@ struct ib_device_modify {
 enum ib_port_modify_flags {
 	IB_PORT_SHUTDOWN		= 1,
 	IB_PORT_INIT_TYPE		= (1<<2),
-	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
+	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
+	IB_PORT_OPA_MASK_CHG		= (1<<4)
 };
 
 struct ib_port_modify {
diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index d0b9f91..0f18ffd 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -75,6 +75,7 @@ struct rvt_ibport {
 	__be64 mkey;
 	u64 tid;
 	u32 port_cap_flags;
+	u16 port_cap3_flags;
 	u32 pma_sample_start;
 	u32 pma_sample_interval;
 	__be16 pma_counter_select[5];

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 6/9] IB/hfi1: Remove atomic SDMA_REQ_SEND_DONE bit operation

[Dennis Dalessandro]

From: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

The atomic SDMA_REQ_SEND_DONE bit is set by the
process-level code, and then the same process-level
code uses the bit to test that all packets have been
submitted incurring a costly atomic read.

Use a bool type with a READ_ONCE/WRITE_ONCE
pairing for this bit, and use the same condition that
is used to set the bit to test that all packets have
been submitted.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index 92517ce..6fb70f0 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -154,10 +154,8 @@
 #define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
 
 /* SDMA request flag bits */
-#define SDMA_REQ_FOR_THREAD 1
-#define SDMA_REQ_SEND_DONE  2
-#define SDMA_REQ_HAS_ERROR  3
-#define SDMA_REQ_DONE_ERROR 4
+#define SDMA_REQ_HAS_ERROR  1
+#define SDMA_REQ_DONE_ERROR 2
 
 #define SDMA_PKT_Q_INACTIVE BIT(0)
 #define SDMA_PKT_Q_ACTIVE   BIT(1)
@@ -258,6 +256,7 @@ struct user_sdma_request {
 	u16 tididx;
 	/* progress index moving along the iovs array */
 	u8 iov_idx;
+	u8 done;
 
 	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
 } ____cacheline_aligned_in_smp;
@@ -628,6 +627,7 @@ int hfi1_user_sdma_process_request(struct hfi1_filedata *fd,
 	req->seqsubmitted = 0;
 	req->flags = 0;
 	req->tids = NULL;
+	req->done = 0;
 	INIT_LIST_HEAD(&req->txps);
 
 	memcpy(&req->info, &info, sizeof(info));
@@ -809,7 +809,7 @@ int hfi1_user_sdma_process_request(struct hfi1_filedata *fd,
 	 * request have been submitted to the SDMA engine. However, it
 	 * will not wait for send completions.
 	 */
-	while (!test_bit(SDMA_REQ_SEND_DONE, &req->flags)) {
+	while (req->seqsubmitted != req->info.npkts) {
 		ret = user_sdma_send_pkts(req, pcount);
 		if (ret < 0) {
 			if (ret != -EBUSY) {
@@ -1118,7 +1118,7 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 	ret = sdma_send_txlist(req->sde, &pq->busy, &req->txps, &count);
 	req->seqsubmitted += count;
 	if (req->seqsubmitted == req->info.npkts) {
-		set_bit(SDMA_REQ_SEND_DONE, &req->flags);
+		WRITE_ONCE(req->done, 1);
 		/*
 		 * The txreq has already been submitted to the HW queue
 		 * so we can free the AHG entry now. Corruption will not
@@ -1585,7 +1585,7 @@ static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status)
 		if (status != SDMA_TXREQ_S_OK)
 			req->status = status;
 		if (req->seqcomp == (ACCESS_ONCE(req->seqsubmitted) - 1) &&
-		    (test_bit(SDMA_REQ_SEND_DONE, &req->flags) ||
+		    (READ_ONCE(req->done) ||
 		     test_bit(SDMA_REQ_DONE_ERROR, &req->flags))) {
 			user_sdma_free_request(req, false);
 			pq_update(pq);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 7/9] IB/hfi1: Remove atomic SDMA_REQ_HAS_ERROR bit operation

[Dennis Dalessandro]

From: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Atomic bit tests are used to single errors and the
completion of request submissions. These operations
don't need to be atomic and show to be expensive on
the profile.

Replace each atomic bit operation with a bool type
and a READ_ONCE/WRITE_ONCE pairing.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |   22 +++++++---------------
 1 files changed, 7 insertions(+), 15 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index 6fb70f0..fcadbb9 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -153,10 +153,6 @@
 #define TXREQ_FLAGS_REQ_ACK   BIT(0)      /* Set the ACK bit in the header */
 #define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
 
-/* SDMA request flag bits */
-#define SDMA_REQ_HAS_ERROR  1
-#define SDMA_REQ_DONE_ERROR 2
-
 #define SDMA_PKT_Q_INACTIVE BIT(0)
 #define SDMA_PKT_Q_ACTIVE   BIT(1)
 #define SDMA_PKT_Q_DEFERRED BIT(2)
@@ -232,7 +228,6 @@ struct user_sdma_request {
 	/* Writeable fields shared with interrupt */
 	u64 seqcomp ____cacheline_aligned_in_smp;
 	u64 seqsubmitted;
-	unsigned long flags;
 	/* status of the last txreq completed */
 	int status;
 
@@ -257,6 +252,7 @@ struct user_sdma_request {
 	/* progress index moving along the iovs array */
 	u8 iov_idx;
 	u8 done;
+	u8 has_error;
 
 	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
 } ____cacheline_aligned_in_smp;
@@ -625,9 +621,9 @@ int hfi1_user_sdma_process_request(struct hfi1_filedata *fd,
 	req->seqnum = 0;
 	req->seqcomp = 0;
 	req->seqsubmitted = 0;
-	req->flags = 0;
 	req->tids = NULL;
 	req->done = 0;
+	req->has_error = 0;
 	INIT_LIST_HEAD(&req->txps);
 
 	memcpy(&req->info, &info, sizeof(info));
@@ -814,7 +810,7 @@ int hfi1_user_sdma_process_request(struct hfi1_filedata *fd,
 		if (ret < 0) {
 			if (ret != -EBUSY) {
 				req->status = ret;
-				set_bit(SDMA_REQ_DONE_ERROR, &req->flags);
+				WRITE_ONCE(req->has_error, 1);
 				if (ACCESS_ONCE(req->seqcomp) ==
 				    req->seqsubmitted - 1)
 					goto free_req;
@@ -916,10 +912,8 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 	pq = req->pq;
 
 	/* If tx completion has reported an error, we are done. */
-	if (test_bit(SDMA_REQ_HAS_ERROR, &req->flags)) {
-		set_bit(SDMA_REQ_DONE_ERROR, &req->flags);
+	if (READ_ONCE(req->has_error))
 		return -EFAULT;
-	}
 
 	/*
 	 * Check if we might have sent the entire request already
@@ -942,10 +936,8 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 		 * with errors. If so, we are not going to process any
 		 * more packets from this request.
 		 */
-		if (test_bit(SDMA_REQ_HAS_ERROR, &req->flags)) {
-			set_bit(SDMA_REQ_DONE_ERROR, &req->flags);
+		if (READ_ONCE(req->has_error))
 			return -EFAULT;
-		}
 
 		tx = kmem_cache_alloc(pq->txreq_cache, GFP_KERNEL);
 		if (!tx)
@@ -1566,7 +1558,7 @@ static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status)
 	if (status != SDMA_TXREQ_S_OK) {
 		SDMA_DBG(req, "SDMA completion with error %d",
 			 status);
-		set_bit(SDMA_REQ_HAS_ERROR, &req->flags);
+		WRITE_ONCE(req->has_error, 1);
 	}
 
 	req->seqcomp = tx->seqnum;
@@ -1586,7 +1578,7 @@ static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status)
 			req->status = status;
 		if (req->seqcomp == (ACCESS_ONCE(req->seqsubmitted) - 1) &&
 		    (READ_ONCE(req->done) ||
-		     test_bit(SDMA_REQ_DONE_ERROR, &req->flags))) {
+		     READ_ONCE(req->has_error))) {
 			user_sdma_free_request(req, false);
 			pq_update(pq);
 			set_comp_state(pq, cq, idx, ERROR, req->status);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 8/9] IB/hfi1: Reclassify type of messages printed for platform config logic

[Dennis Dalessandro]

From: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Reclassify messages printed out to /var/log/messages into
warnings and errors to facilitate debugging in the future
for issues related to the platform config logic.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/platform.c |   20 ++++++++++----------
 1 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/platform.c b/drivers/infiniband/hw/hfi1/platform.c
index 838fe84..cbda9b1 100644
--- a/drivers/infiniband/hw/hfi1/platform.c
+++ b/drivers/infiniband/hw/hfi1/platform.c
@@ -242,7 +242,7 @@ static int qual_power(struct hfi1_pportdata *ppd)
 
 	if (ppd->offline_disabled_reason ==
 			HFI1_ODR_MASK(OPA_LINKDOWN_REASON_POWER_POLICY)) {
-		dd_dev_info(
+		dd_dev_err(
 			ppd->dd,
 			"%s: Port disabled due to system power restrictions\n",
 			__func__);
@@ -268,7 +268,7 @@ static int qual_bitrate(struct hfi1_pportdata *ppd)
 
 	if (ppd->offline_disabled_reason ==
 			HFI1_ODR_MASK(OPA_LINKDOWN_REASON_LINKSPEED_POLICY)) {
-		dd_dev_info(
+		dd_dev_err(
 			ppd->dd,
 			"%s: Cable failed bitrate check, disabling port\n",
 			__func__);
@@ -709,15 +709,15 @@ static void apply_tunings(
 		ret = load_8051_config(ppd->dd, DC_HOST_COMM_SETTINGS,
 				       GENERAL_CONFIG, config_data);
 		if (ret != HCMD_SUCCESS)
-			dd_dev_info(ppd->dd,
-				    "%s: Failed set ext device config params\n",
-				    __func__);
+			dd_dev_err(ppd->dd,
+				   "%s: Failed set ext device config params\n",
+				   __func__);
 	}
 
 	if (tx_preset_index == OPA_INVALID_INDEX) {
 		if (ppd->port_type == PORT_TYPE_QSFP && limiting_active)
-			dd_dev_info(ppd->dd, "%s: Invalid Tx preset index\n",
-				    __func__);
+			dd_dev_err(ppd->dd, "%s: Invalid Tx preset index\n",
+				   __func__);
 		return;
 	}
 
@@ -900,7 +900,7 @@ static int tune_qsfp(struct hfi1_pportdata *ppd,
 	case 0xD: /* fallthrough */
 	case 0xF:
 	default:
-		dd_dev_info(ppd->dd, "%s: Unknown/unsupported cable\n",
+		dd_dev_warn(ppd->dd, "%s: Unknown/unsupported cable\n",
 			    __func__);
 		break;
 	}
@@ -942,7 +942,7 @@ void tune_serdes(struct hfi1_pportdata *ppd)
 	case PORT_TYPE_DISCONNECTED:
 		ppd->offline_disabled_reason =
 			HFI1_ODR_MASK(OPA_LINKDOWN_REASON_DISCONNECTED);
-		dd_dev_info(dd, "%s: Port disconnected, disabling port\n",
+		dd_dev_warn(dd, "%s: Port disconnected, disabling port\n",
 			    __func__);
 		goto bail;
 	case PORT_TYPE_FIXED:
@@ -1027,7 +1027,7 @@ void tune_serdes(struct hfi1_pportdata *ppd)
 		}
 		break;
 	default:
-		dd_dev_info(ppd->dd, "%s: Unknown port type\n", __func__);
+		dd_dev_warn(ppd->dd, "%s: Unknown port type\n", __func__);
 		ppd->port_type = PORT_TYPE_UNKNOWN;
 		tuning_method = OPA_UNKNOWN_TUNING;
 		total_atten = 0;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH for-next 9/9] IB/hfi1: Fix up sdma_init function comment

[Dennis Dalessandro]

From: Ira Weiny <ira.weiny-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

sdma_init does not take a number of sdma engine parameters,
rather it initializes all of the sdma engines.

Signed-off-by: Ira Weiny <ira.weiny-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/hw/hfi1/sdma.c |    6 ++----
 1 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/sdma.c b/drivers/infiniband/hw/hfi1/sdma.c
index bfd0d51..d82ff57 100644
--- a/drivers/infiniband/hw/hfi1/sdma.c
+++ b/drivers/infiniband/hw/hfi1/sdma.c
@@ -1340,10 +1340,8 @@ static void sdma_clean(struct hfi1_devdata *dd, size_t num_engines)
  * @dd: hfi1_devdata
  * @port: port number (currently only zero)
  *
- * sdma_init initializes the specified number of engines.
- *
- * The code initializes each sde, its csrs.  Interrupts
- * are not required to be enabled.
+ * Initializes each sde and its csrs.
+ * Interrupts are not required to be enabled.
  *
  * Returns:
  * 0 - success, -errno on failure

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 5602 bytes --]

On Fri, May 26, 2017 at 05:35:37AM -0700, Dennis Dalessandro wrote:
> From: Michael N. Henry <michael.n.henry-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
> Provide the ability for IB clients to modify the OPA specific
> capability mask and include this mask in the subsequent trap data.
>
> Reviewed-by: Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Michael N. Henry <michael.n.henry-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>  drivers/infiniband/hw/hfi1/mad.c   |    7 ++-----
>  drivers/infiniband/hw/hfi1/mad.h   |    2 +-
>  drivers/infiniband/hw/hfi1/verbs.c |    7 ++++++-
>  drivers/infiniband/sw/rdmavt/vt.c  |    9 +++++++--
>  include/rdma/ib_verbs.h            |    3 ++-
>  include/rdma/rdma_vt.h             |    1 +
>  6 files changed, 19 insertions(+), 10 deletions(-)
>
> diff --git a/drivers/infiniband/hw/hfi1/mad.c b/drivers/infiniband/hw/hfi1/mad.c
> index 5977673..70831ad 100644
> --- a/drivers/infiniband/hw/hfi1/mad.c
> +++ b/drivers/infiniband/hw/hfi1/mad.c
> @@ -260,6 +260,7 @@ void hfi1_cap_mask_chg(struct rvt_dev_info *rdi, u8 port_num)
>  	data.issuer_lid = cpu_to_be32(lid);
>  	data.ntc_144.lid = data.issuer_lid;
>  	data.ntc_144.new_cap_mask = cpu_to_be32(ibp->rvp.port_cap_flags);
> +	data.ntc_144.cap_mask3 = cpu_to_be16(ibp->rvp.port_cap3_flags);
>
>  	send_trap(ibp, &data, sizeof(data));
>  }
> @@ -704,11 +705,7 @@ static int __subn_get_opa_portinfo(struct opa_smp *smp, u32 am, u8 *data,
>  	buffer_units |= (dd->vl15_init << 11) & OPA_PI_MASK_BUF_UNIT_VL15_INIT;
>  	pi->buffer_units = cpu_to_be32(buffer_units);
>
> -	pi->opa_cap_mask = cpu_to_be16(OPA_CAP_MASK3_IsSharedSpaceSupported |
> -				       OPA_CAP_MASK3_IsEthOnFabricSupported);
> -	/* Driver does not support mcast/collective configuration */
> -	pi->opa_cap_mask &=
> -		cpu_to_be16(~OPA_CAP_MASK3_IsAddrRangeConfigSupported);
> +	pi->opa_cap_mask = cpu_to_be16(ibp->rvp.port_cap3_flags);
>  	pi->collectivemask_multicastmask = ((HFI1_COLLECTIVE_NR & 0x7)
>  					    << 3 | (HFI1_MCAST_NR & 0x7));
>
> diff --git a/drivers/infiniband/hw/hfi1/mad.h b/drivers/infiniband/hw/hfi1/mad.h
> index 5aa3fd1..a4e2506 100644
> --- a/drivers/infiniband/hw/hfi1/mad.h
> +++ b/drivers/infiniband/hw/hfi1/mad.h
> @@ -115,7 +115,7 @@ struct opa_mad_notice_attr {
>  			__be32	lid;		/* LID where change occurred */
>  			__be32	new_cap_mask;	/* new capability mask */
>  			__be16	reserved2;
> -			__be16	cap_mask;
> +			__be16	cap_mask3;
>  			__be16	change_flags;	/* low 4 bits only */
>  		} __packed ntc_144;
>
> diff --git a/drivers/infiniband/hw/hfi1/verbs.c b/drivers/infiniband/hw/hfi1/verbs.c
> index af54d3f..2881798 100644
> --- a/drivers/infiniband/hw/hfi1/verbs.c
> +++ b/drivers/infiniband/hw/hfi1/verbs.c
> @@ -1537,9 +1537,14 @@ static void init_ibport(struct hfi1_pportdata *ppd)
>  	/* Set the prefix to the default value (see ch. 4.1.1) */
>  	ibp->rvp.gid_prefix = IB_DEFAULT_GID_PREFIX;
>  	ibp->rvp.sm_lid = 0;
> -	/* Below should only set bits defined in OPA PortInfo.CapabilityMask */
> +	/*
> +	 * Below should only set bits defined in OPA PortInfo.CapabilityMask
> +	 * and PortInfo.CapabilityMask3
> +	 */
>  	ibp->rvp.port_cap_flags = IB_PORT_AUTO_MIGR_SUP |
>  		IB_PORT_CAP_MASK_NOTICE_SUP;
> +	ibp->rvp.port_cap3_flags = OPA_CAP_MASK3_IsSharedSpaceSupported |
> +				   OPA_CAP_MASK3_IsEthOnFabricSupported;
>  	ibp->rvp.pma_counter_select[0] = IB_PMA_PORT_XMIT_DATA;
>  	ibp->rvp.pma_counter_select[1] = IB_PMA_PORT_RCV_DATA;
>  	ibp->rvp.pma_counter_select[2] = IB_PMA_PORT_XMIT_PKTS;
> diff --git a/drivers/infiniband/sw/rdmavt/vt.c b/drivers/infiniband/sw/rdmavt/vt.c
> index 0d7c6bb..64bdd44 100644
> --- a/drivers/infiniband/sw/rdmavt/vt.c
> +++ b/drivers/infiniband/sw/rdmavt/vt.c
> @@ -202,8 +202,13 @@ static int rvt_modify_port(struct ib_device *ibdev, u8 port_num,
>  		return -EINVAL;
>
>  	rvp = rdi->ports[port_index];
> -	rvp->port_cap_flags |= props->set_port_cap_mask;
> -	rvp->port_cap_flags &= ~props->clr_port_cap_mask;
> +	if (port_modify_mask & IB_PORT_OPA_MASK_CHG) {
> +		rvp->port_cap3_flags |= props->set_port_cap_mask;
> +		rvp->port_cap3_flags &= ~props->clr_port_cap_mask;
> +	} else {
> +		rvp->port_cap_flags |= props->set_port_cap_mask;
> +		rvp->port_cap_flags &= ~props->clr_port_cap_mask;
> +	}
>
>  	if (props->set_port_cap_mask || props->clr_port_cap_mask)
>  		rdi->driver_f.cap_mask_chg(rdi, port_num);
> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index 785e927..6c4f6da 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -576,7 +576,8 @@ struct ib_device_modify {
>  enum ib_port_modify_flags {
>  	IB_PORT_SHUTDOWN		= 1,
>  	IB_PORT_INIT_TYPE		= (1<<2),
> -	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
> +	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
> +	IB_PORT_OPA_MASK_CHG		= (1<<4)

How will this flag be set?

>  };
>
>  struct ib_port_modify {
> diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
> index d0b9f91..0f18ffd 100644
> --- a/include/rdma/rdma_vt.h
> +++ b/include/rdma/rdma_vt.h
> @@ -75,6 +75,7 @@ struct rvt_ibport {
>  	__be64 mkey;
>  	u64 tid;
>  	u32 port_cap_flags;
> +	u16 port_cap3_flags;
>  	u32 pma_sample_start;
>  	u32 pma_sample_interval;
>  	__be16 pma_counter_select[5];
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH for-next 2/9] IB/hfi1: Optimize cachelines for user SDMA request structure

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 8084 bytes --]

On Fri, May 26, 2017 at 05:35:18AM -0700, Dennis Dalessandro wrote:
> From: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
> The current user SDMA request structure layout has holes.
> The cachelines can be reduced to improve cacheline trading.
> Separate fields in the following categories: mostly read,
> writable and shared with interrupt.
>
> Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Sebastian Sanchez <sebastian.sanchez-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>  drivers/infiniband/hw/hfi1/user_sdma.c |  106 ++++++++++++++++++--------------
>  1 files changed, 58 insertions(+), 48 deletions(-)
>
> diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
> index 79450cf..92517ce 100644
> --- a/drivers/infiniband/hw/hfi1/user_sdma.c
> +++ b/drivers/infiniband/hw/hfi1/user_sdma.c
> @@ -117,6 +117,7 @@
>
>  #define AHG_KDETH_INTR_SHIFT 12
>  #define AHG_KDETH_SH_SHIFT   13
> +#define AHG_KDETH_ARRAY_SIZE  9
>
>  #define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
>  #define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)
> @@ -204,25 +205,42 @@ struct evict_data {
>  };
>
>  struct user_sdma_request {
> -	struct sdma_req_info info;
> -	struct hfi1_user_sdma_pkt_q *pq;
> -	struct hfi1_user_sdma_comp_q *cq;
>  	/* This is the original header from user space */
>  	struct hfi1_pkt_header hdr;
> +
> +	/* Read mostly fields */
> +	struct hfi1_user_sdma_pkt_q *pq ____cacheline_aligned_in_smp;
> +	struct hfi1_user_sdma_comp_q *cq;
>  	/*
>  	 * Pointer to the SDMA engine for this request.
>  	 * Since different request could be on different VLs,
>  	 * each request will need it's own engine pointer.
>  	 */
>  	struct sdma_engine *sde;
> -	s8 ahg_idx;
> -	u32 ahg[9];
> +	struct sdma_req_info info;
> +	/* TID array values copied from the tid_iov vector */
> +	u32 *tids;
> +	/* total length of the data in the request */
> +	u32 data_len;
> +	/* number of elements copied to the tids array */
> +	u16 n_tids;
>  	/*
> -	 * KDETH.Offset (Eager) field
> -	 * We need to remember the initial value so the headers
> -	 * can be updated properly.
> +	 * We copy the iovs for this request (based on
> +	 * info.iovcnt). These are only the data vectors
>  	 */
> -	u32 koffset;
> +	u8 data_iovs;
> +	s8 ahg_idx;
> +
> +	/* Writeable fields shared with interrupt */
> +	u64 seqcomp ____cacheline_aligned_in_smp;
> +	u64 seqsubmitted;
> +	unsigned long flags;
> +	/* status of the last txreq completed */
> +	int status;
> +
> +	/* Send side fields */
> +	struct list_head txps ____cacheline_aligned_in_smp;
> +	u64 seqnum;
>  	/*
>  	 * KDETH.OFFSET (TID) field
>  	 * The offset can cover multiple packets, depending on the
> @@ -230,29 +248,19 @@ struct user_sdma_request {
>  	 */
>  	u32 tidoffset;
>  	/*
> -	 * We copy the iovs for this request (based on
> -	 * info.iovcnt). These are only the data vectors
> +	 * KDETH.Offset (Eager) field
> +	 * We need to remember the initial value so the headers
> +	 * can be updated properly.
>  	 */
> -	unsigned data_iovs;
> -	/* total length of the data in the request */
> -	u32 data_len;
> +	u32 koffset;
> +	u32 sent;
> +	/* TID index copied from the tid_iov vector */
> +	u16 tididx;
>  	/* progress index moving along the iovs array */
> -	unsigned iov_idx;
> +	u8 iov_idx;
> +
>  	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
> -	/* number of elements copied to the tids array */
> -	u16 n_tids;
> -	/* TID array values copied from the tid_iov vector */
> -	u32 *tids;
> -	u16 tididx;
> -	u32 sent;
> -	u64 seqnum;
> -	u64 seqcomp;
> -	u64 seqsubmitted;
> -	struct list_head txps;
> -	unsigned long flags;
> -	/* status of the last txreq completed */
> -	int status;
> -};
> +} ____cacheline_aligned_in_smp;

Out of curiosity, do you have pahole data before vs. after to share?

>
>  /*
>   * A single txreq could span up to 3 physical pages when the MTU
> @@ -1034,11 +1042,6 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
>  							       datalen);
>  				if (changes < 0)
>  					goto free_tx;
> -				sdma_txinit_ahg(&tx->txreq,
> -						SDMA_TXREQ_F_USE_AHG,
> -						datalen, req->ahg_idx, changes,
> -						req->ahg, sizeof(req->hdr),
> -						user_sdma_txreq_cb);
>  			}
>  		} else {
>  			ret = sdma_txinit(&tx->txreq, 0, sizeof(req->hdr) +
> @@ -1442,21 +1445,22 @@ static int set_txreq_header(struct user_sdma_request *req,
>  }
>
>  static int set_txreq_header_ahg(struct user_sdma_request *req,
> -				struct user_sdma_txreq *tx, u32 len)
> +				struct user_sdma_txreq *tx, u32 datalen)
>  {
> +	u32 ahg[AHG_KDETH_ARRAY_SIZE];
>  	int diff = 0;
>  	u8 omfactor; /* KDETH.OM */
>  	struct hfi1_user_sdma_pkt_q *pq = req->pq;
>  	struct hfi1_pkt_header *hdr = &req->hdr;
>  	u16 pbclen = le16_to_cpu(hdr->pbc[0]);
> -	u32 val32, tidval = 0, lrhlen = get_lrh_len(*hdr, pad_len(len));
> +	u32 val32, tidval = 0, lrhlen = get_lrh_len(*hdr, pad_len(datalen));
>
>  	if (PBC2LRH(pbclen) != lrhlen) {
>  		/* PBC.PbcLengthDWs */
> -		AHG_HEADER_SET(req->ahg, diff, 0, 0, 12,
> +		AHG_HEADER_SET(ahg, diff, 0, 0, 12,
>  			       cpu_to_le16(LRH2PBC(lrhlen)));
>  		/* LRH.PktLen (we need the full 16 bits due to byte swap) */
> -		AHG_HEADER_SET(req->ahg, diff, 3, 0, 16,
> +		AHG_HEADER_SET(ahg, diff, 3, 0, 16,
>  			       cpu_to_be16(lrhlen >> 2));
>  	}
>
> @@ -1468,13 +1472,12 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
>  		(HFI1_CAP_IS_KSET(EXTENDED_PSN) ? 0x7fffffff : 0xffffff);
>  	if (unlikely(tx->flags & TXREQ_FLAGS_REQ_ACK))
>  		val32 |= 1UL << 31;
> -	AHG_HEADER_SET(req->ahg, diff, 6, 0, 16, cpu_to_be16(val32 >> 16));
> -	AHG_HEADER_SET(req->ahg, diff, 6, 16, 16, cpu_to_be16(val32 & 0xffff));
> +	AHG_HEADER_SET(ahg, diff, 6, 0, 16, cpu_to_be16(val32 >> 16));
> +	AHG_HEADER_SET(ahg, diff, 6, 16, 16, cpu_to_be16(val32 & 0xffff));
>  	/* KDETH.Offset */
> -	AHG_HEADER_SET(req->ahg, diff, 15, 0, 16,
> +	AHG_HEADER_SET(ahg, diff, 15, 0, 16,
>  		       cpu_to_le16(req->koffset & 0xffff));
> -	AHG_HEADER_SET(req->ahg, diff, 15, 16, 16,
> -		       cpu_to_le16(req->koffset >> 16));
> +	AHG_HEADER_SET(ahg, diff, 15, 16, 16, cpu_to_le16(req->koffset >> 16));
>  	if (req_opcode(req->info.ctrl) == EXPECTED) {
>  		__le16 val;
>
> @@ -1492,9 +1495,8 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
>  			 * we have to check again.
>  			 */
>  			if (++req->tididx > req->n_tids - 1 ||
> -			    !req->tids[req->tididx]) {
> +			    !req->tids[req->tididx])
>  				return -EINVAL;
> -			}
>  			tidval = req->tids[req->tididx];
>  		}
>  		omfactor = ((EXP_TID_GET(tidval, LEN) *
> @@ -1502,7 +1504,7 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
>  				 KDETH_OM_MAX_SIZE) ? KDETH_OM_LARGE_SHIFT :
>  				 KDETH_OM_SMALL_SHIFT;
>  		/* KDETH.OM and KDETH.OFFSET (TID) */
> -		AHG_HEADER_SET(req->ahg, diff, 7, 0, 16,
> +		AHG_HEADER_SET(ahg, diff, 7, 0, 16,
>  			       ((!!(omfactor - KDETH_OM_SMALL_SHIFT)) << 15 |
>  				((req->tidoffset >> omfactor)
>  				 & 0x7fff)));
> @@ -1522,12 +1524,20 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
>  					     AHG_KDETH_INTR_SHIFT));
>  		}
>
> -		AHG_HEADER_SET(req->ahg, diff, 7, 16, 14, val);
> +		AHG_HEADER_SET(ahg, diff, 7, 16, 14, val);
>  	}
> +	if (diff < 0)
> +		return diff;
>
>  	trace_hfi1_sdma_user_header_ahg(pq->dd, pq->ctxt, pq->subctxt,
>  					req->info.comp_idx, req->sde->this_idx,
> -					req->ahg_idx, req->ahg, diff, tidval);
> +					req->ahg_idx, ahg, diff, tidval);
> +	sdma_txinit_ahg(&tx->txreq,
> +			SDMA_TXREQ_F_USE_AHG,
> +			datalen, req->ahg_idx, diff,
> +			ahg, sizeof(req->hdr),
> +			user_sdma_txreq_cb);
> +
>  	return diff;
>  }
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: [PATCH for-next 2/9] IB/hfi1: Optimize cachelines for user SDMA request structure

[Sanchez, Sebastian]

>Out of curiosity, do you have pahole data before vs. after to share?

Yes, below is the pahole output for this patch.

Sebastian

---------

Before:


struct user_sdma_request {
	struct sdma_req_info       info;                 /*     0     8 */
	struct hfi1_user_sdma_pkt_q * pq;                /*     8     8 */
	struct hfi1_user_sdma_comp_q * cq;               /*    16     8 */
	struct hfi1_pkt_header     hdr;                  /*    24    64 */
	/* --- cacheline 1 boundary (64 bytes) was 24 bytes ago --- */
	struct sdma_engine *       sde;                  /*    88     8 */
	s8                         ahg_idx;              /*    96     1 */

	/* XXX 3 bytes hole, try to pack */

	u32                        ahg[9];               /*   100    36 */
	/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
	u32                        koffset;              /*   136     4 */
	u32                        tidoffset;            /*   140     4 */
	unsigned int               data_iovs;            /*   144     4 */
	u32                        data_len;             /*   148     4 */
	unsigned int               iov_idx;              /*   152     4 */

	/* XXX 4 bytes hole, try to pack */

	struct user_sdma_iovec     iovs[8];              /*   160   512 */
	/* --- cacheline 10 boundary (640 bytes) was 32 bytes ago --- */
	u16                        n_tids;               /*   672     2 */

	/* XXX 6 bytes hole, try to pack */

	u32 *                      tids;                 /*   680     8 */
	u16                        tididx;               /*   688     2 */

	/* XXX 2 bytes hole, try to pack */

	u32                        sent;                 /*   692     4 */
	u64                        seqnum;               /*   696     8 */
	/* --- cacheline 11 boundary (704 bytes) --- */
	u64                        seqcomp;              /*   704     8 */
	u64                        seqsubmitted;         /*   712     8 */
	struct list_head           txps;                 /*   720    16 */
	long unsigned int          flags;                /*   736     8 */
	int                        status;               /*   744     4 */

	/* size: 752, cachelines: 12, members: 23 */
	/* sum members: 733, holes: 4, sum holes: 15 */
	/* padding: 4 */
	/* last cacheline: 48 bytes */
};

After:

struct user_sdma_request {
	struct hfi1_pkt_header     hdr;                  /*     0    64 */
	/* --- cacheline 1 boundary (64 bytes) --- */
	struct hfi1_user_sdma_pkt_q * pq;                /*    64     8 */
	struct hfi1_user_sdma_comp_q * cq;               /*    72     8 */
	struct sdma_engine *       sde;                  /*    80     8 */
	struct sdma_req_info       info;                 /*    88     8 */
	u32 *                      tids;                 /*    96     8 */
	u32                        data_len;             /*   104     4 */
	u16                        n_tids;               /*   108     2 */
	u8                         data_iovs;            /*   110     1 */
	s8                         ahg_idx;              /*   111     1 */

	/* XXX 16 bytes hole, try to pack */

	/* --- cacheline 2 boundary (128 bytes) --- */
	u64                        seqcomp;              /*   128     8 */
	u64                        seqsubmitted;         /*   136     8 */
	long unsigned int          flags;                /*   144     8 */
	int                        status;               /*   152     4 */

	/* XXX 36 bytes hole, try to pack */

	/* --- cacheline 3 boundary (192 bytes) --- */
	struct list_head           txps;                 /*   192    16 */
	u64                        seqnum;               /*   208     8 */
	u32                        tidoffset;            /*   216     4 */
	u32                        koffset;              /*   220     4 */
	u32                        sent;                 /*   224     4 */
	u16                        tididx;               /*   228     2 */
	u8                         iov_idx;              /*   230     1 */

	/* XXX 1 byte hole, try to pack */

	struct user_sdma_iovec     iovs[8];              /*   232   512 */
	/* --- cacheline 11 boundary (704 bytes) was 40 bytes ago --- */

	/* size: 768, cachelines: 12, members: 22 */
	/* sum members: 691, holes: 3, sum holes: 53 */
	/* padding: 24 */
};
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

[Henry, Michael N]

On 5/29/2017 9:14 AM, Leon Romanovsky wrote:
> On Fri, May 26, 2017 at 05:35:37AM -0700, Dennis Dalessandro wrote:
>> From: Michael N. Henry <michael.n.henry-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>
>> Provide the ability for IB clients to modify the OPA specific
>> capability mask and include this mask in the subsequent trap data.
>>
>> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
>> index 785e927..6c4f6da 100644
>> --- a/include/rdma/ib_verbs.h
>> +++ b/include/rdma/ib_verbs.h
>> @@ -576,7 +576,8 @@ struct ib_device_modify {
>>   enum ib_port_modify_flags {
>>   	IB_PORT_SHUTDOWN		= 1,
>>   	IB_PORT_INIT_TYPE		= (1<<2),
>> -	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
>> +	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
>> +	IB_PORT_OPA_MASK_CHG		= (1<<4)
> How will this flag be set?
>
This flag can be set as the port_modify_mask when calling ib_modify_port(..)

Thanks,

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-next 2/9] IB/hfi1: Optimize cachelines for user SDMA request structure

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 4523 bytes --]

On Tue, May 30, 2017 at 05:55:20PM +0000, Sanchez, Sebastian wrote:
> >Out of curiosity, do you have pahole data before vs. after to share?
>
> Yes, below is the pahole output for this patch.

Thanks

>
> Sebastian
>
> ---------
>
> Before:
>
>
> struct user_sdma_request {
> 	struct sdma_req_info       info;                 /*     0     8 */
> 	struct hfi1_user_sdma_pkt_q * pq;                /*     8     8 */
> 	struct hfi1_user_sdma_comp_q * cq;               /*    16     8 */
> 	struct hfi1_pkt_header     hdr;                  /*    24    64 */
> 	/* --- cacheline 1 boundary (64 bytes) was 24 bytes ago --- */
> 	struct sdma_engine *       sde;                  /*    88     8 */
> 	s8                         ahg_idx;              /*    96     1 */
>
> 	/* XXX 3 bytes hole, try to pack */
>
> 	u32                        ahg[9];               /*   100    36 */
> 	/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
> 	u32                        koffset;              /*   136     4 */
> 	u32                        tidoffset;            /*   140     4 */
> 	unsigned int               data_iovs;            /*   144     4 */
> 	u32                        data_len;             /*   148     4 */
> 	unsigned int               iov_idx;              /*   152     4 */
>
> 	/* XXX 4 bytes hole, try to pack */
>
> 	struct user_sdma_iovec     iovs[8];              /*   160   512 */
> 	/* --- cacheline 10 boundary (640 bytes) was 32 bytes ago --- */
> 	u16                        n_tids;               /*   672     2 */
>
> 	/* XXX 6 bytes hole, try to pack */
>
> 	u32 *                      tids;                 /*   680     8 */
> 	u16                        tididx;               /*   688     2 */
>
> 	/* XXX 2 bytes hole, try to pack */
>
> 	u32                        sent;                 /*   692     4 */
> 	u64                        seqnum;               /*   696     8 */
> 	/* --- cacheline 11 boundary (704 bytes) --- */
> 	u64                        seqcomp;              /*   704     8 */
> 	u64                        seqsubmitted;         /*   712     8 */
> 	struct list_head           txps;                 /*   720    16 */
> 	long unsigned int          flags;                /*   736     8 */
> 	int                        status;               /*   744     4 */
>
> 	/* size: 752, cachelines: 12, members: 23 */
> 	/* sum members: 733, holes: 4, sum holes: 15 */
> 	/* padding: 4 */
> 	/* last cacheline: 48 bytes */
> };
>
> After:
>
> struct user_sdma_request {
> 	struct hfi1_pkt_header     hdr;                  /*     0    64 */
> 	/* --- cacheline 1 boundary (64 bytes) --- */
> 	struct hfi1_user_sdma_pkt_q * pq;                /*    64     8 */
> 	struct hfi1_user_sdma_comp_q * cq;               /*    72     8 */
> 	struct sdma_engine *       sde;                  /*    80     8 */
> 	struct sdma_req_info       info;                 /*    88     8 */
> 	u32 *                      tids;                 /*    96     8 */
> 	u32                        data_len;             /*   104     4 */
> 	u16                        n_tids;               /*   108     2 */
> 	u8                         data_iovs;            /*   110     1 */
> 	s8                         ahg_idx;              /*   111     1 */
>
> 	/* XXX 16 bytes hole, try to pack */
>
> 	/* --- cacheline 2 boundary (128 bytes) --- */
> 	u64                        seqcomp;              /*   128     8 */
> 	u64                        seqsubmitted;         /*   136     8 */
> 	long unsigned int          flags;                /*   144     8 */
> 	int                        status;               /*   152     4 */
>
> 	/* XXX 36 bytes hole, try to pack */
>
> 	/* --- cacheline 3 boundary (192 bytes) --- */
> 	struct list_head           txps;                 /*   192    16 */
> 	u64                        seqnum;               /*   208     8 */
> 	u32                        tidoffset;            /*   216     4 */
> 	u32                        koffset;              /*   220     4 */
> 	u32                        sent;                 /*   224     4 */
> 	u16                        tididx;               /*   228     2 */
> 	u8                         iov_idx;              /*   230     1 */
>
> 	/* XXX 1 byte hole, try to pack */
>
> 	struct user_sdma_iovec     iovs[8];              /*   232   512 */
> 	/* --- cacheline 11 boundary (704 bytes) was 40 bytes ago --- */
>
> 	/* size: 768, cachelines: 12, members: 22 */
> 	/* sum members: 691, holes: 3, sum holes: 53 */
> 	/* padding: 24 */
> };

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]

On Tue, May 30, 2017 at 11:32:22AM -0700, Henry, Michael N wrote:
> On 5/29/2017 9:14 AM, Leon Romanovsky wrote:
> > On Fri, May 26, 2017 at 05:35:37AM -0700, Dennis Dalessandro wrote:
> > > From: Michael N. Henry <michael.n.henry-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > >
> > > Provide the ability for IB clients to modify the OPA specific
> > > capability mask and include this mask in the subsequent trap data.
> > >
> > > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> > > index 785e927..6c4f6da 100644
> > > --- a/include/rdma/ib_verbs.h
> > > +++ b/include/rdma/ib_verbs.h
> > > @@ -576,7 +576,8 @@ struct ib_device_modify {
> > >   enum ib_port_modify_flags {
> > >   	IB_PORT_SHUTDOWN		= 1,
> > >   	IB_PORT_INIT_TYPE		= (1<<2),
> > > -	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
> > > +	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
> > > +	IB_PORT_OPA_MASK_CHG		= (1<<4)
> > How will this flag be set?
> >
> This flag can be set as the port_modify_mask when calling ib_modify_port(..)

My question is more general. I'm as a representative of these "IB
clients" would like to know when and why should I use this flag.

Thanks

>
> -Mike
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

[Vishwanathapura, Niranjana]

On Wed, May 31, 2017 at 07:32:35AM +0300, Leon Romanovsky wrote:
>On Tue, May 30, 2017 at 11:32:22AM -0700, Henry, Michael N wrote:
>> On 5/29/2017 9:14 AM, Leon Romanovsky wrote:
>> > On Fri, May 26, 2017 at 05:35:37AM -0700, Dennis Dalessandro wrote:
>> > > From: Michael N. Henry <michael.n.henry-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > >
>> > > Provide the ability for IB clients to modify the OPA specific
>> > > capability mask and include this mask in the subsequent trap data.
>> > >
>> > > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
>> > > index 785e927..6c4f6da 100644
>> > > --- a/include/rdma/ib_verbs.h
>> > > +++ b/include/rdma/ib_verbs.h
>> > > @@ -576,7 +576,8 @@ struct ib_device_modify {
>> > >   enum ib_port_modify_flags {
>> > >   	IB_PORT_SHUTDOWN		= 1,
>> > >   	IB_PORT_INIT_TYPE		= (1<<2),
>> > > -	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
>> > > +	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
>> > > +	IB_PORT_OPA_MASK_CHG		= (1<<4)
>> > How will this flag be set?
>> >
>> This flag can be set as the port_modify_mask when calling ib_modify_port(..)
>
>My question is more general. I'm as a representative of these "IB
>clients" would like to know when and why should I use this flag.
>

Hi,
We have an OPA VNIC use case (patch coming up) to set/clear OPA capmask as 
below.

pm.set/clear_port_cap_mask = OPA_CAP_MASK3_IsEthOnFabricSupported;
ib_modify_port(hca, port, IB_PORT_OPA_MASK_CHG, &pm);

This patch support such usecase and we want to get comments early (if any) for 
that, hence this patch.

Niranjana

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2024 bytes --]

On Wed, May 31, 2017 at 10:58:21AM -0700, Vishwanathapura, Niranjana wrote:
> On Wed, May 31, 2017 at 07:32:35AM +0300, Leon Romanovsky wrote:
> > On Tue, May 30, 2017 at 11:32:22AM -0700, Henry, Michael N wrote:
> > > On 5/29/2017 9:14 AM, Leon Romanovsky wrote:
> > > > On Fri, May 26, 2017 at 05:35:37AM -0700, Dennis Dalessandro wrote:
> > > > > From: Michael N. Henry <michael.n.henry-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > > > >
> > > > > Provide the ability for IB clients to modify the OPA specific
> > > > > capability mask and include this mask in the subsequent trap data.
> > > > >
> > > > > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> > > > > index 785e927..6c4f6da 100644
> > > > > --- a/include/rdma/ib_verbs.h
> > > > > +++ b/include/rdma/ib_verbs.h
> > > > > @@ -576,7 +576,8 @@ struct ib_device_modify {
> > > > >   enum ib_port_modify_flags {
> > > > >   	IB_PORT_SHUTDOWN		= 1,
> > > > >   	IB_PORT_INIT_TYPE		= (1<<2),
> > > > > -	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
> > > > > +	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
> > > > > +	IB_PORT_OPA_MASK_CHG		= (1<<4)
> > > > How will this flag be set?
> > > >
> > > This flag can be set as the port_modify_mask when calling ib_modify_port(..)
> >
> > My question is more general. I'm as a representative of these "IB
> > clients" would like to know when and why should I use this flag.
> >
>
> Hi,
> We have an OPA VNIC use case (patch coming up) to set/clear OPA capmask as
> below.
>
> pm.set/clear_port_cap_mask = OPA_CAP_MASK3_IsEthOnFabricSupported;
> ib_modify_port(hca, port, IB_PORT_OPA_MASK_CHG, &pm);
>
> This patch support such usecase and we want to get comments early (if any)
> for that, hence this patch.

The comment is pretty simple, please don't post patches which are not
backed up by actual usage.

Right now, we have IB_PORT_SHUTDOWN, IB_PORT_INIT_TYPE and
IB_PORT_RESET_QKEY_CNTR are not written in any place. We don't want to
see another one.

And semantic of usage are more than welcomed.

Thanks

>
> Niranjana
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

[Vishwanathapura, Niranjana]

On Thu, Jun 01, 2017 at 08:55:42AM +0300, Leon Romanovsky wrote:
>On Wed, May 31, 2017 at 10:58:21AM -0700, Vishwanathapura, Niranjana wrote:
>> On Wed, May 31, 2017 at 07:32:35AM +0300, Leon Romanovsky wrote:
>> > On Tue, May 30, 2017 at 11:32:22AM -0700, Henry, Michael N wrote:
>> > > On 5/29/2017 9:14 AM, Leon Romanovsky wrote:
>> > > > On Fri, May 26, 2017 at 05:35:37AM -0700, Dennis Dalessandro wrote:
>> > > > > From: Michael N. Henry <michael.n.henry-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > > > >
>> > > > > Provide the ability for IB clients to modify the OPA specific
>> > > > > capability mask and include this mask in the subsequent trap data.
>> > > > >
>> > > > > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
>> > > > > index 785e927..6c4f6da 100644
>> > > > > --- a/include/rdma/ib_verbs.h
>> > > > > +++ b/include/rdma/ib_verbs.h
>> > > > > @@ -576,7 +576,8 @@ struct ib_device_modify {
>> > > > >   enum ib_port_modify_flags {
>> > > > >   	IB_PORT_SHUTDOWN		= 1,
>> > > > >   	IB_PORT_INIT_TYPE		= (1<<2),
>> > > > > -	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
>> > > > > +	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
>> > > > > +	IB_PORT_OPA_MASK_CHG		= (1<<4)
>> > > > How will this flag be set?
>> > > >
>> > > This flag can be set as the port_modify_mask when calling ib_modify_port(..)
>> >
>> > My question is more general. I'm as a representative of these "IB
>> > clients" would like to know when and why should I use this flag.
>> >
>>
>> Hi,
>> We have an OPA VNIC use case (patch coming up) to set/clear OPA capmask as
>> below.
>>
>> pm.set/clear_port_cap_mask = OPA_CAP_MASK3_IsEthOnFabricSupported;
>> ib_modify_port(hca, port, IB_PORT_OPA_MASK_CHG, &pm);
>>
>> This patch support such usecase and we want to get comments early (if any)
>> for that, hence this patch.
>
>The comment is pretty simple, please don't post patches which are not
>backed up by actual usage.
>
>Right now, we have IB_PORT_SHUTDOWN, IB_PORT_INIT_TYPE and
>IB_PORT_RESET_QKEY_CNTR are not written in any place. We don't want to
>see another one.
>
>And semantic of usage are more than welcomed.
>

Revised patch along with OPA-VNIC usage is posted below
https://patchwork.kernel.org/patch/9761375/

Niranjana


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH for-next 0/9] IB/core, hfi1, rdmavt: patches for next 05/19/2017

[Doug Ledford]

On Fri, 2017-05-26 at 05:35 -0700, Dennis Dalessandro wrote:
> Hi Doug,
> 
> Another set of patches for 4.13. These apply on top of the last
> submitted
> series: http://marc.info/?l=linux-rdma&m=149460607918479&w=2. I
> wanted to
> submit this last week but 0-day build service and my GitHub weren't
> seeing
> eye to eye. Things are working now and these have passed 0-day
> builds.
> 
> These are mostly fixes for the hfi1 driver, but there is a core
> change and one
> for rdmavt as well which adds the ability to send OPA cap_mask3 in a
> trap.
> 
> Patches can can also be found in my GitHub repo at:
> https://github.com/ddalessa/kernel/tree/for-4.13

Hi Denny,

Series, with updated patch 5/9 from Niranjana, applied.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html