All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH V7 for-next 0/3] Broadcom's driver add global atomics
@ 2021-06-03 13:15 Devesh Sharma
  2021-06-03 13:15 ` [PATCH V7 for-next 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports Devesh Sharma
                   ` (3 more replies)
  0 siblings, 4 replies; 6+ messages in thread
From: Devesh Sharma @ 2021-06-03 13:15 UTC (permalink / raw)
  To: linux-rdma; +Cc: Devesh Sharma

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

Adding automated detection and enablement of global PCI atomic
operation support.
Updated to ABI to pass wqe-mode to user to support rdma-core
pull request:
https://github.com/linux-rdma/rdma-core/pull/1007

v6->v7
 updated error return type to -EOPNOTSUPP
v5->v6
 dropped fixes tag
v4->v5
 fixed commit msg in patch 0001
 fixed mixing of int with bool
v3->v4
 removed redundant code to enable global atomics
 refactored to honor standard error codes.
v2->v3
 Added additional patch to update ABI. A PR corresponding to this
 is open.

V1->V2
 renamed bnxt_qplib_enable_atomic_ops_to_root to
 bnxt_qplib_determine_atomics

Devesh Sharma (3):
  RDMA/bnxt_re: Enable global atomic ops if platform supports
  bnxt_re: Update maintainers list
  RDMA/bnxt_re: update ABI to pass wqe-mode to user space

 MAINTAINERS                               |  2 --
 drivers/infiniband/hw/bnxt_re/ib_verbs.c  |  7 +++++++
 drivers/infiniband/hw/bnxt_re/main.c      |  3 +++
 drivers/infiniband/hw/bnxt_re/qplib_res.c | 17 +++++++++++++++++
 drivers/infiniband/hw/bnxt_re/qplib_res.h |  1 +
 drivers/infiniband/hw/bnxt_re/qplib_sp.c  | 13 ++++++++++++-
 drivers/infiniband/hw/bnxt_re/qplib_sp.h  |  2 --
 include/uapi/rdma/bnxt_re-abi.h           |  5 ++++-
 8 files changed, 44 insertions(+), 6 deletions(-)

-- 
2.25.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4212 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH V7 for-next 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports
  2021-06-03 13:15 [PATCH V7 for-next 0/3] Broadcom's driver add global atomics Devesh Sharma
@ 2021-06-03 13:15 ` Devesh Sharma
  2021-06-03 13:15 ` [PATCH V7 for-next 2/3] bnxt_re: Update maintainers list Devesh Sharma
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 6+ messages in thread
From: Devesh Sharma @ 2021-06-03 13:15 UTC (permalink / raw)
  To: linux-rdma; +Cc: Devesh Sharma

[-- Attachment #1: Type: text/plain, Size: 4559 bytes --]

Enabling Atomic operations for Gen P5 devices if the underlying
platform supports global atomic ops.

Signed-off-by: Devesh Sharma <devesh.sharma@broadcom.com>
---
 drivers/infiniband/hw/bnxt_re/ib_verbs.c  |  4 ++++
 drivers/infiniband/hw/bnxt_re/main.c      |  3 +++
 drivers/infiniband/hw/bnxt_re/qplib_res.c | 17 +++++++++++++++++
 drivers/infiniband/hw/bnxt_re/qplib_res.h |  1 +
 drivers/infiniband/hw/bnxt_re/qplib_sp.c  | 13 ++++++++++++-
 drivers/infiniband/hw/bnxt_re/qplib_sp.h  |  2 --
 6 files changed, 37 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index 537471ffaa79..a113d8d9e9ed 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -163,6 +163,10 @@ int bnxt_re_query_device(struct ib_device *ibdev,
 	ib_attr->max_qp_init_rd_atom = dev_attr->max_qp_init_rd_atom;
 	ib_attr->atomic_cap = IB_ATOMIC_NONE;
 	ib_attr->masked_atomic_cap = IB_ATOMIC_NONE;
+	if (dev_attr->is_atomic) {
+		ib_attr->atomic_cap = IB_ATOMIC_GLOB;
+		ib_attr->masked_atomic_cap = IB_ATOMIC_GLOB;
+	}
 
 	ib_attr->max_ee_rd_atom = 0;
 	ib_attr->max_res_rd_atom = 0;
diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c
index b090dfa4f4cb..0de4e22f9750 100644
--- a/drivers/infiniband/hw/bnxt_re/main.c
+++ b/drivers/infiniband/hw/bnxt_re/main.c
@@ -128,6 +128,9 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode)
 	rdev->rcfw.res = &rdev->qplib_res;
 
 	bnxt_re_set_drv_mode(rdev, wqe_mode);
+	if (bnxt_qplib_determine_atomics(en_dev->pdev))
+		ibdev_info(&rdev->ibdev,
+			   "platform doesn't support global atomics.");
 	return 0;
 }
 
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.c b/drivers/infiniband/hw/bnxt_re/qplib_res.c
index 3ca47004b752..17f0701b3cee 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_res.c
+++ b/drivers/infiniband/hw/bnxt_re/qplib_res.c
@@ -959,3 +959,20 @@ int bnxt_qplib_alloc_res(struct bnxt_qplib_res *res, struct pci_dev *pdev,
 	bnxt_qplib_free_res(res);
 	return rc;
 }
+
+int bnxt_qplib_determine_atomics(struct pci_dev *dev)
+{
+	int comp;
+	u16 ctl2;
+
+	comp = pci_enable_atomic_ops_to_root(dev,
+					     PCI_EXP_DEVCAP2_ATOMIC_COMP32);
+	if (comp)
+		return -EOPNOTSUPP;
+	comp = pci_enable_atomic_ops_to_root(dev,
+					     PCI_EXP_DEVCAP2_ATOMIC_COMP64);
+	if (comp)
+		return -EOPNOTSUPP;
+	pcie_capability_read_word(dev, PCI_EXP_DEVCTL2, &ctl2);
+	return !(ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ);
+}
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.h b/drivers/infiniband/hw/bnxt_re/qplib_res.h
index 7a1ab38b95da..d2aea52bd1d8 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_res.h
+++ b/drivers/infiniband/hw/bnxt_re/qplib_res.h
@@ -373,6 +373,7 @@ void bnxt_qplib_free_ctx(struct bnxt_qplib_res *res,
 int bnxt_qplib_alloc_ctx(struct bnxt_qplib_res *res,
 			 struct bnxt_qplib_ctx *ctx,
 			 bool virt_fn, bool is_p5);
+int bnxt_qplib_determine_atomics(struct pci_dev *dev);
 
 static inline void bnxt_qplib_hwq_incr_prod(struct bnxt_qplib_hwq *hwq, u32 cnt)
 {
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.c b/drivers/infiniband/hw/bnxt_re/qplib_sp.c
index 049b3576302b..3d9259632eb3 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_sp.c
+++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.c
@@ -54,6 +54,17 @@ const struct bnxt_qplib_gid bnxt_qplib_gid_zero = {{ 0, 0, 0, 0, 0, 0, 0, 0,
 
 /* Device */
 
+static bool bnxt_qplib_is_atomic_cap(struct bnxt_qplib_rcfw *rcfw)
+{
+	u16 pcie_ctl2 = 0;
+
+	if (!bnxt_qplib_is_chip_gen_p5(rcfw->res->cctx))
+		return false;
+
+	pcie_capability_read_word(rcfw->pdev, PCI_EXP_DEVCTL2, &pcie_ctl2);
+	return (pcie_ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ);
+}
+
 static void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw,
 				     char *fw_ver)
 {
@@ -162,7 +173,7 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw,
 		attr->tqm_alloc_reqs[i * 4 + 3] = *(++tqm_alloc);
 	}
 
-	attr->is_atomic = false;
+	attr->is_atomic = bnxt_qplib_is_atomic_cap(rcfw);
 bail:
 	bnxt_qplib_rcfw_free_sbuf(rcfw, sbuf);
 	return rc;
diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.h b/drivers/infiniband/hw/bnxt_re/qplib_sp.h
index bc228340684f..260104783691 100644
--- a/drivers/infiniband/hw/bnxt_re/qplib_sp.h
+++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.h
@@ -42,8 +42,6 @@
 
 #define BNXT_QPLIB_RESERVED_QP_WRS	128
 
-#define PCI_EXP_DEVCTL2_ATOMIC_REQ      0x0040
-
 struct bnxt_qplib_dev_attr {
 #define FW_VER_ARR_LEN			4
 	u8				fw_ver[FW_VER_ARR_LEN];
-- 
2.25.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4212 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH V7 for-next 2/3] bnxt_re: Update maintainers list
  2021-06-03 13:15 [PATCH V7 for-next 0/3] Broadcom's driver add global atomics Devesh Sharma
  2021-06-03 13:15 ` [PATCH V7 for-next 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports Devesh Sharma
@ 2021-06-03 13:15 ` Devesh Sharma
  2021-06-03 13:15 ` [PATCH V7 for-next 3/3] RDMA/bnxt_re: update ABI to pass wqe-mode to user space Devesh Sharma
  2021-06-03 20:01 ` [PATCH V7 for-next 0/3] Broadcom's driver add global atomics Jason Gunthorpe
  3 siblings, 0 replies; 6+ messages in thread
From: Devesh Sharma @ 2021-06-03 13:15 UTC (permalink / raw)
  To: linux-rdma; +Cc: Devesh Sharma

[-- Attachment #1: Type: text/plain, Size: 693 bytes --]

Updated the maintainers list and removed
non-active members.

Signed-off-by: Devesh Sharma <devesh.sharma@broadcom.com>
---
 MAINTAINERS | 2 --
 1 file changed, 2 deletions(-)

diff --git a/MAINTAINERS b/MAINTAINERS
index bd7aff0c120f..786573a556b9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3736,8 +3736,6 @@ F:	drivers/gpio/gpio-bcm-kona.c
 BROADCOM NETXTREME-E ROCE DRIVER
 M:	Selvin Xavier <selvin.xavier@broadcom.com>
 M:	Devesh Sharma <devesh.sharma@broadcom.com>
-M:	Somnath Kotur <somnath.kotur@broadcom.com>
-M:	Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com>
 M:	Naresh Kumar PBS <nareshkumar.pbs@broadcom.com>
 L:	linux-rdma@vger.kernel.org
 S:	Supported
-- 
2.25.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4212 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH V7 for-next 3/3] RDMA/bnxt_re: update ABI to pass wqe-mode to user space
  2021-06-03 13:15 [PATCH V7 for-next 0/3] Broadcom's driver add global atomics Devesh Sharma
  2021-06-03 13:15 ` [PATCH V7 for-next 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports Devesh Sharma
  2021-06-03 13:15 ` [PATCH V7 for-next 2/3] bnxt_re: Update maintainers list Devesh Sharma
@ 2021-06-03 13:15 ` Devesh Sharma
  2021-06-03 19:58   ` Jason Gunthorpe
  2021-06-03 20:01 ` [PATCH V7 for-next 0/3] Broadcom's driver add global atomics Jason Gunthorpe
  3 siblings, 1 reply; 6+ messages in thread
From: Devesh Sharma @ 2021-06-03 13:15 UTC (permalink / raw)
  To: linux-rdma; +Cc: Devesh Sharma

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

Changing ucontext ABI response structure to pass wqe_mode
to user library.
A flag in comp_mask has been set to indicate presence of
wqe_mode.

Signed-off-by: Devesh Sharma <devesh.sharma@broadcom.com>
---
 drivers/infiniband/hw/bnxt_re/ib_verbs.c | 3 +++
 include/uapi/rdma/bnxt_re-abi.h          | 5 ++++-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index a113d8d9e9ed..5955713234cb 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -3882,6 +3882,9 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata)
 	resp.max_cqd = dev_attr->max_cq_wqes;
 	resp.rsvd    = 0;
 
+	resp.comp_mask |= BNXT_RE_UCNTX_CMASK_HAVE_MODE;
+	resp.mode = rdev->chip_ctx->modes.wqe_mode;
+
 	rc = ib_copy_to_udata(udata, &resp, min(udata->outlen, sizeof(resp)));
 	if (rc) {
 		ibdev_err(ibdev, "Failed to copy user context");
diff --git a/include/uapi/rdma/bnxt_re-abi.h b/include/uapi/rdma/bnxt_re-abi.h
index dc52e3cf574c..52205ed2b898 100644
--- a/include/uapi/rdma/bnxt_re-abi.h
+++ b/include/uapi/rdma/bnxt_re-abi.h
@@ -49,7 +49,8 @@
 #define BNXT_RE_CHIP_ID0_CHIP_MET_SFT		0x18
 
 enum {
-	BNXT_RE_UCNTX_CMASK_HAVE_CCTX = 0x1ULL
+	BNXT_RE_UCNTX_CMASK_HAVE_CCTX = 0x1ULL,
+	BNXT_RE_UCNTX_CMASK_HAVE_MODE = 0x02ULL
 };
 
 struct bnxt_re_uctx_resp {
@@ -62,6 +63,8 @@ struct bnxt_re_uctx_resp {
 	__aligned_u64 comp_mask;
 	__u32 chip_id0;
 	__u32 chip_id1;
+	__u32 mode;
+	__u32 rsvd1; /* padding */
 };
 
 /*
-- 
2.25.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4212 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH V7 for-next 3/3] RDMA/bnxt_re: update ABI to pass wqe-mode to user space
  2021-06-03 13:15 ` [PATCH V7 for-next 3/3] RDMA/bnxt_re: update ABI to pass wqe-mode to user space Devesh Sharma
@ 2021-06-03 19:58   ` Jason Gunthorpe
  0 siblings, 0 replies; 6+ messages in thread
From: Jason Gunthorpe @ 2021-06-03 19:58 UTC (permalink / raw)
  To: Devesh Sharma; +Cc: linux-rdma

On Thu, Jun 03, 2021 at 06:45:34PM +0530, Devesh Sharma wrote:
> Changing ucontext ABI response structure to pass wqe_mode
> to user library.
> A flag in comp_mask has been set to indicate presence of
> wqe_mode.
> 
> Signed-off-by: Devesh Sharma <devesh.sharma@broadcom.com>
>  drivers/infiniband/hw/bnxt_re/ib_verbs.c | 3 +++
>  include/uapi/rdma/bnxt_re-abi.h          | 5 ++++-
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
> index a113d8d9e9ed..5955713234cb 100644
> +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
> @@ -3882,6 +3882,9 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata)
>  	resp.max_cqd = dev_attr->max_cq_wqes;
>  	resp.rsvd    = 0;
>  
> +	resp.comp_mask |= BNXT_RE_UCNTX_CMASK_HAVE_MODE;
> +	resp.mode = rdev->chip_ctx->modes.wqe_mode;

The enum for this value is not in bnxt_re-abi.h and needs to be moved
there if you are going to start using it as uABI

In fact it looks like several of the things in
providers/bnxt_re/bnxt_re-abi.h needs to be moved to the kernel ABI
header and harmonized with the kernel driver.

Jason


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH V7 for-next 0/3] Broadcom's driver add global atomics
  2021-06-03 13:15 [PATCH V7 for-next 0/3] Broadcom's driver add global atomics Devesh Sharma
                   ` (2 preceding siblings ...)
  2021-06-03 13:15 ` [PATCH V7 for-next 3/3] RDMA/bnxt_re: update ABI to pass wqe-mode to user space Devesh Sharma
@ 2021-06-03 20:01 ` Jason Gunthorpe
  3 siblings, 0 replies; 6+ messages in thread
From: Jason Gunthorpe @ 2021-06-03 20:01 UTC (permalink / raw)
  To: Devesh Sharma; +Cc: linux-rdma

On Thu, Jun 03, 2021 at 06:45:31PM +0530, Devesh Sharma wrote:
> Adding automated detection and enablement of global PCI atomic
> operation support.
> Updated to ABI to pass wqe-mode to user to support rdma-core
> pull request:
> https://github.com/linux-rdma/rdma-core/pull/1007
> 
> v6->v7
>  updated error return type to -EOPNOTSUPP
> v5->v6
>  dropped fixes tag
> v4->v5
>  fixed commit msg in patch 0001
>  fixed mixing of int with bool
> v3->v4
>  removed redundant code to enable global atomics
>  refactored to honor standard error codes.
> v2->v3
>  Added additional patch to update ABI. A PR corresponding to this
>  is open.
> 
> V1->V2
>  renamed bnxt_qplib_enable_atomic_ops_to_root to
>  bnxt_qplib_determine_atomics
> 
> Devesh Sharma (3):
>   RDMA/bnxt_re: Enable global atomic ops if platform supports
>   bnxt_re: Update maintainers list

I took these two to for-next, thanks

Jason

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2021-06-03 20:01 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-03 13:15 [PATCH V7 for-next 0/3] Broadcom's driver add global atomics Devesh Sharma
2021-06-03 13:15 ` [PATCH V7 for-next 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports Devesh Sharma
2021-06-03 13:15 ` [PATCH V7 for-next 2/3] bnxt_re: Update maintainers list Devesh Sharma
2021-06-03 13:15 ` [PATCH V7 for-next 3/3] RDMA/bnxt_re: update ABI to pass wqe-mode to user space Devesh Sharma
2021-06-03 19:58   ` Jason Gunthorpe
2021-06-03 20:01 ` [PATCH V7 for-next 0/3] Broadcom's driver add global atomics Jason Gunthorpe

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.