* [PATCH for-next V3 0/3] Broadcom's driver add global atomics
@ 2021-05-26 15:36 Devesh Sharma
2021-05-26 15:36 ` [PATCH for-next V3 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports Devesh Sharma
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Devesh Sharma @ 2021-05-26 15:36 UTC (permalink / raw)
To: linux-rdma; +Cc: Devesh Sharma
[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]
Adding automated detection and enablement of global PCI atomic
operation support.
Updated to ABI to pass wqe-mode to user to support rdma-core
pull request:
https://github.com/linux-rdma/rdma-core/pull/1007
v2->v3
Added additional patch to update ABI. A PR corresponding to this
is open.
V1->V2
renamed bnxt_qplib_enable_atomic_ops_to_root to
bnxt_qplib_determine_atomics
Devesh Sharma (3):
RDMA/bnxt_re: Enable global atomic ops if platform supports
bnxt_re: Update maintainers list
RDMA/bnxt_re: update ABI to pass wqe-mode to user space
MAINTAINERS | 2 --
drivers/infiniband/hw/bnxt_re/ib_verbs.c | 7 +++++++
drivers/infiniband/hw/bnxt_re/main.c | 4 ++++
drivers/infiniband/hw/bnxt_re/qplib_res.c | 15 +++++++++++++++
drivers/infiniband/hw/bnxt_re/qplib_res.h | 1 +
drivers/infiniband/hw/bnxt_re/qplib_sp.c | 13 ++++++++++++-
drivers/infiniband/hw/bnxt_re/qplib_sp.h | 2 --
include/uapi/rdma/bnxt_re-abi.h | 5 ++++-
8 files changed, 43 insertions(+), 6 deletions(-)
--
2.25.1
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4212 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH for-next V3 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports 2021-05-26 15:36 [PATCH for-next V3 0/3] Broadcom's driver add global atomics Devesh Sharma @ 2021-05-26 15:36 ` Devesh Sharma 2021-06-01 14:46 ` Jason Gunthorpe 2021-05-26 15:36 ` [PATCH for-next V3 2/3] bnxt_re: Update maintainers list Devesh Sharma 2021-05-26 15:36 ` [PATCH for-next V3 3/3] RDMA/bnxt_re: update ABI to pass wqe-mode to user space Devesh Sharma 2 siblings, 1 reply; 7+ messages in thread From: Devesh Sharma @ 2021-05-26 15:36 UTC (permalink / raw) To: linux-rdma; +Cc: Devesh Sharma [-- Attachment #1: Type: text/plain, Size: 4681 bytes --] Enabling Atomic operations for Gen P5 devices if the underlying platform supports global atomic ops. Fixes:7ff662b76167 ("Disable atomic capability on bnxt_re adapters") Signed-off-by: Devesh Sharma <devesh.sharma@broadcom.com> --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 4 ++++ drivers/infiniband/hw/bnxt_re/main.c | 4 ++++ drivers/infiniband/hw/bnxt_re/qplib_res.c | 15 +++++++++++++++ drivers/infiniband/hw/bnxt_re/qplib_res.h | 1 + drivers/infiniband/hw/bnxt_re/qplib_sp.c | 13 ++++++++++++- drivers/infiniband/hw/bnxt_re/qplib_sp.h | 2 -- 6 files changed, 36 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 537471ffaa79..a113d8d9e9ed 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -163,6 +163,10 @@ int bnxt_re_query_device(struct ib_device *ibdev, ib_attr->max_qp_init_rd_atom = dev_attr->max_qp_init_rd_atom; ib_attr->atomic_cap = IB_ATOMIC_NONE; ib_attr->masked_atomic_cap = IB_ATOMIC_NONE; + if (dev_attr->is_atomic) { + ib_attr->atomic_cap = IB_ATOMIC_GLOB; + ib_attr->masked_atomic_cap = IB_ATOMIC_GLOB; + } ib_attr->max_ee_rd_atom = 0; ib_attr->max_res_rd_atom = 0; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index b090dfa4f4cb..ee0cdf89eb1a 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -128,6 +128,10 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) rdev->rcfw.res = &rdev->qplib_res; bnxt_re_set_drv_mode(rdev, wqe_mode); + if (bnxt_qplib_determine_atomics(en_dev->pdev)) + ibdev_info(&rdev->ibdev, + "platform doesn't support global atomics."); + return 0; } diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.c b/drivers/infiniband/hw/bnxt_re/qplib_res.c index 3ca47004b752..4de9359c2028 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_res.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_res.c @@ -959,3 +959,18 @@ int bnxt_qplib_alloc_res(struct bnxt_qplib_res *res, struct pci_dev *pdev, bnxt_qplib_free_res(res); return rc; } + +bool bnxt_qplib_determine_atomics(struct pci_dev *dev) +{ + u16 ctl2; + + if(pci_enable_atomic_ops_to_root(dev, PCI_EXP_DEVCAP2_ATOMIC_COMP32) && + pci_enable_atomic_ops_to_root(dev, PCI_EXP_DEVCAP2_ATOMIC_COMP64)) + return true; /* Failure */ + pcie_capability_read_word(dev, PCI_EXP_DEVCTL2, &ctl2); + if (ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ) + return 0; /* Success */ + pcie_capability_set_word(dev, PCI_EXP_DEVCTL2, + PCI_EXP_DEVCTL2_ATOMIC_REQ); + return 0; /* Success */ +} diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.h b/drivers/infiniband/hw/bnxt_re/qplib_res.h index 7a1ab38b95da..fb7fde4fed56 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_res.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_res.h @@ -373,6 +373,7 @@ void bnxt_qplib_free_ctx(struct bnxt_qplib_res *res, int bnxt_qplib_alloc_ctx(struct bnxt_qplib_res *res, struct bnxt_qplib_ctx *ctx, bool virt_fn, bool is_p5); +bool bnxt_qplib_determine_atomics(struct pci_dev *dev); static inline void bnxt_qplib_hwq_incr_prod(struct bnxt_qplib_hwq *hwq, u32 cnt) { diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.c b/drivers/infiniband/hw/bnxt_re/qplib_sp.c index 049b3576302b..57407be16f27 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_sp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.c @@ -54,6 +54,17 @@ const struct bnxt_qplib_gid bnxt_qplib_gid_zero = {{ 0, 0, 0, 0, 0, 0, 0, 0, /* Device */ +static u8 bnxt_qplib_is_atomic_cap(struct bnxt_qplib_rcfw *rcfw) +{ + u16 pcie_ctl2 = 0; + + if (!bnxt_qplib_is_chip_gen_p5(rcfw->res->cctx)) + return false; + + pcie_capability_read_word(rcfw->pdev, PCI_EXP_DEVCTL2, &pcie_ctl2); + return (pcie_ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ); +} + static void bnxt_qplib_query_version(struct bnxt_qplib_rcfw *rcfw, char *fw_ver) { @@ -162,7 +173,7 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw, attr->tqm_alloc_reqs[i * 4 + 3] = *(++tqm_alloc); } - attr->is_atomic = false; + attr->is_atomic = bnxt_qplib_is_atomic_cap(rcfw); bail: bnxt_qplib_rcfw_free_sbuf(rcfw, sbuf); return rc; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.h b/drivers/infiniband/hw/bnxt_re/qplib_sp.h index bc228340684f..260104783691 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_sp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.h @@ -42,8 +42,6 @@ #define BNXT_QPLIB_RESERVED_QP_WRS 128 -#define PCI_EXP_DEVCTL2_ATOMIC_REQ 0x0040 - struct bnxt_qplib_dev_attr { #define FW_VER_ARR_LEN 4 u8 fw_ver[FW_VER_ARR_LEN]; -- 2.25.1 [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4212 bytes --] ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH for-next V3 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports 2021-05-26 15:36 ` [PATCH for-next V3 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports Devesh Sharma @ 2021-06-01 14:46 ` Jason Gunthorpe 2021-06-02 3:43 ` Devesh Sharma 0 siblings, 1 reply; 7+ messages in thread From: Jason Gunthorpe @ 2021-06-01 14:46 UTC (permalink / raw) To: Devesh Sharma; +Cc: linux-rdma On Wed, May 26, 2021 at 09:06:27PM +0530, Devesh Sharma wrote: > +bool bnxt_qplib_determine_atomics(struct pci_dev *dev) > +{ > + u16 ctl2; > + > + if(pci_enable_atomic_ops_to_root(dev, PCI_EXP_DEVCAP2_ATOMIC_COMP32) && > + pci_enable_atomic_ops_to_root(dev, PCI_EXP_DEVCAP2_ATOMIC_COMP64)) > + return true; /* Failure */ > + pcie_capability_read_word(dev, PCI_EXP_DEVCTL2, &ctl2); > + if (ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ) > + return 0; /* Success */ > + pcie_capability_set_word(dev, PCI_EXP_DEVCTL2, > + PCI_EXP_DEVCTL2_ATOMIC_REQ); > + return 0; /* Success */ > +} true on failure, 0 on success and the error code is thrown away?? Please return -ENOTSUPP or something on error Jason ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH for-next V3 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports 2021-06-01 14:46 ` Jason Gunthorpe @ 2021-06-02 3:43 ` Devesh Sharma 2021-06-02 14:44 ` Jason Gunthorpe 0 siblings, 1 reply; 7+ messages in thread From: Devesh Sharma @ 2021-06-02 3:43 UTC (permalink / raw) To: Jason Gunthorpe; +Cc: linux-rdma [-- Attachment #1: Type: text/plain, Size: 920 bytes --] On Tue, Jun 1, 2021 at 8:16 PM Jason Gunthorpe <jgg@nvidia.com> wrote: > > On Wed, May 26, 2021 at 09:06:27PM +0530, Devesh Sharma wrote: > > +bool bnxt_qplib_determine_atomics(struct pci_dev *dev) > > +{ > > + u16 ctl2; > > + > > + if(pci_enable_atomic_ops_to_root(dev, PCI_EXP_DEVCAP2_ATOMIC_COMP32) && > > + pci_enable_atomic_ops_to_root(dev, PCI_EXP_DEVCAP2_ATOMIC_COMP64)) > > + return true; /* Failure */ > > + pcie_capability_read_word(dev, PCI_EXP_DEVCTL2, &ctl2); > > + if (ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ) > > + return 0; /* Success */ > > + pcie_capability_set_word(dev, PCI_EXP_DEVCTL2, > > + PCI_EXP_DEVCTL2_ATOMIC_REQ); > > + return 0; /* Success */ > > +} > > true on failure, 0 on success and the error code is thrown away?? > Please return -ENOTSUPP or something on error make sense. > > Jason -- -Regards Devesh [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4212 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH for-next V3 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports 2021-06-02 3:43 ` Devesh Sharma @ 2021-06-02 14:44 ` Jason Gunthorpe 0 siblings, 0 replies; 7+ messages in thread From: Jason Gunthorpe @ 2021-06-02 14:44 UTC (permalink / raw) To: Devesh Sharma; +Cc: linux-rdma On Wed, Jun 02, 2021 at 09:13:43AM +0530, Devesh Sharma wrote: > On Tue, Jun 1, 2021 at 8:16 PM Jason Gunthorpe <jgg@nvidia.com> wrote: > > > > On Wed, May 26, 2021 at 09:06:27PM +0530, Devesh Sharma wrote: > > > +bool bnxt_qplib_determine_atomics(struct pci_dev *dev) > > > +{ > > > + u16 ctl2; > > > + > > > + if(pci_enable_atomic_ops_to_root(dev, PCI_EXP_DEVCAP2_ATOMIC_COMP32) && > > > + pci_enable_atomic_ops_to_root(dev, PCI_EXP_DEVCAP2_ATOMIC_COMP64)) > > > + return true; /* Failure */ > > > + pcie_capability_read_word(dev, PCI_EXP_DEVCTL2, &ctl2); > > > + if (ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ) > > > + return 0; /* Success */ > > > + pcie_capability_set_word(dev, PCI_EXP_DEVCTL2, > > > + PCI_EXP_DEVCTL2_ATOMIC_REQ); > > > + return 0; /* Success */ > > > +} > > > > true on failure, 0 on success and the error code is thrown away?? > > Please return -ENOTSUPP or something on error > make sense. I just noticed another mixing of types with return bool/return 0 in the same function, please check them all Jason ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH for-next V3 2/3] bnxt_re: Update maintainers list 2021-05-26 15:36 [PATCH for-next V3 0/3] Broadcom's driver add global atomics Devesh Sharma 2021-05-26 15:36 ` [PATCH for-next V3 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports Devesh Sharma @ 2021-05-26 15:36 ` Devesh Sharma 2021-05-26 15:36 ` [PATCH for-next V3 3/3] RDMA/bnxt_re: update ABI to pass wqe-mode to user space Devesh Sharma 2 siblings, 0 replies; 7+ messages in thread From: Devesh Sharma @ 2021-05-26 15:36 UTC (permalink / raw) To: linux-rdma; +Cc: Devesh Sharma [-- Attachment #1: Type: text/plain, Size: 693 bytes --] Updated the maintainers list and removed non-active members. Signed-off-by: Devesh Sharma <devesh.sharma@broadcom.com> --- MAINTAINERS | 2 -- 1 file changed, 2 deletions(-) diff --git a/MAINTAINERS b/MAINTAINERS index bd7aff0c120f..786573a556b9 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3736,8 +3736,6 @@ F: drivers/gpio/gpio-bcm-kona.c BROADCOM NETXTREME-E ROCE DRIVER M: Selvin Xavier <selvin.xavier@broadcom.com> M: Devesh Sharma <devesh.sharma@broadcom.com> -M: Somnath Kotur <somnath.kotur@broadcom.com> -M: Sriharsha Basavapatna <sriharsha.basavapatna@broadcom.com> M: Naresh Kumar PBS <nareshkumar.pbs@broadcom.com> L: linux-rdma@vger.kernel.org S: Supported -- 2.25.1 [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4212 bytes --] ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH for-next V3 3/3] RDMA/bnxt_re: update ABI to pass wqe-mode to user space 2021-05-26 15:36 [PATCH for-next V3 0/3] Broadcom's driver add global atomics Devesh Sharma 2021-05-26 15:36 ` [PATCH for-next V3 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports Devesh Sharma 2021-05-26 15:36 ` [PATCH for-next V3 2/3] bnxt_re: Update maintainers list Devesh Sharma @ 2021-05-26 15:36 ` Devesh Sharma 2 siblings, 0 replies; 7+ messages in thread From: Devesh Sharma @ 2021-05-26 15:36 UTC (permalink / raw) To: linux-rdma; +Cc: Devesh Sharma [-- Attachment #1: Type: text/plain, Size: 1584 bytes --] Changing ucontext ABI response structure to pass wqe_mode to user library. A flag in comp_mask has been set to indicate presence of wqe_mode. Signed-off-by: Devesh Sharma <devesh.sharma@broadcom.com> --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 3 +++ include/uapi/rdma/bnxt_re-abi.h | 5 ++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index a113d8d9e9ed..5955713234cb 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -3882,6 +3882,9 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata) resp.max_cqd = dev_attr->max_cq_wqes; resp.rsvd = 0; + resp.comp_mask |= BNXT_RE_UCNTX_CMASK_HAVE_MODE; + resp.mode = rdev->chip_ctx->modes.wqe_mode; + rc = ib_copy_to_udata(udata, &resp, min(udata->outlen, sizeof(resp))); if (rc) { ibdev_err(ibdev, "Failed to copy user context"); diff --git a/include/uapi/rdma/bnxt_re-abi.h b/include/uapi/rdma/bnxt_re-abi.h index dc52e3cf574c..52205ed2b898 100644 --- a/include/uapi/rdma/bnxt_re-abi.h +++ b/include/uapi/rdma/bnxt_re-abi.h @@ -49,7 +49,8 @@ #define BNXT_RE_CHIP_ID0_CHIP_MET_SFT 0x18 enum { - BNXT_RE_UCNTX_CMASK_HAVE_CCTX = 0x1ULL + BNXT_RE_UCNTX_CMASK_HAVE_CCTX = 0x1ULL, + BNXT_RE_UCNTX_CMASK_HAVE_MODE = 0x02ULL }; struct bnxt_re_uctx_resp { @@ -62,6 +63,8 @@ struct bnxt_re_uctx_resp { __aligned_u64 comp_mask; __u32 chip_id0; __u32 chip_id1; + __u32 mode; + __u32 rsvd1; /* padding */ }; /* -- 2.25.1 [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 4212 bytes --] ^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-06-02 14:44 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-26 15:36 [PATCH for-next V3 0/3] Broadcom's driver add global atomics Devesh Sharma 2021-05-26 15:36 ` [PATCH for-next V3 1/3] RDMA/bnxt_re: Enable global atomic ops if platform supports Devesh Sharma 2021-06-01 14:46 ` Jason Gunthorpe 2021-06-02 3:43 ` Devesh Sharma 2021-06-02 14:44 ` Jason Gunthorpe 2021-05-26 15:36 ` [PATCH for-next V3 2/3] bnxt_re: Update maintainers list Devesh Sharma 2021-05-26 15:36 ` [PATCH for-next V3 3/3] RDMA/bnxt_re: update ABI to pass wqe-mode to user space Devesh Sharma
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.