[rdma-rc PATCH 0/2] iw_cxgb4: Adjust the cq/qp mask

[rdma-rc PATCH 0/2] iw_cxgb4: Adjust the cq/qp mask

[Raju Rangoju]

Export the LLD sge_host_page_size field to ULDs via
cxgb4_lld_info, so that iw_cxgb4 can adjust the cq/qp
mask based on no.of bar2 pages in a host page.

This series has both net(cxgb4) and rdma(iw_cxgb4) changes,
and I would request this merge via rdma repo.

I have made sure this series applies cleanly on both net-next
and rdma-for-rc and doesn't cause any merge conflicts.

Raju Rangoju (2):
  cxgb4: export sge_host_page_size to ulds
  iw_cxgb4: cq/qp mask depends on bar2 pages in a host page

 drivers/infiniband/hw/cxgb4/device.c           | 15 +++++++++++++--
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.c |  1 +
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h |  1 +
 3 files changed, 15 insertions(+), 2 deletions(-)

-- 
2.13.0

[rdma-rc PATCH 1/2] cxgb4: export sge_host_page_size to ulds

[Raju Rangoju]

Export the sge_host_page_size field to ULDs via
cxgb4_lld_info, so that iw_cxgb4 can make use of
this in calculating the correct qp/cq mask.

Fixes: 2391b0030e ("cxgb4: Remove SGE_HOST_PAGE_SIZE dependency on page
size")
Signed-off-by: Raju Rangoju <rajur@chelsio.com>
---
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.c | 1 +
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.c
index c041f44324db..b3654598a2d5 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.c
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.c
@@ -660,6 +660,7 @@ static void uld_init(struct adapter *adap, struct cxgb4_lld_info *lld)
 	lld->cclk_ps = 1000000000 / adap->params.vpd.cclk;
 	lld->udb_density = 1 << adap->params.sge.eq_qpp;
 	lld->ucq_density = 1 << adap->params.sge.iq_qpp;
+	lld->sge_host_page_size = 1 << (adap->params.sge.hps + 10);
 	lld->filt_mode = adap->params.tp.vlan_pri_map;
 	/* MODQ_REQ_MAP sets queues 0-3 to chan 0-3 */
 	for (i = 0; i < NCHAN; i++)
diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h
index 5fa9a2d5fc4b..21da34a4ca24 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h
@@ -336,6 +336,7 @@ struct cxgb4_lld_info {
 	unsigned int cclk_ps;                /* Core clock period in psec */
 	unsigned short udb_density;          /* # of user DB/page */
 	unsigned short ucq_density;          /* # of user CQs/page */
+	unsigned int sge_host_page_size;     /* SGE host page size */
 	unsigned short filt_mode;            /* filter optional components */
 	unsigned short tx_modq[NCHAN];       /* maps each tx channel to a */
 					     /* scheduler queue */
-- 
2.13.0

[rdma-rc PATCH 2/2] iw_cxgb4: cq/qp mask depends on bar2 pages in a host page

[Raju Rangoju]

Adjust the cq/qp mask based on no.of bar2 pages in a host page.

For user-mode rdma, the granularity of the BAR2 memory mapped
to a user rdma process during queue allocation must be based
on the host page size. The lld attributes udb_density and
ucq_density are used to figure out how many sge contexts are
in a bar2 page. So the rdev->qpmask and rdev->cqmask in
iw_cxgb4 need to now be adjusted based on how many sge bar2
pages are in a host page.

Fixes: 2391b0030e ("cxgb4: Remove SGE_HOST_PAGE_SIZE dependency on page size")
Signed-off-by: Raju Rangoju <rajur@chelsio.com>
---
 drivers/infiniband/hw/cxgb4/device.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/cxgb4/device.c b/drivers/infiniband/hw/cxgb4/device.c
index c13c0ba30f63..d499cd61c0e8 100644
--- a/drivers/infiniband/hw/cxgb4/device.c
+++ b/drivers/infiniband/hw/cxgb4/device.c
@@ -783,6 +783,7 @@ void c4iw_init_dev_ucontext(struct c4iw_rdev *rdev,
 static int c4iw_rdev_open(struct c4iw_rdev *rdev)
 {
 	int err;
+	unsigned int factor;
 
 	c4iw_init_dev_ucontext(rdev, &rdev->uctx);
 
@@ -806,8 +807,18 @@ static int c4iw_rdev_open(struct c4iw_rdev *rdev)
 		return -EINVAL;
 	}
 
-	rdev->qpmask = rdev->lldi.udb_density - 1;
-	rdev->cqmask = rdev->lldi.ucq_density - 1;
+	/* This implementation requires a sge_host_page_size <= PAGE_SIZE. */
+	if (rdev->lldi.sge_host_page_size > PAGE_SIZE) {
+		pr_err("%s: unsupported sge host page size %u\n",
+		       pci_name(rdev->lldi.pdev),
+		       rdev->lldi.sge_host_page_size);
+		return -EINVAL;
+	}
+
+	factor = PAGE_SIZE / rdev->lldi.sge_host_page_size;
+	rdev->qpmask = (rdev->lldi.udb_density * factor) - 1;
+	rdev->cqmask = (rdev->lldi.ucq_density * factor) - 1;
+
 	pr_debug("dev %s stag start 0x%0x size 0x%0x num stags %d pbl start 0x%0x size 0x%0x rq start 0x%0x size 0x%0x qp qid start %u size %u cq qid start %u size %u srq size %u\n",
 		 pci_name(rdev->lldi.pdev), rdev->lldi.vr->stag.start,
 		 rdev->lldi.vr->stag.size, c4iw_num_stags(rdev),
-- 
2.13.0

Re: [rdma-rc PATCH 2/2] iw_cxgb4: cq/qp mask depends on bar2 pages in a host page

[Jason Gunthorpe]

On Thu, Feb 14, 2019 at 05:40:54PM +0530, Raju Rangoju wrote:
> Adjust the cq/qp mask based on no.of bar2 pages in a host page.
> 
> For user-mode rdma, the granularity of the BAR2 memory mapped
> to a user rdma process during queue allocation must be based
> on the host page size. The lld attributes udb_density and
> ucq_density are used to figure out how many sge contexts are
> in a bar2 page. So the rdev->qpmask and rdev->cqmask in
> iw_cxgb4 need to now be adjusted based on how many sge bar2
> pages are in a host page.

Why is this rc? Do certain arches fail to work or something?

Jason

Re: [rdma-rc PATCH 1/2] cxgb4: export sge_host_page_size to ulds

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 334 bytes --]

On Thu, Feb 14, 2019 at 05:40:53PM +0530, Raju Rangoju wrote:
> Export the sge_host_page_size field to ULDs via
> cxgb4_lld_info, so that iw_cxgb4 can make use of
> this in calculating the correct qp/cq mask.
>
> Fixes: 2391b0030e ("cxgb4: Remove SGE_HOST_PAGE_SIZE dependency on page
> size")

Please don't break Fixes line.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [rdma-rc PATCH 2/2] iw_cxgb4: cq/qp mask depends on bar2 pages in a host page

[Raju Rangoju]

On Thursday, February 02/14/19, 2019 at 15:41:34 +0000, Jason Gunthorpe wrote:
> On Thu, Feb 14, 2019 at 05:40:54PM +0530, Raju Rangoju wrote:
> > Adjust the cq/qp mask based on no.of bar2 pages in a host page.
> > 
> > For user-mode rdma, the granularity of the BAR2 memory mapped
> > to a user rdma process during queue allocation must be based
> > on the host page size. The lld attributes udb_density and
> > ucq_density are used to figure out how many sge contexts are
> > in a bar2 page. So the rdev->qpmask and rdev->cqmask in
> > iw_cxgb4 need to now be adjusted based on how many sge bar2
> > pages are in a host page.
> 
> Why is this rc? Do certain arches fail to work or something?
>

Yes, this series fixes a regression that was introduced by commit
2391b0030e (v5.0-rc1~129^2~272)

> Jason

RE: [rdma-rc PATCH 2/2] iw_cxgb4: cq/qp mask depends on bar2 pages in a host page

[Steve Wise]

> -----Original Message-----
> From: linux-rdma-owner@vger.kernel.org <linux-rdma-
> owner@vger.kernel.org> On Behalf Of Raju Rangoju
> Sent: Thursday, February 14, 2019 11:28 AM
> To: Jason Gunthorpe <jgg@mellanox.com>
> Cc: davem@davemloft.net; linux-rdma@vger.kernel.org;
> netdev@vger.kernel.org; swise@opengridcomputing.com
> Subject: Re: [rdma-rc PATCH 2/2] iw_cxgb4: cq/qp mask depends on bar2
> pages in a host page
> 
> On Thursday, February 02/14/19, 2019 at 15:41:34 +0000, Jason Gunthorpe
> wrote:
> > On Thu, Feb 14, 2019 at 05:40:54PM +0530, Raju Rangoju wrote:
> > > Adjust the cq/qp mask based on no.of bar2 pages in a host page.
> > >
> > > For user-mode rdma, the granularity of the BAR2 memory mapped
> > > to a user rdma process during queue allocation must be based
> > > on the host page size. The lld attributes udb_density and
> > > ucq_density are used to figure out how many sge contexts are
> > > in a bar2 page. So the rdev->qpmask and rdev->cqmask in
> > > iw_cxgb4 need to now be adjusted based on how many sge bar2
> > > pages are in a host page.
> >
> > Why is this rc? Do certain arches fail to work or something?
> >
> 
> Yes, this series fixes a regression that was introduced by commit
> 2391b0030e (v5.0-rc1~129^2~272)
> 
> > Jason

Rdma over cxgb4 on arches with a non-4K page size are busted w/o this fix.
That was the motivation for -rc.

Re: [rdma-rc PATCH 0/2] iw_cxgb4: Adjust the cq/qp mask

[Jason Gunthorpe]

On Thu, Feb 14, 2019 at 05:40:52PM +0530, Raju Rangoju wrote:
> Export the LLD sge_host_page_size field to ULDs via
> cxgb4_lld_info, so that iw_cxgb4 can adjust the cq/qp
> mask based on no.of bar2 pages in a host page.
> 
> This series has both net(cxgb4) and rdma(iw_cxgb4) changes,
> and I would request this merge via rdma repo.
> 
> I have made sure this series applies cleanly on both net-next
> and rdma-for-rc and doesn't cause any merge conflicts.
> 
> Raju Rangoju (2):
>   cxgb4: export sge_host_page_size to ulds
>   iw_cxgb4: cq/qp mask depends on bar2 pages in a host page

As requested, applied to for-rc with some revisions to the commit
messages.

Thanks,
Jason