* [PATCH rdma-next 0/2] IB/rxe: Replace in kernel memory registration support
@ 2016-07-05 16:05 Moni Shoua
[not found] ` <1467734750-20298-1-git-send-email-monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
0 siblings, 1 reply; 4+ messages in thread
From: Moni Shoua @ 2016-07-05 16:05 UTC (permalink / raw)
To: dledford-H+wXaHxf7aLQT0dZR+AlfA
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, matanb-VPRAkNaXOzVWk0Htik3J/w,
talal-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w,
majd-VPRAkNaXOzVWk0Htik3J/w, Moni Shoua
Registering memory regions in ULPs is implemented with 2 sets of API, FMR
and FastReg. FMR is considered obsolete and FastReg is what modern drivers
should implement.
The patches in this series add FastReg and remove FMR support form the
SoftRoCE driver.
Moni Shoua (1):
IB/rxe: Remove FMR support
Sagi Grimberg (1):
IB/rxe: Add fast memory registraion and remote invalidation) support
drivers/infiniband/hw/rxe/rxe.c | 18 ++-----
drivers/infiniband/hw/rxe/rxe_loc.h | 3 --
drivers/infiniband/hw/rxe/rxe_mr.c | 44 ++-------------
drivers/infiniband/hw/rxe/rxe_opcode.c | 4 +-
drivers/infiniband/hw/rxe/rxe_opcode.h | 1 +
drivers/infiniband/hw/rxe/rxe_param.h | 11 ++--
drivers/infiniband/hw/rxe/rxe_pool.c | 10 +---
drivers/infiniband/hw/rxe/rxe_pool.h | 1 -
drivers/infiniband/hw/rxe/rxe_req.c | 36 +++++++++++++
drivers/infiniband/hw/rxe/rxe_resp.c | 16 ++++++
drivers/infiniband/hw/rxe/rxe_verbs.c | 98 +++++++++++++---------------------
drivers/infiniband/hw/rxe/rxe_verbs.h | 8 +--
include/uapi/rdma/ib_user_rxe.h | 5 ++
13 files changed, 114 insertions(+), 141 deletions(-)
--
1.8.3.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH rdma-next 1/2] IB/rxe: Add fast memory registraion and remote invalidation support
[not found] ` <1467734750-20298-1-git-send-email-monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2016-07-05 16:05 ` Moni Shoua
2016-07-05 16:05 ` [PATCH rdma-next 2/2] IB/rxe: Remove FMR support Moni Shoua
2016-07-12 14:55 ` [PATCH rdma-next 0/2] IB/rxe: Replace in kernel memory registration support Doug Ledford
2 siblings, 0 replies; 4+ messages in thread
From: Moni Shoua @ 2016-07-05 16:05 UTC (permalink / raw)
To: dledford-H+wXaHxf7aLQT0dZR+AlfA
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, matanb-VPRAkNaXOzVWk0Htik3J/w,
talal-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w,
majd-VPRAkNaXOzVWk0Htik3J/w, Sagi Grimberg, Sagi Grimberg,
Moni Shoua
From: Sagi Grimberg <sagig-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
1. Implement ib_map_mr_sg() and populate the private
rxe memory region with the page addresses.
2. Implement the registration (MR state) in the requester
routine.
3. Have the responder support remote invalidation.
Signed-off-by: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org>
Signed-off-by: Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/hw/rxe/rxe_mr.c | 3 ++
drivers/infiniband/hw/rxe/rxe_opcode.c | 4 +--
drivers/infiniband/hw/rxe/rxe_opcode.h | 1 +
drivers/infiniband/hw/rxe/rxe_param.h | 3 +-
drivers/infiniband/hw/rxe/rxe_req.c | 36 +++++++++++++++++++++++
drivers/infiniband/hw/rxe/rxe_resp.c | 16 +++++++++++
drivers/infiniband/hw/rxe/rxe_verbs.c | 52 ++++++++++++++++++++++++++++++++++
drivers/infiniband/hw/rxe/rxe_verbs.h | 1 +
include/uapi/rdma/ib_user_rxe.h | 5 ++++
9 files changed, 118 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/hw/rxe/rxe_mr.c b/drivers/infiniband/hw/rxe/rxe_mr.c
index e533249..6f9bf13 100644
--- a/drivers/infiniband/hw/rxe/rxe_mr.c
+++ b/drivers/infiniband/hw/rxe/rxe_mr.c
@@ -246,6 +246,9 @@ int rxe_mem_init_fast(struct rxe_dev *rxe, struct rxe_pd *pd,
rxe_mem_init(0, mem);
+ /* In fastreg, we also set the rkey */
+ mem->ibmr.rkey = mem->ibmr.lkey;
+
err = rxe_mem_alloc(rxe, mem, max_pages);
if (err)
goto err1;
diff --git a/drivers/infiniband/hw/rxe/rxe_opcode.c b/drivers/infiniband/hw/rxe/rxe_opcode.c
index 4293768..61927c1 100644
--- a/drivers/infiniband/hw/rxe/rxe_opcode.c
+++ b/drivers/infiniband/hw/rxe/rxe_opcode.c
@@ -114,13 +114,13 @@ struct rxe_wr_opcode_info rxe_wr_opcode_info[] = {
[IB_WR_LOCAL_INV] = {
.name = "IB_WR_LOCAL_INV",
.mask = {
- /* not supported */
+ [IB_QPT_RC] = WR_REG_MASK,
},
},
[IB_WR_REG_MR] = {
.name = "IB_WR_REG_MR",
.mask = {
- /* not supported */
+ [IB_QPT_RC] = WR_REG_MASK,
},
},
};
diff --git a/drivers/infiniband/hw/rxe/rxe_opcode.h b/drivers/infiniband/hw/rxe/rxe_opcode.h
index 0c5f979..307604e 100644
--- a/drivers/infiniband/hw/rxe/rxe_opcode.h
+++ b/drivers/infiniband/hw/rxe/rxe_opcode.h
@@ -47,6 +47,7 @@ enum rxe_wr_mask {
WR_READ_MASK = BIT(3),
WR_WRITE_MASK = BIT(4),
WR_LOCAL_MASK = BIT(5),
+ WR_REG_MASK = BIT(6),
WR_READ_OR_WRITE_MASK = WR_READ_MASK | WR_WRITE_MASK,
WR_READ_WRITE_OR_SEND_MASK = WR_READ_OR_WRITE_MASK | WR_SEND_MASK,
diff --git a/drivers/infiniband/hw/rxe/rxe_param.h b/drivers/infiniband/hw/rxe/rxe_param.h
index 656a1a1..27ac76c 100644
--- a/drivers/infiniband/hw/rxe/rxe_param.h
+++ b/drivers/infiniband/hw/rxe/rxe_param.h
@@ -77,7 +77,8 @@ enum rxe_device_param {
| IB_DEVICE_PORT_ACTIVE_EVENT
| IB_DEVICE_SYS_IMAGE_GUID
| IB_DEVICE_RC_RNR_NAK_GEN
- | IB_DEVICE_SRQ_RESIZE,
+ | IB_DEVICE_SRQ_RESIZE
+ | IB_DEVICE_MEM_MGT_EXTENSIONS,
RXE_MAX_SGE = 32,
RXE_MAX_SGE_RD = 32,
RXE_MAX_CQ = 16384,
diff --git a/drivers/infiniband/hw/rxe/rxe_req.c b/drivers/infiniband/hw/rxe/rxe_req.c
index f78efa6..33b2d9d 100644
--- a/drivers/infiniband/hw/rxe/rxe_req.c
+++ b/drivers/infiniband/hw/rxe/rxe_req.c
@@ -251,6 +251,9 @@ static int next_opcode_rc(struct rxe_qp *qp, unsigned opcode, int fits)
else
return fits ? IB_OPCODE_RC_SEND_ONLY_WITH_INVALIDATE :
IB_OPCODE_RC_SEND_FIRST;
+ case IB_WR_REG_MR:
+ case IB_WR_LOCAL_INV:
+ return opcode;
}
return -EINVAL;
@@ -592,6 +595,39 @@ next_wqe:
if (unlikely(!wqe))
goto exit;
+ if (wqe->mask & WR_REG_MASK) {
+ if (wqe->wr.opcode == IB_WR_LOCAL_INV) {
+ struct rxe_dev *rxe = to_rdev(qp->ibqp.device);
+ struct rxe_mem *rmr;
+
+ rmr = rxe_pool_get_index(&rxe->mr_pool,
+ wqe->wr.ex.invalidate_rkey >> 8);
+ if (!rmr) {
+ pr_err("No mr for key %#x\n", wqe->wr.ex.invalidate_rkey);
+ wqe->state = wqe_state_error;
+ wqe->status = IB_WC_MW_BIND_ERR;
+ goto exit;
+ }
+ rmr->state = RXE_MEM_STATE_FREE;
+ wqe->state = wqe_state_done;
+ wqe->status = IB_WC_SUCCESS;
+ } else if (wqe->wr.opcode == IB_WR_REG_MR) {
+ struct rxe_mem *rmr = to_rmr(wqe->wr.wr.reg.mr);
+
+ rmr->state = RXE_MEM_STATE_VALID;
+ rmr->access = wqe->wr.wr.reg.access;
+ rmr->lkey = wqe->wr.wr.reg.key;
+ rmr->rkey = wqe->wr.wr.reg.key;
+ wqe->state = wqe_state_done;
+ wqe->status = IB_WC_SUCCESS;
+ } else {
+ goto exit;
+ }
+ qp->req.wqe_index = next_index(qp->sq.queue,
+ qp->req.wqe_index);
+ goto next_wqe;
+ }
+
if (unlikely(qp_type(qp) == IB_QPT_RC &&
qp->req.psn > (qp->comp.psn + RXE_MAX_UNACKED_PSNS))) {
qp->req.wait_psn = 1;
diff --git a/drivers/infiniband/hw/rxe/rxe_resp.c b/drivers/infiniband/hw/rxe/rxe_resp.c
index a00a743..ebb03b4 100644
--- a/drivers/infiniband/hw/rxe/rxe_resp.c
+++ b/drivers/infiniband/hw/rxe/rxe_resp.c
@@ -455,6 +455,11 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
goto err1;
}
+ if (unlikely(mem->state == RXE_MEM_STATE_FREE)) {
+ state = RESPST_ERR_RKEY_VIOLATION;
+ goto err1;
+ }
+
if (mem_check_range(mem, va, resid)) {
state = RESPST_ERR_RKEY_VIOLATION;
goto err2;
@@ -867,8 +872,19 @@ static enum resp_states do_complete(struct rxe_qp *qp,
}
if (pkt->mask & RXE_IETH_MASK) {
+ struct rxe_dev *rxe = to_rdev(qp->ibqp.device);
+ struct rxe_mem *rmr;
+
wc->wc_flags |= IB_WC_WITH_INVALIDATE;
wc->ex.invalidate_rkey = ieth_rkey(pkt);
+
+ rmr = rxe_pool_get_index(&rxe->mr_pool,
+ wc->ex.invalidate_rkey >> 8);
+ if (unlikely(!rmr)) {
+ pr_err("Bad rkey %#x invalidation\n", wc->ex.invalidate_rkey);
+ return RESPST_ERROR;
+ }
+ rmr->state = RXE_MEM_STATE_FREE;
}
wc->qp = &qp->ibqp;
diff --git a/drivers/infiniband/hw/rxe/rxe_verbs.c b/drivers/infiniband/hw/rxe/rxe_verbs.c
index 898b3bb..c5d48c4 100644
--- a/drivers/infiniband/hw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/hw/rxe/rxe_verbs.c
@@ -692,6 +692,14 @@ static void init_send_wr(struct rxe_qp *qp, struct rxe_send_wr *wr,
wr->wr.atomic.swap = atomic_wr(ibwr)->swap;
wr->wr.atomic.rkey = atomic_wr(ibwr)->rkey;
break;
+ case IB_WR_LOCAL_INV:
+ wr->ex.invalidate_rkey = ibwr->ex.invalidate_rkey;
+ break;
+ case IB_WR_REG_MR:
+ wr->wr.reg.mr = reg_wr(ibwr)->mr;
+ wr->wr.reg.key = reg_wr(ibwr)->key;
+ wr->wr.reg.access = reg_wr(ibwr)->access;
+ break;
default:
break;
}
@@ -729,6 +737,10 @@ static int init_send_wqe(struct rxe_qp *qp, struct ib_send_wr *ibwr,
p += sge->length;
}
+ } else if (mask & WR_REG_MASK) {
+ wqe->mask = mask;
+ wqe->state = wqe_state_posted;
+ return 0;
} else
memcpy(wqe->dma.sge, ibwr->sg_list,
num_sge * sizeof(struct ib_sge));
@@ -1102,6 +1114,45 @@ err1:
return ERR_PTR(err);
}
+static int rxe_set_page(struct ib_mr *ibmr, u64 addr)
+{
+ struct rxe_mem *mr = to_rmr(ibmr);
+ struct rxe_map *map;
+ struct rxe_phys_buf *buf;
+
+ if (unlikely(mr->nbuf == mr->num_buf))
+ return -ENOMEM;
+
+ map = mr->map[mr->nbuf / RXE_BUF_PER_MAP];
+ buf = &map->buf[mr->nbuf % RXE_BUF_PER_MAP];
+
+ buf->addr = addr;
+ buf->size = ibmr->page_size;
+ mr->nbuf++;
+
+ return 0;
+}
+
+static int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents,
+ unsigned int *sg_offset)
+{
+ struct rxe_mem *mr = to_rmr(ibmr);
+ int n;
+
+ mr->nbuf = 0;
+
+ n = ib_sg_to_pages(ibmr, sg, sg_nents, sg_offset, rxe_set_page);
+
+ mr->va = ibmr->iova;
+ mr->iova = ibmr->iova;
+ mr->length = ibmr->length;
+ mr->page_shift = ilog2(ibmr->page_size);
+ mr->page_mask = ibmr->page_size - 1;
+ mr->offset = mr->iova & mr->page_mask;
+
+ return n;
+}
+
static struct ib_fmr *rxe_alloc_fmr(struct ib_pd *ibpd,
int access, struct ib_fmr_attr *attr)
{
@@ -1308,6 +1359,7 @@ int rxe_register_device(struct rxe_dev *rxe)
dev->reg_user_mr = rxe_reg_user_mr;
dev->dereg_mr = rxe_dereg_mr;
dev->alloc_mr = rxe_alloc_mr;
+ dev->map_mr_sg = rxe_map_mr_sg;
dev->alloc_fmr = rxe_alloc_fmr;
dev->map_phys_fmr = rxe_map_phys_fmr;
dev->unmap_fmr = rxe_unmap_fmr;
diff --git a/drivers/infiniband/hw/rxe/rxe_verbs.h b/drivers/infiniband/hw/rxe/rxe_verbs.h
index ef73edb..d34c056 100644
--- a/drivers/infiniband/hw/rxe/rxe_verbs.h
+++ b/drivers/infiniband/hw/rxe/rxe_verbs.h
@@ -334,6 +334,7 @@ struct rxe_mem {
int map_mask;
u32 num_buf;
+ u32 nbuf;
u32 max_buf;
u32 num_map;
diff --git a/include/uapi/rdma/ib_user_rxe.h b/include/uapi/rdma/ib_user_rxe.h
index ee17d49..19f9615 100644
--- a/include/uapi/rdma/ib_user_rxe.h
+++ b/include/uapi/rdma/ib_user_rxe.h
@@ -87,6 +87,11 @@ struct rxe_send_wr {
__u32 remote_qkey;
__u16 pkey_index;
} ud;
+ struct {
+ struct ib_mr *mr;
+ __u32 key;
+ int access;
+ } reg;
} wr;
};
--
1.8.3.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH rdma-next 2/2] IB/rxe: Remove FMR support
[not found] ` <1467734750-20298-1-git-send-email-monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2016-07-05 16:05 ` [PATCH rdma-next 1/2] IB/rxe: Add fast memory registraion and remote invalidation support Moni Shoua
@ 2016-07-05 16:05 ` Moni Shoua
2016-07-12 14:55 ` [PATCH rdma-next 0/2] IB/rxe: Replace in kernel memory registration support Doug Ledford
2 siblings, 0 replies; 4+ messages in thread
From: Moni Shoua @ 2016-07-05 16:05 UTC (permalink / raw)
To: dledford-H+wXaHxf7aLQT0dZR+AlfA
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, matanb-VPRAkNaXOzVWk0Htik3J/w,
talal-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w,
majd-VPRAkNaXOzVWk0Htik3J/w, Moni Shoua
Fast memory registration via the send queue is the preferred method for
memory registration inside the kernel. FMR interface is considered
obsolete and shouldn't be supported with in modern drivers.
Signed-off-by: Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/hw/rxe/rxe.c | 18 +++------
drivers/infiniband/hw/rxe/rxe_loc.h | 3 --
drivers/infiniband/hw/rxe/rxe_mr.c | 41 +------------------
drivers/infiniband/hw/rxe/rxe_param.h | 8 ++--
drivers/infiniband/hw/rxe/rxe_pool.c | 10 +----
drivers/infiniband/hw/rxe/rxe_pool.h | 1 -
drivers/infiniband/hw/rxe/rxe_verbs.c | 74 -----------------------------------
drivers/infiniband/hw/rxe/rxe_verbs.h | 7 ----
8 files changed, 10 insertions(+), 152 deletions(-)
diff --git a/drivers/infiniband/hw/rxe/rxe.c b/drivers/infiniband/hw/rxe/rxe.c
index d49418b..55f0e8f 100644
--- a/drivers/infiniband/hw/rxe/rxe.c
+++ b/drivers/infiniband/hw/rxe/rxe.c
@@ -59,7 +59,6 @@ static void rxe_cleanup(struct rxe_dev *rxe)
rxe_pool_cleanup(&rxe->qp_pool);
rxe_pool_cleanup(&rxe->cq_pool);
rxe_pool_cleanup(&rxe->mr_pool);
- rxe_pool_cleanup(&rxe->fmr_pool);
rxe_pool_cleanup(&rxe->mw_pool);
rxe_pool_cleanup(&rxe->mc_grp_pool);
rxe_pool_cleanup(&rxe->mc_elem_pool);
@@ -226,34 +225,27 @@ static int rxe_init_pools(struct rxe_dev *rxe)
if (err)
goto err7;
- err = rxe_pool_init(rxe, &rxe->fmr_pool, RXE_TYPE_FMR,
- rxe->attr.max_fmr);
- if (err)
- goto err8;
-
err = rxe_pool_init(rxe, &rxe->mw_pool, RXE_TYPE_MW,
rxe->attr.max_mw);
if (err)
- goto err9;
+ goto err8;
err = rxe_pool_init(rxe, &rxe->mc_grp_pool, RXE_TYPE_MC_GRP,
rxe->attr.max_mcast_grp);
if (err)
- goto err10;
+ goto err9;
err = rxe_pool_init(rxe, &rxe->mc_elem_pool, RXE_TYPE_MC_ELEM,
rxe->attr.max_total_mcast_qp_attach);
if (err)
- goto err11;
+ goto err10;
return 0;
-err11:
- rxe_pool_cleanup(&rxe->mc_grp_pool);
err10:
- rxe_pool_cleanup(&rxe->mw_pool);
+ rxe_pool_cleanup(&rxe->mc_grp_pool);
err9:
- rxe_pool_cleanup(&rxe->fmr_pool);
+ rxe_pool_cleanup(&rxe->mw_pool);
err8:
rxe_pool_cleanup(&rxe->mr_pool);
err7:
diff --git a/drivers/infiniband/hw/rxe/rxe_loc.h b/drivers/infiniband/hw/rxe/rxe_loc.h
index 34ac568..4a5484e 100644
--- a/drivers/infiniband/hw/rxe/rxe_loc.h
+++ b/drivers/infiniband/hw/rxe/rxe_loc.h
@@ -115,9 +115,6 @@ int rxe_mem_init_user(struct rxe_dev *rxe, struct rxe_pd *pd, u64 start,
int rxe_mem_init_fast(struct rxe_dev *rxe, struct rxe_pd *pd,
int max_pages, struct rxe_mem *mem);
-int rxe_mem_init_fmr(struct rxe_dev *rxe, struct rxe_pd *pd, int access,
- struct ib_fmr_attr *attr, struct rxe_mem *fmr);
-
int rxe_mem_copy(struct rxe_mem *mem, u64 iova, void *addr,
int length, enum copy_direction dir, u32 *crcp);
diff --git a/drivers/infiniband/hw/rxe/rxe_mr.c b/drivers/infiniband/hw/rxe/rxe_mr.c
index 6f9bf13..f3dab65 100644
--- a/drivers/infiniband/hw/rxe/rxe_mr.c
+++ b/drivers/infiniband/hw/rxe/rxe_mr.c
@@ -80,9 +80,6 @@ static void rxe_mem_init(int access, struct rxe_mem *mem)
if (mem->pelem.pool->type == RXE_TYPE_MR) {
mem->ibmr.lkey = lkey;
mem->ibmr.rkey = rkey;
- } else {
- mem->ibfmr.lkey = lkey;
- mem->ibfmr.rkey = rkey;
}
mem->lkey = lkey;
@@ -264,38 +261,6 @@ err1:
return err;
}
-int rxe_mem_init_fmr(struct rxe_dev *rxe, struct rxe_pd *pd, int access,
- struct ib_fmr_attr *attr, struct rxe_mem *mem)
-{
- int err;
-
- if (attr->max_maps > rxe->attr.max_map_per_fmr) {
- pr_warn("max_mmaps = %d too big, max_map_per_fmr = %d\n",
- attr->max_maps, rxe->attr.max_map_per_fmr);
- err = -EINVAL;
- goto err1;
- }
-
- rxe_mem_init(access, mem);
-
- err = rxe_mem_alloc(rxe, mem, attr->max_pages);
- if (err)
- goto err1;
-
- mem->pd = pd;
- mem->access = access;
- mem->page_shift = attr->page_shift;
- mem->page_mask = (1 << attr->page_shift) - 1;
- mem->max_buf = attr->max_pages;
- mem->state = RXE_MEM_STATE_FREE;
- mem->type = RXE_MEM_TYPE_FMR;
-
- return 0;
-
-err1:
- return err;
-}
-
static void lookup_iova(
struct rxe_mem *mem,
u64 iova,
@@ -591,7 +556,7 @@ int advance_dma_data(struct rxe_dma_info *dma, unsigned int length)
return 0;
}
-/* (1) find the mem (mr, fmr or mw) corresponding to lkey/rkey
+/* (1) find the mem (mr or mw) corresponding to lkey/rkey
* depending on lookup_type
* (2) verify that the (qp) pd matches the mem pd
* (3) verify that the mem can support the requested access
@@ -608,10 +573,6 @@ struct rxe_mem *lookup_mem(struct rxe_pd *pd, int access, u32 key,
mem = rxe_pool_get_index(&rxe->mr_pool, index);
if (!mem)
goto err1;
- } else if (index >= RXE_MIN_FMR_INDEX && index <= RXE_MAX_FMR_INDEX) {
- mem = rxe_pool_get_index(&rxe->fmr_pool, index);
- if (!mem)
- goto err1;
} else {
goto err1;
}
diff --git a/drivers/infiniband/hw/rxe/rxe_param.h b/drivers/infiniband/hw/rxe/rxe_param.h
index 27ac76c..f459c43 100644
--- a/drivers/infiniband/hw/rxe/rxe_param.h
+++ b/drivers/infiniband/hw/rxe/rxe_param.h
@@ -100,8 +100,8 @@ enum rxe_device_param {
RXE_MAX_MCAST_QP_ATTACH = 56,
RXE_MAX_TOT_MCAST_QP_ATTACH = 0x70000,
RXE_MAX_AH = 100,
- RXE_MAX_FMR = 2 * 1024,
- RXE_MAX_MAP_PER_FMR = 100,
+ RXE_MAX_FMR = 0,
+ RXE_MAX_MAP_PER_FMR = 0,
RXE_MAX_SRQ = 960,
RXE_MAX_SRQ_WR = 0x4000,
RXE_MIN_SRQ_WR = 1,
@@ -123,9 +123,7 @@ enum rxe_device_param {
RXE_MAX_SRQ_INDEX = 0x00040000,
RXE_MIN_MR_INDEX = 0x00000001,
- RXE_MAX_MR_INDEX = 0x00020000,
- RXE_MIN_FMR_INDEX = 0x00020001,
- RXE_MAX_FMR_INDEX = 0x00040000,
+ RXE_MAX_MR_INDEX = 0x00040000,
RXE_MIN_MW_INDEX = 0x00040001,
RXE_MAX_MW_INDEX = 0x00060000,
RXE_MAX_PKT_PER_ACK = 64,
diff --git a/drivers/infiniband/hw/rxe/rxe_pool.c b/drivers/infiniband/hw/rxe/rxe_pool.c
index cd37d4f..6bac071 100644
--- a/drivers/infiniband/hw/rxe/rxe_pool.c
+++ b/drivers/infiniband/hw/rxe/rxe_pool.c
@@ -35,7 +35,7 @@
#include "rxe_loc.h"
/* info about object pools
- * note that mr, fmr and mw share a single index space
+ * note that mr and mw share a single index space
* so that one can map an lkey to the correct type of object
*/
struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = {
@@ -80,14 +80,6 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = {
.max_index = RXE_MAX_MR_INDEX,
.min_index = RXE_MIN_MR_INDEX,
},
- [RXE_TYPE_FMR] = {
- .name = "rxe-fmr",
- .size = sizeof(struct rxe_mem),
- .cleanup = rxe_mem_cleanup,
- .flags = RXE_POOL_INDEX,
- .max_index = RXE_MAX_FMR_INDEX,
- .min_index = RXE_MIN_FMR_INDEX,
- },
[RXE_TYPE_MW] = {
.name = "rxe-mw",
.size = sizeof(struct rxe_mem),
diff --git a/drivers/infiniband/hw/rxe/rxe_pool.h b/drivers/infiniband/hw/rxe/rxe_pool.h
index 43476b5..4d04830 100644
--- a/drivers/infiniband/hw/rxe/rxe_pool.h
+++ b/drivers/infiniband/hw/rxe/rxe_pool.h
@@ -52,7 +52,6 @@ enum rxe_elem_type {
RXE_TYPE_CQ,
RXE_TYPE_MR,
RXE_TYPE_MW,
- RXE_TYPE_FMR,
RXE_TYPE_MC_GRP,
RXE_TYPE_MC_ELEM,
RXE_NUM_TYPES, /* keep me last */
diff --git a/drivers/infiniband/hw/rxe/rxe_verbs.c b/drivers/infiniband/hw/rxe/rxe_verbs.c
index c5d48c4..4552be9 100644
--- a/drivers/infiniband/hw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/hw/rxe/rxe_verbs.c
@@ -1153,76 +1153,6 @@ static int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nent
return n;
}
-static struct ib_fmr *rxe_alloc_fmr(struct ib_pd *ibpd,
- int access, struct ib_fmr_attr *attr)
-{
- struct rxe_dev *rxe = to_rdev(ibpd->device);
- struct rxe_pd *pd = to_rpd(ibpd);
- struct rxe_mem *fmr;
- int err;
-
- fmr = rxe_alloc(&rxe->fmr_pool);
- if (!fmr) {
- err = -ENOMEM;
- goto err1;
- }
-
- rxe_add_index(fmr);
-
- rxe_add_ref(pd);
-
- err = rxe_mem_init_fmr(rxe, pd, access, attr, fmr);
- if (err)
- goto err2;
-
- return &fmr->ibfmr;
-
-err2:
- rxe_drop_ref(pd);
- rxe_drop_index(fmr);
- rxe_drop_ref(fmr);
-err1:
- return ERR_PTR(err);
-}
-
-static int rxe_map_phys_fmr(struct ib_fmr *ibfmr,
- u64 *page_list, int list_length, u64 iova)
-{
- struct rxe_mem *fmr = to_rfmr(ibfmr);
- struct rxe_dev *rxe = to_rdev(ibfmr->device);
-
- return rxe_mem_map_pages(rxe, fmr, page_list, list_length, iova);
-}
-
-static int rxe_unmap_fmr(struct list_head *fmr_list)
-{
- struct rxe_mem *fmr;
-
- list_for_each_entry(fmr, fmr_list, ibfmr.list) {
- if (fmr->state != RXE_MEM_STATE_VALID)
- continue;
-
- fmr->va = 0;
- fmr->iova = 0;
- fmr->length = 0;
- fmr->num_buf = 0;
- fmr->state = RXE_MEM_STATE_FREE;
- }
-
- return 0;
-}
-
-static int rxe_dealloc_fmr(struct ib_fmr *ibfmr)
-{
- struct rxe_mem *fmr = to_rfmr(ibfmr);
-
- fmr->state = RXE_MEM_STATE_ZOMBIE;
- rxe_drop_ref(fmr->pd);
- rxe_drop_index(fmr);
- rxe_drop_ref(fmr);
- return 0;
-}
-
static int rxe_attach_mcast(struct ib_qp *ibqp, union ib_gid *mgid, u16 mlid)
{
int err;
@@ -1360,10 +1290,6 @@ int rxe_register_device(struct rxe_dev *rxe)
dev->dereg_mr = rxe_dereg_mr;
dev->alloc_mr = rxe_alloc_mr;
dev->map_mr_sg = rxe_map_mr_sg;
- dev->alloc_fmr = rxe_alloc_fmr;
- dev->map_phys_fmr = rxe_map_phys_fmr;
- dev->unmap_fmr = rxe_unmap_fmr;
- dev->dealloc_fmr = rxe_dealloc_fmr;
dev->attach_mcast = rxe_attach_mcast;
dev->detach_mcast = rxe_detach_mcast;
diff --git a/drivers/infiniband/hw/rxe/rxe_verbs.h b/drivers/infiniband/hw/rxe/rxe_verbs.h
index d34c056..feeb41e 100644
--- a/drivers/infiniband/hw/rxe/rxe_verbs.h
+++ b/drivers/infiniband/hw/rxe/rxe_verbs.h
@@ -310,7 +310,6 @@ struct rxe_mem {
struct rxe_pool_entry pelem;
union {
struct ib_mr ibmr;
- struct ib_fmr ibfmr;
struct ib_mw ibmw;
};
@@ -415,7 +414,6 @@ struct rxe_dev {
struct rxe_pool cq_pool;
struct rxe_pool mr_pool;
struct rxe_pool mw_pool;
- struct rxe_pool fmr_pool;
struct rxe_pool mc_grp_pool;
struct rxe_pool mc_elem_pool;
@@ -469,11 +467,6 @@ static inline struct rxe_mem *to_rmr(struct ib_mr *mr)
return mr ? container_of(mr, struct rxe_mem, ibmr) : NULL;
}
-static inline struct rxe_mem *to_rfmr(struct ib_fmr *fmr)
-{
- return fmr ? container_of(fmr, struct rxe_mem, ibfmr) : NULL;
-}
-
static inline struct rxe_mem *to_rmw(struct ib_mw *mw)
{
return mw ? container_of(mw, struct rxe_mem, ibmw) : NULL;
--
1.8.3.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH rdma-next 0/2] IB/rxe: Replace in kernel memory registration support
[not found] ` <1467734750-20298-1-git-send-email-monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2016-07-05 16:05 ` [PATCH rdma-next 1/2] IB/rxe: Add fast memory registraion and remote invalidation support Moni Shoua
2016-07-05 16:05 ` [PATCH rdma-next 2/2] IB/rxe: Remove FMR support Moni Shoua
@ 2016-07-12 14:55 ` Doug Ledford
2 siblings, 0 replies; 4+ messages in thread
From: Doug Ledford @ 2016-07-12 14:55 UTC (permalink / raw)
To: Moni Shoua
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, matanb-VPRAkNaXOzVWk0Htik3J/w,
talal-VPRAkNaXOzVWk0Htik3J/w, leonro-VPRAkNaXOzVWk0Htik3J/w,
majd-VPRAkNaXOzVWk0Htik3J/w
[-- Attachment #1.1: Type: text/plain, Size: 1382 bytes --]
On 7/5/2016 12:05 PM, Moni Shoua wrote:
> Registering memory regions in ULPs is implemented with 2 sets of API, FMR
> and FastReg. FMR is considered obsolete and FastReg is what modern drivers
> should implement.
> The patches in this series add FastReg and remove FMR support form the
> SoftRoCE driver.
>
> Moni Shoua (1):
> IB/rxe: Remove FMR support
>
> Sagi Grimberg (1):
> IB/rxe: Add fast memory registraion and remote invalidation) support
>
> drivers/infiniband/hw/rxe/rxe.c | 18 ++-----
> drivers/infiniband/hw/rxe/rxe_loc.h | 3 --
> drivers/infiniband/hw/rxe/rxe_mr.c | 44 ++-------------
> drivers/infiniband/hw/rxe/rxe_opcode.c | 4 +-
> drivers/infiniband/hw/rxe/rxe_opcode.h | 1 +
> drivers/infiniband/hw/rxe/rxe_param.h | 11 ++--
> drivers/infiniband/hw/rxe/rxe_pool.c | 10 +---
> drivers/infiniband/hw/rxe/rxe_pool.h | 1 -
> drivers/infiniband/hw/rxe/rxe_req.c | 36 +++++++++++++
> drivers/infiniband/hw/rxe/rxe_resp.c | 16 ++++++
> drivers/infiniband/hw/rxe/rxe_verbs.c | 98 +++++++++++++---------------------
> drivers/infiniband/hw/rxe/rxe_verbs.h | 8 +--
> include/uapi/rdma/ib_user_rxe.h | 5 ++
> 13 files changed, 114 insertions(+), 141 deletions(-)
>
Series applied.
--
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
GPG Key ID: 0E572FDD
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-07-12 14:55 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-07-05 16:05 [PATCH rdma-next 0/2] IB/rxe: Replace in kernel memory registration support Moni Shoua
[not found] ` <1467734750-20298-1-git-send-email-monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2016-07-05 16:05 ` [PATCH rdma-next 1/2] IB/rxe: Add fast memory registraion and remote invalidation support Moni Shoua
2016-07-05 16:05 ` [PATCH rdma-next 2/2] IB/rxe: Remove FMR support Moni Shoua
2016-07-12 14:55 ` [PATCH rdma-next 0/2] IB/rxe: Replace in kernel memory registration support Doug Ledford
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.