* [PATCH v3 0/3] RDMA/rxe: Add RDMA Atomic Write operation
@ 2022-03-11 11:52 Xiao Yang
2022-03-11 11:52 ` [PATCH v3 1/3] RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res Xiao Yang
` (2 more replies)
0 siblings, 3 replies; 15+ messages in thread
From: Xiao Yang @ 2022-03-11 11:52 UTC (permalink / raw)
To: linux-rdma
Cc: yanjun.zhu, rpearsonhpe, jgg, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny, Xiao Yang
The IB SPEC v1.5[1] defined new RDMA Atomic Write operation. This
patchset makes SoftRoCE support new RDMA Atomic Write on RC service.
I add ibv_wr_rdma_atomic_write() and a rdma_atomic_write example on my
rdma-core repository[2]. You can verify the patchset by building and
running the rdma_atomic_write example.
server:
$ ./rdma_atomic_write_server -s [server_address] -p [port_number]
client:
$ ./rdma_atomic_write_client -s [server_address] -p [port_number]
[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/yangx-jy/rdma-core/tree/new_api
V2->V3:
1) Rebase
2) Add RDMA Atomic Write attribute for rxe device
V1->V2:
1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D
2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value
3) Use smp_store_release() to ensure that all prior operations have completed
Xiao Yang (3):
RDMA/rxe: Rename send_atomic_ack() and atomic member of struct
resp_res
RDMA/rxe: Support RDMA Atomic Write operation
RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++
drivers/infiniband/sw/rxe/rxe_opcode.c | 19 +++++++++
drivers/infiniband/sw/rxe/rxe_opcode.h | 3 ++
drivers/infiniband/sw/rxe/rxe_param.h | 3 +-
drivers/infiniband/sw/rxe/rxe_qp.c | 5 ++-
drivers/infiniband/sw/rxe/rxe_req.c | 11 +++++-
drivers/infiniband/sw/rxe/rxe_resp.c | 55 ++++++++++++++++++++------
drivers/infiniband/sw/rxe/rxe_verbs.h | 2 +-
include/rdma/ib_pack.h | 2 +
include/rdma/ib_verbs.h | 4 ++
include/uapi/rdma/ib_user_verbs.h | 2 +
include/uapi/rdma/rdma_user_rxe.h | 1 +
12 files changed, 92 insertions(+), 19 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread* [PATCH v3 1/3] RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res 2022-03-11 11:52 [PATCH v3 0/3] RDMA/rxe: Add RDMA Atomic Write operation Xiao Yang @ 2022-03-11 11:52 ` Xiao Yang 2022-03-11 11:52 ` [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation Xiao Yang 2022-03-11 11:52 ` [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device Xiao Yang 2 siblings, 0 replies; 15+ messages in thread From: Xiao Yang @ 2022-03-11 11:52 UTC (permalink / raw) To: linux-rdma Cc: yanjun.zhu, rpearsonhpe, jgg, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny, Xiao Yang send_atomic_ack() and atomic member of struct resp_res will be common in the future so rename them. Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com> --- drivers/infiniband/sw/rxe/rxe_qp.c | 2 +- drivers/infiniband/sw/rxe/rxe_resp.c | 10 +++++----- drivers/infiniband/sw/rxe/rxe_verbs.h | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 5018b9387694..ca07e37d30a6 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -136,7 +136,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp) void free_rd_atomic_resource(struct rxe_qp *qp, struct resp_res *res) { if (res->type == RXE_ATOMIC_MASK) { - kfree_skb(res->atomic.skb); + kfree_skb(res->resp.skb); } else if (res->type == RXE_READ_MASK) { if (res->read.mr) rxe_drop_ref(res->read.mr); diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index e8f435fa6e4d..e015860e8c34 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -957,7 +957,7 @@ static int send_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt, return err; } -static int send_atomic_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt, +static int send_resp(struct rxe_qp *qp, struct rxe_pkt_info *pkt, u8 syndrome) { int rc = 0; @@ -979,7 +979,7 @@ static int send_atomic_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt, skb_get(skb); res->type = RXE_ATOMIC_MASK; - res->atomic.skb = skb; + res->resp.skb = skb; res->first_psn = ack_pkt.psn; res->last_psn = ack_pkt.psn; res->cur_psn = ack_pkt.psn; @@ -1002,7 +1002,7 @@ static enum resp_states acknowledge(struct rxe_qp *qp, if (qp->resp.aeth_syndrome != AETH_ACK_UNLIMITED) send_ack(qp, pkt, qp->resp.aeth_syndrome, pkt->psn); else if (pkt->mask & RXE_ATOMIC_MASK) - send_atomic_ack(qp, pkt, AETH_ACK_UNLIMITED); + send_resp(qp, pkt, AETH_ACK_UNLIMITED); else if (bth_ack(pkt)) send_ack(qp, pkt, AETH_ACK_UNLIMITED, pkt->psn); @@ -1111,9 +1111,9 @@ static enum resp_states duplicate_request(struct rxe_qp *qp, /* Find the operation in our list of responder resources. */ res = find_resource(qp, pkt->psn); if (res) { - skb_get(res->atomic.skb); + skb_get(res->resp.skb); /* Resend the result. */ - rc = rxe_xmit_packet(qp, pkt, res->atomic.skb); + rc = rxe_xmit_packet(qp, pkt, res->resp.skb); if (rc) { pr_err("Failed resending result. This flow is not handled - skb ignored\n"); rc = RESPST_CLEANUP; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index e48969e8d4c8..01421286ed07 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -155,7 +155,7 @@ struct resp_res { union { struct { struct sk_buff *skb; - } atomic; + } resp; struct { struct rxe_mr *mr; u64 va_org; -- 2.34.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation 2022-03-11 11:52 [PATCH v3 0/3] RDMA/rxe: Add RDMA Atomic Write operation Xiao Yang 2022-03-11 11:52 ` [PATCH v3 1/3] RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res Xiao Yang @ 2022-03-11 11:52 ` Xiao Yang 2022-03-11 23:56 ` kernel test robot 2022-03-11 11:52 ` [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device Xiao Yang 2 siblings, 1 reply; 15+ messages in thread From: Xiao Yang @ 2022-03-11 11:52 UTC (permalink / raw) To: linux-rdma Cc: yanjun.zhu, rpearsonhpe, jgg, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny, Xiao Yang This patch implements RDMA Atomic Write operation for RC service. Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com> --- drivers/infiniband/sw/rxe/rxe_comp.c | 4 +++ drivers/infiniband/sw/rxe/rxe_opcode.c | 19 +++++++++++ drivers/infiniband/sw/rxe/rxe_opcode.h | 3 ++ drivers/infiniband/sw/rxe/rxe_qp.c | 3 +- drivers/infiniband/sw/rxe/rxe_req.c | 11 +++++-- drivers/infiniband/sw/rxe/rxe_resp.c | 45 +++++++++++++++++++++----- include/rdma/ib_pack.h | 2 ++ include/rdma/ib_verbs.h | 2 ++ include/uapi/rdma/ib_user_verbs.h | 2 ++ include/uapi/rdma/rdma_user_rxe.h | 1 + 10 files changed, 81 insertions(+), 11 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index f363fe3fa414..6de7ac0b94ed 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -104,6 +104,7 @@ static enum ib_wc_opcode wr_to_wc_opcode(enum ib_wr_opcode opcode) case IB_WR_LOCAL_INV: return IB_WC_LOCAL_INV; case IB_WR_REG_MR: return IB_WC_REG_MR; case IB_WR_BIND_MW: return IB_WC_BIND_MW; + case IB_WR_RDMA_ATOMIC_WRITE: return IB_WC_RDMA_ATOMIC_WRITE; default: return 0xff; @@ -256,6 +257,9 @@ static inline enum comp_state check_ack(struct rxe_qp *qp, if ((syn & AETH_TYPE_MASK) != AETH_ACK) return COMPST_ERROR; + if (wqe->wr.opcode == IB_WR_RDMA_ATOMIC_WRITE) + return COMPST_WRITE_SEND; + fallthrough; /* (IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE doesn't have an AETH) */ diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.c b/drivers/infiniband/sw/rxe/rxe_opcode.c index df596ba7527d..9b1ca0b3618a 100644 --- a/drivers/infiniband/sw/rxe/rxe_opcode.c +++ b/drivers/infiniband/sw/rxe/rxe_opcode.c @@ -103,6 +103,12 @@ struct rxe_wr_opcode_info rxe_wr_opcode_info[] = { [IB_QPT_UC] = WR_LOCAL_OP_MASK, }, }, + [IB_WR_RDMA_ATOMIC_WRITE] = { + .name = "IB_WR_RDMA_ATOMIC_WRITE", + .mask = { + [IB_QPT_RC] = WR_ATOMIC_WRITE_MASK, + }, + }, }; struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = { @@ -380,6 +386,19 @@ struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = { RXE_IETH_BYTES, } }, + [IB_OPCODE_RC_RDMA_ATOMIC_WRITE] = { + .name = "IB_OPCODE_RC_RDMA_ATOMIC_WRITE", + .mask = RXE_RETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK | + RXE_ATOMIC_WRITE_MASK | RXE_START_MASK | + RXE_END_MASK, + .length = RXE_BTH_BYTES + RXE_RETH_BYTES, + .offset = { + [RXE_BTH] = 0, + [RXE_RETH] = RXE_BTH_BYTES, + [RXE_PAYLOAD] = RXE_BTH_BYTES + + RXE_RETH_BYTES, + } + }, /* UC */ [IB_OPCODE_UC_SEND_FIRST] = { diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.h b/drivers/infiniband/sw/rxe/rxe_opcode.h index 8f9aaaf260f2..a470e9b0b884 100644 --- a/drivers/infiniband/sw/rxe/rxe_opcode.h +++ b/drivers/infiniband/sw/rxe/rxe_opcode.h @@ -20,6 +20,7 @@ enum rxe_wr_mask { WR_READ_MASK = BIT(3), WR_WRITE_MASK = BIT(4), WR_LOCAL_OP_MASK = BIT(5), + WR_ATOMIC_WRITE_MASK = BIT(7), WR_READ_OR_WRITE_MASK = WR_READ_MASK | WR_WRITE_MASK, WR_WRITE_OR_SEND_MASK = WR_WRITE_MASK | WR_SEND_MASK, @@ -81,6 +82,8 @@ enum rxe_hdr_mask { RXE_LOOPBACK_MASK = BIT(NUM_HDR_TYPES + 12), + RXE_ATOMIC_WRITE_MASK = BIT(NUM_HDR_TYPES + 14), + RXE_READ_OR_ATOMIC_MASK = (RXE_READ_MASK | RXE_ATOMIC_MASK), RXE_WRITE_OR_SEND_MASK = (RXE_WRITE_MASK | RXE_SEND_MASK), RXE_READ_OR_WRITE_MASK = (RXE_READ_MASK | RXE_WRITE_MASK), diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index ca07e37d30a6..be231907ac73 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -135,7 +135,8 @@ static void free_rd_atomic_resources(struct rxe_qp *qp) void free_rd_atomic_resource(struct rxe_qp *qp, struct resp_res *res) { - if (res->type == RXE_ATOMIC_MASK) { + if (res->type == RXE_ATOMIC_MASK || + res->type == RXE_ATOMIC_WRITE_MASK) { kfree_skb(res->resp.skb); } else if (res->type == RXE_READ_MASK) { if (res->read.mr) diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 5eb89052dd66..893c1e88b956 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -237,6 +237,10 @@ static int next_opcode_rc(struct rxe_qp *qp, u32 opcode, int fits) else return fits ? IB_OPCODE_RC_SEND_ONLY_WITH_INVALIDATE : IB_OPCODE_RC_SEND_FIRST; + + case IB_WR_RDMA_ATOMIC_WRITE: + return IB_OPCODE_RC_RDMA_ATOMIC_WRITE; + case IB_WR_REG_MR: case IB_WR_LOCAL_INV: return opcode; @@ -479,6 +483,9 @@ static int finish_packet(struct rxe_qp *qp, struct rxe_send_wqe *wqe, } } + if (pkt->mask & RXE_ATOMIC_WRITE_MASK) + memcpy(payload_addr(pkt), &wqe->wr.wr.rdma.atomic_wr, paylen); + return 0; } @@ -674,13 +681,13 @@ int rxe_requester(void *arg) } mask = rxe_opcode[opcode].mask; - if (unlikely(mask & RXE_READ_OR_ATOMIC_MASK)) { + if (unlikely(mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK))) { if (check_init_depth(qp, wqe)) goto exit; } mtu = get_mtu(qp); - payload = (mask & RXE_WRITE_OR_SEND_MASK) ? wqe->dma.resid : 0; + payload = (mask & (RXE_WRITE_OR_SEND_MASK | RXE_ATOMIC_WRITE_MASK)) ? wqe->dma.resid : 0; if (payload > mtu) { if (qp_type(qp) == IB_QPT_UD) { /* C10-93.1.1: If the total sum of all the buffer lengths specified for a diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index e015860e8c34..b21e9475db0f 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -258,7 +258,7 @@ static enum resp_states check_op_valid(struct rxe_qp *qp, case IB_QPT_RC: if (((pkt->mask & RXE_READ_MASK) && !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_READ)) || - ((pkt->mask & RXE_WRITE_MASK) && + ((pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) && !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_WRITE)) || ((pkt->mask & RXE_ATOMIC_MASK) && !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_ATOMIC))) { @@ -362,7 +362,7 @@ static enum resp_states check_resource(struct rxe_qp *qp, } } - if (pkt->mask & RXE_READ_OR_ATOMIC_MASK) { + if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK)) { /* it is the requesters job to not send * too many read/atomic ops, we just * recycle the responder resource queue @@ -413,7 +413,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, enum resp_states state; int access; - if (pkt->mask & RXE_READ_OR_WRITE_MASK) { + if (pkt->mask & (RXE_READ_OR_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) { if (pkt->mask & RXE_RETH_MASK) { qp->resp.va = reth_va(pkt); qp->resp.offset = 0; @@ -479,7 +479,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, goto err; } - if (pkt->mask & RXE_WRITE_MASK) { + if (pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) { if (resid > mtu) { if (pktlen != mtu || bth_pad(pkt)) { state = RESPST_ERR_LENGTH; @@ -591,6 +591,28 @@ static enum resp_states process_atomic(struct rxe_qp *qp, return ret; } +static enum resp_states process_atomic_write(struct rxe_qp *qp, + struct rxe_pkt_info *pkt) +{ + struct rxe_mr *mr = qp->resp.mr; + + u64 *src = payload_addr(pkt); + + u64 *dst = iova_to_vaddr(mr, qp->resp.va + qp->resp.offset, sizeof(u64)); + + /* check vaddr is 8 bytes aligned. */ + if (!dst || (uintptr_t)dst & 7) + return RESPST_ERR_MISALIGNED_ATOMIC; + + /* Do atomic write after all prior operations have completed */ + smp_store_release(dst, *src); + + /* decrease resp.resid to zero */ + qp->resp.resid -= sizeof(u64); + + return RESPST_NONE; +} + static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, struct rxe_pkt_info *ack, @@ -801,6 +823,10 @@ static enum resp_states execute(struct rxe_qp *qp, struct rxe_pkt_info *pkt) err = process_atomic(qp, pkt); if (err) return err; + } else if (pkt->mask & RXE_ATOMIC_WRITE_MASK) { + err = process_atomic_write(qp, pkt); + if (err) + return err; } else { /* Unreachable */ WARN_ON_ONCE(1); @@ -965,9 +991,12 @@ static int send_resp(struct rxe_qp *qp, struct rxe_pkt_info *pkt, struct sk_buff *skb; struct resp_res *res; + int opcode = pkt->mask & RXE_ATOMIC_MASK ? + IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE : + IB_OPCODE_RC_RDMA_READ_RESPONSE_ONLY; + skb = prepare_ack_packet(qp, pkt, &ack_pkt, - IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE, 0, pkt->psn, - syndrome); + opcode, 0, pkt->psn, syndrome); if (!skb) { rc = -ENOMEM; goto out; @@ -978,7 +1007,7 @@ static int send_resp(struct rxe_qp *qp, struct rxe_pkt_info *pkt, rxe_advance_resp_resource(qp); skb_get(skb); - res->type = RXE_ATOMIC_MASK; + res->type = pkt->mask & (RXE_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK); res->resp.skb = skb; res->first_psn = ack_pkt.psn; res->last_psn = ack_pkt.psn; @@ -1001,7 +1030,7 @@ static enum resp_states acknowledge(struct rxe_qp *qp, if (qp->resp.aeth_syndrome != AETH_ACK_UNLIMITED) send_ack(qp, pkt, qp->resp.aeth_syndrome, pkt->psn); - else if (pkt->mask & RXE_ATOMIC_MASK) + else if (pkt->mask & (RXE_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK)) send_resp(qp, pkt, AETH_ACK_UNLIMITED); else if (bth_ack(pkt)) send_ack(qp, pkt, AETH_ACK_UNLIMITED, pkt->psn); diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h index a9162f25beaf..519ec6b841e7 100644 --- a/include/rdma/ib_pack.h +++ b/include/rdma/ib_pack.h @@ -84,6 +84,7 @@ enum { /* opcode 0x15 is reserved */ IB_OPCODE_SEND_LAST_WITH_INVALIDATE = 0x16, IB_OPCODE_SEND_ONLY_WITH_INVALIDATE = 0x17, + IB_OPCODE_RDMA_ATOMIC_WRITE = 0x1D, /* real constants follow -- see comment about above IB_OPCODE() macro for more details */ @@ -112,6 +113,7 @@ enum { IB_OPCODE(RC, FETCH_ADD), IB_OPCODE(RC, SEND_LAST_WITH_INVALIDATE), IB_OPCODE(RC, SEND_ONLY_WITH_INVALIDATE), + IB_OPCODE(RC, RDMA_ATOMIC_WRITE), /* UC */ IB_OPCODE(UC, SEND_FIRST), diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 69d883f7fb41..abd1c5d3dc66 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -971,6 +971,7 @@ enum ib_wc_opcode { IB_WC_REG_MR, IB_WC_MASKED_COMP_SWAP, IB_WC_MASKED_FETCH_ADD, + IB_WC_RDMA_ATOMIC_WRITE = IB_UVERBS_WC_RDMA_ATOMIC_WRITE, /* * Set value of IB_WC_RECV so consumers can test if a completion is a * receive by testing (opcode & IB_WC_RECV). @@ -1311,6 +1312,7 @@ enum ib_wr_opcode { IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP, IB_WR_MASKED_ATOMIC_FETCH_AND_ADD = IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD, + IB_WR_RDMA_ATOMIC_WRITE = IB_UVERBS_WR_RDMA_ATOMIC_WRITE, /* These are kernel only and can not be issued by userspace */ IB_WR_REG_MR = 0x20, diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h index 7ee73a0652f1..3b0b509fb96f 100644 --- a/include/uapi/rdma/ib_user_verbs.h +++ b/include/uapi/rdma/ib_user_verbs.h @@ -466,6 +466,7 @@ enum ib_uverbs_wc_opcode { IB_UVERBS_WC_BIND_MW = 5, IB_UVERBS_WC_LOCAL_INV = 6, IB_UVERBS_WC_TSO = 7, + IB_UVERBS_WC_RDMA_ATOMIC_WRITE = 9, }; struct ib_uverbs_wc { @@ -784,6 +785,7 @@ enum ib_uverbs_wr_opcode { IB_UVERBS_WR_RDMA_READ_WITH_INV = 11, IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP = 12, IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD = 13, + IB_UVERBS_WR_RDMA_ATOMIC_WRITE = 15, /* Review enum ib_wr_opcode before modifying this */ }; diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index f09c5c9e3dd5..7e02c614d826 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -86,6 +86,7 @@ struct rxe_send_wr { __aligned_u64 remote_addr; __u32 rkey; __u32 reserved; + __aligned_u64 atomic_wr; } rdma; struct { __aligned_u64 remote_addr; -- 2.34.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation 2022-03-11 11:52 ` [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation Xiao Yang @ 2022-03-11 23:56 ` kernel test robot 0 siblings, 0 replies; 15+ messages in thread From: kernel test robot @ 2022-03-11 23:56 UTC (permalink / raw) To: Xiao Yang, linux-rdma Cc: kbuild-all, yanjun.zhu, rpearsonhpe, jgg, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny, Xiao Yang Hi Xiao, Thank you for the patch! Yet something to improve: [auto build test ERROR on rdma/for-next] [also build test ERROR on v5.17-rc7 next-20220310] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/Xiao-Yang/RDMA-rxe-Add-RDMA-Atomic-Write-operation/20220311-195406 base: https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next config: parisc-allyesconfig (https://download.01.org/0day-ci/archive/20220312/202203120740.G02NgRLo-lkp@intel.com/config) compiler: hppa-linux-gcc (GCC) 11.2.0 reproduce (this is a W=1 build): wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # https://github.com/0day-ci/linux/commit/e2d651ea47e6ed9341efeec5b1cb960c5c12f31c git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review Xiao-Yang/RDMA-rxe-Add-RDMA-Atomic-Write-operation/20220311-195406 git checkout e2d651ea47e6ed9341efeec5b1cb960c5c12f31c # save the config file to linux build tree mkdir build_dir COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=parisc SHELL=/bin/bash If you fix the issue, kindly add following tag as appropriate Reported-by: kernel test robot <lkp@intel.com> All errors (new ones prefixed by >>): In file included from <command-line>: In function 'process_atomic_write', inlined from 'execute' at drivers/infiniband/sw/rxe/rxe_resp.c:828:9: >> include/linux/compiler_types.h:346:45: error: call to '__compiletime_assert_463' declared with attribute error: Need native word sized stores/loads for atomicity. 346 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__) | ^ include/linux/compiler_types.h:327:25: note: in definition of macro '__compiletime_assert' 327 | prefix ## suffix(); \ | ^~~~~~ include/linux/compiler_types.h:346:9: note: in expansion of macro '_compiletime_assert' 346 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__) | ^~~~~~~~~~~~~~~~~~~ include/linux/compiler_types.h:349:9: note: in expansion of macro 'compiletime_assert' 349 | compiletime_assert(__native_word(t), \ | ^~~~~~~~~~~~~~~~~~ arch/parisc/include/asm/barrier.h:38:9: note: in expansion of macro 'compiletime_assert_atomic_type' 38 | compiletime_assert_atomic_type(*p); \ | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ include/asm-generic/barrier.h:164:55: note: in expansion of macro '__smp_store_release' 164 | #define smp_store_release(p, v) do { kcsan_release(); __smp_store_release(p, v); } while (0) | ^~~~~~~~~~~~~~~~~~~ drivers/infiniband/sw/rxe/rxe_resp.c:609:9: note: in expansion of macro 'smp_store_release' 609 | smp_store_release(dst, *src); | ^~~~~~~~~~~~~~~~~ vim +/__compiletime_assert_463 +346 include/linux/compiler_types.h eb5c2d4b45e3d2 Will Deacon 2020-07-21 332 eb5c2d4b45e3d2 Will Deacon 2020-07-21 333 #define _compiletime_assert(condition, msg, prefix, suffix) \ eb5c2d4b45e3d2 Will Deacon 2020-07-21 334 __compiletime_assert(condition, msg, prefix, suffix) eb5c2d4b45e3d2 Will Deacon 2020-07-21 335 eb5c2d4b45e3d2 Will Deacon 2020-07-21 336 /** eb5c2d4b45e3d2 Will Deacon 2020-07-21 337 * compiletime_assert - break build and emit msg if condition is false eb5c2d4b45e3d2 Will Deacon 2020-07-21 338 * @condition: a compile-time constant condition to check eb5c2d4b45e3d2 Will Deacon 2020-07-21 339 * @msg: a message to emit if condition is false eb5c2d4b45e3d2 Will Deacon 2020-07-21 340 * eb5c2d4b45e3d2 Will Deacon 2020-07-21 341 * In tradition of POSIX assert, this macro will break the build if the eb5c2d4b45e3d2 Will Deacon 2020-07-21 342 * supplied condition is *false*, emitting the supplied error message if the eb5c2d4b45e3d2 Will Deacon 2020-07-21 343 * compiler has support to do so. eb5c2d4b45e3d2 Will Deacon 2020-07-21 344 */ eb5c2d4b45e3d2 Will Deacon 2020-07-21 345 #define compiletime_assert(condition, msg) \ eb5c2d4b45e3d2 Will Deacon 2020-07-21 @346 _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__) eb5c2d4b45e3d2 Will Deacon 2020-07-21 347 --- 0-DAY CI Kernel Test Service https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org ^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-11 11:52 [PATCH v3 0/3] RDMA/rxe: Add RDMA Atomic Write operation Xiao Yang 2022-03-11 11:52 ` [PATCH v3 1/3] RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res Xiao Yang 2022-03-11 11:52 ` [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation Xiao Yang @ 2022-03-11 11:52 ` Xiao Yang 2022-03-15 18:53 ` Jason Gunthorpe 2 siblings, 1 reply; 15+ messages in thread From: Xiao Yang @ 2022-03-11 11:52 UTC (permalink / raw) To: linux-rdma Cc: yanjun.zhu, rpearsonhpe, jgg, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny, Xiao Yang The attribute shows that rxe device supports RDMA Atomic Write operation. Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com> --- drivers/infiniband/sw/rxe/rxe_param.h | 3 ++- include/rdma/ib_verbs.h | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h index 918270e34a35..6ae6c4079639 100644 --- a/drivers/infiniband/sw/rxe/rxe_param.h +++ b/drivers/infiniband/sw/rxe/rxe_param.h @@ -53,7 +53,8 @@ enum rxe_device_param { | IB_DEVICE_ALLOW_USER_UNREG | IB_DEVICE_MEM_WINDOW | IB_DEVICE_MEM_WINDOW_TYPE_2A - | IB_DEVICE_MEM_WINDOW_TYPE_2B, + | IB_DEVICE_MEM_WINDOW_TYPE_2B + | IB_DEVICE_ATOMIC_WRITE, RXE_MAX_SGE = 32, RXE_MAX_WQE_SIZE = sizeof(struct rxe_send_wqe) + sizeof(struct ib_sge) * RXE_MAX_SGE, diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index abd1c5d3dc66..580b5cacec09 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -291,6 +291,8 @@ enum ib_device_cap_flags { /* The device supports padding incoming writes to cacheline. */ IB_DEVICE_PCI_WRITE_END_PADDING = (1ULL << 36), IB_DEVICE_ALLOW_USER_UNREG = (1ULL << 37), + /* Atomic write attributes */ + IB_DEVICE_ATOMIC_WRITE = (1ULL << 40), }; enum ib_atomic_cap { -- 2.34.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-11 11:52 ` [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device Xiao Yang @ 2022-03-15 18:53 ` Jason Gunthorpe 2022-03-17 5:58 ` yangx.jy 2022-03-21 3:55 ` yangx.jy 0 siblings, 2 replies; 15+ messages in thread From: Jason Gunthorpe @ 2022-03-15 18:53 UTC (permalink / raw) To: Xiao Yang Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny On Fri, Mar 11, 2022 at 07:52:47PM +0800, Xiao Yang wrote: > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h > index abd1c5d3dc66..580b5cacec09 100644 > +++ b/include/rdma/ib_verbs.h > @@ -291,6 +291,8 @@ enum ib_device_cap_flags { > /* The device supports padding incoming writes to cacheline. */ > IB_DEVICE_PCI_WRITE_END_PADDING = (1ULL << 36), > IB_DEVICE_ALLOW_USER_UNREG = (1ULL << 37), > + /* Atomic write attributes */ > + IB_DEVICE_ATOMIC_WRITE = (1ULL << 40), > }; Can you make a patch to clean this up too? The right parts of it need to get moved to the uapi header and it should work similarly to the ib_uverbs_wr_opcode / ib_wr_opcode thing. You'll also need to do something about the 32 bit compatability that kbuild detected - I suppose this can't work on 32 bit platforms? So IS_ENABLED() it off or something? Jason ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-15 18:53 ` Jason Gunthorpe @ 2022-03-17 5:58 ` yangx.jy 2022-03-21 3:55 ` yangx.jy 1 sibling, 0 replies; 15+ messages in thread From: yangx.jy @ 2022-03-17 5:58 UTC (permalink / raw) To: Jason Gunthorpe Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny On 2022/3/16 2:53, Jason Gunthorpe wrote: > Can you make a patch to clean this up too? The right parts of it need > to get moved to the uapi header and it should work similarly to the > ib_uverbs_wr_opcode / ib_wr_opcode thing. > > You'll also need to do something about the 32 bit compatability that > kbuild detected - I suppose this can't work on 32 bit platforms? So > IS_ENABLED() it off or something? Hi Jason, Thanks for your reminder. I will do the cleanup and fix the issue detected by kbuild. Best Regards, Xiao Yang ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-15 18:53 ` Jason Gunthorpe 2022-03-17 5:58 ` yangx.jy @ 2022-03-21 3:55 ` yangx.jy 2022-03-21 15:32 ` Jason Gunthorpe 1 sibling, 1 reply; 15+ messages in thread From: yangx.jy @ 2022-03-21 3:55 UTC (permalink / raw) To: Jason Gunthorpe Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny On 2022/3/16 2:53, Jason Gunthorpe wrote: > You'll also need to do something about the 32 bit compatability that > kbuild detected - I suppose this can't work on 32 bit platforms? So > IS_ENABLED() it off or something? Hi Jason, Is it possible to fix the issue by atomic64_set_release()? If not, we may need to add a check for __native_word(*dst) and return an unsupported error when __native_word(*dst) is false. Best Regards, Xiao Yang ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-21 3:55 ` yangx.jy @ 2022-03-21 15:32 ` Jason Gunthorpe 2022-03-25 11:44 ` yangx.jy 0 siblings, 1 reply; 15+ messages in thread From: Jason Gunthorpe @ 2022-03-21 15:32 UTC (permalink / raw) To: yangx.jy Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny On Mon, Mar 21, 2022 at 03:55:01AM +0000, yangx.jy@fujitsu.com wrote: > On 2022/3/16 2:53, Jason Gunthorpe wrote: > > You'll also need to do something about the 32 bit compatability that > > kbuild detected - I suppose this can't work on 32 bit platforms? So > > IS_ENABLED() it off or something? > Hi Jason, > > Is it possible to fix the issue by atomic64_set_release()? No > If not, we may need to add a check for __native_word(*dst) and return an > unsupported error when __native_word(*dst) is false. The whole feature, including the cap bits should be turned off for 32 bit builds because it cannot possibly work Jason ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-21 15:32 ` Jason Gunthorpe @ 2022-03-25 11:44 ` yangx.jy 2022-03-25 13:22 ` Jason Gunthorpe 0 siblings, 1 reply; 15+ messages in thread From: yangx.jy @ 2022-03-25 11:44 UTC (permalink / raw) To: Jason Gunthorpe Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny On 2022/3/21 23:32, Jason Gunthorpe write: > On Mon, Mar 21, 2022 at 03:55:01AM +0000, yangx.jy@fujitsu.com wrote: >> On 2022/3/16 2:53, Jason Gunthorpe wrote: >>> You'll also need to do something about the 32 bit compatability that >>> kbuild detected - I suppose this can't work on 32 bit platforms? So >>> IS_ENABLED() it off or something? >> Hi Jason, >> >> Is it possible to fix the issue by atomic64_set_release()? > No > >> If not, we may need to add a check for __native_word(*dst) and return an >> unsupported error when __native_word(*dst) is false. > The whole feature, including the cap bits should be turned off for 32 > bit builds because it cannot possibly work Hi Jason, Is it ok to disable the whole atomic write by checking CONFIG_64BIT? Best Regards, Xiao Yang > > Jason ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-25 11:44 ` yangx.jy @ 2022-03-25 13:22 ` Jason Gunthorpe 2022-03-28 10:07 ` yangx.jy 0 siblings, 1 reply; 15+ messages in thread From: Jason Gunthorpe @ 2022-03-25 13:22 UTC (permalink / raw) To: yangx.jy Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny On Fri, Mar 25, 2022 at 11:44:53AM +0000, yangx.jy@fujitsu.com wrote: > On 2022/3/21 23:32, Jason Gunthorpe write: > > On Mon, Mar 21, 2022 at 03:55:01AM +0000, yangx.jy@fujitsu.com wrote: > >> On 2022/3/16 2:53, Jason Gunthorpe wrote: > >>> You'll also need to do something about the 32 bit compatability that > >>> kbuild detected - I suppose this can't work on 32 bit platforms? So > >>> IS_ENABLED() it off or something? > >> Hi Jason, > >> > >> Is it possible to fix the issue by atomic64_set_release()? > > No > > > >> If not, we may need to add a check for __native_word(*dst) and return an > >> unsupported error when __native_word(*dst) is false. > > The whole feature, including the cap bits should be turned off for 32 > > bit builds because it cannot possibly work > > Hi Jason, > > Is it ok to disable the whole atomic write by checking CONFIG_64BIT? It is not great, but there is not another choice I can see.. Jason ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-25 13:22 ` Jason Gunthorpe @ 2022-03-28 10:07 ` yangx.jy 2022-03-28 11:39 ` Jason Gunthorpe 0 siblings, 1 reply; 15+ messages in thread From: yangx.jy @ 2022-03-28 10:07 UTC (permalink / raw) To: Jason Gunthorpe Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny On 2022/3/25 21:22, Jason Gunthorpe wrote: > It is not great, but there is not another choice I can see.. Hi Jason, I plan to only disable the key places by the following change so that user cannot use the atomic write: ----------------------------------------------------------------------------------------------------------- diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index e2d332a5af0a..7f83b5e39ace 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -238,8 +238,10 @@ static int next_opcode_rc(struct rxe_qp *qp, u32 opcode, int fits) return fits ? IB_OPCODE_RC_SEND_ONLY_WITH_INVALIDATE : IB_OPCODE_RC_SEND_FIRST; +#ifdef CONFIG_64BIT case IB_WR_RDMA_ATOMIC_WRITE: return IB_OPCODE_RC_RDMA_ATOMIC_WRITE; +#endif /* CONFIG_64BIT */ case IB_WR_REG_MR: case IB_WR_LOCAL_INV: diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index ddc791b199e2..311ec523fb57 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -592,6 +592,7 @@ static enum resp_states process_atomic(struct rxe_qp *qp, return ret; } +#ifdef CONFIG_64BIT static enum resp_states process_atomic_write(struct rxe_qp *qp, struct rxe_pkt_info *pkt) { @@ -613,6 +614,7 @@ static enum resp_states process_atomic_write(struct rxe_qp *qp, return RESPST_NONE; } +#endif /* CONFIG_64BIT */ static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, @@ -871,10 +873,12 @@ static enum resp_states execute(struct rxe_qp *qp, struct rxe_pkt_info *pkt) err = process_atomic(qp, pkt); if (err) return err; +#ifdef CONFIG_64BIT } else if (pkt->mask & RXE_ATOMIC_WRITE_MASK) { err = process_atomic_write(qp, pkt); if (err) return err; +#endif /* CONFIG_64BIT */ } else { /* Unreachable */ WARN_ON_ONCE(1); ----------------------------------------------------------------------------------------------------------- Best Regards, Xiao Yang > > Jason ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-28 10:07 ` yangx.jy @ 2022-03-28 11:39 ` Jason Gunthorpe 2022-03-29 2:36 ` yangx.jy 0 siblings, 1 reply; 15+ messages in thread From: Jason Gunthorpe @ 2022-03-28 11:39 UTC (permalink / raw) To: yangx.jy Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny On Mon, Mar 28, 2022 at 10:07:26AM +0000, yangx.jy@fujitsu.com wrote: > On 2022/3/25 21:22, Jason Gunthorpe wrote: > > It is not great, but there is not another choice I can see.. > > Hi Jason, > > I plan to only disable the key places by the following change so that > user cannot use the atomic write: Isn't there a cap flag too? Jason ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-28 11:39 ` Jason Gunthorpe @ 2022-03-29 2:36 ` yangx.jy 2022-03-29 2:39 ` yangx.jy 0 siblings, 1 reply; 15+ messages in thread From: yangx.jy @ 2022-03-29 2:36 UTC (permalink / raw) To: Jason Gunthorpe Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny On 2022/3/28 19:39, Jason Gunthorpe wrote: > On Mon, Mar 28, 2022 at 10:07:26AM +0000, yangx.jy@fujitsu.com wrote: >> On 2022/3/25 21:22, Jason Gunthorpe wrote: >>> It is not great, but there is not another choice I can see.. >> Hi Jason, >> >> I plan to only disable the key places by the following change so that >> user cannot use the atomic write: > Isn't there a cap flag too? Hi Jason, I will disable the atomic write cap flag as well, like this: ----------------------------------- diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h index 918270e34a35..88953f9c26e4 100644 --- a/drivers/infiniband/sw/rxe/rxe_param.h +++ b/drivers/infiniband/sw/rxe/rxe_param.h @@ -53,7 +53,12 @@ enum rxe_device_param { | IB_DEVICE_ALLOW_USER_UNREG | IB_DEVICE_MEM_WINDOW | IB_DEVICE_MEM_WINDOW_TYPE_2A +#ifdef CONFIG_64BIT + | IB_DEVICE_MEM_WINDOW_TYPE_2B + | IB_DEVICE_ATOMIC_WRITE, +#else | IB_DEVICE_MEM_WINDOW_TYPE_2B, +#endif /* CONFIG_64BIT */ RXE_MAX_SGE = 32, RXE_MAX_WQE_SIZE = sizeof(struct rxe_send_wqe) + sizeof(struct ib_sge) * RXE_MAX_SGE, ----------------------------------- BTW: I hope we can review and merge my cleanup patchset[1][2] first. So that I will update the third patch[3] based on it. ^_^ [1]: [PATCH v2 1/2] IB/uverbs: Move enum ib_raw_packet_caps to uapi [2]: [PATCH v2 2/2] IB/uverbs: Move part of enum ib_device_cap_flags to uapi [3]: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device Best Regards, Xiao Yang > > Jason ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device 2022-03-29 2:36 ` yangx.jy @ 2022-03-29 2:39 ` yangx.jy 0 siblings, 0 replies; 15+ messages in thread From: yangx.jy @ 2022-03-29 2:39 UTC (permalink / raw) To: Jason Gunthorpe Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian, tomasz.gromadzki, tom, ira.weiny On 2022/3/29 10:36, yangx.jy@fujitsu.com wrote: > On 2022/3/28 19:39, Jason Gunthorpe wrote: >> On Mon, Mar 28, 2022 at 10:07:26AM +0000, yangx.jy@fujitsu.com wrote: >>> On 2022/3/25 21:22, Jason Gunthorpe wrote: >>>> It is not great, but there is not another choice I can see.. >>> Hi Jason, >>> >>> I plan to only disable the key places by the following change so that >>> user cannot use the atomic write: >> Isn't there a cap flag too? > Hi Jason, > > I will disable the atomic write cap flag as well, like this: > > ----------------------------------- > > diff --git a/drivers/infiniband/sw/rxe/rxe_param.h > b/drivers/infiniband/sw/rxe/rxe_param.h > index 918270e34a35..88953f9c26e4 100644 > --- a/drivers/infiniband/sw/rxe/rxe_param.h > +++ b/drivers/infiniband/sw/rxe/rxe_param.h > @@ -53,7 +53,12 @@ enum rxe_device_param { > | IB_DEVICE_ALLOW_USER_UNREG > | IB_DEVICE_MEM_WINDOW > | IB_DEVICE_MEM_WINDOW_TYPE_2A > +#ifdef CONFIG_64BIT > + | IB_DEVICE_MEM_WINDOW_TYPE_2B > + | IB_DEVICE_ATOMIC_WRITE, > +#else > | IB_DEVICE_MEM_WINDOW_TYPE_2B, > +#endif /* CONFIG_64BIT */ > RXE_MAX_SGE = 32, > RXE_MAX_WQE_SIZE = sizeof(struct rxe_send_wqe) + > sizeof(struct ib_sge) * > RXE_MAX_SGE, > > ----------------------------------- > > BTW: > > I hope we can review and merge my cleanup patchset[1][2] first. So that > I will update the third patch[3] based on it. ^_^ > > [1]: [PATCH v2 1/2] IB/uverbs: Move enum ib_raw_packet_caps to uapi > > [2]: [PATCH v2 2/2] IB/uverbs: Move part of enum ib_device_cap_flags to uapi Sorry, the correct version is my v3 cleanup patchset [1]: [PATCH v3 1/2] IB/uverbs: Move enum ib_raw_packet_caps to uapi [2]: [PATCH v3 2/2] IB/uverbs: Move part of enum ib_device_cap_flags to uapi > > [3]: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device > > Best Regards, > > Xiao Yang > >> Jason ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2022-03-29 2:40 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-03-11 11:52 [PATCH v3 0/3] RDMA/rxe: Add RDMA Atomic Write operation Xiao Yang 2022-03-11 11:52 ` [PATCH v3 1/3] RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res Xiao Yang 2022-03-11 11:52 ` [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation Xiao Yang 2022-03-11 23:56 ` kernel test robot 2022-03-11 11:52 ` [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device Xiao Yang 2022-03-15 18:53 ` Jason Gunthorpe 2022-03-17 5:58 ` yangx.jy 2022-03-21 3:55 ` yangx.jy 2022-03-21 15:32 ` Jason Gunthorpe 2022-03-25 11:44 ` yangx.jy 2022-03-25 13:22 ` Jason Gunthorpe 2022-03-28 10:07 ` yangx.jy 2022-03-28 11:39 ` Jason Gunthorpe 2022-03-29 2:36 ` yangx.jy 2022-03-29 2:39 ` yangx.jy
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.