* [PATCH v3 0/3] RDMA/rxe: Add RDMA Atomic Write operation
@ 2022-03-11 11:52 Xiao Yang
2022-03-11 11:52 ` [PATCH v3 1/3] RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res Xiao Yang
` (2 more replies)
0 siblings, 3 replies; 15+ messages in thread
From: Xiao Yang @ 2022-03-11 11:52 UTC (permalink / raw)
To: linux-rdma
Cc: yanjun.zhu, rpearsonhpe, jgg, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny, Xiao Yang
The IB SPEC v1.5[1] defined new RDMA Atomic Write operation. This
patchset makes SoftRoCE support new RDMA Atomic Write on RC service.
I add ibv_wr_rdma_atomic_write() and a rdma_atomic_write example on my
rdma-core repository[2]. You can verify the patchset by building and
running the rdma_atomic_write example.
server:
$ ./rdma_atomic_write_server -s [server_address] -p [port_number]
client:
$ ./rdma_atomic_write_client -s [server_address] -p [port_number]
[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/yangx-jy/rdma-core/tree/new_api
V2->V3:
1) Rebase
2) Add RDMA Atomic Write attribute for rxe device
V1->V2:
1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D
2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value
3) Use smp_store_release() to ensure that all prior operations have completed
Xiao Yang (3):
RDMA/rxe: Rename send_atomic_ack() and atomic member of struct
resp_res
RDMA/rxe: Support RDMA Atomic Write operation
RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++
drivers/infiniband/sw/rxe/rxe_opcode.c | 19 +++++++++
drivers/infiniband/sw/rxe/rxe_opcode.h | 3 ++
drivers/infiniband/sw/rxe/rxe_param.h | 3 +-
drivers/infiniband/sw/rxe/rxe_qp.c | 5 ++-
drivers/infiniband/sw/rxe/rxe_req.c | 11 +++++-
drivers/infiniband/sw/rxe/rxe_resp.c | 55 ++++++++++++++++++++------
drivers/infiniband/sw/rxe/rxe_verbs.h | 2 +-
include/rdma/ib_pack.h | 2 +
include/rdma/ib_verbs.h | 4 ++
include/uapi/rdma/ib_user_verbs.h | 2 +
include/uapi/rdma/rdma_user_rxe.h | 1 +
12 files changed, 92 insertions(+), 19 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v3 1/3] RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res
2022-03-11 11:52 [PATCH v3 0/3] RDMA/rxe: Add RDMA Atomic Write operation Xiao Yang
@ 2022-03-11 11:52 ` Xiao Yang
2022-03-11 11:52 ` [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation Xiao Yang
2022-03-11 11:52 ` [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device Xiao Yang
2 siblings, 0 replies; 15+ messages in thread
From: Xiao Yang @ 2022-03-11 11:52 UTC (permalink / raw)
To: linux-rdma
Cc: yanjun.zhu, rpearsonhpe, jgg, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny, Xiao Yang
send_atomic_ack() and atomic member of struct resp_res will be common
in the future so rename them.
Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
drivers/infiniband/sw/rxe/rxe_qp.c | 2 +-
drivers/infiniband/sw/rxe/rxe_resp.c | 10 +++++-----
drivers/infiniband/sw/rxe/rxe_verbs.h | 2 +-
3 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c
index 5018b9387694..ca07e37d30a6 100644
--- a/drivers/infiniband/sw/rxe/rxe_qp.c
+++ b/drivers/infiniband/sw/rxe/rxe_qp.c
@@ -136,7 +136,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
void free_rd_atomic_resource(struct rxe_qp *qp, struct resp_res *res)
{
if (res->type == RXE_ATOMIC_MASK) {
- kfree_skb(res->atomic.skb);
+ kfree_skb(res->resp.skb);
} else if (res->type == RXE_READ_MASK) {
if (res->read.mr)
rxe_drop_ref(res->read.mr);
diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index e8f435fa6e4d..e015860e8c34 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -957,7 +957,7 @@ static int send_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
return err;
}
-static int send_atomic_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
+static int send_resp(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
u8 syndrome)
{
int rc = 0;
@@ -979,7 +979,7 @@ static int send_atomic_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
skb_get(skb);
res->type = RXE_ATOMIC_MASK;
- res->atomic.skb = skb;
+ res->resp.skb = skb;
res->first_psn = ack_pkt.psn;
res->last_psn = ack_pkt.psn;
res->cur_psn = ack_pkt.psn;
@@ -1002,7 +1002,7 @@ static enum resp_states acknowledge(struct rxe_qp *qp,
if (qp->resp.aeth_syndrome != AETH_ACK_UNLIMITED)
send_ack(qp, pkt, qp->resp.aeth_syndrome, pkt->psn);
else if (pkt->mask & RXE_ATOMIC_MASK)
- send_atomic_ack(qp, pkt, AETH_ACK_UNLIMITED);
+ send_resp(qp, pkt, AETH_ACK_UNLIMITED);
else if (bth_ack(pkt))
send_ack(qp, pkt, AETH_ACK_UNLIMITED, pkt->psn);
@@ -1111,9 +1111,9 @@ static enum resp_states duplicate_request(struct rxe_qp *qp,
/* Find the operation in our list of responder resources. */
res = find_resource(qp, pkt->psn);
if (res) {
- skb_get(res->atomic.skb);
+ skb_get(res->resp.skb);
/* Resend the result. */
- rc = rxe_xmit_packet(qp, pkt, res->atomic.skb);
+ rc = rxe_xmit_packet(qp, pkt, res->resp.skb);
if (rc) {
pr_err("Failed resending result. This flow is not handled - skb ignored\n");
rc = RESPST_CLEANUP;
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h
index e48969e8d4c8..01421286ed07 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.h
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.h
@@ -155,7 +155,7 @@ struct resp_res {
union {
struct {
struct sk_buff *skb;
- } atomic;
+ } resp;
struct {
struct rxe_mr *mr;
u64 va_org;
--
2.34.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation
2022-03-11 11:52 [PATCH v3 0/3] RDMA/rxe: Add RDMA Atomic Write operation Xiao Yang
2022-03-11 11:52 ` [PATCH v3 1/3] RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res Xiao Yang
@ 2022-03-11 11:52 ` Xiao Yang
2022-03-11 23:56 ` kernel test robot
2022-03-11 11:52 ` [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device Xiao Yang
2 siblings, 1 reply; 15+ messages in thread
From: Xiao Yang @ 2022-03-11 11:52 UTC (permalink / raw)
To: linux-rdma
Cc: yanjun.zhu, rpearsonhpe, jgg, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny, Xiao Yang
This patch implements RDMA Atomic Write operation for RC service.
Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
drivers/infiniband/sw/rxe/rxe_comp.c | 4 +++
drivers/infiniband/sw/rxe/rxe_opcode.c | 19 +++++++++++
drivers/infiniband/sw/rxe/rxe_opcode.h | 3 ++
drivers/infiniband/sw/rxe/rxe_qp.c | 3 +-
drivers/infiniband/sw/rxe/rxe_req.c | 11 +++++--
drivers/infiniband/sw/rxe/rxe_resp.c | 45 +++++++++++++++++++++-----
include/rdma/ib_pack.h | 2 ++
include/rdma/ib_verbs.h | 2 ++
include/uapi/rdma/ib_user_verbs.h | 2 ++
include/uapi/rdma/rdma_user_rxe.h | 1 +
10 files changed, 81 insertions(+), 11 deletions(-)
diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c
index f363fe3fa414..6de7ac0b94ed 100644
--- a/drivers/infiniband/sw/rxe/rxe_comp.c
+++ b/drivers/infiniband/sw/rxe/rxe_comp.c
@@ -104,6 +104,7 @@ static enum ib_wc_opcode wr_to_wc_opcode(enum ib_wr_opcode opcode)
case IB_WR_LOCAL_INV: return IB_WC_LOCAL_INV;
case IB_WR_REG_MR: return IB_WC_REG_MR;
case IB_WR_BIND_MW: return IB_WC_BIND_MW;
+ case IB_WR_RDMA_ATOMIC_WRITE: return IB_WC_RDMA_ATOMIC_WRITE;
default:
return 0xff;
@@ -256,6 +257,9 @@ static inline enum comp_state check_ack(struct rxe_qp *qp,
if ((syn & AETH_TYPE_MASK) != AETH_ACK)
return COMPST_ERROR;
+ if (wqe->wr.opcode == IB_WR_RDMA_ATOMIC_WRITE)
+ return COMPST_WRITE_SEND;
+
fallthrough;
/* (IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE doesn't have an AETH)
*/
diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.c b/drivers/infiniband/sw/rxe/rxe_opcode.c
index df596ba7527d..9b1ca0b3618a 100644
--- a/drivers/infiniband/sw/rxe/rxe_opcode.c
+++ b/drivers/infiniband/sw/rxe/rxe_opcode.c
@@ -103,6 +103,12 @@ struct rxe_wr_opcode_info rxe_wr_opcode_info[] = {
[IB_QPT_UC] = WR_LOCAL_OP_MASK,
},
},
+ [IB_WR_RDMA_ATOMIC_WRITE] = {
+ .name = "IB_WR_RDMA_ATOMIC_WRITE",
+ .mask = {
+ [IB_QPT_RC] = WR_ATOMIC_WRITE_MASK,
+ },
+ },
};
struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = {
@@ -380,6 +386,19 @@ struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = {
RXE_IETH_BYTES,
}
},
+ [IB_OPCODE_RC_RDMA_ATOMIC_WRITE] = {
+ .name = "IB_OPCODE_RC_RDMA_ATOMIC_WRITE",
+ .mask = RXE_RETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK |
+ RXE_ATOMIC_WRITE_MASK | RXE_START_MASK |
+ RXE_END_MASK,
+ .length = RXE_BTH_BYTES + RXE_RETH_BYTES,
+ .offset = {
+ [RXE_BTH] = 0,
+ [RXE_RETH] = RXE_BTH_BYTES,
+ [RXE_PAYLOAD] = RXE_BTH_BYTES +
+ RXE_RETH_BYTES,
+ }
+ },
/* UC */
[IB_OPCODE_UC_SEND_FIRST] = {
diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.h b/drivers/infiniband/sw/rxe/rxe_opcode.h
index 8f9aaaf260f2..a470e9b0b884 100644
--- a/drivers/infiniband/sw/rxe/rxe_opcode.h
+++ b/drivers/infiniband/sw/rxe/rxe_opcode.h
@@ -20,6 +20,7 @@ enum rxe_wr_mask {
WR_READ_MASK = BIT(3),
WR_WRITE_MASK = BIT(4),
WR_LOCAL_OP_MASK = BIT(5),
+ WR_ATOMIC_WRITE_MASK = BIT(7),
WR_READ_OR_WRITE_MASK = WR_READ_MASK | WR_WRITE_MASK,
WR_WRITE_OR_SEND_MASK = WR_WRITE_MASK | WR_SEND_MASK,
@@ -81,6 +82,8 @@ enum rxe_hdr_mask {
RXE_LOOPBACK_MASK = BIT(NUM_HDR_TYPES + 12),
+ RXE_ATOMIC_WRITE_MASK = BIT(NUM_HDR_TYPES + 14),
+
RXE_READ_OR_ATOMIC_MASK = (RXE_READ_MASK | RXE_ATOMIC_MASK),
RXE_WRITE_OR_SEND_MASK = (RXE_WRITE_MASK | RXE_SEND_MASK),
RXE_READ_OR_WRITE_MASK = (RXE_READ_MASK | RXE_WRITE_MASK),
diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c
index ca07e37d30a6..be231907ac73 100644
--- a/drivers/infiniband/sw/rxe/rxe_qp.c
+++ b/drivers/infiniband/sw/rxe/rxe_qp.c
@@ -135,7 +135,8 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
void free_rd_atomic_resource(struct rxe_qp *qp, struct resp_res *res)
{
- if (res->type == RXE_ATOMIC_MASK) {
+ if (res->type == RXE_ATOMIC_MASK ||
+ res->type == RXE_ATOMIC_WRITE_MASK) {
kfree_skb(res->resp.skb);
} else if (res->type == RXE_READ_MASK) {
if (res->read.mr)
diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c
index 5eb89052dd66..893c1e88b956 100644
--- a/drivers/infiniband/sw/rxe/rxe_req.c
+++ b/drivers/infiniband/sw/rxe/rxe_req.c
@@ -237,6 +237,10 @@ static int next_opcode_rc(struct rxe_qp *qp, u32 opcode, int fits)
else
return fits ? IB_OPCODE_RC_SEND_ONLY_WITH_INVALIDATE :
IB_OPCODE_RC_SEND_FIRST;
+
+ case IB_WR_RDMA_ATOMIC_WRITE:
+ return IB_OPCODE_RC_RDMA_ATOMIC_WRITE;
+
case IB_WR_REG_MR:
case IB_WR_LOCAL_INV:
return opcode;
@@ -479,6 +483,9 @@ static int finish_packet(struct rxe_qp *qp, struct rxe_send_wqe *wqe,
}
}
+ if (pkt->mask & RXE_ATOMIC_WRITE_MASK)
+ memcpy(payload_addr(pkt), &wqe->wr.wr.rdma.atomic_wr, paylen);
+
return 0;
}
@@ -674,13 +681,13 @@ int rxe_requester(void *arg)
}
mask = rxe_opcode[opcode].mask;
- if (unlikely(mask & RXE_READ_OR_ATOMIC_MASK)) {
+ if (unlikely(mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK))) {
if (check_init_depth(qp, wqe))
goto exit;
}
mtu = get_mtu(qp);
- payload = (mask & RXE_WRITE_OR_SEND_MASK) ? wqe->dma.resid : 0;
+ payload = (mask & (RXE_WRITE_OR_SEND_MASK | RXE_ATOMIC_WRITE_MASK)) ? wqe->dma.resid : 0;
if (payload > mtu) {
if (qp_type(qp) == IB_QPT_UD) {
/* C10-93.1.1: If the total sum of all the buffer lengths specified for a
diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index e015860e8c34..b21e9475db0f 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -258,7 +258,7 @@ static enum resp_states check_op_valid(struct rxe_qp *qp,
case IB_QPT_RC:
if (((pkt->mask & RXE_READ_MASK) &&
!(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_READ)) ||
- ((pkt->mask & RXE_WRITE_MASK) &&
+ ((pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) &&
!(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_WRITE)) ||
((pkt->mask & RXE_ATOMIC_MASK) &&
!(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_ATOMIC))) {
@@ -362,7 +362,7 @@ static enum resp_states check_resource(struct rxe_qp *qp,
}
}
- if (pkt->mask & RXE_READ_OR_ATOMIC_MASK) {
+ if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK)) {
/* it is the requesters job to not send
* too many read/atomic ops, we just
* recycle the responder resource queue
@@ -413,7 +413,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
enum resp_states state;
int access;
- if (pkt->mask & RXE_READ_OR_WRITE_MASK) {
+ if (pkt->mask & (RXE_READ_OR_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) {
if (pkt->mask & RXE_RETH_MASK) {
qp->resp.va = reth_va(pkt);
qp->resp.offset = 0;
@@ -479,7 +479,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
goto err;
}
- if (pkt->mask & RXE_WRITE_MASK) {
+ if (pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) {
if (resid > mtu) {
if (pktlen != mtu || bth_pad(pkt)) {
state = RESPST_ERR_LENGTH;
@@ -591,6 +591,28 @@ static enum resp_states process_atomic(struct rxe_qp *qp,
return ret;
}
+static enum resp_states process_atomic_write(struct rxe_qp *qp,
+ struct rxe_pkt_info *pkt)
+{
+ struct rxe_mr *mr = qp->resp.mr;
+
+ u64 *src = payload_addr(pkt);
+
+ u64 *dst = iova_to_vaddr(mr, qp->resp.va + qp->resp.offset, sizeof(u64));
+
+ /* check vaddr is 8 bytes aligned. */
+ if (!dst || (uintptr_t)dst & 7)
+ return RESPST_ERR_MISALIGNED_ATOMIC;
+
+ /* Do atomic write after all prior operations have completed */
+ smp_store_release(dst, *src);
+
+ /* decrease resp.resid to zero */
+ qp->resp.resid -= sizeof(u64);
+
+ return RESPST_NONE;
+}
+
static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp,
struct rxe_pkt_info *pkt,
struct rxe_pkt_info *ack,
@@ -801,6 +823,10 @@ static enum resp_states execute(struct rxe_qp *qp, struct rxe_pkt_info *pkt)
err = process_atomic(qp, pkt);
if (err)
return err;
+ } else if (pkt->mask & RXE_ATOMIC_WRITE_MASK) {
+ err = process_atomic_write(qp, pkt);
+ if (err)
+ return err;
} else {
/* Unreachable */
WARN_ON_ONCE(1);
@@ -965,9 +991,12 @@ static int send_resp(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
struct sk_buff *skb;
struct resp_res *res;
+ int opcode = pkt->mask & RXE_ATOMIC_MASK ?
+ IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE :
+ IB_OPCODE_RC_RDMA_READ_RESPONSE_ONLY;
+
skb = prepare_ack_packet(qp, pkt, &ack_pkt,
- IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE, 0, pkt->psn,
- syndrome);
+ opcode, 0, pkt->psn, syndrome);
if (!skb) {
rc = -ENOMEM;
goto out;
@@ -978,7 +1007,7 @@ static int send_resp(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
rxe_advance_resp_resource(qp);
skb_get(skb);
- res->type = RXE_ATOMIC_MASK;
+ res->type = pkt->mask & (RXE_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK);
res->resp.skb = skb;
res->first_psn = ack_pkt.psn;
res->last_psn = ack_pkt.psn;
@@ -1001,7 +1030,7 @@ static enum resp_states acknowledge(struct rxe_qp *qp,
if (qp->resp.aeth_syndrome != AETH_ACK_UNLIMITED)
send_ack(qp, pkt, qp->resp.aeth_syndrome, pkt->psn);
- else if (pkt->mask & RXE_ATOMIC_MASK)
+ else if (pkt->mask & (RXE_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK))
send_resp(qp, pkt, AETH_ACK_UNLIMITED);
else if (bth_ack(pkt))
send_ack(qp, pkt, AETH_ACK_UNLIMITED, pkt->psn);
diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h
index a9162f25beaf..519ec6b841e7 100644
--- a/include/rdma/ib_pack.h
+++ b/include/rdma/ib_pack.h
@@ -84,6 +84,7 @@ enum {
/* opcode 0x15 is reserved */
IB_OPCODE_SEND_LAST_WITH_INVALIDATE = 0x16,
IB_OPCODE_SEND_ONLY_WITH_INVALIDATE = 0x17,
+ IB_OPCODE_RDMA_ATOMIC_WRITE = 0x1D,
/* real constants follow -- see comment about above IB_OPCODE()
macro for more details */
@@ -112,6 +113,7 @@ enum {
IB_OPCODE(RC, FETCH_ADD),
IB_OPCODE(RC, SEND_LAST_WITH_INVALIDATE),
IB_OPCODE(RC, SEND_ONLY_WITH_INVALIDATE),
+ IB_OPCODE(RC, RDMA_ATOMIC_WRITE),
/* UC */
IB_OPCODE(UC, SEND_FIRST),
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 69d883f7fb41..abd1c5d3dc66 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -971,6 +971,7 @@ enum ib_wc_opcode {
IB_WC_REG_MR,
IB_WC_MASKED_COMP_SWAP,
IB_WC_MASKED_FETCH_ADD,
+ IB_WC_RDMA_ATOMIC_WRITE = IB_UVERBS_WC_RDMA_ATOMIC_WRITE,
/*
* Set value of IB_WC_RECV so consumers can test if a completion is a
* receive by testing (opcode & IB_WC_RECV).
@@ -1311,6 +1312,7 @@ enum ib_wr_opcode {
IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP,
IB_WR_MASKED_ATOMIC_FETCH_AND_ADD =
IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD,
+ IB_WR_RDMA_ATOMIC_WRITE = IB_UVERBS_WR_RDMA_ATOMIC_WRITE,
/* These are kernel only and can not be issued by userspace */
IB_WR_REG_MR = 0x20,
diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h
index 7ee73a0652f1..3b0b509fb96f 100644
--- a/include/uapi/rdma/ib_user_verbs.h
+++ b/include/uapi/rdma/ib_user_verbs.h
@@ -466,6 +466,7 @@ enum ib_uverbs_wc_opcode {
IB_UVERBS_WC_BIND_MW = 5,
IB_UVERBS_WC_LOCAL_INV = 6,
IB_UVERBS_WC_TSO = 7,
+ IB_UVERBS_WC_RDMA_ATOMIC_WRITE = 9,
};
struct ib_uverbs_wc {
@@ -784,6 +785,7 @@ enum ib_uverbs_wr_opcode {
IB_UVERBS_WR_RDMA_READ_WITH_INV = 11,
IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP = 12,
IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD = 13,
+ IB_UVERBS_WR_RDMA_ATOMIC_WRITE = 15,
/* Review enum ib_wr_opcode before modifying this */
};
diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h
index f09c5c9e3dd5..7e02c614d826 100644
--- a/include/uapi/rdma/rdma_user_rxe.h
+++ b/include/uapi/rdma/rdma_user_rxe.h
@@ -86,6 +86,7 @@ struct rxe_send_wr {
__aligned_u64 remote_addr;
__u32 rkey;
__u32 reserved;
+ __aligned_u64 atomic_wr;
} rdma;
struct {
__aligned_u64 remote_addr;
--
2.34.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-11 11:52 [PATCH v3 0/3] RDMA/rxe: Add RDMA Atomic Write operation Xiao Yang
2022-03-11 11:52 ` [PATCH v3 1/3] RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res Xiao Yang
2022-03-11 11:52 ` [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation Xiao Yang
@ 2022-03-11 11:52 ` Xiao Yang
2022-03-15 18:53 ` Jason Gunthorpe
2 siblings, 1 reply; 15+ messages in thread
From: Xiao Yang @ 2022-03-11 11:52 UTC (permalink / raw)
To: linux-rdma
Cc: yanjun.zhu, rpearsonhpe, jgg, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny, Xiao Yang
The attribute shows that rxe device supports RDMA Atomic Write operation.
Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
drivers/infiniband/sw/rxe/rxe_param.h | 3 ++-
include/rdma/ib_verbs.h | 2 ++
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h
index 918270e34a35..6ae6c4079639 100644
--- a/drivers/infiniband/sw/rxe/rxe_param.h
+++ b/drivers/infiniband/sw/rxe/rxe_param.h
@@ -53,7 +53,8 @@ enum rxe_device_param {
| IB_DEVICE_ALLOW_USER_UNREG
| IB_DEVICE_MEM_WINDOW
| IB_DEVICE_MEM_WINDOW_TYPE_2A
- | IB_DEVICE_MEM_WINDOW_TYPE_2B,
+ | IB_DEVICE_MEM_WINDOW_TYPE_2B
+ | IB_DEVICE_ATOMIC_WRITE,
RXE_MAX_SGE = 32,
RXE_MAX_WQE_SIZE = sizeof(struct rxe_send_wqe) +
sizeof(struct ib_sge) * RXE_MAX_SGE,
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index abd1c5d3dc66..580b5cacec09 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -291,6 +291,8 @@ enum ib_device_cap_flags {
/* The device supports padding incoming writes to cacheline. */
IB_DEVICE_PCI_WRITE_END_PADDING = (1ULL << 36),
IB_DEVICE_ALLOW_USER_UNREG = (1ULL << 37),
+ /* Atomic write attributes */
+ IB_DEVICE_ATOMIC_WRITE = (1ULL << 40),
};
enum ib_atomic_cap {
--
2.34.1
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation
2022-03-11 11:52 ` [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation Xiao Yang
@ 2022-03-11 23:56 ` kernel test robot
0 siblings, 0 replies; 15+ messages in thread
From: kernel test robot @ 2022-03-11 23:56 UTC (permalink / raw)
To: Xiao Yang, linux-rdma
Cc: kbuild-all, yanjun.zhu, rpearsonhpe, jgg, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny, Xiao Yang
Hi Xiao,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on rdma/for-next]
[also build test ERROR on v5.17-rc7 next-20220310]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Xiao-Yang/RDMA-rxe-Add-RDMA-Atomic-Write-operation/20220311-195406
base: https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next
config: parisc-allyesconfig (https://download.01.org/0day-ci/archive/20220312/202203120740.G02NgRLo-lkp@intel.com/config)
compiler: hppa-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/e2d651ea47e6ed9341efeec5b1cb960c5c12f31c
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Xiao-Yang/RDMA-rxe-Add-RDMA-Atomic-Write-operation/20220311-195406
git checkout e2d651ea47e6ed9341efeec5b1cb960c5c12f31c
# save the config file to linux build tree
mkdir build_dir
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=parisc SHELL=/bin/bash
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
In file included from <command-line>:
In function 'process_atomic_write',
inlined from 'execute' at drivers/infiniband/sw/rxe/rxe_resp.c:828:9:
>> include/linux/compiler_types.h:346:45: error: call to '__compiletime_assert_463' declared with attribute error: Need native word sized stores/loads for atomicity.
346 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^
include/linux/compiler_types.h:327:25: note: in definition of macro '__compiletime_assert'
327 | prefix ## suffix(); \
| ^~~~~~
include/linux/compiler_types.h:346:9: note: in expansion of macro '_compiletime_assert'
346 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/linux/compiler_types.h:349:9: note: in expansion of macro 'compiletime_assert'
349 | compiletime_assert(__native_word(t), \
| ^~~~~~~~~~~~~~~~~~
arch/parisc/include/asm/barrier.h:38:9: note: in expansion of macro 'compiletime_assert_atomic_type'
38 | compiletime_assert_atomic_type(*p); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/asm-generic/barrier.h:164:55: note: in expansion of macro '__smp_store_release'
164 | #define smp_store_release(p, v) do { kcsan_release(); __smp_store_release(p, v); } while (0)
| ^~~~~~~~~~~~~~~~~~~
drivers/infiniband/sw/rxe/rxe_resp.c:609:9: note: in expansion of macro 'smp_store_release'
609 | smp_store_release(dst, *src);
| ^~~~~~~~~~~~~~~~~
vim +/__compiletime_assert_463 +346 include/linux/compiler_types.h
eb5c2d4b45e3d2 Will Deacon 2020-07-21 332
eb5c2d4b45e3d2 Will Deacon 2020-07-21 333 #define _compiletime_assert(condition, msg, prefix, suffix) \
eb5c2d4b45e3d2 Will Deacon 2020-07-21 334 __compiletime_assert(condition, msg, prefix, suffix)
eb5c2d4b45e3d2 Will Deacon 2020-07-21 335
eb5c2d4b45e3d2 Will Deacon 2020-07-21 336 /**
eb5c2d4b45e3d2 Will Deacon 2020-07-21 337 * compiletime_assert - break build and emit msg if condition is false
eb5c2d4b45e3d2 Will Deacon 2020-07-21 338 * @condition: a compile-time constant condition to check
eb5c2d4b45e3d2 Will Deacon 2020-07-21 339 * @msg: a message to emit if condition is false
eb5c2d4b45e3d2 Will Deacon 2020-07-21 340 *
eb5c2d4b45e3d2 Will Deacon 2020-07-21 341 * In tradition of POSIX assert, this macro will break the build if the
eb5c2d4b45e3d2 Will Deacon 2020-07-21 342 * supplied condition is *false*, emitting the supplied error message if the
eb5c2d4b45e3d2 Will Deacon 2020-07-21 343 * compiler has support to do so.
eb5c2d4b45e3d2 Will Deacon 2020-07-21 344 */
eb5c2d4b45e3d2 Will Deacon 2020-07-21 345 #define compiletime_assert(condition, msg) \
eb5c2d4b45e3d2 Will Deacon 2020-07-21 @346 _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
eb5c2d4b45e3d2 Will Deacon 2020-07-21 347
---
0-DAY CI Kernel Test Service
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-11 11:52 ` [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device Xiao Yang
@ 2022-03-15 18:53 ` Jason Gunthorpe
2022-03-17 5:58 ` yangx.jy
2022-03-21 3:55 ` yangx.jy
0 siblings, 2 replies; 15+ messages in thread
From: Jason Gunthorpe @ 2022-03-15 18:53 UTC (permalink / raw)
To: Xiao Yang
Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny
On Fri, Mar 11, 2022 at 07:52:47PM +0800, Xiao Yang wrote:
> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index abd1c5d3dc66..580b5cacec09 100644
> +++ b/include/rdma/ib_verbs.h
> @@ -291,6 +291,8 @@ enum ib_device_cap_flags {
> /* The device supports padding incoming writes to cacheline. */
> IB_DEVICE_PCI_WRITE_END_PADDING = (1ULL << 36),
> IB_DEVICE_ALLOW_USER_UNREG = (1ULL << 37),
> + /* Atomic write attributes */
> + IB_DEVICE_ATOMIC_WRITE = (1ULL << 40),
> };
Can you make a patch to clean this up too? The right parts of it need
to get moved to the uapi header and it should work similarly to the
ib_uverbs_wr_opcode / ib_wr_opcode thing.
You'll also need to do something about the 32 bit compatability that
kbuild detected - I suppose this can't work on 32 bit platforms? So
IS_ENABLED() it off or something?
Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-15 18:53 ` Jason Gunthorpe
@ 2022-03-17 5:58 ` yangx.jy
2022-03-21 3:55 ` yangx.jy
1 sibling, 0 replies; 15+ messages in thread
From: yangx.jy @ 2022-03-17 5:58 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny
On 2022/3/16 2:53, Jason Gunthorpe wrote:
> Can you make a patch to clean this up too? The right parts of it need
> to get moved to the uapi header and it should work similarly to the
> ib_uverbs_wr_opcode / ib_wr_opcode thing.
>
> You'll also need to do something about the 32 bit compatability that
> kbuild detected - I suppose this can't work on 32 bit platforms? So
> IS_ENABLED() it off or something?
Hi Jason,
Thanks for your reminder.
I will do the cleanup and fix the issue detected by kbuild.
Best Regards,
Xiao Yang
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-15 18:53 ` Jason Gunthorpe
2022-03-17 5:58 ` yangx.jy
@ 2022-03-21 3:55 ` yangx.jy
2022-03-21 15:32 ` Jason Gunthorpe
1 sibling, 1 reply; 15+ messages in thread
From: yangx.jy @ 2022-03-21 3:55 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny
On 2022/3/16 2:53, Jason Gunthorpe wrote:
> You'll also need to do something about the 32 bit compatability that
> kbuild detected - I suppose this can't work on 32 bit platforms? So
> IS_ENABLED() it off or something?
Hi Jason,
Is it possible to fix the issue by atomic64_set_release()?
If not, we may need to add a check for __native_word(*dst) and return an
unsupported error when __native_word(*dst) is false.
Best Regards,
Xiao Yang
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-21 3:55 ` yangx.jy
@ 2022-03-21 15:32 ` Jason Gunthorpe
2022-03-25 11:44 ` yangx.jy
0 siblings, 1 reply; 15+ messages in thread
From: Jason Gunthorpe @ 2022-03-21 15:32 UTC (permalink / raw)
To: yangx.jy
Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny
On Mon, Mar 21, 2022 at 03:55:01AM +0000, yangx.jy@fujitsu.com wrote:
> On 2022/3/16 2:53, Jason Gunthorpe wrote:
> > You'll also need to do something about the 32 bit compatability that
> > kbuild detected - I suppose this can't work on 32 bit platforms? So
> > IS_ENABLED() it off or something?
> Hi Jason,
>
> Is it possible to fix the issue by atomic64_set_release()?
No
> If not, we may need to add a check for __native_word(*dst) and return an
> unsupported error when __native_word(*dst) is false.
The whole feature, including the cap bits should be turned off for 32
bit builds because it cannot possibly work
Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-21 15:32 ` Jason Gunthorpe
@ 2022-03-25 11:44 ` yangx.jy
2022-03-25 13:22 ` Jason Gunthorpe
0 siblings, 1 reply; 15+ messages in thread
From: yangx.jy @ 2022-03-25 11:44 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny
On 2022/3/21 23:32, Jason Gunthorpe write:
> On Mon, Mar 21, 2022 at 03:55:01AM +0000, yangx.jy@fujitsu.com wrote:
>> On 2022/3/16 2:53, Jason Gunthorpe wrote:
>>> You'll also need to do something about the 32 bit compatability that
>>> kbuild detected - I suppose this can't work on 32 bit platforms? So
>>> IS_ENABLED() it off or something?
>> Hi Jason,
>>
>> Is it possible to fix the issue by atomic64_set_release()?
> No
>
>> If not, we may need to add a check for __native_word(*dst) and return an
>> unsupported error when __native_word(*dst) is false.
> The whole feature, including the cap bits should be turned off for 32
> bit builds because it cannot possibly work
Hi Jason,
Is it ok to disable the whole atomic write by checking CONFIG_64BIT?
Best Regards,
Xiao Yang
>
> Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-25 11:44 ` yangx.jy
@ 2022-03-25 13:22 ` Jason Gunthorpe
2022-03-28 10:07 ` yangx.jy
0 siblings, 1 reply; 15+ messages in thread
From: Jason Gunthorpe @ 2022-03-25 13:22 UTC (permalink / raw)
To: yangx.jy
Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny
On Fri, Mar 25, 2022 at 11:44:53AM +0000, yangx.jy@fujitsu.com wrote:
> On 2022/3/21 23:32, Jason Gunthorpe write:
> > On Mon, Mar 21, 2022 at 03:55:01AM +0000, yangx.jy@fujitsu.com wrote:
> >> On 2022/3/16 2:53, Jason Gunthorpe wrote:
> >>> You'll also need to do something about the 32 bit compatability that
> >>> kbuild detected - I suppose this can't work on 32 bit platforms? So
> >>> IS_ENABLED() it off or something?
> >> Hi Jason,
> >>
> >> Is it possible to fix the issue by atomic64_set_release()?
> > No
> >
> >> If not, we may need to add a check for __native_word(*dst) and return an
> >> unsupported error when __native_word(*dst) is false.
> > The whole feature, including the cap bits should be turned off for 32
> > bit builds because it cannot possibly work
>
> Hi Jason,
>
> Is it ok to disable the whole atomic write by checking CONFIG_64BIT?
It is not great, but there is not another choice I can see..
Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-25 13:22 ` Jason Gunthorpe
@ 2022-03-28 10:07 ` yangx.jy
2022-03-28 11:39 ` Jason Gunthorpe
0 siblings, 1 reply; 15+ messages in thread
From: yangx.jy @ 2022-03-28 10:07 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny
On 2022/3/25 21:22, Jason Gunthorpe wrote:
> It is not great, but there is not another choice I can see..
Hi Jason,
I plan to only disable the key places by the following change so that
user cannot use the atomic write:
-----------------------------------------------------------------------------------------------------------
diff --git a/drivers/infiniband/sw/rxe/rxe_req.c
b/drivers/infiniband/sw/rxe/rxe_req.c
index e2d332a5af0a..7f83b5e39ace 100644
--- a/drivers/infiniband/sw/rxe/rxe_req.c
+++ b/drivers/infiniband/sw/rxe/rxe_req.c
@@ -238,8 +238,10 @@ static int next_opcode_rc(struct rxe_qp *qp, u32
opcode, int fits)
return fits ?
IB_OPCODE_RC_SEND_ONLY_WITH_INVALIDATE :
IB_OPCODE_RC_SEND_FIRST;
+#ifdef CONFIG_64BIT
case IB_WR_RDMA_ATOMIC_WRITE:
return IB_OPCODE_RC_RDMA_ATOMIC_WRITE;
+#endif /* CONFIG_64BIT */
case IB_WR_REG_MR:
case IB_WR_LOCAL_INV:
diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c
b/drivers/infiniband/sw/rxe/rxe_resp.c
index ddc791b199e2..311ec523fb57 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -592,6 +592,7 @@ static enum resp_states process_atomic(struct rxe_qp
*qp,
return ret;
}
+#ifdef CONFIG_64BIT
static enum resp_states process_atomic_write(struct rxe_qp *qp,
struct rxe_pkt_info *pkt)
{
@@ -613,6 +614,7 @@ static enum resp_states process_atomic_write(struct
rxe_qp *qp,
return RESPST_NONE;
}
+#endif /* CONFIG_64BIT */
static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp,
struct rxe_pkt_info *pkt,
@@ -871,10 +873,12 @@ static enum resp_states execute(struct rxe_qp *qp,
struct rxe_pkt_info *pkt)
err = process_atomic(qp, pkt);
if (err)
return err;
+#ifdef CONFIG_64BIT
} else if (pkt->mask & RXE_ATOMIC_WRITE_MASK) {
err = process_atomic_write(qp, pkt);
if (err)
return err;
+#endif /* CONFIG_64BIT */
} else {
/* Unreachable */
WARN_ON_ONCE(1);
-----------------------------------------------------------------------------------------------------------
Best Regards,
Xiao Yang
>
> Jason
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-28 10:07 ` yangx.jy
@ 2022-03-28 11:39 ` Jason Gunthorpe
2022-03-29 2:36 ` yangx.jy
0 siblings, 1 reply; 15+ messages in thread
From: Jason Gunthorpe @ 2022-03-28 11:39 UTC (permalink / raw)
To: yangx.jy
Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny
On Mon, Mar 28, 2022 at 10:07:26AM +0000, yangx.jy@fujitsu.com wrote:
> On 2022/3/25 21:22, Jason Gunthorpe wrote:
> > It is not great, but there is not another choice I can see..
>
> Hi Jason,
>
> I plan to only disable the key places by the following change so that
> user cannot use the atomic write:
Isn't there a cap flag too?
Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-28 11:39 ` Jason Gunthorpe
@ 2022-03-29 2:36 ` yangx.jy
2022-03-29 2:39 ` yangx.jy
0 siblings, 1 reply; 15+ messages in thread
From: yangx.jy @ 2022-03-29 2:36 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny
On 2022/3/28 19:39, Jason Gunthorpe wrote:
> On Mon, Mar 28, 2022 at 10:07:26AM +0000, yangx.jy@fujitsu.com wrote:
>> On 2022/3/25 21:22, Jason Gunthorpe wrote:
>>> It is not great, but there is not another choice I can see..
>> Hi Jason,
>>
>> I plan to only disable the key places by the following change so that
>> user cannot use the atomic write:
> Isn't there a cap flag too?
Hi Jason,
I will disable the atomic write cap flag as well, like this:
-----------------------------------
diff --git a/drivers/infiniband/sw/rxe/rxe_param.h
b/drivers/infiniband/sw/rxe/rxe_param.h
index 918270e34a35..88953f9c26e4 100644
--- a/drivers/infiniband/sw/rxe/rxe_param.h
+++ b/drivers/infiniband/sw/rxe/rxe_param.h
@@ -53,7 +53,12 @@ enum rxe_device_param {
| IB_DEVICE_ALLOW_USER_UNREG
| IB_DEVICE_MEM_WINDOW
| IB_DEVICE_MEM_WINDOW_TYPE_2A
+#ifdef CONFIG_64BIT
+ | IB_DEVICE_MEM_WINDOW_TYPE_2B
+ | IB_DEVICE_ATOMIC_WRITE,
+#else
| IB_DEVICE_MEM_WINDOW_TYPE_2B,
+#endif /* CONFIG_64BIT */
RXE_MAX_SGE = 32,
RXE_MAX_WQE_SIZE = sizeof(struct rxe_send_wqe) +
sizeof(struct ib_sge) *
RXE_MAX_SGE,
-----------------------------------
BTW:
I hope we can review and merge my cleanup patchset[1][2] first. So that
I will update the third patch[3] based on it. ^_^
[1]: [PATCH v2 1/2] IB/uverbs: Move enum ib_raw_packet_caps to uapi
[2]: [PATCH v2 2/2] IB/uverbs: Move part of enum ib_device_cap_flags to uapi
[3]: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
Best Regards,
Xiao Yang
>
> Jason
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
2022-03-29 2:36 ` yangx.jy
@ 2022-03-29 2:39 ` yangx.jy
0 siblings, 0 replies; 15+ messages in thread
From: yangx.jy @ 2022-03-29 2:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: linux-rdma, yanjun.zhu, rpearsonhpe, y-goto, lizhijian,
tomasz.gromadzki, tom, ira.weiny
On 2022/3/29 10:36, yangx.jy@fujitsu.com wrote:
> On 2022/3/28 19:39, Jason Gunthorpe wrote:
>> On Mon, Mar 28, 2022 at 10:07:26AM +0000, yangx.jy@fujitsu.com wrote:
>>> On 2022/3/25 21:22, Jason Gunthorpe wrote:
>>>> It is not great, but there is not another choice I can see..
>>> Hi Jason,
>>>
>>> I plan to only disable the key places by the following change so that
>>> user cannot use the atomic write:
>> Isn't there a cap flag too?
> Hi Jason,
>
> I will disable the atomic write cap flag as well, like this:
>
> -----------------------------------
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_param.h
> b/drivers/infiniband/sw/rxe/rxe_param.h
> index 918270e34a35..88953f9c26e4 100644
> --- a/drivers/infiniband/sw/rxe/rxe_param.h
> +++ b/drivers/infiniband/sw/rxe/rxe_param.h
> @@ -53,7 +53,12 @@ enum rxe_device_param {
> | IB_DEVICE_ALLOW_USER_UNREG
> | IB_DEVICE_MEM_WINDOW
> | IB_DEVICE_MEM_WINDOW_TYPE_2A
> +#ifdef CONFIG_64BIT
> + | IB_DEVICE_MEM_WINDOW_TYPE_2B
> + | IB_DEVICE_ATOMIC_WRITE,
> +#else
> | IB_DEVICE_MEM_WINDOW_TYPE_2B,
> +#endif /* CONFIG_64BIT */
> RXE_MAX_SGE = 32,
> RXE_MAX_WQE_SIZE = sizeof(struct rxe_send_wqe) +
> sizeof(struct ib_sge) *
> RXE_MAX_SGE,
>
> -----------------------------------
>
> BTW:
>
> I hope we can review and merge my cleanup patchset[1][2] first. So that
> I will update the third patch[3] based on it. ^_^
>
> [1]: [PATCH v2 1/2] IB/uverbs: Move enum ib_raw_packet_caps to uapi
>
> [2]: [PATCH v2 2/2] IB/uverbs: Move part of enum ib_device_cap_flags to uapi
Sorry, the correct version is my v3 cleanup patchset
[1]: [PATCH v3 1/2] IB/uverbs: Move enum ib_raw_packet_caps to uapi
[2]: [PATCH v3 2/2] IB/uverbs: Move part of enum ib_device_cap_flags to uapi
>
> [3]: [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device
>
> Best Regards,
>
> Xiao Yang
>
>> Jason
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2022-03-29 2:40 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-11 11:52 [PATCH v3 0/3] RDMA/rxe: Add RDMA Atomic Write operation Xiao Yang
2022-03-11 11:52 ` [PATCH v3 1/3] RDMA/rxe: Rename send_atomic_ack() and atomic member of struct resp_res Xiao Yang
2022-03-11 11:52 ` [PATCH v3 2/3] RDMA/rxe: Support RDMA Atomic Write operation Xiao Yang
2022-03-11 23:56 ` kernel test robot
2022-03-11 11:52 ` [PATCH v3 3/3] RDMA/rxe: Add RDMA Atomic Write attribute for rxe device Xiao Yang
2022-03-15 18:53 ` Jason Gunthorpe
2022-03-17 5:58 ` yangx.jy
2022-03-21 3:55 ` yangx.jy
2022-03-21 15:32 ` Jason Gunthorpe
2022-03-25 11:44 ` yangx.jy
2022-03-25 13:22 ` Jason Gunthorpe
2022-03-28 10:07 ` yangx.jy
2022-03-28 11:39 ` Jason Gunthorpe
2022-03-29 2:36 ` yangx.jy
2022-03-29 2:39 ` yangx.jy
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).