[PATCH v6 0/8] RDMA/rxe: Add atomic write operation

[PATCH v6 0/8] RDMA/rxe: Add atomic write operation

[yangx.jy]

The IB SPEC v1.5[1] defined new atomic write operation. This patchset
makes SoftRoCE support new atomic write on RC service.

On my rdma-core repository[2], I have introduced atomic write API
for libibverbs and Pyverbs. I also have provided a rdma_atomic_write
example and test_qp_ex_rc_atomic_write python test to verify
the patchset.

The steps to run the rdma_atomic_write example:
server:
$ ./rdma_atomic_write_server -s [server_address] -p [port_number]
client:
$ ./rdma_atomic_write_client -s [server_address] -p [port_number]

The steps to run test_qp_ex_rc_atomic_write test:
run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write
test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.008s

OK

[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point

v5->v6:
1) Rebase on current for-next
2) Split the implementation of atomic write into 7 patches
3) Replace all "RDMA Atomic Write" with "atomic write"
4) Save 8-byte value in struct rxe_dma_info instead
5) Remove the print in atomic_write_reply()

v4->v5:
1) Rebase on current wip/jgg-for-next
2) Rewrite the implementation on responder

v3->v4:
1) Rebase on current wip/jgg-for-next
2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write
3) Replace 64-bit value with 8-byte array for atomic write

V2->V3:
1) Rebase
2) Add RDMA Atomic Write attribute for rxe device

V1->V2:
1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D
2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value
3) Use smp_store_release() to ensure that all prior operations have completed

Xiao Yang (8):
  RDMA: Extend RDMA user ABI to support atomic write
  RDMA: Extend RDMA kernel ABI to support atomic write
  RDMA/rxe: Extend rxe user ABI to support atomic write
  RDMA/rxe: Extend rxe packet format to support atomic write
  RDMA/rxe: Make requester support atomic write on RC service
  RDMA/rxe: Make responder support atomic write on RC service
  RDMA/rxe: Implement atomic write completion
  RDMA/rxe: Enable atomic write capability for rxe device

 drivers/infiniband/sw/rxe/rxe_comp.c   |  4 ++
 drivers/infiniband/sw/rxe/rxe_opcode.c | 18 ++++++
 drivers/infiniband/sw/rxe/rxe_opcode.h |  3 +
 drivers/infiniband/sw/rxe/rxe_param.h  |  5 ++
 drivers/infiniband/sw/rxe/rxe_req.c    | 15 ++++-
 drivers/infiniband/sw/rxe/rxe_resp.c   | 84 ++++++++++++++++++++++++--
 include/rdma/ib_pack.h                 |  2 +
 include/rdma/ib_verbs.h                |  3 +
 include/uapi/rdma/ib_user_verbs.h      |  4 ++
 include/uapi/rdma/rdma_user_rxe.h      |  1 +
 10 files changed, 132 insertions(+), 7 deletions(-)

-- 
2.34.1

[PATCH v6 1/8] RDMA: Extend RDMA user ABI to support atomic write

[yangx.jy]

1) Define new atomic write request/completion in userspace.
2) Define new atomic write capability in userspace.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 include/uapi/rdma/ib_user_verbs.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h
index 43672cb1fd57..237814815544 100644
--- a/include/uapi/rdma/ib_user_verbs.h
+++ b/include/uapi/rdma/ib_user_verbs.h
@@ -466,6 +466,7 @@ enum ib_uverbs_wc_opcode {
 	IB_UVERBS_WC_BIND_MW = 5,
 	IB_UVERBS_WC_LOCAL_INV = 6,
 	IB_UVERBS_WC_TSO = 7,
+	IB_UVERBS_WC_ATOMIC_WRITE = 9,
 };
 
 struct ib_uverbs_wc {
@@ -784,6 +785,7 @@ enum ib_uverbs_wr_opcode {
 	IB_UVERBS_WR_RDMA_READ_WITH_INV = 11,
 	IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP = 12,
 	IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD = 13,
+	IB_UVERBS_WR_ATOMIC_WRITE = 15,
 	/* Review enum ib_wr_opcode before modifying this */
 };
 
@@ -1331,6 +1333,8 @@ enum ib_uverbs_device_cap_flags {
 	/* Deprecated. Please use IB_UVERBS_RAW_PACKET_CAP_SCATTER_FCS. */
 	IB_UVERBS_DEVICE_RAW_SCATTER_FCS = 1ULL << 34,
 	IB_UVERBS_DEVICE_PCI_WRITE_END_PADDING = 1ULL << 36,
+	/* Atomic write attributes */
+	IB_UVERBS_DEVICE_ATOMIC_WRITE = 1ULL << 40,
 };
 
 enum ib_uverbs_raw_packet_caps {
-- 
2.34.1

[PATCH v6 2/8] RDMA: Extend RDMA kernel ABI to support atomic write

[yangx.jy]

1) Define new atomic write request/completion in kernel.
2) Define new atomic write capability in kernel.
3) Define new atomic write opcode for RC service in packet.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 include/rdma/ib_pack.h  | 2 ++
 include/rdma/ib_verbs.h | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h
index a9162f25beaf..f932d164af63 100644
--- a/include/rdma/ib_pack.h
+++ b/include/rdma/ib_pack.h
@@ -84,6 +84,7 @@ enum {
 	/* opcode 0x15 is reserved */
 	IB_OPCODE_SEND_LAST_WITH_INVALIDATE         = 0x16,
 	IB_OPCODE_SEND_ONLY_WITH_INVALIDATE         = 0x17,
+	IB_OPCODE_ATOMIC_WRITE                      = 0x1D,
 
 	/* real constants follow -- see comment about above IB_OPCODE()
 	   macro for more details */
@@ -112,6 +113,7 @@ enum {
 	IB_OPCODE(RC, FETCH_ADD),
 	IB_OPCODE(RC, SEND_LAST_WITH_INVALIDATE),
 	IB_OPCODE(RC, SEND_ONLY_WITH_INVALIDATE),
+	IB_OPCODE(RC, ATOMIC_WRITE),
 
 	/* UC */
 	IB_OPCODE(UC, SEND_FIRST),
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 975d6e9efbcb..19bed1fb8e55 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -270,6 +270,7 @@ enum ib_device_cap_flags {
 	/* The device supports padding incoming writes to cacheline. */
 	IB_DEVICE_PCI_WRITE_END_PADDING =
 		IB_UVERBS_DEVICE_PCI_WRITE_END_PADDING,
+	IB_DEVICE_ATOMIC_WRITE = IB_UVERBS_DEVICE_ATOMIC_WRITE,
 };
 
 enum ib_kernel_cap_flags {
@@ -985,6 +986,7 @@ enum ib_wc_opcode {
 	IB_WC_REG_MR,
 	IB_WC_MASKED_COMP_SWAP,
 	IB_WC_MASKED_FETCH_ADD,
+	IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
 /*
  * Set value of IB_WC_RECV so consumers can test if a completion is a
  * receive by testing (opcode & IB_WC_RECV).
@@ -1325,6 +1327,7 @@ enum ib_wr_opcode {
 		IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP,
 	IB_WR_MASKED_ATOMIC_FETCH_AND_ADD =
 		IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD,
+	IB_WR_ATOMIC_WRITE = IB_UVERBS_WR_ATOMIC_WRITE,
 
 	/* These are kernel only and can not be issued by userspace */
 	IB_WR_REG_MR = 0x20,
-- 
2.34.1

[PATCH v6 3/8] RDMA/rxe: Extend rxe user ABI to support atomic write

[yangx.jy]

Define an atomic_wr array to store 8-byte value.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 include/uapi/rdma/rdma_user_rxe.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h
index 73f679dfd2df..d20d1ecf046f 100644
--- a/include/uapi/rdma/rdma_user_rxe.h
+++ b/include/uapi/rdma/rdma_user_rxe.h
@@ -146,6 +146,7 @@ struct rxe_dma_info {
 	__u32			reserved;
 	union {
 		__DECLARE_FLEX_ARRAY(__u8, inline_data);
+		__DECLARE_FLEX_ARRAY(__u8, atomic_wr);
 		__DECLARE_FLEX_ARRAY(struct rxe_sge, sge);
 	};
 };
-- 
2.34.1

[PATCH v6 4/8] RDMA/rxe: Extend rxe packet format to support atomic write

[yangx.jy]

Extend rxe_wr_opcode_info[] and rxe_opcode[] for new atomic write opcode.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 drivers/infiniband/sw/rxe/rxe_opcode.c | 18 ++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_opcode.h |  3 +++
 2 files changed, 21 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.c b/drivers/infiniband/sw/rxe/rxe_opcode.c
index d4ba4d506f17..fb196029048e 100644
--- a/drivers/infiniband/sw/rxe/rxe_opcode.c
+++ b/drivers/infiniband/sw/rxe/rxe_opcode.c
@@ -101,6 +101,12 @@ struct rxe_wr_opcode_info rxe_wr_opcode_info[] = {
 			[IB_QPT_UC]	= WR_LOCAL_OP_MASK,
 		},
 	},
+	[IB_WR_ATOMIC_WRITE]                       = {
+		.name   = "IB_WR_ATOMIC_WRITE",
+		.mask   = {
+			[IB_QPT_RC]     = WR_ATOMIC_WRITE_MASK,
+		},
+	},
 };
 
 struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = {
@@ -378,6 +384,18 @@ struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = {
 					  RXE_IETH_BYTES,
 		}
 	},
+	[IB_OPCODE_RC_ATOMIC_WRITE]                        = {
+		.name   = "IB_OPCODE_RC_ATOMIC_WRITE",
+		.mask   = RXE_RETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK |
+			  RXE_ATOMIC_WRITE_MASK | RXE_START_MASK |
+			  RXE_END_MASK,
+		.length = RXE_BTH_BYTES + RXE_RETH_BYTES,
+		.offset = {
+			[RXE_BTH]       = 0,
+			[RXE_RETH]      = RXE_BTH_BYTES,
+			[RXE_PAYLOAD]   = RXE_BTH_BYTES + RXE_RETH_BYTES,
+		}
+	},
 
 	/* UC */
 	[IB_OPCODE_UC_SEND_FIRST]			= {
diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.h b/drivers/infiniband/sw/rxe/rxe_opcode.h
index 8f9aaaf260f2..a470e9b0b884 100644
--- a/drivers/infiniband/sw/rxe/rxe_opcode.h
+++ b/drivers/infiniband/sw/rxe/rxe_opcode.h
@@ -20,6 +20,7 @@ enum rxe_wr_mask {
 	WR_READ_MASK			= BIT(3),
 	WR_WRITE_MASK			= BIT(4),
 	WR_LOCAL_OP_MASK		= BIT(5),
+	WR_ATOMIC_WRITE_MASK		= BIT(7),
 
 	WR_READ_OR_WRITE_MASK		= WR_READ_MASK | WR_WRITE_MASK,
 	WR_WRITE_OR_SEND_MASK		= WR_WRITE_MASK | WR_SEND_MASK,
@@ -81,6 +82,8 @@ enum rxe_hdr_mask {
 
 	RXE_LOOPBACK_MASK	= BIT(NUM_HDR_TYPES + 12),
 
+	RXE_ATOMIC_WRITE_MASK   = BIT(NUM_HDR_TYPES + 14),
+
 	RXE_READ_OR_ATOMIC_MASK	= (RXE_READ_MASK | RXE_ATOMIC_MASK),
 	RXE_WRITE_OR_SEND_MASK	= (RXE_WRITE_MASK | RXE_SEND_MASK),
 	RXE_READ_OR_WRITE_MASK	= (RXE_READ_MASK | RXE_WRITE_MASK),
-- 
2.34.1

[PATCH v6 6/8] RDMA/rxe: Make responder support atomic write on RC service

[yangx.jy]

Make responder process an atomic write request and send a read response
on RC service.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 drivers/infiniband/sw/rxe/rxe_resp.c | 84 ++++++++++++++++++++++++++--
 1 file changed, 79 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index ed5a09e86417..7d313d9a8d88 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -22,6 +22,7 @@ enum resp_states {
 	RESPST_EXECUTE,
 	RESPST_READ_REPLY,
 	RESPST_ATOMIC_REPLY,
+	RESPST_ATOMIC_WRITE_REPLY,
 	RESPST_COMPLETE,
 	RESPST_ACKNOWLEDGE,
 	RESPST_CLEANUP,
@@ -57,6 +58,7 @@ static char *resp_state_name[] = {
 	[RESPST_EXECUTE]			= "EXECUTE",
 	[RESPST_READ_REPLY]			= "READ_REPLY",
 	[RESPST_ATOMIC_REPLY]			= "ATOMIC_REPLY",
+	[RESPST_ATOMIC_WRITE_REPLY]		= "ATOMIC_WRITE_REPLY",
 	[RESPST_COMPLETE]			= "COMPLETE",
 	[RESPST_ACKNOWLEDGE]			= "ACKNOWLEDGE",
 	[RESPST_CLEANUP]			= "CLEANUP",
@@ -260,7 +262,7 @@ static enum resp_states check_op_valid(struct rxe_qp *qp,
 	case IB_QPT_RC:
 		if (((pkt->mask & RXE_READ_MASK) &&
 		     !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_READ)) ||
-		    ((pkt->mask & RXE_WRITE_MASK) &&
+		    ((pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) &&
 		     !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_WRITE)) ||
 		    ((pkt->mask & RXE_ATOMIC_MASK) &&
 		     !(qp->attr.qp_access_flags & IB_ACCESS_REMOTE_ATOMIC))) {
@@ -364,7 +366,7 @@ static enum resp_states check_resource(struct rxe_qp *qp,
 		}
 	}
 
-	if (pkt->mask & RXE_READ_OR_ATOMIC_MASK) {
+	if (pkt->mask & (RXE_READ_OR_ATOMIC_MASK | RXE_ATOMIC_WRITE_MASK)) {
 		/* it is the requesters job to not send
 		 * too many read/atomic ops, we just
 		 * recycle the responder resource queue
@@ -415,7 +417,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
 	enum resp_states state;
 	int access;
 
-	if (pkt->mask & RXE_READ_OR_WRITE_MASK) {
+	if (pkt->mask & (RXE_READ_OR_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) {
 		if (pkt->mask & RXE_RETH_MASK) {
 			qp->resp.va = reth_va(pkt);
 			qp->resp.offset = 0;
@@ -483,7 +485,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
 		goto err;
 	}
 
-	if (pkt->mask & RXE_WRITE_MASK)	 {
+	if (pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) {
 		if (resid > mtu) {
 			if (pktlen != mtu || bth_pad(pkt)) {
 				state = RESPST_ERR_LENGTH;
@@ -583,6 +585,7 @@ static struct resp_res *rxe_prepare_res(struct rxe_qp *qp,
 		res->state = rdatm_res_state_new;
 		break;
 	case RXE_ATOMIC_MASK:
+	case RXE_ATOMIC_WRITE_MASK:
 		res->first_psn = pkt->psn;
 		res->last_psn = pkt->psn;
 		res->cur_psn = pkt->psn;
@@ -652,6 +655,55 @@ static enum resp_states atomic_reply(struct rxe_qp *qp,
 	return ret;
 }
 
+static enum resp_states atomic_write_reply(struct rxe_qp *qp,
+						struct rxe_pkt_info *pkt)
+{
+	u64 src, *dst;
+	struct resp_res *res = qp->resp.res;
+	struct rxe_mr *mr = qp->resp.mr;
+	int payload = payload_size(pkt);
+
+	if (!res) {
+		res = rxe_prepare_res(qp, pkt, RXE_ATOMIC_WRITE_MASK);
+		qp->resp.res = res;
+	}
+
+	if (!res->replay) {
+#ifdef CONFIG_64BIT
+		if (mr->state != RXE_MR_STATE_VALID)
+			return RESPST_ERR_RKEY_VIOLATION;
+
+		memcpy(&src, payload_addr(pkt), payload);
+
+		dst = iova_to_vaddr(mr, qp->resp.va + qp->resp.offset, payload);
+		/* check vaddr is 8 bytes aligned. */
+		if (!dst || (uintptr_t)dst & 7)
+			return RESPST_ERR_MISALIGNED_ATOMIC;
+
+		/* Do atomic write after all prior operations have completed */
+		smp_store_release(dst, src);
+
+		/* decrease resp.resid to zero */
+		qp->resp.resid -= sizeof(payload);
+
+		qp->resp.msn++;
+
+		/* next expected psn, read handles this separately */
+		qp->resp.psn = (pkt->psn + 1) & BTH_PSN_MASK;
+		qp->resp.ack_psn = qp->resp.psn;
+
+		qp->resp.opcode = pkt->opcode;
+		qp->resp.status = IB_WC_SUCCESS;
+
+		return RESPST_ACKNOWLEDGE;
+#else
+		return RESPST_ERR_UNSUPPORTED_OPCODE;
+#endif /* CONFIG_64BIT */
+	}
+
+	return RESPST_ACKNOWLEDGE;
+}
+
 static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp,
 					  struct rxe_pkt_info *ack,
 					  int opcode,
@@ -888,6 +940,8 @@ static enum resp_states execute(struct rxe_qp *qp, struct rxe_pkt_info *pkt)
 		return RESPST_READ_REPLY;
 	} else if (pkt->mask & RXE_ATOMIC_MASK) {
 		return RESPST_ATOMIC_REPLY;
+	} else if (pkt->mask & RXE_ATOMIC_WRITE_MASK) {
+		return RESPST_ATOMIC_WRITE_REPLY;
 	} else {
 		/* Unreachable */
 		WARN_ON_ONCE(1);
@@ -1061,6 +1115,19 @@ static int send_atomic_ack(struct rxe_qp *qp, u8 syndrome, u32 psn)
 	return ret;
 }
 
+static int send_read_response_ack(struct rxe_qp *qp, u8 syndrome, u32 psn)
+{
+	int ret = send_common_ack(qp, syndrome, psn,
+			IB_OPCODE_RC_RDMA_READ_RESPONSE_ONLY,
+			"RDMA READ response of length zero ACK");
+
+	/* have to clear this since it is used to trigger
+	 * long read replies
+	 */
+	qp->resp.res = NULL;
+	return ret;
+}
+
 static enum resp_states acknowledge(struct rxe_qp *qp,
 				    struct rxe_pkt_info *pkt)
 {
@@ -1071,6 +1138,8 @@ static enum resp_states acknowledge(struct rxe_qp *qp,
 		send_ack(qp, qp->resp.aeth_syndrome, pkt->psn);
 	else if (pkt->mask & RXE_ATOMIC_MASK)
 		send_atomic_ack(qp, AETH_ACK_UNLIMITED, pkt->psn);
+	else if (pkt->mask & RXE_ATOMIC_WRITE_MASK)
+		send_read_response_ack(qp, AETH_ACK_UNLIMITED, pkt->psn);
 	else if (bth_ack(pkt))
 		send_ack(qp, AETH_ACK_UNLIMITED, pkt->psn);
 
@@ -1182,7 +1251,9 @@ static enum resp_states duplicate_request(struct rxe_qp *qp,
 			res->replay = 1;
 			res->cur_psn = pkt->psn;
 			qp->resp.res = res;
-			rc = RESPST_ATOMIC_REPLY;
+			rc = pkt->mask & RXE_ATOMIC_MASK ?
+					RESPST_ATOMIC_REPLY :
+					RESPST_ATOMIC_WRITE_REPLY;
 			goto out;
 		}
 
@@ -1320,6 +1391,9 @@ int rxe_responder(void *arg)
 		case RESPST_ATOMIC_REPLY:
 			state = atomic_reply(qp, pkt);
 			break;
+		case RESPST_ATOMIC_WRITE_REPLY:
+			state = atomic_write_reply(qp, pkt);
+			break;
 		case RESPST_ACKNOWLEDGE:
 			state = acknowledge(qp, pkt);
 			break;
-- 
2.34.1

[PATCH v6 5/8] RDMA/rxe: Make requester support atomic write on RC service

[yangx.jy]

Make requester process and send an atomic write request on RC service.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 drivers/infiniband/sw/rxe/rxe_req.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c
index f63771207970..50d794910a3f 100644
--- a/drivers/infiniband/sw/rxe/rxe_req.c
+++ b/drivers/infiniband/sw/rxe/rxe_req.c
@@ -258,6 +258,10 @@ static int next_opcode_rc(struct rxe_qp *qp, u32 opcode, int fits)
 		else
 			return fits ? IB_OPCODE_RC_SEND_ONLY_WITH_INVALIDATE :
 				IB_OPCODE_RC_SEND_FIRST;
+
+	case IB_WR_ATOMIC_WRITE:
+		return IB_OPCODE_RC_ATOMIC_WRITE;
+
 	case IB_WR_REG_MR:
 	case IB_WR_LOCAL_INV:
 		return opcode;
@@ -486,6 +490,11 @@ static int finish_packet(struct rxe_qp *qp, struct rxe_av *av,
 		}
 	}
 
+	if (pkt->mask & RXE_ATOMIC_WRITE_MASK) {
+		memcpy(payload_addr(pkt), wqe->dma.atomic_wr, payload);
+		wqe->dma.resid -= payload;
+	}
+
 	return 0;
 }
 
@@ -709,13 +718,15 @@ int rxe_requester(void *arg)
 	}
 
 	mask = rxe_opcode[opcode].mask;
-	if (unlikely(mask & RXE_READ_OR_ATOMIC_MASK)) {
+	if (unlikely(mask & (RXE_READ_OR_ATOMIC_MASK |
+			RXE_ATOMIC_WRITE_MASK))) {
 		if (check_init_depth(qp, wqe))
 			goto exit;
 	}
 
 	mtu = get_mtu(qp);
-	payload = (mask & RXE_WRITE_OR_SEND_MASK) ? wqe->dma.resid : 0;
+	payload = (mask & (RXE_WRITE_OR_SEND_MASK | RXE_ATOMIC_WRITE_MASK)) ?
+			wqe->dma.resid : 0;
 	if (payload > mtu) {
 		if (qp_type(qp) == IB_QPT_UD) {
 			/* C10-93.1.1: If the total sum of all the buffer lengths specified for a
-- 
2.34.1

[PATCH v6 7/8] RDMA/rxe: Implement atomic write completion

[yangx.jy]

Generate an atomic write completion when the atomic write request
has been finished.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 drivers/infiniband/sw/rxe/rxe_comp.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c
index fb0c008af78c..46dba6f163ca 100644
--- a/drivers/infiniband/sw/rxe/rxe_comp.c
+++ b/drivers/infiniband/sw/rxe/rxe_comp.c
@@ -104,6 +104,7 @@ static enum ib_wc_opcode wr_to_wc_opcode(enum ib_wr_opcode opcode)
 	case IB_WR_LOCAL_INV:			return IB_WC_LOCAL_INV;
 	case IB_WR_REG_MR:			return IB_WC_REG_MR;
 	case IB_WR_BIND_MW:			return IB_WC_BIND_MW;
+	case IB_WR_ATOMIC_WRITE:		return IB_WC_ATOMIC_WRITE;
 
 	default:
 		return 0xff;
@@ -258,6 +259,9 @@ static inline enum comp_state check_ack(struct rxe_qp *qp,
 		if ((syn & AETH_TYPE_MASK) != AETH_ACK)
 			return COMPST_ERROR;
 
+		if (wqe->wr.opcode == IB_WR_ATOMIC_WRITE)
+			return COMPST_WRITE_SEND;
+
 		fallthrough;
 		/* (IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE doesn't have an AETH)
 		 */
-- 
2.34.1

[PATCH v6 8/8] RDMA/rxe: Enable atomic write capability for rxe device

[yangx.jy]

The capability shows that rxe device supports atomic write operation.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 drivers/infiniband/sw/rxe/rxe_param.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h
index 86c7a8bf3cbb..bbc88cd71d95 100644
--- a/drivers/infiniband/sw/rxe/rxe_param.h
+++ b/drivers/infiniband/sw/rxe/rxe_param.h
@@ -51,7 +51,12 @@ enum rxe_device_param {
 					| IB_DEVICE_SRQ_RESIZE
 					| IB_DEVICE_MEM_MGT_EXTENSIONS
 					| IB_DEVICE_MEM_WINDOW
+#ifdef CONFIG_64BIT
+					| IB_DEVICE_MEM_WINDOW_TYPE_2B
+					| IB_DEVICE_ATOMIC_WRITE,
+#else
 					| IB_DEVICE_MEM_WINDOW_TYPE_2B,
+#endif /* CONFIG_64BIT */
 	RXE_MAX_SGE			= 32,
 	RXE_MAX_WQE_SIZE		= sizeof(struct rxe_send_wqe) +
 					  sizeof(struct ib_sge) * RXE_MAX_SGE,
-- 
2.34.1

Re: [PATCH v6 0/8] RDMA/rxe: Add atomic write operation

[Yang, Xiao/杨 晓]

Hi Jason, Bob

Thanks a lot for your review. I have updated the patch set according to 
your all comments except the missing kmap issue.

I didn't understand why current iova_to_vaddr() has been broken so I 
hope we can discuss the issue fully and then find a suitable solution.

Best Regards,
Xiao Yang

On 2022/10/15 14:37, Yang, Xiao/杨 晓 wrote:
> The IB SPEC v1.5[1] defined new atomic write operation. This patchset
> makes SoftRoCE support new atomic write on RC service.
> 
> On my rdma-core repository[2], I have introduced atomic write API
> for libibverbs and Pyverbs. I also have provided a rdma_atomic_write
> example and test_qp_ex_rc_atomic_write python test to verify
> the patchset.
> 
> The steps to run the rdma_atomic_write example:
> server:
> $ ./rdma_atomic_write_server -s [server_address] -p [port_number]
> client:
> $ ./rdma_atomic_write_client -s [server_address] -p [port_number]
> 
> The steps to run test_qp_ex_rc_atomic_write test:
> run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write
> test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok
> 
> ----------------------------------------------------------------------
> Ran 1 test in 0.008s
> 
> OK
> 
> [1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
> [2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point
> 
> v5->v6:
> 1) Rebase on current for-next
> 2) Split the implementation of atomic write into 7 patches
> 3) Replace all "RDMA Atomic Write" with "atomic write"
> 4) Save 8-byte value in struct rxe_dma_info instead
> 5) Remove the print in atomic_write_reply()
> 
> v4->v5:
> 1) Rebase on current wip/jgg-for-next
> 2) Rewrite the implementation on responder
> 
> v3->v4:
> 1) Rebase on current wip/jgg-for-next
> 2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write
> 3) Replace 64-bit value with 8-byte array for atomic write
> 
> V2->V3:
> 1) Rebase
> 2) Add RDMA Atomic Write attribute for rxe device
> 
> V1->V2:
> 1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D
> 2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value
> 3) Use smp_store_release() to ensure that all prior operations have completed
> 
> Xiao Yang (8):
>    RDMA: Extend RDMA user ABI to support atomic write
>    RDMA: Extend RDMA kernel ABI to support atomic write
>    RDMA/rxe: Extend rxe user ABI to support atomic write
>    RDMA/rxe: Extend rxe packet format to support atomic write
>    RDMA/rxe: Make requester support atomic write on RC service
>    RDMA/rxe: Make responder support atomic write on RC service
>    RDMA/rxe: Implement atomic write completion
>    RDMA/rxe: Enable atomic write capability for rxe device
> 
>   drivers/infiniband/sw/rxe/rxe_comp.c   |  4 ++
>   drivers/infiniband/sw/rxe/rxe_opcode.c | 18 ++++++
>   drivers/infiniband/sw/rxe/rxe_opcode.h |  3 +
>   drivers/infiniband/sw/rxe/rxe_param.h  |  5 ++
>   drivers/infiniband/sw/rxe/rxe_req.c    | 15 ++++-
>   drivers/infiniband/sw/rxe/rxe_resp.c   | 84 ++++++++++++++++++++++++--
>   include/rdma/ib_pack.h                 |  2 +
>   include/rdma/ib_verbs.h                |  3 +
>   include/uapi/rdma/ib_user_verbs.h      |  4 ++
>   include/uapi/rdma/rdma_user_rxe.h      |  1 +
>   10 files changed, 132 insertions(+), 7 deletions(-)
> 

Re: [PATCH v6 1/8] RDMA: Extend RDMA user ABI to support atomic write

[Jason Gunthorpe]

On Sat, Oct 15, 2022 at 06:37:04AM +0000, yangx.jy@fujitsu.com wrote:
> 1) Define new atomic write request/completion in userspace.
> 2) Define new atomic write capability in userspace.
> 
> Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
> ---
>  include/uapi/rdma/ib_user_verbs.h | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h
> index 43672cb1fd57..237814815544 100644
> --- a/include/uapi/rdma/ib_user_verbs.h
> +++ b/include/uapi/rdma/ib_user_verbs.h
> @@ -466,6 +466,7 @@ enum ib_uverbs_wc_opcode {
>  	IB_UVERBS_WC_BIND_MW = 5,
>  	IB_UVERBS_WC_LOCAL_INV = 6,
>  	IB_UVERBS_WC_TSO = 7,
> +	IB_UVERBS_WC_ATOMIC_WRITE = 9,
>  };

Why is this 9? The following patch does

@@ -985,6 +986,7 @@ enum ib_wc_opcode {
        IB_WC_REG_MR,
        IB_WC_MASKED_COMP_SWAP,
        IB_WC_MASKED_FETCH_ADD,
+       IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,

Which corrupts the enum.

It should be like this:

+++ b/include/rdma/ib_verbs.h
@@ -983,10 +983,10 @@ enum ib_wc_opcode {
        IB_WC_BIND_MW = IB_UVERBS_WC_BIND_MW,
        IB_WC_LOCAL_INV = IB_UVERBS_WC_LOCAL_INV,
        IB_WC_LSO = IB_UVERBS_WC_TSO,
+       IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
        IB_WC_REG_MR,
        IB_WC_MASKED_COMP_SWAP,
        IB_WC_MASKED_FETCH_ADD,
-       IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
 /*
  * Set value of IB_WC_RECV so consumers can test if a completion is a
  * receive by testing (opcode & IB_WC_RECV).
+++ b/include/uapi/rdma/ib_user_verbs.h
@@ -466,7 +466,7 @@ enum ib_uverbs_wc_opcode {
        IB_UVERBS_WC_BIND_MW = 5,
        IB_UVERBS_WC_LOCAL_INV = 6,
        IB_UVERBS_WC_TSO = 7,
-       IB_UVERBS_WC_ATOMIC_WRITE = 9,
+       IB_UVERBS_WC_ATOMIC_WRITE = 8,
 };
 
 struct ib_uverbs_wc {

Jason

Re: [PATCH v6 0/8] RDMA/rxe: Add atomic write operation

[Jason Gunthorpe]

On Sat, Oct 15, 2022 at 06:37:03AM +0000, yangx.jy@fujitsu.com wrote:
> The IB SPEC v1.5[1] defined new atomic write operation. This patchset
> makes SoftRoCE support new atomic write on RC service.
> 
> On my rdma-core repository[2], I have introduced atomic write API
> for libibverbs and Pyverbs. I also have provided a rdma_atomic_write
> example and test_qp_ex_rc_atomic_write python test to verify
> the patchset.
> 
> The steps to run the rdma_atomic_write example:
> server:
> $ ./rdma_atomic_write_server -s [server_address] -p [port_number]
> client:
> $ ./rdma_atomic_write_client -s [server_address] -p [port_number]
> 
> The steps to run test_qp_ex_rc_atomic_write test:
> run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write
> test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok
> 
> ----------------------------------------------------------------------
> Ran 1 test in 0.008s
> 
> OK
> 
> [1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
> [2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point
> 
> v5->v6:
> 1) Rebase on current for-next
> 2) Split the implementation of atomic write into 7 patches
> 3) Replace all "RDMA Atomic Write" with "atomic write"
> 4) Save 8-byte value in struct rxe_dma_info instead
> 5) Remove the print in atomic_write_reply()

I think this looked OK, please fix the enum thing and also resolve all
the remarks on the github and rebase/repost/retest both series.

Thanks,
Jason

Re: [PATCH v6 1/8] RDMA: Extend RDMA user ABI to support atomic write

[Xiao Yang]

On 2022/11/23 3:49, Jason Gunthorpe wrote:
> On Sat, Oct 15, 2022 at 06:37:04AM +0000, yangx.jy@fujitsu.com wrote:
>> 1) Define new atomic write request/completion in userspace.
>> 2) Define new atomic write capability in userspace.
>>
>> Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
>> ---
>>   include/uapi/rdma/ib_user_verbs.h | 4 ++++
>>   1 file changed, 4 insertions(+)
>>
>> diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h
>> index 43672cb1fd57..237814815544 100644
>> --- a/include/uapi/rdma/ib_user_verbs.h
>> +++ b/include/uapi/rdma/ib_user_verbs.h
>> @@ -466,6 +466,7 @@ enum ib_uverbs_wc_opcode {
>>   	IB_UVERBS_WC_BIND_MW = 5,
>>   	IB_UVERBS_WC_LOCAL_INV = 6,
>>   	IB_UVERBS_WC_TSO = 7,
>> +	IB_UVERBS_WC_ATOMIC_WRITE = 9,
>>   };
> 
> Why is this 9? The following patch does
Hi Jason,

I reserve 8 for IB_UVERBS_WC_FLUSH and 14 for IB_UVERBS_WR_FLUSH.

> 
> @@ -985,6 +986,7 @@ enum ib_wc_opcode {
>          IB_WC_REG_MR,
>          IB_WC_MASKED_COMP_SWAP,
>          IB_WC_MASKED_FETCH_ADD,
> +       IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
> 
> Which corrupts the enum.

Good catch. I will correct it now.

Best Regards,
Xiao Yang
> 
> It should be like this:
> 
> +++ b/include/rdma/ib_verbs.h
> @@ -983,10 +983,10 @@ enum ib_wc_opcode {
>          IB_WC_BIND_MW = IB_UVERBS_WC_BIND_MW,
>          IB_WC_LOCAL_INV = IB_UVERBS_WC_LOCAL_INV,
>          IB_WC_LSO = IB_UVERBS_WC_TSO,
> +       IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
>          IB_WC_REG_MR,
>          IB_WC_MASKED_COMP_SWAP,
>          IB_WC_MASKED_FETCH_ADD,
> -       IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
>   /*
>    * Set value of IB_WC_RECV so consumers can test if a completion is a
>    * receive by testing (opcode & IB_WC_RECV).
> +++ b/include/uapi/rdma/ib_user_verbs.h
> @@ -466,7 +466,7 @@ enum ib_uverbs_wc_opcode {
>          IB_UVERBS_WC_BIND_MW = 5,
>          IB_UVERBS_WC_LOCAL_INV = 6,
>          IB_UVERBS_WC_TSO = 7,
> -       IB_UVERBS_WC_ATOMIC_WRITE = 9,
> +       IB_UVERBS_WC_ATOMIC_WRITE = 8,
>   };
>   
>   struct ib_uverbs_wc {
> 
> Jason

Re: [PATCH v6 0/8] RDMA/rxe: Add atomic write operation

[Xiao Yang]

On 2022/11/23 3:54, Jason Gunthorpe wrote:
> On Sat, Oct 15, 2022 at 06:37:03AM +0000, yangx.jy@fujitsu.com wrote:
>> The IB SPEC v1.5[1] defined new atomic write operation. This patchset
>> makes SoftRoCE support new atomic write on RC service.
>>
>> On my rdma-core repository[2], I have introduced atomic write API
>> for libibverbs and Pyverbs. I also have provided a rdma_atomic_write
>> example and test_qp_ex_rc_atomic_write python test to verify
>> the patchset.
>>
>> The steps to run the rdma_atomic_write example:
>> server:
>> $ ./rdma_atomic_write_server -s [server_address] -p [port_number]
>> client:
>> $ ./rdma_atomic_write_client -s [server_address] -p [port_number]
>>
>> The steps to run test_qp_ex_rc_atomic_write test:
>> run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write
>> test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok
>>
>> ----------------------------------------------------------------------
>> Ran 1 test in 0.008s
>>
>> OK
>>
>> [1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
>> [2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point
>>
>> v5->v6:
>> 1) Rebase on current for-next
>> 2) Split the implementation of atomic write into 7 patches
>> 3) Replace all "RDMA Atomic Write" with "atomic write"
>> 4) Save 8-byte value in struct rxe_dma_info instead
>> 5) Remove the print in atomic_write_reply()
> 
> I think this looked OK, please fix the enum thing and also resolve all
> the remarks on the github and rebase/repost/retest both series.
Hi Jason,

Thanks for your reminder. I will do it soon.
In addition, I have resolved all remarks except the following one on github:
EdwardSro: "keep an empty line at EoF"
I: "I wonder why we need to add an empty line at EoF? I think there is 
no empty line at EOF in other files."

I hope you or EdwardSro can answer my question. ^_^

Best Regards,
Xiao Yang
> 
> Thanks,
> Jason

Re: [PATCH v6 0/8] RDMA/rxe: Add atomic write operation

[Jason Gunthorpe]

On Thu, Dec 01, 2022 at 07:58:44PM +0800, Xiao Yang wrote:
> On 2022/11/23 3:54, Jason Gunthorpe wrote:
> > On Sat, Oct 15, 2022 at 06:37:03AM +0000, yangx.jy@fujitsu.com wrote:
> > > The IB SPEC v1.5[1] defined new atomic write operation. This patchset
> > > makes SoftRoCE support new atomic write on RC service.
> > > 
> > > On my rdma-core repository[2], I have introduced atomic write API
> > > for libibverbs and Pyverbs. I also have provided a rdma_atomic_write
> > > example and test_qp_ex_rc_atomic_write python test to verify
> > > the patchset.
> > > 
> > > The steps to run the rdma_atomic_write example:
> > > server:
> > > $ ./rdma_atomic_write_server -s [server_address] -p [port_number]
> > > client:
> > > $ ./rdma_atomic_write_client -s [server_address] -p [port_number]
> > > 
> > > The steps to run test_qp_ex_rc_atomic_write test:
> > > run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write
> > > test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok
> > > 
> > > ----------------------------------------------------------------------
> > > Ran 1 test in 0.008s
> > > 
> > > OK
> > > 
> > > [1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
> > > [2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point
> > > 
> > > v5->v6:
> > > 1) Rebase on current for-next
> > > 2) Split the implementation of atomic write into 7 patches
> > > 3) Replace all "RDMA Atomic Write" with "atomic write"
> > > 4) Save 8-byte value in struct rxe_dma_info instead
> > > 5) Remove the print in atomic_write_reply()
> > 
> > I think this looked OK, please fix the enum thing and also resolve all
> > the remarks on the github and rebase/repost/retest both series.
> Hi Jason,
> 
> Thanks for your reminder. I will do it soon.
> In addition, I have resolved all remarks except the following one on github:
> EdwardSro: "keep an empty line at EoF"
> I: "I wonder why we need to add an empty line at EoF? I think there is no
> empty line at EOF in other files."

It is not really "empty line" it is that the last character in the
file should be '\n', and all files are like that.

Jason

Re: [PATCH v6 0/8] RDMA/rxe: Add atomic write operation

[Xiao Yang]

On 2022/12/1 20:58, Jason Gunthorpe wrote:
> It is not really "empty line" it is that the last character in the
> file should be '\n', and all files are like that.
Hi Jason,

Thanks for your explanation.

I think my latest patch has added an "empty line", right?
diff --git a/pyverbs/libibverbs_enums.pxd b/pyverbs/libibverbs_enums.pxd
index 6a875fdd..c78a2284 100644
--- a/pyverbs/libibverbs_enums.pxd
+++ b/pyverbs/libibverbs_enums.pxd
...
@@ -507,4 +510,4 @@ cdef extern from "<infiniband/tm_types.h>":
          IBV_TMH_NO_TAG
          IBV_TMH_RNDV
          IBV_TMH_FIN
-        IBV_TMH_EAGER
\ No newline at end of file
+        IBV_TMH_EAGER

# cat -A pyverbs/libibverbs_enums.pxd | tail -2
         IBV_TMH_FIN$
         IBV_TMH_EAGER$

Best Regards,
Xiao Yang
> 
> Jason

Re: [PATCH v6 0/8] RDMA/rxe: Add atomic write operation

[Jason Gunthorpe]

On Thu, Dec 01, 2022 at 10:16:15PM +0800, Xiao Yang wrote:
> On 2022/12/1 20:58, Jason Gunthorpe wrote:
> > It is not really "empty line" it is that the last character in the
> > file should be '\n', and all files are like that.
> Hi Jason,
> 
> Thanks for your explanation.
> 
> I think my latest patch has added an "empty line", right?

yes

Jason