[PATCH v7 0/8] RDMA/rxe: Add atomic write operation

[PATCH v7 0/8] RDMA/rxe: Add atomic write operation

[Xiao Yang]

The IB SPEC v1.5[1] defined new atomic write operation. This patchset
makes SoftRoCE support new atomic write on RC service.

On my rdma-core repository[2], I have introduced atomic write API
for libibverbs and Pyverbs. I also have provided a rdma_atomic_write
example and test_qp_ex_rc_atomic_write python test to verify
the patchset.

The steps to run the rdma_atomic_write example:
server:
$ ./rdma_atomic_write_server -s [server_address] -p [port_number]
client:
$ ./rdma_atomic_write_client -s [server_address] -p [port_number]

The steps to run test_qp_ex_rc_atomic_write test:
run_tests.py --dev rxe_enp0s3 --gid 1 -v test_qpex.QpExTestCase.test_qp_ex_rc_atomic_write
test_qp_ex_rc_atomic_write (tests.test_qpex.QpExTestCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.013s

OK

[1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx
[2]: https://github.com/yangx-jy/rdma-core/tree/new_api_with_point

v6->v7:
1) Rebase on current for-next
2) Define IB_WC_ATOMIC_WRITE in order

v5->v6:
1) Rebase on current for-next
2) Split the implementation of atomic write into 7 patches
3) Replace all "RDMA Atomic Write" with "atomic write"
4) Save 8-byte value in struct rxe_dma_info instead
5) Remove the print in atomic_write_reply()

v4->v5:
1) Rebase on current wip/jgg-for-next
2) Rewrite the implementation on responder

v3->v4:
1) Rebase on current wip/jgg-for-next
2) Fix a compiler error on 32-bit arch (e.g. parisc) by disabling RDMA Atomic Write
3) Replace 64-bit value with 8-byte array for atomic write

V2->V3:
1) Rebase
2) Add RDMA Atomic Write attribute for rxe device

V1->V2:
1) Set IB_OPCODE_RDMA_ATOMIC_WRITE to 0x1D
2) Add rdma.atomic_wr in struct rxe_send_wr and use it to pass the atomic write value
3) Use smp_store_release() to ensure that all prior operations have completed

Xiao Yang (8):
  RDMA: Extend RDMA user ABI to support atomic write
  RDMA: Extend RDMA kernel ABI to support atomic write
  RDMA/rxe: Extend rxe user ABI to support atomic write
  RDMA/rxe: Extend rxe packet format to support atomic write
  RDMA/rxe: Make requester support atomic write on RC service
  RDMA/rxe: Make responder support atomic write on RC service
  RDMA/rxe: Implement atomic write completion
  RDMA/rxe: Enable atomic write capability for rxe device

 drivers/infiniband/sw/rxe/rxe_comp.c   |  4 ++
 drivers/infiniband/sw/rxe/rxe_opcode.c | 18 ++++++
 drivers/infiniband/sw/rxe/rxe_opcode.h |  3 +
 drivers/infiniband/sw/rxe/rxe_param.h  |  5 ++
 drivers/infiniband/sw/rxe/rxe_req.c    | 15 ++++-
 drivers/infiniband/sw/rxe/rxe_resp.c   | 84 ++++++++++++++++++++++++--
 include/rdma/ib_pack.h                 |  2 +
 include/rdma/ib_verbs.h                |  3 +
 include/uapi/rdma/ib_user_verbs.h      |  4 ++
 include/uapi/rdma/rdma_user_rxe.h      |  1 +
 10 files changed, 132 insertions(+), 7 deletions(-)

-- 
2.34.1

[PATCH v7 1/8] RDMA: Extend RDMA user ABI to support atomic write

[Xiao Yang]

1) Define new atomic write request/completion in userspace.
2) Define new atomic write capability in userspace.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 include/uapi/rdma/ib_user_verbs.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h
index 43672cb1fd57..237814815544 100644
--- a/include/uapi/rdma/ib_user_verbs.h
+++ b/include/uapi/rdma/ib_user_verbs.h
@@ -466,6 +466,7 @@ enum ib_uverbs_wc_opcode {
 	IB_UVERBS_WC_BIND_MW = 5,
 	IB_UVERBS_WC_LOCAL_INV = 6,
 	IB_UVERBS_WC_TSO = 7,
+	IB_UVERBS_WC_ATOMIC_WRITE = 9,
 };
 
 struct ib_uverbs_wc {
@@ -784,6 +785,7 @@ enum ib_uverbs_wr_opcode {
 	IB_UVERBS_WR_RDMA_READ_WITH_INV = 11,
 	IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP = 12,
 	IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD = 13,
+	IB_UVERBS_WR_ATOMIC_WRITE = 15,
 	/* Review enum ib_wr_opcode before modifying this */
 };
 
@@ -1331,6 +1333,8 @@ enum ib_uverbs_device_cap_flags {
 	/* Deprecated. Please use IB_UVERBS_RAW_PACKET_CAP_SCATTER_FCS. */
 	IB_UVERBS_DEVICE_RAW_SCATTER_FCS = 1ULL << 34,
 	IB_UVERBS_DEVICE_PCI_WRITE_END_PADDING = 1ULL << 36,
+	/* Atomic write attributes */
+	IB_UVERBS_DEVICE_ATOMIC_WRITE = 1ULL << 40,
 };
 
 enum ib_uverbs_raw_packet_caps {
-- 
2.34.1

[PATCH v7 2/8] RDMA: Extend RDMA kernel ABI to support atomic write

[Xiao Yang]

1) Define new atomic write request/completion in kernel.
2) Define new atomic write capability in kernel.
3) Define new atomic write opcode for RC service in packet.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 include/rdma/ib_pack.h  | 2 ++
 include/rdma/ib_verbs.h | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h
index a9162f25beaf..f932d164af63 100644
--- a/include/rdma/ib_pack.h
+++ b/include/rdma/ib_pack.h
@@ -84,6 +84,7 @@ enum {
 	/* opcode 0x15 is reserved */
 	IB_OPCODE_SEND_LAST_WITH_INVALIDATE         = 0x16,
 	IB_OPCODE_SEND_ONLY_WITH_INVALIDATE         = 0x17,
+	IB_OPCODE_ATOMIC_WRITE                      = 0x1D,
 
 	/* real constants follow -- see comment about above IB_OPCODE()
 	   macro for more details */
@@ -112,6 +113,7 @@ enum {
 	IB_OPCODE(RC, FETCH_ADD),
 	IB_OPCODE(RC, SEND_LAST_WITH_INVALIDATE),
 	IB_OPCODE(RC, SEND_ONLY_WITH_INVALIDATE),
+	IB_OPCODE(RC, ATOMIC_WRITE),
 
 	/* UC */
 	IB_OPCODE(UC, SEND_FIRST),
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 77dd9148815b..df6bb26ba0be 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -270,6 +270,7 @@ enum ib_device_cap_flags {
 	/* The device supports padding incoming writes to cacheline. */
 	IB_DEVICE_PCI_WRITE_END_PADDING =
 		IB_UVERBS_DEVICE_PCI_WRITE_END_PADDING,
+	IB_DEVICE_ATOMIC_WRITE = IB_UVERBS_DEVICE_ATOMIC_WRITE,
 };
 
 enum ib_kernel_cap_flags {
@@ -982,6 +983,7 @@ enum ib_wc_opcode {
 	IB_WC_BIND_MW = IB_UVERBS_WC_BIND_MW,
 	IB_WC_LOCAL_INV = IB_UVERBS_WC_LOCAL_INV,
 	IB_WC_LSO = IB_UVERBS_WC_TSO,
+	IB_WC_ATOMIC_WRITE = IB_UVERBS_WC_ATOMIC_WRITE,
 	IB_WC_REG_MR,
 	IB_WC_MASKED_COMP_SWAP,
 	IB_WC_MASKED_FETCH_ADD,
@@ -1325,6 +1327,7 @@ enum ib_wr_opcode {
 		IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP,
 	IB_WR_MASKED_ATOMIC_FETCH_AND_ADD =
 		IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD,
+	IB_WR_ATOMIC_WRITE = IB_UVERBS_WR_ATOMIC_WRITE,
 
 	/* These are kernel only and can not be issued by userspace */
 	IB_WR_REG_MR = 0x20,
-- 
2.34.1

[PATCH v7 3/8] RDMA/rxe: Extend rxe user ABI to support atomic write

[Xiao Yang]

Define an atomic_wr array to store 8-byte value.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 include/uapi/rdma/rdma_user_rxe.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h
index 73f679dfd2df..d20d1ecf046f 100644
--- a/include/uapi/rdma/rdma_user_rxe.h
+++ b/include/uapi/rdma/rdma_user_rxe.h
@@ -146,6 +146,7 @@ struct rxe_dma_info {
 	__u32			reserved;
 	union {
 		__DECLARE_FLEX_ARRAY(__u8, inline_data);
+		__DECLARE_FLEX_ARRAY(__u8, atomic_wr);
 		__DECLARE_FLEX_ARRAY(struct rxe_sge, sge);
 	};
 };
-- 
2.34.1

[PATCH v7 4/8] RDMA/rxe: Extend rxe packet format to support atomic write

[Xiao Yang]

Extend rxe_wr_opcode_info[] and rxe_opcode[] for new atomic write opcode.

Signed-off-by: Xiao Yang <yangx.jy@fujitsu.com>
---
 drivers/infiniband/sw/rxe/rxe_opcode.c | 18 ++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_opcode.h |  3 +++
 2 files changed, 21 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.c b/drivers/infiniband/sw/rxe/rxe_opcode.c
index d4ba4d506f17..fb196029048e 100644
--- a/drivers/infiniband/sw/rxe/rxe_opcode.c
+++ b/drivers/infiniband/sw/rxe/rxe_opcode.c
@@ -101,6 +101,12 @@ struct rxe_wr_opcode_info rxe_wr_opcode_info[] = {
 			[IB_QPT_UC]	= WR_LOCAL_OP_MASK,
 		},
 	},
+	[IB_WR_ATOMIC_WRITE]                       = {
+		.name   = "IB_WR_ATOMIC_WRITE",
+		.mask   = {
+			[IB_QPT_RC]     = WR_ATOMIC_WRITE_MASK,
+		},
+	},
 };
 
 struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = {
@@ -378,6 +384,18 @@ struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] = {
 					  RXE_IETH_BYTES,
 		}
 	},
+	[IB_OPCODE_RC_ATOMIC_WRITE]                        = {
+		.name   = "IB_OPCODE_RC_ATOMIC_WRITE",
+		.mask   = RXE_RETH_MASK | RXE_PAYLOAD_MASK | RXE_REQ_MASK |
+			  RXE_ATOMIC_WRITE_MASK | RXE_START_MASK |
+			  RXE_END_MASK,
+		.length = RXE_BTH_BYTES + RXE_RETH_BYTES,
+		.offset = {
+			[RXE_BTH]       = 0,
+			[RXE_RETH]      = RXE_BTH_BYTES,
+			[RXE_PAYLOAD]   = RXE_BTH_BYTES + RXE_RETH_BYTES,
+		}
+	},
 
 	/* UC */
 	[IB_OPCODE_UC_SEND_FIRST]			= {
diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.h b/drivers/infiniband/sw/rxe/rxe_opcode.h
index 8f9aaaf260f2..a470e9b0b884 100644
--- a/drivers/infiniband/sw/rxe/rxe_opcode.h
+++ b/drivers/infiniband/sw/rxe/rxe_opcode.h
@@ -20,6 +20,7 @@ enum rxe_wr_mask {
 	WR_READ_MASK			= BIT(3),
 	WR_WRITE_MASK			= BIT(4),
 	WR_LOCAL_OP_MASK		= BIT(5),
+	WR_ATOMIC_WRITE_MASK		= BIT(7),
 
 	WR_READ_OR_WRITE_MASK		= WR_READ_MASK | WR_WRITE_MASK,
 	WR_WRITE_OR_SEND_MASK		= WR_WRITE_MASK | WR_SEND_MASK,
@@ -81,6 +82,8 @@ enum rxe_hdr_mask {
 
 	RXE_LOOPBACK_MASK	= BIT(NUM_HDR_TYPES + 12),
 
+	RXE_ATOMIC_WRITE_MASK   = BIT(NUM_HDR_TYPES + 14),
+
 	RXE_READ_OR_ATOMIC_MASK	= (RXE_READ_MASK | RXE_ATOMIC_MASK),
 	RXE_WRITE_OR_SEND_MASK	= (RXE_WRITE_MASK | RXE_SEND_MASK),
 	RXE_READ_OR_WRITE_MASK	= (RXE_READ_MASK | RXE_WRITE_MASK),
-- 
2.34.1

Re: [PATCH v7 0/8] RDMA/rxe: Add atomic write operation

[Jason Gunthorpe]

On Thu, Dec 01, 2022 at 02:37:04PM +0000, Xiao Yang wrote:

> Xiao Yang (8):
>   RDMA: Extend RDMA user ABI to support atomic write
>   RDMA: Extend RDMA kernel ABI to support atomic write
>   RDMA/rxe: Extend rxe user ABI to support atomic write
>   RDMA/rxe: Extend rxe packet format to support atomic write
>   RDMA/rxe: Make requester support atomic write on RC service
>   RDMA/rxe: Make responder support atomic write on RC service
>   RDMA/rxe: Implement atomic write completion
>   RDMA/rxe: Enable atomic write capability for rxe device

Applied to for-next, thanks

Jason

Re: [PATCH v7 0/8] RDMA/rxe: Add atomic write operation

[Xiao Yang]

On 2022/12/2 7:55, Jason Gunthorpe wrote:
> On Thu, Dec 01, 2022 at 02:37:04PM +0000, Xiao Yang wrote:
> 
>> Xiao Yang (8):
>>    RDMA: Extend RDMA user ABI to support atomic write
>>    RDMA: Extend RDMA kernel ABI to support atomic write
>>    RDMA/rxe: Extend rxe user ABI to support atomic write
>>    RDMA/rxe: Extend rxe packet format to support atomic write
>>    RDMA/rxe: Make requester support atomic write on RC service
>>    RDMA/rxe: Make responder support atomic write on RC service
>>    RDMA/rxe: Implement atomic write completion
>>    RDMA/rxe: Enable atomic write capability for rxe device
> 
> Applied to for-next, thanks
> 
> Jason

Hi Jason,

Cool, Thank you very much for reviewing and applying the patch set.
Besides, I have removed the rdma_atomic_write example as you suggested 
on rdma-core repo.

Best Regards,
Xiao Yang