[PATCH] Provider/rxe: Fix regression to UD traffic

[PATCH] Provider/rxe: Fix regression to UD traffic

[Bob Pearson]

Update enum rdma_network_type copy to match kernel version.
Without this change provider/rxe will send incorrect
network types to the kernel in send WQEs.

This fix keeps rxe functional but should be replaced by a better
implementation.

Signed-off-by: Bob Pearson <rpearson@hpe.com>
---
 providers/rxe/rxe.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/providers/rxe/rxe.h b/providers/rxe/rxe.h
index 96f4ee9c..46e6ce72 100644
--- a/providers/rxe/rxe.h
+++ b/providers/rxe/rxe.h
@@ -44,6 +44,7 @@
 
 enum rdma_network_type {
 	RDMA_NETWORK_IB,
+	RDMA_NETWORK_ROCE_V1,
 	RDMA_NETWORK_IPV4,
 	RDMA_NETWORK_IPV6
 };
-- 
2.25.1

Re: [PATCH] Provider/rxe: Fix regression to UD traffic

[Jason Gunthorpe]

On Thu, Oct 15, 2020 at 03:17:51PM -0500, Bob Pearson wrote:
> Update enum rdma_network_type copy to match kernel version.
> Without this change provider/rxe will send incorrect
> network types to the kernel in send WQEs.
> 
> This fix keeps rxe functional but should be replaced by a better
> implementation.
> 
> Signed-off-by: Bob Pearson <rpearson@hpe.com>
> ---
>  providers/rxe/rxe.h | 1 +
>  1 file changed, 1 insertion(+)

Well, we can't just break user space so the kernel has to change in
some way to accommodate this.

Obviously rxe should not have uAPI stuff that is not in
include/uapi/rdma, so lets just fix that directly.

Please confirm I did this right. The PR for this merge window must be
sent Friday.

From 8b20e1ceea413c98cb98930cb549be390226320f Mon Sep 17 00:00:00 2001
From: Jason Gunthorpe <jgg@nvidia.com>
Date: Thu, 15 Oct 2020 20:42:18 -0300
Subject: [PATCH] RDMA/rxe: Move the definitions for rxe_av.network_type to
 uAPI

RXE was wrongly using an internal kernel enum as part of its uAPI, split
this out into a dedicated uAPI enum just for RXE. It only uses the IPv4
and IPv6 values.

This was exposed by changing the internal kernel enum definition which
broke RXE.

Fixes: 1c15b4f2a42f ("RDMA/core: Modify enum ib_gid_type and enum rdma_network_type")
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---
 drivers/infiniband/sw/rxe/rxe_net.c | 8 ++++----
 include/uapi/rdma/rdma_user_rxe.h   | 6 ++++++
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c
index 31b93e7e1e2f41..575e1a4ec82121 100644
--- a/drivers/infiniband/sw/rxe/rxe_net.c
+++ b/drivers/infiniband/sw/rxe/rxe_net.c
@@ -133,14 +133,14 @@ static struct dst_entry *rxe_find_route(struct net_device *ndev,
 		if (dst)
 			dst_release(dst);
 
-		if (av->network_type == RDMA_NETWORK_IPV4) {
+		if (av->network_type == RXE_NETWORK_TYPE_IPV4) {
 			struct in_addr *saddr;
 			struct in_addr *daddr;
 
 			saddr = &av->sgid_addr._sockaddr_in.sin_addr;
 			daddr = &av->dgid_addr._sockaddr_in.sin_addr;
 			dst = rxe_find_route4(ndev, saddr, daddr);
-		} else if (av->network_type == RDMA_NETWORK_IPV6) {
+		} else if (av->network_type == RXE_NETWORK_TYPE_IPV6) {
 			struct in6_addr *saddr6;
 			struct in6_addr *daddr6;
 
@@ -442,7 +442,7 @@ struct sk_buff *rxe_init_packet(struct rxe_dev *rxe, struct rxe_av *av,
 	if (IS_ERR(attr))
 		return NULL;
 
-	if (av->network_type == RDMA_NETWORK_IPV4)
+	if (av->network_type == RXE_NETWORK_TYPE_IPV6)
 		hdr_len = ETH_HLEN + sizeof(struct udphdr) +
 			sizeof(struct iphdr);
 	else
@@ -469,7 +469,7 @@ struct sk_buff *rxe_init_packet(struct rxe_dev *rxe, struct rxe_av *av,
 	skb->dev	= ndev;
 	rcu_read_unlock();
 
-	if (av->network_type == RDMA_NETWORK_IPV4)
+	if (av->network_type == RXE_NETWORK_TYPE_IPV4)
 		skb->protocol = htons(ETH_P_IP);
 	else
 		skb->protocol = htons(ETH_P_IPV6);
diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h
index d8f2e0e46daba7..e591d8c1f3cf10 100644
--- a/include/uapi/rdma/rdma_user_rxe.h
+++ b/include/uapi/rdma/rdma_user_rxe.h
@@ -39,6 +39,11 @@
 #include <linux/in.h>
 #include <linux/in6.h>
 
+enum {
+	RXE_NETWORK_TYPE_IPV4 = 1,
+	RXE_NETWORK_TYPE_IPV6 = 2,
+};
+
 union rxe_gid {
 	__u8	raw[16];
 	struct {
@@ -57,6 +62,7 @@ struct rxe_global_route {
 
 struct rxe_av {
 	__u8			port_num;
+	/* From RXE_NETWORK_TYPE_* */
 	__u8			network_type;
 	__u8			dmac[6];
 	struct rxe_global_route	grh;
-- 
2.28.0

Re: [PATCH] Provider/rxe: Fix regression to UD traffic

[Bob Pearson]

On 10/15/20 6:47 PM, Jason Gunthorpe wrote:
> On Thu, Oct 15, 2020 at 03:17:51PM -0500, Bob Pearson wrote:
>> Update enum rdma_network_type copy to match kernel version.
>> Without this change provider/rxe will send incorrect
>> network types to the kernel in send WQEs.
>>
>> This fix keeps rxe functional but should be replaced by a better
>> implementation.
>>
>> Signed-off-by: Bob Pearson <rpearson@hpe.com>
>> ---
>>  providers/rxe/rxe.h | 1 +
>>  1 file changed, 1 insertion(+)
> 
> Well, we can't just break user space so the kernel has to change in
> some way to accommodate this.
> 
> Obviously rxe should not have uAPI stuff that is not in
> include/uapi/rdma, so lets just fix that directly.
> 
> Please confirm I did this right. The PR for this merge window must be
> sent Friday.

That works too. I am figuring out how to convert ah_handle to ib_ah in rdma_rxe. I think I've mostly got it.
uobj_get_obj_read(type, id, attr) is close to what I need. The problem is that all the code in this area assumes
it is getting called in a user verb API call so there is an attr to pass around. What we need is an API like
rdma_get_obj(ucontext, type, id) since we need a way to get to the user api data structures. Better would be
a pointer from qp to ucontext or ufile that would be set for user QPs. If we did this rxe just use handles from
the core instead of creating its own databases of objects.

Your patch will get us going for now. The above is a longer project that won't fit in your short term schedule.

Bob