[PATCH RFC] Verbs requirement for GRH when working in GID based addressing

[PATCH RFC] Verbs requirement for GRH when working in GID based addressing

[Alex Rosenbaum]

RDMA applications require an indication they are operating in an
environment which is based on GID addressing, such as SR-IOV IB
Virtualization. In this mode, all sent traffic must include a GRH.

This RFC exposes a new verbs port capability flag: GRH_REQUIRED.
When GRH_REQUIRED port capability flag is set, the applications must
create all AH with GRH configured. Meaning calling ibv_create_ah() with
'struct ibv_ah_attr { is_global = 1 }'

Signed-off-by: Alex Rosenbaum <alexr-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 libibverbs/man/ibv_create_ah.3 | 4 ++++
 libibverbs/verbs.h             | 3 ++-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/libibverbs/man/ibv_create_ah.3 b/libibverbs/man/ibv_create_ah.3
index 0ca2198..b645ea3 100644
--- a/libibverbs/man/ibv_create_ah.3
+++ b/libibverbs/man/ibv_create_ah.3
@@ -53,6 +53,10 @@ destroys the AH
 .SH "RETURN VALUE"
 .B ibv_create_ah()
 returns a pointer to the created AH, or NULL if the request fails.
+.SH "NOTES"
+If port cap flag IBV_PORT_GRH_REQUIRED is set then
+.B ibv_create_ah()
+must be created with definition of 'struct ibv_ah_attr { .is_global =
1; .grh = {...}; }'.
 .PP
 .B ibv_destroy_ah()
 returns 0 on success, or the value of errno on failure (which
indicates the failure reason).
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index 997597a..12e80b3 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -338,7 +338,8 @@ enum ibv_port_cap_flags {
        IBV_PORT_BOOT_MGMT_SUP                  = 1 << 23,
        IBV_PORT_LINK_LATENCY_SUP               = 1 << 24,
        IBV_PORT_CLIENT_REG_SUP                 = 1 << 25,
-       IBV_PORT_IP_BASED_GIDS                  = 1 << 26
+       IBV_PORT_IP_BASED_GIDS                  = 1 << 26,
+       IBV_PORT_GRH_REQUIRED                   = 1 << 27
 };

 struct ibv_port_attr {
-- 
1.8.3.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC] Verbs requirement for GRH when working in GID based addressing

[Or Gerlitz]

On Wed, Jan 10, 2018 at 9:28 PM, Alex Rosenbaum <rosenbaumalex-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> RDMA applications require an indication they are operating in an
> environment which is based on GID addressing, such as SR-IOV IB
> Virtualization. In this mode, all sent traffic must include a GRH.

Hi Alex,

Isn't this the case for RoCE since it's day one? how does it work there? maybe
a hop limit > 1 is used and the kernel rdma stack tweaks it over RoCE ports?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RFC] Verbs requirement for GRH when working in GID based addressing

[Jason Gunthorpe]

On Wed, Jan 10, 2018 at 11:59:18PM +0200, Or Gerlitz wrote:
> On Wed, Jan 10, 2018 at 9:28 PM, Alex Rosenbaum <rosenbaumalex-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > RDMA applications require an indication they are operating in an
> > environment which is based on GID addressing, such as SR-IOV IB
> > Virtualization. In this mode, all sent traffic must include a GRH.
> 
> Hi Alex,
> 
> Isn't this the case for RoCE since it's day one? how does it work there? maybe
> a hop limit > 1 is used and the kernel rdma stack tweaks it over RoCE ports?

roce requires GRH because it is roce. Everything should just use
that as the criteria.

The hop limit trick is only for IB and it is only for path records
from the SA. It allows the SA to say 'a GRH is required for this path'
even if the path has local GIDs.

The SA *should* be returning hop limit > 1 for any GID that has
grh_required=1 in this patch, otherwise CM won't work right if the
node is a target.

Presumably the attr is required for non SA based schemes?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html