[PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Or Gerlitz]

enforce local binding is specified for unmapped multicast addresses, otherwise mckey
crashes when attempting to use the cma_id->verbs pointer in the port query verb.

Signed-off-by: Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org>

Sean, using unmapped multicast addresses I see that a different broacast group is
created by the SM such that mckey doesn't manage to join the ipv4 broadcast group

$ ./mckey -M ff12:401b:ffff:0:0:0:ffff:ffff -b 10.10.5.62 -p 0x2

mckey: joined dgid: ff12:401b:ffff:: mlid c00b sl 0

looking in the SA, I see that the MGID used by the rdma-cm is a bif different
from the one used by IPoIB, since the former uses/set only the lower 28 bits where
the latter sets the lower 32 bits for this mgid, any idea what can be  done here?

$ saquery $THIS_NODE_LID

MCMemberRecord group dump:
                MGID....................ff12:401b:ffff::ffff:ffff
                Mlid....................0xC000
                Mtu.....................0x84
                pkey....................0xFFFF
                Rate....................0x83
                SL......................0x0
....

MCMemberRecord group dump:
                MGID....................ff12:401b:ffff::fff:ffff
                Mlid....................0xC00B
                Mtu.....................0x84
                pkey....................0xFFFF
                Rate....................0x83
                SL......................0x0


Index: librdmacm/examples/mckey.c
===================================================================
--- librdmacm.orig/examples/mckey.c
+++ librdmacm/examples/mckey.c
@@ -273,7 +273,7 @@ static int join_handler(struct cmatest_n
 	char buf[40];

 	inet_ntop(AF_INET6, param->ah_attr.grh.dgid.raw, buf, 40);
-	printf("mckey: joined dgid: %s\n", buf);
+	printf("mckey: joined dgid: %s mlid %x sl %d\n", buf, param->ah_attr.dlid, param->ah_attr.sl);

 	node->remote_qpn = param->qp_num;
 	node->remote_qkey = param->qkey;
@@ -556,6 +556,11 @@ int main(int argc, char **argv)
 		}
 	}

+	if (unmapped_addr && !src_addr) {
+		printf("unmapped multicast address requires binding to source address\n");
+		exit(1);
+	}
+
 	test.dst_addr = (struct sockaddr *) &test.dst_in;
 	test.connects_left = connections;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Sean Hefty]

>Sean, using unmapped multicast addresses I see that a different broacast group
>is
>created by the SM such that mckey doesn't manage to join the ipv4 broadcast
>group
>
>$ ./mckey -M ff12:401b:ffff:0:0:0:ffff:ffff -b 10.10.5.62 -p 0x2

Unmapped multicast groups only support the case where the SA has created the
group with the MGID undefined.  The MGID must be in this format:

0xff1 scope 0xA01B

(see figure 196 on page 928 of the spec).  The kernel checks for this specific
address format to see if it needs to convert the address or not.  (The original
feature request for this came from the US national labs, which wanted the
ability to create a group a get back a unique group ID.)

>the latter sets the lower 32 bits for this mgid, any idea what can be  done
>here?

I thought about this, and once support for AF_IB is added, then the format of
the address becomes clear and the full range of unmapped addresses becomes
available.

I'll add your patch into my git tree - thanks.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Jason Gunthorpe]

On Sun, Nov 01, 2009 at 11:31:22AM +0200, Or Gerlitz wrote:

> Sean, using unmapped multicast addresses I see that a different broacast group is
> created by the SM such that mckey doesn't manage to join the ipv4 broadcast group
> 
> $ ./mckey -M ff12:401b:ffff:0:0:0:ffff:ffff -b 10.10.5.62 -p 0x2
> 
> mckey: joined dgid: ff12:401b:ffff:: mlid c00b sl 0

Erm, I'm not sure what is going on by the time things get to the SA,
but the above output shows this kernel bug. The joined DGID for that
AF_INET6 address should be FF12:601b::FFFF:FFFF

The AF_INET6 -M argument to mckey is not treated as a MGID unless it is
prefixed with FF10:A01B:: (so ugly..)

If you want to join the IPv4 all hosts group I think you need to use
-M 255.255.255.255

Your result does show that something else is wrong too, the group with
MLID 0xC00B should have been MGID ff12:401b:ffff:: like mckey reported
..

>From 9f3a76deb5bfda0f8243eadfa024eb547c03f583 Mon Sep 17 00:00:00 2001
From: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
Date: Mon, 2 Nov 2009 11:23:38 -0700
Subject: [PATCH] RDMA CM: Fix AF_INET6 support in multicast joining

If joining to an AF_INET6 address we need to map the address to a MGID
in the same way as the IP stack. The old code would just fall through to
the IPv4 case and generate garbage.

Signed-off-by: Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
---
 drivers/infiniband/core/cma.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

compile tested only.

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index a0fa241..1e9a78a 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -2724,6 +2724,11 @@ static void cma_set_mgid(struct rdma_id_private *id_priv,
 								 0xFF10A01B)) {
 		/* IPv6 address is an SA assigned MGID. */
 		memcpy(mgid, &sin6->sin6_addr, sizeof *mgid);
+	} else if ((addr->sa_family == AF_INET6)) {
+		ipv6_ib_mc_map(&sin6->sin6_addr, dev_addr->broadcast, mc_map);
+		if (id_priv->id.ps == RDMA_PS_UDP)
+			mc_map[7] = 0x01;	/* Use RDMA CM signature */
+		*mgid = *(union ib_gid *) (mc_map + 4);
 	} else {
 		ip_ib_mc_map(sin->sin_addr.s_addr, dev_addr->broadcast, mc_map);
 		if (id_priv->id.ps == RDMA_PS_UDP)
-- 
1.5.4.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Or Gerlitz]

Sean Hefty wrote:
> Unmapped multicast groups only support the case where the SA has created the
> group with the MGID undefined.  The MGID must be in this format: 0xff1 scope 0xA01B (see figure 196 on page 928 of the spec).  The kernel checks for this specific address format to see if it needs to convert the address or not [...] wanted the ability to create a group a get back a unique group ID
I am still not sure to follow you. My basic thought was that unmapped 
multicast addresses are MGIDs specified by the application such that 
rdma-cm doesn't treat them as IPv6 multicast address and no mapping is 
applied on them. From the spec location you have pointed me I understand 
that the intention is for a request to the SA to generate a unique MGID:

1. "if SA receives a request to create a multicast group with the MGID 
undefined"
2.  "the MGID that it creates shall be of the following format"

so there are two parts here, 1st request the SA to create a new group, 
assign it an MGID (what about joining this node/port to the group), 2nd, 
getting back the MGID created by the SA. Looking on the rdma-cm kernel 
code, I don't see where/how it specifies to the SA  that the MGID is 
undefined? shouldn't it not set the MGID bit in the component mask in 
this case? next, I don't see where the MGID created by the SA is given 
back to the application. I guess still miss something here, can you 
clarify, thanks

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Sean Hefty]

>I am still not sure to follow you. My basic thought was that unmapped
>multicast addresses are MGIDs specified by the application such that
>rdma-cm doesn't treat them as IPv6 multicast address and no mapping is
>applied on them. 

This is basically correct, but the MGIDs must be of the format 0xFF1-scope-A01B.
The kernel keys off this in the address to determine if the mapping should be
made.

>1. "if SA receives a request to create a multicast group with the MGID
>undefined"
>2.  "the MGID that it creates shall be of the following format"
>
>so there are two parts here, 1st request the SA to create a new group,
>assign it an MGID (what about joining this node/port to the group), 2nd,
>getting back the MGID created by the SA. Looking on the rdma-cm kernel
>code, I don't see where/how it specifies to the SA  that the MGID is
>undefined? shouldn't it not set the MGID bit in the component mask in
>this case? next, I don't see where the MGID created by the SA is given

>From 15.2.5.17.2:

"MGID undefined" means either the MGID bit in the component mask is not set, or
if the MGID bit in the component mask is set, the MGID component of
MCMemberRecord:MGID is 0.

If the join request is for an address of 0, then the MGID will be 0 in the SA
join request.  The SA will assign an MGID starting with 0xFF1-scope-A01B.  When
the join request completes, the MCMemberRecord information is given to the user
as part of the multicast join event in the form of address handle attributes.
(See cma_ib_mc_handler.)

What's missing is Jason's patch to fix the IPv6 mapping, and a way to extend the
rdma_cm to support the full range of unmapped addresses.  I just haven't been
able to get to either of these yet.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Jason Gunthorpe]

On Tue, Nov 03, 2009 at 08:43:01AM -0800, Sean Hefty wrote:

> What's missing is Jason's patch to fix the IPv6 mapping, and a way to extend the
> rdma_cm to support the full range of unmapped addresses.  I just haven't been
> able to get to either of these yet.

My feeling is when AF_IB is introduced the hacky overload of AF_INET6
should be deprecated and MGIDs must be passed in as AF_IB - and some
ancillary optional data to set the pkey, rate, etc for the join request.

That way a 0 AF_IB MGID would pass straight through to the SA, and
come back as a unique SA assigned address.

This fills in an API hole, there is no way to join a MGID with kernel
help to refcount and unjoin.

BTW, the kernel test for 0xFF1-scope-A01B is busted:

(s6_addr32[0]) & 0xFF10A01B) == 0xFF10A01B

I was wondering what that was about. Should have been

(s6_addr32[0]) & 0xFFF0FFFF) == 0xFF10A01B

Right?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Sean Hefty]

>My feeling is when AF_IB is introduced the hacky overload of AF_INET6
>should be deprecated and MGIDs must be passed in as AF_IB - and some
>ancillary optional data to set the pkey, rate, etc for the join request.

Yes, I expect AF_IB to fix this.

>BTW, the kernel test for 0xFF1-scope-A01B is busted:
>
>(s6_addr32[0]) & 0xFF10A01B) == 0xFF10A01B
>
>I was wondering what that was about. Should have been
>
>(s6_addr32[0]) & 0xFFF0FFFF) == 0xFF10A01B
>
>Right?

The check is wrong and should match the latter.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Or Gerlitz]

Sean Hefty wrote:
> This is basically correct, but the MGIDs must be of the format 0xFF1-scope-A01B.
> The kernel keys off this in the address to determine if the mapping should be made.

okay

> From 15.2.5.17.2:
> "MGID undefined" means either the MGID bit in the component mask is not set, or
> if the MGID bit in the component mask is set, the MGID component of MCMemberRecord:MGID is 0

okay, I was confused between generating an SA assigned MGID to using it as unmapped multicast address with mckey, lets document how to do the generation

Or.


document how SA assigned MGIDs can be created via mckey

Signed-off-by: Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org>

Index: librdmacm/man/mckey.1
===================================================================
--- librdmacm.orig/man/mckey.1
+++ librdmacm/man/mckey.1
@@ -47,6 +47,9 @@ UDP (0x0111) or IPoIB (0x0002) port spac
 Basic usage is to start mckey -m multicast_address on a server system,
 then run mckey -m multicast_address -s on a client system.
 .P
+Unique SA assigned multicast GIDs can be retreived by invoking mckey
+with a zero MGID using the unmapped option, i.e  -M ::0
+.P
 Because this test maps RDMA resources to userspace, users must ensure
 that they have available system resources and permissions.  See the
 libibverbs README file for additional details.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Sean Hefty]

>document how SA assigned MGIDs can be created via mckey
>
>Signed-off-by: Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org>

Thanks - I merged this with your other patch to mckey and applied them to my
tree.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Or Gerlitz]

Sean Hefty wrote:
> I merged this with your other patch to mckey and applied them to my tree
>   
I don't see this @ 
http://www.openfabrics.org/git/?p=~shefty/librdmacm.git, were you 
referring a local clone?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] librdmacm/mckey: enforce local binding for unmapped multicast addresses

[Hefty, Sean]

>I don't see this @
>http://www.openfabrics.org/git/?p=~shefty/librdmacm.git, were you
>referring a local clone?

Yes - I just haven't pushed it back upstream yet.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html