[PATCH] IB/mlx4: Allow to always block UD multicast loopback

[PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

We need this option for many hosts to avoid backflow of multicast
packages. Could we get that merged?



>From 18ceae090b02b3055382e11c305dcb334d938122 Mon Sep 17 00:00:00 2001
From: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Date: Tue, 4 Mar 2014 17:20:00 +0200
Subject: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

Currently, there's no way for user-space applications to specify
the IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK QP creation flags defined
by commit 47ee1b9 "IB/core: Add support for multicast loopback blocking".

As a result, applications who send and recieve on the same QP over
the same multicast group get all their packets bouncded back to them,
which is bad..

Add a module param to the mlx4_ib driver which allows to force all QP
attachements to be carried out with multicast loopback prevented.

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/hw/mlx4/main.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index e81c554..fa01a9e 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -68,6 +68,10 @@ int mlx4_ib_sm_guid_assign = 1;
 module_param_named(sm_guid_assign, mlx4_ib_sm_guid_assign, int, 0444);
 MODULE_PARM_DESC(sm_guid_assign, "Enable SM alias_GUID assignment if sm_guid_assign > 0 (Default: 1)");

+int mlx4_block_loopback = 0;
+module_param_named(block_loopback, mlx4_block_loopback, int, 0644);
+MODULE_PARM_DESC(block_loopback, "Always block UD multicast loopback packets if > 0");
+
 static const char mlx4_ib_version[] =
 	DRV_NAME ": Mellanox ConnectX InfiniBand driver v"
 	DRV_VERSION " (" DRV_RELDATE ")\n";
@@ -1163,6 +1167,8 @@ static int mlx4_ib_mcg_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
 	struct mlx4_ib_steering *ib_steering = NULL;
 	enum mlx4_protocol prot = (gid->raw[1] == 0x0e) ?
 		MLX4_PROT_IB_IPV4 : MLX4_PROT_IB_IPV6;
+	int block_mcast_loopback = (!!(mqp->flags & MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK)) |
+				   mlx4_block_loopback;

 	if (mdev->dev->caps.steering_mode ==
 	    MLX4_STEERING_MODE_DEVICE_MANAGED) {
@@ -1172,8 +1178,7 @@ static int mlx4_ib_mcg_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
 	}

 	err = mlx4_multicast_attach(mdev->dev, &mqp->mqp, gid->raw, mqp->port,
-				    !!(mqp->flags &
-				       MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK),
+				    block_mcast_loopback,
 				    prot, &reg_id);
 	if (err)
 		goto err_malloc;
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Or Gerlitz]

On Mon, Apr 21, 2014 at 9:09 PM, Christoph Lameter <cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org> wrote:
> We need this option for many hosts to avoid backflow of multicast
> packages. Could we get that merged?
[...]

> Currently, there's no way for user-space applications to specify
> the IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK QP creation flags
> defined by commit 47ee1b9 "IB/core: Add support for multicast loopback blocking".
> As a result, applications who send and recieve on the same QP over
> the same multicast group get all their packets bouncded back to them,
> which is bad..

Hi Christoph, I do see your P&F (Pain and Frustration), however, this
patch was only meant to be bandaid (*) for the fact that whole ~six
years elapsed since a basically proper patch that does the task was
posted http://lists.openfabrics.org/pipermail/general/2008-July/053021.html
and we still don't have uverbs support for QP creation flags!!

Roland, how you prefer to move on here? we can come up with quick
patch that just uses the reserved field of struct ib_uverbs_create_qp
and would allow to support upto 8 creation flags, this would not
require new uverbs command, OK?

Roland, we must fix this lack of user space to provide creation flags.
The bandaid I did after a whole production system was successfully
rebased to the upstream kernel sans this PITA which I has to solve
quickly in this uglish way.

Or.

(*) this is just bandaid and not for upstream since it touches just
one HW driver, i.e a similar patch you would need for every other HW
driver, and deals with only one QP creation flag.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

On Mon, 21 Apr 2014, Or Gerlitz wrote:

> > Currently, there's no way for user-space applications to specify
> > the IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK QP creation flags
> > defined by commit 47ee1b9 "IB/core: Add support for multicast loopback blocking".
> > As a result, applications who send and recieve on the same QP over
> > the same multicast group get all their packets bouncded back to them,
> > which is bad..
>
> Hi Christoph, I do see your P&F (Pain and Frustration), however, this
> patch was only meant to be bandaid (*) for the fact that whole ~six
> years elapsed since a basically proper patch that does the task was
> posted http://lists.openfabrics.org/pipermail/general/2008-July/053021.html
> and we still don't have uverbs support for QP creation flags!!
>
> Roland, how you prefer to move on here? we can come up with quick
> patch that just uses the reserved field of struct ib_uverbs_create_qp
> and would allow to support upto 8 creation flags, this would not
> require new uverbs command, OK?

Urgh. So we have all these flags and we cannot use them?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Or Gerlitz]

On Tue, Apr 22, 2014 at 12:06 AM, Christoph Lameter <cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org> wrote:
> On Mon, 21 Apr 2014, Or Gerlitz wrote:
>
>> > Currently, there's no way for user-space applications to specify
>> > the IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK QP creation flags
>> > defined by commit 47ee1b9 "IB/core: Add support for multicast loopback blocking".
>> > As a result, applications who send and recieve on the same QP over
>> > the same multicast group get all their packets bouncded back to them,
>> > which is bad..
>>
>> Hi Christoph, I do see your P&F (Pain and Frustration), however, this
>> patch was only meant to be bandaid (*) for the fact that whole ~six
>> years elapsed since a basically proper patch that does the task was
>> posted http://lists.openfabrics.org/pipermail/general/2008-July/053021.html
>> and we still don't have uverbs support for QP creation flags!!
>>
>> Roland, how you prefer to move on here? we can come up with quick
>> patch that just uses the reserved field of struct ib_uverbs_create_qp
>> and would allow to support upto 8 creation flags, this would not
>> require new uverbs command, OK?
>
> Urgh. So we have all these flags and we cannot use them?

Christoph, the support (kernel IB core API and HW drivers e.g mlx4 and
now also mlx5) is there for **kernel** consumers (e.g IPoIB). and
missing for **user-space** consumers whose calls are tunneled through
uverbs. The uverbs piece is the missing one.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

On Tue, 22 Apr 2014, Or Gerlitz wrote:

> > Urgh. So we have all these flags and we cannot use them?
>
> Christoph, the support (kernel IB core API and HW drivers e.g mlx4 and
> now also mlx5) is there for **kernel** consumers (e.g IPoIB). and
> missing for **user-space** consumers whose calls are tunneled through
> uverbs. The uverbs piece is the missing one.

Could you please post an updated patch? We will test.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

Note that I am pretty unhappy about Mellanox right now. 8 year old patches
that have never been refreshed? If I submit patchsets then I will post
these things regularly until they are accepted. You need to have a patch
queue or a git tree with up to date patches that contains all the patches
you intend to merge.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Or Gerlitz]

On 22/04/2014 16:48, Christoph Lameter wrote:
> Note that I am pretty unhappy about Mellanox right now. 8 year old patches
> that have never been refreshed? If I submit patchsets then I will post
> these things regularly until they are accepted. You need to have a patch
> queue or a git tree with up to date patches that contains all the patches
> you intend to merge.

Chritoph, again I can hear your Pain and Frustration (BTW the patch is 
from 2008 so only six years old...), when patches are repeatedly 
ignored, there's no much point in reposting them again and again. We are 
making sure that large patche sets are posted in a public git tree, for 
example the T10 signature patches were on the floor for 5-6 months and 
eventually went into 3.15 and during that hang time, we have put them in 
public git tree, I will make sure we do it right away for the ODP 
patches and other user space patches (RoCE UD and Flow-steering) that 
were posted here and still not accepted.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

On Tue, 22 Apr 2014, Or Gerlitz wrote:

> Chritoph, again I can hear your Pain and Frustration (BTW the patch is from
> 2008 so only six years old...), when patches are repeatedly ignored, there's
> no much point in reposting them again and again. We are making sure that large
> patche sets are posted in a public git tree, for example the T10 signature
> patches were on the floor for 5-6 months and eventually went into 3.15 and
> during that hang time, we have put them in public git tree, I will make sure
> we do it right away for the ODP patches and other user space patches (RoCE UD
> and Flow-steering) that were posted here and still not accepted.

Ok I saw the ODP patches. But where are the multicast block patches? Is
there any way to move this forward?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Or Gerlitz]

On Tue, Apr 29, 2014 at 8:32 PM, Christoph Lameter <cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org> wrote:

> Ok I saw the ODP patches. But where are the multicast block patches? Is
> there any way to move this forward?

We can try, earlier on this thread I wrote

"Roland, how you prefer to move on here? we can come up with quick
patch that just uses the reserved field of struct ib_uverbs_create_qp
and would allow to support upto 8 creation flags, this would not
require new uverbs command, OK?"

and no reply...

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

On Wed, 30 Apr 2014, Or Gerlitz wrote:

> On Tue, Apr 29, 2014 at 8:32 PM, Christoph Lameter <cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org> wrote:
>
> > Ok I saw the ODP patches. But where are the multicast block patches? Is
> > there any way to move this forward?
>
> We can try, earlier on this thread I wrote
>
> "Roland, how you prefer to move on here? we can come up with quick
> patch that just uses the reserved field of struct ib_uverbs_create_qp
> and would allow to support upto 8 creation flags, this would not
> require new uverbs command, OK?"

Post the patch please?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Doug Ledford]

----- Original Message -----
> We need this option for many hosts to avoid backflow of multicast
> packages. Could we get that merged?
> 
> 
> 
> From 18ceae090b02b3055382e11c305dcb334d938122 Mon Sep 17 00:00:00
> 2001
> From: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Date: Tue, 4 Mar 2014 17:20:00 +0200
> Subject: [PATCH] IB/mlx4: Allow to always block UD multicast loopback
> 
> Currently, there's no way for user-space applications to specify
> the IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK QP creation flags defined
> by commit 47ee1b9 "IB/core: Add support for multicast loopback
> blocking".
> 
> As a result, applications who send and recieve on the same QP over
> the same multicast group get all their packets bouncded back to them,
> which is bad..
> 
> Add a module param to the mlx4_ib driver which allows to force all QP
> attachements to be carried out with multicast loopback prevented.

This seems like driving a thumb tack with a sledge hammer.  No doubt
the correct path here is to get the proper flag added to verbs multicast
attach command.  Or, do you still have a patch that does this the
right way?  Did the one from 2008 do it the right way or this way?

> Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> ---
>  drivers/infiniband/hw/mlx4/main.c |    9 +++++++--
>  1 files changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/infiniband/hw/mlx4/main.c
> b/drivers/infiniband/hw/mlx4/main.c
> index e81c554..fa01a9e 100644
> --- a/drivers/infiniband/hw/mlx4/main.c
> +++ b/drivers/infiniband/hw/mlx4/main.c
> @@ -68,6 +68,10 @@ int mlx4_ib_sm_guid_assign = 1;
>  module_param_named(sm_guid_assign, mlx4_ib_sm_guid_assign, int,
>  0444);
>  MODULE_PARM_DESC(sm_guid_assign, "Enable SM alias_GUID assignment if
>  sm_guid_assign > 0 (Default: 1)");
> 
> +int mlx4_block_loopback = 0;
> +module_param_named(block_loopback, mlx4_block_loopback, int, 0644);
> +MODULE_PARM_DESC(block_loopback, "Always block UD multicast loopback
> packets if > 0");
> +
>  static const char mlx4_ib_version[] =
>  	DRV_NAME ": Mellanox ConnectX InfiniBand driver v"
>  	DRV_VERSION " (" DRV_RELDATE ")\n";
> @@ -1163,6 +1167,8 @@ static int mlx4_ib_mcg_attach(struct ib_qp
> *ibqp, union ib_gid *gid, u16 lid)
>  	struct mlx4_ib_steering *ib_steering = NULL;
>  	enum mlx4_protocol prot = (gid->raw[1] == 0x0e) ?
>  		MLX4_PROT_IB_IPV4 : MLX4_PROT_IB_IPV6;
> +	int block_mcast_loopback = (!!(mqp->flags &
> MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK)) |
> +				   mlx4_block_loopback;
> 
>  	if (mdev->dev->caps.steering_mode ==
>  	    MLX4_STEERING_MODE_DEVICE_MANAGED) {
> @@ -1172,8 +1178,7 @@ static int mlx4_ib_mcg_attach(struct ib_qp
> *ibqp, union ib_gid *gid, u16 lid)
>  	}
> 
>  	err = mlx4_multicast_attach(mdev->dev, &mqp->mqp, gid->raw,
>  	mqp->port,
> -				    !!(mqp->flags &
> -				       MLX4_IB_QP_BLOCK_MULTICAST_LOOPBACK),
> +				    block_mcast_loopback,
>  				    prot, &reg_id);
>  	if (err)
>  		goto err_malloc;
> --
> 1.7.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Or Gerlitz]

On 01/05/2014 02:28, Doug Ledford wrote:
> This seems like driving a thumb tack with a sledge hammer.  No doubt
> the correct path here is to get the proper flag added to verbs multicast
> attach command.  Or, do you still have a patch that does this the
> right way?  Did the one from 2008 do it the right way or this way?

Doug, as I commented earlier on this thread, the correct way here is to 
carry out the task is to repeat the QP creation flags which we have in 
the kernel verbs API for years (see 
IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK in ib_verbs.h and usage in IPoIB) 
for the user space verbs API. The patch I provided to Christophand CO 
was a band-aid to bring up a whole system on upstream and mlx4 where 
this (avoiding multicast loopback) was the only/major missing piece.

The patch posted back in 2008 is proper, but it added a whole new uverb 
call/structure, where I think we can get along here with just re-using a 
reserved field in the existing uverbs API, see my other post on this thread.

Or.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Doug Ledford]

----- Original Message -----
> On 01/05/2014 02:28, Doug Ledford wrote:
> > This seems like driving a thumb tack with a sledge hammer.  No
> > doubt
> > the correct path here is to get the proper flag added to verbs
> > multicast
> > attach command.  Or, do you still have a patch that does this the
> > right way?  Did the one from 2008 do it the right way or this way?
> 
> Doug, as I commented earlier on this thread, the correct way here is
> to
> carry out the task is to repeat the QP creation flags which we have
> in
> the kernel verbs API for years (see
> IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK in ib_verbs.h and usage in
> IPoIB)

I'm curious why this is done at QP creation time.  Should this not
be done at multicast join time?  A single QP can be a member of more
than one MC group, would it not seem more correct to you that the
loopback should be specified on a per group basis and not a per QP
basis?

> for the user space verbs API. The patch I provided to Christophand CO
> was a band-aid to bring up a whole system on upstream and mlx4 where
> this (avoiding multicast loopback) was the only/major missing piece.
> 
> The patch posted back in 2008 is proper, but it added a whole new
> uverb
> call/structure, where I think we can get along here with just
> re-using a
> reserved field in the existing uverbs API, see my other post on this
> thread.
> 
> Or.
> 
> Or.
> 

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Hefty, Sean]

> I'm curious why this is done at QP creation time.  Should this not
> be done at multicast join time?  A single QP can be a member of more
> than one MC group, would it not seem more correct to you that the
> loopback should be specified on a per group basis and not a per QP
> basis?

I agree with Doug here.  This makes more sense to specify on multicast attach time, not QP creation time. 

RE: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

??

On Mon, 5 May 2014, Hefty, Sean wrote:

> PiBJJ20gY3VyaW91cyB3aHkgdGhpcyBpcyBkb25lIGF0IFFQIGNyZWF0aW9uIHRpbWUuICBTaG91
> bGQgdGhpcyBub3QNCj4gYmUgZG9uZSBhdCBtdWx0aWNhc3Qgam9pbiB0aW1lPyAgQSBzaW5nbGUg
> UVAgY2FuIGJlIGEgbWVtYmVyIG9mIG1vcmUNCj4gdGhhbiBvbmUgTUMgZ3JvdXAsIHdvdWxkIGl0
> IG5vdCBzZWVtIG1vcmUgY29ycmVjdCB0byB5b3UgdGhhdCB0aGUNCj4gbG9vcGJhY2sgc2hvdWxk
> IGJlIHNwZWNpZmllZCBvbiBhIHBlciBncm91cCBiYXNpcyBhbmQgbm90IGEgcGVyIFFQDQo+IGJh
> c2lzPw0KDQpJIGFncmVlIHdpdGggRG91ZyBoZXJlLiAgVGhpcyBtYWtlcyBtb3JlIHNlbnNlIHRv
> IHNwZWNpZnkgb24gbXVsdGljYXN0IGF0dGFjaCB0aW1lLCBub3QgUVAgY3JlYXRpb24gdGltZS4g
> DQo=
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Hefty, Sean]

I have no earthly idea what this is or where it came from.  It doesn't show up in my sent items folder as this.

> PiBJJ20gY3VyaW91cyB3aHkgdGhpcyBpcyBkb25lIGF0IFFQIGNyZWF0aW9uIHRpbWUuICBTaG9
> 1
> >
> bGQgdGhpcyBub3QNCj4gYmUgZG9uZSBhdCBtdWx0aWNhc3Qgam9pbiB0aW1lPyAgQSBzaW5nbGU
> g
> >
> UVAgY2FuIGJlIGEgbWVtYmVyIG9mIG1vcmUNCj4gdGhhbiBvbmUgTUMgZ3JvdXAsIHdvdWxkIGl
> 0
> >
> IG5vdCBzZWVtIG1vcmUgY29ycmVjdCB0byB5b3UgdGhhdCB0aGUNCj4gbG9vcGJhY2sgc2hvdWx
> k
> >
> IGJlIHNwZWNpZmllZCBvbiBhIHBlciBncm91cCBiYXNpcyBhbmQgbm90IGEgcGVyIFFQDQo+IGJ
> h
> >
> c2lzPw0KDQpJIGFncmVlIHdpdGggRG91ZyBoZXJlLiAgVGhpcyBtYWtlcyBtb3JlIHNlbnNlIHR
> v
> >
> IHNwZWNpZnkgb24gbXVsdGljYXN0IGF0dGFjaCB0aW1lLCBub3QgUVAgY3JlYXRpb24gdGltZS4
> g
> > DQo=
> >

What I said was:

I agree with Doug here.  This makes more sense to specify on multicast attach time, not QP creation time.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

On Mon, 5 May 2014, Hefty, Sean wrote:

> I agree with Doug here.  This makes more sense to specify on multicast atta=
> ch time, not QP creation time.

I read somewhere that some android apps have a bug an do not properly set
the fields in the header when sendiung a base 64 encoded message.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

On Mon, 5 May 2014, Hefty, Sean wrote:

> I agree with Doug here.  This makes more sense to specify on multicast atta=
> ch time, not QP creation time.

On topic response: Do devices support multicast loopback blocking on a
multicast group? From what I can tell the multicast support in NICs is
based on having a table with MAC addresses of the multicast groups that
the NIC is able to receive.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Doug Ledford]

----- Original Message -----
> On Mon, 5 May 2014, Hefty, Sean wrote:
> 
> > I agree with Doug here.  This makes more sense to specify on
> > multicast atta=
> > ch time, not QP creation time.
> 
> On topic response: Do devices support multicast loopback blocking on
> a
> multicast group?

I would think this is very doubtful...see below...

> From what I can tell the multicast support in NICs
> is
> based on having a table with MAC addresses of the multicast groups
> that
> the NIC is able to receive.

That table only tells the NIC to listen to specific multicast
addresses on the wire.  This is roughly equivalent to telling
the SM to subscribe the port to the multicast groups it needs
subscribed too.  In both cases, this merely gets the packets
to the card.  From there, the card (or the OS as is usually the
case on NICs and Ethernet multicast) must redistribute the
packet to all queue pairs that have subscribed to the group
that the packet was received from.  If you were to block it at
the group level, then it universally effect all applications
that subscribed to that group, and there might well be a number
of applications that did not request this behavior and would
be rightfully confused at the card/OS not sending them their
multicast packets.  So, I would suggest that the blocking of
loopback multicast packets needs to be "opt in" for all
applications.  The big hammer of blocking all loopback on an
entire card or an entire group, while possible, should be
highly discouraged.  It might work with limited applications
that know it is being done, but it can also lead to hard to
diagnose problems if you add a new application into the mix
and it is unaware of this hammer being used and unable to
handle the situation.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

On Tue, 6 May 2014, Doug Ledford wrote:

> That table only tells the NIC to listen to specific multicast
> addresses on the wire.  This is roughly equivalent to telling
> the SM to subscribe the port to the multicast groups it needs
> subscribed too.  In both cases, this merely gets the packets
> to the card.  From there, the card (or the OS as is usually the
> case on NICs and Ethernet multicast) must redistribute the
> packet to all queue pairs that have subscribed to the group
> that the packet was received from.  If you were to block it at
> the group level, then it universally effect all applications
> that subscribed to that group, and there might well be a number
> of applications that did not request this behavior and would
> be rightfully confused at the card/OS not sending them their
> multicast packets.  So, I would suggest that the blocking of
> loopback multicast packets needs to be "opt in" for all
> applications.  The big hammer of blocking all loopback on an
> entire card or an entire group, while possible, should be
> highly discouraged.  It might work with limited applications
> that know it is being done, but it can also lead to hard to
> diagnose problems if you add a new application into the mix
> and it is unaware of this hammer being used and unable to
> handle the situation.

Right the multicast blocking occurs at the socket level for regular
networking (see IP_MULTICAST_LOOP socket options). Socket are owned
by the application.


A QP is roughly the equivalent thing at the RDMA level. So it seems to me
that blocking needs to occur at the QP level and not at the multicast
group level as suggested by Sean.

The 2008 patch does exactly that and allow setting the multicast loopback
blocking from user space. Moreover the support is already there for in
kernel users. Could we just get that 2008 patch updated and merged?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Hefty, Sean]

> Right the multicast blocking occurs at the socket level for regular
> networking (see IP_MULTICAST_LOOP socket options). Socket are owned
> by the application.
> 
> 
> A QP is roughly the equivalent thing at the RDMA level. So it seems to me
> that blocking needs to occur at the QP level and not at the multicast
> group level as suggested by Sean.

I was unaware of the IP_MULTICAST_LOOP option.  Given that it makes sense to apply this at the QP level.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Doug Ledford]

----- Original Message -----
> On Tue, 6 May 2014, Doug Ledford wrote:
> 
> > That table only tells the NIC to listen to specific multicast
> > addresses on the wire.  This is roughly equivalent to telling
> > the SM to subscribe the port to the multicast groups it needs
> > subscribed too.  In both cases, this merely gets the packets
> > to the card.  From there, the card (or the OS as is usually the
> > case on NICs and Ethernet multicast) must redistribute the
> > packet to all queue pairs that have subscribed to the group
> > that the packet was received from.  If you were to block it at
> > the group level, then it universally effect all applications
> > that subscribed to that group, and there might well be a number
> > of applications that did not request this behavior and would
> > be rightfully confused at the card/OS not sending them their
> > multicast packets.  So, I would suggest that the blocking of
> > loopback multicast packets needs to be "opt in" for all
> > applications.  The big hammer of blocking all loopback on an
> > entire card or an entire group, while possible, should be
> > highly discouraged.  It might work with limited applications
> > that know it is being done, but it can also lead to hard to
> > diagnose problems if you add a new application into the mix
> > and it is unaware of this hammer being used and unable to
> > handle the situation.
> 
> Right the multicast blocking occurs at the socket level for regular
> networking (see IP_MULTICAST_LOOP socket options). Socket are owned
> by the application.
> 
> 
> A QP is roughly the equivalent thing at the RDMA level. So it seems
> to me
> that blocking needs to occur at the QP level and not at the multicast
> group level as suggested by Sean.

Nobody that I know of suggested that this should occur at the multicast
group level.  I suggested, and Sean agreed with, the idea that this
should  happen at multicast join time.  That means it would be on a
per queue pair, per multicast join basis.

Setting the IP_MULTICAST_LOOP on the socket effects all joins on that
socket, so this would be a slightly different API.

Right now, the IP stack is somewhat limited in that you would have
to create two different sockets and set IP_MULTICAST_LOOP differently
on the two sockets in order to have some joins reflect your data back
and some not.  You could sort of create what I'm talking about by
using the source address block, but that would block all applications
from your IP address, not just your own sent data, and so wouldn't
really work.

My original preference would have been to allow one queue pair to
have joins in either blocked or unblocked state, that way you would
only need one queue pair for both your reflected and non-reflected
joins.  But it would be easier to have a program capable of both
sockets and RDMA connections if we make the queue pairs follow the
sockets semantics here, so I'll withdraw my concerns over the current
patch (but also wouldn't object to it being done on a per queue
pair basis either, since we don't exactly follow socket semantics
either way given that the socket semantics separate the join and
the ioctl to control this behavior, so separating qp creation
and the setting of this behavior in the RDMA API seems reasonable
too).

> The 2008 patch does exactly that and allow setting the multicast
> loopback
> blocking from user space. Moreover the support is already there for
> in
> kernel users. Could we just get that 2008 patch updated and merged?
> 
> 

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

On Wed, 30 Apr 2014, Doug Ledford wrote:

> > Add a module param to the mlx4_ib driver which allows to force all QP
> > attachements to be carried out with multicast loopback prevented.
>
> This seems like driving a thumb tack with a sledge hammer.  No doubt
> the correct path here is to get the proper flag added to verbs multicast
> attach command.  Or, do you still have a patch that does this the
> right way?  Did the one from 2008 do it the right way or this way?

This is the least invasive approach to do what we need and this approach
has been a part of OFED for long years.

The 2008 approach would be cleaner but it depends on an infrastructure
change. I really would like to see that patch brought up to date and
discussed.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Doug Ledford]

----- Original Message -----
> On Wed, 30 Apr 2014, Doug Ledford wrote:
> 
> > > Add a module param to the mlx4_ib driver which allows to force
> > > all QP
> > > attachements to be carried out with multicast loopback prevented.
> >
> > This seems like driving a thumb tack with a sledge hammer.  No
> > doubt
> > the correct path here is to get the proper flag added to verbs
> > multicast
> > attach command.  Or, do you still have a patch that does this the
> > right way?  Did the one from 2008 do it the right way or this way?
> 
> This is the least invasive approach to do what we need and this
> approach
> has been a part of OFED for long years.

I get that it's non-invasive and when doing something to get around the
fact that it doesn't exist upstream, that makes sense.

> The 2008 approach would be cleaner but it depends on an
> infrastructure
> change. I really would like to see that patch brought up to date and
> discussed.

Indeed.  The non-invasive approach is the wrong one for upstream, it's
really only suited for being a bandaid out of tree.  For upstream, it
needs to be done right.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

On Thu, 1 May 2014, Doug Ledford wrote:

> Indeed.  The non-invasive approach is the wrong one for upstream, it's
> really only suited for being a bandaid out of tree.  For upstream, it
> needs to be done right.

Can we get the bandaid into RH7 for now?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mlx4: Allow to always block UD multicast loopback

[Christoph Lameter]

Tried to forwardport the 2008 patch but there is XRC stuff and other
things in there. The main function ib_uverbs_create_qp_expanded needs some
attention.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html