All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH REPOST FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB
@ 2013-07-17 14:22 Or Gerlitz
       [not found] ` <1374070962-328-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
  0 siblings, 1 reply; 7+ messages in thread
From: Or Gerlitz @ 2013-07-17 14:22 UTC (permalink / raw)
  To: roland-DgEjT+Ai2ygdnm+yROfE0A
  Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w,
	jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb, Or Gerlitz

- Did quick reposting as of one hunk in the last patch that went by mistake 
  from ipoib_ib.c to ipoib_multicast.c, sorry for the spam

Hi Roland,

This set of fixes is critical for Virtualization environments when the VM 
para-virtualized PKEY table isn't fully configured at the time the 
VF is probed, or when the management pkey is provisioned to non-zero index
in the VF pkey table.

The first three patches are pretty much few liners (two of them with somehow 
long change log...). The forth one a bit larger. Would be happy to see them 
all going to -stable, either by you adding a Cc: stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org when 
you push them or I can send them to Greg after they spend some time upstream.



Or.


Erez Shitrit (1):
  IB/ipoib: Fix pkey-change flow for Virtualization environments

Jack Morgenstein (2):
  IB/core: Create QP1 using the pkey index which contains the default pkey
  IB/mlx4: Use default pkey when creating tunnel QPs

Or Gerlitz (1):
  IB/ipoib: Make sure child devices use valid/proper pkeys

 drivers/infiniband/core/mad.c                |    8 +++-
 drivers/infiniband/hw/mlx4/mad.c             |   10 +++-
 drivers/infiniband/ulp/ipoib/ipoib_ib.c      |   76 +++++++++++++++++++++-----
 drivers/infiniband/ulp/ipoib/ipoib_main.c    |    2 +-
 drivers/infiniband/ulp/ipoib/ipoib_netlink.c |    9 +++
 5 files changed, 88 insertions(+), 17 deletions(-)


 drivers/infiniband/ulp/ipoib/ipoib_netlink.c   |    3 +
 6 files changed, 85 insertions(+), 18 deletions(-)

This is the sequence of events when the IPoIB patch is applied at the guest:

<-- the VM pkey table contains 0x0000 in index 0, and hence the mgid has 0x8000 as the pkey

[root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0

0x0000

[root@xena017-3 ~]# ip addr show ib0
22: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast state DOWN qlen 256
    link/infiniband 80:00:05:8b:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:00:00:00:00:00:00:00:ff:ff:ff:ff
    inet 192.168.20.199/24 brd 192.168.20.255 scope global ib0

<--  the hypervisor changed pkey value in index 0  of the VM pkey table to contain 0x8001

[root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0

0x8001

[root@xena017-3 ~]# dmesg
ib0: bringing up interface
IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready
ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22
ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22
[...]
ib0: Event 12 on device mlx4_0 port 1
ib0: pkey changed from 0x8000 to 0x8001
ib0: downing ib_dev
ib0: All sends and receives done.
ib0: Created ah ffff88011a11f4e0
IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
ib0: Created ah ffff88011a11f660
ib0: Created ah ffff88011a11f780

<--- mgid changed to use 0x8001

[root@xena017-3 ~]# ip addr show ib0
22: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
    link/infiniband 80:00:05:8b:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:01:00:00:00:00:00:00:ff:ff:ff:ff
    inet 192.168.20.199/24 brd 192.168.20.255 scope global ib0


ping works etc

--> when the mlx4 patch isn't applied on the host we see these errors

<mlx4_ib> create_pv_sqp: Couldn't change tunnel qp state to INIT (-22)
<mlx4_ib> create_pv_resources: Couldn't create tunnel for QP1 (-22)

--> this is the sequence of events when the IPoIB patch is not applied at the guest:

<-- the VM pkey table contains 0x0000 in index 0, and hence the mgid has 0x8000 as the pkey

[root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0

0x0000

[root@xena017-3 infiniband]# modprobe ib_ipoib debug_level=1

[root@xena017-3 infiniband]# ip a s ib0

30: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast state DOWN qlen 256
    link/infiniband 80:00:05:93:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:00:00:00:00:00:00:00:ff:ff:ff:ff

[root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0

0x8001

<-- ipoib got the event, but nothing changed

[root@xena017-3 infiniband]# ip a s ib0

30: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast state DOWN qlen 256
    link/infiniband 80:00:05:93:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:00:00:00:00:00:00:00:ff:ff:ff:ff
    inet 192.168.20.199/24 brd 192.168.20.255 scope global ib0

[root@xena017-3 infiniband]# dmesg
ib%d: max_srq_sge=31
ib%d: max_cm_mtu = 0xfff0, num_frags=16
ib%d: max_srq_sge=31
ib%d: max_cm_mtu = 0xfff0, num_frags=16
ib0: bringing up interface
IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready
ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22
ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22
ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22
ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22
ib0: Event 12 on device mlx4_0 port 1
ib0: downing ib_dev
ib0: All sends and receives done.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH REPOST FIXES for-3.11 1/4] IB/core: Create QP1 using the pkey index which contains the default pkey
       [not found] ` <1374070962-328-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2013-07-17 14:22   ` Or Gerlitz
       [not found]     ` <1374070962-328-2-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
  2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 2/4] IB/mlx4: Use default pkey when creating tunnel QPs Or Gerlitz
                     ` (2 subsequent siblings)
  3 siblings, 1 reply; 7+ messages in thread
From: Or Gerlitz @ 2013-07-17 14:22 UTC (permalink / raw)
  To: roland-DgEjT+Ai2ygdnm+yROfE0A
  Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w,
	jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb, Or Gerlitz

From: Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>

Currently, QP1 is created using pkey index 0. This patch simply looks for
the index containing the default pkey, rather than hard-coding pkey index 0.

This change will have no effect in Native mode, since QP0 and QP1 are created
before the SM configures the port, so pkey table will still be the default
table defined by the IB Spec, in C10-123: "If non-volatile storage is not used
to hold P_Key Table contents, then if a PM (Partition Manager) is not present,
and prior to PM initialization of the P_Key Table, the P_Key Table must act as
if it contains a single valid entry, at P_Key_ix = 0, containing the default
partition key. All other entries in the P_Key Table must be invalid."

Thus, in the native mode case, the driver will find the default pkey
at index 0 (so it will be no different than the hard-coding).

However, in SRIOV mode, for VFs, the pkey table may be paravirtualized, so
that the VF's pkey index zero may not necessarily be mapped to the real pkey
index 0. For VFs, therefore, it is important to find the virtual index which
maps to the real default pkey.

This commit does the following for QP1 creation:

1. Find the pkey index containing the default pkey, and use that index if found.
   ib_find_pkey() returns the index of the limited-membership default pkey
   (0x7FFF) if the full-member default pkey is not in the table.

2. If neither form of the default pkey is found, use pkey index 0 (previous behavior).

Signed-off-by: Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/mad.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c
index dc3fd1e..9be6754 100644
--- a/drivers/infiniband/core/mad.c
+++ b/drivers/infiniband/core/mad.c
@@ -2663,6 +2663,7 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv)
 	int ret, i;
 	struct ib_qp_attr *attr;
 	struct ib_qp *qp;
+	u16 pkey_index = 0;
 
 	attr = kmalloc(sizeof *attr, GFP_KERNEL);
 	if (!attr) {
@@ -2670,6 +2671,11 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv)
 		return -ENOMEM;
 	}
 
+	ret = ib_find_pkey(port_priv->device, port_priv->port_num,
+			   IB_DEFAULT_PKEY_FULL, &pkey_index);
+	if (ret)
+		pkey_index = 0;
+
 	for (i = 0; i < IB_MAD_QPS_CORE; i++) {
 		qp = port_priv->qp_info[i].qp;
 		if (!qp)
@@ -2680,7 +2686,7 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv)
 		 * one is needed for the Reset to Init transition
 		 */
 		attr->qp_state = IB_QPS_INIT;
-		attr->pkey_index = 0;
+		attr->pkey_index = pkey_index;
 		attr->qkey = (qp->qp_num == 0) ? 0 : IB_QP1_QKEY;
 		ret = ib_modify_qp(qp, attr, IB_QP_STATE |
 					     IB_QP_PKEY_INDEX | IB_QP_QKEY);
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH REPOST FIXES for-3.11 2/4] IB/mlx4: Use default pkey when creating tunnel QPs
       [not found] ` <1374070962-328-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
  2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 1/4] IB/core: Create QP1 using the pkey index which contains the default pkey Or Gerlitz
@ 2013-07-17 14:22   ` Or Gerlitz
  2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 3/4] IB/ipoib: Make sure child devices use valid/proper pkeys Or Gerlitz
  2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 4/4] IB/ipoib: Fix pkey-change flow for Virtualization environments Or Gerlitz
  3 siblings, 0 replies; 7+ messages in thread
From: Or Gerlitz @ 2013-07-17 14:22 UTC (permalink / raw)
  To: roland-DgEjT+Ai2ygdnm+yROfE0A
  Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w,
	jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb, Or Gerlitz

From: Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>

When creating tunnel QPs for special QP tunneling, look for the default pkey
in the slave's virtual pkey table. If it is present, use the real pkey index
where the default pkey is located.

If the default pkey is not found in the pkey table, use the real pkey index
which is stored at index 0 in the slave's virtual pkey table (this is the
current behavior).

This change is required to support cloud computing, where the paravirtualized
index of the default pkey is moved to index 1 or higher. The pkey at
paravirtualized index 0 is used for the default IPoIB interface created by the VF.

Its possible for the pkey value at paravirtualized index 0 to be invalid (zero) at
VF probe time (pkey index 0 is mapped to real pkey index 127, which contains pkey = 0).

At some point after the VF probe, the cloud computing interface at the Hypervisor
maps virtual index 0 for the VF to the pkey index containing the pkey that IPoIB
will use in its operation.  However, when the tunnel QP is created, the pkey at
the slave's virtual index 0 is still mapped to the invalid pkey index, so tunnel
QP creation fails.

This commit causes the Hypervisor to search for the default pkey in the slave's
pkey table -- and this pkey is present in the table (at index > 0) at tunnel QP
creation time, so that the tunnel QP creation will succeed.

Signed-off-by: Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/hw/mlx4/mad.c |   10 ++++++++--
 1 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/hw/mlx4/mad.c b/drivers/infiniband/hw/mlx4/mad.c
index 4d599ce..f2a3f48 100644
--- a/drivers/infiniband/hw/mlx4/mad.c
+++ b/drivers/infiniband/hw/mlx4/mad.c
@@ -1511,8 +1511,14 @@ static int create_pv_sqp(struct mlx4_ib_demux_pv_ctx *ctx,
 
 	memset(&attr, 0, sizeof attr);
 	attr.qp_state = IB_QPS_INIT;
-	attr.pkey_index =
-		to_mdev(ctx->ib_dev)->pkeys.virt2phys_pkey[ctx->slave][ctx->port - 1][0];
+	ret = 0;
+	if (create_tun)
+		ret = find_slave_port_pkey_ix(to_mdev(ctx->ib_dev), ctx->slave,
+					      ctx->port, IB_DEFAULT_PKEY_FULL,
+					      &attr.pkey_index);
+	if (ret || !create_tun)
+		attr.pkey_index =
+			to_mdev(ctx->ib_dev)->pkeys.virt2phys_pkey[ctx->slave][ctx->port - 1][0];
 	attr.qkey = IB_QP1_QKEY;
 	attr.port_num = ctx->port;
 	ret = ib_modify_qp(tun_qp->qp, &attr, qp_attr_mask_INIT);
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH REPOST FIXES for-3.11 3/4] IB/ipoib: Make sure child devices use valid/proper pkeys
       [not found] ` <1374070962-328-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
  2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 1/4] IB/core: Create QP1 using the pkey index which contains the default pkey Or Gerlitz
  2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 2/4] IB/mlx4: Use default pkey when creating tunnel QPs Or Gerlitz
@ 2013-07-17 14:22   ` Or Gerlitz
  2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 4/4] IB/ipoib: Fix pkey-change flow for Virtualization environments Or Gerlitz
  3 siblings, 0 replies; 7+ messages in thread
From: Or Gerlitz @ 2013-07-17 14:22 UTC (permalink / raw)
  To: roland-DgEjT+Ai2ygdnm+yROfE0A
  Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w,
	jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb, Or Gerlitz

Make sure that the IB invalid pkey (0x0000 or 0x8000) isn't used for child devices.

Also, make sure to always set the full membership bit for the pkey of devices
created by rtnl link ops.

Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/ulp/ipoib/ipoib_main.c    |    2 +-
 drivers/infiniband/ulp/ipoib/ipoib_netlink.c |    9 +++++++++
 2 files changed, 10 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c
index b6e049a..c6f71a8 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -1461,7 +1461,7 @@ static ssize_t create_child(struct device *dev,
 	if (sscanf(buf, "%i", &pkey) != 1)
 		return -EINVAL;
 
-	if (pkey < 0 || pkey > 0xffff)
+	if (pkey <= 0 || pkey > 0xffff || pkey == 0x8000)
 		return -EINVAL;
 
 	/*
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_netlink.c b/drivers/infiniband/ulp/ipoib/ipoib_netlink.c
index 7468593..f81abe1 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_netlink.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_netlink.c
@@ -119,6 +119,15 @@ static int ipoib_new_child_link(struct net *src_net, struct net_device *dev,
 	} else
 		child_pkey  = nla_get_u16(data[IFLA_IPOIB_PKEY]);
 
+	if (child_pkey == 0 || child_pkey == 0x8000)
+		return -EINVAL;
+
+	/*
+	 * Set the full membership bit, so that we join the right
+	 * broadcast group, etc.
+	 */
+	child_pkey |= 0x8000;
+
 	err = __ipoib_vlan_add(ppriv, netdev_priv(dev), child_pkey, IPOIB_RTNL_CHILD);
 
 	if (!err && data)
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH REPOST FIXES for-3.11 4/4] IB/ipoib: Fix pkey-change flow for Virtualization environments
       [not found] ` <1374070962-328-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
                     ` (2 preceding siblings ...)
  2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 3/4] IB/ipoib: Make sure child devices use valid/proper pkeys Or Gerlitz
@ 2013-07-17 14:22   ` Or Gerlitz
  3 siblings, 0 replies; 7+ messages in thread
From: Or Gerlitz @ 2013-07-17 14:22 UTC (permalink / raw)
  To: roland-DgEjT+Ai2ygdnm+yROfE0A
  Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w,
	jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb, Or Gerlitz

From: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

IPoIB's required behaviour w.r.t to the pkey used by the device is the following:

- For "parent" interfaces (e.g ib0, ib1, etc) who are created automatically as a
  result of hot-plug events from the IB core, the driver needs to take whatever
  pkey vlaue it finds in index 0, and stick to that index.

- For child interfaces (e.g ib0.8001, etc) created by admin directive, the driver
  needs to use and stick to the value provided during its creation.

In SR-IOV environment its possible for the VF probe to take place before the
cloud management software provisions the suitable pkey for the VF in the
paravirtualed PKEY table index 0. When this is the case, the VF IB stack will
find in index 0 an invalide pkey, which is all zeros.

Moreover, the cloud managment can assign the pkey value at index 0 at any
time of the guest life cycle.

The correct behavior for IPoIB to address these requirements for parent
interfaces is to use PKEY_CHANGE event as trigger to optionally re-init the
device pkey value and re-create all the relevant resources accordingly, if
the value of the pkey in index 0 has changed (from invalid to valid or from
valid value X to invalid value Y).

This patch enhances the heavy flushing code which is triggered by pkey change
event, to behave correctly for parent devices. For child devices, the code
remains the same, namely chases pkey value and not index.

Signed-off-by: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/ulp/ipoib/ipoib_ib.c |   76 +++++++++++++++++++++++++-----
 1 files changed, 63 insertions(+), 13 deletions(-)

diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 2cfa76f..196b1d1 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -932,12 +932,47 @@ int ipoib_ib_dev_init(struct net_device *dev, struct ib_device *ca, int port)
 	return 0;
 }
 
+/*
+ * Takes whatever value which is in pkey index 0 and updates priv->pkey
+ * returns 0 if the pkey value was changed.
+ */
+static inline int update_parent_pkey(struct ipoib_dev_priv *priv)
+{
+	int result;
+	u16 prev_pkey;
+
+	prev_pkey = priv->pkey;
+	result = ib_query_pkey(priv->ca, priv->port, 0, &priv->pkey);
+	if (result) {
+		ipoib_warn(priv, "ib_query_pkey port %d failed (ret = %d)\n",
+			   priv->port, result);
+		return result;
+	}
+
+	priv->pkey |= 0x8000;
+
+	if (prev_pkey != priv->pkey) {
+		ipoib_dbg(priv, "pkey changed from 0x%x to 0x%x\n",
+			  prev_pkey, priv->pkey);
+		/*
+		 * Update the pkey in the broadcast address, while making sure to set
+		 * the full membership bit, so that we join the right broadcast group.
+		 */
+		priv->dev->broadcast[8] = priv->pkey >> 8;
+		priv->dev->broadcast[9] = priv->pkey & 0xff;
+		return 0;
+	}
+
+	return 1;
+}
+
 static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv,
 				enum ipoib_flush_level level)
 {
 	struct ipoib_dev_priv *cpriv;
 	struct net_device *dev = priv->dev;
 	u16 new_index;
+	int result;
 
 	mutex_lock(&priv->vlan_mutex);
 
@@ -951,6 +986,10 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv,
 	mutex_unlock(&priv->vlan_mutex);
 
 	if (!test_bit(IPOIB_FLAG_INITIALIZED, &priv->flags)) {
+		/* for non-child devices must check/update the pkey value here */
+		if (level == IPOIB_FLUSH_HEAVY &&
+		    !test_bit(IPOIB_FLAG_SUBINTERFACE, &priv->flags))
+			update_parent_pkey(priv);
 		ipoib_dbg(priv, "Not flushing - IPOIB_FLAG_INITIALIZED not set.\n");
 		return;
 	}
@@ -961,21 +1000,32 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv,
 	}
 
 	if (level == IPOIB_FLUSH_HEAVY) {
-		if (ib_find_pkey(priv->ca, priv->port, priv->pkey, &new_index)) {
-			clear_bit(IPOIB_PKEY_ASSIGNED, &priv->flags);
-			ipoib_ib_dev_down(dev, 0);
-			ipoib_ib_dev_stop(dev, 0);
-			if (ipoib_pkey_dev_delay_open(dev))
+		/* child devices chase their origin pkey value, while non-child
+		 * (parent) devices should always takes what present in pkey index 0
+		 */
+		if (test_bit(IPOIB_FLAG_SUBINTERFACE, &priv->flags)) {
+			if (ib_find_pkey(priv->ca, priv->port, priv->pkey, &new_index)) {
+				clear_bit(IPOIB_PKEY_ASSIGNED, &priv->flags);
+				ipoib_ib_dev_down(dev, 0);
+				ipoib_ib_dev_stop(dev, 0);
+				if (ipoib_pkey_dev_delay_open(dev))
+					return;
+			}
+			/* restart QP only if P_Key index is changed */
+			if (test_and_set_bit(IPOIB_PKEY_ASSIGNED, &priv->flags) &&
+			    new_index == priv->pkey_index) {
+				ipoib_dbg(priv, "Not flushing - P_Key index not changed.\n");
 				return;
+			}
+			priv->pkey_index = new_index;
+		} else {
+			result = update_parent_pkey(priv);
+			/* restart QP only if P_Key value changed */
+			if (result) {
+				ipoib_dbg(priv, "Not flushing - P_Key value not changed.\n");
+				return;
+			}
 		}
-
-		/* restart QP only if P_Key index is changed */
-		if (test_and_set_bit(IPOIB_PKEY_ASSIGNED, &priv->flags) &&
-		    new_index == priv->pkey_index) {
-			ipoib_dbg(priv, "Not flushing - P_Key index not changed.\n");
-			return;
-		}
-		priv->pkey_index = new_index;
 	}
 
 	if (level == IPOIB_FLUSH_LIGHT) {
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply related	[flat|nested] 7+ messages in thread

* RE: [PATCH REPOST FIXES for-3.11 1/4] IB/core: Create QP1 using the pkey index which contains the default pkey
       [not found]     ` <1374070962-328-2-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2013-07-17 16:31       ` Hefty, Sean
       [not found]         ` <1828884A29C6694DAF28B7E6B8A82373805B82BE-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
  0 siblings, 1 reply; 7+ messages in thread
From: Hefty, Sean @ 2013-07-17 16:31 UTC (permalink / raw)
  To: Or Gerlitz, roland-DgEjT+Ai2ygdnm+yROfE0A
  Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w,
	jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb

> diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c
> index dc3fd1e..9be6754 100644
> --- a/drivers/infiniband/core/mad.c
> +++ b/drivers/infiniband/core/mad.c
> @@ -2663,6 +2663,7 @@ static int ib_mad_port_start(struct ib_mad_port_private
> *port_priv)
>  	int ret, i;
>  	struct ib_qp_attr *attr;
>  	struct ib_qp *qp;
> +	u16 pkey_index = 0;

This shouldn't need to be initialized, as it is always set further down in the patch.

Reviewed-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH REPOST FIXES for-3.11 1/4] IB/core: Create QP1 using the pkey index which contains the default pkey
       [not found]         ` <1828884A29C6694DAF28B7E6B8A82373805B82BE-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
@ 2013-07-18  6:29           ` Or Gerlitz
  0 siblings, 0 replies; 7+ messages in thread
From: Or Gerlitz @ 2013-07-18  6:29 UTC (permalink / raw)
  To: Hefty, Sean
  Cc: roland-DgEjT+Ai2ygdnm+yROfE0A, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
	erezsh-VPRAkNaXOzVWk0Htik3J/w,
	jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb

On 17/07/2013 19:31, Hefty, Sean wrote:
>> diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c
>> index dc3fd1e..9be6754 100644
>> --- a/drivers/infiniband/core/mad.c
>> +++ b/drivers/infiniband/core/mad.c
>> @@ -2663,6 +2663,7 @@ static int ib_mad_port_start(struct ib_mad_port_private
>> *port_priv)
>>   	int ret, i;
>>   	struct ib_qp_attr *attr;
>>   	struct ib_qp *qp;
>> +	u16 pkey_index = 0;
> This shouldn't need to be initialized, as it is always set further down in the patch.
>
> Reviewed-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
OK, for V1 will fix && will add your Reviewed-by
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2013-07-18  6:29 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-07-17 14:22 [PATCH REPOST FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB Or Gerlitz
     [not found] ` <1374070962-328-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 1/4] IB/core: Create QP1 using the pkey index which contains the default pkey Or Gerlitz
     [not found]     ` <1374070962-328-2-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2013-07-17 16:31       ` Hefty, Sean
     [not found]         ` <1828884A29C6694DAF28B7E6B8A82373805B82BE-P5GAC/sN6hkd3b2yrw5b5LfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-07-18  6:29           ` Or Gerlitz
2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 2/4] IB/mlx4: Use default pkey when creating tunnel QPs Or Gerlitz
2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 3/4] IB/ipoib: Make sure child devices use valid/proper pkeys Or Gerlitz
2013-07-17 14:22   ` [PATCH REPOST FIXES for-3.11 4/4] IB/ipoib: Fix pkey-change flow for Virtualization environments Or Gerlitz

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.