* [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB @ 2013-07-18 11:02 Or Gerlitz [not found] ` <1374145352-27816-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Or Gerlitz @ 2013-07-18 11:02 UTC (permalink / raw) To: roland-DgEjT+Ai2ygdnm+yROfE0A Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w, jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb, Or Gerlitz Hi Roland, This set of fixes is critical for Virtualization environments when the VM para-virtualized PKEY table isn't fully configured at the time the VF is probed, or when the management pkey is provisioned to non-zero index in the VF pkey table. The first three patches are pretty much few liners (two of them with somehow long change log...). The forth one a bit larger. Would be happy to see them all going to -stable, either by you adding a Cc: stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org when you push them or I can send them to Greg after they spend some time upstream. Changes from V0: - applied feedback from Sean on the IB core patch and added his review-by Or. Erez Shitrit (1): IB/ipoib: Fix pkey-change flow for Virtualization environments Jack Morgenstein (2): IB/core: Create QP1 using the pkey index which contains the default pkey IB/mlx4: Use default pkey when creating tunnel QPs Or Gerlitz (1): IB/ipoib: Make sure child devices use valid/proper pkeys drivers/infiniband/core/mad.c | 8 +++- drivers/infiniband/hw/mlx4/mad.c | 10 +++- drivers/infiniband/ulp/ipoib/ipoib_ib.c | 76 +++++++++++++++++++++----- drivers/infiniband/ulp/ipoib/ipoib_main.c | 2 +- drivers/infiniband/ulp/ipoib/ipoib_netlink.c | 9 +++ 5 files changed, 88 insertions(+), 17 deletions(-) This is the sequence of events when the IPoIB patch is applied at the guest: <-- the VM pkey table contains 0x0000 in index 0, and hence the mgid has 0x8000 as the pkey [root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 0x0000 [root@xena017-3 ~]# ip addr show ib0 22: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast state DOWN qlen 256 link/infiniband 80:00:05:8b:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:00:00:00:00:00:00:00:ff:ff:ff:ff inet 192.168.20.199/24 brd 192.168.20.255 scope global ib0 <-- the hypervisor changed pkey value in index 0 of the VM pkey table to contain 0x8001 [root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 0x8001 [root@xena017-3 ~]# dmesg ib0: bringing up interface IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 [...] ib0: Event 12 on device mlx4_0 port 1 ib0: pkey changed from 0x8000 to 0x8001 ib0: downing ib_dev ib0: All sends and receives done. ib0: Created ah ffff88011a11f4e0 IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready ib0: Created ah ffff88011a11f660 ib0: Created ah ffff88011a11f780 <--- mgid changed to use 0x8001 [root@xena017-3 ~]# ip addr show ib0 22: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256 link/infiniband 80:00:05:8b:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:01:00:00:00:00:00:00:ff:ff:ff:ff inet 192.168.20.199/24 brd 192.168.20.255 scope global ib0 ping works etc --> when the mlx4 patch isn't applied on the host we see these errors <mlx4_ib> create_pv_sqp: Couldn't change tunnel qp state to INIT (-22) <mlx4_ib> create_pv_resources: Couldn't create tunnel for QP1 (-22) --> this is the sequence of events when the IPoIB patch is not applied at the guest: <-- the VM pkey table contains 0x0000 in index 0, and hence the mgid has 0x8000 as the pkey [root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 0x0000 [root@xena017-3 infiniband]# modprobe ib_ipoib debug_level=1 [root@xena017-3 infiniband]# ip a s ib0 30: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast state DOWN qlen 256 link/infiniband 80:00:05:93:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:00:00:00:00:00:00:00:ff:ff:ff:ff [root@xena017-3 infiniband]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 0x8001 <-- ipoib got the event, but nothing changed [root@xena017-3 infiniband]# ip a s ib0 30: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 2044 qdisc pfifo_fast state DOWN qlen 256 link/infiniband 80:00:05:93:fe:80:00:00:00:00:00:00:00:14:05:00:00:00:04:f9 brd 00:ff:ff:ff:ff:12:40:1b:80:00:00:00:00:00:00:00:ff:ff:ff:ff inet 192.168.20.199/24 brd 192.168.20.255 scope global ib0 [root@xena017-3 infiniband]# dmesg ib%d: max_srq_sge=31 ib%d: max_cm_mtu = 0xfff0, num_frags=16 ib%d: max_srq_sge=31 ib%d: max_cm_mtu = 0xfff0, num_frags=16 ib0: bringing up interface IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 ib0: multicast join failed for ff12:401b:8000:0000:0000:0000:ffff:ffff, status -22 ib0: Event 12 on device mlx4_0 port 1 ib0: downing ib_dev ib0: All sends and receives done. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <1374145352-27816-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* [PATCH V1 FIXES for-3.11 1/4] IB/core: Create QP1 using the pkey index which contains the default pkey [not found] ` <1374145352-27816-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> @ 2013-07-18 11:02 ` Or Gerlitz 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 2/4] IB/mlx4: Use default pkey when creating tunnel QPs Or Gerlitz ` (3 subsequent siblings) 4 siblings, 0 replies; 14+ messages in thread From: Or Gerlitz @ 2013-07-18 11:02 UTC (permalink / raw) To: roland-DgEjT+Ai2ygdnm+yROfE0A Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w, jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb, Or Gerlitz From: Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> Currently, QP1 is created using pkey index 0. This patch simply looks for the index containing the default pkey, rather than hard-coding pkey index 0. This change will have no effect in Native mode, since QP0 and QP1 are created before the SM configures the port, so pkey table will still be the default table defined by the IB Spec, in C10-123: "If non-volatile storage is not used to hold P_Key Table contents, then if a PM (Partition Manager) is not present, and prior to PM initialization of the P_Key Table, the P_Key Table must act as if it contains a single valid entry, at P_Key_ix = 0, containing the default partition key. All other entries in the P_Key Table must be invalid." Thus, in the native mode case, the driver will find the default pkey at index 0 (so it will be no different than the hard-coding). However, in SRIOV mode, for VFs, the pkey table may be paravirtualized, so that the VF's pkey index zero may not necessarily be mapped to the real pkey index 0. For VFs, therefore, it is important to find the virtual index which maps to the real default pkey. This commit does the following for QP1 creation: 1. Find the pkey index containing the default pkey, and use that index if found. ib_find_pkey() returns the index of the limited-membership default pkey (0x7FFF) if the full-member default pkey is not in the table. 2. If neither form of the default pkey is found, use pkey index 0 (previous behavior). Reviewed-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Signed-off-by: Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- drivers/infiniband/core/mad.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c index dc3fd1e..4c837e6 100644 --- a/drivers/infiniband/core/mad.c +++ b/drivers/infiniband/core/mad.c @@ -2663,6 +2663,7 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv) int ret, i; struct ib_qp_attr *attr; struct ib_qp *qp; + u16 pkey_index; attr = kmalloc(sizeof *attr, GFP_KERNEL); if (!attr) { @@ -2670,6 +2671,11 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv) return -ENOMEM; } + ret = ib_find_pkey(port_priv->device, port_priv->port_num, + IB_DEFAULT_PKEY_FULL, &pkey_index); + if (ret) + pkey_index = 0; + for (i = 0; i < IB_MAD_QPS_CORE; i++) { qp = port_priv->qp_info[i].qp; if (!qp) @@ -2680,7 +2686,7 @@ static int ib_mad_port_start(struct ib_mad_port_private *port_priv) * one is needed for the Reset to Init transition */ attr->qp_state = IB_QPS_INIT; - attr->pkey_index = 0; + attr->pkey_index = pkey_index; attr->qkey = (qp->qp_num == 0) ? 0 : IB_QP1_QKEY; ret = ib_modify_qp(qp, attr, IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_QKEY); -- 1.7.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH V1 FIXES for-3.11 2/4] IB/mlx4: Use default pkey when creating tunnel QPs [not found] ` <1374145352-27816-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 1/4] IB/core: Create QP1 using the pkey index which contains the default pkey Or Gerlitz @ 2013-07-18 11:02 ` Or Gerlitz 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 3/4] IB/ipoib: Make sure child devices use valid/proper pkeys Or Gerlitz ` (2 subsequent siblings) 4 siblings, 0 replies; 14+ messages in thread From: Or Gerlitz @ 2013-07-18 11:02 UTC (permalink / raw) To: roland-DgEjT+Ai2ygdnm+yROfE0A Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w, jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb, Or Gerlitz From: Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> When creating tunnel QPs for special QP tunneling, look for the default pkey in the slave's virtual pkey table. If it is present, use the real pkey index where the default pkey is located. If the default pkey is not found in the pkey table, use the real pkey index which is stored at index 0 in the slave's virtual pkey table (this is the current behavior). This change is required to support cloud computing, where the paravirtualized index of the default pkey is moved to index 1 or higher. The pkey at paravirtualized index 0 is used for the default IPoIB interface created by the VF. Its possible for the pkey value at paravirtualized index 0 to be invalid (zero) at VF probe time (pkey index 0 is mapped to real pkey index 127, which contains pkey = 0). At some point after the VF probe, the cloud computing interface at the Hypervisor maps virtual index 0 for the VF to the pkey index containing the pkey that IPoIB will use in its operation. However, when the tunnel QP is created, the pkey at the slave's virtual index 0 is still mapped to the invalid pkey index, so tunnel QP creation fails. This commit causes the Hypervisor to search for the default pkey in the slave's pkey table -- and this pkey is present in the table (at index > 0) at tunnel QP creation time, so that the tunnel QP creation will succeed. Signed-off-by: Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org> Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- drivers/infiniband/hw/mlx4/mad.c | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/mad.c b/drivers/infiniband/hw/mlx4/mad.c index 4d599ce..f2a3f48 100644 --- a/drivers/infiniband/hw/mlx4/mad.c +++ b/drivers/infiniband/hw/mlx4/mad.c @@ -1511,8 +1511,14 @@ static int create_pv_sqp(struct mlx4_ib_demux_pv_ctx *ctx, memset(&attr, 0, sizeof attr); attr.qp_state = IB_QPS_INIT; - attr.pkey_index = - to_mdev(ctx->ib_dev)->pkeys.virt2phys_pkey[ctx->slave][ctx->port - 1][0]; + ret = 0; + if (create_tun) + ret = find_slave_port_pkey_ix(to_mdev(ctx->ib_dev), ctx->slave, + ctx->port, IB_DEFAULT_PKEY_FULL, + &attr.pkey_index); + if (ret || !create_tun) + attr.pkey_index = + to_mdev(ctx->ib_dev)->pkeys.virt2phys_pkey[ctx->slave][ctx->port - 1][0]; attr.qkey = IB_QP1_QKEY; attr.port_num = ctx->port; ret = ib_modify_qp(tun_qp->qp, &attr, qp_attr_mask_INIT); -- 1.7.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH V1 FIXES for-3.11 3/4] IB/ipoib: Make sure child devices use valid/proper pkeys [not found] ` <1374145352-27816-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 1/4] IB/core: Create QP1 using the pkey index which contains the default pkey Or Gerlitz 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 2/4] IB/mlx4: Use default pkey when creating tunnel QPs Or Gerlitz @ 2013-07-18 11:02 ` Or Gerlitz 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 4/4] IB/ipoib: Fix pkey-change flow for Virtualization environments Or Gerlitz 2013-07-31 20:29 ` [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB Or Gerlitz 4 siblings, 0 replies; 14+ messages in thread From: Or Gerlitz @ 2013-07-18 11:02 UTC (permalink / raw) To: roland-DgEjT+Ai2ygdnm+yROfE0A Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w, jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb, Or Gerlitz Make sure that the IB invalid pkey (0x0000 or 0x8000) isn't used for child devices. Also, make sure to always set the full membership bit for the pkey of devices created by rtnl link ops. Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- drivers/infiniband/ulp/ipoib/ipoib_main.c | 2 +- drivers/infiniband/ulp/ipoib/ipoib_netlink.c | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index b6e049a..c6f71a8 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -1461,7 +1461,7 @@ static ssize_t create_child(struct device *dev, if (sscanf(buf, "%i", &pkey) != 1) return -EINVAL; - if (pkey < 0 || pkey > 0xffff) + if (pkey <= 0 || pkey > 0xffff || pkey == 0x8000) return -EINVAL; /* diff --git a/drivers/infiniband/ulp/ipoib/ipoib_netlink.c b/drivers/infiniband/ulp/ipoib/ipoib_netlink.c index 7468593..f81abe1 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_netlink.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_netlink.c @@ -119,6 +119,15 @@ static int ipoib_new_child_link(struct net *src_net, struct net_device *dev, } else child_pkey = nla_get_u16(data[IFLA_IPOIB_PKEY]); + if (child_pkey == 0 || child_pkey == 0x8000) + return -EINVAL; + + /* + * Set the full membership bit, so that we join the right + * broadcast group, etc. + */ + child_pkey |= 0x8000; + err = __ipoib_vlan_add(ppriv, netdev_priv(dev), child_pkey, IPOIB_RTNL_CHILD); if (!err && data) -- 1.7.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH V1 FIXES for-3.11 4/4] IB/ipoib: Fix pkey-change flow for Virtualization environments [not found] ` <1374145352-27816-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> ` (2 preceding siblings ...) 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 3/4] IB/ipoib: Make sure child devices use valid/proper pkeys Or Gerlitz @ 2013-07-18 11:02 ` Or Gerlitz 2013-07-31 20:29 ` [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB Or Gerlitz 4 siblings, 0 replies; 14+ messages in thread From: Or Gerlitz @ 2013-07-18 11:02 UTC (permalink / raw) To: roland-DgEjT+Ai2ygdnm+yROfE0A Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w, jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb, Or Gerlitz From: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> IPoIB's required behaviour w.r.t to the pkey used by the device is the following: - For "parent" interfaces (e.g ib0, ib1, etc) who are created automatically as a result of hot-plug events from the IB core, the driver needs to take whatever pkey vlaue it finds in index 0, and stick to that index. - For child interfaces (e.g ib0.8001, etc) created by admin directive, the driver needs to use and stick to the value provided during its creation. In SR-IOV environment its possible for the VF probe to take place before the cloud management software provisions the suitable pkey for the VF in the paravirtualed PKEY table index 0. When this is the case, the VF IB stack will find in index 0 an invalide pkey, which is all zeros. Moreover, the cloud managment can assign the pkey value at index 0 at any time of the guest life cycle. The correct behavior for IPoIB to address these requirements for parent interfaces is to use PKEY_CHANGE event as trigger to optionally re-init the device pkey value and re-create all the relevant resources accordingly, if the value of the pkey in index 0 has changed (from invalid to valid or from valid value X to invalid value Y). This patch enhances the heavy flushing code which is triggered by pkey change event, to behave correctly for parent devices. For child devices, the code remains the same, namely chases pkey value and not index. Signed-off-by: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> Signed-off-by: Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> --- drivers/infiniband/ulp/ipoib/ipoib_ib.c | 76 +++++++++++++++++++++++++----- 1 files changed, 63 insertions(+), 13 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c index 2cfa76f..196b1d1 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c @@ -932,12 +932,47 @@ int ipoib_ib_dev_init(struct net_device *dev, struct ib_device *ca, int port) return 0; } +/* + * Takes whatever value which is in pkey index 0 and updates priv->pkey + * returns 0 if the pkey value was changed. + */ +static inline int update_parent_pkey(struct ipoib_dev_priv *priv) +{ + int result; + u16 prev_pkey; + + prev_pkey = priv->pkey; + result = ib_query_pkey(priv->ca, priv->port, 0, &priv->pkey); + if (result) { + ipoib_warn(priv, "ib_query_pkey port %d failed (ret = %d)\n", + priv->port, result); + return result; + } + + priv->pkey |= 0x8000; + + if (prev_pkey != priv->pkey) { + ipoib_dbg(priv, "pkey changed from 0x%x to 0x%x\n", + prev_pkey, priv->pkey); + /* + * Update the pkey in the broadcast address, while making sure to set + * the full membership bit, so that we join the right broadcast group. + */ + priv->dev->broadcast[8] = priv->pkey >> 8; + priv->dev->broadcast[9] = priv->pkey & 0xff; + return 0; + } + + return 1; +} + static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, enum ipoib_flush_level level) { struct ipoib_dev_priv *cpriv; struct net_device *dev = priv->dev; u16 new_index; + int result; mutex_lock(&priv->vlan_mutex); @@ -951,6 +986,10 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, mutex_unlock(&priv->vlan_mutex); if (!test_bit(IPOIB_FLAG_INITIALIZED, &priv->flags)) { + /* for non-child devices must check/update the pkey value here */ + if (level == IPOIB_FLUSH_HEAVY && + !test_bit(IPOIB_FLAG_SUBINTERFACE, &priv->flags)) + update_parent_pkey(priv); ipoib_dbg(priv, "Not flushing - IPOIB_FLAG_INITIALIZED not set.\n"); return; } @@ -961,21 +1000,32 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv, } if (level == IPOIB_FLUSH_HEAVY) { - if (ib_find_pkey(priv->ca, priv->port, priv->pkey, &new_index)) { - clear_bit(IPOIB_PKEY_ASSIGNED, &priv->flags); - ipoib_ib_dev_down(dev, 0); - ipoib_ib_dev_stop(dev, 0); - if (ipoib_pkey_dev_delay_open(dev)) + /* child devices chase their origin pkey value, while non-child + * (parent) devices should always takes what present in pkey index 0 + */ + if (test_bit(IPOIB_FLAG_SUBINTERFACE, &priv->flags)) { + if (ib_find_pkey(priv->ca, priv->port, priv->pkey, &new_index)) { + clear_bit(IPOIB_PKEY_ASSIGNED, &priv->flags); + ipoib_ib_dev_down(dev, 0); + ipoib_ib_dev_stop(dev, 0); + if (ipoib_pkey_dev_delay_open(dev)) + return; + } + /* restart QP only if P_Key index is changed */ + if (test_and_set_bit(IPOIB_PKEY_ASSIGNED, &priv->flags) && + new_index == priv->pkey_index) { + ipoib_dbg(priv, "Not flushing - P_Key index not changed.\n"); return; + } + priv->pkey_index = new_index; + } else { + result = update_parent_pkey(priv); + /* restart QP only if P_Key value changed */ + if (result) { + ipoib_dbg(priv, "Not flushing - P_Key value not changed.\n"); + return; + } } - - /* restart QP only if P_Key index is changed */ - if (test_and_set_bit(IPOIB_PKEY_ASSIGNED, &priv->flags) && - new_index == priv->pkey_index) { - ipoib_dbg(priv, "Not flushing - P_Key index not changed.\n"); - return; - } - priv->pkey_index = new_index; } if (level == IPOIB_FLUSH_LIGHT) { -- 1.7.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB [not found] ` <1374145352-27816-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> ` (3 preceding siblings ...) 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 4/4] IB/ipoib: Fix pkey-change flow for Virtualization environments Or Gerlitz @ 2013-07-31 20:29 ` Or Gerlitz [not found] ` <CAJZOPZLVQhZtWyCJLt3W-COdHZnsrBaN+KOpj83d0GyS+RY+QA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 4 siblings, 1 reply; 14+ messages in thread From: Or Gerlitz @ 2013-07-31 20:29 UTC (permalink / raw) To: roland-DgEjT+Ai2ygdnm+yROfE0A Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, erezsh-VPRAkNaXOzVWk0Htik3J/w, jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb On Thu, Jul 18, 2013 at 2:02 PM, Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote: > This set of fixes is critical for Virtualization environments when the VM > para-virtualized PKEY table isn't fully configured at the time the > VF is probed, or when the management pkey is provisioned to non-zero index > in the VF pkey table. Roland, So what's about this series, any reason not to push it for 3.11 too with the other changes you have picked into the for-next branch? ditto for the iser patches to use fast-reg. Or. > The first three patches are pretty much few liners (two of them with somehow > long change log...). The forth one a bit larger. Would be happy to see them > all going to -stable, either by you adding a Cc: stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org when > you push them or I can send them to Greg after they spend some time upstream. > Changes from V0: > - applied feedback from Sean on the IB core patch and added his > review-by -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAJZOPZLVQhZtWyCJLt3W-COdHZnsrBaN+KOpj83d0GyS+RY+QA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB [not found] ` <CAJZOPZLVQhZtWyCJLt3W-COdHZnsrBaN+KOpj83d0GyS+RY+QA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-01 9:42 ` Or Gerlitz [not found] ` <51FA2D98.5000707-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Or Gerlitz @ 2013-08-01 9:42 UTC (permalink / raw) To: roland-DgEjT+Ai2ygdnm+yROfE0A; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA On 31/07/2013 23:29, Or Gerlitz wrote: > Roland, so what's about this series, any reason not to push it for 3.11 too > with the other changes you have picked into the for-next branch? ditto > for the iser patches to use fast-reg. Roland, thanks for picking this series too, do you want to add CC: stable-u79uwXL29TY76Z2rM5mHXA@public.gmane.org # v3.10+ on the four patches or prefer I will submit them later to -stable? I think its perfectly OK to expect a written answer from you on the iser patches. Are you OK with merging them for 3.11 or they will have to wait for 3.12? They bring in the benefit of avoiding bounce buffer when used over ConnectX Virtual Functions or ConnectIB devices AND ONLY touch iser code, so no risk to other kernel components. Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <51FA2D98.5000707-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>]
* Re: [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB [not found] ` <51FA2D98.5000707-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> @ 2013-08-03 22:12 ` Or Gerlitz [not found] ` <CAJZOPZ+_MbfUVjvdjuBBWexGx-++PRoW=7rb8bofaeTNGZqVPQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Or Gerlitz @ 2013-08-03 22:12 UTC (permalink / raw) To: Or Gerlitz Cc: roland-DgEjT+Ai2ygdnm+yROfE0A, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Thu, Aug 1, 2013 at 12:42 PM, Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote: > I think its perfectly OK to expect a written answer from you on the iser > patches. Are you OK with merging them for 3.11 or they will have to wait for 3.12? > > They bring in the benefit of avoiding bounce buffer when used over > ConnectX Virtual Functions or ConnectIB devices AND ONLY touch > iser code, so no risk to other kernel components. OK, Roland, I see that you didn't pick this for 3.11, probably as you consider this non-fix, well the patches makes the initiator to avoid bounce buffer for IOs > 4K, which saves kernel memory and CPU - which I think can be considered a fix for the system behaviour. If you still don't agree, let it be, tough life... but you're the maintainer. So in that respect && taking into account 3.11-rc3 is already here, can you the door (== for-next branch) be opened right away (please) for 3.12 patches, specifically this series? its impossible to let kernel work towards a merge window be done only in the 1-2 weeks before the window open. E.g I'd like to know that the iser series is safe and ready to go from your POV, and f not fix what ever needs to be changed now and not in two months just a little before 3.11 is out, fair-enough? Or. Or. Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAJZOPZ+_MbfUVjvdjuBBWexGx-++PRoW=7rb8bofaeTNGZqVPQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB [not found] ` <CAJZOPZ+_MbfUVjvdjuBBWexGx-++PRoW=7rb8bofaeTNGZqVPQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-05 18:40 ` Roland Dreier [not found] ` <CAL1RGDWoiO+eTjheA-CZrK_O0LrhbpDFxDn-vSWxZda3j9e+oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Roland Dreier @ 2013-08-05 18:40 UTC (permalink / raw) To: Or Gerlitz; +Cc: Or Gerlitz, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Sat, Aug 3, 2013 at 3:12 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > OK, Roland, I see that you didn't pick this for 3.11, probably as you > consider this non-fix, well the patches makes the initiator to avoid > bounce buffer for IOs > 4K, which saves kernel memory and CPU - which > I think can be considered a fix for the system behaviour. Seriously? Please, let's have some intellectual honesty -- there's no way that iSER series could possibly be suitable as post-merge window material. The whole point is that we want to be stabilizing things, not introducing new code, even if we believe it's an improvement. > If you still don't agree, let it be, tough life... but you're the > maintainer. So in that respect && taking into account 3.11-rc3 is > already here, can you the door (== for-next branch) be opened right > away (please) for 3.12 patches, specifically this series? its > impossible to let kernel work towards a merge window be done only in > the 1-2 weeks before the window open. E.g I'd like to know that the > iser series is safe and ready to go from your POV, and f not fix what > ever needs to be changed now and not in two months just a little > before 3.11 is out, fair-enough? Yes, the iSER stuff looks fine for the 3.12 merge window. - R. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAL1RGDWoiO+eTjheA-CZrK_O0LrhbpDFxDn-vSWxZda3j9e+oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB [not found] ` <CAL1RGDWoiO+eTjheA-CZrK_O0LrhbpDFxDn-vSWxZda3j9e+oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-05 21:02 ` Or Gerlitz [not found] ` <CAJZOPZJbjAC=u0vJdUFGO66KBunoZhX1b=y6UudTEgpi9mbA0g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Or Gerlitz @ 2013-08-05 21:02 UTC (permalink / raw) To: Roland Dreier; +Cc: Or Gerlitz, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Mon, Aug 5, 2013 at 9:40 PM, Roland Dreier <roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote: > Yes, the iSER stuff looks fine for the 3.12 merge window. OK, good. So we're post 3.11-rc4 now, which means the merge window can be three weeks away. Are you planning to pick 3.12 materials such as the iser series and more into your for-next branch any time soon? Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAJZOPZJbjAC=u0vJdUFGO66KBunoZhX1b=y6UudTEgpi9mbA0g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB [not found] ` <CAJZOPZJbjAC=u0vJdUFGO66KBunoZhX1b=y6UudTEgpi9mbA0g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-06 3:13 ` Roland Dreier [not found] ` <CAL1RGDWz81ZqkTjf74ske-i3U5yKH0z3aBr_cSxgcKeHo7Y1Lw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Roland Dreier @ 2013-08-06 3:13 UTC (permalink / raw) To: Or Gerlitz; +Cc: Or Gerlitz, linux-rdma-u79uwXL29TY76Z2rM5mHXA On Mon, Aug 5, 2013 at 2:02 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > OK, good. So we're post 3.11-rc4 now, which means the merge window can > be three weeks away. Are you planning to pick 3.12 materials such as > the iser series and more into your for-next branch any time soon? yes -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAL1RGDWz81ZqkTjf74ske-i3U5yKH0z3aBr_cSxgcKeHo7Y1Lw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB [not found] ` <CAL1RGDWz81ZqkTjf74ske-i3U5yKH0z3aBr_cSxgcKeHo7Y1Lw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-06 13:14 ` Or Gerlitz [not found] ` <CAJZOPZ+6Lq8bMmzWttKQufuoZ2er9hejvT-Zz_CFxbfOvsmbOw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Or Gerlitz @ 2013-08-06 13:14 UTC (permalink / raw) To: Roland Dreier Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Sean Hefty, Christoph Lameter, Jeff Squyres, Steve Wise, Doug Ledford On Tue, Aug 6, 2013 at 6:13 AM, Roland Dreier <roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote: > On Mon, Aug 5, 2013 at 2:02 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> OK, good. So we're post 3.11-rc4 now, which means the merge window can >> be three weeks away. Are you planning to pick 3.12 materials such as >> the iser series and more into your for-next branch any time soon? > yes Any time estimate for when you start accepting 3.12 patches into your for-next branch? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAJZOPZ+6Lq8bMmzWttKQufuoZ2er9hejvT-Zz_CFxbfOvsmbOw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB [not found] ` <CAJZOPZ+6Lq8bMmzWttKQufuoZ2er9hejvT-Zz_CFxbfOvsmbOw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-06 17:40 ` Roland Dreier [not found] ` <CAL1RGDWtz4Ku+1KCphYEmg95tUw8BuKTWKOrRfgJ_3gUJOA8Lw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 14+ messages in thread From: Roland Dreier @ 2013-08-06 17:40 UTC (permalink / raw) To: Or Gerlitz Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Sean Hefty, Christoph Lameter, Jeff Squyres, Steve Wise, Doug Ledford On Tue, Aug 6, 2013 at 6:14 AM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > Any time estimate for when you start accepting 3.12 patches into your > for-next branch? No. - R. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <CAL1RGDWtz4Ku+1KCphYEmg95tUw8BuKTWKOrRfgJ_3gUJOA8Lw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB [not found] ` <CAL1RGDWtz4Ku+1KCphYEmg95tUw8BuKTWKOrRfgJ_3gUJOA8Lw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2013-08-06 20:22 ` Or Gerlitz 0 siblings, 0 replies; 14+ messages in thread From: Or Gerlitz @ 2013-08-06 20:22 UTC (permalink / raw) To: Roland Dreier Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Sean Hefty, Christoph Lameter, Jeff Squyres, Steve Wise, Doug Ledford On Tue, Aug 6, 2013 at 8:40 PM, Roland Dreier <roland-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote: > On Tue, Aug 6, 2013 at 6:14 AM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> Any time estimate for when you start accepting 3.12 patches into your >> for-next branch? > No. So we are five days from RC5 and you don't have even estimate when you start accepting patches for the merge window?! I don't think real progress can be made if a kernel sub-system has this slowness. E.g we don't have time to have discussion on patches, its either silently accepted during the week before the merge window, or not accepted without any time to discuss why and how to fix. If e.g the iser series is OK for 3.12, why wait till rc7+ to accept it? would it help you if I send you from where to pull besides the list posting? Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2013-08-06 20:22 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-07-18 11:02 [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB Or Gerlitz [not found] ` <1374145352-27816-1-git-send-email-ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 1/4] IB/core: Create QP1 using the pkey index which contains the default pkey Or Gerlitz 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 2/4] IB/mlx4: Use default pkey when creating tunnel QPs Or Gerlitz 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 3/4] IB/ipoib: Make sure child devices use valid/proper pkeys Or Gerlitz 2013-07-18 11:02 ` [PATCH V1 FIXES for-3.11 4/4] IB/ipoib: Fix pkey-change flow for Virtualization environments Or Gerlitz 2013-07-31 20:29 ` [PATCH V1 FIXES for-3.11 0/4] Pkey fixes for IB core and IPoIB Or Gerlitz [not found] ` <CAJZOPZLVQhZtWyCJLt3W-COdHZnsrBaN+KOpj83d0GyS+RY+QA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-01 9:42 ` Or Gerlitz [not found] ` <51FA2D98.5000707-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2013-08-03 22:12 ` Or Gerlitz [not found] ` <CAJZOPZ+_MbfUVjvdjuBBWexGx-++PRoW=7rb8bofaeTNGZqVPQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-05 18:40 ` Roland Dreier [not found] ` <CAL1RGDWoiO+eTjheA-CZrK_O0LrhbpDFxDn-vSWxZda3j9e+oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-05 21:02 ` Or Gerlitz [not found] ` <CAJZOPZJbjAC=u0vJdUFGO66KBunoZhX1b=y6UudTEgpi9mbA0g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-06 3:13 ` Roland Dreier [not found] ` <CAL1RGDWz81ZqkTjf74ske-i3U5yKH0z3aBr_cSxgcKeHo7Y1Lw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-06 13:14 ` Or Gerlitz [not found] ` <CAJZOPZ+6Lq8bMmzWttKQufuoZ2er9hejvT-Zz_CFxbfOvsmbOw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-06 17:40 ` Roland Dreier [not found] ` <CAL1RGDWtz4Ku+1KCphYEmg95tUw8BuKTWKOrRfgJ_3gUJOA8Lw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2013-08-06 20:22 ` Or Gerlitz
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.