* [PATCH v4 for-next 00/14] RoCE GID management
@ 2015-05-19 14:27 Matan Barak
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
Hi Doug,
RoCE type per GID patch-set aims to add a mechanism that supports
multiple GID types simultaneously while freeing the vendor from
managing its own GID table.
Previously, every vendor implemented its net device notifiers in its own
driver. This introduces a huge code duplication as figuring
whether an event is related to the vendor's net device in the
various cases (bonding, vlan or any other upper device) is
similar for all vendors. In the future, when multiple GID types will
be supported, this code duplication would have gotten even worse.
Therefore, we decided moving this into a common core core.
roce_gid_table and roce_gid_mgmt were created in order to store and
manage the new GID table, by filling it when getting the related events.
Vendors now only have to implement modify_gid and get_netdev IB
device calls, which are truly unique for each vendor.
roce_gid_table is implemented as IB client that manages the GID
table of the IB device. Each GID is associated with a GID type and a
network device (which is mandatory for management of the GID table).
The GID table is populated by using roce_gid_mgmt. roce_gid_mgmt
registers to net device/inet/inet events and calls roce_gid_table
in order to populate the GID table accordingly.
Patch 0001 creates a new infrastructure for storing GIDs and their attributes
in IB/core. This infrastructure support lock-less read of GIDs using a
seqcounter. The data structure is initialized only for RoCE ports.
Every gid has meta information describes its related net device and its
type.
Patch 0002 replaces the locking schema for IB devices. Previously, device_mutex
was used in order to lock the devices/clients list against every modification.
However, downstream patches add new functions which iterate over the device
list. Those functions could be executed for a workqueue contexts on behalf
of IB clients. Thus, when a client is removed, we need to wait for all works
to be finished. Since a client removal was done in device_mutex lock, we'll
be in fact waiting for a work which requires to lock the device_mutex itself
(=DEADLOCK). In order to mitigate this problem, we use rw semaphore to allow
multiple readers. We use modify_mutex in order to solve races between adding
(or removing) a client and a device simultaneously, which could have resulted
in calling client->add (or client->remove) twice for the same device and client.
Patches 0003, 0004 and 0006 add population of this table for various cases
based on net device events. We always enable default gids for an active
device (an active device is defined here as a device that doesn't have
a bonding master or is the current active slave). This is done in order
to allow loopback traffic. Patch 0006 adds proper bonding support -
only the active slaves retain their master's IP based gids and default gids.
Patch 0005 adds the required information for the bonding case.
This whole concept needs to fit the existing sysfs model, thus patch 0008
adds sysfs entries that represent the net device and gid type related to
each gid.
Patch 0009 adds a new API for RoCE gid table lookup. Since users might
want to find a GID which matches a net device with a specific attributes,
the new API allows them to pass a filter function. This function is a bit
slower than the "regular" find by gid, gid_type, if_index and namespace -
thus it should be used only when necessary.
Patches 0007 and 0010 changes the rest of IB/core to fit the new
model. Instead of storing smac and vlan, we store either if_index, net and
gid or sgid_index. Either set suffices in order to resolve all
the required Ethernet parameters. ib_init_ah_from_wc was changed, such
that when a wc is arrived, we search our RoCE gid table in order to
find a suitable sgid_index that matches the net device. Matching is
done based on GID and VLAN.
The rest of the patches add support for ocrdma and mlx4 devices.
Thanks,
Devesh, Somnath, Moni and Matan
Changes from V3:
(1) Remove RoCE V2 functionality (it will be sent at later patchset).
(2) Instead of removing qp_attr_mask flags, reserve them.
(3) Remove the kref from IB devices in favor of rwsem.
(4) Change the name of roce_gid_cache to roce_gid_table.
(5) Fix a race when roce_gid_table is free'd while getting events.
(6) Remove the roce_gid_cache active/inactive flag/API.
Changes from V2:
(1) When creating multiple vlans over an interface,
only the last created vlan's GID was populated in the table
(regression from V2).
(2) Inactive slave of bonding sometimes lost GIDs related to IPs
that were directly applied to it.
(3) Memory leak in mlx4
(4) roce_gid_cache now calls modify_gid with zgid in order to cause
the provider to delete all the information it allocated for those
GIDs.
(4) A mlx4 patch didn't compile and a downstream patch fixed it.
(5) cma_configfs should depend on both address translation and configfs.
(6) ocrdma driver redefined zgid.
(7) Added event information for NETDEV_CHANGEUPPER event.
Changes from V1:
(1) Addressed Shachar and Haggai's comments
(2) Fixed multicast support
(3) Generalized bonding support
(4) Added default GID after the IB device's net device was removed from bonding
(5) Fixed bugs in mlx4 implementation regarding multicast
(6) Fixed bugs in mlx4 when using XRC QPs after this patchset was applied
(7) Fixed bug when the RoCE gid cache didn't exist
(8) Moved the bonding's DRV macros to a private header
(9) Support non-configfs configurations
Matan Barak (10):
IB/core: Add RoCE GID table
IB/core: Replace device_mutex with rwsem
IB/core: Add RoCE GID population
IB/core: Add default GID for RoCE GID table
net: Add info for NETDEV_CHANGEUPPER event
IB/core: Add RoCE table bonding support
IB/core: GID attribute should be returned from verbs API and cache API
IB/core: Report gid_type and gid_ndev through sysfs
IB/core: Support find sgid index using a filter function
IB/core: Modify ib_verbs and cma in order to use roce_gid_table
Moni Shoua (3):
net/mlx4: Postpone the registration of net_device
IB/mlx4: Implement ib_device callbacks
IB/mlx4: Replace mechanism for RoCE GID management
Somnath Kotur (1):
RDMA/ocrdma: Changes in driver to incorporate the moving of GID Table
mgmt to IB/Core.
drivers/infiniband/core/Makefile | 3 +-
drivers/infiniband/core/addr.c | 3 +-
drivers/infiniband/core/cache.c | 313 ++++++++--
drivers/infiniband/core/cm.c | 36 +-
drivers/infiniband/core/cma.c | 93 +--
drivers/infiniband/core/core_priv.h | 75 ++-
drivers/infiniband/core/device.c | 162 ++++-
drivers/infiniband/core/mad.c | 2 +-
drivers/infiniband/core/multicast.c | 3 +-
drivers/infiniband/core/roce_gid_mgmt.c | 782 +++++++++++++++++++++++++
drivers/infiniband/core/roce_gid_table.c | 771 ++++++++++++++++++++++++
drivers/infiniband/core/sa_query.c | 11 +-
drivers/infiniband/core/sysfs.c | 186 +++++-
drivers/infiniband/core/ucma.c | 1 -
drivers/infiniband/core/uverbs_cmd.c | 3 +-
drivers/infiniband/core/uverbs_marshall.c | 4 +-
drivers/infiniband/core/verbs.c | 161 +++--
drivers/infiniband/hw/mlx4/ah.c | 17 +-
drivers/infiniband/hw/mlx4/mad.c | 12 +-
drivers/infiniband/hw/mlx4/main.c | 711 +++++++---------------
drivers/infiniband/hw/mlx4/mcg.c | 2 +-
drivers/infiniband/hw/mlx4/mlx4_ib.h | 24 +-
drivers/infiniband/hw/mlx4/qp.c | 63 +-
drivers/infiniband/hw/mthca/mthca_av.c | 2 +-
drivers/infiniband/hw/ocrdma/ocrdma.h | 11 +
drivers/infiniband/hw/ocrdma/ocrdma_ah.c | 20 +-
drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 22 +-
drivers/infiniband/hw/ocrdma/ocrdma_main.c | 233 +-------
drivers/infiniband/hw/ocrdma/ocrdma_sli.h | 13 +
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 31 +-
drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 4 +
drivers/infiniband/ulp/ipoib/ipoib_main.c | 2 +-
drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 2 +-
drivers/infiniband/ulp/srp/ib_srp.c | 2 +-
drivers/infiniband/ulp/srpt/ib_srpt.c | 3 +-
drivers/net/bonding/bond_options.c | 13 -
drivers/net/ethernet/mellanox/mlx4/en_main.c | 36 +-
drivers/net/ethernet/mellanox/mlx4/intf.c | 3 +
include/linux/mlx4/device.h | 3 +-
include/linux/mlx4/driver.h | 1 +
include/linux/netdevice.h | 14 +
include/net/addrconf.h | 31 +
include/net/bonding.h | 7 +
include/rdma/ib_addr.h | 4 +-
include/rdma/ib_cache.h | 71 ++-
include/rdma/ib_sa.h | 4 +-
include/rdma/ib_verbs.h | 85 ++-
net/core/dev.c | 12 +-
net/ipv6/addrconf.c | 31 -
49 files changed, 3030 insertions(+), 1068 deletions(-)
create mode 100644 drivers/infiniband/core/roce_gid_mgmt.c
create mode 100644 drivers/infiniband/core/roce_gid_table.c
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 01/14] IB/core: Add RoCE GID table
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-05-19 14:27 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 02/14] IB/core: Replace device_mutex with rwsem Matan Barak
` (12 subsequent siblings)
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
In order to manage multiple types, vlans and MACs per GID, we
need to store them along the GID itself. We store the net device
as well, as sometimes GIDs should be handled according to the
net device they came from. Since populating the GID table should
be identical for every RoCE provider, the GIDs table should be
handled in ib_core.
Adding a GID table that supports a lockless find, add and
delete gids. The lockless nature comes from using a unique
sequence number per table entry and detecting that while reading/
writing this sequence wasn't changed.
By using this RoCE GID table, providers must implement a
modify_gid callback. The table is managed exclusively by
this roce_gid_table and the provider just need to write
the data to the hardware.
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/core/Makefile | 3 +-
drivers/infiniband/core/core_priv.h | 22 ++
drivers/infiniband/core/roce_gid_table.c | 492 +++++++++++++++++++++++++++++++
drivers/infiniband/hw/mlx4/main.c | 2 -
include/rdma/ib_verbs.h | 54 +++-
5 files changed, 569 insertions(+), 4 deletions(-)
create mode 100644 drivers/infiniband/core/roce_gid_table.c
diff --git a/drivers/infiniband/core/Makefile b/drivers/infiniband/core/Makefile
index acf7367..fbeb72a 100644
--- a/drivers/infiniband/core/Makefile
+++ b/drivers/infiniband/core/Makefile
@@ -9,7 +9,8 @@ obj-$(CONFIG_INFINIBAND_USER_ACCESS) += ib_uverbs.o ib_ucm.o \
$(user_access-y)
ib_core-y := packer.o ud_header.o verbs.o sysfs.o \
- device.o fmr_pool.o cache.o netlink.o
+ device.o fmr_pool.o cache.o netlink.o \
+ roce_gid_table.o
ib_core-$(CONFIG_INFINIBAND_USER_MEM) += umem.o
ib_core-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += umem_odp.o umem_rbtree.o
diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h
index 87d1936..e7c7a7c 100644
--- a/drivers/infiniband/core/core_priv.h
+++ b/drivers/infiniband/core/core_priv.h
@@ -35,6 +35,7 @@
#include <linux/list.h>
#include <linux/spinlock.h>
+#include <net/net_namespace.h>
#include <rdma/ib_verbs.h>
@@ -51,4 +52,25 @@ void ib_cache_cleanup(void);
int ib_resolve_eth_l2_attrs(struct ib_qp *qp,
struct ib_qp_attr *qp_attr, int *qp_attr_mask);
+
+int roce_gid_table_get_gid(struct ib_device *ib_dev, u8 port, int index,
+ union ib_gid *gid, struct ib_gid_attr *attr);
+
+int roce_gid_table_find_gid(struct ib_device *ib_dev, union ib_gid *gid,
+ enum ib_gid_type gid_type, struct net *net,
+ int if_index, u8 *port, u16 *index);
+
+int roce_gid_table_find_gid_by_port(struct ib_device *ib_dev, union ib_gid *gid,
+ enum ib_gid_type gid_type, u8 port,
+ struct net *net, int if_index, u16 *index);
+
+int roce_add_gid(struct ib_device *ib_dev, u8 port,
+ union ib_gid *gid, struct ib_gid_attr *attr);
+
+int roce_del_gid(struct ib_device *ib_dev, u8 port,
+ union ib_gid *gid, struct ib_gid_attr *attr);
+
+int roce_del_all_netdev_gids(struct ib_device *ib_dev, u8 port,
+ struct net_device *ndev);
+
#endif /* _CORE_PRIV_H */
diff --git a/drivers/infiniband/core/roce_gid_table.c b/drivers/infiniband/core/roce_gid_table.c
new file mode 100644
index 0000000..30e9e04
--- /dev/null
+++ b/drivers/infiniband/core/roce_gid_table.c
@@ -0,0 +1,492 @@
+/*
+ * Copyright (c) 2015, Mellanox Technologies inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <linux/slab.h>
+#include <linux/netdevice.h>
+#include <linux/rtnetlink.h>
+#include <rdma/ib_cache.h>
+
+#include "core_priv.h"
+
+union ib_gid zgid;
+EXPORT_SYMBOL_GPL(zgid);
+
+static const struct ib_gid_attr zattr;
+
+enum gid_attr_find_mask {
+ GID_ATTR_FIND_MASK_GID = 1UL << 0,
+ GID_ATTR_FIND_MASK_GID_TYPE = 1UL << 1,
+ GID_ATTR_FIND_MASK_NETDEV = 1UL << 2,
+};
+
+static inline int start_port(struct ib_device *ib_dev)
+{
+ return (ib_dev->node_type == RDMA_NODE_IB_SWITCH) ? 0 : 1;
+}
+
+struct dev_put_rcu {
+ struct rcu_head rcu;
+ struct net_device *ndev;
+};
+
+static void put_ndev(struct rcu_head *rcu)
+{
+ struct dev_put_rcu *put_rcu =
+ container_of(rcu, struct dev_put_rcu, rcu);
+
+ dev_put(put_rcu->ndev);
+ kfree(put_rcu);
+}
+
+static int write_gid(struct ib_device *ib_dev, u8 port,
+ struct ib_roce_gid_table *table, int ix,
+ const union ib_gid *gid,
+ const struct ib_gid_attr *attr)
+{
+ int ret;
+ struct dev_put_rcu *put_rcu;
+ struct net_device *old_net_dev;
+
+ write_seqcount_begin(&table->data_vec[ix].seq);
+
+ ret = ib_dev->modify_gid(ib_dev, port, ix, gid, attr,
+ &table->data_vec[ix].context);
+
+ old_net_dev = table->data_vec[ix].attr.ndev;
+ if (old_net_dev && old_net_dev != attr->ndev) {
+ put_rcu = kmalloc(sizeof(*put_rcu), GFP_KERNEL);
+ if (put_rcu) {
+ put_rcu->ndev = old_net_dev;
+ call_rcu(&put_rcu->rcu, put_ndev);
+ } else {
+ pr_warn("roce_gid_table: can't allocate rcu context, using synchronize\n");
+ synchronize_rcu();
+ dev_put(old_net_dev);
+ }
+ }
+ /* if modify_gid failed, just delete the old gid */
+ if (ret || !memcmp(gid, &zgid, sizeof(*gid))) {
+ gid = &zgid;
+ attr = &zattr;
+ table->data_vec[ix].context = NULL;
+ }
+ memcpy(&table->data_vec[ix].gid, gid, sizeof(*gid));
+ memcpy(&table->data_vec[ix].attr, attr, sizeof(*attr));
+ if (table->data_vec[ix].attr.ndev &&
+ table->data_vec[ix].attr.ndev != old_net_dev)
+ dev_hold(table->data_vec[ix].attr.ndev);
+
+ write_seqcount_end(&table->data_vec[ix].seq);
+
+ if (!ret) {
+ struct ib_event event;
+
+ event.device = ib_dev;
+ event.element.port_num = port;
+ event.event = IB_EVENT_GID_CHANGE;
+
+ ib_dispatch_event(&event);
+ }
+ return ret;
+}
+
+static int find_gid(struct ib_roce_gid_table *table, union ib_gid *gid,
+ const struct ib_gid_attr *val, unsigned long mask)
+{
+ int i;
+
+ for (i = 0; i < table->sz; i++) {
+ struct ib_gid_attr *attr = &table->data_vec[i].attr;
+ unsigned int orig_seq = read_seqcount_begin(&table->data_vec[i].seq);
+
+ if (mask & GID_ATTR_FIND_MASK_GID_TYPE &&
+ attr->gid_type != val->gid_type)
+ continue;
+
+ if (memcmp(gid, &table->data_vec[i].gid, sizeof(*gid)))
+ continue;
+
+ if (mask & GID_ATTR_FIND_MASK_NETDEV &&
+ attr->ndev != val->ndev)
+ continue;
+
+ if (!read_seqcount_retry(&table->data_vec[i].seq, orig_seq))
+ return i;
+ /* The sequence number changed under our feet,
+ * the GID entry is invalid. Continue to the
+ * next entry.
+ */
+ }
+
+ return -1;
+}
+
+int roce_add_gid(struct ib_device *ib_dev, u8 port,
+ union ib_gid *gid, struct ib_gid_attr *attr)
+{
+ struct ib_roce_gid_table **ports_table =
+ READ_ONCE(ib_dev->cache.roce_gid_table);
+ struct ib_roce_gid_table *table;
+ int ix;
+ int ret = 0;
+
+ /* make sure we read the ports_table */
+ smp_rmb();
+
+ if (!ports_table)
+ return -EOPNOTSUPP;
+
+ table = ports_table[port - start_port(ib_dev)];
+
+ if (!table)
+ return -EPROTONOSUPPORT;
+
+ if (!memcmp(gid, &zgid, sizeof(*gid)))
+ return -EINVAL;
+
+ mutex_lock(&table->lock);
+
+ ix = find_gid(table, gid, attr, GID_ATTR_FIND_MASK_GID_TYPE |
+ GID_ATTR_FIND_MASK_NETDEV);
+ if (ix >= 0)
+ goto out_unlock;
+
+ ix = find_gid(table, &zgid, NULL, 0);
+ if (ix < 0) {
+ ret = -ENOSPC;
+ goto out_unlock;
+ }
+
+ write_gid(ib_dev, port, table, ix, gid, attr);
+
+out_unlock:
+ mutex_unlock(&table->lock);
+ return ret;
+}
+
+int roce_del_gid(struct ib_device *ib_dev, u8 port,
+ union ib_gid *gid, struct ib_gid_attr *attr)
+{
+ struct ib_roce_gid_table **ports_table =
+ READ_ONCE(ib_dev->cache.roce_gid_table);
+ struct ib_roce_gid_table *table;
+ int ix;
+
+ /* make sure we read the ports_table */
+ smp_rmb();
+
+ if (!ports_table)
+ return 0;
+
+ table = ports_table[port - start_port(ib_dev)];
+
+ if (!table)
+ return -EPROTONOSUPPORT;
+
+ mutex_lock(&table->lock);
+
+ ix = find_gid(table, gid, attr,
+ GID_ATTR_FIND_MASK_GID_TYPE |
+ GID_ATTR_FIND_MASK_NETDEV);
+ if (ix < 0)
+ goto out_unlock;
+
+ write_gid(ib_dev, port, table, ix, &zgid, &zattr);
+
+out_unlock:
+ mutex_unlock(&table->lock);
+ return 0;
+}
+
+int roce_del_all_netdev_gids(struct ib_device *ib_dev, u8 port,
+ struct net_device *ndev)
+{
+ struct ib_roce_gid_table **ports_table =
+ READ_ONCE(ib_dev->cache.roce_gid_table);
+ struct ib_roce_gid_table *table;
+ int ix;
+
+ /* make sure we read the ports_table */
+ smp_rmb();
+
+ if (!ports_table)
+ return 0;
+
+ table = ports_table[port - start_port(ib_dev)];
+
+ if (!table)
+ return -EPROTONOSUPPORT;
+
+ mutex_lock(&table->lock);
+
+ for (ix = 0; ix < table->sz; ix++)
+ if (table->data_vec[ix].attr.ndev == ndev)
+ write_gid(ib_dev, port, table, ix, &zgid, &zattr);
+
+ mutex_unlock(&table->lock);
+ return 0;
+}
+
+int roce_gid_table_get_gid(struct ib_device *ib_dev, u8 port, int index,
+ union ib_gid *gid, struct ib_gid_attr *attr)
+{
+ struct ib_roce_gid_table **ports_table =
+ READ_ONCE(ib_dev->cache.roce_gid_table);
+ struct ib_roce_gid_table *table;
+ union ib_gid local_gid;
+ struct ib_gid_attr local_attr;
+ unsigned int orig_seq;
+
+ /* make sure we read the ports_table */
+ smp_rmb();
+
+ if (!ports_table)
+ return -EOPNOTSUPP;
+
+ table = ports_table[port - start_port(ib_dev)];
+
+ if (!table)
+ return -EPROTONOSUPPORT;
+
+ if (index < 0 || index >= table->sz)
+ return -EINVAL;
+
+ orig_seq = read_seqcount_begin(&table->data_vec[index].seq);
+
+ memcpy(&local_gid, &table->data_vec[index].gid, sizeof(local_gid));
+ memcpy(&local_attr, &table->data_vec[index].attr, sizeof(local_attr));
+
+ if (read_seqcount_retry(&table->data_vec[index].seq, orig_seq))
+ return -EAGAIN;
+
+ memcpy(gid, &local_gid, sizeof(*gid));
+ if (attr)
+ memcpy(attr, &local_attr, sizeof(*attr));
+ return 0;
+}
+
+static int _roce_gid_table_find_gid(struct ib_device *ib_dev, union ib_gid *gid,
+ const struct ib_gid_attr *val,
+ unsigned long mask,
+ u8 *port, u16 *index)
+{
+ struct ib_roce_gid_table **ports_table =
+ READ_ONCE(ib_dev->cache.roce_gid_table);
+ struct ib_roce_gid_table *table;
+ u8 p;
+ int local_index;
+
+ /* make sure we read the ports_table */
+ smp_rmb();
+
+ if (!ports_table)
+ return -ENOENT;
+
+ for (p = 0; p < ib_dev->phys_port_cnt; p++) {
+ if (rdma_port_get_link_layer(ib_dev, p + start_port(ib_dev)) !=
+ IB_LINK_LAYER_ETHERNET)
+ continue;
+ table = ports_table[p];
+ if (!table)
+ continue;
+ local_index = find_gid(table, gid, val, mask);
+ if (local_index >= 0) {
+ if (index)
+ *index = local_index;
+ if (port)
+ *port = p + start_port(ib_dev);
+ return 0;
+ }
+ }
+
+ return -ENOENT;
+}
+
+static int get_netdev_from_ifindex(struct net *net, int if_index,
+ struct ib_gid_attr *gid_attr_val)
+{
+ if (if_index && net) {
+ rcu_read_lock();
+ gid_attr_val->ndev = dev_get_by_index_rcu(net, if_index);
+ rcu_read_unlock();
+ if (gid_attr_val->ndev)
+ return GID_ATTR_FIND_MASK_NETDEV;
+ }
+ return 0;
+}
+
+int roce_gid_table_find_gid(struct ib_device *ib_dev, union ib_gid *gid,
+ enum ib_gid_type gid_type, struct net *net,
+ int if_index, u8 *port, u16 *index)
+{
+ unsigned long mask = GID_ATTR_FIND_MASK_GID |
+ GID_ATTR_FIND_MASK_GID_TYPE;
+ struct ib_gid_attr gid_attr_val = {.gid_type = gid_type};
+
+ mask |= get_netdev_from_ifindex(net, if_index, &gid_attr_val);
+
+ return _roce_gid_table_find_gid(ib_dev, gid, &gid_attr_val,
+ mask, port, index);
+}
+
+int roce_gid_table_find_gid_by_port(struct ib_device *ib_dev, union ib_gid *gid,
+ enum ib_gid_type gid_type, u8 port,
+ struct net *net, int if_index, u16 *index)
+{
+ int local_index;
+ struct ib_roce_gid_table **ports_table =
+ READ_ONCE(ib_dev->cache.roce_gid_table);
+ struct ib_roce_gid_table *table;
+ unsigned long mask = GID_ATTR_FIND_MASK_GID_TYPE;
+ struct ib_gid_attr val = {.gid_type = gid_type};
+
+ /* make sure we read the ports_table */
+ smp_rmb();
+
+ if (!ports_table || port < start_port(ib_dev) ||
+ port >= (start_port(ib_dev) + ib_dev->phys_port_cnt))
+ return -ENOENT;
+
+ table = ports_table[port - start_port(ib_dev)];
+ if (!table)
+ return -ENOENT;
+
+ mask |= get_netdev_from_ifindex(net, if_index, &val);
+
+ local_index = find_gid(table, gid, &val, mask);
+ if (local_index >= 0) {
+ if (index)
+ *index = local_index;
+ return 0;
+ }
+
+ return -ENOENT;
+}
+
+static struct ib_roce_gid_table *alloc_roce_gid_table(int sz)
+{
+ unsigned int i;
+ struct ib_roce_gid_table *table =
+ kzalloc(sizeof(struct ib_roce_gid_table), GFP_KERNEL);
+ if (!table)
+ return NULL;
+
+ table->data_vec = kcalloc(sz, sizeof(*table->data_vec), GFP_KERNEL);
+ if (!table->data_vec)
+ goto err_free_table;
+
+ mutex_init(&table->lock);
+
+ table->sz = sz;
+
+ for (i = 0; i < sz; i++)
+ seqcount_init(&table->data_vec[i].seq);
+
+ return table;
+
+err_free_table:
+ kfree(table);
+ return NULL;
+}
+
+static void free_roce_gid_table(struct ib_device *ib_dev, u8 port,
+ struct ib_roce_gid_table *table)
+{
+ int i;
+
+ if (!table)
+ return;
+
+ for (i = 0; i < table->sz; ++i) {
+ if (memcmp(&table->data_vec[i].gid, &zgid,
+ sizeof(table->data_vec[i].gid)))
+ write_gid(ib_dev, port, table, i, &zgid, &zattr);
+ }
+ kfree(table->data_vec);
+ kfree(table);
+}
+
+static int roce_gid_table_setup_one(struct ib_device *ib_dev)
+{
+ u8 port;
+ struct ib_roce_gid_table **table;
+ int err = 0;
+
+ if (!ib_dev->modify_gid)
+ return -EOPNOTSUPP;
+
+ table = kcalloc(ib_dev->phys_port_cnt, sizeof(*table), GFP_KERNEL);
+
+ if (!table) {
+ pr_warn("failed to allocate roce addr table for %s\n",
+ ib_dev->name);
+ return -ENOMEM;
+ }
+
+ for (port = 0; port < ib_dev->phys_port_cnt; port++) {
+ if (rdma_port_get_link_layer(ib_dev, port + start_port(ib_dev))
+ != IB_LINK_LAYER_ETHERNET)
+ continue;
+ table[port] =
+ alloc_roce_gid_table(ib_dev->gid_tbl_len[port]);
+ if (!table[port]) {
+ err = -ENOMEM;
+ goto rollback_table_setup;
+ }
+ }
+
+ ib_dev->cache.roce_gid_table = table;
+ return 0;
+
+rollback_table_setup:
+ for (port = 1; port <= ib_dev->phys_port_cnt; port++)
+ free_roce_gid_table(ib_dev, port, table[port]);
+
+ kfree(table);
+ return err;
+}
+
+static void roce_gid_table_cleanup_one(struct ib_device *ib_dev,
+ struct ib_roce_gid_table **table)
+{
+ u8 port;
+
+ if (!table)
+ return;
+
+ for (port = 0; port < ib_dev->phys_port_cnt; port++)
+ free_roce_gid_table(ib_dev, port + start_port(ib_dev),
+ table[port]);
+
+ kfree(table);
+}
+
diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 57070c5..abd0cf74 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -93,8 +93,6 @@ static void init_query_mad(struct ib_smp *mad)
mad->method = IB_MGMT_METHOD_GET;
}
-static union ib_gid zgid;
-
static int check_flow_steering_support(struct mlx4_dev *dev)
{
int eth_num_ports = 0;
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 65994a1..5cf40f4 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -64,6 +64,35 @@ union ib_gid {
} global;
};
+extern union ib_gid zgid;
+
+enum ib_gid_type {
+ /* If link layer is Ethernet, this is RoCE V1 */
+ IB_GID_TYPE_IB = 0,
+ IB_GID_TYPE_ROCE = 0,
+ IB_GID_TYPE_SIZE
+};
+
+struct ib_gid_attr {
+ enum ib_gid_type gid_type;
+ struct net_device *ndev;
+};
+
+struct ib_roce_gid_table_entry {
+ seqcount_t seq;
+ union ib_gid gid;
+ struct ib_gid_attr attr;
+ void *context;
+};
+
+struct ib_roce_gid_table {
+ int active;
+ int sz;
+ /* locking against multiple writes in data_vec */
+ struct mutex lock;
+ struct ib_roce_gid_table_entry *data_vec;
+};
+
enum rdma_node_type {
/* IB values map to NodeInfo:NodeType. */
RDMA_NODE_IB_CA = 1,
@@ -265,7 +294,8 @@ enum ib_port_cap_flags {
IB_PORT_BOOT_MGMT_SUP = 1 << 23,
IB_PORT_LINK_LATENCY_SUP = 1 << 24,
IB_PORT_CLIENT_REG_SUP = 1 << 25,
- IB_PORT_IP_BASED_GIDS = 1 << 26
+ IB_PORT_IP_BASED_GIDS = 1 << 26,
+ IB_PORT_ROCE = 1 << 27,
};
enum ib_port_width {
@@ -1431,6 +1461,7 @@ struct ib_cache {
struct ib_pkey_cache **pkey_cache;
struct ib_gid_cache **gid_cache;
u8 *lmc_cache;
+ struct ib_roce_gid_table **roce_gid_table;
};
struct ib_dma_mapping_ops {
@@ -1506,6 +1537,27 @@ struct ib_device {
int (*query_gid)(struct ib_device *device,
u8 port_num, int index,
union ib_gid *gid);
+ /* When calling modify_gid, the HW vendor's driver should
+ * modify the gid of device @device at gid index @index of
+ * port @port to be @gid. Meta-info of that gid (for example,
+ * the network device related to this gid is available
+ * at @attr. @context allows the HW vendor driver to store extra
+ * information together with a GID entry. The HW vendor may allocate
+ * memory to contain this information and store it in @context when a
+ * new GID entry is written to. Upon the deletion of a GID entry,
+ * the HW vendor must free any allocated memory. The caller will clear
+ * @context afterwards.GID deletion is done by passing the zero gid.
+ * Params are consistent until the next call of modify_gid.
+ * The function should return 0 on success or error otherwise.
+ * The function could be called concurrently for different ports.
+ * This function is only called when roce_gid_table is used.
+ */
+ int (*modify_gid)(struct ib_device *device,
+ u8 port_num,
+ unsigned int index,
+ const union ib_gid *gid,
+ const struct ib_gid_attr *attr,
+ void **context);
int (*query_pkey)(struct ib_device *device,
u8 port_num, u16 index, u16 *pkey);
int (*modify_device)(struct ib_device *device,
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 02/14] IB/core: Replace device_mutex with rwsem
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-05-19 14:27 ` [PATCH v4 for-next 01/14] IB/core: Add RoCE GID table Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
[not found] ` <1432045637-9090-3-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-05-19 14:27 ` [PATCH v4 for-next 03/14] IB/core: Add RoCE GID population Matan Barak
` (11 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
device_mutex is replaced rwsem in order to allow
clients' work to run concurrently with register_device,
unregister_device, register_client and unregister_client.
Clients might want to iterate the devices list in work
contexts. Saying that, a client must finish its operations
on a device when client->remove(device) is called. Finishing
the client's operation requires waiting for all works it
created regarding this device. However, a work could
depend on the device_mutex to iterate the devices list
and we would have got a deadlock. Replacing this lock
with rwlock solves this problem.
rwlock isn't enough in this scenario. This is because running
register_client and register_device could result in running
client->add(device) twice (if we protect the list with down_read).
A similar problem arises for unregister_client and unregister_device
concurrently. In order to solve this, we mutually exclude those
operations with a mutex.
This follows Jason Gunthorpe's suggestion in:
http://www.spinics.net/lists/linux-rdma/msg25174.html
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/core/device.c | 55 ++++++++++++++++++++++++++++++----------
1 file changed, 41 insertions(+), 14 deletions(-)
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 18c1ece..bf19358 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -59,13 +59,27 @@ static LIST_HEAD(device_list);
static LIST_HEAD(client_list);
/*
- * device_mutex protects access to both device_list and client_list.
- * There's no real point to using multiple locks or something fancier
- * like an rwsem: we always access both lists, and we're always
- * modifying one list or the other list. In any case this is not a
- * hot path so there's no point in trying to optimize.
+ * modify_mutex protects adding/removing devices and clients.
+ * Adding a client and a device simultaneously might cause calling
+ * client->add twice for the same device. Removing a client could
+ * cause calling client->remove twice. modify_mutex synchronizes
+ * those actions.
*/
-static DEFINE_MUTEX(device_mutex);
+static DEFINE_MUTEX(modify_mutex);
+
+/*
+ * lists_rwsem protects the client and device lists from
+ * read-while-write scenario. Adding/removing a device/client
+ * is protected by modify_mutex. However, other contexts of
+ * clients might want to iterate the device list. Client should
+ * cease all actions after client->remove returns. For that purpose,
+ * clients need to wait for their contexts which might need to
+ * iterate that list. Summing all that up, we have to use RCU/rwsem
+ * as either the iteration over client->remove could deadlock with
+ * the clients' works. We don't expect adding/removing of devices/clients
+ * to be done frequently, thus rwsem is favored here.
+ */
+static DECLARE_RWSEM(lists_rwsem);
static int ib_device_check_mandatory(struct ib_device *device)
{
@@ -276,7 +290,7 @@ int ib_register_device(struct ib_device *device,
{
int ret;
- mutex_lock(&device_mutex);
+ mutex_lock(&modify_mutex);
if (strchr(device->name, '%')) {
ret = alloc_name(device->name);
@@ -310,20 +324,24 @@ int ib_register_device(struct ib_device *device,
goto out;
}
+ down_write(&lists_rwsem);
list_add_tail(&device->core_list, &device_list);
+ up_write(&lists_rwsem);
+
device->reg_state = IB_DEV_REGISTERED;
{
struct ib_client *client;
+ /* down_read(&lists_rwsem) is implied by modify_mutex */
list_for_each_entry(client, &client_list, list)
if (client->add && !add_client_context(device, client))
client->add(device);
}
out:
- mutex_unlock(&device_mutex);
+ mutex_unlock(&modify_mutex);
return ret;
}
EXPORT_SYMBOL(ib_register_device);
@@ -340,18 +358,21 @@ void ib_unregister_device(struct ib_device *device)
struct ib_client_data *context, *tmp;
unsigned long flags;
- mutex_lock(&device_mutex);
+ mutex_lock(&modify_mutex);
+ /* down_read(&lists_rwsem) is implied by modify_mutex */
list_for_each_entry_reverse(client, &client_list, list)
if (client->remove)
client->remove(device);
+ down_write(&lists_rwsem);
list_del(&device->core_list);
+ up_write(&lists_rwsem);
kfree(device->gid_tbl_len);
kfree(device->pkey_tbl_len);
- mutex_unlock(&device_mutex);
+ mutex_unlock(&modify_mutex);
ib_device_unregister_sysfs(device);
@@ -381,14 +402,17 @@ int ib_register_client(struct ib_client *client)
{
struct ib_device *device;
- mutex_lock(&device_mutex);
+ mutex_lock(&modify_mutex);
+ down_write(&lists_rwsem);
list_add_tail(&client->list, &client_list);
+ up_write(&lists_rwsem);
+ /* down_read(&lists_rwsem) is implied by modify_mutex */
list_for_each_entry(device, &device_list, core_list)
if (client->add && !add_client_context(device, client))
client->add(device);
- mutex_unlock(&device_mutex);
+ mutex_unlock(&modify_mutex);
return 0;
}
@@ -408,8 +432,9 @@ void ib_unregister_client(struct ib_client *client)
struct ib_device *device;
unsigned long flags;
- mutex_lock(&device_mutex);
+ mutex_lock(&modify_mutex);
+ /* down_read(&lists_rwsem) is implied by modify_mutex */
list_for_each_entry(device, &device_list, core_list) {
if (client->remove)
client->remove(device);
@@ -422,9 +447,11 @@ void ib_unregister_client(struct ib_client *client)
}
spin_unlock_irqrestore(&device->client_data_lock, flags);
}
+ down_write(&lists_rwsem);
list_del(&client->list);
+ up_write(&lists_rwsem);
- mutex_unlock(&device_mutex);
+ mutex_unlock(&modify_mutex);
}
EXPORT_SYMBOL(ib_unregister_client);
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 03/14] IB/core: Add RoCE GID population
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-05-19 14:27 ` [PATCH v4 for-next 01/14] IB/core: Add RoCE GID table Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 02/14] IB/core: Replace device_mutex with rwsem Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 04/14] IB/core: Add default GID for RoCE GID table Matan Barak
` (10 subsequent siblings)
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
In order to populate the GID table, we need to listen for
events:
(a) IB device has been added or removed - used in order
to allocate/deallocate the table and populate
the GID table internally.
(b) inet events - add new GIDs (according to the IP addresses)
to the table.
(c) netdev up/down/change_addr - if a netdev is built onto our
RoCE device, we need to add/delete its IPs.
When an event is received, multiple entries (each with
different GID type) are added.
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/core/Makefile | 2 +-
drivers/infiniband/core/core_priv.h | 26 ++
drivers/infiniband/core/device.c | 78 +++++
drivers/infiniband/core/roce_gid_mgmt.c | 494 +++++++++++++++++++++++++++++++
drivers/infiniband/core/roce_gid_table.c | 52 ++++
include/rdma/ib_addr.h | 2 +-
include/rdma/ib_verbs.h | 8 +
7 files changed, 660 insertions(+), 2 deletions(-)
create mode 100644 drivers/infiniband/core/roce_gid_mgmt.c
diff --git a/drivers/infiniband/core/Makefile b/drivers/infiniband/core/Makefile
index fbeb72a..3ceb3f8 100644
--- a/drivers/infiniband/core/Makefile
+++ b/drivers/infiniband/core/Makefile
@@ -10,7 +10,7 @@ obj-$(CONFIG_INFINIBAND_USER_ACCESS) += ib_uverbs.o ib_ucm.o \
ib_core-y := packer.o ud_header.o verbs.o sysfs.o \
device.o fmr_pool.o cache.o netlink.o \
- roce_gid_table.o
+ roce_gid_table.o roce_gid_mgmt.o
ib_core-$(CONFIG_INFINIBAND_USER_MEM) += umem.o
ib_core-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += umem_odp.o umem_rbtree.o
diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h
index e7c7a7c..eb094f6 100644
--- a/drivers/infiniband/core/core_priv.h
+++ b/drivers/infiniband/core/core_priv.h
@@ -39,6 +39,8 @@
#include <rdma/ib_verbs.h>
+extern struct workqueue_struct *roce_gid_mgmt_wq;
+
int ib_device_register_sysfs(struct ib_device *device,
int (*port_callback)(struct ib_device *,
u8, struct kobject *));
@@ -53,6 +55,22 @@ void ib_cache_cleanup(void);
int ib_resolve_eth_l2_attrs(struct ib_qp *qp,
struct ib_qp_attr *qp_attr, int *qp_attr_mask);
+typedef void (*roce_netdev_callback)(struct ib_device *device, u8 port,
+ struct net_device *idev, void *cookie);
+
+typedef int (*roce_netdev_filter)(struct ib_device *device, u8 port,
+ struct net_device *idev, void *cookie);
+
+void ib_dev_roce_ports_of_netdev(struct ib_device *ib_dev,
+ roce_netdev_filter filter,
+ void *filter_cookie,
+ roce_netdev_callback cb,
+ void *cookie);
+void ib_enum_roce_ports_of_netdev(roce_netdev_filter filter,
+ void *filter_cookie,
+ roce_netdev_callback cb,
+ void *cookie);
+
int roce_gid_table_get_gid(struct ib_device *ib_dev, u8 port, int index,
union ib_gid *gid, struct ib_gid_attr *attr);
@@ -64,6 +82,9 @@ int roce_gid_table_find_gid_by_port(struct ib_device *ib_dev, union ib_gid *gid,
enum ib_gid_type gid_type, u8 port,
struct net *net, int if_index, u16 *index);
+int roce_gid_table_setup(void);
+void roce_gid_table_cleanup(void);
+
int roce_add_gid(struct ib_device *ib_dev, u8 port,
union ib_gid *gid, struct ib_gid_attr *attr);
@@ -73,4 +94,9 @@ int roce_del_gid(struct ib_device *ib_dev, u8 port,
int roce_del_all_netdev_gids(struct ib_device *ib_dev, u8 port,
struct net_device *ndev);
+int roce_gid_mgmt_init(void);
+void roce_gid_mgmt_cleanup(void);
+
+int roce_rescan_device(struct ib_device *ib_dev);
+
#endif /* _CORE_PRIV_H */
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index bf19358..697c715 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -39,6 +39,7 @@
#include <linux/init.h>
#include <linux/mutex.h>
#include <rdma/rdma_netlink.h>
+#include <rdma/ib_addr.h>
#include "core_priv.h"
@@ -626,6 +627,80 @@ int ib_query_gid(struct ib_device *device,
EXPORT_SYMBOL(ib_query_gid);
/**
+ * ib_dev_roce_ports_of_netdev - enumerate RoCE ports of ibdev in
+ * respect of netdev
+ * @ib_dev : IB device we want to query
+ * @filter: Should we call the callback?
+ * @filter_cookie: Cookie passed to filter
+ * @cb: Callback to call for each found RoCE ports
+ * @cookie: Cookie passed back to the callback
+ *
+ * Enumerates all of the physical RoCE ports of ib_dev RoCE ports
+ * which are relaying Ethernet packets to a specific
+ * (possibly virtual) netdevice according to filter.
+ */
+void ib_dev_roce_ports_of_netdev(struct ib_device *ib_dev,
+ roce_netdev_filter filter,
+ void *filter_cookie,
+ roce_netdev_callback cb,
+ void *cookie)
+{
+ u8 port;
+
+ if (ib_dev->modify_gid)
+ for (port = start_port(ib_dev); port <= end_port(ib_dev);
+ port++)
+ if (ib_dev->get_link_layer(ib_dev, port) ==
+ IB_LINK_LAYER_ETHERNET) {
+ struct net_device *idev = NULL;
+
+ rcu_read_lock();
+ if (ib_dev->get_netdev)
+ idev = ib_dev->get_netdev(ib_dev, port);
+
+ if (idev &&
+ idev->reg_state >= NETREG_UNREGISTERED)
+ idev = NULL;
+
+ if (idev)
+ dev_hold(idev);
+
+ rcu_read_unlock();
+
+ if (filter(ib_dev, port, idev, filter_cookie))
+ cb(ib_dev, port, idev, cookie);
+
+ if (idev)
+ dev_put(idev);
+ }
+}
+
+/**
+ * ib_enum_roce_ports_of_netdev - enumerate RoCE ports of a netdev
+ * @filter: Should we call the callback?
+ * @filter_cookie: Cookie passed to filter
+ * @cb: Callback to call for each found RoCE ports
+ * @cookie: Cookie passed back to the callback
+ *
+ * Enumerates all of the physical RoCE ports which are relaying
+ * Ethernet packets to a specific (possibly virtual) netdevice
+ * according to filter.
+ */
+void ib_enum_roce_ports_of_netdev(roce_netdev_filter filter,
+ void *filter_cookie,
+ roce_netdev_callback cb,
+ void *cookie)
+{
+ struct ib_device *dev;
+
+ down_read(&lists_rwsem);
+ list_for_each_entry_rcu(dev, &device_list, core_list)
+ ib_dev_roce_ports_of_netdev(dev, filter, filter_cookie, cb,
+ cookie);
+ up_read(&lists_rwsem);
+}
+
+/**
* ib_query_pkey - Get P_Key table entry
* @device:Device to query
* @port_num:Port number to query
@@ -780,6 +855,8 @@ static int __init ib_core_init(void)
goto err_sysfs;
}
+ roce_gid_table_setup();
+
ret = ib_cache_setup();
if (ret) {
printk(KERN_WARNING "Couldn't set up InfiniBand P_Key/GID cache\n");
@@ -801,6 +878,7 @@ err:
static void __exit ib_core_cleanup(void)
{
+ roce_gid_table_cleanup();
ib_cache_cleanup();
ibnl_cleanup();
ib_sysfs_cleanup();
diff --git a/drivers/infiniband/core/roce_gid_mgmt.c b/drivers/infiniband/core/roce_gid_mgmt.c
new file mode 100644
index 0000000..9aff044
--- /dev/null
+++ b/drivers/infiniband/core/roce_gid_mgmt.c
@@ -0,0 +1,494 @@
+/*
+ * Copyright (c) 2015, Mellanox Technologies inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include "core_priv.h"
+
+#include <linux/in.h>
+#include <linux/in6.h>
+
+/* For in6_dev_get/in6_dev_put */
+#include <net/addrconf.h>
+
+#include <rdma/ib_cache.h>
+#include <rdma/ib_addr.h>
+
+struct workqueue_struct *roce_gid_mgmt_wq;
+
+enum gid_op_type {
+ GID_DEL = 0,
+ GID_ADD
+};
+
+struct update_gid_event_work {
+ struct work_struct work;
+ union ib_gid gid;
+ struct ib_gid_attr gid_attr;
+ enum gid_op_type gid_op;
+};
+
+#define ROCE_NETDEV_CALLBACK_SZ 2
+struct netdev_event_work_cmd {
+ roce_netdev_callback cb;
+ roce_netdev_filter filter;
+};
+
+struct netdev_event_work {
+ struct work_struct work;
+ struct netdev_event_work_cmd cmds[ROCE_NETDEV_CALLBACK_SZ];
+ struct net_device *ndev;
+};
+
+static const struct {
+ int flag_mask;
+ enum ib_gid_type gid_type;
+} PORT_CAP_TO_GID_TYPE[] = {
+ {IB_PORT_ROCE, IB_GID_TYPE_ROCE},
+};
+
+#define CAP_TO_GID_TABLE_SIZE ARRAY_SIZE(PORT_CAP_TO_GID_TYPE)
+
+static void update_gid(enum gid_op_type gid_op, struct ib_device *ib_dev,
+ u8 port, union ib_gid *gid,
+ struct ib_gid_attr *gid_attr)
+{
+ struct ib_port_attr pattr;
+ int i;
+ int err;
+
+ err = ib_query_port(ib_dev, port, &pattr);
+ if (err) {
+ pr_warn("update_gid: ib_query_port() failed for %s, %d\n",
+ ib_dev->name, err);
+ }
+
+ for (i = 0; i < CAP_TO_GID_TABLE_SIZE; i++) {
+ if (pattr.port_cap_flags & PORT_CAP_TO_GID_TYPE[i].flag_mask) {
+ gid_attr->gid_type =
+ PORT_CAP_TO_GID_TYPE[i].gid_type;
+ switch (gid_op) {
+ case GID_ADD:
+ roce_add_gid(ib_dev, port,
+ gid, gid_attr);
+ break;
+ case GID_DEL:
+ roce_del_gid(ib_dev, port,
+ gid, gid_attr);
+ break;
+ }
+ }
+ }
+}
+
+static int is_eth_port_of_netdev(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie)
+{
+ struct net_device *rdev;
+ struct net_device *mdev;
+ struct net_device *ndev = (struct net_device *)cookie;
+
+ if (!idev)
+ return 0;
+
+ rcu_read_lock();
+ mdev = netdev_master_upper_dev_get_rcu(idev);
+ rdev = rdma_vlan_dev_real_dev(ndev);
+ rcu_read_unlock();
+
+ return (rdev ? rdev : ndev) == (mdev ? mdev : idev);
+}
+
+static int pass_all_filter(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie)
+{
+ return 1;
+}
+
+static void update_gid_ip(enum gid_op_type gid_op,
+ struct ib_device *ib_dev,
+ u8 port, struct net_device *ndev,
+ const struct sockaddr *addr)
+{
+ union ib_gid gid;
+ struct ib_gid_attr gid_attr;
+
+ rdma_ip2gid(addr, &gid);
+ memset(&gid_attr, 0, sizeof(gid_attr));
+ gid_attr.ndev = ndev;
+
+ update_gid(gid_op, ib_dev, port, &gid, &gid_attr);
+}
+
+static void enum_netdev_ipv4_ips(struct ib_device *ib_dev,
+ u8 port, struct net_device *ndev)
+{
+ struct in_device *in_dev;
+
+ if (ndev->reg_state >= NETREG_UNREGISTERING)
+ return;
+
+ in_dev = in_dev_get(ndev);
+ if (!in_dev)
+ return;
+
+ for_ifa(in_dev) {
+ struct sockaddr_in ip;
+
+ ip.sin_family = AF_INET;
+ ip.sin_addr.s_addr = ifa->ifa_address;
+ update_gid_ip(GID_ADD, ib_dev, port, ndev,
+ (struct sockaddr *)&ip);
+ }
+ endfor_ifa(in_dev);
+
+ in_dev_put(in_dev);
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+static void enum_netdev_ipv6_ips(struct ib_device *ib_dev,
+ u8 port, struct net_device *ndev)
+{
+ struct inet6_ifaddr *ifp;
+ struct inet6_dev *in6_dev;
+ struct sin6_list {
+ struct list_head list;
+ struct sockaddr_in6 sin6;
+ };
+ struct sin6_list *sin6_iter;
+ struct sin6_list *sin6_temp;
+ struct ib_gid_attr gid_attr = {.ndev = ndev};
+ LIST_HEAD(sin6_list);
+
+ if (ndev->reg_state >= NETREG_UNREGISTERING)
+ return;
+
+ in6_dev = in6_dev_get(ndev);
+ if (!in6_dev)
+ return;
+
+ read_lock_bh(&in6_dev->lock);
+ list_for_each_entry(ifp, &in6_dev->addr_list, if_list) {
+ struct sin6_list *entry = kzalloc(sizeof(*entry), GFP_ATOMIC);
+
+ if (!entry) {
+ pr_warn("roce_gid_mgmt: couldn't allocate entry for IPv6 update\n");
+ continue;
+ }
+
+ entry->sin6.sin6_family = AF_INET6;
+ entry->sin6.sin6_addr = ifp->addr;
+ list_add_tail(&entry->list, &sin6_list);
+ }
+ read_unlock_bh(&in6_dev->lock);
+
+ in6_dev_put(in6_dev);
+
+ list_for_each_entry_safe(sin6_iter, sin6_temp, &sin6_list, list) {
+ union ib_gid gid;
+
+ rdma_ip2gid((const struct sockaddr *)&sin6_iter->sin6, &gid);
+ update_gid(GID_ADD, ib_dev, port, &gid, &gid_attr);
+ list_del(&sin6_iter->list);
+ kfree(sin6_iter);
+ }
+}
+#endif
+
+static void add_netdev_ips(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie)
+{
+ struct net_device *ndev = (struct net_device *)cookie;
+
+ enum_netdev_ipv4_ips(ib_dev, port, ndev);
+#if IS_ENABLED(CONFIG_IPV6)
+ enum_netdev_ipv6_ips(ib_dev, port, ndev);
+#endif
+}
+
+static void del_netdev_ips(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie)
+{
+ struct net_device *ndev = (struct net_device *)cookie;
+
+ roce_del_all_netdev_gids(ib_dev, port, ndev);
+}
+
+static void enum_all_gids_of_dev_cb(struct ib_device *ib_dev,
+ u8 port,
+ struct net_device *idev,
+ void *cookie)
+{
+ struct net *net;
+ struct net_device *ndev;
+
+ /* Lock the rtnl to make sure the netdevs does not move under
+ * our feet
+ */
+ rtnl_lock();
+ for_each_net(net)
+ for_each_netdev(net, ndev)
+ if (is_eth_port_of_netdev(ib_dev, port, idev, ndev))
+ add_netdev_ips(ib_dev, port, idev, ndev);
+ rtnl_unlock();
+}
+
+/* This function will rescan all of the network devices in the system
+ * and add their gids, as needed, to the relevant RoCE devices. Will
+ * take rtnl and the IB device list mutexes. Must not be called from
+ * ib_wq or deadlock will happen. */
+int roce_rescan_device(struct ib_device *ib_dev)
+{
+ ib_dev_roce_ports_of_netdev(ib_dev, pass_all_filter, NULL,
+ enum_all_gids_of_dev_cb, NULL);
+
+ return 0;
+}
+
+static void callback_for_addr_gid_device_scan(struct ib_device *device,
+ u8 port,
+ struct net_device *idev,
+ void *cookie)
+{
+ struct update_gid_event_work *parsed = cookie;
+
+ return update_gid(parsed->gid_op, device,
+ port, &parsed->gid,
+ &parsed->gid_attr);
+}
+
+/* The following functions operate on all IB devices. netdevice_event and
+ * addr_event execute ib_enum_roce_ports_of_netdev through a work.
+ * ib_enum_roce_ports_of_netdev iterates through all IB devices, thus proper
+ * usage of SRCU is required
+ */
+
+static void netdevice_event_work_handler(struct work_struct *_work)
+{
+ struct netdev_event_work *work =
+ container_of(_work, struct netdev_event_work, work);
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(work->cmds) && work->cmds[i].cb; i++)
+ ib_enum_roce_ports_of_netdev(work->cmds[i].filter, work->ndev,
+ work->cmds[i].cb, work->ndev);
+
+ dev_put(work->ndev);
+ kfree(work);
+}
+
+static int netdevice_event(struct notifier_block *this, unsigned long event,
+ void *ptr)
+{
+ static const struct netdev_event_work_cmd add_cmd = {
+ .cb = add_netdev_ips, .filter = is_eth_port_of_netdev};
+ static const struct netdev_event_work_cmd del_cmd = {
+ .cb = del_netdev_ips, .filter = pass_all_filter};
+ struct net_device *ndev = netdev_notifier_info_to_dev(ptr);
+ struct netdev_event_work *ndev_work;
+ struct netdev_event_work_cmd cmds[ROCE_NETDEV_CALLBACK_SZ] = { {NULL} };
+
+ if (ndev->type != ARPHRD_ETHER)
+ return NOTIFY_DONE;
+
+ switch (event) {
+ case NETDEV_REGISTER:
+ case NETDEV_UP:
+ cmds[0] = add_cmd;
+ break;
+
+ case NETDEV_UNREGISTER:
+ if (ndev->reg_state < NETREG_UNREGISTERED)
+ cmds[0] = del_cmd;
+ else
+ return NOTIFY_DONE;
+ break;
+
+ case NETDEV_CHANGEADDR:
+ cmds[0] = del_cmd;
+ cmds[1] = add_cmd;
+ break;
+ default:
+ return NOTIFY_DONE;
+ }
+
+ ndev_work = kmalloc(sizeof(*ndev_work), GFP_KERNEL);
+ if (!ndev_work) {
+ pr_warn("roce_gid_mgmt: can't allocate work for netdevice_event\n");
+ return NOTIFY_DONE;
+ }
+
+ memcpy(ndev_work->cmds, cmds, sizeof(ndev_work->cmds));
+ ndev_work->ndev = ndev;
+ dev_hold(ndev);
+ INIT_WORK(&ndev_work->work, netdevice_event_work_handler);
+
+ queue_work(roce_gid_mgmt_wq, &ndev_work->work);
+
+ return NOTIFY_DONE;
+}
+
+static void update_gid_event_work_handler(struct work_struct *_work)
+{
+ struct update_gid_event_work *work =
+ container_of(_work, struct update_gid_event_work, work);
+
+ ib_enum_roce_ports_of_netdev(is_eth_port_of_netdev, work->gid_attr.ndev,
+ callback_for_addr_gid_device_scan, work);
+
+ dev_put(work->gid_attr.ndev);
+ kfree(work);
+}
+
+static int addr_event(struct notifier_block *this, unsigned long event,
+ struct sockaddr *sa, struct net_device *ndev)
+{
+ struct update_gid_event_work *work;
+ enum gid_op_type gid_op;
+
+ if (ndev->type != ARPHRD_ETHER)
+ return NOTIFY_DONE;
+
+ switch (event) {
+ case NETDEV_UP:
+ gid_op = GID_ADD;
+ break;
+
+ case NETDEV_DOWN:
+ gid_op = GID_DEL;
+ break;
+
+ default:
+ return NOTIFY_DONE;
+ }
+
+ work = kmalloc(sizeof(*work), GFP_ATOMIC);
+ if (!work) {
+ pr_warn("roce_gid_mgmt: Couldn't allocate work for addr_event\n");
+ return NOTIFY_DONE;
+ }
+
+ INIT_WORK(&work->work, update_gid_event_work_handler);
+
+ rdma_ip2gid(sa, &work->gid);
+ work->gid_op = gid_op;
+
+ memset(&work->gid_attr, 0, sizeof(work->gid_attr));
+ dev_hold(ndev);
+ work->gid_attr.ndev = ndev;
+
+ queue_work(roce_gid_mgmt_wq, &work->work);
+
+ return NOTIFY_DONE;
+}
+
+static int inetaddr_event(struct notifier_block *this, unsigned long event,
+ void *ptr)
+{
+ struct sockaddr_in in;
+ struct net_device *ndev;
+ struct in_ifaddr *ifa = ptr;
+
+ in.sin_family = AF_INET;
+ in.sin_addr.s_addr = ifa->ifa_address;
+ ndev = ifa->ifa_dev->dev;
+
+ return addr_event(this, event, (struct sockaddr *)&in, ndev);
+}
+
+#if IS_ENABLED(CONFIG_IPV6)
+static int inet6addr_event(struct notifier_block *this, unsigned long event,
+ void *ptr)
+{
+ struct sockaddr_in6 in6;
+ struct net_device *ndev;
+ struct inet6_ifaddr *ifa6 = ptr;
+
+ in6.sin6_family = AF_INET6;
+ in6.sin6_addr = ifa6->addr;
+ ndev = ifa6->idev->dev;
+
+ return addr_event(this, event, (struct sockaddr *)&in6, ndev);
+}
+#endif
+
+static struct notifier_block nb_netdevice = {
+ .notifier_call = netdevice_event
+};
+
+static struct notifier_block nb_inetaddr = {
+ .notifier_call = inetaddr_event
+};
+
+#if IS_ENABLED(CONFIG_IPV6)
+static struct notifier_block nb_inet6addr = {
+ .notifier_call = inet6addr_event
+};
+#endif
+
+int __init roce_gid_mgmt_init(void)
+{
+ roce_gid_mgmt_wq = alloc_ordered_workqueue("roce_gid_mgmt_wq", 0);
+
+ if (!roce_gid_mgmt_wq) {
+ pr_warn("roce_gid_mgmt: can't allocate work queue\n");
+ return -ENOMEM;
+ }
+
+ register_inetaddr_notifier(&nb_inetaddr);
+#if IS_ENABLED(CONFIG_IPV6)
+ register_inet6addr_notifier(&nb_inet6addr);
+#endif
+ /* We relay on the netdevice notifier to enumerate all
+ * existing devices in the system. Register to this notifier
+ * last to make sure we will not miss any IP add/del
+ * callbacks.
+ */
+ register_netdevice_notifier(&nb_netdevice);
+
+ return 0;
+}
+
+void __exit roce_gid_mgmt_cleanup(void)
+{
+#if IS_ENABLED(CONFIG_IPV6)
+ unregister_inet6addr_notifier(&nb_inet6addr);
+#endif
+ unregister_inetaddr_notifier(&nb_inetaddr);
+ unregister_netdevice_notifier(&nb_netdevice);
+ /* Ensure all gid deletion tasks complete before we go down,
+ * to avoid any reference to free'd memory. By the time
+ * ib-core is removed, all physical devices have been removed,
+ * so no issue with remaining hardware contexts.
+ */
+ synchronize_rcu();
+ drain_workqueue(roce_gid_mgmt_wq);
+ destroy_workqueue(roce_gid_mgmt_wq);
+}
diff --git a/drivers/infiniband/core/roce_gid_table.c b/drivers/infiniband/core/roce_gid_table.c
index 30e9e04..d5d3ca6 100644
--- a/drivers/infiniband/core/roce_gid_table.c
+++ b/drivers/infiniband/core/roce_gid_table.c
@@ -490,3 +490,55 @@ static void roce_gid_table_cleanup_one(struct ib_device *ib_dev,
kfree(table);
}
+static void roce_gid_table_client_cleanup_one(struct ib_device *ib_dev)
+{
+ struct ib_roce_gid_table **table = ib_dev->cache.roce_gid_table;
+
+ if (!table)
+ return;
+
+ ib_dev->cache.roce_gid_table = NULL;
+ /* smp_wmb is mandatory in order to make sure all executing works
+ * realize we're freeing this roce_gid_table. Every function which
+ * could be executed in a work, fetches ib_dev->cache.roce_gid_table
+ * once (READ_ONCE + smp_rmb) into a local variable.
+ * If it fetched a value != NULL, we wait for this work to finish by
+ * calling flush_workqueue. If it fetches NULL, it'll return immediately.
+ */
+ smp_wmb();
+ /* Make sure no gid update task is still referencing this device */
+ flush_workqueue(roce_gid_mgmt_wq);
+
+ roce_gid_table_cleanup_one(ib_dev, table);
+}
+
+static void roce_gid_table_client_setup_one(struct ib_device *ib_dev)
+{
+ if (!roce_gid_table_setup_one(ib_dev))
+ if (roce_rescan_device(ib_dev))
+ roce_gid_table_client_cleanup_one(ib_dev);
+}
+
+static struct ib_client table_client = {
+ .name = "roce_gid_table",
+ .add = roce_gid_table_client_setup_one,
+ .remove = roce_gid_table_client_cleanup_one
+};
+
+int __init roce_gid_table_setup(void)
+{
+ roce_gid_mgmt_init();
+
+ return ib_register_client(&table_client);
+}
+
+void __exit roce_gid_table_cleanup(void)
+{
+ ib_unregister_client(&table_client);
+
+ roce_gid_mgmt_cleanup();
+
+ flush_workqueue(system_wq);
+
+ rcu_barrier();
+}
diff --git a/include/rdma/ib_addr.h b/include/rdma/ib_addr.h
index ce55906..3cf32d1 100644
--- a/include/rdma/ib_addr.h
+++ b/include/rdma/ib_addr.h
@@ -142,7 +142,7 @@ static inline u16 rdma_vlan_dev_vlan_id(const struct net_device *dev)
vlan_dev_vlan_id(dev) : 0xffff;
}
-static inline int rdma_ip2gid(struct sockaddr *addr, union ib_gid *gid)
+static inline int rdma_ip2gid(const struct sockaddr *addr, union ib_gid *gid)
{
switch (addr->sa_family) {
case AF_INET:
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 5cf40f4..3554e32 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -1534,6 +1534,14 @@ struct ib_device {
struct ib_port_attr *port_attr);
enum rdma_link_layer (*get_link_layer)(struct ib_device *device,
u8 port_num);
+ /* When calling get_netdev, the HW vendor's driver should return the
+ * net device of device @device at port @port_num. The function
+ * is called in rtnl_lock. The HW vendor's device driver must guarantee
+ * to return NULL before the net device has reached
+ * NETDEV_UNREGISTER_FINAL state.
+ */
+ struct net_device *(*get_netdev)(struct ib_device *device,
+ u8 port_num);
int (*query_gid)(struct ib_device *device,
u8 port_num, int index,
union ib_gid *gid);
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 04/14] IB/core: Add default GID for RoCE GID table
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (2 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 03/14] IB/core: Add RoCE GID population Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
[not found] ` <1432045637-9090-5-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-05-19 14:27 ` [PATCH v4 for-next 05/14] net: Add info for NETDEV_CHANGEUPPER event Matan Barak
` (9 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
When RoCE is used, a default GID address should be generated
for every supported RoCE type. These default GID addresses are
generated based on the IPv6 link-local address, but in contrast
to the GID based on the regular IPv6 link-local (as we generate
GID per IP address), these GIDs are also available if the net
device is down (in order to support loopback).
Moreover, these default GID addresses can't be deleted.
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/core/core_priv.h | 12 ++
drivers/infiniband/core/roce_gid_mgmt.c | 43 ++++++--
drivers/infiniband/core/roce_gid_table.c | 182 ++++++++++++++++++++++++++++---
include/net/addrconf.h | 31 ++++++
include/rdma/ib_verbs.h | 1 +
net/ipv6/addrconf.c | 31 ------
6 files changed, 248 insertions(+), 52 deletions(-)
diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h
index eb094f6..0383643 100644
--- a/drivers/infiniband/core/core_priv.h
+++ b/drivers/infiniband/core/core_priv.h
@@ -82,6 +82,16 @@ int roce_gid_table_find_gid_by_port(struct ib_device *ib_dev, union ib_gid *gid,
enum ib_gid_type gid_type, u8 port,
struct net *net, int if_index, u16 *index);
+enum roce_gid_table_default_mode {
+ ROCE_GID_TABLE_DEFAULT_MODE_SET,
+ ROCE_GID_TABLE_DEFAULT_MODE_DELETE
+};
+
+void roce_gid_table_set_default_gid(struct ib_device *ib_dev, u8 port,
+ struct net_device *ndev,
+ unsigned long gid_type_mask,
+ enum roce_gid_table_default_mode mode);
+
int roce_gid_table_setup(void);
void roce_gid_table_cleanup(void);
@@ -98,5 +108,7 @@ int roce_gid_mgmt_init(void);
void roce_gid_mgmt_cleanup(void);
int roce_rescan_device(struct ib_device *ib_dev);
+unsigned long roce_gid_type_mask_support(struct ib_device *ib_dev, u8 port);
+
#endif /* _CORE_PRIV_H */
diff --git a/drivers/infiniband/core/roce_gid_mgmt.c b/drivers/infiniband/core/roce_gid_mgmt.c
index 9aff044..0912934 100644
--- a/drivers/infiniband/core/roce_gid_mgmt.c
+++ b/drivers/infiniband/core/roce_gid_mgmt.c
@@ -76,24 +76,37 @@ static const struct {
#define CAP_TO_GID_TABLE_SIZE ARRAY_SIZE(PORT_CAP_TO_GID_TYPE)
-static void update_gid(enum gid_op_type gid_op, struct ib_device *ib_dev,
- u8 port, union ib_gid *gid,
- struct ib_gid_attr *gid_attr)
+unsigned long roce_gid_type_mask_support(struct ib_device *ib_dev, u8 port)
{
struct ib_port_attr pattr;
int i;
int err;
+ unsigned int ret_flags = 0;
err = ib_query_port(ib_dev, port, &pattr);
if (err) {
pr_warn("update_gid: ib_query_port() failed for %s, %d\n",
ib_dev->name, err);
+ return 0;
}
- for (i = 0; i < CAP_TO_GID_TABLE_SIZE; i++) {
- if (pattr.port_cap_flags & PORT_CAP_TO_GID_TYPE[i].flag_mask) {
- gid_attr->gid_type =
- PORT_CAP_TO_GID_TYPE[i].gid_type;
+ for (i = 0; i < CAP_TO_GID_TABLE_SIZE; i++)
+ if (pattr.port_cap_flags & PORT_CAP_TO_GID_TYPE[i].flag_mask)
+ ret_flags |= 1UL << PORT_CAP_TO_GID_TYPE[i].gid_type;
+
+ return ret_flags;
+}
+
+static void update_gid(enum gid_op_type gid_op, struct ib_device *ib_dev,
+ u8 port, union ib_gid *gid,
+ struct ib_gid_attr *gid_attr)
+{
+ int i;
+ unsigned long gid_type_mask = roce_gid_type_mask_support(ib_dev, port);
+
+ for (i = 0; i < IB_GID_TYPE_SIZE; i++) {
+ if ((1UL << i) & gid_type_mask) {
+ gid_attr->gid_type = i;
switch (gid_op) {
case GID_ADD:
roce_add_gid(ib_dev, port,
@@ -147,6 +160,21 @@ static void update_gid_ip(enum gid_op_type gid_op,
update_gid(gid_op, ib_dev, port, &gid, &gid_attr);
}
+static void enum_netdev_default_gids(struct ib_device *ib_dev,
+ u8 port, struct net_device *ndev,
+ struct net_device *idev)
+{
+ unsigned long gid_type_mask;
+
+ if (idev != ndev)
+ return;
+
+ gid_type_mask = roce_gid_type_mask_support(ib_dev, port);
+
+ roce_gid_table_set_default_gid(ib_dev, port, idev, gid_type_mask,
+ ROCE_GID_TABLE_DEFAULT_MODE_SET);
+}
+
static void enum_netdev_ipv4_ips(struct ib_device *ib_dev,
u8 port, struct net_device *ndev)
{
@@ -227,6 +255,7 @@ static void add_netdev_ips(struct ib_device *ib_dev, u8 port,
{
struct net_device *ndev = (struct net_device *)cookie;
+ enum_netdev_default_gids(ib_dev, port, ndev, idev);
enum_netdev_ipv4_ips(ib_dev, port, ndev);
#if IS_ENABLED(CONFIG_IPV6)
enum_netdev_ipv6_ips(ib_dev, port, ndev);
diff --git a/drivers/infiniband/core/roce_gid_table.c b/drivers/infiniband/core/roce_gid_table.c
index d5d3ca6..5692bb8 100644
--- a/drivers/infiniband/core/roce_gid_table.c
+++ b/drivers/infiniband/core/roce_gid_table.c
@@ -34,6 +34,7 @@
#include <linux/netdevice.h>
#include <linux/rtnetlink.h>
#include <rdma/ib_cache.h>
+#include <net/addrconf.h>
#include "core_priv.h"
@@ -46,6 +47,7 @@ enum gid_attr_find_mask {
GID_ATTR_FIND_MASK_GID = 1UL << 0,
GID_ATTR_FIND_MASK_GID_TYPE = 1UL << 1,
GID_ATTR_FIND_MASK_NETDEV = 1UL << 2,
+ GID_ATTR_FIND_MASK_DEFAULT = 1UL << 3,
};
static inline int start_port(struct ib_device *ib_dev)
@@ -70,7 +72,8 @@ static void put_ndev(struct rcu_head *rcu)
static int write_gid(struct ib_device *ib_dev, u8 port,
struct ib_roce_gid_table *table, int ix,
const union ib_gid *gid,
- const struct ib_gid_attr *attr)
+ const struct ib_gid_attr *attr,
+ bool default_gid)
{
int ret;
struct dev_put_rcu *put_rcu;
@@ -78,6 +81,7 @@ static int write_gid(struct ib_device *ib_dev, u8 port,
write_seqcount_begin(&table->data_vec[ix].seq);
+ table->data_vec[ix].default_gid = default_gid;
ret = ib_dev->modify_gid(ib_dev, port, ix, gid, attr,
&table->data_vec[ix].context);
@@ -120,7 +124,8 @@ static int write_gid(struct ib_device *ib_dev, u8 port,
}
static int find_gid(struct ib_roce_gid_table *table, union ib_gid *gid,
- const struct ib_gid_attr *val, unsigned long mask)
+ const struct ib_gid_attr *val, bool default_gid,
+ unsigned long mask)
{
int i;
@@ -132,13 +137,18 @@ static int find_gid(struct ib_roce_gid_table *table, union ib_gid *gid,
attr->gid_type != val->gid_type)
continue;
- if (memcmp(gid, &table->data_vec[i].gid, sizeof(*gid)))
+ if (mask & GID_ATTR_FIND_MASK_GID &&
+ memcmp(gid, &table->data_vec[i].gid, sizeof(*gid)))
continue;
if (mask & GID_ATTR_FIND_MASK_NETDEV &&
attr->ndev != val->ndev)
continue;
+ if (mask & GID_ATTR_FIND_MASK_DEFAULT &&
+ table->data_vec[i].default_gid != default_gid)
+ continue;
+
if (!read_seqcount_retry(&table->data_vec[i].seq, orig_seq))
return i;
/* The sequence number changed under our feet,
@@ -150,6 +160,12 @@ static int find_gid(struct ib_roce_gid_table *table, union ib_gid *gid,
return -1;
}
+static void make_default_gid(struct net_device *dev, union ib_gid *gid)
+{
+ gid->global.subnet_prefix = cpu_to_be64(0xfe80000000000000LL);
+ addrconf_ifid_eui48(&gid->raw[8], dev);
+}
+
int roce_add_gid(struct ib_device *ib_dev, u8 port,
union ib_gid *gid, struct ib_gid_attr *attr)
{
@@ -158,6 +174,7 @@ int roce_add_gid(struct ib_device *ib_dev, u8 port,
struct ib_roce_gid_table *table;
int ix;
int ret = 0;
+ struct net_device *idev;
/* make sure we read the ports_table */
smp_rmb();
@@ -173,20 +190,38 @@ int roce_add_gid(struct ib_device *ib_dev, u8 port,
if (!memcmp(gid, &zgid, sizeof(*gid)))
return -EINVAL;
+ if (ib_dev->get_netdev) {
+ rcu_read_lock();
+ idev = ib_dev->get_netdev(ib_dev, port);
+ if (idev && attr->ndev != idev) {
+ union ib_gid default_gid;
+
+ /* Adding default GIDs in not permitted */
+ make_default_gid(idev, &default_gid);
+ if (!memcmp(gid, &default_gid, sizeof(*gid))) {
+ rcu_read_unlock();
+ return -EPERM;
+ }
+ }
+ rcu_read_unlock();
+ }
+
mutex_lock(&table->lock);
- ix = find_gid(table, gid, attr, GID_ATTR_FIND_MASK_GID_TYPE |
+ ix = find_gid(table, gid, attr, false, GID_ATTR_FIND_MASK_GID |
+ GID_ATTR_FIND_MASK_GID_TYPE |
GID_ATTR_FIND_MASK_NETDEV);
if (ix >= 0)
goto out_unlock;
- ix = find_gid(table, &zgid, NULL, 0);
+ ix = find_gid(table, &zgid, NULL, false, GID_ATTR_FIND_MASK_GID |
+ GID_ATTR_FIND_MASK_DEFAULT);
if (ix < 0) {
ret = -ENOSPC;
goto out_unlock;
}
- write_gid(ib_dev, port, table, ix, gid, attr);
+ write_gid(ib_dev, port, table, ix, gid, attr, false);
out_unlock:
mutex_unlock(&table->lock);
@@ -199,6 +234,7 @@ int roce_del_gid(struct ib_device *ib_dev, u8 port,
struct ib_roce_gid_table **ports_table =
READ_ONCE(ib_dev->cache.roce_gid_table);
struct ib_roce_gid_table *table;
+ union ib_gid default_gid;
int ix;
/* make sure we read the ports_table */
@@ -212,15 +248,24 @@ int roce_del_gid(struct ib_device *ib_dev, u8 port,
if (!table)
return -EPROTONOSUPPORT;
+ if (attr->ndev) {
+ /* Deleting default GIDs in not permitted */
+ make_default_gid(attr->ndev, &default_gid);
+ if (!memcmp(gid, &default_gid, sizeof(*gid)))
+ return -EPERM;
+ }
+
mutex_lock(&table->lock);
- ix = find_gid(table, gid, attr,
+ ix = find_gid(table, gid, attr, false,
+ GID_ATTR_FIND_MASK_GID |
GID_ATTR_FIND_MASK_GID_TYPE |
- GID_ATTR_FIND_MASK_NETDEV);
+ GID_ATTR_FIND_MASK_NETDEV |
+ GID_ATTR_FIND_MASK_DEFAULT);
if (ix < 0)
goto out_unlock;
- write_gid(ib_dev, port, table, ix, &zgid, &zattr);
+ write_gid(ib_dev, port, table, ix, &zgid, &zattr, false);
out_unlock:
mutex_unlock(&table->lock);
@@ -250,7 +295,7 @@ int roce_del_all_netdev_gids(struct ib_device *ib_dev, u8 port,
for (ix = 0; ix < table->sz; ix++)
if (table->data_vec[ix].attr.ndev == ndev)
- write_gid(ib_dev, port, table, ix, &zgid, &zattr);
+ write_gid(ib_dev, port, table, ix, &zgid, &zattr, false);
mutex_unlock(&table->lock);
return 0;
@@ -318,7 +363,7 @@ static int _roce_gid_table_find_gid(struct ib_device *ib_dev, union ib_gid *gid,
table = ports_table[p];
if (!table)
continue;
- local_index = find_gid(table, gid, val, mask);
+ local_index = find_gid(table, gid, val, false, mask);
if (local_index >= 0) {
if (index)
*index = local_index;
@@ -366,7 +411,8 @@ int roce_gid_table_find_gid_by_port(struct ib_device *ib_dev, union ib_gid *gid,
struct ib_roce_gid_table **ports_table =
READ_ONCE(ib_dev->cache.roce_gid_table);
struct ib_roce_gid_table *table;
- unsigned long mask = GID_ATTR_FIND_MASK_GID_TYPE;
+ unsigned long mask = GID_ATTR_FIND_MASK_GID |
+ GID_ATTR_FIND_MASK_GID_TYPE;
struct ib_gid_attr val = {.gid_type = gid_type};
/* make sure we read the ports_table */
@@ -382,7 +428,7 @@ int roce_gid_table_find_gid_by_port(struct ib_device *ib_dev, union ib_gid *gid,
mask |= get_netdev_from_ifindex(net, if_index, &val);
- local_index = find_gid(table, gid, &val, mask);
+ local_index = find_gid(table, gid, &val, false, mask);
if (local_index >= 0) {
if (index)
*index = local_index;
@@ -429,12 +475,114 @@ static void free_roce_gid_table(struct ib_device *ib_dev, u8 port,
for (i = 0; i < table->sz; ++i) {
if (memcmp(&table->data_vec[i].gid, &zgid,
sizeof(table->data_vec[i].gid)))
- write_gid(ib_dev, port, table, i, &zgid, &zattr);
+ write_gid(ib_dev, port, table, i, &zgid, &zattr,
+ table->data_vec[i].default_gid);
}
kfree(table->data_vec);
kfree(table);
}
+void roce_gid_table_set_default_gid(struct ib_device *ib_dev, u8 port,
+ struct net_device *ndev,
+ unsigned long gid_type_mask,
+ enum roce_gid_table_default_mode mode)
+{
+ struct ib_roce_gid_table **ports_table =
+ READ_ONCE(ib_dev->cache.roce_gid_table);
+ union ib_gid gid;
+ struct ib_gid_attr gid_attr;
+ struct ib_gid_attr zattr_type = zattr;
+ struct ib_roce_gid_table *table;
+ unsigned int gid_type;
+
+ /* make sure we read the ports_table */
+ smp_rmb();
+
+ if (!ports_table)
+ return;
+
+ table = ports_table[port - start_port(ib_dev)];
+
+ if (!table)
+ return;
+
+ make_default_gid(ndev, &gid);
+ memset(&gid_attr, 0, sizeof(gid_attr));
+ gid_attr.ndev = ndev;
+ for (gid_type = 0; gid_type < IB_GID_TYPE_SIZE; ++gid_type) {
+ int ix;
+ union ib_gid current_gid;
+ struct ib_gid_attr current_gid_attr;
+
+ if (1UL << gid_type & ~gid_type_mask)
+ continue;
+
+ gid_attr.gid_type = gid_type;
+
+ ix = find_gid(table, &gid, &gid_attr, true,
+ GID_ATTR_FIND_MASK_GID_TYPE |
+ GID_ATTR_FIND_MASK_DEFAULT);
+
+ if (ix < 0) {
+ pr_warn("roce_gid_table: couldn't find index for default gid type %u\n",
+ gid_type);
+ continue;
+ }
+
+ zattr_type.gid_type = gid_type;
+
+ mutex_lock(&table->lock);
+ if (!roce_gid_table_get_gid(ib_dev, port, ix,
+ ¤t_gid, ¤t_gid_attr) &&
+ mode == ROCE_GID_TABLE_DEFAULT_MODE_SET &&
+ !memcmp(&gid, ¤t_gid, sizeof(gid)) &&
+ !memcmp(&gid_attr, ¤t_gid_attr, sizeof(gid_attr))) {
+ mutex_unlock(&table->lock);
+ continue;
+ }
+
+ if ((memcmp(¤t_gid, &zgid, sizeof(current_gid)) ||
+ memcmp(¤t_gid_attr, &zattr_type,
+ sizeof(current_gid_attr))) &&
+ write_gid(ib_dev, port, table, ix, &zgid, &zattr, true)) {
+ pr_warn("roce_gid_table: can't delete index %d for default gid %pI6\n",
+ ix, gid.raw);
+ mutex_unlock(&table->lock);
+ continue;
+ }
+
+ if (mode == ROCE_GID_TABLE_DEFAULT_MODE_SET)
+ if (write_gid(ib_dev, port, table, ix, &gid, &gid_attr,
+ true))
+ pr_warn("roce_gid_table: unable to add default gid %pI6\n",
+ gid.raw);
+
+ mutex_unlock(&table->lock);
+ }
+}
+
+static int roce_gid_table_reserve_default(struct ib_device *ib_dev, u8 port,
+ struct ib_roce_gid_table *table)
+{
+ unsigned int i;
+ unsigned long roce_gid_type_mask;
+ unsigned int num_default_gids;
+
+ roce_gid_type_mask = roce_gid_type_mask_support(ib_dev, port);
+ num_default_gids = hweight_long(roce_gid_type_mask);
+ for (i = 0; i < num_default_gids && i < table->sz; i++) {
+ struct ib_roce_gid_table_entry *entry =
+ &table->data_vec[i];
+
+ entry->default_gid = true;
+ entry->attr.gid_type = find_next_bit(&roce_gid_type_mask,
+ BITS_PER_LONG,
+ i);
+ }
+
+ return 0;
+}
+
static int roce_gid_table_setup_one(struct ib_device *ib_dev)
{
u8 port;
@@ -462,6 +610,12 @@ static int roce_gid_table_setup_one(struct ib_device *ib_dev)
err = -ENOMEM;
goto rollback_table_setup;
}
+
+ err = roce_gid_table_reserve_default(ib_dev,
+ port + start_port(ib_dev),
+ table[port]);
+ if (err)
+ goto rollback_table_setup;
}
ib_dev->cache.roce_gid_table = table;
diff --git a/include/net/addrconf.h b/include/net/addrconf.h
index 80456f7..89890e7 100644
--- a/include/net/addrconf.h
+++ b/include/net/addrconf.h
@@ -91,6 +91,37 @@ int ipv6_rcv_saddr_equal(const struct sock *sk, const struct sock *sk2);
void addrconf_join_solict(struct net_device *dev, const struct in6_addr *addr);
void addrconf_leave_solict(struct inet6_dev *idev, const struct in6_addr *addr);
+static inline int addrconf_ifid_eui48(u8 *eui, struct net_device *dev)
+{
+ if (dev->addr_len != ETH_ALEN)
+ return -1;
+ memcpy(eui, dev->dev_addr, 3);
+ memcpy(eui + 5, dev->dev_addr + 3, 3);
+
+ /*
+ * The zSeries OSA network cards can be shared among various
+ * OS instances, but the OSA cards have only one MAC address.
+ * This leads to duplicate address conflicts in conjunction
+ * with IPv6 if more than one instance uses the same card.
+ *
+ * The driver for these cards can deliver a unique 16-bit
+ * identifier for each instance sharing the same card. It is
+ * placed instead of 0xFFFE in the interface identifier. The
+ * "u" bit of the interface identifier is not inverted in this
+ * case. Hence the resulting interface identifier has local
+ * scope according to RFC2373.
+ */
+ if (dev->dev_id) {
+ eui[3] = (dev->dev_id >> 8) & 0xFF;
+ eui[4] = dev->dev_id & 0xFF;
+ } else {
+ eui[3] = 0xFF;
+ eui[4] = 0xFE;
+ eui[0] ^= 2;
+ }
+ return 0;
+}
+
static inline unsigned long addrconf_timeout_fixup(u32 timeout,
unsigned int unit)
{
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 3554e32..f9afa91 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -83,6 +83,7 @@ struct ib_roce_gid_table_entry {
union ib_gid gid;
struct ib_gid_attr attr;
void *context;
+ bool default_gid;
};
struct ib_roce_gid_table {
diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index 37b70e8..7170c7b 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -1845,37 +1845,6 @@ static void addrconf_leave_anycast(struct inet6_ifaddr *ifp)
__ipv6_dev_ac_dec(ifp->idev, &addr);
}
-static int addrconf_ifid_eui48(u8 *eui, struct net_device *dev)
-{
- if (dev->addr_len != ETH_ALEN)
- return -1;
- memcpy(eui, dev->dev_addr, 3);
- memcpy(eui + 5, dev->dev_addr + 3, 3);
-
- /*
- * The zSeries OSA network cards can be shared among various
- * OS instances, but the OSA cards have only one MAC address.
- * This leads to duplicate address conflicts in conjunction
- * with IPv6 if more than one instance uses the same card.
- *
- * The driver for these cards can deliver a unique 16-bit
- * identifier for each instance sharing the same card. It is
- * placed instead of 0xFFFE in the interface identifier. The
- * "u" bit of the interface identifier is not inverted in this
- * case. Hence the resulting interface identifier has local
- * scope according to RFC2373.
- */
- if (dev->dev_id) {
- eui[3] = (dev->dev_id >> 8) & 0xFF;
- eui[4] = dev->dev_id & 0xFF;
- } else {
- eui[3] = 0xFF;
- eui[4] = 0xFE;
- eui[0] ^= 2;
- }
- return 0;
-}
-
static int addrconf_ifid_eui64(u8 *eui, struct net_device *dev)
{
if (dev->addr_len != IEEE802154_ADDR_LEN)
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 05/14] net: Add info for NETDEV_CHANGEUPPER event
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (3 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 04/14] IB/core: Add default GID for RoCE GID table Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 06/14] IB/core: Add RoCE table bonding support Matan Barak
` (8 subsequent siblings)
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
Consumers of NETDEV_CHANGEUPPER event sometimes want
to know which upper device was linked/unlinked and which
operation was carried. Adding extra information in the
notifier info block.
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
include/linux/netdevice.h | 14 ++++++++++++++
net/core/dev.c | 12 ++++++++++--
2 files changed, 24 insertions(+), 2 deletions(-)
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index dbad4d7..637cc21 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -3554,6 +3554,20 @@ struct sk_buff *__skb_gso_segment(struct sk_buff *skb,
struct sk_buff *skb_mac_gso_segment(struct sk_buff *skb,
netdev_features_t features);
+enum netdev_changeupper_event {
+ NETDEV_CHANGEUPPER_LINK,
+ NETDEV_CHANGEUPPER_UNLINK,
+};
+
+struct netdev_changeupper_info {
+ struct netdev_notifier_info info; /* must be first */
+ enum netdev_changeupper_event event;
+ struct net_device *upper;
+};
+
+void netdev_changeupper_info_change(struct net_device *dev,
+ struct netdev_changeupper_info *info);
+
struct netdev_bonding_info {
ifslave slave;
ifbond master;
diff --git a/net/core/dev.c b/net/core/dev.c
index c7ba038..b52f648 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -5198,6 +5198,7 @@ static int __netdev_upper_dev_link(struct net_device *dev,
void *private)
{
struct netdev_adjacent *i, *j, *to_i, *to_j;
+ struct netdev_changeupper_info changeupper_info;
int ret = 0;
ASSERT_RTNL();
@@ -5253,7 +5254,10 @@ static int __netdev_upper_dev_link(struct net_device *dev,
goto rollback_lower_mesh;
}
- call_netdevice_notifiers(NETDEV_CHANGEUPPER, dev);
+ changeupper_info.event = NETDEV_CHANGEUPPER_LINK;
+ changeupper_info.upper = upper_dev;
+ call_netdevice_notifiers_info(NETDEV_CHANGEUPPER, dev,
+ &changeupper_info.info);
return 0;
rollback_lower_mesh:
@@ -5349,6 +5353,7 @@ void netdev_upper_dev_unlink(struct net_device *dev,
struct net_device *upper_dev)
{
struct netdev_adjacent *i, *j;
+ struct netdev_changeupper_info changeupper_info;
ASSERT_RTNL();
__netdev_adjacent_dev_unlink_neighbour(dev, upper_dev);
@@ -5370,7 +5375,10 @@ void netdev_upper_dev_unlink(struct net_device *dev,
list_for_each_entry(i, &upper_dev->all_adj_list.upper, list)
__netdev_adjacent_dev_unlink(dev, i->dev);
- call_netdevice_notifiers(NETDEV_CHANGEUPPER, dev);
+ changeupper_info.event = NETDEV_CHANGEUPPER_UNLINK;
+ changeupper_info.upper = upper_dev;
+ call_netdevice_notifiers_info(NETDEV_CHANGEUPPER, dev,
+ &changeupper_info.info);
}
EXPORT_SYMBOL(netdev_upper_dev_unlink);
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 06/14] IB/core: Add RoCE table bonding support
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (4 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 05/14] net: Add info for NETDEV_CHANGEUPPER event Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API Matan Barak
` (7 subsequent siblings)
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
Bonding is a unique behavior since when working in
active-backup mode, only the current selected slave
should occupy the default GIDs and the master's GID.
Listening to bonding events and only adding the
required GIDs to the active slave in the RoCE table
GID table.
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/core/roce_gid_mgmt.c | 291 ++++++++++++++++++++++++++++++--
drivers/net/bonding/bond_options.c | 13 --
include/net/bonding.h | 7 +
3 files changed, 282 insertions(+), 29 deletions(-)
diff --git a/drivers/infiniband/core/roce_gid_mgmt.c b/drivers/infiniband/core/roce_gid_mgmt.c
index 0912934..cac0da1 100644
--- a/drivers/infiniband/core/roce_gid_mgmt.c
+++ b/drivers/infiniband/core/roce_gid_mgmt.c
@@ -37,6 +37,7 @@
/* For in6_dev_get/in6_dev_put */
#include <net/addrconf.h>
+#include <net/bonding.h>
#include <rdma/ib_cache.h>
#include <rdma/ib_addr.h>
@@ -55,16 +56,17 @@ struct update_gid_event_work {
enum gid_op_type gid_op;
};
-#define ROCE_NETDEV_CALLBACK_SZ 2
+#define ROCE_NETDEV_CALLBACK_SZ 3
struct netdev_event_work_cmd {
roce_netdev_callback cb;
roce_netdev_filter filter;
+ struct net_device *ndev;
+ struct net_device *f_ndev;
};
struct netdev_event_work {
struct work_struct work;
struct netdev_event_work_cmd cmds[ROCE_NETDEV_CALLBACK_SZ];
- struct net_device *ndev;
};
static const struct {
@@ -121,22 +123,96 @@ static void update_gid(enum gid_op_type gid_op, struct ib_device *ib_dev,
}
}
+#define IS_NETDEV_BONDING_MASTER(ndev) \
+ (((ndev)->priv_flags & IFF_BONDING) && \
+ ((ndev)->flags & IFF_MASTER))
+
+enum bonding_slave_state {
+ BONDING_SLAVE_STATE_ACTIVE = 1UL << 0,
+ BONDING_SLAVE_STATE_INACTIVE = 1UL << 1,
+ BONDING_SLAVE_STATE_NA = 1UL << 2,
+};
+
+static enum bonding_slave_state is_eth_active_slave_of_bonding(struct net_device *idev,
+ struct net_device *upper)
+{
+ if (upper && IS_NETDEV_BONDING_MASTER(upper)) {
+ struct net_device *pdev;
+
+ rcu_read_lock();
+ pdev = bond_option_active_slave_get_rcu(netdev_priv(upper));
+ rcu_read_unlock();
+ if (pdev)
+ return idev == pdev ? BONDING_SLAVE_STATE_ACTIVE :
+ BONDING_SLAVE_STATE_INACTIVE;
+ }
+
+ return BONDING_SLAVE_STATE_NA;
+}
+
+static bool is_upper_dev_rcu(struct net_device *dev, struct net_device *upper)
+{
+ struct net_device *_upper = NULL;
+ struct list_head *iter;
+
+ rcu_read_lock();
+ netdev_for_each_all_upper_dev_rcu(dev, _upper, iter) {
+ if (_upper == upper)
+ break;
+ }
+
+ rcu_read_unlock();
+ return _upper == upper;
+}
+
+static int _is_eth_port_of_netdev(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie,
+ unsigned long bond_state)
+{
+ struct net_device *ndev = (struct net_device *)cookie;
+ struct net_device *rdev;
+ int res;
+
+ if (!idev)
+ return 0;
+
+ rcu_read_lock();
+ rdev = rdma_vlan_dev_real_dev(ndev);
+ if (!rdev)
+ rdev = ndev;
+
+ res = ((is_upper_dev_rcu(idev, ndev) &&
+ (is_eth_active_slave_of_bonding(idev, rdev) &
+ bond_state)) ||
+ rdev == idev);
+
+ rcu_read_unlock();
+ return res;
+}
+
static int is_eth_port_of_netdev(struct ib_device *ib_dev, u8 port,
struct net_device *idev, void *cookie)
{
- struct net_device *rdev;
- struct net_device *mdev;
- struct net_device *ndev = (struct net_device *)cookie;
+ return _is_eth_port_of_netdev(ib_dev, port, idev, cookie,
+ BONDING_SLAVE_STATE_ACTIVE |
+ BONDING_SLAVE_STATE_NA);
+}
+static int is_eth_port_inactive_slave(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie)
+{
+ struct net_device *mdev;
+ int res;
if (!idev)
return 0;
rcu_read_lock();
mdev = netdev_master_upper_dev_get_rcu(idev);
- rdev = rdma_vlan_dev_real_dev(ndev);
+ res = is_eth_active_slave_of_bonding(idev, mdev) ==
+ BONDING_SLAVE_STATE_INACTIVE;
rcu_read_unlock();
- return (rdev ? rdev : ndev) == (mdev ? mdev : idev);
+ return res;
}
static int pass_all_filter(struct ib_device *ib_dev, u8 port,
@@ -145,6 +221,34 @@ static int pass_all_filter(struct ib_device *ib_dev, u8 port,
return 1;
}
+static int upper_device_filter(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie)
+{
+ struct net_device *ndev = (struct net_device *)cookie;
+
+ return idev == ndev || is_upper_dev_rcu(idev, ndev);
+}
+
+static int bonding_slaves_filter(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie)
+{
+ struct net_device *rdev;
+ struct net_device *ndev = (struct net_device *)cookie;
+ int res;
+
+ rdev = rdma_vlan_dev_real_dev(ndev);
+
+ ndev = rdev ? rdev : ndev;
+ if (!idev || !IS_NETDEV_BONDING_MASTER(ndev))
+ return 0;
+
+ rcu_read_lock();
+ res = is_upper_dev_rcu(idev, ndev);
+ rcu_read_unlock();
+
+ return res;
+}
+
static void update_gid_ip(enum gid_op_type gid_op,
struct ib_device *ib_dev,
u8 port, struct net_device *ndev,
@@ -166,8 +270,16 @@ static void enum_netdev_default_gids(struct ib_device *ib_dev,
{
unsigned long gid_type_mask;
- if (idev != ndev)
+ rcu_read_lock();
+ if (!idev ||
+ ((idev != ndev && !is_upper_dev_rcu(idev, ndev)) ||
+ is_eth_active_slave_of_bonding(idev,
+ netdev_master_upper_dev_get_rcu(idev)) ==
+ BONDING_SLAVE_STATE_INACTIVE)) {
+ rcu_read_unlock();
return;
+ }
+ rcu_read_unlock();
gid_type_mask = roce_gid_type_mask_support(ib_dev, port);
@@ -175,6 +287,37 @@ static void enum_netdev_default_gids(struct ib_device *ib_dev,
ROCE_GID_TABLE_DEFAULT_MODE_SET);
}
+static void bond_delete_netdev_default_gids(struct ib_device *ib_dev,
+ u8 port, struct net_device *ndev,
+ struct net_device *idev)
+{
+ struct net_device *rdev = rdma_vlan_dev_real_dev(ndev);
+
+ if (!idev)
+ return;
+
+ if (!rdev)
+ rdev = ndev;
+
+ rcu_read_lock();
+
+ if (is_upper_dev_rcu(idev, ndev) &&
+ is_eth_active_slave_of_bonding(idev, rdev) ==
+ BONDING_SLAVE_STATE_INACTIVE) {
+ unsigned long gid_type_mask;
+
+ rcu_read_unlock();
+
+ gid_type_mask = roce_gid_type_mask_support(ib_dev, port);
+
+ roce_gid_table_set_default_gid(ib_dev, port, idev,
+ gid_type_mask,
+ ROCE_GID_TABLE_DEFAULT_MODE_DELETE);
+ } else {
+ rcu_read_unlock();
+ }
+}
+
static void enum_netdev_ipv4_ips(struct ib_device *ib_dev,
u8 port, struct net_device *ndev)
{
@@ -313,6 +456,72 @@ static void callback_for_addr_gid_device_scan(struct ib_device *device,
&parsed->gid_attr);
}
+static void del_netdev_upper_ips(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie)
+{
+ struct net_device *ndev = (struct net_device *)cookie;
+ struct upper_list {
+ struct list_head list;
+ struct net_device *upper;
+ };
+ struct net_device *upper;
+ struct list_head *iter;
+ struct upper_list *upper_iter;
+ struct upper_list *upper_temp;
+ LIST_HEAD(upper_list);
+
+ rcu_read_lock();
+ netdev_for_each_all_upper_dev_rcu(ndev, upper, iter) {
+ struct upper_list *entry = kmalloc(sizeof(*entry),
+ GFP_ATOMIC);
+
+ if (!entry) {
+ pr_info("roce_gid_mgmt: couldn't allocate entry to delete ndev\n");
+ continue;
+ }
+
+ list_add_tail(&entry->list, &upper_list);
+ dev_hold(upper);
+ entry->upper = upper;
+ }
+ rcu_read_unlock();
+
+ roce_del_all_netdev_gids(ib_dev, port, ndev);
+ list_for_each_entry_safe(upper_iter, upper_temp, &upper_list,
+ list) {
+ roce_del_all_netdev_gids(ib_dev, port,
+ upper_iter->upper);
+ dev_put(upper_iter->upper);
+ list_del(&upper_iter->list);
+ kfree(upper_iter);
+ }
+}
+
+static void del_netdev_default_ips_join(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie)
+{
+ struct net_device *mdev;
+
+ rcu_read_lock();
+ mdev = netdev_master_upper_dev_get_rcu(idev);
+ if (mdev)
+ dev_hold(mdev);
+ rcu_read_unlock();
+
+ if (mdev) {
+ bond_delete_netdev_default_gids(ib_dev, port, mdev, idev);
+ dev_put(mdev);
+ }
+}
+
+static void del_netdev_default_ips(struct ib_device *ib_dev, u8 port,
+ struct net_device *idev, void *cookie)
+{
+ struct net_device *ndev = (struct net_device *)cookie;
+
+ bond_delete_netdev_default_gids(ib_dev, port, ndev, idev);
+}
+
/* The following functions operate on all IB devices. netdevice_event and
* addr_event execute ib_enum_roce_ports_of_netdev through a work.
* ib_enum_roce_ports_of_netdev iterates through all IB devices, thus proper
@@ -325,11 +534,15 @@ static void netdevice_event_work_handler(struct work_struct *_work)
container_of(_work, struct netdev_event_work, work);
unsigned int i;
- for (i = 0; i < ARRAY_SIZE(work->cmds) && work->cmds[i].cb; i++)
- ib_enum_roce_ports_of_netdev(work->cmds[i].filter, work->ndev,
- work->cmds[i].cb, work->ndev);
+ for (i = 0; i < ARRAY_SIZE(work->cmds) && work->cmds[i].cb; i++) {
+ ib_enum_roce_ports_of_netdev(work->cmds[i].filter,
+ work->cmds[i].f_ndev,
+ work->cmds[i].cb,
+ work->cmds[i].ndev);
+ dev_put(work->cmds[i].ndev);
+ dev_put(work->cmds[i].f_ndev);
+ }
- dev_put(work->ndev);
kfree(work);
}
@@ -340,9 +553,20 @@ static int netdevice_event(struct notifier_block *this, unsigned long event,
.cb = add_netdev_ips, .filter = is_eth_port_of_netdev};
static const struct netdev_event_work_cmd del_cmd = {
.cb = del_netdev_ips, .filter = pass_all_filter};
+ static const struct netdev_event_work_cmd bonding_default_del_cmd_join = {
+ .cb = del_netdev_default_ips_join, .filter = is_eth_port_inactive_slave};
+ static const struct netdev_event_work_cmd bonding_default_del_cmd = {
+ .cb = del_netdev_default_ips, .filter = is_eth_port_inactive_slave};
+ static const struct netdev_event_work_cmd default_del_cmd = {
+ .cb = del_netdev_default_ips, .filter = pass_all_filter};
+ static const struct netdev_event_work_cmd bonding_event_ips_del_cmd = {
+ .cb = del_netdev_ips, .filter = bonding_slaves_filter};
+ static const struct netdev_event_work_cmd upper_ips_del_cmd = {
+ .cb = del_netdev_upper_ips, .filter = upper_device_filter};
struct net_device *ndev = netdev_notifier_info_to_dev(ptr);
struct netdev_event_work *ndev_work;
struct netdev_event_work_cmd cmds[ROCE_NETDEV_CALLBACK_SZ] = { {NULL} };
+ unsigned int i;
if (ndev->type != ARPHRD_ETHER)
return NOTIFY_DONE;
@@ -350,7 +574,8 @@ static int netdevice_event(struct notifier_block *this, unsigned long event,
switch (event) {
case NETDEV_REGISTER:
case NETDEV_UP:
- cmds[0] = add_cmd;
+ cmds[0] = bonding_default_del_cmd_join;
+ cmds[1] = add_cmd;
break;
case NETDEV_UNREGISTER:
@@ -361,9 +586,37 @@ static int netdevice_event(struct notifier_block *this, unsigned long event,
break;
case NETDEV_CHANGEADDR:
- cmds[0] = del_cmd;
+ cmds[0] = default_del_cmd;
cmds[1] = add_cmd;
break;
+
+ case NETDEV_CHANGEUPPER:
+ {
+ struct netdev_changeupper_info *changeupper_info =
+ container_of(ptr, struct netdev_changeupper_info, info);
+
+ if (changeupper_info->event ==
+ NETDEV_CHANGEUPPER_UNLINK) {
+ cmds[0] = upper_ips_del_cmd;
+ cmds[0].ndev = changeupper_info->upper;
+ cmds[1] = add_cmd;
+ } else if (changeupper_info->event ==
+ NETDEV_CHANGEUPPER_LINK) {
+ cmds[0] = bonding_default_del_cmd;
+ cmds[0].ndev = changeupper_info->upper;
+ cmds[1] = add_cmd;
+ cmds[1].ndev = changeupper_info->upper;
+ cmds[1].f_ndev = changeupper_info->upper;
+ }
+ }
+ break;
+
+ case NETDEV_BONDING_FAILOVER:
+ cmds[0] = bonding_event_ips_del_cmd;
+ cmds[1] = bonding_default_del_cmd_join;
+ cmds[2] = add_cmd;
+ break;
+
default:
return NOTIFY_DONE;
}
@@ -375,8 +628,14 @@ static int netdevice_event(struct notifier_block *this, unsigned long event,
}
memcpy(ndev_work->cmds, cmds, sizeof(ndev_work->cmds));
- ndev_work->ndev = ndev;
- dev_hold(ndev);
+ for (i = 0; i < ARRAY_SIZE(ndev_work->cmds) && ndev_work->cmds[i].cb; i++) {
+ if (!ndev_work->cmds[i].ndev)
+ ndev_work->cmds[i].ndev = ndev;
+ if (!ndev_work->cmds[i].f_ndev)
+ ndev_work->cmds[i].f_ndev = ndev;
+ dev_hold(ndev_work->cmds[i].ndev);
+ dev_hold(ndev_work->cmds[i].f_ndev);
+ }
INIT_WORK(&ndev_work->work, netdevice_event_work_handler);
queue_work(roce_gid_mgmt_wq, &ndev_work->work);
diff --git a/drivers/net/bonding/bond_options.c b/drivers/net/bonding/bond_options.c
index 4df2894..c4fe29a8 100644
--- a/drivers/net/bonding/bond_options.c
+++ b/drivers/net/bonding/bond_options.c
@@ -689,19 +689,6 @@ static int bond_option_mode_set(struct bonding *bond,
return 0;
}
-static struct net_device *__bond_option_active_slave_get(struct bonding *bond,
- struct slave *slave)
-{
- return bond_uses_primary(bond) && slave ? slave->dev : NULL;
-}
-
-struct net_device *bond_option_active_slave_get_rcu(struct bonding *bond)
-{
- struct slave *slave = rcu_dereference(bond->curr_active_slave);
-
- return __bond_option_active_slave_get(bond, slave);
-}
-
static int bond_option_active_slave_set(struct bonding *bond,
const struct bond_opt_value *newval)
{
diff --git a/include/net/bonding.h b/include/net/bonding.h
index 78ed135..81a94ed 100644
--- a/include/net/bonding.h
+++ b/include/net/bonding.h
@@ -307,6 +307,13 @@ static inline bool bond_uses_primary(struct bonding *bond)
return bond_mode_uses_primary(BOND_MODE(bond));
}
+static inline struct net_device *bond_option_active_slave_get_rcu(struct bonding *bond)
+{
+ struct slave *slave = rcu_dereference(bond->curr_active_slave);
+
+ return bond_uses_primary(bond) && slave ? slave->dev : NULL;
+}
+
static inline bool bond_slave_is_up(struct slave *slave)
{
return netif_running(slave->dev) && netif_carrier_ok(slave->dev);
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (5 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 06/14] IB/core: Add RoCE table bonding support Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
[not found] ` <1432045637-9090-8-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-05-19 14:27 ` [PATCH v4 for-next 08/14] IB/core: Report gid_type and gid_ndev through sysfs Matan Barak
` (6 subsequent siblings)
13 siblings, 1 reply; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
Along with the GID itself, we now store GIDs attribute.
This GID attribute contains important meta information regarding
the GID itself, for example the netdevice. Thus, this information
needs to be returned in APIs. This patch changes the following APIs:
(a) ib_get_cached_gid
(b) ib_find_cached_gid
(c) ib_find_cached_gid_by_port
(d) ib_query_gid
It changes the usage of those APIs and use the RoCE GID table
when needed.
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/core/cache.c | 289 ++++++++++++++++++++-----
drivers/infiniband/core/cm.c | 6 +-
drivers/infiniband/core/cma.c | 84 ++++---
drivers/infiniband/core/device.c | 29 ++-
drivers/infiniband/core/mad.c | 2 +-
drivers/infiniband/core/multicast.c | 3 +-
drivers/infiniband/core/sa_query.c | 7 +-
drivers/infiniband/core/sysfs.c | 2 +-
drivers/infiniband/core/uverbs_marshall.c | 4 +-
drivers/infiniband/core/verbs.c | 7 +-
drivers/infiniband/hw/mlx4/qp.c | 5 +-
drivers/infiniband/hw/mthca/mthca_av.c | 2 +-
drivers/infiniband/ulp/ipoib/ipoib_main.c | 2 +-
drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 2 +-
drivers/infiniband/ulp/srp/ib_srp.c | 2 +-
drivers/infiniband/ulp/srpt/ib_srpt.c | 3 +-
include/rdma/ib_cache.h | 44 +++-
include/rdma/ib_sa.h | 4 +-
include/rdma/ib_verbs.h | 7 +-
19 files changed, 401 insertions(+), 103 deletions(-)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 80f6cf2..5af1e6a 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -42,6 +42,8 @@
#include "core_priv.h"
+#define __IB_ONLY
+
struct ib_pkey_cache {
int table_len;
u16 table[0];
@@ -69,68 +71,215 @@ static inline int end_port(struct ib_device *device)
0 : device->phys_port_cnt;
}
-int ib_get_cached_gid(struct ib_device *device,
- u8 port_num,
- int index,
- union ib_gid *gid)
+static int __IB_ONLY __ib_get_cached_gid(struct ib_device *device,
+ u8 port_num,
+ int index,
+ union ib_gid *gid)
{
struct ib_gid_cache *cache;
unsigned long flags;
- int ret = 0;
+ int ret = -ENOENT;
if (port_num < start_port(device) || port_num > end_port(device))
return -EINVAL;
+ if (!device->cache.gid_cache)
+ return -ENOENT;
read_lock_irqsave(&device->cache.lock, flags);
cache = device->cache.gid_cache[port_num - start_port(device)];
-
- if (index < 0 || index >= cache->table_len)
- ret = -EINVAL;
- else
+ if (cache && index >= 0 && index < cache->table_len) {
*gid = cache->table[index];
+ ret = 0;
+ }
read_unlock_irqrestore(&device->cache.lock, flags);
+ return ret;
+}
+
+int ib_cache_use_roce_gid_table(struct ib_device *device, u8 port_num)
+{
+ if (rdma_port_get_link_layer(device, port_num) ==
+ IB_LINK_LAYER_ETHERNET) {
+ if (device->cache.roce_gid_table)
+ return 0;
+ else
+ return -EAGAIN;
+ }
+
+ return -EINVAL;
+}
+EXPORT_SYMBOL(ib_cache_use_roce_gid_table);
+
+int ib_get_cached_gid(struct ib_device *device,
+ u8 port_num,
+ int index,
+ union ib_gid *gid,
+ struct ib_gid_attr *attr)
+{
+ int ret;
+
+ if (port_num < start_port(device) || port_num > end_port(device))
+ return -EINVAL;
+
+ ret = ib_cache_use_roce_gid_table(device, port_num);
+ if (!ret)
+ return roce_gid_table_get_gid(device, port_num, index, gid,
+ attr);
+
+ if (ret == -EAGAIN)
+ return ret;
+
+ ret = __ib_get_cached_gid(device, port_num, index, gid);
+
+ if (!ret && attr) {
+ memset(attr, 0, sizeof(*attr));
+ attr->gid_type = IB_GID_TYPE_IB;
+ }
return ret;
}
EXPORT_SYMBOL(ib_get_cached_gid);
-int ib_find_cached_gid(struct ib_device *device,
- union ib_gid *gid,
- u8 *port_num,
- u16 *index)
+static int __IB_ONLY ___ib_find_cached_gid_by_port(struct ib_device *device,
+ u8 port_num,
+ const union ib_gid *gid,
+ u16 *index)
{
struct ib_gid_cache *cache;
+ u8 p = port_num - start_port(device);
+ int i;
+
+ if (port_num < start_port(device) || port_num > end_port(device))
+ return -EINVAL;
+ if (!ib_cache_use_roce_gid_table(device, port_num))
+ return -EPROTONOSUPPORT;
+ if (!device->cache.gid_cache)
+ return -ENOENT;
+
+ cache = device->cache.gid_cache[p];
+ if (!cache)
+ return -ENOENT;
+
+ for (i = 0; i < cache->table_len; ++i) {
+ if (!memcmp(gid, &cache->table[i], sizeof(*gid))) {
+ if (index)
+ *index = i;
+ return 0;
+ }
+ }
+
+ return -ENOENT;
+}
+
+static int __IB_ONLY __ib_find_cached_gid_by_port(struct ib_device *device,
+ u8 port_num,
+ union ib_gid *gid,
+ u16 *index)
+{
unsigned long flags;
- int p, i;
+ u16 found_index;
+ int ret;
+
+ if (index)
+ *index = -1;
+
+ read_lock_irqsave(&device->cache.lock, flags);
+
+ ret = ___ib_find_cached_gid_by_port(device, port_num, gid,
+ &found_index);
+
+ read_unlock_irqrestore(&device->cache.lock, flags);
+
+ if (!ret && index)
+ *index = found_index;
+
+ return ret;
+}
+
+static int __IB_ONLY __ib_find_cached_gid(struct ib_device *device,
+ union ib_gid *gid,
+ u8 *port_num,
+ u16 *index)
+{
+ unsigned long flags;
+ u16 found_index;
+ int p;
int ret = -ENOENT;
- *port_num = -1;
+ if (port_num)
+ *port_num = -1;
if (index)
*index = -1;
read_lock_irqsave(&device->cache.lock, flags);
- for (p = 0; p <= end_port(device) - start_port(device); ++p) {
- cache = device->cache.gid_cache[p];
- for (i = 0; i < cache->table_len; ++i) {
- if (!memcmp(gid, &cache->table[i], sizeof *gid)) {
- *port_num = p + start_port(device);
- if (index)
- *index = i;
- ret = 0;
- goto found;
- }
+ for (p = start_port(device); p <= end_port(device); ++p) {
+ if (!___ib_find_cached_gid_by_port(device, p, gid,
+ &found_index)) {
+ if (port_num)
+ *port_num = p;
+ ret = 0;
+ break;
}
}
-found:
+
read_unlock_irqrestore(&device->cache.lock, flags);
+ if (!ret && index)
+ *index = found_index;
+
+ return ret;
+}
+
+int ib_find_cached_gid(struct ib_device *device,
+ union ib_gid *gid,
+ enum ib_gid_type gid_type,
+ struct net *net,
+ int if_index,
+ u8 *port_num,
+ u16 *index)
+{
+ int ret = -ENOENT;
+
+ /* Look for a RoCE device with the specified GID. */
+ if (device->cache.roce_gid_table)
+ ret = roce_gid_table_find_gid(device, gid, gid_type, net,
+ if_index, port_num, index);
+
+ /* If no RoCE devices with the specified GID, look for IB device. */
+ if (ret && gid_type == IB_GID_TYPE_IB)
+ ret = __ib_find_cached_gid(device, gid, port_num, index);
+
return ret;
}
EXPORT_SYMBOL(ib_find_cached_gid);
+int ib_find_cached_gid_by_port(struct ib_device *device,
+ union ib_gid *gid,
+ enum ib_gid_type gid_type,
+ u8 port_num,
+ struct net *net,
+ int if_index,
+ u16 *index)
+{
+ int ret = -ENOENT;
+
+ /* Look for a RoCE device with the specified GID. */
+ if (!ib_cache_use_roce_gid_table(device, port_num))
+ return roce_gid_table_find_gid_by_port(device, gid, gid_type,
+ port_num, net, if_index,
+ index);
+
+ /* If no RoCE devices with the specified GID, look for IB device. */
+ if (gid_type == IB_GID_TYPE_IB)
+ ret = __ib_find_cached_gid_by_port(device, port_num,
+ gid, index);
+
+ return ret;
+}
+EXPORT_SYMBOL(ib_find_cached_gid_by_port);
+
int ib_get_cached_pkey(struct ib_device *device,
u8 port_num,
int index,
@@ -138,22 +287,23 @@ int ib_get_cached_pkey(struct ib_device *device,
{
struct ib_pkey_cache *cache;
unsigned long flags;
- int ret = 0;
+ int ret = -ENOENT;
if (port_num < start_port(device) || port_num > end_port(device))
return -EINVAL;
+ if (!device->cache.pkey_cache)
+ return -ENOENT;
+
read_lock_irqsave(&device->cache.lock, flags);
cache = device->cache.pkey_cache[port_num - start_port(device)];
-
- if (index < 0 || index >= cache->table_len)
- ret = -EINVAL;
- else
+ if (cache && index >= 0 && index < cache->table_len) {
*pkey = cache->table[index];
+ ret = 0;
+ }
read_unlock_irqrestore(&device->cache.lock, flags);
-
return ret;
}
EXPORT_SYMBOL(ib_get_cached_pkey);
@@ -172,9 +322,14 @@ int ib_find_cached_pkey(struct ib_device *device,
if (port_num < start_port(device) || port_num > end_port(device))
return -EINVAL;
+ if (!device->cache.pkey_cache)
+ return -ENOENT;
+
read_lock_irqsave(&device->cache.lock, flags);
cache = device->cache.pkey_cache[port_num - start_port(device)];
+ if (!cache)
+ goto out;
*index = -1;
@@ -193,8 +348,8 @@ int ib_find_cached_pkey(struct ib_device *device,
ret = 0;
}
+out:
read_unlock_irqrestore(&device->cache.lock, flags);
-
return ret;
}
EXPORT_SYMBOL(ib_find_cached_pkey);
@@ -212,9 +367,14 @@ int ib_find_exact_cached_pkey(struct ib_device *device,
if (port_num < start_port(device) || port_num > end_port(device))
return -EINVAL;
+ if (!device->cache.pkey_cache)
+ return -ENOENT;
+
read_lock_irqsave(&device->cache.lock, flags);
cache = device->cache.pkey_cache[port_num - start_port(device)];
+ if (!cache)
+ goto out;
*index = -1;
@@ -224,9 +384,8 @@ int ib_find_exact_cached_pkey(struct ib_device *device,
ret = 0;
break;
}
-
+out:
read_unlock_irqrestore(&device->cache.lock, flags);
-
return ret;
}
EXPORT_SYMBOL(ib_find_exact_cached_pkey);
@@ -236,13 +395,16 @@ int ib_get_cached_lmc(struct ib_device *device,
u8 *lmc)
{
unsigned long flags;
- int ret = 0;
+ int ret = -ENOENT;
if (port_num < start_port(device) || port_num > end_port(device))
return -EINVAL;
read_lock_irqsave(&device->cache.lock, flags);
- *lmc = device->cache.lmc_cache[port_num - start_port(device)];
+ if (device->cache.lmc_cache) {
+ *lmc = device->cache.lmc_cache[port_num - start_port(device)];
+ ret = 0;
+ }
read_unlock_irqrestore(&device->cache.lock, flags);
return ret;
@@ -254,9 +416,19 @@ static void ib_cache_update(struct ib_device *device,
{
struct ib_port_attr *tprops = NULL;
struct ib_pkey_cache *pkey_cache = NULL, *old_pkey_cache;
- struct ib_gid_cache *gid_cache = NULL, *old_gid_cache;
+ struct ib_gid_cache *gid_cache = NULL, *old_gid_cache = NULL;
int i;
int ret;
+ bool use_roce_gid_table =
+ !ib_cache_use_roce_gid_table(device,
+ port);
+
+ if (port < start_port(device) || port > end_port(device))
+ return;
+
+ if (!(device->cache.pkey_cache && device->cache.gid_cache &&
+ device->cache.lmc_cache))
+ return;
tprops = kmalloc(sizeof *tprops, GFP_KERNEL);
if (!tprops)
@@ -276,12 +448,14 @@ static void ib_cache_update(struct ib_device *device,
pkey_cache->table_len = tprops->pkey_tbl_len;
- gid_cache = kmalloc(sizeof *gid_cache + tprops->gid_tbl_len *
- sizeof *gid_cache->table, GFP_KERNEL);
- if (!gid_cache)
- goto err;
+ if (!use_roce_gid_table) {
+ gid_cache = kmalloc(sizeof(*gid_cache) + tprops->gid_tbl_len *
+ sizeof(*gid_cache->table), GFP_KERNEL);
+ if (!gid_cache)
+ goto err;
- gid_cache->table_len = tprops->gid_tbl_len;
+ gid_cache->table_len = tprops->gid_tbl_len;
+ }
for (i = 0; i < pkey_cache->table_len; ++i) {
ret = ib_query_pkey(device, port, i, pkey_cache->table + i);
@@ -292,22 +466,28 @@ static void ib_cache_update(struct ib_device *device,
}
}
- for (i = 0; i < gid_cache->table_len; ++i) {
- ret = ib_query_gid(device, port, i, gid_cache->table + i);
- if (ret) {
- printk(KERN_WARNING "ib_query_gid failed (%d) for %s (index %d)\n",
- ret, device->name, i);
- goto err;
+ if (!use_roce_gid_table) {
+ for (i = 0; i < gid_cache->table_len; ++i) {
+ ret = ib_query_gid(device, port, i,
+ gid_cache->table + i, NULL);
+ if (ret) {
+ printk(KERN_WARNING "ib_query_gid failed (%d) for %s (index %d)\n",
+ ret, device->name, i);
+ goto err;
+ }
}
}
write_lock_irq(&device->cache.lock);
old_pkey_cache = device->cache.pkey_cache[port - start_port(device)];
- old_gid_cache = device->cache.gid_cache [port - start_port(device)];
+ if (!use_roce_gid_table)
+ old_gid_cache =
+ device->cache.gid_cache[port - start_port(device)];
device->cache.pkey_cache[port - start_port(device)] = pkey_cache;
- device->cache.gid_cache [port - start_port(device)] = gid_cache;
+ if (!use_roce_gid_table)
+ device->cache.gid_cache[port - start_port(device)] = gid_cache;
device->cache.lmc_cache[port - start_port(device)] = tprops->lmc;
@@ -403,12 +583,19 @@ err:
kfree(device->cache.pkey_cache);
kfree(device->cache.gid_cache);
kfree(device->cache.lmc_cache);
+ device->cache.pkey_cache = NULL;
+ device->cache.gid_cache = NULL;
+ device->cache.lmc_cache = NULL;
}
static void ib_cache_cleanup_one(struct ib_device *device)
{
int p;
+ if (!(device->cache.pkey_cache && device->cache.gid_cache &&
+ device->cache.lmc_cache))
+ return;
+
ib_unregister_event_handler(&device->cache.event_handler);
flush_workqueue(ib_wq);
diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
index e28a494..d88f2ae 100644
--- a/drivers/infiniband/core/cm.c
+++ b/drivers/infiniband/core/cm.c
@@ -360,6 +360,8 @@ static int cm_init_av_by_path(struct ib_sa_path_rec *path, struct cm_av *av)
read_lock_irqsave(&cm.device_lock, flags);
list_for_each_entry(cm_dev, &cm.device_list, list) {
if (!ib_find_cached_gid(cm_dev->ib_device, &path->sgid,
+ IB_GID_TYPE_IB, path->net,
+ path->ifindex,
&p, NULL)) {
port = cm_dev->port[p-1];
break;
@@ -379,7 +381,6 @@ static int cm_init_av_by_path(struct ib_sa_path_rec *path, struct cm_av *av)
ib_init_ah_from_path(cm_dev->ib_device, port->port_num, path,
&av->ah_attr);
av->timeout = path->packet_life_time + 1;
- memcpy(av->smac, path->smac, sizeof(av->smac));
av->valid = 1;
return 0;
@@ -1566,7 +1567,8 @@ static int cm_req_handler(struct cm_work *work)
ret = cm_init_av_by_path(&work->path[0], &cm_id_priv->av);
if (ret) {
ib_get_cached_gid(work->port->cm_dev->ib_device,
- work->port->port_num, 0, &work->path[0].sgid);
+ work->port->port_num, 0, &work->path[0].sgid,
+ NULL);
ib_send_cm_rej(cm_id, IB_CM_REJ_INVALID_GID,
&work->path[0].sgid, sizeof work->path[0].sgid,
NULL, 0);
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 06441a4..7e6a0b2 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -356,7 +356,7 @@ static int cma_acquire_dev(struct rdma_id_private *id_priv,
struct cma_device *cma_dev;
union ib_gid gid, iboe_gid;
int ret = -ENODEV;
- u8 port, found_port;
+ u8 port;
enum rdma_link_layer dev_ll = dev_addr->dev_type == ARPHRD_INFINIBAND ?
IB_LINK_LAYER_INFINIBAND : IB_LINK_LAYER_ETHERNET;
@@ -375,16 +375,28 @@ static int cma_acquire_dev(struct rdma_id_private *id_priv,
listen_id_priv->id.port_num) == dev_ll) {
cma_dev = listen_id_priv->cma_dev;
port = listen_id_priv->id.port_num;
- if (rdma_node_get_transport(cma_dev->device->node_type) == RDMA_TRANSPORT_IB &&
- rdma_port_get_link_layer(cma_dev->device, port) == IB_LINK_LAYER_ETHERNET)
- ret = ib_find_cached_gid(cma_dev->device, &iboe_gid,
- &found_port, NULL);
- else
- ret = ib_find_cached_gid(cma_dev->device, &gid,
- &found_port, NULL);
+ if (rdma_node_get_transport(cma_dev->device->node_type) ==
+ RDMA_TRANSPORT_IB &&
+ rdma_port_get_link_layer(cma_dev->device, port) ==
+ IB_LINK_LAYER_ETHERNET) {
+ int if_index =
+ id_priv->id.route.addr.dev_addr.bound_dev_if;
+
+ ret = ib_find_cached_gid_by_port(cma_dev->device,
+ &iboe_gid,
+ IB_GID_TYPE_IB,
+ port,
+ &init_net,
+ if_index,
+ NULL);
+ } else {
+ ret = ib_find_cached_gid_by_port(cma_dev->device, &gid,
+ IB_GID_TYPE_IB, port,
+ NULL, 0, NULL);
+ }
- if (!ret && (port == found_port)) {
- id_priv->id.port_num = found_port;
+ if (!ret) {
+ id_priv->id.port_num = port;
goto out;
}
}
@@ -394,15 +406,34 @@ static int cma_acquire_dev(struct rdma_id_private *id_priv,
listen_id_priv->cma_dev == cma_dev &&
listen_id_priv->id.port_num == port)
continue;
- if (rdma_port_get_link_layer(cma_dev->device, port) == dev_ll) {
- if (rdma_node_get_transport(cma_dev->device->node_type) == RDMA_TRANSPORT_IB &&
- rdma_port_get_link_layer(cma_dev->device, port) == IB_LINK_LAYER_ETHERNET)
- ret = ib_find_cached_gid(cma_dev->device, &iboe_gid, &found_port, NULL);
- else
- ret = ib_find_cached_gid(cma_dev->device, &gid, &found_port, NULL);
-
- if (!ret && (port == found_port)) {
- id_priv->id.port_num = found_port;
+ if (rdma_port_get_link_layer(cma_dev->device, port) ==
+ dev_ll) {
+ if (rdma_node_get_transport(cma_dev->device->node_type) ==
+ RDMA_TRANSPORT_IB &&
+ rdma_port_get_link_layer(cma_dev->device, port) ==
+ IB_LINK_LAYER_ETHERNET) {
+ int if_index =
+ id_priv->id.route.addr.dev_addr.bound_dev_if;
+
+ ret = ib_find_cached_gid_by_port(cma_dev->device,
+ &iboe_gid,
+ IB_GID_TYPE_IB,
+ port,
+ &init_net,
+ if_index,
+ NULL);
+ } else {
+ ret = ib_find_cached_gid_by_port(cma_dev->device,
+ &gid,
+ IB_GID_TYPE_IB,
+ port,
+ NULL,
+ 0,
+ NULL);
+ }
+
+ if (!ret) {
+ id_priv->id.port_num = port;
goto out;
}
}
@@ -442,7 +473,9 @@ static int cma_resolve_ib_dev(struct rdma_id_private *id_priv)
if (ib_find_cached_pkey(cur_dev->device, p, pkey, &index))
continue;
- for (i = 0; !ib_get_cached_gid(cur_dev->device, p, i, &gid); i++) {
+ for (i = 0; !ib_get_cached_gid(cur_dev->device, p, i,
+ &gid, NULL);
+ i++) {
if (!memcmp(&gid, dgid, sizeof(gid))) {
cma_dev = cur_dev;
sgid = gid;
@@ -629,7 +662,7 @@ static int cma_modify_qp_rtr(struct rdma_id_private *id_priv,
goto out;
ret = ib_query_gid(id_priv->id.device, id_priv->id.port_num,
- qp_attr.ah_attr.grh.sgid_index, &sgid);
+ qp_attr.ah_attr.grh.sgid_index, &sgid, NULL);
if (ret)
goto out;
@@ -1915,16 +1948,17 @@ static int cma_resolve_iboe_route(struct rdma_id_private *id_priv)
route->num_paths = 1;
- if (addr->dev_addr.bound_dev_if)
+ if (addr->dev_addr.bound_dev_if) {
ndev = dev_get_by_index(&init_net, addr->dev_addr.bound_dev_if);
+ route->path_rec->net = &init_net;
+ route->path_rec->ifindex = addr->dev_addr.bound_dev_if;
+ }
if (!ndev) {
ret = -ENODEV;
goto err2;
}
- route->path_rec->vlan_id = rdma_vlan_dev_vlan_id(ndev);
memcpy(route->path_rec->dmac, addr->dev_addr.dst_dev_addr, ETH_ALEN);
- memcpy(route->path_rec->smac, ndev->dev_addr, ndev->addr_len);
rdma_ip2gid((struct sockaddr *)&id_priv->id.route.addr.src_addr,
&route->path_rec->sgid);
@@ -2058,7 +2092,7 @@ static int cma_bind_loopback(struct rdma_id_private *id_priv)
p = 1;
port_found:
- ret = ib_get_cached_gid(cma_dev->device, p, 0, &gid);
+ ret = ib_get_cached_gid(cma_dev->device, p, 0, &gid, NULL);
if (ret)
goto out;
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 697c715..cacbf1a 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -40,6 +40,7 @@
#include <linux/mutex.h>
#include <rdma/rdma_netlink.h>
#include <rdma/ib_addr.h>
+#include <rdma/ib_cache.h>
#include "core_priv.h"
@@ -616,12 +617,21 @@ EXPORT_SYMBOL(ib_query_port);
* @port_num:Port number to query
* @index:GID table index to query
* @gid:Returned GID
+ * @attr: Returned GID's attribute (only in RoCE)
*
* ib_query_gid() fetches the specified GID table entry.
*/
int ib_query_gid(struct ib_device *device,
- u8 port_num, int index, union ib_gid *gid)
+ u8 port_num, int index, union ib_gid *gid,
+ struct ib_gid_attr *attr)
{
+ if (!ib_cache_use_roce_gid_table(device, port_num))
+ return roce_gid_table_get_gid(device, port_num, index, gid,
+ attr);
+
+ if (attr)
+ return -EINVAL;
+
return device->query_gid(device, port_num, index, gid);
}
EXPORT_SYMBOL(ib_query_gid);
@@ -768,19 +778,32 @@ EXPORT_SYMBOL(ib_modify_port);
* a specified GID value occurs.
* @device: The device to query.
* @gid: The GID value to search for.
+ * @gid_type: Type of GID.
+ * @net: The namespace to search this GID in (RoCE only).
+ * Valid only if if_index != 0.
+ * @if_index: The if_index assigned with this GID (RoCE only).
* @port_num: The port number of the device where the GID value was found.
* @index: The index into the GID table where the GID was found. This
* parameter may be NULL.
*/
int ib_find_gid(struct ib_device *device, union ib_gid *gid,
- u8 *port_num, u16 *index)
+ enum ib_gid_type gid_type, struct net *net,
+ int if_index, u8 *port_num, u16 *index)
{
union ib_gid tmp_gid;
int ret, port, i;
+ if (device->cache.roce_gid_table &&
+ !roce_gid_table_find_gid(device, gid, gid_type, net, if_index,
+ port_num, index))
+ return 0;
+
for (port = start_port(device); port <= end_port(device); ++port) {
+ if (!ib_cache_use_roce_gid_table(device, port))
+ continue;
+
for (i = 0; i < device->gid_tbl_len[port - start_port(device)]; ++i) {
- ret = ib_query_gid(device, port, i, &tmp_gid);
+ ret = ib_query_gid(device, port, i, &tmp_gid, NULL);
if (ret)
return ret;
if (!memcmp(&tmp_gid, gid, sizeof *gid)) {
diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c
index 74c30f4..5d59cce 100644
--- a/drivers/infiniband/core/mad.c
+++ b/drivers/infiniband/core/mad.c
@@ -1791,7 +1791,7 @@ static inline int rcv_has_same_gid(struct ib_mad_agent_private *mad_agent_priv,
((1 << lmc) - 1)));
} else {
if (ib_get_cached_gid(device, port_num,
- attr.grh.sgid_index, &sgid))
+ attr.grh.sgid_index, &sgid, NULL))
return 0;
return !memcmp(sgid.raw, rwc->recv_buf.grh->dgid.raw,
16);
diff --git a/drivers/infiniband/core/multicast.c b/drivers/infiniband/core/multicast.c
index fa17b55..f1927f1 100644
--- a/drivers/infiniband/core/multicast.c
+++ b/drivers/infiniband/core/multicast.c
@@ -729,7 +729,8 @@ int ib_init_ah_from_mcmember(struct ib_device *device, u8 port_num,
u16 gid_index;
u8 p;
- ret = ib_find_cached_gid(device, &rec->port_gid, &p, &gid_index);
+ ret = ib_find_cached_gid(device, &rec->port_gid, IB_GID_TYPE_IB,
+ NULL, 0, &p, &gid_index);
if (ret)
return ret;
diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c
index c38f030..5b20237 100644
--- a/drivers/infiniband/core/sa_query.c
+++ b/drivers/infiniband/core/sa_query.c
@@ -546,7 +546,8 @@ int ib_init_ah_from_path(struct ib_device *device, u8 port_num,
ah_attr->ah_flags = IB_AH_GRH;
ah_attr->grh.dgid = rec->dgid;
- ret = ib_find_cached_gid(device, &rec->sgid, &port_num,
+ ret = ib_find_cached_gid(device, &rec->sgid, IB_GID_TYPE_IB,
+ rec->net, rec->ifindex, &port_num,
&gid_index);
if (ret)
return ret;
@@ -677,9 +678,9 @@ static void ib_sa_path_rec_callback(struct ib_sa_query *sa_query,
ib_unpack(path_rec_table, ARRAY_SIZE(path_rec_table),
mad->data, &rec);
- rec.vlan_id = 0xffff;
+ rec.net = NULL;
+ rec.ifindex = 0;
memset(rec.dmac, 0, ETH_ALEN);
- memset(rec.smac, 0, ETH_ALEN);
query->callback(status, &rec, query->context);
} else
query->callback(status, NULL, query->context);
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index cbd0383..5cee246 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -289,7 +289,7 @@ static ssize_t show_port_gid(struct ib_port *p, struct port_attribute *attr,
union ib_gid gid;
ssize_t ret;
- ret = ib_query_gid(p->ibdev, p->port_num, tab_attr->index, &gid);
+ ret = ib_query_gid(p->ibdev, p->port_num, tab_attr->index, &gid, NULL);
if (ret)
return ret;
diff --git a/drivers/infiniband/core/uverbs_marshall.c b/drivers/infiniband/core/uverbs_marshall.c
index abd9724..7d2f14c 100644
--- a/drivers/infiniband/core/uverbs_marshall.c
+++ b/drivers/infiniband/core/uverbs_marshall.c
@@ -141,8 +141,8 @@ void ib_copy_path_rec_from_user(struct ib_sa_path_rec *dst,
dst->preference = src->preference;
dst->packet_life_time_selector = src->packet_life_time_selector;
- memset(dst->smac, 0, sizeof(dst->smac));
memset(dst->dmac, 0, sizeof(dst->dmac));
- dst->vlan_id = 0xffff;
+ dst->net = NULL;
+ dst->ifindex = 0;
}
EXPORT_SYMBOL(ib_copy_path_rec_from_user);
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index f93eb8d..1fe3e71 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -229,8 +229,8 @@ int ib_init_ah_from_wc(struct ib_device *device, u8 port_num, struct ib_wc *wc,
ah_attr->ah_flags = IB_AH_GRH;
ah_attr->grh.dgid = grh->sgid;
- ret = ib_find_cached_gid(device, &grh->dgid, &port_num,
- &gid_index);
+ ret = ib_find_cached_gid(device, &grh->dgid, IB_GID_TYPE_IB,
+ NULL, 0, &port_num, &gid_index);
if (ret)
return ret;
@@ -873,7 +873,8 @@ int ib_resolve_eth_l2_attrs(struct ib_qp *qp,
if ((*qp_attr_mask & IB_QP_AV) &&
(rdma_port_get_link_layer(qp->device, qp_attr->ah_attr.port_num) == IB_LINK_LAYER_ETHERNET)) {
ret = ib_query_gid(qp->device, qp_attr->ah_attr.port_num,
- qp_attr->ah_attr.grh.sgid_index, &sgid);
+ qp_attr->ah_attr.grh.sgid_index, &sgid,
+ NULL);
if (ret)
goto out;
if (rdma_link_local_addr((struct in6_addr *)qp_attr->ah_attr.grh.dgid.raw)) {
diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c
index 02fc91c6..58a95b5 100644
--- a/drivers/infiniband/hw/mlx4/qp.c
+++ b/drivers/infiniband/hw/mlx4/qp.c
@@ -2192,7 +2192,8 @@ static int build_mlx_header(struct mlx4_ib_sqp *sqp, struct ib_send_wr *wr,
} else {
err = ib_get_cached_gid(ib_dev,
be32_to_cpu(ah->av.ib.port_pd) >> 24,
- ah->av.ib.gid_index, &sgid);
+ ah->av.ib.gid_index, &sgid,
+ NULL);
if (err)
return err;
}
@@ -2234,7 +2235,7 @@ static int build_mlx_header(struct mlx4_ib_sqp *sqp, struct ib_send_wr *wr,
ib_get_cached_gid(ib_dev,
be32_to_cpu(ah->av.ib.port_pd) >> 24,
ah->av.ib.gid_index,
- &sqp->ud_header.grh.source_gid);
+ &sqp->ud_header.grh.source_gid, NULL);
}
memcpy(sqp->ud_header.grh.destination_gid.raw,
ah->av.ib.dgid, 16);
diff --git a/drivers/infiniband/hw/mthca/mthca_av.c b/drivers/infiniband/hw/mthca/mthca_av.c
index 32f6c63..bcac294 100644
--- a/drivers/infiniband/hw/mthca/mthca_av.c
+++ b/drivers/infiniband/hw/mthca/mthca_av.c
@@ -281,7 +281,7 @@ int mthca_read_ah(struct mthca_dev *dev, struct mthca_ah *ah,
ib_get_cached_gid(&dev->ib_dev,
be32_to_cpu(ah->av->port_pd) >> 24,
ah->av->gid_index % dev->limits.gid_table_len,
- &header->grh.source_gid);
+ &header->grh.source_gid, NULL);
memcpy(header->grh.destination_gid.raw,
ah->av->dgid, 16);
}
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c
index 9e1b203..16242aa 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -1610,7 +1610,7 @@ static struct net_device *ipoib_add_port(const char *format,
priv->dev->broadcast[8] = priv->pkey >> 8;
priv->dev->broadcast[9] = priv->pkey & 0xff;
- result = ib_query_gid(hca, port, 0, &priv->local_gid);
+ result = ib_query_gid(hca, port, 0, &priv->local_gid, NULL);
if (result) {
printk(KERN_WARNING "%s: ib_query_gid port %d failed (ret = %d)\n",
hca->name, port, result);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
index 0d23e05..0323c2f 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
@@ -530,7 +530,7 @@ void ipoib_mcast_join_task(struct work_struct *work)
}
priv->local_lid = port_attr.lid;
- if (ib_query_gid(priv->ca, priv->port, 0, &priv->local_gid))
+ if (ib_query_gid(priv->ca, priv->port, 0, &priv->local_gid, NULL))
ipoib_warn(priv, "ib_query_gid() failed\n");
else
memcpy(priv->dev->dev_addr + 4, priv->local_gid.raw, sizeof (union ib_gid));
diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 918814c..79977aa 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -3206,7 +3206,7 @@ static ssize_t srp_create_target(struct device *dev,
INIT_WORK(&target->tl_err_work, srp_tl_err_work);
INIT_WORK(&target->remove_work, srp_remove_work);
spin_lock_init(&target->lock);
- ret = ib_query_gid(ibdev, host->port, 0, &target->sgid);
+ ret = ib_query_gid(ibdev, host->port, 0, &target->sgid, NULL);
if (ret)
goto err;
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 9b84b4c..8e13050 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -546,7 +546,8 @@ static int srpt_refresh_port(struct srpt_port *sport)
sport->sm_lid = port_attr.sm_lid;
sport->lid = port_attr.lid;
- ret = ib_query_gid(sport->sdev->device, sport->port, 0, &sport->gid);
+ ret = ib_query_gid(sport->sdev->device, sport->port, 0, &sport->gid,
+ NULL);
if (ret)
goto err_query_port;
diff --git a/include/rdma/ib_cache.h b/include/rdma/ib_cache.h
index ad9a3c2..96009ca 100644
--- a/include/rdma/ib_cache.h
+++ b/include/rdma/ib_cache.h
@@ -36,6 +36,17 @@
#define _IB_CACHE_H
#include <rdma/ib_verbs.h>
+#include <net/net_namespace.h>
+
+/**
+ * ib_cache_use_roce_gid_table - Returns whether the device uses roce gid table
+ * @device: The device to query
+ * @port_num: The port number of the device to query.
+ *
+ * ib_cache_use_roce_gid_table() returns 0 if this port uses the roce_gid_table
+ * to store GIDs and error otherwise.
+ */
+int ib_cache_use_roce_gid_table(struct ib_device *device, u8 port_num);
/**
* ib_get_cached_gid - Returns a cached GID table entry
@@ -43,6 +54,7 @@
* @port_num: The port number of the device to query.
* @index: The index into the cached GID table to query.
* @gid: The GID value found at the specified index.
+ * @attr: The GID attribute found at the specified index (only in RoCE).
*
* ib_get_cached_gid() fetches the specified GID table entry stored in
* the local software cache.
@@ -50,13 +62,17 @@
int ib_get_cached_gid(struct ib_device *device,
u8 port_num,
int index,
- union ib_gid *gid);
+ union ib_gid *gid,
+ struct ib_gid_attr *attr);
/**
* ib_find_cached_gid - Returns the port number and GID table index where
* a specified GID value occurs.
* @device: The device to query.
* @gid: The GID value to search for.
+ * @gid_type: The GID type to search for.
+ * @net: In RoCE, the namespace of the device.
+ * @if_index: In RoCE, the if_index of the device. Zero means ignore.
* @port_num: The port number of the device where the GID value was found.
* @index: The index into the cached GID table where the GID was found. This
* parameter may be NULL.
@@ -66,10 +82,36 @@ int ib_get_cached_gid(struct ib_device *device,
*/
int ib_find_cached_gid(struct ib_device *device,
union ib_gid *gid,
+ enum ib_gid_type gid_type,
+ struct net *net,
+ int if_index,
u8 *port_num,
u16 *index);
/**
+ * ib_find_cached_gid_by_port - Returns the GID table index where a specified
+ * GID value occurs
+ * @device: The device to query.
+ * @gid: The GID value to search for.
+ * @gid_type: The GID type to search for.
+ * @port_num: The port number of the device where the GID value sould be
+ * searched.
+ * @net: In RoCE, the namespace of the device.
+ * @if_index: In RoCE, the if_index of the device. Zero means ignore.
+ * @index: The index into the cached GID table where the GID was found. This
+ * parameter may be NULL.
+ *
+ * ib_find_cached_gid() searches for the specified GID value in
+ * the local software cache.
+ */
+int ib_find_cached_gid_by_port(struct ib_device *device,
+ union ib_gid *gid,
+ enum ib_gid_type gid_type,
+ u8 port_num,
+ struct net *net,
+ int if_index,
+ u16 *index);
+/**
* ib_get_cached_pkey - Returns a cached PKey table entry
* @device: The device to query.
* @port_num: The port number of the device to query.
diff --git a/include/rdma/ib_sa.h b/include/rdma/ib_sa.h
index 7e071a6..6a1b994 100644
--- a/include/rdma/ib_sa.h
+++ b/include/rdma/ib_sa.h
@@ -156,7 +156,9 @@ struct ib_sa_path_rec {
u8 preference;
u8 smac[ETH_ALEN];
u8 dmac[ETH_ALEN];
- u16 vlan_id;
+ u16 vlan_id;
+ int ifindex;
+ struct net *net;
};
#define IB_SA_MCMEMBER_REC_MGID IB_SA_COMP_MASK( 0)
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index f9afa91..9e7505a 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -48,6 +48,7 @@
#include <linux/rwsem.h>
#include <linux/scatterlist.h>
#include <linux/workqueue.h>
+#include <net/net_namespace.h>
#include <uapi/linux/if_ether.h>
#include <linux/atomic.h>
@@ -1805,7 +1806,8 @@ enum rdma_link_layer rdma_port_get_link_layer(struct ib_device *device,
u8 port_num);
int ib_query_gid(struct ib_device *device,
- u8 port_num, int index, union ib_gid *gid);
+ u8 port_num, int index, union ib_gid *gid,
+ struct ib_gid_attr *attr);
int ib_query_pkey(struct ib_device *device,
u8 port_num, u16 index, u16 *pkey);
@@ -1819,7 +1821,8 @@ int ib_modify_port(struct ib_device *device,
struct ib_port_modify *port_modify);
int ib_find_gid(struct ib_device *device, union ib_gid *gid,
- u8 *port_num, u16 *index);
+ enum ib_gid_type gid_type, struct net *net,
+ int if_index, u8 *port_num, u16 *index);
int ib_find_pkey(struct ib_device *device,
u8 port_num, u16 pkey, u16 *index);
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 08/14] IB/core: Report gid_type and gid_ndev through sysfs
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (6 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 09/14] IB/core: Support find sgid index using a filter function Matan Barak
` (5 subsequent siblings)
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
Since we've added GID attributes to the RoCE GID table,
the users need a convenient way to query them.
Adding the GID type and relate net device to IB's sysfs.
The new attributes are available in:
/sys/class/infiniband/<device>/ports/<port>/gid_attrs/ndevs/<index>
/sys/class/infiniband/<device>/ports/<port>/gid_attrs/types/<index>
The <index> corresponds to the index of the respective GID in:
/sys/class/infiniband/<device>/ports/<port>/gids/<index>
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/core/core_priv.h | 2 +
drivers/infiniband/core/roce_gid_table.c | 12 ++
drivers/infiniband/core/sysfs.c | 184 ++++++++++++++++++++++++++++++-
3 files changed, 196 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h
index 0383643..9eebd09 100644
--- a/drivers/infiniband/core/core_priv.h
+++ b/drivers/infiniband/core/core_priv.h
@@ -71,6 +71,8 @@ void ib_enum_roce_ports_of_netdev(roce_netdev_filter filter,
roce_netdev_callback cb,
void *cookie);
+const char *roce_gid_table_type_str(enum ib_gid_type gid_type);
+
int roce_gid_table_get_gid(struct ib_device *ib_dev, u8 port, int index,
union ib_gid *gid, struct ib_gid_attr *attr);
diff --git a/drivers/infiniband/core/roce_gid_table.c b/drivers/infiniband/core/roce_gid_table.c
index 5692bb8..ecaaaf4 100644
--- a/drivers/infiniband/core/roce_gid_table.c
+++ b/drivers/infiniband/core/roce_gid_table.c
@@ -50,6 +50,10 @@ enum gid_attr_find_mask {
GID_ATTR_FIND_MASK_DEFAULT = 1UL << 3,
};
+static const char * const gid_type_str[] = {
+ [IB_GID_TYPE_IB] = "IB/RoCE v1\n",
+};
+
static inline int start_port(struct ib_device *ib_dev)
{
return (ib_dev->node_type == RDMA_NODE_IB_SWITCH) ? 0 : 1;
@@ -60,6 +64,14 @@ struct dev_put_rcu {
struct net_device *ndev;
};
+const char *roce_gid_table_type_str(enum ib_gid_type gid_type)
+{
+ if (gid_type < ARRAY_SIZE(gid_type_str) && gid_type_str[gid_type])
+ return gid_type_str[gid_type];
+
+ return "Invalid GID type";
+}
+
static void put_ndev(struct rcu_head *rcu)
{
struct dev_put_rcu *put_rcu =
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 5cee246..cfcddf0 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -37,12 +37,22 @@
#include <linux/slab.h>
#include <linux/stat.h>
#include <linux/string.h>
+#include <linux/netdevice.h>
#include <rdma/ib_mad.h>
+struct ib_port;
+
+struct gid_attr_group {
+ struct ib_port *port;
+ struct kobject kobj;
+ struct attribute_group ndev;
+ struct attribute_group type;
+};
struct ib_port {
struct kobject kobj;
struct ib_device *ibdev;
+ struct gid_attr_group *gid_attr_group;
struct attribute_group gid_group;
struct attribute_group pkey_group;
u8 port_num;
@@ -84,6 +94,24 @@ static const struct sysfs_ops port_sysfs_ops = {
.show = port_attr_show
};
+static ssize_t gid_attr_show(struct kobject *kobj,
+ struct attribute *attr, char *buf)
+{
+ struct port_attribute *port_attr =
+ container_of(attr, struct port_attribute, attr);
+ struct ib_port *p = container_of(kobj, struct gid_attr_group,
+ kobj)->port;
+
+ if (!port_attr->show)
+ return -EIO;
+
+ return port_attr->show(p, port_attr, buf);
+}
+
+static const struct sysfs_ops gid_attr_sysfs_ops = {
+ .show = gid_attr_show
+};
+
static ssize_t state_show(struct ib_port *p, struct port_attribute *unused,
char *buf)
{
@@ -281,6 +309,46 @@ static struct attribute *port_default_attrs[] = {
NULL
};
+static size_t print_ndev(struct ib_gid_attr *gid_attr, char *buf)
+{
+ if (!gid_attr->ndev)
+ return -EINVAL;
+
+ return sprintf(buf, "%s\n", gid_attr->ndev->name);
+}
+
+static size_t print_gid_type(struct ib_gid_attr *gid_attr, char *buf)
+{
+ return sprintf(buf, "%s", roce_gid_table_type_str(gid_attr->gid_type));
+}
+
+static ssize_t _show_port_gid_attr(struct ib_port *p,
+ struct port_attribute *attr,
+ char *buf,
+ size_t (*print)(struct ib_gid_attr *gid_attr,
+ char *buf))
+{
+ struct port_table_attribute *tab_attr =
+ container_of(attr, struct port_table_attribute, attr);
+ union ib_gid gid;
+ struct ib_gid_attr gid_attr;
+ ssize_t ret;
+ va_list args;
+
+ rcu_read_lock();
+ ret = ib_query_gid(p->ibdev, p->port_num, tab_attr->index, &gid,
+ &gid_attr);
+ if (ret)
+ goto err;
+
+ ret = print(&gid_attr, buf);
+
+err:
+ va_end(args);
+ rcu_read_unlock();
+ return ret;
+}
+
static ssize_t show_port_gid(struct ib_port *p, struct port_attribute *attr,
char *buf)
{
@@ -296,6 +364,19 @@ static ssize_t show_port_gid(struct ib_port *p, struct port_attribute *attr,
return sprintf(buf, "%pI6\n", gid.raw);
}
+static ssize_t show_port_gid_attr_ndev(struct ib_port *p,
+ struct port_attribute *attr, char *buf)
+{
+ return _show_port_gid_attr(p, attr, buf, print_ndev);
+}
+
+static ssize_t show_port_gid_attr_gid_type(struct ib_port *p,
+ struct port_attribute *attr,
+ char *buf)
+{
+ return _show_port_gid_attr(p, attr, buf, print_gid_type);
+}
+
static ssize_t show_port_pkey(struct ib_port *p, struct port_attribute *attr,
char *buf)
{
@@ -446,12 +527,41 @@ static void ib_port_release(struct kobject *kobj)
kfree(p);
}
+static void ib_port_gid_attr_release(struct kobject *kobj)
+{
+ struct gid_attr_group *g = container_of(kobj, struct gid_attr_group,
+ kobj);
+ struct attribute *a;
+ int i;
+
+ if (g->ndev.attrs) {
+ for (i = 0; (a = g->ndev.attrs[i]); ++i)
+ kfree(a);
+
+ kfree(g->ndev.attrs);
+ }
+
+ if (g->type.attrs) {
+ for (i = 0; (a = g->type.attrs[i]); ++i)
+ kfree(a);
+
+ kfree(g->type.attrs);
+ }
+
+ kfree(g);
+}
+
static struct kobj_type port_type = {
.release = ib_port_release,
.sysfs_ops = &port_sysfs_ops,
.default_attrs = port_default_attrs
};
+static struct kobj_type gid_attr_type = {
+ .sysfs_ops = &gid_attr_sysfs_ops,
+ .release = ib_port_gid_attr_release
+};
+
static void ib_device_release(struct device *device)
{
struct ib_device *dev = container_of(device, struct ib_device, dev);
@@ -545,9 +655,23 @@ static int add_port(struct ib_device *device, int port_num,
return ret;
}
+ p->gid_attr_group = kzalloc(sizeof(*p->gid_attr_group), GFP_KERNEL);
+ if (!p->gid_attr_group) {
+ ret = -ENOMEM;
+ goto err_put;
+ }
+
+ p->gid_attr_group->port = p;
+ ret = kobject_init_and_add(&p->gid_attr_group->kobj, &gid_attr_type,
+ &p->kobj, "gid_attrs");
+ if (ret) {
+ kfree(p->gid_attr_group);
+ goto err_put;
+ }
+
ret = sysfs_create_group(&p->kobj, &pma_group);
if (ret)
- goto err_put;
+ goto err_put_gid_attrs;
p->gid_group.name = "gids";
p->gid_group.attrs = alloc_group_attrs(show_port_gid, attr.gid_tbl_len);
@@ -560,12 +684,38 @@ static int add_port(struct ib_device *device, int port_num,
if (ret)
goto err_free_gid;
+ p->gid_attr_group->ndev.name = "ndevs";
+ p->gid_attr_group->ndev.attrs = alloc_group_attrs(show_port_gid_attr_ndev,
+ attr.gid_tbl_len);
+ if (!p->gid_attr_group->ndev.attrs) {
+ ret = -ENOMEM;
+ goto err_remove_gid;
+ }
+
+ ret = sysfs_create_group(&p->gid_attr_group->kobj,
+ &p->gid_attr_group->ndev);
+ if (ret)
+ goto err_free_gid_ndev;
+
+ p->gid_attr_group->type.name = "types";
+ p->gid_attr_group->type.attrs = alloc_group_attrs(show_port_gid_attr_gid_type,
+ attr.gid_tbl_len);
+ if (!p->gid_attr_group->type.attrs) {
+ ret = -ENOMEM;
+ goto err_remove_gid_ndev;
+ }
+
+ ret = sysfs_create_group(&p->gid_attr_group->kobj,
+ &p->gid_attr_group->type);
+ if (ret)
+ goto err_free_gid_type;
+
p->pkey_group.name = "pkeys";
p->pkey_group.attrs = alloc_group_attrs(show_port_pkey,
attr.pkey_tbl_len);
if (!p->pkey_group.attrs) {
ret = -ENOMEM;
- goto err_remove_gid;
+ goto err_remove_gid_type;
}
ret = sysfs_create_group(&p->kobj, &p->pkey_group);
@@ -593,6 +743,28 @@ err_free_pkey:
kfree(p->pkey_group.attrs);
p->pkey_group.attrs = NULL;
+err_remove_gid_type:
+ sysfs_remove_group(&p->gid_attr_group->kobj,
+ &p->gid_attr_group->type);
+
+err_free_gid_type:
+ for (i = 0; i < attr.gid_tbl_len; ++i)
+ kfree(p->gid_attr_group->type.attrs[i]);
+
+ kfree(p->gid_attr_group->type.attrs);
+ p->gid_attr_group->type.attrs = NULL;
+
+err_remove_gid_ndev:
+ sysfs_remove_group(&p->gid_attr_group->kobj,
+ &p->gid_attr_group->ndev);
+
+err_free_gid_ndev:
+ for (i = 0; i < attr.gid_tbl_len; ++i)
+ kfree(p->gid_attr_group->ndev.attrs[i]);
+
+ kfree(p->gid_attr_group->ndev.attrs);
+ p->gid_attr_group->ndev.attrs = NULL;
+
err_remove_gid:
sysfs_remove_group(&p->kobj, &p->gid_group);
@@ -606,6 +778,9 @@ err_free_gid:
err_remove_pma:
sysfs_remove_group(&p->kobj, &pma_group);
+err_put_gid_attrs:
+ kobject_put(&p->gid_attr_group->kobj);
+
err_put:
kobject_put(&p->kobj);
return ret;
@@ -826,6 +1001,11 @@ static void free_port_list_attributes(struct ib_device *device)
sysfs_remove_group(p, &pma_group);
sysfs_remove_group(p, &port->pkey_group);
sysfs_remove_group(p, &port->gid_group);
+ sysfs_remove_group(&port->gid_attr_group->kobj,
+ &port->gid_attr_group->ndev);
+ sysfs_remove_group(&port->gid_attr_group->kobj,
+ &port->gid_attr_group->type);
+ kobject_put(&port->gid_attr_group->kobj);
kobject_put(p);
}
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 09/14] IB/core: Support find sgid index using a filter function
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (7 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 08/14] IB/core: Report gid_type and gid_ndev through sysfs Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 10/14] IB/core: Modify ib_verbs and cma in order to use roce_gid_table Matan Barak
` (4 subsequent siblings)
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
Sometimes a sgid index need to be found based on variable parameters.
For example, when the CM gets a packet from network, it needs to
match a sgid_index that matches the appropriate L2 attributes
of a packet. Extending the cache's API to include Ethernet L2
attribute is problematic, since they may be vastly extended
in the future. As a result, we add a find function that
gets a user filter function and searches the GID table
until a match is found.
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/core/cache.c | 24 +++++++++++++
drivers/infiniband/core/core_priv.h | 9 +++++
drivers/infiniband/core/roce_gid_table.c | 61 ++++++++++++++++++++++++++++++++
include/rdma/ib_cache.h | 27 ++++++++++++++
4 files changed, 121 insertions(+)
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index 5af1e6a..020006e 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -280,6 +280,30 @@ int ib_find_cached_gid_by_port(struct ib_device *device,
}
EXPORT_SYMBOL(ib_find_cached_gid_by_port);
+int ib_find_gid_by_filter(struct ib_device *device,
+ union ib_gid *gid,
+ u8 port_num,
+ bool (*filter)(const union ib_gid *gid,
+ const struct ib_gid_attr *,
+ void *),
+ void *context, u16 *index)
+{
+ /* Look for a RoCE device with the specified GID. */
+ if (!ib_cache_use_roce_gid_table(device, port_num))
+ return roce_gid_table_find_gid_by_filter(device, gid,
+ port_num, filter,
+ context, index);
+
+ /* Only RoCE GID table supports filter function */
+ if (filter)
+ return -EPROTONOSUPPORT;
+
+ /* If no RoCE devices with the specified GID, look for IB device. */
+ return __ib_find_cached_gid_by_port(device, port_num,
+ gid, index);
+}
+EXPORT_SYMBOL(ib_find_gid_by_filter);
+
int ib_get_cached_pkey(struct ib_device *device,
u8 port_num,
int index,
diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h
index 9eebd09..de795ef 100644
--- a/drivers/infiniband/core/core_priv.h
+++ b/drivers/infiniband/core/core_priv.h
@@ -84,6 +84,15 @@ int roce_gid_table_find_gid_by_port(struct ib_device *ib_dev, union ib_gid *gid,
enum ib_gid_type gid_type, u8 port,
struct net *net, int if_index, u16 *index);
+int roce_gid_table_find_gid_by_filter(struct ib_device *ib_dev,
+ union ib_gid *gid,
+ u8 port,
+ bool (*filter)(const union ib_gid *gid,
+ const struct ib_gid_attr *,
+ void *),
+ void *context,
+ u16 *index);
+
enum roce_gid_table_default_mode {
ROCE_GID_TABLE_DEFAULT_MODE_SET,
ROCE_GID_TABLE_DEFAULT_MODE_DELETE
diff --git a/drivers/infiniband/core/roce_gid_table.c b/drivers/infiniband/core/roce_gid_table.c
index ecaaaf4..db6b1df 100644
--- a/drivers/infiniband/core/roce_gid_table.c
+++ b/drivers/infiniband/core/roce_gid_table.c
@@ -450,6 +450,67 @@ int roce_gid_table_find_gid_by_port(struct ib_device *ib_dev, union ib_gid *gid,
return -ENOENT;
}
+int roce_gid_table_find_gid_by_filter(struct ib_device *ib_dev,
+ union ib_gid *gid,
+ u8 port,
+ bool (*filter)(const union ib_gid *,
+ const struct ib_gid_attr *,
+ void *),
+ void *context,
+ u16 *index)
+{
+ struct ib_roce_gid_table **ports_table =
+ READ_ONCE(ib_dev->cache.roce_gid_table);
+ struct ib_roce_gid_table *table;
+ unsigned int i;
+ bool found = false;
+
+ /* make sure we read the ports_table */
+ smp_rmb();
+
+ if (!ports_table)
+ return -EOPNOTSUPP;
+
+ if (port < start_port(ib_dev) ||
+ port > start_port(ib_dev) + ib_dev->phys_port_cnt ||
+ rdma_port_get_link_layer(ib_dev, port) !=
+ IB_LINK_LAYER_ETHERNET)
+ return -EPROTONOSUPPORT;
+
+ table = ports_table[port - start_port(ib_dev)];
+
+ if (!table)
+ return -ENOENT;
+
+ for (i = 0; i < table->sz; i++) {
+ struct ib_gid_attr attr;
+ unsigned int orig_seq = read_seqcount_begin(&table->data_vec[i].seq);
+
+ if (memcmp(gid, &table->data_vec[i].gid, sizeof(*gid)))
+ continue;
+
+ memcpy(&attr, &table->data_vec[i].attr, sizeof(attr));
+
+ rcu_read_lock();
+
+ if (!read_seqcount_retry(&table->data_vec[i].seq, orig_seq))
+ if (!filter || filter(gid, &attr, context))
+ found = true;
+
+ rcu_read_unlock();
+
+ if (found)
+ break;
+ }
+
+ if (!found)
+ return -ENOENT;
+
+ if (index)
+ *index = i;
+ return 0;
+}
+
static struct ib_roce_gid_table *alloc_roce_gid_table(int sz)
{
unsigned int i;
diff --git a/include/rdma/ib_cache.h b/include/rdma/ib_cache.h
index 96009ca..347c28b 100644
--- a/include/rdma/ib_cache.h
+++ b/include/rdma/ib_cache.h
@@ -111,6 +111,33 @@ int ib_find_cached_gid_by_port(struct ib_device *device,
struct net *net,
int if_index,
u16 *index);
+
+/**
+ * ib_find_gid_by_filter - Returns the GID table index where a specified
+ * GID value occurs
+ * @device: The device to query.
+ * @gid: The GID value to search for.
+ * @port_num: The port number of the device where the GID value could be
+ * searched.
+ * @filter: The filter function is executed on any matching GID in the table.
+ * If the filter function returns true, the corresponding index is returned,
+ * otherwise, we continue searching the GID table. It's guaranteed that
+ * while filter is executed, ndev field is valid and the structure won't
+ * change. filter is executed in an atomic context. filter must be NULL
+ * when RoCE GID table isn't supported on the respective device's port.
+ * @index: The index into the cached GID table where the GID was found. This
+ * parameter may be NULL.
+ *
+ * ib_find_gid_by_filter() searches for the specified GID value in
+ * the local software cache.
+ */
+int ib_find_gid_by_filter(struct ib_device *device,
+ union ib_gid *gid,
+ u8 port_num,
+ bool (*filter)(const union ib_gid *gid,
+ const struct ib_gid_attr *,
+ void *),
+ void *context, u16 *index);
/**
* ib_get_cached_pkey - Returns a cached PKey table entry
* @device: The device to query.
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 10/14] IB/core: Modify ib_verbs and cma in order to use roce_gid_table
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (8 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 09/14] IB/core: Support find sgid index using a filter function Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 11/14] RDMA/ocrdma: Changes in driver to incorporate the moving of GID Table mgmt to IB/Core Matan Barak
` (3 subsequent siblings)
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
Previously, we resolved the dmac and took the smac and vlan
from the resolved address. Changing that into finding a net
device that matches the IP and vlan of the network packet
and querying the RoCE GID table for this net device,
GID and GID type.
ocrdma driver changes were done by Somnath Kotur <Somnath.Kotur-iH1Dq9VlAzfQT0dZR+AlfA@public.gmane.org>
Signed-off-by: Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/core/addr.c | 3 +-
drivers/infiniband/core/cm.c | 30 ------
drivers/infiniband/core/cma.c | 9 --
drivers/infiniband/core/core_priv.h | 4 +-
drivers/infiniband/core/sa_query.c | 4 -
drivers/infiniband/core/ucma.c | 1 -
drivers/infiniband/core/uverbs_cmd.c | 3 +-
drivers/infiniband/core/verbs.c | 162 ++++++++++++++++++-------------
drivers/infiniband/hw/mlx4/ah.c | 15 ++-
drivers/infiniband/hw/mlx4/mad.c | 12 ++-
drivers/infiniband/hw/mlx4/mcg.c | 2 +-
drivers/infiniband/hw/mlx4/mlx4_ib.h | 2 +-
drivers/infiniband/hw/mlx4/qp.c | 48 +++++++--
drivers/infiniband/hw/ocrdma/ocrdma.h | 1 +
drivers/infiniband/hw/ocrdma/ocrdma_ah.c | 20 ++--
drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 19 ++--
include/rdma/ib_addr.h | 2 +-
include/rdma/ib_sa.h | 2 -
include/rdma/ib_verbs.h | 15 ++-
19 files changed, 195 insertions(+), 159 deletions(-)
diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c
index f80da50..43af7f5 100644
--- a/drivers/infiniband/core/addr.c
+++ b/drivers/infiniband/core/addr.c
@@ -458,7 +458,7 @@ static void resolve_cb(int status, struct sockaddr *src_addr,
}
int rdma_addr_find_dmac_by_grh(union ib_gid *sgid, union ib_gid *dgid, u8 *dmac,
- u16 *vlan_id)
+ u16 *vlan_id, int if_index)
{
int ret = 0;
struct rdma_dev_addr dev_addr;
@@ -481,6 +481,7 @@ int rdma_addr_find_dmac_by_grh(union ib_gid *sgid, union ib_gid *dgid, u8 *dmac,
return ret;
memset(&dev_addr, 0, sizeof(dev_addr));
+ dev_addr.bound_dev_if = if_index;
ctx.addr = &dev_addr;
init_completion(&ctx.comp);
diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
index d88f2ae..7974e74 100644
--- a/drivers/infiniband/core/cm.c
+++ b/drivers/infiniband/core/cm.c
@@ -178,8 +178,6 @@ struct cm_av {
struct ib_ah_attr ah_attr;
u16 pkey_index;
u8 timeout;
- u8 valid;
- u8 smac[ETH_ALEN];
};
struct cm_work {
@@ -382,7 +380,6 @@ static int cm_init_av_by_path(struct ib_sa_path_rec *path, struct cm_av *av)
&av->ah_attr);
av->timeout = path->packet_life_time + 1;
- av->valid = 1;
return 0;
}
@@ -1563,7 +1560,6 @@ static int cm_req_handler(struct cm_work *work)
cm_format_paths_from_req(req_msg, &work->path[0], &work->path[1]);
memcpy(work->path[0].dmac, cm_id_priv->av.ah_attr.dmac, ETH_ALEN);
- work->path[0].vlan_id = cm_id_priv->av.ah_attr.vlan_id;
ret = cm_init_av_by_path(&work->path[0], &cm_id_priv->av);
if (ret) {
ib_get_cached_gid(work->port->cm_dev->ib_device,
@@ -3511,32 +3507,6 @@ static int cm_init_qp_rtr_attr(struct cm_id_private *cm_id_priv,
*qp_attr_mask = IB_QP_STATE | IB_QP_AV | IB_QP_PATH_MTU |
IB_QP_DEST_QPN | IB_QP_RQ_PSN;
qp_attr->ah_attr = cm_id_priv->av.ah_attr;
- if (!cm_id_priv->av.valid) {
- spin_unlock_irqrestore(&cm_id_priv->lock, flags);
- return -EINVAL;
- }
- if (cm_id_priv->av.ah_attr.vlan_id != 0xffff) {
- qp_attr->vlan_id = cm_id_priv->av.ah_attr.vlan_id;
- *qp_attr_mask |= IB_QP_VID;
- }
- if (!is_zero_ether_addr(cm_id_priv->av.smac)) {
- memcpy(qp_attr->smac, cm_id_priv->av.smac,
- sizeof(qp_attr->smac));
- *qp_attr_mask |= IB_QP_SMAC;
- }
- if (cm_id_priv->alt_av.valid) {
- if (cm_id_priv->alt_av.ah_attr.vlan_id != 0xffff) {
- qp_attr->alt_vlan_id =
- cm_id_priv->alt_av.ah_attr.vlan_id;
- *qp_attr_mask |= IB_QP_ALT_VID;
- }
- if (!is_zero_ether_addr(cm_id_priv->alt_av.smac)) {
- memcpy(qp_attr->alt_smac,
- cm_id_priv->alt_av.smac,
- sizeof(qp_attr->alt_smac));
- *qp_attr_mask |= IB_QP_ALT_SMAC;
- }
- }
qp_attr->path_mtu = cm_id_priv->path_mtu;
qp_attr->dest_qp_num = be32_to_cpu(cm_id_priv->remote_qpn);
qp_attr->rq_psn = be32_to_cpu(cm_id_priv->rq_psn);
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 7e6a0b2..194ea82 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -666,15 +666,6 @@ static int cma_modify_qp_rtr(struct rdma_id_private *id_priv,
if (ret)
goto out;
- if (rdma_node_get_transport(id_priv->cma_dev->device->node_type)
- == RDMA_TRANSPORT_IB &&
- rdma_port_get_link_layer(id_priv->id.device, id_priv->id.port_num)
- == IB_LINK_LAYER_ETHERNET) {
- ret = rdma_addr_find_smac_by_sgid(&sgid, qp_attr.smac, NULL);
-
- if (ret)
- goto out;
- }
if (conn_param)
qp_attr.max_dest_rd_atomic = conn_param->responder_resources;
ret = ib_modify_qp(id_priv->id.qp, &qp_attr, qp_attr_mask);
diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h
index de795ef..033a381 100644
--- a/drivers/infiniband/core/core_priv.h
+++ b/drivers/infiniband/core/core_priv.h
@@ -52,8 +52,8 @@ void ib_sysfs_cleanup(void);
int ib_cache_setup(void);
void ib_cache_cleanup(void);
-int ib_resolve_eth_l2_attrs(struct ib_qp *qp,
- struct ib_qp_attr *qp_attr, int *qp_attr_mask);
+int ib_resolve_eth_dmac(struct ib_qp *qp,
+ struct ib_qp_attr *qp_attr, int *qp_attr_mask);
typedef void (*roce_netdev_callback)(struct ib_device *device, u8 port,
struct net_device *idev, void *cookie);
diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c
index 5b20237..705b6b8 100644
--- a/drivers/infiniband/core/sa_query.c
+++ b/drivers/infiniband/core/sa_query.c
@@ -559,11 +559,7 @@ int ib_init_ah_from_path(struct ib_device *device, u8 port_num,
}
if (force_grh) {
memcpy(ah_attr->dmac, rec->dmac, ETH_ALEN);
- ah_attr->vlan_id = rec->vlan_id;
- } else {
- ah_attr->vlan_id = 0xffff;
}
-
return 0;
}
EXPORT_SYMBOL(ib_init_ah_from_path);
diff --git a/drivers/infiniband/core/ucma.c b/drivers/infiniband/core/ucma.c
index 45d67e9..5eacda4 100644
--- a/drivers/infiniband/core/ucma.c
+++ b/drivers/infiniband/core/ucma.c
@@ -1125,7 +1125,6 @@ static int ucma_set_ib_path(struct ucma_context *ctx,
return -EINVAL;
memset(&sa_path, 0, sizeof(sa_path));
- sa_path.vlan_id = 0xffff;
ib_sa_unpack_path(path_data->path_rec, &sa_path);
ret = rdma_set_ib_paths(ctx->cm_id, &sa_path, 1);
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index a9f0489..0cf1360 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -2095,7 +2095,7 @@ ssize_t ib_uverbs_modify_qp(struct ib_uverbs_file *file,
attr->alt_ah_attr.port_num = cmd.alt_dest.port_num;
if (qp->real_qp == qp) {
- ret = ib_resolve_eth_l2_attrs(qp, attr, &cmd.attr_mask);
+ ret = ib_resolve_eth_dmac(qp, attr, &cmd.attr_mask);
if (ret)
goto release_qp;
ret = qp->device->modify_qp(qp, attr,
@@ -2559,7 +2559,6 @@ ssize_t ib_uverbs_create_ah(struct ib_uverbs_file *file,
attr.grh.sgid_index = cmd.attr.grh.sgid_index;
attr.grh.hop_limit = cmd.attr.grh.hop_limit;
attr.grh.traffic_class = cmd.attr.grh.traffic_class;
- attr.vlan_id = 0;
memset(&attr.dmac, 0, sizeof(attr.dmac));
memcpy(attr.grh.dgid.raw, cmd.attr.grh.dgid, 16);
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 1fe3e71..2f5fd7a 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -41,6 +41,9 @@
#include <linux/export.h>
#include <linux/string.h>
#include <linux/slab.h>
+#include <linux/in.h>
+#include <linux/in6.h>
+#include <net/addrconf.h>
#include <rdma/ib_verbs.h>
#include <rdma/ib_cache.h>
@@ -192,6 +195,35 @@ struct ib_ah *ib_create_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr)
}
EXPORT_SYMBOL(ib_create_ah);
+struct find_gid_index_context {
+ u16 vlan_id;
+};
+
+static bool find_gid_index(const union ib_gid *gid,
+ const struct ib_gid_attr *gid_attr,
+ void *context)
+{
+ struct find_gid_index_context *ctx =
+ (struct find_gid_index_context *)context;
+
+ if ((!!(ctx->vlan_id != 0xffff) == !is_vlan_dev(gid_attr->ndev)) ||
+ (is_vlan_dev(gid_attr->ndev) &&
+ vlan_dev_vlan_id(gid_attr->ndev) != ctx->vlan_id))
+ return false;
+
+ return true;
+}
+
+static int get_sgid_index_from_eth(struct ib_device *device, u8 port_num,
+ u16 vlan_id, union ib_gid *sgid,
+ u16 *gid_index)
+{
+ struct find_gid_index_context context = {.vlan_id = vlan_id};
+
+ return ib_find_gid_by_filter(device, sgid, port_num, find_gid_index,
+ &context, gid_index);
+}
+
int ib_init_ah_from_wc(struct ib_device *device, u8 port_num, struct ib_wc *wc,
struct ib_grh *grh, struct ib_ah_attr *ah_attr)
{
@@ -203,21 +235,30 @@ int ib_init_ah_from_wc(struct ib_device *device, u8 port_num, struct ib_wc *wc,
memset(ah_attr, 0, sizeof *ah_attr);
if (is_eth) {
+ u16 vlan_id = wc->wc_flags & IB_WC_WITH_VLAN ?
+ wc->vlan_id : 0xffff;
+
if (!(wc->wc_flags & IB_WC_GRH))
return -EPROTOTYPE;
- if (wc->wc_flags & IB_WC_WITH_SMAC &&
- wc->wc_flags & IB_WC_WITH_VLAN) {
- memcpy(ah_attr->dmac, wc->smac, ETH_ALEN);
- ah_attr->vlan_id = wc->vlan_id;
- } else {
+ if (!(wc->wc_flags & IB_WC_WITH_SMAC) ||
+ !(wc->wc_flags & IB_WC_WITH_VLAN)) {
ret = rdma_addr_find_dmac_by_grh(&grh->dgid, &grh->sgid,
- ah_attr->dmac, &ah_attr->vlan_id);
+ ah_attr->dmac,
+ wc->wc_flags & IB_WC_WITH_VLAN ?
+ NULL : &vlan_id,
+ 0);
if (ret)
return ret;
}
- } else {
- ah_attr->vlan_id = 0xffff;
+
+ ret = get_sgid_index_from_eth(device, port_num, vlan_id,
+ &grh->dgid, &gid_index);
+ if (ret)
+ return ret;
+
+ if (wc->wc_flags & IB_WC_WITH_SMAC)
+ memcpy(ah_attr->dmac, wc->smac, ETH_ALEN);
}
ah_attr->dlid = wc->slid;
@@ -229,10 +270,14 @@ int ib_init_ah_from_wc(struct ib_device *device, u8 port_num, struct ib_wc *wc,
ah_attr->ah_flags = IB_AH_GRH;
ah_attr->grh.dgid = grh->sgid;
- ret = ib_find_cached_gid(device, &grh->dgid, IB_GID_TYPE_IB,
- NULL, 0, &port_num, &gid_index);
- if (ret)
- return ret;
+ if (!is_eth) {
+ ret = ib_find_cached_gid_by_port(device, &grh->dgid,
+ IB_GID_TYPE_IB,
+ port_num, NULL, 0,
+ &gid_index);
+ if (ret)
+ return ret;
+ }
ah_attr->grh.sgid_index = (u8) gid_index;
flow_class = be32_to_cpu(grh->version_tclass_flow);
@@ -502,9 +547,7 @@ EXPORT_SYMBOL(ib_create_qp);
static const struct {
int valid;
enum ib_qp_attr_mask req_param[IB_QPT_MAX];
- enum ib_qp_attr_mask req_param_add_eth[IB_QPT_MAX];
enum ib_qp_attr_mask opt_param[IB_QPT_MAX];
- enum ib_qp_attr_mask opt_param_add_eth[IB_QPT_MAX];
} qp_state_table[IB_QPS_ERR + 1][IB_QPS_ERR + 1] = {
[IB_QPS_RESET] = {
[IB_QPS_RESET] = { .valid = 1 },
@@ -585,12 +628,6 @@ static const struct {
IB_QP_MAX_DEST_RD_ATOMIC |
IB_QP_MIN_RNR_TIMER),
},
- .req_param_add_eth = {
- [IB_QPT_RC] = (IB_QP_SMAC),
- [IB_QPT_UC] = (IB_QP_SMAC),
- [IB_QPT_XRC_INI] = (IB_QP_SMAC),
- [IB_QPT_XRC_TGT] = (IB_QP_SMAC)
- },
.opt_param = {
[IB_QPT_UD] = (IB_QP_PKEY_INDEX |
IB_QP_QKEY),
@@ -611,21 +648,7 @@ static const struct {
[IB_QPT_GSI] = (IB_QP_PKEY_INDEX |
IB_QP_QKEY),
},
- .opt_param_add_eth = {
- [IB_QPT_RC] = (IB_QP_ALT_SMAC |
- IB_QP_VID |
- IB_QP_ALT_VID),
- [IB_QPT_UC] = (IB_QP_ALT_SMAC |
- IB_QP_VID |
- IB_QP_ALT_VID),
- [IB_QPT_XRC_INI] = (IB_QP_ALT_SMAC |
- IB_QP_VID |
- IB_QP_ALT_VID),
- [IB_QPT_XRC_TGT] = (IB_QP_ALT_SMAC |
- IB_QP_VID |
- IB_QP_ALT_VID)
- }
- }
+ },
},
[IB_QPS_RTR] = {
[IB_QPS_RESET] = { .valid = 1 },
@@ -847,13 +870,6 @@ int ib_modify_qp_is_ok(enum ib_qp_state cur_state, enum ib_qp_state next_state,
req_param = qp_state_table[cur_state][next_state].req_param[type];
opt_param = qp_state_table[cur_state][next_state].opt_param[type];
- if (ll == IB_LINK_LAYER_ETHERNET) {
- req_param |= qp_state_table[cur_state][next_state].
- req_param_add_eth[type];
- opt_param |= qp_state_table[cur_state][next_state].
- opt_param_add_eth[type];
- }
-
if ((mask & req_param) != req_param)
return 0;
@@ -864,41 +880,55 @@ int ib_modify_qp_is_ok(enum ib_qp_state cur_state, enum ib_qp_state next_state,
}
EXPORT_SYMBOL(ib_modify_qp_is_ok);
-int ib_resolve_eth_l2_attrs(struct ib_qp *qp,
- struct ib_qp_attr *qp_attr, int *qp_attr_mask)
+int ib_resolve_eth_dmac(struct ib_qp *qp,
+ struct ib_qp_attr *qp_attr, int *qp_attr_mask)
{
int ret = 0;
- union ib_gid sgid;
+ u8 start_port = qp->device->node_type == RDMA_NODE_IB_SWITCH ? 0 : 1;
if ((*qp_attr_mask & IB_QP_AV) &&
- (rdma_port_get_link_layer(qp->device, qp_attr->ah_attr.port_num) == IB_LINK_LAYER_ETHERNET)) {
- ret = ib_query_gid(qp->device, qp_attr->ah_attr.port_num,
- qp_attr->ah_attr.grh.sgid_index, &sgid,
- NULL);
- if (ret)
- goto out;
+ (qp_attr->ah_attr.port_num >= start_port) &&
+ (qp_attr->ah_attr.port_num < start_port + qp->device->phys_port_cnt) &&
+ (rdma_port_get_link_layer(qp->device, qp_attr->ah_attr.port_num) ==
+ IB_LINK_LAYER_ETHERNET)) {
if (rdma_link_local_addr((struct in6_addr *)qp_attr->ah_attr.grh.dgid.raw)) {
- rdma_get_ll_mac((struct in6_addr *)qp_attr->ah_attr.grh.dgid.raw, qp_attr->ah_attr.dmac);
- rdma_get_ll_mac((struct in6_addr *)sgid.raw, qp_attr->smac);
- if (!(*qp_attr_mask & IB_QP_VID))
- qp_attr->vlan_id = rdma_get_vlan_id(&sgid);
+ rdma_get_ll_mac((struct in6_addr *)qp_attr->ah_attr.grh.dgid.raw,
+ qp_attr->ah_attr.dmac);
} else {
- ret = rdma_addr_find_dmac_by_grh(&sgid, &qp_attr->ah_attr.grh.dgid,
- qp_attr->ah_attr.dmac, &qp_attr->vlan_id);
- if (ret)
- goto out;
- ret = rdma_addr_find_smac_by_sgid(&sgid, qp_attr->smac, NULL);
- if (ret)
+ union ib_gid sgid;
+ struct ib_gid_attr sgid_attr;
+ int ifindex;
+
+ rcu_read_lock();
+ ret = ib_query_gid(qp->device,
+ qp_attr->ah_attr.port_num,
+ qp_attr->ah_attr.grh.sgid_index,
+ &sgid, &sgid_attr);
+
+ if (ret || !sgid_attr.ndev) {
+ if (!ret)
+ ret = -ENXIO;
+ rcu_read_unlock();
goto out;
+ }
+
+ dev_hold(sgid_attr.ndev);
+ ifindex = sgid_attr.ndev->ifindex;
+
+ rcu_read_unlock();
+
+ ret = rdma_addr_find_dmac_by_grh(&sgid,
+ &qp_attr->ah_attr.grh.dgid,
+ qp_attr->ah_attr.dmac,
+ NULL, ifindex);
+
+ dev_put(sgid_attr.ndev);
}
- *qp_attr_mask |= IB_QP_SMAC;
- if (qp_attr->vlan_id < 0xFFFF)
- *qp_attr_mask |= IB_QP_VID;
}
out:
return ret;
}
-EXPORT_SYMBOL(ib_resolve_eth_l2_attrs);
+EXPORT_SYMBOL(ib_resolve_eth_dmac);
int ib_modify_qp(struct ib_qp *qp,
@@ -907,7 +937,7 @@ int ib_modify_qp(struct ib_qp *qp,
{
int ret;
- ret = ib_resolve_eth_l2_attrs(qp, qp_attr, &qp_attr_mask);
+ ret = ib_resolve_eth_dmac(qp, qp_attr, &qp_attr_mask);
if (ret)
return ret;
diff --git a/drivers/infiniband/hw/mlx4/ah.c b/drivers/infiniband/hw/mlx4/ah.c
index f50a546..aaeeb60 100644
--- a/drivers/infiniband/hw/mlx4/ah.c
+++ b/drivers/infiniband/hw/mlx4/ah.c
@@ -76,7 +76,9 @@ static struct ib_ah *create_iboe_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr
struct mlx4_dev *dev = ibdev->dev;
int is_mcast = 0;
struct in6_addr in6;
- u16 vlan_tag;
+ u16 vlan_tag = 0xffff;
+ union ib_gid sgid;
+ struct ib_gid_attr gid_attr;
memcpy(&in6, ah_attr->grh.dgid.raw, sizeof(in6));
if (rdma_is_multicast_addr(&in6)) {
@@ -85,7 +87,16 @@ static struct ib_ah *create_iboe_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr
} else {
memcpy(ah->av.eth.mac, ah_attr->dmac, ETH_ALEN);
}
- vlan_tag = ah_attr->vlan_id;
+ rcu_read_lock();
+ ib_get_cached_gid(pd->device, ah_attr->port_num,
+ ah_attr->grh.sgid_index, &sgid, &gid_attr);
+ memset(ah->av.eth.s_mac, 0, ETH_ALEN);
+ if (gid_attr.ndev) {
+ if (is_vlan_dev(gid_attr.ndev))
+ vlan_tag = vlan_dev_vlan_id(gid_attr.ndev);
+ memcpy(ah->av.eth.s_mac, gid_attr.ndev->dev_addr, ETH_ALEN);
+ }
+ rcu_read_unlock();
if (vlan_tag < 0x1000)
vlan_tag |= (ah_attr->sl & 7) << 13;
ah->av.eth.port_pd = cpu_to_be32(to_mpd(pd)->pdn | (ah_attr->port_num << 24));
diff --git a/drivers/infiniband/hw/mlx4/mad.c b/drivers/infiniband/hw/mlx4/mad.c
index 9cd2b00..42b033a 100644
--- a/drivers/infiniband/hw/mlx4/mad.c
+++ b/drivers/infiniband/hw/mlx4/mad.c
@@ -1166,7 +1166,7 @@ static int is_proxy_qp0(struct mlx4_ib_dev *dev, int qpn, int slave)
int mlx4_ib_send_to_wire(struct mlx4_ib_dev *dev, int slave, u8 port,
enum ib_qp_type dest_qpt, u16 pkey_index,
u32 remote_qpn, u32 qkey, struct ib_ah_attr *attr,
- u8 *s_mac, struct ib_mad *mad)
+ u8 *s_mac, u16 vlan_id, struct ib_mad *mad)
{
struct ib_sge list;
struct ib_send_wr wr, *bad_wr;
@@ -1253,6 +1253,9 @@ int mlx4_ib_send_to_wire(struct mlx4_ib_dev *dev, int slave, u8 port,
wr.send_flags = IB_SEND_SIGNALED;
if (s_mac)
memcpy(to_mah(ah)->av.eth.s_mac, s_mac, 6);
+ if (vlan_id < 0x1000)
+ vlan_id |= (attr->sl & 7) << 13;
+ to_mah(ah)->av.eth.vlan = cpu_to_be16(vlan_id);
ret = ib_post_send(send_qp, &wr, &bad_wr);
@@ -1289,6 +1292,7 @@ static void mlx4_ib_multiplex_mad(struct mlx4_ib_demux_pv_ctx *ctx, struct ib_wc
u8 *slave_id;
int slave;
int port;
+ u16 vlan_id;
/* Get slave that sent this packet */
if (wc->src_qp < dev->dev->phys_caps.base_proxy_sqpn ||
@@ -1374,10 +1378,10 @@ static void mlx4_ib_multiplex_mad(struct mlx4_ib_demux_pv_ctx *ctx, struct ib_wc
return;
ah_attr.port_num = port;
memcpy(ah_attr.dmac, tunnel->hdr.mac, 6);
- ah_attr.vlan_id = be16_to_cpu(tunnel->hdr.vlan);
+ vlan_id = be16_to_cpu(tunnel->hdr.vlan);
/* if slave have default vlan use it */
mlx4_get_slave_default_vlan(dev->dev, ctx->port, slave,
- &ah_attr.vlan_id, &ah_attr.sl);
+ &vlan_id, &ah_attr.sl);
mlx4_ib_send_to_wire(dev, slave, ctx->port,
is_proxy_qp0(dev, wc->src_qp, slave) ?
@@ -1385,7 +1389,7 @@ static void mlx4_ib_multiplex_mad(struct mlx4_ib_demux_pv_ctx *ctx, struct ib_wc
be16_to_cpu(tunnel->hdr.pkey_index),
be32_to_cpu(tunnel->hdr.remote_qpn),
be32_to_cpu(tunnel->hdr.qkey),
- &ah_attr, wc->smac, &tunnel->mad);
+ &ah_attr, wc->smac, vlan_id, &tunnel->mad);
}
static int mlx4_ib_alloc_pv_bufs(struct mlx4_ib_demux_pv_ctx *ctx,
diff --git a/drivers/infiniband/hw/mlx4/mcg.c b/drivers/infiniband/hw/mlx4/mcg.c
index ed327e6..86bc158 100644
--- a/drivers/infiniband/hw/mlx4/mcg.c
+++ b/drivers/infiniband/hw/mlx4/mcg.c
@@ -217,7 +217,7 @@ static int send_mad_to_wire(struct mlx4_ib_demux_ctx *ctx, struct ib_mad *mad)
spin_unlock(&dev->sm_lock);
return mlx4_ib_send_to_wire(dev, mlx4_master_func_num(dev->dev),
ctx->port, IB_QPT_GSI, 0, 1, IB_QP1_QKEY,
- &ah_attr, NULL, mad);
+ &ah_attr, NULL, 0xffff, mad);
}
static int send_mad_to_slave(int slave, struct mlx4_ib_demux_ctx *ctx,
diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
index fce39343..3eee5da 100644
--- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
+++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
@@ -766,7 +766,7 @@ int mlx4_ib_send_to_slave(struct mlx4_ib_dev *dev, int slave, u8 port,
int mlx4_ib_send_to_wire(struct mlx4_ib_dev *dev, int slave, u8 port,
enum ib_qp_type dest_qpt, u16 pkey_index, u32 remote_qpn,
u32 qkey, struct ib_ah_attr *attr, u8 *s_mac,
- struct ib_mad *mad);
+ u16 vlan_id, struct ib_mad *mad);
__be64 mlx4_ib_get_new_demux_tid(struct mlx4_ib_demux_ctx *ctx);
diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c
index 58a95b5..37305f5 100644
--- a/drivers/infiniband/hw/mlx4/qp.c
+++ b/drivers/infiniband/hw/mlx4/qp.c
@@ -1387,11 +1387,12 @@ static int _mlx4_set_path(struct mlx4_ib_dev *dev, const struct ib_ah_attr *ah,
static int mlx4_set_path(struct mlx4_ib_dev *dev, const struct ib_qp_attr *qp,
enum ib_qp_attr_mask qp_attr_mask,
struct mlx4_ib_qp *mqp,
- struct mlx4_qp_path *path, u8 port)
+ struct mlx4_qp_path *path, u8 port,
+ u16 vlan_id, u8 *smac)
{
return _mlx4_set_path(dev, &qp->ah_attr,
- mlx4_mac_to_u64((u8 *)qp->smac),
- (qp_attr_mask & IB_QP_VID) ? qp->vlan_id : 0xffff,
+ mlx4_mac_to_u64(smac),
+ vlan_id,
path, &mqp->pri, port);
}
@@ -1402,9 +1403,8 @@ static int mlx4_set_alt_path(struct mlx4_ib_dev *dev,
struct mlx4_qp_path *path, u8 port)
{
return _mlx4_set_path(dev, &qp->alt_ah_attr,
- mlx4_mac_to_u64((u8 *)qp->alt_smac),
- (qp_attr_mask & IB_QP_ALT_VID) ?
- qp->alt_vlan_id : 0xffff,
+ 0,
+ 0xffff,
path, &mqp->alt, port);
}
@@ -1420,7 +1420,8 @@ static void update_mcg_macs(struct mlx4_ib_dev *dev, struct mlx4_ib_qp *qp)
}
}
-static int handle_eth_ud_smac_index(struct mlx4_ib_dev *dev, struct mlx4_ib_qp *qp, u8 *smac,
+static int handle_eth_ud_smac_index(struct mlx4_ib_dev *dev,
+ struct mlx4_ib_qp *qp,
struct mlx4_qp_context *context)
{
u64 u64_mac;
@@ -1560,9 +1561,34 @@ static int __mlx4_ib_modify_qp(struct ib_qp *ibqp,
}
if (attr_mask & IB_QP_AV) {
+ u8 port_num = attr_mask & IB_QP_PORT ? attr->port_num : qp->port;
+ union ib_gid gid;
+ struct ib_gid_attr gid_attr = {.gid_type = IB_GID_TYPE_IB};
+ u16 vlan = 0xffff;
+ u8 smac[ETH_ALEN];
+ int status = 0;
+
+ if (rdma_port_get_link_layer(&dev->ib_dev, qp->port) ==
+ IB_LINK_LAYER_ETHERNET &&
+ attr->ah_attr.ah_flags & IB_AH_GRH) {
+ int index = attr->ah_attr.grh.sgid_index;
+
+ rcu_read_lock();
+ status = ib_get_cached_gid(ibqp->device, port_num,
+ index, &gid, &gid_attr);
+ if (!status && !memcmp(&gid, &zgid, sizeof(gid)))
+ status = -ENOENT;
+ if (!status) {
+ vlan = rdma_vlan_dev_vlan_id(gid_attr.ndev);
+ memcpy(smac, gid_attr.ndev->dev_addr, ETH_ALEN);
+ }
+ rcu_read_unlock();
+ }
+ if (status)
+ goto out;
+
if (mlx4_set_path(dev, attr, attr_mask, qp, &context->pri_path,
- attr_mask & IB_QP_PORT ?
- attr->port_num : qp->port))
+ port_num, vlan, smac))
goto out;
optpar |= (MLX4_QP_OPTPAR_PRIMARY_ADDR_PATH |
@@ -1699,7 +1725,7 @@ static int __mlx4_ib_modify_qp(struct ib_qp *ibqp,
if (qp->mlx4_ib_qp_type == MLX4_IB_QPT_UD ||
qp->mlx4_ib_qp_type == MLX4_IB_QPT_PROXY_GSI ||
qp->mlx4_ib_qp_type == MLX4_IB_QPT_TUN_GSI) {
- err = handle_eth_ud_smac_index(dev, qp, (u8 *)attr->smac, context);
+ err = handle_eth_ud_smac_index(dev, qp, context);
if (err) {
err = -EINVAL;
goto out;
@@ -2194,6 +2220,8 @@ static int build_mlx_header(struct mlx4_ib_sqp *sqp, struct ib_send_wr *wr,
be32_to_cpu(ah->av.ib.port_pd) >> 24,
ah->av.ib.gid_index, &sgid,
NULL);
+ if (!err && !memcmp(&sgid, &zgid, sizeof(sgid)))
+ err = -ENOENT;
if (err)
return err;
}
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma.h b/drivers/infiniband/hw/ocrdma/ocrdma.h
index c9780d9..16ee36e 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma.h
+++ b/drivers/infiniband/hw/ocrdma/ocrdma.h
@@ -36,6 +36,7 @@
#include <rdma/ib_verbs.h>
#include <rdma/ib_user_verbs.h>
#include <rdma/ib_addr.h>
+#include <rdma/ib_cache.h>
#include <be_roce.h>
#include "ocrdma_sli.h"
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_ah.c b/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
index d812904..7ecd230 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_ah.c
@@ -41,10 +41,9 @@
static inline int set_av_attr(struct ocrdma_dev *dev, struct ocrdma_ah *ah,
struct ib_ah_attr *attr, union ib_gid *sgid,
- int pdid, bool *isvlan)
+ int pdid, bool *isvlan, u16 vlan_tag)
{
int status = 0;
- u16 vlan_tag;
struct ocrdma_eth_vlan eth;
struct ocrdma_grh grh;
int eth_sz;
@@ -53,7 +52,6 @@ static inline int set_av_attr(struct ocrdma_dev *dev, struct ocrdma_ah *ah,
memset(&grh, 0, sizeof(grh));
/* VLAN */
- vlan_tag = attr->vlan_id;
if (!vlan_tag || (vlan_tag > 0xFFF))
vlan_tag = dev->pvid;
if (vlan_tag && (vlan_tag < 0x1000)) {
@@ -94,9 +92,11 @@ static inline int set_av_attr(struct ocrdma_dev *dev, struct ocrdma_ah *ah,
struct ib_ah *ocrdma_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr)
{
u32 *ahid_addr;
- bool isvlan = false;
int status;
struct ocrdma_ah *ah;
+ bool isvlan = false;
+ u16 vlan_tag = 0xffff;
+ struct ib_gid_attr sgid_attr;
struct ocrdma_pd *pd = get_ocrdma_pd(ibpd);
struct ocrdma_dev *dev = get_ocrdma_dev(ibpd->device);
union ib_gid sgid;
@@ -114,16 +114,22 @@ struct ib_ah *ocrdma_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr)
if (status)
goto av_err;
- status = ocrdma_query_gid(&dev->ibdev, 1, attr->grh.sgid_index, &sgid);
+ rcu_read_lock();
+ status = ib_get_cached_gid(&dev->ibdev, 1, attr->grh.sgid_index, &sgid,
+ &sgid_attr);
if (status) {
pr_err("%s(): Failed to query sgid, status = %d\n",
__func__, status);
goto av_conf_err;
}
+ if (sgid_attr.ndev && is_vlan_dev(sgid_attr.ndev))
+ vlan_tag = vlan_dev_vlan_id(sgid_attr.ndev);
+ rcu_read_unlock();
if (pd->uctx) {
status = rdma_addr_find_dmac_by_grh(&sgid, &attr->grh.dgid,
- attr->dmac, &attr->vlan_id);
+ attr->dmac, &vlan_tag,
+ sgid_attr.ndev->ifindex);
if (status) {
pr_err("%s(): Failed to resolve dmac from gid."
"status = %d\n", __func__, status);
@@ -131,7 +137,7 @@ struct ib_ah *ocrdma_create_ah(struct ib_pd *ibpd, struct ib_ah_attr *attr)
}
}
- status = set_av_attr(dev, ah, attr, &sgid, pd->id, &isvlan);
+ status = set_av_attr(dev, ah, attr, &sgid, pd->id, &isvlan, vlan_tag);
if (status)
goto av_conf_err;
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
index 0c9e959..057ed53 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
@@ -2428,7 +2428,8 @@ static int ocrdma_set_av_params(struct ocrdma_qp *qp,
int status;
struct ib_ah_attr *ah_attr = &attrs->ah_attr;
union ib_gid sgid, zgid;
- u32 vlan_id;
+ struct ib_gid_attr sgid_attr;
+ u32 vlan_id = 0xffff;
u8 mac_addr[6];
struct ocrdma_dev *dev = get_ocrdma_dev(qp->ibqp.device);
@@ -2446,10 +2447,15 @@ static int ocrdma_set_av_params(struct ocrdma_qp *qp,
cmd->flags |= OCRDMA_QP_PARA_FLOW_LBL_VALID;
memcpy(&cmd->params.dgid[0], &ah_attr->grh.dgid.raw[0],
sizeof(cmd->params.dgid));
- status = ocrdma_query_gid(&dev->ibdev, 1,
- ah_attr->grh.sgid_index, &sgid);
- if (status)
- return status;
+
+ rcu_read_lock();
+ status = ib_get_cached_gid(&dev->ibdev, 1, ah_attr->grh.sgid_index,
+ &sgid, &sgid_attr);
+ if (!status) {
+ vlan_id = rdma_vlan_dev_vlan_id(sgid_attr.ndev);
+ memcpy(mac_addr, sgid_attr.ndev->dev_addr, ETH_ALEN);
+ }
+ rcu_read_unlock();
memset(&zgid, 0, sizeof(zgid));
if (!memcmp(&sgid, &zgid, sizeof(zgid)))
@@ -2466,8 +2472,7 @@ static int ocrdma_set_av_params(struct ocrdma_qp *qp,
ocrdma_cpu_to_le32(&cmd->params.dgid[0], sizeof(cmd->params.dgid));
ocrdma_cpu_to_le32(&cmd->params.sgid[0], sizeof(cmd->params.sgid));
cmd->params.vlan_dmac_b4_to_b5 = mac_addr[4] | (mac_addr[5] << 8);
- if (attr_mask & IB_QP_VID) {
- vlan_id = attrs->vlan_id;
+ if (vlan_id < 0x1000) {
cmd->params.vlan_dmac_b4_to_b5 |=
vlan_id << OCRDMA_QP_PARAMS_VLAN_SHIFT;
cmd->flags |= OCRDMA_QP_PARA_VLAN_EN_VALID;
diff --git a/include/rdma/ib_addr.h b/include/rdma/ib_addr.h
index 3cf32d1..0dfaaa7 100644
--- a/include/rdma/ib_addr.h
+++ b/include/rdma/ib_addr.h
@@ -112,7 +112,7 @@ int rdma_addr_size(struct sockaddr *addr);
int rdma_addr_find_smac_by_sgid(union ib_gid *sgid, u8 *smac, u16 *vlan_id);
int rdma_addr_find_dmac_by_grh(union ib_gid *sgid, union ib_gid *dgid, u8 *smac,
- u16 *vlan_id);
+ u16 *vlan_id, int if_index);
static inline u16 ib_addr_get_pkey(struct rdma_dev_addr *dev_addr)
{
diff --git a/include/rdma/ib_sa.h b/include/rdma/ib_sa.h
index 6a1b994..eea01e6 100644
--- a/include/rdma/ib_sa.h
+++ b/include/rdma/ib_sa.h
@@ -154,9 +154,7 @@ struct ib_sa_path_rec {
u8 packet_life_time_selector;
u8 packet_life_time;
u8 preference;
- u8 smac[ETH_ALEN];
u8 dmac[ETH_ALEN];
- u16 vlan_id;
int ifindex;
struct net *net;
};
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 9e7505a..020f5c2 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -74,6 +74,8 @@ enum ib_gid_type {
IB_GID_TYPE_SIZE
};
+#define ROCE_V2_UDP_DPORT 4791
+
struct ib_gid_attr {
enum ib_gid_type gid_type;
struct net_device *ndev;
@@ -667,7 +669,6 @@ struct ib_ah_attr {
u8 ah_flags;
u8 port_num;
u8 dmac[ETH_ALEN];
- u16 vlan_id;
};
enum ib_wc_status {
@@ -925,10 +926,10 @@ enum ib_qp_attr_mask {
IB_QP_PATH_MIG_STATE = (1<<18),
IB_QP_CAP = (1<<19),
IB_QP_DEST_QPN = (1<<20),
- IB_QP_SMAC = (1<<21),
- IB_QP_ALT_SMAC = (1<<22),
- IB_QP_VID = (1<<23),
- IB_QP_ALT_VID = (1<<24),
+ IB_QP_RESERVED1 = (1<<21),
+ IB_QP_RESERVED2 = (1<<22),
+ IB_QP_RESERVED3 = (1<<23),
+ IB_QP_RESERVED4 = (1<<24),
};
enum ib_qp_state {
@@ -978,10 +979,6 @@ struct ib_qp_attr {
u8 rnr_retry;
u8 alt_port_num;
u8 alt_timeout;
- u8 smac[ETH_ALEN];
- u8 alt_smac[ETH_ALEN];
- u16 vlan_id;
- u16 alt_vlan_id;
};
enum ib_wr_opcode {
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 11/14] RDMA/ocrdma: Changes in driver to incorporate the moving of GID Table mgmt to IB/Core.
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (9 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 10/14] IB/core: Modify ib_verbs and cma in order to use roce_gid_table Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 12/14] net/mlx4: Postpone the registration of net_device Matan Barak
` (2 subsequent siblings)
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty,
Somnath Kotur, Devesh Sharma
From: Somnath Kotur <somnath.kotur-laKkSmNT4hbQT0dZR+AlfA@public.gmane.org>
1.Check and set port capability flags to indicate RoCEV2 support.
2.Change query_gid hook to return value from IB/Core GID Mgmt APIs.
3.Get rid of all the netdev notifier chain subscription code as well as
maintenance of SGID Table in memory.
4.Implement get_netdev hook in driver.
Signed-off-by: Somnath Kotur <somnath.kotur-laKkSmNT4hbQT0dZR+AlfA@public.gmane.org>
Signed-off-by: Devesh Sharma <devesh.sharma-laKkSmNT4hbQT0dZR+AlfA@public.gmane.org>
---
drivers/infiniband/hw/ocrdma/ocrdma.h | 10 ++
drivers/infiniband/hw/ocrdma/ocrdma_hw.c | 3 +
drivers/infiniband/hw/ocrdma/ocrdma_main.c | 233 +---------------------------
drivers/infiniband/hw/ocrdma/ocrdma_sli.h | 13 ++
drivers/infiniband/hw/ocrdma/ocrdma_verbs.c | 31 +++-
drivers/infiniband/hw/ocrdma/ocrdma_verbs.h | 4 +
6 files changed, 62 insertions(+), 232 deletions(-)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma.h b/drivers/infiniband/hw/ocrdma/ocrdma.h
index 16ee36e..97f971a 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma.h
+++ b/drivers/infiniband/hw/ocrdma/ocrdma.h
@@ -100,6 +100,7 @@ struct ocrdma_dev_attr {
u8 local_ca_ack_delay;
u8 ird;
u8 num_ird_pages;
+ u8 roce_flags;
};
struct ocrdma_dma_mem {
@@ -575,4 +576,13 @@ static inline u8 ocrdma_is_enabled_and_synced(u32 state)
(state & OCRDMA_STATE_FLAG_SYNC);
}
+static inline bool ocrdma_is_rocev2_supported(struct ocrdma_dev *dev)
+{
+ return (dev->attr.roce_flags & (OCRDMA_L3_TYPE_IPV4 <<
+ OCRDMA_ROUDP_FLAGS_SHIFT) ||
+ dev->attr.roce_flags & (OCRDMA_L3_TYPE_IPV6 <<
+ OCRDMA_ROUDP_FLAGS_SHIFT)) ?
+ true : false;
+}
+
#endif
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
index 057ed53..8e37968 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_hw.c
@@ -1112,6 +1112,9 @@ static void ocrdma_get_attr(struct ocrdma_dev *dev,
attr->local_ca_ack_delay = (rsp->max_pd_ca_ack_delay &
OCRDMA_MBX_QUERY_CFG_CA_ACK_DELAY_MASK) >>
OCRDMA_MBX_QUERY_CFG_CA_ACK_DELAY_SHIFT;
+ attr->roce_flags = (rsp->max_pd_ca_ack_delay &
+ OCRDMA_MBX_QUERY_CFG_L3_TYPE_MASK) >>
+ OCRDMA_MBX_QUERY_CFG_L3_TYPE_SHIFT;
attr->max_mw = rsp->max_mw;
attr->max_mr = rsp->max_mr;
attr->max_mr_size = ((u64)rsp->max_mr_size_hi << 32) |
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_main.c b/drivers/infiniband/hw/ocrdma/ocrdma_main.c
index 7a2b59a..a81492f 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_main.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_main.c
@@ -51,8 +51,6 @@ static LIST_HEAD(ocrdma_dev_list);
static DEFINE_SPINLOCK(ocrdma_devlist_lock);
static DEFINE_IDR(ocrdma_dev_id);
-static union ib_gid ocrdma_zero_sgid;
-
void ocrdma_get_guid(struct ocrdma_dev *dev, u8 *guid)
{
u8 mac_addr[6];
@@ -67,135 +65,6 @@ void ocrdma_get_guid(struct ocrdma_dev *dev, u8 *guid)
guid[6] = mac_addr[4];
guid[7] = mac_addr[5];
}
-
-static bool ocrdma_add_sgid(struct ocrdma_dev *dev, union ib_gid *new_sgid)
-{
- int i;
- unsigned long flags;
-
- memset(&ocrdma_zero_sgid, 0, sizeof(union ib_gid));
-
-
- spin_lock_irqsave(&dev->sgid_lock, flags);
- for (i = 0; i < OCRDMA_MAX_SGID; i++) {
- if (!memcmp(&dev->sgid_tbl[i], &ocrdma_zero_sgid,
- sizeof(union ib_gid))) {
- /* found free entry */
- memcpy(&dev->sgid_tbl[i], new_sgid,
- sizeof(union ib_gid));
- spin_unlock_irqrestore(&dev->sgid_lock, flags);
- return true;
- } else if (!memcmp(&dev->sgid_tbl[i], new_sgid,
- sizeof(union ib_gid))) {
- /* entry already present, no addition is required. */
- spin_unlock_irqrestore(&dev->sgid_lock, flags);
- return false;
- }
- }
- spin_unlock_irqrestore(&dev->sgid_lock, flags);
- return false;
-}
-
-static bool ocrdma_del_sgid(struct ocrdma_dev *dev, union ib_gid *sgid)
-{
- int found = false;
- int i;
- unsigned long flags;
-
-
- spin_lock_irqsave(&dev->sgid_lock, flags);
- /* first is default sgid, which cannot be deleted. */
- for (i = 1; i < OCRDMA_MAX_SGID; i++) {
- if (!memcmp(&dev->sgid_tbl[i], sgid, sizeof(union ib_gid))) {
- /* found matching entry */
- memset(&dev->sgid_tbl[i], 0, sizeof(union ib_gid));
- found = true;
- break;
- }
- }
- spin_unlock_irqrestore(&dev->sgid_lock, flags);
- return found;
-}
-
-static int ocrdma_addr_event(unsigned long event, struct net_device *netdev,
- union ib_gid *gid)
-{
- struct ib_event gid_event;
- struct ocrdma_dev *dev;
- bool found = false;
- bool updated = false;
- bool is_vlan = false;
-
- is_vlan = netdev->priv_flags & IFF_802_1Q_VLAN;
- if (is_vlan)
- netdev = rdma_vlan_dev_real_dev(netdev);
-
- rcu_read_lock();
- list_for_each_entry_rcu(dev, &ocrdma_dev_list, entry) {
- if (dev->nic_info.netdev == netdev) {
- found = true;
- break;
- }
- }
- rcu_read_unlock();
-
- if (!found)
- return NOTIFY_DONE;
-
- mutex_lock(&dev->dev_lock);
- switch (event) {
- case NETDEV_UP:
- updated = ocrdma_add_sgid(dev, gid);
- break;
- case NETDEV_DOWN:
- updated = ocrdma_del_sgid(dev, gid);
- break;
- default:
- break;
- }
- if (updated) {
- /* GID table updated, notify the consumers about it */
- gid_event.device = &dev->ibdev;
- gid_event.element.port_num = 1;
- gid_event.event = IB_EVENT_GID_CHANGE;
- ib_dispatch_event(&gid_event);
- }
- mutex_unlock(&dev->dev_lock);
- return NOTIFY_OK;
-}
-
-static int ocrdma_inetaddr_event(struct notifier_block *notifier,
- unsigned long event, void *ptr)
-{
- struct in_ifaddr *ifa = ptr;
- union ib_gid gid;
- struct net_device *netdev = ifa->ifa_dev->dev;
-
- ipv6_addr_set_v4mapped(ifa->ifa_address, (struct in6_addr *)&gid);
- return ocrdma_addr_event(event, netdev, &gid);
-}
-
-static struct notifier_block ocrdma_inetaddr_notifier = {
- .notifier_call = ocrdma_inetaddr_event
-};
-
-#if IS_ENABLED(CONFIG_IPV6)
-
-static int ocrdma_inet6addr_event(struct notifier_block *notifier,
- unsigned long event, void *ptr)
-{
- struct inet6_ifaddr *ifa = (struct inet6_ifaddr *)ptr;
- union ib_gid *gid = (union ib_gid *)&ifa->addr;
- struct net_device *netdev = ifa->idev->dev;
- return ocrdma_addr_event(event, netdev, gid);
-}
-
-static struct notifier_block ocrdma_inet6addr_notifier = {
- .notifier_call = ocrdma_inet6addr_event
-};
-
-#endif /* IPV6 and VLAN */
-
static enum rdma_link_layer ocrdma_link_layer(struct ib_device *device,
u8 port_num)
{
@@ -246,6 +115,8 @@ static int ocrdma_register_device(struct ocrdma_dev *dev)
dev->ibdev.query_port = ocrdma_query_port;
dev->ibdev.modify_port = ocrdma_modify_port;
dev->ibdev.query_gid = ocrdma_query_gid;
+ dev->ibdev.get_netdev = ocrdma_get_netdev;
+ dev->ibdev.modify_gid = ocrdma_modify_gid;
dev->ibdev.get_link_layer = ocrdma_link_layer;
dev->ibdev.alloc_pd = ocrdma_alloc_pd;
dev->ibdev.dealloc_pd = ocrdma_dealloc_pd;
@@ -307,12 +178,6 @@ static int ocrdma_register_device(struct ocrdma_dev *dev)
static int ocrdma_alloc_resources(struct ocrdma_dev *dev)
{
mutex_init(&dev->dev_lock);
- dev->sgid_tbl = kzalloc(sizeof(union ib_gid) *
- OCRDMA_MAX_SGID, GFP_KERNEL);
- if (!dev->sgid_tbl)
- goto alloc_err;
- spin_lock_init(&dev->sgid_lock);
-
dev->cq_tbl = kzalloc(sizeof(struct ocrdma_cq *) *
OCRDMA_MAX_CQ, GFP_KERNEL);
if (!dev->cq_tbl)
@@ -344,7 +209,6 @@ static void ocrdma_free_resources(struct ocrdma_dev *dev)
kfree(dev->stag_arr);
kfree(dev->qp_tbl);
kfree(dev->cq_tbl);
- kfree(dev->sgid_tbl);
}
/* OCRDMA sysfs interface */
@@ -390,68 +254,6 @@ static void ocrdma_remove_sysfiles(struct ocrdma_dev *dev)
device_remove_file(&dev->ibdev.dev, ocrdma_attributes[i]);
}
-static void ocrdma_add_default_sgid(struct ocrdma_dev *dev)
-{
- /* GID Index 0 - Invariant manufacturer-assigned EUI-64 */
- union ib_gid *sgid = &dev->sgid_tbl[0];
-
- sgid->global.subnet_prefix = cpu_to_be64(0xfe80000000000000LL);
- ocrdma_get_guid(dev, &sgid->raw[8]);
-}
-
-static void ocrdma_init_ipv4_gids(struct ocrdma_dev *dev,
- struct net_device *net)
-{
- struct in_device *in_dev;
- union ib_gid gid;
- in_dev = in_dev_get(net);
- if (in_dev) {
- for_ifa(in_dev) {
- ipv6_addr_set_v4mapped(ifa->ifa_address,
- (struct in6_addr *)&gid);
- ocrdma_add_sgid(dev, &gid);
- }
- endfor_ifa(in_dev);
- in_dev_put(in_dev);
- }
-}
-
-static void ocrdma_init_ipv6_gids(struct ocrdma_dev *dev,
- struct net_device *net)
-{
-#if IS_ENABLED(CONFIG_IPV6)
- struct inet6_dev *in6_dev;
- union ib_gid *pgid;
- struct inet6_ifaddr *ifp;
- in6_dev = in6_dev_get(net);
- if (in6_dev) {
- read_lock_bh(&in6_dev->lock);
- list_for_each_entry(ifp, &in6_dev->addr_list, if_list) {
- pgid = (union ib_gid *)&ifp->addr;
- ocrdma_add_sgid(dev, pgid);
- }
- read_unlock_bh(&in6_dev->lock);
- in6_dev_put(in6_dev);
- }
-#endif
-}
-
-static void ocrdma_init_gid_table(struct ocrdma_dev *dev)
-{
- struct net_device *net_dev;
-
- for_each_netdev(&init_net, net_dev) {
- struct net_device *real_dev = rdma_vlan_dev_real_dev(net_dev) ?
- rdma_vlan_dev_real_dev(net_dev) : net_dev;
-
- if (real_dev == dev->nic_info.netdev) {
- ocrdma_add_default_sgid(dev);
- ocrdma_init_ipv4_gids(dev, net_dev);
- ocrdma_init_ipv6_gids(dev, net_dev);
- }
- }
-}
-
static struct ocrdma_dev *ocrdma_add(struct be_dev_info *dev_info)
{
int status = 0, i;
@@ -480,7 +282,6 @@ static struct ocrdma_dev *ocrdma_add(struct be_dev_info *dev_info)
goto alloc_err;
ocrdma_init_service_level(dev);
- ocrdma_init_gid_table(dev);
status = ocrdma_register_device(dev);
if (status)
goto alloc_err;
@@ -627,34 +428,12 @@ static struct ocrdma_driver ocrdma_drv = {
.be_abi_version = OCRDMA_BE_ROCE_ABI_VERSION,
};
-static void ocrdma_unregister_inet6addr_notifier(void)
-{
-#if IS_ENABLED(CONFIG_IPV6)
- unregister_inet6addr_notifier(&ocrdma_inet6addr_notifier);
-#endif
-}
-
-static void ocrdma_unregister_inetaddr_notifier(void)
-{
- unregister_inetaddr_notifier(&ocrdma_inetaddr_notifier);
-}
-
static int __init ocrdma_init_module(void)
{
int status;
ocrdma_init_debugfs();
- status = register_inetaddr_notifier(&ocrdma_inetaddr_notifier);
- if (status)
- return status;
-
-#if IS_ENABLED(CONFIG_IPV6)
- status = register_inet6addr_notifier(&ocrdma_inet6addr_notifier);
- if (status)
- goto err_notifier6;
-#endif
-
status = be_roce_register_driver(&ocrdma_drv);
if (status)
goto err_be_reg;
@@ -662,19 +441,13 @@ static int __init ocrdma_init_module(void)
return 0;
err_be_reg:
-#if IS_ENABLED(CONFIG_IPV6)
- ocrdma_unregister_inet6addr_notifier();
-err_notifier6:
-#endif
- ocrdma_unregister_inetaddr_notifier();
+
return status;
}
static void __exit ocrdma_exit_module(void)
{
be_roce_unregister_driver(&ocrdma_drv);
- ocrdma_unregister_inet6addr_notifier();
- ocrdma_unregister_inetaddr_notifier();
ocrdma_rem_debugfs();
}
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_sli.h b/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
index 243c87c..6b74eb9 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_sli.h
@@ -125,6 +125,14 @@ enum {
OCRDMA_DB_RQ_SHIFT = 24
};
+enum {
+ OCRDMA_L3_TYPE_IB_GRH = 0x00,
+ OCRDMA_L3_TYPE_IPV4 = 0x01,
+ OCRDMA_L3_TYPE_IPV6 = 0x02
+};
+
+#define OCRDMA_ROUDP_FLAGS_SHIFT 0x03
+
#define OCRDMA_DB_CQ_RING_ID_MASK 0x3FF /* bits 0 - 9 */
#define OCRDMA_DB_CQ_RING_ID_EXT_MASK 0x0C00 /* bits 10-11 of qid at 12-11 */
/* qid #2 msbits at 12-11 */
@@ -488,6 +496,9 @@ enum {
OCRDMA_MBX_QUERY_CFG_CA_ACK_DELAY_SHIFT = 8,
OCRDMA_MBX_QUERY_CFG_CA_ACK_DELAY_MASK = 0xFF <<
OCRDMA_MBX_QUERY_CFG_CA_ACK_DELAY_SHIFT,
+ OCRDMA_MBX_QUERY_CFG_L3_TYPE_SHIFT = 0,
+ OCRDMA_MBX_QUERY_CFG_L3_TYPE_MASK = 0xFF <<
+ OCRDMA_MBX_QUERY_CFG_L3_TYPE_SHIFT,
OCRDMA_MBX_QUERY_CFG_MAX_SEND_SGE_SHIFT = 0,
OCRDMA_MBX_QUERY_CFG_MAX_SEND_SGE_MASK = 0xFFFF,
@@ -1049,6 +1060,8 @@ enum {
OCRDMA_QP_PARAMS_STATE_MASK = BIT(5) | BIT(6) | BIT(7),
OCRDMA_QP_PARAMS_FLAGS_SQD_ASYNC = BIT(8),
OCRDMA_QP_PARAMS_FLAGS_INB_ATEN = BIT(9),
+ OCRDMA_QP_PARAMS_FLAGS_L3_TYPE_SHIFT = 11,
+ OCRDMA_QP_PARAMS_FLAGS_L3_TYPE_MASK = BIT(11) | BIT(12) | BIT(13),
OCRDMA_QP_PARAMS_MAX_SGE_RECV_SHIFT = 16,
OCRDMA_QP_PARAMS_MAX_SGE_RECV_MASK = 0xFFFF <<
OCRDMA_QP_PARAMS_MAX_SGE_RECV_SHIFT,
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
index 8771755..ac0411d 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.c
@@ -31,6 +31,7 @@
#include <rdma/iw_cm.h>
#include <rdma/ib_umem.h>
#include <rdma/ib_addr.h>
+#include <rdma/ib_cache.h>
#include "ocrdma.h"
#include "ocrdma_hw.h"
@@ -49,6 +50,7 @@ int ocrdma_query_pkey(struct ib_device *ibdev, u8 port, u16 index, u16 *pkey)
int ocrdma_query_gid(struct ib_device *ibdev, u8 port,
int index, union ib_gid *sgid)
{
+ int ret;
struct ocrdma_dev *dev;
dev = get_ocrdma_dev(ibdev);
@@ -56,7 +58,22 @@ int ocrdma_query_gid(struct ib_device *ibdev, u8 port,
if (index >= OCRDMA_MAX_SGID)
return -EINVAL;
- memcpy(sgid, &dev->sgid_tbl[index], sizeof(*sgid));
+ ret = ib_get_cached_gid(ibdev, port, index, sgid, NULL);
+ if (ret == -EAGAIN) {
+ memcpy(sgid, &zgid, sizeof(*sgid));
+ return 0;
+ }
+
+ return ret;
+}
+
+int ocrdma_modify_gid(struct ib_device *ibdev, u8 port_num, unsigned int index,
+ const union ib_gid *gid, const struct ib_gid_attr *attr,
+ void **context)
+{
+ struct ocrdma_dev *dev;
+
+ dev = get_ocrdma_dev(ibdev);
return 0;
}
@@ -106,6 +123,15 @@ int ocrdma_query_device(struct ib_device *ibdev, struct ib_device_attr *attr)
return 0;
}
+struct net_device *ocrdma_get_netdev(struct ib_device *ibdev, u8 port_num)
+{
+ struct ocrdma_dev *dev = get_ocrdma_dev(ibdev);
+
+ if (dev)
+ return dev->nic_info.netdev;
+
+ return NULL;
+}
static inline void get_link_speed_and_width(struct ocrdma_dev *dev,
u8 *ib_speed, u8 *ib_width)
{
@@ -175,7 +201,8 @@ int ocrdma_query_port(struct ib_device *ibdev,
props->port_cap_flags =
IB_PORT_CM_SUP |
IB_PORT_REINIT_SUP |
- IB_PORT_DEVICE_MGMT_SUP | IB_PORT_VENDOR_CLASS_SUP | IB_PORT_IP_BASED_GIDS;
+ IB_PORT_DEVICE_MGMT_SUP | IB_PORT_VENDOR_CLASS_SUP |
+ IB_PORT_IP_BASED_GIDS | IB_PORT_ROCE;
props->gid_tbl_len = OCRDMA_MAX_SGID;
props->pkey_tbl_len = 1;
props->bad_pkey_cntr = 0;
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.h b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.h
index b8f7853..8204182 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_verbs.h
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_verbs.h
@@ -44,6 +44,10 @@ int ocrdma_modify_port(struct ib_device *, u8 port, int mask,
void ocrdma_get_guid(struct ocrdma_dev *, u8 *guid);
int ocrdma_query_gid(struct ib_device *, u8 port,
int index, union ib_gid *gid);
+struct net_device *ocrdma_get_netdev(struct ib_device *device, u8 port_num);
+int ocrdma_modify_gid(struct ib_device *ibdev, u8 port_num, unsigned int index,
+ const union ib_gid *gid, const struct ib_gid_attr *attr,
+ void **context);
int ocrdma_query_pkey(struct ib_device *, u8 port, u16 index, u16 *pkey);
struct ib_ucontext *ocrdma_alloc_ucontext(struct ib_device *,
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 12/14] net/mlx4: Postpone the registration of net_device
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (10 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 11/14] RDMA/ocrdma: Changes in driver to incorporate the moving of GID Table mgmt to IB/Core Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 13/14] IB/mlx4: Implement ib_device callbacks Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 14/14] IB/mlx4: Replace mechanism for RoCE GID management Matan Barak
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
From: Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
The mlx4 network driver was registered in the context of the 'add'
function of the core driver (called when HW should be registered).
This makes the netdev event NETDEV_REGISTER to be sent in a context
where the answer to get_protocol_dev() callback returns NULL. This may
be confusing to listeners of netdev events.
This patch is a preparation to the patch that implements the
get_netdev() callback in the IB/mlx4 driver.
Signed-off-by: Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/net/ethernet/mellanox/mlx4/en_main.c | 36 ++++++++++++++++------------
drivers/net/ethernet/mellanox/mlx4/intf.c | 3 +++
include/linux/mlx4/driver.h | 1 +
3 files changed, 25 insertions(+), 15 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c
index 913b716..a946e4b 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c
@@ -224,6 +224,26 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr)
kfree(mdev);
}
+static void mlx4_en_activate(struct mlx4_dev *dev, void *ctx)
+{
+ int i;
+ struct mlx4_en_dev *mdev = ctx;
+
+ /* Create a netdev for each port */
+ mlx4_foreach_port(i, dev, MLX4_PORT_TYPE_ETH) {
+ mlx4_info(mdev, "Activating port:%d\n", i);
+ if (mlx4_en_init_netdev(mdev, i, &mdev->profile.prof[i]))
+ mdev->pndev[i] = NULL;
+ }
+
+ /* register notifier */
+ mdev->nb.notifier_call = mlx4_en_netdev_event;
+ if (register_netdevice_notifier(&mdev->nb)) {
+ mdev->nb.notifier_call = NULL;
+ mlx4_err(mdev, "Failed to create notifier\n");
+ }
+}
+
static void *mlx4_en_add(struct mlx4_dev *dev)
{
struct mlx4_en_dev *mdev;
@@ -297,21 +317,6 @@ static void *mlx4_en_add(struct mlx4_dev *dev)
mutex_init(&mdev->state_lock);
mdev->device_up = true;
- /* Setup ports */
-
- /* Create a netdev for each port */
- mlx4_foreach_port(i, dev, MLX4_PORT_TYPE_ETH) {
- mlx4_info(mdev, "Activating port:%d\n", i);
- if (mlx4_en_init_netdev(mdev, i, &mdev->profile.prof[i]))
- mdev->pndev[i] = NULL;
- }
- /* register notifier */
- mdev->nb.notifier_call = mlx4_en_netdev_event;
- if (register_netdevice_notifier(&mdev->nb)) {
- mdev->nb.notifier_call = NULL;
- mlx4_err(mdev, "Failed to create notifier\n");
- }
-
return mdev;
err_mr:
@@ -335,6 +340,7 @@ static struct mlx4_interface mlx4_en_interface = {
.event = mlx4_en_event,
.get_dev = mlx4_en_get_netdev,
.protocol = MLX4_PROT_ETH,
+ .activate = mlx4_en_activate,
};
static void mlx4_en_verify_params(void)
diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c
index 6fce587..09e94c6 100644
--- a/drivers/net/ethernet/mellanox/mlx4/intf.c
+++ b/drivers/net/ethernet/mellanox/mlx4/intf.c
@@ -63,8 +63,11 @@ static void mlx4_add_device(struct mlx4_interface *intf, struct mlx4_priv *priv)
spin_lock_irq(&priv->ctx_lock);
list_add_tail(&dev_ctx->list, &priv->ctx_list);
spin_unlock_irq(&priv->ctx_lock);
+ if (intf->activate)
+ intf->activate(&priv->dev, dev_ctx->context);
} else
kfree(dev_ctx);
+
}
static void mlx4_remove_device(struct mlx4_interface *intf, struct mlx4_priv *priv)
diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h
index 9553a73..5a06d96 100644
--- a/include/linux/mlx4/driver.h
+++ b/include/linux/mlx4/driver.h
@@ -59,6 +59,7 @@ struct mlx4_interface {
void (*event) (struct mlx4_dev *dev, void *context,
enum mlx4_dev_event event, unsigned long param);
void * (*get_dev)(struct mlx4_dev *dev, void *context, u8 port);
+ void (*activate)(struct mlx4_dev *dev, void *context);
struct list_head list;
enum mlx4_protocol protocol;
int flags;
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 13/14] IB/mlx4: Implement ib_device callbacks
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (11 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 12/14] net/mlx4: Postpone the registration of net_device Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 14/14] IB/mlx4: Replace mechanism for RoCE GID management Matan Barak
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
From: Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
get_netdev: get the net_device on the physical port of the IB transport port. In
port aggregation mode it is required to return the netdev of the active port.
modify_gid: note for a change in the RoCE gid cache. Handle this by writing to
the harsware GID table. It is possible that indexes in cahce and hardware tables
won't match so a translation is required when modifying a QP or creating an
address handle.
Signed-off-by: Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/hw/mlx4/main.c | 199 +++++++++++++++++++++++++++++++++++
drivers/infiniband/hw/mlx4/mlx4_ib.h | 18 ++++
include/linux/mlx4/device.h | 3 +-
3 files changed, 219 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index abd0cf74..08ec498 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -45,6 +45,9 @@
#include <rdma/ib_smi.h>
#include <rdma/ib_user_verbs.h>
#include <rdma/ib_addr.h>
+#include <rdma/ib_cache.h>
+
+#include <net/bonding.h>
#include <linux/mlx4/driver.h>
#include <linux/mlx4/cmd.h>
@@ -129,6 +132,202 @@ static int num_ib_ports(struct mlx4_dev *dev)
return ib_ports;
}
+static struct net_device *mlx4_ib_get_netdev(struct ib_device *device, u8 port_num)
+{
+ struct mlx4_ib_dev *ibdev = to_mdev(device);
+
+ if (mlx4_is_bonded(ibdev->dev)) {
+ struct net_device *dev;
+ struct net_device *upper = NULL;
+
+ rcu_read_lock();
+
+ dev = mlx4_get_protocol_dev(ibdev->dev, MLX4_PROT_ETH, port_num);
+ if (dev)
+ upper = netdev_master_upper_dev_get_rcu(dev);
+ else
+ goto unlock;
+ if (upper)
+ dev = bond_option_active_slave_get_rcu(netdev_priv(upper));
+unlock:
+ rcu_read_unlock();
+
+ return dev;
+ }
+
+ return mlx4_get_protocol_dev(ibdev->dev, MLX4_PROT_ETH, port_num);
+}
+
+static int mlx4_ib_update_gids(struct gid_entry *gids,
+ struct mlx4_ib_dev *ibdev,
+ u8 port_num)
+{
+ struct mlx4_cmd_mailbox *mailbox;
+ int err;
+ struct mlx4_dev *dev = ibdev->dev;
+ int i;
+ union ib_gid *gid_tbl;
+
+ mailbox = mlx4_alloc_cmd_mailbox(dev);
+ if (IS_ERR(mailbox))
+ return -ENOMEM;
+
+ gid_tbl = mailbox->buf;
+
+ for (i = 0; i < MLX4_MAX_PORT_GIDS; ++i)
+ memcpy(&gid_tbl[i], &gids[i].gid, sizeof(union ib_gid));
+
+ err = mlx4_cmd(dev, mailbox->dma,
+ MLX4_SET_PORT_GID_TABLE << 8 | port_num,
+ 1, MLX4_CMD_SET_PORT, MLX4_CMD_TIME_CLASS_B,
+ MLX4_CMD_WRAPPED);
+ if (mlx4_is_bonded(dev))
+ err += mlx4_cmd(dev, mailbox->dma,
+ MLX4_SET_PORT_GID_TABLE << 8 | 2,
+ 1, MLX4_CMD_SET_PORT, MLX4_CMD_TIME_CLASS_B,
+ MLX4_CMD_WRAPPED);
+
+ mlx4_free_cmd_mailbox(dev, mailbox);
+ return err;
+}
+
+static int mlx4_ib_modify_gid(struct ib_device *device,
+ u8 port_num, unsigned int index,
+ const union ib_gid *gid,
+ const struct ib_gid_attr *attr,
+ void **context)
+{
+ struct mlx4_ib_dev *ibdev = to_mdev(device);
+ struct mlx4_ib_iboe *iboe = &ibdev->iboe;
+ struct mlx4_port_gid_table *port_gid_table;
+ int free = -1, found = -1;
+ int ret = 0;
+ int clear = !memcmp(&zgid, gid, sizeof(*gid));
+ int hw_update = 0;
+ int i;
+ struct gid_entry *gids = NULL;
+
+ if (ib_cache_use_roce_gid_table(device, port_num))
+ return -EINVAL;
+
+ if (port_num > MLX4_MAX_PORTS)
+ return -EINVAL;
+
+ if (!context)
+ return -EINVAL;
+
+ spin_lock_bh(&iboe->lock);
+ port_gid_table = &iboe->gids[port_num - 1];
+
+ if (clear) {
+ struct gid_cache_context *ctx = *context;
+
+ if (ctx) {
+ ctx->refcount--;
+ if (!ctx->refcount) {
+ unsigned int real_index = ctx->real_index;
+
+ memcpy(&port_gid_table->gids[real_index].gid, &zgid, sizeof(*gid));
+ kfree(port_gid_table->gids[real_index].ctx);
+ port_gid_table->gids[real_index].ctx = NULL;
+ hw_update = 1;
+ }
+ }
+ } else {
+ for (i = 0; i < MLX4_MAX_PORT_GIDS; ++i) {
+ if (!memcmp(&port_gid_table->gids[i].gid, gid, sizeof(*gid))) {
+ found = (port_gid_table->gids[i].gid_type == attr->gid_type) ? i : -1;
+ if (found >= 0)
+ break;
+ }
+ if (free < 0 && !memcmp(&port_gid_table->gids[i].gid, &zgid, sizeof(*gid)))
+ free = i; /* HW has space */
+ }
+
+ if (found < 0) {
+ if (free < 0) {
+ ret = -ENOSPC;
+ } else {
+ port_gid_table->gids[free].ctx = kmalloc(sizeof(*port_gid_table->gids[free].ctx), GFP_ATOMIC);
+ if (!port_gid_table->gids[free].ctx) {
+ ret = -ENOMEM;
+ } else {
+ *context = port_gid_table->gids[free].ctx;
+ memcpy(&port_gid_table->gids[free].gid, gid, sizeof(*gid));
+ port_gid_table->gids[free].gid_type = attr->gid_type;
+ port_gid_table->gids[free].ctx->real_index = free;
+ port_gid_table->gids[free].ctx->refcount = 1;
+ hw_update = 1;
+ }
+ }
+ } else {
+ struct gid_cache_context *ctx = port_gid_table->gids[found].ctx;
+ *context = ctx;
+ ctx->refcount++;
+ }
+ }
+ if (!ret && hw_update) {
+ gids = kmalloc(sizeof(*gids) * MLX4_MAX_PORT_GIDS, GFP_ATOMIC);
+ if (!gids) {
+ ret = -ENOMEM;
+ } else {
+ for (i = 0; i < MLX4_MAX_PORT_GIDS; i++) {
+ memcpy(&gids[i].gid, &port_gid_table->gids[i].gid, sizeof(union ib_gid));
+ gids[i].gid_type = port_gid_table->gids[i].gid_type;
+ }
+ }
+ }
+ spin_unlock_bh(&iboe->lock);
+
+ if (!ret && hw_update) {
+ ret = mlx4_ib_update_gids(gids, ibdev, port_num);
+ kfree(gids);
+ }
+
+ return ret;
+}
+
+int mlx4_ib_gid_index_to_real_index(struct mlx4_ib_dev *ibdev,
+ u8 port_num, int index)
+{
+ struct mlx4_ib_iboe *iboe = &ibdev->iboe;
+ struct gid_cache_context *ctx = NULL;
+ union ib_gid gid;
+ struct mlx4_port_gid_table *port_gid_table;
+ int real_index = -EINVAL;
+ int i;
+ int ret;
+ struct ib_gid_attr attr;
+ unsigned long flags;
+
+ if (port_num > MLX4_MAX_PORTS)
+ return -EINVAL;
+
+ if (ib_cache_use_roce_gid_table(&ibdev->ib_dev, port_num))
+ return index;
+
+ ret = ib_get_cached_gid(&ibdev->ib_dev, port_num, index, &gid, &attr);
+ if (ret)
+ return ret;
+
+ if (!memcmp(&gid, &zgid, sizeof(gid)))
+ return -EINVAL;
+
+ spin_lock_irqsave(&iboe->lock, flags);
+ port_gid_table = &iboe->gids[port_num - 1];
+
+ for (i = 0; i < MLX4_MAX_PORT_GIDS; ++i)
+ if (!memcmp(&port_gid_table->gids[i].gid, &gid, sizeof(gid)) &&
+ (attr.gid_type == port_gid_table->gids[i].gid_type)) {
+ ctx = port_gid_table->gids[i].ctx;
+ break;
+ }
+ if (ctx)
+ real_index = ctx->real_index;
+ spin_unlock_irqrestore(&iboe->lock, flags);
+ return real_index;
+}
+
static int mlx4_ib_query_device(struct ib_device *ibdev,
struct ib_device_attr *props)
{
diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
index 3eee5da..0a32552 100644
--- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
+++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
@@ -456,6 +456,21 @@ struct mlx4_ib_sriov {
struct idr pv_id_table;
};
+struct gid_cache_context {
+ int real_index;
+ int refcount;
+};
+
+struct gid_entry {
+ union ib_gid gid;
+ enum ib_gid_type gid_type;
+ struct gid_cache_context *ctx;
+};
+
+struct mlx4_port_gid_table {
+ struct gid_entry gids[MLX4_MAX_PORT_GIDS];
+};
+
struct mlx4_ib_iboe {
spinlock_t lock;
struct net_device *netdevs[MLX4_MAX_PORTS];
@@ -465,6 +480,7 @@ struct mlx4_ib_iboe {
struct notifier_block nb_inet;
struct notifier_block nb_inet6;
union ib_gid gid_table[MLX4_MAX_PORTS][128];
+ struct mlx4_port_gid_table gids[MLX4_MAX_PORTS];
};
struct pkey_mgt {
@@ -815,5 +831,7 @@ int mlx4_ib_rereg_user_mr(struct ib_mr *mr, int flags,
u64 start, u64 length, u64 virt_addr,
int mr_access_flags, struct ib_pd *pd,
struct ib_udata *udata);
+int mlx4_ib_gid_index_to_real_index(struct mlx4_ib_dev *ibdev,
+ u8 port_num, int index);
#endif /* MLX4_IB_H */
diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
index 83e80ab..d439949 100644
--- a/include/linux/mlx4/device.h
+++ b/include/linux/mlx4/device.h
@@ -78,7 +78,8 @@ enum {
enum {
MLX4_MAX_PORTS = 2,
- MLX4_MAX_PORT_PKEYS = 128
+ MLX4_MAX_PORT_PKEYS = 128,
+ MLX4_MAX_PORT_GIDS = 128
};
/* base qkey for use in sriov tunnel-qp/proxy-qp communication.
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* [PATCH v4 for-next 14/14] IB/mlx4: Replace mechanism for RoCE GID management
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
` (12 preceding siblings ...)
2015-05-19 14:27 ` [PATCH v4 for-next 13/14] IB/mlx4: Implement ib_device callbacks Matan Barak
@ 2015-05-19 14:27 ` Matan Barak
13 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-19 14:27 UTC (permalink / raw)
To: Doug Ledford
Cc: Matan Barak, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Jason Gunthorpe, Sean Hefty
From: Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Manage RoCE gid table with logic in IB/core, which is common to all
vendors, and remove the mechanism from the mlx4 IB driver.
Since management of the GID cache may lead to index mismatch with the
hardware GID table, a translation between indexes is required when
modifying a QP or creating an address handle.
Signed-off-by: Moni Shoua <monis-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
drivers/infiniband/hw/mlx4/ah.c | 2 +-
drivers/infiniband/hw/mlx4/main.c | 510 ++---------------------------------
drivers/infiniband/hw/mlx4/mlx4_ib.h | 4 -
drivers/infiniband/hw/mlx4/qp.c | 10 +-
4 files changed, 30 insertions(+), 496 deletions(-)
diff --git a/drivers/infiniband/hw/mlx4/ah.c b/drivers/infiniband/hw/mlx4/ah.c
index aaeeb60..82f5387 100644
--- a/drivers/infiniband/hw/mlx4/ah.c
+++ b/drivers/infiniband/hw/mlx4/ah.c
@@ -100,7 +100,7 @@ static struct ib_ah *create_iboe_ah(struct ib_pd *pd, struct ib_ah_attr *ah_attr
if (vlan_tag < 0x1000)
vlan_tag |= (ah_attr->sl & 7) << 13;
ah->av.eth.port_pd = cpu_to_be32(to_mpd(pd)->pdn | (ah_attr->port_num << 24));
- ah->av.eth.gid_index = ah_attr->grh.sgid_index;
+ ah->av.eth.gid_index = mlx4_ib_gid_index_to_real_index(ibdev, ah_attr->port_num, ah_attr->grh.sgid_index);
ah->av.eth.vlan = cpu_to_be16(vlan_tag);
if (ah_attr->static_rate) {
ah->av.eth.stat_rate = ah_attr->static_rate + MLX4_STAT_RATE_OFFSET;
diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 08ec498..4c47cc9 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -77,13 +77,6 @@ static const char mlx4_ib_version[] =
DRV_NAME ": Mellanox ConnectX InfiniBand driver v"
DRV_VERSION " (" DRV_RELDATE ")\n";
-struct update_gid_work {
- struct work_struct work;
- union ib_gid gids[128];
- struct mlx4_ib_dev *dev;
- int port;
-};
-
static void do_slave_init(struct mlx4_ib_dev *ibdev, int slave, int do_init);
static struct workqueue_struct *wq;
@@ -563,7 +556,8 @@ static int eth_link_query_port(struct ib_device *ibdev, u8 port,
props->active_width = (((u8 *)mailbox->buf)[5] == 0x40) ?
IB_WIDTH_4X : IB_WIDTH_1X;
props->active_speed = IB_SPEED_QDR;
- props->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_IP_BASED_GIDS;
+ props->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_IP_BASED_GIDS |
+ IB_PORT_ROCE;
props->gid_tbl_len = mdev->dev->caps.gid_table_len[port];
props->max_msg_sz = mdev->dev->caps.max_msg_sz;
props->pkey_tbl_len = 1;
@@ -572,12 +566,13 @@ static int eth_link_query_port(struct ib_device *ibdev, u8 port,
props->state = IB_PORT_DOWN;
props->phys_state = state_to_phys_state(props->state);
props->active_mtu = IB_MTU_256;
- if (is_bonded)
- rtnl_lock(); /* required to get upper dev */
spin_lock_bh(&iboe->lock);
ndev = iboe->netdevs[port - 1];
- if (ndev && is_bonded)
- ndev = netdev_master_upper_dev_get(ndev);
+ if (ndev && is_bonded) {
+ rcu_read_lock(); /* required to get upper dev */
+ ndev = netdev_master_upper_dev_get_rcu(ndev);
+ rcu_read_unlock();
+ }
if (!ndev)
goto out_unlock;
@@ -589,8 +584,6 @@ static int eth_link_query_port(struct ib_device *ibdev, u8 port,
props->phys_state = state_to_phys_state(props->state);
out_unlock:
spin_unlock_bh(&iboe->lock);
- if (is_bonded)
- rtnl_unlock();
out:
mlx4_free_cmd_mailbox(mdev->dev, mailbox);
return err;
@@ -673,23 +666,21 @@ out:
return err;
}
-static int iboe_query_gid(struct ib_device *ibdev, u8 port, int index,
- union ib_gid *gid)
-{
- struct mlx4_ib_dev *dev = to_mdev(ibdev);
-
- *gid = dev->iboe.gid_table[port - 1][index];
-
- return 0;
-}
-
static int mlx4_ib_query_gid(struct ib_device *ibdev, u8 port, int index,
union ib_gid *gid)
{
- if (rdma_port_get_link_layer(ibdev, port) == IB_LINK_LAYER_INFINIBAND)
+ int ret;
+
+ if (ib_cache_use_roce_gid_table(ibdev, port))
return __mlx4_ib_query_gid(ibdev, port, index, gid, 0);
- else
- return iboe_query_gid(ibdev, port, index, gid);
+
+ ret = ib_get_cached_gid(ibdev, port, index, gid, NULL);
+ if (ret == -EAGAIN) {
+ memcpy(gid, &zgid, sizeof(*gid));
+ return 0;
+ }
+
+ return ret;
}
int __mlx4_ib_query_pkey(struct ib_device *ibdev, u8 port, u16 index,
@@ -1686,273 +1677,6 @@ static struct device_attribute *mlx4_class_attributes[] = {
&dev_attr_board_id
};
-static void mlx4_addrconf_ifid_eui48(u8 *eui, u16 vlan_id,
- struct net_device *dev)
-{
- memcpy(eui, dev->dev_addr, 3);
- memcpy(eui + 5, dev->dev_addr + 3, 3);
- if (vlan_id < 0x1000) {
- eui[3] = vlan_id >> 8;
- eui[4] = vlan_id & 0xff;
- } else {
- eui[3] = 0xff;
- eui[4] = 0xfe;
- }
- eui[0] ^= 2;
-}
-
-static void update_gids_task(struct work_struct *work)
-{
- struct update_gid_work *gw = container_of(work, struct update_gid_work, work);
- struct mlx4_cmd_mailbox *mailbox;
- union ib_gid *gids;
- int err;
- struct mlx4_dev *dev = gw->dev->dev;
- int is_bonded = mlx4_is_bonded(dev);
-
- if (!gw->dev->ib_active)
- return;
-
- mailbox = mlx4_alloc_cmd_mailbox(dev);
- if (IS_ERR(mailbox)) {
- pr_warn("update gid table failed %ld\n", PTR_ERR(mailbox));
- return;
- }
-
- gids = mailbox->buf;
- memcpy(gids, gw->gids, sizeof gw->gids);
-
- err = mlx4_cmd(dev, mailbox->dma, MLX4_SET_PORT_GID_TABLE << 8 | gw->port,
- MLX4_SET_PORT_ETH_OPCODE, MLX4_CMD_SET_PORT,
- MLX4_CMD_TIME_CLASS_B, MLX4_CMD_WRAPPED);
- if (err)
- pr_warn("set port command failed\n");
- else
- if ((gw->port == 1) || !is_bonded)
- mlx4_ib_dispatch_event(gw->dev,
- is_bonded ? 1 : gw->port,
- IB_EVENT_GID_CHANGE);
-
- mlx4_free_cmd_mailbox(dev, mailbox);
- kfree(gw);
-}
-
-static void reset_gids_task(struct work_struct *work)
-{
- struct update_gid_work *gw =
- container_of(work, struct update_gid_work, work);
- struct mlx4_cmd_mailbox *mailbox;
- union ib_gid *gids;
- int err;
- struct mlx4_dev *dev = gw->dev->dev;
-
- if (!gw->dev->ib_active)
- return;
-
- mailbox = mlx4_alloc_cmd_mailbox(dev);
- if (IS_ERR(mailbox)) {
- pr_warn("reset gid table failed\n");
- goto free;
- }
-
- gids = mailbox->buf;
- memcpy(gids, gw->gids, sizeof(gw->gids));
-
- if (mlx4_ib_port_link_layer(&gw->dev->ib_dev, gw->port) ==
- IB_LINK_LAYER_ETHERNET) {
- err = mlx4_cmd(dev, mailbox->dma,
- MLX4_SET_PORT_GID_TABLE << 8 | gw->port,
- MLX4_SET_PORT_ETH_OPCODE, MLX4_CMD_SET_PORT,
- MLX4_CMD_TIME_CLASS_B,
- MLX4_CMD_WRAPPED);
- if (err)
- pr_warn(KERN_WARNING
- "set port %d command failed\n", gw->port);
- }
-
- mlx4_free_cmd_mailbox(dev, mailbox);
-free:
- kfree(gw);
-}
-
-static int update_gid_table(struct mlx4_ib_dev *dev, int port,
- union ib_gid *gid, int clear,
- int default_gid)
-{
- struct update_gid_work *work;
- int i;
- int need_update = 0;
- int free = -1;
- int found = -1;
- int max_gids;
-
- if (default_gid) {
- free = 0;
- } else {
- max_gids = dev->dev->caps.gid_table_len[port];
- for (i = 1; i < max_gids; ++i) {
- if (!memcmp(&dev->iboe.gid_table[port - 1][i], gid,
- sizeof(*gid)))
- found = i;
-
- if (clear) {
- if (found >= 0) {
- need_update = 1;
- dev->iboe.gid_table[port - 1][found] =
- zgid;
- break;
- }
- } else {
- if (found >= 0)
- break;
-
- if (free < 0 &&
- !memcmp(&dev->iboe.gid_table[port - 1][i],
- &zgid, sizeof(*gid)))
- free = i;
- }
- }
- }
-
- if (found == -1 && !clear && free >= 0) {
- dev->iboe.gid_table[port - 1][free] = *gid;
- need_update = 1;
- }
-
- if (!need_update)
- return 0;
-
- work = kzalloc(sizeof(*work), GFP_ATOMIC);
- if (!work)
- return -ENOMEM;
-
- memcpy(work->gids, dev->iboe.gid_table[port - 1], sizeof(work->gids));
- INIT_WORK(&work->work, update_gids_task);
- work->port = port;
- work->dev = dev;
- queue_work(wq, &work->work);
-
- return 0;
-}
-
-static void mlx4_make_default_gid(struct net_device *dev, union ib_gid *gid)
-{
- gid->global.subnet_prefix = cpu_to_be64(0xfe80000000000000LL);
- mlx4_addrconf_ifid_eui48(&gid->raw[8], 0xffff, dev);
-}
-
-
-static int reset_gid_table(struct mlx4_ib_dev *dev, u8 port)
-{
- struct update_gid_work *work;
-
- work = kzalloc(sizeof(*work), GFP_ATOMIC);
- if (!work)
- return -ENOMEM;
-
- memset(dev->iboe.gid_table[port - 1], 0, sizeof(work->gids));
- memset(work->gids, 0, sizeof(work->gids));
- INIT_WORK(&work->work, reset_gids_task);
- work->dev = dev;
- work->port = port;
- queue_work(wq, &work->work);
- return 0;
-}
-
-static int mlx4_ib_addr_event(int event, struct net_device *event_netdev,
- struct mlx4_ib_dev *ibdev, union ib_gid *gid)
-{
- struct mlx4_ib_iboe *iboe;
- int port = 0;
- struct net_device *real_dev = rdma_vlan_dev_real_dev(event_netdev) ?
- rdma_vlan_dev_real_dev(event_netdev) :
- event_netdev;
- union ib_gid default_gid;
-
- mlx4_make_default_gid(real_dev, &default_gid);
-
- if (!memcmp(gid, &default_gid, sizeof(*gid)))
- return 0;
-
- if (event != NETDEV_DOWN && event != NETDEV_UP)
- return 0;
-
- if ((real_dev != event_netdev) &&
- (event == NETDEV_DOWN) &&
- rdma_link_local_addr((struct in6_addr *)gid))
- return 0;
-
- iboe = &ibdev->iboe;
- spin_lock_bh(&iboe->lock);
-
- for (port = 1; port <= ibdev->dev->caps.num_ports; ++port)
- if ((netif_is_bond_master(real_dev) &&
- (real_dev == iboe->masters[port - 1])) ||
- (!netif_is_bond_master(real_dev) &&
- (real_dev == iboe->netdevs[port - 1])))
- update_gid_table(ibdev, port, gid,
- event == NETDEV_DOWN, 0);
-
- spin_unlock_bh(&iboe->lock);
- return 0;
-
-}
-
-static u8 mlx4_ib_get_dev_port(struct net_device *dev,
- struct mlx4_ib_dev *ibdev)
-{
- u8 port = 0;
- struct mlx4_ib_iboe *iboe;
- struct net_device *real_dev = rdma_vlan_dev_real_dev(dev) ?
- rdma_vlan_dev_real_dev(dev) : dev;
-
- iboe = &ibdev->iboe;
-
- for (port = 1; port <= ibdev->dev->caps.num_ports; ++port)
- if ((netif_is_bond_master(real_dev) &&
- (real_dev == iboe->masters[port - 1])) ||
- (!netif_is_bond_master(real_dev) &&
- (real_dev == iboe->netdevs[port - 1])))
- break;
-
- if ((port == 0) || (port > ibdev->dev->caps.num_ports))
- return 0;
- else
- return port;
-}
-
-static int mlx4_ib_inet_event(struct notifier_block *this, unsigned long event,
- void *ptr)
-{
- struct mlx4_ib_dev *ibdev;
- struct in_ifaddr *ifa = ptr;
- union ib_gid gid;
- struct net_device *event_netdev = ifa->ifa_dev->dev;
-
- ipv6_addr_set_v4mapped(ifa->ifa_address, (struct in6_addr *)&gid);
-
- ibdev = container_of(this, struct mlx4_ib_dev, iboe.nb_inet);
-
- mlx4_ib_addr_event(event, event_netdev, ibdev, &gid);
- return NOTIFY_DONE;
-}
-
-#if IS_ENABLED(CONFIG_IPV6)
-static int mlx4_ib_inet6_event(struct notifier_block *this, unsigned long event,
- void *ptr)
-{
- struct mlx4_ib_dev *ibdev;
- struct inet6_ifaddr *ifa = ptr;
- union ib_gid *gid = (union ib_gid *)&ifa->addr;
- struct net_device *event_netdev = ifa->idev->dev;
-
- ibdev = container_of(this, struct mlx4_ib_dev, iboe.nb_inet6);
-
- mlx4_ib_addr_event(event, event_netdev, ibdev, gid);
- return NOTIFY_DONE;
-}
-#endif
-
#define MLX4_IB_INVALID_MAC ((u64)-1)
static void mlx4_ib_update_qps(struct mlx4_ib_dev *ibdev,
struct net_device *dev,
@@ -2011,94 +1735,6 @@ unlock:
mutex_unlock(&ibdev->qp1_proxy_lock[port - 1]);
}
-static void mlx4_ib_get_dev_addr(struct net_device *dev,
- struct mlx4_ib_dev *ibdev, u8 port)
-{
- struct in_device *in_dev;
-#if IS_ENABLED(CONFIG_IPV6)
- struct inet6_dev *in6_dev;
- union ib_gid *pgid;
- struct inet6_ifaddr *ifp;
- union ib_gid default_gid;
-#endif
- union ib_gid gid;
-
-
- if ((port == 0) || (port > ibdev->dev->caps.num_ports))
- return;
-
- /* IPv4 gids */
- in_dev = in_dev_get(dev);
- if (in_dev) {
- for_ifa(in_dev) {
- /*ifa->ifa_address;*/
- ipv6_addr_set_v4mapped(ifa->ifa_address,
- (struct in6_addr *)&gid);
- update_gid_table(ibdev, port, &gid, 0, 0);
- }
- endfor_ifa(in_dev);
- in_dev_put(in_dev);
- }
-#if IS_ENABLED(CONFIG_IPV6)
- mlx4_make_default_gid(dev, &default_gid);
- /* IPv6 gids */
- in6_dev = in6_dev_get(dev);
- if (in6_dev) {
- read_lock_bh(&in6_dev->lock);
- list_for_each_entry(ifp, &in6_dev->addr_list, if_list) {
- pgid = (union ib_gid *)&ifp->addr;
- if (!memcmp(pgid, &default_gid, sizeof(*pgid)))
- continue;
- update_gid_table(ibdev, port, pgid, 0, 0);
- }
- read_unlock_bh(&in6_dev->lock);
- in6_dev_put(in6_dev);
- }
-#endif
-}
-
-static void mlx4_ib_set_default_gid(struct mlx4_ib_dev *ibdev,
- struct net_device *dev, u8 port)
-{
- union ib_gid gid;
- mlx4_make_default_gid(dev, &gid);
- update_gid_table(ibdev, port, &gid, 0, 1);
-}
-
-static int mlx4_ib_init_gid_table(struct mlx4_ib_dev *ibdev)
-{
- struct net_device *dev;
- struct mlx4_ib_iboe *iboe = &ibdev->iboe;
- int i;
- int err = 0;
-
- for (i = 1; i <= ibdev->num_ports; ++i) {
- if (rdma_port_get_link_layer(&ibdev->ib_dev, i) ==
- IB_LINK_LAYER_ETHERNET) {
- err = reset_gid_table(ibdev, i);
- if (err)
- goto out;
- }
- }
-
- read_lock(&dev_base_lock);
- spin_lock_bh(&iboe->lock);
-
- for_each_netdev(&init_net, dev) {
- u8 port = mlx4_ib_get_dev_port(dev, ibdev);
- /* port will be non-zero only for ETH ports */
- if (port) {
- mlx4_ib_set_default_gid(ibdev, dev, port);
- mlx4_ib_get_dev_addr(dev, ibdev, port);
- }
- }
-
- spin_unlock_bh(&iboe->lock);
- read_unlock(&dev_base_lock);
-out:
- return err;
-}
-
static void mlx4_ib_scan_netdevs(struct mlx4_ib_dev *ibdev,
struct net_device *dev,
unsigned long event)
@@ -2108,81 +1744,22 @@ static void mlx4_ib_scan_netdevs(struct mlx4_ib_dev *ibdev,
int update_qps_port = -1;
int port;
+ ASSERT_RTNL();
+
iboe = &ibdev->iboe;
spin_lock_bh(&iboe->lock);
mlx4_foreach_ib_transport_port(port, ibdev->dev) {
- enum ib_port_state port_state = IB_PORT_NOP;
- struct net_device *old_master = iboe->masters[port - 1];
- struct net_device *curr_netdev;
- struct net_device *curr_master;
iboe->netdevs[port - 1] =
mlx4_get_protocol_dev(ibdev->dev, MLX4_PROT_ETH, port);
- if (iboe->netdevs[port - 1])
- mlx4_ib_set_default_gid(ibdev,
- iboe->netdevs[port - 1], port);
- curr_netdev = iboe->netdevs[port - 1];
-
- if (iboe->netdevs[port - 1] &&
- netif_is_bond_slave(iboe->netdevs[port - 1])) {
- iboe->masters[port - 1] = netdev_master_upper_dev_get(
- iboe->netdevs[port - 1]);
- } else {
- iboe->masters[port - 1] = NULL;
- }
- curr_master = iboe->masters[port - 1];
if (dev == iboe->netdevs[port - 1] &&
(event == NETDEV_CHANGEADDR || event == NETDEV_REGISTER ||
event == NETDEV_UP || event == NETDEV_CHANGE))
update_qps_port = port;
- if (curr_netdev) {
- port_state = (netif_running(curr_netdev) && netif_carrier_ok(curr_netdev)) ?
- IB_PORT_ACTIVE : IB_PORT_DOWN;
- mlx4_ib_set_default_gid(ibdev, curr_netdev, port);
- if (curr_master) {
- /* if using bonding/team and a slave port is down, we
- * don't want the bond IP based gids in the table since
- * flows that select port by gid may get the down port.
- */
- if (port_state == IB_PORT_DOWN &&
- !mlx4_is_bonded(ibdev->dev)) {
- reset_gid_table(ibdev, port);
- mlx4_ib_set_default_gid(ibdev,
- curr_netdev,
- port);
- } else {
- /* gids from the upper dev (bond/team)
- * should appear in port's gid table
- */
- mlx4_ib_get_dev_addr(curr_master,
- ibdev, port);
- }
- }
- /* if bonding is used it is possible that we add it to
- * masters only after IP address is assigned to the
- * net bonding interface.
- */
- if (curr_master && (old_master != curr_master)) {
- reset_gid_table(ibdev, port);
- mlx4_ib_set_default_gid(ibdev,
- curr_netdev, port);
- mlx4_ib_get_dev_addr(curr_master, ibdev, port);
- }
-
- if (!curr_master && (old_master != curr_master)) {
- reset_gid_table(ibdev, port);
- mlx4_ib_set_default_gid(ibdev,
- curr_netdev, port);
- mlx4_ib_get_dev_addr(curr_netdev, ibdev, port);
- }
- } else {
- reset_gid_table(ibdev, port);
- }
}
-
spin_unlock_bh(&iboe->lock);
if (update_qps_port > 0)
@@ -2365,6 +1942,8 @@ static void *mlx4_ib_add(struct mlx4_dev *dev)
1 : ibdev->num_ports;
ibdev->ib_dev.num_comp_vectors = dev->caps.num_comp_vectors;
ibdev->ib_dev.dma_device = &dev->persist->pdev->dev;
+ ibdev->ib_dev.get_netdev = mlx4_ib_get_netdev;
+ ibdev->ib_dev.modify_gid = mlx4_ib_modify_gid;
if (dev->caps.userspace_caps)
ibdev->ib_dev.uverbs_abi_ver = MLX4_IB_UVERBS_ABI_VERSION;
@@ -2558,26 +2137,6 @@ static void *mlx4_ib_add(struct mlx4_dev *dev)
goto err_notif;
}
}
- if (!iboe->nb_inet.notifier_call) {
- iboe->nb_inet.notifier_call = mlx4_ib_inet_event;
- err = register_inetaddr_notifier(&iboe->nb_inet);
- if (err) {
- iboe->nb_inet.notifier_call = NULL;
- goto err_notif;
- }
- }
-#if IS_ENABLED(CONFIG_IPV6)
- if (!iboe->nb_inet6.notifier_call) {
- iboe->nb_inet6.notifier_call = mlx4_ib_inet6_event;
- err = register_inet6addr_notifier(&iboe->nb_inet6);
- if (err) {
- iboe->nb_inet6.notifier_call = NULL;
- goto err_notif;
- }
- }
-#endif
- if (mlx4_ib_init_gid_table(ibdev))
- goto err_notif;
}
for (j = 0; j < ARRAY_SIZE(mlx4_class_attributes); ++j) {
@@ -2608,18 +2167,6 @@ err_notif:
pr_warn("failure unregistering notifier\n");
ibdev->iboe.nb.notifier_call = NULL;
}
- if (ibdev->iboe.nb_inet.notifier_call) {
- if (unregister_inetaddr_notifier(&ibdev->iboe.nb_inet))
- pr_warn("failure unregistering notifier\n");
- ibdev->iboe.nb_inet.notifier_call = NULL;
- }
-#if IS_ENABLED(CONFIG_IPV6)
- if (ibdev->iboe.nb_inet6.notifier_call) {
- if (unregister_inet6addr_notifier(&ibdev->iboe.nb_inet6))
- pr_warn("failure unregistering notifier\n");
- ibdev->iboe.nb_inet6.notifier_call = NULL;
- }
-#endif
flush_workqueue(wq);
mlx4_ib_close_sriov(ibdev);
@@ -2743,19 +2290,6 @@ static void mlx4_ib_remove(struct mlx4_dev *dev, void *ibdev_ptr)
kfree(ibdev->ib_uc_qpns_bitmap);
}
- if (ibdev->iboe.nb_inet.notifier_call) {
- if (unregister_inetaddr_notifier(&ibdev->iboe.nb_inet))
- pr_warn("failure unregistering notifier\n");
- ibdev->iboe.nb_inet.notifier_call = NULL;
- }
-#if IS_ENABLED(CONFIG_IPV6)
- if (ibdev->iboe.nb_inet6.notifier_call) {
- if (unregister_inet6addr_notifier(&ibdev->iboe.nb_inet6))
- pr_warn("failure unregistering notifier\n");
- ibdev->iboe.nb_inet6.notifier_call = NULL;
- }
-#endif
-
iounmap(ibdev->uar_map);
for (p = 0; p < ibdev->num_ports; ++p)
if (ibdev->counters[p] != -1)
diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
index 0a32552..d6ce6a1 100644
--- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
+++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
@@ -474,12 +474,8 @@ struct mlx4_port_gid_table {
struct mlx4_ib_iboe {
spinlock_t lock;
struct net_device *netdevs[MLX4_MAX_PORTS];
- struct net_device *masters[MLX4_MAX_PORTS];
atomic64_t mac[MLX4_MAX_PORTS];
struct notifier_block nb;
- struct notifier_block nb_inet;
- struct notifier_block nb_inet6;
- union ib_gid gid_table[MLX4_MAX_PORTS][128];
struct mlx4_port_gid_table gids[MLX4_MAX_PORTS];
};
diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c
index 37305f5..a6f44ba 100644
--- a/drivers/infiniband/hw/mlx4/qp.c
+++ b/drivers/infiniband/hw/mlx4/qp.c
@@ -1292,14 +1292,18 @@ static int _mlx4_set_path(struct mlx4_ib_dev *dev, const struct ib_ah_attr *ah,
path->static_rate = 0;
if (ah->ah_flags & IB_AH_GRH) {
- if (ah->grh.sgid_index >= dev->dev->caps.gid_table_len[port]) {
+ int real_sgid_index = mlx4_ib_gid_index_to_real_index(dev,
+ port,
+ ah->grh.sgid_index);
+
+ if (real_sgid_index >= dev->dev->caps.gid_table_len[port]) {
pr_err("sgid_index (%u) too large. max is %d\n",
- ah->grh.sgid_index, dev->dev->caps.gid_table_len[port] - 1);
+ real_sgid_index, dev->dev->caps.gid_table_len[port] - 1);
return -1;
}
path->grh_mylmc |= 1 << 7;
- path->mgid_index = ah->grh.sgid_index;
+ path->mgid_index = real_sgid_index;
path->hop_limit = ah->grh.hop_limit;
path->tclass_flowlabel =
cpu_to_be32((ah->grh.traffic_class << 20) |
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 02/14] IB/core: Replace device_mutex with rwsem
[not found] ` <1432045637-9090-3-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-05-19 17:36 ` Jason Gunthorpe
[not found] ` <20150519173647.GA18675-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Jason Gunthorpe @ 2015-05-19 17:36 UTC (permalink / raw)
To: Matan Barak
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Sean Hefty
On Tue, May 19, 2015 at 05:27:05PM +0300, Matan Barak wrote:
> device_mutex is replaced rwsem in order to allow
> clients' work to run concurrently with register_device,
> unregister_device, register_client and unregister_client.
Can you guys work together? Why are there two different versions of
these patches now?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 04/14] IB/core: Add default GID for RoCE GID table
[not found] ` <1432045637-9090-5-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-05-19 17:41 ` Jason Gunthorpe
[not found] ` <20150519174159.GB18675-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Jason Gunthorpe @ 2015-05-19 17:41 UTC (permalink / raw)
To: Matan Barak
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Sean Hefty
On Tue, May 19, 2015 at 05:27:07PM +0300, Matan Barak wrote:
> When RoCE is used, a default GID address should be generated
> for every supported RoCE type. These default GID addresses are
> generated based on the IPv6 link-local address, but in contrast
> to the GID based on the regular IPv6 link-local (as we generate
> GID per IP address), these GIDs are also available if the net
> device is down (in order to support loopback).
> Moreover, these default GID addresses can't be deleted.
That addrconf stuff needs to be in a dedicated patch
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API
[not found] ` <1432045637-9090-8-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-05-19 18:06 ` Jason Gunthorpe
[not found] ` <20150519180649.GC18675-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Jason Gunthorpe @ 2015-05-19 18:06 UTC (permalink / raw)
To: Matan Barak
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Or Gerlitz,
Moni Shoua, Somnath Kotur, Sean Hefty
On Tue, May 19, 2015 at 05:27:10PM +0300, Matan Barak wrote:
> +#define __IB_ONLY
What is this?
> +/**
> + * ib_cache_use_roce_gid_table - Returns whether the device uses roce gid table
> + * @device: The device to query
> + * @port_num: The port number of the device to query.
> + *
> + * ib_cache_use_roce_gid_table() returns 0 if this port uses the roce_gid_table
> + * to store GIDs and error otherwise.
> + */
> +int ib_cache_use_roce_gid_table(struct ib_device *device, u8 port_num);
This needs to be in the same place and format as the new items from
Michael's patch set. In fact this whole thing will need to be rebased
ontop of it.
> /**
> * ib_find_cached_gid - Returns the port number and GID table index where
> * a specified GID value occurs.
> * @device: The device to query.
> * @gid: The GID value to search for.
> + * @gid_type: The GID type to search for.
> + * @net: In RoCE, the namespace of the device.
> + * @if_index: In RoCE, the if_index of the device. Zero means ignore.
> * @port_num: The port number of the device where the GID value was found.
> * @index: The index into the cached GID table where the GID was found. This
> * parameter may be NULL.
> @@ -66,10 +82,36 @@ int ib_get_cached_gid(struct ib_device *device,
> */
> int ib_find_cached_gid(struct ib_device *device,
> union ib_gid *gid,
> + enum ib_gid_type gid_type,
> + struct net *net,
> + int if_index,
> u8 *port_num,
> u16 *index);
Why are we adding net namespace stuff here? We don't even have a
proposal for roce net namespaces. Same comment for all net adds.
This patch is similarly muddled with the gid_type, we don't have
rocev2 in this series, so why do we have gid_type being introduced
*at all*?
I'm not certain you should be changing these existing APIs like this,
it doesn't make alot of sense to change a signature then go around to
all callers and not make any use of the additional parameters.
Adding a roce specific call might make more sense, I'm assuming it is
rarely needed?
> int ib_query_gid(struct ib_device *device,
> - u8 port_num, int index, union ib_gid *gid);
> + u8 port_num, int index, union ib_gid *gid,
> + struct ib_gid_attr *attr);
>
> int ib_query_pkey(struct ib_device *device,
> u8 port_num, u16 index, u16 *pkey);
> @@ -1819,7 +1821,8 @@ int ib_modify_port(struct ib_device *device,
> struct ib_port_modify *port_modify);
>
> int ib_find_gid(struct ib_device *device, union ib_gid *gid,
> - u8 *port_num, u16 *index);
> + enum ib_gid_type gid_type, struct net *net,
> + int if_index, u8 *port_num, u16 *index);
None of this makes sense unless the device is roce, again, it probably
should have a roce specific call.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 02/14] IB/core: Replace device_mutex with rwsem
[not found] ` <20150519173647.GA18675-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
@ 2015-05-20 16:07 ` Matan Barak
0 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-20 16:07 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Matan Barak, Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
Or Gerlitz, Moni Shoua, Somnath Kotur, Sean Hefty
On Tue, May 19, 2015 at 8:36 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Tue, May 19, 2015 at 05:27:05PM +0300, Matan Barak wrote:
>> device_mutex is replaced rwsem in order to allow
>> clients' work to run concurrently with register_device,
>> unregister_device, register_client and unregister_client.
>
> Can you guys work together? Why are there two different versions of
> these patches now?
I missed Haggai's v4 submission.
Anyway, they are both based on your idea of splitting the locking into
a mutex and rwsem, so each is a valid candidate.
Matan
>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 04/14] IB/core: Add default GID for RoCE GID table
[not found] ` <20150519174159.GB18675-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
@ 2015-05-20 16:09 ` Matan Barak
0 siblings, 0 replies; 28+ messages in thread
From: Matan Barak @ 2015-05-20 16:09 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Matan Barak, Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
Or Gerlitz, Moni Shoua, Somnath Kotur, Sean Hefty
On Tue, May 19, 2015 at 8:41 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Tue, May 19, 2015 at 05:27:07PM +0300, Matan Barak wrote:
>> When RoCE is used, a default GID address should be generated
>> for every supported RoCE type. These default GID addresses are
>> generated based on the IPv6 link-local address, but in contrast
>> to the GID based on the regular IPv6 link-local (as we generate
>> GID per IP address), these GIDs are also available if the net
>> device is down (in order to support loopback).
>> Moreover, these default GID addresses can't be deleted.
>
> That addrconf stuff needs to be in a dedicated patch
Ok, I'll split this patch.
Thanks for taking a look at this patch.
Matan
>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API
[not found] ` <20150519180649.GC18675-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
@ 2015-05-20 16:27 ` Matan Barak
[not found] ` <CAAKD3BAifeMcHbDwcaK7F6cjj=mAn3W0Ms2mOmoXDhaquFqKTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Matan Barak @ 2015-05-20 16:27 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Matan Barak, Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
Or Gerlitz, Moni Shoua, Somnath Kotur, Sean Hefty
On Tue, May 19, 2015 at 9:06 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Tue, May 19, 2015 at 05:27:10PM +0300, Matan Barak wrote:
>
>> +#define __IB_ONLY
>
> What is this?
>
I annotated functions that should be called with IB link layer only as
__IB_ONLY. I think it's make the code more readable and it could be
verified by tools/scripts (like coccinelle) that we're calling those
functions under an IB link layer if statement.
>> +/**
>> + * ib_cache_use_roce_gid_table - Returns whether the device uses roce gid table
>> + * @device: The device to query
>> + * @port_num: The port number of the device to query.
>> + *
>> + * ib_cache_use_roce_gid_table() returns 0 if this port uses the roce_gid_table
>> + * to store GIDs and error otherwise.
>> + */
>> +int ib_cache_use_roce_gid_table(struct ib_device *device, u8 port_num);
>
> This needs to be in the same place and format as the new items from
> Michael's patch set. In fact this whole thing will need to be rebased
> ontop of it.
>
I'll take a look Michael's patch set more carefully and change that accordingly.
>> /**
>> * ib_find_cached_gid - Returns the port number and GID table index where
>> * a specified GID value occurs.
>> * @device: The device to query.
>> * @gid: The GID value to search for.
>> + * @gid_type: The GID type to search for.
>> + * @net: In RoCE, the namespace of the device.
>> + * @if_index: In RoCE, the if_index of the device. Zero means ignore.
>> * @port_num: The port number of the device where the GID value was found.
>> * @index: The index into the cached GID table where the GID was found. This
>> * parameter may be NULL.
>> @@ -66,10 +82,36 @@ int ib_get_cached_gid(struct ib_device *device,
>> */
>> int ib_find_cached_gid(struct ib_device *device,
>> union ib_gid *gid,
>> + enum ib_gid_type gid_type,
>> + struct net *net,
>> + int if_index,
>> u8 *port_num,
>> u16 *index);
>
> Why are we adding net namespace stuff here? We don't even have a
> proposal for roce net namespaces. Same comment for all net adds.
>
In order to get a netdev, you need to have if_index and namespace.
I prefer to introduce a correct API from the beginning than to assume
we only use &init_net and changing the API afterwards.
roce_gid_table and ib_cache should present a clean API no matter
if their consumers use a custom namespace or init_net.
> This patch is similarly muddled with the gid_type, we don't have
> rocev2 in this series, so why do we have gid_type being introduced
> *at all*?
>
> I'm not certain you should be changing these existing APIs like this,
> it doesn't make alot of sense to change a signature then go around to
> all callers and not make any use of the additional parameters.
>
The alternative is to change this API all over again when RoCE v2 support
is added. In addition, roce_gid_table as an infrastructure is able to manage
multiple GID types, so the API should encapsulate the type you want to
search for.
> Adding a roce specific call might make more sense, I'm assuming it is
> rarely needed?
>
I don't think so. Consumers of this API usually don't care which link layer is
used. Asking them to query the link layer and choose the right function
just breaks the abstraction.
>> int ib_query_gid(struct ib_device *device,
>> - u8 port_num, int index, union ib_gid *gid);
>> + u8 port_num, int index, union ib_gid *gid,
>> + struct ib_gid_attr *attr);
>>
>> int ib_query_pkey(struct ib_device *device,
>> u8 port_num, u16 index, u16 *pkey);
>> @@ -1819,7 +1821,8 @@ int ib_modify_port(struct ib_device *device,
>> struct ib_port_modify *port_modify);
>>
>> int ib_find_gid(struct ib_device *device, union ib_gid *gid,
>> - u8 *port_num, u16 *index);
>> + enum ib_gid_type gid_type, struct net *net,
>> + int if_index, u8 *port_num, u16 *index);
>
> None of this makes sense unless the device is roce, again, it probably
> should have a roce specific call.
>
The key point here is abstraction. In IB, the path will contain NULL
in the namespace field and 0 in the if_index field, but the rest of
the consumers
code will remain the same. The path and the API is a superset of IB and RoCE.
Thanks for your comments.
Matan
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API
[not found] ` <CAAKD3BAifeMcHbDwcaK7F6cjj=mAn3W0Ms2mOmoXDhaquFqKTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-05-20 18:17 ` Jason Gunthorpe
[not found] ` <20150520181718.GD28496-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Jason Gunthorpe @ 2015-05-20 18:17 UTC (permalink / raw)
To: Matan Barak
Cc: Matan Barak, Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
Or Gerlitz, Moni Shoua, Somnath Kotur, Sean Hefty
On Wed, May 20, 2015 at 07:27:15PM +0300, Matan Barak wrote:
> On Tue, May 19, 2015 at 9:06 PM, Jason Gunthorpe
> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > On Tue, May 19, 2015 at 05:27:10PM +0300, Matan Barak wrote:
> >
> >> +#define __IB_ONLY
> >
> > What is this?
> >
>
> I annotated functions that should be called with IB link layer only as
> __IB_ONLY. I think it's make the code more readable and it could be
> verified by tools/scripts (like coccinelle) that we're calling those
> functions under an IB link layer if statement.
Why not add:
BUG_ON(!rdma_is_ib(...))
[or whatever the name ended up as]
To the function prolog?
Then it actually does something..
> >> + * @gid_type: The GID type to search for.
> >> + * @net: In RoCE, the namespace of the device.
> >> + * @if_index: In RoCE, the if_index of the device. Zero means ignore.
> >> * @port_num: The port number of the device where the GID value was found.
> >> * @index: The index into the cached GID table where the GID was found. This
> >> * parameter may be NULL.
> >> @@ -66,10 +82,36 @@ int ib_get_cached_gid(struct ib_device *device,
> >> */
> >> int ib_find_cached_gid(struct ib_device *device,
> >> union ib_gid *gid,
> >> + enum ib_gid_type gid_type,
> >> + struct net *net,
> >> + int if_index,
> >> u8 *port_num,
> >> u16 *index);
> >
> > Why are we adding net namespace stuff here? We don't even have a
> > proposal for roce net namespaces. Same comment for all net adds.
> >
>
> In order to get a netdev, you need to have if_index and namespace.
> I prefer to introduce a correct API from the beginning than to assume
> we only use &init_net and changing the API afterwards.
> roce_gid_table and ib_cache should present a clean API no matter
> if their consumers use a custom namespace or init_net.
No, this needs to be a cleanup series - get to the point where we are
sharing the gid table code between the drivers that need it.
Don't change the APIs speculatively for rocev2 or namespaces, those
series can deal with it on their own.
To be more clear, the goal of this series should only be to get
patches 11,12,13,14 merged, and the minimum set of other patches to
get there.
That means we don't need these API changes. We don't need ib_gid_attr.
Drop patch 8 and 9.
I'm also not sure about patch 10, that looks like a functional
change? It should not be in a cleanup series.
Is 6 new functionality? Are drivers already handling bonding?
A cleanup series should have *no functional change*.
So, it looks roughly like 7,8,9,10 can be dropped?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API
[not found] ` <20150520181718.GD28496-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
@ 2015-05-28 13:50 ` Matan Barak
[not found] ` <CAAKD3BDJtpEhfhsB=o4hc=ARWMt7JgpqvbjXJVvsL_vLZRn8yQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Matan Barak @ 2015-05-28 13:50 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Matan Barak, Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
Or Gerlitz, Moni Shoua, Somnath Kotur, Sean Hefty
On Wed, May 20, 2015 at 9:17 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Wed, May 20, 2015 at 07:27:15PM +0300, Matan Barak wrote:
>> On Tue, May 19, 2015 at 9:06 PM, Jason Gunthorpe
>> <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>> > On Tue, May 19, 2015 at 05:27:10PM +0300, Matan Barak wrote:
>> >
>> >> +#define __IB_ONLY
>> >
>> > What is this?
>> >
>>
>> I annotated functions that should be called with IB link layer only as
>> __IB_ONLY. I think it's make the code more readable and it could be
>> verified by tools/scripts (like coccinelle) that we're calling those
>> functions under an IB link layer if statement.
>
> Why not add:
>
> BUG_ON(!rdma_is_ib(...))
>
> [or whatever the name ended up as]
>
> To the function prolog?
>
> Then it actually does something..
>
I think static checks suffice here - they have a clear advantage by
forcing users to call these functions under rdma_is_ib if statement.
However, since static checks (still) aren't widely used by all
developers, I can change that to BUG_ON like you suggested.
>> >> + * @gid_type: The GID type to search for.
>> >> + * @net: In RoCE, the namespace of the device.
>> >> + * @if_index: In RoCE, the if_index of the device. Zero means ignore.
>> >> * @port_num: The port number of the device where the GID value was found.
>> >> * @index: The index into the cached GID table where the GID was found. This
>> >> * parameter may be NULL.
>> >> @@ -66,10 +82,36 @@ int ib_get_cached_gid(struct ib_device *device,
>> >> */
>> >> int ib_find_cached_gid(struct ib_device *device,
>> >> union ib_gid *gid,
>> >> + enum ib_gid_type gid_type,
>> >> + struct net *net,
>> >> + int if_index,
>> >> u8 *port_num,
>> >> u16 *index);
>> >
>> > Why are we adding net namespace stuff here? We don't even have a
>> > proposal for roce net namespaces. Same comment for all net adds.
>> >
>>
>> In order to get a netdev, you need to have if_index and namespace.
>> I prefer to introduce a correct API from the beginning than to assume
>> we only use &init_net and changing the API afterwards.
>> roce_gid_table and ib_cache should present a clean API no matter
>> if their consumers use a custom namespace or init_net.
>
> No, this needs to be a cleanup series - get to the point where we are
> sharing the gid table code between the drivers that need it.
>
> Don't change the APIs speculatively for rocev2 or namespaces, those
> series can deal with it on their own.
>
The argument for removing the gid_type seems reasonable to me.
However, I don't think we should be removing net.
if_index should always come with net - passing only if_index makes
roce_gid_table's API a bit broken.
> To be more clear, the goal of this series should only be to get
> patches 11,12,13,14 merged, and the minimum set of other patches to
> get there.
>
> That means we don't need these API changes. We don't need ib_gid_attr.
> Drop patch 8 and 9.
>
Patch 8 (the ndev part) is relevant. GID is now related to a ndev and
we would like
to expose this information to the user.
In non rdma-cm applications, how would a user select the gid_index he wants?
Patch 9 is used by 10, so if we don't drop 10 - we need 9.
> I'm also not sure about patch 10, that looks like a functional
> change? It should not be in a cleanup series.
>
We drop smac and vlan_id from path and av. Instead, it's taken
from the netdev in roce_gid_table. That's a crucial part of the new
architecture.
It's not worth adding roce_gid_table and not taking the parameters
based on its information.
> Is 6 new functionality? Are drivers already handling bonding?
>
The mlx4 driver currently supports bonding - dropping this patch means
functionality regression.
> A cleanup series should have *no functional change*.
>
> So, it looks roughly like 7,8,9,10 can be dropped?
>
So if we sum everything up -
do you agree with:
Patch 7 - Change to BUG_ON, delete gid_type from API
Patch 8 - delete gid_type from sysfs
> Jason
Thanks for looking at this patch.
Matan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API
[not found] ` <CAAKD3BDJtpEhfhsB=o4hc=ARWMt7JgpqvbjXJVvsL_vLZRn8yQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-05-28 16:07 ` Jason Gunthorpe
[not found] ` <20150528160748.GC2962-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Jason Gunthorpe @ 2015-05-28 16:07 UTC (permalink / raw)
To: Matan Barak
Cc: Matan Barak, Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA,
Or Gerlitz, Moni Shoua, Somnath Kotur, Sean Hefty
On Thu, May 28, 2015 at 04:50:09PM +0300, Matan Barak wrote:
> The argument for removing the gid_type seems reasonable to me.
> However, I don't think we should be removing net.
> if_index should always come with net - passing only if_index makes
> roce_gid_table's API a bit broken.
Well, get rid of if_index too? Isn't all of that rocev2 stuff? Todays
roce drivers do not need if_index at this point to do gid index lookup.
It isn't clear to me why that should change in a refactoring exercise.
> Patch 8 (the ndev part) is relevant. GID is now related to a ndev and
> we would like
> to expose this information to the user.
> In non rdma-cm applications, how would a user select the gid_index he wants?
I don't mean drop forever, I mean, concentrate on getting this clean
up done, then start discussing UAPI changes separately. Please don't
bury UAPI changes, new features, etc in a cleanup patch series.
> > I'm also not sure about patch 10, that looks like a functional
> > change? It should not be in a cleanup series.
>
> We drop smac and vlan_id from path and av. Instead, it's taken
> from the netdev in roce_gid_table. That's a crucial part of the new
> architecture.
Again, I didn't say drop it forever, these patches belong in a later
series, perhaps as a precursor series to rocev2.
> It's not worth adding roce_gid_table and not taking the parameters
> based on its information.
It should be worth adding roce_gid_table because it factors duplicate
existing code out of several drivers. Get that merged, then deal with
adding rocev2, adding new UAPIs, etc.
If that isn't true then we have a problem :)
You started with a giagantic series that added new UAPIs, refactored
code, clean ups, added rocev2, and maybe more I haven't noticed. That
is too big to review. Make it smaller:
- Refactor the gid table code from the drivers - as is. Minimize
changes elsewhere
- Add rocev2 in stages
- New UAPIs for rocev2
- Anything else I missed.
Don't try and do so much at once.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API
[not found] ` <20150528160748.GC2962-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
@ 2015-05-28 16:34 ` Or Gerlitz
[not found] ` <5567439C.7080700-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Or Gerlitz @ 2015-05-28 16:34 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Matan Barak, Matan Barak, Doug Ledford,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Moni Shoua, Somnath Kotur,
Sean Hefty
On 5/28/2015 7:07 PM, Jason Gunthorpe wrote:
>> Patch 8 (the ndev part) is relevant. GID is now related to a ndev and
>> >we would like
>> >to expose this information to the user.
>> >In non rdma-cm applications, how would a user select the gid_index he wants?
> I don't mean drop forever, I mean, concentrate on getting this clean
> up done, then start discussing UAPI changes separately. Please don't
> bury UAPI changes, new features, etc in a cleanup patch series.
Jason,
I agree. This series can be perfectly made without UAPI changes, Matan
can drop patch #8 and have user-space to just work as they did before,
for both librdmacm and libibverbs consumers.
As for the RoCE GID table itself, adding in properly net-devices in
their native Linux kernel form, namely with if_index and name-space --
seems to me the correct way to go. This for itself goes just a bit
beyond refactoring, doesn't add special complexity which wasn't there
before and has the advantage of doing things right and solid.
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API
[not found] ` <5567439C.7080700-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
@ 2015-05-28 17:07 ` Jason Gunthorpe
[not found] ` <20150528170711.GA4345-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Jason Gunthorpe @ 2015-05-28 17:07 UTC (permalink / raw)
To: Or Gerlitz
Cc: Matan Barak, Matan Barak, Doug Ledford,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Moni Shoua, Somnath Kotur,
Sean Hefty
On Thu, May 28, 2015 at 07:34:36PM +0300, Or Gerlitz wrote:
> As for the RoCE GID table itself, adding in properly net-devices in their
> native Linux kernel form, namely with if_index and name-space -- seems to me
> the correct way to go.
Well, no, it is goofy. Callers with a path have a if_index/net pair.
Several other callers have an actual struct netdevice.
The first thing roce_gid_table_find_gid_by_port does is convert the
if_index/net pair back to a netdevice. This tells me the API is wrong,
callers that already have a netdevice should just pass it in, and path
is one that should be special cased to search.
But I don't even want to talk about that for a clean up series.
I want to know that the behavior of ib_find_cached_gid does not
change. If it hasn't changed then we don't need to change call
sites to pass in the optional netdev, because it must work as it does
today. Otherwise the cleanup is broken.
So, my specific advice is:
- roce_gid_table_find_gid_by_port should accept a netdevice or null
- Always pass null for the netdevice for all call sites
- Do not change the ib_* APIs during the clean up.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API
[not found] ` <20150528170711.GA4345-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
@ 2015-06-02 16:13 ` Matan Barak
[not found] ` <CAAKD3BALO6ts66Q0n-BVFz4Niu7L6jX0N-GHLXGp0DQ7Z7c9HA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 28+ messages in thread
From: Matan Barak @ 2015-06-02 16:13 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Or Gerlitz, Matan Barak, Doug Ledford,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Moni Shoua, Somnath Kotur,
Sean Hefty
On Thu, May 28, 2015 at 8:07 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Thu, May 28, 2015 at 07:34:36PM +0300, Or Gerlitz wrote:
>
>> As for the RoCE GID table itself, adding in properly net-devices in their
>> native Linux kernel form, namely with if_index and name-space -- seems to me
>> the correct way to go.
>
> Well, no, it is goofy. Callers with a path have a if_index/net pair.
> Several other callers have an actual struct netdevice.
>
> The first thing roce_gid_table_find_gid_by_port does is convert the
> if_index/net pair back to a netdevice. This tells me the API is wrong,
> callers that already have a netdevice should just pass it in, and path
> is one that should be special cased to search.
>
> But I don't even want to talk about that for a clean up series.
>
ib_find_cached_gid is used in cm_init_av_by_path. ndev is necessary
in order to find the correct GID in the GID table, as otherwise if two upper
devices has the same IP (hence GID), we might not found the bounded
device correctly.
Therefore, I suggest modifying ib_find_cached_gid (add netdevice),
In addition, vendors need to get the MAC of the ndev that relates to the GID
in order to output a correct Ethernet header. This is currently done though
ib_get_cached_gid. Without adding gid_attr, you get a buggy implementation
that upper devices with different HW addresses would all map to the same MAC.
> I want to know that the behavior of ib_find_cached_gid does not
> change. If it hasn't changed then we don't need to change call
> sites to pass in the optional netdev, because it must work as it does
> today. Otherwise the cleanup is broken.
>
Prior to those changes, the vendor resolved the MAC address twice.
The current patchset *cleans* this up - instead of resolving all over again
we map a GID entry from ndev.
This conforms to a cleanup series as *no new functionality is added*.
Modified API doesn't always imply new functionality but sometimes better design.
> So, my specific advice is:
>
> - roce_gid_table_find_gid_by_port should accept a netdevice or null
I agree
> - Always pass null for the netdevice for all call sites
I think we should stick with the path->net and path->if_index.
They are used to indicate the bounded device of the cma.
This device is then used in the ib_find_cached_gid.
> - Do not change the ib_* APIs during the clean up.
>
I think we should keep the modified
(a) ib_get_cached_gid (output gid_attr)
(b) ib_find_cached_gid (input ndev)
(c) ib_find_cached_gid_by_port (input ndev)
(d) ib_query_gid (output gid_attr)
> Jason
Thanks for your review and looking for your comments regarding this.
Matan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API
[not found] ` <CAAKD3BALO6ts66Q0n-BVFz4Niu7L6jX0N-GHLXGp0DQ7Z7c9HA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-06-02 17:39 ` Jason Gunthorpe
0 siblings, 0 replies; 28+ messages in thread
From: Jason Gunthorpe @ 2015-06-02 17:39 UTC (permalink / raw)
To: Matan Barak
Cc: Or Gerlitz, Matan Barak, Doug Ledford,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, Moni Shoua, Somnath Kotur,
Sean Hefty
On Tue, Jun 02, 2015 at 07:13:00PM +0300, Matan Barak wrote:
> ib_find_cached_gid is used in cm_init_av_by_path. ndev is necessary
> in order to find the correct GID in the GID table, as otherwise if two upper
> devices has the same IP (hence GID), we might not found the bounded
> device correctly.
But this is exactly what the current state of affairs is, right?
> In addition, vendors need to get the MAC of the ndev that relates to the GID
> in order to output a correct Ethernet header. This is currently done though
> ib_get_cached_gid. Without adding gid_attr, you get a buggy implementation
> that upper devices with different HW addresses would all map to the same MAC.
Again, that is current state of affairs, so this change is a bug fix.
And, we don't expect macvlan to work today with RDMA, so it isn't even
a bug fix, it is a feature add.
And.. given the patches I've seen for roce namespaces, I'm not even
convinced this is the right API approach.
So, save it for the roce namespace support discussion, and we can
figure out how to comprehsively deal with multiple netdevs and
roce.
Otherwise, keep with the current code's assumption that there is
on netdev for each roce device.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2015-06-02 17:39 UTC | newest]
Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-05-19 14:27 [PATCH v4 for-next 00/14] RoCE GID management Matan Barak
[not found] ` <1432045637-9090-1-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-05-19 14:27 ` [PATCH v4 for-next 01/14] IB/core: Add RoCE GID table Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 02/14] IB/core: Replace device_mutex with rwsem Matan Barak
[not found] ` <1432045637-9090-3-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-05-19 17:36 ` Jason Gunthorpe
[not found] ` <20150519173647.GA18675-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-05-20 16:07 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 03/14] IB/core: Add RoCE GID population Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 04/14] IB/core: Add default GID for RoCE GID table Matan Barak
[not found] ` <1432045637-9090-5-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-05-19 17:41 ` Jason Gunthorpe
[not found] ` <20150519174159.GB18675-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-05-20 16:09 ` Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 05/14] net: Add info for NETDEV_CHANGEUPPER event Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 06/14] IB/core: Add RoCE table bonding support Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 07/14] IB/core: GID attribute should be returned from verbs API and cache API Matan Barak
[not found] ` <1432045637-9090-8-git-send-email-matanb-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-05-19 18:06 ` Jason Gunthorpe
[not found] ` <20150519180649.GC18675-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-05-20 16:27 ` Matan Barak
[not found] ` <CAAKD3BAifeMcHbDwcaK7F6cjj=mAn3W0Ms2mOmoXDhaquFqKTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-05-20 18:17 ` Jason Gunthorpe
[not found] ` <20150520181718.GD28496-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-05-28 13:50 ` Matan Barak
[not found] ` <CAAKD3BDJtpEhfhsB=o4hc=ARWMt7JgpqvbjXJVvsL_vLZRn8yQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-05-28 16:07 ` Jason Gunthorpe
[not found] ` <20150528160748.GC2962-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-05-28 16:34 ` Or Gerlitz
[not found] ` <5567439C.7080700-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2015-05-28 17:07 ` Jason Gunthorpe
[not found] ` <20150528170711.GA4345-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
2015-06-02 16:13 ` Matan Barak
[not found] ` <CAAKD3BALO6ts66Q0n-BVFz4Niu7L6jX0N-GHLXGp0DQ7Z7c9HA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-06-02 17:39 ` Jason Gunthorpe
2015-05-19 14:27 ` [PATCH v4 for-next 08/14] IB/core: Report gid_type and gid_ndev through sysfs Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 09/14] IB/core: Support find sgid index using a filter function Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 10/14] IB/core: Modify ib_verbs and cma in order to use roce_gid_table Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 11/14] RDMA/ocrdma: Changes in driver to incorporate the moving of GID Table mgmt to IB/Core Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 12/14] net/mlx4: Postpone the registration of net_device Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 13/14] IB/mlx4: Implement ib_device callbacks Matan Barak
2015-05-19 14:27 ` [PATCH v4 for-next 14/14] IB/mlx4: Replace mechanism for RoCE GID management Matan Barak
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.